Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Fix KSTACK_PAGES issue when the default value was changed in KERNCONF

If KSTACK_PAGES was changed to anything alse than the default,
the value from param.h was taken instead in some places and
the value from KENRCONF in some others. This resulted in
inconsistency which caused corruption in SMP envorinment.

Ensure all places where KSTACK_PAGES are used the opt_kstack_pages.h
is included.

The file opt_kstack_pages.h could not be included in param.h
because was breaking the toolchain compilation.

Reviewed by: kib
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3094

Fix typo revealed by using newer binutils

Differential Revision: https://reviews.freebsd.org/D933
Reviewed by: marius

Add a global MD macro for the VIS block size instead of duplicating
it and using magic values all over the place.

MFC after: 1 week

Fully disable interrupts while we fiddle with the FP context in the
VIS-based block copy/zero implementations. While with 4BSD it's
sufficient to just disable the tick interrupts, with ULE+PREEMPTION
it's otherwise also possible that these are preempted via IPIs.

- Move the macros for generating load and store instructions to asmacros.h
so they can be shared by different source files and extend them by a
variant for atomic compare and swap.
- Consistently use EMPTY.

Add a VIS-based block copy function for SPARC64 V and later, which
additionally takes advantage of the prefetch cache of these CPUs.
Unlike the uncommitted US-III version, which provide no measurable
speedup or even resulted in a slight slowdown on certain CPUs models
compared to using the US-I version with these, the SPARC64 version
actually results in a slight improvement.

- The firmware of Sun Fire V1280 has a misfeature of setting %wstate to
7 which corresponds to WSTATE_KMIX in OpenSolaris whenever calling into
it which totally screws us even when restoring %wstate afterwards as
spill/fill traps can happen while in OFW. The rather hackish OpenBSD
approach of just setting the equivalent of WSTATE_KERNEL to 7 also is
no option as we treat %wstate as a bit field. So in order to deal with
this problem actually implement spill/fill handlers for %wstate 7 which
just act as the WSTATE_KERNEL ones except of theoretically also handling
32-bit, turn off interrupts completely so we don't even take IPIs while
in OFW which should ensure we only take spill/fill traps at most and
restore %wstate after calling into OFW once we have taken over the trap
table. While at it, actually set WSTATE_{,PROM}_KMIX before calling into
OFW just like OpenSolaris does, which should at least help testing this
change on non-V1280.
- Remove comments referring to the %wstate usage in BSD/OS.
- Remove the no longer used RSF_ALIGN_RETRY macro.
- Correct some trap table addresses in comments.
- Ensure %wstate is set to WSTATE_KERNEL when taking over the trap table.
- Ensure PSTATE_AM is off when entering or exiting to OFW as well as that
interrupts are also completely off when exiting to OFW as the firmware
trap table shouldn't be used to handle our interrupts.

Modularize the Open Firmware client interface to allow run-time switching
of OFW access semantics, in order to allow future support for real-mode
OF access and flattened device frees. OF client interface modules are
implemented using KOBJ, in a similar way to the PPC PMAP modules.

Because we need Open Firmware to be available before mutexes can be used on
sparc64, changes are also included to allow KOBJ to be used very early in
the boot process by only using the mutex once we know it has been initialized.

Reviewed by: marius, grehan

Micro-optimize spitfire_block_{copy,zero}():
- Predict the loop as taken as it's more likely that there's still
data to copy and memory to zero respectively.
- Don't waste the delay slot.

USIII and beyond CPUs have stricter requirements when it comes
to synchronization needed after stores to internal ASIs in order
to make side-effects visible. This mainly requires the MEMBAR #Sync
after such stores to be replaced with a FLUSH. We use KERNBASE as
the address to FLUSH as it is guaranteed to not trap. Actually,
the USII synchronization rules also already require a FLUSH in
pretty much all of the cases changed.
We're also hitting an additional USIII synchronization rule which
requires stores to AA_IMMU_SFSR to be immediately followed by a DONE,
FLUSH or RETRY. Doing so triggers a RED state exception though so
leave the MEMBAR #Sync. Linux apparently also has gotten away with
doing the same for quite some time now, apart from the fact that
it's not clear to me why we need to clear the valid bit from the
SFSR in the first place.

Reviewed by: nwhitehorn

cosmetic changes and style fixes

Convert the remainder of the low hanging fruits regarding including
headers in .S directly rather than getting to their macros through
genassym.c/assym.s so there are less headers genassym.c has to be
kept in sync with.
While at it fix some stytle(9) bugs (indentation, prototype format,
sort headers, etc) and remove trailing whitespace.

o Add keyword volatile for user mutex owner field.
o Fix type consistent problem by using type long for old
umtx and wait channel.
o Rename casuptr to casuword.

Implement casuword32, compare and set user integer, thank Marcel Moolenarr
who wrote the IA64 version of casuword32.

Fix a bug in the data access error recorvery. Before re-enabling the data
cache after a data access error we must discard all cache lines. When
disabled existing cache lines are not invalidated by stores to memory, so
we risk reading stale data that was cached before the data access error if
we don't flush them. This is especially fatal when the memory involved
is the active part of the kernel or user stack. For good measure we also
flush the instruction cache.

This fixes random crashes when the X server probes the PCI bus through
/dev/pci.

reenable the caches when a PCI peek faults. Takes my kernel compile
from 3770 real down to 1250 real.

Submitted by: jake

add support for peeking at pci busses on UltraSparc systems. This prevents
data access errors when trying to read/write to non-existant PCI devices.

fix the psycho bridge to use peek for probing devices. This no longer
fakes it if the OFW node doesn't exist (and the reg == 0).

Reviewed by: jake, tmm

Avoid using v8 opcodes; use ba instead of b for unconditional branches.

Use 16 byte alignment for internal labels, 32 bytes is excessive.

- Fix placement of cvs ids in previous commit to match .S files in libc.
- gcc uses 32 byte alignment for functions regardless of profiling, so
follow suit.

I was wrong, the ENTRY bits in asm.h did have a purpose -- for userland.
Restore the bits and remove them from asmacros.h. *.S will now be asm.h
consumers.

Approved by: jake

Define ovbcopy() as a macro which expands to the equivalent bcopy() call,
to take care of the KAME IPv6 code which needs ovbcopy() because NetBSD's
bcopy() doesn't handle overlap like ours.

Remove all implementations of ovbcopy().

Previously, bzero was a function pointer on i386, to save a jmp to
bzero_vector. Get rid of this microoptimization as it only confuses
things, adds machine-dependent code to an MD header, and doesn't really
save all that much.

This commit does not add my pagezero() / pagecopy() code.

Add optimized block copy and zero functions using vis instructions, which
can do 64 bytes at a time and don't allocate lines in the L2 cache. These
assume that everything is 64 byte aligned, and that there's more than 128
bytes of data (best for whole pages). The block load and store instructions
don't follow normal memory ordering rules and require either a memory barrier
or move between registers before the data can actually be used. This
implementation correctly shuffles around 3 out of the 4 sets of registers
in order to avoid memory barriers expect for the last 2 blocks.

Implement casuptr.

Demark sections of code that need special fault handling with labels.
Check if the trapped pc is inside of the demarked sections to implement
fault recovery for copyin etc, instead of pcb_onfault. Handle recovery
from data access exceptions as well as page faults.

Inspired by: bde's sys.dif

Add definitions for statistical and high-resolution profiling. The calling
conventions for _mcount and __cyg_profile_func_enter are different, so
statistical profiling kernels build and link but don't actually work.
IWBNI one could tell gcc to only generate calls to the former.

Define uintfptr_t properly for userland, but not for the kernel (I hope).

Use _ALIGN_TEXT. Implement __cyg_profile_func_enter and
__cyg_profile_func_exit for GUPROF.

Remove debug code.

{f,s}usword -> {f,s}uword16. Implement {f,s}uword32.

Requested by: peter

Deorbit suibyte(). It was only used for split address space systems
for supporting UIO_USERISPACE (ie: it wasn't used).

Add declarations of suword32 and suword64. Add implementations of one or
the other (or both) to all the platforms. Similar for fuword32 and
fuword64.

Use PCB_REG instead of loading the pcb from curthread. This fixes a bug
where %g6 could be inconsistent for 1 instruction.

Add .register directives for gcc3.
Add some traces that can be useful but are also very loud.
Use defines for offsets into jmpbuf instead of magic numbers.
Fix a style bug.
Fixup comments.

1. Implement ascopyto() and ascopyfrom() for copying to an alternate address
space from kernel space and from an alternate address space to kernel
space.
2. Remove the unused and unprototyped physcopy() and physzero() and replace
with the more versatile ascopy() and aszero(), inspired by the above.
These can be used to copy and zero physical pages of memory without mapping
them into kernel space first.
3. Use magic numbers for the offsets in the jmpbuf structure like other
platforms.
4. Use SET.

Submitted by: tmm (1, 4)

Add a special OpenFirmware entry point for terminating the kernel (in
this case, the firmware trap table needs to be restored). Make use of
it in cpu_halt() and cpu_reset(), and make cpu_reset() reboot the kernel
that was used previously insead of behaving like cpu_halt().
Add a shutdown_final event handler that turns the power off if requested.

Return zero on success from su*. Apparently no one checks the return
values.
Add traces to fubyte, subyte, etc. These are useful for catching errors.
due to alignment since its usually not checked for by the caller.

Optimize bcopy and bzero etc to use 64 bit loads and stores if possible.
Handle overlap in bcopy.
Add routines for copying and zeroing pages using physical addresses
directly.
Remove all the hacks to account for calling the firmware on its own
trap table, we use the kernel trap table. There is still a problem
with OF_exit().

Add ktr traces to copy{in,out} and cpu_switch.

Context switch the cwp value. The register usage in cpu_switch will
be updated shortly to better reflect the fact that the current window
may change.

Disable interrupts when calling openfirmware.

Store 8 bytes instead of 4 in suword. Use a temporary stack that's known
to be locked in the tlb for calling openfirmware.

Submitted by: tmm

The author isn't a [UC] Regents. Correct the copyright language.

Save the primary mmu context around calls to the prom, and install
nucleus context. The prom runs at trap level 0, so there's no
implicit nucleus context and we have to force it.

Flesh out the sparc64 port considerably. This contains:
- mostly complete kernel pmap support, and tested but currently turned
off userland pmap support
- low level assembly language trap, context switching and support code
- fully implemented atomic.h and supporting cpufunc.h
- some support for kernel debugging with ddb
- various header tweaks and filling out of machine dependent structures