Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r236991:

Final whitespace trim.

Submitted by: imp
Approved by: marius
Obtained from: Netflix

MFC r227293 (by ed):
Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs.

This means that their use is restricted to a single C file.

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Make md_tp a register_t not a void *. This will keep us from
accidentally dereferencng it and might be one fewer things to change
if arm64 happens...

Submitted by: rwatson's question on irc...

For architectures not using direct map , and requiring real KVA page for
sf buf allocation, use wakeup() instead of wakeup_one() to notify sf
buffer waiters about free buffer.

sf_buf_alloc() calls msleep(PCATCH) when SFB_CATCH flag was given,
and for simultaneous wakeup and signal delivery, msleep() returns
EINTR/ERESTART despite the thread was selected for wakeup_one(). As
result, we loose a wakeup, and some other waiter will not be woken up.

Reported and tested by: az
Reviewed by: alc, jhb
MFC after: 1 week

Fix ARM cache handling yet more.

1) vm_machdep.c: remove the dangling allocations so they do not
un-necessarily turn off the cache upon consecutive access.

2) busdma_machdep.c: remove the same amount than shadow mapped.

Reported by: Maks Verver
Submitted by: Mark Tinguely
Reviewed by: Grzegorz Bernacki
MFC after: 3 days

Extract the code that records syscall results in the frame into MD
function cpu_set_syscall_retval().

Suggested by: marcel
Reviewed by: marcel, davidxu
PowerPC, ARM, ia64 changes: marcel
Sparc64 tested and reviewed by: marius, also sunv reviewed
MIPS tested by: gonzo
MFC after: 1 month

Reintroduce the r196640, after fixing the problem with my testing.

Remove the altkstacks, instead instantiate threads with kernel stack
allocated with the right size from the start. For the thread that has
kernel stack cached, verify that requested stack size is equial to the
actual, and reallocate the stack if sizes differ [1].

This fixes the bug introduced by r173361 that was committed several days
after r173004 and consisted of kthread_add(9) ignoring the non-default
kernel stack size.

Also, r173361 removed the caching of the kernel stacks for a non-first
thread in the process. Introduce separate kernel stack cache that keeps
some limited amount of preallocated kernel stacks to lower the latency
of thread allocation. Add vm_lowmem handler to prune the cache on
low memory condition. This way, system with reasonable amount of the
threads get lower latency of thread creation, while still not exhausting
significant portion of KVA for unused kstacks.

Submitted by: peter [1]
Discussed with: jhb, julian, peter
Reviewed by: jhb
Tested by: pho (and retested according to new test scenarious)
MFC after: 1 week

Reverse r196640 and r196644 for now.

Remove the altkstacks, instead instantiate threads with kernel stack
allocated with the right size from the start. For the thread that has
kernel stack cached, verify that requested stack size is equial to the
actual, and reallocate the stack if sizes differ [1].

This fixes the bug introduced by r173361 that was committed several days
after r173004 and consisted of kthread_add(9) ignoring the non-default
kernel stack size.

Also, r173361 removed the caching of the kernel stacks for a non-first
thread in the process. Introduce separate kernel stack cache that keeps
some limited amount of preallocated kernel stacks to lower the latency
of thread allocation. Add vm_lowmem handler to prune the cache on
low memory condition. This way, system with reasonable amount of the
threads get lower latency of thread creation, while still not exhausting
significant portion of KVA for unused kstacks.

Submitted by: peter [1]
Discussed with: jhb, julian, peter
Reviewed by: jhb
Tested by: pho
MFC after: 1 week

ARM pmap fixes.

a) nocache-remap problem

When a page is remapped into a non-cacheable virtual memory region there
was no associated write-back invalidate operation performed. We remove
writeback of the original buffer size from bus_dmamem_alloc() and add
appropriate L1/L2 flush operation.

b) missing write-back invalidate operation

In pmap_kremove a page is removed so we must do a write-back
invalidate operation aligned to the page virtual address.

Submitted by: Michal Hajduk
Reviewed by: Mark Tinguely, rpaulo, stas
Approved by: re (kib)
Obtained from: Semihalf

Fix typo.

Remove unused variables.

Spotted out by: Christoph Mallon <christoph d0t mallon AT gmx d0t de>

Fix ARM nocache allocator:

- let the loop iterate every page (as intended), and not some multiplies
(which led to a fake exhaustion of the ARM_NOCACHE_KVA_SIZE)

- eliminate using MIN(): it compared number of pages vs. address
(ARM_TP_ADDRESS), which was bogus

Reviewed by: cognet, imp
Obtained from: Piotr Ziecik kosmo ! semihalf dot com
MFC after: 3 days

In keeping with style(9)'s recommendations on macros, use a ';'
after each SYSINIT() macro invocation. This makes a number of
lightweight C parsers much happier with the FreeBSD kernel
source, including cflow's prcc and lxr.

MFC after: 1 month
Discussed with: imp, rink

o Rename cpu_thread_setup() to cpu_thread_alloc() to better
communicate that it relates to (is called by) thread_alloc()
o Add cpu_thread_free() which is called from thread_free()
to counter-act cpu_thread_alloc().

i386: Have cpu_thread_free() call cpu_thread_clean() to
preserve behaviour.
ia64: Have cpu_thread_free() call mtx_destroy() for the
mutex initialized in cpu_thread_alloc().

PR: ia64/118024

It has been observed on the mailing lists that the different categories
of pages don't sum to anywhere near the total number of pages on amd64.
This is for the most part because uma_small_alloc() pages have never been
counted as wired pages, like their kmem_malloc() brethren. They should
be. This changes fixes that.

It is no longer necessary for the page queues lock to be held to free
pages allocated by uma_small_alloc(). I removed the acquisition and
release of the page queues lock from uma_small_free() on amd64 and ia64
weeks ago. This patch updates the other architectures that have
uma_small_alloc() and uma_small_free().

Approved by: re (kensmith)

Use supersection instead of standard sections to map the whole memory
when available.

Approved by: re (blanket)

- Change comments and asserts to reflect the removal of the global
scheduler lock.

Tested by: kris, current@
Tested on: i386, amd64, ULE, 4BSD, libthr, libkse, PREEMPTION, etc.
Discussed with: kris, attilio, kmacy, jhb, julian, bde (small parts each)

Remove duplicate includes.

Submitted by: Cyril Nguyen Huu <cyril ci0 org>

- Add bounce pages for arm, largely based on the i386 implementation.
- Add a default parent dma tag, similar to what has been done for sparc64.
- Before invalidating the dcache in POSTREAD, save the bits which are in the
same cachelines than our buffers, but not part of it, and restore them after
the invalidation.

In the ARM_USE_SMALL_ALLOC case, vm_page_t may have an address < KERNBASE,
so adjust the KASSERT to reflect this.

Ooops, dump_avail[i] can be 0 if the RAM starts at 0x00000000, so check that
dump_avail[i + 1] is == 0 as a stop condition instead.
MFC after: 3 days

Rewrite ARM_USE_SMALL_ALLOC so that instead of the current behavior, it maps
whole the physical memory, cached, using 1MB section mappings. This reduces
the address space available for user processes a bit, but given the amount of
memory a typical arm machine has, it is not (yet) a big issue.
It then provides a uma_small_alloc() that works as it does for architectures
which have a direct mapping.

Oops bring back code that shouldn't have been removed by the previous
commit.

Make sure that if uma_small_alloc() gets called recursively, we just give up
and call kmem_malloc(), to avoid a deadlock.

vm_page_alloc_contig() can sleep, so don't even think about using it
in the M_NOWAIT case.

Use 8 * sizeof(int) instead of hardcoding 32, for the unlikely case this
code ever get used on a plateform where sizeof(int) != 4.

Suggested by: jmg

Try to honor BUS_DMA_COHERENT : if the flag is set, normally allocate memory
with malloc() or contigmalloc() as usual, but try to re-map the allocated
memory into a VA outside the KVA, non-cached, thus making the calls to
bus_dmamap_sync() for these buffers useless.

Try harder not to recurse.

Try to use contigmalloc() even if M_NOWAIT has been specified.

Normalize a significant number of kernel malloc type names:

- Prefer '_' to ' ', as it results in more easily parsed results in
memory monitoring tools such as vmstat.

- Remove punctuation that is incompatible with using memory type names
as file names, such as '/' characters.

- Disambiguate some collisions by adding subsystem prefixes to some
memory types.

- Generally prefer lower case to upper case.

- If the same type is defined in multiple architecture directories,
attempt to use the same name in additional cases.

Not all instances were caught in this change, so more work is required to
finish this conversion. Similar changes are required for UMA zone names.

If a thread already tries to allocate a new memory range, wait for it
instead of trying to do the same.

Validate if the value written into {FS,GS}.base is a canonical
address, writting non-canonical address can cause kernel a panic,
by restricting base values to 0..VM_MAXUSER_ADDRESS, ensuring
only canonical values get written to the registers.

Reviewed by: peter, Josepha Koshy < joseph.koshy at gmail dot com >
Approved by: re (scottl)

Don't abuse UMA_SLAB_KMEM.

Approved by: re (blanket)

Add a new arm-specific option, ARM_USE_SMALL_ALLOC. If defined, it provides
an implementation of uma_small_alloc() which tries to preallocate memory
1MB per 1MB, and maps it into a section mapping.

Correctly setup the UND stack in cpu_set_upcall(), and the trapframe in
cpu_thread_setup(), as done in cpu_fork().

Don't forget to copy the TP when forking, or bad things will happen to the
child process if it tries to use threads.

Change cpu_set_kse_upcall to more generic style, so we can reuse it
in other codes. Add cpu_set_user_tls, use it to tweak user register
and setup user TLS. I ever wanted to merge it into cpu_set_kse_upcall,
but since cpu_set_kse_upcall is also used by M:N threads which may
not need this feature, so I wrote a separated cpu_set_user_tls.

Divorce critical sections from spinlocks. Critical sections as denoted by
critical_enter() and critical_exit() are now solely a mechanism for
deferring kernel preemptions. They no longer have any affect on
interrupts. This means that standalone critical sections are now very
cheap as they are simply unlocked integer increments and decrements for the
common case.

Spin mutexes now use a separate KPI implemented in MD code: spinlock_enter()
and spinlock_exit(). This KPI is responsible for providing whatever MD
guarantees are needed to ensure that a thread holding a spin lock won't
be preempted by any other code that will try to lock the same lock. For
now all archs continue to block interrupts in a "spinlock section" as they
did formerly in all critical sections. Note that I've also taken this
opportunity to push a few things into MD code rather than MI. For example,
critical_fork_exit() no longer exists. Instead, MD code ensures that new
threads have the correct state when they are created. Also, we no longer
try to fixup the idlethreads for APs in MI code. Instead, each arch sets
the initial curthread and adjusts the state of the idle thread it borrows
in order to perform the initial context switch.

This change is largely a big NOP, but the cleaner separation it provides
will allow for more efficient alternative locking schemes in other parts
of the kernel (bare critical sections rather than per-CPU spin mutexes
for per-CPU data for example).

Reviewed by: grehan, cognet, arch@, others
Tested on: i386, alpha, sparc64, powerpc, arm, possibly more

Start all license statements with /*-

Don't include sys/user.h merely for its side-effect of recursively
including other headers.

Get the kernel stack right now that the u-area is gone.

Remove some references to U area here while trying not to break
anything. Someone with ARM hardware could do a lot more to untangle
this code.

Reviewed by: arch@

Introduce two new options, "CPU private" and "no wait", to sf_buf_alloc().
Change the spelling of the "catch" option to be consistent with the new
options. Implement the "no wait" option. An implementation of the "CPU
private" for i386 will be committed at a later date.

Implement cpu_set_upcall and cpu_set_upcall_kse.
Calculate td_frame and td_pcb the right way in cpu_thread_setup.

On Xscale, use the minicache for the kernel stack.

Define NSFBUFS and use it.

Retire cpu_sched_exit(); it is not used any more.

Import FreeBSD/arm kernel bits.
It only supports sa1110 (on simics) right now, but xscale support should come
soon.
Some of the initial work has been provided by :
Stephane Potvin <sepotvin at videotron.ca>
Most of this comes from NetBSD.