Copy stable/11@r303970 to releng/11.0 as part of the 11.0-RELEASE
cycle.

Prune svn:mergeinfo from the new branch, and rename it to RC1.

Update __FreeBSD_version.

Use the quarterly branch for the default FreeBSD.conf pkg(8) repo and
the dvd1.iso packages population.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r303393:
Remove empty initializer for the once facility.

Approved by: re (gjb)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Add implementation of robust mutexes, hopefully close enough to the
intention of the POSIX IEEE Std 1003.1TM-2008/Cor 1-2013.

A robust mutex is guaranteed to be cleared by the system upon either
thread or process owner termination while the mutex is held. The next
mutex locker is then notified about inconsistent mutex state and can
execute (or abandon) corrective actions.

The patch mostly consists of small changes here and there, adding
neccessary checks for the inconsistent and abandoned conditions into
existing paths. Additionally, the thread exit handler was extended to
iterate over the userspace-maintained list of owned robust mutexes,
unlocking and marking as terminated each of them.

The list of owned robust mutexes cannot be maintained atomically
synchronous with the mutex lock state (it is possible in kernel, but
is too expensive). Instead, for the duration of lock or unlock
operation, the current mutex is remembered in a special slot that is
also checked by the kernel at thread termination.

Kernel must be aware about the per-thread location of the heads of
robust mutex lists and the current active mutex slot. When a thread
touches a robust mutex for the first time, a new umtx op syscall is
issued which informs about location of lists heads.

The umtx sleep queues for PP and PI mutexes are split between
non-robust and robust.

Somewhat unrelated changes in the patch:
1. Style.
2. The fix for proper tdfind() call use in umtxq_sleep_pi() for shared
pi mutexes.
3. Removal of the userspace struct pthread_mutex m_owner field.
4. The sysctl kern.ipc.umtx_vnode_persistent is added, which controls
the lifetime of the shared mutex associated with a vnode' page.

Reviewed by: jilles (previous version, supposedly the objection was fixed)
Discussed with: brooks, Martin Simmons <martin@lispworks.com> (some aspects)
Tested by: pho
Sponsored by: The FreeBSD Foundation

Use __FBSDID() for .c files from lib/libthr/thread.

Sponsored by: The FreeBSD Foundation

Remove unused variable. It was write-only before r297139.

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

Lock pshared_lock shared around fork, to ensure that the COW snapshot
of the pshared hash in child is consistent and can be safely used.

Reported and tested by: "Oleg V. Nauman" <oleg@opentransfer.com>
Sponsored by: The FreeBSD Foundation

From libthr, remove special and strange code to set up session and
control terminal, activated when running with pid 1. It is
application duty to handle this, and unsuspecting init replacements
which are linked with libthr would be broken by this.

The pre-resolving of getpid() is restored, just in case.

Reviewed by: jilles
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks

Implement process-shared locks support for libthr.so.3, without
breaking the ABI. Special value is stored in the lock pointer to
indicate shared lock, and offline page in the shared memory is
allocated to store the actual lock.

Reviewed by: vangyzen (previous version)
Discussed with: deischen, emaste, jhb, rwatson,
Martin Simmons <martin@lispworks.com>
Tested by: pho
Sponsored by: The FreeBSD Foundation

libthr: Don't use both __sys_open() and __sys_openat().

Fix known issues which blow up the process after dlopen("libthr.so")
(or loading a dso linked to libthr.so into process which was not
linked against threading library).

- Remove libthr interposers of the libc functions, including
__error(). Instead, functions calls are indirected through the
interposing table, similar to how pthread stubs in libc are already
done. Libc by default points either to syscall trampolines or to
existing libc implementations. On libthr load, libthr rewrites the
pointers to the cancellable implementations already in libthr. The
interposition table is separate from pthreads stubs indirection
table to not pull pthreads stubs into static binaries.

- Postpone the malloc(3) internal mutexes initialization until libthr
is loaded. This avoids recursion between calloc(3) and static
pthread_mutex_t initialization.

- Reinstall signal handlers with wrapper on libthr load. The
_rtld_is_dlopened(3) is used to avoid useless calls to sigaction(2)
when libthr is statically referenced from the main binary.

In the process, fix openat(2), swapcontext(2) and setcontext(2)
interposing. The libc symbols were exported at different versions
than libthr interposers. Export both libc and libthr versions from
libc now, with default set to the higher version from libthr.

Remove unused and disconnected swapcontext(3) userspace implementation
from libc/gen.

No objections from: deischen
Tested by: pho, antoine (exp-run) (previous versions)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

Switch the defaults to not split the RLIMIT_STACK-sized initial thread
stack into the stacks of the created threads. Add knob
LIBPTHREAD_SPLITSTACK_MAIN to restore the older behaviour.

Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks

Add a knob LIBPTHREAD_BIGSTACK_MAIN, which instructs libthr to leave
the whole RLIMIT_STACK-sized region of the kernel-allocated stack as
the stack of main thread.

By default, the main thread stack is clamped at 2MB (4MB on 64bit
ABIs) and the rest is used for other threads stack allocation. Since
there is no programmatic way to adjust the size of the main thread
stack, pthread_attr_setstacksize() is too late, the knob allows user
to manage the main stack size both for single-threaded and
multi-threaded processes with the rlimit.

Reported by: "Ivan A. Kosarev" <ivan@ivan-labs.com>
Tested by: dim
Sponsored by: The FreeBSD Foundation
MFC after: 3 days

libthr: Always use the threaded rtld lock implementation.

The threaded rtld lock implementation is faster even in the single-threaded
case because it postpones signal handlers via THR_CRITICAL_ENTER and
THR_CRITICAL_LEAVE instead of calling sigprocmask(2).

As a result, exception handling becomes faster in single-threaded
applications linked with libthr.

Reviewed by: kib

In suspend_common(), don't wait for a thread which is in creation, because
pthread_suspend_all_np() may have already suspended its parent thread.
Add locking code in pthread_suspend_all_np() to only allow one thread
to suspend other threads, this eliminates a deadlock where two or more
threads try to suspend each others.

MFp4:
Enqueue thread in LIFO, this can cause starvation, but it gives better
performance. Use _thr_queuefifo to control the frequency of FIFO vs LIFO,
you can use environment string LIBPTHREAD_QUEUE_FIFO to configure the
variable.

Do not set thread name to less than informative 'initial thread'.

MFp4:

- Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based
condition variable, this should eliminate an extra system call to get
current time.

- Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single
system call. Create userland sleep queue for condition variable, in most
cases, thread will wait in the queue, the pthread_cond_signal will defer
thread wakeup until the mutex is unlocked, it tries to avoid an extra
system call and a extra context switch in time window of pthread_cond_signal
and pthread_mutex_unlock.

The changes are part of process-shared mutex project.

In current code, statically initialized and destroyed object have
same null value, the code can not distinguish between them, to
fix the problem, now a destroyed object is assigned to a non-null
value, and it will be rejected by some pthread functions.
PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that
adaptive mutex can be statically initialized correctly.

To support stack unwinding for cancellation points, add -fexceptions flag
for them, two functions _pthread_cancel_enter and _pthread_cancel_leave
are added to let thread enter and leave a cancellation point, it also
makes it possible that other functions can be cancellation points in
libraries without having to be rewritten in libthr.

add code to support stack unwinding when thread exits. note that only
defer-mode cancellation works, asynchrnous mode does not work because
it lacks of libuwind's support. stack unwinding is not enabled unless
LIBTHR_UNWIND_STACK is defined in Makefile.

Convert thread list lock from mutex to rwlock.

Change atfork lock from mutex to rwlock, also make mutexes used by malloc()
module private type, when private type mutex is locked/unlocked, thread
critical region is entered or leaved. These changes makes fork()
async-signal safe which required by POSIX. Note that user's atfork handler
still needs to be async-signal safe, but it is not problem of libthr, it
is user's responsiblity.

Add signal handler wrapper, the reason to add it becauses there are
some cases we want to improve:
1) if a thread signal got a signal while in cancellation point,
it is possible the TDP_WAKEUP may be eaten by signal handler
if the handler called some interruptibly system calls.
2) In signal handler, we want to disable cancellation.
3) When thread holding some low level locks, it is better to
disable signal, those code need not to worry reentrancy,
sigprocmask system call is avoided because it is a bit expensive.
The signal handler wrapper works in this way:
1) libthr installs its signal handler if user code invokes sigaction
to install its handler, the user handler is recorded in internal
array.
2) when a signal is delivered, libthr's signal handler is invoke,
libthr checks if thread holds some low level lock or is in critical
region, if it is true, the signal is buffered, and all signals are
masked, once the thread leaves critical region, correct signal
mask is restored and buffered signal is processed.
3) before user signal handler is invoked, cancellation is temporarily
disabled, after user signal handler is returned, cancellation state
is restored, and pending cancellation is rescheduled.

Use umtx to implement process sharable semaphore, to make this work,
now type sema_t is a structure which can be put in a shared memory area,
and multiple processes can operate it concurrently.
User can either use mmap(MAP_SHARED) + sem_init(pshared=1) or use sem_open()
to initialize a shared semaphore.
Named semaphore uses file system and is located in /tmp directory, and its
file name is prefixed with 'SEMD', so now it is chroot or jail friendly.
In simplist cases, both for named and un-named semaphore, userland code
does not have to enter kernel to reduce/increase semaphore's count.
The semaphore is designed to be crash-safe, it means even if an application
is crashed in the middle of operating semaphore, the semaphore state is
still safely recovered by later use, there is no waiter counter maintained
by userland code.
The main semaphore code is in libc and libthr only has some necessary stubs,
this makes it possible that a non-threaded application can use semaphore
without linking to thread library.
Old semaphore implementation is kept libc to maintain binary compatibility.
The kernel ksem API is no longer used in the new implemenation.

Discussed on: threads@

- Reduce function call overhead for uncontended case.
- Remove unused flags MUTEX_FLAGS_* and their code.
- Check validity of the timeout parameter in mutex_self_lock().

_vfork is not in libthr, remove the reference.

Use cpuset defined in pthread_attr for newly created thread, for now,
we set scheduling parameters and cpu binding fully in userland, and
because default scheduling policy is SCHED_RR (time-sharing), we set
default sched_inherit to PTHREAD_SCHED_INHERIT, this saves a system
call.

implement pthread_attr_getaffinity_np and pthread_attr_setaffinity_np.

Add my recent work of adaptive spin mutex code. Use two environments variable
to tune pthread mutex performance:
1. LIBPTHREAD_SPINLOOPS
If a pthread mutex is being locked by another thread, this environment
variable sets total number of spin loops before the current thread
sleeps in kernel, this saves a syscall overhead if the mutex will be
unlocked very soon (well written application code).
2. LIBPTHREAD_YIELDLOOPS
If a pthread mutex is being locked by other threads, this environment
variable sets total number of sched_yield() loops before the currrent
thread sleeps in kernel. if a pthread mutex is locked, the current thread
gives up cpu, but will not sleep in kernel, this means, current thread
does not set contention bit in mutex, but let lock owner to run again
if the owner is on kernel's run queue, and when lock owner unlocks the
mutex, it does not need to enter kernel and do lots of work to resume
mutex waiters, in some cases, this saves lots of syscall overheads for
mutex owner.

In my practice, sometimes LIBPTHREAD_YIELDLOOPS can massively improve performance
than LIBPTHREAD_SPINLOOPS, this depends on application. These two environments
are global to all pthread mutex, there is no interface to set them for each
pthread mutex, the default values are zero, this means spinning is turned off
by default.

backout experimental adaptive spinning mutex for product use.

get LIBPTHREAD_ADAPTIVE_SPIN early, so it can be used for some global
mutexes.

Check environment variable PTHREAD_ADAPTIVE_SPIN, if it is set, use
it as a default spin cycle count.

- Remove variable _thr_scope_system, all threads are system scope.
- Rename _thr_smp_cpus to boolean variable _thr_is_smp.
- Define CPU_SPINWAIT macro for each arch, only X86 supports it.

Eliminate atomic operations in thread cancellation functions, it should
reduce overheads of cancellation points.

use rtprio_thread system call to get or set thread priority.

Replace internal usage of struct umtx with umutex which can supports
real-time if we want, no functionality is changed.

Use umutex APIs to implement pthread_mutex, member pp_mutexq is added
into pthread structure to keep track of locked PTHREAD_PRIO_PROTECT mutex,
no real mutex code is changed, the mutex locking and unlocking code should
has same performance as before.

Get number of CPUs and ignore spin count on single processor machine.

1. Don't override underscore version of aio_suspend(), system(),
wait(), waitpid() and usleep(), they are internal versions and
should not be cancellation points.
2. Make wait3() as a cancellation point.
3. Move raise() and pause() into file thr_sig.c.
4. Add functions _sigsuspend, _sigwait, _sigtimedwait and _sigwaitinfo,
remove SIGCANCEL bit in wait-set for those functions, the signal is
used internally to implement thread cancellation.

Caching scheduling policy and priority in userland, a critical but baddly
written application is frequently changing thread priority for SCHED_OTHER
policy.

Use kernel facilities to support real-time scheduling.

- Use same priority range returned by kernel's sched_get_priority_min()
and sched_get_priority_max() syscalls.
- Remove unused fields from structure pthread_attr.

WARNS level 4 cleanup.

Remove priority mutex code because it does not work correctly,
to make it work, turnstile like mechanism to support priority
propagating and other realtime scheduling options in kernel
should be available to userland mutex, for the moment, I just
want to make libthr be simple and efficient thread library.

Discussed with: deischen, julian

Set default contention scope to system.

Add some more pthread stubs so that librt can use them.
The thread jump table has been resorted, so you need to
keep libc, libpthread, and libthr in sync.

Submitted by: xu

Rework last change of pthread_once, create a function _thr_once_init to
reinitialize its internal locks.

After fork(), reinitialize internal locks for pthread_once().

Now, thread name is stored in kernel, userland no longer has to keep it.

Use macro STATIC_LIB_REQUIRE to declare a symbol should be linked into
static binary.

1. Retire macro SCLASS, instead simply use language keyword and
put variables in thr_init.c.
2. Hide all global symbols which won't be exported.

Add code to handle timer_delete(). The timer wrapper code is completely
rewritten, now timers created with same sigev_notify_attributes will
run in same thread, this allows user to organize which timers can
run in same thread to save some thread resource.

Conditionally report initial thread event.

Add debugger event reporting support, current only TD_CREATE and TD_DEATH
events are reported.

Remove unique id field which is no longer used by debugger.

Import my recent 1:1 threading working. some features improved includes:
1. fast simple type mutex.
2. __thread tls works.
3. asynchronous cancellation works ( using signal ).
4. thread synchronization is fully based on umtx, mainly, condition
variable and other synchronization objects were rewritten by using
umtx directly. those objects can be shared between processes via
shared memory, it has to change ABI which does not happen yet.
5. default stack size is increased to 1M on 32 bits platform, 2M for
64 bits platform.
As the result, some mysql super-smack benchmarks show performance is
improved massivly.

Okayed by: jeff, mtm, rwatson, scottl

Increase the default stacksizes:

32-bit 64-bit
main thread 2 MB 4 MB
other threads 1 MB 2 MB

Approved by: mtm
Adapted from: libpthread

Don't include sys/user.h merely for its side-effect of recursively
including other headers.

Implement pthread_atfork in libthr. This is mostly from deichen's
work in libpthread.

Submitted by: Dan Nelson <dnelson@allantgroup.com>

In the case that the global thread list is being re-initialized after
a fork, make sure that the current thread isn't detached and freed. As
a consequence the thread should be inserted into the head of the
active list only once (in the beginning).

Make libthr async-signal-safe without costly signal masking. The guidlines I
followed are: Only 3 functions (pthread_cancel, pthread_setcancelstate,
pthread_setcanceltype) are required to be async-signal-safe by POSIX. None of
the rest of the pthread api is required to be async-signal-safe. This means
that only the three mentioned functions are safe to use from inside
signal handlers.
However, there are certain system/libc calls that are
cancellation points that a caller may call from within a signal handler,
and since they are cancellation points calls have to be made into libthr
to test for cancellation and exit the thread if necessary. So, the
cancellation test and thread exit code paths must be async-signal-safe
as well. A summary of the changes follows:

o Almost all of the code paths that masked signals, as well as locking the
pthread structure now lock only the pthread structure.
o Signals are masked (and left that way) as soon as a thread enters
pthread_exit().
o The active and dead threads locks now explicitly require that signals
are masked.
o Access to the isdead field of the pthread structure is protected by both
the active and dead list locks for writing. Either one is sufficient for
reading.
o The thread state and type fields have been combined into one three-state
switch to make it easier to read without requiring a lock. It doesn't need
a lock for writing (and therefore for reading either) because only the
current thread can write to it and it is an integer value.
o The thread state field of the pthread structure has been eliminated. It
was an unnecessary field that mostly duplicated the flags field, but
required additional locking that would make a lot more code paths require
signal masking. Any truly unique values (such as PS_DEAD) have been
reborn as separate members of the pthread structure.
o Since the mutex and condvar pthread functions are not async-signal-safe
there is no need to muck about with the wait queues when handling
a signal ...
o ... which also removes the need for wrapping signal handlers and sigaction(2).
o The condvar and mutex async-cancellation code had to be revised as a result
of some of these changes, which resulted in semi-unrelated changes which
would have been difficult to work on as a separate commit, so they are
included as well.

The only part of the changes I am worried about is related to locking for
the pthread joining fields. But, I will take a closer look at them once this
mega-patch is committed.

o Remove more references to SIGTHR
o Remove clock resolution information left over from libc_r

Remove the garbage collector thread. All resources are freed
in-line. If the exiting thread cannot release a resource, then
the next thread to exit will release it.

Move the initialization of thread priority to a common function.

Preparations to make libthr work in multi-threaded fork()ing applications.

o Remove some code duplication between _thread_init(), which is run once
to initialize libthr and the intitial thread, and pthread_create(), which
initializes newly created threads, into a new function called from both
places: init_td_common()
o Move initialization of certain parts of libthr into a separate
function. These include:
- Active threads list and it's lock
- Dead threads list and it's lock & condition variable
- Naming and insertion of the initial thread into the
active threads list.

When _PTHREADSINVARIANTS is defined SIGABRT is not included
in the set of signals to block.
Also, make the PANIC macro call abort() instead of simply
exiting.

Make _thread_suspend work with both the old broken sigtimedwait
implementation and the new improved one. We now precompute the
signal set passed to sigtimedwait, using an inverted set when
necessary for compatibility with older kernels.

Make C applications statically compiled with libthr work. Previously,
an application compiled -static with libthr would dump core in
malloc(3) because the stub thread initialization routine in libc would
be used instead of the libthr supplied one.

Return gracefully, rather than aborting, when the maximum concurrent
threads per process has been reached. Return EAGAIN, as per spec.

Approved by: re/blanket libthr

Start locking up the active and dead threads lists. The active threads
list is protected by a spinlock_t, but the dead list uses a pthread_mutex
because it is necessary to synchronize other threads with the garbage
collector thread. Lock/Unlock macros are used so it's easier to make
changes to the locks in the future.

The 'dead thread list' lock is intended to replace the gc mutex.
This doesn't have any practical ramifications. It simply makes it
clearer what the purpose of the lock is. The gc will use this lock,
instead of the gc mutex, to synchronize access to the dead list with
other threads.

Modify _pthread_exit() to use these two new locks instead of GIANT_LOCK,
and also to properly lock and protect thread state changes,
especially with respect to a joining thread.

The gc thread was also re-arranged to be more organized and less nested.

_pthread_join() was also modified to use the thread list locks. However,
locking and unlocking here needs special care because a thread could find
itself in a position where it's joining an exiting thread that is
waiting on the dead list lock, which this thread (joiner) holds. If the
joiner doesn't take care to lock *and* unlock in the same order they
(the joiner and the joinee) could deadlock against each other.

Approved by: re/blanket libthr

Make WARNS2 clean. The fixes mostly included:
o removed unused variables
o explicit inclusion of header files
o prototypes for externally defined functions

Approved by: re/blanket libthr

The thread id was being set *before* zeroing out the thread. Reverse
the order.

Approved by: markm/mentor, re/blanket libthr

- Pass a ucontext_t to _set_curthread. If non-NULL the new thread is set
as curthread in the new context, so that it will be set automatically when
the thread is switched to. This fixes a race where we'd run for a little
while with curthread unset in _thread_start.

Reviewed by: jeff

- Define curthread as _get_curthread() and remove all direct calls to
_get_curthread(). This is similar to the kernel's curthread. Doing
this saves stack overhead and is more convenient to the programmer.
- Pass the pointer to the newly created thread to _thread_init().
- Remove _get_curthread_slow().

- Add libthr but don't hook it up to the regular build yet. This is an
adaptation of libc_r for the thr system call interface. This is beta
quality code.