pthread: Nix trailing whitespace.

libpthread: Move namespacing include to top of .c files.

Stuff like libc's namespace.h, or atomic_op_namespace.h, which does
namespacing tricks like `#define atomic_cas_uint _atomic_cas_uint',
has to go at the top of each .c file. If it goes in the middle, it
might be too late to affect the declarations, and result in compile
errors.

I tripped over this by including <sys/atomic.h> in mips
<machine/lock.h>.

(Maybe we should create a new pthread_namespace.h file for the
purpose, but this'll do for now.)

Adjust memory barriers.

- Make pthread_condvar and pthread_mutex work on the stack rather than in
pthread_t, so there's less chance of bad things happening if someone calls
(for example) pthread_cond_broadcast() from a signal handler.

- Remove all the deferred waiter handling except for the one case that really
matters which is transferring waiters from condvar -> mutex on wakeup, and
do that by splicing the condvar's waiters onto the mutex.

- Remove the mutex waiters bit as it's another complication that's not
strictly needed.

Deal with a couple of problems with threads being awoken early due to
timeouts or cancellation where:

- The restarting thread calls _lwp_exit() before another thread gets around
to waking it with _lwp_unpark(), leading to ESRCH (observed by joerg@).
(I may have removed a similar check mistakenly over the weekend.)

- The restarting thread considers itself gone off the sleep queue but
at the same time another thread is part way through waking it, and hasn't
fully completed that operation yet by setting thread->pt_mutexwait = 0.
I think that could have potentially lead to the list of waiters getting
messed up given the right circumstances.

In the interests of reliability simplify waiter handling more and redo
condvars to manage the list of waiters with atomic ops.

- Try to eliminate a hang in "parked" I've been seeing while stress testing.
Centralise wakeup of deferred waiters in pthread__clear_waiters() and use
throughout libpthread. Make fewer assumptions. Be more conservative in
pthread_mutex when dealing with pending waiters.

- Remove the "hint" argument everywhere since the kernel doesn't use it any
more.

Revert "Enhance the pthread(3) + malloc(3) init model"

It is reported to hand on aarch64 with gzip.

Enhance the pthread(3) + malloc(3) init model

Separate the pthread_atfork(3) call from pthread_tsd_init()
and move it into a distinct function.

Call inside pthread__init() late TSD initialization route, just after
"pthread_atfork(NULL, NULL, pthread__fork_callback);".

Document that malloc(3) initialization is now controlled again and called
during the first pthread_atfork(3) call.

Remove #if 0 code from pthread_mutex.c as we no longer initialize malloc
prematurely.

Revert previous

'git grep' breaks now.

Remove 'ifdef 0' hacks

It is no longer needed as the proper fix avoiding premature malloc()
landed the sources.

Refactor libpthread checks for invalid arguments

Switch from manual functions to pthread__error().

In the same spirit as the previous pthread_mutex_init change for jemalloc,
make pthread_mutexattr_init do always a full initialization, so that the
attribute that will be used later when we become threaded is properly
initialized.

Use pthread_mutexattr_t and pthread_mutex_t magic fields

Validate _PT_MUTEX_MAGIC in pthread_mutex_t and _PT_MUTEXATTR_MAGIC
in pthread_mutexattr_t accordingly.

Mark destroyed pthread_mutexattr_t as dead

Adjustment to previous: don't call _lwp_unpark_all() with nwaiters == 0.

pthread__mutex_unlock_slow(): ignore the DEFERRED bit. It's only purpose
is to get the thread to go through the slow path. If there are waiters,
process them there and then. Should not affect well behaved apps. Maybe
of help for:

PR bin/50350: rump/rumpkern/t_sp/stress_{long,short} fail on Core 2 Quad

Rip out some very ambitious optimisations around pthread_mutex that are
don't buy much. This stuff is hard enough to get right in the kernel let
alone userspace, and I don't trust that it's right.

branches: 1.65.2;
Jemalloc initializes mutexes before we become threaded and expects to use
them later.

branches: 1.64.4;

Deal with more lwp_park() timestamp unconsting

Don't spin if we already own the mutex, otherwise we will get stuck spinning
forever, fixes timemutex{1,2} tests.

Use anonymous union for ptm_ceiling and old __pthread_spin_t field to
maintain backward compatibility and fix hppa build. hppa has an non-
integer type __pthread_spin_t

KNF

branches: 1.60.2;
GSoC 2016 Charles Cui: Implement thread priority protection based on work
by Andy Doran. Also document the get/set pshared thread calls as not
implemented, and add a skeleton implementation that is disabled.
XXX: document _sched_protect(2).

pthread__mutex_lock_slow: fix the handling of a potential race with the
non-interlocked CAS in the fast unlock path -- it is unsafe to test for
the waiters-bit while the owner thread is running, we have to spin for
the owner or its state change to be sure about the presence of the bit.
Split off the logic into the pthread__mutex_setwaiters() routine.

This is a partial fix to the named lockup problem (also see PR/44756).
It seems there is another race which can be reproduced on faster CPUs.

remove compatibility code for handling CLOCK_MONOTONIC and handle it in the
syscall directly.

PR/44756: Sad Clouds: Prevent leakage of errno = ESRCH from _lwp_park. This
has two parts:
- in pthread_cond_timedwait() if the thread we are trying to unpark
exited, retry the the _lwp_park call without it.
- pthread_mutex() was affecting errno since it is calling _lwp_park()
from pthread_mutex_lock_slow(). preserve the original errno.
Note that the example problem still causes an occassional deadlock on machines
with many CPUs and it is the same deadlock we observe with named.

- Allow libpthread to be dlopened again, by providing libc stubs to libpthread.
- Fail if the dlopened libpthread does pthread_create(). From manu@
- Discussed at length in the mailing lists; approved by core@
- This was chosen as the least intrusive patch that will provide
the necessary functionality.
XXX: pullup to 6

add comments
whitespace

branches: 1.54.2;
Add a pthread__smt_wake and add support for it on arm along with
pthread__smt_pause. These are implemented using the ARM instructions
SEV (wake) and WFE (pause). These are treated as NOPs on ARM CPUs that
don't support them.

Move the pthread_once alias where it belongs.

Move pthread_once implementation into a separate file, it doesn't depend
on the mutex implementation in any way.

branches: 1.51.2; 1.51.4; 1.51.6; 1.51.22;
Change pthread_mutex_t to use the amount of space as and be congruent to
the version used in the SA version of pthreads. This preserves binary
compatibility between both versions of the library.

PR lib/38741 priority inversion in libpthread breaks apps that use
SCHED_FIFO threads

- Change condvar sync so that we never take the condvar's spinlock without
first holding the caller-provided mutex. Previously, the spinlock was only
taken without the mutex in an error path, but it was enough to trigger the
problem described in the PR.

- Even with this change, applications calling pthread_cond_signal/broadcast
without holding the interlocking mutex are still subject to the problem
described in the PR. POSIX discourages this saying that it leads to
undefined scheduling behaviour, which seems good enough for the time being.

- Elsewhere, use a hash of mutexes instead of per-object spinlocks to
synchronize entry/exit from sleep queues.

- Simplify how sleep queues are maintained.

Add some general comments about the mutex implementation.

branches: 1.48.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.47.2;
pthread__mutex_lock_slow:

- don't stop spinning if there are already waiters on the mutex, there is
no point.
- cosmetic change.

Fix pthread_mutex_trylock() for recursive mutexes. PR lib/38087.
Based on a patch provided by rafal@.

branches: 1.45.2;
Adjust mutex/rwlock definitions to match reality now that there is only
one implementation of each. PR lib/38030.

- Remove libpthread's atomic ops.
- Remove the old spinlock-based mutex and rwlock implementations.
- Use the atomic ops from libc.

Make this compile after Andrew's fix (remove now-unused variables)

pthread_mutex_lock_slow: remove "deliberate deadlock" code that's not
needed with 1:1. PR lib/37524.

add missing static decls.

machine/lock.h, not sys/lock.h

- Fix pthread_rwlock_trywrlock() which was broken.

- Add new functions: pthread_mutex_held_np, mutex_owner_np, rwlock_held_np,
rwlock_wrheld_np, rwlock_rdheld_np. These match the kernel's locking
primitives and can be used when porting kernel code to userspace.

- Always create LWPs detached. Do join/exit sync mostly in userland. When
looped on a dual core box this seems ~30% quicker than using lwp_wait().
Reduce number of lock acquire/release ops during thread exit.

Remove the debuglog stuff. ktrace is more useful now.

For PR bin/37347:

- Override __libc_thr_init() instead of using our own constructor.
- Add pthread__getenv() and use instead of getenv(). This is used before
we are up and running and unfortunatley getenv() takes locks.

Other changes:

- Cache the spinlock vectors in pthread__st. Internal spinlock operations
now take 1 function call instead of 3 (i386).
- Use pthread__self() internally, not pthread_self().
- Use __attribute__ ((visibility("hidden"))) in some places.
- Kill PTHREAD_MAIN_DEBUG.

Add a per-mutex deferred wakeup flag so that threads doing something like
the following do not wake other threads early:

pthread_mutex_lock(&mutex);
pthread_cond_broadcast(&cond);
foo = malloc(100); /* takes libc mutexes */
pthread_mutex_unlock(&mutex);

Fix inverted test after merge of nick-csl-alignment.

Merge nick-csl-alignment.

- Get rid of self->pt_mutexhint and use pthread__mutex_owned() instead.
- Update some comments and fix minor bugs. Minor cosmetic changes.
- Replace some spinlocks with mutexes and rwlocks.
- Change the process private semaphores to use mutexes and condition
variables instead of doing the synchronization directly. Spinlocks
are no longer used by the semaphore code.

- Don't take the mutex's spinlock (ptr_interlock) in pthread_cond_wait().
Instead, make the deferred wakeup list a per-thread array and pass down
the lwpid_t's that way.

- In pthread_cond_wait(), take the mutex before dealing with early wakeup.
In this way there should never be contention on the CV's spinlock if
the app follows POSIX rules (there should only be contention on the
user-provided mutex).

- Add a port of the kernel's rwlocks. The rwlock's spinlock is only taken if
there is contention. This is enabled where atomic ops are available. Right
now that is only i386 and amd64 because I don't have other hardware to
test with. It's trivial to add stubs for other architectures as long as
they have compare-and-swap. When we have proper atomic ops the old rwlock
code can be removed.

- Add a new mutex implementation that's similar to the kernel's mutexes, but
uses compare-and-swap to maintain the waiters list, so no spinlocks are
involved. Same caveats apply as for the rwlocks.

branches: 1.31.2;
One more micro-optimization..

Trim fat off libpthread internal spinlock operations. Makes a mesurable
improvement across the board.

branches: 1.29.2;
Some significant performance improvements, and a fix for a race with pthread
detach/join.

- Make mutex acquire spin for a short time, as done with spinlocks.
- Make the number of spins controllable with the env var PTHREAD_NSPINS.
- Reduce the amount of time that libpthread internal spinlocks are held.
- Rely more on the barrier effects of park/unpark to avoid taking spinlocks.
- Simplify the locking around pthreads and the global queues.
- Align per-thread sync data on a 128 byte boundary.
- Offset thread stacks by a small amount to try and reduce cache thrash.

branches: 1.28.2;
- Test+branch is usually cheaper than making an indirect function call,
so avoid making them.
- When parking an LWP on a condition variable, point the hint argument at
the mutex's waiters queue. Chances are we will be awoken from that later.

- Maintain a per-thread pointer to the last mutex acquired by the app, to
be used only as as a hint. Clear the pointer when releasing the mutex.
- When releasing a mutex, wake all waiters. Makes it possible to tranfer
waiters from another object to a mutex.

Update to match changed pthread__park() interface.

Remove the PTHREAD_SA option. If M:N threads is reimplemented it's
better off done with a seperate library.

Fix bugs with and improve upon previous.

Conditionalised support for 1:1 threads. Needs associated kernel changes
and more work to be useful.

branches: 1.22.2; 1.22.4;
Close a window in which we can not notice a recently-slept-on-our
mutex thread, thus leaving a thread sleeping on an unlocked mutex.

Reviewed by myself and Christos.

Problem reported by Arne H. Juul, arnej at pvv dot ntnu dot no,
in PR 26208. This fix represents option 1 presented in the PR.

branches: 1.21.2;
starting the pthread library (ie. calling pthread__start()) before
any threads are created turned out to be not such a good idea.
there are stronger requirements on what has to work in a forked child
while a process is still single-threaded. so take all that stuff
back out and fix the problems with single-threaded programs that
are linked with libpthread differently, by checking if the library
has been started and doing completely different stuff if it hasn't been:
- for pthread_rwlock_timedrdlock(), just fail with EDEADLK immediately.
- for sem_wait(), the only thing that can unlock the semaphore is a
signal handler, so use sigsuspend() to wait for a signal.
- for pthread_mutex_lock_slow(), just go into an infinite loop
waiting for signals.

I also noticed that there's a "sem2" test that has never worked in its
single-threaded form. the problem there is that a signal handler tries
to take a sem_t interlock which is already held when the signal is received.
fix this too, by adding a single-threaded case for sig_trywait() that
blocks signals instead of using the userland interlock.

in pthread_mutex_lock_slow(), pthread_rwlock_timedrdlock() and sem_wait(),
call pthread__start() if it hasn't already been called. this avoids
an internal assertion from the library if these routines are used
before any threads are created and they need to sleep.
fixes PR 20256, PR 24241, PR 25722, PR 26096.

Add cancellation protection to pthread_once() as per the standard; if
the once routine is cancelled, the effect on once_control is as if
pthread_once() was never called.

Bug report and fix from PR lib/30734.

branches: 1.18.6;
add libpthread part of concurrency support for SA on MP systems
- enable concurrency according to environment variable PTHREAD_CONCURRENCY
- add idle VP wakeup if there are additional jobs and idle VPs
- make reidlequeue per VP
- enable spinning for locks
- fix race condition in alarm processing
- fix race condition in mutex locking
- make debugging output line buffered and add VP prefix to debug lines

Update mutex/rwlock/sem code to match recent change in cond code.

add missing notreached lint comment.

Separately report unlocking an unlocked mutex and unlocking a mutex
owned by another thread.

Use pthread__error() instead of pthread__abort().

More agressive assertions. Change a number of detected conditions, which
officially have undefined behavior, from returning an error code to raising
an assertion failure.
Also, don't bother to explicitly test for (illegal) null pointers and return
an error; they'll bomb out soon enough.

pthread_mutex_trylock(): It's not an error to call trylock() on a
mutex already locked by the calling thread, even for non-recursive
mutexes.

When a thread sleeps on a mutex or cv, have it put itself on the
front of the sleep queue rather than the back. This is more
cache-friendly behavior and within the (lack of) constraints on wakeup
ordering imposed on equal-priority threads.

add __RCSID()

Remove unnecessary inclusion of <assert.h>.

Mutex tuneup.

* Use a double-checked locking technique to avoid taking
the interlock in pthread_mutex_unlock().

* In pthread_mutex_lock() and pthread_mutex_trylock(), only store the
stack pointer, not the thread ID, in ptm_owner. Do the translation
to a thread ID in the slow-lock, errorcheck, and recursive mutex
cases rather than in the common path.

* Juggle where pthread__self() is called, to move it out of the fast path.

Overall, this means that neither pthread_self() nor
pthread_spin[un]lock() are used in the course of locking and unlocking
an uncontested mutex. Speeds up the fast path by 40-50%, and
eliminates about 98% of spinlocks used by a couple of large threaded
applications.

(Still a GET_MUTEX_PRIVATE() in the fast path... perhaps the type
should be in the main body of the mutex).

Add debug counters for mutex and condvar operations.

Need to include <string.h> for memcmp() prototype.

Fix typo.

Add mutexattr_init(), mutexattr_destroy(), and mutexattr_settype() (recursive
mutexes are used by the X libraries).

de-lint

Merge the nathanw_sa branch.

branches: 1.1.2;
file pthread_mutex.c was initially added on branch nathanw_sa.

libpthread: Move namespacing include to top of .c files.

Stuff like libc's namespace.h, or atomic_op_namespace.h, which does
namespacing tricks like `#define atomic_cas_uint _atomic_cas_uint',
has to go at the top of each .c file. If it goes in the middle, it
might be too late to affect the declarations, and result in compile
errors.

I tripped over this by including <sys/atomic.h> in mips
<machine/lock.h>.

(Maybe we should create a new pthread_namespace.h file for the
purpose, but this'll do for now.)

Adjust memory barriers.

- Make pthread_condvar and pthread_mutex work on the stack rather than in
pthread_t, so there's less chance of bad things happening if someone calls
(for example) pthread_cond_broadcast() from a signal handler.

- Remove all the deferred waiter handling except for the one case that really
matters which is transferring waiters from condvar -> mutex on wakeup, and
do that by splicing the condvar's waiters onto the mutex.

- Remove the mutex waiters bit as it's another complication that's not
strictly needed.

Deal with a couple of problems with threads being awoken early due to
timeouts or cancellation where:

- The restarting thread calls _lwp_exit() before another thread gets around
to waking it with _lwp_unpark(), leading to ESRCH (observed by joerg@).
(I may have removed a similar check mistakenly over the weekend.)

- The restarting thread considers itself gone off the sleep queue but
at the same time another thread is part way through waking it, and hasn't
fully completed that operation yet by setting thread->pt_mutexwait = 0.
I think that could have potentially lead to the list of waiters getting
messed up given the right circumstances.

In the interests of reliability simplify waiter handling more and redo
condvars to manage the list of waiters with atomic ops.

- Try to eliminate a hang in "parked" I've been seeing while stress testing.
Centralise wakeup of deferred waiters in pthread__clear_waiters() and use
throughout libpthread. Make fewer assumptions. Be more conservative in
pthread_mutex when dealing with pending waiters.

- Remove the "hint" argument everywhere since the kernel doesn't use it any
more.

Revert "Enhance the pthread(3) + malloc(3) init model"

It is reported to hand on aarch64 with gzip.

Enhance the pthread(3) + malloc(3) init model

Separate the pthread_atfork(3) call from pthread_tsd_init()
and move it into a distinct function.

Call inside pthread__init() late TSD initialization route, just after
"pthread_atfork(NULL, NULL, pthread__fork_callback);".

Document that malloc(3) initialization is now controlled again and called
during the first pthread_atfork(3) call.

Remove #if 0 code from pthread_mutex.c as we no longer initialize malloc
prematurely.

Revert previous

'git grep' breaks now.

Remove 'ifdef 0' hacks

It is no longer needed as the proper fix avoiding premature malloc()
landed the sources.

Refactor libpthread checks for invalid arguments

Switch from manual functions to pthread__error().

In the same spirit as the previous pthread_mutex_init change for jemalloc,
make pthread_mutexattr_init do always a full initialization, so that the
attribute that will be used later when we become threaded is properly
initialized.

Use pthread_mutexattr_t and pthread_mutex_t magic fields

Validate _PT_MUTEX_MAGIC in pthread_mutex_t and _PT_MUTEXATTR_MAGIC
in pthread_mutexattr_t accordingly.

Mark destroyed pthread_mutexattr_t as dead

Adjustment to previous: don't call _lwp_unpark_all() with nwaiters == 0.

pthread__mutex_unlock_slow(): ignore the DEFERRED bit. It's only purpose
is to get the thread to go through the slow path. If there are waiters,
process them there and then. Should not affect well behaved apps. Maybe
of help for:

PR bin/50350: rump/rumpkern/t_sp/stress_{long,short} fail on Core 2 Quad

Rip out some very ambitious optimisations around pthread_mutex that are
don't buy much. This stuff is hard enough to get right in the kernel let
alone userspace, and I don't trust that it's right.

branches: 1.65.2;
Jemalloc initializes mutexes before we become threaded and expects to use
them later.

branches: 1.64.4;

Deal with more lwp_park() timestamp unconsting

Don't spin if we already own the mutex, otherwise we will get stuck spinning
forever, fixes timemutex{1,2} tests.

Use anonymous union for ptm_ceiling and old __pthread_spin_t field to
maintain backward compatibility and fix hppa build. hppa has an non-
integer type __pthread_spin_t

KNF

branches: 1.60.2;
GSoC 2016 Charles Cui: Implement thread priority protection based on work
by Andy Doran. Also document the get/set pshared thread calls as not
implemented, and add a skeleton implementation that is disabled.
XXX: document _sched_protect(2).

pthread__mutex_lock_slow: fix the handling of a potential race with the
non-interlocked CAS in the fast unlock path -- it is unsafe to test for
the waiters-bit while the owner thread is running, we have to spin for
the owner or its state change to be sure about the presence of the bit.
Split off the logic into the pthread__mutex_setwaiters() routine.

This is a partial fix to the named lockup problem (also see PR/44756).
It seems there is another race which can be reproduced on faster CPUs.

remove compatibility code for handling CLOCK_MONOTONIC and handle it in the
syscall directly.

PR/44756: Sad Clouds: Prevent leakage of errno = ESRCH from _lwp_park. This
has two parts:
- in pthread_cond_timedwait() if the thread we are trying to unpark
exited, retry the the _lwp_park call without it.
- pthread_mutex() was affecting errno since it is calling _lwp_park()
from pthread_mutex_lock_slow(). preserve the original errno.
Note that the example problem still causes an occassional deadlock on machines
with many CPUs and it is the same deadlock we observe with named.

- Allow libpthread to be dlopened again, by providing libc stubs to libpthread.
- Fail if the dlopened libpthread does pthread_create(). From manu@
- Discussed at length in the mailing lists; approved by core@
- This was chosen as the least intrusive patch that will provide
the necessary functionality.
XXX: pullup to 6

add comments
whitespace

branches: 1.54.2;
Add a pthread__smt_wake and add support for it on arm along with
pthread__smt_pause. These are implemented using the ARM instructions
SEV (wake) and WFE (pause). These are treated as NOPs on ARM CPUs that
don't support them.

Move the pthread_once alias where it belongs.

Move pthread_once implementation into a separate file, it doesn't depend
on the mutex implementation in any way.

branches: 1.51.2; 1.51.4; 1.51.6; 1.51.22;
Change pthread_mutex_t to use the amount of space as and be congruent to
the version used in the SA version of pthreads. This preserves binary
compatibility between both versions of the library.

PR lib/38741 priority inversion in libpthread breaks apps that use
SCHED_FIFO threads

- Change condvar sync so that we never take the condvar's spinlock without
first holding the caller-provided mutex. Previously, the spinlock was only
taken without the mutex in an error path, but it was enough to trigger the
problem described in the PR.

- Even with this change, applications calling pthread_cond_signal/broadcast
without holding the interlocking mutex are still subject to the problem
described in the PR. POSIX discourages this saying that it leads to
undefined scheduling behaviour, which seems good enough for the time being.

- Elsewhere, use a hash of mutexes instead of per-object spinlocks to
synchronize entry/exit from sleep queues.

- Simplify how sleep queues are maintained.

Add some general comments about the mutex implementation.

branches: 1.48.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.47.2;
pthread__mutex_lock_slow:

- don't stop spinning if there are already waiters on the mutex, there is
no point.
- cosmetic change.

Fix pthread_mutex_trylock() for recursive mutexes. PR lib/38087.
Based on a patch provided by rafal@.

branches: 1.45.2;
Adjust mutex/rwlock definitions to match reality now that there is only
one implementation of each. PR lib/38030.

- Remove libpthread's atomic ops.
- Remove the old spinlock-based mutex and rwlock implementations.
- Use the atomic ops from libc.

Make this compile after Andrew's fix (remove now-unused variables)

pthread_mutex_lock_slow: remove "deliberate deadlock" code that's not
needed with 1:1. PR lib/37524.

add missing static decls.

machine/lock.h, not sys/lock.h

- Fix pthread_rwlock_trywrlock() which was broken.

- Add new functions: pthread_mutex_held_np, mutex_owner_np, rwlock_held_np,
rwlock_wrheld_np, rwlock_rdheld_np. These match the kernel's locking
primitives and can be used when porting kernel code to userspace.

- Always create LWPs detached. Do join/exit sync mostly in userland. When
looped on a dual core box this seems ~30% quicker than using lwp_wait().
Reduce number of lock acquire/release ops during thread exit.

Remove the debuglog stuff. ktrace is more useful now.

For PR bin/37347:

- Override __libc_thr_init() instead of using our own constructor.
- Add pthread__getenv() and use instead of getenv(). This is used before
we are up and running and unfortunatley getenv() takes locks.

Other changes:

- Cache the spinlock vectors in pthread__st. Internal spinlock operations
now take 1 function call instead of 3 (i386).
- Use pthread__self() internally, not pthread_self().
- Use __attribute__ ((visibility("hidden"))) in some places.
- Kill PTHREAD_MAIN_DEBUG.

Add a per-mutex deferred wakeup flag so that threads doing something like
the following do not wake other threads early:

pthread_mutex_lock(&mutex);
pthread_cond_broadcast(&cond);
foo = malloc(100); /* takes libc mutexes */
pthread_mutex_unlock(&mutex);

Fix inverted test after merge of nick-csl-alignment.

Merge nick-csl-alignment.

- Get rid of self->pt_mutexhint and use pthread__mutex_owned() instead.
- Update some comments and fix minor bugs. Minor cosmetic changes.
- Replace some spinlocks with mutexes and rwlocks.
- Change the process private semaphores to use mutexes and condition
variables instead of doing the synchronization directly. Spinlocks
are no longer used by the semaphore code.

- Don't take the mutex's spinlock (ptr_interlock) in pthread_cond_wait().
Instead, make the deferred wakeup list a per-thread array and pass down
the lwpid_t's that way.

- In pthread_cond_wait(), take the mutex before dealing with early wakeup.
In this way there should never be contention on the CV's spinlock if
the app follows POSIX rules (there should only be contention on the
user-provided mutex).

- Add a port of the kernel's rwlocks. The rwlock's spinlock is only taken if
there is contention. This is enabled where atomic ops are available. Right
now that is only i386 and amd64 because I don't have other hardware to
test with. It's trivial to add stubs for other architectures as long as
they have compare-and-swap. When we have proper atomic ops the old rwlock
code can be removed.

- Add a new mutex implementation that's similar to the kernel's mutexes, but
uses compare-and-swap to maintain the waiters list, so no spinlocks are
involved. Same caveats apply as for the rwlocks.

branches: 1.31.2;
One more micro-optimization..

Trim fat off libpthread internal spinlock operations. Makes a mesurable
improvement across the board.

branches: 1.29.2;
Some significant performance improvements, and a fix for a race with pthread
detach/join.

- Make mutex acquire spin for a short time, as done with spinlocks.
- Make the number of spins controllable with the env var PTHREAD_NSPINS.
- Reduce the amount of time that libpthread internal spinlocks are held.
- Rely more on the barrier effects of park/unpark to avoid taking spinlocks.
- Simplify the locking around pthreads and the global queues.
- Align per-thread sync data on a 128 byte boundary.
- Offset thread stacks by a small amount to try and reduce cache thrash.

branches: 1.28.2;
- Test+branch is usually cheaper than making an indirect function call,
so avoid making them.
- When parking an LWP on a condition variable, point the hint argument at
the mutex's waiters queue. Chances are we will be awoken from that later.

- Maintain a per-thread pointer to the last mutex acquired by the app, to
be used only as as a hint. Clear the pointer when releasing the mutex.
- When releasing a mutex, wake all waiters. Makes it possible to tranfer
waiters from another object to a mutex.

Update to match changed pthread__park() interface.

Remove the PTHREAD_SA option. If M:N threads is reimplemented it's
better off done with a seperate library.

Fix bugs with and improve upon previous.

Conditionalised support for 1:1 threads. Needs associated kernel changes
and more work to be useful.

branches: 1.22.2; 1.22.4;
Close a window in which we can not notice a recently-slept-on-our
mutex thread, thus leaving a thread sleeping on an unlocked mutex.

Reviewed by myself and Christos.

Problem reported by Arne H. Juul, arnej at pvv dot ntnu dot no,
in PR 26208. This fix represents option 1 presented in the PR.

branches: 1.21.2;
starting the pthread library (ie. calling pthread__start()) before
any threads are created turned out to be not such a good idea.
there are stronger requirements on what has to work in a forked child
while a process is still single-threaded. so take all that stuff
back out and fix the problems with single-threaded programs that
are linked with libpthread differently, by checking if the library
has been started and doing completely different stuff if it hasn't been:
- for pthread_rwlock_timedrdlock(), just fail with EDEADLK immediately.
- for sem_wait(), the only thing that can unlock the semaphore is a
signal handler, so use sigsuspend() to wait for a signal.
- for pthread_mutex_lock_slow(), just go into an infinite loop
waiting for signals.

I also noticed that there's a "sem2" test that has never worked in its
single-threaded form. the problem there is that a signal handler tries
to take a sem_t interlock which is already held when the signal is received.
fix this too, by adding a single-threaded case for sig_trywait() that
blocks signals instead of using the userland interlock.

in pthread_mutex_lock_slow(), pthread_rwlock_timedrdlock() and sem_wait(),
call pthread__start() if it hasn't already been called. this avoids
an internal assertion from the library if these routines are used
before any threads are created and they need to sleep.
fixes PR 20256, PR 24241, PR 25722, PR 26096.

Add cancellation protection to pthread_once() as per the standard; if
the once routine is cancelled, the effect on once_control is as if
pthread_once() was never called.

Bug report and fix from PR lib/30734.

branches: 1.18.6;
add libpthread part of concurrency support for SA on MP systems
- enable concurrency according to environment variable PTHREAD_CONCURRENCY
- add idle VP wakeup if there are additional jobs and idle VPs
- make reidlequeue per VP
- enable spinning for locks
- fix race condition in alarm processing
- fix race condition in mutex locking
- make debugging output line buffered and add VP prefix to debug lines

Update mutex/rwlock/sem code to match recent change in cond code.

add missing notreached lint comment.

Separately report unlocking an unlocked mutex and unlocking a mutex
owned by another thread.

Use pthread__error() instead of pthread__abort().

More agressive assertions. Change a number of detected conditions, which
officially have undefined behavior, from returning an error code to raising
an assertion failure.
Also, don't bother to explicitly test for (illegal) null pointers and return
an error; they'll bomb out soon enough.

pthread_mutex_trylock(): It's not an error to call trylock() on a
mutex already locked by the calling thread, even for non-recursive
mutexes.

When a thread sleeps on a mutex or cv, have it put itself on the
front of the sleep queue rather than the back. This is more
cache-friendly behavior and within the (lack of) constraints on wakeup
ordering imposed on equal-priority threads.

add __RCSID()

Remove unnecessary inclusion of <assert.h>.

Mutex tuneup.

* Use a double-checked locking technique to avoid taking
the interlock in pthread_mutex_unlock().

* In pthread_mutex_lock() and pthread_mutex_trylock(), only store the
stack pointer, not the thread ID, in ptm_owner. Do the translation
to a thread ID in the slow-lock, errorcheck, and recursive mutex
cases rather than in the common path.

* Juggle where pthread__self() is called, to move it out of the fast path.

Overall, this means that neither pthread_self() nor
pthread_spin[un]lock() are used in the course of locking and unlocking
an uncontested mutex. Speeds up the fast path by 40-50%, and
eliminates about 98% of spinlocks used by a couple of large threaded
applications.

(Still a GET_MUTEX_PRIVATE() in the fast path... perhaps the type
should be in the main body of the mutex).

Add debug counters for mutex and condvar operations.

Need to include <string.h> for memcmp() prototype.

Fix typo.

Add mutexattr_init(), mutexattr_destroy(), and mutexattr_settype() (recursive
mutexes are used by the X libraries).

de-lint

Merge the nathanw_sa branch.

branches: 1.1.2;
file pthread_mutex.c was initially added on branch nathanw_sa.

Adjust memory barriers.

- Make pthread_condvar and pthread_mutex work on the stack rather than in
pthread_t, so there's less chance of bad things happening if someone calls
(for example) pthread_cond_broadcast() from a signal handler.

- Remove all the deferred waiter handling except for the one case that really
matters which is transferring waiters from condvar -> mutex on wakeup, and
do that by splicing the condvar's waiters onto the mutex.

- Remove the mutex waiters bit as it's another complication that's not
strictly needed.

Deal with a couple of problems with threads being awoken early due to
timeouts or cancellation where:

- The restarting thread calls _lwp_exit() before another thread gets around
to waking it with _lwp_unpark(), leading to ESRCH (observed by joerg@).
(I may have removed a similar check mistakenly over the weekend.)

- The restarting thread considers itself gone off the sleep queue but
at the same time another thread is part way through waking it, and hasn't
fully completed that operation yet by setting thread->pt_mutexwait = 0.
I think that could have potentially lead to the list of waiters getting
messed up given the right circumstances.

In the interests of reliability simplify waiter handling more and redo
condvars to manage the list of waiters with atomic ops.

- Try to eliminate a hang in "parked" I've been seeing while stress testing.
Centralise wakeup of deferred waiters in pthread__clear_waiters() and use
throughout libpthread. Make fewer assumptions. Be more conservative in
pthread_mutex when dealing with pending waiters.

- Remove the "hint" argument everywhere since the kernel doesn't use it any
more.

Revert "Enhance the pthread(3) + malloc(3) init model"

It is reported to hand on aarch64 with gzip.

Enhance the pthread(3) + malloc(3) init model

Separate the pthread_atfork(3) call from pthread_tsd_init()
and move it into a distinct function.

Call inside pthread__init() late TSD initialization route, just after
"pthread_atfork(NULL, NULL, pthread__fork_callback);".

Document that malloc(3) initialization is now controlled again and called
during the first pthread_atfork(3) call.

Remove #if 0 code from pthread_mutex.c as we no longer initialize malloc
prematurely.

Revert previous

'git grep' breaks now.

Remove 'ifdef 0' hacks

It is no longer needed as the proper fix avoiding premature malloc()
landed the sources.

Refactor libpthread checks for invalid arguments

Switch from manual functions to pthread__error().

In the same spirit as the previous pthread_mutex_init change for jemalloc,
make pthread_mutexattr_init do always a full initialization, so that the
attribute that will be used later when we become threaded is properly
initialized.

Use pthread_mutexattr_t and pthread_mutex_t magic fields

Validate _PT_MUTEX_MAGIC in pthread_mutex_t and _PT_MUTEXATTR_MAGIC
in pthread_mutexattr_t accordingly.

Mark destroyed pthread_mutexattr_t as dead

Adjustment to previous: don't call _lwp_unpark_all() with nwaiters == 0.

pthread__mutex_unlock_slow(): ignore the DEFERRED bit. It's only purpose
is to get the thread to go through the slow path. If there are waiters,
process them there and then. Should not affect well behaved apps. Maybe
of help for:

PR bin/50350: rump/rumpkern/t_sp/stress_{long,short} fail on Core 2 Quad

Rip out some very ambitious optimisations around pthread_mutex that are
don't buy much. This stuff is hard enough to get right in the kernel let
alone userspace, and I don't trust that it's right.

branches: 1.65.2;
Jemalloc initializes mutexes before we become threaded and expects to use
them later.

branches: 1.64.4;

Deal with more lwp_park() timestamp unconsting

Don't spin if we already own the mutex, otherwise we will get stuck spinning
forever, fixes timemutex{1,2} tests.

Use anonymous union for ptm_ceiling and old __pthread_spin_t field to
maintain backward compatibility and fix hppa build. hppa has an non-
integer type __pthread_spin_t

KNF

branches: 1.60.2;
GSoC 2016 Charles Cui: Implement thread priority protection based on work
by Andy Doran. Also document the get/set pshared thread calls as not
implemented, and add a skeleton implementation that is disabled.
XXX: document _sched_protect(2).

pthread__mutex_lock_slow: fix the handling of a potential race with the
non-interlocked CAS in the fast unlock path -- it is unsafe to test for
the waiters-bit while the owner thread is running, we have to spin for
the owner or its state change to be sure about the presence of the bit.
Split off the logic into the pthread__mutex_setwaiters() routine.

This is a partial fix to the named lockup problem (also see PR/44756).
It seems there is another race which can be reproduced on faster CPUs.

remove compatibility code for handling CLOCK_MONOTONIC and handle it in the
syscall directly.

PR/44756: Sad Clouds: Prevent leakage of errno = ESRCH from _lwp_park. This
has two parts:
- in pthread_cond_timedwait() if the thread we are trying to unpark
exited, retry the the _lwp_park call without it.
- pthread_mutex() was affecting errno since it is calling _lwp_park()
from pthread_mutex_lock_slow(). preserve the original errno.
Note that the example problem still causes an occassional deadlock on machines
with many CPUs and it is the same deadlock we observe with named.

- Allow libpthread to be dlopened again, by providing libc stubs to libpthread.
- Fail if the dlopened libpthread does pthread_create(). From manu@
- Discussed at length in the mailing lists; approved by core@
- This was chosen as the least intrusive patch that will provide
the necessary functionality.
XXX: pullup to 6

add comments
whitespace

branches: 1.54.2;
Add a pthread__smt_wake and add support for it on arm along with
pthread__smt_pause. These are implemented using the ARM instructions
SEV (wake) and WFE (pause). These are treated as NOPs on ARM CPUs that
don't support them.

Move the pthread_once alias where it belongs.

Move pthread_once implementation into a separate file, it doesn't depend
on the mutex implementation in any way.

branches: 1.51.2; 1.51.4; 1.51.6; 1.51.22;
Change pthread_mutex_t to use the amount of space as and be congruent to
the version used in the SA version of pthreads. This preserves binary
compatibility between both versions of the library.

PR lib/38741 priority inversion in libpthread breaks apps that use
SCHED_FIFO threads

- Change condvar sync so that we never take the condvar's spinlock without
first holding the caller-provided mutex. Previously, the spinlock was only
taken without the mutex in an error path, but it was enough to trigger the
problem described in the PR.

- Even with this change, applications calling pthread_cond_signal/broadcast
without holding the interlocking mutex are still subject to the problem
described in the PR. POSIX discourages this saying that it leads to
undefined scheduling behaviour, which seems good enough for the time being.

- Elsewhere, use a hash of mutexes instead of per-object spinlocks to
synchronize entry/exit from sleep queues.

- Simplify how sleep queues are maintained.

Add some general comments about the mutex implementation.

branches: 1.48.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.47.2;
pthread__mutex_lock_slow:

- don't stop spinning if there are already waiters on the mutex, there is
no point.
- cosmetic change.

Fix pthread_mutex_trylock() for recursive mutexes. PR lib/38087.
Based on a patch provided by rafal@.

branches: 1.45.2;
Adjust mutex/rwlock definitions to match reality now that there is only
one implementation of each. PR lib/38030.

- Remove libpthread's atomic ops.
- Remove the old spinlock-based mutex and rwlock implementations.
- Use the atomic ops from libc.

Make this compile after Andrew's fix (remove now-unused variables)

pthread_mutex_lock_slow: remove "deliberate deadlock" code that's not
needed with 1:1. PR lib/37524.

add missing static decls.

machine/lock.h, not sys/lock.h

- Fix pthread_rwlock_trywrlock() which was broken.

- Add new functions: pthread_mutex_held_np, mutex_owner_np, rwlock_held_np,
rwlock_wrheld_np, rwlock_rdheld_np. These match the kernel's locking
primitives and can be used when porting kernel code to userspace.

- Always create LWPs detached. Do join/exit sync mostly in userland. When
looped on a dual core box this seems ~30% quicker than using lwp_wait().
Reduce number of lock acquire/release ops during thread exit.

Remove the debuglog stuff. ktrace is more useful now.

For PR bin/37347:

- Override __libc_thr_init() instead of using our own constructor.
- Add pthread__getenv() and use instead of getenv(). This is used before
we are up and running and unfortunatley getenv() takes locks.

Other changes:

- Cache the spinlock vectors in pthread__st. Internal spinlock operations
now take 1 function call instead of 3 (i386).
- Use pthread__self() internally, not pthread_self().
- Use __attribute__ ((visibility("hidden"))) in some places.
- Kill PTHREAD_MAIN_DEBUG.

Add a per-mutex deferred wakeup flag so that threads doing something like
the following do not wake other threads early:

pthread_mutex_lock(&mutex);
pthread_cond_broadcast(&cond);
foo = malloc(100); /* takes libc mutexes */
pthread_mutex_unlock(&mutex);

Fix inverted test after merge of nick-csl-alignment.

Merge nick-csl-alignment.

- Get rid of self->pt_mutexhint and use pthread__mutex_owned() instead.
- Update some comments and fix minor bugs. Minor cosmetic changes.
- Replace some spinlocks with mutexes and rwlocks.
- Change the process private semaphores to use mutexes and condition
variables instead of doing the synchronization directly. Spinlocks
are no longer used by the semaphore code.

- Don't take the mutex's spinlock (ptr_interlock) in pthread_cond_wait().
Instead, make the deferred wakeup list a per-thread array and pass down
the lwpid_t's that way.

- In pthread_cond_wait(), take the mutex before dealing with early wakeup.
In this way there should never be contention on the CV's spinlock if
the app follows POSIX rules (there should only be contention on the
user-provided mutex).

- Add a port of the kernel's rwlocks. The rwlock's spinlock is only taken if
there is contention. This is enabled where atomic ops are available. Right
now that is only i386 and amd64 because I don't have other hardware to
test with. It's trivial to add stubs for other architectures as long as
they have compare-and-swap. When we have proper atomic ops the old rwlock
code can be removed.

- Add a new mutex implementation that's similar to the kernel's mutexes, but
uses compare-and-swap to maintain the waiters list, so no spinlocks are
involved. Same caveats apply as for the rwlocks.

branches: 1.31.2;
One more micro-optimization..

Trim fat off libpthread internal spinlock operations. Makes a mesurable
improvement across the board.

branches: 1.29.2;
Some significant performance improvements, and a fix for a race with pthread
detach/join.

- Make mutex acquire spin for a short time, as done with spinlocks.
- Make the number of spins controllable with the env var PTHREAD_NSPINS.
- Reduce the amount of time that libpthread internal spinlocks are held.
- Rely more on the barrier effects of park/unpark to avoid taking spinlocks.
- Simplify the locking around pthreads and the global queues.
- Align per-thread sync data on a 128 byte boundary.
- Offset thread stacks by a small amount to try and reduce cache thrash.

branches: 1.28.2;
- Test+branch is usually cheaper than making an indirect function call,
so avoid making them.
- When parking an LWP on a condition variable, point the hint argument at
the mutex's waiters queue. Chances are we will be awoken from that later.

- Maintain a per-thread pointer to the last mutex acquired by the app, to
be used only as as a hint. Clear the pointer when releasing the mutex.
- When releasing a mutex, wake all waiters. Makes it possible to tranfer
waiters from another object to a mutex.

Update to match changed pthread__park() interface.

Remove the PTHREAD_SA option. If M:N threads is reimplemented it's
better off done with a seperate library.

Fix bugs with and improve upon previous.

Conditionalised support for 1:1 threads. Needs associated kernel changes
and more work to be useful.

branches: 1.22.2; 1.22.4;
Close a window in which we can not notice a recently-slept-on-our
mutex thread, thus leaving a thread sleeping on an unlocked mutex.

Reviewed by myself and Christos.

Problem reported by Arne H. Juul, arnej at pvv dot ntnu dot no,
in PR 26208. This fix represents option 1 presented in the PR.

branches: 1.21.2;
starting the pthread library (ie. calling pthread__start()) before
any threads are created turned out to be not such a good idea.
there are stronger requirements on what has to work in a forked child
while a process is still single-threaded. so take all that stuff
back out and fix the problems with single-threaded programs that
are linked with libpthread differently, by checking if the library
has been started and doing completely different stuff if it hasn't been:
- for pthread_rwlock_timedrdlock(), just fail with EDEADLK immediately.
- for sem_wait(), the only thing that can unlock the semaphore is a
signal handler, so use sigsuspend() to wait for a signal.
- for pthread_mutex_lock_slow(), just go into an infinite loop
waiting for signals.

I also noticed that there's a "sem2" test that has never worked in its
single-threaded form. the problem there is that a signal handler tries
to take a sem_t interlock which is already held when the signal is received.
fix this too, by adding a single-threaded case for sig_trywait() that
blocks signals instead of using the userland interlock.

in pthread_mutex_lock_slow(), pthread_rwlock_timedrdlock() and sem_wait(),
call pthread__start() if it hasn't already been called. this avoids
an internal assertion from the library if these routines are used
before any threads are created and they need to sleep.
fixes PR 20256, PR 24241, PR 25722, PR 26096.

Add cancellation protection to pthread_once() as per the standard; if
the once routine is cancelled, the effect on once_control is as if
pthread_once() was never called.

Bug report and fix from PR lib/30734.

branches: 1.18.6;
add libpthread part of concurrency support for SA on MP systems
- enable concurrency according to environment variable PTHREAD_CONCURRENCY
- add idle VP wakeup if there are additional jobs and idle VPs
- make reidlequeue per VP
- enable spinning for locks
- fix race condition in alarm processing
- fix race condition in mutex locking
- make debugging output line buffered and add VP prefix to debug lines

Update mutex/rwlock/sem code to match recent change in cond code.

add missing notreached lint comment.

Separately report unlocking an unlocked mutex and unlocking a mutex
owned by another thread.

Use pthread__error() instead of pthread__abort().

More agressive assertions. Change a number of detected conditions, which
officially have undefined behavior, from returning an error code to raising
an assertion failure.
Also, don't bother to explicitly test for (illegal) null pointers and return
an error; they'll bomb out soon enough.

pthread_mutex_trylock(): It's not an error to call trylock() on a
mutex already locked by the calling thread, even for non-recursive
mutexes.

When a thread sleeps on a mutex or cv, have it put itself on the
front of the sleep queue rather than the back. This is more
cache-friendly behavior and within the (lack of) constraints on wakeup
ordering imposed on equal-priority threads.

add __RCSID()

Remove unnecessary inclusion of <assert.h>.

Mutex tuneup.

* Use a double-checked locking technique to avoid taking
the interlock in pthread_mutex_unlock().

* In pthread_mutex_lock() and pthread_mutex_trylock(), only store the
stack pointer, not the thread ID, in ptm_owner. Do the translation
to a thread ID in the slow-lock, errorcheck, and recursive mutex
cases rather than in the common path.

* Juggle where pthread__self() is called, to move it out of the fast path.

Overall, this means that neither pthread_self() nor
pthread_spin[un]lock() are used in the course of locking and unlocking
an uncontested mutex. Speeds up the fast path by 40-50%, and
eliminates about 98% of spinlocks used by a couple of large threaded
applications.

(Still a GET_MUTEX_PRIVATE() in the fast path... perhaps the type
should be in the main body of the mutex).

Add debug counters for mutex and condvar operations.

Need to include <string.h> for memcmp() prototype.

Fix typo.

Add mutexattr_init(), mutexattr_destroy(), and mutexattr_settype() (recursive
mutexes are used by the X libraries).

de-lint

Merge the nathanw_sa branch.

branches: 1.1.2;
file pthread_mutex.c was initially added on branch nathanw_sa.

- Make pthread_condvar and pthread_mutex work on the stack rather than in
pthread_t, so there's less chance of bad things happening if someone calls
(for example) pthread_cond_broadcast() from a signal handler.

- Remove all the deferred waiter handling except for the one case that really
matters which is transferring waiters from condvar -> mutex on wakeup, and
do that by splicing the condvar's waiters onto the mutex.

- Remove the mutex waiters bit as it's another complication that's not
strictly needed.

Deal with a couple of problems with threads being awoken early due to
timeouts or cancellation where:

- The restarting thread calls _lwp_exit() before another thread gets around
to waking it with _lwp_unpark(), leading to ESRCH (observed by joerg@).
(I may have removed a similar check mistakenly over the weekend.)

- The restarting thread considers itself gone off the sleep queue but
at the same time another thread is part way through waking it, and hasn't
fully completed that operation yet by setting thread->pt_mutexwait = 0.
I think that could have potentially lead to the list of waiters getting
messed up given the right circumstances.

In the interests of reliability simplify waiter handling more and redo
condvars to manage the list of waiters with atomic ops.

- Try to eliminate a hang in "parked" I've been seeing while stress testing.
Centralise wakeup of deferred waiters in pthread__clear_waiters() and use
throughout libpthread. Make fewer assumptions. Be more conservative in
pthread_mutex when dealing with pending waiters.

- Remove the "hint" argument everywhere since the kernel doesn't use it any
more.

Revert "Enhance the pthread(3) + malloc(3) init model"

It is reported to hand on aarch64 with gzip.

Enhance the pthread(3) + malloc(3) init model

Separate the pthread_atfork(3) call from pthread_tsd_init()
and move it into a distinct function.

Call inside pthread__init() late TSD initialization route, just after
"pthread_atfork(NULL, NULL, pthread__fork_callback);".

Document that malloc(3) initialization is now controlled again and called
during the first pthread_atfork(3) call.

Remove #if 0 code from pthread_mutex.c as we no longer initialize malloc
prematurely.

Revert previous

'git grep' breaks now.

Remove 'ifdef 0' hacks

It is no longer needed as the proper fix avoiding premature malloc()
landed the sources.

Refactor libpthread checks for invalid arguments

Switch from manual functions to pthread__error().

In the same spirit as the previous pthread_mutex_init change for jemalloc,
make pthread_mutexattr_init do always a full initialization, so that the
attribute that will be used later when we become threaded is properly
initialized.

Use pthread_mutexattr_t and pthread_mutex_t magic fields

Validate _PT_MUTEX_MAGIC in pthread_mutex_t and _PT_MUTEXATTR_MAGIC
in pthread_mutexattr_t accordingly.

Mark destroyed pthread_mutexattr_t as dead

Adjustment to previous: don't call _lwp_unpark_all() with nwaiters == 0.

pthread__mutex_unlock_slow(): ignore the DEFERRED bit. It's only purpose
is to get the thread to go through the slow path. If there are waiters,
process them there and then. Should not affect well behaved apps. Maybe
of help for:

PR bin/50350: rump/rumpkern/t_sp/stress_{long,short} fail on Core 2 Quad

Rip out some very ambitious optimisations around pthread_mutex that are
don't buy much. This stuff is hard enough to get right in the kernel let
alone userspace, and I don't trust that it's right.

branches: 1.65.2;
Jemalloc initializes mutexes before we become threaded and expects to use
them later.

branches: 1.64.4;

Deal with more lwp_park() timestamp unconsting

Don't spin if we already own the mutex, otherwise we will get stuck spinning
forever, fixes timemutex{1,2} tests.

Use anonymous union for ptm_ceiling and old __pthread_spin_t field to
maintain backward compatibility and fix hppa build. hppa has an non-
integer type __pthread_spin_t

KNF

branches: 1.60.2;
GSoC 2016 Charles Cui: Implement thread priority protection based on work
by Andy Doran. Also document the get/set pshared thread calls as not
implemented, and add a skeleton implementation that is disabled.
XXX: document _sched_protect(2).

pthread__mutex_lock_slow: fix the handling of a potential race with the
non-interlocked CAS in the fast unlock path -- it is unsafe to test for
the waiters-bit while the owner thread is running, we have to spin for
the owner or its state change to be sure about the presence of the bit.
Split off the logic into the pthread__mutex_setwaiters() routine.

This is a partial fix to the named lockup problem (also see PR/44756).
It seems there is another race which can be reproduced on faster CPUs.

remove compatibility code for handling CLOCK_MONOTONIC and handle it in the
syscall directly.

PR/44756: Sad Clouds: Prevent leakage of errno = ESRCH from _lwp_park. This
has two parts:
- in pthread_cond_timedwait() if the thread we are trying to unpark
exited, retry the the _lwp_park call without it.
- pthread_mutex() was affecting errno since it is calling _lwp_park()
from pthread_mutex_lock_slow(). preserve the original errno.
Note that the example problem still causes an occassional deadlock on machines
with many CPUs and it is the same deadlock we observe with named.

- Allow libpthread to be dlopened again, by providing libc stubs to libpthread.
- Fail if the dlopened libpthread does pthread_create(). From manu@
- Discussed at length in the mailing lists; approved by core@
- This was chosen as the least intrusive patch that will provide
the necessary functionality.
XXX: pullup to 6

add comments
whitespace

branches: 1.54.2;
Add a pthread__smt_wake and add support for it on arm along with
pthread__smt_pause. These are implemented using the ARM instructions
SEV (wake) and WFE (pause). These are treated as NOPs on ARM CPUs that
don't support them.

Move the pthread_once alias where it belongs.

Move pthread_once implementation into a separate file, it doesn't depend
on the mutex implementation in any way.

branches: 1.51.2; 1.51.4; 1.51.6; 1.51.22;
Change pthread_mutex_t to use the amount of space as and be congruent to
the version used in the SA version of pthreads. This preserves binary
compatibility between both versions of the library.

PR lib/38741 priority inversion in libpthread breaks apps that use
SCHED_FIFO threads

- Change condvar sync so that we never take the condvar's spinlock without
first holding the caller-provided mutex. Previously, the spinlock was only
taken without the mutex in an error path, but it was enough to trigger the
problem described in the PR.

- Even with this change, applications calling pthread_cond_signal/broadcast
without holding the interlocking mutex are still subject to the problem
described in the PR. POSIX discourages this saying that it leads to
undefined scheduling behaviour, which seems good enough for the time being.

- Elsewhere, use a hash of mutexes instead of per-object spinlocks to
synchronize entry/exit from sleep queues.

- Simplify how sleep queues are maintained.

Add some general comments about the mutex implementation.

branches: 1.48.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.47.2;
pthread__mutex_lock_slow:

- don't stop spinning if there are already waiters on the mutex, there is
no point.
- cosmetic change.

Fix pthread_mutex_trylock() for recursive mutexes. PR lib/38087.
Based on a patch provided by rafal@.

branches: 1.45.2;
Adjust mutex/rwlock definitions to match reality now that there is only
one implementation of each. PR lib/38030.

- Remove libpthread's atomic ops.
- Remove the old spinlock-based mutex and rwlock implementations.
- Use the atomic ops from libc.

Make this compile after Andrew's fix (remove now-unused variables)

pthread_mutex_lock_slow: remove "deliberate deadlock" code that's not
needed with 1:1. PR lib/37524.

add missing static decls.

machine/lock.h, not sys/lock.h

- Fix pthread_rwlock_trywrlock() which was broken.

- Add new functions: pthread_mutex_held_np, mutex_owner_np, rwlock_held_np,
rwlock_wrheld_np, rwlock_rdheld_np. These match the kernel's locking
primitives and can be used when porting kernel code to userspace.

- Always create LWPs detached. Do join/exit sync mostly in userland. When
looped on a dual core box this seems ~30% quicker than using lwp_wait().
Reduce number of lock acquire/release ops during thread exit.

Remove the debuglog stuff. ktrace is more useful now.

For PR bin/37347:

- Override __libc_thr_init() instead of using our own constructor.
- Add pthread__getenv() and use instead of getenv(). This is used before
we are up and running and unfortunatley getenv() takes locks.

Other changes:

- Cache the spinlock vectors in pthread__st. Internal spinlock operations
now take 1 function call instead of 3 (i386).
- Use pthread__self() internally, not pthread_self().
- Use __attribute__ ((visibility("hidden"))) in some places.
- Kill PTHREAD_MAIN_DEBUG.

Add a per-mutex deferred wakeup flag so that threads doing something like
the following do not wake other threads early:

pthread_mutex_lock(&mutex);
pthread_cond_broadcast(&cond);
foo = malloc(100); /* takes libc mutexes */
pthread_mutex_unlock(&mutex);

Fix inverted test after merge of nick-csl-alignment.

Merge nick-csl-alignment.

- Get rid of self->pt_mutexhint and use pthread__mutex_owned() instead.
- Update some comments and fix minor bugs. Minor cosmetic changes.
- Replace some spinlocks with mutexes and rwlocks.
- Change the process private semaphores to use mutexes and condition
variables instead of doing the synchronization directly. Spinlocks
are no longer used by the semaphore code.

- Don't take the mutex's spinlock (ptr_interlock) in pthread_cond_wait().
Instead, make the deferred wakeup list a per-thread array and pass down
the lwpid_t's that way.

- In pthread_cond_wait(), take the mutex before dealing with early wakeup.
In this way there should never be contention on the CV's spinlock if
the app follows POSIX rules (there should only be contention on the
user-provided mutex).

- Add a port of the kernel's rwlocks. The rwlock's spinlock is only taken if
there is contention. This is enabled where atomic ops are available. Right
now that is only i386 and amd64 because I don't have other hardware to
test with. It's trivial to add stubs for other architectures as long as
they have compare-and-swap. When we have proper atomic ops the old rwlock
code can be removed.

- Add a new mutex implementation that's similar to the kernel's mutexes, but
uses compare-and-swap to maintain the waiters list, so no spinlocks are
involved. Same caveats apply as for the rwlocks.

branches: 1.31.2;
One more micro-optimization..

Trim fat off libpthread internal spinlock operations. Makes a mesurable
improvement across the board.

branches: 1.29.2;
Some significant performance improvements, and a fix for a race with pthread
detach/join.

- Make mutex acquire spin for a short time, as done with spinlocks.
- Make the number of spins controllable with the env var PTHREAD_NSPINS.
- Reduce the amount of time that libpthread internal spinlocks are held.
- Rely more on the barrier effects of park/unpark to avoid taking spinlocks.
- Simplify the locking around pthreads and the global queues.
- Align per-thread sync data on a 128 byte boundary.
- Offset thread stacks by a small amount to try and reduce cache thrash.

branches: 1.28.2;
- Test+branch is usually cheaper than making an indirect function call,
so avoid making them.
- When parking an LWP on a condition variable, point the hint argument at
the mutex's waiters queue. Chances are we will be awoken from that later.

- Maintain a per-thread pointer to the last mutex acquired by the app, to
be used only as as a hint. Clear the pointer when releasing the mutex.
- When releasing a mutex, wake all waiters. Makes it possible to tranfer
waiters from another object to a mutex.

Update to match changed pthread__park() interface.

Remove the PTHREAD_SA option. If M:N threads is reimplemented it's
better off done with a seperate library.

Fix bugs with and improve upon previous.

Conditionalised support for 1:1 threads. Needs associated kernel changes
and more work to be useful.

branches: 1.22.2; 1.22.4;
Close a window in which we can not notice a recently-slept-on-our
mutex thread, thus leaving a thread sleeping on an unlocked mutex.

Reviewed by myself and Christos.

Problem reported by Arne H. Juul, arnej at pvv dot ntnu dot no,
in PR 26208. This fix represents option 1 presented in the PR.

branches: 1.21.2;
starting the pthread library (ie. calling pthread__start()) before
any threads are created turned out to be not such a good idea.
there are stronger requirements on what has to work in a forked child
while a process is still single-threaded. so take all that stuff
back out and fix the problems with single-threaded programs that
are linked with libpthread differently, by checking if the library
has been started and doing completely different stuff if it hasn't been:
- for pthread_rwlock_timedrdlock(), just fail with EDEADLK immediately.
- for sem_wait(), the only thing that can unlock the semaphore is a
signal handler, so use sigsuspend() to wait for a signal.
- for pthread_mutex_lock_slow(), just go into an infinite loop
waiting for signals.

I also noticed that there's a "sem2" test that has never worked in its
single-threaded form. the problem there is that a signal handler tries
to take a sem_t interlock which is already held when the signal is received.
fix this too, by adding a single-threaded case for sig_trywait() that
blocks signals instead of using the userland interlock.

in pthread_mutex_lock_slow(), pthread_rwlock_timedrdlock() and sem_wait(),
call pthread__start() if it hasn't already been called. this avoids
an internal assertion from the library if these routines are used
before any threads are created and they need to sleep.
fixes PR 20256, PR 24241, PR 25722, PR 26096.

Add cancellation protection to pthread_once() as per the standard; if
the once routine is cancelled, the effect on once_control is as if
pthread_once() was never called.

Bug report and fix from PR lib/30734.

branches: 1.18.6;
add libpthread part of concurrency support for SA on MP systems
- enable concurrency according to environment variable PTHREAD_CONCURRENCY
- add idle VP wakeup if there are additional jobs and idle VPs
- make reidlequeue per VP
- enable spinning for locks
- fix race condition in alarm processing
- fix race condition in mutex locking
- make debugging output line buffered and add VP prefix to debug lines

Update mutex/rwlock/sem code to match recent change in cond code.

add missing notreached lint comment.

Separately report unlocking an unlocked mutex and unlocking a mutex
owned by another thread.

Use pthread__error() instead of pthread__abort().

More agressive assertions. Change a number of detected conditions, which
officially have undefined behavior, from returning an error code to raising
an assertion failure.
Also, don't bother to explicitly test for (illegal) null pointers and return
an error; they'll bomb out soon enough.

pthread_mutex_trylock(): It's not an error to call trylock() on a
mutex already locked by the calling thread, even for non-recursive
mutexes.

When a thread sleeps on a mutex or cv, have it put itself on the
front of the sleep queue rather than the back. This is more
cache-friendly behavior and within the (lack of) constraints on wakeup
ordering imposed on equal-priority threads.

add __RCSID()

Remove unnecessary inclusion of <assert.h>.

Mutex tuneup.

* Use a double-checked locking technique to avoid taking
the interlock in pthread_mutex_unlock().

* In pthread_mutex_lock() and pthread_mutex_trylock(), only store the
stack pointer, not the thread ID, in ptm_owner. Do the translation
to a thread ID in the slow-lock, errorcheck, and recursive mutex
cases rather than in the common path.

* Juggle where pthread__self() is called, to move it out of the fast path.

Overall, this means that neither pthread_self() nor
pthread_spin[un]lock() are used in the course of locking and unlocking
an uncontested mutex. Speeds up the fast path by 40-50%, and
eliminates about 98% of spinlocks used by a couple of large threaded
applications.

(Still a GET_MUTEX_PRIVATE() in the fast path... perhaps the type
should be in the main body of the mutex).

Add debug counters for mutex and condvar operations.

Need to include <string.h> for memcmp() prototype.

Fix typo.

Add mutexattr_init(), mutexattr_destroy(), and mutexattr_settype() (recursive
mutexes are used by the X libraries).

de-lint

Merge the nathanw_sa branch.

branches: 1.1.2;
file pthread_mutex.c was initially added on branch nathanw_sa.

Deal with a couple of problems with threads being awoken early due to
timeouts or cancellation where:

- The restarting thread calls _lwp_exit() before another thread gets around
to waking it with _lwp_unpark(), leading to ESRCH (observed by joerg@).
(I may have removed a similar check mistakenly over the weekend.)

- The restarting thread considers itself gone off the sleep queue but
at the same time another thread is part way through waking it, and hasn't
fully completed that operation yet by setting thread->pt_mutexwait = 0.
I think that could have potentially lead to the list of waiters getting
messed up given the right circumstances.

In the interests of reliability simplify waiter handling more and redo
condvars to manage the list of waiters with atomic ops.

- Try to eliminate a hang in "parked" I've been seeing while stress testing.
Centralise wakeup of deferred waiters in pthread__clear_waiters() and use
throughout libpthread. Make fewer assumptions. Be more conservative in
pthread_mutex when dealing with pending waiters.

- Remove the "hint" argument everywhere since the kernel doesn't use it any
more.

Revert "Enhance the pthread(3) + malloc(3) init model"

It is reported to hand on aarch64 with gzip.

Enhance the pthread(3) + malloc(3) init model

Separate the pthread_atfork(3) call from pthread_tsd_init()
and move it into a distinct function.

Call inside pthread__init() late TSD initialization route, just after
"pthread_atfork(NULL, NULL, pthread__fork_callback);".

Document that malloc(3) initialization is now controlled again and called
during the first pthread_atfork(3) call.

Remove #if 0 code from pthread_mutex.c as we no longer initialize malloc
prematurely.

Revert previous

'git grep' breaks now.

Remove 'ifdef 0' hacks

It is no longer needed as the proper fix avoiding premature malloc()
landed the sources.

Refactor libpthread checks for invalid arguments

Switch from manual functions to pthread__error().

In the same spirit as the previous pthread_mutex_init change for jemalloc,
make pthread_mutexattr_init do always a full initialization, so that the
attribute that will be used later when we become threaded is properly
initialized.

Use pthread_mutexattr_t and pthread_mutex_t magic fields

Validate _PT_MUTEX_MAGIC in pthread_mutex_t and _PT_MUTEXATTR_MAGIC
in pthread_mutexattr_t accordingly.

Mark destroyed pthread_mutexattr_t as dead

Adjustment to previous: don't call _lwp_unpark_all() with nwaiters == 0.

pthread__mutex_unlock_slow(): ignore the DEFERRED bit. It's only purpose
is to get the thread to go through the slow path. If there are waiters,
process them there and then. Should not affect well behaved apps. Maybe
of help for:

PR bin/50350: rump/rumpkern/t_sp/stress_{long,short} fail on Core 2 Quad

Rip out some very ambitious optimisations around pthread_mutex that are
don't buy much. This stuff is hard enough to get right in the kernel let
alone userspace, and I don't trust that it's right.

branches: 1.65.2;
Jemalloc initializes mutexes before we become threaded and expects to use
them later.

branches: 1.64.4;

Deal with more lwp_park() timestamp unconsting

Don't spin if we already own the mutex, otherwise we will get stuck spinning
forever, fixes timemutex{1,2} tests.

Use anonymous union for ptm_ceiling and old __pthread_spin_t field to
maintain backward compatibility and fix hppa build. hppa has an non-
integer type __pthread_spin_t

KNF

branches: 1.60.2;
GSoC 2016 Charles Cui: Implement thread priority protection based on work
by Andy Doran. Also document the get/set pshared thread calls as not
implemented, and add a skeleton implementation that is disabled.
XXX: document _sched_protect(2).

pthread__mutex_lock_slow: fix the handling of a potential race with the
non-interlocked CAS in the fast unlock path -- it is unsafe to test for
the waiters-bit while the owner thread is running, we have to spin for
the owner or its state change to be sure about the presence of the bit.
Split off the logic into the pthread__mutex_setwaiters() routine.

This is a partial fix to the named lockup problem (also see PR/44756).
It seems there is another race which can be reproduced on faster CPUs.

remove compatibility code for handling CLOCK_MONOTONIC and handle it in the
syscall directly.

PR/44756: Sad Clouds: Prevent leakage of errno = ESRCH from _lwp_park. This
has two parts:
- in pthread_cond_timedwait() if the thread we are trying to unpark
exited, retry the the _lwp_park call without it.
- pthread_mutex() was affecting errno since it is calling _lwp_park()
from pthread_mutex_lock_slow(). preserve the original errno.
Note that the example problem still causes an occassional deadlock on machines
with many CPUs and it is the same deadlock we observe with named.

- Allow libpthread to be dlopened again, by providing libc stubs to libpthread.
- Fail if the dlopened libpthread does pthread_create(). From manu@
- Discussed at length in the mailing lists; approved by core@
- This was chosen as the least intrusive patch that will provide
the necessary functionality.
XXX: pullup to 6

add comments
whitespace

branches: 1.54.2;
Add a pthread__smt_wake and add support for it on arm along with
pthread__smt_pause. These are implemented using the ARM instructions
SEV (wake) and WFE (pause). These are treated as NOPs on ARM CPUs that
don't support them.

Move the pthread_once alias where it belongs.

Move pthread_once implementation into a separate file, it doesn't depend
on the mutex implementation in any way.

branches: 1.51.2; 1.51.4; 1.51.6; 1.51.22;
Change pthread_mutex_t to use the amount of space as and be congruent to
the version used in the SA version of pthreads. This preserves binary
compatibility between both versions of the library.

PR lib/38741 priority inversion in libpthread breaks apps that use
SCHED_FIFO threads

- Change condvar sync so that we never take the condvar's spinlock without
first holding the caller-provided mutex. Previously, the spinlock was only
taken without the mutex in an error path, but it was enough to trigger the
problem described in the PR.

- Even with this change, applications calling pthread_cond_signal/broadcast
without holding the interlocking mutex are still subject to the problem
described in the PR. POSIX discourages this saying that it leads to
undefined scheduling behaviour, which seems good enough for the time being.

- Elsewhere, use a hash of mutexes instead of per-object spinlocks to
synchronize entry/exit from sleep queues.

- Simplify how sleep queues are maintained.

Add some general comments about the mutex implementation.

branches: 1.48.2;
Remove clause 3 and 4 from TNF licenses

branches: 1.47.2;
pthread__mutex_lock_slow:

- don't stop spinning if there are already waiters on the mutex, there is
no point.
- cosmetic change.

Fix pthread_mutex_trylock() for recursive mutexes. PR lib/38087.
Based on a patch provided by rafal@.

branches: 1.45.2;
Adjust mutex/rwlock definitions to match reality now that there is only
one implementation of each. PR lib/38030.

- Remove libpthread's atomic ops.
- Remove the old spinlock-based mutex and rwlock implementations.
- Use the atomic ops from libc.

Make this compile after Andrew's fix (remove now-unused variables)

pthread_mutex_lock_slow: remove "deliberate deadlock" code that's not
needed with 1:1. PR lib/37524.

add missing static decls.

machine/lock.h, not sys/lock.h

- Fix pthread_rwlock_trywrlock() which was broken.

- Add new functions: pthread_mutex_held_np, mutex_owner_np, rwlock_held_np,
rwlock_wrheld_np, rwlock_rdheld_np. These match the kernel's locking
primitives and can be used when porting kernel code to userspace.

- Always create LWPs detached. Do join/exit sync mostly in userland. When
looped on a dual core box this seems ~30% quicker than using lwp_wait().
Reduce number of lock acquire/release ops during thread exit.

Remove the debuglog stuff. ktrace is more useful now.

For PR bin/37347:

- Override __libc_thr_init() instead of using our own constructor.
- Add pthread__getenv() and use instead of getenv(). This is used before
we are up and running and unfortunatley getenv() takes locks.

Other changes:

- Cache the spinlock vectors in pthread__st. Internal spinlock operations
now take 1 function call instead of 3 (i386).
- Use pthread__self() internally, not pthread_self().
- Use __attribute__ ((visibility("hidden"))) in some places.
- Kill PTHREAD_MAIN_DEBUG.

Add a per-mutex deferred wakeup flag so that threads doing something like
the following do not wake other threads early:

pthread_mutex_lock(&mutex);
pthread_cond_broadcast(&cond);
foo = malloc(100); /* takes libc mutexes */
pthread_mutex_unlock(&mutex);

Fix inverted test after merge of nick-csl-alignment.

Merge nick-csl-alignment.

- Get rid of self->pt_mutexhint and use pthread__mutex_owned() instead.
- Update some comments and fix minor bugs. Minor cosmetic changes.
- Replace some spinlocks with mutexes and rwlocks.
- Change the process private semaphores to use mutexes and condition
variables instead of doing the synchronization directly. Spinlocks
are no longer used by the semaphore code.

- Don't take the mutex's spinlock (ptr_interlock) in pthread_cond_wait().
Instead, make the deferred wakeup list a per-thread array and pass down
the lwpid_t's that way.

- In pthread_cond_wait(), take the mutex before dealing with early wakeup.
In this way there should never be contention on the CV's spinlock if
the app follows POSIX rules (there should only be contention on the
user-provided mutex).

- Add a port of the kernel's rwlocks. The rwlock's spinlock is only taken if
there is contention. This is enabled where atomic ops are available. Right
now that is only i386 and amd64 because I don't have other hardware to
test with. It's trivial to add stubs for other architectures as long as
they have compare-and-swap. When we have proper atomic ops the old rwlock
code can be removed.

- Add a new mutex implementation that's similar to the kernel's mutexes, but
uses compare-and-swap to maintain the waiters list, so no spinlocks are
involved. Same caveats apply as for the rwlocks.

branches: 1.31.2;
One more micro-optimization..

Trim fat off libpthread internal spinlock operations. Makes a mesurable
improvement across the board.

branches: 1.29.2;
Some significant performance improvements, and a fix for a race with pthread
detach/join.

- Make mutex acquire spin for a short time, as done with spinlocks.
- Make the number of spins controllable with the env var PTHREAD_NSPINS.
- Reduce the amount of time that libpthread internal spinlocks are held.
- Rely more on the barrier effects of park/unpark to avoid taking spinlocks.
- Simplify the locking around pthreads and the global queues.
- Align per-thread sync data on a 128 byte boundary.
- Offset thread stacks by a small amount to try and reduce cache thrash.

branches: 1.28.2;
- Test+branch is usually cheaper than making an indirect function call,
so avoid making them.
- When parking an LWP on a condition variable, point the hint argument at
the mutex's waiters queue. Chances are we will be awoken from that later.

- Maintain a per-thread pointer to the last mutex acquired by the app, to
be used only as as a hint. Clear the pointer when releasing the mutex.
- When releasing a mutex, wake all waiters. Makes it possible to tranfer
waiters from another object to a mutex.

Update to match changed pthread__park() interface.

Remove the PTHREAD_SA option. If M:N threads is reimplemented it's
better off done with a seperate library.

Fix bugs with and improve upon previous.

Conditionalised support for 1:1 threads. Needs associated kernel changes
and more work to be useful.

branches: 1.22.2; 1.22.4;
Close a window in which we can not notice a recently-slept-on-our
mutex thread, thus leaving a thread sleeping on an unlocked mutex.

Reviewed by myself and Christos.

Problem reported by Arne H. Juul, arnej at pvv dot ntnu dot no,
in PR 26208. This fix represents option 1 presented in the PR.

branches: 1.21.2;
starting the pthread library (ie. calling pthread__start()) before
any threads are created turned out to be not such a good idea.
there are stronger requirements on what has to work in a forked child
while a process is still single-threaded. so take all that stuff
back out and fix the problems with single-threaded programs that
are linked with libpthread differently, by checking if the library
has been started and doing completely different stuff if it hasn't been:
- for pthread_rwlock_timedrdlock(), just fail with EDEADLK immediately.
- for sem_wait(), the only thing that can unlock the semaphore is a
signal handler, so use sigsuspend() to wait for a signal.
- for pthread_mutex_lock_slow(), just go into an infinite loop
waiting for signals.

I also noticed that there's a "sem2" test that has never worked in its
single-threaded form. the problem there is that a signal handler tries
to take a sem_t interlock which is already held when the signal is received.
fix this too, by adding a single-threaded case for sig_trywait() that
blocks signals instead of using the userland interlock.

in pthread_mutex_lock_slow(), pthread_rwlock_timedrdlock() and sem_wait(),
call pthread__start() if it hasn't already been called. this avoids
an internal assertion from the library if these routines are used
before any threads are created and they need to sleep.
fixes PR 20256, PR 24241, PR 25722, PR 26096.

Add cancellation protection to pthread_once() as per the standard; if
the once routine is cancelled, the effect on once_control is as if
pthread_once() was never called.

Bug report and fix from PR lib/30734.

branches: 1.18.6;
add libpthread part of concurrency support for SA on MP systems
- enable concurrency according to environment variable PTHREAD_CONCURRENCY
- add idle VP wakeup if there are additional jobs and idle VPs
- make reidlequeue per VP
- enable spinning for locks
- fix race condition in alarm processing
- fix race condition in mutex locking
- make debugging output line buffered and add VP prefix to debug lines

Update mutex/rwlock/sem code to match recent change in cond code.

add missing notreached lint comment.

Separately report unlocking an unlocked mutex and unlocking a mutex
owned by another thread.

Use pthread__error() instead of pthread__abort().

More agressive assertions. Change a number of detected conditions, which
officially have undefined behavior, from returning an error code to raising
an assertion failure.
Also, don't bother to explicitly test for (illegal) null pointers and return
an error; they'll bomb out soon enough.

pthread_mutex_trylock(): It's not an error to call trylock() on a
mutex already locked by the calling thread, even for non-recursive
mutexes.

When a thread sleeps on a mutex or cv, have it put itself on the
front of the sleep queue rather than the back. This is more
cache-friendly behavior and within the (lack of) constraints on wakeup
ordering imposed on equal-priority threads.

add __RCSID()

Remove unnecessary inclusion of <assert.h>.

Mutex tuneup.

* Use a double-checked locking technique to avoid taking
the interlock in pthread_mutex_unlock().

* In pthread_mutex_lock() and pthread_mutex_trylock(), only store the
stack pointer, not the thread ID, in ptm_owner. Do the translation
to a thread ID in the slow-lock, errorcheck, and recursive mutex
cases rather than in the common path.

* Juggle where pthread__self() is called, to move it out of the fast path.

Overall, this means that neither pthread_self() nor
pthread_spin[un]lock() are used in the course of locking and unlocking
an uncontested mutex. Speeds up the fast path by 40-50%, and
eliminates about 98% of spinlocks used by a couple of large threaded
applications.

(Still a GET_MUTEX_PRIVATE() in the fast path... perhaps the type
should be in the main body of the mutex).

Add debug counters for mutex and condvar operations.

Need to include <string.h> for memcmp() prototype.

Fix typo.

Add mutexattr_init(), mutexattr_destroy(), and mutexattr_settype() (recursive
mutexes are used by the X libraries).

de-lint

Merge the nathanw_sa branch.