- Copy stable/10 (r259064) to releng/10.0 as part of the
10.0-RELEASE cycle.
- Update __FreeBSD_version [1]
- Set branch name to -RC1

[1] 10.0-CURRENT __FreeBSD_version value ended at '55', so
start releng/10.0 at '100' so the branch is started with
a value ending in zero.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r258499:
Fix for the spurious signal handler call with zero signo in the threaded
process.

Approved by: re (hrs)

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

libthr: Remove _thr_rtld_fini(), unused since r245630.

In suspend_common(), don't wait for a thread which is in creation, because
pthread_suspend_all_np() may have already suspended its parent thread.
Add locking code in pthread_suspend_all_np() to only allow one thread
to suspend other threads, this eliminates a deadlock where two or more
threads try to suspend each others.

MFp4:
Further decreases unexpected context switches by defering mutex wakeup
until internal sleep queue lock is released.

MFp4:
Enqueue thread in LIFO, this can cause starvation, but it gives better
performance. Use _thr_queuefifo to control the frequency of FIFO vs LIFO,
you can use environment string LIBPTHREAD_QUEUE_FIFO to configure the
variable.

Set SIGCANCEL to SIGTHR as part of some cleanup of DTrace code.

Reviewed by: davidxu@
MFC after: 1 week

umtx operation UMTX_OP_MUTEX_WAKE has a side-effect that it accesses
a mutex after a thread has unlocked it, it event writes data to the mutex
memory to clear contention bit, there is a race that other threads
can lock it and unlock it, then destroy it, so it should not write
data to the mutex memory if there isn't any waiter.
The new operation UMTX_OP_MUTEX_WAKE2 try to fix the problem. It
requires thread library to clear the lock word entirely, then
call the WAKE2 operation to check if there is any waiter in kernel,
and try to wake up a thread, if necessary, the contention bit is set again
by the operation. This also mitgates the chance that other threads find
the contention bit and try to enter kernel to compete with each other
to wake up sleeping thread, this is unnecessary. With this change, the
mutex owner is no longer holding the mutex until it reaches a point
where kernel umtx queue is locked, it releases the mutex as soon as
possible.
Performance is improved when the mutex is contensted heavily. On Intel
i3-2310M, the runtime of a benchmark program is reduced from 26.87 seconds
to 2.39 seconds, it even is better than UMTX_OP_MUTEX_WAKE which is
deprecated now. http://people.freebsd.org/~davidxu/bench/mutex_perf.c

When destroying a barrier, waiting all threads exit the barrier,
this makes it possible a thread received PTHREAD_BARRIER_SERIAL_THREAD
immediately free memory area of the barrier.

Plug a memory leak. When a cached thread is reused, don't clear sleep
queue pointers, just reuse it.

PR: 164828
MFC after: 1 week

Implement the __pthread_map_stacks_exec() for libthr.

Stack creation code is changed to call _rtld_get_stack_prot() to get
the stack protection right. There is a race where thread is created
during dlopen() of dso that requires executable stacks. Then,
_rtld_get_stack_prot() may return PROT_READ | PROT_WRITE, but thread
is still not linked into the thread list. In this case, the callback
misses the thread stack, and rechecks the required protection
afterward.

Reviewed by: davidxu

MFp4:

- Add flags CVWAIT_ABSTIME and CVWAIT_CLOCKID for umtx kernel based
condition variable, this should eliminate an extra system call to get
current time.

- Add sub-function UMTX_OP_NWAKE_PRIVATE to wake up N channels in single
system call. Create userland sleep queue for condition variable, in most
cases, thread will wait in the queue, the pthread_cond_signal will defer
thread wakeup until the mutex is unlocked, it tries to avoid an extra
system call and a extra context switch in time window of pthread_cond_signal
and pthread_mutex_unlock.

The changes are part of process-shared mutex project.

Get cpuset in pthread_attr_get_np() and free it in pthread_attr_destroy().

MFC after: 7 days

Revert revision 214007, I realized that MySQL wants to resolve
a silly rwlock deadlock problem, the deadlock is caused by writer
waiters, if a thread has already locked a reader lock, and wants to
acquire another reader lock, it will be blocked by writer waiters,
but we had already fixed it years ago.

Add pthread_rwlockattr_setkind_np and pthread_rwlockattr_getkind_np, the
functions set or get pthread_rwlock type, current supported types are:
PTHREAD_RWLOCK_PREFER_READER_NP,
PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP,
PTHREAD_RWLOCK_PREFER_WRITER_NP,
default is PTHREAD_RWLOCK_PREFER_WRITER_NONCECURSIVE_NP, this maintains
binary compatible with old code.

change code to use unwind.h.

In current code, statically initialized and destroyed object have
same null value, the code can not distinguish between them, to
fix the problem, now a destroyed object is assigned to a non-null
value, and it will be rejected by some pthread functions.
PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that
adaptive mutex can be statically initialized correctly.

Add missing field.

To support stack unwinding for cancellation points, add -fexceptions flag
for them, two functions _pthread_cancel_enter and _pthread_cancel_leave
are added to let thread enter and leave a cancellation point, it also
makes it possible that other functions can be cancellation points in
libraries without having to be rewritten in libthr.

In most cases, cancel_point and cancel_async needn't be checked again,
because cancellation is almostly checked at cancellation points.

Because atfork lock is held while forking, a thread cancellation triggered
by atfork handler is unsafe, use intenal flag no_cancel to disable it.

add code to support stack unwinding when thread exits. note that only
defer-mode cancellation works, asynchrnous mode does not work because
it lacks of libuwind's support. stack unwinding is not enabled unless
LIBTHR_UNWIND_STACK is defined in Makefile.

Move back IN_GCLIST flag into field tlflags, since thread list and gc list
still share same lock.

Fix copy&paste problem.

Convert thread list lock from mutex to rwlock.

Change atfork lock from mutex to rwlock, also make mutexes used by malloc()
module private type, when private type mutex is locked/unlocked, thread
critical region is entered or leaved. These changes makes fork()
async-signal safe which required by POSIX. Note that user's atfork handler
still needs to be async-signal safe, but it is not problem of libthr, it
is user's responsiblity.

Add signal handler wrapper, the reason to add it becauses there are
some cases we want to improve:
1) if a thread signal got a signal while in cancellation point,
it is possible the TDP_WAKEUP may be eaten by signal handler
if the handler called some interruptibly system calls.
2) In signal handler, we want to disable cancellation.
3) When thread holding some low level locks, it is better to
disable signal, those code need not to worry reentrancy,
sigprocmask system call is avoided because it is a bit expensive.
The signal handler wrapper works in this way:
1) libthr installs its signal handler if user code invokes sigaction
to install its handler, the user handler is recorded in internal
array.
2) when a signal is delivered, libthr's signal handler is invoke,
libthr checks if thread holds some low level lock or is in critical
region, if it is true, the signal is buffered, and all signals are
masked, once the thread leaves critical region, correct signal
mask is restored and buffered signal is processed.
3) before user signal handler is invoked, cancellation is temporarily
disabled, after user signal handler is returned, cancellation state
is restored, and pending cancellation is rescheduled.

Unregister thread specific data destructor when a corresponding dso
is unloaded.

eliminate unused code.

The __hidden definition is provided by sys/cdefs.h.

MFC after: 2 weeks

Add wrapper for setcontext() and swapcontext(), the wrappers
unblock SIGCANCEL which is needed by thread cancellation.

On shared object unload, in __cxa_finalize, call and clear all installed
atexit and __cxa_atexit handlers that are either installed by unloaded
dso, or points to the functions provided by the dso.

Use _rtld_addr_phdr to locate segment information from the address of
private variable belonging to the dso, supplied by crtstuff.c. Provide
utility function __elf_phdr_match_addr to do the match of address against
dso executable segment.

Call back into libthr from __cxa_finalize using weak
__pthread_cxa_finalize symbol to remove any atfork handler which
function points into unloaded object.

The rtld needs private __pthread_cxa_finalize symbol to not require
resolution of the weak undefined symbol at initialization time. This
cannot work, since rtld is relocated before sym_zero is set up.

Idea by: kan
Reviewed by: kan (previous version)
MFC after: 3 weeks

In current implementation, thread cancellation is done in signal handler,
which does not know what is the state of interrupted system call, for
example, open() system call opened a file and the thread is still cancelled,
result is descriptor leak, there are other problems which can cause resource
leak or undeterminable side effect when a thread is cancelled. However, this
is no longer true in new implementation.

In defering mode, a thread is canceled if cancellation request is pending and
later the thread enters a cancellation point, otherwise, a later
pthread_cancel() just causes SIGCANCEL to be sent to the target thread, and
causes target thread to abort system call, userland code in libthr then checks
cancellation state, and cancels the thread if needed. For example, the
cancellation point open(), the thread may be canceled at start,
but later, if it opened a file descriptor, it is not canceled, this avoids
file handle leak. Another example is read(), a thread may be canceled at start
of the function, but later, if it read some bytes from a socket, the thread
is not canceled, the caller then can decide if it should still enable cancelling
or disable it and continue reading data until it thinks it has read all
bytes of a packet, and keeps a protocol stream in health state, if user ignores
partly reading of a packet without disabling cancellation, then second iteration
of read loop cause the thread to be cancelled.
An exception is that the close() cancellation point always closes a file handle
despite whether the thread is cancelled or not.

The old mechanism is still kept, for a functions which is not so easily to
fix a cancellation problem, the rough mechanism is used.

Reviewed by: kib@

Use umtx to implement process sharable semaphore, to make this work,
now type sema_t is a structure which can be put in a shared memory area,
and multiple processes can operate it concurrently.
User can either use mmap(MAP_SHARED) + sem_init(pshared=1) or use sem_open()
to initialize a shared semaphore.
Named semaphore uses file system and is located in /tmp directory, and its
file name is prefixed with 'SEMD', so now it is chroot or jail friendly.
In simplist cases, both for named and un-named semaphore, userland code
does not have to enter kernel to reduce/increase semaphore's count.
The semaphore is designed to be crash-safe, it means even if an application
is crashed in the middle of operating semaphore, the semaphore state is
still safely recovered by later use, there is no waiter counter maintained
by userland code.
The main semaphore code is in libc and libthr only has some necessary stubs,
this makes it possible that a non-threaded application can use semaphore
without linking to thread library.
Old semaphore implementation is kept libc to maintain binary compatibility.
The kernel ksem API is no longer used in the new implemenation.

Discussed on: threads@

Make openat(2) a cancellation point.

This is required by POSIX and matches open(2).

Reviewed by: kib, jhb
MFC after: 1 month

Make pthread_cleanup_push() and pthread_cleanup_pop() as a pair of macros,
use stack space to keep cleanup information, this eliminates overhead of
calling malloc() and free() in thread library.

Discussed on: thread@

- Reduce function call overhead for uncontended case.
- Remove unused flags MUTEX_FLAGS_* and their code.
- Check validity of the timeout parameter in mutex_self_lock().

Use UMTX_OP_WAIT_UINT_PRIVATE and UMTX_OP_WAKE_PRIVATE to save
time in kernel(avoid VM lookup).

put THR_CRITICAL_LEAVE into do .. while statement.

add __hidden suffix to _umtx_op_err, this eliminates PLT.

Add pthread_setaffinity_np and pthread_getaffinity_np to libc namespace.

Remove unused functions.

Replace function _umtx_op with _umtx_op_err, the later function directly
returns errno, because errno can be mucked by user's signal handler and
most of pthread api heavily depends on errno to be correct, this change
should improve stability of the thread library.

Replace userland rwlock with a pure kernel based rwlock, the new
implementation does not switch pointers when it resumes waiters.

Asked by: jeff

Rewrite rwlock to user atomic operations to change rwlock state, this
eliminates internal mutex lock contention when most rwlock operations
are read.

Orignal patch provided by: jeff

Use cpuset defined in pthread_attr for newly created thread, for now,
we set scheduling parameters and cpu binding fully in userland, and
because default scheduling policy is SCHED_RR (time-sharing), we set
default sched_inherit to PTHREAD_SCHED_INHERIT, this saves a system
call.

If a new thread is created, it inherits current thread's signal masks,
however if current thread is executing cancellation handler, signal
SIGCANCEL may have already been blocked, this is unexpected, unblock the
signal in new thread if this happens.

MFC after: 1 week

Include cpuset.h, unbreak compiling.

implement pthread_attr_getaffinity_np and pthread_attr_setaffinity_np.

1. Add function pthread_mutex_setspinloops_np to turn a mutex's spin
loop count.
2. Add function pthread_mutex_setyieldloops_np to turn a mutex's yield
loop count.
3. Make environment variables PTHREAD_SPINLOOPS and PTHREAD_YIELDLOOPS
to be only used for turnning PTHREAD_MUTEX_ADAPTIVE_NP mutex.

Remove umtx_t definition, use type long directly, add wrapper function
_thr_umtx_wait_uint() for umtx operation UMTX_OP_WAIT_UINT, use the
function in semaphore operations, this fixed compiler warnings.

Add my recent work of adaptive spin mutex code. Use two environments variable
to tune pthread mutex performance:
1. LIBPTHREAD_SPINLOOPS
If a pthread mutex is being locked by another thread, this environment
variable sets total number of spin loops before the current thread
sleeps in kernel, this saves a syscall overhead if the mutex will be
unlocked very soon (well written application code).
2. LIBPTHREAD_YIELDLOOPS
If a pthread mutex is being locked by other threads, this environment
variable sets total number of sched_yield() loops before the currrent
thread sleeps in kernel. if a pthread mutex is locked, the current thread
gives up cpu, but will not sleep in kernel, this means, current thread
does not set contention bit in mutex, but let lock owner to run again
if the owner is on kernel's run queue, and when lock owner unlocks the
mutex, it does not need to enter kernel and do lots of work to resume
mutex waiters, in some cases, this saves lots of syscall overheads for
mutex owner.

In my practice, sometimes LIBPTHREAD_YIELDLOOPS can massively improve performance
than LIBPTHREAD_SPINLOOPS, this depends on application. These two environments
are global to all pthread mutex, there is no interface to set them for each
pthread mutex, the default values are zero, this means spinning is turned off
by default.

Output error message to STDERR_FILENO.

Approved by: re (bmah)

Check environment variable PTHREAD_ADAPTIVE_SPIN, if it is set, use
it as a default spin cycle count.

- Remove variable _thr_scope_system, all threads are system scope.
- Rename _thr_smp_cpus to boolean variable _thr_is_smp.
- Define CPU_SPINWAIT macro for each arch, only X86 supports it.

Use ucond to implement barrier.

Use kernel provided userspace condition variable to implement pthread
condition variable.

Eliminate atomic operations in thread cancellation functions, it should
reduce overheads of cancellation points.

Use type pthread_state for thread state.

use rtprio_thread system call to get or set thread priority.

Replace internal usage of struct umtx with umutex which can supports
real-time if we want, no functionality is changed.

Use umutex APIs to implement pthread_mutex, member pp_mutexq is added
into pthread structure to keep track of locked PTHREAD_PRIO_PROTECT mutex,
no real mutex code is changed, the mutex locking and unlocking code should
has same performance as before.

Axe unused member field.

Get number of CPUs and ignore spin count on single processor machine.

Use kernel facilities to support real-time scheduling.

Remove unused member.

Remove unused member field m_queue.

s/long/int.

- Use same priority range returned by kernel's sched_get_priority_min()
and sched_get_priority_max() syscalls.
- Remove unused fields from structure pthread_attr.

Do not check validity of timeout if a mutex can be acquired immediately.
Completly drop recursive mutex in pthread_cond_wait and restore recursive
after resumption. Reorganize code to make gcc to generate better code.

WARNS level 4 cleanup.

Remove priority mutex code because it does not work correctly,
to make it work, turnstile like mechanism to support priority
propagating and other realtime scheduling options in kernel
should be available to userland mutex, for the moment, I just
want to make libthr be simple and efficient thread library.

Discussed with: deischen, julian

Add locking support for rtld.

Rework last change of pthread_once, create a function _thr_once_init to
reinitialize its internal locks.

After fork(), reinitialize internal locks for pthread_once().

Now, thread name is stored in kernel, userland no longer has to keep it.

Use macro STATIC_LIB_REQUIRE to declare a symbol should be linked into
static binary.

Tweak macro THR_LOCK_RELEASE a bit for non-PTHREAD_INVARIANTS case.

Refine thread suspension code, now thread suspension is a blockable
operation, the caller is blocked util target threads are really
suspended, also avoid suspending a thread when it is holding a
critical lock.
Fix a bug in _thr_ref_delete which tests a never set flag.

Hide umtx API symbols as well.

1. Retire macro SCLASS, instead simply use language keyword and
put variables in thr_init.c.
2. Hide all global symbols which won't be exported.

Update copyright.

Add code to handle timer_delete(). The timer wrapper code is completely
rewritten, now timers created with same sigev_notify_attributes will
run in same thread, this allows user to organize which timers can
run in same thread to save some thread resource.

Fix off-by-one nanosecond bug in macro TIMESPEC_ADD.

Reviewed by: deischen
Approved by: re (dwhite)
MFC after : 4 days

Add debugger event reporting support, current only TD_CREATE and TD_DEATH
events are reported.

Remove unique id field which is no longer used by debugger.

Import my recent 1:1 threading working. some features improved includes:
1. fast simple type mutex.
2. __thread tls works.
3. asynchronous cancellation works ( using signal ).
4. thread synchronization is fully based on umtx, mainly, condition
variable and other synchronization objects were rewritten by using
umtx directly. those objects can be shared between processes via
shared memory, it has to change ABI which does not happen yet.
5. default stack size is increased to 1M on 32 bits platform, 2M for
64 bits platform.
As the result, some mysql super-smack benchmarks show performance is
improved massivly.

Okayed by: jeff, mtm, rwatson, scottl

Increase the default stacksizes:

32-bit 64-bit
main thread 2 MB 4 MB
other threads 1 MB 2 MB

Approved by: mtm
Adapted from: libpthread

Don't include sys/user.h merely for its side-effect of recursively
including other headers.

Implement cancellation points in libc interfaces, as specified by POSIX.

Adjust code to support AMD64, on AMD64, thread needs to set fsbase by
itself before it can execute any other code, so new thread should be
created with all signals are masked until after fsbase is set.

Define _libthr_debug for use by libthread_db.

Change the thread ID (thr_id_t) used for 1:1 threading from being a
pointer to the corresponding struct thread to the thread ID (lwpid_t)
assigned to that thread. The primary reason for this change is that
libthr now internally uses the same ID as the debugger and the kernel
when referencing to a kernel thread. This allows us to implement the
support for debugging without additional translations and/or mappings.

To preserve the ABI, the 1:1 threading syscalls, including the umtx
locking API have not been changed to work on a lwpid_t. Instead the
1:1 threading syscalls operate on long and the umtx locking API has
not been changed except for the contested bit. Previously this was
the least significant bit. Now it's the most significant bit. Since
the contested bit should not be tested by userland, this change is
not expected to be visible. Just to be sure, UMTX_CONTESTED has been
removed from <sys/umtx.h>.

Reviewed by: mtm@
ABI preservation tested on: i386, ia64

Implement pthread_atfork in libthr. This is mostly from deichen's
work in libpthread.

Submitted by: Dan Nelson <dnelson@allantgroup.com>

Make libthr async-signal-safe without costly signal masking. The guidlines I
followed are: Only 3 functions (pthread_cancel, pthread_setcancelstate,
pthread_setcanceltype) are required to be async-signal-safe by POSIX. None of
the rest of the pthread api is required to be async-signal-safe. This means
that only the three mentioned functions are safe to use from inside
signal handlers.
However, there are certain system/libc calls that are
cancellation points that a caller may call from within a signal handler,
and since they are cancellation points calls have to be made into libthr
to test for cancellation and exit the thread if necessary. So, the
cancellation test and thread exit code paths must be async-signal-safe
as well. A summary of the changes follows:

o Almost all of the code paths that masked signals, as well as locking the
pthread structure now lock only the pthread structure.
o Signals are masked (and left that way) as soon as a thread enters
pthread_exit().
o The active and dead threads locks now explicitly require that signals
are masked.
o Access to the isdead field of the pthread structure is protected by both
the active and dead list locks for writing. Either one is sufficient for
reading.
o The thread state and type fields have been combined into one three-state
switch to make it easier to read without requiring a lock. It doesn't need
a lock for writing (and therefore for reading either) because only the
current thread can write to it and it is an integer value.
o The thread state field of the pthread structure has been eliminated. It
was an unnecessary field that mostly duplicated the flags field, but
required additional locking that would make a lot more code paths require
signal masking. Any truly unique values (such as PS_DEAD) have been
reborn as separate members of the pthread structure.
o Since the mutex and condvar pthread functions are not async-signal-safe
there is no need to muck about with the wait queues when handling
a signal ...
o ... which also removes the need for wrapping signal handlers and sigaction(2).
o The condvar and mutex async-cancellation code had to be revised as a result
of some of these changes, which resulted in semi-unrelated changes which
would have been difficult to work on as a separate commit, so they are
included as well.

The only part of the changes I am worried about is related to locking for
the pthread joining fields. But, I will take a closer look at them once this
mega-patch is committed.

o Remove more references to SIGTHR
o Remove clock resolution information left over from libc_r

Remove the garbage collector thread. All resources are freed
in-line. If the exiting thread cannot release a resource, then
the next thread to exit will release it.

Stop using signals for synchronizing threads. The performance penalty
was too much.

Implement PThreads barriers and barrier attributes.

o Refactor and, among other things, get rid of insane nesting levels.
o Fix mutex priority protocols. Keep separate counts of priority
inheritance and protection mutexes to make things easier.
This will not have much affect since this is only the
userland side, and the rest involves kernel scheduling.

o Implement the pthread_spin_* functions in libthr.
o Man pages

Implement reference counting of read-write locks. This uses
a list in the thread structure to keep track of the locks and
how many times they have been locked. This list is checked
on every lock and unlock. The traversal through the list is
O(n). Most applications don't hold so many locks at once that
this will become a problem. However, if it does become a problem
it might be a good idea to review this once libthr is
off probation and in the optimization cycle.
This fixes:
o deadlock when a thread tries to recursively acquire a
read lock when a writer is waiting on the lock.
o a thread could previously successfully unlock a lock it did not own
o deadlock when a thread tries to acquire a write lock on
a lock it already owns for reading or writing [ this is admittedly
not required by POSIX, but is nice to have ]

Make it possible for the library to specify a timeout value when
waiting on a locked mutex. This involves passing a struct timespec
from the pthread mutex locking interfaces all the way down to the
function that suspends the thread until the mutex is released.
The timeout is assumed to be an absolute time (i.e. not relative to
the current time).

Also, in _thread_suspend() make the passed in timespec const.

Preparations to make libthr work in multi-threaded fork()ing applications.

o Remove some code duplication between _thread_init(), which is run once
to initialize libthr and the intitial thread, and pthread_create(), which
initializes newly created threads, into a new function called from both
places: init_td_common()
o Move initialization of certain parts of libthr into a separate
function. These include:
- Active threads list and it's lock
- Dead threads list and it's lock & condition variable
- Naming and insertion of the initial thread into the
active threads list.

Remove _giant_mutex and its associated macros.

Take a stab at fixing some of the macro-nightmare.
PTHREAD_NEW_STATE should work as expected now: a thread
marked PS_RUNNING will get sent a SIGTHR.
Still more cleanups necessary.

Fix the wrapper function around signals so that a signal handling
thread on one of the mutex or condition variable queues is removed
from those queues before the real signal handler is called.

o Add a wrapper around sigaction(2), so we can insert our own wrapper
around signals.
o Lock the process global signal action table.

When _PTHREADSINVARIANTS is defined SIGABRT is not included
in the set of signals to block.
Also, make the PANIC macro call abort() instead of simply
exiting.

Change all instances of THR_LOCK/UNLOCK, etc to UMTX_*.
It is a more acurate description of the locks they
operate on.

There's no need for _umtxtrylock to be a separate function.
Roll it into the pre-existing macro that's used to call it.

Locking primitives and operations in libthr should use struct umtx,
not spinlock_t. Spinlock_t and the associated functions and macros may
require blocking signals in order for async-safe libc functions to behave
appropriately in libthr. This is undesriable for libthr internal locking.
So, this is the first step in completely separating libthr from libc's
locking primitives.

Three new macros should be used for internal libthr locking from now on:
THR_LOCK, THR_TRYLOCK, THR_UNLOCK.

In a critical section, separate the aquisition of the thread lock
and the disabling of signals. What we are really interested in is
keeping track of recursive disabling of signals. We should not
be recursively acquiring thread locks. Any such situations should
be reorganized to not require a recursive lock.

Separating the two out also allows us to block signals independent of
acquiring thread locks. This will be needed in libthr in the near future when
we put the pieces together to protect libc functions that use pthread mutexes
and low level locks.

Make _thread_suspend work with both the old broken sigtimedwait
implementation and the new improved one. We now precompute the
signal set passed to sigtimedwait, using an inverted set when
necessary for compatibility with older kernels.

The move to _retire() a thread in the GC instead of in the thread's
exit function has invalidated the need for _spin[un]lock_pthread().
The _spin[un]lock() functions can now dereference curthread without
the danger that the ldtentry containing the pointer to the thread
has been cleared out from under them.

Teach recent changes in the umtx structure in the kernel to the libthr
initialiazer.

Found by: tinderbox

Decouple the thread stack [de]allocating functions from the 'dead threads list'
lock. It's not really necessary and we don't need the added complexity
or potential for deadlocks.

Approved by: re/blanket libthr

Return gracefully, rather than aborting, when the maximum concurrent
threads per process has been reached. Return EAGAIN, as per spec.

Approved by: re/blanket libthr

_pthread_cancel() breaks the normal lock order of first locking the
joined and then the joiner thread. There isn't an easy (sane?) way
to make it use the correct order without introducing races involving
the target thread and finding which (active or dead) list it is on. So,
after locking the canceled thread it will try to lock the joined thread
and if it fails release the first lock and try again from the top.

Introduce a new function, _spintrylock, which is simply a wrapper arround
umtx_trylock(), to help accomplish this.

Approved by: re/blanket libthr

Start locking up the active and dead threads lists. The active threads
list is protected by a spinlock_t, but the dead list uses a pthread_mutex
because it is necessary to synchronize other threads with the garbage
collector thread. Lock/Unlock macros are used so it's easier to make
changes to the locks in the future.

The 'dead thread list' lock is intended to replace the gc mutex.
This doesn't have any practical ramifications. It simply makes it
clearer what the purpose of the lock is. The gc will use this lock,
instead of the gc mutex, to synchronize access to the dead list with
other threads.

Modify _pthread_exit() to use these two new locks instead of GIANT_LOCK,
and also to properly lock and protect thread state changes,
especially with respect to a joining thread.

The gc thread was also re-arranged to be more organized and less nested.

_pthread_join() was also modified to use the thread list locks. However,
locking and unlocking here needs special care because a thread could find
itself in a position where it's joining an exiting thread that is
waiting on the dead list lock, which this thread (joiner) holds. If the
joiner doesn't take care to lock *and* unlock in the same order they
(the joiner and the joinee) could deadlock against each other.

Approved by: re/blanket libthr

The libthr code makes use of higher-level primitives (pthread_mutex_t and
pthread_cond_t) internaly in addition to the low-level spinlock_t. The
garbage collector mutex and condition variable are two such examples. This
might lead to critical sections nested within critical sections. Implement
a reference counting mechanism so that signals are masked only on the first
entry and unmasked on the last exit.

I'm not sure I like the idea of nested critical sections, but if
the library is going to use the pthread primitives it might be necessary.

Approved by: re/blanket libthr

Add two functions: _spinlock_pthread() and _spinunlock_pthread()
that take the address of a struct pthread as their first argument.
_spin[un]lock() just become wrappers arround these two functions.
These new functions are for use in situations where curthread can't be
used. One example is _thread_retire(), where we invalidate the array index
curthread uses to get its pointer..

Approved by: re/blanket libthr

EDOOFUS
Prevent one thread from messing up another thread's saved signal
mask by saving it in struct pthread instead of leaving it as a
global variable. D'oh!

Approved by: re/blanket libthr

Make WARNS2 clean. The fixes mostly included:
o removed unused variables
o explicit inclusion of header files
o prototypes for externally defined functions

Approved by: re/blanket libthr

Forced commit, for previous revision.

Make state transitions of a thread on a mutex queue
atomic (with respect to other threads and signal handlers).
This includes:
o Introduce two functions to implement atomicity with respect
to other threads and signal handlers. Basically,
_thread_critical_enter() locks the calling thread and blocks
signals. _thread_critical_exit() unblocks signals and unlocks
the thread.

o Introduce two new functions:
get_muncontested() locks a mutex that is not owned by
another thread.
get_mcontested() places a thread on a contested mutex's
queue, taking care to use the _thread_critical_enter/exit
functions to protect thread state.

o Modify mutex_unlock_common() to also protect state transitions.
In this case it needs the cooperation of mutex_queue_deq(), which
must return with the thread locked and signals disabled *before*
it takes the thread off the queue.

Combine _pthread_mutex_lock() and _pthread_mutex_trylock()
into one function: mutex_lock_common(), that can handle
both cases. Its behaviour is controlled by an argument,
int nonblock, which if not zero means do not attempt
to acquire a contested mutex if the uncontested case fails.

BTW, when I write about contested and uncontested mutexes, I'm writing
about it from the application's point of view. I'm not writing about
internal locking of pthread_mutex->lock, which is achieved differently.

While internal mutex locking is mostly done, there's still a bit more
work left in this area.

Approved by: markm/mentor, re/blanket libthr
Reviewed by: jeff (slightly diff. revision)

msg1

- Pass a ucontext_t to _set_curthread. If non-NULL the new thread is set
as curthread in the new context, so that it will be set automatically when
the thread is switched to. This fixes a race where we'd run for a little
while with curthread unset in _thread_start.

Reviewed by: jeff

- Define curthread as _get_curthread() and remove all direct calls to
_get_curthread(). This is similar to the kernel's curthread. Doing
this saves stack overhead and is more convenient to the programmer.
- Pass the pointer to the newly created thread to _thread_init().
- Remove _get_curthread_slow().

- Restore old mutex code from libc_r. It is more standards compliant.
This was changed because originally we were blocking on the umtx and
allowing the kernel to do the queueing. It was decided that the
lib should queue and start the threads in the order it decides and the
umtx code would just be used like spinlocks.

- Add libthr but don't hook it up to the regular build yet. This is an
adaptation of libc_r for the thr system call interface. This is beta
quality code.