MFC r337992, r338125:
POSIX compliance improvements in the pthread(3) functions.

This basically adds makes use of the C99 restrict keyword, and also
adds some 'const's to four threading functions: pthread_mutexattr_gettype(),
pthread_mutexattr_getprioceiling(), pthread_mutexattr_getprotocol(), and
pthread_mutex_getprioceiling. The changes are in accordance to POSIX/SUSv4-2018.

Hinted by: DragonFlyBSD

Relnotes: yes

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

MFC r318539 r318580 r318581 r318584 r318749 r318952 r318953 r318955

libthr: fix warnings at WARNS=6

Fix warnings about the following when WARNS=6 (which I will commit soon):

- casting away const
- no previous 'extern' declaration for non-static variable
- others as explained by #pragmas and comments
- unused parameters

libthr: disable thread-safety warnings

These warnings don't make sense for code that implements
the locking primitives.

libthr: change CHECK_AND_INIT_RWLOCK to an inline function

This was prompted by a compiler warning about 'ret' shadowing
a local variable in the callers of the macro.

libthr: Use CLI flags instead of pragmas to disable warnings

People tweaking the build system or compilers tend to look into
the Makefile and not into the source. Having some warning controls
in the Makefile and some in the source code is surprising.

Pragmas have the advantage that they leave the warnings enabled
for more code, but that advantage isn't very relevant in these cases.

libthr: fix warnings from GCC when WARNS=6

Fix warnings about:
- redundant declarations
- a local variable shadowing a global function (dlinfo)
- an old-style function definition (with an empty parameter list)
- a variable that is possibly used uninitialized

libthr: prevent setcontext() from masking SIGTHR

__thr_setcontext() mistakenly tested for the presence of SIGCANCEL
in its local ucontext_t instead of the parameter. Therefore,
if a thread calls setcontext() with a context whose signal mask
contains SIGTHR (a.k.a. SIGCANCEL), that signal will be blocked,
preventing the thread from being cancelled or suspended.

Sponsored by: Dell EMC

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Use __FBSDID() for .c files from lib/libthr/thread.

Sponsored by: The FreeBSD Foundation

Assert that the lock objects put into the off-page, fit into the page.

Sponsored by: The FreeBSD Foundation

Implement process-shared locks support for libthr.so.3, without
breaking the ABI. Special value is stored in the lock pointer to
indicate shared lock, and offline page in the shared memory is
allocated to store the actual lock.

Reviewed by: vangyzen (previous version)
Discussed with: deischen, emaste, jhb, rwatson,
Martin Simmons <martin@lispworks.com>
Tested by: pho
Sponsored by: The FreeBSD Foundation

Follow changes made in revision 232144, pass absolute timeout to kernel,
this eliminates a clock_gettime() syscall.

Revert revision 214007, I realized that MySQL wants to resolve
a silly rwlock deadlock problem, the deadlock is caused by writer
waiters, if a thread has already locked a reader lock, and wants to
acquire another reader lock, it will be blocked by writer waiters,
but we had already fixed it years ago.

Add pthread_rwlockattr_setkind_np and pthread_rwlockattr_getkind_np, the
functions set or get pthread_rwlock type, current supported types are:
PTHREAD_RWLOCK_PREFER_READER_NP,
PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP,
PTHREAD_RWLOCK_PREFER_WRITER_NP,
default is PTHREAD_RWLOCK_PREFER_WRITER_NONCECURSIVE_NP, this maintains
binary compatible with old code.

In current code, statically initialized and destroyed object have
same null value, the code can not distinguish between them, to
fix the problem, now a destroyed object is assigned to a non-null
value, and it will be rejected by some pthread functions.
PTHREAD_ADAPTIVE_MUTEX_INITIALIZER_NP is changed to number 1, so that
adaptive mutex can be statically initialized correctly.

In the current code, rdlock_count is not correctly handled for some cases.
The most notable is that it is not bumped in rwlock_rdlock_common() when
the hard path (__thr_rwlock_rdlock()) returns successfully.
This can lead to deadlocks in libthr when rwlocks recursion in read mode
happens.
Fix the interested parts by correctly handling rdlock_count.

PR: threads/136345
Reported by: rink
Tested by: rink
Reviewed by: jeff
Approved by: re (kib)
MFC: 2 weeks

don't include pthread_np.h, it is not used.

Replace userland rwlock with a pure kernel based rwlock, the new
implementation does not switch pointers when it resumes waiters.

Asked by: jeff

Restore normal pthread_cond_signal path to avoid some obscure races.

return EAGAIN early rather than running bunch of code later, micro optimize
static branch prediction.

Rewrite rwlock to user atomic operations to change rwlock state, this
eliminates internal mutex lock contention when most rwlock operations
are read.

Orignal patch provided by: jeff

Remove multiple _get_curthread() calls.

WARNS level 4 cleanup.

Import my recent 1:1 threading working. some features improved includes:
1. fast simple type mutex.
2. __thread tls works.
3. asynchronous cancellation works ( using signal ).
4. thread synchronization is fully based on umtx, mainly, condition
variable and other synchronization objects were rewritten by using
umtx directly. those objects can be shared between processes via
shared memory, it has to change ABI which does not happen yet.
5. default stack size is increased to 1M on 32 bits platform, 2M for
64 bits platform.
As the result, some mysql super-smack benchmarks show performance is
improved massivly.

Okayed by: jeff, mtm, rwatson, scottl

o Assertions to catch that stuff that shouldn't happen is not happening.
o In the rwlock code: move a duplicated check inside an if..else to after
the if...else clause.
o When initializing a static rwlock move the initialization check
inside the lock.
o In thr_setschedparam.c: When breaking out of the trylock...retry if busy
loop make sure to reset the mtx pointer to null if the mutex is nolonger
in a queue.

There are consumers of rwlocks, inluding our own libc, that depend on
a PTHREAD_RWLOCK_INITIALIZER to do for rwlocks what
a similarly named symbol does for statically initialized mutexes.
This symbol was dropped in The Open Group Base Specifications Issue 6
and does not exist in IEEE Std 1003.1, 2003, but it should still be
supported for backwards compatibility.

Pointy hat: mtm

I update the rwlock code in libthr to be more standards compliant and
what do I get for my troubles? libc breaks offcourse!

Reimplement a hack (in libthr) that allows libc to use
rwlocks without initializing them first. The hack was reimplemented
so that only a private libc version of the rwlock locking functions
initializes an uninitialized rwlock. The application version will
correctly fail.

Implement reference counting of read-write locks. This uses
a list in the thread structure to keep track of the locks and
how many times they have been locked. This list is checked
on every lock and unlock. The traversal through the list is
O(n). Most applications don't hold so many locks at once that
this will become a problem. However, if it does become a problem
it might be a good idea to review this once libthr is
off probation and in the optimization cycle.
This fixes:
o deadlock when a thread tries to recursively acquire a
read lock when a writer is waiting on the lock.
o a thread could previously successfully unlock a lock it did not own
o deadlock when a thread tries to acquire a write lock on
a lock it already owns for reading or writing [ this is admittedly
not required by POSIX, but is nice to have ]

Add an implementation of pthread_rwlock_timed{rd,wr}lock() to libthr with
attendant documentation.

o We are not required to initialize an invalid rwlock. So axe all that
code and simply return EINVAL (which is allowed by the standard) in
all those pthread functions that previously initialized it.

o Refactor the pthread_rwlock_[try]rdlock() and pthread_rwlock_[try]wrlock()
functions. They are now completeley condensed into rwlock_rdlock_common()
and rwlock_wrlock_common(), respectively.

o If the application tries to destroy an rwlock that is currently
held by a thread return EBUSY where it previously went ahead and
freed all resources associated with the lock.

o Refactor _pthread_rwlock_init() to make it look (relatively) sane.

o When obtaining a read lock on an rwlock the check for whether it
would exceed the maximum allowed read locks should happen *before*
we obtain the lock.

o The pthread_rwlock_* functions shall *never* return EINTR, so make
sure to requeue/resuspend the thread if it encounters such an error.

o Make a note that pthread_rwlock_unlock() needs to ensure it holds a
lock on an rwlock it tries to unlock. It will be implemented in a
separate commit because it requires some additional rwlock infrastructure.

- Add libthr but don't hook it up to the regular build yet. This is an
adaptation of libc_r for the thr system call interface. This is beta
quality code.