Copy stable/11@r303970 to releng/11.0 as part of the 11.0-RELEASE
cycle.

Prune svn:mergeinfo from the new branch, and rename it to RC1.

Update __FreeBSD_version.

Use the quarterly branch for the default FreeBSD.conf pkg(8) repo and
the dvd1.iso packages population.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

sem: Don't free nameinfo that is still in list when open() fails.

This bug could be reproduced easily by calling sem_open() with O_CREAT |
O_EXCL on a semaphore that is already open in the process. The struct
sem_nameinfo would be freed while still in sem_list and later calls to
sem_open() or sem_close() could access freed memory.

PR: 206396
MFC after: 5 days

sem_post(): Fix and document semaphore value overflow error.

The error code is per Austin Group issue #315.

I provided different wording for the manual page change.

Submitted by: pluknet
MFC after: 1 week

The current POSIX semaphore implementation stores the _has_waiters flag
in a separate word from the _count. This does not permit both items to
be updated atomically in a portable manner. As a result, sem_post()
must always perform a system call to safely clear _has_waiters.

This change removes the _has_waiters field and instead uses the high bit
of _count as the _has_waiters flag. A new umtx object type (_usem2) and
two new umtx operations are added (SEM_WAIT2 and SEM_WAKE2) to implement
these semantics. The older operations are still supported under the
COMPAT_FREEBSD9/10 options. The POSIX semaphore API in libc has
been updated to use the new implementation. Note that the new
implementation is not compatible with the previous implementation.
However, this only affects static binaries (which cannot be helped by
symbol versioning). Binaries using a dynamic libc will continue to work
fine. SEM_MAGIC has been bumped so that mismatched binaries will error
rather than corrupting a shared semaphore. In addition, a padding field
has been added to sem_t so that it remains the same size.

Differential Revision: https://reviews.freebsd.org/D961
Reported by: adrian
Reviewed by: kib, jilles (earlier version)
Sponsored by: Norse

Invalidate the cache for the named posix semaphore when opened and
actual file storing the semaphore object is different from the file
created on the first open. Store the file st_dev and st_ino members
of the struct stat in the semaphore structure on open, and compare
them with the attributes of the opened file to detect unlink and
re-creation.

This fixes an issue of sem_unlink(3) failing to flush the named entry
in the semaphore list for the current or remote process, making
sem_unlink(3) not correctly operating if the unlinked semaphore is
still opened.

Reported by: Joris Giovannangeli <joris@giovannangeli.fr>
PR: standards/189353
Reviewed by: jilles (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

Style.

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

To avoid missing a chance to cancel thread, call _pthread_testcancel at the
beginning of _sem_timedwait.

Submitted by: Eric van Gyzen &lt; eric at vangyzen dot net &gt;
MFC after: 3 days

Make more code be protected by internal mutex, and now it is fork-safe, in
error case, the file exclusive lock is now released as soon as possible,
in previous code, child process can still hold the exclusive lock.

Simplify code by using flag O_EXLOCK.

PR: kern/175674

libc: Use O_CLOEXEC for various internal file descriptors.

This fixes a race condition where another thread may fork() before CLOEXEC
is set, unintentionally passing the descriptor to the child process.

This commit only adds O_CLOEXEC flags to open() or openat() calls where no
fcntl(fd, F_SETFD, FD_CLOEXEC) follows. The separate fcntl() call still
leaves a race window so it should be fixed later.

sem_open: Make sure to fail an O_CREAT|O_EXCL open, even if that semaphore
is already open in this process.

If the named semaphore is already open, sem_open() only increments a
reference count and did not take the flags into account (which otherwise
happens by passing them to open()). Add an extra check for O_CREAT|O_EXCL.

PR: kern/166706
Reviewed by: davidxu
MFC after: 10 days

In sem_post, the field _has_waiters is no longer used, because some
application destroys semaphore after sem_wait returns. Just enter
kernel to wake up sleeping threads, only update _has_waiters if
it is safe. While here, check if the value exceed SEM_VALUE_MAX and
return EOVERFLOW if this is true.

Revert previous change. It is an incomplete change from old branch. :-(

Use version 2 of semaphore provided by kernel umtx code, now if there is
no waiters, we still increase and decrease count in user mode without
entering kernel, once there is a waiter, sem_post will enter kernel to
increase count and wake thread up, this is atomicy and allow us to
gracefully destroy semaphore after sem_wait returned.

In revision 231989, we pass a 16-bit clock ID into kernel, however
according to POSIX document, the clock ID may be dynamically allocated,
it unlikely will be in 64K forever. To make it future compatible, we
pack all timeout information into a new structure called _umtx_time, and
use fourth argument as a size indication, a zero means it is old code
using timespec as timeout value, but the new structure also includes flags
and a clock ID, so the size argument is different than before, and it is
non-zero. With this change, it is possible that a thread can sleep
on any supported clock, though current kernel code does not have such a
POSIX clock driver system.

Insert read memory barriers.

To support stack unwinding for cancellation points, add -fexceptions flag
for them, two functions _pthread_cancel_enter and _pthread_cancel_leave
are added to let thread enter and leave a cancellation point, it also
makes it possible that other functions can be cancellation points in
libraries without having to be rewritten in libthr.

preserve errno when processing error cases.

Also call sem_module_init in sem_close to initialize mutex
with some attributes.

Return SEM_FAILED instead of NULL, though there are same, but the
SEM_FAILED is more suitable name.
In function, sem_close(), always set errno on error.

Don't forget to use fourth argument if O_CREAT is set in argument oflag.
The fourth specifies initial value for the semaphore.

More cleanup, remove _libc prefix because libthr no longer has stubs
referencing them.

Don't check has_waiters twice, inline some small functions.
performance result on my machine:
mutex Elapsed: 902115 us; per iteration: 90 ns.
semaphore Elapsed: 958780 us; per iteration: 95 ns.

Use umtx to implement process sharable semaphore, to make this work,
now type sema_t is a structure which can be put in a shared memory area,
and multiple processes can operate it concurrently.
User can either use mmap(MAP_SHARED) + sem_init(pshared=1) or use sem_open()
to initialize a shared semaphore.
Named semaphore uses file system and is located in /tmp directory, and its
file name is prefixed with 'SEMD', so now it is chroot or jail friendly.
In simplist cases, both for named and un-named semaphore, userland code
does not have to enter kernel to reduce/increase semaphore's count.
The semaphore is designed to be crash-safe, it means even if an application
is crashed in the middle of operating semaphore, the semaphore state is
still safely recovered by later use, there is no waiter counter maintained
by userland code.
The main semaphore code is in libc and libthr only has some necessary stubs,
this makes it possible that a non-threaded application can use semaphore
without linking to thread library.
Old semaphore implementation is kept libc to maintain binary compatibility.
The kernel ksem API is no longer used in the new implemenation.

Discussed on: threads@