MFC r349220: Add wakeup_any(), cheaper wakeup_one() for taskqueue(9).

wakeup_one() and underlying sleepq_signal() spend additional time trying
to be fair, waking thread with highest priority, sleeping longest time.
But in case of taskqueue there are many absolutely identical threads, and
any fairness between them is quite pointless. It makes even worse, since
round-robin wakeups not only make previous CPU affinity in scheduler quite
useless, but also hide from user chance to see CPU bottlenecks, when
sequential workload with one request at a time looks evenly distributed
between multiple threads.

This change adds new SLEEPQ_UNFAIR flag to sleepq_signal(), making it wakeup
thread that went to sleep last, but no longer in context switch (to avoid
immediate spinning on the thread lock). On top of that new wakeup_any()
function is added, equivalent to wakeup_one(), but setting the flag.
On top of that taskqueue(9) is switchied to wakeup_any() to wakeup its
threads.

As result, on 72-core Xeon v4 machine sequential ZFS write to 12 ZVOLs
with 16KB block size spend 34% less time in wakeup_any() and descendants
then it was spending in wakeup_one(), and total write throughput increased
by ~10% with the same as before CPU usage.

MFC r330349 and r330362:
Allow pause_sbt() to catch signals during sleep by passing C_CATCH flag.
Define pause_sig() function macro helper similarly to other kernel functions
which catch signals. Update outdated function description.

Document pause_sig(9) and update prototypes for existing pause(9) and
pause_sbt(9) functions.

Discussed with: kib@
Suggested by: cem@
Sponsored by: Mellanox Technologies

MFC r315280 r315287

When the RTC is adjusted, reevaluate absolute sleep times based on the RTC

POSIX 2008 says this about clock_settime(2):

If the value of the CLOCK_REALTIME clock is set via clock_settime(),
the new value of the clock shall be used to determine the time
of expiration for absolute time services based upon the
CLOCK_REALTIME clock. This applies to the time at which armed
absolute timers expire. If the absolute time requested at the
invocation of such a time service is before the new value of
the clock, the time service shall expire immediately as if the
clock had reached the requested time normally.

Setting the value of the CLOCK_REALTIME clock via clock_settime()
shall have no effect on threads that are blocked waiting for
a relative time service based upon this clock, including the
nanosleep() function; nor on the expiration of relative timers
based upon this clock. Consequently, these time services shall
expire when the requested relative interval elapses, independently
of the new or old value of the clock.

When the real-time clock is adjusted, such as by clock_settime(3),
wake any threads sleeping until an absolute real-clock time.
Such a sleep is indicated by a non-zero td_rtcgen. The sleep functions
will set that field to zero and return zero to tell the caller
to reevaluate its sleep duration based on the new value of the clock.

At present, this affects the following functions:

pthread_cond_timedwait(3)
pthread_mutex_timedlock(3)
pthread_rwlock_timedrdlock(3)
pthread_rwlock_timedwrlock(3)
sem_timedwait(3)
sem_clockwait_np(3)

I'm working on adding clock_nanosleep(2), which will also be affected.

Reported by: Sebastian Huber <sebastian.huber@embedded-brains.de>
Relnotes: yes
Sponsored by: Dell EMC

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Remove a reference to the PBDRY flag, which itself was removed in r248470.

MFC after: 3 days

use .Mt to mark up email addresses consistently (part6)

PR: 191174
Submitted by: Franco Fichtner <franco at lastsummer.de>

Correct sentence syntax too.

Correct spelling.

MFcalloutng:
Dcoument the new functions added to condvar(9), sleep(9), sleepqueue(9)
KPIs. Also document recent changes in timeout(9) and eventtimers(4).

Fix some minor inaccuracies introduced in r243251.
Also correct the comment in kern_synch.c which was the source of the
problematic text.

Reviewed by: kib (previous version)
Approved by: hrs (mentor)

Improve grammar.

Approved by: hrs (mentor)

Remove trailing whitespace per mdoc lint warning

Disussed with: gavin
No objection from: doc
Approved by: joel
MFC after: 3 days

document that even when wakeup_one is used to wakeup, msleep and friends
may still return a non-zero value... You are not guaranteed to get a one
to one mapping between wakeup_one and zero return values...

Reviewed by: kib
MFC after: 3 days

mdoc: order prologue macros consistently by Dd/Dt/Os

Although groff_mdoc(7) gives another impression, this is the ordering
most widely used and also required by mdocml/mandoc.

Reviewed by: ru
Approved by: philip, ed (mentors)

Document PBDRY and SLEEPQ_STOP_ON_BDRY.

Requested and reviewed by: attilio
MFC after: 3 days

Permit Giant to be passed as the explicit interlock either to
msleep/mtx_sleep or the various cv_*wait*() routines. Currently, the
"unlock" behavior of PDROP and cv_wait_unlock() with Giant is not
permitted as it is will be confusing since Giant is fully unrecursed and
unlocked during a thread sleep.

This is handy for subsystems which wish to allow unlocked drivers to
continue to use Giant such as CAM, the new TTY layer, and the new USB
stack. CAM currently uses a hack that I told Scott to use because I
really didn't want to permit this behavior, and the TTY and USB patches
both have various patches to permit this.

MFC after: 2 weeks

Add note about PZERO being obsolete, because so much code uses it.
Feel free to improve the verbage, since this was a compromise between
conflicting feedback I got on my original version.

Add some cross references to locking.9 from related pages.

Typo.

Allow threads to atomically release rw and sx locks while waiting for an
event. Locking primitives that support this (mtx, rw, and sx) now each
include their own foo_sleep() routine.
- Rename msleep() to _sleep() and change it's 'struct mtx' object to a
'struct lock_object' pointer. _sleep() uses the recently added
lc_unlock() and lc_lock() function pointers for the lock class of the
specified lock to release the lock while the thread is suspended.
- Add wrappers around _sleep() for mutexes (mtx_sleep()), rw locks
(rw_sleep()), and sx locks (sx_sleep()). msleep() still exists and
is now identical to mtx_sleep(), but it is deprecated.
- Rename SLEEPQ_MSLEEP to SLEEPQ_SLEEP.
- Rewrite much of sleep.9 to not be msleep(9) centric.
- Flesh out the 'RETURN VALUES' section in sleep.9 and add an 'ERRORS'
section.
- Add __nonnull(1) to _sleep() and msleep_spin() so that the compiler will
warn if you try to pass a NULL wait channel. The functions already have
a KASSERT to that effect.

Another typo.

Spotted by: ru
Another pointy hat to: grog

Cosmetics: remove trailing white space, wrap over-long lines.

Pointy hat to: grog

Rearrange function descriptions in more logical order.
Be less alarmist about the dangers of abusing wakeup_one().

Requested by: kris, imp

Furhter clarifications:

- the issues with wakeup_one are due to address space clashes between
unrelated groups of threads.
- sleep() was removed in FreeBSD 2.2.
- date the page today, not 4 days ago.
- replace grammatically correct "woken" with "woken up" for
consistency with the function name.

In FreeBSD 5.x+, sleep/wakeup operate on threads, not processes.

Fix markup and grammar bugs in recent revisions.

Update HISTORY.

Reviewed by: dmr

Add warning about deadlocks created by use of wakeup_one.

Add a new kernel sleep function pause(9). pause(9) is for places that
want an equivalent of DELAY(9) that sleeps instead of spins. It accepts
a wmesg and a timeout and is not interrupted by signals. It uses a private
wait channel that should never be woken up by wakeup(9) or wakeup_one(9).

Glanced at by: phk

Markup fixes.

- Sleep/wakeup operate on threads now, not processes.
- Describe msleep() as the primary sleep function now rather than tsleep()
and describe tsleep() and msleep_spin() as variations.
- Try to make the description of msleep() a bit closer to English
(sentences with actual subjects, etc.)
- Document that a priority of 0 now prevents the thread's priority from
being altered.
- Add a history note for wakeup_one().

allow timo to be 0 when mtx is not NULL (KASSERT allowed this as
pointed out by jhb)

Reviewed by: jhb
MFC after: 3 days

- Don't list sleep() in the NAME section as we haven't had a sleep()
function in years.
- Change the 'ident' paramters to 'wchan' to match <sys/systm.h>.
- Use 'otherwise' in place of 'else' in one place so that this reads like
English rather than C.
- Document the new msleep_spin() function.
- Add history notes for msleep() and msleep_spin().

Use 'manual page' instead of 'man page' for consistency.

Approved by: re (hrs)

Point out that tsleep(... , 0) is not permitted if the Giant lock is not
held. (It causes a "sleeping without a mutex" assertion failure in
msleep.)

MFC after: 3 days

o Be consistent with the declaration: pri -> priority.

Spotted by: Dmitry Rzhavin
MFC after: 1 week

Mdoc Janitor:

* Fix hard sentence breaks.

Some mdoc(7) fixes:

Nuke incorrect usage of .Ar; replace it with .Vt, .Va or .Fa appropriately.
The .Ar mdoc(7) specifier should only be used when displaying command line
arguments.

Approved by: des (mentor)

- Use better terminology when describing mutex operations in msleep(9)'s
description.
- Remove some bogus commas.
- Use the past tense when referring to the removal of the sleep() function
since it happened quite a while ago and since the previous sentence in the
paragraph already uses the past tense.

Approved by: re (rwatson)

mdoc(7) police: Scheduled sweep.

sleep -> msleep

mdoc(4) police: Markup. Fix 1 hard sentence break.

mdoc(7) police: Remove hard sentence breaks.

mdoc(7) police: markup nits.

mdoc(7) police: macroize UNIX.

Fix speling errur.

mdoc(7) police: Use the new .In macro for #include statements.

Remove references to asleep/await.

mdoc(7) police:

Avoid using parenthesis enclosure macros (.Pq and .Po/.Pc) with plain text.
Not only this slows down the mdoc(7) processing significantly, but it also
has an undesired (in this case) effect of disabling hyphenation within the
entire enclosed block.

Removed whitespace at end-of-line; no content changes. I simply did
cd src/share; find man[1-9] -type f|xargs perl -pi -e 's/[ \t]+$//'

BTW, what editors are the culprits? I'm using vim and it shows
me whitespace at EOL in troff files with a thick blue block...

Reviewed by: Silence from cvs diff -b
MFC after: 7 days

mdoc(7) police: split punctuation characters + misc fixes.

mdoc(7) police: Er macro usage cleanup.

mdoc(7) police: do not split author names in the AUTHORS section.

mdoc(7) police: use the new features of the Nm macro.

Avoid use of direct troff requests in mdoc(7) manual pages.

Replace non-existent 'mtx_t' typedef'd type with 'struct mtx'.

Add scant documentation for msleep.

Improve the cross-references.

Fix grammar and a function name.

Small grammar gnit.

Fix a typo (simple --> simply).

Move the description of the traditional sleep() form out into the history
section, we no longer have it.

Fix mdoc style violations caused by a previous committer.

$Id$ -> $FreeBSD$

Various man page cleanup:

- Sort xrefs
- FreeBSD.ORG -> FreeBSD.org
- Be consistent with section names as outlined in mdoc(7).
- Other misc mdoc cleanup.

PR: doc/13144
Submitted by: Alexey M. Zelkin <phantom@cris.net>

Fix some spelling errors.

update asleep/await sections

update HISTORY section

Add manual page for experimental kernel asleep() and await() routines

Fixed prototype for tsleep() (const poisoning).

Fixed synopsis:
- don't declare nonexistent function sleep().
- don't include <sys/errno.h> explicitly.

sleep() should be nuked some more, e.g., this man page should not be
named after a nonexistent function.

Document wakeup_one().

Revert $FreeBSD$ back to $Id$

Make the long-awaited change from $Id$ to $FreeBSD$

This will make a number of things easier in the future, as well as (finally!)
avoiding the Id-smashing problem which has plagued developers for so long.

Boy, I'm glad we're not using sup anymore. This update would have been
insane otherwise.

Fix some typos.

Fix a typo.

.Ql --> .Dv

Submitted by: wollman

List tsleep & wakeup in the summary line so that they can
be found with "man -k".

Populate this. :)

Add a man page for tsleep()/wakeup().