MFC r349220: Add wakeup_any(), cheaper wakeup_one() for taskqueue(9).

wakeup_one() and underlying sleepq_signal() spend additional time trying
to be fair, waking thread with highest priority, sleeping longest time.
But in case of taskqueue there are many absolutely identical threads, and
any fairness between them is quite pointless. It makes even worse, since
round-robin wakeups not only make previous CPU affinity in scheduler quite
useless, but also hide from user chance to see CPU bottlenecks, when
sequential workload with one request at a time looks evenly distributed
between multiple threads.

This change adds new SLEEPQ_UNFAIR flag to sleepq_signal(), making it wakeup
thread that went to sleep last, but no longer in context switch (to avoid
immediate spinning on the thread lock). On top of that new wakeup_any()
function is added, equivalent to wakeup_one(), but setting the flag.
On top of that taskqueue(9) is switchied to wakeup_any() to wakeup its
threads.

As result, on 72-core Xeon v4 machine sequential ZFS write to 12 ZVOLs
with 16KB block size spend 34% less time in wakeup_any() and descendants
then it was spending in wakeup_one(), and total write throughput increased
by ~10% with the same as before CPU usage.

MFC r339703, r347365, r347703, r347940

r339703:
Deprecate a number of less used 10 and 10/100 Ethernet devices.

The current deprecated list is: ae, bm, cs, de, dme, ed, ep, ex, fe,
pcn, sf, sn, tl, tx, txp, vx, wb, xe

The list as refined as part of FCP-0101. Per the FCP, devices may be
removed from the deprecation list if enough users are found or they are
converted to iflib.

FCP: https://github.com/freebsd/fcp/blob/master/fcp-0101.md

r347365:
Update dme(4) to reflect that it will not be removed due to FCP-101.

dme(4) is the built-in NIC on a couple non-expandable mips platforms and
thus should remain. The FCP has been updated to reflect this fact.

Discussed with: imp

r347703:
FCP-101: ae(4) is sufficently popular to be moved to the keep list.

r347940:
Remove the notice that ae(4) will be removed in FreeBSD 13.

MFC requested by: rgrimes
Approved by: re (kib)

MFC r336152-r336154, r336157

r336152:
subr_hints: Use goto/label instead of series of conditionals

r336153:
subr_hints: Convert some bool-like ints to bools

r336154:
subr_hints: Skip static_env and static_hints if they don't contain hints

This is possible because, well, they're static. Both the dynamic environment
and the MD-environment (generally loader(8) environment) can potentially
have room for new variables to be set, and thus do not receive this
treatment.

r336157:
kern_environment: bool'itize dynamic_kenv; fix small style(9) nit

kenv MFC: r335998, r336019, r336026, r336036, r336217, r336335, r336337,
r336415-r336416, r336419

As an aside- this has been slightly altered from the version in head to keep
the MD and config-static environments mutually exclusive by default.

This difference is a one-line change in init_static_kenv to setup the MD
environment if the config-static environment is empty or if
loader_env.disabled is explicitly set to 0.

r335998:
kern_environment: use any provided environments, evict hintmode/envmode

At the moment, hintmode and envmode are used to indicate whether static
hints or static env have been provided in the kernel config(5) and the
static versions are mutually exclusive with loader(8)-provided environment.
hintmode *can* be reconfigured later to pull from the dynamic environment,
thus taking advantage of the loader(8) or post-kmem environment setting.

This changeset fixes both problems at once to move us from a semi-confusing
state to a consistent state: if an environment file, hints file, or
loader(8) environment are provided, we use them in a well-known order of
precedence:

- loader(8) environment
- static environment
- static hints file

Once the dynamic environment is setup this becomes a moot point. The
loader(8) and static environments are merged (respecting the above order of
precedence), and the static hints are merged in on an as-needed basis after
the dynamic environment has been setup.

Hints lookup are changed to respect all of the above. Before the dynamic
environment is setup, lookups use the above-mentioned order and fallback to
the next environment if a matching hint is not found. Once the dynamic
environment is setup, that is used on its own since it captures all of the
above information plus any dynamic kenv settings that came up later in boot.

The following tangentially related changes were made to res_find:

- A hintp cookie is now passed in so that related searches continue using
the chain of environments (or dynamic environment) without relying on
global state
- All three environments will be searched if they actually have valid hints
to use, rather than just choosing the first environment that actually had
a hint and rolling with that only

The hintmode sysctl has been ripped out. static_{env,hints}.disabled are
still honored and will disable their respective environments from being used
for hint lookups and from being merged into the dynamic environment, as
expected.

r336019:
config(8): De-dupe hint/env vars within a single file

r335653 flipped the order in which hints/env files are concatenated to match
the order in which vars are processed by the kernel. This is the other
hammer to drop.

Use nv(9) to de-dupe entries within a single `hint` or `env` file, using the
latest value specified for a key. This leaves some duplicates if a variable
is specified in multiple hint/env files or via `envvar` in a kernel config,
but the reversed order of concatenation (from r335653) makes this a
non-issue as the latest-specified version will be seen first.

This change also silently rewrote hint bits to use the same sanitization
process that ian@ wrote for r335642. To the kernel, hints and env vars are
basically the same thing through early boot, then get merged into the
dynamic environment once kmem becomes available and the dynamic environment
is created. They should be subjected to the same restrictions.

libnv has been added to -legacy for the time being to support the build of
config(8) with the new cnvlist API.

r336026:
config(8): Fix broken ABI

r336019 introduced ${SRCTOP}/sys to the include paths in order to pull in a
new sys/{c,}nv.h. This is wrong, because the build tree's ABI isn't
guaranteed to match what's running on the host system.

Fix instead by removing -I${SRCTOP}/sys and installing the libnv headers
with `make -C lib/libnv includes`... this may or may not get re-worked in
the future so that a userland lib isn't installing includes from sys/.

r336036:
kern_environment: Fix SYSINIT ordering

The dynamic environment was being initialized at SI_SUB_KMEM, SI_ORDER_ANY.
I added the hint-merging at SI_SUB_KMEM, SI_ORDER_ANY as well in r335998 -
this can only work by coincidence.

Re-do both to operate at SI_SUB_KMEM + 1, SI_ORDER_FIRST and SI_ORDER_SECOND
respectively to be safe. It's sufficiently obfuscated away as to when in
SU_SUB_KMEM malloc will be available, and the dynamic environment cannot be
relied upon there anyways since it's initialized at SI_ORDER_ANY.

r336217:
kern_environment: Give the static environment a chance to disable MD env

This variable has been given the name "loader_env.disabled" as it's the
primary way most people will have an MD environment. This restores the
previously-default behavior of ignoring the loader(8) environment, which may
be useful for vendor distributions or other scenarios where inheriting the
loader environment may be considered a security issue or potentially
breaking of a more locked-down environment.

As the change to config(5) indicates, disabling the loader environment
should not be a choice made lightly since it may provide ACPI hints and
other useful things that the system can rely on to boot.

An UPDATING entry has been added to mention an upgrade path for those that
may have relied on the previous behavior.

r336335 by arichardson:
No longer install sys/nv.h and sys/cnv.h in lib/libnv/Makefile

Use tools/build/Makefile to install the headers into ${WORLDTMP}/legacy
instead. Compared to r336026 this has the minor advantage that it avoids
unncessary header installation when building the non-bootstrap libnv.

r336337:
Unconditionally build libnv in legacy

Rather than using a config(8) built from new tree linking libnv built on
host.

r336415:
config(8): Add compatibility shims for r335998

Plumb the %VERSREQ from Makefile.<arch> through to the rest of config(8).
We've recorded the config(8) version that we're calling "the end of
envmode and hintmode," and we'll write them out for earlier versions. Later
kernel version bumps will remove envmode/hintmode from the kernel as needed,
which is OK since the current kernel does not use them at all.

These compatibility shims really need to go away when the major version
rolls over...

r336416:
Fix GCC 4.2 build after r336415, proper declaration and prototype

r336419:
config(8): Invert checks; envmode/hintmode reflect "env provided"

Relnotes: yes (maybe) [The loader environment may now be used with
the config-static environment by specifying loader_env.disabled=0 in the
config-static environment]

MFC r335461:
Permit the kernel environment to set an array of numeric values for a single
sysctl(9) node.

Reviewed by: kib@, imp@, jhb@
Differential Revision: https://reviews.freebsd.org/D15802
Sponsored by: Mellanox Technologies

MFC r333994: stop and restart kernel event timers in the suspend / resume cycle

MFC r328522:

Create deprecation management functions.

gone_in(majar, msg); If we're running in FreeBSD major, tell
the user this code may be deleted soon.
If we're running in FreeBSD major - 1,
the the user is deprecated and will
be gone in major.
Otherwise say nothing.

gone_in_dev(dev, major, msg) Just like gone_in, except use device_printf.

New tunable / sysctl debug.oboslete_panic: 0 - don't panic,
1 - panic in major or newer , 2 - panic in major - 1 or newer
default: 0

if NO_OBSOLETE_CODE is defined, then both of these turn into compile
time errors when building for major. Add options NO_OBSOLETE_CODE to
kernel build system.

This lets us tag code that's going away so users know it will be gone,
as well as automatically manage things.

Differential Review: https://reviews.freebsd.org/D13818

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

MFC r330349 and r330362:
Allow pause_sbt() to catch signals during sleep by passing C_CATCH flag.
Define pause_sig() function macro helper similarly to other kernel functions
which catch signals. Update outdated function description.

Document pause_sig(9) and update prototypes for existing pause(9) and
pause_sbt(9) functions.

Discussed with: kib@
Suggested by: cem@
Sponsored by: Mellanox Technologies

MFC r330285:
Remove _Nonnull attributes from user addresses arguments for
copyout(9) family.

MFC r323235,r323236,r324789,r324863:

Introduce __read_frequently

While __read_mostly groups variables together, their placement is not
specified. In particular 2 frequently used variables can end up in
different lines.

This annotation is only expected to be used for variables read all the time,
e.g. on each syscall entry.

=============

Sprinkle __read_frequently on few obvious places.

Note that some of annotated variables should probably change their types
to something smaller, preferably bit-sized.

=============

Mark kdb_active as __read_frequently and switch to bool to eat less space.

=============

Change kdb_active type to u_char.

Fixes warnings from gcc and keeps the small size. Perhaps nesting should be moved
to another variablle.

MFC r319678:

Allow sysctl kern.vm_guest to return bhyve when running under bhyve.

Submitted by: Sean Fagan <sef@ixsystems.com>
Reviewed by: grehan
MFH: 4 weeks.
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D11090

MFC r313853,r313859:

locks: remove SCHEDULER_STOPPED checks from primitives for modules

They all fallback to the slow path if necessary and the check is there.

This means a panicked kernel executing code from modules will be able to
succeed doing actual lock/unlock, but this was already the case for core code
which has said primitives inlined.

==

Introduce SCHEDULER_STOPPED_TD for use when the thread pointer was already read

Sprinkle in few places.

MFC r312888:

Introduce __read_mostly and __exclusive_cache_line macros.

The intended use is to annotate frequently used globals which either rarely
change (and thus can be grouped in the same cacheline) or are an atomic counter
(which means it may benefit from being the only variable in the cacheline).

Linker script support is provided only for amd64. Architectures without it risk
having other variables put in, i.e. as if they were not annotated. This is
harmless from correctness point of view.

MFC r312934:
Make use of clang nullability attributes in C headers.

Replace uses of the GCC __nonnull__ attribute with the clang nullability
qualifiers. These are starting to get use in clang's static analyzer.

Replacement should be transparent for developers using clang. GCC ports
from older FreeBSD versions may need updating if the compiler was built
before r312860 (Jan-27-2017).

Hinted by: Apple's Libc-1158.20.4, Bionic libc

Relnotes: yes

MFC r314253:
Do not leak mount references for dying threads.

MFC r314211:
Remove cpu_deepest_sleep variable.

MFC r302783:

Add explicit detection of KVM hypervisor

Set vm_guest to a new enum value (VM_GUEST_KVM) when kvm is detected and use
vm_guest in conditionals testing for KVM.

Also, fix a conditional checking if we're running in a VM which caught only
the generic VM case, but not more specific VMs (KVM, VMWare, etc.). (Spotted
by: vangyzen).

Sponsored by: Dell Inc.
Approved by: vangyzen (mentor)

MFC r303151:
Provide counter_warning(9) KPI.

MFC r303155:
Hide counted_warning(9) under #ifdef _KERNEL braces.

Approved by: re (gjb)