Merge iflib changes to 11-STABLE

MFC r300147, r300153, r300154, r300215, r301563, r301567,
r302372, r307560, r307562, r307563, r307568, r308792, r311039,
r311837, r312755, r312903, r312905, r312924, r313248, r315217,
r315245, r315288, r316278, r316281, r316502, r316596, r317756,
r319917, r319921, r319984, r319989, r320059, r320609, r320611,
r321253, r321629, r321630, r322337, r322338, r322823, r323077,
r323825, r323876, r323879, r323887, r323941, r323942, r323943,
r323944, r323954, r324038, r324318, r324937, r325166, r325167,
r325168, r325201, r325241, r325245, r325487, r325494, r325901,
r326033, r326369, r326370, r326432, r326577, r326578, r326702,
r326706, r326775, r327013, r327017, r327052, r327072, r327098,
r327242, r327244, r327247, r329651, r329742, r330289, r330715,
r330721, r332419, r332422, r332729

Reviewed by: sbruno
Approved by: re (delphij@)
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D15142

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

MFC r304021: Update iflib to support more NIC designs

- Move group task queue into kern/subr_gtaskqueue.c
- Change intr_enable to return an int so it can be detected if it's not
implemented
- Allow different TX/RX queues per set to be different sizes
- Don't split up TX mbufs before transmit
- Allow a completion queue for TX as well as RX
- Pass the RX budget to isc_rxd_available() to allow an earlier return
and avoid multiple calls

Approved by: sbruno

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

fix loss of taskqueue wakeups (introduced in r300113)

Submitted by: kmacy
Tested by: dchagin

Import the 'iflib' API library for network drivers. From the author:

"iflib is a library to eliminate the need for frequently duplicated device
independent logic propagated (poorly) across many network drivers."

Participation is purely optional. The IFLIB kernel config option is
provided for drivers that want to transition between legacy and iflib
modes of operation. ixl and ixgbe driver conversions will be committed
shortly. We hope to see participation from the Broadcom and maybe
Chelsio drivers in the near future.

Submitted by: mmacy@nextbsd.org
Reviewed by: gallatin
Differential Revision: D5211

Revert r256587.

Requested by: zec

For VIMAGE kernels store vnet in the struct task, and set vnet context
during task processing.

Reported & tested by: mm

Use a safer mechanism for determining if a task is currently running,
that does not rely on the lifetime of pointers being the same. This also
restores the task KBI.

Suggested by: jhb
MFC after: 1 month

Fix taskqueue_drain(9) to not have false negatives. For threaded
taskqueues, more than one task can be running simultaneously.

Also make taskqueue_run(9) static to the file, since there are no
consumers in the base kernel and the function signature needs to change
with this fix.

Remove mention of taskqueue_run(9) and taskqueue_run_fast(9) from the
taskqueue(9) man page.

Reviewed by: jhb
Approved by: zml (mentor)

Revert taskqueue(9) related commits until mdf@ is approved and can
resolve issues.

This reverts commits r207439, r208623, r208624

Revert r207439 and solve the problem differently. The task handler
ta_func may free the task structure, so no references to its members
are valid after the handler has been called. Using a per-queue member
and having waits longer than strictly necessary was suggested by jhb.

Submitted by: Matthew Fleming <matthew.fleming@isilon.com>
Reviewed by: zml, jhb

Handle taskqueue_drain(9) correctly on a threaded taskqueue:

taskqueue_drain(9) will not correctly detect whether a task is
currently running. The check is against a field in the taskqueue
struct, but for a threaded queue with more than one thread, multiple
threads can simultaneously be running a task, thus stomping over the
tq_running field.

Submitted by: Matthew Fleming <matthew.fleming@isilon.com>
Reviewed by: jhb
Approved by: dfr (mentor)

revert changes accidentally included in last commit

import vendor fixes to cxgb

o eliminate modification of task structures after their run to avoid
modify-after-free races when the task structure is malloc'd
o shrink task structure by removing ta_flags (no longer needed with
avoid fix) and combining ta_pending and ta_priority

Reviewed by: dwhite, dfr
MFC after: 4 days

Add taskqueue_drain. This waits for the specified task to finish, if
running, or returns. The calling program is responsible for making sure
that nothing new is enqueued.

# man page coming soon.

Remove (at least temporarily) the check that prevents us from including
this file from userland. Since we export struct ifnet to userland, and
that struct ifnet now contains a struct task, userland needs to know
what struct task looks like.

We need to consider having a pointer to a struct task here instead and
forward declare struct task in the !_KERNEL case.

Defer the vrele() on a jail's root vnode reference from prison_free()
to a new prison_complete() task run by a task queue. This removes
a requirement for grabbing Giant in crfree(). Embed the 'struct task'
in 'struct prison' so that we don't have to allocate memory from
prison_free() (which means we also defer the FREE()).

With this change, I believe grabbing Giant from crfree() can now be
removed, but need to check the uidinfo code paths.

To avoid header pollution, move the definition of 'struct task'
to _task.h, and recursively include from taskqueue.h and jail.h; much
preferably to all files including jail.h picking up a requirement to
include taskqueue.h.

Bumped into by: sam
Reviewed by: bde, tjr