Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r254262 Further improve the msix setup, make sure pci_alloc_msix() gives us
the vectors we requested, and fall back to MSI when not, also release
any allocated resources before the fallback.

MFC r254008 Make the fallback from MSIX to MSI interrupt usage more graceful.

Merge r254263:

Update PCI drivers to no longer look at the MEMIO-enabled bit in the PCI
command register. The lazy BAR allocation code in FreeBSD sometimes
disables this bit when it detects a range conflict, and will re-enable
it on demand when a driver allocates the BAR. Thus, the bit is no longer
a reliable indication of capability, and should not be checked. This
results in the elimination of a lot of code from drivers, and also gives
the opportunity to simplify a lot of drivers to use a helper API to set
the busmaster enable bit.

This changes fixes some recent reports of disk controllers and their
associated drives/enclosures disappearing during boot.

Candidate for 9.2

Submitted by: jhb
Reviewed by: jfv, marius, adrian, achim

MFC: r253284, r253285, r253303:

Correct the Intel network driver module builds. They were not
defining INET or INET6, and in the case of ixgbe this will cause
a panic in the TSO setup code, but in all cases the ioctl behavior
is different, this change makes the module and static consistent.

Approved by: re

MFC of the ixgbe driver including the follow revisions:

235547,235964,241616,241646,241856,241885,242403,
242421,243714,243716,243718,243721,243724,243725,
243728,243729,243733,243735,243736,243741,243833,
244514,246128,246482,247035,247056,247822,247823

Note: 246482 includes only the changes in the ixgbe driver
and the buf_ring supporting code in if_var.h and buf_ring.h
igb and em will be seperate commits, and non-Intel drivers
can commit the changes at will.

Reviewed by:rss, jhb

MFC: r243857 (partial)

Mechanically substitute flags from historic mbuf allocator with
malloc(9) flags in sys/dev.

Merge r241037 from head:
The drbr(9) API appeared to be so unclear, that most drivers in
tree used it incorrectly, which lead to inaccurate overrated
if_obytes accounting. The drbr(9) used to update ifnet stats on
drbr_enqueue(), which is not accurate since enqueuing doesn't
imply successful processing by driver. Dequeuing neither mean
that. Most drivers also called drbr_stats_update() which did
accounting again, leading to doubled if_obytes statistics. And
in case of severe transmitting, when a packet could be several
times enqueued and dequeued it could have been accounted several
times.

o Thus, make drbr(9) API thinner. Now drbr(9) merely chooses between
ALTQ queueing or buf_ring(9) queueing.
- It doesn't touch the buf_ring stats any more.
- It doesn't touch ifnet stats anymore.
- drbr_stats_update() no longer exists.

o buf_ring(9) handles its stats itself:
- It handles br_drops itself.
- br_prod_bytes stats are dropped. Rationale: no one ever
reads them but update of a common counter on every packet
negatively affects performance due to excessive cache
invalidation.
- buf_ring_enqueue_bytes() reduced to buf_ring_enqueue(), since
we no longer account bytes.

o Drivers handle their stats theirselves: if_obytes, if_omcasts.

o mlx4(4), igb(4), em(4), vxge(4), oce(4) and ixv(4) no longer
use drbr_stats_update(), and update ifnet stats theirselves.

o bxe(4) was the most correct driver, it didn't call
drbr_stats_update(), thus it was the only driver accurate under
moderate load. Now it also maintains stats itself.

o ixgbe(4) had already taken stats from hardware, so just
- drop software stats updating.
- take multicast packet count from hardware as well.

o mxge(4) just no longer needs NO_SLOW_STATS define.

o cxgb(4), cxgbe(4) need no change, since they obtain stats
from hardware.

Reviewed by: jfv, gnn

MFC r241917:
Now that device disabling is generic, remove extraneous code from the
device drivers that used to provide this feature.

Approved by: cperciva (implicit)

MFC of 238149

Approved by:re

MFC of the ixgbe driver. Including revisions:
228276,229767,229939,230329,
230572,230775,230790,231796,
232238,234137,234229,234620

MFC 232854,232874,232882,232883,232886 for bus_get_dma_tag()

MFC r228440:

Consistently use types in ixgbe driver code:

- {ixgbe,ixv}_header_split is passed to TUNABLE_INT, so delcare it
int, not bool.
- {ixgbe,ixv}_tx_ctx_setup() returns a boolean value, so declare it
bool, not int.
- {ixgbe,ixv}_tso_setup() returns a bool, so declare it bool, not boolean_t.
- {ixgbe,ixv}_txeof() returns a bool, so declare it bool, not boolean_t.
- Do not re-define bool if the symbol already exists.

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Cut and paste mistake corrected.

First off: update the driver README, the old one was horribly
crusty, and this still isn't perfect, but its at least a bit
more recent.

Secondly, a few improvements to the driver from Andrew Boyer,
support hint to allow devices to not attach, add VLAN_HWTSO
capability so vlans can use TSO, fix in the interrupt handler
to make sure the stack TX queue is processed. Oh, and also
make sure IPv6 does not cause a re-init in the ioctl routine.
Thanks for your efforts Andrew!

Thanks to Claudio Jeker for noticing the ixgbe_xmit() routine
was not correctly swapping the dma map from the first to the
last descriptor in a multi-descriptor transmission, corrected
this.

Do a sweep of the tree replacing calls to pci_find_extcap() with calls to
pci_find_cap() instead.

Update ixgbe driver to verion 2.3.6
- This adds a VM SRIOV interface, ixv, it is however
transparent to the user, it links with the ixgbe.ko,
but when ixgbe is loaded in a virtualized guest with
SRIOV configured this will be detected.
- Sync shared code to latest
- Many bug fixes and improvements, thanks to everyone
who has been using the driver and reporting issues.