MFC r322248:
Fix for mlx4en(4) to properly call m_defrag().

The m_defrag() function can only defrag mbuf chains which have a valid
mbuf packet header. In r291699 when the mlx4en(4) driver was converted
into using BUSDMA(9), the call to m_defrag() was moved after the part
of the transmit routine which strips the header from the mbuf chain.
This effectivly disabled the mbuf defrag mechanism and such packets
simply got dropped.

This patch removes the stripping of mbufs from a chain and loads all
mbufs using busdma. If busdma finds there are no segments, unload
the DMA map and free the mbuf right away, because that means all
data in the mbuf has been inlined in the TX ring. Else proceed
as usual.

Add a per-ring rounter for the number of defrag attempts and
make sure the oversized_packets counter gets zeroed while at it.

The counters are per-ring to avoid excessive cache misses in the
TX path.

Approved by: re (kib)
Submitted by: mjoras@
Differential Revision: https://reviews.freebsd.org/D11683
Sponsored by: Mellanox Technologies

MFC r322304:
Add support for RX and TX statistics when the mlx4en(4) PCI device
is in VF or SRIOV mode typically in a virtual machine environment.

Approved by: re (kib)
Submitted by: Sepherosa Ziehau <sephe@dragonflybsd.org>
Sponsored by: Mellanox Technologies

MFC r296987:
Add missing curly brackets in for loop.

Sponsored by: Mellanox Technologies

MFC r290710, r291694, r291699 and r291793:
- Fix print formatting compile warnings for Sparc64 and PowerPC platforms.
- Updated the mlx4 and mlxen drivers to the latest version, v2.1.6:
- Added support for dumping the SFP EEPROM content to dmesg.
- Fixed handling of network interface capability IOCTLs.
- Fixed race when loading and unloading the mlxen driver by applying
appropriate locking.
- Removed two unused C-files.
- Convert the mlxen driver to use the BUSDMA(9) APIs instead of
vtophys() when loading mbufs for transmission and reception. While at
it all pointer arithmetic and cast qualifier issues were fixed, mostly
related to transmission and reception.
- Fix i386 build WITH_OFED=YES. Remove some redundant KASSERTs.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4283
Differential Revision: https://reviews.freebsd.org/D4284

MFC r272027:

Hardware driver update from Mellanox Technologies, including:
- improved performance
- better stability
- new features
- bugfixes

Supported HCAs:
- ConnectX-2
- ConnectX-3
- ConnectX-3 Pro

NOTE:
- TSO feature needs r271946, which is not yet merged.

Sponsored by: Mellanox Technologies
Approved by: re, glebius

MFC r272027:

Hardware driver update from Mellanox Technologies, including:
- improved performance
- better stability
- new features
- bugfixes

Supported HCAs:
- ConnectX-2
- ConnectX-3
- ConnectX-3 Pro

NOTE:
- TSO feature needs r271946, which is not yet merged.

Sponsored by: Mellanox Technologies
Approved by: re, glebius