MFC r359653:
Count number of times transmit ring is out of buffers in mlx5en(4).

Differential Revision: https://reviews.freebsd.org/D24273
Sponsored by: Mellanox Technologies

MFC r338492:
Add support for receive side scaling stride, RSSS, in mlx5en(4).

The receive side scaling stride parameter is a value which define the interval
between active receive side queues. The traffic for the inactive queues is
redirected to the nearest active queue by use of modulus. The default value
of this parameter is one, which means all receive side queues are used.

The point of this feature is to redirect more traffic to fewer receive side
queues in order to take more advantage of sorted large receive offload,
sorted LRO. The sorted LRO works better when more packets are accumulated
per service interval.

Sponsored by: Mellanox Technologies

MFC r338489:
Maximum number of mbuf frags is off-by-one for worst case scenario in mlx5en(4).

Inspecting the PRM no more than 0x3F data segments, DS, of size 16 bytes is
allowed.

Worst case scenario summary of DS usage:
Header is fixed: 2 DS
Maximum inlining: 98 => (98 - 2) / 16 = 6 DS
Remainder: 0x3F - 2 - 6 = 55 DS (mbuf frags)

Previously a value of 56 DS was used and this would work in the
normal case because not all inline data area was used up.

Sponsored by: Mellanox Technologies

MFC r325661:
Expose the current hardware MTU in mlx5en(4) as a separate entry
in the sysctl tree.

Sponsored by: Mellanox Technologies

MFC r336450:
Do not inline transmit headers and use HW VLAN tagging if supported by mlx5en(4).

Query the minimal inline mode supported by the card.
When creating a send queue, cache the queried mode and optimize the transmit
if no inlining is required. In this case, we can avoid touching the headers
cache line and avoid dirtying several more lines by copying headers into
the send WQEs. Also, if no inline headers are used, hardware assists in
the VLAN tag framing.

Submitted by: kib@, slavash@
Sponsored by: Mellanox Technologies

MFC r336407:
Handle jumbo frames without requiring big clusters in mlx5en(4).

The scatter list is formed by the chunks of MCLBYTES each, and larger
than default packets are returned to the stack as the mbuf chain.

Submitted by: kib@
Sponsored by: Mellanox Technologies

MFC r328591:
Move the mlx5 core device pointer first in the mlx5en priv. This help simplify
checks to recognize own network devices when using mlx5ib. This patch fixes
an issues where mlx5ib fails to recognize mceX network devices for use with
RoCE.

Sponsored by: Mellanox Technologies

MFC r324202:
Make sure the doorbell lock is valid for the i386 version
of the mlx5en(4) driver.

Tested by: gallatin @
Sponsored by: Mellanox Technologies

MFC r312872:
Add support for reading advanced diagnostic counters.

By default reading the diagnostic counters is disabled. The firmware
decides which counters are supported and only those supported show up
in the dev.mce.X.diagnostics sysctl tree.

To enable reading of diagnostic counters set one or more of the
following sysctls to one:

dev.mce.X.conf.diag_general_enable=1
dev.mce.X.conf.diag_pci_enable=1

Sponsored by: Mellanox Technologies

MFC r312865:
Enforce reading the consumer and producer counters once to ensure
consistent return values from the mlx5e_sq_has_room_for()
function. The two counters are incremented by different threads under
different locks.

Sponsored by: Mellanox Technologies

MFC r312536:
Allow transmit packet bufring in software to be disabled.

- Add new sysctl node to control the transmit packet bufring.

- Add optimised version of the transmit routine which output packets
directly to the DMA ring instead of using bufring in case the transmit
lock is congested. This can reduce the number of taskswitches which in
turn influence the overall system CPU usage, depending on the
workload.

- Add " TX" suffix to debug name for transmit mutexes to silence some
witness warnings about aquiring duplicate locks having same name.

Sponsored by: Mellanox Technologies
Suggested by: gallatin @

MFC r312528:
Make draining a sendqueue more robust.

Add own state variable to track if a sendqueue is stopped or not.
This will prevent traffic from entering the sendqueue while it is
being destroyed.

Update drain function to wait for traffic to be transmitted before
returning when the link state is active.

Add extra checks in transmit path for stopped SQ's.

While at it:
- Use likely() for a mbuf pointer check.
- Remove redundant IFF_DRV_RUNNING check.

Sponsored by: Mellanox Technologies

MFC r308416:
Add timer to watch the RQ when we are out of mbufs.

The firmware/hardware does not generate additional completion
events unless we post new buffers. Use a timer to try to post
more buffers in case we are temporarily out of mbufs. Else
the receive schedule completely stops.

Sponsored by: Mellanox Technologies

MFC r305876:
mlx5en: Remove unused pdev pointer.

Sponsored by: Mellanox Technologies

MFC r305873:
mlx5en: Factor out common sendqueue code for use with rate limiting SQs.

Try to reuse code to setup sendqueues when possible by making some static
functions global. Further split the mlx5e_close_sq_wait() function to
separate out reusable parts.

Sponsored by: Mellanox Technologies

MFC r305872:
mlx5en: Properly declare doorbell lock for 32-bit CPUs.

Sponsored by: Mellanox Technologies

MFC r305871:
mlx5en: Optimise away duplicate UAR pointers.

This change also reduces the size of the mlx5e_sq structure so that the last
queue_state element will fit into the previous cacheline and then the mlx5e_sq
structure becomes one cacheline less for amd64.

Sponsored by: Mellanox Technologies

MFC r305870:
mlx5en: Make the mlx5e_open_cq() and mlx5e_close_cq() functions global.

Make some functions and structures global to allow for code reuse
when creating rate limiting sendqueues.

Sponsored by: Mellanox Technologies

MFC r305868:
mlx5en: Separate the sendqueue from using the mlx5e_channel structure.

This change allows for reusing the transmit path for so called
rate limited senqueues. While at it optimise some pointer lookups
in the fast path.

Sponsored by: Mellanox Technologies

MFC r304601:
Increase the maximum RX/TX queue size. This allows for a RX/TX queue
size of 16384 mbufs. Previously the limit was 8192.

Sponsored by: Mellanox Technologies

MFC r300280:
Optimise use of doorbell and remove redundant NOPs

Store the last doorbell write in the mlx5e_sq structure and write the
doorbell to the hardware when the transmit routine finishes
transmitting all queued mbufs.

Sponsored by: Mellanox Technologies
Tested by: Netflix

MFC r300277:
Implement TX completion event interleaving.

This patch implements a sysctl which allows setting a factor, N, for
how many work queue elements can be generated before requiring a
completion event. When a completion event happens the code simulates N
completion events instead of only one. When draining a transmit queue,
N-1 NOPs are transmitted at most, to force generation of the final
completion event. Further a timer is running every HZ ticks to flush
any remaining data off the transmit queue when the tx_completion_fact
> 1.

The goal of this feature is to reduce the PCI bandwidth needed when
transmitting data.

Sponsored by: Mellanox Technologies
Tested by: Netflix

MFC r294314:
Allow RX and TX pause frames to be set through ifconfig.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4817

MFC r292949:
Add support for modifying coalescing parameters runtime.

Sponsored by: Mellanox Technologies

MFC r292946:
10G ER/LR should present itself as LR.

MFC after: 1 week
Submitted by: Shahar Klein <shahark@mellanox.com>
Sponsored by: Mellanox Technologies

MFC r292838:
Add support for CQE zipping. CQE zipping reduces PCI overhead by
coalescing and zipping multiple CQEs into a single merged CQE. The
feature is enabled by default and can be disabled by a sysctl.

Implementing this feature mlx5_cqwq_pop() has been separated from
mlx5e_get_cqe().

Submitted by: Mark Bloch <markb@mellanox.com>
Differential Revision: https://reviews.freebsd.org/D4598
Sponsored by: Mellanox Technologies

MFC r291938:
Add full support for Receive Side Scaling, RSS, to the mlx5en
driver. This includes binding all interrupt and worker threads
according to the RSS configuration, setting up correct Toeplitz
hashing keys as given by RSS and setting the correct mbuf
hashtype for all received traffic.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4410

MFC r291932:
Add support for setting the TX moderation mode via a sysctl entry. TX
completion events can be moderated in the same way like RX completion
events. Expose this functionality by a sysctl variable.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4409

MFC r290650, r290709, r291067, r291068, r291069 and r291070:
Add mlx5 and mlx5en driver(s) for ConnectX-4 and ConnectX-4LX cards
from Mellanox Technologies. The current driver supports ethernet
speeds up to and including 100 GBit/s. Infiniband support will be
done later.

The code added is not compiled by default, which will be done by a
separate commit.

Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4177
Differential Revision: https://reviews.freebsd.org/D4178
Differential Revision: https://reviews.freebsd.org/D4179
Differential Revision: https://reviews.freebsd.org/D4180