MFC 064009e79462:
Add support for enabling and disabling IFCAP_VLAN_HWTSO via
ifconfig(8) in mlx5en(4).

Sponsored by: Mellanox Technologies // NVIDIA Networking

(cherry picked from commit 064009e79462dea517aa7f1a857fb4d5393caa69)

Git Hash: 8340ed8900e4ff48e7a428f30a98c0e91c91da03
Git Author: hselasky@FreeBSD.org

MFC 9a47ae044b48:
Bump driver versions for mlx5en(4) and mlx4en(4).

Sponsored by: Mellanox Technologies // NVIDIA Networking

Git Hash: e87e3e82f3a062856118ed42751b498277eb09a5
Git Author: hselasky@FreeBSD.org

MFC r367612:
mlx5en: Set ifmr_current same as ifmr_active.

Git Hash: 2695791408f5da6df47cdc83307f9bd2af3b6750
Git Author: kib@FreeBSD.org

MFC r367611:
mlx5en: stop ignoring pauses and flow in the media reqs.

Git Hash: 2cd3c13be316974ac580ad55809e619ef230712f
Git Author: kib@FreeBSD.org

MFC r367610:
mlx5en: Register all combinations of FDX/RXPAUSE/TXPAUSE as valid media types.

Git Hash: 65dd92f6043da77865e2900b6aab75e75cecdb4c
Git Author: kib@FreeBSD.org

MFC r367609:
mlx5en: Refactor repeated code to register media type to mlx5e_ifm_add().

Git Hash: c408d03b378aeebba1faa09fe62c1562ad34295a
Git Author: kib@FreeBSD.org

MFC a00718e1dfcd:
Implement SIOCGIFRSSKEY and SIOCGIFRSSHASH and mlx5en(4).

Sponsored by: Mellanox Technologies // NVIDIA Networking

Git Hash: 43bf82ace68d9a9bb9315e829b0a548b33d2cfb3
Git Author: hselasky@FreeBSD.org

MFC 89c0b4fa1172:
Bump some copyrights in mlx5en(4).

Sponsored by: Mellanox Technologies // NVIDIA Networking

Git Hash: 198471ed6b15b6f0a1ed4b7ebf21d0a5c910d67e
Git Author: hselasky@FreeBSD.org

MFC 376479200760:
Fix whitespace in mlx5en(4).

Sponsored by: Mellanox Technologies // NVIDIA Networking

Git Hash: 026b14030bdff05587c0db680bfba255b5df01b3
Git Author: hselasky@FreeBSD.org

MFC r367717:
Enhance the mlx5_core_create_cq() function in mlx5core.

Enhance mlx5_core_create_cq() to get the command out buffer from the
callers to let them use the output.

Linux commit:
38164b771947be9baf06e78ffdfb650f8f3e908e

Sponsored by: Mellanox Technologies // NVIDIA Networking

MFC r365719:
Poll statistics more frequently in mlx5en(4).

This makes traffic steering algorithms more accurate.

Submitted by: gallatin @
Sponsored by: Mellanox Technologies // NVIDIA Networking

MFC r365001:
mlx5en: Implement SIOCGIFDOWNREASON.

MFC r362953:
Infiniband clients must be attached and detached in a specific order in ibcore.

Currently the linking order of the infiniband, IB, modules decide in which
order the clients are attached and detached. For example one IB client may
use resources from another IB client. This can lead to a potential deadlock
at shutdown. For example if the ipoib is unregistered after the ib_multicast
client is detached, then if ipoib is using multicast addresses a deadlock may
happen, because ib_multicast will wait for all its resources to be freed before
returning from the remove method.

Fix this by using module_xxx_order() instead of module_xxx().

Differential Revision: https://reviews.freebsd.org/D23973
Sponsored by: Mellanox Technologies

MFC r362201:

Avoid trying to toggle TSO twice

Remove TSO from the toggle mask when automatically disabled by TXCKSUM* in
various NIC drivers.

Reviewed by: hselasky, np, gallatin, jpaetzel
Approved by: mav (mentor)
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25120

MFC r362045:
Make sure packets generated by raw IP code is let through by mlx5en(4).

Allow the TCP header to reside in the mbuf following the IP header.
Else such packets will get dropped.

Backtrace:
mlx5e_sq_xmit()
mlx5e_xmit()
ether_output_frame()
ether_output()
ip_output_send()
ip_output()
rip_output()
sosend_generic()
sosend()
kern_sendit()
sendit()
sys_sendto()
amd64_syscall()
fast_syscall_common()

Sponsored by: Mellanox Technologies

MFC r362044:
Extend use of unlikely() in the fast path, in mlx5en(4).

Typically the TCP/IP headers fit within the first mbuf and should not
trigger any of the error cases. Use unlikely() for these cases.

No functional change.

Sponsored by: Mellanox Technologies

MFC r362043:
Use const keyword when parsing the TCP/IP header in the fast path in mlx5en(4).

When parsing the TCP/IP header in the fast path, make it clear by using
the const keyword, no fields are to be modified inside the transmitted
packet.

No functional change.

Sponsored by: Mellanox Technologies

MFC r360377:
Add support for reading temperature in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r359724:
Account out of buffer as dropped packets in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r359723:
Remove obsolete bufring stats in mlx5en(4).

Leftover from when DRBR was removed.

Sponsored by: Mellanox Technologies

MFC r359653:
Count number of times transmit ring is out of buffers in mlx5en(4).

Differential Revision: https://reviews.freebsd.org/D24273
Sponsored by: Mellanox Technologies

MFC r359655:
Don't drop packets having too many TCP option headers in mlx5en(4).

When using SACK it can happen there are multiple option headers.
Don't drop these packets, but instead limit the amount of inlining
to the maximum supported.

Sponsored by: Mellanox Technologies

MFC r359654:
Ensure a minimum inline size of 16 bytes in mlx5en(4).

This includes 14 bytes of ethernet header and 2 bytes of VLAN header.

This allows for making assumptions about the inline size limit
in the fast transmit path later on.

Use a signed integer variable to catch underflow.

Sponsored by: Mellanox Technologies

MFC r359101:
mlx5: Integrate eswitch and mpfs management code.

MFC r355422:
mlx5: Do not poke hardware for statistic after teardown is started.

MFC r353321:
Fix regression issue after r352989:

As noted by the commit message, callouts are now persistant
and should not be in the auto-zero section of the RQ's and SQ's.
This fixes an assert when using the TX completion event
factor feature with mlx5en(4).

Found by: gallatin@
Sponsored by: Mellanox Technologies

MFC r352998:
Bump driver version for mlx5core, mlx5en(4) and mlx5ib(4).

Sponsored by: Mellanox Technologies

MFC r352996:
Add print to show user a reason for rejecting buffer size change in mlx5en(4).

Submitted by: kib@
Sponsored by: Mellanox Technologies

MFC r352995:
Only update lossy buffers config when manual PFC configuration was done
in mlx5en(4).

Submitted by: kib@
Sponsored by: Mellanox Technologies

MFC r352989:
Seal transmit path with regards to using destroyed mutex in mlx5en(4).

It may happen during link down that the running state may be observed
non-zero in the transmit routine, right before the running state is
cleared. This may end up using a destroyed mutex.

Make all channel mutexes and callouts persistant.

Preserve receive and send queue statistics during link toggle.

Sponsored by: Mellanox Technologies

MFC r352988:
Remove unused cpu field from channel structure in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r352987:
Remove mkey_be from channel structure in mlx5en(4).
Use value from priv structure instead.
This saves some space in the channel structure.

Sponsored by: Mellanox Technologies

MFC r352985:
Add sysctl(8) to get and set forward error correction, FEC, configuration
in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r352983 and r353001:
Move EEPROM information query from a sysctl in mlx5en(4) to an ioctl
in mlx5core. The EEPROM information is not only a property of the
mlx5en(4) driver.

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r352982:
Add support for buffer parameter manipulations in mlx5en(4).

The following sysctls are added:
dev.mce.N.conf.qos.cable_length
dev.mce.N.conf.qos.buffers_size
dev.mce.N.conf.qos.buffers_prio

Submitted by: kib@
Sponsored by: Mellanox Technologies

MFC r352981:
Import Linux code to query/set buffer state in mlx5en(4).

Submitted by: kib@
Sponsored by: Mellanox Technologies

MFC r352980:
Add mlx5e_dbg() compatibility macro.

Submitted by: kib@
Sponsored by: Mellanox Technologies

MFC r352976:
Unify prints in mlx5en(4).

All prints in mlx5en(4) should use on of the macros:
mlx5_en_err/dbg/warn

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r352966:
Add port module event software counters in mlx5core.
While at it, fixup PME based on latest PRM defines.

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r352965:
Correct and update some counter names in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r352964:
Export channel IRQ number as part of the "hw_ctx_debug" sysctl(8) in mlx5en(4).

Sponsored by: Mellanox Technologies

Fix compilation after MFC r352962.

This is a direct commit.

Sponsored by: Mellanox Technologies

MFC r352962:
Add support for Multi-Physical Function Switch, MPFS, in mlx5en.

MPFS is a logical switch in the Mellanox device which forward packets
based on a hardware driven L2 address table, to one or more physical-
or virtual- functions. The physical- or virtual- function is required
to tell the MPFS by using the MPFS firmware commands, which unicast
MAC addresses it is requesting from the physical port's traffic.
Broadcast and multicast traffic however, is copied to all listening
physical- and virtual- functions and does not need a rule in the MPFS
switching table.

Linux commit: eeb66cdb682678bfd1f02a4547e3649b38ffea7e
Sponsored by: Mellanox Technologies

MFC r352961:
Implement macro for asserting priv lock in mlx5en.

Sponsored by: Mellanox Technologies

MFC r352959:
Check return value of mlx5_vector2eqn() function in mlx5en.

Sponsored by: Mellanox Technologies

MFC r348603:
Make sure the DMA tags get freed in mlx5en(4).

Approved by: re (gjb)
Sponsored by: Mellanox Technologies

MFC r347325:
Bump the Mellanox driver version numbers and the FreeBSD version number.

Sponsored by: Mellanox Technologies

MFC r347322:
Ensure the flowtable rules are not freed twice in mlx5en(4).
This can happen when re-loading the driver.

Sponsored by: Mellanox Technologies

MFC r347321:
Undo previous steps upon returning failure in mlx5en(4).

Else flowtable resources may not be properly freed.

Sponsored by: Mellanox Technologies

MFC r347320:
Make sure the flow destination structure does not use values off the stack
in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r347317:
Remove non-functional MLX5E_MAX_RX_SEGS macro in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r347316:
Fix for compilation warning in mlx5en(4).

Function 'mlx5e_alloc_rx_wqe' can never be inlined because it uses alloca
(override using the always_inline attribute)

Sponsored by: Mellanox Technologies

MFC r347310:
Expose per-lane counters before correction mechanism in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r347309:
Add support for extended PCIe counters in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r347308:
Extend the counters framework in mlx5en(4).

Allow more macro arguments and split the variable type and name into
separate arguments. This allows simple and powerful copy and extraction
of values from IFC based structures into SYSCTLs with the use of a single
macro.

Sponsored by: Mellanox Technologies

MFC r347305:
Move workqueue from mlx5en(4) to mlx5core.

This avoids creating more workqueues in mlx5core to do
simple firmware command polling tasks.

Sponsored by: Mellanox Technologies

MFC r347300:
Do not add IFM_10G_LR and IFM_40G_ER4 to supported media types by default in
mlx5en(4).

IFM_10G_LR and IFM_40G_ER4 media should be added only if the device
has the needed capability bit set for it.

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r347299:
Add support for 200Gb ethernet speeds to mlx5core.

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r347295:
Let rx_out_of_buffer be a 32-bit counter in mlx5en(4).

This fixes counting issues when the firmware resets the counter during
allocation of the counter set where the counter belongs.

Sponsored by: Mellanox Technologies

MFC r347294:
Add vnic steering drop statistics in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r347293:
Use software counters for rx_packets and rx_bytes in mlx5en(4).

The physical- and virtual- port counters might not reflect the amount
of data received after address filtering. Use the software counters
instead for rx_packets and rx_bytes to know exactly how much data
was received.

Sponsored by: Mellanox Technologies

MFC r347279:
Fix netstat counters mapping in mlx5en(4).

The current mapping of driver counters to netstat counters is wrong.
For example, a single jabber packet, will cause the Ierrs counter to
count three times.

The work for mapping the hardware and software counters to their right
place in netstat counters were already done in Linux, take that as is
to the FreeBSD driver.

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r347277:
Avoid leaking send queue mbufs during error recovery in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r347264:
Configure firmware to use RX hash format in mini CQE in mlx5en(4).

When using CQE zipping, one can choose between RX hash and Checksum.
This will indicate the parameter on which a zipping session should be
stopped.

While porting the Linux code, Checksum was chosen. However, the value
of Checksum is not being used anywhere.
For the FreeBSD driver, we prefer to use the RX hash format which will
guarantee the RX hash value for all the mini CQEs.
While at it, make sure to initialize the Checksum value in the
decompressed CQE.

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r347263:
Disable CQE zipping by default in mlx5en(4).

After doing performance measurements, it seems like CQE zipping doesn't
have any significant benefit.
Moreover, we know that this feature is disabled by default on other
operating systems (Linux for example).

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r347262:
Split mlx5e_update_stats_work() in mlx5en(4).

Split the function into the mlx5e_update_stats_locked() core and make
mlx5e_update_stats_work() call the _locked helper, similar to many other
places in the kernel. This improves the code structure, making the
locking clean.

Submitted by: kib@
Sponsored by: Mellanox Technologies

MFC r347261:
Implement fast close of RX channel in mlx5en(4).

Instead of waiting for all jobs to be cancelled, simply close the completion
queue to prevent more completion events and let mlx5e_destroy_rq() cleanup
the remaining mbufs.

Sponsored by: Mellanox Technologies

MFC r347260 and r347326:
Correct number of elements for priority to traffic class mappings in mlx5en(4).

The number of priorities is always 8, while the number of traffic classes
supported can vary. While at it convert the sysctl node into an array.

Sponsored by: Mellanox Technologies

MFC r347256:
Destroy port stats debug context in correct order in mlx5en(4).
Destroy children nodes before parent nodes.

Sponsored by: Mellanox Technologies

MFC r347255:
Fix tx_jumbo_packets counter in mlx5en(4).

Instead of reading Ethernet RFC 2819 pXtoYoctets counters from
hardware which counts RX octets, count tx_stat_pXtoYoctets from
Ethernet extended counters which counts TX octets.

TX jumbo counters should be accumulated only after the PPCNT
counters were fetched from hardware with their latest value.

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r347254:
Update Ethernet extended counters in mlx5en(4).

Expose all Ethernet extended counters those counters via debug_stats
sysctl:
dev.mce.X.debug_stats

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r347246:
Add support for Dynamic Interrupt Moderation, DIM, in mlx5en(4).

Add support for DIM based on Linux,
with some minor adaptions specific to FreeBSD.

Linux commit
f97c3dc3c0e8d23a5c4357d182afeef4c67f5c33

Sponsored by: Mellanox Technologies

MFC r341587:
mlx4/mlx5: Updated driver version to 3.5.0

Sponsored by: Mellanox Technologies

MFC r341585:
mlx5en: Improve configuration of HW LRO.

In order to enable HW LRO, both the "hw_lro" sysctl in the mlx5en(4) config
space must be set, and the ifconfig(8) LRO capability must be set. Any other
settings will disable HW LRO.

Sponsored by: Mellanox Technologies

MFC r341584:
mlx5en: Count all transmitted and received bytes.

Add counter for all transmitted and received bytes. Currently only all
transmitted and received packets were counted. Fix description of RX LRO
counters while at it.

Sponsored by: Mellanox Technologies

MFC r341583:
mlx5en: Statically allocate and free the channel structure(s).

By allocating the worst case size channel structure array
at attach time we can eliminate various NULL checks in the
fast path. And also reduce the chance for use-after-free
issues in the transmit fast path.

This change is also a requirement for implementing
backpressure support.

Sponsored by: Mellanox Technologies

MFC r341582:
mlx5en: Fix race in mlx5e_ethtool_debug_stats().

Writing to the debug stats variable must be locked,
else serialization will be lost which might cause
various kernel panics due to creating and destroying
sysctls out of order.

Make sure the sysctl context is initialized after freeing
the sysctl nodes, else they can be freed twice.

Sponsored by: Mellanox Technologies

MFC r341581:
mlx5en: Add support for IFM_10G_LR and IFM_40G_ER4 media types.

Inspect the ethernet compliance code to figure out actual cable type by reading
the PDDR module info register.

Sponsored by: Mellanox Technologies

MFC r341579:
mlx5en: Fix for inlining issues in transmit path

1) Don't exceed the drivers own hardcoded TX inline limit.

The blueflame register size can be much greater than the hardcoded limit
for inlining. Make sure we don't exceed the drivers own limit, because this
also means that the maximum number of TX fragments becomes invalid and
then memory size assumptions in the TX path no longer hold up.

2) Make sure the mlx5_query_min_inline() function returns an error code.

3) Header inlining is required when using TSO.

4) Catch failure to compute inline header size for TSO.

5) Add support for UDP when computing inline header size.

6) Fix for inlining issues with regards to DSCP.

Make sure we inline 4 bytes beyond the ethernet and/or
VLAN header to workaround a hardware bug extracting
the DSCP field from the IPv4/v6 header.

Sponsored by: Mellanox Technologies

MFC r341578 and r341655:
mlx5en: Remove the DRBR and associated logic in the transmit path.

The hardware queues are deep enough currently and using the DRBR and associated
callbacks only leads to more task switching in the TX path. The is also a race
setting the queue_state which can lead to hung TX rings.

Sponsored by: Mellanox Technologies

MFC r341577:
mlx5en: Implement support for bandwidth limiting in by ratio, ETS.

Add support for setting the bandwidth limit as a ratio rather than in bits per
second. The ratio must be an integer number between 1 and 100 inclusivly.

Implement the needed firmware commands and SYSCTLs through mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r341567:
mlx5: Fix driver version location

Driver description should be set by core and not by the Ethernet driver.

Sponsored by: Mellanox Technologies

MFC r341564:
mlx5: Add software tx_jumbo_packets counter

This counter will represent transmitted packets which has more than
1518 octets.
The NIC has multiple hardware counters for counting transmitted
packets larger than 1518 octets. Each counter counts the packets
in specific range.
We accumulate those counters to have a single counter.

Sponsored by: Mellanox Technologies

MFC r338492:
Add support for receive side scaling stride, RSSS, in mlx5en(4).

The receive side scaling stride parameter is a value which define the interval
between active receive side queues. The traffic for the inactive queues is
redirected to the nearest active queue by use of modulus. The default value
of this parameter is one, which means all receive side queues are used.

The point of this feature is to redirect more traffic to fewer receive side
queues in order to take more advantage of sorted large receive offload,
sorted LRO. The sorted LRO works better when more packets are accumulated
per service interval.

Sponsored by: Mellanox Technologies

MFC r338490:
Don't stall transmit queue on drops in mlx5en(4).

When a transmitted packet is dropped don't stall the transmit queue.

Sponsored by: Mellanox Technologies

MFC r338489:
Maximum number of mbuf frags is off-by-one for worst case scenario in mlx5en(4).

Inspecting the PRM no more than 0x3F data segments, DS, of size 16 bytes is
allowed.

Worst case scenario summary of DS usage:
Header is fixed: 2 DS
Maximum inlining: 98 => (98 - 2) / 16 = 6 DS
Remainder: 0x3F - 2 - 6 = 55 DS (mbuf frags)

Previously a value of 56 DS was used and this would work in the
normal case because not all inline data area was used up.

Sponsored by: Mellanox Technologies

MFC r336451:
Update version information for the mlx5 and mlx5en(4) modules.

While at it bump some copyright dates.

Sponsored by: Mellanox Technologies

MFC r336450:
Do not inline transmit headers and use HW VLAN tagging if supported by mlx5en(4).

Query the minimal inline mode supported by the card.
When creating a send queue, cache the queried mode and optimize the transmit
if no inlining is required. In this case, we can avoid touching the headers
cache line and avoid dirtying several more lines by copying headers into
the send WQEs. Also, if no inline headers are used, hardware assists in
the VLAN tag framing.

Submitted by: kib@, slavash@
Sponsored by: Mellanox Technologies

MFC r336411:
Use a mbuf header instead of a mbuf cluster for debugging interrupts in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r336407:
Handle jumbo frames without requiring big clusters in mlx5en(4).

The scatter list is formed by the chunks of MCLBYTES each, and larger
than default packets are returned to the stack as the mbuf chain.

Submitted by: kib@
Sponsored by: Mellanox Technologies

MFC r336404:
Enable both receive and transmit pauseframes by default in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r336403:
Add context numbers for HW elements in mlx5en(4).

To access the data, set sysctl dev.mce.N.conf.debug_stats to 1.
This enables the sysctl node dev.mce.N.hw_ctx_debug. Its content is
the mapping of each channel' number to used receive queue and associated
completion queue, set of the transmit queues numbers and corresponding
completion queues.

Trimmed example output:
channel 30 rq 188 cq 1085
channel 30 tc 0 sq 187 cq 1084
channel 31 rq 191 cq 1087
channel 31 tc 0 sq 190 cq 1086

Sponsored by: Mellanox Technologies

MFC r336402:
Do not hint about 'trust both' mode when the mlx5en(4) hardware does not support it.

Sponsored by: Mellanox Technologies

MFC r336401:
Correctly write atomic variable in mlx5en(4).

Sponsored by: Mellanox Technologies

MFC r336392:
Implement support for Differentiated Service Code Point, DSCP, in mlx5en(4).

The DSCP feature is controlled using a set of sysctl(8) fields under
the qos sysctl directory entry for mlx5en(4).

For Routable RoCE QPs, the DSCP should be set in the QP's address path.
The DSCP's value is derived from the traffic class.

Linux commit:
ed88451e1f2d400fd6a743d0a481631cf9f97550

Sponsored by: Mellanox Technologies

MFC r333182:
mlx5en: Always allow VLAN id 0.

Sponsored by: Mellanox Technologies
Approved by: re (marius)

MFC r332003:
Bump driver version number in mlx5en(4).

Approved by: hselasky (mentor), kib (mentor)
Sponsored by: Mellanox Technologies

MFC r331797:

Use an accessor function to access ifr_data.

This fixes 32-bit compat (no ioctl command defintions are required
as struct ifreq is the same size).

Reviewed by: kib
Obtained from: CheriBSD
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D14900

MFC r331456:
Don't wait for completions when a mlx5en(4) device is in internal
error state.

If the device is in internal error state the hardware will not
generate completions. Just move on to destroy the resources.

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r330647:
Use the autogenerated interface file for all commands in mlx5core.

This patch accumulates the following Linux commits:
- 90b3e38d048f09b22fb50bcd460cea65fd00b2d7
mlx5_core: Modify CQ moderation parameters
- 09a7d9eca1a6cf5eb4f9abfdf8914db9dbd96f08
mlx5_core: QP/XRCD commands via mlx5 ifc
- 1a412fb1caa2c1b77719ccb5ed8b0c3c2bc65da7
mlx5_core: Modify QP commands via mlx5 ifc
- ec22eb53106be1472ba6573dc900943f52f8fd1e
mlx5_core: MKey/PSV commands via mlx5 ifc
- 73b626c182dff06867ceba996a819e8372c9b2ce
mlx5_core: EQ commands via mlx5 ifc
- 20ed51c643b6296789a48adc3bc2cc875a1612cf
mlx5_core: Access register and MAD IFC commands via mlx5 ifc
- a533ed5e179cd15512d40282617909d3482a771c
mlx5_core: Pages management commands via mlx5 ifc
- b8a4ddb2e8f44f872fb93bbda2d541b27079fd2b
mlx5_core: Add MLX5_ARRAY_SET64 to fix BUILD_BUG_ON
- af1ba291c5e498973cc325c501dd8da80b234571
mlx5_core: Refactor internal SRQ API
- b06e7de8a9d8d1d540ec122bbdf2face2a211634
mlx5_core: Refactor device capability function
- c4f287c4a6ac489c18afc4acc4353141a8c53070
mlx5_core: Unify and improve command interface

Submitted by: Matthew Finlay <matt@mellanox.com>
Sponsored by: Mellanox Technologies

MFC r331437:
Create designated workqueue for each mlx5en(4) device instance.

The mlx5e_destroy_ifp() function may be called from the system workqueue and
in this case trying to flush all works will cause a dead lock.
Instead of using the system workqueue, create a designated workqueue
for each mlx5en(4) device instance.

Submitted by: slavash@
Sponsored by: Mellanox Technologies

MFC r330659:
Avoid more LFENCE/SFENCe on x86 in mlx5en(4),
by using the FreeBSD native fences.

Submitted by: kib@
Sponsored by: Mellanox Technologies

MFC r330658:
Fix mlx5en(4) driver to properly call m_defrag().

When the mlx5en(4) driver was converted to using BUSDMA(9) the call to
m_defrag() was moved after the part of the TX routine that strips the
header from the mbuf chain. Before it called m_defrag it first trimmed
off the now-empty mbufs from the start of the chain. This has the side
effect of also removing the head of the chain that has M_PKTHDR set.
m_defrag() will not defrag a chain that does not have M_PKTHDR set,
thus it was effectively never defragging the mbuf chains.

As it turns out, trimming the mbufs in this fashion is unnecessary since
the call to bus_dmamap_load_mbuf_sg doesn't map empty mbufs anyway, so
remove it.

Differential Revision: https://reviews.freebsd.org/D12050
Submitted by: mjoras@
Sponsored by: Mellanox Technologies

MFC r330657:
Use vport rather than physical-port MTU in mlx5en(4).

Set and report vport MTU rather than physical MTU,
The driver will set both vport and physical port mtu
and will rely on the query of vport mtu.

SRIOV VFs have to report their MTU to their vport manager (PF),
and this will allow them to work with any MTU they need
without failing the request.

Also for some cases where the PF is not a port owner, PF can
work with MTU less than the physical port mtu if set physical
port mtu didn't take effect.

Based on Linux upstream commit:
cd255efff9baadd654d6160e52d17ae7c568c9d3

Submitted by: Meny Yossefi <menyy@mellanox.com>
Sponsored by: Mellanox Technologies

MFC r330656:
Use the device unit number for naming the ifnet interface in mlx5en(4).

Currently the ifnet interface is named mceX, where X is a monotonically
incremented value. If the device is reset due to a fatal error, then the
interface name will change. Using the device unit number will keep the
naming consistent across the reset logic.

Submitted by: Matthew Finlay <matt@mellanox.com>
Sponsored by: Mellanox Technologies

MFC r330649:
Add support for per priority flow control, PFC, to mlx5en(4).

Add support for PFC and implement reading the per priority statistics
using the sysctl(8) interface. PFC is used together with VLAN priority
and can be enabled and disabled on a per priority basis.

Global pause frames and PFC are incompatible features and surrounding
logic has been added to warn the user about misconfiguration.

Update relevant mlx5core APIs for PFC configuration.

Sponsored by: Mellanox Technologies

MFC r330608:
Implement priority to traffic class mapping in mlx5core.

Add support for mapping priority to traffic class via sysctl

Submitted by: Slava Shwartsman <slavash@mellanox.com>
Sponsored by: Mellanox Technologies

MFC r330607:
Implement rate limit per traffic class in mlx5core.

Add support for rate limiting traffic class via sysctl.

Submitted by: Slava Shwartsman <slavash@mellanox.com>
Sponsored by: Mellanox Technologies

MFC r330606:
Implement support for querying the current port rate in mlx5core.
The mlx5ib(4) part will be merged separately.

- Factor out port speed definitions into new port.h header file,
similarly as done in Linux upstream.
- Correct two existing port speed definitions in mlx5en according to
Linux upstream.

Sponsored by: Mellanox Technologies

MFC r330604:
Add log message for unsupported QSFPs in mlx5core.

Submitted by: Matthew Finlay <matt@mellanox.com>
Sponsored by: Mellanox Technologies

MFC r325661:
Expose the current hardware MTU in mlx5en(4) as a separate entry
in the sysctl tree.

Sponsored by: Mellanox Technologies

MFC r325660:
Add support for configuring local multicast and unicast data traffic loopback
in mlx5en(4) driver via the sysctl interface.

Sponsored by: Mellanox Technologies

MFC r325659:
Add support for disabling and enabling RX and TX DMA rings in mlx5en(4).
This is useful for supporting setups similar to Netmap.

Sponsored by: Mellanox Technologies

MFC r328591:
Move the mlx5 core device pointer first in the mlx5en priv. This help simplify
checks to recognize own network devices when using mlx5ib. This patch fixes
an issues where mlx5ib fails to recognize mceX network devices for use with
RoCE.

Sponsored by: Mellanox Technologies

MFC r325638 and r325976:
Refactor the flowsteering APIs used by mlx5en(4). This change is needed by
the coming ibcore and mlx5ib updates in order to support traffic redirection
to so-called raw ethernet QPs.

Remove unused E-switch related routines and files while at it.

Sponsored by: Mellanox Technologies

MFC r326977:
mlx5en: Avoid SFENCe on x86.

MFC r324202:
Make sure the doorbell lock is valid for the i386 version
of the mlx5en(4) driver.

Tested by: gallatin @
Sponsored by: Mellanox Technologies

MFC r322250:
Count drop events due to lack of PCI bandwidth as queue drops and not as
input errors in the mlx5en(4) driver. This improves the sysadmin view of
physical port errors.

Submitted by: gallatin@
Sponsored by: Mellanox Technologies

MFC r322251:
Make sure the received IP header gets 32-bit aligned for short packets
in the mlx5en(4) driver.

Sponsored by: Mellanox Technologies

MFC r312872:
Add support for reading advanced diagnostic counters.

By default reading the diagnostic counters is disabled. The firmware
decides which counters are supported and only those supported show up
in the dev.mce.X.diagnostics sysctl tree.

To enable reading of diagnostic counters set one or more of the
following sysctls to one:

dev.mce.X.conf.diag_general_enable=1
dev.mce.X.conf.diag_pci_enable=1

Sponsored by: Mellanox Technologies

MFC r312865:
Enforce reading the consumer and producer counters once to ensure
consistent return values from the mlx5e_sq_has_room_for()
function. The two counters are incremented by different threads under
different locks.

Sponsored by: Mellanox Technologies

MFC r312537:
Remove superfluous return statement.

Sponsored by: Mellanox Technologies

MFC r312536:
Allow transmit packet bufring in software to be disabled.

- Add new sysctl node to control the transmit packet bufring.

- Add optimised version of the transmit routine which output packets
directly to the DMA ring instead of using bufring in case the transmit
lock is congested. This can reduce the number of taskswitches which in
turn influence the overall system CPU usage, depending on the
workload.

- Add " TX" suffix to debug name for transmit mutexes to silence some
witness warnings about aquiring duplicate locks having same name.

Sponsored by: Mellanox Technologies
Suggested by: gallatin @

MFC r312528:
Make draining a sendqueue more robust.

Add own state variable to track if a sendqueue is stopped or not.
This will prevent traffic from entering the sendqueue while it is
being destroyed.

Update drain function to wait for traffic to be transmitted before
returning when the link state is active.

Add extra checks in transmit path for stopped SQ's.

While at it:
- Use likely() for a mbuf pointer check.
- Remove redundant IFF_DRV_RUNNING check.

Sponsored by: Mellanox Technologies

MFC r312527:
Add runtime support for modifying the SQ and RQ completion event
moderation mode. The presence of this feature is indicated through the
firmware capabilities.

Sponsored by: Mellanox Technologies

MFC r317568:

Improve sysadmin visibility of physical port error counters in the
mlx5en driver.

Sponsored by: Mellanox Technologies

MFC r310387:
Add more comments regarding collection of statistics counters.

Sponsored by: Mellanox Technologies

MFC r310388:
Make a read only pointer constant.

Sponsored by: Mellanox Technologies

MFC r309406:
Remove useless NULL checks.

NULL is not returned when allocating memory passing the M_WAITOK flag.

Submitted by: trasz @
Differential Revision: https://reviews.freebsd.org/D5772
Sponsored by: Mellanox Technologies

MFC r308416:
Add timer to watch the RQ when we are out of mbufs.

The firmware/hardware does not generate additional completion
events unless we post new buffers. Use a timer to try to post
more buffers in case we are temporarily out of mbufs. Else
the receive schedule completely stops.

Sponsored by: Mellanox Technologies

MFC r306453:
Set hardware stats flag to avoid double counting the number of incoming bytes.

Found by: Ben RUBSON <ben.rubson@gmail.com>

MFC r305877:
mlx5en: Fix duplicate mbuf free-by-code.

When mlx5e_sq_xmit() returns an error code and the mbuf pointer is set,
we should not free the mbuf, because the caller will keep the mbuf in
the drbr. Make sure the mbuf pointer is correctly set upon function
exit.

Sponsored by: Mellanox Technologies

MFC r305876:
mlx5en: Remove unused pdev pointer.

Sponsored by: Mellanox Technologies

MFC r305875:
mlx5en: Verify port type is ethernet before creating network device

Else the mlx5en driver might attach to infiniband ports.

Sponsored by: Mellanox Technologies

MFC r305874:
mlx5en: Allow setting the software MTU size below 1500 bytes

The hardware MTU size can't be set to a value less than 1500 bytes due
to side-band management support. Allow setting the software MTU size
below 1500 bytes, thus creating a mismatch between hardware and
software MTU sizes.

Sponsored by: Mellanox Technologies

MFC r305873:
mlx5en: Factor out common sendqueue code for use with rate limiting SQs.

Try to reuse code to setup sendqueues when possible by making some static
functions global. Further split the mlx5e_close_sq_wait() function to
separate out reusable parts.

Sponsored by: Mellanox Technologies

MFC r305872:
mlx5en: Properly declare doorbell lock for 32-bit CPUs.

Sponsored by: Mellanox Technologies

MFC r305871:
mlx5en: Optimise away duplicate UAR pointers.

This change also reduces the size of the mlx5e_sq structure so that the last
queue_state element will fit into the previous cacheline and then the mlx5e_sq
structure becomes one cacheline less for amd64.

Sponsored by: Mellanox Technologies

MFC r305870:
mlx5en: Make the mlx5e_open_cq() and mlx5e_close_cq() functions global.

Make some functions and structures global to allow for code reuse
when creating rate limiting sendqueues.

Sponsored by: Mellanox Technologies

MFC r305869:
mlx5en: Minor completion queue control path code refactor.

Move setting of CQ moderation mode together with the other
CQ moderation parameters. Pass completion event vector as
a separate argument to mlx5e_open_cq(), because its value is
different for each call. Pass mlx5e_priv pointer instead of
mlx5e_channel pointer so that code can be used by rate
limiting sendqueues.

Sponsored by: Mellanox Technologies

MFC r305868:
mlx5en: Separate the sendqueue from using the mlx5e_channel structure.

This change allows for reusing the transmit path for so called
rate limited senqueues. While at it optimise some pointer lookups
in the fast path.

Sponsored by: Mellanox Technologies

MFC r305867:
Update the MLX5 core module:
- Add new firmware commands and update existing ones.
- Add more firmware related structures and update existing ones.
- Some minor fixes, like adding missing \n to some prints.

Sponsored by: Mellanox Technologies

MFC r304601:
Increase the maximum RX/TX queue size. This allows for a RX/TX queue
size of 16384 mbufs. Previously the limit was 8192.

Sponsored by: Mellanox Technologies

MFC r303870:
Fix for use after free.

Clear the device description to avoid use after free because the
bsddev is not destroyed when the mlx5en module is unloaded. Only when
the parent mlx5 module is unloaded the bsddev is destroyed. This fixes
a panic on listing sysctls which refer strings in the bsddev after the
mlx5en module has been unloaded.

Sponsored by: Mellanox Technologies

MFC r303837:
Switch to the new block based LRO input function for the mlx5en
driver. This change significantly increases the overall RX aggregation
ratio for heavily loaded networks handling 10-80 thousand simultaneous
connections.

Remove the turbo LRO code and all references to it which has now been
superceeded by the tcp_lro_queue_mbuf() function.

Tested by: Netflix
Sponsored by: Mellanox Technologies

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Add SR-IOV guest support to the mlx5en driver.

This patch adds the missing pieces needed for device setup using the
mlx5en driver inside a virtual machine which is providing hardware
access through SR-IOV.

Sponsored by: Mellanox Technologies
MFC after: 1 week

net: Use M_HASHTYPE_OPAQUE_HASH if the mbuf flowid has hash properties

Reviewed by: hps, erj, tuexen
Sponsored by: Microsoft OSTC
Differential Revision: https://reviews.freebsd.org/D6688

Verify one sysctl parameter at a time. When a mlx5en sysctl parameter
is updated only verify the changed one instead of all.

No functional change.

Sponsored by: Mellanox Technologies
Tested by: Netflix
MFC after: 1 week

Optimise use of doorbell and remove redundant NOPs

Store the last doorbell write in the mlx5e_sq structure and write the
doorbell to the hardware when the transmit routine finishes
transmitting all queued mbufs.

Sponsored by: Mellanox Technologies
Tested by: Netflix
MFC after: 1 week

Implement TX completion event interleaving.

This patch implements a sysctl which allows setting a factor, N, for
how many work queue elements can be generated before requiring a
completion event. When a completion event happens the code simulates N
completion events instead of only one. When draining a transmit queue,
N-1 NOPs are transmitted at most, to force generation of the final
completion event. Further a timer is running every HZ ticks to flush
any remaining data off the transmit queue when the tx_completion_fact
> 1.

The goal of this feature is to reduce the PCI bandwidth needed when
transmitting data.

Sponsored by: Mellanox Technologies
Tested by: Netflix
MFC after: 1 week

Correct some error codes to native FreeBSD ones.

Sponsored by: Mellanox Technologies
Tested by: Netflix
MFC after: 1 week

Add function to detect the presence of a port module and use this
function to error out early when no port module is present and doing
eeprom access. This also prevents error codes from filling up in
dmesg.

Sponsored by: Mellanox Technologies
Tested by: Netflix
MFC after: 1 week

tcp/lro: Use tcp_lro_flush_all in device drivers to avoid code duplication

And factor out tcp_lro_rx_done, which deduplicates the same logic with
netinet/tcp_lro.c

Reviewed by: gallatin (1st version), hps, zbb, np, Dexuan Cui <decui microsoft com>
Sponsored by: Microsoft OSTC
Differential Revision: https://reviews.freebsd.org/D5725

Fix an issue where the network adapter could be left in down state
after changing the HW LRO sysctl when previously in up state.

Reviewed by: gnn
Sponsored by: Mellanox Technologies
MFC after: 5 days
Differential Revision: https://reviews.freebsd.org/D4941

Add clarifying comment about CQE zipping.

Reviewed by: gnn
Sponsored by: Mellanox Technologies
MFC after: 5 days
Differential Revision: https://reviews.freebsd.org/D4940

Declare local variables at top of function.

Reviewed by: gnn
Sponsored by: Mellanox Technologies
MFC after: 5 days
Differential Revision: https://reviews.freebsd.org/D4939

Allow RX and TX pause frames to be set through ifconfig.

Reviewed by: gnn
Sponsored by: Mellanox Technologies
MFC after: 5 days
Differential Revision: https://reviews.freebsd.org/D4817

Add support for modifying coalescing parameters runtime.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Allow I2C to read address 0x51 as well as address 0x50.

MFC after: 1 week
Submitted by: Shahar Klein <shahark@mellanox.com>
Sponsored by: Mellanox Technologies

10G ER/LR should present itself as LR.

MFC after: 1 week
Submitted by: Shahar Klein <shahark@mellanox.com>
Sponsored by: Mellanox Technologies

Add support for CQE zipping. CQE zipping reduces PCI overhead by
coalescing and zipping multiple CQEs into a single merged CQE. The
feature is enabled by default and can be disabled by a sysctl.

Implementing this feature mlx5_cqwq_pop() has been separated from
mlx5e_get_cqe().

MFC after: 1 week
Submitted by: Mark Bloch <markb@mellanox.com>
Differential Revision: https://reviews.freebsd.org/D4598
Sponsored by: Mellanox Technologies

Add support for sysctl tunables to 10-stable and older. Pushed through
head first to simplify driver maintenance.

MFC after: 1 week
Submitted by: Drew Gallatin <gallatin@freebsd.org>
Differential Revision: https://reviews.freebsd.org/D4552
Sponsored by: Mellanox Technologies

Make the eeprom dump function more readable and rename variables for
better clarity.

MFC after: 1 week
Submitted by: Daria Genzel <dariaz@mellanox.com>
Differential Revision: https://reviews.freebsd.org/D4551
Sponsored by: Mellanox Technologies

Add full support for Receive Side Scaling, RSS, to the mlx5en
driver. This includes binding all interrupt and worker threads
according to the RSS configuration, setting up correct Toeplitz
hashing keys as given by RSS and setting the correct mbuf
hashtype for all received traffic.

MFC after: 1 week
Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4410

Add support for setting the TX moderation mode via a sysctl entry. TX
completion events can be moderated in the same way like RX completion
events. Expose this functionality by a sysctl variable.

MFC after: 1 week
Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4409

The firmware no longer supports setting a port MTU of zero bytes.
Set the port MTU and then query it and report if any problems instead.

MFC after: 1 week
Submitted by: Shahar Klein <shahark@mellanox.com>
Sponsored by: Mellanox Technologies
Differential Revision: https://reviews.freebsd.org/D4408

Style changes, mostly automated.

Differential Revision: https://reviews.freebsd.org/D4179
Submitted by: Daria Genzel <dariaz@mellanox.com>
Sponsored by: Mellanox Technologies
MFC after: 3 days

Accumulate out of RX buffers into a 64-bit value and subtract out of
RX buffers from number of received packets.

Differential Revision: https://reviews.freebsd.org/D4178
Submitted by: Drew Gallatin <gallatin@freebsd.org>
Sponsored by: Mellanox Technologies
MFC after: 3 days

Maintain the "hw_lro" configuration variable correctly.

Setting sysctl dev....conf.hw_lro may fail if the net device lro is
turned off. Due to the nature of our sysctl handler we need to set the
values back to 0 and issue an error.

Differential Revision: https://reviews.freebsd.org/D4177
Submitted by: Shahar Klein <shahark@mellanox.com>
Sponsored by: Mellanox Technologies
MFC after: 3 days

Print cable name, if cable type is not recognized.

Differential Revision: https://reviews.freebsd.org/D4180
Submitted by: Mark Bloch <markb@mellanox.com>
Sponsored by: Mellanox Technologies
MFC after: 3 days

Add mlx5 and mlx5en driver(s) for ConnectX-4 and ConnectX-4LX cards
from Mellanox Technologies. The current driver supports ethernet
speeds up to and including 100 GBit/s. Infiniband support will be
done later.

The code added is not compiled by default, which will be done by a
separate commit.

Sponsored by: Mellanox Technologies
MFC after: 2 weeks