mlx5ib: adjust for the mlx5_create_auto_grouped_flow_table() interface change

Sponsored by: NVidia networking
MFC after: 1 week

mlx5: Use software enum in APIs instead of PRM

Users of the steering APIs shouldn't use the PRM directly.
Create an software enum to be used instead.

Signed-off-by: Mark Bloch <mbloch@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

mlx5: Refactor flow actions into a struct

Create a struct to hold flow actions to be used when creating
a flow rule.

Signed-off-by: Mark Bloch <mbloch@nvidia.com>
Sponsored by: NVidia networking
MFC after: 1 week

mlx5ib: Fix ethertype to be ETH_P_IPV6

Sponsored by: Juniper Networks, Inc.
MFC after: 7 days
Reviewed by: ae, kib, zlei
Differential Revision: https://reviews.freebsd.org/D42184

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

mlx5: Make mlx5_del_flow_rule() NULL safe.

This change factors out repeated NULL checks.

No functional change intended.

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5ib(4): Mark driver knows net epoch

This driver has already been EPOCH(9) aware since e48813009c6a.

Reviewed by: hselasky
Tested by: hselasky
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D39406

mlx5: Enter network epoch when using if_foreach()

Summary: IFNET_RLOCK() is not sufficient, the epoch needs entered.

Reviewed by: hselasky
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D38711

Mechanically convert mlx5en(4) to IfAPI

Reviewed by: zlei
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D38595

mlx5ib: Add support for NDR link speed.

The IBTA specification has new speed - NDR. That speed supports signaling
rate of 100Gb. mlx5 IB driver translates link modes reported by ConnectX
device to IB speed and width. Added translation of new 100Gb, 200Gb and
400Gb link modes to NDR IB type and width of x1, x2 or x4 respectively.

Linux commits:
f946e45f59ef01ff54ffb3b1eba3a8e7915e7326

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5ib: Add support for parsing udata in mlx5_ib_create_flow().

Backport from Linux 5.17 (drivers/infiniband/hw/mlx5/fs.c)

This fixes creating flow rules from user-space after the
kernel space update based on Linux 5.7-rc1 .

Sponsored by: NVIDIA Networking

mlx5: Add raw ethernet local loopback support.

Currently, unicast/multicast loopback raw ethernet (non-RDMA) packets
are sent back to the vport. A unicast loopback packet is the packet
with destination MAC address the same as the source MAC address. For
multicast, the destination MAC address is in the vport's multicast
filter list.

Moreover, the local loopback is not needed if there is one or none
user space context.

After this patch, the raw ethernet unicast and multicast local
loopback are disabled by default. When there is more than one user
space context, the local loopback is enabled.

Note that when local loopback is disabled, raw ethernet packets are
not looped back to the vport and are forwarded to the next routing
level (eswitch, or multihost switch, or out to the wire depending on
the configuration).

Linux commits:
c85023e153e3824661d07307138fdeff41f6d86a
8978cc921fc7fad3f4d6f91f1da01352aeeeff25

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5ib: idiomatic use of preprocessor, in particular paths

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5ib: normalize use of the opt_*.h files

MFC after: 1 week
Sponsored by: NVIDIA Networking

mlx5/mlx4: Bump driver version to 3.7

While at it only output driver version to dmesg(8) when hardware is present.

Differential Revision: https://reviews.freebsd.org/D29100
MFC after: 1 week
Reviewed by: kib and markj
Sponsored by: NVIDIA Networking

ibcore: Kernel space update based on Linux 5.7-rc1.

Overview:

This is the first stage of a RDMA stack upgrade introducing kernel
changes only based on Linux 5.7-rc1.

This patch is based on about four main areas of work:
- Update of the IB uobjects system:
- The memory holding so-called AH, CQ, PD, SRQ and UCONTEXT objects
is now managed by ibcore. This also require some changes in the
kernel verbs API. The updated verbs changes are typically about
initialize and deinitialize objects, and remove allocation and
free of memory.

- Update of the uverbs IOCTL framework:
- The parsing and handling of user-space commands has been
completely refactored to integrate with the updated IB uobjects
system.

- Various changes and updates to the generic uverbs interfaces in
device drivers including the new uAPI surface.

- The mlx5_ib_devx.c in mlx5ib and related mlx5 core changes.

Dependencies:

- The mlx4ib driver code has been updated with the minimum changes
needed.

- The mlx5ib driver code has been updated with the minimum changes
needed including DV support.

Compatibility:

- All user-space facing APIs are backwards compatible after this
change.

- All kernel-space facing RDMA APIs are backwards compatible after
this change, with exception of ib_create_ah() and ib_destroy_ah()
which takes a new flag.

- The "ib_device_ops" structure exist, but only contains the driver ID
and some structure sizes.

Differences from Linux:

- Infiniband drivers must use the INIT_IB_DEVICE_OPS() macro to set
the sizes needed for allocating various IB objects, when adding
IB device instances.

Security:

- PRIV_NET_RAW is needed to use raw ethernet transmit features.
- PRIV_DRIVER is needed to use other privileged operations.

Based on upstream Linux, Torvalds (5.7-rc1):
8632e9b5645bbc2331d21d892b0d6961c1a08429

MFC after: 1 week
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D31149
Sponsored by: NVIDIA Networking

ibcore: Introduce ib_port_phys_state enum.

In order to improve readability, add ib_port_phys_state enum to replace
the use of magic numbers.

Linux commit:
72a7720fca37fec0daf295923f17ac5d88a613e1

MFC after: 1 week
Reviewed by: kib
Sponsored by: Mellanox Technologies // NVIDIA Networking

mlx5ib: Support RAW Ethernet when RoCE is disabled in mlx5ib(4)

On some environments, such as certain SRIOV VF configurations, RoCE is
not supported for mlx5 Ethernet ports. Currently, the driver will not
open IB device on that port.

This is problematic, since we do want user-space RAW Ethernet (RAW_PACKET
QPs) functionality to remain in place. For that end, enhance the relevant
driver flows such that we do create a device instance in that case.

Linux commit:
ca5b91d63192ceaa41a6145f8c923debb64c71fa

MFC after: 1 week
Reviewed by: kib
Sponsored by: Mellanox Technologies // NVIDIA Networking

mlx4/OFED: replace the struct net_device with struct ifnet

Given all the code does operate on struct ifnet, the last step in this
longer series of changes now is to rename struct net_device to
struct ifnet (that is what it was defined to in the LinuxKPi code).
While mlx4 and OFED are "shared" code the decision was made years ago
to not write it based on the netdevice KPI but the native ifnet KPI
for most of it. This commit simply spells this out and with that
frees "struct netdevice" to be re-done on LinuxKPI to become a more
native/mixed implementation over time as needed by, e.g., wireless
drivers.

Sponsored by: The FreeBSD Foundation
MFC after: 10 days
Reviewed by: hselasky
Differential Revision: https://reviews.freebsd.org/D30515

mlx5: remove dependency on ifnet specifics of linux/netdevice.h

Rename the last remaining bits depending on ifnet from linux/netdevice.h
instead of using the compat macros. This helps clearing up
struct netdevice being struct ifnet from linux/netdevice.h.

Sponsored-by: The FreeBSD Foundation
MFC-after: 2 weeks
Reviewed-by: hselasky, kib
X-D-R: D29366
Differential Revision: https://reviews.freebsd.org/D29497

Add missing init of new fields after new UAR API was introduced
by f8f5b459d21e in mlx5ib.

MFC after: 1 week
Sponsored by: Mellanox Technologies // NVIDIA Networking

Update user access region, UAR, APIs in the core in mlx5core.

This change include several changes as listed below all related to UAR.
UAR is a special PCI memory area where the so-called doorbell register and
blue flame register live. Blue flame is a feature for sending small packets
more efficiently via a PCI memory page, instead of using PCI DMA.

- All structures and functions named xxx_uuars were renamed into xxx_bfreg.
- Remove partially implemented Blueflame support from mlx5en(4) and mlx5ib.
- Implement blue flame register allocator.
- Use blue flame register allocator in mlx5ib.
- A common UAR page is now allocated by the core to support doorbell register
writes for all of mlx5en and mlx5ib, instead of allocating one UAR per
sendqueue.
- Add support for DEVX query UAR.
- Add support for 4K UAR for libmlx5.

Linux commits:
7c043e908a74ae0a935037cdd984d0cb89b2b970
2f5ff26478adaff5ed9b7ad4079d6a710b5f27e7
0b80c14f009758cefeed0edff4f9141957964211
30aa60b3bd12bd79b5324b7b595bd3446ab24b52
5fe9dec0d045437e48f112b8fa705197bd7bc3c0
0118717583cda6f4f36092853ad0345e8150b286
a6d51b68611e98f05042ada662aed5dbe3279c1e

MFC after: 1 week
Sponsored by: Mellanox Technologies // NVIDIA Networking

Bump driver versions for mlx5en(4) and mlx4en(4).

MFC after: 1 week
Sponsored by: Mellanox Technologies // NVIDIA Networking

Report EQE data upon CQ completion in mlx5core.

Report EQE data upon CQ completion to let upper layers use this data.

Linux commit:
4e0e2ea1886afe8c001971ff767f6670312a9b04

MFC after: 1 week
Sponsored by: Mellanox Technologies // NVIDIA Networking

Infiniband clients must be attached and detached in a specific order in ibcore.

Currently the linking order of the infiniband, IB, modules decide in which
order the clients are attached and detached. For example one IB client may
use resources from another IB client. This can lead to a potential deadlock
at shutdown. For example if the ipoib is unregistered after the ib_multicast
client is detached, then if ipoib is using multicast addresses a deadlock may
happen, because ib_multicast will wait for all its resources to be freed before
returning from the remove method.

Fix this by using module_xxx_order() instead of module_xxx().

Differential Revision: https://reviews.freebsd.org/D23973
MFC after: 1 week
Sponsored by: Mellanox Technologies

mlx5en: Support 50GBase-KR4 media type in mlx5en driver.

Submitted by: Adam Peace <adam.e.peace@gmail.com>
Reviewed by: hselasky
Sponsored by: Mellanox Technologies
MFC after: 1 week

Bump driver version for mlx5core, mlx5en(4) and mlx5ib(4).

MFC after: 3 days
Sponsored by: Mellanox Technologies

Fix reported max SGE calculation in mlx5ib.

Add the 512 bytes limit of RDMA READ and the size of remote address to the max
SGE calculation.

Submitted by: slavash@
Linux commit: 288c01b746aa
MFC after: 3 days
Sponsored by: Mellanox Technologies

Bump the Mellanox driver version numbers and the FreeBSD version number.

MFC after: 3 days
Sponsored by: Mellanox Technologies

Always return success for RoCE modify port in mlx5ib.

CM layer calls ib_modify_port() regardless of the link layer.

For the Ethernet ports, qkey violation and Port capabilities
are meaningless. Therefore, always return success for ib_modify_port
calls on the Ethernet ports.

Linux Commit:
ec2558796d25e6024071b6bcb8e11392538d57bf

Submitted by: slavash@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Add support for new rates to mlx5ib.

Submitted by: slavash@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Add support for 200Gb ethernet speeds to mlx5core.

Submitted by: slavash@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Remove unused module parameter in mlx5ib.

MFC after: 3 days
Sponsored by: Mellanox Technologies

Import Linux code to implement mlx5_ib_disassociate_ucontext() in mlx5ib.

Submitted by: kib@
MFC after: 3 days
Sponsored by: Mellanox Technologies

mlx4/mlx5: Updated driver version to 3.5.0

Approved by: hselasky (mentor)
MFC after: 1 week
Sponsored by: Mellanox Technologies

mlx5ib: Set default active width and speed when querying port.

Make sure the active width and speed is set in case the
translate_eth_proto_oper() function doesn't recognize the
current port operation mask.

Linux commit:
7672ed33c4c15dbe9d56880683baaba4227cf940

Submitted by: hselasky@
Approved by: hselasky (mentor)
MFC after: 1 week
Sponsored by: Mellanox Technologies

mlx5ib: Fix sign extension in mlx5_ib_query_device

"fw_rev_min(dev->mdev)" with type "unsigned short" (16 bits, unsigned) is
promoted in "fw_rev_min(dev->mdev) << 16" to type "int" (32 bits, signed), then
sign-extended to type "unsigned long" (64 bits, unsigned). If
"fw_rev_min(dev->mdev) << 16" is greater than 0x7FFFFFFF, the upper bits of the
result will all be 1.

Approved by: hselasky (mentor)
MFC after: 1 week
Sponsored by: Mellanox Technologies

mlx5: Fix driver version location

Driver description should be set by core and not by the Ethernet driver.

Approved by: hselasky (mentor)
MFC after: 1 week
Sponsored by: Mellanox Technologies

mlx5: Raise fatal IB event when sys error occurs

All other mlx5_events report the port number as 1 based, which is how FW
reports it in the port event EQE. Reporting 0 for this event causes
mlx5_ib to not raise a fatal event notification to registered clients
due to a seemingly invalid port.

All switch cases in mlx5_ib_event that go through the port check are
supposed to set the port now, so just do it once at variable
declaration.

Linux commit:
aba462134634b502d720e15b23154f21cfa277e5

Approved by: hselasky (mentor)
MFC after: 1 week
Sponsored by: Mellanox Technologies

Update version information for the mlx5ib module.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Don't pass unsupported events to ibcore from mlx5ib.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Use static device naming instead of dynamic one in mlx5ib.

When resetting mlx5core instances it can happen that the order of attach and
detach for mlx5ib instances is changed. Take the unit number for mlx5_%d from
the parent PCI device, similarly to what is done in mlx5en(4), so that there
is a direct relationship between mce<N> and mlx5_<N>.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Implement support for Differentiated Service Code Point, DSCP, in mlx5en(4).

The DSCP feature is controlled using a set of sysctl(8) fields under
the qos sysctl directory entry for mlx5en(4).

For Routable RoCE QPs, the DSCP should be set in the QP's address path.
The DSCP's value is derived from the traffic class.

Linux commit:
ed88451e1f2d400fd6a743d0a481631cf9f97550

MFC after: 1 week
Sponsored by: Mellanox Technologies

Add support for prio-tagged traffic for RDMA in ibcore.

When receiving a PCP change all GID entries are reloaded.
This ensures the relevant GID entries use prio tagging,
by setting VLAN present and VLAN ID to zero.

The priority for prio tagged traffic is set using the regular
rdma_set_service_type() function.

Fake the real network device to have a VLAN ID of zero
when prio tagging is enabled. This is logic is hidden inside
the rdma_vlan_dev_vlan_id() function which must always be used
to retrieve the VLAN ID throughout all of ibcore and the
infiniband network drivers.

The VLAN presence information then propagates through all
of ibcore and so incoming connections will have the VLAN
bit set. The incoming VLAN ID is then checked against the
return value of rdma_vlan_dev_vlan_id().

MFC after: 1 week
Sponsored by: Mellanox Technologies

UDP: further performance improvements on tx

Cumulative throughput while running 64
netperf -H $DUT -t UDP_STREAM -- -m 1
on a 2x8x2 SKL went from 1.1Mpps to 2.5Mpps

Single stream throughput increases from 910kpps to 1.18Mpps

Baseline:
https://people.freebsd.org/~mmacy/2018.05.11/udpsender2.svg

- Protect read access to global ifnet list with epoch
https://people.freebsd.org/~mmacy/2018.05.11/udpsender3.svg

- Protect short lived ifaddr references with epoch
https://people.freebsd.org/~mmacy/2018.05.11/udpsender4.svg

- Convert if_afdata read lock path to epoch
https://people.freebsd.org/~mmacy/2018.05.11/udpsender5.svg

A fix for the inpcbhash contention is pending sufficient time
on a canary at LLNW.

Reviewed by: gallatin
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D15409

Add support for explicit congestion notification, ECN, to mlx5ib(4).

ECN configuration and statistics is available through a set of sysctl(8)
nodes under sys.class.infiniband.mlx5_X.cong . The ECN configuration
nodes can also be used as loader tunables.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Implement missing query for current port rate in mlx5ib(4).

- Factor out port speed definitions into new port.h header file,
similarly as done in Linux upstream.
- Correct two existing port speed definitions in mlx5en according to
Linux upstream.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Disable unsupported disassociate ucontext functionality in mlx5ib(4).

MFC after: 1 week
Sponsored by: Mellanox Technologies

Optimize ibcore RoCE address handle creation from user-space.

Creating a UD address handle from user-space or from the kernel-space,
when the link layer is ethernet, requires resolving the remote L3
address into a L2 address. Doing this from the kernel is easy because
the required ARP(IPv4) and ND6(IPv6) address resolving APIs are readily
available. In userspace such an interface does not exist and kernel
help is required.

It should be noted that in an IP-based GID environment, the GID itself
does not contain all the information needed to resolve the destination
IP address. For example information like VLAN ID and SCOPE ID, is not
part of the GID and must be fetched from the GID attributes. Therefore
a source GID should always be referred to as a GID index. Instead of
going through various racy steps to obtain information about the
GID attributes from user-space, this is now all done by the kernel.

This patch optimises the L3 to L2 address resolving using the existing
create address handle uverbs interface, retrieving back the L2 address
as an additional user-space information structure.

This commit combines the following Linux upstream commits:

IB/core: Let create_ah return extended response to user
IB/core: Change ib_resolve_eth_dmac to use it in create AH
IB/mlx5: Make create/destroy_ah available to userspace
IB/mlx5: Use kernel driver to help userspace create ah
IB/mlx5: Report that device has udata response in create_ah

MFC after: 1 week
Sponsored by: Mellanox Technologies

Compile fixes for 32-bit architectures.

Sponsored by: Mellanox Technologies

Update mlx5ib(4) to match Linux 4.9 and the new ibcore APIs.

Sponsored by: Mellanox Technologies

The remote DMA TCP portspace selector, RDMA_PS_TCP, is used for both
iWarp and RoCE in ibcore. The selection of RDMA_PS_TCP can not be used
to indicate iWarp protocol use. Backport the proper IB device
capabilities from Linux upstream to distinguish between iWarp and
RoCE. Only allocate the additional socket required for iWarp for RDMA
IDs when at least one iWarp device present. This resolves
interopability issues between iWarp and RoCE in ibcore

Reviewed by: np @
Differential Revision: https://reviews.freebsd.org/D12563
Sponsored by: Mellanox Technologies
MFC after: 3 days

Use common rdma_ip2gid() function instead of custom mlx5_ip2gid() one.

Sponsored by: Mellanox Technologies
MFC after: 1 week

Compile fixes for LINT on 32-bit platforms.

MFC after: 2 weeks
Sponsored by: Mellanox Technologies

Add new mlx5ib(4) driver to the kernel source tree which supports
Remote DMA over Converged Ethernet, RoCE, for the ConnectX-4 series of
PCI express network cards.

There is currently no user-space support and this driver only supports
kernel side non-routable RoCE V1. The krping kernel module can be used
to test this driver. Full user-space support including RoCE V2 will be
added as part of the ongoing upgrade to ibcore from Linux 4.9. Otherwise
this driver is feature equivalent to mlx4ib(4). The mlx5ib(4) kernel
module will only be built when WITH_OFED=YES is specified.

MFC after: 2 weeks
Sponsored by: Mellanox Technologies