MFC r333153, r333394, r333442, r333472, r333620, r334058, r334447,
r334452, and r335684. These revisions added hashfilters, NAT offload,
and SMAC/DMAC swapping filters to cxgbe.

r333153:
cxgbe(4): Move all TCAM filter code into a separate file.

Sponsored by: Chelsio Communications

r333394:
cxgbe(4): Add support for hash filters.

These filters reside in the card's memory instead of its TCAM and can be
configured via a new "hashfilter" subcommand in cxgbetool. Hash and
normal TCAM filters can be used together. The hardware does an
exact-match of packet fields for hash filters, unlike the masked match
performed for TCAM filters. Any T5/T6 card with memory can support at
least half a million hash filters. The sample config file with the
driver configures 512K of these, it is possible to double this to 1
million+ in some cases.

The chip does an exact-match of fields of incoming datagrams with hash
filters and performs the action configured for the filter if it matches.
The fields to match are specified in a "filter mask" in the firmware
config file. The filter mask always includes the 5-tuple (sip, dip,
sport, dport, ipproto). It can, optionally, also include any subset of
the filter mode (see filterMode and filterMask in the firmware config
file).

For example:
filterMode = fragmentation, mpshittype, protocol, vlan, port, fcoe
filterMask = protocol, port, vlan

Exact values of the 5-tuple, the physical port, and VLAN tag would have
to be provided while setting up a hash filter with the chip
configuration above.

Hash filters support all actions supported by TCAM filters. A packet
that hits a hash filter can be dropped, let through (with optional
steering to a specific queue or RSS region), switched out of another
port (with optional L2 rewrite of DMAC, SMAC, VLAN tag), or get NAT'ed.
(Support for some of these will show up in the driver in a follow-up
commit very shortly).

Sponsored by: Chelsio Communications

r333442:
cxgbe(4): Determine whether the firmware supports the FILTER2 work
request, which can be used to configure hardware NAT and swapmac.

All firmwares released after Jan 2017 support this work request.

Sponsored by: Chelsio Communications

r333472:
cxgbe(4): Add fields to support configuration of hardware NAT and
swapmac (SMAC/DMAC switcheroo) from userspace.

Sponsored by: Chelsio Communications

r333620:
cxgbe(4): Filtering related features and fixes.

- Driver support for hardware NAT.
- Driver support for swapmac action.
- Validate a request to create a hashfilter against the filter mask.
- Add a hashfilter config file for T5.

Sponsored by: Chelsio Communications

r334058:
cxgbe(4): Only valid filters are expected to have a valid tid.

r334447:
cxgbe(4): Add code to deal with the chip's source MAC table (aka SMT).

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

r334452:
cxgbe(4): Add support for SMAC-rewriting filters.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

r335684:
cxgbe(4): Do not leak the filters in the hashfilter table on module
unload.

Sponsored by: Chelsio Communications

Relnotes: Yes

MFC r317849 (partial), r332506, and r332787.

r317849 (partial, required by r332506):
cxgbe/t4_tom: Per-connection rate limiting for TCP sockets handled by
the TOE.

Sponsored by: Chelsio Communications

r332506:
cxgbe(4): Add support for Connection Offload Policy (aka COP).

COP allows fine-grained control on whether to offload a TCP connection
using t4_tom, and what settings to apply to a connection selected for
offload. t4_tom must still be loaded and IFCAP_TOE must still be
enabled for full TCP offload to take place on an interface. The
difference is that IFCAP_TOE used to be the only knob and would enable
TOE for all new connections on the inteface, but now the driver will
also consult the COP, if any, before offloading to the hardware TOE.

A policy is a plain text file with any number of rules, one per line.
Each rule has a "match" part consisting of a socket-type (L = listen,
A = active open, P = passive open, D = don't care) and a pcap-filter(7)
expression, and a "settings" part that specifies whether to offload the
connection or not and the parameters to use if so. The general format
of a rule is: [socket-type] expr => settings

Example. See cxgbetool(8) for more information.
[L] ip && port http => offload
[L] port 443 => !offload
[L] port ssh => offload
[P] src net 192.168/16 && dst port ssh => offload !nagle !timestamp cong newreno
[P] dst port ssh => offload !nagle ecn cong tahoe
[P] dst port http => offload
[A] dst port 443 => offload tls
[A] dst net 192.168/16 => offload !timestamp cong highspeed

The driver processes the rules for each new listen, active open, or
passive open and stops at the first match. There is an implicit rule at
the end of every policy that prohibits offload when no rule in the
policy matches:
[D] all => !offload

This is a reworked and expanded version of a patch submitted by
Krishnamraju Eraparaju @ Chelsio.

Sponsored by: Chelsio Communications

r332787:
cxgbe(4): Fix bugs in the handling of COP rules that match on VLAN tag.

Retrieve the tag from the correct ifnet and use the provided tag
(instead of hardcoded 0xffff, implying no tag) in the routines that
process offload policy.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

MFC r319506, r319872, r321063, r321103, r321179, r321390, r321435,
r321582, r321671, r322014, r322034, r322055, r322123, r322167, r322425,
r322549, r322914, r322960, r322962, r322964, r322985, r322990, r323006,
r323026, r323041, r323069, r323078, r323343, r323514, r323520, r324296,
r324379, r324386, r324443, r324945, r325596, r325680, r325880,
r325883-r325884, r325961, r326026, r326042, r327062, r327093, r327332,
r327528, r328420, and r328423.

r319506:
cxgbe(4): Update the statistics for compound tx work requests once per
work request, not once per frame.

r319872:
cxgbe(4): Do not request an FEC setting that the port does not support.

r321063:
cxgbe(4): Various link/media related improvements.

- Deal with changes to port_type, and not just port_mod when a
transceiver is changed. This fixes hot swapping of transceivers of
different types (QSFP+ or QSA or QSFP28 in a QSFP28 port, SFP+ or
SFP28 in a SFP28 port, etc.).

- Always refresh media information for ifconfig if the port is down.
The firmware does not generate tranceiver-change interrupts unless at
least one VI is enabled on the physical port. Before this change
ifconfig diplayed potentially stale information for ports that were
administratively down.

- Always recalculate and reapply L1 config on a transceiver change.

- Display PAUSE settings in ifconfig. The driver sysctls for this
continue to work as well.

r321103:
cxgbe(4): New ioctls to flash bootrom and boot config to the card.

r321179:
cxgbe/t4_tom: Log more details about the newly ESTABLISHED tid to the
trace buffer.

r321390:
cxgbe(4): Install the firmware bundled with the driver to the card if it
doesn't seem to have one. This lets the driver recover automatically
from incomplete firmware upgrades (panic, reboot, power loss, etc. in
the middle of an upgrade).

r321435:
cxgbe(4): Display some more TOE parameters related to retransmission
and keepalive in the sysctl MIB. Provide tunables to change some of
these parameters. These are supposed to be setup by the firmware so
these tunables are for experimentation only.

r321582:
cxgbe(4): Some updates to the common code.

- Updated register ranges.
- Helper routines for access to TP registers.
- Updated routine to read flash parameters.

r321671:
cxgbe/iw_cxgbe: Log the end point's history and flags to the trace
buffer just before it's freed.

r322014:
cxgbe(4): Initial import of the "collect" component of Chelsio unified
debug (cudbg) code, hooked up to the main driver via an ioctl.

The ioctl can be used to collect the chip's internal state in a
compressed dump file. These dumps can be decoded with the "view"
component of cudbg.

r322034:
cxgbe(4): Always use the first and not the last virtual interface
associated with a port in begin_synchronized_op.

r322055:
cxgbe(4): Allow the TOE timer tunables to be set with microsecond
precision. These timers are already displayed in microseconds in the
sysctl MIB. Add variables to track these tunables while here.

r322123:
cxgbe(4): Avoid a NULL dereference that would occur during module unload
if there were problems earlier during attach.

r322167:
cxgbe(4): Add the T6 and T5 Unified Wire configuration files to the
kernel, just like for T4, when the driver is compiled into the kernel.

r322425:
cxgbe(4): Save the last reported link parameters and compare them with
the current state to determine whether to generate a link-state change
notification. This fixes a bug introduced in r321063 that caused the
driver to sometimes skip these notifications.

r322549:
cxgbe/t4_tom: Use correct name for the ISS-valid bit in options2.

r322914:
cxgbe(4): Dump the mailbox contents in the same format as CH_DUMP_MBOX.

r322960:
cxgbe(4): Verify that the driver accesses the firmware mailbox in a
thread-safe manner.

r322962:
cxgbe(4): Remove write only variable from t4_port_init.

r322964:
cxgbe(4): vi_mac_funcs should include the base Ethernet function. It is
already used in the driver as if it does.

r322985:
cxgbe(4): Maintain one ifmedia per physical port instead of one per
Virtual Interface (VI). All autonomous VIs that share a port share the
same media.

r322990:
cxgbe(4): Do not access the mailbox without appropriate locks while
creating hardware VIs.

This fixes a bad race on systems with hw.cxgbe.num_vis > 1.

r323006:
cxgbe(4): Update T6/T5/T4 firmwares to 1.16.59.0.

r323026:
cxgbe(4): Zero out the memory allocated for the debug dump.
cudbg_collect seems to expect it this way.

r323041:
cxgbe(4): Add two new debug flags -- one to allow manual firmware
install after full initialization, and another to disable the TCB
cache (T6+). The latter works as a tunable only.

Note that debug_flags are for debugging only and should not be set
normally.

r323069:
cxgbe/t4_tom: Add a knob to select the congestion control algorigthm
used by the TOE hardware for fully offloaded connections. The knob
affects new connections only.

r323078:
cxgbe/t4_tom: There may not be a tid to update if the connection isn't
established.

r323343:
cxgbe(4): Fix a couple of problems in the sge_wrq data path.

- start_wrq_wr must not drain the wr_list if there are incomplete_wrs
pending. This can happen when a t4_wrq_tx runs between two
start_wrq_wr.

- commit_wrq_wr must examine the cookie's pidx and ndesc with the
queue's lock held. Otherwise there is a bad race when incomplete WRs
are being completed and commit_wrq_wr for the WR that is ahead in the
queue updates the next incomplete WR's cookie's pidx/ndesc but the
commit_wrq_wr for the second one is using stale values that it read
without the lock.

r323514:
cxgbetool(8): mode must be specified when creating the dump file.

r323520:
cxgbe(4): Ignore capabilities that depend on TOE when the firmware
reports TOE is not available.

r324296:
cxgbe(4): Provide knobs to set the holdoff parameters of TOE rx queues
separately from NIC rx queues instead of using the same parameters for
both types of queues.

r324379:
cxgbetool(8): Do not create a large file devoid of useful content when
the dumpstate ioctl fails. Make the file world-readable while here.

r324386:
cxgbe(4): Update T6, T5, and T4 firmwares to 1.16.63.0.

r324443:
cxgbetool(8): Do not close uninitialized fd on malloc failure.

r324945:
cxgbe(4): Read the MPS buffer group map from the firmware as it could be
different from hardware defaults. The congestion channel map, which is
still fixed, needs to be tracked separately now. Change the congestion
setting for TOE rx queues to match the drivers on other OSes while here.

r325596:
cxgbe(4): Do not request settings not supported by the port.

r325680:
cxgbe(4): Excluce mdi from the check against port capabilities.

r325880:
cxgbe(4): Combine all _10g and _1g tunables and drop the suffix from
their names. The finer-grained knobs weren't practically useful.

r325883:
cxgbe(4): Sanitize t4_num_vis during MOD_LOAD like all other t4_*
tunables. Add num_vis to the intrs_and_queues structure as it affects
the number of interrupts requested and queues created. In future
cfg_itype_and_nqueues might lower it incrementally instead of going
straight to 1 when enough interrupts aren't available.

r325884:
cxgbe(4): Remove rsrv_noflowq from intrs_and_queues structure as it does
not influence or get affected by the number of interrupts or queues.

r325961:
cxgbe(4): Add core Vdd to the sysctl MIB.

r326026:
cxgbe(4): Add a custom board to the device id list.

r326042:
cxgbe(4): Fix unsafe mailbox access in cudbg.

r327062:
cxgbe(4): Read the MFG diags version from the VPD and make it available
in the sysctl MIB.

r327093:
cxgbe(4): Do not forward interrupts to queues with freelists. This
leaves the firmware event queue (fwq) as the only queue that can take
interrupts for others.

This simplifies cfg_itype_and_nqueues and queue allocation in the driver
at the cost of a little (never?) used configuration. It also allows
service_iq to be split into two specialized variants in the future.

r327332:
cxgbe(4): Reduce duplication by consolidating minor variations of the
same code into a single routine.

r327528:
cxgbe(4): Add a knob to enable/disable PCIe relaxed ordering. Disable it by
default when running on Intel CPUs.

r328420:
cxgbe(4): Do not display harmless warning in non-debug builds.

r328423:
cxgbe(4): Accept old names of a couple of tunables.

Sponsored by: Chelsio Communications

MFC 306821,306823: Permit updating firmware config file in flash.

306821:
cxgbe(4): Add an ioctl to copy a firmware config file to the card's flash.

306823:
cxgbetool: Add a loadcfg subcommand to allow a user to upload a firmware
configuration file to the card.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

cxgbe(4): Create a reusable struct type for scheduling class parameters.

Sponsored by: Chelsio Communications

sys/dev: minor spelling fixes.

Most affect comments, very few have user-visible effects.

cxgbe(4): Overhaul the shared code that deals with the chip's TP block,
which is responsible for filtering and RSS.

Add the ability to use filters that match on PF/VF (aka "VNIC id") while
here. This is mutually exclusive with filtering on outer VLAN tag with
Q-in-Q.

Sponsored by: Chelsio Communications

cxgbe(4): T4_SET_SCHED_CLASS and T4_SET_SCHED_QUEUE ioctls to program
scheduling classes in the chip and to bind tx queue(s) to a scheduling
class respectively. These can be used for various kinds of tx traffic
throttling (to force selected tx queues to drain at a fixed Kbps rate,
or a % of the port's total bandwidth, or at a fixed pps rate, etc.).

Obtained from: Chelsio

Add support for packet-sniffing tracers to cxgbe(4). This works with
all T4 and T5 based cards and is useful for analyzing TSO, LRO, TOE, and
for general purpose monitoring without tapping any cxgbe or cxl ifnet
directly.

Tracers on the T4/T5 chips provide access to Ethernet frames exactly as
they were received from or transmitted on the wire. On transmit, a
tracer will capture a frame after TSO segmentation, hw VLAN tag
insertion, hw L3 & L4 checksum insertion, etc. It will also capture
frames generated by the TCP offload engine (TOE traffic is normally
invisible to the kernel). On receive, a tracer will capture a frame
before hw VLAN extraction, runt filtering, other badness filtering,
before the steering/drop/L2-rewrite filters or the TOE have had a go at
it, and of course before sw LRO in the driver.

There are 4 tracers on a chip. A tracer can trace only in one direction
(tx or rx). For now cxgbetool will set up tracers to capture the first
128B of every transmitted or received frame on a given port. This is a
small subset of what the hardware can do. A pseudo ifnet with the same
name as the nexus driver (t4nex0 or t5nex0) will be created for tracing.
The data delivered to this ifnet is an additional copy made inside the
chip. Normal delivery to cxgbe<n> or cxl<n> will be made as usual.

/* watch cxl0, which is the first port hanging off t5nex0. */
# cxgbetool t5nex0 tracer 0 tx0 (watch what cxl0 is transmitting)
# cxgbetool t5nex0 tracer 1 rx0 (watch what cxl0 is receiving)
# cxgbetool t5nex0 tracer list
# tcpdump -i t5nex0 <== all that cxl0 sees and puts on the wire

If you were doing TSO, a tcpdump on cxl0 may have shown you ~64K
"frames" with no L3/L4 checksum but this will show you the frames that
were actually transmitted.

/* all done */
# cxgbetool t5nex0 tracer 0 disable
# cxgbetool t5nex0 tracer 1 disable
# cxgbetool t5nex0 tracer list
# ifconfig t5nex0 destroy

Reserve room for ioctls that aren't in this copy of the driver yet.

cxgbe(4): Add support for Chelsio's Terminator 5 (aka T5) ASIC. This
includes support for the NIC and TOE features of the 40G, 10G, and
1G/100M cards based on the T5.

The ASIC is mostly backward compatible with the Terminator 4 so cxgbe(4)
has been updated instead of writing a brand new driver. T5 cards will
show up as cxl (short for cxlgb) ports attached to the t5nex bus driver.

Sponsored by: Chelsio

Add a driver ioctl to clear a port's MAC statistics.

Submitted by: gnn@
MFC after: 3 days

Add a driver ioctl to read a byte from any device on a port's i2c bus.
This lets userspace read arbitrary information from the SFP+ modules
etc. on this bus.

Reading multiple bytes in the same transaction isn't possible right now.
I'll update the driver once the chip's firmware supports this.

MFC after: 3 days

Many updates to cxgbe(4)

- Device configuration via plain text config file. Also able to operate
when not attached to the chip as the master driver.

- Generic "work request" queue that serves as the base for both ctrl and
ofld tx queues.

- Generic interrupt handler routine that can process any event on any
kind of ingress queue (via a dispatch table).

- A couple of new driver ioctls. cxgbetool can now install a firmware
to the card ("loadfw" command) and can read the card's memory
("memdump" and "tcb" commands).

- Lots of assorted information within dev.t4nex.X.misc.* This is
primarily for debugging and won't show up in sysctl -a.

- Code to manage the L2 tables on the chip.

- Updates to cxgbe(4) man page to go with the tunables that have changed.

- Updates to the shared code in common/

- Updates to the driver-firmware interface (now at fw 1.4.16.0)

MFC after: 1 month

- driver ioctl to get SGE context for any given queue.
- sysctls to display the context id, cidx, and pidx of all kinds of queues.

MFC after: 3 days

L2 table code. This is enough to get the T4's switch + L2 rewrite
filters working. (All other filters - switch without L2 info rewrite,
steer, and drop - were already fully-functional).

Some contrived examples of "switch" filters with L2 rewriting:

# cxgbetool t4nex0 iport 0 dport 80 action switch vlan +9 eport 3
Intercept all packets received on physical port 0 with TCP port 80 as
destination, insert a vlan tag with VID 9, and send them out of port 3.

# cxgbetool t4nex0 sip 192.168.1.1/32 ivlan 5 action switch \
vlan =9 smac aa:bb:cc:dd:ee:ff eport 0
Intercept all packets (received on any port) with source IP address
192.168.1.1 and VLAN id 5, rewrite the VLAN id to 9, rewrite source mac
to aa:bb:cc:dd:ee:ff, and send it out of port 0.

MFC after: 1 week

T4 packet filtering/steering.

- Enable 5-tuple and every-packet lookup.

- Setup the default filter mode to allow filtering/steering based on IP
protocol, ingress port, inner VLAN ID, IP frag, FCoE, and MPS match
type; all combined together. You can also filter based on MAC index,
Ethernet type, IP TOS/IPv6 Traffic Class, and outer VLAN ID but you'll
have to modify the default filter mode and exclude some of the
match-fields in it.

IPv4 and IPv6 SIP/DIP/SPORT/DPORT are always available in all filter
rules.

- Add driver ioctls to get/set the global filter mode.

- Add driver ioctls to program and delete hardware filters. A couple of
the "switch" actions that rewrite Ethernet and VLAN information and
switch the packet out of another port may not work as the L2 code is not
yet in place. Everything else, including all "drop" and "pass" rules
with RSS or absolute qid, should work.

Obtained from: Chelsio Communications

Modify read/write ioctls to work with 64 bit registers too.

MFC after: 3 days

cxgbe(4) - NIC driver for Chelsio T4 (Terminator 4) based 10Gb/1Gb adapters.

MFC after: 3 weeks