MFC r362201:

Avoid trying to toggle TSO twice

Remove TSO from the toggle mask when automatically disabled by TXCKSUM* in
various NIC drivers.

Reviewed by: hselasky, np, gallatin, jpaetzel
Approved by: mav (mentor)
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25120

MFC r354742:

cxgbev(4): Catch up with the pciids in the PF driver.

Sponsored by: Chelsio Communications

MFC r354522:

cxgbe(4): Query Vdd from the firmware if its last known value is 0.

TVSENSE may not be ready by the time t4_fw_initialize returns and the
firmware returns 0 if the driver asks for the Vdd before the sensor is
ready.

Sponsored by: Chelsio Communications

MFC r354106:

cxgbe(4): Use correct FetchBurstMin values for T6.

Sponsored by: Chelsio Communications

MFC r351524:

cxgbe/t4_tom: Limit work requests with immediate payload to a single
descriptor. The per-tid tx credits are in demand during active Tx and
it's best not to use too many just for payload.

Sponsored by: Chelsio Communications

MFC r351446:

cxgbe/t4_tom: Any invalid scaling factor in the hardware's wsf field
implies that window scaling is not in use.

Sponsored by: Chelsio Communications

MFC r351445:

whitespace nit.

MFC r349956:

cxgbe(4): Completely ignore all top level interrupts that are not enabled.

The driver used to log any non-zero cause and when running with a single
line interrupt it would spam the console/logs with reports of interrupts
that are of no interest to anyone.

Sponsored by: Chelsio Communications

MFC r349865:

cxgbe(4): Use the simplest configuration possible when falling back from
the default configuration.

Sponsored by: Chelsio Communications

MFC r349500:

cxgbe/t4_tom: Fix regression in t_maxseg usage within t4_tom.

t_maxseg was changed in r293284 to not have any adjustment for TCP
timestamps. t4_tom inadvertently went back to pre-r293284 semantics
in r332506.

Sponsored by: Chelsio Communications

MFC r349499:

cxgbe/iw_cxgbe: Remove unused field from the endpoint structure.

MFC r349242:

cxgbe/t4_tom: DDP_DEAD is a ddp flag and not a toepcb flag.

The driver was in effect setting TPF_ABORT_SHUTDOWN on the toepcb
instead of what was intended.

Sponsored by: Chelsio Communications

MFC 353369: Remove adapters from t4_list earlier during detach.

This ensures the clip task won't race with t4_destroy_clip_table.

While here, make some mutex destroys unconditional since attach always
initializes them.

Sponsored by: Chelsio Communications

MFC 353323: Set the FID field in lookaside crypto requests to the rx queue ID.

The PCI block in the adapter requires this field to be set to a valid
queue ID. It is not clear why it did not fail on all machines, but
the effect was that crypto operations reading input data via DMA
failed with an internal PCI read error on machines with 128G or more
of RAM.

MFC 349467: Hold an explicit reference on the socket for the aiotx task.

Previously, the aiotx task relied on the aio jobs in the queue to hold
a reference on the socket. However, when the last job is completed,
there is nothing left to hold a reference to the socket buffer lock
used to check if the queue is empty. In addition, if the last job on
the queue is cancelled, the task can run with no queued jobs holding a
reference to the socket buffer lock the task uses to notice the queue
is empty.

Fix these races by holding an explicit reference on the socket when
the task is queued and dropping that reference when the task
completes.

MFC 348791: Fix debug trace after removal of pdu_overhead.

MFC r348491:

cxgbe/t4_tom: adjust the hardware receive window to match changes to the
receive sockbuf's high water mark.

Calculate rx credits on the spot instead of tracking sbused/sb_cc and
rx_credits in the toepcb. The previous method worked when the high
water mark changed due to SB_AUTOSIZE but not when it was adjusted
directly (for example, by the soreserve in nfsrvd_addsock).

This fixes a connection hang while running iozone over an NFS mounted
share where nfsd's TCP sockets are being handled by t4_tom.

Sponsored by: Chelsio Communications

Approved by: re@ (gjb@)

MFC r342208:

cxgbe/t4_tom: fixes for issues on the passive open side.

- Fix PR 227760 by getting the TOE to respond to the SYN after the call
to toe_syncache_add, not during it. The kernel syncache code calls
syncache_respond just before syncache_insert. If the ACK to the
syncache_respond is processed in another thread it may run before the
syncache_insert and won't find the entry. Note that this affects only
t4_tom because it's the only driver trying to insert and expand
syncache entries from different threads.

- Do not leak resources if an embryonic connection terminates at
SYN_RCVD because of L2 lookup failures.

- Retire lctx->synq and associated code because there is never a need to
walk the list of embryonic connections associated with a listener.
The per-tid state is still called a synq entry in the driver even
though the synq itself is now gone.

PR: 227760
Sponsored by: Chelsio Communications

MFC r345334:

cxgbe(4): Treat the viid as an opaque identifier.

Recent firmwares prefer to use a different format for viid internally
and this change allows them to do so.

Sponsored by: Chelsio Communications

MFC r344654:

cxgbe(4): Request high priority filter support explicitly, as required
by recent firmwares.

Sponsored by: Chelsio Communications

MFC r343889, r344519, r344682, r344719

r343889:
cxgbev(4): Initialize debug_flags from the environment like in the PF driver.

r344519:
cxgbe(4): Use correct port_info in the call to is_bt().

This fixes a panic during configuration if the tx channel of a port
isn't the same as its port id.

Reported by: Fabrice Bruel
Sponsored by: Chelsio Communications

r344682:
cxgbe(4): Don't forget to report link state to the kernel if the link is
already up at attach.

Reported by: Fabrice Bruel @ Orange Business Service
Sponsored by: Chelsio Communications

r344719:
cxgbev(4): Enable 32b port capabilities in the VF driver.

Sponsored by: Chelsio Communications

MFC r343666, r343861-r343862, r343923, r343968, r345660, r345810

r343666:
cxgbe(4): Improved error reporting and diagnostics.

"slow" interrupt handler:
- Expand the list of INT_CAUSE registers known to the driver.
- Add decode information for many more bits but decouple it from the
rest of intr_info so that it is entirely optional.
- Call t4_fatal_err exactly once, and from the top level PL intr handler.

t4_fatal_err:
- Use t4_shutdown_adapter from the common code to stop the adapter.
- Stop servicing slow interrupts after the first fatal one.

Driver/firmware interaction:
- CH_DUMP_MBOX: note whether the mailbox being dumped is a command or a
reply or something else.
- Log the raw value of pcie_fw for some errors.
- Use correct log levels (debug vs. error).

Sponsored by: Chelsio Communications

r343861:
cxgbe(4): Auto-dump the device log on a mailbox timeout or when the
firmware reports an error in pcie_fw.

Sponsored by: Chelsio Communications

r343862:
cxgbe(4): Auto-dump the CIM block's logic analyzer on a TIMER0 interrupt.

Sponsored by: Chelsio Communications

r343923:
cxgbe(4): Delay the panic due to a fatal error by 30s.

This lets information logged by the interrupt handler reach the system
log before the system goes down.

r343968:
cxgbe(4): Ignore unused interrupts.

Sponsored by: Chelsio Communications

r345660:
cxgbe(4): Count and clear interrupts generated at the software's request.

An interrupt can be requested by setting the F_SWINT bit in PL_PF_CTL.

Sponsored by: Chelsio Communications

r345810:
cxgbe(4): Add a flag to indicate that bits in interrupt cause but not in
interrupt enable are not fatal.

The firmware sets up all the interrupt enables based on run time
configuration, which means the information in the enables is more
accurate than what's compiled into the driver. This change also allows
the fatal bits to be updated without any changes in the driver in some
cases.

Sponsored by: Chelsio Communications

MFC r343539:

cxgbe(4): Add adapter information to messages logged by the OS-agnostic
code in t4_hw.c.

Sponsored by: Chelsio Communications

MFC r343269, r346567

r343269:
cxgbe(4): Allow negative values in hw.cxgbe.fw_install and take them to
mean that the driver should taste the firmware in the KLD and use that
firmware's version for all its fw_install checks.

The driver gets firmware version information from compiled-in values by
default and this change allows custom (or older/newer) firmware modules
to be used with the stock driver.

There is no change in default behavior.

Sponsored by: Chelsio Communications

r346567:
cxgbe(4): Make sure bundled_fw is always initialized before use.

This fixes a bug that prevented the driver from auto-flashing the
firmware when it didn't see one on the card. This feature was
introduced in r321390 and this bug was introduced in r343269.

Reported by: gallatin@
Sponsored by: Chelsio Communications

MFC r343264:

cxgbe(4): Use a truncated firmware header for version checks. All the
version numbers are towards the begining of the header.

Sponsored by: Chelsio Communications

MFC r343233:

cxgbe(4): Clear the reply-pending status of a hashfilter when the reply
indicates an error. Also, do not remove it twice from the hf list in
this case.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communicatons

MFC r343569, r345307

r343569:
cxgbe/iw_cxgbe: Fix an address calculation in the memory registration code that
was added in r342266.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

r345307:
iw_cxgbe: Remove unused smac_idx from the ep structure.

Submitted by: Krishnamraju Eraparaju @ Chelsio

MFC r342954:

cxgbe(4): Move some INTx specific code to a more appropriate place.

MFC r342758:

cxgbe(4): Clear FW_OK if the firmware reports an error.

Sponsored by: Chelsio Communications

MFC r342356:

Remove unused macros from t4_tom.h.

MFC r342234:

cxgbe(4): Do not issue mbox commands after t4_fw_bye.

Sponsored by: Chelsio Communications

MFC r341654:

cxgbe(4): Get Linux cxgb4vf working in bhyve VMs with VFs passed
through.

cxgb4vf doesn't own the buffer size list but still expects the first two
entries to be 4K and some power of 2 respectively. The BSD cxgbe
doesn't care where its preferred buffer sizes are as long as they're in
the list somewhere, so just move its entries towards the end as a
workaround.

Sponsored by: Chelsio Communications

MFC r341620:

cxgbe(4): Fall back to a basic configuration in case of any error during
card initialization. This is an expanded version of r333682.

Break up prep_firmware into simpler routines while here. Load the
firmware/config KLD only if needed.

Sponsored by: Chelsio Communications

MFC r338954, r340651, r344524, r345083.

r338954:
cxgbe(4): Enable support for per-connection rate limiting in the default
firmware configuration files.

Approved by: re@ (gjb@)
Sponsored by: Chelsio Communications

r340651:
cxgbe(4): Update T4/5/6 firmwares to 1.22.0.3.

Obtained from: Chelsio Communications
Sponsored by: Chelsio Communications

r344524:
cxgbe(4): Updates to the default and hashfilter configurations.

- Do not use nvf = 4 as it is not really supported by the firmware.
Firmwares 1.23.3.0 and above will ignore it silently.
- Increase PF4's share of the VIs and let it use all of the RSS table.

Sponsored by: Chelsio Communications

r345083:
cxgbe(4): Update T4/5/6 firmwares to 1.23.0.0.

Obtained from: Chelsio Communications
Sponsored by: Chelsio Communications

MFC r341172, r341270.
t4_clip.c had to be manually adjusted because Concurrency Kit is not
available in stable/11.

r341172:
Move CLIP table handling out of TOM and into the base driver.

- Store the clip table in 'struct adapter' instead of in the TOM softc.
- Init the clip table during attach and teardown during detach.
- While here, add a dev.<nexus>.<unit>.misc.clip sysctl to dump the
CLIP table.

This does mean that we update the clip table even if TOE is not enabled,
but non-TOE things need the CLIP table anyway.

Reviewed by: np, Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D18010

r341270:
Make most of the CLIP code conditional on #ifdef INET6.

This fixes builds of kernels without INET6 such as LINT-NOINET6.

Reported by: arybchik
Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D18384

MFC r339700:

cxgbe(4): new sysctl to display the start of the RSS region for a VI.

dev.<ifname>.<inst>.rss_base

For example:
dev.cc.0.rss_base: 0
dev.cc.1.rss_base: 128
dev.vcc.0.rss_base: 256
dev.vcc.1.rss_base: 384

Sponsored by: Chelsio Communications

MFC r339628, r339965

r339628:
cxgbe(4): improve the accuracy of various TSO limits reported to the kernel.

Sponsored by: Chelsio Communications

r339965:
cxgbe(4): Report a reasonable non-zero if_hw_tsomaxsegsize to the
kernel.

This reverts an accidental change that snuck in with r339628.

Sponsored by: Chelsio Communications

MFC r339891, r340063, r342266, r342270, r342272, r342288-r342289

r339891:
cxgbe/iw_cxgbe: Install the socket upcall before calling soconnect to
ensure that it always runs when soisconnected does.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

r340063:
cxgbe/iw_cxgbe: Suppress spurious "Unexpected streaming data ..."
messages.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

r342266:
cxgbe/iw_cxgbe: Use DSGLs to write to card's memory when appropriate.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

r342270:
cxgbe/iw_cxgbe: Add a knob for testing that lets iWARP connections cycle
through 4-tuples quickly.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

r342272:
cxgbe/iw_cxgbe: Use -ve errno when interfacing with linuxkpi/OFED.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

r342288:
cxgbe/iw_cxgbe: Do not terminate CTRx messages with \n.

r342289:
cxgbe/iw_cxgbe: Remove redundant CTRs from c4iw_alloc/c4iw_rdev_open.
This information is readily available elsewhere.

Sponsored by: Chelsio Communications

MFC r339667:

cxgbe/iw_cxgbe: save the ep in the driver-private provider_data field.

Submitted By: Lily Wang @ Netapp

MFC r339626:

cxgbe(4): Use automatic cidx updates with ofld and ctrl queues.

The bits that explicitly request cidx updates do not work reliably with
all possible WRs that can be sent over the queue. The F_FW_WR_EQUIQ
requests that still remain may also have to be replaced with explicit
credit flush WRs in the future.

Sponsored by: Chelsio Communications

MFC r338940:

cxgbe(4): Treat base/end of firmware parameters as signed integers when
figuring out whether the range is valid or not.

MFC r338874:

cxgbe(4): Reuse existing "switching" L2T entries when possible.

MFC r338669:

cxgbe(4): Use the correct number of parameters when querying the tid
range for hashfilters.

MFC r338355:

cxgbe/tom: Unregister shared CPL handlers on module unload. This fixes
a panic with INVARIANTS that occurs when t4_tom is unloaded and reloaded.

MFC r338218:

cxgbev(4): Updates to the VF driver to cope with recent ifmedia and
ctrlq changes in the base driver.

Sponsored by: Chelsio Communications

MFC r338156, r338158-r338161, r338166.

r338156:
cxgbe(4): Avoid overflow while calculating channel rate.

Reported by: Coverity (CID 1008352)

r338158:
cxgbe(4): Check the RO bit properly before disabling relaxed ordering.

Reported by: Coverity (CID 1384286)

r338159:
cxgbe(4): Make it clear that VI_INIT_DONE implies vi->ntxq > 0, and so
rc will never be returned uninitialized.

Reported by: Coverity (CID 1394884). This is a false positive though.

r338160:
cxgbe(4): Do not leak memory in case of errors during VI initialization.

Reported by: Coverity (CID 1392026)

r338161:
cxgbe/tom: Make sure 'matched' is always initialized before use.

Reported by: Coverity (CID 1390894)

r338166:
cxgbe(4): Be explicit about ignoring the return value of cmpset in some
cases.

Reported by: Coverity (CIDs 1009398, 1009400, 1009401, 1357325, 1394783). All false positives.

MFC r337873:

cxgbe(4): Use VLAN_TRUNKDEV instead of private cookie to figure out the
parent of a VLAN ifnet.

Sponsored by: Chelsio Communications

MFC r337830:

cxgbe(4): Use two hashes instead of a table to keep track of
hashfilters. Two because the driver needs to look up a hashfilter by
its 4-tuple or tid.

A couple of fixes while here:
- Reject attempts to add duplicate hashfilters.
- Do not assume that any part of the 4-tuple that isn't specified is 0.
This makes it consistent with all other mandatory parameters that
already require explicit user input.

MFC r337659:

cxgbe(4): Move all control queues to the adapter.

There used to be one control queue per adapter (the mgmtq) that was
initialized during adapter init and one per port that was initialized
later during port init. This change moves all the control queues (one
per port/channel) to the adapter so that they are initialized during
adapter init and are available before any port is up. This allows the
driver to issue ctrlq work requests over any channel without having to
bring up any port.

MFC r337609:

cxgbe(4): Create two variants of service_iq, one for queues with
freelists and one for those without.

MFH: 3 weeks
Sponsored by: Chelsio Communications

MFC r337538, r337987

r337538:
cxgbe(4): Add support for high priority filters on T6+. They have their
own region in the TCAM starting with T6, unlike previous chips where
they were in the same region as normal filters.

These filters "hit" before anything else in the LE's lookup. The exact
order is:
a) High priority filters
b) TOE's active region (TCAM and/or hash)
c) Servers (TOE hw listeners)
d) Normal filters

Sponsored by: Chelsio Communications

r337987:
cxgbe(4): Adjust ntids to account for nhptids in the TOE case too.
This should have been part of r337538.

MFC r337192:

cxgbe(4): Improvements in TID management.

- Ignore any type of TID where the start/end values are not in the
correct order. There are situations where the firmware isn't able to
reserve room for the number requested in the config file but doesn't
report a failure during configuration and instead sets end <= start.

- Track start/end in tid_tab and remove some redundant copies from
adapter->params.

- Move all the start/end and other read-only parameters to a quiet part
of tid_tab, away from the tid locks.

Sponsored by: Chelsio Communications

MFC r336718, r336720, r336734-r336735, r337398, r337439, and r337540.
These are all related to tx rate limiting in cxgbe.

r336718:
cxgbe(4): Validate only those parameters that are relevant to the
type of rate limiter being programmed. Skip the ones that are not
applicable.

Sponsored by: Chelsio Communications

r336720:
cxgbe(4): Remove useless code that crept in with r336718.

X-MFC With: 336718

r336734:
cxgbe(4): Better defaults for all cl-rl rate limiters.

Start in "class" instead of "flow" mode. This eliminates the need to
specify an MTU, which is not available that early anyway. It also
allows the user to manually configure ch-rl rate limiting after attach.
This used to fail because ch-rl isn't supported if cl-rl "flow" mode is
configured.

Set all traffic classes to 1Gbps during initialization. The goal is to
start off with _any_ valid configuration and 1Gbps works even for
gigabit cards.

Sponsored by: Chelsio Communications

r336735:
cxgbe(4): Consider rateunit before ratemode when displaying information
about a traffic class. This matches the order in which the firmware
evaluates unit and mode internally.

Sponsored by: Chelsio Communications

r337398:
cxgbe(4): Allow user-configured and driver-configured traffic classes to
be used simultaneously. Move sysctl_tc and sysctl_tc_params to
t4_sched.c while here.

Sponsored by: Chelsio Communications

r337439:
cxgbe(4): Allow the driver to specify a burst size when configuring a
traffic class for rate limiting.

Add experimental knobs that allow the user to specify a default pktsize
and burstsize for traffic classes associated with a port:

dev.<ifname>.<instance>.tc.pktsize
dev.<ifname>.<instance>.tc.burstsize

Sponsored by: Chelsio Communications

r337540:
cxgbe(4): Display pkt-size and burst-size in traffic class parameters.

MFC r337397:

cxgbe(4): Break up sysctl_bitfield into 8 bit and 16 bit variants. Have
them display the current value of the bitfield rather than the fixed
value that was provided when the sysctl node was created.

Sponsored by: Chelsio Communications

MFC r335223:

cxgbe(4): sysctls to display the local and intr CPUs for the adapter.

The driver assumes the list can change (even though it does't right now)
and queries it every time the sysctl runs.

sysctl dev.<nexus>.<inst>.local_cpus
sysctl dev.<nexus>.<inst>.intr_cpus

sysctl dev.t6nex.0.local_cpus
sysctl dev.t6nex.0.intr_cpus

Sponsored by: Chelsio Communications

MFC r335701:

cxgbe/cxgbei: Fix harmful typo in the iSCSI offload driver.

Reported by: gcc8 (via mmacy@)
Sponsored by: Chelsio Communications

MFC r334467:

cxgbe(4): Retire an old check.

MFC r334139:

cxgbe/t4_tom: ABORT_RPL_RSS is a shared CPL and t4_tom shouldn't remove
the global handler when it's being unloaded.

MFC r334138:

r334138: cxgbe(4): Make FW4_ACK a shared CPL.

MFC r334137:

cxgbe(4): Fix range checks in is_etid.

MFC r334136:

cxgbe(4): Slightly simpler needs_<foo> functions.

MFC r334132:

cxgbe(4): Make sure that the egress queue's cidx is updated periodically
when the driver is writing WRs using start_wrq_wr/commit_wrq_wr all the
time.

Sponsored by: Chelsio Communications

MFC r333141 (by gallatin@):

Optionally panic when cxgbe encounters a fatal error

Sometimes it is better to panic than to leave a machine
unreachable.

Reviewed by: np
Sponsored by: Netflix

MFC r333153, r333394, r333442, r333472, r333620, r334058, r334447,
r334452, and r335684. These revisions added hashfilters, NAT offload,
and SMAC/DMAC swapping filters to cxgbe.

r333153:
cxgbe(4): Move all TCAM filter code into a separate file.

Sponsored by: Chelsio Communications

r333394:
cxgbe(4): Add support for hash filters.

These filters reside in the card's memory instead of its TCAM and can be
configured via a new "hashfilter" subcommand in cxgbetool. Hash and
normal TCAM filters can be used together. The hardware does an
exact-match of packet fields for hash filters, unlike the masked match
performed for TCAM filters. Any T5/T6 card with memory can support at
least half a million hash filters. The sample config file with the
driver configures 512K of these, it is possible to double this to 1
million+ in some cases.

The chip does an exact-match of fields of incoming datagrams with hash
filters and performs the action configured for the filter if it matches.
The fields to match are specified in a "filter mask" in the firmware
config file. The filter mask always includes the 5-tuple (sip, dip,
sport, dport, ipproto). It can, optionally, also include any subset of
the filter mode (see filterMode and filterMask in the firmware config
file).

For example:
filterMode = fragmentation, mpshittype, protocol, vlan, port, fcoe
filterMask = protocol, port, vlan

Exact values of the 5-tuple, the physical port, and VLAN tag would have
to be provided while setting up a hash filter with the chip
configuration above.

Hash filters support all actions supported by TCAM filters. A packet
that hits a hash filter can be dropped, let through (with optional
steering to a specific queue or RSS region), switched out of another
port (with optional L2 rewrite of DMAC, SMAC, VLAN tag), or get NAT'ed.
(Support for some of these will show up in the driver in a follow-up
commit very shortly).

Sponsored by: Chelsio Communications

r333442:
cxgbe(4): Determine whether the firmware supports the FILTER2 work
request, which can be used to configure hardware NAT and swapmac.

All firmwares released after Jan 2017 support this work request.

Sponsored by: Chelsio Communications

r333472:
cxgbe(4): Add fields to support configuration of hardware NAT and
swapmac (SMAC/DMAC switcheroo) from userspace.

Sponsored by: Chelsio Communications

r333620:
cxgbe(4): Filtering related features and fixes.

- Driver support for hardware NAT.
- Driver support for swapmac action.
- Validate a request to create a hashfilter against the filter mask.
- Add a hashfilter config file for T5.

Sponsored by: Chelsio Communications

r334058:
cxgbe(4): Only valid filters are expected to have a valid tid.

r334447:
cxgbe(4): Add code to deal with the chip's source MAC table (aka SMT).

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

r334452:
cxgbe(4): Add support for SMAC-rewriting filters.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

r335684:
cxgbe(4): Do not leak the filters in the hashfilter table on module
unload.

Sponsored by: Chelsio Communications

Relnotes: Yes

MFC r333121, r333128

r333121:
cxgbe/t4_tom: Use appropriate macros instead of magic math while
constructing the atid of an active open work request.

Sponsored by: Chelsio Communications

r333128:
cxgbe(4): Convert ACT_OPEN_RPL to a shared CPL.

Reserve 3b in the 14b atid to identify the owner and use it to dispatch
the CPL. This allows all CPLs that use an atid to be used as shared
CPLs, although ACT_OPEN_RPL is the only one being converted in this
revision.

Sponsored by: Chelsio Communications

MFC r333114:

cxgbe(4): Use opaque cookies or tid range-checks to determine the
intended recipient of a CPL when it can't be determined solely from the
opcode. Retire the per-queue handlers for such CPLs in favor of the new
scheme.

Sponsored by: Chelsio Communications

MFC r333043:

cxgbe(4): Move release_tid to the base NIC driver for future consumers.

Sponsored by: Chelsio Communications.

MFC r333030:

cxgbe(4): Break up alloc_tid_tabs and move the atid routines to the base
NIC driver. The atid services will be used by new features (hashfilters
and inline TLS) that do not involve TOE.

Sponsored by: Chelsio Communications

MFC r331902:

r331902: cxgbe: Implement tcp_info handler for connections handled by t4_tom.

MFC r317849 (partial), r332506, and r332787.

r317849 (partial, required by r332506):
cxgbe/t4_tom: Per-connection rate limiting for TCP sockets handled by
the TOE.

Sponsored by: Chelsio Communications

r332506:
cxgbe(4): Add support for Connection Offload Policy (aka COP).

COP allows fine-grained control on whether to offload a TCP connection
using t4_tom, and what settings to apply to a connection selected for
offload. t4_tom must still be loaded and IFCAP_TOE must still be
enabled for full TCP offload to take place on an interface. The
difference is that IFCAP_TOE used to be the only knob and would enable
TOE for all new connections on the inteface, but now the driver will
also consult the COP, if any, before offloading to the hardware TOE.

A policy is a plain text file with any number of rules, one per line.
Each rule has a "match" part consisting of a socket-type (L = listen,
A = active open, P = passive open, D = don't care) and a pcap-filter(7)
expression, and a "settings" part that specifies whether to offload the
connection or not and the parameters to use if so. The general format
of a rule is: [socket-type] expr => settings

Example. See cxgbetool(8) for more information.
[L] ip && port http => offload
[L] port 443 => !offload
[L] port ssh => offload
[P] src net 192.168/16 && dst port ssh => offload !nagle !timestamp cong newreno
[P] dst port ssh => offload !nagle ecn cong tahoe
[P] dst port http => offload
[A] dst port 443 => offload tls
[A] dst net 192.168/16 => offload !timestamp cong highspeed

The driver processes the rules for each new listen, active open, or
passive open and stops at the first match. There is an implicit rule at
the end of every policy that prohibits offload when no rule in the
policy matches:
[D] all => !offload

This is a reworked and expanded version of a patch submitted by
Krishnamraju Eraparaju @ Chelsio.

Sponsored by: Chelsio Communications

r332787:
cxgbe(4): Fix bugs in the handling of COP rules that match on VLAN tag.

Retrieve the tag from the correct ifnet and use the provided tag
(instead of hardcoded 0xffff, implying no tag) in the routines that
process offload policy.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

MFC 330040,330041,330079,330884,330946,330947,331649,333068,333810,337722,
340466,340468,340469,340473: Add TOE-based TLS offload.

Note that this requires a modified OpenSSL library.

330040:
Fetch TLS key parameters from the firmware.

The parameters describe how much of the adapter's memory is reserved for
storing TLS keys. The 'meminfo' sysctl now lists this region of adapter
memory as 'TLS keys' if present.

330041:
Move ccr_aes_getdeckey() from ccr(4) to the cxgbe(4) driver.

This routine will also be used by the TOE module to manage TLS keys.

330079:
Move #include for rijndael.h out of x86-specific region.

The #include was added inside of the conditional by accident and the lack
of it broke non-x86 builds.

330884:
Support for TLS offload of TOE connections on T6 adapters.

The TOE engine in Chelsio T6 adapters supports offloading of TLS
encryption and TCP segmentation for offloaded connections. Sockets
using TLS are required to use a set of custom socket options to upload
RX and TX keys to the NIC and to enable RX processing. Currently
these socket options are implemented as TCP options in the vendor
specific range. A patched OpenSSL library will be made available in a
port / package for use with the TLS TOE support.

TOE sockets can either offload both transmit and reception of TLS
records or just transmit. TLS offload (both RX and TX) is enabled by
setting the dev.t6nex.<x>.tls sysctl to 1 and requires TOE to be
enabled on the relevant interface. Transmit offload can be used on
any "normal" or TLS TOE socket by using the custom socket option to
program a transmit key. This permits most TOE sockets to
transparently offload TLS when applications use a patched SSL library
(e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
library). Receive offload can only be used with TOE sockets using the
TLS mode. The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
list of TCP port numbers. Any connection with either a local or
remote port number in that list will be created as a TLS socket rather
than a plain TOE socket. Note that although this sysctl accepts an
arbitrary list of port numbers, the sysctl(8) tool is only able to set
sysctl nodes to a single value. A TLS socket will hang without
receiving data if used by an application that is not using a patched
SSL library. Thus, the tls_rx_ports node should be used with care.
For a server mostly concerned with offloading TLS transmit, this node
is not needed as plain TOE sockets will fall back to software crypto
when using an unpatched SSL library.

New per-interface statistics nodes are added giving counts of TLS
packets and payload bytes (payload bytes do not include TLS headers or
authentication tags/MACs) offloaded via the TOE engine, e.g.:

dev.cc.0.stats.rx_tls_octets: 149
dev.cc.0.stats.rx_tls_records: 13
dev.cc.0.stats.tx_tls_octets: 26501823
dev.cc.0.stats.tx_tls_records: 1620

TLS transmit work requests are constructed by a new variant of
t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.

TLS transmit work requests require a buffer containing IVs. If the
IVs are too large to fit into the work request, a separate buffer is
allocated when constructing a work request. This buffer is associated
with the transmit descriptor and freed when the descriptor is ACKed by
the adapter.

Received TLS frames use two new CPL messages. The first message is a
CPL_TLS_DATA containing the decryped payload of a single TLS record.
The handler places the mbuf containing the received payload on an
mbufq in the TOE pcb. The second message is a CPL_RX_TLS_CMP message
which includes a copy of the TLS header and indicates if there were
any errors. The handler for this message places the TLS header into
the socket buffer followed by the saved mbuf with the payload data.
Both of these handlers are contained in tom/t4_tls.c.

A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
including send_rx_credits(), a new send_rx_modulate(), and
t4_close_conn().

TLS keys for both transmit and receive are stored in onboard memory
in the NIC in the "TLS keys" memory region.

In some cases a TLS socket can hang with pending data available in the
NIC that is not delivered to the host. As a workaround, TLS sockets
are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
any data is read from a TLS socket. In addition, a fallback timer will
periodically send CPL_RX_DATA_ACK messages to the NIC for connections
that are still in the handshake phase. Once the connection has
finished the handshake and programmed RX keys via the socket option,
the timer is stopped.

A new function select_ulp_mode() is used to determine what sub-mode a
given TOE socket should use (plain TOE, DDP, or TLS). The existing
set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
handles initialization of TLS-specific state when necessary in
addition to DDP-specific state.

Since TLS sockets do not receive individual TCP segments but always
receive full TLS records, they can receive more data than is available
in the current window (e.g. if a 16k TLS record is received but the
socket buffer is itself 16k). To cope with this, just drop the window
to 0 when this happens, but track the overage and "eat" the overage as
it is read from the socket buffer not opening the window (or adding
rx_credits) for the overage bytes.

330946:
Remove TLS-related inlines from t4_tom.h to fix iw_cxgbe(4) build.

- Remove the one use of is_tls_offload() and the function. AIO special
handling only needs to be disabled when a TOE socket is actively doing
TLS offload on transmit. The TOE socket's mode (which affects receive
operation) doesn't matter, so remove the check for the socket's mode and
only check if a TOE socket has TLS transmit keys configured to determine
if an AIO write request should fall back to the normal socket handling
instead of the TOE fast path.
- Move can_tls_offload() into t4_tls.c. It is not used in critical paths,
so inlining isn't that important. Change return type to bool while here.

330947:
Fix the check for an empty send socket buffer on a TOE TLS socket.

Compare sbavail() with the cached sb_off of already-sent data instead of
always comparing with zero. This will correctly close the connection and
send the FIN if the socket buffer contains some previously-sent data but
no unsent data.

331649:
Use the offload transmit queue to set flags on TLS connections.

Requests to modify the state of TLS connections need to be sent on the
same queue as TLS record transmit requests to ensure ordering.

However, in order to use the offload transmit queue in t4_set_tcb_field(),
the function needs to be updated to do proper flow control / credit
management when queueing a request to an offload queue. This required
passing a pointer to the toepcb itself to this function, so while here
remove the 'tid' and 'iqid' parameters and obtain those values from the
toepcb in t4_set_tcb_field() itself.

333068:
Use the correct key address when renegotiating the transmit key.

Previously, get_keyid() was returning the address of the receive key
instead of the transmit key when renegotiating the transmit key. This
could either hang the card (if a connection was only offloading TLS TX
and thus had a receive key address of -1) or cause the connection to
fail by overwriting the wrong key (if both RX and TX TLS were
offloaded).

333810:
Be more robust against garbage input on a TOE TLS TX socket.

If a socket is closed or shutdown and a partial record (or what
appears to be a partial record) is waiting in the socket buffer,
discard the partial record and close the connection rather than
waiting forever for the rest of the record.

337722:
Whitespace nit in t4_tom.h

340466:
Move the TLS key map into the adapter softc so non-TOE code can use it.

340468:
Change the quantum for TLS key addresses to 32 bytes.

The addresses passed when reading and writing keys are always shifted
right by 5 as the memory locations are addressed in 32-byte chunks, so
the quantum needs to be 32, not 8.

340469:
Remove bogus roundup2() of the key programming work request header.

The key context is always placed immediately after the work request
header. The total work request length has to be rounded up by 16
however.

340473:
Restore the <sys/vmem.h> header to fix build of cxgbe(4) TOM.

vmem's are not just used for TLS memory in TOM and the #include actually
predates the TLS code so should not have been removed when the TLS vmem
moved in r340466.

Sponsored by: Chelsio Communications

MFC r329788 (by jhb@):

Bring in additional constants and message fields for TLS-related messages.

Sponsored by: Chelsio Communications

MFC 318429,318967,319721,319723,323600,323724,328353-328361,330042,343056:
Add a driver for the Chelsio T6 crypto accelerator engine.

Note that with the set of commits in this batch, no additional tunables
are needed to use the driver once it is loaded.

318429:
Add a driver for the Chelsio T6 crypto accelerator engine.

The ccr(4) driver supports use of the crypto accelerator engine on
Chelsio T6 NICs in "lookaside" mode via the opencrypto framework.

Currently, the driver supports AES-CBC, AES-CTR, AES-GCM, and AES-XTS
cipher algorithms as well as the SHA1-HMAC, SHA2-256-HMAC, SHA2-384-HMAC,
and SHA2-512-HMAC authentication algorithms. The driver also supports
chaining one of AES-CBC, AES-CTR, or AES-XTS with an authentication
algorithm for encrypt-then-authenticate operations.

Note that this driver is still under active development and testing and
may not yet be ready for production use. It does pass the tests in
tests/sys/opencrypto with the exception that the AES-GCM implementation
in the driver does not yet support requests with a zero byte payload.

To use this driver currently, the "uwire" configuration must be used
along with explicitly enabling support for lookaside crypto capabilities
in the cxgbe(4) driver. These can be done by setting the following
tunables before loading the cxgbe(4) driver:

hw.cxgbe.config_file=uwire
hw.cxgbe.cryptocaps_allowed=-1

318967:
Fail large requests with EFBIG.

The adapter firmware in general does not accept PDUs larger than 64k - 1
bytes in size. Sending crypto requests larger than this size result in
hangs or incorrect output, so reject them with EFBIG. For requests
chaining an AES cipher with an HMAC, the firmware appears to require
slightly smaller requests (around 512 bytes).

319721:
Add explicit handling for requests with an empty payload.

- For HMAC requests, construct a special input buffer to request an empty
hash result.
- For plain cipher requests and requests that chain an AES cipher with an
HMAC, fail with EINVAL if there is no cipher payload. If needed in
the future, chained requests that only contain AAD could be serviced as
HMAC-only requests.
- For GCM requests, the hardware does not support generating the tag for
an AAD-only request. Instead, complete these requests synchronously
in software on the assumption that such requests are rare.

319723:
Fix the software fallback for GCM to validate the existing tag for decrypts.

323600:
Fix some incorrect sysctl pointers for some error stats.

The bad_session, sglist_error, and process_error sysctl nodes were
returning the value of the pad_error node instead of the appropriate
error counters.

323724:
Enable support for lookaside crypto operations by default.

This permits ccr(4) to be used with the default firmware configuration
file.

328353:
Always store the IV in the immediate portion of a work request.

Combined authentication-encryption and GCM requests already stored the
IV in the immediate explicitly. This extends this behavior to block
cipher requests to work around a firmware bug. While here, simplify
the AEAD and GCM handlers to not include always-true conditions.

328354:
Always set the IV location to IV_NOP.

The firmware ignores this field in the FW_CRYPTO_LOOKASIDE_WR work
request.

328355:
Reject requests with AAD and IV larger than 511 bytes.

The T6 crypto engine's control messages only support a total AAD
length (including the prefixed IV) of 511 bytes. Reject requests with
large AAD rather than returning incorrect results.

328356:
Don't discard AAD and IV output data for AEAD requests.

The T6 can hang when processing certain AEAD requests if the request
sets a flag asking the crypto engine to discard the input IV and AAD
rather than copying them into the output buffer. The existing driver
always discards the IV and AAD as we do not need it. As a workaround,
allocate a single "dummy" buffer when the ccr driver attaches and
change all AEAD requests to write the IV and AAD to this scratch
buffer. The contents of the scratch buffer are never used (similar to
"bogus_page"), and it is ok for multiple in-flight requests to share
this dummy buffer.

328357:
Fail crypto requests when the resulting work request is too large.

Most crypto requests will not trigger this condition, but a request
with a highly-fragmented data buffer (and a resulting "large" S/G
list) could trigger it.

328358:
Clamp DSGL entries to a length of 2KB.

This works around an issue in the T6 that can result in DMA engine
stalls if an error occurs while processing a DSGL entry with a length
larger than 2KB.

328359:
Expand the software fallback for GCM to cover more cases.

- Extend ccr_gcm_soft() to handle requests with a non-empty payload.
While here, switch to allocating the GMAC context instead of placing
it on the stack since it is over 1KB in size.
- Allow ccr_gcm() to return a special error value (EMSGSIZE) which
triggers a fallback to ccr_gcm_soft(). Move the existing empty
payload check into ccr_gcm() and change a few other cases
(e.g. large AAD) to fallback to software via EMSGSIZE as well.
- Add a new 'sw_fallback' stat to count the number of requests
processed via the software fallback.

328360:
Don't read or generate an IV until all error checking is complete.

In particular, this avoids edge cases where a generated IV might be
written into the output buffer even though the request is failed with
an error.

328361:
Store IV in output buffer in GCM software fallback when requested.

Properly honor the lack of the CRD_F_IV_PRESENT flag in the GCM
software fallback case for encryption requests.

330042:
Don't overflow the ipad[] array when clearing the remainder.

After the auth key is copied into the ipad[] array, any remaining bytes
are cleared to zero (in case the key is shorter than one block size).
The full block size was used as the length of the zero rather than the
size of the remaining ipad[]. In practice this overflow was harmless as
it could only clear bytes in the following opad[] array which is
initialized with a copy of ipad[] in the next statement.

343056:
Reject new sessions if the necessary queues aren't initialized.

ccr reuses the control queue and first rx queue from the first port on
each adapter. The driver cannot send requests until those queues are
initialized. Refuse to create sessions for now if the queues aren't
ready. This is a workaround until cxgbe allocates one or more
dedicated queues for ccr.

Relnotes: yes
Sponsored by: Chelsio Communications

MFC 344671: Don't assume all children of a nexus are ports.

Specifically, ccr(4) devices are also children of cxgbe nexus devices.
Rather than making assumptions about the child device's softc, walk
the list of ports from the nexus' softc to determine if a child is a
port in t4_child_location_str(). This fixes a panic when detaching a
ccr device.

MFC 341098: Add read-only sysctls for all tunables in the cxgbe(4) driver.

MFC 330882: Simplify error handling in t4_tom.ko module loading.

- Change t4_ddp_mod_load() to return void instead of always returning
success. This avoids having to pretend to have proper support for
unloading when only part of t4_tom_mod_load() has run.
- If t4_register_uld() fails, don't invoke t4_tom_mod_unload() directly.
The module handling code in the kernel invokes MOD_UNLOAD on a module
whose MOD_LOAD fails with an error already.

MFC r342603:
cxgbe(4): Attach to two T540 variants.

Sponsored by: Chelsio Communications

MFC 340206: Treat the memory lengths for CHELSIO_T4_GET_MEM as unsigned.

Previously attempts to read the MC region were failing since the
length was greater than 2^31.

MFC 340022: Add support for port unit wiring to cxgbe(4).

- Add a bus_child_location_str method to the nexus drivers that prints
out 'port=N' as the location string exported via devinfo and the
'%location' sysctl node.

- We can't use a bus_hint_device_unit to wire the unit numbers of
devices with a fixed devclass as the device gets assigned a unit in
make_device() before the device creator can set softc, etc.
Instead, when adding a child device, use a helper function much like
a bus_hint_device_unit method to look for wiring hints or to return
-1 to let the system choose a unit number. This function requires
an "at" hint for the port pointing to the nexus device and a "port"
hint listing the port number. For example:

hint.cxl.4.at="t5nex0"
hint.cxl.4.port="0"

wires cxl4 to the first port on the t5nex0 adapter.

MFC 340021: Assert that reclaim_tx_descs() is always making forward progress.

MFC 340304: Use tcp_state_change() in the cxgbe(4) TOE module.

r254889 added tcp_state_change() as a centralized place to log state
changes in TCP connections for DTrace. r294869 and r296881 took
advantage of this central location to manage per-state counters.
However, TOE sockets were still performing some (but not all) state
change updates via direct assignments to t_state. This resulted in
state counters underflowing when TOE was in use. Fix by using
tcp_state_change() when changing a TOE connection's state.

MFC r341145

cxgbe: revert r309725

After the fix contained in r341144, cxgbe does not need anymore
to set the IFCAP_NETMAP flag manually.

Reviewed by: np
Approved by: gnn (mentor)
Differential Revision: https://reviews.freebsd.org/D17987

MFC r339639

netmap: align codebase to the current upstream (sha 8374e1a7e6941)

Changelist:
- Move large parts of VALE code to a new file and header netmap_bdg.[ch].
This is useful to reuse the code within upcoming projects.
- Improvements and bug fixes to pipes and monitors.
- Introduce nm_os_onattach(), nm_os_onenter() and nm_os_onexit() to
handle differences between FreeBSD and Linux.
- Introduce some new helper functions to handle more host rings and fake
rings (netmap_all_rings(), netmap_real_rings(), ...)
- Added new sysctl to enable/disable hw checksum in emulated netmap mode.
- nm_inject: add support for NS_MOREFRAG

Approved by: gnn (mentor)
Differential Revision: https://reviews.freebsd.org/D17364

MFC r336159:

cxgbe(4): Add a sysctl to report the chip's microprocessor's load
averages. This works with debug or custom firmwares only.

sysctl dev.<nexus>.<instance>.loadavg
sysctl dev.t6nex.0.loadavg

MFC r335352:

cxgbe(4): Some mailbox commands require access to the Tx pipeline and
can time out if it's backed up due to a non-stop deluge of PAUSE frames
from a misbehaving peer. Detect this situation and toggle MPS TxEn
to allow forward progress.

MFC r334987:

cxgbe(4): Remove homemade version of htobe32 from the driver.

It was needed only for ia64 where it was implemented as a call to
bswapXX, which was always a real function. htobeXX with a constant
argument is calculated at compile-time everywhere else.

MFC r320426:

cxgbe/t4_tom: Do not include space taken by the TCP timestamp option in
the "effective MSS" for the connection. The chip expects it this way.

MFC r338254:

cxgbe(4): Use fcmpset instead of cmpset when appropriate.

MFC r338924:

cxgbe(4): Link related changes.

- Switch to using 32b port/link capabilities in the driver. The 32b
format is used internally by firmwares > 1.16.45.0 and the driver will
now interact with the firmware in its native format, whether it's 16b
or 32b. Note that the 16b format doesn't have room for 50G, 200G, or
400G speeds.

- Add a bit in the pause_settings knobs to allow negotiated PAUSE
settings to override manual settings.

- Ensure that manual link settings persist across an administrative
down/up as well as transceiver unplug/replug.

- Remove unused is_*G_port() functions.

Sponsored by: Chelsio Communications

MFC r336042:

cxgbe(4): Assume that any unknown flash on the card is 4MB and has 64KB
sectors, instead of refusing to attach to the card.

Submitted by: Casey Leedom @ Chelsio
Sponsored by: Chelsio Communications

MFC r333139:

cxgbe(4): Destroy the cdev before disabling interrupts in driver detach.

Filter work requests are submitted in the nexus cdev's ioctl which then
blocks waiting for a reply. If driver detach runs in this state and
disables interrupts the ioctl will never complete and detach will hang
in destroy_cdev.

MFC r325840, r327811, and r329701.

r325840:
CXGBE: fix big-endian behaviour

The setbit/clearbit pair casts the bitfield pointer
to uint8_t* which effectively treats its contents as
little-endian variable. The ffs() function accepts int as
the parameter, which is big-endian. Use uint8_t here to
avoid mismatch, as we have only 4 doorbells.

Submitted by: Wojciech Macek <wma@freebsd.org>
Reviewed by: np
Obtained from: Semihalf
Sponsored by: QCM Technologies
Differential revision: https://reviews.freebsd.org/D13084

r327811:
CXGBE: fix get_filt to be endianness-aware

Unconditional 32-bit shift is not endianness-safe.
Modify the logic to work both on LE and BE.

Submitted by: Wojciech Macek <wma@freebsd.org>
Reviewed by: np
Obtained from: Semihalf
Sponsored by: IBM, QCM Technologies
Differential revision: https://reviews.freebsd.org/D13102

r329701:
CXGBE: implement prefetch on non-Intel architectures

Submitted by: Michal Stanek <mst@semihalf.com>
Obtained from: Semihalf
Reviewed by: np, pdk@semihalf.com
Sponsored by: IBM, QCM Technologies
Differential revision: https://reviews.freebsd.org/D14452

MFC r320419, r337679, r338366, and r338652.

r320419:
cxgbe/iw_cxgbe: Disable debug output by default. The help text for the sysctl
already says that the default is 0.

r337679:
Remove unused stuff from iw_cxgbe.h

r338366:
cxgbe/iw_cxgbe: Fix iWARP RDMA + VIMAGE operation by setting the VNET
properly in a couple of places in the driver.

r338652:
cxgbe/iw_cxgbe: Fix reported build breakage when the kernel
configuration has "device cxgbe' but no VIMAGE.

Sponsored by: Chelsio Communications

MFC r327254, r327904, and r328994.

r327254:
cxgbe/iw_cxgbe: Fix iWARP over VLANs (catch up with r326169).

r327904:
cxgbe/iw_cxgbe: Remove duplicates to fix compilation with recent gcc.

r328994:
iw_cxgbe: Remove declaration of a function that no longer exists.

Sponsored by: Chelsio Communications

cxgbe/iw_cxgbe: Do not call soaccept twice on the same socket.

This is a direct commit to stable/11.

Reported by: Sai Tallamraju @ Netapp
Sponsored by: Chelsio Communications

cxgbe(4): Determine early in the ioctl whether it is allowed to sleep or
not, instead of always starting a non-sleepable operation and
re-adjusting later. This ensures that an operation that is allowed to
sleep (ifconfig up/down) never fails with EBUSY on the initial attempt
to start a synchronized operation.

This is a direct commit to stable/11. The driver ioctl is always
allowed to sleep in head.

Sponsored by: Chelsio Communications

MFC r333650, r333652, r333682, r334406, r334409-r334410, and r334489.

r333650:
cxgbe(4): Claim some more T5 and T6 boards.

r333652:
cxgbe(4): Add support for two more flash parts.

r333682:
cxgbe(4): Fall back to a failsafe configuration built into the firmware
if an error is reported while pre-processing the configuration file that
the driver attempted to use.

Also, allow the user to explicitly use the built-in configuration with
hw.cxgbe.config_file="built-in"

r334406:
cxgbe(4): Consider all supported speeds when building the ifmedia list
for a port. Fix other related issues while here:
- Require port lock for access to link_config.
- Allow 100Mbps operation by tracking the speed in Mbps. Yes, really.
- New port flag to indicate that the media list is immutable. It will
be used in future refinements.

This also fixes a bug where the driver reports incorrect media with
recent firmwares.

r334409:
cxgbe(4): Implement ifm_change callback.

r334410:
cxgbe(4): Use ifm for ifmedia just like the rest of the kernel.

No functional change.

r334489:
cxgbe(4): Include full duplex mediaopt in media that can be reported as
active. Always report full duplex in active media.

Sponsored by: Chelsio Communications

MFC r331340, r331342, r331472, r332050, r333276, r333448:

r331340:
cxgbe(4): Tunnel congestion drops on a port should be cleared when the
stats for that port are cleared.

r331342:
cxgbe(4): Do not read MFG diags information from custom boards.

r331472:
cxgbe(4): Always initialize requested_speed to a valid value.

This fixes an avoidable EINVAL when the user tries to disable AN after
the port is initialized but l1cfg doesn't have a valid speed to use.

r332050:
cxgbe(4): Always display an error message if SIOCSIFFLAGS will leave
IFF_UP and IFF_DRV_RUNNING out of sync. ifhwioctl in the kernel pays no
attention to the return code from the driver ioctl during SIOCSIFFLAGS
so these messages are the only indication that the ioctl was called but
failed.

r333276:
cxgbe(4): Update all firmwares to 1.19.1.0.

r333448:
cxgbe(4): Disable write-combined doorbells by default.

This had been the default behavior but was changed accidentally as part
of the recent iw_cxgbe+OFED overhaul. Fix another bug in that change
while here: the global knob affects all the adapters in the system and
should be left alone by per-adapter code.

Approved by: re@ (marius@)
Sponsored by: Chelsio Communications

MFC r331797:

Use an accessor function to access ifr_data.

This fixes 32-bit compat (no ioctl command defintions are required
as struct ifreq is the same size).

Reviewed by: kib
Obtained from: CheriBSD
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D14900

MFC r330508:
Optimize ibcore RoCE address handle creation from user-space.

Creating a UD address handle from user-space or from the kernel-space,
when the link layer is ethernet, requires resolving the remote L3
address into a L2 address. Doing this from the kernel is easy because
the required ARP(IPv4) and ND6(IPv6) address resolving APIs are readily
available. In userspace such an interface does not exist and kernel
help is required.

It should be noted that in an IP-based GID environment, the GID itself
does not contain all the information needed to resolve the destination
IP address. For example information like VLAN ID and SCOPE ID, is not
part of the GID and must be fetched from the GID attributes. Therefore
a source GID should always be referred to as a GID index. Instead of
going through various racy steps to obtain information about the
GID attributes from user-space, this is now all done by the kernel.

This patch optimises the L3 to L2 address resolving using the existing
create address handle uverbs interface, retrieving back the L2 address
as an additional user-space information structure.

This commit combines the following Linux upstream commits:

IB/core: Let create_ah return extended response to user
IB/core: Change ib_resolve_eth_dmac to use it in create AH
IB/mlx5: Make create/destroy_ah available to userspace
IB/mlx5: Use kernel driver to help userspace create ah
IB/mlx5: Report that device has udata response in create_ah

Sponsored by: Mellanox Technologies

MFC r303505, r303506, r303512, r303513, r303646, r320418, r323082,
r326169, r326563, r326649, r326716, r326764, r326765 and r329222:

RoCE/infiniband/iWarp upgrade to Linux 4.9 for kernel and userspace.
This commit merges projects/bsd_rdma_4_9 to 11-stable.

Compatibility wrappers have been made for existing 11-stable ibcore
APIs, including ib_reg_phys_mr().
Refer to "sys/ofed/include/rdma/ib_verbs_compat.h" for more information.

The iw_cxgb driver has not been updated and has been disconnected from
the build.

Sponsored by: Mellanox Technologies

MFC r326169 and r326563:
RoCE/infiniband upgrade to Linux v4.9 for kernel and userspace.

List of kernel sources used:
============================

1) kernel sources were cloned from git://github.com/torvalds/linux.git
Top commit 69973b830859bc6529a7a0468ba0d80ee5117826 - tag: v4.9, linux-4.9

2) krping was cloned from https://github.com/larrystevenwise/krping
Top commit 292a2f1abf0348285e678a82264740d52e4dcfe4

List of userspace sources used:
===============================

1) rdma-core was cloned from https://github.com/linux-rdma/rdma-core.git
Top commit d65138ef93af30b3ea249f3a84aa6a24ba7f8a75

2) OpenSM was cloned from git://git.openfabrics.org/~halr/opensm.git
Top commit 85f841cf209f791c89a075048a907020e924528d

3) libibmad was cloned from git://git.openfabrics.org/~iraweiny/libibmad.git
Tag 1.3.13 with some additional patches from Mellanox.

4) infiniband-diags was cloned from git://git.openfabrics.org/~iraweiny/infiniband-diags.git
Tag 1.6.7 with some additional patches from Mellanox.

NOTES:
======

1) The mthca driver has been removed from userspace.
2) All GPLv2 only sources have been removed and where applicable
rewritten from scratch under a BSD license.
3) List of fully supported drivers in userspace and kernel:
a) iw_cxgbe (Chelsio)
b) mlx4ib (Mellanox)
c) mlx5ib (Mellanox)
4) WITH_OFED=YES is still required by make in order to build
OFED userspace and kernel code.
5) Full support has been added for routable RoCE, RoCE v2.

MFC r326649:
Disconnect OFED after r326169 broke all DIRDEPS support for it.

MFC r326716:
Correctly define the unordered_map namespace in ofed/libibnetdisc .

This should fix ofed/libibnetdisc compilation with C-compilers
different from clang and GCC v4.2.1.

Submitted by: kib
Sponsored by: Mellanox Technologies

MFC r326764:
ofed: Remove duplicated symbols from the version file.

ld.bfd accepts multiple listing of the same symbol in the version script.
lld is stricter and errors out. Since arm64 and sometimes amd64 use lld,
we should correct this cosmetic issue.

Sponsored by: Mellanox Technologies
Reviewed by: hselasky
Differential revision: https://reviews.freebsd.org/D13329

MFC r326765:
ofed: Define barriers for mips and arm.

I used the strongest barriers available on the architectures, so if
the future analysis show that it is excessive, the barriers could be
relaxed. Still, it is unlikely that it is meaningful to run IB on 32bit
ARM or current MIPS machines, so the change is to make WITH_OFED to pass
tinderbox.

Sponsored by: Mellanox Technologies
Reviewed by: hselasky
Differential revision: https://reviews.freebsd.org/D13329

MFC r303505:
sdp: Use an mbufq for received control packets.

This is simpler than the hand-rolled queue, and fixes a use-after-free.

Sponsored by: EMC / Isilon Storage Division

MFC r303506:
sdp: Destroy the PCB lock before freeing to the zone.

Sponsored by: EMC / Isilon Storage Division

MFC r303512:
sdp: Use malloc(9) instead of the Linux compat layer.

SDP transmit and receive rings are always created in a sleepable context,
so we can use M_WAITOK and remove error checks.

Sponsored by: EMC / Isilon Storage Division

MFC r303513:
sdp: Destroy the RDMA ID after destroying the connection's queue pair.

This is the ordering documented by rdma_destroy_qp(). Also add a useful
KASSERT to sdp_pcbfree().

Sponsored by: EMC / Isilon Storage Division

MFC r303646:
ipoib: Bound the number of egress mbufs buffered during pathrec lookups.

In pathological situations where the master subnet manager becomes
unresponsive for an extended period, we may otherwise end up queuing all
of the system's mbufs while waiting for a response to a path record lookup.

This addresses the same issue as commit 1e85b806f9 in Linux.

Reviewed by: cem, ngie
Sponsored by: EMC / Isilon Storage Division

MFC r329222:
Import the mthca kernel side infiniband driver from Linux 4.9 and fix
compilation under FreeBSD. The mthca driver was temporarily removed as
part of the Linux 4.9 RoCE/infinband upgrade.

Top commit in Linux source tree:
69973b830859bc6529a7a0468ba0d80ee5117826

Sponsored by: Mellanox Technologies

MFC r320418. Note that the socket lock _is_ the same as so_rcv's lock
in 11 and this is a no-op in this branch.

Sponsored by: Chelsio Communications

MFC r323082:
cxgbe/iw_cxgbe: Set TCP_NODELAY before initiating connection so that
t4_tom picks it up right away. This is less work than waiting for
the connection to be established before applying the setting.

Sponsored by: Chelsio Communications

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)