cxgbe(4): New knob to limit driver to the specified types of doorbells.

hw.cxgbe.doorbells_allowed="0xf"

The adapter's doorbells bitmap is clipped to the value specified in the
tunable, which is meant for debug and workarounds only. There is no
change in default behavior.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Do not issue an L1CFG command from a VF.

It is pointless to attempt an operation that is not permitted. It spams
the firmware devlog with "insufficient caps" errors that distract from
real errors.

78 2463625358 ERR CORE insufficient caps to process mailbox cmd: pfn 0x0 vfn 0x1; r_caps 0x86 wx_caps 0x82 required r_caps 0x81 w_caps 0x5

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe: Use device_set_descf()

No functional change intended.

MFC after: 1 week

cxgbe(4): Query TPCHMAP once and not once per port.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Rename rx_c_chan to rx_chan.

It is the equivalent of tx_chan but for receive so rx_chan is a better
name. Initialize both using helper functions and make sure both are
displayed in the sysctl MIB.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Minor tweaks to comments.

No functional change intended.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Initialize mps_bg_map to an invalid value if it's not known.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Allocate a taskqueue per port instead of per channel.

All the channels are not used on all boards and there's no point
allocating taskqueues that will never be used.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Reword the comment explaining the atid/cookie split.

Avoid a magic constant while here. No functional change intended.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Retire t4_intr_clear.

The firmware clears the interrupts already and it has a better idea of
exactly what to clear for which generation of the ASIC. There is no
need for the driver to get involved.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Add a helper function to locate MPS/MAC registers.

These register blocks are at different locations in different chips.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe: Support TCP_USE_DDP on offloaded TOE connections

When this socket option is enabled, relatively large contiguous
buffers are allocated and used to receive data from the remote
connection. When data is received a wrapper M_EXT mbuf is queued to
the socket's receive buffer. This reduces the length of the linked
list of received mbufs and allows consumers to consume receive data in
larger chunks.

To minimize reprogramming the page pods in the adapter, receive
buffers for a given connection are recycled. When a buffer has been
fully consumed by the receiver and freed, the buffer is placed on a
per-connection free buffers list.

The size of the receive buffers defaults to 256k and can be set via
the hw.cxgbe.toe.ddp_rcvbuf_len sysctl. The
hw.cxgbe.toe.ddp_rcvbuf_cache sysctl (defaults to 4) determines the
maximum number of free buffers cached per connection. Note that this
limit does not apply to "in-flight" receive buffers that are
associated with mbufs in the socket's receive buffer.

Co-authored-by: Navdeep Parhar <np@FreeBSD.org>
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D44001

cxgbe tom: Limit TOE connections to 2 reassembly islands always

Previously this was only limited on T6 cards to support switching from
ULP_MODE_NONE to ULP_MODE_TLS. To support switching to
ULP_MODE_TCPDDP, enable this for all adapters.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D43669

cxgbe: Add counters for POSIX async I/O requests handled by the driver

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D43668

cxgbe(4): Fix virtual interface reattach.

Replace the DOOMED flag with a transient DETACHING flag that is cleared
when VI is detached. This fixes VI reattach when only the VI and not
the parent nexus is detached. The old flag was never cleared and
prevented subsequent synch op's related to the VI.

PR: 275260
Reviewed by: jhb
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43287
Sponsored by: Chelsio Communications

cxgbe(4): Destroy the tick mutex during VI detach.

This avoids a mutex reinitialization when the VI is detached and
reattached.

Fixes: 516fe911a6b7 cxgbe(4): Always use the per-VI callout to read interface stats.
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Avoid hang on kldunload on netlink enabled kernels.

netlink(4) calls back into the driver during detach and it attempts to
start an internal synchronized op recursively, causing an interruptible
hang. Fix it by failing the ioctl if the VI has been marked as DOOMED
by cxgbe_detach.

Here's the stack for the hang for reference.
#6 begin_synchronized_op
#7 cxgbe_media_status
#8 ifmedia_ioctl
#9 cxgbe_ioctl
#10 if_ioctl
#11 get_operstate_ether
#12 get_operstate
#13 dump_iface
#14 rtnl_handle_ifevent
#15 rtnl_handle_ifnet_event
#16 rt_ifmsg
#17 if_unroute
#18 if_down
#19 if_detach_internal
#20 if_detach
#21 ether_ifdetach
#22 cxgbe_vi_detach
#23 cxgbe_detach
#24 DEVICE_DETACH

MFC after: 3 days
Sponsored by: Chelsio Communications

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

ddb: Rework macros to make it easier to add new command tables.

- Add new DB_DEFINE_TABLE and DB_DECLARE_TABLE macros to define new
command tables. DB_DECLARE_TABLE is intended for use in headers
similar to MALLOC_DECLARE and SYSCTL_DECL.

DB_DEFINE_TABLE takes three arguments, the name of the parent table,
the command name, and the name of the table itself, e.g.
DB_DEFINE_TABLE(show, foo, show_foo) defines a new "show foo" table.

- DB_TABLE_COMMAND, DB_TABLE_COMMAND_FLAGS, DB_TABLE_ALIAS, and
DB_ALIAS_FLAGS allow new commands and aliases to be defined. These
are similar to the existing DB_COMMAND, etc. except that they take
an initial argument giving the name of the parent table, e.g.:

DB_TABLE_COMMAND(show_foo, bar, db_show_foo_bar)

defines a new "show foo bar" command.

This provides a cleaner interface than the ad-hoc use of internal
macros like _DB_SET that was required previously (e.g. in cxgbe(4)).

This retires DB_FUNC macro as well as the internal _DB_FUNC macro.

Reviewed by: melifaro, kib, markj
Differential Revision: https://reviews.freebsd.org/D40819

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

cxgbe(4): Dump the firmware log before falling back to a minimal config.

It might have errors that explain why the attempted configuration
failed.

MFC after: 1 week
Sponsored by: Chelsio Communications

Mechanically convert cxgb(4) and cxgbe(4) to IfAPI

Reviewed by: np
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D38597

cxgbe: Allow parse_pkt to internally queue a packet.

If parse_pkt returns EINPROGRESS, return from cxgbe_transmit
without queueing the packet in a txq. Use this to move the call
to ethofld_transmit for packet pacing into parse_pkt.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D38577

cxgbe: fix module loading

After c03414326909, if_cxgbe.ko fails to load if crypto is not compiled
in kernel, e.g., MINIMAL.

link_elf_obj: symbol hmac_init_ipad undefined
linker_load_file: /boot/kernel/if_cxgbe.ko - unsupported file type
kldload: an error occurred while loading module if_cxgbe. Please check dmesg(8) for more details.

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D38482

cxgbe: Enable TOE TLS RX when an RX key is provided via setsockopt().

Rather than requiring a socket to be created as a TLS socket from the
get go, switch a TOE socket from "plain" TOE to TLS mode when a
receive key is added to the socket.

The firmware is only able to switch a "plain" TOE connection to TLS
mode if the head of the pending socket data is the start of a TLS
record, so the connection is migrated to TLS mode as a multi-step
process.

When TOE TLS RX is enabled, the associated connection's receive side
is frozen via a flag in the TCB. The state of the socket buffer is
then examined to determine if the pending data in the socket buffer
ends on a TLS record boundary. If so, the connection is migrated to
TLS mode and unfrozen. Otherwise, the connection is unfrozen
temporarily until more data arrives. Once more data arrives, the
receive queue is frozen again and rechecked. This continues until the
connection is paused at a record boundary. Any records received
before TLS mode is enabled are decrypted as software records.

Note that this removes the 'rx_tls_ports' sysctl. TOE TLS offload for
receive is now enabled automatically on existing TOE connections when
using a KTLS-aware SSL library just as it was previously enabled
automatically for TLS transmit. This also enables TLS offload for TOE
connections which enable TLS after passing initial data in the clear
(e.g. STARTTLS with SMTP).

Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D37351

cxgbe: Various whitespace fixes.

Mostly trailing whitespace and spaces before tabs.

Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D37350

cxgbe: Use secq(9) to manage the timestamp generations.

This is mostly cosmetic, but it also doesn't leave a gap of time where
no structures are valid. Instead, we permit the ISR to continue to
use the previous structure if the write to update cal_current isn't
yet visible.

Reviewed by: gallatin
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D36669

cxgbe: Compute timestamps via sbintime_t.

This uses fixed-point math already used elsewhere in the kernel for
sub-second time values. To avoid overflows this does require updating
the calibration once a second rather than once every 30 seconds. Note
that the cxgbe driver already queries multiple registers once a second
for the statistics timers. This version also uses fewer instructions
with no branches (for the math portion) in the per-packet fast path.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D36663

cxgbe: fix kernel build without TCP_OFFLOAD or RATELIMIT

Reviewed by: np
Fixes: e398922eaf6
Sponsored by: Nvidia networking
Differential revision: https://reviews.freebsd.org/D36648

cxgbe: fix the build after e398922eaf66978b5e556f6b4b095693c865f329

Enable M_TSTMP in Chelsio cxgbe driver by creating a mechanism that can sync the time.

Chelsio has always been recording a timestamp in the mbuf (rcv_tstmp) but
not setting the M_TSTMP bit in the mbuf flags. This is because the timestamp
was just the free running 60bit clock. This change fixes that so that
we keep a synchronization by periodically (every 30 seconds after startup)
getting the timestamp and the current nanosecond time. We always keep
several sets around and the current one we always keep the current pair
and the previous pair of timestamps. This allows us to setup a ratio
between the two so we can correctly translate the time. Note that
we use special care to split the timestamp into seconds (per the clock tick)
and nanoseconds otherwise 64bit math would overflow.

Reviewed by: np
Sponsored by: Netflix Inc
Differential Revision: https://reviews.freebsd.org/D36315

cxgbe(4): Add a knob to request that clocks be gated on suspend.

MFC after: 3 months
Sponsored by: Chelsio Communications

cxgbe: Rename t4_kern_tls.c to t6_kern_tls.c.

This implementation of NIC TLS is specific to T6 adapters.

Sponsored by: Chelsio Communications

ddb: namespacing of struct command

'command' is too generic for something specific to the kernel debugger;
change this so it is less likely to collide with local variable names.
Also rename struct command_table to struct db_command_table.

Reviewed by: markj
MFC after: 1 week
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D35367

cxgbe: Deactivate upper layer drivers (like TOE) during detach.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D35237

cxgbe: Remove unused devclass arguments to *DRIVER_MODULE().

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D34964

cxgbe(4): Add support for PPOD_EDRAM feature.

The driver queries the firmware to find out if it supports this feature
and enables it if it does. The firmware moves the iSCSI page pod region
to a lower address so that some of it is located in the faster on-chip
memory instead of external DDR.

Reviewed by: jhb@
MFC after: 3 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D34895

cxgbe(4): Fix control flow issues reported by Coverity.

CID 1487932: Control flow issues (NESTING_INDENT_MISMATCH).
The macro on this line expands into multiple statements, only the first
of which is nested within the preceding parent while the rest are not.
9828 ulp_region(RX_TLS_KEY);

Reported by: Coverity (CID 1487932)
Fixes: f88b31885c4 cxgbe(4): meminfo should get the TLS region's limits from the hardware.
MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Display the number of free PM pages and p-structs in meminfo.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): meminfo should get the TLS region's limits from the hardware.

meminfo is meant for debugging and it should read hardware configuration
directly when possible instead of relying on values queried from the
firmware. Rename the region to "TLSKey region" to match other drivers
while here.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Fix the sorting order of the regions listed in meminfo.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Handle FORCE_FEC in pcaps correctly.

The firmware doesn't report FORCE_FEC in pcaps if the transceiver
plugged in at that time does not support a speed that may use FEC. It
is incorrect for the driver to assume that the FORCE_FEC value it read
during attach (in init_link_config) is permanent. Instead, it should
check pcaps just before issuing the L1CFG command.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Allow dump_cimla and dump_devlog to sleep.

This has been safe since e9e7bc82505, which moved parts of error
handling from the ithread to a taskqueue.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): dump_devlog should never fail silently.

Do the same thing as dump_cimla and log a warning on failure.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Dump some more debug registers in cim_dump_regs.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Changes to the fatal error handler.

* New error_flags that can be used from the error ithread and elsewhere
without a synch_op.
* Stop the adapter immediately in t4_fatal_err but defer most of the
rest of the handling to a task. The task is allowed to sleep, unlike
the ithread. Remove async_event_task as it is no longer needed.
* Dump the devlog, CIMLA, and PCIE_FW exactly once on any fatal error
involving the firmware or the CIM block. While here, dump some
additional info (see dump_cim_regs) for these errors.
* If both reset_on_fatal_err and panic_on_fatal_err are set then attempt
a reset first and do not panic the system if it is successful.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Fix illegal hardware access in cxgbe_refresh_stats.

cxgbe_refresh_stats takes into account VI_SKIP_STATS but not
VI_INIT_DONE when deciding whether to read the hardware stats. But
before this change VI_SKIP_STATS was set only for VIs with VI_INIT_DONE.
That meant that cxgbe_refresh_stats always accessed the hardware for
uninitialized VIs, and this is a problem if the adapter is suspended or
in the middle of a reset.

Fix this by setting VI_SKIP_STATS on all VIs during suspend. While
here, ignore VI_INIT_DONE in vi_refresh_stats too to be consistent with
cxgbe_refresh_stats.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Avoid unsafe hardware access in the ifmedia ioctls.

The hardware is unavailable when the device is suspended or in the
middle of a reset.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Fix bad races between sysctl and driver detach.

The default sysctl context setup by newbus for a device is eventually
freed by device_sysctl_fini, which runs after the device driver's detach
routine. sysctl nodes associated with this context must not use any
resources (like driver locks, hardware access, counters, etc.) that are
released by driver detach.

There are a lot of sysctl nodes like this in cxgbe(4) and the fix is to
hang them off a context that is explicitly freed by the driver before it
releases any resource that might be used by a sysctl.

This fixes panics when running "sysctl dev.t6nex dev.cc" in a tight loop
and loading/unloading the driver in parallel.

Reported by: Suhas Lokesha
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Do not ignore the return value of ifmedia_ioctl.

This ensures that the driver reports an error instead of failing
silently when an invalid media is requested.

Reported by: Suhas Lokesha @ Chelsio
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Fix "set but not used [-Wunused-but-set-variable]" warnings.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Fix stats collection for ports with port_id != tx_chan

This fixes a driver panic during stats collection when a port's id does
not match its tx channel. The bug affected only the T580 card running
with a non-default VPD.

Reported by: Suhas Lokesha @ Chelsio
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Change the way t4_shutdown_adapter brings the link(s) down.

Modify the GPIO pins only on the Base-T cards and even there drive all
of them low instead of putting them in hi-z state. For the rest (this
is the common case), directly power off the PLLs of the high speed
serdes. This is the simplest method that does not involve or conflict
with the firmware but still works with all T4-T6 cards regardless of
what's plugged into the port.

This fixes a problem where the peer wouldn't always see a link down if
it is connected to the device using a -CR4 copper cable.

MFC after: 3 weeks
Sponsored by: Chelsio Communications

cxgbe(4): internal knob for flexible control over FEC selection.

Recent firmwares have support for autonomous FEC selection and a "force"
knob to let the driver control this behavior (or not) in a fine grained
manner. This change adds a driver knob so that all the different ways of
configuring the link FEC can be exercised. Note that this controls the
internal driver/firmware interaction for link configuration and is not
meant for general use.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): separate sysctls for user-requested and in-use FEC.

Recent firmwares have more leeway in FEC selection and there is a need
to track the FECs requested by the driver separately from the FEC in use
on the link. The existing dev.<port>.<inst>.fec sysctl can read both but
its behavior depends on the link state and it is sometimes hard to find
out what was requested when the link is up.

Split the fec sysctl into two (requested_fec and link_fec) to get access
to both pieces of information regardless of the link state.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): sysctl to track the last L1_CFG32 requested by the driver.

dev.<port>.<inst>.rcaps

# sysctl dev.cc | grep rcaps
dev.cc.1.rcaps: 581107776
dev.cc.0.rcaps: 582156414

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe: Only run ktls_tick when NIC TLS is enabled.

Previously the body of ktls_tick was a nop when NIC TLS was disabled,
but the callout was still scheduled consuming power on otherwise-idle
systems with Chelsio T6 adapters. Now the callout only runs while NIC
TLS is enabled on at least one interface of an adapter.

Reported by: mav
Reviewed by: np, mav
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D32491

Add a switch structure for send tags.

Move the type and function pointers for operations on existing send
tags (modify, query, next, free) out of 'struct ifnet' and into a new
'struct if_snd_tag_sw'. A pointer to this structure is added to the
generic part of send tags and is initialized by m_snd_tag_init()
(which now accepts a switch structure as a new argument in place of
the type).

Previously, device driver ifnet methods switched on the type to call
type-specific functions. Now, those type-specific functions are saved
in the switch structure and invoked directly. In addition, this more
gracefully permits multiple implementations of the same tag within a
driver. In particular, NIC TLS for future Chelsio adapters will use a
different implementation than the existing NIC TLS support for T6
adapters.

Reviewed by: gallatin, hselasky, kib (older version)
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D31572

cxgbe(4): Fix the decode and display of the DBVFIFO region in meminfo.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Display HMA information in meminfo.

This should have been added with initial T6 support many years ago.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbei: Support for ISO (iSCSI segmentation offload).

ISO can be disabled before establishing a connection by setting
dev.tNnex.N.toe.iso to 0.

Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D31223

cxgbe(4): Do not configure traffic classes automatically on attach.

The driver used to configure all available classes with some default
parameters on attach and the rest of t4_sched.c was written with the
assumption that all traffic classes are always valid in the hardware.
But this resulted in a lot of informational messages being logged in the
firmware's circular log, crowding out other more useful messages.

This change leaves the tx scheduler alone during attach to reduce the
spam in the devlog. The state of every class is now tracked separately
from its flags and there is support for an 'uninitialized' state.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Get the number of usable traffic classes from the firmware.

Recent firmwares are able to utilize the traffic classes of tx channels
that were previously unused. This effectively doubles the number of
traffic classes available per port for 2 port cards. Stop using the raw
per-channel value in the driver and ask the firmware for the number of
usable traffic classes instead.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

newbus: Move from bus_child_{pnpinfo,location}_src to bus_child_{pnpinfo,location} with sbuf

Now that the upper layers all go through a layer to tie into these
information functions that translates an sbuf into char * and len. The
current interface suffers issues of what to do in cases of truncation,
etc. Instead, migrate all these functions to using struct sbuf and these
issues go away. The caller is also in charge of any memory allocation
and/or expansion that's needed during this process.

Create a bus_generic_child_{pnpinfo,location} and make it default. It
just returns success. This is for those busses that have no information
for these items. Migrate the now-empty routines to using this as
appropriate.

Document these new interfaces with man pages, and oversight from before.

Reviewed by: jhb, bcr
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D29937

cxgbe(4): Check if the firmware supports 512 SGL per FR MR.

Firmwares >= 1.25.6.0 support 512 SGL entries in a single memory
registration request.

Obtained from: Chelsio Communications
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Overhaul CLIP (Compressed Local IPv6) table management.

- Process the list of local IPs once instead of once per adapter. Add
addresses from all VNETs to the driver's list but leave hardware
updates for later when the global VNET/IFADDR list locks have been
released.

- Add address to the hardware table synchronously when a CLIP entry is
requested for an address that's not already in there.

- Provide ioctls that allow userspace tools to manage addresses in the
CLIP table.

- Add a knob (hw.cxgbe.clip_db_auto) that controls whether local IPs are
automatically added to the CLIP table or not.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe: Make the TOE ISCSI RX stats per-queue instead of per adapter.

Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D29903

cxgbe(4): Add support for NIC suspend/resume and live reset.

Add suspend/resume callbacks to the driver and a live reset built around
them. This commit covers the basic NIC and future commits will expand
this functionality to other stateful parts of the chip. Suspend and
resume operate on the chip (the t?nex nexus device) and affect all its
ports. It is not possible to suspend/resume or reset individual ports.
All these operations can be performed on a running NIC. A reset will
look like a link bounce to the networking stack.

Here are some ways to exercise this functionality:

/* Manual suspend and resume. */
# devctl suspend t6nex0
# devctl resume t6nex0

/* Manual reset. */
# devctl reset t6nex0

/* Manual reset with driver sysctl. */
# sysctl dev.t6nex.0.reset=1

/* Automatic adapter reset on any fatal error. */
# hw.cxgbe.reset_on_fatal_err=1

Suspend disables the adapter (DMA, interrupts, and the port PHYs) and
marks the hardware as unavailable to the driver. All ifnets associated
with the adapter are still visible to the kernel but operations that
require hardware interaction will fail with ENXIO. All ifnets report
link-down while the adapter is suspended.

Resume will reattach to the card, reconfigure it as before, and recreate
the queues servicing the existing ifnets. The ifnets are able to send
and receive traffic as soon as the link comes back up.

Reset is roughly the same as a suspend and a resume with at least one of
these events in between: D0->D3Hot->D0, FLR, PCIe link retrain.

MFC after: 1 month
Relnotes: yes
Sponsored by: Chelsio Communications

cxgbe(4): Separate the sw- and hw-specific parts of resource allocations

The driver uses both software resources (locks, callouts, memory for
descriptors and for bookkeeping, sysctls, etc.) and hardware resources
(VIs, DMA queues, TCAM entries, etc.) to operate the NIC. This commit
splits the single *_ALLOCATED flag used to track all these resources
into separate *_SW_ALLOCATED and *_HW_ALLOCATED flags.

This is the simplified pseudocode that now applies to most queues (foo
can be ctrlq/txq/rxq/ofld_txq/ofld_rxq):

/* Idempotent */
alloc_foo
{
if (!SW_ALLOCATED)
init_iq/init_eq/init_fl no-fail sw init
alloc_iq_fl/alloc_eq/alloc_wrq may-fail sw alloc
add_foo_sysctls, etc. no-fail post-alloc items
if (!HW_ALLOCATED)
alloc_iq_fl_hwq/alloc_eq_hwq hw resource allocation
}

/* Idempotent */
free_foo
{
if (!HW_ALLOCATED)
free_iq_fl_hwq/free_eq_hwq release hw resources
if (!SW_ALLOCATED)
free_iq_fl/free_eq/free_wrq release sw resources
}

The routines that take the driver to FULL_INIT_DONE and VI_INIT_DONE and
back are now all idempotent. The quiesce routines pay attention to the
HW_ALLOCATED flag and will not wait on the hardware for pidx/cidx
updates and other completions if this flag is not set.

MFC after: 1 month
Sponsored by: Chelsio Communications

cxgbe(4): hw.cxgbe.panic_on_fatal_err can be changed any time.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Move the hw-specific parts of VXLAN setup to a separate function.

It can be called to (re)apply the settings in the driver softc to the
hardware.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Add flag to reliably stop the driver from accessing hw stats.

There are two kinds of routines in the driver that read statistics from
the hardware: the cxgbe_* variants read the per-port MPS/MAC registers
and the vi_* variants read the per-VI registers. They can be called
from the 1Hz callout or if_get_counter. All stats collection now takes
place under the callout lock and there is a new flag to indicate that
these routines should not access any hardware register.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Fix minor nit in the display of MPS TCAM entries.

MFC after: 3 days

cxgbe/t4_tom: Implement tod_pmtu_update.

tod_pmtu_update was added to the kernel in 01d74fe1ffc.

Sponsored by: Chelsio Communications

cxgbe: Add counters for iSCSI PDUs transmitted via TOE.

Reviewed by: np
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D29297

cxgbe(4): Always use the per-VI callout to read interface stats.

There is no change in the source of the stats (t4_get_port_stats or
t4_get_vi_stats) but the per-port callout is gone.

Sponsored by: Chelsio Communications
Reviewed by: jhb@
Differential Revision: https://reviews.freebsd.org/D29527

cxgbe: Make the TOE TLS stats per-queue instead of per-port.

This avoids some atomics by using counter_u64 for TX and relying on
existing single-threading (single ithread per rxq) for RX.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D29383

cxgbe: Add a struct sge_ofld_txq type.

This type mirrors struct sge_ofld_rxq and holds state for TCP offload
transmit queues. Currently it only holds a work queue but will
include additional state in future changes.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D29382

cxgbe(4): Allow a T6 adapter to switch between TOE and NIC TLS mode.

The hw.cxgbe.kern_tls tunable was used for this in the past and if it
was set then all T6 adapters would be configured for NIC TLS operation
and could not be reconfigured for TOE without a reload. With this
change ifconfig can be used to manipulate toe and txtls caps like any
other caps. hw.cxgbe.kern_tls continues to work as usual but its
effects are not permanent any more.

* Enable nic_ktls_ofld in the default configuration file and use the
firmware instead of direct register manipulation to apply/rollback
NIC TLS configuration. This allows the driver to switch the hardware
between TOE and NIC TLS mode in a safe manner. Note that the
configuration is adapter-wide and not per-port.

* Remove the kern_tls config file as it works with 100G T6 cards only
and leads to firmware crashes with 25G cards. The configurations
included with the driver (with the exception of the FPGA configs) are
supposed to work with all adapters.

Reported by: Veeresh U.K. at Chelsio
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Reviewed by: jhb@
Differential Revision: https://reviews.freebsd.org/D29291

cxgbe(4): create a separate helper routine to write the global RSS key.

While here, make sure only the PF driver attempts to program the global
RSS key (with options RSS). The VF driver doesn't have access to those
device registers.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): make it safe to call setup_memwin repeatedly.

A repeat call will recreate the memory windows in the hardware and move
them to their last-known positions without repeating any of the software
initialization.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): catch up with the latest cryptocaps.

There are two crypto capabilities that the driver didn't know about.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Fix an assertion that is not valid during attach.

Firmware access from t4_attach takes place without any synchronization.
The driver should not panic (debug kernels) if something goes wrong in
early communication with the firmware. It should still load so that
it's possible to poke around with cxgbetool.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Read the rx 'c' channel for a port and make it available.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Add a driver ioctl to set the filter mask.

Allow the filter mask (aka the hashfilter mode when hashfilters are
in use) to be set any time it is safe to do so. The requested mask
must be a subset of the filter mode already. The driver will not change
the mode or ingress config just to support a new mask.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Break up t4_read_chip_settings.

Read the PF-only hardware settings directly in get_params__post_init.
Split the rest into two routines used by both the PF and VF drivers: one
that reads the SGE rx buffer configuration and another that verifies
miscellaneous hardware configuration.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Fixes to tx coalescing.

- The behavior implemented in r362905 resulted in delayed transmission
of packets in some cases, causing performance issues. Use a different
heuristic to predict tx requests.

- Add a tunable/sysctl (hw.cxgbe.tx_coalesce) to disable tx coalescing
entirely. It can be changed at any time. There is no change in
default behavior.

Catch up with 6edfd179c86: mechanically rename IFCAP_NOMAP to IFCAP_MEXTPG.

Originally IFCAP_NOMAP meant that the mbuf has external storage pointer
that points to unmapped address. Then, this was extended to array of
such pointers. Then, such mbufs were augmented with header/trailer.
Basically, extended mbufs are extended, and set of features is subject
to change. The new name should be generic enough to avoid further
renaming.

Clear TLS offload mode if a TLS socket hangs without receiving data.

By default, if a TOE TLS socket stops receiving data for more than 5
seconds, revert the connection back to plain TOE mode. This provides
a fallback if the userland SSL library does not support KTLS. In
addition, for client TLS 1.3 sockets using connect(), the TOE socket
blocks before the handshake has completed since the socket option is
only invoked for the final handshake.

The timeout defaults to 5 seconds, but can be changed at boot via the
hw.cxgbe.toe.tls_rx_timeout tunable or for an individual interface via
the dev.<nexus>.toe.tls_rx_timeout sysctl.

Reviewed by: np
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27470

cxgbe(4): Fix vertical alignment in sysctl_cpl_stats.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): two new debug sysctls.

dev.<nexus>.<instance>.misc.tid_stats
dev.<nexus>.<instance>.misc.tnl_stats

MFC after: 3 days
Sponsored by: Chelsio Communications

Fix downgrading of TOE TLS sockets to plain TOE.

If a TOE TLS socket ends up using an unsupported TLS version or
ciphersuite, it must be downgraded to a "plain" TOE socket with TLS
encryption/decryption performed on the host. The previous
implementation of this fallback was incomplete and resulted in hung
connections.

Reviewed by: np
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27467

cxgbev(4): Make sure that the iq/eq map sizes are correct for VFs.

This should have been part of r366929.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): fix the size of the iq/eq maps.

The firmware can allocate ingress and egress context ids anywhere from
its configured range. Size the iq/eq maps to match the entire range
instead of assuming that the firmware always allocates the first
available context id.

Reported by: Baptiste Wicht @ Verisign
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): display correct tid range for T6 based -SO cards.

Reported by: Chelsio QA
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Updates to the drop features from r366532.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): More fixes for the T6 FCS error counter.

r365732 was the first attempt to get an accurate count but it was
writing to some read-only registers to clear them and that obviously
didn't work. Instead, note the counter's value when it is supposed to
be cleared and subtract it from future readings.

dev.<port>.stats.rx_fcs_error should not be serviced from the MPS
register for T6.

The stats.* sysctls should all use T5_PORT_REG for T5 and above. This
must have been missed in the initial T5 support years ago. Fix it while
here.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): knobs to drop various kinds of undesirable frames on ingress.

These kind of drops come for free in the sense that they do not use the
filter TCAM or any other resource that wouldn't normally be used during
rx. Frames dropped by the hardware get counted in the MAC's rx stats
but are not delivered to the driver.

hw.cxgbe.attack_filter
Set to 1 to enable the "attack filter". Default is 0. The attack
filter will drop an incoming frame if any of these conditions is true:
src ip/ip6 == dst ip/ip6; tcp and src/dst ip is not unicast; src/dst ip
is loopback (127.x.y.z); src ip6 is not unicast; src/dst ip6 is loopback
(::1/128) or unspecified (::/128); tcp and src/dst ip6 is mcast
(ff00::/8).

hw.cxgbe.drop_ip_fragments
Set to 1 to drop all incoming IP fragments. Default is 0. Note that
this drops valid frames.

hw.cxgbe.drop_pkts_with_l2_errors
Set to 1 to drop incoming frames with Layer 2 length or checksum errors.
Default is 1.

hw.cxgbe.drop_pkts_with_l3_errors
Set to 1 to drop incoming frames with IP version, length, or checksum
errors. Default is 0.

hw.cxgbe.drop_pkts_with_l4_errors
Set to 1 to drop incoming frames with Layer 4 length, checksum, or other
errors. Default is 0.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

Store the send tag type in the common send tag header.

Both cxgbe(4) and mlx5(4) wrapped the existing send tag header with
their own identical headers that stored the type that the
type-specific tag structures inherited from, so in practice it seems
drivers need this in the tag anyway. This permits removing these
extra header indirections (struct cxgbe_snd_tag and struct
mlx5e_snd_tag).

In addition, this permits driver-independent code to query the type of
a tag, e.g. to know what type of tag is being queried via
if_snd_query.

Reviewed by: gallatin, hselasky, np, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D26689

cxgbe(4) sysctls do not need Giant.

Sponsored by: Chelsio Communications

cxgbe(4): let the PF driver use VM work requests for transmit.

This allows the PF interfaces to communicate with the VF interfaces over
the internal switch in the ASIC. Fix the GL limits for VM work requests
while here.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): add support for stateless offloads for VXLAN traffic.

Hardware assistance includes checksumming (tx and rx), TSO, and RSS on
the inner traffic in a VXLAN tunnel.

Relnotes: Yes
Sponsored by: Chelsio Communications

cxgbe(4): Use large clusters for TOE rx queues when TOE+TLS is enabled.

Rx is more efficient within the chip when the receive buffer size
matches the TLS PDU size.

MFC after: 3 days
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D26127

cxgbe(4): Stop checking for failures from malloc(M_WAITOK).

PR: 240545
Submitted by: Andrew Reiter <arr@watson.org>
Reviewed by: np
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D25767

cxgbe(4): changes in the Tx path to help increase tx coalescing.

- Ask the firmware for the number of frames that can be stuffed in one
work request.

- Modify mp_ring to increase the likelihood of tx coalescing when there
are just one or two threads that are doing most of the tx. Add teeth
to the abdication mechanism by pushing the consumer lock into mp_ring.
This reduces the likelihood that a consumer will get stuck with all
the work even though it is above its budget.

- Add support for coalesced tx WR to the VF driver. This, with the
changes above, results in a 7x improvement in the tx pps of the VF
driver for some common cases. The firmware vets the L2 headers
submitted by the VF driver and it's a big win if the checks are
performed for a batch of packets and not each one individually.

Reviewed by: jhb@
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D25454

cxgbe(4): Add a pointer to the adapter softc in vi_info.

There were quite a few places where port_info was being accessed only to
get to the adapter.

Reviewed by: jhb@
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D25432

Avoid trying to toggle TSO twice

Remove TSO from the toggle mask when automatically disabled by TXCKSUM* in
various NIC drivers.

Reviewed by: hselasky, np, gallatin, jpaetzel
Approved by: mav (mentor)
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25120

cxgbe/iw_cxgbe: Add an async callback to notify iw_cxgbe in case of a
fatal error.

Submitted by: Krishnamraju Eraparaju @ Chelsio
MFC after: 2 weeks
Sponsored by: Chelsio Communications

Rename TOE TLS stats from [rt]x_tls_* to [rt]x_toe_tls_*.

This more clearly differentiates TLS records encrypted and decrypted
in TOE connections from those encrypted via NIC TLS.

MFC after: 1 week
Sponsored by: Chelsio Communications

Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)

r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.

This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.

Mark all obvious cases as MPSAFE. All entries that haven't been marked
as MPSAFE before are by default marked as NEEDGIANT

Approved by: kib (mentor, blanket)
Commented by: kib, gallatin, melifaro
Differential Revision: https://reviews.freebsd.org/D23718

cxgbe(4): Congestion drops are maintained per E-channel and not per
buffer group.

This fixes a bug where congestion drops on port 1 of a T6 card would
incorrectly be counted as drops on port 0.

MFC after: 1 week
Sponsored by: Chelsio Communications

Remove the per-TXQ tls_wrs stat.

It duplicated the kern_tls_records stat and was not conditional on NIC
TLS being enabled.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D23670

cxgbe(4): Add native netmap support to the main interface.

This means that extra virtual interfaces (VIs) created with
hw.cxgbe.num_vis are no longer required to use netmap. Use this
tunable to enable native netmap support on the main interface:

hw.cxgbe.native_netmap="3"

There is no change in default behavior.

Suggested by: jch@
MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Add pfil(9) hooks to the driver's rx.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Retire the allow_mbufs_in_cluster optimization.

This simplifies the driver's rx fast path as well as the bookkeeping
code that tracks various rx buffer sizes and layouts.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): check if the firmware supports FW_RI_FR_NSMR_TPTE_WR work
request.

This is used by iw_cxgbe to figure out how best to register memory.

MFC after: 1 month
Sponsored by: Chelsio Communications

cxgbe(4): Simplify the firmware version checks a bit.

No functional change.

MFC after: 1 week

cxgbe(4): Allow the driver to specify multiple FECs that the firmware
should try in order to link up with the peer.

Various FEC variables within the driver can now have multiple bits set
instead of being powers of 2. 0 and -1 in the user knobs still mean no
FEC and auto (driver decides) respectively for backward compatibility,
but no-FEC and auto now have their own bits in the internal
representation. There is a new bit that can be set to request the FEC
recommended by the cable/transceiver module.

Add sysctls to display link related capabilities of the local side as
well as the link partner.

Note that all this needs a new firmware and the documentation for the
driver FEC knobs will be updated after that firmware is added to the
driver.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): sysctl to reset the temperature/voltage sensor.

# sysctl dev.<nexus>.<inst>.reset_sensor=1
# sysctl dev.t6nex.0.reset_sensor=1

MFC after: 1 week
Sponsored by: Chelsio Communications

NIC KTLS for Chelsio T6 adapters.

This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.

NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment
the encrypted TLS frames output by the crypto engine. Instead, the
TOE is placed into a special setup to permit "dummy" connections to be
associated with regular sockets using KTLS. This permits using the
TOE to segment the encrypted TLS records. However, this approach does
have some limitations:

1) Regular TOE sockets cannot be used when the TOE is in this special
mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but
not both at the same time.

2) In NIC KTLS mode, the TOE is only able to accept a per-connection
timestamp offset that varies in the upper 4 bits. Put another way,
only connections whose timestamp offset has the 28 lower bits
cleared can use NIC KTLS and generate correct timestamps. The
driver will refuse to enable NIC KTLS on connections with a
timestamp offset with any of the lower 28 bits set. To use NIC
KTLS, users can either disable TCP timestamps by setting the
net.inet.tcp.rfc1323 sysctl to 0, or apply a local patch to the
tcp_new_ts_offset() function to clear the lower 28 bits of the
generated offset.

3) Because the TCP segmentation relies on fields mirrored in a TCB in
the TOE, not all fields in a TCP packet can be sent in the TCP
segments generated from a TLS record. Specifically, for packets
containing TCP options other than timestamps, the driver will
inject an "empty" TCP packet holding the requested options (e.g. a
SACK scoreboard) along with the segments from the TLS record.
These empty TCP packets are counted by the
dev.cc.N.txq.M.kern_tls_options sysctls.

Unlike TOE TLS which is able to buffer encrypted TLS records in
on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS
records for retransmit requests as well as non-retransmit requests
that do not include the start of a TLS record but do include the
trailer. The T6 NIC KTLS code tries to optimize some of the cases for
requests to transmit partial TLS records. In particular it attempts
to minimize sending "waste" bytes that have to be given as input to
the crypto engine but are not needed on the wire to satisfy mbufs sent
from the TCP stack down to the driver.

TCP packets for TLS requests are broken down into the following
classes (with associated counters):

- Mbufs that send an entire TLS record in full do not have any waste
bytes (dev.cc.N.txq.M.kern_tls_full).

- Mbufs that send a short TLS record that ends before the end of the
trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC,
the encryption must always start at the beginning, so if the mbuf
starts at an offset into the TLS record, the offset bytes will be
"waste" bytes. For sockets using AES-GCM, the encryption can start
at the 16 byte block before the starting offset capping the waste at
15 bytes.

- Mbufs that send a partial TLS record that has a non-zero starting
offset but ends at the end of the trailer
(dev.cc.N.txq.M.kern_tls_partial). In order to compute the
authentication hash stored in the trailer, the entire TLS record
must be sent as input to the crypto engine, so the bytes before the
offset are always "waste" bytes.

In addition, other per-txq sysctls are provided:

- dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq
using AES-CBC.

- dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq
using AES-GCM.

- dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to
compensate for the TOE engine not being able to set FIN on the last
segment of a TLS record if the TLS record mbuf had FIN set.

- dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this
txq including full, short, and partial records.

- dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header
and payload) sent for TLS record requests.

- dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS
record requests.

To enable NIC KTLS with T6, set the following tunables prior to
loading the cxgbe(4) driver:

hw.cxgbe.config_file=kern_tls
hw.cxgbe.kern_tls=1

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21962

Fix regression from r353841: ctx.rc needs to be initialized,
otherwise driver might silently fail to initialize.

Pointy hat to: glebius

Create a file to hold shared routines for dealing with T6 key contexts.

ccr(4) and TLS support in cxgbe(4) construct key contexts used by the
crypto engine in the T6. This consolidates some duplicated code for
helper functions used to build key contexts.

Reviewed by: np
MFC after: 1 month
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D22156

cxgbe(4): Query Vdd from the firmware if its last known value is 0.

TVSENSE may not be ready by the time t4_fw_initialize returns and the
firmware returns 0 if the driver asks for the Vdd before the sensor is
ready.

MFC after: 1 week
Sponsored by: Chelsio Communications

Split Chelsio send tags into a generic base tag and a ratelimit tag.

NIC KTLS will add a new TLS send tag type in cxgbe(4) that is a
distinct tag from a ratelimit tag. To support this, refactor
cxgbe_snd_tag to be a simple send tag with a type and convert the
existing ratelimit tag to a new cxgbe_rate_tag structure.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D22072

Always allocate the atid table during attach.

Previously the table was allocated on first use by TOE and the
ratelimit code. The forthcoming NIC KTLS code also uses this table.
Allocate it unconditionally during attach to simplify consumers.

Reviewed by: np
Differential Revision: https://reviews.freebsd.org/D22028

Convert to if_foreach_llmaddr() KPI.

Remove adapters from t4_list earlier during detach.

This ensures the clip task won't race with t4_destroy_clip_table.

While here, make some mutex destroys unconditional since attach always
initializes them.

Reviewed by: np
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21952

cxgbe/t4_tom: Initialize all TOE connection parameters in one place.
Remove now-redundant items from toepcb and synq_entry and the code to
support them.

Let the driver calculate tx_align, rx_coalesce, and sndbuf by default.

Reviewed by: jhb@
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21387

This adds the third step in getting BBR into the tree. BBR and
an updated rack depend on having access to the new
ratelimit api in this commit.

Sponsored by: Netflix Inc.
Differential Revision: https://reviews.freebsd.org/D20953

cxgbe(4): Clear the freelist statistics in the clearstats ioctl.

Move all clearstats code into its own function while here.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Use the simplest configuration possible when falling back from
the default configuration.

MFC after: 1 week
Sponsored by: Chelsio Communications

Add support for IFCAP_NOMAP to cxgbe(4).

Since cxgbe(4) uses sglist instead of bus_dma, this required updates
to the code that generates scatter/gather lists for packets. Also,
unmapped mbufs are always sent via DMA and never as immediate data in
the payload of a work request.

Submitted by: gallatin (earlier version)
Reviewed by: gallatin, hselasky, rrs
Discussed with: np
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20616

cxgbe(4): Rename the DDP sysctl to rx_zcopy to match the tx_zcopy sysctl
and update its description. The old name continues to work for now.

Sponsored by: Chelsio Communications

Fix two errors reported by PVS Studio: V646 Consider inspecting the
application's logic. It's possible that 'else' keyword is missing.

Reviewed by: gallatin, np, pfg
Approved by: pfg
Differential Revision: https://reviews.freebsd.org/D20396

Restructure mbuf send tags to provide stronger guarantees.

- Perform ifp mismatch checks (to determine if a send tag is allocated
for a different ifp than the one the packet is being output on), in
ip_output() and ip6_output(). This avoids sending packets with send
tags to ifnet drivers that don't support send tags.

Since we are now checking for ifp mismatches before invoking
if_output, we can now try to allocate a new tag before invoking
if_output sending the original packet on the new tag if allocation
succeeds.

To avoid code duplication for the fragment and unfragmented cases,
add ip_output_send() and ip6_output_send() as wrappers around
if_output and nd6_output_ifp, respectively. All of the logic for
setting send tags and dealing with send tag-related errors is done
in these wrapper functions.

For pseudo interfaces that wrap other network interfaces (vlan and
lagg), wrapper send tags are now allocated so that ip*_output see
the wrapper ifp as the ifp in the send tag. The if_transmit
routines rewrite the send tags after performing an ifp mismatch
check. If an ifp mismatch is detected, the transmit routines fail
with EAGAIN.

- To provide clearer life cycle management of send tags, especially
in the presence of vlan and lagg wrapper tags, add a reference count
to send tags managed via m_snd_tag_ref() and m_snd_tag_rele().
Provide a helper function (m_snd_tag_init()) for use by drivers
supporting send tags. m_snd_tag_init() takes care of the if_ref
on the ifp meaning that code alloating send tags via if_snd_tag_alloc
no longer has to manage that manually. Similarly, m_snd_tag_rele
drops the refcount on the ifp after invoking if_snd_tag_free when
the last reference to a send tag is dropped.

This also closes use after free races if there are pending packets in
driver tx rings after the socket is closed (e.g. from tcpdrop).

In order for m_free to work reliably, add a new CSUM_SND_TAG flag in
csum_flags to indicate 'snd_tag' is set (rather than 'rcvif').
Drivers now also check this flag instead of checking snd_tag against
NULL. This avoids false positive matches when a forwarded packet
has a non-NULL rcvif that was treated as a send tag.

- cxgbe was relying on snd_tag_free being called when the inp was
detached so that it could kick the firmware to flush any pending
work on the flow. This is because the driver doesn't require ACK
messages from the firmware for every request, but instead does a
kind of manual interrupt coalescing by only setting a flag to
request a completion on a subset of requests. If all of the
in-flight requests don't have the flag when the tag is detached from
the inp, the flow might never return the credits. The current
snd_tag_free command issues a flush command to force the credits to
return. However, the credit return is what also frees the mbufs,
and since those mbufs now hold references on the tag, this meant
that snd_tag_free would never be called.

To fix, explicitly drop the mbuf's reference on the snd tag when the
mbuf is queued in the firmware work queue. This means that once the
inp's reference on the tag goes away and all in-flight mbufs have
been queued to the firmware, tag's refcount will drop to zero and
snd_tag_free will kick in and send the flush request. Note that we
need to avoid doing this in the middle of ethofld_tx(), so the
driver grabs a temporary reference on the tag around that loop to
defer the free to the end of the function in case it sends the last
mbuf to the queue after the inp has dropped its reference on the
tag.

- mlx5 preallocates send tags and was using the ifp pointer even when
the send tag wasn't in use. Explicitly use the ifp from other data
structures instead.

- Sprinkle some assertions in various places to assert that received
packets don't have a send tag, and that other places that overwrite
rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer.

Reviewed by: gallatin, hselasky, rgrimes, ae
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20117

Track device's NUMA domain in ifnet & alloc ifnet from NUMA local memory

This commit adds new if_alloc_domain() and if_alloc_dev() methods to
allocate ifnets. When called with a domain on a NUMA machine,
ifalloc_domain() will record the NUMA domain in the ifnet, and it will
allocate the ifnet struct from memory which is local to that NUMA
node. Similarly, if_alloc_dev() is a wrapper for if_alloc_domain
which uses a driver supplied device_t to call ifalloc_domain() with
the appropriate domain.

Note that the new if_numa_domain field fits in an alignment pad in
struct ifnet, and so does not alter the size of the structure.

Reviewed by: glebius, kib, markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D19930

cxgbe(4): Make sure bundled_fw is always initialized before use.

This fixes a bug that prevented the driver from auto-flashing the
firmware when it didn't see one on the card. This feature was
introduced in r321390 and this bug was introduced in r343269.

Reported by: gallatin@
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe/t4_tom: Catch up with r344433, which removed tcb_autorcvbuf_inc.

The declaration in tcp_var.h is still around so t4_tom continued to
compile but wouldn't load. A separate commit will fix tcp_var.h

Reported By: Dustin Marquess (dmarquess at gmail)

Sponsored by: Chelsio Communications

cxgbe(4): Count and clear interrupts generated at the software's request.

An interrupt can be requested by setting the F_SWINT bit in PL_PF_CTL.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Treat the viid as an opaque identifier.

Recent firmwares prefer to use a different format for viid internally
and this change allows them to do so.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Don't forget to report link state to the kernel if the link is
already up at attach.

Reported by: Fabrice Bruel @ Orange Business Service
MFC after: 1 week
Sponsored by: Chelsio Communications

Don't assume all children of a nexus are ports.

Specifically, ccr(4) devices are also children of cxgbe nexus devices.
Rather than making assumptions about the child device's softc, walk
the list of ports from the nexus' softc to determine if a child is a
port in t4_child_location_str(). This fixes a panic when detaching a
ccr device.

Reviewed by: np
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D19399

cxgbe(4): Request high priority filter support explicitly, as required
by recent firmwares.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Delay the panic due to a fatal error by 30s.

This lets information logged by the interrupt handler reach the system
log before the system goes down.

cxgbe(4): Auto-dump the CIM block's logic analyzer on a TIMER0 interrupt.

Sponsored by: Chelsio Communications

cxgbe(4): Auto-dump the device log on a mailbox timeout or when the
firmware reports an error in pcie_fw.

Sponsored by: Chelsio Communications

cxgbe(4): Improved error reporting and diagnostics.

"slow" interrupt handler:
- Expand the list of INT_CAUSE registers known to the driver.
- Add decode information for many more bits but decouple it from the
rest of intr_info so that it is entirely optional.
- Call t4_fatal_err exactly once, and from the top level PL intr handler.

t4_fatal_err:
- Use t4_shutdown_adapter from the common code to stop the adapter.
- Stop servicing slow interrupts after the first fatal one.

Driver/firmware interaction:
- CH_DUMP_MBOX: note whether the mailbox being dumped is a command or a
reply or something else.
- Log the raw value of pcie_fw for some errors.
- Use correct log levels (debug vs. error).

Sponsored by: Chelsio Communications

Fix a few more places to handle ofld tx queues for RATELIMIT.

- Drain offload transmit queues when RATELIMIT is enabled but
TCP_OFFLOAD is not.
- Expose the per-VI nofldtxq and first_ofld_txq sysctls when
RATELIMIT is enabled but TCP_OFFLOAD is not.
- Clear offload transmit queue stats as part of a 'cxgbetool clearstats'
request when RATELIMIT is enabled but TCP_OFFLOAD is not.

Reviewed by: np
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D18966

cxgbe(4): Allow negative values in hw.cxgbe.fw_install and take them to
mean that the driver should taste the firmware in the KLD and use that
firmware's version for all its fw_install checks.

The driver gets firmware version information from compiled-in values by
default and this change allows custom (or older/newer) firmware modules
to be used with the stock driver.

There is no change in default behavior.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Use a truncated firmware header for version checks. All the
version numbers are towards the begining of the header.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Attach to two T540 variants.

MFC after: 1 week

cxgbe/iw_cxgbe: Use DSGLs to write to card's memory when appropriate.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

cxgbe(4): Fall back to a basic configuration in case of any error during
card initialization. This is an expanded version of r333682.

Break up prep_firmware into simpler routines while here. Load the
firmware/config KLD only if needed.

MFC after: 1 month
Sponsored by: Chelsio Communications

Make most of the CLIP code conditional on #ifdef INET6.

This fixes builds of kernels without INET6 such as LINT-NOINET6.

Reported by: arybchik
Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D18384

Move CLIP table handling out of TOM and into the base driver.

- Store the clip table in 'struct adapter' instead of in the TOM softc.
- Init the clip table during attach and teardown during detach.
- While here, add a dev.<nexus>.<unit>.misc.clip sysctl to dump the
CLIP table.

This does mean that we update the clip table even if TOE is not enabled,
but non-TOE things need the CLIP table anyway.

Reviewed by: np, Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D18010

cxgbe: revert r309725

After the fix contained in r341144, cxgbe does not need anymore
to set the IFCAP_NETMAP flag manually.

Reviewed by: np
Approved by: gnn (mentor)
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D17987

Add read-only sysctls for all tunables in the cxgbe(4) driver.

Reviewed by: np
MFC after: 1 month
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D18360

Change the quantum for TLS key addresses to 32 bytes.

The addresses passed when reading and writing keys are always shifted
right by 5 as the memory locations are addressed in 32-byte chunks, so
the quantum needs to be 32, not 8.

MFC after: 1 month
Sponsored by: Chelsio Communications

Move the TLS key map into the adapter softc so non-TOE code can use it.

Sponsored by: Chelsio Communications

Treat the memory lengths for CHELSIO_T4_GET_MEM as unsigned.

Previously attempts to read the MC region were failing since the
length was greater than 2^31.

Reviewed by: np
MFC after: 2 months
Differential Revision: https://reviews.freebsd.org/D17857

Add a facility for transmitting "raw" work requests on regular NIC queues.

- Use PH_loc.eight[1] as a general 'cflags' (Chelsio flags) field to
describe properties of a queued packet. The MC_RAW_WR flag
indicates an mbuf holding a raw work request. mbuf_cflags() returns
the current flags.
- Raw work request mbufs are allocated via alloc_wr_mbuf() which will
allocate a single contiguous range to hold the mbuf data. The
consumer can use mtod() to obtain the start of the work request and
write the required work request in the buffer. The mbuf can then be
enqueued directly to the txq via mp_ring_enqueue().
- Since raw work requests might potentially send arbitrary work
requests, only set the EQUIQ and EQUEQ bits on work requests that
support them such as the normal tunneled Ethernet packet work
requests.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D17811

Add support for port unit wiring to cxgbe(4).

- Add a bus_child_location_str method to the nexus drivers that prints
out 'port=N' as the location string exported via devinfo and the
'%location' sysctl node.

- We can't use a bus_hint_device_unit to wire the unit numbers of
devices with a fixed devclass as the device gets assigned a unit in
make_device() before the device creator can set softc, etc.
Instead, when adding a child device, use a helper function much like
a bus_hint_device_unit method to look for wiring hints or to return
-1 to let the system choose a unit number. This function requires
an "at" hint for the port pointing to the nexus device and a "port"
hint listing the port number. For example:

hint.cxl.4.at="t5nex0"
hint.cxl.4.port="0"

wires cxl4 to the first port on the t5nex0 adapter.

Requested by: gallatin
MFC after: 2 months

cxgbe(4): Report a reasonable non-zero if_hw_tsomaxsegsize to the
kernel.

This reverts an accidental change that snuck in with r339628.

Sponsored by: Chelsio Communications

cxgbe(4): Allow "pass" filters to distribute matching traffic using a
subset of a VI's RSS indirection table.

This makes it possible to make groups out of rx queues and steer
different kinds of traffic to different groups. For example, an
interface with 8 rx queues could have all non-TCP traffic delivered to
queues 0-3 and all TCP traffic to queues 4-7.

Note that it is already possible for filters to steer traffic to a
particular queue or to distribute it using the full indirection table
(much like normal rx does).

Sponsored by: Chelsio Communications

cxgbe(4): Update the VI's default queue when netmap is enabled/disabled.

Sponsored by: Chelsio Communications

cxgbe(4): new sysctl to display the start of the RSS region for a VI.

dev.<ifname>.<inst>.rss_base

For example:
dev.cc.0.rss_base: 0
dev.cc.1.rss_base: 128
dev.vcc.0.rss_base: 256
dev.vcc.1.rss_base: 384

Sponsored by: Chelsio Communications

cxgbe(4): improve the accuracy of various TSO limits reported to the kernel.

Sponsored by: Chelsio Communications

cxgbe(4): Enable support for per-connection rate limiting in the default
firmware configuration files.

Approved by: re@ (gjb@)
Sponsored by: Chelsio Communications

cxgbe(4): Treat base/end of firmware parameters as signed integers when
figuring out whether the range is valid or not.

Approved by: re@ (rgrimes@)
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Link related changes.

- Switch to using 32b port/link capabilities in the driver. The 32b
format is used internally by firmwares > 1.16.45.0 and the driver will
now interact with the firmware in its native format, whether it's 16b
or 32b. Note that the 16b format doesn't have room for 50G, 200G, or
400G speeds.

- Add a bit in the pause_settings knobs to allow negotiated PAUSE
settings to override manual settings.

- Ensure that manual link settings persist across an administrative
down/up as well as transceiver unplug/replug.

- Remove unused is_*G_port() functions.

Approved by: re@ (gjb@)
MFC after: 1 month
Sponsored by: Chelsio Communications

cxgbe(4): Enable TXRTLMT by default when the feature is available in the
kernel (options RATELIMIT) and provisioned in the driver's configuration
file (nethofld > 0).

Submitted by: gallatin@
Approved by: re@ (kib@)

cxgbe(4): Use the correct number of parameters when querying the tid
range for hashfilters.

Approved by: re@ (gjb@)

cxgbev(4): Updates to the VF driver to cope with recent ifmedia and
ctrlq changes in the base driver.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Do not leak memory in case of errors during VI initialization.

Reported by: Coverity (CID 1392026)
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Check the RO bit properly before disabling relaxed ordering.

Reported by: Coverity (CID 1384286)
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Replace T4_PKT_TIMESTAMP with something slightly less hackish.

cxgbe(4): Adjust ntids to account for nhptids in the TOE case too.
This should have been part of r337538.

cxgbe(4): Use VLAN_TRUNKDEV instead of private cookie to figure out the
parent of a VLAN ifnet.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Use two hashes instead of a table to keep track of
hashfilters. Two because the driver needs to look up a hashfilter by
its 4-tuple or tid.

A couple of fixes while here:
- Reject attempts to add duplicate hashfilters.
- Do not assume that any part of the 4-tuple that isn't specified is 0.
This makes it consistent with all other mandatory parameters that
already require explicit user input.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Move all control queues to the adapter.

There used to be one control queue per adapter (the mgmtq) that was
initialized during adapter init and one per port that was initialized
later during port init. This change moves all the control queues (one
per port/channel) to the adapter so that they are initialized during
adapter init and are available before any port is up. This allows the
driver to issue ctrlq work requests over any channel without having to
bring up any port.

MFH: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Create two variants of service_iq, one for queues with
freelists and one for those without.

MFH: 3 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Add support for high priority filters on T6+. They have their
own region in the TCAM starting with T6, unlike previous chips where
they were in the same region as normal filters.

These filters "hit" before anything else in the LE's lookup. The exact
order is:
a) High priority filters
b) TOE's active region (TCAM and/or hash)
c) Servers (TOE hw listeners)
d) Normal filters

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Allow the driver to specify a burst size when configuring a
traffic class for rate limiting.

Add experimental knobs that allow the user to specify a default pktsize
and burstsize for traffic classes associated with a port:

dev.<ifname>.<instance>.tc.pktsize
dev.<ifname>.<instance>.tc.burstsize

Sponsored by: Chelsio Communications

cxgbe(4): Allow user-configured and driver-configured traffic classes to
be used simultaneously. Move sysctl_tc and sysctl_tc_params to
t4_sched.c while here.

MFC after: 3 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Break up sysctl_bitfield into 8 bit and 16 bit variants. Have
them display the current value of the bitfield rather than the fixed
value that was provided when the sysctl node was created.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Improvements in TID management.

- Ignore any type of TID where the start/end values are not in the
correct order. There are situations where the firmware isn't able to
reserve room for the number requested in the config file but doesn't
report a failure during configuration and instead sets end <= start.

- Track start/end in tid_tab and remove some redundant copies from
adapter->params.

- Move all the start/end and other read-only parameters to a quiet part
of tid_tab, away from the tid locks.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Consider rateunit before ratemode when displaying information
about a traffic class. This matches the order in which the firmware
evaluates unit and mode internally.

Sponsored by: Chelsio Communications

Fix vertical whitespace nit in cxgbe.

cxgbe(4): Add a sysctl to report the chip's microprocessor's load
averages. This works with debug or custom firmwares only.

sysctl dev.<nexus>.<instance>.loadavg
sysctl dev.t6nex.0.loadavg

MFC after: 1 month
Sponsored by: Chelsio Communications

cxgbe(4): Do not leak the filters in the hashfilter table on module
unload.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): sysctls to display the local and intr CPUs for the adapter.

The driver assumes the list can change (even though it does't right now)
and queries it every time the sysctl runs.

sysctl dev.<nexus>.<inst>.local_cpus
sysctl dev.<nexus>.<inst>.intr_cpus

sysctl dev.t6nex.0.local_cpus
sysctl dev.t6nex.0.intr_cpus

Sponsored by: Chelsio Communications

cxgbe(4): Catch up with recent changes in the kernel -- it no longer
holds non-sleepable locks around any of the driver ioctls.

Sponsored by: Chelsio Communications

cxgbe(4): Remove homemade version of htobe32 from the driver.

It was needed only for ia64 where it was implemented as a call to
bswapXX, which was always a real function. htobeXX with a constant
argument is calculated at compile-time everywhere else.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Include full duplex mediaopt in media that can be reported as
active. Always report full duplex in active media.

Sponsored by: Chelsio Communications

cxgbe(4): Retire an old check.

cxgbe(4): Add code to deal with the chip's source MAC table (aka SMT).

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

cxgbe(4): Use ifm for ifmedia just like the rest of the kernel.

No functional change.

cxgbe(4): Implement ifm_change callback.

Sponsored by: Chelsio Communications

cxgbe(4): Consider all supported speeds when building the ifmedia list
for a port. Fix other related issues while here:
- Require port lock for access to link_config.
- Allow 100Mbps operation by tracking the speed in Mbps. Yes, really.
- New port flag to indicate that the media list is immutable. It will
be used in future refinements.

This also fixes a bug where the driver reports incorrect media with
recent firmwares.

MFC after: 2 days
Sponsored by: Chelsio Communications

cxgbe(4): Report IFCAP_TXRTLMT to kernels built with RATELIMIT if the
firmware has provisioned resources for this feature.

Sponsored by: Chelsio Communications

cxgbe(4): Data path for rate-limited tx.

This is hardware support for the SO_MAX_PACING_RATE sockopt (see
setsockopt(2)), which is available in kernels built with "options
RATELIMIT".

Relnotes: Yes
Sponsored by: Chelsio Communications

cxgbe(4): Fix range checks in is_etid.

ifnet: Replace if_addr_lock rwlock with epoch + mutex

Run on LLNW canaries and tested by pho@

gallatin:
Using a 14-core, 28-HTT single socket E5-2697 v3 with a 40GbE MLX5
based ConnectX 4-LX NIC, I see an almost 12% improvement in received
packet rate, and a larger improvement in bytes delivered all the way
to userspace.

When the host receiving 64 streams of netperf -H $DUT -t UDP_STREAM -- -m 1,
I see, using nstat -I mce0 1 before the patch:

InMpps OMpps InGbs OGbs err TCP Est %CPU syscalls csw irq GBfree
4.98 0.00 4.42 0.00 4235592 33 83.80 4720653 2149771 1235 247.32
4.73 0.00 4.20 0.00 4025260 33 82.99 4724900 2139833 1204 247.32
4.72 0.00 4.20 0.00 4035252 33 82.14 4719162 2132023 1264 247.32
4.71 0.00 4.21 0.00 4073206 33 83.68 4744973 2123317 1347 247.32
4.72 0.00 4.21 0.00 4061118 33 80.82 4713615 2188091 1490 247.32
4.72 0.00 4.21 0.00 4051675 33 85.29 4727399 2109011 1205 247.32
4.73 0.00 4.21 0.00 4039056 33 84.65 4724735 2102603 1053 247.32

After the patch

InMpps OMpps InGbs OGbs err TCP Est %CPU syscalls csw irq GBfree
5.43 0.00 4.20 0.00 3313143 33 84.96 5434214 1900162 2656 245.51
5.43 0.00 4.20 0.00 3308527 33 85.24 5439695 1809382 2521 245.51
5.42 0.00 4.19 0.00 3316778 33 87.54 5416028 1805835 2256 245.51
5.42 0.00 4.19 0.00 3317673 33 90.44 5426044 1763056 2332 245.51
5.42 0.00 4.19 0.00 3314839 33 88.11 5435732 1792218 2499 245.52
5.44 0.00 4.19 0.00 3293228 33 91.84 5426301 1668597 2121 245.52

Similarly, netperf reports 230Mb/s before the patch, and 270Mb/s after the patch

Reviewed by: gallatin
Sponsored by: Limelight Networks
Differential Revision: https://reviews.freebsd.org/D15366

cxgbe(4): Implement ifnet callbacks that deal with send tags.

An etid (ethoffload tid) is allocated for a send tag and it acquires a
reference on the traffic class that matches the send parameters
associated with the tag.

Sponsored by: Chelsio Communications

cxgbe(4): Fix s->neq miscalculation in r333698.

cxgbe(4): Allocate offload Tx queues when a card has resources
provisioned for NIC_ETHOFLD and the kernel has option RATELIMIT.

It is possible to use the chip's offload queues for normal NIC Tx and
not just TOE Tx. The difference is that these queues support out of
order processing of work requests and have a per-"flowid" mechanism for
tracking credits between the driver and hardware. This allows Tx for
any number of flows bound to different rate limits to be submitted to a
single Tx queue and the work requests for slow flows won't cause HOL
blocking for the rest.

Sponsored by: Chelsio Communications

cxgbe(4): Add NIC_ETHOFLD to the NIC capabilities allowed by the driver
by default.

This is the first of a series of commits that will add support for
RATELIMIT kernel option to the base if_cxgbe driver, for use with
ordinary NIC traffic "flows". RATELIMIT is already supported by t4_tom
for the fully-offloaded TCP connections that it handles.

Sponsored by: Chelsio Communications

cxgbe(4): Fall back to a failsafe configuration built into the firmware
if an error is reported while pre-processing the configuration file that
the driver attempted to use.

Also, allow the user to explicitly use the built-in configuration with
hw.cxgbe.config_file="built-in"

MFC after: 2 days
Sponsored by: Chelsio Communications

cxgbe(4): Claim some more T5 and T6 boards.

MFC after: 2 days
Sponsored by: Chelsio Communications

cxgbe(4): Disable write-combined doorbells by default.

This had been the default behavior but was changed accidentally as part
of the recent iw_cxgbe+OFED overhaul. Fix another bug in that change
while here: the global knob affects all the adapters in the system and
should be left alone by per-adapter code.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Determine whether the firmware supports the FILTER2 work
request, which can be used to configure hardware NAT and swapmac.

All firmwares released after Jan 2017 support this work request.

Sponsored by: Chelsio Communications

cxgbe(4): Add support for hash filters.

These filters reside in the card's memory instead of its TCAM and can be
configured via a new "hashfilter" subcommand in cxgbetool. Hash and
normal TCAM filters can be used together. The hardware does an
exact-match of packet fields for hash filters, unlike the masked match
performed for TCAM filters. Any T5/T6 card with memory can support at
least half a million hash filters. The sample config file with the
driver configures 512K of these, it is possible to double this to 1
million+ in some cases.

The chip does an exact-match of fields of incoming datagrams with hash
filters and performs the action configured for the filter if it matches.
The fields to match are specified in a "filter mask" in the firmware
config file. The filter mask always includes the 5-tuple (sip, dip,
sport, dport, ipproto). It can, optionally, also include any subset of
the filter mode (see filterMode and filterMask in the firmware config
file).

For example:
filterMode = fragmentation, mpshittype, protocol, vlan, port, fcoe
filterMask = protocol, port, vlan

Exact values of the 5-tuple, the physical port, and VLAN tag would have
to be provided while setting up a hash filter with the chip
configuration above.

Hash filters support all actions supported by TCAM filters. A packet
that hits a hash filter can be dropped, let through (with optional
steering to a specific queue or RSS region), switched out of another
port (with optional L2 rewrite of DMAC, SMAC, VLAN tag), or get NAT'ed.
(Support for some of these will show up in the driver in a follow-up
commit very shortly).

Sponsored by: Chelsio Communications

cxgbe(4): Update all firmwares to 1.19.1.0.

These firmwares and the following list of changes are from the public
ChelsioUwire-3.7.1.0 release.

T6 Firmware
================================================================================
Version : 1.19.1.0
Date : 04/23/2018
================================================================================

Fixes
-----

BASE:
- Fixed traffic stall when rate-limit is modified while running traffic.
- Fixes a firmware crash in FW_ETH_TX_EO_WR handling.
- Fixes host DCB support when FW_PORT_CMD is used.

ETH:
- Exit Auto-Negotiation if we don't receive base page from peer within 10s.
This fixes some cases where in we keep on restarting auto negotiation without
ever exiting, resulting in link failure.
- Fixes an issue where VF packets counter were not increasing if VF packets
coalesced WR is used by driver.

OFLD:
- Kernel and user mode NVMEoF performance enhancements.

FOiSCSI:
- Fixes fw crash when trying to connect to non-existence IPv6 iSNS target.

================================================================================
Version : 1.18.9.0
Date : 03/27/2018
================================================================================

Fixes
-----

BASE:
- For Ethernet frames less than 64B, pad them with zero bytes as per IEEE spec
(RFC 894).
- Added a new parameter iqtype to FW_IQ_CMD to identify the ingress NIC or offload
queues. This fixes an issue where driver was receiving interrupt with no new
messages in queue.
- FW_PARAMS_CMD processes all the valaid paramaters and returns value 0UL for
any unknown parameter.

OFLD:
- Fixes connection failure during SRQ reuse.
- Fixes incorrect cqe in case of WRITE with immediate operation.

FOiSCSI:
- Fixes a fw crash when wrong node-id is passed to FW_FOISCSI_CTRL_WR.

FOFCoE:
- Fixes a fw hang while creating NPIV.

Enhancements
------------

ETH:
- A new WR FW_ETH_TX_PKTS_VM_WR added to support VM packet coalescing.

================================================================================
Version : 1.18.4.0
Date : 02/28/2018
================================================================================

Fixes
-----

BASE:
- Fixed Rate limiting not working for 101Mbps<=rate limit<=163Mbps range.
- Fixed starting more than 32 VMs on PF4 causing firmware hang.

ETH:
- Fixed link failure due to FEC mismatch with optics.
- Fixed link failure with link toggle stress tests.
- Only BaseR FEC is supported for 50G.
- Fixed a bug in next page handling which sometimes causes link down.
- Fixed port down due to failre to read eeprom contents of some modules.
- Fixed a bug causing adapter to fail with spider configuration.

FOiSCSI:
- Fixed a bug causing login failure when connecting to multiple targets.

Enhancements
------------

BASE:
- Added a new firmware API to retrieve the maximum temperaturethreshold for
the chip (FW_PARAM_DEV_DIAG_MAXTMPTHRESH).

ETH:
- Added support for user to contol pause negotiation during auto negotiation.

FOiSCSI:
- Added a new facility to redirect few fw events to offload rx queue
(based on driver's configration)
- Driver can ignore providing ipv6 prefix len during ipv6 address configuration.

================================================================================
Version : 1.17.14.0
Date : 12/27/2017
================================================================================

FIXES
-----

BASE:
- Fixed an FLR failure during simulteneous power up of VM.
- Fixed an issue in vlan acl which was limiting vlan range to 1024.

ETH:
- Enabled RS-FEC for 25G active copper cable and 25GBASE-SR.
- When auto negotiation is enabled, final pause settings are resolved
based on local and peer pause settings.
- Handle NACK for an I2C access.

OFLD
- Fixed rdma connection cleanup in SO adpater.
- Fixed rdma connections during read invalidate.
- Fixed the crash when invalid BW rate is passed to fw.
- Fixed the traffic hang when BW allocation is changed from switch during traffic.

FOFCoE:
- Fixed an issue where initiator remains logged-in even after LLDP is disabled
on switch.

ENHANCEMENTS
------------

BASE:
- Added support for 248 VFs.
- Added fw driver periodic calibration for MC.

ETH:
- Added XLAUI port type support.
- Added raw mac entry deletion support (FW_VI_MAC_ID_BASED_FREE).

OFLD:
- Inline IPSec support added (flag F_FW_ULPTX_WR_DATA indicates the inline
IPSec WR).
- New work request FW_RI_RDMA_WRITE_CMPL_WR (write with completion) added to

T5 Firmware
================================================================================
Version : 1.19.1.0
Date : 04/23/2018
================================================================================

Fixes
-----

BASE:
- Fixes a firmware crash in FW_ETH_TX_EO_WR handling.
- Fixes host DCB support when FW_PORT_CMD is used.

ETH:
- Fixes an issue where VF packets counter were not increasing if VF packets
coalesced WR is used by driver.

OFLD:
- Fixes an issue where fw hangs if max traffic rate passed is 0.

FOiSCSI:
- Fixes fw crash when trying to connect to non-existence IPv6 iSNS target.

================================================================================
Version : 1.18.9.0
Date : 03/27/2018
================================================================================

Fixes
-----

BASE:
- For Ethernet frames less than 64B, pad them with zero bytes as per IEEE spec
(RFC 894).
- Added a new parameter iqtype to FW_IQ_CMD to identify the ingress NIC or offload
queues. This fixes an issue where driver was receiving interrupt with no new
messages in queue.

ETH:
- Pad the Ethernet packets of size less than 64B with zeros. This fixes the
incorrect checksum generation of packets less then 64B.

FOiSCSI:
- Fixes a fw crash when wrong node-id is passed to FW_FOISCSI_CTRL_WR.

FOFCoE:
- Fixes a fw hang while creating NPIV.

Enhancements
------------

ETH:
- A new WR FW_ETH_TX_PKTS_VM_WR added to support VM packet coalescing.

================================================================================
Version : 1.18.4.0
Date : 02/28/2018
================================================================================

Fixes
-----

BASE:
- Fixed starting more than 32 VMs on PF4 causing firmware hang.

FOiSCSI:
- Fixed a bug causing login failure when connecting to multiple targets.

Enhancements
------------

BASE:
- Added a new firmware API to retrieve the maximum temperaturethreshold for
the chip (FW_PARAM_DEV_DIAG_MAXTMPTHRESH).

ETH:
- Added support for user to contol pause negotiation during auto negotiation.

FOiSCSI:
- Added a new facility to redirect few fw events to offload rx queue
(based on driver's configration)
- Driver can ignore providing ipv6 prefix len during ipv6 address configuration.

================================================================================
Version : 1.17.14.0
Date : 12/27/2017
================================================================================

FIXES
-----

BASE:
- Fixed an issue in vlan acl which was limiting vlan range to 1024.

ETH:
- Corrected lane inversion logic.
- Fixed improper LED behavior in T580 cards.
- When auto negotiation is enabled, final pause settings are resolved
based on local and peer pause settings.
- Handle NACK for an I2C access.

OFLD
- Fixed rdma connections during read invalidate.

FOiSCSI:
- Fixed a connections hang when link is toggled frequently.

FOFCoE:
- Fixed an issue where initiator remains logged-in even after LLDP is disabled
on switch.

ENHANCEMENTS
------------

BASE:
- Added support for 124 VFs.

ETH:
- Added XLAUI port type support.
- Added raw mac entry deletion support (FW_VI_MAC_ID_BASED_FREE).

OFLD:
- New work request FW_RI_RDMA_WRITE_CMPL_WR (write with completion) added to
optimize NVMEoF write.

T4 Firmware
================================================================================
Version : 1.19.1.0
Date : 04/23/2018
================================================================================

Fixes
-----

BASE:
- Fixes a firmware crash in FW_ETH_TX_EO_WR handling.
- Fixes host DCB support when FW_PORT_CMD is used.

FOiSCSI:
- Fixes fw crash when trying to connect to non-existence IPv6 iSNS target.

================================================================================
Version : 1.18.9.0
Date : 03/27/2018
================================================================================

Fixes
-----

BASE:
- Added a new paramter iqtype to FW_IQ_CMD to identify the ingress NIC or
offload queues. This fixes an issue where driver was receiving interrupt with
no new messages in queue.

FOFCoE:
- Fixes a fw hang while creating NPIV.

Enhancements
------------

ETH:
- A new WR FW_ETH_TX_PKTS_VM_WR added to support VM packet coalescing.

================================================================================
Version : 1.18.4.0
Date : 02/28/2018
================================================================================

Enhancements
------------

BASE:
- Added a new firmware API to retrieve the maximum temperaturethreshold for
the chip (FW_PARAM_DEV_DIAG_MAXTMPTHRESH).

================================================================================
Version : 1.17.14.0
Date : 12/27/2017
================================================================================

FIXES
-----

BASE:
- Fixed an issue in vlan acl which was limiting vlan range to 1024.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Move all TCAM filter code into a separate file.

Sponsored by: Chelsio Communications

Optionally panic when cxgbe encounters a fatal error

Sometimes it is better to panic than to leave a machine
unreachable.

Reviewed by: np
Sponsored by: Netflix

cxgbe(4): Destroy the cdev before disabling interrupts in driver detach.

Filter work requests are submitted in the nexus cdev's ioctl which then
blocks waiting for a reply. If driver detach runs in this state and
disables interrupts the ioctl will never complete and detach will hang
in destroy_cdev.

Sponsored by: Chelsio Communications

cxgbe(4): Convert ACT_OPEN_RPL to a shared CPL.

Reserve 3b in the 14b atid to identify the owner and use it to dispatch
the CPL. This allows all CPLs that use an atid to be used as shared
CPLs, although ACT_OPEN_RPL is the only one being converted in this
revision.

Sponsored by: Chelsio Communications

cxgbe(4): Use opaque cookies or tid range-checks to determine the
intended recipient of a CPL when it can't be determined solely from the
opcode. Retire the per-queue handlers for such CPLs in favor of the new
scheme.

Sponsored by: Chelsio Communications

cxgbe(4): Move release_tid to the base NIC driver for future consumers.

Sponsored by: Chelsio Communications.

cxgbe(4): Break up alloc_tid_tabs and move the atid routines to the base
NIC driver. The atid services will be used by new features (hashfilters
and inline TLS) that do not involve TOE.

Sponsored by: Chelsio Communications

cxgbe(4): Add support for Connection Offload Policy (aka COP).

COP allows fine-grained control on whether to offload a TCP connection
using t4_tom, and what settings to apply to a connection selected for
offload. t4_tom must still be loaded and IFCAP_TOE must still be
enabled for full TCP offload to take place on an interface. The
difference is that IFCAP_TOE used to be the only knob and would enable
TOE for all new connections on the inteface, but now the driver will
also consult the COP, if any, before offloading to the hardware TOE.

A policy is a plain text file with any number of rules, one per line.
Each rule has a "match" part consisting of a socket-type (L = listen,
A = active open, P = passive open, D = don't care) and a pcap-filter(7)
expression, and a "settings" part that specifies whether to offload the
connection or not and the parameters to use if so. The general format
of a rule is: [socket-type] expr => settings

Example. See cxgbetool(8) for more information.
[L] ip && port http => offload
[L] port 443 => !offload
[L] port ssh => offload
[P] src net 192.168/16 && dst port ssh => offload !nagle !timestamp cong newreno
[P] dst port ssh => offload !nagle ecn cong tahoe
[P] dst port http => offload
[A] dst port 443 => offload tls
[A] dst net 192.168/16 => offload !timestamp cong highspeed

The driver processes the rules for each new listen, active open, or
passive open and stops at the first match. There is an implicit rule at
the end of every policy that prohibits offload when no rule in the
policy matches:
[D] all => !offload

This is a reworked and expanded version of a patch submitted by
Krishnamraju Eraparaju @ Chelsio.

Sponsored by: Chelsio Communications

cxgbe(4): Always display an error message if SIOCSIFFLAGS will leave
IFF_UP and IFF_DRV_RUNNING out of sync. ifhwioctl in the kernel pays no
attention to the return code from the driver ioctl during SIOCSIFFLAGS
so these messages are the only indication that the ioctl was called but
failed.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe: Implement tcp_info handler for connections handled by t4_tom.

The TCB is read using a memory window right now. A better alternate to
get self-consistent, uncached information would be to use a GET_TCB
request but waiting for a reply from hw while holding non-sleepable
locks is quite inconvenient.

Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D14817

Use an accessor function to access ifr_data.

This fixes 32-bit compat (no ioctl command defintions are required
as struct ifreq is the same size). This is believed to be sufficent to
fully support ifconfig on 32-bit systems.

Reviewed by: kib
Obtained from: CheriBSD
MFC after: 1 week
Relnotes: yes
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D14900

cxgbe(4): Always initialize requested_speed to a valid value.

This fixes an avoidable EINVAL when the user tries to disable AN after
the port is initialized but l1cfg doesn't have a valid speed to use.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Tunnel congestion drops on a port should be cleared when the
stats for that port are cleared.

MFC after: 1 week
Sponsored by: Chelsio Communications

Support for TLS offload of TOE connections on T6 adapters.

The TOE engine in Chelsio T6 adapters supports offloading of TLS
encryption and TCP segmentation for offloaded connections. Sockets
using TLS are required to use a set of custom socket options to upload
RX and TX keys to the NIC and to enable RX processing. Currently
these socket options are implemented as TCP options in the vendor
specific range. A patched OpenSSL library will be made available in a
port / package for use with the TLS TOE support.

TOE sockets can either offload both transmit and reception of TLS
records or just transmit. TLS offload (both RX and TX) is enabled by
setting the dev.t6nex.<x>.tls sysctl to 1 and requires TOE to be
enabled on the relevant interface. Transmit offload can be used on
any "normal" or TLS TOE socket by using the custom socket option to
program a transmit key. This permits most TOE sockets to
transparently offload TLS when applications use a patched SSL library
(e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
library). Receive offload can only be used with TOE sockets using the
TLS mode. The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
list of TCP port numbers. Any connection with either a local or
remote port number in that list will be created as a TLS socket rather
than a plain TOE socket. Note that although this sysctl accepts an
arbitrary list of port numbers, the sysctl(8) tool is only able to set
sysctl nodes to a single value. A TLS socket will hang without
receiving data if used by an application that is not using a patched
SSL library. Thus, the tls_rx_ports node should be used with care.
For a server mostly concerned with offloading TLS transmit, this node
is not needed as plain TOE sockets will fall back to software crypto
when using an unpatched SSL library.

New per-interface statistics nodes are added giving counts of TLS
packets and payload bytes (payload bytes do not include TLS headers or
authentication tags/MACs) offloaded via the TOE engine, e.g.:

dev.cc.0.stats.rx_tls_octets: 149
dev.cc.0.stats.rx_tls_records: 13
dev.cc.0.stats.tx_tls_octets: 26501823
dev.cc.0.stats.tx_tls_records: 1620

TLS transmit work requests are constructed by a new variant of
t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.

TLS transmit work requests require a buffer containing IVs. If the
IVs are too large to fit into the work request, a separate buffer is
allocated when constructing a work request. This buffer is associated
with the transmit descriptor and freed when the descriptor is ACKed by
the adapter.

Received TLS frames use two new CPL messages. The first message is a
CPL_TLS_DATA containing the decryped payload of a single TLS record.
The handler places the mbuf containing the received payload on an
mbufq in the TOE pcb. The second message is a CPL_RX_TLS_CMP message
which includes a copy of the TLS header and indicates if there were
any errors. The handler for this message places the TLS header into
the socket buffer followed by the saved mbuf with the payload data.
Both of these handlers are contained in tom/t4_tls.c.

A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
including send_rx_credits(), a new send_rx_modulate(), and
t4_close_conn().

TLS keys for both transmit and receive are stored in onboard memory
in the NIC in the "TLS keys" memory region.

In some cases a TLS socket can hang with pending data available in the
NIC that is not delivered to the host. As a workaround, TLS sockets
are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
any data is read from a TLS socket. In addition, a fallback timer will
periodically send CPL_RX_DATA_ACK messages to the NIC for connections
that are still in the handshake phase. Once the connection has
finished the handshake and programmed RX keys via the socket option,
the timer is stopped.

A new function select_ulp_mode() is used to determine what sub-mode a
given TOE socket should use (plain TOE, DDP, or TLS). The existing
set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
handles initialization of TLS-specific state when necessary in
addition to DDP-specific state.

Since TLS sockets do not receive individual TCP segments but always
receive full TLS records, they can receive more data than is available
in the current window (e.g. if a 16k TLS record is received but the
socket buffer is itself 16k). To cope with this, just drop the window
to 0 when this happens, but track the overage and "eat" the overage as
it is read from the socket buffer not opening the window (or adding
rx_credits) for the overage bytes.

Reviewed by: np (earlier version)
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D14529

Move #include for rijndael.h out of x86-specific region.

The #include was added inside of the conditional by accident and the lack
of it broke non-x86 builds.

Reported by: lwhsu (jenkins), andrew

Move ccr_aes_getdeckey() from ccr(4) to the cxgbe(4) driver.

This routine will also be used by the TOE module to manage TLS keys.

Sponsored by: Chelsio Communications

Fetch TLS key parameters from the firmware.

The parameters describe how much of the adapter's memory is reserved for
storing TLS keys. The 'meminfo' sysctl now lists this region of adapter
memory as 'TLS keys' if present.

Sponsored by: Chelsio Communications

cxgbe(4): Accept old names of a couple of tunables.

cxgbe(4): Add a knob to enable/disable PCIe relaxed ordering. Disable it by
default when running on Intel CPUs.

This is a crude fix for the performance issues alluded to in these Linux commits:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=87e09cdec4dae08acdb4aa49beb793c19d73e73e
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a99b646afa8a02571ea298bedca6592d818229cd

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Do not forward interrupts to queues with freelists. This
leaves the firmware event queue (fwq) as the only queue that can take
interrupts for others.

This simplifies cfg_itype_and_nqueues and queue allocation in the driver
at the cost of a little (never?) used configuration. It also allows
service_iq to be split into two specialized variants in the future.

MFC after: 2 months
Sponsored by: Chelsio Communications

cxgbe(4): Read the MFG diags version from the VPD and make it available
in the sysctl MIB.

MFC after: 1 week
Sponsored by: Chelsio Communications

sys/dev: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

cxgbe(4): Add a custom board to the device id list.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Add core Vdd to the sysctl MIB.

Sponsored by: Chelsio Communications

cxgbe(4): Remove rsrv_noflowq from intrs_and_queues structure as it does
not influence or get affected by the number of interrupts or queues.

Sponsored by: Chelsio Communications

cxgbe(4): Sanitize t4_num_vis during MOD_LOAD like all other t4_*
tunables. Add num_vis to the intrs_and_queues structure as it affects
the number of interrupts requested and queues created. In future
cfg_itype_and_nqueues might lower it incrementally instead of going
straight to 1 when enough interrupts aren't available.

Sponsored by: Chelsio Communications

cxgbe(4): Combine all _10g and _1g tunables and drop the suffix from
their names. The finer-grained knobs weren't practically useful.

Sponsored by: Chelsio Communications

Update the iw_cxgbe bits in the projects branch.

Submitted by: Krishnamraju Eraparaju @ Chelsio
Sponsored by: Chelsio Communications

cxgbe(4): Read the MPS buffer group map from the firmware as it could be
different from hardware defaults. The congestion channel map, which is
still fixed, needs to be tracked separately now. Change the congestion
setting for TOE rx queues to match the drivers on other OSes while here.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Provide knobs to set the holdoff parameters of TOE rx queues
separately from NIC rx queues instead of using the same parameters for
both types of queues.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

Enable support for lookaside crypto operations by default.

This permits ccr(4) to be used with the default firmware configuration
file.

Discussed with: np
Sponsored by: Chelsio Communications

cxgbe(4): Ignore capabilities that depend on TOE when the firmware
reports TOE is not available.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe/t4_tom: Add a knob to select the congestion control algorigthm
used by the TOE hardware for fully offloaded connections. The knob
affects new connections only.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Add two new debug flags -- one to allow manual firmware
install after full initialization, and another to disable the TCB
cache (T6+). The latter works as a tunable only.

Note that debug_flags are for debugging only and should not be set
normally.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Zero out the memory allocated for the debug dump.
cudbg_collect seems to expect it this way.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Do not access the mailbox without appropriate locks while
creating hardware VIs.

This fixes a bad race on systems with hw.cxgbe.num_vis > 1.

Reported by: olivier@
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Maintain one ifmedia per physical port instead of one per
Virtual Interface (VI). All autonomous VIs that share a port share the
same media.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): vi_mac_funcs should include the base Ethernet function. It is
already used in the driver as if it does.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Verify that the driver accesses the firmware mailbox in a
thread-safe manner.

MFC after: 3 days

cxgbe(4): Save the last reported link parameters and compare them with
the current state to determine whether to generate a link-state change
notification. This fixes a bug introduced in r321063 that caused the
driver to sometimes skip these notifications.

Reported by: Jason Eggleston @ LLNW
MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Allow the TOE timer tunables to be set with microsecond
precision. These timers are already displayed in microseconds in the
sysctl MIB. Add variables to track these tunables while here.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Always use the first and not the last virtual interface
associated with a port in begin_synchronized_op.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Initial import of the "collect" component of Chelsio unified
debug (cudbg) code, hooked up to the main driver via an ioctl.

The ioctl can be used to collect the chip's internal state in a
compressed dump file. These dumps can be decoded with the "view"
component of cudbg.

Obtained from: Chelsio Communications
MFC after: 2 months
Sponsored by: Chelsio Communications

cxgbe(4): Some updates to the common code.

- Updated register ranges.
- Helper routines for access to TP registers.
- Updated routine to read flash parameters.

Obtained from: Chelsio Communications
MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Display some more TOE parameters related to retransmission
and keepalive in the sysctl MIB. Provide tunables to change some of
these parameters. These are supposed to be setup by the firmware so
these tunables are for experimentation only.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Install the firmware bundled with the driver to the card if it
doesn't seem to have one. This lets the driver recover automatically
from incomplete firmware upgrades (panic, reboot, power loss, etc. in
the middle of an upgrade).

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): New ioctls to flash bootrom and boot config to the card.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

cxgbe(4): Various link/media related improvements.

- Deal with changes to port_type, and not just port_mod when a
transceiver is changed. This fixes hot swapping of transceivers of
different types (QSFP+ or QSA or QSFP28 in a QSFP28 port, SFP+ or
SFP28 in a SFP28 port, etc.).

- Always refresh media information for ifconfig if the port is down.
The firmware does not generate tranceiver-change interrupts unless at
least one VI is enabled on the physical port. Before this change
ifconfig diplayed potentially stale information for ports that were
administratively down.

- Always recalculate and reapply L1 config on a transceiver change.

- Display PAUSE settings in ifconfig. The driver sysctls for this
continue to work as well.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

Add support for child devices that aren't ports.

Invoke any identify routines of child drivers during attach before attaching
children, and delete any remaining devices after deleting ports.

MFC after: 1 month
Sponsored by: Chelsio Communications

cxgbe(4): Do not assume that if_qflush is always followed by inteface-down.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Fixes related to the knob that controls link autonegotiation.

- Do not leak the adapter lock in sysctl_autoneg.
- Accept only 0 or 1 as valid settings for autonegotiation.
- A fixed speed must be requested by the driver when autonegotiation is
disabled otherwise the firmware will reject the l1cfg command. Use
the top speed supported by the port for now.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Update the list of PCIe devices claimed by the driver. At
this point any board with a T6 should just work.

Obtained from: Chelsio Communications
MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Support routines for Tx traffic scheduling.

- Create a new file, t4_sched.c, and move all of the code related to
traffic management from t4_main.c and t4_sge.c to this file.
- Track both Channel Rate Limiter (ch_rl) and Class Rate Limiter (cl_rl)
parameters in the PF driver.
- Initialize all the cl_rl limiters with somewhat arbitrary default
rates and provide routines to update them on the fly.
- Provide routines to reserve and release traffic classes.

MFC after: 1 month
Sponsored by: Chelsio Communications

cxgbe: Query some more RDMA related parameters from the firmware.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Program the global RSS key once instead of once per ifnet.

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Fix an always-true assertion (reported by PVS-Studio).

sys/dev/cxgbe/t4_main.c: PVS-Studio: Expression is Always True (CWE-571) (3)

PR: 217746
Submitted by: Svyatoslav (razmyslov at viva64 com)
MFC after: 1 week

cxgbe(4): Allow tunables that control the number of queues to be set to
'-n' to tell the driver to create _up to_ 'n' queues if enough cores are
available. For example, setting hw.cxgbe.nrxq10g="-32" will result in
16 queues if the system has 16 cores, 32 if it has 32.

There is no change in the default number of queues of any type.

MFC after: 1 week
Sponsored by: Chelsio Communications

cxgbe(4): Attach to the 2x25 debug card. This is for internal use only.

MFC after: 3 days

cxgbe(4): Updates to link configuration.

- Update struct link_settings and associated shared code.

- Add tunables to control FEC and autonegotiation. All ports inherit
these values as their initial settings.
hw.cxgbe.fec
hw.cxgbe.autoneg

- Add per-port sysctls to control FEC and autonegotiation. These can be
modified at any time.
dev.<port>.<n>.fec
dev.<port>.<n>.autoneg

MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Fix the tid range shown for T6 cards in misc.tids.

MFC after: 3 days

cxgbe(4): netmap does not set IFCAP_NETMAP in an ifnet's if_capabilities
any more (since r307394). Do it in the driver instead.

MFC after: 1 week

cxgbe(4): Update firmwares from version 1.16.12.0 to 1.16.22.0.

Obtained from: Chelsio Communications
MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Include firmware for T6 cards in the driver. Update all
firmwares to 1.16.12.0.

Obtained from: Chelsio Communications
MFC after: 3 days
Sponsored by: Chelsio Communications

cxgbe(4): Adjust whitespace to line up the column titles in cim_qcfg
with the values displayed.

cxgbe(4): Allow the interface MTU to be set as high as the actual
hardware limit.

Submitted by: jpaetzel@
Differential Revision: https://reviews.freebsd.org/D8237

cxgbe(4): Add an ioctl to copy a firmware config file to the card's flash.

cxgbe(4): Fix whitespace in the pm_stats display.

cxgbe(4): Claim the T6 -DBG card.

cxgbe(4): Use the port's top speed to figure out whether it is "high
speed" or not (for the purpose of calculating the number of queues etc.)
This does the right thing for 25Gbps and 100Gbps ports.

cxgbe(4): Support SIOGIFXMEDIA so that ifconfig displays correct media
for 25Gbps and 100Gbps ports. This should have been part of r305713,
which is when the driver first started reporting extended media types.

cxgbe(4): Fix the output of the "tids" sysctl on T6.

cxgbe(4): Catch up with the different layout of WHOAMI in T6.

Note that the code moved below t4_prep_adapter() as part of this change
because now it needs a working chip_id().

cxgbe(4): Show wcwr_stats for T6 cards.

cxgbe/t4_tom: The SMAC entry for a VI is at a different location in the T6.

Sponsored by: Chelsio Communications

cxgbe(4): Attach to cards with the Terminator 6 ASIC. T6 cards will
come up as 't6nex' nexus devices with 'cc' ports hanging off them.

The T6 firmware and configuration files will be added as soon as they
are released. For now the driver will try to work with whatever
firmware and configuration is on the card's flash.

Sponsored by: Chelsio Communications

cxgbe(4): Catch up with the rename of tlscaps -> cryptocaps. TLS is one
of the capabilities of the crypto engine in T6.

Sponsored by: Chelsio Communications

cxgbe(4): Add support for additional port types and link speeds.

Sponsored by: Chelsio Communications.

cxgbe(4): Rename the debug_flags driver tunable/sysctl to dflags.
Tunables that end with _flags are special.

Sponsored by: Chelsio Communications

cxgbe(4): Deal with the slightly different SGE_STAT_CFG in T6.

Sponsored by: Chelsio Communications

cxgbe(4): Avoid a NULL dereference in the clearstats ioctl handler.
Port softc's are not initialized when the adapter is in recovery mode.

Chelsio T4/T5 VF driver.

The cxgbev/cxlv driver supports Virtual Function devices for Chelsio
T4 and T4 adapters. The VF devices share most of their code with the
existing PF4 driver (cxgbe/cxl) and as such the VF device driver
currently depends on the PF4 driver.

Similar to the cxgbe/cxl drivers, the VF driver includes a t4vf/t5vf
PCI device driver that attaches to the VF device. It then creates
child cxgbev/cxlv devices representing ports assigned to the VF.
By default, the PF driver assigns a single port to each VF.

t4vf_hw.c contains VF-specific routines from the shared code used to
fetch VF-specific parameters from the firmware.

t4_vf.c contains the VF-specific PCI device driver and includes its
own attach routine.

VF devices are required to use a different firmware request when
transmitting packets (which in turn requires a different CPL message
to encapsulate messages). This alternate firmware request does not
permit chaining multiple packets in a single message, so each packet
results in a firmware request. In addition, the different CPL message
requires more detailed information when enabling hardware checksums,
so parse_pkt() on VF devices must examine L2 and L3 headers for all
packets (not just TSO packets) for VF devices. Finally, L2 checksums
on non-UDP/non-TCP packets do not work reliably (the firmware trashes
the IPv4 fragment field), so IPv4 checksums for such packets are
calculated in software.

Most of the other changes in the non-VF-specific code are to expose
various variables and functions private to the PF driver so that they
can be used by the VF driver.

Note that a limited subset of cxgbetool functions are supported on VF
devices including register dumps, scheduler classes, and clearing of
statistics. In addition, TOE is not supported on VF devices, only for
the PF interfaces.

Reviewed by: np
MFC after: 2 months
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7599

cxgbe/cxgbei: Retire all DDP related code from cxgbei and switch to
routines available in t4_tom to manage the iSCSI DDP page pod region.

This adds the ability to use multiple DDP page sizes to the iSCSI
driver, among other improvements.

Sponsored by: Chelsio Communications

cxgbe(4): Provide more details about the card in the sysctl MIB.

dev.t5nex.0.%desc: Chelsio T580-CR
dev.t5nex.0.hw_revision: 1
dev.t5nex.0.sn: PT13140042
dev.t5nex.0.pn: 110117150A0
dev.t5nex.0.ec: 0000000000000000
dev.t5nex.0.na: 0007432AF490
dev.t5nex.0.vpd_version: 3
dev.t5nex.0.scfg_version: 53255
dev.t5nex.0.bs_version: 1.1.0.0
dev.t5nex.0.er_version: 1.0.0.68
dev.t5nex.0.tp_version: 0.1.4.9
dev.t5nex.0.firmware_version: 1.16.2.0

Sponsored by: Chelsio Communications

Reorder sysctls so that nodes shared with the VF driver are added first.

This permits a single early return for VF devices in the routines that
add sysctl nodes.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7512

Add support for register dumps on VF devices.

- Add handling of VF register sets to t4_get_regs_len() and t4_get_regs().
- While here, use t4_get_regs_len() in the ioctl handler for regdump
instead of inlining it.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7484

Make SGE parameter handling more VF-friendly.

Add fields to hold the SGE control register and free list buffer sizes to
the sge_params structure. Populate these new fields in
t4_init_sge_params() for PF devices and change t4_read_chip_settings() to
pull these values out of the params structure instead of reading
registers directly. This will permit t4_read_chip_settings() to be reused
for VF devices which cannot read SGE registers directly.

While here, move the call to t4_init_sge_params() to
get_params__post_init(). The VF driver will populate the SGE parameters
structure via a different method before calling t4_read_chip_settings().

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7476

Add __printflike() to bus_describe_intr() to enable -Wformat checks.

Fix a few places that were passing a raw string as the format to use
a "%s" format string instead.

MFC after: 2 months

Use the port device name for the iov device for Chelsio T4/T5 cards.

Chelsio T4/T5 adapters are multifunction cards. The main driver uses
physical function 4 (PF4). However, VF devices for SR-IOV are only
supported on physical functions 0 through 3, where PF0 creates VFs tied
to port 0, etc. The t4iov/t5iov driver was previously added to
create VF devices for ports that are present on each adapter. This
change uses the recently added pci_iov_attach_name() function to
name the character device in /dev/iov after the associated port on
the card (e.g. /dev/iov/cxl0 is used to create VFs that share the
cxl0 port). With this in place, mark the t4iov/t5iov devices quiet
to prevent them from cluttering dmesg.

Reviewed by: rstone
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7402

Store the offset of the KDOORBELL and GTS registers in the softc.

VF devices use a different register layout than PF devices. Storing
the offset in a value in the softc allows code to be shared between the
PF and VF drivers.

Reviewed by: np
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D7389

Various fixes to the t4/5nex character device.

- Remove null open/close methods.
- Don't set d_flags to 0 explicitly.
- Remove t5_cdevsw as the .d_name member isn't really used and doesn't
warrant a separate cdevsw just for the name.
- Use ENOTTY as the error value for an unknown ioctl request.
- Use make_dev_s() to close race with setting si_drv1.

Sponsored by: Chelsio Communications

Add support for zero-copy aio_write() on TOE sockets.

AIO write requests for a TOE socket on a Chelsio T4+ adapter can now
DMA directly from the user-supplied buffer. This is implemented by
wiring the pages backing the user-supplied buffer and queueing special
mbufs backed by raw VM pages to the socket buffer. The TOE code
recognizes these special mbufs and builds a sglist from the VM page
array associated with the mbuf when queueing a work request to the TOE.

Because these mbufs do not have an associated virtual address, m_data
is not valid. Thus, the AIO handler does not invoke sosend() directly
for these mbufs but instead inlines portions of sosend_generic() and
tcp_usr_send().

An aiotx_buffer structure is used to describe the user buffer (e.g.
it holds the array of VM pages and a reference to the AIO job). The
special mbufs reference this structure via m_ext. Note that a single
job might be split across multiple mbufs (e.g. if it is larger than
the socket buffer size). The 'ext_arg2' member of each mbuf gives an
offset relative to the backing aiotx_buffer. The AIO job associated
with an aiotx_buffer structure is completed when the last reference to
the structure is released.

Zero-copy aio_write()'s for connections associated with a given
adapter can be enabled/disabled at runtime via the
'dev.t[45]nex.N.toe.tx_zcopy' sysctl.

MFC after: 1 month
Relnotes: yes
Sponsored by: Chelsio Communications

cxgbe(4): Initialize the adapter queues (fwq and mgmtq) instead of
returning EAGAIN if they aren't available when the user tries to program
a filter. Do this after validating the filter so that the driver
doesn't bring up the queues if it doesn't have to.

Add a driver to create VF devices on Chelsio T4/T5 NICs.

Chelsio NICs are a bit unique compared to some other NICs in that they
expose different functionality on different physical functions. In
particular, PF4 is used to manage the NIC interfaces ('t4nex' and 't5nex').
However, PF4 is not able to create VF devices. Instead, VFs are only
supported by physical functions 0 through 3. This commit adds 't4iov'
and 't5iov' drivers that attach to PF0-3.

One extra wrinkle is that the iov devices cannot enable SR-IOV until the
firwmare has been initialized by the main PF4 driver. To handle this
case, a new t4_if kobj interface has been added to permit cross-calls
between the PF drivers. The PF4 driver notifies sibling drivers when it
is fully attached. It also requests sibling drivers to detach before it
detaches. Sibling drivers query the PF4 driver during their attach
routine to see if it is attached. If not, the sibling drivers defer
their attach actions until the PF4 driver informs them it is attached.

VF devices are associated with a single port on the NIC. VF devices
created from PF0 are associated with the first port on the NIC, VFs
from PF1 are associated with the second port, etc. VF devices can
only be created from a PF device that has an associated port. Thus,
on a 2-port card, VFs are only supported on PF0 and PF1.

Reviewed by: np (earlier versions)
MFC after: 1 month
Sponsored by: Chelsio Communications

cxgbe(4): Add sysctl to display the RSS indirection table size for an
interface.

dev.cxl.<n>.rss_size
dev.vcxl.<n>.rss_size

MFC after: 3 days

cxgbe(4): Changes to the CPL-handler registration mechanism and code
related to "shared" CPLs.

a) Combine t4_set_tcb_field and t4_set_tcb_field_rpl into a single
function. Allow callers to direct the response to any iq. Tidy up
set_ulp_mode_iscsi while there to use names from t4_tcb.h instead of
magic constants.

b) Remove all CPL handler tables from struct adapter. This reduces its
size by around 2KB. All handlers are now registered at MOD_LOAD instead
of attach or some kind of initialization/activation. The registration
functions do not need an adapter parameter any more.

c) Add per-iq handlers to deal with CPLs whose destination cannot be
determined solely from the opcode. There are 2 such CPLs in use right
now: SET_TCB_RPL and L2T_WRITE_RPL. The base driver continues to send
filter and L2T_WRITEs over the mgmtq and solicits the reply on fwq.
t4_tom (including the DDP code) now uses the port's ctrlq to send
L2T_WRITEs and SET_TCB_FIELDs and solicits the reply on an ofld_rxq.
fwq and ofld_rxq have different handlers that know what kind of tid to
expect in the reply. Update t4_write_l2e and callers to to support any
wrq/iq combination.

Approved by: re@ (kib@)
Sponsored by: Chelsio Communications

cxgbe(4): Do not bring up an interface when IFCAP_TOE is enabled on it.
The interface's queues are functional after VI_INIT_DONE (which is short
of interface-up) and that's all that's needed for t4_tom to communicate
with the chip.

Approved by: re@ (gjb@)
Sponsored by: Chelsio Communications

cxgbe(4): Merge netmap support from the ncxgbe/ncxl interfaces to the
vcxgbe/vcxl interfaces and retire the 'n' interfaces. The main
cxgbe/cxl interfaces and tunables related to them are not affected by
any of this and will continue to operate as usual.

The driver used to create an additional 'n' interface for every
cxgbe/cxl interface if "device netmap" was in the kernel. The 'n'
interface shared the wire with the main interface but was otherwise
autonomous (with its own MAC address, etc.). It did not have normal
tx/rx but had a specialized netmap-only data path. r291665 added
another set of virtual interfaces (the 'v' interfaces) to the driver.
These had normal tx/rx but no netmap support.

This revision consolidates the features of both the interfaces into the
'v' interface which now has a normal data path, TOE support, and native
netmap support. The 'v' interfaces need to be created explicitly with
the hw.cxgbe.num_vis tunable. This means "device netmap" will not
result in the automatic creation of any virtual interfaces.

The following tunables can be used to override the default number of
queues allocated for each 'v' interface. nofld* = 0 will disable TOE on
the virtual interface and nnm* = 0 to will disable native netmap
support.

# number of normal NIC queues
hw.cxgbe.ntxq_vi
hw.cxgbe.nrxq_vi

# number of TOE queues
hw.cxgbe.nofldtxq_vi
hw.cxgbe.nofldrxq_vi

# number of netmap queues
hw.cxgbe.nnmtxq_vi
hw.cxgbe.nnmrxq_vi

hw.cxgbe.nnm{t,r}xq{10,1}g tunables have been removed.

--- tl;dr version ---
The workflow for netmap on cxgbe starting with FreeBSD 11 is:
1) "device netmap" in the kernel config.
2) "hw.cxgbe.num_vis=2" in loader.conf. num_vis > 2 is ok too, you'll
end up with multiple autonomous netmap-capable interfaces for every
port.
3) "dmesg | grep vcxl | grep netmap" to verify that the interface has
netmap queues.
4) Use any of the 'v' interfaces for netmap. pkt-gen -i vcxl<n>... .
One major improvement is that the netmap interface has a normal data
path as expected.
5) Just ignore the cxl interfaces if you want to use netmap only. No
need to bring them up. The vcxl interfaces are completely independent
and everything should just work.
---------------------

Approved by: re@ (gjb@)
Relnotes: Yes
Sponsored by: Chelsio Communications

cxgbe(4): A couple of fixes to set_sched_queue.

- Validate the scheduling class against the actual limit (which is chip
specific) instead of a magic number.

- Return an error if an attempt is made to manipulate the tx queues of a
VI that hasn't been initialized.

Sponsored by: Chelsio Communications

cxgbe(4): Provide information about traffic classes in the sysctl mib.

Sponsored by: Chelsio Communications

cxgbe(4): Track the state of the hardware traffic schedulers in the
driver. This works as long as everyone uses set_sched_class_params
to program them.

Sponsored by: Chelsio Communications

cxgbe(4): Break up set_sched_class. Validate the channel number and
min/max rates against their actual limits (which are chip and port
specific) instead of hardcoded constants.

Sponsored by: Chelsio Communications

Use DDP to implement zerocopy TCP receive with aio_read().

Chelsio's TCP offload engine supports direct DMA of received TCP payload
into wired user buffers. This feature is known as Direct-Data Placement.
However, to scale well the adapter needs to prepare buffers for DDP
before data arrives. aio_read() is more amenable to this requirement than
read() as applications often call read() only after data is available in
the socket buffer.

When DDP is enabled, TOE sockets use the recently added pru_aio_queue
protocol hook to claim aio_read(2) requests instead of letting them use
the default AIO socket logic. The DDP feature supports scheduling DMA
to two buffers at a time so that the second buffer is ready for use
after the first buffer is filled. The aio/DDP code optimizes the case
of an application ping-ponging between two buffers (similar to the
zero-copy bpf(4) code) by keeping the two most recently used AIO buffers
wired. If a buffer is reused, the aio/DDP code is able to reuse the
vm_page_t array as well as page pod mappings (a kind of MMU mapping the
Chelsio NIC uses to describe user buffers). The generation of the
vmspace of the calling process is used in conjunction with the user
buffer's address and length to determine if a user buffer matches a
previously used buffer. If an application queues a buffer for AIO that
does not match a previously used buffer then the least recently used
buffer is unwired before the new buffer is wired. This ensures that no
more than two user buffers per socket are ever wired.

Note that this feature is best suited to applications sending a steady
stream of data vs short bursts of traffic.

Discussed with: np
Relnotes: yes
Sponsored by: Chelsio Communications

sys/dev: minor spelling fixes.

Most affect comments, very few have user-visible effects.

Cleanup unnecessary semicolons from the kernel.

Found with devel/coccinelle.

Add a 'show t4 devlog <nexus>' DDB command.

This command displays the adapter's firmware device log similar to the
dev.<nexus>.misc.devlog sysctl.

Sponsored by: Chelsio Communications

Add a 'show t4 tcb <nexus> <tid>' command to dump a TCB from DDB.

This allows the contents of a TCB to be extracted from a T4/T5 card in
DDB after a panic.

cxgbe(4): Tidy up PAUSE frame accounting.

Figure out if the chip is counting PAUSE frames in the "normal" stats
and take them out if it is. This fixes a bug in the tx stats because
the default hardware behavior is different for Tx and Rx but the driver
was treating both the same way. The result was that OPACKETS, OBYTES,
and OMCASTS were under-reported (if tx_pause > 0) before this change.

Note that the mac_stats sysctl still gives you the raw value of these
statistics straight from the device registers.

cxgbe(4): Enable additional capabilities in the default configuration
files. All features with FreeBSD drivers of some kind are now in the
default configuration.

cxgbe(4): Remove a couple of pointless assignments in sysctl_meminfo.
Do not display range if start = stop (this is a workaround for some
unused regions).