Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Re-enable IFCAP_TSO6 in cxgb(4) and cxgbe(4) in stable/9. The kernel
changes needed for all this to work have now been MFC'd to 9 by bz@.

This is a direct commit to stable/9 that removes earlier changes made to
drivers in this branch only.

Approved by: re (kib)

MFC r235944:

Significantly update tcp_lro for mostly two things:
1) introduce basic support for IPv6 without extension headers.
2) try hard to also get the incremental checksum updates right,
especially also in the IPv4 case for the IP and TCP header.

Move variables around for better locality, factor things out into
functions, allow checksum updates to be compiled out, ...

Leave a few comments on further things to look at in the future,
though that is not the full list.

Update drivers with appropriate #includes as needed for IPv6 data
type in LRO.

Approved by: re

MFC r237832, r237436, r237439, r237463, r237512, r237587, r237799,
r237819, r237831.

r237832:
cxgb(4): IPv6 rx/tx hw checksum, IPv6 TSO and LRO too.

r237436:
cxgbe(4): update to firmware interface 1.5.2.0; updates to shared code.

r237439:
Do not read registers with read side effects while performing a register
dump for cxgbetool.

r237463:
Do not allocate extra vectors when adapter is not TOE
capable (or toecaps have been disallowed by the user).

r237512:
Better way to determine the status page length and rx pad boundary.

r237587:
Allow cxgbe(4) running within a VM to attach to its devices that have been
exported via PCI passthrough.

r237799:
cxgbe(4): support for IPv6 hardware checksumming (rx and tx).

r237819:
cxgbe(4): support for IPv6 TSO and LRO.

r237831:
- Assign (don't OR) the CSUM_XXX bits to csum_flags in the rx checksum code.
- Fix TSO/TSO4 mixup.
- Add IFCAP_LINKSTATE to the available/enabled capabilities.

Backport just the sys/{dev,modules}/cxgb{,e}/ parts of r237263, and then
disable the TOE and iWARP modules in the Makefiles (they won't compile
without the rest of r237263).

This reduces diffs between the cxgb/cxgbe drivers in head and 9 and
makes it easy to MFC other fixes to 9.

MFC r231116:
Remove if_start from cxgb and cxgbe.

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

t3_free_sge_resources should be given the number of qsets it needs to free.

MFC after: 1 week

Improve cxgb(4)'s behaviour when faced with temporarily "bouncy" links:
- Run the adapter's tick at 1Hz and remove link state checks from it.
Instead, have each port check its link state. Delay the check so that
it takes place slightly after the driver is notified of a change in
link state. This is a cheap way to debounce these notifications if
many are received in rapid succession. POLL_LINK_1ST_TIME flag can
also be eliminated as a side effect of these changes.
- Do not reset the PHY when link goes down.
- Clear port's link_fault flag if the PHY indicates link is down.
- get_link_status_r should leave speed and duplex alone when link is down.

MFC after: 1 month

Eliminate ext_intr_task. The "slow" interrupt handler is already
running on the adapter's task queue. Just do what the task does
instead of enqueueing it.

MFC after: 3 days

cxgb(4): add knob to get packet timestamps from the hardware.

The T3 ASIC can provide an incoming packet's timestamp instead of its RSS hash.
The timestamp is just a counter running off the card's clock. With a 175MHz
clock an increment represents ~5.7ns and the 32 bit value wraps around in ~25s.

# sysctl -d dev.cxgbc.0.pkt_timestamp
dev.cxgbc.0.pkt_timestamp: provide packet timestamp instead of connection hash

# sysctl -d dev.cxgbc.0.core_clock
dev.cxgbc.0.core_clock: core clock frequency (in KHz)
# sysctl dev.cxgbc.0.core_clock
dev.cxgbc.0.core_clock: 175000

Don't ring the tx doorbell for every frame when we know more frames
will follow. Adjust the freelist and response queue doorbells too.

Discussed with: kmacy

Increase response queue size to avoid starvation, add a counter
to track it when it does occur.

Multiple fixes related to queue set sizing and resources:

- Only the tunnelq (TXQ_ETH) requires a buf_ring, an ifq, and the watchdog/timer
callouts. Do not allocate these for the other tx queues.

- Use 16k jumbo clusters only on offload capable cards by default.

- Do not allocate a full tx ring for the offload queue if the card is not
offload capable.

- Slightly better freelist size calculation.

- Fix nmbjumbo4 typo, remove unneeded global variables.

MFC after: 3 days

Fix tx drop statistics.

MFC after: 3 days

There is no need to test __FreeBSD_version for features that have
been around for a long time now (7.1-ish or even earlier); assume
they are present. These includes MSI, TSO, LRO, VLAN, INTR_FILTERS,
FIRMWARE, etc.

Also, eliminate some dead code and clean up in other places as part
of this quick once-over.

MFC after: 1 week

Fix for a cxgb(4) panic. cxgb_ioctl can be called by the IP and IPv6
layers with non-sleepable locks held. Don't (potentially) sleep in
those situations.

cxgb(4) updates, including:
- support for the new Gen-2, BT, and LP-CR cards.
- T3 firmware 7.7.0
- shared "common code" updates.

Approved by: gnn (mentor)
Obtained from: Chelsio
MFC after: 1 month

Use if_maddr_rlock() instead of IF_ADDR_LOCK() to protect access to
if_multiaddrs in if_cxgb.

Approved by: re (kib)
MFC after: 6 weeks

Greatly simplify cxgb by removing almost all of the custom mbuf management logic

- remove mbuf iovec - useful, but adds too much complexity when isolated to
the driver

- remove driver private caching - insufficient benefit over UMA to justify
the added complexity and maintenance overhead

- remove separate logic for managing multiple transmit queues, with the
new drbr routines the control flow can be made to much more closely resemble
legacy drivers

- remove dedicated service threads, with per-cpu callouts one can get the same
benefit much more simply by registering a callout 1 tick in the future if there
are still buffered packets

- remove embedded mbuf usage - Jeffr's changes will (I hope) soon be integrated
greatly reducing the overhead of using kernel APIs for reference counting
clusters

- add hysteresis to descriptor coalescing logic

- add coalesce threshold sysctls to allow users to decide at run-time
between optimizing for forwarding / UDP or optimizing for TCP

- add once per second watchdog to effectively close the very rare races
occurring from coalescing

- incorporate Navdeep's changes to the initialization path required to
convert port and adapter locks back to ordinary mutexes (silencing BPF
LOR complaints)

- enable prefetches in get_packet and tx cleaning

Reviewed by: navdeep@
MFC after: 2 weeks

Integrate three changes from Chelsio.

1) Add a sysctl that will say what type of PHYs exist on the card.
2) Fix a bug that occurs when an AEL 2005 PHY resets without a transciever
in the card.
3) Unify the PHY link detection code.

Obtained from: Navdeep Parhar
MFC after: 10 days

Update the Chelsio driver to the latest bits from Chelsio

Firmware upgraded to 7.1.0 (from 5.0.0).
T3C EEPROM and SRAM added; Code to update eeprom/sram fixed.
fl_empty and rx_fifo_ovfl counters can be observed via sysctl.
Two new cxgbtool commands to get uP logic analyzer info and uP IOQs
Synced up with Chelsio's "common code" (as of 03/03/09)

Submitted by: Navdeep Parhar at Chelsio
Reviewed by: gnn
MFC after: 2 weeks

- enable multiple transmit queues
- invert sense of hw.cxgb.singleq tunable to hw.cxgb.multiq
- don't wake up transmitting thread by default
- add per tx queue ifaltq to handle ALTQ
- remove several unused functions in cxgb_multiq.c
- add several sysctls: multiq_tx_enable, coalesce_tx_enable,
and wakeup_tx_thread
- this obsoletes the hw.cxgb.snd_queue_len as ifq is replaced
by a buf_ring

- bump __FreeBSD version to reflect added buf_ring, memory barriers,
and ifnet functions

- add memory barriers to <machine/atomic.h>
- update drivers to only conditionally define their own

- add lockless producer / consumer ring buffer
- remove ring buffer implementation from cxgb and update its callers

- add if_transmit(struct ifnet *ifp, struct mbuf *m) to ifnet to
allow drivers to efficiently manage multiple hardware queues
(i.e. not serialize all packets through one ifq)
- expose if_qflush to allow drivers to flush any driver managed queues

This work was supported by Bitgravity Inc. and Chelsio Inc.

- Remove default NIC dependency on ulp headers
- make toe module build dependent on kernel support

Submitted by: Chelsio Inc.
MFC after: 1 week

Update cxgb include paths to not require prefixing with dev/cxgb

Submitted by: Chelsio Inc.

Allow cxgb to be unified across versions by making newer features conditional

Submitted by: Chelsio Inc
MFC after: 3 days

White space cleanups to bring closer to RELENG_7

Remove some dead code along with gratuitous differences between HEAD and 7

Fix issue with recovering from transient jumbo mbuf shortage.

Submitted by: Chelsio Inc.
MFC after: 3 days

Add LRO and MAC statistics to exported sysctls.

Obtained from: Chelsio Inc.
MFC after: 1 week

Remove cxgb private lro implementation and switch to using system implementation.

Obtained from: Chelsio Inc.
MFC after: 1 week

Vendor fix for PHY problem.

Obtained from: Chelsio Inc.
MFC after: 3 days

import vendor fixes to cxgb

reduce the size of the jumbo ring on i386 and disable pcpu cluster caching

- update firmware to 5.0
- add support for T3C
- add DDP support (zero-copy receive)
- fix TOE transmit of large requests
- fix shutdown so that sockets don't remain in CLOSING state indefinitely
- register listeners when an interface is brought up after tom is loaded
- fix setting of multicast filter
- enable link at device attach
- exit tick handler if shutdown is in progress
- add helper for logging TCB
- add sysctls for dumping transmit queues

- note that TOE wxill not be MFC'd until after 7.0 has been finalized

MFC after: 3 days

put tx queue size back to 1024

- Simplify mb_free_ext_fast
- increase asserts for mbuf accounting
- track outstanding mbufs (maps very closely to leaked)
- actually only create one thread per port if !multiq
Oddly enough this fixes the use after free

- move txq_segs to stack in t3_encap
- add checks that pidx doesn't move pass cidx
- simplify mbuf free logic in collapse mbufs routine

- move WR_LEN in to cxgb_adapter.h add PIO_LEN to make intent clearer
- move cxgb_tx_common in to cxgb_multiq.c and rename to cxgb_tx
- move cxgb_tx_common dependencies
- further simplify cxgb_dequeue_packet for the non-multiqueue case
- only launch one service thread per port in the non-multiq case
- remove dead cleaning code from cxgb_sge.c
- simplify PIO case substantially in by returning directly from mbuf collapse
and just using m_copydata
- remove gratuitous m_gethdr in the rx path
- clarify freeing of mbufs in collapse

Convert over to using the multiqueue infrastructure although all calls going
through cxgb_start still end up using queue 0

Be more aggressive about tx cleaning - when multiples streams were running the tx
queue could fill up and stop getting cleaned.

Add support for selectively dumping the state of the hardware response queue.
Change ordering of a couple of types.

make nqsets a uint32_t so that sysctl will work
add 2 fields for allowing queue dumping

Make TCP offload work on HEAD (modulo negative interaction between sbcompress
and t3_push_frames).
- Import latest changes to cxgb_main.c and cxgb_sge.c from toestack p4 branch
- make driver local copy of tcp_subr.c and tcp_usrreq.c and override tcp_usrreqs so
TOE can also functions on versions with unmodified TCP

- add cxgb back to the build

Import updated support code for the TOM (tcp offload module).

pull in changes made to RELENG_6 version in the process of doing the MFC

Supported by: Chelsio
Approved by: re (blanket)

Add back in support for normal mbuf chaining on RX under DISABLE_MBUF_IOVEC

Approved by: re(blanket)
Supported by: Chelsio

- fix qset to port binding as a proper fix for the problems encountered on the 4-port
- fix the use after free seen when sending packets small enough to fit as an immediate
and bpf peers are present
- update to firmware rev 4.7 along with various small vendor fixes

Supported by: Chelsio
Approved by: re (blanket)
MFC after: 3 days

Fixes for 4 port and small packet optimization

- remove cpl->iff panic - we can't know the port number from the rspq on the 4-port
- pick the ifnet based on the interface in the CPL header
- switch to using qset 0 for egress on the 4-port for now - may change
when we start using RSS
- move ether_ifdetach to before the port lock gets deinitialized to avoid
hang in the case where there are BPF peers (cxgb_ioctl is called indirectly
when BPF peers are present)
- don't call t3_mac_reset if multiport is set, this was causing tx errors
by misconfiguring the MAC on the 4-port
- change V_TXPKT_INTF to use txpkt_intf as the interfaces are not contiguous
- free the mbuf immediately in the case where the payload is small enough to be copied
into the rspq
- only update the coalesce timer if for a queue if packets were taken off of it
- add in missed 20ms DELAY in initializaton vsc8211

- prompt MFC as this only applies to the 4-port which is currently completely
broken - OK'd by kensmith

Supported by: Chelsio
Approved by: re (blanket)
MFC after: 0 days

forward port signedness fixes from RELENG_6
fix compile error for case where MSI_SUPPORTED not defined

Approved by: re (blanket)

- In all structures other than port info port is a pointer to a port info,
make the code less confusing by renaming the port number to port_id

Approved by: re (blanket)

- integrate most recent changes from vendor branch and upgrade to firmware revision 4.5.5
- add filter support
- further improvements for T304
- recover gracefully from spurious immediate packets

Approved by: re(blanket)
Supported by: Chelsio
MFC after: 3 days

MFp4 122896
- reduce cpu usage by as much as 25% (40% -> 30) by doing txq reclaim more efficiently
- use mtx_trylock when trying to grab the lock to avoid spinning during long encap loop
- add per-txq reclaim task
- if mbufs were successfully re-claimed try another pass
- track txq overruns with sysctl

Approved by: re (blanket)

- switch adapter and port lock over to using sx so that resources
can be allocated atomically
- add debug macros for printing lock initialization / teardown
- add buffers to port_info and adapter to allow each lock to have a
unique name
- destroy mutexes initialized by cxgb_offload_init
- remove recursive calls to ADAPTER_LOCK
- move callout_drain calls so that they don't occur with the lock held
- ensure that only as many qsets as are needed are initialized and
destroyed

MFC after: 3 days
Sponsored by: Chelsio Inc.

- import new common code for the T304
- update to firmware version 4.1.0

- switch over to standard method for initializing cdevs (contributed by scottl@)
- break out timer_reclaim_task to be per-port
- move msix teardown into separate function
- fix bus_setup_intr for msi-x for the multi-port case so that msi-x resources
are not corrupted on unload
- handle 10/100/1000 base-T media and auto negotiation
- bind qset to cpu even for singleq case
- white space cleanups
- remove recursive PORT_LOCK
- move mtu setting to separate function
- stop and re-init port when changing mtu
- replace all direct references to m_data with calls to mtod
- handle attach failure better by not trying to de-initialize
taskqueues when they have not been allocated
- no longer default to jumbo frames

Sponsored by: Chelsio
MFC after: 3 days

When building cxgb as a module make include paths relative to the driver's root.
This will make it possible to build the module out of tree against an older src tree.

MFC after: 3 days

(MFp4)
- upgrade to reflect state of 1.0.0.86
- move from firmware rev 3.2 to 4.0.0
- import driver bits for offload functionality
- remove binary distribution clause from top level files as it
runs counter to the intent of purely supporting the hardware

MFC after: 3 days

Convert driver RX path over to using mbuf iovec

Make DMA tags per-queue to facilate parallel mappings
Defer mbuf allocation and initialization until after data has already been
received in a cluster

This reduces cpu utilization somewhat, but it only improves the rx path.
Recent changes to TCP appear to make us rate limited by the TX path.

This is the first step in reducing mbuf management overhead for manipulating
clusters.

MFC after: 3 days

bus_size_t is a bad cross-architectural type with respect to printf, use uint32_t instead

commit missed change

make MSI-X the default and allocate up to mp_ncpus queues per port

MFC after: 3 days

Synchronize with version 1.0.071 of Chelsio's common code
(with the notable exception of improvements for using multiple TX queues)

This adds support for the T3B2 ASIC rev

Obtained from: Chelsio
MFC after: 3 days

make desc_reclaimable macro safe to arbitrary arguments

First of several commits for driver support for the Chelsio T3B 10 Gigabit
Ethernet adapter.

Reviewed by: scottl, sam

For those interested in the preliminary performance work see below.

Plots of mxge vs. cxgb running netpipe:

blocksize vs. bandwidth:
http://www.fsmware.com/chelsio.random/bsvsbw.gif

blocksize vs. RTT:
First of several commits for driver support for the Chelsio T3B 10 Gigabit
Ethernet adapter.

Reviewed by: scottl, sam

For those interested in the preliminary performance work see below.

Plots of mxge vs. cxgb running netpipe:

blocksize vs. bandwidth:
http://www.fsmware.com/chelsio.random/bsvsbw.gif

blocksize vs. RTT:
http://www.fsmware.com/chelsio.random/bsvstime.gif

blocksize vs. RTT for block sizes <= 10kb:
http://www.fsmware.com/chelsio.random/bsvstime_10kb.gif
http://www.fsmware.com/chelsio.random/bsvstime_10kb3.gif