MFC r341785

sfxge(4): use n Tx queues instead of n + 2 on EF10 HW

On EF10 HW we can avoid sending packets without checksum offload
or with IP-only checksum offload to dedicated queues. Instead, we
can use option descriptors to change offload policy on any queue
during runtime. Thus, we don't need to create two dedicated queues.

Submitted by: Ivan Malov <Ivan.Malov at oktetlabs.ru>
Sponsored by: Solarflare Communications, Inc.
Differential Revision: https://reviews.freebsd.org/D18390

MFC r341784

sfxge(4): prepare the number of Tx queues on event queue 0 to become
variable

The number of Tx queues on event queue 0 can depend on the NIC family
type, and this property will be leveraged by future patches.
This patch prepares the code for this change.

Submitted by: Ivan Malov <Ivan.Malov at oktetlabs.ru>
Sponsored by: Solarflare Communications, Inc.
Differential Revision: https://reviews.freebsd.org/D18389

MFC r341327

sfxge(4): rollback last seen VLAN TCI if Tx packet is dropped

Early processing of a packet on transmit may change last seen
VLAN TCI in the queue context. If such a packet is eventually
dropped, last seen VLAN TCI must be set to its previous value.

Submitted by: Ivan Malov <Ivan.Malov at oktetlabs.ru>
Sponsored by: Solarflare Communications, Inc.
Differential Revision: https://reviews.freebsd.org/D18288

MFC r312883

sfxge(4): fix invalid VLAN tagging after stop/start

TxQ is destroyed on stop and last used tag should be reset to default 0
on the next start.

Sponsored by: Solarflare Communications, Inc.
Differential Revision: https://reviews.freebsd.org/D9358

MFC r312866

sfxge(4): cleanup: remove unused txq_index TxQ control structure member

Sponsored by: Solarflare Communications, Inc.

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

MFC r311877

sfxge(4): avoid unnecessary mbuf data prefetch

Unnecessary prefetch just loads HW prefetcher and displaces other
cache entries (which could be really useful).

If we parse mbuf for TSO early and use firmware-assisted TSO, we do not
expect mbuf data access when we compose firmware-assisted TSO (v1 or v2)
option descriptors. If packet header needs to be linearized or finally
FATSO cannot be used because of, for example, too big header, we do not
care about a bit more performance degradation because of prefetch
absence (it is better to optimize more common case).

Sponsored by: Solarflare Communications, Inc.

MFC r311640

sfxge(4): allow DMA descs to cross 4k boundary on EF10

Siena has limitation on maximum byte count and 4k boundary crosssing
(which is stricter than maximum byte count).
EF10 has limitation on maximum byte count only.

Sponsored by: Solarflare Communications, Inc.

MFC r310810

sfxge(4): cleanup: check deferred packet list tunables once

Sponsored by: Solarflare Communications, Inc.

MFC r310770

sfxge(4): cleanup: clarify/unify variable name used for put-list length

get_count is used for get-list.

Sponsored by: Solarflare Communications, Inc.

MFC r310627

sfxge(4): do not limit driver RSS table to RSS channels max

Specification of entire RSS table in the driver allows to spread traffic
more equally across CPUs/RSS channels if number of RSS channels is not
power of 2.

Sponsored by: Solarflare Communications, Inc.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

sfxge(4): host byte order is required for IP ID in TSO descriptors

Submitted by: Artem V. Andreev <Artem.Andreev at oktetlabs.ru>
Sponsored by: Solarflare Communications, Inc.
MFC after: 1 week

sfxge(4): pick an RSS bucket for the packet enqueued and select TXQ accordingly

Submitted by: Ivan Malov <Ivan.Malov at oktetlabs.ru>
Reviewed by: gnn
Sponsored by: Solarflare Communications, Inc.
Differential Revision: https://reviews.freebsd.org/D6723

sfxge(4): cleanup: update copyright to 2016

Sponsored by: Solarflare Communications, Inc.
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D6509

sys/dev: minor spelling fixes.

Most affect comments, very few have user-visible effects.

sys/dev: extend use of the howmany() macro when available.

We have a howmany() macro in the <sys/param.h> header that is
convenient to re-use as it makes things easier to read.

These files were getting sys/malloc.h and vm/uma.h with header pollution
via sys/mbuf.h

sfxge: support FATSOv2

Reviewed by: gnn
Sponsored by: Solarflare Communications, Inc.
MFC after: 2 days
Differential Revision: https://reviews.freebsd.org/D4934

sfxge: switch to TxQ creation specific flags

It is better do not mix TxQ creation and receive event flags since only
checksum flags are applicable to TxQ.
Also it will allow to add a new TxQ creation specific flags.

Reviewed by: gnn
Sponsored by: Solarflare Communications, Inc.
MFC after: 2 days
Differential Revision: https://reviews.freebsd.org/D4389

sfxge: parse packets for TSO early in if_transmit

Submitted by: Artem V. Andreev <Artem.Andreev at oktetlabs.ru>
Sponsored by: Solarflare Communications, Inc.
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D4309

sfxge: add 7xxx NICs family support

Support 7xxx adapters including firmware-assisted TSO and VLAN tagging:

- Solarflare Flareon Ultra 7000 series 10/40G adapters:
- Solarflare SFN7042Q QSFP+ Server Adapter
- Solarflare SFN7142Q QSFP+ Server Adapter

- Solarflare Flareon Ultra 7000 series 10G adapters:
- Solarflare SFN7022F SFP+ Server Adapter
- Solarflare SFN7122F SFP+ Server Adapter
- Solarflare SFN7322F Precision Time Synchronization Server Adapter

- Solarflare Flareon 7000 series 10G adapters:
- Solarflare SFN7002F SFP+ Server Adapter

Support utilities to configure adapters and update firmware.

The work is done by Solarflare developers
(Andy Moreton, Andrew Lee and many others),
Artem V. Andreev <Artem.Andreev at oktetlabs.ru> and me.

Sponsored by: Solarflare Communications, Inc.
MFC after: 2 weeks
Causually read by: gnn
Differential Revision: https://reviews.freebsd.org/D2618

sfxge: relax assertion to allow RST flag in TSO packets

Kernel under stress load, mixed MC reboot and sfupdate really
generates TSO packet with RST flag.
It will generate many TCP packets with RST flag set.
May be RST flag should be set in the last segment only, but it could be
dropped. So, it is safer to keep the flag in all packets to be sure that
connection is reset.

Reviewed by: gnn
Sponsored by: Solarflare Communications, Inc.
MFC after: 2 days
Differential Revision: https://reviews.freebsd.org/D2609

sfxge: fix overflow queue freeze

If TxQ lock is obtained, deferred packet list shold be serviced even if
the packet addition fails because of overflow.

Without the patch freeze happens if:
- queue is not blocked (i.e. completion does not trigger unblock and service)
- put-list overflow (1024 entries)
- sfxge_tx_packet_add() acquires TxQ lock just as it is released it in
sfxge_tx_qdpl_service() on the second CPU but before pending check
- sfxge_tx_packet_add() swizzles put-list to get-list, fails because of
non-tcp get-list overflow and returns without packet list service
- sfxge_tx_qdpl_service() on the second CPU checks that there are no
pending packets in the put-list and returns

Other possible solution is to guaranee that maximum length of the put-list
is less than maximum length of any get-list.

Reviewed by: gnn
Sponsored by: Solarflare Communications, Inc.
MFC after: 2 days
Differential Revision: https://reviews.freebsd.org/D2562

sfxge: move mbuf free to sfxge_if_transmit()

It is a preparation to the next patch which will service packet queue even
if packet addtion fails.

Reviewed by: gnn
Sponsored by: Solarflare Communications, Inc.
MFC after: 2 days
Differential Revision: https://reviews.freebsd.org/D2552

sfxge: get rid of locked variable in sfxge_tx_packet_add()

Now each branch has one and only one possible TxQ lock state.
It simplifies understanding of the code.

Reviewed by: gnn
Sponsored by: Solarflare Communications, Inc.
MFC after: 2 days
Differential Revision: https://reviews.freebsd.org/D2551

sfxge: split sfxge_tx_qdpl_put() into *_locked() and *_unlocked()

It simplifies understanding of the sfxge_tx_packet_add() logic and
avoids passing of 'locked' to called function.

Reviewed by: gnn
Sponsored by: Solarflare Communications, Inc.
MFC after: 2 days
Differential Revision: https://reviews.freebsd.org/D2547

sfxge: advertise IPv6 Rx and Tx checksum offload support

Tx checksum offload may be enabled/disabled.

Sponsored by: Solarflare Communications, Inc.
MFC after: 2 days
Differential Revision: https://reviews.freebsd.org/D2543

sfxge: fix bug in TSO when a DMA segment has both header and data

Reviewed by: gnn
Sponsored by: Solarflare Communications, Inc.
MFC after: 2 days
Differential Revision: https://reviews.freebsd.org/D2163

sfxge: cleanup: fix index variable type to match upper boundary type

Sponsored by: Solarflare Communications, Inc.

sfxge: add statistics for each Tx queue

Sponsored by: Solarflare Communications, Inc.
Differential Revision: https://reviews.freebsd.org/D2082

sfxge: remove obsolete Tx non-multi queue support

Tx multi queue is added in FreeBSD 8.0. So, the changeset drops earlier
versions support.

Sponsored by: Solarflare Communications, Inc.
Differential Revision: https://reviews.freebsd.org/D2081

sfxge: assert either kernel or internal copy of interface flags

ioctl to put interface down sets ifp->if_flags which holds the intended
administratively defined state and calls driver callback to apply it.
When everything is done, driver updates internal copy of
interface flags sc->if_flags which holds the operational state.
So, transmit from Rx path is possible when interface is intended to be
administratively down in accordance with ifp->if_flags, but not applied
yet and the operational state is up in accordance with sc->if_flags.

Sponsored by: Solarflare Communications, Inc.
Differential Revision: https://reviews.freebsd.org/D2075

sfxge: prefetch txq->common if TxQ is started only

Transmit may be called when TxQ is not started yet (i.e. txq->common is
invalid). TxQ state is checked below when mbuf is processed and dropped
if TxQ is not started.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: move deferred packet list statistics to dedicated node

It is done to structure sysctl and do not mix with Tx queue statistics
to be added.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: expect required init_state on data path and in periodic calls

With the patch applied the number of instruction events is 1% less and
number of mispredicted branch events is 5% less under multistream TCP
traffic load close to line rate.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: add put-list high watermark

It is interesting to know how long put-list grows.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: implement if_get_counter callback

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: DMA allocated memory is set to zeros because of BUS_DMA_ZERO flag

It is not required to set it to zeros once again.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: TxQ block level should use EFX_TXQ_LIMIT as maximum TxQ size

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: packet can't require more than n_dma_seg descriptors plus one

Use remaining number of DMA segment instead of maximum number in mapping
when checking space for one more TSO segment packet.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: style fixes and cleanup

Sync endif comment with conditional.
BOOTROM and SIENA_BOOTROM are the same, but highlight that it is Siena.
Restore commented out assertion.
Sync comments with out-of-tree driver.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: more accurate calculation of maximum number of TSO segments

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: reap Tx descriptors on unblock

Otherwise when processing finally comes to efx_tx_qdesc_post() it could
be insufficient space between reaped and added to post pending
descriptors.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: style fixes

Use nitem() to get number of array elements.
Remove unused define.
Use TAB to indent.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: handle fragmented TCP header in mbuf

TCP header is fragmented in the case of VLAN tagged IPv6 traffic without
HW VLAN tagging.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: add driver context member with number of transmit queues

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: add TCP segment size to sfxge_tso_state

It avoids access to m_pkthdr when TSO packet is started and also makes
tso_start_new_packet() function smaller.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: flush all Tx queues from if_qflush

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor), glebius

sfxge: remove inline specifiers

Now compiler does not need any help.
The patch does not change generated code.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor), glebius

sfxge: remove used sfxge_tso_state member dma_seg_i

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: remove full_packet_size from sfxge_tso_state

It makes sfxge_tso_state smaller and even makes tso_start_new_packet()
few bytes smaller. Data used to calculate packet size are used nearby,
so it should be no problems with cache etc.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor), glebius

sfxge: remove unused variable

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor), glebius

sfxge: Add statistics for partially dropped TSO packets

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: make lock names unique

Lock name should include interface name.
Tx queue and event queue lock name should include queue number.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: Add macros to init, destroy, acquire, release and assert locks

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: Separate software Tx queue limit for non-TCP traffic

Add separate software Tx queue limit for non-TCP traffic to make total
limit higher and avoid local drops of TCP packets because of no
backpressure.
There is no point to make non-TCP limit high since without backpressure
UDP stream easily overflows any sensible limit.

Split early drops statistics since it is better to have separate counter
for each drop reason to make it unabmiguous.

Add software Tx queue high watermark. The information is very useful to
understand how big queues grow under traffic load.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: Add evq argument to sfxge_tx_qcomplete()

It removes necessity to get evq pointer by its index in soft context.

Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

sfxge: fixed TSO code to cope with VLAN headers

Submitted by: Artem V. Andreev <Artem.Andreev at oktetlabs.ru>
Sponsored by: Solarflare Communications, Inc.
Approved by: gnn (mentor)

Start process of removing the use of the deprecated "M_FLOWID" flag
from the FreeBSD network code. The flag is still kept around in the
"sys/mbuf.h" header file, but does no longer have any users. Instead
the "m_pkthdr.rsstype" field in the mbuf structure is now used to
decide the meaning of the "m_pkthdr.flowid" field. To modify the
"m_pkthdr.rsstype" field please use the existing "M_HASHTYPE_XXX"
macros as defined in the "sys/mbuf.h" header file.

This patch introduces new behaviour in the transmit direction.
Previously network drivers checked if "M_FLOWID" was set in "m_flags"
before using the "m_pkthdr.flowid" field. This check has now now been
replaced by checking if "M_HASHTYPE_GET(m)" is different from
"M_HASHTYPE_NONE". In the future more hashtypes will be added, for
example hashtypes for hardware dedicated flows.

"M_HASHTYPE_OPAQUE" indicates that the "m_pkthdr.flowid" value is
valid and has no particular type. This change removes the need for an
"if" statement in TCP transmit code checking for the presence of a
valid flowid value. The "if" statement mentioned above is now a direct
variable assignment which is then later checked by the respective
network drivers like before.

Additional notes:
- The SCTP code changes will be committed as a separate patch.
- Removal of the "M_FLOWID" flag will also be done separately.
- The FreeBSD version has been bumped.

MFC after: 1 month
Sponsored by: Mellanox Technologies

Support tunable to control Tx deferred packet list limits

Also increase default for Tx queue get-list limit.
Too small limit results in TCP packets drops especiall when many
streams are running simultaneously.
Put list may be kept small enough since it is just a temporary
location if transmit function can't get Tx queue lock.

Submitted by: Andrew Rybchenko <arybchenko at solarflare.com>
Sponsored by: Solarflare Communications, Inc.

The patch allows to check state of the software Tx queues at run time.

Submitted by: Andrew Rybchenko <arybchenko at solarflare.com>
Sponsored by: Solarflare Communications, Inc.

Make size of Tx and Rx rings configurable

Required size of event queue is calculated now.

Submitted by: Andrew Rybchenko <arybchenko at solarflare.com>
Sponsored by: Solarflare Communications, Inc.

cleanup: code style fixes

Remove trailing whitespaces and tabs.
Enclose value in return statements in parentheses.
Use tabs after #define.
Do not skip comparison with 0/NULL in boolean expressions.

Submitted by: Andrew Rybchenko <arybchenko at solarflare.com>
Sponsored by: Solarflare Communications, Inc.

Commit various fixes for the SolarFlare drivers, in particular
this set of patches fixes support for systems with > 32 cores.

Details include

sfxge: RXQ index (not label) comes from FW in flush done/failed events

Change the second argument name of the efx_rxq_flush_done_ev_t and
efx_rxq_flush_failed_ev_t prototypes to highlight that RXQ index (not label)
comes from FW in flush done and failed events.

sfxge: TXQ index (not label) comes from FW in flush done event

Change the second argument name of the efx_txq_flush_done_ev_t prototype to
highlight that TXQ index (not label) comes from FW in flush done event.

sfxge: use TXQ type as label to support more than 32 TXQs

There are 3 TXQs in event queue 0 and 1 TXQ (with TCP/UDP checksum offload)
in all other event queues.

Submitted by: Andrew Rybchenko <Andrew.Rybchenko at oktetlabs.ru>
Sponsored by: Solarflare Communications, Inc.

sfxge: limit software Tx queue size.

Previous implementation limits put queue size only (when Tx lock can't
be acquired), but get queue may grow unboundedly which results in mbuf
pools exhaustion and latency growth.

Submitted by: Andrew Rybchenko <Andrew.Rybchenko at oktetlabs.ru>
Sponsored by: Solarflare Communications, Inc.

Add counter for Tx errors returned from if_transmit.

Submitted-by: Boris Misenov <Boris.Misenov@oktetlabs.ru>
Sponsored by: Solarflare Communications, Inc.

fix mbuf leak if it does not fit in software queue

mbuf should be owned by if_transmit function in any case.

Submitted-by: Andrew Rybchenko <Andrew.Rybchenko at oktetlabs.ru>
Sponsored by: Solarflare Communications, Inc.

Return error when packet is dropped because of link down.

Submitted-by: Boris Misenov <Boris.Misenov@oktetlabs.ru>
Sponsored by: Solarflare Communications, Inc.

Mechanically substitute flags from historic mbuf allocator with
malloc(9) flags in sys/dev.

Fix compilation on ILP32.

Add the sfxge(4) device driver, providing support for 10Gb Ethernet adapters
based on Solarflare SFC9000 family controllers. The driver supports jumbo
frames, transmit/receive checksum offload, TCP Segmentation Offload (TSO),
Large Receive Offload (LRO), VLAN checksum offload, VLAN TSO, and Receive Side
Scaling (RSS) using MSI-X interrupts.

This work was sponsored by Solarflare Communications, Inc.

My sincere thanks to Ben Hutchings for doing a lot of the hard work!

Sponsored by: Solarflare Communications, Inc.
MFC after: 3 weeks