vmx(4): add native netmap support

This change adds native netmap support for the vmx(4) adapter
(vmxnet3). Native support comes for free in 12, since the driver has been
ported to iflib. To make it minimally intrusive, the native support is
only enabled if vmxnet3.netmap_native is set at boot (e.g., in loader.conf).

Tested on stable/11 running inside vmplayer.

Submitted by: Giuseppe Lettieri <g.lettieri@iet.unipi.it>
Reviewed by: vmaffione, bryanv
Sponsored by: Sunny Valley Networks
Differential Revision: https://reviews.freebsd.org/D19104

MFC r313911:

Spell VMXNET3_LEGACY_TX correctly.

MFC r326744:

Correctly report the vmxnet3 link down media status

MFC r318867: fix vmxnet3 crash when LRO is enabled

The crash can occur when all of the following conditions are true:
- a packet consists of multiple segments (requires LRO enabled)
- there has been a failure to allocate an mbuf for the packet and
the packet has to be dropped
- a host (vmware) still owned at least one segment of the packet,
so the driver had to wait for another interrupt to proceed to
discarding the remaining segment(s)

Reviewed by: rstone
Approved by: re (gjb)
Sponsored by: Panzura

Update if_hwassist when interface options are changed.

In particular for me this fixes checksum problem when if_bridge attached
to the interface requests TXCSUM to be disabled, but effectively ignored.

Approved by: re (kib)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Only use a power of 2 for the number of receive and transmit queues.
Using other values causes VMXNET3_CMD_ENABLE to fail. The Linux
driver also enforces this restriction.

Reviewed by: bryanv
MFC after: 1 week
Sponsored by: Norse
Differential Revision: https://reviews.freebsd.org/D4139

Start process of removing the use of the deprecated "M_FLOWID" flag
from the FreeBSD network code. The flag is still kept around in the
"sys/mbuf.h" header file, but does no longer have any users. Instead
the "m_pkthdr.rsstype" field in the mbuf structure is now used to
decide the meaning of the "m_pkthdr.flowid" field. To modify the
"m_pkthdr.rsstype" field please use the existing "M_HASHTYPE_XXX"
macros as defined in the "sys/mbuf.h" header file.

This patch introduces new behaviour in the transmit direction.
Previously network drivers checked if "M_FLOWID" was set in "m_flags"
before using the "m_pkthdr.flowid" field. This check has now now been
replaced by checking if "M_HASHTYPE_GET(m)" is different from
"M_HASHTYPE_NONE". In the future more hashtypes will be added, for
example hashtypes for hardware dedicated flows.

"M_HASHTYPE_OPAQUE" indicates that the "m_pkthdr.flowid" value is
valid and has no particular type. This change removes the need for an
"if" statement in TCP transmit code checking for the presence of a
valid flowid value. The "if" statement mentioned above is now a direct
variable assignment which is then later checked by the respective
network drivers like before.

Additional notes:
- The SCTP code changes will be committed as a separate patch.
- Removal of the "M_FLOWID" flag will also be done separately.
- The FreeBSD version has been bumped.

MFC after: 1 month
Sponsored by: Mellanox Technologies

Provide vmxnet3_get_counter() to return counters that are not collected,
but taken from hardware.

Improve transmit sending offload, TSO, algorithm in general.

The current TSO limitation feature only takes the total number of
bytes in an mbuf chain into account and does not limit by the number
of mbufs in a chain. Some kinds of hardware is limited by two
factors. One is the fragment length and the second is the fragment
count. Both of these limits need to be taken into account when doing
TSO. Else some kinds of hardware might have to drop completely valid
mbuf chains because they cannot loaded into the given hardware's DMA
engine. The new way of doing TSO limitation has been made backwards
compatible as input from other FreeBSD developers and will use
defaults for values not set.

Reviewed by: adrian, rmacklem
Sponsored by: Mellanox Technologies
MFC after: 1 week

Revert r271504. A new patch to solve this issue will be made.

Suggested by: adrian @

Improve transmit sending offload, TSO, algorithm in general.

The current TSO limitation feature only takes the total number of
bytes in an mbuf chain into account and does not limit by the number
of mbufs in a chain. Some kinds of hardware is limited by two
factors. One is the fragment length and the second is the fragment
count. Both of these limits need to be taken into account when doing
TSO. Else some kinds of hardware might have to drop completely valid
mbuf chains because they cannot loaded into the given hardware's DMA
engine. The new way of doing TSO limitation has been made backwards
compatible as input from other FreeBSD developers and will use
defaults for values not set.

MFC after: 1 week
Sponsored by: Mellanox Technologies

Remove some write only variables

MFC after: 3 days

Give each interrupt a descriptive name when using MSIX

MFC after: 3 days

Increment the pending packets more aggressively for TSO

Assume the number of description used is reasonable value to
increment this otherwise opaque field by.

While here, reduce a minor difference between the legacy and
multiqueue transmit paths.

MFC after: 1 week

Handle multiple calls to rxq_eof for single packet completion

This requires the VMware vmxnet3 device to flip the start of packet
descriptor's generation before the rest of the packet's descriptors
have been loaded into the Rx ring. I've never observed this behavior,
and it seems to make the most sense not to do it this way. But it is
not a lot of work for the driver to handle this situation just in case.

MFC after: 1 week

Remove an unnecessary kick of the host at the end of transmitting

MFC after: 1 week

Fix GCC compile warning: Variable(s) can be used uninitialized.

Don't check the dma map address for a static DMA mapping against NULL
to determine if the mapping is valid.

Submitted by: jhb

Fix TSO support on VMware Fusion

Apparently for VMware Fusion (and presumably VMware Workstation/Player
since the PR states TSO is broken there too, but I cannot test), the
TCP header pseudo checksum calculated should only include the protocol
(IPPROTO_TCP) value, not also the lengths as the stack does instead.

VMware ESXi seems to ignore whatever value is in the TCP header checksum,
and it is a bit surprising there is a different behavior between the
VMware products. And it is unfortunate that on ESXi we are forced to do
this extra bit of work.

PR: kern/185849
MFC after: 3 days

Remove an unnecessary variable reassignment

And it would be bad if 'm' was different from '*m0' at this
point, since we've already populated the SG list.

MFC after: 3 days

make sure if_transmit returns 0 if the mbuf is enqueued.
ixgbe/ixv.c still needs a similar fix but it takes a little
more restructuring of the code.

MFC after: 3 days

Add Tx/Rx multiqueue support to vmx(4)

As a prerequisite for multiple queues, the guest must have MSIX enabled.
Unfortunately, to work around device passthrough bugs, FreeBSD disables
MSIX when running as a VMWare guest due to the hw.pci.honor_msi_blacklist
tunable; this tunable must be disabled for multiple queues.

Also included is various minor changes from the projects/vmxnet branch.

MFC after: 1 month

Since 32-bit if_baudrate isn't enough to describe a baud rate of a 10 Gbit
interface, in the r241616 a crutch was provided. It didn't work well, and
finally we decided that it is time to break ABI and simply make if_baudrate
a 64-bit value. Meanwhile, the entire struct if_data was reviewed.

o Remove the if_baudrate_pf crutch.

o Make all fields of struct if_data fixed machine independent size. The
notion of data (packet counters, etc) are by no means MD. And it is a
bug that on amd64 we've got a 64-bit counters, while on i386 32-bit,
which at modern speeds overflow within a second.

This also removes quite a lot of COMPAT_FREEBSD32 code.

o Give 16 bit for the ifi_datalen field. This field was provided to
make future changes to if_data less ABI breaking. Unfortunately the
8 bit size of it had effectively limited sizeof if_data to 256 bytes.

o Give 32 bits to ifi_mtu and ifi_metric.
o Give 64 bits to the rest of fields, since they are counters.

__FreeBSD_version bumped.

Discussed with: emax
Sponsored by: Netflix
Sponsored by: Nginx, Inc.

Include necessary headers that now are available due to pollution
via if_var.h.

Sponsored by: Netflix
Sponsored by: Nginx, Inc.

The r48589 promised to remove implicit inclusion of if_var.h soon. Prepare
to this event, adding if_var.h to files that do need it. Also, include
all includes that now are included due to implicit pollution via if_var.h

Sponsored by: Netflix
Sponsored by: Nginx, Inc.

Do not provide a hint of the guest's OS version

The calculation can overflow if __FreeBSD_version is big
enough, and it does not appear to be required.

Reported by: grehan
Approved by: re (gjb)

Use correct type for the vmx vlan filter table

Approved by: re (glebius, gjb)

Few more minor if_vmx tweaks

- Allow the Rx/Tx queue sizes to be configured by tunables
- Bail out earlier if the Tx queue unlikely has enough free
descriptors to hold the frame
- Cleanup some of the offloading capabilities handling

Couple minor if_vmx tweaks

- Use queue size fields from the Tx/Rx queues in various places
instead of (currently the same values) from the softc.
- Fix potential crash in detach if the attached failed to alloc
queue memory.
- Move the VMXNET3_MAX_RX_SEGS define to a better spot.
- Tweak frame size calculation w.r.t. ETHER_ALIGN. This could be
tweaked some more, or removed since it probably doesn't matter
much for x86 (and the x86 class of machines this driver will
be used on).

Add vmx(4), a VMware VMXNET3 ethernet driver ported from OpenBSD