MFC r303971:

Fix vtnet hang with max_virtqueue_pairs > VTNET_MAX_QUEUE_PAIRS

Sponsored by: Multiplay

MFC r276491:

Add softc flag for when the indirect descriptor feature was negotiated

MFC r268481:

Rework when the Tx queue completion interrupt is enabled

The Tx interrupt is now kept disabled in the common case, only
enabled when the number of free descriptors in the queue falls
below a threshold. Transmitted frames are cleared from the VQ
before subsequent transmit, or in the watchdog timer.

This was a very big performance improvement for an experimental
Netmap bhyve backend.

MFC r267520, r267521, r267522, r267523, r267524:

- Remove two write-only local variables
- Remove unused element in the vtnet_rxq structure
- Remove kernel specific macro out of the VirtIO PCI header file
- Move the VIRTIO_RING_F_* defines out of virtqueue.h into
virtio_config.h
- Make the feature negotiation code easier to follow
- Force two byte alignment for all control message headers

MFC r261150, r261151, r261164, r261166, r261167, r261168, r261394, r261395:

- Read and write the MAC address in the config space byte by byte
- Also include the mbuf's csum_flags in an assert message
- Remove stray space
- Move duplicated transmit start code into a single function
- Avoid queue unlock followed by relock when the enable interrupt race
is lost
- Check for a full virtqueue in the multiqueue transmit path
- Do not place the sglist used for Rx/Tx on the stack
- Use m_defrag() instead of m_collapse() to compact a long mbuf chain

MFC r276491:

Add softc flag for when the indirect descriptor feature was negotiated

MFC r268481:

Rework when the Tx queue completion interrupt is enabled

The Tx interrupt is now kept disabled in the common case, only
enabled when the number of free descriptors in the queue falls
below a threshold. Transmitted frames are cleared from the VQ
before subsequent transmit, or in the watchdog timer.

This was a very big performance improvement for an experimental
Netmap bhyve backend.

MFC r267520, r267521, r267522, r267523, r267524:

- Remove two write-only local variables
- Remove unused element in the vtnet_rxq structure
- Remove kernel specific macro out of the VirtIO PCI header file
- Move the VIRTIO_RING_F_* defines out of virtqueue.h into
virtio_config.h
- Make the feature negotiation code easier to follow
- Force two byte alignment for all control message headers

MFC r261150, r261151, r261164, r261166, r261167, r261168, r261394, r261395:

- Read and write the MAC address in the config space byte by byte
- Also include the mbuf's csum_flags in an assert message
- Remove stray space
- Move duplicated transmit start code into a single function
- Avoid queue unlock followed by relock when the enable interrupt race
is lost
- Check for a full virtqueue in the multiqueue transmit path
- Do not place the sglist used for Rx/Tx on the stack
- Use m_defrag() instead of m_collapse() to compact a long mbuf chain