MFC r293328:

nvme: do not revert to single I/O queue when per-CPU queues not available

Previously nvme(4) would revert to a single I/O queue if it could not
allocate enought interrupt vectors or NVMe submission/completion queues
to have one I/O queue per core. This patch determines how to utilize a
smaller number of available interrupt vectors, and assigns (as closely
as possible) an equal number of cores to each associated I/O queue.

MFC r293326:

nvme: do not pre-allocate MSI-X IRQ resources

The issue referenced here was resolved by other changes
in recent commits, so this code is no longer needed.

MFC r293325:

nvme: remove per_cpu_io_queues from struct nvme_controller

Instead just use num_io_queues to make this determination.

This prepares for some future changes enabling use of multiple
queues when we do not have enough queues or MSI-X vectors
for one queue per CPU.

MFC r281283:

nvme: remove CHATHAM related code

Chatham was an internal NVMe prototype board used for
early driver development.

Sponsored by: Intel

MFC r281281, r281285:

nvme: create separate DMA tag for non-payload DMA buffers

Submission and completion queue memory need to use a
separate DMA tag for mappings than payload buffers,
to ensure mappings remain contiguous even with DMAR
enabled.

Sponsored by: Intel

MFC r263311:

nvme: Allocate all MSI resources up front so that we can fall back to
INTx if necessary.

MFC r263310:

nvme: Close hole where nvd(4) would not be notified of all nvme(4)
instances if modules loaded during boot.

MFC r281283:

nvme: remove CHATHAM related code

Chatham was an internal NVMe prototype board used for
early driver development.

Sponsored by: Intel

MFC r281281, r281285:

nvme: create separate DMA tag for non-payload DMA buffers

Submission and completion queue memory need to use a
separate DMA tag for mappings than payload buffers,
to ensure mappings remain contiguous even with DMAR
enabled.

Sponsored by: Intel

MFC r263311:

nvme: Allocate all MSI resources up front so that we can fall back to
INTx if necessary.

MFC r263310:

nvme: Close hole where nvd(4) would not be notified of all nvme(4)
instances if modules loaded during boot.