MFC r295944:

nvme: fix intx handler to not dereference ioq during initialization

This was a regression from r293328, which deferred allocation
of the controller's ioq array until after interrupts are enabled
during boot.

Approved by: re (gjb)
Sponsored by: Intel

MFC r295532:

nvme: avoid duplicate SET_NUM_QUEUES commands

nvme(4) issues a SET_NUM_QUEUES command during device
initialization to ensure enough I/O queues exists for each
of the MSI-X vectors we have allocated. The SET_NUM_QUEUES
command is then issued again during nvme_ctrlr_start(), to
ensure that is properly set after any controller reset.

At least one NVMe drive exists which fails this second
SET_NUM_QUEUES command during device initialization. So
change nvme_ctrlr_start() to only issue its SET_NUM_QUEUES
command when it is coming out of a reset - avoiding the
duplicate SET_NUM_QUEUES during device initialization.

Approved by: re (glebius)
Sponsored by: Intel

MFC r293354:

nvme: replace NVME_CEILING macro with howmany()

MFC r293352:

nvme: add hw.nvme.min_cpus_per_ioq tunable

Due to FreeBSD system-wide limits on number of MSI-X vectors
(https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199321),
it may be desirable to allocate fewer than the maximum number
of vectors for an NVMe device, in order to save vectors for
other devices (usually Ethernet) that can take better
advantage of them and may be probed after NVMe.

This tunable is expressed in terms of minimum number of CPUs
per I/O queue instead of max number of queues per controller,
to allow for a more even distribution of CPUs per queue. This
avoids cases where some number of CPUs have a dedicated queue,
but other CPUs need to share queues. Ideally the PR referenced
above will eventually be fixed and the mechanism implemented
here becomes obsolete anyways.

While here, fix a bug in the CPUs per I/O queue calculation to
properly account for the admin queue's MSI-X vector.

MFC r293328:

nvme: do not revert to single I/O queue when per-CPU queues not available

Previously nvme(4) would revert to a single I/O queue if it could not
allocate enought interrupt vectors or NVMe submission/completion queues
to have one I/O queue per core. This patch determines how to utilize a
smaller number of available interrupt vectors, and assigns (as closely
as possible) an equal number of cores to each associated I/O queue.

MFC r293327:

nvme: break out interrupt setup code into a separate function

MFC r293326:

nvme: do not pre-allocate MSI-X IRQ resources

The issue referenced here was resolved by other changes
in recent commits, so this code is no longer needed.

MFC r293325:

nvme: remove per_cpu_io_queues from struct nvme_controller

Instead just use num_io_queues to make this determination.

This prepares for some future changes enabling use of multiple
queues when we do not have enough queues or MSI-X vectors
for one queue per CPU.

MFC r293324:

nvme: simplify some of the nested ifs in interrupt setup code

This prepares for some follow-up commits which do more work in
this area.

MFC r285816:

nvme: ensure csts.rdy bit is cleared before returning from nvme_ctrlr_disable

Sponsored by: Intel

MFC r285815:

nvme: properly handle case where pci_alloc_msix does not alloc all vectors

Sponsored by: Intel

MFC r281283:

nvme: remove CHATHAM related code

Chatham was an internal NVMe prototype board used for
early driver development.

Sponsored by: Intel

MFC r281280:

nvme: fall back to a smaller MSI-X vector allocation if necessary

Previously, if per-CPU MSI-X vectors could not be allocated,
nvme(4) would fall back to INTx with a single I/O queue pair.
This change will still fall back to a single I/O queue pair, but
allocate MSI-X vectors instead of reverting to INTx.

Sponsored by: Intel

MFC r263311:

nvme: Allocate all MSI resources up front so that we can fall back to
INTx if necessary.

MFC r263310:

nvme: Close hole where nvd(4) would not be notified of all nvme(4)
instances if modules loaded during boot.

MFC r263277:

nvme: Remove the software progress marker SET_FEATURE command during
controller initialization.

The spec says OS drivers should send this command after controller
initialization completes successfully, but other NVMe OS drivers are
not sending this command. This change will therefore reduce differences
between the FreeBSD and other OS drivers.

MFC r260382:

For IDENTIFY passthrough commands to Chatham prototype controllers, copy
the spoofed identify data into the user buffer rather than issuing the
command to the controller, since Chatham IDENTIFY data is always spoofed.

While here, fix a bug in the spoofed data for Chatham submission and
completion queue entry sizes.

MFC r257534:

Create a unique unit number for each controller and namespace cdev.

Sponsored by: Intel
Approved by: re (glebius)

MFC r285816:

nvme: ensure csts.rdy bit is cleared before returning from nvme_ctrlr_disable

Sponsored by: Intel

MFC r285815:

nvme: properly handle case where pci_alloc_msix does not alloc all vectors

Sponsored by: Intel

MFC r281283:

nvme: remove CHATHAM related code

Chatham was an internal NVMe prototype board used for
early driver development.

Sponsored by: Intel

MFC r281280:

nvme: fall back to a smaller MSI-X vector allocation if necessary

Previously, if per-CPU MSI-X vectors could not be allocated,
nvme(4) would fall back to INTx with a single I/O queue pair.
This change will still fall back to a single I/O queue pair, but
allocate MSI-X vectors instead of reverting to INTx.

Sponsored by: Intel

MFC r263311:

nvme: Allocate all MSI resources up front so that we can fall back to
INTx if necessary.

MFC r263310:

nvme: Close hole where nvd(4) would not be notified of all nvme(4)
instances if modules loaded during boot.

MFC r263277:

nvme: Remove the software progress marker SET_FEATURE command during
controller initialization.

The spec says OS drivers should send this command after controller
initialization completes successfully, but other NVMe OS drivers are
not sending this command. This change will therefore reduce differences
between the FreeBSD and other OS drivers.

MFC r260382:

For IDENTIFY passthrough commands to Chatham prototype controllers, copy
the spoofed identify data into the user buffer rather than issuing the
command to the controller, since Chatham IDENTIFY data is always spoofed.

While here, fix a bug in the spoofed data for Chatham submission and
completion queue entry sizes.

MFC r257534:

Create a unique unit number for each controller and namespace cdev.

Sponsored by: Intel
Approved by: re (glebius)