MFC r356216: Don't spin on cleanup_lock if we are not interrupt.

If somebody else holds that lock, it will likely do the work for us.
If it won't, then we return here later and retry.

Under heavy load it allows to avoid lock congestion between interrupt and
polling threads.

MFC r345812 (by tychon):
ioatcontrol(8) could exercise 8k-aligned copy with page-break, crc and
crc-copy modes.

MFC r354753: Initialize *comp_update with valid value.

I've noticed that sometimes with enabled DMAR initial write from device
to this address is somehow getting delayed, triggering assertion due to
zero default being invalid.

MFC r354703: Pass more reasonable WAIT flags to bus_dma(9) calls.

MFC r354841: Add ioat_get_domain() to ioat(4) KPI.

This allows NUMA-aware consumers to reduce inter-domain traffic.

MFC r352787: Replace argument checks with assertions.

Those functions are used by kernel, and we can't check all possible argument
errors in production kernel. Plus according to docs many of those errors
are checked by hardware. Assertions should just help with code debugging.

MFC r344441: Fix few issues in ioat(4) driver.

- Do not explicitly count active descriptors. It allows hardware reset
to happen while device is still referenced, plus simplifies locking.
- Do not stop/start callout each time the queue becomes empty. Let it
run to completion and rearm if needed, that is much cheaper then to touch
it every time, plus also simplifies locking.
- Decouple submit and cleanup locks, making driver reentrant.
- Avoid memory mapped status register read on every interrupt.
- Improve locking during device attach/detach.
- Remove some no longer used variables.

Sponsored by: iXsystems, Inc.

MFC r302669,302677-302686,303761,304602,304603,305027-305028,305259,
305710,305711,308067-308070,308178,308179,308230,308553,309338,309526,
343125 (by cem): Synchronize ioat(4) with head.

Most of these changes are 3 years old, just never got merged.

MFC r326572:

ioat(4): Add Skylake Xeon PCI-ID

SKX IOAT is just another 3.2 version of the CBDMA engine.

MFC r314666: ioat: don't specify inline for function with variable argument list

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

ioat(4): Block asynchronous work during HW reset

Fix the race between ioat_reset_hw and ioat_process_events.

HW reset isn't protected by a lock because it can sleep for a long time
(40.1 ms). This resulted in a race where we would process bogus parts
of the descriptor ring as if it had completed. This looked like
duplicate completions on old events, if your ring had looped at least
once.

Block callout and interrupt work while reset runs so the completion end
of things does not observe indeterminate state and process invalid parts
of the ring.

Start the channel with a manually implemented ioat_null() to keep other
submitters quiesced while we wait for the channel to start (100 us).

r295605 may have made the race between ioat_reset_hw and
ioat_process_events wider, but I believe it already existed before that
revision. ioat_process_events can be invoked by two asynchronous
sources: callout (softclock) and device interrupt. Those could race
each other, to the same effect.

Reviewed by: markj
Approved by: re
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D7097

ioat(4): Serialize ioat_reset_hw invocations

Reviewed by: markj
Approved by: re
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D7097

ioat(4): Split timer into poll and shrink functions

Poll should happen quickly, while shrink should happen infrequently.

Protect is_completion_pending with submit_lock.

Reviewed by: markj
Approved by: re
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D7097

ioat(4): Add ddb "show ioat <unit>" debugger command

Sponsored by: EMC / Isilon Storage Division

ioat(4): Always log capabilities on attach

Different, relatively recent Intel Xeon hardware support radically different
features. E.g., BDX support CRC32 while BDX-DE does not.

Reviewed by: rpokala (spiritually)
Sponsored by: EMC / Isilon Storage Division

ioat(4): Export the number of available channels

Sponsored by: EMC / Isilon Storage Division

ioat(4): Make channel indices unsigned

Sponsored by: EMC / Isilon Storage Division

Remove misc NULL checks after M_WAITOK allocations.

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Use DEVMETHOD_END ({ NULL, NULL }) instead of hardcoding { 0, 0 }

Sponsored by: EMC / Isilon Storage Division

ioat(4): Implement CRC and MOVECRC APIs

And document them in ioat.4.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add CRC descriptor structure

Add CRC/MOVECRC operations, as well as the TEST and STORE variants.

With these operations, a CRC32C can be computed over one or more
descriptors' source data. When the STORE operation is encountered, the
accumulated CRC32C is emitted to memory. A TEST operations triggers an
IOAT channel error if the accumulated CRC32C does not match one in
memory.

These operations are not exposed through any API yet.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Limit descriptor allocation to low 40 bits

The IOAT engine can only address the low 40 bits (1 TB) of physmem via
the 'next descriptor' pointer. Restrict acceptable range given to
bus_dma_tag_create to match.

Sponsored by: EMC / Isilon Storage Division

ioat(4): ioat_get_dmaengine(): Add M_WAITOK mode

Sponsored by: EMC / Isilon Storage Division

ioat(4): On error detected in ithread, defer HW reset to taskqueue

The I/OAT HW reset process may sleep, so it is invalid to perform a
channel reset from the software interrupt thread.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Also check for errors if the channel is suspended

Sponsored by: EMC / Isilon Storage Division

ioat(4): Decode/define more capabilities, operations

These are defined in the Intel Haswell EDS volume 2 (registers) (507849
v2.1).

Sponsored by: EMC / Isilon Storage Division

ioat(4): Recheck status register on zero-descriptor wakeups

Errors that halt the channel don't necessarily result in a completion
update, apparently.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add support for 'fence' bit with DMA_FENCE flag

Some classes of IOAT hardware prefetch reads. DMA operations that
depend on the result of prior DMA operations must use the DMA_FENCE flag
to prevent stale reads.

(E.g., I've hit this personally on Broadwell-EP. The Broadwell-DE has a
different IOAT unit that is documented to not pipeline DMA operations.)

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add ioat_acquire_reserve() KPI

ioat_acquire_reserve() is an extended version of ioat_acquire(). It
allows users to reserve space in the channel for some number of
descriptors. If this succeeds, it guarantees that at least submission
of N valid descriptors will succeed.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add ioat_get_max_io_size() KPI

Consumers need to know the permitted IO size to send maximally sized
chunks to the hardware.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add an API to get HW revision

Different revisions support different operations. Refer to Intel
External Design Specifications to figure out what your hardware
supports.

Sponsored by: EMC / Isilon Storage Division

ioatcontrol(8): Add support for interrupt coalescing

The new flag, -c <period>, sets the interrupt coalescing period in
microseconds through the new ioat(4) API ioat_set_interrupt_coalesce().

Also add a -z flag to zero ioat statistics before tests, to make it easy
to measure results.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add support for interrupt coalescing

In I/OAT, this is done through the INTRDELAY register. On supported
platforms, this register can coalesce interrupts in a set period to
avoid excessive interrupt load for small descriptor workflows. The
period is configurable anywhere from 1 microsecond to 16.38
milliseconds, in microsecond granularity.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Gather and expose DMA statistics via sysctl

Organize the dev.ioat sysctl node into a tree while we're here.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add ioatcontrol(8) testing for copy_8k

Add -E ("Eight k") and -m ("Memcpy") modes to the ioatcontrol(8) tool.

Prompted by: rpokala
Sponsored by: EMC / Isilon Storage Division

ioat(4): Add Broadwell-EP PCI IDs

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add ioat_copy_8k_aligned KPI

The hardware supports descriptors with two non-contiguous pages. This
allows issuing one descriptor for an 8k copy from/to non-contiguous but
otherwise page-aligned memory.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add MODULE_VERSION so MODULE_DEPEND works

Suggested by: jhb
Review in progress: cc
Sponsored by: EMC / Isilon Storage Division

ioat: Handle channel-fatal HW errors safely

Certain invalid operations trigger hardware error conditions. Error
conditions that only halt one channel can be detected and recovered by
resetting the channel. Error conditions that halt the whole device are
generally not recoverable.

Add a sysctl to inject channel-fatal HW errors,
'dev.ioat.<N>.force_hw_error=1'.

When a halt due to a channel error is detected, ioat(4) blocks new
operations from being queued on the channel, completes any outstanding
operations with an error status, and resets the channel before allowing
new operations to be queued again.

Update ioat.4 to document error recovery; document blockfill introduced
in r290021 while we are here; document ioat_put_dmaengine() added in
r289907; document DMA_NO_WAIT added in r289982.

Sponsored by: EMC / Isilon Storage Division

ioat_test: Handled forced hardware resets gracefully

Sponsored by: EMC / Isilon Storage Division

ioat: Drain/quiesce the device less racily

On detach and during a forced HW reset.

Sponsored by: EMC / Isilon Storage Division

ioatcontrol(8): Add and document "raw" testing mode

Allows DMA from/to arbitrary KVA or physical address. /dev/ioat_test
must be enabled by root and is only R/W root, so this is approximately
as dangerous as /dev/mem and /dev/kmem.

Sponsored by: EMC / Isilon Storage Division

ioat: Define DMACAPABILITY bits

Check for BFILL capability before initiating blockfill operations.

Sponsored by: EMC / Isilon Storage Division

ioat: Add support for Block Fill operations

The IOAT hardware supports writing a 64-bit pattern to some destination
buffer. The same limitations on buffer length apply as for copy
operations. Throughput is a bit higher (probably because fill does not
have to spend bandwidth reading from a source in memory).

Support for testing Block Fill has been added to ioatcontrol(8) and the
ioat_test device. ioatcontrol(8) accepts the '-f' flag, which tests
Block Fill. (If the flag is omitted, the tool tests copy by default.)
The '-V' flag, in conjunction with '-f', verifies that buffers are
filled in the expected pattern.

Tested on: Broadwell DE (Xeon D-1500)
Sponsored by: EMC / Isilon Storage Division

ioat: Dedupe operation enqueue logic

Add generic hw descriptor struct and generic control flags struct, in
preparation for other kinds of IOAT operation.

Sponsored by: EMC / Isilon Storage Division

ioat: Add %b format string for CHANERR codes

Sponsored by: EMC / Isilon Storage Division

ioat: Allocate memory for ring resize sanely

Add a new flag for DMA operations, DMA_NO_WAIT. It behaves much like
other NOWAIT flags -- if queueing an operation would sleep, abort and
return NULL instead.

When growing the internal descriptor ring, the memory allocation is
performed outside of all locks. A lock-protected flag is used to avoid
duplicated work. Threads that cannot sleep and attempt to queue
operations when the descriptor ring is full allocate a larger ring with
M_NOWAIT, or bail if that fails.

ioat_reserve_space() could become an external API if is important to
callers that they have room for a sequence of operations, or that those
operations succeed each other directly in the hardware ring.

This patch splits the internal head index (->head) from the hardware's
head-of-chain (DMACOUNT) register (->hw_head). In the future, for
simplicity's sake, we could drop the 'ring' array entirely and just use
a linked list (with head and tail pointers rather than indices).

Suggested by: Witness
Sponsored by: EMC / Isilon Storage Division

ioat: Expose more softc members in sysctls

Kill some unused softc variables while we're here.

Sponsored by: EMC / Isilon Storage Division

ioat: Introduce KTR probes

Sponsored by: EMC / Isilon Storage Division

ioat: Actually bring the hardware back online after reset

We need to reset the chancmp and chainaddr MMIO registers to bring the
device back to a working state.

Name the chanerr bits while we're here.

Sponsored by: EMC / Isilon Storage Division

ioat: Use bus_alloc_resource_any(9)

Sponsored by: EMC / Isilon Storage Division

ioat: Extract halted error-debugging to a function

Sponsored by: EMC / Isilon Storage Division

ioat: Always re-arm interrupts in process_events

It doesn't hurt, even if there is nothing to do.

Sponsored by: EMC / Isilon Storage Division

ioat: Add sysctl to force hw reset

To enable controlled testing.

Sponsored by: EMC / Isilon Storage Division

ioat: refcnt users so we can drain them at detach

We only need to borrow a mutex for the drain sleep and the 0->1
transition, so just reuse an existing one for now.

The wchan is arbitrary. Using refcount itself would have required
__DEVOLATILE(), so use the lock's address instead.

Different uses are tagged by kind, although we only do anything with
that information in INVARIANTS builds.

Sponsored by: EMC / Isilon Storage Division

ioat: When queueing operations, assert the submit lock

Callers should have acquired this lock when they invoked ioat_acquire()
before issuing operations. Assert it is held.

Sponsored by: EMC / Isilon Storage Division

ioat: Don't use sleeping allocation in lock path

This is still the worst possible way to allocate memory if it will ever
be under pressure, but at least it won't deadlock.

Suggested by: WITNESS
Sponsored by: EMC / Isilon Storage Division

ioat: Pull out timer callout delay into a constant

Pull out the timer callout delay into IOAT_INTR_TIMO and shorten it
considerably (5s -> 100ms). Single operations do not take 5-10 seconds
and when interrupts aren't working, waiting 100ms sucks a lot less than
5s.

Sponsored by: EMC / Isilon Storage Division

ioat_test: Add a colon (':') for style

Missed in r289776.

Sponsored by: EMC / Isilon Storage Division

ioat: Clean up logging

Replace custom Linux-like logging with a thin shim around
device_printf(), when the softc is available.

In ioat_test, shim around printf(9) instead.

Sponsored by: EMC / Isilon Storage Division

ioat: Fix some attach/detach issues

Don't run the selftest until after we've enabled bus mastering, or the
DMA engine can't copy anything for our test.

Create the ioat_test device on attach, if so tuned. Destroy the
ioat_test device on teardown.

Replace deprecated 'CALLOUT_MPSAFE' with correct '1' in callout_init().

Sponsored by: EMC / Isilon Storage Division

Improve flexibility of ioat_test / ioatcontrol(8)

The test logic now preallocates memory before running the test.

The buffer size is now configurable. Post-copy verification is
configurable. The number of copies to chain into one transaction (one
interrupt) is configurable.

A 'duration' mode is added, which repeats the test until the duration
has elapsed, reporting the B/s and transactions completed.

ioatcontrol.8 has been updated to document the new arguments.

Initial limits (on this particular Broadwell-DE) (and when the
interrupts are working) seem to be: 256 interrupts/sec or ~6 GB/s,
whichever limit is more restrictive.

Unfortunately, it seems the interrupt-reset handling on Broadwell isn't
working as intended. That will be fixed in a later commit.

Sponsored by: EMC / Isilon Storage Division

ioat: Define IOAT_XFERCAP_VALID_MASK and use in ioat_read_xfercap

Instead of ANDing a magic constant later.

Sponsored by: EMC / Isilon Storage Division

ioat: Use correct macro, fix build on i386

Sponsored by: EMC / Isilon Storage Division

ioat(4): pci_save/restore_state to persist MSI-X registers over BDXDE reset

Also for BWD devices, per jimharris@.

Reviewed by: jhb
Approved by: markj (mentor)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D3552

ioat: re-initialize interrupts after resetting hw on BDXDE

Resetting some generations of the I/OAT hardware (just BDXDE for now)
resets the corresponding MSI-X registers. So, teardown and
re-initialize interrupts after resetting the hardware.

Reviewed by: jimharris
Approved by: markj (mentor)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D3549

ioat(4): Minor style cleanups

Suggested by: ngie
Reviewed by: jimharris
Approved by: markj (mentor)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D3481

Import ioat(4) driver

I/OAT is also referred to as Crystal Beach DMA and is a Platform Storage
Extension (PSE) on some Intel server platforms.

This driver currently supports DMA descriptors only and is part of a
larger effort to upstream an interconnect between multiple systems using
the Non-Transparent Bridge (NTB) PSE.

For now, this driver is only built on AMD64 platforms. It may be ported
to work on i386 later, if that is desired. The hardware is exclusive to
x86.

Further documentation on ioat(4), including API documentation and usage,
can be found in the new manual page.

Bring in a test tool, ioatcontrol(8), in tools/tools/ioat. The test
tool is not hooked up to the build and is not intended for end users.

Submitted by: jimharris, Carl Delsey <carl.r.delsey@intel.com>
Reviewed by: jimharris (reviewed my changes)
Approved by: markj (mentor)
Relnotes: yes
Sponsored by: Intel
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D3456