sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

Cleanup address range checks in ioat(4).

- Deduce allowed address range for bus_dma(9) from the hardware version.
Different versions (CPU generations) have different documented limits.
- Remove difference between address ranges for src/dst and crc. At least
docs for few recent generations of CPUs do not mention anything like that,
while older are already limited with above limits.
- Remove address assertions from arguments. While I do not think the
addresses out of allowed ranges should realistically happen there due to
the platforms physical address limitations, there is now bus_dma(9) to
make sure of that, preferably via IOMMU.
- Since crc now has the same address range as src/dst, remove crc_dmamap,
reusing dst2_dmamap instead.

Discussed with: cem
MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Improve ioat(4) NUMA-awareness.

Allocate ioat->ring memory from the device domain.
Schedule ioat->poll_timer to the first CPU of the device domain.

According to pcm-numa tool from intel-pcm port, this reduces number of
remote DRAM accesses while copying data by 75%. And unless it is a noise,
I've noticed some speed improvement when copying data to other domain.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

ioat(4) should use bus_dma(9) for the operation source and destination
addresses

Reviewed by: cem
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D19725

Fix few issues in ioat(4) driver.

- Do not explicitly count active descriptors. It allows hardware reset
to happen while device is still referenced, plus simplifies locking.
- Do not stop/start callout each time the queue becomes empty. Let it
run to completion and rearm if needed, that is much cheaper then to touch
it every time, plus also simplifies locking.
- Decouple submit and cleanup locks, making driver reentrant.
- Avoid memory mapped status register read on every interrupt.
- Improve locking during device attach/detach.
- Remove some no longer used variables.

Reviewed by: cem
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D19231

ioat(4): Fix 'bogus completion_pending' KASSERT

Fix ioat_release to only set is_completion_pending if DMAs were actually
queued. Otherwise, the spurious flag could trigger an assert in the
reset path on INVARIANTS kernels.

Reviewed by: bdrewery, Suraj Raju @ Isilon
Sponsored by: Dell EMC Isilon

ioat(4): Allocate contiguous descriptors

This allows us to make strong assertions about descriptor address
validity. Additionally, future generations of the ioat(4) hardware will
require contiguous descriptors.

Reviewed by: markj
Sponsored by: Dell EMC Isilon

ioat(4): Simplify by removing dynamic scaling

This paves the way for a contiguous descriptor array.

A contiguous descriptor array has the benefit that we can make strong
assertions about whether an address is a valid descriptor or not. The
other benefit is that future generations of I/OAT hardware will require
a contiguous descriptor array anyway. The downside is that after system
boot, big chunks of contiguous memory is much harder to find. So
dynamic scaling after boot is basically impossible.

Reviewed by: markj
Sponsored by: Dell EMC Isilon

ioat(4): Use memory completion rather than device register

The CHANSTS register is a split 64-bit register on CBDMA units before
hardware v3.3. If a torn read happens during ioat_process_events(),
software cannot know when to stop completing descriptors correctly.

So, just use the device-pushed main memory channel status instead.

Remove the ioat_get_active() seatbelt as well. It does nothing if the
completion address is valid.

Sponsored by: Dell EMC Isilon

ioat(4): Submitters pick up a shovel if queue is too full

Before attempting to grow the ring.

ioat(4): Block asynchronous work during HW reset

Fix the race between ioat_reset_hw and ioat_process_events.

HW reset isn't protected by a lock because it can sleep for a long time
(40.1 ms). This resulted in a race where we would process bogus parts
of the descriptor ring as if it had completed. This looked like
duplicate completions on old events, if your ring had looped at least
once.

Block callout and interrupt work while reset runs so the completion end
of things does not observe indeterminate state and process invalid parts
of the ring.

Start the channel with a manually implemented ioat_null() to keep other
submitters quiesced while we wait for the channel to start (100 us).

r295605 may have made the race between ioat_reset_hw and
ioat_process_events wider, but I believe it already existed before that
revision. ioat_process_events can be invoked by two asynchronous
sources: callout (softclock) and device interrupt. Those could race
each other, to the same effect.

Reviewed by: markj
Approved by: re
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D7097

ioat(4): Serialize ioat_reset_hw invocations

Reviewed by: markj
Approved by: re
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D7097

ioat(4): Split timer into poll and shrink functions

Poll should happen quickly, while shrink should happen infrequently.

Protect is_completion_pending with submit_lock.

Reviewed by: markj
Approved by: re
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D7097

ioat(4): Make channel indices unsigned

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add CRC descriptor structure

Add CRC/MOVECRC operations, as well as the TEST and STORE variants.

With these operations, a CRC32C can be computed over one or more
descriptors' source data. When the STORE operation is encountered, the
accumulated CRC32C is emitted to memory. A TEST operations triggers an
IOAT channel error if the accumulated CRC32C does not match one in
memory.

These operations are not exposed through any API yet.

Sponsored by: EMC / Isilon Storage Division

ioat(4): ioat_get_dmaengine(): Add M_WAITOK mode

Sponsored by: EMC / Isilon Storage Division

ioat(4): On error detected in ithread, defer HW reset to taskqueue

The I/OAT HW reset process may sleep, so it is invalid to perform a
channel reset from the software interrupt thread.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Decode/define more capabilities, operations

These are defined in the Intel Haswell EDS volume 2 (registers) (507849
v2.1).

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add support for interrupt coalescing

In I/OAT, this is done through the INTRDELAY register. On supported
platforms, this register can coalesce interrupts in a set period to
avoid excessive interrupt load for small descriptor workflows. The
period is configurable anywhere from 1 microsecond to 16.38
milliseconds, in microsecond granularity.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Gather and expose DMA statistics via sysctl

Organize the dev.ioat sysctl node into a tree while we're here.

Sponsored by: EMC / Isilon Storage Division

ioat(4): Add ioat_copy_8k_aligned KPI

The hardware supports descriptors with two non-contiguous pages. This
allows issuing one descriptor for an 8k copy from/to non-contiguous but
otherwise page-aligned memory.

Sponsored by: EMC / Isilon Storage Division

ioat: Handle channel-fatal HW errors safely

Certain invalid operations trigger hardware error conditions. Error
conditions that only halt one channel can be detected and recovered by
resetting the channel. Error conditions that halt the whole device are
generally not recoverable.

Add a sysctl to inject channel-fatal HW errors,
'dev.ioat.<N>.force_hw_error=1'.

When a halt due to a channel error is detected, ioat(4) blocks new
operations from being queued on the channel, completes any outstanding
operations with an error status, and resets the channel before allowing
new operations to be queued again.

Update ioat.4 to document error recovery; document blockfill introduced
in r290021 while we are here; document ioat_put_dmaengine() added in
r289907; document DMA_NO_WAIT added in r289982.

Sponsored by: EMC / Isilon Storage Division

ioat: Drain/quiesce the device less racily

On detach and during a forced HW reset.

Sponsored by: EMC / Isilon Storage Division

ioat: Define DMACAPABILITY bits

Check for BFILL capability before initiating blockfill operations.

Sponsored by: EMC / Isilon Storage Division

ioat: Dedupe operation enqueue logic

Add generic hw descriptor struct and generic control flags struct, in
preparation for other kinds of IOAT operation.

Sponsored by: EMC / Isilon Storage Division

ioat: Allocate memory for ring resize sanely

Add a new flag for DMA operations, DMA_NO_WAIT. It behaves much like
other NOWAIT flags -- if queueing an operation would sleep, abort and
return NULL instead.

When growing the internal descriptor ring, the memory allocation is
performed outside of all locks. A lock-protected flag is used to avoid
duplicated work. Threads that cannot sleep and attempt to queue
operations when the descriptor ring is full allocate a larger ring with
M_NOWAIT, or bail if that fails.

ioat_reserve_space() could become an external API if is important to
callers that they have room for a sequence of operations, or that those
operations succeed each other directly in the hardware ring.

This patch splits the internal head index (->head) from the hardware's
head-of-chain (DMACOUNT) register (->hw_head). In the future, for
simplicity's sake, we could drop the 'ring' array entirely and just use
a linked list (with head and tail pointers rather than indices).

Suggested by: Witness
Sponsored by: EMC / Isilon Storage Division

ioat: Expose more softc members in sysctls

Kill some unused softc variables while we're here.

Sponsored by: EMC / Isilon Storage Division

ioat: Introduce KTR probes

Sponsored by: EMC / Isilon Storage Division

ioat: refcnt users so we can drain them at detach

We only need to borrow a mutex for the drain sleep and the 0->1
transition, so just reuse an existing one for now.

The wchan is arbitrary. Using refcount itself would have required
__DEVOLATILE(), so use the lock's address instead.

Different uses are tagged by kind, although we only do anything with
that information in INVARIANTS builds.

Sponsored by: EMC / Isilon Storage Division

ioat: Clean up logging

Replace custom Linux-like logging with a thin shim around
device_printf(), when the softc is available.

In ioat_test, shim around printf(9) instead.

Sponsored by: EMC / Isilon Storage Division

ioat: Fix some attach/detach issues

Don't run the selftest until after we've enabled bus mastering, or the
DMA engine can't copy anything for our test.

Create the ioat_test device on attach, if so tuned. Destroy the
ioat_test device on teardown.

Replace deprecated 'CALLOUT_MPSAFE' with correct '1' in callout_init().

Sponsored by: EMC / Isilon Storage Division

ioat: Define IOAT_XFERCAP_VALID_MASK and use in ioat_read_xfercap

Instead of ANDing a magic constant later.

Sponsored by: EMC / Isilon Storage Division

ioat: Use correct macro, fix build on i386

Sponsored by: EMC / Isilon Storage Division

Import ioat(4) driver

I/OAT is also referred to as Crystal Beach DMA and is a Platform Storage
Extension (PSE) on some Intel server platforms.

This driver currently supports DMA descriptors only and is part of a
larger effort to upstream an interconnect between multiple systems using
the Non-Transparent Bridge (NTB) PSE.

For now, this driver is only built on AMD64 platforms. It may be ported
to work on i386 later, if that is desired. The hardware is exclusive to
x86.

Further documentation on ioat(4), including API documentation and usage,
can be found in the new manual page.

Bring in a test tool, ioatcontrol(8), in tools/tools/ioat. The test
tool is not hooked up to the build and is not intended for end users.

Submitted by: jimharris, Carl Delsey <carl.r.delsey@intel.com>
Reviewed by: jimharris (reviewed my changes)
Approved by: markj (mentor)
Relnotes: yes
Sponsored by: Intel
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D3456