#
356430 |
|
07-Jan-2020 |
mav |
MFC r356216: Don't spin on cleanup_lock if we are not interrupt.
If somebody else holds that lock, it will likely do the work for us. If it won't, then we return here later and retry.
Under heavy load it allows to avoid lock congestion between interrupt and polling threads.
|
#
355196 |
|
29-Nov-2019 |
mav |
MFC r354753: Initialize *comp_update with valid value.
I've noticed that sometimes with enabled DMAR initial write from device to this address is somehow getting delayed, triggering assertion due to zero default being invalid.
|
#
355154 |
|
28-Nov-2019 |
mav |
MFC r354703: Pass more reasonable WAIT flags to bus_dma(9) calls.
|
#
355112 |
|
26-Nov-2019 |
mav |
MFC r354841: Add ioat_get_domain() to ioat(4) KPI.
This allows NUMA-aware consumers to reduce inter-domain traffic.
|
#
353581 |
|
15-Oct-2019 |
mav |
MFC r352787: Replace argument checks with assertions.
Those functions are used by kernel, and we can't check all possible argument errors in production kernel. Plus according to docs many of those errors are checked by hardware. Assertions should just help with code debugging.
|
#
344650 |
|
28-Feb-2019 |
mav |
MFC r344441: Fix few issues in ioat(4) driver.
- Do not explicitly count active descriptors. It allows hardware reset to happen while device is still referenced, plus simplifies locking. - Do not stop/start callout each time the queue becomes empty. Let it run to completion and rearm if needed, that is much cheaper then to touch it every time, plus also simplifies locking. - Decouple submit and cleanup locks, making driver reentrant. - Avoid memory mapped status register read on every interrupt. - Improve locking during device attach/detach. - Remove some no longer used variables.
Sponsored by: iXsystems, Inc.
|
#
344401 |
|
21-Feb-2019 |
mav |
MFC r302669,302677-302686,303761,304602,304603,305027-305028,305259, 305710,305711,308067-308070,308178,308179,308230,308553,309338,309526, 343125 (by cem): Synchronize ioat(4) with head.
Most of these changes are 3 years old, just never got merged.
|
#
330450 |
|
05-Mar-2018 |
eadler |
MFC r326572:
ioat(4): Add Skylake Xeon PCI-ID
SKX IOAT is just another 3.2 version of the CBDMA engine.
|
#
302408 |
|
07-Jul-2016 |
gjb |
Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle. Prune svn:mergeinfo from the new branch, as nothing has been merged here.
Additional commits post-branch will follow.
Approved by: re (implicit) Sponsored by: The FreeBSD Foundation |
#
302354 |
|
05-Jul-2016 |
cem |
ioat(4): Block asynchronous work during HW reset
Fix the race between ioat_reset_hw and ioat_process_events.
HW reset isn't protected by a lock because it can sleep for a long time (40.1 ms). This resulted in a race where we would process bogus parts of the descriptor ring as if it had completed. This looked like duplicate completions on old events, if your ring had looped at least once.
Block callout and interrupt work while reset runs so the completion end of things does not observe indeterminate state and process invalid parts of the ring.
Start the channel with a manually implemented ioat_null() to keep other submitters quiesced while we wait for the channel to start (100 us).
r295605 may have made the race between ioat_reset_hw and ioat_process_events wider, but I believe it already existed before that revision. ioat_process_events can be invoked by two asynchronous sources: callout (softclock) and device interrupt. Those could race each other, to the same effect.
Reviewed by: markj Approved by: re Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D7097
|
#
302353 |
|
05-Jul-2016 |
cem |
ioat(4): Serialize ioat_reset_hw invocations
Reviewed by: markj Approved by: re Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D7097
|
#
302352 |
|
05-Jul-2016 |
cem |
ioat(4): Split timer into poll and shrink functions
Poll should happen quickly, while shrink should happen infrequently.
Protect is_completion_pending with submit_lock.
Reviewed by: markj Approved by: re Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D7097
|
#
301712 |
|
08-Jun-2016 |
cem |
ioat(4): Add ddb "show ioat <unit>" debugger command
Sponsored by: EMC / Isilon Storage Division
|
#
301300 |
|
04-Jun-2016 |
cem |
ioat(4): Always log capabilities on attach
Different, relatively recent Intel Xeon hardware support radically different features. E.g., BDX support CRC32 while BDX-DE does not.
Reviewed by: rpokala (spiritually) Sponsored by: EMC / Isilon Storage Division
|
#
301297 |
|
04-Jun-2016 |
cem |
ioat(4): Export the number of available channels
Sponsored by: EMC / Isilon Storage Division
|
#
301296 |
|
04-Jun-2016 |
cem |
ioat(4): Make channel indices unsigned
Sponsored by: EMC / Isilon Storage Division
|
#
299353 |
|
10-May-2016 |
trasz |
Remove misc NULL checks after M_WAITOK allocations.
MFC after: 1 month Sponsored by: The FreeBSD Foundation
|
#
299015 |
|
03-May-2016 |
ngie |
Use DEVMETHOD_END ({ NULL, NULL }) instead of hardcoding { 0, 0 }
Sponsored by: EMC / Isilon Storage Division
|
#
298989 |
|
03-May-2016 |
cem |
ioat(4): Implement CRC and MOVECRC APIs
And document them in ioat.4.
Sponsored by: EMC / Isilon Storage Division
|
#
298987 |
|
03-May-2016 |
cem |
ioat(4): Limit descriptor allocation to low 40 bits
The IOAT engine can only address the low 40 bits (1 TB) of physmem via the 'next descriptor' pointer. Restrict acceptable range given to bus_dma_tag_create to match.
Sponsored by: EMC / Isilon Storage Division
|
#
297746 |
|
09-Apr-2016 |
cem |
ioat(4): ioat_get_dmaengine(): Add M_WAITOK mode
Sponsored by: EMC / Isilon Storage Division
|
#
295605 |
|
13-Feb-2016 |
cem |
ioat(4): On error detected in ithread, defer HW reset to taskqueue
The I/OAT HW reset process may sleep, so it is invalid to perform a channel reset from the software interrupt thread.
Sponsored by: EMC / Isilon Storage Division
|
#
295604 |
|
13-Feb-2016 |
cem |
ioat(4): Also check for errors if the channel is suspended
Sponsored by: EMC / Isilon Storage Division
|
#
295588 |
|
13-Feb-2016 |
cem |
ioat(4): Recheck status register on zero-descriptor wakeups
Errors that halt the channel don't necessarily result in a completion update, apparently.
Sponsored by: EMC / Isilon Storage Division
|
#
294062 |
|
14-Jan-2016 |
cem |
ioat(4): Add support for 'fence' bit with DMA_FENCE flag
Some classes of IOAT hardware prefetch reads. DMA operations that depend on the result of prior DMA operations must use the DMA_FENCE flag to prevent stale reads.
(E.g., I've hit this personally on Broadwell-EP. The Broadwell-DE has a different IOAT unit that is documented to not pipeline DMA operations.)
Sponsored by: EMC / Isilon Storage Division
|
#
293390 |
|
07-Jan-2016 |
cem |
ioat(4): Add ioat_acquire_reserve() KPI
ioat_acquire_reserve() is an extended version of ioat_acquire(). It allows users to reserve space in the channel for some number of descriptors. If this succeeds, it guarantees that at least submission of N valid descriptors will succeed.
Sponsored by: EMC / Isilon Storage Division
|
#
293221 |
|
05-Jan-2016 |
cem |
ioat(4): Add ioat_get_max_io_size() KPI
Consumers need to know the permitted IO size to send maximally sized chunks to the hardware.
Sponsored by: EMC / Isilon Storage Division
|
#
292413 |
|
17-Dec-2015 |
cem |
ioat(4): Add an API to get HW revision
Different revisions support different operations. Refer to Intel External Design Specifications to figure out what your hardware supports.
Sponsored by: EMC / Isilon Storage Division
|
#
292228 |
|
14-Dec-2015 |
cem |
ioat(4): Add support for interrupt coalescing
In I/OAT, this is done through the INTRDELAY register. On supported platforms, this register can coalesce interrupts in a set period to avoid excessive interrupt load for small descriptor workflows. The period is configurable anywhere from 1 microsecond to 16.38 milliseconds, in microsecond granularity.
Sponsored by: EMC / Isilon Storage Division
|
#
292226 |
|
14-Dec-2015 |
cem |
ioat(4): Gather and expose DMA statistics via sysctl
Organize the dev.ioat sysctl node into a tree while we're here.
Sponsored by: EMC / Isilon Storage Division
|
#
292032 |
|
09-Dec-2015 |
cem |
ioat(4): Add Broadwell-EP PCI IDs
Sponsored by: EMC / Isilon Storage Division
|
#
292031 |
|
09-Dec-2015 |
cem |
ioat(4): Add ioat_copy_8k_aligned KPI
The hardware supports descriptors with two non-contiguous pages. This allows issuing one descriptor for an 8k copy from/to non-contiguous but otherwise page-aligned memory.
Sponsored by: EMC / Isilon Storage Division
|
#
291826 |
|
04-Dec-2015 |
cem |
ioat(4): Add MODULE_VERSION so MODULE_DEPEND works
Suggested by: jhb Review in progress: cc Sponsored by: EMC / Isilon Storage Division
|
#
290229 |
|
31-Oct-2015 |
cem |
ioat: Handle channel-fatal HW errors safely
Certain invalid operations trigger hardware error conditions. Error conditions that only halt one channel can be detected and recovered by resetting the channel. Error conditions that halt the whole device are generally not recoverable.
Add a sysctl to inject channel-fatal HW errors, 'dev.ioat.<N>.force_hw_error=1'.
When a halt due to a channel error is detected, ioat(4) blocks new operations from being queued on the channel, completes any outstanding operations with an error status, and resets the channel before allowing new operations to be queued again.
Update ioat.4 to document error recovery; document blockfill introduced in r290021 while we are here; document ioat_put_dmaengine() added in r289907; document DMA_NO_WAIT added in r289982.
Sponsored by: EMC / Isilon Storage Division
|
#
290131 |
|
29-Oct-2015 |
cem |
ioat: Drain/quiesce the device less racily
On detach and during a forced HW reset.
Sponsored by: EMC / Isilon Storage Division
|
#
290087 |
|
28-Oct-2015 |
cem |
ioat: Define DMACAPABILITY bits
Check for BFILL capability before initiating blockfill operations.
Sponsored by: EMC / Isilon Storage Division
|
#
290021 |
|
26-Oct-2015 |
cem |
ioat: Add support for Block Fill operations
The IOAT hardware supports writing a 64-bit pattern to some destination buffer. The same limitations on buffer length apply as for copy operations. Throughput is a bit higher (probably because fill does not have to spend bandwidth reading from a source in memory).
Support for testing Block Fill has been added to ioatcontrol(8) and the ioat_test device. ioatcontrol(8) accepts the '-f' flag, which tests Block Fill. (If the flag is omitted, the tool tests copy by default.) The '-V' flag, in conjunction with '-f', verifies that buffers are filled in the expected pattern.
Tested on: Broadwell DE (Xeon D-1500) Sponsored by: EMC / Isilon Storage Division
|
#
290020 |
|
26-Oct-2015 |
cem |
ioat: Dedupe operation enqueue logic
Add generic hw descriptor struct and generic control flags struct, in preparation for other kinds of IOAT operation.
Sponsored by: EMC / Isilon Storage Division
|
#
289983 |
|
26-Oct-2015 |
cem |
ioat: Add %b format string for CHANERR codes
Sponsored by: EMC / Isilon Storage Division
|
#
289982 |
|
26-Oct-2015 |
cem |
ioat: Allocate memory for ring resize sanely
Add a new flag for DMA operations, DMA_NO_WAIT. It behaves much like other NOWAIT flags -- if queueing an operation would sleep, abort and return NULL instead.
When growing the internal descriptor ring, the memory allocation is performed outside of all locks. A lock-protected flag is used to avoid duplicated work. Threads that cannot sleep and attempt to queue operations when the descriptor ring is full allocate a larger ring with M_NOWAIT, or bail if that fails.
ioat_reserve_space() could become an external API if is important to callers that they have room for a sequence of operations, or that those operations succeed each other directly in the hardware ring.
This patch splits the internal head index (->head) from the hardware's head-of-chain (DMACOUNT) register (->hw_head). In the future, for simplicity's sake, we could drop the 'ring' array entirely and just use a linked list (with head and tail pointers rather than indices).
Suggested by: Witness Sponsored by: EMC / Isilon Storage Division
|
#
289980 |
|
26-Oct-2015 |
cem |
ioat: Expose more softc members in sysctls
Kill some unused softc variables while we're here.
Sponsored by: EMC / Isilon Storage Division
|
#
289979 |
|
26-Oct-2015 |
cem |
ioat: Introduce KTR probes
Sponsored by: EMC / Isilon Storage Division
|
#
289912 |
|
24-Oct-2015 |
cem |
ioat: Actually bring the hardware back online after reset
We need to reset the chancmp and chainaddr MMIO registers to bring the device back to a working state.
Name the chanerr bits while we're here.
Sponsored by: EMC / Isilon Storage Division
|
#
289911 |
|
24-Oct-2015 |
cem |
ioat: Use bus_alloc_resource_any(9)
Sponsored by: EMC / Isilon Storage Division
|
#
289910 |
|
24-Oct-2015 |
cem |
ioat: Extract halted error-debugging to a function
Sponsored by: EMC / Isilon Storage Division
|
#
289909 |
|
24-Oct-2015 |
cem |
ioat: Always re-arm interrupts in process_events
It doesn't hurt, even if there is nothing to do.
Sponsored by: EMC / Isilon Storage Division
|
#
289908 |
|
24-Oct-2015 |
cem |
ioat: Add sysctl to force hw reset
To enable controlled testing.
Sponsored by: EMC / Isilon Storage Division
|
#
289907 |
|
24-Oct-2015 |
cem |
ioat: refcnt users so we can drain them at detach
We only need to borrow a mutex for the drain sleep and the 0->1 transition, so just reuse an existing one for now.
The wchan is arbitrary. Using refcount itself would have required __DEVOLATILE(), so use the lock's address instead.
Different uses are tagged by kind, although we only do anything with that information in INVARIANTS builds.
Sponsored by: EMC / Isilon Storage Division
|
#
289906 |
|
24-Oct-2015 |
cem |
ioat: When queueing operations, assert the submit lock
Callers should have acquired this lock when they invoked ioat_acquire() before issuing operations. Assert it is held.
Sponsored by: EMC / Isilon Storage Division
|
#
289905 |
|
24-Oct-2015 |
cem |
ioat: Don't use sleeping allocation in lock path
This is still the worst possible way to allocate memory if it will ever be under pressure, but at least it won't deadlock.
Suggested by: WITNESS Sponsored by: EMC / Isilon Storage Division
|
#
289904 |
|
24-Oct-2015 |
cem |
ioat: Pull out timer callout delay into a constant
Pull out the timer callout delay into IOAT_INTR_TIMO and shorten it considerably (5s -> 100ms). Single operations do not take 5-10 seconds and when interrupts aren't working, waiting 100ms sucks a lot less than 5s.
Sponsored by: EMC / Isilon Storage Division
|
#
289776 |
|
22-Oct-2015 |
cem |
ioat: Clean up logging
Replace custom Linux-like logging with a thin shim around device_printf(), when the softc is available.
In ioat_test, shim around printf(9) instead.
Sponsored by: EMC / Isilon Storage Division
|
#
289760 |
|
22-Oct-2015 |
cem |
ioat: Fix some attach/detach issues
Don't run the selftest until after we've enabled bus mastering, or the DMA engine can't copy anything for our test.
Create the ioat_test device on attach, if so tuned. Destroy the ioat_test device on teardown.
Replace deprecated 'CALLOUT_MPSAFE' with correct '1' in callout_init().
Sponsored by: EMC / Isilon Storage Division
|
#
289732 |
|
22-Oct-2015 |
cem |
ioat: Define IOAT_XFERCAP_VALID_MASK and use in ioat_read_xfercap
Instead of ANDing a magic constant later.
Sponsored by: EMC / Isilon Storage Division
|
#
287414 |
|
02-Sep-2015 |
cem |
ioat(4): pci_save/restore_state to persist MSI-X registers over BDXDE reset
Also for BWD devices, per jimharris@.
Reviewed by: jhb Approved by: markj (mentor) Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3552
|
#
287403 |
|
02-Sep-2015 |
cem |
ioat: re-initialize interrupts after resetting hw on BDXDE
Resetting some generations of the I/OAT hardware (just BDXDE for now) resets the corresponding MSI-X registers. So, teardown and re-initialize interrupts after resetting the hardware.
Reviewed by: jimharris Approved by: markj (mentor) Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3549
|
#
287138 |
|
25-Aug-2015 |
cem |
ioat(4): Minor style cleanups
Suggested by: ngie Reviewed by: jimharris Approved by: markj (mentor) Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3481
|
#
287117 |
|
24-Aug-2015 |
cem |
Import ioat(4) driver
I/OAT is also referred to as Crystal Beach DMA and is a Platform Storage Extension (PSE) on some Intel server platforms.
This driver currently supports DMA descriptors only and is part of a larger effort to upstream an interconnect between multiple systems using the Non-Transparent Bridge (NTB) PSE.
For now, this driver is only built on AMD64 platforms. It may be ported to work on i386 later, if that is desired. The hardware is exclusive to x86.
Further documentation on ioat(4), including API documentation and usage, can be found in the new manual page.
Bring in a test tool, ioatcontrol(8), in tools/tools/ioat. The test tool is not hooked up to the build and is not intended for end users.
Submitted by: jimharris, Carl Delsey <carl.r.delsey@intel.com> Reviewed by: jimharris (reviewed my changes) Approved by: markj (mentor) Relnotes: yes Sponsored by: Intel Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3456
|