busdma: better handling of small segment bouncing

Typically, when a DMA transaction requires bouncing, we will break up
the request into segments that are, at maximum, page-sized.

However, in the atypical case of a driver whose maximum segment size is
smaller than PAGE_SIZE, we end up inefficiently assigning each segment
its own bounce page. For example, the dwmmc driver has a maximum segment
size of 2048 (PAGE_SIZE / 2); a 4-page transfer ends up requiring 8
bounce pages in the current scheme.

We should attempt to batch segments into bounce pages more efficiently.
This is achieved by pushing all considerations of the maximum segment
size into the new _bus_dmamap_addsegs() function, which wraps
_bus_dmamap_addseg(). Thus we allocate the minimal number of bounce
pages required to complete the entire transfer, while still performing
the transfer with smaller-sized transactions.

For most drivers with a segment size >= PAGE_SIZE, this will have no
impact. For drivers like dwmmc mentioned above, this improves the memory
and performance efficiency when bouncing a large transfer.

Co-authored-by: jhb
Reviewed by: jhb
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45048

busdma: deduplicate _bus_dmamap_addseg() function

It is functionally identical in all implementations, so move the
function to subr_busdma_bounce.c. The KASSERT present in the x86 version
is now enabled for all architectures. It should be universally
applicable.

Reviewed by: jhb
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45047

arm64: Switch the address argument to cpu_*cache* to a pointer

No functional change, but this reduces diffs with CheriBSD downstream.

Reviewed by: andrew
Sponsored by: University of Cambridge, Google, Inc.
Differential Revision: https://reviews.freebsd.org/D44342

busdma: fix page miscount for small segment sizes

For small segments (< PAGE_SIZE) there is a mismatch between how
required bounce pages are counted in _bus_dmamap_count_pages() and
bounce_bus_dmamap_load_buffer().

This problem has been observed on the RISC-V VisionFive v2 SoC (and
earlier revisions of the hardware) which has memory physically addressed
above 4GB. This requires some bouncing for the dwmmc driver, which has
has a maximum segment size of 2048 bytes. When attempting to load a
page-aligned 4-page buffer that requires bouncing, we can end up
counting 4 bounce pages for an 8-segment transfer. These pages will be
incorrectly configured to cover only the first half of the transfer (4 x
2048 bytes).

Fix the immediate issue by adding the maxsegsz check to
_bus_dmamap_count_pages(); this is what _bus_dmamap_count_phys() does
already. The result is that we will inefficiently allocate a separate
bounce page for each segment (8 pages for the example above), but the
transfer will proceed in its entirety.

The more complete fix is to address the shortcomings in how small
segments are assigned to bounce pages, so that we opportunistically
batch multiple segments to a page whenever they fit (e.g. two 2048 bytes
segments per 4096 page). This will be addressed more holistically in the
future. For now this change will prevent the (silent) incomplete
transfers that have been observed.

PR: 273694
Reported by: Jari Sihvola <jsihv@gmx.com>
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34118

arm64: Implement busdma bits for KMSAN

This works identically to amd64. In particular, only the
bus_dma_bounce_impl busdma implementation handles KMSAN at the moment.

MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43157

busdma: tidy bus_dma_run_filter() functions

After removing filter functionality, the naming doesn't clearly
represent what the function does, so try to address this. Include some
code clarity and style improvements.

Create a common version in subr_busdma_bounce.c, used by most
implementations. powerpc still needs its own version of the function,
due to its dmat->iommu == NULL check.

No functional change intended.

Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D42896

busdma: remove parent tag tracking

Without filter functions, we do not need to keep track of tag ancestry.
All inheritance of the parent tag's parameters occurs when creating the
new child tag.

Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D42895

busdma: kill filter functionality internally

Address filter functions are unused, unsupported, and now rejected.
Simplify some busdma code by removing filter functionality completely.

Note that the chains of parent tags become useless, and will be cleaned
up in the next commit.

No functional change intended.

Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D42894

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

busdma: On systmes that use subr_busdma_bounce, measure deferred time

Measure the total deferred time (from the time we decide to defer until
we try again) for busdma_load requests. On systems that don't ever
defer, there is no performnce change. Add new sysctl
hw.busdma.zoneX.total_deferred_time to report this (in
microseconds).

Normally, deferrals don't happen in modern hardware... Except there's a
lot of buggy hardware that can't cope with memory > 4GB or that can't
cross a 4GB boundary (or even more restrictive values), necessitating
bouncing. This will measure the effect on the I/Os of this deferral.

Sponsored by: Netflix
Reviewed by: gallatin, mav
Differential Revision: https://reviews.freebsd.org/D42550

arm64: Teach bus_dma on arm64 about NUMA

When allocating memory we should try to allocate from the NUMA node
closest to the device to reduce cross domain memory traffic. Teach the
arm64 bus_dma code to do this.

While here use mallocarray to guard against an unlikely integer
overflow.

Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42187

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

kmem_malloc/free: Use void * instead of vm_offset_t for kernel pointers.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36549

busdma_bounce: Add free_bounce_pages helper function.

Deduplicate code to iterate over the bpages list in a bus_dmamap_t
freeing bounce pages during bus_dmamap_unload.

Reviewed by: imp
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D34967

Fix a coherent bus check in the arm64 busdma

In the arm64 busdma we have an internal flag to signal when a tag is
for a cache-coherent device. In this case we don't need to adjust the
size and alignment of allocated buffers to be within a cache line.

The cache line adjustment was incorrectly using the coherent flag
passed in to bus_dma_tag_create and not the internal flag. Fix it to
use the latter to reduce the memory usage slightly.

Reviewed by: bz
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34763

Deduplicate bus_dma bounce code.

Move mostly duplicated code in various MD bus_dma backends to support
bounce pages into sys/kern/subr_busdma_bounce.c. This file is
currently #include'd into the backends rather than compiled standalone
since it requires access to internal members of opaque bus_dma
structures such as bus_dmamap_t and bus_dma_tag_t.

Reviewed by: kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33684

busdma: _bus_dmamap_addseg repaired

A recent change introduced a one-off error into a test allowing
coalescing chunks into segments. This fixes that error.

broke a check in _bus_dmamap_addseg on many architectures. This change makes it clear that it is not a particular range that is being boundary-checked, but the proposed union of the two adjacent ranges.
Reported by: se
Reviewed by: se
Fixes: c606ab59e7f9 vm_extern: use standard address checkers everywhere
Differential Revision: https://reviews.freebsd.org/D33715

vm_extern: use standard address checkers everywhere

Define simple functions for alignment and boundary checks and use them
everywhere instead of having slightly different implementations
scattered about. Define them in vm_extern.h and use them where
possible where vm_extern.h is included.

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D33685

Simplify swi for bus_dma.

When a DMA request using bounce pages completes, a swi is triggered to
schedule pending DMA requests using the just-freed bounce pages. For
a long time this bus_dma swi has been tied to a "virtual memory" swi
(swi_vm). However, all of the swi_vm implementations are the same and
consist of checking a flag (busdma_swi_pending) which is always true
and if set calling busdma_swi. I suspect this dates back to the
pre-SMPng days and that the intention was for swi_vm to serve as a
mux. However, in the current scheme there's no need for the mux.

Instead, remove swi_vm and vm_ih. Each bus_dma implementation that
uses bounce pages is responsible for creating its own swi (busdma_ih)
which it now schedules directly. This swi invokes busdma_swi directly
removing the need for busdma_swi_pending.

One consequence is that the swi now works on RISC-V which had previously
failed to invoke busdma_swi from swi_vm.

Reviewed by: imp, kib
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33447

Fix set but not used warnings in arm64 core code

arm64 busdma: Fix loading of small bounced buffers.

- Don't oversize the buffer fragment. PAGE_SIZE - (curaddr & PAGE_MASK)
may be greater than the total length of the buffer.
- Don't use roundup2(len, alignment) to calculate the buffer fragment
size. The length of current bounced fragment is not subject to alignment
restriction, and next fragment should start at the page boundary.

Tested by: bz, s199p.wa1k9r@gmail.com

Fix whitespace in comment modified by r368697

Allocate right number of pages for the bounced buffers crossing the page.

One of the disadvantages of our current busdma code is the fact that
we process the bounced buffer in a page-by-page manner. This means that
the short (subpage) buffer allocated across page boundaries is bounced
to 2 separate pages.

This suboptimal behavior is consistent across all platforms and can be
related to (probably unimplementable or incompatible with bouncing)
BUS_DMA_KEEP_PG_OFFSET flag.

Therefore, allocate one additional page to be fully comply with this
requirement.

Discused with: markj
PR: 251018

arm64: Check if we have a map before checking the flags

This fixes amdgpu on arm64 where linuxkpi is calling id_mapped
and we call might_bounce without a map.

Stop trying to bounce in memory allocated by bus dma

Memory allocated by bus_dmamem_alloc will take into account any alignment
requirements of the CPU it's running on. Stop trying to bounce in this case
as there is no bounce zone allocated.

Reported by: manu, tuexen
Tested by: manu
Sponsored by: Innovate UK

Improve loading of multipage aligned buffers.

The multipage alignment requirements is incompatible with many aspects
of actual busdma code. Multi-page alignment requests are incompatible
with many aspects of current busdma code. Mainly with partially bounced
buffer segments and per-page loop in bus_dmamap_load_buffer(). Because
proper implementation would be a major restructuring of the code, add
the fix only for already known uses and do KASSERT for all other cases.

For this reason, bus_dmamap_load_buffer () should take the memory allocated
by bus_dmam_alloc () as one segment bypassing per page segmentation. We can
do this because it is guaranteed that the memory is physically continuous.

Reviewed by: bz
Tested by: imp, mv, daniel.engberg.lists_pyret.net, kjopek_gmail.com
Differential Revision: https://reviews.freebsd.org/D26735

Bounce in more cases in the arm64 busdma

We need to use a bounce buffer when the memory we are operating on is not
aligned to a cacheline, and not aligned to the maps alignment.

The former is to stop other threads from dirtying the cacheline while we
are performing DMA operations with it. The latter is to check memory
passed in by a driver is correctly aligned for the device.

Reviewed by: mmel
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D26496

Ensure we always align and size arm64 busdma allocations to a cacheline

This will ensure nothing modifies the cacheline while DMA is in progress
so we won't need to bounce the data.

Reviewed by: mmel
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D26495

Add a coherent flag on the arm64 dma map struct

Use it to decide if we can skip cache management.

While here remove the DMAMAP_COULD_BOUNCE flag as it's unneeded.

Reviewed by: mmel
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D26494

Add bounce helpers to the arm64 busdma

Add helper functions to the arm64 busdma for common cases of checking if
we may need to bounce, and if we must bounce for a given address.

These will be expanded later as we handle cache-misaligned memory.

Reported by: mmel
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D26493

arm64: clean up empty lines in .c and .h files

Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)

r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.

This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.

Mark all obvious cases as MPSAFE. All entries that haven't been marked
as MPSAFE before are by default marked as NEEDGIANT

Approved by: kib (mentor, blanket)
Commented by: kib, gallatin, melifaro
Differential Revision: https://reviews.freebsd.org/D23718

arm64: busdma_bounce: fix BUS_DMA_ALLOCNOW for non-paged aligned sizes

For any size that isn't page-aligned, we end up not pre-allocating enough
for a single mapping because we truncate the size instead of rounding up to
make sure the last bit is accounted for, leaving us one page shy of what we
need to fulfill a request.

Differential Revision: https://reviews.freebsd.org/D22288

very large dma mappings can cause integer overflow

Reviewed by: kib
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D20505

Allow loading the same DMA address multiple times without any prior
unload for the LinuxKPI.

Reviewed by: kib, zeising
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D20181

Eliminate the arena parameter to kmem_free(). Implicitly this corrects an
error in the function hypercall_memfree(), where the wrong arena was being
passed to kmem_free().

Introduce a per-page flag, VPO_KMEM_EXEC, to mark physical pages that are
mapped in kmem with execute permissions. Use this flag to determine which
arena the kmem virtual addresses are returned to.

Eliminate UMA_SLAB_KRWX. The introduction of VPO_KMEM_EXEC makes it
redundant.

Update the nearby comment for UMA_SLAB_KERNEL.

Reviewed by: kib, markj
Discussed with: jeff
Approved by: re (marius)
Differential Revision: https://reviews.freebsd.org/D16845

Eliminate kmem_alloc_contig()'s unused arena parameter.

Reviewed by: hselasky, kib, markj
Discussed with: jeff
Differential Revision: https://reviews.freebsd.org/D16799

Oops. r338030 didn't eliminate the unused arena argument from all of
kmem_alloc_attr()'s callers. Correct that mistake.

Revert r327828, r327949, r327953, r328016-r328026, r328041:
Uses of mallocarray(9).

The use of mallocarray(9) has rocketed the required swap to build FreeBSD.
This is likely caused by the allocation size attributes which put extra pressure
on the compiler.

Given that most of these checks are superfluous we have to choose better
where to use mallocarray(9). We still have more uses of mallocarray(9) but
hopefully this is enough to bring swap usage to a reasonable level.

Reported by: wosch
PR: 225197

arm: make some use of mallocarray(9).

Focus on code where we are doing multiplications within malloc(9). None of
these ire likely to overflow, however the change is still useful as some
static checkers can benefit from the allocation attributes we use for
mallocarray.

This initial sweep only covers malloc(9) calls with M_NOWAIT. No good
reason but I started doing the changes before r327796 and at that time it
was convenient to make sure the sorrounding code could handle NULL values.

X-Differential revision: https://reviews.freebsd.org/D13837

Avoid use of contiguous memory allocations in busdma when possible.

This patch improves the boundary checks in busdma to allow more cases
using the regular page based kernel memory allocator. Especially in
the case of having a non-zero boundary in the parent DMA tag. For
example AMD64 based platforms set the PCI DMA tag boundary to
PCI_DMA_BOUNDARY, 4GB, which before this patch caused contiguous
memory allocations to be preferred when allocating more than PAGE_SIZE
bytes. Even if the required alignment was less than PAGE_SIZE bytes.

This patch also fixes the nsegments check for using kmem_alloc_attr()
when the maximum segment size is less than PAGE_SIZE bytes.

Updated some comments describing the code in question.

Differential Revision: https://reviews.freebsd.org/D10645
Reviewed by: kib, jhb, gallatin, scottl
MFC after: 1 week
Sponsored by: Mellanox Technologies

Map coherent memory in a non-coherent dma tag as uncached. This is similar
to what the 32-bit arm code does, with the exception that it always assumes
the tag is non-coherent.

Tested by: jmcneill
Obtained from: ABT Systems Ltd
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Enable setting BF_COHERENT on DMA tags. This allows the kernel to start
using the cache handling functions.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Filter out BUS_DMASYNC_POSTWRITE sync operations, there is nothing for us
to do on these.

Reported by: wma
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add support to the arm64 busdma to handle the cache. For now this is
disabled, however when we enable it it will default to assume memory is
not cache-coherent, unless either the tag was created or the parent was
marked as cache-coherent.

Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation

Rename the internal BUC_DMA_* flags to BF_* so they won't conflict with
the flags in sys/bus_dma.h.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Restrict the memory barriers in bus_dmamap_sync to just the operations
where it's needed.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

On arm64 always create a bus_dmamap_t object. This will be use to hold the
list of memory that the kernel will need to sync when operating with a
non-cache coherent DMA engine.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add data barriers to the arm64 bus_dmamap_sync function. We need these
to ensure ordering between the CPU and device. As the CPU and DMA target
may be in different shareability domains they need to be full system
barriers.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Revert r291142.

The not quite consistent logic for bounce pages allocation is utilizited
by re(4) interface which can hang now.

Approved by: kib (mentor)

Fix BUS_DMA_MIN_ALLOC_COMP flag logic. When bus_dmamap_t map is being
created for bus_dma_tag_t tag, bounce pages should be allocated
only if needed.

Before the fix, they were allocated always if BUS_DMA_COULD_BOUNCE flag
was set but BUS_DMA_MIN_ALLOC_COMP not. As bounce pages are never freed,
it could cause memory exhaustion when a lot of such tags together with
their maps were created.

Note that there could be more maps in one tag by current design.
However BUS_DMA_MIN_ALLOC_COMP flag is tag's flag. It's set after
bounce pages are allocated. Thus, they are allocated only for first
tag's map which needs them.

Approved by: kib (mentor)

Avoid a NULL pointer dereference in bounce_bus_dmamap_unload() when
the map has been created via bounce_bus_dmamem_alloc(). In that case
bus_dmamap_unload(9) typically isn't called during normal operation
but still should be during detach, cleanup from failed attach etc.

Submitted by: yongari
MFC after: 3 days

Avoid a NULL pointer dereference in bounce_bus_dmamap_sync() when the
map has been created via bounce_bus_dmamem_alloc(). Even for coherent
DMA - which bus_dmamem_alloc(9) typically is used for -, calling of
bus_dmamap_sync(9) isn't optional.

PR: 188899 (non-original problem)
MFC after: 3 days

Fix an alignment check that is wrong in half the busdma implementations.
This will enable the elimination of a workaround in the USB driver that
artifically allocates buffers twice as big as they need to be (which
actually saves memory for very small buffers on the buggy platforms).

When deciding how to allocate a dma buffer, armv4, armv6, mips, and
x86/iommu all correctly check for the tag alignment <= maxsize as enabling
simple uma/malloc based allocation. Powerpc, sparc64, x86/bounce, and
arm64/bounce were all checking for alignment < maxsize; on those platforms
when alignment was equal to the max size it would fall back to page-based
allocators even for very small buffers.

This change makes all platforms use the <= check. It should be noted that
on all platforms other than arm[v6] and mips, this check is relying on
undocumented behavior in malloc(9) that if you allocate a block of a given
size it will be aligned to the next larger power-of-2 boundary. There is
nothing in the malloc(9) man page that makes that explicit promise (but the
busdma code has been relying on this behavior all along so I guess it works).

Arm and mips code uses the allocator in kern/subr_busdma_buffalloc.c, which
does explicitly implement this promise about size and alignment. Other
platforms probably should switch to the aligned allocator.

Remove unclear comment about address truncation in busdma. Add (hopefully much clearer) comment at declaration of PHYS_TO_VM_PAGE().

Noted by: avg

Use pmap_quick* functions in arm64 busdma to make bounce buffer synchronization more flexible and avoid borrowing UVAs for userspace buffers. This is mostly equivalent to r286785 and r286787 for x86.

Differential Revision: https://reviews.freebsd.org/D3870

Port x86 busdma to ARM64

The x86 busdma subsystem allows using multiple implementations.
By default the classic bounce buffer approach is used, however
on systems with IOMMU it could be in runtime switched to more
efficient hardware accelerated implementation.

This commit adds ARM64 port of the x86 busdma framework and bounce
buffer backend. It is ready to use on IO coherent systems. If the
IO coherency cannot be guaranteed, the cache management operations have
to be added to this code in places marked by /* XXX ARM64TODO (...) */
comments. Also IOMMU support might be added by registering another
busdma implementation like it is already done on the x86.

Reviewed by: andrew, emaste
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation