MFC r318353:
Avoid use of contiguous memory allocations in busdma when possible.

This patch improves the boundary checks in busdma to allow more cases
using the regular page based kernel memory allocator. Especially in
the case of having a non-zero boundary in the parent DMA tag. For
example AMD64 based platforms set the PCI DMA tag boundary to
PCI_DMA_BOUNDARY, 4GB, which before this patch caused contiguous
memory allocations to be preferred when allocating more than PAGE_SIZE
bytes. Even if the required alignment was less than PAGE_SIZE bytes.

This patch also fixes the nsegments check for using kmem_alloc_attr()
when the maximum segment size is less than PAGE_SIZE bytes.

Updated some comments describing the code in question.

Differential Revision: https://reviews.freebsd.org/D10645
Reviewed by: kib, jhb, gallatin, scottl
Sponsored by: Mellanox Technologies

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Rename busdma_machdep.c to busdma_machdep-v4.c, pmap.c to pmap-v4.c
and trap.c to trap-v4.c to be plain and consistent with other armv4
specific files.

Revert r291142.

The not quite consistent logic for bounce pages allocation is utilizited
by re(4) interface which can hang now.

Approved by: kib (mentor)

Fix BUS_DMA_MIN_ALLOC_COMP flag logic. When bus_dmamap_t map is being
created for bus_dma_tag_t tag, bounce pages should be allocated
only if needed.

Before the fix, they were allocated always if BUS_DMA_COULD_BOUNCE flag
was set but BUS_DMA_MIN_ALLOC_COMP not. As bounce pages are never freed,
it could cause memory exhaustion when a lot of such tags together with
their maps were created.

Note that there could be more maps in one tag by current design.
However BUS_DMA_MIN_ALLOC_COMP flag is tag's flag. It's set after
bounce pages are allocated. Thus, they are allocated only for first
tag's map which needs them.

Approved by: kib (mentor)

Eliminate the last dregs of the old global arm_root_dma_tag.

In the old days, device drivers passed NULL for the parent tag when creating
a new tag, and on arm platforms that resulted in a global tag representing
overall platform constraints being substituted in the busdma code. Now all
drivers use bus_get_dma_tag() and if there is a need to represent overall
platform constraints they will be inherited from a tag supplied by nexus or
some bus driver in the hierarchy.

The only arm platforms still relying on the old global-tag scheme were some
xscale boards with special PCI-bus constraints. This change provides those
constraints through a tag supplied by the xscale PCI bus driver, and
eliminates the few remaining references to the old global var.

Reviewed by: cognet

Bring in all the new(-ish) statistics code from armv6.

Change the preallocation of a busdma segment mapping array from per-tag to
per-map. The per-tag scheme is not safe, and a mutex can't be used to
protect it because the mapping routines can't sleep. Code brought in
from armv6 implementation.

Instead of all memory allocations using M_DEVBUF, use new categories
M_BUSDMA for allocations of metadata (tags, maps, segment tracking lists),
and M_BOUNCE for bounce pages.

Whitespace and style nits, no functional changes.

The goal is to make these two files cosmetically alike so that the actual
implementation differences are visible. The only changes which aren't
spaces<->tabs and rewrapping and reindenting lines are a couple fields
shuffled around in the tag and map structs so that everything is in the same
order in both versions (which should amount to no functional change).

Remove unclear comment about address truncation in busdma. Add (hopefully much clearer) comment at declaration of PHYS_TO_VM_PAGE().

Noted by: avg

Use pmap_quick* for out-of-context bounce buffers and (limited) cache maintenance of unmapped buffers in armv5 busdma.

Tested by: Mattia Rossi <mattia.rossi.mailinglists@gmail.com>
Differential Revision: https://reviews.freebsd.org/D3522

Follow the advice of the misplaced comment and don't access the map struct
after freeing it. Remove the comment whose uselessness has been revealed.

Remove trailing whitespace from sys/arm/arm

The add_bounce_page() function can be called when loading physical
pages which pass a NULL virtual address. If the BUS_DMA_KEEP_PG_OFFSET
flag is set, use the physical address to compute the page offset
instead. The physical address should always be valid when adding
bounce pages and should contain the same page offset like the virtual
address.

Submitted by: Svatopluk Kraus <onwahe@gmail.com>
MFC after: 1 week
Reviewed by: jhb@

Fix multiple incorrect SYSCTL arguments in the kernel:

- Wrong integer type was specified.

- Wrong or missing "access" specifier. The "access" specifier
sometimes included the SYSCTL type, which it should not, except for
procedural SYSCTL nodes.

- Logical OR where binary OR was expected.

- Properly assert the "access" argument passed to all SYSCTL macros,
using the CTASSERT macro. This applies to both static- and dynamically
created SYSCTLs.

- Properly assert the the data type for both static and dynamic
SYSCTLs. In the case of static SYSCTLs we only assert that the data
pointed to by the SYSCTL data pointer has the correct size, hence
there is no easy way to assert types in the C language outside a
C-function.

- Rewrote some code which doesn't pass a constant "access" specifier
when creating dynamic SYSCTL nodes, which is now a requirement.

- Updated "EXAMPLES" section in SYSCTL manual page.

MFC after: 3 days
Sponsored by: Mellanox Technologies

Pull in r267961 and r267973 again. Fix for issues reported will follow.

Revert r267961, r267973:

These changes prevent sysctl(8) from returning proper output,
such as:

1) no output from sysctl(8)
2) erroneously returning ENOMEM with tools like truss(1)
or uname(1)
truss: can not get etype: Cannot allocate memory

Extend the meaning of the CTLFLAG_TUN flag to automatically check if
there is an environment variable which shall initialize the SYSCTL
during early boot. This works for all SYSCTL types both statically and
dynamically created ones, except for the SYSCTL NODE type and SYSCTLs
which belong to VNETs. A new flag, CTLFLAG_NOFETCH, has been added to
be used in the case a tunable sysctl has a custom initialisation
function allowing the sysctl to still be marked as a tunable. The
kernel SYSCTL API is mostly the same, with a few exceptions for some
special operations like iterating childrens of a static/extern SYSCTL
node. This operation should probably be made into a factored out
common macro, hence some device drivers use this. The reason for
changing the SYSCTL API was the need for a SYSCTL parent OID pointer
and not only the SYSCTL parent OID list pointer in order to quickly
generate the sysctl path. The motivation behind this patch is to avoid
parameter loading cludges inside the OFED driver subsystem. Instead of
adding special code to the OFED driver subsystem to post-load tunables
into dynamically created sysctls, we generalize this in the kernel.

Other changes:
- Corrected a possibly incorrect sysctl name from "hw.cbb.intr_mask"
to "hw.pcic.intr_mask".
- Removed redundant TUNABLE statements throughout the kernel.
- Some minor code rewrites in connection to removing not needed
TUNABLE statements.
- Added a missing SYSCTL_DECL().
- Wrapped two very long lines.
- Avoid malloc()/free() inside sysctl string handling, in case it is
called to initialize a sysctl from a tunable, hence malloc()/free() is
not ready when sysctls from the sysctl dataset are registered.
- Bumped FreeBSD version to indicate SYSCTL API change.

MFC after: 2 weeks
Sponsored by: Mellanox Technologies

Add bus_dmamap_load_ma() function to load map with the array of
vm_pages. Provide trivial implementation which forwards the load to
_bus_dmamap_load_phys() page by page. Right now all architectures use
bus_dmamap_load_ma_triv().

Tested by: pho (as part of the functional patch)
Sponsored by: The FreeBSD Foundation
MFC after: 1 month

Eliminate a compiler warning about extraneous parens.

Retire arm_remap_nocache() and the data and constants associated with it.

The only remaining user was the code that allocates bounce pages for armv4
busdma. It's not clear why bounce pages would need uncached memory, but
if that ever changes, kmem_alloc_attr() would be the way to get it.

Replace kernel virtual address space allocation with vmem. This provides
transparent layering and better fragmentation.

- Normalize functions that allocate memory to use kmem_*
- Those that allocate address space are named kva_*
- Those that operate on maps are named kmap_*
- Implement recursive allocation handling for kmem_arena in vmem.

Reviewed by: alc
Tested by: pho
Sponsored by: EMC / Isilon Storage Division

In _bus_dmamap_addseg(), the return value must be zero for error, or the size
actually added to the segment (possibly smaller than the requested size if
boundary crossings had to be avoided).

Reform the busdma API so that new types may be added without modifying
every architecture's busdma_machdep.c. It is done by unifying the
bus_dmamap_load_buffer() routines so that they may be called from MI
code. The MD busdma is then given a chance to do any final processing
in the complete() callback.

The cam changes unify the bus_dmamap_load* handling in cam drivers.

The arm and mips implementations are updated to track virtual
addresses for sync(). Previously this was done in a type specific
way. Now it is done in a generic way by recording the list of
virtuals in the map.

Submitted by: jeff (sponsored by EMC/Isilon)
Reviewed by: kan (previous version), scottl,
mjacob (isp(4), no objections for target mode changes)
Discussed with: ian (arm changes)
Tested by: marius (sparc64), mips (jmallet), isci(4) on x86 (jharris),
amd64 (Fabian Keil <freebsd-listen@fabiankeil.de>)

Use pmap_kextract() instead of inlining the page table walk.
Remove the comment referencing non-existing code.

Reviewed by: cognet, ian (previous version)
Tested by: ian

The manpage states that bus_dmamap_create(9) returns ENOMEM if it can't
allocate a map or mapping resources. That seems to imply that any memory
allocations it does must use M_NOWAIT and check for NULL.

Submitted by: Ian Lepore <freebsd@damnhippie.dyndns.org>

Use C comments instead of C++ comments.

Spotted out by: gonzo (thanks, man)

Busdma enhancements, especially for managing small uncacheable buffers.

- Use the new architecture-agnostic buffer pool manager that uses uma(9)
to manage a set of power-of-2 sized buffers for bus_dmamem_alloc().

- Create pools of buffers backed by both regular and uncacheable memory,
and use them to handle regular versus BUS_DMA_COHERENT allocations.

- Use uma(9) to manage a pool of bus_dmamap structs instead of local code
to manage a static list of 500 items (it took 3300 maps to get to
multi-user mode, so the static pool wasn't much of an optimization).

- Small BUS_DMA_COHERENT allocations no longer waste an entire page per
allocation, or set pages to uncached when they contain data other than
DMA buffers. There's no longer a need for drivers to work around the
inefficiency by allocing large buffers then sub-dividing them.

- Because we know the alignment and padding of buffers allocated by
bus_dmamem_alloc() (whether coherent or regular memory, and whether
obtained from the pool allocator or directly from the kernel) we
can avoid doing partial cacheline flushes on them.

- Add a fast-out to _bus_dma_could_bounce() (and some comments about
what the routine really does because the old misplaced comment was wrong).

- Everywhere the dma tag alignment is used, the interpretation is that
an alignment of 1 means no special alignment. If the tag is created
with an alignment argument of zero, store it in the tag as one, and
remove all the code scattered around that changed 0->1 at point of use.

- Remove stack-allocated arrays of segments, use a local array of two
segments within the tag struct, or dynamically allocate an array at first
use if nsegments > 2. On an arm system I tested, only 5 of 97 tags used
more than two segments. On my x86 desktop it was only 7 of 111 tags.

Submitted by: Ian Lepore <freebsd@damnhippie.dyndns.org>

Dynamically allocate the S/G lists passed to callback routines rather than
allocating them on the stack of various bus_dmamap_load*() functions. The
S/G lists are stored in the DMA tags. This matches the implementation on
all other platforms.

Discussed with: scottl, gibbs
Tested by: stas (arm@)

Final whitespace trim.

Interrupts must be disabled while handling a partial cache line flush,
as otherwise the interrupt handling code may modify data in the non-DMA
part of the cache line while we have it stashed away in the temporary
stack buffer, then we end up restoring a stale value.

PR: 160431
Submitted by: Ian Lepore
MFC after: 1 week

- Change contigmalloc() to use the vm_paddr_t type instead of an unsigned
long for specifying a boundary constraint.
- Change bus_dma tags to use bus_addr_t instead of bus_size_t for boundary
constraints.

These allow boundary constraints to be fully expressed for cases where
sizeof(bus_addr_t) != sizeof(bus_size_t). Specifically, it allows a
driver to properly specify a 4GB boundary in a PAE kernel.

Note that this cannot be safely MFC'd without a lot of compat shims due
to KBI changes, so I do not intend to merge it.

Reviewed by: scottl

Mark all SYSCTL_NODEs static that have no corresponding SYSCTL_DECLs.

The SYSCTL_NODE macro defines a list that stores all child-elements of
that node. If there's no SYSCTL_DECL macro anywhere else, there's no
reason why it shouldn't be static.

Fix ARM cache handling yet more.

1) vm_machdep.c: remove the dangling allocations so they do not
un-necessarily turn off the cache upon consecutive access.

2) busdma_machdep.c: remove the same amount than shadow mapped.

Reported by: Maks Verver
Submitted by: Mark Tinguely
Reviewed by: Grzegorz Bernacki
MFC after: 3 days

Do not free the dmamap if it is still busy.

Submitted by: Mark Tinguely
MFC after: 3 days

ARM pmap fixes.

a) nocache-remap problem

When a page is remapped into a non-cacheable virtual memory region there
was no associated write-back invalidate operation performed. We remove
writeback of the original buffer size from bus_dmamem_alloc() and add
appropriate L1/L2 flush operation.

b) missing write-back invalidate operation

In pmap_kremove a page is removed so we must do a write-back
invalidate operation aligned to the page virtual address.

Submitted by: Michal Hajduk
Reviewed by: Mark Tinguely, rpaulo, stas
Approved by: re (kib)
Obtained from: Semihalf

Reduce the number of bounce zones (and thus the number of bounce pages
used in some cases):
- Ignore DMA tag boundaries when allocating bounce pages. The boundaries
don't determine whether or not parts of a DMA request bounce. Instead,
they are just used to carve up segments.
- Allow tags with sub-page alignment to share bounce pages since bounce
pages are always page aligned.

Reviewed by: scottl (amd64)
MFC after: 1 month

Restore bus DMA bounce pages to an offset of 0 when they are released by
a tag that has BUS_DMA_KEEP_PG_OFFSET set. Otherwise the page could be
reused with a non-zero offset by a tag that doesn't have
BUS_DMA_KEEP_PG_OFFSET leading to data corruption.

Sleuthing by: avg
Reviewed by: scottl

The bus_dmamap_load_uio(9) shall use pmap of the thread recorded in the
uio_td to extract pages from, instead of unconditionally use kernel
pmap.

Submitted by: Jason Harmening <jason.harmening gmail com> (amd64 version)
PR: amd64/133592
Reviewed by: scottl (original patch), jhb
MFC after: 2 weeks

The bounce zone sees its page number increased if multiple dma maps use it in
the same dma tag. However, it can happen multiple dma tags share the same
bounce zone too, so add a per-bounce zone map counter, and check it instead of
the dma tag map counter, to know if we have to alloc more pages.

Reported by: miwi
Reviewed by: scottl

When bouncing pages, allow a new option to preserve the intra-page
offset. This is needed for the ehci hardware buffer rings that assume
this behavior.

This is an interim solution, and a more general one is being worked
on. This solution doesn't break anything that doesn't ask for it
directly. The mbuf and uio variants with this flag likely don't work
and haven't been tested.

Universe builds with these changes. I don't have a huge-memory
machine to test these changes with, but will be happy to work with
folks that do and hps if this changes turns out not to be sufficient.

Submitted by: alfred@ from Hans Peter Selasky's original

Erm... Report the buffer as being bounced even when it's the entire buffer,
or we would end up invalidating the cache line for what we just copied...

Reported by: thompsa
Pointy at to: cognet

MFC after: 3 days

Increment total_bounced busdma stat as required.

- Get rid of unused variable in KTR checks. This allows ktr(4) enabled
ARM kernel to compile.

PR: arm/128897
Submitted by: Pankov Pavel <pankov_p@mail.ru>
Reviewed by: raj
Approved by: kib (mentor, implicit)
MFC after: 1 week

Provide L2 cache synchronization (write back + invalidation) on ARM.

Note the cpu_l2cache_wbinv_* routines are no-ops on systems not populated with
L2 caches.

Obtained from: Marvell, Semihalf

Improve ARM bus_dmamap_load_buffer() error handling.

Reviewed by: imp
Approved by: cognet (mentor)
Spotted by: Grzegorz Bernacki gjb AT semihalf DOT com

Correct the logic : we can just invalidate the cache lines, and not
write-back them, only if PREWRITE is not set, and if the buffer is
cache-line aligned.

MFC After: 1 week

Remove the 'needbounce' variable from the _bus_dmamap_load_buffer()
routine. It is not needed as the existing tests for segment coalescing
already handle bounced addresses and it prevents legal segment coalescing
in certain edge cases.

MFC after: 1 week
Reviewed by: scottl

Just wbinv if both PREREAD and PREWRITE are set.
In PREREAD, just invalidate the cache lines, and do not write back them, if
the buffer is properly aligned.

Approved by: re (blanket)

Handle supersections and L2 cache.

Approved by: re (blanket)

Initialize the dma tag's bounce_zone to NULL if we didn't allocate it.

There's no nobounce_dmamap on arm.

Honor maxsegsz of less than a page size in a DMA tag. Previously it
used to return PAGE_SIZE without respect to restrictions of a DMA tag.
This affected all of the busdma load functions that use
_bus_dmamap_loader_buffer() as their back-end.

Reviewed by: scottl

Do not try to inline bus_dmamap_sync_buf(), gcc 4.2 doesn't want to do so
because it uses alloca().
Initialize lastaddr in bus_dmamap_load_uio().

Fix typo: MacPPC -> ARM

- Add bounce pages for arm, largely based on the i386 implementation.
- Add a default parent dma tag, similar to what has been done for sparc64.
- Before invalidating the dcache in POSTREAD, save the bits which are in the
same cachelines than our buffers, but not part of it, and restore them after
the invalidation.

Explicitely set the "allocbuffer" field to NULL when creating a new dmamap.

If our buffer is not aligned on the cache line size, write back/invalidate
the first and last cache line in PREREAD, and just invalidate the cache
lines in POSTREAD, instead of write-back/invalidating in POSTREAD, which
could lead to stale data overriding what has been transfered by DMA.

Try to honor BUS_DMA_COHERENT : if the flag is set, normally allocate memory
with malloc() or contigmalloc() as usual, but try to re-map the allocated
memory into a VA outside the KVA, non-cached, thus making the calls to
bus_dmamap_sync() for these buffers useless.

Bring in the good version of this file.

*blush*
Don't try to dereference map if it's NULL.
While I'm there, increase the minimum value to write-back/invalidate the
whole dcache in bus_dmamap_sync().

- Use a TAILQ instead of parsing the array to find a free dmamap.
- Inline busdma_alloc_dmamap, busdma_free_dmamap and bus_dmamap_sync_buf.

Approved by: re (blanket)

- Try to avoid calling malloc() in bus_dmamap_create() and bus_dmamem_alloc()
for the dmamap by using static dmamaps.
- Don't do anything for BUS_DMASYNC_PREREAD and BUS_DMASYNC_POSTWRITE in
bus_dmamap_sync(), it's not needed anymore.

make bus_dmamem_alloc always allocate a new map like we are suppose to..
This was found when I tried to run the usb code on my arm board...

Approved by: cognet

fix arm wrt to busdma...

also wrap the two macros that have bare if's w/ do {} while(0) so that
my epe driver doesn't get a warning about braces around confused else...

Fixup KTR traces.

Use __func__ in the KTR_BUSDMA traces. This avoids copy and paste
errors like in the bus_dmamap_load_mbuf_sg() case where we were wrongly
displaying the function name as bus_dmamap_load_mbuf.

Fix typo.

netchild's mega-patch to isolate compiler dependencies into a central
place.

This moves the dependency on GCC's and other compiler's features into
the central sys/cdefs.h file, while the individual source files can
then refer to #ifdef __COMPILER_FEATURE_FOO where they by now used to
refer to #if __GNUC__ > 3.1415 && __BARC__ <= 42.

By now, GCC and ICC (the Intel compiler) have been actively tested on
IA32 platforms by netchild. Extension to other compilers is supposed
to be possible, of course.

Submitted by: netchild
Reviewed by: various developers on arch@, some time ago

Define bus_dmamap_load_buffer before bus_dmamap_load to make gcc happy.

Fix compile for !KTR.

Erm, don't forget to store the mbuf in the dmamap in bus_dmamap_load_mbuf_sg(),
so that bus_dmamap_sync() knows what to invalidate. This makes em(4) work again.
While I'm there, remove the unused "first" variable.

Add support for KTR_BUSDMA.

MFi386: add bus_dmamap_load_mbuf_sg().

Start all license statements with /*-

Fix compilation with INVARIANTS.

Simplify a bit bus_dmamap_load_buffer by removing the "first" parameter, use
nseg == -1 instead.

Obtained from: NetBSD

MFi386:
- inlina bus_dmamap_load_buffer
- Directly pass the pmap to bus_dmamap_load_buffer, instead of the struct thread

We want to ignore BUS_DMASYNC_POSTWRITE, not BUS_DMASYNC_POSTREAD.
Spotted out by: mux
Pointy hat to: cognet

Don't attempt to manage our own segment list, and just remember the buffers
provided.

Obtained from: NetBSD

Fix a problem with tag->boundary inheritence that has existed since day one
and was propagated to nearly every platform. The boundary of the child needs
to consider the boundary of the parent and pick the minimum of the two, not
the maximum. However, if either is 0 then pick the appropriate one.
This bug was exposed by a recent change to ATA, which should now be fixed by
this change. The alignment and maxsegsz tag attributes likely also need
a similar review in the near future.

This is a MT5 candidate.

Reviewed by: marcel
Submitted by: sos (in part)

Use the kernel pmap if no thread is provided.

Import FreeBSD/arm kernel bits.
It only supports sa1110 (on simics) right now, but xscale support should come
soon.
Some of the initial work has been provided by :
Stephane Potvin <sepotvin at videotron.ca>
Most of this comes from NetBSD.