spelling

add support for bypassing iommu translation

managing the translation table entries (TTEs) on an iommu is not
free, and is in fact extremely expensive on some platforms. the
flip side of this is that forcing dma through TTEs does provide
some safety and can help during the development of drivers. however,
this has been less true in recent years than it used to be and is
less of a concern now, especially considering the performance
differences on some platforms.

devices have to create dmamaps with BUS_DMA_64BIT to bypass the
iommu because the memory window presented to hardware with direct
access to memory is at an extremely high address. there's no 32bit
bypass access to memory, it has to go through TTEs otherwise.

on an m4000 there are several orders of magnitude performance
difference between a driver with BUS_DMA_64BIT set and one without
it.

hilariously, sun used a pci bridge on a whole generation of machines
that had broken support for dma addresses over 40 bits (or around
there), so devices behind those pci bridges need to have their
dmamap_creates intercepted and any potential BUS_DMA_64BIT flags
cleared on the way to the iommu drivers. this affects at least v215,
v245, and v445, and probably u25s and u45s. it probably explains
why all their onboard nics and disk controllers feel super slow,
and why there was a meme at sun that bcopy was cheaper than dma
when moving packets on and off a nic.

ok kettenis@ deraadt@

Some hardware (such as the onboard dc(4) of the Netra X1) has a broken DMA
engine that might attempt to read beyond the end of the buffer that was
programmed. The IOMMU catches this "DMA overrun" and throws an unrecoverable
error at us, at which point we have no choice but to panic. To avoid this
implement a BUS_DMA_OVERRUN flag that maps an additional scratch page at the
end of the vdma address range. DMA requests will spill over into this page,
which just returns zeroes.

Thanks to matthieu@ for giving me access to a machine with the problem.

ok deraadt@, beck@

Use extent_alloc_subregion_with_descr(9). This should make bus_dmamap_load(9)
and bus_dmamap_unload(9) "mpsafe".

type pedantry.

the type we bind to an iommu or a GART is paddr_t, by definition, on the
other hand, the type we get out of it is not a vaddr_t, it's bus_addr_t.

fix up sparc64 iommu, amd64 iommu and the sg_dma backedn that uses it to
realise this.

ok kettenis@

Make IOMMU code use the generic _bus_dmamem_map() and _bus_dmamem_unmap().

Perform IOMMU cache flushes on Oberon.

Add some members needed by the sunv IOMMU code.

use mutexes to protect the iommu's extent map and the streaming buffer,
so that busdma is most likely MP_SAFE now. (while there, fix an extent
map race... the ranges were selected outside splhigh)
ok kettenis

use the right capitalization for `SBus'

ok jmc@

Much of the sbus, psycho, and schizo bus_dma code is the same, so let's
call the real implementation functions directly instead of duplicating
lots of code that only calls the real stuff anyway.

tested by miod@ henning@
ok jason@

The existing IOMMU code had a rounding problem that was most noticeable
on faster systems under heavy network load. This replaces some of the
unreadable iommu functions with something a little less dense and a lot
less crash prone.

The bus_dma function pointer/cookie handling was broken. Change them
to work like the stacked bus_space drivers (where "work" is the key
word).

Tested my many (thanks).

ok jason@ deraadt@

Add support for the Sun Enterprise 450
Reduce the size of a GENERIC kernel by ~190k
Remove the nasty pointer/bus_space_handle_t casts
Adds debug bus_space code including the ability to trace
bus operations (it actually works now).

The following rules are now followed (and verfified by the debug
code):

1. A "bus_space_handle_t" may only be used with the
"bus_space_tag_t" that created it.
2. Only "bus_space_map()" may create "bus_space_handle_t"s.
3. A "bus_space_handle_t" may not be modified after it has
been created (other than being destroyed by "bus_space_unmap()").


Thanks to help from mcbride, marc, jason, drahn, to anyone that might
have slipped my mind at the moment.

ok jason@, deraadt@

First round of __P removal in sys

From NetBSD:
- Fix pr sparc64/15633: datafault at tlp_start causes panic
- Also clean up some additional 32-bit kernel printf issues.
- Handle the case where a DMA operation wraps from the end of a page to the
beginning of the same page properly by double-mapping that page.
- Move a brace so this will compile w/o DEBUG.
- Overhaul iommu_dvmamap_load_raw().
From Me:
- Verify the streaming cache exists before waiting on it to clear during flush.
- Force sync of flush status (hardware sets it to 1 when its done)
- Force sync flush of registers (bus_space_barrier)
- Flush ALL segments on BUS_DMA_POSTREAD operations (not just the first!)

branches: 1.4.4; 1.4.6;
Pull in several changes from NetBSD:
- Clear the dmamap inside splhigh() protection.
- Manage both streaming caches on psycho/psycho+.
And fix a botch I created when merging iommu changes last time.

Merge with NetBSD:
better handling of boundary conditions
add iommu_extract debugging function

RCSids

enough to get sbus.o from NetBSD

add support for bypassing iommu translation

managing the translation table entries (TTEs) on an iommu is not
free, and is in fact extremely expensive on some platforms. the
flip side of this is that forcing dma through TTEs does provide
some safety and can help during the development of drivers. however,
this has been less true in recent years than it used to be and is
less of a concern now, especially considering the performance
differences on some platforms.

devices have to create dmamaps with BUS_DMA_64BIT to bypass the
iommu because the memory window presented to hardware with direct
access to memory is at an extremely high address. there's no 32bit
bypass access to memory, it has to go through TTEs otherwise.

on an m4000 there are several orders of magnitude performance
difference between a driver with BUS_DMA_64BIT set and one without
it.

hilariously, sun used a pci bridge on a whole generation of machines
that had broken support for dma addresses over 40 bits (or around
there), so devices behind those pci bridges need to have their
dmamap_creates intercepted and any potential BUS_DMA_64BIT flags
cleared on the way to the iommu drivers. this affects at least v215,
v245, and v445, and probably u25s and u45s. it probably explains
why all their onboard nics and disk controllers feel super slow,
and why there was a meme at sun that bcopy was cheaper than dma
when moving packets on and off a nic.

ok kettenis@ deraadt@

Some hardware (such as the onboard dc(4) of the Netra X1) has a broken DMA
engine that might attempt to read beyond the end of the buffer that was
programmed. The IOMMU catches this "DMA overrun" and throws an unrecoverable
error at us, at which point we have no choice but to panic. To avoid this
implement a BUS_DMA_OVERRUN flag that maps an additional scratch page at the
end of the vdma address range. DMA requests will spill over into this page,
which just returns zeroes.

Thanks to matthieu@ for giving me access to a machine with the problem.

ok deraadt@, beck@

Use extent_alloc_subregion_with_descr(9). This should make bus_dmamap_load(9)
and bus_dmamap_unload(9) "mpsafe".

type pedantry.

the type we bind to an iommu or a GART is paddr_t, by definition, on the
other hand, the type we get out of it is not a vaddr_t, it's bus_addr_t.

fix up sparc64 iommu, amd64 iommu and the sg_dma backedn that uses it to
realise this.

ok kettenis@

Make IOMMU code use the generic _bus_dmamem_map() and _bus_dmamem_unmap().

Perform IOMMU cache flushes on Oberon.

Add some members needed by the sunv IOMMU code.

use mutexes to protect the iommu's extent map and the streaming buffer,
so that busdma is most likely MP_SAFE now. (while there, fix an extent
map race... the ranges were selected outside splhigh)
ok kettenis

use the right capitalization for `SBus'

ok jmc@

Much of the sbus, psycho, and schizo bus_dma code is the same, so let's
call the real implementation functions directly instead of duplicating
lots of code that only calls the real stuff anyway.

tested by miod@ henning@
ok jason@

The existing IOMMU code had a rounding problem that was most noticeable
on faster systems under heavy network load. This replaces some of the
unreadable iommu functions with something a little less dense and a lot
less crash prone.

The bus_dma function pointer/cookie handling was broken. Change them
to work like the stacked bus_space drivers (where "work" is the key
word).

Tested my many (thanks).

ok jason@ deraadt@

Add support for the Sun Enterprise 450
Reduce the size of a GENERIC kernel by ~190k
Remove the nasty pointer/bus_space_handle_t casts
Adds debug bus_space code including the ability to trace
bus operations (it actually works now).

The following rules are now followed (and verfified by the debug
code):

1. A "bus_space_handle_t" may only be used with the
"bus_space_tag_t" that created it.
2. Only "bus_space_map()" may create "bus_space_handle_t"s.
3. A "bus_space_handle_t" may not be modified after it has
been created (other than being destroyed by "bus_space_unmap()").


Thanks to help from mcbride, marc, jason, drahn, to anyone that might
have slipped my mind at the moment.

ok jason@, deraadt@

First round of __P removal in sys

From NetBSD:
- Fix pr sparc64/15633: datafault at tlp_start causes panic
- Also clean up some additional 32-bit kernel printf issues.
- Handle the case where a DMA operation wraps from the end of a page to the
beginning of the same page properly by double-mapping that page.
- Move a brace so this will compile w/o DEBUG.
- Overhaul iommu_dvmamap_load_raw().
From Me:
- Verify the streaming cache exists before waiting on it to clear during flush.
- Force sync of flush status (hardware sets it to 1 when its done)
- Force sync flush of registers (bus_space_barrier)
- Flush ALL segments on BUS_DMA_POSTREAD operations (not just the first!)

branches: 1.4.4; 1.4.6;
Pull in several changes from NetBSD:
- Clear the dmamap inside splhigh() protection.
- Manage both streaming caches on psycho/psycho+.
And fix a botch I created when merging iommu changes last time.

Merge with NetBSD:
better handling of boundary conditions
add iommu_extract debugging function

RCSids

enough to get sbus.o from NetBSD

Some hardware (such as the onboard dc(4) of the Netra X1) has a broken DMA
engine that might attempt to read beyond the end of the buffer that was
programmed. The IOMMU catches this "DMA overrun" and throws an unrecoverable
error at us, at which point we have no choice but to panic. To avoid this
implement a BUS_DMA_OVERRUN flag that maps an additional scratch page at the
end of the vdma address range. DMA requests will spill over into this page,
which just returns zeroes.

Thanks to matthieu@ for giving me access to a machine with the problem.

ok deraadt@, beck@

Use extent_alloc_subregion_with_descr(9). This should make bus_dmamap_load(9)
and bus_dmamap_unload(9) "mpsafe".

type pedantry.

the type we bind to an iommu or a GART is paddr_t, by definition, on the
other hand, the type we get out of it is not a vaddr_t, it's bus_addr_t.

fix up sparc64 iommu, amd64 iommu and the sg_dma backedn that uses it to
realise this.

ok kettenis@

Make IOMMU code use the generic _bus_dmamem_map() and _bus_dmamem_unmap().

Perform IOMMU cache flushes on Oberon.

Add some members needed by the sunv IOMMU code.

use mutexes to protect the iommu's extent map and the streaming buffer,
so that busdma is most likely MP_SAFE now. (while there, fix an extent
map race... the ranges were selected outside splhigh)
ok kettenis

use the right capitalization for `SBus'

ok jmc@

Much of the sbus, psycho, and schizo bus_dma code is the same, so let's
call the real implementation functions directly instead of duplicating
lots of code that only calls the real stuff anyway.

tested by miod@ henning@
ok jason@

The existing IOMMU code had a rounding problem that was most noticeable
on faster systems under heavy network load. This replaces some of the
unreadable iommu functions with something a little less dense and a lot
less crash prone.

The bus_dma function pointer/cookie handling was broken. Change them
to work like the stacked bus_space drivers (where "work" is the key
word).

Tested my many (thanks).

ok jason@ deraadt@

Add support for the Sun Enterprise 450
Reduce the size of a GENERIC kernel by ~190k
Remove the nasty pointer/bus_space_handle_t casts
Adds debug bus_space code including the ability to trace
bus operations (it actually works now).

The following rules are now followed (and verfified by the debug
code):

1. A "bus_space_handle_t" may only be used with the
"bus_space_tag_t" that created it.
2. Only "bus_space_map()" may create "bus_space_handle_t"s.
3. A "bus_space_handle_t" may not be modified after it has
been created (other than being destroyed by "bus_space_unmap()").


Thanks to help from mcbride, marc, jason, drahn, to anyone that might
have slipped my mind at the moment.

ok jason@, deraadt@

First round of __P removal in sys

From NetBSD:
- Fix pr sparc64/15633: datafault at tlp_start causes panic
- Also clean up some additional 32-bit kernel printf issues.
- Handle the case where a DMA operation wraps from the end of a page to the
beginning of the same page properly by double-mapping that page.
- Move a brace so this will compile w/o DEBUG.
- Overhaul iommu_dvmamap_load_raw().
From Me:
- Verify the streaming cache exists before waiting on it to clear during flush.
- Force sync of flush status (hardware sets it to 1 when its done)
- Force sync flush of registers (bus_space_barrier)
- Flush ALL segments on BUS_DMA_POSTREAD operations (not just the first!)

branches: 1.4.4; 1.4.6;
Pull in several changes from NetBSD:
- Clear the dmamap inside splhigh() protection.
- Manage both streaming caches on psycho/psycho+.
And fix a botch I created when merging iommu changes last time.

Merge with NetBSD:
better handling of boundary conditions
add iommu_extract debugging function

RCSids

enough to get sbus.o from NetBSD