With the introduction of the PMAP_PREFER_{ALIGN,OFFSET} macros a long time ago,
there are actually no more uses of the PMAP_PREFER() macro left in the kernel.
Remove that macro but keep PMAP_PREFER as a simple #define for it to let uvm
knows the PMAP_PREFER_{ALIGN,OFFSET} macros are available.

ok mpi@

Drop orphaned pv_flags values.

Remove pmap_collect() when a no-op, define __HAVE_PMAP_COLLECT otherwise.
Use that define to shunt uvm_swapout_threads(), which is a noop when
pmap_collect() does nothing.

ok mpi@

Convert KVA allocation to km_alloc(9).

ok mpi@

Make sure we always pass a page-aligned address to pmap_grow_map().
Fixes an issue uncovered by the recent change to enlarge kva space where
an unaligned address was passed resulting in memset() writing past the
end of the newly allocated page.

ok miod@

remove uneeded includes in md armv7 files

based on include-what-you-use suggestions

spelling

Convert boolean_t/TRUE/FALSE to int/1/0 for coherency with the rest of
the kernel.

ok patrick@

revert revert revert. there are many other archs that use custom allocs.

make it possible to reduce kmem pressure by letting some pools use a more
accomodating allocator. an interrupt safe pool may also be used in process
context, as indicated by waitok flags. thanks to the garbage collector, we
can always free pages in process context. the only complication is where
to put the pages. solve this by saving the allocation flags in the pool
page header so the free function can examine them.
not actually used in this diff. (coming soon.)
arm testing and compile fixes from phessler

If you use sys/param.h, you don't need sys/types.h

If an Access Flag fault happens while we were running the kernel and
it happened on a kernel page, we need to consult the kernel pmap
instead of the current proc's pmap. Fixes panic when using tmpfs.

ok kettenis@

Change pmap_proc_iflush() to take a process instead of a proc
powerpc: rename second argument of pmap_proc_iflush() to match other archs

ok kettenis@

Remove a couple of unsused static inline functions. Also remove a comparis
of an array to a null pointer that is always false. Found with clang.

ok jsg@

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

In pmap_activate instead of doing disable_interrupts/enable_interrupts
nested inside of splhigh/splx just disable and enable interrupts
once with inline cps instructions.

Remove uneeded pcb_pl1vec block as well, suggested by kettenis.

ok kettenis@

Add support for the PXN bit in level 1 translation table descriptors and
enable it on CPUs that support it. When enabled, this prevents the kernel
from executing userland code.

ok jsg@, tom@

Remove cpu_cpwait() calls; they are no-ops on armv7.

ok tom@

Remove the code that switches around MMU domains on armv7. MMU domains are
basically a relic from the past. Using them doesn't make a lot of sense
the way our pmaps work. Support for MMU domains isn't present in
long-descriptor translation table format, so it is clearly on its way out.

Based on a diff from Artituri Alm.

ok patrick@

Replace pmap_fault_fixup() with an access flag fault handler on armv7.

ok tom@

Fix indentation.

Argh, commit from the wrong tree. Revert previous commit.

Fix indentation.

Correctly enter a mapping as writable if no "page modified" emulation
is needed.

ok visa@, patrick@

Start using to XN flag to enforce that mappings without PROT_EXEC are
non-executable.

ok visa@, deraadt@

Mark device memory as execute-never to prevent a speculative instruction fetch
to access it.

Use Access Flag to do page reference emulation.

ok visa@

Separate out the Access Flag bit from the Access Permission bits in the
armv7 pmap.

ok tom@

Fix typo/inconsistensy where L1_S_DOMAIN was used instead of L1_C_DOMAIN.
These are functionally equivolent so it didn't matter and the resulting
code doesn't change.

The ARMv7 ARM says that the TLB may hold translation table entries at any
level of the translation table, including entries that point to further
levels of the tables. This means that we have to do a TLB flush whenever
we invalidate an L1 slot too. Doing so fixes the pmap_fault_fixup
issue on Cortex-A7 processors.

The page tables are cached now, and given the significant speedup, I
don't think we'll ever go back. So let's ditch the code that tries to
check and patch up incorrect memory attributes.

Also realize that pmap_clean_page(pg, FALSE) doesn't do anything
anymore so remove those calls and drop the 2nd argument from
pmap_clean_page(pg, TRUE) calls.

Last but not least, get rid of pmap_pte_init_generic() here. The only
useful thing it did was setting pmap_copy_page_func() and
pmap_zero_page_func().

This diff should not introduce any change in behaviour.

ok visa@

Mapping non-cachable memory as cachable and subsequently changing the mapping
to non-cachable is retarded. Fix this by introducing PMAP_NOCACHE and
PMAP_DEVICE flags that can be or'ed into the physical address passed to
pmap_kenter(9), like we have on many of our other architectures. This way we
can also properly distinguish between device memory and normal (non-cachable)
memory.

ok visa@

ARMv7 data caches are "effectively" PIPT. This means there is in general
no need to clean and/or invalidate cached pages. So remove most of the
cache cleaning and invalidation from the pmap. We still need to synchronize
the instruction cache with the data cache in various places though. And we
also need to make sure that we clean and invalidate when we make a page
non-cachable.

Tested by Daniel Bolgheroni, mglocker@ and jsg@. on Cortex-A8 and myself on
Cortex-A9.

ok visa@

Put page tables in normal cachable memory on armv7. Check if the MMU walks
the page tables coherently and also skip flushing modified ptes out of the
cache in that case. Speeds up building a kernel with a factor of two on
Cortex-A9 (tested by me) and Cortex-A8 (tested by mglocker@).

ok patrick@

Simplify the way we handle TLB flushes. Since ARMv7 effectively has a
unified TLB there is not much point in optimizing TLB flushing for pages
that have never been executable. The only difference is a flush of the
branch predictor and even that isn't necessary anymore on all but the oldest
Cortex cores.

ok patrick@

Remove devmap stuff which is unused on armv7.

ok patrick@

According to te armv7 ARM TLB entries that caused a Permission fault might
be held in the TLB. On top of that valid page table entries might be
speculatively loaded into the TLB. As a result we need to flush TLB entries
even when the page in question has not been referenced.

Fixes pmap_fault_fixup messages on Cortex-A53, and presumably also on
Cortex-A7.

ok patrick@, guenther@

Only flush the virtual page if it was actually mapped. Otherwise
we will run into translation faults.

ok tom@

When pmap_page_remove() is called by UVM, a physical page is to be
removed from pmaps it currently is in. To check if a virtual address
pointing to that physical page has been mapped, the code uses
the l2pte_valid() function. Unfortunately there is a difference
between being valid and the PTE being zero. If a page is mapped
but has never been accessed, it will be non-zero but invalid.

In that case the PTE for that virtual address will not be zeroed
and the virtual address will be removed from the vm page struct.

The next time someone tries to map a page to that virtual address,
other pmap code will consider the virtual address to be already
mapped, even though that assumption is completely wrong.

To make sure this does not happen, check the PTE for zero. This way
the PTE will be zeroed correctly. The check for zero is how other
ARM pmap code also handles this issue.

ok kettenis@ tom@

Remove what appears to be a copy-paste error setting cur_ttb
in pmap_free_l1().

from aalm@ - thanks

ok patrick@

Don't need a separate flags variable in armv7 pmap_clean_page() -
just use the pv_flags. ('Twas a copy-paste from arm's pmap_clean_page(),
which did need it.)

Also remove even less used flags variable from pmap_page_remove().

First part from a diff from aalm@ - thanks

ok kettenis@ "looks good" patrick@

consistently set ipls on pmap pools.

this is a step toward making ipls unconditionaly on pools.

ok deraadt@ kettenis@

When a physical address is needed to flush the secondary cache use
VM_PAGE_TO_PHYS() instead of unnecessarily calling pmap_extract().

From Patrick Wildt.

Fix the encoding of AP bits for large page second-level
short-descriptors with arm v7 (same as small page encoding, except XN is
in a different bit for the mask).

Expanded version of a diff from Patrick Wildt who also tested and
reviewed this.

Switch from PSR_X_bit and X32_bit PSR macro names to just PSR_X.
This matches FreeBSD and makes things a bit more consistent.
Discussed with Patrick.

Give the pool page allocator backends more sensible names. We now have:
* pool_allocator_single: single page allocator, always interrupt safe
* pool_allocator_multi: multi-page allocator, interrupt safe
* pool_allocator_multi_ni: multi-page allocator, not interrupt-safe

ok deraadt@, dlg@

Remove some pmap locks that were #defined to be nothing (empty). Discussed
with many, ok kettenis@.

remove no-op simple locks
tested by jsg, ok miod

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

use #ifdef DDB for Debugger()

Correct the l1 pte permission bits for armv7. Problem pointed out
by Patrick Wildt who made a similiar change in Bitrig.

ok miod@ rapha@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

Format string fixes and removal of -Wno-format for arm kernels.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

move the arm pmaps away from pool ctors to just initting memory after its
been allocated. also gets rid of a potential bug where pool_get could
return NULL and pmap_alloc_l2_ptp tried to blindly init it.

tests, tweaks, and ok patrick@

The cache mask for ARMv7 is a little bit different from the one inited by
the generic function. While there, also set the cache bits manually.

ok aalm@

Some current boards do not have write-through caching. For those,
we enable write-back, but it seems this is not working and those
boards hang on bootup. Until that is fixed, do not cache PTEs
on those boards.

Rewrite the ARMv7 cache discovery, as some assumptions in the previous one
were utterly wrong. Fix Log2(), correct one taken from the scheduler code.

Tested by rapha@ and Artturi Alm.

typo

We're handling L2 there, so use the corresponding define, not the L1 one.

ok bmercer@

When mapping a new entry, map it read-only, even though it should be
writable. This will cause a pmap fault on first write, so that we can
mark the page as modified. Also mask the bits used for the protection
settings, so that there aren't any leftovers.

ok bmercer@

On ARMv7 we can't use the cache mask to check for coherency.
Therefore we add new macros to be able to check for it properly.

ok miod@

Revert the ARMv7 header split introduced in pmap7.

ok bmercer@

Add secondary cache flushes to armv7's pmap.

ok bmercer@

Further updates to pmap7. Fixes some problems and removes debug printfs.

ok miod@

New pmap for panda boards. Work from lots of folks.
OK miod@

Drop orphaned pv_flags values.

Remove pmap_collect() when a no-op, define __HAVE_PMAP_COLLECT otherwise.
Use that define to shunt uvm_swapout_threads(), which is a noop when
pmap_collect() does nothing.

ok mpi@

Convert KVA allocation to km_alloc(9).

ok mpi@

Make sure we always pass a page-aligned address to pmap_grow_map().
Fixes an issue uncovered by the recent change to enlarge kva space where
an unaligned address was passed resulting in memset() writing past the
end of the newly allocated page.

ok miod@

remove uneeded includes in md armv7 files

based on include-what-you-use suggestions

spelling

Convert boolean_t/TRUE/FALSE to int/1/0 for coherency with the rest of
the kernel.

ok patrick@

revert revert revert. there are many other archs that use custom allocs.

make it possible to reduce kmem pressure by letting some pools use a more
accomodating allocator. an interrupt safe pool may also be used in process
context, as indicated by waitok flags. thanks to the garbage collector, we
can always free pages in process context. the only complication is where
to put the pages. solve this by saving the allocation flags in the pool
page header so the free function can examine them.
not actually used in this diff. (coming soon.)
arm testing and compile fixes from phessler

If you use sys/param.h, you don't need sys/types.h

If an Access Flag fault happens while we were running the kernel and
it happened on a kernel page, we need to consult the kernel pmap
instead of the current proc's pmap. Fixes panic when using tmpfs.

ok kettenis@

Change pmap_proc_iflush() to take a process instead of a proc
powerpc: rename second argument of pmap_proc_iflush() to match other archs

ok kettenis@

Remove a couple of unsused static inline functions. Also remove a comparis
of an array to a null pointer that is always false. Found with clang.

ok jsg@

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

In pmap_activate instead of doing disable_interrupts/enable_interrupts
nested inside of splhigh/splx just disable and enable interrupts
once with inline cps instructions.

Remove uneeded pcb_pl1vec block as well, suggested by kettenis.

ok kettenis@

Add support for the PXN bit in level 1 translation table descriptors and
enable it on CPUs that support it. When enabled, this prevents the kernel
from executing userland code.

ok jsg@, tom@

Remove cpu_cpwait() calls; they are no-ops on armv7.

ok tom@

Remove the code that switches around MMU domains on armv7. MMU domains are
basically a relic from the past. Using them doesn't make a lot of sense
the way our pmaps work. Support for MMU domains isn't present in
long-descriptor translation table format, so it is clearly on its way out.

Based on a diff from Artituri Alm.

ok patrick@

Replace pmap_fault_fixup() with an access flag fault handler on armv7.

ok tom@

Fix indentation.

Argh, commit from the wrong tree. Revert previous commit.

Fix indentation.

Correctly enter a mapping as writable if no "page modified" emulation
is needed.

ok visa@, patrick@

Start using to XN flag to enforce that mappings without PROT_EXEC are
non-executable.

ok visa@, deraadt@

Mark device memory as execute-never to prevent a speculative instruction fetch
to access it.

Use Access Flag to do page reference emulation.

ok visa@

Separate out the Access Flag bit from the Access Permission bits in the
armv7 pmap.

ok tom@

Fix typo/inconsistensy where L1_S_DOMAIN was used instead of L1_C_DOMAIN.
These are functionally equivolent so it didn't matter and the resulting
code doesn't change.

The ARMv7 ARM says that the TLB may hold translation table entries at any
level of the translation table, including entries that point to further
levels of the tables. This means that we have to do a TLB flush whenever
we invalidate an L1 slot too. Doing so fixes the pmap_fault_fixup
issue on Cortex-A7 processors.

The page tables are cached now, and given the significant speedup, I
don't think we'll ever go back. So let's ditch the code that tries to
check and patch up incorrect memory attributes.

Also realize that pmap_clean_page(pg, FALSE) doesn't do anything
anymore so remove those calls and drop the 2nd argument from
pmap_clean_page(pg, TRUE) calls.

Last but not least, get rid of pmap_pte_init_generic() here. The only
useful thing it did was setting pmap_copy_page_func() and
pmap_zero_page_func().

This diff should not introduce any change in behaviour.

ok visa@

Mapping non-cachable memory as cachable and subsequently changing the mapping
to non-cachable is retarded. Fix this by introducing PMAP_NOCACHE and
PMAP_DEVICE flags that can be or'ed into the physical address passed to
pmap_kenter(9), like we have on many of our other architectures. This way we
can also properly distinguish between device memory and normal (non-cachable)
memory.

ok visa@

ARMv7 data caches are "effectively" PIPT. This means there is in general
no need to clean and/or invalidate cached pages. So remove most of the
cache cleaning and invalidation from the pmap. We still need to synchronize
the instruction cache with the data cache in various places though. And we
also need to make sure that we clean and invalidate when we make a page
non-cachable.

Tested by Daniel Bolgheroni, mglocker@ and jsg@. on Cortex-A8 and myself on
Cortex-A9.

ok visa@

Put page tables in normal cachable memory on armv7. Check if the MMU walks
the page tables coherently and also skip flushing modified ptes out of the
cache in that case. Speeds up building a kernel with a factor of two on
Cortex-A9 (tested by me) and Cortex-A8 (tested by mglocker@).

ok patrick@

Simplify the way we handle TLB flushes. Since ARMv7 effectively has a
unified TLB there is not much point in optimizing TLB flushing for pages
that have never been executable. The only difference is a flush of the
branch predictor and even that isn't necessary anymore on all but the oldest
Cortex cores.

ok patrick@

Remove devmap stuff which is unused on armv7.

ok patrick@

According to te armv7 ARM TLB entries that caused a Permission fault might
be held in the TLB. On top of that valid page table entries might be
speculatively loaded into the TLB. As a result we need to flush TLB entries
even when the page in question has not been referenced.

Fixes pmap_fault_fixup messages on Cortex-A53, and presumably also on
Cortex-A7.

ok patrick@, guenther@

Only flush the virtual page if it was actually mapped. Otherwise
we will run into translation faults.

ok tom@

When pmap_page_remove() is called by UVM, a physical page is to be
removed from pmaps it currently is in. To check if a virtual address
pointing to that physical page has been mapped, the code uses
the l2pte_valid() function. Unfortunately there is a difference
between being valid and the PTE being zero. If a page is mapped
but has never been accessed, it will be non-zero but invalid.

In that case the PTE for that virtual address will not be zeroed
and the virtual address will be removed from the vm page struct.

The next time someone tries to map a page to that virtual address,
other pmap code will consider the virtual address to be already
mapped, even though that assumption is completely wrong.

To make sure this does not happen, check the PTE for zero. This way
the PTE will be zeroed correctly. The check for zero is how other
ARM pmap code also handles this issue.

ok kettenis@ tom@

Remove what appears to be a copy-paste error setting cur_ttb
in pmap_free_l1().

from aalm@ - thanks

ok patrick@

Don't need a separate flags variable in armv7 pmap_clean_page() -
just use the pv_flags. ('Twas a copy-paste from arm's pmap_clean_page(),
which did need it.)

Also remove even less used flags variable from pmap_page_remove().

First part from a diff from aalm@ - thanks

ok kettenis@ "looks good" patrick@

consistently set ipls on pmap pools.

this is a step toward making ipls unconditionaly on pools.

ok deraadt@ kettenis@

When a physical address is needed to flush the secondary cache use
VM_PAGE_TO_PHYS() instead of unnecessarily calling pmap_extract().

From Patrick Wildt.

Fix the encoding of AP bits for large page second-level
short-descriptors with arm v7 (same as small page encoding, except XN is
in a different bit for the mask).

Expanded version of a diff from Patrick Wildt who also tested and
reviewed this.

Switch from PSR_X_bit and X32_bit PSR macro names to just PSR_X.
This matches FreeBSD and makes things a bit more consistent.
Discussed with Patrick.

Give the pool page allocator backends more sensible names. We now have:
* pool_allocator_single: single page allocator, always interrupt safe
* pool_allocator_multi: multi-page allocator, interrupt safe
* pool_allocator_multi_ni: multi-page allocator, not interrupt-safe

ok deraadt@, dlg@

Remove some pmap locks that were #defined to be nothing (empty). Discussed
with many, ok kettenis@.

remove no-op simple locks
tested by jsg, ok miod

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

use #ifdef DDB for Debugger()

Correct the l1 pte permission bits for armv7. Problem pointed out
by Patrick Wildt who made a similiar change in Bitrig.

ok miod@ rapha@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

Format string fixes and removal of -Wno-format for arm kernels.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

move the arm pmaps away from pool ctors to just initting memory after its
been allocated. also gets rid of a potential bug where pool_get could
return NULL and pmap_alloc_l2_ptp tried to blindly init it.

tests, tweaks, and ok patrick@

The cache mask for ARMv7 is a little bit different from the one inited by
the generic function. While there, also set the cache bits manually.

ok aalm@

Some current boards do not have write-through caching. For those,
we enable write-back, but it seems this is not working and those
boards hang on bootup. Until that is fixed, do not cache PTEs
on those boards.

Rewrite the ARMv7 cache discovery, as some assumptions in the previous one
were utterly wrong. Fix Log2(), correct one taken from the scheduler code.

Tested by rapha@ and Artturi Alm.

typo

We're handling L2 there, so use the corresponding define, not the L1 one.

ok bmercer@

When mapping a new entry, map it read-only, even though it should be
writable. This will cause a pmap fault on first write, so that we can
mark the page as modified. Also mask the bits used for the protection
settings, so that there aren't any leftovers.

ok bmercer@

On ARMv7 we can't use the cache mask to check for coherency.
Therefore we add new macros to be able to check for it properly.

ok miod@

Revert the ARMv7 header split introduced in pmap7.

ok bmercer@

Add secondary cache flushes to armv7's pmap.

ok bmercer@

Further updates to pmap7. Fixes some problems and removes debug printfs.

ok miod@

New pmap for panda boards. Work from lots of folks.
OK miod@

Remove pmap_collect() when a no-op, define __HAVE_PMAP_COLLECT otherwise.
Use that define to shunt uvm_swapout_threads(), which is a noop when
pmap_collect() does nothing.

ok mpi@

Convert KVA allocation to km_alloc(9).

ok mpi@

Make sure we always pass a page-aligned address to pmap_grow_map().
Fixes an issue uncovered by the recent change to enlarge kva space where
an unaligned address was passed resulting in memset() writing past the
end of the newly allocated page.

ok miod@

remove uneeded includes in md armv7 files

based on include-what-you-use suggestions

spelling

Convert boolean_t/TRUE/FALSE to int/1/0 for coherency with the rest of
the kernel.

ok patrick@

revert revert revert. there are many other archs that use custom allocs.

make it possible to reduce kmem pressure by letting some pools use a more
accomodating allocator. an interrupt safe pool may also be used in process
context, as indicated by waitok flags. thanks to the garbage collector, we
can always free pages in process context. the only complication is where
to put the pages. solve this by saving the allocation flags in the pool
page header so the free function can examine them.
not actually used in this diff. (coming soon.)
arm testing and compile fixes from phessler

If you use sys/param.h, you don't need sys/types.h

If an Access Flag fault happens while we were running the kernel and
it happened on a kernel page, we need to consult the kernel pmap
instead of the current proc's pmap. Fixes panic when using tmpfs.

ok kettenis@

Change pmap_proc_iflush() to take a process instead of a proc
powerpc: rename second argument of pmap_proc_iflush() to match other archs

ok kettenis@

Remove a couple of unsused static inline functions. Also remove a comparis
of an array to a null pointer that is always false. Found with clang.

ok jsg@

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

In pmap_activate instead of doing disable_interrupts/enable_interrupts
nested inside of splhigh/splx just disable and enable interrupts
once with inline cps instructions.

Remove uneeded pcb_pl1vec block as well, suggested by kettenis.

ok kettenis@

Add support for the PXN bit in level 1 translation table descriptors and
enable it on CPUs that support it. When enabled, this prevents the kernel
from executing userland code.

ok jsg@, tom@

Remove cpu_cpwait() calls; they are no-ops on armv7.

ok tom@

Remove the code that switches around MMU domains on armv7. MMU domains are
basically a relic from the past. Using them doesn't make a lot of sense
the way our pmaps work. Support for MMU domains isn't present in
long-descriptor translation table format, so it is clearly on its way out.

Based on a diff from Artituri Alm.

ok patrick@

Replace pmap_fault_fixup() with an access flag fault handler on armv7.

ok tom@

Fix indentation.

Argh, commit from the wrong tree. Revert previous commit.

Fix indentation.

Correctly enter a mapping as writable if no "page modified" emulation
is needed.

ok visa@, patrick@

Start using to XN flag to enforce that mappings without PROT_EXEC are
non-executable.

ok visa@, deraadt@

Mark device memory as execute-never to prevent a speculative instruction fetch
to access it.

Use Access Flag to do page reference emulation.

ok visa@

Separate out the Access Flag bit from the Access Permission bits in the
armv7 pmap.

ok tom@

Fix typo/inconsistensy where L1_S_DOMAIN was used instead of L1_C_DOMAIN.
These are functionally equivolent so it didn't matter and the resulting
code doesn't change.

The ARMv7 ARM says that the TLB may hold translation table entries at any
level of the translation table, including entries that point to further
levels of the tables. This means that we have to do a TLB flush whenever
we invalidate an L1 slot too. Doing so fixes the pmap_fault_fixup
issue on Cortex-A7 processors.

The page tables are cached now, and given the significant speedup, I
don't think we'll ever go back. So let's ditch the code that tries to
check and patch up incorrect memory attributes.

Also realize that pmap_clean_page(pg, FALSE) doesn't do anything
anymore so remove those calls and drop the 2nd argument from
pmap_clean_page(pg, TRUE) calls.

Last but not least, get rid of pmap_pte_init_generic() here. The only
useful thing it did was setting pmap_copy_page_func() and
pmap_zero_page_func().

This diff should not introduce any change in behaviour.

ok visa@

Mapping non-cachable memory as cachable and subsequently changing the mapping
to non-cachable is retarded. Fix this by introducing PMAP_NOCACHE and
PMAP_DEVICE flags that can be or'ed into the physical address passed to
pmap_kenter(9), like we have on many of our other architectures. This way we
can also properly distinguish between device memory and normal (non-cachable)
memory.

ok visa@

ARMv7 data caches are "effectively" PIPT. This means there is in general
no need to clean and/or invalidate cached pages. So remove most of the
cache cleaning and invalidation from the pmap. We still need to synchronize
the instruction cache with the data cache in various places though. And we
also need to make sure that we clean and invalidate when we make a page
non-cachable.

Tested by Daniel Bolgheroni, mglocker@ and jsg@. on Cortex-A8 and myself on
Cortex-A9.

ok visa@

Put page tables in normal cachable memory on armv7. Check if the MMU walks
the page tables coherently and also skip flushing modified ptes out of the
cache in that case. Speeds up building a kernel with a factor of two on
Cortex-A9 (tested by me) and Cortex-A8 (tested by mglocker@).

ok patrick@

Simplify the way we handle TLB flushes. Since ARMv7 effectively has a
unified TLB there is not much point in optimizing TLB flushing for pages
that have never been executable. The only difference is a flush of the
branch predictor and even that isn't necessary anymore on all but the oldest
Cortex cores.

ok patrick@

Remove devmap stuff which is unused on armv7.

ok patrick@

According to te armv7 ARM TLB entries that caused a Permission fault might
be held in the TLB. On top of that valid page table entries might be
speculatively loaded into the TLB. As a result we need to flush TLB entries
even when the page in question has not been referenced.

Fixes pmap_fault_fixup messages on Cortex-A53, and presumably also on
Cortex-A7.

ok patrick@, guenther@

Only flush the virtual page if it was actually mapped. Otherwise
we will run into translation faults.

ok tom@

When pmap_page_remove() is called by UVM, a physical page is to be
removed from pmaps it currently is in. To check if a virtual address
pointing to that physical page has been mapped, the code uses
the l2pte_valid() function. Unfortunately there is a difference
between being valid and the PTE being zero. If a page is mapped
but has never been accessed, it will be non-zero but invalid.

In that case the PTE for that virtual address will not be zeroed
and the virtual address will be removed from the vm page struct.

The next time someone tries to map a page to that virtual address,
other pmap code will consider the virtual address to be already
mapped, even though that assumption is completely wrong.

To make sure this does not happen, check the PTE for zero. This way
the PTE will be zeroed correctly. The check for zero is how other
ARM pmap code also handles this issue.

ok kettenis@ tom@

Remove what appears to be a copy-paste error setting cur_ttb
in pmap_free_l1().

from aalm@ - thanks

ok patrick@

Don't need a separate flags variable in armv7 pmap_clean_page() -
just use the pv_flags. ('Twas a copy-paste from arm's pmap_clean_page(),
which did need it.)

Also remove even less used flags variable from pmap_page_remove().

First part from a diff from aalm@ - thanks

ok kettenis@ "looks good" patrick@

consistently set ipls on pmap pools.

this is a step toward making ipls unconditionaly on pools.

ok deraadt@ kettenis@

When a physical address is needed to flush the secondary cache use
VM_PAGE_TO_PHYS() instead of unnecessarily calling pmap_extract().

From Patrick Wildt.

Fix the encoding of AP bits for large page second-level
short-descriptors with arm v7 (same as small page encoding, except XN is
in a different bit for the mask).

Expanded version of a diff from Patrick Wildt who also tested and
reviewed this.

Switch from PSR_X_bit and X32_bit PSR macro names to just PSR_X.
This matches FreeBSD and makes things a bit more consistent.
Discussed with Patrick.

Give the pool page allocator backends more sensible names. We now have:
* pool_allocator_single: single page allocator, always interrupt safe
* pool_allocator_multi: multi-page allocator, interrupt safe
* pool_allocator_multi_ni: multi-page allocator, not interrupt-safe

ok deraadt@, dlg@

Remove some pmap locks that were #defined to be nothing (empty). Discussed
with many, ok kettenis@.

remove no-op simple locks
tested by jsg, ok miod

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

use #ifdef DDB for Debugger()

Correct the l1 pte permission bits for armv7. Problem pointed out
by Patrick Wildt who made a similiar change in Bitrig.

ok miod@ rapha@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

Format string fixes and removal of -Wno-format for arm kernels.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

move the arm pmaps away from pool ctors to just initting memory after its
been allocated. also gets rid of a potential bug where pool_get could
return NULL and pmap_alloc_l2_ptp tried to blindly init it.

tests, tweaks, and ok patrick@

The cache mask for ARMv7 is a little bit different from the one inited by
the generic function. While there, also set the cache bits manually.

ok aalm@

Some current boards do not have write-through caching. For those,
we enable write-back, but it seems this is not working and those
boards hang on bootup. Until that is fixed, do not cache PTEs
on those boards.

Rewrite the ARMv7 cache discovery, as some assumptions in the previous one
were utterly wrong. Fix Log2(), correct one taken from the scheduler code.

Tested by rapha@ and Artturi Alm.

typo

We're handling L2 there, so use the corresponding define, not the L1 one.

ok bmercer@

When mapping a new entry, map it read-only, even though it should be
writable. This will cause a pmap fault on first write, so that we can
mark the page as modified. Also mask the bits used for the protection
settings, so that there aren't any leftovers.

ok bmercer@

On ARMv7 we can't use the cache mask to check for coherency.
Therefore we add new macros to be able to check for it properly.

ok miod@

Revert the ARMv7 header split introduced in pmap7.

ok bmercer@

Add secondary cache flushes to armv7's pmap.

ok bmercer@

Further updates to pmap7. Fixes some problems and removes debug printfs.

ok miod@

New pmap for panda boards. Work from lots of folks.
OK miod@

Convert KVA allocation to km_alloc(9).

ok mpi@

Make sure we always pass a page-aligned address to pmap_grow_map().
Fixes an issue uncovered by the recent change to enlarge kva space where
an unaligned address was passed resulting in memset() writing past the
end of the newly allocated page.

ok miod@

remove uneeded includes in md armv7 files

based on include-what-you-use suggestions

spelling

Convert boolean_t/TRUE/FALSE to int/1/0 for coherency with the rest of
the kernel.

ok patrick@

revert revert revert. there are many other archs that use custom allocs.

make it possible to reduce kmem pressure by letting some pools use a more
accomodating allocator. an interrupt safe pool may also be used in process
context, as indicated by waitok flags. thanks to the garbage collector, we
can always free pages in process context. the only complication is where
to put the pages. solve this by saving the allocation flags in the pool
page header so the free function can examine them.
not actually used in this diff. (coming soon.)
arm testing and compile fixes from phessler

If you use sys/param.h, you don't need sys/types.h

If an Access Flag fault happens while we were running the kernel and
it happened on a kernel page, we need to consult the kernel pmap
instead of the current proc's pmap. Fixes panic when using tmpfs.

ok kettenis@

Change pmap_proc_iflush() to take a process instead of a proc
powerpc: rename second argument of pmap_proc_iflush() to match other archs

ok kettenis@

Remove a couple of unsused static inline functions. Also remove a comparis
of an array to a null pointer that is always false. Found with clang.

ok jsg@

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

In pmap_activate instead of doing disable_interrupts/enable_interrupts
nested inside of splhigh/splx just disable and enable interrupts
once with inline cps instructions.

Remove uneeded pcb_pl1vec block as well, suggested by kettenis.

ok kettenis@

Add support for the PXN bit in level 1 translation table descriptors and
enable it on CPUs that support it. When enabled, this prevents the kernel
from executing userland code.

ok jsg@, tom@

Remove cpu_cpwait() calls; they are no-ops on armv7.

ok tom@

Remove the code that switches around MMU domains on armv7. MMU domains are
basically a relic from the past. Using them doesn't make a lot of sense
the way our pmaps work. Support for MMU domains isn't present in
long-descriptor translation table format, so it is clearly on its way out.

Based on a diff from Artituri Alm.

ok patrick@

Replace pmap_fault_fixup() with an access flag fault handler on armv7.

ok tom@

Fix indentation.

Argh, commit from the wrong tree. Revert previous commit.

Fix indentation.

Correctly enter a mapping as writable if no "page modified" emulation
is needed.

ok visa@, patrick@

Start using to XN flag to enforce that mappings without PROT_EXEC are
non-executable.

ok visa@, deraadt@

Mark device memory as execute-never to prevent a speculative instruction fetch
to access it.

Use Access Flag to do page reference emulation.

ok visa@

Separate out the Access Flag bit from the Access Permission bits in the
armv7 pmap.

ok tom@

Fix typo/inconsistensy where L1_S_DOMAIN was used instead of L1_C_DOMAIN.
These are functionally equivolent so it didn't matter and the resulting
code doesn't change.

The ARMv7 ARM says that the TLB may hold translation table entries at any
level of the translation table, including entries that point to further
levels of the tables. This means that we have to do a TLB flush whenever
we invalidate an L1 slot too. Doing so fixes the pmap_fault_fixup
issue on Cortex-A7 processors.

The page tables are cached now, and given the significant speedup, I
don't think we'll ever go back. So let's ditch the code that tries to
check and patch up incorrect memory attributes.

Also realize that pmap_clean_page(pg, FALSE) doesn't do anything
anymore so remove those calls and drop the 2nd argument from
pmap_clean_page(pg, TRUE) calls.

Last but not least, get rid of pmap_pte_init_generic() here. The only
useful thing it did was setting pmap_copy_page_func() and
pmap_zero_page_func().

This diff should not introduce any change in behaviour.

ok visa@

Mapping non-cachable memory as cachable and subsequently changing the mapping
to non-cachable is retarded. Fix this by introducing PMAP_NOCACHE and
PMAP_DEVICE flags that can be or'ed into the physical address passed to
pmap_kenter(9), like we have on many of our other architectures. This way we
can also properly distinguish between device memory and normal (non-cachable)
memory.

ok visa@

ARMv7 data caches are "effectively" PIPT. This means there is in general
no need to clean and/or invalidate cached pages. So remove most of the
cache cleaning and invalidation from the pmap. We still need to synchronize
the instruction cache with the data cache in various places though. And we
also need to make sure that we clean and invalidate when we make a page
non-cachable.

Tested by Daniel Bolgheroni, mglocker@ and jsg@. on Cortex-A8 and myself on
Cortex-A9.

ok visa@

Put page tables in normal cachable memory on armv7. Check if the MMU walks
the page tables coherently and also skip flushing modified ptes out of the
cache in that case. Speeds up building a kernel with a factor of two on
Cortex-A9 (tested by me) and Cortex-A8 (tested by mglocker@).

ok patrick@

Simplify the way we handle TLB flushes. Since ARMv7 effectively has a
unified TLB there is not much point in optimizing TLB flushing for pages
that have never been executable. The only difference is a flush of the
branch predictor and even that isn't necessary anymore on all but the oldest
Cortex cores.

ok patrick@

Remove devmap stuff which is unused on armv7.

ok patrick@

According to te armv7 ARM TLB entries that caused a Permission fault might
be held in the TLB. On top of that valid page table entries might be
speculatively loaded into the TLB. As a result we need to flush TLB entries
even when the page in question has not been referenced.

Fixes pmap_fault_fixup messages on Cortex-A53, and presumably also on
Cortex-A7.

ok patrick@, guenther@

Only flush the virtual page if it was actually mapped. Otherwise
we will run into translation faults.

ok tom@

When pmap_page_remove() is called by UVM, a physical page is to be
removed from pmaps it currently is in. To check if a virtual address
pointing to that physical page has been mapped, the code uses
the l2pte_valid() function. Unfortunately there is a difference
between being valid and the PTE being zero. If a page is mapped
but has never been accessed, it will be non-zero but invalid.

In that case the PTE for that virtual address will not be zeroed
and the virtual address will be removed from the vm page struct.

The next time someone tries to map a page to that virtual address,
other pmap code will consider the virtual address to be already
mapped, even though that assumption is completely wrong.

To make sure this does not happen, check the PTE for zero. This way
the PTE will be zeroed correctly. The check for zero is how other
ARM pmap code also handles this issue.

ok kettenis@ tom@

Remove what appears to be a copy-paste error setting cur_ttb
in pmap_free_l1().

from aalm@ - thanks

ok patrick@

Don't need a separate flags variable in armv7 pmap_clean_page() -
just use the pv_flags. ('Twas a copy-paste from arm's pmap_clean_page(),
which did need it.)

Also remove even less used flags variable from pmap_page_remove().

First part from a diff from aalm@ - thanks

ok kettenis@ "looks good" patrick@

consistently set ipls on pmap pools.

this is a step toward making ipls unconditionaly on pools.

ok deraadt@ kettenis@

When a physical address is needed to flush the secondary cache use
VM_PAGE_TO_PHYS() instead of unnecessarily calling pmap_extract().

From Patrick Wildt.

Fix the encoding of AP bits for large page second-level
short-descriptors with arm v7 (same as small page encoding, except XN is
in a different bit for the mask).

Expanded version of a diff from Patrick Wildt who also tested and
reviewed this.

Switch from PSR_X_bit and X32_bit PSR macro names to just PSR_X.
This matches FreeBSD and makes things a bit more consistent.
Discussed with Patrick.

Give the pool page allocator backends more sensible names. We now have:
* pool_allocator_single: single page allocator, always interrupt safe
* pool_allocator_multi: multi-page allocator, interrupt safe
* pool_allocator_multi_ni: multi-page allocator, not interrupt-safe

ok deraadt@, dlg@

Remove some pmap locks that were #defined to be nothing (empty). Discussed
with many, ok kettenis@.

remove no-op simple locks
tested by jsg, ok miod

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

use #ifdef DDB for Debugger()

Correct the l1 pte permission bits for armv7. Problem pointed out
by Patrick Wildt who made a similiar change in Bitrig.

ok miod@ rapha@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

Format string fixes and removal of -Wno-format for arm kernels.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

move the arm pmaps away from pool ctors to just initting memory after its
been allocated. also gets rid of a potential bug where pool_get could
return NULL and pmap_alloc_l2_ptp tried to blindly init it.

tests, tweaks, and ok patrick@

The cache mask for ARMv7 is a little bit different from the one inited by
the generic function. While there, also set the cache bits manually.

ok aalm@

Some current boards do not have write-through caching. For those,
we enable write-back, but it seems this is not working and those
boards hang on bootup. Until that is fixed, do not cache PTEs
on those boards.

Rewrite the ARMv7 cache discovery, as some assumptions in the previous one
were utterly wrong. Fix Log2(), correct one taken from the scheduler code.

Tested by rapha@ and Artturi Alm.

typo

We're handling L2 there, so use the corresponding define, not the L1 one.

ok bmercer@

When mapping a new entry, map it read-only, even though it should be
writable. This will cause a pmap fault on first write, so that we can
mark the page as modified. Also mask the bits used for the protection
settings, so that there aren't any leftovers.

ok bmercer@

On ARMv7 we can't use the cache mask to check for coherency.
Therefore we add new macros to be able to check for it properly.

ok miod@

Revert the ARMv7 header split introduced in pmap7.

ok bmercer@

Add secondary cache flushes to armv7's pmap.

ok bmercer@

Further updates to pmap7. Fixes some problems and removes debug printfs.

ok miod@

New pmap for panda boards. Work from lots of folks.
OK miod@

Make sure we always pass a page-aligned address to pmap_grow_map().
Fixes an issue uncovered by the recent change to enlarge kva space where
an unaligned address was passed resulting in memset() writing past the
end of the newly allocated page.

ok miod@

remove uneeded includes in md armv7 files

based on include-what-you-use suggestions

spelling

Convert boolean_t/TRUE/FALSE to int/1/0 for coherency with the rest of
the kernel.

ok patrick@

revert revert revert. there are many other archs that use custom allocs.

make it possible to reduce kmem pressure by letting some pools use a more
accomodating allocator. an interrupt safe pool may also be used in process
context, as indicated by waitok flags. thanks to the garbage collector, we
can always free pages in process context. the only complication is where
to put the pages. solve this by saving the allocation flags in the pool
page header so the free function can examine them.
not actually used in this diff. (coming soon.)
arm testing and compile fixes from phessler

If you use sys/param.h, you don't need sys/types.h

If an Access Flag fault happens while we were running the kernel and
it happened on a kernel page, we need to consult the kernel pmap
instead of the current proc's pmap. Fixes panic when using tmpfs.

ok kettenis@

Change pmap_proc_iflush() to take a process instead of a proc
powerpc: rename second argument of pmap_proc_iflush() to match other archs

ok kettenis@

Remove a couple of unsused static inline functions. Also remove a comparis
of an array to a null pointer that is always false. Found with clang.

ok jsg@

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

In pmap_activate instead of doing disable_interrupts/enable_interrupts
nested inside of splhigh/splx just disable and enable interrupts
once with inline cps instructions.

Remove uneeded pcb_pl1vec block as well, suggested by kettenis.

ok kettenis@

Add support for the PXN bit in level 1 translation table descriptors and
enable it on CPUs that support it. When enabled, this prevents the kernel
from executing userland code.

ok jsg@, tom@

Remove cpu_cpwait() calls; they are no-ops on armv7.

ok tom@

Remove the code that switches around MMU domains on armv7. MMU domains are
basically a relic from the past. Using them doesn't make a lot of sense
the way our pmaps work. Support for MMU domains isn't present in
long-descriptor translation table format, so it is clearly on its way out.

Based on a diff from Artituri Alm.

ok patrick@

Replace pmap_fault_fixup() with an access flag fault handler on armv7.

ok tom@

Fix indentation.

Argh, commit from the wrong tree. Revert previous commit.

Fix indentation.

Correctly enter a mapping as writable if no "page modified" emulation
is needed.

ok visa@, patrick@

Start using to XN flag to enforce that mappings without PROT_EXEC are
non-executable.

ok visa@, deraadt@

Mark device memory as execute-never to prevent a speculative instruction fetch
to access it.

Use Access Flag to do page reference emulation.

ok visa@

Separate out the Access Flag bit from the Access Permission bits in the
armv7 pmap.

ok tom@

Fix typo/inconsistensy where L1_S_DOMAIN was used instead of L1_C_DOMAIN.
These are functionally equivolent so it didn't matter and the resulting
code doesn't change.

The ARMv7 ARM says that the TLB may hold translation table entries at any
level of the translation table, including entries that point to further
levels of the tables. This means that we have to do a TLB flush whenever
we invalidate an L1 slot too. Doing so fixes the pmap_fault_fixup
issue on Cortex-A7 processors.

The page tables are cached now, and given the significant speedup, I
don't think we'll ever go back. So let's ditch the code that tries to
check and patch up incorrect memory attributes.

Also realize that pmap_clean_page(pg, FALSE) doesn't do anything
anymore so remove those calls and drop the 2nd argument from
pmap_clean_page(pg, TRUE) calls.

Last but not least, get rid of pmap_pte_init_generic() here. The only
useful thing it did was setting pmap_copy_page_func() and
pmap_zero_page_func().

This diff should not introduce any change in behaviour.

ok visa@

Mapping non-cachable memory as cachable and subsequently changing the mapping
to non-cachable is retarded. Fix this by introducing PMAP_NOCACHE and
PMAP_DEVICE flags that can be or'ed into the physical address passed to
pmap_kenter(9), like we have on many of our other architectures. This way we
can also properly distinguish between device memory and normal (non-cachable)
memory.

ok visa@

ARMv7 data caches are "effectively" PIPT. This means there is in general
no need to clean and/or invalidate cached pages. So remove most of the
cache cleaning and invalidation from the pmap. We still need to synchronize
the instruction cache with the data cache in various places though. And we
also need to make sure that we clean and invalidate when we make a page
non-cachable.

Tested by Daniel Bolgheroni, mglocker@ and jsg@. on Cortex-A8 and myself on
Cortex-A9.

ok visa@

Put page tables in normal cachable memory on armv7. Check if the MMU walks
the page tables coherently and also skip flushing modified ptes out of the
cache in that case. Speeds up building a kernel with a factor of two on
Cortex-A9 (tested by me) and Cortex-A8 (tested by mglocker@).

ok patrick@

Simplify the way we handle TLB flushes. Since ARMv7 effectively has a
unified TLB there is not much point in optimizing TLB flushing for pages
that have never been executable. The only difference is a flush of the
branch predictor and even that isn't necessary anymore on all but the oldest
Cortex cores.

ok patrick@

Remove devmap stuff which is unused on armv7.

ok patrick@

According to te armv7 ARM TLB entries that caused a Permission fault might
be held in the TLB. On top of that valid page table entries might be
speculatively loaded into the TLB. As a result we need to flush TLB entries
even when the page in question has not been referenced.

Fixes pmap_fault_fixup messages on Cortex-A53, and presumably also on
Cortex-A7.

ok patrick@, guenther@

Only flush the virtual page if it was actually mapped. Otherwise
we will run into translation faults.

ok tom@

When pmap_page_remove() is called by UVM, a physical page is to be
removed from pmaps it currently is in. To check if a virtual address
pointing to that physical page has been mapped, the code uses
the l2pte_valid() function. Unfortunately there is a difference
between being valid and the PTE being zero. If a page is mapped
but has never been accessed, it will be non-zero but invalid.

In that case the PTE for that virtual address will not be zeroed
and the virtual address will be removed from the vm page struct.

The next time someone tries to map a page to that virtual address,
other pmap code will consider the virtual address to be already
mapped, even though that assumption is completely wrong.

To make sure this does not happen, check the PTE for zero. This way
the PTE will be zeroed correctly. The check for zero is how other
ARM pmap code also handles this issue.

ok kettenis@ tom@

Remove what appears to be a copy-paste error setting cur_ttb
in pmap_free_l1().

from aalm@ - thanks

ok patrick@

Don't need a separate flags variable in armv7 pmap_clean_page() -
just use the pv_flags. ('Twas a copy-paste from arm's pmap_clean_page(),
which did need it.)

Also remove even less used flags variable from pmap_page_remove().

First part from a diff from aalm@ - thanks

ok kettenis@ "looks good" patrick@

consistently set ipls on pmap pools.

this is a step toward making ipls unconditionaly on pools.

ok deraadt@ kettenis@

When a physical address is needed to flush the secondary cache use
VM_PAGE_TO_PHYS() instead of unnecessarily calling pmap_extract().

From Patrick Wildt.

Fix the encoding of AP bits for large page second-level
short-descriptors with arm v7 (same as small page encoding, except XN is
in a different bit for the mask).

Expanded version of a diff from Patrick Wildt who also tested and
reviewed this.

Switch from PSR_X_bit and X32_bit PSR macro names to just PSR_X.
This matches FreeBSD and makes things a bit more consistent.
Discussed with Patrick.

Give the pool page allocator backends more sensible names. We now have:
* pool_allocator_single: single page allocator, always interrupt safe
* pool_allocator_multi: multi-page allocator, interrupt safe
* pool_allocator_multi_ni: multi-page allocator, not interrupt-safe

ok deraadt@, dlg@

Remove some pmap locks that were #defined to be nothing (empty). Discussed
with many, ok kettenis@.

remove no-op simple locks
tested by jsg, ok miod

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

use #ifdef DDB for Debugger()

Correct the l1 pte permission bits for armv7. Problem pointed out
by Patrick Wildt who made a similiar change in Bitrig.

ok miod@ rapha@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

Format string fixes and removal of -Wno-format for arm kernels.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

move the arm pmaps away from pool ctors to just initting memory after its
been allocated. also gets rid of a potential bug where pool_get could
return NULL and pmap_alloc_l2_ptp tried to blindly init it.

tests, tweaks, and ok patrick@

The cache mask for ARMv7 is a little bit different from the one inited by
the generic function. While there, also set the cache bits manually.

ok aalm@

Some current boards do not have write-through caching. For those,
we enable write-back, but it seems this is not working and those
boards hang on bootup. Until that is fixed, do not cache PTEs
on those boards.

Rewrite the ARMv7 cache discovery, as some assumptions in the previous one
were utterly wrong. Fix Log2(), correct one taken from the scheduler code.

Tested by rapha@ and Artturi Alm.

typo

We're handling L2 there, so use the corresponding define, not the L1 one.

ok bmercer@

When mapping a new entry, map it read-only, even though it should be
writable. This will cause a pmap fault on first write, so that we can
mark the page as modified. Also mask the bits used for the protection
settings, so that there aren't any leftovers.

ok bmercer@

On ARMv7 we can't use the cache mask to check for coherency.
Therefore we add new macros to be able to check for it properly.

ok miod@

Revert the ARMv7 header split introduced in pmap7.

ok bmercer@

Add secondary cache flushes to armv7's pmap.

ok bmercer@

Further updates to pmap7. Fixes some problems and removes debug printfs.

ok miod@

New pmap for panda boards. Work from lots of folks.
OK miod@

remove uneeded includes in md armv7 files

based on include-what-you-use suggestions

spelling

Convert boolean_t/TRUE/FALSE to int/1/0 for coherency with the rest of
the kernel.

ok patrick@

revert revert revert. there are many other archs that use custom allocs.

make it possible to reduce kmem pressure by letting some pools use a more
accomodating allocator. an interrupt safe pool may also be used in process
context, as indicated by waitok flags. thanks to the garbage collector, we
can always free pages in process context. the only complication is where
to put the pages. solve this by saving the allocation flags in the pool
page header so the free function can examine them.
not actually used in this diff. (coming soon.)
arm testing and compile fixes from phessler

If you use sys/param.h, you don't need sys/types.h

If an Access Flag fault happens while we were running the kernel and
it happened on a kernel page, we need to consult the kernel pmap
instead of the current proc's pmap. Fixes panic when using tmpfs.

ok kettenis@

Change pmap_proc_iflush() to take a process instead of a proc
powerpc: rename second argument of pmap_proc_iflush() to match other archs

ok kettenis@

Remove a couple of unsused static inline functions. Also remove a comparis
of an array to a null pointer that is always false. Found with clang.

ok jsg@

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

In pmap_activate instead of doing disable_interrupts/enable_interrupts
nested inside of splhigh/splx just disable and enable interrupts
once with inline cps instructions.

Remove uneeded pcb_pl1vec block as well, suggested by kettenis.

ok kettenis@

Add support for the PXN bit in level 1 translation table descriptors and
enable it on CPUs that support it. When enabled, this prevents the kernel
from executing userland code.

ok jsg@, tom@

Remove cpu_cpwait() calls; they are no-ops on armv7.

ok tom@

Remove the code that switches around MMU domains on armv7. MMU domains are
basically a relic from the past. Using them doesn't make a lot of sense
the way our pmaps work. Support for MMU domains isn't present in
long-descriptor translation table format, so it is clearly on its way out.

Based on a diff from Artituri Alm.

ok patrick@

Replace pmap_fault_fixup() with an access flag fault handler on armv7.

ok tom@

Fix indentation.

Argh, commit from the wrong tree. Revert previous commit.

Fix indentation.

Correctly enter a mapping as writable if no "page modified" emulation
is needed.

ok visa@, patrick@

Start using to XN flag to enforce that mappings without PROT_EXEC are
non-executable.

ok visa@, deraadt@

Mark device memory as execute-never to prevent a speculative instruction fetch
to access it.

Use Access Flag to do page reference emulation.

ok visa@

Separate out the Access Flag bit from the Access Permission bits in the
armv7 pmap.

ok tom@

Fix typo/inconsistensy where L1_S_DOMAIN was used instead of L1_C_DOMAIN.
These are functionally equivolent so it didn't matter and the resulting
code doesn't change.

The ARMv7 ARM says that the TLB may hold translation table entries at any
level of the translation table, including entries that point to further
levels of the tables. This means that we have to do a TLB flush whenever
we invalidate an L1 slot too. Doing so fixes the pmap_fault_fixup
issue on Cortex-A7 processors.

The page tables are cached now, and given the significant speedup, I
don't think we'll ever go back. So let's ditch the code that tries to
check and patch up incorrect memory attributes.

Also realize that pmap_clean_page(pg, FALSE) doesn't do anything
anymore so remove those calls and drop the 2nd argument from
pmap_clean_page(pg, TRUE) calls.

Last but not least, get rid of pmap_pte_init_generic() here. The only
useful thing it did was setting pmap_copy_page_func() and
pmap_zero_page_func().

This diff should not introduce any change in behaviour.

ok visa@

Mapping non-cachable memory as cachable and subsequently changing the mapping
to non-cachable is retarded. Fix this by introducing PMAP_NOCACHE and
PMAP_DEVICE flags that can be or'ed into the physical address passed to
pmap_kenter(9), like we have on many of our other architectures. This way we
can also properly distinguish between device memory and normal (non-cachable)
memory.

ok visa@

ARMv7 data caches are "effectively" PIPT. This means there is in general
no need to clean and/or invalidate cached pages. So remove most of the
cache cleaning and invalidation from the pmap. We still need to synchronize
the instruction cache with the data cache in various places though. And we
also need to make sure that we clean and invalidate when we make a page
non-cachable.

Tested by Daniel Bolgheroni, mglocker@ and jsg@. on Cortex-A8 and myself on
Cortex-A9.

ok visa@

Put page tables in normal cachable memory on armv7. Check if the MMU walks
the page tables coherently and also skip flushing modified ptes out of the
cache in that case. Speeds up building a kernel with a factor of two on
Cortex-A9 (tested by me) and Cortex-A8 (tested by mglocker@).

ok patrick@

Simplify the way we handle TLB flushes. Since ARMv7 effectively has a
unified TLB there is not much point in optimizing TLB flushing for pages
that have never been executable. The only difference is a flush of the
branch predictor and even that isn't necessary anymore on all but the oldest
Cortex cores.

ok patrick@

Remove devmap stuff which is unused on armv7.

ok patrick@

According to te armv7 ARM TLB entries that caused a Permission fault might
be held in the TLB. On top of that valid page table entries might be
speculatively loaded into the TLB. As a result we need to flush TLB entries
even when the page in question has not been referenced.

Fixes pmap_fault_fixup messages on Cortex-A53, and presumably also on
Cortex-A7.

ok patrick@, guenther@

Only flush the virtual page if it was actually mapped. Otherwise
we will run into translation faults.

ok tom@

When pmap_page_remove() is called by UVM, a physical page is to be
removed from pmaps it currently is in. To check if a virtual address
pointing to that physical page has been mapped, the code uses
the l2pte_valid() function. Unfortunately there is a difference
between being valid and the PTE being zero. If a page is mapped
but has never been accessed, it will be non-zero but invalid.

In that case the PTE for that virtual address will not be zeroed
and the virtual address will be removed from the vm page struct.

The next time someone tries to map a page to that virtual address,
other pmap code will consider the virtual address to be already
mapped, even though that assumption is completely wrong.

To make sure this does not happen, check the PTE for zero. This way
the PTE will be zeroed correctly. The check for zero is how other
ARM pmap code also handles this issue.

ok kettenis@ tom@

Remove what appears to be a copy-paste error setting cur_ttb
in pmap_free_l1().

from aalm@ - thanks

ok patrick@

Don't need a separate flags variable in armv7 pmap_clean_page() -
just use the pv_flags. ('Twas a copy-paste from arm's pmap_clean_page(),
which did need it.)

Also remove even less used flags variable from pmap_page_remove().

First part from a diff from aalm@ - thanks

ok kettenis@ "looks good" patrick@

consistently set ipls on pmap pools.

this is a step toward making ipls unconditionaly on pools.

ok deraadt@ kettenis@

When a physical address is needed to flush the secondary cache use
VM_PAGE_TO_PHYS() instead of unnecessarily calling pmap_extract().

From Patrick Wildt.

Fix the encoding of AP bits for large page second-level
short-descriptors with arm v7 (same as small page encoding, except XN is
in a different bit for the mask).

Expanded version of a diff from Patrick Wildt who also tested and
reviewed this.

Switch from PSR_X_bit and X32_bit PSR macro names to just PSR_X.
This matches FreeBSD and makes things a bit more consistent.
Discussed with Patrick.

Give the pool page allocator backends more sensible names. We now have:
* pool_allocator_single: single page allocator, always interrupt safe
* pool_allocator_multi: multi-page allocator, interrupt safe
* pool_allocator_multi_ni: multi-page allocator, not interrupt-safe

ok deraadt@, dlg@

Remove some pmap locks that were #defined to be nothing (empty). Discussed
with many, ok kettenis@.

remove no-op simple locks
tested by jsg, ok miod

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

use #ifdef DDB for Debugger()

Correct the l1 pte permission bits for armv7. Problem pointed out
by Patrick Wildt who made a similiar change in Bitrig.

ok miod@ rapha@

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

Format string fixes and removal of -Wno-format for arm kernels.

It's been a quarter century: we can assume volatile is present with that name.

ok dlg@ mpi@ deraadt@

move the arm pmaps away from pool ctors to just initting memory after its
been allocated. also gets rid of a potential bug where pool_get could
return NULL and pmap_alloc_l2_ptp tried to blindly init it.

tests, tweaks, and ok patrick@

The cache mask for ARMv7 is a little bit different from the one inited by
the generic function. While there, also set the cache bits manually.

ok aalm@

Some current boards do not have write-through caching. For those,
we enable write-back, but it seems this is not working and those
boards hang on bootup. Until that is fixed, do not cache PTEs
on those boards.

Rewrite the ARMv7 cache discovery, as some assumptions in the previous one
were utterly wrong. Fix Log2(), correct one taken from the scheduler code.

Tested by rapha@ and Artturi Alm.

typo

We're handling L2 there, so use the corresponding define, not the L1 one.

ok bmercer@

When mapping a new entry, map it read-only, even though it should be
writable. This will cause a pmap fault on first write, so that we can
mark the page as modified. Also mask the bits used for the protection
settings, so that there aren't any leftovers.

ok bmercer@

On ARMv7 we can't use the cache mask to check for coherency.
Therefore we add new macros to be able to check for it properly.

ok miod@

Revert the ARMv7 header split introduced in pmap7.

ok bmercer@

Add secondary cache flushes to armv7's pmap.

ok bmercer@

Further updates to pmap7. Fixes some problems and removes debug printfs.

ok miod@

New pmap for panda boards. Work from lots of folks.
OK miod@

spelling

Convert boolean_t/TRUE/FALSE to int/1/0 for coherency with the rest of
the kernel.

ok patrick@

revert revert revert. there are many other archs that use custom allocs.

make it possible to reduce kmem pressure by letting some pools use a more
accomodating allocator. an interrupt safe pool may also be used in process
context, as indicated by waitok flags. thanks to the garbage collector, we
can always free pages in process context. the only complication is where
to put the pages. solve this by saving the allocation flags in the pool
page header so the free function can examine them.
not actually used in this diff. (coming soon.)
arm testing and compile fixes from phessler

If you use sys/param.h, you don't need sys/types.h

If an Access Flag fault happens while we were running the kernel and
it happened on a kernel page, we need to consult the kernel pmap
instead of the current proc's pmap. Fixes panic when using tmpfs.

ok kettenis@

Change pmap_proc_iflush() to take a process instead of a proc
powerpc: rename second argument of pmap_proc_iflush() to match other archs

ok kettenis@

Remove a couple of unsused static inline functions. Also remove a comparis
of an array to a null pointer that is always false. Found with clang.

ok jsg@

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

In pmap_activate instead of doing disable_interrupts/enable_interrupts
nested inside of splhigh/splx just disable and enable interrupts
once with inline cps instructions.

Remove uneeded pcb_pl1vec block as well, suggested by kettenis.

ok kettenis@

Add support for the PXN bit in level 1 translation table descriptors and
enable it on CPUs that support it. When enabled, this prevents the kernel
from executing userland code.

ok jsg@, tom@

Remove cpu_cpwait() calls; they are no-ops on armv7.

ok tom@

Remove the code that switches around MMU domains on armv7. MMU domains are
basically a relic from the past. Using them doesn't make a lot of sense
the way our pmaps work. Support for MMU domains isn't present in
long-descriptor translation table format, so it is clearly on its way out.

Based on a diff from Artituri Alm.

ok patrick@

Replace pmap_fault_fixup() with an access flag fault handler on armv7.

ok tom@

Fix indentation.

Argh, commit from the wrong tree. Revert previous commit.

Fix indentation.

Correctly enter a mapping as writable if no "page modified" emulation
is needed.

ok visa@, patrick@

Start using to XN flag to enforce that mappings without PROT_EXEC are
non-executable.

ok visa@, deraadt@

Mark device memory as execute-never to prevent a speculative instruction fetch
to access it.

Use Access Flag to do page reference emulation.

ok visa@

Separate out the Access Flag bit from the Access Permission bits in the
armv7 pmap.

ok tom@

Fix typo/inconsistensy where L1_S_DOMAIN was used instead of L1_C_DOMAIN.
These are functionally equivolent so it didn't matter and the resulting
code doesn't change.

The ARMv7 ARM says that the TLB may hold translation table entries at any
level of the translation table, including entries that point to further
levels of the tables. This means that we have to do a TLB flush whenever
we invalidate an L1 slot too. Doing so fixes the pmap_fault_fixup
issue on Cortex-A7 processors.

The page tables are cached now, and given the significant speedup, I
don't think we'll ever go back. So let's ditch the code that tries to
check and patch up incorrect memory attributes.

Also realize that pmap_clean_page(pg, FALSE) doesn't do anything
anymore so remove those calls and drop the 2nd argument from
pmap_clean_page(pg, TRUE) calls.

Last but not least, get rid of pmap_pte_init_generic() here. The only
useful thing it did was setting pmap_copy_page_func() and
pmap_zero_page_func().

This diff should not introduce any change in behaviour.

ok visa@

Mapping non-cachable memory as cachable and subsequently changing the mapping
to non-cachable is retarded. Fix this by introducing PMAP_NOCACHE and
PMAP_DEVICE flags that can be or'ed into the physical address passed to
pmap_kenter(9), like we have on many of our other architectures. This way we
can also properly distinguish between device memory and normal (non-cachable)
memory.

ok visa@