Replace obsolete pv_dump() call with pmap_db_mdpg_print().

It was rewritten on rev 1.107, but not replaced with new implementation in
PMAP_PV_DEBUG guarded block.

Remove ryo@'s mail addresses.

Provide a shared pmap_devmap implementation and convert all pmap_devmap
arrays to use DEVMAP_ENTRY{,_END}

Rename VM_KERNEL_IO_ADDRESS to VM_KERNEL_IO_BASE to match RISC-V

It's less letters, matches other similar variables and will help with
sharing code between the two architectures.

NFCI.

aarch64/pmap: Fix criterion in previous.

Use the pte bit that says whether this is a PMAP_WIRED page, not the
bit that says whether this is a non-global page.

(Forgot to git commit --amend before exporting to CVS, sorry!)

aarch64/pmap(9): Teach pmap_protect about pmap_kenter_pa mappings.

Pages mapped with pmap_kenter_pa are necessarily unmanaged, so there
are no P->V records, and pmap_kenter_pa leaves pp->pp_pv.pv_va zero
with no modified/referenced state.

However, pmap_protect erroneously examined pp->pp_pv.pv_va to
ascertain the modified/referenced state -- and if the page was not
marked referenced, pmap_protect would clear the LX_BLKPAG_AF bit
(Access Flag), with the effect that subsequent uses of the page fault
and require a detour through pmap_fault_fixup.

This caused problems for the kernel module loader:

- When loading the text section, kobj_load first allocates kva with
uvm_km_alloc(UVM_KMF_WIRED|UVM_KMF_EXEC), which creates ptes with
pmap_kenter_pa. These ptes are writable, so we can copy the text
section into them, and have LX_BLKPAG_AF set so there will be no
fault when they are used by the kernel.

- But then kobj_affix makes the text section read/execute-only (and
nonwritable) with uvm_km_protect(VM_PROT_READ|VM_PROT_EXECUTE),
which updates the ptes with pmap_protect. This _should_ leave
LX_BLKPAG_AF set, but by inadvertently treating the page as managed
when it should be unmanaged, pmap_protect cleared it instead.

- Most of the time, clearing LX_BLKPAG_AF caused no problem, because
pmap_fault_fixup would silently resolve it. But if a hard
interrupt handler tried to use any page in the module's text (or
rodata, I suspect) that was not yet fixed up, the CPU would fault
and enter pmap_fault_fixup -- which would promptly crash (or hang)
by trying to take the pmap lock in interrupt context, which is
forbidden.

I observed this by loading dtrace.kmod early at boot and trying to
dtrace hard interrupt handlers.

With this change, pmap_protect now recognizes wired mappings (as
created by pmap_kenter_pa) before consulting pp->pp_pv.pv_va, and
preserves then LX_BLKPAG_AF bit in that case.

ok skrll

fix a spello in a comment

Remove some empty lines

Use UVMHIST_CALLARGS in pmap_bootstrap

Only define the EFI variable if EFI_RUNTIME

KNF

Use "non-posted" instead of "strongly ordered" to describe nGnRnE mappings

Rename the following defines:
- _ARM_BUS_SPACE_MAP_STRONGLY_ORDERED to BUS_SPACE_MAP_NONPOSTED
- PMAP_DEV_SO to PMAP_DEV_NP
- LX_BLKPAG_ATTR_DEVICE_MEM_SO to LX_BLKPAG_ATTR_DEVICE_MEM_NP
Rename the following option:
- AARCH64_DEVICE_MEM_STRONGLY_ORDERED to AARCH64_DEVICE_MEM_NONPOSTED

Fixed a bug that pte's __BIT(63,48) could be set when accessing addresses above 0x0001000000000000 in /dev/mem with mmap().

When accessed in mmap by the device pager, pmap_enter() may be called with prot == PROT_WRITE.

Sprinkle some KASSERT(kpreempt_disabled());

since pmap_activate_efirt() rewrites TTBR0, it is necessary to pmap_activate() again after pmap_deactivate_efirt() to restore the original TTBR0.

- Fix to do pmap_{de,}activate() before/after pmap_{,de}activate_efirt().
- moved kpreempt_{disable,enable}() to the caller since everything between
arm_efirt_md_enter() and arm_efirt_md_exit() should be kpreempt disabled.

ok skrll@

Add the missing kpreempt_enable to pmap_deactivate_efirt

No need to flush icache for EFI RT mappings as bootaa64.efi flushed
the full icache for us. (Also this avoids traps)

sys: Use membar_release/acquire around reference drop.

This just goes through my recent reference count membar audit and
changes membar_exit to membar_release and membar_enter to
membar_acquire -- this should make everything cheaper on most CPUs
without hurting correctness, because membar_acquire is generally
cheaper than membar_enter.

Update to support EFI runtime outside the kernel virtual address space
by creating an EFI RT pmap that can be activated / deactivated when
required.

Adds support for EFI RT to ARM_MMU_EXTENDED (ASID) 32-bit Arm machines.

On Arm64 the usage of pmapboot_enter is reduced and the mappings are
created much later in the boot process -- now in cpu_startup_hook.
Backward compatiblity for KVA mapped RT from old bootaa64.efi is
maintained.

Adding support to other platforms should be easier as a result.

Slight code re-organisation. NFCI.

sys: Membar audit around reference count releases.

If two threads are using an object that is freed when the reference
count goes to zero, we need to ensure that all memory operations
related to the object happen before freeing the object.

Using an atomic_dec_uint_nv(&refcnt) == 0 ensures that only one
thread takes responsibility for freeing, but it's not enough to
ensure that the other thread's memory operations happen before the
freeing.

Consider:

Thread A Thread B
obj->foo = 42; obj->baz = 73;
mumble(&obj->bar); grumble(&obj->quux);
/* membar_exit(); */ /* membar_exit(); */
atomic_dec -- not last atomic_dec -- last
/* membar_enter(); */
KASSERT(invariant(obj->foo,
obj->bar));
free_stuff(obj);

The memory barriers ensure that

obj->foo = 42;
mumble(&obj->bar);

in thread A happens before

KASSERT(invariant(obj->foo, obj->bar));
free_stuff(obj);

in thread B. Without them, this ordering is not guaranteed.

So in general it is necessary to do

membar_exit();
if (atomic_dec_uint_nv(&obj->refcnt) != 0)
return;
membar_enter();

to release a reference, for the `last one out hit the lights' style
of reference counting. (This is in contrast to the style where one
thread blocks new references and then waits under a lock for existing
ones to drain with a condvar -- no membar needed thanks to mutex(9).)

I searched for atomic_dec to find all these. Obviously we ought to
have a better abstraction for this because there's so much copypasta.
This is a stop-gap measure to fix actual bugs until we have that. It
would be nice if an abstraction could gracefully handle the different
styles of reference counting in use -- some years ago I drafted an
API for this, but making it cover everything got a little out of hand
(particularly with struct vnode::v_usecount) and I ended up setting
it aside to work on psref/localcount instead for better scalability.

I got bored of adding #ifdef __HAVE_ATOMIC_AS_MEMBAR everywhere, so I
only put it on things that look performance-critical on 5sec review.
We should really adopt membar_enter_preatomic/membar_exit_postatomic
or something (except they are applicable only to atomic r/m/w, not to
atomic_load/store_*, making the naming annoying) and get rid of all
the ifdefs.

Slight comment improvement.

fix various typos, mainly in comments.

add support Hardware updates to Access flag and Dirty state (FEAT_HAFDBS)

- The DBM bit of the PTE is now used to determine if it is writable, and
the AF bit is treated entirely as a reference bit. A valid PTE is always
treated as readable. There can be no valid PTE that is not readable.
- LX_BLKPAG_OS_{READ,WRITE} are used only for debugging purposes,
and has been superseded by LX_BLKPAG_AF and LX_BLKPAG_DBM.
- Improve comment

The need for reference/modify emulation has been eliminated,
and access/permission faults have been reduced, however,
there has been little change in overall performance.

Skip unnecessary PTE operations and TLB invalidation.

aarch64/pmap: remove stray semicolon

No binary change.

The translations that need invalidation are always last level so remove
the (dubious) logic around tracking the level. The "any level" TLB
invalidation maintenance operation are used, but this may change after
further testing.

before
======
1661.0u 420.2s 2:53.82 1197.3% 231+29k 10+33918io 102pf+0w
1646.8u 425.2s 2:52.96 1198.0% 232+29k 1+33937io 49pf+0w
1647.9u 425.7s 2:52.58 1201.6% 232+29k 0+33940io 32pf+0w

After
=====
1602.5u 420.8s 2:49.09 1196.6% 238+30k 24+33893io 54pf+0w
1600.7u 421.3s 2:51.53 1178.8% 238+30k 1+33914io 33pf+0w
1597.5u 424.3s 2:50.46 1186.1% 238+30k 0+33915io 17pf+0w

LGTM from ryo@

Restore the previous pmap_remove_all behaviour as the new method meant
the n1sdp couldn't complete a build.

No noticeable change in kernel build performance.

KNF

s/occured/occurred/ in comments, log messages and man pages.

fix various typos, mainly in comments.

Fix non-UVMHIST build

fix non-MULTIPROCESSOR build

Use sys/uvm/pmap/pmap_tlb.c on Aarch64 in the same way that some Arm, MIPS,
and some PPC kernels do. This removes the limitation of 256 processes on
CPUs with 8bit ASID field, e.g. Apple M1.

Additionally the following changes have been made

- removed a couple of unnecessary aarch64_tlbi_all calls
- removed any invalidation after freeing page tables due to
_pmap_sweep_pdp. This was never necessary afaict.
- all kernel mappings are marked global and userland mapping not-global.

Performance testing hasn't show a significant difference. The data here
is from building a kernel on an lx2k system with nvme.

before
1489.6u 400.4s 2:40.65 1176.5% 228+224k 0+32289io 57pf+0w
1482.6u 403.2s 2:38.49 1189.9% 228+222k 0+32274io 46pf+0w
1485.4u 402.2s 2:37.27 1200.2% 228+222k 0+32275io 12pf+0w

after
1493.9u 404.6s 2:37.50 1205.4% 227+221k 0+32265io 48pf+0w
1485.0u 408.0s 2:38.54 1194.0% 227+222k 0+32272io 36pf+0w
1484.3u 407.0s 2:35.88 1213.3% 228+224k 0+32268io 14pf+0w

>>> stats.ttest_ind([160.65,158.49,157.27], [157.5,158.54,155.88])
Ttest_indResult(statistic=1.1923622711296888, pvalue=0.2990182944606766)
>>>

Ensure TCR_EPD0 is set on entry to pmap_activate and ensure it is set as
early as possible for APs.

Use UVMHIST_CALLARGS

'\n' is not required in KASSERTMSG either.

"\n" is not required in KERNHIST

Use __SHIFTIN. Same code before and after.

pmap_page_remove: simply and reduce the code size slightly.

In pmap_icache_sync_range change

for (...) {
...
if (condition) {
// do stuff
}
}

to

for (...) {
...
if (!conditional)
continue;
// do stuff
}

to save on indentation. Same code (modulo register usage) before and
after.

KNF

Deal with the pmap limitation of maxproc in a more complete way and
recognise CPUs with only 8bit ASIDs.

branches: 1.107.2;
Make the ddb for pmap / pte information pmap agnostic

Remove some unnecessary tlb invalidate in pmap_growkernel and ASAN shadow
map. Ensure the shadow map mappings are visible to the TLB walkers.

branches: 1.105.2;
added more attributes of PTE displayed by "ddb>machine pte"

remove KERNHIST_INIT_STATIC(). it stradles the line between usable
early in boot and broken early in boot by requiring a partly static
structure with another structure that must be present by the time
any uses are performed. theoretically platform code could allocate
a chunk while seting up memory and assign it here, giving a dynamic
sizing for the entry list, but the reality is that all users have
a statically allocated entry list as well.

the existing KERNHIST_LINK_STATIC() is used in conjunction with
KERNHIST_INITIALIZER() instead.

this stops a NULL pointer deref when the _LOG() macro is called
before the storage is linked in, which happens with GCC 10 on OCTEON
with UVMHIST enabled, crashing in very early kernel init.

branches: 1.103.2;
fix build error without options DDB.

kvtopte() is referenced from arm/acpi/acpi_machdep.c

No assignment is needed here.

the loop in pmap_page_remove() always removes the first pv,
and since the list is managed by _pmap_remove_pv(), pp->pp_pv.pv_next always points to the first.

It is enough to make a page accessible instead of writable.
same fix as r1.76

implement pmap_remove_all().

The size of struct pv_entry has increased, but speed of kernel build has improved by about 1%
exec and exit should have been improved.

Improve the English in the previous comment fix.

Tweak a comment

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

AA64 is not MIPS.

Change all KSEG references to directmap

In pmap_devmap_bootstrap only set pmap_devmap_bootstrap_done if there
is an entry and ALL of the entries have been done. The entry required
for EARLYCONS might not be the first/only one...

branches: 1.94.2;
No need to disable translation table walks in pmap_activate().

Use the isb macro - missed in previous commit

Use the dmb/dsb/isb macros... if nothing else they're all now consistent
about the "memory" assembler contraint.

No binary change

Only set pmap_devmap_bootstrap_done if something gets mapped.

Think acpi_platform_devmap

Make __md_palloc pmap agnostic (think sys/uvm/pmap)

PID_MAX is just an initial value (soft maximum). Don't use it for CTASSERT.
defined __HAVE_CPU_MAXPROC to use function cpu_maxproc().

pointed out by mrg@, thanks.

Fix panic caused by modload. http://mail-index.netbsd.org/port-arm/2020/08/30/msg006960.html

The address space reserved for modules may not be mapped in L1-L3.

Whitespace

Part IV of ad's performance improvements for aarch64

- Implement pmap_growkernel(), and update kernel pmap's stats with atomics.

- Then, pmap_kenter_pa() and pmap_kremove() no longer need to allocate
memory nor take pm_lock, because they only modify L3 PTEs.

- Then, pm_lock and pp_lock can be adaptive mutexes at IPL_NONE which are
cheaper than spin mutexes.

- Take the pmap's lock in pmap_extract() if not the kernel's pmap, otherwise
pmap_extract() might see inconsistent state.

Fix another UVMHIST so it doesn't use %s

pmapboot_enter simplication
- bootpage_alloc in asm becomes pmapboot_pagealloc in C
- PMAPBOOT_ENTER_NOBLOCK is removed as it's not used
- PMAPBOOT_ENTER_NOOVERWRITE is removed as it's now always on
- physpage_allocator argument is removed as it's always
pmapboot_pagealloc
- Support for EARLYCONS without CONSADDR is removed so that the identity
map for CONSADDR is always known.

For the assembly files:
2 files changed, 40 insertions(+), 89 deletions(-)

LGTM ryo

Use tlen for temporary length variable instead of l, which is usually
used for struct lwp *.

No binary changes.

pmap_procwr(): sync icache even if p != curproc. This fixes applications
like GDB for arm32, that rewrite text of other process.

Thanks to ryo@ for discussion.

Set uvmexp.ncolors appropriately, which is required for some CPU
models with VIPT icache.

Otherwise, alias in virtual address results in inconsistent results,
at least for applications that rewrite text of other process, e.g.,
GDB for arm32.

Also, this hopefully fixes other unexpected failures due to alias.

Confirmed that there's no observable regression in performance;
difference in ``time make -j8'' for GENERIC64 kernel on BCM2837
with and without setting uvmexp.ncolors is within 0.1%.

Thanks to ryo@ for discussion.

Fix typo in name of evcnt(4) counter.

Fix bug with incorrect range calculation when doing icache sync.
This is called by sysarch(ARM_SYNC_ICACHE) from aarch32 (compat_netbsd32) emul process.

pointed out by rin@, thanks.

XXX pullup-9

- Fix a lock order reversal in pmap_page_protect().

- Make sure pmap is always locked when updating stats; atomics no longer
needed to do that.

- Remove unneeded traversal of pv list in pmap_enter_pv().

- Shrink struct vm_page from 136 to 128 bytes (cache line sized) and struct
pv_entry from 48 to 32 bytes (power of 2 sized).

- Embed a pv_entry in each vm_page. This means PV entries don't need to
be allocated for private anonymous memory / COW pages / most UBC mappings.
Dynamic PV entries are then used only for stuff like shared libraries and
shared memory.

Proposed on port-arm@.

- Wired/resident stats shouldn't covered by PMAPCOUNTERS.
- Rename need_update_pv -> need_enter_pv.

Ok ryo@

no need to make the PTE writable to do icache_sync, enough to accessible.

Use __diagused

fix non-diag build

Use MUTEX_NODEBUG for PV locks as is commonly done. OK ryo.

Implement pmap_extract_coherency

PMAP_DEBUG has been deleted on arm

Add support for Branch Target Identification (BTI).

On the executable pages that have the GP (Guarded Page) bit, the semantic
of the "br" and "blr" instructions is changed: the CPU expects the first
instruction of the jump/call target to be "bti", and faults if it isn't.

We add the GP bit on the kernel .text pages (and incidentally the .rodata
pages, but we don't care). The compiler adds a "bti c" instruction at the
beginning of each C function. We modify the ENTRY() macros to manually add
"bti c" in the asm functions.

cpuswitch.S needs a specific change: with "br x27" the CPU expects "bti j",
which is bad because the functions begin with "bti c"; switch to "br x16",
for the CPU to accept "bti c".

BTI helps defend against JOP/COP. Tested on Qemu.

branches: 1.69.2;
use PMAP_PAGE_INIT() to initialize mutex in pmap_page.

VM_MDPAGE_INIT() in pmap_free_pdp() had initialized pp_flags,
so it unintentionally cleared PMAP_PAGE_FLAGS_PV_TRACKED.
use PMAP_PAGE_INIT to avoid using PMAP_PAGE_FLAGS_PV_TRACKED.

pointed out by tnn@, thanks

pmap_remove_all(): Return a boolean value to indicate the behaviour. If
true, all mappings have been removed, the pmap is totally cleared out, and
UVM can then avoid doing the work to call pmap_remove() for each map entry.
If false, either nothing has been done, or some helpful arch-specific voodoo
has taken place.

oops, fix incorrect usage of daif_enable() in my previous commit.

Fix pmap to work correctly with tagged addresses

- when fault, untag from address before passing to uvm/pmap functions
- pmap_extract() checks more strictly and consider the address tag

use pmapboot_enter_range()

use LIST(3) instead of TAILQ(3) to save one word in struct vm_page and struct pmap.

pointed out by riastradh@. thanks

add support pmap_pv(9)

Patch originally from jmcneill@. thanks

separate struct vm_page_md into vm_page_md and pmap_page
for preparation pmap_pv(9)

fix behaviour mmap()/mprotect() when passed only PROT_EXEC.

when mmap()/mprotect() with only PROT_EXEC, syscall will be successful,
but the page actually hadn't been mapped.
it should be mapped with PROT_READ|PROT_EXEC implicitly. (r-x)

branches: 1.60.2;
Update pmap_map_chunk to allow L[12] block mappings and L3 page mappings

Remove unnecessary brackets and unwrap a conditional. Same code before
and after.

Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

G/C kasan_shadow_map call in pmap_enter

pmap_growkernel calls kasan_shadow_map for KVA

atomic_add_16() is not used in pmap.c anymore. no need decl here.

space to tab

Fix build... wire_count probably doesn't need atomics

Fix KASAN support by calling kasan_shadow_map in pmap_growkernel

pg->phys_addr -> VM_PAGE_TO_PHYS(pg)

Mark several kASan functions with __nothing, to avoid annoying #ifdefs.
Same as kCSan and kMSan.

Don't include "opt_kasan.h" when there's already <sys/asan.h> included.

Define PMAP_NEED_PROCWR, providing strategically placed i-cache
synchronization where just-changed memory is about to be executed.

Fixes SIGILLs seen when running Mono 6 on QEMU Cortex-A57.

ok ryo

Disable translation table walks using TTBR0 while changing its value and
when deactivating a pmap. Fixes stability issues on Ampere eMAG CPUs.

ref/mod bit should be set according to 'flags' argument, not 'prot'. r1.44 was incomplete.

In pmap_devmap_bootstrap(), cpu_earlydevice_va_p() must not return true until *all* devmap tables have been enabled.
console mapping may be present in the last table.

- remove incorrect KASSERT. mmap(2) with prot=PROT_WRITE calls pmap_enter(..., PROT_WRITE) internally.
- fix to update page reference flags when only PROT_WRITE or PROT_EXECUTE specified

Make pmap_db_pte_print more terse so it's quicker on serial consoles

Use PMAP_DEV in DEVMAP_ENTRY rather than pmap_map_chunk. It's clearer and
means pmap_map_chunk can be made to map other memory types.

branches: 1.41.2;
apply some __diagused.

- free empty page tables pages if reach a certain usage.
- need to lock at removing an old pg (_pmap_remove_pv) in _pmap_enter()

Fix race conditions about pmap_page_protect() and pmap_enter().

while handling same PTE by these functions in same time, there
is a critical path that the number of valid PTEs and wire_count
are inconsistent, and it caused KASSERT.
Need to hold a pv_lock while modifying them.

spinkle __printflike(), and use PRIxxx

- add ddb command "machine ttbr" to dump MMU tables.
- tidy up descriptions, usages and messages.

- free L1-L3 pages that has been emptied by pmap_remove().
- if no memories, pmap_enter will return correctly ENOMEM if PMAP_CANFAIL, or wait until available any memories if !PMAP_CANFAIL.

These changes improves the stability when we use a huge virtual memory spaces with mmap.

improve pmap_remove
- don't lock/unlock per page in pmap_remove()
- speedup pte lookup for continuous addresses
- bring out pool_cache_put(&_pmap_pv_pool, pv) from lock/unlock section

- add workaround for Cavium ThunderX errata 27456.
- add cpufuncs table in cpu_info. each cpu clusters may have different erratum. (e.g. big.LITTLE)

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

invalidate icache correctly.
l3pte_executable() should be used for only valid pte.

Provide generic start code that assumes the MMU is off and caches are
disabled as per the linux booting protocol for ARMv6 and ARMv7 boards.
u-boot image type should be changed to 'linux' for correct behaviour.

The new start code builds a minimal "bootstrap" L1PT with cached access
disabled and uses the same table for all processors. AP startup is
performed in less steps and more code is written in C.

The bootstrap tables and stack are placed into an (orphaned) section
"_init_memory" which is given to uvm when it is no longer used.

Various kernels have been converted to use this code and tested. Some
boards were provided by TNF. Thanks!

The GENERIC kernel now boots on boards using the TEGRA, SUNXI and EXYNOS
kernels. The GENERIC kernel will also work on RPI2 using u-boot.

Thanks to martin@ and aymeric@ for testing on parallella and nanosoc
respectively

Use __nothing

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

- cleanup checking address ranges with IN_RANGE macro
- change PM_ADDR_CHECK macro to KASSERTMSG
- restore fast lookup cases with IN_RANGE macro for pmap_extract changed in my previous commit.

rewrite pmap_pte_lookup() to share similar code.

cleanup locore, and changed the way to map memories during boot.
- add functions bootpage_enter() and bootpage_alloc() to adapt various layout
of physical memory map. especially for 64bit physical memory layout.
pmapboot_alloc() allocates pagetable pages from _end[].
- changed to map only the required amount for PA=VA identity mapping
(kernel image, UART device, and FDT blob) with L2_BLOCK(2Mbyte).
- changing page permission for kernel image, and making KSEG mapping are done
at cpu_kernel_vm_init() instead of at locore.
- optimize PTE entries with PTE Contiguous bit. it is enabled on devmap only for now.

reviewed by skrll@, thanks.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

delete debug printf and KASSERT.

Replace KDASSERT by panic.

Rename _pmap_alloc_pdp -> pmap_alloc_pdp, and make it public.

cleanup aarch64 mpstart and fdt bootstrap
* arm_cpu_hatch_arg is a bad idea. avoid serializing CPU startup, and eliminate arm_cpu_hatch_arg.
in mpstart, resolve own cpu index using array of cpu_mpidr[] (aarch64)
* add support fdt enable-method "spin-table"
* add support fdt enable-method "brcm,bcm2836-smp" (for 32bit RaspberryPi)
* use arm_fdt_cpu_bootstrap() instead of psci_fdt_bootstrap()
* rename "arm/fdt/psci_fdt.h" to "arm/fdt/psci_fdtvar.h" because of conflict of include file for needs-flag
* add devmap for cpu spin-table of raspberrypi3/aarch64
* no need to force hatch APs for raspberrypi3/arm32 ifndef MULTIPROCESSOR.
* fix to work pmap_extract(kerneltext/data/bss) even if before calling pmap_bootstrap

idea to use cpu_mpidr[] by jmcneill@. reviewd by skrll@. thanks.

need to add VM_PROT_READ when pmap_kenter_pa(va, pa, VM_PROT_WRITE, 0) or pmap_kenter_pa(va, pa, VM_PROT_EXECUTE, 0).
VM_PROT_READ is treated as an access permission inernally.

change to minimum invalidation of TLB.
specifying not only va but also asid, and not invalidate L0-L2 entry using tlbi_*_ll() if needed.

treat kernel-exec attr and user-exec attr separately.
kernel cannot execute userland exec page, and user cannot execute kernel page.

set kernel text/rodata readonly by default.
add function db_write_text() for setting ddb breakpoint.

Define and use VPRINTF

changes of pmap.c r1.13 seems to be unstable.
In order to invalidate icache, not to invalidate all icache,
but temporary to make the page writable and invalidate target address only.

don't call pool_cache_put with locking pmap. pool_cache_put call pmap_kenter_pa internally.
(pool_cache_put_paddr -> pool_cache_put_slow -> pool_get -> pmap_kenter_pa)

* fix icache invalidations.
* "ic ivau" (aarch64_icache_sync_range) with VA generates permission fault in some situations, therefore use KSEG address for now.

rather than using flags to resolve nested locks, reserve pool_cache before locking.

* avoid deadlock. mutex_owned() works only for adaptive lock, therefore we cannot use it for spinlock...
* add more NULL check
* clear pte when pmap_enter() fails

Use __debugused

Add missing casts, remove unused variables.

need locks in pmap_kremove() and pmap_fault_fixup()

branches: 1.7.2;
pmap_enter() must update modified/referenced flags by 'flags' not 'prot'.

Fix memory leak. it was leaking one page every pmap_create().
pm->pm_vmlist must be initialized before calling _pmap_alloc_pdp().

fix KASSERT panic. pv_entry may not be exists in pvlist when pmap_remove().

delete unused code

fix instability behavior of bufcache on aarch64.
* fix to return correct ref/mod when PMAP_WIRED.
* changed to keep wired flags in pte instead of pv_entry, and cleanup.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove ryo@'s mail addresses.

Provide a shared pmap_devmap implementation and convert all pmap_devmap
arrays to use DEVMAP_ENTRY{,_END}

Rename VM_KERNEL_IO_ADDRESS to VM_KERNEL_IO_BASE to match RISC-V

It's less letters, matches other similar variables and will help with
sharing code between the two architectures.

NFCI.

aarch64/pmap: Fix criterion in previous.

Use the pte bit that says whether this is a PMAP_WIRED page, not the
bit that says whether this is a non-global page.

(Forgot to git commit --amend before exporting to CVS, sorry!)

aarch64/pmap(9): Teach pmap_protect about pmap_kenter_pa mappings.

Pages mapped with pmap_kenter_pa are necessarily unmanaged, so there
are no P->V records, and pmap_kenter_pa leaves pp->pp_pv.pv_va zero
with no modified/referenced state.

However, pmap_protect erroneously examined pp->pp_pv.pv_va to
ascertain the modified/referenced state -- and if the page was not
marked referenced, pmap_protect would clear the LX_BLKPAG_AF bit
(Access Flag), with the effect that subsequent uses of the page fault
and require a detour through pmap_fault_fixup.

This caused problems for the kernel module loader:

- When loading the text section, kobj_load first allocates kva with
uvm_km_alloc(UVM_KMF_WIRED|UVM_KMF_EXEC), which creates ptes with
pmap_kenter_pa. These ptes are writable, so we can copy the text
section into them, and have LX_BLKPAG_AF set so there will be no
fault when they are used by the kernel.

- But then kobj_affix makes the text section read/execute-only (and
nonwritable) with uvm_km_protect(VM_PROT_READ|VM_PROT_EXECUTE),
which updates the ptes with pmap_protect. This _should_ leave
LX_BLKPAG_AF set, but by inadvertently treating the page as managed
when it should be unmanaged, pmap_protect cleared it instead.

- Most of the time, clearing LX_BLKPAG_AF caused no problem, because
pmap_fault_fixup would silently resolve it. But if a hard
interrupt handler tried to use any page in the module's text (or
rodata, I suspect) that was not yet fixed up, the CPU would fault
and enter pmap_fault_fixup -- which would promptly crash (or hang)
by trying to take the pmap lock in interrupt context, which is
forbidden.

I observed this by loading dtrace.kmod early at boot and trying to
dtrace hard interrupt handlers.

With this change, pmap_protect now recognizes wired mappings (as
created by pmap_kenter_pa) before consulting pp->pp_pv.pv_va, and
preserves then LX_BLKPAG_AF bit in that case.

ok skrll

fix a spello in a comment

Remove some empty lines

Use UVMHIST_CALLARGS in pmap_bootstrap

Only define the EFI variable if EFI_RUNTIME

KNF

Use "non-posted" instead of "strongly ordered" to describe nGnRnE mappings

Rename the following defines:
- _ARM_BUS_SPACE_MAP_STRONGLY_ORDERED to BUS_SPACE_MAP_NONPOSTED
- PMAP_DEV_SO to PMAP_DEV_NP
- LX_BLKPAG_ATTR_DEVICE_MEM_SO to LX_BLKPAG_ATTR_DEVICE_MEM_NP
Rename the following option:
- AARCH64_DEVICE_MEM_STRONGLY_ORDERED to AARCH64_DEVICE_MEM_NONPOSTED

Fixed a bug that pte's __BIT(63,48) could be set when accessing addresses above 0x0001000000000000 in /dev/mem with mmap().

When accessed in mmap by the device pager, pmap_enter() may be called with prot == PROT_WRITE.

Sprinkle some KASSERT(kpreempt_disabled());

since pmap_activate_efirt() rewrites TTBR0, it is necessary to pmap_activate() again after pmap_deactivate_efirt() to restore the original TTBR0.

- Fix to do pmap_{de,}activate() before/after pmap_{,de}activate_efirt().
- moved kpreempt_{disable,enable}() to the caller since everything between
arm_efirt_md_enter() and arm_efirt_md_exit() should be kpreempt disabled.

ok skrll@

Add the missing kpreempt_enable to pmap_deactivate_efirt

No need to flush icache for EFI RT mappings as bootaa64.efi flushed
the full icache for us. (Also this avoids traps)

sys: Use membar_release/acquire around reference drop.

This just goes through my recent reference count membar audit and
changes membar_exit to membar_release and membar_enter to
membar_acquire -- this should make everything cheaper on most CPUs
without hurting correctness, because membar_acquire is generally
cheaper than membar_enter.

Update to support EFI runtime outside the kernel virtual address space
by creating an EFI RT pmap that can be activated / deactivated when
required.

Adds support for EFI RT to ARM_MMU_EXTENDED (ASID) 32-bit Arm machines.

On Arm64 the usage of pmapboot_enter is reduced and the mappings are
created much later in the boot process -- now in cpu_startup_hook.
Backward compatiblity for KVA mapped RT from old bootaa64.efi is
maintained.

Adding support to other platforms should be easier as a result.

Slight code re-organisation. NFCI.

sys: Membar audit around reference count releases.

If two threads are using an object that is freed when the reference
count goes to zero, we need to ensure that all memory operations
related to the object happen before freeing the object.

Using an atomic_dec_uint_nv(&refcnt) == 0 ensures that only one
thread takes responsibility for freeing, but it's not enough to
ensure that the other thread's memory operations happen before the
freeing.

Consider:

Thread A Thread B
obj->foo = 42; obj->baz = 73;
mumble(&obj->bar); grumble(&obj->quux);
/* membar_exit(); */ /* membar_exit(); */
atomic_dec -- not last atomic_dec -- last
/* membar_enter(); */
KASSERT(invariant(obj->foo,
obj->bar));
free_stuff(obj);

The memory barriers ensure that

obj->foo = 42;
mumble(&obj->bar);

in thread A happens before

KASSERT(invariant(obj->foo, obj->bar));
free_stuff(obj);

in thread B. Without them, this ordering is not guaranteed.

So in general it is necessary to do

membar_exit();
if (atomic_dec_uint_nv(&obj->refcnt) != 0)
return;
membar_enter();

to release a reference, for the `last one out hit the lights' style
of reference counting. (This is in contrast to the style where one
thread blocks new references and then waits under a lock for existing
ones to drain with a condvar -- no membar needed thanks to mutex(9).)

I searched for atomic_dec to find all these. Obviously we ought to
have a better abstraction for this because there's so much copypasta.
This is a stop-gap measure to fix actual bugs until we have that. It
would be nice if an abstraction could gracefully handle the different
styles of reference counting in use -- some years ago I drafted an
API for this, but making it cover everything got a little out of hand
(particularly with struct vnode::v_usecount) and I ended up setting
it aside to work on psref/localcount instead for better scalability.

I got bored of adding #ifdef __HAVE_ATOMIC_AS_MEMBAR everywhere, so I
only put it on things that look performance-critical on 5sec review.
We should really adopt membar_enter_preatomic/membar_exit_postatomic
or something (except they are applicable only to atomic r/m/w, not to
atomic_load/store_*, making the naming annoying) and get rid of all
the ifdefs.

Slight comment improvement.

fix various typos, mainly in comments.

add support Hardware updates to Access flag and Dirty state (FEAT_HAFDBS)

- The DBM bit of the PTE is now used to determine if it is writable, and
the AF bit is treated entirely as a reference bit. A valid PTE is always
treated as readable. There can be no valid PTE that is not readable.
- LX_BLKPAG_OS_{READ,WRITE} are used only for debugging purposes,
and has been superseded by LX_BLKPAG_AF and LX_BLKPAG_DBM.
- Improve comment

The need for reference/modify emulation has been eliminated,
and access/permission faults have been reduced, however,
there has been little change in overall performance.

Skip unnecessary PTE operations and TLB invalidation.

aarch64/pmap: remove stray semicolon

No binary change.

The translations that need invalidation are always last level so remove
the (dubious) logic around tracking the level. The "any level" TLB
invalidation maintenance operation are used, but this may change after
further testing.

before
======
1661.0u 420.2s 2:53.82 1197.3% 231+29k 10+33918io 102pf+0w
1646.8u 425.2s 2:52.96 1198.0% 232+29k 1+33937io 49pf+0w
1647.9u 425.7s 2:52.58 1201.6% 232+29k 0+33940io 32pf+0w

After
=====
1602.5u 420.8s 2:49.09 1196.6% 238+30k 24+33893io 54pf+0w
1600.7u 421.3s 2:51.53 1178.8% 238+30k 1+33914io 33pf+0w
1597.5u 424.3s 2:50.46 1186.1% 238+30k 0+33915io 17pf+0w

LGTM from ryo@

Restore the previous pmap_remove_all behaviour as the new method meant
the n1sdp couldn't complete a build.

No noticeable change in kernel build performance.

KNF

s/occured/occurred/ in comments, log messages and man pages.

fix various typos, mainly in comments.

Fix non-UVMHIST build

fix non-MULTIPROCESSOR build

Use sys/uvm/pmap/pmap_tlb.c on Aarch64 in the same way that some Arm, MIPS,
and some PPC kernels do. This removes the limitation of 256 processes on
CPUs with 8bit ASID field, e.g. Apple M1.

Additionally the following changes have been made

- removed a couple of unnecessary aarch64_tlbi_all calls
- removed any invalidation after freeing page tables due to
_pmap_sweep_pdp. This was never necessary afaict.
- all kernel mappings are marked global and userland mapping not-global.

Performance testing hasn't show a significant difference. The data here
is from building a kernel on an lx2k system with nvme.

before
1489.6u 400.4s 2:40.65 1176.5% 228+224k 0+32289io 57pf+0w
1482.6u 403.2s 2:38.49 1189.9% 228+222k 0+32274io 46pf+0w
1485.4u 402.2s 2:37.27 1200.2% 228+222k 0+32275io 12pf+0w

after
1493.9u 404.6s 2:37.50 1205.4% 227+221k 0+32265io 48pf+0w
1485.0u 408.0s 2:38.54 1194.0% 227+222k 0+32272io 36pf+0w
1484.3u 407.0s 2:35.88 1213.3% 228+224k 0+32268io 14pf+0w

>>> stats.ttest_ind([160.65,158.49,157.27], [157.5,158.54,155.88])
Ttest_indResult(statistic=1.1923622711296888, pvalue=0.2990182944606766)
>>>

Ensure TCR_EPD0 is set on entry to pmap_activate and ensure it is set as
early as possible for APs.

Use UVMHIST_CALLARGS

'\n' is not required in KASSERTMSG either.

"\n" is not required in KERNHIST

Use __SHIFTIN. Same code before and after.

pmap_page_remove: simply and reduce the code size slightly.

In pmap_icache_sync_range change

for (...) {
...
if (condition) {
// do stuff
}
}

to

for (...) {
...
if (!conditional)
continue;
// do stuff
}

to save on indentation. Same code (modulo register usage) before and
after.

KNF

Deal with the pmap limitation of maxproc in a more complete way and
recognise CPUs with only 8bit ASIDs.

branches: 1.107.2;
Make the ddb for pmap / pte information pmap agnostic

Remove some unnecessary tlb invalidate in pmap_growkernel and ASAN shadow
map. Ensure the shadow map mappings are visible to the TLB walkers.

branches: 1.105.2;
added more attributes of PTE displayed by "ddb>machine pte"

remove KERNHIST_INIT_STATIC(). it stradles the line between usable
early in boot and broken early in boot by requiring a partly static
structure with another structure that must be present by the time
any uses are performed. theoretically platform code could allocate
a chunk while seting up memory and assign it here, giving a dynamic
sizing for the entry list, but the reality is that all users have
a statically allocated entry list as well.

the existing KERNHIST_LINK_STATIC() is used in conjunction with
KERNHIST_INITIALIZER() instead.

this stops a NULL pointer deref when the _LOG() macro is called
before the storage is linked in, which happens with GCC 10 on OCTEON
with UVMHIST enabled, crashing in very early kernel init.

branches: 1.103.2;
fix build error without options DDB.

kvtopte() is referenced from arm/acpi/acpi_machdep.c

No assignment is needed here.

the loop in pmap_page_remove() always removes the first pv,
and since the list is managed by _pmap_remove_pv(), pp->pp_pv.pv_next always points to the first.

It is enough to make a page accessible instead of writable.
same fix as r1.76

implement pmap_remove_all().

The size of struct pv_entry has increased, but speed of kernel build has improved by about 1%
exec and exit should have been improved.

Improve the English in the previous comment fix.

Tweak a comment

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

AA64 is not MIPS.

Change all KSEG references to directmap

In pmap_devmap_bootstrap only set pmap_devmap_bootstrap_done if there
is an entry and ALL of the entries have been done. The entry required
for EARLYCONS might not be the first/only one...

branches: 1.94.2;
No need to disable translation table walks in pmap_activate().

Use the isb macro - missed in previous commit

Use the dmb/dsb/isb macros... if nothing else they're all now consistent
about the "memory" assembler contraint.

No binary change

Only set pmap_devmap_bootstrap_done if something gets mapped.

Think acpi_platform_devmap

Make __md_palloc pmap agnostic (think sys/uvm/pmap)

PID_MAX is just an initial value (soft maximum). Don't use it for CTASSERT.
defined __HAVE_CPU_MAXPROC to use function cpu_maxproc().

pointed out by mrg@, thanks.

Fix panic caused by modload. http://mail-index.netbsd.org/port-arm/2020/08/30/msg006960.html

The address space reserved for modules may not be mapped in L1-L3.

Whitespace

Part IV of ad's performance improvements for aarch64

- Implement pmap_growkernel(), and update kernel pmap's stats with atomics.

- Then, pmap_kenter_pa() and pmap_kremove() no longer need to allocate
memory nor take pm_lock, because they only modify L3 PTEs.

- Then, pm_lock and pp_lock can be adaptive mutexes at IPL_NONE which are
cheaper than spin mutexes.

- Take the pmap's lock in pmap_extract() if not the kernel's pmap, otherwise
pmap_extract() might see inconsistent state.

Fix another UVMHIST so it doesn't use %s

pmapboot_enter simplication
- bootpage_alloc in asm becomes pmapboot_pagealloc in C
- PMAPBOOT_ENTER_NOBLOCK is removed as it's not used
- PMAPBOOT_ENTER_NOOVERWRITE is removed as it's now always on
- physpage_allocator argument is removed as it's always
pmapboot_pagealloc
- Support for EARLYCONS without CONSADDR is removed so that the identity
map for CONSADDR is always known.

For the assembly files:
2 files changed, 40 insertions(+), 89 deletions(-)

LGTM ryo

Use tlen for temporary length variable instead of l, which is usually
used for struct lwp *.

No binary changes.

pmap_procwr(): sync icache even if p != curproc. This fixes applications
like GDB for arm32, that rewrite text of other process.

Thanks to ryo@ for discussion.

Set uvmexp.ncolors appropriately, which is required for some CPU
models with VIPT icache.

Otherwise, alias in virtual address results in inconsistent results,
at least for applications that rewrite text of other process, e.g.,
GDB for arm32.

Also, this hopefully fixes other unexpected failures due to alias.

Confirmed that there's no observable regression in performance;
difference in ``time make -j8'' for GENERIC64 kernel on BCM2837
with and without setting uvmexp.ncolors is within 0.1%.

Thanks to ryo@ for discussion.

Fix typo in name of evcnt(4) counter.

Fix bug with incorrect range calculation when doing icache sync.
This is called by sysarch(ARM_SYNC_ICACHE) from aarch32 (compat_netbsd32) emul process.

pointed out by rin@, thanks.

XXX pullup-9

- Fix a lock order reversal in pmap_page_protect().

- Make sure pmap is always locked when updating stats; atomics no longer
needed to do that.

- Remove unneeded traversal of pv list in pmap_enter_pv().

- Shrink struct vm_page from 136 to 128 bytes (cache line sized) and struct
pv_entry from 48 to 32 bytes (power of 2 sized).

- Embed a pv_entry in each vm_page. This means PV entries don't need to
be allocated for private anonymous memory / COW pages / most UBC mappings.
Dynamic PV entries are then used only for stuff like shared libraries and
shared memory.

Proposed on port-arm@.

- Wired/resident stats shouldn't covered by PMAPCOUNTERS.
- Rename need_update_pv -> need_enter_pv.

Ok ryo@

no need to make the PTE writable to do icache_sync, enough to accessible.

Use __diagused

fix non-diag build

Use MUTEX_NODEBUG for PV locks as is commonly done. OK ryo.

Implement pmap_extract_coherency

PMAP_DEBUG has been deleted on arm

Add support for Branch Target Identification (BTI).

On the executable pages that have the GP (Guarded Page) bit, the semantic
of the "br" and "blr" instructions is changed: the CPU expects the first
instruction of the jump/call target to be "bti", and faults if it isn't.

We add the GP bit on the kernel .text pages (and incidentally the .rodata
pages, but we don't care). The compiler adds a "bti c" instruction at the
beginning of each C function. We modify the ENTRY() macros to manually add
"bti c" in the asm functions.

cpuswitch.S needs a specific change: with "br x27" the CPU expects "bti j",
which is bad because the functions begin with "bti c"; switch to "br x16",
for the CPU to accept "bti c".

BTI helps defend against JOP/COP. Tested on Qemu.

branches: 1.69.2;
use PMAP_PAGE_INIT() to initialize mutex in pmap_page.

VM_MDPAGE_INIT() in pmap_free_pdp() had initialized pp_flags,
so it unintentionally cleared PMAP_PAGE_FLAGS_PV_TRACKED.
use PMAP_PAGE_INIT to avoid using PMAP_PAGE_FLAGS_PV_TRACKED.

pointed out by tnn@, thanks

pmap_remove_all(): Return a boolean value to indicate the behaviour. If
true, all mappings have been removed, the pmap is totally cleared out, and
UVM can then avoid doing the work to call pmap_remove() for each map entry.
If false, either nothing has been done, or some helpful arch-specific voodoo
has taken place.

oops, fix incorrect usage of daif_enable() in my previous commit.

Fix pmap to work correctly with tagged addresses

- when fault, untag from address before passing to uvm/pmap functions
- pmap_extract() checks more strictly and consider the address tag

use pmapboot_enter_range()

use LIST(3) instead of TAILQ(3) to save one word in struct vm_page and struct pmap.

pointed out by riastradh@. thanks

add support pmap_pv(9)

Patch originally from jmcneill@. thanks

separate struct vm_page_md into vm_page_md and pmap_page
for preparation pmap_pv(9)

fix behaviour mmap()/mprotect() when passed only PROT_EXEC.

when mmap()/mprotect() with only PROT_EXEC, syscall will be successful,
but the page actually hadn't been mapped.
it should be mapped with PROT_READ|PROT_EXEC implicitly. (r-x)

branches: 1.60.2;
Update pmap_map_chunk to allow L[12] block mappings and L3 page mappings

Remove unnecessary brackets and unwrap a conditional. Same code before
and after.

Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

G/C kasan_shadow_map call in pmap_enter

pmap_growkernel calls kasan_shadow_map for KVA

atomic_add_16() is not used in pmap.c anymore. no need decl here.

space to tab

Fix build... wire_count probably doesn't need atomics

Fix KASAN support by calling kasan_shadow_map in pmap_growkernel

pg->phys_addr -> VM_PAGE_TO_PHYS(pg)

Mark several kASan functions with __nothing, to avoid annoying #ifdefs.
Same as kCSan and kMSan.

Don't include "opt_kasan.h" when there's already <sys/asan.h> included.

Define PMAP_NEED_PROCWR, providing strategically placed i-cache
synchronization where just-changed memory is about to be executed.

Fixes SIGILLs seen when running Mono 6 on QEMU Cortex-A57.

ok ryo

Disable translation table walks using TTBR0 while changing its value and
when deactivating a pmap. Fixes stability issues on Ampere eMAG CPUs.

ref/mod bit should be set according to 'flags' argument, not 'prot'. r1.44 was incomplete.

In pmap_devmap_bootstrap(), cpu_earlydevice_va_p() must not return true until *all* devmap tables have been enabled.
console mapping may be present in the last table.

- remove incorrect KASSERT. mmap(2) with prot=PROT_WRITE calls pmap_enter(..., PROT_WRITE) internally.
- fix to update page reference flags when only PROT_WRITE or PROT_EXECUTE specified

Make pmap_db_pte_print more terse so it's quicker on serial consoles

Use PMAP_DEV in DEVMAP_ENTRY rather than pmap_map_chunk. It's clearer and
means pmap_map_chunk can be made to map other memory types.

branches: 1.41.2;
apply some __diagused.

- free empty page tables pages if reach a certain usage.
- need to lock at removing an old pg (_pmap_remove_pv) in _pmap_enter()

Fix race conditions about pmap_page_protect() and pmap_enter().

while handling same PTE by these functions in same time, there
is a critical path that the number of valid PTEs and wire_count
are inconsistent, and it caused KASSERT.
Need to hold a pv_lock while modifying them.

spinkle __printflike(), and use PRIxxx

- add ddb command "machine ttbr" to dump MMU tables.
- tidy up descriptions, usages and messages.

- free L1-L3 pages that has been emptied by pmap_remove().
- if no memories, pmap_enter will return correctly ENOMEM if PMAP_CANFAIL, or wait until available any memories if !PMAP_CANFAIL.

These changes improves the stability when we use a huge virtual memory spaces with mmap.

improve pmap_remove
- don't lock/unlock per page in pmap_remove()
- speedup pte lookup for continuous addresses
- bring out pool_cache_put(&_pmap_pv_pool, pv) from lock/unlock section

- add workaround for Cavium ThunderX errata 27456.
- add cpufuncs table in cpu_info. each cpu clusters may have different erratum. (e.g. big.LITTLE)

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

invalidate icache correctly.
l3pte_executable() should be used for only valid pte.

Provide generic start code that assumes the MMU is off and caches are
disabled as per the linux booting protocol for ARMv6 and ARMv7 boards.
u-boot image type should be changed to 'linux' for correct behaviour.

The new start code builds a minimal "bootstrap" L1PT with cached access
disabled and uses the same table for all processors. AP startup is
performed in less steps and more code is written in C.

The bootstrap tables and stack are placed into an (orphaned) section
"_init_memory" which is given to uvm when it is no longer used.

Various kernels have been converted to use this code and tested. Some
boards were provided by TNF. Thanks!

The GENERIC kernel now boots on boards using the TEGRA, SUNXI and EXYNOS
kernels. The GENERIC kernel will also work on RPI2 using u-boot.

Thanks to martin@ and aymeric@ for testing on parallella and nanosoc
respectively

Use __nothing

add initial support of COMPAT_NETBSD32 on AArch64.
arm ELF32 EABI binaries could be execute in AArch32 state on AArch64. A32 THUMB mode is not supported yet.

- cleanup checking address ranges with IN_RANGE macro
- change PM_ADDR_CHECK macro to KASSERTMSG
- restore fast lookup cases with IN_RANGE macro for pmap_extract changed in my previous commit.

rewrite pmap_pte_lookup() to share similar code.

cleanup locore, and changed the way to map memories during boot.
- add functions bootpage_enter() and bootpage_alloc() to adapt various layout
of physical memory map. especially for 64bit physical memory layout.
pmapboot_alloc() allocates pagetable pages from _end[].
- changed to map only the required amount for PA=VA identity mapping
(kernel image, UART device, and FDT blob) with L2_BLOCK(2Mbyte).
- changing page permission for kernel image, and making KSEG mapping are done
at cpu_kernel_vm_init() instead of at locore.
- optimize PTE entries with PTE Contiguous bit. it is enabled on devmap only for now.

reviewed by skrll@, thanks.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

delete debug printf and KASSERT.

Replace KDASSERT by panic.

Rename _pmap_alloc_pdp -> pmap_alloc_pdp, and make it public.

cleanup aarch64 mpstart and fdt bootstrap
* arm_cpu_hatch_arg is a bad idea. avoid serializing CPU startup, and eliminate arm_cpu_hatch_arg.
in mpstart, resolve own cpu index using array of cpu_mpidr[] (aarch64)
* add support fdt enable-method "spin-table"
* add support fdt enable-method "brcm,bcm2836-smp" (for 32bit RaspberryPi)
* use arm_fdt_cpu_bootstrap() instead of psci_fdt_bootstrap()
* rename "arm/fdt/psci_fdt.h" to "arm/fdt/psci_fdtvar.h" because of conflict of include file for needs-flag
* add devmap for cpu spin-table of raspberrypi3/aarch64
* no need to force hatch APs for raspberrypi3/arm32 ifndef MULTIPROCESSOR.
* fix to work pmap_extract(kerneltext/data/bss) even if before calling pmap_bootstrap

idea to use cpu_mpidr[] by jmcneill@. reviewd by skrll@. thanks.

need to add VM_PROT_READ when pmap_kenter_pa(va, pa, VM_PROT_WRITE, 0) or pmap_kenter_pa(va, pa, VM_PROT_EXECUTE, 0).
VM_PROT_READ is treated as an access permission inernally.

change to minimum invalidation of TLB.
specifying not only va but also asid, and not invalidate L0-L2 entry using tlbi_*_ll() if needed.

treat kernel-exec attr and user-exec attr separately.
kernel cannot execute userland exec page, and user cannot execute kernel page.

set kernel text/rodata readonly by default.
add function db_write_text() for setting ddb breakpoint.

Define and use VPRINTF

changes of pmap.c r1.13 seems to be unstable.
In order to invalidate icache, not to invalidate all icache,
but temporary to make the page writable and invalidate target address only.

don't call pool_cache_put with locking pmap. pool_cache_put call pmap_kenter_pa internally.
(pool_cache_put_paddr -> pool_cache_put_slow -> pool_get -> pmap_kenter_pa)

* fix icache invalidations.
* "ic ivau" (aarch64_icache_sync_range) with VA generates permission fault in some situations, therefore use KSEG address for now.

rather than using flags to resolve nested locks, reserve pool_cache before locking.

* avoid deadlock. mutex_owned() works only for adaptive lock, therefore we cannot use it for spinlock...
* add more NULL check
* clear pte when pmap_enter() fails

Use __debugused

Add missing casts, remove unused variables.

need locks in pmap_kremove() and pmap_fault_fixup()

branches: 1.7.2;
pmap_enter() must update modified/referenced flags by 'flags' not 'prot'.

Fix memory leak. it was leaking one page every pmap_create().
pm->pm_vmlist must be initialized before calling _pmap_alloc_pdp().

fix KASSERT panic. pv_entry may not be exists in pvlist when pmap_remove().

delete unused code

fix instability behavior of bufcache on aarch64.
* fix to return correct ref/mod when PMAP_WIRED.
* changed to keep wired flags in pte instead of pv_entry, and cleanup.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Provide a shared pmap_devmap implementation and convert all pmap_devmap
arrays to use DEVMAP_ENTRY{,_END}

Rename VM_KERNEL_IO_ADDRESS to VM_KERNEL_IO_BASE to match RISC-V

It's less letters, matches other similar variables and will help with
sharing code between the two architectures.

NFCI.

aarch64/pmap: Fix criterion in previous.

Use the pte bit that says whether this is a PMAP_WIRED page, not the
bit that says whether this is a non-global page.

(Forgot to git commit --amend before exporting to CVS, sorry!)

aarch64/pmap(9): Teach pmap_protect about pmap_kenter_pa mappings.

Pages mapped with pmap_kenter_pa are necessarily unmanaged, so there
are no P->V records, and pmap_kenter_pa leaves pp->pp_pv.pv_va zero
with no modified/referenced state.

However, pmap_protect erroneously examined pp->pp_pv.pv_va to
ascertain the modified/referenced state -- and if the page was not
marked referenced, pmap_protect would clear the LX_BLKPAG_AF bit
(Access Flag), with the effect that subsequent uses of the page fault
and require a detour through pmap_fault_fixup.

This caused problems for the kernel module loader:

- When loading the text section, kobj_load first allocates kva with
uvm_km_alloc(UVM_KMF_WIRED|UVM_KMF_EXEC), which creates ptes with
pmap_kenter_pa. These ptes are writable, so we can copy the text
section into them, and have LX_BLKPAG_AF set so there will be no
fault when they are used by the kernel.

- But then kobj_affix makes the text section read/execute-only (and
nonwritable) with uvm_km_protect(VM_PROT_READ|VM_PROT_EXECUTE),
which updates the ptes with pmap_protect. This _should_ leave
LX_BLKPAG_AF set, but by inadvertently treating the page as managed
when it should be unmanaged, pmap_protect cleared it instead.

- Most of the time, clearing LX_BLKPAG_AF caused no problem, because
pmap_fault_fixup would silently resolve it. But if a hard
interrupt handler tried to use any page in the module's text (or
rodata, I suspect) that was not yet fixed up, the CPU would fault
and enter pmap_fault_fixup -- which would promptly crash (or hang)
by trying to take the pmap lock in interrupt context, which is
forbidden.

I observed this by loading dtrace.kmod early at boot and trying to
dtrace hard interrupt handlers.

With this change, pmap_protect now recognizes wired mappings (as
created by pmap_kenter_pa) before consulting pp->pp_pv.pv_va, and
preserves then LX_BLKPAG_AF bit in that case.

ok skrll

fix a spello in a comment

Remove some empty lines

Use UVMHIST_CALLARGS in pmap_bootstrap

Only define the EFI variable if EFI_RUNTIME

KNF

Use "non-posted" instead of "strongly ordered" to describe nGnRnE mappings

Rename the following defines:
- _ARM_BUS_SPACE_MAP_STRONGLY_ORDERED to BUS_SPACE_MAP_NONPOSTED
- PMAP_DEV_SO to PMAP_DEV_NP
- LX_BLKPAG_ATTR_DEVICE_MEM_SO to LX_BLKPAG_ATTR_DEVICE_MEM_NP
Rename the following option:
- AARCH64_DEVICE_MEM_STRONGLY_ORDERED to AARCH64_DEVICE_MEM_NONPOSTED

Fixed a bug that pte's __BIT(63,48) could be set when accessing addresses above 0x0001000000000000 in /dev/mem with mmap().

When accessed in mmap by the device pager, pmap_enter() may be called with prot == PROT_WRITE.

Sprinkle some KASSERT(kpreempt_disabled());

since pmap_activate_efirt() rewrites TTBR0, it is necessary to pmap_activate() again after pmap_deactivate_efirt() to restore the original TTBR0.

- Fix to do pmap_{de,}activate() before/after pmap_{,de}activate_efirt().
- moved kpreempt_{disable,enable}() to the caller since everything between
arm_efirt_md_enter() and arm_efirt_md_exit() should be kpreempt disabled.

ok skrll@

Add the missing kpreempt_enable to pmap_deactivate_efirt

No need to flush icache for EFI RT mappings as bootaa64.efi flushed
the full icache for us. (Also this avoids traps)

sys: Use membar_release/acquire around reference drop.

This just goes through my recent reference count membar audit and
changes membar_exit to membar_release and membar_enter to
membar_acquire -- this should make everything cheaper on most CPUs
without hurting correctness, because membar_acquire is generally
cheaper than membar_enter.

Update to support EFI runtime outside the kernel virtual address space
by creating an EFI RT pmap that can be activated / deactivated when
required.

Adds support for EFI RT to ARM_MMU_EXTENDED (ASID) 32-bit Arm machines.

On Arm64 the usage of pmapboot_enter is reduced and the mappings are
created much later in the boot process -- now in cpu_startup_hook.
Backward compatiblity for KVA mapped RT from old bootaa64.efi is
maintained.

Adding support to other platforms should be easier as a result.

Slight code re-organisation. NFCI.

sys: Membar audit around reference count releases.

If two threads are using an object that is freed when the reference
count goes to zero, we need to ensure that all memory operations
related to the object happen before freeing the object.

Using an atomic_dec_uint_nv(&refcnt) == 0 ensures that only one
thread takes responsibility for freeing, but it's not enough to
ensure that the other thread's memory operations happen before the
freeing.

Consider:

Thread A Thread B
obj->foo = 42; obj->baz = 73;
mumble(&obj->bar); grumble(&obj->quux);
/* membar_exit(); */ /* membar_exit(); */
atomic_dec -- not last atomic_dec -- last
/* membar_enter(); */
KASSERT(invariant(obj->foo,
obj->bar));
free_stuff(obj);

The memory barriers ensure that

obj->foo = 42;
mumble(&obj->bar);

in thread A happens before

KASSERT(invariant(obj->foo, obj->bar));
free_stuff(obj);

in thread B. Without them, this ordering is not guaranteed.

So in general it is necessary to do

membar_exit();
if (atomic_dec_uint_nv(&obj->refcnt) != 0)
return;
membar_enter();

to release a reference, for the `last one out hit the lights' style
of reference counting. (This is in contrast to the style where one
thread blocks new references and then waits under a lock for existing
ones to drain with a condvar -- no membar needed thanks to mutex(9).)

I searched for atomic_dec to find all these. Obviously we ought to
have a better abstraction for this because there's so much copypasta.
This is a stop-gap measure to fix actual bugs until we have that. It
would be nice if an abstraction could gracefully handle the different
styles of reference counting in use -- some years ago I drafted an
API for this, but making it cover everything got a little out of hand
(particularly with struct vnode::v_usecount) and I ended up setting
it aside to work on psref/localcount instead for better scalability.

I got bored of adding #ifdef __HAVE_ATOMIC_AS_MEMBAR everywhere, so I
only put it on things that look performance-critical on 5sec review.
We should really adopt membar_enter_preatomic/membar_exit_postatomic
or something (except they are applicable only to atomic r/m/w, not to
atomic_load/store_*, making the naming annoying) and get rid of all
the ifdefs.

Slight comment improvement.

fix various typos, mainly in comments.

add support Hardware updates to Access flag and Dirty state (FEAT_HAFDBS)

- The DBM bit of the PTE is now used to determine if it is writable, and
the AF bit is treated entirely as a reference bit. A valid PTE is always
treated as readable. There can be no valid PTE that is not readable.
- LX_BLKPAG_OS_{READ,WRITE} are used only for debugging purposes,
and has been superseded by LX_BLKPAG_AF and LX_BLKPAG_DBM.
- Improve comment

The need for reference/modify emulation has been eliminated,
and access/permission faults have been reduced, however,
there has been little change in overall performance.

Skip unnecessary PTE operations and TLB invalidation.

aarch64/pmap: remove stray semicolon

No binary change.

The translations that need invalidation are always last level so remove
the (dubious) logic around tracking the level. The "any level" TLB
invalidation maintenance operation are used, but this may change after
further testing.

before
======
1661.0u 420.2s 2:53.82 1197.3% 231+29k 10+33918io 102pf+0w
1646.8u 425.2s 2:52.96 1198.0% 232+29k 1+33937io 49pf+0w
1647.9u 425.7s 2:52.58 1201.6% 232+29k 0+33940io 32pf+0w

After
=====
1602.5u 420.8s 2:49.09 1196.6% 238+30k 24+33893io 54pf+0w
1600.7u 421.3s 2:51.53 1178.8% 238+30k 1+33914io 33pf+0w
1597.5u 424.3s 2:50.46 1186.1% 238+30k 0+33915io 17pf+0w

LGTM from ryo@

Restore the previous pmap_remove_all behaviour as the new method meant
the n1sdp couldn't complete a build.

No noticeable change in kernel build performance.

KNF

s/occured/occurred/ in comments, log messages and man pages.

fix various typos, mainly in comments.

Fix non-UVMHIST build

fix non-MULTIPROCESSOR build

Use sys/uvm/pmap/pmap_tlb.c on Aarch64 in the same way that some Arm, MIPS,
and some PPC kernels do. This removes the limitation of 256 processes on
CPUs with 8bit ASID field, e.g. Apple M1.

Additionally the following changes have been made

- removed a couple of unnecessary aarch64_tlbi_all calls
- removed any invalidation after freeing page tables due to
_pmap_sweep_pdp. This was never necessary afaict.
- all kernel mappings are marked global and userland mapping not-global.

Performance testing hasn't show a significant difference. The data here
is from building a kernel on an lx2k system with nvme.

before
1489.6u 400.4s 2:40.65 1176.5% 228+224k 0+32289io 57pf+0w
1482.6u 403.2s 2:38.49 1189.9% 228+222k 0+32274io 46pf+0w
1485.4u 402.2s 2:37.27 1200.2% 228+222k 0+32275io 12pf+0w

after
1493.9u 404.6s 2:37.50 1205.4% 227+221k 0+32265io 48pf+0w
1485.0u 408.0s 2:38.54 1194.0% 227+222k 0+32272io 36pf+0w
1484.3u 407.0s 2:35.88 1213.3% 228+224k 0+32268io 14pf+0w

>>> stats.ttest_ind([160.65,158.49,157.27], [157.5,158.54,155.88])
Ttest_indResult(statistic=1.1923622711296888, pvalue=0.2990182944606766)
>>>

Ensure TCR_EPD0 is set on entry to pmap_activate and ensure it is set as
early as possible for APs.

Use UVMHIST_CALLARGS

'\n' is not required in KASSERTMSG either.

"\n" is not required in KERNHIST

Use __SHIFTIN. Same code before and after.

pmap_page_remove: simply and reduce the code size slightly.

In pmap_icache_sync_range change

for (...) {
...
if (condition) {
// do stuff
}
}

to

for (...) {
...
if (!conditional)
continue;
// do stuff
}

to save on indentation. Same code (modulo register usage) before and
after.

KNF

Deal with the pmap limitation of maxproc in a more complete way and
recognise CPUs with only 8bit ASIDs.

branches: 1.107.2;
Make the ddb for pmap / pte information pmap agnostic

Remove some unnecessary tlb invalidate in pmap_growkernel and ASAN shadow
map. Ensure the shadow map mappings are visible to the TLB walkers.

branches: 1.105.2;
added more attributes of PTE displayed by "ddb>machine pte"

remove KERNHIST_INIT_STATIC(). it stradles the line between usable
early in boot and broken early in boot by requiring a partly static
structure with another structure that must be present by the time
any uses are performed. theoretically platform code could allocate
a chunk while seting up memory and assign it here, giving a dynamic
sizing for the entry list, but the reality is that all users have
a statically allocated entry list as well.

the existing KERNHIST_LINK_STATIC() is used in conjunction with
KERNHIST_INITIALIZER() instead.

this stops a NULL pointer deref when the _LOG() macro is called
before the storage is linked in, which happens with GCC 10 on OCTEON
with UVMHIST enabled, crashing in very early kernel init.

branches: 1.103.2;
fix build error without options DDB.

kvtopte() is referenced from arm/acpi/acpi_machdep.c

No assignment is needed here.

the loop in pmap_page_remove() always removes the first pv,
and since the list is managed by _pmap_remove_pv(), pp->pp_pv.pv_next always points to the first.

It is enough to make a page accessible instead of writable.
same fix as r1.76

implement pmap_remove_all().

The size of struct pv_entry has increased, but speed of kernel build has improved by about 1%
exec and exit should have been improved.

Improve the English in the previous comment fix.

Tweak a comment

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

AA64 is not MIPS.

Change all KSEG references to directmap

In pmap_devmap_bootstrap only set pmap_devmap_bootstrap_done if there
is an entry and ALL of the entries have been done. The entry required
for EARLYCONS might not be the first/only one...

branches: 1.94.2;
No need to disable translation table walks in pmap_activate().

Use the isb macro - missed in previous commit

Use the dmb/dsb/isb macros... if nothing else they're all now consistent
about the "memory" assembler contraint.

No binary change

Only set pmap_devmap_bootstrap_done if something gets mapped.

Think acpi_platform_devmap

Make __md_palloc pmap agnostic (think sys/uvm/pmap)

PID_MAX is just an initial value (soft maximum). Don't use it for CTASSERT.
defined __HAVE_CPU_MAXPROC to use function cpu_maxproc().

pointed out by mrg@, thanks.

Fix panic caused by modload. http://mail-index.netbsd.org/port-arm/2020/08/30/msg006960.html

The address space reserved for modules may not be mapped in L1-L3.

Whitespace

Part IV of ad's performance improvements for aarch64

- Implement pmap_growkernel(), and update kernel pmap's stats with atomics.

- Then, pmap_kenter_pa() and pmap_kremove() no longer need to allocate
memory nor take pm_lock, because they only modify L3 PTEs.

- Then, pm_lock and pp_lock can be adaptive mutexes at IPL_NONE which are
cheaper than spin mutexes.

- Take the pmap's lock in pmap_extract() if not the kernel's pmap, otherwise
pmap_extract() might see inconsistent state.

Fix another UVMHIST so it doesn't use %s

pmapboot_enter simplication
- bootpage_alloc in asm becomes pmapboot_pagealloc in C
- PMAPBOOT_ENTER_NOBLOCK is removed as it's not used
- PMAPBOOT_ENTER_NOOVERWRITE is removed as it's now always on
- physpage_allocator argument is removed as it's always
pmapboot_pagealloc
- Support for EARLYCONS without CONSADDR is removed so that the identity
map for CONSADDR is always known.

For the assembly files:
2 files changed, 40 insertions(+), 89 deletions(-)

LGTM ryo

Use tlen for temporary length variable instead of l, which is usually
used for struct lwp *.

No binary changes.

pmap_procwr(): sync icache even if p != curproc. This fixes applications
like GDB for arm32, that rewrite text of other process.

Thanks to ryo@ for discussion.

Set uvmexp.ncolors appropriately, which is required for some CPU
models with VIPT icache.

Otherwise, alias in virtual address results in inconsistent results,
at least for applications that rewrite text of other process, e.g.,
GDB for arm32.

Also, this hopefully fixes other unexpected failures due to alias.

Confirmed that there's no observable regression in performance;
difference in ``time make -j8'' for GENERIC64 kernel on BCM2837
with and without setting uvmexp.ncolors is within 0.1%.

Thanks to ryo@ for discussion.

Fix typo in name of evcnt(4) counter.

Fix bug with incorrect range calculation when doing icache sync.
This is called by sysarch(ARM_SYNC_ICACHE) from aarch32 (compat_netbsd32) emul process.

pointed out by rin@, thanks.

XXX pullup-9

- Fix a lock order reversal in pmap_page_protect().

- Make sure pmap is always locked when updating stats; atomics no longer
needed to do that.

- Remove unneeded traversal of pv list in pmap_enter_pv().

- Shrink struct vm_page from 136 to 128 bytes (cache line sized) and struct
pv_entry from 48 to 32 bytes (power of 2 sized).

- Embed a pv_entry in each vm_page. This means PV entries don't need to
be allocated for private anonymous memory / COW pages / most UBC mappings.
Dynamic PV entries are then used only for stuff like shared libraries and
shared memory.

Proposed on port-arm@.

- Wired/resident stats shouldn't covered by PMAPCOUNTERS.
- Rename need_update_pv -> need_enter_pv.

Ok ryo@

no need to make the PTE writable to do icache_sync, enough to accessible.

Use __diagused

fix non-diag build

Use MUTEX_NODEBUG for PV locks as is commonly done. OK ryo.

Implement pmap_extract_coherency

PMAP_DEBUG has been deleted on arm

Add support for Branch Target Identification (BTI).

On the executable pages that have the GP (Guarded Page) bit, the semantic
of the "br" and "blr" instructions is changed: the CPU expects the first
instruction of the jump/call target to be "bti", and faults if it isn't.

We add the GP bit on the kernel .text pages (and incidentally the .rodata
pages, but we don't care). The compiler adds a "bti c" instruction at the
beginning of each C function. We modify the ENTRY() macros to manually add
"bti c" in the asm functions.

cpuswitch.S needs a specific change: with "br x27" the CPU expects "bti j",
which is bad because the functions begin with "bti c"; switch to "br x16",
for the CPU to accept "bti c".

BTI helps defend against JOP/COP. Tested on Qemu.

branches: 1.69.2;
use PMAP_PAGE_INIT() to initialize mutex in pmap_page.

VM_MDPAGE_INIT() in pmap_free_pdp() had initialized pp_flags,
so it unintentionally cleared PMAP_PAGE_FLAGS_PV_TRACKED.
use PMAP_PAGE_INIT to avoid using PMAP_PAGE_FLAGS_PV_TRACKED.

pointed out by tnn@, thanks

pmap_remove_all(): Return a boolean value to indicate the behaviour. If
true, all mappings have been removed, the pmap is totally cleared out, and
UVM can then avoid doing the work to call pmap_remove() for each map entry.
If false, either nothing has been done, or some helpful arch-specific voodoo
has taken place.

oops, fix incorrect usage of daif_enable() in my previous commit.

Fix pmap to work correctly with tagged addresses

- when fault, untag from address before passing to uvm/pmap functions
- pmap_extract() checks more strictly and consider the address tag

use pmapboot_enter_range()

use LIST(3) instead of TAILQ(3) to save one word in struct vm_page and struct pmap.

pointed out by riastradh@. thanks

add support pmap_pv(9)

Patch originally from jmcneill@. thanks

separate struct vm_page_md into vm_page_md and pmap_page
for preparation pmap_pv(9)

fix behaviour mmap()/mprotect() when passed only PROT_EXEC.

when mmap()/mprotect() with only PROT_EXEC, syscall will be successful,
but the page actually hadn't been mapped.
it should be mapped with PROT_READ|PROT_EXEC implicitly. (r-x)

branches: 1.60.2;
Update pmap_map_chunk to allow L[12] block mappings and L3 page mappings

Remove unnecessary brackets and unwrap a conditional. Same code before
and after.

Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

G/C kasan_shadow_map call in pmap_enter

pmap_growkernel calls kasan_shadow_map for KVA

atomic_add_16() is not used in pmap.c anymore. no need decl here.

space to tab

Fix build... wire_count probably doesn't need atomics

Fix KASAN support by calling kasan_shadow_map in pmap_growkernel

pg->phys_addr -> VM_PAGE_TO_PHYS(pg)

Mark several kASan functions with __nothing, to avoid annoying #ifdefs.
Same as kCSan and kMSan.

Don't include "opt_kasan.h" when there's already <sys/asan.h> included.

Define PMAP_NEED_PROCWR, providing strategically placed i-cache
synchronization where just-changed memory is about to be executed.

Fixes SIGILLs seen when running Mono 6 on QEMU Cortex-A57.

ok ryo

Disable translation table walks using TTBR0 while changing its value and
when deactivating a pmap. Fixes stability issues on Ampere eMAG CPUs.

ref/mod bit should be set according to 'flags' argument, not 'prot'. r1.44 was incomplete.

In pmap_devmap_bootstrap(), cpu_earlydevice_va_p() must not return true until *all* devmap tables have been enabled.
console mapping may be present in the last table.

- remove incorrect KASSERT. mmap(2) with prot=PROT_WRITE calls pmap_enter(..., PROT_WRITE) internally.
- fix to update page reference flags when only PROT_WRITE or PROT_EXECUTE specified

Make pmap_db_pte_print more terse so it's quicker on serial consoles

Use PMAP_DEV in DEVMAP_ENTRY rather than pmap_map_chunk. It's clearer and
means pmap_map_chunk can be made to map other memory types.

branches: 1.41.2;
apply some __diagused.

- free empty page tables pages if reach a certain usage.
- need to lock at removing an old pg (_pmap_remove_pv) in _pmap_enter()

Fix race conditions about pmap_page_protect() and pmap_enter().

while handling same PTE by these functions in same time, there
is a critical path that the number of valid PTEs and wire_count
are inconsistent, and it caused KASSERT.
Need to hold a pv_lock while modifying them.

spinkle __printflike(), and use PRIxxx

- add ddb command "machine ttbr" to dump MMU tables.
- tidy up descriptions, usages and messages.

- free L1-L3 pages that has been emptied by pmap_remove().
- if no memories, pmap_enter will return correctly ENOMEM if PMAP_CANFAIL, or wait until available any memories if !PMAP_CANFAIL.

These changes improves the stability when we use a huge virtual memory spaces with mmap.

improve pmap_remove
- don't lock/unlock per page in pmap_remove()
- speedup pte lookup for continuous addresses
- bring out pool_cache_put(&_pmap_pv_pool, pv) from lock/unlock section

- add workaround for Cavium ThunderX errata 27456.
- add cpufuncs table in cpu_info. each cpu clusters may have different erratum. (e.g. big.LITTLE)

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

invalidate icache correctly.
l3pte_executable() should be used for only valid pte.

Provide generic start code that assumes the MMU is off and caches are
disabled as per the linux booting protocol for ARMv6 and ARMv7 boards.
u-boot image type should be changed to 'linux' for correct behaviour.

The new start code builds a minimal "bootstrap" L1PT with cached access
disabled and uses the same table for all processors. AP startup is
performed in less steps and more code is written in C.

The bootstrap tables and stack are placed into an (orphaned) section
"_init_memory" which is given to uvm when it is no longer used.

Various kernels have been converted to use this code and tested. Some
boards were provided by TNF. Thanks!

The GENERIC kernel now boots on boards using the TEGRA, SUNXI and EXYNOS
kernels. The GENERIC kernel will also work on RPI2 using u-boot.

Thanks to martin@ and aymeric@ for testing on parallella and nanosoc
respectively