remove prototypes with no matching function and externs with no var

Initialize earlier macppc's and powerpc64's pmap_hash_lock

At boot, the powerpc64 kernel was calling
pmap_bootstrap -> pmap_kenter_pa -> mtx_enter(&pmap_hash_lock)
before it did
pmap_init -> mtx_init(&pmap_hash_lock, IPL_HIGH)

Change from mtx_init to MUTEX_INITIALIZER. This allows an option
WITNESS kernel to boot without warning of an uninitialized mutex.

Also change macppc's pmap_hash_lock from __ppc_lock_init to
PPC_LOCK_INITIALIZER, though WITNESS doesn't see this lock.

ok mpi@

Set the current pmap in macppc's pmap_activate

This fixes a possible freeze in execve(2). It sometimes froze when a
dual-cpu macppc started daemons during boot. There is a chance that
uvm_map.c uvmspace_exec sees ovm->vm_refcnt != 1 and switches curproc
to a new pmap. If this happened, then execve froze by trying to
copyout to the wrong pmap; curpcb->pcb_pm was old. Fix by setting
pointers when uvmspace_exec calls pmap_activate.

ok miod@

Add missing check for pg != NULL

The code was reading pg->pg_flags, so clang assumed pg != NULL, then
optimized a later "if (pg != NULL)" to "if (1)", and allowed a call to
pmap_enter_pv(pted, NULL). Such a call can freeze bsd.mp by trying to
lock NULL's ((struct mutex *)0x3c). I froze bsd.mp this way by
starting Xorg on a macppc with nv(4) or r128(4) video, as it tried to
mmap the xf86(4) aperture.

ok miod@

On systems without xonly mmu hardware-enforcement, we can still mitigate
against classic BROP with a range-checking wrapper in front of copyin() and
copyinstr() which ensures the userland source doesn't overlap the main program
text, ld.so text, signal tramp text (it's mapping is hard to distinguish
so it comes along for the ride), or libc.so text. ld.so tells the kernel
libc.so text range with msyscall(2). The range checking for 2-4 elements is
done without locking (because all 4 ranges are immutable!) and is inexpensive.

write(sock, &open, 400) now fails with EFAULT. No programs have been
discovered which require reading their own text segments with a system call.

On a machine without mmu enforcement, a test program reports the following:
userland kernel
ld.so readable unreadable
mmap xz unreadable unreadable
mmap x readable readable
mmap nrx readable readable
mmap nwx readable readable
mmap xnwx readable readable
main readable unreadable
libc unmapped? readable unreadable
libc mapped readable unreadable

ok kettenis, additional help from miod

Execute-only for macppc G5

The G5 PowerPC 970 has a Data Address Compare mechanism that can trap
loads and stores to pages with PTE_AC_64, while allowing instruction
fetches. Use this for execute-only mappings, like we do on powerpc64.

Add a check to pte_spill_v for execute-only mappings. Without this,
we would forever retry reading an execute-only page.

In altivec_assist, copyin would fail to read the instruction from an
execute-only page. Add copyinsn to bypass x-only, like sparc64.

with help from abieber@ deraadt@ kettenis@
ok deraadt@

Use atomic ops on the set of used segment registers

Each pmap sets a bit in usedsr to claim 16 unique VSIDs for its
segment registers. Use atomic_cas_uint to set this bit (checking that
the other cpu didn't steal it) and atomic_clearbits_int to clear it.
Stop using splvm.

ok miod@

Remove pmap_collect() when a no-op, define __HAVE_PMAP_COLLECT otherwise.
Use that define to shunt uvm_swapout_threads(), which is a noop when
pmap_collect() does nothing.

ok mpi@

Allow writes to rw pages in pte_spill_v

In the powerpc pmap, hash collisions can spill page table entries.
Page faults can use pte_spill_v to reinsert a spilled pte. If the
fault is a write (DSISR_STORE), then pte_spill_v tries to check for a
read-only page. The existing check (pte_lo & PTE_RO_64) also matched
rw pages, because PTE_RO_64 is 3 and PTE_RW_64 is 2. This caused
pte_spill_v to deny writes to rw pages. Then uvm_fault might allow
the write; but uvm_fault can't handle some pages in the kernel. Such
faults caused, "panic: uvm_fault: fault on non-pageable map", or
"panic: trap type 300".

Change it to ((pte_lo & PTE_PP_64) == PTE_RO_64). This seems to fix
one reason why bsd.mp on a macppc dual G5 might panic.

ok kettenis@ miod@

Don't put an extern variable (ppc_kvm_stolen) into vmparam.h, other instances
of this file are only doing cpp #define

spelling

pmap_avail_setup() is the only place physmem is calculated, delete a bunch
of code which thinks it could be done elsewhere.
ok kurt

Switch powerpc to MI mplock implementation.

Reduce differences with others architectures and make it possible to use
WITNESS on it.

Rename & keep the current recursive lock implementation as it is used by
the pmap.

Tested by Peter J. Philipp, otto@ and cwen@.

ok kettenis@

Prepare the bat for kernels greater > 8MB of code, why because clang.
ok kettenis

some cleanup for clang; ok kettenis

In pmap_page_protect(), zap the PTE before unlinking. At that point the
PTED_VA_MANAGED_M flag is still set so proper MOD/REF accounting will
happen. Fixes memory corruption that would invariably happen when a
machine started swapping.

Giant cluestick from George Koehler.
ok visa@, mpi@

More "explicitely" -> "explicitly" in various comments.

ok guenther@ tb@ deraadt@

Implement copyin32(9).

ok mpi@, visa@

Change pmap_proc_iflush() to take a process instead of a proc
powerpc: rename second argument of pmap_proc_iflush() to match other archs

ok kettenis@

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

consistently set ipls on pmap pools.

this is a step toward making ipls unconditionaly on pools.

ok deraadt@ kettenis@

Add a per-page flag to indicate that all mappings of that page should be
uncached. To be used in the drm code.

ok mpi@

Make the powerpc pmap (more) mpsafe by protecting both the pmap itself and the
pv lists with a mutex. This should make pmap_enter(9), pmap_remove(9) and
pmap_page_protect(9) safe to use without holding the kernel lock.

ok visa@, mpi@, deraadt@

Give the pool page allocator backends more sensible names. We now have:
* pool_allocator_single: single page allocator, always interrupt safe
* pool_allocator_multi: multi-page allocator, interrupt safe
* pool_allocator_multi_ni: multi-page allocator, not interrupt-safe

ok deraadt@, dlg@

Make pmap_remove() grab the kernel lock. This is a big hammer but makes MP
machines work again with the unlocked reaper.

ok mpi@, deraadt@
no objection from miod@

Finally protect VP lookups to guarantee that a pted won't be freed or
reused by a CPU while another CPU is manipulating it.

This races occurs because the virtual spill handlers are run without
taking the KERNEL_LOCK for obvious reasons. So use a per-pmap mutex
that CPUs must hold when modifying a pted in order to guarantee the
atomicity of operations *and* the coherence between pmap VPs tree and
what's in the HASH.

Thanks to dlg@ for assisting me debugging this. This change ends your
PowerPC pmap SMP show of the week. GENERIC.MP on macppc should now be
stable enough to build ports without corrupting its own memory.

ok kettenis@, deraadt@, dlg@

Don't try to be clever when unrolling the loop in pmap_remove().

Needed for upcoming locking.

Replace the per-entry locks by a global HASH lock.

Since this lock is recursive we can now guarantee the atomicity of
pte_inser{32,64}() when a pted has to be removed first. This fixes
one of the races.

Using a __mp_lock here also allowed dlg@ to provide me useful traces
to fix the next race. Thanks for your help!

ok kettenis@, deraadt@, dlg@

Call pte_spill_v() from the real mode fault handler instead of rerolling
it. This will reduce the number of places to audit for locking.

Note that for profiling purposes pte_spill_v() is now marked a __noprof
since per-CPU profiling buffers are not guaranteed to be 1:1 mapped and
cannot be accessed from the real mode fault handler.

ok kettenis@, deraadt@, dlg@

Rewrite PTE manipulation routines to better match the PEM.

Document every operation, make sure to call "sync" when appropriate so
that other CPUs see the bit changes and finally grab a lock where it was
missing to grantee atomicity.

ok kettenis@, deraadt@, dlg@

Split pteclrbits() into pmap_{test,clear}_attrs().

This should not introduce any behavior change but makes the code easier
to read and later easier to protect. This also brings this pmap closer
to what others do.

Thanks to kettenis@ for spotting a bad typo!

ok kettenis@, deraadt@, dlg@

More usages of pmap_ptedinhash().

If you wonder why pte_insert{32,64}() is not using pmap_hash_remove() if
it finds a conflicting PTE in the HASH, it's because in the current state
trying to grab the same lock a second time would lead to a deadlock.

This is much easier to reproduce on G5 (or G4 with BAT disabled).

ok kettenis@, deraadt@, dlg@

Remove DEBUG stuff.

Make use of ptesr() instead of rerolling it.

Merge various copies of the same code into a new function to determine
if a PTE is present in the HASH.

Note that atomicity is currently not guaranteed between this check and
the following operations.

ok kettenis@, deraadt@, dlg@

Introduce pmap_pted_ro() a simple wrapper for the 32/64 bits versions
that does not call pmap_vp_lookup().

Carreful readers would have notice the removal of the bits on the virtual
address with a page mask, this change allows me to find the 13 years old
bug fixed in r1.145.

ok kettenis@, deraadt@, dlg@

Do only one VP lookup when removing a page.

This simplify pmap_remove() & friends by re-using an already fetched PTE
descriptor.

There's currently a race on MP system where one CPU can reuse a pted
while another one is still trying to insert it in the HASH. This commit
starts reducing the number of pmap_vp_lookup() calls to help fix this
race.

ok kettenis@, deraadt@, dlg@

Remove the MANAGED flag when removing a PV entry.

Even if this change is not strickly needed, because the memory will be
returned to the pool it helped me track the use-after-free.

Remove unneeded splvm() calls and the pool_setipl(9) hack of r1.140.

By instructing spl(9) calls on MP machines I figured out that their high
cost was hiding a race condition involving PTE reuse in our pmap. Thanks
to deraadt@ for finding a way to trigger such panic by adding a couple of
splvm().

This should make the races easier to trigger but will be addressed
shortly.

This commit starts your PowerPC pmap SMP show of the week.

ok kettenis@, deraadt@, dlg@

Fix 13 years old typo that should be responsible for the unhappiness
of UVM on PowerPC architectures by breaking pmap_is_referenced() and
friends.

ok kettenis@

Make it possisble to disable block address translation mechanism on
processors that support it.

Due to the way trap code is patched it is currently not possible to
enabled/disable BAT at runtime.

ok miod@, kettenis@

Merge two versions of ppc_check_procid().

ok miod@, kettenis@ as part of a larger diff

oops, accidental commit

sync

pool_setipl() on both pmap pools as a workaround for some sort of MP
race. This will certainly be revisited, but too much time has been
spent on it for now.
ok mpi

Let powerpc's bus_space(9) use the same pmap and uvm interfaces than the
other archs.

Specify the caching policy by passing PMAP_* flags to pmap_kenter_pa()
like the majority of our archs do and kill pmap_kenter_cache().

Spread some pmap_update() along the way.

While here remove the unused flag argument from pmap_fill_pte().

Finally convert the bus map/unmap functions to km_alloc/free() instead
of uvm_km_valloc/free().

Inputs from kettenis@ and miod@, ok miod@

Even without BATs memory under ``physmaxaddr'' is mapped 1:1 in the
kernel, so update pmap_extract() accordingly and save a VP lookup.

While here unify pted checks after the VP lookups.

ok miod@

Various cleanups. Explicitly include <sys/atomic.h>, Use pmap_remove_pg()
for the kernel pmap and kill pmap_kremove_pg(). Finally guard the hash
lock code under "MULTIPROCESSOR" to explicit which part of the code
received some MP love.

ok kettenis@

force the pool of pmapvp onto PAGE_SIZE allocations by specifying a
pool allocator. pmapvp is 1024 bytes, and the size * 8 change in pools
without an allocator being specified tries to place it on large pages.
you need pmap to use large pages, and pmap isnt set up yet.

fixed a very early fault on macppc.
debugged with and tested by krw@
ok deraadt@ krw@

remove simplelocks use
ok kettenis mpi

Speed up page zeroing by using a loop of dcbz/dcbzl instead of bzero().

While here, use the direct map for pmap_copy_page() and remove the now
unused stolen page addresses.

No objection from the usual suspects, "it works, commit" deraadt@

make pmap_zero_page MP-safe, by using the directmap
mpi will investigate speedups after this.
ok mpi kettenis

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

Only mark segment 0 as executable on 64-bit systems. There it is harmless as
we have a proper X bit in the page tables. On 32-bit systems kernel .text is
handled by an IBAT, so we don't need page table entries that are executable
in the kernel pmap.

ok mpi@

Remove execute permission from most pages in the kernel pmap. This is a first
step towards W^X in the kernel, even though it is only effective on machines
with a G5 processor.

ok mpi@

Format string fixes and removal of -Wno-format for *ppc kernels.

Allow to compile with DEBUG_PMAP defined.

Remove the almost unused abstraction around "struct firmware" and use
instead a single function ppc_mem_regions() required by the ppc pmap.

ok kettenis@

Including <uvm/uvm_extern.h> is enough, no need for <uvm/uvm.h> or more.

Use syncicache() instead of rerolling an almost identical version.

ok miod@

Do not bzero() the available memory in pmap_bootstrap(); allocations in
pmap_bootstrap explicitely bzero them, and there is no need to clear the
remaining memory.

ok mpi@

Some (if not all) G5 systems use a different layout for the physical memory
information (property `reg' of the `/memory' node). Fortunately the available
physical memory information still uses the same format, so this only affects
the computation of physmem.

Detect this case and parse the information correctly, converting to the format
expected by pmap, ignoring physical memory beyond 4GB.

Compute physmem from all the physical memory information, even memory not
usable by the kernel. Let pmap not recompute physmem in pmap_bootstrap() if
physmem is != 0 upon entry.

This should allow G5 systems fitted with more than 2GB of physical memory to
report the correct amount of memory, even though the kernel will only use
the lower 2GB.

Prompted by a dmesg@ submission by Greg Marsh, owner of a 3.5GB G5

help and tweaks kettenis@, ok mpi@

Remove excessive parentheses.

pmap.c:1061:13: error: equality comparison with extraneous parentheses [-Werror,-Wparentheses-equality]

ok mpi@

Mark all the C functions called in real mode as non instrumented and
remove the check for address relocation from MCOUNT_ENTER.

This fix kernel profiling on powerpc architectures, broken since the
buffers are per cpu.

ok miod@

Managed device mappings should be uncached by default.

ok mpi@

Add the possibility to map DMA memory non-cached, based on the i386/amd64
implementation. For the moment only the BUS_DMA_NOCACHE macro is required
to build drm on macppc but it will be used soon.

ok kettenis@

Remove the freelist member from vm_physseg

The new world order of pmemrange makes this data completely redundant
(being dealt with by the pmemrange constraints instead). Remove all code
that messes with the freelist.

While touching every caller of uvm_page_physload() anyway, add the flags
argument to all callers (all but one is 0 and that one already used
PHYSLOAD_DEVICE) and remove the macro magic to allow callers to continue
without it.

Should shrink the code a bit, as well.

matthew@ pointed out some mistakes i'd made.
``freelist death, I like. Ok.' ariane@
`I agree with the general direction, go ahead and i'll fix any fallout
shortly'' miod@ (68k 88k and vax i could not check would build)

No "\n" needed at the end of panic() strings.

Bogus chunks pointed out by matthew@ and miod@. No cookies for
marco@ and jasper@.

ok deraadt@ miod@ matthew@ jasper@ macro@

We never create or destroy pmaps from interrupt context, so wrapping the
associated pool calls in splvm()/splx() is unnecessary and confusing.

ok deraadt@, drahn@

Don't #include <sys/user.h> into files that don't need the stuff
it defines. In some cases, this means pulling in uvm.h or pcb.h
instead, but most of the inclusions were just noise. Tested on
alpha, amd64, armish, hppa, i386, macpcc, sgi, sparc64, and vax,
mostly by krw and naddy.
ok krw@

There is no reason to protect the pmap_vp_pool with splvm(). The only pmap
that gets manipulated in interrupt context is the kernel pmap, and we fully
populate its VP mappings during pmap_bootstrap(). Gets rid of the excessive
spl's at pmap_destroy() time noticed by deraadt@

ok deraadt@, drahn@

two missing splx in error path; ok drahn

Prevent an interrupt from causing recursion while holding the pmap hash lock,
Otherwise a pmap_remove from a completed I/O may deadlock.

fix an ugly construct

Clear the PG_PMAP_EXE flags whenever writable mappings are created.
ok deraadt@ kettenis@

More carefully manage PG_PMAP_EXE bit and cache flushing on pmap_protect
operations, where X or W is taken away. ok deraadt@ kettenis@

Make pmap_enter respect the PMAP_CANFAIL flag. With and essential
memory leak plug from drahn@

Handle pool allocation failures slightly better. ok kettenis@

Kernel map is supposed to only allocate from the limited kernel addresses,
panic if the kernel attempts to map an improper address.

A bunch of pool_get() + bzero() -> pool_get(..., .. | PR_ZERO)
conversions that should shave a few bytes off the kernel.

ok henning, krw, jsing, oga, miod, and thib (``even though i usually prefer
FOO|BAR''; thanks for looking.

Changes to get closer to SMP.
add biglock before interrupt calls into the kernel.
switch the clock to using cpuinfo variables instead of globals
move cpu_switchto into C code so that on multiprocessor the FPU
and Altivec can be saved before switching CPUs.
add a lock into pmap when modifying the hash table.

replace even more ctob/btoc with ptoa/atop

[fF]uther -> [fF]urther in comments and man page. First one spotted on
tech@ by Jung.

Move powerpc to vm_page_md, 'throw it in' kettenis@

Ansi prototypes, not K&R. no binary difference.

Implement pmap_steal_memory() on powerpc. With some help from art@.

Relax the cache flags logic in pmap_kenter_pa, to make sure that mappings
entered before vm_physmem[] are initialized will be cached. This is a
temporary measure until this pmap implements pmap_steal_memory().

Help and ok drahn@

Dont pass seemingly random numbers down as the flag
parameter of pool_init()

ok drahn@
no objections miod@

W^X for G5
ok drahn@, totally groovy deraadt@

Get rid of deprecated vm_{offset,size}_t types for good, use {p,v}{addr,size}_t
instead; looked at millert@

splimp -> splvm

ok drahn@

Nearly functional crashdump support for macppc. Because savecore
does not recognize the resulting crashdumps, the writing has been disabled.
Better here than in my forest of trees.

64 bit compat. clean up the 32/64 code paths so less decision points exist
allow more than 256MB ram on G5, (still 2G limit) by creating PTE entries
dynamically for all physical memory.

Handle segment register restore at context enter/exit instead of
deep in the kernel. Based on code from two years ago, now necessary
for G5. removes the 1GB ram limit on 32bit processors, temporarily
sets a 256MB limit on G5.

G5 pmap support, most of this G5 work has been done by kettenis@
without his forging ahead, it would barely be started.
Again this is one step of many, but needs to be tested, this is
independant of the locore change just committed which kettenis@ and
deraadt@ significantly wrote.

Avoid infite loop.
ok drahn@

Do a better job at containing powerpc specific #defines to PPC_...
ok deraadt@

Properly flush instruction cache for ptrace(PT_WRTIE_{DI}, ...) on powerpc
and m68k.
ok drahn@, millert@

Various typos in comments.

backout segment register restore diff which causes reproducible hangs; ok deraadt

Pass -Wformat

Fix ppc segment register restores, this fixes the 1GB ram limit and
cleans up pieces in the pmap code.
tested otto, brad, miod, pval.

Reduce the amount of asm code in powerpc/macppc by replacing it with
inlined functions, helps improve readability and fix a couple of bugs.
ok miod@

kill clause 3 and 4 from several of my copyrights, cleanup.

Remove an unnecessary structure copy from useage of setfault(), call
by reference, not by value, ok matthieu#, miod@

Track if a physical page has been previously mapped executable. If it
has not been previously mapped EXE, flush it. If a writeable mapping
which is not executable occurs for the page, clear this bit.
Solves a problem where an executable page is double mapped, first without
EXE then accessed for execute at a different physical page, the cache
will behave properly.

Eliminate the use of KERN_SUCCESS outside of uvm/

Also uvm_map returns KERN_* codes that are directly mapped to
errnos, so we can return them instead of doing some attempt to
translation.

drahn@ "I see no problem" pval@ "makes sense"

Remove more '\n's from panic() statements. From Chris Kuethe.

backout premature

KNF

Change the pmap_zero_page and pmap_copy_page API to take the struct vm_page *
instead of the pa. Most callers already had it handy and those who didn't
only called it for managed pages and were outside time-critical code.

This will allow us to make those functions clean and fast on sparc and
sparc64 letting us to avoid unnecessary cache flushes.

deraadt@ miod@ drahn@ ok.

- change pte_spill_X() to take an extra parameter to determine if
the fault is a EXE fault or R/W fault.

- mask/or the SR_NOEXEC bit into the segment register value
when the number of executable pages becomes 0/non-zero.

- create segments with SR_NOEXEC set, will be cleared when first
exec mapping in the segment is created.

- allow pte_spill_X() to deal with a new type of fault, page mapped
but non executable, when execute was requested.

Adds up to - non-exec stack support for powerpc.

Perform accounting for executable pages on powerpc, prepare for
non-executeable stack.

Cleanup: use less _t typedefs, use the structure itself.

pmap_t is the exception, it is required by the MI code so pmap_t will
be used instead of using 'struct pmap *' in the code. (consistency)

argh, no last minute changes...

pmap cleanup and KNF.

This doesn't happen, and Debugger should not be used here anyway...

spelling; raj@cerias.purdue.edu

Optimize pmap_remove(). It frequently is called with no mappings to
remove eg mmap() ok miod@ art@

Attribute table must be allocated 1-1 because it is accessed from
pte_spill_r(). ok art@

Remove dead code, ifdef code which should be dead, KNF. Cleanup.

Be more consistant about pted zeroing (the whole structure)
Fix missing 'attr' initialization.
Zero available memory.
Raise available memory limit from 256MB to 1GB. This code has only been
tested up to 512MB, but should be fine to 1GB. Ram modules are not avail
to the developers to test out the machines up to their 1.5GB physical limit.

Do not include headers twice. Pointed out by Dries Schellekens.

Final __P removal plus some cosmetic fixups

Complete rewrite of the powerpc pmap handling, Instead of keeping
the spill list for each PTEG, the V->P translations are stored in
trees for each pmap. All valid kernel mappings are preallocated
in 1-1 memory so that tlb spill/loads for kernel accesses can be
looked up while physical, user mappings are not guaranteed to
be 1-1 mapped, thus the kernel must go virtual to look up user
mappings. While this is more expensive, the tree search is much
lower cost than the long linked list search. Also on each pmap_remove()
it was necessary to search the linked lists for each possible mapping,
now it just looks up the entry in the tree.
This change gives a 25-36% speedup in 'make build' time. What was
around 2:50 is now around 1:55 on a 733MHz G4.

This change causes a likely existing bug to appear quite often,
it deals with the segment register invalidation in kernel mode.
Because of that problem, currently this change limits the physical
memory used to 256MB. This limitation will be fixed soon, it is not
an error in the pmap code.

* Effort sponsored in part by the Defense Advanced Research Projects
* Agency (DARPA) and Air Force Research Laboratory, Air Force
* Materiel Command, USAF, under agreement number F30602-01-2-0537.

Move the message buffer out of low memory, Openfirmware clears the area
on reboot. perhaps OF uses it at other times?
Since OF always use the same memory addresses, this should always allocate
the same ram to the msgbuf, and allow it to be preserved across reboot.

Instead of finding pages matching this physical page, match this specific
entry. Also terminate the search as soon as this entry is found.

How did this work? It used to attempt the tlb entry for a mapping it is
removing by using the va it is replacing it with, NO!.
Calculate the va of the mapping by inverting the pte_hi calculation
producing bits 4-19 of the address. This is enough to correctly invalidate
the tlb entry for the mapping being removed.

Pool deals fairly well with physical memory shortage, but it doesn't deal
well (not at all) with shortages of the vm_map where the pages are mapped
(usually kmem_map).

Try to deal with it:
- group all information the backend allocator for a pool in a separate
struct. The pool will only have a pointer to that struct.
- change the pool_init API to reflect that.
- link all pools allocating from the same allocator on a linked list.
- Since an allocator is responsible to wait for physical memory it will
only fail (waitok) when it runs out of its backing vm_map, carefully
drain pools using the same allocator so that va space is freed.
(see comments in code for caveats and details).
- change pool_reclaim to return if it actually succeeded to free some
memory, use that information to make draining easier and more efficient.
- get rid of PR_URGENT, noone uses it.

Regress this one additional change, with this change more systems
successfully ran 'make build'.

That was no fix, that broke things. If the pte entry is currently found
in the po lists, it will NOT have the PTE_VALID bit set. Thus valid
mappings could be ignored if enough mappings existed for that PTEG pair.
This explains the bus_dma panics.

pte_spill() is executed on a special stack in real mode (vm not enabled).
It is not valid to call pool_put() from that context.
If called from that context, put the freed item on one of two lists
(race safe), poalloc() will attempt to fetch from there, and pofree()
will clean up if called from a normal context.

branches: 1.52.2;
Fix for pmap extract from art.

Kernel mappings (pmap_kenter_pa) should not be entered into the pv list.

Make pmap_update functions into nops so that we can have a consistent
pmap_update API (right now it's nop).

pmap_kenter_pgs is not used and not really useful. remove.

Sync in more uvm changes from NetBSD.
This time we're getting rid of KERN_* and VM_PAGER_* error codes and
use errnos instead.

Replace inclusion of <vm/foo.h> with the correct <uvm/bar.h> when necessary.
(Look ma, I might have broken the tree)

No need for those prototypes here.

merge vm/vm_kern.h into uvm/uvm_extern.h; art@ ok

Changing the way the pmap code works again.
Changes to the pmap_enter code so that the pmap_kenter/pmap_kremove
has a method to create mappings without adding them to the _pv lists
(part of the point of pmap_k* functions). Also adds an interface
so that device mappings can be created with cacheable attributes.
So that devices such as display memory can be mapped writethru
greatly increasing their speed.

Rewrite of powerpc pmap_page_protect(), the old version had a couple of
possible bugs in it which could cause the code to spin indefinately
attempting to remove all mappings for a page.

This is now able to survive a paging death program and additional other
testing.

Zero pages before handing them over to the VM layer.
This seems to improve the reliablity of the system.
Thanks to those who tested this.

The VP cache code, while giving the powerpc port a signficant speed
increase is contributing to the instability of the port.
This ifdef's the code, disabling it.

remove pv_table, it is not referenced any longer.

convert V->P table memory allocations to pool.

Correct misinitialization of a variable. This worked before?

Change the pmap_enter interface to merge access_type and the wired boolean
and arbitrary flags into one argument.

One new flag is PMAP_CANFAIL that tells pmap_enter that it can fail if there
are not enough resources to satisfy the request. If this flag is not passed,
pmap_enter should panic as it should have done before this change (XXX - many
pmaps are still not doing that).

Only i386 and alpha implement CANFAIL for now.

Includes uvm updates from NetBSD.

Get rid of the PMAP_NEW option by making it mandatory for all archs.
The archs that didn't have a proper PMAP_NEW now have a dummy implementation
with wrappers around the old functions.

more spacees, includes, protos

MNN is no longer an option.

kill old vm

-Warn cleanups for powerpc, still not done.

Post pmap_extract() changes cleanup.
bus_addr_t vs vaddr_t/paddr_t
Return correct value for poalloc();

Change the paddr_t pmap_extract(struct pmap *, vaddr_t) interface to
boolean_t pmap_extract(struct pmap *, vaddr_t, paddr_t *).
Matches NetBSD. Tested by various people on various platforms.

More sync to NetBSD.

- Change pmap_change_wiring to pmap_unwire because it's only called that way.
- Remove pmap_pageable because it's seldom implemented and when it is, it's
either almost useless or incorrect. The same information is already passed
to the pmap anyway by pmap_enter and pmap_unwire.

Remove the (vaddr_t) casts inside the round_page and trunc_page macros.
We might want to use them on types that are bigger than vaddr_t.

Fix all callers that pass pointers without casts.

Remove last change to powerpc pmap. This has a slight chance of being the
cause of a frequent, but not easy to reproduce crash.
The reason for making this change is to support functionality that will
not be in 2.9.

If in pmap_kernel() allow vtop for all 1-1 mapped pages.

Quiet down pmap code, this is mostly shaken out now, remove some of
the debug code.

Improve the page mapped check algorithm in the powerpc pmap module,
before it was looking through two arrays of 8 and a linked list of
undetermined size, before deciding that a mapping was not valid.
Now it allocates a data structure and caches that data.

This improves both pmap_enter and pmap_remove because both check
to see if a mapping is valid before taking the appropriate actions.

Also in pmap_remove, if the va mapping is found, stop searching for
it in the rest of this array, the alternate array and the linked list.
only one valid mapping of each va is allowed.

This change improved lat_mmap (from lmbench) from 1300 to 720
and fork+exit from 7320 to 2724 microseconds.

Adhere to VM/UVM pager requirements, do not unmap pager mappings.
recognize pager_sva/pager_eva. Does not seem to change anything under UVM
but is recommended, may have been the cause for the "pmap" bug under VM.
Test compiled for VM, but not run.
Ok'd by art.

Allow siop driver to work on powerpc.
pmap_extract should work for all accessable memory.
Since powerpc maps the kernel va=pa without using the pte table,
these addresses need to be handled seperately.

Attempt to update powerpc pmap module to MACHINE_NEW_NONCONTIG
configuration. modernize for UVM. Does not yet work with UVM,
but does seem more stable than older version with old VM.
This may be in part due to a kludge that only uses the largest
memory region instead of all of the memory regions. a bug in
the MD MNN code is suspected.

Verify that memory regions are always page aligned and multiple of page size.
The kernel does not want to deal with memory that is not page aligned.

size htab according to system memory size, not constant in header file.

Rework some code in the powerpc pmap module.
Previously it was possible to remove multiple entries on a single pass thru
the pv deletion code in pmap_page_protect. Also when it did this, it
was not properly tracking of decrementing pm->pm_stats.resident_count.

By storing an additional piece of information in the pv structure,
the pmap pointer it is possible to call pmap_remove on the
entries rather than duplicating the pte removal code (again).

This fixes a problem seen where the system panics or hangs in
pmap_remove_pv due to the item not being on the list. Now
it is not possible for mix entries.

Because the pv entry only contained the pteidx, it was possible
to remove an incorrect entry due to ambiguity. multiple pmaps
having mappings at the same va of the same pa. Multipe pmap containing similar
entries will occur frequenty with shared libaries. Because of the hash
entries that have the same result of (sr & ptab_mask) will alias in the pv
list..

Since the pv_idx is now recomputable, should it be removed?

Clean up one of the screwy things about the powerpc pmap, it could
not properly track the count of mapped pages. Fix the count
at a higher level. From NetBSD.

comment out memory region debugging.
make certain that physmem is initialized.
improve readablity of code.

add first version of bus_dma for powerpc.
changes to trap handler to print out better information for jump to 0 bugs.
changes to pmap.c and machdep.c to debug a duplicate memory region
bug occasionally observed on imac with compressed kernels.

branches: 1.14.2;


UVM changes mainly. As of this checkin UVM is still not working for powerpc
it has a copyin bug after device configuration. However to get these diffs
out of my tree.

All of the UVM code is currently inside ifdef UVM the kernel works fine
without option UVM. Config files have been left without UVM for now.

Prelimiary changes for busdma, (what UVM was wanted for).

autoconf.c:
calculate delay time for delay() before it is acutally used.
add support for md_diskconf come closer to supporting crashdumps,
eventually this code should be un if 0 ed and supported.
add the wd device as a supported device, fix some comments.
clock.c:
support calculation of delay loop earlier, do the spin loop correcly,
unsigned math on the lower half, not signed math.
conf.c:
addd support for wd driver, block major 0, char major 11.
machdep.c:
bus_space_map becomes a real function, not just inlined function.
Support devices that are not mapped with bats (most still currently
are mapped with bats,...). BAT mapping does not allow proper
mapping of cachable devices.
mapiodev HACK, NEEDS TO BE REMOVED. added for quicker import
of BROKEN mac drivers. the drivers NEED to be rewritten in
a busified manner. it would FIX all of the endian swabbing
done by each driver. (Is that emphasized enough?)

bus_space_(read|write)_raw_multi as functions, should these
be turned into inline functions and put in bus.h?
ofw_machdep.c:
removed extranious variable.
openfirm.c:
telling openfirmware to "boot" will put the system
in somewhat of a strange state, try reset-all, but that
typically fails, therefore, try OF_exit before spinning.
pmap.c:
support stealing memory from kernel address space so that
mappings can be created before vm is initalized.
vm_machdep.c:
maybe the meaning of removing this will later become obvious. ???

clean up a global pointer/array reference for OF_buf.
fix typo someone made.

Change the pmap_enter api to pass down an argument that indicates
the access type that caused this mapping. This is to simplify pmaps
with mod/ref emulation (none for the moment) and in some cases speed
up pmap_is_{referenced,modified}.
At the same time, clean up some mappings that had too high protection.

XXX - the access type is incorrect in old vm, it's only used by uvm and MD code.
The actual use of this in pmap_enter implementations is not in this commit.

Several changes here:
(Some of these changes are work in progress and may change more later)
locore.S:
rearranged to remove most of the direct openfirmware references in
the attempt to move all of the openfirmware pieces into ofw_ files.
This could allow other firmware type to be supported easier. Also
this keeps the openfirmware code grouped in the same files.

OF_buf is now statically allocated in the data/bss section instead
of allocated during initialization.

machdep.c:
change the order of vm initialization, Still considering removing
the BATs from use. instead of calls directly to ppc_exit and ppc_boot
these are now called via a firmware function pointer structure.
Add iMac recognition to systems
ofw_machdep.c:
function pointer structure to allow different firmware to supply
specific system functionality, normally startup and reset,
including a hook to notify when bsd is about to go virtual,
in case firmware calls need to act different after that time.

Allow BSD to handle the virtual memory operations for openfirmware.
this idea was copied from NetBSD macppc, It is not fully implemented,
among other problems, openfirmware does not have a mechanism to
add new mappings.

ofwreal.S:
Major rewrite of the firmware call code, It still copies
a portion of the stack, but now does not restore exeception vectors.
Modified to be similar in idea to NetBSD macppc with BSD handling
the openfirmware VM faults/TLB misses.
This still needs to be reviewed, Should be possible to not require
any stack copy.

opendev.c:
OF_bus is not a pointer to the buffer, but is the buffer itself now.

openfirm.c:
OF_bus is not a pointer to the buffer, but is the buffer itself now.
Dont panic if OF_boot fails, OF_boot can be called by panic.
instead print and the hang in a spin loop.

pmap.c:
call the firmware function to get memory regions.
Scale the PowerPC hash table size by size of real memory.
Properly align the hash table based on the start, not just
the size.

Remove diagnotic that could (was frequently) causing crashes.
this whole pmap could use replacing.

panic prints a newline for you, don't do it in the panic string

Various changes to allow mixing of ofw drivers and real drivers.
NCR driver seems to work.
Major changes are isa can be child of pci or mainbus.
ofroot is child of mainbus not root.
ofw bus configured before pci bus
Note that if a pci device configures accessing of driver will crash
the system. they need to be exclusive.

Adapt comments to reality

Fix problems pointed out by Andrew Cagney,
These didn't seem to have adverse effects, but were wrong.

Now that ELF symbols are working (not leading '_'), dont put them
in for these symbols either.

support resident page count.
Attempt at least, pmap doesn't allow
for exact tracking.

Adding OpenBSD tags to files.

branches: 1.1.1;
Initial revision

Initialize earlier macppc's and powerpc64's pmap_hash_lock

At boot, the powerpc64 kernel was calling
pmap_bootstrap -> pmap_kenter_pa -> mtx_enter(&pmap_hash_lock)
before it did
pmap_init -> mtx_init(&pmap_hash_lock, IPL_HIGH)

Change from mtx_init to MUTEX_INITIALIZER. This allows an option
WITNESS kernel to boot without warning of an uninitialized mutex.

Also change macppc's pmap_hash_lock from __ppc_lock_init to
PPC_LOCK_INITIALIZER, though WITNESS doesn't see this lock.

ok mpi@

Set the current pmap in macppc's pmap_activate

This fixes a possible freeze in execve(2). It sometimes froze when a
dual-cpu macppc started daemons during boot. There is a chance that
uvm_map.c uvmspace_exec sees ovm->vm_refcnt != 1 and switches curproc
to a new pmap. If this happened, then execve froze by trying to
copyout to the wrong pmap; curpcb->pcb_pm was old. Fix by setting
pointers when uvmspace_exec calls pmap_activate.

ok miod@

Add missing check for pg != NULL

The code was reading pg->pg_flags, so clang assumed pg != NULL, then
optimized a later "if (pg != NULL)" to "if (1)", and allowed a call to
pmap_enter_pv(pted, NULL). Such a call can freeze bsd.mp by trying to
lock NULL's ((struct mutex *)0x3c). I froze bsd.mp this way by
starting Xorg on a macppc with nv(4) or r128(4) video, as it tried to
mmap the xf86(4) aperture.

ok miod@

On systems without xonly mmu hardware-enforcement, we can still mitigate
against classic BROP with a range-checking wrapper in front of copyin() and
copyinstr() which ensures the userland source doesn't overlap the main program
text, ld.so text, signal tramp text (it's mapping is hard to distinguish
so it comes along for the ride), or libc.so text. ld.so tells the kernel
libc.so text range with msyscall(2). The range checking for 2-4 elements is
done without locking (because all 4 ranges are immutable!) and is inexpensive.

write(sock, &open, 400) now fails with EFAULT. No programs have been
discovered which require reading their own text segments with a system call.

On a machine without mmu enforcement, a test program reports the following:
userland kernel
ld.so readable unreadable
mmap xz unreadable unreadable
mmap x readable readable
mmap nrx readable readable
mmap nwx readable readable
mmap xnwx readable readable
main readable unreadable
libc unmapped? readable unreadable
libc mapped readable unreadable

ok kettenis, additional help from miod

Execute-only for macppc G5

The G5 PowerPC 970 has a Data Address Compare mechanism that can trap
loads and stores to pages with PTE_AC_64, while allowing instruction
fetches. Use this for execute-only mappings, like we do on powerpc64.

Add a check to pte_spill_v for execute-only mappings. Without this,
we would forever retry reading an execute-only page.

In altivec_assist, copyin would fail to read the instruction from an
execute-only page. Add copyinsn to bypass x-only, like sparc64.

with help from abieber@ deraadt@ kettenis@
ok deraadt@

Use atomic ops on the set of used segment registers

Each pmap sets a bit in usedsr to claim 16 unique VSIDs for its
segment registers. Use atomic_cas_uint to set this bit (checking that
the other cpu didn't steal it) and atomic_clearbits_int to clear it.
Stop using splvm.

ok miod@

Remove pmap_collect() when a no-op, define __HAVE_PMAP_COLLECT otherwise.
Use that define to shunt uvm_swapout_threads(), which is a noop when
pmap_collect() does nothing.

ok mpi@

Allow writes to rw pages in pte_spill_v

In the powerpc pmap, hash collisions can spill page table entries.
Page faults can use pte_spill_v to reinsert a spilled pte. If the
fault is a write (DSISR_STORE), then pte_spill_v tries to check for a
read-only page. The existing check (pte_lo & PTE_RO_64) also matched
rw pages, because PTE_RO_64 is 3 and PTE_RW_64 is 2. This caused
pte_spill_v to deny writes to rw pages. Then uvm_fault might allow
the write; but uvm_fault can't handle some pages in the kernel. Such
faults caused, "panic: uvm_fault: fault on non-pageable map", or
"panic: trap type 300".

Change it to ((pte_lo & PTE_PP_64) == PTE_RO_64). This seems to fix
one reason why bsd.mp on a macppc dual G5 might panic.

ok kettenis@ miod@

Don't put an extern variable (ppc_kvm_stolen) into vmparam.h, other instances
of this file are only doing cpp #define

spelling

pmap_avail_setup() is the only place physmem is calculated, delete a bunch
of code which thinks it could be done elsewhere.
ok kurt

Switch powerpc to MI mplock implementation.

Reduce differences with others architectures and make it possible to use
WITNESS on it.

Rename & keep the current recursive lock implementation as it is used by
the pmap.

Tested by Peter J. Philipp, otto@ and cwen@.

ok kettenis@

Prepare the bat for kernels greater > 8MB of code, why because clang.
ok kettenis

some cleanup for clang; ok kettenis

In pmap_page_protect(), zap the PTE before unlinking. At that point the
PTED_VA_MANAGED_M flag is still set so proper MOD/REF accounting will
happen. Fixes memory corruption that would invariably happen when a
machine started swapping.

Giant cluestick from George Koehler.
ok visa@, mpi@

More "explicitely" -> "explicitly" in various comments.

ok guenther@ tb@ deraadt@

Implement copyin32(9).

ok mpi@, visa@

Change pmap_proc_iflush() to take a process instead of a proc
powerpc: rename second argument of pmap_proc_iflush() to match other archs

ok kettenis@

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

consistently set ipls on pmap pools.

this is a step toward making ipls unconditionaly on pools.

ok deraadt@ kettenis@

Add a per-page flag to indicate that all mappings of that page should be
uncached. To be used in the drm code.

ok mpi@

Make the powerpc pmap (more) mpsafe by protecting both the pmap itself and the
pv lists with a mutex. This should make pmap_enter(9), pmap_remove(9) and
pmap_page_protect(9) safe to use without holding the kernel lock.

ok visa@, mpi@, deraadt@

Give the pool page allocator backends more sensible names. We now have:
* pool_allocator_single: single page allocator, always interrupt safe
* pool_allocator_multi: multi-page allocator, interrupt safe
* pool_allocator_multi_ni: multi-page allocator, not interrupt-safe

ok deraadt@, dlg@

Make pmap_remove() grab the kernel lock. This is a big hammer but makes MP
machines work again with the unlocked reaper.

ok mpi@, deraadt@
no objection from miod@

Finally protect VP lookups to guarantee that a pted won't be freed or
reused by a CPU while another CPU is manipulating it.

This races occurs because the virtual spill handlers are run without
taking the KERNEL_LOCK for obvious reasons. So use a per-pmap mutex
that CPUs must hold when modifying a pted in order to guarantee the
atomicity of operations *and* the coherence between pmap VPs tree and
what's in the HASH.

Thanks to dlg@ for assisting me debugging this. This change ends your
PowerPC pmap SMP show of the week. GENERIC.MP on macppc should now be
stable enough to build ports without corrupting its own memory.

ok kettenis@, deraadt@, dlg@

Don't try to be clever when unrolling the loop in pmap_remove().

Needed for upcoming locking.

Replace the per-entry locks by a global HASH lock.

Since this lock is recursive we can now guarantee the atomicity of
pte_inser{32,64}() when a pted has to be removed first. This fixes
one of the races.

Using a __mp_lock here also allowed dlg@ to provide me useful traces
to fix the next race. Thanks for your help!

ok kettenis@, deraadt@, dlg@

Call pte_spill_v() from the real mode fault handler instead of rerolling
it. This will reduce the number of places to audit for locking.

Note that for profiling purposes pte_spill_v() is now marked a __noprof
since per-CPU profiling buffers are not guaranteed to be 1:1 mapped and
cannot be accessed from the real mode fault handler.

ok kettenis@, deraadt@, dlg@

Rewrite PTE manipulation routines to better match the PEM.

Document every operation, make sure to call "sync" when appropriate so
that other CPUs see the bit changes and finally grab a lock where it was
missing to grantee atomicity.

ok kettenis@, deraadt@, dlg@

Split pteclrbits() into pmap_{test,clear}_attrs().

This should not introduce any behavior change but makes the code easier
to read and later easier to protect. This also brings this pmap closer
to what others do.

Thanks to kettenis@ for spotting a bad typo!

ok kettenis@, deraadt@, dlg@

More usages of pmap_ptedinhash().

If you wonder why pte_insert{32,64}() is not using pmap_hash_remove() if
it finds a conflicting PTE in the HASH, it's because in the current state
trying to grab the same lock a second time would lead to a deadlock.

This is much easier to reproduce on G5 (or G4 with BAT disabled).

ok kettenis@, deraadt@, dlg@

Remove DEBUG stuff.

Make use of ptesr() instead of rerolling it.

Merge various copies of the same code into a new function to determine
if a PTE is present in the HASH.

Note that atomicity is currently not guaranteed between this check and
the following operations.

ok kettenis@, deraadt@, dlg@

Introduce pmap_pted_ro() a simple wrapper for the 32/64 bits versions
that does not call pmap_vp_lookup().

Carreful readers would have notice the removal of the bits on the virtual
address with a page mask, this change allows me to find the 13 years old
bug fixed in r1.145.

ok kettenis@, deraadt@, dlg@

Do only one VP lookup when removing a page.

This simplify pmap_remove() & friends by re-using an already fetched PTE
descriptor.

There's currently a race on MP system where one CPU can reuse a pted
while another one is still trying to insert it in the HASH. This commit
starts reducing the number of pmap_vp_lookup() calls to help fix this
race.

ok kettenis@, deraadt@, dlg@

Remove the MANAGED flag when removing a PV entry.

Even if this change is not strickly needed, because the memory will be
returned to the pool it helped me track the use-after-free.

Remove unneeded splvm() calls and the pool_setipl(9) hack of r1.140.

By instructing spl(9) calls on MP machines I figured out that their high
cost was hiding a race condition involving PTE reuse in our pmap. Thanks
to deraadt@ for finding a way to trigger such panic by adding a couple of
splvm().

This should make the races easier to trigger but will be addressed
shortly.

This commit starts your PowerPC pmap SMP show of the week.

ok kettenis@, deraadt@, dlg@

Fix 13 years old typo that should be responsible for the unhappiness
of UVM on PowerPC architectures by breaking pmap_is_referenced() and
friends.

ok kettenis@

Make it possisble to disable block address translation mechanism on
processors that support it.

Due to the way trap code is patched it is currently not possible to
enabled/disable BAT at runtime.

ok miod@, kettenis@

Merge two versions of ppc_check_procid().

ok miod@, kettenis@ as part of a larger diff

oops, accidental commit

sync

pool_setipl() on both pmap pools as a workaround for some sort of MP
race. This will certainly be revisited, but too much time has been
spent on it for now.
ok mpi

Let powerpc's bus_space(9) use the same pmap and uvm interfaces than the
other archs.

Specify the caching policy by passing PMAP_* flags to pmap_kenter_pa()
like the majority of our archs do and kill pmap_kenter_cache().

Spread some pmap_update() along the way.

While here remove the unused flag argument from pmap_fill_pte().

Finally convert the bus map/unmap functions to km_alloc/free() instead
of uvm_km_valloc/free().

Inputs from kettenis@ and miod@, ok miod@

Even without BATs memory under ``physmaxaddr'' is mapped 1:1 in the
kernel, so update pmap_extract() accordingly and save a VP lookup.

While here unify pted checks after the VP lookups.

ok miod@

Various cleanups. Explicitly include <sys/atomic.h>, Use pmap_remove_pg()
for the kernel pmap and kill pmap_kremove_pg(). Finally guard the hash
lock code under "MULTIPROCESSOR" to explicit which part of the code
received some MP love.

ok kettenis@

force the pool of pmapvp onto PAGE_SIZE allocations by specifying a
pool allocator. pmapvp is 1024 bytes, and the size * 8 change in pools
without an allocator being specified tries to place it on large pages.
you need pmap to use large pages, and pmap isnt set up yet.

fixed a very early fault on macppc.
debugged with and tested by krw@
ok deraadt@ krw@

remove simplelocks use
ok kettenis mpi

Speed up page zeroing by using a loop of dcbz/dcbzl instead of bzero().

While here, use the direct map for pmap_copy_page() and remove the now
unused stolen page addresses.

No objection from the usual suspects, "it works, commit" deraadt@

make pmap_zero_page MP-safe, by using the directmap
mpi will investigate speedups after this.
ok mpi kettenis

Replace a plethora of historical protection options with just
PROT_NONE, PROT_READ, PROT_WRITE, and PROT_EXEC from mman.h.
PROT_MASK is introduced as the one true way of extracting those bits.
Remove UVM_ADV_* wrapper, using the standard names.
ok doug guenther kettenis

Only mark segment 0 as executable on 64-bit systems. There it is harmless as
we have a proper X bit in the page tables. On 32-bit systems kernel .text is
handled by an IBAT, so we don't need page table entries that are executable
in the kernel pmap.

ok mpi@

Remove execute permission from most pages in the kernel pmap. This is a first
step towards W^X in the kernel, even though it is only effective on machines
with a G5 processor.

ok mpi@

Format string fixes and removal of -Wno-format for *ppc kernels.

Allow to compile with DEBUG_PMAP defined.

Remove the almost unused abstraction around "struct firmware" and use
instead a single function ppc_mem_regions() required by the ppc pmap.

ok kettenis@

Including <uvm/uvm_extern.h> is enough, no need for <uvm/uvm.h> or more.

Use syncicache() instead of rerolling an almost identical version.

ok miod@

Do not bzero() the available memory in pmap_bootstrap(); allocations in
pmap_bootstrap explicitely bzero them, and there is no need to clear the
remaining memory.

ok mpi@

Some (if not all) G5 systems use a different layout for the physical memory
information (property `reg' of the `/memory' node). Fortunately the available
physical memory information still uses the same format, so this only affects
the computation of physmem.

Detect this case and parse the information correctly, converting to the format
expected by pmap, ignoring physical memory beyond 4GB.

Compute physmem from all the physical memory information, even memory not
usable by the kernel. Let pmap not recompute physmem in pmap_bootstrap() if
physmem is != 0 upon entry.

This should allow G5 systems fitted with more than 2GB of physical memory to
report the correct amount of memory, even though the kernel will only use
the lower 2GB.

Prompted by a dmesg@ submission by Greg Marsh, owner of a 3.5GB G5

help and tweaks kettenis@, ok mpi@

Remove excessive parentheses.

pmap.c:1061:13: error: equality comparison with extraneous parentheses [-Werror,-Wparentheses-equality]

ok mpi@

Mark all the C functions called in real mode as non instrumented and
remove the check for address relocation from MCOUNT_ENTER.

This fix kernel profiling on powerpc architectures, broken since the
buffers are per cpu.

ok miod@

Managed device mappings should be uncached by default.

ok mpi@

Add the possibility to map DMA memory non-cached, based on the i386/amd64
implementation. For the moment only the BUS_DMA_NOCACHE macro is required
to build drm on macppc but it will be used soon.

ok kettenis@

Remove the freelist member from vm_physseg

The new world order of pmemrange makes this data completely redundant
(being dealt with by the pmemrange constraints instead). Remove all code
that messes with the freelist.

While touching every caller of uvm_page_physload() anyway, add the flags
argument to all callers (all but one is 0 and that one already used
PHYSLOAD_DEVICE) and remove the macro magic to allow callers to continue
without it.

Should shrink the code a bit, as well.

matthew@ pointed out some mistakes i'd made.
``freelist death, I like. Ok.' ariane@
`I agree with the general direction, go ahead and i'll fix any fallout
shortly'' miod@ (68k 88k and vax i could not check would build)

No "\n" needed at the end of panic() strings.

Bogus chunks pointed out by matthew@ and miod@. No cookies for
marco@ and jasper@.

ok deraadt@ miod@ matthew@ jasper@ macro@

We never create or destroy pmaps from interrupt context, so wrapping the
associated pool calls in splvm()/splx() is unnecessary and confusing.

ok deraadt@, drahn@

Don't #include <sys/user.h> into files that don't need the stuff
it defines. In some cases, this means pulling in uvm.h or pcb.h
instead, but most of the inclusions were just noise. Tested on
alpha, amd64, armish, hppa, i386, macpcc, sgi, sparc64, and vax,
mostly by krw and naddy.
ok krw@

There is no reason to protect the pmap_vp_pool with splvm(). The only pmap
that gets manipulated in interrupt context is the kernel pmap, and we fully
populate its VP mappings during pmap_bootstrap(). Gets rid of the excessive
spl's at pmap_destroy() time noticed by deraadt@

ok deraadt@, drahn@

two missing splx in error path; ok drahn

Prevent an interrupt from causing recursion while holding the pmap hash lock,
Otherwise a pmap_remove from a completed I/O may deadlock.

fix an ugly construct

Clear the PG_PMAP_EXE flags whenever writable mappings are created.
ok deraadt@ kettenis@

More carefully manage PG_PMAP_EXE bit and cache flushing on pmap_protect
operations, where X or W is taken away. ok deraadt@ kettenis@

Make pmap_enter respect the PMAP_CANFAIL flag. With and essential
memory leak plug from drahn@

Handle pool allocation failures slightly better. ok kettenis@

Kernel map is supposed to only allocate from the limited kernel addresses,
panic if the kernel attempts to map an improper address.

A bunch of pool_get() + bzero() -> pool_get(..., .. | PR_ZERO)
conversions that should shave a few bytes off the kernel.

ok henning, krw, jsing, oga, miod, and thib (``even though i usually prefer
FOO|BAR''; thanks for looking.

Changes to get closer to SMP.
add biglock before interrupt calls into the kernel.
switch the clock to using cpuinfo variables instead of globals
move cpu_switchto into C code so that on multiprocessor the FPU
and Altivec can be saved before switching CPUs.
add a lock into pmap when modifying the hash table.

replace even more ctob/btoc with ptoa/atop

[fF]uther -> [fF]urther in comments and man page. First one spotted on
tech@ by Jung.

Move powerpc to vm_page_md, 'throw it in' kettenis@

Ansi prototypes, not K&R. no binary difference.

Implement pmap_steal_memory() on powerpc. With some help from art@.

Relax the cache flags logic in pmap_kenter_pa, to make sure that mappings
entered before vm_physmem[] are initialized will be cached. This is a
temporary measure until this pmap implements pmap_steal_memory().

Help and ok drahn@

Dont pass seemingly random numbers down as the flag
parameter of pool_init()

ok drahn@
no objections miod@

W^X for G5
ok drahn@, totally groovy deraadt@

Get rid of deprecated vm_{offset,size}_t types for good, use {p,v}{addr,size}_t
instead; looked at millert@

splimp -> splvm

ok drahn@

Nearly functional crashdump support for macppc. Because savecore
does not recognize the resulting crashdumps, the writing has been disabled.
Better here than in my forest of trees.

64 bit compat. clean up the 32/64 code paths so less decision points exist
allow more than 256MB ram on G5, (still 2G limit) by creating PTE entries
dynamically for all physical memory.

Handle segment register restore at context enter/exit instead of
deep in the kernel. Based on code from two years ago, now necessary
for G5. removes the 1GB ram limit on 32bit processors, temporarily
sets a 256MB limit on G5.

G5 pmap support, most of this G5 work has been done by kettenis@
without his forging ahead, it would barely be started.
Again this is one step of many, but needs to be tested, this is
independant of the locore change just committed which kettenis@ and
deraadt@ significantly wrote.

Avoid infite loop.
ok drahn@

Do a better job at containing powerpc specific #defines to PPC_...
ok deraadt@

Properly flush instruction cache for ptrace(PT_WRTIE_{DI}, ...) on powerpc
and m68k.
ok drahn@, millert@

Various typos in comments.

backout segment register restore diff which causes reproducible hangs; ok deraadt

Pass -Wformat

Fix ppc segment register restores, this fixes the 1GB ram limit and
cleans up pieces in the pmap code.
tested otto, brad, miod, pval.

Reduce the amount of asm code in powerpc/macppc by replacing it with
inlined functions, helps improve readability and fix a couple of bugs.
ok miod@

kill clause 3 and 4 from several of my copyrights, cleanup.

Remove an unnecessary structure copy from useage of setfault(), call
by reference, not by value, ok matthieu#, miod@

Track if a physical page has been previously mapped executable. If it
has not been previously mapped EXE, flush it. If a writeable mapping
which is not executable occurs for the page, clear this bit.
Solves a problem where an executable page is double mapped, first without
EXE then accessed for execute at a different physical page, the cache
will behave properly.

Eliminate the use of KERN_SUCCESS outside of uvm/

Also uvm_map returns KERN_* codes that are directly mapped to
errnos, so we can return them instead of doing some attempt to
translation.

drahn@ "I see no problem" pval@ "makes sense"

Remove more '\n's from panic() statements. From Chris Kuethe.

backout premature

KNF

Change the pmap_zero_page and pmap_copy_page API to take the struct vm_page *
instead of the pa. Most callers already had it handy and those who didn't
only called it for managed pages and were outside time-critical code.

This will allow us to make those functions clean and fast on sparc and
sparc64 letting us to avoid unnecessary cache flushes.

deraadt@ miod@ drahn@ ok.

- change pte_spill_X() to take an extra parameter to determine if
the fault is a EXE fault or R/W fault.

- mask/or the SR_NOEXEC bit into the segment register value
when the number of executable pages becomes 0/non-zero.

- create segments with SR_NOEXEC set, will be cleared when first
exec mapping in the segment is created.

- allow pte_spill_X() to deal with a new type of fault, page mapped
but non executable, when execute was requested.

Adds up to - non-exec stack support for powerpc.

Perform accounting for executable pages on powerpc, prepare for
non-executeable stack.

Cleanup: use less _t typedefs, use the structure itself.

pmap_t is the exception, it is required by the MI code so pmap_t will
be used instead of using 'struct pmap *' in the code. (consistency)

argh, no last minute changes...

pmap cleanup and KNF.

This doesn't happen, and Debugger should not be used here anyway...

spelling; raj@cerias.purdue.edu

Optimize pmap_remove(). It frequently is called with no mappings to
remove eg mmap() ok miod@ art@

Attribute table must be allocated 1-1 because it is accessed from
pte_spill_r(). ok art@

Remove dead code, ifdef code which should be dead, KNF. Cleanup.

Be more consistant about pted zeroing (the whole structure)
Fix missing 'attr' initialization.
Zero available memory.
Raise available memory limit from 256MB to 1GB. This code has only been
tested up to 512MB, but should be fine to 1GB. Ram modules are not avail
to the developers to test out the machines up to their 1.5GB physical limit.

Do not include headers twice. Pointed out by Dries Schellekens.

Final __P removal plus some cosmetic fixups

Complete rewrite of the powerpc pmap handling, Instead of keeping
the spill list for each PTEG, the V->P translations are stored in
trees for each pmap. All valid kernel mappings are preallocated
in 1-1 memory so that tlb spill/loads for kernel accesses can be
looked up while physical, user mappings are not guaranteed to
be 1-1 mapped, thus the kernel must go virtual to look up user
mappings. While this is more expensive, the tree search is much
lower cost than the long linked list search. Also on each pmap_remove()
it was necessary to search the linked lists for each possible mapping,
now it just looks up the entry in the tree.
This change gives a 25-36% speedup in 'make build' time. What was
around 2:50 is now around 1:55 on a 733MHz G4.

This change causes a likely existing bug to appear quite often,
it deals with the segment register invalidation in kernel mode.
Because of that problem, currently this change limits the physical
memory used to 256MB. This limitation will be fixed soon, it is not
an error in the pmap code.

* Effort sponsored in part by the Defense Advanced Research Projects
* Agency (DARPA) and Air Force Research Laboratory, Air Force
* Materiel Command, USAF, under agreement number F30602-01-2-0537.

Move the message buffer out of low memory, Openfirmware clears the area
on reboot. perhaps OF uses it at other times?
Since OF always use the same memory addresses, this should always allocate
the same ram to the msgbuf, and allow it to be preserved across reboot.

Instead of finding pages matching this physical page, match this specific
entry. Also terminate the search as soon as this entry is found.

How did this work? It used to attempt the tlb entry for a mapping it is
removing by using the va it is replacing it with, NO!.
Calculate the va of the mapping by inverting the pte_hi calculation
producing bits 4-19 of the address. This is enough to correctly invalidate
the tlb entry for the mapping being removed.

Pool deals fairly well with physical memory shortage, but it doesn't deal
well (not at all) with shortages of the vm_map where the pages are mapped
(usually kmem_map).

Try to deal with it:
- group all information the backend allocator for a pool in a separate
struct. The pool will only have a pointer to that struct.
- change the pool_init API to reflect that.
- link all pools allocating from the same allocator on a linked list.
- Since an allocator is responsible to wait for physical memory it will
only fail (waitok) when it runs out of its backing vm_map, carefully
drain pools using the same allocator so that va space is freed.
(see comments in code for caveats and details).
- change pool_reclaim to return if it actually succeeded to free some
memory, use that information to make draining easier and more efficient.
- get rid of PR_URGENT, noone uses it.

Regress this one additional change, with this change more systems
successfully ran 'make build'.

That was no fix, that broke things. If the pte entry is currently found
in the po lists, it will NOT have the PTE_VALID bit set. Thus valid
mappings could be ignored if enough mappings existed for that PTEG pair.
This explains the bus_dma panics.

pte_spill() is executed on a special stack in real mode (vm not enabled).
It is not valid to call pool_put() from that context.
If called from that context, put the freed item on one of two lists
(race safe), poalloc() will attempt to fetch from there, and pofree()
will clean up if called from a normal context.

branches: 1.52.2;
Fix for pmap extract from art.

Kernel mappings (pmap_kenter_pa) should not be entered into the pv list.

Make pmap_update functions into nops so that we can have a consistent
pmap_update API (right now it's nop).

pmap_kenter_pgs is not used and not really useful. remove.

Sync in more uvm changes from NetBSD.
This time we're getting rid of KERN_* and VM_PAGER_* error codes and
use errnos instead.

Replace inclusion of <vm/foo.h> with the correct <uvm/bar.h> when necessary.
(Look ma, I might have broken the tree)

No need for those prototypes here.

merge vm/vm_kern.h into uvm/uvm_extern.h; art@ ok

Changing the way the pmap code works again.
Changes to the pmap_enter code so that the pmap_kenter/pmap_kremove
has a method to create mappings without adding them to the _pv lists
(part of the point of pmap_k* functions). Also adds an interface
so that device mappings can be created with cacheable attributes.
So that devices such as display memory can be mapped writethru
greatly increasing their speed.

Rewrite of powerpc pmap_page_protect(), the old version had a couple of
possible bugs in it which could cause the code to spin indefinately
attempting to remove all mappings for a page.

This is now able to survive a paging death program and additional other
testing.

Zero pages before handing them over to the VM layer.
This seems to improve the reliablity of the system.
Thanks to those who tested this.

The VP cache code, while giving the powerpc port a signficant speed
increase is contributing to the instability of the port.
This ifdef's the code, disabling it.

remove pv_table, it is not referenced any longer.

convert V->P table memory allocations to pool.

Correct misinitialization of a variable. This worked before?

Change the pmap_enter interface to merge access_type and the wired boolean
and arbitrary flags into one argument.

One new flag is PMAP_CANFAIL that tells pmap_enter that it can fail if there
are not enough resources to satisfy the request. If this flag is not passed,
pmap_enter should panic as it should have done before this change (XXX - many
pmaps are still not doing that).

Only i386 and alpha implement CANFAIL for now.

Includes uvm updates from NetBSD.

Get rid of the PMAP_NEW option by making it mandatory for all archs.
The archs that didn't have a proper PMAP_NEW now have a dummy implementation
with wrappers around the old functions.

more spacees, includes, protos

MNN is no longer an option.

kill old vm

-Warn cleanups for powerpc, still not done.

Post pmap_extract() changes cleanup.
bus_addr_t vs vaddr_t/paddr_t
Return correct value for poalloc();

Change the paddr_t pmap_extract(struct pmap *, vaddr_t) interface to
boolean_t pmap_extract(struct pmap *, vaddr_t, paddr_t *).
Matches NetBSD. Tested by various people on various platforms.

More sync to NetBSD.

- Change pmap_change_wiring to pmap_unwire because it's only called that way.
- Remove pmap_pageable because it's seldom implemented and when it is, it's
either almost useless or incorrect. The same information is already passed
to the pmap anyway by pmap_enter and pmap_unwire.

Remove the (vaddr_t) casts inside the round_page and trunc_page macros.
We might want to use them on types that are bigger than vaddr_t.

Fix all callers that pass pointers without casts.

Remove last change to powerpc pmap. This has a slight chance of being the
cause of a frequent, but not easy to reproduce crash.
The reason for making this change is to support functionality that will
not be in 2.9.

If in pmap_kernel() allow vtop for all 1-1 mapped pages.

Quiet down pmap code, this is mostly shaken out now, remove some of
the debug code.

Improve the page mapped check algorithm in the powerpc pmap module,
before it was looking through two arrays of 8 and a linked list of
undetermined size, before deciding that a mapping was not valid.
Now it allocates a data structure and caches that data.

This improves both pmap_enter and pmap_remove because both check
to see if a mapping is valid before taking the appropriate actions.

Also in pmap_remove, if the va mapping is found, stop searching for
it in the rest of this array, the alternate array and the linked list.
only one valid mapping of each va is allowed.

This change improved lat_mmap (from lmbench) from 1300 to 720
and fork+exit from 7320 to 2724 microseconds.

Adhere to VM/UVM pager requirements, do not unmap pager mappings.
recognize pager_sva/pager_eva. Does not seem to change anything under UVM
but is recommended, may have been the cause for the "pmap" bug under VM.
Test compiled for VM, but not run.
Ok'd by art.

Allow siop driver to work on powerpc.
pmap_extract should work for all accessable memory.
Since powerpc maps the kernel va=pa without using the pte table,
these addresses need to be handled seperately.

Attempt to update powerpc pmap module to MACHINE_NEW_NONCONTIG
configuration. modernize for UVM. Does not yet work with UVM,
but does seem more stable than older version with old VM.
This may be in part due to a kludge that only uses the largest
memory region instead of all of the memory regions. a bug in
the MD MNN code is suspected.

Verify that memory regions are always page aligned and multiple of page size.
The kernel does not want to deal with memory that is not page aligned.

size htab according to system memory size, not constant in header file.

Rework some code in the powerpc pmap module.
Previously it was possible to remove multiple entries on a single pass thru
the pv deletion code in pmap_page_protect. Also when it did this, it
was not properly tracking of decrementing pm->pm_stats.resident_count.

By storing an additional piece of information in the pv structure,
the pmap pointer it is possible to call pmap_remove on the
entries rather than duplicating the pte removal code (again).

This fixes a problem seen where the system panics or hangs in
pmap_remove_pv due to the item not being on the list. Now
it is not possible for mix entries.

Because the pv entry only contained the pteidx, it was possible
to remove an incorrect entry due to ambiguity. multiple pmaps
having mappings at the same va of the same pa. Multipe pmap containing similar
entries will occur frequenty with shared libaries. Because of the hash
entries that have the same result of (sr & ptab_mask) will alias in the pv
list..

Since the pv_idx is now recomputable, should it be removed?

Clean up one of the screwy things about the powerpc pmap, it could
not properly track the count of mapped pages. Fix the count
at a higher level. From NetBSD.

comment out memory region debugging.
make certain that physmem is initialized.
improve readablity of code.

add first version of bus_dma for powerpc.
changes to trap handler to print out better information for jump to 0 bugs.
changes to pmap.c and machdep.c to debug a duplicate memory region
bug occasionally observed on imac with compressed kernels.

branches: 1.14.2;


UVM changes mainly. As of this checkin UVM is still not working for powerpc
it has a copyin bug after device configuration. However to get these diffs
out of my tree.

All of the UVM code is currently inside ifdef UVM the kernel works fine
without option UVM. Config files have been left without UVM for now.

Prelimiary changes for busdma, (what UVM was wanted for).

autoconf.c:
calculate delay time for delay() before it is acutally used.
add support for md_diskconf come closer to supporting crashdumps,
eventually this code should be un if 0 ed and supported.
add the wd device as a supported device, fix some comments.
clock.c:
support calculation of delay loop earlier, do the spin loop correcly,
unsigned math on the lower half, not signed math.
conf.c:
addd support for wd driver, block major 0, char major 11.
machdep.c:
bus_space_map becomes a real function, not just inlined function.
Support devices that are not mapped with bats (most still currently
are mapped with bats,...). BAT mapping does not allow proper
mapping of cachable devices.
mapiodev HACK, NEEDS TO BE REMOVED. added for quicker import
of BROKEN mac drivers. the drivers NEED to be rewritten in
a busified manner. it would FIX all of the endian swabbing
done by each driver. (Is that emphasized enough?)

bus_space_(read|write)_raw_multi as functions, should these
be turned into inline functions and put in bus.h?
ofw_machdep.c:
removed extranious variable.
openfirm.c:
telling openfirmware to "boot" will put the system
in somewhat of a strange state, try reset-all, but that
typically fails, therefore, try OF_exit before spinning.
pmap.c:
support stealing memory from kernel address space so that
mappings can be created before vm is initalized.
vm_machdep.c:
maybe the meaning of removing this will later become obvious. ???

clean up a global pointer/array reference for OF_buf.
fix typo someone made.

Change the pmap_enter api to pass down an argument that indicates
the access type that caused this mapping. This is to simplify pmaps
with mod/ref emulation (none for the moment) and in some cases speed
up pmap_is_{referenced,modified}.
At the same time, clean up some mappings that had too high protection.

XXX - the access type is incorrect in old vm, it's only used by uvm and MD code.
The actual use of this in pmap_enter implementations is not in this commit.

Several changes here:
(Some of these changes are work in progress and may change more later)
locore.S:
rearranged to remove most of the direct openfirmware references in
the attempt to move all of the openfirmware pieces into ofw_ files.
This could allow other firmware type to be supported easier. Also
this keeps the openfirmware code grouped in the same files.

OF_buf is now statically allocated in the data/bss section instead
of allocated during initialization.

machdep.c:
change the order of vm initialization, Still considering removing
the BATs from use. instead of calls directly to ppc_exit and ppc_boot
these are now called via a firmware function pointer structure.
Add iMac recognition to systems
ofw_machdep.c:
function pointer structure to allow different firmware to supply
specific system functionality, normally startup and reset,
including a hook to notify when bsd is about to go virtual,
in case firmware calls need to act different after that time.

Allow BSD to handle the virtual memory operations for openfirmware.
this idea was copied from NetBSD macppc, It is not fully implemented,
among other problems, openfirmware does not have a mechanism to
add new mappings.

ofwreal.S:
Major rewrite of the firmware call code, It still copies
a portion of the stack, but now does not restore exeception vectors.
Modified to be similar in idea to NetBSD macppc with BSD handling
the openfirmware VM faults/TLB misses.
This still needs to be reviewed, Should be possible to not require
any stack copy.

opendev.c:
OF_bus is not a pointer to the buffer, but is the buffer itself now.

openfirm.c:
OF_bus is not a pointer to the buffer, but is the buffer itself now.
Dont panic if OF_boot fails, OF_boot can be called by panic.
instead print and the hang in a spin loop.

pmap.c:
call the firmware function to get memory regions.
Scale the PowerPC hash table size by size of real memory.
Properly align the hash table based on the start, not just
the size.

Remove diagnotic that could (was frequently) causing crashes.
this whole pmap could use replacing.

panic prints a newline for you, don't do it in the panic string

Various changes to allow mixing of ofw drivers and real drivers.
NCR driver seems to work.
Major changes are isa can be child of pci or mainbus.
ofroot is child of mainbus not root.
ofw bus configured before pci bus
Note that if a pci device configures accessing of driver will crash
the system. they need to be exclusive.

Adapt comments to reality

Fix problems pointed out by Andrew Cagney,
These didn't seem to have adverse effects, but were wrong.

Now that ELF symbols are working (not leading '_'), dont put them
in for these symbols either.

support resident page count.
Attempt at least, pmap doesn't allow
for exact tracking.

Adding OpenBSD tags to files.

branches: 1.1.1;
Initial revision

Set the current pmap in macppc's pmap_activate

This fixes a possible freeze in execve(2). It sometimes froze when a
dual-cpu macppc started daemons during boot. There is a chance that
uvm_map.c uvmspace_exec sees ovm->vm_refcnt != 1 and switches curproc
to a new pmap. If this happened, then execve froze by trying to
copyout to the wrong pmap; curpcb->pcb_pm was old. Fix by setting
pointers when uvmspace_exec calls pmap_activate.

ok miod@

Add missing check for pg != NULL

The code was reading pg->pg_flags, so clang assumed pg != NULL, then
optimized a later "if (pg != NULL)" to "if (1)", and allowed a call to
pmap_enter_pv(pted, NULL). Such a call can freeze bsd.mp by trying to
lock NULL's ((struct mutex *)0x3c). I froze bsd.mp this way by
starting Xorg on a macppc with nv(4) or r128(4) video, as it tried to
mmap the xf86(4) aperture.

ok miod@

On systems without xonly mmu hardware-enforcement, we can still mitigate
against classic BROP with a range-checking wrapper in front of copyin() and
copyinstr() which ensures the userland source doesn't overlap the main program
text, ld.so text, signal tramp text (it's mapping is hard to distinguish
so it comes along for the ride), or libc.so text. ld.so tells the kernel
libc.so text range with msyscall(2). The range checking for 2-4 elements is
done without locking (because all 4 ranges are immutable!) and is inexpensive.

write(sock, &open, 400) now fails with EFAULT. No programs have been
discovered which require reading their own text segments with a system call.

On a machine without mmu enforcement, a test program reports the following:
userland kernel
ld.so readable unreadable
mmap xz unreadable unreadable
mmap x readable readable
mmap nrx readable readable
mmap nwx readable readable
mmap xnwx readable readable
main readable unreadable
libc unmapped? readable unreadable
libc mapped readable unreadable

ok kettenis, additional help from miod

Execute-only for macppc G5

The G5 PowerPC 970 has a Data Address Compare mechanism that can trap
loads and stores to pages with PTE_AC_64, while allowing instruction
fetches. Use this for execute-only mappings, like we do on powerpc64.

Add a check to pte_spill_v for execute-only mappings. Without this,
we would forever retry reading an execute-only page.

In altivec_assist, copyin would fail to read the instruction from an
execute-only page. Add copyinsn to bypass x-only, like sparc64.

with help from abieber@ deraadt@ kettenis@
ok deraadt@

Use atomic ops on the set of used segment registers

Each pmap sets a bit in usedsr to claim 16 unique VSIDs for its
segment registers. Use atomic_cas_uint to set this bit (checking that
the other cpu didn't steal it) and atomic_clearbits_int to clear it.
Stop using splvm.

ok miod@

Remove pmap_collect() when a no-op, define __HAVE_PMAP_COLLECT otherwise.
Use that define to shunt uvm_swapout_threads(), which is a noop when
pmap_collect() does nothing.

ok mpi@

Allow writes to rw pages in pte_spill_v

In the powerpc pmap, hash collisions can spill page table entries.
Page faults can use pte_spill_v to reinsert a spilled pte. If the
fault is a write (DSISR_STORE), then pte_spill_v tries to check for a
read-only page. The existing check (pte_lo & PTE_RO_64) also matched
rw pages, because PTE_RO_64 is 3 and PTE_RW_64 is 2. This caused
pte_spill_v to deny writes to rw pages. Then uvm_fault might allow
the write; but uvm_fault can't handle some pages in the kernel. Such
faults caused, "panic: uvm_fault: fault on non-pageable map", or
"panic: trap type 300".

Change it to ((pte_lo & PTE_PP_64) == PTE_RO_64). This seems to fix
one reason why bsd.mp on a macppc dual G5 might panic.

ok kettenis@ miod@

Don't put an extern variable (ppc_kvm_stolen) into vmparam.h, other instances
of this file are only doing cpp #define

spelling

pmap_avail_setup() is the only place physmem is calculated, delete a bunch
of code which thinks it could be done elsewhere.
ok kurt

Switch powerpc to MI mplock implementation.

Reduce differences with others architectures and make it possible to use
WITNESS on it.

Rename & keep the current recursive lock implementation as it is used by
the pmap.

Tested by Peter J. Philipp, otto@ and cwen@.

ok kettenis@

Prepare the bat for kernels greater > 8MB of code, why because clang.
ok kettenis

some cleanup for clang; ok kettenis

In pmap_page_protect(), zap the PTE before unlinking. At that point the
PTED_VA_MANAGED_M flag is still set so proper MOD/REF accounting will
happen. Fixes memory corruption that would invariably happen when a
machine started swapping.

Giant cluestick from George Koehler.
ok visa@, mpi@

More "explicitely" -> "explicitly" in various comments.

ok guenther@ tb@ deraadt@

Implement copyin32(9).

ok mpi@, visa@

Change pmap_proc_iflush() to take a process instead of a proc
powerpc: rename second argument of pmap_proc_iflush() to match other archs

ok kettenis@

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

consistently set ipls on pmap pools.

this is a step toward making ipls unconditionaly on pools.

ok deraadt@ kettenis@

Add a per-page flag to indicate that all mappings of that page should be
uncached. To be used in the drm code.

ok mpi@

Make the powerpc pmap (more) mpsafe by protecting both the pmap itself and the
pv lists with a mutex. This should make pmap_enter(9), pmap_remove(9) and
pmap_page_protect(9) safe to use without holding the kernel lock.

ok visa@, mpi@, deraadt@

Give the pool page allocator backends more sensible names. We now have:
* pool_allocator_single: single page allocator, always interrupt safe
* pool_allocator_multi: multi-page allocator, interrupt safe
* pool_allocator_multi_ni: multi-page allocator, not interrupt-safe

ok deraadt@, dlg@

Make pmap_remove() grab the kernel lock. This is a big hammer but makes MP
machines work again with the unlocked reaper.

ok mpi@, deraadt@
no objection from miod@

Finally protect VP lookups to guarantee that a pted won't be freed or
reused by a CPU while another CPU is manipulating it.

This races occurs because the virtual spill handlers are run without
taking the KERNEL_LOCK for obvious reasons. So use a per-pmap mutex
that CPUs must hold when modifying a pted in order to guarantee the
atomicity of operations *and* the coherence between pmap VPs tree and
what's in the HASH.

Thanks to dlg@ for assisting me debugging this. This change ends your
PowerPC pmap SMP show of the week. GENERIC.MP on macppc should now be
stable enough to build ports without corrupting its own memory.

ok kettenis@, deraadt@, dlg@

Don't try to be clever when unrolling the loop in pmap_remove().

Needed for upcoming locking.