Counter tweaks:

- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
each kind of page can be summed to get the totals.

- Track the number of free pages with a counter so that it's one less thing
for the allocator to do, which opens up further options there.

- Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot.
For the cheap option, give cpu_count_sync() a boolean parameter indicating
that a cached value is okay, and rate limit the updates for cached values
to hz.

uvm_loanuobjpages():

- there are no pages to unbusy in the error case
- always clear the caller's page array

uvm_loanuobjpages():

- vmobjlock is shared between tmpfs vnodes and UAOs now
- split into two routines, to simplify
- fix error recovery

Start trying to reduce cache misses on vm_page during fault processing.

- Make PGO_LOCKED getpages imply PGO_NOBUSY and remove the latter. Mark
pages busy only when there's actually I/O to do.

- When doing COW on a uvm_object, don't mess with neighbouring pages. In
all likelyhood they're already entered.

- Don't mess with neighbouring VAs that have existing mappings as replacing
those mappings with same can be quite costly.

- Don't enqueue pages for neighbour faults unless not enqueued already, and
don't activate centre pages unless uvmpdpol says its useful.

Also:

- Make PGO_LOCKED getpages on UAOs work more like vnodes: do gang lookup in
the radix tree, and don't allocate new pages.

- Fix many assertion failures around faults/loans with tmpfs.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Go back to freeing struct vm_anon one by one. There may have been an
advantage circa ~2008 but there isn't now.

Tweak the March 14th change to make page waits interlocked by pg->interlock.
Remove unneeded changes and only deal with the PQ_WANTED flag, to exclude
possible bugs.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

uvm_unloanpage(): fix a screwup in previous. slock must be set NULL if
it can't be acquired.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.93.2;
- Add and use wrapper functions that take and acquire page interlocks, and pairs
of page interlocks. Require that the page interlock be held over calls to
uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state. Rather than
updating the global state synchronously, set an intended state on
individual pages (active, inactive, enqueued, dequeued) while holding the
page interlock. After the interlock is released put the pages on a 128
entry per-CPU queue for their state changes to be made real in batch.
This results in in a ~400 fold decrease in contention on my test system.
Proposed on tech-kern but modified to use the page interlock rather than
atomics to synchronise as it's much easier to maintain that way, and
cheaper.

PR kern/54783: t_mmap crahes the kernel

- Fix various locking & sequencing errors with breaking loans.

- Don't call uvm_pageremove_tree() while holding pg->interlock as radixtree
can take further locks when freeing nodes.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Don't call uvm_pagedequeue() while holding pg->interlock.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

- Adjust uvmexp.swpgonly with atomics, and make uvm_swap_data_lock static.
- A bit more __cacheline_aligned on mutexes.

branches: 1.87.2;
add the KASSERT() for loan_count wrap-around to all places which increase it

detect wraparound when bumping page wire_count and loan_count

branches: 1.85.2;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

branches: 1.84.6;
__diagused police

branches: 1.83.2; 1.83.16; 1.83.20; 1.83.24;
-fno-common broke kernhist since it used commons.
Add a KERNHIST_DEFINE which is define the kernel history.
Change UVM to deal with the new usage.

Remove VM_MAP_INTRSAFE and related code. Not used since the "kmem changes".

branches: 1.81.2; 1.81.6;
- Rework uvm_anfree() into uvm_anon_freelst(), which always drops the lock.
- Free anons in uvm_anon_freelst() without lock held.
- Mechanic sync to unused loaning code.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.79.2;
Replace "malloc" in comments, remove unnecessary header inclusions.

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

branches: 1.77.2; 1.77.4; 1.77.6; 1.77.8;
A few assertions & comments.

Don't pass an unnecessary reference to uvm_loanbreak_anon().

Requested by rmind@.

Move A->K loan break code to uvm_loan.c.

Convert tsleep(&lbolt) to kpause(). Make ltsleep/mtsleep on lbolt
illegal. I examined all places where lbolt is referenced to make
sure there were pointer aliases of it passed to tsleep, but put a
KASSERT in m/ltsleep() just to be sure.

uvm_loanuobjpages(): "nfsread" -> "loanuopg" in tsleep wmesg

branches: 1.72.2; 1.72.4; 1.72.10;
initialize uvm_loanzero_object correctly after page-cache rbtree changes.

branches: 1.71.2;
listq -> listq.queue

branches: 1.70.6; 1.70.8; 1.70.10; 1.70.12;
Merge vmlocking2 to head.

branches: 1.69.2; 1.69.6;
constify pagerops.

use designated initiaizers for uvm_pagerops.

branches: 1.67.4;
Remove LOCK_ASSERT(!simple_lock_held(&foo));

branches: 1.66.4; 1.66.6; 1.66.8; 1.66.10;
Merge unobtrusive locking changes from the vmlocking branch.

branches: 1.65.4; 1.65.12;
TRUE -> true, FALSE -> false

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.63.2;
put ->K loaned pages on the page queue, so that page loaning doesn't
disturb pagedaemon/pdpolicy.

remove some __unused from function parameters.

uvm_loanbreak: transfer dirtiness of the old page to the new page,
rather than blindly mark it dirty. fix a part of PR/33513.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.59.8; 1.59.10;
from Christian Ehrhardt:
* uvm_loanzero may call uvm_analloc which will return with anon->an_lock
locked. This lock is never dropped by uvm_loanzero and AFAICS the caller
doesn't drop it either.

branches: 1.58.2; 1.58.4; 1.58.6; 1.58.8; 1.58.10;
handle "strange" filesystems like layered filesystems and tmpfs,
where pgo_get returns pages which don't belong to the uobj.
also fix an XXX in uvm_loananon and lock-unlock mismatch in uvm_loanuobj.

PR/28372, PR/32665 (Alan Barrett).

branches: 1.57.2;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

branches: 1.55.2;
Make a note about why a large function like uvm_loanentry() can be
an inline in this case.

Use ANSI function decls.

allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

introduce UVMHIST_LOANHIST and sprinkle UVMHIST_LOGs.

- prevent wired pages from being loaned, rather than just panicking.
caller should take care of failure by eg. falling back to dumb copy.
PR/23285.
- add some related assertions.

- Nuke __P().
- Drop trailing spaces.

make this compile whether DIAGNOSTIC is defined or not

when breaking a loan from uobj,
insert the replacement page into the same position
as the original page on the object memq so that
genfs_putpages (and lfs) won't be confused.

noted by Stephan Uphoff (PR/24328)

uvm_loanentry: add a missing uvmfault_unlockall.

uvm_loanuobjpages: fix a comment.

- get pages to loan out in uvm_loanuobjpages() rather than
having caller (nfsd, in this case) do so.
- tweak locking so that nfs loaned READ works on layered filesystems.

uvm_loanzero:
- after sleeping for memory, re-check if we have a page.
- put the allocated page to pageq to appease UVM_PAGE_TRKOWN.
- dequeue the page when doing ->K loan.

whitespace.

branches: 1.42.2;
- export raw page loan out routine as uvm_loanuobjpages. (for nfsd)
- put code for loan-breaking into a function, uvm_loanbreak.

Implement a minimal pager for the uvm_loanzero_object, which simply has
a "put" method which reactivates or dequeues the page.

Need for pager pointed out by enami tsugutomo.

Fix the following pathological scanario:
* User allocates ZFOD region, but does not actually touch the buffer
to fault in the pages.
* In a loop, user writes this buffer to a network socket, triggering
sosend_loan().
* uvm_loan() calls uvm_loanzero() once for each page in the loaned
region (since the pages have not yet faulted in). This causes a
page to be allocated and zero'd. The result is the kernel spends
a lot of time allocating and zero'ing pages.

This fixes creates a special object which owns a single zero'd page.
This single zero'd page is used to satisfy all loans of non-resident
ZFOD mappings.

Thanks to Allen Briggs for discovering the problem and for providing
an initial patch.

when dropping a kernel loan, if this was the last loan-to-kernel but
the page is still loaned to an anon, we should put the page back on a
paging queue. this is because while pages loaned to the kernel really
do need to stay resident (since the kernel is accessing the physical
memory directly), pages loaned to anons can be paged out just fine.
(the page will be paged out twice, first to the object and then again
to the anon, but after that the page can be reused.)

Add missing pageq lock while uvm_pagefree() is called (either directly
or indirectly). Reviewed by chuq.

branches: 1.37.2; 1.37.4;
Fetch the right page from a file even if it is mapped from middle of it.
This makes `tail -<N> <FILE> | cat > file' correctly, where <FILE> is
a regular file larger than 10Mbytes (makes tail to map part of file)
and <N> is big enough to produce output larger than 8kbytes (makes pipe
to use page loan facility). Problem reported by FUKAUMI Naoki on japanese
local mailing list.

fix locking for loaning. in general we should be looking at the page's
uobject and uanon pointers rather than at the PQ_ANON flag to determine
which lock to hold, since PQ_ANON can be clear even when the anon's lock
is the one which we should hold (if the page was loaned from an object
and then freed by the object).

add RCSIDs, and in some cases, slightly cleanup #include order

several changes prompted by loaning problems:
- fix the loaned case in uvm_pagefree().
- redo uvmexp.swpgonly accounting to work with page loaning.
add an assertion before each place we adjust uvmexp.swpgonly.
- fix uvm_km_pgremove() to always free any swap space associated with
the range being removed.
- get rid of UVM_LOAN_WIRED flag. instead, we just make sure that
pages loaned to the kernel are never on the page queues.
this allows us to assert that pages are not loaned and wired
at the same time.
- add yet more assertions.

branches: 1.33.2;
add new UVM_LOAN_WIRED flag - the memory pages loaned in TOPAGE case
are only wired if this flag is present (i.e. they are not wired by default now)
loaned pages are unloaned via new uvm_unloan(), uvm_unloananon() and
uvm_unloanpage() are no longer exported
adjust uvm_unloanpage() to unwire the pages if UVM_LOAN_WIRED is specified
mark uvm_loanuobj() and uvm_loanzero() static also in function implementation

kern/sys_pipe.c: uvm_unloanpage() --> uvm_unloan()

a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

branches: 1.31.2;
handle a locking problem where the second (or later) call in the loanentry
loop returns 0. loanentry was returning >0, but was unlocking the maps
(because of the zero). reworked to avoid this. problem reported by
chuck silvers. also clarify a comment that jdolecek asked about.

when fetching an object page to loan out, do so synchronously.

branches: 1.29.2;
remove trailing whitespace.

fix locking problem noted by Jaromir Dolecek. also, add more comments
on locking rules to make code easier to understand. locking in
uvm_loananon still needs some work on fringe cases where anon's page
is actually on loan from a uobj.

Upon Chuck Cranor request, revert rev. 1.26. There is indeed a bug in way
locking is done, but this fix is not the right way to fix it.

Remove superflous uvmfault_unlockmaps() in uvm_loan(), only call it
if uvm_loanentry() returned 0; otherwise, the unlocking would already
have been done by uvmfault_unlockall() call in uvm_loanentry().
Okay'ed by Chuck Silvers

eliminate the KERN_* error codes in favor of the traditional E* codes.
the mapping is:

KERN_SUCCESS 0
KERN_INVALID_ADDRESS EFAULT
KERN_PROTECTION_FAILURE EACCES
KERN_NO_SPACE ENOMEM
KERN_INVALID_ARGUMENT EINVAL
KERN_FAILURE various, mostly turn into KASSERTs
KERN_RESOURCE_SHORTAGE ENOMEM
KERN_NOT_RECEIVER <unused>
KERN_NO_ACCESS <unused>
KERN_PAGES_LOCKED <unused>

eliminate the VM_PAGER_* error codes in favor of the traditional E* codes.
the mapping is:

VM_PAGER_OK 0
VM_PAGER_BAD <unused>
VM_PAGER_FAIL <unused>
VM_PAGER_PEND 0 (see below)
VM_PAGER_ERROR EIO
VM_PAGER_AGAIN EAGAIN
VM_PAGER_UNLOCK EBUSY
VM_PAGER_REFAULT ERESTART

for async i/o requests, it used to be possible for the request to
be convert to sync, and the pager would return VM_PAGER_OK or VM_PAGER_PEND
to indicate whether the caller should perform post-i/o cleanup.
this is no longer allowed; pagers must now return 0 to indicate that
the async i/o was successfully started, and the caller never needs to
worry about doing the post-i/o cleanup.

branches: 1.23.2;
Change uvm_analloc() to return a locked anon, update all callers,
and fix an anon locking protocol error in uvm_loanzero().

remove include of <vm/vm.h>

remove/move more mach vm header files:

<vm/pglist.h> -> <uvm/uvm_pglist.h>
<vm/vm_inherit.h> -> <uvm/uvm_inherit.h>
<vm/vm_kern.h> -> into <uvm/uvm_extern.h>
<vm/vm_object.h> -> nothing
<vm/vm_pager.h> -> into <uvm/uvm_pager.h>

also includes a bunch of <vm/vm_page.h> include removals (due to redudancy
with <vm/vm.h>), and a scattering of other similar headers.

Use UVM_PGA_ZERO in a few (easy) places.

branches: 1.19.2;
eliminate the PMAP_NEW option by making it required for all ports.
ports which previously had no support for PMAP_NEW now implement
the pmap_k* interfaces as wrappers around the non-k versions.

Garbage collect thread_sleep()/thread_wakeup() left over from the old
Mach VM code. Also nuke iprintf(), which was no longer used anywhere.

Add proclist locking where appropriate.

Just say no to interrupt-safe maps.

Change the main comment block to indicate why PMAP_NEW (specifically,
pmap_kenter*()) is not required for {O,A}->K page loans.

add a `flags' argument to uvm_pagealloc_strat().
define a flag UVM_PGA_USERESERVE to allow non-kernel object
allocations to use pages from the reserve.
use the new flag for allocations in pmap modules.

branches: 1.14.2;
remove now >1 year old pre-release message.

cleanup/reorg:
- break anon related functions out of uvm_amap.c and put them in their own
file (uvm_anon.c). includes break up uvm_anon_init into an amap and an
an anon init function
- ensure that only functions within the amap module access amap structure
fields (add macros to amap api as needed)

branches: 1.12.2;
be consistent with locking of amaps and anons when freeing them.

shift by PAGE_SHIFT instead of multiplying or dividing by PAGE_SIZE.

remove unused share map code from UVM:
- update uvm_faultinfo's rvaddr to orig_rvaddr to match changes from
uvm_fault.h

Merge paddr_t changes into the main branch.

branches: 1.8.2;
Remove inclusions of syscall (and syscall argument) related header files;
we don't need them here.

remove tmpwire arg from uvm_pagewire() -- it isn't needed anymore.
noted by chuck s.

KNF.

- add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.

restore rcsids

fix typoes in locking.

RCS ID police.

branches: 1.1.1;
Initial revision

uvm_loanuobjpages():

- there are no pages to unbusy in the error case
- always clear the caller's page array

uvm_loanuobjpages():

- vmobjlock is shared between tmpfs vnodes and UAOs now
- split into two routines, to simplify
- fix error recovery

Start trying to reduce cache misses on vm_page during fault processing.

- Make PGO_LOCKED getpages imply PGO_NOBUSY and remove the latter. Mark
pages busy only when there's actually I/O to do.

- When doing COW on a uvm_object, don't mess with neighbouring pages. In
all likelyhood they're already entered.

- Don't mess with neighbouring VAs that have existing mappings as replacing
those mappings with same can be quite costly.

- Don't enqueue pages for neighbour faults unless not enqueued already, and
don't activate centre pages unless uvmpdpol says its useful.

Also:

- Make PGO_LOCKED getpages on UAOs work more like vnodes: do gang lookup in
the radix tree, and don't allocate new pages.

- Fix many assertion failures around faults/loans with tmpfs.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Go back to freeing struct vm_anon one by one. There may have been an
advantage circa ~2008 but there isn't now.

Tweak the March 14th change to make page waits interlocked by pg->interlock.
Remove unneeded changes and only deal with the PQ_WANTED flag, to exclude
possible bugs.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

uvm_unloanpage(): fix a screwup in previous. slock must be set NULL if
it can't be acquired.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.93.2;
- Add and use wrapper functions that take and acquire page interlocks, and pairs
of page interlocks. Require that the page interlock be held over calls to
uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state. Rather than
updating the global state synchronously, set an intended state on
individual pages (active, inactive, enqueued, dequeued) while holding the
page interlock. After the interlock is released put the pages on a 128
entry per-CPU queue for their state changes to be made real in batch.
This results in in a ~400 fold decrease in contention on my test system.
Proposed on tech-kern but modified to use the page interlock rather than
atomics to synchronise as it's much easier to maintain that way, and
cheaper.

PR kern/54783: t_mmap crahes the kernel

- Fix various locking & sequencing errors with breaking loans.

- Don't call uvm_pageremove_tree() while holding pg->interlock as radixtree
can take further locks when freeing nodes.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Don't call uvm_pagedequeue() while holding pg->interlock.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

- Adjust uvmexp.swpgonly with atomics, and make uvm_swap_data_lock static.
- A bit more __cacheline_aligned on mutexes.

branches: 1.87.2;
add the KASSERT() for loan_count wrap-around to all places which increase it

detect wraparound when bumping page wire_count and loan_count

branches: 1.85.2;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

branches: 1.84.6;
__diagused police

branches: 1.83.2; 1.83.16; 1.83.20; 1.83.24;
-fno-common broke kernhist since it used commons.
Add a KERNHIST_DEFINE which is define the kernel history.
Change UVM to deal with the new usage.

Remove VM_MAP_INTRSAFE and related code. Not used since the "kmem changes".

branches: 1.81.2; 1.81.6;
- Rework uvm_anfree() into uvm_anon_freelst(), which always drops the lock.
- Free anons in uvm_anon_freelst() without lock held.
- Mechanic sync to unused loaning code.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.79.2;
Replace "malloc" in comments, remove unnecessary header inclusions.

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

branches: 1.77.2; 1.77.4; 1.77.6; 1.77.8;
A few assertions & comments.

Don't pass an unnecessary reference to uvm_loanbreak_anon().

Requested by rmind@.

Move A->K loan break code to uvm_loan.c.

Convert tsleep(&lbolt) to kpause(). Make ltsleep/mtsleep on lbolt
illegal. I examined all places where lbolt is referenced to make
sure there were pointer aliases of it passed to tsleep, but put a
KASSERT in m/ltsleep() just to be sure.

uvm_loanuobjpages(): "nfsread" -> "loanuopg" in tsleep wmesg

branches: 1.72.2; 1.72.4; 1.72.10;
initialize uvm_loanzero_object correctly after page-cache rbtree changes.

branches: 1.71.2;
listq -> listq.queue

branches: 1.70.6; 1.70.8; 1.70.10; 1.70.12;
Merge vmlocking2 to head.

branches: 1.69.2; 1.69.6;
constify pagerops.

use designated initiaizers for uvm_pagerops.

branches: 1.67.4;
Remove LOCK_ASSERT(!simple_lock_held(&foo));

branches: 1.66.4; 1.66.6; 1.66.8; 1.66.10;
Merge unobtrusive locking changes from the vmlocking branch.

branches: 1.65.4; 1.65.12;
TRUE -> true, FALSE -> false

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.63.2;
put ->K loaned pages on the page queue, so that page loaning doesn't
disturb pagedaemon/pdpolicy.

remove some __unused from function parameters.

uvm_loanbreak: transfer dirtiness of the old page to the new page,
rather than blindly mark it dirty. fix a part of PR/33513.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.59.8; 1.59.10;
from Christian Ehrhardt:
* uvm_loanzero may call uvm_analloc which will return with anon->an_lock
locked. This lock is never dropped by uvm_loanzero and AFAICS the caller
doesn't drop it either.

branches: 1.58.2; 1.58.4; 1.58.6; 1.58.8; 1.58.10;
handle "strange" filesystems like layered filesystems and tmpfs,
where pgo_get returns pages which don't belong to the uobj.
also fix an XXX in uvm_loananon and lock-unlock mismatch in uvm_loanuobj.

PR/28372, PR/32665 (Alan Barrett).

branches: 1.57.2;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

branches: 1.55.2;
Make a note about why a large function like uvm_loanentry() can be
an inline in this case.

Use ANSI function decls.

allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

introduce UVMHIST_LOANHIST and sprinkle UVMHIST_LOGs.

- prevent wired pages from being loaned, rather than just panicking.
caller should take care of failure by eg. falling back to dumb copy.
PR/23285.
- add some related assertions.

- Nuke __P().
- Drop trailing spaces.

make this compile whether DIAGNOSTIC is defined or not

when breaking a loan from uobj,
insert the replacement page into the same position
as the original page on the object memq so that
genfs_putpages (and lfs) won't be confused.

noted by Stephan Uphoff (PR/24328)

uvm_loanentry: add a missing uvmfault_unlockall.

uvm_loanuobjpages: fix a comment.

- get pages to loan out in uvm_loanuobjpages() rather than
having caller (nfsd, in this case) do so.
- tweak locking so that nfs loaned READ works on layered filesystems.

uvm_loanzero:
- after sleeping for memory, re-check if we have a page.
- put the allocated page to pageq to appease UVM_PAGE_TRKOWN.
- dequeue the page when doing ->K loan.

whitespace.

branches: 1.42.2;
- export raw page loan out routine as uvm_loanuobjpages. (for nfsd)
- put code for loan-breaking into a function, uvm_loanbreak.

Implement a minimal pager for the uvm_loanzero_object, which simply has
a "put" method which reactivates or dequeues the page.

Need for pager pointed out by enami tsugutomo.

Fix the following pathological scanario:
* User allocates ZFOD region, but does not actually touch the buffer
to fault in the pages.
* In a loop, user writes this buffer to a network socket, triggering
sosend_loan().
* uvm_loan() calls uvm_loanzero() once for each page in the loaned
region (since the pages have not yet faulted in). This causes a
page to be allocated and zero'd. The result is the kernel spends
a lot of time allocating and zero'ing pages.

This fixes creates a special object which owns a single zero'd page.
This single zero'd page is used to satisfy all loans of non-resident
ZFOD mappings.

Thanks to Allen Briggs for discovering the problem and for providing
an initial patch.

when dropping a kernel loan, if this was the last loan-to-kernel but
the page is still loaned to an anon, we should put the page back on a
paging queue. this is because while pages loaned to the kernel really
do need to stay resident (since the kernel is accessing the physical
memory directly), pages loaned to anons can be paged out just fine.
(the page will be paged out twice, first to the object and then again
to the anon, but after that the page can be reused.)

Add missing pageq lock while uvm_pagefree() is called (either directly
or indirectly). Reviewed by chuq.

branches: 1.37.2; 1.37.4;
Fetch the right page from a file even if it is mapped from middle of it.
This makes `tail -<N> <FILE> | cat > file' correctly, where <FILE> is
a regular file larger than 10Mbytes (makes tail to map part of file)
and <N> is big enough to produce output larger than 8kbytes (makes pipe
to use page loan facility). Problem reported by FUKAUMI Naoki on japanese
local mailing list.

fix locking for loaning. in general we should be looking at the page's
uobject and uanon pointers rather than at the PQ_ANON flag to determine
which lock to hold, since PQ_ANON can be clear even when the anon's lock
is the one which we should hold (if the page was loaned from an object
and then freed by the object).

add RCSIDs, and in some cases, slightly cleanup #include order

several changes prompted by loaning problems:
- fix the loaned case in uvm_pagefree().
- redo uvmexp.swpgonly accounting to work with page loaning.
add an assertion before each place we adjust uvmexp.swpgonly.
- fix uvm_km_pgremove() to always free any swap space associated with
the range being removed.
- get rid of UVM_LOAN_WIRED flag. instead, we just make sure that
pages loaned to the kernel are never on the page queues.
this allows us to assert that pages are not loaned and wired
at the same time.
- add yet more assertions.

branches: 1.33.2;
add new UVM_LOAN_WIRED flag - the memory pages loaned in TOPAGE case
are only wired if this flag is present (i.e. they are not wired by default now)
loaned pages are unloaned via new uvm_unloan(), uvm_unloananon() and
uvm_unloanpage() are no longer exported
adjust uvm_unloanpage() to unwire the pages if UVM_LOAN_WIRED is specified
mark uvm_loanuobj() and uvm_loanzero() static also in function implementation

kern/sys_pipe.c: uvm_unloanpage() --> uvm_unloan()

a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

branches: 1.31.2;
handle a locking problem where the second (or later) call in the loanentry
loop returns 0. loanentry was returning >0, but was unlocking the maps
(because of the zero). reworked to avoid this. problem reported by
chuck silvers. also clarify a comment that jdolecek asked about.

when fetching an object page to loan out, do so synchronously.

branches: 1.29.2;
remove trailing whitespace.

fix locking problem noted by Jaromir Dolecek. also, add more comments
on locking rules to make code easier to understand. locking in
uvm_loananon still needs some work on fringe cases where anon's page
is actually on loan from a uobj.

Upon Chuck Cranor request, revert rev. 1.26. There is indeed a bug in way
locking is done, but this fix is not the right way to fix it.

Remove superflous uvmfault_unlockmaps() in uvm_loan(), only call it
if uvm_loanentry() returned 0; otherwise, the unlocking would already
have been done by uvmfault_unlockall() call in uvm_loanentry().
Okay'ed by Chuck Silvers

eliminate the KERN_* error codes in favor of the traditional E* codes.
the mapping is:

KERN_SUCCESS 0
KERN_INVALID_ADDRESS EFAULT
KERN_PROTECTION_FAILURE EACCES
KERN_NO_SPACE ENOMEM
KERN_INVALID_ARGUMENT EINVAL
KERN_FAILURE various, mostly turn into KASSERTs
KERN_RESOURCE_SHORTAGE ENOMEM
KERN_NOT_RECEIVER <unused>
KERN_NO_ACCESS <unused>
KERN_PAGES_LOCKED <unused>

eliminate the VM_PAGER_* error codes in favor of the traditional E* codes.
the mapping is:

VM_PAGER_OK 0
VM_PAGER_BAD <unused>
VM_PAGER_FAIL <unused>
VM_PAGER_PEND 0 (see below)
VM_PAGER_ERROR EIO
VM_PAGER_AGAIN EAGAIN
VM_PAGER_UNLOCK EBUSY
VM_PAGER_REFAULT ERESTART

for async i/o requests, it used to be possible for the request to
be convert to sync, and the pager would return VM_PAGER_OK or VM_PAGER_PEND
to indicate whether the caller should perform post-i/o cleanup.
this is no longer allowed; pagers must now return 0 to indicate that
the async i/o was successfully started, and the caller never needs to
worry about doing the post-i/o cleanup.

branches: 1.23.2;
Change uvm_analloc() to return a locked anon, update all callers,
and fix an anon locking protocol error in uvm_loanzero().

remove include of <vm/vm.h>

remove/move more mach vm header files:

<vm/pglist.h> -> <uvm/uvm_pglist.h>
<vm/vm_inherit.h> -> <uvm/uvm_inherit.h>
<vm/vm_kern.h> -> into <uvm/uvm_extern.h>
<vm/vm_object.h> -> nothing
<vm/vm_pager.h> -> into <uvm/uvm_pager.h>

also includes a bunch of <vm/vm_page.h> include removals (due to redudancy
with <vm/vm.h>), and a scattering of other similar headers.

Use UVM_PGA_ZERO in a few (easy) places.

branches: 1.19.2;
eliminate the PMAP_NEW option by making it required for all ports.
ports which previously had no support for PMAP_NEW now implement
the pmap_k* interfaces as wrappers around the non-k versions.

Garbage collect thread_sleep()/thread_wakeup() left over from the old
Mach VM code. Also nuke iprintf(), which was no longer used anywhere.

Add proclist locking where appropriate.

Just say no to interrupt-safe maps.

Change the main comment block to indicate why PMAP_NEW (specifically,
pmap_kenter*()) is not required for {O,A}->K page loans.

add a `flags' argument to uvm_pagealloc_strat().
define a flag UVM_PGA_USERESERVE to allow non-kernel object
allocations to use pages from the reserve.
use the new flag for allocations in pmap modules.

branches: 1.14.2;
remove now >1 year old pre-release message.

cleanup/reorg:
- break anon related functions out of uvm_amap.c and put them in their own
file (uvm_anon.c). includes break up uvm_anon_init into an amap and an
an anon init function
- ensure that only functions within the amap module access amap structure
fields (add macros to amap api as needed)

branches: 1.12.2;
be consistent with locking of amaps and anons when freeing them.

shift by PAGE_SHIFT instead of multiplying or dividing by PAGE_SIZE.

remove unused share map code from UVM:
- update uvm_faultinfo's rvaddr to orig_rvaddr to match changes from
uvm_fault.h

Merge paddr_t changes into the main branch.

branches: 1.8.2;
Remove inclusions of syscall (and syscall argument) related header files;
we don't need them here.

remove tmpwire arg from uvm_pagewire() -- it isn't needed anymore.
noted by chuck s.

KNF.

- add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.

restore rcsids

fix typoes in locking.

RCS ID police.

branches: 1.1.1;
Initial revision

uvm_loanuobjpages():

- vmobjlock is shared between tmpfs vnodes and UAOs now
- split into two routines, to simplify
- fix error recovery

Start trying to reduce cache misses on vm_page during fault processing.

- Make PGO_LOCKED getpages imply PGO_NOBUSY and remove the latter. Mark
pages busy only when there's actually I/O to do.

- When doing COW on a uvm_object, don't mess with neighbouring pages. In
all likelyhood they're already entered.

- Don't mess with neighbouring VAs that have existing mappings as replacing
those mappings with same can be quite costly.

- Don't enqueue pages for neighbour faults unless not enqueued already, and
don't activate centre pages unless uvmpdpol says its useful.

Also:

- Make PGO_LOCKED getpages on UAOs work more like vnodes: do gang lookup in
the radix tree, and don't allocate new pages.

- Fix many assertion failures around faults/loans with tmpfs.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Go back to freeing struct vm_anon one by one. There may have been an
advantage circa ~2008 but there isn't now.

Tweak the March 14th change to make page waits interlocked by pg->interlock.
Remove unneeded changes and only deal with the PQ_WANTED flag, to exclude
possible bugs.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

uvm_unloanpage(): fix a screwup in previous. slock must be set NULL if
it can't be acquired.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.93.2;
- Add and use wrapper functions that take and acquire page interlocks, and pairs
of page interlocks. Require that the page interlock be held over calls to
uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state. Rather than
updating the global state synchronously, set an intended state on
individual pages (active, inactive, enqueued, dequeued) while holding the
page interlock. After the interlock is released put the pages on a 128
entry per-CPU queue for their state changes to be made real in batch.
This results in in a ~400 fold decrease in contention on my test system.
Proposed on tech-kern but modified to use the page interlock rather than
atomics to synchronise as it's much easier to maintain that way, and
cheaper.

PR kern/54783: t_mmap crahes the kernel

- Fix various locking & sequencing errors with breaking loans.

- Don't call uvm_pageremove_tree() while holding pg->interlock as radixtree
can take further locks when freeing nodes.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Don't call uvm_pagedequeue() while holding pg->interlock.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

- Adjust uvmexp.swpgonly with atomics, and make uvm_swap_data_lock static.
- A bit more __cacheline_aligned on mutexes.

branches: 1.87.2;
add the KASSERT() for loan_count wrap-around to all places which increase it

detect wraparound when bumping page wire_count and loan_count

branches: 1.85.2;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

branches: 1.84.6;
__diagused police

branches: 1.83.2; 1.83.16; 1.83.20; 1.83.24;
-fno-common broke kernhist since it used commons.
Add a KERNHIST_DEFINE which is define the kernel history.
Change UVM to deal with the new usage.

Remove VM_MAP_INTRSAFE and related code. Not used since the "kmem changes".

branches: 1.81.2; 1.81.6;
- Rework uvm_anfree() into uvm_anon_freelst(), which always drops the lock.
- Free anons in uvm_anon_freelst() without lock held.
- Mechanic sync to unused loaning code.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.79.2;
Replace "malloc" in comments, remove unnecessary header inclusions.

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

branches: 1.77.2; 1.77.4; 1.77.6; 1.77.8;
A few assertions & comments.

Don't pass an unnecessary reference to uvm_loanbreak_anon().

Requested by rmind@.

Move A->K loan break code to uvm_loan.c.

Convert tsleep(&lbolt) to kpause(). Make ltsleep/mtsleep on lbolt
illegal. I examined all places where lbolt is referenced to make
sure there were pointer aliases of it passed to tsleep, but put a
KASSERT in m/ltsleep() just to be sure.

uvm_loanuobjpages(): "nfsread" -> "loanuopg" in tsleep wmesg

branches: 1.72.2; 1.72.4; 1.72.10;
initialize uvm_loanzero_object correctly after page-cache rbtree changes.

branches: 1.71.2;
listq -> listq.queue

branches: 1.70.6; 1.70.8; 1.70.10; 1.70.12;
Merge vmlocking2 to head.

branches: 1.69.2; 1.69.6;
constify pagerops.

use designated initiaizers for uvm_pagerops.

branches: 1.67.4;
Remove LOCK_ASSERT(!simple_lock_held(&foo));

branches: 1.66.4; 1.66.6; 1.66.8; 1.66.10;
Merge unobtrusive locking changes from the vmlocking branch.

branches: 1.65.4; 1.65.12;
TRUE -> true, FALSE -> false

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.63.2;
put ->K loaned pages on the page queue, so that page loaning doesn't
disturb pagedaemon/pdpolicy.

remove some __unused from function parameters.

uvm_loanbreak: transfer dirtiness of the old page to the new page,
rather than blindly mark it dirty. fix a part of PR/33513.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.59.8; 1.59.10;
from Christian Ehrhardt:
* uvm_loanzero may call uvm_analloc which will return with anon->an_lock
locked. This lock is never dropped by uvm_loanzero and AFAICS the caller
doesn't drop it either.

branches: 1.58.2; 1.58.4; 1.58.6; 1.58.8; 1.58.10;
handle "strange" filesystems like layered filesystems and tmpfs,
where pgo_get returns pages which don't belong to the uobj.
also fix an XXX in uvm_loananon and lock-unlock mismatch in uvm_loanuobj.

PR/28372, PR/32665 (Alan Barrett).

branches: 1.57.2;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

branches: 1.55.2;
Make a note about why a large function like uvm_loanentry() can be
an inline in this case.

Use ANSI function decls.

allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

introduce UVMHIST_LOANHIST and sprinkle UVMHIST_LOGs.

- prevent wired pages from being loaned, rather than just panicking.
caller should take care of failure by eg. falling back to dumb copy.
PR/23285.
- add some related assertions.

- Nuke __P().
- Drop trailing spaces.

make this compile whether DIAGNOSTIC is defined or not

when breaking a loan from uobj,
insert the replacement page into the same position
as the original page on the object memq so that
genfs_putpages (and lfs) won't be confused.

noted by Stephan Uphoff (PR/24328)

uvm_loanentry: add a missing uvmfault_unlockall.

uvm_loanuobjpages: fix a comment.

- get pages to loan out in uvm_loanuobjpages() rather than
having caller (nfsd, in this case) do so.
- tweak locking so that nfs loaned READ works on layered filesystems.

uvm_loanzero:
- after sleeping for memory, re-check if we have a page.
- put the allocated page to pageq to appease UVM_PAGE_TRKOWN.
- dequeue the page when doing ->K loan.

whitespace.

branches: 1.42.2;
- export raw page loan out routine as uvm_loanuobjpages. (for nfsd)
- put code for loan-breaking into a function, uvm_loanbreak.

Implement a minimal pager for the uvm_loanzero_object, which simply has
a "put" method which reactivates or dequeues the page.

Need for pager pointed out by enami tsugutomo.

Fix the following pathological scanario:
* User allocates ZFOD region, but does not actually touch the buffer
to fault in the pages.
* In a loop, user writes this buffer to a network socket, triggering
sosend_loan().
* uvm_loan() calls uvm_loanzero() once for each page in the loaned
region (since the pages have not yet faulted in). This causes a
page to be allocated and zero'd. The result is the kernel spends
a lot of time allocating and zero'ing pages.

This fixes creates a special object which owns a single zero'd page.
This single zero'd page is used to satisfy all loans of non-resident
ZFOD mappings.

Thanks to Allen Briggs for discovering the problem and for providing
an initial patch.

when dropping a kernel loan, if this was the last loan-to-kernel but
the page is still loaned to an anon, we should put the page back on a
paging queue. this is because while pages loaned to the kernel really
do need to stay resident (since the kernel is accessing the physical
memory directly), pages loaned to anons can be paged out just fine.
(the page will be paged out twice, first to the object and then again
to the anon, but after that the page can be reused.)

Add missing pageq lock while uvm_pagefree() is called (either directly
or indirectly). Reviewed by chuq.

branches: 1.37.2; 1.37.4;
Fetch the right page from a file even if it is mapped from middle of it.
This makes `tail -<N> <FILE> | cat > file' correctly, where <FILE> is
a regular file larger than 10Mbytes (makes tail to map part of file)
and <N> is big enough to produce output larger than 8kbytes (makes pipe
to use page loan facility). Problem reported by FUKAUMI Naoki on japanese
local mailing list.

fix locking for loaning. in general we should be looking at the page's
uobject and uanon pointers rather than at the PQ_ANON flag to determine
which lock to hold, since PQ_ANON can be clear even when the anon's lock
is the one which we should hold (if the page was loaned from an object
and then freed by the object).

add RCSIDs, and in some cases, slightly cleanup #include order

several changes prompted by loaning problems:
- fix the loaned case in uvm_pagefree().
- redo uvmexp.swpgonly accounting to work with page loaning.
add an assertion before each place we adjust uvmexp.swpgonly.
- fix uvm_km_pgremove() to always free any swap space associated with
the range being removed.
- get rid of UVM_LOAN_WIRED flag. instead, we just make sure that
pages loaned to the kernel are never on the page queues.
this allows us to assert that pages are not loaned and wired
at the same time.
- add yet more assertions.

branches: 1.33.2;
add new UVM_LOAN_WIRED flag - the memory pages loaned in TOPAGE case
are only wired if this flag is present (i.e. they are not wired by default now)
loaned pages are unloaned via new uvm_unloan(), uvm_unloananon() and
uvm_unloanpage() are no longer exported
adjust uvm_unloanpage() to unwire the pages if UVM_LOAN_WIRED is specified
mark uvm_loanuobj() and uvm_loanzero() static also in function implementation

kern/sys_pipe.c: uvm_unloanpage() --> uvm_unloan()

a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

branches: 1.31.2;
handle a locking problem where the second (or later) call in the loanentry
loop returns 0. loanentry was returning >0, but was unlocking the maps
(because of the zero). reworked to avoid this. problem reported by
chuck silvers. also clarify a comment that jdolecek asked about.

when fetching an object page to loan out, do so synchronously.

branches: 1.29.2;
remove trailing whitespace.

fix locking problem noted by Jaromir Dolecek. also, add more comments
on locking rules to make code easier to understand. locking in
uvm_loananon still needs some work on fringe cases where anon's page
is actually on loan from a uobj.

Upon Chuck Cranor request, revert rev. 1.26. There is indeed a bug in way
locking is done, but this fix is not the right way to fix it.

Remove superflous uvmfault_unlockmaps() in uvm_loan(), only call it
if uvm_loanentry() returned 0; otherwise, the unlocking would already
have been done by uvmfault_unlockall() call in uvm_loanentry().
Okay'ed by Chuck Silvers

eliminate the KERN_* error codes in favor of the traditional E* codes.
the mapping is:

KERN_SUCCESS 0
KERN_INVALID_ADDRESS EFAULT
KERN_PROTECTION_FAILURE EACCES
KERN_NO_SPACE ENOMEM
KERN_INVALID_ARGUMENT EINVAL
KERN_FAILURE various, mostly turn into KASSERTs
KERN_RESOURCE_SHORTAGE ENOMEM
KERN_NOT_RECEIVER <unused>
KERN_NO_ACCESS <unused>
KERN_PAGES_LOCKED <unused>

eliminate the VM_PAGER_* error codes in favor of the traditional E* codes.
the mapping is:

VM_PAGER_OK 0
VM_PAGER_BAD <unused>
VM_PAGER_FAIL <unused>
VM_PAGER_PEND 0 (see below)
VM_PAGER_ERROR EIO
VM_PAGER_AGAIN EAGAIN
VM_PAGER_UNLOCK EBUSY
VM_PAGER_REFAULT ERESTART

for async i/o requests, it used to be possible for the request to
be convert to sync, and the pager would return VM_PAGER_OK or VM_PAGER_PEND
to indicate whether the caller should perform post-i/o cleanup.
this is no longer allowed; pagers must now return 0 to indicate that
the async i/o was successfully started, and the caller never needs to
worry about doing the post-i/o cleanup.

branches: 1.23.2;
Change uvm_analloc() to return a locked anon, update all callers,
and fix an anon locking protocol error in uvm_loanzero().

remove include of <vm/vm.h>

remove/move more mach vm header files:

<vm/pglist.h> -> <uvm/uvm_pglist.h>
<vm/vm_inherit.h> -> <uvm/uvm_inherit.h>
<vm/vm_kern.h> -> into <uvm/uvm_extern.h>
<vm/vm_object.h> -> nothing
<vm/vm_pager.h> -> into <uvm/uvm_pager.h>

also includes a bunch of <vm/vm_page.h> include removals (due to redudancy
with <vm/vm.h>), and a scattering of other similar headers.

Use UVM_PGA_ZERO in a few (easy) places.

branches: 1.19.2;
eliminate the PMAP_NEW option by making it required for all ports.
ports which previously had no support for PMAP_NEW now implement
the pmap_k* interfaces as wrappers around the non-k versions.

Garbage collect thread_sleep()/thread_wakeup() left over from the old
Mach VM code. Also nuke iprintf(), which was no longer used anywhere.

Add proclist locking where appropriate.

Just say no to interrupt-safe maps.

Change the main comment block to indicate why PMAP_NEW (specifically,
pmap_kenter*()) is not required for {O,A}->K page loans.

add a `flags' argument to uvm_pagealloc_strat().
define a flag UVM_PGA_USERESERVE to allow non-kernel object
allocations to use pages from the reserve.
use the new flag for allocations in pmap modules.

branches: 1.14.2;
remove now >1 year old pre-release message.

cleanup/reorg:
- break anon related functions out of uvm_amap.c and put them in their own
file (uvm_anon.c). includes break up uvm_anon_init into an amap and an
an anon init function
- ensure that only functions within the amap module access amap structure
fields (add macros to amap api as needed)

branches: 1.12.2;
be consistent with locking of amaps and anons when freeing them.

shift by PAGE_SHIFT instead of multiplying or dividing by PAGE_SIZE.

remove unused share map code from UVM:
- update uvm_faultinfo's rvaddr to orig_rvaddr to match changes from
uvm_fault.h

Merge paddr_t changes into the main branch.

branches: 1.8.2;
Remove inclusions of syscall (and syscall argument) related header files;
we don't need them here.

remove tmpwire arg from uvm_pagewire() -- it isn't needed anymore.
noted by chuck s.

KNF.

- add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.

restore rcsids

fix typoes in locking.

RCS ID police.

branches: 1.1.1;
Initial revision

Start trying to reduce cache misses on vm_page during fault processing.

- Make PGO_LOCKED getpages imply PGO_NOBUSY and remove the latter. Mark
pages busy only when there's actually I/O to do.

- When doing COW on a uvm_object, don't mess with neighbouring pages. In
all likelyhood they're already entered.

- Don't mess with neighbouring VAs that have existing mappings as replacing
those mappings with same can be quite costly.

- Don't enqueue pages for neighbour faults unless not enqueued already, and
don't activate centre pages unless uvmpdpol says its useful.

Also:

- Make PGO_LOCKED getpages on UAOs work more like vnodes: do gang lookup in
the radix tree, and don't allocate new pages.

- Fix many assertion failures around faults/loans with tmpfs.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Go back to freeing struct vm_anon one by one. There may have been an
advantage circa ~2008 but there isn't now.

Tweak the March 14th change to make page waits interlocked by pg->interlock.
Remove unneeded changes and only deal with the PQ_WANTED flag, to exclude
possible bugs.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

uvm_unloanpage(): fix a screwup in previous. slock must be set NULL if
it can't be acquired.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.93.2;
- Add and use wrapper functions that take and acquire page interlocks, and pairs
of page interlocks. Require that the page interlock be held over calls to
uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state. Rather than
updating the global state synchronously, set an intended state on
individual pages (active, inactive, enqueued, dequeued) while holding the
page interlock. After the interlock is released put the pages on a 128
entry per-CPU queue for their state changes to be made real in batch.
This results in in a ~400 fold decrease in contention on my test system.
Proposed on tech-kern but modified to use the page interlock rather than
atomics to synchronise as it's much easier to maintain that way, and
cheaper.

PR kern/54783: t_mmap crahes the kernel

- Fix various locking & sequencing errors with breaking loans.

- Don't call uvm_pageremove_tree() while holding pg->interlock as radixtree
can take further locks when freeing nodes.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Don't call uvm_pagedequeue() while holding pg->interlock.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

- Adjust uvmexp.swpgonly with atomics, and make uvm_swap_data_lock static.
- A bit more __cacheline_aligned on mutexes.

branches: 1.87.2;
add the KASSERT() for loan_count wrap-around to all places which increase it

detect wraparound when bumping page wire_count and loan_count

branches: 1.85.2;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

branches: 1.84.6;
__diagused police

branches: 1.83.2; 1.83.16; 1.83.20; 1.83.24;
-fno-common broke kernhist since it used commons.
Add a KERNHIST_DEFINE which is define the kernel history.
Change UVM to deal with the new usage.

Remove VM_MAP_INTRSAFE and related code. Not used since the "kmem changes".

branches: 1.81.2; 1.81.6;
- Rework uvm_anfree() into uvm_anon_freelst(), which always drops the lock.
- Free anons in uvm_anon_freelst() without lock held.
- Mechanic sync to unused loaning code.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.79.2;
Replace "malloc" in comments, remove unnecessary header inclusions.

udpate license clauses on my code to match the new-style BSD licenses.
based on diff that rmind@ sent me.

no functional change with this commit.

branches: 1.77.2; 1.77.4; 1.77.6; 1.77.8;
A few assertions & comments.

Don't pass an unnecessary reference to uvm_loanbreak_anon().

Requested by rmind@.

Move A->K loan break code to uvm_loan.c.

Convert tsleep(&lbolt) to kpause(). Make ltsleep/mtsleep on lbolt
illegal. I examined all places where lbolt is referenced to make
sure there were pointer aliases of it passed to tsleep, but put a
KASSERT in m/ltsleep() just to be sure.

uvm_loanuobjpages(): "nfsread" -> "loanuopg" in tsleep wmesg

branches: 1.72.2; 1.72.4; 1.72.10;
initialize uvm_loanzero_object correctly after page-cache rbtree changes.

branches: 1.71.2;
listq -> listq.queue

branches: 1.70.6; 1.70.8; 1.70.10; 1.70.12;
Merge vmlocking2 to head.

branches: 1.69.2; 1.69.6;
constify pagerops.

use designated initiaizers for uvm_pagerops.

branches: 1.67.4;
Remove LOCK_ASSERT(!simple_lock_held(&foo));

branches: 1.66.4; 1.66.6; 1.66.8; 1.66.10;
Merge unobtrusive locking changes from the vmlocking branch.

branches: 1.65.4; 1.65.12;
TRUE -> true, FALSE -> false

Replace the Mach-derived boolean_t type with the C99 bool type. A
future commit will replace use of TRUE and FALSE with true and false.

branches: 1.63.2;
put ->K loaned pages on the page queue, so that page loaning doesn't
disturb pagedaemon/pdpolicy.

remove some __unused from function parameters.

uvm_loanbreak: transfer dirtiness of the old page to the new page,
rather than blindly mark it dirty. fix a part of PR/33513.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.59.8; 1.59.10;
from Christian Ehrhardt:
* uvm_loanzero may call uvm_analloc which will return with anon->an_lock
locked. This lock is never dropped by uvm_loanzero and AFAICS the caller
doesn't drop it either.

branches: 1.58.2; 1.58.4; 1.58.6; 1.58.8; 1.58.10;
handle "strange" filesystems like layered filesystems and tmpfs,
where pgo_get returns pages which don't belong to the uobj.
also fix an XXX in uvm_loananon and lock-unlock mismatch in uvm_loanuobj.

PR/28372, PR/32665 (Alan Barrett).

branches: 1.57.2;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

branches: 1.55.2;
Make a note about why a large function like uvm_loanentry() can be
an inline in this case.

Use ANSI function decls.

allocate anons on-demand, rather than reserving static amount of
them on boot/swapon.

introduce UVMHIST_LOANHIST and sprinkle UVMHIST_LOGs.

- prevent wired pages from being loaned, rather than just panicking.
caller should take care of failure by eg. falling back to dumb copy.
PR/23285.
- add some related assertions.

- Nuke __P().
- Drop trailing spaces.

make this compile whether DIAGNOSTIC is defined or not

when breaking a loan from uobj,
insert the replacement page into the same position
as the original page on the object memq so that
genfs_putpages (and lfs) won't be confused.

noted by Stephan Uphoff (PR/24328)

uvm_loanentry: add a missing uvmfault_unlockall.

uvm_loanuobjpages: fix a comment.

- get pages to loan out in uvm_loanuobjpages() rather than
having caller (nfsd, in this case) do so.
- tweak locking so that nfs loaned READ works on layered filesystems.

uvm_loanzero:
- after sleeping for memory, re-check if we have a page.
- put the allocated page to pageq to appease UVM_PAGE_TRKOWN.
- dequeue the page when doing ->K loan.

whitespace.

branches: 1.42.2;
- export raw page loan out routine as uvm_loanuobjpages. (for nfsd)
- put code for loan-breaking into a function, uvm_loanbreak.

Implement a minimal pager for the uvm_loanzero_object, which simply has
a "put" method which reactivates or dequeues the page.

Need for pager pointed out by enami tsugutomo.

Fix the following pathological scanario:
* User allocates ZFOD region, but does not actually touch the buffer
to fault in the pages.
* In a loop, user writes this buffer to a network socket, triggering
sosend_loan().
* uvm_loan() calls uvm_loanzero() once for each page in the loaned
region (since the pages have not yet faulted in). This causes a
page to be allocated and zero'd. The result is the kernel spends
a lot of time allocating and zero'ing pages.

This fixes creates a special object which owns a single zero'd page.
This single zero'd page is used to satisfy all loans of non-resident
ZFOD mappings.

Thanks to Allen Briggs for discovering the problem and for providing
an initial patch.

when dropping a kernel loan, if this was the last loan-to-kernel but
the page is still loaned to an anon, we should put the page back on a
paging queue. this is because while pages loaned to the kernel really
do need to stay resident (since the kernel is accessing the physical
memory directly), pages loaned to anons can be paged out just fine.
(the page will be paged out twice, first to the object and then again
to the anon, but after that the page can be reused.)

Add missing pageq lock while uvm_pagefree() is called (either directly
or indirectly). Reviewed by chuq.

branches: 1.37.2; 1.37.4;
Fetch the right page from a file even if it is mapped from middle of it.
This makes `tail -<N> <FILE> | cat > file' correctly, where <FILE> is
a regular file larger than 10Mbytes (makes tail to map part of file)
and <N> is big enough to produce output larger than 8kbytes (makes pipe
to use page loan facility). Problem reported by FUKAUMI Naoki on japanese
local mailing list.

fix locking for loaning. in general we should be looking at the page's
uobject and uanon pointers rather than at the PQ_ANON flag to determine
which lock to hold, since PQ_ANON can be clear even when the anon's lock
is the one which we should hold (if the page was loaned from an object
and then freed by the object).

add RCSIDs, and in some cases, slightly cleanup #include order

several changes prompted by loaning problems:
- fix the loaned case in uvm_pagefree().
- redo uvmexp.swpgonly accounting to work with page loaning.
add an assertion before each place we adjust uvmexp.swpgonly.
- fix uvm_km_pgremove() to always free any swap space associated with
the range being removed.
- get rid of UVM_LOAN_WIRED flag. instead, we just make sure that
pages loaned to the kernel are never on the page queues.
this allows us to assert that pages are not loaned and wired
at the same time.
- add yet more assertions.

branches: 1.33.2;
add new UVM_LOAN_WIRED flag - the memory pages loaned in TOPAGE case
are only wired if this flag is present (i.e. they are not wired by default now)
loaned pages are unloaned via new uvm_unloan(), uvm_unloananon() and
uvm_unloanpage() are no longer exported
adjust uvm_unloanpage() to unwire the pages if UVM_LOAN_WIRED is specified
mark uvm_loanuobj() and uvm_loanzero() static also in function implementation

kern/sys_pipe.c: uvm_unloanpage() --> uvm_unloan()

a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

branches: 1.31.2;
handle a locking problem where the second (or later) call in the loanentry
loop returns 0. loanentry was returning >0, but was unlocking the maps
(because of the zero). reworked to avoid this. problem reported by
chuck silvers. also clarify a comment that jdolecek asked about.

when fetching an object page to loan out, do so synchronously.

branches: 1.29.2;
remove trailing whitespace.

fix locking problem noted by Jaromir Dolecek. also, add more comments
on locking rules to make code easier to understand. locking in
uvm_loananon still needs some work on fringe cases where anon's page
is actually on loan from a uobj.

Upon Chuck Cranor request, revert rev. 1.26. There is indeed a bug in way
locking is done, but this fix is not the right way to fix it.

Remove superflous uvmfault_unlockmaps() in uvm_loan(), only call it
if uvm_loanentry() returned 0; otherwise, the unlocking would already
have been done by uvmfault_unlockall() call in uvm_loanentry().
Okay'ed by Chuck Silvers