uvm: Expand v_size <= v_writesize assertions to help diagnostics.

PR kern/58117

genfs: KASSERT(A && B) -> KASSERT(A); KASSERT(B)

genfs(9): Prune dead branch.

Remove trailing \n from UVMHIST_LOG() format strings.

centralize calls from UVM to radixtree into a few functions.
in those functions, assert that the object lock is held in
the correct mode.

Output offsets in hex for UVMHIST.

genfs_putpages(): when building a cluster make use of pages in the in the
existing uvm_page_array.

- Alter the convention for uvm_page_array slightly, so the basic search
parameters can't change part way through a search: move the "uobj" and
"flags" arguments over to uvm_page_array_init() and store those with the
array.

- With that, detect when it's not possible to find any more pages in the
tree with the given search parameters, and avoid repeated tree lookups if
the caller loops over uvm_page_array_fill_and_peek().

Start trying to reduce cache misses on vm_page during fault processing.

- Make PGO_LOCKED getpages imply PGO_NOBUSY and remove the latter. Mark
pages busy only when there's actually I/O to do.

- When doing COW on a uvm_object, don't mess with neighbouring pages. In
all likelyhood they're already entered.

- Don't mess with neighbouring VAs that have existing mappings as replacing
those mappings with same can be quite costly.

- Don't enqueue pages for neighbour faults unless not enqueued already, and
don't activate centre pages unless uvmpdpol says its useful.

Also:

- Make PGO_LOCKED getpages on UAOs work more like vnodes: do gang lookup in
the radix tree, and don't allocate new pages.

- Fix many assertion failures around faults/loans with tmpfs.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Tweak the March 14th change to make page waits interlocked by pg->interlock.
Remove unneeded changes and only deal with the PQ_WANTED flag, to exclude
possible bugs.

Make uvm_pagemarkdirty() responsible for putting vnodes onto the syncer
work list. Proposed on tech-kern@.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

Unused variable.

- Hide the details of SPCF_SHOULDYIELD and related behind a couple of small
functions: preempt_point() and preempt_needed().

- preempt(): if the LWP has exceeded its timeslice in kernel, strip it of
any priority boost gained earlier from blocking.

OR into bp->b_cflags; don't overwrite.

Tighten up the locking around vp->v_iflag a little more after the recent
split of vmobjlock & v_interlock.

v_interlock -> vmobjlock

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

remove the aiodoned thread. I originally added this to provide a thread context
for doing page cache iodone work, but since then biodone() has changed to
hand off all iodone work to a softint thread, so we no longer need the
special-purpose aiodoned thread.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.83.2;
- Add and use wrapper functions that take and acquire page interlocks, and pairs
of page interlocks. Require that the page interlock be held over calls to
uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state. Rather than
updating the global state synchronously, set an intended state on
individual pages (active, inactive, enqueued, dequeued) while holding the
page interlock. After the interlock is released put the pages on a 128
entry per-CPU queue for their state changes to be made real in batch.
This results in in a ~400 fold decrease in contention on my test system.
Proposed on tech-kern but modified to use the page interlock rather than
atomics to synchronise as it's much easier to maintain that way, and
cheaper.

Rename uvm_page_locked_p() -> uvm_page_owner_locked_p()

genfs_do_putpages(): add a missing call to uvm_page_array_advance().

Spotted by the automated test runs and:

Reported-by: syzbot+adc1f0ce21bcece5307d@syzkaller.appspotmail.com

Correction to previous for DEBUG case.

Fix DEBUG build.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

Defer to synchronous I/O before the aiodone work queue exists.

Fix (harmless) uninitialized variable: 'pg' could be 'endm', in which case
'pg->uobject' would not be initialized. Just invert the two last conditions
of the KASSERT.

ok hannken@

assert that WAPBL journal write lock is actually held when called with
PGO_JOURNALLOCKED or IO_JOURNALLOCKED

suggested by mrg@, thanks

support flag PGO_JOURNALLOCKED also for genfs_getpages()

branches: 1.72.2;
add a genfs method to allow a file system to limit the range of pages
that are given to a single GOP_WRITE() call. needed by ZFS.

branches: 1.71.2;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

Add missing check for dead or dying vnode to the entry of genfs_getpages().

Operations fstrans_start() and fstrans_start_nowait() now always
use FSTRANS_SHARED as lock type so remove the lock type argument.

File system state FSTRANS_SUSPENDING is now unused so remove it.

Regen vnode_if files.

Ride 8.99.1 less than a hour ago.

branches: 1.68.6;
Clarify meaning of "glocked" argument of genfs_putpages_read.

Simplify genfs_getpages_read async/unlock protocol.

Previously the caller unlocked for error or sync I/O, whereas
genfs_getpages_read unlocked on successful async.

Now caller unlocks in every case, and genfs_getpages_read doesn't
touch the lock.

Change last users of FSTRANS_LAZY to FSTRANS_SHARED and change
genfs_suspendctl() to move from FSTRANS_NORMAL to FSTRANS_SUSPENDED
and vice versa.

Protect genfs_do_putpages() against vnodes disappearing during
a forced mount update from read-write to read-only.

Protect genfs_getpages() against vnodes disappearing during a
forced mount update from read-write to read-only.

branches: 1.63.2;
don't change the loop counts; noted by mrg@

Allow sparc kernels to build with SSP by using a constant PAGE_SIZE...

branches: 1.61.2;
Remove miscfs/syncfs and

- move the syncer into kern/vfs_subr.c.

- change the syncer to process the mountlist and VFS_SYNC as appropriate.

- use an API for mount points similiar to the API for vnodes:
- vfs_syncer_add_to_worklist(struct mount *mp) to add
- vfs_syncer_remove_from_worklist(struct mount *mp) to remove a mount.

No objections on tech-kern@

Fix UVMHIST build.

Pull VOP_BMAP/VOP_STRATEGY loop from getpages into its own function.

No functional change.

In preparation for a gop_read like the existing gop_write.

branches: 1.58.6;
Turn a few __unused into __diagused

Mark a potentially unused variable

Mark a potentially unused (if an arch implements pmap_update as empty
macro) variable accordingly.

branches: 1.55.2; 1.55.4;
don't block on pager map for read-ahead.
reduce code duplication.

change vflushbuf() to take the full FSYNC_* flags.
translate FSYNC_LAZY into PGO_LAZY for VOP_PUTPAGES() so that
genfs_do_io() can set the appropriate io priority for the I/O.
this is the first part of addressing PR 46325.

branches: 1.53.2; 1.53.6; 1.53.8;
typo in a comment

Trim unused headers.

Use the new UVM_KMF_COLORMATCH flag to get a congruent mappings of the user
buffer so we can use unmanaged mappings (pmap_kenter_pa/pmap_kremove).

genfs_do_directio: acquire the lock of page owner for now and fix PR/45177.
Will be revisited to avoid locking dance and be more efficient, e.g. we can
use unmanaged-mapping by allocating with colouring in mind.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.48.2;
Move some #ifdefs to prevent a code path change when DEBUG .vs. !DEBUG
Solves problem an assert firing when using NFS on MIPS.

G/C unused speedup_syncer() mechanism and thus simplify some code.
Update some comments to reflect the reality. No actual changes to
the (used) syncer logic.

OK ad@

branches: 1.46.2;
Correct an assertion; pointed out by mrg@ and pooka@, thanks.

genfs_do_putpages(): When testing an uobject for dirty or modified
pages skip uninitialized (PG_FAKE) pages (DEBUG only).

Always take the object lock before changing vmpage flags. Fixes a deadlock
where a thread is waiting on "genput" but the page in question is neither
BUSY nor WANTED.

No objections from tech-kern@.

Whitespace.

Genfs_getpages(): Break a deadlock where one thread runs VOP_GETPAGES(),
has busy pages and wants the wapbl lock as reader from wapbl_begin(),
another thread has the wapbl lock as reader and waits for a page from
the first thread. Now a third thread calls wapbl_flush() and wants the
wapbl lock as writer.

Move the wapbl_begin() up to a point where genfs_getpages() has no busy
pages yet.

genfs_getpages: restore vm_page array correctly in PGO_LOCKED error
code path.

replace the earlier workaround for PR 40389 with a better fix.
the earlier change caused data corruption by freeing pages
without invaliding their mappings. instead of the trylock/retry,
just take the genfs-node lock before calling VOP_GETPAGES()
and pass a new flag to tell it that we're already holding this lock.

print more info in the "past eof" panic

in genfs_getpages(), mark the vnode dirty (ie. add to syncer worklist
and set VI_WRMAPDIRTY) after we have busied the pages rather than
before. this prevents other threads calling genfs_do_putpages() from
marking the vnode clean again while we're in the process of creating
new writable mappings, since such threads will wait for the page(s) to
become unbusy before proceeding.
fixes the problem recently reported by hannken@ on tech-kern.

Add vm page flag PG_MARKER and use it to tag dummy marker pages
in genfs_do_putpages() and uao_put().
Use 'v_uobj.uo_npages' to check for an empty memq.
Put some assertions where these marker pages may not appear.

Ok: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.36.2; 1.36.4;
Reduce the diff between genfs_getpages() and genfs_do_io(). These should be
merged eventually.

Slightly more descriptive local variable names.

genfs_getpages: Narrow & clarify the context where I/O happens & vmobjlock is dropped.

genfs_getpages: Redo previous with a better goto label.

Revert part which variable initializations within interleaved gotos.

again: if (...) goto err;
void *ptr = alloc();
if (...) goto again;
if (...) goto err1;
...
err1: if (ptr) free(ptr);
err:
return;

This leaks memory if exited with "goto again; -> goto err;".

genfs_getpages: More constification & localization.

genfs_getpages: Constify 2 variables, move one. No functional changes.

genfs_getpages: Constify orignpages. Don't override its meaning by the value
re-calucated from GOP_SIZE(GOP_SIZE_MEM), but assign another variable
(orignmempages).

Unbreak modules build.

genfs_getpages: Constify & localize more variables.

genfs_getpages: Move local variable declarations that are used only for I/O
to where they're used. This helps to track what's going in this lengthy
function.

genfs_getpages: Localize a few more variables.

genfs_putpages: Localize a few variables. No functional changes.

Use genfs_node_*lock().

Constify some pointers in genfs_getpages() and genfs_do_putpages().

Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Move genfs_null_putpages() from genfs_io.c to genfs_vnops.c -- it does
not really do i/o.

genfs_getpages: rework 1.18 revision - move uvm_pagermapout() back.
It is useful to make KVA available ASAP. Per discussion with <yamt>.

branches: 1.18.2;
genfs_getpages: move putiobuf() and uvm_pagermapout() outside the glock.
OK by <ad>.

- g/c stale function prototypes.
- rename UVM_PAGE_HASH_PENALTY to UVM_PAGE_TREE_PENALTY.

Check that the filesystem acutally uses WAPBL before initiating a
transaction for the directio case. Fixes PR 39929 and similiar issues
seen with PostgreSQL.

more <sys/buf.h> police

- allocate 8 pointers on the stack to avoid stack overflow in nfs.
- make that 8 a constant
- remove bogus panic

branches: 1.13.2; 1.13.4;
Make genfs_directio() IO_JOURNALLOCKED aware. DirectIO no longer triggers
"locking against myself" panic in wapbl_begin().

Observed and tested by: Frank Kardel <kardel@netbsd.org>

Break a deadlock where one thread has a wapbl transaction, calls VOP_GETPAGES
and wants to busy a page while another thread calls VOP_PUTPAGES on the same
vnode, takes pages busy and wants to start a wapbl transaction.

Reviewed by: Jason Thorpe <thorpej@netbsd.org>

remove always-true conditionals.

constify

Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

branches: 1.8.2; 1.8.4;
vm_page: put TAILQ_ENTRY into a union with LIST_ENTRY, so we can use both.

Import writing part of the UDF file system making optical media like CD's
and DVD's behave like floppy discs. Writing is supported upto and including
version 2.01; version 2.50 and 2.60 will follow.

Also extending the UDF implementation to support symbolic links and
hardlinks.

Added are the mmcformat(8) tool to format rewritable CD/DVD discs and
newfs_udf(8).

Limitations:
all operations can be performed on the file system though the
sheduling is currently optimised for archiving workloads.

mv(1)/rename(2) is currently only implemented for non-directories.

branches: 1.6.2; 1.6.4;
Remove a race when pages are released while waiting for fstrans_start().

Fixes PR #38460

branches: 1.5.6; 1.5.8;
genfs_do_putpages: DEBUG checks.

genfs_do_putpages: ensure that we clean the vnode in the case of PGO_RECLAIM.

push pmap_clear_reference calls into pdpolicy code, where reference bits
actually matter.

Merge vmlocking2 to head.

branches: 1.1.4; 1.1.6; 1.1.8; 1.1.10; 1.1.12; 1.1.14; 1.1.16; 1.1.20;
Split I/O-related routines (getpages, putpages, etc.) which are heavily
tied to uvm out of genfs_vnops into genfs_io.c

genfs: KASSERT(A && B) -> KASSERT(A); KASSERT(B)

genfs(9): Prune dead branch.

Remove trailing \n from UVMHIST_LOG() format strings.

centralize calls from UVM to radixtree into a few functions.
in those functions, assert that the object lock is held in
the correct mode.

Output offsets in hex for UVMHIST.

genfs_putpages(): when building a cluster make use of pages in the in the
existing uvm_page_array.

- Alter the convention for uvm_page_array slightly, so the basic search
parameters can't change part way through a search: move the "uobj" and
"flags" arguments over to uvm_page_array_init() and store those with the
array.

- With that, detect when it's not possible to find any more pages in the
tree with the given search parameters, and avoid repeated tree lookups if
the caller loops over uvm_page_array_fill_and_peek().

Start trying to reduce cache misses on vm_page during fault processing.

- Make PGO_LOCKED getpages imply PGO_NOBUSY and remove the latter. Mark
pages busy only when there's actually I/O to do.

- When doing COW on a uvm_object, don't mess with neighbouring pages. In
all likelyhood they're already entered.

- Don't mess with neighbouring VAs that have existing mappings as replacing
those mappings with same can be quite costly.

- Don't enqueue pages for neighbour faults unless not enqueued already, and
don't activate centre pages unless uvmpdpol says its useful.

Also:

- Make PGO_LOCKED getpages on UAOs work more like vnodes: do gang lookup in
the radix tree, and don't allocate new pages.

- Fix many assertion failures around faults/loans with tmpfs.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Tweak the March 14th change to make page waits interlocked by pg->interlock.
Remove unneeded changes and only deal with the PQ_WANTED flag, to exclude
possible bugs.

Make uvm_pagemarkdirty() responsible for putting vnodes onto the syncer
work list. Proposed on tech-kern@.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

Unused variable.

- Hide the details of SPCF_SHOULDYIELD and related behind a couple of small
functions: preempt_point() and preempt_needed().

- preempt(): if the LWP has exceeded its timeslice in kernel, strip it of
any priority boost gained earlier from blocking.

OR into bp->b_cflags; don't overwrite.

Tighten up the locking around vp->v_iflag a little more after the recent
split of vmobjlock & v_interlock.

v_interlock -> vmobjlock

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

remove the aiodoned thread. I originally added this to provide a thread context
for doing page cache iodone work, but since then biodone() has changed to
hand off all iodone work to a softint thread, so we no longer need the
special-purpose aiodoned thread.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.83.2;
- Add and use wrapper functions that take and acquire page interlocks, and pairs
of page interlocks. Require that the page interlock be held over calls to
uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state. Rather than
updating the global state synchronously, set an intended state on
individual pages (active, inactive, enqueued, dequeued) while holding the
page interlock. After the interlock is released put the pages on a 128
entry per-CPU queue for their state changes to be made real in batch.
This results in in a ~400 fold decrease in contention on my test system.
Proposed on tech-kern but modified to use the page interlock rather than
atomics to synchronise as it's much easier to maintain that way, and
cheaper.

Rename uvm_page_locked_p() -> uvm_page_owner_locked_p()

genfs_do_putpages(): add a missing call to uvm_page_array_advance().

Spotted by the automated test runs and:

Reported-by: syzbot+adc1f0ce21bcece5307d@syzkaller.appspotmail.com

Correction to previous for DEBUG case.

Fix DEBUG build.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

Defer to synchronous I/O before the aiodone work queue exists.

Fix (harmless) uninitialized variable: 'pg' could be 'endm', in which case
'pg->uobject' would not be initialized. Just invert the two last conditions
of the KASSERT.

ok hannken@

assert that WAPBL journal write lock is actually held when called with
PGO_JOURNALLOCKED or IO_JOURNALLOCKED

suggested by mrg@, thanks

support flag PGO_JOURNALLOCKED also for genfs_getpages()

branches: 1.72.2;
add a genfs method to allow a file system to limit the range of pages
that are given to a single GOP_WRITE() call. needed by ZFS.

branches: 1.71.2;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

Add missing check for dead or dying vnode to the entry of genfs_getpages().

Operations fstrans_start() and fstrans_start_nowait() now always
use FSTRANS_SHARED as lock type so remove the lock type argument.

File system state FSTRANS_SUSPENDING is now unused so remove it.

Regen vnode_if files.

Ride 8.99.1 less than a hour ago.

branches: 1.68.6;
Clarify meaning of "glocked" argument of genfs_putpages_read.

Simplify genfs_getpages_read async/unlock protocol.

Previously the caller unlocked for error or sync I/O, whereas
genfs_getpages_read unlocked on successful async.

Now caller unlocks in every case, and genfs_getpages_read doesn't
touch the lock.

Change last users of FSTRANS_LAZY to FSTRANS_SHARED and change
genfs_suspendctl() to move from FSTRANS_NORMAL to FSTRANS_SUSPENDED
and vice versa.

Protect genfs_do_putpages() against vnodes disappearing during
a forced mount update from read-write to read-only.

Protect genfs_getpages() against vnodes disappearing during a
forced mount update from read-write to read-only.

branches: 1.63.2;
don't change the loop counts; noted by mrg@

Allow sparc kernels to build with SSP by using a constant PAGE_SIZE...

branches: 1.61.2;
Remove miscfs/syncfs and

- move the syncer into kern/vfs_subr.c.

- change the syncer to process the mountlist and VFS_SYNC as appropriate.

- use an API for mount points similiar to the API for vnodes:
- vfs_syncer_add_to_worklist(struct mount *mp) to add
- vfs_syncer_remove_from_worklist(struct mount *mp) to remove a mount.

No objections on tech-kern@

Fix UVMHIST build.

Pull VOP_BMAP/VOP_STRATEGY loop from getpages into its own function.

No functional change.

In preparation for a gop_read like the existing gop_write.

branches: 1.58.6;
Turn a few __unused into __diagused

Mark a potentially unused variable

Mark a potentially unused (if an arch implements pmap_update as empty
macro) variable accordingly.

branches: 1.55.2; 1.55.4;
don't block on pager map for read-ahead.
reduce code duplication.

change vflushbuf() to take the full FSYNC_* flags.
translate FSYNC_LAZY into PGO_LAZY for VOP_PUTPAGES() so that
genfs_do_io() can set the appropriate io priority for the I/O.
this is the first part of addressing PR 46325.

branches: 1.53.2; 1.53.6; 1.53.8;
typo in a comment

Trim unused headers.

Use the new UVM_KMF_COLORMATCH flag to get a congruent mappings of the user
buffer so we can use unmanaged mappings (pmap_kenter_pa/pmap_kremove).

genfs_do_directio: acquire the lock of page owner for now and fix PR/45177.
Will be revisited to avoid locking dance and be more efficient, e.g. we can
use unmanaged-mapping by allocating with colouring in mind.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.48.2;
Move some #ifdefs to prevent a code path change when DEBUG .vs. !DEBUG
Solves problem an assert firing when using NFS on MIPS.

G/C unused speedup_syncer() mechanism and thus simplify some code.
Update some comments to reflect the reality. No actual changes to
the (used) syncer logic.

OK ad@

branches: 1.46.2;
Correct an assertion; pointed out by mrg@ and pooka@, thanks.

genfs_do_putpages(): When testing an uobject for dirty or modified
pages skip uninitialized (PG_FAKE) pages (DEBUG only).

Always take the object lock before changing vmpage flags. Fixes a deadlock
where a thread is waiting on "genput" but the page in question is neither
BUSY nor WANTED.

No objections from tech-kern@.

Whitespace.

Genfs_getpages(): Break a deadlock where one thread runs VOP_GETPAGES(),
has busy pages and wants the wapbl lock as reader from wapbl_begin(),
another thread has the wapbl lock as reader and waits for a page from
the first thread. Now a third thread calls wapbl_flush() and wants the
wapbl lock as writer.

Move the wapbl_begin() up to a point where genfs_getpages() has no busy
pages yet.

genfs_getpages: restore vm_page array correctly in PGO_LOCKED error
code path.

replace the earlier workaround for PR 40389 with a better fix.
the earlier change caused data corruption by freeing pages
without invaliding their mappings. instead of the trylock/retry,
just take the genfs-node lock before calling VOP_GETPAGES()
and pass a new flag to tell it that we're already holding this lock.

print more info in the "past eof" panic

in genfs_getpages(), mark the vnode dirty (ie. add to syncer worklist
and set VI_WRMAPDIRTY) after we have busied the pages rather than
before. this prevents other threads calling genfs_do_putpages() from
marking the vnode clean again while we're in the process of creating
new writable mappings, since such threads will wait for the page(s) to
become unbusy before proceeding.
fixes the problem recently reported by hannken@ on tech-kern.

Add vm page flag PG_MARKER and use it to tag dummy marker pages
in genfs_do_putpages() and uao_put().
Use 'v_uobj.uo_npages' to check for an empty memq.
Put some assertions where these marker pages may not appear.

Ok: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.36.2; 1.36.4;
Reduce the diff between genfs_getpages() and genfs_do_io(). These should be
merged eventually.

Slightly more descriptive local variable names.

genfs_getpages: Narrow & clarify the context where I/O happens & vmobjlock is dropped.

genfs_getpages: Redo previous with a better goto label.

Revert part which variable initializations within interleaved gotos.

again: if (...) goto err;
void *ptr = alloc();
if (...) goto again;
if (...) goto err1;
...
err1: if (ptr) free(ptr);
err:
return;

This leaks memory if exited with "goto again; -> goto err;".

genfs_getpages: More constification & localization.

genfs_getpages: Constify 2 variables, move one. No functional changes.

genfs_getpages: Constify orignpages. Don't override its meaning by the value
re-calucated from GOP_SIZE(GOP_SIZE_MEM), but assign another variable
(orignmempages).

Unbreak modules build.

genfs_getpages: Constify & localize more variables.

genfs_getpages: Move local variable declarations that are used only for I/O
to where they're used. This helps to track what's going in this lengthy
function.

genfs_getpages: Localize a few more variables.

genfs_putpages: Localize a few variables. No functional changes.

Use genfs_node_*lock().

Constify some pointers in genfs_getpages() and genfs_do_putpages().

Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Move genfs_null_putpages() from genfs_io.c to genfs_vnops.c -- it does
not really do i/o.

genfs_getpages: rework 1.18 revision - move uvm_pagermapout() back.
It is useful to make KVA available ASAP. Per discussion with <yamt>.

branches: 1.18.2;
genfs_getpages: move putiobuf() and uvm_pagermapout() outside the glock.
OK by <ad>.

- g/c stale function prototypes.
- rename UVM_PAGE_HASH_PENALTY to UVM_PAGE_TREE_PENALTY.

Check that the filesystem acutally uses WAPBL before initiating a
transaction for the directio case. Fixes PR 39929 and similiar issues
seen with PostgreSQL.

more <sys/buf.h> police

- allocate 8 pointers on the stack to avoid stack overflow in nfs.
- make that 8 a constant
- remove bogus panic

branches: 1.13.2; 1.13.4;
Make genfs_directio() IO_JOURNALLOCKED aware. DirectIO no longer triggers
"locking against myself" panic in wapbl_begin().

Observed and tested by: Frank Kardel <kardel@netbsd.org>

Break a deadlock where one thread has a wapbl transaction, calls VOP_GETPAGES
and wants to busy a page while another thread calls VOP_PUTPAGES on the same
vnode, takes pages busy and wants to start a wapbl transaction.

Reviewed by: Jason Thorpe <thorpej@netbsd.org>

remove always-true conditionals.

constify

Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

branches: 1.8.2; 1.8.4;
vm_page: put TAILQ_ENTRY into a union with LIST_ENTRY, so we can use both.

Import writing part of the UDF file system making optical media like CD's
and DVD's behave like floppy discs. Writing is supported upto and including
version 2.01; version 2.50 and 2.60 will follow.

Also extending the UDF implementation to support symbolic links and
hardlinks.

Added are the mmcformat(8) tool to format rewritable CD/DVD discs and
newfs_udf(8).

Limitations:
all operations can be performed on the file system though the
sheduling is currently optimised for archiving workloads.

mv(1)/rename(2) is currently only implemented for non-directories.

branches: 1.6.2; 1.6.4;
Remove a race when pages are released while waiting for fstrans_start().

Fixes PR #38460

branches: 1.5.6; 1.5.8;
genfs_do_putpages: DEBUG checks.

genfs_do_putpages: ensure that we clean the vnode in the case of PGO_RECLAIM.

push pmap_clear_reference calls into pdpolicy code, where reference bits
actually matter.

Merge vmlocking2 to head.

branches: 1.1.4; 1.1.6; 1.1.8; 1.1.10; 1.1.12; 1.1.14; 1.1.16; 1.1.20;
Split I/O-related routines (getpages, putpages, etc.) which are heavily
tied to uvm out of genfs_vnops into genfs_io.c

genfs(9): Prune dead branch.

Remove trailing \n from UVMHIST_LOG() format strings.

centralize calls from UVM to radixtree into a few functions.
in those functions, assert that the object lock is held in
the correct mode.

Output offsets in hex for UVMHIST.

genfs_putpages(): when building a cluster make use of pages in the in the
existing uvm_page_array.

- Alter the convention for uvm_page_array slightly, so the basic search
parameters can't change part way through a search: move the "uobj" and
"flags" arguments over to uvm_page_array_init() and store those with the
array.

- With that, detect when it's not possible to find any more pages in the
tree with the given search parameters, and avoid repeated tree lookups if
the caller loops over uvm_page_array_fill_and_peek().

Start trying to reduce cache misses on vm_page during fault processing.

- Make PGO_LOCKED getpages imply PGO_NOBUSY and remove the latter. Mark
pages busy only when there's actually I/O to do.

- When doing COW on a uvm_object, don't mess with neighbouring pages. In
all likelyhood they're already entered.

- Don't mess with neighbouring VAs that have existing mappings as replacing
those mappings with same can be quite costly.

- Don't enqueue pages for neighbour faults unless not enqueued already, and
don't activate centre pages unless uvmpdpol says its useful.

Also:

- Make PGO_LOCKED getpages on UAOs work more like vnodes: do gang lookup in
the radix tree, and don't allocate new pages.

- Fix many assertion failures around faults/loans with tmpfs.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Tweak the March 14th change to make page waits interlocked by pg->interlock.
Remove unneeded changes and only deal with the PQ_WANTED flag, to exclude
possible bugs.

Make uvm_pagemarkdirty() responsible for putting vnodes onto the syncer
work list. Proposed on tech-kern@.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

Unused variable.

- Hide the details of SPCF_SHOULDYIELD and related behind a couple of small
functions: preempt_point() and preempt_needed().

- preempt(): if the LWP has exceeded its timeslice in kernel, strip it of
any priority boost gained earlier from blocking.

OR into bp->b_cflags; don't overwrite.

Tighten up the locking around vp->v_iflag a little more after the recent
split of vmobjlock & v_interlock.

v_interlock -> vmobjlock

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

remove the aiodoned thread. I originally added this to provide a thread context
for doing page cache iodone work, but since then biodone() has changed to
hand off all iodone work to a softint thread, so we no longer need the
special-purpose aiodoned thread.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.83.2;
- Add and use wrapper functions that take and acquire page interlocks, and pairs
of page interlocks. Require that the page interlock be held over calls to
uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state. Rather than
updating the global state synchronously, set an intended state on
individual pages (active, inactive, enqueued, dequeued) while holding the
page interlock. After the interlock is released put the pages on a 128
entry per-CPU queue for their state changes to be made real in batch.
This results in in a ~400 fold decrease in contention on my test system.
Proposed on tech-kern but modified to use the page interlock rather than
atomics to synchronise as it's much easier to maintain that way, and
cheaper.

Rename uvm_page_locked_p() -> uvm_page_owner_locked_p()

genfs_do_putpages(): add a missing call to uvm_page_array_advance().

Spotted by the automated test runs and:

Reported-by: syzbot+adc1f0ce21bcece5307d@syzkaller.appspotmail.com

Correction to previous for DEBUG case.

Fix DEBUG build.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

Defer to synchronous I/O before the aiodone work queue exists.

Fix (harmless) uninitialized variable: 'pg' could be 'endm', in which case
'pg->uobject' would not be initialized. Just invert the two last conditions
of the KASSERT.

ok hannken@

assert that WAPBL journal write lock is actually held when called with
PGO_JOURNALLOCKED or IO_JOURNALLOCKED

suggested by mrg@, thanks

support flag PGO_JOURNALLOCKED also for genfs_getpages()

branches: 1.72.2;
add a genfs method to allow a file system to limit the range of pages
that are given to a single GOP_WRITE() call. needed by ZFS.

branches: 1.71.2;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

Add missing check for dead or dying vnode to the entry of genfs_getpages().

Operations fstrans_start() and fstrans_start_nowait() now always
use FSTRANS_SHARED as lock type so remove the lock type argument.

File system state FSTRANS_SUSPENDING is now unused so remove it.

Regen vnode_if files.

Ride 8.99.1 less than a hour ago.

branches: 1.68.6;
Clarify meaning of "glocked" argument of genfs_putpages_read.

Simplify genfs_getpages_read async/unlock protocol.

Previously the caller unlocked for error or sync I/O, whereas
genfs_getpages_read unlocked on successful async.

Now caller unlocks in every case, and genfs_getpages_read doesn't
touch the lock.

Change last users of FSTRANS_LAZY to FSTRANS_SHARED and change
genfs_suspendctl() to move from FSTRANS_NORMAL to FSTRANS_SUSPENDED
and vice versa.

Protect genfs_do_putpages() against vnodes disappearing during
a forced mount update from read-write to read-only.

Protect genfs_getpages() against vnodes disappearing during a
forced mount update from read-write to read-only.

branches: 1.63.2;
don't change the loop counts; noted by mrg@

Allow sparc kernels to build with SSP by using a constant PAGE_SIZE...

branches: 1.61.2;
Remove miscfs/syncfs and

- move the syncer into kern/vfs_subr.c.

- change the syncer to process the mountlist and VFS_SYNC as appropriate.

- use an API for mount points similiar to the API for vnodes:
- vfs_syncer_add_to_worklist(struct mount *mp) to add
- vfs_syncer_remove_from_worklist(struct mount *mp) to remove a mount.

No objections on tech-kern@

Fix UVMHIST build.

Pull VOP_BMAP/VOP_STRATEGY loop from getpages into its own function.

No functional change.

In preparation for a gop_read like the existing gop_write.

branches: 1.58.6;
Turn a few __unused into __diagused

Mark a potentially unused variable

Mark a potentially unused (if an arch implements pmap_update as empty
macro) variable accordingly.

branches: 1.55.2; 1.55.4;
don't block on pager map for read-ahead.
reduce code duplication.

change vflushbuf() to take the full FSYNC_* flags.
translate FSYNC_LAZY into PGO_LAZY for VOP_PUTPAGES() so that
genfs_do_io() can set the appropriate io priority for the I/O.
this is the first part of addressing PR 46325.

branches: 1.53.2; 1.53.6; 1.53.8;
typo in a comment

Trim unused headers.

Use the new UVM_KMF_COLORMATCH flag to get a congruent mappings of the user
buffer so we can use unmanaged mappings (pmap_kenter_pa/pmap_kremove).

genfs_do_directio: acquire the lock of page owner for now and fix PR/45177.
Will be revisited to avoid locking dance and be more efficient, e.g. we can
use unmanaged-mapping by allocating with colouring in mind.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.48.2;
Move some #ifdefs to prevent a code path change when DEBUG .vs. !DEBUG
Solves problem an assert firing when using NFS on MIPS.

G/C unused speedup_syncer() mechanism and thus simplify some code.
Update some comments to reflect the reality. No actual changes to
the (used) syncer logic.

OK ad@

branches: 1.46.2;
Correct an assertion; pointed out by mrg@ and pooka@, thanks.

genfs_do_putpages(): When testing an uobject for dirty or modified
pages skip uninitialized (PG_FAKE) pages (DEBUG only).

Always take the object lock before changing vmpage flags. Fixes a deadlock
where a thread is waiting on "genput" but the page in question is neither
BUSY nor WANTED.

No objections from tech-kern@.

Whitespace.

Genfs_getpages(): Break a deadlock where one thread runs VOP_GETPAGES(),
has busy pages and wants the wapbl lock as reader from wapbl_begin(),
another thread has the wapbl lock as reader and waits for a page from
the first thread. Now a third thread calls wapbl_flush() and wants the
wapbl lock as writer.

Move the wapbl_begin() up to a point where genfs_getpages() has no busy
pages yet.

genfs_getpages: restore vm_page array correctly in PGO_LOCKED error
code path.

replace the earlier workaround for PR 40389 with a better fix.
the earlier change caused data corruption by freeing pages
without invaliding their mappings. instead of the trylock/retry,
just take the genfs-node lock before calling VOP_GETPAGES()
and pass a new flag to tell it that we're already holding this lock.

print more info in the "past eof" panic

in genfs_getpages(), mark the vnode dirty (ie. add to syncer worklist
and set VI_WRMAPDIRTY) after we have busied the pages rather than
before. this prevents other threads calling genfs_do_putpages() from
marking the vnode clean again while we're in the process of creating
new writable mappings, since such threads will wait for the page(s) to
become unbusy before proceeding.
fixes the problem recently reported by hannken@ on tech-kern.

Add vm page flag PG_MARKER and use it to tag dummy marker pages
in genfs_do_putpages() and uao_put().
Use 'v_uobj.uo_npages' to check for an empty memq.
Put some assertions where these marker pages may not appear.

Ok: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.36.2; 1.36.4;
Reduce the diff between genfs_getpages() and genfs_do_io(). These should be
merged eventually.

Slightly more descriptive local variable names.

genfs_getpages: Narrow & clarify the context where I/O happens & vmobjlock is dropped.

genfs_getpages: Redo previous with a better goto label.

Revert part which variable initializations within interleaved gotos.

again: if (...) goto err;
void *ptr = alloc();
if (...) goto again;
if (...) goto err1;
...
err1: if (ptr) free(ptr);
err:
return;

This leaks memory if exited with "goto again; -> goto err;".

genfs_getpages: More constification & localization.

genfs_getpages: Constify 2 variables, move one. No functional changes.

genfs_getpages: Constify orignpages. Don't override its meaning by the value
re-calucated from GOP_SIZE(GOP_SIZE_MEM), but assign another variable
(orignmempages).

Unbreak modules build.

genfs_getpages: Constify & localize more variables.

genfs_getpages: Move local variable declarations that are used only for I/O
to where they're used. This helps to track what's going in this lengthy
function.

genfs_getpages: Localize a few more variables.

genfs_putpages: Localize a few variables. No functional changes.

Use genfs_node_*lock().

Constify some pointers in genfs_getpages() and genfs_do_putpages().

Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Move genfs_null_putpages() from genfs_io.c to genfs_vnops.c -- it does
not really do i/o.

genfs_getpages: rework 1.18 revision - move uvm_pagermapout() back.
It is useful to make KVA available ASAP. Per discussion with <yamt>.

branches: 1.18.2;
genfs_getpages: move putiobuf() and uvm_pagermapout() outside the glock.
OK by <ad>.

- g/c stale function prototypes.
- rename UVM_PAGE_HASH_PENALTY to UVM_PAGE_TREE_PENALTY.

Check that the filesystem acutally uses WAPBL before initiating a
transaction for the directio case. Fixes PR 39929 and similiar issues
seen with PostgreSQL.

more <sys/buf.h> police

- allocate 8 pointers on the stack to avoid stack overflow in nfs.
- make that 8 a constant
- remove bogus panic

branches: 1.13.2; 1.13.4;
Make genfs_directio() IO_JOURNALLOCKED aware. DirectIO no longer triggers
"locking against myself" panic in wapbl_begin().

Observed and tested by: Frank Kardel <kardel@netbsd.org>

Break a deadlock where one thread has a wapbl transaction, calls VOP_GETPAGES
and wants to busy a page while another thread calls VOP_PUTPAGES on the same
vnode, takes pages busy and wants to start a wapbl transaction.

Reviewed by: Jason Thorpe <thorpej@netbsd.org>

remove always-true conditionals.

constify

Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

branches: 1.8.2; 1.8.4;
vm_page: put TAILQ_ENTRY into a union with LIST_ENTRY, so we can use both.

Import writing part of the UDF file system making optical media like CD's
and DVD's behave like floppy discs. Writing is supported upto and including
version 2.01; version 2.50 and 2.60 will follow.

Also extending the UDF implementation to support symbolic links and
hardlinks.

Added are the mmcformat(8) tool to format rewritable CD/DVD discs and
newfs_udf(8).

Limitations:
all operations can be performed on the file system though the
sheduling is currently optimised for archiving workloads.

mv(1)/rename(2) is currently only implemented for non-directories.

branches: 1.6.2; 1.6.4;
Remove a race when pages are released while waiting for fstrans_start().

Fixes PR #38460

branches: 1.5.6; 1.5.8;
genfs_do_putpages: DEBUG checks.

genfs_do_putpages: ensure that we clean the vnode in the case of PGO_RECLAIM.

push pmap_clear_reference calls into pdpolicy code, where reference bits
actually matter.

Merge vmlocking2 to head.

branches: 1.1.4; 1.1.6; 1.1.8; 1.1.10; 1.1.12; 1.1.14; 1.1.16; 1.1.20;
Split I/O-related routines (getpages, putpages, etc.) which are heavily
tied to uvm out of genfs_vnops into genfs_io.c

Remove trailing \n from UVMHIST_LOG() format strings.

centralize calls from UVM to radixtree into a few functions.
in those functions, assert that the object lock is held in
the correct mode.

Output offsets in hex for UVMHIST.

genfs_putpages(): when building a cluster make use of pages in the in the
existing uvm_page_array.

- Alter the convention for uvm_page_array slightly, so the basic search
parameters can't change part way through a search: move the "uobj" and
"flags" arguments over to uvm_page_array_init() and store those with the
array.

- With that, detect when it's not possible to find any more pages in the
tree with the given search parameters, and avoid repeated tree lookups if
the caller loops over uvm_page_array_fill_and_peek().

Start trying to reduce cache misses on vm_page during fault processing.

- Make PGO_LOCKED getpages imply PGO_NOBUSY and remove the latter. Mark
pages busy only when there's actually I/O to do.

- When doing COW on a uvm_object, don't mess with neighbouring pages. In
all likelyhood they're already entered.

- Don't mess with neighbouring VAs that have existing mappings as replacing
those mappings with same can be quite costly.

- Don't enqueue pages for neighbour faults unless not enqueued already, and
don't activate centre pages unless uvmpdpol says its useful.

Also:

- Make PGO_LOCKED getpages on UAOs work more like vnodes: do gang lookup in
the radix tree, and don't allocate new pages.

- Fix many assertion failures around faults/loans with tmpfs.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Tweak the March 14th change to make page waits interlocked by pg->interlock.
Remove unneeded changes and only deal with the PQ_WANTED flag, to exclude
possible bugs.

Make uvm_pagemarkdirty() responsible for putting vnodes onto the syncer
work list. Proposed on tech-kern@.

Make page waits (WANTED vs BUSY) interlocked by pg->interlock. Gets RW
locks out of the equation for sleep/wakeup, and allows observing+waiting
for busy pages when holding only a read lock. Proposed on tech-kern.

Unused variable.

- Hide the details of SPCF_SHOULDYIELD and related behind a couple of small
functions: preempt_point() and preempt_needed().

- preempt(): if the LWP has exceeded its timeslice in kernel, strip it of
any priority boost gained earlier from blocking.

OR into bp->b_cflags; don't overwrite.

Tighten up the locking around vp->v_iflag a little more after the recent
split of vmobjlock & v_interlock.

v_interlock -> vmobjlock

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

remove the aiodoned thread. I originally added this to provide a thread context
for doing page cache iodone work, but since then biodone() has changed to
hand off all iodone work to a softint thread, so we no longer need the
special-purpose aiodoned thread.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

branches: 1.83.2;
- Add and use wrapper functions that take and acquire page interlocks, and pairs
of page interlocks. Require that the page interlock be held over calls to
uvm_pageactivate(), uvm_pagewire() and similar.

- Solve the concurrency problem with page replacement state. Rather than
updating the global state synchronously, set an intended state on
individual pages (active, inactive, enqueued, dequeued) while holding the
page interlock. After the interlock is released put the pages on a 128
entry per-CPU queue for their state changes to be made real in batch.
This results in in a ~400 fold decrease in contention on my test system.
Proposed on tech-kern but modified to use the page interlock rather than
atomics to synchronise as it's much easier to maintain that way, and
cheaper.

Rename uvm_page_locked_p() -> uvm_page_owner_locked_p()

genfs_do_putpages(): add a missing call to uvm_page_array_advance().

Spotted by the automated test runs and:

Reported-by: syzbot+adc1f0ce21bcece5307d@syzkaller.appspotmail.com

Correction to previous for DEBUG case.

Fix DEBUG build.

Merge from yamt-pagecache:

- do gang lookup of pages using radixtree.
- remove now unused uvm_object::uo_memq and vm_page::listq.queue.

Break the global uvm_pageqlock into a per-page identity lock and a private
lock for use of the pagedaemon policy code. Discussed on tech-kern.

PR kern/54209: NetBSD 8 large memory performance extremely low
PR kern/54210: NetBSD-8 processes presumably not exiting
PR kern/54727: writing a large file causes unreasonable system behaviour

Defer to synchronous I/O before the aiodone work queue exists.

Fix (harmless) uninitialized variable: 'pg' could be 'endm', in which case
'pg->uobject' would not be initialized. Just invert the two last conditions
of the KASSERT.

ok hannken@

assert that WAPBL journal write lock is actually held when called with
PGO_JOURNALLOCKED or IO_JOURNALLOCKED

suggested by mrg@, thanks

support flag PGO_JOURNALLOCKED also for genfs_getpages()

branches: 1.72.2;
add a genfs method to allow a file system to limit the range of pages
that are given to a single GOP_WRITE() call. needed by ZFS.

branches: 1.71.2;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

Add missing check for dead or dying vnode to the entry of genfs_getpages().

Operations fstrans_start() and fstrans_start_nowait() now always
use FSTRANS_SHARED as lock type so remove the lock type argument.

File system state FSTRANS_SUSPENDING is now unused so remove it.

Regen vnode_if files.

Ride 8.99.1 less than a hour ago.

branches: 1.68.6;
Clarify meaning of "glocked" argument of genfs_putpages_read.

Simplify genfs_getpages_read async/unlock protocol.

Previously the caller unlocked for error or sync I/O, whereas
genfs_getpages_read unlocked on successful async.

Now caller unlocks in every case, and genfs_getpages_read doesn't
touch the lock.

Change last users of FSTRANS_LAZY to FSTRANS_SHARED and change
genfs_suspendctl() to move from FSTRANS_NORMAL to FSTRANS_SUSPENDED
and vice versa.

Protect genfs_do_putpages() against vnodes disappearing during
a forced mount update from read-write to read-only.

Protect genfs_getpages() against vnodes disappearing during a
forced mount update from read-write to read-only.

branches: 1.63.2;
don't change the loop counts; noted by mrg@

Allow sparc kernels to build with SSP by using a constant PAGE_SIZE...

branches: 1.61.2;
Remove miscfs/syncfs and

- move the syncer into kern/vfs_subr.c.

- change the syncer to process the mountlist and VFS_SYNC as appropriate.

- use an API for mount points similiar to the API for vnodes:
- vfs_syncer_add_to_worklist(struct mount *mp) to add
- vfs_syncer_remove_from_worklist(struct mount *mp) to remove a mount.

No objections on tech-kern@

Fix UVMHIST build.

Pull VOP_BMAP/VOP_STRATEGY loop from getpages into its own function.

No functional change.

In preparation for a gop_read like the existing gop_write.

branches: 1.58.6;
Turn a few __unused into __diagused

Mark a potentially unused variable

Mark a potentially unused (if an arch implements pmap_update as empty
macro) variable accordingly.

branches: 1.55.2; 1.55.4;
don't block on pager map for read-ahead.
reduce code duplication.

change vflushbuf() to take the full FSYNC_* flags.
translate FSYNC_LAZY into PGO_LAZY for VOP_PUTPAGES() so that
genfs_do_io() can set the appropriate io priority for the I/O.
this is the first part of addressing PR 46325.

branches: 1.53.2; 1.53.6; 1.53.8;
typo in a comment

Trim unused headers.

Use the new UVM_KMF_COLORMATCH flag to get a congruent mappings of the user
buffer so we can use unmanaged mappings (pmap_kenter_pa/pmap_kremove).

genfs_do_directio: acquire the lock of page owner for now and fix PR/45177.
Will be revisited to avoid locking dance and be more efficient, e.g. we can
use unmanaged-mapping by allocating with colouring in mind.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.48.2;
Move some #ifdefs to prevent a code path change when DEBUG .vs. !DEBUG
Solves problem an assert firing when using NFS on MIPS.

G/C unused speedup_syncer() mechanism and thus simplify some code.
Update some comments to reflect the reality. No actual changes to
the (used) syncer logic.

OK ad@

branches: 1.46.2;
Correct an assertion; pointed out by mrg@ and pooka@, thanks.

genfs_do_putpages(): When testing an uobject for dirty or modified
pages skip uninitialized (PG_FAKE) pages (DEBUG only).

Always take the object lock before changing vmpage flags. Fixes a deadlock
where a thread is waiting on "genput" but the page in question is neither
BUSY nor WANTED.

No objections from tech-kern@.

Whitespace.

Genfs_getpages(): Break a deadlock where one thread runs VOP_GETPAGES(),
has busy pages and wants the wapbl lock as reader from wapbl_begin(),
another thread has the wapbl lock as reader and waits for a page from
the first thread. Now a third thread calls wapbl_flush() and wants the
wapbl lock as writer.

Move the wapbl_begin() up to a point where genfs_getpages() has no busy
pages yet.

genfs_getpages: restore vm_page array correctly in PGO_LOCKED error
code path.

replace the earlier workaround for PR 40389 with a better fix.
the earlier change caused data corruption by freeing pages
without invaliding their mappings. instead of the trylock/retry,
just take the genfs-node lock before calling VOP_GETPAGES()
and pass a new flag to tell it that we're already holding this lock.

print more info in the "past eof" panic

in genfs_getpages(), mark the vnode dirty (ie. add to syncer worklist
and set VI_WRMAPDIRTY) after we have busied the pages rather than
before. this prevents other threads calling genfs_do_putpages() from
marking the vnode clean again while we're in the process of creating
new writable mappings, since such threads will wait for the page(s) to
become unbusy before proceeding.
fixes the problem recently reported by hannken@ on tech-kern.

Add vm page flag PG_MARKER and use it to tag dummy marker pages
in genfs_do_putpages() and uao_put().
Use 'v_uobj.uo_npages' to check for an empty memq.
Put some assertions where these marker pages may not appear.

Ok: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.36.2; 1.36.4;
Reduce the diff between genfs_getpages() and genfs_do_io(). These should be
merged eventually.

Slightly more descriptive local variable names.

genfs_getpages: Narrow & clarify the context where I/O happens & vmobjlock is dropped.

genfs_getpages: Redo previous with a better goto label.

Revert part which variable initializations within interleaved gotos.

again: if (...) goto err;
void *ptr = alloc();
if (...) goto again;
if (...) goto err1;
...
err1: if (ptr) free(ptr);
err:
return;

This leaks memory if exited with "goto again; -> goto err;".

genfs_getpages: More constification & localization.

genfs_getpages: Constify 2 variables, move one. No functional changes.

genfs_getpages: Constify orignpages. Don't override its meaning by the value
re-calucated from GOP_SIZE(GOP_SIZE_MEM), but assign another variable
(orignmempages).

Unbreak modules build.

genfs_getpages: Constify & localize more variables.

genfs_getpages: Move local variable declarations that are used only for I/O
to where they're used. This helps to track what's going in this lengthy
function.

genfs_getpages: Localize a few more variables.

genfs_putpages: Localize a few variables. No functional changes.

Use genfs_node_*lock().

Constify some pointers in genfs_getpages() and genfs_do_putpages().

Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Move genfs_null_putpages() from genfs_io.c to genfs_vnops.c -- it does
not really do i/o.

genfs_getpages: rework 1.18 revision - move uvm_pagermapout() back.
It is useful to make KVA available ASAP. Per discussion with <yamt>.

branches: 1.18.2;
genfs_getpages: move putiobuf() and uvm_pagermapout() outside the glock.
OK by <ad>.

- g/c stale function prototypes.
- rename UVM_PAGE_HASH_PENALTY to UVM_PAGE_TREE_PENALTY.

Check that the filesystem acutally uses WAPBL before initiating a
transaction for the directio case. Fixes PR 39929 and similiar issues
seen with PostgreSQL.

more <sys/buf.h> police

- allocate 8 pointers on the stack to avoid stack overflow in nfs.
- make that 8 a constant
- remove bogus panic

branches: 1.13.2; 1.13.4;
Make genfs_directio() IO_JOURNALLOCKED aware. DirectIO no longer triggers
"locking against myself" panic in wapbl_begin().

Observed and tested by: Frank Kardel <kardel@netbsd.org>

Break a deadlock where one thread has a wapbl transaction, calls VOP_GETPAGES
and wants to busy a page while another thread calls VOP_PUTPAGES on the same
vnode, takes pages busy and wants to start a wapbl transaction.

Reviewed by: Jason Thorpe <thorpej@netbsd.org>

remove always-true conditionals.

constify

Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

branches: 1.8.2; 1.8.4;
vm_page: put TAILQ_ENTRY into a union with LIST_ENTRY, so we can use both.