Revert "kern: Sprinkle biglock-slippage assertions."

Got the diagnostic information I needed from this, and it's holding
up releng tests of everything else, so let's back this out until I
need more diagnostics or track down the original source of the
problem.

kern: Sprinkle biglock-slippage assertions.

We seem to have a poltergeist that occasionally messes with the
biglock depth, but it's very hard to reproduce and only manifests as
some other CPU spinning out on the kernel lock which is no good for
diagnostics.

If we're only doing a count-only kern.buf sysctl, just return the number
of active members in the pool cache (plus some slop) instead of looking
in all the free buffer list. Should reduce CPU usage for "systat vm"
to << 1% especially for machines with a larger number of buffers.

Expose KERN_BUFSLOP in <sys/sysctl.h>.

Pad out the slop for kern.buf based on the passed in element size,
rather than a size of an unrelated struct.

branches: 1.298.2;
Add a sysctl hashstat collector for bufhash.

branches: 1.297.2; 1.297.4;
fix the UFS2 extattr truncate code to play nice with wapbl.
also, rather than pull in the FreeBSD V_NORMAL/V_ALT flags to
vinvalbuf() and the buf b_xflags field and BX_ALTDATA flag,
add a binvalbuf() function to invalid a specific buffer
and use that to invalidate the two possible exattr bufs
during IO_EXT truncations.

uvm_availmem(): give it a boolean argument to specify whether a recent
cached value will do, or if the very latest total must be fetched. It can
be called thousands of times a second and fetching the totals impacts not
only the calling LWP but other CPUs doing unrelated activity in the VM
system.

pass B_PHYS|B_RAW also in nestio_setup(), courtesy to e.g. xbd(4), which
wants to know whether the buf came from user space or bio subsystem

Rename buf_syncwait() to vfs_syncwait(), and have it wait on v_numoutput
rather than BC_BUSY. Removes the dependency on bufhash.

for bmempools set align, not ioff

explicitly use DEV_BSIZE align for all bmempools

this is required for Xen xbd(4) in order to not have to use bounce buffers

the alignment is implicitly provided when POOL_REDZONE is not active,
this change makes it also aligned when POOL_REDZONE _is_ active - that is
when (!KMSAN && (DIAGNOSTIC || KASAN))

Remove buffer reference counting, now that it's safe to destroy b_busy after
waking any waiters.

branches: 1.290.2;
- Hide the details of SPCF_SHOULDYIELD and related behind a couple of small
functions: preempt_point() and preempt_needed().

- preempt(): if the LWP has exceeded its timeslice in kernel, strip it of
any priority boost gained earlier from blocking.

OOPS -- fix mistake in previous commit.

bbusy really needs to return the error; otherwise things are very
bad!

Buffer cache SDT probes.

biodone2(): don't acquire kernel_lock for anybody anymore.

branches: 1.286.2;
Rename uvm_free() -> uvm_availmem().

s/transfered/transferred/

uvmexp.free -> uvm_free()

Add a comment.

For safety, cv_broadcast(&bp->b_busy) in more places where the buffer is
changing identity or moving from one vnode list to another.

Adjustment to previous: if we're going to toss the buffer, then wake
everybody.

- Avoid thundering herd: cv_broadcast(&bp->b_busy) -> cv_signal(&bp->b_busy)
- Sprinkle __cacheline_aligned.

Change buf_nbuf()'s return value from int to u_int to avoid undefined
behavior in wapbl_start() which extended int to size_t.

Error message was:
> UBSan: Undefined Behavior in ../../../../kern/vfs_wapbl.c:609:41, signed integer overflow: 3345138 * 1024 cannot be represented in type 'int'

> /* XXX maybe use filesystem fragment size instead of 1024 */
> /* XXX fix actual number of buffers reserved per filesystem. */
> wl->wl_bufcount_max = (buf_nbuf() / 2) * 1024;

Need more work?

Fix kernel pointer leaks in sysctl_dobuf. While here constify argument.

Also memset the buffer, to prevent leaks (even if there doesn't seem to
be currently).

Make sure getnewbuf() runs bawrite() inside fstrans.
Use fstrans_start_nowait() to skip buffers that would block.

branches: 1.276.2; 1.276.4;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

normalise a BIOHIST log message

move some buffer cache internals declarations from buf.h to vfs_bio.c.
this is needed to avoid name conflicts with ZFS and also
makes it clearer that other code shouldn't be messing with these.
remove the LFS debug code that poked around in bufqueues and
remove the BQ_EMPTY bufqueue since nothing uses it anymore.
provide a function to let LFS and wapbl read the value of nbuf for now.

branches: 1.273.2;
When logging a history record for biowait(), include the return address
as a parameter, to identify to which of the many calls to biowait() the
record refers.

expose disk device FUA/DPO support via DIOCGCACHE, and allow the flags
to be set for I/O; implement support in sd(4) and nvme(4)

discussed on tech-kern

Use brelsel while the bufcache_lock is held rather than dropping it
and re-taking / dropping it in brelse

Nix trailing whitespace.

Fix build

Simplify getiobuf. buf_init already does bp->b_objlock == &buffer_lock

branches: 1.267.2;
Remove some extraneous whitespace

Decouple BIOHIST from other users of KERNHIST.

Fix locking so we don't release the lock between the time we check the
tailq (for being non-empty) and the time we remove an entry.

Add a BIOHIST option. As mentioned on tech-kern.

typo in comment

fixup comment

Allow sparc kernels to build with SSP by using a constant PAGE_SIZE...

In bwrite, add assertion that vp != NULL. (vp is the vnode from the
buffer being written.)

There's some logic here that carefully checks for vp being null, and
other logic that will crash if it is. It appears that it's all
needless paranoia. See tech-kern for more info.

Unless someone sees the assertion go off (in which case a lot more
investigation is needed) I or someone will clean out the logic at some
future point.

Spotted by coypu.

branches: 1.259.2;
Implement the 'io' provider for DTrace. From riastradh@, with
fixes from me.

Whatever the point of this "biodone_vfs" global function pointer is
(something rumpity?) declare it properly in a header file instead of
in secret where its types can diverge.

KASSERT->KASSERTMSG to allow debugging a double-free'd buffer in ddb.

to garnish, dust with _KERNEL_OPT

Remove the 'cred' argument from bread(). Remove a now unused var in
ffs_snapshot.c. Update the man page accordingly.

ok hannken@

Remove the 'cred' argument from breadn(), and update the man page
accordingly.

ok hannken@

Remove the 'cred' argument from bio_doread().

branches: 1.252.2;
Replace random with cprng_fast32. Reorganise computation to replace
(32bit) division with (long) multiplication.

Don't next structure and enum definitions.
Don't use C++ keywords new, try, class, private, etc.

Call biodone() in the bdev_strategy() error via a pointer. Decouples
subr_devsw from VFS -- not that I/O buffers are _VFS_ entities -- and
eliminates the last weak alias from librump, which means things now
fully work on glibc (w/o LD_DYNAMIC_WEAK) and musl.

The whole code path is suspect anyway, since nothing prevents the device
from escaping after the lookup, suggesting that the whole error path
should be handled by the caller, but oh well.

branches: 1.249.2;
Ensure that the top level sysctl nodes (kern, vfs, net, ...) exist before
the sysctl link sets are processed, and remove redundancy.

Shaves >13kB off of an amd64 GENERIC, not to mention >1k duplicate
lines of code.

Mark a diagnostic-only variable

Replace macro v_specmountpoint with two functions spec_node_getmountedfs()
and spec_node_setmountedfs() to manage the file system mounted on a device.
Assert the device is a block device.

Welcome to 6.99.24

Discussed on tech-kern@ some time ago.

Reviewed by: David Holland <dholland@netbsd.org>

Remove unused variable

branches: 1.245.2;
remove useless initialization
http://m00nbsd.net/ae123a9bae03f7dde5c6d654412daf5a.html

Redo case where buf_map is not yet mapped when buf_memcalc() is called
early from pmap_bootstrap(). Makes alpha, and probably others, boot
again.

calculate vnode cache size based on the resource it gets allocated from
this stops setting kern.maxvnodes to high so it exhausts available space in kmem

http://mail-index.netbsd.org/tech-kern/2013/03/08/msg015095.html

Always call brelse() on error for breadn() too.

Always call brelse() on error. Otherwise a possible error from bread() will
cause the buffer to stay lock and we end up blocking forever in
VOP_CLOSE->spec_close->vinvalbuf->bbysy since the buffer is marked busy
but there is no I/O pending.
This caused my laptop to hang on boot_findwedge because:
findroot: unable to read block 358331527 of dev dk0 (22)

Change bread() and breadn() to never return a buffer on
error and modify all callers to not brelse() on error.

Welcome to 6.99.16

PR kern/46282 (6.0_BETA crash: msdosfs_bmap -> pcbmap -> bread -> bio_doread)

branches: 1.239.2;
Use separate temporaries for the 'int' percentage and the 'long'
water marks.
Previous paniced on sparc64 due to a misaligned copy.

Fix processing of vm.bufmem_lowater and vm.bufmem_hiwater on 64bit systems.
The values are 'u_long' so copying them into an 'int' temporary
(to avoid writing an out of range value into the actual variable)
doesn't work too well at all.
Shows up on amd64 now that the sysctl values are marked as 64bit.
sparc64 must have been badly broken for ages.

Add some pre-processor magic to verify that the type of the data item
passed to sysctl_createv() actually matches the declared type for
the item itself.
In the places where the caller specifies a function and a structure
address (typically the 'softc') an explicit (void *) cast is now needed.
Fixes bugs in sys/dev/acpi/asus_acpi.c sys/dev/bluetooth/bcsp.c
sys/kern/vfs_bio.c sys/miscfs/syncfs/sync_subr.c and setting
AcpiGbl_EnableAmlDebugObject.
(mostly passing the address of a uint64_t when typed as CTLTYPE_INT).
I've test built quite a few kernels, but there may be some unfixed MD
fallout. Most likely passing &char[] to char *.
Also add CTLFLAG_UNSIGNED for unsiged decimals - not set yet.

branches: 1.236.2;
allocate uareas and buffers from kernel_map again
add code to drain pools if kmem_arena runs out of space

pool_page_alloc, pool_page_alloc_meta: avoid extra compare, use const.
ffs_mountfs,sys_swapctl: replace memset with kmem_zalloc.
sys_swapctl: move kmem_free outside the lock path.
uvm_init: fix comment, remove pointless numeration of steps.
uvm_map_enter: remove meflagval variable.
Fix some indentation.

extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged

sysctl_dobuf: re-acquire the sysctl lock on retry path. PR/45827.

branches: 1.232.2; 1.232.6;
Make parts of this this somewhat less gross by using ilog2() from
<sys/bitops.h>. Additionally, concoct shorter wchan names for >=1 MiB
buf pools using the irrelevantly-more-correct format specifier for u_int.

Change VOP_BWRITE() to take a vnode as its first argument like all other
VOPs do. Layered file systems no longer have to modify bp->b_vp and run
into trouble when an async VOP_BWRITE() uses the wrong vnode.

- change all occurences of VOP_BWRITE(bp) to VOP_BWRITE(bp->b_vp, bp).
- remove layer_bwrite().
- welcome to 5.99.55

Adresses PR kern/38762 panic: vwakeup: neg numoutput

No objections from tech-kern@.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.229.2;
- Replace few malloc(9) uses with kmem(9).
- Rename buf_malloc() to buf_alloc(), fix comments.
- Remove some unnecessary inclusions.

G/C count_lock_queue (unused for 12 years)

Include internal definitions (uvm/uvm.h) only where necessary.

branches: 1.226.2;
Fix nestio's behavior on error.

The mbp->b_resid is used to track if all the nested buffers have been issued
and reported back. When the last buffer calls in, mbp->b_resid becomes zero
and biodone(mbp) is called. This is fine as long as there are no errors.

If a read-error does occure in one of the nested buffers, the mbp->b_error is
set and on its call to biodone(mbp), with mbp->b_resid is zero, physio()
panics since it asserts that IF an error is set on a buffer, there should be a
residual amount of data left to transfered.

The patch fixes this case by setting mbp->b_resid back to mbp->b_bcount on
mbp->b_error just before biodone(mbp).

This behaviour is consistent with normal buffer issueing. It either succeeds
or doesn't succeed.

brelsel: Clear B_COWDONE flag on clean (! BO_DELWRI) buffer. B_COWDONE is set
if the buffer was read with intention to modify but the caller changed its mind.

This error could lead to snapshot corruption when a buffer with B_COWDONE set
resides on the freelist and we create a new snapshot.

Don't sleep forever if hz < 25.

from Alessandro Forin

branches: 1.223.2;
fs_ffs.h is no longer required (since the death of bufops / softdep)

branches: 1.222.2;
Add a comment saying "name" to pool_init() is never freed (fixing
requires touching pool implementation). No biggie, though, since
the pools themselves are never freed.

Add a small comment on buffer cache locking, fix mark letter b_objlock.

G/C unused breada() and bdirty().

Excommunicate comment not abiding to the 80col dogma.
(well, turns out it was no longer valid either)

like KERN_FILE2: *do* update "needed" when there is no count. we want
userland to know what sort of size to provide..

while here, slightly normalise the previous to init_sysctl.c.

Fix some comments.

PR kern/26878 FFSv2 + softdep = livelock (no free ram)
PR kern/16942 panic with softdep and quotas
PR kern/19565 panic: softdep_write_inodeblock: indirect pointer #1 mismatch
PR kern/26274 softdep panic: allocdirect_merge: ...
PR kern/26374 Long delay before non-root users can write to softdep partitions
PR kern/28621 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
PR kern/29513 FFS+Softdep panic with unfsck-able file-corruption
PR kern/31544 The ffs softdep code appears to fail to write dirty bits to disk
PR kern/31981 stopping scsi disk can cause panic (softdep)
PR kern/32116 kernel panic in softdep (assertion failure)
PR kern/32532 softdep_trackbufs deadlock
PR kern/37191 softdep: locking against myself
PR kern/40474 Kernel panic after remounting raid root with softdep

Retire softdep, pass 2. As discussed and later formally announced on the
mailing lists.

branches: 1.215.2;
Move some sysctl node creations away from linksets and into the
constructors for subsystems.

XXX: CTLFLAG_PERMANENT is non-sensible.

Backout revision 1.212 and add a comment that short-cutting the WAPBL
case is not possible. The buffer length has changed and the rounded size
may not have, essentially changing the transaction size. Reported by
various users and in PR 39898.

Move WAPL replay handling from bread() into ufs_strategy.
This changes the order of hook processing as the copy-on-write handlers
are called after the journal processing. This makes more sense as the
journal overwrite is logically part of the disk IO.

If the size of the buffer didn't change, don't bother updating the WAPBL
accounting as it won't change either.

Don't dereference bp->b_vp->v_mount if its vnode type is VT_VNON. I dont
know if this masks a bug but with a machine having a ffs+wapbl mount, NFS
mounts and a ntfs mount this paniced the machine on suspend.

branches: 1.210.2; 1.210.4; 1.210.6; 1.210.10;
nestiobuf_setup(): Initialize b_dev from master buffer.

Accidental commit, but asserts buffer cache lock held

Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

bdwrite(): The COWDONE check may fire for special vnodes with mounted
non-ffs file system. Remove VBLK vnodes from the check.

Should fix PR kern/38892

branches: 1.206.2;
kern/39052: fix broken assertion. We can have a BC_BUSY buffer in the LRU
queue, if it's being flushed. But in this case, we'll also have B_VFLUSH.

While there fix checkfreelist() so that it can be used to check that a
buffer is not in the free lists.

Drop !cv_has_waiters assertion.

bdirty() is called from within biodone() processing before
waiters have been woken up and removed.

N.B. it is also used by smbfs.

Mark a buffer `busy` in getnewbuf() when it came from the pool_cache since
its not on a free list.

Also change buf_init() to not automatically mark buffers `busy' since this
only makes sense for bufcache buffers.

Mark all buf_init'd buffers 'busy' on the places where they ought to be
flagged as such to not confuse the buffer cache.

Fixes PR 38923.

Sprinkle more assertions.

branches: 1.202.2;
- Make getiobuf() return buffers marked BUSY.
- Sprinkle more assertions.

biodone2: if the buffer is async or has a callback method, assert that
there are no waiters on b_done (threads in biowait()).

tsleep -> kpause

brelse: always wakeup on b_busy, in case BC_WANTED is not set.

Make sure all cached buffers with valid, not yet written data have been
run through copy-on-write. Call fscow_run() with valid data where possible.

The LP_UFSCOW hack is no longer needed to protect ffs_copyonwrite() against
endless recursion.

- Add a flag B_MODIFY to bread(), breada() and breadn(). If set the caller
intends to modify the buffer returned.

- Always run copy-on-write on buffers returned from ffs_balloc().

- Add new function ffs_getblk() that gets a buffer, assigns a new blkno,
may clear the buffer and runs copy-on-write. Process possible errors
from getblk() or fscow_run(). Part of PR kern/38664.

Welcome to 4.99.63

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.197.2;
- Convert hashinit() to use kmem_alloc(). The hash tables can be large
and it's better to not have them in kmem_map.
- Convert a couple of minor items along the way to kmem_alloc().
- Fix some memory leaks.

Remove clause 3 and 4 from TNF licenses

branches: 1.195.2;
When using nested buffers, allow one erroring-out nested buffer to
error-out the master buffer.

The old setup was undeterministic since a later sheduled nested buffer
could clear the error again since there is no B_ERROR flag anymore. It also
would discard the error the nested buffer returned.

branches: 1.194.2;
Make rusage collection per-LWP and collate in the appropriate places.
cloned threads need a little bit more work but the locking needs to
be fixed first.

- for some ports, especially for ones without pmap_growkernel,
buf_memcalc is used by bootstrap as well. fix NULL dereference for them.
- limit kva usage for each cache to 20% of vm_map. XXX a bit arbitrary.
- add a comment.

when calculating some cache sizes, consider the amount of available kva.
PR/33185.

make buf_map static.

update a comment.

branches: 1.189.2; 1.189.6;
Merge all the *different* definitions of bufqueues into one common one.

Give bbusy() an interlock argument. If the we need to wait for the buffer,
the interlock is dropped and reacquired when awoken. This allows for
busying buffers attached to a list that is not locked by bufcache_lock.

The buffer LOCKED flag need not be under the protection of bufcache_lock,
BUSY is enough.

BO_COWDONE -> B_COWDONE: this flag is tested/modified from the thread owning
the buffer and therefore needs no protection by a mutex.

Ok: Andrew Doran <ad@netbsd.org>

Patch up sysctl locking:

- Lock processes, credentials, filehead etc correctly.
- Acquire a read hold on sysctl_treelock if only doing a query.
- Don't wire down the output buffer. It doesn't work correctly and the code
regularly does long term sleeps with it held - it's not worth it.
- Don't hold locks other than sysctl_lock while doing copyout().
- Drop sysctl_lock while doing copyout / allocating memory in a few places.
- Don't take kernel_lock for sysctl.
- Fix a number of bugs spotted along the way

bwrite, bdwrite: bufcache_lock must be held for reassignbuf.

Merge vmlocking2 to head.

b_un.b_addr -> b_data

branches: 1.181.2; 1.181.6;
Fscow_run(): add a flag "bool data_valid" to note still valid data.
Buffers run through copy-on-write are marked B_COWDONE. This condition
is valid until the buffer has run through bwrite() and gets cleared from
biodone().

Welcome to 4.99.39.

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

branches: 1.180.2;
Fix vm.bufmem* sysctl nodes for 64bit archs by making them uint64_t,
as discussed on tech-kern. No requests for binary compat - so don't
bother to version them.

branches: 1.179.2;
Merge brelse() changes from the vmlocking branch.

branches: 1.178.2;
Put the RCSID before any other headers

Make bioops a pointer and point it to the softdeps struct in softdep
init. Decouples "options SOFTDEP" from the main kernel and ffs code.

branches: 1.176.2;
POOL_INIT -> pool_init, we need to call bufinit() anyway

branches: 1.175.4; 1.175.6;
B_ERROR is gone.

It's not a good idea for device drivers to modify b_flags, as they don't
need to understand the locking around that field. Instead of setting
B_ERROR, set b_error instead. b_error is 'owned' by whoever completes
the I/O request.

branches: 1.173.2;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

branches: 1.171.2; 1.171.4;
Pass an ipl argument to pool_init/POOL_INIT to be used when initializing
the pool's lock.

branches: 1.170.2;
Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

TRUE -> true, FALSE -> false

branches: 1.168.2;
Merge newlock2 to head.

branches: 1.167.2; 1.167.4;
Export nestiobuf_iodone(). This allows nested iobufs to have a custom
call-back function that can then call the nestiobuf_iodone() to propagate.

remove some __unused from function parameters.

with the introduction of 512 byte buffers, the index in the array is not
the number of kilobytes anymore, so name the pools appropriately.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

branches: 1.163.2;
unexport getnewbuf.

branches: 1.162.2;
use c99 initializers

move wait points for kva from upper layers to vm_map. PR/33185 #1.

XXX there is a concern about interaction with kva fragmentation.
see: http://mail-index.NetBSD.org/tech-kern/2006/05/11/0000.html

branches: 1.160.2;
integrate kauth.

Tell config to generate fs_ffs.h as vfs_bio.c checks for defined(FFS).
Include that header in vfs_bio.c so that bioops are not redefined.

Add one more buffer pool, for 512-byte buffers. On the one hand, most
systems will never, ever need this -- because they use 8k/1k or even,
these days, 16k/2k or 32k/4k filesystems. On the other hand, when you
do need this, you *really* need it: on anoncvs.netbsd.org, for instance,
where /tmp is 4k/512 and the filesystem contains tens or even hundreds
of thousands of single-frag directories, this essentially doubles the
efficiency of the allocator. Since the overhead of keeping one extra
pool around is minimal, just add it by default.

branches: 1.157.2; 1.157.4;
Move ISSET/SET/CLR macros to sys/types.h

branches: 1.156.2;
nestiobuf_iodone: remove a comment which is no longer true.

branches: 1.155.2; 1.155.4;
Propagate an appropiate error code in nestiobuf_iodone() to the master
buffer when the passed nested buffer has no B_ERROR flag set but not all
was transfered for the nested iobuf extent.

Discussed on tech-kern and ok'd by Takashi

allocbuf: yield cpu in a loop.

- use POOL_INIT for bufpool.
- make bufiopool static.

add nestiobuf api for convenience when splitting a request to several pieces.

remove B_EINTR as it isn't used anymore.

use a dedicated buf pool for getiobuf.

- add simple functions to allocate/free a buffer for i/o.
- make bufpool static.

branches: 1.148.2;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

branches: 1.146.2;
Properly fix the constipated lossage wrt -Wcast-qual and the sysctl
code. I know it's not the prettiest code, but it seems to work rather
well in spite of itself.

- add const.
- remove unnecessary casts.
- add __UNCONST casts and mark them with XXXUNCONST as necessary.

merge yamt-km branch.
- don't use managed mappings/backing objects for wired memory allocations.
save some resources like pv_entry. also fix (most of) PR/27030.
- simplify kernel memory management API.
- simplify pmap bootstrap of some ports.
- some related cleanups.

fix validation of new values when setting vm.{hi,low}water. fixes PR 29651.

branches: 1.142.2;
nuke trailing whitespace

branches: 1.141.2; 1.141.4;
Users have observed that the amount of memory used by the metadata cache
can in some situations exceed the high-water mark, and stay there once it
gets there. Adjust the canrelease function so that it will immediately
bring us back down to the high-water mark in this situation.

How can this happen at all? Consider a machine with two filesystems, one
with a much larger blocksize than the other. If the small-block filesystem
is very busy, growing the cache up to the high-water mark, and then the
large-block filesystem becomes busy, buffers will be recycled (since we
are at the high-water mark) but _grow each time they're recycled_. Once
we're above the high-water mark, the canrelease call in allocbuf (without
this change) doesn't shrink us back down below it; so things get worse and
worse.

also define bioops if FFS is not defined.

Fix previous commit, got bufcache and bufmem messages reversed.

Change sysctl -d vm.bufcache to say percent of physical memory not
kernel memory. Addresses PR misc/27233. Approved by atatat@netbsd.org.

PR/25749: Peter Postma: Missing splx() in kernel.

- Testing low memory condition to see if we should alloc or not doesn't make
sense, since 1) the condition is quite normal condition and 2) there is
pool between us and uvm.
- Make the step of allocation possibility a bit seamless by moving the origin
of curve from 0 to lowater mark.
Simon told that this helps for interactive performance when there is heavy
disk activity in PR#27057.

Factor out code to set watermark and ensure high > low.

- Don't let pagedaemon sleep while draining buf.
- Estimate amount of memory to free at a time.
Address PR#27057 (and similar hangs I saw several months ago).

x > 15 is always false if x is 0 .. 15.
# XXX: testing free memory here is quite doubtful. also, I guess lowater
# XXX: is better than 0 as origin.

Cheap test first.

fix allocbuf() O(n**2) behaviour where n is number of AGE buffers
by always tracking amount of buffers on a queue.
bump to 2.0H.

- add missing function prototypes.
- fix prototype mismatches.

buf_trim: a buffer grabbed by getnewbuf() should be clean and anonymous.
thus, there's no need to check and handle B_WANTED here.

- Add flag L_COWINPROGRESS to struct lwp to avoid recursion when
doing copy-on-write.

- Change VFS_SNAPSHOT() to return the snapshot vnode locked.

- Make the IO path for copy-on-write and snapshot-read more lightweight.
Avoids deadlocks where vn_rdwr(...READ...) has a shared lock and needs
to copy-on-write.
Avoids deadlocks/panics where to clean pages the copy-on-write needs
to allocate pages for its VOP_PUTPAGES().

L_COWINPROGRESS part approved by: Jason R. Thorpe <thorpej@netbsd.org>

When initializing the buffer cache memory pools where the size <= PAGE_SIZE,
also use the standard allocator on systems that use a direct-mapped memory
segment for mapping pool pages.

Don't use PR_IMMEDRELEASE on buffer cache pools. Instead, set a high
water mark of 1, which will have the same effect.

Pointed out back in January by YAMAMOTO Takashi.

Remaining sysctl descriptions under kern subtree

bio_doread: vp is always non-NULL here.

Replace the statfs() family of system calls with statvfs().
Retain binary compatibility.

branches: 1.122.2;
Give buf_lotsfree() a bit of a service:
- Fix a 32-bit overflow that could erroneously return true even if the
currently allocated buffer memory was greater than the high water mark.
- Add an early check for bufmem > hiwater to avoid a needless call to
random().
- Sprinkle some comments.

Add a vm.bufmem sysctl so the current bufmem value can be easily queried
from userland.

Reviewed by Thor Simon.

More white space nits.

White-space nit.

Tango on sysctl_createv() and flags. The flags have all been renamed,
and sysctl_createv() now uses more arguments.

micro-optimisation - if we're going to return 0, do so before doing
other unnecessary work

Add PTRTOUINT64() and UINT64TOPTR() macros to sys/sysctl.h for use by
kern.proc, kern.proc2, kern.lwp, and kern.buf.

Define more MIB for kern.buf so that specific buffers can be selected
(only all/all is supported right now), and use a 32/64 bit agnostic
structure for communcating buffer information to userland.

Convert systat to the new kern.buf method.

Clean up the vm.buf* handling a little. There's no actual need to
record the dynamically assigned OIDs, since sysctl_data can tell us
what we're looking at.

Oh, and fix a typo in a comment.

- raise ipl when calling buf_canrelease() because it traverses buffer queue.
- correct/add comments on buf_canrelease().

Fix bug noted by yamt@netbsd.org: the UVM free target is in *pages*,
so the last change has us comparing pages to bytes instead of pages
to buffers! The consequence was to try to free radically less memory
than UVM wanted us to -- though always at least one buffer, which is
probably why the results weren't dire.

This does suggest that buf_canrelease() could be a *lot* more
conservative about how much to release than "2 * page deficit". In
fact, serious trouble seems to ensue if it's not -- when anything
else on the system demands enough pages, we slam down to the low
water mark nd stay there. I've adjusted it to use min(page defecit,
buffer memory / 16), which still isn't quite right but seems better.

Another change: consider the case of an infinite loop that does
"tar xzf pkgsrc.tar.gz ; rm -rf pkgsrc". Each time the rm runs,
all the dead metadata will go on the AGE list -- and, until we hit
the high-water mark, stay there, at which point it may be slowly
recycled. Two adjustments seem to solve this: 1) whack buf_lotsfree()
to return 0 if there's anything on the AGE list; 2) whack buf_canrelease()
to count the memory used by the AGE list and always return at least
that much.

This basically turns the AGE list into a "delayed free" list, since we
can't entirely eliminate it as we can't free pool items from interrupt
context (e.g. from biodone()).

To consider: with the bookkeeping corrected, should buf_drain() move
back to the _end_ of the pagedaemon, and should the calculation then
try to give back at least the current defecit?

Buffer cache fixes to avoid thrashing between high and low water marks
and uncontrolled growth.

The key fix is from Dan Carasone, who noticed that buf_canfree() was
counting in _bytes_ but freeing in _buffers_, which caused the instant
drop to lowater observed by some users.

We now control the rate of growth; the probability of getting a new
allocation is inversely proportional to the current size of the
cache. This idea is from a long-ago conversation with Kirk McKusick
and, if memory serves, was used for the file-system cache in some
other BSD variant at some point in history.

With growth and shrinkage more or less dealt with, we return the
default maximum cache size to 15%. The default _minimum_ cache size
is raised from 1/16 of the maximum cache size to 1/8, since 1/16 was
chosen when the maximum size was 30% of memory.

Finally, after observing the behaviour of the pagedaemon and the
buffer cache drainer under pathological workloads (e.g. a benchmark
that steps through 75% of available memory backwards) I have moved
the call to buf_drain() to the beginning of the pagedaemon from the
end; if the pagedaemon bogs down, it still won't get run as often
as it should, but at least this way it will see the state of the
free count and free target _before_ the scan step does its thing.

Reduce the default BUFCACHE to 10% for now. Too many users are
tripping over this getting too large, and suffering other performance
problems due to the lack of good backpressure shrinking the bufcache
when other memory is required. Again, this tunable should be
revisited when the backpressure mechanism has been improved.

sysctl vm.bufcache can be used to manually tune those rare machines
that might need more than this.

See comments in rev 1.106 for more detail.

Make VOP_STRATEGY(bp) a real VOP as discussed on tech-kern.

VOP_STRATEGY(bp) is replaced by one of two new functions:

- VOP_STRATEGY(vp, bp) Call the strategy routine of vp for bp.
- DEV_STRATEGY(bp) Call the d_strategy routine of bp->b_dev for bp.

DEV_STRATEGY(bp) is used only for block-to-block device situations.

bufpool_page_alloc: for no-wait allocations, specify UVM_KMF_TRYLOCK as well.

Obviously, sizeof(u_int) is not enough to copy struct buf.
Prevents ``sysctl -a'' from dumping core.

reset i/o priority in geteblk() as well.

store a i/o priority hint in struct buf for buffer queue discipline.

Initialize buffer pools with PR_IMMEDRELEASE. Don't use pool_reclaim()
on those pools; it is no longer necessary.

Change BUFCACHE (default hard limit on physmem consumption by metadata
cache) from 30% to 20%. This seems to significantly smooth the oscillation
between "almost no memory available" and "UVM free target available" caused
by the current sudden, heavy backpressure on the metadata cache. We should
revisit this again once the backpressure mechanism is better tuned; ideally,
the hard limit should almost never come into play, because the metadata
cache should gradually give back pages as buffers hit the AGE list and as
the page cache demands them, rather than giving back a big slug of pages
all at once when UVM decides it's in a hurry and fires off the page daemon.

Just how well this adjustment works is likely to vary significantly from
machine to machine depending on I/O mix, filesystem frag size, and total
memory. However, 20% seems to be quite a bit better than 30% on several
systems I've tested and is, coincidentally, more than enough to cache
the entire metadata working set of the AnonCVS server with 100 clients,
which is a useful worst-case stake in the ground...

Add pool_reclaim() on pool to which we just pool_put() a buffer in
buf_mrelease(). Without this, though the pages are returned to the
relevant *pool*, they are never available for any other use in the
system.

Now the backpressure on the physical size of the buffer cache through
the buf_drain() call in the pagedaemon works correctly. If anything,
it may be a bit more aggressive than intended. On my 256MB system,
with vm.bufcache set to the default 30% of physmem, a kernel with this
fix can do 5 simultaneous config/makedep/builds of different NetBSD
kernels in 1313 seconds; with the "traditional" buffer cache code it
requires 1320 seconds. Running "find / -type d -exec ls -l {}" while
the build is going demonstrates that the backpressure is working
correctly: free memory oscillates slowly between close to none and
the UVM target free, and vmstat -m shows a large number of releases
for the buffer pools.

For future work: how is "bufpl" memory returned to the system? This
is not obvious to me (I must be looking in the wrong place). Also,
buf_mrelease() is also called from brelse() in some cases. Would it
be better to add a pool flag causing automatic release of full pages
as they become available (not fragmented)? Jason Thorpe proposed this
and it seems more elegant than cleaning the _entire_ pool only upon
memory pressure.

Greg Oster did a lot of the work of figuring this out. Jason proposed
the use of pool_reclaim as a way to fix it.

Expose the buf_map symbol so that pmap(1) can find it.

Split the sysctl setup routine into two routines, one for each
"subtree". Perhaps it's a little pedantic, but it's cleaner. Also,
assert that the "kern" and "vm" nodes exist.

bufpool_page_free: pass `buf_map' to uvm_km_free().

getnewbuf: return buffer locked.

Consistently use ANSI-style function decls.

Replace the traditional buffer memory management -- based on fixed per buffer
virtual memory reservation and a private pool of memory pages -- by a scheme
based on memory pools.

This allows better utilization of memory because buffers can now be allocated
with a granularity finer than the system's native page size (useful for
filesystems with e.g. 1k or 2k fragment sizes). It also avoids fragmentation
of virtual to physical memory mappings (due to the former fixed virtual
address reservation) resulting in better utilization of MMU resources on some
platforms. Finally, the scheme is more flexible by allowing run-time decisions
on the amount of memory to be used for buffers.

On the other hand, the effectiveness of the LRU queue for buffer recycling
may be somewhat reduced compared to the traditional method since, due to the
nature of the pool based memory allocation, the actual least recently used
buffer may release its memory to a pool different from the one needed by a
newly allocated buffer. However, this effect will kick in only if the
system is under memory pressure.

when ifdef DEBUG and debug_verify_freelist != 0
then perform an expensive search of the buffer freelists
in brelse and bremfree to verify consistency

add explanatory comment in bremfree:
We break the TAILQ abstraction in order to efficiently remove a
buffer from its freelist without having to know exactly which
freelist it is on.

protect a few uses of buf's b_flags with b_interlock

in getblk(), don't call allocbuf() for B_LOCKED buffers.

LFS misses total size of B_LOCKED buffer (locked_queue_bytes) when
getblk() re-size them.

XXX maybe needs a better fix.

buffer with B_CALL shouldn't be brelse'ed. assert it.

bremfree needs bqueue_slock held. assert it.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

branches: 1.92.2;
remove B_NEEDCOMMIT as it's no longer used.

Add a new BUF_INIT() macro which initializes b_dep and b_interlock, and
use it. This fixes a few places where either b_dep or b_interlock were
not properly initialized.

bdwrite(): remove check for MFS major device number (why was 255 changed
to 4096?). In any case, bdevsw_lookup() will take care of it.

In getnewbuf(), release the buffer queue lock before calling bawrite() and
re-acquire it afterward.

Require the bdirty() be called at splbio() and with the buffer interlock held.
This is essentially just a helper routine called from biodone() through
ffs softdep's I/O completion, to re-queue the buffer.

Make the buffer cache code MP-safe.

Merge the nathanw_sa branch.

Merge the gehenna-devsw branch into the trunk.

This merge changes the device switch tables from static array to
dynamically generated by config(8).

- All device switches is defined as a constant structure in device drivers.

- The new grammer ``device-major'' is introduced to ``files''.

device-major <prefix> char <num> [block <num>] [<rules>]

- All device major numbers must be listed up in port dependent majors.<arch>
by using this grammer.

- Added the new naming convention.
The name of the device switch must be <prefix>_[bc]devsw for auto-generation
of device switch tables.

- The backward compatibility of loading block/character device
switch by LKM framework is broken. This is necessary to convert
from block/character device major to device name in runtime and vice versa.

- The restriction to assign device major by LKM is completely removed.
We don't need to reserve LKM entries for dynamic loading of device switch.

- In compile time, device major numbers list is packed into the kernel and
the LKM framework will refer it to assign device major number dynamically.

Use the queue macros from <sys/queue.h> instead of referring to the queue
members directly. Use *_FOREACH whenever possible.

Remove the old device buffer queue interface.

Approved by: Jason R. Thorpe <thorpej@wasabisystems.com>

Make nbuf, nswbuf, and bufpages unsigned. Make all operations on these
variables unsigned, and update places where their values are printed.

branches: 1.81.2;
Eliminate commons.

fix bread() to return errors from reading past the end of the device.
back in rev. 1.51, bread() and breadn() were changed to assume that
if B_DONE is set on a buffer returned by bio_doread(), that the buffer
must have already been in the cache, and thus the overall bread() should
return success. but if the requested buffer is not in the cache and
is past the end of the device, bounds_check_with_label() will set B_ERROR
on the buffer and the caller will call biodone(), which will cause bread()
to think the buffer was already in the cache and thus return success.
to fix this, undo rev. 1.51 and instead have biowait() treat both B_DONE
and B_DELWRI as indicators that it doesn't need to sleep waiting for an
i/o to complete.

Pool deals fairly well with physical memory shortage, but it doesn't
deal with shortages of the VM maps where the backing pages are mapped
(usually kmem_map). Try to deal with this:

* Group all information about the backend allocator for a pool in a
separate structure. The pool references this structure, rather than
the individual fields.
* Change the pool_init() API accordingly, and adjust all callers.
* Link all pools using the same backend allocator on a list.
* The backend allocator is responsible for waiting for physical memory
to become available, but will still fail if it cannot callocate KVA
space for the pages. If this happens, carefully drain all pools using
the same backend allocator, so that some KVA space can be freed.
* Change pool_reclaim() to indicate if it actually succeeded in freeing
some pages, and use that information to make draining easier and more
efficient.
* Get rid of PR_URGENT. There was only one use of it, and it could be
dealt with by the caller.

From art@openbsd.org.

getblk()'s "block size invariant" isn't valid for VBLK vnodes
since bounds_check_with_label() will truncate a buffer that crosses
the end of the partition. adjust the assertion to account for this.
fixes PRs 7938, 12156, 12698, 13076, 13210 and 13288.

add RCSIDs

branches: 1.76.2; 1.76.4; 1.76.6;
in bwrite(), when deciding whether to convert sync writes into delayed writes,
examine the filesystem contained in a block device rather than the filesystem
containing the block device vnode. fixes PR 12484.

in getnewbuf(), when we need to write a buffer before reusing it,
return NULL instead of restarting the loop since we might sleep
while starting the i/o. this tells getblk() to check if someone else
created the buffer while we slept. from OpenBSD.

branches: 1.74.2;
this doesn't need <sys/trace.h>

Initial integration of the Unified Buffer Cache project.

Don't use alloca() - breaks compile on alpha (alloca is not prototyped
anywhere).

NBPG -> PAGE_SIZE

Update for hashinit() change.

use round_page(...) instead of roundup(..., NBPG).

remove include of <vm/vm.h>

branches: 1.67.4;
Fix from Ethan Solomita <ethan@geocast.com> to avoid a livelock problem
where the buffer cache code would be recycling B_AGE buffers with
dependencies.

Get rid of register declarations.

One small piece from UBC: create a pool for I/O buffers. One small piece
not from UBC: make physio use it instead of its own home-grown thing.

Fix a bug in disksort_*() which caused non-optimal ordering when multiple
active partitions were on a single spindle. Add a b_rawblkno member to
struct buf which contains the non-partition-relative block number to sort
by.

- Implement bowrite() -- perform an asynchronous, ordered write.

First round of discarding the CL* macros.

Clear B_AGE in bdirty(), this buffer must go through the LRU again
to be back on the AGE queue. Otherwise we risk recycling a set
of buffers with (soft) dependencies on the AGE list, which may
last forever if the vnode they belong to is locked (i.e. the syncer
won't get to the buffers they depend on, so their dependencies
are never flushed).

Be more careful to block bio interrupts for some data structures. There
were at least a few missed cases where vp->v_{clean,dirty}blkhd were
unprotected since the softdep/trickle sync merge.

Add Kirk McKusick's soft updates code to the trunk. Not enabled by
default, as the copyright on the main file (ffs_softdep.c) is such
that is has been put into gnusrc. options SOFTDEP will pull this
in. This code also contains the trickle syncer.

Bump version number to 1.4O

branches: 1.58.6; 1.58.10; 1.58.12; 1.58.16;
GC the B_CACHE bit.

branches: 1.57.2;
Several things:
* Change the usage of B_DONE so that it is only set when a buffer is in sync
with the data on disk.
* If a buffer is being waited for, don't put it on the age queue.
* Make sure to clear B_DONE when pages are stolen from a buffer.
* Make sure to clear B_CACHE after each use.
* If we find a buffer for the block we want with valid data, but it is too
small, panic. (This isn't supposed to happen.)
Fixes potential file corruption problems with clustering.

Merge paddr_t changes into the main branch.

Abolition of bcopy, ovbcopy, bcmp, and bzero, phase one.
bcopy(x, y, z) -> memcpy(y, x, z)
ovbcopy(x, y, z) -> memmove(y, x, z)
bcmp(x, y, z) -> memcmp(x, y, z)
bzero(x, y) -> memset(x, 0, y)

fix sizeofs so they comply with the KNF style guide. yes, it is pedantic.

branches: 1.53.2;
add flags arg to hashinit(), to pass to malloc().

branches: 1.52.6;
Check `b_dev' field in bdwrite() before using it as an index into bdevsw[].
`b_dev' value of NODEV happens and is normal if the buffer is on its way
to the underlying device strategy function for the first time.

Also, MFS sillily uses a major device number (255) which cannot be used
to index bdevsw[]. Check marked with XXXs.

In bread() and breadn(): if getblk() returns a DELWRI buffer, don't
call biowait() but return `success' immediately. We can return `success'
because buffers with recorded errors are not returned by getblk().
(Takes care of PR#3694).

Fix two performance issues:
* When a delayed write buffer falls off the LRU queue, arrange for it to go on
the AGE queue after being flushed out to disk.
* When a delayed write buffer is synced, leave it in its relative position in
the LRU queue.

curproc was being used directly for ru_{in,ou}block counting. Instead
of using it directly, use a local, and set that local to be curproc
if curproc is not NULL else a pointer to process 0's proc struct.
If syncing disks while handling a panic that occurred while 'curproc'
was NULL, the old code would dereference NULL and die.

backout previous kprintf change

printf -> kprintf, sprintf -> ksprintf

In the sync case of bwrite(), move the accounting earlier so that so that the
delayed write is logically converted to a sync write, mirroring the async case.
In bdwrite(), move the tape case earlier to avoid needless reassignbuf()s.

Call reassignbuf() at splbio in bdwrite().

Protect vnode when updating for started IO on buffers.

branches: 1.43.4;
remove include of <sys/cpu.h>

Changes for NVSv3 code: pull in more NFS include files into kern_time.c
to get types right (overkill for just one function call, but oh well).
Clear B_NEEDCOMMIT in bdwrite().

More proto fixes

First pass at prototyping

fix bug pointed out by, and do the cleanup suggested by
Alasdair Baird <alasdair@wildcat.demon.co.uk>. From pr 1301.

bdwrite() should upgrade writes to tape devices by sending them to
bawrite(). it's logically more correct (doesn't return an error code,
because it's async; bdwrite is also async), it still writes things
in-order, it makes sure the proper accountins is done (see the
wasdelayed cases in bwrite()), and it allows writes to vnodes on volumes
mountd with the MNT_ASYNC to be converted into delayed writes the way
God, err, Kirk intended. Convert synchronous bwrite()s on MNT_ASYNC
file systems to delayed writes.

fix long-standing XXX in getblk(): NFS does funky things (somewhat
explained in comments), which can cause a race condition. amazingly,
the _only_ time i've ever seen or heard of this problem was in some
comments and sources by Rick Macklem, and when running against the
a DEC OSF/1 NFS server running on an Alpha.

fix pr 1128; change vfs_bufstats defn from DIAGNOSTIC - > DEBUG

Use the new d_type field.

Various code rearrangement.

be more careful with types, also pull in headers where necessary.

Patch to fix `reassignbuf: NULL' messages, from cgd.

branches: 1.31.2;
light clean up, use some macros

New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'

Minor cleanup.

Update to 4.4-Lite fs code.

minor type pointed out by Onno van der Linden

Clear more flags in brelse().

Cluster routines want 0-sized bufs.

some paranoia, also, clean up spaces vs. tabs

my two favorite reference books

forgot the damned rcsid

significant rework, to match 4.4-Lite interfaces, and to comment more
closely from Bach.

notdef out vn_bwrite out, for now, so that kernels compile, until new fs stuff

Implement new functions for 4.4-Lite file systems, and some general cleanup.

rearrange some splfoo

clean up a little bit, and minor optimization...

Convert mount, vnode, and buf structs to use <sys/queue.h>. Also,
some knf and structure frobbing to do along with it.

Make FFS optional

get rid of jolitz hack, and add panic() where appropriate

Canonicalize all #includes.

new specfs.h and fifo.h locations

if you try to allocate a buffer larger than MAXBSIZE, panic.

fix my last change; for some reason i thought that 'p' was defined
in these functions. use curproc instead.

pay for block i/o. slightly different than how done by Mark Tinguely.

changed the Debugger() call, which not all kernels have, to panic(),
but only when DIAGNOSTIC is defined.

new, improved, and rationally-implemented vfs_bio. no more serious
structural changes should happen, as it now does the right thing
w.r.t. buffer resizing and having lots of buffers vs. relatively
little buffer space. Ports can now "do the standard thing", re:
nbuf and bufpages, which is make nbuf = bufpages by default.

patch from Christoph Badura <bad@flatlin.ka.sub.org> to fix credential
use by read-ahead blocks. This fixes those weak NFS authentication
messages, and allows us to use BSDI NFS servers again...

convert to use the buffers which are (now) statically allocated at
startup in machdep.c... buffers are now *never* allocated after boot.
currently, the limitation that says bufpages must cover nbuf*MAXBSIZE
still exists, but this is one step closer to removing that limitation.

rewrite biodone to the spec in the daemon book, and to account for
the fact that buffers with B_CALL set shouldn't be brelse()'d.

branches: 1.3.2;
merge in changes from netbsd-0-9-ALPHA2

Use all of freebufspace

branches: 1.1.1; 1.1.2;
replace jolitz's vfs__bio with a better one from CMU via mw.
so, replace vfs__bio, and deal with attendant changes.

kern: Sprinkle biglock-slippage assertions.

We seem to have a poltergeist that occasionally messes with the
biglock depth, but it's very hard to reproduce and only manifests as
some other CPU spinning out on the kernel lock which is no good for
diagnostics.

If we're only doing a count-only kern.buf sysctl, just return the number
of active members in the pool cache (plus some slop) instead of looking
in all the free buffer list. Should reduce CPU usage for "systat vm"
to << 1% especially for machines with a larger number of buffers.

Expose KERN_BUFSLOP in <sys/sysctl.h>.

Pad out the slop for kern.buf based on the passed in element size,
rather than a size of an unrelated struct.

branches: 1.298.2;
Add a sysctl hashstat collector for bufhash.

branches: 1.297.2; 1.297.4;
fix the UFS2 extattr truncate code to play nice with wapbl.
also, rather than pull in the FreeBSD V_NORMAL/V_ALT flags to
vinvalbuf() and the buf b_xflags field and BX_ALTDATA flag,
add a binvalbuf() function to invalid a specific buffer
and use that to invalidate the two possible exattr bufs
during IO_EXT truncations.

uvm_availmem(): give it a boolean argument to specify whether a recent
cached value will do, or if the very latest total must be fetched. It can
be called thousands of times a second and fetching the totals impacts not
only the calling LWP but other CPUs doing unrelated activity in the VM
system.

pass B_PHYS|B_RAW also in nestio_setup(), courtesy to e.g. xbd(4), which
wants to know whether the buf came from user space or bio subsystem

Rename buf_syncwait() to vfs_syncwait(), and have it wait on v_numoutput
rather than BC_BUSY. Removes the dependency on bufhash.

for bmempools set align, not ioff

explicitly use DEV_BSIZE align for all bmempools

this is required for Xen xbd(4) in order to not have to use bounce buffers

the alignment is implicitly provided when POOL_REDZONE is not active,
this change makes it also aligned when POOL_REDZONE _is_ active - that is
when (!KMSAN && (DIAGNOSTIC || KASAN))

Remove buffer reference counting, now that it's safe to destroy b_busy after
waking any waiters.

branches: 1.290.2;
- Hide the details of SPCF_SHOULDYIELD and related behind a couple of small
functions: preempt_point() and preempt_needed().

- preempt(): if the LWP has exceeded its timeslice in kernel, strip it of
any priority boost gained earlier from blocking.

OOPS -- fix mistake in previous commit.

bbusy really needs to return the error; otherwise things are very
bad!

Buffer cache SDT probes.

biodone2(): don't acquire kernel_lock for anybody anymore.

branches: 1.286.2;
Rename uvm_free() -> uvm_availmem().

s/transfered/transferred/

uvmexp.free -> uvm_free()

Add a comment.

For safety, cv_broadcast(&bp->b_busy) in more places where the buffer is
changing identity or moving from one vnode list to another.

Adjustment to previous: if we're going to toss the buffer, then wake
everybody.

- Avoid thundering herd: cv_broadcast(&bp->b_busy) -> cv_signal(&bp->b_busy)
- Sprinkle __cacheline_aligned.

Change buf_nbuf()'s return value from int to u_int to avoid undefined
behavior in wapbl_start() which extended int to size_t.

Error message was:
> UBSan: Undefined Behavior in ../../../../kern/vfs_wapbl.c:609:41, signed integer overflow: 3345138 * 1024 cannot be represented in type 'int'

> /* XXX maybe use filesystem fragment size instead of 1024 */
> /* XXX fix actual number of buffers reserved per filesystem. */
> wl->wl_bufcount_max = (buf_nbuf() / 2) * 1024;

Need more work?

Fix kernel pointer leaks in sysctl_dobuf. While here constify argument.

Also memset the buffer, to prevent leaks (even if there doesn't seem to
be currently).

Make sure getnewbuf() runs bawrite() inside fstrans.
Use fstrans_start_nowait() to skip buffers that would block.

branches: 1.276.2; 1.276.4;
Update the kernhist(9) kernel history code to address issues identified
in PR kern/52639, as well as some general cleaning-up...

(As proposed on tech-kern@ with additional changes and enhancements.)

Details of changes:

* All history arguments are now stored as uintmax_t values[1], both in
the kernel and in the structures used for exporting the history data
to userland via sysctl(9). This avoids problems on some architectures
where passing a 64-bit (or larger) value to printf(3) can cause it to
process the value as multiple arguments. (This can be particularly
problematic when printf()'s format string is not a literal, since in
that case the compiler cannot know how large each argument should be.)

* Update the data structures used for exporting kernel history data to
include a version number as well as the length of history arguments.

* All [2] existing users of kernhist(9) have had their format strings
updated. Each format specifier now includes an explicit length
modifier 'j' to refer to numeric values of the size of uintmax_t.

* All [2] existing users of kernhist(9) have had their format strings
updated to replace uses of "%p" with "%#jx", and the pointer
arguments are now cast to (uintptr_t) before being subsequently cast
to (uintmax_t). This is needed to avoid compiler warnings about
casting "pointer to integer of a different size."

* All [2] existing users of kernhist(9) have had instances of "%s" or
"%c" format strings replaced with numeric formats; several instances
of mis-match between format string and argument list have been fixed.

* vmstat(1) has been modified to handle the new size of arguments in the
history data as exported by sysctl(9).

* vmstat(1) now provides a warning message if the history requested with
the -u option does not exist (previously, this condition was silently
ignored, with only a single blank line being printed).

* vmstat(1) now checks the version and argument length included in the
data exported via sysctl(9) and exits if they do not match the values
with which vmstat was built.

* The kernhist(9) man-page has been updated to note the additional
requirements imposed on the format strings, along with several other
minor changes and enhancements.

[1] It would have been possible to use an explicit length (for example,
uint64_t) for the history arguments. But that would require another
"rototill" of all the users in the future when we add support for an
architecture that supports a larger size. Also, the printf(3) format
specifiers for explicitly-sized values, such as "%"PRIu64, are much
more verbose (and less aesthetically appealing, IMHO) than simply
using "%ju".

[2] I've tried very hard to find "all [the] existing users of kernhist(9)"
but it is possible that I've missed some of them. I would be glad to
update any stragglers that anyone identifies.

normalise a BIOHIST log message

move some buffer cache internals declarations from buf.h to vfs_bio.c.
this is needed to avoid name conflicts with ZFS and also
makes it clearer that other code shouldn't be messing with these.
remove the LFS debug code that poked around in bufqueues and
remove the BQ_EMPTY bufqueue since nothing uses it anymore.
provide a function to let LFS and wapbl read the value of nbuf for now.

branches: 1.273.2;
When logging a history record for biowait(), include the return address
as a parameter, to identify to which of the many calls to biowait() the
record refers.

expose disk device FUA/DPO support via DIOCGCACHE, and allow the flags
to be set for I/O; implement support in sd(4) and nvme(4)

discussed on tech-kern

Use brelsel while the bufcache_lock is held rather than dropping it
and re-taking / dropping it in brelse

Nix trailing whitespace.

Fix build

Simplify getiobuf. buf_init already does bp->b_objlock == &buffer_lock

branches: 1.267.2;
Remove some extraneous whitespace

Decouple BIOHIST from other users of KERNHIST.

Fix locking so we don't release the lock between the time we check the
tailq (for being non-empty) and the time we remove an entry.

Add a BIOHIST option. As mentioned on tech-kern.

typo in comment

fixup comment

Allow sparc kernels to build with SSP by using a constant PAGE_SIZE...

In bwrite, add assertion that vp != NULL. (vp is the vnode from the
buffer being written.)

There's some logic here that carefully checks for vp being null, and
other logic that will crash if it is. It appears that it's all
needless paranoia. See tech-kern for more info.

Unless someone sees the assertion go off (in which case a lot more
investigation is needed) I or someone will clean out the logic at some
future point.

Spotted by coypu.

branches: 1.259.2;
Implement the 'io' provider for DTrace. From riastradh@, with
fixes from me.

Whatever the point of this "biodone_vfs" global function pointer is
(something rumpity?) declare it properly in a header file instead of
in secret where its types can diverge.

KASSERT->KASSERTMSG to allow debugging a double-free'd buffer in ddb.

to garnish, dust with _KERNEL_OPT

Remove the 'cred' argument from bread(). Remove a now unused var in
ffs_snapshot.c. Update the man page accordingly.

ok hannken@

Remove the 'cred' argument from breadn(), and update the man page
accordingly.

ok hannken@

Remove the 'cred' argument from bio_doread().

branches: 1.252.2;
Replace random with cprng_fast32. Reorganise computation to replace
(32bit) division with (long) multiplication.

Don't next structure and enum definitions.
Don't use C++ keywords new, try, class, private, etc.

Call biodone() in the bdev_strategy() error via a pointer. Decouples
subr_devsw from VFS -- not that I/O buffers are _VFS_ entities -- and
eliminates the last weak alias from librump, which means things now
fully work on glibc (w/o LD_DYNAMIC_WEAK) and musl.

The whole code path is suspect anyway, since nothing prevents the device
from escaping after the lookup, suggesting that the whole error path
should be handled by the caller, but oh well.

branches: 1.249.2;
Ensure that the top level sysctl nodes (kern, vfs, net, ...) exist before
the sysctl link sets are processed, and remove redundancy.

Shaves >13kB off of an amd64 GENERIC, not to mention >1k duplicate
lines of code.

Mark a diagnostic-only variable

Replace macro v_specmountpoint with two functions spec_node_getmountedfs()
and spec_node_setmountedfs() to manage the file system mounted on a device.
Assert the device is a block device.

Welcome to 6.99.24

Discussed on tech-kern@ some time ago.

Reviewed by: David Holland <dholland@netbsd.org>

Remove unused variable

branches: 1.245.2;
remove useless initialization
http://m00nbsd.net/ae123a9bae03f7dde5c6d654412daf5a.html

Redo case where buf_map is not yet mapped when buf_memcalc() is called
early from pmap_bootstrap(). Makes alpha, and probably others, boot
again.

calculate vnode cache size based on the resource it gets allocated from
this stops setting kern.maxvnodes to high so it exhausts available space in kmem

http://mail-index.netbsd.org/tech-kern/2013/03/08/msg015095.html

Always call brelse() on error for breadn() too.

Always call brelse() on error. Otherwise a possible error from bread() will
cause the buffer to stay lock and we end up blocking forever in
VOP_CLOSE->spec_close->vinvalbuf->bbysy since the buffer is marked busy
but there is no I/O pending.
This caused my laptop to hang on boot_findwedge because:
findroot: unable to read block 358331527 of dev dk0 (22)

Change bread() and breadn() to never return a buffer on
error and modify all callers to not brelse() on error.

Welcome to 6.99.16

PR kern/46282 (6.0_BETA crash: msdosfs_bmap -> pcbmap -> bread -> bio_doread)

branches: 1.239.2;
Use separate temporaries for the 'int' percentage and the 'long'
water marks.
Previous paniced on sparc64 due to a misaligned copy.

Fix processing of vm.bufmem_lowater and vm.bufmem_hiwater on 64bit systems.
The values are 'u_long' so copying them into an 'int' temporary
(to avoid writing an out of range value into the actual variable)
doesn't work too well at all.
Shows up on amd64 now that the sysctl values are marked as 64bit.
sparc64 must have been badly broken for ages.

Add some pre-processor magic to verify that the type of the data item
passed to sysctl_createv() actually matches the declared type for
the item itself.
In the places where the caller specifies a function and a structure
address (typically the 'softc') an explicit (void *) cast is now needed.
Fixes bugs in sys/dev/acpi/asus_acpi.c sys/dev/bluetooth/bcsp.c
sys/kern/vfs_bio.c sys/miscfs/syncfs/sync_subr.c and setting
AcpiGbl_EnableAmlDebugObject.
(mostly passing the address of a uint64_t when typed as CTLTYPE_INT).
I've test built quite a few kernels, but there may be some unfixed MD
fallout. Most likely passing &char[] to char *.
Also add CTLFLAG_UNSIGNED for unsiged decimals - not set yet.

branches: 1.236.2;
allocate uareas and buffers from kernel_map again
add code to drain pools if kmem_arena runs out of space

pool_page_alloc, pool_page_alloc_meta: avoid extra compare, use const.
ffs_mountfs,sys_swapctl: replace memset with kmem_zalloc.
sys_swapctl: move kmem_free outside the lock path.
uvm_init: fix comment, remove pointless numeration of steps.
uvm_map_enter: remove meflagval variable.
Fix some indentation.

extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged

sysctl_dobuf: re-acquire the sysctl lock on retry path. PR/45827.

branches: 1.232.2; 1.232.6;
Make parts of this this somewhat less gross by using ilog2() from
<sys/bitops.h>. Additionally, concoct shorter wchan names for >=1 MiB
buf pools using the irrelevantly-more-correct format specifier for u_int.

Change VOP_BWRITE() to take a vnode as its first argument like all other
VOPs do. Layered file systems no longer have to modify bp->b_vp and run
into trouble when an async VOP_BWRITE() uses the wrong vnode.

- change all occurences of VOP_BWRITE(bp) to VOP_BWRITE(bp->b_vp, bp).
- remove layer_bwrite().
- welcome to 5.99.55

Adresses PR kern/38762 panic: vwakeup: neg numoutput

No objections from tech-kern@.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.229.2;
- Replace few malloc(9) uses with kmem(9).
- Rename buf_malloc() to buf_alloc(), fix comments.
- Remove some unnecessary inclusions.

G/C count_lock_queue (unused for 12 years)

Include internal definitions (uvm/uvm.h) only where necessary.

branches: 1.226.2;
Fix nestio's behavior on error.

The mbp->b_resid is used to track if all the nested buffers have been issued
and reported back. When the last buffer calls in, mbp->b_resid becomes zero
and biodone(mbp) is called. This is fine as long as there are no errors.

If a read-error does occure in one of the nested buffers, the mbp->b_error is
set and on its call to biodone(mbp), with mbp->b_resid is zero, physio()
panics since it asserts that IF an error is set on a buffer, there should be a
residual amount of data left to transfered.

The patch fixes this case by setting mbp->b_resid back to mbp->b_bcount on
mbp->b_error just before biodone(mbp).

This behaviour is consistent with normal buffer issueing. It either succeeds
or doesn't succeed.

brelsel: Clear B_COWDONE flag on clean (! BO_DELWRI) buffer. B_COWDONE is set
if the buffer was read with intention to modify but the caller changed its mind.

This error could lead to snapshot corruption when a buffer with B_COWDONE set
resides on the freelist and we create a new snapshot.

Don't sleep forever if hz < 25.

from Alessandro Forin

branches: 1.223.2;
fs_ffs.h is no longer required (since the death of bufops / softdep)

branches: 1.222.2;
Add a comment saying "name" to pool_init() is never freed (fixing
requires touching pool implementation). No biggie, though, since
the pools themselves are never freed.

Add a small comment on buffer cache locking, fix mark letter b_objlock.

G/C unused breada() and bdirty().

Excommunicate comment not abiding to the 80col dogma.
(well, turns out it was no longer valid either)

like KERN_FILE2: *do* update "needed" when there is no count. we want
userland to know what sort of size to provide..

while here, slightly normalise the previous to init_sysctl.c.

Fix some comments.

PR kern/26878 FFSv2 + softdep = livelock (no free ram)
PR kern/16942 panic with softdep and quotas
PR kern/19565 panic: softdep_write_inodeblock: indirect pointer #1 mismatch
PR kern/26274 softdep panic: allocdirect_merge: ...
PR kern/26374 Long delay before non-root users can write to softdep partitions
PR kern/28621 1.6.x "vp != NULL" panic in ffs_softdep.c:4653 while unmounting a softdep (+quota) filesystem
PR kern/29513 FFS+Softdep panic with unfsck-able file-corruption
PR kern/31544 The ffs softdep code appears to fail to write dirty bits to disk
PR kern/31981 stopping scsi disk can cause panic (softdep)
PR kern/32116 kernel panic in softdep (assertion failure)
PR kern/32532 softdep_trackbufs deadlock
PR kern/37191 softdep: locking against myself
PR kern/40474 Kernel panic after remounting raid root with softdep

Retire softdep, pass 2. As discussed and later formally announced on the
mailing lists.

branches: 1.215.2;
Move some sysctl node creations away from linksets and into the
constructors for subsystems.

XXX: CTLFLAG_PERMANENT is non-sensible.

Backout revision 1.212 and add a comment that short-cutting the WAPBL
case is not possible. The buffer length has changed and the rounded size
may not have, essentially changing the transaction size. Reported by
various users and in PR 39898.

Move WAPL replay handling from bread() into ufs_strategy.
This changes the order of hook processing as the copy-on-write handlers
are called after the journal processing. This makes more sense as the
journal overwrite is logically part of the disk IO.

If the size of the buffer didn't change, don't bother updating the WAPBL
accounting as it won't change either.

Don't dereference bp->b_vp->v_mount if its vnode type is VT_VNON. I dont
know if this masks a bug but with a machine having a ffs+wapbl mount, NFS
mounts and a ntfs mount this paniced the machine on suspend.

branches: 1.210.2; 1.210.4; 1.210.6; 1.210.10;
nestiobuf_setup(): Initialize b_dev from master buffer.

Accidental commit, but asserts buffer cache lock held

Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

bdwrite(): The COWDONE check may fire for special vnodes with mounted
non-ffs file system. Remove VBLK vnodes from the check.

Should fix PR kern/38892

branches: 1.206.2;
kern/39052: fix broken assertion. We can have a BC_BUSY buffer in the LRU
queue, if it's being flushed. But in this case, we'll also have B_VFLUSH.

While there fix checkfreelist() so that it can be used to check that a
buffer is not in the free lists.

Drop !cv_has_waiters assertion.

bdirty() is called from within biodone() processing before
waiters have been woken up and removed.

N.B. it is also used by smbfs.

Mark a buffer `busy` in getnewbuf() when it came from the pool_cache since
its not on a free list.

Also change buf_init() to not automatically mark buffers `busy' since this
only makes sense for bufcache buffers.

Mark all buf_init'd buffers 'busy' on the places where they ought to be
flagged as such to not confuse the buffer cache.

Fixes PR 38923.

Sprinkle more assertions.

branches: 1.202.2;
- Make getiobuf() return buffers marked BUSY.
- Sprinkle more assertions.

biodone2: if the buffer is async or has a callback method, assert that
there are no waiters on b_done (threads in biowait()).

tsleep -> kpause

brelse: always wakeup on b_busy, in case BC_WANTED is not set.