Add memfd_create(2) from GSoC 2023 by Theodore Preduta

file(9): New fo_posix_fadvise operation.

XXX kernel revbump -- changes struct fileops API and ABI

file(9): New fo_fpathconf operation.

XXX kernel revbump -- struct fileops API and ABI change

file(9): New fo_advlock operation.

This moves the vnode-specific logic from sys_descrip.c into
vfs_vnode.c, like we did for fo_seek.

XXX kernel revbump -- struct fileops API and ABI change

readdir(2), lseek(2): Fix races in access to struct file::f_offset.

For non-directory vnodes:
- reading f_offset requires a shared or exclusive vnode lock
- writing f_offset requires an exclusive vnode lock

For directory vnodes, access (read or write) requires either:
- a shared vnode lock AND f_lock, or
- an exclusive vnode lock.

This way, two files for the same underlying directory vnode can still
do VOP_READDIR in parallel, but if two readdir(2) or lseek(2) calls
run in parallel on the same file, the load and store of f_offset is
atomic (otherwise, e.g., on 32-bit systems it might be torn and lead
to corrupt offsets).

There is still a potential problem: the _whole transaction_ of
readdir(2) may not be atomic. For example, if thread A and thread B
read n bytes of directory content, thread A might get bytes [0,n) and
thread B might get bytes [n,2n) but f_offset might end up at n
instead of 2n once both operations complete. (However, f_offset
wouldn't be some corrupt garbled number like n & 0xffffffff00000000.)
Fixing this would require either:
(a) using an exclusive vnode lock in vn_readdir,
(b) introducing a new lock that serializes vn_readdir on the same
file (but ont necessarily the same vnode), or
(c) proving it is safe to hold f_lock across VOP_READDIR, VOP_SEEK,
and VOP_GETATTR.

vn_open(9): Add assertion that vp is locked on return.

Null out vp internally out of paranoia so we'll crash in evaluating
the assertion if we ever reach it via one of the vput paths.

vn_open(9): Clarify that this returns a locked vnode.

Comment only, no functional change intended.

vnodeops(9): Take exclusive lock in read/seek for f_offset update.

Otherwise concurrent readers/seekers might clobber it.

Make kqueue event status for vnodes shareable, and for stacked file systems
like nullfs, make the upper vnode share that status with the lower vnode.

And, lo, NetBSD 9.99.99.

Fixes PR kern/56713.

kern: Work around spurious -Wtype-limits warnings.

This useless garbage warning is apparently designed to make it
painful to write portable safe arithmetic and I think we ought to
just disable it.

kern/vfs_vnops.c: Fix missing semicolon in previous.

Neglected to build and amend commit, oops.

kern/vfs_vnops.c: Sprinkle KNF.

No functional change intended.

mmap(2): Avoid overflow in overflow check in vn_mmap.

uvm(9): fo_mmap caller guarantees positive size.

No functional change intended, just sprinkling assertions to make it
clearer.

fix various small typos, mainly in comments.

It is impossible for VOP_LOCK() to return ENOENT with LK_RETRY flag.
Remove the second call to VOP_LOCK().

Enable assertion "vrefcnt(vp) > 0" and assert all possible errors
for all LK_RETRY/LK_NOWAIT combinations.

Lock vnode across VOP_OPEN.

vfs(9): Avoid arithmetic overflow in vn_seek.

Reported-by: syzbot+b9f9a02148a40675c38a@syzkaller.appspotmail.com

Overhaul of the EVFILT_VNODE kevent(2) filter:

- Centralize vnode kevent handling in the VOP_*() wrappers, rather than
forcing each individual file system to deal with it (except VOP_RENAME(),
because VOP_RENAME() is a mess and we currently have 2 different ways
of handling it; at least it's reasonably well-centralized in the "new"
way).
- Add support for NOTE_OPEN, NOTE_CLOSE, NOTE_CLOSE_WRITE, and NOTE_READ,
compatible with the same events in FreeBSD.
- Track which kevent notifications clients are interested in receiving
to avoid doing work for events no one cares about (avoiding, e.g.
taking locks and traversing the klist to send a NOTE_WRITE when
someone is merely watching for a file to be deleted, for example).

In support of the above:

- Add support in vnode_if.sh for specifying PRE- and POST-op handlers,
to be invoked before and after vop_pre() and vop_post(), respectively.
Basic idea from FreeBSD, but implemented differently.
- Add support in vnode_if.sh for specifying CONTEXT fields in the
vop_*_args structures. These context fields are used to convey information
between the file system VOP function and the VOP wrapper, but do not
occupy an argument slot in the VOP_*() call itself. These context fields
are initialized and subsequently interpreted by PRE- and POST-op handlers.
- Version VOP_REMOVE(), uses the a context field for the file system to report
back the resulting link count of the target vnode. Return this in tmpfs,
udf, nfs, chfs, ext2fs, lfs, and ufs.

NetBSD 9.99.92.

sys/kern: Avoid fp->f_offset without the object (here, vnode) lock.

sys/kern: Allow custom fileops to specify fo_seek method.

Previously only vnodes allowed lseek/pread[v]/pwrite[v], which meant
converting a regular device to a cloning device doesn't always work.

Semantics is:

(*fp->f_ops->fo_seek)(fp, delta, whence, newoffp, flags)

1. Compute a new offset according to whence + delta -- that is, if
whence is SEEK_CUR, add delta to fp->f_offset; if whence is
SEEK_END, add delta to end of file; if whence is SEEK_CUR, use delta
as is.

2. If newoffp is nonnull, return the new offset in *newoffp.

3. If flags & FOF_UPDATE_OFFSET, set fp->f_offset to the new offset.

Access to fp->f_offset, and *newoffp if newoffp = &fp->f_offset, must
happen under the object lock (e.g., vnode lock), in order to
synchronize fp->f_offset reads and writes.

This change has the side effect that every call to VOP_SEEK happens
under the vnode lock now, when previously it didn't. However, from a
review of all the VOP_SEEK implementations, it does not appear that
any file system even examines the vnode, let alone locks it. So I
think this is safe -- and essentially the only reasonable way to do
things, given that it is used to validate a change from oldoff to
newoff, and oldoff becomes stale the moment we unlock the vnode.

No kernel bump because this reuses a spare entry in struct fileops,
and it is safe for the entry to be null, so all existing fileops will
continue to work as before (rejecting seek).

Fix confusion arising from whether FOLLOW or NOFOLLOW is 0.

In vn_open, don't set and then throw away FOLLOW, and clarify the
comment about requesting FOLLOW/NOFOLLOW behavior.

Related to PR 56316.

gcc (with some options) eroneously claims we would use "vp" uninitialized,
so initialize it as NULL.

don't clear the error before we use it to determine if we are moving or duping.

Improve Christos's vn_open fix.

- assert about api misuse up front (suggested by riastradh)
- restore the behavior of returning EOPNOTSUPP if ret_fd is NULL and we
get a fd back (otherwise things like ktruss -o /dev/stderr panic)
- clear error to 0 for the EDUPFD and EMOVEFD cases so opening a
cloner succeeds

PR/56286: Martin Husemann: Fix NULL deref on kmod load.
- No need to set ret_domove and ret_fd in the regular case, they are meaningless
- KASSERT instead of setting errno and then doing the NULL deref.

Add containment for the cloning devices hack in vn_open.

Cloning devices (and also things like /dev/stderr) work by allocating
a struct file, stuffing it in the file table (which is a layer
violation), stuffing the file descriptor number for it in a magic
field of struct lwp (which is gross), and then "failing" with one of
two magic errnos, EDUPFD or EMOVEFD.

Before this commit, all callers of vn_open in the kernel (there are
quite a few) were expected to check for these errors and handle the
situation. Needless to say, none of them except for open() itself did,
resulting in internal negative errnos being returned to userspace.

This hack is fairly deeply rooted and cannot be eliminated all at
once. This commit adds logic to handle the magic errnos inside
vn_open; now on success vn_open returns either a vnode or an integer
file descriptor, along with a flag that says whether the underlying
code requested EDUPFD or EMOVEFD. Callers not prepared to cope with
file descriptors can pass NULL for the extra return values, in which
case if a file descriptor would be produced vn_open fails with
EOPNOTSUPP.

Since I'm rearranging vn_open's signature anyway, stop exposing struct
nameidata. Instead, take three arguments: an optional vnode to use as
the starting point (like openat()), the path, and additional namei
flags to use, restricted to NOCHROOT and TRYEMULROOT. (Other namei
behavior, e.g. NOFOLLOW, can be requested via the open flags.)

This change requires a kernel bump. Ride the one an hour ago.
(That was supposed to be coordinated; did not intend to let an hour
slip by. My fault.)

Add a new namei flag NONEXCLHACK for open with O_CREAT and not O_EXCL.

This case needs to be distinguished from the other CREATE operations
because it is supposed to successfully return (and open) the target if
it exists. In the case where that target is the root, or a mount
point, such that there's no parent dir, "real" CREATE operations fail,
but O_CREAT without O_EXCL needs to succeed.

So (a) add the flag, (b) test for it in namei in the situation
described above, (c) set it in open under the appropriate
circumstances, and (d) because this can result in namei returning
ni_dvp of NULL, cope with that case.

Should get into -9 and maybe even -8, because it was prompted by
issues with 3rd-party code. The use of a flag (vs. adding an
additional nameiop, which would be more appropriate) was deliberate to
make the patch small and noninvasive.

branches: 1.214.4;
Lock the vnode while calling VOP_BMAP() for FIOGETBMAP.

Reported-by: syzbot+cfa1b773be7337250428@syzkaller.appspotmail.com

branches: 1.213.2;
Counter tweaks:

- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
each kind of page can be summed to get the totals.

- Track the number of free pages with a counter so that it's one less thing
for the allocator to do, which opens up further options there.

- Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot.
For the cheap option, give cpu_count_sync() a boolean parameter indicating
that a cached value is okay, and rate limit the updates for cached values
to hz.

Move proc_lock into the data segment. It was dynamically allocated because
at the time we had mutex_obj_alloc() but not __cacheline_aligned.

Replace most uses of vp->v_usecount with a call to vrefcnt(vp), a function
that hides the details and does atomic_load_relaxed(). Signature matches
FreeBSD.

Oops missed one more NULL -> NOCRED

delete debugging printf.

Pass NOCRED instead of NULL for credentials. These routines are supposed
to be accessing system ACL's on behalf of the kernel. This code appears
to be copied from FreeBSD, but there it works because in FreeBSD NOCRED
is 0, ours is -1. I guess nobody has used system extended attributes on
NetBSD yet :-)

branches: 1.207.4;
Tighten up the locking around vp->v_iflag a little more after the recent
split of vmobjlock & v_interlock.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

- Shuffle some items around in struct lwp to save space. Remove an unused
item or two.

- For lockstat, get a useful callsite for vnode locks (caller to vn_lock()).

branches: 1.204.2;
- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

Minor vnode locking changes:

- Stop using atomics to maniupulate v_usecount. It was a mistake to begin
with. It doesn't work as intended unless the XLOCK bit is incorporated in
v_usecount and we don't have that any more. When I introduced this 10+
years ago it was to reduce pressure on v_interlock but it doesn't do that,
it just makes stuff disappear from lockstat output and introduces problems
elsewhere. We could do atomic usecounts on vnodes but there has to be a
well thought out scheme.

- Resurrect LK_UPGRADE/LK_DOWNGRADE which will be needed to work effectively
when there is increased use of shared locks on vnodes.

- Allocate the vnode lock using rw_obj_alloc() to reduce false sharing of
struct vnode.

- Put all of the LRU lists into a single cache line, and do not requeue a
vnode if it's already on the correct list and was requeued recently (less
than a second ago).

Kernel build before and after:

119.63s real 1453.16s user 2742.57s system
115.29s real 1401.52s user 2690.94s system

Add functions to open devices by device number or path.

set VEXEC if FEXEC is set.

branches: 1.200.4;
Change vn_openchk() to fail VNON and VBAD with error ENXIO.

Reported-by: syzbot+d66b1be08516a4d2d2b2@syzkaller.appspotmail.com
Reported-by: syzbot+c5eaef5a8af535c3b217@syzkaller.appspotmail.com

s/fall into .../FALLTHROUGH/

Rename min/max -> uimin/uimax for better honesty.

These functions are defined on unsigned int. The generic name
min/max should not silently truncate to 32 bits on 64-bit systems.
This is purely a name change -- no functional change intended.

HOWEVER! Some subsystems have

#define min(a, b) ((a) < (b) ? (a) : (b))
#define max(a, b) ((a) > (b) ? (a) : (b))

even though our standard name for that is MIN/MAX. Although these
may invite multiple evaluation bugs, these do _not_ cause integer
truncation.

To avoid `fixing' these cases, I first changed the name in libkern,
and then compile-tested every file where min/max occurred in order to
confirm that it failed -- and thus confirm that nothing shadowed
min/max -- before changing it.

I have left a handful of bootloaders that are too annoying to
compile-test, and some dead code:

cobalt ews4800mips hp300 hppa ia64 luna68k vax
acorn32/if_ie.c (not included in any kernels)
macppc/if_gm.c (superseded by gem(4))

It should be easy to fix the fallout once identified -- this way of
doing things fails safe, and the goal here, after all, is to _avoid_
silent integer truncations, not introduce them.

Maybe one day we can reintroduce min/max as type-generic things that
never silently truncate. But we should avoid doing that for a while,
so that existing code has a chance to be detected by the compiler for
conversion to uimin/uimax without changing the semantics until we can
properly audit it all. (Who knows, maybe in some cases integer
truncation is actually intended!)

branches: 1.197.2; 1.197.4;
add fo_name so we can identify the fileops in a simple way.

Add O_REGULAR to enforce opening of only regular files
(like we have O_DIRECTORY for directories).
This is better than open(, O_NONBLOCK), fstat()+S_ISREG() because opening
devices can have side effects.

branches: 1.195.6;
Lock the vnode before changing its writecount.

Must always lock the parent -> lock the child -> unlock the parent.

branches: 1.193.2; 1.193.4;
Remove useless semicolon reported by Henning Petersen in PR#49634.

add a new "fo_mmap" fileops method to allow use of arbitrary uvm_objects for
mappings of file objects. move vnode-specific details of mmap()ing a vnode
from uvm_mmap() to the new vnode-specific vn_mmap(). add new uvm_mmap_dev()
and uvm_mmap_anon() convenience functions for mapping character devices
and anonymous memory, and replace all other calls to uvm_mmap() with those.
use the new fileop in drm2 so that libdrm can use mmap() to map things
like on other platforms (instead of the ioctl that we have used so far).

branches: 1.191.2;
Try not to use f_data, use f_{vnode,socket,pipe,mqueue,kqueue,ksem} to get
a correctly typed pointer.

branches: 1.190.2;
Fix a NULL pointer dereference after a loooong discussion with dholland@,
hannken@, blymn@ and martin@.

This bug would panic the system when veriexec is set to the VERIEXEC_LOCKDOWN
mode (only settable from root).

branches: 1.189.2;
The current implementation of vn_lock() is racy. Modification of
the vnode operations vector for active vnodes is unsafe because it
is not known whether deadfs or the original file system will be
called.

- Pass down LK_RETRY to the lock operation (hint for deadfs only).

- Change deadfs lock operation to return ENOENT if LK_RETRY is unset.

- Change all other lock operations to check for dead vnode once
the vnode is locked and unlock and return ENOENT in this case.

With these changes in place vnode lock operations will never succeed
after vclean() has marked the vnode as VI_XLOCK and before vclean()
has changed the operations vector.

Adresses PR kern/37706 (Forced unmount of file systems is unsafe)

Discussed on tech-kern.

Welcome to 6.99.33

Change vnode operations create, mknod, mkdir and symlink to return
the resulting vnode *vpp unlocked.

Discussed on tech-kern@

Welcome to 6.99.30

Change vnode operations create, mknod, mkdir and symlink to keep the
directory node dvp locked on return.

Discussed on tech-kern@

Welcome to 6.99.29

branches: 1.186.2;
Bring back Manuel Bouyers patch to resolve races between vget() and vrelel()
resulting in vget() returning dead vnodes.
It is impossible to resolve these races in vn_lock().

Needs pullup to NetBSD-6.

branches: 1.185.2;
don't truncate size_t to int

Fix vn_lock() to return an invalid (dead, clean) vnode
only if the caller requested it by setting LK_RETRY.

Should fix PR #46221: Kernel panic in NFS server code

branches: 1.183.2; 1.183.6; 1.183.8;
Change the vnode locking protocol of VOP_GETATTR() to request at least
a shared lock. Make all calls outside of file systems respect it.

The calls from file systems need review.

No objections from tech-kern.

vn_close: add an assertion

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.180.6;
Introduce struct pathbuf. This is an abstraction to hold a pathname
and the metadata required to interpret it. Callers of namei must now
create a pathbuf and pass it to NDINIT (instead of a string and a
uio_seg), then destroy the pathbuf after the namei session is
complete.

Update all namei call sites accordingly. Add a pathbuf(9) man page and
update namei(9).

The pathbuf interface also now appears in a couple of related
additional places that were passing string/uio_seg pairs that were
later fed into NDINIT. Update other call sites accordingly.

Zero entire stat structure before filling in contents to avoid
leaking kernel memory -- the elements are no longer packed now that
dev_t is 64bit.

from pgoyette

implement O_DIRECTORY as standardized in POSIX-2008,
for both native and linux emulations.
this fixes the rest of PR 43695.

I'm not even going to describe this change. I'll just say that
churn creates interesting code.

Fixes open(O_CREAT|O_TRUNC) on at least tmpfs and nfs to not fail
with ENOENT due to a racy removal of the newly created file.

Caught, as most bugs these days are, by a test run.

Modify vn_lock():
- Take v_interlock before examining v_iflag
- Must always be called without v_interlock taken,
LK_INTERLOCK flag is no longer allowed.

Don't leak kernel stack into userspace.

Clean up vnode lock operations pass 2:

VOP_UNLOCK(vp, flags) -> VOP_UNLOCK(vp): Remove the unneeded flags argument.

Welcome to 5.99.32.

Discussed on tech-kern.

Remove the concept of recursive vnode locks by eliminating
vn_setrecurse(), vn_restorerecurse() and LK_CANRECURSE.
Welcome to 5.99.31

Discussed on tech-kern.

Change layered file systems to always pass the locking VOP's down to the
leaf file system. Remove now unused member v_vnlock from struct vnode.
Welcome to 5.99.30

Discussed on tech-kern.

Enforce RLIMIT_FSIZE before VOP_WRITE. This adds support to file
system drivers where it was missing from and fixes one buggy
implementation. The arguably weird semantics of the check are
maintained (v_size vs. va_bytes, overwrite).

Stop exposing fifofs internals and leave only fifo_vnodeop_p visible.

branches: 1.169.2; 1.169.4;
The VATTR_NULL/VREF/VHOLD/HOLDRELE() macros lost their will to live
years ago when the kernel was modified to not alter ABI based on
DIAGNOSTIC, and now just call the respective function interfaces
(in lowercase). Plenty of mix'n match upper/lowercase has creeped
into the tree since then. Nuke the macros and convert all callsites
to lowercase.

no functional change

If a multithreaded app closes an fd while another thread is blocked in
read/write/accept, then the expectation is that the blocked thread will
exit and the close complete.
Since only one fd is affected, but many fd can refer to the same file,
the close code can only request the fs code unblock with ERESTART.
Fixed for pipes and sockets, ERESTART will only be generated after such
a close - so there should be no change for other programs.
Also rename fo_abort() to fo_restart() (this used to be fo_drain()).
Fixes PR/26567

Rename fo_drain() to fo_abort(), 'drain' is used to mean 'wait for output
do drain' in many places, whereas fo_drain() was called in order to force
blocking read()/write() etc calls to return to userspace so that a close()
call from a different thread can complete.
In the sockets code comment out the broken code in the inner function,
it was being called from compat code.

remove FILE_LOCK and FILE_UNLOCK.

Fix locking as Andy explained. Also fill in uid and gid like sys_pipe did.

Add fileops::fo_drain(), to be called from fd_close() when there is more
than one active reference to a file descriptor. It should dislodge threads
sleeping while holding a reference to the descriptor. Implemented only for
sockets but should be extended to pipes, fifos, etc.

Fixes the case of a multithreaded process doing something like the
following, which would have hung until the process got a signal.

thr0 accept(fd, ...)
thr1 close(fd)

Make module (auto)loading under chroot envrionment actually work:
- NOCHROOT flag must be assigned to different bit from TRYEMULROOT
since the code expected to be executed is in the else clase of
if (flags & TRYEMULROOT).
- Necessary variables aren't set.

branches: 1.162.2;
malloc -> kmem_alloc.

Remove LKMs and switch to the module framework, pass 1.

Proposed on tech-kern@.

branches: 1.160.2; 1.160.4;
Writing 0 bytes on an O_APPEND file should not affect the offset

Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

branches: 1.158.2; 1.158.4;
Don't needlessly acquire v_interlock.

vn_marktext, vn_lock: don't needlessly acquire v_interlock.

branches: 1.156.2; 1.156.4;
Network protocol interrupts can now block on locks, so merge the globals
proclist_mutex and proclist_lock into a single adaptive mutex (proc_lock).
Implications:

- Inspecting process state requires thread context, so signals can no longer
be sent from a hardware interrupt handler. Signal activity must be
deferred to a soft interrupt or kthread.

- As the proc state locking is simplified, it's now safe to take exit()
and wait() out from under kernel_lock.

- The system spends less time at IPL_SCHED, and there is less lock activity.

branches: 1.155.2;
Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

branches: 1.154.6;
Replace struct lock on vnodes with a simpler lock object built on
krwlock_t. This is a step towards removing lockmgr and simplifying
vnode locking. Discussed on tech-kern.

vn_setrecurse: if no lock is exported, use v_lock. Works around issue
described in PR kern/37808. The ideal solution here is to kill vnode
lock recursion, which should not be hard once it is understood what
the two remaining callers of vn_setrecurse() are doing.

Remove VOP_LEASE. Discussed on tech-kern.

vn_write: include f_advice in VOP_WRITE

Use FILE_LOCK() and FILE_UNLOCK()

Merge vmlocking2 to head.

branches: 1.148.4;
Remove cn_lwp from struct componentname. curlwp should be used
from on. The NDINIT() macro no longer takes the lwp parameter and
associates the credentials of the calling thread with the namei
structure.

branches: 1.147.2;
Fscow_run(): add a flag "bool data_valid" to note still valid data.
Buffers run through copy-on-write are marked B_COWDONE. This condition
is valid until the buffer has run through bwrite() and gets cleared from
biodone().

Welcome to 4.99.39.

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

- reduce the number of VOP_ACCESS calls for O_RDWR. for nfs, it reduces
the number of rpcs.
- reduce code duplication.

Use atomics to maintain uvmexp.{anon,exec,file}pages.

Remove the "struct lwp *" argument from all VFS and VOP interfaces.
The general trend is to remove it from all kernel interfaces and
this is a start. In case the calling lwp is desired, curlwp should
be used.

quick consensus on tech-kern

branches: 1.143.4;
Merge from vmlocking:

- Split vnode::v_flag into three fields, depending on field locking.
- simple_lock -> kmutex in a few places.
- Fix some simple locking problems.

Merge file descriptor locking, cwdi locking and cross-call changes
from the vmlocking branch.

Update the file system copy-on-write handler.

- Instead of hooking the handler on the specdev of a mounted file system
hook directly on the `struct mount'.

- Rename from `vn_cow_*' to `fscow_*' and move to `kern/vfs_trans.c'. Use
`mount_*specific' instead of clobbering `struct mount' or `struct specinfo'.

- Replace the hand-made reader/writer lock with a krwlock.

- Keep `vn_cow_*' functions and mark as obsolete.

- Welcome to NetBSD 4.99.32 - `struct specinfo' changed size.

Reviewed by: Jason Thorpe <thorpej@netbsd.org>

branches: 1.140.4; 1.140.6; 1.140.8; 1.140.10;
Retire uvn_attach() - it abuses VXLOCK and its functionality,
setting vnode sizes, is handled elsewhere: file system vnode creation
or spec_open() for regular files or block special files, respectively.

Add a call to VOP_MMAP() to the pagedvn exec path, since the vnode
is being memory mapped.

reviewed by tech-kern & wrstuden

branches: 1.139.2;
- remove pathname_ interface.
- use macros to deal with pathnames in userspace, when veriexec is used.
- reorder the veriexec_ call arguments for consistency.
With help from elad@ finding the last bug.

Change the way that emulations locate files within the emulation root to
avoid having to allocate space in the 'stackgap'
- which is very LWP unfriendly.
The additional code for non-emulation namei() is trivial, the reduction for
the emulations is massive.
The vnode for a processes emulation root is saved in the cwdi structure
during process exec.
If the emulation root the TRYEMULROOT flag are set, namei() will do an initial
search for absolute pathnames in the emulation root, if that fails it will
retry from the normal root.
".." at the emulation root will always go to the real root, even in the middle
of paths and when expanding symlinks.
Absolute symlinks found using absolute paths in the emulation root will be
relative to the emulation root (so /usr/lib/xxx.so -> /lib/xxx.so links
inside the emulation root don't need changing).
If the root of the emulation would be returned (for an emulation lookup), then
the real root is returned instead (matching the behaviour of emul_lookup,
but being a cheap comparison here) so that programs that scan "../.."
looking for the root dircetory don't loop forever.
The target for symbolic links is no longer mangled (it used to get the
CHECK_ALT_xxx() treatment, so could get /emul/xxx prepended).
CHECK_ALT_xxx() are no more. Most of the change is deleting them, and adding
TRYEMULROOT to the flags to NDINIT().
A lot of the emulation system call stubs could now be deleted.

Remove now obsolete vn_start_write() and vn_finished_write() and
corresponding flags.

Revert softdep_trackbufs() to its state before vn_start_write() was added.

Remove from struct mount now unneeded flags IMNT_SUSPEND* and
members mnt_writeopcountupper, mnt_writeopcountlower and mnt_leaf.

Welcome to 4.99.17

Remove calls to now obsolete vn_start_write() and vn_finished_write().

branches: 1.135.2; 1.135.4;
- Make the proclist_lock a mutex. The write:read ratio is unfavourable,
and mutexes are cheaper use than RW locks.
- LOCK_ASSERT -> KASSERT in some places.
- Hold proclist_lock/kernel_lock longer in a couple of places.

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.133.2;
Make fstrans(9) the default helper for file system suspension.
Replaces the now obsolete vn_start_write()/vn_finished_write().

Merge newlock2 to head.

New file system suspension API to replace vn_start_write and vn_finished_write.
The suspension helpers are now put into file system specific operations.
This means every file system not supporting these helpers cannot be suspended
and therefore snapshots are no longer possible.

Implemented for file systems of type ffs.

The new API is enabled on a kernel option NEWVNGATE. This option is
not enabled by default in any kernel config.

Presented and discussed on tech-kern with much input from
Bill Studenmund <wrstuden@netbsd.org> and YAMAMOTO Takashi <yamt@netbsd.org>.

Welcome to 4.99.9 (new vfs op vfs_suspendctl).

Avoid TOCTOU in Veriexec by introducing veriexec_openchk() to enforce
the policy and using a single namei() call in vn_open().

branches: 1.129.2;
Massive restructuring and cleanup of Veriexec, mainly in preparation
for work on some future functionality.

- Veriexec data-structures are no longer exposed.

- Thanks to using proplib for data passing now, the interface
changes further to accomodate that.

Introduce four new functions. First, veriexec_file_add(), to add
a new file to be monitored by Veriexec, to replace both
veriexec_load() and veriexec_hashadd(). veriexec_table_add(), to
replace veriexec_newtable(), will be used to optimize hash table
size (during preload), and finally, veriexec_convert(), to convert
an internal entry to one userland can read.

- Introduce veriexec_unmountchk(), to enforce Veriexec unmount
policy. This cleans up a bit of code in kern/vfs_syscalls.c.

- Rename veriexec_tblfind() with veriexec_table_lookup(), and make
it static. More functions that became static: veriexec_fp_cmp(),
veriexec_fp_calc().

- veriexec_verify() no longer returns the entry as well, but just
sets a boolean indicating whether an entry was found or not.

- veriexec_purge() now takes a struct vnode *.

- veriexec_add_fp_name() was merged into veriexec_add_fp_ops(), that
changed its name to veriexec_fpops_add(). veriexec_find_ops() was
also renamed to veriexec_fpops_lookup().

Also on the fp-ops front, the three function types used to initialize,
update, and finalize a hash context were renamed to
veriexec_fpop_init_t, veriexec_fpop_update_t, and veriexec_fpop_final_t
respectively.

- Introduce a new malloc(9) type, M_VERIEXEC, and use it instead of
M_TEMP, so we can tell exactly how much memory is used by Veriexec.

- And, most importantly, whitespace and indentation nits.

Built successfuly for amd64, i386, sparc, and sparc64. Tested on amd64.

printf() -> log().

Adapt to changes suggested by yamt@ to get rid of __UNCONST() stuff.

While here, don't leak pathbuf on success.

Don't allocate MAXPATHLEN on the stack.

Prompted by and initial diff okay yamt@

add support for O_DIRECT (I/O directly to application memory,
bypassing any kernel caching for file data).

branches: 1.124.2;
Fix typo.

Prevent a veriexec file from being truncated.

branches: 1.122.4;
at the request of elad, as veriexec.h has returned, revert the changes
from 2006-07-25.

mechanically go through and
s,include "veriexec.h",include <sys/verified_exec.h>,
as the former has apparently gone away.

replace magic numbers for strict levels (0-3) with defines.

finally do things properly. veriexec_report() takes flags, not three ints.

some fixes:
- adapt to NVERIEXEC in init_sysctl.c.
- we now need "veriexec.h" for NVERIEXEC.
- "opt_verified_exec.h" -> "opt_veriexec.h", and include it only where
it is needed.

Use the LWP cached credentials where sane.

kill a VOP_GETATTR() we don't need for veriexec.

deprecate the VERIFIED_EXEC option; now we only need the pseudo-device to
enable it. while here, some config file tweaks.

tons of input from cube@ (thanks!) and okay blymn@.

oops, forgot to commit that one. thanks Arnaud Lacombe.

okay, since there was no way to divide this to two commits, here it goes..

introduce fileassoc(9), a kernel interface for associating meta-data with
files using in-kernel memory. this is very similar to what we had in
veriexec till now, only abstracted so it can be used more easily by more
consumers.

this also prompted the redesign of the interface, making it work on vnodes
and mounts and not directly on devices and inodes. internally, we still
use file-id but that's gonna change soon... the interface will remain
consistent.

as a result, veriexec went under some heavy changes to conform to the new
interface. since we no longer use device numbers to identify file-systems,
the veriexec sysctl stuff changed too: kern.veriexec.count.dev_N is now
kern.veriexec.tableN.* where 'N' is NOT the device number but rather a
way to distinguish several mounts.

also worth noting is the plugging of unmount/delete operations
wrt/fileassoc and veriexec.

tons of input from yamt@, wrstuden@, martin@, and christos@.

Limit the size of any kernel buffers allocated by the VOP_READDIR
routines to MAXBSIZE.

branches: 1.111.2;
integrate kauth.

XXX: GCC uninitialized.

Change VOP_FCNTL to take an unlocked vnode. Approved by wrstuden@.

vn_rdwr(): Initialize `mp' to NULL. vn_finished_write() would be called
with uninitialized `mp' if `vp->v_type == VCHR'.

From Coverity CID 2475.

branches: 1.107.2;
remove a wrong assertion.

branches: 1.106.2; 1.106.4;
merge yamt-uio_vmspace branch.

- use vmspace rather than proc or lwp where appropriate.
the latter is more natural to specify an address space.
(and less likely to be abused for random purposes.)
- fix a swdmover race.

vn_read: don't bother to allocate read-ahead context here.
it will be done in uvn_get if necessary.

branches: 1.104.2; 1.104.4;
vn_lock: LK_CANRECURSE is used by layered filesystems. pointed by cube@.

vn_lock: assert that only a limited set of LK_* flags is used.

branches: 1.102.2;
Catch up with ktrace-lwp merge.

While I'm here, stop using cur{lwp,proc}.

merge ktrace-lwp.

merge yamt-readahead branch.

branches: 1.99.2;
vput() -> vrele(). Vnode is already unlocked.
With much help from Pavel Cahyna.

Fixes PR 32005.

copystr and copyinstr return int, not void.

No need for __UNCONST in previous commit; factor out the function call.

Copy the path to a kernel buffer before using it from ndp, as it may be a
pointer to userspace.

uninline vn_start_write and vn_finished_write as they are big enough.

Fix a null vp panic when creating a file at veriexec strict level 3.

defopt verified_exec.

branches: 1.92.2;
- Avoid pollution of struct vnode. Save the fingerprint evaluation status
in the veriexec table entry; the lookups are very cheap now. Suggested
by Chuq.

- Handle non-regular (!VREG) files correctly).

- Remove (no longer needed) FINGERPRINT_NOENTRY.

More veriexec changes:

- Better organize strict level. Now we have 4 levels:
- Level 0, learning mode: Warnings only about anything that might've
resulted in 'access denied' or similar in a higher strict level.

- Level 1, IDS mode:
- Deny access on fingerprint mismatch.
- Deny modification of veriexec tables.

- Level 2, IPS mode:
- All implications of strict level 1.
- Deny write access to monitored files.
- Prevent removal of monitored files.
- Enforce access type - 'direct', 'indirect', or 'file'.

- Level 3, lockdown mode:
- All implications of strict level 2.
- Prevent creation of new files.
- Deny access to non-monitored files.

- Update sysctl(3) man-page with above. (date bumped too :)

- Remove FINGERPRINT_INDIRECT from possible fp_status values; it's no
longer needed.

- Simplify veriexec_removechk() in light of new strict level policies.

- Eliminate use of 'securelevel'; veriexec now behaves according to
its strict level only.

Work according to veriexec strict level, not securelevel. Also, use the
veriexec_report() routine when possible; and when opening a file for writing,
only invalidate the fingerprint - not always the data will be changed.

Use ANSI function decls.

- add const.
- remove unnecessary casts.
- add __UNCONST casts and mark them with XXXUNCONST as necessary.

Rototill of the verified exec functionality.
* We now use hash tables instead of a list to store the in kernel
fingerprints.
* Fingerprint methods handling has been made more flexible, it is now
even simpler to add new methods.
* the loader no longer passes in magic numbers representing the
fingerprint method so veriexecctl is not longer kernel specific.
* fingerprint methods can be tailored out using options in the kernel
config file.
* more fingerprint methods added - rmd160, sha256/384/512
* veriexecctl can now report the fingerprint methods supported by the
running kernel.
* regularised the naming of some portions of veriexec.

branches: 1.86.2;
nuke trailing whitespace

branches: 1.85.2; 1.85.4;
Add the system call and VFS infrastructure for file system extended
attributes.

From FreeBSD.

vn_lock: #if 0 out an assertion for now. (until PR/27021 is fixed)

Cloning cleanup:
1. make fileops const
2. add 2 new negative errno's to `officially' support the cloning hack:
- EDUPFD (used to overload ENODEV)
- EMOVEFD (used to overload ENXIO)
3. Created an fdclone() function to encapsulate the operations needed for
EMOVEFD, and made all cloners use it.
4. Centralize the local noop/badop fileops functions to:
fnullop_fcntl, fnullop_poll, fnullop_kqfilter, fbadop_stat

Fix another stupid typo.

Add support for FIONWRITE and FIONSPACE ioctls. FIONWRITE reports
the number of bytes in the send queue, and FIONSPACE reports the
number of free bytes in the send queue. These ioctls permit applications
to monitor file descriptor transmission dynamics.

In examining prior art, FIONWRITE exists with the semantics given
here. FIONSPACE is provided so that programs may easily determine how
much space is left in the send queue; they do not need to know the
send queue size.

The fact that a write may block even if there is enough space in the
send queue for it is noted in the documentation.

FIONWRITE functionality may be used to implement TIOCOUTQ for Linux
emulation - Linux extended this ioctl to sockets, even though they are
not ttys.

vn_lock: add an assertion about usecount.

vn_lock: don't pass LK_RETRY to VOP_LOCK.

Add ffs internal snapshots. Written by Marshall Kirk McKusick for FreeBSD.

- Not enabled by default. Needs kernel option FFS_SNAPSHOT.
- Change parameters of ffs_blkfree.
- Let the copy-on-write functions return an error so spec_strategy
may fail if the copy-on-write fails.
- Change genfs_*lock*() to use vp->v_vnlock instead of &vp->v_lock.
- Add flag B_METAONLY to VOP_BALLOC to return indirect block buffer.
- Add a function ffs_checkfreefile needed for snapshot creation.
- Add special handling of snapshot files:
Snapshots may not be opened for writing and the attributes are read-only.
Use the mtime as the time this snapshot was taken.
Deny mtime updates for snapshot files.
- Add function transferlockers to transfer any waiting processes from
one lock to another.
- Add vfsop VFS_SNAPSHOT to take a snapshot and make it accessible through
a vnode.
- Add snapshot support to ls, fsck_ffs and dump.

Welcome to 2.0F.

Approved by: Jason R. Thorpe <thorpej@netbsd.org>

branches: 1.77.2; 1.77.4; 1.77.6;
Add a generic copy-on-write hook to add/remove functions that will be
called with every buffer written through spec_strategy().

Used by fss(4). Future file-system-internal snapshots will need them too.

Welcome to 1.6ZK

Approved by: Jason R. Thorpe <thorpej@netbsd.org>

Allow vfs_write_suspend() to wait if the file system is already
suspending.

Move vfs_write_suspend() and vfs_write_resume() from kern/vfs_vnops.c
to kern/vfs_subr.c.

Change vnode write gating in ufs/ffs/ffs_softdep.c (from FreeBSD).

When vnodes are throttled in softdep_trackbufs() check for
file system suspension every 10 msecs to avoid a deadlock.

Add the gating of system calls that cause modifications to the underlying
file system.
The function vfs_write_suspend stops all new write operations to a file
system, allows any file system modifying system calls already in progress
to complete, then sync's the file system to disk and returns. The
function vfs_write_resume allows the suspended write operations to
complete.

From FreeBSD with slight modifications.

Approved by: Frank van der Linden <fvdl@netbsd.org>

fix O_NOFOLLOW for non-O_CREAT case.

Reviewed by: christos@ (some time ago)

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

branches: 1.72.2;
Back out the lwp/ktrace changes. They contained a lot of colateral damage,
and need to be examined and discussed more.

Adjust to ktrace/lwp changes.

Pass lwp pointers throughtout the kernel, as required, so that the lwpid can
be inserted into ktrace records. The general change has been to replace
"struct proc *" with "struct lwp *" in various function prototypes, pass
the lwp through and use l_proc to get the process pointer when needed.

Bump the kernel rev up to 1.6V

Copy birthtime in vn_stat.

Use 'void *' instead of 'caddr_t' in prototypes of VOP_IOCTL, VOP_FCNTL
and VOP_ADVLOCK, delete casts from callers (and some to copyin/out).

Change 'data' argument to fo_ioctl and fo_fcntl from 'caddr_t' to 'void *'.
Avoids a lot of casting and removes the need for some line breaks.
Removed a load of (caddr_t) casts from calls to copyin/copyout as well.
(approved by christos - he has a plan to remove caddr_t...)

make it possible for UNION fs to be loaded via LKM - instead of
having some #ifdef UNION code in vfs_vnops.c, introduce variable
'vn_union_readdir_hook' which is set to address of appropriate
vn_readdir() hook by union filesystem when it's loaded & mounted

move union filesystem code from sys/miscfs/union to sys/fs/union

only pull in/declare veriexec related stuff with VERIFIED_EXEC

Allow filesystems' VOP_IOCTL to catch ioctl calls on directories and
regular files. Approved thorpej, fvdl.

Check for (and deny) negative values passed to FIOGETBMAP.

Bump daddr_t to 64 bits. Replace it with int32_t in all places where
it was used on-disk, so that on-disk formats remain the same.
Remove ufs_daddr_t and ufs_lbn_t for the time being.

Provide a ioctl called FIOGETBMAP (there are some who call
it...FIBMAP) that translates a logical block number to a physical
block number from the underlying device. Via VOP_BMAP().

s/NOSYMLINK/O_NOFOLLOW/

Added support for fingerprinted executables aka verified exec

merge kqueue branch into -current

kqueue provides a stateful and efficient event notification framework
currently supported events include socket, file, directory, fifo,
pipe, tty and device changes, and monitoring of processes and signals

kqueue is supported by all writable filesystems in NetBSD tree
(with exception of Coda) and all device drivers supporting poll(2)

based on work done by Jonathan Lemon for FreeBSD
initial NetBSD port done by Luke Mewburn and Jason Thorpe

vn_stat() can now take a struct vnode * for consistency. Hide away
the opaque file descriptor operations.

count executable image pages as executable for vm-usage purposes.
also, always do the VTEXT vs. v_writecount mutual exclusion
(which we previously skipped if the text or data segment was empty).

branches: 1.54.6;
Convert ioctl code to use EPASSTHROUGH instead of -1 or ENOTTY for
indicating an unhandled "command". ERESTART is -1, which can lead to
confusion. ERESTART has been moved to -3 and EPASSTHROUGH has been
placed at -4. No ioctl code should now return -1 anywhere. The
ioctl() system call is now properly restartable.

replace "vnode" and "vtext" with "file" and "exec" in uvmexp field names.

add RCSIDs

- Add a new vnode flag VEXECMAP, which indicates that a vnode has
executable mappings. Stop overloading VTEXT for this purpose (VTEXT
also has another meaning).
- Rename vn_marktext() to vn_markexec(), and use it when executable
mappings of a vnode are established.
- In places where we want to set VTEXT, set it in v_flag directly, rather
than making a function call to do this (it no longer makes sense to
use a function call, since we no longer overload VTEXT with VEXECMAP's
meaning).

VEXECMAP suggested by Chuq Silvers.

branches: 1.50.2;
use shared locks instead of exclusive for VOP_READ() and VOP_READDIR().

a whole bunch of changes to improve performance and robustness under load:

- remove special treatment of pager_map mappings in pmaps. this is
required now, since I've removed the globals that expose the address range.
pager_map now uses pmap_kenter_pa() instead of pmap_enter(), so there's
no longer any need to special-case it.
- eliminate struct uvm_vnode by moving its fields into struct vnode.
- rewrite the pageout path. the pager is now responsible for handling the
high-level requests instead of only getting control after a bunch of work
has already been done on its behalf. this will allow us to UBCify LFS,
which needs tighter control over its pages than other filesystems do.
writing a page to disk no longer requires making it read-only, which
allows us to write wired pages without causing all kinds of havoc.
- use a new PG_PAGEOUT flag to indicate that a page should be freed
on behalf of the pagedaemon when it's unlocked. this flag is very similar
to PG_RELEASED, but unlike PG_RELEASED, PG_PAGEOUT can be cleared if the
pageout fails due to eg. an indirect-block buffer being locked.
this allows us to remove the "version" field from struct vm_page,
and together with shrinking "loan_count" from 32 bits to 16,
struct vm_page is now 4 bytes smaller.
- no longer use PG_RELEASED for swap-backed pages. if the page is busy
because it's being paged out, we can't release the swap slot to be
reallocated until that write is complete, but unlike with vnodes we
don't keep a count of in-progress writes so there's no good way to
know when the write is done. instead, when we need to free a busy
swap-backed page, just sleep until we can get it busy ourselves.
- implement a fast-path for extending writes which allows us to avoid
zeroing new pages. this substantially reduces cpu usage.
- encapsulate the data used by the genfs code in a struct genfs_node,
which must be the first element of the filesystem-specific vnode data
for filesystems which use genfs_{get,put}pages().
- eliminate many of the UVM pagerops, since they aren't needed anymore
now that the pager "put" operation is a higher-level operation.
- enhance the genfs code to allow NFS to use the genfs_{get,put}pages
instead of a modified copy.
- clean up struct vnode by removing all the fields that used to be used by
the vfs_cluster.c code (which we don't use anymore with UBC).
- remove kmem_object and mb_object since they were useless.
instead of allocating pages to these objects, we now just allocate
pages with no object. such pages are mapped in the kernel until they
are freed, so we can use the mapping to find the page to free it.
this allows us to remove splvm() protection in several places.

The sum of all these changes improves write throughput on my
decstation 5000/200 to within 1% of the rate of NetBSD 1.5
and reduces the elapsed time for "make release" of a NetBSD 1.5
source tree on my 128MB pc to 10% less than a 1.5 kernel took.

branches: 1.48.2; 1.48.4;
Change the first arg to fileops fo_stat routine to struct file *, adjust
callers and appropriate routines to cope. This makes fo_stat more
consistent with rest of fileops routines and also makes the fo_stat
match FreeBSD as an added bonus.
Discussed with Luke Mewburn on tech-kern@.

Add new 'stat' fileop and call the stat function via f_ops rather
than directly.
For compat syscalls, also add necessary FILE_USE()/FILE_UNUSE().
Now that soo_stat() gets a proc arg, pass it on to usrreq function.

add UBC memory-usage balancing. we track the number of pages in use for
each of the basic types (anonymous data, executable image, cached files)
and prevent the pagedaemon from reusing a given page if that would reduce
the count of that type of page below a sysctl-setable minimum threshold.
the thresholds are controlled via three new sysctl tunables:
vm.anonmin, vm.vnodemin, and vm.vtextmin. these tunables are the
percentages of pageable memory reserved for each usage, and we do not allow
the sum of the minimums to be more than 95% so that there's always some
memory that can be reused.

branches: 1.45.2;
Initial integration of the Unified Buffer Cache project.

Use ltsleep(...,PNORELOCK..) instead of simple_unlock()/tsleep()

remove include of <vm/vm.h>

add a new function vn_marktext() for exec code to let others know
that the vnode is now being used as process text.

Get rid of register declarations.

Delete redundant decl of union_vnodeop_p, it's in <miscfs/union/union.h>.

Fixes to the softdep code from Ethan Solomita <ethan@geocast.com>.
* Fix buffer ordering when it has dependencies.
* Alleviate memory problems.
* Deal with some recursive vnode locks (sigh).
* Fix other bugs.

branches: 1.38.2;
Add a new flag, used by vn_open() which prevent symlinks from being followed
at open time. Use this to prevent coredump to follow symlinks when the
kernel opens/creates the file.

Add support for fcntl(2) to generate VOP_FCNTL calls. Any fcntl
call with F_FSCTL set and F_SETFL calls generate calls to a new
fileop fo_fcntl. Add genfs_fcntl() and soo_fcntl() which return 0
for F_SETFL and EOPNOTSUPP otherwise. Have all leaf filesystems
use genfs_fcntl().

Reviewed by: thorpej
Tested by: wrstuden

branches: 1.36.2; 1.36.4;
Previous change to vn_lock() was bogus. If we got EDEADLK, it was from
lockmgr(), and it already unlocked v_interlock. So, just return in this case.

The mode for a node is a mode_t in both struct stat and struct vattr -
don't use a u_short for intermediate storage in vn_stat.

Prevent deadlock cited in PR4629 from crashing the system. (copyout
and system call now just return EFAULT). A complete fix will
presumably have to wait for UBC and/or for vnode locking protocols to
be revamped to allow use of shared locks.

completely remove Mach VM support. all that is left is the all the
header files as UVM still uses (most of) these.

Modify VOP_CLOSE vnode op to always take a locked vnode. Change vn_close
to pass down a locked node. Modify union_copyup() to call VOP_CLOSE
locked nodes.

Also fix a bug in union_copyup() where a lock on the lower vnode would
only be released if VOP_OPEN didn't fail.

branches: 1.31.2;
Implement support for IEEE Std 1003.1b-1993 syncronous I/O:
* if synchronized I/O file integrity completion of read operations was
requested, set IO_SYNC in the ioflag passed to the read vnode operator.
* if synchronized I/O data integrity completion of write operations was
requested, set IO_DSYNC in the ioflag passed to the write vnode operator.

Change the "aresid" argument of vn_rdwr() from an int * to a size_t *,
to match the new uio_resid type.

Add two additional arguments to the fileops read and write calls, a
pointer to the offset to use, and a flags word. Define a flag that
specifies whether or not to update the offset passed by reference.

Merge with Lite2 + local changes

Include the UNION option header.

- add defopt's for UVM, UVMHIST and PMAP_NEW.
- remove unnecessary UVMHIST_DECL's.

initial import of the new virtual memory system, UVM, into -current.

UVM was written by chuck cranor <chuck@maria.wustl.edu>, with some
minor portions derived from the old Mach code. i provided some help
getting swap and paging working, and other bug fixes/ideas. chuck
silvers <chuq@chuq.com> also provided some other fixes.

this is the rest of the MI portion changes.

this will be KNF'd shortly. :-)

Grab a fix from 4.4BSD-Lite2: open(2) with O_FSYNC and MNT_SYNCHRONOUS
had not effect. Fix: check for either of these flags in vn_write(),
and pass IO_SYNC down if they're set.

branches: 1.23.2;
Add vn_readdir function for use in both the old getdirentries and
the new getdents(). Add getdents().

branches: 1.22.4;
Do not return generation counts to the user.

Implement poll(2).

First pass at prototyping

Remove gratuitous extra indirections.

Remove extra arg to vn_open().

LEASE_CHECK -> VOP_LEASE

added extra argument in vn_open and VOP_OPEN to allow cloning devices

be more careful with types, also pull in headers where necessary.

Fix space change in last commit.

from Kirk McKusick: release old ctty if acquiring a new one.
also: prettiness police!

branches: 1.12.2;
New RCS ID's, take two. they're more aesthecially pleasant, and use 'NetBSD'

Update to 4.4-Lite fs code.

copyright foo

some prototype cleanup, eliminate/replace bogus types (e.g. quad and
u_quad) -> use better types (e.g. quad_t & u_quad_t in inodes),
some cleanup.

FIONREAD returns int not off_t. (ssize_t prefered, but standards may
dictate otherwise)

kill two wrong 'case's

Canonicalize all #includes.

branches: 1.5.2;
Changes to VFS readdir semantics
NFS changes for better cookie support
ISOFS changes for better Rockridge support and support for generation numbers

Support added for proc filesystem.

add include of select.h if necessary for protos, or delete if extraneous

make kernel select interface be one-stop shopping & clean it all up.

branches: 1.1.1;
Initial revision

file(9): New fo_posix_fadvise operation.

XXX kernel revbump -- changes struct fileops API and ABI

file(9): New fo_fpathconf operation.

XXX kernel revbump -- struct fileops API and ABI change

file(9): New fo_advlock operation.

This moves the vnode-specific logic from sys_descrip.c into
vfs_vnode.c, like we did for fo_seek.

XXX kernel revbump -- struct fileops API and ABI change

readdir(2), lseek(2): Fix races in access to struct file::f_offset.

For non-directory vnodes:
- reading f_offset requires a shared or exclusive vnode lock
- writing f_offset requires an exclusive vnode lock

For directory vnodes, access (read or write) requires either:
- a shared vnode lock AND f_lock, or
- an exclusive vnode lock.

This way, two files for the same underlying directory vnode can still
do VOP_READDIR in parallel, but if two readdir(2) or lseek(2) calls
run in parallel on the same file, the load and store of f_offset is
atomic (otherwise, e.g., on 32-bit systems it might be torn and lead
to corrupt offsets).

There is still a potential problem: the _whole transaction_ of
readdir(2) may not be atomic. For example, if thread A and thread B
read n bytes of directory content, thread A might get bytes [0,n) and
thread B might get bytes [n,2n) but f_offset might end up at n
instead of 2n once both operations complete. (However, f_offset
wouldn't be some corrupt garbled number like n & 0xffffffff00000000.)
Fixing this would require either:
(a) using an exclusive vnode lock in vn_readdir,
(b) introducing a new lock that serializes vn_readdir on the same
file (but ont necessarily the same vnode), or
(c) proving it is safe to hold f_lock across VOP_READDIR, VOP_SEEK,
and VOP_GETATTR.

vn_open(9): Add assertion that vp is locked on return.

Null out vp internally out of paranoia so we'll crash in evaluating
the assertion if we ever reach it via one of the vput paths.

vn_open(9): Clarify that this returns a locked vnode.

Comment only, no functional change intended.

vnodeops(9): Take exclusive lock in read/seek for f_offset update.

Otherwise concurrent readers/seekers might clobber it.

Make kqueue event status for vnodes shareable, and for stacked file systems
like nullfs, make the upper vnode share that status with the lower vnode.

And, lo, NetBSD 9.99.99.

Fixes PR kern/56713.

kern: Work around spurious -Wtype-limits warnings.

This useless garbage warning is apparently designed to make it
painful to write portable safe arithmetic and I think we ought to
just disable it.

kern/vfs_vnops.c: Fix missing semicolon in previous.

Neglected to build and amend commit, oops.

kern/vfs_vnops.c: Sprinkle KNF.

No functional change intended.

mmap(2): Avoid overflow in overflow check in vn_mmap.

uvm(9): fo_mmap caller guarantees positive size.

No functional change intended, just sprinkling assertions to make it
clearer.

fix various small typos, mainly in comments.

It is impossible for VOP_LOCK() to return ENOENT with LK_RETRY flag.
Remove the second call to VOP_LOCK().

Enable assertion "vrefcnt(vp) > 0" and assert all possible errors
for all LK_RETRY/LK_NOWAIT combinations.

Lock vnode across VOP_OPEN.

vfs(9): Avoid arithmetic overflow in vn_seek.

Reported-by: syzbot+b9f9a02148a40675c38a@syzkaller.appspotmail.com

Overhaul of the EVFILT_VNODE kevent(2) filter:

- Centralize vnode kevent handling in the VOP_*() wrappers, rather than
forcing each individual file system to deal with it (except VOP_RENAME(),
because VOP_RENAME() is a mess and we currently have 2 different ways
of handling it; at least it's reasonably well-centralized in the "new"
way).
- Add support for NOTE_OPEN, NOTE_CLOSE, NOTE_CLOSE_WRITE, and NOTE_READ,
compatible with the same events in FreeBSD.
- Track which kevent notifications clients are interested in receiving
to avoid doing work for events no one cares about (avoiding, e.g.
taking locks and traversing the klist to send a NOTE_WRITE when
someone is merely watching for a file to be deleted, for example).

In support of the above:

- Add support in vnode_if.sh for specifying PRE- and POST-op handlers,
to be invoked before and after vop_pre() and vop_post(), respectively.
Basic idea from FreeBSD, but implemented differently.
- Add support in vnode_if.sh for specifying CONTEXT fields in the
vop_*_args structures. These context fields are used to convey information
between the file system VOP function and the VOP wrapper, but do not
occupy an argument slot in the VOP_*() call itself. These context fields
are initialized and subsequently interpreted by PRE- and POST-op handlers.
- Version VOP_REMOVE(), uses the a context field for the file system to report
back the resulting link count of the target vnode. Return this in tmpfs,
udf, nfs, chfs, ext2fs, lfs, and ufs.

NetBSD 9.99.92.

sys/kern: Avoid fp->f_offset without the object (here, vnode) lock.

sys/kern: Allow custom fileops to specify fo_seek method.

Previously only vnodes allowed lseek/pread[v]/pwrite[v], which meant
converting a regular device to a cloning device doesn't always work.

Semantics is:

(*fp->f_ops->fo_seek)(fp, delta, whence, newoffp, flags)

1. Compute a new offset according to whence + delta -- that is, if
whence is SEEK_CUR, add delta to fp->f_offset; if whence is
SEEK_END, add delta to end of file; if whence is SEEK_CUR, use delta
as is.

2. If newoffp is nonnull, return the new offset in *newoffp.

3. If flags & FOF_UPDATE_OFFSET, set fp->f_offset to the new offset.

Access to fp->f_offset, and *newoffp if newoffp = &fp->f_offset, must
happen under the object lock (e.g., vnode lock), in order to
synchronize fp->f_offset reads and writes.

This change has the side effect that every call to VOP_SEEK happens
under the vnode lock now, when previously it didn't. However, from a
review of all the VOP_SEEK implementations, it does not appear that
any file system even examines the vnode, let alone locks it. So I
think this is safe -- and essentially the only reasonable way to do
things, given that it is used to validate a change from oldoff to
newoff, and oldoff becomes stale the moment we unlock the vnode.

No kernel bump because this reuses a spare entry in struct fileops,
and it is safe for the entry to be null, so all existing fileops will
continue to work as before (rejecting seek).

Fix confusion arising from whether FOLLOW or NOFOLLOW is 0.

In vn_open, don't set and then throw away FOLLOW, and clarify the
comment about requesting FOLLOW/NOFOLLOW behavior.

Related to PR 56316.

gcc (with some options) eroneously claims we would use "vp" uninitialized,
so initialize it as NULL.

don't clear the error before we use it to determine if we are moving or duping.

Improve Christos's vn_open fix.

- assert about api misuse up front (suggested by riastradh)
- restore the behavior of returning EOPNOTSUPP if ret_fd is NULL and we
get a fd back (otherwise things like ktruss -o /dev/stderr panic)
- clear error to 0 for the EDUPFD and EMOVEFD cases so opening a
cloner succeeds

PR/56286: Martin Husemann: Fix NULL deref on kmod load.
- No need to set ret_domove and ret_fd in the regular case, they are meaningless
- KASSERT instead of setting errno and then doing the NULL deref.

Add containment for the cloning devices hack in vn_open.

Cloning devices (and also things like /dev/stderr) work by allocating
a struct file, stuffing it in the file table (which is a layer
violation), stuffing the file descriptor number for it in a magic
field of struct lwp (which is gross), and then "failing" with one of
two magic errnos, EDUPFD or EMOVEFD.

Before this commit, all callers of vn_open in the kernel (there are
quite a few) were expected to check for these errors and handle the
situation. Needless to say, none of them except for open() itself did,
resulting in internal negative errnos being returned to userspace.

This hack is fairly deeply rooted and cannot be eliminated all at
once. This commit adds logic to handle the magic errnos inside
vn_open; now on success vn_open returns either a vnode or an integer
file descriptor, along with a flag that says whether the underlying
code requested EDUPFD or EMOVEFD. Callers not prepared to cope with
file descriptors can pass NULL for the extra return values, in which
case if a file descriptor would be produced vn_open fails with
EOPNOTSUPP.

Since I'm rearranging vn_open's signature anyway, stop exposing struct
nameidata. Instead, take three arguments: an optional vnode to use as
the starting point (like openat()), the path, and additional namei
flags to use, restricted to NOCHROOT and TRYEMULROOT. (Other namei
behavior, e.g. NOFOLLOW, can be requested via the open flags.)

This change requires a kernel bump. Ride the one an hour ago.
(That was supposed to be coordinated; did not intend to let an hour
slip by. My fault.)

Add a new namei flag NONEXCLHACK for open with O_CREAT and not O_EXCL.

This case needs to be distinguished from the other CREATE operations
because it is supposed to successfully return (and open) the target if
it exists. In the case where that target is the root, or a mount
point, such that there's no parent dir, "real" CREATE operations fail,
but O_CREAT without O_EXCL needs to succeed.

So (a) add the flag, (b) test for it in namei in the situation
described above, (c) set it in open under the appropriate
circumstances, and (d) because this can result in namei returning
ni_dvp of NULL, cope with that case.

Should get into -9 and maybe even -8, because it was prompted by
issues with 3rd-party code. The use of a flag (vs. adding an
additional nameiop, which would be more appropriate) was deliberate to
make the patch small and noninvasive.

branches: 1.214.4;
Lock the vnode while calling VOP_BMAP() for FIOGETBMAP.

Reported-by: syzbot+cfa1b773be7337250428@syzkaller.appspotmail.com

branches: 1.213.2;
Counter tweaks:

- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
each kind of page can be summed to get the totals.

- Track the number of free pages with a counter so that it's one less thing
for the allocator to do, which opens up further options there.

- Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot.
For the cheap option, give cpu_count_sync() a boolean parameter indicating
that a cached value is okay, and rate limit the updates for cached values
to hz.

Move proc_lock into the data segment. It was dynamically allocated because
at the time we had mutex_obj_alloc() but not __cacheline_aligned.

Replace most uses of vp->v_usecount with a call to vrefcnt(vp), a function
that hides the details and does atomic_load_relaxed(). Signature matches
FreeBSD.

Oops missed one more NULL -> NOCRED

delete debugging printf.

Pass NOCRED instead of NULL for credentials. These routines are supposed
to be accessing system ACL's on behalf of the kernel. This code appears
to be copied from FreeBSD, but there it works because in FreeBSD NOCRED
is 0, ours is -1. I guess nobody has used system extended attributes on
NetBSD yet :-)

branches: 1.207.4;
Tighten up the locking around vp->v_iflag a little more after the recent
split of vmobjlock & v_interlock.

UVM locking changes, proposed on tech-kern:

- Change the lock on uvm_object, vm_amap and vm_anon to be a RW lock.
- Break v_interlock and vmobjlock apart. v_interlock remains a mutex.
- Do partial PV list locking in the x86 pmap. Others to follow later.

- Shuffle some items around in struct lwp to save space. Remove an unused
item or two.

- For lockstat, get a useful callsite for vnode locks (caller to vn_lock()).

branches: 1.204.2;
- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

Minor vnode locking changes:

- Stop using atomics to maniupulate v_usecount. It was a mistake to begin
with. It doesn't work as intended unless the XLOCK bit is incorporated in
v_usecount and we don't have that any more. When I introduced this 10+
years ago it was to reduce pressure on v_interlock but it doesn't do that,
it just makes stuff disappear from lockstat output and introduces problems
elsewhere. We could do atomic usecounts on vnodes but there has to be a
well thought out scheme.

- Resurrect LK_UPGRADE/LK_DOWNGRADE which will be needed to work effectively
when there is increased use of shared locks on vnodes.

- Allocate the vnode lock using rw_obj_alloc() to reduce false sharing of
struct vnode.

- Put all of the LRU lists into a single cache line, and do not requeue a
vnode if it's already on the correct list and was requeued recently (less
than a second ago).

Kernel build before and after:

119.63s real 1453.16s user 2742.57s system
115.29s real 1401.52s user 2690.94s system

Add functions to open devices by device number or path.

set VEXEC if FEXEC is set.

branches: 1.200.4;
Change vn_openchk() to fail VNON and VBAD with error ENXIO.

Reported-by: syzbot+d66b1be08516a4d2d2b2@syzkaller.appspotmail.com
Reported-by: syzbot+c5eaef5a8af535c3b217@syzkaller.appspotmail.com

s/fall into .../FALLTHROUGH/

Rename min/max -> uimin/uimax for better honesty.

These functions are defined on unsigned int. The generic name
min/max should not silently truncate to 32 bits on 64-bit systems.
This is purely a name change -- no functional change intended.

HOWEVER! Some subsystems have

#define min(a, b) ((a) < (b) ? (a) : (b))
#define max(a, b) ((a) > (b) ? (a) : (b))

even though our standard name for that is MIN/MAX. Although these
may invite multiple evaluation bugs, these do _not_ cause integer
truncation.

To avoid `fixing' these cases, I first changed the name in libkern,
and then compile-tested every file where min/max occurred in order to
confirm that it failed -- and thus confirm that nothing shadowed
min/max -- before changing it.

I have left a handful of bootloaders that are too annoying to
compile-test, and some dead code:

cobalt ews4800mips hp300 hppa ia64 luna68k vax
acorn32/if_ie.c (not included in any kernels)
macppc/if_gm.c (superseded by gem(4))

It should be easy to fix the fallout once identified -- this way of
doing things fails safe, and the goal here, after all, is to _avoid_
silent integer truncations, not introduce them.

Maybe one day we can reintroduce min/max as type-generic things that
never silently truncate. But we should avoid doing that for a while,
so that existing code has a chance to be detected by the compiler for
conversion to uimin/uimax without changing the semantics until we can
properly audit it all. (Who knows, maybe in some cases integer
truncation is actually intended!)

branches: 1.197.2; 1.197.4;
add fo_name so we can identify the fileops in a simple way.

Add O_REGULAR to enforce opening of only regular files
(like we have O_DIRECTORY for directories).
This is better than open(, O_NONBLOCK), fstat()+S_ISREG() because opening
devices can have side effects.

branches: 1.195.6;
Lock the vnode before changing its writecount.

Must always lock the parent -> lock the child -> unlock the parent.

branches: 1.193.2; 1.193.4;
Remove useless semicolon reported by Henning Petersen in PR#49634.

add a new "fo_mmap" fileops method to allow use of arbitrary uvm_objects for
mappings of file objects. move vnode-specific details of mmap()ing a vnode
from uvm_mmap() to the new vnode-specific vn_mmap(). add new uvm_mmap_dev()
and uvm_mmap_anon() convenience functions for mapping character devices
and anonymous memory, and replace all other calls to uvm_mmap() with those.
use the new fileop in drm2 so that libdrm can use mmap() to map things
like on other platforms (instead of the ioctl that we have used so far).

branches: 1.191.2;
Try not to use f_data, use f_{vnode,socket,pipe,mqueue,kqueue,ksem} to get
a correctly typed pointer.

branches: 1.190.2;
Fix a NULL pointer dereference after a loooong discussion with dholland@,
hannken@, blymn@ and martin@.

This bug would panic the system when veriexec is set to the VERIEXEC_LOCKDOWN
mode (only settable from root).

branches: 1.189.2;
The current implementation of vn_lock() is racy. Modification of
the vnode operations vector for active vnodes is unsafe because it
is not known whether deadfs or the original file system will be
called.

- Pass down LK_RETRY to the lock operation (hint for deadfs only).

- Change deadfs lock operation to return ENOENT if LK_RETRY is unset.

- Change all other lock operations to check for dead vnode once
the vnode is locked and unlock and return ENOENT in this case.

With these changes in place vnode lock operations will never succeed
after vclean() has marked the vnode as VI_XLOCK and before vclean()
has changed the operations vector.

Adresses PR kern/37706 (Forced unmount of file systems is unsafe)

Discussed on tech-kern.

Welcome to 6.99.33

Change vnode operations create, mknod, mkdir and symlink to return
the resulting vnode *vpp unlocked.

Discussed on tech-kern@

Welcome to 6.99.30

Change vnode operations create, mknod, mkdir and symlink to keep the
directory node dvp locked on return.

Discussed on tech-kern@

Welcome to 6.99.29

branches: 1.186.2;
Bring back Manuel Bouyers patch to resolve races between vget() and vrelel()
resulting in vget() returning dead vnodes.
It is impossible to resolve these races in vn_lock().

Needs pullup to NetBSD-6.

branches: 1.185.2;
don't truncate size_t to int

Fix vn_lock() to return an invalid (dead, clean) vnode
only if the caller requested it by setting LK_RETRY.

Should fix PR #46221: Kernel panic in NFS server code

branches: 1.183.2; 1.183.6; 1.183.8;
Change the vnode locking protocol of VOP_GETATTR() to request at least
a shared lock. Make all calls outside of file systems respect it.

The calls from file systems need review.

No objections from tech-kern.

vn_close: add an assertion

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.180.6;
Introduce struct pathbuf. This is an abstraction to hold a pathname
and the metadata required to interpret it. Callers of namei must now
create a pathbuf and pass it to NDINIT (instead of a string and a
uio_seg), then destroy the pathbuf after the namei session is
complete.

Update all namei call sites accordingly. Add a pathbuf(9) man page and
update namei(9).

The pathbuf interface also now appears in a couple of related
additional places that were passing string/uio_seg pairs that were
later fed into NDINIT. Update other call sites accordingly.

Zero entire stat structure before filling in contents to avoid
leaking kernel memory -- the elements are no longer packed now that
dev_t is 64bit.

from pgoyette

implement O_DIRECTORY as standardized in POSIX-2008,
for both native and linux emulations.
this fixes the rest of PR 43695.

I'm not even going to describe this change. I'll just say that
churn creates interesting code.

Fixes open(O_CREAT|O_TRUNC) on at least tmpfs and nfs to not fail
with ENOENT due to a racy removal of the newly created file.

Caught, as most bugs these days are, by a test run.

Modify vn_lock():
- Take v_interlock before examining v_iflag
- Must always be called without v_interlock taken,
LK_INTERLOCK flag is no longer allowed.

Don't leak kernel stack into userspace.

Clean up vnode lock operations pass 2:

VOP_UNLOCK(vp, flags) -> VOP_UNLOCK(vp): Remove the unneeded flags argument.

Welcome to 5.99.32.

Discussed on tech-kern.

Remove the concept of recursive vnode locks by eliminating
vn_setrecurse(), vn_restorerecurse() and LK_CANRECURSE.
Welcome to 5.99.31

Discussed on tech-kern.

Change layered file systems to always pass the locking VOP's down to the
leaf file system. Remove now unused member v_vnlock from struct vnode.
Welcome to 5.99.30

Discussed on tech-kern.

Enforce RLIMIT_FSIZE before VOP_WRITE. This adds support to file
system drivers where it was missing from and fixes one buggy
implementation. The arguably weird semantics of the check are
maintained (v_size vs. va_bytes, overwrite).

Stop exposing fifofs internals and leave only fifo_vnodeop_p visible.

branches: 1.169.2; 1.169.4;
The VATTR_NULL/VREF/VHOLD/HOLDRELE() macros lost their will to live
years ago when the kernel was modified to not alter ABI based on
DIAGNOSTIC, and now just call the respective function interfaces
(in lowercase). Plenty of mix'n match upper/lowercase has creeped
into the tree since then. Nuke the macros and convert all callsites
to lowercase.

no functional change

If a multithreaded app closes an fd while another thread is blocked in
read/write/accept, then the expectation is that the blocked thread will
exit and the close complete.
Since only one fd is affected, but many fd can refer to the same file,
the close code can only request the fs code unblock with ERESTART.
Fixed for pipes and sockets, ERESTART will only be generated after such
a close - so there should be no change for other programs.
Also rename fo_abort() to fo_restart() (this used to be fo_drain()).
Fixes PR/26567

Rename fo_drain() to fo_abort(), 'drain' is used to mean 'wait for output
do drain' in many places, whereas fo_drain() was called in order to force
blocking read()/write() etc calls to return to userspace so that a close()
call from a different thread can complete.
In the sockets code comment out the broken code in the inner function,
it was being called from compat code.

remove FILE_LOCK and FILE_UNLOCK.

Fix locking as Andy explained. Also fill in uid and gid like sys_pipe did.

Add fileops::fo_drain(), to be called from fd_close() when there is more
than one active reference to a file descriptor. It should dislodge threads
sleeping while holding a reference to the descriptor. Implemented only for
sockets but should be extended to pipes, fifos, etc.

Fixes the case of a multithreaded process doing something like the
following, which would have hung until the process got a signal.

thr0 accept(fd, ...)
thr1 close(fd)

Make module (auto)loading under chroot envrionment actually work:
- NOCHROOT flag must be assigned to different bit from TRYEMULROOT
since the code expected to be executed is in the else clase of
if (flags & TRYEMULROOT).
- Necessary variables aren't set.

branches: 1.162.2;
malloc -> kmem_alloc.

Remove LKMs and switch to the module framework, pass 1.

Proposed on tech-kern@.

branches: 1.160.2; 1.160.4;
Writing 0 bytes on an O_APPEND file should not affect the offset

Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

branches: 1.158.2; 1.158.4;
Don't needlessly acquire v_interlock.

vn_marktext, vn_lock: don't needlessly acquire v_interlock.

branches: 1.156.2; 1.156.4;
Network protocol interrupts can now block on locks, so merge the globals
proclist_mutex and proclist_lock into a single adaptive mutex (proc_lock).
Implications:

- Inspecting process state requires thread context, so signals can no longer
be sent from a hardware interrupt handler. Signal activity must be
deferred to a soft interrupt or kthread.

- As the proc state locking is simplified, it's now safe to take exit()
and wait() out from under kernel_lock.

- The system spends less time at IPL_SCHED, and there is less lock activity.

branches: 1.155.2;
Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

branches: 1.154.6;
Replace struct lock on vnodes with a simpler lock object built on
krwlock_t. This is a step towards removing lockmgr and simplifying
vnode locking. Discussed on tech-kern.

vn_setrecurse: if no lock is exported, use v_lock. Works around issue
described in PR kern/37808. The ideal solution here is to kill vnode
lock recursion, which should not be hard once it is understood what
the two remaining callers of vn_setrecurse() are doing.

Remove VOP_LEASE. Discussed on tech-kern.

vn_write: include f_advice in VOP_WRITE

Use FILE_LOCK() and FILE_UNLOCK()

Merge vmlocking2 to head.

branches: 1.148.4;
Remove cn_lwp from struct componentname. curlwp should be used
from on. The NDINIT() macro no longer takes the lwp parameter and
associates the credentials of the calling thread with the namei
structure.

branches: 1.147.2;
Fscow_run(): add a flag "bool data_valid" to note still valid data.
Buffers run through copy-on-write are marked B_COWDONE. This condition
is valid until the buffer has run through bwrite() and gets cleared from
biodone().

Welcome to 4.99.39.

Reviewed by: YAMAMOTO Takashi <yamt@netbsd.org>

- reduce the number of VOP_ACCESS calls for O_RDWR. for nfs, it reduces
the number of rpcs.
- reduce code duplication.

Use atomics to maintain uvmexp.{anon,exec,file}pages.

Remove the "struct lwp *" argument from all VFS and VOP interfaces.
The general trend is to remove it from all kernel interfaces and
this is a start. In case the calling lwp is desired, curlwp should
be used.

quick consensus on tech-kern

branches: 1.143.4;
Merge from vmlocking:

- Split vnode::v_flag into three fields, depending on field locking.
- simple_lock -> kmutex in a few places.
- Fix some simple locking problems.

Merge file descriptor locking, cwdi locking and cross-call changes
from the vmlocking branch.

Update the file system copy-on-write handler.

- Instead of hooking the handler on the specdev of a mounted file system
hook directly on the `struct mount'.

- Rename from `vn_cow_*' to `fscow_*' and move to `kern/vfs_trans.c'. Use
`mount_*specific' instead of clobbering `struct mount' or `struct specinfo'.

- Replace the hand-made reader/writer lock with a krwlock.

- Keep `vn_cow_*' functions and mark as obsolete.

- Welcome to NetBSD 4.99.32 - `struct specinfo' changed size.

Reviewed by: Jason Thorpe <thorpej@netbsd.org>

branches: 1.140.4; 1.140.6; 1.140.8; 1.140.10;
Retire uvn_attach() - it abuses VXLOCK and its functionality,
setting vnode sizes, is handled elsewhere: file system vnode creation
or spec_open() for regular files or block special files, respectively.

Add a call to VOP_MMAP() to the pagedvn exec path, since the vnode
is being memory mapped.

reviewed by tech-kern & wrstuden

branches: 1.139.2;
- remove pathname_ interface.
- use macros to deal with pathnames in userspace, when veriexec is used.
- reorder the veriexec_ call arguments for consistency.
With help from elad@ finding the last bug.

Change the way that emulations locate files within the emulation root to
avoid having to allocate space in the 'stackgap'
- which is very LWP unfriendly.
The additional code for non-emulation namei() is trivial, the reduction for
the emulations is massive.
The vnode for a processes emulation root is saved in the cwdi structure
during process exec.
If the emulation root the TRYEMULROOT flag are set, namei() will do an initial
search for absolute pathnames in the emulation root, if that fails it will
retry from the normal root.
".." at the emulation root will always go to the real root, even in the middle
of paths and when expanding symlinks.
Absolute symlinks found using absolute paths in the emulation root will be
relative to the emulation root (so /usr/lib/xxx.so -> /lib/xxx.so links
inside the emulation root don't need changing).
If the root of the emulation would be returned (for an emulation lookup), then
the real root is returned instead (matching the behaviour of emul_lookup,
but being a cheap comparison here) so that programs that scan "../.."
looking for the root dircetory don't loop forever.
The target for symbolic links is no longer mangled (it used to get the
CHECK_ALT_xxx() treatment, so could get /emul/xxx prepended).
CHECK_ALT_xxx() are no more. Most of the change is deleting them, and adding
TRYEMULROOT to the flags to NDINIT().
A lot of the emulation system call stubs could now be deleted.

Remove now obsolete vn_start_write() and vn_finished_write() and
corresponding flags.

Revert softdep_trackbufs() to its state before vn_start_write() was added.

Remove from struct mount now unneeded flags IMNT_SUSPEND* and
members mnt_writeopcountupper, mnt_writeopcountlower and mnt_leaf.

Welcome to 4.99.17

Remove calls to now obsolete vn_start_write() and vn_finished_write().

branches: 1.135.2; 1.135.4;
- Make the proclist_lock a mutex. The write:read ratio is unfavourable,
and mutexes are cheaper use than RW locks.
- LOCK_ASSERT -> KASSERT in some places.
- Hold proclist_lock/kernel_lock longer in a couple of places.

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

branches: 1.133.2;
Make fstrans(9) the default helper for file system suspension.
Replaces the now obsolete vn_start_write()/vn_finished_write().

Merge newlock2 to head.

New file system suspension API to replace vn_start_write and vn_finished_write.
The suspension helpers are now put into file system specific operations.
This means every file system not supporting these helpers cannot be suspended
and therefore snapshots are no longer possible.

Implemented for file systems of type ffs.

The new API is enabled on a kernel option NEWVNGATE. This option is
not enabled by default in any kernel config.

Presented and discussed on tech-kern with much input from
Bill Studenmund <wrstuden@netbsd.org> and YAMAMOTO Takashi <yamt@netbsd.org>.

Welcome to 4.99.9 (new vfs op vfs_suspendctl).

Avoid TOCTOU in Veriexec by introducing veriexec_openchk() to enforce
the policy and using a single namei() call in vn_open().

branches: 1.129.2;
Massive restructuring and cleanup of Veriexec, mainly in preparation
for work on some future functionality.

- Veriexec data-structures are no longer exposed.

- Thanks to using proplib for data passing now, the interface
changes further to accomodate that.

Introduce four new functions. First, veriexec_file_add(), to add
a new file to be monitored by Veriexec, to replace both
veriexec_load() and veriexec_hashadd(). veriexec_table_add(), to
replace veriexec_newtable(), will be used to optimize hash table
size (during preload), and finally, veriexec_convert(), to convert
an internal entry to one userland can read.

- Introduce veriexec_unmountchk(), to enforce Veriexec unmount
policy. This cleans up a bit of code in kern/vfs_syscalls.c.

- Rename veriexec_tblfind() with veriexec_table_lookup(), and make
it static. More functions that became static: veriexec_fp_cmp(),
veriexec_fp_calc().

- veriexec_verify() no longer returns the entry as well, but just
sets a boolean indicating whether an entry was found or not.

- veriexec_purge() now takes a struct vnode *.

- veriexec_add_fp_name() was merged into veriexec_add_fp_ops(), that
changed its name to veriexec_fpops_add(). veriexec_find_ops() was
also renamed to veriexec_fpops_lookup().

Also on the fp-ops front, the three function types used to initialize,
update, and finalize a hash context were renamed to
veriexec_fpop_init_t, veriexec_fpop_update_t, and veriexec_fpop_final_t
respectively.

- Introduce a new malloc(9) type, M_VERIEXEC, and use it instead of
M_TEMP, so we can tell exactly how much memory is used by Veriexec.

- And, most importantly, whitespace and indentation nits.

Built successfuly for amd64, i386, sparc, and sparc64. Tested on amd64.

printf() -> log().

Adapt to changes suggested by yamt@ to get rid of __UNCONST() stuff.

While here, don't leak pathbuf on success.

Don't allocate MAXPATHLEN on the stack.

Prompted by and initial diff okay yamt@

add support for O_DIRECT (I/O directly to application memory,
bypassing any kernel caching for file data).

branches: 1.124.2;
Fix typo.

Prevent a veriexec file from being truncated.

branches: 1.122.4;
at the request of elad, as veriexec.h has returned, revert the changes
from 2006-07-25.

mechanically go through and
s,include "veriexec.h",include <sys/verified_exec.h>,
as the former has apparently gone away.

replace magic numbers for strict levels (0-3) with defines.

finally do things properly. veriexec_report() takes flags, not three ints.

some fixes:
- adapt to NVERIEXEC in init_sysctl.c.
- we now need "veriexec.h" for NVERIEXEC.
- "opt_verified_exec.h" -> "opt_veriexec.h", and include it only where
it is needed.

Use the LWP cached credentials where sane.

kill a VOP_GETATTR() we don't need for veriexec.

deprecate the VERIFIED_EXEC option; now we only need the pseudo-device to
enable it. while here, some config file tweaks.

tons of input from cube@ (thanks!) and okay blymn@.

oops, forgot to commit that one. thanks Arnaud Lacombe.

okay, since there was no way to divide this to two commits, here it goes..

introduce fileassoc(9), a kernel interface for associating meta-data with
files using in-kernel memory. this is very similar to what we had in
veriexec till now, only abstracted so it can be used more easily by more
consumers.

this also prompted the redesign of the interface, making it work on vnodes
and mounts and not directly on devices and inodes. internally, we still
use file-id but that's gonna change soon... the interface will remain
consistent.

as a result, veriexec went under some heavy changes to conform to the new
interface. since we no longer use device numbers to identify file-systems,
the veriexec sysctl stuff changed too: kern.veriexec.count.dev_N is now
kern.veriexec.tableN.* where 'N' is NOT the device number but rather a
way to distinguish several mounts.

also worth noting is the plugging of unmount/delete operations
wrt/fileassoc and veriexec.

tons of input from yamt@, wrstuden@, martin@, and christos@.

Limit the size of any kernel buffers allocated by the VOP_READDIR
routines to MAXBSIZE.

branches: 1.111.2;
integrate kauth.

XXX: GCC uninitialized.

Change VOP_FCNTL to take an unlocked vnode. Approved by wrstuden@.

vn_rdwr(): Initialize `mp' to NULL. vn_finished_write() would be called
with uninitialized `mp' if `vp->v_type == VCHR'.

From Coverity CID 2475.

branches: 1.107.2;
remove a wrong assertion.

branches: 1.106.2; 1.106.4;
merge yamt-uio_vmspace branch.

- use vmspace rather than proc or lwp where appropriate.
the latter is more natural to specify an address space.
(and less likely to be abused for random purposes.)
- fix a swdmover race.

vn_read: don't bother to allocate read-ahead context here.
it will be done in uvn_get if necessary.