rump: Sprinkle sys/syncobj.h here too.

Fix root search in RAID 1 sets

We use the wedge information given by bootstrap, where the kernel was
found. This requires src/sys/arch/i386/stand/i386/lib/biosdisk.c 1.59
to work in all cases.

rumpkern: Provide stub pnbuf_cache weak alias.

Needed only by sys_module.c handle_modctl_load, which won't work if
there's no rumpvfs linked in anyway.

rumpkern: Provide fs_filtops weak alias.

We should introduce a link set for sys_kfilters, but this works
around the problem for now -- librump's dependency on librumpvfs
symbols.

nkmempages should be size_t

Rewrite entropy subsystem.

Primary goals:

1. Use cryptography primitives designed and vetted by cryptographers.
2. Be honest about entropy estimation.
3. Propagate full entropy as soon as possible.
4. Simplify the APIs.
5. Reduce overhead of rnd_add_data and cprng_strong.
6. Reduce side channels of HWRNG data and human input sources.
7. Improve visibility of operation with sysctl and event counters.

Caveat: rngtest is no longer used generically for RND_TYPE_RNG
rndsources. Hardware RNG devices should have hardware-specific
health tests. For example, checking for two repeated 256-bit outputs
works to detect AMD's 2019 RDRAND bug. Not all hardware RNGs are
necessarily designed to produce exactly uniform output.

ENTROPY POOL

- A Keccak sponge, with test vectors, replaces the old LFSR/SHA-1
kludge as the cryptographic primitive.

- `Entropy depletion' is available for testing purposes with a sysctl
knob kern.entropy.depletion; otherwise it is disabled, and once the
system reaches full entropy it is assumed to stay there as far as
modern cryptography is concerned.

- No `entropy estimation' based on sample values. Such `entropy
estimation' is a contradiction in terms, dishonest to users, and a
potential source of side channels. It is the responsibility of the
driver author to study the entropy of the process that generates
the samples.

- Per-CPU gathering pools avoid contention on a global queue.

- Entropy is occasionally consolidated into global pool -- as soon as
it's ready, if we've never reached full entropy, and with a rate
limit afterward. Operators can force consolidation now by running
sysctl -w kern.entropy.consolidate=1.

- rndsink(9) API has been replaced by an epoch counter which changes
whenever entropy is consolidated into the global pool.
. Usage: Cache entropy_epoch() when you seed. If entropy_epoch()
has changed when you're about to use whatever you seeded, reseed.
. Epoch is never zero, so initialize cache to 0 if you want to reseed
on first use.
. Epoch is -1 iff we have never reached full entropy -- in other
words, the old rnd_initial_entropy is (entropy_epoch() != -1) --
but it is better if you check for changes rather than for -1, so
that if the system estimated its own entropy incorrectly, entropy
consolidation has the opportunity to prevent future compromise.

- Sysctls and event counters provide operator visibility into what's
happening:
. kern.entropy.needed - bits of entropy short of full entropy
. kern.entropy.pending - bits known to be pending in per-CPU pools,
can be consolidated with sysctl -w kern.entropy.consolidate=1
. kern.entropy.epoch - number of times consolidation has happened,
never 0, and -1 iff we have never reached full entropy

CPRNG_STRONG

- A cprng_strong instance is now a collection of per-CPU NIST
Hash_DRBGs. There are only two in the system: user_cprng for
/dev/urandom and sysctl kern.?random, and kern_cprng for kernel
users which may need to operate in interrupt context up to IPL_VM.

(Calling cprng_strong in interrupt context does not strike me as a
particularly good idea, so I added an event counter to see whether
anything actually does.)

- Event counters provide operator visibility into when reseeding
happens.

INTEL RDRAND/RDSEED, VIA C3 RNG (CPU_RNG)

- Unwired for now; will be rewired in a subsequent commit.

- Eliminate the global "boottime" variable, which was being accessed
without any synchronization against changes by e.g. clock_settime().
- Replace with new getbinboottime() / getnanoboottime() / getmicroboottime()
functions (naming mirrors that of other time access functions in kern_tc.c).
It returns the (maybe-converted) value of timebasebin, which also tracks
our estimate of when the system was booted (i.e. the legacy "boottime" was
redundant).

XXX There needs to be a lockless synchronization mechanism for reading
timebasebin, but this is a problem in kern_tc.c that pre-existed these
"boottime" changes. At least now the problem is centralized in one location.

Add shutting_down variable for rump.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

Provide a weak alias for vnode_to_path to be used unless librumpvfs is present.

Apply a patch from hannken@ which adds a weak alias for rump_getcwd_common()
allowing -lrump to be used without -lrumpvfs.

This is an alternate fix to the earluer one which added -lrumvfs to
many rump based tests (and the rump server) which might be undone soon.

This also fixes the sun2 build.

Rumpkernel has its own thread deallocation. Add missing fstrans_lwp_dtor()
to lwproc_freelwp().

PR bin/50350: rump/rumpkern/t_sp/stress_{long,short} fail on Core 2 Quad

no more need for get_expose_address() here.

add get_expose_address()

Bring back three state file system suspension:

NORMAL -> SUSPENDING -> SUSPENDED

and add operation fstrans_start_lazy() that only blocks while SUSPENDED.

Change vndthread() support operation handle_with_rdwr() to bracket
its file system operations by fstrans_start_lazy() and fstrans_done().

PR kern/53624 (dom0 freeze on domU exit)

Allow syscall_establish() to install new syscalls when the existing
entry-point is either sys_nomodule or sys_nosys. Update the
makesyscalls.sh script to create a const array of bits to allow
syscall_disestablish() to properly restore the original entry-point.
Update all the initializers of struct emul to initialize the pointer
to the bit array struct emul.

XXX Regen of all files created by makesyscalls.sh will come soon,
XXX followed by a kernel version bump (since struct emul is being
XXX modified).

This commit should address PR kern/45781 and also removes the need
for the work-around for that PR in file

sys/arch/usermode/modules/syscallemu/syscallemu.c

branches: 1.185.2; 1.185.4;
Add missing inclusion of pserialize.h (fix build)

Implement debugging feature for pserialize(9)

The debugging feature detects violations of pserialize constraints.
It causes a panic:
- if a context switch happens in a read section, or
- if a sleepable function is called in a read section.

The feature is enabled only if LOCKDEBUG is on.

Discussed on tech-kern@

added booted_method

Operations fstrans_start() and fstrans_start_nowait() now always
use FSTRANS_SHARED as lock type so remove the lock type argument.

File system state FSTRANS_SUSPENDING is now unused so remove it.

Regen vnode_if files.

Ride 8.99.1 less than a hour ago.

branches: 1.181.6;
Add weak aliases for _fstrans_start() and fstrans_done().

branches: 1.180.2;
physmem should be of type psize_t

Also, use PRIxPSIZE when printf(9)ing physmem.

branches: 1.179.2;
Put the kernelside rump kernel headers into <rump-sys> instead of
sprinkling them around the faction directories. Avoids having
to add a CPPFLAGS (or several) to pretty much every component
Makefile.

Leave compat headers around in the old locations.

The commit changes some autogenerated files, but I'll fix the
generators shortly and regen.

Fix build with RUMP_LOCKDEBUG=yes

put lwp/proc stuff into the same source module (emul.c -> lwproc.c)

massively reduce header pollution from times prehistoric

boottime is a timespec, not timeval

Now that the table of auto-loadable syscalls is per-emulation, make sure
that the rump-kernel has its own list. Otherwise, missing syscalls won't
trigger a module auto-load.

This commit finishes the work to get tests/lib/librumphijack/t_tcpip
nfs_autoload test case working again after 16 months of failures! (see
PR bin/49153).

add cpu_getmodel()

Since the rump kernel does not know when the container it's running in
actually halts, print "halted" in the hypercall.

Include kern_clock.c in rump kernels.

Give remote clients struct pmap storage. Although the pmap is unused,
that way we can sure that the pmap macro framework doesn't access all
the wrong places.

Put all sysproxy routines to their own C module, sysproxy.c

branches: 1.168.2;
delay() can't use rumpuser_clock_sleep(), because that may unschedule().

Instead, busyloop with calls to rumpuser_clock_gettime(). Fixing delay()
properly should addressed in hypercall rev.18 by introducing a nowrap
variant/flag for rumpuser_clock_sleep().

Attempt to make sure that DELAY()/delay() is available in all MD names.

Add missing __diagused in rump code

cpu_reboot() is more of an emul.c thing than a rump.c thing

branches: 1.164.2;
Put the syncobjs in emul.c instead of locks.c so that they are
present for both locks.c and locks_up.c alike.

+ resettodr

lwp0 needs l_proc set for panic->cpu_reboot to dump core the right way

Use subr_cprng.c instead of stub implementation. Rijndael migrates from
rumpkern_crypto to rumpkern due to it being mandatory for cprng.

Translate return values for emulations, e.g. Linux. For ports without
__HAVE_MINIMAL_EMUL, we simply look up the values from p->p_emul->e_errno.
For ports which cannot afford to keep an extra pointer per emul structure
around, we hope there is __HAVE_SYSCALL_INTERN support and thread the
errno values through p_emuldata. Notably, we cannot alter the syscall
method like most ports do with syscall_intern, since they do it via
p_mdproc, so MI code is not possible there.

Support ktrace for rump kernels.

Requested by Justin Cormack on rumpkernel-users.

branches: 1.158.4;
Flip parameter order in the clock hypercalls to make them consistent
with everything else wrt in/out parameters.

Make hypercall calling conventions consistent: iff a hypercall can fail,
it returns an int containing the error value.

Improve the time-related hypercalls so that's it's possible to
sleep until an absolute time on the host's monotonic clock (should
something like that be supported).

calculate vnode cache size based on the resource it gets allocated from
this stops setting kern.maxvnodes to high so it exhausts available space in kmem

http://mail-index.netbsd.org/tech-kern/2013/03/08/msg015095.html

set e_nsysent

Stopgap fix to make rump cooperate with pserialize, may be revisited later.
Patch from pooka, ok: rmind. No related regressions in a complete atf test
run (which works again with this, even on non x86 SMP machines).

Provide xc_send_ipi() routine in RUMP, which is required for high-priority
xcall(9) mechanism. It is emulated using low-priority xcall(9).

split device_t/softc for all remaining drivers.
replace "struct device *" with "device_t".
use device_xname(), device_unit(), etc.

branches: 1.150.4; 1.150.10; 1.150.14;
Update copyright statements.

no functional change.

Since "physmem" is largely unused except for zfs wanting to know
if it has enough memory available, just pick a number which makes
zfs happy.

We *could* use a MIN() of available host mem and rump_memlimit,
though ...

branches: 1.148.2; 1.148.4;
Support LOCKDEBUG. To use it, compile sys/rump with RUMP_LOCKDEBUG=yes.

requested by martin (sparc64 gdb cannot reliably produce a stack trace)

Realize the >1yo comment above rump_reboot and retire them to make
room for sys_reboot.

Make common kernel module binaries work on both sun3 and sun3x.
Tested on 3/160 (on TME) and (real) 3/80.

XXX: module files can be loaded only on single user?

Implement rump_lwproc: the new lwp/proc management routines for
rump. These move the management of the pid/lwpid space from the
application into the kernel, make code more robust, and make it
possible to attach multiple lwp's to non-proc0 processes.

As normal, fix breakage from untested commits by rmind.

Remove comment which wasn't supposed to get committed.

Make some stubby tty routines used by printf weak aliases, so that
the full ones will be used if we have the upcoming tty component
loaded.

Fix previous in emul.c -- only numbers are operands for cpp comparisons.
Apparently non-numbers logically produce arch-dependent behaviour.

deal with sun3's module disability

add rest of sparc pagesize variables

Use kern_proc.c instead of a collection of stubs. But what we
really wanted from this commit was the support for proc_specificdata.

TODO: make creating a new process actually use kern_proc and
maybe even add an interface which starts a process with
"any pid you don't like"

adjust comment in previous.
XXX: should make that (and physmem) mean something here

remove unnecessary #ifdef

Now that we have a big devsw_conv0, need at least as many entries
in devsw as the highest device number in the conv table. Do a
"good for a hundred years" guess now, fix properly later.

Include devsw_conv0 from an i386 kernel compilation (no, we don't
care about the arch as long as all the devices we care about are
present). The file should be autogenerated, but that requires some
more changes to config(1).

Make initial devsw a little more believable. Especially adjust
max_sys_devsw, since it was previously a bit too much, given that
the correct value is 0.

remove stuff which now comes via std. kernel source modules

support kern_resource

Move all signal-related from emul.c to signals.c. Additionally,
define a few alternate signal models for the rump kernel, including
ones where signals are ignored or sent to host processes.

no \n in panic

One emul is enough and since we need emul_netbsd, retire emul_rump.

Move scheduling related routines from emul.c to scheduler.c

Use kern_syscall.c instead of homegrown syscall dis/establishment routines.

Include kern_tc and use a timecounter driver instead of homerolled
kern_tc implementation.

We don't have a real rootdev (by default at least), so set it to NODEV.

branches: 1.123.2;
add signal stubs required by mfs

Remove separate mb_map. The nmbclusters is computed at boot time based
on the amount of physical memory and limited by NMBCLUSTERS if present.
Architectures without direct mapping also limit it based on the kmem_map
size, which is used as backing store. On i386 and ARM, the maximum KVA
used for mbuf clusters is limited to 64MB by default.

The old default limits and limits based on GATEWAY have been removed.
key_registered_sb_max is hard-wired to a value derived from 2048
clusters.

branches: 1.121.2;
Include newly-created subr_device.c and remove few special case
device accessor copypastes. This makes it possible to link static
binaries which use -lrumpdev.

include kern_hook.c

Use subr_percpu.c instead of homegrown implementation. ...except
when using malloc(3)-relegated allocators (happens in production
e.g. on Linux), since subr_percpu.c uses vmem and i don't want to
reimplement vmem.

Minimize unnecessary differences in rump.

Add a few symbols used by the tty code.

Include kern_pmf.c in rumpdev.

Expand PMF_FN_* macros.

allow rump to build

Generate vers.c and include it in the kernel component.

Decide it's not worth the fuss to have rumpfs as a module and just
hardcode attach into rump_vfs_init(). Saves us from a lot of
pingpong init bouncing from one component to another to get the order
right.

Include cpu crosscall support (instead of stubbing it out).

For rumpfs, do mountroot instead of the bunch of homegrown hacks
currently there. Still needs a little massage to get the kernel
interfaces right and avoid copypaste especially from main().

Also, move it a bit more into the direction of a real file system
(finally!) by giving it a vfsops. Most ops are still unimplemented,
though.

misc_stub and emul have been the same thing for a looong time now,
so just move the few remaining routines in misc_stub to emul.

Give the kthread->pthread interface emulation its own module.

Pull all relegating memory allocators under a common roof in memalloc.c

move copy-related routines to their own module

Use std. uiomove() & friends.

Sync rump with kernel changes.

Include sys_select.c for proper select()/poll() support.

Give lwp usage some much-needed love: stop treating lwp0 as the
all-sink and make sure each separate thread in rump has its own
lwp. Happy-go-lucky callers will get scheduled a temporary lwp
on entry, while true lwp connoisseurs may request a stable lwp
for their purposes. Some more love may be required later down the
road, but for now different threads will stepping on each others
toes.

Add initial work on a rump virtual cpu scheduler. This is necessary
for kernel code which has been written to avoid MP contention by
using cpu-local storage (most prominently, select and pool_cache).

Instead of always assuming rump_cpu, the scheduler must now be run
(and unrun) on all entry points into rump. Likewise, rumpuser
unruns and re-runs the scheduler around each potentially blocking
operation. As an optimization, I modified some locking primitives
to try to get the lock without blocking before releasing the cpu.

Also, ltsleep was modified to assume that it is never called without
the biglock held and made to use the biglock as the sleep interlock.
Otherwise there is just too much drama with deadlocks. If some
kernel code wants to call ltsleep without the biglock, then, *snif*,
it's no longer supported and rump and should be modified to support
newstyle locks anyway.

"rumppriv" goes back to "rump" per internal interface naming change.

Adjust rump sources for external/internal interfaces.
No functional change.

Provide an interface for reboot.

Need proc_uidmatch

remove a no longer necessary hack

default to AB_SILENT

provide pmf stubs

move hppa cpp kludge to the end of the file

add a few global symbols required by kernel code

realloc works much better if it actually uses realloc instead of malloc

Nuke some bss stuff which is coming from vfs_bio via rumpvfs.

Don't create unpgc thread if RUMP_THREADS is 0. The benefit is the
usual: possible to run programs using AF_LOCAL without threads.

Add proof-of-concept code for enabling system calls to rump virtual
kernels running in other processes on the same machine or on an
entirely different host. I wrote this a while ago and am now
committing it mainly to avoid losing it. It works, but could do
with a little tuning here and there.

What this will hopefully eventually buy us is the ability to use
standard userland tools to configure rump kernels, e.g. ifconfig(8)
and route(8) could be used to configure the networking stack provided
by a rump kernel. Also some distributed OS implications may apply.

fun fact: a system call which just does copyin/copyout takes >1000x
longer when made over the LAN as compared to when made on the same
machine.

Fix getnano/microuptime to report actual uptime.

Catch up with recent changes to sys/proc.h (Hello, rmind!)

we want microuptime too.

need getnanouptime not getmicrouptime

one more timeval -> timespec line

Catch up with kernel changes

Support kqueue in the rump virtual kernel.

Ansify function definitions w/o arguments. Generated with sed.

decouple rumpuser gettime from struct timeval

Make rumpuser stat and nanosleep independent of the host system
stat and timespec definitions.

branches: 1.77.2;
Make the clock interrupt tick based on absolute time instead of
relative time. This prevents drifting. Also, keep track of time
within userspace, so we do not have to make a syscall to get the
clock value. This is approximately 7 times cheaper, but on the
negative side is limited to the clock interrupt frequency.

Implement block I/O as a real block driver instead of a hacked copy
of specfs. That was easier years ago when rump didn't support
devices, but brings no gain now. This allows us to include the
real specfs in rump.

Shove in some hppa love to help with spcopy.S (why does that need
to be in libkern?).

Protect against nfssilly in RUMP_THREADS=0

merge christos-time_t

provide kern_realloc

Rename malloc() to kern_malloc() to avoid name conflict with libc.
Now it is possible to use the kernel linker with rump.

Include libkern contents in librump.

Sauce with some kludges:
* revert rev 1.68 of emul.c: delay() is hopelessly MD and needs
whipping (not the fun kind)
* #undef DELAY in cpu.h

This will hopefully fix build for all archs and give me time to
hop through our 46521 fine archs and edit the headers.

Export delay, _delay and delay_func symbols to attempt to appease
the macro mania set over all architectures.

Include kernel printf routines in rump instead of relying on the
magic libc symbol. This also allows to bid farewell to subr_prf2.c
and merge the contents back to subr_prf.c. The host kernel bridging
is now done via rumpuser_putchar().

Define MODULAR for rump core components. This enables module
loading via the kernel module framework (instead of dlopen()).
For now it only works on amd64 and i386, but for the rest it should
just be a matter of including the relevant kobj_machdep.c modules
from the kernel sources.

properly handle kthread_exit()

include subr_devsw in rumpkern

branches: 1.63.2;
undo turn malloc flags to enums.
Requested by christos.

Keep malloc flag defines in kernel section.

Turn malloc flags into an enum. Fixes build breakage.

__KERNEL_RCSID

If available (__NetBSD__), use pthread_setname_np() to set the
thread name for kthread_create().

If the arch uses __BSWAP_RENAME, provide non-namespaced kernel
symbols which just call the libc symbols. Allows to get rid of
rump machine/bswap.h.

XXX:
1) this is unlikely to work on non-NetBSD
==>
2) should look into including libkern in librump

Make non-mpsafe threads KERNEL_LOCK() before starting execution.

Remove the whole magic ubc window thingie. All file systems use
ubc_uiomove() now, so we can hook ourselves there.

Add a few symbols required by nfsd

Add more verbose commentage to the module thread non-creation.

Ignore the module unload thread.

branches: 1.53.2; 1.53.4; 1.53.8;
Add some more trivial emulations.

Provide psignal() so that we don't get linked against psignal(3).
As there is no real process model currently, just "trap" the signals
appropriately.

Support callouts and call callout_hardclock() from the timer
interrupt thread.

The sleepq implementation required for callouts is horrible, kludged
only for callouts, and generally unacceptable. It needs revisiting,
but I'm not sure yet should rump or kern_timeout be improved. It's
almost untested as of now, but committing this will give me some
maneuvering space while letting application compile.

Remove rumpuser_yield(). Not only doesn't it really make sense
here, some kind soul made it completely empty.

Switch to std kern_auth.

Move global malloc types from kern_malloc into a separate module.

Make it possible to control starting of threads per env variable
instead of only at compile-time.

Add support for using real kmem/vmem. Don't enable it by default,
though, since it a) is a lot of unnecessary indirection in rump
b) requires callouts which are so far unimplemented.

support real sysctls

Install rump libraries and utilities to the base system and remove the
private non-installed build infrastructure from sys/rump.

breakdown of commit:
* install relevant headers into /usr/include/rump
* build sys/rump/librump/rumpuser and sys/rump/librump/rumpkern
from src/lib and install as librumpuser and librump, respectively
+ this retains the ability to test a librump build with just the
kernel sources at hand
* move sys/rump/fs/lib/libukfs and sys/rump/fs/lib/libp2k to src/lib
for general consumption, they are not kernel-space dwellers anyway
* build and install sys/rump/fs/lib/lib$fs as librumpfs_$fs
* add chapter 3 manual pages for rump, rumpuser, ukfs and p2k
* build and install userspace kernel file system daemons if MKPUFFS=yes
is spexified
* retire fsconsole for now, it will make a comeback with an actually
implemented version shortly

Repeat after me: do not pollute sys with #ifdef _RUMPKERNEL

emulate vlog()

branches: 1.41.2;
Don't compile kern_lock for rump any more, it's no longer required.
Allows us to get rid of the incorrect _RUMPKERNEL ifdefs outside sys/rump.

Make kpause() use nanosleep() instead of usleep(). Fixes >=1s sleeps
to actually sleep a bit too.

from Arnaud Ysmal

branches: 1.39.2;
Also fake the namecache g/c thread in kthread_create() so that file
systems can again be run without threads (and hence gdb can be used
on them).

branches: 1.38.2; 1.38.4;
Catch up with process locking changes.

branches: 1.37.2;
Add a device_xname() stub.

- for some ports, especially for ones without pmap_growkernel,
buf_memcalc is used by bootstrap as well. fix NULL dereference for them.
- limit kva usage for each cache to 20% of vm_map. XXX a bit arbitrary.
- add a comment.

Adapt to sel* changes

when calculating some cache sizes, consider the amount of available kva.
PR/33185.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Add lwp_unsleep from kern/kern_lwp.c so that this builds. ok ad@

Support multiple file systems within one process with ukfs by using
a "chroot" for each file system.

Backup some fixes for recent breakage from local tree. Also some
other improvements such as exporting the real kernel namei and
using that in ukfs instead of the homegrown heap'o hacks namei.
"etcetc".

branches: 1.29.2; 1.29.6;
Implement yield().

Add dummy hardclock_ticks.

Use vfs_subr.c from sys/kern. This brings differences in the vnode
life cycle between rump and a real kernel to a minimum.

Use namei() etc. from kernel sources instead of a reimplementation.
To accommodate, give the rootvnode its own vnode op vector with a
simple lookup operation. This is used for looking up the file
system's device vnode instead of doing that directly in a homesmoked
namei().

Don't use PAGE_SIZE when faking physmem, as it's not always available
at compile-time. We could init it runtime, but it's just a random
number anyway.

Default physmem was too tight, increase to 256megs from 0 bytes.

Until debugging threaded programs in NetBSD is fixed, supply the
cpp option RUMP_WITHOUT_THREADS as a workaround. If defined, it
makes rump itself operate single-threaded and prevents kthread_create()
from working.

Now that __HAVE_TIMECOUNTER and __HAVE_GENERIC_TODR are invariants,
remove the conditionals and the code associated with the undef case.

Print "panic: " before panicstr when panicking.

Add the ability to run puffs in userspace. This means that puffs
can now be developed in userspace using puffs for development
(I hate emulators, they are annoyingly clumsy).

To e.g. mount psshfs using puffs-on-puffs, run fs/bin/syspuffs/syspuffs
with the regular mount_psshfs command line as an argument:

golem> ./syspuffs /usr/sbin/mount_psshfs ftp.netbsd.org:/pub /puffs

This will make the mount appear as usual, with the exception that the
requests will be passed through puffs both in the kernel and userspace:

ftp.netbsd.org:/pub on /puffs type puffs|p2k|puffs|psshfs

branches: 1.19.6;
fix build (hi rmind!)

branches: 1.18.2;
* sprinkle some locking into the vm code
* avoid extra insert+search+remove step in file systems using
ubc_uiomove() instead of standard uiomove()

Make it possible to run rumps multithreaded. This brings real
locking and makes it possible to run file systems which create
threads. It also makes rump file system behaviour better match
file system behaviour in the kernel.

branches: 1.16.2;
"flags * (M_CANFAIL | M_NOWAIT)" is probably not correct. Use & instead.
And while committing, add other random cruft I've needed recently.

machine/{bus,cpu,intr}.h -> sys/{bus,cpu,intr}.h

branches: 1.14.2;
Don't whine about wakeup() not being implemented every time it's
called. Nobody cares unless ltsleep() is called (and it panics).

branches: 1.13.2;
If !_HAVE_TIMECOUNTER, initialize 'time' instead of 'time_second.'

Provide errno from rumpuser_{gettimeofday,close}() to be consistent.

branches: 1.11.2; 1.11.4;
few panicky functions

branches: 1.10.2;
Hide NetBSD kernel headers completely from ukfs. This includes creating
accessors for:
* struct mount & VFS ops
* struct uio
* struct vnode
* struct vattr

and some namespace games for:
* namei flags
* VOPs
* enum vtype

Also, split rump services into two categories: library private and public
(rump_private.h and rump.h, respectively).

As a result, it is now possible to compile and use the NetBSD kernel
file systems on Linux (and probably other systems too with very
little work), although the makefiles need a bit of work to make it
a pleasureable experience.

Wrap malloc() so that we catch the kernel arguments (namely M_ZERO)
properly. It's fairly amusing that this wasn't noticed until now.

branches: 1.8.2;
Kill handrolled buffercache and use vfs_bio from the kernel. This is
mostly to get the flag jungle in sync with the kernel.

Use supermarket variety vfs_vnops.c instead of homecooked routines
(that's actually a good thing here).

compile in libkern __assert.c, argument order differs from libc model

Add some stubs for lfs.

Provide rumpmachine bswap.h, which makes bswapxx() call rumpuser
directly instead of relying on a symbol in rumpkern. I would like
to make it call the libc symbol directly, but I don't currently know
how to make it do that MI.

Makes hfs work (on i386), as it avoids endless recursion in bswap64().
Thanks to dillo for the image!

* compile param.c
* move opts to a separate directory

add microtime

branches: 1.1.2;
Introduce RUMPs - Runnable Userspace Meta-Programs

/sys/rump contains programs which run unmodified kernel code in an
emulated userspace environment. The kernel environment is provided
by librump. Currently supported are a number of file systems,
which by using puffs integrate seamlessly into the system and
provide a similar user experience to if the code was running as
part of the kernel. Potential future rumpification targets include
for example parts of the networking stack and some device drivers.

This work was supported by Google Summer of Code 2007.

Fix root search in RAID 1 sets

We use the wedge information given by bootstrap, where the kernel was
found. This requires src/sys/arch/i386/stand/i386/lib/biosdisk.c 1.59
to work in all cases.

rumpkern: Provide stub pnbuf_cache weak alias.

Needed only by sys_module.c handle_modctl_load, which won't work if
there's no rumpvfs linked in anyway.

rumpkern: Provide fs_filtops weak alias.

We should introduce a link set for sys_kfilters, but this works
around the problem for now -- librump's dependency on librumpvfs
symbols.

nkmempages should be size_t

Rewrite entropy subsystem.

Primary goals:

1. Use cryptography primitives designed and vetted by cryptographers.
2. Be honest about entropy estimation.
3. Propagate full entropy as soon as possible.
4. Simplify the APIs.
5. Reduce overhead of rnd_add_data and cprng_strong.
6. Reduce side channels of HWRNG data and human input sources.
7. Improve visibility of operation with sysctl and event counters.

Caveat: rngtest is no longer used generically for RND_TYPE_RNG
rndsources. Hardware RNG devices should have hardware-specific
health tests. For example, checking for two repeated 256-bit outputs
works to detect AMD's 2019 RDRAND bug. Not all hardware RNGs are
necessarily designed to produce exactly uniform output.

ENTROPY POOL

- A Keccak sponge, with test vectors, replaces the old LFSR/SHA-1
kludge as the cryptographic primitive.

- `Entropy depletion' is available for testing purposes with a sysctl
knob kern.entropy.depletion; otherwise it is disabled, and once the
system reaches full entropy it is assumed to stay there as far as
modern cryptography is concerned.

- No `entropy estimation' based on sample values. Such `entropy
estimation' is a contradiction in terms, dishonest to users, and a
potential source of side channels. It is the responsibility of the
driver author to study the entropy of the process that generates
the samples.

- Per-CPU gathering pools avoid contention on a global queue.

- Entropy is occasionally consolidated into global pool -- as soon as
it's ready, if we've never reached full entropy, and with a rate
limit afterward. Operators can force consolidation now by running
sysctl -w kern.entropy.consolidate=1.

- rndsink(9) API has been replaced by an epoch counter which changes
whenever entropy is consolidated into the global pool.
. Usage: Cache entropy_epoch() when you seed. If entropy_epoch()
has changed when you're about to use whatever you seeded, reseed.
. Epoch is never zero, so initialize cache to 0 if you want to reseed
on first use.
. Epoch is -1 iff we have never reached full entropy -- in other
words, the old rnd_initial_entropy is (entropy_epoch() != -1) --
but it is better if you check for changes rather than for -1, so
that if the system estimated its own entropy incorrectly, entropy
consolidation has the opportunity to prevent future compromise.

- Sysctls and event counters provide operator visibility into what's
happening:
. kern.entropy.needed - bits of entropy short of full entropy
. kern.entropy.pending - bits known to be pending in per-CPU pools,
can be consolidated with sysctl -w kern.entropy.consolidate=1
. kern.entropy.epoch - number of times consolidation has happened,
never 0, and -1 iff we have never reached full entropy

CPRNG_STRONG

- A cprng_strong instance is now a collection of per-CPU NIST
Hash_DRBGs. There are only two in the system: user_cprng for
/dev/urandom and sysctl kern.?random, and kern_cprng for kernel
users which may need to operate in interrupt context up to IPL_VM.

(Calling cprng_strong in interrupt context does not strike me as a
particularly good idea, so I added an event counter to see whether
anything actually does.)

- Event counters provide operator visibility into when reseeding
happens.

INTEL RDRAND/RDSEED, VIA C3 RNG (CPU_RNG)

- Unwired for now; will be rewired in a subsequent commit.

- Eliminate the global "boottime" variable, which was being accessed
without any synchronization against changes by e.g. clock_settime().
- Replace with new getbinboottime() / getnanoboottime() / getmicroboottime()
functions (naming mirrors that of other time access functions in kern_tc.c).
It returns the (maybe-converted) value of timebasebin, which also tracks
our estimate of when the system was booted (i.e. the legacy "boottime" was
redundant).

XXX There needs to be a lockless synchronization mechanism for reading
timebasebin, but this is a problem in kern_tc.c that pre-existed these
"boottime" changes. At least now the problem is centralized in one location.

Add shutting_down variable for rump.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

Provide a weak alias for vnode_to_path to be used unless librumpvfs is present.

Apply a patch from hannken@ which adds a weak alias for rump_getcwd_common()
allowing -lrump to be used without -lrumpvfs.

This is an alternate fix to the earluer one which added -lrumvfs to
many rump based tests (and the rump server) which might be undone soon.

This also fixes the sun2 build.

Rumpkernel has its own thread deallocation. Add missing fstrans_lwp_dtor()
to lwproc_freelwp().

PR bin/50350: rump/rumpkern/t_sp/stress_{long,short} fail on Core 2 Quad

no more need for get_expose_address() here.

add get_expose_address()

Bring back three state file system suspension:

NORMAL -> SUSPENDING -> SUSPENDED

and add operation fstrans_start_lazy() that only blocks while SUSPENDED.

Change vndthread() support operation handle_with_rdwr() to bracket
its file system operations by fstrans_start_lazy() and fstrans_done().

PR kern/53624 (dom0 freeze on domU exit)

Allow syscall_establish() to install new syscalls when the existing
entry-point is either sys_nomodule or sys_nosys. Update the
makesyscalls.sh script to create a const array of bits to allow
syscall_disestablish() to properly restore the original entry-point.
Update all the initializers of struct emul to initialize the pointer
to the bit array struct emul.

XXX Regen of all files created by makesyscalls.sh will come soon,
XXX followed by a kernel version bump (since struct emul is being
XXX modified).

This commit should address PR kern/45781 and also removes the need
for the work-around for that PR in file

sys/arch/usermode/modules/syscallemu/syscallemu.c

branches: 1.185.2; 1.185.4;
Add missing inclusion of pserialize.h (fix build)

Implement debugging feature for pserialize(9)

The debugging feature detects violations of pserialize constraints.
It causes a panic:
- if a context switch happens in a read section, or
- if a sleepable function is called in a read section.

The feature is enabled only if LOCKDEBUG is on.

Discussed on tech-kern@

added booted_method

Operations fstrans_start() and fstrans_start_nowait() now always
use FSTRANS_SHARED as lock type so remove the lock type argument.

File system state FSTRANS_SUSPENDING is now unused so remove it.

Regen vnode_if files.

Ride 8.99.1 less than a hour ago.

branches: 1.181.6;
Add weak aliases for _fstrans_start() and fstrans_done().

branches: 1.180.2;
physmem should be of type psize_t

Also, use PRIxPSIZE when printf(9)ing physmem.

branches: 1.179.2;
Put the kernelside rump kernel headers into <rump-sys> instead of
sprinkling them around the faction directories. Avoids having
to add a CPPFLAGS (or several) to pretty much every component
Makefile.

Leave compat headers around in the old locations.

The commit changes some autogenerated files, but I'll fix the
generators shortly and regen.

Fix build with RUMP_LOCKDEBUG=yes

put lwp/proc stuff into the same source module (emul.c -> lwproc.c)

massively reduce header pollution from times prehistoric

boottime is a timespec, not timeval

Now that the table of auto-loadable syscalls is per-emulation, make sure
that the rump-kernel has its own list. Otherwise, missing syscalls won't
trigger a module auto-load.

This commit finishes the work to get tests/lib/librumphijack/t_tcpip
nfs_autoload test case working again after 16 months of failures! (see
PR bin/49153).

add cpu_getmodel()

Since the rump kernel does not know when the container it's running in
actually halts, print "halted" in the hypercall.

Include kern_clock.c in rump kernels.

Give remote clients struct pmap storage. Although the pmap is unused,
that way we can sure that the pmap macro framework doesn't access all
the wrong places.

Put all sysproxy routines to their own C module, sysproxy.c

branches: 1.168.2;
delay() can't use rumpuser_clock_sleep(), because that may unschedule().

Instead, busyloop with calls to rumpuser_clock_gettime(). Fixing delay()
properly should addressed in hypercall rev.18 by introducing a nowrap
variant/flag for rumpuser_clock_sleep().

Attempt to make sure that DELAY()/delay() is available in all MD names.

Add missing __diagused in rump code

cpu_reboot() is more of an emul.c thing than a rump.c thing

branches: 1.164.2;
Put the syncobjs in emul.c instead of locks.c so that they are
present for both locks.c and locks_up.c alike.

+ resettodr

lwp0 needs l_proc set for panic->cpu_reboot to dump core the right way

Use subr_cprng.c instead of stub implementation. Rijndael migrates from
rumpkern_crypto to rumpkern due to it being mandatory for cprng.

Translate return values for emulations, e.g. Linux. For ports without
__HAVE_MINIMAL_EMUL, we simply look up the values from p->p_emul->e_errno.
For ports which cannot afford to keep an extra pointer per emul structure
around, we hope there is __HAVE_SYSCALL_INTERN support and thread the
errno values through p_emuldata. Notably, we cannot alter the syscall
method like most ports do with syscall_intern, since they do it via
p_mdproc, so MI code is not possible there.

Support ktrace for rump kernels.

Requested by Justin Cormack on rumpkernel-users.

branches: 1.158.4;
Flip parameter order in the clock hypercalls to make them consistent
with everything else wrt in/out parameters.

Make hypercall calling conventions consistent: iff a hypercall can fail,
it returns an int containing the error value.

Improve the time-related hypercalls so that's it's possible to
sleep until an absolute time on the host's monotonic clock (should
something like that be supported).

calculate vnode cache size based on the resource it gets allocated from
this stops setting kern.maxvnodes to high so it exhausts available space in kmem

http://mail-index.netbsd.org/tech-kern/2013/03/08/msg015095.html

set e_nsysent

Stopgap fix to make rump cooperate with pserialize, may be revisited later.
Patch from pooka, ok: rmind. No related regressions in a complete atf test
run (which works again with this, even on non x86 SMP machines).

Provide xc_send_ipi() routine in RUMP, which is required for high-priority
xcall(9) mechanism. It is emulated using low-priority xcall(9).

split device_t/softc for all remaining drivers.
replace "struct device *" with "device_t".
use device_xname(), device_unit(), etc.

branches: 1.150.4; 1.150.10; 1.150.14;
Update copyright statements.

no functional change.

Since "physmem" is largely unused except for zfs wanting to know
if it has enough memory available, just pick a number which makes
zfs happy.

We *could* use a MIN() of available host mem and rump_memlimit,
though ...

branches: 1.148.2; 1.148.4;
Support LOCKDEBUG. To use it, compile sys/rump with RUMP_LOCKDEBUG=yes.

requested by martin (sparc64 gdb cannot reliably produce a stack trace)

Realize the >1yo comment above rump_reboot and retire them to make
room for sys_reboot.

Make common kernel module binaries work on both sun3 and sun3x.
Tested on 3/160 (on TME) and (real) 3/80.

XXX: module files can be loaded only on single user?

Implement rump_lwproc: the new lwp/proc management routines for
rump. These move the management of the pid/lwpid space from the
application into the kernel, make code more robust, and make it
possible to attach multiple lwp's to non-proc0 processes.

As normal, fix breakage from untested commits by rmind.

Remove comment which wasn't supposed to get committed.

Make some stubby tty routines used by printf weak aliases, so that
the full ones will be used if we have the upcoming tty component
loaded.

Fix previous in emul.c -- only numbers are operands for cpp comparisons.
Apparently non-numbers logically produce arch-dependent behaviour.

deal with sun3's module disability

add rest of sparc pagesize variables

Use kern_proc.c instead of a collection of stubs. But what we
really wanted from this commit was the support for proc_specificdata.

TODO: make creating a new process actually use kern_proc and
maybe even add an interface which starts a process with
"any pid you don't like"

adjust comment in previous.
XXX: should make that (and physmem) mean something here

remove unnecessary #ifdef

Now that we have a big devsw_conv0, need at least as many entries
in devsw as the highest device number in the conv table. Do a
"good for a hundred years" guess now, fix properly later.

Include devsw_conv0 from an i386 kernel compilation (no, we don't
care about the arch as long as all the devices we care about are
present). The file should be autogenerated, but that requires some
more changes to config(1).

Make initial devsw a little more believable. Especially adjust
max_sys_devsw, since it was previously a bit too much, given that
the correct value is 0.

remove stuff which now comes via std. kernel source modules

support kern_resource

Move all signal-related from emul.c to signals.c. Additionally,
define a few alternate signal models for the rump kernel, including
ones where signals are ignored or sent to host processes.

no \n in panic

One emul is enough and since we need emul_netbsd, retire emul_rump.

Move scheduling related routines from emul.c to scheduler.c

Use kern_syscall.c instead of homegrown syscall dis/establishment routines.

Include kern_tc and use a timecounter driver instead of homerolled
kern_tc implementation.

We don't have a real rootdev (by default at least), so set it to NODEV.

branches: 1.123.2;
add signal stubs required by mfs

Remove separate mb_map. The nmbclusters is computed at boot time based
on the amount of physical memory and limited by NMBCLUSTERS if present.
Architectures without direct mapping also limit it based on the kmem_map
size, which is used as backing store. On i386 and ARM, the maximum KVA
used for mbuf clusters is limited to 64MB by default.

The old default limits and limits based on GATEWAY have been removed.
key_registered_sb_max is hard-wired to a value derived from 2048
clusters.

branches: 1.121.2;
Include newly-created subr_device.c and remove few special case
device accessor copypastes. This makes it possible to link static
binaries which use -lrumpdev.

include kern_hook.c

Use subr_percpu.c instead of homegrown implementation. ...except
when using malloc(3)-relegated allocators (happens in production
e.g. on Linux), since subr_percpu.c uses vmem and i don't want to
reimplement vmem.

Minimize unnecessary differences in rump.

Add a few symbols used by the tty code.

Include kern_pmf.c in rumpdev.

Expand PMF_FN_* macros.

allow rump to build

Generate vers.c and include it in the kernel component.

Decide it's not worth the fuss to have rumpfs as a module and just
hardcode attach into rump_vfs_init(). Saves us from a lot of
pingpong init bouncing from one component to another to get the order
right.

Include cpu crosscall support (instead of stubbing it out).

For rumpfs, do mountroot instead of the bunch of homegrown hacks
currently there. Still needs a little massage to get the kernel
interfaces right and avoid copypaste especially from main().

Also, move it a bit more into the direction of a real file system
(finally!) by giving it a vfsops. Most ops are still unimplemented,
though.

misc_stub and emul have been the same thing for a looong time now,
so just move the few remaining routines in misc_stub to emul.

Give the kthread->pthread interface emulation its own module.

Pull all relegating memory allocators under a common roof in memalloc.c

move copy-related routines to their own module

Use std. uiomove() & friends.

Sync rump with kernel changes.

Include sys_select.c for proper select()/poll() support.

Give lwp usage some much-needed love: stop treating lwp0 as the
all-sink and make sure each separate thread in rump has its own
lwp. Happy-go-lucky callers will get scheduled a temporary lwp
on entry, while true lwp connoisseurs may request a stable lwp
for their purposes. Some more love may be required later down the
road, but for now different threads will stepping on each others
toes.

Add initial work on a rump virtual cpu scheduler. This is necessary
for kernel code which has been written to avoid MP contention by
using cpu-local storage (most prominently, select and pool_cache).

Instead of always assuming rump_cpu, the scheduler must now be run
(and unrun) on all entry points into rump. Likewise, rumpuser
unruns and re-runs the scheduler around each potentially blocking
operation. As an optimization, I modified some locking primitives
to try to get the lock without blocking before releasing the cpu.

Also, ltsleep was modified to assume that it is never called without
the biglock held and made to use the biglock as the sleep interlock.
Otherwise there is just too much drama with deadlocks. If some
kernel code wants to call ltsleep without the biglock, then, *snif*,
it's no longer supported and rump and should be modified to support
newstyle locks anyway.

"rumppriv" goes back to "rump" per internal interface naming change.

Adjust rump sources for external/internal interfaces.
No functional change.

Provide an interface for reboot.

Need proc_uidmatch

remove a no longer necessary hack

default to AB_SILENT

provide pmf stubs

move hppa cpp kludge to the end of the file

add a few global symbols required by kernel code

realloc works much better if it actually uses realloc instead of malloc

Nuke some bss stuff which is coming from vfs_bio via rumpvfs.

Don't create unpgc thread if RUMP_THREADS is 0. The benefit is the
usual: possible to run programs using AF_LOCAL without threads.

Add proof-of-concept code for enabling system calls to rump virtual
kernels running in other processes on the same machine or on an
entirely different host. I wrote this a while ago and am now
committing it mainly to avoid losing it. It works, but could do
with a little tuning here and there.

What this will hopefully eventually buy us is the ability to use
standard userland tools to configure rump kernels, e.g. ifconfig(8)
and route(8) could be used to configure the networking stack provided
by a rump kernel. Also some distributed OS implications may apply.

fun fact: a system call which just does copyin/copyout takes >1000x
longer when made over the LAN as compared to when made on the same
machine.

Fix getnano/microuptime to report actual uptime.

Catch up with recent changes to sys/proc.h (Hello, rmind!)

we want microuptime too.

need getnanouptime not getmicrouptime

one more timeval -> timespec line

Catch up with kernel changes

Support kqueue in the rump virtual kernel.

Ansify function definitions w/o arguments. Generated with sed.

decouple rumpuser gettime from struct timeval

Make rumpuser stat and nanosleep independent of the host system
stat and timespec definitions.

branches: 1.77.2;
Make the clock interrupt tick based on absolute time instead of
relative time. This prevents drifting. Also, keep track of time
within userspace, so we do not have to make a syscall to get the
clock value. This is approximately 7 times cheaper, but on the
negative side is limited to the clock interrupt frequency.

Implement block I/O as a real block driver instead of a hacked copy
of specfs. That was easier years ago when rump didn't support
devices, but brings no gain now. This allows us to include the
real specfs in rump.

Shove in some hppa love to help with spcopy.S (why does that need
to be in libkern?).

Protect against nfssilly in RUMP_THREADS=0

merge christos-time_t

provide kern_realloc

Rename malloc() to kern_malloc() to avoid name conflict with libc.
Now it is possible to use the kernel linker with rump.

Include libkern contents in librump.

Sauce with some kludges:
* revert rev 1.68 of emul.c: delay() is hopelessly MD and needs
whipping (not the fun kind)
* #undef DELAY in cpu.h

This will hopefully fix build for all archs and give me time to
hop through our 46521 fine archs and edit the headers.

Export delay, _delay and delay_func symbols to attempt to appease
the macro mania set over all architectures.

Include kernel printf routines in rump instead of relying on the
magic libc symbol. This also allows to bid farewell to subr_prf2.c
and merge the contents back to subr_prf.c. The host kernel bridging
is now done via rumpuser_putchar().

Define MODULAR for rump core components. This enables module
loading via the kernel module framework (instead of dlopen()).
For now it only works on amd64 and i386, but for the rest it should
just be a matter of including the relevant kobj_machdep.c modules
from the kernel sources.

properly handle kthread_exit()

include subr_devsw in rumpkern

branches: 1.63.2;
undo turn malloc flags to enums.
Requested by christos.

Keep malloc flag defines in kernel section.

Turn malloc flags into an enum. Fixes build breakage.

__KERNEL_RCSID

If available (__NetBSD__), use pthread_setname_np() to set the
thread name for kthread_create().

If the arch uses __BSWAP_RENAME, provide non-namespaced kernel
symbols which just call the libc symbols. Allows to get rid of
rump machine/bswap.h.

XXX:
1) this is unlikely to work on non-NetBSD
==>
2) should look into including libkern in librump

Make non-mpsafe threads KERNEL_LOCK() before starting execution.

Remove the whole magic ubc window thingie. All file systems use
ubc_uiomove() now, so we can hook ourselves there.

Add a few symbols required by nfsd

Add more verbose commentage to the module thread non-creation.

Ignore the module unload thread.

branches: 1.53.2; 1.53.4; 1.53.8;
Add some more trivial emulations.

Provide psignal() so that we don't get linked against psignal(3).
As there is no real process model currently, just "trap" the signals
appropriately.

Support callouts and call callout_hardclock() from the timer
interrupt thread.

The sleepq implementation required for callouts is horrible, kludged
only for callouts, and generally unacceptable. It needs revisiting,
but I'm not sure yet should rump or kern_timeout be improved. It's
almost untested as of now, but committing this will give me some
maneuvering space while letting application compile.

Remove rumpuser_yield(). Not only doesn't it really make sense
here, some kind soul made it completely empty.

Switch to std kern_auth.

Move global malloc types from kern_malloc into a separate module.

Make it possible to control starting of threads per env variable
instead of only at compile-time.

Add support for using real kmem/vmem. Don't enable it by default,
though, since it a) is a lot of unnecessary indirection in rump
b) requires callouts which are so far unimplemented.

support real sysctls

Install rump libraries and utilities to the base system and remove the
private non-installed build infrastructure from sys/rump.

breakdown of commit:
* install relevant headers into /usr/include/rump
* build sys/rump/librump/rumpuser and sys/rump/librump/rumpkern
from src/lib and install as librumpuser and librump, respectively
+ this retains the ability to test a librump build with just the
kernel sources at hand
* move sys/rump/fs/lib/libukfs and sys/rump/fs/lib/libp2k to src/lib
for general consumption, they are not kernel-space dwellers anyway
* build and install sys/rump/fs/lib/lib$fs as librumpfs_$fs
* add chapter 3 manual pages for rump, rumpuser, ukfs and p2k
* build and install userspace kernel file system daemons if MKPUFFS=yes
is spexified
* retire fsconsole for now, it will make a comeback with an actually
implemented version shortly

Repeat after me: do not pollute sys with #ifdef _RUMPKERNEL

emulate vlog()

branches: 1.41.2;
Don't compile kern_lock for rump any more, it's no longer required.
Allows us to get rid of the incorrect _RUMPKERNEL ifdefs outside sys/rump.

Make kpause() use nanosleep() instead of usleep(). Fixes >=1s sleeps
to actually sleep a bit too.

from Arnaud Ysmal

branches: 1.39.2;
Also fake the namecache g/c thread in kthread_create() so that file
systems can again be run without threads (and hence gdb can be used
on them).

branches: 1.38.2; 1.38.4;
Catch up with process locking changes.

branches: 1.37.2;
Add a device_xname() stub.

- for some ports, especially for ones without pmap_growkernel,
buf_memcalc is used by bootstrap as well. fix NULL dereference for them.
- limit kva usage for each cache to 20% of vm_map. XXX a bit arbitrary.
- add a comment.

Adapt to sel* changes

when calculating some cache sizes, consider the amount of available kva.
PR/33185.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Add lwp_unsleep from kern/kern_lwp.c so that this builds. ok ad@

Support multiple file systems within one process with ukfs by using
a "chroot" for each file system.

Backup some fixes for recent breakage from local tree. Also some
other improvements such as exporting the real kernel namei and
using that in ukfs instead of the homegrown heap'o hacks namei.
"etcetc".

branches: 1.29.2; 1.29.6;
Implement yield().

Add dummy hardclock_ticks.

Use vfs_subr.c from sys/kern. This brings differences in the vnode
life cycle between rump and a real kernel to a minimum.

Use namei() etc. from kernel sources instead of a reimplementation.
To accommodate, give the rootvnode its own vnode op vector with a
simple lookup operation. This is used for looking up the file
system's device vnode instead of doing that directly in a homesmoked
namei().

Don't use PAGE_SIZE when faking physmem, as it's not always available
at compile-time. We could init it runtime, but it's just a random
number anyway.

Default physmem was too tight, increase to 256megs from 0 bytes.

Until debugging threaded programs in NetBSD is fixed, supply the
cpp option RUMP_WITHOUT_THREADS as a workaround. If defined, it
makes rump itself operate single-threaded and prevents kthread_create()
from working.

Now that __HAVE_TIMECOUNTER and __HAVE_GENERIC_TODR are invariants,
remove the conditionals and the code associated with the undef case.

Print "panic: " before panicstr when panicking.

Add the ability to run puffs in userspace. This means that puffs
can now be developed in userspace using puffs for development
(I hate emulators, they are annoyingly clumsy).

To e.g. mount psshfs using puffs-on-puffs, run fs/bin/syspuffs/syspuffs
with the regular mount_psshfs command line as an argument:

golem> ./syspuffs /usr/sbin/mount_psshfs ftp.netbsd.org:/pub /puffs

This will make the mount appear as usual, with the exception that the
requests will be passed through puffs both in the kernel and userspace:

ftp.netbsd.org:/pub on /puffs type puffs|p2k|puffs|psshfs

branches: 1.19.6;
fix build (hi rmind!)

branches: 1.18.2;
* sprinkle some locking into the vm code
* avoid extra insert+search+remove step in file systems using
ubc_uiomove() instead of standard uiomove()

Make it possible to run rumps multithreaded. This brings real
locking and makes it possible to run file systems which create
threads. It also makes rump file system behaviour better match
file system behaviour in the kernel.

branches: 1.16.2;
"flags * (M_CANFAIL | M_NOWAIT)" is probably not correct. Use & instead.
And while committing, add other random cruft I've needed recently.

machine/{bus,cpu,intr}.h -> sys/{bus,cpu,intr}.h

branches: 1.14.2;
Don't whine about wakeup() not being implemented every time it's
called. Nobody cares unless ltsleep() is called (and it panics).

branches: 1.13.2;
If !_HAVE_TIMECOUNTER, initialize 'time' instead of 'time_second.'

Provide errno from rumpuser_{gettimeofday,close}() to be consistent.

branches: 1.11.2; 1.11.4;
few panicky functions

branches: 1.10.2;
Hide NetBSD kernel headers completely from ukfs. This includes creating
accessors for:
* struct mount & VFS ops
* struct uio
* struct vnode
* struct vattr

and some namespace games for:
* namei flags
* VOPs
* enum vtype

Also, split rump services into two categories: library private and public
(rump_private.h and rump.h, respectively).

As a result, it is now possible to compile and use the NetBSD kernel
file systems on Linux (and probably other systems too with very
little work), although the makefiles need a bit of work to make it
a pleasureable experience.

Wrap malloc() so that we catch the kernel arguments (namely M_ZERO)
properly. It's fairly amusing that this wasn't noticed until now.

branches: 1.8.2;
Kill handrolled buffercache and use vfs_bio from the kernel. This is
mostly to get the flag jungle in sync with the kernel.

Use supermarket variety vfs_vnops.c instead of homecooked routines
(that's actually a good thing here).

compile in libkern __assert.c, argument order differs from libc model

Add some stubs for lfs.

Provide rumpmachine bswap.h, which makes bswapxx() call rumpuser
directly instead of relying on a symbol in rumpkern. I would like
to make it call the libc symbol directly, but I don't currently know
how to make it do that MI.

Makes hfs work (on i386), as it avoids endless recursion in bswap64().
Thanks to dillo for the image!

* compile param.c
* move opts to a separate directory

add microtime

branches: 1.1.2;
Introduce RUMPs - Runnable Userspace Meta-Programs

/sys/rump contains programs which run unmodified kernel code in an
emulated userspace environment. The kernel environment is provided
by librump. Currently supported are a number of file systems,
which by using puffs integrate seamlessly into the system and
provide a similar user experience to if the code was running as
part of the kernel. Potential future rumpification targets include
for example parts of the networking stack and some device drivers.

This work was supported by Google Summer of Code 2007.

rumpkern: Provide stub pnbuf_cache weak alias.

Needed only by sys_module.c handle_modctl_load, which won't work if
there's no rumpvfs linked in anyway.

rumpkern: Provide fs_filtops weak alias.

We should introduce a link set for sys_kfilters, but this works
around the problem for now -- librump's dependency on librumpvfs
symbols.

nkmempages should be size_t

Rewrite entropy subsystem.

Primary goals:

1. Use cryptography primitives designed and vetted by cryptographers.
2. Be honest about entropy estimation.
3. Propagate full entropy as soon as possible.
4. Simplify the APIs.
5. Reduce overhead of rnd_add_data and cprng_strong.
6. Reduce side channels of HWRNG data and human input sources.
7. Improve visibility of operation with sysctl and event counters.

Caveat: rngtest is no longer used generically for RND_TYPE_RNG
rndsources. Hardware RNG devices should have hardware-specific
health tests. For example, checking for two repeated 256-bit outputs
works to detect AMD's 2019 RDRAND bug. Not all hardware RNGs are
necessarily designed to produce exactly uniform output.

ENTROPY POOL

- A Keccak sponge, with test vectors, replaces the old LFSR/SHA-1
kludge as the cryptographic primitive.

- `Entropy depletion' is available for testing purposes with a sysctl
knob kern.entropy.depletion; otherwise it is disabled, and once the
system reaches full entropy it is assumed to stay there as far as
modern cryptography is concerned.

- No `entropy estimation' based on sample values. Such `entropy
estimation' is a contradiction in terms, dishonest to users, and a
potential source of side channels. It is the responsibility of the
driver author to study the entropy of the process that generates
the samples.

- Per-CPU gathering pools avoid contention on a global queue.

- Entropy is occasionally consolidated into global pool -- as soon as
it's ready, if we've never reached full entropy, and with a rate
limit afterward. Operators can force consolidation now by running
sysctl -w kern.entropy.consolidate=1.

- rndsink(9) API has been replaced by an epoch counter which changes
whenever entropy is consolidated into the global pool.
. Usage: Cache entropy_epoch() when you seed. If entropy_epoch()
has changed when you're about to use whatever you seeded, reseed.
. Epoch is never zero, so initialize cache to 0 if you want to reseed
on first use.
. Epoch is -1 iff we have never reached full entropy -- in other
words, the old rnd_initial_entropy is (entropy_epoch() != -1) --
but it is better if you check for changes rather than for -1, so
that if the system estimated its own entropy incorrectly, entropy
consolidation has the opportunity to prevent future compromise.

- Sysctls and event counters provide operator visibility into what's
happening:
. kern.entropy.needed - bits of entropy short of full entropy
. kern.entropy.pending - bits known to be pending in per-CPU pools,
can be consolidated with sysctl -w kern.entropy.consolidate=1
. kern.entropy.epoch - number of times consolidation has happened,
never 0, and -1 iff we have never reached full entropy

CPRNG_STRONG

- A cprng_strong instance is now a collection of per-CPU NIST
Hash_DRBGs. There are only two in the system: user_cprng for
/dev/urandom and sysctl kern.?random, and kern_cprng for kernel
users which may need to operate in interrupt context up to IPL_VM.

(Calling cprng_strong in interrupt context does not strike me as a
particularly good idea, so I added an event counter to see whether
anything actually does.)

- Event counters provide operator visibility into when reseeding
happens.

INTEL RDRAND/RDSEED, VIA C3 RNG (CPU_RNG)

- Unwired for now; will be rewired in a subsequent commit.

- Eliminate the global "boottime" variable, which was being accessed
without any synchronization against changes by e.g. clock_settime().
- Replace with new getbinboottime() / getnanoboottime() / getmicroboottime()
functions (naming mirrors that of other time access functions in kern_tc.c).
It returns the (maybe-converted) value of timebasebin, which also tracks
our estimate of when the system was booted (i.e. the legacy "boottime" was
redundant).

XXX There needs to be a lockless synchronization mechanism for reading
timebasebin, but this is a problem in kern_tc.c that pre-existed these
"boottime" changes. At least now the problem is centralized in one location.

Add shutting_down variable for rump.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

Provide a weak alias for vnode_to_path to be used unless librumpvfs is present.

Apply a patch from hannken@ which adds a weak alias for rump_getcwd_common()
allowing -lrump to be used without -lrumpvfs.

This is an alternate fix to the earluer one which added -lrumvfs to
many rump based tests (and the rump server) which might be undone soon.

This also fixes the sun2 build.

Rumpkernel has its own thread deallocation. Add missing fstrans_lwp_dtor()
to lwproc_freelwp().

PR bin/50350: rump/rumpkern/t_sp/stress_{long,short} fail on Core 2 Quad

no more need for get_expose_address() here.

add get_expose_address()

Bring back three state file system suspension:

NORMAL -> SUSPENDING -> SUSPENDED

and add operation fstrans_start_lazy() that only blocks while SUSPENDED.

Change vndthread() support operation handle_with_rdwr() to bracket
its file system operations by fstrans_start_lazy() and fstrans_done().

PR kern/53624 (dom0 freeze on domU exit)

Allow syscall_establish() to install new syscalls when the existing
entry-point is either sys_nomodule or sys_nosys. Update the
makesyscalls.sh script to create a const array of bits to allow
syscall_disestablish() to properly restore the original entry-point.
Update all the initializers of struct emul to initialize the pointer
to the bit array struct emul.

XXX Regen of all files created by makesyscalls.sh will come soon,
XXX followed by a kernel version bump (since struct emul is being
XXX modified).

This commit should address PR kern/45781 and also removes the need
for the work-around for that PR in file

sys/arch/usermode/modules/syscallemu/syscallemu.c

branches: 1.185.2; 1.185.4;
Add missing inclusion of pserialize.h (fix build)

Implement debugging feature for pserialize(9)

The debugging feature detects violations of pserialize constraints.
It causes a panic:
- if a context switch happens in a read section, or
- if a sleepable function is called in a read section.

The feature is enabled only if LOCKDEBUG is on.

Discussed on tech-kern@

added booted_method

Operations fstrans_start() and fstrans_start_nowait() now always
use FSTRANS_SHARED as lock type so remove the lock type argument.

File system state FSTRANS_SUSPENDING is now unused so remove it.

Regen vnode_if files.

Ride 8.99.1 less than a hour ago.