Revert previous until I can diagnose a failure reported by gson.

Early in main(), assert that curcpu() evaluates as the primary CPU and
stash away a pointer to it as the boot CPU for quick reference later.

Protect kernel hooks exechook, exithook and forkhook with rwlock.
Lock as writer on establish/disestablish and as reader on list traverse.

For exechook ride "exec_lock" as it is already take as reader when
traversing the list. Add local locks for exithook and forkhook.

Move exec_init before signal_init as signal_init calls exechook_establish()
that needs "exec_lock".

PR kern/39913 "exec, fork, exit hooks need locking"

Repply this change with a couple of bugs fixed:

- Do away with separate pool_cache for some kernel objects that have no special
requirements and use the general purpose allocator instead. On one of my
test systems this makes for a small (~1%) but repeatable reduction in system
time during builds presumably because it decreases the kernel's cache /
memory bandwidth footprint a little.
- vfs_lockf: cache a pointer to the uidinfo and put mutex in the data segment.

Back out recent change to replace pool_cache with then general allocator.
Will return to this when I have time again.

- Do away with separate pool_cache for some kernel objects that have no special
requirements and use the general purpose allocator instead. On one of my
test systems this makes for a small (~1%) but repeatable reduction in system
time during builds presumably because it decreases the kernel's cache /
memory bandwidth footprint a little.
- vfs_lockf: cache a pointer to the uidinfo and put mutex in the data segment.

heartbeat(9): Move #ifdef HEARTBEAT to sys/heartbeat.h.

Less error-prone this way, and the callers are less cluttered.

heartbeat(9): New mechanism to check progress of kernel.

This uses hard interrupts to check progress of low-priority soft
interrupts, and one CPU to check progress of another CPU.

If no progress has been made after a configurable number of seconds
(kern.heartbeat.max_period, default 15), then the system panics --
preferably on the CPU that is stuck so we get a stack trace in dmesg
of where it was stuck, but if the stuckness was detected by another
CPU and the stuck CPU doesn't acknowledge the request to panic within
one second, the detecting CPU panics instead.

This doesn't supplant hardware watchdog timers. It is possible for
hard interrupts to be stuck on all CPUs for some reason too; in that
case heartbeat(9) has no opportunity to complete.

Downside: heartbeat(9) relies on hardclock to run at a reasonably
consistent rate, which might cause trouble for the glorious tickless
future. However, it could be adapted to take a parameter for an
approximate number of units that have elapsed since the last call on
the current CPU, rather than treating that as a constant 1.

XXX kernel revbump -- changes struct cpu_info layout

kern/init_main.c: Get extern lwp0 from sys/lwp.h.

Removed unused opt_wapbl.h include.

fix typos in word "functions" in comments, mainly s/fuctions/functions/.

Fix locking after opendisk(), VOP_IOCTL() needs an unlocked vnode,
vn_rdwr() needs flag IO_NODELOCKED.

entropy(9): Establish the softint a little earlier.

Just need to wait until softint_establish and high-priority xcalls
will work, no later than that. Doing this earlier gives us slightly
more of a chance to ensure cprng_fast and ssp get entropy from
hardware RNG devices that rely on interrupts.

remove double t from targeted, add missing r to arbitrary
And fix few more typos along the way in comments and man pages.

Expose olde style intrcnt interrupt accounting via event counters.
This code will be garbage collected once our last legacy intrcnt
user is update to native evcnts.

branches: 1.534.2;
Refactor interval timers to make it possible to support types other than
the BSD/POSIX per-process timers:

- "struct ptimer" is split into "struct itimer" (common interval timer
data) and "struct ptimer" (per-process timer data, which contains a
"struct itimer").

- Introduce a new "struct itimer_ops" that supplies information about
the specific kind of interval timer, including it's processing
queue, the softint handle used to schedule processing, the function
to call when the timer fires (which adds it to the queue), and an
optional function to call when the CLOCK_REALTIME clock is changed by
a call to clock_settime() or settimeofday().

- Rename some fuctions to clearly identify what they're operating on
(ptimer vs itimer).

- Use kmem(9) to allocate ptimer-related structures, rather than having
dedicated pools for them.

Welcome to NetBSD 9.99.77.

Set a better default for MAXFILES on larger RAM machines if not
otherwise specified the kernel config file. Arbitary numbers are
20,000 files for 16GB RAM or more and 10,000 files for 1GB RAM or
more.

TODO: Adjust this and other values totally dynamically.

In uvmpd_tryownerlock(), if the initial try-lock of the owner lock fails
then rather than do more try-locks and eventually sleep for a tick,
take a hold on the current owner's lock, drop the page interlock,
and acquire the lock that we took the hold on in a blocking fashion.
After we get the lock, check if the lock that we acquired is still
the lock for the owner of the page that we're interested in.
If the owner hasn't changed then can proceed with this page,
otherwise we will skip this page and move on to a different page.
This dramatically reduces the amount of time that the pagedaemon
sleeps trying to get locks, since even 1 tick is an eternity to sleep
in this context and it was easy to trigger that case in practice,
and with this new method the pagedaemon only very rarely actually blocks
to acquire the lock that it wants since the object locks are adaptive,
and when the pagedaemon does block then the amount of time it spends
sleeping will be generally be much less than 1 tick.

branches: 1.531.2;
ipi: Split up initialization into two parts.

First part runs early so ipi_register can be used in module
initialization, e.g. via pktqueue_create; second part runs after CPUs
have been detected.

Add the ability to set an alternate cnmagic in the kernel config
file, e.g.:

options CNMAGIC="\"+++++\""

Move address hashing from init_main.c to kern_sysctl.c.

This way rump gets it automatically. Make sure blake2s is in
librumpkern.so, not just in librumpkern_crypto.so, for this to work.

Instead of returning 0 when sysctl kern.expose_address=0, return a random
hashed value of the data. This allows sockstat to work without exposing
kernel addresses or being setgid kmem.

uvm_availmem(): give it a boolean argument to specify whether a recent
cached value will do, or if the very latest total must be fetched. It can
be called thousands of times a second and fetching the totals impacts not
only the calling LWP but other CPUs doing unrelated activity in the VM
system.

Move proc_lock into the data segment. It was dynamically allocated because
at the time we had mutex_obj_alloc() but not __cacheline_aligned.

Move cprng_init before configure.

This makes it available to device drivers, e.g. to generate MAC
addresses at random, without initialization order hacks.

Requires a minor initialization hack for cpu_name(primary cpu) early
on, since that doesn't get set until mi_cpu_attach which may not run
until the middle of configure. But this hack is less bad than other
initialization order hacks.

Rewrite entropy subsystem.

Primary goals:

1. Use cryptography primitives designed and vetted by cryptographers.
2. Be honest about entropy estimation.
3. Propagate full entropy as soon as possible.
4. Simplify the APIs.
5. Reduce overhead of rnd_add_data and cprng_strong.
6. Reduce side channels of HWRNG data and human input sources.
7. Improve visibility of operation with sysctl and event counters.

Caveat: rngtest is no longer used generically for RND_TYPE_RNG
rndsources. Hardware RNG devices should have hardware-specific
health tests. For example, checking for two repeated 256-bit outputs
works to detect AMD's 2019 RDRAND bug. Not all hardware RNGs are
necessarily designed to produce exactly uniform output.

ENTROPY POOL

- A Keccak sponge, with test vectors, replaces the old LFSR/SHA-1
kludge as the cryptographic primitive.

- `Entropy depletion' is available for testing purposes with a sysctl
knob kern.entropy.depletion; otherwise it is disabled, and once the
system reaches full entropy it is assumed to stay there as far as
modern cryptography is concerned.

- No `entropy estimation' based on sample values. Such `entropy
estimation' is a contradiction in terms, dishonest to users, and a
potential source of side channels. It is the responsibility of the
driver author to study the entropy of the process that generates
the samples.

- Per-CPU gathering pools avoid contention on a global queue.

- Entropy is occasionally consolidated into global pool -- as soon as
it's ready, if we've never reached full entropy, and with a rate
limit afterward. Operators can force consolidation now by running
sysctl -w kern.entropy.consolidate=1.

- rndsink(9) API has been replaced by an epoch counter which changes
whenever entropy is consolidated into the global pool.
. Usage: Cache entropy_epoch() when you seed. If entropy_epoch()
has changed when you're about to use whatever you seeded, reseed.
. Epoch is never zero, so initialize cache to 0 if you want to reseed
on first use.
. Epoch is -1 iff we have never reached full entropy -- in other
words, the old rnd_initial_entropy is (entropy_epoch() != -1) --
but it is better if you check for changes rather than for -1, so
that if the system estimated its own entropy incorrectly, entropy
consolidation has the opportunity to prevent future compromise.

- Sysctls and event counters provide operator visibility into what's
happening:
. kern.entropy.needed - bits of entropy short of full entropy
. kern.entropy.pending - bits known to be pending in per-CPU pools,
can be consolidated with sysctl -w kern.entropy.consolidate=1
. kern.entropy.epoch - number of times consolidation has happened,
never 0, and -1 iff we have never reached full entropy

CPRNG_STRONG

- A cprng_strong instance is now a collection of per-CPU NIST
Hash_DRBGs. There are only two in the system: user_cprng for
/dev/urandom and sysctl kern.?random, and kern_cprng for kernel
users which may need to operate in interrupt context up to IPL_VM.

(Calling cprng_strong in interrupt context does not strike me as a
particularly good idea, so I added an event counter to see whether
anything actually does.)

- Event counters provide operator visibility into when reseeding
happens.

INTEL RDRAND/RDSEED, VIA C3 RNG (CPU_RNG)

- Unwired for now; will be rewired in a subsequent commit.

Add a NetBSD native futex implementation, mostly written by riastradh@.
Map the COMPAT_LINUX futex calls to the native ones.

move config_init_mi() call before vfsinit(), which can trigger loading
of VFS modules

fixes crash with LOCKDEBUG due to uninitialized mutex when zfs
module is loaded in boot, because zfs's spa_init() calls config_mountroot()
which now requires the config init having been done

remove the aiodoned thread. I originally added this to provide a thread context
for doing page cache iodone work, but since then biodone() has changed to
hand off all iodone work to a softint thread, so we no longer need the
special-purpose aiodoned thread.

- Move the LW_RUNNING flag back into l_pflag: updating l_flag without lock
in softint_dispatch() is risky. May help with the "softint screwup"
panic.

- Correct the memory barriers around zombies switching into oblivion.

Call radix_tree_init() earlier, so more stuff can make use of radixtree.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.517.2;
- Eliminate the global "boottime" variable, which was being accessed
without any synchronization against changes by e.g. clock_settime().
- Replace with new getbinboottime() / getnanoboottime() / getmicroboottime()
functions (naming mirrors that of other time access functions in kern_tc.c).
It returns the (maybe-converted) value of timebasebin, which also tracks
our estimate of when the system was booted (i.e. the legacy "boottime" was
redundant).

XXX There needs to be a lockless synchronization mechanism for reading
timebasebin, but this is a problem in kern_tc.c that pre-existed these
"boottime" changes. At least now the problem is centralized in one location.

- Introduce a new global kernel variable "shutting_down" to indicate that
the system is shutting down or rebooting.
- Set this global in a new function called kern_reboot(), which is currently
just a basic wrapper around cpu_reboot().
- Call kern_reboot() instead of cpu_reboot() almost everywhere; a few
places remain where it's still called directly, but those are in early
pre-main() machdep locations.

Eventually, all of the various cpu_reboot() functions should be re-factored
and common functionality moved to kern_reboot(), but that's for another day.

First steps towards properly serializing access to the TOD clock.
- Add a mutex around the TODR, and provide lock/unlock/lock-owned
functions to manipulate it.
- Rename inittodr() to todr_set_systime() and resettodr() to
todr_save_systime() to better reflect what they do. These functions
are intended to be called with the TODR lock held, which will allow
for a pattern like:
-> todr_lock()
-> todr_save_systime()
-> [do machine-dependent stuff to sleep/suspend]
-> [magically awaken]
-> todr_set_systime(...)
-> todr_unlock()
- Provide historically-named wrappers inittodr() and resettodr() that
do the dance of acquiring / releasing the lock around the actual
substance.

NOTE: resettodr()'s use of the TODR lock is currently disabled (and
todr_save_systime() does not assert it's held) until such time as
issues around shutdown / reboot under duress can be addressed.

Rename uvm_free() -> uvm_availmem().

Redo the page allocator to perform better, especially on multi-core and
multi-socket systems. Proposed on tech-kern. While here:

- add rudimentary NUMA support - needs more work.
- remove now unused "listq" from vm_page.

Fix integer overflow when printing available memory size (resulting from
a cast lost during merges).

Reported-by: syzbot+f02ca5f83ac7196b8afd@syzkaller.appspotmail.com

uvmexp.free -> uvm_free()

Include radixtree in the kernel.

Eliminate per-hook duplication of common code as suggested by
(and with major contributions from) riastradh@

Welcome to 9.99.23

Take the basic CPU topology information we already collect, and use it
to make circular lists of CPU siblings in the same core, and in the
same package. Nothing fancy, just enough to have a bit of fun in the
scheduler trying out different tactics.

Init kern_runq and kern_synch before booting secondary CPUs.

Remove compile-time asserts checking whether intptr_t and void* are compat

The checks were requested by core@ as a prerequisite for kevent::udata type
switch from intptr_t to void*.

Add a temporary ctassert checking whether void* and intptr_t are compatible

branches: 1.504.2;
Implement an aggressive psref leak detector

It is yet another psref leak detector that enables to tell where a leak occurs
while a simpler version that is already committed just tells an occurrence of a
leak.

Investigating of psref leaks is hard because once a leak occurs a percpu list of
psref that tracks references can be corrupted. A reference to a tracking object
is memorized in the list via an intermediate object (struct psref) that is
normally allocated on a stack of a thread. Thus, the intermediate object can be
overwritten on a leak resulting in corruption of the list.

The tracker makes a shadow entry to an intermediate object and stores some hints
into it (currently it's a caller address of psref_acquire). We can detect a
leak by checking the entries on certain points where any references should be
released such as the return point of syscalls and the end of each softint
handler.

The feature is expensive and enabled only if the kernel is built with
PSREF_DEBUG.

Proposed on tech-kern

Attach "mnt_transinfo" to "dead_rootmount" so every mount has a
valid "mnt_transinfo" and remove now unneeded flag IMNT_HAS_TRANS.

Run fstrans_start()/fstrans_done() on dead_rootmount if FSTRANS_DEAD_ENABLED.
Should become the default for DIAGNOSTIC in the future.

Change the place of initproc initialization

The initproc variable cannot be initialized in start_init as there
is a race between vfs_mountroot and start_init.

PR kern/53817 by Andreas Gustafsson

Rather than performing lazy initialization, statically initialize early
in the respective kernel startup routines.

Correct the 6 second offset issue between the time reported by
dmesg -T and the actual time a message was produced, noted on
current-users by Geoff Wing (Oct 27, 2018).

The size of the offset would depend upon architecture, and processor,
but was the delay from starting the clocks to initialising the time
of day (after mounting root, in case that is needed).

Change the kernel to set boottime to be the time at which the
clocks were started, rather than the time at which it is init'd
(by subtracting the interval between).

Correct dmesg to properly compute the ToD based upon the
boottime (which is a timespec, not a timeval, and has been
since Jan 2009) and the time logged in the message.

Note that this can (rarely) be 1 second earlier than date reports.
This occurs when the time when the message was logged was actually
in the next second, but the timecounters have not yet processed
the tick, and so the time of the last tick, near the end of the
previous second, is reported instead. Since times are always
truncated, rather than rounded, it is occasionally possible to
observe that disparity (if you try hard enough).

IOW: sys/kern/subr_prf.c:addtstamp() uses getnanouptime() rather
than nanouptime().

Note in dmesg(8) that -T conversions are gibberish other than
when the message comes from current the running kernel. (It
could be fixed when -M is used, for messages generated by the
kernel whose corpse is being observed. But hasn't been...)

Only print the "no console" warning when booting verbose or debug.
It is a normal condition in many setups and has no consequences for
the user, so do not scare them.

Fix net.inet6.ip6.ifq node doesn't exist

The node (and child nodes) is initialized in sysctl_net_pktq_setup, but the call
of sysctl_net_pktq_setup is skipped unexpectedly.

sysctl_net_pktq_setup is skipped if in6_present is false that indicates the
netinet6 component isn't loaded on rump kernels. However the flag is
accidentally always false because the flag is turned on in in6_dom_init that is
called after if_sysctl_setup on both normal and rump kernels.

Fix the issue by moving if_sysctl_setup after in6_dom_init (domaininit on normal
kernels). This fix is ad-hoc but good enough for netbsd-8. We should refine
the initialization order of network components in the future.

Pointed out by hikaru@

branches: 1.497.2;
Remove the rnewprocp argument from fork1(9)

It's now unused and it can cause use-after-free scenarios as noted by
<Mateusz Guzik>.

Reference: http://mail-index.netbsd.org/tech-kern/2017/09/08/msg022267.html

Sponsored by <The NetBSD Foundation>

Set initproc inside start_init()

This allows us to stop using the rnewprocp argument in fork1(9).

The rnewprocp argument will be removed soon from the API, as it can cause
use-after-free scenarios.

No functional change intended.

Noted by <Mateusz Guzik>
Reference: http://mail-index.netbsd.org/tech-kern/2017/09/08/msg022267.html

Sponsored by <The NetBSD Foundation>

branches: 1.495.2;
Add a proper defflag for GPROF, and include opt_gprof.h, otherwise we're
not gonna go very far.

Make cold __read_mostly like mp_online.

add some assertions to verify that CPU_INFO_FOREACH() works right
early in the boot process. this detects existing bugs on some platforms.

Revert printf return value change.

[syzkaller] Cast all the printf's to (void *)
> as a result of new printf(9) declaration.

branches: 1.490.4; 1.490.6;
Make pfil(9) MP-safe (applying psref(9))

branches: 1.489.2;
Actually initialize the sysctl stuff for kernhist! Missed this file
in earlier commits.

Add a BIOHIST option. As mentioned on tech-kern.

Initialize the bufq code right before we're ready to load the strategy
modules.

Define a new module class for the bufq_strategy modules. These need to
be loaded and intialized before autoconfigure runs, since some devices
(like disks and floppy drives) want to call bufq_alloc().

Modularize the various bufq strategies

* Split sys/kern/sys_process.c into three parts:
1 - ptrace(2) syscall for native emulation
2 - common ptrace(2) syscall code (shared with compat_netbsd32)
3 - support routines that are shared with PROCFS and/or KTRACE

* Add module glue for #1 and #2. Both modules will be built-in to the
kernel if "options PTRACE" is included in the config file (this is
the default, defined in sys/conf/std).

* Mark the ptrace(2) syscall as modular in syscalls.master (generated
files will be committed shortly).

* Conditionalize all remaining portions of PTRACE code on a new kernel
option PTRACE_HOOKS.

XXX Instead of PROCFS depending on 'options PTRACE', we should probably
just add a procfs attribute to the sys/kern/sys_process.c file's
entry in files.kern, and add PROCFS to the "#if defineds" for
process_domem(). It's really confusing to have two different ways
of requiring this file.

This is just a temporary stack that holds fake arguments, and that gets
remapped as RW in sys_execve. Still, in this small window, it does not need
to be executable.

branches: 1.482.2;
KNF. Remove extra spaces. No functional change.

Added missing "it" to comment in start_init()

reduce #ifdef mess caused by PaX

fix this properly.
uap is supposed to hold init's argv[], so it's 3 * sizeof(char *), the bug
was to copyout(..., sizeof(args)) which is an array of syscallargs, not argv
*

do not assume that syscallarg(const char *) and (char *) are the same size
first step to make n32 kernels run binaries again

Replace DIOCGPART -> DIOCGPARTINFO which returns the data needed instead of
pointers.

Modularize drvctl(4)

Fix build dependency of if_llatbl.c

if_llatbl.c is required if inet or inet6 is enabled. Depending on ether
doesn't suit for NDP case.

get rid of the suword {m,j}umbo and check return of copyout.

In sysv_sem.c, defer establishment of exithook so we can initialize the
module code from module_init() rather than waiting until after calling
exec_init(). Use a RUN_ONCE routine at entry to each sys_sem* syscall
to establish the exithook, and no longer KASSERT that the hook has
been set before removing it. (A manually loaded module can be unloaded
before any syscalls have been invoked.)

Remove the conditional calls to the various xxx_init() routines from
init_main.c - we now rely on module_init() to handle initialization.

Let each sub-component's xxx_init() routine handle its own sysctl
sub-tree initialization; this removes another set of #ifdef ugliness.

Tested both built-in and loadable versions and verified that atf
test kernel/t_sysv passes.

introduce a new way of handling SYSCALL_DEBUG messages -- send them to
a kernel history, settable via the SCDEBUG_KERNHIST flag.

this requires a fairly significantly different set of messages than the
normal debug as histories are restricted:
- each message can take one literal format string and upto 4
arguments
- the arguments can not be strings if you want vmstat -u to
work (this could be fixed, and i might, as it would be nice
if we could print syscall names as well as numbers.)

introduce SCDEBUG_DEFAULT that is settable in the kernel config.

fix a problem in kernhist_dump_histories() where it would crash when a
history with no allocated entries was found.

extend kernhist_dumpmask() to handle the usbhist and scdebughist.

initialize MODULE_CLASS_DRIVER modules before the drivers themselves are loaded during autoconfiguration

Handle splash image generation better.

Fix building kernels w/o ether

Hook up lltable/llentry with the kernel (and rumpkernel)

It is built and initialized on bootup, but there is no user for now.

Most codes in in.c are imported from FreeBSD as well as lltable/llentry.

Remove miscfs/syncfs and

- move the syncer into kern/vfs_subr.c.

- change the syncer to process the mountlist and VFS_SYNC as appropriate.

- use an API for mount points similiar to the API for vnodes:
- vfs_syncer_add_to_worklist(struct mount *mp) to add
- vfs_syncer_remove_from_worklist(struct mount *mp) to remove a mount.

No objections on tech-kern@

Remove unintended whitespace.

Added a new option for embedding a splash screen into kernel.
Add: options SPLASHSCREEN
makeoptions SPLASHSCREEN_IMAGE="path/to/image"
to your config file. So far it will work on amd64 and RPI/RPI2.

This commit was with ideas, help, and OK from jmcneill@.

Replace a home-grown run-once implementation with the real RUN_ONCE()

Update initialization of sysmon and its components. These are now handled as part of module initialization, and do not require manual invocation. sysmon_taskq is special, since it is potentially used by several non-module users who may need the facility before modules are fully ready.

wait for config_mountroot threads to complete before we tell init it
can start up. this solves the problem where a console device needs
mountroot to complete attaching, and must create wsdisplay0 before
init tries to open /dev/console. fixes PR#49709.

XXX: pullup-7

branches: 1.461.2;
Yield the main thread only after exiting critical section.

Make uuidgen(2) generate v4 (random) uuids.

Rip out all the needless MAC address and date/time leakage. No more
uuid_init necessary, nor contention over a global uuid state.

While here, simplify uuid_snprintf and fix a strict aliasing
violation.

Defer cprng_fast_init until CPUs are detected.

branches: 1.458.2;
Merge tls-earlyentropy branch into HEAD.

Initialize ubchist earlier.

Move ipi_sysinit() after configure2(); we want secondary CPUs attached.
Might revisit if the there will be a need to use this interface earlier.

Implement MI IPI interface with cross-call support.

branches: 1.454.2;
Add "/rescue/init" to the end of the initpaths list, which
now contains: { "/sbin/init", "/sbin/oinit", "/sbin/init.bak",
"/rescue/init", NULL }.

XXX: The kernel's use of initpaths is not documented.

Tighten initialization of rnd softints.

- Do rnd_init_softint as early as possible in main, after configure2,
and before networking is initialized.

- Initialize the rnd_wakeup softint in rnd_init_softint, not lazily in
rnd_schedule_wakeup.

ok tls

Back out the recent rnd stop-gap/stop-gap/stop-gap measures.

This reverts

sys/dev/rnd_private.h -> r1.1
sys/kern/init_main.c -> r1.450
sys/kern/kern_rndq.c -> r1.14
sys/kern/kern_rndsink.c -> r1.2

Parts of these changes will be added back, and the rndsource
callbacks will be fixed to avoid the lock recursion bug that
motivated the stop-gaps in the first place.

ok tls

Attempt to resolve locking issues at kernel startup on platforms with
hardware RNGs using the polling mode of operation:

1) Initialize the rng subsystem soft interrupts as early in kernel startup
as seems safe (we have no MI guarantee that softints are working at all
until configure2() returns, AFAICT).

This should have the rnd subsystem able to process events via softint
before the network subsystem (a notorious early user of entropy) starts.

2) Remove the shortcut calls to rnd_process_events() from
rnd_schedule_process(), with the result that until the softint is installed
rnd_process_events() is a NOP.

3) Directly call rnd_process_events() in rnd_extract_data(),
rnd_maybe_extract(), and rnd_init_softint(). This should suck up any
samples actually collected as early as possible.

branches: 1.450.2;
Initialize the rnd softint explicitly via a function late in main. Avoids
LOCKDEBUG panic since softint_establish() was called via wdcintr -> wddone
from an interrupt context and tried to acquire a non-spin mutex.

IPSEC has not come in two speeds for a long time now (IPSEC == kame,
FAST_IPSEC). Make everything refer to IPSEC to avoid confusion.

calculate vnode cache size based on the resource it gets allocated from
this stops setting kern.maxvnodes to high so it exhausts available space in kmem

http://mail-index.netbsd.org/tech-kern/2013/03/08/msg015095.html

Move boottime50 and its associated sysctl into the compat module. As
noted on tech-kern. Should fix PR/47579.

OK christos@

Will request pull-up to 6.0 in a few days.

printflike maintenance.

branches: 1.445.2;
Do not call setroot() from MD code and from MI code, which has
unwanted sideeffects in the RB_ASKNAME case. This fixes PR/46732.

No longer wrap MD cpu_rootconf(), as hp300 port stores reboot information
as a side effect. Instead call MI rootconf() from MD code which makes
rootconf() now a wrapper to setroot().

Adjust several MD routines to set the global booted_device,booted_partition
variables instead of passing partial information to setroot().

Make cpu_rootconf(9) describe the calling order.

Do not try to find the wedge we booted from if opendisk(booted_device)
failed.

Make detection of root on wedges (dk(4)) machine independent. Remove
MD code for x86, xen, sparc64.

Remove COMPAT_SA / KERN_SA. Welcome to 6.99.3!
Approved by core@.

branches: 1.441.2;
Entropy-pool implementation move and cleanup.

1) Move core entropy-pool code and source/sink/sample management code
to sys/kern from sys/dev.

2) Remove use of NRND as test for presence of entropy-pool code throughout
source tree.

3) Remove use of RND_ENABLED in device drivers as microoptimization to
avoid expensive operations on disabled entropy sources; make the
rnd_add calls do this directly so all callers benefit.

4) Fix bug in recent rnd_add_data()/rnd_add_uint32() changes that might
have lead to slight entropy overestimation for some sources.

5) Add new source types for environmental sensors, power sensors, VM
system events, and skew between clocks, with a sample implementation
for each.

ok releng to go in before the branch due to the difficulty of later
pullup (widespread #ifdef removal and moved files). Tested with release
builds on amd64 and evbarm and live testing on amd64.

- Add mi_cpu_init() and initialise cpu_lock and kcpuset_attached/running there.
- Add kcpuset_running which gets set in idle_loop().
- Use kcpuset_running in pserialize_perform().

Use and define ALIGN() ALIGN_POINTER() and STACK_ALIGN() consistently,
and avoid definining them in 10 different places if not needed.

Implement the register/deregister/evaluation API for secmodel(9). It
allows registration of callbacks that can be used later for
cross-secmodel "safe" communication.

When a secmodel wishes to know a property maintained by another
secmodel, it has to submit a request to it so the other secmodel can
proceed to evaluating the request. This is done through the
secmodel_eval(9) call; example:

bool isroot;
error = secmodel_eval("org.netbsd.secmodel.suser", "is-root",
cred, &isroot);
if (error == 0 && !isroot)
result = KAUTH_RESULT_DENY;

This one asks the suser module if the credentials are assumed to be root
when evaluated by suser module. If the module is present, it will
respond. If absent, the call will return an error.

Args and command are arbitrarily defined; it's up to the secmodel(9) to
document what it expects.

Typical example is securelevel testing: when someone wants to know
whether securelevel is raised above a certain level or not, the caller
has to request this property to the secmodel_securelevel(9) module.
Given that securelevel module may be absent from system's context (thus
making access to the global "securelevel" variable impossible or
unsafe), this API can cope with this absence and return an error.

We are using secmodel_eval(9) to implement a secmodel_extensions(9)
module, which plugs with the bsd44, suser and securelevel secmodels
to provide the logic behind curtain, usermount and user_set_cpu_affinity
modes, without adding hooks to traditional secmodels. This solves a
real issue with the current secmodel(9) code, as usermount or
user_set_cpu_affinity are not really tied to secmodel_suser(9).

The secmodel_eval(9) is also used to restrict security.models settings
when securelevel is above 0, through the "is-securelevel-above"
evaluation:
- curtain can be enabled any time, but cannot be disabled if
securelevel is above 0.
- usermount/user_set_cpu_affinity can be disabled any time, but cannot
be enabled if securelevel is above 0.

Regarding sysctl(7) entries:
curtain and usermount are now found under security.models.extensions
tree. The security.curtain and vfs.generic.usermount are still
accessible for backwards compat.

Documentation is incoming, I am proof-reading my writings.

Written by elad@, reviewed and tested (anita test + interact for rights
tests) by me. ok elad@.

See also
http://mail-index.netbsd.org/tech-security/2011/11/29/msg000422.html

XXX might consider va0 mapping too.

XXX Having a secmodel(9) specific printf (like aprint_*) for reporting
secmodel(9) errors might be a good idea, but I am not sure on how
to design such a function right now.

branches: 1.437.2;
First step of random number subsystem rework described in
<20111022023242.BA26F14A158@mail.netbsd.org>. This change includes
the following:

An initial cleanup and minor reorganization of the entropy pool
code in sys/dev/rnd.c and sys/dev/rndpool.c. Several bugs are
fixed. Some effort is made to accumulate entropy more quickly at
boot time.

A generic interface, "rndsink", is added, for stream generators to
request that they be re-keyed with good quality entropy from the pool
as soon as it is available.

The arc4random()/arc4randbytes() implementation in libkern is
adjusted to use the rndsink interface for rekeying, which helps
address the problem of low-quality keys at boot time.

An implementation of the FIPS 140-2 statistical tests for random
number generator quality is provided (libkern/rngtest.c). This
is based on Greg Rose's implementation from Qualcomm.

A new random stream generator, nist_ctr_drbg, is provided. It is
based on an implementation of the NIST SP800-90 CTR_DRBG by
Henric Jungheim. This generator users AES in a modified counter
mode to generate a backtracking-resistant random stream.

An abstraction layer, "cprng", is provided for in-kernel consumers
of randomness. The arc4random/arc4randbytes API is deprecated for
in-kernel use. It is replaced by "cprng_strong". The current
cprng_fast implementation wraps the existing arc4random
implementation. The current cprng_strong implementation wraps the
new CTR_DRBG implementation. Both interfaces are rekeyed from
the entropy pool automatically at intervals justifiable from best
current cryptographic practice.

In some quick tests, cprng_fast() is about the same speed as
the old arc4randbytes(), and cprng_strong() is about 20% faster
than rnd_extract_data(). Performance is expected to improve.

The AES code in src/crypto/rijndael is no longer an optional
kernel component, as it is required by cprng_strong, which is
not an optional kernel component.

The entropy pool output is subjected to the rngtest tests at
startup time; if it fails, the system will reboot. There is
approximately a 3/10000 chance of a false positive from these
tests. Entropy pool _input_ from hardware random numbers is
subjected to the rngtest tests at attach time, as well as the
FIPS continuous-output test, to detect bad or stuck hardware
RNGs; if any are detected, they are detached, but the system
continues to run.

A problem with rndctl(8) is fixed -- datastructures with
pointers in arrays are no longer passed to userspace (this
was not a security problem, but rather a major issue for
compat32). A new kernel will require a new rndctl.

The sysctl kern.arandom() and kern.urandom() nodes are hooked
up to the new generators, but the /dev/*random pseudodevices
are not, yet.

Manual pages for the new kernel interfaces are forthcoming.

branches: 1.436.2;
Initialize cpufreq(9) normally from main().

Add kcpuset(9) - a reworked dynamic CPU set implementation for kernel.
Suitable for use during the early boot. MD and other implementations
should be replaced with this interface.

Discussed on: tech-kern@

Add an implementation of passive serialization as described in expired
US patent 4809168. This is a reader / writer synchronization mechanism,
designed for lock-less read operations.

Fix kern/45093 as discussed on tech-kern@:
http://mail-index.netbsd.org/tech-kern/2011/06/17/msg010734.html

The cause of the problem is that the so_pendfree is processed with
the softnet_lock held at one point, and processing the list
calls sodoloanfree() which may kpause(). As the thread sleeps with
softnet_lock held, it ultimately cause a deadlock (see the PR or tech-kern
thread for details).
Although it should be possible to call sodopendfree() after releasing
the socket lock, it's not so easy to know where he socket lock is held and
where it's not, so we may hit the issue again later.
Add a kernel thread to handle the so_pendfree list, and wake up this
thread when adding mbufs to this list. Get rid of the various sodopendfree()
calls, hopefully fixing definitively the problem.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.431.2;
Don't use the C preprocessor to configure USERCONF. Instead, either do
or do not link in subr_userconf.c and x86_userconf.c.

Provide no-op stubs for userconf_bootinfo(), userconf_init(), and
userconf_prompt().

Delete all occurrences of #include "opt_userconf.h" as well as USERCONF
and __HAVE_USERCONF_BOOTINFO #ifdef'age.

Support userconf(4) command in boot(8)/boot.cfg(5) on i386/amd64.

From jmmv@, no objections seen in the proposed thread:

http://mail-index.netbsd.org/tech-kern/2009/01/22/msg004081.html

Re-implement kthread_join(9), so that it actually works (hi haad@).

comment

Move sysctl routines from init_sysctl.c to kern_descrip.c (for
descriptors) and kern_proc.c (for processes). This makes them
usable in a rump kernel, in case somebody was wondering.

branches: 1.426.2;
Move up evcnt_init to before cpu_startup()

Include internal definitions (uvm/uvm.h) only where necessary.

branches: 1.424.2;
ubc_init needs to run while we're still cold or uvmhist breaks.

Define a set of new kernel locking primitives to implement the recursive
kernconfig_mutex. Update module subsystem to use this mutex rather than
its own internal (non-recursive) mutex. Make module_autoload() do its
own locking to be consistent with the rest of the module_xxx() calls.
Update module(9) man page appropriately.

As discussed on tech-kern over the last few weeks.

Welcome to NetBSD 5.99.39 !

1. Add an allocator for 'struct module *' and use it instead of local
allocations.

2. Add a new member mod_flags to the 'struct module *' and define
MODFLG_MUST_FORCE. If this flag is set and the entry is on the list
of builtins, it means that the module has been explicitly unloaded
and any re-loads will require the MODCTL_LOAD_FORCE flag. Provide a
module_require_force() method to set this flag; once set, it should
never be unset.

3. Rename original module_init2() to module_start_unload_thread() to be
more descriptive of what it does.

4. Add a new module_builtin_require_force() routine that sets the
MODFLG_MUST_FORCE flag for any module that has not yet successfully
been initialized. Call it after module_init_class(MODULE_CLASS_ANY)
to disable remaining built-in modules.

This makes built-in versions of the xxxVERBOSE modules work once more,
resolving breakage reported by jruoho@ and njoly@.

Discussed on tech-kern, and comments and suggestions implemented. No
additional discussion for last week. Tested only on amd64 systems, but
there's nothing here that should be port- or architecture-specific (no
more specific than existing module implementation) so others should not
break.

Add config_mountroot(9), which defers device configuration
after mountroot(), like config_interrupt(9) that defers
configuration after interrupts are enabled.
This will be used for devices that require firmware loaded
from the root file system by firmload(9) to complete device
initialization (getting MAC address etc).

No objection on tech-kern@:
http://mail-index.NetBSD.org/tech-kern/2010/06/18/msg008370.html
and will also fix PR kern/43125.

lwp0 seems like an lwp instead of a process, so move bits related
to it from kern_proc.c to kern_lwp.c. This makes kern_proc
"scheduling-clean" and more easily usable in environments with a
non-integrated scheduler (like, to take a random example, rump).

Reduce #ifdef spew by attaching wapbl as a module.
(no, it's still too ifdef-ridden to be able to actually do anything
useful and module-like like load into any kernel)

branches: 1.418.2; 1.418.4;
fix LOCKDEBUG panic 'uninitialized lock'.
seminit() calls exithook_establish(). exithook_establish() uses the exec_lock.
exec_lock is initialzed by exec_init(1).
Call exec_init(1) before seminit().

uncommit part which wasn't supposed to get committed yet

Pass root device as a parameter to domountroothook().

Replace more printfs with aprint_normal / aprint_verbose
Makes "boot -z" go mostly silent for me.

Redefine bpf linkage through an always present op vector, i.e.
#if NBPFILTER is no longer required in the client. This change
doesn't yet add support for loading bpf as a module, since drivers
can register before bpf is attached. However, callers of bpf can
now be modularized.

Dynamically loadable bpf could probably be done fairly easily with
coordination from the stub driver and the real driver by registering
attachments in the stub before the real driver is loaded and doing
a handoff. ... and I'm not going to ponder the depths of unload
here.

Tested with i386/MONOLITHIC, modified MONOLITHIC without bpf and rump.

Including sysctl.h once is enough.

Replace few USER_TO_UAREA/UAREA_TO_USER uses, reduce sys/user.h inclusions.

Move rootfs-related init from init_main() to vfs_mountroot().
Reduces code re-written in rump.

Include miscfs/specfs/specdev.h for spec_init().

- Move kauth_init() a little bit higher.

- Add spec_init() to authorize special device actions (and passthru too for
the time being). Move policy out of secmodel_suser.

Add a kernel configuration flag, SPLDEBUG, that activates a per-CPU log
of transitions to IPL_HIGH from lower IPLs. SPLDEBUG is only available
on i386 and Xen kernels, today.

'options SPLDEBUG' adds instrumentation to spllower() and splraise() as
well as routines to start/stop debugging and to record IPL transitions:
spldebug_start(), spldebug_stop(), spldebug_raise(), spldebug_lower().

Update comment about proc0_init().

Add a (weak aliased) machdep_init() as a place to do machdep initialization
that can't happen as early as the other init functions as called from
cpu_startup() -- for example, register kauth(9) listeners.

Put unprivileged policy in the x86 code; used by i386, amd64, and xen.

- Move sched_listener and co. from kern_synch.c to sys_sched.c, where it
really belongs (suggested by rmind@),

- Rename sched_init() to synch_init(), and introduce a new sched_init()
in sys_sched.c where we (a) initialize the sysctl node (no more
link-set) and (b) listen on the process scope with sched_listener.

Reviewed by and okay rmind@.

Move ptrace's security policy back to the subsystem itself.

Add a ptrace_init() so we have a place to register the listener; called
next to ktrinit().

First part of secmodel cleanup and other misc. changes:

- Separate the suser part of the bsd44 secmodel into its own secmodel
and directory, pending even more cleanups. For revision history
purposes, the original location of the files was

src/sys/secmodel/bsd44/secmodel_bsd44_suser.c
src/sys/secmodel/bsd44/suser.h

- Add a man-page for secmodel_suser(9) and update the one for
secmodel_bsd44(9).

- Add a "secmodel" module class and use it. Userland program and
documentation updated.

- Manage secmodel count (nsecmodels) through the module framework.
This eliminates the need for secmodel_{,de}register() calls in
secmodel code.

- Prepare for secmodel modularization by adding relevant module bits.
The secmodels don't allow auto unload. The bsd44 secmodel depends
on the suser and securelevel secmodels. The overlay secmodel depends
on the bsd44 secmodel. As the module class is only cosmetic, and to
prevent ambiguity, the bsd44 and overlay secmodels are prefixed with
"secmodel_".

- Adapt the overlay secmodel to recent changes (mainly vnode scope).

- Stop using link-sets for the sysctl node(s) creation.

- Keep sysctl variables under nodes of their relevant secmodels. In
other words, don't create duplicates for the suser/securelevel
secmodels under the bsd44 secmodel, as the latter is merely used
for "grouping".

- For the suser and securelevel secmodels, "advertise presence" in
relevant sysctl nodes (sysctl.security.models.{suser,securelevel}).

- Get rid of the LKM preprocessor stuff.

- As secmodels are now modules, there's no need for an explicit call
to secmodel_start(); it's handled by the module framework. That
said, the module framework was adjusted to properly load secmodels
early during system startup.

- Adapt rump to changes: Instead of using empty stubs for securelevel,
simply use the suser secmodel. Also replace secmodel_start() with a
call to secmodel_suser_start().

- 5.99.20.

Testing was done on i386 ("release" build). Spearated module_init()
changes were tested on sparc and sparc64 as well by martin@ (thanks!).

Mailing list reference:

http://mail-index.netbsd.org/tech-kern/2009/09/25/msg006135.html

#include "drvctl.h" for the NDRVCTL definition. Without the NDRVCTL
definition, drvctl_init() is not called, the drvctl_eventq is not
initialized, and the kernel will panic in devmon_insert() when a
device is detached.

Thanks to Jared McNeill for pointing out the panic.

Split config_init() into config_init() and config_init_mi() to help
platforms which want to call config_init() very early in the boot.

Replace a large number of link set based sysctl node creations with
calls from subsystem constructors. Benefits both future kernel
modules and rump.

no change to sysctl nodes on i386/MONOLITHIC & build tested i386/ALL

Wipe out the last vestiges of POOL_INIT with one swift stroke. In
most cases, use a proper constructor. For proplib, give a local
equivalent of POOL_INIT for the kernel object implementation. This
way the code structure can be preserved, and a local link set is
not hazardous anyway (unless proplib is split to several modules,
but that'll be the day).

tested by booting a kernel in qemu and compile-testing i386/ALL

Move configure() and configure2() from subr_autoconf.c to init_main.c,
since they are only peripherially related to the autoconf subsystem
and more related to boot initialization. Also, apply _KERNEL_OPT
to autoconf where necessary.

Initialize devsw (lock) early so that subsystems may play with it.

Do not attach gpiosim(4) at root, but make it a pseudo device.
With help from Matthias Drochner, thanks!

Allow gpiosim(4) to attach if configured in the kernel configuration.

set LP_RUNNING when starting lwp0 and idle lwps.
add assertions.

Make POSIX message queues a kernel module.

Don't send the quiet banner to the log, since the usual noise gets dumped
there anyway.

Convert 67 namei call sites to use namei_simple, in these functions:

check_console, veriexecclose, veriexec_delete, veriexec_file_add,
emul_find_root, coff_load_shlib (sh3 version), coff_load_shlib,
compat_20_sys_statfs, compat_20_netbsd32_statfs,
ELFNAME2(netbsd32,probe_noteless), darwin_sys_statfs,
ibcs2_sys_statfs, ibcs2_sys_statvfs, linux_sys_uselib,
osf1_sys_statfs, sunos_sys_statfs, sunos32_sys_statfs,
ultrix_sys_statfs, do_sys_mount, fss_create_files (3 of 4),
adosfs_mount, cd9660_mount, coda_ioctl, coda_mount, ext2fs_mount,
ffs_mount, filecore_mount, hfs_mount, lfs_mount, msdosfs_mount,
ntfs_mount, sysvbfs_mount, udf_mount, union_mount, sys_chflags,
sys_lchflags, sys_chmod, sys_lchmod, sys_chown, sys_lchown,
sys___posix_chown, sys___posix_lchown, sys_link, do_sys_pstatvfs,
sys_quotactl, sys_revoke, sys_truncate, do_sys_utimes, sys_extattrctl,
sys_extattr_set_file, sys_extattr_set_link, sys_extattr_get_file,
sys_extattr_get_link, sys_extattr_delete_file,
sys_extattr_delete_link, sys_extattr_list_file, sys_extattr_list_link,
sys_setxattr, sys_lsetxattr, sys_getxattr, sys_lgetxattr,
sys_listxattr, sys_llistxattr, sys_removexattr, sys_lremovexattr

All have been scrutinized (several times, in fact) and compile-tested,
but not all have been explicitly tested in action.

XXX: While I haven't (intentionally) changed the use or nonuse of
XXX: TRYEMULROOT in any of these places, I'm not convinced all the
XXX: uses are correct; an audit might be desirable.

Make domaininit() take an argument which determines if it should
add the special PF_ROUTE domain or not (if available).

call rw_obj_init()

Explicitly pass a specific buffer length to format_bytes() to
make it print memory sizes in humanized readable digits.

banner: fix a minor bug.

Remove debug code from previous.

Add a cental banner() function to print the copyright and version info.

PR port-i386/40143 Viewing an mpeg transport stream with mplayer causes crash

Fix numerous problems:

1. LDT updates are not atomic.

2. Number of processes running with private LDTs and/or I/O bitmaps
is not capped. System with high maxprocs can be paniced.

3. LDTR can be leaked over context switch.

4. GDT slot allocations can race, giving the same LDT slot to two procs.

5. Incomplete interrupt/trap frames can be stacked.

6. In some rare cases segment faults are not handled correctly.

main: disable kerenel preemption during early on boot. namely, between
configure() and configure2(). some kernel threads are not expected
to be run before "cold = 0". fixes cache_thread() busy-loop.

Use "defopt MODULAR" in sys/conf/files, and #include "opt_modular.h"
in all kernel sources that use the MODULAR option.
Proposed in tech-kern on 18 Jan 2009.

Unbreak ssp kernels. The issue here that when the ssp_init() call was deferred,
it caused the return from the enclosing function to break, as well as the
ssp return on i386. To fix both issues, split configure in two pieces
the one before calling ssp_init and the one after, and move the ssp_init()
call back in main. Put ssp_init() in its own file, and compile this new file
with -fno-stack-protector. Tested on amd64.
XXX: If we want to have ssp kernels working on 5.0, this change needs to
be pulled up.

branches: 1.380.2;
merge christos-time_t

* unexpose kprintf locking internals
* migrate from simplelock to kmutex

Don't bother to bump kernel version, since nothing outside of subr_prf
used KPRINTF_MUTEX_ENXIT()

Move some sysctl node creations away from linksets and into the
constructors for subsystems.

XXX: CTLFLAG_PERMANENT is non-sensible.

Ksyms are optional, so make the call to ksyms_init() dependent
on the same conditionals which are defined in sys/conf/files.

As discussed on tech-kern: mutex_init is too heavyweight for early bootstrap
phases, so move the initialization of the ksyms mutex back into main via
a function called ksyms_init. Rename the existing (but quite different)
ksyms_init* variations into ksyms_addsyms_elf() and ksyms_addsyms_explicit()
and adapt machdep code accordingly.

cwd is logically a vfs concept, so take it out from the bosom of
kern_descrip and into vfs_cwd. No functional change.

Make POSIX AIO loadable as a module.

Allow the POSIX semaphore code to be loaded as a module.

Remove LKMs and switch to the module framework, pass 1.

Proposed on tech-kern@.

branches: 1.371.2;
On the prompt for init path, print a simple usage line
if input strings are not valid path or command.
Per comments from perry@ and pgoyette@.

branches: 1.370.2;
- if no usable init(8) program (listed in *initpaths[]) can be found,
set the RB_ASKNAME flag and prompt users for the init path, rather than
panicking with "no init".
- when prompting for the init path, support the special strings
"halt", "reboot", and "ddb", as well as a prompt for the root device.

Dissussed and no objection on tech-kern. Changes summary by apb@.

don't expose ksyms_lock

Only init ksyms mutex if ksyms is present in the kernel

PR kern/38814 ksyms needs locking

- Make ksyms MT safe.
- Fix deadlock from an operation like "modload foo.lkm < /dev/ksyms".
- Fix uninitialized structure members.
- Reduce memory footprint for loaded modules.
- Export ksyms structures for kernel grovellers like savecore.
- Some KNF.

- Initialize pool subsystem and kmem(9) earlier, when UVM is up enough.
- Remove uao_hashinit() workaround used for anon-objects.
- Replace malloc with kmem.

OK by <yamt>.

Move uidinfo to its own module in kern_uidinfo.c and include in rump.
No functional change to uidinfo.

Rewrite once to use global locks and atomic ops to get rid of the
static simplelock initializer (and simplelock too). The fastpath
is still lockless, so doesn't make a difference in terms of
performance.

Also fixes a hanging bug if the once routine returned an error.
It does not retry after an error occurs, as I can't really imagine
fruitful semantics for that.

Revert previous change and clarify meaning of RNG

Fix simple typo:
- rnd_init(); /* initialize RNG */
+ rnd_init(); /* initialize RND */

Merge the simonb-wapbl branch. From the original branch commit:

Add Wasabi System's WAPBL (Write Ahead Physical Block Logging)
journaling code. Originally written by Darrin B. Jewell while
at Wasabi and updated to -current by Antti Kantee, Andy Doran,
Greg Oster and Simon Burge.

OK'd by core@, releng@.

branches: 1.360.2;
merge yamt-pf42 branch.
(import newer pf from OpenBSD 4.2)

ok'ed by peter@. requested by core@

PR kern/38927: processes getting stuck in uvm_map (cv_timedwait), hanging
machine

Assume that a vnode (and associated data structures) costs 2kB in the
worst imaginable case. Don't allow sysctl to set desiredvnodes to a
value that would use more than 75% of KVA or 75% of physical memory.

branches: 1.358.2;
- Put in place module compatibility check against __NetBSD_Version__,
as discussed on tech-kern.

- Remove unused module_jettison().

PR kern/38355 lockf deadlock detection is broken after vmlocking

- Fix it; tested with Sun's libMicro.
- Use pool_cache.
- Use a global lock, so the deadlock detection code is safer.

Replace a couple of tsleep calls with cv_wait.

branches: 1.355.2;
Get the pre-loaded module code working.

Remove clause 3 and 4 from TNF licenses

branches: 1.353.2;
Merge proc::p_mutex and proc::p_smutex into a single adaptive mutex, since
we no longer need to guard against access from hardware interrupt handlers.

Additionally, if cloning a process with CLONE_SIGHAND, arrange to have the
child process share the parent's lock so that signal state may be kept in
sync. Partially addresses PR kern/37437.

Network protocol interrupts can now block on locks, so merge the globals
proclist_mutex and proclist_lock into a single adaptive mutex (proc_lock).
Implications:

- Inspecting process state requires thread context, so signals can no longer
be sent from a hardware interrupt handler. Signal activity must be
deferred to a soft interrupt or kthread.

- As the proc state locking is simplified, it's now safe to take exit()
and wait() out from under kernel_lock.

- The system spends less time at IPL_SCHED, and there is less lock activity.

It's only a typo in a comment, but it reduces the number of diffs in my local
tree :-)

timer fixes for PR 37093:

- Fix serious concurrency problems, making the code MT and MP safe in
the process.
- Don't allocate memory or inspect process state from hardclock().

branches: 1.349.2;
SSP: block interrupts when enabling, and move the init to just before
starting secondary processors.

Use multiple kthreads to process config_interrupts tasks. Proposed on
tech-kern.

branches: 1.347.2;
Add code for dynamically allocated mutexes, as posted on tech-kern.

- for some ports, especially for ones without pmap_growkernel,
buf_memcalc is used by bootstrap as well. fix NULL dereference for them.
- limit kva usage for each cache to 20% of vm_map. XXX a bit arbitrary.
- add a comment.

when calculating some cache sizes, consider the amount of available kva.
PR/33185.

Commit the "per-CPU" select patch. This is the result of much work and
testing by rmind@ and myself.

Which approach to use is still being discussed, but I would like to get
this out of my working tree. If we decide to use a different approach
there is no problem with revisiting this.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Remove include of sys/pset.h in sys/lwp.h header.
Include it in few appropriate sources.

branches: 1.341.2; 1.341.6;
Now that __HAVE_TIMECOUNTER and __HAVE_GENERIC_TODR are invariants,
remove the conditionals and the code associated with the undef case.

Pull in my modules code for review/test/hacking.

Implementation of processor-sets, affinity and POSIX real-time extensions.
Add schedctl(8) - a program to control scheduling of processes and threads.

Notes:
- This is supported only by SCHED_M2;
- Migration of LWP mechanism will be revisited;

Proposed on: <tech-kern>. Reviewed by: <ad>.

add a per-cpu storage allocator.

Remove curlwp check, all ports should hopefully be doing the right thing
now (NOTICE: curlwp should be set before main()).

Merge vmlocking2 to head.

Remove systrace. Ok core@.

Call pax_init() for PAX_ASLR.

Merge more changes from vmlocking2, mainly:

- Locking improvements.
- Use pool_cache for more items.

use binuptime for l_stime/l_rtime.

branches: 1.331.2; 1.331.4;
Remove cn_lwp from struct componentname. curlwp should be used
from on. The NDINIT() macro no longer takes the lwp parameter and
associates the credentials of the calling thread with the namei
structure.

branches: 1.330.2;
Add a bit of locking around timecounter attachment / selection.

Boot the secondary processors just before the interrupt-enabled section
of autoconfig. This is needed if APs are able to take interrupts.

call debug_init earlier. ie. before malloc is used.

Use C99 structures initializers when possible.
[from matt-armv6]

Merge tty changes from the vmlocking branch.

Merge from vmlocking.

Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
number and type of priority levels into bands. Add new bands like
'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

branches: 1.323.2;
machine/{bus,cpu,intr}.h -> sys/{bus,cpu,intr}.h

branches: 1.322.2;
Add _SC_NPROCESSORS_ONLN and _SC_NPROCESSORS_CONF for sysconf(). These
are extensions but are provided by many Unix systems.

Merge from vmlocking:

- G/C spinlockmgr() and simple_lock debugging.
- Always include the kernel_lock functions, for LKMs.
- Slightly improved subr_lockdebug code.
- Keep sizeof(struct lock) the same if LOCKDEBUG.

Merge run time accounting changes from the vmlocking branch. These make
the LWP "start time" per-thread instead of per-CPU.

Merge file descriptor locking, cwdi locking and cross-call changes
from the vmlocking branch.

No need to db_init_commands() early any more - it will happen on first
entry to ddb.

Make previous conditional upon !__i386__ && !__x86_64__. I know this is
gross but it's a debug check that's not intended to live very long.
curlwp is about to become a function on x86 (and so can't be assigned to).

If curlwp is not set before main(), moan about it but continue to set it.
curlwp will need to be available earlier for UVM/pmap bootstrap.

db_init_commands() early

branches: 1.314.2;
Implementation of POSIX message queues.

Reviewed by: <ad>, <tech-kern>

Convert the sysmon watchdog framework to use mutex(9) rather than
simple_locks and initialize them on init_main via sysmon_wdog_init().

All the sysmon code now is cleaned up and doesn't use old style locking.

branches: 1.312.2; 1.312.4;
Add cpuctl(8). For now this is not much more than a toy for debugging and
benchmarking that allows taking CPUs online/offline.

branches: 1.311.4;
move setrootfstime() from init_main.c to vfs_subr2.c

Convert sysmon_taskqueue to use mutex(9) and condvar(9) and initialize
them in init_main.c via sysmon_task_queue_preinit().

Reviewed and ok by ad@.

Replace some uses of lockmgr().

Defer callout_startup2() (which calls softintr_establish(9)) call
after cpu_configure(9) for now because softintr(9) is initialized
in cpu_configure(9) on some ports.

Ok'ed by ad@ on current-users, and fixes hangs on m68k ports
during scsi probe.

branches: 1.307.2;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

Remove kcont:

- There are no users in tree.
- Its functionality has largely been replaced by workqueues and generic
soft interrupts.
- It's not MP friendly.

Imported envsys 2, a brief description of the new features:
(Part 1: API)

* Support for detachable sensors.
* Cleaned up the API for simplicity and efficiency.
* Ability to send capacity/critical/warning events to powerd(8).
* Adapted all the code to the new locking order.
* Compatibility with the old envsys API: the ENVSYS_GTREINFO
and ENVSYS_GTREDATA ioctl(2)s are supported.
* Added support for a 'dictionary based communication channel' between
sysmon_power(9) and powerd(8), that means there is no 32 bytes event
size restriction anymore.
* Binary compatibility with old envstat(8) and powerd(8) via COMPAT_40.
* All drivers with the n^2 gtredata bug were fixed, PR kern/36226.

Tested by:

blymn: smsc(4).
bouyer: ipmi(4), mfi(4).
kefren: ug(4).
njoly: viaenv(4), adt7463.c.
riz: owtemp(4).
xtraeme: acpiacad(4), acpibat(4), acpitz(4), aiboost(4), it(4), lm(4).

periodically resize vmem hash table.

- Make aio_worker to handle pending exits and coredumps
- Allow aio_suspend() to be ended early by a signal
- Fix reference counting on LIO structures (remove hack)
- Use two global pools for AIO structures
- Minor cleanups

Patch provided by <ad>. Some additional modifications by me.
Reviewed by <yamt>.

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

Don't call pipe_init if PIPE_SOCKETPAIR is defined.

Put a lock around pipe->pipe_peer.

branches: 1.299.2; 1.299.4;
lockdebug:

- Initialize on the first allocation.
- Handle overflow better. PR kern/35723.

- Make the proclist_lock a mutex. The write:read ratio is unfavourable,
and mutexes are cheaper use than RW locks.
- LOCK_ASSERT -> KASSERT in some places.
- Hold proclist_lock/kernel_lock longer in a couple of places.

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

Remove extra space so that symbol renaming works properly.

Change the process/lwp flags seen by userland via sysctl back to the
P_*/L_* naming convention, and rename the in-kernel flags to avoid
conflict. (P_ -> PK_, L_ -> LW_ ). Add back the (now unused) LSDEAD
constant.

Restores source compatibility with pre-newlock2 tools like ps or top.

Reviewed by Andrew Doran.

branches: 1.294.2;
Count the number of CPUs at boot and stash in 'ncpu'. Eventually should
have each CPU register at attach, so we can figure out the topology for
the scheduler.

remove a duplicated inclusion of sleepq.h.

Merge newlock2 to head.

Add a comment to indicate the reason for kauth_init() and secmodel_start()
being where they are. Suggested by and okay christos@.

Start the security model sooner.

As with previous commit, we want to allow the secmodel code to control
the credential inheritance, etc., so we need it started earlier (also
before proc0_init()).

Initialize kauth(9) sooner.

Since we'll soon want to be able to control the inheritance of credentials,
kauth(9) needs to be ready for use much sooner -- at least before the call
to proc0_init().

New file system suspension API to replace vn_start_write and vn_finished_write.
The suspension helpers are now put into file system specific operations.
This means every file system not supporting these helpers cannot be suspended
and therefore snapshots are no longer possible.

Implemented for file systems of type ffs.

The new API is enabled on a kernel option NEWVNGATE. This option is
not enabled by default in any kernel config.

Presented and discussed on tech-kern with much input from
Bill Studenmund <wrstuden@netbsd.org> and YAMAMOTO Takashi <yamt@netbsd.org>.

Welcome to 4.99.9 (new vfs op vfs_suspendctl).

Oops - this should have gone in a long time ago.

Weak alias secmodel_start to a nop routine, for building without a secmodel
in the kernel.

merge yamt-splraiseipl branch.

- finish implementing splraiseipl (and makeiplcookie).
http://mail-index.NetBSD.org/tech-kern/2006/07/01/0000.html
- complete workqueue(9) and fix its ipl problem, which is reported
to cause audio skipping.
- fix netbt (at least compilation problems) for some ports.
- fix PR/33218.

- remove a static configuration, FILEASSOC_NHOOKS. do it dynamically instead.
- make fileassoc_t a pointer and remove FILEASSOC_INVAL.
- clean up kern_fileassoc.c. unify duplicated code.
- unexport fileassoc_init using RUN_ONCE(9).
- plug memory leaks in fileassoc_file_delete and fileassoc_table_delete.
- always call callbacks, regardless of the value of the associated data.

ok'ed by elad.

iostat: avoid sleeping with a held simple_lock.

I wanted to do this for so long: veriexec_init_fp_ops() -> veriexec_init().

Initial implementation of PaX Segvguard (this is still work-in-progress,
it's just to get it out of my local tree).

Make PaX MPROTECT use specificdata(9), freeing up two P_* flags.
While here, make more generic for upcoming PaX features.

Add SSP support.
XXX: This is broken for me right now, because my kernel resets after fxp0
is probed, but it could be some bug in the driver/compiler.

Add specificdata support to procs and lwps, each providing their own
wrappers around the speicificdata subroutines. Also:
- Call the new lwpinit() function from main() after calling procinit().
- Move some pool initialization out of kern_proc.c and into files that
are directly related to the pools in question (kern_lwp.c and kern_ras.c).
- Convert uipc_sem.c to proc_{get,set}specific(), and eliminate the p_ksems
member from struct proc.

Move the kauth_init() call above auto-configuration; this will fix some
recent bugs introduced with the usage of kauth(9) in MD/device code.

While here, change the sanity checks to KASSERT(), because they're really
bugs we should fix if triggered.

branches: 1.277.2;
First take at security model abstraction.

- Add a few scopes to the kernel: system, network, and machdep.

- Add a few more actions/sub-actions (requests), and start using them as
opposed to the KAUTH_GENERIC_ISSUSER place-holders.

- Introduce a basic set of listeners that implement our "traditional"
security model, called "bsd44". This is the default (and only) model we
have at the moment.

- Update all relevant documentation.

- Add some code and docs to help folks who want to actually use this stuff:

* There's a sample overlay model, sitting on-top of "bsd44", for
fast experimenting with tweaking just a subset of an existing model.

This is pretty cool because it's *really* straightforward to do stuff
you had to use ugly hacks for until now...

* And of course, documentation describing how to do the above for quick
reference, including code samples.

All of these changes were tested for regressions using a Python-based
testsuite that will be (I hope) available soon via pkgsrc. Information
about the tests, and how to write new ones, can be found on:

http://kauth.linbsd.org/kauthwiki

NOTE FOR DEVELOPERS: *PLEASE* don't add any code that does any of the
following:

- Uses a KAUTH_GENERIC_ISSUSER kauth(9) request,
- Checks 'securelevel' directly,
- Checks a uid/gid directly.

(or if you feel you have to, contact me first)

This is still work in progress; It's far from being done, but now it'll
be a lot easier.

Relevant mailing list threads:

http://mail-index.netbsd.org/tech-security/2006/01/25/0011.html
http://mail-index.netbsd.org/tech-security/2006/03/24/0001.html
http://mail-index.netbsd.org/tech-security/2006/04/18/0000.html
http://mail-index.netbsd.org/tech-security/2006/05/15/0000.html
http://mail-index.netbsd.org/tech-security/2006/08/01/0000.html
http://mail-index.netbsd.org/tech-security/2006/08/25/0000.html

Many thanks to YAMAMOTO Takashi, Matt Thomas, and Christos Zoulas for help
stablizing kauth(9).

Full credit for the regression tests, making sure these changes didn't break
anything, goes to Matt Fleming and Jaime Fournier.

Happy birthday Randi! :)

branches: 1.276.4;
at the request of elad, as veriexec.h has returned, revert the changes
from 2006-07-25.

mechanically go through and
s,include "veriexec.h",include <sys/verified_exec.h>,
as the former has apparently gone away.

some fixes:
- adapt to NVERIEXEC in init_sysctl.c.
- we now need "veriexec.h" for NVERIEXEC.
- "opt_verified_exec.h" -> "opt_veriexec.h", and include it only where
it is needed.

deprecate the VERIFIED_EXEC option; now we only need the pseudo-device to
enable it. while here, some config file tweaks.

tons of input from cube@ (thanks!) and okay blymn@.

keep NetBSD boottime semantics:
- only set at boot
- only tracking delta of set-time operations
-> will keep boottime stable across ACPI sleeps
uptime(1) will report the time since last boot

okay, since there was no way to divide this to two commits, here it goes..

introduce fileassoc(9), a kernel interface for associating meta-data with
files using in-kernel memory. this is very similar to what we had in
veriexec till now, only abstracted so it can be used more easily by more
consumers.

this also prompted the redesign of the interface, making it work on vnodes
and mounts and not directly on devices and inodes. internally, we still
use file-id but that's gonna change soon... the interface will remain
consistent.

as a result, veriexec went under some heavy changes to conform to the new
interface. since we no longer use device numbers to identify file-systems,
the veriexec sysctl stuff changed too: kern.veriexec.count.dev_N is now
kern.veriexec.tableN.* where 'N' is NOT the device number but rather a
way to distinguish several mounts.

also worth noting is the plugging of unmount/delete operations
wrt/fileassoc and veriexec.

tons of input from yamt@, wrstuden@, martin@, and christos@.

always call ntp initialisation for timecounter systems as
the ntp code degenerates to the adjtime implementation in the
non NTP case

1. implement solaris-like vmem. (still primitive, though)
2. implement solaris-like kmem_alloc/free api, using #1.
(note: this implementation is backed by kernel_map, thus can't be
used from interrupt context.)

branches: 1.268.2;
re-order initialization sequence to have real counters available during autoconfig

merge FreeBSD timecounters from branch simonb-timecounters
- struct timeval time is gone
time.tv_sec -> time_second
- struct timeval mono_time is gone
mono_time.tv_sec -> time_uptime
- access to time via
{get,}{micro,nano,bin}time()
get* versions are fast but less precise
- support NTP nanokernel implementation (NTP API 4)
- further reading:
Timecounter Paper: http://phk.freebsd.dk/pubs/timecounter.pdf
NTP Nanokernel: http://www.eecis.udel.edu/~mills/ntp/html/kern.html

branches: 1.266.2;
integrate kauth.

Move "opt_maxuprc.h" from init_main.c to kern_proc.c, as the definition
of maxuprc has been moved to kern_proc.c (rev. 1.80).

Remove useless whitepsace.

This commit is dedicated to Dan Langille.

branches: 1.263.2; 1.263.4;
Clean up fallout proc_is_traced_p() change:
- proc_is_traced_p() -> trace_is_enabled(), to match trace_enter() and
trace_exit().
- trace_is_enabled() becomes a real function.
- Remove unnecessary include files from various files that used to care
about KTRACE and SYSTRACE, but do no more.

branches: 1.262.2;
convert "magiclinks" from a per-fs mount option to a system-wide sysctl.
as discussed on tech-kern quite some time ago.

branches: 1.261.2; 1.261.4; 1.261.6;
Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

Overhaul how TTY line disciplines are handled:
- Replace references to linesw[0] with a ttyldisc_default() function
that returns the default ("termios") line discipline.
- The linesw[] array is gone, replaced by a linked list.
- ttyldisc_add() and ttyldisc_remove() have been replaced by
ttyldisc_attach() and ttyldisc_detach().
- Things that provide line disciplines are now responsible for
registering those disciplines with the system. The linesw
structures are no longer declared in tty_conf.c
- Line disciplines are now refcounted; a lookup causes a reference to
be held. ttyldisc_release() releases the reference. Attempts to
detach an in-use line discipline result in EBUSY.
- Fix function signature lossage in if_sl.c, if_strip.c, and tty_tb.c
that was masked by the old tty_conf.c
- tty_init() is no longer necessary; delete it and its call from main().

Statically initalize the systrace lock. systrace_init() is now not
needed on NetBSD. Remove the call from main().

Use a once control to initialize the LKM subsystem on first open. Remove
the lkm_init() call from main().

Use a once control to initialize the NFS server / client shared data
from nfs_vfs_init() or sys_nfssvc(). Remove the nfs_init() call from
main().

Use a once control to call initialize the 802.11 layer when
ieee80211_ifattach() is called. "wlan" no longer needs-flag,
and remove the ieee80211_init() call from main().

- De-couple the software crypto implementation from the rest of the
framework. There is no need to waste the space if you are only using
algoritms provided by hardware accelerators. To get the software
implementations, add "pseudo-device swcr" to your kernel config.
- Lazily initialize the opencrypto framework when crypto drivers
(either hardware or swcr) register themselves with the framework.

Only call ieee80211_init() in kernels that include some wlan stuff.

Resolve conflicts and adapt to NetBSD.

Thanks to dyoung@, scw@, and perry@ for help testing.

2005-08-30 15:27 avatar

Properly set ic_curchan before calling back to device driver to do channel
switching(ifconfig devX channel Y). This fix should make channel changing
works again in monitor mode.

Submitted by: sam
X-MFC-With: other ic_curchan changes

2005-08-13 18:50 sam

revert 1.64: we cannot use the channel characteristics to decide when to
do 11g erp sta accounting because b/g channels show up as false positives
when operating in 11b.

Noticed by: Michal Mertl

2005-08-13 18:31 sam

Extend acl support to pass ioctl requests through and use this to
add support for getting the current policy setting and collecting
the list of mac addresses in the acl table.

Submitted by: Michal Mertl (original version)
MFC after: 2 weeks

2005-08-10 18:42 sam

Don't use ic_curmode to decide when to do 11g station accounting,
use the station channel properties. Fixes assert failure/bogus
operation when an ap is operating in 11a and has associated stations
then switches to 11g.

Noticed by: Michal Mertl
Reviewed by: avatar
MFC after: 2 weeks

2005-08-10 17:22 sam

Clarify/fix handling of the current channel:
o add ic_curchan and use it uniformly for specifying the current
channel instead of overloading ic->ic_bss->ni_chan (or in some
drivers ic_ibss_chan)
o add ieee80211_scanparams structure to encapsulate scanning-related
state captured for rx frames
o move rx beacon+probe response frame handling into separate routines
o change beacon+probe response handling to treat the scan table
more like a scan cache--look for an existing entry before adding
a new one; this combined with ic_curchan use corrects handling of
stations that were previously found at a different channel
o move adhoc neighbor discovery by beacon+probe response frames to
a new ieee80211_add_neighbor routine

Reviewed by: avatar
Tested by: avatar, Michal Mertl
MFC after: 2 weeks

2005-08-09 11:19 rwatson

Propagate rename of IFF_OACTIVE and IFF_RUNNING to IFF_DRV_OACTIVE and
IFF_DRV_RUNNING, as well as the move from ifnet.if_flags to
ifnet.if_drv_flags. Device drivers are now responsible for
synchronizing access to these flags, as they are in if_drv_flags. This
helps prevent races between the network stack and device driver in
maintaining the interface flags field.

Many __FreeBSD__ and __FreeBSD_version checks maintained and continued;
some less so.

Reviewed by: pjd, bz
MFC after: 7 days

2005-08-08 19:46 sam

Split crypto tx+rx key indices and add a key index -> node mapping table:

Crypto changes:
o change driver/net80211 key_alloc api to return tx+rx key indices; a
driver can leave the rx key index set to IEEE80211_KEYIX_NONE or set
it to be the same as the tx key index (the former disables use of
the key index in building the keyix->node mapping table and is the
default setup for naive drivers by null_key_alloc)
o add cs_max_keyid to crypto state to specify the max h/w key index a
driver will return; this is used to allocate the key index mapping
table and to bounds check table loookups
o while here introduce ieee80211_keyix (finally) for the type of a h/w
key index
o change crypto notifiers for rx failures to pass the rx key index up
as appropriate (michael failure, replay, etc.)

Node table changes:
o optionally allocate a h/w key index to node mapping table for the
station table using the max key index setting supplied by drivers
(note the scan table does not get a map)
o defer node table allocation to lateattach so the driver has a chance
to set the max key id to size the key index map
o while here also defer the aid bitmap allocation
o add new ieee80211_find_rxnode_withkey api to find a sta/node entry
on frame receive with an optional h/w key index to use in checking
mapping table; also updates the map if it does a hash lookup and the
found node has a rx key index set in the unicast key; note this work
is separated from the old ieee80211_find_rxnode call so drivers do
not need to be aware of the new mechanism
o move some node table manipulation under the node table lock to close
a race on node delete
o add ieee80211_node_delucastkey to do the dirty work of deleting
unicast key state for a node (deletes any key and handles key map
references)

Ath driver:
o nuke private sc_keyixmap mechansim in favor of net80211 support
o update key alloc api

These changes close several race conditions for the ath driver operating
in ap mode. Other drivers should see no change. Station mode operation
for ath no longer uses the key index map but performance tests show no
noticeable change and this will be fixed when the scan table is eliminated
with the new scanning support.

Tested by: Michal Mertl, avatar, others
Reviewed by: avatar, others
MFC after: 2 weeks

2005-08-08 06:49 sam

use ieee80211_iterate_nodes to retrieve station data; the previous
code walked the list w/o locking

MFC after: 1 week

2005-08-08 04:30 sam

Cleanup beacon/listen interval handling:
o separate configured beacon interval from listen interval; this
avoids potential use of one value for the other (e.g. setting
powersavesleep to 0 clobbers the beacon interval used in hostap
or ibss mode)
o bounds check the beacon interval received in probe response and
beacon frames and drop frames with bogus settings; not clear
if we should instead clamp the value as any alteration would
result in mismatched sta+ap configuration and probably be more
confusing (don't want to log to the console but perhaps ok with
rate limiting)
o while here up max beacon interval to reflect WiFi standard

Noticed by: Martin <nakal@nurfuerspam.de>
MFC after: 1 week

2005-08-06 05:57 sam

fix debug msg typo

MFC after: 3 days

2005-08-06 05:56 sam

Fix handling of frames sent prior to a station being authorized
when operating in ap mode. Previously we allocated a node from the
station table, sent the frame (using the node), then released the
reference that "held the frame in the table". But while the frame
was in flight the node might be reclaimed which could lead to
problems. The solution is to add an ieee80211_tmp_node routine
that crafts a node that does exist in a table and so isn't ever
reclaimed; it exists only so long as the associated frame is in flight.

MFC after: 5 days

2005-07-31 07:12 sam

close a race between reclaiming a node when a station is inactive
and sending the null data frame used to probe inactive stations

MFC after: 5 days

2005-07-27 05:41 sam

when bridging internally bypass the bss node as traffic to it
must follow the normal input path

Submitted by: Michal Mertl
MFC after: 5 days

2005-07-27 03:53 sam

bandaid ni_fails handling so ap's with association failures are
reconsidered after a bit; a proper fix involves more changes to
the scanning infrastructure

Reviewed by: avatar, David Young
MFC after: 5 days

2005-07-23 01:16 sam

the AREF flag is only meaningful in ap mode; adhoc neighbors now
are timed out of the sta/neighbor table

2005-07-23 00:25 sam

o move inactivity-related debug msgs under IEEE80211_MSG_INACT
o probe inactive neighbors in adhoc mode (they don't have an
association id so previously were being timed out)

MFC after: 3 days

2005-07-22 22:11 sam

split xmit of probe request frame out into a separate routine that
takes explicit parameters; this will be needed when scanning is
decoupled from the state machine to do bg scanning

MFC after: 3 days

2005-07-22 21:48 sam

split 802.11 frame xmit setup code into ieee80211_send_setup

MFC after: 3 days

2005-07-22 18:57 sam

simplify ic_newassoc callback

MFC after: 3 days

2005-07-22 18:54 sam

simplify ieee80211_ibss_merge api

MFC after: 3 days

2005-07-22 18:50 sam

add stats we know we'll need soon and some spare fields for future expansion

MFC after: 3 days

2005-07-22 18:45 sam

simplify tim callback api

MFC after: 3 days

2005-07-22 18:42 sam

don't include 802.3 header in min frame length calculation as it may
not be present for a frag; fixes problem with small (fragmented) frames
being dropped

Obtained from: Atheros
MFC after: 3 days

2005-07-22 18:36 sam

simplify ieee80211_node_authorize and ieee80211_node_unauthorize api's

MFC after: 3 days

2005-07-22 18:31 sam

simplifiy ieee80211_send_nulldata api

MFC after: 3 days

2005-07-22 18:29 sam

simplify rate set api's by removing ic parameter (implicit in node reference)

MFC after: 3 days

2005-07-22 18:21 sam

reject association requests with a wpa/rsn ie when wpa/rsn is not
configured on the ap; previously we either ignored the ie or (possibly)
failed an assertion

Obtained from: Atheros
MFC after: 3 days

2005-07-22 18:16 sam

missed one in last commit; add device name to discard msgs

2005-07-22 18:13 sam

include device name in discard msgs

2005-07-22 18:12 sam

add diag msgs for frames discarded because the direction field is wrong

2005-07-22 18:08 sam

split data frame delivery out to a new function ieee80211_deliver_data

2005-07-22 18:00 sam

o add IEEE80211_IOC_FRAGTHRESHOLD for getting+setting the
tx fragmentation threshold
o fix bounds checking on IEEE80211_IOC_RTSTHRESHOLD

MFC after: 3 days

2005-07-22 17:55 sam

o add IEEE80211_FRAG_DEFAULT
o move default settings for RTS and frag thresholds to ieee80211_var.h

2005-07-22 17:50 sam

diff reduction against p4: define IEEE80211_FIXED_RATE_NONE and use
it instead of -1

2005-07-22 17:37 sam

add flags missed in last merge

2005-07-22 17:36 sam

Diff reduction against p4:
o add ic_flags_ext for eventual extention of ic_flags
o define/reserve flag+capabilities bits for superg,
bg scan, and roaming support
o refactor debug msg macros

MFC after: 3 days

2005-07-22 06:17 sam

send a response when an auth request is denied due to an acl;
might be better to silently ignore the frame but this way we
give stations a chance of figuring out what's wrong

2005-07-22 06:15 sam

remove excess whitespace

2005-07-22 05:55 sam

use IF_HANDOFF when bridging frames internally so if_start gets
called; fixes communication between associated sta's

MFC after: 3 days

2005-07-11 04:06 sam

Handle encrypt of arbitarily fragmented mbuf chains: previously
we bailed if we couldn't collect the 16-bytes of data required
for an aes block cipher in 2 mbufs; now we deal with it. While
here make space accounting signed so a sanity check does the
right thing for malformed mbuf chains.

Approved by: re (scottl)

2005-07-11 04:00 sam

nuke assert that duplicates real check

Reviewed by: avatar
Approved by: re (scottl)

branches: 1.251.6;
Move proc0 initialization from main() in init_main.c and proc0_insert() in
kern_proc.c into a new function proc0_init() in kern_proc.c, as suggested
on tech-kern@ days ago.

defopt verified_exec.

White space KNF nit.

branches: 1.248.2;
Implement expansion of special "magic" strings in symlinks into
system-specific values. Submitted by Chris Demetriou in Nov 1995 (!)
in PR kern/1781, modified only slighly by me.

This is enabled on a per-mount basis with the MNT_MAGICLINKS mount
flag. It can be enabled at mountroot() time by building the kernel
with the ROOTFS_MAGICLINKS option.

The following magic strings are supported by the implementation:

@machine value of MACHINE for the system
@machine_arch value of MACHINE_ARCH for the system
@hostname the system host name, as set with sethostname()
@domainname the system domain name, as set with setdomainname()
@kernel_ident the kernel config file name
@osrelease the releaes number of the OS
@ostype the name of the OS (always "NetBSD" for NetBSD)

Example usage:

mkdir /arch/i386/bin
mkdir /arch/sparc/bin
ln -s /arch/@machine_arch/bin /bin

- add const.
- remove unnecessary casts.
- add __UNCONST casts and mark them with XXXUNCONST as necessary.

Move the MI printing of `copyright' to the MD cpu_startup() code
where the printing of `version' is already performed.
This has the benefit of allowing the copyright to be available
via dmesg(8) on platforms which need the `msgbuf' to be setup
in cpu_startup() before printed output is remembered.

Rototill of the verified exec functionality.
* We now use hash tables instead of a list to store the in kernel
fingerprints.
* Fingerprint methods handling has been made more flexible, it is now
even simpler to add new methods.
* the loader no longer passes in magic numbers representing the
fingerprint method so veriexecctl is not longer kernel specific.
* fingerprint methods can be tailored out using options in the kernel
config file.
* more fingerprint methods added - rmd160, sha256/384/512
* veriexecctl can now report the fingerprint methods supported by the
running kernel.
* regularised the naming of some portions of veriexec.

branches: 1.244.6;
move the call to link_pool_init() to the end of uvm_init(). needed for sun3.

branches: 1.243.2;
Rework the mountroot interface so that vfs_mountroot() opens the root device
and just passes it on to the file system functions. This avoids opening and
closing the device several times.

Mentioned on tech-kern some time ago, IIRC. I've been running this for a
long time.

No longer need <sys/disk.h>

- Eliminate the need to call disk_init().
- disk_count needs to be protected with disklist_slock, too.

introduce a function, proclist_foreach_call, to iterate all procs on
a proclist and call the specified function for each of them.
primarily to fix a procfs locking problem, but i think that it's useful for
others as well.

while i'm here, introduce PROCLIST_FOREACH macro, which is similar to
LIST_FOREACH but skips marker entries which are used by proclist_foreach_call.

Call inittodr() from main(). Let file system code set the recorded `last
update' time (if any) through the new function setrootfstime().

Initialize simple_lock in struct cwd; otherwise, one gets an
uninitialized lock panic at the first use of cwdshare().

Provide a mutex for the process limits data structure.

Initialise (most) pools from a link set instead of explicit calls
to pool_init. Untouched pools are ones that either in arch-specific
code, or aren't initialiased during initial system startup.

Convert struct session, ucred and lockf to pools.

Make kernel continuations optional for now.

Use proper NetBSD conventions for deferred kthread creation, not the
other semantics from an earlier incarnation.

Call kcont_init() from init_main before device autoconfiguration,
so kcont is availble to device drivers if required.

Also ensure the kthread process runs any pending continuations once
the kthread is finally up and running. For now, use a non-null timeout
to poll the queue periodically. Draining any pending requests just
before the kthread enters its ltsleep()/kc_run loop is cleaner, but
this is the version I tested with an early-in-boot kcont request.)

Whitespaces.

Bump default size of vnode cache to 1% of physical memory, instead of
0.5%, based on some quick measurements on a number of workstations and
small fileservers (including my home fileserver running simultaneous
builds of the NetBSD source tree and several NetBSD kernels). This
brings the hit rate on my machines from below 70% to above 90%. We
should be able to tune this as we run, by tracking the hit rate and
increasing the size of the cache if memory permits.

Some systems will still require significantly larger cache sizes. Some
ports -- notably the 64-bit ones -- probably should use more than 1% of
physmem as the default due to the larger size of struct vnode.

Store the copyright text in conf/copyright, and use conf/newvers.sh
to generate the appropriate const char copyright[] = "...";
statement instead of hard coding it into kern/init_main.c.
Idea from Simon Burge.

Rearrange process exit path to avoid need to free resources from different
process context ('reaper').

From within the exiting process context:
* deactivate pmap and free vmspace while we can still block
* introduce MD cpu_lwp_free() - this cleans all MD-specific context (such
as FPU state), and is the last potentially blocking operation;
all of cpu_wait(), and most of cpu_exit(), is now folded into cpu_lwp_free()
* process is now immediatelly marked as zombie and made available for pickup
by parent; the remaining last lwp continues the exit as fully detached
* MI (rather than MD) code bumps uvmexp.swtch, cpu_exit() is now same
for both 'process' and 'lwp' exit

uvm_lwp_exit() is modified to never block; the u-area memory is now
always just linked to the list of available u-areas. Introduce (blocking)
uvm_uarea_drain(), which is called to release the excessive u-area memory;
this is called by parent within wait4(), or by pagedaemon on memory shortage.
uvm_uarea_free() is now private function within uvm_glue.c.

MD process/lwp exit code now always calls lwp_exit2() immediatelly after
switching away from the exiting lwp.

g/c now unneeded routines and variables, including the reaper kernel thread

Welcome to 2004!

Replace the traditional buffer memory management -- based on fixed per buffer
virtual memory reservation and a private pool of memory pages -- by a scheme
based on memory pools.

This allows better utilization of memory because buffers can now be allocated
with a granularity finer than the system's native page size (useful for
filesystems with e.g. 1k or 2k fragment sizes). It also avoids fragmentation
of virtual to physical memory mappings (due to the former fixed virtual
address reservation) resulting in better utilization of MMU resources on some
platforms. Finally, the scheme is more flexible by allowing run-time decisions
on the amount of memory to be used for buffers.

On the other hand, the effectiveness of the LRU queue for buffer recycling
may be somewhat reduced compared to the traditional method since, due to the
nature of the pool based memory allocation, the actual least recently used
buffer may release its memory to a pool different from the one needed by a
newly allocated buffer. However, this effect will kick in only if the
system is under memory pressure.

include <sys/mbuf.h> before FAST_IPSEC-dependent headers.

Remove p_nras from struct proc - use LIST_EMPTY(&p->p_raslist) instead.
Remove p_raslock and rename p_lwplock p_lock (one lock is enough).
Simplify window test when adding a ras and correct test on VM_MAXUSER_ADDRESS.
Avoid unpredictable branch in i386 locore.S
(pad fields left in struct proc to avoid kernel bump)

use LIST_FOREACH() as appropriate

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

(FAST_IPSEC): Pull in option header-test for FAST_IPSEC (and IPSEC).

If FAST_IPSEC is configured, attach fast-ipsec transforms after
autoconfiguring devices (perhaps including crypto hardware)
but before starting up network-device packet input.

Move the initialization of the crypto framework from the userland
pseudo-device to init_main(), so the framework is ready for
registration requests at autoconfiguration time.

Thanks to Quentin Garnier for confirming the change was required, and
for testing a similar fix.

branches: 1.221.2;
Back out the lwp/ktrace changes. They contained a lot of colateral damage,
and need to be examined and discussed more.

Undo part of the ktrace/lwp changes. In particular:
* Remove the "lwp *" argument that was added to vget(). Turns out
that nothing actually used it!
* Remove the "lwp *" arguments that were added to VFS_ROOT(), VFS_VGET(),
and VFS_FHTOVP(); all they did was pass it to vget() (which, as noted
above, didn't use it).
* Remove all of the "lwp *" arguments to internal functions that were added
just to appease the above.

Pass lwp pointers throughtout the kernel, as required, so that the lwpid can
be inserted into ktrace records. The general change has been to replace
"struct proc *" with "struct lwp *" in various function prototypes, pass
the lwp through and use l_proc to get the process pointer when needed.

Bump the kernel rev up to 1.6V

Alternative pid/proc allocater, removes all searches associated with pid
lookup and allocation, and any dependency on NPROC or MAXUSERS.
NO_PID changed to -1 (and renamed NO_PGID) to remove artificial limit
on PID_MAX.
As discussed on tech-kern.

add support for p1003.1b semaphores. From FreeBSD

Merge the nathanw_sa branch.

Update copyright notice.

branches: 1.214.2;
Define nofile and maxuprc variables (set to NOFILE and MAXUPRC), so they can
be patched in a compiled kernel.

initialize uvm.aiodoned_proc.

Add an EVCNT_ATTACH_STATIC() macro which gathers static evcnts
into a link set, which are added to the list of event counters
at boot time.

add support for __MACHINE_STACK_GROWS_UP platforms. from fredette@

Add a new VM map, lkm_map, which machine-dependent code can provide
in the event that it needs to use a special VM range (x86_64 falls
into this category). We fall back onto kernel_map if machine-dependent
code doesn't create a special map.

merge kqueue branch into -current

kqueue provides a stateful and efficient event notification framework
currently supported events include socket, file, directory, fifo,
pipe, tty and device changes, and monitoring of processes and signals

kqueue is supported by all writable filesystems in NetBSD tree
(with exception of Coda) and all device drivers supporting poll(2)

based on work done by Jonathan Lemon for FreeBSD
initial NetBSD port done by Luke Mewburn and Jason Thorpe

Add a generic config finalization hook, to be called once all real
devices have been discovered. All finalizer routines are iteratively
invoked until all of them report that they have done no work.

Use this hook to fix a latent bug in RAIDframe autoconfiguration of
RAID sets exposed by the rework of SCSI device discovery.

Don't include <sys/map.h>.

Use the queue macros from <sys/queue.h> instead of referring to the queue
members directly. Use *_FOREACH whenever possible.

Initialize proc0.p_raslock to avoid a lock assertion on the first fork().

Fix a signed/unsigned comparison warning from GCC 3.3.

only print "init: trying /some/init" if RB_ASKNAME or if it's not the first
path we're trying. (the intent but not the behaviour of the previous rev.)

in start_init(), if RB_ASKNAME is set in boothowto, ask for the path
name to start up as init (rather than just cycling thru initpaths[]
and panicing when out of options). if RB_ASKNAME isn't set, the old
behaviour remains. inspired by changes in der Mouse's patchtree.
resolves [kern/18027] from me.

Niels Provos systrace work, ported to NetBSD by kittenz and reworked...

re-scan all ifnet after domaininit() for if_afdata initialization.

branches: 1.199.2; 1.199.6; 1.199.8;
Use <sys/disk.h> for the prototype of disk_init() rather than declaring
our own locally.

Switch default for pipes to the faster John S. Dyson's implementation.
Old, socketpair-based ones are available with option PIPE_SOCKETPAIR.

Happy New Year!

add RCSIDs

branches: 1.195.4;
user maps are always pageable.

When we auto size the vnode cache, make sure we do it *before* we
init vfs so it can the size into account when creating its hash lists.
This means that for a 2GB system, it'll have a default of 65536 buckets
instead of 2048 and when you have 200,000+ vnodes that makes a significant
difference.

Remove initial newline from copyright[], which was mistakely added in rev.1.191.
Fixes kern/13470 by Tetsuya Isaki.

branches: 1.192.2;
Add port of high performance pipe implementation written by John S. Dyson
for FreeBSD project. Besides huge speed boost compared with socketpair-based
pipes, this implementation also uses pagable kernel memory instead of mbufs.

Significant differences to FreeBSD version:
* uses uvm_loan() facility for direct write
* async/SIGIO handling correct also for sync writer, async reader
* limits settable via sysctl, amountpipekva and nbigpipes available via sysctl
* pipes are unidirectional - this is enforced on file descriptor level
for now only, the code would be updated to take advantage of it
eventually
* uses lockmgr(9)-based locks instead of home brew variant
* scatter-gather write is handled correctly for direct write case, data
is transferred by PIPE_DIRECT_CHUNK bytes maximum, to avoid running out of kva

All FreeBSD/NetBSD specific code is within appropriate #ifdef, in preparation
to feed changes back to FreeBSD tree.

This pipe implementation is optional for now, add 'options NEW_PIPE'
to your kernel config to use it.

use real \n's copyright[]; avoids gcc 3.0-prerelease warnings.

Remove the use of splimp() from the NetBSD kernel. splnet()
and only splnet() is allowed for the protection of data structures
used by network devices.

eliminate the KERN_* error codes in favor of the traditional E* codes.
the mapping is:

KERN_SUCCESS 0
KERN_INVALID_ADDRESS EFAULT
KERN_PROTECTION_FAILURE EACCES
KERN_NO_SPACE ENOMEM
KERN_INVALID_ARGUMENT EINVAL
KERN_FAILURE various, mostly turn into KASSERTs
KERN_RESOURCE_SHORTAGE ENOMEM
KERN_NOT_RECEIVER <unused>
KERN_NO_ACCESS <unused>
KERN_PAGES_LOCKED <unused>

branches: 1.188.2;
Happy new year!

Introduce 2 new flags in types.h:
* __HAVE_SYSCALL_INTERN. If this is defined, e_syscall is replaced by
e_syscall_intern, which is called at key places in the kernel. This can be
used to set a MD syscall handler pointer. This obsoletes and replaces the
*_HAS_SEPARATED_SYSCALL flags.
* __HAVE_MINIMAL_EMUL. If this is defined, certain (deprecated) elements in
struct emul are omitted.