Teach percpu to run constructors on CPU attach.

This should make percpu work reliably for things like cprng_strong
much earlier -- not just after all CPUs have been detected. This is
also necessary (though not on its own sufficient) for CPU hotplug.

Add percpu_foreach_xcall(), which is like percpu_foreach(), except it
runs the callback on the target CPU.

KNF

Use __read_mostly and gather related __cacheline_aligned together.

Should save a few bytes of kernel.

New function percpu_create.

Associates a constructor and destructor with the percpu. Currently
the constructor runs immediately, but in principle we could use the
same API for future CPU hotplug support.

This lets you sleep for allocation or draining users before
deallocation when setting up or tearing down a percpu -- currently we
have many abuses of percpu_foreach in tree for that purpose.

Proposed on tech-kern:
https://mail-index.NetBSD.org/tech-kern/2020/01/30/msg026036.html

branches: 1.20.2;
Allow equality in this assertion.

This can happen if we lose the race mentioned in percpu_cpu_swap.

Decorate percpu_cpu_swap() with __noubsan

branches: 1.18.10;
vmem_alloc() with VM_SLEEP cannot fail, so percpu_alloc() cannot fail either.

branches: 1.17.2;
Consistently use kpreempt_*() outside scheduler path.

branches: 1.16.6;
extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged

branches: 1.15.2; 1.15.6;
Report vmem(9) errors out-of-band so that we can use vmem(9) to manage
ranges that include the least and the greatest vmem_addr_t. Update
vmem(9) uses throughout the kernel. Slightly expand on the tests in
subr_vmem.c, which still pass. I've been running a kernel with this
patch without any trouble.

These don't need uvm/uvm_extern.h.

Sprinkle __cacheline_aligned and __read_mostly.

Relax an assertion

Add a KASSERT

branches: 1.10.4; 1.10.6;
Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Start percpu allocation at (ALIGNBYTES + 1) to avoid problem with importing
offset zero to vmem.

branches: 1.8.8; 1.8.10;
- encrypt/decrypt offsets if DIAGNOSTIC.
- add an assertion.
these changes allow to detect a use of uninitialized percpu_t *.

Add MI code to support in-kernel preemption. Preemption is deferred by
one of the following:

- Holding kernel_lock (indicating that the code is not MT safe).
- Bracketing critical sections with kpreempt_disable/kpreempt_enable.
- Holding the interrupt priority level above IPL_NONE.

Statistics on kernel preemption are reported via event counters, and
where preemption is deferred for some reason, it's also reported via
lockstat. The LWP priority at which preemption is triggered is tuneable
via sysctl.

branches: 1.6.2;
- Rename crit_enter/crit_exit to kpreempt_disable/kpreempt_enable.
DragonflyBSD uses the crit names for something quite different.
- Add a kpreempt_disabled function for diagnostic assertions.
- Add inline versions of kpreempt_enable/kpreempt_disable for primitives.
- Make some more changes for preemption safety to the x86 pmap.

fix a comment.

branches: 1.4.2;
Make the percpu API a little more friendly:
- percpu_getptr() is now called percpu_getref() and implicitly disables
preemption (via crit_enter()) when it is called.
- Added percpu_putref() which implicitly reenables preemption (via
crit_exit()).

branches: 1.3.2;
- simplify ASSERT_SLEEPABLE.
- move it from proc.h to systm.h.
- add some more checks.
- make it a little more lkm friendly.

branches: 1.2.2; 1.2.4; 1.2.6; 1.2.8; 1.2.12;
- add a cpu_info pointer argument to percpu_callback_t.
- unexport percpu_zero.
- add some comments.

add a per-cpu storage allocator.

Add percpu_foreach_xcall(), which is like percpu_foreach(), except it
runs the callback on the target CPU.

KNF

Use __read_mostly and gather related __cacheline_aligned together.

Should save a few bytes of kernel.

New function percpu_create.

Associates a constructor and destructor with the percpu. Currently
the constructor runs immediately, but in principle we could use the
same API for future CPU hotplug support.

This lets you sleep for allocation or draining users before
deallocation when setting up or tearing down a percpu -- currently we
have many abuses of percpu_foreach in tree for that purpose.

Proposed on tech-kern:
https://mail-index.NetBSD.org/tech-kern/2020/01/30/msg026036.html

Allow equality in this assertion.

This can happen if we lose the race mentioned in percpu_cpu_swap.

Decorate percpu_cpu_swap() with __noubsan

vmem_alloc() with VM_SLEEP cannot fail, so percpu_alloc() cannot fail either.

branches: 1.17.2;
Consistently use kpreempt_*() outside scheduler path.

branches: 1.16.6;
extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged

branches: 1.15.2; 1.15.6;
Report vmem(9) errors out-of-band so that we can use vmem(9) to manage
ranges that include the least and the greatest vmem_addr_t. Update
vmem(9) uses throughout the kernel. Slightly expand on the tests in
subr_vmem.c, which still pass. I've been running a kernel with this
patch without any trouble.

These don't need uvm/uvm_extern.h.

Sprinkle __cacheline_aligned and __read_mostly.

Relax an assertion

Add a KASSERT

branches: 1.10.4; 1.10.6;
Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Start percpu allocation at (ALIGNBYTES + 1) to avoid problem with importing
offset zero to vmem.

branches: 1.8.8; 1.8.10;
- encrypt/decrypt offsets if DIAGNOSTIC.
- add an assertion.
these changes allow to detect a use of uninitialized percpu_t *.

Add MI code to support in-kernel preemption. Preemption is deferred by
one of the following:

- Holding kernel_lock (indicating that the code is not MT safe).
- Bracketing critical sections with kpreempt_disable/kpreempt_enable.
- Holding the interrupt priority level above IPL_NONE.

Statistics on kernel preemption are reported via event counters, and
where preemption is deferred for some reason, it's also reported via
lockstat. The LWP priority at which preemption is triggered is tuneable
via sysctl.

branches: 1.6.2;
- Rename crit_enter/crit_exit to kpreempt_disable/kpreempt_enable.
DragonflyBSD uses the crit names for something quite different.
- Add a kpreempt_disabled function for diagnostic assertions.
- Add inline versions of kpreempt_enable/kpreempt_disable for primitives.
- Make some more changes for preemption safety to the x86 pmap.

fix a comment.

branches: 1.4.2;
Make the percpu API a little more friendly:
- percpu_getptr() is now called percpu_getref() and implicitly disables
preemption (via crit_enter()) when it is called.
- Added percpu_putref() which implicitly reenables preemption (via
crit_exit()).

branches: 1.3.2;
- simplify ASSERT_SLEEPABLE.
- move it from proc.h to systm.h.
- add some more checks.
- make it a little more lkm friendly.

branches: 1.2.2; 1.2.4; 1.2.6; 1.2.8; 1.2.12;
- add a cpu_info pointer argument to percpu_callback_t.
- unexport percpu_zero.
- add some comments.

add a per-cpu storage allocator.

KNF

Use __read_mostly and gather related __cacheline_aligned together.

Should save a few bytes of kernel.

New function percpu_create.

Associates a constructor and destructor with the percpu. Currently
the constructor runs immediately, but in principle we could use the
same API for future CPU hotplug support.

This lets you sleep for allocation or draining users before
deallocation when setting up or tearing down a percpu -- currently we
have many abuses of percpu_foreach in tree for that purpose.

Proposed on tech-kern:
https://mail-index.NetBSD.org/tech-kern/2020/01/30/msg026036.html

Allow equality in this assertion.

This can happen if we lose the race mentioned in percpu_cpu_swap.

Decorate percpu_cpu_swap() with __noubsan

vmem_alloc() with VM_SLEEP cannot fail, so percpu_alloc() cannot fail either.

branches: 1.17.2;
Consistently use kpreempt_*() outside scheduler path.

branches: 1.16.6;
extending vmem(9) to be able to allocated resources for it's own needs.
simplifying uvm_map handling (no special kernel entries anymore no relocking)
make malloc(9) a thin wrapper around kmem(9)
(with private interface for interrupt safety reasons)

releng@ acknowledged

branches: 1.15.2; 1.15.6;
Report vmem(9) errors out-of-band so that we can use vmem(9) to manage
ranges that include the least and the greatest vmem_addr_t. Update
vmem(9) uses throughout the kernel. Slightly expand on the tests in
subr_vmem.c, which still pass. I've been running a kernel with this
patch without any trouble.

These don't need uvm/uvm_extern.h.

Sprinkle __cacheline_aligned and __read_mostly.

Relax an assertion

Add a KASSERT

branches: 1.10.4; 1.10.6;
Remove uarea swap-out functionality:

- Addresses the issue described in PR/38828.
- Some simplification in threading and sleepq subsystems.
- Eliminates pmap_collect() and, as a side note, allows pmap optimisations.
- Eliminates XS_CTL_DATA_ONSTACK in scsipi code.
- Avoids few scans on LWP list and thus potentially long holds of proc_lock.
- Cuts ~1.5k lines of code. Reduces amd64 kernel size by ~4k.
- Removes __SWAP_BROKEN cases.

Tested on x86, mips, acorn32 (thanks <mpumford>) and partly tested on
acorn26 (thanks to <bjh21>).

Discussed on <tech-kern>, reviewed by <ad>.

Start percpu allocation at (ALIGNBYTES + 1) to avoid problem with importing
offset zero to vmem.

branches: 1.8.8; 1.8.10;
- encrypt/decrypt offsets if DIAGNOSTIC.
- add an assertion.
these changes allow to detect a use of uninitialized percpu_t *.

Add MI code to support in-kernel preemption. Preemption is deferred by
one of the following:

- Holding kernel_lock (indicating that the code is not MT safe).
- Bracketing critical sections with kpreempt_disable/kpreempt_enable.
- Holding the interrupt priority level above IPL_NONE.

Statistics on kernel preemption are reported via event counters, and
where preemption is deferred for some reason, it's also reported via
lockstat. The LWP priority at which preemption is triggered is tuneable
via sysctl.

branches: 1.6.2;
- Rename crit_enter/crit_exit to kpreempt_disable/kpreempt_enable.
DragonflyBSD uses the crit names for something quite different.
- Add a kpreempt_disabled function for diagnostic assertions.
- Add inline versions of kpreempt_enable/kpreempt_disable for primitives.
- Make some more changes for preemption safety to the x86 pmap.

fix a comment.

branches: 1.4.2;
Make the percpu API a little more friendly:
- percpu_getptr() is now called percpu_getref() and implicitly disables
preemption (via crit_enter()) when it is called.
- Added percpu_putref() which implicitly reenables preemption (via
crit_exit()).

branches: 1.3.2;
- simplify ASSERT_SLEEPABLE.
- move it from proc.h to systm.h.
- add some more checks.
- make it a little more lkm friendly.

branches: 1.2.2; 1.2.4; 1.2.6; 1.2.8; 1.2.12;
- add a cpu_info pointer argument to percpu_callback_t.
- unexport percpu_zero.
- add some comments.

add a per-cpu storage allocator.