sys/sys: remove CONSTCOND comment from do-while-0 loop

Lint doesn't need this comment anymore, as there is nothing surprising
or unusual about this idiom.

heartbeat(9): Make heartbeat_suspend/resume nestable.

And make them bind to the CPU as a side effect, instead of requiring
the caller to have already done so.

This lets us eliminate the assertions so we can use them in ddb even
when things are going haywire and we just want to get diagnostics.

XXX kernel revbump -- struct cpu_info change

Break cycle by using `struct kmutex *' instead of `kmutex_t *'.

sys/sched.h included sys/mutex.h
which includes sys/intr.h
which includes machine/intr.h
which on cats includes arm/footbridge/footbridge_intr.h
which includes arm/cpu.h
which includes sys/cpu_data.h
which includes sys/sched.h

But there was never any real need for sys/mutex.h in sys/sched.h,
because it only uses pointers to the opaque struct kmutex. Cycle
broken by using `struct kmutex *' instead of pulling in sys/mutex.h
for the definition of kmutex_t.

Side effect: This revealed that sys/cpu_data.h needed sys/intr.h
(which was pulled in accidentally by sys/mutex.h via sys/sched.h) for
SOFTINT_COUNT. Also revealed some other machine/cpu.h header files
were missing includes of sys/mutex.h for kmutex_t.

heartbeat(9): New mechanism to check progress of kernel.

This uses hard interrupts to check progress of low-priority soft
interrupts, and one CPU to check progress of another CPU.

If no progress has been made after a configurable number of seconds
(kern.heartbeat.max_period, default 15), then the system panics --
preferably on the CPU that is stuck so we get a stack trace in dmesg
of where it was stuck, but if the stuckness was detected by another
CPU and the stuck CPU doesn't acknowledge the request to panic within
one second, the detecting CPU panics instead.

This doesn't supplant hardware watchdog timers. It is possible for
hard interrupts to be stuck on all CPUs for some reason too; in that
case heartbeat(9) has no opportunity to complete.

Downside: heartbeat(9) relies on hardclock to run at a reasonably
consistent rate, which might cause trouble for the glorious tickless
future. However, it could be adapted to take a parameter for an
approximate number of units that have elapsed since the last call on
the current CPU, rather than treating that as a constant 1.

XXX kernel revbump -- changes struct cpu_info layout

branches: 1.52.20;
Remove PG_ZERO. It worked brilliantly on x86 machines from the mid-90s but
having spent an age experimenting with it over the last 6 months on various
machines and with different use cases it's always either break-even or a
slight net loss for me.

Counter tweaks:

- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
each kind of page can be summed to get the totals.

- Track the number of free pages with a counter so that it's one less thing
for the allocator to do, which opens up further options there.

- Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot.
For the cheap option, give cpu_count_sync() a boolean parameter indicating
that a cached value is okay, and rate limit the updates for cached values
to hz.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

cpu topology:

- Fix a stupid bug where it complained about non-existent SMT on assymetric
systems.

- Give each CPU a pointer back to the first CPU in same package.

- Many small tweaks to the SMT awareness in the scheduler. It does a much
better job now at keeping all physical CPUs busy, while using the extra
threads to help out. In particular, during preempt() if we're using SMT,
try to find a better CPU to run on and teleport curlwp there.

- Change the CPU topology stuff so it can work on asymmetric systems. This
mainly entails rearranging one of the CPU lists so it makes sense in all
configurations.

- Add a parameter to cpu_topology_set() to note that a CPU is "slow", for
where there are fast CPUs and slow CPUs, like with the Rockwell RK3399.
Extend the SMT awareness to try and handle that situation too (keep fast
CPUs busy, use slow CPUs as helpers).

branches: 1.46.2;
Counter tweaks:

"zeroaborts" + "free" don't need to be per-CPU counters, and "bucketmiss"
wasn't used. Remove those and cluster by usage.

Some more CPU topology stuff:

- Use cegger@'s ACPI SRAT parsing code to figure out NUMA node ID for each
CPU as it is attached.

- For scheduler experiments with SMT, flag CPUs with the lowest numbered SMT
IDs as "primaries", link back to the primaries from secondaries, and build
a circular list of CPUs in each package with identical SMT IDs.

- No need for package/core/smt/numa IDs to be anything other than a u_int.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

- Add some more failsafes to the CPU topology stuff, and build a 3rd
circular list of peer CPUs in other packages, so we might scroll through
them in the scheduler when looking to distribute or steal jobs.

- Fold the run queue data structure into spc_schedstate. Makes kern_runq.c
a far more pleasant place to work.

- Remove the code in sched_nextlwp() that tries to steal jobs from other
CPUs. It's not needed, because we do the very same thing in the idle LWP
anyway. Outside the VM system this was one of the the main causes of L3
cache misses I saw during builds. On my machine, this change yields a
60%-70% drop in time on the "hackbench" benchmark (there's clearly a bit
more going on here, but basically being less aggressive helps).

Rip out pserialize(9) logic now that the RCU patent has expired.

pserialize_perform() is now basically just xc_barrier(XC_HIGHPRI).
No more tentacles throughout the scheduler. Simplify the psz read
count for diagnostic assertions by putting it unconditionally into
cpu_info.

From rmind@, tidied up by me.

Take the basic CPU topology information we already collect, and use it
to make circular lists of CPU siblings in the same core, and in the
same package. Nothing fancy, just enough to have a bit of fun in the
scheduler trying out different tactics.

Fix false sharing problems with cpu_info. Identified with tprof(8).
This was a very nice win in my tests on a 48 CPU box.

- Reorganise cpu_data slightly according to usage.
- Put cpu_onproc into struct cpu_info alongside ci_curlwp (now is ci_onproc).
- On x86, put some items in their own cache lines according to usage, like
the IPI bitmask and ci_want_resched.

Remove stale comment. There are more than just compat definitions.

branches: 1.38.18; 1.38.20;
Change type of cpu_uvm to struct uvm_cpu *

MI IPI interface:
- Implement support for the asynchronous IPI calls.
- Rework synchronous IPI code to reuse the asynchronous mechanism.
- Add ipi(9) manual page; needs wizd(8).

Note: MD code can now provide a low level primitive for the ipi(9) and
reuse this interface instead of open-coding. Portmasters are encouraged
to convert. Ride 6.99.43!

branches: 1.36.2;
excise unused qchain

branches: 1.35.2; 1.35.4;
Add a kcpuset_t which just includes ourself.
Add a ci_cpuname for convenience

- Add mi_cpu_init() and initialise cpu_lock and kcpuset_attached/running there.
- Add kcpuset_running which gets set in idle_loop().
- Use kcpuset_running in pserialize_perform().

branches: 1.33.4; 1.33.8;
Add PCU (Per-CPU Unit) - an interface to manage synchronization of any
per CPU context tied with an LWP. Main use - lazy FPU handling on SMP.

Requested by matt@, will be used on mips64. Note: implementation will
be improved to use IPIs before adopting on x86. OK ad@.

branches: 1.32.2; 1.32.4;
Move counting of faults, traps, intrs, soft[intr]s, syscalls, and nswtch
from uvmexp to per-cpu cpu_data and move them to 64bits. Remove unneeded
includes of <uvm/uvm_extern.h> and/or <uvm/uvm.h>.

Rename ci_node_id to ci_package_id, as some claim that the former might
be confused with NUMA node.

Add node/core/smt ID numbers to struct cpu_data, will be filled by MD.
OK matt@.

Make select/poll work with more than 32 CPUs.
No ABI change.

branches: 1.28.2; 1.28.4;
introduce a new function that returns a unique string for each cpu:

char *cpu_name(struct cpu_info *);

and use it when setting up the runq event counters, avoiding an 8 byte
kmem(4) allocation for each cpu. there are more places the cpuname is
used that can be converted to using this new interface, but that can
and will be done as future work.

as discussed with rmind.

branches: 1.27.16; 1.27.20;
vfs_cache:

- Don't use goto in critical paths, it can confuse the compiler.
- Sprinkle some branch hints.
- Make namecache stats per-CPU and collate once per second.
- Use vtryget().

Make trap counters per-cpu, like syscalls.

+ uint64_t cpu_cc_freq; /* cycle counter frequency */
+ int64_t cpu_cc_skew; /* counter skew vs cpu0 */

branches: 1.24.2;
LOCKDEBUG: try to speed it up a bit by not using so much global state.

This will break the build briefly but will be followed by another commit
to fix that..

Remove clause 3 and 4 from TNF licenses

branches: 1.22.2;
Implement MP callouts as discussed on tech-kern. The CPU binding code is
disabled for the moment until we figure out what we want to do with CPUs
being offlined.

branches: 1.21.2;
Maintain a circular queue of cpu_info's.

Restructure the name cache code to eliminate most lock contention
resulting from forward lookups. Discussed on tech-kern@.

Commit the "per-CPU" select patch. This is the result of much work and
testing by rmind@ and myself.

Which approach to use is still being discussed, but I would like to get
this out of my working tree. If we decide to use a different approach
there is no problem with revisiting this.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Make context switch + syscall counters optionally per-CPU and accumulate
in schedclock() at "about 16 hz".

branches: 1.16.2; 1.16.6;
remove a special allocator for uareas, which is no longer necessary.
use pool_cache instead.

add a per-cpu storage allocator.

branches: 1.14.2; 1.14.6;
Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
number and type of priority levels into bands. Add new bands like
'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

branches: 1.13.2; 1.13.4;
Merge file descriptor locking, cwdi locking and cross-call changes
from the vmlocking branch.

branches: 1.12.2; 1.12.4;
Make the uarea cache per-CPU and drain in batches of 4.

branches: 1.11.2;
cpu_index doesn't need to be a cpuid_t, since it's a cardinal

Improve per-CPU support for the workqueue(9):
- Make structures CPU-cache friendly, as suggested and explained
by Andrew Doran. CACHE_LINE_SIZE definition is invented.
- Use current CPU if NULL is passed to the workqueue_enqueue().
- Implemented MI CPU index, which could be used as an index of array.
Removed linked-lists usage for work queues.

The roundup2() function avoids division, but works only with power of 2.

Reviewed by: <ad>, <yamt>, <tech-kern>

branches: 1.9.2; 1.9.6;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

branches: 1.7.2; 1.7.6; 1.7.8;
Merge newlock2 to head.

branches: 1.6.2;
Add the 'lockstat' driver, which collects basic information about lock events.

branches: 1.5.4; 1.5.8;
changes for making DIAGNOSTIC not change the kernel ABI:
- for structure fields that are conditionally present,
make those fields always present.
- for functions which are conditionally inline, make them never inline.
- remove some other functions which are conditionally defined but
don't actually do anything anymore.
- make a lock-debugging function conditional on only LOCKDEBUG.

as discussed on tech-kern some time back.

Make sure that multiple header inclusion is done, it uses consistent names,
and it is around the whole code of the file, not parts of it.

branches: 1.3.12;
a relatively lightweight implementation of kernel_lock.

remove an unneeded #include.

branches: 1.1.2;
move some per-cpu data definitions to MI place so that they can be modified
without touching all ports. discussed on tech-kern@.

heartbeat(9): Make heartbeat_suspend/resume nestable.

And make them bind to the CPU as a side effect, instead of requiring
the caller to have already done so.

This lets us eliminate the assertions so we can use them in ddb even
when things are going haywire and we just want to get diagnostics.

XXX kernel revbump -- struct cpu_info change

Break cycle by using `struct kmutex *' instead of `kmutex_t *'.

sys/sched.h included sys/mutex.h
which includes sys/intr.h
which includes machine/intr.h
which on cats includes arm/footbridge/footbridge_intr.h
which includes arm/cpu.h
which includes sys/cpu_data.h
which includes sys/sched.h

But there was never any real need for sys/mutex.h in sys/sched.h,
because it only uses pointers to the opaque struct kmutex. Cycle
broken by using `struct kmutex *' instead of pulling in sys/mutex.h
for the definition of kmutex_t.

Side effect: This revealed that sys/cpu_data.h needed sys/intr.h
(which was pulled in accidentally by sys/mutex.h via sys/sched.h) for
SOFTINT_COUNT. Also revealed some other machine/cpu.h header files
were missing includes of sys/mutex.h for kmutex_t.

heartbeat(9): New mechanism to check progress of kernel.

This uses hard interrupts to check progress of low-priority soft
interrupts, and one CPU to check progress of another CPU.

If no progress has been made after a configurable number of seconds
(kern.heartbeat.max_period, default 15), then the system panics --
preferably on the CPU that is stuck so we get a stack trace in dmesg
of where it was stuck, but if the stuckness was detected by another
CPU and the stuck CPU doesn't acknowledge the request to panic within
one second, the detecting CPU panics instead.

This doesn't supplant hardware watchdog timers. It is possible for
hard interrupts to be stuck on all CPUs for some reason too; in that
case heartbeat(9) has no opportunity to complete.

Downside: heartbeat(9) relies on hardclock to run at a reasonably
consistent rate, which might cause trouble for the glorious tickless
future. However, it could be adapted to take a parameter for an
approximate number of units that have elapsed since the last call on
the current CPU, rather than treating that as a constant 1.

XXX kernel revbump -- changes struct cpu_info layout

branches: 1.52.20;
Remove PG_ZERO. It worked brilliantly on x86 machines from the mid-90s but
having spent an age experimenting with it over the last 6 months on various
machines and with different use cases it's always either break-even or a
slight net loss for me.

Counter tweaks:

- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
each kind of page can be summed to get the totals.

- Track the number of free pages with a counter so that it's one less thing
for the allocator to do, which opens up further options there.

- Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot.
For the cheap option, give cpu_count_sync() a boolean parameter indicating
that a cached value is okay, and rate limit the updates for cached values
to hz.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

cpu topology:

- Fix a stupid bug where it complained about non-existent SMT on assymetric
systems.

- Give each CPU a pointer back to the first CPU in same package.

- Many small tweaks to the SMT awareness in the scheduler. It does a much
better job now at keeping all physical CPUs busy, while using the extra
threads to help out. In particular, during preempt() if we're using SMT,
try to find a better CPU to run on and teleport curlwp there.

- Change the CPU topology stuff so it can work on asymmetric systems. This
mainly entails rearranging one of the CPU lists so it makes sense in all
configurations.

- Add a parameter to cpu_topology_set() to note that a CPU is "slow", for
where there are fast CPUs and slow CPUs, like with the Rockwell RK3399.
Extend the SMT awareness to try and handle that situation too (keep fast
CPUs busy, use slow CPUs as helpers).

branches: 1.46.2;
Counter tweaks:

"zeroaborts" + "free" don't need to be per-CPU counters, and "bucketmiss"
wasn't used. Remove those and cluster by usage.

Some more CPU topology stuff:

- Use cegger@'s ACPI SRAT parsing code to figure out NUMA node ID for each
CPU as it is attached.

- For scheduler experiments with SMT, flag CPUs with the lowest numbered SMT
IDs as "primaries", link back to the primaries from secondaries, and build
a circular list of CPUs in each package with identical SMT IDs.

- No need for package/core/smt/numa IDs to be anything other than a u_int.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

- Add some more failsafes to the CPU topology stuff, and build a 3rd
circular list of peer CPUs in other packages, so we might scroll through
them in the scheduler when looking to distribute or steal jobs.

- Fold the run queue data structure into spc_schedstate. Makes kern_runq.c
a far more pleasant place to work.

- Remove the code in sched_nextlwp() that tries to steal jobs from other
CPUs. It's not needed, because we do the very same thing in the idle LWP
anyway. Outside the VM system this was one of the the main causes of L3
cache misses I saw during builds. On my machine, this change yields a
60%-70% drop in time on the "hackbench" benchmark (there's clearly a bit
more going on here, but basically being less aggressive helps).

Rip out pserialize(9) logic now that the RCU patent has expired.

pserialize_perform() is now basically just xc_barrier(XC_HIGHPRI).
No more tentacles throughout the scheduler. Simplify the psz read
count for diagnostic assertions by putting it unconditionally into
cpu_info.

From rmind@, tidied up by me.

Take the basic CPU topology information we already collect, and use it
to make circular lists of CPU siblings in the same core, and in the
same package. Nothing fancy, just enough to have a bit of fun in the
scheduler trying out different tactics.

Fix false sharing problems with cpu_info. Identified with tprof(8).
This was a very nice win in my tests on a 48 CPU box.

- Reorganise cpu_data slightly according to usage.
- Put cpu_onproc into struct cpu_info alongside ci_curlwp (now is ci_onproc).
- On x86, put some items in their own cache lines according to usage, like
the IPI bitmask and ci_want_resched.

Remove stale comment. There are more than just compat definitions.

branches: 1.38.18; 1.38.20;
Change type of cpu_uvm to struct uvm_cpu *

MI IPI interface:
- Implement support for the asynchronous IPI calls.
- Rework synchronous IPI code to reuse the asynchronous mechanism.
- Add ipi(9) manual page; needs wizd(8).

Note: MD code can now provide a low level primitive for the ipi(9) and
reuse this interface instead of open-coding. Portmasters are encouraged
to convert. Ride 6.99.43!

branches: 1.36.2;
excise unused qchain

branches: 1.35.2; 1.35.4;
Add a kcpuset_t which just includes ourself.
Add a ci_cpuname for convenience

- Add mi_cpu_init() and initialise cpu_lock and kcpuset_attached/running there.
- Add kcpuset_running which gets set in idle_loop().
- Use kcpuset_running in pserialize_perform().

branches: 1.33.4; 1.33.8;
Add PCU (Per-CPU Unit) - an interface to manage synchronization of any
per CPU context tied with an LWP. Main use - lazy FPU handling on SMP.

Requested by matt@, will be used on mips64. Note: implementation will
be improved to use IPIs before adopting on x86. OK ad@.

branches: 1.32.2; 1.32.4;
Move counting of faults, traps, intrs, soft[intr]s, syscalls, and nswtch
from uvmexp to per-cpu cpu_data and move them to 64bits. Remove unneeded
includes of <uvm/uvm_extern.h> and/or <uvm/uvm.h>.

Rename ci_node_id to ci_package_id, as some claim that the former might
be confused with NUMA node.

Add node/core/smt ID numbers to struct cpu_data, will be filled by MD.
OK matt@.

Make select/poll work with more than 32 CPUs.
No ABI change.

branches: 1.28.2; 1.28.4;
introduce a new function that returns a unique string for each cpu:

char *cpu_name(struct cpu_info *);

and use it when setting up the runq event counters, avoiding an 8 byte
kmem(4) allocation for each cpu. there are more places the cpuname is
used that can be converted to using this new interface, but that can
and will be done as future work.

as discussed with rmind.

branches: 1.27.16; 1.27.20;
vfs_cache:

- Don't use goto in critical paths, it can confuse the compiler.
- Sprinkle some branch hints.
- Make namecache stats per-CPU and collate once per second.
- Use vtryget().

Make trap counters per-cpu, like syscalls.

+ uint64_t cpu_cc_freq; /* cycle counter frequency */
+ int64_t cpu_cc_skew; /* counter skew vs cpu0 */

branches: 1.24.2;
LOCKDEBUG: try to speed it up a bit by not using so much global state.

This will break the build briefly but will be followed by another commit
to fix that..

Remove clause 3 and 4 from TNF licenses

branches: 1.22.2;
Implement MP callouts as discussed on tech-kern. The CPU binding code is
disabled for the moment until we figure out what we want to do with CPUs
being offlined.

branches: 1.21.2;
Maintain a circular queue of cpu_info's.

Restructure the name cache code to eliminate most lock contention
resulting from forward lookups. Discussed on tech-kern@.

Commit the "per-CPU" select patch. This is the result of much work and
testing by rmind@ and myself.

Which approach to use is still being discussed, but I would like to get
this out of my working tree. If we decide to use a different approach
there is no problem with revisiting this.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Make context switch + syscall counters optionally per-CPU and accumulate
in schedclock() at "about 16 hz".

branches: 1.16.2; 1.16.6;
remove a special allocator for uareas, which is no longer necessary.
use pool_cache instead.

add a per-cpu storage allocator.

branches: 1.14.2; 1.14.6;
Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
number and type of priority levels into bands. Add new bands like
'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

branches: 1.13.2; 1.13.4;
Merge file descriptor locking, cwdi locking and cross-call changes
from the vmlocking branch.

branches: 1.12.2; 1.12.4;
Make the uarea cache per-CPU and drain in batches of 4.

branches: 1.11.2;
cpu_index doesn't need to be a cpuid_t, since it's a cardinal

Improve per-CPU support for the workqueue(9):
- Make structures CPU-cache friendly, as suggested and explained
by Andrew Doran. CACHE_LINE_SIZE definition is invented.
- Use current CPU if NULL is passed to the workqueue_enqueue().
- Implemented MI CPU index, which could be used as an index of array.
Removed linked-lists usage for work queues.

The roundup2() function avoids division, but works only with power of 2.

Reviewed by: <ad>, <yamt>, <tech-kern>

branches: 1.9.2; 1.9.6;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

branches: 1.7.2; 1.7.6; 1.7.8;
Merge newlock2 to head.

branches: 1.6.2;
Add the 'lockstat' driver, which collects basic information about lock events.

branches: 1.5.4; 1.5.8;
changes for making DIAGNOSTIC not change the kernel ABI:
- for structure fields that are conditionally present,
make those fields always present.
- for functions which are conditionally inline, make them never inline.
- remove some other functions which are conditionally defined but
don't actually do anything anymore.
- make a lock-debugging function conditional on only LOCKDEBUG.

as discussed on tech-kern some time back.

Make sure that multiple header inclusion is done, it uses consistent names,
and it is around the whole code of the file, not parts of it.

branches: 1.3.12;
a relatively lightweight implementation of kernel_lock.

remove an unneeded #include.

branches: 1.1.2;
move some per-cpu data definitions to MI place so that they can be modified
without touching all ports. discussed on tech-kern@.

Break cycle by using `struct kmutex *' instead of `kmutex_t *'.

sys/sched.h included sys/mutex.h
which includes sys/intr.h
which includes machine/intr.h
which on cats includes arm/footbridge/footbridge_intr.h
which includes arm/cpu.h
which includes sys/cpu_data.h
which includes sys/sched.h

But there was never any real need for sys/mutex.h in sys/sched.h,
because it only uses pointers to the opaque struct kmutex. Cycle
broken by using `struct kmutex *' instead of pulling in sys/mutex.h
for the definition of kmutex_t.

Side effect: This revealed that sys/cpu_data.h needed sys/intr.h
(which was pulled in accidentally by sys/mutex.h via sys/sched.h) for
SOFTINT_COUNT. Also revealed some other machine/cpu.h header files
were missing includes of sys/mutex.h for kmutex_t.

heartbeat(9): New mechanism to check progress of kernel.

This uses hard interrupts to check progress of low-priority soft
interrupts, and one CPU to check progress of another CPU.

If no progress has been made after a configurable number of seconds
(kern.heartbeat.max_period, default 15), then the system panics --
preferably on the CPU that is stuck so we get a stack trace in dmesg
of where it was stuck, but if the stuckness was detected by another
CPU and the stuck CPU doesn't acknowledge the request to panic within
one second, the detecting CPU panics instead.

This doesn't supplant hardware watchdog timers. It is possible for
hard interrupts to be stuck on all CPUs for some reason too; in that
case heartbeat(9) has no opportunity to complete.

Downside: heartbeat(9) relies on hardclock to run at a reasonably
consistent rate, which might cause trouble for the glorious tickless
future. However, it could be adapted to take a parameter for an
approximate number of units that have elapsed since the last call on
the current CPU, rather than treating that as a constant 1.

XXX kernel revbump -- changes struct cpu_info layout

Remove PG_ZERO. It worked brilliantly on x86 machines from the mid-90s but
having spent an age experimenting with it over the last 6 months on various
machines and with different use cases it's always either break-even or a
slight net loss for me.

Counter tweaks:

- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
each kind of page can be summed to get the totals.

- Track the number of free pages with a counter so that it's one less thing
for the allocator to do, which opens up further options there.

- Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot.
For the cheap option, give cpu_count_sync() a boolean parameter indicating
that a cached value is okay, and rate limit the updates for cached values
to hz.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

cpu topology:

- Fix a stupid bug where it complained about non-existent SMT on assymetric
systems.

- Give each CPU a pointer back to the first CPU in same package.

- Many small tweaks to the SMT awareness in the scheduler. It does a much
better job now at keeping all physical CPUs busy, while using the extra
threads to help out. In particular, during preempt() if we're using SMT,
try to find a better CPU to run on and teleport curlwp there.

- Change the CPU topology stuff so it can work on asymmetric systems. This
mainly entails rearranging one of the CPU lists so it makes sense in all
configurations.

- Add a parameter to cpu_topology_set() to note that a CPU is "slow", for
where there are fast CPUs and slow CPUs, like with the Rockwell RK3399.
Extend the SMT awareness to try and handle that situation too (keep fast
CPUs busy, use slow CPUs as helpers).

branches: 1.46.2;
Counter tweaks:

"zeroaborts" + "free" don't need to be per-CPU counters, and "bucketmiss"
wasn't used. Remove those and cluster by usage.

Some more CPU topology stuff:

- Use cegger@'s ACPI SRAT parsing code to figure out NUMA node ID for each
CPU as it is attached.

- For scheduler experiments with SMT, flag CPUs with the lowest numbered SMT
IDs as "primaries", link back to the primaries from secondaries, and build
a circular list of CPUs in each package with identical SMT IDs.

- No need for package/core/smt/numa IDs to be anything other than a u_int.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

- Add some more failsafes to the CPU topology stuff, and build a 3rd
circular list of peer CPUs in other packages, so we might scroll through
them in the scheduler when looking to distribute or steal jobs.

- Fold the run queue data structure into spc_schedstate. Makes kern_runq.c
a far more pleasant place to work.

- Remove the code in sched_nextlwp() that tries to steal jobs from other
CPUs. It's not needed, because we do the very same thing in the idle LWP
anyway. Outside the VM system this was one of the the main causes of L3
cache misses I saw during builds. On my machine, this change yields a
60%-70% drop in time on the "hackbench" benchmark (there's clearly a bit
more going on here, but basically being less aggressive helps).

Rip out pserialize(9) logic now that the RCU patent has expired.

pserialize_perform() is now basically just xc_barrier(XC_HIGHPRI).
No more tentacles throughout the scheduler. Simplify the psz read
count for diagnostic assertions by putting it unconditionally into
cpu_info.

From rmind@, tidied up by me.

Take the basic CPU topology information we already collect, and use it
to make circular lists of CPU siblings in the same core, and in the
same package. Nothing fancy, just enough to have a bit of fun in the
scheduler trying out different tactics.

Fix false sharing problems with cpu_info. Identified with tprof(8).
This was a very nice win in my tests on a 48 CPU box.

- Reorganise cpu_data slightly according to usage.
- Put cpu_onproc into struct cpu_info alongside ci_curlwp (now is ci_onproc).
- On x86, put some items in their own cache lines according to usage, like
the IPI bitmask and ci_want_resched.

Remove stale comment. There are more than just compat definitions.

branches: 1.38.18; 1.38.20;
Change type of cpu_uvm to struct uvm_cpu *

MI IPI interface:
- Implement support for the asynchronous IPI calls.
- Rework synchronous IPI code to reuse the asynchronous mechanism.
- Add ipi(9) manual page; needs wizd(8).

Note: MD code can now provide a low level primitive for the ipi(9) and
reuse this interface instead of open-coding. Portmasters are encouraged
to convert. Ride 6.99.43!

branches: 1.36.2;
excise unused qchain

branches: 1.35.2; 1.35.4;
Add a kcpuset_t which just includes ourself.
Add a ci_cpuname for convenience

- Add mi_cpu_init() and initialise cpu_lock and kcpuset_attached/running there.
- Add kcpuset_running which gets set in idle_loop().
- Use kcpuset_running in pserialize_perform().

branches: 1.33.4; 1.33.8;
Add PCU (Per-CPU Unit) - an interface to manage synchronization of any
per CPU context tied with an LWP. Main use - lazy FPU handling on SMP.

Requested by matt@, will be used on mips64. Note: implementation will
be improved to use IPIs before adopting on x86. OK ad@.

branches: 1.32.2; 1.32.4;
Move counting of faults, traps, intrs, soft[intr]s, syscalls, and nswtch
from uvmexp to per-cpu cpu_data and move them to 64bits. Remove unneeded
includes of <uvm/uvm_extern.h> and/or <uvm/uvm.h>.

Rename ci_node_id to ci_package_id, as some claim that the former might
be confused with NUMA node.

Add node/core/smt ID numbers to struct cpu_data, will be filled by MD.
OK matt@.

Make select/poll work with more than 32 CPUs.
No ABI change.

branches: 1.28.2; 1.28.4;
introduce a new function that returns a unique string for each cpu:

char *cpu_name(struct cpu_info *);

and use it when setting up the runq event counters, avoiding an 8 byte
kmem(4) allocation for each cpu. there are more places the cpuname is
used that can be converted to using this new interface, but that can
and will be done as future work.

as discussed with rmind.

branches: 1.27.16; 1.27.20;
vfs_cache:

- Don't use goto in critical paths, it can confuse the compiler.
- Sprinkle some branch hints.
- Make namecache stats per-CPU and collate once per second.
- Use vtryget().

Make trap counters per-cpu, like syscalls.

+ uint64_t cpu_cc_freq; /* cycle counter frequency */
+ int64_t cpu_cc_skew; /* counter skew vs cpu0 */

branches: 1.24.2;
LOCKDEBUG: try to speed it up a bit by not using so much global state.

This will break the build briefly but will be followed by another commit
to fix that..

Remove clause 3 and 4 from TNF licenses

branches: 1.22.2;
Implement MP callouts as discussed on tech-kern. The CPU binding code is
disabled for the moment until we figure out what we want to do with CPUs
being offlined.

branches: 1.21.2;
Maintain a circular queue of cpu_info's.

Restructure the name cache code to eliminate most lock contention
resulting from forward lookups. Discussed on tech-kern@.

Commit the "per-CPU" select patch. This is the result of much work and
testing by rmind@ and myself.

Which approach to use is still being discussed, but I would like to get
this out of my working tree. If we decide to use a different approach
there is no problem with revisiting this.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Make context switch + syscall counters optionally per-CPU and accumulate
in schedclock() at "about 16 hz".

branches: 1.16.2; 1.16.6;
remove a special allocator for uareas, which is no longer necessary.
use pool_cache instead.

add a per-cpu storage allocator.

branches: 1.14.2; 1.14.6;
Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
number and type of priority levels into bands. Add new bands like
'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

branches: 1.13.2; 1.13.4;
Merge file descriptor locking, cwdi locking and cross-call changes
from the vmlocking branch.

branches: 1.12.2; 1.12.4;
Make the uarea cache per-CPU and drain in batches of 4.

branches: 1.11.2;
cpu_index doesn't need to be a cpuid_t, since it's a cardinal

Improve per-CPU support for the workqueue(9):
- Make structures CPU-cache friendly, as suggested and explained
by Andrew Doran. CACHE_LINE_SIZE definition is invented.
- Use current CPU if NULL is passed to the workqueue_enqueue().
- Implemented MI CPU index, which could be used as an index of array.
Removed linked-lists usage for work queues.

The roundup2() function avoids division, but works only with power of 2.

Reviewed by: <ad>, <yamt>, <tech-kern>

branches: 1.9.2; 1.9.6;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

branches: 1.7.2; 1.7.6; 1.7.8;
Merge newlock2 to head.

branches: 1.6.2;
Add the 'lockstat' driver, which collects basic information about lock events.

branches: 1.5.4; 1.5.8;
changes for making DIAGNOSTIC not change the kernel ABI:
- for structure fields that are conditionally present,
make those fields always present.
- for functions which are conditionally inline, make them never inline.
- remove some other functions which are conditionally defined but
don't actually do anything anymore.
- make a lock-debugging function conditional on only LOCKDEBUG.

as discussed on tech-kern some time back.

Make sure that multiple header inclusion is done, it uses consistent names,
and it is around the whole code of the file, not parts of it.

branches: 1.3.12;
a relatively lightweight implementation of kernel_lock.

remove an unneeded #include.

branches: 1.1.2;
move some per-cpu data definitions to MI place so that they can be modified
without touching all ports. discussed on tech-kern@.

heartbeat(9): New mechanism to check progress of kernel.

This uses hard interrupts to check progress of low-priority soft
interrupts, and one CPU to check progress of another CPU.

If no progress has been made after a configurable number of seconds
(kern.heartbeat.max_period, default 15), then the system panics --
preferably on the CPU that is stuck so we get a stack trace in dmesg
of where it was stuck, but if the stuckness was detected by another
CPU and the stuck CPU doesn't acknowledge the request to panic within
one second, the detecting CPU panics instead.

This doesn't supplant hardware watchdog timers. It is possible for
hard interrupts to be stuck on all CPUs for some reason too; in that
case heartbeat(9) has no opportunity to complete.

Downside: heartbeat(9) relies on hardclock to run at a reasonably
consistent rate, which might cause trouble for the glorious tickless
future. However, it could be adapted to take a parameter for an
approximate number of units that have elapsed since the last call on
the current CPU, rather than treating that as a constant 1.

XXX kernel revbump -- changes struct cpu_info layout

Remove PG_ZERO. It worked brilliantly on x86 machines from the mid-90s but
having spent an age experimenting with it over the last 6 months on various
machines and with different use cases it's always either break-even or a
slight net loss for me.

Counter tweaks:

- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
each kind of page can be summed to get the totals.

- Track the number of free pages with a counter so that it's one less thing
for the allocator to do, which opens up further options there.

- Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot.
For the cheap option, give cpu_count_sync() a boolean parameter indicating
that a cached value is okay, and rate limit the updates for cached values
to hz.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

cpu topology:

- Fix a stupid bug where it complained about non-existent SMT on assymetric
systems.

- Give each CPU a pointer back to the first CPU in same package.

- Many small tweaks to the SMT awareness in the scheduler. It does a much
better job now at keeping all physical CPUs busy, while using the extra
threads to help out. In particular, during preempt() if we're using SMT,
try to find a better CPU to run on and teleport curlwp there.

- Change the CPU topology stuff so it can work on asymmetric systems. This
mainly entails rearranging one of the CPU lists so it makes sense in all
configurations.

- Add a parameter to cpu_topology_set() to note that a CPU is "slow", for
where there are fast CPUs and slow CPUs, like with the Rockwell RK3399.
Extend the SMT awareness to try and handle that situation too (keep fast
CPUs busy, use slow CPUs as helpers).

branches: 1.46.2;
Counter tweaks:

"zeroaborts" + "free" don't need to be per-CPU counters, and "bucketmiss"
wasn't used. Remove those and cluster by usage.

Some more CPU topology stuff:

- Use cegger@'s ACPI SRAT parsing code to figure out NUMA node ID for each
CPU as it is attached.

- For scheduler experiments with SMT, flag CPUs with the lowest numbered SMT
IDs as "primaries", link back to the primaries from secondaries, and build
a circular list of CPUs in each package with identical SMT IDs.

- No need for package/core/smt/numa IDs to be anything other than a u_int.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

- Add some more failsafes to the CPU topology stuff, and build a 3rd
circular list of peer CPUs in other packages, so we might scroll through
them in the scheduler when looking to distribute or steal jobs.

- Fold the run queue data structure into spc_schedstate. Makes kern_runq.c
a far more pleasant place to work.

- Remove the code in sched_nextlwp() that tries to steal jobs from other
CPUs. It's not needed, because we do the very same thing in the idle LWP
anyway. Outside the VM system this was one of the the main causes of L3
cache misses I saw during builds. On my machine, this change yields a
60%-70% drop in time on the "hackbench" benchmark (there's clearly a bit
more going on here, but basically being less aggressive helps).

Rip out pserialize(9) logic now that the RCU patent has expired.

pserialize_perform() is now basically just xc_barrier(XC_HIGHPRI).
No more tentacles throughout the scheduler. Simplify the psz read
count for diagnostic assertions by putting it unconditionally into
cpu_info.

From rmind@, tidied up by me.

Take the basic CPU topology information we already collect, and use it
to make circular lists of CPU siblings in the same core, and in the
same package. Nothing fancy, just enough to have a bit of fun in the
scheduler trying out different tactics.

Fix false sharing problems with cpu_info. Identified with tprof(8).
This was a very nice win in my tests on a 48 CPU box.

- Reorganise cpu_data slightly according to usage.
- Put cpu_onproc into struct cpu_info alongside ci_curlwp (now is ci_onproc).
- On x86, put some items in their own cache lines according to usage, like
the IPI bitmask and ci_want_resched.

Remove stale comment. There are more than just compat definitions.

branches: 1.38.18; 1.38.20;
Change type of cpu_uvm to struct uvm_cpu *

MI IPI interface:
- Implement support for the asynchronous IPI calls.
- Rework synchronous IPI code to reuse the asynchronous mechanism.
- Add ipi(9) manual page; needs wizd(8).

Note: MD code can now provide a low level primitive for the ipi(9) and
reuse this interface instead of open-coding. Portmasters are encouraged
to convert. Ride 6.99.43!

branches: 1.36.2;
excise unused qchain

branches: 1.35.2; 1.35.4;
Add a kcpuset_t which just includes ourself.
Add a ci_cpuname for convenience

- Add mi_cpu_init() and initialise cpu_lock and kcpuset_attached/running there.
- Add kcpuset_running which gets set in idle_loop().
- Use kcpuset_running in pserialize_perform().

branches: 1.33.4; 1.33.8;
Add PCU (Per-CPU Unit) - an interface to manage synchronization of any
per CPU context tied with an LWP. Main use - lazy FPU handling on SMP.

Requested by matt@, will be used on mips64. Note: implementation will
be improved to use IPIs before adopting on x86. OK ad@.

branches: 1.32.2; 1.32.4;
Move counting of faults, traps, intrs, soft[intr]s, syscalls, and nswtch
from uvmexp to per-cpu cpu_data and move them to 64bits. Remove unneeded
includes of <uvm/uvm_extern.h> and/or <uvm/uvm.h>.

Rename ci_node_id to ci_package_id, as some claim that the former might
be confused with NUMA node.

Add node/core/smt ID numbers to struct cpu_data, will be filled by MD.
OK matt@.

Make select/poll work with more than 32 CPUs.
No ABI change.

branches: 1.28.2; 1.28.4;
introduce a new function that returns a unique string for each cpu:

char *cpu_name(struct cpu_info *);

and use it when setting up the runq event counters, avoiding an 8 byte
kmem(4) allocation for each cpu. there are more places the cpuname is
used that can be converted to using this new interface, but that can
and will be done as future work.

as discussed with rmind.

branches: 1.27.16; 1.27.20;
vfs_cache:

- Don't use goto in critical paths, it can confuse the compiler.
- Sprinkle some branch hints.
- Make namecache stats per-CPU and collate once per second.
- Use vtryget().

Make trap counters per-cpu, like syscalls.

+ uint64_t cpu_cc_freq; /* cycle counter frequency */
+ int64_t cpu_cc_skew; /* counter skew vs cpu0 */

branches: 1.24.2;
LOCKDEBUG: try to speed it up a bit by not using so much global state.

This will break the build briefly but will be followed by another commit
to fix that..

Remove clause 3 and 4 from TNF licenses

branches: 1.22.2;
Implement MP callouts as discussed on tech-kern. The CPU binding code is
disabled for the moment until we figure out what we want to do with CPUs
being offlined.

branches: 1.21.2;
Maintain a circular queue of cpu_info's.

Restructure the name cache code to eliminate most lock contention
resulting from forward lookups. Discussed on tech-kern@.

Commit the "per-CPU" select patch. This is the result of much work and
testing by rmind@ and myself.

Which approach to use is still being discussed, but I would like to get
this out of my working tree. If we decide to use a different approach
there is no problem with revisiting this.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Make context switch + syscall counters optionally per-CPU and accumulate
in schedclock() at "about 16 hz".

branches: 1.16.2; 1.16.6;
remove a special allocator for uareas, which is no longer necessary.
use pool_cache instead.

add a per-cpu storage allocator.

branches: 1.14.2; 1.14.6;
Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
number and type of priority levels into bands. Add new bands like
'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

branches: 1.13.2; 1.13.4;
Merge file descriptor locking, cwdi locking and cross-call changes
from the vmlocking branch.

branches: 1.12.2; 1.12.4;
Make the uarea cache per-CPU and drain in batches of 4.

branches: 1.11.2;
cpu_index doesn't need to be a cpuid_t, since it's a cardinal

Improve per-CPU support for the workqueue(9):
- Make structures CPU-cache friendly, as suggested and explained
by Andrew Doran. CACHE_LINE_SIZE definition is invented.
- Use current CPU if NULL is passed to the workqueue_enqueue().
- Implemented MI CPU index, which could be used as an index of array.
Removed linked-lists usage for work queues.

The roundup2() function avoids division, but works only with power of 2.

Reviewed by: <ad>, <yamt>, <tech-kern>

branches: 1.9.2; 1.9.6;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

branches: 1.7.2; 1.7.6; 1.7.8;
Merge newlock2 to head.

branches: 1.6.2;
Add the 'lockstat' driver, which collects basic information about lock events.

branches: 1.5.4; 1.5.8;
changes for making DIAGNOSTIC not change the kernel ABI:
- for structure fields that are conditionally present,
make those fields always present.
- for functions which are conditionally inline, make them never inline.
- remove some other functions which are conditionally defined but
don't actually do anything anymore.
- make a lock-debugging function conditional on only LOCKDEBUG.

as discussed on tech-kern some time back.

Make sure that multiple header inclusion is done, it uses consistent names,
and it is around the whole code of the file, not parts of it.

branches: 1.3.12;
a relatively lightweight implementation of kernel_lock.

remove an unneeded #include.

branches: 1.1.2;
move some per-cpu data definitions to MI place so that they can be modified
without touching all ports. discussed on tech-kern@.

Remove PG_ZERO. It worked brilliantly on x86 machines from the mid-90s but
having spent an age experimenting with it over the last 6 months on various
machines and with different use cases it's always either break-even or a
slight net loss for me.

Counter tweaks:

- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
each kind of page can be summed to get the totals.

- Track the number of free pages with a counter so that it's one less thing
for the allocator to do, which opens up further options there.

- Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot.
For the cheap option, give cpu_count_sync() a boolean parameter indicating
that a cached value is okay, and rate limit the updates for cached values
to hz.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

cpu topology:

- Fix a stupid bug where it complained about non-existent SMT on assymetric
systems.

- Give each CPU a pointer back to the first CPU in same package.

- Many small tweaks to the SMT awareness in the scheduler. It does a much
better job now at keeping all physical CPUs busy, while using the extra
threads to help out. In particular, during preempt() if we're using SMT,
try to find a better CPU to run on and teleport curlwp there.

- Change the CPU topology stuff so it can work on asymmetric systems. This
mainly entails rearranging one of the CPU lists so it makes sense in all
configurations.

- Add a parameter to cpu_topology_set() to note that a CPU is "slow", for
where there are fast CPUs and slow CPUs, like with the Rockwell RK3399.
Extend the SMT awareness to try and handle that situation too (keep fast
CPUs busy, use slow CPUs as helpers).

branches: 1.46.2;
Counter tweaks:

"zeroaborts" + "free" don't need to be per-CPU counters, and "bucketmiss"
wasn't used. Remove those and cluster by usage.

Some more CPU topology stuff:

- Use cegger@'s ACPI SRAT parsing code to figure out NUMA node ID for each
CPU as it is attached.

- For scheduler experiments with SMT, flag CPUs with the lowest numbered SMT
IDs as "primaries", link back to the primaries from secondaries, and build
a circular list of CPUs in each package with identical SMT IDs.

- No need for package/core/smt/numa IDs to be anything other than a u_int.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

- Add some more failsafes to the CPU topology stuff, and build a 3rd
circular list of peer CPUs in other packages, so we might scroll through
them in the scheduler when looking to distribute or steal jobs.

- Fold the run queue data structure into spc_schedstate. Makes kern_runq.c
a far more pleasant place to work.

- Remove the code in sched_nextlwp() that tries to steal jobs from other
CPUs. It's not needed, because we do the very same thing in the idle LWP
anyway. Outside the VM system this was one of the the main causes of L3
cache misses I saw during builds. On my machine, this change yields a
60%-70% drop in time on the "hackbench" benchmark (there's clearly a bit
more going on here, but basically being less aggressive helps).

Rip out pserialize(9) logic now that the RCU patent has expired.

pserialize_perform() is now basically just xc_barrier(XC_HIGHPRI).
No more tentacles throughout the scheduler. Simplify the psz read
count for diagnostic assertions by putting it unconditionally into
cpu_info.

From rmind@, tidied up by me.

Take the basic CPU topology information we already collect, and use it
to make circular lists of CPU siblings in the same core, and in the
same package. Nothing fancy, just enough to have a bit of fun in the
scheduler trying out different tactics.

Fix false sharing problems with cpu_info. Identified with tprof(8).
This was a very nice win in my tests on a 48 CPU box.

- Reorganise cpu_data slightly according to usage.
- Put cpu_onproc into struct cpu_info alongside ci_curlwp (now is ci_onproc).
- On x86, put some items in their own cache lines according to usage, like
the IPI bitmask and ci_want_resched.

Remove stale comment. There are more than just compat definitions.

branches: 1.38.18; 1.38.20;
Change type of cpu_uvm to struct uvm_cpu *

MI IPI interface:
- Implement support for the asynchronous IPI calls.
- Rework synchronous IPI code to reuse the asynchronous mechanism.
- Add ipi(9) manual page; needs wizd(8).

Note: MD code can now provide a low level primitive for the ipi(9) and
reuse this interface instead of open-coding. Portmasters are encouraged
to convert. Ride 6.99.43!

branches: 1.36.2;
excise unused qchain

branches: 1.35.2; 1.35.4;
Add a kcpuset_t which just includes ourself.
Add a ci_cpuname for convenience

- Add mi_cpu_init() and initialise cpu_lock and kcpuset_attached/running there.
- Add kcpuset_running which gets set in idle_loop().
- Use kcpuset_running in pserialize_perform().

branches: 1.33.4; 1.33.8;
Add PCU (Per-CPU Unit) - an interface to manage synchronization of any
per CPU context tied with an LWP. Main use - lazy FPU handling on SMP.

Requested by matt@, will be used on mips64. Note: implementation will
be improved to use IPIs before adopting on x86. OK ad@.

branches: 1.32.2; 1.32.4;
Move counting of faults, traps, intrs, soft[intr]s, syscalls, and nswtch
from uvmexp to per-cpu cpu_data and move them to 64bits. Remove unneeded
includes of <uvm/uvm_extern.h> and/or <uvm/uvm.h>.

Rename ci_node_id to ci_package_id, as some claim that the former might
be confused with NUMA node.

Add node/core/smt ID numbers to struct cpu_data, will be filled by MD.
OK matt@.

Make select/poll work with more than 32 CPUs.
No ABI change.

branches: 1.28.2; 1.28.4;
introduce a new function that returns a unique string for each cpu:

char *cpu_name(struct cpu_info *);

and use it when setting up the runq event counters, avoiding an 8 byte
kmem(4) allocation for each cpu. there are more places the cpuname is
used that can be converted to using this new interface, but that can
and will be done as future work.

as discussed with rmind.

branches: 1.27.16; 1.27.20;
vfs_cache:

- Don't use goto in critical paths, it can confuse the compiler.
- Sprinkle some branch hints.
- Make namecache stats per-CPU and collate once per second.
- Use vtryget().

Make trap counters per-cpu, like syscalls.

+ uint64_t cpu_cc_freq; /* cycle counter frequency */
+ int64_t cpu_cc_skew; /* counter skew vs cpu0 */

branches: 1.24.2;
LOCKDEBUG: try to speed it up a bit by not using so much global state.

This will break the build briefly but will be followed by another commit
to fix that..

Remove clause 3 and 4 from TNF licenses

branches: 1.22.2;
Implement MP callouts as discussed on tech-kern. The CPU binding code is
disabled for the moment until we figure out what we want to do with CPUs
being offlined.

branches: 1.21.2;
Maintain a circular queue of cpu_info's.

Restructure the name cache code to eliminate most lock contention
resulting from forward lookups. Discussed on tech-kern@.

Commit the "per-CPU" select patch. This is the result of much work and
testing by rmind@ and myself.

Which approach to use is still being discussed, but I would like to get
this out of my working tree. If we decide to use a different approach
there is no problem with revisiting this.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Make context switch + syscall counters optionally per-CPU and accumulate
in schedclock() at "about 16 hz".

branches: 1.16.2; 1.16.6;
remove a special allocator for uareas, which is no longer necessary.
use pool_cache instead.

add a per-cpu storage allocator.

branches: 1.14.2; 1.14.6;
Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
number and type of priority levels into bands. Add new bands like
'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

branches: 1.13.2; 1.13.4;
Merge file descriptor locking, cwdi locking and cross-call changes
from the vmlocking branch.

branches: 1.12.2; 1.12.4;
Make the uarea cache per-CPU and drain in batches of 4.

branches: 1.11.2;
cpu_index doesn't need to be a cpuid_t, since it's a cardinal

Improve per-CPU support for the workqueue(9):
- Make structures CPU-cache friendly, as suggested and explained
by Andrew Doran. CACHE_LINE_SIZE definition is invented.
- Use current CPU if NULL is passed to the workqueue_enqueue().
- Implemented MI CPU index, which could be used as an index of array.
Removed linked-lists usage for work queues.

The roundup2() function avoids division, but works only with power of 2.

Reviewed by: <ad>, <yamt>, <tech-kern>

branches: 1.9.2; 1.9.6;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

branches: 1.7.2; 1.7.6; 1.7.8;
Merge newlock2 to head.

branches: 1.6.2;
Add the 'lockstat' driver, which collects basic information about lock events.

branches: 1.5.4; 1.5.8;
changes for making DIAGNOSTIC not change the kernel ABI:
- for structure fields that are conditionally present,
make those fields always present.
- for functions which are conditionally inline, make them never inline.
- remove some other functions which are conditionally defined but
don't actually do anything anymore.
- make a lock-debugging function conditional on only LOCKDEBUG.

as discussed on tech-kern some time back.

Make sure that multiple header inclusion is done, it uses consistent names,
and it is around the whole code of the file, not parts of it.

branches: 1.3.12;
a relatively lightweight implementation of kernel_lock.

remove an unneeded #include.

branches: 1.1.2;
move some per-cpu data definitions to MI place so that they can be modified
without touching all ports. discussed on tech-kern@.

Counter tweaks:

- Don't need to count anonpages+filepages any more; clean+unknown+dirty for
each kind of page can be summed to get the totals.

- Track the number of free pages with a counter so that it's one less thing
for the allocator to do, which opens up further options there.

- Remove cpu_count_sync_one(). It has no users and doesn't save a whole lot.
For the cheap option, give cpu_count_sync() a boolean parameter indicating
that a cached value is okay, and rate limit the updates for cached values
to hz.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

cpu topology:

- Fix a stupid bug where it complained about non-existent SMT on assymetric
systems.

- Give each CPU a pointer back to the first CPU in same package.

- Many small tweaks to the SMT awareness in the scheduler. It does a much
better job now at keeping all physical CPUs busy, while using the extra
threads to help out. In particular, during preempt() if we're using SMT,
try to find a better CPU to run on and teleport curlwp there.

- Change the CPU topology stuff so it can work on asymmetric systems. This
mainly entails rearranging one of the CPU lists so it makes sense in all
configurations.

- Add a parameter to cpu_topology_set() to note that a CPU is "slow", for
where there are fast CPUs and slow CPUs, like with the Rockwell RK3399.
Extend the SMT awareness to try and handle that situation too (keep fast
CPUs busy, use slow CPUs as helpers).

branches: 1.46.2;
Counter tweaks:

"zeroaborts" + "free" don't need to be per-CPU counters, and "bucketmiss"
wasn't used. Remove those and cluster by usage.

Some more CPU topology stuff:

- Use cegger@'s ACPI SRAT parsing code to figure out NUMA node ID for each
CPU as it is attached.

- For scheduler experiments with SMT, flag CPUs with the lowest numbered SMT
IDs as "primaries", link back to the primaries from secondaries, and build
a circular list of CPUs in each package with identical SMT IDs.

- No need for package/core/smt/numa IDs to be anything other than a u_int.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

- Add some more failsafes to the CPU topology stuff, and build a 3rd
circular list of peer CPUs in other packages, so we might scroll through
them in the scheduler when looking to distribute or steal jobs.

- Fold the run queue data structure into spc_schedstate. Makes kern_runq.c
a far more pleasant place to work.

- Remove the code in sched_nextlwp() that tries to steal jobs from other
CPUs. It's not needed, because we do the very same thing in the idle LWP
anyway. Outside the VM system this was one of the the main causes of L3
cache misses I saw during builds. On my machine, this change yields a
60%-70% drop in time on the "hackbench" benchmark (there's clearly a bit
more going on here, but basically being less aggressive helps).

Rip out pserialize(9) logic now that the RCU patent has expired.

pserialize_perform() is now basically just xc_barrier(XC_HIGHPRI).
No more tentacles throughout the scheduler. Simplify the psz read
count for diagnostic assertions by putting it unconditionally into
cpu_info.

From rmind@, tidied up by me.

Take the basic CPU topology information we already collect, and use it
to make circular lists of CPU siblings in the same core, and in the
same package. Nothing fancy, just enough to have a bit of fun in the
scheduler trying out different tactics.

Fix false sharing problems with cpu_info. Identified with tprof(8).
This was a very nice win in my tests on a 48 CPU box.

- Reorganise cpu_data slightly according to usage.
- Put cpu_onproc into struct cpu_info alongside ci_curlwp (now is ci_onproc).
- On x86, put some items in their own cache lines according to usage, like
the IPI bitmask and ci_want_resched.

Remove stale comment. There are more than just compat definitions.

branches: 1.38.18; 1.38.20;
Change type of cpu_uvm to struct uvm_cpu *

MI IPI interface:
- Implement support for the asynchronous IPI calls.
- Rework synchronous IPI code to reuse the asynchronous mechanism.
- Add ipi(9) manual page; needs wizd(8).

Note: MD code can now provide a low level primitive for the ipi(9) and
reuse this interface instead of open-coding. Portmasters are encouraged
to convert. Ride 6.99.43!

branches: 1.36.2;
excise unused qchain

branches: 1.35.2; 1.35.4;
Add a kcpuset_t which just includes ourself.
Add a ci_cpuname for convenience

- Add mi_cpu_init() and initialise cpu_lock and kcpuset_attached/running there.
- Add kcpuset_running which gets set in idle_loop().
- Use kcpuset_running in pserialize_perform().

branches: 1.33.4; 1.33.8;
Add PCU (Per-CPU Unit) - an interface to manage synchronization of any
per CPU context tied with an LWP. Main use - lazy FPU handling on SMP.

Requested by matt@, will be used on mips64. Note: implementation will
be improved to use IPIs before adopting on x86. OK ad@.

branches: 1.32.2; 1.32.4;
Move counting of faults, traps, intrs, soft[intr]s, syscalls, and nswtch
from uvmexp to per-cpu cpu_data and move them to 64bits. Remove unneeded
includes of <uvm/uvm_extern.h> and/or <uvm/uvm.h>.

Rename ci_node_id to ci_package_id, as some claim that the former might
be confused with NUMA node.

Add node/core/smt ID numbers to struct cpu_data, will be filled by MD.
OK matt@.

Make select/poll work with more than 32 CPUs.
No ABI change.

branches: 1.28.2; 1.28.4;
introduce a new function that returns a unique string for each cpu:

char *cpu_name(struct cpu_info *);

and use it when setting up the runq event counters, avoiding an 8 byte
kmem(4) allocation for each cpu. there are more places the cpuname is
used that can be converted to using this new interface, but that can
and will be done as future work.

as discussed with rmind.

branches: 1.27.16; 1.27.20;
vfs_cache:

- Don't use goto in critical paths, it can confuse the compiler.
- Sprinkle some branch hints.
- Make namecache stats per-CPU and collate once per second.
- Use vtryget().

Make trap counters per-cpu, like syscalls.

+ uint64_t cpu_cc_freq; /* cycle counter frequency */
+ int64_t cpu_cc_skew; /* counter skew vs cpu0 */

branches: 1.24.2;
LOCKDEBUG: try to speed it up a bit by not using so much global state.

This will break the build briefly but will be followed by another commit
to fix that..

Remove clause 3 and 4 from TNF licenses

branches: 1.22.2;
Implement MP callouts as discussed on tech-kern. The CPU binding code is
disabled for the moment until we figure out what we want to do with CPUs
being offlined.

branches: 1.21.2;
Maintain a circular queue of cpu_info's.

Restructure the name cache code to eliminate most lock contention
resulting from forward lookups. Discussed on tech-kern@.

Commit the "per-CPU" select patch. This is the result of much work and
testing by rmind@ and myself.

Which approach to use is still being discussed, but I would like to get
this out of my working tree. If we decide to use a different approach
there is no problem with revisiting this.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Make context switch + syscall counters optionally per-CPU and accumulate
in schedclock() at "about 16 hz".

branches: 1.16.2; 1.16.6;
remove a special allocator for uareas, which is no longer necessary.
use pool_cache instead.

add a per-cpu storage allocator.

branches: 1.14.2; 1.14.6;
Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
number and type of priority levels into bands. Add new bands like
'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

branches: 1.13.2; 1.13.4;
Merge file descriptor locking, cwdi locking and cross-call changes
from the vmlocking branch.

branches: 1.12.2; 1.12.4;
Make the uarea cache per-CPU and drain in batches of 4.

branches: 1.11.2;
cpu_index doesn't need to be a cpuid_t, since it's a cardinal

Improve per-CPU support for the workqueue(9):
- Make structures CPU-cache friendly, as suggested and explained
by Andrew Doran. CACHE_LINE_SIZE definition is invented.
- Use current CPU if NULL is passed to the workqueue_enqueue().
- Implemented MI CPU index, which could be used as an index of array.
Removed linked-lists usage for work queues.

The roundup2() function avoids division, but works only with power of 2.

Reviewed by: <ad>, <yamt>, <tech-kern>

branches: 1.9.2; 1.9.6;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

branches: 1.7.2; 1.7.6; 1.7.8;
Merge newlock2 to head.

branches: 1.6.2;
Add the 'lockstat' driver, which collects basic information about lock events.

branches: 1.5.4; 1.5.8;
changes for making DIAGNOSTIC not change the kernel ABI:
- for structure fields that are conditionally present,
make those fields always present.
- for functions which are conditionally inline, make them never inline.
- remove some other functions which are conditionally defined but
don't actually do anything anymore.
- make a lock-debugging function conditional on only LOCKDEBUG.

as discussed on tech-kern some time back.

Make sure that multiple header inclusion is done, it uses consistent names,
and it is around the whole code of the file, not parts of it.

branches: 1.3.12;
a relatively lightweight implementation of kernel_lock.

remove an unneeded #include.

branches: 1.1.2;
move some per-cpu data definitions to MI place so that they can be modified
without touching all ports. discussed on tech-kern@.

Process concurrent page faults on individual uvm_objects / vm_amaps in
parallel, where the relevant pages are already in-core. Proposed on
tech-kern.

Temporarily disabled on MP architectures with __HAVE_UNLOCKED_PMAP until
adjustments are made to their pmaps.

Merge from yamt-pagecache (after much testing):

- Reduce unnecessary page scan in putpages esp. when an object has a ton of
pages cached but only a few of them are dirty.

- Reduce the number of pmap operations by tracking page dirtiness more
precisely in uvm layer.

cpu topology:

- Fix a stupid bug where it complained about non-existent SMT on assymetric
systems.

- Give each CPU a pointer back to the first CPU in same package.

- Many small tweaks to the SMT awareness in the scheduler. It does a much
better job now at keeping all physical CPUs busy, while using the extra
threads to help out. In particular, during preempt() if we're using SMT,
try to find a better CPU to run on and teleport curlwp there.

- Change the CPU topology stuff so it can work on asymmetric systems. This
mainly entails rearranging one of the CPU lists so it makes sense in all
configurations.

- Add a parameter to cpu_topology_set() to note that a CPU is "slow", for
where there are fast CPUs and slow CPUs, like with the Rockwell RK3399.
Extend the SMT awareness to try and handle that situation too (keep fast
CPUs busy, use slow CPUs as helpers).

branches: 1.46.2;
Counter tweaks:

"zeroaborts" + "free" don't need to be per-CPU counters, and "bucketmiss"
wasn't used. Remove those and cluster by usage.

Some more CPU topology stuff:

- Use cegger@'s ACPI SRAT parsing code to figure out NUMA node ID for each
CPU as it is attached.

- For scheduler experiments with SMT, flag CPUs with the lowest numbered SMT
IDs as "primaries", link back to the primaries from secondaries, and build
a circular list of CPUs in each package with identical SMT IDs.

- No need for package/core/smt/numa IDs to be anything other than a u_int.

- Extend the per-CPU counters matt@ did to include all of the hot counters
in UVM, excluding uvmexp.free, which needs special treatment and will be
done with a separate commit. Cuts system time for a build by 20-25% on
a 48 CPU machine w/DIAGNOSTIC.

- Avoid 64-bit integer divide on every fault (for rnd_add_uint32).

- Add some more failsafes to the CPU topology stuff, and build a 3rd
circular list of peer CPUs in other packages, so we might scroll through
them in the scheduler when looking to distribute or steal jobs.

- Fold the run queue data structure into spc_schedstate. Makes kern_runq.c
a far more pleasant place to work.

- Remove the code in sched_nextlwp() that tries to steal jobs from other
CPUs. It's not needed, because we do the very same thing in the idle LWP
anyway. Outside the VM system this was one of the the main causes of L3
cache misses I saw during builds. On my machine, this change yields a
60%-70% drop in time on the "hackbench" benchmark (there's clearly a bit
more going on here, but basically being less aggressive helps).

Rip out pserialize(9) logic now that the RCU patent has expired.

pserialize_perform() is now basically just xc_barrier(XC_HIGHPRI).
No more tentacles throughout the scheduler. Simplify the psz read
count for diagnostic assertions by putting it unconditionally into
cpu_info.

From rmind@, tidied up by me.

Take the basic CPU topology information we already collect, and use it
to make circular lists of CPU siblings in the same core, and in the
same package. Nothing fancy, just enough to have a bit of fun in the
scheduler trying out different tactics.

Fix false sharing problems with cpu_info. Identified with tprof(8).
This was a very nice win in my tests on a 48 CPU box.

- Reorganise cpu_data slightly according to usage.
- Put cpu_onproc into struct cpu_info alongside ci_curlwp (now is ci_onproc).
- On x86, put some items in their own cache lines according to usage, like
the IPI bitmask and ci_want_resched.

Remove stale comment. There are more than just compat definitions.

branches: 1.38.18; 1.38.20;
Change type of cpu_uvm to struct uvm_cpu *

MI IPI interface:
- Implement support for the asynchronous IPI calls.
- Rework synchronous IPI code to reuse the asynchronous mechanism.
- Add ipi(9) manual page; needs wizd(8).

Note: MD code can now provide a low level primitive for the ipi(9) and
reuse this interface instead of open-coding. Portmasters are encouraged
to convert. Ride 6.99.43!

branches: 1.36.2;
excise unused qchain

branches: 1.35.2; 1.35.4;
Add a kcpuset_t which just includes ourself.
Add a ci_cpuname for convenience

- Add mi_cpu_init() and initialise cpu_lock and kcpuset_attached/running there.
- Add kcpuset_running which gets set in idle_loop().
- Use kcpuset_running in pserialize_perform().

branches: 1.33.4; 1.33.8;
Add PCU (Per-CPU Unit) - an interface to manage synchronization of any
per CPU context tied with an LWP. Main use - lazy FPU handling on SMP.

Requested by matt@, will be used on mips64. Note: implementation will
be improved to use IPIs before adopting on x86. OK ad@.

branches: 1.32.2; 1.32.4;
Move counting of faults, traps, intrs, soft[intr]s, syscalls, and nswtch
from uvmexp to per-cpu cpu_data and move them to 64bits. Remove unneeded
includes of <uvm/uvm_extern.h> and/or <uvm/uvm.h>.

Rename ci_node_id to ci_package_id, as some claim that the former might
be confused with NUMA node.

Add node/core/smt ID numbers to struct cpu_data, will be filled by MD.
OK matt@.

Make select/poll work with more than 32 CPUs.
No ABI change.

branches: 1.28.2; 1.28.4;
introduce a new function that returns a unique string for each cpu:

char *cpu_name(struct cpu_info *);

and use it when setting up the runq event counters, avoiding an 8 byte
kmem(4) allocation for each cpu. there are more places the cpuname is
used that can be converted to using this new interface, but that can
and will be done as future work.

as discussed with rmind.

branches: 1.27.16; 1.27.20;
vfs_cache:

- Don't use goto in critical paths, it can confuse the compiler.
- Sprinkle some branch hints.
- Make namecache stats per-CPU and collate once per second.
- Use vtryget().

Make trap counters per-cpu, like syscalls.

+ uint64_t cpu_cc_freq; /* cycle counter frequency */
+ int64_t cpu_cc_skew; /* counter skew vs cpu0 */

branches: 1.24.2;
LOCKDEBUG: try to speed it up a bit by not using so much global state.

This will break the build briefly but will be followed by another commit
to fix that..

Remove clause 3 and 4 from TNF licenses

branches: 1.22.2;
Implement MP callouts as discussed on tech-kern. The CPU binding code is
disabled for the moment until we figure out what we want to do with CPUs
being offlined.

branches: 1.21.2;
Maintain a circular queue of cpu_info's.

Restructure the name cache code to eliminate most lock contention
resulting from forward lookups. Discussed on tech-kern@.

Commit the "per-CPU" select patch. This is the result of much work and
testing by rmind@ and myself.

Which approach to use is still being discussed, but I would like to get
this out of my working tree. If we decide to use a different approach
there is no problem with revisiting this.

Catch up with descriptor handling changes. See kern_descrip.c revision
1.173 for details.

Make context switch + syscall counters optionally per-CPU and accumulate
in schedclock() at "about 16 hz".

branches: 1.16.2; 1.16.6;
remove a special allocator for uareas, which is no longer necessary.
use pool_cache instead.

add a per-cpu storage allocator.

branches: 1.14.2; 1.14.6;
Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
number and type of priority levels into bands. Add new bands like
'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

branches: 1.13.2; 1.13.4;
Merge file descriptor locking, cwdi locking and cross-call changes
from the vmlocking branch.

branches: 1.12.2; 1.12.4;
Make the uarea cache per-CPU and drain in batches of 4.

branches: 1.11.2;
cpu_index doesn't need to be a cpuid_t, since it's a cardinal

Improve per-CPU support for the workqueue(9):
- Make structures CPU-cache friendly, as suggested and explained
by Andrew Doran. CACHE_LINE_SIZE definition is invented.
- Use current CPU if NULL is passed to the workqueue_enqueue().
- Implemented MI CPU index, which could be used as an index of array.
Removed linked-lists usage for work queues.

The roundup2() function avoids division, but works only with power of 2.

Reviewed by: <ad>, <yamt>, <tech-kern>

branches: 1.9.2; 1.9.6;
Merge some of the less invasive changes from the vmlocking branch:

- kthread, callout, devsw API changes
- select()/poll() improvements
- miscellaneous MT safety improvements

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

branches: 1.7.2; 1.7.6; 1.7.8;
Merge newlock2 to head.

branches: 1.6.2;
Add the 'lockstat' driver, which collects basic information about lock events.