Surround db_stacktrace() with "#ifdef DDB" check.

Fixes LOCKDEBUG enabled build without DDB option.

Eliminate l->l_ncsw and l->l_nivcsw. From memory think they were added
before we had per-LWP struct rusage; the same is now tracked there.

Revert unintentional changes to kern_lock.c in previous commit.

heartbeat(9): Test whether curcpu is stable, not kpreempt_disabled.

kpreempt_disabled worked for my testing because I tested on aarch64,
which doesn't have kpreemption.

XXX Should move curcpu_stable() to somewhere that other things can
use it.

ASSERT_SLEEPABLE(9): Micro-optimize this a little bit.

This convinces gcc to do less -- make a smaller stack frame, compute
fewer conditional moves in favour of predicted-not-taken branches --
in the fast path where we are sleepable as the caller expects.

Wasn't able to convince it to do the ncsw loop with a
predicted-not-taken branch, but let's leave the __predict_false in
there anyway because it's still a good prediction.

KERNEL_LOCK(9): Minor tweaks to ci->ci_biglock_wanted access.

1. Use atomic_load_relaxed to read ci->ci_biglock_wanted from another
CPU, for clarity and to avoid the appearance of data races in thread
sanitizers. (Reading ci->ci_biglock_wanted on the local CPU need
not be atomic because no other CPU can be writing to it.)

2. Use atomic_store_relaxed to update ci->ci_biglock_wanted when we
start to spin, to avoid the appearance of data races.

3. Add comments to explain what's going on and cross-reference the
specific matching membars in mutex_vector_enter.

related to PR kern/57240

Sprinkle __predict_{true,false} for panicstr checks

branches: 1.181.2;
kern/kern_lock.c: We get start_init_exec from sys/kernel.h now.

KERNEL_LOCK(9): Avoid spinning out until 10sec have passed.

This means we'll never spin out if the hardclock timer is stuck. But
the hardclock timer never runs with the kernel lock held itself, so
it's not immediately clear that's important.

KERNEL_LOCK(9): Restore backoff while spinning in !LOCKDEBUG case.

When the spinout logic was put under LOCKDEBUG among a series of
other changes that got reverted, the backoff was inadvertently made
LOCKDEBUG-only too.

KERNEL_LOCK(9): Limit ipi trace diagnostic to after init has started.

KERNEL_LOCK(9): Fix previous for non-LOCKDEBUG builds.

KERNEL_LOCK(9): Record kernel lock holder in fast path too.

KERNEL_LOCK(9): Need kpreempt_disable to ipi_send, oops.

KERNEL_LOCK(9): Send an IPI to print holder's stack trace on spinout.

Revert the 2015 change I made that allowed sleeping in the idle lwp
if it wasn't running yet, e.g. in cpu_hatch
---
sys/kern/kern_lock.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/sys/kern/kern_lock.c b/sys/kern/kern_lock.c
index 40557427de86..c0c9d8adaf9e 100644
--- a/sys/kern/kern_lock.c
+++ b/sys/kern/kern_lock.c
@@ -89,8 +89,7 @@ assert_sleepable(void)
} while (pctr != lwp_pctr());

reason = NULL;
- if (idle && !cold &&
- kcpuset_isset(kcpuset_running, cpu_index(curcpu()))) {
+ if (idle && !cold) {
reason = "idle";
}
if (cpu_intr_p()) {
--
2.25.1

Comments.

branches: 1.171.2;
Fix inverted condition in r1.136 - we do want LOCKDEBUG spinouts of the
kernel lock to assert as soon as we have userland running - not in
the early boot phase (where firmware loading and device init could take
a long time).

Kill off kernel_lock_plug_leak(), and go back to dropping kernel_lock in
exit1(), since there seems little hope of finding the leaking code any
time soon. Can still be caught with LOCKDEBUG.

Put back the delay hooks which were deleted before. Without them VirtualBox
spins out.

Add a kernel_lock_plug_leak() that drops any holds and tries to identify
the baddy.

Carefully put kernel_lock back the way it was, and add a comment hinting
that changing it is not a good idea, and hopefully nobody will ever try to
change it ever again.

- DIAGNOSTIC: check for leaked kernel_lock in mi_switch().

- Now that ci_biglock_wanted is set later, explicitly disable preemption
while acquiring kernel_lock. It was blocked in a roundabout way
previously.

Reported-by: syzbot+43111d810160fb4b978b@syzkaller.appspotmail.com
Reported-by: syzbot+f5b871bd00089bf97286@syzkaller.appspotmail.com
Reported-by: syzbot+cd1f15eee5b1b6d20078@syzkaller.appspotmail.com
Reported-by: syzbot+fb945a331dabd0b6ba9e@syzkaller.appspotmail.com
Reported-by: syzbot+53a0c2342b361db25240@syzkaller.appspotmail.com
Reported-by: syzbot+552222a952814dede7d1@syzkaller.appspotmail.com
Reported-by: syzbot+c7104a72172b0f9093a4@syzkaller.appspotmail.com
Reported-by: syzbot+efbd30c6ca0f7d8440e8@syzkaller.appspotmail.com
Reported-by: syzbot+330a421bd46794d8b750@syzkaller.appspotmail.com

kernel_lock:

- Defer setting ci_biglock_wanted for a bit, because if curlwp holds a mutex
or rwlock, and otherlwp is spinning waiting for the mutex/rwlock, setting
ci_biglock_wanted causes otherlwp to block to avoid deadlock. If the spin
on kernel_lock is short there's no point causing trouble.

- Do exponential backoff.

- Put the spinout check under LOCKDEBUG to match the others.

branches: 1.164.2;
Use __insn_barrier to enforce ordering in l_ncsw loops.

(Only need ordering observable by interruption, not by other CPUs.)

branches: 1.163.2;
Avoid prepending a timestamp to lock debug outputs on ddb

Lock printer functions (lockops_t#lo_dump) use printf_nolog to print, but
printf_nolog now prepends a timestamp which is unnecessary for ddb:

db{0}> show all locks/t
[Locks tracked through LWPs]
Locks held by an LWP (iperf):
Lock 0 (initialized at soinit)
lock address : 0xffffedeb84b06080 type : sleep/adaptive
initialized : 0xffffffff806d8c3f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 11
current cpu : 0 last held: 1
current lwp : 0xffffedeb849ff040 last held: 0xffffedeb7dfdb240
last locked* : 0xffffffff806d8335 unlocked : 0xffffffff806d8385
[ 79103.0868574] owner field : 0xffffedeb7dfdb240 wait/spin: 1/0

Fix it by passing a printer function to lo_dump functions, i.e., make the
functions use db_printf on ddb.

Make _kernel_lock_dump static

branches: 1.161.4;
Apply C99-style struct initialization to lockops_t

Implement debugging feature for pserialize(9)

The debugging feature detects violations of pserialize constraints.
It causes a panic:
- if a context switch happens in a read section, or
- if a sleepable function is called in a read section.

The feature is enabled only if LOCKDEBUG is on.

Discussed on tech-kern@

more const

branches: 1.158.6;
For LOCKDEBUG:
Always provide the location of the caller of the lock as __func__, __LINE__.

branches: 1.157.2; 1.157.4;
Trailing whitespace

Allow sleeping in the idle lwp if the cpu isn't running yet.

OK'ed by rmind a while ago.

branches: 1.155.4; 1.155.6;
Move a CTASSERT to global scope (easiest way to avoid gcc 4.8.1 local unused
typedef warnings)

branches: 1.154.4;
Revert change that allowed rw_tryenter(&lock, RW_READER) to recurse
for vfs_busy(). This is no longer necessary.

branches: 1.153.2;
Use __cacheline_aligned

add KERNEL_LOCKED_P() macro

branches: 1.151.2;
Retire varargs.h support. Move machine/stdarg.h logic into MI
sys/stdarg.h and expect compiler to provide proper builtins, defaulting
to the GCC interface. lint still has a special fallback.
Reduce abuse of _BSD_VA_LIST_ by defining __va_list by default and
derive va_list as required by standards.

remove dated and wrong comments about curlwp being NULL.
_kernel_{,un}lock() always assume it is valid now.

Fix spelling: situatations -> situations.

- Add lwp_pctr(), get an LWP's preemption/ctxsw counter.
- Fix a preemption bug in CURCPU_IDLE_P() that can lead to a bogus
assertion failure on DEBUG kernels.
- Fix MP/preemption races with timecounter detachment.

branches: 1.147.4;
Remove LKMs and switch to the module framework, pass 1.

Proposed on tech-kern@.

branches: 1.146.2; 1.146.4;
Switch from KASSERT to CTASSERT for those asserts testing sizes of types.

Don't compile kern_lock for rump any more, it's no longer required.
Allows us to get rid of the incorrect _RUMPKERNEL ifdefs outside sys/rump.

branches: 1.144.2;
LOCKDEBUG:

- Tweak it so it can also catch common errors with condition variables.
The change to kern_condvar.c is not included in this commit and will
come later.

- Don't call kmem_alloc() if operating in interrupt context, just fail
the allocation and disable debugging for the object. Makes it safe
to do mutex_init/rw_init/cv_init in interrupt context, when running
a LOCKDEBUG kernel.

Replace a couple of tsleep calls with cv_wait.

Reduce ifdefs due to MULTIPROCESSOR slightly.

branches: 1.141.2;
Allow rw_tryenter(&lock, RW_READER) to recurse, for vfs_busy().

Remove clause 3 and 4 from TNF licenses

Add MI code to support in-kernel preemption. Preemption is deferred by
one of the following:

- Holding kernel_lock (indicating that the code is not MT safe).
- Bracketing critical sections with kpreempt_disable/kpreempt_enable.
- Holding the interrupt priority level above IPL_NONE.

Statistics on kernel preemption are reported via event counters, and
where preemption is deferred for some reason, it's also reported via
lockstat. The LWP priority at which preemption is triggered is tuneable
via sysctl.

Extend spl protection to keep all kernel_lock state in sync. There could
have been problems before. This might help with the assertion failures
seen on sparc64.

branches: 1.137.2; 1.137.4;
remove useless passing of the lwp from the KERNEL_LOCK() ABI
(not the API; this would be easy as well)
agreed (a while ago) by ad

Don't report kernel lock spinouts if init has not yet started.
XXX This should be backed out when we are sure that the drivers
are good citizens and configure nicely with interrupts enabled /
the system running.

- simplify ASSERT_SLEEPABLE.
- move it from proc.h to systm.h.
- add some more checks.
- make it a little more lkm friendly.

branches: 1.134.2; 1.134.6;
Goodbye lockmgr().

lockstat: no longer track lockmgr() events.

- Fix a memory order problem with non-interlocked mutex release.
- Give kernel_lock its own cache line.

Start detangling lock.h from intr.h. This is likely to cause short term
breakage, but the mess of dependencies has been regularly breaking the
build recently anyhow.

Merge vmlocking2 to head.

branches: 1.129.4;
Nothing uses shared -> exclusive upgrades any more, so remove the code.
This is good since they are effectively the same as ...

lockmgr(&lock, LK_RELEASE);
lockmgr(&lock, LK_EXCLUSIVE);

.. and therefore don't behave as expected.

branches: 1.128.2;
Use membar_*().

make kmutex_t and krwlock_t smaller by killing lock id.
ok'ed by Andrew Doran.

Remove KERNEL_LOCK_ASSERT_LOCKED, KERNEL_LOCK_ASSERT_UNLOCKED since the
kernel_lock functions can be patched out at runtime now. Assertions are
provided by the existing functions and by LOCKDEBUG_BARRIER.

Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
number and type of priority levels into bands. Add new bands like
'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

branches: 1.124.2;
Wrap parts dealing with kernel_lock behind #ifndef _RUMPKERNEL.
I don't like doing this, but there's too much pain to get this file
to compile clean due to how SPINLOCK_{BACKOFF,SPIN}_HOOK and
mb_write() are spread out in weird weird places throughout MD code.

branches: 1.123.2;
Use __SIMPLELOCK_LOCKED_P().

Merge from vmlocking:

- G/C spinlockmgr() and simple_lock debugging.
- Always include the kernel_lock functions, for LKMs.
- Slightly improved subr_lockdebug code.
- Keep sizeof(struct lock) the same if LOCKDEBUG.

Kill transferlockers() now that it's unused.

branches: 1.120.2;
__FUNCTION__ -> __func__

Merge nick-csl-alignment.

branches: 1.118.4; 1.118.6; 1.118.8;
Define a new lockmgr flag LK_RESURRECT which can be used in
conjunction with LK_DRAIN. This has the same effect as LK_DRAIN
except it atomically does NOT mark the lock as drained. This
guarantees that when we got the lock, we were the last one currently
waiting for the lock.

Use LK_DRAIN|LK_RESURRECT in vclean() to make sure there are no
waiters for the lock. This should fix behaviour theoretized to be
caused by vfs_subr.c 1.289 which caused vclean() to run into
completion and free the vnode before all lock-waiters had been
processed. Should therefore fix the "simple_lock: unitialized lock"
problems seen recently.

thanks to Juergen Hannken-Illjes for some analysis of the problem
and Erik Bertelsen for testing

Be more forgiving if panicstr != NULL.

branches: 1.116.2;
Re-apply rev 1.111:
Always include kernel_lock so that LOCKDEBUG checks can find the symbol.

Nuke __HAVE_SPLBIGLOCK.

splstatclock, spllock -> splhigh

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

Include the lwpid in the lock panic message, so we don't see silly messages
like
lockmgr: pid 17, not exclusive lock holder 17 unlocking

Always include kernel_lock so that LOCKDEBUG checks can find the symbol.

branches: 1.110.2; 1.110.4; 1.110.6;
add a lockpanic function that prints more detailed error messages.

typedef pri_t and use it instead of int and u_char.

TRUE -> true, FALSE -> false

kernel_lock():

- Fix error in previous.
- Call LOCKDEBUG_WANTLOCK() so the "exclusive wanted" count isn't off.

_kernel_lock(): we can recurse here if we take an interrupt while spinning.
Don't double book the time spent with lockstat.

branches: 1.105.2;
Merge newlock2 to head.

lockstat: improve reporting slightly, and fix a bug where the command
could spin while resorting lists.

in lockstatus(), report LK_EXCLOTHER if LK_WANT_EXCL or LK_WANT_UPGRADE
is set, since the thread that set either of those flags will be the next
one to get the lock. fixes PR 35143.

branches: 1.102.2; 1.102.4;
remove some __unused from function parameters.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

- KERNEL_LOCK_ASSERT_LOCKED: check cpu_biglock_count as well.
- implement KERNEL_LOCK_ASSERT_UNLOCKED.

branches: 1.99.2; 1.99.4;
Add lock_owner_onproc().

Track lockmgr() sleep events for lockstat.

assert_sleepable: panic if curlwp == NULL.

add ASSERT_SLEEPABLE() macro to assert we can sleep.

Fix call to simple_lock_assert_held() so builds with -DDEBUG work.

Add simple_lock_assert_locked/simple_lock_assert_unlocked to provide additional
useful information when panic'ing because the assertion fails.
Use these to define the SCHED_ASSERT_LOCKED/SCHED_ASSERT_UNLOCKED macros.

branches: 1.93.2;
Check db_onpanic before dropping into the debugger on lock errors.

branches: 1.92.4; 1.92.6; 1.92.8; 1.92.10;
changes for making DIAGNOSTIC not change the kernel ABI:
- for structure fields that are conditionally present,
make those fields always present.
- for functions which are conditionally inline, make them never inline.
- remove some other functions which are conditionally defined but
don't actually do anything anymore.
- make a lock-debugging function conditional on only LOCKDEBUG.

as discussed on tech-kern some time back.

Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

default to simple_lock_debugger=1 with LOCKDEBUG.

branches: 1.88.2;
Fix function variable names shadowing global declarations.

Now we can fix the volatile cast-aways.
Rename some shadowed variables while here.

branches: 1.86.2;
nuke trailing whitespace

branches: 1.85.4; 1.85.6;
a relatively lightweight implementation of kernel_lock.

don't reference kernel_lock directly.

add missing wakeups in the cases of lock failure.
from Stephan Uphoff, FreeBSD PR/69964.

(http://www.freebsd.org/cgi/query-pr.cgi?pr=69964)
> The LK_WANT_EXCL and LK_WANT_UPGRADE bits act as mini-locks and can block
> other threads.
> Normally this is not a problem since the mini locks are upgraded to full loc
> and the release of the locks will unblock the other threads.
> However if a thread reset the bits without optaining a full lock
> other threads are not awoken.
> This can happens if obtaining the full lock fails because of a LK_SLEEPFAIL,
> or a signal (if lock priority includes PCATCH .. don't think this is used).

- revert a part of the previous which breaks LK_SPIN locks.
(reported by Nicolas Joly on current-users@)
- propagate the previous to spinlock_acquire_count.

when acquiring an exclusive lock,
ensure that no one else have the same lock.
a patch from Stephan Uphoff, FreeBSD PR/69934.

(http://www.freebsd.org/cgi/query-pr.cgi?pr=69934)
> Upgrading a lock does not play well together with acquiring
> an exclusive lock and can lead to two threads being
> granted exclusive access.
>
> Problematic sequence:
> Thread A acquires a previous unlocked lock in shared mode.
> Thread B tries to acquire the same lock in exclusive mode
> and blocks.
> Thread A upgrades its lock - waking up thread B.
> Thread B wakes up and also acquires the same lock as it only checks
> if the lock is not shared or if someone wants to upgrade the lock
> and not if someone already upgraded the lock to an exclusive lock.

lockmgr: add a comment about LK_RETRY.

lockmgr: assert that LK_RETRY is not specified.

Add ffs internal snapshots. Written by Marshall Kirk McKusick for FreeBSD.

- Not enabled by default. Needs kernel option FFS_SNAPSHOT.
- Change parameters of ffs_blkfree.
- Let the copy-on-write functions return an error so spec_strategy
may fail if the copy-on-write fails.
- Change genfs_*lock*() to use vp->v_vnlock instead of &vp->v_lock.
- Add flag B_METAONLY to VOP_BALLOC to return indirect block buffer.
- Add a function ffs_checkfreefile needed for snapshot creation.
- Add special handling of snapshot files:
Snapshots may not be opened for writing and the attributes are read-only.
Use the mtime as the time this snapshot was taken.
Deny mtime updates for snapshot files.
- Add function transferlockers to transfer any waiting processes from
one lock to another.
- Add vfsop VFS_SNAPSHOT to take a snapshot and make it accessible through
a vnode.
- Add snapshot support to ls, fsck_ffs and dump.

Welcome to 2.0F.

Approved by: Jason R. Thorpe <thorpej@netbsd.org>

use lockstatus() instead of L_BIGLOCK to check if we're holding a biglock.
fix PR/25595.

introduce LK_EXCLOTHER for lockstatus().
from FreeBSD, but a little differently. instead of letting lockstatus()
take an additional thread argument, always use curlwp/curcpu.

branches: 1.75.2;
Uppercase CPU, plural is CPUs.

Fix last commit. The current spl was an implicit argument to the ACQUIRE
macro. With help and approval from YAMAMOTO Takashi <yamt@netbsd.org>

turn ACQUIRE macro into a function by introducing new internal
flags, LK_SHARE_NONZERO and LK_WAIT_NONZERO. from FreeBSD.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

branches: 1.71.2;
Use lock_printf() in SPINLOCK_SPINCHECK() and SLOCK_TRACE().

_simple_lock(): revert to IPL at entry while spinning on the lock; raise
to spllock() again after we get it.

Merge the nathanw_sa branch.

lock_printf(): use vsnprintf/printf_nolog to avoid covertly using the system
log and thereby invoking scheduler code.

Quell uninitialised variable warnings.

/*CONTCOND*/ while (0)'ed macros

For INTERLOCK_ACQUIRE, s/splsched/spllock/.

remove trailing \n in panic(). approved perry.

print a stack trace in the "spinout" case too.

Move kernel_lock manipulation info functions so that they will
show up in a profile.

branches: 1.61.2;
Remove #ifdef DIAGNOSTIC around panic(). It is better than NULL pointer
dereference.

add RCSIDs

branches: 1.59.2;
replace wakeup_one() with wakeup(). wakeup_one() can only be used if the
woken-up thread is guaranteed to pass the buck to the next guy before
going back to sleep, and the rest of the lockmgr() code doesn't do that.
from Bill Sommerfeld. fixes PR 14097.

print a stack trace in more LOCKDEBUG cases.
add a blank line between complaints.
use TAILQ_FOREACH where appropriate.

Correct comment to match code

branches: 1.56.2; 1.56.4;
synchron*, not sychron*

Add a simple_lock_only_held() LOCKDEBUG routine, which allows code
to assert that exactly zero or one (and a specific one) locks are
held.

From Bill Sommerfeld.

Define local variable cpu_id only when either MULTIPROCESSOR or DIAGNOSTIC
is defined since it isn't used otherwise.

STDC cleanup: volatile needs to be cast away for lk_flags as well.

SPINLOCK_INTERLOCK_RELEASE_HOOK should actually be
SPINLOCK_SPIN_HOOK, so that we actually check for
pending IPIs on the Alpha more than once. Also,
when we call alpha_ipi_process(), make sure to go
to splipi().

branches: 1.51.2;
Default lock_printf to syslog rather than printf. Some of the lock debug checks
are done inside of wakeup which is holding the sched lock. Printf can cause
wakeup to get called again (pty redirection of console message) which will
panic with sched lock already held.

This isn't a long term fix as not being able to printf vs. sched lock should
be cleaned up better but this avoids continual panics with lockdebug running
and an xterm -C.

Add a LOCKDEBUG check for a r/w spinlock spinning out of control.
Partially from Bill Sommerfeld.

Allow machine dependent code to specify a hook to be run when a
spinlock's interlock is released.

Idea from Bill Sommerfeld.

Fix !LOCKDEBUG && !DIAGNOSTIC case

Since the spinlock count is per-cpu, we don't need atomic operations
to update it, so don't bother with <machine/atomic.h>

Flush kernel_lock_release_all() and kernel_lock_acquire_count() (which
didn't do spinlock accounting correctly), and replace them with
spinlock_release_all() and spinlock_acquire_count().

Fix a printf format (for Alpha).

Default simple_lock_debugger to "on" on MULTIPROCESSOR.
Change uninitialized simple_lock check from KASSERT to use SLOCK_WHERE
(to show the "real" source line where the error was detected).

Use spllock() rather than splhigh().

Slight adjustment to INTERLOCK_*() macros to make it easier
for the compiler to optimize.

- Clean up _simple_lock_held()
- In simple_lock_switchcheck(), allow/enforce exactly one lock to be
held: sched_lock.
- Per e-mail to tech-smp from Bill Sommerfeld, r/w spin locks have
an interlock at splsched(), rather than splhigh().

Lock debugging fix: Make sure a simplelock's lock_holder gets
initialized properly, and consistently tracks the owning CPU's
cpuid. Add some diagnostic assertions to enforce this.

For spinlocks, block interrupts while holding the interlock. Partially
from Bill Sommerfeld.

Add a DIAGNOSTIC check for release of an unlocked lock.

From Bill Sommerfeld.

Some more lock debugging support:
- LOCK_ASSERT(), which expands to KASSERT() if LOCKDEBUG.
- new simple_lock_held(), which tests if the calling CPU holds
the specified simple lock.

From Bill Sommerfeld, modified slightly by me.

Nother __kprintf_attribute__ to be removed.

Fix printf format error pointed out by Steve Woodford.

Add a DIAGNOSTIC or LOCKDEBUG check for held spin locks.

It doesn't make sense to charge simple locks to proc's, because
simple locks are held by CPUs. Remove p_simple_locks (which was
unused anyway, really), and add a LOCKDEBUG check for held simple
locks in mi_switch(). Grow p_locks to an int to take up the space
previously used by p_simple_locks so that the proc structure doens't
change size.

ANSI'ify.

branches: 1.32.2;
Fix assorted bugs around shutdown/reboot/panic time.
- add a new global variable, doing_shutdown, which is nonzero if
vfs_shutdown() or panic() have been called.
- in panic, set RB_NOSYNC if doing_shutdown is already set on entry
so we don't reenter vfs_shutdown if we panic'ed there.
- in vfs_shutdown, don't use proc0's process for sys_sync unless
curproc is NULL.
- in lockmgr, attribute successful locks to proc0 if doing_shutdown
&& curproc==NULL, and panic if we can't get the lock right away; avoids the
spurious lockmgr DIAGNOSTIC panic from the ddb reboot command.
- in subr_pool, deal with curproc==NULL in the doing_shutdown case.
- in mfs_strategy, bitbucket writes if doing_shutdown, so we don't
wedge waiting for the mfs process.
- in ltsleep, treat ((curproc == NULL) && doing_shutdown) like the
panicstr case.

Appears to fix: kern/9239, kern/10187, kern/9367.
May also fix kern/10122.

Use ltsleep().

branches: 1.30.2;
Fix a typo, and add some lint comments.

Let MULTIPROCESSOR && LOCKDEBUG case compile again

- If a platform defines __HAVE_ATOMIC_OPERATIONS, use them for counting
in the MULTIPROCESSOR case.
- Move a misplaced #ifdef so that LK_REENABLE actually works.

Require that each each MACHINE/MACHINE_ARCH supply a lock.h. This file
contains the values __SIMPLELOCK_LOCKED and __SIMPLELOCK_UNLOCKED, which
replace the old SIMPLELOCK_LOCKED and SIMPLELOCK_UNLOCKED. These files
are also required to supply inline functions __cpu_simple_lock(),
__cpu_simple_lock_try(), and __cpu_simple_unlock() if locking is to be
supported on that platform (i.e. if MULTIPROCESSOR is defined in the
_KERNEL case). Change these functions to take an int * (&alp->lock_data)
rather than the struct simplelock * itself.

These changes make it possible for userland to use the locking primitives
by including <machine/lock.h>.

Three MULTIPROCESSOR + LOCKDEBUG fixes:
1) fix typo preventing compilation (missing comma).
2) in SLOCK_WHERE, display cpu number in the MP case.
3) the folowing race condition was observed in _simple_lock:
cpu 1 releases lock,
cpu 0 grabs lock
cpu 1 sees it's already locked.
cpu 1 sees that lock_holder== "cpu 1"
cpu 1 assumes that it already holds it and barfs.
cpu 0 sets lock_holder == "cpu 0"
Fix: set lock_holder to LK_NOCPU in _simple_unlock().

branches: 1.25.2;
Make it possible to direct LOCKDEBUG messages to syslog only.

Use cpuid_t and cpu_number().

Fix a thinko in draining of spin locks: bump waitcount in the spin case,
too. Remove some needless code duplication by adding a "drain" argument
to the ACQUIRE() macro (compiler can [and does] optimize the constant
conditional).

- Correct the definition of the COUNT macro so that it takes the same
number of arguments when compiled without DIAGNOSTIC as with.

Improve the LOCKDEBUG code:
- Now compatible with MULTIPROCESSOR (requires other changes not yet
committed, but which will be later today).
- In addition to tracking simple locks, track exclusive spin locks.
- Count spin locks like we do sleep locks (in the cpu_info for this
CPU).
- Lock debug lists are now TAILQs, so as to make the locking order
more obvious when dumping the list.

Also, some suggestions from Bill Sommerfeld:
- SIMPLELOCK_LOCKED and SIMPLELOCK_UNLOCKED constants, which may be
defined in <machine/lock.h> (default to 1 and 0, respectively). This
makes it easier to support architectures which use test-and-clear
rather than test-and-set.
- Add __attribute__((__aligned__)) to the `lock_data' member of the
simplelock structure. This makes it easier to support architectures
which can only perform atomic operations on very-well-aligned memory
locations. NOTE: This changes the size of struct simplelock, and
will cause a version bump.

Use wakeup_one() for waking up sleep lock sleepers.

Add a spin lock mode to the lock manager. Provides a read/write
spin lock facility. Some code and ideas from Ross Harvey.

more cleanup:
remove simplelockrecurse, lockpausetime and PAUSE():
none of these serve any purpose anymore.
in the LOCKDEBUG functions, expand the splhigh() region to
cover the entire function. without this there can still be races.

Count lockmgr locks held by process if LOCKDEBUG || DIAGNOSTIC.
(previously, it was just under LOCKDEBUG).

branches: 1.16.2;
Prevent deadlock cited in PR4629 from crashing the system. (copyout
and system call now just return EFAULT). A complete fix will
presumably have to wait for UBC and/or for vnode locking protocols to
be revamped to allow use of shared locks.

Recursive locks were previously only available with LK_CANRECURSE. This
could be done in one of 2 ways:

* call lk_init with LK_CANRECURSE, resulting in a lock that
always can be used recursively.
* call lockmgr with LK_CANRECURSE, meaning that it's ok if this
lock is already held by us.

Sometimes we need a locking type that says: take this lock now, exclusively,
but while I am holding it, I may go through a code path which could attempt
to get the lock again, and which is unaware that the lock might already
be taken. Implement LK_SETRECURSE for this purpose. Assume that locks and
unlocks come in matching pairs (they should), and check for this 'level'
using SETRECURSE locks.

print a little more info in simple_lock_freecheck().

Cosmectic change in a panic(), so that the panic string printed by savecore
has more meaning.

branches: 1.12.2;
LOCKDEBUG enhancements for non-MP:
keep a list of locked locks.
use this to print where the lock was locked
when we either go to sleep with a lock held
or try to free a locked lock.

Disable the daft PAUSE() macro, which manages to skip all the relevant
code in lockmgr() most of the time. This a no doubt a case of Bad Coding Style.

Initialize the CPU ID in the simplelock.

Key off MULTIPROCESSOR, not NCPUS > 1. Pull in <machine/lock.h> if
MULTIPROCESSOR is defined, and rely on it to define the simple lock
operations.

Abolition of bcopy, ovbcopy, bcmp, and bzero, phase one.
bcopy(x, y, z) -> memcpy(y, x, z)
ovbcopy(x, y, z) -> memmove(y, x, z)
bcmp(x, y, z) -> memcmp(x, y, z)
bzero(x, y) -> memset(x, 0, y)

defopt LOCKDEBUG

Merge with Lite2 + local changes

snazzier LOCKDEBUG code.

Make wmesg arguments to various functions const.

branches: 1.3.2;
There appear to be spinlock bugs in the VM code. They are not a problem
now, as we're always one on CPU (they will be later, though). With DEBUG,
they cause a lot of output, so DEBUG -> LOCKDEBUG for now.

Add NetBSD RCS Id, and a few minor changes to make it compile.

branches: 1.1.1;
Initial revision

Eliminate l->l_ncsw and l->l_nivcsw. From memory think they were added
before we had per-LWP struct rusage; the same is now tracked there.

Revert unintentional changes to kern_lock.c in previous commit.

heartbeat(9): Test whether curcpu is stable, not kpreempt_disabled.

kpreempt_disabled worked for my testing because I tested on aarch64,
which doesn't have kpreemption.

XXX Should move curcpu_stable() to somewhere that other things can
use it.

ASSERT_SLEEPABLE(9): Micro-optimize this a little bit.

This convinces gcc to do less -- make a smaller stack frame, compute
fewer conditional moves in favour of predicted-not-taken branches --
in the fast path where we are sleepable as the caller expects.

Wasn't able to convince it to do the ncsw loop with a
predicted-not-taken branch, but let's leave the __predict_false in
there anyway because it's still a good prediction.

KERNEL_LOCK(9): Minor tweaks to ci->ci_biglock_wanted access.

1. Use atomic_load_relaxed to read ci->ci_biglock_wanted from another
CPU, for clarity and to avoid the appearance of data races in thread
sanitizers. (Reading ci->ci_biglock_wanted on the local CPU need
not be atomic because no other CPU can be writing to it.)

2. Use atomic_store_relaxed to update ci->ci_biglock_wanted when we
start to spin, to avoid the appearance of data races.

3. Add comments to explain what's going on and cross-reference the
specific matching membars in mutex_vector_enter.

related to PR kern/57240

Sprinkle __predict_{true,false} for panicstr checks

branches: 1.181.2;
kern/kern_lock.c: We get start_init_exec from sys/kernel.h now.

KERNEL_LOCK(9): Avoid spinning out until 10sec have passed.

This means we'll never spin out if the hardclock timer is stuck. But
the hardclock timer never runs with the kernel lock held itself, so
it's not immediately clear that's important.

KERNEL_LOCK(9): Restore backoff while spinning in !LOCKDEBUG case.

When the spinout logic was put under LOCKDEBUG among a series of
other changes that got reverted, the backoff was inadvertently made
LOCKDEBUG-only too.

KERNEL_LOCK(9): Limit ipi trace diagnostic to after init has started.

KERNEL_LOCK(9): Fix previous for non-LOCKDEBUG builds.

KERNEL_LOCK(9): Record kernel lock holder in fast path too.

KERNEL_LOCK(9): Need kpreempt_disable to ipi_send, oops.

KERNEL_LOCK(9): Send an IPI to print holder's stack trace on spinout.

Revert the 2015 change I made that allowed sleeping in the idle lwp
if it wasn't running yet, e.g. in cpu_hatch
---
sys/kern/kern_lock.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/sys/kern/kern_lock.c b/sys/kern/kern_lock.c
index 40557427de86..c0c9d8adaf9e 100644
--- a/sys/kern/kern_lock.c
+++ b/sys/kern/kern_lock.c
@@ -89,8 +89,7 @@ assert_sleepable(void)
} while (pctr != lwp_pctr());

reason = NULL;
- if (idle && !cold &&
- kcpuset_isset(kcpuset_running, cpu_index(curcpu()))) {
+ if (idle && !cold) {
reason = "idle";
}
if (cpu_intr_p()) {
--
2.25.1

Comments.

branches: 1.171.2;
Fix inverted condition in r1.136 - we do want LOCKDEBUG spinouts of the
kernel lock to assert as soon as we have userland running - not in
the early boot phase (where firmware loading and device init could take
a long time).

Kill off kernel_lock_plug_leak(), and go back to dropping kernel_lock in
exit1(), since there seems little hope of finding the leaking code any
time soon. Can still be caught with LOCKDEBUG.

Put back the delay hooks which were deleted before. Without them VirtualBox
spins out.

Add a kernel_lock_plug_leak() that drops any holds and tries to identify
the baddy.

Carefully put kernel_lock back the way it was, and add a comment hinting
that changing it is not a good idea, and hopefully nobody will ever try to
change it ever again.

- DIAGNOSTIC: check for leaked kernel_lock in mi_switch().

- Now that ci_biglock_wanted is set later, explicitly disable preemption
while acquiring kernel_lock. It was blocked in a roundabout way
previously.

Reported-by: syzbot+43111d810160fb4b978b@syzkaller.appspotmail.com
Reported-by: syzbot+f5b871bd00089bf97286@syzkaller.appspotmail.com
Reported-by: syzbot+cd1f15eee5b1b6d20078@syzkaller.appspotmail.com
Reported-by: syzbot+fb945a331dabd0b6ba9e@syzkaller.appspotmail.com
Reported-by: syzbot+53a0c2342b361db25240@syzkaller.appspotmail.com
Reported-by: syzbot+552222a952814dede7d1@syzkaller.appspotmail.com
Reported-by: syzbot+c7104a72172b0f9093a4@syzkaller.appspotmail.com
Reported-by: syzbot+efbd30c6ca0f7d8440e8@syzkaller.appspotmail.com
Reported-by: syzbot+330a421bd46794d8b750@syzkaller.appspotmail.com

kernel_lock:

- Defer setting ci_biglock_wanted for a bit, because if curlwp holds a mutex
or rwlock, and otherlwp is spinning waiting for the mutex/rwlock, setting
ci_biglock_wanted causes otherlwp to block to avoid deadlock. If the spin
on kernel_lock is short there's no point causing trouble.

- Do exponential backoff.

- Put the spinout check under LOCKDEBUG to match the others.

branches: 1.164.2;
Use __insn_barrier to enforce ordering in l_ncsw loops.

(Only need ordering observable by interruption, not by other CPUs.)

branches: 1.163.2;
Avoid prepending a timestamp to lock debug outputs on ddb

Lock printer functions (lockops_t#lo_dump) use printf_nolog to print, but
printf_nolog now prepends a timestamp which is unnecessary for ddb:

db{0}> show all locks/t
[Locks tracked through LWPs]
Locks held by an LWP (iperf):
Lock 0 (initialized at soinit)
lock address : 0xffffedeb84b06080 type : sleep/adaptive
initialized : 0xffffffff806d8c3f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 11
current cpu : 0 last held: 1
current lwp : 0xffffedeb849ff040 last held: 0xffffedeb7dfdb240
last locked* : 0xffffffff806d8335 unlocked : 0xffffffff806d8385
[ 79103.0868574] owner field : 0xffffedeb7dfdb240 wait/spin: 1/0

Fix it by passing a printer function to lo_dump functions, i.e., make the
functions use db_printf on ddb.

Make _kernel_lock_dump static

branches: 1.161.4;
Apply C99-style struct initialization to lockops_t

Implement debugging feature for pserialize(9)

The debugging feature detects violations of pserialize constraints.
It causes a panic:
- if a context switch happens in a read section, or
- if a sleepable function is called in a read section.

The feature is enabled only if LOCKDEBUG is on.

Discussed on tech-kern@

more const

branches: 1.158.6;
For LOCKDEBUG:
Always provide the location of the caller of the lock as __func__, __LINE__.

branches: 1.157.2; 1.157.4;
Trailing whitespace

Allow sleeping in the idle lwp if the cpu isn't running yet.

OK'ed by rmind a while ago.

branches: 1.155.4; 1.155.6;
Move a CTASSERT to global scope (easiest way to avoid gcc 4.8.1 local unused
typedef warnings)

branches: 1.154.4;
Revert change that allowed rw_tryenter(&lock, RW_READER) to recurse
for vfs_busy(). This is no longer necessary.

branches: 1.153.2;
Use __cacheline_aligned

add KERNEL_LOCKED_P() macro

branches: 1.151.2;
Retire varargs.h support. Move machine/stdarg.h logic into MI
sys/stdarg.h and expect compiler to provide proper builtins, defaulting
to the GCC interface. lint still has a special fallback.
Reduce abuse of _BSD_VA_LIST_ by defining __va_list by default and
derive va_list as required by standards.

remove dated and wrong comments about curlwp being NULL.
_kernel_{,un}lock() always assume it is valid now.

Fix spelling: situatations -> situations.

- Add lwp_pctr(), get an LWP's preemption/ctxsw counter.
- Fix a preemption bug in CURCPU_IDLE_P() that can lead to a bogus
assertion failure on DEBUG kernels.
- Fix MP/preemption races with timecounter detachment.

branches: 1.147.4;
Remove LKMs and switch to the module framework, pass 1.

Proposed on tech-kern@.

branches: 1.146.2; 1.146.4;
Switch from KASSERT to CTASSERT for those asserts testing sizes of types.

Don't compile kern_lock for rump any more, it's no longer required.
Allows us to get rid of the incorrect _RUMPKERNEL ifdefs outside sys/rump.

branches: 1.144.2;
LOCKDEBUG:

- Tweak it so it can also catch common errors with condition variables.
The change to kern_condvar.c is not included in this commit and will
come later.

- Don't call kmem_alloc() if operating in interrupt context, just fail
the allocation and disable debugging for the object. Makes it safe
to do mutex_init/rw_init/cv_init in interrupt context, when running
a LOCKDEBUG kernel.

Replace a couple of tsleep calls with cv_wait.

Reduce ifdefs due to MULTIPROCESSOR slightly.

branches: 1.141.2;
Allow rw_tryenter(&lock, RW_READER) to recurse, for vfs_busy().

Remove clause 3 and 4 from TNF licenses

Add MI code to support in-kernel preemption. Preemption is deferred by
one of the following:

- Holding kernel_lock (indicating that the code is not MT safe).
- Bracketing critical sections with kpreempt_disable/kpreempt_enable.
- Holding the interrupt priority level above IPL_NONE.

Statistics on kernel preemption are reported via event counters, and
where preemption is deferred for some reason, it's also reported via
lockstat. The LWP priority at which preemption is triggered is tuneable
via sysctl.

Extend spl protection to keep all kernel_lock state in sync. There could
have been problems before. This might help with the assertion failures
seen on sparc64.

branches: 1.137.2; 1.137.4;
remove useless passing of the lwp from the KERNEL_LOCK() ABI
(not the API; this would be easy as well)
agreed (a while ago) by ad

Don't report kernel lock spinouts if init has not yet started.
XXX This should be backed out when we are sure that the drivers
are good citizens and configure nicely with interrupts enabled /
the system running.

- simplify ASSERT_SLEEPABLE.
- move it from proc.h to systm.h.
- add some more checks.
- make it a little more lkm friendly.

branches: 1.134.2; 1.134.6;
Goodbye lockmgr().

lockstat: no longer track lockmgr() events.

- Fix a memory order problem with non-interlocked mutex release.
- Give kernel_lock its own cache line.

Start detangling lock.h from intr.h. This is likely to cause short term
breakage, but the mess of dependencies has been regularly breaking the
build recently anyhow.

Merge vmlocking2 to head.

branches: 1.129.4;
Nothing uses shared -> exclusive upgrades any more, so remove the code.
This is good since they are effectively the same as ...

lockmgr(&lock, LK_RELEASE);
lockmgr(&lock, LK_EXCLUSIVE);

.. and therefore don't behave as expected.

branches: 1.128.2;
Use membar_*().

make kmutex_t and krwlock_t smaller by killing lock id.
ok'ed by Andrew Doran.

Remove KERNEL_LOCK_ASSERT_LOCKED, KERNEL_LOCK_ASSERT_UNLOCKED since the
kernel_lock functions can be patched out at runtime now. Assertions are
provided by the existing functions and by LOCKDEBUG_BARRIER.

Merge scheduler changes from the vmlocking branch. All discussed on
tech-kern:

- Invert priority space so that zero is the lowest priority. Rearrange
number and type of priority levels into bands. Add new bands like
'kernel real time'.
- Ignore the priority level passed to tsleep. Compute priority for
sleep dynamically.
- For SCHED_4BSD, make priority adjustment per-LWP, not per-process.

branches: 1.124.2;
Wrap parts dealing with kernel_lock behind #ifndef _RUMPKERNEL.
I don't like doing this, but there's too much pain to get this file
to compile clean due to how SPINLOCK_{BACKOFF,SPIN}_HOOK and
mb_write() are spread out in weird weird places throughout MD code.

branches: 1.123.2;
Use __SIMPLELOCK_LOCKED_P().

Merge from vmlocking:

- G/C spinlockmgr() and simple_lock debugging.
- Always include the kernel_lock functions, for LKMs.
- Slightly improved subr_lockdebug code.
- Keep sizeof(struct lock) the same if LOCKDEBUG.

Kill transferlockers() now that it's unused.

branches: 1.120.2;
__FUNCTION__ -> __func__

Merge nick-csl-alignment.

branches: 1.118.4; 1.118.6; 1.118.8;
Define a new lockmgr flag LK_RESURRECT which can be used in
conjunction with LK_DRAIN. This has the same effect as LK_DRAIN
except it atomically does NOT mark the lock as drained. This
guarantees that when we got the lock, we were the last one currently
waiting for the lock.

Use LK_DRAIN|LK_RESURRECT in vclean() to make sure there are no
waiters for the lock. This should fix behaviour theoretized to be
caused by vfs_subr.c 1.289 which caused vclean() to run into
completion and free the vnode before all lock-waiters had been
processed. Should therefore fix the "simple_lock: unitialized lock"
problems seen recently.

thanks to Juergen Hannken-Illjes for some analysis of the problem
and Erik Bertelsen for testing

Be more forgiving if panicstr != NULL.

branches: 1.116.2;
Re-apply rev 1.111:
Always include kernel_lock so that LOCKDEBUG checks can find the symbol.

Nuke __HAVE_SPLBIGLOCK.

splstatclock, spllock -> splhigh

merge yamt-idlelwp branch. asked by core@. some ports still needs work.

from doc/BRANCHES:

idle lwp, and some changes depending on it.

1. separate context switching and thread scheduling.
(cf. gmcgarry_ctxsw)
2. implement idle lwp.
3. clean up related MD/MI interfaces.
4. make scheduler(s) modular.

Include the lwpid in the lock panic message, so we don't see silly messages
like
lockmgr: pid 17, not exclusive lock holder 17 unlocking

Always include kernel_lock so that LOCKDEBUG checks can find the symbol.

branches: 1.110.2; 1.110.4; 1.110.6;
add a lockpanic function that prints more detailed error messages.

typedef pri_t and use it instead of int and u_char.

TRUE -> true, FALSE -> false

kernel_lock():

- Fix error in previous.
- Call LOCKDEBUG_WANTLOCK() so the "exclusive wanted" count isn't off.

_kernel_lock(): we can recurse here if we take an interrupt while spinning.
Don't double book the time spent with lockstat.

branches: 1.105.2;
Merge newlock2 to head.

lockstat: improve reporting slightly, and fix a bug where the command
could spin while resorting lists.

in lockstatus(), report LK_EXCLOTHER if LK_WANT_EXCL or LK_WANT_UPGRADE
is set, since the thread that set either of those flags will be the next
one to get the lock. fixes PR 35143.

branches: 1.102.2; 1.102.4;
remove some __unused from function parameters.

- sprinkle __unused on function decls.
- fix a couple of unused bugs
- no more -Wno-unused for i386

- KERNEL_LOCK_ASSERT_LOCKED: check cpu_biglock_count as well.
- implement KERNEL_LOCK_ASSERT_UNLOCKED.

branches: 1.99.2; 1.99.4;
Add lock_owner_onproc().

Track lockmgr() sleep events for lockstat.

assert_sleepable: panic if curlwp == NULL.

add ASSERT_SLEEPABLE() macro to assert we can sleep.

Fix call to simple_lock_assert_held() so builds with -DDEBUG work.

Add simple_lock_assert_locked/simple_lock_assert_unlocked to provide additional
useful information when panic'ing because the assertion fails.
Use these to define the SCHED_ASSERT_LOCKED/SCHED_ASSERT_UNLOCKED macros.

branches: 1.93.2;
Check db_onpanic before dropping into the debugger on lock errors.

branches: 1.92.4; 1.92.6; 1.92.8; 1.92.10;
changes for making DIAGNOSTIC not change the kernel ABI:
- for structure fields that are conditionally present,
make those fields always present.
- for functions which are conditionally inline, make them never inline.
- remove some other functions which are conditionally defined but
don't actually do anything anymore.
- make a lock-debugging function conditional on only LOCKDEBUG.

as discussed on tech-kern some time back.

Remove leading __ from __(const|inline|signed|volatile) -- it is obsolete.

merge ktrace-lwp.

default to simple_lock_debugger=1 with LOCKDEBUG.

branches: 1.88.2;
Fix function variable names shadowing global declarations.

Now we can fix the volatile cast-aways.
Rename some shadowed variables while here.

branches: 1.86.2;
nuke trailing whitespace

branches: 1.85.4; 1.85.6;
a relatively lightweight implementation of kernel_lock.

don't reference kernel_lock directly.

add missing wakeups in the cases of lock failure.
from Stephan Uphoff, FreeBSD PR/69964.

(http://www.freebsd.org/cgi/query-pr.cgi?pr=69964)
> The LK_WANT_EXCL and LK_WANT_UPGRADE bits act as mini-locks and can block
> other threads.
> Normally this is not a problem since the mini locks are upgraded to full loc
> and the release of the locks will unblock the other threads.
> However if a thread reset the bits without optaining a full lock
> other threads are not awoken.
> This can happens if obtaining the full lock fails because of a LK_SLEEPFAIL,
> or a signal (if lock priority includes PCATCH .. don't think this is used).

- revert a part of the previous which breaks LK_SPIN locks.
(reported by Nicolas Joly on current-users@)
- propagate the previous to spinlock_acquire_count.

when acquiring an exclusive lock,
ensure that no one else have the same lock.
a patch from Stephan Uphoff, FreeBSD PR/69934.

(http://www.freebsd.org/cgi/query-pr.cgi?pr=69934)
> Upgrading a lock does not play well together with acquiring
> an exclusive lock and can lead to two threads being
> granted exclusive access.
>
> Problematic sequence:
> Thread A acquires a previous unlocked lock in shared mode.
> Thread B tries to acquire the same lock in exclusive mode
> and blocks.
> Thread A upgrades its lock - waking up thread B.
> Thread B wakes up and also acquires the same lock as it only checks
> if the lock is not shared or if someone wants to upgrade the lock
> and not if someone already upgraded the lock to an exclusive lock.

lockmgr: add a comment about LK_RETRY.

lockmgr: assert that LK_RETRY is not specified.

Add ffs internal snapshots. Written by Marshall Kirk McKusick for FreeBSD.

- Not enabled by default. Needs kernel option FFS_SNAPSHOT.
- Change parameters of ffs_blkfree.
- Let the copy-on-write functions return an error so spec_strategy
may fail if the copy-on-write fails.
- Change genfs_*lock*() to use vp->v_vnlock instead of &vp->v_lock.
- Add flag B_METAONLY to VOP_BALLOC to return indirect block buffer.
- Add a function ffs_checkfreefile needed for snapshot creation.
- Add special handling of snapshot files:
Snapshots may not be opened for writing and the attributes are read-only.
Use the mtime as the time this snapshot was taken.
Deny mtime updates for snapshot files.
- Add function transferlockers to transfer any waiting processes from
one lock to another.
- Add vfsop VFS_SNAPSHOT to take a snapshot and make it accessible through
a vnode.
- Add snapshot support to ls, fsck_ffs and dump.

Welcome to 2.0F.

Approved by: Jason R. Thorpe <thorpej@netbsd.org>

use lockstatus() instead of L_BIGLOCK to check if we're holding a biglock.
fix PR/25595.

introduce LK_EXCLOTHER for lockstatus().
from FreeBSD, but a little differently. instead of letting lockstatus()
take an additional thread argument, always use curlwp/curcpu.

branches: 1.75.2;
Uppercase CPU, plural is CPUs.

Fix last commit. The current spl was an implicit argument to the ACQUIRE
macro. With help and approval from YAMAMOTO Takashi <yamt@netbsd.org>

turn ACQUIRE macro into a function by introducing new internal
flags, LK_SHARE_NONZERO and LK_WAIT_NONZERO. from FreeBSD.

Move UCB-licensed code from 4-clause to 3-clause licence.

Patches provided by Joel Baker in PR 22364, verified by myself.

branches: 1.71.2;
Use lock_printf() in SPINLOCK_SPINCHECK() and SLOCK_TRACE().

_simple_lock(): revert to IPL at entry while spinning on the lock; raise
to spllock() again after we get it.

Merge the nathanw_sa branch.

lock_printf(): use vsnprintf/printf_nolog to avoid covertly using the system
log and thereby invoking scheduler code.

Quell uninitialised variable warnings.

/*CONTCOND*/ while (0)'ed macros

For INTERLOCK_ACQUIRE, s/splsched/spllock/.

remove trailing \n in panic(). approved perry.

print a stack trace in the "spinout" case too.

Move kernel_lock manipulation info functions so that they will
show up in a profile.

branches: 1.61.2;
Remove #ifdef DIAGNOSTIC around panic(). It is better than NULL pointer
dereference.

add RCSIDs

branches: 1.59.2;
replace wakeup_one() with wakeup(). wakeup_one() can only be used if the
woken-up thread is guaranteed to pass the buck to the next guy before
going back to sleep, and the rest of the lockmgr() code doesn't do that.
from Bill Sommerfeld. fixes PR 14097.

print a stack trace in more LOCKDEBUG cases.
add a blank line between complaints.
use TAILQ_FOREACH where appropriate.

Correct comment to match code

branches: 1.56.2; 1.56.4;
synchron*, not sychron*

Add a simple_lock_only_held() LOCKDEBUG routine, which allows code
to assert that exactly zero or one (and a specific one) locks are
held.

From Bill Sommerfeld.

Define local variable cpu_id only when either MULTIPROCESSOR or DIAGNOSTIC
is defined since it isn't used otherwise.

STDC cleanup: volatile needs to be cast away for lk_flags as well.

SPINLOCK_INTERLOCK_RELEASE_HOOK should actually be
SPINLOCK_SPIN_HOOK, so that we actually check for
pending IPIs on the Alpha more than once. Also,
when we call alpha_ipi_process(), make sure to go
to splipi().

branches: 1.51.2;
Default lock_printf to syslog rather than printf. Some of the lock debug checks
are done inside of wakeup which is holding the sched lock. Printf can cause
wakeup to get called again (pty redirection of console message) which will
panic with sched lock already held.

This isn't a long term fix as not being able to printf vs. sched lock should
be cleaned up better but this avoids continual panics with lockdebug running
and an xterm -C.

Add a LOCKDEBUG check for a r/w spinlock spinning out of control.
Partially from Bill Sommerfeld.

Allow machine dependent code to specify a hook to be run when a
spinlock's interlock is released.

Idea from Bill Sommerfeld.

Fix !LOCKDEBUG && !DIAGNOSTIC case

Since the spinlock count is per-cpu, we don't need atomic operations
to update it, so don't bother with <machine/atomic.h>

Flush kernel_lock_release_all() and kernel_lock_acquire_count() (which
didn't do spinlock accounting correctly), and replace them with
spinlock_release_all() and spinlock_acquire_count().

Fix a printf format (for Alpha).

Default simple_lock_debugger to "on" on MULTIPROCESSOR.
Change uninitialized simple_lock check from KASSERT to use SLOCK_WHERE
(to show the "real" source line where the error was detected).

Use spllock() rather than splhigh().

Slight adjustment to INTERLOCK_*() macros to make it easier
for the compiler to optimize.

- Clean up _simple_lock_held()
- In simple_lock_switchcheck(), allow/enforce exactly one lock to be
held: sched_lock.
- Per e-mail to tech-smp from Bill Sommerfeld, r/w spin locks have
an interlock at splsched(), rather than splhigh().

Lock debugging fix: Make sure a simplelock's lock_holder gets
initialized properly, and consistently tracks the owning CPU's
cpuid. Add some diagnostic assertions to enforce this.

For spinlocks, block interrupts while holding the interlock. Partially
from Bill Sommerfeld.

Add a DIAGNOSTIC check for release of an unlocked lock.

From Bill Sommerfeld.

Some more lock debugging support:
- LOCK_ASSERT(), which expands to KASSERT() if LOCKDEBUG.
- new simple_lock_held(), which tests if the calling CPU holds
the specified simple lock.

From Bill Sommerfeld, modified slightly by me.

Nother __kprintf_attribute__ to be removed.

Fix printf format error pointed out by Steve Woodford.

Add a DIAGNOSTIC or LOCKDEBUG check for held spin locks.

It doesn't make sense to charge simple locks to proc's, because
simple locks are held by CPUs. Remove p_simple_locks (which was
unused anyway, really), and add a LOCKDEBUG check for held simple
locks in mi_switch(). Grow p_locks to an int to take up the space
previously used by p_simple_locks so that the proc structure doens't
change size.

ANSI'ify.

branches: 1.32.2;
Fix assorted bugs around shutdown/reboot/panic time.
- add a new global variable, doing_shutdown, which is nonzero if
vfs_shutdown() or panic() have been called.
- in panic, set RB_NOSYNC if doing_shutdown is already set on entry
so we don't reenter vfs_shutdown if we panic'ed there.
- in vfs_shutdown, don't use proc0's process for sys_sync unless
curproc is NULL.
- in lockmgr, attribute successful locks to proc0 if doing_shutdown
&& curproc==NULL, and panic if we can't get the lock right away; avoids the
spurious lockmgr DIAGNOSTIC panic from the ddb reboot command.
- in subr_pool, deal with curproc==NULL in the doing_shutdown case.
- in mfs_strategy, bitbucket writes if doing_shutdown, so we don't
wedge waiting for the mfs process.
- in ltsleep, treat ((curproc == NULL) && doing_shutdown) like the
panicstr case.

Appears to fix: kern/9239, kern/10187, kern/9367.
May also fix kern/10122.

Use ltsleep().

branches: 1.30.2;
Fix a typo, and add some lint comments.

Let MULTIPROCESSOR && LOCKDEBUG case compile again

- If a platform defines __HAVE_ATOMIC_OPERATIONS, use them for counting
in the MULTIPROCESSOR case.
- Move a misplaced #ifdef so that LK_REENABLE actually works.

Require that each each MACHINE/MACHINE_ARCH supply a lock.h. This file
contains the values __SIMPLELOCK_LOCKED and __SIMPLELOCK_UNLOCKED, which
replace the old SIMPLELOCK_LOCKED and SIMPLELOCK_UNLOCKED. These files
are also required to supply inline functions __cpu_simple_lock(),
__cpu_simple_lock_try(), and __cpu_simple_unlock() if locking is to be
supported on that platform (i.e. if MULTIPROCESSOR is defined in the
_KERNEL case). Change these functions to take an int * (&alp->lock_data)
rather than the struct simplelock * itself.

These changes make it possible for userland to use the locking primitives
by including <machine/lock.h>.

Three MULTIPROCESSOR + LOCKDEBUG fixes:
1) fix typo preventing compilation (missing comma).
2) in SLOCK_WHERE, display cpu number in the MP case.
3) the folowing race condition was observed in _simple_lock:
cpu 1 releases lock,
cpu 0 grabs lock
cpu 1 sees it's already locked.
cpu 1 sees that lock_holder== "cpu 1"
cpu 1 assumes that it already holds it and barfs.
cpu 0 sets lock_holder == "cpu 0"
Fix: set lock_holder to LK_NOCPU in _simple_unlock().

branches: 1.25.2;
Make it possible to direct LOCKDEBUG messages to syslog only.

Use cpuid_t and cpu_number().

Fix a thinko in draining of spin locks: bump waitcount in the spin case,
too. Remove some needless code duplication by adding a "drain" argument
to the ACQUIRE() macro (compiler can [and does] optimize the constant
conditional).

- Correct the definition of the COUNT macro so that it takes the same
number of arguments when compiled without DIAGNOSTIC as with.

Improve the LOCKDEBUG code:
- Now compatible with MULTIPROCESSOR (requires other changes not yet
committed, but which will be later today).
- In addition to tracking simple locks, track exclusive spin locks.
- Count spin locks like we do sleep locks (in the cpu_info for this
CPU).
- Lock debug lists are now TAILQs, so as to make the locking order
more obvious when dumping the list.

Also, some suggestions from Bill Sommerfeld:
- SIMPLELOCK_LOCKED and SIMPLELOCK_UNLOCKED constants, which may be
defined in <machine/lock.h> (default to 1 and 0, respectively). This
makes it easier to support architectures which use test-and-clear
rather than test-and-set.
- Add __attribute__((__aligned__)) to the `lock_data' member of the
simplelock structure. This makes it easier to support architectures
which can only perform atomic operations on very-well-aligned memory
locations. NOTE: This changes the size of struct simplelock, and
will cause a version bump.

Use wakeup_one() for waking up sleep lock sleepers.

Add a spin lock mode to the lock manager. Provides a read/write
spin lock facility. Some code and ideas from Ross Harvey.

more cleanup:
remove simplelockrecurse, lockpausetime and PAUSE():
none of these serve any purpose anymore.
in the LOCKDEBUG functions, expand the splhigh() region to
cover the entire function. without this there can still be races.

Count lockmgr locks held by process if LOCKDEBUG || DIAGNOSTIC.
(previously, it was just under LOCKDEBUG).

branches: 1.16.2;
Prevent deadlock cited in PR4629 from crashing the system. (copyout
and system call now just return EFAULT). A complete fix will
presumably have to wait for UBC and/or for vnode locking protocols to
be revamped to allow use of shared locks.

Recursive locks were previously only available with LK_CANRECURSE. This
could be done in one of 2 ways:

* call lk_init with LK_CANRECURSE, resulting in a lock that
always can be used recursively.
* call lockmgr with LK_CANRECURSE, meaning that it's ok if this
lock is already held by us.

Sometimes we need a locking type that says: take this lock now, exclusively,
but while I am holding it, I may go through a code path which could attempt
to get the lock again, and which is unaware that the lock might already
be taken. Implement LK_SETRECURSE for this purpose. Assume that locks and
unlocks come in matching pairs (they should), and check for this 'level'
using SETRECURSE locks.

print a little more info in simple_lock_freecheck().

Cosmectic change in a panic(), so that the panic string printed by savecore
has more meaning.

branches: 1.12.2;
LOCKDEBUG enhancements for non-MP:
keep a list of locked locks.
use this to print where the lock was locked
when we either go to sleep with a lock held
or try to free a locked lock.

Disable the daft PAUSE() macro, which manages to skip all the relevant
code in lockmgr() most of the time. This a no doubt a case of Bad Coding Style.

Initialize the CPU ID in the simplelock.

Key off MULTIPROCESSOR, not NCPUS > 1. Pull in <machine/lock.h> if
MULTIPROCESSOR is defined, and rely on it to define the simple lock
operations.

Abolition of bcopy, ovbcopy, bcmp, and bzero, phase one.
bcopy(x, y, z) -> memcpy(y, x, z)
ovbcopy(x, y, z) -> memmove(y, x, z)
bcmp(x, y, z) -> memcmp(x, y, z)
bzero(x, y) -> memset(x, 0, y)

defopt LOCKDEBUG

Merge with Lite2 + local changes

snazzier LOCKDEBUG code.

Make wmesg arguments to various functions const.

branches: 1.3.2;
There appear to be spinlock bugs in the VM code. They are not a problem
now, as we're always one on CPU (they will be later, though). With DEBUG,
they cause a lot of output, so DEBUG -> LOCKDEBUG for now.

Add NetBSD RCS Id, and a few minor changes to make it compile.

branches: 1.1.1;
Initial revision

Revert unintentional changes to kern_lock.c in previous commit.

heartbeat(9): Test whether curcpu is stable, not kpreempt_disabled.

kpreempt_disabled worked for my testing because I tested on aarch64,
which doesn't have kpreemption.

XXX Should move curcpu_stable() to somewhere that other things can
use it.

ASSERT_SLEEPABLE(9): Micro-optimize this a little bit.

This convinces gcc to do less -- make a smaller stack frame, compute
fewer conditional moves in favour of predicted-not-taken branches --
in the fast path where we are sleepable as the caller expects.

Wasn't able to convince it to do the ncsw loop with a
predicted-not-taken branch, but let's leave the __predict_false in
there anyway because it's still a good prediction.

KERNEL_LOCK(9): Minor tweaks to ci->ci_biglock_wanted access.

1. Use atomic_load_relaxed to read ci->ci_biglock_wanted from another
CPU, for clarity and to avoid the appearance of data races in thread
sanitizers. (Reading ci->ci_biglock_wanted on the local CPU need
not be atomic because no other CPU can be writing to it.)

2. Use atomic_store_relaxed to update ci->ci_biglock_wanted when we
start to spin, to avoid the appearance of data races.

3. Add comments to explain what's going on and cross-reference the
specific matching membars in mutex_vector_enter.

related to PR kern/57240

Sprinkle __predict_{true,false} for panicstr checks

kern/kern_lock.c: We get start_init_exec from sys/kernel.h now.

KERNEL_LOCK(9): Avoid spinning out until 10sec have passed.

This means we'll never spin out if the hardclock timer is stuck. But
the hardclock timer never runs with the kernel lock held itself, so
it's not immediately clear that's important.

KERNEL_LOCK(9): Restore backoff while spinning in !LOCKDEBUG case.

When the spinout logic was put under LOCKDEBUG among a series of
other changes that got reverted, the backoff was inadvertently made
LOCKDEBUG-only too.

KERNEL_LOCK(9): Limit ipi trace diagnostic to after init has started.

KERNEL_LOCK(9): Fix previous for non-LOCKDEBUG builds.

KERNEL_LOCK(9): Record kernel lock holder in fast path too.

KERNEL_LOCK(9): Need kpreempt_disable to ipi_send, oops.

KERNEL_LOCK(9): Send an IPI to print holder's stack trace on spinout.

Revert the 2015 change I made that allowed sleeping in the idle lwp
if it wasn't running yet, e.g. in cpu_hatch
---
sys/kern/kern_lock.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/sys/kern/kern_lock.c b/sys/kern/kern_lock.c
index 40557427de86..c0c9d8adaf9e 100644
--- a/sys/kern/kern_lock.c
+++ b/sys/kern/kern_lock.c
@@ -89,8 +89,7 @@ assert_sleepable(void)
} while (pctr != lwp_pctr());

reason = NULL;
- if (idle && !cold &&
- kcpuset_isset(kcpuset_running, cpu_index(curcpu()))) {
+ if (idle && !cold) {
reason = "idle";
}
if (cpu_intr_p()) {
--
2.25.1

Comments.

branches: 1.171.2;
Fix inverted condition in r1.136 - we do want LOCKDEBUG spinouts of the
kernel lock to assert as soon as we have userland running - not in
the early boot phase (where firmware loading and device init could take
a long time).

Kill off kernel_lock_plug_leak(), and go back to dropping kernel_lock in
exit1(), since there seems little hope of finding the leaking code any
time soon. Can still be caught with LOCKDEBUG.

Put back the delay hooks which were deleted before. Without them VirtualBox
spins out.

Add a kernel_lock_plug_leak() that drops any holds and tries to identify
the baddy.

Carefully put kernel_lock back the way it was, and add a comment hinting
that changing it is not a good idea, and hopefully nobody will ever try to
change it ever again.

- DIAGNOSTIC: check for leaked kernel_lock in mi_switch().

- Now that ci_biglock_wanted is set later, explicitly disable preemption
while acquiring kernel_lock. It was blocked in a roundabout way
previously.

Reported-by: syzbot+43111d810160fb4b978b@syzkaller.appspotmail.com
Reported-by: syzbot+f5b871bd00089bf97286@syzkaller.appspotmail.com
Reported-by: syzbot+cd1f15eee5b1b6d20078@syzkaller.appspotmail.com
Reported-by: syzbot+fb945a331dabd0b6ba9e@syzkaller.appspotmail.com
Reported-by: syzbot+53a0c2342b361db25240@syzkaller.appspotmail.com
Reported-by: syzbot+552222a952814dede7d1@syzkaller.appspotmail.com
Reported-by: syzbot+c7104a72172b0f9093a4@syzkaller.appspotmail.com
Reported-by: syzbot+efbd30c6ca0f7d8440e8@syzkaller.appspotmail.com
Reported-by: syzbot+330a421bd46794d8b750@syzkaller.appspotmail.com

kernel_lock:

- Defer setting ci_biglock_wanted for a bit, because if curlwp holds a mutex
or rwlock, and otherlwp is spinning waiting for the mutex/rwlock, setting
ci_biglock_wanted causes otherlwp to block to avoid deadlock. If the spin
on kernel_lock is short there's no point causing trouble.

- Do exponential backoff.

- Put the spinout check under LOCKDEBUG to match the others.

branches: 1.164.2;
Use __insn_barrier to enforce ordering in l_ncsw loops.

(Only need ordering observable by interruption, not by other CPUs.)

Avoid prepending a timestamp to lock debug outputs on ddb

Lock printer functions (lockops_t#lo_dump) use printf_nolog to print, but
printf_nolog now prepends a timestamp which is unnecessary for ddb:

db{0}> show all locks/t
[Locks tracked through LWPs]
Locks held by an LWP (iperf):
Lock 0 (initialized at soinit)
lock address : 0xffffedeb84b06080 type : sleep/adaptive
initialized : 0xffffffff806d8c3f
shared holds : 0 exclusive: 1
shares wanted: 0 exclusive: 11
current cpu : 0 last held: 1
current lwp : 0xffffedeb849ff040 last held: 0xffffedeb7dfdb240
last locked* : 0xffffffff806d8335 unlocked : 0xffffffff806d8385
[ 79103.0868574] owner field : 0xffffedeb7dfdb240 wait/spin: 1/0

Fix it by passing a printer function to lo_dump functions, i.e., make the
functions use db_printf on ddb.

Make _kernel_lock_dump static

branches: 1.161.4;
Apply C99-style struct initialization to lockops_t

Implement debugging feature for pserialize(9)

The debugging feature detects violations of pserialize constraints.
It causes a panic:
- if a context switch happens in a read section, or
- if a sleepable function is called in a read section.

The feature is enabled only if LOCKDEBUG is on.

Discussed on tech-kern@

more const

branches: 1.158.6;
For LOCKDEBUG:
Always provide the location of the caller of the lock as __func__, __LINE__.

branches: 1.157.2; 1.157.4;
Trailing whitespace

Allow sleeping in the idle lwp if the cpu isn't running yet.

OK'ed by rmind a while ago.

branches: 1.155.4; 1.155.6;
Move a CTASSERT to global scope (easiest way to avoid gcc 4.8.1 local unused
typedef warnings)

branches: 1.154.4;
Revert change that allowed rw_tryenter(&lock, RW_READER) to recurse
for vfs_busy(). This is no longer necessary.

branches: 1.153.2;
Use __cacheline_aligned