#
331722 |
|
29-Mar-2018 |
eadler |
Revert r330897:
This was intended to be a non-functional change. It wasn't. The commit message was thus wrong. In addition it broke arm, and merged crypto related code.
Revert with prejudice.
This revert skips files touched in r316370 since that commit was since MFCed. This revert also skips files that require $FreeBSD$ property changes.
Thank you to those who helped me get out of this mess including but not limited to gonzo, kevans, rgrimes.
Requested by: gjb (re)
|
#
330897 |
|
14-Mar-2018 |
eadler |
Partial merge of the SPDX changes
These changes are incomplete but are making it difficult to determine what other changes can/should be merged.
No objections from: pfg
|
#
327413 |
|
31-Dec-2017 |
mjg |
MFC r320561,r323236,r324041,r324314,r324609,r324613,r324778,r324780,r324787, r324803,r324836,r325469,r325706,r325917,r325918,r325919,r325920,r325921, r325922,r325925,r325963,r326106,r326107,r326110,r326111,r326112,r326194, r326195,r326196,r326197,r326198,r326199,r326200,r326237:
rwlock: perform the typically false td_rw_rlocks check later
Check if the lock is available first instead.
=============
Sprinkle __read_frequently on few obvious places.
Note that some of annotated variables should probably change their types to something smaller, preferably bit-sized.
=============
mtx: drop the tid argument from _mtx_lock_sleep
tid must be equal to curthread and the target routine was already reading it anyway, which is not a problem. Not passing it as a parameter allows for a little bit shorter code in callers.
=============
locks: partially tidy up waiting on readers
spin first instant of instantly re-readoing and don't re-read after spinning is finished - the state is already known.
Note the code is subject to significant changes later.
=============
locks: take the number of readers into account when waiting
Previous code would always spin once before checking the lock. But a lock with e.g. 6 readers is not going to become free in the duration of once spin even if they start draining immediately.
Conservatively perform one for each reader.
Note that the total number of allowed spins is still extremely small and is subject to change later.
=============
mtx: change MTX_UNOWNED from 4 to 0
The value is spread all over the kernel and zeroing a register is cheaper/shorter than setting it up to an arbitrary value.
Reduces amd64 GENERIC-NODEBUG .text size by 0.4%.
=============
mtx: fix up owner_mtx after r324609
Now that MTX_UNOWNED is 0 the test was alwayas false.
=============
mtx: clean up locking spin mutexes
1) shorten the fast path by pushing the lockstat probe to the slow path 2) test for kernel panic only after it turns out we will have to spin, in particular test only after we know we are not recursing
=============
mtx: stop testing SCHEDULER_STOPPED in kabi funcs for spin mutexes
There is nothing panic-breaking to do in the unlock case and the lock case will fallback to the slow path doing the check already.
=============
rwlock: reduce lockstat branches in the slowpath
=============
mtx: fix up UP build after r324778
=============
mtx: implement thread lock fastpath
=============
rwlock: fix up compilation without KDTRACE_HOOKS after r324787
=============
rwlock: use fcmpset for setting RW_LOCK_WRITE_SPINNER
=============
sx: avoid branches if in the slow path if lockstat is disabled
=============
rwlock: avoid branches in the slow path if lockstat is disabled
=============
locks: pull up PMC_SOFT_CALLs out of slow path loops
=============
mtx: unlock before traversing threads to wake up
This shortens the lock hold time while not affecting corretness. All the woken up threads end up competing can lose the race against a completely unrelated thread getting the lock anyway.
=============
rwlock: unlock before traversing threads to wake up
While here perform a minor cleanup of the unlock path.
=============
sx: perform a minor cleanup of the unlock slowpath
No functional changes.
=============
mtx: add missing parts of the diff in r325920
Fixes build breakage.
=============
locks: fix compilation issues without SMP or KDTRACE_HOOKS
=============
locks: remove the file + line argument from internal primitives when not used
The pair is of use only in debug or LOCKPROF kernels, but was passed (zeroed) for many locks even in production kernels.
While here whack the tid argument from wlock hard and xlock hard.
There is no kbi change of any sort - "external" primitives still accept the pair.
=============
locks: pass the found lock value to unlock slow path
This avoids an explicit read later.
While here whack the cheaply obtainable 'tid' argument.
=============
rwlock: don't check for curthread's read lock count in the fast path
=============
rwlock: unbreak WITNESS builds after r326110
=============
sx: unbreak debug after r326107
An assertion was modified to use the found value, but it was not updated to handle a race where blocked threads appear after the entrance to the func.
Move the assertion down to the area protected with sleepq lock where the lock is read anyway. This does not affect coverage of the assertion and is consistent with what rw locks are doing.
=============
rwlock: stop re-reading the owner when going to sleep
=============
locks: retry turnstile/sleepq loops on failed cmpset
In order to go to sleep threads set waiter flags, but that can spuriously fail e.g. when a new reader arrives. Instead of unlocking everything and looping back, re-evaluate the new state while still holding the lock necessary to go to sleep.
=============
sx: change sunlock to wake waiters up if it locked sleepq
sleepq is only locked if the curhtread is the last reader. By the time the lock gets acquired new ones could have arrived. The previous code would unlock and loop back. This results spurious relocking of sleepq.
This is a step towards xadd-based unlock routine.
=============
rwlock: add __rw_try_{r,w}lock_int
=============
rwlock: fix up compilation of the previous change
commmitted wrong version of the patch
=============
Convert in-kernel thread_lock_flags calls to thread_lock when debug is disabled
The flags argument is not used in this case.
=============
Add the missing lockstat check for thread lock.
=============
rw: fix runlock_hard when new readers show up
When waiters/writer spinner flags are set no new readers can show up unless they already have a different rw rock read locked. The change in r326195 failed to take that into account - in presence of new readers it would spin until they all drain, which would be lead to trouble if e.g. they go off cpu and can get scheduled because of this thread.
|
#
315394 |
|
16-Mar-2017 |
mjg |
MFC,r313855,r313865,r313875,r313877,r313878,r313901,r313908,r313928,r313944,r314185,r314476,r314187
locks: let primitives for modules unlock without always goging to the slsow path
It is only needed if the LOCK_PROFILING is enabled. It has to always check if the lock is about to be released which requires an avoidable read if the option is not specified..
==
sx: fix compilation on UP kernels after r313855
sx primitives use inlines as opposed to macros. Change the tested condition to LOCK_DEBUG which covers the case, but is slightly overzelaous.
commit a39b839d16cd72b1df284ccfe6706fcdf362706e Author: mjg <mjg@ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f> Date: Sat Feb 18 22:06:03 2017 +0000
locks: clean up trylock primitives
In particular thius reduces accesses of the lock itself.
git-svn-id: svn+ssh://svn.freebsd.org/base/head@313928 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
commit 013560e742a5a276b0deef039bc18078d51d6eb0 Author: mjg <mjg@ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f> Date: Sat Feb 18 01:52:10 2017 +0000
mtx: plug the 'opts' argument when not used
git-svn-id: svn+ssh://svn.freebsd.org/base/head@313908 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
commit 9a507901162fb476b9809da2919905735cd605af Author: mjg <mjg@ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f> Date: Fri Feb 17 22:09:55 2017 +0000
sx: fix mips builld after r313855
The namespace in this file really needs cleaning up. In the meantime let inline primitives be defined as long as LOCK_DEBUG is not enabled.
Reported by: kib
git-svn-id: svn+ssh://svn.freebsd.org/base/head@313901 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
commit aa6243a5124b9ceb3b1683ea4dbb0a133ce70095 Author: mjg <mjg@ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f> Date: Fri Feb 17 15:40:24 2017 +0000
mtx: get rid of file/line args from slow paths if they are unused
This denotes changes which went in by accident in r313877.
On most production kernels both said parameters are zeroed and have nothing reading them in either __mtx_lock_sleep or __mtx_unlock_sleep. Thus this change stops passing them by internal consumers which this is the case.
Kernel modules use _flags variants which are not affected kbi-wise.
git-svn-id: svn+ssh://svn.freebsd.org/base/head@313878 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
commit 688545a6af7ed0972653d6e2c6ca406ac511f39d Author: mjg <mjg@ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f> Date: Fri Feb 17 15:34:40 2017 +0000
mtx: restrict r313875 to kernels without LOCK_PROFILING
git-svn-id: svn+ssh://svn.freebsd.org/base/head@313877 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f
commit bbe6477138713da2d503f93cb5dd602e14152a08 Author: mjg <mjg@ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f> Date: Fri Feb 17 14:55:59 2017 +0000
mtx: microoptimize lockstat handling in __mtx_lock_sleep
This saves a function call and multiple branches after the lock is acquired.
overzelaous
|
#
315378 |
|
16-Mar-2017 |
mjg |
MFC r313275,r313280,r313282,r313335:
mtx: move lockstat handling out of inline primitives
Lockstat requires checking if it is enabled and if so, calling a 6 argument function. Further, determining whether to call it on unlock requires pre-reading the lock value.
This is problematic in at least 3 ways: - more branches in the hot path than necessary - additional cacheline ping pong under contention - bigger code
Instead, check first if lockstat handling is necessary and if so, just fall back to regular locking routines. For this purpose a new macro is introduced (LOCKSTAT_PROFILE_ENABLED).
LOCK_PROFILING uninlines all primitives. Fold in the current inline lock variant into the _mtx_lock_flags to retain the support. With this change the inline variants are not used when LOCK_PROFILING is defined and thus can ignore its existence.
This results in: text data bss dec hex filename 22259667 1303208 4994976 28557851 1b3c21b kernel.orig 21797315 1303208 4994976 28095499 1acb40b kernel.patched
i.e. about 3% reduction in text size.
A remaining action is to remove spurious arguments for internal kernel consumers.
==
sx: move lockstat handling out of inline primitives
See r313275 for details.
==
rwlock: move lockstat handling out of inline primitives
See r313275 for details.
One difference here is that recursion handling was removed from the fallback routine. As it is it was never supposed to see a recursed lock in the first place. Future changes will move it out of inline variants, but right now there is no easy to way to test if the lock is recursed without reading additional words.
==
locks: fix recursion support after recent changes
When a relevant lockstat probe is enabled the fallback primitive is called with a constant signifying a free lock. This works fine for typical cases but breaks with recursion, since it checks if the passed value is that of the executing thread.
Read the value if necessary.
|
#
315377 |
|
16-Mar-2017 |
mjg |
MFC r313269,r313270,r313271,r313272,r313274,r313278,r313279,r313996,r314474
mtx: switch to fcmpset
The found value is passed to locking routines in order to reduce cacheline accesses.
mtx_unlock grows an explicit check for regular unlock. On ll/sc architectures the routine can fail even if the lock could have been handled by the inline primitive.
==
rwlock: switch to fcmpset
==
sx: switch to fcmpset
==
sx: uninline slock/sunlock
Shared locking routines explicitly read the value and test it. If the change attempt fails, they fall back to a regular function which would retry in a loop.
The problem is that with many concurrent readers the risk of failure is pretty high and even the value returned by fcmpset is very likely going to be stale by the time the loop in the fallback routine is reached.
Uninline said primitives. It gives a throughput increase when doing concurrent slocks/sunlocks with 80 hardware threads from ~50 mln/s to ~56 mln/s.
Interestingly, rwlock primitives are already not inlined.
==
sx: add witness support missed in r313272
==
mtx: fix up _mtx_obtain_lock_fetch usage in thread lock
Since _mtx_obtain_lock_fetch no longer sets the argument to MTX_UNOWNED, callers have to do it on their own.
==
mtx: fixup r313278, the assignemnt was supposed to go inside the loop
==
mtx: fix spin mutexes interaction with failed fcmpset
While doing so move recursion support down to the fallback routine.
==
locks: ensure proper barriers are used with atomic ops when necessary
Unclear how, but the locking routine for mutexes was using the *release* barrier instead of acquire. This must have been either a copy-pasto or bad completion.
Going through other uses of atomics shows no barriers in: - upgrade routines (addressed in this patch) - sections protected with turnstile locks - this should be fine as necessary barriers are in the worst case provided by turnstile unlock
I would like to thank Mark Millard and andreast@ for reporting the problem and testing previous patches before the issue got identified.
|
#
315341 |
|
16-Mar-2017 |
mjg |
MFC r311172,r311194,r311226,r312389,r312390:
mtx: reduce lock accesses
Instead of spuriously re-reading the lock value, read it once.
This change also has a side effect of fixing a performance bug: on failed _mtx_obtain_lock, it was possible that re-read would find the lock is unowned, but in this case the primitive would make a trip through turnstile code.
This is diff reduction to a variant which uses atomic_fcmpset.
==
Reduce lock accesses in thread lock similarly to r311172
==
mtx: plug open-coded mtx_lock access missed in r311172
==
rwlock: reduce lock accesses similarly to r311172
==
sx: reduce lock accesses similarly to r311172
|
#
302408 |
|
07-Jul-2016 |
gjb |
Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle. Prune svn:mergeinfo from the new branch, as nothing has been merged here.
Additional commits post-branch will follow.
Approved by: re (implicit) Sponsored by: The FreeBSD Foundation |
#
301157 |
|
01-Jun-2016 |
mjg |
Microoptimize locking primitives by avoiding unnecessary atomic ops.
Inline version of primitives do an atomic op and if it fails they fallback to actual primitives, which immediately retry the atomic op.
The obvious optimisation is to check if the lock is free and only then proceed to do an atomic op.
Reviewed by: jhb, vangyzen
|
#
298981 |
|
03-May-2016 |
pfg |
sys/sys: minor spelling fixes.
While the changes are minor, these headers are very visible.
MFC after: 2 weeks
|
#
285704 |
|
19-Jul-2015 |
markj |
Consistently use a reader/writer flag for lockstat probes in rwlock(9) and sx(9), rather than using the probe function name to determine whether a given lock is a read lock or a write lock. Update lockstat(1) accordingly.
|
#
285703 |
|
19-Jul-2015 |
markj |
Implement the lockstat provider using SDT(9) instead of the custom provider in lockstat.ko. This means that lockstat probes now have typed arguments and will utilize SDT probe hot-patching support when it arrives.
Reviewed by: gnn Differential Revision: https://reviews.freebsd.org/D2993
|
#
275751 |
|
13-Dec-2014 |
dchagin |
Add _NEW flag to mtx(9), sx(9), rmlock(9) and rwlock(9). A _NEW flag passed to _init_flags() to avoid check for double-init.
Differential Revision: https://reviews.freebsd.org/D1208 Reviewed by: jhb, wblock MFC after: 1 Month
|
#
258541 |
|
25-Nov-2013 |
attilio |
- For kernel compiled only with KDTRACE_HOOKS and not any lock debugging option, unbreak the lock tracing release semantic by embedding calls to LOCKSTAT_PROFILE_RELEASE_LOCK() direclty in the inlined version of the releasing functions for mutex, rwlock and sxlock. Failing to do so skips the lockstat_probe_func invokation for unlocking. - As part of the LOCKSTAT support is inlined in mutex operation, for kernel compiled without lock debugging options, potentially every consumer must be compiled including opt_kdtrace.h. Fix this by moving KDTRACE_HOOKS into opt_global.h and remove the dependency by opt_kdtrace.h for all files, as now only KDTRACE_FRAMES is linked there and it is only used as a compile-time stub [0].
[0] immediately shows some new bug as DTRACE-derived support for debug in sfxge is broken and it was never really tested. As it was not including correctly opt_kdtrace.h before it was never enabled so it was kept broken for a while. Fix this by using a protection stub, leaving sfxge driver authors the responsibility for fixing it appropriately [1].
Sponsored by: EMC / Isilon storage division Discussed with: rstone [0] Reported by: rstone [1] Discussed with: philip
|
#
247787 |
|
04-Mar-2013 |
davide |
MFcalloutng: Introduce sbt variants of msleep(), msleep_spin(), pause(), tsleep() in the KPI, allowing to specify timeout in 'sbintime_t' rather than ticks.
Sponsored by: Google Summer of Code 2012, iXsystems inc. Tested by: flo, marius, ian, markj, Fabian Keil
|
#
227788 |
|
21-Nov-2011 |
attilio |
Introduce the same mutex-wise fix in r227758 for sx locks.
The functions that offer file and line specifications are: - sx_assert_ - sx_downgrade_ - sx_slock_ - sx_slock_sig_ - sx_sunlock_ - sx_try_slock_ - sx_try_xlock_ - sx_try_upgrade_ - sx_unlock_ - sx_xlock_ - sx_xlock_sig_ - sx_xunlock_
Now vm_map locking is fully converted and can avoid to know specifics about locking procedures. Reviewed by: kib MFC after: 1 month
|
#
227588 |
|
16-Nov-2011 |
pjd |
Constify arguments for locking KPIs where possible.
This enables locking consumers to pass their own structures around as const and be able to assert locks embedded into those structures.
Reviewed by: ed, kib, jhb
|
#
219819 |
|
21-Mar-2011 |
jeff |
- Merge changes to the base system to support OFED. These include a wider arg2 for sysctl, updates to vlan code, IFT_INFINIBAND, and other miscellaneous small features.
|
#
197643 |
|
30-Sep-2009 |
attilio |
When releasing a read/shared lock we need to use a write memory barrier in order to avoid, on architectures which doesn't have strong ordered writes, CPU instructions reordering.
Diagnosed by: fabio Reviewed by: jhb Tested by: Giovanni Trematerra <giovanni dot trematerra at gmail dot com>
|
#
194578 |
|
21-Jun-2009 |
rdivacky |
In non-debugging mode make this define (void)0 instead of nothing. This helps to catch bugs like the below with clang.
if (cond); <--- note the trailing ; something();
Approved by: ed (mentor) Discussed on: current@
|
#
193011 |
|
28-May-2009 |
attilio |
Reverse the logic for ADAPTIVE_SX option and enable it by default. Introduce for this operation the reverse NO_ADAPTIVE_SX option. The flag SX_ADAPTIVESPIN to be passed to sx_init_flags(9) gets suppressed and the new flag, offering the reversed logic, SX_NOADAPTIVE is added.
Additively implements adaptive spininning for sx held in shared mode. The spinning limit can be handled through sysctls in order to be tuned while the code doesn't reach the release, after which time they should be dropped probabilly.
This change has made been necessary by recent benchmarks where it does improve concurrency of workloads in presence of high contention (ie. ZFS).
KPI breakage is documented by __FreeBSD_version bumping, manpage and UPDATING updates.
Requested by: jeff, kmacy Reviewed by: jeff Tested by: pho
|
#
192853 |
|
26-May-2009 |
sson |
Add the OpenSolaris dtrace lockstat provider. The lockstat provider adds probes for mutexes, reader/writer and shared/exclusive locks to gather contention statistics and other locking information for dtrace scripts, the lockstat(1M) command and other potential consumers.
Reviewed by: attilio jhb jb Approved by: gnn (mentor)
|
#
181682 |
|
13-Aug-2008 |
ed |
Fix compilation of arm's AVILA.
Compilation of the AVILA kernel failed because of two reasons:
- It needed curthread, which is defined through <sys/pcpu.h>.
- It still referred the softc's sc_mtx field, which has been replaced by sc_lock three weeks ago.
To solve the first problem, I decided to include <sys/pcpu.h> in <sys/sx.h>, which also seems to be done by <sys/mutex.h> and <sys/rwlock.h>. Those header files also require curthread.
Approved by: jhb
|
#
174629 |
|
15-Dec-2007 |
jeff |
- Re-implement lock profiling in such a way that it no longer breaks the ABI when enabled. There is no longer an embedded lock_profile_object in each lock. Instead a list of lock_profile_objects is kept per-thread for each lock it may own. The cnt_hold statistic is now always 0 to facilitate this. - Support shared locking by tracking individual lock instances and statistics in the per-thread per-instance lock_profile_object. - Make the lock profiling hash table a per-cpu singly linked list with a per-cpu static lock_prof allocator. This removes the need for an array of spinlocks and reduces cache contention between cores. - Use a seperate hash for spinlocks and other locks so that only a critical_enter() is required and not a spinlock_enter() to modify the per-cpu tables. - Count time spent spinning in the lock statistics. - Remove the LOCK_PROFILE_SHARED option as it is always supported now. - Specifically drop and release the scheduler locks in both schedulers since we track owners now.
In collaboration with: Kip Macy Sponsored by: Nokia
|
#
171277 |
|
06-Jul-2007 |
attilio |
Fix some problems with lock_profiling in sx locks: - Adjust lock_profiling stubs semantic in the hard functions in order to be more accurate and trustable - Disable shared paths for lock_profiling. Actually, lock_profiling has a subtle race which makes results caming from shared paths not completely trustable. A macro stub (LOCK_PROFILING_SHARED) can be actually used for re-enabling this paths, but is currently intended for developing use only. - Use homogeneous names for automatic variables in hard functions regarding lock_profiling - Style fixes - Add a CTASSERT for some flags building
Discussed with: kmacy, kris Approved by: jeff (mentor) Approved by: re
|
#
170149 |
|
31-May-2007 |
attilio |
Add functions sx_xlock_sig() and sx_slock_sig(). These functions are intended to do the same actions of sx_xlock() and sx_slock() but with the difference to perform an interruptible sleep, so that sleep can be interrupted by external events. In order to support these new featueres, some code renstruction is needed, but external API won't be affected at all.
Note: use "void" cast for "int" returning functions in order to avoid tools like Coverity prevents to whine.
Requested by: rwatson Tested by: rwatson Reviewed by: jhb Approved by: jeff (mentor)
|
#
170115 |
|
29-May-2007 |
attilio |
style(9) fixes for sx locks.
Approved by: jeff (mentor)
|
#
169780 |
|
19-May-2007 |
jhb |
Rename the macros for assertion flags passed to sx_assert() from SX_* to SA_* to match mutexes and rwlocks. The old flags still exist for backwards compatiblity.
Requested by: attilio
|
#
169776 |
|
19-May-2007 |
jhb |
Expose sx_xholder() as a public macro. It returns a pointer to the thread that holds the current exclusive lock, or NULL if no thread holds an exclusive lock.
Requested by: pjd
|
#
169769 |
|
19-May-2007 |
jhb |
Add a new SX_RECURSE flag to make support for recursive exclusive locks conditional. By default, sx(9) locks are back to not supporting recursive exclusive locks.
Submitted by: attilio
|
#
169394 |
|
08-May-2007 |
jhb |
Add destroyed cookie values for sx locks and rwlocks as well as extra KASSERTs so that any lock operations on a destroyed lock will panic or hang.
|
#
168330 |
|
03-Apr-2007 |
kmacy |
Fixes to sx for newsx - fix recursed case and move out of inline
Submitted by: Attilio Rao <attilio@freebsd.org>
|
#
168191 |
|
31-Mar-2007 |
jhb |
Optimize sx locks to use simple atomic operations for the common cases of obtaining and releasing shared and exclusive locks. The algorithms for manipulating the lock cookie are very similar to that rwlocks. This patch also adds support for exclusive locks using the same algorithm as mutexes.
A new sx_init_flags() function has been added so that optional flags can be specified to alter a given locks behavior. The flags include SX_DUPOK, SX_NOWITNESS, SX_NOPROFILE, and SX_QUITE which are all identical in nature to the similar flags for mutexes.
Adaptive spinning on select locks may be enabled by enabling the ADAPTIVE_SX kernel option. Only locks initialized with the SX_ADAPTIVESPIN flag via sx_init_flags() will adaptively spin.
The common cases for sx_slock(), sx_sunlock(), sx_xlock(), and sx_xunlock() are now performed inline in non-debug kernels. As a result, <sys/sx.h> now requires <sys/lock.h> to be included prior to <sys/sx.h>.
The new kernel option SX_NOINLINE can be used to disable the aforementioned inlining in non-debug kernels.
The size of struct sx has changed, so the kernel ABI is probably greatly disturbed.
MFC after: 1 month Submitted by: attilio Tested by: kris, pjd
|
#
167787 |
|
21-Mar-2007 |
jhb |
Rename the 'mtx_object', 'rw_object', and 'sx_object' members of mutexes, rwlocks, and sx locks to 'lock_object'.
|
#
167387 |
|
09-Mar-2007 |
jhb |
Allow threads to atomically release rw and sx locks while waiting for an event. Locking primitives that support this (mtx, rw, and sx) now each include their own foo_sleep() routine. - Rename msleep() to _sleep() and change it's 'struct mtx' object to a 'struct lock_object' pointer. _sleep() uses the recently added lc_unlock() and lc_lock() function pointers for the lock class of the specified lock to release the lock while the thread is suspended. - Add wrappers around _sleep() for mutexes (mtx_sleep()), rw locks (rw_sleep()), and sx locks (sx_sleep()). msleep() still exists and is now identical to mtx_sleep(), but it is deprecated. - Rename SLEEPQ_MSLEEP to SLEEPQ_SLEEP. - Rewrite much of sleep.9 to not be msleep(9) centric. - Flesh out the 'RETURN VALUES' section in sleep.9 and add an 'ERRORS' section. - Add __nonnull(1) to _sleep() and msleep_spin() so that the compiler will warn if you try to pass a NULL wait channel. The functions already have a KASSERT to that effect.
|
#
161721 |
|
29-Aug-2006 |
jhb |
The _sx_assert() prototype should exist if either of INVARIANTS or INVARIANT_SUPPORT is defined so you can build a kernel with INVARIANT_SUPPORT, but build a module with just INVARIANTS on.
MFC after: 3 days Reported by: kuriyama
|
#
161337 |
|
15-Aug-2006 |
jhb |
Add a new 'show sleepchain' ddb command similar to 'show lockchain' except that it operates on lockmgr and sx locks. This can be useful for tracking down vnode deadlocks in VFS for example. Note that this command is a bit more fragile than 'show lockchain' as we have to poke around at the wait channel of a thread to see if it points to either a struct lock or a condition variable inside of a struct sx. If td_wchan points to something unmapped, then this command will terminate early due to a fault, but no harm will be done.
|
#
159844 |
|
21-Jun-2006 |
jhb |
Add a sx_xlocked() macro which returns true if the current thread holds an exclusive lock on the specified sx lock.
|
#
157296 |
|
30-Mar-2006 |
jhb |
Style fix.
|
#
149739 |
|
02-Sep-2005 |
jhb |
Add a SYSUNINIT() to SX_SYSINIT() to call sx_destroy() to destroy the sx lock when a module is unloaded similar to the recent change made to MTX_SYSINIT().
Suggested by: pjd MFC after: 3 days
|
#
139825 |
|
07-Jan-2005 |
imp |
/* -> /*- for license, minor formatting changes
|
#
131984 |
|
11-Jul-2004 |
darrenr |
Add sx_unlock() macro as a frontend to both sx_sunlock() and sx_xunlock(), using sx_cnt to determine what state the lock is in and call the respective function appropriately.
|
#
125444 |
|
04-Feb-2004 |
bde |
Include <sys/queue.h> before <sys/_lock.h> instead of depending on namespace pollution in other headers. <sys/types.h> is now the only prerequisite for <sys/sx.h>.
Fixed some style bugs: - removed bogus LOCORE ifdef. Including this C header in assembler sources is just nonsense. - removed unused include of <sys/_mutex.h>. It finished rotting when the mutex in struct sx became indirect in rev.1.15. - removed most comments on #else and #endif's and cleaned up the others. All were misindented...
|
#
125419 |
|
04-Feb-2004 |
pjd |
Add SX_UNLOCKED define. It will be used with sx_assert(9) to be sure that current thread does not hold given sx(9) lock.
Reviewed by: jhb Approved by: jhb, scottl (mentor)
|
#
93688 |
|
02-Apr-2002 |
arr |
- Make this compile if INVARIANTS support is not enabled.
|
#
93672 |
|
02-Apr-2002 |
arr |
- Add MTX_SYSINIT and SX_SYSINIT as macro glue for allowing sx and mtx locks to be able to setup a SYSINIT call. This helps in places where a lock is needed to protect some data, but the data is not truly associated with a subsystem that can properly initialize it's lock. The macros use the mtx_sysinit() and sx_sysinit() functions, respectively, as the handler argument to SYSINIT().
Reviewed by: alfred, jhb, smp@
|
#
86333 |
|
13-Nov-2001 |
dillon |
Create a mutex pool API for short term leaf mutexes. Replace the manual mutex pool in kern_lock.c (lockmgr locks) with the new API. Replace the mutexes embedded in sxlocks with the new API.
|
#
85412 |
|
24-Oct-2001 |
jhb |
Fix this to actually compile in the !INVARIANTS case.
Reported by: Maxime Henrion <mux@qualys.com>
|
#
85388 |
|
23-Oct-2001 |
jhb |
Change the sx(9) assertion API to use a sx_assert() function similar to mtx_assert(9) rather than several SX_ASSERT_* macros.
|
#
83593 |
|
17-Sep-2001 |
jhb |
Use NULL instead of __FILE__ in the !LOCK_DEBUG case in the locking code since the filenames are only used in the LOCK_DEBUG case and are just bloat in the !LOCK_DEBUG case.
|
#
83366 |
|
12-Sep-2001 |
julian |
KSE Milestone 2 Note ALL MODULES MUST BE RECOMPILED make the kernel aware that there are smaller units of scheduling than the process. (but only allow one thread per process at this time). This is functionally equivalent to teh previousl -current except that there is a thread associated with each process.
Sorry john! (your next MFC will be a doosie!)
Reviewed by: peter@freebsd.org, dillon@freebsd.org
X-MFC after: ha ha ha ha
|
#
83103 |
|
05-Sep-2001 |
jhb |
Include <sys/_lock.h> for the definition of struct lock_object. Don't understand why this wasn't added when _mutex.h was added.
Noticed by: jlemon
|
#
81599 |
|
13-Aug-2001 |
jasone |
Add sx_try_upgrade() and sx_downgrade().
Submitted by: Alexander Kabaev <ak03@gte.com>
|
#
78872 |
|
27-Jun-2001 |
jhb |
- Add trylock variants of shared and exclusive locks. - The sx assertions don't actually need the internal sx mutex lock, so don't bother doing so. - Add a new assertion SX_ASSERT_LOCKED() that asserts that either a shared or exclusive lock should be held. This assertion should be used instead of SX_ASSERT_SLOCKED() in almost all cases. - Adjust some KASSERT()'s to include file and line information. - Use the new witness_assert() function in the WITNESS case for sx slock asserts to verify that the current thread actually owns a slock.
|
#
76166 |
|
01-May-2001 |
markm |
Undo part of the tangle of having sys/lock.h and sys/mutex.h included in other "system" header files.
Also help the deprecation of lockmgr.h by making it a sub-include of sys/lock.h and removing sys/lockmgr.h form kernel .c files.
Sort sys/*.h includes where possible in affected files.
OK'ed by: bde (with reservations)
|
#
74912 |
|
28-Mar-2001 |
jhb |
Rework the witness code to work with sx locks as well as mutexes. - Introduce lock classes and lock objects. Each lock class specifies a name and set of flags (or properties) shared by all locks of a given type. Currently there are three lock classes: spin mutexes, sleep mutexes, and sx locks. A lock object specifies properties of an additional lock along with a lock name and all of the extra stuff needed to make witness work with a given lock. This abstract lock stuff is defined in sys/lock.h. The lockmgr constants, types, and prototypes have been moved to sys/lockmgr.h. For temporary backwards compatability, sys/lock.h includes sys/lockmgr.h. - Replace proc->p_spinlocks with a per-CPU list, PCPU(spinlocks), of spin locks held. By making this per-cpu, we do not have to jump through magic hoops to deal with sched_lock changing ownership during context switches. - Replace proc->p_heldmtx, formerly a list of held sleep mutexes, with proc->p_sleeplocks, which is a list of held sleep locks including sleep mutexes and sx locks. - Add helper macros for logging lock events via the KTR_LOCK KTR logging level so that the log messages are consistent. - Add some new flags that can be passed to mtx_init(): - MTX_NOWITNESS - specifies that this lock should be ignored by witness. This is used for the mutex that blocks a sx lock for example. - MTX_QUIET - this is not new, but you can pass this to mtx_init() now and no events will be logged for this lock, so that one doesn't have to change all the individual mtx_lock/unlock() operations. - All lock objects maintain an initialized flag. Use this flag to export a mtx_initialized() macro that can be safely called from drivers. Also, we on longer walk the all_mtx list if MUTEX_DEBUG is defined as witness performs the corresponding checks using the initialized flag. - The lock order reversal messages have been improved to output slightly more accurate file and line numbers.
|
#
73901 |
|
06-Mar-2001 |
jhb |
In order to avoid recursing on the backing mutex for sx locks in the INVARIANTS case, define the actual KASSERT() in _SX_ASSERT_[SX]LOCKED macros that are used in the sx code itself and convert the SX_ASSERT_[SX]LOCKED macros to simple wrappers that grab the mutex for the duration of the check.
|
#
73900 |
|
06-Mar-2001 |
jhb |
Get the arguments to the KASSERT() printf() in SX_ASSERT_XLOCKED() in the proper order.
|
#
73874 |
|
06-Mar-2001 |
dwmalone |
Fix typo: define SX_ASSERT_XLOCKED not SX_ASSERT_XLOCKER in non-INVARIANTS case.
PR: 25567 Submitted by: nnd@mail.nsk.ru
|
#
73863 |
|
06-Mar-2001 |
bmilekic |
- Add sx_descr description member to sx lock structure - Add sx_xholder member to sx struct which is used for INVARIANTS-enabled assertions. It indicates the thread that presently owns the xlock. - Add some assertions to the sx lock code that will detect the fatal API abuse: xlock --> xlock xlock --> slock which now works thanks to sx_xholder. Notice that the remaining two problematic cases: slock --> xlock slock --> slock (a little less problematic, but still recursion) will need to be handled by witness eventually, as they are more involved.
Reviewed by: jhb, jake, jasone
|
#
73782 |
|
05-Mar-2001 |
jasone |
Implement shared/exclusive locks.
Reviewed by: bmilekic, jake, jhb
|