Introduce membar_acquire/release. Deprecate membar_enter/exit.

The names membar_enter/exit were unclear, and the documentation of
membar_enter has disagreed with the implementations on sparc,
powerpc, and even x86(!) for the entire time it has been in NetBSD.

The terms `acquire' and `release' are ubiquitous in the literature
today, and have been adopted in the C and C++ standards to mean
load-before-load/store and load/store-before-store, respectively,
which are exactly the orderings required by acquiring and releasing a
mutex, as well as other useful applications like decrementing a
reference count and then freeing the underlying object if it went to
zero.

Originally I proposed changing one word in the documentation for
membar_enter to make it load-before-load/store instead of
store-before-load/store, i.e., to make it an acquire barrier. I
proposed this on the grounds that

(a) all implementations guarantee load-before-load/store,
(b) some implementations fail to guarantee store-before-load/store,
and
(c) all uses in-tree assume load-before-load/store.

I verified parts (a) and (b) (except, for (a), powerpc didn't even
guarantee load-before-load/store -- isync isn't necessarily enough;
need lwsync in general -- but it _almost_ did, and it certainly didn't
guarantee store-before-load/store).

Part (c) might not be correct, however: under the mistaken assumption
that atomic-r/m/w then membar-w/rw is equivalent to atomic-r/m/w then
membar-r/rw, I only audited the cases of membar_enter that _aren't_
immediately after an atomic-r/m/w. All of those cases assume
load-before-load/store. But my assumption was wrong -- there are
cases of atomic-r/m/w then membar-w/rw that would be broken by
changing to atomic-r/m/w then membar-r/rw:

https://mail-index.netbsd.org/tech-kern/2022/03/29/msg028044.html

Furthermore, the name membar_enter has been adopted in other places
like OpenBSD where it actually does follow the documentation and
guarantee store-before-load/store, even if that order is not useful.
So the name membar_enter currently lives in a bad place where it
means either of two things -- r/rw or w/rw.

With this change, we deprecate membar_enter/exit, introduce
membar_acquire/release as better names for the useful pair (r/rw and
rw/w), and make sure the implementation of membar_enter guarantees
both what was documented _and_ what was implemented, making it an
alias for membar_sync.

While here, rework all of the membar_* definitions and aliases. The
new logic follows a rule to make it easier to audit:

membar_X is defined as an alias for membar_Y iff membar_X is
guaranteed by membar_Y.

The `no stronger than' relation is (the transitive closure of):

- membar_consumer (r/r) is guaranteed by membar_acquire (r/rw)
- membar_producer (w/w) is guaranteed by membar_release (rw/w)
- membar_acquire (r/rw) is guaranteed by membar_sync (rw/rw)
- membar_release (rw/w) is guaranteed by membar_sync (rw/rw)

And, for the deprecated membars:

- membar_enter (whether r/rw, w/rw, or rw/rw) is guaranteed by
membar_sync (rw/rw)
- membar_exit (rw/w) is guaranteed by membar_release (rw/w)

(membar_exit is identical to membar_release, but the name is
deprecated.)

Finally, while here, annotate some of the instructions with their
semantics. For powerpc, leave an essay with citations on the
unfortunate but -- as far as I can tell -- necessary decision to use
lwsync, not isync, for membar_acquire and membar_consumer.

Also add membar(3) and atomic(3) man page links.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Improve the membar_ops barriers - no need to use dsb and wait for
completion. Also, we only to act on the inner shareability domain.

branches: 1.6.26; 1.6.28;
Ensure SBZ register is zero

Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)

Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions

branches: 1.3.2;
Actually use the assembly version of the atomic function if compiling
for armv6 or armv7 cpus. Use atomic_cas_ptr instead of _lock_cas so
we pick up the assembly version when it's used.

branches: 1.2.2; 1.2.4; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex

branches: 1.1.2;
file membar_ops.S was initially added on branch thorpej-atomic.

Remove memory barriers from the atomic_ops(3) atomic operations. They're
not needed for correctness.

Add the correct memory barriers to the gcc legacy __sync built-in
functions for atomic memory access. From the gcc documentation:

In most cases, these built-in functions are considered a full barrier.
That is, no memory operand is moved across the operation, either forward
or backward. Further, instructions are issued as necessary to prevent the
processor from speculating loads across the operation and from queuing
stores after the operation.

type __sync_lock_test_and_set (type *ptr, type value, ...)

This built-in function is not a full barrier, but rather an acquire
barrier. This means that references after the operation cannot move to
(or be speculated to) before the operation, but previous memory stores
may not be globally visible yet, and previous memory loads may not yet
be satisfied.

void __sync_lock_release (type *ptr, ...)

This built-in function is not a full barrier, but rather a release
barrier. This means that all previous memory stores are globally
visible, and all previous memory loads have been satisfied, but
following memory reads are not prevented from being speculated to
before the barrier.

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Improve the membar_ops barriers - no need to use dsb and wait for
completion. Also, we only to act on the inner shareability domain.

branches: 1.6.26; 1.6.28;
Ensure SBZ register is zero

Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)

Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions

branches: 1.3.2;
Actually use the assembly version of the atomic function if compiling
for armv6 or armv7 cpus. Use atomic_cas_ptr instead of _lock_cas so
we pick up the assembly version when it's used.

branches: 1.2.2; 1.2.4; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex

branches: 1.1.2;
file membar_ops.S was initially added on branch thorpej-atomic.

s/ifdef _ARM_ARCH_6/if defined(_ARM_ARCH_6)/ for consistency. NFCI.

Improve the membar_ops barriers - no need to use dsb and wait for
completion. Also, we only to act on the inner shareability domain.

branches: 1.6.26; 1.6.28;
Ensure SBZ register is zero

Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)

Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions

branches: 1.3.2;
Actually use the assembly version of the atomic function if compiling
for armv6 or armv7 cpus. Use atomic_cas_ptr instead of _lock_cas so
we pick up the assembly version when it's used.

branches: 1.2.2; 1.2.4; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex

branches: 1.1.2;
file membar_ops.S was initially added on branch thorpej-atomic.

Improve the membar_ops barriers - no need to use dsb and wait for
completion. Also, we only to act on the inner shareability domain.

branches: 1.6.26;
Ensure SBZ register is zero

Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)

Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions

branches: 1.3.2;
Actually use the assembly version of the atomic function if compiling
for armv6 or armv7 cpus. Use atomic_cas_ptr instead of _lock_cas so
we pick up the assembly version when it's used.

branches: 1.2.2; 1.2.4; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex

branches: 1.1.2;
file membar_ops.S was initially added on branch thorpej-atomic.

Ensure SBZ register is zero

Don't export __sync* if _KERNEL || _STANDALONE are defined.
(except if _RUMPKERNEL is defined)

Add support for the gcc __sync builtins.
Note that these need earmv6 or later to get the ldrex/strex instructions

branches: 1.3.2;
Actually use the assembly version of the atomic function if compiling
for armv6 or armv7 cpus. Use atomic_cas_ptr instead of _lock_cas so
we pick up the assembly version when it's used.

branches: 1.2.2; 1.2.4; 1.2.24;
Add assembly versions of atomic ops with ldrex/strex

branches: 1.1.2;
file membar_ops.S was initially added on branch thorpej-atomic.

Add armv6 variants of atomic ops which use ldrex/strex.
Add atomic_swap (all arm architectures) which uses swp and swpb.
Add membar_ops which uses armv6 data {sync, memory} barrier.
XXX let thorpej figure out the Makefile magic.

Pull atomic ops from HEAD.

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs. ("Protocol error: too many arguments")

sync with head

Sync with wrstuden-revivesa-base-2.

file membar_ops.S was added on branch wrstuden-revivesa on 2008-09-18 04:54:18 +0000

Rebase to HEAD as of a few days ago.