MFC 332891,332892: Fixes for atomic_*cmpset() on arm.

332891:
Fix some harmless type mismatches in the ARM atomic_cmpset implementations.

The return value of atomic_cmpset() and atomic_fcmpset() is an int (which
is really a bool) that has the values 0 or 1. Some of the inlines were
using the type being operated on (e.g. uint32_t) as either the return type
of the function, or the type of a local 'ret' variable used to hold the
return value. Fix all of these to just use plain 'int'. Due to C promotion
rules and the fact that the value can only be 0 or 1, these should all be
harmless.

332892:
Implement 32-bit atomic_fcmpset() in userland for armv4/v5.

- Add an implementation of atomic_fcmpset_32() using RAS for armv4/v5.
This fixes recent world breakage due to use of atomic_fcmpset() in
userland.
- While here, be more careful to not expose wrapper macros for 64-bit
atomic_*cmpset to userland for armv4/v5 as only 32-bit cmpset is
implemented.

This has been reviewed, but not runtime-tested, but should fix the arm.arm
and arm.armeb worlds that have been broken for a while.

Approved by: re (kib)

MFC r314568 (by emaste):

kern_sig.c: ANSIfy and remove archaic register keyword

Sponsored by: The FreeBSD Foundation

MFC r318389 (by emaste):

Remove register keyword from sys/ and ANSIfy prototypes

A long long time ago the register keyword told the compiler to store
the corresponding variable in a CPU register, but it is not relevant
for any compiler used in the FreeBSD world today.

ANSIfy related prototypes while here.

Reviewed by: cem, jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10193

MFC r312932,r312933,r312949,r313141

(by cognet)

Use strexeq instead of needlessly branch.

==

(by cognet)

Remove useless labels.

==

(by cognet)

Correct the IT instruction in atomic_fcmpset_64().

==

(by andrew)

Only define atomic_fcmpset_long in the kernel. We may include
machine/atomic.h in userspace, however atomic_fcmpset_32 is unimplemented
there.

MFC r311169,r311898,r312925,r312973,r312975,r313007,r313040,r313080,
r313254,r313341

amd64: add atomic_fcmpset

==

sparc64: add atomic_fcmpset

==

Implement atomic_fcmpset_* for arm and arm64.

==

Add atomic_fcmpset_*() inlines for powerpc

Summary:
atomic_fcmpset_*() is analogous to atomic_cmpset(), but saves off the read value
from the target memory location into the 'old' pointer in the case of failure.

==

i386: add atomic_fcmpset

==

Don't retry a lost reservation in atomic_fcmpset()

The desired behavior of atomic_fcmpset_() is to always exit on error. Instead
of retrying on lost reservation, leave the retry to the caller, and return

==

Add atomic_fcmpset_*() inlines for MIPS

atomic_fcmpset_*() is analogous to atomic_cmpset(), but saves off the
read value from the target memory location into the 'old' pointer.

==

i386: fixup fcmpset

An incorrect output specifier was used which worked with clang by accident,
but breaks with the in-tree gcc version.

While here plug a whitespace nit.

==

Implement atomic_fcmpset_*() for RISC-V.

==

Use 64bit store instruction in atomic_fcmpset_64.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Implement atomic_cmpset_acq_64 and atomic_cmpset_rel_64 on arm and armeb.
This should allow r300113 to build there.

Sponsored by: ABT Systems Ltd

ARM: Implement atomic_swap_int(9). It's used in DRM2 code.

Approved by: kib (mentor)

ARM: Remove trailing whitespace from sys/arm/include
No functional changes.

Approved by: kib (mentor)

FreeBSD does not support SMP on ARMv5. Since processor is always
self-consistent, there is no need in anything but compiler barrier in
the implementation of atomic_thread_fence_*() on ARMv5. Split
implementation of fences for ARMv4/5 and ARMv6; the former use
compiler barriers, the later also perform hardware barriers.

An issue which is fixed by the change is the faults from the CP15
coprocessor accesses in the user mode. This was uncovered by the
pthread_once() changes in r287556.

Reported by: Mattia Rossi <mattia.rossi.mailinglists@gmail.com>
Discussed with: alc, cognet, jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

Sort the ARM atomic functions to be in alphabetical order.

Sponsored by: ABT Systems Ltd

Split out the arm and armv6 parts of atomic.h to new files. While here use
__ARM_ARCH to determine which revision of the architecture is applicable.

Sponsored by: ABT Systems Ltd

Add the atomic_thread_fence() family of functions with intent to
provide a semantic defined by the C11 fences with corresponding
memory_order.

atomic_thread_fence_acq() gives r | r, w, where r and w are read and
write accesses, and | denotes the fence itself.

atomic_thread_fence_rel() is r, w | w.

atomic_thread_fence_acq_rel() is the combination of the acquire and
release in single operation. Note that reads after the acq+rel fence
could be made visible before writes preceeding the fence.

atomic_thread_fence_seq_cst() orders all accesses before/after the
fence, and the fence itself is globally ordered against other
sequentially consistent atomic operations.

Reviewed by: alc
Discussed with: bde
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks

List both registers to use in the 64-bit atomic instructions. We will need
these to build for Thumb-2.

Revert r279338. The casts are apparently bogus, despite the fact that
they've been working in i386 (where this change came from).

Add casting to make atomic ops work for pointers. (Apparently nobody has
ever done atomic ops on pointers before now on arm).

Submitted by: Svatopluk Kraus <onwahe@gmail.com>

Correct a comment which was exactly backwards from reality.

Unify interrupts bit definition and usage. While here remove PSR_C_bit.

Submitted by: Svatopluk Kraus <onwahe at gmail.com>,
Michal Meloun <meloun at miracle.cz>
Differential Revision: https://reviews.freebsd.org/D754

Rename new to newval in inline asm code, to avoid clashes with C++ new.
Also rename cmp to cmpval just to keep the asm variable names similar to
the C variable names.

When arm 64-bit atomic ops are available, define ARM_HAVE_ATOMIC64. Use
that symbol (which will be correct in both kernel and userland contexts)
rather than just __arm__ to decide whether to use a local implementation.

Add 64-bit atomic ops for armv4, only for kernel code, mostly so that we
don't need any #ifdef stuff to use atomic_load/store_64() elsewhere in
the kernel. For armv4 the atomics are trivial to implement for kernel
code (just disable interrupts), less so for user mode, so this only has
the kernel mode implementations for now.

Add 64-bit atomic ops for armv6. The only safe way to access a 64-bit
value shared across multiple cores is with atomic_load_64() and
atomic_store_64(), because the normal 64-bit load/store instructions
are not atomic on 32-bit arm. Luckily the ldrexd/strexd instructions
that are atomic are fairly cheap on armv6. Because it's fairly simple
to do, this implements all the ops for 64-bit, not just load/store.

Reviewed by: andrew, cognet

Make the hardware memory and instruction barrier functions work on armv4
and armv5 as well.

Update all arm code that manipulates the PSR registers to use modern syntax.

It turns out the version of gas we're using interprets the old '_all' mask
as 'fc' instead of 'fsxc'. That is, "all" doesn't really mean "all".

This was the cause of the "wrong-endian register restore" bug that's
been causing problems with some cortex-a9 chips. The 'endian' bit in the
spsr register would never get changed (it falls into the 'x' mask group)
and the first return-from-exception would fail if the chip had powered on
with garbage in the spsr register that included the big-endian bit. It's
unknown why this affected only certain cortex-a9 chips.

Fix an itt instruction. We need to execute both the mov and b instructions
when building for Thumb.

Start adding support to build bits of our code using the Thumb-2
instruction set. Thumb-2 requires an if-then instruction to implement
conditional codes.

When building for ARM mode the it-then instructions do not generate any
assembled instruction as per the ARMv7-A Architecture Reference Manual, and
are safe to use.

While this allows the atomic instructions to be built, it doesn't mean we
fully support Thumb code. It works in small tests, but is still known to
fail in a large number of places.

While here add a check for the armv6t2 architecture.

Don't define rel/acq variants of some atomic operations as the regular
version for armv6.

Implement barriers for AMRv6 and ARMv7

Submitted by: Daisuke Aoyama <aoyama at peach.ne.jp>
Reviewed by: ian, cognet

Fix the clobber list on the atomic operators that do comparisons. Without
this some compilers will place a cmp instruction before the atomic operation
and expect to be able to use the result afterwards. By adding "cc" to the
list of used registers we tell the compiler to not do this.

Merging projects/armv6, part 1

Cummulative patch of changes that are not vendor-specific:
- ARMv6 and ARMv7 architecture support
- ARM SMP support
- VFP/Neon support
- ARM Generic Interrupt Controller driver
- Simplification of startup code for all platforms

Revert committal of local change accidentally swept up in r238329.

Remove some unused variables/externs that have been copied too many times...

trim trailing whitespace

Fix the userland, RAS, version of atomic_fetchadd_32 :
return the correct value, and do not store the wrong one in the supplied
pointer.

Submitted by: Mark Tinguely <tinguely casselton net>

force atomic_cmpset_ptr types to match atomic_cmpset_32;
this matches what powerpc does

Submitted by: stass
MFC after: 2 weeks

- bump __FreeBSD version to reflect added buf_ring, memory barriers,
and ifnet functions

- add memory barriers to <machine/atomic.h>
- update drivers to only conditionally define their own

- add lockless producer / consumer ring buffer
- remove ring buffer implementation from cxgb and update its callers

- add if_transmit(struct ifnet *ifp, struct mbuf *m) to ifnet to
allow drivers to efficiently manage multiple hardware queues
(i.e. not serialize all packets through one ifq)
- expose if_qflush to allow drivers to flush any driver managed queues

This work was supported by Bitgravity Inc. and Chelsio Inc.

Improve ARM_TP_ADDRESS and RAS area.

De-hardcode usage of ARM_TP_ADDRESS and RAS local storage, and move this
special purpose page to a more convenient place i.e. after the vectors high
page, more towards the end of address space. Previous location (0xe000_0000)
caused grief if KVA was to go beyond the default limit.

Note that ARM world rebuilding is required after this change since the
location of ARM_TP_ADDRESS is shared between kernel and userland.

Submitted by: Grzegorz Bernacki (gjb AT semihalf dot com)
Reviewed by: imp
Approved by: cognet (mentor)

Close a race.

The RAS implementation would set the end address, then the start
address. These were used by the kernel to restart a RAS sequence if
it was interrupted. When the thread switching code ran, it would
check these values and adjust the PC and clear them if it did.

However, there's a small flaw in this scheme. Thread T1, sets the end
address and gets preempted. Thread T2 runs and also does a RAS
operation. This resets end to zero. Thread T1 now runs again and
sets start and then begins the RAS sequence, but is preempted before
the RAS sequence executes its last instruction. The kernel code that
would ordinarily restart the RAS sequence doesn't because the PC isn't
between start and 0, so the PC isn't set to the start of the sequence.
So when T1 is resumed again, it is at the wrong location for RAS to
produce the correct results. This causes the wrong results for the
atomic sequence.

The window for the first race is 3 instructions. The window for the
second race is 5-10 instructions depending on the atomic operation.
This makes this failure fairly rare and hard to reproduce.

Mutexs are implemented in libthr using atomic operations. When the
above race would occur, a lock could get stuck locked, causing many
downstream problems, as you might expect.

Also, make sure to reset the start and end address when doing a syscall, or
a malicious process could set them before doing a syscall.

Reviewed by: imp, ups (thanks guys)
Pointy hat to: cognet
MFC After: 3 days

In atomic_fetchadd_32(), do not blindly increase the value of %3.
It should just contain the value we want to add, as if we're interrupted
between the add and the str, we will restart from the beginning. Just use
a register we can scratch instead.

MFC After: 1 week

MFp4: Add missing atomic functions
Based on a patch by: des

Add atomic_cmpset_acq_32.

Add definitions for atomic_subtract_rel_32, atomic_add_rel_32 and
atomic_load_acq_32, needed for hwpmc.

Disable/enable fiqs as well as irqs.

Use memory clobbers, to be on the safe side.
Suggested by: jhb

Backout rev 1.12. It would have been a good thing, if gcc was smart enough
not to generate bad code.

A #define is not enough, we need to cast from u_long * to uint32_t *.

Define atomic_whatever_long

Whitespace.

Change the userland atomic operations on arm to use memory operands for
the modified memory rather than using register operands that held a pointer
to the memory. The biggest effect is that we now correctly tell the
compiler that these functions change the memory that these functions
modify.

Reviewed by: cognet

Add a new atomic_fetchadd() primitive that atomically adds a value to a
variable and returns the previous value of the variable.

Tested on: i386, alpha, sparc64, arm (cognet)
Reviewed by: arch@
Submitted by: cognet (arm)
MFC after: 1 week

Add extra constraints to tell the compiler that the memory be modified
in the arm __swp() and sparc64 casa() and casax() functions is actually
being used as an input and output and not just the value of the register
that points to the memory location. This was the underlying source of
the mbuf refcount problems on sparc64 a while back. For arm this should be
a nop because __swp() has a constraint to clobber all memory which can
probably be removed now.

Reviewed by: alc, cognet
MFC after: 1 week

Convert the atomic_ptr() operations over to operating on uintptr_t
variables rather than void * variables. This makes it easier and simpler
to get asm constraints and volatile keywords correct.

MFC after: 3 days
Tested on: i386, alpha, sparc64
Compiled on: ia64, powerpc, amd64
Kernel toolchain busted on: arm

Fix a typo.

Approved by: re (scottl)

Make sure we clean the RAS start address once we're done.
This fixes the random segfaults which occurs at high interrupts rate.

Import a basic implementation of the restartable atomic sequences to provide
atomic operations to userland (this is OK for UP only, but SMP is still so
far away).

Start all license statements with /*-

Make sure gcc doesn't generate something such as swp r3, r4, [r3] for __swp,
as it has unpredictable results.

Disable interrupts for atomic_cmpset_32, this one is just not atomic.
Don't export it to userland.

Try to implement atomic operations using swp, instead of disabling interrupts.

Import FreeBSD/arm kernel bits.
It only supports sa1110 (on simics) right now, but xscale support should come
soon.
Some of the initial work has been provided by :
Stephane Potvin <sepotvin at videotron.ca>
Most of this comes from NetBSD.