arm: add missing atomic-pointer functions

The pointer function types were missing for these functions, so add
them. Valuable for places where type sizes vary by architecture.

Differential Revision: https://reviews.freebsd.org/D37778
Reviewed by: imp, andrew
Pull Request: https://github.com/freebsd/freebsd-src/pull/1126

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

arm: Move contents of atomic-v6.h into atomic.h

Previously we had an armv4/v5 and armv6/v7 implementation of the atomic
operations. As we have removed armv4/v5 support we can now merge the
armv6/v7 code into atomic.h

Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41135

arm: Explain why _atomic_subword.h is needed

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41134

Remove left over includes from armv4/v5

These includes were added for armv4 and armv5 support. Remove them as
this has been removed from the tree.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41133

Remove now unused armv4 and not-INTRNG files.

Expand generic subword atomic primitives

The goal of this change is to make the atomic_load_acq_{8,16},
atomic_testandset{,_acq}_long, and atomic_testandclear_long primitives
available in MI-namespace.

The second goal is to get this draft out of my local tree, as anything that
requires a full tinderbox is a big burden out of tree. MD specifics can be
refined individually afterwards.

The generic implementations may not be ideal for your architecture; feel
free to implement better versions. If no subword_atomic definitions are
needed, the include can be removed from your arch's machine/atomic.h.
Generic definitions are guarded by defined macros of the same name. To
avoid picking up conflicting generic definitions, some macro defines are
added to various MD machine/atomic.h to register an existing implementation.

Include _atomic_subword.h in arm and arm64 machine/atomic.h.

For some odd reason, KCSAN only generates some versions of primitives.
Generate the _acq variants of atomic_load.*_8, atomic_load.*_16, and
atomic_testandset.*_long. There are other questionably disabled primitives,
but I didn't run into them, so I left them alone. KCSAN is only built for
amd64 in tinderbox for now.

Add atomic_subword implementations of atomic_load_acq_{8,16} implemented
using masking and atomic_load_acq_32.

Add generic atomic_subword implementations of atomic_testandset_long(),
atomic_testandclear_long(), and atomic_testandset_acq_long(), using
atomic_fcmpset_long() and atomic_fcmpset_acq_long().

On x86, add atomic_testandset_acq_long as an alias for
atomic_testandset_long.

Reviewed by: kevans, rlibby (previous versions both)
Differential Revision: https://reviews.freebsd.org/D22963

Implement atomic_swap_xxx() for all platforms.

Differential Revision: https://reviews.freebsd.org/D18450
Reviewed by: kib@
MFC after: 3 days
Sponsored by: Mellanox Technologies

Remove arm-specific implementations of atomic_load/store_xxx() now that
they are provided by sys/atomic_common.h.

Add atomic_load(9) and atomic_store(9) operations.

They provide relaxed-ordered atomic access semantic. Due to the
FreeBSD memory model, the operations are syntaxical wrappers around
the volatile accesses. The volatile qualifier is used to ensure that
the access not optimized out and in turn depends on the volatile
semantic as implemented by supported compilers.

The motivation for adding the operation is to help people coming from
other systems or knowing the C11/C++ standards where atomics have
special type and require use of the special access operations. It is
still the case that FreeBSD requires plain load and stores of aligned
integer types to be atomic.

Suggested by: jhb
Reviewed by: alc, jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D13534

sys/arm: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Stop including sys/types.h from arm's machine/atomic.h, fix the places
where atomic.h was being included without ensuring that types.h (via
param.h) was included first, as required by atomic(9).

Implement atomic_fcmpset_* for arm and arm64.

Remove arm's cpuconf.h, and references to it, after moving a few lines from
it into pmap-v4.h where they are used. Other than those few lines of
support for different MMU types, nothing in cpuconf.h has been used in our
code for quite a while.

The file existed to set up a variety of symbols to describe the
architecture. Over the past few years we have converted all of our source
to use the new architecture symbols standardized by ARM Inc, and predefined
by both clang and gcc.

PR: 216104

Add some missing atomic_*_ptr #defines for arm.

MFC after: 1 week

Include machine/acle-compat.h in cdefs.h on arm if the compiler doesn't
have ACLE support built in. The ACLE (ARM C Language Extensions) defines
a set of standardized symbols which indicate the architecture version and
features available. ACLE support is built in to modern compilers (both
clang and gcc), but absent from gcc prior to 4.4.

ARM (the company) provides the acle-compat.h header file to define the
right symbols for older versions of gcc. Basically, acle-compat.h does
for arm about the same thing cdefs.h does for freebsd: defines
standardized macros that work no matter which compiler you use. If ARM
hadn't provided this file we would have ended up with a big #ifdef __arm__
section in cdefs.h with our own compatibility shims.

Remove #include <machine/acle-compat.h> from the zillion other places (an
ever-growing list) that it appears. Since style(9) requires sys/types.h
or sys/param.h early in the include list, and both of those lead to
including cdefs.h, only a couple special cases still need to include
acle-compat.h directly.

Loves it: imp

ARM: Implement atomic_swap_int(9). It's used in DRM2 code.

Approved by: kib (mentor)

FreeBSD does not support SMP on ARMv5. Since processor is always
self-consistent, there is no need in anything but compiler barrier in
the implementation of atomic_thread_fence_*() on ARMv5. Split
implementation of fences for ARMv4/5 and ARMv6; the former use
compiler barriers, the later also perform hardware barriers.

An issue which is fixed by the change is the faults from the CP15
coprocessor accesses in the user mode. This was uncovered by the
pthread_once() changes in r287556.

Reported by: Mattia Rossi <mattia.rossi.mailinglists@gmail.com>
Discussed with: alc, cognet, jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week

Split out the arm and armv6 parts of atomic.h to new files. While here use
__ARM_ARCH to determine which revision of the architecture is applicable.

Sponsored by: ABT Systems Ltd

Add the atomic_thread_fence() family of functions with intent to
provide a semantic defined by the C11 fences with corresponding
memory_order.

atomic_thread_fence_acq() gives r | r, w, where r and w are read and
write accesses, and | denotes the fence itself.

atomic_thread_fence_rel() is r, w | w.

atomic_thread_fence_acq_rel() is the combination of the acquire and
release in single operation. Note that reads after the acq+rel fence
could be made visible before writes preceeding the fence.

atomic_thread_fence_seq_cst() orders all accesses before/after the
fence, and the fence itself is globally ordered against other
sequentially consistent atomic operations.

Reviewed by: alc
Discussed with: bde
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks

List both registers to use in the 64-bit atomic instructions. We will need
these to build for Thumb-2.

Revert r279338. The casts are apparently bogus, despite the fact that
they've been working in i386 (where this change came from).

Add casting to make atomic ops work for pointers. (Apparently nobody has
ever done atomic ops on pointers before now on arm).

Submitted by: Svatopluk Kraus <onwahe@gmail.com>

Correct a comment which was exactly backwards from reality.

Unify interrupts bit definition and usage. While here remove PSR_C_bit.

Submitted by: Svatopluk Kraus <onwahe at gmail.com>,
Michal Meloun <meloun at miracle.cz>
Differential Revision: https://reviews.freebsd.org/D754

Rename new to newval in inline asm code, to avoid clashes with C++ new.
Also rename cmp to cmpval just to keep the asm variable names similar to
the C variable names.

When arm 64-bit atomic ops are available, define ARM_HAVE_ATOMIC64. Use
that symbol (which will be correct in both kernel and userland contexts)
rather than just __arm__ to decide whether to use a local implementation.

Add 64-bit atomic ops for armv4, only for kernel code, mostly so that we
don't need any #ifdef stuff to use atomic_load/store_64() elsewhere in
the kernel. For armv4 the atomics are trivial to implement for kernel
code (just disable interrupts), less so for user mode, so this only has
the kernel mode implementations for now.

Add 64-bit atomic ops for armv6. The only safe way to access a 64-bit
value shared across multiple cores is with atomic_load_64() and
atomic_store_64(), because the normal 64-bit load/store instructions
are not atomic on 32-bit arm. Luckily the ldrexd/strexd instructions
that are atomic are fairly cheap on armv6. Because it's fairly simple
to do, this implements all the ops for 64-bit, not just load/store.

Reviewed by: andrew, cognet

Make the hardware memory and instruction barrier functions work on armv4
and armv5 as well.

Update all arm code that manipulates the PSR registers to use modern syntax.

It turns out the version of gas we're using interprets the old '_all' mask
as 'fc' instead of 'fsxc'. That is, "all" doesn't really mean "all".

This was the cause of the "wrong-endian register restore" bug that's
been causing problems with some cortex-a9 chips. The 'endian' bit in the
spsr register would never get changed (it falls into the 'x' mask group)
and the first return-from-exception would fail if the chip had powered on
with garbage in the spsr register that included the big-endian bit. It's
unknown why this affected only certain cortex-a9 chips.

Fix an itt instruction. We need to execute both the mov and b instructions
when building for Thumb.

Start adding support to build bits of our code using the Thumb-2
instruction set. Thumb-2 requires an if-then instruction to implement
conditional codes.

When building for ARM mode the it-then instructions do not generate any
assembled instruction as per the ARMv7-A Architecture Reference Manual, and
are safe to use.

While this allows the atomic instructions to be built, it doesn't mean we
fully support Thumb code. It works in small tests, but is still known to
fail in a large number of places.

While here add a check for the armv6t2 architecture.

Don't define rel/acq variants of some atomic operations as the regular
version for armv6.

Implement barriers for AMRv6 and ARMv7

Submitted by: Daisuke Aoyama <aoyama at peach.ne.jp>
Reviewed by: ian, cognet

Fix the clobber list on the atomic operators that do comparisons. Without
this some compilers will place a cmp instruction before the atomic operation
and expect to be able to use the result afterwards. By adding "cc" to the
list of used registers we tell the compiler to not do this.

Merging projects/armv6, part 1

Cummulative patch of changes that are not vendor-specific:
- ARMv6 and ARMv7 architecture support
- ARM SMP support
- VFP/Neon support
- ARM Generic Interrupt Controller driver
- Simplification of startup code for all platforms

Revert committal of local change accidentally swept up in r238329.

Remove some unused variables/externs that have been copied too many times...

trim trailing whitespace

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Fix the userland, RAS, version of atomic_fetchadd_32 :
return the correct value, and do not store the wrong one in the supplied
pointer.

Submitted by: Mark Tinguely <tinguely casselton net>

force atomic_cmpset_ptr types to match atomic_cmpset_32;
this matches what powerpc does

Submitted by: stass
MFC after: 2 weeks

- bump __FreeBSD version to reflect added buf_ring, memory barriers,
and ifnet functions

- add memory barriers to <machine/atomic.h>
- update drivers to only conditionally define their own

- add lockless producer / consumer ring buffer
- remove ring buffer implementation from cxgb and update its callers

- add if_transmit(struct ifnet *ifp, struct mbuf *m) to ifnet to
allow drivers to efficiently manage multiple hardware queues
(i.e. not serialize all packets through one ifq)
- expose if_qflush to allow drivers to flush any driver managed queues

This work was supported by Bitgravity Inc. and Chelsio Inc.

- Import the HEAD csup code which is the basis for the cvsmode work.

Improve ARM_TP_ADDRESS and RAS area.

De-hardcode usage of ARM_TP_ADDRESS and RAS local storage, and move this
special purpose page to a more convenient place i.e. after the vectors high
page, more towards the end of address space. Previous location (0xe000_0000)
caused grief if KVA was to go beyond the default limit.

Note that ARM world rebuilding is required after this change since the
location of ARM_TP_ADDRESS is shared between kernel and userland.

Submitted by: Grzegorz Bernacki (gjb AT semihalf dot com)
Reviewed by: imp
Approved by: cognet (mentor)

Close a race.

The RAS implementation would set the end address, then the start
address. These were used by the kernel to restart a RAS sequence if
it was interrupted. When the thread switching code ran, it would
check these values and adjust the PC and clear them if it did.

However, there's a small flaw in this scheme. Thread T1, sets the end
address and gets preempted. Thread T2 runs and also does a RAS
operation. This resets end to zero. Thread T1 now runs again and
sets start and then begins the RAS sequence, but is preempted before
the RAS sequence executes its last instruction. The kernel code that
would ordinarily restart the RAS sequence doesn't because the PC isn't
between start and 0, so the PC isn't set to the start of the sequence.
So when T1 is resumed again, it is at the wrong location for RAS to
produce the correct results. This causes the wrong results for the
atomic sequence.

The window for the first race is 3 instructions. The window for the
second race is 5-10 instructions depending on the atomic operation.
This makes this failure fairly rare and hard to reproduce.

Mutexs are implemented in libthr using atomic operations. When the
above race would occur, a lock could get stuck locked, causing many
downstream problems, as you might expect.

Also, make sure to reset the start and end address when doing a syscall, or
a malicious process could set them before doing a syscall.

Reviewed by: imp, ups (thanks guys)
Pointy hat to: cognet
MFC After: 3 days

In atomic_fetchadd_32(), do not blindly increase the value of %3.
It should just contain the value we want to add, as if we're interrupted
between the add and the str, we will restart from the beginning. Just use
a register we can scratch instead.

MFC After: 1 week

MFp4: Add missing atomic functions
Based on a patch by: des

Add atomic_cmpset_acq_32.

Add definitions for atomic_subtract_rel_32, atomic_add_rel_32 and
atomic_load_acq_32, needed for hwpmc.

Disable/enable fiqs as well as irqs.

Use memory clobbers, to be on the safe side.
Suggested by: jhb

Backout rev 1.12. It would have been a good thing, if gcc was smart enough
not to generate bad code.

A #define is not enough, we need to cast from u_long * to uint32_t *.

Define atomic_whatever_long

Whitespace.

Change the userland atomic operations on arm to use memory operands for
the modified memory rather than using register operands that held a pointer
to the memory. The biggest effect is that we now correctly tell the
compiler that these functions change the memory that these functions
modify.

Reviewed by: cognet

Add a new atomic_fetchadd() primitive that atomically adds a value to a
variable and returns the previous value of the variable.

Tested on: i386, alpha, sparc64, arm (cognet)
Reviewed by: arch@
Submitted by: cognet (arm)
MFC after: 1 week

Add extra constraints to tell the compiler that the memory be modified
in the arm __swp() and sparc64 casa() and casax() functions is actually
being used as an input and output and not just the value of the register
that points to the memory location. This was the underlying source of
the mbuf refcount problems on sparc64 a while back. For arm this should be
a nop because __swp() has a constraint to clobber all memory which can
probably be removed now.

Reviewed by: alc, cognet
MFC after: 1 week

Convert the atomic_ptr() operations over to operating on uintptr_t
variables rather than void * variables. This makes it easier and simpler
to get asm constraints and volatile keywords correct.

MFC after: 3 days
Tested on: i386, alpha, sparc64
Compiled on: ia64, powerpc, amd64
Kernel toolchain busted on: arm

Fix a typo.

Approved by: re (scottl)

Make sure we clean the RAS start address once we're done.
This fixes the random segfaults which occurs at high interrupts rate.

Import a basic implementation of the restartable atomic sequences to provide
atomic operations to userland (this is OK for UP only, but SMP is still so
far away).

Start all license statements with /*-

Make sure gcc doesn't generate something such as swp r3, r4, [r3] for __swp,
as it has unpredictable results.

Disable interrupts for atomic_cmpset_32, this one is just not atomic.
Don't export it to userland.

Try to implement atomic operations using swp, instead of disabling interrupts.

Import FreeBSD/arm kernel bits.
It only supports sa1110 (on simics) right now, but xscale support should come
soon.
Some of the initial work has been provided by :
Stephane Potvin <sepotvin at videotron.ca>
Most of this comes from NetBSD.