#
95ee2897 |
|
16-Aug-2023 |
Warner Losh <imp@FreeBSD.org> |
sys: Remove $FreeBSD$: two-line .h pattern Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/
|
#
d5d97bed |
|
25-Jul-2023 |
Mike Karels <karels@FreeBSD.org> |
arm64 lib32: prepare arm64 headers to redirect to arm In order to compile lib32 libraries and other 32-bit code on arm64, <machine/foo.h> needs to be redirected to an arm header rather than arm64 when building with -m32. Ifdef the arm64 headers that are installed in /usr/include/machine and used by user-level software (including references from /usr/include/*.h) so that if __arm__ is defined when including the arm64 version, <arm/foo.h> is included rather than using the rest of the file's contents. Some arm headers had no arm64 equivalent; headers were added just to do the redirection. These files use #error if __arm__ is not defined to guard against confusion. Also add an include/arm Makefile, and modify Makefiles as needed to install everything, including the arm files in /usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h. The new arm64 headers are: acle-compat.h cpuinfo.h sysreg.h Reviewed by: jrtc27, imp Differential Revision: https://reviews.freebsd.org/D40944
|
#
89c52f9d |
|
23-Mar-2023 |
Kyle Evans <kevans@FreeBSD.org> |
arm64: add KASAN support This entails: - Marking some obvious candidates for __nosanitizeaddress - Similar trap frame markings as amd64, for similar reasons - Shadow map implementation The shadow map implementation is roughly similar to what was done on amd64, with some exceptions. Attempting to use available space at preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range, as depicted in the physmap) results in odd failures, so we instead search the physmap for free regions that we can carve out, fragmenting the shadow map as necessary to try and fit as much as we need for the initial kernel map. pmap_bootstrap_san() is thus after pmap_bootstrap(), which still included some technically reserved areas of the memory map that needed to be included in the DMAP. The odd failure noted above may be a bug, but I haven't investigated it all that much. Initial work by mhorne with additional fixes from kevans and markj. Reviewed by: andrew, markj Sponsored by: Juniper Networks, Inc. Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D36701
|
#
d8da59dd |
|
12-Jan-2022 |
Andrew Turner <andrew@FreeBSD.org> |
Fix the sanitizer header checks on arm64 We need to not include the MI _san.h files when builing some parts of the kernel. Fix these checks in the arm64 header files. Sponsored by: The FreeBSD Foundation
|
#
ec1ecf78a |
|
12-Jan-2022 |
Andrew Turner <andrew@FreeBSD.org> |
Add the char and short atomic operations on arm64 Sponsored by: The FreeBSD Foundation
|
#
02c16e21 |
|
20-Dec-2021 |
Andrew Turner <andrew@FreeBSD.org> |
Add atomic_testandset_acq_* on arm64 We only need to include sys/_atomic_subword.h on arm64 to provide atomic_testandset_acq_long. Add an implementation in the arm64 atomic.h based on the existing atomic_testandset macro. Reviewed by: kib Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D33587
|
#
435c7cfb |
|
07-Mar-2021 |
Mark Johnston <markj@FreeBSD.org> |
Rename _cscan_atomic.h and _cscan_bus.h to atomic_san.h and bus_san.h Other kernel sanitizers (KMSAN, KASAN) require interceptors as well, so put these in a more generic place as a step towards importing the other sanitizers. No functional change intended. MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D29103
|
#
82661227 |
|
02-Jan-2021 |
Ryan Libby <rlibby@FreeBSD.org> |
arm64: fix mask in atomic_testand{set,clear}_64 These macros generate both the 32- and 64-bit ops, but the mask was hard coded for 32-bit ops, causing the 64-bit ops always to affect only the low 32 bits. PR: 252324 Reported by: gbe, mmel Reviewed by: markj, mmel Tested by: mmel, rwatson Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D27886
|
#
ca0ec73c |
|
25-Mar-2020 |
Conrad Meyer <cem@FreeBSD.org> |
Expand generic subword atomic primitives The goal of this change is to make the atomic_load_acq_{8,16}, atomic_testandset{,_acq}_long, and atomic_testandclear_long primitives available in MI-namespace. The second goal is to get this draft out of my local tree, as anything that requires a full tinderbox is a big burden out of tree. MD specifics can be refined individually afterwards. The generic implementations may not be ideal for your architecture; feel free to implement better versions. If no subword_atomic definitions are needed, the include can be removed from your arch's machine/atomic.h. Generic definitions are guarded by defined macros of the same name. To avoid picking up conflicting generic definitions, some macro defines are added to various MD machine/atomic.h to register an existing implementation. Include _atomic_subword.h in arm and arm64 machine/atomic.h. For some odd reason, KCSAN only generates some versions of primitives. Generate the _acq variants of atomic_load.*_8, atomic_load.*_16, and atomic_testandset.*_long. There are other questionably disabled primitives, but I didn't run into them, so I left them alone. KCSAN is only built for amd64 in tinderbox for now. Add atomic_subword implementations of atomic_load_acq_{8,16} implemented using masking and atomic_load_acq_32. Add generic atomic_subword implementations of atomic_testandset_long(), atomic_testandclear_long(), and atomic_testandset_acq_long(), using atomic_fcmpset_long() and atomic_fcmpset_acq_long(). On x86, add atomic_testandset_acq_long as an alias for atomic_testandset_long. Reviewed by: kevans, rlibby (previous versions both) Differential Revision: https://reviews.freebsd.org/D22963
|
#
a83c682b |
|
03-Feb-2020 |
Mark Johnston <markj@FreeBSD.org> |
Dynamically select LSE-based atomic(9)s on arm64. Once all CPUs are online, determine if they all support LSE atomics and set lse_supported to indicate this. For now the atomic(9) implementations are still always inlined, though it would be preferable to create out-of-line functions to avoid text bloat. This was not done here since big.little systems exist in which some CPUs implement LSE while others do not, and ifunc resolution must occur well before this scenario can be detected. It does seem unlikely that FreeBSD will ever run on such platforms, however, so converting atomic(9) to use ifuncs is probably a good next step. Add a LSE_ATOMICS arm64 kernel configuration option to unconditionally select LSE-based atomic(9) implementations when the target system is known. Reviewed by: andrew, kib MFC after: 1 month Sponsored by: The FreeBSD Foundation, Amazon (hardware) Differential Revision: https://reviews.freebsd.org/D23325
|
#
920de6a1 |
|
03-Feb-2020 |
Mark Johnston <markj@FreeBSD.org> |
Add LSE-based atomic(9) implementations. These make use of the cas*, ld* and swp instructions added in ARMv8.1. Testing shows them to be significantly more performant than LL/SC-based implementations. No functional change here since the wrappers still unconditionally select the _llsc variants. Reviewed by: andrew, kib MFC after: 1 month Submitted by: Ali Saidi <alisaidi@amazon.com> (original version) Differential Revision: https://reviews.freebsd.org/D23324
|
#
c1fced68 |
|
03-Feb-2020 |
Mark Johnston <markj@FreeBSD.org> |
Add wrappers for arm64 atomics. Add a _llsc suffix for the existing LL/SC-based implementations and add trivial wrappers. This is in preparation for supporting LSE-based atomic(9) implementations. No functional change intended. Reviewed by: andrew, kib MFC after: 1 month Sponsored by: The FreeBSD Foundation, Amazon (hardware) Differential Revision: https://reviews.freebsd.org/D23323
|
#
3ad6c736 |
|
03-Feb-2020 |
Mark Johnston <markj@FreeBSD.org> |
Provide a single implementation for each of the arm64 atomic(9) ops. Parameterize the macros by type width as well as acq/rel semantics. This makes modifying the implementations much less tedious and error-prone and makes it easier to support alternate LSE-based implementations. No functional change intended. Reviewed by: andrew, kib MFC after: 1 month Sponsored by: The FreeBSD Foundation, Amazon (hardware) Differential Revision: https://reviews.freebsd.org/D23322
|
#
f2792e5e |
|
09-Jan-2020 |
Andrew Turner <andrew@FreeBSD.org> |
Add atomic_testandset/clear on arm64. These will reportedly be used in future uma changes. MFC after: 2 weeks Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D23019
|
#
6d26116b |
|
03-Jan-2020 |
Andrew Turner <andrew@FreeBSD.org> |
Add the 8 and 16 bit atomic load/store functions with a barrier on arm64. Reviewed by: cem MFC after: 2 weeks Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D22966
|
#
849aef49 |
|
21-Nov-2019 |
Andrew Turner <andrew@FreeBSD.org> |
Port the NetBSD KCSAN runtime to FreeBSD. Update the NetBSD Kernel Concurrency Sanitizer (KCSAN) runtime to work in the FreeBSD kernel. It is a useful tool for finding data races between threads executing on different CPUs. This can be enabled by enabling KCSAN in the kernel config, or by using the GENERIC-KCSAN amd64 kernel. It works on amd64 and arm64, however the later needs a compiler change to allow -fsanitize=thread that KCSAN uses. Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D22315
|
#
4ffa494e |
|
07-Nov-2019 |
Andrew Turner <andrew@FreeBSD.org> |
Add more 8 and 16 bit variants of the the atomic(9) functions on arm64. These are direct copies of the 32 bit functions, adjusted ad needed. While here fix atomic_fcmpset_16 to use the valid load and store exclusive instructions. Sponsored by: DARPA, AFRL
|
#
44654b75 |
|
28-May-2018 |
Emmanuel Vadot <manu@FreeBSD.org> |
arm64: fix atomic_fcmpset_16 newval needs to be uint16_t Reported by: andrew
|
#
e39ce4ca |
|
28-May-2018 |
Emmanuel Vadot <manu@FreeBSD.org> |
arm64: Add atomic_fcmpset_8 and atomic_fcmpset_16 Reviewed by: cognet
|
#
30d4f9e8 |
|
19-Dec-2017 |
Konstantin Belousov <kib@FreeBSD.org> |
Add atomic_load(9) and atomic_store(9) operations. They provide relaxed-ordered atomic access semantic. Due to the FreeBSD memory model, the operations are syntaxical wrappers around the volatile accesses. The volatile qualifier is used to ensure that the access not optimized out and in turn depends on the volatile semantic as implemented by supported compilers. The motivation for adding the operation is to help people coming from other systems or knowing the C11/C++ standards where atomics have special type and require use of the special access operations. It is still the case that FreeBSD requires plain load and stores of aligned integer types to be atomic. Suggested by: jhb Reviewed by: alc, jhb Sponsored by: The FreeBSD Foundation MFC after: 1 week Differential revision: https://reviews.freebsd.org/D13534
|
#
2da3e34e |
|
30-Jun-2017 |
Andrew Turner <andrew@FreeBSD.org> |
Remove a blank line accidentally added in r320403.
|
#
c90baf68 |
|
27-Jun-2017 |
Andrew Turner <andrew@FreeBSD.org> |
Some of the atomic_clear_* functions were incorrectly defined to be an atomic add. Correct these, fixing a NULL-pointer dereference in netgraph. PR: 220273 MFC after: 3 days Sponsored by: DARPA, AFRL
|
#
dc5f9fcd |
|
28-Jan-2017 |
Olivier Houchard <cognet@FreeBSD.org> |
Implement atomic_fcmpset_* for arm and arm64.
|
#
119a353e |
|
30-Nov-2015 |
Andrew Turner <andrew@FreeBSD.org> |
Rework the atomic code to reduce the repetition. This merges some of the atomic functions where they are almost identical, or have acquire/release semantics. While here clean these function up. The cbnz instruction doesn't change the condition flags so drop cc, however they should have memory added to the clobber list. Reviewed by: kib Sponsored by: ABT Systems Ltd Differential Revision: https://reviews.freebsd.org/D4318
|
#
9b8c3c4f |
|
31-Jul-2015 |
Andrew Turner <andrew@FreeBSD.org> |
Add more atomic_swap_* functions. Obtained from: ABT Systems Ltd Sponsored by: The FreeBSD Foundation
|
#
8954a9a4 |
|
08-Jul-2015 |
Konstantin Belousov <kib@FreeBSD.org> |
Add the atomic_thread_fence() family of functions with intent to provide a semantic defined by the C11 fences with corresponding memory_order. atomic_thread_fence_acq() gives r | r, w, where r and w are read and write accesses, and | denotes the fence itself. atomic_thread_fence_rel() is r, w | w. atomic_thread_fence_acq_rel() is the combination of the acquire and release in single operation. Note that reads after the acq+rel fence could be made visible before writes preceeding the fence. atomic_thread_fence_seq_cst() orders all accesses before/after the fence, and the fence itself is globally ordered against other sequentially consistent atomic operations. Reviewed by: alc Discussed with: bde Sponsored by: The FreeBSD Foundation MFC after: 3 weeks
|
#
458f2175 |
|
09-Jun-2015 |
Zbigniew Bodek <zbb@FreeBSD.org> |
Add options to dmb() and dsb() macros on ARM64 Using plain dsb()/dmb() as full system barriers is usually to much. Adding proper options to those barriers (instead of full system - sy) will most likely reduce the cost of the instructions and will benefit in performance improvement. This commit adds options to barrier macro definitions. Obtained from: Semihalf Reviewed by: andrew, ian Sponsored by: The FreeBSD Foundation
|
#
46f52b02 |
|
06-Apr-2015 |
Andrew Turner <andrew@FreeBSD.org> |
Split out the _acq and _rel functions. These were the same, but there is no need for them to be this strong, we only need to provide one or the other. While here replace atomic_load_acq_* and atomic_store_rel_* with a single instruction version, and fix the definition of atomic_clear_* to point to the correct functions. Sponsored by: The FreeBSD Foundation
|
#
412042e2 |
|
23-Mar-2015 |
Andrew Turner <andrew@FreeBSD.org> |
Add the start of the arm64 machine headers. This is the subset needed to start getting userland libraries building. Reviewed by: imp Sponsored by: The FreeBSD Foundation
|