arm64: Fix indentation to be consistent

Adjust the mair_el1 macro indentation to be consistent with the
surrounding macros.

Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45524

arm64: Add the pointer auth registers to armreg.h

Add the pointer authentication registers to armreg.h. These will be
used to support pointer authentication in a kernel built with GCC.

Reviewed by: jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45262

arm64: Use the UL macro in TCR_EL1 defines

While clang can handle numbers with a UL suffix in assembly files
gcc/gas is unable to. Switch to use the UL macro for TCR_EL1 defines as
some are used in locore.S

Reviewed by: brooks, jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45261

arm64: add CONTEXTIDR_EL1 reg

CONTEXTIDR_EL1 is used in debug and trace features to identify the
current process or context.

Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45173

arm64: add PMBSR_MSS_{BSC,FSC} status code field

Bits [5:0] of PMBSR_MSS encodes either Buffer Status Code (BSC) or Fault
Status Code (FSC) depending on PMBSR_EC value.

Add PMBSR_MSS_{BSC,FSC} to cover this field.

Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45172

arm64: make SPE regs use ALT_NAME macro

When the register is not defined in Armv8.0 i.e. added in a later
extension, like SPE added in v8.2, the alternative name format of:
S<op0>_<op1>_C<crn>_C<crm>_<op2>
should be used; otherwise, calls to {READ,WRITE}_SPECIALREG() will
fail.

Use the MRS_REG_ALT_NAME() macro for SPE changing hex to decimal as
required by the macro.

Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45171

arm64: Add MRS_REG_ALT_NAME ID register macros

These can be used even when the compiler is too old for the register
to be included.

Reviewed by: Zachary Leaf <zachary.leaf@arm.com>
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45176

arm64: Add EL1 hardware breakpoint exceptions

Reviewed by: jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44353

arm64: Add ISS_MSR_REG for ESR_ELx.ISS values

Add a macro to get the ESR_ELx ISS value when we trap accessing a
special register.

arm64: Add more spsr_el1 register values

Sponsored by: Arm Ltd

arm64: Add CurrentEL register definitions

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43971

armv8rng: Don't require toolchain to support FEAT_RNG

We have the mechanism in place to support encoding system registers
explicitly, so use that rather than requiring LLVM 13+, which breaks our
current set of GitHub CI builds.

Fixes: 9eecef052155 ("Add an Armv8 rndr random number provider")

Add BTI exceptions

We could hit these when executing code marked as using BTI but jumps
to a non-branch target instruction.

Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39450

arm64: Add TCR register masks

These will be used by bhyve to implement page table walking.

Sponsored by: Arm Ltd

arm64: Fix the TCR_TBI0 macro to use UL

This needs to be 1UL to ensure the type is large enough for the shift.

Sponsored by: Arm Ltd

arm64: Fix the TCR_EPD0 definition

Sponsored by: Arm Ltd

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

arm64: Add constants for decoding ISS fields for WF* exceptions

WFI and WFIT trap to EL2 when executed in a vmm guest. (Currently
WFE/WFET are not configured to trap.) We only handle WFI at the moment,
so these constants are useful when handling the exception.

Reviewed by: andrew
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41199

arm64: Decode the ID_AA64PFR2_EL1 register

No fields have been defined, but it has been documented in the
Architecture Reference Manual.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40897

arm64: Update the ID_AA64PFR1_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40896

arm64: Update the ID_AA64PFR0_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40895

arm64: Decode the ID_AA64MMFR4_EL1 register

No fields have been defined, but it has been documented in the
Architecture Reference Manual.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40894

arm64: Decode the ID_AA64MMFR3_EL1 register

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40893

arm64: Don't use hex for ID_AA64MMFR2_EL1_op/CR*

It breaks a future macro that creates the alternative register name
for old compilers.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40892

arm64: Update the ID_AA64MMFR1_EL1 fields

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40891

arm64: Update the ID_AA64MMFR0_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40890

arm64: Update the ID_AA64ISAR1_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40889

arm64: Update the ID_AA64ISAR0_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40888

arm64: Update the ID_AA64DFR0_EL1 fields

While here move to decimal for the _op and _CR definitions to be used
by a future macro to define the register when the assembler doesn't
know about it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40887

arm64 lib32: prepare arm64 headers to redirect to arm

In order to compile lib32 libraries and other 32-bit code on arm64,
<machine/foo.h> needs to be redirected to an arm header rather
than arm64 when building with -m32. Ifdef the arm64 headers that
are installed in /usr/include/machine and used by user-level software
(including references from /usr/include/*.h) so that if __arm__ is
defined when including the arm64 version, <arm/foo.h> is included
rather than using the rest of the file's contents. Some arm headers
had no arm64 equivalent; headers were added just to do the redirection.
These files use #error if __arm__ is not defined to guard against
confusion. Also add an include/arm Makefile, and modify Makefiles
as needed to install everything, including the arm files in
/usr/include/arm. fenv.h comes from lib/msun/arm/fenv.h.

The new arm64 headers are:
acle-compat.h
cpuinfo.h
sysreg.h

Reviewed by: jrtc27, imp
Differential Revision: https://reviews.freebsd.org/D40944

Add more arm64 special register values

These will be used to simplify the kernel special register handling.

Sponsored by: Arm Ltd

arm64: Fix the definition of ID_AA64DFR1_EL1

Add more arm64 ID registers

These will be used by bhyve to emulate these registers.

Sponsored by: Arm Ltd

arm64: Remove CNTHCTL_EL2 from arm64.h

It is also in hypervisor.h where it belongs.

Sponsored by: Arm Ltd

Add more arm64 special registers

These will be used by bhyve

Reviewed by: markj
Sponsored by: Arm Ltd
Sponsored by: Innovate UK
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40128

Mark the arm64 PSR register fields with UL

These are for a 64 bit register. Make them 64 bit values on arm64.

Sponsored by: Arm Ltd

Mark arm64 mair_el1 fields as unsigned long

The register is 64-bit so the upper bits could be shifted past the
signed 32-bit size of an int the values were before.

Sponsored by: Arm Ltd

Decode the arm64 ID_AA64ISAR1_EL1 register

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35627

Decode the arm64 SVE ID register

The field values are only valid when the ID_AA64PFR0_EL1.SVE or
ID_AA64PFR1_EL1.SME vields are non-zero. When this is not the case
the register is reserved as zero so is safe to read, but the SVEver
field will be incorrect so only print the decoded register when
the SVE or SME fields indicate it is valid.

Sponsored by: The FreeBSD Foundation

Allow use of the arm64 unnamed register form

On arm64 all registers have a name that encodes op0, op1, CRn, CRm, and
op2 that are used to encode the register in the instruction. As some
registers we need to access may not be supportedby older compilers, or
are only supported when specific extensions are enabled support this
alternative form.

Sponsored by: The FreeBSD Foundation

Add the SVE reigster definitions

Sponsored by: The FreeBSD Foundation

Trap SVE instructions until we have SVE support

When running on hardware that supports SVE send the correct signal when
an SVE instruction is run.

Sponsored by: The FreeBSD Foundation

arm64: Print per-CPU cache summary

Summary:
It can be useful to see a summary of CPU caches on bootup. This is done
for most platforms already, so add this to arm64, in the form of (taken
from Apple M1 pro test):

L1 cache: 192KB (instruction), 128KB (data)
L2 cache: 12288KB (unified)

This is printed out per-CPU, only under bootverbose.

Future refinements could instead determine if a cache level is shared
with other cores (L2 is shared among cores on some SoCs, for instance),
and perform a better calculation to the full true cache sizes. For
instance, it's known that the M1 pro, on which this test was done, has 2
12MB L2 clusters, for a total of 24MB. Seeing each CPU with 12288KB L2
would make one think that there's 12MB * NCPUs, for possibly 120MB
cache, which is incorrect.

Sponsored by: Juniper Networks, Inc.
Reviewed by: #arm64, andrew
Differential Revision: https://reviews.freebsd.org/D35366

Decode all Arm GIC feature ID bits

The AWS Graviton3 CPU features a GIC 4.1 CPU Interface. Teach the CPU
identift code to decode it.

Sponsored by: The FreeBSD Foundation

arm64: Enable the floating-point exception traps

To enable it user-space needs to call feenableexcept().

FPE_FLTIDO has been added as the IDF bit can't be mapped to any existing
FPE code.

Reviewed by: andrew@
Differential revision: https://reviews.freebsd.org/D35247
MFC after: 2 weeks

aarch64: Add constants for fields in the PMEVTYPERn_EL0 event registers.

Reviewed by: andrew
Sponsored by: University of Cambridge, Google, Inc.
Differential Revision: https://reviews.freebsd.org/D34526

Revert "Add the PMCR_EL0.N arm64 register field"

This reverts commit b6fd96ed0b40e5a42eaea662b5461070c93a1c6c.

Add more arm64 register op* and CR* values

These will be used by bhyve/arm64.

Sponsored by: Innovate UK

Add the PMCR_EL0.N arm64 register field

Sponsored by: Innovate UK

Correct the location of the arm64 PMCR register

This was one of the last registers to not be in alphabetical order in
armreg.h. Fix this to make it easier to find.

Sponsored by: Innovate UK

Sort the M* and P* arm64 registers

Sponsored by: Innovate UK

Fix the TCR_TG0 values

They are in a different order to the TCR_TG1 values but appear to have
been copied incorrectly.

While here use TCR_TG0_4K in locore.S to make it explicit the userspace
page size is 4K.

Sponsored by: The FreeBSD Foundation

Add more arm64 PAC identification fields

Sponsored by: The FreeBSD Foundation

Stop single stepping in signal handers on arm64

We should clear the single step flag when entering a signal hander and
set it when returning. This fixes the ptrace__PT_STEP_with_signal test.

While here add support for userspace to set the single step bit as on
x86. This can be used by userspace for self tracing.

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34170

Sort the names of the arm64 debug registers

While here clean up the names for the naming convention of the other
registers in this file.

Reviewed by: kib, mhorne (earlier version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34060

Add the Armv8.3-SPE registers

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier and check if the PAC is correct. If the check fails it will
either return an invalid address or fault to the kernel.

As many of these instructions are a NOP when disabled and in earlier
revisions of the architecture this can be used, for example, to sign
the return address before pushing it to the stack making Return-oriented
programming (ROP) attack more difficult on hardware that supports them.

The kernel manages five 128 bit signing keys: 2 instruction keys, 2 data
keys, and a generic key. The instructions then use one of these when
signing the registers. Instructions that use the first four store the
PAC in the register being signed, however the instructions that use the
generic key store the PAC in a separate register.

Currently all userspace threads share all the keys within a process
with a new set of userspace keys being generated when executing a new
process. This means a forked child will share its keys with its parent
until it calls an appropriate exec system call.

In the kernel we allow the use of one of the instruction keys, the ia
key. This will be used to sign return addresses in function calls.
Unlike userspace each kernel thread has its own randomly generated.

Thread0 has a static key as does the early code on secondary CPUs.
This should be safe as there is minimal user interaction with these
threads, however we could generate random keys when the Armv8.5
Random number generation instructions are present.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31261

Add the arm64 TTBR fields to armreg.h

Sponsored by: The FreeBSD Foundation

arm64: Handle 32bits breakpoint exception.

A different exception is raised when we hit a 32bits breakpoint, rather than
a 64bits one, so handle those as well when COMPAT_FREEBSD32 is defined.
This should fix SIGBUS at least when using breakpoints with thumb2 code.

PR: 256468
MFC After: 1 week

Enable arm64 SError exceptions in the kernel

These are needed to signal to the kernel when a Reliability,
Availability, and Serviceability (RAS) exception has triggered.

Reviewed by: mhorne
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31477

Add macros for the arm64 daifset/daifclr flags

Sponsored by: The FreeBSD Foundation

Add macros for arm64 special reg op and CR values

Use these to simplify the definition of the user_regs array.

Reviewed by: imp, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31333

arm64: HWCAP/HWCAP2 aux args support for 32-bit ARM binaries.

This fixes build/run of golang under COMPAT32 emulation.

PR: 256897
Reviewed by: andrew, mmel, manu, jhb, cognet, Robert Clausecker
Tested by: brd, andrew, Robert Clausecker
MFC after: 3 weeks
Relnotes: yes
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D31175

Update the SCTLR_EL1 register definitions

They are valid as of the ARMv8.7 XML.

While here remove SCTLR_RES0 as it's unused and depends on which CPU
the kernel is running on and switch to shifted values as they are
easier to compare with the documentation.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31120

Fix the name of the arm64 SCTLR_E0E register

The character between the E's was the letter O, however in the Arm
Documentation and XML the character is the number 0 (zero).

Sponsored by: The FreeBSD Foundation

Sync the arm64 special registers with the Armv8.5 XML

Add the missing macros and decode all the fields as described in the
Arm Architecture System Registers XML corresponding to Armv8.5.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D30983

arm64: extend struct db_reg to include watchpoint registers

The motivation is to provide access to these registers from userspace
via ptrace(2) requests PT_GETDBREGS and PT_SETDBREGS.

This change breaks the ABI of these particular requests, but is
justified by the fact that the intended consumers (debuggers) have not
been taught to use them yet. Making this change now enables active
upstream work on lldb to begin using this interface, and take advantage
of the hardware debugging registers available on the platform.

PR: 252860
Reported by: Michał Górny (mgorny@gentoo.org)
Reviewed by: andrew, markj (earlier version)
Tested by: Michał Górny (mgorny@gentoo.org)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28415

arm64: handle watchpoint exceptions from EL0

This is a prerequisite to allowing the use of hardware watchpoints for
userspace debuggers.

This is also a slight departure from the x86 behaviour, since `si_addr`
returns the data address that triggered the watchpoint, not the
address of the instruction that was executed. Otherwise, there is no
straightforward way for the application to determine which watchpoint
was triggered. Make a note of this in the siginfo(3) man page.

Reviewed by: jhb, markj (earlier version)
Tested by: Michał Górny (mgorny@gentoo.org)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28561

arm64: validate breakpoint registers

In particular, we want to disallow setting breakpoints on kernel
addresses from userspace. The control register fields are validated or
ignored as appropriate.

Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28560

Add support for arm64 nGnRE device memory

On arm64 we can select how strongly we order device memory. Currently
we use the strongest type of non-Gathering, non-Reordering, no Early
write acknowledgement. This is equivalent to VM_MEMATTR_SO in the 32-bit
arm code.

Create a new memory type to remove the no Early write acknowledgement
option to create a memory attribute that is equivalent to the arm
VM_MEMATTR_DEVICE.

Keep the the old nGnRnE memory as what we provide for VM_MEMATTR_DEVICE
until we can test nGnRE on more hardware. A method for dynamically
switching back may be needed as at least one vendor is known to have
broken nGnRE memory.

Sponsored by: Innovate UK

Extract the logic from pmap_kextract

This allows us to use it when we only need to check if the virtual address
is valid. For example when checking if an address in the DMAP region is
mapped.

Reviewed by: kib, markj
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D27621

Update the ID_AA64MMFR2_EL1 register definitions

This brings these definitions in sync with the ARMv8.6 version of the
architecture reference manual.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26706

arm64: clean up empty lines in .c and .h files

arm64: update instruction set attribute register definitions

This adds definitions for the latest additions to the AA64ISAR[01] ID
registers. This brings these registers in sync with ARMv8.6 initial spec
release.

An future change will parse many of these fields for HWCAP features.

Reviewed by: andrew, manu, markj (all previous versions)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26029

Add support for Cortex-A76/Neoverse-N1 to hwpmc

This adds support for the Cortex-A76 and Neoverse-N1 PMU counters to pmc.

While here add more PMCR_IDCODE values and check the implementers code is
correct before setting the PMU type.

Reviewed by: bz, emaste (looks reasonable to me)
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D25959

arm64 armreg.h: fix TCR_TBI1 definition

Submitted by: Greg V <greg@unrelenting.technology>
Differential Revision: https://reviews.freebsd.org/D25411

Add all the TCR_EL1 fields

These will be used when adding support for new Armv8 extensions.

Sponsored by: Innovate UK

Add more are64 special register fields

Obtained from: https://github.com/FreeBSD-UPB/freebsd

Update the hypervisor registers

- Add more registers needed by bhyve [1]
- Move EL2 registers from armreg.h to hypervisor.h
- Add the register name to hypervisor.h

Obtained from: https://github.com/FreeBSD-UPB/freebsd [1]

Fix the spelling of the VIPT cache type field

Sponsored by: Innovate UK

Add more arm64 CTR_EL0 register fields

While here make the _SIZE macros return the size in bytes, not the log2
of the size

Sponsored by: Innovate UK

Print missing ID_AA64PFR{0,1}_EL1 register fields.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23213

Determine whether the MMU hardware is capable of updating a page table
entry's access flag and dirty state, and enable this feature when it's
available.

Ensure that we don't overlook a dirty state update that is concurrent
with a call to pmap_enter(). (Previously, all dirty state updates would
have occurred with the containing pmap's lock held, so a page table entry's
dirty state could not have changed while pmap_enter() held that same lock.)

Reviewed by: andrew, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D22907

Add comments and macros to the tcr_el1 setting code to help understand it.

This code is non-obvious when reading for the first time. To help with
understanding of it add comments explaining what it's doing.

While here use macros from armreg.h rather than magic numbers.

Sponsored by: DARPA, AFRL

Utilize ASIDs to reduce both the direct and indirect costs of context
switching. The indirect costs being unnecessary TLB misses that are
incurred when ASIDs are not used. In fact, currently, when we perform a
context switch on one processor, we issue a broadcast TLB invalidation that
flushes the TLB contents on every processor.

Mark all user-space ("ttbr0") page table entries with the non-global flag so
that they are cached in the TLB under their ASID.

Correct an error in pmap_pinit0(). The pointer to the root of the page
table was being initialized to the root of the kernel-space page table
rather than a user-space page table. However, the root of the page table
that was being cached in process 0's md_l0addr field correctly pointed to a
user-space page table. As long as ASIDs weren't being used, this was
harmless, except that it led to some unnecessary page table switches in
pmap_switch(). Specifically, other kernel processes besides process 0 would
have their md_l0addr field set to the root of the kernel-space page table,
and so pmap_switch() would actually change page tables when switching
between process 0 and other kernel processes.

Implement a workaround for Cavium erratum 27456 affecting ThunderX machines.
(I would like to thank andrew@ for providing the code to detect the affected
machines.)

Address integer overflow in the definition of TCR_ASID_16.

Setup TCR according to the PARange and ASIDBits fields from
ID_AA64MMFR0_EL1. Previously, TCR_ASID_16 was unconditionally set.

Modify build_l1_block_pagetable so that lower attributes, such as ATTR_nG,
can be specified as a parameter.

Eliminate some unused code.

Earlier versions were tested to varying degrees by: andrew, emaste, markj

MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D21922

Add support for setting hardware breakpoints from ptrace on arm64.

Implement get/fill_dbregs on arm64. This is used by ptrace with the
PT_GETDBREGS and PT_SETDBREGS requests. It allows userspace to set hardware
breakpoints.

The struct dbreg is based on Linux to ease adding hardware breakpoint
support to debuggers.

Reviewed by: jhb
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22195

Allow exceptions to be masked when in userspace

We may want to mask exceptions when in userspace. This was previously
impossible as threads are created with all exceptions unmasked and
signals expected userspace to mask any. Fix these by copying the
mask state on thread creation and allow exceptions to be masked on
signal return, as long as they don't change.

Sponsored by: DARPA, AFRL

Allow the userspace ID register fields to be read from the kernel

To allow consistent values to be used in both the kernel and userspace
create a function for these to be read from the kernel. They use a newly
created macro with the name of the ID register to read. For now there is
redundant information in the user_regs array as it still holds the CRm and
Op2 values, however this will be fixed in a later change.

This will be used by ptrace to allow hardware breakpoints in userspace.

Sponsored by: DARPA, AFRL

Move the MRS instruction decode macros to armreg.h

These instructions are used to access the registers described in armreg.h,
and will be used in a future change to create a per-register identification
macro.

Sponsored by: DARPA, AFRL

Rename the macros to extract a single arm64 ID field.

Because of the previous naming scheme the old ID_AA64PFR0_EL1 macro
collided with a potential macro for the register of the same name. To fix
this collision rename these macros.

Sponsored by: DARPA, AFRL

Remove the arm4 ID register masks, they are not needed after r353641.

Sponsored by: DARPA, AFRL

As with r350241 use the new UL macro on the main register mask.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Ensure the arm64 ID register fields are 64 bit types.

Previously only some of the ID register fields were 64 bit. To allow
for a script to generate these mark them all 64 bit. To allow for their
use in assembly we need to use the UINT64_C macro via a new UL macro
to stop the lines from being too long.

MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D20977

Rename arm64 macros in preperation for a script to generate them.

I have a script to generate most of the ID_AA64* macros from the Arm
XML source [1]. In preperation for using this we need to clean up the
macros to be in line with what the script will generate. This is the
first step, rename the macros to follow the names in said XML.

[1] https://developer.arm.com/architectures/cpu-architecture/a-profile/exploration-tools

MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D20976

Fix some ISS bit definitions for data aborts.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Lightly hide the 'var' inside the macros to read the arm special registers.
I just happenned to have 3rd party code using 'var' as the output variable
which drew my attention to this. variables defined inside macros should be
prefixed to avoid getting shadowed varable wanrings from clang.

Impleent COMPAT_FREEBSD32 for arm64.
This is based on early work by andrew@.

Add the ARMv8.3 SCTLR_EL1 fields.

While here tag which architecture release fields were added and remove a
field that only existed in very early releases of the ARMv8 spec.

Sponsored by: DARPA, AFRL

Fix the ID_AA64ISAR0_EL1 dot product field shift.

It's 44 in the documentation, use this correct value.

MFC after: 3 days

Correctly set the DAIF bits in new threads

We should only unmask interrupts when creating a new thread and leave the
other exceptions in teh same state as before creating the thread.

Reported by: jhibbits
Reviewed by: jhibbits
MFC after: 1 month
Sponsored by: https://reviews.freebsd.org/D17497

Add a Saved Process Status Register bit for AArch32 execution mode.

The documentation on the Saved Process Status Register (SPSR) is a bit
weird; the M[4] bit is documented separately from M[3:0]. The M[4] bit
can be toggled to switch to 32-bit execution mode. This functionality is
orthogonal to M[3:0].

Change the definition of PSR_M_MASK to no longer include M[4]. Add a new
definition, PSR_AARCH32 that can be used to toggle 32-bit independently.
This bit will be used by the cloudabi32 code to force execution of
userspace code in 32-bit mode.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13148

Make 32-bit system calls end up in svc_handler().

The nice thing about ARM64 is that it's pretty elegant to install
separate trap/exception handlers for 32-bit and 64-bit processes. That
said, for all other architectures (e.g., i386 on amd64) we always let
32-bit counterparts go through the regular system call codepath. Let's
do the same on ARM64.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13146

Ensure we check the program state set in the trap frame on arm and arm64.
This value may be set by userspace so we need to check it before using it.
If this is not done correctly on exception return the kernel may continue
in kernel mode with all registers set to a userspace controlled value. Fix
this by moving the check into set_mcontext, and also add the missing
sanitisation from the arm64 set_regs.

Discussed with: security-officer@
MFC after: 3 days
Sponsored by: DARPA, AFRL

Add the ARMv8.3 ID register fields. These were found in the A-Profile
exploration tools documentation:
https://developer.arm.com/products/architecture/a-profile/exploration-tools

Sponsored by: DARPA, AFRL

Fix the SVE ID field shift.

Sponsored by: DARPA, AFRL

Fix the value of ID_AA64ISAR1_DPB_SHIFT, the field is bits 3:0.

Sponsored by: DARPA, AFRL

Add the ARMv8.2 ID register additions and use them to decode the register
values. As not all assemblers understand the new ID_AA64MMFR2_EL1 register
add a macro to access it. This seems to be safe for older CPUs to read this
new register, with them returning zero.

Sponsored by: DARPA, AFRL

Allocate a cacheline when reading or writing to write through memory. The
hardware will still write to memory, however following reads will be from
the cache.

MFC after: 1 week
Sponsored by: DARPA, AFRL

In ARMv8.1 ARM has added a process state bit to disable access to userspace
from the kernel. Make use of this to restrict accessing userspace to just
the functions that explicitly handle crossing the user kernel boundary.

Reported by: kib
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D10371

Add SCTLR bits added in ARMv8.1 and ARMv8.2 and start to use them in the
early boot code.

Sponsored by: DARPA, AFRL

Fix ISS_DATA_DFSC_MASK, it should include all bits in the field.

Fix the TLB conflict abort value. This should be a no-op as we don't use
this value in the code.

Create macros for the MAIR memory attributes. While here add an uncached
memory type, however the VM code still needs to be taught about this.

MFC after: 1 week
Sponsored by: ABT Systems Ltd

Add the ARMv8.1 identification registers to the list we print when booting.

MFC after: 1 week
Sponsored by: ABT Systems Ltd

Add the fields for the PAR_EL1 register. This is used when performing an
address lookup with the AT instructions.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Add the Data Fault Status Code values to the ESR_ELx registers for when the
fault code is a Data Abort.

Obtained from: AT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

arm64: bzero optimization

This optimization attempts to utylize as wide as possible register store instructions to zero large buffers.
The implementation, if possible, will use 'dc zva' to zero buffer by cache lines.

Speedup: 60x faster memory zeroing

Submitted by: Dominik Ermel <der@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D5726

Enable SRE_EL2 on ARM64

Enable system register access for EL2. Alpine-V2 is
the first device requiring this to be enabled.
It is also in-sync with Linux initialization code,
and compatible with Alpine-V2 uboot requirements.

Obtained from: Semihalf
Submitted by: Michal Stanek <mst@semihalf.com>
Sponsored by: Annapurna Labs
Approved by: cognet (mentor)
Reviewed by: wma
Differential revision: https://reviews.freebsd.org/D5394

Minor clean-ups for ARM64 GICv3 and GIC drivers

GICv3:
- move ICC_SGI1R_EL1 definitions to armreg.h and use proper system
register's names
GIC:
- remove unused functions

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5119

Implement single stepping on arm64. We need to set the single step bits in
the processor and debug state registers. A flag has been added to the pcb
to tell us when to enable single stepping for a given thread.

Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4730

Framework for ARM64 instruction disassembler

Provide an easy to use framework for ARM64 DDB disassembler.
This commit does not contain full list of instruction opcodes.

Obtained from: Semihalf
Sponsored by: Cavium
Approved by: cognet (mentor)
Reviewed by: zbb, andrew, cognet
Differential revision: https://reviews.freebsd.org/D5114

Decode and print the ID_AA64* registers on boot. These registers hold
information on what the core supports. In most cases these will be
identical across most CPUs in the SoC, however there may be the case where,
with a big.LITTLE setup they may differ. In this case we print the
decoded data on all CPUs.

Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4725

Use 4 levels of page tables when enabling the MMU. This will allow us to
boot on an SoC that places physical memory at an address past where three
levels of page tables can access in an identity mapping.

Submitted by: Wojciech Macek <wma@semihalf.com>,
Patrick Wildt <patrick@bitrig.org>
Differential Revision: https://reviews.freebsd.org/D3885 (partial)
Differential Revision: https://reviews.freebsd.org/D3744

Add the CNTHCTL_EL2 register bits missed in r286674

Set-up proper TCR values for memory related to Translation Table Walking

This commit adds proper cache and shareability attributes to
the TCR register.
Set memory attributes to Normal, outer and inner cacheable WBWA.
Set shareability to inner and outer shareable when SMP is enabled.

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3093

Add Performance Monitoring Counters support for AArch64.
Family-common and CPU-specific counters implemented.

Supported CPUs: ARM Cortex A53/57/72.

Reviewed by: andrew, bz, emaste, gnn, jhb
Sponsored by: ARM Limited
Differential Revision: https://reviews.freebsd.org/D2555

Add support for ARM GICv3 interrupt controller used in some ARM64 chips

GICv3 allows to distribute interrupts to more than 8 cores served by
the previous GIC revisions. GICv3 introduces additional logic in form
of Re-Distributors associated with particular CPUs to determine
the highest priority interrupts and manage PPIs and LPIs
(Locality-specific Peripheral Interrupts). Interrupts routing is
based on CPUs' affinity numbers. CPU interface was changed to be
accessible via CPU System Registers and this is the preferred
(and supported) method in this driver.

Obtained from: Semihalf
Reviewed by: andrew, emaste, ian, imp
Sponsored by: The FreeBSD Foundation

Bring in the start of the arm64 kernel.

This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.

The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.

This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.

Differential Revision: https://reviews.freebsd.org/D2199
Reviewed by: emaste, imp
Relnotes: yes
Sponsored by: The FreeBSD Foundation