arm64: Support BTI checking in most of the kernel

LLD has the -zbti-report=error argument to check if the BTI note is
present when linking. To allow for this to be used when linking the
kernel and modules:
- Add the BTI note to the remaining assembly files
- Mark ptrauth.c as protected by BTI
- Disable -zbti-report for vmm hypervisor switching code as it's not
used there.

The linux64 module doesn't build with the flag as it includes vdso code
that doesn't include the note.

Reviewed by: imp, kib, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45466

arm64: Separate serror handler to EL1H and EL0 versions.

In order to ensure the registers are saved and restored properly for
the exception level, we need separate handlers serror at each of
EL1H and EL0.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D44630

arm64: Make KMSAN aware of exceptions

- Call kmsan_intr_enter() when an exception occurs. This ensures that
code running in the exception context does not clobber thread-local
KMSAN state.
- Ensure that stack memory containing trap frames is treated as
initialized.

Co-authored-by: Alexander Stetsenko <alex.stetsenko@klarasystems.com>
Reviewed by: imp
MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43155

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

arm64: fix stack unwinding past exception handlers

Commit 281402e0a563 ("arm64: Shave off two instructions in exceptions")
removed the instruction that set the frame pointer (x29) as it appeared
to be unused.

The frame pointer is used in arm64/db_trace.c:db_stack_trace_cmd() when
unwinding state, and hence still needs to be set.

Add back the instruction to save_registers to properly update frame
pointer.

Reported by: andrew
Sponsored by: Arm Ltd

arm64: add fault address to trapframe

It was previously possible for the fault address register to get
clobbered before it was saved. This small window occurred when an
additional exception was encountered inside the exception handler,
overwriting the previous value.

Commit f29942229d24 ("Read the arm64 far early in el0 exceptions")
patched this issue, but avoided changing the trapframe since this could
be considered a KBI change in FreeBSD 13.

Revert the above fix and save the fault address in the trapframe
instead. This saves the fault address even earlier in the exception
handling process, and is a more robust and simple fix.

Reviewed by: andrew, jhb, jrtc27
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38984

arm64: extend ESR/SPSR registers to 64b

For the Exception Syndrome Register, ESR_ELx, the upper 32b were
previously unused, but now may contain additional exception info as of
Armv8.7 (FEAT_LS64).

Extend ESR from u32->u64 in exception handling code to support this. In
addition, also extend Saved Program Status Register SPSR_ELx in the same
way to allow for future extensions.

Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38983

arm64: Shave off two instructions in exceptions

This patch shaves off up to two three instructions in
save_registers_head in exception.S for arm64, which would make more
space for instructions that could be added in CheriBSD.

This is done by:
1. Combining pointer arithmetic with pre-incrementing STP instructions
2. Removing the instruction that sets the frame pointer (x29) as its
content is unused

Differential Revision: https://reviews.freebsd.org/D34631

Read the arm64 far early in el0 exceptions

When handling userspace exceptions on arm64 we need to dereference the
current thread pointer. If this is being promoted/demoted there is a
small window where it will cause another exception to be hit. As this
second exception will set the fault address register we will read the
incorrect value in the userspace exception handler.

Fix this be always reading the fault address before dereferencing the
current thread pointer.

Reported by: olivier@
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38196

arm64: Don't include td_inhibitors when checking td_ast in do_ast

The td_ast member is an int so only 4 bytes, yet we were using an 8 byte
load and thus also got td_inhibitors in the upper bits. The code prior
to the commit that introduced td_ast did also do a bogus 8 byte load of
td_flags but masked the flags so arguably was correct, if dodgy. Now
that we're using the right width for the load we can also fold the
immediate offset back into the load; because td_ast is at an odd
multiple of 4 bytes from the start of struct thread the normal scaled
load couldn't be used with such an immediate offset when doing an 8 byte
load due to its limited immediate range, but we can use a scaled load
once more now that the offset is a multiple of the load width.

Reviewed by: andrew, kib
Fixes: c6d31b8306eb ("AST: rework")
Differential Revision: https://reviews.freebsd.org/D37751

AST: rework

Make most AST handlers dynamically registered. This allows to have
subsystem-specific handler source located in the subsystem files,
instead of making subr_trap.c aware of it. For instance, signal
delivery code on return to userspace is now moved to kern_sig.c.

Also, it allows to have some handlers designated as the cleanup (kclear)
type, which are called both at AST and on thread/process exit. For
instance, ast(), exit1(), and NFS server no longer need to be aware
about UFS softdep processing.

The dynamic registration also allows third-party modules to register AST
handlers if needed. There is one caveat with loadable modules: the
code does not make any effort to ensure that the module is not unloaded
before all threads processed through AST handler in it. In fact, this
is already present behavior for hwpmc.ko and ufs.ko. I do not think it
is worth the efforts and the runtime overhead to try to fix it.

Reviewed by: markj
Tested by: emaste (arm64), pho
Discussed with: jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D35888

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier and check if the PAC is correct. If the check fails it will
either return an invalid address or fault to the kernel.

As many of these instructions are a NOP when disabled and in earlier
revisions of the architecture this can be used, for example, to sign
the return address before pushing it to the stack making Return-oriented
programming (ROP) attack more difficult on hardware that supports them.

The kernel manages five 128 bit signing keys: 2 instruction keys, 2 data
keys, and a generic key. The instructions then use one of these when
signing the registers. Instructions that use the first four store the
PAC in the register being signed, however the instructions that use the
generic key store the PAC in a separate register.

Currently all userspace threads share all the keys within a process
with a new set of userspace keys being generated when executing a new
process. This means a forked child will share its keys with its parent
until it calls an appropriate exec system call.

In the kernel we allow the use of one of the instruction keys, the ia
key. This will be used to sign return addresses in function calls.
Unlike userspace each kernel thread has its own randomly generated.

Thread0 has a static key as does the early code on secondary CPUs.
This should be safe as there is minimal user interaction with these
threads, however we could generate random keys when the Armv8.5
Random number generation instructions are present.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31261

Fix a typo in an arm64 comment

This was pointed out by markj in the review, but I missed it and forgot
to fix before pushing.

Reported by: markj
Sponsored by: The FreeBSD Foundation

Move instructions into the arm64 exception vectors

We have 32 instructions in each exception vector on arm64. Previously
only one was used to branch to the handler function. We can split the
start of these functions and move some of the instructions into the
vectors.

Reviewed by: kib, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33751

Per-thread stack canary on arm64

With the update to llvm 13 we are able to tell the compiler it can find
the SSP canary relative to the register that holds the userspace stack
pointer. As this is unused in most of the kernel it can be used here
to point to a per-thread SSP canary.

As the kernel could be built with an old toolchain, e.g. when upgrading
from 13, add a warning that the options was enabled but the compiler
doesn't support it to both the build and kernel boot.

Discussed with: emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33079

arm64: Pass the right label to END() for handle_empty_exception.

GNU as reported an error for the argument to .size not being a constant.

Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D31950

Enable arm64 SError exceptions in the kernel

These are needed to signal to the kernel when a Reliability,
Availability, and Serviceability (RAS) exception has triggered.

Reviewed by: mhorne
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31477

Add macros for the arm64 daifset/daifclr flags

Sponsored by: The FreeBSD Foundation

arm64: fix hardware single-stepping from EL1

The main issue is that debug exceptions must to be disabled for the
entire duration that SS bit in MDSCR_EL1 is set. Otherwise, a
single-step exception will be generated immediately. This can occur
before returning from the debugger (when MDSCR is written to) or before
re-entering it after the single-step (when debug exceptions are unmasked
in the exception handler).

Solve this by delaying the unmask to C code for EL1, and avoid unmasking
at all while handling debug exceptions, thus avoiding any recursive
debug traps.

Reviewed by: markj, jhb
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28944

arm64: Improve DDB backtrace support

The existing implementation relies on each trap handler saving a normal
stack frame record, which is a waste of time and space when we're
already saving a trapframe to the stack. It's also wrong as it currently
saves LR not ELR.

Instead of patching it up, rewrite it based on the RISC-V implementation
with inspiration from the amd64 implementation for how to handle
vectored traps to provide an improved implementation. This includes
compressing the information down to one line like other architectures
rather than the highly-verbose old form that repeats itself by printing
LR and FP in one frame only to print them as PC and SP in the next. It
also includes printing out actually useful information about the traps
that occurred, though FAR is not saved in the trapframe so we cannot
print it (in general it can be clobbered between when the trap happened
and now), only ESR.

The AAPCS also allows the stack frame record to be located anywhere in
the frame, not just the top, so the caller's SP is not at a fixed offset
from the callee's FP like on almost all other architectures in
existence. This means there is no way to derive the caller's SP in the
unwinder, and so we have to drop that bit of (unused) state everywhere.

Reviewed by: jhb, markj
Differential Revision: https://reviews.freebsd.org/D28026

arm64: remove pcb_pc

The program counter field in the PCB is written in exactly one place,
makectx(), upon entry to the debugger. For threads other than curthread,
its value will be empty, or bogus. Rather than writing to this field in
more places, it can be removed in favor of using the value in the link
register.

To make this clearer, pcb->pcb_x[30] is renamed to pcb->pcb_lr, similar
to what already exists in struct trapframe. Also, prefer lr to x30 in
assembly, as it better conveys intention.

This improves PC_REGS() for kdb_thread != curthread. It is required for
a functional gdb(4) stub, fixing the output of `info threads`, in
particular.

The space occupied by pcb_pc is retained, for compatibility with kgdb.

Reviewed by: markj, jhb
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D27720

Stop speculation past an eret instruction

On arm64 the eret instruction is used to return from an exception handler.
Some implementations may speculate past this instruction into the next
function. As the user may control many registers in these functions add
a synchronisation barrier sequence after the eret instruction to stop these
CPUs from speculating out of the exception handler.

PR: 242676
Submitted by: Anthony Steinhauser <asteinhauser@google.com> (previous version)
MFC after: 1 week

Add support for setting hardware breakpoints from ptrace on arm64.

Implement get/fill_dbregs on arm64. This is used by ptrace with the
PT_GETDBREGS and PT_SETDBREGS requests. It allows userspace to set hardware
breakpoints.

The struct dbreg is based on Linux to ease adding hardware breakpoint
support to debuggers.

Reviewed by: jhb
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22195

Update the debug monitor handling to work after userspace has started

The debug monitor register state is now stored in a struct and updated
when required. Currently there is only a kernel state, however a
per-process state will be added in a future change.

Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22128

Implement the SSBD (CVE-2018-3639) workaround on arm64

This calls into the Arm Trusted Firmware to enable and disable the
workaround for the Speculative Store Bypass Disable (SSBD) issue, also
known as Spectre Variant 4.

As this may have a large performance overhead, and how exploitable SSBD is
is unknown we follow the Linux lead of allowing the administrator to select
between always on, always off, or only enabled in the kernel, with the
latter being the default.

PR: 228955
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D15819

Clean up handling of unexpected exceptions. Previously we would issue a
breakpoint instruction, however this would lose information that may be
useful for debugging.

These are now handled in a similar way to other exceptions, however it
won't exit out of the exception handler until it is known if we can
handle these exceptions in a useful way.

Sponsored by: DARPA, AFRL

Rename assym.s to assym.inc

assym is only to be included by other .s files, and should never
actually be assembled by itself.

Reviewed by: imp, bdrewery (earlier)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D14180

In do_ast, make sure the interrupts are enabled before calling ast().
We can reach that point with IRQs disabled, and calling ast() with IRQs 
disabled can lead to a deadlock.
This should fix the freezes on arm64 under load.

Reviewed by: andrew

Make 32-bit system calls end up in svc_handler().

The nice thing about ARM64 is that it's pretty elegant to install
separate trap/exception handlers for 32-bit and 64-bit processes. That
said, for all other architectures (e.g., i386 on amd64) we always let
32-bit counterparts go through the regular system call codepath. Let's
do the same on ARM64.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13146

Push loading curthread into assembly in the synchronous exception handlers.
This will help investigating the performance impact of moving parts of the
switch statement in do_el0_sync into assembly.

Sponsored by: DARPA, AFRL

Push reading of ESR_EL1 to assembly. Among other uses this will allow us
to expose this to signal handlers, e.g. for the clang sanitizers.

Sponsored by: DARPA, AFRL

Reduce the diff with intrng by renaming similar functions. This is a noop,
but will help move to use the common interrupt handling code later.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Skip restoring more registers when returning from an exception taken in
the kernel. These registers are all callee saved, and as such will be
restored before returning to the exception handler.

Userland still needs these registers to be restored as they may be changed
by the kernel and we don't currently track these places.

Rework the exception entry/return functions to make them valid frames to be
unwound through. For this we need the frame pointer (x29) to point to the
location on the stack where we stored the previous frame pointer, and link
register. To simplify this the stack pointer is only adjusted by addition
and subtraction, and not through the use of post increment on loads and
stores.

The updated frame layout is:

+------------+
| lr -- x30 |
+------------+
| fp -- x29 | <-- x29 points at this
+------------+
| Trap frame |
| ... |
| | <-- sp points at this
+------------+

The only difference is the first two items, and setting of x29.

Sponsored by: ABT Systems Ltd

Shorten conditional branch code.

Reviewed by: andrew
Sponsored by: The FreeBSD Foundation

Add the ENTRY/END entries around the exception handlers.

Obtained from: EuroBSDCon Devsummit
Sponsored by: ABT Systems Ltd

Call ast when handling irq from userspace, otherwise we could miss
reschedule. Right now arm_cpu_intr() does critical_exit() as the last
action, so the impact is not serious.

Remove duplicated interrupt disable in restore_registers macro, when
returning to usermode. The do_ast macro disabled interrupts for us.

Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3714

Don't restore interrupts when we are about to disable them in the next
instruction.

Re-check for new ast after ast was handled. We should not return to
usermode with pending asts.

Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3667

Remove arm64 workaround for Clang 3.4 crash

First cut of DTrace for AArch64.

Reviewed by: andrew, emaste
Sponsored by: ARM Limited
Differential Revision: https://reviews.freebsd.org/D2738

Rework exception entry to help with DTrace. We now store the stack pointer
before adjusting it to store any registers. This is needed as DTrace may
need to adjust the kernel stack pointer, and previously the new stack
pointer would have needed to be checked incase it was changed.

Bring in the start of the arm64 kernel.

This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.

The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.

This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.

Differential Revision: https://reviews.freebsd.org/D2199
Reviewed by: emaste, imp
Relnotes: yes
Sponsored by: The FreeBSD Foundation