arm64: Add EL1 hardware breakpoint exceptions

Reviewed by: jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44353

arm64: Use a switch to decide when to enable debug

Use a switch statement to decide which exceptions we need to call
dbg_enable for. This simplifies adding more esceptions to the list in
the future.

Reviewed by: jhb
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44352

arm64: Make KMSAN aware of exceptions

- Call kmsan_intr_enter() when an exception occurs. This ensures that
code running in the exception context does not clobber thread-local
KMSAN state.
- Ensure that stack memory containing trap frames is treated as
initialized.

Co-authored-by: Alexander Stetsenko <alex.stetsenko@klarasystems.com>
Reviewed by: imp
MFC after: 2 weeks
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43155

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

arm64: Raise a SIGBUS on a user external abort

When userspace triggers an external abort allow it to handle the abort
by raising a SIGBUS.

Reviewed by: markj, kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42018

syscalls: fix missing SIGSYS for several ENOSYS errors

In particular, when the syscall number is too large, or when syscall is
dynamic. For that, add nosys_sysent structure to pass fake sysent to
syscall top code.

Reviewed by: dchagin, markj
Discussed with: jhb
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D41976

Add BTI exceptions

We could hit these when executing code marked as using BTI but jumps
to a non-branch target instruction.

Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39450

arm64: initialize pcb in the TBI/PAC/etc. fault case

After 2c10be9e06d, we may jump to the bad_far label without `pcb` being
set, resulting in a follow-up fault as we may dereference it immediately
after the jump if td_intr_nesting_level == 0. In this branch, it should
be safe to dereference `td` as we're not handling the special case
mentioned below of accessing it during promotion/demotion.

This seems to fix a null ptr deref I hit during my most recent pkgbase
build attempt on the Windows DevKit, though that was admittedly
encountered while we were on the way to a panic from an apparent
use-after-free in ZFS bits.

Reviewed by: andrew, markj
Fixes: 2c10be9e06d ("arm64: Handle translation faults for thread [..]")
Differential Revision: https://reviews.freebsd.org/D41677

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

arm64: improve KTR_TRAP trace entries

Follow-up of 7756232199ad ("riscv: improve KTR_TRAP trace entries").

Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40696

arm64: print '0x' before hex values

Not making it explicit that we're printing values in hex can be
misleading when the number doesn't contain hex-only symbols (a-f). A
good example of this is print_gp_register(), where we print "(func +
offset)"; if the offset doesn't contain a-f symbols, it's not
immediately clear if that value is in decimal or hex. Using '%#' instead
of '0x%' also isn't a better option, it doesn't print '0x' if the value
is 0, and it also messes up column alignment.

Reviewed by: imp, markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40583

arm64: Print the spinlock count on panic

When the spinlock count is non-zero while taking a data abort we panic.
Print this count to help debugging.

Sponsored by: Arm Ltd

Print registers on an arm64 spinlock data abort

When checking if the spinlock count is correct in a data abort we can
panic without printing the registers. These are useful to debug the
abort, e.g. by giving the fault address register.

Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40228

Don't print leading 0's in the arm64 esr

We don't print leading zeros for other registers, so do the same with
ESR_EL1.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D40129

Handle the arm64 unknown exception separately

Rather than falling through to the default case handle the unknown
exception with its own panic message. As ESR_EL1 is zero for this
exception stop printing it.

Sponsored by: Arm Ltd

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow map implementation is roughly similar to what was done on
amd64, with some exceptions. Attempting to use available space at
preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range,
as depicted in the physmap) results in odd failures, so we instead
search the physmap for free regions that we can carve out, fragmenting
the shadow map as necessary to try and fit as much as we need for the
initial kernel map. pmap_bootstrap_san() is thus after
pmap_bootstrap(), which still included some technically reserved areas
of the memory map that needed to be included in the DMAP.

The odd failure noted above may be a bug, but I haven't investigated it
all that much.

Initial work by mhorne with additional fixes from kevans and markj.

Reviewed by: andrew, markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36701

arm64: add fault address to trapframe

It was previously possible for the fault address register to get
clobbered before it was saved. This small window occurred when an
additional exception was encountered inside the exception handler,
overwriting the previous value.

Commit f29942229d24 ("Read the arm64 far early in el0 exceptions")
patched this issue, but avoided changing the trapframe since this could
be considered a KBI change in FreeBSD 13.

Revert the above fix and save the fault address in the trapframe
instead. This saves the fault address even earlier in the exception
handling process, and is a more robust and simple fix.

Reviewed by: andrew, jhb, jrtc27
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38984

arm64: extend ESR/SPSR registers to 64b

For the Exception Syndrome Register, ESR_ELx, the upper 32b were
previously unused, but now may contain additional exception info as of
Armv8.7 (FEAT_LS64).

Extend ESR from u32->u64 in exception handling code to support this. In
addition, also extend Saved Program Status Register SPSR_ELx in the same
way to allow for future extensions.

Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38983

Read the arm64 far early in el0 exceptions

When handling userspace exceptions on arm64 we need to dereference the
current thread pointer. If this is being promoted/demoted there is a
small window where it will cause another exception to be hit. As this
second exception will set the fault address register we will read the
incorrect value in the userspace exception handler.

Fix this be always reading the fault address before dereferencing the
current thread pointer.

Reported by: olivier@
Reviewed by: markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38196

arm64: Disable per-thread stack-smashing protection in data_abort()

With PERTHREAD_SSP configured, the compiler's stack-smashing protection
uses a per-thread canary value instead of a global value. The value is
stored in td->td_md.md_canary; the sp_el0 register always contains a
pointer to that value, and certain functions selected by the compiler
will store the canary value on the stack as a part of the function
prologue (and will verify the copy as part of the epilogue). In
particular, the thread structure may be accessed.

This happens to occur in data_abort(), which leads to the same problem
addressed by commit 2c10be9e06d4 ("arm64: Handle translation faults for
thread structures"). This commit fixes that directly, by disabling SSP
in data_abort() and a couple of related functions by using a function
attribute. It also moves the update of sp_el0 out of C code in case
the compiler decides to start checking the canary in pmap_switch()
someday.

A different solution might be to move the canary value to the PCB, which
currently lives on the kernel stack and isn't subject to the same
problem as thread structures (if only because guard pages inhibit
superpage promotion). However, there isn't any particular reason the
PCB has to live on the stack today; on amd64 it is embedded in struct
thread, reintroducing the same problem. Keeping the reference canary
value at the top of the stack is also rather dubious since it could be
clobbered by a sufficiently large stack overflow.

A third solution could be to go back to the approach of commit
5aa5420ff2e8, and modify UMA to use the direct map for thread structures
even if KASAN is enabled. But, transient promotions and demotions in
the direct map are possible too.

Reviewed by: alc, kib, andrew
MFC after: 1 month
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D37255

arm64: Handle translation faults for thread structures

The break-before-make requirement poses a problem when promoting or
demoting mappings containing thread structures: a CPU may raise a
translation fault while accessing curthread, and data_abort() accesses
the thread again before pmap_fault() can translate the address and
return.

Normally this isn't a problem because we have a hack to ensure that
slabs used by the thread zone are always accessed via the direct map,
where promotions and demotions are rare. However, this hack doesn't
work properly with UMA_MD_SMALL_ALLOC disabled, as is the case with
KASAN configured (since our KASAN implementation does not shadow the
direct map and so tries to force the use of the kernel map wherever
possible).

Fix the problem by modifying data_abort() to handle translation faults
in the kernel map without dereferencing "td", i.e., curthread, and
without enabling interrupts. pmap_klookup() has special handling for
translation faults which makes it safe to call in this context. Then,
revert the aforementioned hack.

Reviewed by: kevans, alc, kib, andrew
MFC after: 1 month
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D37231

Trap SVE instructions until we have SVE support

When running on hardware that supports SVE send the correct signal when
an SVE instruction is run.

Sponsored by: The FreeBSD Foundation

arm64: Enable the floating-point exception traps

To enable it user-space needs to call feenableexcept().

FPE_FLTIDO has been added as the IDF bit can't be mapped to any existing
FPE code.

Reviewed by: andrew@
Differential revision: https://reviews.freebsd.org/D35247
MFC after: 2 weeks

syscallarg_t: Add a type for system call arguments

This more clearly differentiates system call arguments from integer
registers and return values. On current architectures it has no effect,
but on architectures where pointers are not integers (CHERI) and may
not even share registers (CHERI-MIPS) it is necessiary to differentiate
between system call arguments (syscallarg_t) and integer register values
(register_t).

Obtained from: CheriBSD

Reviewed by: imp, kib
Differential Revision: https://reviews.freebsd.org/D33780

Treat cache write as a read in arm64 data faults

On arm64 we can ask the hardware to perform cache operations from
userspace. These require read permission however when the memory is
unmapped the kernel will receive a write exception. Add a check to
see if the cause of the exception is from the cache and pass a memory
read fault type to the vm subsystem.

PR: 262836
Reported by: dch
Sponsored by: The FreeBSD Foundation

Print the instruction for the unknown exception

The arm64 unknown exception will be raised when we execute an
instruction that id invalid or disabled. To help debug these print
the instruction that failed.

Sponsored by: The FreeBSD Foundation

Stop single stepping in signal handers on arm64

We should clear the single step flag when entering a signal hander and
set it when returning. This fixes the ptrace__PT_STEP_with_signal test.

While here add support for userspace to set the single step bit as on
x86. This can be used by userspace for self tracing.

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34170

Sort the names of the arm64 debug registers

While here clean up the names for the naming convention of the other
registers in this file.

Reviewed by: kib, mhorne (earlier version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34060

Revert "syscallarg_t: Add a type for system call arguments"

Missed issues in truss on at least armv7 and powerpcspe need to be
resolved before recommit.

This reverts commit 3889fb8af0b611e3126dc250ebffb01805152104.
This reverts commit 1544e0f5d1f1e3b8c10a64cb899a936976ca7ea4.

syscallarg_t: Add a type for system call arguments

This more clearly differentiates system call arguments from integer
registers and return values. On current architectures it has no effect,
but on architectures where pointers are not integers (CHERI) and may
not even share registers (CHERI-MIPS) it is necessiary to differentiate
between system call arguments (syscallarg_t) and integer register values
(register_t).

Obtained from: CheriBSD

Reviewed by: imp, kib
Differential Revision: https://reviews.freebsd.org/D33780

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier and check if the PAC is correct. If the check fails it will
either return an invalid address or fault to the kernel.

As many of these instructions are a NOP when disabled and in earlier
revisions of the architecture this can be used, for example, to sign
the return address before pushing it to the stack making Return-oriented
programming (ROP) attack more difficult on hardware that supports them.

The kernel manages five 128 bit signing keys: 2 instruction keys, 2 data
keys, and a generic key. The instructions then use one of these when
signing the registers. Instructions that use the first four store the
PAC in the register being signed, however the instructions that use the
generic key store the PAC in a separate register.

Currently all userspace threads share all the keys within a process
with a new set of userspace keys being generated when executing a new
process. This means a forked child will share its keys with its parent
until it calls an appropriate exec system call.

In the kernel we allow the use of one of the instruction keys, the ia
key. This will be used to sign return addresses in function calls.
Unlike userspace each kernel thread has its own randomly generated.

Thread0 has a static key as does the early code on secondary CPUs.
This should be safe as there is minimal user interaction with these
threads, however we could generate random keys when the Armv8.5
Random number generation instructions are present.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31261

syscall_args: remove MAXARGS define

Use nitems instead and just use a magic `8` for the size of the args
array. MAXARGS was rarely used (only in arm64 code) and is an overly
generic name to polute the namespace with.

Requested by: kib in D33308

Print the correct register for the arm64 elr

In 7ec86b6609912 ("Also print symbols when printing arm64 registers")
a new function was created to print most registers. Unfortunately the
Link Register (LR) was being printed when we should have printed the
Exception Link Register (ELR).

Fix this by adding the missing 'e'.

Sponsored by: The FreeBSD Foundation

Also print symbols when printing arm64 registers

When printing arm64 registers because of an exception in the kernel
also print the symbol and offset. This can be used to track down why
the exception occured without needing external tools.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32077

arm64: Handle 32bits breakpoint exception.

A different exception is raised when we hit a 32bits breakpoint, rather than
a 64bits one, so handle those as well when COMPAT_FREEBSD32 is defined.
This should fix SIGBUS at least when using breakpoints with thumb2 code.

PR: 256468
MFC After: 1 week

Add more arm64 external abort sources

These will be used when we support the Arm Reliability, Availability,
and Serviceability extension.

Sponsored by: The FreeBSD Foundation

Start to clean up arm64 address space selection

On arm64 we should use bit 55 of the address to decide if aan address
is a user or kernel address. Add a new macro with this check and a
second to ensure the address is in teh canonical form, i.e.
the top bits are all zero or all one.

This will help with supporting future cpu features, including Top
Byte Ignore, Pointer Authentication, and Memory Tagging.

Reviewed by: kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31179

Pass the syscall number to capsicum permission-denied signals

The syscall number is stored in the same register as the syscall return
on amd64 (and possibly other architectures) and so it is impossible to
recover in the signal handler after the call has returned. This small
tweak delivers it in the `si_value` field of the signal, which is
sufficient to catch capability violations and emulate them with a call
to a more-privileged process in the signal handler.

This reapplies 3a522ba1bc852c3d4660a4fa32e4a94999d09a47 with a fix for
the static assertion failure on i386.

Approved by: markj (mentor)

Reviewed by: kib, bcr (manpages)

Differential Revision: https://reviews.freebsd.org/D29185

Remove the newline from arm64 trap panics

The panic function will already print a newline so there is no need to
add it here.

Sponsored by: The FreeBSD Foundation

Revert "Pass the syscall number to capsicum permission-denied signals"

This broke the i386 build.

This reverts commit 3a522ba1bc852c3d4660a4fa32e4a94999d09a47.

Pass the syscall number to capsicum permission-denied signals

The syscall number is stored in the same register as the syscall return
on amd64 (and possibly other architectures) and so it is impossible to
recover in the signal handler after the call has returned. This small
tweak delivers it in the `si_value` field of the signal, which is
sufficient to catch capability violations and emulate them with a call
to a more-privileged process in the signal handler.

Approved by: markj (mentor)

Reviewed by: kib, bcr (manpages)

Differential Revision: https://reviews.freebsd.org/D29185

arm64, riscv: remove reference to fsu_intr_fault

This variable no longer exists.

MFC after: 3 days

arm64: log vm_fault error for data_abort

Summary:
Log the vm_fault() error in the data_abort panic so it is easier to
find the reason vm_fault() failed (e.g., invalid address).

Reviewed by: andrew
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D30362

arm64: fix hardware single-stepping from EL1

The main issue is that debug exceptions must to be disabled for the
entire duration that SS bit in MDSCR_EL1 is set. Otherwise, a
single-step exception will be generated immediately. This can occur
before returning from the debugger (when MDSCR is written to) or before
re-entering it after the single-step (when debug exceptions are unmasked
in the exception handler).

Solve this by delaying the unmask to C code for EL1, and avoid unmasking
at all while handling debug exceptions, thus avoiding any recursive
debug traps.

Reviewed by: markj, jhb
MFC after: 5 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28944

arm64: handle watchpoint exceptions from EL0

This is a prerequisite to allowing the use of hardware watchpoints for
userspace debuggers.

This is also a slight departure from the x86 behaviour, since `si_addr`
returns the data address that triggered the watchpoint, not the
address of the instruction that was executed. Otherwise, there is no
straightforward way for the application to determine which watchpoint
was triggered. Make a note of this in the siginfo(3) man page.

Reviewed by: jhb, markj (earlier version)
Tested by: Michał Górny (mgorny@gentoo.org)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D28561

Handle arm64 undefied instructions on msr exceptions

When userspace tries to access a special register that it doesn't have
access to the kernel receives an exception. On most cores this exception
has been observed to be the undefined instruction exception, however on
the Apple M1 under a QEMU based hypervisor it can be the MSR exception.

Handle this second case by also running the undefined exception handler
on these exceptions.

Sponsored by: Innovate UK

arm64: don't pass user trapframe to kdb_trap()

This effectively undoes the changes made in r321571. While useful, it is
inconsistent with how other architectures pass trapframes to kdb. This
change is also required to get a working gdb(4) stub on arm64, as
otherwise the backtrace will begin too early.

As of 088a7eef95b1, this information can still be obtained via
"show registers/u".

Reviewed by: jhb (slightly earlier version)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Pull Request: https://reviews.freebsd.org/D27719

Fix write only mappings on arm64

When trapping on a wrote access to a buffer the kernel has mapped as write
only we should only pass the VM_PROT_WRITE flag. Previously the call to
vm_fault_trap as the VM_PROT_READ flag was unexpected.

Reported by: manu
Sponsored by: Innovate UK

Tweak arm64's cpu_fetch_syscall_args(). This should make it possible
for the compiler to inline the memcpy().

Reviewed by: andrew
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D26629

Get rid of sa->narg. It serves no purpose; use sa->callp->sy_narg instead.

Reviewed by: kib
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D26458

Add NetBSD compatible bus_space_peek_N() and bus_space_poke_N() functions.
One problem with the bus_space_read_N() and bus_space_write_N() family of
functions is that they provide no protection against exceptions which can
occur when no physical hardware or device responds to the read or write
cycles. In such a situation, the system typically would panic due to a
kernel-mode bus error. The bus_space_peek_N() and bus_space_poke_N() family
of functions provide a mechanism to handle these exceptions gracefully
without the risk of crashing the system.

Typical example is access to PCI(e) configuration space in bus enumeration
function on badly implemented PCI(e) root complexes (RK3399 or Neoverse
N1 N1SDP and/or access to PCI(e) register when device is in deep sleep state.

This commit adds a real implementation for arm64 only. The remaining
architectures have bus_space_peek()/bus_space_poke() emulated by using
bus_space_read()/bus_space_write() (without exception handling).

MFC after: 1 month
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D25371

Set si_trapno to the exception code from esr.

Reviewed by: kib
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D25771

Add missing space after switch.

Reviewed by: br, emaste
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D25778

Print the arm64 registers in more exception handling panics

It can be useful to get a dump of all registers when investigating why we
received an exception that we are unable to handle. In these cases we
already call panic, however we don't always print the registers.

Add calls to print_registers and print esr and far when applicable.

Sponsored by: Innovate UK

Retire procfs-based process debugging.

Modern debuggers and process tracers use ptrace() rather than procfs
for debugging. ptrace() has a supserset of functionality available
via procfs and new debugging features are only added to ptrace().
While the two debugging services share some fields in struct proc,
they each use dedicated fields and separate code. This results in
extra complexity to support a feature that hasn't been enabled in the
default install for several years.

PR: 244939 (exp-run)
Reviewed by: kib, mjg (earlier version)
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D23837

Print useful debug data on unhandled kernel fault on arm64

When panicing because of an unhandled data abort from the kernel it is
useful to know the register state and faulting address to aid debugging.
Print these registers before calling panic.

Sponsored by: DARPA, AFRL

Add support for setting hardware breakpoints from ptrace on arm64.

Implement get/fill_dbregs on arm64. This is used by ptrace with the
PT_GETDBREGS and PT_SETDBREGS requests. It allows userspace to set hardware
breakpoints.

The struct dbreg is based on Linux to ease adding hardware breakpoint
support to debuggers.

Reviewed by: jhb
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22195

ARM64: Treat alignment faults as bus errors

Summary:
ARM64 currently treats all data abort exceptions as page faults. This
can cause infinite loops on non-page fault faults, such as alignment faults.

Since kernel-side alignment faults should be avoided, this adds support directly
to the el0 fault handler, instead of the data_abort() handler.

Test Plan: Tested on rpi3, with a misaligned ldm test.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D22133

Use a lowercase name for arm64 special registers so they don't conflict
with macros of the same name.

Sponsored by: DARPA, AFRL

Use an array of handlers in the data and instruction aborts

Previously we would call data_abort on all data and instruction aborts
however this is incorrect for most abort types. Move to use an array
of function pointers to allow for more handlers to be easily added.

Reviewed by: jhibbits
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22170

Stop enabling interrupts when reentering kdb on arm64

When we raise a data abort from the kernel we need to enable interrupts,
however we shouldn't be doing this when in the kernel debugger. In this
case interrupts can lead to a further panic as they don't expect to be
run from such a context.

MFC after: 2 weeks
Sponsored by: DARPA, AFRL

Improve MD page fault handlers.

Centralize calculation of signal and ucode delivered on unhandled page
fault in new function vm_fault_trap(). MD trap_pfault() now almost
always uses the signal numbers and error codes calculated in
consistent MI way.

This introduces the protection fault compatibility sysctls to all
non-x86 architectures which did not have that bug, but apparently they
were already much more wrong in selecting delivered signals on
protection violations.

Change the delivered signal for accesses to mapped area after the
backing object was truncated. According to POSIX description for
mmap(2):
The system shall always zero-fill any partial page at the end of an
object. Further, the system shall never write out any modified
portions of the last page of an object which are beyond its
end. References within the address range starting at pa and
continuing for len bytes to whole pages following the end of an
object shall result in delivery of a SIGBUS signal.

An implementation may generate SIGBUS signals when a reference
would cause an error in the mapped object, such as out-of-space
condition.
Adjust according to the description, keeping the existing
compatibility code for SIGSEGV/SIGBUS on protection failures.

For situations where kernel cannot handle page fault due to resource
limit enforcement, SIGBUS with a new error code BUS_OBJERR is
delivered. Also, provide a new error code SEGV_PKUERR for SIGSEGV on
amd64 due to protection key access violation.

vm_fault_hold() is renamed to vm_fault(). Fixed some nits in
trap_pfault()s like mis-interpreting Mach errors as errnos. Removed
unneeded truncations of the fault addresses reported by hardware.

PR: 211924
Reviewed by: alc
Discussed with: jilles, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D21566

Don't pass error from syscallenter() to syscallret().

syscallret() doesn't use error anymore. Fix a few other places to permit
removing the return value from syscallenter() entirely.
- Remove a duplicated assertion from arm's syscall().
- Use td_errno for amd64_syscall_ret_flush_l1d.

Reviewed by: kib
MFC after: 1 month
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D2090

Implement software access and dirty bit management for arm64.

Previously the arm64 pmap did no reference or modification tracking;
all mappings were treated as referenced and all read-write mappings
were treated as dirty. This change implements software management
of these attributes.

Dirty bit management is implemented to emulate ARMv8.1's optional
hardware dirty bit modifier management, following a suggestion from alc.
In particular, a mapping with ATTR_SW_DBM set is logically writeable and
is dirty if the ATTR_AP_RW_BIT bit is clear. Mappings with
ATTR_AP_RW_BIT set are write-protected, and a write access will trigger
a permission fault. pmap_fault() handles permission faults for such
mappings and marks the page dirty by clearing ATTR_AP_RW_BIT, thus
mapping the page read-write.

Reviewed by: alc
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D20907

Include ktr.h in more compilation units

Similar to r348026, exhaustive search for uses of CTRn() and cross reference
ktr.h includes. Where it was obvious that an OS compat header of some kind
included ktr.h indirectly, .c files were left alone. Some of these files
clearly got ktr.h via header pollution in some scenarios, or tinderbox would
not be passing prior to this revision, but go ahead and explicitly include it
in files using it anyway.

Like r348026, these CUs did not show up in tinderbox as missing the include.

Reported by: peterj (arm64/mp_machdep.c)
X-MFC-With: r347984
Sponsored by: Dell EMC Isilon

Pass VM_PROT_EXECUTE to vm_fault for instruction faults.

We need to tell vm_fault the reason for the fault was because we tried to
execute from the memory location. Without this it may return with success
as we only request read-only memory, then we return to the same location
and try to execute from the same memory address. This leads to an infinite
loop raising the same fault and returning to the same invalid location.

MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D18511

Remove iBCS2, part2: general kernel

Reviewed by: kib (previous version)
Sponsored by: The FreeBSD Foundation

Don't enter DDB for fatal traps before panic by default.

Add a new 'debugger_on_trap' knob separate from 'debugger_on_panic'
and make the calls to kdb_trap() in MD fatal trap handlers prior to
calling panic() conditional on this new knob instead of
'debugger_on_panic'. Disable the new knob by default. Developers who
wish to recover from a fatal fault by adjusting saved register state
and retrying the faulting instruction can still do so by enabling the
new knob. However, for the more common case this makes the user
experience for panics due to a fatal fault match the user experience
for other panics, e.g. 'c' in DDB will generate a crash dump and
reboot the system rather than being stuck in an infinite loop of fatal
fault messages and DDB prompts.

Reviewed by: kib, avg
MFC after: 2 months
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D17768

Clean up handling of unexpected exceptions. Previously we would issue a
breakpoint instruction, however this would lose information that may be
useful for debugging.

These are now handled in a similar way to other exceptions, however it
won't exit out of the exception handler until it is known if we can
handle these exceptions in a useful way.

Sponsored by: DARPA, AFRL

Further limit when we call pmap_fault.

We should only call pmap_fault in the kernel when accessing a userspace
address. As this should always happen through specific functions that set
a fault handler we can use this to limit calls to pmap_fault to when this
is set.

This should help with NULL pointer dereferences when we are unable to sleep
so we fall into the correct case.

Sponsored by: DARPA, AFRL

Restrict the faulting addresses we call pmap_fault from to just those that
may fault due to superpage mappings being changed.

Sponsored by: DARPA, AFRL

followup to r332730/r332752: set kdb_why to "trap" for fatal traps

This change updates arm, arm64 and mips achitectures. Additionally, it
removes redundant checks for kdb_active where it already results in
kdb_reenter() and adds kdb_reenter() calls where they were missing.

Some architectures check the return value of kdb_trap(), but some don't.
I haven't changed any of that.

Some trap handling routines have a return code. I am not sure if I
provided correct ones for returns after kdb_reenter(). kdb_reenter
should never return unless kdb_jmpbufp is NULL for some reason.

Only compile tested for all affected architectures. There can be bugs
resulting from my poor understanding of architecture specific details.

Reported by: jhb
Reviewed by: jhb, eadler
MFC after: 4 weeks
Differential Revision: https://reviews.freebsd.org/D15431

Remove the unused fuwintr() and suiwintr() functions.

Half of implementations always failed (returned (-1)) and they were
previously used in only one place.

Reviewed by: kib, andrew
Obtained from: CheriBSD
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D15102

Fix arm64 buildkernel target with "nooptions KDB"

Make kdb_trap in breakpoint exception handler conditional. If "options KDB"
is not present just panic with message that debugger is not enabled.

PR: 224653

Workaround Spectre Variant 2 on arm64.

We need to handle two cases:

1. One process attacking another process.
2. A process attacking the kernel.

For the first case we clear the branch predictor state on context switch
between different processes. For the second we do this when taking an
instruction abort on a non-userspace address.

To clear the branch predictor state a per-CPU function pointer has been
added. This is set by the new cpu errata code based on if the CPU is
known to be affected.

On Cortex-A57, A72, A73, and A75 we call into the PSCI firmware as newer
versions of this will clear the branch predictor state for us.

It has been reported the ThunderX is unaffected, however the ThunderX2 is
vulnerable. The Qualcomm Falkor core is also affected. As FreeBSD doesn't
yet run on the ThunderX2 or Falkor no workaround is included for these CPUs.

MFC after: 3 days
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D13812

Fix the direction we move the instruction to retuirn to after exiting the
kernel debugger. We should skip the breakpoint instruction, not execute the
instruction before it.

Pointy-hat to: andrew
MFC after: 1 week
Sponsored by: DARPA, AFRL

When we exit the kernel debugger having entered because of a breakpoint
instruction we need to jump over the instruction. Without this we will
execute the same instruction again and enter into the debugger again.

PR: 223917
Reported by: emaste
MFC after: 1 week
Sponsored by: DARPA, AFRL

Make 32-bit system calls end up in svc_handler().

The nice thing about ARM64 is that it's pretty elegant to install
separate trap/exception handlers for 32-bit and 64-bit processes. That
said, for all other architectures (e.g., i386 on amd64) we always let
32-bit counterparts go through the regular system call codepath. Let's
do the same on ARM64.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13146

Add support for handling undefined instructions in userspace and the
kernel. We can register callbacks to perform the required operation on the
saved registers before returning.

This is initially used to work around a bug in old versions of QEMU that
trigger such an exception when reading from an ID register when it should
load z zero value.

I expect this could be used with other exception types, e.g. to emulate
special register access from userland.

Sponsored by: DARPA, AFRL

Pass the last exception trap frame to kdb_trap. This allows show registers
in ddb to show the traps registers, and not the registers from within the
panic call.

Sponsored by: DARPA, AFRL

Move struct syscall_args syscall arguments parameters container into
struct thread.

For all architectures, the syscall trap handlers have to allocate the
structure on the stack. The structure takes 88 bytes on 64bit arches
which is not negligible. Also, it cannot be easily found by other
code, which e.g. caused duplication of some members of the structure
to struct thread already. The change removes td_dbg_sc_code and
td_dbg_sc_nargs which were directly copied from syscall_args.

The structure is put into the copied on fork part of the struct thread
to make the syscall arguments information correct in the child after
fork.

This move will also allow several more uses shortly.

Reviewed by: jhb (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
X-Differential revision: https://reviews.freebsd.org/D11080

Restrict the arm64 supervisor all instructions to only allow a zero
immediate value for system calls. We may wish to use other values in the
future for other purposes.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Push loading curthread into assembly in the synchronous exception handlers.
This will help investigating the performance impact of moving parts of the
switch statement in do_el0_sync into assembly.

Sponsored by: DARPA, AFRL

Rather than checking if the top bit in a virtual address is a 0 or 1
compare against VM_MAXUSER_ADDRESS as we should have been doing.

Sponsored by: DARPA, AFRL

Push reading of ESR_EL1 to assembly. Among other uses this will allow us
to expose this to signal handlers, e.g. for the clang sanitizers.

Sponsored by: DARPA, AFRL

Use nitems to get the correct number of registers to read when dumping
them. Previously this would walk past the end of the array and print
whatever happened to be after the trapframe struct.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Add support for the fpu_kern(9) KPI on arm64. It hooks into the existing
VFP code to store the old context, with lazy loading of the new context
when needed.

FPU_KERN_NOCTX is missing as this is unused in the crypto code this has
been tested with, and I am unsure on the requirements of the UEFI
Runtime Services.

Reviewed by: kib
Obtained from: ABT Systeems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D8276

Add PCB_FP_USERMASK so we can mask off floating point flags that should
not be sent to userspace, for example the future flag to tell when we are
using floating point in the kernel.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Trap msr/mrs instructions. These are privileged arm64 instructions and
shouldn't normally be used.

Obtained from: ABT Systems Ltd
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Don't panic when we don't handle a userland exception, not all we may see
are currently handled.

Obtained from: ABT Systems Ltd
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Also handle instruction traps. We might hit these when the page we are
executing is being promoted to a superpage.

Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Ensure map is valid, even before userland exists and the fault address
register points to an address in the userland range.

Obtained from: ABT Systems Ltd
MFC after: 1 week
Sponsored by: the FreeBSD Foundation

Implement promotions and demotions in the arm64 pmap code. For now we don't
promote memory as I am not sure all the demotion cases are handled, however
it is useful to implement pmap_page_set_memattr. This is used, for example,
when mapping uncached memory for bus_dma(9).

pmap_page_set_memattr needs to demote the DMAP region as on ARM we need to
ensure all mappings to the same physical address have the same attributes.

Reviewed by: kib
Obtained from: ABT Systems Ltd
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6987

Fix a race when the hardware has raised an exception with an unknown or
uncategorised reason. We need to read the fault address register before
enabling interrupts as the interrupt handler may cause this register to
change.

Approved by: re (marius, kib)
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

As <machine/vmparam.h> is included from <vm/vm_param.h>, there is no
need to include it explicitly when <vm/vm_param.h> is already included.

Suggested by: alc
Reviewed by: alc
Differential Revision: https://reviews.freebsd.org/D5379

Handle a misaligned stack pointer exception from userspace. The exception
still needs to be enabled, but this will help with testing.

Sponsored by: ABT Systems Ltd

Implement single stepping on arm64. We need to set the single step bits in
the processor and debug state registers. A flag has been added to the pcb
to tell us when to enable single stepping for a given thread.

Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4730

Print useful information when we hit a data abort we can't handle. This
prints the trap frame, along with the exception syndrome and fault address
registers. Even though esr is 64-bits here it is only 32-bits in hardware
so only print the valid 32-bits.

While here also print esr and far when appropriate after printing the trap
frame.

Sponsored by: ABT Systems Ltd

arm64: remove exception instruction length assertion

From the (now removed) comment:

* It is unclear in some cases if the bit is implementation defined.
* The Foundation Model and QEMU disagree on if the IL bit should
* be set when we are in a data fault from the same EL and the ISV
* bit (bit 24) is also set.

Instead of adding even more special cases just remove the assertion.

Approved by: andrew
Sponsored by: The FreeBSD Foundation

When trying to execute from a misaligned address raise a SIGBUS with the
invalid address alignment code.

Obtained from: EuroBSDCon
Sponsored by: ABT Systems Ltd

When asserting IL bit in exception syndrome register, print the raw
register value.

Reviewed by: andrew
Sponsored by: The FreeBSD Foundation

Do not execute exception handlers with disabled interrupts.

We should not call vm_fault(), or send a signal, with interrupts
disabled. MI kernel code is not prepared for such environment, not to
mention that this increases system latency, since code appears to be
executing as being under spinlock.

The FAR register for data aborts is read before the interrupts are
enabled, to avoid its corruption due to nested exception or context
switch.

Add asserts, similar to the checks done by other architectures, about
not taking page faults in non-sleepable contexts, rather than die with
late and somewhat confusing witness diagnostic.

Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3669

Clear exclusive monitors when handling data aborts, the monitors are
in unknown state per spec.

Reviewed by: andrew (previous version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3668

Do not hold the process around the vm_fault() call from the trap()s.
The only operation which is prevented by the hold is the kernel stack
swapout for the faulted thread, which should be fine to allow.

Remove useless checks for NULL curproc or curproc->p_vmspace from the
trap_pfault() wrappers on x86 and powerpc.

Reviewed by: alc (previous version)
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks

Call into kdb on a data abort when it's enabled. This is needed to
correctly handle trying to access an invalid address in the debugger.

While here document that the breakpoint handler is supposed to fall
through to the following case.

Sponsored by: ABT Systems Ltd

Handle the breakpoint instruction in userland by sending a SIGTRAP signal.

Correct comment typo noted by erik

Remove register dump from arm64 el0 unknown exception

An exception with an unknown reasion is the expected result of the
attempted execution of an instruction bit pattern that has no allocated
instruction.

Sponsored by: The FreeBSD Foundation

arm64: turn unknown el0 exception into a SIGILL

It seems we get EXCP_UNKNOWN from QEMU when executing zeroed memory.
Print a register dump here and signal illegal instruction. Also print
a register dump for other invalid exceptions, before panic.

Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3370

Clean out some externally visible "more then" grammar

MFC after: 3 days

Add missing exception number to EL0 sync. abort on ARM64

When doing a data abort from userland it is possible to get
more than one data abort inside the same exception level.
Add an appropriate exception number to allow nesting of
data_abort handler for EL0.

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3276

Add ARM64TODO markers to unimplemented functionality

Reviewed by: andrew
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D2389

Enable IRQ during syscalls on ARM64

FreeBSD provides a feature called Adaptive Mutexes, which allows
a thread to spin for a while when the mutex is taken instead of
immediately going to sleep. This causes issues when called from
syscall handler if interrupts are masked. If every other core
also attempts to access the same mutex there is a chance that
all of them are spinning on the same lock at the same time.
If interrupts are disabled, no kernel preemtion can occur and
the system becomes unresponsive.

This patch enables interrupts when syscall is being executed
and masks them as soon as it is completed.

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3246

Add logging of synchronous exceptions.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Send the correct signal when vm_fault fails. While here also set the code
and address fields.

Sponsored by: ABT Systems Ltd

First cut of DTrace for AArch64.

Reviewed by: andrew, emaste
Sponsored by: ARM Limited
Differential Revision: https://reviews.freebsd.org/D2738

Use the address of fsu_intr_fault, not the first two instructions in it as
the address to check if we are handling a trap for fuswintr or suswintr.

Sponsored by: The FreeBSD Foundation

Bring in the start of the arm64 kernel.

This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.

The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.

This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.

Differential Revision: https://reviews.freebsd.org/D2199
Reviewed by: emaste, imp
Relnotes: yes
Sponsored by: The FreeBSD Foundation