remove prototypes for removed functions

The feature is called SSBS instead of SBSS.

Expose BTI support to userland.

ok deraadt@

handle /reserved-memory nodes from device trees on arm64.

u-boot is supposed to take these entries and put them in the efi
memory map, but i keep hitting machines where an otherwise functional
u-boot does not do this, resulting in weird errors.

i have an espressobin with a vendor u-boot that has a reserved-memory
region for psci. without this diff the machine faults when the
kernel tries to reboot using a psci handler.

a macchiatobin with an otherwise working u-boot throws SErrors or
panics on weird memory corruption problems without this. i thought
it was bad RAM, but the problems persisted with completely different
ram, and very underclocked and well cooled ram.

riscv64 already has code to handle reserved-memory regions. the
riscv64 change is to add handling for the "no-map" property.

ok kettenis@

Account for nkmempages as well in the pmap_growkernel() call during
initalisation. This way there is enough KVA mapped that kmeminit()
succeeds even with large nkmempages. This is similar to e.g. alpha.
OK miod@ kettenis@

Take advantage of the fact that the WFI instruction does continue
immediately if there is a pending interrupt to fix a potential race
in the idle loop.

ok guenther@

Use the deep idle state available on Apple M1/M2 cores in the idle loop and
for suspend. This state makes the CPU lose some of its register state so
we need to save these registers before putting the core to sleep and
restore them when we wake up. This deep idle state has a higher wakeup
latency than the normal WFI idle state. Use similar logic as acpucpu(4) to
decide which idle state to pick.

If some cores of a cluster are in this deep idle state, turbo states become
available to the cores that remain active. So stop skipping these states.
This improves single-core performance a little bit.

The main win is in power savings when running in a state with a high clock
frequency. My M2 Pro mini goes from 14W to 6.5W when idle at the maximum
clock frequency. But event at the lowest clock frequency there are small
but significant power savings.

ok deraadt@, tobhe@

Implement support for pointer authentication (PAC) in userland. With PAC
it is possible to "sign" pointers with a hidden key. The signature is
placed in unused bits of the pointer and can be checked later. This can
be used to provide "tail CFI" that is similar to what retguard provides.

Debuggers need to be aware of the fact that pointers can be signed. For
this purpose a new PT_PACMASK ptrace(2) request is introduced that returns
as mask that indicates the bits used for the signature. Separate masks
are provided for code and data pointers even though the masks are identical
in the current implementation. These masks are also written into a special
note section in the core dump.

ok patrick@

Enable branch target control flow enforcement on arm64.

ok robert@, deraadt@

Map device tree read/write to unbreak root on softraid

Since r1.76 "Get rid of pmap_map_early()" the FDT was mapped read-only,
but CRYPTO softraid code writes it to zero out the key.

Found and tested by me; explanation and fix from kettenis.
OK kettenis

Set the default for machdep.lidaction to 1 like on amd64.

ok tobhe@, deraadt@

Add arm64 lid_action sysctl for Apple Silicon laptops.

ok kettenis@

Expose the complete set of ID registers as defined in the current version
of ARMv8/ARMv9. Make sure we only expose the features that we know about
and support in our kernel. This matches what Linux does. For now, mostly
restrict ourselves to features defined in ARMv8.5 which means that we only
actually implement support for ID_AA64ISAR0_EL1, ID_AA64ISAR1_EL1,
ID_AA64PFR0_EL1 and ID_AA64PFR1_EL1. For the other registers we simply
always return 0.

ok deraadt@

Get rid of pmap_map_early(). This is part of my crusade against the use of
1G mappings for the PA = VA identity mapping used in the early boot phase
of the kernel and when spinning up CPUs. The mappings are dangerous since
they might (unintentially) covering address ranges that should not be
mapped (i.e. secure memory) which is dangerous on arm64 since the
architecture allows speculative access to any address for which a valid
mapping exists and even speculative access may cause the machine to
misbehave.

So instead of relying on the PA = VA identity mapping, call pmap_bootstrap()
earlier such that we can use pmap_kenter_cache() to enter mappings for the
FDT.

ok miod@

Simplfity setregs() by passing it the ps_strings and switching
sys_execve() to return EJUSTRETURN.

setregs() is the MD routine used by sys_execve() to set up the
thread's trapframe and PCB such that, on 'return' to userspace, it
has the register values defined by the ABI and otherwise zero. It
had to set the syscall retval[] values previously because the normal
syscall return path overwrites a couple registers with the retval[]
values. By instead returning EJUSTRETURN that and some complexity
with program-counter handling on m88k and sparc64 goes away.

Also, give setregs() add a 'struct ps_strings *arginfo' argument
so powerpc, powerpc64, and sh can directly get argc/argv/envp
values for registers instead of copyin()ing the one in userspace.

Improvements from miod@ and millert@
Testing assistance miod@, kettenis@, and aoyama@
ok miod@ kettenis@

On CPUs that support the Data Independent Timing feature, enable this feature
by default in both the kernel and userland. At this point, this feature has
only been seen on Apple's CPU cores, where turning it on has no measurable
impact on performance. Turning this feature on should help to mitigate
timing side-channel attacks.

ok deraadt@, beck@

Reorganize the EFI code a bit. Move the efi.h header from dev/acpi to
dev/efi and rename the arm64 efi.c to efi_machdep.c, preparing the way
for MI EFI code and an amd64 implementation of EFI runtime support.

ok deraadt@, mlarkin@

Don't ignore memory blocks smaller than 64KB. Some EFI implementations
(such as the one on the x13s) allocate memory with the EfiBootSevicesData
type in a semi-random fashion. Ignoring small regions with that type
results in different memory maps between boots of the same kernel.
This causes problems with upcoming hibernate support.

The decision to ignore small regions was made because we do this on
amd64 to work arounmd broken BIOS implementations and because of fears
that we would run out of physical memory segments in UVM. We have some
reasons to believe that we can trust the EFI memory map on arm64 and the
risk of running out of physical memory segments is mitigated by the fact
that we try to merge memory regions before loading them into UVM. If
for some reason we see a significant drop in physical memory on certain
machines, we should probably increase the number of items in the array
we use to store memory regions and/or increase the number of physical
memory segments in UVM.

ok mlarkin@

Put printing the EFI memory map behind and #ifdef.

ok deraadt@

Postpone calling the EFI SetVirtualAddressMap() interface until efi(4)
attaches. This prevents us from having to map the various EFI memory regions
early on. These early mappings use 1G blocks. On Apple M1 systems, these
blocks may overlap memory that isn't accessable. The CPU may speculatively
access this inaccessable memory which will result in SError exceptions.

ok deraadt@

Export the ID_AA64ISARn_EL1 registers to userspace through sysctl(2) such
that we can detect which instruction set extensions are supported without
relying in catching SIGILL.

ok deraadt@

Enable cduart(4) on arm64.

OK kettenis@

Rewrite the kernel FPU handling code. The new code saves the FPU state
in cpu_switch() instead of at the kernel edge and gets rid of the FPU
state tracking in struct cpu_info and struct pcb. This fixes the random
crashes seen with SMP kernels on Apple M1.

ok patrick@

zero correct var in cpu_dump()
ok guenther@ deraadt@

Avoid local variable definitions in the middle of code.

Make memreg_add() a bit smarter and have it merge adjacent regions.
This should reduce the number of physical memory segments that we pass
to uvm reducing the chance that we run out of physical memory segments
and should make physical memory allocation a bit more efficient.

ok patrick@

Build a list of memory regions and call yvm_physload(9) on those like
we do on riscv64.

ok patrick@

Initialize per-CPU pointer register earlier.

ok patrick@

Add missing memory clobbers to "data" barriers.

We can use memory marked as EfiBootServicesCode or EfiBootServicesData
as well.

ok drahn@, kn@

spelling

Introduce BUS_SPACE_MAP_POSTED such that we can distinguish between
posted and non-posted device memory mappings and set the right memory
attributes for them. Needed because on the Apple M1 using the wrong
mapping will fault.

ok patrick@, dlg@

Call exuart(4) early attach on arm64.

ok kettenis@

Give machdep.c a thorough cleanup that is long overdue.

ok patrick@

Clean up arm64's autoconf code. There have been quite a few unused
variables and includes. While there get rid of the boot_file support
and (void) casts.

ok kettenis@

Include <sys/systm.h> directly instead of relying on hidden UVM includes.

The header is being pulled via db_machdep.h -> uvm_extern.h -> uvm_map.h

Implement cpu_rnd_messybits() as a read of the virtual counter xored
with a bit-reversed copy of itself. There is progressively less
entropy in the higher bits of a counter than in the lower bits, so
bit-reverse one half in order to extract maximal entropy.

style fixes and ok kettenis@

introduce "cpu_rnd_messybits" for use instead of nanotime in dev/rnd.c.

rnd.c uses nanotime to get access to some bits that change quickly
between events that it can mix into the entropy pool. it doesn't
use nanotime to get a monotonically increasing set or ordered and
accurate timestamps, it just wants something with bits that change.

there's been discussions for years about letting rnd use a clock
that's super fast to read, but not necessarily accurate, but it
wasn't until recently that i figured out it wasn't interested in
time at all, so things like keeping a fast clock coherent between
cpu cores or correct according to ntp is unecessary. this means we
can just let rnd read the cycle counters on cpus and things will
be fine. cpus with cycle counters that vary in their speed and
arent kept consistent between cores may even be desirable in this
context.

so this is the first step in converting rnd.c to reading cycle
counter. it copies the nanotime backend to each arch, and they can
replace it with something MD as a second step later on.

djm@ suggested rnd_messybytes, but we landed on cpu_rnd_messybits.
thanks to visa for his eyes.
ok deraadt@ visa@
deraadt@ says he will help handle any MD fallout that occurs.

Add a little bit of const-ness.

Look at the openbsd,boothowto property.

ok visa@

Add machdep.compatible.

ok jsg@

Make inittodr() and resettodr() MI.

ok deraadt@, mpi@, visa@
ok cheloha@ as well (would have preferred in new file for this code)

Initialize the timeval passed to todr_gettime() with the base time from
the file system such that implementations can use it to guess the right
century.

ok mpi@

Move inittodr() and resettodr() to the end of the file to easy move an MI
implementation.

ok mpi@

Sanitize inittodr()/resettodr() implementation.
- move implementations next to eachother in the same file
- remove pointless call to resettodr() in inittodr()
- use OpenBSD define to get a minimum plausible time
- if RTC time is before minimum plausible time, reject it
- don't print "clock gained N days" on ramdisk

Hopefully this implementation can serve as a model for unification
into an MI implementation.

ok mpi@

Set DMA constraint based on the openbsd,dma-constraint property if
present. Make sure bus_dmamap_alloc(9) allocates memory that meets that
constraint.

ok patrick@

Add amluart(4), a driver for the UARTs found on various Amlogic SoCs.

ok jsg@

Make sure the virtual addresses that are used for EFI runtime services
fit into the 39-bit address space that we currently use for the runtime
mappings.

ok patrick@

On machines with large amounts of physical memory we fail to initialize uvm
because we don't have enough kernel memory available in the early bootstrap
phase to allocate the vm_page structures. Fix this by making uvm_growkernel()
work before uvm is initialized like we do on other architectures that don't
use a direct map and explicitly call it after enumerating the available
physical memory with an estimate of how much KVA we need to initialize uvm.

ok patrick@

On bigger arm64 machines with plenty of memory it is possible that
the kernel, which is allocated in low memory by efiboot, is not in
the same 512 GiB L0 address space as the EFI/FDT/ACPI tables. Since
we only identity-map a single L0 entry for bootstrapping purposes we
can't reach those tables. Keep a set of spare L1 tables, currently
one, which we can dynamically enter into the identity map so that we
are able to reach those tables. While there, do the FDT mapping a
bit later in C code by calling pmap_map_early(). This allows us to
boot further on Packet's Ampere eMAGs.

Discussed with drahn@
ok kettenis@

Add mvuart(4) to support the Armada 3720's serial console.

ok kettenis@

fast track ddb> reboot command to skip anything which might panic again.
ok deraadt

Implement booting from softraid on arm64. This consists of creating a
list of available block devices, so we can also boot from another block
device than the one that efiboot was loaded from. Softraid will then
create a list of volumes by checking the partition table for RAID and
matching softraid metadata. If efiboot was loaded from the same
physical drive as a softraid volume, it will use the soft- raid volume
as boot device and it will ask you to unlock it. The UUID and key of
that boot volume will be passed as FDT properties to the kernel. Those
FDT properties will be zeroed explicitly by the kernel after they have
been passed to the softraid stack.

ok kettenis@

It is possible for a memory write to not complete if no barrier exists
between the write and a wfi instruction. Make certain system state
is coherent when waiting on interrupts.
ok kettenis@

Remove comcnspeed variable, which gets set but is never actually used.
This removes the (no-op) -1 and -9 kernel boot arguments.

ok visa@

Cleanup bootconfig.h and use strlcpy() to copy boot arguments.

ok drahn@

Remove a4x bus space hack.

ok patrick@

Use speed from device tree for serial console. Makes it possible to boot
and install systems where the firmware uses a non-standard speed. This is
important for various boards that use a Rockchip SoC where the vendor thinks
using 1500000 bps as the serial console speed is a good idea (it isn't).

ok deraadt@, visa@, patrick@

Call imxuart(4) early attach on arm64.

Remove bogus copying of bootargs.

From Arrtturi Alm.

Allow the kernel to recognize that it has been netbooted and to add the
boot interface to the "netboot" group. efiboot grabs the MAC address
from the PXE environment, passes it to the kernel, where it is matched
against the list of ethernet interfaces and the boot device is set.
Concept and most of the code cribbed from amd64.
ok kettenis@

Make sure each CPU uses its own set of VAs for pmap_zero_page() and
pmap_copy_page().

ok patrick@

Spin up secondary CPUs and add IPI infrastructure. Heavily based on diffs
from dran@.

ok patrick@

Runtime services may (and do) use device mappings on some UEFI implementations.
Skip these mappings during the remap-pahse as they are likely to be in
a different 512G bloch as memory and SetVirtualAddressMap() shouldn't need
them. But do assign a new virtual address and let efi(4) create a mapping.
Add a PMAP_DEVICE flag such that pmap_enter() can continue to be used
to create these mappings.

ok patrick@

Implement FUCKWIT for arm64; unmap the kernel almost entirely while userland
is running. This provides protection against meltown on cores that are
vilnerable (just Cortex-A75 so far) but also seems to be an essential to
protect against spectre-like attacks against the kernel.

This implementation only exposes a single treampoline page that does not
contain any kernel virtual addresses and also hides the real virtual address
of the exception vectors, which helps on cores vulnerable to "variant 3a"
(Cortex-A57, Cortex-A72). The implementation is inspired by the work done
by Will Deacon for Linux, but there are no knobs to turn it off. The
overhead is fairly limited: around 3-4% slowdown on Cortex-A57.

ok patrick@, deraadt@

Implement support for calling EFI runtime services and use it to implement
a time-of-day clock device based on the GetTime() and SetTime() services.
The virtual memory mappings for the runtime services calls are implemented
through a separate pmap that is only activated when we make a runtime services
call.

ok tom@, visa@
tested by naddy@

In uvm Chuck decided backing store would not be allocated proactively
for blocks re-fetchable from the filesystem. However at reboot time,
filesystems are unmounted, and since processes lack backing store they
are killed. Since the scheduler is still running, in some cases init is
killed... which drops us to ddb [noted by bluhm]. Solution is to convert
filesystems to read-only [proposed by kettenis]. The tale follows:
sys_reboot() should pass proc * to MD boot() to vfs_shutdown() which
completes current IO with vfs_busy VB_WRITE|VB_WAIT, then calls VFS_MOUNT()
with MNT_UPDATE | MNT_RDONLY, soon teaching us that *fs_mount() calls a
copyin() late... so store the sizes in vfsconflist[] and move the copyin()
to sys_mount()... and notice nfs_mount copyin() is size-variant, so kill
legacy struct nfs_args3. Next we learn ffs_mount()'s MNT_UPDATE code is
sharp and rusty especially wrt softdep, so fix some bugs adn add
~MNT_SOFTDEP to the downgrade. Some vnodes need a little more help,
so tie them to &dead_vnops.

ffs_mount calling DIOCCACHESYNC is causing a bit of grief still but
this issue is seperate and will be dealt with in time.
couple hundred reboots by bluhm and myself, advice from guenther and
others at the hut

If you use sys/param.h, you don't need sys/types.h

Add glass console support for arm64. This uses the "stdout-path" property
of the /chosen node in the device tree to decide whether the framebuffer
should be used as the console device. Most, if not all, machines will
have that set to use a serial console and there is no easy way yet to
change that.

ok jsg@

Clear second return register on exec.

ok guenther@

fix format strings
ok kettenis@

Use EFI memory map to discover available physical memory.
Fall back on improved code that parses the FDT /memory node.
However, on machines with "real" UEFI firmware, /memory may include
reserved memory that we shouldn't use (it does on the Overdrive 1000).
On those machines, make sure that you use BOOTAA64 0.3 or later such that
the EFI memory map gets used.

For now, print out the EFI memory map to help debug any issues.

ok patrick@

Bring boot() in line with our other architectures.

ok mpi@, deraadt@

Zap silly diagnostic reboot message.

ok bmercer@, deraadt@

Rename Debugger() into db_enter().

Using a name with the 'db_' prefix makes it invisible from the dynamic
profiler.

ok deraadt@, kettenis@, visa@

store cpu model information in the buffer used by the hw.model sysctl
ok kettenis@

Revise 'struct fpreg' and dump floating-point register in core dumps. Also
reset the floating-point register state upon exec.

ok guenther@

Don't limit physmem to 2GB confirmed to work with 16GB by deraadt@.

Fix up and use the device memory attribute for device mappings. Add
inner shareable attributes while there.

Move cache and tlb flush functions, which were mostly inline assembly,
into separate functions. This makes them reusable from other parts in
the kernel. Assembly and header are taken from FreeBSD, but modified
to fit our requirements and with some unnecessary stuff removed. While
there remove micro optimization for uniprocessor kernels.

A bit of code and comment cleanup.

Remove gdb waitcount debug code and dummy print.

Enter UKC when built with BOOT_CONFIG and -c is specified.

ok kettenis@ patrick@

Initialize DDB if compiled with option DDB.

ok kettenis@

Exterminate comcnspeed and comcnmode which were uninitialized and
caused issues in the tty layer. Use the tty defaults for the cflags,
while still overwriting the terminal speed with B115200.

Found with lots of help from deraadt@

Make sure to also backup and restore the a4x bus space map function
so that we can initialize the console early.

Ansify cpu_sysctl() on alpha, arm, arm64, luna88k and sparc64.

OK mpi@

fix uninitialised variable warnings from clang
ok patrick@

Import of OpenBSD/arm64

This commit contains all the kernel files related to the OpenBSD/arm64
port. It is based on the PowerPC pmap, loongson, arm/armv7 code and
FreeBSD aarch64 code. Hard work done by Dale Rahn.

The feature is called SSBS instead of SBSS.

Expose BTI support to userland.

ok deraadt@

handle /reserved-memory nodes from device trees on arm64.

u-boot is supposed to take these entries and put them in the efi
memory map, but i keep hitting machines where an otherwise functional
u-boot does not do this, resulting in weird errors.

i have an espressobin with a vendor u-boot that has a reserved-memory
region for psci. without this diff the machine faults when the
kernel tries to reboot using a psci handler.

a macchiatobin with an otherwise working u-boot throws SErrors or
panics on weird memory corruption problems without this. i thought
it was bad RAM, but the problems persisted with completely different
ram, and very underclocked and well cooled ram.

riscv64 already has code to handle reserved-memory regions. the
riscv64 change is to add handling for the "no-map" property.

ok kettenis@

Account for nkmempages as well in the pmap_growkernel() call during
initalisation. This way there is enough KVA mapped that kmeminit()
succeeds even with large nkmempages. This is similar to e.g. alpha.
OK miod@ kettenis@

Take advantage of the fact that the WFI instruction does continue
immediately if there is a pending interrupt to fix a potential race
in the idle loop.

ok guenther@

Use the deep idle state available on Apple M1/M2 cores in the idle loop and
for suspend. This state makes the CPU lose some of its register state so
we need to save these registers before putting the core to sleep and
restore them when we wake up. This deep idle state has a higher wakeup
latency than the normal WFI idle state. Use similar logic as acpucpu(4) to
decide which idle state to pick.

If some cores of a cluster are in this deep idle state, turbo states become
available to the cores that remain active. So stop skipping these states.
This improves single-core performance a little bit.

The main win is in power savings when running in a state with a high clock
frequency. My M2 Pro mini goes from 14W to 6.5W when idle at the maximum
clock frequency. But event at the lowest clock frequency there are small
but significant power savings.

ok deraadt@, tobhe@

Implement support for pointer authentication (PAC) in userland. With PAC
it is possible to "sign" pointers with a hidden key. The signature is
placed in unused bits of the pointer and can be checked later. This can
be used to provide "tail CFI" that is similar to what retguard provides.

Debuggers need to be aware of the fact that pointers can be signed. For
this purpose a new PT_PACMASK ptrace(2) request is introduced that returns
as mask that indicates the bits used for the signature. Separate masks
are provided for code and data pointers even though the masks are identical
in the current implementation. These masks are also written into a special
note section in the core dump.

ok patrick@

Enable branch target control flow enforcement on arm64.

ok robert@, deraadt@

Map device tree read/write to unbreak root on softraid

Since r1.76 "Get rid of pmap_map_early()" the FDT was mapped read-only,
but CRYPTO softraid code writes it to zero out the key.

Found and tested by me; explanation and fix from kettenis.
OK kettenis

Set the default for machdep.lidaction to 1 like on amd64.

ok tobhe@, deraadt@

Add arm64 lid_action sysctl for Apple Silicon laptops.

ok kettenis@

Expose the complete set of ID registers as defined in the current version
of ARMv8/ARMv9. Make sure we only expose the features that we know about
and support in our kernel. This matches what Linux does. For now, mostly
restrict ourselves to features defined in ARMv8.5 which means that we only
actually implement support for ID_AA64ISAR0_EL1, ID_AA64ISAR1_EL1,
ID_AA64PFR0_EL1 and ID_AA64PFR1_EL1. For the other registers we simply
always return 0.

ok deraadt@

Get rid of pmap_map_early(). This is part of my crusade against the use of
1G mappings for the PA = VA identity mapping used in the early boot phase
of the kernel and when spinning up CPUs. The mappings are dangerous since
they might (unintentially) covering address ranges that should not be
mapped (i.e. secure memory) which is dangerous on arm64 since the
architecture allows speculative access to any address for which a valid
mapping exists and even speculative access may cause the machine to
misbehave.

So instead of relying on the PA = VA identity mapping, call pmap_bootstrap()
earlier such that we can use pmap_kenter_cache() to enter mappings for the
FDT.

ok miod@

Simplfity setregs() by passing it the ps_strings and switching
sys_execve() to return EJUSTRETURN.

setregs() is the MD routine used by sys_execve() to set up the
thread's trapframe and PCB such that, on 'return' to userspace, it
has the register values defined by the ABI and otherwise zero. It
had to set the syscall retval[] values previously because the normal
syscall return path overwrites a couple registers with the retval[]
values. By instead returning EJUSTRETURN that and some complexity
with program-counter handling on m88k and sparc64 goes away.

Also, give setregs() add a 'struct ps_strings *arginfo' argument
so powerpc, powerpc64, and sh can directly get argc/argv/envp
values for registers instead of copyin()ing the one in userspace.

Improvements from miod@ and millert@
Testing assistance miod@, kettenis@, and aoyama@
ok miod@ kettenis@

On CPUs that support the Data Independent Timing feature, enable this feature
by default in both the kernel and userland. At this point, this feature has
only been seen on Apple's CPU cores, where turning it on has no measurable
impact on performance. Turning this feature on should help to mitigate
timing side-channel attacks.

ok deraadt@, beck@

Reorganize the EFI code a bit. Move the efi.h header from dev/acpi to
dev/efi and rename the arm64 efi.c to efi_machdep.c, preparing the way
for MI EFI code and an amd64 implementation of EFI runtime support.

ok deraadt@, mlarkin@

Don't ignore memory blocks smaller than 64KB. Some EFI implementations
(such as the one on the x13s) allocate memory with the EfiBootSevicesData
type in a semi-random fashion. Ignoring small regions with that type
results in different memory maps between boots of the same kernel.
This causes problems with upcoming hibernate support.

The decision to ignore small regions was made because we do this on
amd64 to work arounmd broken BIOS implementations and because of fears
that we would run out of physical memory segments in UVM. We have some
reasons to believe that we can trust the EFI memory map on arm64 and the
risk of running out of physical memory segments is mitigated by the fact
that we try to merge memory regions before loading them into UVM. If
for some reason we see a significant drop in physical memory on certain
machines, we should probably increase the number of items in the array
we use to store memory regions and/or increase the number of physical
memory segments in UVM.

ok mlarkin@

Put printing the EFI memory map behind and #ifdef.

ok deraadt@

Postpone calling the EFI SetVirtualAddressMap() interface until efi(4)
attaches. This prevents us from having to map the various EFI memory regions
early on. These early mappings use 1G blocks. On Apple M1 systems, these
blocks may overlap memory that isn't accessable. The CPU may speculatively
access this inaccessable memory which will result in SError exceptions.

ok deraadt@

Export the ID_AA64ISARn_EL1 registers to userspace through sysctl(2) such
that we can detect which instruction set extensions are supported without
relying in catching SIGILL.

ok deraadt@

Enable cduart(4) on arm64.

OK kettenis@

Rewrite the kernel FPU handling code. The new code saves the FPU state
in cpu_switch() instead of at the kernel edge and gets rid of the FPU
state tracking in struct cpu_info and struct pcb. This fixes the random
crashes seen with SMP kernels on Apple M1.

ok patrick@

zero correct var in cpu_dump()
ok guenther@ deraadt@

Avoid local variable definitions in the middle of code.

Make memreg_add() a bit smarter and have it merge adjacent regions.
This should reduce the number of physical memory segments that we pass
to uvm reducing the chance that we run out of physical memory segments
and should make physical memory allocation a bit more efficient.

ok patrick@

Build a list of memory regions and call yvm_physload(9) on those like
we do on riscv64.

ok patrick@

Initialize per-CPU pointer register earlier.

ok patrick@

Add missing memory clobbers to "data" barriers.

We can use memory marked as EfiBootServicesCode or EfiBootServicesData
as well.

ok drahn@, kn@

spelling

Introduce BUS_SPACE_MAP_POSTED such that we can distinguish between
posted and non-posted device memory mappings and set the right memory
attributes for them. Needed because on the Apple M1 using the wrong
mapping will fault.

ok patrick@, dlg@

Call exuart(4) early attach on arm64.

ok kettenis@

Give machdep.c a thorough cleanup that is long overdue.

ok patrick@

Clean up arm64's autoconf code. There have been quite a few unused
variables and includes. While there get rid of the boot_file support
and (void) casts.

ok kettenis@

Include <sys/systm.h> directly instead of relying on hidden UVM includes.

The header is being pulled via db_machdep.h -> uvm_extern.h -> uvm_map.h

Implement cpu_rnd_messybits() as a read of the virtual counter xored
with a bit-reversed copy of itself. There is progressively less
entropy in the higher bits of a counter than in the lower bits, so
bit-reverse one half in order to extract maximal entropy.

style fixes and ok kettenis@

introduce "cpu_rnd_messybits" for use instead of nanotime in dev/rnd.c.

rnd.c uses nanotime to get access to some bits that change quickly
between events that it can mix into the entropy pool. it doesn't
use nanotime to get a monotonically increasing set or ordered and
accurate timestamps, it just wants something with bits that change.

there's been discussions for years about letting rnd use a clock
that's super fast to read, but not necessarily accurate, but it
wasn't until recently that i figured out it wasn't interested in
time at all, so things like keeping a fast clock coherent between
cpu cores or correct according to ntp is unecessary. this means we
can just let rnd read the cycle counters on cpus and things will
be fine. cpus with cycle counters that vary in their speed and
arent kept consistent between cores may even be desirable in this
context.

so this is the first step in converting rnd.c to reading cycle
counter. it copies the nanotime backend to each arch, and they can
replace it with something MD as a second step later on.

djm@ suggested rnd_messybytes, but we landed on cpu_rnd_messybits.
thanks to visa for his eyes.
ok deraadt@ visa@
deraadt@ says he will help handle any MD fallout that occurs.

Add a little bit of const-ness.

Look at the openbsd,boothowto property.

ok visa@

Add machdep.compatible.

ok jsg@

Make inittodr() and resettodr() MI.

ok deraadt@, mpi@, visa@
ok cheloha@ as well (would have preferred in new file for this code)

Initialize the timeval passed to todr_gettime() with the base time from
the file system such that implementations can use it to guess the right
century.

ok mpi@

Move inittodr() and resettodr() to the end of the file to easy move an MI
implementation.

ok mpi@

Sanitize inittodr()/resettodr() implementation.
- move implementations next to eachother in the same file
- remove pointless call to resettodr() in inittodr()
- use OpenBSD define to get a minimum plausible time
- if RTC time is before minimum plausible time, reject it
- don't print "clock gained N days" on ramdisk

Hopefully this implementation can serve as a model for unification
into an MI implementation.

ok mpi@

Set DMA constraint based on the openbsd,dma-constraint property if
present. Make sure bus_dmamap_alloc(9) allocates memory that meets that
constraint.

ok patrick@

Add amluart(4), a driver for the UARTs found on various Amlogic SoCs.

ok jsg@

Make sure the virtual addresses that are used for EFI runtime services
fit into the 39-bit address space that we currently use for the runtime
mappings.

ok patrick@

On machines with large amounts of physical memory we fail to initialize uvm
because we don't have enough kernel memory available in the early bootstrap
phase to allocate the vm_page structures. Fix this by making uvm_growkernel()
work before uvm is initialized like we do on other architectures that don't
use a direct map and explicitly call it after enumerating the available
physical memory with an estimate of how much KVA we need to initialize uvm.

ok patrick@

On bigger arm64 machines with plenty of memory it is possible that
the kernel, which is allocated in low memory by efiboot, is not in
the same 512 GiB L0 address space as the EFI/FDT/ACPI tables. Since
we only identity-map a single L0 entry for bootstrapping purposes we
can't reach those tables. Keep a set of spare L1 tables, currently
one, which we can dynamically enter into the identity map so that we
are able to reach those tables. While there, do the FDT mapping a
bit later in C code by calling pmap_map_early(). This allows us to
boot further on Packet's Ampere eMAGs.

Discussed with drahn@
ok kettenis@

Add mvuart(4) to support the Armada 3720's serial console.

ok kettenis@

fast track ddb> reboot command to skip anything which might panic again.
ok deraadt

Implement booting from softraid on arm64. This consists of creating a
list of available block devices, so we can also boot from another block
device than the one that efiboot was loaded from. Softraid will then
create a list of volumes by checking the partition table for RAID and
matching softraid metadata. If efiboot was loaded from the same
physical drive as a softraid volume, it will use the soft- raid volume
as boot device and it will ask you to unlock it. The UUID and key of
that boot volume will be passed as FDT properties to the kernel. Those
FDT properties will be zeroed explicitly by the kernel after they have
been passed to the softraid stack.

ok kettenis@

It is possible for a memory write to not complete if no barrier exists
between the write and a wfi instruction. Make certain system state
is coherent when waiting on interrupts.
ok kettenis@

Remove comcnspeed variable, which gets set but is never actually used.
This removes the (no-op) -1 and -9 kernel boot arguments.

ok visa@

Cleanup bootconfig.h and use strlcpy() to copy boot arguments.

ok drahn@

Remove a4x bus space hack.

ok patrick@

Use speed from device tree for serial console. Makes it possible to boot
and install systems where the firmware uses a non-standard speed. This is
important for various boards that use a Rockchip SoC where the vendor thinks
using 1500000 bps as the serial console speed is a good idea (it isn't).

ok deraadt@, visa@, patrick@

Call imxuart(4) early attach on arm64.

Remove bogus copying of bootargs.

From Arrtturi Alm.

Allow the kernel to recognize that it has been netbooted and to add the
boot interface to the "netboot" group. efiboot grabs the MAC address
from the PXE environment, passes it to the kernel, where it is matched
against the list of ethernet interfaces and the boot device is set.
Concept and most of the code cribbed from amd64.
ok kettenis@

Make sure each CPU uses its own set of VAs for pmap_zero_page() and
pmap_copy_page().

ok patrick@

Spin up secondary CPUs and add IPI infrastructure. Heavily based on diffs
from dran@.

ok patrick@

Runtime services may (and do) use device mappings on some UEFI implementations.
Skip these mappings during the remap-pahse as they are likely to be in
a different 512G bloch as memory and SetVirtualAddressMap() shouldn't need
them. But do assign a new virtual address and let efi(4) create a mapping.
Add a PMAP_DEVICE flag such that pmap_enter() can continue to be used
to create these mappings.

ok patrick@

Implement FUCKWIT for arm64; unmap the kernel almost entirely while userland
is running. This provides protection against meltown on cores that are
vilnerable (just Cortex-A75 so far) but also seems to be an essential to
protect against spectre-like attacks against the kernel.

This implementation only exposes a single treampoline page that does not
contain any kernel virtual addresses and also hides the real virtual address
of the exception vectors, which helps on cores vulnerable to "variant 3a"
(Cortex-A57, Cortex-A72). The implementation is inspired by the work done
by Will Deacon for Linux, but there are no knobs to turn it off. The
overhead is fairly limited: around 3-4% slowdown on Cortex-A57.

ok patrick@, deraadt@

Implement support for calling EFI runtime services and use it to implement
a time-of-day clock device based on the GetTime() and SetTime() services.
The virtual memory mappings for the runtime services calls are implemented
through a separate pmap that is only activated when we make a runtime services
call.

ok tom@, visa@
tested by naddy@

In uvm Chuck decided backing store would not be allocated proactively
for blocks re-fetchable from the filesystem. However at reboot time,
filesystems are unmounted, and since processes lack backing store they
are killed. Since the scheduler is still running, in some cases init is
killed... which drops us to ddb [noted by bluhm]. Solution is to convert
filesystems to read-only [proposed by kettenis]. The tale follows:
sys_reboot() should pass proc * to MD boot() to vfs_shutdown() which
completes current IO with vfs_busy VB_WRITE|VB_WAIT, then calls VFS_MOUNT()
with MNT_UPDATE | MNT_RDONLY, soon teaching us that *fs_mount() calls a
copyin() late... so store the sizes in vfsconflist[] and move the copyin()
to sys_mount()... and notice nfs_mount copyin() is size-variant, so kill
legacy struct nfs_args3. Next we learn ffs_mount()'s MNT_UPDATE code is
sharp and rusty especially wrt softdep, so fix some bugs adn add
~MNT_SOFTDEP to the downgrade. Some vnodes need a little more help,
so tie them to &dead_vnops.

ffs_mount calling DIOCCACHESYNC is causing a bit of grief still but
this issue is seperate and will be dealt with in time.
couple hundred reboots by bluhm and myself, advice from guenther and
others at the hut

If you use sys/param.h, you don't need sys/types.h

Add glass console support for arm64. This uses the "stdout-path" property
of the /chosen node in the device tree to decide whether the framebuffer
should be used as the console device. Most, if not all, machines will
have that set to use a serial console and there is no easy way yet to
change that.

ok jsg@

Clear second return register on exec.

ok guenther@

fix format strings
ok kettenis@

Use EFI memory map to discover available physical memory.
Fall back on improved code that parses the FDT /memory node.
However, on machines with "real" UEFI firmware, /memory may include
reserved memory that we shouldn't use (it does on the Overdrive 1000).
On those machines, make sure that you use BOOTAA64 0.3 or later such that
the EFI memory map gets used.

For now, print out the EFI memory map to help debug any issues.

ok patrick@

Bring boot() in line with our other architectures.

ok mpi@, deraadt@

Zap silly diagnostic reboot message.

ok bmercer@, deraadt@

Rename Debugger() into db_enter().

Using a name with the 'db_' prefix makes it invisible from the dynamic
profiler.

ok deraadt@, kettenis@, visa@

store cpu model information in the buffer used by the hw.model sysctl
ok kettenis@

Revise 'struct fpreg' and dump floating-point register in core dumps. Also
reset the floating-point register state upon exec.

ok guenther@

Don't limit physmem to 2GB confirmed to work with 16GB by deraadt@.

Fix up and use the device memory attribute for device mappings. Add
inner shareable attributes while there.

Move cache and tlb flush functions, which were mostly inline assembly,
into separate functions. This makes them reusable from other parts in
the kernel. Assembly and header are taken from FreeBSD, but modified
to fit our requirements and with some unnecessary stuff removed. While
there remove micro optimization for uniprocessor kernels.

A bit of code and comment cleanup.

Remove gdb waitcount debug code and dummy print.

Enter UKC when built with BOOT_CONFIG and -c is specified.

ok kettenis@ patrick@

Initialize DDB if compiled with option DDB.

ok kettenis@

Exterminate comcnspeed and comcnmode which were uninitialized and
caused issues in the tty layer. Use the tty defaults for the cflags,
while still overwriting the terminal speed with B115200.

Found with lots of help from deraadt@

Make sure to also backup and restore the a4x bus space map function
so that we can initialize the console early.

Ansify cpu_sysctl() on alpha, arm, arm64, luna88k and sparc64.

OK mpi@

fix uninitialised variable warnings from clang
ok patrick@

Import of OpenBSD/arm64

This commit contains all the kernel files related to the OpenBSD/arm64
port. It is based on the PowerPC pmap, loongson, arm/armv7 code and
FreeBSD aarch64 code. Hard work done by Dale Rahn.

Expose BTI support to userland.

ok deraadt@

handle /reserved-memory nodes from device trees on arm64.

u-boot is supposed to take these entries and put them in the efi
memory map, but i keep hitting machines where an otherwise functional
u-boot does not do this, resulting in weird errors.

i have an espressobin with a vendor u-boot that has a reserved-memory
region for psci. without this diff the machine faults when the
kernel tries to reboot using a psci handler.

a macchiatobin with an otherwise working u-boot throws SErrors or
panics on weird memory corruption problems without this. i thought
it was bad RAM, but the problems persisted with completely different
ram, and very underclocked and well cooled ram.

riscv64 already has code to handle reserved-memory regions. the
riscv64 change is to add handling for the "no-map" property.

ok kettenis@

Account for nkmempages as well in the pmap_growkernel() call during
initalisation. This way there is enough KVA mapped that kmeminit()
succeeds even with large nkmempages. This is similar to e.g. alpha.
OK miod@ kettenis@

Take advantage of the fact that the WFI instruction does continue
immediately if there is a pending interrupt to fix a potential race
in the idle loop.

ok guenther@

Use the deep idle state available on Apple M1/M2 cores in the idle loop and
for suspend. This state makes the CPU lose some of its register state so
we need to save these registers before putting the core to sleep and
restore them when we wake up. This deep idle state has a higher wakeup
latency than the normal WFI idle state. Use similar logic as acpucpu(4) to
decide which idle state to pick.

If some cores of a cluster are in this deep idle state, turbo states become
available to the cores that remain active. So stop skipping these states.
This improves single-core performance a little bit.

The main win is in power savings when running in a state with a high clock
frequency. My M2 Pro mini goes from 14W to 6.5W when idle at the maximum
clock frequency. But event at the lowest clock frequency there are small
but significant power savings.

ok deraadt@, tobhe@

Implement support for pointer authentication (PAC) in userland. With PAC
it is possible to "sign" pointers with a hidden key. The signature is
placed in unused bits of the pointer and can be checked later. This can
be used to provide "tail CFI" that is similar to what retguard provides.

Debuggers need to be aware of the fact that pointers can be signed. For
this purpose a new PT_PACMASK ptrace(2) request is introduced that returns
as mask that indicates the bits used for the signature. Separate masks
are provided for code and data pointers even though the masks are identical
in the current implementation. These masks are also written into a special
note section in the core dump.

ok patrick@

Enable branch target control flow enforcement on arm64.

ok robert@, deraadt@

Map device tree read/write to unbreak root on softraid

Since r1.76 "Get rid of pmap_map_early()" the FDT was mapped read-only,
but CRYPTO softraid code writes it to zero out the key.

Found and tested by me; explanation and fix from kettenis.
OK kettenis

Set the default for machdep.lidaction to 1 like on amd64.

ok tobhe@, deraadt@

Add arm64 lid_action sysctl for Apple Silicon laptops.

ok kettenis@

Expose the complete set of ID registers as defined in the current version
of ARMv8/ARMv9. Make sure we only expose the features that we know about
and support in our kernel. This matches what Linux does. For now, mostly
restrict ourselves to features defined in ARMv8.5 which means that we only
actually implement support for ID_AA64ISAR0_EL1, ID_AA64ISAR1_EL1,
ID_AA64PFR0_EL1 and ID_AA64PFR1_EL1. For the other registers we simply
always return 0.

ok deraadt@

Get rid of pmap_map_early(). This is part of my crusade against the use of
1G mappings for the PA = VA identity mapping used in the early boot phase
of the kernel and when spinning up CPUs. The mappings are dangerous since
they might (unintentially) covering address ranges that should not be
mapped (i.e. secure memory) which is dangerous on arm64 since the
architecture allows speculative access to any address for which a valid
mapping exists and even speculative access may cause the machine to
misbehave.

So instead of relying on the PA = VA identity mapping, call pmap_bootstrap()
earlier such that we can use pmap_kenter_cache() to enter mappings for the
FDT.

ok miod@

Simplfity setregs() by passing it the ps_strings and switching
sys_execve() to return EJUSTRETURN.

setregs() is the MD routine used by sys_execve() to set up the
thread's trapframe and PCB such that, on 'return' to userspace, it
has the register values defined by the ABI and otherwise zero. It
had to set the syscall retval[] values previously because the normal
syscall return path overwrites a couple registers with the retval[]
values. By instead returning EJUSTRETURN that and some complexity
with program-counter handling on m88k and sparc64 goes away.

Also, give setregs() add a 'struct ps_strings *arginfo' argument
so powerpc, powerpc64, and sh can directly get argc/argv/envp
values for registers instead of copyin()ing the one in userspace.

Improvements from miod@ and millert@
Testing assistance miod@, kettenis@, and aoyama@
ok miod@ kettenis@

On CPUs that support the Data Independent Timing feature, enable this feature
by default in both the kernel and userland. At this point, this feature has
only been seen on Apple's CPU cores, where turning it on has no measurable
impact on performance. Turning this feature on should help to mitigate
timing side-channel attacks.

ok deraadt@, beck@

Reorganize the EFI code a bit. Move the efi.h header from dev/acpi to
dev/efi and rename the arm64 efi.c to efi_machdep.c, preparing the way
for MI EFI code and an amd64 implementation of EFI runtime support.

ok deraadt@, mlarkin@

Don't ignore memory blocks smaller than 64KB. Some EFI implementations
(such as the one on the x13s) allocate memory with the EfiBootSevicesData
type in a semi-random fashion. Ignoring small regions with that type
results in different memory maps between boots of the same kernel.
This causes problems with upcoming hibernate support.

The decision to ignore small regions was made because we do this on
amd64 to work arounmd broken BIOS implementations and because of fears
that we would run out of physical memory segments in UVM. We have some
reasons to believe that we can trust the EFI memory map on arm64 and the
risk of running out of physical memory segments is mitigated by the fact
that we try to merge memory regions before loading them into UVM. If
for some reason we see a significant drop in physical memory on certain
machines, we should probably increase the number of items in the array
we use to store memory regions and/or increase the number of physical
memory segments in UVM.

ok mlarkin@

Put printing the EFI memory map behind and #ifdef.

ok deraadt@

Postpone calling the EFI SetVirtualAddressMap() interface until efi(4)
attaches. This prevents us from having to map the various EFI memory regions
early on. These early mappings use 1G blocks. On Apple M1 systems, these
blocks may overlap memory that isn't accessable. The CPU may speculatively
access this inaccessable memory which will result in SError exceptions.

ok deraadt@

Export the ID_AA64ISARn_EL1 registers to userspace through sysctl(2) such
that we can detect which instruction set extensions are supported without
relying in catching SIGILL.

ok deraadt@

Enable cduart(4) on arm64.

OK kettenis@

Rewrite the kernel FPU handling code. The new code saves the FPU state
in cpu_switch() instead of at the kernel edge and gets rid of the FPU
state tracking in struct cpu_info and struct pcb. This fixes the random
crashes seen with SMP kernels on Apple M1.

ok patrick@

zero correct var in cpu_dump()
ok guenther@ deraadt@

Avoid local variable definitions in the middle of code.

Make memreg_add() a bit smarter and have it merge adjacent regions.
This should reduce the number of physical memory segments that we pass
to uvm reducing the chance that we run out of physical memory segments
and should make physical memory allocation a bit more efficient.

ok patrick@

Build a list of memory regions and call yvm_physload(9) on those like
we do on riscv64.

ok patrick@

Initialize per-CPU pointer register earlier.

ok patrick@

Add missing memory clobbers to "data" barriers.

We can use memory marked as EfiBootServicesCode or EfiBootServicesData
as well.

ok drahn@, kn@

spelling

Introduce BUS_SPACE_MAP_POSTED such that we can distinguish between
posted and non-posted device memory mappings and set the right memory
attributes for them. Needed because on the Apple M1 using the wrong
mapping will fault.

ok patrick@, dlg@

Call exuart(4) early attach on arm64.

ok kettenis@

Give machdep.c a thorough cleanup that is long overdue.

ok patrick@

Clean up arm64's autoconf code. There have been quite a few unused
variables and includes. While there get rid of the boot_file support
and (void) casts.

ok kettenis@

Include <sys/systm.h> directly instead of relying on hidden UVM includes.

The header is being pulled via db_machdep.h -> uvm_extern.h -> uvm_map.h

Implement cpu_rnd_messybits() as a read of the virtual counter xored
with a bit-reversed copy of itself. There is progressively less
entropy in the higher bits of a counter than in the lower bits, so
bit-reverse one half in order to extract maximal entropy.

style fixes and ok kettenis@

introduce "cpu_rnd_messybits" for use instead of nanotime in dev/rnd.c.

rnd.c uses nanotime to get access to some bits that change quickly
between events that it can mix into the entropy pool. it doesn't
use nanotime to get a monotonically increasing set or ordered and
accurate timestamps, it just wants something with bits that change.

there's been discussions for years about letting rnd use a clock
that's super fast to read, but not necessarily accurate, but it
wasn't until recently that i figured out it wasn't interested in
time at all, so things like keeping a fast clock coherent between
cpu cores or correct according to ntp is unecessary. this means we
can just let rnd read the cycle counters on cpus and things will
be fine. cpus with cycle counters that vary in their speed and
arent kept consistent between cores may even be desirable in this
context.

so this is the first step in converting rnd.c to reading cycle
counter. it copies the nanotime backend to each arch, and they can
replace it with something MD as a second step later on.

djm@ suggested rnd_messybytes, but we landed on cpu_rnd_messybits.
thanks to visa for his eyes.
ok deraadt@ visa@
deraadt@ says he will help handle any MD fallout that occurs.

Add a little bit of const-ness.

Look at the openbsd,boothowto property.

ok visa@

Add machdep.compatible.

ok jsg@

Make inittodr() and resettodr() MI.

ok deraadt@, mpi@, visa@
ok cheloha@ as well (would have preferred in new file for this code)

Initialize the timeval passed to todr_gettime() with the base time from
the file system such that implementations can use it to guess the right
century.

ok mpi@

Move inittodr() and resettodr() to the end of the file to easy move an MI
implementation.

ok mpi@

Sanitize inittodr()/resettodr() implementation.
- move implementations next to eachother in the same file
- remove pointless call to resettodr() in inittodr()
- use OpenBSD define to get a minimum plausible time
- if RTC time is before minimum plausible time, reject it
- don't print "clock gained N days" on ramdisk

Hopefully this implementation can serve as a model for unification
into an MI implementation.

ok mpi@

Set DMA constraint based on the openbsd,dma-constraint property if
present. Make sure bus_dmamap_alloc(9) allocates memory that meets that
constraint.

ok patrick@

Add amluart(4), a driver for the UARTs found on various Amlogic SoCs.

ok jsg@

Make sure the virtual addresses that are used for EFI runtime services
fit into the 39-bit address space that we currently use for the runtime
mappings.

ok patrick@

On machines with large amounts of physical memory we fail to initialize uvm
because we don't have enough kernel memory available in the early bootstrap
phase to allocate the vm_page structures. Fix this by making uvm_growkernel()
work before uvm is initialized like we do on other architectures that don't
use a direct map and explicitly call it after enumerating the available
physical memory with an estimate of how much KVA we need to initialize uvm.

ok patrick@

On bigger arm64 machines with plenty of memory it is possible that
the kernel, which is allocated in low memory by efiboot, is not in
the same 512 GiB L0 address space as the EFI/FDT/ACPI tables. Since
we only identity-map a single L0 entry for bootstrapping purposes we
can't reach those tables. Keep a set of spare L1 tables, currently
one, which we can dynamically enter into the identity map so that we
are able to reach those tables. While there, do the FDT mapping a
bit later in C code by calling pmap_map_early(). This allows us to
boot further on Packet's Ampere eMAGs.

Discussed with drahn@
ok kettenis@

Add mvuart(4) to support the Armada 3720's serial console.

ok kettenis@

fast track ddb> reboot command to skip anything which might panic again.
ok deraadt

Implement booting from softraid on arm64. This consists of creating a
list of available block devices, so we can also boot from another block
device than the one that efiboot was loaded from. Softraid will then
create a list of volumes by checking the partition table for RAID and
matching softraid metadata. If efiboot was loaded from the same
physical drive as a softraid volume, it will use the soft- raid volume
as boot device and it will ask you to unlock it. The UUID and key of
that boot volume will be passed as FDT properties to the kernel. Those
FDT properties will be zeroed explicitly by the kernel after they have
been passed to the softraid stack.

ok kettenis@

It is possible for a memory write to not complete if no barrier exists
between the write and a wfi instruction. Make certain system state
is coherent when waiting on interrupts.
ok kettenis@

Remove comcnspeed variable, which gets set but is never actually used.
This removes the (no-op) -1 and -9 kernel boot arguments.

ok visa@

Cleanup bootconfig.h and use strlcpy() to copy boot arguments.

ok drahn@

Remove a4x bus space hack.

ok patrick@

Use speed from device tree for serial console. Makes it possible to boot
and install systems where the firmware uses a non-standard speed. This is
important for various boards that use a Rockchip SoC where the vendor thinks
using 1500000 bps as the serial console speed is a good idea (it isn't).

ok deraadt@, visa@, patrick@

Call imxuart(4) early attach on arm64.

Remove bogus copying of bootargs.

From Arrtturi Alm.

Allow the kernel to recognize that it has been netbooted and to add the
boot interface to the "netboot" group. efiboot grabs the MAC address
from the PXE environment, passes it to the kernel, where it is matched
against the list of ethernet interfaces and the boot device is set.
Concept and most of the code cribbed from amd64.
ok kettenis@

Make sure each CPU uses its own set of VAs for pmap_zero_page() and
pmap_copy_page().

ok patrick@

Spin up secondary CPUs and add IPI infrastructure. Heavily based on diffs
from dran@.

ok patrick@

Runtime services may (and do) use device mappings on some UEFI implementations.
Skip these mappings during the remap-pahse as they are likely to be in
a different 512G bloch as memory and SetVirtualAddressMap() shouldn't need
them. But do assign a new virtual address and let efi(4) create a mapping.
Add a PMAP_DEVICE flag such that pmap_enter() can continue to be used
to create these mappings.

ok patrick@

Implement FUCKWIT for arm64; unmap the kernel almost entirely while userland
is running. This provides protection against meltown on cores that are
vilnerable (just Cortex-A75 so far) but also seems to be an essential to
protect against spectre-like attacks against the kernel.

This implementation only exposes a single treampoline page that does not
contain any kernel virtual addresses and also hides the real virtual address
of the exception vectors, which helps on cores vulnerable to "variant 3a"
(Cortex-A57, Cortex-A72). The implementation is inspired by the work done
by Will Deacon for Linux, but there are no knobs to turn it off. The
overhead is fairly limited: around 3-4% slowdown on Cortex-A57.

ok patrick@, deraadt@

Implement support for calling EFI runtime services and use it to implement
a time-of-day clock device based on the GetTime() and SetTime() services.
The virtual memory mappings for the runtime services calls are implemented
through a separate pmap that is only activated when we make a runtime services
call.

ok tom@, visa@
tested by naddy@

In uvm Chuck decided backing store would not be allocated proactively
for blocks re-fetchable from the filesystem. However at reboot time,
filesystems are unmounted, and since processes lack backing store they
are killed. Since the scheduler is still running, in some cases init is
killed... which drops us to ddb [noted by bluhm]. Solution is to convert
filesystems to read-only [proposed by kettenis]. The tale follows:
sys_reboot() should pass proc * to MD boot() to vfs_shutdown() which
completes current IO with vfs_busy VB_WRITE|VB_WAIT, then calls VFS_MOUNT()
with MNT_UPDATE | MNT_RDONLY, soon teaching us that *fs_mount() calls a
copyin() late... so store the sizes in vfsconflist[] and move the copyin()
to sys_mount()... and notice nfs_mount copyin() is size-variant, so kill
legacy struct nfs_args3. Next we learn ffs_mount()'s MNT_UPDATE code is
sharp and rusty especially wrt softdep, so fix some bugs adn add
~MNT_SOFTDEP to the downgrade. Some vnodes need a little more help,
so tie them to &dead_vnops.

ffs_mount calling DIOCCACHESYNC is causing a bit of grief still but
this issue is seperate and will be dealt with in time.
couple hundred reboots by bluhm and myself, advice from guenther and
others at the hut

If you use sys/param.h, you don't need sys/types.h

Add glass console support for arm64. This uses the "stdout-path" property
of the /chosen node in the device tree to decide whether the framebuffer
should be used as the console device. Most, if not all, machines will
have that set to use a serial console and there is no easy way yet to
change that.

ok jsg@

Clear second return register on exec.

ok guenther@

fix format strings
ok kettenis@

Use EFI memory map to discover available physical memory.
Fall back on improved code that parses the FDT /memory node.
However, on machines with "real" UEFI firmware, /memory may include
reserved memory that we shouldn't use (it does on the Overdrive 1000).
On those machines, make sure that you use BOOTAA64 0.3 or later such that
the EFI memory map gets used.

For now, print out the EFI memory map to help debug any issues.

ok patrick@

Bring boot() in line with our other architectures.

ok mpi@, deraadt@

Zap silly diagnostic reboot message.

ok bmercer@, deraadt@

Rename Debugger() into db_enter().

Using a name with the 'db_' prefix makes it invisible from the dynamic
profiler.

ok deraadt@, kettenis@, visa@

store cpu model information in the buffer used by the hw.model sysctl
ok kettenis@

Revise 'struct fpreg' and dump floating-point register in core dumps. Also
reset the floating-point register state upon exec.

ok guenther@

Don't limit physmem to 2GB confirmed to work with 16GB by deraadt@.

Fix up and use the device memory attribute for device mappings. Add
inner shareable attributes while there.

Move cache and tlb flush functions, which were mostly inline assembly,
into separate functions. This makes them reusable from other parts in
the kernel. Assembly and header are taken from FreeBSD, but modified
to fit our requirements and with some unnecessary stuff removed. While
there remove micro optimization for uniprocessor kernels.

A bit of code and comment cleanup.

Remove gdb waitcount debug code and dummy print.

Enter UKC when built with BOOT_CONFIG and -c is specified.

ok kettenis@ patrick@

Initialize DDB if compiled with option DDB.

ok kettenis@

Exterminate comcnspeed and comcnmode which were uninitialized and
caused issues in the tty layer. Use the tty defaults for the cflags,
while still overwriting the terminal speed with B115200.

Found with lots of help from deraadt@

Make sure to also backup and restore the a4x bus space map function
so that we can initialize the console early.

Ansify cpu_sysctl() on alpha, arm, arm64, luna88k and sparc64.

OK mpi@

fix uninitialised variable warnings from clang
ok patrick@

Import of OpenBSD/arm64

This commit contains all the kernel files related to the OpenBSD/arm64
port. It is based on the PowerPC pmap, loongson, arm/armv7 code and
FreeBSD aarch64 code. Hard work done by Dale Rahn.

handle /reserved-memory nodes from device trees on arm64.

u-boot is supposed to take these entries and put them in the efi
memory map, but i keep hitting machines where an otherwise functional
u-boot does not do this, resulting in weird errors.

i have an espressobin with a vendor u-boot that has a reserved-memory
region for psci. without this diff the machine faults when the
kernel tries to reboot using a psci handler.

a macchiatobin with an otherwise working u-boot throws SErrors or
panics on weird memory corruption problems without this. i thought
it was bad RAM, but the problems persisted with completely different
ram, and very underclocked and well cooled ram.

riscv64 already has code to handle reserved-memory regions. the
riscv64 change is to add handling for the "no-map" property.

ok kettenis@

Account for nkmempages as well in the pmap_growkernel() call during
initalisation. This way there is enough KVA mapped that kmeminit()
succeeds even with large nkmempages. This is similar to e.g. alpha.
OK miod@ kettenis@

Take advantage of the fact that the WFI instruction does continue
immediately if there is a pending interrupt to fix a potential race
in the idle loop.

ok guenther@

Use the deep idle state available on Apple M1/M2 cores in the idle loop and
for suspend. This state makes the CPU lose some of its register state so
we need to save these registers before putting the core to sleep and
restore them when we wake up. This deep idle state has a higher wakeup
latency than the normal WFI idle state. Use similar logic as acpucpu(4) to
decide which idle state to pick.

If some cores of a cluster are in this deep idle state, turbo states become
available to the cores that remain active. So stop skipping these states.
This improves single-core performance a little bit.

The main win is in power savings when running in a state with a high clock
frequency. My M2 Pro mini goes from 14W to 6.5W when idle at the maximum
clock frequency. But event at the lowest clock frequency there are small
but significant power savings.

ok deraadt@, tobhe@

Implement support for pointer authentication (PAC) in userland. With PAC
it is possible to "sign" pointers with a hidden key. The signature is
placed in unused bits of the pointer and can be checked later. This can
be used to provide "tail CFI" that is similar to what retguard provides.

Debuggers need to be aware of the fact that pointers can be signed. For
this purpose a new PT_PACMASK ptrace(2) request is introduced that returns
as mask that indicates the bits used for the signature. Separate masks
are provided for code and data pointers even though the masks are identical
in the current implementation. These masks are also written into a special
note section in the core dump.

ok patrick@

Enable branch target control flow enforcement on arm64.

ok robert@, deraadt@

Map device tree read/write to unbreak root on softraid

Since r1.76 "Get rid of pmap_map_early()" the FDT was mapped read-only,
but CRYPTO softraid code writes it to zero out the key.

Found and tested by me; explanation and fix from kettenis.
OK kettenis

Set the default for machdep.lidaction to 1 like on amd64.

ok tobhe@, deraadt@

Add arm64 lid_action sysctl for Apple Silicon laptops.

ok kettenis@

Expose the complete set of ID registers as defined in the current version
of ARMv8/ARMv9. Make sure we only expose the features that we know about
and support in our kernel. This matches what Linux does. For now, mostly
restrict ourselves to features defined in ARMv8.5 which means that we only
actually implement support for ID_AA64ISAR0_EL1, ID_AA64ISAR1_EL1,
ID_AA64PFR0_EL1 and ID_AA64PFR1_EL1. For the other registers we simply
always return 0.

ok deraadt@

Get rid of pmap_map_early(). This is part of my crusade against the use of
1G mappings for the PA = VA identity mapping used in the early boot phase
of the kernel and when spinning up CPUs. The mappings are dangerous since
they might (unintentially) covering address ranges that should not be
mapped (i.e. secure memory) which is dangerous on arm64 since the
architecture allows speculative access to any address for which a valid
mapping exists and even speculative access may cause the machine to
misbehave.

So instead of relying on the PA = VA identity mapping, call pmap_bootstrap()
earlier such that we can use pmap_kenter_cache() to enter mappings for the
FDT.

ok miod@

Simplfity setregs() by passing it the ps_strings and switching
sys_execve() to return EJUSTRETURN.

setregs() is the MD routine used by sys_execve() to set up the
thread's trapframe and PCB such that, on 'return' to userspace, it
has the register values defined by the ABI and otherwise zero. It
had to set the syscall retval[] values previously because the normal
syscall return path overwrites a couple registers with the retval[]
values. By instead returning EJUSTRETURN that and some complexity
with program-counter handling on m88k and sparc64 goes away.

Also, give setregs() add a 'struct ps_strings *arginfo' argument
so powerpc, powerpc64, and sh can directly get argc/argv/envp
values for registers instead of copyin()ing the one in userspace.

Improvements from miod@ and millert@
Testing assistance miod@, kettenis@, and aoyama@
ok miod@ kettenis@

On CPUs that support the Data Independent Timing feature, enable this feature
by default in both the kernel and userland. At this point, this feature has
only been seen on Apple's CPU cores, where turning it on has no measurable
impact on performance. Turning this feature on should help to mitigate
timing side-channel attacks.

ok deraadt@, beck@

Reorganize the EFI code a bit. Move the efi.h header from dev/acpi to
dev/efi and rename the arm64 efi.c to efi_machdep.c, preparing the way
for MI EFI code and an amd64 implementation of EFI runtime support.

ok deraadt@, mlarkin@

Don't ignore memory blocks smaller than 64KB. Some EFI implementations
(such as the one on the x13s) allocate memory with the EfiBootSevicesData
type in a semi-random fashion. Ignoring small regions with that type
results in different memory maps between boots of the same kernel.
This causes problems with upcoming hibernate support.

The decision to ignore small regions was made because we do this on
amd64 to work arounmd broken BIOS implementations and because of fears
that we would run out of physical memory segments in UVM. We have some
reasons to believe that we can trust the EFI memory map on arm64 and the
risk of running out of physical memory segments is mitigated by the fact
that we try to merge memory regions before loading them into UVM. If
for some reason we see a significant drop in physical memory on certain
machines, we should probably increase the number of items in the array
we use to store memory regions and/or increase the number of physical
memory segments in UVM.

ok mlarkin@

Put printing the EFI memory map behind and #ifdef.

ok deraadt@

Postpone calling the EFI SetVirtualAddressMap() interface until efi(4)
attaches. This prevents us from having to map the various EFI memory regions
early on. These early mappings use 1G blocks. On Apple M1 systems, these
blocks may overlap memory that isn't accessable. The CPU may speculatively
access this inaccessable memory which will result in SError exceptions.

ok deraadt@

Export the ID_AA64ISARn_EL1 registers to userspace through sysctl(2) such
that we can detect which instruction set extensions are supported without
relying in catching SIGILL.

ok deraadt@

Enable cduart(4) on arm64.

OK kettenis@

Rewrite the kernel FPU handling code. The new code saves the FPU state
in cpu_switch() instead of at the kernel edge and gets rid of the FPU
state tracking in struct cpu_info and struct pcb. This fixes the random
crashes seen with SMP kernels on Apple M1.

ok patrick@

zero correct var in cpu_dump()
ok guenther@ deraadt@

Avoid local variable definitions in the middle of code.

Make memreg_add() a bit smarter and have it merge adjacent regions.
This should reduce the number of physical memory segments that we pass
to uvm reducing the chance that we run out of physical memory segments
and should make physical memory allocation a bit more efficient.

ok patrick@

Build a list of memory regions and call yvm_physload(9) on those like
we do on riscv64.

ok patrick@

Initialize per-CPU pointer register earlier.

ok patrick@

Add missing memory clobbers to "data" barriers.

We can use memory marked as EfiBootServicesCode or EfiBootServicesData
as well.

ok drahn@, kn@

spelling

Introduce BUS_SPACE_MAP_POSTED such that we can distinguish between
posted and non-posted device memory mappings and set the right memory
attributes for them. Needed because on the Apple M1 using the wrong
mapping will fault.

ok patrick@, dlg@

Call exuart(4) early attach on arm64.

ok kettenis@

Give machdep.c a thorough cleanup that is long overdue.

ok patrick@

Clean up arm64's autoconf code. There have been quite a few unused
variables and includes. While there get rid of the boot_file support
and (void) casts.

ok kettenis@

Include <sys/systm.h> directly instead of relying on hidden UVM includes.

The header is being pulled via db_machdep.h -> uvm_extern.h -> uvm_map.h

Implement cpu_rnd_messybits() as a read of the virtual counter xored
with a bit-reversed copy of itself. There is progressively less
entropy in the higher bits of a counter than in the lower bits, so
bit-reverse one half in order to extract maximal entropy.

style fixes and ok kettenis@

introduce "cpu_rnd_messybits" for use instead of nanotime in dev/rnd.c.

rnd.c uses nanotime to get access to some bits that change quickly
between events that it can mix into the entropy pool. it doesn't
use nanotime to get a monotonically increasing set or ordered and
accurate timestamps, it just wants something with bits that change.

there's been discussions for years about letting rnd use a clock
that's super fast to read, but not necessarily accurate, but it
wasn't until recently that i figured out it wasn't interested in
time at all, so things like keeping a fast clock coherent between
cpu cores or correct according to ntp is unecessary. this means we
can just let rnd read the cycle counters on cpus and things will
be fine. cpus with cycle counters that vary in their speed and
arent kept consistent between cores may even be desirable in this
context.

so this is the first step in converting rnd.c to reading cycle
counter. it copies the nanotime backend to each arch, and they can
replace it with something MD as a second step later on.

djm@ suggested rnd_messybytes, but we landed on cpu_rnd_messybits.
thanks to visa for his eyes.
ok deraadt@ visa@
deraadt@ says he will help handle any MD fallout that occurs.

Add a little bit of const-ness.

Look at the openbsd,boothowto property.

ok visa@

Add machdep.compatible.

ok jsg@

Make inittodr() and resettodr() MI.

ok deraadt@, mpi@, visa@
ok cheloha@ as well (would have preferred in new file for this code)

Initialize the timeval passed to todr_gettime() with the base time from
the file system such that implementations can use it to guess the right
century.

ok mpi@

Move inittodr() and resettodr() to the end of the file to easy move an MI
implementation.

ok mpi@

Sanitize inittodr()/resettodr() implementation.
- move implementations next to eachother in the same file
- remove pointless call to resettodr() in inittodr()
- use OpenBSD define to get a minimum plausible time
- if RTC time is before minimum plausible time, reject it
- don't print "clock gained N days" on ramdisk

Hopefully this implementation can serve as a model for unification
into an MI implementation.

ok mpi@

Set DMA constraint based on the openbsd,dma-constraint property if
present. Make sure bus_dmamap_alloc(9) allocates memory that meets that
constraint.

ok patrick@

Add amluart(4), a driver for the UARTs found on various Amlogic SoCs.

ok jsg@

Make sure the virtual addresses that are used for EFI runtime services
fit into the 39-bit address space that we currently use for the runtime
mappings.

ok patrick@

On machines with large amounts of physical memory we fail to initialize uvm
because we don't have enough kernel memory available in the early bootstrap
phase to allocate the vm_page structures. Fix this by making uvm_growkernel()
work before uvm is initialized like we do on other architectures that don't
use a direct map and explicitly call it after enumerating the available
physical memory with an estimate of how much KVA we need to initialize uvm.

ok patrick@

On bigger arm64 machines with plenty of memory it is possible that
the kernel, which is allocated in low memory by efiboot, is not in
the same 512 GiB L0 address space as the EFI/FDT/ACPI tables. Since
we only identity-map a single L0 entry for bootstrapping purposes we
can't reach those tables. Keep a set of spare L1 tables, currently
one, which we can dynamically enter into the identity map so that we
are able to reach those tables. While there, do the FDT mapping a
bit later in C code by calling pmap_map_early(). This allows us to
boot further on Packet's Ampere eMAGs.

Discussed with drahn@
ok kettenis@

Add mvuart(4) to support the Armada 3720's serial console.

ok kettenis@

fast track ddb> reboot command to skip anything which might panic again.
ok deraadt

Implement booting from softraid on arm64. This consists of creating a
list of available block devices, so we can also boot from another block
device than the one that efiboot was loaded from. Softraid will then
create a list of volumes by checking the partition table for RAID and
matching softraid metadata. If efiboot was loaded from the same
physical drive as a softraid volume, it will use the soft- raid volume
as boot device and it will ask you to unlock it. The UUID and key of
that boot volume will be passed as FDT properties to the kernel. Those
FDT properties will be zeroed explicitly by the kernel after they have
been passed to the softraid stack.

ok kettenis@

It is possible for a memory write to not complete if no barrier exists
between the write and a wfi instruction. Make certain system state
is coherent when waiting on interrupts.
ok kettenis@

Remove comcnspeed variable, which gets set but is never actually used.
This removes the (no-op) -1 and -9 kernel boot arguments.

ok visa@

Cleanup bootconfig.h and use strlcpy() to copy boot arguments.

ok drahn@

Remove a4x bus space hack.

ok patrick@

Use speed from device tree for serial console. Makes it possible to boot
and install systems where the firmware uses a non-standard speed. This is
important for various boards that use a Rockchip SoC where the vendor thinks
using 1500000 bps as the serial console speed is a good idea (it isn't).

ok deraadt@, visa@, patrick@

Call imxuart(4) early attach on arm64.

Remove bogus copying of bootargs.

From Arrtturi Alm.

Allow the kernel to recognize that it has been netbooted and to add the
boot interface to the "netboot" group. efiboot grabs the MAC address
from the PXE environment, passes it to the kernel, where it is matched
against the list of ethernet interfaces and the boot device is set.
Concept and most of the code cribbed from amd64.
ok kettenis@

Make sure each CPU uses its own set of VAs for pmap_zero_page() and
pmap_copy_page().

ok patrick@

Spin up secondary CPUs and add IPI infrastructure. Heavily based on diffs
from dran@.

ok patrick@

Runtime services may (and do) use device mappings on some UEFI implementations.
Skip these mappings during the remap-pahse as they are likely to be in
a different 512G bloch as memory and SetVirtualAddressMap() shouldn't need
them. But do assign a new virtual address and let efi(4) create a mapping.
Add a PMAP_DEVICE flag such that pmap_enter() can continue to be used
to create these mappings.

ok patrick@

Implement FUCKWIT for arm64; unmap the kernel almost entirely while userland
is running. This provides protection against meltown on cores that are
vilnerable (just Cortex-A75 so far) but also seems to be an essential to
protect against spectre-like attacks against the kernel.

This implementation only exposes a single treampoline page that does not
contain any kernel virtual addresses and also hides the real virtual address
of the exception vectors, which helps on cores vulnerable to "variant 3a"
(Cortex-A57, Cortex-A72). The implementation is inspired by the work done
by Will Deacon for Linux, but there are no knobs to turn it off. The
overhead is fairly limited: around 3-4% slowdown on Cortex-A57.

ok patrick@, deraadt@

Implement support for calling EFI runtime services and use it to implement
a time-of-day clock device based on the GetTime() and SetTime() services.
The virtual memory mappings for the runtime services calls are implemented
through a separate pmap that is only activated when we make a runtime services
call.

ok tom@, visa@
tested by naddy@

In uvm Chuck decided backing store would not be allocated proactively
for blocks re-fetchable from the filesystem. However at reboot time,
filesystems are unmounted, and since processes lack backing store they
are killed. Since the scheduler is still running, in some cases init is
killed... which drops us to ddb [noted by bluhm]. Solution is to convert
filesystems to read-only [proposed by kettenis]. The tale follows:
sys_reboot() should pass proc * to MD boot() to vfs_shutdown() which
completes current IO with vfs_busy VB_WRITE|VB_WAIT, then calls VFS_MOUNT()
with MNT_UPDATE | MNT_RDONLY, soon teaching us that *fs_mount() calls a
copyin() late... so store the sizes in vfsconflist[] and move the copyin()
to sys_mount()... and notice nfs_mount copyin() is size-variant, so kill
legacy struct nfs_args3. Next we learn ffs_mount()'s MNT_UPDATE code is
sharp and rusty especially wrt softdep, so fix some bugs adn add
~MNT_SOFTDEP to the downgrade. Some vnodes need a little more help,
so tie them to &dead_vnops.

ffs_mount calling DIOCCACHESYNC is causing a bit of grief still but
this issue is seperate and will be dealt with in time.
couple hundred reboots by bluhm and myself, advice from guenther and
others at the hut

If you use sys/param.h, you don't need sys/types.h

Add glass console support for arm64. This uses the "stdout-path" property
of the /chosen node in the device tree to decide whether the framebuffer
should be used as the console device. Most, if not all, machines will
have that set to use a serial console and there is no easy way yet to
change that.

ok jsg@

Clear second return register on exec.

ok guenther@

fix format strings
ok kettenis@

Use EFI memory map to discover available physical memory.
Fall back on improved code that parses the FDT /memory node.
However, on machines with "real" UEFI firmware, /memory may include
reserved memory that we shouldn't use (it does on the Overdrive 1000).
On those machines, make sure that you use BOOTAA64 0.3 or later such that
the EFI memory map gets used.

For now, print out the EFI memory map to help debug any issues.

ok patrick@

Bring boot() in line with our other architectures.

ok mpi@, deraadt@

Zap silly diagnostic reboot message.

ok bmercer@, deraadt@

Rename Debugger() into db_enter().

Using a name with the 'db_' prefix makes it invisible from the dynamic
profiler.

ok deraadt@, kettenis@, visa@

store cpu model information in the buffer used by the hw.model sysctl
ok kettenis@

Revise 'struct fpreg' and dump floating-point register in core dumps. Also
reset the floating-point register state upon exec.

ok guenther@

Don't limit physmem to 2GB confirmed to work with 16GB by deraadt@.

Fix up and use the device memory attribute for device mappings. Add
inner shareable attributes while there.

Move cache and tlb flush functions, which were mostly inline assembly,
into separate functions. This makes them reusable from other parts in
the kernel. Assembly and header are taken from FreeBSD, but modified
to fit our requirements and with some unnecessary stuff removed. While
there remove micro optimization for uniprocessor kernels.

A bit of code and comment cleanup.

Remove gdb waitcount debug code and dummy print.

Enter UKC when built with BOOT_CONFIG and -c is specified.

ok kettenis@ patrick@

Initialize DDB if compiled with option DDB.

ok kettenis@

Exterminate comcnspeed and comcnmode which were uninitialized and
caused issues in the tty layer. Use the tty defaults for the cflags,
while still overwriting the terminal speed with B115200.

Found with lots of help from deraadt@

Make sure to also backup and restore the a4x bus space map function
so that we can initialize the console early.

Ansify cpu_sysctl() on alpha, arm, arm64, luna88k and sparc64.

OK mpi@

fix uninitialised variable warnings from clang
ok patrick@

Import of OpenBSD/arm64

This commit contains all the kernel files related to the OpenBSD/arm64
port. It is based on the PowerPC pmap, loongson, arm/armv7 code and
FreeBSD aarch64 code. Hard work done by Dale Rahn.

Account for nkmempages as well in the pmap_growkernel() call during
initalisation. This way there is enough KVA mapped that kmeminit()
succeeds even with large nkmempages. This is similar to e.g. alpha.
OK miod@ kettenis@

Take advantage of the fact that the WFI instruction does continue
immediately if there is a pending interrupt to fix a potential race
in the idle loop.

ok guenther@

Use the deep idle state available on Apple M1/M2 cores in the idle loop and
for suspend. This state makes the CPU lose some of its register state so
we need to save these registers before putting the core to sleep and
restore them when we wake up. This deep idle state has a higher wakeup
latency than the normal WFI idle state. Use similar logic as acpucpu(4) to
decide which idle state to pick.

If some cores of a cluster are in this deep idle state, turbo states become
available to the cores that remain active. So stop skipping these states.
This improves single-core performance a little bit.

The main win is in power savings when running in a state with a high clock
frequency. My M2 Pro mini goes from 14W to 6.5W when idle at the maximum
clock frequency. But event at the lowest clock frequency there are small
but significant power savings.

ok deraadt@, tobhe@

Implement support for pointer authentication (PAC) in userland. With PAC
it is possible to "sign" pointers with a hidden key. The signature is
placed in unused bits of the pointer and can be checked later. This can
be used to provide "tail CFI" that is similar to what retguard provides.

Debuggers need to be aware of the fact that pointers can be signed. For
this purpose a new PT_PACMASK ptrace(2) request is introduced that returns
as mask that indicates the bits used for the signature. Separate masks
are provided for code and data pointers even though the masks are identical
in the current implementation. These masks are also written into a special
note section in the core dump.

ok patrick@

Enable branch target control flow enforcement on arm64.

ok robert@, deraadt@

Map device tree read/write to unbreak root on softraid

Since r1.76 "Get rid of pmap_map_early()" the FDT was mapped read-only,
but CRYPTO softraid code writes it to zero out the key.

Found and tested by me; explanation and fix from kettenis.
OK kettenis

Set the default for machdep.lidaction to 1 like on amd64.

ok tobhe@, deraadt@

Add arm64 lid_action sysctl for Apple Silicon laptops.

ok kettenis@

Expose the complete set of ID registers as defined in the current version
of ARMv8/ARMv9. Make sure we only expose the features that we know about
and support in our kernel. This matches what Linux does. For now, mostly
restrict ourselves to features defined in ARMv8.5 which means that we only
actually implement support for ID_AA64ISAR0_EL1, ID_AA64ISAR1_EL1,
ID_AA64PFR0_EL1 and ID_AA64PFR1_EL1. For the other registers we simply
always return 0.

ok deraadt@

Get rid of pmap_map_early(). This is part of my crusade against the use of
1G mappings for the PA = VA identity mapping used in the early boot phase
of the kernel and when spinning up CPUs. The mappings are dangerous since
they might (unintentially) covering address ranges that should not be
mapped (i.e. secure memory) which is dangerous on arm64 since the
architecture allows speculative access to any address for which a valid
mapping exists and even speculative access may cause the machine to
misbehave.

So instead of relying on the PA = VA identity mapping, call pmap_bootstrap()
earlier such that we can use pmap_kenter_cache() to enter mappings for the
FDT.

ok miod@

Simplfity setregs() by passing it the ps_strings and switching
sys_execve() to return EJUSTRETURN.

setregs() is the MD routine used by sys_execve() to set up the
thread's trapframe and PCB such that, on 'return' to userspace, it
has the register values defined by the ABI and otherwise zero. It
had to set the syscall retval[] values previously because the normal
syscall return path overwrites a couple registers with the retval[]
values. By instead returning EJUSTRETURN that and some complexity
with program-counter handling on m88k and sparc64 goes away.

Also, give setregs() add a 'struct ps_strings *arginfo' argument
so powerpc, powerpc64, and sh can directly get argc/argv/envp
values for registers instead of copyin()ing the one in userspace.

Improvements from miod@ and millert@
Testing assistance miod@, kettenis@, and aoyama@
ok miod@ kettenis@

On CPUs that support the Data Independent Timing feature, enable this feature
by default in both the kernel and userland. At this point, this feature has
only been seen on Apple's CPU cores, where turning it on has no measurable
impact on performance. Turning this feature on should help to mitigate
timing side-channel attacks.

ok deraadt@, beck@

Reorganize the EFI code a bit. Move the efi.h header from dev/acpi to
dev/efi and rename the arm64 efi.c to efi_machdep.c, preparing the way
for MI EFI code and an amd64 implementation of EFI runtime support.

ok deraadt@, mlarkin@

Don't ignore memory blocks smaller than 64KB. Some EFI implementations
(such as the one on the x13s) allocate memory with the EfiBootSevicesData
type in a semi-random fashion. Ignoring small regions with that type
results in different memory maps between boots of the same kernel.
This causes problems with upcoming hibernate support.

The decision to ignore small regions was made because we do this on
amd64 to work arounmd broken BIOS implementations and because of fears
that we would run out of physical memory segments in UVM. We have some
reasons to believe that we can trust the EFI memory map on arm64 and the
risk of running out of physical memory segments is mitigated by the fact
that we try to merge memory regions before loading them into UVM. If
for some reason we see a significant drop in physical memory on certain
machines, we should probably increase the number of items in the array
we use to store memory regions and/or increase the number of physical
memory segments in UVM.

ok mlarkin@

Put printing the EFI memory map behind and #ifdef.

ok deraadt@

Postpone calling the EFI SetVirtualAddressMap() interface until efi(4)
attaches. This prevents us from having to map the various EFI memory regions
early on. These early mappings use 1G blocks. On Apple M1 systems, these
blocks may overlap memory that isn't accessable. The CPU may speculatively
access this inaccessable memory which will result in SError exceptions.

ok deraadt@

Export the ID_AA64ISARn_EL1 registers to userspace through sysctl(2) such
that we can detect which instruction set extensions are supported without
relying in catching SIGILL.

ok deraadt@

Enable cduart(4) on arm64.

OK kettenis@

Rewrite the kernel FPU handling code. The new code saves the FPU state
in cpu_switch() instead of at the kernel edge and gets rid of the FPU
state tracking in struct cpu_info and struct pcb. This fixes the random
crashes seen with SMP kernels on Apple M1.

ok patrick@

zero correct var in cpu_dump()
ok guenther@ deraadt@

Avoid local variable definitions in the middle of code.

Make memreg_add() a bit smarter and have it merge adjacent regions.
This should reduce the number of physical memory segments that we pass
to uvm reducing the chance that we run out of physical memory segments
and should make physical memory allocation a bit more efficient.

ok patrick@

Build a list of memory regions and call yvm_physload(9) on those like
we do on riscv64.

ok patrick@

Initialize per-CPU pointer register earlier.

ok patrick@

Add missing memory clobbers to "data" barriers.

We can use memory marked as EfiBootServicesCode or EfiBootServicesData
as well.

ok drahn@, kn@

spelling

Introduce BUS_SPACE_MAP_POSTED such that we can distinguish between
posted and non-posted device memory mappings and set the right memory
attributes for them. Needed because on the Apple M1 using the wrong
mapping will fault.

ok patrick@, dlg@

Call exuart(4) early attach on arm64.

ok kettenis@

Give machdep.c a thorough cleanup that is long overdue.

ok patrick@

Clean up arm64's autoconf code. There have been quite a few unused
variables and includes. While there get rid of the boot_file support
and (void) casts.

ok kettenis@

Include <sys/systm.h> directly instead of relying on hidden UVM includes.

The header is being pulled via db_machdep.h -> uvm_extern.h -> uvm_map.h

Implement cpu_rnd_messybits() as a read of the virtual counter xored
with a bit-reversed copy of itself. There is progressively less
entropy in the higher bits of a counter than in the lower bits, so
bit-reverse one half in order to extract maximal entropy.

style fixes and ok kettenis@

introduce "cpu_rnd_messybits" for use instead of nanotime in dev/rnd.c.

rnd.c uses nanotime to get access to some bits that change quickly
between events that it can mix into the entropy pool. it doesn't
use nanotime to get a monotonically increasing set or ordered and
accurate timestamps, it just wants something with bits that change.

there's been discussions for years about letting rnd use a clock
that's super fast to read, but not necessarily accurate, but it
wasn't until recently that i figured out it wasn't interested in
time at all, so things like keeping a fast clock coherent between
cpu cores or correct according to ntp is unecessary. this means we
can just let rnd read the cycle counters on cpus and things will
be fine. cpus with cycle counters that vary in their speed and
arent kept consistent between cores may even be desirable in this
context.

so this is the first step in converting rnd.c to reading cycle
counter. it copies the nanotime backend to each arch, and they can
replace it with something MD as a second step later on.

djm@ suggested rnd_messybytes, but we landed on cpu_rnd_messybits.
thanks to visa for his eyes.
ok deraadt@ visa@
deraadt@ says he will help handle any MD fallout that occurs.

Add a little bit of const-ness.

Look at the openbsd,boothowto property.

ok visa@

Add machdep.compatible.

ok jsg@

Make inittodr() and resettodr() MI.

ok deraadt@, mpi@, visa@
ok cheloha@ as well (would have preferred in new file for this code)

Initialize the timeval passed to todr_gettime() with the base time from
the file system such that implementations can use it to guess the right
century.

ok mpi@

Move inittodr() and resettodr() to the end of the file to easy move an MI
implementation.

ok mpi@

Sanitize inittodr()/resettodr() implementation.
- move implementations next to eachother in the same file
- remove pointless call to resettodr() in inittodr()
- use OpenBSD define to get a minimum plausible time
- if RTC time is before minimum plausible time, reject it
- don't print "clock gained N days" on ramdisk

Hopefully this implementation can serve as a model for unification
into an MI implementation.

ok mpi@