arm64: Support BTI checking in most of the kernel

LLD has the -zbti-report=error argument to check if the BTI note is
present when linking. To allow for this to be used when linking the
kernel and modules:
- Add the BTI note to the remaining assembly files
- Mark ptrauth.c as protected by BTI
- Disable -zbti-report for vmm hypervisor switching code as it's not
used there.

The linux64 module doesn't build with the flag as it includes vdso code
that doesn't include the note.

Reviewed by: imp, kib, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45466

arm64: map kernel using large pages when page size is 16K

When the page size is 16K, use ATTR_CONTIGUOUS to map the kernel code
and data sections using 2M pages. Previously, they were mapped using
16K pages.

Reviewed by: markj
Tested by: markj
Differential Revision: https://reviews.freebsd.org/D45162

arm64: Update the page table list in locore

The comment describing the page tables was out of date. Update it with
the current list.

Sponsored by: Arm Ltd

arm64: Merge common page table creation code

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45061

arm64: Support a shared release for spin-table

When releasing multiple CPUs that share a release address we need them
to wait for their turn to boot. Add a mechanism to do this by booting
them until they enable the TLB before waiting their turn to enter
init_secondary.

Reviewed by: jhibbits, kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45082

arm64: Rename drop_to_el1 to enter_kernel_el

In the future we may not drop to EL1, e.g. when we support FEAT_VHE
where the kernel runs in EL2.

Reviewed by: emaste, imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43976

arm64: Fix typo in pagetable_l0_ttbr0_bootstrap symbol name

Obtained from: CheriBSD

arm64: Use the new CurrentEL macros

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43972

arm64: Remove pmap_san_bootstrap() and call kasan_init_early() directly

pmap_san_bootstrap() doesn't really do much, and it was hard-coding the
the bootstrap stack size defined in locore.S. Moreover, the name is a
bit confusing given the existence of pmap_bootstrap_san(). Just remove
it and call kasan_init_early() directly like we do on amd64. It will
not be used by KMSAN in a forthcoming patch series.

No functional change intended.

MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D43403

arm64: Set the Guarded Page flag in the kernel

Now the kernel and modules are built with branch protection we can
enablethe Guarded Page flag in the page tables. This causes indirect
branches to a location without a correct landing pad instruction to
raise an exception.

This should help mitigate some attacks where a function pointer is
changed to point somewhere other than the start of the function,
however it doesn't stop an attacker pointing it to an unintended
function.

Reviewed by: alc, scottph (both earlier version), markj
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation (earlier version)
Differential Revision: https://reviews.freebsd.org/D42080

arm64: Remove an old comment

This was missed when removing kern_delta

Sponsored by: Arm Ltd

arm64: Correct a comment in locore.S

We now use the physical address of get_load_phys_addr. Use it in a
comment rather than the old symbol.

Sponsored by: Arm Ltd

arm64: Clean up finding our load address

Use the linker to pre-calculate the offset of a known symbol from
KERNBASE, and use this to find the physical address KERNBASE should
map to.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42568

arm64: Make kern_delta unneeded in the boot params

Use pmap_early_vtophys to translate from a virtual to physical where
we were previously using the calculated delta. This means that, while
we still calculate it, we don't need to pass it to initarm or either
pmap bootstrap functions.

While here remove an unneeded printf that indirectly used it or was
related to the previous printf.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42567

arm64: Use adrp + :lo12: to load globals from asm

When loading a global variable we can use a pseudo-instruction similar
to "ldr, xn, =global" to load the address of the symbol. As this is
unlikely to be supported by a mov instruction a pc-relative load is
used, with the absolute address written at the end of the function so
it will be loaded.

This load can be partially replaced with an adrp instruction. This
generates the address, aligned to a 4k boundary, using a pc-relative
addition. Because the address is 4k-aligned we then update reading the
global variable using a load with the offset of the load the low
12-bits of the global. Arm64 assemblers have :lo12: to support this,
e.g. "ldr xn, [xn, :lo12:global]".

The only remaining users of "ldr, xn, =global" that I can find are
executed from the physical address space the kernel was loaded in and
need an address in the kernels virtual address space. Because of this
they can't use adrp.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42565

arm64: Initialize x18 for APs earlier during boot

When KMSAN is configured, the instrumentation inserts calls to
__msan_get_context_state() into all function prologues. The
implementation dereferences curthread and thus assumes that x18 points
to the PCPU area. This applies in particular to init_secondary(), which
currently is responsible for initializing x18 for APs.

Move initialization into locore to avoid this problem. No functional
change intended.

Reviewed by: kib, andrew
MFC after: 2 weeks
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D42533

arm64: Save an instruction in locore.S

We can move and sub at the same time, so let's do that.

Reviewed by: andrew, kevans
Pull Request: https://github.com/freebsd/freebsd-src/pull/794

arm64 locore.S: fix typos

Reviewed by: jhb, markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D42143

arm64: Add BTI landing pads to assembly functions

When we enable BTI iboth the first instruction in a function that could
be called indirectly, and a branch within a function need a valid
landing pad instruction.

There are three options for these instructions:
1. A breakpoint instruction
2. A pointer authentication PACIASP/PACIBSP
3. A BTI instruction

Option 1 will raise a breakpoint exception so isn't useable in either
cases. Option 2 could be used in some function entry cases, but needs
to be paired with an authentication instruction, and is normally only
used in non-leaf functions we can't use it in this case. This leaves
option 3.

There are four variants of the instruction, the C variant is used on
function entry and the J variant is for jumping within a function.
There is also a JC that works with both and one with no target that
works with neither.

Reviewed by: markj
Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation (earlier version)
Differential Revision: https://reviews.freebsd.org/D42078

Add BTI exceptions

We could hit these when executing code marked as using BTI but jumps
to a non-branch target instruction.

Sponsored by: Arm Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39450

sys: Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

arm64: use PSR_DAIF instead of each individual flag

No functional change intended.

Reviewed by: mhorne, andrew
Approved by: markj (mentor)
Differential Revision: https://reviews.freebsd.org/D40165

Move the arm64 sigcode to .rodata

The kernel doesn't execute this code, it's only ever copied to
userspace. Move it to .rodata as we don't need to modify it.

Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39399

arm64: Move the initial kernel stack out of the init_pagetables section

init_pagetables is mapped into the segment containing the BSS, but does
not get zeroed by locore. It is used for bootstrap page table pages.

It happens that the bootstrap kernel stack is also placed in that
section, but there's no reason it shouldn't live in the BSS, so move it
there. No functional change intended.

Reviewed by: andrew
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Juniper Networks
Differential Revision: https://reviews.freebsd.org/D39367

arm64: add KASAN support

This entails:
- Marking some obvious candidates for __nosanitizeaddress
- Similar trap frame markings as amd64, for similar reasons
- Shadow map implementation

The shadow map implementation is roughly similar to what was done on
amd64, with some exceptions. Attempting to use available space at
preinit_map_va + PMAP_PREINIT_MAPPING_SIZE (up to the end of that range,
as depicted in the physmap) results in odd failures, so we instead
search the physmap for free regions that we can carve out, fragmenting
the shadow map as necessary to try and fit as much as we need for the
initial kernel map. pmap_bootstrap_san() is thus after
pmap_bootstrap(), which still included some technically reserved areas
of the memory map that needed to be included in the DMAP.

The odd failure noted above may be a bug, but I haven't investigated it
all that much.

Initial work by mhorne with additional fixes from kevans and markj.

Reviewed by: andrew, markj
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36701

arm64: disable the physical timer for now if HCR_EL2.E2H is set

On some hardware, we can't clear HCR_EL2.E2H so accesses to the physical
timer hopelessly trap to EL2. Stash off the value of HCR_EL2 and use it
in has_hyp() to avoid this.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D38884

arm64: set FPEN if we're stuck with HCR_EL2.E2H

On Apple Silicon systems, E2H can't actually be cleared; we're stuck
with it. Check it again when we're setting up CPTR_EL2 and set FPEN
appropriately to avoid later trapping to EL2 on writes to SIMD
registers.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D38819

Split out the arm64 EL2 exception vectors

These were originally in locore.S as they are only needed so we have
a valid value to put into the vbar_el2 register. As these will soon
be used by bhyve so move them to a new file as we already have with
the EL1 exception vectors in exception.S.

Obtained from: https://github.com/FreeBSD-UPB/freebsd-src (earlier version)
Sponsored by: Innovate UK
Sponsored by: The FreeBSD Foundation

Put the arm64 vttbr_el2 register into a state

Zero the vttbr_el2 register on each CPU so we can tell if we are
running the host or guest kernel from a hypervisor.

Obtained from: https://github.com/FreeBSD-UPB/freebsd-src (earlier version)
Sponsored by: Innovate UK
Sponsored by: The FreeBSD Foundation

Remove unneeded variables in the arm64 pmap bootstrap

These are now unneeded after cleaning up the pmap bootstrap process.
Remove them and the variables that set them.

Sponsored by: The FreeBSD Foundation

Add experimental 16k page support on arm64

Add initial 16k page support on arm64. It is considered experimental,
with no guarantee of compatibility with a userspace or kernel modules
built with the current a 4k page size as code will likely try to pass
in a too small size when working with APIs that take a multiple of a
page, e.g. mmap.

As this is experimental, and because userspace and the kernel need to
have the PAGE_SIZE macro kept in sync there is no kernel option to
enable this. To test a new image should be built with the
PAGE_{SIZE,SHIFT,MASK} macros changed to the 16k versions.

There are currently known issues with loading modules from an old
loader as it can misalign them to load on a non-16k boundary.

Testing has shown good results in kernel workloads that allocate and
free large amounts of memory as only a quarter of the number of calls
into the VM subsystem are needed in the best case.

Reviewed by: markj
Tested by: gallatin
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34793

arm64: disable the EL2 MMU before dropping to EL1

An earlier stage may have set HCR_EL2.E2H, the clearing of which may
break address translation. We don't need the EL2 MMU at this point, so
we can avoid re-enabling it for now and just drop to EL1 as usual.

Suggested by: andrew
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D34644

Move the arm64 DMAP creation to C

To simplify the creation of the direct map (DMAP) region on arm64 move
it from the pre-C code into pmap. This simplifies the DMAP creation
as we can use the notmal index macros, and should reduce the number
of pages needed to hold the level 1 tables to just those needed.

Reviewed by: alc, dch
Tested by: dch, kevans
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34568

Remove L2_BLOCK_MASK from arm64

It's unneeded as it was just used to align KERNBASE to a level 2
block start address. KERNBASE was already aligned correctly.

Sponsored by: The FreeBSD Foundation

Fix the TCR_TG0 values

They are in a different order to the TCR_TG1 values but appear to have
been copied incorrectly.

While here use TCR_TG0_4K in locore.S to make it explicit the userspace
page size is 4K.

Sponsored by: The FreeBSD Foundation

Add arm64 pointer authentication support

Pointer authentication allows userspace to add instructions to insert
a Pointer Authentication Code (PAC) into a register based on an address
and modifier and check if the PAC is correct. If the check fails it will
either return an invalid address or fault to the kernel.

As many of these instructions are a NOP when disabled and in earlier
revisions of the architecture this can be used, for example, to sign
the return address before pushing it to the stack making Return-oriented
programming (ROP) attack more difficult on hardware that supports them.

The kernel manages five 128 bit signing keys: 2 instruction keys, 2 data
keys, and a generic key. The instructions then use one of these when
signing the registers. Instructions that use the first four store the
PAC in the register being signed, however the instructions that use the
generic key store the PAC in a separate register.

Currently all userspace threads share all the keys within a process
with a new set of userspace keys being generated when executing a new
process. This means a forked child will share its keys with its parent
until it calls an appropriate exec system call.

In the kernel we allow the use of one of the instruction keys, the ia
key. This will be used to sign return addresses in function calls.
Unlike userspace each kernel thread has its own randomly generated.

Thread0 has a static key as does the early code on secondary CPUs.
This should be safe as there is minimal user interaction with these
threads, however we could generate random keys when the Armv8.5
Random number generation instructions are present.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31261

Extract the arm64 signal code to a new file

This will be used by the vdso signal trampoline on arm64.

While here fix the license as this part of locore.S to correct the
copyright owner.

Sponsored by: The FreeBSD Foundation

Per-thread stack canary on arm64

With the update to llvm 13 we are able to tell the compiler it can find
the SSP canary relative to the register that holds the userspace stack
pointer. As this is unused in most of the kernel it can be used here
to point to a per-thread SSP canary.

As the kernel could be built with an old toolchain, e.g. when upgrading
from 13, add a warning that the options was enabled but the compiler
doesn't support it to both the build and kernel boot.

Discussed with: emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33079

Add the arm64 table attributes and use them

Add the table page table attributes on arm64 and use them to add
restrictions to the block and page entries below them. This ensures
we are unable to increase the permissions in these last level entries
without also changing them in the upper levels.

Use the attributes to ensure the kernel can't execute from userspace
memory and vice versa, userspace has no access to read or write kernel
memory, and that the DMAP region is non-executable.

Reviewed by: alc, kib
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32081

Add macros for the arm64 daifset/daifclr flags

Sponsored by: The FreeBSD Foundation

Fix the name of the arm64 SCTLR_E0E register

The character between the E's was the letter O, however in the Arm
Documentation and XML the character is the number 0 (zero).

Sponsored by: The FreeBSD Foundation

Clean up early arm64 pmap code

Early in the arm64 pmap code we need to translate between a virtual
address and a physical address. Rather than manually walking the page
table we can ask the hardware to do it for us.

Reviewed by: kib, markj
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D30357

arm64: Fix COMPAT_FREEBSD32.

The ENTRY() macro was modified by commit
28d945204ea1014d7de6906af8470ed8b3311335 to add an optional NOP instruction
at the beginning of the function. It is of course an arm64 instruction, so
unsuitable for the 32bits sigcode. So just use EENTRY() instead for
aarch32_sigcode. This should fix receiving signals when running 32bits
binaries on FreeBSD/arm64.

MFC After: 1 week

Fix creating the early arm64 level 2 blocks

In 48ba9b2669e6 we switched from creating level 1 blocks to smaller
level 2 blocks when creating the early arm64 page tables. On issue
was that they had a different meaning for register x7. The former used
it to hold page table attributes, while the latter held just the memory
type. This caused these attributes to be incorrectly shifted.

Fix this by changing the meaning of x7 to hold the block attributes
and fix the only caller that used the old meaning.

Most hardware seems to have handled the bits being off however qemu
failed to boot as reserved bits that should be zero were being set and
qemu fails to clear these when translating from a virtual address to a
physical address.

Sponsored by: Innovate UK

Use L2 blocks when in the identity map

This reduces the memory mapped to be closer to the minimal memory
needed to enable the MMU.

Reviewed by: mmel
Sponsored by: Innovate UK
Differential Revision:://reviews.freebsd.org/D27765

Add support for arm64 nGnRE device memory

On arm64 we can select how strongly we order device memory. Currently
we use the strongest type of non-Gathering, non-Reordering, no Early
write acknowledgement. This is equivalent to VM_MEMATTR_SO in the 32-bit
arm code.

Create a new memory type to remove the no Early write acknowledgement
option to create a memory attribute that is equivalent to the arm
VM_MEMATTR_DEVICE.

Keep the the old nGnRnE memory as what we provide for VM_MEMATTR_DEVICE
until we can test nGnRE on more hardware. A method for dynamically
switching back may be needed as at least one vendor is known to have
broken nGnRE memory.

Sponsored by: Innovate UK

arm64: Use new arm_kernel_boothdr script for generating booti images.

Use the base address for early arm64 page tables

Use the kernel physical base rather than the ttbr0 base when building
the kernel identity map. The latter is correct with current assumptions
but may not always be the case.

Sponsored by: Innovate UK

Mark all arm64 locore functions with ENTRY/LENTRY

It is useful to know where these are within the code, and will be
needed by later changes.

Sponsored by: Innovate UK

More the arm64 early page tables and stack to .bss

This removes 806k from the kernel ELF file that is only needed while
the kernel is running, not in the static file.

Sponsored by: Innovate UK

Improve address generation in the early arm64 boot

The adr instruction allows for an address of +-1M from the instruction.
If we replace these with an adrp and an add instruction we can generate
an address +-4G. The adrp will get an address of the 4k page the label
is within, and the add uses the :lo12: prefix to add just the low bits
to this address.

This will allow us to move things around with fewer issues than if we
needed to keep them within the +-1MB range.

Sponsored by: Innovate UK

Use a macro to find the offset of kern_ttbr0

Rather than hard coding the offset of kern_ttbr0 within arm64_bootparams
use a macro like the other fields.

Sponsored by: Innovate UK

Free the arm64 bootparams memory after initarm

This is only needed in initarm, we can return this memory to the stack
used by mi_startup.

Sponsored by: Innivate UK

Fix booting arm64 EFI with LINUX_BOOT_ABI enabled.
Use address of the pointer passed to kernel to determine whether the pointer
is a FDT block (physical address) or a module pointer (virtual kernel address).
This fragment was supposed to be committed before r366196, but I accidentally
skipped it in a patch series.

Reported by: bz

Reapply r366193 with proper commit log.

Don't map same physical memory multiple times with different cache attributes.
This is explicitly stated as architectural undefined behavior, leading to
coherency issues sooner or later.

Revert r366193, it was committed with unsaved commit log.

Don't map same physical memory multiple times with different cache attributes.
This is explicitly stated as architectural undefined behavior, leadint to
coherencz issues sonner or later.

Use ATTR_DEFAULT in the arm64 locore.S

We can use ATTR_DEFAULT directly in locore.S as it fits within an orr
instruction operand.

Sponsored by: Innovate UK

Switch to an empty ttbr0 pagetable when the MMU is enabled

We don't need these pagetables after the early boot. Remove the chance we
write to memory we didn't expect to and remove architectural undefined
behaviour.

Reviewed by: alc (earlier version), mmel
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D22606

Ensure the tlbi has completed before setting SCTLR

When enabling the MMU on arm64 we need to ensure the tlb invalidation has
completed before setting the enable bit in the SCTLR register.

Reported by: alc
Sponsored by: Innovate UK

Remove the secondary_stacks array in arm64 and riscv kernels.

Instead, dynamically allocate a page for the boot stack of each AP when
starting them up, like we do on x86. This shrinks the bss by
MAXCPU*KSTACK_PAGES pages, which corresponds to 4MB on arm64 and 256KB
on riscv.

Duplicate the logic used on x86 to free the bootstacks, by using a
sysinit to wait for each AP to switch to a thread before freeing its
stack.

While here, mark some static MD variables as such.

Reviewed by: kib
MFC after: 1 month
Sponsored by: Juniper Networks, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D24158

Store the boot exception level on arm64 so it can be queried later

A hypervisor, e.g. bhyve, will need to know what exception levelthe kernel
was in when it started booting. If it was EL2 we can then enable said
hypervisor.

Store the boot exception level and allow the kernel to later query it.

Obtained from: https://github.com/FreeBSD-UPB/freebsd (earlier version)
Sponsored by: Innovate UK

Generate the offsets for struct arm64_bootparams and use it in locore.S

This removes one place with hard coded offsets in locore.S

Sponsored by: Innovate UK

Split out the stage 1 pte bits and add the stage 2 bits

In preperation for adding bhyve support to arm64 we need to split the
stage 1 and stage 2 pte fields to allow future changes to create stage 2
page tables.

MFC after: 1 month
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D23669

Determine whether the MMU hardware is capable of updating a page table
entry's access flag and dirty state, and enable this feature when it's
available.

Ensure that we don't overlook a dirty state update that is concurrent
with a call to pmap_enter(). (Previously, all dirty state updates would
have occurred with the containing pmap's lock held, so a page table entry's
dirty state could not have changed while pmap_enter() held that same lock.)

Reviewed by: andrew, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D22907

Add comments and macros to the tcr_el1 setting code to help understand it.

This code is non-obvious when reading for the first time. To help with
understanding of it add comments explaining what it's doing.

While here use macros from armreg.h rather than magic numbers.

Sponsored by: DARPA, AFRL

Add support for booting kernel directly from U-Boot using booti command.

In some cases, like is locked bootstrap or device's inability to boot from
removable media, we cannot use standard boot sequence and is necessary to
boot kernel directly from U-Boot.

Discussed with: jhibbits
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D13861

revert r354714 "Boot arm64 kernel using booti command from U-boot."

After discussing with mmel@, it was clear this is insufficient to address
all the needs. mmel@ will commit his original patch, from
https://reviews.freebsd.org/D13861, and the additions needed from r354714
will be made afterward.

Requested by: mmel
Sponsored by: Juniper Networks, Inc.

Use the VM_MEMATTR macros to describe the MAIR offsets.

Remove the duplicate macros that defined a subset of the VM_MEMATTR values.
While here use VM_MEMATTR macros when filling in the MAIR register.

Reviewed by: alc, markj
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22241

Boot arm64 kernel using booti command from U-boot.

Summary:
Boot arm64 kernel using booti command from U-boot. booti can relocate initrd
image into higher ram addresses, therefore align the initrd load address to 1GiB
and create VA = PA map for it. Create L2 pagetable entries to copy the initrd
image into KVA.
(parts of the code in https://reviews.freebsd.org/D13861 was referred and used
as appropriate)

Submitted by: Siddharth Tuli <siddharthtuli_gmail.com>
Reviewed by: manu
Sponsored by: Juniper Networks, Inc
Differential Revision: https://reviews.freebsd.org/D22255

Utilize ASIDs to reduce both the direct and indirect costs of context
switching. The indirect costs being unnecessary TLB misses that are
incurred when ASIDs are not used. In fact, currently, when we perform a
context switch on one processor, we issue a broadcast TLB invalidation that
flushes the TLB contents on every processor.

Mark all user-space ("ttbr0") page table entries with the non-global flag so
that they are cached in the TLB under their ASID.

Correct an error in pmap_pinit0(). The pointer to the root of the page
table was being initialized to the root of the kernel-space page table
rather than a user-space page table. However, the root of the page table
that was being cached in process 0's md_l0addr field correctly pointed to a
user-space page table. As long as ASIDs weren't being used, this was
harmless, except that it led to some unnecessary page table switches in
pmap_switch(). Specifically, other kernel processes besides process 0 would
have their md_l0addr field set to the root of the kernel-space page table,
and so pmap_switch() would actually change page tables when switching
between process 0 and other kernel processes.

Implement a workaround for Cavium erratum 27456 affecting ThunderX machines.
(I would like to thank andrew@ for providing the code to detect the affected
machines.)

Address integer overflow in the definition of TCR_ASID_16.

Setup TCR according to the PARange and ASIDBits fields from
ID_AA64MMFR0_EL1. Previously, TCR_ASID_16 was unconditionally set.

Modify build_l1_block_pagetable so that lower attributes, such as ATTR_nG,
can be specified as a parameter.

Eliminate some unused code.

Earlier versions were tested to varying degrees by: andrew, emaste, markj

MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D21922

Set the userspace execute never bit on kernel mappings.

Arm64 allows us to create execute only mappings. To make sure userspace is
unable to accidentally execute kernel code set the user execute never
bit in the kernel page tables.

MFC after: 1 week
Sponsored by: DARPA, AFRL

Apply r353893 to arm64.

Reported by: Jenkins (hardware CI lab)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Impleent COMPAT_FREEBSD32 for arm64.
This is based on early work by andrew@.

Add the ARMv8.3 SCTLR_EL1 fields.

While here tag which architecture release fields were added and remove a
field that only existed in very early releases of the ARMv8 spec.

Sponsored by: DARPA, AFRL

Increase the number of pages we allocate in the arm64 early boot. We are
already close to the limit so increasing the kernel size may cause it to
fail to boot when it runs past the end of allocated memory.

Reported by: manu
Sponsored by: DARPA, AFRL

Rename assym.s to assym.inc

assym is only to be included by other .s files, and should never
actually be assembled by itself.

Reviewed by: imp, bdrewery (earlier)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D14180

Initialize CONTEXTIDR register on secondary cores by zero,
not with undefined value from X1 register.

MFC after: 1 month

Use the module pointer to find the address we need to map to in the early
arm64 boot sequence. This will be a virtual address in the kernel space
after the kernel and any modules loaded by loader so we can use this to
find the size of the kernel + modules. We can then add on a level 2 page for
the module data and round up the size to be aligned to a level 2 page.

This allows more than 8 MiB of modules to be loaded by loader, e.g. zfs.ko
and opensolaris.ko.

Reported by: Shawn Webb
MFC after: 1 week
Sponsored by: DARPA, AFRL

Set CP15BEN in SCTLR to make memory barriers work in 32-bit mode.

Binaries generated by Clang for ARMv6 may contain these instructions:

MCR p15, 0, <Rd>, c7, c10, 5

These instructions are deprecated as of ARMv7, which is why modern
processors have a way of toggling support for them. On FreeBSD/arm64 we
currently disable support for these instructions, meaning that if 32-bit
executables with these instructions are run, they would crash with
SIGILL. This is likely not what we want.

Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D13145

Add SCTLR bits added in ARMv8.1 and ARMv8.2 and start to use them in the
early boot code.

Sponsored by: DARPA, AFRL

Create macros for the MAIR memory attributes. While here add an uncached
memory type, however the VM code still needs to be taught about this.

MFC after: 1 week
Sponsored by: ABT Systems Ltd

Increase the arm64 kernel address space to 512GB, and the DMAP region to
2TB. The latter can be increased in 512GB chunks by adjusting the lower
address, however more work will be needed to increase the former.

There is still some work needed to only create a DMAP region for the RAM
address space as on ARM architectures all mappings should have the same
memory attributes, and these will be different for device and normal memory.

Reviewed by: kib
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5859

Cleanup the early pagetable creation code in preperation for increasing
the size of the arm64 DMAP region.

Approved by: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Allow vmparam.h to be included from assembly files on arm64.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add support for 4 level pagetables. The userland address space has been
increased to 256TiB. The kernel address space can also be increased to be
the same size, but this will be performed in a later change.

To help work with an extra level of page tables two new functions have
been added, one to file the lowest level table entry, and one to find the
block/page level. Both of these find the entry for a given pmap and virtual
address.

This has been tested with a combination of buildworld, stress2 tests, and
by using sort to consume a large amount of memory by sorting /dev/zero. No
new issues are known to be present from this change.

Reviewed by: kib
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5720

Enable SRE_EL2 on ARM64

Enable system register access for EL2. Alpine-V2 is
the first device requiring this to be enabled.
It is also in-sync with Linux initialization code,
and compatible with Alpine-V2 uboot requirements.

Obtained from: Semihalf
Submitted by: Michal Stanek <mst@semihalf.com>
Sponsored by: Annapurna Labs
Approved by: cognet (mentor)
Reviewed by: wma
Differential revision: https://reviews.freebsd.org/D5394

Enable checking of the stack alignment. The stack should be aligned to a
16-byte value. With this the hardware will check if a memory access uses
an incorrectly aligned stack pointer as the base address.

Sponsored by: ABT Systems Ltd

Increase the space we use after the kernel to 8MiB. On 2GiB HiKey board we
would try to access data past this point stopping the boot.

Sponsored by: ABT Systems Ltd

Use 4 levels of page tables when enabling the MMU. This will allow us to
boot on an SoC that places physical memory at an address past where three
levels of page tables can access in an identity mapping.

Submitted by: Wojciech Macek <wma@semihalf.com>,
Patrick Wildt <patrick@bitrig.org>
Differential Revision: https://reviews.freebsd.org/D3885 (partial)
Differential Revision: https://reviews.freebsd.org/D3744

Replace build_section_pagetable with build_l1_block_pagetable as it takes
an extra argument to specify the number of 1GiB pages to map. This should
be a nop as we are only mapping a single page, but when we move to use an
extra level of page tables we will be able to map a second block, e.g. if
the kernel was loaded over a 1GiB boundary.

Rename build_block_pagetable to build_l2_block_pagetable in preperation
for adding support for 4 levels of page tables.

Obtained from: Patrick Wildt <patrick@bitrig.org>

When dropping to EL1 ensure we have written to all special registers by
moving the instruction barrier to just before we drop exception level.

Sponsored by: ABT Systems Ltd

Set the counter-timer virtual offset to a know value, it may not have been
set by the boot code and are reset to an implementation defined value that
may be unknown.

Sponsored by: ABT Systems Ltd

Fix secondary stacks calculation on ARM64

Secondary stack calculation is modified to provide
stack_top = secondary_stacks + (cpu_id) * PAGE_SIZE * KSTACK_PAGES
because on ARM64 the stack grows to lower memory addresses.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3107

Fix KSTACK_PAGES issue when the default value was changed in KERNCONF

If KSTACK_PAGES was changed to anything alse than the default,
the value from param.h was taken instead in some places and
the value from KENRCONF in some others. This resulted in
inconsistency which caused corruption in SMP envorinment.

Ensure all places where KSTACK_PAGES are used the opt_kstack_pages.h
is included.

The file opt_kstack_pages.h could not be included in param.h
because was breaking the toolchain compilation.

Reviewed by: kib
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3094

Set-up proper TCR values for memory related to Translation Table Walking

This commit adds proper cache and shareability attributes to
the TCR register.
Set memory attributes to Normal, outer and inner cacheable WBWA.
Set shareability to inner and outer shareable when SMP is enabled.

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3093

Set memory to be inner-sharable. This isn't needed on device memory as the
MMU will ignore the attribute there, howeverit simplifies to code to alwas
set it.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add support for SMP. This uses the FDT data to find the CPUs to start on,
and psci to start them. I expect ACPI support to be added later.

This has been tested on qemu with 2 cpus as that is the current value of
MAXCPUS. This is expected to be increased in the future as FreeBSD has
been tested on 48 cores on the Cavium ThunderX hardware.

Partially based on a patch from Robin Randhawa from ARM.

Approved by: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3024

Add support for ARM GICv3 interrupt controller used in some ARM64 chips

GICv3 allows to distribute interrupts to more than 8 cores served by
the previous GIC revisions. GICv3 introduces additional logic in form
of Re-Distributors associated with particular CPUs to determine
the highest priority interrupts and manage PPIs and LPIs
(Locality-specific Peripheral Interrupts). Interrupts routing is
based on CPUs' affinity numbers. CPU interface was changed to be
accessible via CPU System Registers and this is the preferred
(and supported) method in this driver.

Obtained from: Semihalf
Reviewed by: andrew, emaste, ian, imp
Sponsored by: The FreeBSD Foundation

Bring in the start of the arm64 kernel.

This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.

The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.

This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.

Differential Revision: https://reviews.freebsd.org/D2199
Reviewed by: emaste, imp
Relnotes: yes
Sponsored by: The FreeBSD Foundation