Simplify fix for PR toolchain/57146

Introduce ARCH_STRIP_SYMBOLS variable to centralize logic for debug
symbols from MD Makefile's to Makefile.kern.inc.

Fix kernel size inflation for arm and aarch64 (PR toolchain/57146)

For some conditions, SYSTEM_LD_TAIL is set for arm and aarch64.
Then, ctfmerge(1) in default SYSTEM_LD_TAIL is unintentionally
skipped, which results in the catastrophic kernel size inflation,
as reported in the PR.

Also, introduce and use OBJCOPY_STRIPFLAGS variable instead of
STRIPFLAGS, as strip(1) is replaced by objcopy(1) during MI
kernel build procedure.

XXX
For Makefile.{arm,aarch64}, weird logic is used to determine how
to handle debug symbols; MKDEBUG{,KERNEL} are taken into account
later in sys/conf/Makefile.kern.inc.

In gcc10, -msign-return-address is no longer supported.
Instead, (LLVM-compatible) -mbranch-protection option is supported.

branches: 1.22.4; 1.22.6;
add support kernel profiling on aarch64

- add MCOUNT_ENTER, MCOUNT_EXIT macro
- __mcount() function should be aligned
- add "-fno-optimize-sibling-calls" option when PROF. for accurate profiling, it is better to suppress the tail call.

branches: 1.21.2;
"options ARMV83_PAC" is now supported for gcc as well.

- add "-msign-return-address=all" to CFLAGS for gcc when specified options ARMV83_PAC
- AARCH64REG_{READ,WRITE}_INLINE3 macro can now use the APIAKey registers in both gcc and llvm.
llvm requires asm(".arch armv8.3-a"), whereas gcc requires __attribute__((target("arch=armv8.3-a"))).
- use ".arch armv8.3-a" rather than ".arch armv8.3-a+pac" in *.S for llvm.

Add KASAN instrumentation on on-stack VLAs, same as amd64.

Add support for Branch Target Identification (BTI).

On the executable pages that have the GP (Guarded Page) bit, the semantic
of the "br" and "blr" instructions is changed: the CPU expects the first
instruction of the jump/call target to be "bti", and faults if it isn't.

We add the GP bit on the kernel .text pages (and incidentally the .rodata
pages, but we don't care). The compiler adds a "bti c" instruction at the
beginning of each C function. We modify the ENTRY() macros to manually add
"bti c" in the asm functions.

cpuswitch.S needs a specific change: with "br x27" the CPU expects "bti j",
which is bad because the functions begin with "bti c"; switch to "br x16",
for the CPU to accept "bti c".

BTI helps defend against JOP/COP. Tested on Qemu.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.17.2;
change kernel vm base address to use more than 256GB of memory. (up to 64TB)

also enlarge KSEG(direct map) region from 512GB to 64TB.
KASAN works ok.

Note: -fasan-shadow-offset=
KASAN_SHADOW_START - (CANONICAL_BASE >> 3) =
0xFFFF400000000000 - (0xFFFF000000000000 >> 3) =
0xDFFF600000000000

Fix alignment of .text section by changing load address to
0xffffffc000000000 and adding 64 bytes of padding before the entry point.

branches: 1.15.4;
Add KASAN use-after-scope detection in aarch64, tested by Ryo Shimizu,
thanks.

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

- Determine KERN_AS automatically depending on whether OPT_MODULAR is
set or not, in the same way as libcompat.

- Specify OPT_MODULAR in the port Makefile instead of KERN_AS.

Now, KERN_AS=library is used for kernels without module(7) for all ports.

OK christos

s/A64/ARM/

no functional change

branches: 1.10.2;
locore.S is a MD_SFILES.

This keeps the dependency handling in the loop, so rebuilds after
changing options, say EARLYCONS, don't fail.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.8.2;
- Allow multiple .BEGIN targets
- Make their protection consistent

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Fix kernel size inflation for arm and aarch64 (PR toolchain/57146)

For some conditions, SYSTEM_LD_TAIL is set for arm and aarch64.
Then, ctfmerge(1) in default SYSTEM_LD_TAIL is unintentionally
skipped, which results in the catastrophic kernel size inflation,
as reported in the PR.

Also, introduce and use OBJCOPY_STRIPFLAGS variable instead of
STRIPFLAGS, as strip(1) is replaced by objcopy(1) during MI
kernel build procedure.

XXX
For Makefile.{arm,aarch64}, weird logic is used to determine how
to handle debug symbols; MKDEBUG{,KERNEL} are taken into account
later in sys/conf/Makefile.kern.inc.

In gcc10, -msign-return-address is no longer supported.
Instead, (LLVM-compatible) -mbranch-protection option is supported.

branches: 1.22.4; 1.22.6;
add support kernel profiling on aarch64

- add MCOUNT_ENTER, MCOUNT_EXIT macro
- __mcount() function should be aligned
- add "-fno-optimize-sibling-calls" option when PROF. for accurate profiling, it is better to suppress the tail call.

branches: 1.21.2;
"options ARMV83_PAC" is now supported for gcc as well.

- add "-msign-return-address=all" to CFLAGS for gcc when specified options ARMV83_PAC
- AARCH64REG_{READ,WRITE}_INLINE3 macro can now use the APIAKey registers in both gcc and llvm.
llvm requires asm(".arch armv8.3-a"), whereas gcc requires __attribute__((target("arch=armv8.3-a"))).
- use ".arch armv8.3-a" rather than ".arch armv8.3-a+pac" in *.S for llvm.

Add KASAN instrumentation on on-stack VLAs, same as amd64.

Add support for Branch Target Identification (BTI).

On the executable pages that have the GP (Guarded Page) bit, the semantic
of the "br" and "blr" instructions is changed: the CPU expects the first
instruction of the jump/call target to be "bti", and faults if it isn't.

We add the GP bit on the kernel .text pages (and incidentally the .rodata
pages, but we don't care). The compiler adds a "bti c" instruction at the
beginning of each C function. We modify the ENTRY() macros to manually add
"bti c" in the asm functions.

cpuswitch.S needs a specific change: with "br x27" the CPU expects "bti j",
which is bad because the functions begin with "bti c"; switch to "br x16",
for the CPU to accept "bti c".

BTI helps defend against JOP/COP. Tested on Qemu.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.17.2;
change kernel vm base address to use more than 256GB of memory. (up to 64TB)

also enlarge KSEG(direct map) region from 512GB to 64TB.
KASAN works ok.

Note: -fasan-shadow-offset=
KASAN_SHADOW_START - (CANONICAL_BASE >> 3) =
0xFFFF400000000000 - (0xFFFF000000000000 >> 3) =
0xDFFF600000000000

Fix alignment of .text section by changing load address to
0xffffffc000000000 and adding 64 bytes of padding before the entry point.

branches: 1.15.4;
Add KASAN use-after-scope detection in aarch64, tested by Ryo Shimizu,
thanks.

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

- Determine KERN_AS automatically depending on whether OPT_MODULAR is
set or not, in the same way as libcompat.

- Specify OPT_MODULAR in the port Makefile instead of KERN_AS.

Now, KERN_AS=library is used for kernels without module(7) for all ports.

OK christos

s/A64/ARM/

no functional change

branches: 1.10.2;
locore.S is a MD_SFILES.

This keeps the dependency handling in the loop, so rebuilds after
changing options, say EARLYCONS, don't fail.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.8.2;
- Allow multiple .BEGIN targets
- Make their protection consistent

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

In gcc10, -msign-return-address is no longer supported.
Instead, (LLVM-compatible) -mbranch-protection option is supported.

add support kernel profiling on aarch64

- add MCOUNT_ENTER, MCOUNT_EXIT macro
- __mcount() function should be aligned
- add "-fno-optimize-sibling-calls" option when PROF. for accurate profiling, it is better to suppress the tail call.

branches: 1.21.2;
"options ARMV83_PAC" is now supported for gcc as well.

- add "-msign-return-address=all" to CFLAGS for gcc when specified options ARMV83_PAC
- AARCH64REG_{READ,WRITE}_INLINE3 macro can now use the APIAKey registers in both gcc and llvm.
llvm requires asm(".arch armv8.3-a"), whereas gcc requires __attribute__((target("arch=armv8.3-a"))).
- use ".arch armv8.3-a" rather than ".arch armv8.3-a+pac" in *.S for llvm.

Add KASAN instrumentation on on-stack VLAs, same as amd64.

Add support for Branch Target Identification (BTI).

On the executable pages that have the GP (Guarded Page) bit, the semantic
of the "br" and "blr" instructions is changed: the CPU expects the first
instruction of the jump/call target to be "bti", and faults if it isn't.

We add the GP bit on the kernel .text pages (and incidentally the .rodata
pages, but we don't care). The compiler adds a "bti c" instruction at the
beginning of each C function. We modify the ENTRY() macros to manually add
"bti c" in the asm functions.

cpuswitch.S needs a specific change: with "br x27" the CPU expects "bti j",
which is bad because the functions begin with "bti c"; switch to "br x16",
for the CPU to accept "bti c".

BTI helps defend against JOP/COP. Tested on Qemu.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.17.2;
change kernel vm base address to use more than 256GB of memory. (up to 64TB)

also enlarge KSEG(direct map) region from 512GB to 64TB.
KASAN works ok.

Note: -fasan-shadow-offset=
KASAN_SHADOW_START - (CANONICAL_BASE >> 3) =
0xFFFF400000000000 - (0xFFFF000000000000 >> 3) =
0xDFFF600000000000

Fix alignment of .text section by changing load address to
0xffffffc000000000 and adding 64 bytes of padding before the entry point.

branches: 1.15.4;
Add KASAN use-after-scope detection in aarch64, tested by Ryo Shimizu,
thanks.

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

- Determine KERN_AS automatically depending on whether OPT_MODULAR is
set or not, in the same way as libcompat.

- Specify OPT_MODULAR in the port Makefile instead of KERN_AS.

Now, KERN_AS=library is used for kernels without module(7) for all ports.

OK christos

s/A64/ARM/

no functional change

branches: 1.10.2;
locore.S is a MD_SFILES.

This keeps the dependency handling in the loop, so rebuilds after
changing options, say EARLYCONS, don't fail.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.8.2;
- Allow multiple .BEGIN targets
- Make their protection consistent

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

add support kernel profiling on aarch64

- add MCOUNT_ENTER, MCOUNT_EXIT macro
- __mcount() function should be aligned
- add "-fno-optimize-sibling-calls" option when PROF. for accurate profiling, it is better to suppress the tail call.

"options ARMV83_PAC" is now supported for gcc as well.

- add "-msign-return-address=all" to CFLAGS for gcc when specified options ARMV83_PAC
- AARCH64REG_{READ,WRITE}_INLINE3 macro can now use the APIAKey registers in both gcc and llvm.
llvm requires asm(".arch armv8.3-a"), whereas gcc requires __attribute__((target("arch=armv8.3-a"))).
- use ".arch armv8.3-a" rather than ".arch armv8.3-a+pac" in *.S for llvm.

Add KASAN instrumentation on on-stack VLAs, same as amd64.

Add support for Branch Target Identification (BTI).

On the executable pages that have the GP (Guarded Page) bit, the semantic
of the "br" and "blr" instructions is changed: the CPU expects the first
instruction of the jump/call target to be "bti", and faults if it isn't.

We add the GP bit on the kernel .text pages (and incidentally the .rodata
pages, but we don't care). The compiler adds a "bti c" instruction at the
beginning of each C function. We modify the ENTRY() macros to manually add
"bti c" in the asm functions.

cpuswitch.S needs a specific change: with "br x27" the CPU expects "bti j",
which is bad because the functions begin with "bti c"; switch to "br x16",
for the CPU to accept "bti c".

BTI helps defend against JOP/COP. Tested on Qemu.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.17.2;
change kernel vm base address to use more than 256GB of memory. (up to 64TB)

also enlarge KSEG(direct map) region from 512GB to 64TB.
KASAN works ok.

Note: -fasan-shadow-offset=
KASAN_SHADOW_START - (CANONICAL_BASE >> 3) =
0xFFFF400000000000 - (0xFFFF000000000000 >> 3) =
0xDFFF600000000000

Fix alignment of .text section by changing load address to
0xffffffc000000000 and adding 64 bytes of padding before the entry point.

branches: 1.15.4;
Add KASAN use-after-scope detection in aarch64, tested by Ryo Shimizu,
thanks.

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

- Determine KERN_AS automatically depending on whether OPT_MODULAR is
set or not, in the same way as libcompat.

- Specify OPT_MODULAR in the port Makefile instead of KERN_AS.

Now, KERN_AS=library is used for kernels without module(7) for all ports.

OK christos

s/A64/ARM/

no functional change

branches: 1.10.2;
locore.S is a MD_SFILES.

This keeps the dependency handling in the loop, so rebuilds after
changing options, say EARLYCONS, don't fail.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.8.2;
- Allow multiple .BEGIN targets
- Make their protection consistent

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

"options ARMV83_PAC" is now supported for gcc as well.

- add "-msign-return-address=all" to CFLAGS for gcc when specified options ARMV83_PAC
- AARCH64REG_{READ,WRITE}_INLINE3 macro can now use the APIAKey registers in both gcc and llvm.
llvm requires asm(".arch armv8.3-a"), whereas gcc requires __attribute__((target("arch=armv8.3-a"))).
- use ".arch armv8.3-a" rather than ".arch armv8.3-a+pac" in *.S for llvm.

Add KASAN instrumentation on on-stack VLAs, same as amd64.

Add support for Branch Target Identification (BTI).

On the executable pages that have the GP (Guarded Page) bit, the semantic
of the "br" and "blr" instructions is changed: the CPU expects the first
instruction of the jump/call target to be "bti", and faults if it isn't.

We add the GP bit on the kernel .text pages (and incidentally the .rodata
pages, but we don't care). The compiler adds a "bti c" instruction at the
beginning of each C function. We modify the ENTRY() macros to manually add
"bti c" in the asm functions.

cpuswitch.S needs a specific change: with "br x27" the CPU expects "bti j",
which is bad because the functions begin with "bti c"; switch to "br x16",
for the CPU to accept "bti c".

BTI helps defend against JOP/COP. Tested on Qemu.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.17.2;
change kernel vm base address to use more than 256GB of memory. (up to 64TB)

also enlarge KSEG(direct map) region from 512GB to 64TB.
KASAN works ok.

Note: -fasan-shadow-offset=
KASAN_SHADOW_START - (CANONICAL_BASE >> 3) =
0xFFFF400000000000 - (0xFFFF000000000000 >> 3) =
0xDFFF600000000000

Fix alignment of .text section by changing load address to
0xffffffc000000000 and adding 64 bytes of padding before the entry point.

branches: 1.15.4;
Add KASAN use-after-scope detection in aarch64, tested by Ryo Shimizu,
thanks.

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

- Determine KERN_AS automatically depending on whether OPT_MODULAR is
set or not, in the same way as libcompat.

- Specify OPT_MODULAR in the port Makefile instead of KERN_AS.

Now, KERN_AS=library is used for kernels without module(7) for all ports.

OK christos

s/A64/ARM/

no functional change

branches: 1.10.2;
locore.S is a MD_SFILES.

This keeps the dependency handling in the loop, so rebuilds after
changing options, say EARLYCONS, don't fail.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.8.2;
- Allow multiple .BEGIN targets
- Make their protection consistent

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add KASAN instrumentation on on-stack VLAs, same as amd64.

Add support for Branch Target Identification (BTI).

On the executable pages that have the GP (Guarded Page) bit, the semantic
of the "br" and "blr" instructions is changed: the CPU expects the first
instruction of the jump/call target to be "bti", and faults if it isn't.

We add the GP bit on the kernel .text pages (and incidentally the .rodata
pages, but we don't care). The compiler adds a "bti c" instruction at the
beginning of each C function. We modify the ENTRY() macros to manually add
"bti c" in the asm functions.

cpuswitch.S needs a specific change: with "br x27" the CPU expects "bti j",
which is bad because the functions begin with "bti c"; switch to "br x16",
for the CPU to accept "bti c".

BTI helps defend against JOP/COP. Tested on Qemu.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

change kernel vm base address to use more than 256GB of memory. (up to 64TB)

also enlarge KSEG(direct map) region from 512GB to 64TB.
KASAN works ok.

Note: -fasan-shadow-offset=
KASAN_SHADOW_START - (CANONICAL_BASE >> 3) =
0xFFFF400000000000 - (0xFFFF000000000000 >> 3) =
0xDFFF600000000000

Fix alignment of .text section by changing load address to
0xffffffc000000000 and adding 64 bytes of padding before the entry point.

branches: 1.15.4;
Add KASAN use-after-scope detection in aarch64, tested by Ryo Shimizu,
thanks.

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

- Determine KERN_AS automatically depending on whether OPT_MODULAR is
set or not, in the same way as libcompat.

- Specify OPT_MODULAR in the port Makefile instead of KERN_AS.

Now, KERN_AS=library is used for kernels without module(7) for all ports.

OK christos

s/A64/ARM/

no functional change

branches: 1.10.2;
locore.S is a MD_SFILES.

This keeps the dependency handling in the loop, so rebuilds after
changing options, say EARLYCONS, don't fail.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.8.2;
- Allow multiple .BEGIN targets
- Make their protection consistent

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

change kernel vm base address to use more than 256GB of memory. (up to 64TB)

also enlarge KSEG(direct map) region from 512GB to 64TB.
KASAN works ok.

Note: -fasan-shadow-offset=
KASAN_SHADOW_START - (CANONICAL_BASE >> 3) =
0xFFFF400000000000 - (0xFFFF000000000000 >> 3) =
0xDFFF600000000000

Fix alignment of .text section by changing load address to
0xffffffc000000000 and adding 64 bytes of padding before the entry point.

branches: 1.15.4;
Add KASAN use-after-scope detection in aarch64, tested by Ryo Shimizu,
thanks.

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

- Determine KERN_AS automatically depending on whether OPT_MODULAR is
set or not, in the same way as libcompat.

- Specify OPT_MODULAR in the port Makefile instead of KERN_AS.

Now, KERN_AS=library is used for kernels without module(7) for all ports.

OK christos

s/A64/ARM/

no functional change

branches: 1.10.2;
locore.S is a MD_SFILES.

This keeps the dependency handling in the loop, so rebuilds after
changing options, say EARLYCONS, don't fail.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.8.2;
- Allow multiple .BEGIN targets
- Make their protection consistent

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

change kernel vm base address to use more than 256GB of memory. (up to 64TB)

also enlarge KSEG(direct map) region from 512GB to 64TB.
KASAN works ok.

Note: -fasan-shadow-offset=
KASAN_SHADOW_START - (CANONICAL_BASE >> 3) =
0xFFFF400000000000 - (0xFFFF000000000000 >> 3) =
0xDFFF600000000000

Fix alignment of .text section by changing load address to
0xffffffc000000000 and adding 64 bytes of padding before the entry point.

branches: 1.15.4;
Add KASAN use-after-scope detection in aarch64, tested by Ryo Shimizu,
thanks.

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

- Determine KERN_AS automatically depending on whether OPT_MODULAR is
set or not, in the same way as libcompat.

- Specify OPT_MODULAR in the port Makefile instead of KERN_AS.

Now, KERN_AS=library is used for kernels without module(7) for all ports.

OK christos

s/A64/ARM/

no functional change

branches: 1.10.2;
locore.S is a MD_SFILES.

This keeps the dependency handling in the loop, so rebuilds after
changing options, say EARLYCONS, don't fail.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.8.2;
- Allow multiple .BEGIN targets
- Make their protection consistent

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Fix alignment of .text section by changing load address to
0xffffffc000000000 and adding 64 bytes of padding before the entry point.

Add KASAN use-after-scope detection in aarch64, tested by Ryo Shimizu,
thanks.

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

- Determine KERN_AS automatically depending on whether OPT_MODULAR is
set or not, in the same way as libcompat.

- Specify OPT_MODULAR in the port Makefile instead of KERN_AS.

Now, KERN_AS=library is used for kernels without module(7) for all ports.

OK christos

s/A64/ARM/

no functional change

branches: 1.10.2;
locore.S is a MD_SFILES.

This keeps the dependency handling in the loop, so rebuilds after
changing options, say EARLYCONS, don't fail.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.8.2;
- Allow multiple .BEGIN targets
- Make their protection consistent

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add KASAN use-after-scope detection in aarch64, tested by Ryo Shimizu,
thanks.

Track the stack with kASan on aarch64. Same principle as on amd64. Illegal
accesses occurring there are now detected.

Originally written by me, but reworked by ryo@, thanks.

Add kASan support for aarch64. Stack tracking needs more investigation
and will come in a separate commit.

Reviewed by ryo@ jmcneill@ skrll@.

- Determine KERN_AS automatically depending on whether OPT_MODULAR is
set or not, in the same way as libcompat.

- Specify OPT_MODULAR in the port Makefile instead of KERN_AS.

Now, KERN_AS=library is used for kernels without module(7) for all ports.

OK christos

s/A64/ARM/

no functional change

locore.S is a MD_SFILES.

This keeps the dependency handling in the loop, so rebuilds after
changing options, say EARLYCONS, don't fail.

Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.8.2;
- Allow multiple .BEGIN targets
- Make their protection consistent

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

- Allow multiple .BEGIN targets
- Make their protection consistent

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Fix config(1) errors and warnings.

Set up arm headers for the build.

Define ${LINKSCRIPT} in one place.

Use ${KERNLDSCRIPT}.

branches: 1.4.2;
Use LINKSCRIPT.

branches: 1.3.2;
Reorganize symbol table embedding. The existing option SYMTAB_SPACE is
replaced by the make option COPY_SYMTAB set to any value. The copy of
the symbol table is no longer put into a buffer in kern_ksyms.o, but a
small helper object. This object is build first with a dummy size, then
the kernel is linked to compute the real dimension of the symbol table
buffer. After that, the helper object is rebuild and the kernel linked
again.

Use wildcards for stripping/preserving the mapping symbols on ARM and
AArch64. LLVM creates unique symbols in each file of the form $a.n etc.

branches: 1.1.2;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.