Rename VM_KERNEL_IO_ADDRESS to VM_KERNEL_IO_BASE to match RISC-V

It's less letters, matches other similar variables and will help with
sharing code between the two architectures.

NFCI.

Remove GIC_SPLFUNCS.

Add __HAVE_PREEMPTION support to gic_splfuncs asm funcs.

"looks right to me" - thorpej

Add CI_SPLX_SAVEDIPL and CI_HWPL

Ensure TCR_EPD0 is set on entry to pmap_activate and ensure it is set as
early as possible for APs.

gic_splx: performance optimizations

Avoid any kind of register access (DAIF, PMR, etc), barriers, and atomic
operations in the common case where no interrupt fires between spl being
raised and lowered.

This introduces a per-CPU return address (ci_splx_restart) used by the
vector handler to restart a sequence in splx that compares the new ipl
with the per-CPU hardware priority state stored in ci_hwpl.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

AA64 is not MIPS.

Change all KSEG references to directmap

branches: 1.31.2;
fix typo

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

fix build failure without options MULTIPROCESSOR

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

set kernel text/rodata readonly when not defined DDB.
set readonly segment on 2Mbytes aligned. (kernel image is mapped with 2Mbytes L2 block)

fix build with aarch64 gcc/gas

allow to read CNTVCT_EL0 and CNTFRQ_EL0 from EL0

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Remove GIC_SPLFUNCS.

Add __HAVE_PREEMPTION support to gic_splfuncs asm funcs.

"looks right to me" - thorpej

Add CI_SPLX_SAVEDIPL and CI_HWPL

Ensure TCR_EPD0 is set on entry to pmap_activate and ensure it is set as
early as possible for APs.

gic_splx: performance optimizations

Avoid any kind of register access (DAIF, PMR, etc), barriers, and atomic
operations in the common case where no interrupt fires between spl being
raised and lowered.

This introduces a per-CPU return address (ci_splx_restart) used by the
vector handler to restart a sequence in splx that compares the new ipl
with the per-CPU hardware priority state stored in ci_hwpl.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

AA64 is not MIPS.

Change all KSEG references to directmap

branches: 1.31.2;
fix typo

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

fix build failure without options MULTIPROCESSOR

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

set kernel text/rodata readonly when not defined DDB.
set readonly segment on 2Mbytes aligned. (kernel image is mapped with 2Mbytes L2 block)

fix build with aarch64 gcc/gas

allow to read CNTVCT_EL0 and CNTFRQ_EL0 from EL0

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Add __HAVE_PREEMPTION support to gic_splfuncs asm funcs.

"looks right to me" - thorpej

Add CI_SPLX_SAVEDIPL and CI_HWPL

Ensure TCR_EPD0 is set on entry to pmap_activate and ensure it is set as
early as possible for APs.

gic_splx: performance optimizations

Avoid any kind of register access (DAIF, PMR, etc), barriers, and atomic
operations in the common case where no interrupt fires between spl being
raised and lowered.

This introduces a per-CPU return address (ci_splx_restart) used by the
vector handler to restart a sequence in splx that compares the new ipl
with the per-CPU hardware priority state stored in ci_hwpl.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

AA64 is not MIPS.

Change all KSEG references to directmap

branches: 1.31.2;
fix typo

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

fix build failure without options MULTIPROCESSOR

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

set kernel text/rodata readonly when not defined DDB.
set readonly segment on 2Mbytes aligned. (kernel image is mapped with 2Mbytes L2 block)

fix build with aarch64 gcc/gas

allow to read CNTVCT_EL0 and CNTFRQ_EL0 from EL0

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Ensure TCR_EPD0 is set on entry to pmap_activate and ensure it is set as
early as possible for APs.

gic_splx: performance optimizations

Avoid any kind of register access (DAIF, PMR, etc), barriers, and atomic
operations in the common case where no interrupt fires between spl being
raised and lowered.

This introduces a per-CPU return address (ci_splx_restart) used by the
vector handler to restart a sequence in splx that compares the new ipl
with the per-CPU hardware priority state stored in ci_hwpl.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

AA64 is not MIPS.

Change all KSEG references to directmap

branches: 1.31.2;
fix typo

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

fix build failure without options MULTIPROCESSOR

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

set kernel text/rodata readonly when not defined DDB.
set readonly segment on 2Mbytes aligned. (kernel image is mapped with 2Mbytes L2 block)

fix build with aarch64 gcc/gas

allow to read CNTVCT_EL0 and CNTFRQ_EL0 from EL0

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

gic_splx: performance optimizations

Avoid any kind of register access (DAIF, PMR, etc), barriers, and atomic
operations in the common case where no interrupt fires between spl being
raised and lowered.

This introduces a per-CPU return address (ci_splx_restart) used by the
vector handler to restart a sequence in splx that compares the new ipl
with the per-CPU hardware priority state stored in ci_hwpl.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

AA64 is not MIPS.

Change all KSEG references to directmap

branches: 1.31.2;
fix typo

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

fix build failure without options MULTIPROCESSOR

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

set kernel text/rodata readonly when not defined DDB.
set readonly segment on 2Mbytes aligned. (kernel image is mapped with 2Mbytes L2 block)

fix build with aarch64 gcc/gas

allow to read CNTVCT_EL0 and CNTFRQ_EL0 from EL0

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

s:aarch64/cpufunc.h:arm/cpufunc.h:

a baby step in the grand arm header unification challenge

AA64 is not MIPS.

Change all KSEG references to directmap

fix typo

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

fix build failure without options MULTIPROCESSOR

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

set kernel text/rodata readonly when not defined DDB.
set readonly segment on 2Mbytes aligned. (kernel image is mapped with 2Mbytes L2 block)

fix build with aarch64 gcc/gas

allow to read CNTVCT_EL0 and CNTFRQ_EL0 from EL0

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

AA64 is not MIPS.

Change all KSEG references to directmap

fix typo

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

fix build failure without options MULTIPROCESSOR

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

set kernel text/rodata readonly when not defined DDB.
set readonly segment on 2Mbytes aligned. (kernel image is mapped with 2Mbytes L2 block)

fix build with aarch64 gcc/gas

allow to read CNTVCT_EL0 and CNTFRQ_EL0 from EL0

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

fix typo

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

fix build failure without options MULTIPROCESSOR

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

set kernel text/rodata readonly when not defined DDB.
set readonly segment on 2Mbytes aligned. (kernel image is mapped with 2Mbytes L2 block)

fix build with aarch64 gcc/gas

allow to read CNTVCT_EL0 and CNTFRQ_EL0 from EL0

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Part II of ad's aarch64 performance improvements (cpu_switch.S bugs are
all mine)

- Use tpidr_el1 to hold curlwp and not curcpu, because curlwp is accessed
much more often by MI code. It also makes curlwp preemption safe and
allows aarch64_curlwp() to be a const function (curcpu must be volatile).

- Make ASTs operate per-LWP rather than per-CPU, otherwise sometimes LWPs
can see spurious ASTs (which doesn't cause a problem, it just means some
time may be wasted).

- Use plain stores to set/clear ASTs. Make sure ASTs are always set on the
same CPU as the target LWP, and delivered via IPI if posted from a remote
CPU so that they are resolved quickly.

- Add some cache line padding to struct cpu_info, to match x86.

- Add a memory barrier in a couple of places where ci_curlwp is set. This
is needed whenever an LWP that is resuming on the CPU could hold an
adaptive mutex. The barrier needs to drain the CPU's store buffer, so
that the update to ci_curlwp becomes globally visible before the LWP can
resume and call mutex_exit(). By my reading of the ARM docs it looks like
the instruction I used will do the right thing, but I'm not 100% sure.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

fix build failure without options MULTIPROCESSOR

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

set kernel text/rodata readonly when not defined DDB.
set readonly segment on 2Mbytes aligned. (kernel image is mapped with 2Mbytes L2 block)

fix build with aarch64 gcc/gas

allow to read CNTVCT_EL0 and CNTFRQ_EL0 from EL0

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

revert the changes of http://mail-index.netbsd.org/source-changes/2020/08/03/msg120183.html

This change is overengineered.
bus_space_{peek,poke}_N does not have to be reentrant nor available for interrupt context.

requested by skrll@

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

fix build failure without options MULTIPROCESSOR

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

set kernel text/rodata readonly when not defined DDB.
set readonly segment on 2Mbytes aligned. (kernel image is mapped with 2Mbytes L2 block)

fix build with aarch64 gcc/gas

allow to read CNTVCT_EL0 and CNTFRQ_EL0 from EL0

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

Implement MD ucas(9) (__HAVE_UCAS_FULL)

Fix a problem in which a fault occured in an interrupt handler during copyin/copyout was erroneously detected as being occured by copyin.

- keep idepth in faultbuf and compare it to avoid unnecessary fault recovery
- make cpu_set_onfault() nestable to use bus_space_{peek,poke}()
in hardware interrupt handlers during copyin & copyout.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C

make savecore for arm64 basically work.

- move MD lwp "md_ktf" member into struct pcb. the pcb is used by
the gdb "bsd-kvm" target code to find the stack of each thread
and needs to be available in a well known location.
- implement aarch64_nbsd_supply_pcb() in GDB. makes basic gdb work
on a crash dump.
- remove '#if L_MD_KTF + 8 == L_MD_CPACR' conditional code, as there
is no more L_MD_KTF.

with this gdb has minimal working functionality with "target kvm",
and crash can at least "ps" on a crash dump.

ok skrll.

add support PT_STEP

fix build failure without options MULTIPROCESSOR

rewrite the CPU identification on arm64:

- publish per-cpu data
- publish a whole bunch of info in struct aarch64_sysctl_cpu_id
instead of various individual nodes (there are 16 total.)
- add MIDR extractor bits
- define ARMv8.2-A id_aa64mmfr2_el1 and id_aa64zfr0_el1 regs,
but avoid using them until we make sure they exist. (these
members are added to aarch64_sysctl_cpu_id to avoid future
compat issues.)

the arm32 and aarch32 version of these need to be adjusted as
well (and aarch32 data published at all.) still trying to
work out how to make the same userland binary running on a
real arm32 or an aarch32 system can work sanely here.

ok ryo@.

* define LX_BLKPAG_{OS,ATTR}_* for OS dependent PTE attributes in pmap.h
* cleanup macros

add support multiple cpu clusters.
* pass cpu index as an argument to secondary processors when hatching.
* keep cpu cache confituration per cpu clusters.

Hello big.LITTLE!

set kernel text/rodata readonly when not defined DDB.
set readonly segment on 2Mbytes aligned. (kernel image is mapped with 2Mbytes L2 block)

fix build with aarch64 gcc/gas

allow to read CNTVCT_EL0 and CNTFRQ_EL0 from EL0

add MULTIPROCESSOR support

branches: 1.2.2;
Add initial support for ARMv8 (AARCH64) (by nisimura@ and ryo@)

- sys/arch/evbarm64 is gone and integrated into sys/arch/evbarm. (by skrll@)
- add support fdt. evbarm/conf/GENERIC64 fdt (bcm2837,sunxi,tegra) based generic 64bit kernel config. (by skrll@, jmcneill@)

branches: 1.1.4; 1.1.28;
Preliminary files for AARCH64 (64-bit ARM) support.
Enough for a distribution build.

- make AP{IB,DA,DB}Key are also enabled when ARMV83_PAC.
- If no ARMV83_PAC, clearly disable SCTLR_En{IA,IB,DA,DB}

Not only the kernel thread, but also the userland PAC keys
(APIA,APIB,APDA,APDB,APGA) are now randomly initialized at exec, and switched
when context switch.
userland programs are able to perform pointer authentication on ARMv8.3+PAC cpu.

reviewd by maxv@, thanks.

SCTLR_EnIA should be enabled in the caller(locore).

For some reason, gcc make aarch64_pac_init() function non-leaf, and it uses paciasp/autiasp.

Add support for Pointer Authentication (PAC).

We use the "pac-ret" option, to sign the return instruction pointer on
function entry, and authenticate it on function exit. This acts as a
mitigation against ROP.

The authentication uses a per-lwp (secret) I-A key stored in the 128bit
APIAKey register and part of the lwp context. During lwp creation, the
kernel generates a random key, and during context switches, it installs
the key of the target lwp on the CPU.

Userland cannot read the APIAKey register directly. However, it can sign
its pointers with it, because the register is architecturally shared
between userland and the kernel. Although part of the CPU design, it is
a bit of an undesired behavior, because it allows to forge valid kernel
pointers from userland. To avoid that, we don't share the key with
userland, and rather switch it in EL0<->EL1 transitions. This means that
when userland executes, a different key is loaded in APIAKey than the one
the kernel uses. For now the userland key is a fixed 128bit zero value.

The DDB stack unwinder is changed to strip the authentication code from
the pointers in lr.

Two problems are known:

* Currently the idlelwps' keys are not really secret. This is because
the RNG is not yet available when we spawn these lwps. Not overly
important, but would be nice to fix with UEFI RNG.
* The key switching in EL0<->EL1 transitions is not the most optimized
code on the planet. Instead of checking aarch64_pac_enabled, it would
be better to hot-patch the code at boot time, but there currently is
no hot-patch support on aarch64.

Tested on Qemu.

branches: 1.22.4;
G/C

G/C some more

G/C

Jazelle and T32EE are not part of ARMv8, fix the bits to their real
meanings. No functional change.

Hopefully fix some problems seen with MP support on non-x86, in particular
where curcpu() is defined as curlwp->l_cpu:

- mi_switch(): undo the ~2007ish optimisation to unlock curlwp before
calling cpu_switchto(). It's not safe to let other actors mess with the
LWP (in particular l->l_cpu) while it's still context switching. This
removes l->l_ctxswtch.

- Move the LP_RUNNING flag into l->l_flag and rename to LW_RUNNING since
it's now covered by the LWP's lock.

- Ditch lwp_exit_switchaway() and just call mi_switch() instead. Everything
is in cache anyway so it wasn't buying much by trying to avoid saving old
state. This means cpu_switchto() will never be called with prevlwp ==
NULL.

- Remove some KERNEL_LOCK handling which hasn't been needed for years.

branches: 1.17.2;
Do not use Early Write Acknowledge for PCIe I/O and config space.

Enable early write acknowledge for device memory mappings.

corect #include order

add AARCH64_KSEG_MASK. pmap_page.S refer it. (but no functional changed)

branches: 1.13.2;
G/C