MFC 340164,340168,340170: Add custom cpu_lock_delay() for x86.

340164:
Add a KPI for the delay while spinning on a spin lock.

Replace a call to DELAY(1) with a new cpu_lock_delay() KPI. Currently
cpu_lock_delay() is defined to DELAY(1) on all platforms. However,
platforms with a DELAY() implementation that uses spin locks should
implement a custom cpu_lock_delay() doesn't use locks.

340168:
Add a delay_tsc() static function for when DELAY() uses the TSC.

This uses slightly simpler logic than the existing code by using the
full 64-bit counter and thus not having to worry about counter
overflow.

340170:
Add a custom implementation of cpu_lock_delay() for x86.

Avoid using DELAY() since it can try to use spin locks on CPUs without
a P-state invariant TSC. For cpu_lock_delay(), always use the TSC if
it exists (even if it is not P-state invariant) to delay for a
microsecond. If the TSC does not exist, read from I/O port 0x84 to
delay instead.

PR: 228768

MFC 332906,332907,332976,333679,336053: Expand testing of breakpoints.

332906:
Extend support for ptrace() tests using breakpoints.

- Use a single list of platforms to define HAVE_BREAKPOINT for platforms
that expose a functional breakpoint() inline to userland. Replace
existing lists of platform tests with HAVE_BREAKPOINT instead.
- Add support for advancing PC past a breakpoint inserted via breakpoint()
to support the existing ptrace__PT_CONTINUE_different_thread test on
non-x86 platforms (x86 advances the PC past the breakpoint instruction,
but other platforms do not). This is implemented by defining a new
SKIP_BREAK macro which accepts a pointer to a 'struct reg' as its sole
argument and modifies the contents to advance the PC. The intention is
to use it in between PT_GETREGS and PT_SETREGS.

332907:
Expose breakpoint() to userland from <machine/cpufunc.h> on MIPS.

Enable ptrace() tests using breakpoint on MIPS as well.

332976:
Shorten some recently-added lines that are an extra indent over 80 columns.

333679:
Export a breakpoint() function to userland for riscv.

As a result, enable tests using breakpoint() on riscv.

336053:
Export a breakpoint() function to userland for arm and arm64.

Enable ptrace() tests using breakpoint() on these architectures.

MFC r336460:
Port r324665 and r325285 to arm64.

MFC r334101:
Add GET_STACK_USAGE() for arm64.

MFC r329859,r329860: Float protection in stand

r329859: Do not include float interfaces when using libsa.

We don't support float in the boot loaders, so don't include
interfaces for float or double in systems headers. In addition, take
the unusual step of spiking double and float to prevent any more
accidental seepage.

r329860: Floaty McFloatface is funnier...

Revert r331022: MFC of EFI Runtime Service support on aarch64

Apologies; this is NOT an MFC'able change. It requires a good number of pcb
changes that would break KBI.

Pointy hat to: me

MFC r324495: Support the EFI Runtime Services on arm64. As with amd64 we use
the 1:1 mapping. This uses the new common code shared with amd64.

The RTC should only be accessed via EFI. There is no locking around it as
the spec only has this as a requirement for the PC-AT CMOS device.

NOTE: r326311's changes to arm64/efirt_machdep.c have been hand-applied to
this import of efirt_machdep.c due to r326311 having already been merged.

MFC Loader Fixes 2017q4p6: r324649,r324650,r324651,r324652,r324653,r324654,
r324700,r324702,r324709,r324717,r324719,r324841,r324842,r324843,r324845,
r324850,r324876,r324877,r324878,r324879,r324880,r324881,r324883,r324980,
r324981,r324982,r324995,r325014,r325093,r325094,r325114,r325170,r325171,
r325172,r325173,r325174,r325175,r325176,r325248,r325286,r325310,r325332,
r325338,r325339,r325376,r325377,r325379,r325380,r325382,r325478,r325479,
r325480,r325482,r325483,r325484,r325485,r325556,r325641,r325681,r325685,
r325686,r325687,r325688,r325689,r325690,r325691,r325692,r325693,r325694,
r325743,r325744,r325748,r325775,r325779,r325780

r324649: Move common/Makefile.inc to sys/boot/loader.mk.

r324650: tweak style

r324651: create defs.mk for common definitions

r324652: Move all the ficl common code into ficl.mk

r324653: LOADER_foo_SUPPORTED

r324654: Explicitly inlcude SYSDIR in the include path

r324700: loader: initialize dv_cleanup in md.c to eliminate clang warning

r324702: We need to include disk.o in libuboot.a when we're building with
support for disk access.

r324709: Revert "Unify boot1 with loader" change r324646

r324717: libsa/ip.c: misplaced comment, ip_v is half char, not ip_p

r324719: libsa/ip: stop read loop on bad fragments

r324841: Use BOOTDIR more consistently in defs.mk rather than repeat
sys/boot.

r324842: Introduce BOOTOBJ: The top level object directory for the boot tree

r324843: Stopgap fix to the mistmatch between LOADER_GELI_SUPPORT and
LOADER_NO_GELI_SUPPORT.

r324845: Use BOOTOBJ and BOOTDIR to find geli includes and libraries.

r324850: Define LIBSA32 to LIBSA on i386 to fix build.

r324876: Move fdt and uboot defines into common uboot.mk.

r324877: End source directories with SRC rather than a hodgepodge of names

r324878: Make at91 boot loader compile again.

r324879: Prefer SRCTOP paths for bits we're grabbing from libc.

r324880: Use BOOTSRC here.

r324881: Use SYSDIR instead of ${.CURDIR}/../..<etc>/sys.

r324883: Use preferred defined paths, rather than relative paths in fdt.

r324980: Use BOOTDIR consistently.

r324981: Move BINDIR definition to defs.mk, and override where it isn't
/boot

r324982: Remove sys/boot/arm/at91 and ixp425

r324995: loader.efi: Make framebuffer commands available for arm64

r325014: Add a 'place holder' arm struct efi_fb until a real one comes

r325093: Define new EFI variables

r325094: Cleanup non-arch Makefiles

r325114: Use defs.mk defins in most MD code

r325170: Use defs.mk values for userboot

r325171: Use defs.mk name and prefer bsd.init.mk

r325172: Remove the -nostdlib stuff I added. Instead, fix LDFLAGS to be
honored correctly with the new Makefile.inc include order.

r325173: We don't need to build a special ficl for userboot.

r325174: Minor cleanup

r325175: For amd64, compile both zfs and zfs32 libraries.

r325176: Actually add zfs32/Makefile

r325248: loader ptblread() is broken with >512B sectors

r325286: efipart_strategy is using wrong offset with >512B sectors

r325310: zfs.c:vdev_read() needs to be careful about large sectors

r325332: loader: re-enable gzip support for x86

r325338: loader: fix BOOTSRC -> BOOTOBJ in a library path

r325339: This used to have bzip2 support too.

r325376: WIP: centralize machine links

r325377: mostly libsa

r325379: Revert "mostly libsa"

r325380: Revert "WIP: centralize machine links"

r325382: Cleanup stray libstand names to be libsa names.

r325478: Powerpc is a 32-bit boot loader.

r325479: Define LIBFICL32 to be libficl.a on i386 and libficl32.a on amd64.

r325480: Use DO32 for all the places that we need to flag we're building a
32-bit version of a library.

r325482: Move machine and other link creation to defs.mk

r325483: MACHINE can never be powerpc64, so cleanup code that thinks it can.

r325484: Prefer bsd.init.mk to src.opts.mk

r325485: Centralize all 32-bit builds on 64-bit platform stuff.

r325556: loader: set options before including bsd.init.mk

r325641: loader.efi: efi_devpath_is_prefix should return bool

r325681: boot1: avoid using NULL device path

r325685: libsa32 isn't needed for i386. It's already a 32-bit platform.

r325686: Simplify this if to a direct assignment.

r325687: Remove all the empty help files from the powerpc build.

r325688: FDT support doesn't make sense for ps3, remove it.

r325689: Remove LOADER_FDT_SUPPORT as a Makefile variable.

r325690: Remove LOADER_ZFS_SUPPORT as a Makefile variable

r325691: Remove useless PNP define here.

r325692: Replace LOADER_FIREWIRE_SUPPORT variable

r325693: Move LOADER_{NO,}_GELI_SUPPORT to MK_LOADER_GELI

r325694: Install the 4th files in sys/boot/forth instead of each loader

r325743: Make sure the proper loader.rc gets installed.

r325744: boot1: also check for NULL device

r325748: Use proper include file.

r325775: Add loader.conf to the list of files that are MD.

r325779: Add /boot/dts to the list of default modules.

r325780: Don't add /boot/dt*s* but /boot/dt*b*. Stupid think-o.

MFC r316648:

Corrected misspelled versions of rendezvous.

The MFC maintains smp_no_rendevous_barrier() as a symbol alias of
smp_no_rendezvous_barrier().

__FreeBSD_version bumped to indicate presence of the new name
smp_no_rendezvous_barrier().

Reviewed by: gnn, jhb (email), kib
Differential Revision: https://reviews.freebsd.org/D10313

MFC r326971, r327047 (by ian), r327053 (by marius), r327074, r327097:
Add atomic_load(9) and atomic_store(9) operations.

MFC r326137:

Ensure we check the program state set in the trap frame on arm and arm64.
This value may be set by userspace so we need to check it before using it.
If this is not done correctly on exception return the kernel may continue
in kernel mode with all registers set to a userspace controlled value. Fix
this by moving the check into set_mcontext, and also add the missing
sanitisation from the arm64 set_regs.

Discussed with: security-officer@
Sponsored by: DARPA, AFRL

MFC 323580,323933,323934,324814,324817: Enable AT_HWCAP on arm.

I reused the SV_HWCAP stub to cover the sv_hwcap2 field as well.

323580:
Add AT_HWCAP flags for VFP settings for FreeBSD/arm.

These flags match the meaning and value of flags in Linux, though
Linux has many more flags.

323933:
Correct HWCAP_VFP3* values to match Linux.

323934:
Detect NEON and set HWCAP_NEON if present.

324814:
Add AT_HWCAP2 ELF auxiliary vector.
- allocate value for new AT_HWCAP2 auxiliary vector on all platforms.
- expand 'struct sysentvec' by new 'u_long *sv_hwcap2', in exactly
same way as for AT_HWCAP.

324817:
Fullify implementation of AT_HWCAP and AT_HWCAP2 for ARMv6,7.
This makes elf_aux_info(3) useable for ARM ports.

Tested by: mmel

MFC 323579,323585: Add AT_HWCAP and AT_EHDRFLAGS on all platforms.

To preserve KBI on stable/11, a new SV_HWCAP flag is added which
indicates if the sv_hwcap field is present and valid to avoid examining
the field in old modules. Only sysentvec's which wish to use sv_hwcap
need to set the flag in stable/11.

323579:
Add AT_HWCAP and AT_EHDRFLAGS on all platforms.

A new 'u_long *sv_hwcap' field is added to 'struct sysentvec'. A
process ABI can set this field to point to a value holding a mask of
architecture-specific CPU feature flags. If an ABI does not wish to
supply AT_HWCAP to processes the field can be left as NULL.

The support code for AT_EHDRFLAGS was already present on all systems,
just the #define was not present. This is a step towards unifying the
AT_* constants across platforms.

323585:
Add AT_EHDRFLAGS and AT_HWCAP on amd64.

x86 has two separate (but identical) list of AT_* constants and the
earlier commit to add AT_HWCAP only updated the i386 list.

MFC 322437: Reliably enable debug exceptions on all CPUs.

Previously, debug exceptions were only enabled on the boot CPU if
DDB was enabled in the dbg_monitor_init() function. APs also called
this function, but since mp_machdep.c doesn't include opt_ddb.h, the
APs ended up calling an empty stub defined in <machine/debug_monitor.h>
instead of the real function. Also, if DDB was not enabled in the kernel,
the boot CPU would not enable debug exceptions.

Fix this by adding a new dbg_init() function that always clears the OS
lock to enable debug exceptions which the boot CPU and the APs call.
This function also calls dbg_monitor_init() to enable hardware breakpoints
from DDB on all CPUs if DDB is enabled. Eventually base support for
hardware breakpoints/watchpoints will need to move out of the DDB-only
debug_monitor.c for use by userland debuggers.

MFC r319871:
Make struct syscall_args visible to userspace compilation environment
from machine/proc.h, consistently on all architectures.

MFC r320403, r320497

r320403:
Some of the atomic_clear_* functions were incorrectly defined to be an
atomic add. Correct these, fixing a NULL-pointer dereference in netgraph.

PR: 220273
Sponsored by: DARPA, AFRL

r320497:
Remove a blank line accidentally added in r320403.

MFC r307334, r318252: Support write-through caches on arm64

r307334:
Create macros for the MAIR memory attributes. While here add an uncached
memory type, however the VM code still needs to be taught about this.

r318252:
Add the VM_MEMATTR_WRITE_THROUGH memory type to arm64 and use it to support
VM_MEMATTR_WRITE_COMBINING in the kernel. This fixes a bug where Xorg would
use write back cached memory for its graphics buffers. This would produce
artifacts on the screen as cachelines were written to memory.

MFC r313396:

Push reading of ESR_EL1 to assembly. Among other uses this will allow us
to expose this to signal handlers, e.g. for the clang sanitizers.

MFC r316734, r316761: Support Execute-Never bit in the arm64 pmap.

r316734:
Start to use the User and Privileged execute-never bits in the arm64
pagetables. This sets both bits when entering an address we know shouldn't
be executed.

I expect we could mark all userspace pages as Privileged execute-never to
ensure the kernel doesn't branch to one of these addresses.

While here add the ARMv8.1 upper attributes.

r316761:
Set the arm64 Execute-never bits in more places.

We need to set the Execute-never bits when mapping device memory as the
hardware may perform speculative instruction fetches.

Set the Privileged Execute-ever bit on userspace memory to stop the kernel
if it is tricked into executing it.

MFC r316732, r316756: Enable Privileged Access Never on arm64.

r316732:
Use the unprivileged variant of the load and store instructions most
places possible in the kernel. This forces these functions to fail if
userspace is unable to access a given memory location, even if it is in
the user memory range.

This will simplify adding Privileged Access Never support later.

r316756:
In ARMv8.1 ARM has added a process state bit to disable access to userspace
from the kernel. Make use of this to restrict accessing userspace to just
the functions that explicitly handle crossing the user kernel boundary.

MFC r316755:

Add SCTLR bits added in ARMv8.1 and ARMv8.2 and start to use them in the
early boot code.

MFC efivar(8) (by imp):

List of revisions merged:
r307070
r307071
r307072
r307074
r307189
r307224
r307339
r307390
r307391
r309776
r314231
r314232
r314615
r314616
r314617
r314618
r314619
r314620
r314621
r314623
r314890
r314925
r314926
r314927
r314928
r315770
r315771

Discussed with: gjb (re), imp
Sponsored by: The FreeBSD Foundation

MFC r311169,r311898,r312925,r312973,r312975,r313007,r313040,r313080,
r313254,r313341

amd64: add atomic_fcmpset

==

sparc64: add atomic_fcmpset

==

Implement atomic_fcmpset_* for arm and arm64.

==

Add atomic_fcmpset_*() inlines for powerpc

Summary:
atomic_fcmpset_*() is analogous to atomic_cmpset(), but saves off the read value
from the target memory location into the 'old' pointer in the case of failure.

==

i386: add atomic_fcmpset

==

Don't retry a lost reservation in atomic_fcmpset()

The desired behavior of atomic_fcmpset_() is to always exit on error. Instead
of retrying on lost reservation, leave the retry to the caller, and return

==

Add atomic_fcmpset_*() inlines for MIPS

atomic_fcmpset_*() is analogous to atomic_cmpset(), but saves off the
read value from the target memory location into the 'old' pointer.

==

i386: fixup fcmpset

An incorrect output specifier was used which worked with clang by accident,
but breaks with the in-tree gcc version.

While here plug a whitespace nit.

==

Implement atomic_fcmpset_*() for RISC-V.

==

Use 64bit store instruction in atomic_fcmpset_64.

MFC r313345:
Update arm and arm64 counters MD bits.

MFC r313394 (by manu):
subr_sfbus.c need sys/proc.h for struct thread definition.

MFC r313194:
Define the vm_ooffset_t and vm_pindex_t types as machine-independend.

MFC r306091:
Add a way for the architecture to specify the calling ABI for methods
in the EFI Runtime Services Table. On amd64, the calling conventions
are MS.

MFC 305285:
Add a pc_clock pcpu field and use it to implement cpu_est_clockrate. This
will allow drivers that manage the clock frequency to communicate this with
the reset of the kernel.

Sponsored by: ABT Systems Ltd

MFC 304004, 304596, 304598, 304599, 304600, 304604, 304620, 304685, 304687,
304688, 304689, 304746, 304749, 304750, 304806, 305071, 305191:
Merge arm64 superpage support, however leave it disabled by default.

MFC after: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

MFC r304285:
Implement userspace gettimeofday(2) with HPET timecounter.

MFC 305771, 305772:
Fix the arm64 kernel build when DDB is disabled, debug_monitor.c depends on
DDB, and is unused when it's disabled.

MFC 303594, 303610:
Add more AArch64 special register macros

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

MFC 304140:
Add the ARMv8.1 identification registers to the list we print when booting.

Sponsored by: ABT Systems Ltd

MFC 302847, 302848, 302852, 302853:
Remove the remaining non-INTRNG support from the arm64 code.

MFC 302789:
Add memmmap on arm64 so we can mmap /dev/mem and /dev/kmem.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

ARM64: fix DMAP calculation

Use arithmetic operators instead of logical. This fixes
DMAP ranges calculation for ThunderX Dual Socket.

Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: zbb
Differential Revision: https://reviews.freebsd.org/D7023
Approved by: re (gjb)

Add the GICv3 ITS intrng driver. As the interface to the interrupt
framework has significantly changed the driver has moved to a new file.
While it shares some code with the existing driver this has been modified
to work better with the intrng framework.

This has been tested on the ThunderX servers in the netperf cluster and has
been used to boot them for other testing, including DTrace and hwpmc.

With this we can use intrng on all supported arm64 platforms I was able to
test on. It is expected we will move to intrng soon, and disable the old
arm64 interrupt framework.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6437

Define PCI_RES_BUS for NEW_PCIB

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add support for intrng to arm64. As the GICv3 drivers will need to be
updated, and until further testing can be done, this is disabled for now.

It is expected arm64 will switch to this interface, and the old interface
will be removed before 11.0 is released.

Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation

Add support to the arm64 busdma to handle the cache. For now this is
disabled, however when we enable it it will default to assume memory is
not cache-coherent, unless either the tag was created or the parent was
marked as cache-coherent.

Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation

Call busdma_swi from swi_vm as is done from other architectures.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Fix I/O coherence issues on ThunderX when SMP is disabled

To maintain coherence between cache and DMA memory appropriate
shareability flags need to be set in the PTE regardless of SMP
option.

Reviewed by: wma
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D6231

Move arm's devmap to some generic place, so it can be used
by other architectures.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D6091
Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

Use the yield instruction in the arm64 cpu_spinwait. This instruction is
a hint to the hardware the software is not performing a task.

Sponsored by: ABT Systems Ltd

Set the upper limit of the DMAP region to the limit of RAM as was found in
the physmap. This will reduce the likelihood of an issue where we have
device memory mapped in the DMAP. This can only happen if it is within the
same 1G block of normal memory.

Reviewed by: kib
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5938

Document the memory ranges within the kernel region to help with debugging
to track down which region an address is from.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Increase the arm64 kernel address space to 512GB, and the DMAP region to
2TB. The latter can be increased in 512GB chunks by adjusting the lower
address, however more work will be needed to increase the former.

There is still some work needed to only create a DMAP region for the RAM
address space as on ARM architectures all mappings should have the same
memory attributes, and these will be different for device and normal memory.

Reviewed by: kib
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5859

Allow vmparam.h to be included from assembly files on arm64.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Implement dtrace_getupcstack in ARM64

Allow using DTRACE for performance analysis of userspace
applications - the function call stack can be captured.
This is almost an exact copy of AMD64 solution.

Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: emaste, gnn, jhibbits
Differential Revision: https://reviews.freebsd.org/D5779

arm64: pagezero improvement

This change has been provided to improve pagezero call performance.

Submitted by: Dominik Ermel <der@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D5741

arm64: bzero optimization

This optimization attempts to utylize as wide as possible register store instructions to zero large buffers.
The implementation, if possible, will use 'dc zva' to zero buffer by cache lines.

Speedup: 60x faster memory zeroing

Submitted by: Dominik Ermel <der@semihalf.com>
Obtained from: Semihalf
Sponsored by: Cavium
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D5726

Add support for 4 level pagetables. The userland address space has been
increased to 256TiB. The kernel address space can also be increased to be
the same size, but this will be performed in a later change.

To help work with an extra level of page tables two new functions have
been added, one to file the lowest level table entry, and one to find the
block/page level. Both of these find the entry for a given pmap and virtual
address.

This has been tested with a combination of buildworld, stress2 tests, and
by using sort to consume a large amount of memory by sorting /dev/zero. No
new issues are known to be present from this change.

Reviewed by: kib
Obtained from: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5720

Use the saved program state register to detect when an exception frame is
from userpsace. Previously we could have triggered a panic by trying to
jump to a kernel address from userland as the trap handling code thought we
received an ast in kernel mode.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Rename COUNT_IPI to INTR_IPI_COUNT to reduce the diff with intrng.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Reduce the diff with intrng by renaming similar functions. This is a noop,
but will help move to use the common interrupt handling code later.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Remove the invalid L0_BLOCK definition. ARMv8 doesn't support block
translation in the level 0 descriptor.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Force re-routing PCI interrupts (this is for legacy INTx not MSI).
Need this for gem5, but was not needed on real hadrware (yet) as it
was always MSI.

Reviewed by: andrew, jhb
Discovered by: andrew
Sponsored by: DARPA/AFRL
Differential Revision: https://reviews.freebsd.org/D5494

Enable SRE_EL2 on ARM64

Enable system register access for EL2. Alpine-V2 is
the first device requiring this to be enabled.
It is also in-sync with Linux initialization code,
and compatible with Alpine-V2 uboot requirements.

Obtained from: Semihalf
Submitted by: Michal Stanek <mst@semihalf.com>
Sponsored by: Annapurna Labs
Approved by: cognet (mentor)
Reviewed by: wma
Differential revision: https://reviews.freebsd.org/D5394

Remove pd_prot and pd_cache members from struct arm_devmap_entry.
The struct is used for definition of static device mappings which
should always have same protection and attributes.

Introduce bus_bind_intr method for ARM64

It can be used to bind specific interrupt to a particular CPU.
Requires PIC support for interrupts binding.

Reviewed by: wma
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5122

Minor clean-ups for ARM64 GICv3 and GIC drivers

GICv3:
- move ICC_SGI1R_EL1 definitions to armreg.h and use proper system
register's names
GIC:
- remove unused functions

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5119

Implement kdb_cpu_sync_icache on arm64.

Sponsored by: ABT Systems Ltd

Increase the size of PHYS_AVAIL_SIZE to allow firmware to provide a large
number of physical memory locations we can access. This is the case on
some HiKey boards that may have UEFI reserved memory dispersed through the
physical space.

Sponsored by: ABT Systems Ltd

Implement single stepping on arm64. We need to set the single step bits in
the processor and debug state registers. A flag has been added to the pcb
to tell us when to enable single stepping for a given thread.

Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4730

Framework for ARM64 instruction disassembler

Provide an easy to use framework for ARM64 DDB disassembler.
This commit does not contain full list of instruction opcodes.

Obtained from: Semihalf
Sponsored by: Cavium
Approved by: cognet (mentor)
Reviewed by: zbb, andrew, cognet
Differential revision: https://reviews.freebsd.org/D5114

Make using the #address-cells property on the interrupt parent in device
tree parsing opt-out rather than opt-in. All FDT-based systems as well as
PowerPC systems with real Open Firmware use the CHRP-derived binding that
includes it, which makes SPARC the odd man out here. Making it opt-out
avoids astonishment on new platform bring up.

Decode and print the ID_AA64* registers on boot. These registers hold
information on what the core supports. In most cases these will be
identical across most CPUs in the SoC, however there may be the case where,
with a big.LITTLE setup they may differ. In this case we print the
decoded data on all CPUs.

Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4725

Increase the size and alignment of the setjmp buffer. This will allow for
possible future CPU extentions with larger registers.

jmp_buf's size and alignment are baked into the ABI of third party libraries
and thus are very hard to change later so it is best to waste a small amount
of space now.

Reviewed by: brooks
Differential Revision: https://reviews.freebsd.org/D3956

Support the variant of the interrupt-map property where the parent bus has
the #address-cells property set. For this we need to read more data before
the parent interrupt description.

this is only enabled on arm64 for now as it's not quite compliant with the
ePAPR spec. We should use a default of 2 where the #address-cells property
is missing, however this will need further testing across architectures.

Obtained from: ABT Systems Ltd
Sponsored by: SoftIron Inc
Differential Revision: https://reviews.freebsd.org/D4518

Reduce the numbers of levels of indirection in arm64 pcib drivers by making
the MSI & MSI-X handler functions be directly callable from the driver
methods.

Sponsored by: ABT Systems Ltd

Add support for usermode (vdso-like) gettimeofday(2) and
clock_gettime(2) on ARMv7 and ARMv8 systems which have architectural
generic timer hardware. It is similar how the RDTSC timer is used in
userspace on x86.

Fix a permission problem where generic timer access from EL0 (or
userspace on v7) was not properly initialized on APs.

For ARMv7, mark the stack non-executable. The shared page is added for
all arms (including ARMv8 64bit), and the signal trampoline code is
moved to the page.

Reviewed by: andrew
Discussed with: emaste, mmel
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D4209

Rework the atomic code to reduce the repetition. This merges some of the
atomic functions where they are almost identical, or have acquire/release
semantics.

While here clean these function up. The cbnz instruction doesn't change
the condition flags so drop cc, however they should have memory added to the
clobber list.

Reviewed by: kib
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D4318

Add support for moving the DMAP range. This is needed as some AMD SoCs
place physical memory at an address outside the old DMAP range. This is an
issue as we rely on being able to move from PA -> VA using this range.

Obtained from: Patrick Wildt <patrick@bitrig.org> (earlier version)
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D3885

Mark functions as such. This means we call them directly rather than have
the dynamic linker copy them, but not relocate them at the new location.
This allows us to run sqlite3 without it crashing.

Sponsored by: ABT Systems Ltd

Use 4 levels of page tables when enabling the MMU. This will allow us to
boot on an SoC that places physical memory at an address past where three
levels of page tables can access in an identity mapping.

Submitted by: Wojciech Macek <wma@semihalf.com>,
Patrick Wildt <patrick@bitrig.org>
Differential Revision: https://reviews.freebsd.org/D3885 (partial)
Differential Revision: https://reviews.freebsd.org/D3744

Clear exclusive monitors when handling data aborts, the monitors are
in unknown state per spec.

Reviewed by: andrew (previous version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3668

Block secondary ITS instances from attaching on ARM64

Currently FreeBSD supports only single PIC controller. Some systems
that have more than one (like ThunderX dual-socket) fails to boot.
Disable other PICes until proper handling is implemented in the
generic interrupt code.

Reviewed by: imp
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3682

Remove pmap_kenter from pmap.h, the function doesn't exist.

Add pmap_get_tables to get the page tables for a given virtual address. This
will be used for minidump support.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add the arm64 minidump header. This was missed from r286953.

Increase MAXCPU in ARM64

Increase MAXCPU number to the maximum known value the existing
hardware can support.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3405

Add SMP support to GICv3 and ITS drivers

Introduce supprot for SMP to GICv3 and ITS drivers.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3299

Add the CNTHCTL_EL2 register bits missed in r286674

Rationalize BSD license on sys/*/include/in_cksum.h

Remove the advertising clause from the Regents of the University of
California's license, per the letter dated July 22, 1999.

Update clause numbering.

Rationalize BSD license on sys/*/include/float.h

Remove the advertising clause from the Regents of the University of
California's license, per the letter dated July 22, 1999.

Update clause numbering.

Pass the pcb to store the vfp state in to vfp_save_state. This fixes a bug
in savectx where it will be used to store the current state however will
pass in a pcb when vfp_save_state expected a thread pointer.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add support for uma_small_alloc and uma_small_free, and make use of these.
This is copied from the amd64 version with minor changes. These should be
merged into a single file as from a quick look there are other copies of
the same file in other parts of the tree.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add more atomic_swap_* functions.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add VIRT_IN_DMAP to check if a virtual address is from the DMAP range.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Apply erratum for mrs ICC_IAR1_EL1 speculative execution on ThunderX

ERRATUM: 22978, 23154
PASS (rev.): 1.0/1.1

Reviewed by: imp
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3184

Implement get_cyclecount() on ARM64

Use Vritual Counter register associated with Generic Timer to
read the cyclecount.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3134

Improve ARM64 CPU_MATCH

Add a method to identify CPU based on RAW MIDR value.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3117

Increase DMAP (Direct Map) size on ARM64

Previous DMAP size was too small for systems with more than 64GB
of RAM. Increase it to 128GB to support ThunderX CRB.

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3113

Set-up proper TCR values for memory related to Translation Table Walking

This commit adds proper cache and shareability attributes to
the TCR register.
Set memory attributes to Normal, outer and inner cacheable WBWA.
Set shareability to inner and outer shareable when SMP is enabled.

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3093

Set memory to be inner-sharable. This isn't needed on device memory as the
MMU will ignore the attribute there, howeverit simplifies to code to alwas
set it.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Implement stubs for ACPI PCI routines

ACPI driver requires special functions to be provided by machdep code.
Add temporary stubs to satisfy the compiler when both "pci" and "acpi"
are enabled in the kernel configuration file.

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3028

Add support for SMP. This uses the FDT data to find the CPUs to start on,
and psci to start them. I expect ACPI support to be added later.

This has been tested on qemu with 2 cpus as that is the current value of
MAXCPUS. This is expected to be increased in the future as FreeBSD has
been tested on 48 cores on the Cavium ThunderX hardware.

Partially based on a patch from Robin Randhawa from ARM.

Approved by: ABT Systems Ltd
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3024

Add the definition of the shareable bits in the pagetables

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Clean up the types used in <machine/ucontext.h> on arm64. As some ports
include this file without first including the headers needed for uint32_t
and the like use the __foo type.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Rework CPU identification on ARM64

This commit reworks the code responsible for identification of
the CPUs during runtime.
It is necessary to provide a way for workarounds and erratums
to be applied only for certain HW versions.

The copy of MIDR is now stored in pcpu to provide a fast and
convenient way for assambly code to read it (pcpu is used quite often
so there is a chance it's inside the cache).
The MIDR is also better way of identification than using user-friendly
cpu_desc structure, because it can be compiled into comparision of
single u32 with only one access to the memory - this is crucial
for some erratums which are called from performance-critical
places.

Changes in cpu_identify makes this function safe to be called
on non-boot CPUs.

New function CPU_MATCH was implemented which returns boolean
value based on mathing masked MIDR with chip identification.
Example of usage:

printf("is thunder: %d\n", CPU_MATCH(CPU_IMPL_MASK | CPU_PART_MASK,
CPU_IMPL_CAVIUM, CPU_PART_THUNDER, 0, 0));
printf("is generic: %d\n", CPU_MATCH(CPU_IMPL_MASK | CPU_PART_MASK,
CPU_IMPL_ARM, CPU_PART_FOUNDATION, 0, 0));

Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3030

Add the atomic_thread_fence() family of functions with intent to
provide a semantic defined by the C11 fences with corresponding
memory_order.

atomic_thread_fence_acq() gives r | r, w, where r and w are read and
write accesses, and | denotes the fence itself.

atomic_thread_fence_rel() is r, w | w.

atomic_thread_fence_acq_rel() is the combination of the acquire and
release in single operation. Note that reads after the acq+rel fence
could be made visible before writes preceeding the fence.

atomic_thread_fence_seq_cst() orders all accesses before/after the
fence, and the fence itself is globally ordered against other
sequentially consistent atomic operations.

Reviewed by: alc
Discussed with: bde
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks

Correctly set __WCHAR_MIN, there is no __UINT_MIN, it's 0.

Sponsored by: ABT Systems Ltd

Introduce ITS support for ARM64

Add ARM ITS (Interrupt Translation Services) support required
to bring-up message signalled interrupts on some ARM64 platforms.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation

add floatingpoint.h for arm64

On other architectures floatingpoint.h is a symlink to
machine/floatingpoint.h which in turn includes machine/ieeefp.h.
Do this on arm64 as well for now.

Sponsored by: The FreeBSD Foundation

Fix the floating-point exception values to line up with the hardware
register bits. Nothing in base uses these as they are deprecated, however
third-party applications, such as perl, expect some of these functions to
exist.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add basic support for ACPI. It splits out the nexus driver to two new
drivers, one for fdt, one for acpi. It then uses this to decide if it will
use fdt or acpi.

The GICv2 (interrupt controller) and Generic Timer drivers have been
updated to handle both cases.

As this is early code we still need FDT to find the kernel console, and
some parts are still missing, including PCI support.

Differential Revision: https://reviews.freebsd.org/D2463
Reviewed by: jhb, jkim, emaste
Obtained from: ABT Systems Ltd
Relnotes: Yes
Sponsored by: The FreeBSD Foundation

Split out db_unwind_frame() so it can be used by DTrace.

Sponsored by: ARM Ltd.
Differential Revision: https://reviews.freebsd.org/D2741

Add options to dmb() and dsb() macros on ARM64

Using plain dsb()/dmb() as full system barriers is usually to much.
Adding proper options to those barriers (instead of full system - sy)
will most likely reduce the cost of the instructions and will benefit
in performance improvement.
This commit adds options to barrier macro definitions.

Obtained from: Semihalf
Reviewed by: andrew, ian
Sponsored by: The FreeBSD Foundation

Retire VM_FREEPOOL_CACHE as the next step in eliminating PG_CACHE pages.

Differential Revision: https://reviews.freebsd.org/D2712
Reviewed by: kib
Sponsored by: EMC / Isilon Storage Division

Add Performance Monitoring Counters support for AArch64.
Family-common and CPU-specific counters implemented.

Supported CPUs: ARM Cortex A53/57/72.

Reviewed by: andrew, bz, emaste, gnn, jhb
Sponsored by: ARM Limited
Differential Revision: https://reviews.freebsd.org/D2555

Add support for ARM GICv3 interrupt controller used in some ARM64 chips

GICv3 allows to distribute interrupts to more than 8 cores served by
the previous GIC revisions. GICv3 introduces additional logic in form
of Re-Distributors associated with particular CPUs to determine
the highest priority interrupts and manage PPIs and LPIs
(Locality-specific Peripheral Interrupts). Interrupts routing is
based on CPUs' affinity numbers. CPU interface was changed to be
accessible via CPU System Registers and this is the preferred
(and supported) method in this driver.

Obtained from: Semihalf
Reviewed by: andrew, emaste, ian, imp
Sponsored by: The FreeBSD Foundation

Port x86 busdma to ARM64

The x86 busdma subsystem allows using multiple implementations.
By default the classic bounce buffer approach is used, however
on systems with IOMMU it could be in runtime switched to more
efficient hardware accelerated implementation.

This commit adds ARM64 port of the x86 busdma framework and bounce
buffer backend. It is ready to use on IO coherent systems. If the
IO coherency cannot be guaranteed, the cache management operations have
to be added to this code in places marked by /* XXX ARM64TODO (...) */
comments. Also IOMMU support might be added by registering another
busdma implementation like it is already done on the x86.

Reviewed by: andrew, emaste
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation

Add pmap_mapbios and pmap_unmapbios. These will be needed to support acpi.

Sponsored by: The FreeBSD Foundation

Add support for arm64 to loader.efi and boot1.efi

Reviewed by: emaste
Sponsored by: The FreeBSD Foundation

Bring in the start of the arm64 kernel.

This is only the minimum set of files needed to boot in qemu. As such it is
missing a few things.

The bus_dma code is currently only stub functions with a full implementation
from the development tree to follow.

The gic driver has been copied as the interrupt framework is different. It
is expected the two drivers will be merged by the arm intrng project,
however this will need to be imported into the tree and support for arm64
would need to be added.

This includes code developed by myself, SemiHalf, Ed Maste, and Robin
Randhawa from ARM. This has been funded by the FreeBSD Foundation, with
early development by myself in my spare time with assistance from Robin.

Differential Revision: https://reviews.freebsd.org/D2199
Reviewed by: emaste, imp
Relnotes: yes
Sponsored by: The FreeBSD Foundation

We will have fueword on arm64, mark as such in machine/param.h.

Sponsored by: The FreeBSD Foundation

Split out the _acq and _rel functions. These were the same, but there is
no need for them to be this strong, we only need to provide one or the
other.

While here replace atomic_load_acq_* and atomic_store_rel_* with a single
instruction version, and fix the definition of atomic_clear_* to point to
the correct functions.

Sponsored by: The FreeBSD Foundation

Add space for future expansion [1]. While here clean up a little, spsr is
only 32-bits, and mark the holes in the struct with a pad member.

Suggested by: kib [1]
Sponsored by: The FreeBSD Foundation

Add more arm64 machine dependent headers. With this we now have the minimum
set of machine headers needed to build the userland toolchain.

Differential Revision: https://reviews.freebsd.org/D2148
Reviewed by: imp
Sponsored by: The FreeBSD Foundation

Renumber clauses to avoid missing 3

Add the start of the arm64 machine headers. This is the subset needed to
start getting userland libraries building.

Reviewed by: imp
Sponsored by: The FreeBSD Foundation