MFC r328467:

Implement mitigation for Spectre version 2 attacks on ARMv7.

MFC r319896,r320054:

r319896:
Implement tunable CPU quirks. These quirks are intended for optimizing CPU
performance, not for applying errata workarounds. Nobody can expect that
CPU with unfixed errata is stable enough to execute the kernel until quirks
are applied.
r320054:
Manually load tunable CPU quirks. These are needed too early, far before
SYSINIT is processed.

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

MFC r306756:

ARM: SEV/WFE instructions are implemented starting from ARMv6K, use it
directly.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Move the arm call to intr_pic_init_secondary earlier in the secondary CPU
initialisation. This ensures it will complete before signalling to the boot
CPU it has booted. This fixes a race with the GIC where the arm_gic_map may
not be populated before it is used to bind interrupts leading to some
interrupts becoming bound to no CPUs.

Approved by: re (kib)
Sponsored by: ABT Systems Ltd

Improve ARM debug_monitor for SMP machines

- Reset debug architecture and enable monitor for secondary
CPUs in init_secondary() rather than when configuring watchpoint, etc.
- Disable HW debugging capabilities when one of the CPU cores fails
to set up.
- Use dbg_capable() in a more atomic manner to avoid any mismatch
between CPUs.

Differential Revision: https://reviews.freebsd.org/D6009

Include machine/acle-compat.h in cdefs.h on arm if the compiler doesn't
have ACLE support built in. The ACLE (ARM C Language Extensions) defines
a set of standardized symbols which indicate the architecture version and
features available. ACLE support is built in to modern compilers (both
clang and gcc), but absent from gcc prior to 4.4.

ARM (the company) provides the acle-compat.h header file to define the
right symbols for older versions of gcc. Basically, acle-compat.h does
for arm about the same thing cdefs.h does for freebsd: defines
standardized macros that work no matter which compiler you use. If ARM
hadn't provided this file we would have ended up with a big #ifdef __arm__
section in cdefs.h with our own compatibility shims.

Remove #include <machine/acle-compat.h> from the zillion other places (an
ever-growing list) that it appears. Since style(9) requires sys/types.h
or sys/param.h early in the include list, and both of those lead to
including cdefs.h, only a couple special cases still need to include
acle-compat.h directly.

Loves it: imp

Rename ARM_INTRNG and MIPS_INTRNG to INTRNG. This will help with machine
independent code that needs to know about INTRNG such as PCI drivers.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Generalize IPI support for ARM intrng and use it for interrupt
controller IPI provider.

New struct intr_ipi is defined which keeps all info about an IPI:
its name, counter, send and dispatch methods. Generic intr_ipi_setup(),
intr_ipi_send() and intr_ipi_dispatch() functions are implemented.

An IPI provider must implement two functions:
(1) an intr_ipi_send_t function which is able to send an IPI,
(2) a setup function which initializes itself for an IPI and
calls intr_ipi_setup() with appropriate arguments.

Differential Revision: https://reviews.freebsd.org/D5700

Almost all copies of platform_mp_init_secondary just called
intr_pic_init_secondary. Replace them with a direct call. On BCM2836
and ARMADA XP we need to add this function, but it can be empty.

Reviewed by: ian, imp
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D5460

Remove platform_mp_probe as it's almost identical on most ARM SoCs, and
slightly wrong on the others. We should just check if mp_ncpus is set to
more than one CPU as we may wish to run on a single core even when SMP is
available.

Reviewed by: ian
Sponsored by: ABT Systems Ltd
Differential Revision: https://reviews.freebsd.org/D5458

Remove platform_ipi_send, it's an unneeded as all implementations are
identical.

Sponsored by: ABT Systems Ltd

As <machine/pmap.h> is included from <vm/pmap.h>, there is no need to
include it explicitly when <vm/pmap.h> is already included.

Reviewed by: alc, kib
Differential Revision: https://reviews.freebsd.org/D5373

ARM: Use new ARMv6 naming conventions for cache and TLB functions
in all but ARMv4 specific files.
Expand ARMv6 compatibility stubs in cpu-v4.h. Use physical address
in L2 cache functions if ARM_L2_PIPT is defined.

Remove not needed <machine/pte.h> includes.

ARM: Remove TLB IPI.
We don't support SMP on ARMv6. All ARMv7 multicore cpus already uses
hardware broadcast for TLB and cache operations.

ARM: Cleanup mp_machdep.c. SMP is supported only on ARMv6 and later.

ARM: remove old pmap-v6 code. The new pmap-v6 is mature enough, and
dual implementation is showstopper for major cleanup.

This patch only removes old code from tree. Cleanups will follow asap.

SMP support for ARMv6/v7 HW watchpoints

Use per-CPU structure to store HW watchpoints registers state
for each CPU present in the system. Those registers will be restored
upon wake up from the STOP state if requested by the debug_monitor
code. The method is similar to the one introduced to AMD64.

We store all possible 16 registers for HW watchpoints
(maximum allowed by the architecture).
HW breakpoints are not maintained since they are used for single
stepping only.

Pointed out by: kib
Reviewed by: wma
No strong objections from: kib
Submitted by: Zbigniew Bodek <zbb@semihalf.com>
Obtained from: Semihalf
Sponsored by: Juniper Networks Inc.
Differential Revision: https://reviews.freebsd.org/D4338

[intrng] Migrate the intrng code from sys/arm/arm to sys/kern/subr_intr.c.

The ci20 port (by kan@) is going to reuse almost all of the intrng code
since the SoC in question looks suspiciously like someone took an ARM
SoC design and replaced the ARM core with a MIPS core.

* migrate out the code;
* rename ARM_ -> INTR_;
* rename arm_ -> intr_;
* move the interrupt flush routine from intr.c / intrng.c into
arm/machdep_intr.c - removing the code duplication and removing
the ARM specific bits from here.

Thanks to the Star Wars: The Force Awakens premiere line for allowing
me a couple hours of quiet time to finish the universe builds.

Tested:

* make universe

TODO:

* The structure definitions in subr_intr.c still includes machine/intr.h
which requires one duplicates all of the intrng definitions in
the platform code (which kan has done, and I think we don't have to.)

Instead I should break out the generic things (function declarations,
common intr structures, etc) into a separate header.

* Kan has requested I make the PIC based IPI stuff optional.

Set the correct values in the arm aux control register, based on chip type.

The bits in the aux control register vary based on the processor type. In
the past we've always just set the 'smp' and "broadcast tlb/cache ops' bits,
which worked fine for the first few SoCs we supported. Now that we support
most of the cortex-a series processors, it's important to get the right bits
set based on the processor type.

Submitted by: Svatopluk Kraus <onwahe@gmail.com>

Import ARM_INTRNG, the "next generation" interrupt architecture for arm
and armv6 architecures. The primary enhancement over the old design is
support for hierarchical interrupt controllers (such as a gpio driver
which can receive interrupts from a root PIC and act as a PIC itself for
clients interested in handling a change of gpio pin state as an
interrupt). The new code also provides an infrastructure for mapping
interrupts described in metadata in the form of a "controller reference
plus interrupt number" tuple into the simple "0-n" flat numeric space
understood by rman and the bus resource mechanisms.

Use of the new code is enabled by setting the ARM_INTRNG option, and by
making a few simple changes to the platform's support code. In addition
each existing PIC driver needs changes to be ready for INTRNG; this commit
contains the changes for the arm/gic driver, which most armv6 SoCs use, but
it does not enable the new code yet on any platform.

This project has been many years in the making, starting as a GSoC project
by Jakub Klama (jceel@) in 2012. That didn't get committed right away and
the source base evolved out from under it to some degree. In 2014 I rebased
the diffs to then -current and did some enhancements in the area of mapping
interrupt numbers and storing associated fdt data, then the project went
cold again for a while. Eventually Svata Kraus took that work in progress
and did another big round of work on it, removing most of the remaining
rough edges. Finally I took that and made one more pass through it, mostly
disabling the "INTR_SOLO" feature for now, pending further design
discussions on how to most efficiently dispatch a pending interrupt through
more than one layer of PIC. The current code with the INTR_SOLO feature
disabled uses approximate 100 extra cpu cycles for each cascaded PIC the
interrupt has to be passed to, so what's left to do is about efficiency, not
correct operation.

Differential Revision: https://reviews.freebsd.org/D2047

Remove pc_cpu, it was duplicating pc_cpuid so was unneeded.

Remove trailing whitespace from sys/arm/arm

Use the wait-for-event instruction to put the core we have just enabled
to sleep while it waits to start scheduling. The boot core can then use
the send-event instruction to wake the cores when they should enter the
scheduler.

MFC after: 1 week

Retire pmap_lazyfix(). This function only existed in the new armv6 pmap
because the i386 pmap on which the new armv6 pmap is based had it, and in
r281707 pmap_lazyfix() was removed from the i386 pmap.

Discussed with: kib
Submitted by: Michal Meloun (via Svatopluk Kraus)

Remove the bootconfig parsing. We never used it and always passed either an
empty string or NULL to the setup functions that called into it.

New pmap code for armv6. Disabled by default, option ARM_NEW_PMAP enables it.

This is pretty much a complete rewrite based on the existing i386 code. The
patches have been circulating for a couple years and have been looked at by
plenty of people, but I'm not putting anybody on the hook as having reviewed
this in any formal sense except myself.

After this has gotten wider testing from the user community, ARM_NEW_PMAP
will become the default and various dregs of the old pmap code will be
removed.

Submitted by: Svatopluk Kraus <onwahe@gmail.com>,
Michal Meloun <meloun@miracle.cz>

Rename locore.S to locore-v4.S and add a new locore-v6.S for starting up
armv6/7 systems. We need to use some new armv6/7 features at startup
and splitting the implemenations to separate files will be more maintainable
than adding even more #ifdef sections to locore.S.

Because of the standardized interfaces to cache and MMU operations in armv7,
we can tolerate the kernel being entered with caches enabled. This allows
running u-boot and loader(8) with caches enabled, and the performance
improvement can be dramatic (boot times can be cut from over a minute
to under 30 seconds). The new implementation also has more robust cache
and mmu sequences for launching AP cores, and it paves the way for
upcoming changes to the pmap code which will use the TEX remap feature.

Changes in mp_machdep.c work with the new behavior in locore-v6 mp_entry,
and also reuse the original boot-time page tables to get transitioned
from physical to virtual addressing before installing the normal tables.

Submitted by Svatopluk Kraus and Michal Meloun with some changes by me.

Rename pic_ipi_get to pic_ipi_read for intrng.

Unify interrupts bit definition and usage. While here remove PSR_C_bit.

Submitted by: Svatopluk Kraus <onwahe at gmail.com>,
Michal Meloun <meloun at miracle.cz>
Differential Revision: https://reviews.freebsd.org/D754

Flush and invalidate caches on each CPU as part of handling IPI_STOP.

Flushing the caches is required before doing a panic dump, but ARM
doesn't provide a flavor of flush that gets broadcast to other cores.
However, all cores except one are stopped before doing a dump, so this
works around the lack of a global flush/invalidate by doing it locally
on each CPU as part of stopping.

Discussed with: cognet@

There is no difference between IPI_STOP and IPI_STOP_HARD on ARM, so
map them both to the same interrupt number like other arches do.

Retire smp_active. It was racey and caused demonstrated problems with
the cpufreq code. Replace its use with smp_started. There's at least
one userland tool that still looks at the kern.smp.active sysctl, so
preserve it but point it to smp_started as well.

Discussed with: peter, jhb
MFC after: 3 days
Obtained from: Netflix

All our current ARM multi-core systems have all cores in one package with
a shared L2 cache, reflect that in the common cpu_topo() routine.

Invalidate caches immediately upon entry to init_secondary(). Also set
the Bufferable bit in the PDE entries of the secondary processor startup
pagetables.

The caches really need to be invalidated even earlier than this, but this
is a big step in the right direction. The invalidate needs to happen
before the MMU is enabled, which means it has to be called from asm code
that's running with physical addressing. Fixing that will be handled in
a future change.

It turns out a global variable is the only straightforward way to
communicate the kernel's physical load address from where it's known in
initarm() into cpu_mp_start() which is called from non-arm code and
takes no parameters.

This adds the global variable and ensures that all the various copies
of initarm() set it. It uses the variable in cpu_mp_start(), eliminating
the last uses of KERNPHYSADDR outside of locore.S (where we can now
calculate it instead of relying on the constant).

Change the way pcpu and curthread are stored per-core:
the old way was to store pcpu in a register, and get curthread from pcpu,
which is not very atomic, and led to issues if the thread was migrated
to another core between the time we got the pcpu address and the time we
got curthread.
Instead, we now store curthread where pcpu used to be store, and we
calculate the pcpu address based on the cpu id.

Remove hard-coded mappings related to Armada XP support

Armada XP initialization flow requires SoC registers to be
mapped very early in order to configure Snoop Filter for SMP.
Additional mapping in locore.S is redundant as proper mapping is
made in pmap_devmap_bootstrap() prior to calling cpu_setup() which
configures the Snoop Filter.
For secondaru CPUs it is better to pass VA of the SoC
registers defined in MV_BASE and PA consistent with the value
in the Device Tree.

Tested by: kevlo

Rename device vfp to option VFP and retire the ARM_VFP_SUPPORT option. This
simplifies enabling as previously both options were required to be enabled,
now we only need a single option.

While here enable VFP on the PandaBoard.

Replace kernel virtual address space allocation with vmem. This provides
transparent layering and better fragmentation.

- Normalize functions that allocate memory to use kmem_*
- Those that allocate address space are named kva_*
- Those that operate on maps are named kmap_*
- Implement recursive allocation handling for kmem_arena in vmem.

Reviewed by: alc
Tested by: pho
Sponsored by: EMC / Isilon Storage Division

Avoid calling pcpu_init() simultaneously.

pcpu_init() updates queue, so cannot be called by multiple cores
at the same time

Obtained from: Semihalf

Properly initialize Armada XP MP subsystem.

- correct setting of Auxiliary Control Register for MP mode
- correct setting of Auxiliarty Debug registers
- cleanup management of memory contains bootup code
- early initialization of Coherency Fabric (MP and not-MP mode)
- enable Snoop Filtering

Obtained from: Semihalf

Don't forget to init the VFP stuff for all cores.

Merging projects/armv6, part 1

Cummulative patch of changes that are not vendor-specific:
- ARMv6 and ARMv7 architecture support
- ARM SMP support
- VFP/Neon support
- ARM Generic Interrupt Controller driver
- Simplification of startup code for all platforms