riscv: Remove the unused riscv64_cpu driver

This is a repeat of 63bf2d735ca3 ("Remove the unused arm64_cpu driver.")
for RISC-V, which copied the defunct code from arm64 with no changes
beyond substituting riscv64 for arm64, and made no use of it elsewhere.
It has thus always been entirely superfluous.

Reviewed by: mhorne
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43672

riscv: Convert local interrupt controller to a newbus PIC

Currently the local interrupt controller implementation is based on
pre-INTRNG arm/arm64 code, using hand-rolled event code rather than
INTRNG. This then interacts weirdly with the PLIC, and other future
interrupt controllers like the APLIC and IMSICs in the upcoming AIA
specification, since they become the root PIC despite not being the
logical root. Instead, use a real newbus device for it and register
it as the root PIC.

This also adapts the IPI code to make use of the newly-added INTRNG
generic IPI handling framework, adding a new sbi_ipi as the PIC. In
future there will be alternative devices for sending IPIs that will
register with higher priorities, such as the proposed AIA IMSIC and
ACLINT SSWI.

Reviewed by: mhorne
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D35901

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

smp: Dynamically allocate the stoppcbs array

This avoids bloating the kernel image when MAXCPU is large.

A follow-up patch for kgdb and other kernel debuggers is needed since
the stoppcbs symbol is now a pointer. Bump __FreeBSD_version so that
debuggers can use osreldate to figure out how to handle stoppcbs.

PR: 269572
MFC after: never
Reviewed by: mjg, emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39806

riscv: Rework CPU identification (second part)

Modify when and how we perform parsing and reporting. Most notably,
everything now executes on CPU 0.

The de-facto standard way to enumerate CPU features (ISA extensions) on
RISC-V is by parsing each CPU's ISA string. We currently obtain this
information from the device tree, and in the future will be able to pull
it from ACPI tables.

Eliminate the SYSINIT from identcpu.c. We still need to walk the /cpus
list in the device tree, but now do this one CPU at a time, as a step in
the identify_cpu() procedure. This is slightly less error prone, and
allows us to parse ISA features for CPU 0 much earlier.

Make use of the SMP hooks cpu_mp_start() and cpu_mp_announce() to
identify and print secondary CPU info, respectively. This causes
secondary processor identification to be printed much earlier in boot;
everything is done by SI_SUB_CPU, SI_ORDER_THIRD. Adjust some other
printf() calls so that we get enough useful info to debug under
bootverbose.

Reviewed by: markj (slightly earlier version)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39811

riscv: Call identify_cpu() earlier for CPU 0

It is advantageous to have knowledge of ISA features as early as
possible. For example, the presence of newer virtual memory extensions
may be useful to pmap_bootstrap().

To achieve this, split out the printf() parts of identify_cpu() into a
separate function, printcpuinfo(). This latter function will be called
later in boot after the console has been initialized.

Reviewed by: markj
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39810

ofw_cpu_early_foreach: Change callback to return bool instead of boolean_t.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D39926

riscv: reject CPUs with mmu-type "riscv,none"

According to riscv/cpus.yaml in the device-tree docs, this property may
exist but indicate that the CPU does not have an MMU. Detect this
possibility.

Reviewed by: jhb
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36980

riscv: drop a dead declaration

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

kmem_malloc/free: Use void * instead of vm_offset_t for kernel pointers.

Reviewed by: kib, markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D36549

arm64, riscv: size boot stacks appropriately

In 8db2e8fd16c4 ("Remove the secondary_stacks array in arm64 [...]"),
bootstacks was setup to be allocated dynamically. While this is
generally how x86 does it, it inadvertently shrunk each boot stack from
KSTACK_PAGES pages to a single page.

Resize these back up to the expected size using the kstack_pages
tunable, as we'll need larger stacks with upcoming sanitizer work.

Reviewed by: andrew, imp, markj
Fixes: 8db2e8fd16c4 ("Remove the secondary_stacks array [...]")
Sponsored by: Juniper Networks, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D36475

Fix the test used to wait for AP startup on x86, arm64, riscv

On arm64, testing pc_curpcb != NULL is not correct since pc_curpcb is
set in pmap_switch() while the bootstrap stack is still in use. As a
result, smp_after_idle_runnable() can free the boot stack prematurely.

Take a different approach: use smp_rendezvous() to wait for all APs to
acknowledge an interrupt. Since APs must not enable interrupts until
they've entered the scheduler, i.e., switched off the boot stack, this
provides the right guarantee without depending as much on the
implementation of cpu_throw(). And, this approach applies to all
platforms, so convert x86 and riscv as well.

Reported by: mmel
Tested by: mmel
Reviewed by: kib
Fixes: 8db2e8fd16c4 ("Remove the secondary_stacks array in arm64 and riscv kernels.")
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35435

riscv: Remove unused devclass arguments to DRIVER_MODULE.

sched: split sched_ap_entry() out of sched_throw()

sched_throw() can no longer take a NULL thread, APs enter through
sched_ap_entry() instead. This completely removes branching in the
common case and cleans up both paths. No functional change intended.

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D32829

sched: separate out schedinit_ap()

schedinit_ap() sets up an AP for a later call to sched_throw(NULL).

Currently, ULE sets up some pcpu bits and fixes the idlethread lock with
a call to sched_throw(NULL); this results in a window where curthread is
setup in platforms' init_secondary(), but it has the wrong td_lock.
Typical platform AP startup procedure looks something like:

- Setup curthread
- ... other stuff, including cpu_initclocks_ap()
- Signal smp_started
- sched_throw(NULL) to enter the scheduler

cpu_initclocks_ap() may have callouts to process (e.g., nvme) and
attempt to sched_add() for this AP, but this attempt fails because
of the noted violated assumption leading to locking heartburn in
sched_setpreempt().

Interrupts are still disabled until cpu_throw() so we're not really at
risk of being preempted -- just let the scheduler in on it a little
earlier as part of setting up curthread.

Reviewed by: alfredo, kib, markj
Triage help from: andrew, markj
Smoke-tested by: alfredo (ppc), kevans (arm64, x86), mhorne (arm)
Differential Revision: https://reviews.freebsd.org/D32797

riscv: Add an hw.ncpu tunable to limit the number of cores

Based on a similar change to arm64 in 01a8235ea61c.

Reviewed by: mhorne
MRC after: 1 week
Differential Revision: https://reviews.freebsd.org/D30655

riscv: set kernel_pmap hart mask more precisely

In pmap_bootstrap(), we fill kernel_pmap->pm_active since it is
invariably active on all harts. However, this marks it as active even
for harts that don't exist in the system, which can cause issue when the
mask is passed to the SBI firmware via sbi_remote_sfence_vma().
Specifically, the SBI spec allows SBI_ERR_INVALID_PARAM to be returned
when an invalid hart is set in the mask.

The latest version of OpenSBI does not have this issue, but v0.6 does,
and this is triggering a recently added KASSERT in CI. Switch to only
setting bits in pm_active for harts that enter the system.

Reported by: Jenkins
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D27080

riscv: remove sbi_clear_ipi()

S-mode software has write access to the SIP.SSIP bit, so instead of
making a second round-trip through the SBI we can clear it ourselves.
The SBI spec has deprecated this function for this exactly this reason.

Submitted by: Danjel Q. <danq1222@gmail.com
Reviewed by: kp
Differential Revision: https://reviews.freebsd.org/D26952

Use the HSM SBI extension to start APs

The addition of the HSM SBI extension to OpenSBI introduces a new
breaking change: secondary harts will remain parked in the firmware,
until they are brought up explicitly via sbi_hsm_hart_start(). Add
the call to do this, sending the secondary harts to mpentry.

If the HSM extension is not present, secondary harts are assumed to be
released by the firmware, as is the case for OpenSBI =< v0.6 and BBL.

In the case that the HSM call fails we exclude the CPU, notify the
user, and allow the system to proceed with booting.

Reviewed by: markj (older version)
Differential Revision: https://reviews.freebsd.org/D24497

Remove the secondary_stacks array in arm64 and riscv kernels.

Instead, dynamically allocate a page for the boot stack of each AP when
starting them up, like we do on x86. This shrinks the bss by
MAXCPU*KSTACK_PAGES pages, which corresponds to 4MB on arm64 and 256KB
on riscv.

Duplicate the logic used on x86 to free the bootstacks, by using a
sysinit to wait for each AP to switch to a thread before freeing its
stack.

While here, mark some static MD variables as such.

Reviewed by: kib
MFC after: 1 month
Sponsored by: Juniper Networks, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D24158

riscv: Remove unused variable

Fix the build that was broken by r356221. Pointy hat to myself.

riscv: Remove pointless loop

There's no point in checking for absent CPUs if we're not going to do anything
about either the present or absent case. This loop can just be removed.

Reviewed by: philip
Sponsored by: Axiado

RISC-V: Support EARLY_AP_STARTUP

The EARLY_AP_STARTUP option initializes non-boot processors
much sooner during startup. This adds support for this option
on RISC-V and enables it by default for GENERIC.

Reviewed by: jhb, markj
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D21661

Centralize __pcpu definitions.

Many extern struct pcpu <something>__pcpu declarations were
copied/pasted in sources. The issue is that the definition is MD, but
it cannot be provided by machine/pcpu.h due to actual struct pcpu
defined in sys/pcpu.h later than the inclusion of machine/pcpu.h.
This forced the copying when other code needed direct access to
__pcpu. There is no way around it, due to machine/pcpu.h supplying
part of struct pcpu fields.

To work around the problem, add a new machine/pcpu_aux.h header, which
should fill any needed MD definitions after struct pcpu definition is
completed. This allows to remove copies of __pcpu spread around the
source. Also on x86 it makes it possible to remove work arounds like
OFFSETOF_CURTHREAD or clang specific warnings supressions.

Reported and tested by: lwhsu, bcran
Reviewed by: imp, markj (previous version)
Discussed with: jhb
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D21418

Fix global pointer relaxations in the RISC-V kernel

The gp register is intended to used by the linker as another means of
performing relaxations, and should point to the small data section (.sdata).

Currently gp is being used as the pcpu pointer within the kernel, but the more
appropriate choice for this is the tp register, which is unused.

Swap existing usage of gp with tp within the kernel, and set up gp properly
at boot with the value of __global_pointer$ for all harts.

Additionally, remove some cases of accessing tp from the PCB, as it is not
part of the per-thread state. The user's tp and gp should be tracked only
through the trapframe.

Reviewed by: markj, jhb
Approved by: markj (mentor)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D19893

Extract eventfilter declarations to sys/_eventfilter.h

This allows replacing "sys/eventfilter.h" includes with "sys/_eventfilter.h"
in other header files (e.g., sys/{bus,conf,cpu}.h) and reduces header
pollution substantially.

EVENTHANDLER_DECLARE and EVENTHANDLER_LIST_DECLAREs were moved out of .c
files into appropriate headers (e.g., sys/proc.h, powernv/opal.h).

As a side effect of reduced header pollution, many .c files and headers no
longer contain needed definitions. The remainder of the patch addresses
adding appropriate includes to fix those files.

LOCK_DEBUG and LOCK_FILE_LINE_ARG are moved to sys/_lock.h, as required by
sys/mutex.h since r326106 (but silently protected by header pollution prior
to this change).

No functional change (intended). Of course, any out of tree modules that
relied on header pollution for sys/eventhandler.h, sys/lock.h, or
sys/mutex.h inclusion need to be fixed. __FreeBSD_version has been bumped.

Add support for HiFive Unleashed -- the board with a multi-core RISC-V SoC
from SiFive, Inc.

The first core on this SoC (hart 0) is a 64-bit microcontroller.

o Pick a hart to run boot process using hart lottery.
This allows to exclude hart 0 from running the boot process.
(BBL releases hart 0 after the main harts, so it never wins the lottery).
o Renumber CPUs early on boot.
Exclude non-MMU cores. Store the original hart ID in struct pcpu. This
allows to find out the correct destination for IPIs and remote sfence
calls.

Thanks to SiFive, Inc for the board provided.

Reviewed by: markj
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D20225

Implement per-CPU pmap activation tracking for RISC-V.

This reduces the overhead of TLB invalidations by ensuring that we
only interrupt CPUs which are using the given pmap. Tracking is
performed in pmap_activate(), which gets called during context switches:
from cpu_throw(), if a thread is exiting or an AP is starting, or
cpu_switch() for a regular context switch.

For now, pmap_sync_icache() still must interrupt all CPUs.

Reviewed by: kib (earlier version), jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18874

Don't enable interrupts in init_secondary().

The MI kernel assumes that interrupts will not be enabled on APs until
after the first context switch. In particular, the problem was causing
occasional deadlocks during boot.

Remove an unneeded intr_disable() added in r335005.

Reviewed by: jhb (previous version)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18738

SBI calls expect a pointer to a u_long rather than a pointer.

This is just cosmetic.

A weirder issue is that the SBI doc claims the hart mask pointer should
be a physical address, not a virtual address. However, the implementation
in bbl seems to just dereference the address directly.

Reviewed by: markj
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D17781

Various fixes for TLB management on RISC-V.

- Remove the arm64-specific cpu_*cache* and cpu_tlb_flush* functions.
Instead, add RISC-V specific inline functions in cpufunc.h for the
fence.i and sfence.vma instructions.
- Catch up to changes in the arm64 pmap and remove all the cpu_dcache_*
calls, pmap_is_current, pmap_l3_valid_cacheable, and PTE_NEXT bits from
pmap.
- Remove references to the unimplemented riscv_setttb().
- Remove unused cpu_nullop.
- Add a link to the SBI doc to sbi.h.
- Add support for a 4th argument in SBI calls. It's not documented but
it seems implied for the asid argument to SBI_REMOVE_SFENCE_VMA_ASID.
- Pass the arguments from sbi_remote_sfence*() to the SEE. BBL ignores
them so this is just cosmetic.
- Flush icaches on other CPUs when they resume from kdb in case the
debugger wrote any breakpoints while the CPUs were paused in the IPI_STOP
handler.
- Add SMP vs UP versions of pmap_invalidate_* similar to amd64. The
UP versions just use simple fences. The SMP versions use the
sbi_remove_sfence*() functions to perform TLB shootdowns. Since we
don't have a valid pm_active field in the riscv pmap, just IPI all
CPUs for all invalidations for now.
- Remove an extraneous TLB flush from the end of pmap_bootstrap().
- Don't do a TLB flush when writing new mappings in pmap_enter(), only if
modifying an existing mapping. Note that for COW faults a TLB flush is
only performed after explicitly clearing the old mapping as is done in
other pmaps.
- Sync the i-cache on all harts before updating the PTE for executable
mappings in pmap_enter and pmap_enter_quick. Previously the i-cache was
only sync'd after updating the PTE in pmap_enter.
- Use sbi_remote_fence() instead of smp_rendezvous in pmap_sync_icache().

Reviewed by: markj
Approved by: re (gjb, kib)
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D17414

Eliminate kmem_malloc()'s unused arena parameter. (The arena parameter
became unused in FreeBSD 12.x as a side-effect of the NUMA-related
changes.)

Reviewed by: kib, markj
Discussed with: jeff, re@
Differential Revision: https://reviews.freebsd.org/D16825

o Add driver for PLIC (Platform-Level Interrupt Controller) device.
o Convert interrupt machdep support to use INTRNG code.

Sponsored by: DARPA, AFRL

Release secondary cores from WFI (wait for interrupt) by sending them
an IPI.

This does not work however yet in QEMU. As a temporary workaround set
software interrupt pending bit manually on a local core to ensure WFI
doesn't halt the hart.

This is required to smpboot in QEMU.

Sponsored by: DARPA, AFRL

Add full softfloat and hardfloat support for RISC-V.

Hardfloat is now default (use riscv64sf as TARGET_ARCH
for softfloat).

Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D8529

o Remove operation in machine mode.
Machine privilege level was specially designed to use in vendor's
firmware or bootloader. We have implemented operation in machine
mode in FreeBSD as part of understanding RISC-V ISA, but it is time
to remove it.
We now use BBL (Berkeley Boot Loader) -- standard RISC-V firmware,
which provides operation in machine mode for us.
We now use standard SBI calls to machine mode, instead of handmade
'syscalls'.
o Remove HTIF bus.
HTIF bus is now legacy and no longer exists in RISC-V specification.
HTIF code still exists in Spike simulator, but BBL do not provide
raw interface to it.
Memory disk is only choice for now to have multiuser booted in Spike,
until Spike has implemented more devices (e.g. Virtio, etc).

Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

Add support for symmetric multiprocessing (SMP).

Tested on Spike simulator with 2 and 16 cores (tlb enabled),
so set MAXCPU to 16 at this time.

This uses FDT data to get information about CPUs
(code based on arm64 mp_machdep).

Invalidate entire TLB cache as it is the only way yet.

Sponsored by: DARPA, AFRL
Sponsored by: HEIF5