riscv: Introduce support for APLIC interrupt controller

This patch introduces support for the RISC-V APLIC interrupt controller
[1]. Currently, it is only supports direct mode, i.e. without an IMSIC
and functionally replacing the legacy RISC-V PLIC. Work on IMSIC support
is in progress.

[1] https://github.com/riscv/riscv-aia/releases/tag/1.0

Reviewed by: mhorne
Discussed with: jrtc27
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D43293

riscv: Convert local interrupt controller to a newbus PIC

Currently the local interrupt controller implementation is based on
pre-INTRNG arm/arm64 code, using hand-rolled event code rather than
INTRNG. This then interacts weirdly with the PLIC, and other future
interrupt controllers like the APLIC and IMSICs in the upcoming AIA
specification, since they become the root PIC despite not being the
logical root. Instead, use a real newbus device for it and register
it as the root PIC.

This also adapts the IPI code to make use of the newly-added INTRNG
generic IPI handling framework, adding a new sbi_ipi as the PIC. In
future there will be alternative devices for sending IPIs that will
register with higher priorities, such as the proposed AIA IMSIC and
ACLINT SSWI.

Reviewed by: mhorne
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D35901

sys: Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

Consistently provide ffs/fls using builtins

Use of compiler builtin ffs/ctz functions will result in optimized
instruction sequences when possible, and fall back to calling a function
provided by the compiler run-time library. We have slowly shifted our
platforms to take advantage of these builtins in 60645781d613 (arm64),
1c76d3a9fbef (arm), 9e319462a03a (powerpc, partial).

Some platforms still rely on the libkern implementations of these
functions provided by libkern, namely riscv, powerpc (ffs*, flsll), and
i386 (ffsll and flsll). These routines are slow, as they perform a
linear search for the bit in question. Even on platforms lacking
dedicated bit-search instructions, such as riscv, the compiler library
will provide better-optimized routines, e.g. by using binary search.

Consolidate all definitions of these functions (whether currently using
builtins or not) to libkern.h. This should result in equivalent or
better performing routines in all cases.

One wart in all of this is the existing HAVE_INLINE_F*** macros, which
we use in a few places to conditionally avoid the slow libkern routines.
These aren't easily removed in one commit. For now, provide these
defines unconditionally, but marked for removal after subsequent
cleanup.

Removal of the now unused libkern routines will follow in the next
commit.

Reviewed by: dougm, imp (previous version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40698

riscv: Compile instr_size.c into the kernel when DTrace is configured

Reported by: Jenkins
Fixes: 080e56a6c98c ("dtrace: expose dtrace_instr_size() to userland and implement it for riscv")

Import an optimized str{n}cmp on arm64

These are from the Arm Optimized Routines and don't use the VFP so are
safe to use in the kernel.

Sponsored by: The FreeBSD Foundation

files: Make ext_resources non-optional

EXT_RESOURCES have been introduced in 12-CURRENT and all supported
releases have it enabled in their kernel config.

MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D33834

riscv: gdb(4) support

Add the MD portion required for the gdb stub.

Reviewed by: jhb (earlier version)
Discussed with: jrtc27
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33734

netinet: Deduplicate most in_cksum() implementations

in_cksum() and related routines are implemented separately for each
platform, but only i386 and arm have optimized versions. Other
platforms' copies of in_cksum.c are identical except for style
differences and support for big-endian CPUs.

Deduplicate the implementations for the rest of the platforms. This
will make it easier to implement in_cksum() for unmapped mbufs. On arm
and i386, define HAVE_MD_IN_CKSUM to mean that the MI implementation is
not to be compiled.

No functional change intended.

Reviewed by: kp, glebius
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33095

riscv: move signal delivery code to exec_machdep.c

Reviewed by: emaste, imp
Discussed with: jrtc27
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D32310

riscv: retire bcmp

Unused since ba96f37758412151 ("Use __builtin for various mem* and b* (e.g. bzero)
routines.")

Reviewed by: mhorne
Sponsored by: Rubicon Communications, LLC ("Netgate")

fu740_pci_dw: Add SiFive FU740 PCIe controller driver

Reviewed by: mhorne
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31033

Rename ofwpci.c to ofw_pcib.c

It's a class0 driver that implements some pcib methods and creates
a pci bus as its children.
The "ofw_pci" name will be used by a new driver that will be a subclass
of the pci bus.
No functional changes intended.

Submitted by: Kornel Duleba <mindal@semihalf.com>
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: Alstom Group
Differential Revision: https://reviews.freebsd.org/D30226

riscv: Remove old qemu compatibility code

During early qemu development, the /soc node was marked as compatible
with "riscv-virtio-soc" instead of "simple-bus".

This was changed in qemu 53f54508dae6 in Sep 2018, and predates the
baseline required qemu version (5.0) for riscv by a wide margin.

The generic simplebus code handles attachment in all cases nowadays.

Sponsored by: Tag1 Consulting, Inc.
Reviewed by: jrtc27, mhorne
Differential Revision: https://reviews.freebsd.org/D30011

amd64: implement strlen in assembly

The C variant in libkern performs excessive branching to find the
non-zero byte instead of using the bsfq instruction. The same code
patched to use it is still slower than the routine implemented here
as the compiler keeps neglecting to perform certain optimizations
(like using leaq).

On top of that the routine can is a starting point for copyinstr
which operates on words instead of bytes.

Tested with glibc test suite.

Sample results (calls/s):

Haswell:
$(perl -e "print 'A' x 3"):
stock: 211198039
patched:338626619
asm: 465609618

$(perl -e "print 'A' x 100"):
stock: 83151997
patched: 98285919
asm: 120719888

AMD EPYC 7R32:
$(perl -e "print 'A' x 3"):
stock: 282523617
asm: 491498172

$(perl -e "print 'A' x 100"):
stock: 114857172
asm: 112082057

Move kern_clocksource.c to sys/conf/files

Sponsored by: The FreeBSD Foundation

riscv: Include syscon_power device driver in GENERIC kernel config

QEMU's RISC-V virt machine provides syscon-power and syscon-reset
devices as the means by which to shutdown and reboot. We also need to
ensure that we have attached the syscon_generic device before attaching
any syscon_power devices, and so we introduce a new riscv_syscon device
akin to aw_syscon added in r327936. Currently the SiFive test finisher
is used as the specific implementation of such a syscon device.

Reviewed by: br, brooks (mentor), jhb (mentor)
Approved by: br, brooks (mentor), jhb (mentor)
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D25725

copystr(9): Move to deprecate (attempt #2)

This reapplies logical r360944 and r360946 (reverting r360955), with fixed
copystr() stand-in replacement macro. Eventually the goal is to convert
consumers and kill the macro, but for a first step it helps if the macro is
correct.

Prior commit message:

Unlike the other copy*() functions, it does not serve to copy from one
address space to another or protect against potential faults. It's just
an older incarnation of the now-more-common strlcpy().

Add a coccinelle script to tools/ which can be used to mechanically
convert existing instances where replacement with strlcpy is trivial.
In the two cases which matched, fuse_vfsops.c and union_vfsops.c, the
code was further refactored manually to simplify.

Replace the declaration of copystr() in systm.h with a small macro
wrapper around strlcpy (with correction from brooks@ -- thanks).

Remove N redundant MI implementations of copystr. For MIPS, this
entailed inlining the assembler copystr into the only consumer,
copyinstr, and making the latter a leaf function.

Reviewed by: jhb (earlier version)
Discussed with: brooks (thanks!)
Differential Revision: https://reviews.freebsd.org/D24672

Revert r360944 and r360946 until reported issues can be resolved

Reported by: cy

Remove deleted files from the build

Fix build break introduced in r360944.

Reported by: kevans

Remove support for DES and Triple DES from OCF.

It no longer has any in-kernel consumers via OCF. smbfs still uses
single DES directly, so sys/crypto/des remains for that use case.

Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24773

Remove support for the Blowfish algorithm from OCF.

It no longer has any in-kernel consumers.

Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24772

RISC-V: use physmem to manage physical memory

Replace our hand-rolled functions with the generic ones provided by
kern/subr_physmem.c. This greatly simplifies the initialization of
physical memory regions and kernel globals.

Tested by: nick
Differential Revision: https://reviews.freebsd.org/D24154

Add the Cadence GEM ethernet driver to NOTES so that it gets built with
LINT kernels. Move the config for it from files.<arch> files into the
main config (conf/files), because it works on multiple platforms now.

Add driver for Xilinx XDMA PCIe Bridge found in the U.S. Government
Furnished Equipment (GFE) riscv cores.

GFE cores are synthesized on the Xilinx Virtex UltraScale+ FPGA VCU118
Evaluation Kit.

Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D23337

Include the PCI stack to the riscv GENERIC kernel.

It will be used by an upcoming PCI root complex driver.

Sponsored by: DARPA, AFRL

RISC-V: add support for SBI spec v0.2

The Supervisor Binary Interface (SBI) specification v0.2 is a backwards
incompatible update to the SBI call interface for kernels running in
supervisor mode. The goal of this update was to make it easier for new
and optional functionality to be added to the SBI.

SBI functions are now called by passing an "extension ID" and a
"function ID" which are passed in a7 and a6 respectively. SBI calls
will also return an error and value in the following struct:

struct sbi_ret {
long error;
long value;
}

This version introduces several new functions under the "base"
extension. It is expected that all SBI implementations >= 0.2 will
support this base set of functions, as they implement some essential
services such as obtaining the SBI version, CPU implementation info, and
extension probing.

Existing SBI functions have been designated as "legacy". For the time
being they will remain implemented, but it is expected that in the
future their functionality will be duplicated or replaced by new SBI
extensions. Each legacy function has been assigned its own extension ID,
and for now we simply probe and assert for their existence.

Compatibility with legacy SBI implementations (such as BBL) is
maintained by checking the output of sbi_get_spec_version(). This
function is guaranteed to succeed by the new spec, but will return an
error in legacy implementations. We use this as an indicator of whether
or not we can rely on the new SBI base extensions.

For further info on the Supervisor Binary Interface, see:
https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.adoc

Reviewed by: kp, jhb
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D22326

emulate illumos membar_producer with atomic_thread_fence_rel

membar_producer is supposed to be a store-store barrier.
Also, in the code that FreeBSD has ported from illumos membar_producer
is used only with regular stores to regular memory (with respect to
caching).

We do not have an MI primitive for the store-store barrier, so
atomic_thread_fence_rel is the closest we have as it provides
(load | store) -> store barrier.

Previously, membar_producer was an empty function call on all 32-bit
arm-s, 32-bit powerpc, riscv and all mips variants. I think that it was
inadequate.
On other platforms, such as amd64, arm64, i386, powerpc64, sparc64,
membar_producer was implemented using stronger primitives than required
for a store-store barrier with respect to regular memory access.
For example, it used sfence on amd64 and lock-ed nop in i386 (despite TSO).
On powerpc64 we now use recommended lwsync instead of eieio.
On sparc64 FreeBSD uses TSO mode.
On arm64/aarch64 we now use dmb sy instead of dmb ish. Not sure if this
is an improvement, actually.

After this change we can drop opensolaris_atomic.S for aarch64, amd64,
powerpc64 and sparc64 as all required atomic operations have either
direct or light-weight mapping to FreeBSD native atomic operations.

Discussed with: kib
MFC after: 4 weeks

Connect Xilinx AXI drivers and Cadence Ethernet MAC to the RISC-V build.

Sponsored by: DARPA, AFRL

Provide a template for busdma code for RISC-V.

RISC-V ISA specifies no cache management instructions so leave cache
operations in cpufunc.h as no-op for now.

Note some new hardware comes with their own memory-mapped cache
management controller.

Tested on HiFive Unleashed board with cgem(4).

Reviewed by: markj
Obtained from: arm64
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D20126

Add support for the UART device found in lowRISC system-on-a-chip.

The only source of documentation for this device is verilog,
so driver is minimalistic.

Reviewed by: Dr Jonathan Kimmitt <jrrk2@cam.ac.uk>
Approved by: re (kib)
Sponsored by: DARPA, AFRL

amd64: implement memcmp in assembly

Both the in-kernel C variant and libc asm variant have very poor performance.
The former compiles to a single byte comparison loop, which breaks down even
for small sizes. The latter uses rep cmpsq/b which turn out to have very poor
throughput and are slower than a hand-coded 32-byte comparison loop.

Depending on size this is about 3-4 times faster than the current routines.

Reviewed by: kib
Approved by: re (gjb)
Differential Revision: https://reviews.freebsd.org/D17328

Make ZSTD a real option via ZSTDIO.

It looks like the intent was to allow ZSTD support to be
compiled into the kernel with options ZSTDIO. But it doesn't look
like that was ever implemented or I'm missing how to do it.

I did a cursory audit of kernel config files and made a decision to
enable ZSTDIO in riscv GENERIC and mips MALTA configurations. All other
kernel configurations already had this option in their kernel configs
but they didn't do anything useful as the feature was declared as
"standard" prior to this.

Reviewed by: cem allanjude
Differential Revision: https://reviews.freebsd.org/D16007

o Add driver for PLIC (Platform-Level Interrupt Controller) device.
o Convert interrupt machdep support to use INTRNG code.

Sponsored by: DARPA, AFRL

Add simplebus-like RISC-V SoC bus.

This is required in order to probe and attach devices described under
"riscv-virtio-soc" node of DTS.

Sponsored by: DARPA, AFRL

Simplify things a little

Rather than include a copy for memmove to call bcopy to call memcpy
(which handles overlapping copies), make memmove a strong reference to
memcpy to save the two calls.

Differential Revision: https://reviews.freebsd.org/D15374

Move MI-ish bcopy routine to libkern

riscv and powerpc have nearly identical bcopy.c that's
supposed to be mostly MI. Move it to the MI libkern.

Differential Revision: https://reviews.freebsd.org/D15374

Include ffsll() on riscv kernels.

Finally, fix Zstd kernel build on MIPS and RISC-V

Add an implementation of the intrinsics invoked by __builtin_ctz{,ll} and
__builtin_clz{,ll}, and include this compilation unit on platforms that lack
assembly intrinsics for those builtins (MIPS and RISC-V).

Future cleanup work might involve bringing these into a mini libcompiler-rt
for the standalone kernel environment. Or cleaning up the approach upstream
takes for builtins in standalone environments (or just FreeBSD). For now,
at least this builds, and doesn't require modifying the vendor code.

Reported by: jeff, markj, mizhka
Reviewed by: jhb (earlier version), rpokala (comment text earlier version)
Sponsored by: Dell EMC Isilon

Support for v1.10 (latest) of RISC-V privilege specification.

New version is not compatible on supervisor mode with v1.9.1
(previous version).

Highlights:
o BBL (Berkeley Boot Loader) provides no initial page tables
anymore allowing us to choose VM, to build page tables manually
and enable MMU in S-mode.
o SBI interface changed.
o GENERIC kernel.
FDT is now chosen standard for RISC-V hardware description.
DTB is now provided by Spike (golden model simulator). This
allows us to introduce GENERIC kernel. However, description
for console and timer devices is not provided in DTB, so move
these devices temporary to nexus bus.
o Supervisor can't access userspace by default. Solution is to
set SUM (permit Supervisor User Memory access) bit in sstatus
register.
o Compressed extension is now turned on by default.
o External GCC 7.1 compiler used.
o _gp renamed to __global_pointer$
o Compiler -march= string is now in use allowing us to choose
required extensions (compressed, FPU, atomic, etc).

Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D11800

Merge projects/ipsec into head/.

Small summary
-------------

o Almost all IPsec releated code was moved into sys/netipsec.
o New kernel modules added: ipsec.ko and tcpmd5.ko. New kernel
option IPSEC_SUPPORT added. It enables support for loading
and unloading of ipsec.ko and tcpmd5.ko kernel modules.
o IPSEC_NAT_T option was removed. Now NAT-T support is enabled by
default. The UDP_ENCAP_ESPINUDP_NON_IKE encapsulation type
support was removed. Added TCP/UDP checksum handling for
inbound packets that were decapsulated by transport mode SAs.
setkey(8) modified to show run-time NAT-T configuration of SA.
o New network pseudo interface if_ipsec(4) added. For now it is
build as part of ipsec.ko module (or with IPSEC kernel).
It implements IPsec virtual tunnels to create route-based VPNs.
o The network stack now invokes IPsec functions using special
methods. The only one header file <netipsec/ipsec_support.h>
should be included to declare all the needed things to work
with IPsec.
o All IPsec protocols handlers (ESP/AH/IPCOMP protosw) were removed.
Now these protocols are handled directly via IPsec methods.
o TCP_SIGNATURE support was reworked to be more close to RFC.
o PF_KEY SADB was reworked:
- now all security associations stored in the single SPI namespace,
and all SAs MUST have unique SPI.
- several hash tables added to speed up lookups in SADB.
- SADB now uses rmlock to protect access, and concurrent threads
can do SA lookups in the same time.
- many PF_KEY message handlers were reworked to reflect changes
in SADB.
- SADB_UPDATE message was extended to support new PF_KEY headers:
SADB_X_EXT_NEW_ADDRESS_SRC and SADB_X_EXT_NEW_ADDRESS_DST. They
can be used by IKE daemon to change SA addresses.
o ipsecrequest and secpolicy structures were cardinally changed to
avoid locking protection for ipsecrequest. Now we support
only limited number (4) of bundled SAs, but they are supported
for both INET and INET6.
o INPCB security policy cache was introduced. Each PCB now caches
used security policies to avoid SP lookup for each packet.
o For inbound security policies added the mode, when the kernel does
check for full history of applied IPsec transforms.
o References counting rules for security policies and security
associations were changed. The proper SA locking added into xform
code.
o xform code was also changed. Now it is possible to unregister xforms.
tdb_xxx structures were changed and renamed to reflect changes in
SADB/SPDB, and changed rules for locking and refcounting.

Reviewed by: gnn, wblock
Obtained from: Yandex LLC
Relnotes: yes
Sponsored by: Yandex LLC
Differential Revision: https://reviews.freebsd.org/D9352

o Remove operation in machine mode.
Machine privilege level was specially designed to use in vendor's
firmware or bootloader. We have implemented operation in machine
mode in FreeBSD as part of understanding RISC-V ISA, but it is time
to remove it.
We now use BBL (Berkeley Boot Loader) -- standard RISC-V firmware,
which provides operation in machine mode for us.
We now use standard SBI calls to machine mode, instead of handmade
'syscalls'.
o Remove HTIF bus.
HTIF bus is now legacy and no longer exists in RISC-V specification.
HTIF code still exists in Spike simulator, but BBL do not provide
raw interface to it.
Memory disk is only choice for now to have multiuser booted in Spike,
until Spike has implemented more devices (e.g. Virtio, etc).

Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

Add initial DTrace support for RISC-V.

Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

o Add device tree files and kernel configuration files
for RISC-V cpus synthesized on FPGA hardware.
o Include new files to the build.

Move arm's devmap to some generic place, so it can be used
by other architectures.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D6091
Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

Add support for ddb(4).

Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

Add support for symmetric multiprocessing (SMP).

Tested on Spike simulator with 2 and 16 cores (tlb enabled),
so set MAXCPU to 16 at this time.

This uses FDT data to get information about CPUs
(code based on arm64 mp_machdep).

Invalidate entire TLB cache as it is the only way yet.

Sponsored by: DARPA, AFRL
Sponsored by: HEIF5

Set a dependencies for stack(9) RISC-V MD part.

Pointed out by: andrew

Provide stack(9) MD stubs for RISC-V so ktr(9) can be compiled in.

Welcome the RISC-V 64-bit kernel.

This is the final step required allowing to compile and to run RISC-V
kernel and userland from HEAD.

RISC-V is a completely open ISA that is freely available to academia
and industry.

Thanks to all the people involved! Special thanks to Andrew Turner,
David Chisnall, Ed Maste, Konstantin Belousov, John Baldwin and
Arun Thomas for their help.
Thanks to Robert Watson for organizing this project.

This project sponsored by UK Higher Education Innovation Fund (HEIF5) and
DARPA CTSRD project at the University of Cambridge Computer Laboratory.

FreeBSD/RISC-V project home: https://wiki.freebsd.org/riscv

Reviewed by: andrew, emaste, kib
Relnotes: Yes
Sponsored by: DARPA, AFRL
Sponsored by: HEIF5
Differential Revision: https://reviews.freebsd.org/D4982