libsa: Move hash functions up a level

This should have no functional change. Move compiling the sha256, sha512
and md5 hash functions up into libsa to allow them to be used elsewhere
in the boot loader when geli isn't configured. Since libsa is a .a, these
won't wind up in any boot loader that doesn't reference them, so should
be a nop.

Sponsored by: Netflix

libsa: Move include file creation to the end of the Makefile

The include file symblic links, etc are out of place where they
are. Move them to the end of the file. No functional change intended.

Sponsored by: Netflix

loader: provide a features table for binary compatibility advertisement

liblua now provides a loader.has_feature() function to probe the loader
binary for features advertised. name => desc mappings are provided in
loader.features to get a list of all of the features loader *can*
support. core.hasFeature is provided as a shim to loader.has_feature
so that individual consumers don't need to think about the logic of the
loader module not providing has_feature; we know that means the feature
isn't enabled.

The first consumer of this will be EARLY_ACPI to advertise that the
loader binary probes for ACPI presence before the interpreter has
started, so that we know whether we can trust the presence of acpi.rsdp
as relatively authoritative. In general, it's intended to be used to
avoid breaking new scripts on older loaders within reason.

This will be used in lua as `core.hasFeature("EARLY_ACPI")`, while the
C bits of loader will `feature_enable(FEATURE_EARLY_ACPI)`.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D42695

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

stand: Separate base and cli parts of nvstore

zfs lives in libsa. However, it depends on nvstore (and other things)
that are in common. Fix part of this layering violation by splitting
nvstore into a libsa piece (which is the base implementation) and
keeping a much smaller common piece (to implement the nvstore
command). This just leaves zfs' knowledge of device names that's
specific to common and its calling platform specific init code to
resolve. Add a nvstore.h file for these two parts to communicate private
things and move the public nvstore api from bootstrap.h to stand.h.

Sponsored by: Netflix
Reviewed by: tsoome, kevans
Differential Revision: https://reviews.freebsd.org/D38043

stand: Add inttype.h

libunwind files need inttype.h. It's safe so add it to the safe list.

Sponsored by: Netflix
Reviewed by: jrtc27, kevans
Differential Revision: https://reviews.freebsd.org/D37947

stand/zlib: Document the upstream issue behind NO_DEPRECATED_NON_PROTOTYPE

The zlib project has issue https://github.com/madler/zlib/issues/633 to
document its continued use of old K&R-style function definitions.

Suggested by: delphij@
Sponsored by: Netflix

stand/zlib: Zlib still uses K&R function definitions

So add ${NO_WDEPRECATED_NON_PROTOTYPE} to the CFLAGS of those
files. This can be removed when we import a zlib that's free of this
anachronism.

Sponsored by: Netflix
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D37516

stand: Compile out the extensive superblock diagnostic messages for BIOS loader

The BIOS loader operates in a very constrained environment. The messages
for the super block integrity tests take up about 12k of space. Compile
them out for the BIOS loader, while leaving it intact for all other
loaders that aren't space constrained. These aren't used in the 'super
tiny' *boot* programs, so no adjustment is needed there.

We reply on the fact that (a) i386 doesn't support 32-bit UEFI booting
and (b) LIBSA_CPUARCH is "i386" when building on both i386 and when
we're building the 32-bit libsa32 library.

This saves about 12k of space for this constrained envrionment and will
take a bit of the pressure off some machines where the loader has grown
too big for their BIOS (see comments in i386/loader/Makefile for
details).

Sponsored by: Netflix
Reviewed by: mckusick
Differential Revision: https://reviews.freebsd.org/D36175

stand: Use devformat instead of disk_devfmt

Use devformat instead of disk_devfmt. This allows us to avoid knowing
the details of the device that's underneath us. Remove disk.h include
and the -I${LDRSRC} from the build of ufs.c since they are no longer
needed.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D35922

stand: Use devformat rather than disk_devfmt

Fix layering violation and use devformat to get the string
representation of the device to see if we're mounted yet or not. Remove
added include to pickup disk.h.

Sponsored by: Netflix
Reviewed by: tsoome (prior version)
Differential Revision: https://reviews.freebsd.org/D35919

stand: tftp.c doesn't need bootinfo.h

tftp.c includes bootinfo.h, but doesn't need it. Remove it, and remove
the -Istand/common from CFALGS since that's the only reason we had it.

Sponsored by: Netflix
Reviewed by: tsoome
Differential Revision: https://reviews.freebsd.org/D35928

stand libsa: Restore include path to LDRSRC for disk.h for filesystems.

In theory they shouldn't need anything outside of libsa, but disk.h and
bootstrap.h are currently required. Future work wil address this issue.

This fixes the build with MK_LOADER_ZFS=no. ZFS's Makefile.inc adds
these flags globally to CFLAGS when it should not. This masked the
problem because the tools/boot/universe.sh didn't build MK_LOADER_ZFS=no
as part of its regressions. Future work will also fix this.

Obtained from: CheriBSD
Fixes: 84bf2bbbecc3 stand: constrain zlib/gzip CFLAGS better
Sponsored by: DARPA
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D35860

stand: Add comment about CFLAGS pollution

Add a note that veriexec / bearssl builds will pollute the CFLAGS in a
way that's somewhat hard to fix, so I'm just noting it for now.

Sponsored by: Netflix

stand: constrain zlib/gzip CFLAGS better

Define ZLIB_CFLAGS and use it only for the sources that are in ZLIB or
that include it.

Sponsored by: Netflix

stand: Confine BZIP defines to bzip files

Sponsored by: Netflix

stand: Install libsa.3

Turns out there is a libsa.3. It's a bit out of date, but we reference
it in a number of places so we should install it. We need to do the DO32
dance because this Makefile is included twice and we don't want it
installing twice.

Sponsored by: Netflix

WITHOUT_BOOT build option appears to include stand, failing on libsa

building libsa needs to use -I${LDRSRC} for some files.

PR: 260083
Submitted by: Ivan Rozhuk
MFC: 1 day

loader: Add preload operation to fs_ops

When we load an ELF file (kernel or module) we do seek(2) a lot to
parse/load the different sections of the ELF file.
Protocol like TFTP suffers a lot from this as there is no resume or
a way to start the tranfer from a specified offset in the file.
fs_preload is added to help those protocol.
Call preload just after opening the ELF file that we need to load so
the underlying method can cache the hole file and then read/lseek operations
are faster.

Reviewed by: imp
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D33409

loader: implement mount/unmount rootfs

We want to keep our root file system open to preserve bcache segment
between file accesses, thus reducing physical disk IO.

Reviewed by: imp, allanjude, kevans (previous version)
Differential Revision: https://reviews.freebsd.org/D30848
MFC after: 1 month

libsa: Add support for timestamp logging (tslog)

At present this only supports x86, due to the use of the rdtsc
instruction; and is inert unless a buffer is allocated and passed to
the tslog code (which will be done by a future commit).

Reviewed by: kevans

libc: Use musl's optimized strchr and strchrnul

Parentheses added to HASZERO macro to avoid a GCC warning, and formatted
with clang-format as we have adopted these and don't consider them
'contrib' code.

Obtained from: musl (snapshot at commit 4d0a82170a25)
Reviewed by: kib (libc integration), mjg (both earlier)
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17630

Implement some time variables from kernel

OpenZFS will start using some of the kernel timekeeping bits
shortly. This implements the bare minimum of that which currently
is just the time_seconds variable.

Work around cp breakage in current from last week

There was a small window cp was broken. Work around this by using :>
instead of cp /dev/null. Ideally, we'd keep the cp /dev/null in the
build as a regression test, but doing so breaks people that upgraded
during the cp breakage and this is simpler than bootstrapping a
working cp since there's no good __FreeBSD_version sign posts for
that.

Suggested by: lots of people
Too stubborn for his own good: imp

zalloc_malloc:Free hexdump preceeding buffer when we detect overflow

Move hexdump from stand/common/misc.c to stand/libsa/hexdump.c
(svn cp)
Disable use of pager - causes linking issue for boot1
can be re-enabled by defining HEXDUMP_PAGER.

Reviewed by: stevek, imp
MFC after: 1 week
Sponsored by: Juniper Networks
Differential Revision: https://reviews.freebsd.org/D26235

Remove tests for obsolete compilers in the build system

Assume gcc is at least 6.4, the oldest xtoolchain in the ports tree.
Assume clang is at least 6, which was in 11.2-RELEASE. Drop conditions
for older compilers.

Reviewed by: imp (earlier version), emaste, jhb
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D24802

Move smbios.c to libsa.

smbios used to be an i386 only kinda weird quirk to the x86
architecture. But UEFI picked it up, dusted it off and now it's many
other locations. Make it base technology by moving it to libsa and
fixing up the compliation. The code has issues with unaligned access
still, but that will be addressed in a followup commit.

Differential Revision: https://reviews.freebsd.org/D23660

Move all sources from the llvm project into contrib/llvm-project.

This uses the new layout of the upstream repository, which was recently
migrated to GitHub, and converted into a "monorepo". That is, most of
the earlier separate sub-projects with their own branches and tags were
consolidated into one top-level directory, and are now branched and
tagged together.

Updating the vendor area to match this layout is next.

Remove duplicate lz4 implementations

Port illumos change: https://www.illumos.org/issues/11667

Move lz4.c out of zfs tree to opensolaris/common/lz4, adjust it to be
usable from kernel/stand/userland builds, so we can use just one single
source. Add lz4.h to declare lz4_compress() and lz4_decompress().

MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D22037

Add support for hypervisor check on x86

Add ficl words for isvirtualized
and move ficl inb and outb words to ficl/x86/sysdep.c
so can be shared by i386 and amd64

Reviewed by: imp bdrewery
MFC after: 1 week
Sponsored by: Juniper Networks
Differential Revision: https://reviews.freebsd.org/D22069

stand: gptboot: fix build with xtoolchain-llvm90

ufsread.c grows a dependency on __ashldi3 with llvm90. For gptboot, just
start pulling in ashldi3.c ashrdi3.c lshrdi3.c into libsa for all archs as
the number of archs requiring one or more of them keeps growing. qdivrem.c
and quad.h can be trivially kicked out of libsa if we start pulling these
from compiler-rt as qdivrem was only used to implement umoddi3, divdi3,
moddi3 (also in qdivrem.c).

Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D21291

stand: push LIBC_SRC up into defs.mk

Other parts of stand/ that don't use libsa will need to grab bits from libc
shortly. Push LIBC_SRC up to defs.mk in advance of this so that they can use
it, and rename it to LIBCSRC to match the convention of the rest of the *SRC
variables in this file.

Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D21282

Revert r351078, r351085: stand/quad.h eviction

It did not go well; further examination is required...

stand: kick out quad.h

Use quad.h from libc instead for the time being. This reduces the number of
nearly-identical-quad.h we have in tree to two with only minor changes.

Prototypes for some *sh*di3 have been added to match the copy in libkern.
The differences between the two are likely few enough that they can perhaps
be merged with little additional effort to bring us down to 1.

MFC after: 3 days

Remove NAND and NANDFS support

NANDFS has been broken for years. Remove it. The NAND drivers that
remain are for ancient parts that are no longer relevant. They are
polled, have terrible performance and just for ancient arm
hardware. NAND parts have evolved significantly from this early work
and little to none of it would be relevant should someone need to
update to support raw nand. This code has been off by default for
years and has violated the vnode protocol leading to panics since it
was committed.

Numerous posts to arch@ and other locations have found no actual users
for this software.

Relnotes: Yes
No Objection From: arch@
Differential Revision: https://reviews.freebsd.org/D20745

Move contrib/zlib to sys/contrib/zlib so that we can use it in kernel.
This is a prerequisite of unifying kernel zlib instances.

Submitted by: Yoshihiro Ota <ota at j.email.ne.jp>
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D20191

Enable veriexec for loader

This relies on libbearssl and libsecureboot
to verify files read by loader in a maner equivalent
to how mac_veriexec

Note: disabled by default.
Use is initially expected to be by embeded vendors

Reviewed by: emaste, imp
Sponsored by: Juniper Networks
Differential Revision: D16336

MK_ZFS -> {MK_ZFS|MK_LOADER_ZFS}, this is so we can diable userland / kernel
ZFS but keep the boot-loaders when using ZoL port.

MFC after: 1 week
Reviewed by: rgrimes
Differential Revision: https://reviews.freebsd.org/D18739

Fix stand/ build after r339671.

ffs_subr.c requires calculate_crc32c() from libkern. Unfortunately we
cannot just add libkern/crc32.c to libstand because crc32.o is already
compiled from contrib/zlib/crc32.c. Use the include trick to rename
the source.

Note that libstand also provides crc32.c which seems to be unused.

Reviewed by: imp
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D17677

libsa: Add lshrdi3.c for powerpc* and mips

Add ashldi3 and ashrdi3 to mips.

Now that we're using -Os, mips needs these routines.

There's two files in the sys tree named inflate.c, in addition
to it being a common name elsewhere. Rename the old kzip one
to subr_inflate.c.

This actually fixes the build issues on sparc64 that my inclusion of
.PATH ${SYSDIR}/kern created in r336244, so also revert the broken
workaround I committed in r336249.

This slipped passed me because apparently, I never did a clean build.

Fix sparc64 builds

gcc is complaining about struct infate being defined in a parameter
list. It's inclear how long this has been broken, but the fix is
simple enough.

Create helper functions for parsing boot args.

boot_parse_arg to parse a single arg
boot_parse_cmdline to parse a command line string
boot_parse_args to parse all the args in a vector
boot_howto_to_env Convert howto bits to env vars
boot_env_to_howto Return howto mask mased on what's set in the environment.

All these routines return an int that's the bitmask of the args
translated to RB_* flags. As a special case, the 'S' flag sets the
comconsole_speed env var. Any arg that looks like a=b will set the env
key 'a' to value 'b'. If =b is omitted, 'a' is set to '1'. This
should help us reduce the number of redundant copies of these routines
in the tree. It should also give a more uniform experience between
platforms.

Also, invent a new flag RB_PROBE that's set when 'P' is parsed. On
x86 + BIOS, this means 'probe for the keyboard, and if it's not there
set both RB_MULTIPLE and RB_SERIAL (which means show the output on
both video and serial consoles, but make serial primary). Others it
may be some similar concept of probing, but it's loader dependent
what, exactly, it means.

These routines are suitable for /boot/loader and/or the kernel,
though they may not be suitable for the tightly hand-rolled-for-space
environments like boot2.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D16205

Move ZFS files into libsa

Move the libzfs stuff into libsa. There's no need for it to be a
separate library. The separate library adds to the issues of build
ordering that we see from time to time. Move the filesystem support
into libsa, like all the other filesystem support rather than making
zfs the odd-duck out.

Discussed with: allanjude@

Remove stray debugging line.

Noticed by: ian@

stand: move libgeliboot into libsa.

Reduce by 1 the number of crazy libraries we need in stand by moving
geli into libsa (where architecturally it belonged all along). This
just moves things around without any code changes.

Eliminate bsd.stand.mk and -fPIC 32-bit intel builds

OK. We don't really need a bsd.stand.mk, and it was causing a -fPIC
for the toolchain to be added (bogusly) when building on amd64. Pull
all relevant defs back into defs.mk and delete bsd.stand.mk.

This saves about 15-20k on i386 loader and zfsloader which when
combined with Lua give us a lot more stack space in those constrained
environments.

Centralize several variables.

MK_CTF, MK_SSP, MK_PROFILE, NO_PIC, and INTERNALLIB are always the
same, so set them in defs.mk. MAN= is common, so set it here too.
This removes a lot of boring repetition from the Makefiles that added
almost no value.

Kill copies of strtol and strtoul. Use the ones that are in libc,
since they suffice. Create xlocale_private.h which provides the most
minimal locale implementation we can get away with. Add strtoll and
strtoull from libc.

Provide abs form stdlib.h.

Sponsored by: Netflix

Implement abort() as a call to panic.

Sponsored by: Netflix

Refactoring of reading and writing of the UFS/FFS superblock.
Specifically reading is done if ffs_sbget() and writing is done
in ffs_sbput(). These functions are exported to libufs via the
sbget() and sbput() functions which then used in the various
filesystem utilities. This work is in preparation for adding
subperblock check hashes.

No functional change intended.

Reviewed by: kib

libsa: Pull in strnlen from libc

strnlen is not used at the moment, but it will be when libfdt gets updated.
Prepare for the not-so-distant future by pulling in strnlen.

Noticed because: segfault in ld.bfd due to strnlen missing

Put the files we're copying over into a few variables and add them to
CLEANDIRS and CLEANFILES so make clean removes any divots.

Sponsored by: Netflix

Add partial support signal.h functioanlity. Pull in machine/signal.h
to define sig_atomic_t.

Sponsored by: Netflix

Now that we offer a semi-sane standards-ish set of #include files in
the stand environment that's safe to use (and insulated from whatever
build env you might normally have), stop hacking the bzlib and zlib
sources with sed. There's no longer any need.

Sponsored by: Netflix

Stop building with the standard system headers.

Building with the standard system headers isn't a perfect match to the
stand environment. Instead, copy over the files we know are safe to
use and constrain what else is used. We use -nostdinc to achieve this.

This also fixes issues with building 32-bit libraries on amd64
sometimes pulling in the wrong cpufunc.h giving an error now that we
stop on errors. It will also enable an easier transition to lua boot.

Sponsored by: Netflix

Fix all warnings related to geli and ZFS support on x86.

Default WARNS to 0 still, since there's still some warnings on other
architectures.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D13301

Minor flags cleanup

Move kernel includes and libsa includes together at the top of defs.mk
Move all machine specific defines from Makefile.inc to their friends
in defs.mk.
Add comments and remove now useless junk after the move.

Sponsored by: Netflix

Move sys/boot to stand. Fix all references to new location

Sponsored by: Netflix