Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

loader: install help files only once

Every file should be installed exactly once by `make installworld`.
This is especially important for pkgbase.

Loader help files were being installed by each loader variant (e.g.,
the simp, lua, and 4th EFI loaders). Add a (slightly hacky) mechanism
to skip installing help files for all but one variant.

PR: 271178
Reviewed by: imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40021

loader: always install help files

Address two issues with current help file logic:

The existing condition prevents the common help file from being
installed when there are no additional help files defined. This results
in no loader.help on EFI platforms, for example.

Second, due to the fact that we build and install multiple loader types,
each successive install will clobber the previous loader.help. The
result is that we could lose type-specific commands, or possibly list
them in loaders that do not have such commands.

Instead, give each loader type a uniquely named help file. The EFI
loader will look for /boot/loader.help.efi, userboot will look for
/boot/loader.help.userboot, etc. The interpreter variant has no effect
on which help file is loaded.

This leaves the old /boot/loader.help unused.

Some credit for the final approach goes to Mathieu <sigsys@gmail.com>
for their version of the fix in https://reviews.freebsd.org/D22951.

PR: 267134
Reported by: Daniel O'Connor <darius@dons.net.au>
Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D28591

stand: Move md_copymodules into modinfo.c and reduce copies

md_copymodules, bi_copymdoules, bi_copymodules32 (x2) and
bi_copymodules64 (x2) are all the same routine... Replace them all with
md_copymodules. This saves about 800 bytes on i386 BIOS loader, which is
a nice bonus.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D36572

stand: Fix a common typo in source code comments

- s/independant/independent/

MFC after: 3 days

loader: narrow the scope of gfx frame buffer wrt tg supported kernels

Store whether or not we found a vbefb module (eg, a tg supported kernel)
in the preloaded_file structure. This automatically resets on reload and
eliminates load_elf knowing about any gfx_* interface. Restrict this to
i386, which is the only place it's used. Update libi386 to check in the
preloaded_file struct. Eliminate this from the teken_gfx
structure. Rewrite the parsing code to be more inline. Check this from
the same place we check for a relocatable amd64 kernel.

Sponsored by: Netflix
Reviewed by: manu, tsoome
Differential Revision: https://reviews.freebsd.org/D33427

stand: remove mips support

As part of decommissioning mips support, remove the boot loader
support. Do this in advance of other boot loader work to limit the
amount of work that will be thrown away.

Sponsored by: Netflix
Reviewed by: jrtc27
Differential Revision: https://reviews.freebsd.org/D33377

loader.mk: Simplify a little bit

The elf loader needs to know how to reach into the gfx_fb code, so
simplify how we include files to find that stuff.

Sponsored by: Netflix
Reviewed by: manu, tsoome
Differential Revision: https://reviews.freebsd.org/D33376

stand/common: Add support for timestamp logging (tslog)

This adds tslog_init, which allocates a 2MB buffer for recording
timestamped events; and tslog_publish, which takes the buffer and
passes it to the kernel as a "preloaded module". These functions
will be used in a later commit.

Reviewed by: kevans

EFI secure boot VECTX related changes

When VECTX is enabled as a kernel option and non-EFI loaders are
built, many reads will fail due to the mis-match of whether
LOADER_VERIEXEC_VECTX or not in readin.h. Source that includes
bootstrap.h must ensure the kernel option agrees with the compile
time CFLAGS in the various make related files.

Submitted by: bret_ketchum@dell.com (original revision)
Reviewed by: sjg, bdrewery, dab, bret_ketchum@dell.com
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D29993

loader: start kernel in text mode when there is no vbefb vt driver

If kernel is built without VT vbefb driver, make sure
we start kernel in text mode.

loader: fix build on non-x86 platforms

Need to build pnglite with all.

Reported by: Herbert J. Skuhra

loader: implement framebuffer console

Draw console on efi.
Add vbe framebuffer for BIOS loader (vbe off, vbe on, vbe list,
vbe set xxx).
autoload font (/boot/fonts) based on resolution and font size.
Add command loadfont (set font by file) and
variable screen.font (set font by size). Pass loaded font to kernel.

Export variables:
screen.height
screen.width
screen.depth

Add gfx primitives to draw the screen and put png image on the screen.
Rework menu draw to iterate list of consoles to enamble device specific
output.

Probably something else I forgot...

Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D27420

loader: zfs should support bootonce an nextboot

bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.

By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.

bootonce dataset name is recorded in boot pool labels, bootenv area.

in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.

bootonce and nextboot features are usable in both UEFI and BIOS boot.

To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).

At this time, only lua loader is updated.

Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25512

RISC-V boot1.efi and loader.efi support

This implementation doesn't have any major deviations from the other EFI
ports. I've copied the boilerplate from arm and arm64.

I've tested this with the following boot flows:
OpenSBI (M-mode) -> u-boot (S-mode) -> loader.efi -> FreeBSD
OpenSBI (M-mode) -> u-boot (S-mode) -> boot1.efi -> loader.efi -> FreeBSD

Due to the way that u-boot handles secondary CPUs, OpenSBI >= v0.7 is required,
as the HSM extension is needed to bring them up explicitly. Because of this,
using BBL as the SBI implementation will not be possible. Additionally, there
are a few recent u-boot changes that are required as well, all of which will be
present in the upcoming v2020.07 release.

Looks good: emaste
Differential Revision: https://reviews.freebsd.org/D25135

loader: create single zfs nextboot implementation

We should have nextboot feature implemented in libsa zfs code.
To get there, I have created zfs_nextboot() implementation based on
two sources, our current simple textual string based approach with added
structured boot label PAD structure from OpenZFS.

Secondly, all nvlist details are moved to separate source file and
restructured a bit. This is done to provide base support to add nvlist
add/update feature in followup updates.

And finally, the zfsboot/gptzfsboot disk access functions are swapped to use
libi386 and libsa.

Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25324

loader: remove libsa/crc32.c and use version from zlib

we have crc32(const void *, size_t) in libsa. Unfortunately zlib has
crc32(long, const unigned char *, unsigned) and we have conflict.

Since we do build libsa with zlib, we can use zlib version instead.

Reviewed by: allanjude
Differential Revision: https://reviews.freebsd.org/D24068

veloader use vectx API for kernel and modules

The vectx API, computes the hash for verifying a file as it is read.
This avoids the overhead of reading files twice - once to verify, then
again to load.

For doing an install via loader, avoiding the need to rewind
large files is critical.

This API is only used for modules, kernel and mdimage as these are the
biggest files read by the loader.
The reduction in boot time depends on how expensive the I/O is
on any given platform. On a fast VM we see 6% improvement.

For install via loader the first file to be verified is likely to be the
kernel, so some of the prep work (finding manifest etc) done by
verify_file() needs to be factored so it can be reused for
vectx_open().

For missing or unrecognized fingerprint entries, we fail
in vectx_open() unless verifying is disabled.

Otherwise fingerprint check happens in vectx_close() and
since this API is only used for files which must be verified
(VE_MUST) we panic if we get an incorrect hash.

Reviewed by: imp,tsoome
MFC after: 1 week
Sponsored by: Juniper Networks
Differential Revision: https://reviews.freebsd.org//D23827

Remove sparc64 support from the boot loader.

Remove all the sparc64 specific bits, both files and ifdefs.

Add flua to the base system, install to /usr/libexec

FreeBSDlua ("flua") is a FreeBSD-private lua, flavored with whatever
extensions we need for base system operations. We currently support a subset
of lfs and lposix that are used in the rewrite of makesyscall.sh into lua,
added in r354786.

flua is intentionally written such that one can install standard lua and
some set of lua modules from ports and achieve the same effect.

linit_flua is a copy of linit.c from contrib/lua with lfs and lposix added
in. This is similar to what we do in stand/. linit.c has been renamed to
make it clear that this has flua-specific bits.

luaconf has been slightly obfuscated to make extensions more difficult. Part
of the problem is that flua is already hard enough to use as a bootstrap
tool because it's not in PATH- attempting to do extension loading would
require a special bootstrap version of flua with paths changed to protect
the innocent.

src.lua.mk has been added to make it easy for in-tree stuff to find flua,
whether it's bootstrap-flua or relying on PATH frobbing by Makefile.inc1.

Reviewed by: brooks, emaste (both earlier version), imp
Differential Revision: https://reviews.freebsd.org/D21893

stand: consolidate knowledge of lua path

Multiple places coordinate to 'know' where lua scripts are installed. Knock
this down to being formally defined (and overridable) in exactly one spot,
defs.mk, and spread the knowledge to loaders and liblua alike. A future
commit will expose this to lua as loader.lua_path, so it can build absolute
paths to lua scripts as needed.

MFC after: 1 week

Remove NAND and NANDFS support

NANDFS has been broken for years. Remove it. The NAND drivers that
remain are for ancient parts that are no longer relevant. They are
polled, have terrible performance and just for ancient arm
hardware. NAND parts have evolved significantly from this early work
and little to none of it would be relevant should someone need to
update to support raw nand. This code has been off by default for
years and has violated the vnode protocol leading to panics since it
was committed.

Numerous posts to arch@ and other locations have found no actual users
for this software.

Relnotes: Yes
No Objection From: arch@
Differential Revision: https://reviews.freebsd.org/D20745

loader: implement map-vdisk and unmap-vdisk commands

illumos update: https://www.illumos.org/issues/10598

Add map-vdisk and unmap-vdisk commands to create virtual disk interface on top of file. This will allow to use disk image from file system to load and start the kernel.

By mapping file, we create vdiskX device, the device will be listed by lsdev [-v] and can be accessed directly as ls vdisk0p1:/path or can be used as value for currdev variable.

vdisk strategy function does not use bcache as we have bcache used with backing file. vdisk can be unmapped when all consumers have closed the open files.

In first iteration we do not support the zfs images because zfs pools do keep the device open (there is no "zpool export" mechanism). Adding zfs support is relatively simple, we just need to run zfs disk probe after mapping is done.

Differential Revision: https://reviews.freebsd.org/D19733

Create kernel module to parse Veriexec manifest based on envs

The current approach of injecting manifest into mac_veriexec is to
verify the integrity of it in userspace (veriexec (8)) and pass its
entries into kernel using a char device (/dev/veriexec).
This requires verifying root partition integrity in loader,
for example by using memory disk and checking its hash.
Otherwise if rootfs is compromised an attacker could inject their own data.

This patch introduces an option to parse manifest in kernel based on envs.
The loader sets manifest path and digest.
EVENTHANDLER is used to launch the module right after the rootfs is mounted.
It has to be done this way, since one might want to verify integrity of the init file.
This means that manifest is required to be present on the root partition.
Note that the envs have to be set right before boot to make sure that no one can spoof them.

Submitted by: Kornel Duleba <mindal@semihalf.com>
Reviewed by: sjg
Obtained from: Semihalf
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D19281

Enable veriexec for loader

This relies on libbearssl and libsecureboot
to verify files read by loader in a maner equivalent
to how mac_veriexec

Note: disabled by default.
Use is initially expected to be by embeded vendors

Reviewed by: emaste, imp
Sponsored by: Juniper Networks
Differential Revision: D16336

Create MK_LOADER_VERBOSE and connect it to ELF_VERBOSE in the loader
code.

PR: 18498
Submitted by: mellon at pobox dot com

Create a loader for each interpreter for x86 BIOS and all EFI

Create loader_{4th,lua,simp}{,.efi}. All of these are installed by
default. Create LOADER_DEFAULT_INTERP to specify the default
interpreter when no other is specified. LOADER_INTERP is the current
interpreter language building. Turn building of lua on by default to
match 4th. simploader is a simplified loader build w/o any interpreter
language (but with a simple loader). This is the historic behavir you
got with WITHOUT_FORTH. Make a hard link to the default loader. This
has to be a hard link rather than the more desirable soft link because
older zfsboot blocks don't support symlinks.

RelNotes: Yes
Differential Revision: https://reviews.freebsd.org/D16705

Collapse zfsloader functionality back down into loader.

We no longer really need a separate zfsloader. It was useful when we
were first supporting ZFS and had limited ability to properly boot off
of ZFS without the special boot loader. Now that the boot loader has
matured, go the way loader.efi pioneered and just build one
binary. Change the name of the loader to load in the secondary boot
blocks to be just /boot/loader. Provide a symbolic link from zfsloader
to loader so people who have not upgraded their boot blocks are not
affected. This has the happy benefit of making coexistence easier as
well (fewer binaries in the matrix).

Discussed with: allanjude@, kevans@
RelNotes: Yes
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D16361

Move ZFS files into libsa

Move the libzfs stuff into libsa. There's no need for it to be a
separate library. The separate library adds to the issues of build
ordering that we see from time to time. Move the filesystem support
into libsa, like all the other filesystem support rather than making
zfs the odd-duck out.

Discussed with: allanjude@

Remove redundant defs.mk includes. They aren't needed.

stand: Fix copy-paste-o, unbreaks libi386 lualoader build

The 32-bit version of liblua is, in fact, NOT libficl. =)

Reported by: emaste

Consolidate three copies of ZFS commands into a central location.

There's no reason to have multiple copies of lszfs and
reloadbe. Consolidate them into one location. Also ldi_get_size is the
same everywhere (except sparc64). Make it the same everywhere as the
common definition is more general and will work on spar64.

Unify metadata load files for arm, mips, powerpc, sparc64

Summary:
All metadata.c files are very similar, with only trivial changes. Unify them
into a single common file, with minor special-casing where needed.

Reviewed By: imp
Differential Revision: https://reviews.freebsd.org/D13978

Add Lua as a scripting langauge to /boot/loader

liblua glues the lua run time into the boot loader. It implements all
the runtime routines that lua expects. In addition, it has a few
standard 'C' headers that nueter various aspects of the LUA build that
are too specific to lua to be in libsa. Many refinements from the
original code to improve implementation and the number of included lua
libraries. Use int64_t for lua_Number. Have "/boot/lua" be the default
module path. Numerous cleanups from the original GSoC project,
including hacking libsa to allow lua to be built with only one change
outside luaconf.h.

Add the final bit of lua glue to bring in liblua and plug into the
multiple interpreter framework, previously committed.

Add LOADER_LUA option, currently off by default.

Presently, this is an experimental option. One must opt-in to using
this by defining WITH_LOADER_LUA and WITHOUT_FORTH. It's been
lightly tested, so keep a backup copy of your old loader handy.
The menu code, coming in the next commit, hasn't been exhaustively
tested. A LUA boot loader is 60k larger than a FORTH one, which is
80k larger than a no-interpreter one. Subtle changes in size
may tip things past some subtle limit (the binary is ~430k now
when built with LUA). A future version may offer coexistance.

Bump FreeBSD version to 1200058 to mark the milestone.

Pedro Souza's 2014 Summer of Code project. Rui Paulo, Pedro Arthur,
Zakary Nafziger and Wojciech A. Koszek also contributed. Warner Losh
reworked it extensively into its current form.

Obtained from: https://wiki.freebsd.org/SummerOfCode2014/LuaLoader
Sponsored by: Google Summer of Code
Relnotes: Yes
MFC After: 1 month
Differential Review: https://reviews.freebsd.org/D14295

Invent new LDR_INTERP for the loader interpreter to use. Use this in
preference to LIBFICL{,32}. LIBFICL{,32} are now always defined, but
LDR_INTERP{,32} is defined empty when building w/o forth (aka the
simple interpreter) and defined to LIBFICL{,32} when we are building
forth.

Break out the interpreters (simple and forth) w/o ifdefs. This is
akin to what Pedro Souza and Wojciech Koszek did in the lua GSoC with
interp.h, interp_simple.c and changes to interp.c and interp_forth.c,
but completely redone from scratch.

This effectively restores the spirit of r326712 (my first attempt to
bring in Pedro's and Wojciech's work) updated for new requirements
that had silently broke their original work. This change also differs
by using fixed function names instead of function pointers to simply
things. Only one interpreter at a time may be compiled in.

Also of note: we take a mutable string, pass it in via a const char *
pointer into intrp_forth's interp_run(). We then cast away the const
to pass into ficlExec since ficl would require extensive changes to
properly const-poison. See Sections 6.5.2.5 and 6.7.3 of C11 standard
noting it's only UB if you modify a const object through a non-const
pointer, but not char [] -> const char * -> char * as here.

Fix ubldr. uboot/lib uses defines for the loader. It's part of the
loader, but not compile as loader (it's building a library), so we
can't just include loader.mk for the defines. Move LOADER_DISK_SUPPORT
back to defs.mk for the moment.

Sponsored by: Netflix

libficl is only ever used in a loader (never a boot) program. Move it
to loader.mk.

Sponsored by: Netflix

Move loader help file definitions to being 100% inside of loader.mk.

HELP_FILES is a loader only thing, so move it to loader.mk. Only
generate the help file if HELP_FILES is defined. Adjust Makefiles to
new convention. Fix a few cases where ${.CURDIR}/ was missing
resulting in missing bits from the help files.

Sponsored by: Netflix

Move loader-only defines to loader.mk from defs.mk

Produces the same .o's, verified with md5.

Sponsored by: Netflix

Revert r326792, r326784, r326772, r326712

Something subtle is creating problems for disk access on ubldr. Back
it out unti that can be sorted out.

Sponsored by: Netflix

Create interp class.

Create an interp class. Use it to separate out the different types of
interpreters: forth and simple with function pointers rather than
via #ifdefs.

Obtained from: lua boot loader project
(via https://bsdimp@github.com/bsdimp/freebsd.git lua-bootloader)
Sponsored by: Netflix

Move sys/boot to stand. Fix all references to new location

Sponsored by: Netflix