MFC r361763: lualoader: drop the filename and word "LUA" from errors

The filename is nearly always wrong since it's /boot/lua/loader.lua, which
is not useful for diagnostics. The actual errmsg will include a lua filename
if this is relevant.

Dropping "LUA" while we're here because that's almost universally
irrelevant to whatever error follows, unless the error states that it's
actually a lua problem.

Both of these are minor nits that just detract from identifying the
pertinent information.

loader: fix userboot's ability to detect a guest's interpreter

Some time after r338418, I believe with -Os/-Oz -ffunction-sections
-fdata-sections, the bootprog_interp variable that held our "$Interpreter:"
marker started getting strip from all loaders, with exception to userboot
since it used bootprog_interp to determine what flavor of userboot it was.

At some point, it had been brought to my attention that this was no longer
working and I had worked up some potential solutions to use the variable
that involved printing it out. My vague recollection is that this was
rejected, and I forgot to explore the alternatives; I cannot find records of
this discussion anymore.

Fast forward to present day, Andrew reported that it was non-functional and
offered (effectively) this patch (sans comment) to stop the compiler from
optimizing it out by assigning it to a volatile variable. This removes
concerns about user-facing change while retaining the interpreter marker.
Furthermore, it could certainly be uglier.

MFC r357828: interp_lua: fix bogus indentation, NFC

MFC r357827: lualoader: disable autoboot on high-level interpreter errors

If we hit an error at this level, we are almost certainly not in any kind
of sane state where autoboot can do the right thing. Instead of letting it
try and probably failing, disable autoboot so they immediately get kicked
into a loader prompt for manual remediation/diagnosis.

MFC r354245, r354833, r354837: add flua to the base system

r354245: stand: consolidate knowledge of lua path

Multiple places coordinate to 'know' where lua scripts are installed. Knock
this down to being formally defined (and overridable) in exactly one spot,
defs.mk, and spread the knowledge to loaders and liblua alike. A future
commit will expose this to lua as loader.lua_path, so it can build absolute
paths to lua scripts as needed.

r354833: Add flua to the base system, install to /usr/libexec

FreeBSDlua ("flua") is a FreeBSD-private lua, flavored with whatever
extensions we need for base system operations. We currently support a subset
of lfs and lposix that are used in the rewrite of makesyscall.sh into lua,
added in r354786.

flua is intentionally written such that one can install standard lua and
some set of lua modules from ports and achieve the same effect.

linit_flua is a copy of linit.c from contrib/lua with lfs and lposix added
in. This is similar to what we do in stand/. linit.c has been renamed to
make it clear that this has flua-specific bits.

luaconf has been slightly obfuscated to make extensions more difficult. Part
of the problem is that flua is already hard enough to use as a bootstrap
tool because it's not in PATH- attempting to do extension loading would
require a special bootstrap version of flua with paths changed to protect
the innocent.

src.lua.mk has been added to make it easy for in-tree stuff to find flua,
whether it's bootstrap-flua or relying on PATH frobbing by Makefile.inc1.

r354837: flua: newer GCC complains about format-nonliteral at WARNS=2

Disable that one, too.

MFC r348569: [zfsboot] Fix boot env back compat (#190)

* Fix boot env back compat

zfsboot must try zfsloader before loader in order to remain compatible
with boot environments created prior to zfs functionality being rolled
into loader proper.

* Improve comments in zfsboot

Explain the significance of the load path order, and put the comment
about looping through the paths in the appropriate scope.

Approved by: re (gjb)

MFC r347142:

loader: validate sectorsize argument in disk_open()

The bug and patch is reported against 11.2, but it is good idea to have
the check in place for all versions.

PR: 236585
Submitted by: john@feith.com
Reported by: john@feith.com

MFC r343911, r344238-r344241, r344247, r344254-r344255, r344260, r344268,
r344335, r344839, r345066, r345330

r343911:
Allow reading the UEFI variable size

When loading bigger variables form UEFI it is necessary to know their
size beforehand, so that an appropriate amount of memory can be
allocated. The easiest way to do this is to try to read the variable
with buffer size equal 0, expecting EFI_BUFFER_TOO_SMALL error to be
returned. Allow such possible approach in efi_getenv routine.

Extracted from a bigger patch as suggested by imp.

r344238:
Restore loader(8)'s ability for lsdev to show partitions within a bsd slice.

I'm pretty sure this used to work at one time, perhaps long ago. It has
been failing recently because if you call disk_open() with dev->d_partition
set to -1 when d_slice refers to a bsd slice, it assumes you want it to
open the first partition within that slice. When you then pass that open
dev instance to ptable_open(), it tries to read the start of the 'a'
partition and decides there is no recognizable partition type there.

This restores the old functionality by resetting d_offset to the start
of the raw slice after disk_open() returns. For good measure, d_partition
is also set back to -1, although that doesn't currently affect anything.

I would have preferred to make disk_open() avoid such rude assumptions and
if you ask for partition -1 you get the raw slice. But the commit history
shows that someone already did that once (r239058), and had to revert it
(r239232), so I didn't even try to go down that road.

r344239:
Use a couple local variables to avoid repetitive long expressions that
cause line-wrapping.

r344240:
Make lsdev -v output line up in neat columns by using a fixed width for
the size field and a tab between the partition type and the size.

Changes this

disk devices:
disk0 (MMC)
disk0s1: DOS/Windows 49MB
disk0s2: FreeBSD 14GB
disk0s2a: FreeBSD UFS 14GB
disk0s2b: Unknown 2048KB
disk0s2d: FreeBSD UFS 2040KB

to this

disk devices:
disk0 (MMC)
disk0s1: DOS/Windows 49MB
disk0s2: FreeBSD 14GB
disk0s2a: FreeBSD UFS 14GB
disk0s2b: Unknown 2048KB
disk0s2d: FreeBSD UFS 2040KB

r344241:
Garbage collection no-longer-used constant.

r344247:
Make uboot_devdesc properly alias disk_devdesc, so that parsing the u-boot
loaderdev variable works correctly.

The uboot_devdesc struct is variously cast back and forth between
uboot_devdesc and disk_devdesc as pointers are handed off through various
opaque interfaces. uboot_devdesc attempted to mimic the layout of
disk_devdesc by having a devdesc struct, followed by a union of some
device-specific stuff that included a struct that contains the same fields
as a disk_devdesc. However, one of those fields inside the struct is 64-bit
which causes the entire union to be 64-bit aligned -- 32 bits of padding
is added between the struct devdesc and the union, so the whole mess ends
up NOT properly mimicking a disk_devdesc after all. (In disk_devdesc there
is also 32 bits of padding, but it shows up immediately before the d_offset
field, rather than before the whole collection of d_* fields.)

This fixes the problem by using an anonymous union to overlay the devdesc
field uboot network devices need with the disk_devdesc that uboot storage
devices need. This is a different solution than the one contributed with
the PR (so if anything goes wrong, the blame goes to me), but 95% of the
credit for this fix goes to Pawel Worach and Manuel Stuhn who analyzed the
problem and proposed a fix.

r344254:
Use DEV_TYP_NONE instead of -1 to indicate no device was specified.

DEV_TYP_NONE has a value of zero, which makes more sense since the device
type is a bunch of bits describing the device, crammed into an int.

r344255:
Fix more places to use DEV_TYP_NONE instead of -1 to indicate 'no device'.

r344260:
Allow the u-boot loaderdev env var to be formatted in the "usual" loader(8)
way: device<unit>[s|p]<slice><partition>. E.g., disk0s2a or disk3p12.
The code first tries to parse the variable in this format using the
standard disk_parsedev(). If that fails, it falls back to parsing the
legacy format that has been supported by ubldr for years.

In addition to 'disk', all the valid uboot device names can also be used:
mmc, sata, usb, ide, scsi. The 'disk' device serves as an alias for all
those types and will match the Nth storage-type device found (where N is
the unit number).

r344268:
loader: ptable_close() should check its argument

If the passed in table is NULL, just return.

r344335:
Fix the handling of legacy-format devices in the u-boot loaderdev variable.
When I added support for the standard loader(8) disk0s2a: type formats,
the parsing of legacy format was broken because it also contains a colon,
but it comes before the slice and partition. That would cause disk_parsedev()
to return success with the slice and partition set to wildcard values.

This change examines the string first, and if it contains spaces, dots, or
a colon at any position other than the end, it must be a legacy-format
string and we don't even try to use disk_parsedev() on it.

r344839:
Add retry loop around GetMemoryMap call to fix fragmentation bug

The call to BS->AllocatePages can cause the memory map to become framented,
causing BS->GetMemoryMap to return EFI_BUFFER_TOO_SMALL more than once. For
example this can happen on the MinnowBoard Turbot, causing the boot to stop
with an error. Avoid this by calling GetMemoryMap in a loop.

r345066:
stand: Improve some debugging experience

Some of these files using <FOO>_DEBUG defined a DEBUG() macro to serve as a
debug-printf. -DDEBUG is useful to enable some debugging output across
multiple ELF/common parts, so switch the DEBUG-as-printf macros over to
something more like DPRINTF that is more commonly used for this kind of
thing and less likely to conflict.

userboot/elf64_freebsd debugging also assumed %llx for uint64; use PRIx64
instead.

r345330:
loader: fix loading of kernels with . in path

The loader indended to search the kernel file name (only) for . but
instead searched the entire path, so paths like
"boot/test.elfv2/kernel" would not work.

PR: 233097

MFC r341101, r341231, r341276, r341329, r341433, r341780, r342054-r342055,
r342721, r342742, r342840, r343008, r343225

r341101:
powerpcspe: Don't crash the loader on ubldr with SPE instructions.

-msoft-float seems to be insufficient for disabling the SPE on powerpcspe.
Force it off with -mno-spe as well. This prevents a crash in ubldr on
powerpcspe.

r341231:
loader: command_bcache() should print unsigned values

All bcache counters are unsigned.

r341276:
When handling CMD_CRIT error set command_errmsg to NULL after we dump it out,
so that it does not result in error message printed twice.

OK load doodoo
can't find 'doodoo'
can't find 'doodoo'
OK

r341329:
loader.efi: fix EFI getchar() for multiple consoles

This fix is ported from illumos (issue #9970), the analysis and initial
implementation was done by John Levon.

See also: https://www.illumos.org/issues/9970

Currently, efi_cons_getchar() will wait for a key. While this seems to make
sense, the implementation of getchar() in common/console.c will loop across
getchar() for all consoles without doing ischar() first.

This means that if we've configured multiple consoles, we can't input into
the serial, as getchar() will be sat waiting for input only from efi_console.c

This patch does implement a bit more generic key buffer to support
translation of input keys, and we use generic efi_readkey() to reduce
duplication from calls from getchar() and poll().

r341433:
Move inclusion of src.opts.mk later.

src.opts.mk includes bsd.own.mk. This in turn defines CTFCONVERT_CMD
depending on the MK_CTF value. We then set MK_CTF to no, which has no
real effect. The solution is to set all the MK_foo values before
including src.opts.mk.

This should stop the cdboot binary from exploding in size for releases
built WITH_CTF=yes in src.conf.

r341780:
powerpc/ubldr: Teach powerpc's ubldr to boot 64-bit kernels

This is just a copy of powerpc/ofw's ppc64_elf_freebsd.c modified to fit
ubldr's boot format.

r342054:
Print an error message in efi_main.c if we can't allocate memory for the heap

With the default Qemu parameters, only 128MB RAM gets given to a VM. This causes
the loader to be unable to allocate the 64MB it needs for the heap. This change
makes the cause of the error more obvious.

r342055:
Cast error message in efi_main.c to CHAR16* to avoid build error

r342721:
loader.efi: update memmap command to recognize new attributes

Also move memory type to string translation to libefi for later use.

r342742:
loader.efi: efi variable rework and lsefi command added

This update does add diag and debug capabilities to interpret the efi
variables, configuration and protocols (lsefi).

The side effect is that we add/update bunch of related headers.

r342840:
Create MK_LOADER_VERBOSE and connect it to ELF_VERBOSE in the loader
code.

r343008:
Add Dell Chromebook to the list of devices with E820 extmem quirk enabled

Just like for Acer C270 chromebook the E820 extmem workaround is required for
FreeBSD to boot on Dell chromebook.

r343225:
Unbreak mip64 build after r328437

Add exit and getchar functions to beri/boot2 code. They are required by
panic_action functin introduced in r328437

PR: 18498, 204916

MFC r338262, r339334, r339796, r340240, r340857, r340917, r341007

r338262:
stand: fdt: Drop some write-only assignments/variables and leaked bits

Generally straightforward enough; a copy of argv[1] was being made in
command_fdt_internal, solely used for a comparison within the
handler-search, then promptly leaked.

r339334:
loader.efi: add poweroff command

Add poweroff command to make life a bit easier.

r339796:
Simplify the EFI delay() function by calling BS->Stall()

r340240:
loader: ptable_open() check for ptable_cd9660read result is wrong

The ptable_*read() functions return NULL on read errors (and partition table
closed as an side effect). The ptable_open must check the return value and
act properly.

r340857:
Nuke out buffer overflow safety marker code, it duplicates similar code in
the malloc()/free() as well as having potential of softening the handling
in case error is detected down to a mere warning as compared to hard panic
in free().

r340917:
Update pxeboot(8) manual page to reflect the next-server change in the ISC DHCP v3 server.

r341007:
Bump the date of pxeboot(8) manual page for r340917.

PR: 123484, 232483

MFC r341253, r341328, r342619, r342626, r342707, r342785, r342865

r341253:
The libstand's panic() appends its own '\n' to the message, so that users of the API
don't need to supply one.

r341328:
loader: create separate lists for fd, cd and hd, merge bioscd with biosdisk

Create unified block IO implementation in BIOS version, like it is done in UEFI
side. Implement fd, disk and cd device lists, this will split floppy devices
from disks and will allow us to have consistent, predictable device naming
(modulo BIOS issues).

r342619:
loader: create bio_alloc and bio_free for bios bounce buffer

We do have 16KB buffer space defined in pxe.c, move it to bio.c and implement
bio_alloc()/bio_free() interface to make it possible to use this space for
other BIOS calls (notably, from biosdisk.c).

r342626:
Add Copyright.

r342707:
i386_parsedev() needs to support fd devices

r342785:
With buggy int13 ah=15, we can mis-identify the floppy devices.

We have no option than trust INT13 ah=08 return code during the init phase.

r342865:
biospci_write_config args were backwards

biospci_write_config args swapped length and value to write. Some
hardware coped just fine, while other hardware had issues.

PR: 155441, 234460

MFC r339658, r339959, r340047, r340049, r340215

r339658:
loader: biosdisk interface should be able to cope with 4k sectors

The 4kn support in current bios specific biosdisk.c is broken, as the code
is only implementing the support for the 512B sector size.

This work is building the support for custom size sectors, we still do assume
the requested data to be multiple of 512B blocks and we only do address the
biosdisk.c interface here.

For reference, see also:
https://www.illumos.org/issues/8303
https://www.illumos.org/rb/r/547

As the GELI is moved above biosdisk "layer", the GELI should just work

r339959:
loader: issue edd probe before legacy ah=08 and detect no media

while probing for drives, use int13 extended info before standard one and
provide workaround for case we are not getting needed information in case
of floppy drive.

In case of INT13 errors, there are (at least) 3 error codes appearing in case
of missin media - 20h, 31h and 80h. Flag the no media and do not print an
error.

r340047:
loader: do not probe floppy devices for zfs

The subject is telling it all.

r340049:
loader: biosdisk should check if the media is present

The bd_print/bd_open/bd_strategy need to make sure the device does have
media, before getting into performing IO operations. Some systems can
hung if the device without a media is accessed.

r340215:
loader: always set media size from partition.

The disk access is validated by using partition table definitions, therefore
we have no need for if statements, just set the disk size.

Of course the partition table itself may be incorrect/inconsistent, but if
so, we are in trouble anyhow.

MFC r344161: stand: dev_net: correct net_open's interpretation of params

net_open previously casted the first vararg to a char * and this was
half-OK: at first, it is passed to netif_open, which would cast it back to
the struct devdesc * that it really is and use it properly. It is then
strdup()d and used as the netdev_name, which is objectively wrong.

Correct it so that the first vararg is properly casted to a struct devdesc *
and the netdev_name gets set properly to make it more clear at a glance that
it's not doing something horribly wrong.

MFC Loader Interpreter Coexistance: r337806, r337868, r337914, r337927,
r337989, r338052, r338064, r338407, r338418, r338474

r337806:
Create a loader for each interpreter for x86 BIOS and all EFI

Create loader_{4th,lua,simp}{,.efi}. All of these are installed by
default. Create LOADER_DEFAULT_INTERP to specify the default
interpreter when no other is specified. LOADER_INTERP is the current
interpreter language building. Turn building of lua on by default to
match 4th. simploader is a simplified loader build w/o any interpreter
language (but with a simple loader). This is the historic behavir you
got with WITHOUT_FORTH. Make a hard link to the default loader. This
has to be a hard link rather than the more desirable soft link because
older zfsboot blocks don't support symlinks.

r337868:
stand: Use -Oz/-Os for all loader/stand builds.

While we're not super size constrained, the x86 BIOS /boot/loader has
to be less than about 520k-530k to be reliable. The LUA loader is at
this size today. -Oz saves 15-20% on the size, keeping us safely small
enough (comparable to where we were with the 4th loader). This will
also help with sjg's work on bringing in bearssl, though we may again
be looking for space in the LUA loader.

Size table for clang 6.0.0:
default -O1 -Os -Oz
4th 442368 417792 389120 376832
lua 524288 479232 446464 430080

r337914:
Install links for loader.efi.

r337927:
Add ashldi3 and ashrdi3 to mips.

Now that we're using -Os, mips needs these routines.

r337989:
Copy the boot loader from the new location for the co-existing
loaders.

r338052:
libsa: Add lshrdi3.c for powerpc* and mips

r338064:
Turn back the clock just a little: make userboot.so always be 4th

Turns out there was a hidden dependency we hasn't counted upon. The
host load /boot/userboot.so to boot the VMs it runs. This means that
the change to lua meant suddently that nobody could run their older
VMs because LUA wasn't in 10.0, last month's HardenedBSD, 11.2 or
whatever. Even more than for the /boot/loader* binaries, we need a
good coexistance strategy for this. While that's being designed and
implemented, drop back to always 4th for userboot.so. This will fail
safe in all but the most extreme environments (but lua-only hacks
to .lua files won't be processes in VMs until we fix it).

r338407:
lualoader: Print error messages from command failures at the prompt

Previously lualoader would remain silent, rather than printing
command_errmsg or noting that a command had failed or was not found.

r338418:
userboot: handle guest interpreter mismatches more intelligently

The switch to lualoader creates a problem with userboot: the host is
inclined to build userboot with Lua, but the host userboot's interpreter
must match what's available on the guest. For almost all FreeBSD guests in
the wild, Lua is not yet available and a Lua-based userboot will fail.

This revision updates userboot protocol to version 5, which adds a
swap_interpreter callback to request a different interpreter, and tries to
determine the proper interpreter to be used based on how the guest
/boot/loader is compiled. This is still a bit of a guess, but it's likely
the best possible guess we can make in order to get it right. The
interpreter is now embedded in the resulting executable, so we can open
/boot/loader on the guest and hunt that down to derive the interpreter it
was built with.

Using -l with bhyveload will not allow an intepreter swap, even if the
loader specified happens to be a userboot with the wrong interpreter. We'll
simply complain about the mismatch and bail out.

For legacy guests without the interpreter marker, we assume they're 4th.
For new guests with the interpreter marker, we'll read it and swap over
to the proper interpreter if it doesn't match what the userboot we're using
was compiled with.

Both flavors of userboot are installed by default, userboot_4th.so and
userboot_lua.so. This fixes the build WITHOUT_FORTH as a coincidence, which
was broken by userboot being forced to 4th.

r338474:
Be a little conservative about when to force size optimizations.

Reports have come in that there's issue with powerpc and sparc64 since
we've switched to using -Oz / -Os. We don't strictly need them for
!x86, so be conservative about when we enable them.

MFC GELI Loader Improvements: r336252, r336254, r336256, r336354,
r336532-r336534, r336537, r336626, r337326, r337349, r341071, r341160,
r341420, r341473, r341651, r342793

Note that this MFC contains some seemingly unrelated zfsloader bits -- this
was needed in order to pull in some later fixes for GELI hand-off w/ ZFS
bits included.

r336252:
Extend loader(8) geli support to all architectures and all disk-like devices.

This moves the bulk of the geli support from lib386/biosdisk.c into a new
geli/gelidev.c which implements a devsw-type device whose dv_strategy()
function handles geli decryption. Support for all arches comes from moving
the taste-and-attach code to the devopen() function in libsa.

After opening any DEVT_DISK device, devopen() calls the new function
geli_probe_and_attach(), which will "attach" the geli code to the open_file
struct by creating a geli_devdesc instance to replace the disk_devdesc
instance in the open_file. That routes all IO for the device through the
geli code.

A new public geli_add_key() function is added, to allow arch/vendor-specific
code to add keys obtained from custom hardware or other sources.

With these changes, geli support will be compiled into all variations of
loader(8) on all arches because the default is WITH_LOADER_GELI.

r336254:
Use if rather than case for a simple boolean. gcc thinks blks is
undefined sometimes with the case, but enc is always 0 or 1, so
and if / else is better anyway.

r336256:
Fix glitched indentation (and rewrap as needed due to deeper indent).
No functional changes.

r336354:
zfsboot: fix build with WITHOUT_LOADER_GELI

r336532:
Collapse zfsloader functionality back down into loader.

We no longer really need a separate zfsloader. It was useful when we
were first supporting ZFS and had limited ability to properly boot off
of ZFS without the special boot loader. Now that the boot loader has
matured, go the way loader.efi pioneered and just build one
binary. Change the name of the loader to load in the secondary boot
blocks to be just /boot/loader. Provide a symbolic link from zfsloader
to loader so people who have not upgraded their boot blocks are not
affected. This has the happy benefit of making coexistence easier as
well (fewer binaries in the matrix).

r336533:
Eliminate zfsloader man page.

Remove all cross references to zfsloader.8 and /boot/zfsloader.
Move ZFS specific info into loader.8.

r336534:
NM and OBJCOPY are already defined for all builds. There's no need to
conditionally define them here.

r336537:
Mention zfsloader being folded into loader in UPDATING.

r336626:
Older zfs boot blocks don't support symlinks. install the link to
zfsloader as a hard link. While newer ones do, the whole point of the
link was to transition to the new world order smoothly. A hard link is
less flexible, but it works and will result in fewer bumps. Adjust
UPDATING entry to match.

r337326:
loader: biosdisk.c has leftover geli header.

A small cleanup, remove unneeded #include.

r337349:
zfsboot: Fix startup crash

On a FreeNAS mini XL, with geli encrypted drives the loader crashed in
geli_read().

When we iterate over the list of disks and allocate the zfsdsk structures we
don’t zero out the gdev pointer. In one case that resulted in geli_read()
(called on the bogus pointer) dividing by zero.

Use calloc() to ensure the zfsdsk structure is always zeroed, so the pointer is
initialised to NULL. As a side benefit it gets rid of one #ifdef
LOADER_GELI_SUPPORT.

r341071:
Restore the ability to override the disk unit/partition at the boot: prompt
in gptboot.

When arch-independent geli support was added, a new static 'gdsk' struct
was added, but there was still a static 'dsk' struct, and when you typed
in an alternate disk/partition, the string was parsed into that struct,
which was then never used for anything. Now the string gets parsed into
gdsk.dsk, the struct that's actually used.

r341160:
Add comments describing the bootargs handoff between loader(8) and gptboot
or zfsboot, when loader(8) is the BTX loader. No functional changes.

r341420:
Eliminate duplicated code and struct member definitions in the handoff
of args data between gptboot/zfsboot and loader(8).

Despite what seems like a lot of changes here, there are no actual
changes in behavior, or in the data layout in the structures involved.
This is just eliminating identical code pasted into multiple locations.

In detail, the changes are...

- Move struct zfs_boot_args definition from libsa/zfs/libzfs.h to
i386/common/bootargs.h because it is specific to x86 booting and the
handoff between zfsboot and loader, and has no relation to the zfs
library code in general.

- The geli_boot_args and zfs_boot_args structs both contain an identical
set of member variables containing geli information. Extract this out
to a new geli_boot_data struct, and embed it in the arg-passing structs.

- Provide new routines geli_import_boot_data() and geli_export_boot_data()
that can be shared between gptboot, zfsboot, and loader instead of
pasting identical code into several different .c files.

- Remove some checks for a NULL pointer that can never be true because the
pointer being tested was set using pointer math (kargs + 1) and that can
never result in NULL in this code.

r341473:
Fix args cross-threading between gptboot(8) and loader(8) with zfs support.

When loader(8) is built with zfs support enabled, it assumes that any extarg
data present is a zfs_boot_args struct, but if the first-stage loader was
gptboot(8) the extarg data is actually a geli_boot_args struct. Luckily,
zfsboot(8) and gptzfsboot(8) have always passed KARGS_FLAGS_ZFS along with
KARGS_FLAGS_EXTARG, so we can use KARGS_FLAGS_ZFS to decide whether the
extarg data is a zfs_boot_args struct.

To avoid similar problems in the future, gptboot(8) now passes a new
KARGS_FLAGS_GELI to indicate that extarg data is geli_boot_args. In
loader(8), if the neither KARGS_FLAGS_ZFS nor KARGS_FLAGS_GELI is set but
extarg data is present (which will be the case for gptboot compiled before
this change), we now check for the known size of the geli_boot_args struct
passed by the older versions of gptboot as a way of confirming what type of
extarg data is present.

In a semi-related tidying up, since loader's main() has already decided
what type of extarg data is present and set the global 'zargs' var
accordingly, don't repeat the check in extract_currdev, just check whether
zargs is NULL or not.

r341651:
Don't reference zfs-specific variables if LOADER_ZFS_SUPPORT is undefined
because the variables will be undefined too.

r342793:
MK_ZFS -> {MK_ZFS|MK_LOADER_ZFS}, this is so we can diable userland / kernel
ZFS but keep the boot-loaders when using ZoL port.

Relnotes: yes (GELI support extended)
Relnotes: yes (zfsloader has been collapsed into loader and may be
removed after boot blocks have been updated)

MFC r336244, r336246-r336247: Standardize boot arg parsing

r336244:
Create helper functions for parsing boot args.

boot_parse_arg to parse a single arg
boot_parse_cmdline to parse a command line string
boot_parse_args to parse all the args in a vector
boot_howto_to_env Convert howto bits to env vars
boot_env_to_howto Return howto mask mased on what's set in the environment.

All these routines return an int that's the bitmask of the args
translated to RB_* flags. As a special case, the 'S' flag sets the
comconsole_speed env var. Any arg that looks like a=b will set the env
key 'a' to value 'b'. If =b is omitted, 'a' is set to '1'. This
should help us reduce the number of redundant copies of these routines
in the tree. It should also give a more uniform experience between
platforms.

Also, invent a new flag RB_PROBE that's set when 'P' is parsed. On
x86 + BIOS, this means 'probe for the keyboard, and if it's not there
set both RB_MULTIPLE and RB_SERIAL (which means show the output on
both video and serial consoles, but make serial primary). Others it
may be some similar concept of probing, but it's loader dependent
what, exactly, it means.

These routines are suitable for /boot/loader and/or the kernel,
though they may not be suitable for the tightly hand-rolled-for-space
environments like boot2.

r336246:
Eliminate boot loader copies of boot arg parsing.

Eliminate 4 of the copies of the arg parsing in /boot/laoder
by using boot_parse_cmdline.

r336247:
Transition to boot_env_to_howto and boot_howto_to_env in the boot
loader.

MFC r336084-r336085, r336118: Move zfs impl into libsa

r336084:
Move ZFS files into libsa

Move the libzfs stuff into libsa. There's no need for it to be a
separate library. The separate library adds to the issues of build
ordering that we see from time to time. Move the filesystem support
into libsa, like all the other filesystem support rather than making
zfs the odd-duck out.

Discussed with: allanjude@

r336085:
Stop using ../zfs/libzfs.h but instead use libzfs.h.

While ../zfs/libzfs.h mostly works, there are a few situations where
it does not. Eliminate the problem by using plain libzfs.h, like we do
for ufs support. This fixes the weird cases, and is easier to
understand. It also follows the general style convetion of avoiding
../ in #includes.

r336118:
Remove dirs that git svn should have removed but didn't.

MFC r335245-r335248, r335254, r335276, r335298, r335398, r335868, r335883

r335245:
Correct logic error in biosdisk.c:bd_realstrategy()

The wrong condition is used when evaluating the return of disk_ioctl()
This results in reaching the 'We should not get here' branch in most casts

r335246:
biosdisk.c remove redundant variable

`rdev` and `disk` serve the same purpose, read the partition table without
the `d_offset` or `d_slice` set, so the read is relative to the start of
the disk. Reuse the already initialized `disk` instead of making another
copy later.

r335247:
biosdisk.c: add missing \n to error message

r335248:
biosdisk.c: fix type in debug printf

r335254:
Avoid reading past the end of the disk in zfsboot.c and biosdisk.c

The GELI boot code rounds reads up to 4k, since the encrypted sectors are
4k, and must be decrypted as a unit. With oddball sized disks (almost
always virtual), this can lead to reading past the end of the disk.

r335276:
gptboot, zfsboot, gptzfsboot: Enable the video and serial consoles early

Normally the serial console is not enabled until /boot.config is read and
we know how the serial console should be configured. Initialize the
consoles early in 'dual' mode (serial & keyboard) with a default serial
rate of 115200. Then serial is re-initialized once the disk is decrypted
and the /boot.config file can be read.

This allows the GELIBoot passphrase to be provided via the serial console.

r335298:
stand/common/disk.c: Update debug printf

This was missed in r330809 because it is compiled out by default

r335398:
Revert r335276

This was causing issues for people booting.
I will likely bring this back as an optional feature, similar to
boot0sio, like gptboot-serial or something.

r335868:
stand/common/disk.c: Read partition table relative to the start of the disk

If a disk is of an oddball size, like the 200mb + 512b used in rootgen.sh,
when disk_open() is called on a GELI encrypted partition, attempts to read
the partition table fail, as they pass through the decryption process which
turns the already plaintext data into jibberish.

When reading the partition table, always pass a slice and partition setting
of -1, and an offset of 0. Setting the slice to -1 prevents a false
positive when checking the slice against the cache of GELI encrypted
slices.

r335883:
stand/common/disk.c: dev->d_offset still needs to be set to 0

With r335868, I thought this was no longer necessary. I was wrong.

MFC r335117: stand bootprog_info consolidation

bootprog_info is generated in vers.c. Move it's definition to
bootstrap.h and remove all the redundant copies.

MFC r332557-r332558, r332560, r332565: loader command cleanup-lite

r332557:
loader: make sure we use snprintf() in commands.c

Safeguard against memory corruptions.

r332558:
loader: command_errmsg should be const

Use const char * for command_errmsg.

r332560:
loader: make sure we do not return garbage from help_getnext

Since we do free subtopic and desc in help_getnext(), we need to set them also
NULL, so we make sure we dont get double free().

r332565:
loader: cstyle cleanup of command.c

just clean it up. no functional changes intended.

MFC r332555: loader: provide values in help_getnext()

With r328289 we attempt to make sure we free the resources allocated in
help_getnext(), however, it is possible that we get no resources allocated
and help_getnext() will return early.

Make sure we have pointers set to NULL early in help_getnext().

MFC r330757: stand/interp_lua: correct errorr => error

MFC lualoader: r326353, r328440, r328443, r329166-r329167, r329274, r329329,
r329349-r329352, r329355-r329359, r329366-r329369, r329386-r329387, r329393,
r329413-r329415, r329417, r329424-r329436, r329457, r329473-r329474,
r329496-r329501, r329503-r329504, r329543, r329547-r329551, r329576-r329578,
r329580, r329583, r329585-r329590, r329592-r329596, r329609-r329611,
r329614, r329619, r329621-r329622, r329624, r329626-r329627,
r329629-r329632, r329640-r329641, r329643-r329646, r329649-r329650, r329654,
r329656, r329662, r329669-r329671, r329673-r329674, r329680,
r329684-r329689, r329692-r329693, r329696-r329700, r329709, r329716,
r329731, r329733-r329734, r329747-r329748, r329756, r329779, r329782,
r329784, r329786, r329804, r329806, r329809, r329811, r329836,
r329850-r329852, r329854, r329856-r329858, r329861, r329895,
r329897-r329899, r329901-r329903, r329922-r329924, r329927-r329928,
r329944-r329949, r329986-r329987, r330008-r330010, r330012, r330020,
r330082-r330084, r330087-r330088, r330098-r330101, r330138-r330139,
r330261-r330263, r330267-r330269, r330281-r330284, r330287, r330339-r330342,
r330345-r330346, r330369-r330370, r330434-r330435, r330564, r330616-r330618,
r330620, r330625, r330690, r330701, r330703, r330825, r331211, r331257,
r331259, r331281-r331282, r331304, r331314, r331476-r331477,
r331563-r331564, r331854-r331857, r331859-r331860, r332106, r334723,
r334879, r334891, r334907, r334912, r334939, r334986, r335009, r335371,
r336759, r337711, r337807-r337810, r338054, r338063, r338065-r338067,
r338083, r338085-r338086, r338108, r338167-r338168, r338173, r338203,
r338255, r338259, r338309, r338394, r338438, r338886, r338893, r339173,
r339200, r339218, r339222, r339301, r339307, r339677-r339678, r339702,
r339805, r339831, r339849, r340040, r340152

Some notes for this MFC:
- This is still pre-forth/lua coexistance; that will come shortly-ish so
that forth/lua may be installed together (with forth remaining the default)

- module_blacklist support for lualoader has been MFC'd, but the drm modules
are not blacklisted in this stable branch.

r326353: Import lua 5.3.4 to contrib
r328440: Preserve the original luaconf.h in a convenient place. Clients will
r328443: Gross hack to omit printing hex floating point when the lua number
r329166: Add Lua as a scripting langauge to /boot/loader
r329167: Add the lua scripts from the lua-bootloader SoC
r329274: stand/lua: Exit sub-menus on backspace
r329329: stand/lua: Always boot on 'enter' keypress in menus
r329349: stand/lua: Reduce magic numbers
r329350: stand/lua: Don't descend into an empty kernels submenu
r329351: stand/lua: Set reasonable ACPI default based on presence
r329352: stand/lua: Consistently use semicolons for line endings
r329355: stand/lua: Move kernel selection into main menu
r329356: stand/lua: Allow menu items to not have explicit aliases
r329357: stand/lua: Remove explicit alias from "Back to main menu"
r329358: stand/lua: Say "loader prompt" instead of "lua interpreter"
r329359: stand/lua: Remove a magic number/string (not a trivial literal)
r329366: stand/lua: Set ACPI's default the proper way (setACPI)
r329367: stand/lua: Create a "carousel" menu entry type
r329368: stand/lua: Create/use some MENU_ constants where applicable
r329369: stand/lua: Remove sneaky kernel assignment
r329386: stand/lua: Correct usage and acceptance of BACKSPACE/DELETE keys
r329387: stand/lua: Chop off the decimal for numbers passed to setcursor
r329393: stand/lua: Use escaped dot instead of single character class
r329413: stand/lua: Allow MENU_RETURN items to have func, fix esc. to prompt
r329414: stand/lua: Don't try to divide by 0; do nothing
r329415: stand/lua: Don't reload kernel config if we only have one kernel
r329417: stand/lua: Make CAROUSEL_ENTRY func parameters consistent with name
r329424: stand/lua: Don't set autoboot_delay=NO in menu autoboot sequence
r329425: stand/lua: Enable menu autoboot; it seems to work
r329426: stand/lua: Correct interpretation of autoboot_delay
r329427: stand/lua: Color non-default kernels blue
r329428: stand/lua: Correct some trivial errors in config
r329429: stand/lua: Add debug method to dump modules
r329430: stand/lua: Address some nits
r329431: stand/lua: Correct test sense, this should have been 'not nil'
r329432: stand/lua: Try to load alternate kernels as directories first
r329433: stand/lua: Add optional GELI passphrase prompt
r329434: stand/lua: Check for nil (GELI prompt)
r329435: stand/lua: Style pass
r329436: stand/lua: Debugging string snuck in...
r329457: stand/lua: dumpModules => lsModules
r329473: liblua: Clean up io/loader C module registration
r329474: liblua: Emulate DIR, opendir, fdopendir, closedir
r329496: stand/lua: Fix verbiage and some typos
r329497: stand/lua: Fix module_path handling with multiple kernels
r329498: stand/lua: Remove some debugging bits that snuck in... gr...
r329499: interp_lua: Register io/loader with regular Lua module system
r329500: Lua loader: Add barebones "lfs" module
r329501: lua loader: Auto detect eligible list of kernels to boot
r329503: liblua: Fix missing '}' in lutil.c after r329499
r329504: stand/lua: More style nits, config.lua
r329543: Create style.lua(9)
r329547: stand/lua: Allow menu items to be conditionally (in)visible
r329548: stand/lua: Addres style.lua(9) concern
r329549: stand/lua: Clear the screen before prompting for passwords
r329550: stand/lua: Store the loaded kernel as config.kernel_loaded
r329551: stand/lua: reload previously loaded kernel at config-load/reload
r329576: stand/lua: Defer kernel/module loading until boot or menu escape
r329577: stand/lua: Rename bootserial for clarity
r329578: stand/lua: Menu style.lua(9) nits
r329580: stand/lua: Remove some unused local declarations
r329583: stand/lua: Store menu entries in an "entries" table
r329585: stand/lua: Add core.isSingleUserBoot
r329586: stand/lua: Call menu_entries if it's a function
r329587: stand/lua: Swap single-/multi- user boot entries as needed
r329588: stand/lua: Re-wrap menu.lua now that I've added indentation...
r329589: stand/lua: Track env changes that come in via loader.conf(5)
r329590: stand/lua: Restore environment upon config reload
r329592: stand/lua: Remove inaccurate comment after r329590
r329593: stand/lua: Change boot menu items' names when swapped
r329594: stand/lua: Round up some more style.lua(9) concerns
r329595: stand/lua: Re-order locals after copyright notice; require first
r329596: stand/lua: Add copyright notice in places
r329609: stand/lua: Cache swapped menu, and don't create locals for swapping
r329610: style.lua(9): Note that wrapping at 80-columns is not rigid
r329611: stand/lua: Wrap tuple assignment earlier for readability
r329614: stand/lua: Don't set ACPI off just because we can't detect it.
r329619: stand/lua: Extract menu handlers out into menu.handlers table
r329621: stand/lua: Add and use drawer.menu_name_handlers
r329622: stand/lua: Move drawer.menu_name_handlers further up
r329624: stand/lua: Reduce exposure of the drawer module
r329626: stand/lua: Refactor logos into drawer.logodefs table
r329627: stand/lua: Refactor brands into drawer.branddefs
r329629: stand/lua: Use 'graphic' instead of 'logo' for depicting graphics
r329630: stand/lua: Stick a copyright notice on drawer.lua
r329631: stand/lua: Insert helpful comment for drawer.branddefs
r329632: style.lua(9): Clarify local variable guideline
r329640: stand/lua: Consistently declare local functions at module scope
r329641: stand/lua: Consistently organize modules
r329643: Implement loader.command
r329644: lualoader: Add ability to intercept cli commands
r329645: lualoader: Move carousel storage out into config
r329646: lualoader: Eliminate global namespace pollution in loader.lua
r329649: Lua lfs.attributes: Provide a more consistent error return
r329650: liblua: Add loader.machine and loader.machine_arch properties
r329654: lualoader: Ignore ACPI bits on !i386
r329656: loader.lua: Expose errno table to lua
r329662: lualoader: Replace invalid construct with valid construct
r329669: lualoader: Prefer selected kernel to currently loaded
r329670: lualoader: Don't try so hard to load a kernel
r329671: lualoader: Prepare for interception of "boot" CLI cmd
r329673: lualoader: Intercept boot cli command
r329674: lualoader: Intercept the 'autoboot' cli command
r329680: lualoader: When restoring environment, only restore unchanged vars
r329684: lualoader: Drop terminating semicolons
r329685: lualoader: Drop excessive parenthesizing
r329686: style.lua(9): Drop notes about semicolons
r329687: lualoader: Drop explicit boolean tests; b or not b
r329688: lualoader: Don't return false for failure to open config on silent
r329689: lualoader: Output "Failed to parse" messages
r329692: lualoader: Bring in local.lua module if it exists
r329693: lualoader: Return only argstr if with_kernel not requested
r329696: lualoader: Add "menu.default", initialized to menu.welcome
r329697: lualoader: Drop name requirement for menu separators
r329698: lualoader: Directly reference submenu definition with submenu key
r329699: lualoader: Simplify menu definitions a little further
r329700: lualoader: Allow carousel 'items' to be a table as well
r329709: lualoader: Don't autodetect kernels if 'kernels' is explicitly set
r329716: lualoader: Use the key that interrupts autoboot as a menu choice
r329731: lualoader: Add boot environment support
r329733: lualoader: Make kernel autodetect. contingent on loader.conf(5) var
r329734: lualoader: Don't execute menu.autoboot() for every opened menu
r329747: lualoader: Replace 8-space indentation with a single tab
r329748: lualoader: Drop password length restrictions
r329756: lualoader: Remove nasty hack for not printing out ".0"
r329779: lualoader: Split cli bits out into a cli module
r329782: lualoader: Unbreak 'boot [kernel]' by including config
r329784: lualoader: Pull argument extraction for cli funcs to cli.arguments
r329786: lualoader: Attach cli command functions to cli module
r329804: lualoader: Eliminate some unused locals
r329806: lualoader: Consistently use double quotes
r329809: lualoader: Address some 'luacheck' concerns
r329811: lualoader: Clear up an empty conditional branch
r329836: lualoader: Attend to some 80-col issues, pointed out by luacheck
r329850: lualoader: Drop unused return values; we'll only use the first
r329851: Add SPDX tags to lua files
r329852: Add copyright notice to core.lua
r329854: lualoader: shallowCopyTable => deepCopyTable
r329856: lualoader: Use "local function x()" instead "local x = function()"
r329857: Centralize lua defines
r329858: When the LUA_FLOAT_TYPE != LUA_FLOAT_INT64, we can't ref. float
r329861: lualoader: Track effective line number, use it for drawing
r329895: liblua: Implement write support
r329897: lualoader: Add nextboot support
r329898: lualoader: Plug file handle not properly closed
r329899: lualoader: Correct test and name
r329901: lualoader: Add comment on trailing space, don't operate on nil
r329902: lualoader: Remove unused variable; we now use effective line number
r329903: lualoader: Explain nextboot stuff a little bit more
r329922: lualoader: Split config file I/O out into a separate function
r329923: lualoader: Strip config.parse of its I/O privileges
r329924: lualoader: throw out nextboot's usage of standard config processing
r329927: lualoader: Clean up naming conventions a little bit
r329928: lualoader: Remove inaccurate part of comment
r329944: lualoader: Don't explicitly index tables without reason
r329945: lualoader: menu: Terminate final values in tables with a comma
r329946: lualoader: Clean up menu handling a little bit
r329947: lualoader: Pull menu redrawing specifics out of menu.process
r329948: lualoader: Pull autoboot handling out into menu.run()
r329949: lualoader: Explain deviation from naming guidelines
r329986: lualoader: Invalidate the screen from menu perspective mnu exit
r329987: lualoader: Track the menu currently drawn, instead of validity
r330008: lualoader: A little more general menu cleanup
r330009: lualoader: More argument name expansion, part 2
r330010: lualoader: screen argument fixes
r330012: style.lua(9): Add some additional notes about naming and commas
r330020: lualoader: Re-work menu skipping bits
r330082: lualoader: Add a twiddle at password prompt
r330083: lualoader: Remove remnants of testing...
r330084: lualoader: Replace instances of \027 with KEYSTR_ESCAPE
r330087: lualoader: Convert instances of KEYSTR_ESCAPE .. "[" -> KEYSTR_CSI
r330088: lualoader: Correct test sense, comments, and add some more comments
r330098: lualoader: Re-do twiddle
r330099: lualoader: Further screen cleanup
r330100: lualoader: Remove debug function
r330101: lualoader: Add note that \027 is a decimal representation
r330138: lualoader: Dedup these "Return to main menu" entries
r330139: lualoader: config: Pull some messages out into constants
r330261: lualoader: Fix some lint-mentioned errors
r330262: lualoader: Use #str instead of tracking length with 'n'
r330263: lualoader: Use string literal \xNN instead of string.char()
r330267: Add core.lua(8), but do not add to distribution
r330268: Add menu.lua(8), but do not add to distribution
r330269: core.lua(8): Add missing note about core.KEYSTR_CSI
r330281: lualoader: Steamroll the box-drawing
r330282: lualoader: Register loader.printc as global printc
r330283: lualoader: Use global printc instead of loader.printc
r330284: liblua: Use putc instead of printf for printc
r330287: lualoader: Reset the cursor position after the menu is drawn
r330339: liblua: Add loader.interpret
r330340: lualoader: Execute menu_timeout_command at the end of menu autoboot
r330341: lualoader: Respect loader_menu_title, prepare for align
r330342: lualoader: Respect loader_menu_title_align
r330345: lualoader: Tweak positioning and fix an off-by-one
r330346: lualoader: Shift menu+brand even for logo=none with customized pos
r330369: lualoader: Return meaningful value in cli_execute
r330370: lualoader: logdef -> logodef typo
r330434: lualoader: Add note about importance of including cli module early
r330435: lualoader: Use FILESDIR instead of BINDIR
r330564: lualoader: Only loadelf before boot/autoboot if no kernel loaded
r330616: lualoader: Expose loader.parse and add cli_execute_unparsed
r330617: lualoader: Fix name, cli.execute_unparsed -> cli_execute_unparsed
r330618: lualoader: Use cli_execute_unparsed instead of loader.interpret
r330620: lualoader: Use cli_execute_unparsed for commands via loader.conf
r330625: lualoader: Return status in cli_execute_unparsed properly
r330690: stand: Fix copy-paste-o, unbreaks libi386 lualoader build
r330701: lualoader: Don't redraw the autoboot message every .05s
r330703: lualoader: Cache kernel list
r330825: lualoader: Sprinkle some verbose_loading salt
r331211: lualoader: Setup default color scheme if we're using colors
r331257: lualoader: Reset attributes and color scheme with color.highlight()
r331259: lualoader: Use less atomic options for resetting colors/attributes
r331281: lualoader: Add primitive hook module to untangle bogus reference
r331282: core.lua(8): Update to reflect recently added function
r331304: lualoader: Clear up some possible naming confusion
r331314: lualoader: Use printc when we expect ANSI escape sequences
r331476: lualoader: Make config env-related bits private API
r331477: lualoader: Privatize some more config.lua bits
r331563: lualoader: Implement try_include and use it for including local
r331564: lualoader: Actually re-raise error in try_include
r331854: lualoader: Do case-insensitive comparison of "yes"
r331855: lualoader: Don't assume that {module}_load is set
r331856: lualoader: revert whitespace change that snuck in
r331857: lualoader: Simplify some expressions
r331859: lualoader: Split logodefs out into logo-* files
r331860: lualoader: Don't try to lookup a nil logo
r332106: lualoader: Fix menu skipping with loader.conf(5) vars
r334723: lualoader: Add a loaded hook for others to execute upon config load
r334879: lualoader: Add hook.lua(8) to tree
r334891: lualoader: Add cli.lua(8) to the tree
r334907: lualoader: Process loader_conf_files properly
r334912: lualoader: Support variable substitution in env var settings
r334939: lualoader: Allow brand-*.lua for adding new brands
r334986: lualoader: More black-on-white fixes
r335009: lualoader: Match Forth module-loading behavior w.r.t flags
r335371: lualoader: Correct kernel_options handling
r336759: lualoader: "nextboot_file" should be spelled "nextboot_conf"
r337711: lualoader: Fix parsing of negative number loader.conf(5) variables
r337807: Prevent a wanring about checkdp being unused.
r337808: When the LUA floating point model is INT64, we don't need to do the
r337809: For our INT64 implementation, we can compare integers and numbers
r337810: MFV r337586: lua: Update to 5.3.5
r338054: Add config.lua(8) to the tree
r338063: lualoader: Stop exporting drawer.draw
r338065: lualoader: Hide most of the internal drawing functions
r338066: lualoader: Hide the rest of the private interfaces
r338067: lualoader: Add drawer-exported variables for default logodefs
r338083: Add drawer.lua(8)
r338085: Add color.lua(8), password.lua(8), and screen.lua(8)
r338086: lualoader: Install all manpages
r338108: Serial console menus for lua.
r338167: lualoader: Just compare expression directly
r338168: lualoader: Refactor config line expressions
r338173: lualoader: Fix loader.conf(5) EOL validation for 'exec' lines
r338203: Turn off LOADER_GELI and LOADER_LUA for sparc64, until functional
r338255: lualoader: Fix (add) Xen support
r338259: lualoader: Accept that people use unquoted values in loader.conf
r338309: lualoader: Fix override of module_path on loader prompt
r338394: lualoader: fix color usage
r338438: lualoader: Handle comma-separated kernels as well
r338886: Improve loader passwords:
r338893: Set the default loader for powerpc(32- and 64-bit) back to to forth
r339173: Set the default loader for powerpc64 back to to forth too.
r339200: lualoader: Don't draw loader menu with autoboot_delay=-1
r339218: lualoader: Create a module blacklist, add DRM modules to it
r339222: lualoader: Honor boot_* variables at lua init
r339301: Loader GELI support, like lua loader, seems to be broken on PowerPC
r339307: lualoader: Provide a 'menu' command to redraw the menu at prompt
r339677: lualoader: unload upon kernel change if a kernel was loaded
r339678: menu.lua: Abort autoboot sequence on failed command
r339702: lualoader: Improve module loading diagnostics
r339805: lualoader: Always return a proper dictionary for blacklist
r339831: Move LUA_ROOT to /boot/lua
r339849: lualoader: Fix try_include error handling
r340040: lualoader: Implement boot-conf
r340152: lualoader: Add chainload menu entry

Relnotes: yes ("lualoader has been merged to stable/11, off by
default, to facilitate testing")

Direct commit since these files have gone away in head

Move pc98's biosdisk.c to the new style disk access. This is missing
support in common/part.c, however, for pc98 partitions, so it's
unlikely to actually work. Lack of a pc98 machine that's in sevice
limits my ability to test, but this allows pc98 to compile again.

MFC r334882, r334884-r334885: loader(8) boot flag <-> environment fixes

r334882:
stand: Consolidate checking for boot flags driven by environment vars

e.g. boot_mute, boot_single, boot_verbose, and friends; we checked for these
in multiple places, consolidate into common/ and allow a setting of "NO" for
any of these to turn them off. This allows systems with multiple
loader.conf(5) or loader.conf(5) overlay systems to easily turn off
variables in later processed files by setting it to NO.

Reported by: Nick Wolff @ iXsystems
Reviewed by: imp

r334884:
stand: Fix build after r334882

Not sure how this was not caught in Universe.

r334885:
stand: One more trivial consolidation (setting environment from howto)

MFC r334656, r334665, r334695

r334656:
Add vsnprintf() to libsa. Alpha-sort the printf prototypes in stand.h.

r334665:
Make the v*printf() functions in libsa return int instead of void.
This makes them compatible with the C standard signatures, avoiding
spurious mismatch errors in the places where the oddball requirements
of standalone code end up putting two declarations of the same function
in play.

r334695:
Remove comments and assertions that are no longer valid after r330809.

r330809 replaced duplication of devdesc struct fields with an embedded copy
of the devdesc struct, to avoid fragility. That means all the scattered
comments indicating that structs must match are no longer valid. Likewise
asserts that attempted to mitigate some of the old fragility.

Reviewed by: imp@

MFC r334432:

Fix build of stand with base gcc

* Make autoboot() a static function in stand/common/boot.c, so it does
not shadow local variables in gptboot.c and zfsboot.c.
* Remove -Winline from the Makefiles for gptboot, gptzfsboot and
zfsboot, as gcc will always fail to inline some functions, and there
is nothing we can do about it.
* For gcc <= 4.2.1, silence -Wuninitialized for isoboot, as it produces
a false positive warning.
* Remove deprecated and unnecessary -mcpu=i386 flag from stand/defs.mk,
as there is already a -march=i386 flag further in the file.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D15628

MFC: r333955

- Unbreak booting sparc64 kernels after the metadata unification in
r329190 (MFCed to stable/11 in r332150); sparc64 kernels are always
64-bit but with that revision in place, the loader was treating them
as 32-bit ones.
- In order to reduce the likelihood of this kind of breakage in the
future, #ifdef out md_load() on sparc64 and make md_load_dual() -
which is currently local to metadata.c anyway - static.
- Make md_getboothowto() - also local to metadata.c - static.
- Get rid of the unused DTB pointer on sparc64.

Approved by: re (kib)

MFC r332085

Add an ISO9660 "partition table" type to loader.

When booted via isoboot(8) loader will be handed a disk that simply contains
an ISO9660 image. Currently this confuses it greatly. Teach it how to spot
that it's in this situation and that ISO9660 has one "partition" covering
the whole disk.

Sponsored by: iXsystems, Inc.

MFC 328101,328911: Require SHF_ALLOC for kernel object module sections.

328101:
Require the SHF_ALLOC flag for program sections from kernel object modules.

ELF object files can contain program sections which are not supposed
to be loaded into memory (e.g. .comment). Normally the static linker
uses these flags to decide which sections are allocated to loadable
program segments in ELF binaries and shared objects (including kernels
on all architectures and kernel modules on architectures other than
amd64).

Mapping ELF object files (such as amd64 kernel modules) into memory
directly is a bit of a grey area. ELF object files are intended to be
used as inputs to the static linker. As a result, there is not a
standardized definition for what the memory layout of an ELF object
should be (none of the section headers have valid virtual memory
addresses for example).

The kernel and loader were not checking the SHF_ALLOC flag but loading
any program sections with certain types such as SHT_PROGBITS. As a
result, the kernel and loader would load into RAM some sections that
weren't marked with SHF_ALLOC such as .comment that are not loaded
into RAM for kernel modules on other architectures (which are
implemented as ELF shared objects). Aside from possibly requiring
slightly more RAM to hold a kernel module this does not affect runtime
correctness as the kernel relocates symbols based on the layout it
uses.

Debuggers such as gdb and lldb do not extract symbol tables from a
running process or kernel. Instead, they replicate the memory layout
of ELF executables and shared objects and use that to construct their
own symbol tables. For executables and shared objects this works
fine. For ELF objects the current logic in kgdb (and probably lldb
based on a simple reading) assumes that only sections with SHF_ALLOC
are memory resident when constructing a memory layout. If the
debugger constructs a different memory layout than the kernel, then it
will compute different addresses for symbols causing symbols in the
debugger to appear to have the wrong values (though the kernel itself
is working fine). The current port of mdb does not check SHF_ALLOC as
it replicates the kernel's logic in its existing kernel support.

The bfd linker sorts the sections in ELF object files such that all of
the allocated sections (sections with SHF_ALLOCATED) are placed first
followed by unallocated sections. As a result, when kgdb composed a
memory layout using only the allocated sections, this layout happened
to match the layout used by the kernel and loader. The lld linker
does not sort the sections in ELF object files and mixed allocated and
unallocated sections. This resulted in kgdb composing a different
memory layout than the kernel and loader.

We could either patch kgdb (and possibly in the future lldb) to use
custom handling when generating memory layouts for kernel modules that
are ELF objects, or we could change the kernel and loader to check
SHF_ALLOCATED. I chose the latter as I feel we shouldn't be loading
things into RAM that the module won't use. This should mostly be a
NOP when linking with bfd but will allow the existing kgdb to work
with amd64 kernel modules linked with lld.

Note that we only require SHF_ALLOC for "program" sections for types
like SHT_PROGBITS and SHT_NOBITS. Other section types such as symbol
tables, string tables, and relocations must also be loaded and are not
marked with SHF_ALLOC.

328911:
Ignore relocation tables for non-memory-resident sections.

As a followup to r328101, ignore relocation tables for ELF object
sections that are not memory resident. For modules loaded by the
loader, ignore relocation tables whose associated section was not
loaded by the loader (sh_addr is zero). For modules loaded at runtime
via kldload(2), ignore relocation tables whose associated section is
not marked with SHF_ALLOC.

MFC r331331: Apply r228478 (CTASSERT => _Static_assert()) to bootstrap.h

MFC r330806-r330815, r330837, r330864, r330883

pc98 changes associated with the named commits are also included in this
commit, despite not having been made with the original commits due to its
removal in head.

r330806:
Minor cosmetic changes.

Make sure { on the same line as struct for all struct *devdesc. Move
some type definitions to next to the dv_type define, since that's what
sets the d_type.

r330807:
We can't use d_opendata for blkio storage.

open_disk uses d_opendata for it's own purpse. We can't store blkio
there. Fortunately, blkio is stored elsewhere and we never actually
retrieve blkio from d_opendata. Eliminate it as a source of confusion.
Eliminate all stores of d_opendata in efi since this layer doesn't own
that field.

r330808:
Make struct libi386_devdesc match the struct devdesc better

Move data to top and call it d_opendata.

r330809:
Use the actual struct devdesc at the start of all *_devdesc structs

The current system is fragile and requires very careful layout of all
*_devdesc structures. It also makes it hard to change the base
devdesc. Take a page from CAM and put the 'header' in all the derived
classes and adjust the code to match.

For OFW, move the iHandle h_handle out of a slot conflicting with
d_opendata. Due to quirks in the alignment rules, this worked.
However changing the code to use d_opendata storage now that it's a
pointer is hard, so just have a separate field for it.

All other cleanups were to make the *_devdesc structures match where
they'd taken some liberties that were none-the-less compatible enough
to work.

r330810:
Remove d_type from devdesc. It's not needed as we can fetch it from
d_dev->dv_type when we need it.

r330811:
GC unused routines.

Sponsored by: Netflix

r330812:
Use the one-line-per-file pattern here, and sort the file names.

Sponsored by: Netflix

r330813:
Move the env convenience routines out of boot1.c.

These routines are more generally useful. Even though boot1 is on its
way out, it's better to make these common during the transition than
copy them.

r330814:
Star BootCurrent entry when booting.

Sponsored by: Netflix

r330815:
Print the load and device path as well as BootCurrent and BootOrder

Sponsored by: Netflix

r330837:
biosdisk.c should not set d_opendata.

Same as 330807, d_opendata is owned by open_disk and we should not
set it.

M stand/i386/libi386/biosdisk.c

r330864:
Prefer uintXX_t to u_intXX_t

A foolish consistency is the hobgoblin of little minds, adored by
little statesmen and philosophers and divines. With consistency a
great soul has simply nothing to do. -- Ralph Waldo Emerson

r330883:
Fix typo that misteriously passes compilation.

MFC r329190, r329315, r330131: metadata load file unification

r329190: Unify metadata load files for arm, mips, powerpc, sparc64

Summary:
All metadata.c files are very similar, with only trivial changes. Unify them
into a single common file, with minor special-casing where needed.

r329315: stand: Fix ubldr after r329190

metadata load files were consolidated in r329190, and these relocation fixup
bits were inadvertently dropped in the process. Re-add them to fix boot with
ubldr.

r330131: Fix module loading on arm after the metadata.c unification in
r329190. Arm modules need an additional address fixup not needed by other
platforms.

MFC r329745: load_elf.c: Use consistent indentation

As noted in D14267 load_elf.c has a variety of indentation styles. Move
to standard 8 column hard tab indents, 4 space second level indents.
Also includes some whitespace cleanups found by clang-format.

MFC r329725, r329831

r329725:
Consolidate three copies of ZFS commands into a central location.

There's no reason to have multiple copies of lszfs and
reloadbe. Consolidate them into one location. Also ldi_get_size is the
same everywhere (except sparc64). Make it the same everywhere as the
common definition is more general and will work on spar64.

r329831:
Fix userboot w/ ZFS after r329725

r329725 cleaned up ZFS commands duplicated in multiple places, but userboot
was not setting HAVE_ZFS when MK_ZFS != "no". This resulted in a failure to
boot (as seen in PR 226118) in bhyve, with the following message:

/boot/userboot.so: Undefined symbol "ldi_get_size"

Revert MFC r328911 from r329183

jhb@ had specified an MFC timer of 1-month, and I clobbered his efforts by
MFC'ing it way prematurely. Back out that part of r329183 so it may be MFC'd
when jhb is ready.

MFC Loader Fixes Final: r327612,r327703,r327704,r327878,r327879,r327881,
r328007,r328029,r328030,r328031,r328061,r328156,r328169,r328288,r328289,
r328290,r328291,r328292,r328411,r328536,r328603,r328614,r328642,r328769,
r328779,r328780,r328781,r328782,r328783,r328806,r328808,r328826,r328835,
r328911,r328986,r328987,r328990,r328999,r329000,r329019,r329050,r329054,
r329060

r327612: Invent new #defines for the biospci_{read,write}_config function

r327703: Define __dmadat after #include'ing ufsread.c.

r327704: Fix printf missing format variables warnings.

r327878: Add GUID for UEFI boot manager variables.

r327879: Report the boot order and where we are in that boot order.

r327881: Allow this file to be included

r328007: Fix booting on some arm64 systems after r327879 by fixing the call
to utf8_to_ucs2

r328029: When returning an error and freeing allocated memory from
ucs2_to_utf8, NULL the return pointer.

r328030: Check the return value from utf8_to_ucs2 instead of whether or not
uv is NULL.

r328031: Need to free uv after we're done using it.

r328061: utf8_to_ucs2() should check for malloc failure

r328156: stand: Move sections around to fix stand/ build with ld.lld on
armv7

r328169: Remove extra copy of bootinfo.c. It's a bit rotted copy of the one
in efi/loader.

r328288: Fix some resource leaks.

r328289: Don't leak memory when displaying help.

r328290: On malloc failure, be sure to close the include file that triggered
it.

r328291: getenv does not return tainted data in the boot loader. Attempt to
clue Coverity into that fact.

r328292: There's no tainted data here, tag it as such to avoid false
positives.

r328411: loader.efi: add missing EFI GUIDs

r328536: loader: support for mixed-endianness ELF/loader and POWER8

r328603: Add missing non-POWERPC case to give the scr value something
non-zero.

r328614: Move libstand.3 to libsa.3. Update libsa.3 to include functions

r328642: Break out the interpreters (simple and forth) w/o ifdefs.

r328769: Centralize several variables.

r328779: Retire pnp.4th and the code needed only for 4th words used here.

r328780: These 4th words were an attempt to allow integration into the boot
loader scripts. However, that path won't be taken after all it
seems.

r328781: Remove pcibios forth support.

r328782: Now that we no longer conditionally compile some files outside of
ficl

r328783: Invent new LDR_INTERP for the loader interpreter to use.

r328806: We need more heap space to properly load newer powerpc kernels.

r328808: Implement strcoll as strcmp.

r328826: Make cross-endian loader changes apply only to powerpc

r328835: Fix regression introduced in r328806, preventing boot on many
platforms.

r328911: Ignore relocation tables for non-memory-resident sections.

r328986: Fix relative location of USB sources after recent move.

r328987: A more definitions to kernel emulation shim in order to build
stand/usb.

r328990: Move the stand/usb test loader into its own directory.

r328999: Fix indentation to FreeBSD standard for interp files

r329000: Move simple interpreter 'perform' into interp.c and call it

r329019: Move to tabs for indentation and to 8-space notches, per style(9).

r329050: Fix build of userboot.so

r329054: Set script.lang in the environment to either 'forth' or 'simple' to
reflect what scripting language was compiled into the loader.

r329060: loader: fix endianness conversion

PR: 225323

MFC Loader Fixes 2017q4p7: r324844,r326089,r326926,r326440,r326484,r326494,
r326588,r326708,r326784,r326914,r327390,r328446,r326090,r326143,r326144,
r326182,r326384,r326421,r326440,r326441,r326442,r326443,r326444,r326445,
r326446,r326447,r326448,r326484,r326485,r326486,r326487,r326488,r326490,
r326491,r326492,r326493,r326494,r326495,r326504,r326507,r326509,r326584,
r326585,r326586,r326587,r326588,r326589,r326590,r326591,r326592,r326593,
r326594,r326600,r326616,r326671,r326707,r326708,r326709,r326710,r326711,
r326712,r326714,r326720,r326768,r326772,r326784,r326792,r326812,r326854,
r326855,r326856,r326858,r326886,r326887,r326914,r326926,r326927,r326960,
r326961,r326962,r326963,r327351,r327453,r327390,r327523,r327524,r326489,
r327880,r328437,r328438,r328439,r328441,r328446,r328448,r328449,r328612,
r328613,r328615

While here, undo our libfdt hack of not including <stdlib.h> if we're
compiling _STANDALONE.

r324844: When building standalone, don't define errno. Let the definition from
stand.h override. This is similar to what we do in the kernel.

r326089: loader.efi: efipart does not recognize partitionless disks

r326090: net_parse_rootpath() has no parameters

r326143: Fix theoretical integer overflow issues. If the product here is

r326144: Mark the func pointer as __dead2. It looks up loader_main, which

r326182: Modify all FreeBSD bootloaders on PowerPC AIM (Book-S) systems

r326384: Use const pointers to avoid casting away constness.

r326421: loader.efi: efipart should exclude iPXE stub block protocol

r326440: Remove stale dependency on ufsread.c

r326441: Minor flags cleanup

r326442: Cleanup CFALGS usage here

r326443: We don't need both _STAND and _STANDALONE, use the latter.

r326444: Move geli to common DO32 stuff

r326445: Fix random() and srandom() prototypes to match the standard.

r326446: Undefine _STANDALONE since this is test code.

r326447: Tweaks to the beri boot loader so that it builds w/o warnings.

r326448: Fix all warnings related to geli and ZFS support on x86.

r326484: Const poison the propname.

r326485: Delcare md_load in libofw.h. Make all prototypes match for ofw

r326486: Include machine/md_var to pick up __syncicache prototype.

r326487: Cast mdp (a vm_offset_t) to void * to match prototype.

r326488: e_entry can be smaller than a pointer. Cast it to an intptr_t

r326490: Declare our strange brand of main().

r326491: Disconnet ps3 from the build. There's too many warnings to fix.

r326492: Cast void * pointer to char * so the arg matches the %s format.

r326493: Provide a md_load64 prototype.

r326494: Mark two things as unused (since they are only sometimes used)

r326495: Now it's safe to bump WARNS to 1.

r326504: Switch to proper MK_LOADER_GELI tests.

r326507: increase maximum size of zfsboot

r326509: loader.efi: add note about iPXE into the efipart.c

r326584: When building standalone, include stand.h rather than the kernel

r326585: Include ficl.h before anything else

r326586: No need to include the userland md5.h, the kernel one is just fine.

r326587: Use the kernel relative paths, rather than the userland relative
paths

r326588: Need to include skein in the include path

r326589: Make sure we include the right path for skein.h

r326590: Prefer stdint.h to inttypes.h

r326591: This isn't NetBSD specific code. Include these for any kernel /

r326592: Don't inherit CFLAGS. This a specialized test program.

r326593: Stop building with the standard system headers.

r326594: Now that we offer a semi-sane standards-ish set of #include files,
stop hacking includes with sed.

r326600: Since this is contrib code, create an upstreamable version of my

r326616: dhcp_try_rfc1048() is not used any more

r326671: Avoid setting -Wno-tentative-definition-incomplete-type with gcc.

r326707: Add partial support signal.h functioanlity. Pull in
machine/signal.h

r326708: Remove _KERNEL hack now that errno.h does the right thing when
built standalone.

r326709: Provide implementations for iscntrl, ispunct and isgraph.

r326710: Put the files we're copying over into a few variables and clean
hings up.

r326711: Const poison a couple of interfaces.

r326712: Create interp class.

r326714: boot1.c needs EFI_ZFS_BOOT too, so add it globally.

r326720: This path belongs in ficl/Makefile, not the common defines for
users

r326768: Fix a comment to be more accurate

r326772: Fix regression with lua import

r326784: Revert part of 362772. It was causing problems for includes

r326792: Attempt to unbreak buildworld

r326812: Revert r326792, r326784, r326772, r326712

r326854: libefi: make efichar.h more usable in stand code

r326855: Cargo cut a fix for the regressions r326585 caused.

r326856: Fix comments after bump in size.

r326858: Revert r326855: Cargo cut a fix for the regressions r326585 caused.

r326886: Panic in sbrk if setheap hasn't been called yet. This is preferable
o a mysterious crash

r326887: Remove the 'mini libstand in libstand' that util.[ch] provided.

r326914: Move loader-only defines to loader.mk from defs.mk

r326926: Move loader help file definitions to being 100% inside of
loader.mk.

r326927: libficl is only ever used in a loader (never a boot) program. Move
it.

r326960: Simplify things a little. The RETURN macro isn't required.

r326961: Interact is always called with NULL. Simplify code a little

r326962: Hoist btx include stuff to i386/Makefile.inc

r326963: No need to use relative paths like this here.

r327351: Fix ubldr. uboot/lib uses defines for the loader.

r327453: Add a validbcd() routine that uses the bcd2bin_data[] array

r327390: Garbage-collect loader.ps3. It is currently disconnected from the
build and kboot replaces.

r327523: Don't clobber system LDFLAGS for beri boot loaders.

r327524: Use 'extern uint8_t' instead of 'extern void' for external symbols.

r326489: Allow this file to be used in libsa without warning...

r327880: Move getsecs() prototype to stand.h from net.h so it can be used

r328437: Split panic routine

r328438: Implement abort() as a call to panic.

r328439: Provide abs form stdlib.h.

r328441: abort() should be marked __dead2 since it won't return.

r328446: Now that exit is __dead2, we need to tag ub_exit() as __dead2.

r328448: Make exit() never return until host_exit can be written.

r328449: Tag unreachable places as such. I left the while (1); in place

r328612: Move strtold wrapper from strtol.c to its own strtold.c.

r328613: Kill copies of strtol and strtoul.

r328615: Update stand.h for changes for strto*l

PR: 223969

MFC r325834,r325997,326502: Move sys/boot to stand/

This is effectively a direct commit to stable/11, due to differences between
stable/11 and head. Changes to DTS in sys/boot/fdt/dts were often
accompanied by kernel changes. Many of these were also risc-v updates that
likely had many more dependencies to MFC.

Because of this, sys/boot/fdt/dts remains as-is while everything else in
sys/boot relocates to stand/.

r325834: Move sys/boot to stand. Fix all references to new location

r325997: Remove empty directories.

r326502: Document the sys/boot -> stand move in hier.7 and the top-level README.

MFC Loader Fixes 2017q4p6: r324649,r324650,r324651,r324652,r324653,r324654,
r324700,r324702,r324709,r324717,r324719,r324841,r324842,r324843,r324845,
r324850,r324876,r324877,r324878,r324879,r324880,r324881,r324883,r324980,
r324981,r324982,r324995,r325014,r325093,r325094,r325114,r325170,r325171,
r325172,r325173,r325174,r325175,r325176,r325248,r325286,r325310,r325332,
r325338,r325339,r325376,r325377,r325379,r325380,r325382,r325478,r325479,
r325480,r325482,r325483,r325484,r325485,r325556,r325641,r325681,r325685,
r325686,r325687,r325688,r325689,r325690,r325691,r325692,r325693,r325694,
r325743,r325744,r325748,r325775,r325779,r325780

r324649: Move common/Makefile.inc to sys/boot/loader.mk.

r324650: tweak style

r324651: create defs.mk for common definitions

r324652: Move all the ficl common code into ficl.mk

r324653: LOADER_foo_SUPPORTED

r324654: Explicitly inlcude SYSDIR in the include path

r324700: loader: initialize dv_cleanup in md.c to eliminate clang warning

r324702: We need to include disk.o in libuboot.a when we're building with
support for disk access.

r324709: Revert "Unify boot1 with loader" change r324646

r324717: libsa/ip.c: misplaced comment, ip_v is half char, not ip_p

r324719: libsa/ip: stop read loop on bad fragments

r324841: Use BOOTDIR more consistently in defs.mk rather than repeat
sys/boot.

r324842: Introduce BOOTOBJ: The top level object directory for the boot tree

r324843: Stopgap fix to the mistmatch between LOADER_GELI_SUPPORT and
LOADER_NO_GELI_SUPPORT.

r324845: Use BOOTOBJ and BOOTDIR to find geli includes and libraries.

r324850: Define LIBSA32 to LIBSA on i386 to fix build.

r324876: Move fdt and uboot defines into common uboot.mk.

r324877: End source directories with SRC rather than a hodgepodge of names

r324878: Make at91 boot loader compile again.

r324879: Prefer SRCTOP paths for bits we're grabbing from libc.

r324880: Use BOOTSRC here.

r324881: Use SYSDIR instead of ${.CURDIR}/../..<etc>/sys.

r324883: Use preferred defined paths, rather than relative paths in fdt.

r324980: Use BOOTDIR consistently.

r324981: Move BINDIR definition to defs.mk, and override where it isn't
/boot

r324982: Remove sys/boot/arm/at91 and ixp425

r324995: loader.efi: Make framebuffer commands available for arm64

r325014: Add a 'place holder' arm struct efi_fb until a real one comes

r325093: Define new EFI variables

r325094: Cleanup non-arch Makefiles

r325114: Use defs.mk defins in most MD code

r325170: Use defs.mk values for userboot

r325171: Use defs.mk name and prefer bsd.init.mk

r325172: Remove the -nostdlib stuff I added. Instead, fix LDFLAGS to be
honored correctly with the new Makefile.inc include order.

r325173: We don't need to build a special ficl for userboot.

r325174: Minor cleanup

r325175: For amd64, compile both zfs and zfs32 libraries.

r325176: Actually add zfs32/Makefile

r325248: loader ptblread() is broken with >512B sectors

r325286: efipart_strategy is using wrong offset with >512B sectors

r325310: zfs.c:vdev_read() needs to be careful about large sectors

r325332: loader: re-enable gzip support for x86

r325338: loader: fix BOOTSRC -> BOOTOBJ in a library path

r325339: This used to have bzip2 support too.

r325376: WIP: centralize machine links

r325377: mostly libsa

r325379: Revert "mostly libsa"

r325380: Revert "WIP: centralize machine links"

r325382: Cleanup stray libstand names to be libsa names.

r325478: Powerpc is a 32-bit boot loader.

r325479: Define LIBFICL32 to be libficl.a on i386 and libficl32.a on amd64.

r325480: Use DO32 for all the places that we need to flag we're building a
32-bit version of a library.

r325482: Move machine and other link creation to defs.mk

r325483: MACHINE can never be powerpc64, so cleanup code that thinks it can.

r325484: Prefer bsd.init.mk to src.opts.mk

r325485: Centralize all 32-bit builds on 64-bit platform stuff.

r325556: loader: set options before including bsd.init.mk

r325641: loader.efi: efi_devpath_is_prefix should return bool

r325681: boot1: avoid using NULL device path

r325685: libsa32 isn't needed for i386. It's already a 32-bit platform.

r325686: Simplify this if to a direct assignment.

r325687: Remove all the empty help files from the powerpc build.

r325688: FDT support doesn't make sense for ps3, remove it.

r325689: Remove LOADER_FDT_SUPPORT as a Makefile variable.

r325690: Remove LOADER_ZFS_SUPPORT as a Makefile variable

r325691: Remove useless PNP define here.

r325692: Replace LOADER_FIREWIRE_SUPPORT variable

r325693: Move LOADER_{NO,}_GELI_SUPPORT to MK_LOADER_GELI

r325694: Install the 4th files in sys/boot/forth instead of each loader

r325743: Make sure the proper loader.rc gets installed.

r325744: boot1: also check for NULL device

r325748: Use proper include file.

r325775: Add loader.conf to the list of files that are MD.

r325779: Add /boot/dts to the list of default modules.

r325780: Don't add /boot/dt*s* but /boot/dt*b*. Stupid think-o.

MFC r324648: Rename top level Makefile.ficl to ficl.mk.

MFC Loader Fixes 2017q4p5: r324557,r324558,r324559,r324646,r324647

r324557: Rename exit to efi_exit to avoid clashing with libsa exit
definition

r324558: Define prototype for exit and ensure references

r324559: Move panic back into libsa. It's documented in libstand(3) to
belong there.

r324646: Unify boot1 with loader.

r324647: Move orphaned man pages into new man directory from common.

MFC Loader Fixes 2017q4p4: r324552,r324553,r324554,r324555,r324556

r324552: Kill the userboot copy of libstand.

r324553: Rename libstand.a to libsa.a and libstand32.a to libsa32.a

r324554: Move crc32.c, util.c and gpt.c over to libsa.

r324555: Honor CFLAGS modifications in Makefile.inc by using += here.

r324556: Move ufsread.c

MFC Loader Fixes 2017q4p3: r324460,r324471

r324460: Rather than laying whack-a-mole with including the path to stand.h,
always include it.

r324471: Fix typos: `stand alon{e,g}` should be spelled like `standalone`

MFC Loader Fixes 2017q4p1: r324321,r324359,r324360,r324388,r324449,r324450,
r324451,r324452

r324321: This README file was quite relevant for FreeBSD 3 or 4. No more.

r324359: Add efi_devpath_is_prefix

r324360: Encapsulate ZFS preferences into efi_zfs_is_preferred

r324388: Unbreak building efiboot with MK_ZFS == no

r324449: Prefer ${LIBSTAND} to -lstand

r324450: Define SASRC and use it

r324451: Add missing -I${SASRC} in a couple of places so that stand.h
doesn't accidentally come in via host pollution.

r324452: Define LIBSA* and use them instead of overloaded LIBSTAND

MFC Loader Fixes 2017q3: r320547,r320553,r321621,r321844,r321969,r321991,
r322037,r322038,r322039,r322040,r322056,r322074,r322542,r322592,r322593,
r322896,r322923,r323671,r322930,r322931,r322932,r322933,r322934,r322935,
r322936,r322937,r322938,r322939,r322941,r323062,r323063,r323064,r323065,
r323100,r323131,r323174,r323258,r323261,r323272,r323367,r323379,r323389,
r323407,r323428,r323436,r323494,r323496,r323497,r323541,r323554,r323589,
r323707,r323867,r323885,r323886,r323895,r323896,r323897,r323905,r323906,
r323907,r323908,r323909,r323952,r323991,r324099,r324558,r326445,r326609,
r326610

This batch includes a special kludge to fix powerpc loader build; <stdlib.h>
was included after <stand.h> there, causing problems with DEBUG_MALLOC bits.
Include <stdlib.h> a little bit earlier to fix the build with the intention
of removing this when eventually libsa silently replaces stdlib.h with
stand.h.

r320547: Link EFI/uboot loaders with -znotext

r320553: Integer underflow in efipart_realstrategy when I/O starts after end
of disk

r321621: Always set the receive mask in loader.efi.

r321844: Clean up style in print_state(..) and pager_printf(..)

r321969: Fix the return types for printf and putchar to match their libc

r321991: Revert r321969

r322037: Add stpcpy and stpncpy to libstand

r322038: Add definitions and utilities for EFI drivers

r322039: Move EFI ZFS functions to libefi

r322040: Add EFI utility functions to libefi

r322056: Move EFI fmtdev functionality to libefi

r322074: libefi/time.c cstyle cleanup

r322542: loader.efi: repace XXX with real comments in trap.c

r322592: Remove unused defines.

r322593: Define proposed GUID for FreeBSD boot loader variables.

r322896: Make spinconsole platform independent and hook it up into EFI
loader

r322923: Hide length of geli passphrase during boot.

r323671: Fix language used in the r322923.

r322930: Move efi_main into efi/loader

r322931: Cleanup efi_main return type

r322932: Use the loader.efi conventions for the various EFI tables.

r322933: No need for MK_ZFS around these: they are by their nature only
active when MK_ZFS is true.

r322934: _STAND is sometimes defined on the command line. Make the define
here match.

r322935: Fix warnings due to type mismatch.

r322936: Remove useless 'static' for an enum definition.

r322937: Forward declare struct dsk to avoid warnings when building libi386.

r322938: Link in libefi for boot1

r322939: Use efi_devpath_str for debug path info.

r322941: Eliminate redunant device path matching.

r323062: Make efichar.c routines available to libefi.

r323063: boot1.efi: print more info about where boot1.efi is loaded from

r323064: Exit rather than panic for most errors.

r323065: Save where we're booted from

r323100: libstand: nfs_readlink() should return proper return code

r323131: Revert r322941: Eliminate redundant device matching functions

r323174: Fix loader bug causing too many pages allocation when bootloader
is U-Boot

r323258: ucs2len

r323261: Fix armv6 build

r323272: Be consistent and do return (1);

r323367: Mark init_chroot and init_script variables as deprecated.

r323379: It's been pointed out that init_script at least is useful w/o

r323389: loader.efi: chain loader should provide proper device handle

r323407: boot1 generate-fat: generate all templates at once

r323428: r323389 breaks the kernel build when WITHOUT_ZFS is defined in
src.conf

r323436: boot1: remove BOOT1_MAXSIZE default value

r323494: loader should support large_dnode

r323496: libstand: tftp_open() can leak pkt on error

r323497: libefi: efipart_open should check the status from disk_open

r323541: libefi: efipart_realstrategy rsize pointer may be NULL

r323554: Increase EFI boot file size frok 128k to 384k

r323589: loader: biosmem.c cstyle cleanup

r323707: loader: biosmem allocate heap just below 4GB

r323867: libefi: devicename.c cleanups

r323885: libefi: efi_devpath_match() should return bool

r323886: libefi: efipart.c should use calloc()

r323895: libefi: efi_devpath_match local len should be unsigned

r323896: r323885 did miss efilib.h update

r323897: efilib.h: typo in structure member description

r323905: libefi: pdinfo_t pd_unit and pd_open should be unsigned

r323906: libefi: efipart_strategy() should return ENXIO when there is no
media

r323907: libefi: efipart.c cstyle fix for efipart_print_common()

r323908: libefi: efipart_hdinfo_add_filepath should check strtol result

r323909: libefi: define EISA PNP constants

r323952: After the r317886 support for TFTP and NFS can be enable
simultaneously.

r323991: libefi: efipart_floppy() will should not pass acpi pointer if the
HID test fails

r324099: Compile loader as Little-Endian on PPC64/POWER8

r324558: Define prototype for exit and ensure references

r326445: Fix random() and srandom() prototypes to match the standard.

r326609: Make putenv and getenv match the userland definition

r326610: Fix random() prototype to match the system.

PR: 219000 221001 222215
Relnotes: yes ("The length of the geli passphrase is hidden during boot")

MFC Loader Fixes 2017q2: r316437,r316577,r316578,r316585,r316590,r316612,
r316625,r316628,r316654,r316682,r316704,r316771,r317092,r317097,r317099,
r317652,r317785,r317886,r317887,r318142,r318320,r318356,r318678,r318754,
r318982,r318986,r318987,r318988,r318989,r318990,r318991,r318992,r318993,
r318994,r318999,r319083,r319084,r319085,r320011,r320234,r320288,r320304,
r320467,r320482

r316437: Small cleanup to make i386/loader match efi/loader boot environment
code

r316577: loader: part.c cstyle cleanup

r316578: loader: want mechanism to avoid RA with bcache

r316585: loader: zfs reader should check all labels

r316590: libstand/dosfs: cache FAT32 in 128 Kb blocks to save loader memory

r316612: In r298230 the value of HEAP_MIN was changed from 3MB to 64MB.
Correct a comment.

r316625: Do not use -msoft-float with intention of disabling FP on aarch64

r316628: Silence GCC warning by initializing the local variable.

r316654: loader: r316585 did miss userboot update

r316682: loader: r316585 did miss sparc/ofw

r316704: loader.efi: only fetch zfs pool guid for the actual boot device

r316771: loader: Avoid possible overflow via environment variable

r317092: loader: zfs reader vdev_probe should check for minimum device size

r317097: loader: F_READ/F_WRITE should be checked against masked flag

r317099: loader: uboot disk ioctl should call disk_ioctl

r317652: loader.efi: ResetSystem does not use data with EFI_SUCCESS

r317785: zfsboot: drvsize() may be unusable on some systems

r317886: distinguish NFS versus TFTP boot by rootpath

r317887: loader: network read rework

r318142: libstand: NULL pointer dereference in rarp

r318320: loader: add ip layer code into libstand

r318356: libstand: increase nfs max read size to 16k

r318678: Replacing iterating over rootpath by strsep(3).

r318754: Pass -N directly to ld via -Wl rather than passing it to the
compiler driver.

r318982: Pass a "FREEBSD" user-class in PXE dhcp request

r318986: add a comment on vendor index 19 and 20 to avoid confusion

r318987: Support URI scheme for root-path in netbooting

r318988: Always build tftpfs support along with nfs for pxeboot

r318989: Always issue the pxe request

r318990: Partially revert r314948

r318991: Document recent changes on pxeboot

r318992: Capitalize DHCP

r318993: Use the usual FreeBSD spelling for the DHCP user class

r318994: Catch with the change in the user class

r318999: Update the comments concerning net_parse_rootpath to reflect what
it is now

r319083: Followup on the user-class changes

r319084: Small cleanup in dev_net.c

r319085: use the same option list for dhcp discovery and request

r320011: Add chain loader support for loader

r320234: Make structure padding explicit in EFI_MEMORY_DESCRIPTOR

r320288: Allow Clang's integrated assembler to assemble boot0

r320304: loader.efi: Disable smbios for arm

r320467: loader: chain load relocate data declaration is bad

r320482: As with arm64 mark the EFI PE header as allocated on arm.

PR: 218473

MFC Loader Fixes 2017q1: r311458,r312237,r312314,r312374,r312947,r313042,
r313047,r313166,r313328,r313332,r313333,r313337,r313348,r313349,r313389,
r313442,r313451,r313575,r313645,r313710,r314114,r314213,r314275,r314945,
r314948,r315008,r315408,r315427,r315645,r315646,r315648,r315653,r315850,
r316064,r316078,r316079,r316100,r316104,r316111,r316112,r316171,r316279,
r316280,r316287,r316311,r316343,r316424,r316436

r311458: Use compiler driver to link BERI boot loaders

r312237: loader.efi: find_currdev() can leak memory

r312314: loader: move device path definitions to include/efidevp.h

r312374: loader: efi devpath api usage should be more aware of NULL pointers

r312947: Remove "-Xassembler -G0" from CFLAGS.

r313042: loader.efi environment related cleanups

r313047: loader: disk/part api needs to use uint64_t offsets

r313166: loader: libefi/env.c warnings in arm build

r313328: loader: Implement disk_ioctl() to support DIOCGSECTORSIZE and
DIOCGMEDIASIZE.

r313332: loader: bcache read ahead block count should take account the large
sectors

r313333: loader: Replace EFI part devices.

r313337: loader: 313329 missed ZFS guard in loader/main.c

r313348: loader: biosdisk fix for 2+TB disks

r313349: loader: disk io should not use alloca()

r313389: efipart is also using the '%S' printf format, add -Wno-format for
it.

r313442: loader: possible NULL pointer dereference in efipart.c

r313451: loader: possible NULL pointer dereference in bcache.c

r313575: makefs: make the buffer functions look exactly like the kernel ones

r313645: loader: implement MEDIA_FILEPATH_DP support in efipart

r313710: loader: cstyle fixes and DIOCGMEDIASIZE should use uint64_t

r314114: Use LDFLAGS with CC instead of _LDFLAGS.

r314213: Remove control+r handling from geliboot's pwgets()

r314275: Remove unused macro from common/drv.c.

r314945: Some style(9) fixes. No functional changes.

r314948: Try to extract the RFC1048 data from PXE.

r315008: r314948 seems to be missing a variable or two that will break

r315408: loader: remove open_disk cache

r315427: loader: biosdisk should report IO error from INT13

r315645: loader: disk_cleanup was left in userboot_disk.c

r315646: loader: pxe.h constants have wrong values

r315648: libstand: verify value provided by nfs.read_size

r315653: loader: verify the value from dhcp.interface-mtu and use snprintf
o set mtu

r315850: The original author abused Nd (one-line description, used by
makewhatis)

r316064: Fix build with path names with 'align' or 'nop' in them.

r316078: gpt*boot: Save a bit more memory when LOADER_NO_GELI_SUPPORT is
specified

r316079: Simply retire the sedification of the boot2.s file.

r316100: Remove -fno-guess-branch-probability and -fno-unit-at-a-time.

r316104: Use `NO_WCAST_ALIGN` instead of spelling it out as -Wno-cast-align
in CFLAGS

r316111: loader: move bios getsecs into time.c

r316112: loader: ls command should display file types properly

r316171: xfsread inlined uses more space, so remove the inline tag.

r316279: loader: efipart should check disk size from partition table

r316280: loader: simplify efi_zfs_probe and avoid double probing for zfs.

r316287: Remove OLD_NFSV2 from loader and libstand

r316311: Add explicit_bzero() to libstand, and switch GELIBoot to using it

r316343: Implement boot-time encryption key passing (keybuf)

r316424: Fix sparc64 build broken by r316343 and r316076

r316436: Restore EFI boot environment functionality broken in r313333

PR: 216940 217298 217935

MFC r307322,r307323,r307324,r307326,r307327,r307338,r307879,r307908,r307911,
r307942,r307950,r307951,r307954,r307955,r308125,r308195,r308476,r308534,
r308535,r308776,r308843,r310236,r310726: Loader fixes, 2016q4

r307322: Remove /boot/boot.conf, deprecated for 16 years

r307323: Remove fetching of pInterp.

r307324: Create a new linker set, Xficl_compile_set which contains a list of
functions to call to register new forth words.

r307326: In UEFI mode expose the SMBIOS anchor base address via kenv

r307327: Update i386 build of loader.efi (but leave it disabled) so that we
at least build it now.

r307338: Create a pcibios-version environment FORTH word.

r307879: Preliminary support for EFI in boot loader.

r307908: Fix the build on both arm64 and when WITHOUT_FORTH is defined.

r307911: Add better comment...

r307942: Really make WITHOUT_FORTH (MK_FORTH==no) work.

r307950: Add it (Makefile.ficl) to the right place

r307951: Fix two backwards tests.

r307954: Back out the move to the loader script from -N.

r307955: LIBSTAND goes last, so put it last here too.

r308125: In loader.efi, instead of exiting directly, try to fallback.

r308195: efinet_dev_print should honor verbose option.

r308476: boot/forth spelling issue in forth word

r308534: The file_loadraw function grew an argument, update install function

r308535: Add support for LOADER_RC setting in the pkgfs manifes

r308776: loader: zfs toplevel vdev must have spa set.

r308843: loader: smbios version check is not correct

r310236: Renumber license clauses to avoid skipping #3

r310726: cdboot: add explict suffix to ambiguous or instruction

PR: 214375

MFC r303555,r303556,r303936,r303962,r304317,r304532,r305026,r305107,r305132,
r305178,r305353,r305814,r306159,r306380,r306504: Loader fixes, 2016q3

r303555: bcache should support reads shorter than sector size

r303556: Improve boot loader quote parsing

r303936: Add kernel environment variables under smbios.system

r303962: Add the missing space between .asciz directive and opening quote
for some lines with #ifdef BTXLDR_VERBOSE/#endif

r304317: boot1.efi Free() should check for NULL to provide consistent
behavior

r304532: Replace sprintf -> snprintf for command_errbuf provisioned from
dynamic content.

r305026: Emulate efi_cons_poll when WaitForKey is not available

r305107: Create a hook 'post-initialize' for people that want to define
something to read in .conf files after all other .conf files for the purpose
of overriding.

r305132: Remove accidentally committed stray comment.

r305178: bd_int13probe() should check extended info if sector info is bad

r305353: Don't use -N to set the OMAGIC with data and text writeable and
data not page aligned.

r305814: ufsread: Do not cast struct direct from void *

r306159: Consistently declare getsecs(void) with proper return type and void
when no arguments are present.

r306380: loader command interpreter should reset command_errmsg

r306504: Fix a cluster of bugs in list EFI environment variables

PR: 204602, 211958, 211958
Relnotes: yes ("Improve boot loader quote parsing")

MFC r308434, 308827

MFC r308434: Loader paged/pageable data is not always paged.
MFC r308827: lsdev device name section headers should be printed by dv_print
callback.

MFC r327822:
Skip IRELATIVE relocations when loader processes ELF files.

MFC r324703: loader.mk: clean md.o even if MD_IMAGE_SIZE not defined

We don't normally provide special handling for optionally-included src
files, but md.o depends on both md.c and the value of ${MD_IMAGE_SIZE}.
Previously if one built with MD_IMAGE_SIZE, executed "make clean", and
then built with a different MD_IMAGE_SIZE md.o would not be rebuilt.

Reported by: Zakary Nafziger
Sponsored by: The FreeBSD Foundation

MFC r322628: Fix BSD label partition end sector calculation.

Differential Revision: https://reviews.freebsd.org/D12066

MFC r316102:

Wrap bootcamp DEBUG statement with curly braces

This fixes a -Wempty-body warning with gcc 6.3.0 when PART_DEBUG is undefined.

Tested with: amd64-gcc-6.3.0 (devel/amd64-xtoolchain-gcc)

MFC r316076:

sys/boot/common: Make geli(4) support optional in MI sources

This saves a negligible amount of memory for non-geli enabled
bootloaders.

MFC r316754: loader/multiboot: fix multiboot loading

Sponsored by: Citrix Systems R&D

MFC r316108:

Remove -Wunused-but-set variable, `tail` in `ls_getdir(..)`

This variable has been unused since its inception in r40106.

MFC r313982, r314068:
sys: Replace zero with NULL for pointers.

Found with: devel/coccinelle

MFC r309369,310850,310853:

libstand: dosfs cstyle cleanup for return keyword.
dosfs support in libstand is broken since r298230

PR: 214423
Submitted by: Mikhail Kupchik
Reported by: Mikhail Kupchik
Approved by: imp (mentor)

MFC r311929:

Don't include <errno.h> in reloc_elf.c, as it includes <stand.h> just
after it, which has a conflicting definition of errno. This leads to
the following warning with clang 4.0.0:

In file included from sys/boot/common/reloc_elf32.c:6:
In file included from sys/boot/common/reloc_elf.c:37:
/usr/obj/usr/src/tmp/usr/include/stand.h:155:12: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes]
extern int errno;
^
sys/sys/errno.h:46:26: note: expanded from macro 'errno'
#define errno (* __error())
^

Add WITH_REPRODUCIBLE_BUILD src.conf(5) knob

MFC r310128: Add WITH_REPRODUCIBLE_BUILD src.conf(5) knob to disable kernel
metadata

The kernel builds reproducibly, except for the time, date, user, and
hostname baked into the kernel (reported at startup and via the
kern.version sysctl for uname). Add a build knob to disable the
inclusion of this metadata.

MFC r310268: Build loaders reproducibly when WITH_REPRODUCIBLE_BUILD

When WITH_REPRODUCIBLE_BUILD=yes is set in src.conf(5), eliminate the
time, user, and host from the loader's version information. This allows
builds to produce bit-for-bit identical output.

MFC r310267: Deduplicate loader vers.c Makefile rules

The Makefile rule to create vers.c for loader version info was
previously duplicated in each of the various loader Makefiles.
Instead, share a common rule in Makefile.inc.

Sponsored by: The FreeBSD Foundation

MFC r310225: Reduce boot loader version string duplication

Instead of repeating "%s, Revision %s" "(%s %s)" in each loader, just
create the full version string in vers.c

Fix EFI self relocation code for rela architectures

MFC r306812 (andrew):

The bootloader self relocation code was slightly wrong for the
R_AARCH64_RELATIVE relocation found on arm64. It would try to add the
contents of the memory location being relocated to the base address and
the relocation addend. This worked when the contents was zero, however
this now seems to be set to the value of the addend so we add this twice.
Fix this by just setting the memory to the computed value.

MFC r309360: EFI loaders: parse rela relocations on amd64

Prior to this change the loader self relocation code interpreted amd64's
rela relocations as if they were rel relocations, discarding the addend.
This "works" because GNU ld 2.17.50 stores the addend value in both the
r_addend field of the relocation (as expected) and at the target of the
relocation.

Other linkers, and possibly other versions of GNU ld, won't have this
behaviour, so interpret the relocations correctly.

MFC r307238:

Stop closing the network device when netbooting for loaders using the common
dev_net.c code.

The NETIF_OPEN_CLOSE_ONCE flag was added in r201932 to prevent that behaviour
on some architectures (sparc64 and powerpc64) the default was left to always
open and close the device for each open and close of a file by the loader
because it was necessary for u-boot on arm.

Since it has been added, the flag was turned on for every arches including the
u-boot loader for arm.

This also fixes netbooting on RPi3 (tested by gonzo@)

For the loader.efi it greatly speeds up netbooting

Reviewed by: emaste, gonzo, tsoome
Approved by: gonzo
MFC after: 1 month
Sponsored by: Gandi.net
Differential Revision: https://reviews.freebsd.org/D8230

MFC: 304677, 304680, 305125, 305770, 305769

r304677:
Add tftpfs support for the EFI loader

Allow netbooting on efi without having to setup any NFS server by rebuilding the
loader with LOADER_TFTP_SUPPORT like for the i386 pxeloader

r305125:
Netboot: allow both tftpfs and nfs in both pxeboot and loader.efi

Add a new 'netproto' variable which can be set for now to
NET_TFTP or NET_NFS (default to NET_NONE)

From the dhcp options if one sets the root-path option to:
"ip:path", the loader will consider it is booting over NFS
(meaning same behaviour as the default current behaviour)

if the dhcp option "tftp server address" is set (option 150)
the loader will consider it is booting over tftpfs, it will then
consider the root-path options with 2 possible case
1. "path" then the IP of the tftp server will be the one passed by
the option 150, and the files will be retrieved under "path" on the tftp
server
2. "ip:path" then the IP of the tftp server will be the one passed in
the option "overwritting the IP from the option 150.

We could not "abuse" the rootpath option in the form or tftp://ip:path because
this is already used for other purpose by iPXE preventing any chainload from
iPXE to the FreeBSD loader.

Given at each open(), the loader loops over all available filesystems and keep
the "best" error, we needed to prevent tftpfs to fallback on nfs and vice versa.
the tftpfs and nfs implementation in libstand now return EINVAL early if
'netproto' for that purpose.

Relnotes: yes
Sponsored by: Gandi.net

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Fix typo.

Spotted by: Matteo Riondato

Also add comment about the bug I comments in the forth.

Fix several instances where the boot loader ignored pager_output
return value when it could return 1 (indicating we should stop).
Fix a few instances of pager_open() / pager_close() not being called.
Actually use these routines for the environment variable printing code
I just committed.

It sure would be nice to use printf with wide strings. Implement %S to
do that. The C_WIDEOUT flag indicates that the console supports
it. Mark the EFI console as supporting this.

MFC After: 3 days

sys/boot/common: use of spaces vs. TAB.

No functional change.

bcache read ahead may attempt to read past end of disk

The new bcache code does not know the size of the disk, and therefore may attempt to read past the end of the disk while trying to fill its read-ahead cache.

This is usually not an issue, it fails gracefully on all of my machines, but some BIOSes seem to retry the reads for up to 30 seconds each, resulting in a long stall during boot

Submitted by: Toomas Soome <tsoome@me.com>
Reviewed by: jhb, np
Differential Revision: https://reviews.freebsd.org/D6109

sys/boot: make use of the howmany() macro when available.

We have a howmany() macro in the <sys/param.h> header that is
convenient to re-use as it makes things easier to read.

sys: use our roundup2/rounddown2() macros when param.h is available.

rounddown2 tends to produce longer lines than the original code
and when the code has a high indentation level it was not really
advantageous to do the replacement.

This tries to strike a balance between readability using the macros
and flexibility of having the expressions, so not everything is
converted.

sys/boot: use our nitems() macro when it is available through param.h.

No functional change, only trivial cases are done in this sweep,

Discussed in: freebsd-current

A new implementation of the loader block cache

The block cache implementation in loader has proven to be almost useless, and in worst case even slowing down the disk reads due to insufficient cache size and extra memory copy.
Also the current cache implementation does not cache reads from CDs, or work with zfs built on top of multiple disks.
Instead of an LRU, this code uses a simple hash (O(1) read from cache), and instead of a single global cache, a separate cache per block device.
The cache also implements limited read-ahead to increase performance.
To simplify read ahead management, the read ahead will not wrap over bcache end, so in worst case, single block physical read will be performed to fill the last block in bcache.

Booting from a virtual CD over IPMI:
0ms latency, before: 27 second, after: 7 seconds
60ms latency, before: over 12 minutes, after: under 5 minutes.

Submitted by: Toomas Soome <tsoome@me.com>
Reviewed by: delphij (previous version), emaste (previous version)
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D4713

Cleanup unnecessary semicolons from the kernel.

Found with devel/coccinelle.

Use the ABI-prescribed name for SHT_X86_64_UNWIND in the loader and
kernel linker, after the r297686.

Sponsored by: The FreeBSD Foundation

If the dhcp server provided an interface-mtu option, transcribe the value
to the boot.netif.mtu env var, which will be picked up by pre-existing code
in nfs_mountroot() and used to configure the interface accordingly.

This should bring the same functionality when the bootp/dhcp work is done
by loader(8) as r297150 does for the in-kernel BOOTP case.

Implement GELI (AES-XTS and AES-CBC only) in gptboot and gptzfsboot

Allows booting from a GELI encrypted root file system, via UFS or ZFS

Reviewed by: gnn, smh (previous version), delphij (previous version)
Relnotes: yes
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D4593

Since kernel modules can now contain sections of type SHT_AMD64_UNWIND,
the boot loader should not skip over these anymore while loading images.
Otherwise the kernel can still panic when it doesn't find the .eh_frame
section belonging to the .rela.eh_frame section.

Unfortunately this will require installing boot loaders from sys/boot
before attempting to boot with a new kernel.

Reviewed by: kib
MFC after: 2 weeks
X-MFC-With: r296419

Fix ubldr build failure on mipsn32 and mipsn32el targets.

Approved by: adrian (mentor)

Fix build failure introduced by r296182

Approved by: adrian (mentor)

These changes attempt to put things in order before the introduction of MIPS
ubldr.

The changes are mostly dealing with removing unnecessary casts from the U-Boot
API (we're passing only pointers, no obvious reason to cast them to uint32_t),
cleaning up some compiler warnings and using the proper printf format
specifiers in order to be able to compile cleanly for both 32-bit and 64-bit
MIPS targets.

Reviewed by: imp
Approved by: adrian (mentor)
Sponsored by: Smartcom - Bulgaria AD
Differential Revision: https://reviews.freebsd.org/D5312

Don't set the offset when loading the kernel on the arm loader.efi. The
copyin and copyout code handle virtual addresses such that they will take
a virtual address and convert it into a valid physical address. It may
also mean we fail to boot as the elf files load address could be 0.

Sponsored by: ABT Systems Ltd

Fix EFI platform build failures

With warnings now enabled some plaforms where failing due to warnings.
* Fix st_size printed as a size_t when its actually an off_t.
* Fix pointer conversion in load_elf for some 32bit platforms due to 64bit
off in ef.

MFC after: 2 days
X-MFC-With:
Sponsored by: Multiplay

Fix mistake when transitioning to the new defines with ZFS loader. I
hate adding yet another define, but it is the lessor of the evil
choices available. Kill another evil by removing PATH_BOOT3 and
replacing it with PATH_LOADER or PATH_LOADER_ZFS as appropriate.

PR: 206659

RBX_ defines are in rbx.h, move it there.

Differential Revision: https://reviews.freebsd.org/D5038

Move all the separate copies of the same strings into paths.h. There's
nothing machine specific about these.

Differential Revision: https://reviews.freebsd.org/D5038

Prevent loader.conf load failure due to unknown console entries

When processing loader.conf if console contained an entry for an unsupported
console then cons_set would return an error refusing to set any console.

This has two side effects:

1. Forth would throw a syntax error and stop processing loader.conf at that
point.
2. The value of console is ignored.

#1 Means other important loader.conf entries may not be processed, which is
clearly undesirable.
#2 Means the users preference for console aren't applied even if they did
contain valid options. Now we have support for multi boot paths from a
single image e.g. bios and efi mode the console preference needs to deal
with the need to set preference for more than one source.

Fix this by:
* Returning CMD_OK where possible from cons_set.
* Allowing set with at least one valid console to proceed.

Reviewed by: allanjude
MFC after: 1 week
Sponsored by: Multiplay
Differential Revision: https://reviews.freebsd.org/D5018

Fix EFI UFS caching

EFI was mixing caching in two separate places causing issues when multiple
partitions where tested.

Eliminate this by removing fsstat and re-factoring fsread into fsread_size,
adding basic parameter validation.

Also:
* Enhance some error print outs.
* Fix compilation under UFS1_ONLY and UFS2_ONLY
* Use sizeof on vars instead of structs.
* Add basic parameter validation to fsread_size.

MFC after: 1 week
X-MFC-With: r293268
Sponsored by: Multiplay
Differential Revision: https://reviews.freebsd.org/D4989

Ensure boot fsread correctly probes all partitions

The boot code fsread was caching the result of meta data request and
reusing it even for calls with inode = 0, which is used to partitions
trigger a probe.

The result was that success was incorrectly returned for all partition
probes after the first valid success, even for partitions which are not
UFS.

MFC after: 2 weeks
X-MFC-With: r293268
Sponsored by: Multiplay

Make common boot file_loadraw name parameter const

Fix compiler warnings about dropping const qualifier by changing file_loadraw
name param to const, and updating method to make that the case (it was
abusing the variable).

MFC after: 2 weeks
X-MFC-With: r293268
Sponsored by: Multiplay

Improve non-interactive forth cmd error reporting

Non-interactive forth command errors where silent even for critical issues
e.g. failing to load a required kernel module or mfs_root.

This resulted in later unexplained and hard to trace errors such as mount
root failures.

This introduces additional command return codes that are treated
appropriately by the non-interactive command processor (bf_command).
* CMD_CRIT = print error
* CMD_FATAL = panic

Also fix minor style(9) issues with command_load return codes.

MFC after: 2 weeks
X-MFC-With: r293268
Sponsored by: Multiplay

Enable warnings in EFI boot code

Set WARNS if not set for EFI boot code and fix the issues highlighted by
setting it.

Most components are set to WARNS level 6 with few being left at lower
levels due to the amount of changes needed to fix at higher levels.

Error types fixed:
* Missing / invalid casts
* Missing inner structs
* Unused vars
* Missing static for internal only funcs
* Missing prototypes
* Alignment changes
* Use of uninitialised vars
* Unknown pragma (intrinsic)
* Missing types etc due to missing includes
* printf formatting types

Reviewed by: emaste (in part)
MFC after: 2 weeks
X-MFC-With: r293268
Sponsored by: Multiplay
Differential Revision: https://reviews.freebsd.org/D4839

Remove hidden "Not ufs" printfs from boot code

Remove the printf("Not ufs\n") from the boot code which was hidden by the
local printf implementations, allowing these to have that code removed too.

MFC after: 2 weeks
X-MFC-With: r293268
Sponsored by: Multiplay

Make detection of GPT a bit more reliable.

When we are detecting a partition table and didn't find PMBR, try to
read backup GPT header from the last sector and if it is correct,
assume that we have GPT.

Reviewed by: rpokala
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D4282

META MODE: Update dependencies with 'the-lot' and add missing directories.

This is not properly respecting WITHOUT or ARCH dependencies in target/.
Doing so requires a massive effort to rework targets/ to do so. A
better approach will be to either include the SUBDIR Makefiles directly
and map to DIRDEPS or just dynamically lookup the SUBDIR. These lose
the benefit of having a userland/lib, userland/libexec, etc, though and
results in a massive package. The current implementation of targets/ is
very unmaintainable.

Currently rescue/rescue and sys/modules are still not connected.

Sponsored by: EMC / Isilon Storage Division

Implement simple ops for umass_disk

The initial IOCTL implementation supports reading disk physical
geometry.
Two additional functions were added. They allow reading/writing raw
data to the disk (default partition).

Submitted by: Wojciech Macek <wma@semihalf.com>
Obtained from: Semihalf
Sponsored by: Juniper Networks Inc.
Differential Revision: https://reviews.freebsd.org/D4143

Document loader(8) dumpdev option

Add an entry for dumpdev environment variable to loader(8).

MFC after: 1 week
Sponsored by: Multiplay

Make vers.c creation atomic by using a temporary file, then moving
the temporary file to vers.c at the end of the script

The previous logic wrote out to vers.c multiple times, so the file
could be incorrectly interpreted as being completely written out
after one of the echo calls with recursive make, when in reality it
was only partially written.

Also, in the event the build was interrupted when creating vers.c
(small race window), it would have a leftover file that needed to
be cleaned up before resuming the build.

MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division

Fix a problem which made loader(8) load non-kld files twice.

For example, without this patch, the following three lines
in /boot/loader.conf would result in /boot/root.img being preloaded
twice, and two md(4) devices - md0 and md1 - being created.

initmd_load="YES"
initmd_type="md_image"
initmd_name="/boot/root.img"

Reviewed by: marcel@
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3204

Document md_root in loader(8). The md(4) manual page mentions it,
but it's hard to find and easy to miss.

Reviewed by: wblock@
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3183

Install loader(8) and zfsloader(8) only once

Differential Revision: https://reviews.freebsd.org/D2841
Reviewed by: imp

Refactor net_getparams() to make it easier to get params from sources other
than bootp and rarp.

The code which splits a serverip:/rootpath string into rootip and a plain
pathname is now a separate net_parse_rootpath() function that can be
called by others. The code that sets the kernel env vars needed for
nfs_diskless is moved into net_open() so that the variables get set no
matter where the params came from.

There was already code in net_open() that allowed for the possibility that
some other entity has set up the network-related global variables. It uses
the rootip variable as the key, assuming that if it is set all the other
required variables are set too. These changes don't alter the existing
behavior, they just make it easier to actually write some new code to get
the params from another source (such as the U-Boot environment).

An ARM kernel can be loaded at any 2MB boundary, make ubldr aware of that.

Previously, ubldr would use the virtual addresses in the elf headers by
masking off the high bits and assuming the result was a physical address
where the kernel should be loaded. That would sometimes discard
significant bits of the physical address, but the effects of that were
undone by archsw copy code that would find a large block of memory and
apply an offset to the source/dest copy addresses. The result was that
things were loaded at a different physical address than requested by the
higher code layers, but that worked because other adjustments were applied
later (such as when jumping to the entry point). Very confusing, and
somewhat fragile.

Now the archsw copy routines are just simple copies, and instead
archsw.arch_loadaddr is implemented to choose a load address. The new
routine uses some of the code from the old offset-translation routine to
find the largest block of ram, but it excludes ubldr itself from that
range, and also excludes If ubldr splits the largest block of ram in
two, the kernel is loaded into the bottom of whichever resulting block is
larger.

As part of eliminating ubldr itself from the ram ranges, export the heap
start/end addresses in a pair of new global variables.

This change means that the virtual addresses in the arm kernel elf headers
now have no meaning at all, except for the entry point address. There is
an implicit assumption that the entry point is in the first text page, and
that the address in the the header can be turned into an offset by masking
it with PAGE_MASK. In the future we can link all arm kernels at a virtual
address of 0xC0000000 with no need to use any low-order part of the
address to influence where in ram the kernel gets loaded.

The self-relocation code is not efi-specific, move it to boot/common.

The function was defined as taking 4 parameters and returning EFI_STATUS,
but all existing callers (in asm code) passed only two parameters and don't
use the return value. The function signature now matches that usage, and
doesn't refer to efi-specific types.

Parameters and variables now use the cannonical typenames set up by elf.h
(Elf_Word, Elf_Addr, etc) instead of raw C types. Hopefully this will
prevent suprises as new platforms come along and use this code.

The function was renamed from _reloc() to self_reloc() to emphasize its
difference from the other elf relocation code found in boot/common.

Differential Revision: https://reviews.freebsd.org/D2490

Small change in header order to allow this to compile.

Obtained from: Netflix, Inc.
MFC after: 3 days

Add support for arm64 to loader.efi and boot1.efi

Reviewed by: emaste
Sponsored by: The FreeBSD Foundation

Make sure forth manpages are only installed once.

Differential Revision: https://reviews.freebsd.org/D2224
Reviewed by: imp

Add code to support loading relocatable kernels at offsets that are not
zero.

Add support for booting relocatable kernels on PowerPC.

loader: use correct types for parse_modmetadata

Use the proper types in parse_modmetadata for the p_start and p_end
parameters. This was causing problems in the ARM 32bit loader.

Sponsored by: Citrix Systems R&D
Reported and Tested by: ian

loader: implement multiboot support for Xen Dom0

Implement a subset of the multiboot specification in order to boot Xen
and a FreeBSD Dom0 from the FreeBSD bootloader. This multiboot
implementation is tailored to boot Xen and FreeBSD Dom0, and it will
most surely fail to boot any other multiboot compilant kernel.

In order to detect and boot the Xen microkernel, two new file formats
are added to the bootloader, multiboot and multiboot_obj. Multiboot
support must be tested before regular ELF support, since Xen is a
multiboot kernel that also uses ELF. After a multiboot kernel is
detected, all the other loaded kernels/modules are parsed by the
multiboot_obj format.

The layout of the loaded objects in memory is the following; first the
Xen kernel is loaded as a 32bit ELF into memory (Xen will switch to
long mode by itself), after that the FreeBSD kernel is loaded as a RAW
file (Xen will parse and load it using it's internal ELF loader), and
finally the metadata and the modules are loaded using the native
FreeBSD way. After everything is loaded we jump into Xen's entry point
using a small trampoline. The order of the multiboot modules passed to
Xen is the following, the first module is the RAW FreeBSD kernel, and
the second module is the metadata and the FreeBSD modules.

Since Xen will relocate the memory position of the second
multiboot module (the one that contains the metadata and native
FreeBSD modules), we need to stash the original modulep address inside
of the metadata itself in order to recalculate its position once
booted. This also means the metadata must come before the loaded
modules, so after loading the FreeBSD kernel a portion of memory is
reserved in order to place the metadata before booting.

In order to tell the loader to boot Xen and then the FreeBSD kernel the
following has to be added to the /boot/loader.conf file:

xen_cmdline="dom0_mem=1024M dom0_max_vcpus=2 dom0pvh=1 console=com1,vga"
xen_kernel="/boot/xen"

The first argument contains the command line that will be passed to the Xen
kernel, while the second argument is the path to the Xen kernel itself. This
can also be done manually from the loader command line, by for example
typing the following set of commands:

OK unload
OK load /boot/xen dom0_mem=1024M dom0_max_vcpus=2 dom0pvh=1 console=com1,vga
OK load kernel
OK load zfs
OK load if_tap
OK load ...
OK boot

Sponsored by: Citrix Systems R&D
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D517

For the Forth bits:
Submitted by: Julien Grall <julien.grall AT citrix.com>

Reserve and ignore the a new module metadata type MDT_PNP_INFO for
associating an optional PNP hint table with this module. In the
future, when these are added, these changes will silently ignore the
new type they would otherwise warn about. It will always be safe to
ignore this data. Get this into the builds today for some future
proofing.

MFC After: 3 days

mdoc: remove EOL whitespace.

Use the proper markup for single quotes.

Add a new loader(8) variable, twiddle_divisor, allowing control over the
output frequency of the "twiddle" IO progress indicator. The default
value is 1. For larger values N, the next stage of the animation is only
output on every Nth call to the output routine. A sufficiently large N
effectively disables the animation completely.

The current limit of 100k for the linker hints file is getting a bit
crowded as we now are at about 70k. Bump the limit to 1MB instead
which is still quite a reasonable limit and allows for future growth
of this file and possible future expansion to additional data.

MFC After: 2 weeks

Misc mdoc fixes:

- Remove superfluous paragraph macros.
- Remove/fix empty or incorrect macros.
- Sort sections into conventional order.
- Terminate quoted strings properly.
- Remove EOL whitespace.

Fix incorrect reading of 32-bit modinfo by 64-bit loaders.

The various structures in the mod_metadata set of a FreeBSD kernel and
modules contain pointers. The FreeBSD loader correctly deals with a
mismatch in loader and kernel pointer size (e.g. 32-bit i386/ppc
loader, loading 64-bit amd64/ppc64 kernels), but wasn't dealing with
the inverse case where a 64-bit loader was loading a 32-bit kernel.

Reported by: ktcallbox@gmail.com with a bhyve/i386 and ZFS root install
Differential Revision: https://reviews.freebsd.org/D1129
Reviewed by: neel, jhb
MFC after: 1 week

In alloc_pread() and kern_pread(), print errors only when DEBUG is
defined. An error is not fatal and is supposed to be handled by the
caller.

Obtained from: Juniper Networks, Inc.

Change the order of the arguments to file_loadraw(). They were swapped
as of r262345 when file_loadraw() was made public and this little detail
got overlooked during porting.

Obtained from: Juniper Networks, Inc.

Fully support constructors for the purpose of code coverage analysis.
This involves:
1. Have the loader pass the start and size of the .ctors section to the
kernel in 2 new metadata elements.
2. Have the linker backends look for and record the start and size of
the .ctors section in dynamically loaded modules.
3. Have the linker backends call the constructors as part of the final
work of initializing preloaded or dynamically loaded modules.

Note that LLVM appends the priority of the constructors to the name of
the .ctors section. Not so when compiling with GCC. The code currently
works for GCC and not for LLVM.

Submitted by: Dmitry Mikulin <dmitrym@juniper.net>
Obtained from: Juniper Networks, Inc.

Fix comment.

MFC after: 1 week

Rework bootparttest to use more code from sys/boot.
Use disk_open() call to emulate loader behavior.

Add a bit more debug messages.

Add GUID of FreeBSD slice to GPT scheme.

MFC after: 1 week

add gptzfsboot.8, zfsboot.8 and zfsloader.8 manual pages

Many thanks to Warren Block for his reviews, corrections and additions.

Reviewed by: Warren Block <wblock@FreeBSD.org>
MFC after: 1 week