MFC r347142:

loader: validate sectorsize argument in disk_open()

The bug and patch is reported against 11.2, but it is good idea to have
the check in place for all versions.

PR: 236585
Submitted by: john@feith.com
Reported by: john@feith.com

MFC r343911, r344238-r344241, r344247, r344254-r344255, r344260, r344268,
r344335, r344839, r345066, r345330

r343911:
Allow reading the UEFI variable size

When loading bigger variables form UEFI it is necessary to know their
size beforehand, so that an appropriate amount of memory can be
allocated. The easiest way to do this is to try to read the variable
with buffer size equal 0, expecting EFI_BUFFER_TOO_SMALL error to be
returned. Allow such possible approach in efi_getenv routine.

Extracted from a bigger patch as suggested by imp.

r344238:
Restore loader(8)'s ability for lsdev to show partitions within a bsd slice.

I'm pretty sure this used to work at one time, perhaps long ago. It has
been failing recently because if you call disk_open() with dev->d_partition
set to -1 when d_slice refers to a bsd slice, it assumes you want it to
open the first partition within that slice. When you then pass that open
dev instance to ptable_open(), it tries to read the start of the 'a'
partition and decides there is no recognizable partition type there.

This restores the old functionality by resetting d_offset to the start
of the raw slice after disk_open() returns. For good measure, d_partition
is also set back to -1, although that doesn't currently affect anything.

I would have preferred to make disk_open() avoid such rude assumptions and
if you ask for partition -1 you get the raw slice. But the commit history
shows that someone already did that once (r239058), and had to revert it
(r239232), so I didn't even try to go down that road.

r344239:
Use a couple local variables to avoid repetitive long expressions that
cause line-wrapping.

r344240:
Make lsdev -v output line up in neat columns by using a fixed width for
the size field and a tab between the partition type and the size.

Changes this

disk devices:
disk0 (MMC)
disk0s1: DOS/Windows 49MB
disk0s2: FreeBSD 14GB
disk0s2a: FreeBSD UFS 14GB
disk0s2b: Unknown 2048KB
disk0s2d: FreeBSD UFS 2040KB

to this

disk devices:
disk0 (MMC)
disk0s1: DOS/Windows 49MB
disk0s2: FreeBSD 14GB
disk0s2a: FreeBSD UFS 14GB
disk0s2b: Unknown 2048KB
disk0s2d: FreeBSD UFS 2040KB

r344241:
Garbage collection no-longer-used constant.

r344247:
Make uboot_devdesc properly alias disk_devdesc, so that parsing the u-boot
loaderdev variable works correctly.

The uboot_devdesc struct is variously cast back and forth between
uboot_devdesc and disk_devdesc as pointers are handed off through various
opaque interfaces. uboot_devdesc attempted to mimic the layout of
disk_devdesc by having a devdesc struct, followed by a union of some
device-specific stuff that included a struct that contains the same fields
as a disk_devdesc. However, one of those fields inside the struct is 64-bit
which causes the entire union to be 64-bit aligned -- 32 bits of padding
is added between the struct devdesc and the union, so the whole mess ends
up NOT properly mimicking a disk_devdesc after all. (In disk_devdesc there
is also 32 bits of padding, but it shows up immediately before the d_offset
field, rather than before the whole collection of d_* fields.)

This fixes the problem by using an anonymous union to overlay the devdesc
field uboot network devices need with the disk_devdesc that uboot storage
devices need. This is a different solution than the one contributed with
the PR (so if anything goes wrong, the blame goes to me), but 95% of the
credit for this fix goes to Pawel Worach and Manuel Stuhn who analyzed the
problem and proposed a fix.

r344254:
Use DEV_TYP_NONE instead of -1 to indicate no device was specified.

DEV_TYP_NONE has a value of zero, which makes more sense since the device
type is a bunch of bits describing the device, crammed into an int.

r344255:
Fix more places to use DEV_TYP_NONE instead of -1 to indicate 'no device'.

r344260:
Allow the u-boot loaderdev env var to be formatted in the "usual" loader(8)
way: device<unit>[s|p]<slice><partition>. E.g., disk0s2a or disk3p12.
The code first tries to parse the variable in this format using the
standard disk_parsedev(). If that fails, it falls back to parsing the
legacy format that has been supported by ubldr for years.

In addition to 'disk', all the valid uboot device names can also be used:
mmc, sata, usb, ide, scsi. The 'disk' device serves as an alias for all
those types and will match the Nth storage-type device found (where N is
the unit number).

r344268:
loader: ptable_close() should check its argument

If the passed in table is NULL, just return.

r344335:
Fix the handling of legacy-format devices in the u-boot loaderdev variable.
When I added support for the standard loader(8) disk0s2a: type formats,
the parsing of legacy format was broken because it also contains a colon,
but it comes before the slice and partition. That would cause disk_parsedev()
to return success with the slice and partition set to wildcard values.

This change examines the string first, and if it contains spaces, dots, or
a colon at any position other than the end, it must be a legacy-format
string and we don't even try to use disk_parsedev() on it.

r344839:
Add retry loop around GetMemoryMap call to fix fragmentation bug

The call to BS->AllocatePages can cause the memory map to become framented,
causing BS->GetMemoryMap to return EFI_BUFFER_TOO_SMALL more than once. For
example this can happen on the MinnowBoard Turbot, causing the boot to stop
with an error. Avoid this by calling GetMemoryMap in a loop.

r345066:
stand: Improve some debugging experience

Some of these files using <FOO>_DEBUG defined a DEBUG() macro to serve as a
debug-printf. -DDEBUG is useful to enable some debugging output across
multiple ELF/common parts, so switch the DEBUG-as-printf macros over to
something more like DPRINTF that is more commonly used for this kind of
thing and less likely to conflict.

userboot/elf64_freebsd debugging also assumed %llx for uint64; use PRIx64
instead.

r345330:
loader: fix loading of kernels with . in path

The loader indended to search the kernel file name (only) for . but
instead searched the entire path, so paths like
"boot/test.elfv2/kernel" would not work.

PR: 233097

MFC r339658, r339959, r340047, r340049, r340215

r339658:
loader: biosdisk interface should be able to cope with 4k sectors

The 4kn support in current bios specific biosdisk.c is broken, as the code
is only implementing the support for the 512B sector size.

This work is building the support for custom size sectors, we still do assume
the requested data to be multiple of 512B blocks and we only do address the
biosdisk.c interface here.

For reference, see also:
https://www.illumos.org/issues/8303
https://www.illumos.org/rb/r/547

As the GELI is moved above biosdisk "layer", the GELI should just work

r339959:
loader: issue edd probe before legacy ah=08 and detect no media

while probing for drives, use int13 extended info before standard one and
provide workaround for case we are not getting needed information in case
of floppy drive.

In case of INT13 errors, there are (at least) 3 error codes appearing in case
of missin media - 20h, 31h and 80h. Flag the no media and do not print an
error.

r340047:
loader: do not probe floppy devices for zfs

The subject is telling it all.

r340049:
loader: biosdisk should check if the media is present

The bd_print/bd_open/bd_strategy need to make sure the device does have
media, before getting into performing IO operations. Some systems can
hung if the device without a media is accessed.

r340215:
loader: always set media size from partition.

The disk access is validated by using partition table definitions, therefore
we have no need for if statements, just set the disk size.

Of course the partition table itself may be incorrect/inconsistent, but if
so, we are in trouble anyhow.

MFC r335245-r335248, r335254, r335276, r335298, r335398, r335868, r335883

r335245:
Correct logic error in biosdisk.c:bd_realstrategy()

The wrong condition is used when evaluating the return of disk_ioctl()
This results in reaching the 'We should not get here' branch in most casts

r335246:
biosdisk.c remove redundant variable

`rdev` and `disk` serve the same purpose, read the partition table without
the `d_offset` or `d_slice` set, so the read is relative to the start of
the disk. Reuse the already initialized `disk` instead of making another
copy later.

r335247:
biosdisk.c: add missing \n to error message

r335248:
biosdisk.c: fix type in debug printf

r335254:
Avoid reading past the end of the disk in zfsboot.c and biosdisk.c

The GELI boot code rounds reads up to 4k, since the encrypted sectors are
4k, and must be decrypted as a unit. With oddball sized disks (almost
always virtual), this can lead to reading past the end of the disk.

r335276:
gptboot, zfsboot, gptzfsboot: Enable the video and serial consoles early

Normally the serial console is not enabled until /boot.config is read and
we know how the serial console should be configured. Initialize the
consoles early in 'dual' mode (serial & keyboard) with a default serial
rate of 115200. Then serial is re-initialized once the disk is decrypted
and the /boot.config file can be read.

This allows the GELIBoot passphrase to be provided via the serial console.

r335298:
stand/common/disk.c: Update debug printf

This was missed in r330809 because it is compiled out by default

r335398:
Revert r335276

This was causing issues for people booting.
I will likely bring this back as an optional feature, similar to
boot0sio, like gptboot-serial or something.

r335868:
stand/common/disk.c: Read partition table relative to the start of the disk

If a disk is of an oddball size, like the 200mb + 512b used in rootgen.sh,
when disk_open() is called on a GELI encrypted partition, attempts to read
the partition table fail, as they pass through the decryption process which
turns the already plaintext data into jibberish.

When reading the partition table, always pass a slice and partition setting
of -1, and an offset of 0. Setting the slice to -1 prevents a false
positive when checking the slice against the cache of GELI encrypted
slices.

r335883:
stand/common/disk.c: dev->d_offset still needs to be set to 0

With r335868, I thought this was no longer necessary. I was wrong.

MFC r332085

Add an ISO9660 "partition table" type to loader.

When booted via isoboot(8) loader will be handed a disk that simply contains
an ISO9660 image. Currently this confuses it greatly. Teach it how to spot
that it's in this situation and that ISO9660 has one "partition" covering
the whole disk.

Sponsored by: iXsystems, Inc.

MFC r330806-r330815, r330837, r330864, r330883

pc98 changes associated with the named commits are also included in this
commit, despite not having been made with the original commits due to its
removal in head.

r330806:
Minor cosmetic changes.

Make sure { on the same line as struct for all struct *devdesc. Move
some type definitions to next to the dv_type define, since that's what
sets the d_type.

r330807:
We can't use d_opendata for blkio storage.

open_disk uses d_opendata for it's own purpse. We can't store blkio
there. Fortunately, blkio is stored elsewhere and we never actually
retrieve blkio from d_opendata. Eliminate it as a source of confusion.
Eliminate all stores of d_opendata in efi since this layer doesn't own
that field.

r330808:
Make struct libi386_devdesc match the struct devdesc better

Move data to top and call it d_opendata.

r330809:
Use the actual struct devdesc at the start of all *_devdesc structs

The current system is fragile and requires very careful layout of all
*_devdesc structures. It also makes it hard to change the base
devdesc. Take a page from CAM and put the 'header' in all the derived
classes and adjust the code to match.

For OFW, move the iHandle h_handle out of a slot conflicting with
d_opendata. Due to quirks in the alignment rules, this worked.
However changing the code to use d_opendata storage now that it's a
pointer is hard, so just have a separate field for it.

All other cleanups were to make the *_devdesc structures match where
they'd taken some liberties that were none-the-less compatible enough
to work.

r330810:
Remove d_type from devdesc. It's not needed as we can fetch it from
d_dev->dv_type when we need it.

r330811:
GC unused routines.

Sponsored by: Netflix

r330812:
Use the one-line-per-file pattern here, and sort the file names.

Sponsored by: Netflix

r330813:
Move the env convenience routines out of boot1.c.

These routines are more generally useful. Even though boot1 is on its
way out, it's better to make these common during the transition than
copy them.

r330814:
Star BootCurrent entry when booting.

Sponsored by: Netflix

r330815:
Print the load and device path as well as BootCurrent and BootOrder

Sponsored by: Netflix

r330837:
biosdisk.c should not set d_opendata.

Same as 330807, d_opendata is owned by open_disk and we should not
set it.

M stand/i386/libi386/biosdisk.c

r330864:
Prefer uintXX_t to u_intXX_t

A foolish consistency is the hobgoblin of little minds, adored by
little statesmen and philosophers and divines. With consistency a
great soul has simply nothing to do. -- Ralph Waldo Emerson

r330883:
Fix typo that misteriously passes compilation.

MFC r325834,r325997,326502: Move sys/boot to stand/

This is effectively a direct commit to stable/11, due to differences between
stable/11 and head. Changes to DTS in sys/boot/fdt/dts were often
accompanied by kernel changes. Many of these were also risc-v updates that
likely had many more dependencies to MFC.

Because of this, sys/boot/fdt/dts remains as-is while everything else in
sys/boot relocates to stand/.

r325834: Move sys/boot to stand. Fix all references to new location

r325997: Remove empty directories.

r326502: Document the sys/boot -> stand move in hier.7 and the top-level README.

MFC Loader Fixes 2017q4p6: r324649,r324650,r324651,r324652,r324653,r324654,
r324700,r324702,r324709,r324717,r324719,r324841,r324842,r324843,r324845,
r324850,r324876,r324877,r324878,r324879,r324880,r324881,r324883,r324980,
r324981,r324982,r324995,r325014,r325093,r325094,r325114,r325170,r325171,
r325172,r325173,r325174,r325175,r325176,r325248,r325286,r325310,r325332,
r325338,r325339,r325376,r325377,r325379,r325380,r325382,r325478,r325479,
r325480,r325482,r325483,r325484,r325485,r325556,r325641,r325681,r325685,
r325686,r325687,r325688,r325689,r325690,r325691,r325692,r325693,r325694,
r325743,r325744,r325748,r325775,r325779,r325780

r324649: Move common/Makefile.inc to sys/boot/loader.mk.

r324650: tweak style

r324651: create defs.mk for common definitions

r324652: Move all the ficl common code into ficl.mk

r324653: LOADER_foo_SUPPORTED

r324654: Explicitly inlcude SYSDIR in the include path

r324700: loader: initialize dv_cleanup in md.c to eliminate clang warning

r324702: We need to include disk.o in libuboot.a when we're building with
support for disk access.

r324709: Revert "Unify boot1 with loader" change r324646

r324717: libsa/ip.c: misplaced comment, ip_v is half char, not ip_p

r324719: libsa/ip: stop read loop on bad fragments

r324841: Use BOOTDIR more consistently in defs.mk rather than repeat
sys/boot.

r324842: Introduce BOOTOBJ: The top level object directory for the boot tree

r324843: Stopgap fix to the mistmatch between LOADER_GELI_SUPPORT and
LOADER_NO_GELI_SUPPORT.

r324845: Use BOOTOBJ and BOOTDIR to find geli includes and libraries.

r324850: Define LIBSA32 to LIBSA on i386 to fix build.

r324876: Move fdt and uboot defines into common uboot.mk.

r324877: End source directories with SRC rather than a hodgepodge of names

r324878: Make at91 boot loader compile again.

r324879: Prefer SRCTOP paths for bits we're grabbing from libc.

r324880: Use BOOTSRC here.

r324881: Use SYSDIR instead of ${.CURDIR}/../..<etc>/sys.

r324883: Use preferred defined paths, rather than relative paths in fdt.

r324980: Use BOOTDIR consistently.

r324981: Move BINDIR definition to defs.mk, and override where it isn't
/boot

r324982: Remove sys/boot/arm/at91 and ixp425

r324995: loader.efi: Make framebuffer commands available for arm64

r325014: Add a 'place holder' arm struct efi_fb until a real one comes

r325093: Define new EFI variables

r325094: Cleanup non-arch Makefiles

r325114: Use defs.mk defins in most MD code

r325170: Use defs.mk values for userboot

r325171: Use defs.mk name and prefer bsd.init.mk

r325172: Remove the -nostdlib stuff I added. Instead, fix LDFLAGS to be
honored correctly with the new Makefile.inc include order.

r325173: We don't need to build a special ficl for userboot.

r325174: Minor cleanup

r325175: For amd64, compile both zfs and zfs32 libraries.

r325176: Actually add zfs32/Makefile

r325248: loader ptblread() is broken with >512B sectors

r325286: efipart_strategy is using wrong offset with >512B sectors

r325310: zfs.c:vdev_read() needs to be careful about large sectors

r325332: loader: re-enable gzip support for x86

r325338: loader: fix BOOTSRC -> BOOTOBJ in a library path

r325339: This used to have bzip2 support too.

r325376: WIP: centralize machine links

r325377: mostly libsa

r325379: Revert "mostly libsa"

r325380: Revert "WIP: centralize machine links"

r325382: Cleanup stray libstand names to be libsa names.

r325478: Powerpc is a 32-bit boot loader.

r325479: Define LIBFICL32 to be libficl.a on i386 and libficl32.a on amd64.

r325480: Use DO32 for all the places that we need to flag we're building a
32-bit version of a library.

r325482: Move machine and other link creation to defs.mk

r325483: MACHINE can never be powerpc64, so cleanup code that thinks it can.

r325484: Prefer bsd.init.mk to src.opts.mk

r325485: Centralize all 32-bit builds on 64-bit platform stuff.

r325556: loader: set options before including bsd.init.mk

r325641: loader.efi: efi_devpath_is_prefix should return bool

r325681: boot1: avoid using NULL device path

r325685: libsa32 isn't needed for i386. It's already a 32-bit platform.

r325686: Simplify this if to a direct assignment.

r325687: Remove all the empty help files from the powerpc build.

r325688: FDT support doesn't make sense for ps3, remove it.

r325689: Remove LOADER_FDT_SUPPORT as a Makefile variable.

r325690: Remove LOADER_ZFS_SUPPORT as a Makefile variable

r325691: Remove useless PNP define here.

r325692: Replace LOADER_FIREWIRE_SUPPORT variable

r325693: Move LOADER_{NO,}_GELI_SUPPORT to MK_LOADER_GELI

r325694: Install the 4th files in sys/boot/forth instead of each loader

r325743: Make sure the proper loader.rc gets installed.

r325744: boot1: also check for NULL device

r325748: Use proper include file.

r325775: Add loader.conf to the list of files that are MD.

r325779: Add /boot/dts to the list of default modules.

r325780: Don't add /boot/dt*s* but /boot/dt*b*. Stupid think-o.

MFC Loader Fixes 2017q2: r316437,r316577,r316578,r316585,r316590,r316612,
r316625,r316628,r316654,r316682,r316704,r316771,r317092,r317097,r317099,
r317652,r317785,r317886,r317887,r318142,r318320,r318356,r318678,r318754,
r318982,r318986,r318987,r318988,r318989,r318990,r318991,r318992,r318993,
r318994,r318999,r319083,r319084,r319085,r320011,r320234,r320288,r320304,
r320467,r320482

r316437: Small cleanup to make i386/loader match efi/loader boot environment
code

r316577: loader: part.c cstyle cleanup

r316578: loader: want mechanism to avoid RA with bcache

r316585: loader: zfs reader should check all labels

r316590: libstand/dosfs: cache FAT32 in 128 Kb blocks to save loader memory

r316612: In r298230 the value of HEAP_MIN was changed from 3MB to 64MB.
Correct a comment.

r316625: Do not use -msoft-float with intention of disabling FP on aarch64

r316628: Silence GCC warning by initializing the local variable.

r316654: loader: r316585 did miss userboot update

r316682: loader: r316585 did miss sparc/ofw

r316704: loader.efi: only fetch zfs pool guid for the actual boot device

r316771: loader: Avoid possible overflow via environment variable

r317092: loader: zfs reader vdev_probe should check for minimum device size

r317097: loader: F_READ/F_WRITE should be checked against masked flag

r317099: loader: uboot disk ioctl should call disk_ioctl

r317652: loader.efi: ResetSystem does not use data with EFI_SUCCESS

r317785: zfsboot: drvsize() may be unusable on some systems

r317886: distinguish NFS versus TFTP boot by rootpath

r317887: loader: network read rework

r318142: libstand: NULL pointer dereference in rarp

r318320: loader: add ip layer code into libstand

r318356: libstand: increase nfs max read size to 16k

r318678: Replacing iterating over rootpath by strsep(3).

r318754: Pass -N directly to ld via -Wl rather than passing it to the
compiler driver.

r318982: Pass a "FREEBSD" user-class in PXE dhcp request

r318986: add a comment on vendor index 19 and 20 to avoid confusion

r318987: Support URI scheme for root-path in netbooting

r318988: Always build tftpfs support along with nfs for pxeboot

r318989: Always issue the pxe request

r318990: Partially revert r314948

r318991: Document recent changes on pxeboot

r318992: Capitalize DHCP

r318993: Use the usual FreeBSD spelling for the DHCP user class

r318994: Catch with the change in the user class

r318999: Update the comments concerning net_parse_rootpath to reflect what
it is now

r319083: Followup on the user-class changes

r319084: Small cleanup in dev_net.c

r319085: use the same option list for dhcp discovery and request

r320011: Add chain loader support for loader

r320234: Make structure padding explicit in EFI_MEMORY_DESCRIPTOR

r320288: Allow Clang's integrated assembler to assemble boot0

r320304: loader.efi: Disable smbios for arm

r320467: loader: chain load relocate data declaration is bad

r320482: As with arm64 mark the EFI PE header as allocated on arm.

PR: 218473

MFC Loader Fixes 2017q1: r311458,r312237,r312314,r312374,r312947,r313042,
r313047,r313166,r313328,r313332,r313333,r313337,r313348,r313349,r313389,
r313442,r313451,r313575,r313645,r313710,r314114,r314213,r314275,r314945,
r314948,r315008,r315408,r315427,r315645,r315646,r315648,r315653,r315850,
r316064,r316078,r316079,r316100,r316104,r316111,r316112,r316171,r316279,
r316280,r316287,r316311,r316343,r316424,r316436

r311458: Use compiler driver to link BERI boot loaders

r312237: loader.efi: find_currdev() can leak memory

r312314: loader: move device path definitions to include/efidevp.h

r312374: loader: efi devpath api usage should be more aware of NULL pointers

r312947: Remove "-Xassembler -G0" from CFLAGS.

r313042: loader.efi environment related cleanups

r313047: loader: disk/part api needs to use uint64_t offsets

r313166: loader: libefi/env.c warnings in arm build

r313328: loader: Implement disk_ioctl() to support DIOCGSECTORSIZE and
DIOCGMEDIASIZE.

r313332: loader: bcache read ahead block count should take account the large
sectors

r313333: loader: Replace EFI part devices.

r313337: loader: 313329 missed ZFS guard in loader/main.c

r313348: loader: biosdisk fix for 2+TB disks

r313349: loader: disk io should not use alloca()

r313389: efipart is also using the '%S' printf format, add -Wno-format for
it.

r313442: loader: possible NULL pointer dereference in efipart.c

r313451: loader: possible NULL pointer dereference in bcache.c

r313575: makefs: make the buffer functions look exactly like the kernel ones

r313645: loader: implement MEDIA_FILEPATH_DP support in efipart

r313710: loader: cstyle fixes and DIOCGMEDIASIZE should use uint64_t

r314114: Use LDFLAGS with CC instead of _LDFLAGS.

r314213: Remove control+r handling from geliboot's pwgets()

r314275: Remove unused macro from common/drv.c.

r314945: Some style(9) fixes. No functional changes.

r314948: Try to extract the RFC1048 data from PXE.

r315008: r314948 seems to be missing a variable or two that will break

r315408: loader: remove open_disk cache

r315427: loader: biosdisk should report IO error from INT13

r315645: loader: disk_cleanup was left in userboot_disk.c

r315646: loader: pxe.h constants have wrong values

r315648: libstand: verify value provided by nfs.read_size

r315653: loader: verify the value from dhcp.interface-mtu and use snprintf
o set mtu

r315850: The original author abused Nd (one-line description, used by
makewhatis)

r316064: Fix build with path names with 'align' or 'nop' in them.

r316078: gpt*boot: Save a bit more memory when LOADER_NO_GELI_SUPPORT is
specified

r316079: Simply retire the sedification of the boot2.s file.

r316100: Remove -fno-guess-branch-probability and -fno-unit-at-a-time.

r316104: Use `NO_WCAST_ALIGN` instead of spelling it out as -Wno-cast-align
in CFLAGS

r316111: loader: move bios getsecs into time.c

r316112: loader: ls command should display file types properly

r316171: xfsread inlined uses more space, so remove the inline tag.

r316279: loader: efipart should check disk size from partition table

r316280: loader: simplify efi_zfs_probe and avoid double probing for zfs.

r316287: Remove OLD_NFSV2 from loader and libstand

r316311: Add explicit_bzero() to libstand, and switch GELIBoot to using it

r316343: Implement boot-time encryption key passing (keybuf)

r316424: Fix sparc64 build broken by r316343 and r316076

r316436: Restore EFI boot environment functionality broken in r313333

PR: 216940 217298 217935

MFC r309369,310850,310853:

libstand: dosfs cstyle cleanup for return keyword.
dosfs support in libstand is broken since r298230

PR: 214423
Submitted by: Mikhail Kupchik
Reported by: Mikhail Kupchik
Approved by: imp (mentor)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Fix several instances where the boot loader ignored pager_output
return value when it could return 1 (indicating we should stop).
Fix a few instances of pager_open() / pager_close() not being called.
Actually use these routines for the environment variable printing code
I just committed.

A new implementation of the loader block cache

The block cache implementation in loader has proven to be almost useless, and in worst case even slowing down the disk reads due to insufficient cache size and extra memory copy.
Also the current cache implementation does not cache reads from CDs, or work with zfs built on top of multiple disks.
Instead of an LRU, this code uses a simple hash (O(1) read from cache), and instead of a single global cache, a separate cache per block device.
The cache also implements limited read-ahead to increase performance.
To simplify read ahead management, the read ahead will not wrap over bcache end, so in worst case, single block physical read will be performed to fill the last block in bcache.

Booting from a virtual CD over IPMI:
0ms latency, before: 27 second, after: 7 seconds
60ms latency, before: over 12 minutes, after: under 5 minutes.

Submitted by: Toomas Soome <tsoome@me.com>
Reviewed by: delphij (previous version), emaste (previous version)
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D4713

Implement GELI (AES-XTS and AES-CBC only) in gptboot and gptzfsboot

Allows booting from a GELI encrypted root file system, via UFS or ZFS

Reviewed by: gnn, smh (previous version), delphij (previous version)
Relnotes: yes
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D4593

Implement simple ops for umass_disk

The initial IOCTL implementation supports reading disk physical
geometry.
Two additional functions were added. They allow reading/writing raw
data to the disk (default partition).

Submitted by: Wojciech Macek <wma@semihalf.com>
Obtained from: Semihalf
Sponsored by: Juniper Networks Inc.
Differential Revision: https://reviews.freebsd.org/D4143

Rework bootparttest to use more code from sys/boot.
Use disk_open() call to emulate loader behavior.

When loader tries to open GPT partition, but partition table is not GPT,
then try automatically detect an appropriate partition type.

PR: kern/172550
Tested by: Ralf Wenk

Add the flags parameter to the disk_open() function and DISK_F_NOCACHE
flag, that disables the caching of partition tables metadata.
Use this flag for floppies in the libi386/biosdisk driver.

Fix disk_cleanup() to work without DISK_DEBUG too.

Almost each time when loader opens a file, this leads to calling
disk_open(). Very often this is called several times for one file.
This leads to reading partition table metadata for each call. To
reduce the number of disk I/O we have a simple block cache, but it
is very dumb and more than half of I/O operations related to reading
metadata, misses this cache.

Introduce new cache layer to resolve this problem. It is independent
and doesn't need initialization like bcache, and will work by default
for all loaders which use the new DISK API. A successful disk_open()
call to each new disk or partition produces new entry in the cache.
Even more, when disk was already open, now opening of any nested
partitions does not require reading top level partition table.
So, if without this cache, partition table metadata was read around
20-50 times during boot, now it reads only once. This affects the booting
from GPT and MBR from the UFS.

Make the loader a bit smarter, when it tries to open disk and the slice
number is not exactly specified. When the disk has MBR, also try to read
BSD label after ptable_getpart() call. When the disk has GPT, also set
d_partition to 255. Mostly, this is how it worked before.

Rework r239232 to unbreak ZFS detection on MBR slices.

Restore the old behaviour. If requested partition is a BSD slice,
but d_partition isn't explicitly set, then try to open BSD label and its
first partition.

Remove colons from the debug message, device name returned by the
disk_fmtdev() already has the colons.

Unbreak booting from the true dedicated disks.
When we open the disk, check the type of partition table, that has
been detected. If this is BSD label, then we assume this is DD mode.

Reported by: dim@

Add more debug messages.

Introduce new API to work with disks from the loader's drivers.
It uses new API from the part.c to work with partition tables.

Update userboot's disk driver to use new API. Note that struct
loader_callbacks_v1 has changed.

With LOADER_MBR_SUPPORT defined and LOADER_GPT_SUPPORT undefined we would
never call disk_openmbr().

Submitted by: avg
MFC after: 3 days

Fix missing return when LOADER_GPT_SUPPORT is defined, but LOADER_MBR_SUPPORT
is not.

MFC after: 3 days

Fix r223695 to compile on architectures which don't use the MBR scheme; wrap
the MBR support in the common part of the loader in #ifdef's and enable it
only for userboot for now.

Add a version of the FreeBSD bootloader which can run in userland, packaged
as a shared library. This is intended to be used by BHyVe to load FreeBSD
kernels into new virtual machines.