MFC r325834,r325997,326502: Move sys/boot to stand/

This is effectively a direct commit to stable/11, due to differences between
stable/11 and head. Changes to DTS in sys/boot/fdt/dts were often
accompanied by kernel changes. Many of these were also risc-v updates that
likely had many more dependencies to MFC.

Because of this, sys/boot/fdt/dts remains as-is while everything else in
sys/boot relocates to stand/.

r325834: Move sys/boot to stand. Fix all references to new location

r325997: Remove empty directories.

r326502: Document the sys/boot -> stand move in hier.7 and the top-level README.

MFC r303555,r303556,r303936,r303962,r304317,r304532,r305026,r305107,r305132,
r305178,r305353,r305814,r306159,r306380,r306504: Loader fixes, 2016q3

r303555: bcache should support reads shorter than sector size

r303556: Improve boot loader quote parsing

r303936: Add kernel environment variables under smbios.system

r303962: Add the missing space between .asciz directive and opening quote
for some lines with #ifdef BTXLDR_VERBOSE/#endif

r304317: boot1.efi Free() should check for NULL to provide consistent
behavior

r304532: Replace sprintf -> snprintf for command_errbuf provisioned from
dynamic content.

r305026: Emulate efi_cons_poll when WaitForKey is not available

r305107: Create a hook 'post-initialize' for people that want to define
something to read in .conf files after all other .conf files for the purpose
of overriding.

r305132: Remove accidentally committed stray comment.

r305178: bd_int13probe() should check extended info if sector info is bad

r305353: Don't use -N to set the OMAGIC with data and text writeable and
data not page aligned.

r305814: ufsread: Do not cast struct direct from void *

r306159: Consistently declare getsecs(void) with proper return type and void
when no arguments are present.

r306380: loader command interpreter should reset command_errmsg

r306504: Fix a cluster of bugs in list EFI environment variables

PR: 204602, 211958, 211958
Relnotes: yes ("Improve boot loader quote parsing")

MFC r308434, 308827

MFC r308434: Loader paged/pageable data is not always paged.
MFC r308827: lsdev device name section headers should be printed by dv_print
callback.

MFC r316076:

sys/boot/common: Make geli(4) support optional in MI sources

This saves a negligible amount of memory for non-geli enabled
bootloaders.

MFC r316754: loader/multiboot: fix multiboot loading

Sponsored by: Citrix Systems R&D

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Fix several instances where the boot loader ignored pager_output
return value when it could return 1 (indicating we should stop).
Fix a few instances of pager_open() / pager_close() not being called.
Actually use these routines for the environment variable printing code
I just committed.

sys/boot/common: use of spaces vs. TAB.

No functional change.

sys: use our roundup2/rounddown2() macros when param.h is available.

rounddown2 tends to produce longer lines than the original code
and when the code has a high indentation level it was not really
advantageous to do the replacement.

This tries to strike a balance between readability using the macros
and flexibility of having the expressions, so not everything is
converted.

A new implementation of the loader block cache

The block cache implementation in loader has proven to be almost useless, and in worst case even slowing down the disk reads due to insufficient cache size and extra memory copy.
Also the current cache implementation does not cache reads from CDs, or work with zfs built on top of multiple disks.
Instead of an LRU, this code uses a simple hash (O(1) read from cache), and instead of a single global cache, a separate cache per block device.
The cache also implements limited read-ahead to increase performance.
To simplify read ahead management, the read ahead will not wrap over bcache end, so in worst case, single block physical read will be performed to fill the last block in bcache.

Booting from a virtual CD over IPMI:
0ms latency, before: 27 second, after: 7 seconds
60ms latency, before: over 12 minutes, after: under 5 minutes.

Submitted by: Toomas Soome <tsoome@me.com>
Reviewed by: delphij (previous version), emaste (previous version)
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D4713

Make common boot file_loadraw name parameter const

Fix compiler warnings about dropping const qualifier by changing file_loadraw
name param to const, and updating method to make that the case (it was
abusing the variable).

MFC after: 2 weeks
X-MFC-With: r293268
Sponsored by: Multiplay

Improve non-interactive forth cmd error reporting

Non-interactive forth command errors where silent even for critical issues
e.g. failing to load a required kernel module or mfs_root.

This resulted in later unexplained and hard to trace errors such as mount
root failures.

This introduces additional command return codes that are treated
appropriately by the non-interactive command processor (bf_command).
* CMD_CRIT = print error
* CMD_FATAL = panic

Also fix minor style(9) issues with command_load return codes.

MFC after: 2 weeks
X-MFC-With: r293268
Sponsored by: Multiplay

Enable warnings in EFI boot code

Set WARNS if not set for EFI boot code and fix the issues highlighted by
setting it.

Most components are set to WARNS level 6 with few being left at lower
levels due to the amount of changes needed to fix at higher levels.

Error types fixed:
* Missing / invalid casts
* Missing inner structs
* Unused vars
* Missing static for internal only funcs
* Missing prototypes
* Alignment changes
* Use of uninitialised vars
* Unknown pragma (intrinsic)
* Missing types etc due to missing includes
* printf formatting types

Reviewed by: emaste (in part)
MFC after: 2 weeks
X-MFC-With: r293268
Sponsored by: Multiplay
Differential Revision: https://reviews.freebsd.org/D4839

Fix a problem which made loader(8) load non-kld files twice.

For example, without this patch, the following three lines
in /boot/loader.conf would result in /boot/root.img being preloaded
twice, and two md(4) devices - md0 and md1 - being created.

initmd_load="YES"
initmd_type="md_image"
initmd_name="/boot/root.img"

Reviewed by: marcel@
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3204

loader: implement multiboot support for Xen Dom0

Implement a subset of the multiboot specification in order to boot Xen
and a FreeBSD Dom0 from the FreeBSD bootloader. This multiboot
implementation is tailored to boot Xen and FreeBSD Dom0, and it will
most surely fail to boot any other multiboot compilant kernel.

In order to detect and boot the Xen microkernel, two new file formats
are added to the bootloader, multiboot and multiboot_obj. Multiboot
support must be tested before regular ELF support, since Xen is a
multiboot kernel that also uses ELF. After a multiboot kernel is
detected, all the other loaded kernels/modules are parsed by the
multiboot_obj format.

The layout of the loaded objects in memory is the following; first the
Xen kernel is loaded as a 32bit ELF into memory (Xen will switch to
long mode by itself), after that the FreeBSD kernel is loaded as a RAW
file (Xen will parse and load it using it's internal ELF loader), and
finally the metadata and the modules are loaded using the native
FreeBSD way. After everything is loaded we jump into Xen's entry point
using a small trampoline. The order of the multiboot modules passed to
Xen is the following, the first module is the RAW FreeBSD kernel, and
the second module is the metadata and the FreeBSD modules.

Since Xen will relocate the memory position of the second
multiboot module (the one that contains the metadata and native
FreeBSD modules), we need to stash the original modulep address inside
of the metadata itself in order to recalculate its position once
booted. This also means the metadata must come before the loaded
modules, so after loading the FreeBSD kernel a portion of memory is
reserved in order to place the metadata before booting.

In order to tell the loader to boot Xen and then the FreeBSD kernel the
following has to be added to the /boot/loader.conf file:

xen_cmdline="dom0_mem=1024M dom0_max_vcpus=2 dom0pvh=1 console=com1,vga"
xen_kernel="/boot/xen"

The first argument contains the command line that will be passed to the Xen
kernel, while the second argument is the path to the Xen kernel itself. This
can also be done manually from the loader command line, by for example
typing the following set of commands:

OK unload
OK load /boot/xen dom0_mem=1024M dom0_max_vcpus=2 dom0pvh=1 console=com1,vga
OK load kernel
OK load zfs
OK load if_tap
OK load ...
OK boot

Sponsored by: Citrix Systems R&D
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D517

For the Forth bits:
Submitted by: Julien Grall <julien.grall AT citrix.com>

The current limit of 100k for the linker hints file is getting a bit
crowded as we now are at about 70k. Bump the limit to 1MB instead
which is still quite a reasonable limit and allows for future growth
of this file and possible future expansion to additional data.

MFC After: 2 weeks

When built with FDT support, add /boot/dtb to the list of search directories.

Rename command_unload() to unload() and re-implement command_unload()
in terms of unload() This allows unloading all files by the loader
itself.

Obtained from: Juniper Networks, Inc.

In command_lsmod() prevent overrunning lbuf due to long path
names. Call pager_output() separately for the module name.

Obtained from: Juniper Networks, Inc.

In file_loadraw() print the name of the file as well as its size
so that we know what file is being loaded and how big the file
is once complete. This has ELF modules and disk images emit the
same output.

Change file_loadraw() from static to public. Change the order of its
arguments from type,filename to filename,type to be consistant with other
public file_whatever() functions, and change it to return a pointer to
the preloaded_file struct describing the file. Adjust existing callers.

Since we didn't break the loop, we should set i to -1 to start from the
beginning.

Submitted by: Steven Hartland
MFC after: 1 week

Fix the bug I introduced in r247045.
After digging through more carefully, it looks like there's
no real need to have the DTB in the module directory.
So we can simplify a lot: Just copy DTB into local heap
for "fdt addr" and U-Boot integration, drop all the extra
COPYIN() calls.

I've left one final COPYIN() to update the in-kernel DTB
for consistency with how this code used to work, but I'm
no longer convinced it's appropriate here.

I've also remove the mem_load_raw() utility that I added
to boot/common/module.c with r247045 since it's no longer
necessary.

Add mem_load_raw() for loading data from another location in memory.

This will be used by some upcoming changes to loader(8) FDT
handling to allow it to use an FDT provided by an earlier
boot stage the same as an FDT loaded from disk.

Fix the style.

Remember the file format of the last loaded module and try to use it for
next files.

boot: file_loadraw should strdup name argument

... the same way it's done for type argument.

MFC after: 2 weeks

Fix a long standing bug where file_load() passes down the global loadaddr
to the l_load() method in the file_formats structure, while being passed
an address as an argument (dest). With file_load() calling arch_loadaddr()
now, this bug is a little bit more significant.

Spotted by: nyan@ (nice catch!)

Add 2 new archsw interfaces:
1. arch_loadaddr - used by platform code to adjust the address at which
the object gets loaded. Implement PC98 using this new interface instead
of using conditional compilation. For ELF objects the ELF header is
passed as the data pointer. For raw files it's the filename. Note that
ELF objects are first considered as raw files.
2. arch_loadseg - used by platform code to keep track of actual segments,
so that (instruction) caches can be flushed or translations can be
created. Both the ELF header as well as the program header are passed
to allow platform code to treat the kernel proper differently from any
additional modules and to have all the relevant details of the loaded
segment (e.g. protection).

Formatting nit

Don't use 15M-16M area on pc98. It's reserved for some devices.

MFC after: 2 week

Add a helper function for loading geli keys from the loader.

Also boot *.debug if everything else fails.

Approved by: re(gnn)

In moduledir_readhints() cast the value returned by sizeof() to ssize_t
when checking whether it's greater than a struct stat st_size in order
to also catch the case when st_size is -1. Previously this check didn't
trigger on sparc64 when st_size is -1 (as it's the case for a file on
a bzipfs, TFTP server etc.), causing the content of the linker hints
file to be copied to memory referenced by a null-pointer.

PR: 91231
MFC after: 1 week

Use __FBSDID().
Also some minor style cleanups.

FreeBSD 5.0 has stopped shipping /modules 2.5 years ago. Catch
up with this further by excluding /modules from the (default)
kern.module_path.

Bandaid for a buffer overrun in the module searching code. When breaking
up the module_path string, we would walk one past the end of the buffer.
This hurting ia64 originally, but it was probably also happening on i386
occasionally as well. The effects were usually harmless, it would add
bogus "binary" search directories to the places it actually looked for
files.

Fix a number of misspellings of "dependency" and "dependencies" in
comments and function names.

PR: kern/8589
Submitted by: Rajesh Vaidheeswarran <rv@fore.com>

Implement the long-awaited module->file cache database. A userland
tool (kldxref(8)) keeps a cache of what modules and versions are inside
what .ko files. I have tested this on both Alpha and i386.

Submitted by: bp

Fix a minor style bug in the last commit.

Submitted by: Adrian Steinmann <ast@marabu.ch>
MFC after: 2 days

In sys/boot/common/module.c, near line 105 a request for a raw
file is processed by passing its name in argv[1]:
return(mod_loadobj(typestr, argv[1]));
however, it is not tested to see if argv[1] actually is defined.

At best, mod_loadobj() near line 244 returns an error like
"can't find 'garbage'" but if the "filename" entered is sufficiently
long, some buffer gets overrun. Of course, "load -t filename" is
actually a typo because we meant to type "load -t mfs_root filename";
nevertheless, a hung machine seems like too harsh a punishment for
such a small typo...

PR: i386/27693
Submitted by: Adrian Steinmann <ast@marabu.ch>
MFC after: 1 week

The default search path for kernel and modules was bogus. It makes no
sense for /boot/kernel to come last.

Fix autoboot. Now autoboot *always* show the correct kernel name. It
gets the name from the environment variable kernelname, which is set
when a kernel is loaded. For this reason, autoboot will _first_ try
to load a kernel, and only proceed with the wait prompt after that
succeeds. If it fails, it will abort immediately.

While I understand some may think this behavior undesirable, I think
it is, overall, the best thing to do, even if we do not consider the
aesthetic issue. Notice that anyone using the default loader.rc
already has the kernel loaded before autoboot.

On unload, unset kernelname.

Separate the code that tries to load a kernel from the list of options
to the function loadakernel(). It is used by both boot() and
autoboot().

The kernel is now known as `kernel.ko' and it and its matching modules
live in ``/boot/kernel/''.

Cleanup warnings. Most of these are signed/unsigned warnings, as well as
some added const's.

Update loader logic to distinguish modules vs. files.
Add support for module metadata. The old way of dependancy
handling will be supported for a while.

Reviewed by: peter

Fix the loader to handle module dependencies properly. More fixes
will be provided after modmetadata appears in the kernel.

Reviewed by: msmith
Approved by: jkh

Close a file descriptor leak in the code which loads file objects.

Submitted by: Paul Saab <paul@mu.org>
Approved by: jkh

Add fairings. Do not depend on user actually supplying the arguments
he is supposed to supply.

$Id$ -> $FreeBSD$

Fix a number of memory leaks and other memory-related disorders.

Also, unbreak the breakage introduced at the last revision of module.c.
This changes the semantics of mod_searchfile() (and mod_searchmodule())
to make the caller's responsibility freeing the buffer returned. This
is different from other functions in loader's code, and was done as a
fix for kern/9631. If someone wants to revert this to the original
behavior, don't forget to fix kern/9631 in another way.

This should also fix bin/10462, which was introduced as a result of the
first try at kern/9631 (module.c last revision).

PR: bin/10462
Submitted by: Takanori Saneto <sanewo@ba2.so-net.ne.jp>

Fix assorted memory leak/buffer reuse problems.

Not restricted to, but including:

PR: kern/9631
Submitted by: Bill Fenner <fenner@parc.xerox.com>

No builtin command resets getopt before using it, causing
problems in case a wrong option was given previously, and no option
is given to the next command.

PR: kern/9371
Submitted by: "Daniel C. Sobral" <dcs@newsguy.com>

Use a consistant module search path (same as kernel will be).
Use new dependency mechanism.

* Add old UFS compatibility code to alpha/boot1.
* Fix a raft of warnings, printf and otherwise.
* Allocate the correct amount in mod_searchmodule to prevent an overflow.
* Fix the makefiles so they work outside my home directory (oops).

Resynch with working sources before BTX integration.

- Use format-independant module allocator.
- Conditionalise ISA PnP support.
- Simplify PnP enumerator interface.
- Improve module/object searching.
- Add missing depend/install targets in BTX makefiles.
- Pass the kernel environment and module data in extended bootinfo fields.
- Add a pointer to the end of the kernel + modules in bootinfo.
- Fix parsing of old-style kernel arguments.

Bootstrap updates.

- Move some startup code from MD to MI sections
- Add a 'copyout' and some copyout-related functions. These will be
obsoleted when BTX is available for the 386 and the kernel load
area becomes directly addressable.
- Add the ability load an arbitrary file as a module, associating
and arbitrary type string with it. This can be used eg. for loading
splash-screen images etc.
- Add KLD module dependancy infrastructure. We know how to look for
dependancies inside KLD modules, how to resolve these dependancies
and what to do if things go wrong. Only works for a.out at the
moment, due to lack of an MI ELF loader. Attach KLD module information
to loaded modules as metadata, but don't pass it to the kernel (it
can find it itself).
- Load a.out KLD modules on a page boundary. Only pad the a.out BSS
for the kernel, as it may want to throw symbols away. (We might want
to do this for KLD modules too.)
- Allow commands to be hidden from the '?' display, to avoid cluttering
it with things like 'echo'. Add 'echo'.
- Bring the 'prompt' command into line with the parser syntax.
- Fix the verbose 'ls'; it was using an uninitialised stack variable.
- Add a '-v' flag to 'lsmod' to have it display module metadata as well
(not terribly useful for the average user)
- Support a 'module searchpath' for required modules.
- The bootstrap file on i386 is now called 'loader' to permit the
/boot directory to use that name.
- Discard the old i386 pread() function, as it's replaced by
arch_readin()

Bootloader update.

- Implement a new copyin/readin interface for loading modules.
This allows the module loaders to become MI, reducing code duplication.
- Simplify the search for an image activator for the loaded kernel.
- Use the common module management code for all module metadata.
- Add an 'unload' command that throws everything away.
- Move the a.out module loader to MI code, add support for a.out
kld modules.

Submitted by: Alpha changes fixed by Doug Rabson <dfr@freebsd.org>

This commit was generated by cvs2svn to compensate for changes in r38465,
which included commits to RCS files with non-trunk default branches.

This is the new unified bootstrap, sometimes known previously as the
'three-stage' bootstrap.
There are a number of caveats with the code in its current state:
- The i386 bootstrap only supports booting from a floppy.
- The kernel and kld do not yet know how to deal with the extended
information and module summary passed in.
- PnP-based autodetection and demand loading of modules is not implemented.
- i386 ELF kernel loading is not ready yet.
- The i386 bootstrap is loaded via an ugly blockmap.

On the alpha, both net- and disk-booting (SRM console machines only) is
supported. No blockmaps are used by this code.

Obtained from: Parts from the NetBSD/i386 standalone bootstrap.