Copy stable/11@r303970 to releng/11.0 as part of the 11.0-RELEASE
cycle.

Prune svn:mergeinfo from the new branch, and rename it to RC1.

Update __FreeBSD_version.

Use the quarterly branch for the default FreeBSD.conf pkg(8) repo and
the dvd1.iso packages population.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

boot/forth: spelling fixes.

Reviewed by: dteske
MFC after: 2 weeks

Introduce the ZFS Boot Environments menu to the loader menu

If the system was booted with ZFS, a new menu item (#7) appears
It contains an autogenerated list of ZFS Boot Environments

This allows the user to switch to an alternate root file system
Use Cases:
- Revert a failed upgrade
- Concurrently run different versions of FreeBSD with common home directory
- Easier integration with the sysadmin/beadm utility

Requested by: many
Reviewed by: dteske
MFC after: 10 days
Relnotes: yes
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D3167

Remove supposition comment that code would better live elsewhere.
Thinking this through, and looking at process_assignment, I believe
moving the code would be wrong considering that set_conf_files is
called in one condition while set_nextboot_conf is guarded by a
different condition of having nextboot_enable="YES". So these must
stay separated and not combined.

MFC after: 1 week

Fix a memory leak. nextboot_conf_file is not volatile, as far as I
can tell, and thus the strdup is entirely unnecessary.

Thanks to: Toomas Soome (tsoome at-me dot-com)
MFC after: 1 week

Remove debugging messages. NB: SVN r187143 reads:
comment out some debugging messages that slipped in by mistake
(removing them 7 years after they have been commented out)

MFC after: 1 week

Fix stack leak introduced by SVN r97201 (nextboot_conf support).
Fix erroneous error path in error messages when processing boot_conf.
Fixup stack comments for functions introduced by SVN r97201.
Remove a questioning stack comment introduced by SVN r186789.
NB: Comment removed because strdup usage here is correct/not a leak.

MFC after: 1 week

Add comments to indicate where the flag for later abort" originates.

MFC after: 3 days
X-MFC-to: stable/10

Solve dreaded "dictionary full" issue by segregating words into new
vocabularies delay-processing, password-processing, version-processing,
frame-drawing, menu-infrastructure, menu-namespace, menu-command-helpers,
and menusets-infrastructure. The net effect is to remove almost 200
definitions from the main forth vocabulary reducing the dictionary size
by over 50%. The chances of hitting "dictionary full" should be greatly
reduced by this patch.

MFC after: 3 days
X-MFC-to: stable/10

Comments. No functional change(s).

MFC after: 3 days
X-MFC-to: stable/10

loader: implement multiboot support for Xen Dom0

Implement a subset of the multiboot specification in order to boot Xen
and a FreeBSD Dom0 from the FreeBSD bootloader. This multiboot
implementation is tailored to boot Xen and FreeBSD Dom0, and it will
most surely fail to boot any other multiboot compilant kernel.

In order to detect and boot the Xen microkernel, two new file formats
are added to the bootloader, multiboot and multiboot_obj. Multiboot
support must be tested before regular ELF support, since Xen is a
multiboot kernel that also uses ELF. After a multiboot kernel is
detected, all the other loaded kernels/modules are parsed by the
multiboot_obj format.

The layout of the loaded objects in memory is the following; first the
Xen kernel is loaded as a 32bit ELF into memory (Xen will switch to
long mode by itself), after that the FreeBSD kernel is loaded as a RAW
file (Xen will parse and load it using it's internal ELF loader), and
finally the metadata and the modules are loaded using the native
FreeBSD way. After everything is loaded we jump into Xen's entry point
using a small trampoline. The order of the multiboot modules passed to
Xen is the following, the first module is the RAW FreeBSD kernel, and
the second module is the metadata and the FreeBSD modules.

Since Xen will relocate the memory position of the second
multiboot module (the one that contains the metadata and native
FreeBSD modules), we need to stash the original modulep address inside
of the metadata itself in order to recalculate its position once
booted. This also means the metadata must come before the loaded
modules, so after loading the FreeBSD kernel a portion of memory is
reserved in order to place the metadata before booting.

In order to tell the loader to boot Xen and then the FreeBSD kernel the
following has to be added to the /boot/loader.conf file:

xen_cmdline="dom0_mem=1024M dom0_max_vcpus=2 dom0pvh=1 console=com1,vga"
xen_kernel="/boot/xen"

The first argument contains the command line that will be passed to the Xen
kernel, while the second argument is the path to the Xen kernel itself. This
can also be done manually from the loader command line, by for example
typing the following set of commands:

OK unload
OK load /boot/xen dom0_mem=1024M dom0_max_vcpus=2 dom0pvh=1 console=com1,vga
OK load kernel
OK load zfs
OK load if_tap
OK load ...
OK boot

Sponsored by: Citrix Systems R&D
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D517

For the Forth bits:
Submitted by: Julien Grall <julien.grall AT citrix.com>

Add support for comma-separated values (whitespace-separated still supported).

PR: conf/121064
Submitted by: koitsu
Reviewed by: jh

Use ASCII characters for box/line characters in frames.4th

Committed with changes to support the following from loader.conf(5):
+ console="vidconsole comconsole" (not just console="comconsole")
+ boot_serial="anything" (not just boot_serial="YES")
+ boot_multicons="anything" (unsupported in originally-submitted patch)

PR: conf/121064
Submitted by: koitsu
Reviewed by: gcooper, adrian (co-mentor)
Approved by: adrian (co-mentor)

New boot loader menus from Devin Teske.
Discussed on hackers and recommended for inclusion into 9.0 at the devsummit.
All support email to devin dteske at vicor dot ignoreme dot com .

Submitted by: dteske at vicor dot ignoreme dot com
Reviewed by: me and many others

comment out some debugging messages that slipped in by mistake.

MFC after: 3 days

This patch introduces a number of simplifications to the Forth
functions used in the bootloader. The goal is to make the code more
readable and smaller (especially because we have size issues
in the loader's environment).

High level description of the changes:
+ define some string manipulation functions to improve readability;
+ create functions to manipulate module descriptors, removing some
duplicated code;
+ rename the error codes to ESOMETHING;
+ consistently use set_environment_variable (which evaluates
$variables) when interpreting variable=value assignments;

I have tested the code, but there might be code paths that I have
not traversed so please let me know of any issues.

Details of this change:

--- loader.4th ---
+ add some module operators, to remove duplicated code while parsing
module-related commands:

set-module-flag
enable-module
disable-module
toggle-module
show-module

--- pnp.4th ---
+ move here the definition related to the pnp devices list, e.g.
STAILQ_* , pnpident, pnpinfo

--- support.4th ---
+ rename error codes to capital e.g. ENOMEM EFREE ... and do obvious
changes related to the renaming;
+ remove unused structures (those relevant to pnp are moved to pnp.4th)
+ various string functions
- strlen removed (it is an internal function)
- strchr, defined as the C function
- strtype -- type a string to output
- strref -- assign a reference to the string on the stack
- unquote -- remove quotes from a string

+ remove reset_line_buffer

+ move up the 'set_environment_variable' function (which now
uses the interpreter, so $variables are evaluated).
Use the function in various places

+ add a 'test_file function' for debugging purposes

MFC after: 4 weeks

PROBLEM: putting in a loader config file a line of the form

loader_conf_files="foo bar baz"

should cause loading the files listed, and then resume with the
remaining config files (from previous values of the variable).
Unfortunately, sometimes the line was ignored -- actually even
modifying the line in /boot/default/loader.conf sometimes doesn't work.

ANALYSIS: After much investigation, turned out to be a bug in the logic.
The existing code detected a new assignment by looking at the address
of the the variable containing the string. This only worked by pure
chance, i.e. if the new string is longer than the previous value
then the memory allocator may return a different address
to store the string hence triggering the detection.

SOLUTION: This commit contains a minimal change to fix the problem,
without altering too much the existing structure of the code.
However, as a step towards improving the quality and reliability of
this code, I have introduced a handful of one-line functions
(strget, strset, strfree, string= ) that could be used in dozens
of places in the existing code.

HOWEVER:
There is a much bigger problem here. Even though I am no Forth
expert (as most fellow src committers) I can tell that much of the
forth code (in support.4th at least) is in severe need of a
review/refactoring:

+ pieces of code are replicated multiple times instead of writing
functions (see e.g. set_module_*);

+ a lot of stale code (e.g. "structure" definitions for
preloaded_files, kernel_module, pnp stuff) which is not used
or at least belongs elsewhere.
The code bload is extremely bad as the loader runs with very small
memory constraints, and we already hit the limit once (see

http://svn.freebsd.org/viewvc/base?view=revision&revision=185132
Reducing the footprint of the forth files is critical.

+ two different styles of coding, one using pure stack functions
(maybe beautiful but surely highly unreadable), one using
high level mechanisms to give names to arguments and local
variables (which leads to readable code).

Note that this code is used by default by all FreeBSD installations,
so the fragility and the code bloat are extremely damaging.
I will try to work fixing the three items above, but if others have
time, please have a look at these issues.

MFC after: 4 weeks

Allow negative values to be specified in the loader.

Replace a rarely used "depuration" with "debugging".

PR: docs/85127
Submitted by: Gary W. Swearingen (partially)
MFC after: 3 days

Add support for reading an additional loader configuration file. By default,
this is called /boot/nextboot.conf. This file is required to have it's first
line be nextboot_enable="YES" for it to be read. Also, this file is
rewritten by the loader to nextboot_enable="NO"<space> after it is read.
This makes it so the file is read exactly once. Finally, the nextboot.conf
is removed shortly after the filesystems are mounted r/w.

Caution should be taken as you can shoot yourself in the foot. This is only
the loader piece. There will be a tool called nextboot(8) that will manage
the nextboot.conf file for you. It is coming shortly.

Reviewed by: dcs
Approved by: jake (mentor)

- Add 'fwrite' and 'fseek' words for writing to and seeking on files.
- Change the 'fopen' keyword to accept a mode parameter. Note that this
will break existing 4th scripts that use fopen. Thus, the loader
version has been bumped and loader.4th has been changed to check for a
sufficient version on i386 and alpha. Be sure that you either do a full
world build or install or full build and install of sys/boot after this
since loader.old won't work with the new 4th files and vice versa.

PR: kern/32389
Submitted by: Jonathan Mini <mini@haikugeek.com>
Sponsored by: ClickArray, Inc.

Get rid of garbage left on the stack.

Use _ instead or - where proper, according to the style I have been
using.

Overload "?" so it will also show loader.4th commands.

Solve a name clash.
Add something to help debugging.

Both boot and boot-conf were using a different algorithm from the one
used by start to find the kernel. Fix this.

Also, boot would proceed immediately in the absence of a path as
argument. Check first if a kernel has already been loaded, and, if
not, fall back to load kernel&modules behavior.

Some further factorizing. I deem this code to be mostly readable by
now! :-)

Many thanks to: Makoto MATSUSHITA <matusita@jp.FreeBSD.org>

Factorize, reorganize, and move code around.

The boot-conf and boot code had various bugs, and some of it was big,
ugly, unwieldy, and, sometimes, plain incorrect. I'm just about
completely replaced these ugly parts with something much more manageable.

Minor changes were made to the well-factorized parts of it, to accomodate
the new code.

Of note:

* make sure boot-conf has the exact same behavior wrt boot order
as start.

* Correct both boot and boot-conf so they'll work correctly when
compiled in, as they both had some bugs, minor and major.

* Remove all the crud from loader.4th back into support.4th, for
the first time since boot-conf was first improved. Hurray!

I'm fairly satisfied with the code at this time. Time to see about those
man pages...

Upon reflection, I decided that bootfile must have priority over kernel
as the kernel name. The one very unfortunate consequence is that kernel
as an absolute path loses the priority. It will only be tried after
/boot/${kernel}/${bootfile}. I'll see what can be done about it later.

First tackle at trying to handle the New Deal on kernels.

Load the first of the following kernels to be found:

${kernel} if ${kernel} is an absolute path
/boot/${kernel}/${kernel}
/boot/${kernel}/${bootfile}
${kernel}/${kernel}
${kernel}/${bootfile}
${kernel}
${bootfile}

The last instance of ${kernel} and ${bootfile} will be treated as a
list of semicolon separated file names, and each will be tried in turn,
from left to right.

Also, for each filename loader(8) will try filename, filename.ko,
filename.gz, filename.ko.gz, in that order, but that's not related
to this code.

This resulted in a major reorganization of the code, and much of what
was accumulating on loader.4th was rightly transfered to support.4th.

The semantics of boot-conf and boot also changed. Both will try to load
a kernel the same as above.

After a kernel was loaded, the variable module_path may get changed. Such
change will happen if the kernel was found with a directory prefix. In
that case, the module path will be set to ${directory};${module_path}.

Next, the modules are loaded as usual.

This is intended so kernel="xyzzy" in /boot/loader.conf will load
/boot/xyzzy/kernel.ko, load system modules from /boot/xyzzy/, and
load third party modules from /boot/modules or /modules. If that doesn't
work, it's a bug.

Also, fix a breakage of "boot" which was recently introduced. Boot without
any arguments would fail. No longer. Also, boot will only unload/reload
if the first argument is a path. If no argument exists or the first
argument is a flag, boot will use whatever is already loaded. I hope this
is POLA. That behavior is markedly different from that of boot-conf, which
will always unload/reload.

The semantics introduced here are experimental. Even if the code works,
we might decide this is not the prefered behavior. If you feel so, send
your feedback. (Yeah, this belongs in a HEADS UP or something, but I've
been working for the past 16 hours on this stuff, so gimme a break.)

Fix an error message which was using the wrong variable to get the
kernel name from.

Add constructors to crude structure support. Rework some of the
code into a more modular interface, with hidden vocabularies and
such. Remove the need to a lot of ugly initialization.

Also, add a few structure definitions, from stuff used on the C
part of loader. Some of this will disappear, and the crude structure
support will most likely be replaced by full-blown OOP support
already present on FICL, but not installed by default. But it was
getting increasingly inconvenient to keep this separate on my tree,
and I already lost lots of work once because of the hurdles, so
commit this.

Anyway, it makes support.4th more structured, and I'm not proceeding
with the work on it any time soon, unfortunately.

Remove AGAIN definition, as FICL 2.04 provides it.

Add strlen, to help handling data generated by C code.

Add 2>r 2r>, because OO programming without them sucks.

Add silly password feature. If people want to depend on a flawed
security measures, so be it. It costs us almost nothing.

Document some code in support.4th that I was unable to understand
just by reading.

$Id$ -> $FreeBSD$

New loader.rc stuff.

Reviewed by: jkh