Copy stable/11@r303970 to releng/11.0 as part of the 11.0-RELEASE
cycle.

Prune svn:mergeinfo from the new branch, and rename it to RC1.

Update __FreeBSD_version.

Use the quarterly branch for the default FreeBSD.conf pkg(8) repo and
the dvd1.iso packages population.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Explain why extra sutff always outputs even when it shouldn't for the
'?' command. Wasted a bunch of time tracking it down tonight.

boot/forth: spelling fixes.

Reviewed by: dteske
MFC after: 2 weeks

Re-do proper mode-endings. Antithesis of r281176-281179 which reverted
earlier migration away from sloppy mode-endings (r280925,280974-280976)
due to a red-herring in diagnosing HardenedBSD boot lockup.

Thanks to: lattera (shawn.webb@hardenedbsd<dot>org)
MFC after: 3 days
X-MFC-to: stable/10
X-MFC-with: r280925,280974-280976,281176-281179

Partially revert r280925: Back to previous mode-endings based on feedback

MFC after: 3 days
X-MFC-with: r280925
Reported by: lattera

Solve dreaded "dictionary full" issue by segregating words into new
vocabularies delay-processing, password-processing, version-processing,
frame-drawing, menu-infrastructure, menu-namespace, menu-command-helpers,
and menusets-infrastructure. The net effect is to remove almost 200
definitions from the main forth vocabulary reducing the dictionary size
by over 50%. The chances of hitting "dictionary full" should be greatly
reduced by this patch.

MFC after: 3 days
X-MFC-to: stable/10

Optimize bootmsg to use fg/bg/me from screen.4th
NB: screen.4th is included by way of check-password.4th

MFC after: 3 days
X-MFC-to: stable/10

Comments. No functional change(s).

MFC after: 3 days
X-MFC-to: stable/10

loader: implement multiboot support for Xen Dom0

Implement a subset of the multiboot specification in order to boot Xen
and a FreeBSD Dom0 from the FreeBSD bootloader. This multiboot
implementation is tailored to boot Xen and FreeBSD Dom0, and it will
most surely fail to boot any other multiboot compilant kernel.

In order to detect and boot the Xen microkernel, two new file formats
are added to the bootloader, multiboot and multiboot_obj. Multiboot
support must be tested before regular ELF support, since Xen is a
multiboot kernel that also uses ELF. After a multiboot kernel is
detected, all the other loaded kernels/modules are parsed by the
multiboot_obj format.

The layout of the loaded objects in memory is the following; first the
Xen kernel is loaded as a 32bit ELF into memory (Xen will switch to
long mode by itself), after that the FreeBSD kernel is loaded as a RAW
file (Xen will parse and load it using it's internal ELF loader), and
finally the metadata and the modules are loaded using the native
FreeBSD way. After everything is loaded we jump into Xen's entry point
using a small trampoline. The order of the multiboot modules passed to
Xen is the following, the first module is the RAW FreeBSD kernel, and
the second module is the metadata and the FreeBSD modules.

Since Xen will relocate the memory position of the second
multiboot module (the one that contains the metadata and native
FreeBSD modules), we need to stash the original modulep address inside
of the metadata itself in order to recalculate its position once
booted. This also means the metadata must come before the loaded
modules, so after loading the FreeBSD kernel a portion of memory is
reserved in order to place the metadata before booting.

In order to tell the loader to boot Xen and then the FreeBSD kernel the
following has to be added to the /boot/loader.conf file:

xen_cmdline="dom0_mem=1024M dom0_max_vcpus=2 dom0pvh=1 console=com1,vga"
xen_kernel="/boot/xen"

The first argument contains the command line that will be passed to the Xen
kernel, while the second argument is the path to the Xen kernel itself. This
can also be done manually from the loader command line, by for example
typing the following set of commands:

OK unload
OK load /boot/xen dom0_mem=1024M dom0_max_vcpus=2 dom0pvh=1 console=com1,vga
OK load kernel
OK load zfs
OK load if_tap
OK load ...
OK boot

Sponsored by: Citrix Systems R&D
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D517

For the Forth bits:
Submitted by: Julien Grall <julien.grall AT citrix.com>

Add a try-include word (which acts the same as "include") and use it to
conditionally include (but ignore failures) /boot/loader.rc.local and
/boot/menu.rc.local -- to make customizing the menu easier.

Reviewed by: alfred
Discussed on: -hackers

Extend loader_delay as-documented in beastie.4th(8) and delay.4th(8), making it available to architectures that do not use or load the beastie
menu. This is reported to save headaches on some PPC systems where unload
followed by load does not produce the desired results wherein if-given
the opportunicy to abort the initial loading sequence, you can customize
the first load.

Reviewed by: nwhitehorn, kan
Discussed on: -current

Defer loading of kernel and modules if the beastie menu is enabled. Add a
kernel selection menu to the beastie menu. List of kernels is taken from
`kernels' in loader.conf(5) as a space (or comma) separated list of names
to display (up to 9). If not set, default value is "kernel kernel.old".
Does not validate that kernels exist because the next enhancement will be
to allow selection of the root device.

Discussed on: -current
MFC after: 3 days

Add ontop of my existing boot loader Forth code, including support for
submenus. See menusets.4th(8) for additional details including examples.

Discussed on arch and recommended for inclusion at the devsummit.

This change does not alter the appearance or user experience, only enhances
possibilities.

Reviewed by: adrian (co-mentor)
Approved by: adrian (co-mentor)

Add an echo to say we're "Booting..." when the overloaded "boot" Ficl word is
executed to better differentiate between loader-specific errors and kernel-
specific errors (if ever any of either).

This type of functionality hasn't been required before the introduction of the
advanced menu system (r222417). Adding this functionality will help different-
iate errors at the loader-level such as a BTX halt caused by heap exhaustion
and errors that may be involved with executing the kernel (wrong architecture
for example). A user can learn that messages before "Booting..." are related to
the loader(8) environment and it's Forth-ilk, while those after are not
related to loader(8) -- the point that loader(8) has ``left the building''.

This patch also includes a man-page update to color.4th(8) as the color logic
moves to a lower-level (from being included by beastie.4th to being included
by loader.4th).

After noticing a delay between execution of the overloaded "boot" FICL word and
the display of text on-screen, gcooper confirmed that the introduction of a
builtin memory test (disabled by adding hw.memtest.tests="0" to loader.conf(5))
was the cause of the delay.

This patch adds an echo to produce "Booting..." when the overloaded "boot" word
is executed (this includes from the interactive command-prompt on all arches,
from the menu system on arches that run the beastie menu, and even those arches
that run the menu but disable it by setting beastie_disable="YES" in
loader.conf(5)). When loader_color="YES" in loader.conf(5), the same message is
produced but in white text on a blue background (only the letters produced have
this background -- opposed to perhaps the entire line).

Get rid of a spurious warning on the console when booting the kernel
from the interactive loader(8) prompt and beastie_disable="YES" is set
in loader.conf(5). In this case menu.rc is not evaluated and consequently
menu-unset does not have a body yet. This results in the ficl warning
"menu-unset not found" when try-menu-unset invokes menu-unset.

Check for beastie_disable="YES" explicitly, so that the try-menu-unset
word will not attempt to invoke menu-unset because the menu will have
never been configured. [1]
Use the sfind primitive as a last resort as an additional safer approach
conjuring a foreign word safely. [2]

PR: kern/163938
Submitted by: Devin Teske [1]
Reviewed by: Devin Teske [2]
Reported and tested by: dim
MFC after: 1 week
X-MFC with: r228985

Unset the environment variables associated with individual menu items
before invoking the kernel.

Quoting submitter:

The issue is with the new boot loader menu. It adds many loader variables
including ones that contain ANSI color escapes.

Obviously, these ANSI codes don't play well with serial consoles when
kenv(1) is executed without arguments (reports vary as to what happens,
but it's never pretty).

The net-effect is that kenv(1) no longer reports menu-related variables.

In essence, kenv(1) output should now appear the same as on RELENG_8
(which lacks the new boot loader and didn't use any such variables).
Thus, restoring serial console glory.

Submitted by: Devin Teske <devin dott teske fisglobal.com>
MFC after: 2 weeks

New boot loader menus from Devin Teske.
Discussed on hackers and recommended for inclusion into 9.0 at the devsummit.
All support email to devin dteske at vicor dot ignoreme dot com .

Submitted by: dteske at vicor dot ignoreme dot com
Reviewed by: me and many others

This patch introduces a number of simplifications to the Forth
functions used in the bootloader. The goal is to make the code more
readable and smaller (especially because we have size issues
in the loader's environment).

High level description of the changes:
+ define some string manipulation functions to improve readability;
+ create functions to manipulate module descriptors, removing some
duplicated code;
+ rename the error codes to ESOMETHING;
+ consistently use set_environment_variable (which evaluates
$variables) when interpreting variable=value assignments;

I have tested the code, but there might be code paths that I have
not traversed so please let me know of any issues.

Details of this change:

--- loader.4th ---
+ add some module operators, to remove duplicated code while parsing
module-related commands:

set-module-flag
enable-module
disable-module
toggle-module
show-module

--- pnp.4th ---
+ move here the definition related to the pnp devices list, e.g.
STAILQ_* , pnpident, pnpinfo

--- support.4th ---
+ rename error codes to capital e.g. ENOMEM EFREE ... and do obvious
changes related to the renaming;
+ remove unused structures (those relevant to pnp are moved to pnp.4th)
+ various string functions
- strlen removed (it is an internal function)
- strchr, defined as the C function
- strtype -- type a string to output
- strref -- assign a reference to the string on the stack
- unquote -- remove quotes from a string

+ remove reset_line_buffer

+ move up the 'set_environment_variable' function (which now
uses the interpreter, so $variables are evaluated).
Use the function in various places

+ add a 'test_file function' for debugging purposes

MFC after: 4 weeks

Remove more Alpha bits from the boot code including fixing several
stale comments.

Add support for reading an additional loader configuration file. By default,
this is called /boot/nextboot.conf. This file is required to have it's first
line be nextboot_enable="YES" for it to be read. Also, this file is
rewritten by the loader to nextboot_enable="NO"<space> after it is read.
This makes it so the file is read exactly once. Finally, the nextboot.conf
is removed shortly after the filesystems are mounted r/w.

Caution should be taken as you can shoot yourself in the foot. This is only
the loader piece. There will be a tool called nextboot(8) that will manage
the nextboot.conf file for you. It is coming shortly.

Reviewed by: dcs
Approved by: jake (mentor)

- Add 'fwrite' and 'fseek' words for writing to and seeking on files.
- Change the 'fopen' keyword to accept a mode parameter. Note that this
will break existing 4th scripts that use fopen. Thus, the loader
version has been bumped and loader.4th has been changed to check for a
sufficient version on i386 and alpha. Be sure that you either do a full
world build or install or full build and install of sys/boot after this
since loader.old won't work with the new 4th files and vice versa.

PR: kern/32389
Submitted by: Jonathan Mini <mini@haikugeek.com>
Sponsored by: ClickArray, Inc.

Go back to 1.20 again.

Revert 1.20, as it causes mysterious problems to the Alpha people.

Allocate more memory if necessary.

Now I see the error of my ways.

Previous revision of this file changed the "boot" commands to take
no arguments from the stack. This is only valid in the case where
a kernel has not been loaded. In that case, load_kernel_and_modules
will be called, which takes a list of arguments from the stack.

When a kernel is presently loaded, though, the list of arguments must
be passed to the boot command, which was the behaviour before the last
revision.

Fix things for both cases.

Noticed by: S-Max and others on that chat room

What could possibly have possessed me to forget the "0 (arguments)"
in two of the three boot words in the "boot" redefinition, I have no
clue. Fix it.

Noticed by: bp
Noticed by: adrian

Check for the correct minimum version required by the current code.

I hope I got this right... :-)

Use _ instead or - where proper, according to the style I have been
using.

Overload "?" so it will also show loader.4th commands.

Solve a name clash.
Add something to help debugging.

Both boot and boot-conf were using a different algorithm from the one
used by start to find the kernel. Fix this.

Also, boot would proceed immediately in the absence of a path as
argument. Check first if a kernel has already been loaded, and, if
not, fall back to load kernel&modules behavior.

Some further factorizing. I deem this code to be mostly readable by
now! :-)

Many thanks to: Makoto MATSUSHITA <matusita@jp.FreeBSD.org>

Factorize, reorganize, and move code around.

The boot-conf and boot code had various bugs, and some of it was big,
ugly, unwieldy, and, sometimes, plain incorrect. I'm just about
completely replaced these ugly parts with something much more manageable.

Minor changes were made to the well-factorized parts of it, to accomodate
the new code.

Of note:

* make sure boot-conf has the exact same behavior wrt boot order
as start.

* Correct both boot and boot-conf so they'll work correctly when
compiled in, as they both had some bugs, minor and major.

* Remove all the crud from loader.4th back into support.4th, for
the first time since boot-conf was first improved. Hurray!

I'm fairly satisfied with the code at this time. Time to see about those
man pages...

First tackle at trying to handle the New Deal on kernels.

Load the first of the following kernels to be found:

${kernel} if ${kernel} is an absolute path
/boot/${kernel}/${kernel}
/boot/${kernel}/${bootfile}
${kernel}/${kernel}
${kernel}/${bootfile}
${kernel}
${bootfile}

The last instance of ${kernel} and ${bootfile} will be treated as a
list of semicolon separated file names, and each will be tried in turn,
from left to right.

Also, for each filename loader(8) will try filename, filename.ko,
filename.gz, filename.ko.gz, in that order, but that's not related
to this code.

This resulted in a major reorganization of the code, and much of what
was accumulating on loader.4th was rightly transfered to support.4th.

The semantics of boot-conf and boot also changed. Both will try to load
a kernel the same as above.

After a kernel was loaded, the variable module_path may get changed. Such
change will happen if the kernel was found with a directory prefix. In
that case, the module path will be set to ${directory};${module_path}.

Next, the modules are loaded as usual.

This is intended so kernel="xyzzy" in /boot/loader.conf will load
/boot/xyzzy/kernel.ko, load system modules from /boot/xyzzy/, and
load third party modules from /boot/modules or /modules. If that doesn't
work, it's a bug.

Also, fix a breakage of "boot" which was recently introduced. Boot without
any arguments would fail. No longer. Also, boot will only unload/reload
if the first argument is a path. If no argument exists or the first
argument is a flag, boot will use whatever is already loaded. I hope this
is POLA. That behavior is markedly different from that of boot-conf, which
will always unload/reload.

The semantics introduced here are experimental. Even if the code works,
we might decide this is not the prefered behavior. If you feel so, send
your feedback. (Yeah, this belongs in a HEADS UP or something, but I've
been working for the past 16 hours on this stuff, so gimme a break.)

Enhance boot-conf.

Now boot-conf can also receive parameters to be passed to the kernel
being booted. The syntax is the same as in the boot command, so one
boots /kernel.OLD in single-user mode by typing:

boot-conf /kernel.OLD -s instead of
boot-conf -s /kernel.OLD

The syntax still supports use of directory instead of file name, so

boot-conf kernel.OLD -s

may be used to boot /boot/kernel.OLD/kernel.ko in single-user mode.

Notice that if one passes a flag to boot-conf, it will override the
flags set in .conf files, but only for that invocation. If the user
aborts the countdown and tries again without passing any flags, the
flags set in .conf files will be used.

Some factorization was done in the process of enhancing boot-conf,
as it has been growing steadly as features are getting added, becoming
too big for a Forth word. It still could do with more factorization,
as a matter of fact.

Override the builtin "boot" with something based on boot-conf. It will
behave exactly like boot-conf, but booting directly instead of going
through autoboot.

Since we are now pairing kernel and module set in the same directory,
this change to boot makes sense.

Revert to 1.8

With apologies to dcs, temporarily comment out the version check code. It
is failing for everybody that I have spoken with that has tried it.

FreeBSD/i386 bootstrap loader, Revision 0.8
(root@outback.netplex.com.au, Tue Jun 13 23:26:49 PDT 2000)
Loader version 0.3+ required
Aborted!
start not found

Note that the 0.3+ message is from inside the arch-alpha block, not the
i386 block of code. And even then, 0.8 is higher than 0.3.

This prevents the rest of the loader.conf stuff working. :-/

The word environment? returns a flag indicating whether the variable
was found or not. Fix it's usage. Alas, it caused no problem before,
besides leaving garbage in the stack, because refill, used by [if]
[else] [then], was broken.

Put some version checking.

Modify boot-conf so it can take a kernel or directory name as
a parameter and dtrt.

Also, make boot-conf always unload first. There wasn't really any
point in not doing this, as the kernel _has_ to be loaded before
any other modules.

Tested by: dwhite

Add silly password feature. If people want to depend on a flawed
security measures, so be it. It costs us almost nothing.

Document some code in support.4th that I was unable to understand
just by reading.

$Id$ -> $FreeBSD$

Add "initialize" to loader.4th, so running "start" is not actually needed.

Add loader.4th man page (worst man page I ever wrote -- reviews
are welcomed). Correct a nasty bug in loader.4th before anyone
notices, add a couple of features.

New loader.rc stuff.

Reviewed by: jkh