kboot: Add our own lua bindings

Create a small wrapper around the new flua hash module so we can use it
here too. There's no 4th bindings, nor will they be created.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D43874

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

stand: remove mips support

As part of decommissioning mips support, remove the boot loader
support. Do this in advance of other boot loader work to limit the
amount of work that will be thrown away.

Sponsored by: Netflix
Reviewed by: jrtc27
Differential Revision: https://reviews.freebsd.org/D33377

powerpc64le: stand fixes

Fix boot1 and loader on PowerPC64 little-endian (LE).

Due to endian issues, boot1 couldn't find the UFS boot partition
and loader wasn't able to load the kernel. Most of the issues
happened because boot1 and loader were BE binaries trying to access
LE UFS partitions and because loader expects the kernel ELF image
to use the same endian as itself.

To fix these issues, boot1 and loader are now built as LE binaries
on PPC64LE. To support this, the functions that call OpenFirmware
were enhanced to correctly perform endian conversion on its input
and output arguments and to change the CPU into BE mode before
making the calls, as OpenFirmware always runs in BE. Besides that,
some other small fixes were needed.

Submitted by: bdragon (initial version)
Reviewed by: alfredo, jhibbits
Sponsored by: Instituto de Pesquisas Eldorado (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D32160

stand: Add MK_PIE=no to defs.mk

There's no need to build both pie and non-pie .o's for stand. There's
some other build thing with MK_BEAR_SSL=yes and/or MK_LOADER_VERIEXEC=yes
that causes the pie build to fail that the 'ar' stage now. Since we don't
need the PIE stuff and the non-PIE stuff, disable PIE for the boot loader.

Reviewed by: emaste
Sponsored by: Netflix

Prefer MK_SSP=no to SSP_CFLAGS=

It is more idiomatic. CFLAGS is only augmented with $SSP_CFLAGS when
$MK_SSP != "no".

Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31401

Fix build of stand/ when building world with ASAN

The userboot/test program links against the default userspace libraries
(e.g. shared libgcc_s.so) that will be instrumented if WITH_ASAN is set.
All other programs link against libsa instead of libc and therefore can't
use the sanitizer runtime library. To fix the stand/ build with
sanitizers, we disable MK_ASAN/MK_UBSAN if -nostdlib is found in the
LDFLAGS (i.e. we are using libsa instead of libc).

Reviewed By: imp, tsoome
Differential Revision: https://reviews.freebsd.org/D31047

loader: make sure CPUTYPE is ignored when building

CPUTYPE?=native causes -march=native to be added to the command
line. When the host machine is haswell, this causes some versions of
clang to generate code that can't execute in the efi boot loader
environment. Set _CPUCFLAGS= to undo what's done bsd.cpu.mk. bsd.cpu.mk
is included too early to control with NO_CPU_CFLAGS here. The only other
option is to put that in all the Makefiles, and this is less tedious and
error prone.

PR: 194641
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D31187
MFC After: 1 week

[PowerPC64LE] Set up powerpc.powerpc64le architecture

This is the initial set up for PowerPC64LE.

The current plan is for this arch to remain experimental for FreeBSD 13.

This started as a weekend learning project for me and kinda snowballed from
there.

(More to follow momentarily.)

Reviewed by: imp (earlier version), emaste
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D26399

[PowerPC] Remove obsolete MK_LOADER_FORCE_LE

In D12421, the ability to compile stand/ in little-endian was added, with the
intention to extend loader.kboot to run in Petitboot.

However, no further work was done, as the kernel then gained self-execution
capabilities as Petitboot was taught to load FreeBSD kernels directly.

The FreeBSD installer on powerpc64 (on POWER8 and POWER9) uses
/boot/etc/kboot.conf instead of loader.

As this option does nothing but cause stand/ to be miscompiled and actively
causes confusion, remove it.

(I have a functioning petitboot loader in my local tree, however, it turned
out to be quite inconvient to use due to the current petitboot plugin design
so I put it on hold.)

Reviewed by: emaste, imp, jhibbits
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D26430

Add defines for OpenZFS variables

OZFS is the top of the OpenZFS tree (aka src/sys/contrib/openzfs).
ZFSOSSRC is the path to the OepnZFS sources
ZFSOSINC is the path to the OepnZFS includes

MFC After: 3 days

Create CFLAGS_EARLY.file for boot loader.

Some external code requires a specific set of include paths to work
properly since it emulates the typical environment the code is used
in. Enable this by creating a CFLAGS_EARLY.file variable that can be
used to build this stack. Otherwise the include stack we build for
stand programs may get in the way. Code that uses this feature has to
tolerate the normal stack of inclues being last on the list (and
presumably unused), though.

Generally, it it should only be used for the specific include
directories. Defines and that sort of thing should be done in the
normal CFLAGS variable. There is a global CFLAGS_EARY hook as well for
everything in a Makefile.

stand: use portable ln -n instead of ln -h

This fixes the build on Linux

Differential Revision: https://reviews.freebsd.org/D24783

Really fix cleandir after r362973

I made an attempt to fix this in r362978, but all it really did was
confine the issue to the $MACHINE_CPUARCH == "riscv" case. The real
problem is that LINKER_FEATURES is not defined here, so bsd.linker.mk
needs to be included.

This error with cleandir only occurs when META_MODE is disabled, which
explains why it was missed by both CI and myself.

Note that this effectively reverts r362978.

Reported by: mjg
Reviewed by: imp, kevans (in IRC)

Fix cleandir target post r362973

Reported by: mmacy

RISC-V boot1.efi and loader.efi support

This implementation doesn't have any major deviations from the other EFI
ports. I've copied the boilerplate from arm and arm64.

I've tested this with the following boot flows:
OpenSBI (M-mode) -> u-boot (S-mode) -> loader.efi -> FreeBSD
OpenSBI (M-mode) -> u-boot (S-mode) -> boot1.efi -> loader.efi -> FreeBSD

Due to the way that u-boot handles secondary CPUs, OpenSBI >= v0.7 is required,
as the HSM extension is needed to bring them up explicitly. Because of this,
using BBL as the SBI implementation will not be possible. Additionally, there
are a few recent u-boot changes that are required as well, all of which will be
present in the upcoming v2020.07 release.

Looks good: emaste
Differential Revision: https://reviews.freebsd.org/D25135

Remove tests for obsolete compilers in the build system

Assume gcc is at least 6.4, the oldest xtoolchain in the ports tree.
Assume clang is at least 6, which was in 11.2-RELEASE. Drop conditions
for older compilers.

Reviewed by: imp (earlier version), emaste, jhb
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D24802

Build the arm64 loader with -ffixed-x18

This stops the compiler from using the x18 register. Some UEFI
implementations assume this will be preserved when calling the Boot
Services.

MFC after: 2 weeks
Sponsored by: Innovate UK

[PowerPC] Switch powerpc and powerpcspe to lld

Now that LLD 10 is out, and required patches have landed, we are now ready
to finally switch away from the ancient in-tree ld.bfd.

Special thanks to Fangrui Song for many hours of work on getting the
32-bit powerpc lld ready for prime-time.

Reviewed by: emaste (earlier revision), jhibbits
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D24111

Move stand/ofw/libofw to stand/libofw.

Since rS330365, there has been no particular reason for libofw to be in a
subdirectory of ofw. Move libofw up a level to make it fit in better with
the other top level libraries.

Also add a LIBOFWSRC to stand/defs.mk to match what all the other
libraries are doing.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D23000

[PowerPC64] Use ld.bfd to build LIB32 and STAND - when using llvm

Summary:
This patch is to support ongoing work for replacing "GCC/BFD" by "CLANG/LLD" on
target PowerPC64 [1], by proposing a way to specify and/or locate a secondary
ld.bfd linker.
This is necessary as LLD currently doesn't support PowerPC 32 bits, so we keep
using BFD for the 32 bit stuff on PowePC64(LIB32 compatibility and
STAND/slof/loader.)

- creates LD_BFD variable pointing to ld.bfd
- use LD_BFD as linker for LIB32/compat
- Default behavior for other platforms aren't changed.

[1] https://wiki.freebsd.org/powerpc/llvm-elfv2

Submitted by: alfredo.junior_eldorado.org.br
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D20261

Add flua to the base system, install to /usr/libexec

FreeBSDlua ("flua") is a FreeBSD-private lua, flavored with whatever
extensions we need for base system operations. We currently support a subset
of lfs and lposix that are used in the rewrite of makesyscall.sh into lua,
added in r354786.

flua is intentionally written such that one can install standard lua and
some set of lua modules from ports and achieve the same effect.

linit_flua is a copy of linit.c from contrib/lua with lfs and lposix added
in. This is similar to what we do in stand/. linit.c has been renamed to
make it clear that this has flua-specific bits.

luaconf has been slightly obfuscated to make extensions more difficult. Part
of the problem is that flua is already hard enough to use as a bootstrap
tool because it's not in PATH- attempting to do extension loading would
require a special bootstrap version of flua with paths changed to protect
the innocent.

src.lua.mk has been added to make it easy for in-tree stuff to find flua,
whether it's bootstrap-flua or relying on PATH frobbing by Makefile.inc1.

Reviewed by: brooks, emaste (both earlier version), imp
Differential Revision: https://reviews.freebsd.org/D21893

stand: consolidate knowledge of lua path

Multiple places coordinate to 'know' where lua scripts are installed. Knock
this down to being formally defined (and overridable) in exactly one spot,
defs.mk, and spread the knowledge to loaders and liblua alike. A future
commit will expose this to lua as loader.lua_path, so it can build absolute
paths to lua scripts as needed.

MFC after: 1 week

Add support for hypervisor check on x86

Add ficl words for isvirtualized
and move ficl inb and outb words to ficl/x86/sysdep.c
so can be shared by i386 and amd64

Reviewed by: imp bdrewery
MFC after: 1 week
Sponsored by: Juniper Networks
Differential Revision: https://reviews.freebsd.org/D22069

loader: --gc-sections needs sections to work with

--gc-sections is not really useful unless we generate sections with
-ffunction-sections -fdata-sections

While there, i386/loader would win from --gc-sections too.

loader: use teken teminal emulator for x86 and uefi

Replace mini cons25 emulator with teken, this does enable us proper console
terminal for loader and will make it possible to implement different
back end callbacks to draw to screen.

At this time we still only "draw" in text mode.

stand: push LIBC_SRC up into defs.mk

Other parts of stand/ that don't use libsa will need to grab bits from libc
shortly. Push LIBC_SRC up to defs.mk in advance of this so that they can use
it, and rename it to LIBCSRC to match the convention of the rest of the *SRC
variables in this file.

Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D21282

Move inclusion of src.opts.mk later.

src.opts.mk includes bsd.own.mk. This in turn defines CTFCONVERT_CMD
depending on the MK_CTF value. We then set MK_CTF to no, which has no
real effect. The solution is to set all the MK_foo values before
including src.opts.mk.

This should stop the cdboot binary from exploding in size for releases
built WITH_CTF=yes in src.conf.

Sponsored by: Netflix

powerpcspe: Don't crash the loader on ubldr with SPE instructions.

-msoft-float seems to be insufficient for disabling the SPE on powerpcspe.
Force it off with -mno-spe as well. This prevents a crash in ubldr on
powerpcspe.

Be a little conservative about when to force size optimizations.

Reports have come in that there's issue with powerpc and sparc64 since
we've switched to using -Oz / -Os. We don't strictly need them for
!x86, so be conservative about when we enable them.

Approved by: re@ (gjb)
Differential Revision: https://reviews.freebsd.org/D17016

Enable 'C'-compressed ISA extension.

This was disabled recently due to lack of support in KDB disassembler
and DTrace FBT provider. Support for 'C'-extension to both of these was
added, so we can now enable 'C'-extension.

This reduces size of the kernel important for low-end embedded devices,
and saves cache footprint for high perfomance machines.

Approved by: re (kib)
Sponsored by: DARPA, AFRL

stand: Flip the default interpreter to Lua

After years in the making, lualoader is ready to make its debut. Both
flavors of loader are still built by default, and may be installed as
/boot/loader or /boot/loader.efi as appropriate either by manually creating
hard links or using LOADER_DEFAULT_INTERP as documented in build(7).

Discussed with: imp
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D16795

stand: Use -Oz/-Os for all loader/stand builds.

While we're not super size constrained, the x86 BIOS /boot/loader has
to be less than about 520k-530k to be reliable. The LUA loader is at
this size today. -Oz saves 15-20% on the size, keeping us safely small
enough (comparable to where we were with the 4th loader). This will
also help with sjg's work on bringing in bearssl, though we may again
be looking for space in the LUA loader.

Size table for clang 6.0.0:
default -O1 -Os -Oz
4th 442368 417792 389120 376832
lua 524288 479232 446464 430080

Tested by: kevans91@ (ubldr on armv7), dhw@ (loader on amdy64)
Differential Revision: https://reviews.freebsd.org/D16724

Create a loader for each interpreter for x86 BIOS and all EFI

Create loader_{4th,lua,simp}{,.efi}. All of these are installed by
default. Create LOADER_DEFAULT_INTERP to specify the default
interpreter when no other is specified. LOADER_INTERP is the current
interpreter language building. Turn building of lua on by default to
match 4th. simploader is a simplified loader build w/o any interpreter
language (but with a simple loader). This is the historic behavir you
got with WITHOUT_FORTH. Make a hard link to the default loader. This
has to be a hard link rather than the more desirable soft link because
older zfsboot blocks don't support symlinks.

RelNotes: Yes
Differential Revision: https://reviews.freebsd.org/D16705

Disable 'C'-compressed ISA extension.

It works excellent, but KDB disassembler and DTrace FBT provider for
RISC-V do lack support for it. They currently handle 4-byte instructions
only, while C-compressed ISA extension introduces 2-byte instructions
freely mixing them together.

So disable it for now.

Reviewed by: markj@
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D16436

Extend loader(8) geli support to all architectures and all disk-like devices.

This moves the bulk of the geli support from lib386/biosdisk.c into a new
geli/gelidev.c which implements a devsw-type device whose dv_strategy()
function handles geli decryption. Support for all arches comes from moving
the taste-and-attach code to the devopen() function in libsa.

After opening any DEVT_DISK device, devopen() calls the new function
geli_probe_and_attach(), which will "attach" the geli code to the open_file
struct by creating a geli_devdesc instance to replace the disk_devdesc
instance in the open_file. That routes all IO for the device through the
geli code.

A new public geli_add_key() function is added, to allow arch/vendor-specific
code to add keys obtained from custom hardware or other sources.

With these changes, geli support will be compiled into all variations of
loader(8) on all arches because the default is WITH_LOADER_GELI.

Relnotes: yes
Sponsored by: Microchip Technology Inc
Differential Revision: https://reviews.freebsd.org/D15743

Move ZFS files into libsa

Move the libzfs stuff into libsa. There's no need for it to be a
separate library. The separate library adds to the issues of build
ordering that we see from time to time. Move the filesystem support
into libsa, like all the other filesystem support rather than making
zfs the odd-duck out.

Discussed with: allanjude@

stand: move libgeliboot into libsa.

Reduce by 1 the number of crazy libraries we need in stand by moving
geli into libsa (where architecturally it belonged all along). This
just moves things around without any code changes.

Fix build of stand with base gcc

* Make autoboot() a static function in stand/common/boot.c, so it does
not shadow local variables in gptboot.c and zfsboot.c.
* Remove -Winline from the Makefiles for gptboot, gptzfsboot and
zfsboot, as gcc will always fail to inline some functions, and there
is nothing we can do about it.
* For gcc <= 4.2.1, silence -Wuninitialized for isoboot, as it produces
a false positive warning.
* Remove deprecated and unnecessary -mcpu=i386 flag from stand/defs.mk,
as there is already a -march=i386 flag further in the file.

Reviewed by: imp
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D15628

Don't add links or cleanfiles for NO_OBJ case, in addition to not
creating them. Move them under the if after the all: target. They are
just defines, so it doesn't really matter where we have them.

MFC After: 3 days

Add NO_OBJ to those directories that don't make anything.

For directories that don't many anything, add NO_OBJ=t just before we
include bsd.init.mk. This prevents them from creating an OBJ
directory. In addition, prevent defs.mk from creating the machine
related links in these cases. They aren't needed and break, at least
on stable, the read-only src tree build.

Floaty McFloatface is funnier...

Submitted by: emaste@

Do not include float interfaces when using libsa.

We don't support float in the boot loaders, so don't include
interfaces for float or double in systems headers. In addition, take
the unusual step of spiking double and float to prevent any more
accidental seepage.

Eliminate bsd.stand.mk and -fPIC 32-bit intel builds

OK. We don't really need a bsd.stand.mk, and it was causing a -fPIC
for the toolchain to be added (bogusly) when building on amd64. Pull
all relevant defs back into defs.mk and delete bsd.stand.mk.

This saves about 15-20k on i386 loader and zfsloader which when
combined with Lua give us a lot more stack space in those constrained
environments.

Add Lua as a scripting langauge to /boot/loader

liblua glues the lua run time into the boot loader. It implements all
the runtime routines that lua expects. In addition, it has a few
standard 'C' headers that nueter various aspects of the LUA build that
are too specific to lua to be in libsa. Many refinements from the
original code to improve implementation and the number of included lua
libraries. Use int64_t for lua_Number. Have "/boot/lua" be the default
module path. Numerous cleanups from the original GSoC project,
including hacking libsa to allow lua to be built with only one change
outside luaconf.h.

Add the final bit of lua glue to bring in liblua and plug into the
multiple interpreter framework, previously committed.

Add LOADER_LUA option, currently off by default.

Presently, this is an experimental option. One must opt-in to using
this by defining WITH_LOADER_LUA and WITHOUT_FORTH. It's been
lightly tested, so keep a backup copy of your old loader handy.
The menu code, coming in the next commit, hasn't been exhaustively
tested. A LUA boot loader is 60k larger than a FORTH one, which is
80k larger than a no-interpreter one. Subtle changes in size
may tip things past some subtle limit (the binary is ~430k now
when built with LUA). A future version may offer coexistance.

Bump FreeBSD version to 1200058 to mark the milestone.

Pedro Souza's 2014 Summer of Code project. Rui Paulo, Pedro Arthur,
Zakary Nafziger and Wojciech A. Koszek also contributed. Warner Losh
reworked it extensively into its current form.

Obtained from: https://wiki.freebsd.org/SummerOfCode2014/LuaLoader
Sponsored by: Google Summer of Code
Relnotes: Yes
MFC After: 1 month
Differential Review: https://reviews.freebsd.org/D14295

Fix build of userboot.so

Since it's not possible to unset a variable easily, create a new
variable 'PIC' to signal that we are creating a shared object that we
want to install. defs.mk refains from defining NO_PIC and ITNERALLIB
when PIC is defined. This unbreaks userboot.so building.

Centralize several variables.

MK_CTF, MK_SSP, MK_PROFILE, NO_PIC, and INTERNALLIB are always the
same, so set them in defs.mk. MAN= is common, so set it here too.
This removes a lot of boring repetition from the Makefiles that added
almost no value.

Fix ubldr. uboot/lib uses defines for the loader. It's part of the
loader, but not compile as loader (it's building a library), so we
can't just include loader.mk for the defines. Move LOADER_DISK_SUPPORT
back to defs.mk for the moment.

Sponsored by: Netflix

libficl is only ever used in a loader (never a boot) program. Move it
to loader.mk.

Sponsored by: Netflix

Move loader help file definitions to being 100% inside of loader.mk.

HELP_FILES is a loader only thing, so move it to loader.mk. Only
generate the help file if HELP_FILES is defined. Adjust Makefiles to
new convention. Fix a few cases where ${.CURDIR}/ was missing
resulting in missing bits from the help files.

Sponsored by: Netflix

Move loader-only defines to loader.mk from defs.mk

Produces the same .o's, verified with md5.

Sponsored by: Netflix

Fix a comment to be more accurate

Stop building with the standard system headers.

Building with the standard system headers isn't a perfect match to the
stand environment. Instead, copy over the files we know are safe to
use and constrain what else is used. We use -nostdinc to achieve this.

This also fixes issues with building 32-bit libraries on amd64
sometimes pulling in the wrong cpufunc.h giving an error now that we
stop on errors. It will also enable an easier transition to lua boot.

Sponsored by: Netflix

Now it's safe to bump WARNS to 1.

Sponsored by: Netflix

Fix all warnings related to geli and ZFS support on x86.

Default WARNS to 0 still, since there's still some warnings on other
architectures.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D13301

Cleanup CFALGS usage here

Only define the CFLAGS we need.
SSP_CFLAGS is now defined globally, no need to define it here.
Define -D_STANDALONE globally for src/stand builds.

Sponsored by: Netflix

Minor flags cleanup

Move kernel includes and libsa includes together at the top of defs.mk
Move all machine specific defines from Makefile.inc to their friends
in defs.mk.
Add comments and remove now useless junk after the move.

Sponsored by: Netflix

Move sys/boot to stand. Fix all references to new location

Sponsored by: Netflix