MFC Loader Interpreter Coexistance: r337806, r337868, r337914, r337927,
r337989, r338052, r338064, r338407, r338418, r338474

r337806:
Create a loader for each interpreter for x86 BIOS and all EFI

Create loader_{4th,lua,simp}{,.efi}. All of these are installed by
default. Create LOADER_DEFAULT_INTERP to specify the default
interpreter when no other is specified. LOADER_INTERP is the current
interpreter language building. Turn building of lua on by default to
match 4th. simploader is a simplified loader build w/o any interpreter
language (but with a simple loader). This is the historic behavir you
got with WITHOUT_FORTH. Make a hard link to the default loader. This
has to be a hard link rather than the more desirable soft link because
older zfsboot blocks don't support symlinks.

r337868:
stand: Use -Oz/-Os for all loader/stand builds.

While we're not super size constrained, the x86 BIOS /boot/loader has
to be less than about 520k-530k to be reliable. The LUA loader is at
this size today. -Oz saves 15-20% on the size, keeping us safely small
enough (comparable to where we were with the 4th loader). This will
also help with sjg's work on bringing in bearssl, though we may again
be looking for space in the LUA loader.

Size table for clang 6.0.0:
default -O1 -Os -Oz
4th 442368 417792 389120 376832
lua 524288 479232 446464 430080

r337914:
Install links for loader.efi.

r337927:
Add ashldi3 and ashrdi3 to mips.

Now that we're using -Os, mips needs these routines.

r337989:
Copy the boot loader from the new location for the co-existing
loaders.

r338052:
libsa: Add lshrdi3.c for powerpc* and mips

r338064:
Turn back the clock just a little: make userboot.so always be 4th

Turns out there was a hidden dependency we hasn't counted upon. The
host load /boot/userboot.so to boot the VMs it runs. This means that
the change to lua meant suddently that nobody could run their older
VMs because LUA wasn't in 10.0, last month's HardenedBSD, 11.2 or
whatever. Even more than for the /boot/loader* binaries, we need a
good coexistance strategy for this. While that's being designed and
implemented, drop back to always 4th for userboot.so. This will fail
safe in all but the most extreme environments (but lua-only hacks
to .lua files won't be processes in VMs until we fix it).

r338407:
lualoader: Print error messages from command failures at the prompt

Previously lualoader would remain silent, rather than printing
command_errmsg or noting that a command had failed or was not found.

r338418:
userboot: handle guest interpreter mismatches more intelligently

The switch to lualoader creates a problem with userboot: the host is
inclined to build userboot with Lua, but the host userboot's interpreter
must match what's available on the guest. For almost all FreeBSD guests in
the wild, Lua is not yet available and a Lua-based userboot will fail.

This revision updates userboot protocol to version 5, which adds a
swap_interpreter callback to request a different interpreter, and tries to
determine the proper interpreter to be used based on how the guest
/boot/loader is compiled. This is still a bit of a guess, but it's likely
the best possible guess we can make in order to get it right. The
interpreter is now embedded in the resulting executable, so we can open
/boot/loader on the guest and hunt that down to derive the interpreter it
was built with.

Using -l with bhyveload will not allow an intepreter swap, even if the
loader specified happens to be a userboot with the wrong interpreter. We'll
simply complain about the mismatch and bail out.

For legacy guests without the interpreter marker, we assume they're 4th.
For new guests with the interpreter marker, we'll read it and swap over
to the proper interpreter if it doesn't match what the userboot we're using
was compiled with.

Both flavors of userboot are installed by default, userboot_4th.so and
userboot_lua.so. This fixes the build WITHOUT_FORTH as a coincidence, which
was broken by userboot being forced to 4th.

r338474:
Be a little conservative about when to force size optimizations.

Reports have come in that there's issue with powerpc and sparc64 since
we've switched to using -Oz / -Os. We don't strictly need them for
!x86, so be conservative about when we enable them.

MFC r335026:

style(9) remove unnecessary blank tabs.

Obtained from: TrueOS
Sponsored by: iXsystems Inc.

MFC r326276:

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

MFC r325834,r325997,326502: Move sys/boot to stand/

This is effectively a direct commit to stable/11, due to differences between
stable/11 and head. Changes to DTS in sys/boot/fdt/dts were often
accompanied by kernel changes. Many of these were also risc-v updates that
likely had many more dependencies to MFC.

Because of this, sys/boot/fdt/dts remains as-is while everything else in
sys/boot relocates to stand/.

r325834: Move sys/boot to stand. Fix all references to new location

r325997: Remove empty directories.

r326502: Document the sys/boot -> stand move in hier.7 and the top-level README.

MFC r326615: De-const to match changes in userboot.h

MFC Loader Fixes 2017q3: r320547,r320553,r321621,r321844,r321969,r321991,
r322037,r322038,r322039,r322040,r322056,r322074,r322542,r322592,r322593,
r322896,r322923,r323671,r322930,r322931,r322932,r322933,r322934,r322935,
r322936,r322937,r322938,r322939,r322941,r323062,r323063,r323064,r323065,
r323100,r323131,r323174,r323258,r323261,r323272,r323367,r323379,r323389,
r323407,r323428,r323436,r323494,r323496,r323497,r323541,r323554,r323589,
r323707,r323867,r323885,r323886,r323895,r323896,r323897,r323905,r323906,
r323907,r323908,r323909,r323952,r323991,r324099,r324558,r326445,r326609,
r326610

This batch includes a special kludge to fix powerpc loader build; <stdlib.h>
was included after <stand.h> there, causing problems with DEBUG_MALLOC bits.
Include <stdlib.h> a little bit earlier to fix the build with the intention
of removing this when eventually libsa silently replaces stdlib.h with
stand.h.

r320547: Link EFI/uboot loaders with -znotext

r320553: Integer underflow in efipart_realstrategy when I/O starts after end
of disk

r321621: Always set the receive mask in loader.efi.

r321844: Clean up style in print_state(..) and pager_printf(..)

r321969: Fix the return types for printf and putchar to match their libc

r321991: Revert r321969

r322037: Add stpcpy and stpncpy to libstand

r322038: Add definitions and utilities for EFI drivers

r322039: Move EFI ZFS functions to libefi

r322040: Add EFI utility functions to libefi

r322056: Move EFI fmtdev functionality to libefi

r322074: libefi/time.c cstyle cleanup

r322542: loader.efi: repace XXX with real comments in trap.c

r322592: Remove unused defines.

r322593: Define proposed GUID for FreeBSD boot loader variables.

r322896: Make spinconsole platform independent and hook it up into EFI
loader

r322923: Hide length of geli passphrase during boot.

r323671: Fix language used in the r322923.

r322930: Move efi_main into efi/loader

r322931: Cleanup efi_main return type

r322932: Use the loader.efi conventions for the various EFI tables.

r322933: No need for MK_ZFS around these: they are by their nature only
active when MK_ZFS is true.

r322934: _STAND is sometimes defined on the command line. Make the define
here match.

r322935: Fix warnings due to type mismatch.

r322936: Remove useless 'static' for an enum definition.

r322937: Forward declare struct dsk to avoid warnings when building libi386.

r322938: Link in libefi for boot1

r322939: Use efi_devpath_str for debug path info.

r322941: Eliminate redunant device path matching.

r323062: Make efichar.c routines available to libefi.

r323063: boot1.efi: print more info about where boot1.efi is loaded from

r323064: Exit rather than panic for most errors.

r323065: Save where we're booted from

r323100: libstand: nfs_readlink() should return proper return code

r323131: Revert r322941: Eliminate redundant device matching functions

r323174: Fix loader bug causing too many pages allocation when bootloader
is U-Boot

r323258: ucs2len

r323261: Fix armv6 build

r323272: Be consistent and do return (1);

r323367: Mark init_chroot and init_script variables as deprecated.

r323379: It's been pointed out that init_script at least is useful w/o

r323389: loader.efi: chain loader should provide proper device handle

r323407: boot1 generate-fat: generate all templates at once

r323428: r323389 breaks the kernel build when WITHOUT_ZFS is defined in
src.conf

r323436: boot1: remove BOOT1_MAXSIZE default value

r323494: loader should support large_dnode

r323496: libstand: tftp_open() can leak pkt on error

r323497: libefi: efipart_open should check the status from disk_open

r323541: libefi: efipart_realstrategy rsize pointer may be NULL

r323554: Increase EFI boot file size frok 128k to 384k

r323589: loader: biosmem.c cstyle cleanup

r323707: loader: biosmem allocate heap just below 4GB

r323867: libefi: devicename.c cleanups

r323885: libefi: efi_devpath_match() should return bool

r323886: libefi: efipart.c should use calloc()

r323895: libefi: efi_devpath_match local len should be unsigned

r323896: r323885 did miss efilib.h update

r323897: efilib.h: typo in structure member description

r323905: libefi: pdinfo_t pd_unit and pd_open should be unsigned

r323906: libefi: efipart_strategy() should return ENXIO when there is no
media

r323907: libefi: efipart.c cstyle fix for efipart_print_common()

r323908: libefi: efipart_hdinfo_add_filepath should check strtol result

r323909: libefi: define EISA PNP constants

r323952: After the r317886 support for TFTP and NFS can be enable
simultaneously.

r323991: libefi: efipart_floppy() will should not pass acpi pointer if the
HID test fails

r324099: Compile loader as Little-Endian on PPC64/POWER8

r324558: Define prototype for exit and ensure references

r326445: Fix random() and srandom() prototypes to match the standard.

r326609: Make putenv and getenv match the userland definition

r326610: Fix random() prototype to match the system.

PR: 219000 221001 222215
Relnotes: yes ("The length of the geli passphrase is hidden during boot")

MFV r320195: bhyveload: correctly query size of disks

On FreeBSD fstat(2) works fine for querying sizes of plain files,
but not so much for character devices.
So, use DIOCGMEDIASIZE to try to get the correct size for disks
and disk-like devices (e.g. zvols).

PR: 220186

MFC r314659,r314676:

r314659:

usr.sbin: normalize paths using SRCTOP-relative paths or :H when possible

This simplifies make logic/output

r314676:

Fix build after r314656

Some of the changes I introduced to use .ALLSRC were correct in spirit,
but incorrect in reality -- in particular, ../Makefile.inc hadn't been
pulled in via bsd.init.mk (via bsd.lib.mk, bsd.prog.mk), so the value
of .ALLSRC (evaluated immediately) was empty. .include bsd.init.mk
explicitly so we can be certain that the values used as dependencies in
the targets are defined when the target recipe has been evaluated.

Reminder: thou shalt separate out separate functional changes before
committing them.

(YUGE) Pointyhat to: ngie
In collaboration with: bdrewery

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

bhyve: improve memory size documentation

A couple of minor memory size option related nits:

- use common name 'memsize' (instead of 'max-size' or just 'size')
- bhyve: update usage with memsize unit suffix, drop legacy "MB"
unit
- bhyveload: update usage with memsize unit suffix
- bhyve(8): document default size
- bhyveload(8): use memsize formatting like it's done
in bhyve(8)

Reviewed by: wblock, grehan
Approved by: re (kib), wblock, grehan
Differential Revision: https://reviews.freebsd.org/D6952

Merge the projects/release-pkg branch to head.

This allows packaging the base system with pkg(8), including
but not limited to providing the ability to provide upstream
binary update possibilities for non-tier-1 architectures.

This merge is a requirement of the 11.0-RELEASE, and as such,
thank you to everyone that has tested the project branch.

Documentation in build(7) etc. is still somewhat sparse, but
updates to those parts will follow.

Sponsored by: The FreeBSD Foundation

Final pass through bogus svn:mergeinfo removal.

Note, paths part of 'contrib' were left alone for now.

Sponsored by: The FreeBSD Foundation

bhyveload: fix from loading undefined size.

We were setting an incorrect/undefined size and as it came out the st
struct was not really being used at all. This was actually a bug but
by sheer luck it had no visual effect.

CID: 1194320
Reviewed by: grehan

Upgrade our copies of clang, llvm, lldb and compiler-rt to 3.8.0
release.

Please note that from 3.5.0 onwards, clang, llvm and lldb require C++11
support to build; see UPDATING for more information.

Release notes for llvm and clang will soon be available here:
<http://llvm.org/releases/3.8.0/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.8.0/tools/clang/docs/ReleaseNotes.html>

Thanks to Ed Maste, Roman Divacky, Davide Italiano and Antoine Brodin
for their help.

Relnotes: yes

Add option -C to have the guest memory included in core files.
This aids in debugging OS loaders.

Support version 4 of the userboot structure by implementing the
vm_set_register() and vm_set_desc() callbacks.

META MODE: Update dependencies with 'the-lot' and add missing directories.

This is not properly respecting WITHOUT or ARCH dependencies in target/.
Doing so requires a massive effort to rework targets/ to do so. A
better approach will be to either include the SUBDIR Makefiles directly
and map to DIRDEPS or just dynamically lookup the SUBDIR. These lose
the benefit of having a userland/lib, userland/libexec, etc, though and
results in a massive package. The current implementation of targets/ is
very unmaintainable.

Currently rescue/rescue and sys/modules are still not connected.

Sponsored by: EMC / Isilon Storage Division

Improve collation string and locales support

Merge collation support from Illumos and DragonflyBSD.

Locales are now generated with the new localedef(1) tool from CLDR POSIX files.
The generated files are now identified as "BSD 1.0" format.

The libc now only read "BSD 1.0" locales definitions, all other version will be
set to "C"
The localedef(1) tool has been imported from Illumos and modified to use tree(3)
instead of the CDDL avl(3)
A set of tool created by edwin@ and extended by marino@ for dragonfly has been
added to be able to generate locales and the Makefiles from the vanilla CLDR
unicode databases + a universal UTF-8 charmap (by marino@)
Update the locales to unicode v27
Given our regex(3) does not support multibyte (yet) it has been forced to always
use locale C
Remove now unused colldef(1) and mklocale(1)
Finish implementing the numeric BSD extension for ctypes
The number of supported locales has grown from 175 to 250 locales. Among the new
locales: 6 Arabic locales (AE EG JO MA QA SA), Different variations of spanish
locales.
Added new 3 components locales for mn_Cyrl_MN, sr_Cyrl_RS sr_Latn_RS,
zh_Hans_CN, zh_Hant_HK and zh_Hant_TW. Some aliases has been for 2 components
version when possible.

Thanks: Garrett D'Amore (Illumos) who made sure all his work was done under
BSD license!, Edwin Groothuis (edwin@) for the work he made on tools to be able
to generate locales definition usable in freebsd sources out of vanilla CLDR
definitions, John Marino (DragonflyBSD) who first merge the Illumos work into
Dragonfly and spent hours tracking down bugs.

Merge mpsutil(8) branch

mpsutil(8)/mprutil(8) are new utilities for managing LSI Fusion-MPT
2/3 controllers (mps(4) and mpr(4))

For now only informational commands have been implemented.

This utility has been written by scottl@ [1] and polished by myself[2]

Submitted by: scottl
Discussed with: scottl
Relnotes: yes
Sponsored by: Netflix [1]
Sponsored by: Gandi.net [2]

Add option -l for specifying which OS loader to dlopen(3). By default
this is /boot/userboot.so. This option allows for the development and
use of other OS loaders.

Upgrade our copies of clang, llvm, lldb, compiler-rt and libc++ to 3.7.0
release.

Please note that from 3.5.0 onwards, clang, llvm and lldb require C++11
support to build; see UPDATING for more information.

Release notes for llvm and clang can be found here:
<http://llvm.org/releases/3.7.0/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.7.0/tools/clang/docs/ReleaseNotes.html>

Thanks to Ed Maste, Andrew Turner and Antoine Brodin for their help.

Exp-run: antoine
Relnotes: yes

Fix issues detected by 'mandoc -Tlint bhyveload.8'

Pointed out by: wblock
Differential Revision: https://reviews.freebsd.org/D2762

Restructure memory allocation in bhyve to support "devmem".

devmem is used to represent MMIO devices like the boot ROM or a VESA framebuffer
where doing a trap-and-emulate for every access is impractical. devmem is a
hybrid of system memory (sysmem) and emulated device models.

devmem is mapped in the guest address space via nested page tables similar
to sysmem. However the address range where devmem is mapped may be changed
by the guest at runtime (e.g. by reprogramming a PCI BAR). Also devmem is
usually mapped RO or RW as compared to RWX mappings for sysmem.

Each devmem segment is named (e.g. "bootrom") and this name is used to
create a device node for the devmem segment (e.g. /dev/vmm/testvm.bootrom).
The device node supports mmap(2) and this decouples the host mapping of
devmem from its mapping in the guest address space (which can change).

Reviewed by: tychon
Discussed with: grehan
Differential Revision: https://reviews.freebsd.org/D2762
MFC after: 4 weeks

Fix off-by-one in array index bounds check

bhyveload would allow you to create 33 entries on an array that only has 32 slots

Differential Revision: https://reviews.freebsd.org/D2569
Reviewed by: araujo
Approved by: neel
MFC after: 1 week
Sponsored by: ScaleEngine Inc.

Fix overlinking in bhyve:

libvmmapi is actually needed to be linked to libutil, not bhyve nor bhyveload

Upgrade our copy of clang, llvm and lldb to 3.5.0 release.

Please note that this version now requires C++11 support to build; see
UPDATING for more information.

Release notes for llvm and clang can be found here:
<http://llvm.org/releases/3.5.0/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.5.0/tools/clang/docs/ReleaseNotes.html>

Thanks to Ed Maste, Roman Divacky, Andrew Turner, Justin Hibbits and
Antoine Brodin for their invaluable help with this import.

Approved by: portmgr (antoine)
MFC after: 1 month

Convert usr.sbin to LIBADD
Reduce overlinking

Sort command flags in usage output and the manpages.

Provide APIs to directly get 'lowmem' and 'highmem' size directly.

Previously the sizes were inferred indirectly based on the size of the mappings
at 0 and 4GB respectively. This works fine as long as size of the allocation is
identical to the size of the mapping in the guest's address space. However, if
the mapping is disjoint then this assumption falls apart (e.g., due to the
legacy BIOS hole between 640KB and 1MB).

use .Mt to mark up email addresses consistently (part2)

PR: 191174
Submitted by: Franco Fichtner <franco@lastsummer.de>

Add ioctl(VM_REINIT) to reinitialize the virtual machine state maintained
by vmm.ko. This allows the virtual machine to be restarted without having
to destroy it first.

Reviewed by: grehan

Merge the projects/clang-sparc64 branch back to head. This brings in
several updates from the llvm and clang trunks to make the sparc64
backend fully functional.

Apart from one patch to sys/sparc64/include/pcpu.h which is still under
discussion, this makes it possible to let clang fully build world and
kernel for sparc64.

Any assistance with testing this on actual sparc64 hardware is greatly
appreciated, as there will unavoidably be bugs left.

Many thanks go to Roman Divacky for his upstream work on getting the
sparc64 backend into shape.

MFC after: 1 month

ZFS boot support for bhyveload.

Modelled after the i386 zfsloader. However, with no
2nd stage zfsboot to search for a bootable dataset,
attempt a ZFS boot if there is more than one ZFS
dataset found during the disk probe.

sys/boot/userboot/zfs
- build the ZFS boot library

sys/boot/userboot/userboot/
conf.c
- Add the ZFS pool and filesystem tables
devicename.c
- correctly format ZFS devices
main.c
- increase the size of the libstand malloc pool
to account for the increased usage from ZFS buffers
- probe for a ZFS dataset, and if one is
found, attempt to boot from it.

usr.sbin/bhyveload/bhyveload.c
- allow multiple invocations of the '-d' option
to specify multiple disks e.g. a raidz set.
Up to 32 disks are supported.

Tested with various combinations of GPT, MBR, single
and multiple disks, RAID-Z, mirrors.

Reviewed by: neel
Discussed with: avg
Tested by: Michael Dexter and others
MFC after: 3 weeks

Add support for FreeBSD/i386 guests under bhyve.
- Similar to the hack for bootinfo32.c in userboot, define
_MACHINE_ELF_WANT_32BIT in the load_elf32 file handlers in userboot.
This allows userboot to load 32-bit kernels and modules.
- Copy the SMAP generation code out of bootinfo64.c and into its own
file so it can be shared with bootinfo32.c to pass an SMAP to the i386
kernel.
- Use uint32_t instead of u_long when aligning module metadata in
bootinfo32.c in userboot, as otherwise the metadata used 64-bit
alignment which corrupted the layout.
- Populate the basemem and extmem members of the bootinfo struct passed
to 32-bit kernels.
- Fix the 32-bit stack in userboot to start at the top of the stack
instead of the bottom so that there is room to grow before the
kernel switches to its own stack.
- Push a fake return address onto the 32-bit stack in addition to the
arguments normally passed to exec() in the loader. This return
address is needed to convince recover_bootinfo() in the 32-bit
locore code that it is being invoked from a "new" boot block.
- Add a routine to libvmmapi to setup a 32-bit flat mode register state
including a GDT and TSS that is able to start the i386 kernel and
update bhyveload to use it when booting an i386 kernel.
- Use the guest register state to determine the CPU's current instruction
mode (32-bit vs 64-bit) and paging mode (flat, 32-bit, PAE, or long
mode) in the instruction emulation code. Update the gla2gpa() routine
used when fetching instructions to handle flat mode, 32-bit paging, and
PAE paging in addition to long mode paging. Don't look for a REX
prefix when the CPU is in 32-bit mode, and use the detected mode to
enable the existing 32-bit mode code when decoding the mod r/m byte.

Reviewed by: grehan, neel
MFC after: 1 month

o Fix typo, sort .Xrs.

PR: docs/186191
Submitted by: Andrew (typo fix)
MFC after: 1 week

mdoc: quote string properly.

Don't create an initial value for the host filesystem of "/".
This has the unintended effect of booting the host kernel if
a disk image open fails.

Discussed with: neel

Allow bhyve and bhyveload to attach to tty devices.

bhyveload: introduce the -c <device> parameter
to select a tty for output (or "stdio")

bhyve: allow the puc and lpc-com backends to
accept a tty in addition to "stdio"

When used in conjunction with the null-modem device,
nmdm(4), this allows attach/detach to the guest console
and multiple concurrent serial ports. kgdb on a serial
port is now functional.

Reviewed by: neel
Requested by: Almost everyone that has used bhyve
MFC after: 10.0

Tidy usage messages for bhyve and bhyveload.

Submitted by: jhb

Add an option to bhyveload(8) that allows setting a loader environment variable
from the command line.

The option syntax is "-e <name=value>". It may be used multiple times to set
multiple environment variables.

Reviewed by: grehan
Requested by: alfred

Fix missing .

Approved by: re (blanket)

Parse the memory size parameter using expand_number() to allow specifying
the memory size more intuitively (e.g. 512M, 4G etc).

Submitted by: rodrigc
Reviewed by: grehan
Approved by: re (blanket)

Merge projects/bhyve_npt_pmap into head.

Make the amd64/pmap code aware of nested page table mappings used by bhyve
guests. This allows bhyve to associate each guest with its own vmspace and
deal with nested page faults in the context of that vmspace. This also
enables features like accessed/dirty bit tracking, swapping to disk and
transparent superpage promotions of guest memory.

Guest vmspace:
Each bhyve guest has a unique vmspace to represent the physical memory
allocated to the guest. Each memory segment allocated by the guest is
mapped into the guest's address space via the 'vmspace->vm_map' and is
backed by an object of type OBJT_DEFAULT.

pmap types:
The amd64/pmap now understands two types of pmaps: PT_X86 and PT_EPT.

The PT_X86 pmap type is used by the vmspace associated with the host kernel
as well as user processes executing on the host. The PT_EPT pmap is used by
the vmspace associated with a bhyve guest.

Page Table Entries:
The EPT page table entries as mostly similar in functionality to regular
page table entries although there are some differences in terms of what
bits are used to express that functionality. For e.g. the dirty bit is
represented by bit 9 in the nested PTE as opposed to bit 6 in the regular
x86 PTE. Therefore the bitmask representing the dirty bit is now computed
at runtime based on the type of the pmap. Thus PG_M that was previously a
macro now becomes a local variable that is initialized at runtime using
'pmap_modified_bit(pmap)'.

An additional wrinkle associated with EPT mappings is that older Intel
processors don't have hardware support for tracking accessed/dirty bits in
the PTE. This means that the amd64/pmap code needs to emulate these bits to
provide proper accounting to the VM subsystem. This is achieved by using
the following mapping for EPT entries that need emulation of A/D bits:
Bit Position Interpreted By
PG_V 52 software (accessed bit emulation handler)
PG_RW 53 software (dirty bit emulation handler)
PG_A 0 hardware (aka EPT_PG_RD)
PG_M 1 hardware (aka EPT_PG_WR)

The idea to use the mapping listed above for A/D bit emulation came from
Alan Cox (alc@).

The final difference with respect to x86 PTEs is that some EPT implementations
do not support superpage mappings. This is recorded in the 'pm_flags' field
of the pmap.

TLB invalidation:
The amd64/pmap code has a number of ways to do invalidation of mappings
that may be cached in the TLB: single page, multiple pages in a range or the
entire TLB. All of these funnel into a single EPT invalidation routine called
'pmap_invalidate_ept()'. This routine bumps up the EPT generation number and
sends an IPI to the host cpus that are executing the guest's vcpus. On a
subsequent entry into the guest it will detect that the EPT has changed and
invalidate the mappings from the TLB.

Guest memory access:
Since the guest memory is no longer wired we need to hold the host physical
page that backs the guest physical page before we can access it. The helper
functions 'vm_gpa_hold()/vm_gpa_release()' are available for this purpose.

PCI passthru:
Guest's with PCI passthru devices will wire the entire guest physical address
space. The MMIO BAR associated with the passthru device is backed by a
vm_object of type OBJT_SG. An IOMMU domain is created only for guest's that
have one or more PCI passthru devices attached to them.

Limitations:
There isn't a way to map a guest physical page without execute permissions.
This is because the amd64/pmap code interprets the guest physical mappings as
user mappings since they are numerically below VM_MAXUSER_ADDRESS. Since PG_U
shares the same bit position as EPT_PG_EXECUTE all guest mappings become
automatically executable.

Thanks to Alan Cox and Konstantin Belousov for their rigorous code reviews
as well as their support and encouragement.

Thanks for John Baldwin for reviewing the use of OBJT_SG as the backing
object for pci passthru mmio regions.

Special thanks to Peter Holm for testing the patch on short notice.

Approved by: re
Discussed with: grehan
Reviewed by: alc, kib
Tested by: pho

mdoc: remove superfluous paragraph macro.

Simplify the assignment of memory to virtual machines by requiring a single
command line option "-m <memsize in MB>" to specify the memory size.

Prior to this change the user needed to explicitly specify the amount of
memory allocated below 4G (-m <lowmem>) and the amount above 4G (-M <highmem>).

The "-M" option is no longer supported by 'bhyveload' and 'bhyve'.

The start of the PCI hole is fixed at 3GB and cannot be directly changed
using command line options. However it is still possible to change this in
special circumstances via the 'vm_set_lowmem_limit()' API provided by
libvmmapi.

Submitted by: Dinakar Medavaram (initial version)
Reviewed by: grehan
Obtained from: NetApp

Remove EOL whitespace.

Minor mdoc fixes.

Merge projects/bhyve to head.

'bhyve' was developed by grehan@ and myself at NetApp (thanks!).

Special thanks to Peter Snyder, Joe Caradonna and Michael Dexter for their
support and encouragement.

Obtained from: NetApp

Add the 'bhyveload(8)' man page.

Obtained from: NetApp
Reviewed by: grehan

Reduce the default memory allocation for a VM from 768MB to 128MB.

Obtained from: NetApp

IFC @ r242684

Use the new userboot 'getenv' callback to set a couple of environment variables
in the guest.

The variables are: smbios.bios.vendor=BHYVE and boot_serial=1

The FreeBSD guest uses the "smbios.bios.vendor" environment variable to
detect whether or not it is running as a guest inside a hypervisor.

The "boot_serial=1" is temporary and will be dropped when bhyve can do VGA
emulation.

Obtained from: NetApp

IFC @ r234692

sys/amd64/include/cpufunc.h
sys/amd64/include/fpu.h
sys/amd64/amd64/fpu.c
sys/amd64/vmm/vmm.c

- Add API to allow vmm FPU state init/save/restore.

FP stuff discussed with: kib

'bhyveload' is a userspace FreeBSD loader that can load the kernel + metadata
inside a BHyVe-based virtual machine.

It is a thin wrapper on top of userboot.so which is a variant of the FreeBSD
loader packaged as a shared library. 'bhyveload' provides callbacks that are
utilized by userboot.so to do things like console i/o, disk i/o,
set virtual machine registers etc.

Thanks for Doug Rabson (dfr@) for making this happen.