MFC r335604:
bhyve/vmrun.sh: make -L functional.

MFC: r328695,r328720,r328784,r328795 share/examples/bhyve/vmrun.sh additions
and cleanups

r328695:
Add option '-n' network device emulation type which has a default
of virtio-net
PR: 225558
Submitted by: Johannes Lundberg <johalun0@gmail.com>

r328720:
Add -n <network device emulation type> to usage synopsis
forgotten in r328695.

Fix long lines in usage.
Consistently use "default:" rather than "default is"

r328784:
Clean up long lines.

r328795:
Fold constants and unify vnc variable names.

Approved by: grehan (mentor)

PR: 225558

MFC: r316746 Add UEFI support to vmrun.sh

Adds:
-E: Use UEFI mode
-f: path to UEFI firmware image (default: path to uefi-edk2-bhyve package)
-F: UEFI framebuffer size (default: w=1024,h=768)
-L: IP to listen for VNC connections on (default: 127.0.0.1)
-P: Port to listen for VNC connections on (default: 5900)
-T: Enable tablnet device (for VNC)
-v: Wait for VNC client before booting VM

Submitted by: Shawn Webb <shawn.webb@hardenedbsd.org>
Approved by: grehan(mentor),allanjude(original committer)
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D10378

MFC r326140: vmrun.sh: add -A option for AHCI emulation of disk devices

MFC: r314694

Make vmrun.sh passthrough -u and -w to bhybe

PR: 214273
Submitted by: Martin Birgmeier <la5lb...@aon.at>
Approved by: grehan (mentor)

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Add option -l for specifying which OS loader to dlopen(3). By default
this is /boot/userboot.so. This option allows for the development and
use of other OS loaders.

Restructure memory allocation in bhyve to support "devmem".

devmem is used to represent MMIO devices like the boot ROM or a VESA framebuffer
where doing a trap-and-emulate for every access is impractical. devmem is a
hybrid of system memory (sysmem) and emulated device models.

devmem is mapped in the guest address space via nested page tables similar
to sysmem. However the address range where devmem is mapped may be changed
by the guest at runtime (e.g. by reprogramming a PCI BAR). Also devmem is
usually mapped RO or RW as compared to RWX mappings for sysmem.

Each devmem segment is named (e.g. "bootrom") and this name is used to
create a device node for the devmem segment (e.g. /dev/vmm/testvm.bootrom).
The device node supports mmap(2) and this decouples the host mapping of
devmem from its mapping in the guest address space (which can change).

Reviewed by: tychon
Discussed with: grehan
Differential Revision: https://reviews.freebsd.org/D2762
MFC after: 4 weeks

bhyve/vmrun.sh: pass all existing readable disks to bhyveload(8)

This is done if the installation mode is not requested or auto-detected.
This is useful when a kernel has to be loaded from multiple disks.
For example from a RAID-Z pool or a ZFS pool with multiple top level
vdevs.

Differential Revision: https://reviews.freebsd.org/D2723
Reviewed by: neel
MFC after: 8 days

bhyve/vmrun.sh: support passing disk options same as for bhyve(8)

Differential Revision: https://reviews.freebsd.org/D2723
Reviewed by: neel
MFC after: 8 days

Add -p parameter to list PCI device to pass through to the guest.

Reviewed by: neel

Make the error message explicit instead of just printing the usage if the
virtual machine name is not specified.

Requested by: grehan
MFC after: 1 week

Use '-e' to check if the virtio backing file has already been created.

The '-f' check works fine on a regular file but not if the backing file is
a device (e.g., /dev/md0). In this case it would print a misleading but
otherwise benign message about the backing file not being present.

Submitted by: Marcus Reid (marcus@blazingdot.com)
Discussed with: grehan

Propagate the error status of /usr/sbin/bhyve out of the vmrun.sh script.
Clean up the /dev/vmm entry for this VM when vmrun.sh exits.

Reviewed by: neel
Phabric: D767

Attach the ISO to an ahci-cd emulated device. The
ISO will appear to be mounted on a /dev/cd device
instead of /dev/vtbd. This is similar to how other
virtualization environments handle mounting ISO images.

Reviewed by: neel

Use "file -s", so that we can run vmrun.sh against special devices such
as /dev/md memory file systems

Reviewed by: neel

If the VM was reset via "/sbin/reboot" or "shutdown -r",
then it is no longer necessary to "bhyvectl --destroy" the VM when it reboots.
Move the "bhyvectl --destroy" outside of the while loop.

Reviewed by: neel

Add comment which describes the exit status codes returned from
/usr/sbin/bhyve. These are in src/usr.sbin/bhyve/bhyverun.c.

Reviewed by: neel

After r267897 brought in a new version of file/libmagic, a filesystem image
is identified as "DOS/MBR boot sector" as opposed to "x86 boot sector".

This trips up vmrun.sh when using the new file(1) and makes it want to boot
into the installer instead.

Fix this by just looking for "boot sector" instead.

Support for multiple disks and tap devices.

This allows you to give a bhyve instance multiple network devices
and disk devices easily by specifying additional "-d " and "-t "
options.

Reviewed by: neel
Sponsored by: Norse

- Format the usage so that it fits in 80 cols and follows the standard
convention for long usage lines in manpages.
- Sort the option string passed to getopts and the case statements for
the option returned by getopts.
- Add a -C option to specify the device to be used for the console
(defaults to 'stdio') (This could be let vmrun be run in the background
by using /dev/nmdm0B or the like)
- Add a -H option to specify a host path to pass to bhyveload(8) via
-h to back the host0: filesystem in bhyveload(8) (useful for loading
kernels from the host into the guest without having to copy them into
the guest's disk image first)

Reviewed by: neel
MFC after: 2 weeks

Make the virtual ioapic available unconditionally in a bhyve virtual machine.

This is in preparation for moving the ioapic device model from userspace to
vmm.ko.

Reviewed by: grehan

Add support for PCI-to-ISA LPC bridge emulation. If the LPC bus is attached
to a virtual machine then we implicitly create COM1 and COM2 ISA devices.

Prior to this change the only way of attaching a COM port to the virtual
machine was by presenting it as a PCI device that is mapped at the legacy
I/O address 0x3F8 or 0x2F8.

There were some issues with the original approach:
- It did not work at all with UEFI because UEFI will reprogram the PCI device
BARs and remap the COM1/COM2 ports at non-legacy addresses.
- OpenBSD GENERIC kernel does not create a /dev/console because it expects
the uart device at the legacy 0x3F8/0x2F8 address to be an ISA device.
- It was functional with a FreeBSD guest but caused the console to appear
on /dev/ttyu2 which was not intuitive.

The uart emulation is now independent of the bus on which it resides. Thus it
is possible to have uart devices on the PCI bus in addition to the legacy
COM1/COM2 devices behind the LPC bus.

The command line option to attach ISA COM1/COM2 ports to a virtual machine is
"-s <bus>,lpc -l com1,stdio".

The command line option to create a PCI-attached uart device is:
"-s <bus>,uart[,stdio]"

The command line option to create PCI-attached COM1/COM2 device is:
"-S <bus>,uart[,stdio]". This style of creating COM ports is deprecated.

Discussed with: grehan
Reviewed by: grehan
Submitted by: Tycho Nightingale (tycho.nightingale@pluribusnetworks.com)

M share/examples/bhyve/vmrun.sh
AM usr.sbin/bhyve/legacy_irq.c
AM usr.sbin/bhyve/legacy_irq.h
M usr.sbin/bhyve/Makefile
AM usr.sbin/bhyve/uart_emul.c
M usr.sbin/bhyve/bhyverun.c
AM usr.sbin/bhyve/uart_emul.h
M usr.sbin/bhyve/pci_uart.c
M usr.sbin/bhyve/pci_emul.c
M usr.sbin/bhyve/inout.c
M usr.sbin/bhyve/pci_emul.h
M usr.sbin/bhyve/inout.h
AM usr.sbin/bhyve/pci_lpc.c
AM usr.sbin/bhyve/pci_lpc.h

Add an option to bhyveload(8) that allows setting a loader environment variable
from the command line.

The option syntax is "-e <name=value>". It may be used multiple times to set
multiple environment variables.

Reviewed by: grehan
Requested by: alfred

Parse the memory size parameter using expand_number() to allow specifying
the memory size more intuitively (e.g. 512M, 4G etc).

Submitted by: rodrigc
Reviewed by: grehan
Approved by: re (blanket)

Add the "-g <gdport>" option to the vmrun.sh script that enables remote
debugging of the guest via kgdb.

See also: https://wiki.freebsd.org/BHyVe/gdb

Obtained from: NetApp

Add bhyve to examples.

Requested by: alfred, julian
Obtained from: NetApp