bhyve: Remove init_snapshot() and initialize static vars

vCPU threads are starting before init_snapshot() is called. That can lead
to corruption of vcpu_lock userspace mutex (snapshot.c) and then VM hangs
in acquiring that mutex.

init_snapshot() initializes only static variables (mutex, cv) and that
code can be optimized and removed.

Fixes: 9a9a248964696 ("bhyve: init checkput before caph_enter")
Reviewed by: markj
MFC after: 1 week
Sponsored by: vStack

bhyve: Replace many fprintf(stderr, ...) calls with EPRINTLN

EPRINTLN handles newlines appropriately when stdout/stderr have been
reused as the backend for a serial port.

For bhyverun.c itself, the rule this attempts to follow is to use
regular fprintf/perror/warn/err prior to init_pci() (which is when
serial ports are configured) and to switch to EPRINTLN afterwards.

Reviewed by: corvink, markj
Differential Revision: https://reviews.freebsd.org/D42182

Remove $FreeBSD$: two-line .h pattern

Remove /^\s*\*\n \*\s+\$FreeBSD\$$\n/

bhyve: rename 'user_dev' with 'devices'

Bhyve don't use 'user' specifier for emulated devices. And
using 'user' adds duality.

Reviewed by: corvink, rew
MFC after: 1 week
Sponsored by: vStack
Differential Revision: https://reviews.freebsd.org/D40106

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

bhyve: Remove vmctx member from struct vm_snapshot_meta.

This is a userland-only pointer that isn't relevant to the kernel and
doesn't belong in the ioctl structure shared between userland and the
kernel. For the kernel, the old structure for the ioctl is still
supported under COMPAT_FREEBSD13.

This changes vm_snapshot_req() in libvmmapi to accept an explicit
vmctx argument.

It also changes vm_snapshot_guest2host_addr to take an explicit vmctx
argument. As part of this change, move the declaration for this
function and its wrapper macro from vmm_snapshot.h to snapshot.h as it
is a userland-only API.

Reviewed by: corvink, markj
Differential Revision: https://reviews.freebsd.org/D38125

bhyve: Remove vmctx argument from PCI device model methods.

Most of these arguments were unused. Device models which do need
access to the vmctx in one of these methods can obtain it from the
pi_vmctx member of the pci_devinst argument instead.

Reviewed by: corvink, markj
Differential Revision: https://reviews.freebsd.org/D38096

bhyve/snapshot: use a string for cmd element in the nvlist

The nvlist for a checkpoint request will now look like:

{ cmd="checkpoint", suspend="true/false", filename="afilename" }

Reviewed by: jhb
Suggested by: jhb
Differential Revision: https://reviews.freebsd.org/D34237

bhyve/snapshot: switch to nvlist for snapshot requests

Switch to using an nvlist with nvlist_send()/nvlist_recv() to
communicate from bhyvectl(8) to bhyve(8).

The idea is that a bhyve process receives a command with with a set of
arguments. The nvlist here is structured to reflect that premise.

For example, to snapshot the vm, the expected nvlist looks like:

{ cmd=START_CHECKPOINT, filename="filename" }

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D33977

bhyve/snapshot: split up mutex/cond initialization from socket creation

Move initialization of the mutex/condition variables required by the
save/restore feature to their own function.

The unix domain socket that facilitates communication between bhyvectl
and bhyve doesn't rely on these variables in order to be functional.

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D30281

bhyve/snapshot: provide a way to send other messages/data to bhyve

This is a step towards sending messages (other than suspend/checkpoint)
from bhyvectl to bhyve.

Introduce a new struct, ipc_message - this struct stores the type of
message and a union containing message specific structures for the type
of message being sent.

Reviewed by: grehan
Differential Revision: https://reviews.freebsd.org/D30221

bhyve/snapshot: rename and bump size of MAX_SNAPSHOT_VMNAME

MAX_SNAPSHOT_VMNAME is a macro used to set the size of a character
buffer that stores a filename or the path to a file - this file is used
by the save/restore feature.

Since the file doesn't have anything to do with a vm name, rename
MAX_SNAPSHOT_VMNAME to MAX_SNAPSHOT_FILENAME. Bump the size to PATH_MAX
while here.

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D28879

bhyve/snapshot: rename checkpoint_opcodes to be more generic

Generalize the naming here since the domain socket that uses these codes
might be used for purposes other than the save/restore feature.

- rename checkpoint_opcodes to ipc_opcode

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D28877

bhyve/snapshot: drop mkdir when creating the unix domain socket

Add /var/run/bhyve/ to BSD.var.dist so we don't have to call mkdir when
creating the unix domain socket for a given bhyve vm.

The path to the unix domain socket for a bhyve vm will now be
/var/run/bhyve/vmname instead of /var/run/bhyve/checkpoint/vmname

Move BHYVE_RUN_DIR from snapshot.c to snapshot.h so it can be shared
to bhyvectl(8).

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D28783

libvmm: clean up vmmapi.h

struct checkpoint_op, enum checkpoint_opcodes, and
MAX_SNAPSHOT_VMNAME are not vmm specific, move them out of the vmmapi
header.

They are used for the save/restore functionality that bhyve(8)
provides and are better suited in usr.sbin/bhyve/snapshot.h

Since bhyvectl(8) requires these, the Makefile for bhyvectl has been
modified to include usr.sbin/bhyve/snapshot.h

Reviewed by: kevans, grehan
Differential Revision: https://reviews.freebsd.org/D28410

Initial support for bhyve save and restore.

Save and restore (also known as suspend and resume) permits a snapshot
to be taken of a guest's state that can later be resumed. In the
current implementation, bhyve(8) creates a UNIX domain socket that is
used by bhyvectl(8) to send a request to save a snapshot (and
optionally exit after the snapshot has been taken). A snapshot
currently consists of two files: the first holds a copy of guest RAM,
and the second file holds other guest state such as vCPU register
values and device model state.

To resume a guest, bhyve(8) must be started with a matching pair of
command line arguments to instantiate the same set of device models as
well as a pointer to the saved snapshot.

While the current implementation is useful for several uses cases, it
has a few limitations. The file format for saving the guest state is
tied to the ABI of internal bhyve structures and is not
self-describing (in that it does not communicate the set of device
models present in the system). In addition, the state saved for some
device models closely matches the internal data structures which might
prove a challenge for compatibility of snapshot files across a range
of bhyve versions. The file format also does not currently support
versioning of individual chunks of state. As a result, the current
file format is not a fixed binary format and future revisions to save
and restore will break binary compatiblity of snapshot files. The
goal is to move to a more flexible format that adds versioning,
etc. and at that point to commit to providing a reasonable level of
compatibility. As a result, the current implementation is not enabled
by default. It can be enabled via the WITH_BHYVE_SNAPSHOT=yes option
for userland builds, and the kernel option BHYVE_SHAPSHOT.

Submitted by: Mihai Tiganus, Flavius Anton, Darius Mihai
Submitted by: Elena Mihailescu, Mihai Carabas, Sergiu Weisz
Relnotes: yes
Sponsored by: University Politehnica of Bucharest
Sponsored by: Matthew Grooms (student scholarships)
Sponsored by: iXsystems
Differential Revision: https://reviews.freebsd.org/D19495