MFC r328011,329162

r328011:

Provide some mitigation against CVE-2017-5715 by clearing registers
upon returning from the guest which aren't immediately clobbered by
the host. This eradicates any remaining guest contents limiting their
usefulness in an exploit gadget.

r329162:

Provide further mitigation against CVE-2017-5715 by flushing the
return stack buffer (RSB) upon returning from the guest.

MFC r329364: move vintr_intercept_enabled under INVARIANTS

MFC r328622: vmm/svm: post LAPIC interrupts using event injection

PR: 215972

MFC r327726: vmm/svm: contigmalloc of the whole svm_softc is excessive

MFC r327593: Fix a couple of comments in AMD Virtual Machine Control Block structure

MFC 325039: Rework pass through changes in r305485 to be safer.

Specifically, devices that do not support PCI-e FLR and were not
gracefully shutdown by the guest OS could continue to issue DMA
requests after the VM was terminated. The changes in r305485 meant
that those DMA requests were completed against the host's memory which
could result in random memory corruption. Instead, leave ppt devices
that are not attached to a VM disabled in the IOMMU and only restore
the devices to the host domain if the ppt(4) driver is detached from a
device.

As an added safety belt, disable busmastering for a pass-through device
when before adding it to the host domain during ppt(4) detach.

PR: 222937

MFC r315361 and r315364: Hide MONITORX/MWAITX from guests.

r315361
Add the AMD MONITORX/MWAITX feature definition introduced in
Bulldozer/Ryzen CPUs.

r315364
Hide the AMD MONITORX/MWAITX capability.
Otherwise, recent Linux guests will use these instructions, resulting
in #UD exceptions since bhyve doesn't implement MONITOR/MWAIT exits.

This fixes boot-time hangs in recent Linux guests on Ryzen CPUs
(and probably Bulldozer aka AMD FX as well).

MFC r312531: vmm_dev: work around a bogus error with gcc 6.3.0

MFC r307903,307904,308039,308050: vmm/svm: iopm_bitmap and msr_bitmap
must be contiguous in physical memory

MFC 305502: Reset PCI pass through devices via PCI-e FLR during VM start/end.

Add routines to trigger a function level reset (FLR) of a PCI-express
device via the PCI-express device control register. This also includes
support routines to wait for pending transactions to complete as well
as calculating the maximum completion timeout permitted by a device.

Change the ppt(4) driver to reset pass through devices before attaching
to a VM during startup and before detaching from a VM during shutdown.

Sponsored by: Chelsio Communications

MFC 304858,305485: Fix various issues with PCI pass through and VT-d.

304858:
Enable I/O MMU when PCI pass through is first used.

Rather than enabling the I/O MMU when the vmm module is loaded,
defer initialization until the first attempt to pass a PCI device
through to a guest. If the I/O MMU fails to initialize or is not
present, than fail the attempt to pass a PCI device through to a
guest.

The hw.vmm.force_iommu tunable has been removed since the I/O MMU is
no longer enabled during boot. However, the I/O MMU support can be
disabled by setting the hw.vmm.iommu.enable tunable to 0 to prevent
use of the I/O MMU on any systems where it is buggy.

305485:
Leave ppt devices in the host domain when they are not attached to a VM.

This allows a pass through device to be reset to a normal device driver
on the host and reused on the host. ppt devices are now always active in
some I/O MMU domain when the I/O MMU is active, either the host domain
or the domain of a VM they are attached to.

MFC 303713: Correct assertion on vcpuid argument to vm_gpa_hold().

PR: 208168

MFC r284539, r284630, r284688, r284877, r285217, r285218,
r286837, r286838, r288470, r288522, r288524, r288826,
r289001

Pull in bhyve bug fixes and changes to allow UEFI booting.
This provides Windows support.

Tested on Intel and AMD with:
- Arch Linux i386+amd64 (kernel 4.3.3)
- Ubuntu 15.10 server 64-bit
- FreeBSD-CURRENT/amd64 20160127 snap
- FreeBSD 10.2 i386+amd64
- OpenBSD 5.8 i386+amd64
- SmartOS latest
- Windows 10 build 1511'

Huge thanks to Yamagi Burmeister who submitted the patch
and did the majority of the testing.

r284539 - bootrom mem allocation support
r284630 - Add SO_REUSEADDR when starting debug port
r284688 - Fix a regression in "movs" emulation
r284877 - verify_gla() non-zero segment base fix
r285217 - Always assert DCD and DSR in the uart
r285218 - devmem nodes moved to /dev/vmm.io/
r286837 - Add define for SATA Check-Power-Mode
r286838 - Add simple (no-op) SATA cmd emulations
r288470 - Increase virtio-blk indirect descs
r288522 - Firmware guest query interface
r288524 - Fix post-test typo
r288826 - Clean up SATA unimplemented cmd msg
r289001 - Add -l option to specify userboot path

Submitted by: Yamagi Burmeister
Approved by: re (kib)

MFC r284712:
Restore the host's GS.base before returning from 'svm_launch()' so the Dtrace
FBT provider works with vmm.ko on AMD.

MFC r282209:
Emulate the 'bit test' instruction.

MFC r282259:
Re-implement RTC current time calculation to eliminate the possibility of
losing time.

MFC r282281:
Advertise the MTRR feature via CPUID and emulate the minimal set of MTRR MSRs.

MFC r282284:
When an instruction cannot be decoded just return to userspace so bhyve(8)
can dump the instruction bytes.

MFC r282287:
Don't require <sys/cpuset.h> to be always included before <machine/vmm.h>.

MFC r282296:
Emulate MSR_SYSCFG which is accessed by Linux on AMD cpus when MTRRs are
enabled.

MFC r282301:
Relax limits when transitioning a vector from the IRR to the ISR and also
when extinguishing it from the ISR in response to an EOI.

MFC r282335:
Advertise an additional memory BAR in the "dummy" device emulation.

MFC r282336:
Emulate machine check related MSRs to allow guest OSes like Windows to boot.

MFC r282351:
Don't advertise the Intel SMX capability to the guest.

MFC r282407:
Emulate the 'CMP r/m8, imm8' instruction.

MFC r282519:
Add macros for AMD-specific bits in MSR_EFER: LMSLE, FFXSR and TCE.

MFC r282520:
Emulate guest writes to EFER_MSR properly.

MFC r282558:
Deprecate the 3-way return values from vm_gla2gpa() and vm_copy_setup().

MFC r282571:
Check 'td_owepreempt' and yield the vcpu thread if it is set.

MFC r282595:
Allow byte reads of AHCI registers.

MFC r282784:
Handling indirect descriptors is a capability of the host and not one that
needs to be negotiated. Use the host capabilities field and not the negotiated
field when verifying that indirect descriptors are supported.

MFC r282788:
Allow configuration of the sector size advertised to the guest.

MFC r282865:
Set the subvendor field in config space to the vendor ID. This is required
by the Windows virtio drivers to correctly match a device.

MFC r282922:
Bump the size of the blockif scatter-gather list to 67.

MFC r283075:
Fix off-by-one in array index bounds check. bhyveload would allow you to
create 33 entries on an array that only has 32 slots

MFC r283168:
Temporarily revert r282922 which bumped the max descriptors.

MFC r283255:
Emulate the "CMP r/m, reg" instruction (opcode 39H).

MFC r283256:
Add an option "--get-vmcs-exit-inst-length" to display the instruction length
of the instruction that caused the VM-exit.

MFC r283264:
Change the header type of the emulated host-bridge from type 1 to type 0.

MFC r283293:
Don't rely on the 'VM-exit instruction length' field in the VMCS to always
have an accurate length on an EPT violation.

MFC r283299:
Remove bogus verification of instruction length after instruction decode.

MFC r283308:
Exceptions don't deliver an error code in real mode.

MFC r283657:
Fix non-deterministic delays when accessing a vcpu that was in "running" or
"sleeping" state.

MFC r283973:
Use tunable 'hw.vmm.svm.features' to disable specific SVM features even
though they might be available in hardware. Use tunable 'hw.vmm.svm.num_asids'
to limit the number of ASIDs used by the hypervisor.

MFC r284046:
Fix regression in 'verify_gla()' with the RIP-relative addressing mode.

MFC r284174:
Support guest writes to the TSC by enabling the "use TSC offsetting"
execution control.

MFC r279444:
Allow passthrough devices to be hinted.

MFC r279683:
When ICW1 is issued the edge sense circuit is reset which means that
following an initialization a low-to-high transistion is necesary to
generate an interrupt.

MFC r279925:
Add -p parameter to list PCI device to pass through to the guest.

MFC r281559:
Fix handling of BUS_PROBE_NOWILDCARD in 'device_probe_child()'.

MFC r280447:
When fetching an instruction in non-64bit mode, consider the value of the
code segment base address.

MFC r280725:
Move legacy interrupt allocation for virtio devices to common code.

MFC r280775:
Fix the RTC device model to operate correctly in 12-hour mode.

MFC r280929:
Fix "MOVS" instruction memory to MMIO emulation.

MFC r280968:
Display instruction bytes and %rip prior to aborting due to an instruction
emulation error.

MFC r281145:
Enhance the support for Group 1 Extended opcodes for CMP, AND, OR instructions.

MFC r281542:
Initialize 'error' before use (Coverity IDs 1249748, 1249747, 1249751, 1249749)

MFC r281561:
Prior to aborting due to an ioport error, it is always interesting to see what
the guest's %rip is.

MFC r281611:
If the number of guest vcpus is less than '1' then flag it as an error.

MFC r281612:
Prefer 'vcpu_should_yield()' over checking 'curthread->td_flags' directly.

MFC r281630:
Relax the check on which vectors can be delivered through the APIC. According
to the Intel SDM vectors 16 through 255 are allowed to be delivered via the
local APIC.

MFC r281879:
Missing break in switch case (Coverity ID 1292499)

MFC r281946:
Don't allow guest to modify readonly bits in the PCI config 'status' register.

MFC r281987:
STOS/STOSB/STOSW/STOSD/STOSQ instruction emulation.

MFC r282206:
Implement the century byte in the RTC.

MFC r276428:
Replace bhyve's minimal RTC emulation with a fully featured one in vmm.ko.

MFC r276432:
Initialize all fields of 'struct vm_exception exception' before passing it
to vm_inject_exception().

MFC r276763:
Clear blocking due to STI or MOV SS in the hypervisor when an instruction is
emulated or when the vcpu incurs an exception.

MFC r277149:
Clean up usage of 'struct vm_exception' to only to communicate information
from userspace to vmm.ko when injecting an exception.

MFC r277168:
Fix typo (missing comma).

MFC r277309:
Make the error message explicit instead of just printing the usage if the
virtual machine name is not specified.

MFC r277310:
Simplify instruction restart logic in bhyve.

MFC r277359:
Fix a bug in libvmmapi 'vm_copy_setup()' where it would return success even
if the 'gpa' was in the guest MMIO region.

MFC r277360:
MOVS instruction emulation.

MFC r277626:
Add macro to identify AVIC capability (advanced virtual interrupt controller)
in AMD processors.

MFC r279220:
Don't close a block context if it couldn't be opened avoiding a null deref.

MFC r279225:
Add "-u" option to bhyve(8) to indicate that the RTC should maintain UTC time.

MFC r279227:
Emulate MSR 0xC0011024 when running on AMD processors.

MFC r279228:
Always emulate MSR_PAT on Intel processors and don't rely on PAT save/restore
capability of VT-x. This lets bhyve run nested in older VMware versions that
don't support the PAT save/restore capability.

MFC r279540:
Fix warnings/errors when building vmm.ko with gcc.

MFC r280134:
Report ARAT (APIC-Timer-always-running) feature for virtual CPU.

This makes FreeBSD guest to not avoid using LAPIC timer, preferring HPET
due to worries about non-existing for virtual CPUs deep sleep states.

Benchmarks of usleep(1) on guest and host show such extra latencies:
- 51us for virtual HPET,
- 22us for virtual LAPIC timer,
- 22us for host HPET and
- 3us for host LAPIC timer.

MFC r264007,r264008,r264009,r264011,r264012,r264013

MFC support for PCI Alternate RID Interpretation. ARI is an optional PCIe
feature that allows PCI devices to present up to 256 functions on a bus.
This is effectively a prerequisite for PCI SR-IOV support.

r264007:
Add a method to get the PCI RID for a device.

Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264008:
Re-implement the DMAR I/O MMU code in terms of PCI RIDs

Under the hood the VT-d spec is really implemented in terms of
PCI RIDs instead of bus/slot/function, even though the spec makes
pains to convert back to bus/slot/function in examples. However
working with bus/slot/function is not correct when PCI ARI is
in use, so convert to using RIDs in most cases. bus/slot/function
will only be used when reporting errors to a user.

Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264009:
Re-write bhyve's I/O MMU handling in terms of PCI RID.

Reviewed by: neel
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264011:
Add support for PCIe ARI

PCIe Alternate RID Interpretation (ARI) is an optional feature that
allows devices to have up to 256 different functions. It is
implemented by always setting the PCI slot number to 0 and
re-purposing the 5 bits used to encode the slot number to instead
contain the function number. Combined with the original 3 bits
allocated for the function number, this allows for 256 functions.

This is enabled by default, but it's expected to be a no-op on currently
supported hardware. It's a prerequisite for supporting PCI SR-IOV, and
I want the ARI support to go in early to help shake out any bugs in it.
ARI can be disabled by setting the tunable hw.pci.enable_ari=0.

Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264012:
Print status of ARI capability in pciconf -c

Teach pciconf how to print out the status (enabled/disabled) of the ARI
capability on PCI Root Complexes and Downstream Ports.

MFC after: 2 months
Sponsored by: Sandvine Inc.

r264013:
Add missing copyright date.

MFC after: 2 months

MFC r276323
Implement "special mask mode" in vatpic.

MFC r273683
Move the ACPI PM timer emulation into vmm.ko.

MFC r273706
Change the type of the first argument to the I/O emulation handlers to
'struct vm *'.

MFC r273710
Add a comment explaining the intent behind the I/O reservation [0x72-0x77].

MFC r273744
Add foo_genassym.c files to DPSRCS so dependencies for them are generated.
This ensures these objects are rebuilt to generate an updated header of
assembly constants if needed.

MFC r274045
If the start bit, PxCMD.ST, is cleared and nothing is in-flight then
PxCI, PxSACT, PxCMD.CCS and PxCMD.CR should be 0.

MFC r274076
Improve the ability to cancel an in-flight request by using an interrupt,
via SIGCONT, to force the read or write system call to return prematurely.

MFC r274330
To allow a request to be submitted from within the callback routine of
a completing one increase the total by 1 but don't advertise it.

MFC r274931
Change the lower bound for guest vmspace allocation to 0 instead of using
the VM_MIN_ADDRESS constant.

MFC r275817
For level triggered interrupts clear the PIC IRR bit when the interrupt pin
is deasserted.

MFC r275850
Fix 8259 IRQ priority resolver.

MFC r275952
Various 8259 device model improvements.

MFC r275965
Emulate writes to the IA32_MISC_ENABLE MSR.

MFC r273375
Add support AMD processors with the SVM/AMD-V hardware extensions.

MFC r273749
Remove bhyve SVM feature printf's now that they are available in the general
CPU feature detection code.

MFC r273766
Add missing 'break' pointed out by Coverity CID 1249760.

MFC r276098
Allow ktr(4) tracing of all guest exceptions via the tunable "hw.vmm.trace_guest_exceptions"

MFC r276392
Inject #UD into the guest when it executes either 'MONITOR' or 'MWAIT' on an
AMD/SVM host.

MFC r276402
Remove "svn:mergeinfo" property that was dragged along when these files were
svn copied in r273375.

MFC 261321
Rename the AMD MSR_PERFCTR[0-3] so the Pentium Pro MSR_PERFCTR[0-1] aren't
redefined.

MFC r273214
Fix build to not bogusly always rebuild vmm.ko.

MFC r273338
Add support for AMD's nested page tables in pmap.c:
- Provide the correct bit mask for various bit fields in a PTE (e.g. valid bit)
for a pmap of type PT_RVI.
- Add a function 'pmap_type_guest(pmap)' that returns TRUE if the pmap is of
type PT_EPT or PT_RVI.

Add CPU_SET_ATOMIC_ACQ(num, cpuset):
This is used when activating a vcpu in the nested pmap. Using the 'acquire'
variant guarantees that the load of the 'pm_eptgen' will happen only after
the vcpu is activated in 'pm_active'.

Add defines for various AMD-specific MSRs.

Discussed with: kib (r261321)

MFC r270326
Fix a recursive lock acquisition in vi_reset_dev().

MFC r270434
Return the spurious interrupt vector (IRQ7 or IRQ15) if the atpic cannot find
any unmasked pin with an interrupt asserted.

MFC r270436
Fix a bug in the emulation of CPUID leaf 0x4.

MFC r270437
Add "hw.vmm.topology.threads_per_core" and "hw.vmm.topology.cores_per_package"
tunables to modify the default cpu topology advertised by bhyve.

MFC r270855
Set the 'inst_length' to '0' early on before any error conditions are detected
in the emulation of the task switch. If any exceptions are triggered then the
guest %rip should point to instruction that caused the task switch as opposed
to the one after it.

MFC r270857
The "SUB" instruction used in getcc() actually does 'x -= y' so use the
proper constraint for 'x'. The "+r" constraint indicates that 'x' is an
input and output register operand.

While here generate code for different variants of getcc() using a macro
GETCC(sz) where 'sz' indicates the operand size.

Update the status bits in %rflags when emulating AND and OR opcodes.

MFC r271439
Initialize 'bc_rdonly' to the right value.

MFC r271451
Optimize the common case of injecting an interrupt into a vcpu after a HLT
by explicitly moving it out of the interrupt shadow.

MFC r271888
Restructure the MSR handling so it is entirely handled by processor-specific
code.

MFC r271890
MSR_KGSBASE is no longer saved and restored from the guest MSR save area. This
behavior was changed in r271888 so update the comment block to reflect this.

MFC r271891
Add some more KTR events to help debugging.

MFC r272197
mmap(2) requires either MAP_PRIVATE or MAP_SHARED for non-anonymous mappings.

MFC r272395
Get rid of code that dealt with the hardware not being able to save/restore
the PAT MSR on guest exit/entry. This workaround was done for a beta release
of VMware Fusion 5 but is no longer needed in later versions.

All Intel CPUs since Nehalem have supported saving and restoring MSR_PAT
in the VM exit and entry controls.

MFC r272670
Inject #UD into the guest when it executes either 'MONITOR' or 'MWAIT'.

MFC r272710
Implement the FLUSH operation in the virtio-block emulation.

MFC r272838
iasl(8) expects integer fields in data tables to be specified as hexadecimal
values. Therefore the bit width of the "PM Timer Block" was actually being
interpreted as 50-bits instead of the expected 32-bit.

This eliminates an error message emitted by a Linux 3.17 guest during boot:
"Invalid length for FADT/PmTimerBlock: 50, using default 32"

MFC r272839
Support Intel-specific MSRs that are accessed when booting up a linux in bhyve:
- MSR_PLATFORM_INFO
- MSR_TURBO_RATIO_LIMITx
- MSR_RAPL_POWER_UNIT

MFC r273108
Emulate "POP r/m". This is needed to boot OpenBSD/i386 MP kernel in bhyve.

MFC r273212
Support stopping and restarting the AHCI command list via toggling PxCMD.ST
from '1' to '0' and back. This allows the driver a chance to recover if
for instance a timeout occurred due to activity on the host.

MFC r273666.
Don't pass the 'error' return from an I/O port handler directly to vm_run().

MFC r272193

Allow the PIC's IMR register to be read before ICW initialisation.

As of git submit e179f6914152eca9, the Linux kernel does a simple
probe of the PIC by writing a pattern to the IMR and then reading it
back, prior to the init sequence of ICW words.

The bhyve PIC emulation wasn't allowing the IMR to be read until
the ICW sequence was complete. This limitation isn't required so
relax the test.

With this change, Linux kernels 3.15-rc2 and later won't hang
on boot when calibrating the local APIC.

Approved by: re (gjb)

MFC r270689:
Implement the 0x2B SUB instruction, and the OR variant of 0x81.

Found with local APIC accesses from bitrig/amd64 bsd.rd, 07/15-snap.

Approved by: re (rodrigc)

MFC r267921, r267934, r267949, r267959, r267966, r268202, r268276,
r268427, r268428, r268521, r268638, r268639, r268701, r268777,
r268889, r268922, r269008, r269042, r269043, r269080, r269094,
r269108, r269109, r269281, r269317, r269700, r269896, r269962,
r269989.

Catch bhyve up to CURRENT.

Lightly tested with FreeBSD i386/amd64, Linux i386/amd64, and
OpenBSD/amd64. Still resolving an issue with OpenBSD/i386.

Many thanks to jhb@ for all the hard work on the prior MFCs !

r267921 - support the "mov r/m8, imm8" instruction
r267934 - document options
r267949 - set DMI vers/date to fixed values
r267959 - doc: sort cmd flags
r267966 - EPT misconf post-mortem info
r268202 - use correct flag for event index
r268276 - 64-bit virtio capability api
r268427 - invalidate guest TLB when cr3 is updated, needed for TSS
r268428 - identify vcpu's operating mode
r268521 - use correct offset in guest logical-to-linear translation
r268638 - chs value
r268639 - chs fake values
r268701 - instr emul operand/address size override prefix support
r268777 - emulation for legacy x86 task switching
r268889 - nested exception support
r268922 - fix INVARIANTS build
r269008 - emulate instructions found in the OpenBSD/i386 5.5 kernel
r269042 - fix fault injection
r269043 - Reduce VMEXIT_RESTARTs in task_switch.c
r269080 - fix issues in PUSH emulation
r269094 - simplify return values from the inout handlers
r269108 - don't return -1 from the push emulation handler
r269109 - avoid permanent sleep in vm_handle_hlt()
r269281 - list VT-x features in base kernel dmesg
r269317 - Mark AHCI fatal errors as not completed
r269700 - Support PCI extended config space in bhyve
r269896 - Minor cleanup
r269962 - use max guest memory when creating IOMMU domain
r269989 - fix interrupt mode names

MFC r267311, r267330, r267811, r267884

Turn on interrupt window exiting unconditionally when an ExtINT is being
injected into the guest.

Add helper functions to populate VM exit information for rendezvous and
astpending exits.

Provide APIs to directly get 'lowmem' and 'highmem' size directly.

Expose the amount of resident and wired memory from the guest's vmspace

MFC r267178, r267300

Support guest accesses to %cr8

Add reserved bit checking when doing %CR8 emulation and inject #GP if required.

MFC r267216
Add ioctl(VM_REINIT) to reinitialize the virtual machine state maintained
by vmm.ko. This allows the virtual machine to be restarted without having
to destroy it first.

MFC r266933
Activate vcpus from bhyve(8) using the ioctl VM_ACTIVATE_CPU instead of doing
it implicitly in vmm.ko.

MFC 266424,266476,266524,266573,266595,266626,266627,266633,266641,266642,
266708,266724,266934,266935,268521:
Emulation of the "ins" and "outs" instructions.

Various fixes for translating guest linear addresses to guest physical
addresses.

MFC 266125:
Implement a PCI interrupt router to route PCI legacy INTx interrupts to
the legacy 8259A PICs.

MFC 264353,264509,264768,264770,264825,264846,264988,265114,265165,265365,
265941,265951,266390,266550,266910:
Various bhyve fixes:
- Don't save host's return address in 'struct vmxctx'.
- Permit non-32-bit accesses to local APIC registers.
- Factor out common ioport handler code.
- Use calloc() in favor of malloc + memset.
- Change the vlapic timer frequency to be in the ballpark of contemporary
hardware.
- Allow the guest to read the TSC via MSR 0x10.
- A VMCS is always inactive when it exits the vmx_run() loop. Remove
redundant code and the misleading comment that suggest otherwise.
- Ignore writes to microcode update MSR. This MSR is accessed by RHEL7
guest.
Add KTR tracepoints to annotate wrmsr and rdmsr VM exits.
- Provide an alias for the userboot console and name it 'comconsole'.
- Use EV_ADD to create an mevent and EV_ENABLE to enable it.
- abort(3) the process in response to a VMEXIT_ABORT.
- Don't include the guest memory segments in the bhyve(8) process core dump.
- Make the vmx asm code dtrace-fbt-friendly.
- Allow vmx_getdesc() and vmx_setdesc() to be called for a vcpu that is in
the VCPU_RUNNING state.
- Enable VMX in the IA32_FEATURE_CONTROL MSR if it not enabled and the MSR
isn't locked.

MFC 264347:
Account for the "plus 1" encoding of the CPUID Function 4 reported
core per package and cache sharing values.

MFC 263780,264516,265062,265101,265203,265364:
Add an ioctl to suspend a virtual machine (VM_SUSPEND).

Add logic in the HLT exit handler to detect if the guest has put all vcpus
to sleep permanently by executing a HLT with interrupts disabled.

When this condition is detected the guest with be suspended with a reason of
VM_SUSPEND_HALT and the bhyve(8) process will exit.

This logic can be disabled via the tunable 'hw.vmm.halt_detection'.

MFC 259942,262274,263035,263054,263211,263744,264179,264324,264468,264631,
264648,264650,264651,266572,267558:
Flesh out the AT PIC and 8254 PIT emulations and move them into the kernel.

MFC 262139,262140,262236,262281,262532:
Various x2APIC fixes and enhancements:
- Use spinlocks for the vioapic.
- Handle the SELF_IPI MSR.
- Simplify the APIC mode switching between MMIO and x2APIC. The guest is
no longer allowed to switch modes at runtime. Instead, the desired mode
is set when the virtual machine is created.
- Disallow MMIO access in x2APIC mode and MSR access in xAPIC mode.
- Add support for x2APIC virtualization assist in Intel VT-x.

MFC 262615,262624:
Workaround an apparent bug in VMWare Fusion's nested VT support where it
triggers a VM exit with the exit reason of an external interrupt but
without a valid interrupt set in the exit interrupt information.

MFC 261638,262144,262506,266765:
Add virtualized XSAVE support to bhyve which permits guests to use XSAVE and
XSAVE-enabled features like AVX.
- Store a per-cpu guest xcr0 register and handle xsetbv VM exits by emulating
the instruction.
- Only expose XSAVE to guests if XSAVE is enabled in the host. Only expose
a subset of XSAVE features currently supported by the guest and for which
the proper emulation of xsetbv is known. Currently this includes X87, SSE,
AVX, AVX-512, and Intel MPX.
- Add support for injecting hardware exceptions into the guest and use this
to trigger exceptions in the guest for invalid xsetbv operations instead
of potentially faulting in the host.
- Queue pending exceptions in the 'struct vcpu' instead of directly updating
the processor-specific VMCS or VMCB. The pending exception will be delivered
right before entering the guest.
- Rename the unused ioctl VM_INJECT_EVENT to VM_INJECT_EXCEPTION and restrict
it to only deliver x86 hardware exceptions. This new ioctl is now used to
inject a protection fault when the guest accesses an unimplemented MSR.
- Expose a subset of known-safe features from leaf 0 of the structured
extended features to guests if they are supported on the host including
RDFSBASE/RDGSBASE, BMI1/2, AVX2, AVX-512, HLE, ERMS, and RTM. Aside
from AVX-512, these features are all new instructions available for use
in ring 3 with no additional hypervisor changes needed.

MFC 261504:
Add support for FreeBSD/i386 guests under bhyve.

MFC 261503,264501:
Emulate the byte move and zero/sign extend instructions.

MFC 260239,261268,265058:
Expand the support for PCI INTx interrupts including providing interrupt
routing information for INTx interrupts to I/O APIC pins and enabling
INTx interrupts in the virtio and AHCI backends.

MFC 260972:
There is no need to initialize the IOMMU if no passthru devices have been
configured for bhyve to use.

MFC 260802,260836,260863,261001,261074,261617:
Various fixes for NMI and interrupt injection.
- If a VM-exit happens during an NMI injection then clear the "NMI Blocking"
bit in the Guest Interruptibility-state VMCS field.
- If the guest exits due to a fault while it is executing IRET then restore
the state of "Virtual NMI blocking" in the guest's interruptibility-state
field before resuming the guest.
- Inject a pending NMI only if NMI_BLOCKING, MOVSS_BLOCKING, STI_BLOCKING
are all clear. If any of these bits are set then enable "NMI window
exiting" and inject the NMI in the VM-exit handler.
- Handle a VM-exit due to a NMI properly by vectoring to the host's NMI
handler via a software interrupt.
- Set "Interrupt Window Exiting" in the case where there is a vector to be
injected into the vcpu but the VM-entry interruption information field
already has the valid bit set.
- For VM-exits due to an NMI, handle the NMI with interrupts disabled in
addition to "blocking by NMI" already established by the VM-exit.

MFC 260237:
Fix a bug in the HPET emulation where a timer interrupt could be lost when the
guest disables the HPET.

The HPET timer interrupt is triggered from the callout handler associated with
the timer. It is possible for the callout handler to be delayed before it gets
a chance to execute. If the guest disables the HPET during this window then the
handler never gets a chance to execute and the timer interrupt is lost.

This is now fixed by injecting a timer interrupt into the guest if the callout
time is detected to be in the past when the HPET is disabled.

MFC 259737, 262646:
Fix a couple of issues with vcpu state:
- Add a parameter to 'vcpu_set_state()' to enforce that the vcpu is in the
IDLE state before the requested state transition. This guarantees that
there is exactly one ioctl() operating on a vcpu at any point in time and
prevents unintended state transitions.
- Fix a race between VMRUN() and vcpu_notify_event() due to 'vcpu->hostcpu'
being updated outside of the vcpu_lock().

MFC 259641,259863,259924,259937,259961,259978,260380,260383,260410,260466,
260531,260532,260550,260619,261170,261453,261621,263280,263290,264516:
Add support for local APIC hardware-assist.
- Restructure vlapic access and register handling to support hardware-assist
for the local APIC.
- Use the 'Virtual Interrupt Delivery' and 'Posted Interrupt Processing'
feature of Intel VT-x if supported by hardware.
- Add an API to rendezvous all active vcpus in a virtual machine and use
it to support level triggered interrupts with VT-x 'Virtual Interrupt
Delivery'.
- Use a cheaper IPI handler than IPI_AST for nested page table shootdowns
and avoid doing unnecessary nested TLB invalidations.

Reviewed by: neel

MFC 258860,260167,260238,260397:
- Restructure the VMX code to enter and exit the guest. In large part this
change hides the setjmp/longjmp semantics of VM enter/exit.
vmx_enter_guest() is used to enter guest context and vmx_exit_guest() is
used to transition back into host context.

Fix a longstanding race where a vcpu interrupt notification might be
ignored if it happens after vmx_inject_interrupts() but before host
interrupts are disabled in vmx_resume/vmx_launch. We now call
vmx_inject_interrupts() with host interrupts disabled to prevent this.
- The 'protection' field in the VM exit collateral for the PAGING exit is
not used - get rid of it.

Reviewed by: grehan

MFC 259542:
Use vmcs_read() and vmcs_write() in preference to vmread() and vmwrite()
respectively. The vmcs_xxx() functions provide inline error checking of
all accesses to the VMCS.

MFC 258859,259081,259085,259205,259213,259275,259482,259537,259702,259779:
Several changes to the local APIC support in bhyve:
- Rename 'vm_interrupt_hostcpu()' to 'vcpu_notify_event()'.
- If a vcpu disables its local apic and then executes a 'HLT' then spin
down the vcpu and destroy its thread context. Also modify the 'HLT'
processing to ignore pending interrupts in the IRR if interrupts have
been disabled by the guest. The interrupt cannot be injected into the
guest in any case so resuming it is futile.
- Use callout(9) to drive the vlapic timer instead of clocking it on each
VM exit.
- When the guest is bringing up the APs in the x2APIC mode a write to the
ICR register will now trigger a return to userspace with an exitcode of
VM_EXITCODE_SPINUP_AP.
- Change the vlapic timer lock to be a spinlock because the vlapic can be
accessed from within a critical section (vm run loop) when guest is using
x2apic mode.
- Fix the vlapic version register.
- Add a command to bhyvectl to inject an NMI on a specific vcpu.
- Add an API to deliver message signalled interrupts to vcpus. This allows
callers to treat the MSI 'addr' and 'data' fields as opaque and also lets
bhyve implement multiple destination modes: physical, flat and clustered.
- Rename the ambiguously named 'vm_setup_msi()' and 'vm_setup_msix()' to
'vm_setup_pptdev_msi()' and 'vm_setup_pptdev_msix()' respectively.
- Consolidate the virtual apic initialization in a single function:
vlapic_reset()
- Add a generic routine to trigger an LVT interrupt that supports both
fixed and NMI delivery modes.
- Add an ioctl and bhyvectl command to trigger local interrupts inside a
guest. In particular, a global NMI similar to that raised by SERR# or
PERR# can be simulated by asserting LINT1 on all vCPUs.
- Extend the LVT table in the vCPU local APIC to support CMCI.
- Flesh out the local APIC error reporting a bit to cache errors and
report them via ESR when ESR is written to. Add support for asserting
the error LVT when an error occurs. Raise illegal vector errors when
attempting to signal an invalid vector for an interrupt or when sending
an IPI.
- Export table entries in the MADT and MP Table advertising the stock x86
config of LINT0 set to ExtInt and LINT1 wired to NMI.

MFC 257297:
Remove unnecessary includes of <machine/pmap.h>

MFC r258779,r258780,r258787,r258822:

Fix undefined behavior: (1 << 31) is not defined as 1 is an int and this
shifts into the sign bit. Instead use (1U << 31) which gets the
expected result.

Similar to the (1 << 31) case it is not defined to do (2 << 30).

This fix is not ideal as it assumes a 32 bit int, but does fix the issue
for most cases.

A similar change was made in OpenBSD.

MFC 259782:
Add a resume hook for bhyve that runs a function on all CPUs during
resume. For Intel CPUs, invoke vmxon for CPUs that were in VMX mode
at the time of suspend.

MFC 257422,257661,258075,258476,258494,258579,258609,258699:
Several enhancements to the I/O APIC support in bhyve including:
- Move the I/O APIC device model from userspace into vmm.ko and add
ioctls to assert and deassert I/O APIC pins.
- Add HPET device emulation including a single timer block with 8 timers.
- Remove the 'vdev' abstraction.

Approved by: neel

Hoist all the mergeinfo up to the root in preparation for enforcing merges
to the root only. All MFC's were rerecorded to the root.

Going forward, if an MFC includes mergeinfo, it will need to be made to
the root and committed from the root. Merges with --ignore-ancestry
or diff | patch can go anywhere.

The mergeinfo in HEAD is in a bad state from years of neglect and manual
tampering and this was branched into 10.x. This confuses the coalescing
code and prevents it from doing its job.

Approved by: re (gjb, implicit)

MFC r256645.

Add a new capability, VM_CAP_ENABLE_INVPCID, that can be enabled to expose
'invpcid' instruction to the guest. Currently bhyve will try to enable this
capability unconditionally if it is available.

Consolidate code in bhyve to set the capabilities so it is no longer
duplicated in BSP and AP bringup.

Add a sysctl 'vm.pmap.invpcid_works' to display whether the 'invpcid'
instruction is available.

Approved by: re (hrs)

MFC r256570:

Fix the witness warning that warned against calling uiomove() while holding
the 'vmmdev_mtx' in vmmdev_rw().

Rely on the 'si_threadcount' accounting to ensure that we never destroy the
VM device node while it has operations in progress (e.g. ioctl, mmap etc).

Approved by: re (rodrigc)

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Merge projects/bhyve_npt_pmap into head.

Make the amd64/pmap code aware of nested page table mappings used by bhyve
guests. This allows bhyve to associate each guest with its own vmspace and
deal with nested page faults in the context of that vmspace. This also
enables features like accessed/dirty bit tracking, swapping to disk and
transparent superpage promotions of guest memory.

Guest vmspace:
Each bhyve guest has a unique vmspace to represent the physical memory
allocated to the guest. Each memory segment allocated by the guest is
mapped into the guest's address space via the 'vmspace->vm_map' and is
backed by an object of type OBJT_DEFAULT.

pmap types:
The amd64/pmap now understands two types of pmaps: PT_X86 and PT_EPT.

The PT_X86 pmap type is used by the vmspace associated with the host kernel
as well as user processes executing on the host. The PT_EPT pmap is used by
the vmspace associated with a bhyve guest.

Page Table Entries:
The EPT page table entries as mostly similar in functionality to regular
page table entries although there are some differences in terms of what
bits are used to express that functionality. For e.g. the dirty bit is
represented by bit 9 in the nested PTE as opposed to bit 6 in the regular
x86 PTE. Therefore the bitmask representing the dirty bit is now computed
at runtime based on the type of the pmap. Thus PG_M that was previously a
macro now becomes a local variable that is initialized at runtime using
'pmap_modified_bit(pmap)'.

An additional wrinkle associated with EPT mappings is that older Intel
processors don't have hardware support for tracking accessed/dirty bits in
the PTE. This means that the amd64/pmap code needs to emulate these bits to
provide proper accounting to the VM subsystem. This is achieved by using
the following mapping for EPT entries that need emulation of A/D bits:
Bit Position Interpreted By
PG_V 52 software (accessed bit emulation handler)
PG_RW 53 software (dirty bit emulation handler)
PG_A 0 hardware (aka EPT_PG_RD)
PG_M 1 hardware (aka EPT_PG_WR)

The idea to use the mapping listed above for A/D bit emulation came from
Alan Cox (alc@).

The final difference with respect to x86 PTEs is that some EPT implementations
do not support superpage mappings. This is recorded in the 'pm_flags' field
of the pmap.

TLB invalidation:
The amd64/pmap code has a number of ways to do invalidation of mappings
that may be cached in the TLB: single page, multiple pages in a range or the
entire TLB. All of these funnel into a single EPT invalidation routine called
'pmap_invalidate_ept()'. This routine bumps up the EPT generation number and
sends an IPI to the host cpus that are executing the guest's vcpus. On a
subsequent entry into the guest it will detect that the EPT has changed and
invalidate the mappings from the TLB.

Guest memory access:
Since the guest memory is no longer wired we need to hold the host physical
page that backs the guest physical page before we can access it. The helper
functions 'vm_gpa_hold()/vm_gpa_release()' are available for this purpose.

PCI passthru:
Guest's with PCI passthru devices will wire the entire guest physical address
space. The MMIO BAR associated with the passthru device is backed by a
vm_object of type OBJT_SG. An IOMMU domain is created only for guest's that
have one or more PCI passthru devices attached to them.

Limitations:
There isn't a way to map a guest physical page without execute permissions.
This is because the amd64/pmap code interprets the guest physical mappings as
user mappings since they are numerically below VM_MAXUSER_ADDRESS. Since PG_U
shares the same bit position as EPT_PG_EXECUTE all guest mappings become
automatically executable.

Thanks to Alan Cox and Konstantin Belousov for their rigorous code reviews
as well as their support and encouragement.

Thanks for John Baldwin for reviewing the use of OBJT_SG as the backing
object for pci passthru mmio regions.

Special thanks to Peter Holm for testing the patch on short notice.

Approved by: re
Discussed with: grehan
Reviewed by: alc, kib
Tested by: pho

Return 0 for a rdmsr of MSR_IA32_PLATFORM_ID. This
is enough to get Ubuntu 12.0.4/13.0.4 to boot.

Approved by: re@ (blanket)

Hide TSC-deadline APIC timer support from guests. This mode
isn't yet implemented in bhyve's APIC emulation.

Reviewed by: neel
Approved by: re@ (blanket)

Fix a bug in decoding an instruction that has an SIB byte as well as an
immediate operand. The presence of an SIB byte in decoding the ModR/M field
would cause 'imm_bytes' to not be set to the correct value.

Fix this by initializing 'imm_bytes' independent of the ModR/M decoding.

Reported by: grehan@
Approved by: re@

Fix a limitation in bhyve that would limit the number of virtual machines to
the maximum number of VT-d domains (256 on a Sandybridge). We now allocate a
VT-d domain for a guest only if the administrator has explicitly configured
one or more PCI passthru device(s).

If there are no PCI passthru devices configured (the common case) then the
number of virtual machines is no longer limited by the maximum number of
VT-d domains.

Reviewed by: grehan@
Approved by: re@

Allocate VPIDs by using the unit number allocator to keep do the bookkeeping.

Also deal with VPID exhaustion by allocating out of a reserved range as the
last resort.

Mask off the vector from the MSI-x data word.
Some o/s's set the trigger-mode level bit which
results in an invalid vector and pass-thru interrupts
not being delivered.

Emulate reading of the IA32_MISC_ENABLE MSR, by returning
the host MSR and masking off features that aren't supported.
Linux reads this MSR to detect if NX has been disabled via
BIOS.

Allow CPUID leaf 0xD to be read as zeroes.
Linux reads this even though extended features
aren't exposed.

Support for 0xD will be expanded once AVX[2]
is exposed to the guest in upcoming work.

Add support for emulating the byte move instruction "mov r/m8, r8".

This emulation is required when dumping MMIO space via the ddb "examine"
command.

Do not create superpage mappings in the iommu.

This is a workaround to hide the fact that we do not have any code to
demote a superpage mapping before we unmap a single page that is part
of the superpage.

Extract the location of the remapping hardware units from the ACPI DMAR table.

Submitted by: Gopakumar T (gopakumar_thekkedath@yahoo.co.in)

Follow-up commit to fix CR0 issues. Maintain
architectural state on CR vmexits by guaranteeing
that EFER, CR0 and the VMCS entry controls are
all in sync when transitioning to IA-32e mode.

Submitted by: Tycho Nightingale (tycho.nightingale <at> plurisbusnetworks.com)

Moved clearing of vmm_initialized to avoid the case
of unloading the module while VMs existed. This would
result in EBUSY, but would prevent further operations
on VMs resulting in the module being impossible to
unload.

Submitted by: Tycho Nightingale (tycho.nightingale <at> plurisbusnetworks.com)
Reviewed by: grehan, neel

Correctly maintain the CR0/CR4 shadow registers.
This was exposed with AP spinup of Linux, and
booting OpenBSD, where the CR0 register is unconditionally
written to prior to the longjump to enter protected
mode. The CR-vmexit handling was not updating CPU state which
resulted in a vmentry failure with invalid guest state.

A follow-on submit will fix the CPU state issue, but this
fix prevents the CR-vmexit prior to entering protected
mode by properly initializing and maintaining CR* state.

Reviewed by: neel
Reported by: Gopakumar.T @ netapp

Add support for emulation of the "or r/m, imm8" instruction.

Submitted by: Zhixiang Yu (zxyu.core@gmail.com)
Obtained from: GSoC 2013 (AHCI device emulation for bhyve)

Verify that all bytes in the instruction buffer are consumed during decoding.

Suggested by: grehan

Ignore guest PAT settings by default in EPT mappings.
From experimentation, other hypervisors also do this.

Diagnosed by: tycho nightingale at pluribusnetworks com
Reviewed by: neel

Make sure all CPUID values are handled, instead of exiting the
bhyve process when an unhandled one is encountered.

Hide some additional capabilities from the guest (e.g. debug store).

This fixes the issue with FreeBSD 9.1 MP guests exiting the VM on
AP spinup (where CPUID is used when sync'ing the TSCs) and the
issue with the Java build where CPUIDs are issued from a guest
userspace.

Submitted by: tycho nightingale at pluribusnetworks com
Reviewed by: neel
Reported by: many

Fix a gcc warning uncovered after r251745.

Reported by: Sergey V. Dyatko
Reviewed by: neel

Replace cpusetffs_obj with CPU_FFS, missed in r251703.

Reported by: bdrewery, O. Hartmann

Support array-type of stats in bhyve.

An array-type stat in vmm.ko is defined as follows:
VMM_STAT_ARRAY(IPIS_SENT, VM_MAXCPU, "ipis sent to vcpu");

It is incremented as follows:
vmm_stat_array_incr(vm, vcpuid, IPIS_SENT, array_index, 1);

And output of 'bhyvectl --get-stats' looks like:
ipis sent to vcpu[0] 3114
ipis sent to vcpu[1] 0

Reviewed by: grehan
Obtained from: NetApp

Switch to standard copyright license text

The initial version of this came from Sandvine but had "PROVIDED BY NETAPP,
INC" in the copyright text, presuambly because the license block was copied
from another file. Replace it with standard "AUTHOR AND CONTRIBUTORS" form.

Approvided by: grehan@

Add RIP-relative addressing to the instruction decoder.
Rework the guest register fetch code to allow the RIP to
be extracted from the VMCS while the kernel decoder is
functioning.

Hit by the OpenBSD local-apic code.

Submitted by: neel
Reviewed by: grehan
Obtained from: NetApp

Create sysctl node 'hw.vmm.vmx' and populate it with oids that expose the VMX
hardware capabilities.

Obtained from: NetApp

Use the MAKEDEV_CHECKNAME flag to check for an invalid device name and return
an error instead of panicking.

Obtained from: NetApp

If vmm.ko could not be initialized correctly then prevent the creation of
virtual machines subsequently.

Submitted by: Chris Torek

Make the code to check if VMX is enabled more readable by using macros
instead of magic numbers.

Discussed with: Chris Torek

Unsynchronized TSCs on the host require special handling in bhyve:

- use clock_gettime(2) as the time base for the emulated ACPI timer instead
of directly using rdtsc().

- don't advertise the invariant TSC capability to the guest to discourage it
from using the TSC as its time base.

Discussed with: jhb@ (about making 'smp_tsc' a global)
Reported by: Dan Mack on freebsd-virtualization@
Obtained from: NetApp

Don't panic when a valid divisor of 1 has been requested.

Obtained from: NetApp

Add counter to keep track of the number of timer interrupts generated by
the local apic for each virtual cpu.

Add some more stats to keep track of all the reasons that a vcpu is exiting.

Allow caller to skip 'guest linear address' validation when doing instruction
decode. This is to accomodate hardware assist implementations that do not
provide the 'guest linear address' as part of nested page fault collateral.

Submitted by: Anish Gupta (akgupt3 at gmail dot com)

Fix the '-Wtautological-compare' warning emitted by clang for comparing the
unsigned enum type with a negative value.

Obtained from: NetApp

Allow vmm stats to be specific to the underlying hardware assist technology.
This can be done by using the new macros VMM_STAT_INTEL() and VMM_STAT_AMD().
Statistic counters that are common across the two are defined using VMM_STAT().

Suggested by: Anish Gupta
Discussed with: grehan
Obtained from: NetApp

Requests for invalid CPUID leaves should map to the highest known leaf instead.

Reviewed by: grehan
Obtained from: NetApp

Implement guest vcpu pinning using 'pthread_setaffinity_np(3)'.

Prior to this change pinning was implemented via an ioctl (VM_SET_PINNING)
that called 'sched_bind()' on behalf of the user thread.

The ULE implementation of 'sched_bind()' bumps up 'td_pinned' which in turn
runs afoul of the assertion '(td_pinned == 0)' in userret().

Using the cpuset affinity to implement pinning of the vcpu threads works with
both 4BSD and ULE schedulers and has the happy side-effect of getting rid
of a bunch of code in vmm.ko.

Discussed with: grehan

Fix a broken assumption in the passthru implementation that the MSI-X table
can only be located at the beginning or the end of the BAR.

If the MSI-table is located in the middle of a BAR then we will split the
BAR into two and create two mappings - one before the table and one after
the table - leaving a hole in place of the table so accesses to it can be
trapped and emulated.

Obtained from: NetApp

Increase the number of passthru devices supported by bhyve.

The maximum length of an environment variable puts a limitation on the
number of passthru devices that can be specified via a single variable.
The workaround is to allow user to specify passthru devices via multiple
environment variables instead of a single one.

Obtained from: NetApp

Add emulation support for instruction "88/r: mov r/m8, r8".

This instruction moves a byte from a register to a memory location.

Tested by: tycho nightingale at pluribusnetworks com

Always allow access to the sysenter cs/esp/eip MSRs since they
are automatically saved and restored in the VMCS.

Reviewed by: neel
Obtained from: NetApp

Postpone vmm module initialization until after SMP is initialized - particularly
that 'smp_started != 0'.

This is required because the VT-x initialization calls smp_rendezvous()
to set the CR4_VMXE bit on all the cpus.

With this change we can preload vmm.ko from the loader.

Reported by: alfred@, sbruno@
Obtained from: NetApp

Add svn properties to the recently merged bhyve source files.

The pre-commit hook will not allow any commits without the svn:keywords
property in head.

Merge projects/bhyve to head.

'bhyve' was developed by grehan@ and myself at NetApp (thanks!).

Special thanks to Peter Snyder, Joe Caradonna and Michael Dexter for their
support and encouragement.

Obtained from: NetApp

There is no need for 'start_emulating()' and 'stop_emulating()' to be defined
in <machine/cpufunc.h> so remove them from there.

Obtained from: NetApp

Modify the default behavior of bhyve such that it no longer forces the use of
x2apic mode on the guest.

The guest can decide whether or not it wants to use legacy mmio or x2apic
access to the APIC by writing to the MSR_APICBASE register.

Obtained from: NetApp

Properly screen for the AND 0x81 instruction from the set
of group1 0x81 instructions that use the reg bits as an
extended opcode.

Still todo: properly update rflags.

Pointed out by: jilles@

Remove debug printf.

Pointed out by: emaste

Add support for the 0x81 AND instruction, now generated
by clang in the local APIC code.

0x81 is a read-modify-write instruction - the EPT check
that only allowed read or write and not both has been
relaxed to allow read and write.

Reviewed by: neel
Obtained from: NetApp

Cleanup the user-space paging exit handler now that the unified instruction
emulation is in place.

Obtained from: NetApp

Change emulate_rdmsr() and emulate_wrmsr() to return 0 on sucess and errno on
failure. The conversion from the return value to HANDLED or UNHANDLED can be
done locally in vmx_exit_process().

Obtained from: NetApp

Revamp the x86 instruction emulation in bhyve.

On a nested page table fault the hypervisor will:
- fetch the instruction using the guest %rip and %cr3
- decode the instruction in 'struct vie'
- emulate the instruction in host kernel context for local apic accesses
- any other type of mmio access is punted up to user-space (e.g. ioapic)

The decoded instruction is passed as collateral to the user-space process
that is handling the PAGING exit.

The emulation code is fleshed out to include more addressing modes (e.g. SIB)
and more types of operands (e.g. imm8). The source code is unified into a
single file (vmm_instruction_emul.c) that is compiled into vmm.ko as well
as /usr/sbin/bhyve.

Reviewed by: grehan
Obtained from: NetApp

Fix a bug in the MSI-X resource allocation for PCI passthrough devices.

In the case where the underlying host had disabled MSI-X via the
"hw.pci.enable_msix" tunable, the ppt_setup_msix() function would fail
and return an error without properly cleaning up. This in turn would
cause a page fault on the next boot of the guest.

Fix this by calling ppt_teardown_msix() in all the error return paths.

Obtained from: NetApp

Get rid of redundant comparision which is guaranteed to be "true" for unsigned
integers.

Obtained from: NetApp

Handle CPUID leaf 0x7 now that FreeBSD is using it.
Return 0's for now.

Reviewed by: neel
Obtained from: NetApp

Fix issue found with clang build. Avoid code insertion by the compiler
between inline asm statements that would in turn modify the flags
value set by the first asm, and used by the second.

Solve by making the common error block a string that can be pulled
into the first inline asm, and using symbolic labels for asm variables.

bhyve can now build/run fine when compiled with clang.

Reviewed by: neel
Obtained from: NetApp

Convert VMCS_ENTRY_INTR_INFO field into a vmcs identifier before passing it
to vmcs_getreg(). Without this conversion vmcs_getreg() will return EINVAL.

In particular this prevented injection of the breakpoint exception into the
guest via the "-B" option to /usr/sbin/bhyve which is hugely useful when
debugging guest hangs.

This was broken in r241921.

Pointy hat: me
Obtained from: NetApp

Corral all the host state associated with the virtual machine into its own file.

This state is independent of the type of hardware assist used so there is
really no need for it to be in Intel-specific code.

Obtained from: NetApp

Set the valid field of the newly allocated field as all other
vm page allocators do. This fixes a panic when a virtio block
device is mounted as root, with the host system dying in
vm_page_dirty with invalid bits.

Reviewed by: neel
Obtained from: NetApp

Unconditionally enable fpu emulation by setting CR0.TS in the host after the
guest does a vm exit.

This allows us to trap any fpu access in the host context while the fpu still
has "dirty" state belonging to the guest.

Reported by: "s vas" on freebsd-virtualization@
Obtained from: NetApp

If the guest vcpu wants to idle then use that opportunity to relinquish the
host cpu to the scheduler until the guest is ready to run again.

This implies that the host cpu utilization will now closely mirror the actual
load imposed by the guest vcpu.

Also, the vcpu mutex now needs to be of type MTX_SPIN since we need to acquire
it inside a critical section.

Obtained from: NetApp

Hide the monitor/mwait instruction capability from the guest until we know how
to properly intercept it.

Obtained from: NetApp

Maintain state regarding NMI delivery to guest vcpu in VT-x independent manner.
Also add a stats counter to count the number of NMIs delivered per vcpu.

Obtained from: NetApp

Test for AST pending with interrupts disabled right before entering the guest.

If an IPI was delivered to this cpu before interrupts were disabled
then return right away via vmx_setjmp() with a return value of VMX_RETURN_AST.

Obtained from: NetApp

Calculate the number of host ticks until the next guest timer interrupt.

This information will be used in conjunction with guest "HLT exiting" to
yield the thread hosting the virtual cpu.

Obtained from: NetApp

Add the guest physical address and r/w/x bits to
the paging exit in preparation for a rework of
bhyve MMIO handling.

Reviewed by: neel
Obtained from: NetApp

Provide per-vcpu locks instead of relying on a single big lock.

This also gets rid of all the witness.watch warnings related to calling
malloc(M_WAITOK) while holding a mutex.

Reviewed by: grehan

Fix warnings generated by 'debug.witness.watch' during VM creation and
destruction for calling malloc() with M_WAITOK while holding a mutex.

Do not allow vmm.ko to be unloaded until all virtual machines are destroyed.

Deliver the MSI to the correct guest virtual cpu.

Prior to this change the MSI was being delivered unconditionally to vcpu 0
regardless of how the guest programmed the MSI delivery.

Allocate memory pages for the guest from the host's free page queue.

It is no longer necessary to hard-partition the memory between the host
and guests at boot time.

Change vm_malloc() to map pages in the guest physical address space in 4KB
chunks. This breaks the assumption that the entire memory segment is
contiguously allocated in the host physical address space.

This also paves the way to satisfy the 4KB page allocations by requesting
free pages from the VM subsystem as opposed to hard-partitioning host memory
at boot time.

Get rid of assumptions in the hypervisor that the host physical memory
associated with guest physical memory is contiguous.

Add check to vm_gpa2hpa() that the range indicated by [gpa,gpa+len) is all
contained within a single 4KB page.

Get rid of assumptions in the hypervisor that the host physical memory
associated with guest physical memory is contiguous.

Rewrite vm_gpa2hpa() to get the GPA to HPA mapping by querying the nested
page tables.

Get rid of assumptions in the hypervisor that the host physical memory
associated with guest physical memory is contiguous.

In this case vm_malloc() was using vm_gpa2hpa() to indirectly infer whether
or not the address range had already been allocated.

Replace this instead with an explicit API 'vm_gpa_available()' that returns
TRUE if a page is available for allocation in guest physical address space.

Intel VT-x provides the length of the instruction at the time of the nested
page table fault. Use this when fetching the instruction bytes from the guest
memory.

Also modify the lapic_mmio() API so that a decoded instruction is fed into it
instead of having it fetch the instruction bytes from the guest. This is
useful for hardware assists like SVM that provide the faulting instruction
as part of the vmexit.

Add an option "-a" to present the local apic in the XAPIC mode instead of the
default X2APIC mode to the guest.

Add support for trapping MMIO writes to local apic registers and emulating them.

The default behavior is still to present the local apic to the guest in the
x2apic mode.

Add ioctls to control the X2APIC capability exposed by the virtual machine to
the guest.

At the moment this simply sets the state in the 'vcpu' instance but there is
no code that acts upon these settings.

Add an explicit exit code 'SPINUP_AP' to tell the controlling process that an
AP needs to be activated by spinning up an execution context for it.

The local apic emulation is now completely done in the hypervisor and it will
detect writes to the ICR_LO register that try to bring up the AP. In response
to such writes it will return to userspace with an exit code of SPINUP_AP.

Reviewed by: grehan

Stash the 'vm_exit' information in each 'struct vcpu'.

There is no functional change at this time but this paves the way for vm exit
handler functions to easily modify the exit reason going forward.

Restructure the x2apic access code in preparation for supporting memory mapped
access to the local apic.

The vlapic code is now aware of the mode that the guest is using to access the
local apic.

Reviewed by: grehan@

Add sysctls to display the total and free amount of hard-wired mem for VMs
# sysctl hw.vmm
hw.vmm.mem_free: 2145386496
hw.vmm.mem_total: 2145386496

Submitted by: Takeshi HASEGAWA hasegaw at gmail com

Force certain bits in %cr4 to be hard-wired to '1' or '0' from a guest's
perspective. If we don't do this some guest OSes (e.g. Linux) will reset
the CR4_VMXE bit in %cr4 with disastrous consequences.

Reported by: grehan

Verify that VMX operation has been enabled by BIOS before executing the
VMXON instruction.

Reported by "s vas" on freebsd-virtualization@

Until the issue of how to handle guest XCR0 state is resolved,
prevent CURRENT guests from hitting unhandled xsetbv exits
by hiding the xsave/osxsave/avx cpuid2 bits.

MSI-x interrupt support for PCI pass-thru devices.

Includes instruction emulation for memory r/w access. This
opens the door for io-apic, local apic, hpet timer, and
legacy device emulation.

Submitted by: ryan dot berryhill at sandvine dot com
Reviewed by: grehan
Obtained from: Sandvine

IFC @ r234692

sys/amd64/include/cpufunc.h
sys/amd64/include/fpu.h
sys/amd64/amd64/fpu.c
sys/amd64/vmm/vmm.c

- Add API to allow vmm FPU state init/save/restore.

FP stuff discussed with: kib

Remove duplicated license text.

Add support for running as a nested hypervisor under VMWare Fusion, on
systems with VT-x/EPT (e.g. Sandybridge Macbooks). This will most
likely work on VMWare Workstation8/Player4 as well. See the VMWare app
note at:

http://communities.vmware.com/docs/DOC-8970

Fusion doesn't propagate the PAT MSR auto save-restore entry/exit
control bits. Deal with this by noting that fact and setting up the
PAT MSR to essentially be a no-op - it is init'd to power-on default,
and a software shadow copy maintained.

Since it is treated as a no-op, o/s settings are essentially ignored.
This may not give correct results, but since the hypervisor is running
nested, a number of bets are already off.

On a quad-core/HT-enabled 'MacBook8,2', nested VMs with 1/2/4 vCPUs were
fired up. The more nested vCPUs the worse the performance, unless the VMs
were started up in multiplexed mode where things worked perfectly up to
the limit of 8 vCPUs.

Reviewed by: neel

Get rid of redundant initialization of 'dmask'. It was being re-initialized
shortly afterwards.

IFC @ r222830

Some tweaks to the CPUID support:
- Don't always pass the cpuid request to the current CPU as some nodes
we will emulate purely in software.
- Pass in the APIC ID of the virtual CPU so we can return the proper APIC
ID.
- Always report a completely flat topology with no SMT or multicore.
- Report the CPUID2_HV feature and implement support for the 0x40000000
CPUID level.
- Use existing constants from <machine/specialreg.h> when possible and
use cpu_feature2 when checking for VMX support.

Add a 'show vmcs' DDB command to dump state about the current CPU's
current VMCS.

Fix a long standing bug in VMXCTX_GUEST_RESTORE().

There was an assumption by the "callers" of this macro that on "return" the
%rsp will be pointing to the 'vmxctx'. The macro was not doing this and thus
when trying to restore host state on an error from "vmlaunch" or "vmresume"
we were treating the memory locations on the host stack as 'struct vmxctx'.
This led to all sorts of weird bugs like double faults or invalid instruction
faults.

This bug is exposed by the -O2 option used to compile the kernel module. With
the -O2 flag the compiler will optimize the following piece of code:

int loopstart = 1;
...
if (loopstart) {
loopstart = 0;
vmx_launch();
} else
vmx_resume();

into this:

vmx_launch();

Since vmx_launch() and vmx_resume() are declared to be __dead2 functions the
compiler is free to do this. The compiler has no way to know that the
functions return indirectly through vmx_setjmp(). This optimization in turn
leads us to trigger the bug in VMXCTX_GUEST_RESTORE().

With this change we can boot a 8.1 guest on a 9.0 host.

Reported by: jhb@

Avoid unnecessary sign extension when promoted to a 64-bit integer.

This was benign because the interruption info field is a 32-bit quantity and
the hardware guarantees that the upper 32-bits are all zeros. But it did make
reading the objdump output very confusing.

Changes to allow the GENERIC+bhye kernel built from this branch to
run as a 1/2 CPU guest on an 8.1 bhyve host.

bhyve/inout.c
inout.h
fbsdrun.c
- Rather than exiting on accesses to unhandled i/o ports, emulate
hardware by returning -1 on reads and ignoring writes to unhandled
ports. Support the previous mode by allowing a 'strict' parameter
to be set from the command line.
The 8.1 guest kernel was vastly cut down from GENERIC and had no
ISA devices. Booting GENERIC exposes a massive amount of random
touching of i/o ports (hello syscons/vga/atkbdc).

bhyve/consport.c
dev/bvm/bvm_console.c
- implement a simplistic signature for the bvm console by returning
'bv' for an inw on the port. Also, set the priority of the console
to CN_REMOTE if the signature was returned. This works better in
an environment where multiple consoles are in the kernel (hello syscons)

bhyve/rtc.c
- return 0 for the access to RTC_EQUIPMENT (yes, you syscons)

amd64/vmm/x86.c
x86.h
- hide a bunch more CPUID leaf 1 bits from the guest to prevent
cpufreq drivers from probing.
The next step will be to move CPUID handling completely into
user-space. This will allow the full spectrum of changes from
presenting a lowest-common-denominator CPU type/feature set, to
exposing (almost) everything that the host can support.

Reviewed by: neel
Obtained from: NetApp

Enable handling of 1GB pages in the direct map since HEAD supports those.

Submitted by: neel

First cut at porting the kernel portions of 221828 and 221905 from the
BHyVe reference branch to HEAD.

Import of bhyve hypervisor and utilities, part 1.
vmm.ko - kernel module for VT-x, VT-d and hypervisor control
bhyve - user-space sequencer and i/o emulation
vmmctl - dump of hypervisor register state
libvmm - front-end to vmm.ko chardev interface

bhyve was designed and implemented by Neel Natu.

Thanks to the following folk from NetApp who helped to make this available:
Joe CaraDonna
Peter Snyder
Jeff Heller
Sandeep Mann
Steve Miller
Brian Pawlowski