MFC r358315:
Fix IBRS for machines with IBRS_ALL capability.

MFC r337715, r337751, r337754, r337758, r337813, r338354, r338687,
r339124, r341821:
Add support for boot-time Intel microcode loading.

MFC r334340: add support for console resuming, implement it for uart, use on x86

MFC r332918, r333222: go deeper for ACPI suspend bounce test

debug.acpi.suspend_bounce sysctl now allows a deeper dive into the sleep
abyss. The system will execute the suspend sequence up to the call to
AcpiEnterSleepState(). That includes saving processor contexts and
parking APs. Then, instead of actually entering the sleep state, the
BSP will call resumectx() to emulate the wakeup. The APs should get
restarted by the sequence of Init and Startup IPIs that BSP sends to
them.

AcpiOsEnterSleep() is used to implement this feature.

Joint work with jkim.

MFC r334004:
Add Intel Spec Store Bypass Disable control.

This also includes the i386/include/pcpu.h part of the r334018.

Security: CVE-2018-3639
Approved by: re (gjb)

MFC r333125:
Turn off IBRS on suspend.

Approved by: re (marius)

MFC r327056: Use resume_cpus() instead of restart_cpus() to resume from ACPI suspension.

MFC r319825:
More accurately handle early EFER restoration on resume.

Approved by: re (delphij)

MFC r318318:
Ensure that resume path on amd64 only accesses page tables for normal
operation after processor is configured to allow all required
features.

MFC r313154:
For i386, remove config options CPU_DISABLE_CMPXCHG, CPU_DISABLE_SSE
and device npx.

MFC r302793:
Allow ACPI wakeup code and page tables to be stored in non-contiguous pages.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Use M_NOWAIT when allocating memory for the ACPI wakeup handler.

If the allocation attempt fails, we may otherwise VM_WAIT after a failed
attempt to reclaim contiguous memory in the requested range. After r297466,
this results in the thread going to sleep, causing a hang during boot.

Reviewed by: jkim, kib
Approved by: re (gjb)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D6945

As <machine/pmap.h> is included from <vm/pmap.h>, there is no need to
include it explicitly when <vm/pmap.h> is already included.

Reviewed by: alc, kib
Differential Revision: https://reviews.freebsd.org/D5373

Merge ACPICA 20150619.

Add config option PAE_TABLES for the i386 kernel. It switches pmap to
use PAE format for the page tables, but does not incur other
consequences of the full PAE config. In particular, vm_paddr_t and
bus_addr_t are left 32bit, and max supported memory is still limited
by 4GB.

The option allows to have nx permissions for memory mappings on i386
kernel, while keeping the usual i386 KBI and avoiding the kernel data
sizing problems typical for the PAE config.

Intel documented that the PAE format for page tables is available
starting with the Pentium Pro, but it is possible that the plain
Pentium CPUs have the required support (Appendix H). The goal is to
enable the option and non-exec mappings on i386 for the GENERIC
kernel. Anybody wanting a useful system on 486, have to reconfigure
the modern i386 kernel anyway.

Discussed with: alc, jhb
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks

Fix build on i386 without "device apic"

Reviewed by: kib

Fix UP build.

Sponsored by: The FreeBSD Foundation
MFC after: 2 months

Initialize x2APIC mode on the resume path before accessing LAPIC.

Remove unneeded disable of LAPIC in the native_lapic_xapic_mode(). We
attempt to send wakeup IPI on the resume path right after BSP wakeup,
so disabling is wrong.

Reported and tested by: glebius, "Ranjan1018 ." <214748mv@gmail.com>
Sponsored by: The FreeBSD Foundation
MFC after: 2 months

MFamd64: Add support for extended FPU states on i386. This includes
support for AVX on i386.
- Similar to amd64, move the FPU save area out of the PCB and instead
store saved FPU state in a variable-sized buffer after the PCB on the
stack.
- To support the variable PCB location, alter the locore code to only use
the bottom-most page of proc0stack for init386(). init386() returns
the correct stack pointer to locore which adjusts the stack for thread0
before calling mi_startup().
- Don't bother setting cr3 in thread0's pcb in locore before calling
init386(). It wasn't used (init386() overwrote it at the end) and
it doesn't work with the variable-sized FPU save area.
- Remove the new-bus attachment from npx. This was only ever useful for
external co-processors using IRQ13, but those have not been supported
for several years. npxinit() is now called much earlier during boot
(init386()) similar to amd64.
- Implement PT_{GET,SET}XSTATE and I386_GET_XFPUSTATE.
- npxsave() is now only called from context switch contexts so it can
use XSAVEOPT.

Differential Revision: https://reviews.freebsd.org/D1058
Reviewed by: kib
Tested on: FreeBSD/i386 VM under bhyve on Intel i5-2520

Create a separate structure for per-CPU state saved across suspend and
resume that is a superset of a pcb. Move the FPU state out of the pcb and
into this new structure. As part of this, move the FPU resume code on
amd64 into a C function. This allows resumectx() to still operate only on
a pcb and more closely mirrors the i386 code.

Reviewed by: kib (earlier version)

Save and restore FPU state across suspend and resume. In earlier revisions
of this patch, resumectx() called npxresume() directly, but that doesn't
work because resumectx() runs with a non-standard %cs selector. Instead,
all of the FPU suspend/resume handling is done in C.

MFC after: 1 week

Move fpusave() wrapper for suspend hander to sys/amd64/amd64/fpu.c.

Inspired by: jhb

Fix i386 build.

Pointy hat to: jhb

Add a resume hook for bhyve that runs a function on all CPUs during
resume. For Intel CPUs, invoke vmxon for CPUs that were in VMX mode
at the time of suspend.

Reviewed by: neel

Add support for suspend/resume/migration operations when running as a
Xen PVHVM guest.

Submitted by: Roger Pau Monné
Sponsored by: Citrix Systems R&D
Reviewed by: gibbs
Approved by: re (blanket Xen)
MFC after: 2 weeks

sys/amd64/amd64/mp_machdep.c:
sys/i386/i386/mp_machdep.c:
- Make sure that are no MMU related IPIs pending on migration.
- Reset pending IPI_BITMAP on resume.
- Init vcpu_info on resume.

sys/amd64/include/intr_machdep.h:
sys/i386/include/intr_machdep.h:
sys/x86/acpica/acpi_wakeup.c:
sys/x86/x86/intr_machdep.c:
sys/x86/isa/atpic.c:
sys/x86/x86/io_apic.c:
sys/x86/x86/local_apic.c:
- Add a "suspend_cancelled" parameter to pic_resume(). For the
Xen PIC, restoration of interrupt services differs between
the aborted suspend and normal resume cases, so we must provide
this information.

sys/dev/acpica/acpi_timer.c:
sys/dev/xen/timer/timer.c:
sys/timetc.h:
- Don't swap out "suspend safe" timers across a suspend/resume
cycle. This includes the Xen PV and ACPI timers.

sys/dev/xen/control/control.c:
- Perform proper suspend/resume process for PVHVM:
- Suspend all APs before going into suspension, this allows us
to reset the vcpu_info on resume for each AP.
- Reset shared info page and callback on resume.

sys/dev/xen/timer/timer.c:
- Implement suspend/resume support for the PV timer. Since FreeBSD
doesn't perform a per-cpu resume of the timer, we need to call
smp_rendezvous in order to correctly resume the timer on each CPU.

sys/dev/xen/xenpci/xenpci.c:
- Don't reset the PCI interrupt on each suspend/resume.

sys/kern/subr_smp.c:
- When suspending a PVHVM domain make sure there are no MMU IPIs
in-flight, or we will get a lockup on resume due to the fact that
pending event channels are not carried over on migration.
- Implement a generic version of restart_cpus that can be used by
suspended and stopped cpus.

sys/x86/xen/hvm.c:
- Implement resume support for the hypercall page and shared info.
- Clear vcpu_info so it can be reset by APs when resuming from
suspension.

sys/dev/xen/xenpci/xenpci.c:
sys/x86/xen/hvm.c:
sys/x86/xen/xen_intr.c:
- Support UP kernel configurations.

sys/x86/xen/xen_intr.c:
- Properly rebind per-cpus VIRQs and IPIs on resume.

Merge ACPICA 20120816.

- Remove unused code for CR3 and CR4.
- Fix few style(9) nits while I am here.

Share IPI init and startup code of mp_machdep.c with acpi_wakeup.c
as ipi_startup().

Add x86/acpica/acpi_wakeup.c for amd64 and i386. Difference of
suspend/resume procedures are minimized among them.

common:
- Add global cpuset suspended_cpus to indicate APs are suspended/resumed.
- Remove acpi_waketag and acpi_wakemap from acpivar.h (no longer used).
- Add some variables in acpi_wakecode.S in order to minimize the difference
among amd64 and i386.
- Disable load_cr3() because now CR3 is restored in resumectx().

amd64:
- Add suspend/resume related members (such as MSR) in PCB.
- Modify savectx() for above new PCB members.
- Merge acpi_switch.S into cpu_switch.S as resumectx().

i386:
- Merge(and remove) suspendctx() into savectx() in order to match with
amd64 code.

Reviewed by: attilio@, acpi@