Surround full mp_cpu_start() method with NLAPIC > 0 guard.

Initialization is based on x86_ipi* functions, which are implemented only
when lapic flag is enabled.

x86: Sprinkle extensive commentary about %fs/%gs initialization.

Plus some other side quests like the three-stage GDT metamorphosis
lifecycle.

No functional change intended.

x86: Call fpuinit_mxcsr_mask only once.

No need to call it again and again on the secondary CPUs to compute
what should be the same mxcsr mask. (If it's not, we have deeper
problems!)

x86: Assert kpreempt_disabled() in cpu_load_pmap.

No functional change intended. Just makes it easier to audit
curcpu() usage.

x86: Support EFI runtime services.

This creates a special pmap, efi_runtime_pmap, which avoids setting
PTE_U but allows mappings to lie in what would normally be user VM --
this way we don't fall afoul of SMAP/SMEP when executing EFI runtime
services from CPL 0. SVS does not apply to the EFI runtime pmap.

The mechanism is intended to work with either physical addressing or
virtual addressing; currently the bootloader does physical addressing
but in principle it could be modified to do virtual addressing
instead, if it allocated virtual pages, assigned them in the memory
map, and issued RT->SetVirtualAddressMap.

Not sure pmap_activate_sync and pmap_deactivate_sync are correct,
need more review from an x86 wizard.

If this causes fallout, it can be disabled temporarily without
reverting anything by just making efi_runtime_init return immediately
without doing anything, or by removing options EFI_RUNTIME.

amd64-only for now pending type fixes and testing on i386.

x86: Split most of pmap.h into pmap_private.h or vmparam.h.

This way pmap.h only contains the MD definition of the MI pmap(9)
API, which loads of things in the kernel rely on, so changing x86
pmap internals no longer requires recompiling the entire kernel every
time.

Callers needing these internals must now use machine/pmap_private.h.
Note: This is not x86/pmap_private.h because it contains three parts:

1. CPU-specific (different for i386/amd64) definitions used by...

2. common definitions, including Xenisms like xpmap_ptetomach,
further used by...

3. more CPU-specific inlines for pmap_pte_* operations

So {amd64,i386}/pmap_private.h defines 1, includes x86/pmap_private.h
for 2, and then defines 3. Maybe we should split that out into a new
pmap_pte.h to reduce this trouble.

No functional change intended, other than that some .c files must
include machine/pmap_private.h when previously uvm/uvm_pmap.h
polluted the namespace with pmap internals.

Note: This migrates part of i386/pmap.h into i386/vmparam.h --
specifically the parts that are needed for several constants defined
in vmparam.h:

VM_MAXUSER_ADDRESS
VM_MAX_ADDRESS
VM_MAX_KERNEL_ADDRESS
VM_MIN_KERNEL_ADDRESS

Since i386 needs PDP_SIZE in vmparam.h, I added it there on amd64
too, just to keep things parallel.

Split TSC calibtration into many small steps and disable interrupts
for each step. Also add debug messages.

x86, arm: Allow fpu_kern_enter/leave while cold.

Normally these are forbidden above IPL_VM, so that FPU usage doesn't
block IPL_SCHED or IPL_HIGH interrupts. But while cold, e.g. during
builtin module initialization at boot, all interrupts are blocked
anyway so it's a moot point.

Also initialize x86 cpu_info_primary.ci_kfpu_spl to -1 so we don't
trip over an assertion about it while cold -- the assertion is meant
to detect reentrance into fpu_kern_enter/leave, which is prohibited.

Also initialize cpu0's ci_kfpu_spl.

KNF. No functional change.

Merge thorpej-cfargs2.

branches: 1.200.8;
Merge thorpej-cfargs branch:

Simplify and make extensible the config_search() / config_found() /
config_attach() interfaces: rather than having different variants for
which arguments you want pass along, just have a single call that
takes a variadic list of tag-value arguments.

Adjust all call sites:
- Simplify wherever possible; don't pass along arguments that aren't
actually needed.
- Don't be explicit about what interface attribute is attaching if
the device only has one. (More simplification.)
- Add a config_probe() function to be used in indirect configuiration
situations, making is visibly easier to see when indirect config is
in play, and allowing for future change in semantics. (As of now,
this is just a wrapper around config_match(), but that is an
implementation detail.)

Remove unnecessary or redundant interface attributes where they're not
needed.

There are currently 5 "cfargs" defined:
- CFARG_SUBMATCH (submatch function for direct config)
- CFARG_SEARCH (search function for indirect config)
- CFARG_IATTR (interface attribte)
- CFARG_LOCATORS (locators array)
- CFARG_DEVHANDLE (devhandle_t - wraps OFW, ACPI, etc. handles)

...and a sentinel value CFARG_EOL.

Add some extra sanity checking to ensure that interface attributes
aren't ambiguous.

Use CFARG_DEVHANDLE in MI FDT, OFW, and ACPI code, and macppc and shark
ports to associate those device handles with device_t instance. This
will trickle trough to more places over time (need back-end for pre-OFW
Sun OBP; any others?).

branches: 1.199.4;
Don't do extra work finding the power of 2 for values we are not going to
use. Explain that cpu_hatch has not been called yet, so no cpu_probe either
so the cache info is 0 for AP's.

move lcall sniffer to x86_machdep since xen/pv has its own cpu.c

PR/55547: Dan Plassche: Fix BSD/OS binary emulation.
Centralize lcall sniffer and recognize the BSD/OS flavor.

Use CPU_IS_PRIMARY macro in cpu_stop(), cpu_resume(), and cpu_get_tsc_freq()
on x86.

OK kamil@

Introduce per-cpu IDTs

This is realized by following modifications:
- Add IDT pages and its allocation maps for each cpu in "struct cpu_info"
- Load per-cpu IDTs at cpu_init_idt(struct cpu_info*)
- Copy the IDT entries for cpu0 to other CPUs at attach
- These are, for example, exceptions, db, system calls, etc.

And, added a kernel option named PCPU_IDT to enable the feature.

Serialize rdtsc using with lfence, mfence or cpuid to read TSC more precisely.

x86/x86/tsc.c rev. 1.67 reduced cache problem and got big improvement, but it
still has room. I measured the effect of lfence, mfence, cpuid and rdtscp.
The impact to TSC skew and/or drift is:

AMD: mfence > rdtscp > cpuid > lfence-serialize > lfence = nomodify
Intel: lfence > rdtscp > cpuid > nomodify

So, mfence is the best on AMD and lfence is the best on Intel. If it has no
SSE2, we can use cpuid.

NOTE:
- An AMD's document says DE_CFG_LFENCE_SERIALIZE bit can be used for
serializing, but it's not so good.
- On Intel i386(not amd64), it seems the improvement is very little.
- rdtscp instruct can be used as serializing instruction + rdtsc, but
it's not good as [lm]fence. Both Intel and AMD's document say that
the latency of rdtscp is bigger than rdtsc, so I suspect the difference
of the result comes from it.

g/c vm_page_zero_enable

- Recalibrate the APIC timer using the TSC, once the TSC has in turn been
recalibrated using the HPET. This gets the clock interrupt firing more
closely to HZ.

- Undo change with recent Xen merge and go back to starting the clocks in
initclocks() on the boot CPU, and in cpu_hatch() on secondary CPUs.

- On reflection don't use HPET delay any more, it works very well but means
going over the bus. It's enough to use HPET to calibrate the TSC and
APIC.

Tested on amd64 native, xen and xen PVH.

Don't use TSC freq value from CPUID if calibration works.

- When it's the first call of cpu_get_tsc_freq() the HPET is not initialized,
so try to use CPUID to get TSC freq.
- If it's the 2nd call, don't use CPUID. Instead, print the difference
between the calibrated value and CPUID's value if the verbose mode is set.

Fix the TSC timecounter (on the systems I have access to):

- Make the early i8254-based calculation of frequency a bit more accurate.

- Keep track of how far the HPET & TSC advance between HPET attach and
secondary CPU boot, and use to compute an accurate value before attaching
the timecounter. Initial idea from joerg@.

- When determining skew and drift between CPUs, make each measurement 1000
times and pick the lowest observed value. Increase the error threshold to
1000 clock cycles.

- Use the frequency computed on the boot CPU for secondary CPUs too.

- Remove cpu_counter_serializing().

Introduce Xen PVH support in GENERIC.
This is compiled in with
options XENPVHVM
x86 changes:
- add Xen section and xen pvh entry points to locore.S. Set vm_guest
to VM_GUEST_XENPVH in this entry point.
Most of the boot procedure (especially page table setup and switch to
paged mode) is shared with native.
- change some x86_delay() to delay_func(), which points to x86_delay() for
native/HVM, and xen_delay() for PVH

Xen changes:
- remove Xen bits from init_x86_64_ksyms() and init386_ksyms()
and move to xen_init_ksyms(), used for both PV and PVH
- set ISA no-legacy-devices property for PVH
- factor out code from Xen's cpu_bootconf() to xen_bootconf()
in xen_machdep.c
- set up a specific pvh_consinit() which starts with printk()
(which uses a simple hypercall that is available early) and switch to
xencons when we can use pmap_kenter_pa().

Back out HPET delay & TSC changes to rule them out as the cause for recent
hangs during boot etc.

Merge the bouyer-xenpvh branch, bringing in Xen PV drivers support under HVM
guests in GENERIC.
Xen support can be disabled at runtime with
boot -c
disable hypervisor

- Install HPET based DELAY() before going multiuser then recalibrate the TSC.
Idea from joerg@.

- Take overhead into account when computing CPU frequency.

- Don't flush cache before computing TSC skew.

Get TSC frequency from CPUID 0x15 and/or x16 for newer Intel processors.

- If the max CPUID leaf is >= 0x15, take TSC value from CPUID. Some processors
can take TSC/core crystal clock ratio but core crystal clock frequency
can't be taken. Intel SDM give us the values for some processors.
- It also required to change lapic_per_second to make LAPIC timer correctly.
- Add new file x86/x86/identcpu_subr.c to share common subroutines between
kernel and userland. Some code in x86/x86/identcpu.c and cpuctl/arch/i386.c
will be moved to this file in future.
- Add comment to clarify.

Whitespace fix. No functional change.

Revert, wrong branch

Skip cx8_spllower patch if we're running on any form of Xen PV,
we can't handle PV interrupts with a single atomic op here.
Enable x86_patch() for Xen too.

branches: 1.181.4;
If "application processors" were skipped/disabled at boot time (due to
RB_MD1 being set), don't try to examine the featurebus info, since it
was never retrieved. Addresses kern/54815

XXX pullup-9

Make "mach cpu" in ddb show the IPL for each cpu.

branches: 1.179.2;
Some more CPU topology stuff:

- Use cegger@'s ACPI SRAT parsing code to figure out NUMA node ID for each
CPU as it is attached.

- For scheduler experiments with SMT, flag CPUs with the lowest numbered SMT
IDs as "primaries", link back to the primaries from secondaries, and build
a circular list of CPUs in each package with identical SMT IDs.

- No need for package/core/smt/numa IDs to be anything other than a u_int.

Get a Hyper-V virtual processor id in cpu_hatch().

Currently, it is got in config_interrupts context.
However, since it is required when attaching a device,
it is got earlier than now.

Add a small API for in-kernel FPU operations.

fpu_kern_enter();
/* do FPU stuff */
fpu_kern_leave();

cpu_need_resched():

- Remove all code that should be MI, leaving the bare minimum under arch/.
- Make the required actions very explicit.
- Pass in LWP pointer for convenience.
- When a trap is required on another CPU, have the IPI set it locally.
- Expunge cpu_did_resched().

- On-demand zeroing pages with MOVNTI is crazy. It empties L1/L2/L3.
- Disable zeroing in the idle loop. That needs a cache-friendly strategy.

Result: 3 to 4% reduction in kernel build time on my test system.
Inspired by a discussion with Mateusz Guzik and David Maxwell.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

Rewrite the FPU code on x86. This greatly simplifies the logic and removes
the dependency on IPL_HIGH. NVMM is updated accordingly. Posted on
port-amd64 a week ago.

Bump the kernel version to 9.99.16.

avoid misalignment in 32 bit kernels and "mach cpu".

branches: 1.171.2;
Add PCID support in SVS. This avoids TLB flushes during kernel<->user
transitions, which greatly reduces the performance penalty introduced by
SVS.

We use two ASIDs, 0 (kern) and 1 (user), and use invpcid to flush pages
in both ASIDs.

The read-only machdep.svs.pcid={0,1} sysctl is added, and indicates whether
SVS+PCID is in use.

Change the effect of SVS on the TLB. Keep CR4_PGE set when SVS is enabled,
but don't use PTE_G on the kernel PTEs in general.

Add PTE_G on only a few pages, that are already leaked to userland and do
not contain secrets.

This slightly improves syscall performance.

Remove 'ci_svs_kpdirpa', unused. While here fix a few comments here and
there, reduces a future diff.

Start replacing the x86 PTE bits.

Added Microsoft Hyper-V support. It ported from OpenBSD and FreeBSD.

graphical console is not work on Gen.2 VM yet. To use the serial console,
enter "consdev com,0x3f8,115200" on efiboot.

Welcome XENPVHVM mode.

It is UP only, has xbd(4) and xennet(4) as PV drivers.

The console is com0 at isa and the native portion is very
rudimentary AT architecture, so is probably suboptimal to
run without PV support.

Fix NLAPIC, NISA and NIOAPIC related conditional compile errors.

This will allow us to now compile an amd64 kernel without PCI.

No functional changes.

Hypothetically speaking, if one were to want to compile a

'no options MULTIPROCESSOR'

kernel, these files may trip up the build.

Fix them by moving around the #defines as originally intended.

No Functional Changes.

Stop panic()ing on a UP system.

The reason for the panic is that the cpu_attach() doesn't run to
completion because it thinks it's run past maxcpus (which in the case
of UP), is 1.

This is because on x86 at least, mi_cpu_attach() is called *before*
configure() (and thus the cpu_match()/cpu_attach() pair). Thus ncpu
has already been incremented by the time MD cpu_attach() is called.

Fix this.

Add a comment explaining an important rule. Just to better highlight that
this rule is actually not respected.

Rename min/max -> uimin/uimax for better honesty.

These functions are defined on unsigned int. The generic name
min/max should not silently truncate to 32 bits on 64-bit systems.
This is purely a name change -- no functional change intended.

HOWEVER! Some subsystems have

#define min(a, b) ((a) < (b) ? (a) : (b))
#define max(a, b) ((a) > (b) ? (a) : (b))

even though our standard name for that is MIN/MAX. Although these
may invite multiple evaluation bugs, these do _not_ cause integer
truncation.

To avoid `fixing' these cases, I first changed the name in libkern,
and then compile-tested every file where min/max occurred in order to
confirm that it failed -- and thus confirm that nothing shadowed
min/max -- before changing it.

I have left a handful of bootloaders that are too annoying to
compile-test, and some dead code:

cobalt ews4800mips hp300 hppa ia64 luna68k vax
acorn32/if_ie.c (not included in any kernels)
macppc/if_gm.c (superseded by gem(4))

It should be easy to fix the fallout once identified -- this way of
doing things fails safe, and the goal here, after all, is to _avoid_
silent integer truncations, not introduce them.

Maybe one day we can reintroduce min/max as type-generic things that
never silently truncate. But we should avoid doing that for a while,
so that existing code has a chance to be detected by the compiler for
conversion to uimin/uimax without changing the semantics until we can
properly audit it all. (Who knows, maybe in some cases integer
truncation is actually intended!)

Remove useless/outdated comments. No functional change.

Oh. Don't call svs_pdir_switch if SVS is disabled, that's not needed.

I was playing around with PMCs, and was wondering why some cache misses
were occurring in svs_pdir_switch while I had SVS disabled.

branches: 1.158.2;
Revert jdolecek's changes related to FXSAVE. They just didn't make any
sense and were trying to hide a real bug, which is, that there is for some
reason a wrong stack alignment that causes FXSAVE to fault in
fpuinit_mxcsr_mask. As seen in current-users@ yesterday, rdi % 16 = 8. And
as seen several months ago, as well.

The rest of the changes in XSAVE are wrong too, but I'll let him fix these
ones.

as a stop-gap, make fpuinit_mxcsr_mask() for native independant of
XSAVE as it should be, only xen case checks the flag now; need to
investigate further why exactly the fault happens for the xen
no-xsave case

pointed out by maxv

fix FPU initialization on Xen to allow e.g. AVX when supported by hardware;
only use XSAVE when the the CPUID OSXSAVE bit is set, as this seems to be
reliable indication

tested with Xen 4.2.6 DOM0/DOMU on Intel CPU, without and with no-xsave flag,
so should work also on those AMD CPUs, which have XSAVE disabled by default;
also tested with Xen DOM0 4.8.3

fixes PR kern/50332 by Torbjorn Granlund; sorry it took three years to address

XXX pullup netbsd-8

Call cpu_speculation_init on i386 too. We don't have IBRS for i386, but
we do have the AMD DIS_IND method.

Enable the SpectreV2 mitigation by default at boot time.

Move the SpectreV2 mitigation code into a dedicated spectre.c file. The
content of the file is taken from the end of cpu.c, and is copied as-is.

Remove #ifdef XEN (Xen has its own cpu.c), and add a comment.

Spectre V2 mitigation for certain families of AMD CPUs.

A new sysctl is added, machdep.spectreV2.mitigated, that controls whether
Spectre V2 is mitigated. For now it defaults to "false".

The code is written in such a way that there can be several methods. For
now only one method is supported, on AMD Families 10h, 12h and 16h, where
an MSR is available to disable branch prediction entirely.

Compile-tested on Intel, AMD will be tested soon.

Explain the TSC drift thing.

branches: 1.149.2;
Remove svs_pgg_update(). Instead of manually changing PG_G on each page,
we can disable the global-paging mechanism in %cr4 with CR4_PGE. Do that.

In addition, install CR4_PGE when SVS is disabled manually (via the
sysctl).

Now, doing "sysctl -w machdep.svs_enabled=0" restores the performance
completely, exactly as if SVS hadn't been enabled in the first place.

Add a dynamic detection for SVS.

The SVS_* macros are now compiled as skip-noopt. When the system boots, if
the cpu is from Intel, they are hotpatched to their real content.
Typically:

jmp 1f
int3
int3
int3
... int3 ...
1:

gets hotpatched to:

movq SVS_UTLS+UTLS_KPDIRPA,%rax
movq %rax,%cr3
movq CPUVAR(KRSP0),%rsp

These two chunks of code being of the exact same size. We put int3 (0xCC)
to make sure we never execute there.

In the non-SVS (ie non-Intel) case, all it costs is one jump. Given that
the SVS_* macros are small, this jump will likely leave us in the same
icache line, so it's pretty fast.

The syscall entry point is special, because there we use a scratch uint64_t
not in curcpu but in the UTLS page, and it's difficult to hotpatch this
properly. So instead of hotpatching we declare the entry point as an ASM
macro, and define two functions: syscall and syscall_svs, the latter being
the one used in the SVS case.

While here 'syscall' is optimized not to contain an SVS_ENTER - this way
we don't even need to do a jump on the non-SVS case.

When adding pages in the user page tables, make sure we don't have PG_G,
now that it's dynamic.

A read-only sysctl is added, machdep.svs_enabled, that tells whether the
kernel uses SVS or not.

More changes to come, svs_init() is not very clean.

Add SMAP support for i386.

Introduce a new svs_page_add function, which can be used to map in the user
space a VA from the kernel space.

Use it to replace the PDIR_SLOT_PCPU slot: at boot time each CPU creates
its own slot which maps only its own pcpu_entry plus the common area (IDT+
LDT).

This way, the pcpu areas of the remote CPUs are not mapped in userland.

Changing CR4 register may change cpuid values. For example, setting
CR4_OSXSAVE sets CPUID2_OSXSAVE. The CPUID2_OSXSAVE is in ci_feat_val[1],
so update it after changing CR4.

Add a new option, SVS (for Separate Virtual Space), that unmaps kernel
pages when running in userland. For now, only the PTE area is unmapped.

Sent on tech-kern@.

Use uvm_km_alloc instead of kmem_zalloc.

Add a __HAVE_PCPU_AREA option, enabled by default on native amd64 but not
Xen.

With this option, the CPU structures that must always be present in the
CPU's page tables are moved on L4 slot 384, which means address
0xffffc00000000000.

A new pcpu_area structure is defined. It contains shared structures (IDT,
LDT), and then an array of pcpu_entry structures, indexed by cpu_index(ci).
Theoretically the LDT should be in the array, but this will be done later.

During the boot procedure, cpu0 calls pmap_init_pcpu, which creates a
page tree that is able to map the pcpu_area structure entirely. cpu0 then
immediately maps the shared structures. Later, every CPU goes through
cpu_pcpuarea_init, which allocates physical pages and kenters the relevant
pcpu_entry to them. Finally, each pointer is replaced to point to pcpuarea.

The point of this change is to make sure that the structures that must
always be present in the page tables have their own L4 slot. Until now
their L4 slot was that of pmap_kernel, and making a distinction between
what must be mapped and what does not need to be was complicated.

Even in the non-speculative-bug case this change makes some sense: there
are several x86 instructions that leak the addresses of the CPU structures,
and putting these structures inside pmap_kernel actually offered a way to
compute the address of the kernel heap - which would have made ASLR on it
plainly useless, had we implemented that.

Note that, for now, pcpuarea does not contain rsp0.

Unfortunately this change adds many #ifdefs, and makes the code harder to
understand. There is also some duplication, but that will be solved later.

Recommit

http://mail-index.netbsd.org/source-changes/2017/11/08/msg089525.html

but use __INITIAL_MXCSR_MASK__ on Xen until someone figures out what's
wrong with the Xen fpu.

Revert http://mail-index.netbsd.org/source-changes/2017/11/08/msg089525.html,
it breaks Xen:
http://www-soc.lip6.fr/~bouyer/NetBSD-tests/xen/HEAD/amd64/201711082340Z_anita.txt

Call fpuinit_mxcsr_mask in cpu_init, after cr4 is initialized, but before
touching xcr0. Then use clts/stts instead of modifying cr0, and enable the
mxcsr_mask detection on Xen.

Have the cpu clear PSL_D automatically when entering the kernel via a
syscall. Then, don't clear PSL_D and PSL_AC in the syscall entry point,
they are now both cleared by the cpu (faster). However they still need to
be manually cleared in the interrupt/trap entry points.

Add support for SMAP on amd64.

PSL_AC is cleared from %rflags in each kernel entry point. In the copy
sections, a copy window is opened and the kernel can touch userland
pages. This window is closed when the kernel is done, either at the end
of the copy sections or in the fault-recover functions.

This implementation is not optimized yet, due to the fact that INTRENTRY
is a macro, and we can't hotpatch macros.

Sent on tech-kern@ a month or two ago, tested on a Kabylake.

Pack the useful variables at the end of the trampoline page; eliminates
a hard-coded dependency on KERNBASE. Note that I cannot test this change
on i386 right now, but it seems fine enough.

Remove TRAPLOG from i386. Nowadays there are better instrumentation tools,
in both software and hardware.

style, and move some i386-specific code into i386/

Localify. By the way, we should use a different stack for NMIs.

cpu_trace is no more, remove vestige of it that broke ALL kernel.

Further reduce the loop counter so that hatching completes before the
boot processor times us out.

Add a nice big XXX comment for why the counter is so low.

XXX Will need to pullup to NetBSD-7 branch

branches: 1.130.2;

And now do what 1.128 should have done, and put back the (now re-)used
variable that had earlier been deleted, when it's use was removed in
1.126, but wasn't restored in 1.127.

Revert previous. Removing unused variable declarations is only a good
idea when the variable is, in fact, unused.

Remove unused variabe (I reverted too much in previous commit!)

Partially revert previous. Rather than completely removing the loop
around calls to x86_pause(), just drastically reduce the repeat count.
It's still good to have some real delay here (among other things, for
letting the TSCs drift).

As discussed on IRC

Do not pause many times between testing if the CPU can go.

This only impacts QEMU as QEMU's implementation of pause is
significantly slower than its implementation of nop.

PR kern/51623: running qemu-x86_64 with -smp 4 - the additional
CPUs don't start.

x86: Add preliminary x2APIC support.

x2APIC is used only when x2APIC is enabled in BIOS/UEFI.
LAPIC ID is not supported above 256.

use CR8 instead of LAPIC Task Priority register on x86-64.

Instead of using a global array with per-cpu indexes, embed the tmp VAs
into cpu_info directly. This concerns only {i386, Xen-i386, Xen-amd64},
because amd64 already has a direct map that is way faster than that.

There are two major issues with the global array: maxcpus entries are
allocated while it is unlikely that common i386 machines have so many
cpus, and the base VA of these entries is not cache-line-aligned, which
mostly guarantees cache-line-thrashing each time the VAs are entered.

Now the number of tmp VAs allocated is proportionate to the number of CPUs
attached (which therefore reduces memory consumption), and the base is
properly aligned.

On my 3-core AMD, the number of DC_refills_L2 events triggered when
performing 5x10^6 calls to pmap_zero_page on two dedicated cores is on
average divided by two with this patch.

Discussed on tech-kern a little.

Use __read_mostly on these variables, to reduce the probability of false
sharing.

branches: 1.121.2;
Use the generic i82489_writereg instead of lapic_tpr, for consistency.

branches: 1.120.2;
KNF. Remove extra spaces. No functional change.

Extend SMEP support to i386 (does not require PAE).

Implement amd64 support for SMEP - Supervisor Mode Execution Protection.

Now, on CPUs that support this feature, if the kernel tries to execute
an instruction located in userland, the CPU will trigger a page fault.

Tested on amd64 (Intel Core i5).

Retrieve cpuid7 (Structured Extended Features) into ci_feat_val.

Don't disable/re-enable interrupts if they are already disabled.
Addresses PR 48196.

This commit was improved and approved by christos@

OOOOPS. Revert previous.

Workaround for "lapic_set_lvt: bad pin value %d" panic on some (broken?) BIOS
system. Don't panic when a local APIC's interrput input pin number (LINTx) > 1.
Instead, print warning message and continue. The default is pin 1.
Same as Linux (and perhaps FreeBSD). Tested with Shuttle DS57U.

PR/49104: Jarle Greipsland: Don't touch cr4 in cpus that don't have it.
XXX: pullup-7

Modify around cpu_identify() to not to break the dmesg of cpus with AB_VERBOSE
or AB_DEBUG.

branches: 1.111.2; 1.111.4;
Match lapic conditionals from the primary CPU.

branches: 1.110.2;
Add support for saving the AVX-256 ymm registers during FPU context switches.
Add support for the forthcoming AVX-512 registers.
Code compiled with -mavx seems to work, but I've not tested context
switches with live ymm registers.
There is a small cost on fork/exec (a larger area is copied/zerod),
but I don't think the ymm registers are read/written unless they
have been used.
The code use XSAVE on all cpus, I'm not brave enough to enable XSAVEOPT.

Add explicit #include <x86/fpu.h> instead of relying on pcb.h including it.

Remove support for 'external' floating point units and the MS-DOS
compatible method of handling floating point exceptions.
Make kernel support for teh fpu non-optional (486SX should still work).
Only 386 cpus support external fpu, and i386 support was removed years ago.
This means that the npx code no longer uses port 0xf0 or interupt 13.
All the "npx at isa" lines go from the configs, arch/i386/isa/npx.c
is now mandatory for all i386 kernels.
I've renamed npxinit() to fpuinit() and npxinit_cpu() to fpuinit_cpu()
to match the very similar amd64 functions.
The fpu of the boot cpu is now initialised by a direct call from
cpu_configure(), this enables FP emulation for a 486SX.
(for amd64 the cr0 values are set in locore.S and similar).
This fixes a long-standing bug in linux_setregs() - which did not
save the fpu regsiters if they were active.
I've test booted a single cpu i386 kernel (using anita).
amd64 builds - none of teh changes should affect it.
The i386 XEN kernels build, but I'm not sure where they set cr0, and
it might have got lost!

revert fpu/pcu changes until we figure out what's wrong; they cause random
freezes

Modify some macros and add some new macros for CPU family and model
to reduce code duplication and to avoid bug.

CPUID_TO_STEPPING(cpuid) (not changed)

CPUID_TO_FAMILY(cpuid) (new)
CPUID_TO_MODEL(cpuid) (new)

Return the display family and the display model.
The macro names are the same as FreeBSD.

CPUID_TO_BASEFAMILY(cpuid) (The old name was CPUID2FAMILY)
CPUID_TO_BASEMODEL(cpuid) (The old name was CPUID2MODEL)

Only for the base field.

CPUID_TO_EXTFAMILY(cpuid) (The old name was CPUID2EXTFAMILY)
CPUID_TO_EXTMODEL(cpuid) (The old name was CPUID2EXTMODEL)

Only for the extended field.

See http://mail-index.netbsd.org/port-amd64/2013/11/12/msg001978.html

Revert previos. I accidentally committed a debug code. Sorry.

Fix a bug in last commit. Check correct variable.

Use the MI "pcu" framework for bookkeeping of npx/fpu states on x86.
This reduces the amount of MD code enormously, and makes it easier
to implement support for newer CPU features which require more fpu
state, or for fpu usage by the kernel.
For access to FPU state across CPUs, an xcall kthread is used now
rather than a dedicated IPI.
No user visible changes intended.

branches: 1.102.2;
With recent introduction of conditionals for the various MP options, we
broke the build for x86 systems that have MULTIPROCESSOR but which do not
include MPBIOS. So let's try to untangle things just a bit. Presented
on current-users (and referenced on source-changes-d) without any comment.

XXX We really should find a better method to select kernel options; #ifdef
spaghetti is rather sub-optimal.

#ifdef - #endif-ed. NMCA, NISA, NNPX, NIOAPIC, LAPIC, MPBIOS and MULTIPROCESSOR.

branches: 1.100.2;
in cpu_boot_secondary_processors(), wait until all the other CPUs
have registered themselves in kcpuset_running before returning.
recent changes to the TLB invalidation xcall code assume that
any CPU which will receive a broadcast IPI is registered in
kcpuset_running, so ensure that is true by waiting here.

cpu_load_pmap: disable interrupts. add a comment to explain why. PR/44995

- Convert x86 MD code, mainly pmap(9) e.g. TLB shootdown code, to use
kcpuset(9) and thus replace hardcoded CPU bitmasks. This removes the
limitation of maximum CPUs.

- Support up to 256 CPUs on amd64 architecture by default.

Bug fixes, improvements, completion of Xen part and testing on 64-core
AMD Opteron(tm) Processor 6282 SE (also, as Xen HVM domU with 128 CPUs)
by Manuel Bouyer.

Apply patch proposed in PR port-xen/45975 (this does not solve the exact
problem reported here but is part of the solution):
xen_kpm_sync() is not working as expected,
leading to races between CPUs.
1 the check (xpq_cpu != &x86_curcpu) is always false because we
have different x86_curcpu symbols with different addresses in the kernel.
Fortunably, all addresses dissaemble to the same code.
Because of this we always use the code intended for bootstrap, which doesn't
use cross-calls or lock.

2 once 1 above is fixed, xen_kpm_sync() will use xcalls to sync other CPUs,
which cause it to sleep and pmap.c doesn't like that. It triggers this
KASSERT() in pmap_unmap_ptes():
KASSERT(pmap->pm_ncsw == curlwp->l_ncsw);
3 pmap->pm_cpus is not safe for the purpose of xen_kpm_sync(), which
needs to know on which CPU a pmap is loaded *now*:
pmap->pm_cpus is cleared before cpu_load_pmap() is called to switch
to a new pmap, leaving a window where a pmap is still in a CPU's
ci_kpm_pdir but not in pm_cpus. As a virtual CPU may be preempted
by the hypervisor at any time, it can be large enough to let another
CPU free the PTP and reuse it as a normal page.

To fix 2), avoid cross-calls and IPIs completely, and instead
use a mutex to update all CPU's ci_kpm_pdir from the local CPU.
It's safe because we just need to update the table page, a tlbflush IPI will
happen later. As a side effect, we don't need a different code for bootstrap,
fixing 1). The mutex added to struct cpu needs a small headers reorganisation.

to fix 3), introduce a pm_xen_ptp_cpus which is updated from
cpu_pmap_load(), whith the ci_kpm_mtx mutex held. Checking it with
ci_kpm_mtx held will avoid overwriting the wrong pmap's ci_kpm_pdir.

While there I removed the unused pmap_is_active() function;
and added some more details to DIAGNOSTIC panics.

branches: 1.96.2; 1.96.6; 1.96.8;
As cpu_shutdown() is a wrapper to cpu_suspend(), modify slightly to prevent
setting low frequencies for active non-bootstrap processors during shutdown.

add a "vm" device class for cpufeaturebus

remove a check against uvmexp.ncolors that is done inside uvm_page_recolor()
already anyway.

Call cpufreq_suspend(9) and cpufreq_resume(9) during suspend/resume.

Unbreak the build. (conflicting types in function declaration and definition)

Thanks riz@

Hide the MD details of specific IPIs behind semantically pleasing functions. This cleans up a couple of #ifdef XEN/#endif pairs

Don't shutdown the boostrap processor (BSP) because we may have to run
BIOS methods on it. For example, ACPI requires that we execute the code
for changing sleep state on the BSP.

This may help the problem where folks' machines would hang instead of
powering off when they entered ACPI sleep state 5.

XXX If the BSP is already shut down, we should start it back up.

Add small comment.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.87.2;
Use config_defer(9) for cpu_rescan() in cpu_attach().
Also mark few local functions as static.

Fix autoconf(9) of cpufeaturebus.

Move VIA_C7TEMP to the cpufeaturebus.

Move PowerNow! to the cpufeaturebus.

Move ENHANCED_SPEEDSTEP, or henceforth est(4), to the cpufeaturebus.

Modularize coretemp(4). Ok jmcneill@.

modularize VIA PadLock support
- retire options VIA_PADLOCK, replace with 'padlock0 at cpu0'
- driver supports attach & detach
- support building as a module

Some CPU have cpu counter (CPUID_TSC is there) but don't handle the
rdmsr instruction (CPUID_MSR is not there).
Introduce a cpu_counter_serializing() function to remplace rdmsr(MSR_TSC)
calls, which does a rdmsr(MSR_TSC) if available and cpu_counter() otherwise.
This makes the cpu counter useable on vortex86 CPUs.
OK ad@

branches: 1.79.2; 1.79.4;
Use pmf_device_register1(9) and add cpu_shutdown(), which calls cpu_suspend().

Machine dependent code is considered as part of UVM. Include
internal API header.

Revert all previous changes that were made naively believing that the
existing CPU power management implementations could peacefully coexist with
the acpicpu(4) driver. The following options can not be used with acpicpu(4):
ENHANCED_SPEEDSTEP, INTEL_ONDEMAND_CLOCKMOD, POWERNOW_K7, and POWERNOW_K8.

Revert the previous changes to EST. The used hack had an obvious flaw:
the acpicpu(4) driver should attach even if the existing frequency management
code fails to attach, mainly because ACPI is the only proper way to deal
with EST on new Intel system.

Use a more drastic hack to deal with this: when acpicpu(4) attachs, it tears
down any existing sysctl(8) controls and installs identical ones in place.
Upon detachment, the initialization function of the existing EST is called.

Move the sysctl function pointers used by acpicpu(4) to x86/cpu.c.
Rename these so that the same pointers may be used in other parts.

Store the MADT-derived CPU ID to <x86/cpu.h>. This is required to properly
match the ACPI processor object ID with the ID available in the APIC table.

Welcome PAE inside i386 current.

This patch is inspired by work previously done by Jeremy Morse, ported by me
to -current, merged with the work previously done for port-xen, together with
additionals fixes and improvements.

PAE option is disabled by default in GENERIC (but will be enabled in ALL in
the next few days).

In quick, PAE switches the CPU to a mode where physical addresses become
36 bits (64 GiB). Virtual address space remains at 32 bits (4 GiB). To cope
with the increased size of the physical address, they are manipulated as
64 bits variables by kernel and MMU.

When supported by the CPU, it also allows the use of the NX/XD bit that
provides no-execution right enforcement on a per physical page basis.

Notes:

- reworked locore.S

- introduce cpu_load_pmap(), used to switch pmap for the curcpu. Due to the
different handling of pmap mappings with PAE vs !PAE, Xen vs native, details
are hidden within this function. This helps calling it from assembly,
as some features, like BIOS calls, switch to pmap_kernel before mapping
trampoline code in low memory.

- some changes in bioscall and kvm86_call, to reflect the above.

- the L3 is "pinned" per-CPU, and is only manipulated by a
reduced set of functions within pmap. To track the L3, I added two
elements to struct cpu_info, namely ci_l3_pdirpa (PA of the L3), and
ci_l3_pdir (the L3 VA). Rest of the code considers that it runs "just
like" a normal i386, except that the L2 is 4 pages long (PTP_LEVELS is
still 2).

- similar to the ci_pae_l3_pdir{,pa} variables, amd64's xen_current_user_pgd
becomes an element of cpu_info (slowly paving the way for MP world).

- bootinfo_source struct declaration is modified, to cope with paddr_t size
change with PAE (it is not correct to assume that bs_addr is a paddr_t when
compiled with PAE - it should remain 32 bits). bs_addrs is now a
void * array (in bootloader's code under i386/stand/, the bs_addrs
is a physaddr_t, which is an unsigned long).

- fixes in multiboot code (same reason as bootinfo): paddr_t size
change. I used Elf32_* types, use RELOC() where necessary, and move the
memcpy() functions out of the if/else if (I do not expect sym and str tables
to overlap with ELF).

- 64 bits atomic functions for pmap

- all pmap_pdirpa access are now done through the pmap_pdirpa macro. It
hides the L3/L2 stuff from PAE, as well as the pm_pdirpa change in
struct pmap (it now becomes a PDP_SIZE array, with or without PAE).

- manipulation of recursive mappings ( PDIR_SLOT_{,A}PTEs ) is done via
loops on PDP_SIZE.

See also http://mail-index.netbsd.org/port-i386/2010/07/17/msg002062.html

No objection raised on port-i386@ and port-xen@R for about a week.

XXX kvm(3) will be fixed in another patch to properly handle both PAE and !PAE
kernel dumps (VA => PA macros are slightly different, and need proper 64 bits
PA support in kvm_i386).

XXX Mixing PAE and !PAE modules may lead to unwanted/unexpected results. This
cannot be solved easily, and needs lots of thinking before being declared
safe (paddr_t/bus_addr_t size handling, PD/PT macros abstractions).

cpu_attach: use kmem_zalloc instead of memset.

Turn PMAP_NOCACHE into MI flag.
Add MI flags PMAP_WRITE_COMBINE, PMAP_WRITE_BACK, PMAP_NOCACHE_OVR.
Update pmap(9) manpage.

hppa: Remove MD PMAP_NOCACHE flag as it exists as MI flag
mips: Rename MD PMAP_NOCACHE to PGC_NOCACHE.

x86: Implement new MI flags using Page-Attribute Tables.
x86: Implement BUS_SPACE_MAP_PREFETCHABLE.

Patch presented on tech-kern@:
http://mail-index.netbsd.org/tech-kern/2010/06/30/msg008458.html

No comments on this last version.

This patch fixes the NX regression issue observed on amd64 kernels, where
per-page execution right was disabled (therefore leading to the inability
of the kernel to detect fraudulent use of memory mappings marked as not
being executable).

- replace cpu_feature and ci_feature_flags variables by cpu_feature and
ci_feat_val arrays. This makes it cleaner and brings kernel code closer
to the design of cpuctl(8). A warning will be raised for each CPU that
does not expose the same features as the Boot Processor (BP).

- the blacklist of CPU features is now a macro defined in the
specialreg.h header, instead of hardcoding it inside MD initialization
code; fix comments.

- replace checks against CPUID_TSC with the cpu_hascounter() function.

- clean up the code in init_x86_64(), as cpu_feature variables are set
inside cpu_probe().

- use cpu_init_msrs() for i386. It will be eventually used later for NX
feature under i386 PAE kernels.

- remove code that checks for CPUID_NOX in amd64 mptramp.S, this is already
performed by cpu_hatch() through cpu_init_msrs().

- remove cpu_signature and feature_flags members from struct mpbios_proc
(they were never used).

This patch was tested with i386 MONOLITHIC, XEN3PAE_DOM0 and XEN3_DOM0 under
a native i386 host, and amd64 GENERIC, XEN3_DOM0 via QEMU virtual machines.

XXX Should kernel rev be bumped?

XXX A similar patch should be pulled-up for NetBSD-5, hopefully tomorrow.

branches: 1.69.2;
A pointer typedef entails trading too much flexibility to declare const
and non-const types, and the kernel uses both const and non-const
PMF qualifiers and device suspensors, so change the pmf_qual_t and
device_suspensor_t typedefs from "pointers to const" to non-pointer,
non-const types.

Wrap a comment; add a space after a comma to another (align with next line)

Use roundup2() instead of hardcoding the operation.

branches: 1.66.2;
Expand PMF_FN_* macros.

Use lwp_getpcb() on x86 MD code, clean from struct user usage.

Add a flags argument to pmap_kenter_pa(9).
Patch showed on tech-kern@ http://mail-index.netbsd.org/tech-kern/2009/11/04/msg006434.html
No objections.

Rearrange TSC inter-CPU synchronization code so that the gory details
are dealt with in x86/tsc.c and callers don't have to care that much.
Also add some comments and make some variables static.
approved by ad (a while ago)

branches: 1.62.2;
Make i386 config without NPX work, problem reported and fix tested by
Wojciech Galazka.
While there change a __i386__ to i386 for consistency.

move from malloc to kmem

PR kern/40213 my i386 machine can't boot because of tsc

- Patch in atomic_cas_64() twice. The first patch is early and makes it
the MP-atomic version available if we have cmpxchg8b. The second patch
strips the lock prefix if ncpu==1.

- Fix the i486 atomic_cas_64() to not unconditionally enable interrupts.

Link cpus in the order they are attaching and not in inverse order.

- Avoid the race with CPU online/offline state changes, when setting the
affinity (cpu_lock protects these operations now).
- Disallow setting of state of CPU to to offline, if there are bound LWPs,
which have no CPU to migrate.
- Disallow setting of affinity for the LWP(s), if all CPUs in the dynamic
CPU-set are offline.
- sched_setaffinity: fix invalid check of kcpuset_isset().
- Rename cpu_setonline() to cpu_setstate().

Should fix PR/39349.

branches: 1.57.2; 1.57.4;
- Rename cpu_lookup_byindex() to cpu_lookup(). The hardware ID isn't of
interest to MI code. No functional change.
- Change /dev/cpu to operate on cpu index, not hardware ID. Now cpuctl
shouldn't print confused output.

branches: 1.56.4;
If we boot with RB_MD1, register a NULL pmf handler for APs so we can
still suspend.

- Don't bother using sse to copy/zero pages on demand. It turns out not
to be worth it.
- If the machine has sse, re-enable zeroing pages in the idle loop and
use the sse instructions so that we don't blow out the cache.

Remove X86_MAXPROCS. There is still a 32-cpu limit, but it's now using
the MI constants.

Do the errata patchup after identifying the CPU, to avoid badly formatted
output.

verbose -> debug for # page colours

- cpu_attach: ensure that the boot processor is set up before trying to
initialize APs. We need the lapic set up and the boot processor may
not be attached first.

- mp_cpu_start: write back and invalidate the data cache before starting the
init IPI sequence. If a buggy BIOS has left the AP with cache disabled,
it might not be able to participate in the cache coherency protocol.

Be more conservative during AP startup. Don't let the AP access the lapic
or do any setup until the boot processor has finished the init sequence,
and add a few more delays.

- Make cpu_number() return MI index, otherwise the pmap cannot work on
systems with lapic IDs > X86_MAXPROCS.
- Kill cpu_info[] array and use MI cpu_lookup_byindex().

Don't crash if more than 32 cpus. Hopefully the boot processor will be
within the first 32 attached.

- Complain if unable to reset the lapic ID.
- Minor clean up.

cpu_hatch: hack around problem with multiple CPUs spinning in i8254_delay.

- Decouple the APIC ID from cpu_info[].
- Probe TSC frequency on each AP when hatching.

MP + apics are needed now so kill the #ifdefs

Don't reload LDTR unless a new value, which only happens for USER_LDT.

Stop using APIC IDs to identify CPUs for software purposes. Allows for
APIC IDs beyond 31, which has been possible for some time now.

Share cpu.h between the x86 ports.

Simplify x86 identcpu code, and share between i386/amd64.

If the boot processor's lapic has the wrong ID, reset it.

Improve x86 tsc handling:

- Ditch the cross-CPU calibration stuff. It didn't work properly, and it's
near impossible to synchronize the CPUs in a running system, because bus
traffic will interfere with any calibration attempt, messing up the
timings.

- Only enable the TSC on CPUs where we are sure it does not drift. If we are
On a known good CPU, give the TSC high timecounter quality, making it the
default.

- When booting CPUs, detect TSC skew and account for it. Most Intel MP
systems have synchronized counters, but that need not be true if the
system has a complicated bus structure. As far as I know, AMD systems
do not have synchronized TSCs and so we need to handle skew.

- While an AP is waiting to be set running, try and make the TSC drift by
entering a reduced power state. If we detect drift, ensure that the TSC
does not get a high timecounter quality. This should not happen and is
only for safety.

- Make cpu_counter() stuff LKM safe.

Make cpu_idle a macro calling a function pointer on x86.
Select the Xen idle routine for Xen, mwait if supported by the CPU and
it is not AMD and halt otherwise. As reported by Christoph Egger,
AMD Barcelona keeps the CPU in C0 state with MWAIT, contrary to HLT,
which uses C1 and therefore much less power.

branches: 1.36.2;
Minor correction to previous.

Recognise two new boot flags:

-1 disable MP
-2 disable ACPI

Remove clause 3 and 4 from TNF licenses

branches: 1.33.2;
Gracefully handle a condition where apic id >= X86_MAXPROCS rather than
panicing.

Commit a quick workaround for the not-power-of-two cache colors problem
pointed out by Simon (Simon's option #3): use the greatest power of two
which is a divisor of the desired number of cache colors.

This code might want to stay even after the cache probing code is fixed.

branches: 1.31.2;
g/c unused ioapic_bsp_id.
Per discussion with bouyer.

wrap long line. Requested and OK by simonb.

cpu_debug_dump: s/curproc/curlwp/ in a message.

use aprint_*_dev.
OK simonb

- use aprint_*_dev and device_xname
- use POSIX integer types

use device accessors and other misc cleanups

Add more error reporting to AP startup.

If MPDEBUG and waiting for the CPU to start, dump cpu_trace[] as it changes.

Split device_t/softc.

Use PMF_FN_ARGS, PMF_FN_PROTO.

branches: 1.21.2; 1.21.6;
Align cc_microtime and struct cpu_info to 64b.

pmf: Naively track online/offline state of APs during suspend/resume.

Initialise the Local Vector Table of the primary LAPIC directly after
enabling it. Explicitly initialise LINT0 as ExtInt and LINT1 as NMI,
the platform default. Mask the NMIs on the application processors and
mask the ExtInt if a IOAPIC was found.

With this patch, "disable ioapic" is supposed to work and it will allow
enabling the local APIC on all systems that have one to gain e.g. the
better clock interrupt.

Introduce optional cpu_offline_md to execute MD actions at the end of
cpu_offline. Use this on amd64/i386 to force a FPU save. As this was
triggered by npxsave_cpu/fpusave_cpu not working for a different CPU,
remove the cpu_info argument and adjust npxsave_*/fpusave_* to use bool
for the save.

OK ad@

Ensure that non-primary CPUs save the FPU state on suspend.

- make amd64 use per-cpu tss.
- fix iopl syscall for amd64+xen.

i386:
- make tss per-cpu. this considerably speeds up context switch for,
at least, pentium4, where ltr instruction seems very slow.
i386, xen:
- kill cpu_maxproc.
kvm86:
- adapt to per-cpu tss.
- cleanup and simplify.
- move kvm86_mp_lock to more meaningful place.
- disable preemption during a call.

Add new IPI for saving CPU state explicitly, share high-level part of
ACPI wakeup code and teach it how to start the APs again. As a side
effect the CPU_START interface allows choosing between different
bootstrap codes more easily now.

For now, remove the attempts to shutdown other CPUs and bring them back
online. It runs into issues in the pmap code and will handled
differently. This allows sysctl -w machdep.sleep_state=3 to at least
recover into a working system again.

branches: 1.12.2;
Merge jmcneill-pm branch.

branches: 1.11.2;
- Fix the locking around the i8254. Values for the TSC clock and lapic
delay function were wildly inaccurate due to multiple CPUs competing
in DELAY() during calibration, confusing the clock chip.
- Use i8254_delay() explictly in a few more places.

branches: 1.10.2;
Back out part of patch that got merged accidentally.

Use atomics to adjust ci_flags.

cpu_hatch: change lapic initialization order.

In cpu_hatch(), recompute ci_tsc_freq instead of using the boot CPU's value.

- cpu_vendor was both an int and char[] on amd64 - fix it.
- Run the errata check/patch on all CPUs, not just the boot processor.

- When computing the TSC frequency, call i8254_delay() and not DELAY().
- Use atomics to adjust the pmap reference count, instead of taking locks.
- Implement I386_{SET,GET}_{FS,GS}BASE, allowing %fs and %gs to be used
as per-thread registers. This is compatible with FreeBSD.
- Run patches after we have attached CPUs, since we then know if the
system is uniprocessor or not. Eliminates a lot of #ifdef MULTIPROCESSOR
and makes running MP kernels on UP systems cheaper.
- Patch out many of the 'lock' prefixes to nops if uniprocessor.
- Do a wbinvd after patching to ensure that the trace/instruction cache
is up to date.

branches: 1.4.2; 1.4.4;
merge yamt-x86pmap branch.

- reduce differences between amd64 and i386. notably, share pmap.c
between them. it makes several i386 pmap improvements available to
amd64, including tlb shootdown reduction and bug fixes from Stephan Uphoff.
- implement deferred pmap switching for amd64.
- remove LARGEPAGES option. always use large pages if available.
also, make it work on amd64.

branches: 1.3.2;
x86 changes for pcc and LKMs.

- Replace most inline assembly with proper functions. As a side effect
this reduces the size of amd64 GENERIC by about 120kB, and i386 by a
smaller amount. Nearly all of the inlines did something slow, or something
that does not need to be fast.
- Make curcpu() and curlwp functions proper, unless __GNUC__ && _KERNEL.
In that case make them inlines. Makes curlwp LKM and preemption safe.
- Make bus_space and bus_dma more LKM friendly.
- Share a few more files between the ports.
- Other minor changes.

branches: 1.2.2; 1.2.4; 1.2.6; 1.2.8;
Merge most x86 changes from the vmlocking branch, except the threaded soft
interrupt stuff. This is mostly comprised of changes to the pmap modules to
work on multiprocessor systems without kernel_lock, and changes to speed up
tlb shootdowns.

branches: 1.1.2;
file cpu.c was initially added on branch vmlocking.

x86: Sprinkle extensive commentary about %fs/%gs initialization.

Plus some other side quests like the three-stage GDT metamorphosis
lifecycle.

No functional change intended.

x86: Call fpuinit_mxcsr_mask only once.

No need to call it again and again on the secondary CPUs to compute
what should be the same mxcsr mask. (If it's not, we have deeper
problems!)

x86: Assert kpreempt_disabled() in cpu_load_pmap.

No functional change intended. Just makes it easier to audit
curcpu() usage.

x86: Support EFI runtime services.

This creates a special pmap, efi_runtime_pmap, which avoids setting
PTE_U but allows mappings to lie in what would normally be user VM --
this way we don't fall afoul of SMAP/SMEP when executing EFI runtime
services from CPL 0. SVS does not apply to the EFI runtime pmap.

The mechanism is intended to work with either physical addressing or
virtual addressing; currently the bootloader does physical addressing
but in principle it could be modified to do virtual addressing
instead, if it allocated virtual pages, assigned them in the memory
map, and issued RT->SetVirtualAddressMap.

Not sure pmap_activate_sync and pmap_deactivate_sync are correct,
need more review from an x86 wizard.

If this causes fallout, it can be disabled temporarily without
reverting anything by just making efi_runtime_init return immediately
without doing anything, or by removing options EFI_RUNTIME.

amd64-only for now pending type fixes and testing on i386.

x86: Split most of pmap.h into pmap_private.h or vmparam.h.

This way pmap.h only contains the MD definition of the MI pmap(9)
API, which loads of things in the kernel rely on, so changing x86
pmap internals no longer requires recompiling the entire kernel every
time.

Callers needing these internals must now use machine/pmap_private.h.
Note: This is not x86/pmap_private.h because it contains three parts:

1. CPU-specific (different for i386/amd64) definitions used by...

2. common definitions, including Xenisms like xpmap_ptetomach,
further used by...

3. more CPU-specific inlines for pmap_pte_* operations

So {amd64,i386}/pmap_private.h defines 1, includes x86/pmap_private.h
for 2, and then defines 3. Maybe we should split that out into a new
pmap_pte.h to reduce this trouble.

No functional change intended, other than that some .c files must
include machine/pmap_private.h when previously uvm/uvm_pmap.h
polluted the namespace with pmap internals.

Note: This migrates part of i386/pmap.h into i386/vmparam.h --
specifically the parts that are needed for several constants defined
in vmparam.h:

VM_MAXUSER_ADDRESS
VM_MAX_ADDRESS
VM_MAX_KERNEL_ADDRESS
VM_MIN_KERNEL_ADDRESS

Since i386 needs PDP_SIZE in vmparam.h, I added it there on amd64
too, just to keep things parallel.

Split TSC calibtration into many small steps and disable interrupts
for each step. Also add debug messages.

x86, arm: Allow fpu_kern_enter/leave while cold.

Normally these are forbidden above IPL_VM, so that FPU usage doesn't
block IPL_SCHED or IPL_HIGH interrupts. But while cold, e.g. during
builtin module initialization at boot, all interrupts are blocked
anyway so it's a moot point.

Also initialize x86 cpu_info_primary.ci_kfpu_spl to -1 so we don't
trip over an assertion about it while cold -- the assertion is meant
to detect reentrance into fpu_kern_enter/leave, which is prohibited.

Also initialize cpu0's ci_kfpu_spl.

KNF. No functional change.

Merge thorpej-cfargs2.

branches: 1.200.8;
Merge thorpej-cfargs branch:

Simplify and make extensible the config_search() / config_found() /
config_attach() interfaces: rather than having different variants for
which arguments you want pass along, just have a single call that
takes a variadic list of tag-value arguments.

Adjust all call sites:
- Simplify wherever possible; don't pass along arguments that aren't
actually needed.
- Don't be explicit about what interface attribute is attaching if
the device only has one. (More simplification.)
- Add a config_probe() function to be used in indirect configuiration
situations, making is visibly easier to see when indirect config is
in play, and allowing for future change in semantics. (As of now,
this is just a wrapper around config_match(), but that is an
implementation detail.)

Remove unnecessary or redundant interface attributes where they're not
needed.

There are currently 5 "cfargs" defined:
- CFARG_SUBMATCH (submatch function for direct config)
- CFARG_SEARCH (search function for indirect config)
- CFARG_IATTR (interface attribte)
- CFARG_LOCATORS (locators array)
- CFARG_DEVHANDLE (devhandle_t - wraps OFW, ACPI, etc. handles)

...and a sentinel value CFARG_EOL.

Add some extra sanity checking to ensure that interface attributes
aren't ambiguous.

Use CFARG_DEVHANDLE in MI FDT, OFW, and ACPI code, and macppc and shark
ports to associate those device handles with device_t instance. This
will trickle trough to more places over time (need back-end for pre-OFW
Sun OBP; any others?).

branches: 1.199.4;
Don't do extra work finding the power of 2 for values we are not going to
use. Explain that cpu_hatch has not been called yet, so no cpu_probe either
so the cache info is 0 for AP's.

move lcall sniffer to x86_machdep since xen/pv has its own cpu.c

PR/55547: Dan Plassche: Fix BSD/OS binary emulation.
Centralize lcall sniffer and recognize the BSD/OS flavor.

Use CPU_IS_PRIMARY macro in cpu_stop(), cpu_resume(), and cpu_get_tsc_freq()
on x86.

OK kamil@

Introduce per-cpu IDTs

This is realized by following modifications:
- Add IDT pages and its allocation maps for each cpu in "struct cpu_info"
- Load per-cpu IDTs at cpu_init_idt(struct cpu_info*)
- Copy the IDT entries for cpu0 to other CPUs at attach
- These are, for example, exceptions, db, system calls, etc.

And, added a kernel option named PCPU_IDT to enable the feature.

Serialize rdtsc using with lfence, mfence or cpuid to read TSC more precisely.

x86/x86/tsc.c rev. 1.67 reduced cache problem and got big improvement, but it
still has room. I measured the effect of lfence, mfence, cpuid and rdtscp.
The impact to TSC skew and/or drift is:

AMD: mfence > rdtscp > cpuid > lfence-serialize > lfence = nomodify
Intel: lfence > rdtscp > cpuid > nomodify

So, mfence is the best on AMD and lfence is the best on Intel. If it has no
SSE2, we can use cpuid.

NOTE:
- An AMD's document says DE_CFG_LFENCE_SERIALIZE bit can be used for
serializing, but it's not so good.
- On Intel i386(not amd64), it seems the improvement is very little.
- rdtscp instruct can be used as serializing instruction + rdtsc, but
it's not good as [lm]fence. Both Intel and AMD's document say that
the latency of rdtscp is bigger than rdtsc, so I suspect the difference
of the result comes from it.

g/c vm_page_zero_enable

- Recalibrate the APIC timer using the TSC, once the TSC has in turn been
recalibrated using the HPET. This gets the clock interrupt firing more
closely to HZ.

- Undo change with recent Xen merge and go back to starting the clocks in
initclocks() on the boot CPU, and in cpu_hatch() on secondary CPUs.

- On reflection don't use HPET delay any more, it works very well but means
going over the bus. It's enough to use HPET to calibrate the TSC and
APIC.

Tested on amd64 native, xen and xen PVH.

Don't use TSC freq value from CPUID if calibration works.

- When it's the first call of cpu_get_tsc_freq() the HPET is not initialized,
so try to use CPUID to get TSC freq.
- If it's the 2nd call, don't use CPUID. Instead, print the difference
between the calibrated value and CPUID's value if the verbose mode is set.

Fix the TSC timecounter (on the systems I have access to):

- Make the early i8254-based calculation of frequency a bit more accurate.

- Keep track of how far the HPET & TSC advance between HPET attach and
secondary CPU boot, and use to compute an accurate value before attaching
the timecounter. Initial idea from joerg@.

- When determining skew and drift between CPUs, make each measurement 1000
times and pick the lowest observed value. Increase the error threshold to
1000 clock cycles.

- Use the frequency computed on the boot CPU for secondary CPUs too.

- Remove cpu_counter_serializing().

Introduce Xen PVH support in GENERIC.
This is compiled in with
options XENPVHVM
x86 changes:
- add Xen section and xen pvh entry points to locore.S. Set vm_guest
to VM_GUEST_XENPVH in this entry point.
Most of the boot procedure (especially page table setup and switch to
paged mode) is shared with native.
- change some x86_delay() to delay_func(), which points to x86_delay() for
native/HVM, and xen_delay() for PVH

Xen changes:
- remove Xen bits from init_x86_64_ksyms() and init386_ksyms()
and move to xen_init_ksyms(), used for both PV and PVH
- set ISA no-legacy-devices property for PVH
- factor out code from Xen's cpu_bootconf() to xen_bootconf()
in xen_machdep.c
- set up a specific pvh_consinit() which starts with printk()
(which uses a simple hypercall that is available early) and switch to
xencons when we can use pmap_kenter_pa().

Back out HPET delay & TSC changes to rule them out as the cause for recent
hangs during boot etc.

Merge the bouyer-xenpvh branch, bringing in Xen PV drivers support under HVM
guests in GENERIC.
Xen support can be disabled at runtime with
boot -c
disable hypervisor

- Install HPET based DELAY() before going multiuser then recalibrate the TSC.
Idea from joerg@.

- Take overhead into account when computing CPU frequency.

- Don't flush cache before computing TSC skew.

Get TSC frequency from CPUID 0x15 and/or x16 for newer Intel processors.

- If the max CPUID leaf is >= 0x15, take TSC value from CPUID. Some processors
can take TSC/core crystal clock ratio but core crystal clock frequency
can't be taken. Intel SDM give us the values for some processors.
- It also required to change lapic_per_second to make LAPIC timer correctly.
- Add new file x86/x86/identcpu_subr.c to share common subroutines between
kernel and userland. Some code in x86/x86/identcpu.c and cpuctl/arch/i386.c
will be moved to this file in future.
- Add comment to clarify.

Whitespace fix. No functional change.

Revert, wrong branch

Skip cx8_spllower patch if we're running on any form of Xen PV,
we can't handle PV interrupts with a single atomic op here.
Enable x86_patch() for Xen too.

branches: 1.181.4;
If "application processors" were skipped/disabled at boot time (due to
RB_MD1 being set), don't try to examine the featurebus info, since it
was never retrieved. Addresses kern/54815

XXX pullup-9

Make "mach cpu" in ddb show the IPL for each cpu.

branches: 1.179.2;
Some more CPU topology stuff:

- Use cegger@'s ACPI SRAT parsing code to figure out NUMA node ID for each
CPU as it is attached.

- For scheduler experiments with SMT, flag CPUs with the lowest numbered SMT
IDs as "primaries", link back to the primaries from secondaries, and build
a circular list of CPUs in each package with identical SMT IDs.

- No need for package/core/smt/numa IDs to be anything other than a u_int.

Get a Hyper-V virtual processor id in cpu_hatch().

Currently, it is got in config_interrupts context.
However, since it is required when attaching a device,
it is got earlier than now.

Add a small API for in-kernel FPU operations.

fpu_kern_enter();
/* do FPU stuff */
fpu_kern_leave();

cpu_need_resched():

- Remove all code that should be MI, leaving the bare minimum under arch/.
- Make the required actions very explicit.
- Pass in LWP pointer for convenience.
- When a trap is required on another CPU, have the IPI set it locally.
- Expunge cpu_did_resched().

- On-demand zeroing pages with MOVNTI is crazy. It empties L1/L2/L3.
- Disable zeroing in the idle loop. That needs a cache-friendly strategy.

Result: 3 to 4% reduction in kernel build time on my test system.
Inspired by a discussion with Mateusz Guzik and David Maxwell.

Add Kernel Concurrency Sanitizer (kCSan) support. This sanitizer allows us
to detect race conditions at runtime. It is a variation of TSan that is
easy to implement and more suited to kernel internals, albeit theoretically
less precise than TSan's happens-before.

We do basically two things:

- On every KCSAN_NACCESSES (=2000) memory accesses, we create a cell
describing the access, and delay the calling CPU (10ms).

- On all memory accesses, we verify if the memory we're reading/writing
is referenced in a cell already.

The combination of the two means that, if for example cpu0 does a read that
is selected and cpu1 does a write at the same address, kCSan will fire,
because cpu1's write collides with cpu0's read cell.

The coverage of the instrumentation is the same as that of kASan. Also, the
code is organized in a way similar to kASan, so it is easy to add support
for more architectures than amd64. kCSan is compatible with KCOV.

Reviewed by Kamil.

Rewrite the FPU code on x86. This greatly simplifies the logic and removes
the dependency on IPL_HIGH. NVMM is updated accordingly. Posted on
port-amd64 a week ago.

Bump the kernel version to 9.99.16.

avoid misalignment in 32 bit kernels and "mach cpu".

branches: 1.171.2;
Add PCID support in SVS. This avoids TLB flushes during kernel<->user
transitions, which greatly reduces the performance penalty introduced by
SVS.

We use two ASIDs, 0 (kern) and 1 (user), and use invpcid to flush pages
in both ASIDs.

The read-only machdep.svs.pcid={0,1} sysctl is added, and indicates whether
SVS+PCID is in use.

Change the effect of SVS on the TLB. Keep CR4_PGE set when SVS is enabled,
but don't use PTE_G on the kernel PTEs in general.

Add PTE_G on only a few pages, that are already leaked to userland and do
not contain secrets.

This slightly improves syscall performance.

Remove 'ci_svs_kpdirpa', unused. While here fix a few comments here and
there, reduces a future diff.

Start replacing the x86 PTE bits.

Added Microsoft Hyper-V support. It ported from OpenBSD and FreeBSD.

graphical console is not work on Gen.2 VM yet. To use the serial console,
enter "consdev com,0x3f8,115200" on efiboot.

Welcome XENPVHVM mode.

It is UP only, has xbd(4) and xennet(4) as PV drivers.

The console is com0 at isa and the native portion is very
rudimentary AT architecture, so is probably suboptimal to
run without PV support.

Fix NLAPIC, NISA and NIOAPIC related conditional compile errors.

This will allow us to now compile an amd64 kernel without PCI.

No functional changes.

Hypothetically speaking, if one were to want to compile a

'no options MULTIPROCESSOR'

kernel, these files may trip up the build.

Fix them by moving around the #defines as originally intended.

No Functional Changes.

Stop panic()ing on a UP system.

The reason for the panic is that the cpu_attach() doesn't run to
completion because it thinks it's run past maxcpus (which in the case
of UP), is 1.

This is because on x86 at least, mi_cpu_attach() is called *before*
configure() (and thus the cpu_match()/cpu_attach() pair). Thus ncpu
has already been incremented by the time MD cpu_attach() is called.

Fix this.

Add a comment explaining an important rule. Just to better highlight that
this rule is actually not respected.

Rename min/max -> uimin/uimax for better honesty.

These functions are defined on unsigned int. The generic name
min/max should not silently truncate to 32 bits on 64-bit systems.
This is purely a name change -- no functional change intended.

HOWEVER! Some subsystems have

#define min(a, b) ((a) < (b) ? (a) : (b))
#define max(a, b) ((a) > (b) ? (a) : (b))

even though our standard name for that is MIN/MAX. Although these
may invite multiple evaluation bugs, these do _not_ cause integer
truncation.

To avoid `fixing' these cases, I first changed the name in libkern,
and then compile-tested every file where min/max occurred in order to
confirm that it failed -- and thus confirm that nothing shadowed
min/max -- before changing it.

I have left a handful of bootloaders that are too annoying to
compile-test, and some dead code:

cobalt ews4800mips hp300 hppa ia64 luna68k vax
acorn32/if_ie.c (not included in any kernels)
macppc/if_gm.c (superseded by gem(4))

It should be easy to fix the fallout once identified -- this way of
doing things fails safe, and the goal here, after all, is to _avoid_
silent integer truncations, not introduce them.

Maybe one day we can reintroduce min/max as type-generic things that
never silently truncate. But we should avoid doing that for a while,
so that existing code has a chance to be detected by the compiler for
conversion to uimin/uimax without changing the semantics until we can
properly audit it all. (Who knows, maybe in some cases integer
truncation is actually intended!)

Remove useless/outdated comments. No functional change.

Oh. Don't call svs_pdir_switch if SVS is disabled, that's not needed.

I was playing around with PMCs, and was wondering why some cache misses
were occurring in svs_pdir_switch while I had SVS disabled.

branches: 1.158.2;
Revert jdolecek's changes related to FXSAVE. They just didn't make any
sense and were trying to hide a real bug, which is, that there is for some
reason a wrong stack alignment that causes FXSAVE to fault in
fpuinit_mxcsr_mask. As seen in current-users@ yesterday, rdi % 16 = 8. And
as seen several months ago, as well.

The rest of the changes in XSAVE are wrong too, but I'll let him fix these
ones.

as a stop-gap, make fpuinit_mxcsr_mask() for native independant of
XSAVE as it should be, only xen case checks the flag now; need to
investigate further why exactly the fault happens for the xen
no-xsave case

pointed out by maxv

fix FPU initialization on Xen to allow e.g. AVX when supported by hardware;
only use XSAVE when the the CPUID OSXSAVE bit is set, as this seems to be
reliable indication

tested with Xen 4.2.6 DOM0/DOMU on Intel CPU, without and with no-xsave flag,
so should work also on those AMD CPUs, which have XSAVE disabled by default;
also tested with Xen DOM0 4.8.3

fixes PR kern/50332 by Torbjorn Granlund; sorry it took three years to address

XXX pullup netbsd-8

Call cpu_speculation_init on i386 too. We don't have IBRS for i386, but
we do have the AMD DIS_IND method.

Enable the SpectreV2 mitigation by default at boot time.

Move the SpectreV2 mitigation code into a dedicated spectre.c file. The
content of the file is taken from the end of cpu.c, and is copied as-is.

Remove #ifdef XEN (Xen has its own cpu.c), and add a comment.

Spectre V2 mitigation for certain families of AMD CPUs.

A new sysctl is added, machdep.spectreV2.mitigated, that controls whether
Spectre V2 is mitigated. For now it defaults to "false".

The code is written in such a way that there can be several methods. For
now only one method is supported, on AMD Families 10h, 12h and 16h, where
an MSR is available to disable branch prediction entirely.

Compile-tested on Intel, AMD will be tested soon.

Explain the TSC drift thing.

branches: 1.149.2;
Remove svs_pgg_update(). Instead of manually changing PG_G on each page,
we can disable the global-paging mechanism in %cr4 with CR4_PGE. Do that.

In addition, install CR4_PGE when SVS is disabled manually (via the
sysctl).

Now, doing "sysctl -w machdep.svs_enabled=0" restores the performance
completely, exactly as if SVS hadn't been enabled in the first place.

Add a dynamic detection for SVS.

The SVS_* macros are now compiled as skip-noopt. When the system boots, if
the cpu is from Intel, they are hotpatched to their real content.
Typically:

jmp 1f
int3
int3
int3
... int3 ...
1:

gets hotpatched to:

movq SVS_UTLS+UTLS_KPDIRPA,%rax
movq %rax,%cr3
movq CPUVAR(KRSP0),%rsp

These two chunks of code being of the exact same size. We put int3 (0xCC)
to make sure we never execute there.

In the non-SVS (ie non-Intel) case, all it costs is one jump. Given that
the SVS_* macros are small, this jump will likely leave us in the same
icache line, so it's pretty fast.

The syscall entry point is special, because there we use a scratch uint64_t
not in curcpu but in the UTLS page, and it's difficult to hotpatch this
properly. So instead of hotpatching we declare the entry point as an ASM
macro, and define two functions: syscall and syscall_svs, the latter being
the one used in the SVS case.

While here 'syscall' is optimized not to contain an SVS_ENTER - this way
we don't even need to do a jump on the non-SVS case.

When adding pages in the user page tables, make sure we don't have PG_G,
now that it's dynamic.

A read-only sysctl is added, machdep.svs_enabled, that tells whether the
kernel uses SVS or not.

More changes to come, svs_init() is not very clean.

Add SMAP support for i386.

Introduce a new svs_page_add function, which can be used to map in the user
space a VA from the kernel space.

Use it to replace the PDIR_SLOT_PCPU slot: at boot time each CPU creates
its own slot which maps only its own pcpu_entry plus the common area (IDT+
LDT).

This way, the pcpu areas of the remote CPUs are not mapped in userland.

Changing CR4 register may change cpuid values. For example, setting
CR4_OSXSAVE sets CPUID2_OSXSAVE. The CPUID2_OSXSAVE is in ci_feat_val[1],
so update it after changing CR4.

Add a new option, SVS (for Separate Virtual Space), that unmaps kernel
pages when running in userland. For now, only the PTE area is unmapped.

Sent on tech-kern@.

Use uvm_km_alloc instead of kmem_zalloc.

Add a __HAVE_PCPU_AREA option, enabled by default on native amd64 but not
Xen.

With this option, the CPU structures that must always be present in the
CPU's page tables are moved on L4 slot 384, which means address
0xffffc00000000000.

A new pcpu_area structure is defined. It contains shared structures (IDT,
LDT), and then an array of pcpu_entry structures, indexed by cpu_index(ci).
Theoretically the LDT should be in the array, but this will be done later.

During the boot procedure, cpu0 calls pmap_init_pcpu, which creates a
page tree that is able to map the pcpu_area structure entirely. cpu0 then
immediately maps the shared structures. Later, every CPU goes through
cpu_pcpuarea_init, which allocates physical pages and kenters the relevant
pcpu_entry to them. Finally, each pointer is replaced to point to pcpuarea.

The point of this change is to make sure that the structures that must
always be present in the page tables have their own L4 slot. Until now
their L4 slot was that of pmap_kernel, and making a distinction between
what must be mapped and what does not need to be was complicated.

Even in the non-speculative-bug case this change makes some sense: there
are several x86 instructions that leak the addresses of the CPU structures,
and putting these structures inside pmap_kernel actually offered a way to
compute the address of the kernel heap - which would have made ASLR on it
plainly useless, had we implemented that.

Note that, for now, pcpuarea does not contain rsp0.

Unfortunately this change adds many #ifdefs, and makes the code harder to
understand. There is also some duplication, but that will be solved later.

Recommit

http://mail-index.netbsd.org/source-changes/2017/11/08/msg089525.html

but use __INITIAL_MXCSR_MASK__ on Xen until someone figures out what's
wrong with the Xen fpu.

Revert http://mail-index.netbsd.org/source-changes/2017/11/08/msg089525.html,
it breaks Xen:
http://www-soc.lip6.fr/~bouyer/NetBSD-tests/xen/HEAD/amd64/201711082340Z_anita.txt

Call fpuinit_mxcsr_mask in cpu_init, after cr4 is initialized, but before
touching xcr0. Then use clts/stts instead of modifying cr0, and enable the
mxcsr_mask detection on Xen.

Have the cpu clear PSL_D automatically when entering the kernel via a
syscall. Then, don't clear PSL_D and PSL_AC in the syscall entry point,
they are now both cleared by the cpu (faster). However they still need to
be manually cleared in the interrupt/trap entry points.

Add support for SMAP on amd64.

PSL_AC is cleared from %rflags in each kernel entry point. In the copy
sections, a copy window is opened and the kernel can touch userland
pages. This window is closed when the kernel is done, either at the end
of the copy sections or in the fault-recover functions.

This implementation is not optimized yet, due to the fact that INTRENTRY
is a macro, and we can't hotpatch macros.

Sent on tech-kern@ a month or two ago, tested on a Kabylake.

Pack the useful variables at the end of the trampoline page; eliminates
a hard-coded dependency on KERNBASE. Note that I cannot test this change
on i386 right now, but it seems fine enough.

Remove TRAPLOG from i386. Nowadays there are better instrumentation tools,
in both software and hardware.

style, and move some i386-specific code into i386/

Localify. By the way, we should use a different stack for NMIs.

cpu_trace is no more, remove vestige of it that broke ALL kernel.

Further reduce the loop counter so that hatching completes before the
boot processor times us out.

Add a nice big XXX comment for why the counter is so low.

XXX Will need to pullup to NetBSD-7 branch

branches: 1.130.2;

And now do what 1.128 should have done, and put back the (now re-)used
variable that had earlier been deleted, when it's use was removed in
1.126, but wasn't restored in 1.127.

Revert previous. Removing unused variable declarations is only a good
idea when the variable is, in fact, unused.

Remove unused variabe (I reverted too much in previous commit!)

Partially revert previous. Rather than completely removing the loop
around calls to x86_pause(), just drastically reduce the repeat count.
It's still good to have some real delay here (among other things, for
letting the TSCs drift).

As discussed on IRC

Do not pause many times between testing if the CPU can go.

This only impacts QEMU as QEMU's implementation of pause is
significantly slower than its implementation of nop.

PR kern/51623: running qemu-x86_64 with -smp 4 - the additional
CPUs don't start.

x86: Add preliminary x2APIC support.

x2APIC is used only when x2APIC is enabled in BIOS/UEFI.
LAPIC ID is not supported above 256.

use CR8 instead of LAPIC Task Priority register on x86-64.

Instead of using a global array with per-cpu indexes, embed the tmp VAs
into cpu_info directly. This concerns only {i386, Xen-i386, Xen-amd64},
because amd64 already has a direct map that is way faster than that.

There are two major issues with the global array: maxcpus entries are
allocated while it is unlikely that common i386 machines have so many
cpus, and the base VA of these entries is not cache-line-aligned, which
mostly guarantees cache-line-thrashing each time the VAs are entered.

Now the number of tmp VAs allocated is proportionate to the number of CPUs
attached (which therefore reduces memory consumption), and the base is
properly aligned.

On my 3-core AMD, the number of DC_refills_L2 events triggered when
performing 5x10^6 calls to pmap_zero_page on two dedicated cores is on
average divided by two with this patch.

Discussed on tech-kern a little.

Use __read_mostly on these variables, to reduce the probability of false
sharing.

branches: 1.121.2;
Use the generic i82489_writereg instead of lapic_tpr, for consistency.

branches: 1.120.2;
KNF. Remove extra spaces. No functional change.

Extend SMEP support to i386 (does not require PAE).

Implement amd64 support for SMEP - Supervisor Mode Execution Protection.

Now, on CPUs that support this feature, if the kernel tries to execute
an instruction located in userland, the CPU will trigger a page fault.

Tested on amd64 (Intel Core i5).

Retrieve cpuid7 (Structured Extended Features) into ci_feat_val.

Don't disable/re-enable interrupts if they are already disabled.
Addresses PR 48196.

This commit was improved and approved by christos@

OOOOPS. Revert previous.

Workaround for "lapic_set_lvt: bad pin value %d" panic on some (broken?) BIOS
system. Don't panic when a local APIC's interrput input pin number (LINTx) > 1.
Instead, print warning message and continue. The default is pin 1.
Same as Linux (and perhaps FreeBSD). Tested with Shuttle DS57U.

PR/49104: Jarle Greipsland: Don't touch cr4 in cpus that don't have it.
XXX: pullup-7

Modify around cpu_identify() to not to break the dmesg of cpus with AB_VERBOSE
or AB_DEBUG.

branches: 1.111.2; 1.111.4;
Match lapic conditionals from the primary CPU.

branches: 1.110.2;
Add support for saving the AVX-256 ymm registers during FPU context switches.
Add support for the forthcoming AVX-512 registers.
Code compiled with -mavx seems to work, but I've not tested context
switches with live ymm registers.
There is a small cost on fork/exec (a larger area is copied/zerod),
but I don't think the ymm registers are read/written unless they
have been used.
The code use XSAVE on all cpus, I'm not brave enough to enable XSAVEOPT.

Add explicit #include <x86/fpu.h> instead of relying on pcb.h including it.

Remove support for 'external' floating point units and the MS-DOS
compatible method of handling floating point exceptions.
Make kernel support for teh fpu non-optional (486SX should still work).
Only 386 cpus support external fpu, and i386 support was removed years ago.
This means that the npx code no longer uses port 0xf0 or interupt 13.
All the "npx at isa" lines go from the configs, arch/i386/isa/npx.c
is now mandatory for all i386 kernels.
I've renamed npxinit() to fpuinit() and npxinit_cpu() to fpuinit_cpu()
to match the very similar amd64 functions.
The fpu of the boot cpu is now initialised by a direct call from
cpu_configure(), this enables FP emulation for a 486SX.
(for amd64 the cr0 values are set in locore.S and similar).
This fixes a long-standing bug in linux_setregs() - which did not
save the fpu regsiters if they were active.
I've test booted a single cpu i386 kernel (using anita).
amd64 builds - none of teh changes should affect it.
The i386 XEN kernels build, but I'm not sure where they set cr0, and
it might have got lost!

revert fpu/pcu changes until we figure out what's wrong; they cause random
freezes

Modify some macros and add some new macros for CPU family and model
to reduce code duplication and to avoid bug.

CPUID_TO_STEPPING(cpuid) (not changed)

CPUID_TO_FAMILY(cpuid) (new)
CPUID_TO_MODEL(cpuid) (new)

Return the display family and the display model.
The macro names are the same as FreeBSD.

CPUID_TO_BASEFAMILY(cpuid) (The old name was CPUID2FAMILY)
CPUID_TO_BASEMODEL(cpuid) (The old name was CPUID2MODEL)

Only for the base field.

CPUID_TO_EXTFAMILY(cpuid) (The old name was CPUID2EXTFAMILY)
CPUID_TO_EXTMODEL(cpuid) (The old name was CPUID2EXTMODEL)

Only for the extended field.

See http://mail-index.netbsd.org/port-amd64/2013/11/12/msg001978.html

Revert previos. I accidentally committed a debug code. Sorry.

Fix a bug in last commit. Check correct variable.

Use the MI "pcu" framework for bookkeeping of npx/fpu states on x86.
This reduces the amount of MD code enormously, and makes it easier
to implement support for newer CPU features which require more fpu
state, or for fpu usage by the kernel.
For access to FPU state across CPUs, an xcall kthread is used now
rather than a dedicated IPI.
No user visible changes intended.

branches: 1.102.2;
With recent introduction of conditionals for the various MP options, we
broke the build for x86 systems that have MULTIPROCESSOR but which do not
include MPBIOS. So let's try to untangle things just a bit. Presented
on current-users (and referenced on source-changes-d) without any comment.

XXX We really should find a better method to select kernel options; #ifdef
spaghetti is rather sub-optimal.

#ifdef - #endif-ed. NMCA, NISA, NNPX, NIOAPIC, LAPIC, MPBIOS and MULTIPROCESSOR.

branches: 1.100.2;
in cpu_boot_secondary_processors(), wait until all the other CPUs
have registered themselves in kcpuset_running before returning.
recent changes to the TLB invalidation xcall code assume that
any CPU which will receive a broadcast IPI is registered in
kcpuset_running, so ensure that is true by waiting here.

cpu_load_pmap: disable interrupts. add a comment to explain why. PR/44995

- Convert x86 MD code, mainly pmap(9) e.g. TLB shootdown code, to use
kcpuset(9) and thus replace hardcoded CPU bitmasks. This removes the
limitation of maximum CPUs.

- Support up to 256 CPUs on amd64 architecture by default.

Bug fixes, improvements, completion of Xen part and testing on 64-core
AMD Opteron(tm) Processor 6282 SE (also, as Xen HVM domU with 128 CPUs)
by Manuel Bouyer.

Apply patch proposed in PR port-xen/45975 (this does not solve the exact
problem reported here but is part of the solution):
xen_kpm_sync() is not working as expected,
leading to races between CPUs.
1 the check (xpq_cpu != &x86_curcpu) is always false because we
have different x86_curcpu symbols with different addresses in the kernel.
Fortunably, all addresses dissaemble to the same code.
Because of this we always use the code intended for bootstrap, which doesn't
use cross-calls or lock.

2 once 1 above is fixed, xen_kpm_sync() will use xcalls to sync other CPUs,
which cause it to sleep and pmap.c doesn't like that. It triggers this
KASSERT() in pmap_unmap_ptes():
KASSERT(pmap->pm_ncsw == curlwp->l_ncsw);
3 pmap->pm_cpus is not safe for the purpose of xen_kpm_sync(), which
needs to know on which CPU a pmap is loaded *now*:
pmap->pm_cpus is cleared before cpu_load_pmap() is called to switch
to a new pmap, leaving a window where a pmap is still in a CPU's
ci_kpm_pdir but not in pm_cpus. As a virtual CPU may be preempted
by the hypervisor at any time, it can be large enough to let another
CPU free the PTP and reuse it as a normal page.

To fix 2), avoid cross-calls and IPIs completely, and instead
use a mutex to update all CPU's ci_kpm_pdir from the local CPU.
It's safe because we just need to update the table page, a tlbflush IPI will
happen later. As a side effect, we don't need a different code for bootstrap,
fixing 1). The mutex added to struct cpu needs a small headers reorganisation.

to fix 3), introduce a pm_xen_ptp_cpus which is updated from
cpu_pmap_load(), whith the ci_kpm_mtx mutex held. Checking it with
ci_kpm_mtx held will avoid overwriting the wrong pmap's ci_kpm_pdir.

While there I removed the unused pmap_is_active() function;
and added some more details to DIAGNOSTIC panics.

branches: 1.96.2; 1.96.6; 1.96.8;
As cpu_shutdown() is a wrapper to cpu_suspend(), modify slightly to prevent
setting low frequencies for active non-bootstrap processors during shutdown.

add a "vm" device class for cpufeaturebus

remove a check against uvmexp.ncolors that is done inside uvm_page_recolor()
already anyway.

Call cpufreq_suspend(9) and cpufreq_resume(9) during suspend/resume.

Unbreak the build. (conflicting types in function declaration and definition)

Thanks riz@

Hide the MD details of specific IPIs behind semantically pleasing functions. This cleans up a couple of #ifdef XEN/#endif pairs

Don't shutdown the boostrap processor (BSP) because we may have to run
BIOS methods on it. For example, ACPI requires that we execute the code
for changing sleep state on the BSP.

This may help the problem where folks' machines would hang instead of
powering off when they entered ACPI sleep state 5.

XXX If the BSP is already shut down, we should start it back up.

Add small comment.

Welcome to 5.99.53! Merge rmind-uvmplock branch:

- Reorganize locking in UVM and provide extra serialisation for pmap(9).
New lock order: [vmpage-owner-lock] -> pmap-lock.

- Simplify locking in some pmap(9) modules by removing P->V locking.

- Use lock object on vmobjlock (and thus vnode_t::v_interlock) to share
the locks amongst UVM objects where necessary (tmpfs, layerfs, unionfs).

- Rewrite and optimise x86 TLB shootdown code, make it simpler and cleaner.
Add TLBSTATS option for x86 to collect statistics about TLB shootdowns.

- Unify /dev/mem et al in MI code and provide required locking (removes
kernel-lock on some ports). Also, avoid cache-aliasing issues.

Thanks to Andrew Doran and Joerg Sonnenberger, as their initial patches
formed the core changes of this branch.

branches: 1.87.2;
Use config_defer(9) for cpu_rescan() in cpu_attach().
Also mark few local functions as static.

Fix autoconf(9) of cpufeaturebus.

Move VIA_C7TEMP to the cpufeaturebus.

Move PowerNow! to the cpufeaturebus.

Move ENHANCED_SPEEDSTEP, or henceforth est(4), to the cpufeaturebus.

Modularize coretemp(4). Ok jmcneill@.

modularize VIA PadLock support
- retire options VIA_PADLOCK, replace with 'padlock0 at cpu0'
- driver supports attach & detach
- support building as a module

Some CPU have cpu counter (CPUID_TSC is there) but don't handle the
rdmsr instruction (CPUID_MSR is not there).
Introduce a cpu_counter_serializing() function to remplace rdmsr(MSR_TSC)
calls, which does a rdmsr(MSR_TSC) if available and cpu_counter() otherwise.
This makes the cpu counter useable on vortex86 CPUs.
OK ad@

branches: 1.79.2; 1.79.4;
Use pmf_device_register1(9) and add cpu_shutdown(), which calls cpu_suspend().

Machine dependent code is considered as part of UVM. Include
internal API header.

Revert all previous changes that were made naively believing that the
existing CPU power management implementations could peacefully coexist with
the acpicpu(4) driver. The following options can not be used with acpicpu(4):
ENHANCED_SPEEDSTEP, INTEL_ONDEMAND_CLOCKMOD, POWERNOW_K7, and POWERNOW_K8.

Revert the previous changes to EST. The used hack had an obvious flaw:
the acpicpu(4) driver should attach even if the existing frequency management
code fails to attach, mainly because ACPI is the only proper way to deal
with EST on new Intel system.

Use a more drastic hack to deal with this: when acpicpu(4) attachs, it tears
down any existing sysctl(8) controls and installs identical ones in place.
Upon detachment, the initialization function of the existing EST is called.

Move the sysctl function pointers used by acpicpu(4) to x86/cpu.c.
Rename these so that the same pointers may be used in other parts.

Store the MADT-derived CPU ID to <x86/cpu.h>. This is required to properly
match the ACPI processor object ID with the ID available in the APIC table.

Welcome PAE inside i386 current.

This patch is inspired by work previously done by Jeremy Morse, ported by me
to -current, merged with the work previously done for port-xen, together with
additionals fixes and improvements.

PAE option is disabled by default in GENERIC (but will be enabled in ALL in
the next few days).

In quick, PAE switches the CPU to a mode where physical addresses become
36 bits (64 GiB). Virtual address space remains at 32 bits (4 GiB). To cope
with the increased size of the physical address, they are manipulated as
64 bits variables by kernel and MMU.

When supported by the CPU, it also allows the use of the NX/XD bit that
provides no-execution right enforcement on a per physical page basis.

Notes:

- reworked locore.S

- introduce cpu_load_pmap(), used to switch pmap for the curcpu. Due to the
different handling of pmap mappings with PAE vs !PAE, Xen vs native, details
are hidden within this function. This helps calling it from assembly,
as some features, like BIOS calls, switch to pmap_kernel before mapping
trampoline code in low memory.

- some changes in bioscall and kvm86_call, to reflect the above.

- the L3 is "pinned" per-CPU, and is only manipulated by a
reduced set of functions within pmap. To track the L3, I added two
elements to struct cpu_info, namely ci_l3_pdirpa (PA of the L3), and
ci_l3_pdir (the L3 VA). Rest of the code considers that it runs "just
like" a normal i386, except that the L2 is 4 pages long (PTP_LEVELS is
still 2).

- similar to the ci_pae_l3_pdir{,pa} variables, amd64's xen_current_user_pgd
becomes an element of cpu_info (slowly paving the way for MP world).

- bootinfo_source struct declaration is modified, to cope with paddr_t size
change with PAE (it is not correct to assume that bs_addr is a paddr_t when
compiled with PAE - it should remain 32 bits). bs_addrs is now a
void * array (in bootloader's code under i386/stand/, the bs_addrs
is a physaddr_t, which is an unsigned long).

- fixes in multiboot code (same reason as bootinfo): paddr_t size
change. I used Elf32_* types, use RELOC() where necessary, and move the
memcpy() functions out of the if/else if (I do not expect sym and str tables
to overlap with ELF).

- 64 bits atomic functions for pmap

- all pmap_pdirpa access are now done through the pmap_pdirpa macro. It
hides the L3/L2 stuff from PAE, as well as the pm_pdirpa change in
struct pmap (it now becomes a PDP_SIZE array, with or without PAE).

- manipulation of recursive mappings ( PDIR_SLOT_{,A}PTEs ) is done via
loops on PDP_SIZE.

See also http://mail-index.netbsd.org/port-i386/2010/07/17/msg002062.html

No objection raised on port-i386@ and port-xen@R for about a week.

XXX kvm(3) will be fixed in another patch to properly handle both PAE and !PAE
kernel dumps (VA => PA macros are slightly different, and need proper 64 bits
PA support in kvm_i386).

XXX Mixing PAE and !PAE modules may lead to unwanted/unexpected results. This
cannot be solved easily, and needs lots of thinking before being declared
safe (paddr_t/bus_addr_t size handling, PD/PT macros abstractions).

cpu_attach: use kmem_zalloc instead of memset.

Turn PMAP_NOCACHE into MI flag.
Add MI flags PMAP_WRITE_COMBINE, PMAP_WRITE_BACK, PMAP_NOCACHE_OVR.
Update pmap(9) manpage.

hppa: Remove MD PMAP_NOCACHE flag as it exists as MI flag
mips: Rename MD PMAP_NOCACHE to PGC_NOCACHE.

x86: Implement new MI flags using Page-Attribute Tables.
x86: Implement BUS_SPACE_MAP_PREFETCHABLE.

Patch presented on tech-kern@:
http://mail-index.netbsd.org/tech-kern/2010/06/30/msg008458.html

No comments on this last version.

This patch fixes the NX regression issue observed on amd64 kernels, where
per-page execution right was disabled (therefore leading to the inability
of the kernel to detect fraudulent use of memory mappings marked as not
being executable).

- replace cpu_feature and ci_feature_flags variables by cpu_feature and
ci_feat_val arrays. This makes it cleaner and brings kernel code closer
to the design of cpuctl(8). A warning will be raised for each CPU that
does not expose the same features as the Boot Processor (BP).

- the blacklist of CPU features is now a macro defined in the
specialreg.h header, instead of hardcoding it inside MD initialization
code; fix comments.

- replace checks against CPUID_TSC with the cpu_hascounter() function.

- clean up the code in init_x86_64(), as cpu_feature variables are set
inside cpu_probe().

- use cpu_init_msrs() for i386. It will be eventually used later for NX
feature under i386 PAE kernels.

- remove code that checks for CPUID_NOX in amd64 mptramp.S, this is already
performed by cpu_hatch() through cpu_init_msrs().

- remove cpu_signature and feature_flags members from struct mpbios_proc
(they were never used).

This patch was tested with i386 MONOLITHIC, XEN3PAE_DOM0 and XEN3_DOM0 under
a native i386 host, and amd64 GENERIC, XEN3_DOM0 via QEMU virtual machines.

XXX Should kernel rev be bumped?

XXX A similar patch should be pulled-up for NetBSD-5, hopefully tomorrow.

branches: 1.69.2;
A pointer typedef entails trading too much flexibility to declare const
and non-const types, and the kernel uses both const and non-const
PMF qualifiers and device suspensors, so change the pmf_qual_t and
device_suspensor_t typedefs from "pointers to const" to non-pointer,
non-const types.

Wrap a comment; add a space after a comma to another (align with next line)

Use roundup2() instead of hardcoding the operation.

branches: 1.66.2;
Expand PMF_FN_* macros.

Use lwp_getpcb() on x86 MD code, clean from struct user usage.

Add a flags argument to pmap_kenter_pa(9).
Patch showed on tech-kern@ http://mail-index.netbsd.org/tech-kern/2009/11/04/msg006434.html
No objections.

Rearrange TSC inter-CPU synchronization code so that the gory details
are dealt with in x86/tsc.c and callers don't have to care that much.
Also add some comments and make some variables static.
approved by ad (a while ago)

branches: 1.62.2;
Make i386 config without NPX work, problem reported and fix tested by
Wojciech Galazka.
While there change a __i386__ to i386 for consistency.

move from malloc to kmem

PR kern/40213 my i386 machine can't boot because of tsc

- Patch in atomic_cas_64() twice. The first patch is early and makes it
the MP-atomic version available if we have cmpxchg8b. The second patch
strips the lock prefix if ncpu==1.

- Fix the i486 atomic_cas_64() to not unconditionally enable interrupts.

Link cpus in the order they are attaching and not in inverse order.

- Avoid the race with CPU online/offline state changes, when setting the
affinity (cpu_lock protects these operations now).
- Disallow setting of state of CPU to to offline, if there are bound LWPs,
which have no CPU to migrate.
- Disallow setting of affinity for the LWP(s), if all CPUs in the dynamic
CPU-set are offline.
- sched_setaffinity: fix invalid check of kcpuset_isset().
- Rename cpu_setonline() to cpu_setstate().

Should fix PR/39349.

branches: 1.57.2; 1.57.4;
- Rename cpu_lookup_byindex() to cpu_lookup(). The hardware ID isn't of
interest to MI code. No functional change.
- Change /dev/cpu to operate on cpu index, not hardware ID. Now cpuctl
shouldn't print confused output.

branches: 1.56.4;
If we boot with RB_MD1, register a NULL pmf handler for APs so we can
still suspend.

- Don't bother using sse to copy/zero pages on demand. It turns out not
to be worth it.
- If the machine has sse, re-enable zeroing pages in the idle loop and
use the sse instructions so that we don't blow out the cache.

Remove X86_MAXPROCS. There is still a 32-cpu limit, but it's now using
the MI constants.