MFC r354828:
Add x86 msr tweak KPI.

MFC r345078:
hwpmc/core: Adopt to upcoming Skylake TSX errata.

MFC r345074:
Remove useless version check.

Revert r330897:

This was intended to be a non-functional change. It wasn't. The commit
message was thus wrong. In addition it broke arm, and merged crypto
related code.

Revert with prejudice.

This revert skips files touched in r316370 since that commit was since
MFCed. This revert also skips files that require $FreeBSD$ property
changes.

Thank you to those who helped me get out of this mess including but not
limited to gonzo, kevans, rgrimes.

Requested by: gjb (re)

Partial merge of the SPDX changes

These changes are incomplete but are making it difficult
to determine what other changes can/should be merged.

No objections from: pfg

MFC r323230:
Skylake server core PMC support for hwpmc(4).

MFC 311224

Fix PMC architecture check to handle later IPAs including Skylake
Tested with tools/test/hwpmc/pmctest.py

Obtained from: Oliver Pinter

MFC 303720: Apply the fix from r232612 to fixed function counters.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Remove slightly used const values that can be replaced with nitems().

Suggested by: jhb

If full width writes to the performance monitoring counters are
supported, use full-width aliases MSRs for writes. This fixes the
"[pmc,X] negative increment" assertion on the context switch when
clipped counter value is sign-extended.

Add definitions for the MSR IA32_PERF_CAPABILITIES needed to detect
the feature.

PR: 207068
Submitted by: joss.upton@yahoo.com
MFC after: 2 weeks

More fixes in the various intel processors, fixing missing
IAP_F_FM's as well as incorrect umask specifications for
some of the new Broadwell/Skylake PMC's. Also silvermont
had a *lot* of missing IAP_F_FM.

Sponsored by: Netflix Inc.

Add support for Intel Skylake and Intel Broadwell PMC's. The Broadwell PMC's have been
tested on the Broadwell-Xeon with a hacked up version of pmcstudy -T. I still need
to circle back and add in to pmcstudy all the new tests from the Broadwell Vtune
guide (for the hacked up version I just made it so I could run the -T option). The
Skylake CPU is not yet available (even though Intel is advertising it .. imagine that).
The Skylake PMC's will need to be tested once we can get a sample skylake CPU :-)

Sponsored by: Netflix Inc.

Fix two bugs that could result in PMC sampling effectively stopping.
In both cases, the the effect of the bug was that a very small positive
number was written to the counter. This means that a large number of
events needed to occur before the next sampling interrupt would trigger.
Even with very frequently occurring events like clock cycles wrapping all
the way around could take a long time. Both bugs occurred when updating
the saved reload count for an outgoing thread on a context switch.

First, the counter-independent code compares the current reload count
against the count set when the thread switched in and generates a delta
to apply to the saved count. If this delta causes the reload counter
to go negative, it would add a full reload interval to wrap it around to
a positive value. The fix is to add the full reload interval if the
resulting counter is zero.

Second, occasionally the raw counter value read during a context switch
has actually wrapped, but an interrupt has not yet triggered. In this
case the existing logic would return a very large reload count (e.g.
2^48 - 2 if the counter had overflowed by a count of 2). This was seen
both for fixed-function and programmable counters on an E5-2643.
Workaround this case by returning a reload count of zero.

PR: 198149
Differential Revision: https://reviews.freebsd.org/D2557
Reviewed by: emaste
MFC after: 1 week
Sponsored by: Norse Corp, Inc.

Use the proper mask when reloading sampling PMCs for Core CPUs.

Differential Revision: https://reviews.freebsd.org/D2492
Reviewed by: emaste
MFC after: 1 month

Convert hwpmc(4) debug printfs over to KTR.

Differential Revision: https://reviews.freebsd.org/D2487
Reviewed by: davide, emaste
MFC after: 2 weeks
Sponsored by: Norse Corp, Inc.

hwpmc: add initial Intel Broadwell support.

The full list of aliases and events will follow in a subsequent
commit.

MFC after: 1 month

Remove whitespace.

hwpmc: Fix event number to match enum name

Differential revision: https://reviews.freebsd.org/D1592
Reviewed by: Joseph Kong
MFC after: 1 month

Add missing counter definitions

Differential Revision: https://reviews.freebsd.org/D1591
MFC after: 1 month
Sponsored by: Sandvine Inc

Fix Ivy Bridge+ MEM_UOPS_RETIRED counters

The MEM_UOPS_RETIRED actually work the same way as the Sandy
Bridge counters, but the counters were documented in a different
way and that seemed to cause the Ivy Bridge counters to be
implemented incorrectly. Use the same counter definitions as
Sandy Bridge. While I'm here, rename the counters to match
what's documented in the datasheet.

Differential Revision: https://reviews.freebsd.org/D1590
MFC after: 1 month
Sponsored by: Sandvine Inc.

Fix Sandy Bridge+ hwpmc branch counters

On Sandy Bridge and later, to count branch-related events you
have to or together a mask indicating the type of branch
instruction to count (e.g. direct jump, branch, etc) and a bits
indicating whether to count taken and not-taken branches. The
current counter definitions where defining this bits individually,
so the counters never worked and always just counted 0.

Fix the counter definitions to instead contain the proper
combination of masks. Also update the man pages to reflect the
new counters.

Differential Revision: https://reviews.freebsd.org/D1587
MFC after: 1 month
Sponsored by: Sandvine Inc.

Fix pmc unit restrictions to match documentation

A couple of pmc counters did not work because there were being
restricted to the wrong PMC unit. I've verified that these
counters now work and match the documented restrictions.

Differential Revision: https://reviews.freebsd.org/D1586
MFC after: 1 month
Sponsored by: Sandvine Inc

style(9) cleanup

Update the hwpmc driver to have the new type HASWELL_XEON. Also
go back through HASWELL, IVY_BRIDGE, IVY_BRIDGE_XEON and SANDY_BRIDGE
to straighten out all the missing PMCs. We also add a new pmc tool
pmcstudy, this allows one to run the various formulas from
the documents "Using Intel Vtune Amplifier XE on XXX Generation platforms" for
IB/SB and Haswell. The tool also allows one to postulate your own
formulas with any of the various PMC's. At some point I will enahance
this to work with Brendan Gregg's flame-graphs so we can flamegraph
various PMC interactions. Note the manual page also needs some
work (lots of work) but gnn has committed to help me with that ;-)
Reviewed by: gnn
MFC after:1 month
Sponsored by: Netflix Inc.

Since introducing the extra mapping in r250103 for architectural performance
events we have actually counted 'Branch Instruction Retired' when people
asked for 'Unhalted core cycles' using the 'unhalted-core-cycles' event mask
mnemonic.

Reviewed by: jimharris
Discussed with: gnn, rwatson
MFC after: 3 days
Sponsored by: DARPA/AFRL

For Xeon 7500 and 48XX (Nehalem EX and Westmere EX) variants of the
Core i7 and Westmere processors, the uncore PMC subsystem is
completely different from the uncore PMC on smaller versions of CPUs.
Disable existing uncore hwpmc code for EX, otherwise non-existing MSRs
are accessed.

The cores PMCs seems to be identical for non-EX and EX, according to
the SDM.

Reviewed by: davide, fabient
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks

Update hwpmc to support core events for Atom Silvermont microarchitecture.
(Model 0x4D as per Intel document 330061-001 01/2014)

Tested by: Olivier Cochard-Labbe <olivier@cochatrd.me>
MFC after: 4 weeks

Fix pointer type in call to malloc

Submitted by: Meyer, Conrad conrad.meyer@isilon.com

Move <machine/apicvar.h> to <x86/apicvar.h>.

o Remove assertions on ipa_version as sometimes the version detection
using cpuid can be quirky (this is the case of VMWare without the
vPMC support) but fail to probe hwpmc.
o Apply the fix for XEON family of processors as established by
315338-020 document (bug AJ85).

Sponsored by: EMC / Isilon storage division
Reviewed by: fabient

Fix a >80 character long line, introduced in my previous commit.

Noticed by: hiren

Update the MEM_UOP_RETIRED PMC operation for sandy bridge and sandy
bridge Xeon.

Summary: These are PEBS events but they're also available as normal
counter/sample events. The source table (Table 19-2) lists the
base versions (LOAD, STLB_MISS, SPLIT, ALL) but it says they must
be qualified with other values. This particular commit fleshes
out those umask values.

Source:

* Linux; SDM June 2013, Volume 3B, Table 19-2 and 18-21.

Tested:

* Sandy Bridge (non-Xeon)

Add in missing events for Sandy Bridge Xeon.

* Add in MEM_LOAD_UOPS_LLC_HIT_RETIRED for both sandy bridge and sandy
bridge Xeon. Right now it only is enabled for Sandy Bridge.
* D2/0F is actually a combination rather than a separate counter, so
just flip that on for the CPU types that support it.

There's an errata for using this on SB Xeon hardware - I've documented
it in kern/181346.

Tested:

* Sandy Bridge
* Sandy Bridge Xeon

Sponsored by: Netflix, Inc.

Suppress a GCC warning. This warning is actually bogus and newer GCC
versions than the one in base (dim@ mentioned he tried on 4.7.3 and 4.8.1)
do not whine about it, so, at some point this workaround will be reverted.

Reported by: ache
Discussed with: dim

The Intel PMC architectural events have encodings which are identical to
those of some non-architectural core events. This is not a problem in the
general case as long as there's an 1:1 mapping between the two, but there
are few exceptions. For example, 3CH_01H on Nehalem/Westmere represents
both unhalted-reference-cycles and CPU_CLK_UNHALTED.REF_P.
CPU_CLK_UNHALTED.REF_P on the aforementioned architectures does not measure
reference (i.e. bus) but TSC, so there's the need to disambiguate.
In order to avoid the namespace collision rename all the architectural
events in a way they cannot be ambigous and refactor the architectural
events handling function to reflect this change.
While here, per Jim Harris request, rename
iap_architectural_event_is_unsupported() to iap_event_is_architectural().

Discussed with: jimharris
Reviewed by: jimharris, gnn

Fixup Westmere hwpmc(4) support: add missing CPU flag so that
intrucion-retired, llc-misses and llc-reference events can now be
allocated.

Reviewed by: jimharris, gnn

Improve/correct a comment. We now support a lot more cpu types.

PR: kern/177496
Approved by: sbruno (mentor)

Cosmetic change: make a comment reference Sandy Bridge *Xeon*

Reviewed by: sbruno
MFC after: 1 week

Trailing whitespace cleanup along with 80 column enforcemnt.

Submitted by: hiren.panchasara@gmail.com
Reviewed by: sbruno@freebsd.org
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Update hwpmc to support Haswell class processors.
0x3C: /* Per Intel document 325462-045US 01/2013. */

Add manpage to document all the goodness that is available in this
processor model.

Submitted by: hiren panchasara <hiren.panchasara@gmail.com>
Reviewed by: jimharris, sbruno
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Update hwpmc to support the Xeon class of Ivybridge processors.
case 0x3E: /* Per Intel document 325462-045US 01/2013. */

Add manpage to document all the goodness that is available in this
processor model.

No support for uncore events at this time.

Submitted by: hiren panchasara <hiren.panchasara@gmail.com>
Reviewed by: davide, jimharris, sbruno
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Update hwpmc to support the Xeon class of Sandybridge processors.
(Model 0x2D /* Per Intel document 253669-044US 08/2012. */)

Add manpage to document all the goodness that is available in this
processor model.

No support for uncore events at this time.

Submitted by: hiren panchasara <hiren.panchasara@gmail.com>
Reviewed by: jimharris@ fabient@
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Complete and merge the list between Sandy/Ivy bridge of events
that can run on specific PMC.

MFC after: 1 month

Add Intel Ivy Bridge support to hwpmc(9).
Update offcore RSP token for Sandy Bridge.
Note: No uncore support.

Will works on Family 6 Model 3a.

MFC after: 1 month
Tested by: bapt, grehan

Fix so that ,usr and ,os work correctly with fixed function (IAF)
counters.

MFC after: 1 week

Add software PMC support.

New kernel events can be added at various location for sampling or counting.
This will for example allow easy system profiling whatever the processor is
with known tools like pmcstat(8).

Simultaneous usage of software PMC and hardware PMC is possible, for example
looking at the lock acquire failure, page fault while sampling on
instructions.

Sponsored by: NETASQ
MFC after: 1 month

Fix crash on VirtualBox (and probably on some real hardware):

- Do not cover error returned by pmc_core_initialize with the
result of pmc_uncore_initialize, fail right away.
- Give a user something to report instead failing silently

Reported by: Alexandr Kovalenko <never@nevermind.kiev.ua>

Properly mask off bits that are not supported in the IAP counters.
This fixes a bug where users would see massively large counts, near
to 2**64 -1, due to the bits not being cleared.

MFC after: 3 weeks

- Add support for the Intel Sandy Bridge microarchitecture (both core and uncore counting events)
- New manpages with event lists.
- Add MSRs for the Intel Sandy Bridge microarchitecture

Reviewed by: attilio, brueffer, fabient
Approved by: gnn (mentor)
MFC after: 3 weeks

Update PMC events from October 2011 Intel documentation.

Submitted by: Davide Italiano <davide.italiano@gmail.com>
MFC after: 3 days

- Remove extra space

Submitted by: Davide Italiano <davide.italiano@gmail.com>
Approved by: brucec

There's a small set of events on Nehalem, that are not supported in
processors with CPUID signature 06_1AH, 06_1EH, and 06_1FH.

Refuse to allocate them on unsupported model.

Submitted by: Davide Italiano <davide.italiano@gmail.com>
MFC after: 1 month

Update Westmere uncore event exception list.

Submitted by: Davide Italiano <davide italiano at gmail com>
MFC after: 1 week

Fix invalid class removal when IAF is not the last class.
Keep IAF class with 0 PMC and change the alias in libpmc to IAP.

MFC after: 1 week

Make sure that we clear the correct bits when we turn off
a PMC. It was possible that we could have turned a bit on but
never cleared it.

Extend the calls to rdmsr() to all necessary functions, not
just those which previously caused a panic.

Pointed out by: jhb@
MFC after: 1 week

Fix a panic brought about by writing an MSR without a proper mask.
All of the necessary wrmsr calls are now preceded by a rdmsr
and we leave the reserved bits alone.
Document the bits in the relevant registers for future reference.

Tested by: mdf
MFC after: 1 week

- Fix a typo OFFCORE_REQUESTS.ANY.RFO is B0H10H and not 80H10H.
- Enable missing PARTIAL_ADDRESS_ALIAS for Core i7.

MFC after: 3 days

- Support for uncore counting events: one fixed PMC with the uncore
domain clock, 8 programmable PMC.
- Westmere based CPU (Xeon 5600, Corei7 980X) support.
- New man pages with events list for core and uncore.
- Updated Corei7 events with Intel 253669-033US December 2009 doc.
There is some removed events in the documentation, they have been
kept in the code but documented in the man page as obsolete.
- Offcore response events can be setup with rsp token.

Sponsored by: NETASQ

* Support the L1D_CACHE_LD event on Core2 processors.
* Correct a group of typos: for Core2 programmable events, check
user supplied umask values against the correct event descriptor
field.

Submitted by: Ryan Stone <rysto32 at gmail dot com>

Only claim that the PMC_CLASS_IAF PMCs are supported by a CPU if
there are PMCs on the CPU that belong to the class.

Review and testing by: fabient

Add counters for the i7 architecture which were accidentally left
out of the original commit of i7 support. These are all the counters
on pages A-32 and A-33 of the _Intel(R) 64 and IA32 Architectures
Software Developer's Manual Vol 3B_, June 2009. Almost all
of these counters relate to operations on the L2 cache.

Reviewed by: jkoshy
MFC after: 1 month

Adjust the handling of the local APIC PMC interrupt vector:
- Provide lapic_disable_pmc(), lapic_enable_pmc(), and lapic_reenable_pmc()
routines in the local APIC code that the hwpmc(4) driver can use to
manage the local APIC PMC interrupt vector.
- Do not enable the local APIC PMC interrupt vector by default when
HWPMC_HOOKS is enabled. Instead, the hwpmc(4) driver explicitly
enables the interrupt when it is succesfully initialized and disables
the interrupt when it is unloaded. This avoids enabling the interrupt
on unsupported CPUs which may result in spurious NMIs.

Reported by: rnoland
Reviewed by: jkoshy
Approved by: re (kib)
MFC after: 2 weeks

- Add support for nehalem/corei7 cpus. This supports all of the core
counters defined in the reference manual. It does not support the
'uncore' events.

Reviewed by: jkoshy
Sponsored by: Nokia

Bug fixes:
- Initialize variables before use.
- Remove a KASSERT() that could falsely trigger if there are other sources
of NMIs in the system.

Efficiency tweak:
- When checking PMCs that overflowed, ignore PMCs that were not configured for
sampling.

Fixes for Core2 Extreme support.

Submitted by: "Artem Belevich" <artemb at gmail dot com>

- Add support for PMCs in Intel CPUs of Family 6, model 0xE (Core Solo
and Core Duo), models 0xF (Core2), model 0x17 (Core2Extreme) and
model 0x1C (Atom).

In these CPUs, the actual numbers, kinds and widths of PMCs present
need to queried at run time. Support for specific "architectural"
events also needs to be queried at run time.

Model 0xE CPUs support programmable PMCs, subsequent CPUs
additionally support "fixed-function" counters.

- Use event names that are close to vendor documentation, taking in
account that:
- events with identical semantics on two or more CPUs in this family
can have differing names in vendor documentation,
- identical vendor event names may map to differing events across
CPUs,
- each type of CPU supports a different subset of measurable
events.

Fixed-function and programmable counters both use the same vendor
names for events. The use of a class name prefix ("iaf-" or
"iap-" respectively) permits these to be distinguished.

- In libpmc, refactor pmc_name_of_event() into a public interface
and an internal helper function, for use by log handling code.

- Minor code tweaks: staticize a global, freshen a few comments.

Tested by: gnn