sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

hwpmc: tidy pcd_finalize methods

Make them a little bit more consistent with one another in terms of what
is done:
- Add/reduce assertions to what is most useful: a loop to ensure
pcpu_fini freed everything
- Add PMCDBG trace entries

The exception is the dmc620/cmn600 classes, which behave a little
differently, so leave them untouched.

Reviewed by: jkoshy
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41270

pmc: better distinguish pmu-events allocation path

Background:

The pm_ev field of struct pmc_op_pmcallocate and struct pmc
traditionally contains the index of the chosen event, corresponding to
the __PMC_EVENTS array in pmc_events.h. This is a static list of events,
maintained by FreeBSD.

In the usual case, libpmc translates the user supplied event name
(string) into the pm_ev index, which is passed as an argument to the
allocation syscall. On the kernel side, the allocation method for the
relevant hwpmc class translates the given index into the event code that
will be written to an event selection register.

In 2018, a new source of performance event definitions was introduced:
the pmu-events json files, which are maintained by the Linux kernel. The
result was better coverage for newer Intel processors with a reduced
maintenance burden for libpmc/hwpmc. Intel and AMD CPUs were
unconditionally switched to allocate events from pmu-events instead of
the traditional scheme (959826ca1bb0a, 81eb4dcf9e0d).

Under the pmu-events scheme, the pm_ev field contains an index
corresponding to the selected event from the pmu-events table, something
which the kernel has no knowledge of. The configuration for the
performance counting registers is instead passed via class-dependent
fields (struct pmc_md_op_pmcallocate).

In 2021 I changed the allocation logic so that it would attempt to
pull from the pmu-events table first, and fall-back to the traditional
method (dfb4fb41166bc3). Later, pmu-events support for arm64 and power8
CPUs was added (28dd6730a5d6 and b48a2770d48b).

The problem that remains is that the pm_ev field is overloaded, without
a definitive way to determine whether the event allocation came from the
pmu-events table or FreeBSD's statically-defined PMC events. This
resulted in a recent fix, 21f7397a61f7.

Change:

To disambiguate these two supported but separate use-cases, add a new
flag, PMC_F_EV_PMU, to be set as part of the allocation, indicating that
the event index came from pmu-events.

This is useful in two ways:
1. On the kernel side, we can validate the syscall arguments better.
Some classes support only the traditional event scheme (e.g.
hwpmc_armv7), while others support only the pmu-events method (e.g.
hwpmc_core for Intel). We can now check for this. The hwpmc_arm64
class supports both methods, so the new flag supersedes the existing
MD flag, PM_MD_EVENT_RAW.

2. The flag will be tracked in struct pmc for the duration of its
lifetime, meaning it is communicated back to userspace. This allows
libpmc to perform the reverse index-to-event-name translation
without speculating about the meaning of the index value.

Adding the flag is a backwards-incompatible ABI change. We recently
bumped the major version of the hwpmc module, so this breakage is
acceptable.

Reviewed by: jkoshy
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40753

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

hwpmc: pass pmc pointer to more class methods

In many cases this avoids an extra lookup, since the callers always have
pm at hand. We can also eliminate several assertions, mostly for pm !=
NULL. The class methods are an internal interface, and the callers
already handle such a scenario. No functional change intended.

Reviewed by: jkoshy
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39915

hwpmc: remove pmd_pcpu_{init,fini} callbacks

These are unused on all platforms.

Reviewed by: jkoshy, emaste
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39912

hwpmc: don't use deprecated copystr(9)

It is just wrapper around strlcpy(), but results in more complicated
code. Clean this up to use strlcpy() or snprintf() as appropriate.

Reviewed by: jkoshy
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39910

hwpmc: Fix a typo

MFC after: 1 week

hwpmc: Avoid touching MSR_DEBUGCTLMSR inside VMs.

At least KVM in Proxmox seems not happy about it. Just to be safe
block it for all VMs, since it should just improve profile accuracy.

MFC after: 1 week

hwpmc: Reenable PME before reenabling counters.

Doing otherwise may lead to lost interrupts, that in combination
with hardware PMCs freezing may leave them frozen forever. It may
also slightly improve profiling accuracy.

MFC after: 2 weeks

hwpmc: Add IDs for few more Intel Atom CPUs.

MFC after: 1 month

hwpmc: Skip GLOBAL_CTRL updates on stop_pmc().

After we wipe PMC configuration, including its local enable bit(s),
we don't really care about its global enable bit. Global enable bits
now may only be cleared by interrupt handler in case of error (sample
buffer overflow). Being set is actually a reset default for them.

This saves one WRMSR per process-scope PMC per context switch, that
is clearly visible in profiles.

MFC after: 1 month

hwpmc: Bump Intel's IA32_PERFEVTSELx width to 64 bits.

Haswell added there bits 32/33 for TSX, and AlderLake added bit 34
for Adaptive PEBS Record.

MFC after: 1 month

hwpmc: Add basic Intel Alderlake CPUs support.

The PMC subsystem is not designed for non-uniform CPU capabilities
(P/E-cores are different), but at least several working architectural
events like cpu_clk_unhalted.thread_p should be better than nothing.

MFC after: 1 month

hwpmc: Update Intel's programmable counters restrictions.

Primarily remove unneeded restrictions from later CPUs.

MFC after: 1 month

hwpmc: Correct selection of Intel fixed counters.

Intel json's use event=0 to specify fixed counter number via umask.
Alternatively fixed counters have equivalent programmable event/umask.

MFC after: 1 month

hwpmc: Use hardware PMCs freezing on PMI on Intel v2+.

Since version 2 Intel CPUs can freeze PMCs when intering PMI to reduce
PMI effects on collected statistics. Since version 4 hardware supports
"streamlined" mechanism, not requiring IA_GLOBAL_CTRL MSR access.

MFC after: 1 month

hwpmc: drop 0x before %p printf format string

%p already includes the 0x.

Sponsored by: The FreeBSD Foundation

hwpmc: Add IDs for Intel Comet/Ice/Tiger/Rocketlake CPUs.

MFC after: 1 month

hwpmc: don't validate capabilities in allocation method

These checks were inconsistently applied across the various hwpmc
classes. The condition is already checked by the generic code in
hwpmc_mod.c, so remove them.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31388

hwpmc: consistently validate PMC class in allocation method

It is always a good idea. In one case, attempting to allocate N+1 PMCs
from a class with N hardware counters would incorrectly attempt to
allocate from the next class in the list. Without this validation, this
can lead to all kinds of strange behaviour.

Since powerpc_allocate_pmc() is used by both the mpc7xxx and ppc970
classes, add a new global to track which is active (it will never be
both).

Reviewed by: luporl, ray
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31387

x86_msr_op: extend the KPI to allow MSR read and single-CPU operations

Reivewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D31386

hwpmc: mostly clean up cc --analyze

Sponsored by: Rubicon Communications, LLC ("Netgate")

Add x86 msr tweak KPI.

Use the KPI to tweak MSRs in mitigation code.

Reviewed by: markj, scottl
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D22431

hwpmc/core: Adopt to upcoming Skylake TSX errata.

The forthcoming microcode update will fix a TSX bug by clobbering PMC3
when TSX instructions are executed (even speculatively). There is an
alternate mode where CPU executes all TSX instructions by aborting
them, in which case PMC3 is still available to OS. Any code that
correctly uses TSX must be ready to handle abort anyway.

Since it is believed that FreeBSD population of hwpmc(4) users is
significantly larger than the population of TSX users, switch the
microcode into TSX abort mode whenever a pmc is allocated, and back to
bug avoidance mode when the last pmc is deallocated.

Sponsored by: The FreeBSD Foundation
MFC after: 1 week

Remove useless version check.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days

hwpmc: avoid undefined variable on LINT

hwpmc: simplify calling convention for hwpmc interrupt handling

pmc_process_interrupt takes 5 arguments when only 3 are needed.
cpu is always available in curcpu and inuserspace can always be
derived from the passed trapframe.

While facially a reasonable cleanup this change was motivated
by the need to workaround a compiler bug.

core2_intr(cpu, tf) ->
pmc_process_interrupt(cpu, ring, pmc, tf, inuserspace) ->
pmc_add_sample(cpu, ring, pm, tf, inuserspace)

In the process of optimizing the tail call the tf pointer was getting
clobbered:

(kgdb) up
at /storage/mmacy/devel/freebsd/sys/dev/hwpmc/hwpmc_mod.c:4709
4709 pmc_save_kernel_callchain(ps->ps_pc,
(kgdb) up
1205 error = pmc_process_interrupt(cpu, PMC_HR, pm, tf,

resulting in a crash in pmc_save_kernel_callchain.

hwpmc: fix fixed counters checks

hwpmc: ABI fixes
- increase pmc cpuid field from 8 to 12 bits
- add cpuid version string to initialize entry in the log
so that filter can identify which counter index an
event name maps to
- GC unused config flags
- make fixed counter assignment more robust as well as the
changes needed to be properly identified for filter

hwpmc: remove unused pre-table driven bits for intel

Intel now provides comprehensive tables for all performance counters
and the various valid configuration permutations as text .json files.
Libpmc has been converted to use these and hwpmc_core has been greatly
simplified by moving to passthrough of the table values.

The one gotcha is that said tables don't support pentium pro and and pentium
IV. There's very few users of hwpmc on _amd64_ kernels on new hardware. It is
unlikely that anyone is doing low level optimization on 15 year old Intel
hardware. Nonetheless, if someone feels strongly enough to populate the
corresponding tables for p4 and ppro I will reinstate the files in to the
build.

Code for the K8 counters and !x86 architectures remains unchanged.

hwpmc: don't incrorrectly strip the ANY flag

pmc(3)/hwpmc(4): update supported Intel processors to rely fully on the
vendor provided pmu-events tables and sundry cleanups.

The vendor pmu-events tables provide counter descriptions, default
sample rates, event, umask, and flag values for all the counter
configuration permutations. Using this gives us:

- much simpler kernel code for the MD component
- helpful long and short event descriptions
- simpler user code
- sample rates that won't overload the system

Update man page with newer sample types and remove unused sample type.

Revert r334242 "pmc(3)/hwpmc(4): update supported Intel processors to rely fully on the"
because of squash commit messages

pmc(3)/hwpmc(4): update supported Intel processors to rely fully on the
vendor provided pmu-events tables and sundry cleanups.

The vendor pmu-events tables provide counter descriptions, default
sample rates, event, umask, and flag values for all the counter
configuration permutations. Using this gives us:

- much simpler kernel code for the MD component
- helpful long and short event descriptions
- simpler user code
- sample rates that won't overload the system

Update man page with newer sample types and remove unused sample type.

Squashed commit of the following:

commit 4459d43eff815bec08ccc5533dbe5de846f03128
Author: Matt Macy <mmacy@mattmacy.io>
Date: Sat May 26 00:06:31 2018 -0700

libpmc: fix pmu function signatures for non amd64

commit a2cb8bbc586c65d41f9b291430a2261ec67b59fe
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:38:11 2018 -0700

pmcstat: fix indentation of usage

commit f686954b15ff56a833ac80404898977cb80a265b
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:19:49 2018 -0700

pmclog(3): add callchain and pmcallocatedyn, remove pcsample

commit 73e13a0d2e9498c81c150d14d022050cee7511bb
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:19:00 2018 -0700

pmclog.h: GC pcsample field

commit 3e93ffd65da641fa657539dad3c48e281f8b5798
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:05:57 2018 -0700

hwpmc: make Intel core CPUs use external event tables

commit 634f5fae1e1644ac324003136c66cd9c619d1c93
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:00:06 2018 -0700

pmclog: update log record types, bump PMC_MAJOR
- explicitly make log record types a multiple of 8 bytes
- hook in pmu event types for pmc_allocate records
- remove references to no longer PCSAMPLE record

commit 83d84fcd2d65bdf6ddcb2e155a22f0cfa2a9c225
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 21:52:10 2018 -0700

libpmc: add support for having vendor table driven pmc_allocate

commit 9e6ad63c40c2fce8404847ace5078ca6cb33a736
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 19:11:33 2018 -0700

hwpmc_core: add accessors for EVSEL & UMASK, make IAP_UMASK useful to user

commit 859dceb93daa6419a48c794db99b6758e5b041c9
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 19:09:45 2018 -0700

pmcstat: update usage and man page as well as make -L consistent with pmccontrol

commit 79c7d8597e28c2eb13f5f9113e65ec2792ca57b1
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 18:07:03 2018 -0700

pmu_util: add support for all current intel event keywords

commit d8089c7f6a6c8527f38324252b1ffb47004694c6
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 17:45:00 2018 -0700

add description for new arguments

commit 058336740bab53c62ec88a3a026ea848cf3878c6
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 17:38:15 2018 -0700

libpmc: move pmu_events table and pmu_utils out of libpmcstat so that they can be used by pmc_allocate

commit 049b66b382e2f833c3f47bc8df9e750cb265709f
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 16:12:41 2018 -0700

pmcstat: hook pmu_events counter description utility routines in

commit f5e01e7b37a691dc045e1aa16b3ebdd162515de8
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 16:11:59 2018 -0700

pmu_events: add utility routines for listing counters and their descriptions

commit cba4d4f8907f772279f86f18f915e0d74d33ac56
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 16:09:50 2018 -0700

pmu-events: expand out skylake regex to simplify string matches

hwpmc(9): Make pmclog buffer pcpu and update constants

On non-trivial SMP systems the contention on the pmc_owner mutex leads
to a substantial number of samples captured being from the pmc process
itself. This change a) makes buffers larger to avoid contention on the
global list b) makes the working sample buffer per cpu.

Run pmcstat in the background (default event rate of 64k):
pmcstat -S UNHALTED_CORE_CYCLES -O /dev/null sleep 600 &

Before:
make -j96 buildkernel -s >&/dev/null 3336.68s user 24684.10s system 7442% cpu 6:16.50 total

After:
make -j96 buildkernel -s >&/dev/null 2697.82s user 1347.35s system 6058% cpu 1:06.77 total

For more realistic overhead measurement set the sample rate for ~2khz
on a 2.1Ghz processor:
pmcstat -n 1050000 -S UNHALTED_CORE_CYCLES -O /dev/null sleep 6000 &

Collecting 10 samples of `make -j96 buildkernel` from each:

x before
+ after

real time:
N Min Max Median Avg Stddev
x 10 76.4 127.62 84.845 88.577 15.100031
+ 10 59.71 60.79 60.135 60.179 0.29957192
Difference at 95.0% confidence
-28.398 +/- 10.0344
-32.0602% +/- 7.69825%
(Student's t, pooled s = 10.6794)

system time:
N Min Max Median Avg Stddev
x 10 2277.96 6948.53 2949.47 3341.492 1385.2677
+ 10 1038.7 1081.06 1070.555 1064.017 15.85404
Difference at 95.0% confidence
-2277.47 +/- 920.425
-68.1574% +/- 8.77623%
(Student's t, pooled s = 979.596)

x no pmc
+ pmc running
real time:

HEAD:
N Min Max Median Avg Stddev
x 10 58.38 59.15 58.86 58.847 0.22504567
+ 10 76.4 127.62 84.845 88.577 15.100031
Difference at 95.0% confidence
29.73 +/- 10.0335
50.5208% +/- 17.0525%
(Student's t, pooled s = 10.6785)

patched:
N Min Max Median Avg Stddev
x 10 58.38 59.15 58.86 58.847 0.22504567
+ 10 59.71 60.79 60.135 60.179 0.29957192
Difference at 95.0% confidence
1.332 +/- 0.248939
2.2635% +/- 0.426506%
(Student's t, pooled s = 0.264942)

system time:

HEAD:
N Min Max Median Avg Stddev
x 10 1010.15 1073.31 1025.465 1031.524 18.135705
+ 10 2277.96 6948.53 2949.47 3341.492 1385.2677
Difference at 95.0% confidence
2309.97 +/- 920.443
223.937% +/- 89.3039%
(Student's t, pooled s = 979.616)

patched:
N Min Max Median Avg Stddev
x 10 1010.15 1073.31 1025.465 1031.524 18.135705
+ 10 1038.7 1081.06 1070.555 1064.017 15.85404
Difference at 95.0% confidence
32.493 +/- 16.0042
3.15% +/- 1.5794%
(Student's t, pooled s = 17.0331)

Reviewed by: jeff@
Approved by: sbruno@
Differential Revision: https://reviews.freebsd.org/D15155

sys/dev: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

o Support for Kabylake CPU PMCs (fall down to PMC_CPU_INTEL_SKYLAKE).
o Fix bugs in events descriptions for Skylake, Skylake Xeon and Haswell.

Reviewed by: kib
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D12654

Skylake server core PMC support for hwpmc(4).

Reviewed by: emaste
Sponsored by: The FreeBSD Foundation
Hardware provided by: Intel
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D12221

Fix PMC architecture check to handle later IPAs including Skylake
Tested with tools/test/hwpmc/pmctest.py

Obtained from: Oliver Pinter
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D9036

Apply the fix from r232612 to fixed function counters.

Reviewed by: emaste
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D7397

Remove slightly used const values that can be replaced with nitems().

Suggested by: jhb

If full width writes to the performance monitoring counters are
supported, use full-width aliases MSRs for writes. This fixes the
"[pmc,X] negative increment" assertion on the context switch when
clipped counter value is sign-extended.

Add definitions for the MSR IA32_PERF_CAPABILITIES needed to detect
the feature.

PR: 207068
Submitted by: joss.upton@yahoo.com
MFC after: 2 weeks

More fixes in the various intel processors, fixing missing
IAP_F_FM's as well as incorrect umask specifications for
some of the new Broadwell/Skylake PMC's. Also silvermont
had a *lot* of missing IAP_F_FM.

Sponsored by: Netflix Inc.

Add support for Intel Skylake and Intel Broadwell PMC's. The Broadwell PMC's have been
tested on the Broadwell-Xeon with a hacked up version of pmcstudy -T. I still need
to circle back and add in to pmcstudy all the new tests from the Broadwell Vtune
guide (for the hacked up version I just made it so I could run the -T option). The
Skylake CPU is not yet available (even though Intel is advertising it .. imagine that).
The Skylake PMC's will need to be tested once we can get a sample skylake CPU :-)

Sponsored by: Netflix Inc.

Fix two bugs that could result in PMC sampling effectively stopping.
In both cases, the the effect of the bug was that a very small positive
number was written to the counter. This means that a large number of
events needed to occur before the next sampling interrupt would trigger.
Even with very frequently occurring events like clock cycles wrapping all
the way around could take a long time. Both bugs occurred when updating
the saved reload count for an outgoing thread on a context switch.

First, the counter-independent code compares the current reload count
against the count set when the thread switched in and generates a delta
to apply to the saved count. If this delta causes the reload counter
to go negative, it would add a full reload interval to wrap it around to
a positive value. The fix is to add the full reload interval if the
resulting counter is zero.

Second, occasionally the raw counter value read during a context switch
has actually wrapped, but an interrupt has not yet triggered. In this
case the existing logic would return a very large reload count (e.g.
2^48 - 2 if the counter had overflowed by a count of 2). This was seen
both for fixed-function and programmable counters on an E5-2643.
Workaround this case by returning a reload count of zero.

PR: 198149
Differential Revision: https://reviews.freebsd.org/D2557
Reviewed by: emaste
MFC after: 1 week
Sponsored by: Norse Corp, Inc.

Use the proper mask when reloading sampling PMCs for Core CPUs.

Differential Revision: https://reviews.freebsd.org/D2492
Reviewed by: emaste
MFC after: 1 month

Convert hwpmc(4) debug printfs over to KTR.

Differential Revision: https://reviews.freebsd.org/D2487
Reviewed by: davide, emaste
MFC after: 2 weeks
Sponsored by: Norse Corp, Inc.

hwpmc: add initial Intel Broadwell support.

The full list of aliases and events will follow in a subsequent
commit.

MFC after: 1 month

Remove whitespace.

hwpmc: Fix event number to match enum name

Differential revision: https://reviews.freebsd.org/D1592
Reviewed by: Joseph Kong
MFC after: 1 month

Add missing counter definitions

Differential Revision: https://reviews.freebsd.org/D1591
MFC after: 1 month
Sponsored by: Sandvine Inc

Fix Ivy Bridge+ MEM_UOPS_RETIRED counters

The MEM_UOPS_RETIRED actually work the same way as the Sandy
Bridge counters, but the counters were documented in a different
way and that seemed to cause the Ivy Bridge counters to be
implemented incorrectly. Use the same counter definitions as
Sandy Bridge. While I'm here, rename the counters to match
what's documented in the datasheet.

Differential Revision: https://reviews.freebsd.org/D1590
MFC after: 1 month
Sponsored by: Sandvine Inc.

Fix Sandy Bridge+ hwpmc branch counters

On Sandy Bridge and later, to count branch-related events you
have to or together a mask indicating the type of branch
instruction to count (e.g. direct jump, branch, etc) and a bits
indicating whether to count taken and not-taken branches. The
current counter definitions where defining this bits individually,
so the counters never worked and always just counted 0.

Fix the counter definitions to instead contain the proper
combination of masks. Also update the man pages to reflect the
new counters.

Differential Revision: https://reviews.freebsd.org/D1587
MFC after: 1 month
Sponsored by: Sandvine Inc.

Fix pmc unit restrictions to match documentation

A couple of pmc counters did not work because there were being
restricted to the wrong PMC unit. I've verified that these
counters now work and match the documented restrictions.

Differential Revision: https://reviews.freebsd.org/D1586
MFC after: 1 month
Sponsored by: Sandvine Inc

style(9) cleanup

Update the hwpmc driver to have the new type HASWELL_XEON. Also
go back through HASWELL, IVY_BRIDGE, IVY_BRIDGE_XEON and SANDY_BRIDGE
to straighten out all the missing PMCs. We also add a new pmc tool
pmcstudy, this allows one to run the various formulas from
the documents "Using Intel Vtune Amplifier XE on XXX Generation platforms" for
IB/SB and Haswell. The tool also allows one to postulate your own
formulas with any of the various PMC's. At some point I will enahance
this to work with Brendan Gregg's flame-graphs so we can flamegraph
various PMC interactions. Note the manual page also needs some
work (lots of work) but gnn has committed to help me with that ;-)
Reviewed by: gnn
MFC after:1 month
Sponsored by: Netflix Inc.

Since introducing the extra mapping in r250103 for architectural performance
events we have actually counted 'Branch Instruction Retired' when people
asked for 'Unhalted core cycles' using the 'unhalted-core-cycles' event mask
mnemonic.

Reviewed by: jimharris
Discussed with: gnn, rwatson
MFC after: 3 days
Sponsored by: DARPA/AFRL

For Xeon 7500 and 48XX (Nehalem EX and Westmere EX) variants of the
Core i7 and Westmere processors, the uncore PMC subsystem is
completely different from the uncore PMC on smaller versions of CPUs.
Disable existing uncore hwpmc code for EX, otherwise non-existing MSRs
are accessed.

The cores PMCs seems to be identical for non-EX and EX, according to
the SDM.

Reviewed by: davide, fabient
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks

Update hwpmc to support core events for Atom Silvermont microarchitecture.
(Model 0x4D as per Intel document 330061-001 01/2014)

Tested by: Olivier Cochard-Labbe <olivier@cochatrd.me>
MFC after: 4 weeks

Fix pointer type in call to malloc

Submitted by: Meyer, Conrad conrad.meyer@isilon.com

Move <machine/apicvar.h> to <x86/apicvar.h>.

o Remove assertions on ipa_version as sometimes the version detection
using cpuid can be quirky (this is the case of VMWare without the
vPMC support) but fail to probe hwpmc.
o Apply the fix for XEON family of processors as established by
315338-020 document (bug AJ85).

Sponsored by: EMC / Isilon storage division
Reviewed by: fabient

Fix a >80 character long line, introduced in my previous commit.

Noticed by: hiren

Update the MEM_UOP_RETIRED PMC operation for sandy bridge and sandy
bridge Xeon.

Summary: These are PEBS events but they're also available as normal
counter/sample events. The source table (Table 19-2) lists the
base versions (LOAD, STLB_MISS, SPLIT, ALL) but it says they must
be qualified with other values. This particular commit fleshes
out those umask values.

Source:

* Linux; SDM June 2013, Volume 3B, Table 19-2 and 18-21.

Tested:

* Sandy Bridge (non-Xeon)

Add in missing events for Sandy Bridge Xeon.

* Add in MEM_LOAD_UOPS_LLC_HIT_RETIRED for both sandy bridge and sandy
bridge Xeon. Right now it only is enabled for Sandy Bridge.
* D2/0F is actually a combination rather than a separate counter, so
just flip that on for the CPU types that support it.

There's an errata for using this on SB Xeon hardware - I've documented
it in kern/181346.

Tested:

* Sandy Bridge
* Sandy Bridge Xeon

Sponsored by: Netflix, Inc.

Suppress a GCC warning. This warning is actually bogus and newer GCC
versions than the one in base (dim@ mentioned he tried on 4.7.3 and 4.8.1)
do not whine about it, so, at some point this workaround will be reverted.

Reported by: ache
Discussed with: dim

The Intel PMC architectural events have encodings which are identical to
those of some non-architectural core events. This is not a problem in the
general case as long as there's an 1:1 mapping between the two, but there
are few exceptions. For example, 3CH_01H on Nehalem/Westmere represents
both unhalted-reference-cycles and CPU_CLK_UNHALTED.REF_P.
CPU_CLK_UNHALTED.REF_P on the aforementioned architectures does not measure
reference (i.e. bus) but TSC, so there's the need to disambiguate.
In order to avoid the namespace collision rename all the architectural
events in a way they cannot be ambigous and refactor the architectural
events handling function to reflect this change.
While here, per Jim Harris request, rename
iap_architectural_event_is_unsupported() to iap_event_is_architectural().

Discussed with: jimharris
Reviewed by: jimharris, gnn

Fixup Westmere hwpmc(4) support: add missing CPU flag so that
intrucion-retired, llc-misses and llc-reference events can now be
allocated.

Reviewed by: jimharris, gnn

Improve/correct a comment. We now support a lot more cpu types.

PR: kern/177496
Approved by: sbruno (mentor)

Cosmetic change: make a comment reference Sandy Bridge *Xeon*

Reviewed by: sbruno
MFC after: 1 week

Trailing whitespace cleanup along with 80 column enforcemnt.

Submitted by: hiren.panchasara@gmail.com
Reviewed by: sbruno@freebsd.org
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Update hwpmc to support Haswell class processors.
0x3C: /* Per Intel document 325462-045US 01/2013. */

Add manpage to document all the goodness that is available in this
processor model.

Submitted by: hiren panchasara <hiren.panchasara@gmail.com>
Reviewed by: jimharris, sbruno
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Update hwpmc to support the Xeon class of Ivybridge processors.
case 0x3E: /* Per Intel document 325462-045US 01/2013. */

Add manpage to document all the goodness that is available in this
processor model.

No support for uncore events at this time.

Submitted by: hiren panchasara <hiren.panchasara@gmail.com>
Reviewed by: davide, jimharris, sbruno
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Update hwpmc to support the Xeon class of Sandybridge processors.
(Model 0x2D /* Per Intel document 253669-044US 08/2012. */)

Add manpage to document all the goodness that is available in this
processor model.

No support for uncore events at this time.

Submitted by: hiren panchasara <hiren.panchasara@gmail.com>
Reviewed by: jimharris@ fabient@
Obtained from: Yahoo! Inc.
MFC after: 2 weeks

Complete and merge the list between Sandy/Ivy bridge of events
that can run on specific PMC.

MFC after: 1 month

Add Intel Ivy Bridge support to hwpmc(9).
Update offcore RSP token for Sandy Bridge.
Note: No uncore support.

Will works on Family 6 Model 3a.

MFC after: 1 month
Tested by: bapt, grehan

Fix so that ,usr and ,os work correctly with fixed function (IAF)
counters.

MFC after: 1 week

Add software PMC support.

New kernel events can be added at various location for sampling or counting.
This will for example allow easy system profiling whatever the processor is
with known tools like pmcstat(8).

Simultaneous usage of software PMC and hardware PMC is possible, for example
looking at the lock acquire failure, page fault while sampling on
instructions.

Sponsored by: NETASQ
MFC after: 1 month

Fix crash on VirtualBox (and probably on some real hardware):

- Do not cover error returned by pmc_core_initialize with the
result of pmc_uncore_initialize, fail right away.
- Give a user something to report instead failing silently

Reported by: Alexandr Kovalenko <never@nevermind.kiev.ua>

Properly mask off bits that are not supported in the IAP counters.
This fixes a bug where users would see massively large counts, near
to 2**64 -1, due to the bits not being cleared.

MFC after: 3 weeks

- Add support for the Intel Sandy Bridge microarchitecture (both core and uncore counting events)
- New manpages with event lists.
- Add MSRs for the Intel Sandy Bridge microarchitecture

Reviewed by: attilio, brueffer, fabient
Approved by: gnn (mentor)
MFC after: 3 weeks

Update PMC events from October 2011 Intel documentation.

Submitted by: Davide Italiano <davide.italiano@gmail.com>
MFC after: 3 days

- Remove extra space

Submitted by: Davide Italiano <davide.italiano@gmail.com>
Approved by: brucec

There's a small set of events on Nehalem, that are not supported in
processors with CPUID signature 06_1AH, 06_1EH, and 06_1FH.

Refuse to allocate them on unsupported model.

Submitted by: Davide Italiano <davide.italiano@gmail.com>
MFC after: 1 month

Update Westmere uncore event exception list.

Submitted by: Davide Italiano <davide italiano at gmail com>
MFC after: 1 week

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Fix invalid class removal when IAF is not the last class.
Keep IAF class with 0 PMC and change the alias in libpmc to IAP.

MFC after: 1 week

Make sure that we clear the correct bits when we turn off
a PMC. It was possible that we could have turned a bit on but
never cleared it.

Extend the calls to rdmsr() to all necessary functions, not
just those which previously caused a panic.

Pointed out by: jhb@
MFC after: 1 week

Fix a panic brought about by writing an MSR without a proper mask.
All of the necessary wrmsr calls are now preceded by a rdmsr
and we leave the reserved bits alone.
Document the bits in the relevant registers for future reference.

Tested by: mdf
MFC after: 1 week

MFC r206089, r206684:

- Support for uncore counting events: one fixed PMC with the uncore
domain clock, 8 programmable PMC.
- Westmere based CPU (Xeon 5600, Corei7 980X) support.
- New man pages with events list for core and uncore.
- Updated Corei7 events with Intel 253669-033US December 2009 doc.
There is some removed events in the documentation, they have been
kept in the code but documented in the man page as obsolete.
- Offcore response events can be setup with rsp token.

Sponsored by: NETASQ

- Fix a typo OFFCORE_REQUESTS.ANY.RFO is B0H10H and not 80H10H.
- Enable missing PARTIAL_ADDRESS_ALIAS for Core i7.

MFC after: 3 days

- Support for uncore counting events: one fixed PMC with the uncore
domain clock, 8 programmable PMC.
- Westmere based CPU (Xeon 5600, Corei7 980X) support.
- New man pages with events list for core and uncore.
- Updated Corei7 events with Intel 253669-033US December 2009 doc.
There is some removed events in the documentation, they have been
kept in the code but documented in the man page as obsolete.
- Offcore response events can be setup with rsp token.

Sponsored by: NETASQ

MFC 201023:
* Support the L1D_CACHE_LD event on Core2 processors.
* Correct a group of typos: for Core2 programmable events, check
user supplied umask values against the correct event descriptor
field.

* Support the L1D_CACHE_LD event on Core2 processors.
* Correct a group of typos: for Core2 programmable events, check
user supplied umask values against the correct event descriptor
field.

Submitted by: Ryan Stone <rysto32 at gmail dot com>

MFC 196739:
Add counters for the i7 architecture which were accidentally left
out of the original commit of i7 support. These are all the counters
on pages A-32 and A-33 of the _Intel(R) 64 and IA32 Architectures
Software Developer's Manual Vol 3B_, June 2009. Almost all
of these counters relate to operations on the L2 cache.

MFC 198432:
Only claim that the PMC_CLASS_IAF PMCs are supported by a CPU if
there are PMCs on the CPU that belong to the class.

Only claim that the PMC_CLASS_IAF PMCs are supported by a CPU if
there are PMCs on the CPU that belong to the class.

Review and testing by: fabient

Add counters for the i7 architecture which were accidentally left
out of the original commit of i7 support. These are all the counters
on pages A-32 and A-33 of the _Intel(R) 64 and IA32 Architectures
Software Developer's Manual Vol 3B_, June 2009. Almost all
of these counters relate to operations on the L2 cache.

Reviewed by: jkoshy
MFC after: 1 month

Adjust the handling of the local APIC PMC interrupt vector:
- Provide lapic_disable_pmc(), lapic_enable_pmc(), and lapic_reenable_pmc()
routines in the local APIC code that the hwpmc(4) driver can use to
manage the local APIC PMC interrupt vector.
- Do not enable the local APIC PMC interrupt vector by default when
HWPMC_HOOKS is enabled. Instead, the hwpmc(4) driver explicitly
enables the interrupt when it is succesfully initialized and disables
the interrupt when it is unloaded. This avoids enabling the interrupt
on unsupported CPUs which may result in spurious NMIs.

Reported by: rnoland
Reviewed by: jkoshy
Approved by: re (kib)
MFC after: 2 weeks

Adjust the handling of the local APIC PMC interrupt vector:
- Provide lapic_disable_pmc(), lapic_enable_pmc(), and lapic_reenable_pmc()
routines in the local APIC code that the hwpmc(4) driver can use to
manage the local APIC PMC interrupt vector.
- Do not enable the local APIC PMC interrupt vector by default when
HWPMC_HOOKS is enabled. Instead, the hwpmc(4) driver explicitly
enables the interrupt when it is succesfully initialized and disables
the interrupt when it is unloaded. This avoids enabling the interrupt
on unsupported CPUs which may result in spurious NMIs.

Reported by: rnoland
Reviewed by: jkoshy
Approved by: re (kib)
MFC after: 2 weeks

- Add support for nehalem/corei7 cpus. This supports all of the core
counters defined in the reference manual. It does not support the
'uncore' events.

Reviewed by: jkoshy
Sponsored by: Nokia

Bug fixes:
- Initialize variables before use.
- Remove a KASSERT() that could falsely trigger if there are other sources
of NMIs in the system.

Efficiency tweak:
- When checking PMCs that overflowed, ignore PMCs that were not configured for
sampling.

Fixes for Core2 Extreme support.

Submitted by: "Artem Belevich" <artemb at gmail dot com>

- Add support for PMCs in Intel CPUs of Family 6, model 0xE (Core Solo
and Core Duo), models 0xF (Core2), model 0x17 (Core2Extreme) and
model 0x1C (Atom).

In these CPUs, the actual numbers, kinds and widths of PMCs present
need to queried at run time. Support for specific "architectural"
events also needs to be queried at run time.

Model 0xE CPUs support programmable PMCs, subsequent CPUs
additionally support "fixed-function" counters.

- Use event names that are close to vendor documentation, taking in
account that:
- events with identical semantics on two or more CPUs in this family
can have differing names in vendor documentation,
- identical vendor event names may map to differing events across
CPUs,
- each type of CPU supports a different subset of measurable
events.

Fixed-function and programmable counters both use the same vendor
names for events. The use of a class name prefix ("iaf-" or
"iap-" respectively) permits these to be distinguished.

- In libpmc, refactor pmc_name_of_event() into a public interface
and an internal helper function, for use by log handling code.

- Minor code tweaks: staticize a global, freshen a few comments.

Tested by: gnn