Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r240650: hwpmc amd_pcpu_fini: fix a bug in code locked under DEBUG

MFC r233628, r234598, r235229, r235831, r226986.

Add software PMC support.

New kernel events can be added at various location for sampling or counting.
This will for example allow easy system profiling whatever the processor is
with known tools like pmcstat(8).

Simultaneous usage of software PMC and hardware PMC is possible, for example
looking at the lock acquire failure, page fault while sampling on
instructions.

Sponsored by: NETASQ

MFC r229076:

In sys/dev/hwpmc/hwpmc_amd.c, fix a clang warning about invalid enum
conversions.

Reviewed by: jkoshy

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

- Efficiency tweak: when checking for PMC overflows, only go to
hardware for PMCs that have been configured for sampling.

- Bug fix: acknowledge PMC hardware overflows irrespective of the
the (software) PMC's state.

Correct an indexing error (a change missed out in #184802).

- Separate PMC class dependent code from other kinds of machine
dependencies. A 'struct pmc_classdep' structure describes operations
on PMCs; 'struct pmc_mdep' contains one or more 'struct pmc_classdep'
structures depending on the CPU in question.

Inside PMC class dependent code, row indices are relative to the
PMCs supported by the PMC class; MI code in "hwpmc_mod.c" translates
global row indices before invoking class dependent operations.

- Augment the OP_GETCPUINFO request with the number of PMCs present
in a PMC class.

- Move code common to Intel CPUs to file "hwpmc_intel.c".

- Move TSC handling to file "hwpmc_tsc.c".

Retire the MALLOC and FREE macros. They are an abomination unto style(9).

MFC after: 3 months

Support sparsely numbered CPUs.

Requested by: obrien, alfred (long ago)

Sign-extend the 48-bit AMD PMC counter before treating it to a 64-bit
2's compliment.

The 2's compliment transform is done so a "count down" sampling interval
can be converted into a "count up" PMC value. a 2's complimented 'count down'
value is written to the PMC counter; then the read-back counter is reverted
via another 2's compliment.

PR: kern/121660
Reviewed by: jkoshy
Approved by: jkoshy
MFC after: 1 week

Fix the debugging output - the '0x' was duplicated from the %p option.

Kernel and hwpmc(4) support for callchain capture.

Sponsored by: FreeBSD Foundation and Google Inc.

Fix -Wundef warnings found when compiling i386 LINT, GENERIC and
custom kernels.

Process one NMI interrupt per handler invocation as the processor
'buffers' pending NMIs from multiple interrupting PMCs and delivers
them serially.

Reported by: Olivier Crameri <olivier.crameri@epfl.ch>
MFC after: 3 days

Re-enable sampling on the AMD64.

Turn off sampling modes on the AMD64 till the time I can track down
the reason for the double fault seen when sampling under load.

MFC after: 3 days

sys/dev/hwpmc/hwpmc_{amd,piv,ppro}.c:
- Update driver interrupt statistics correctly.

sys/sys/pmc.h, sys/dev/hwpmc/hwpmc_mod.c:
- Fix a bug affecting debug printfs.
- Move the 'stalled' flag from being in a bit in the
'pm_flags' field of a 'struct pmc' to a field of its own in the
same structure. This flag is updated from the NMI handler and
keeping it separate makes it easier to avoid races with other
parts of the code.

sys/dev/hwpmc/hwpmc_logging.c:
- Do arithmetic with 'uintptr_t' types rather that casting
to and from 'char *'.

Approved by: re (scottl)

Fix a -Wuninitialized warning reported by rwatson.

Approved by: re (blanket hwpmc)

MFP4:

- Implement sampling modes and logging support in hwpmc(4).

- Separate MI and MD parts of hwpmc(4) and allow sharing of
PMC implementations across different architectures.
Add support for P4 (EMT64) style PMCs to the amd64 code.

- New pmcstat(8) options: -E (exit time counts) -W (counts
every context switch), -R (print log file).

- pmc(3) API changes, improve our ability to keep ABI compatibility
in the future. Add more 'alias' names for commonly used events.

- bug fixes & documentation.

Kernel hooks to support PMC sampling modes.

Reviewed by: alc

Add convenience APIs pmc_width() and pmc_capabilities() to -lpmc.
Have pmcstat(8) and pmccontrol(8) use these APIs.

Return PMC class-related constants (PMC widths and capabilities)
with the OP GETCPUINFO call leaving OP PMCINFO to return only the
dynamic information associated with a PMC (i.e., whether enabled,
owner pid, reload count etc.).

Allow pmc_read() (i.e., OPS PMCRW) on active self-attached PMCs to
get upto-date values from hardware since we can guarantee that the
hardware is running the correct PMC at the time of the call.

Bug fixes:
- (x86 class processors) Fix a bug that prevented an RDPMC
instruction from being recognized as permitted till after the
attached process had context switched out and back in again after
a pmc_start() call.

Tighten the rules for using RDPMC class instructions: a GETMSR
OP is now allowed only after an OP ATTACH has been done by the
PMC's owner to itself. OP GETMSR is not allowed for PMCs that
track descendants, for PMCs attached to processes other than
their owner processes.

- (P4/HTT processors only) Fix a bug that caused the MI and MD
layers to get out of sync. Add a new MD operation 'get_config()'
as part of this fix.

- Allow multiple system-mode PMCs at the same row-index but on
different CPUs to be allocated.

- Reject allocation of an administratively disabled PMC.

Misc. code cleanups and refactoring. Improve a few comments.

Return the correct register number in the 'get_msr()' MD function.

Only allow a process to use the x86 RDPMC instruction if it has
allocated and attached a PMC to itself.

Inform the MD layer of the "pseudo context switch out" that needs
to be done when the last thread of a process is exiting.

Forced commit to note the move from "src/sys/hwpmc" to "src/sys/dev/hwpmc".

Include <sys/pmc.h> instead of <machine/pmc_mdep.h>. The MI header
includes the MD header for us. Do not include <machine/specialreg.h>
as it is not a header file that can be included from MI files. It
is included from <machine/pmc_mdep.h> if so needed and possible.

Ok'd: jkoshy@

Bring a working snapshot of hwpmc(4), its associated libraries, userland utilities
and documentation into -CURRENT.

Bump FreeBSD_version.

Reviewed by: alc, jhb (kernel changes)