sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

powerpc powernv: Remove unused devclass arguments to DRIVER_MODULE.

Rename ofwpci.c to ofw_pcib.c

It's a class0 driver that implements some pcib methods and creates
a pci bus as its children.
The "ofw_pci" name will be used by a new driver that will be a subclass
of the pci bus.
No functional changes intended.

Submitted by: Kornel Duleba <mindal@semihalf.com>
Reviewed by: andrew
Obtained from: Semihalf
Sponsored by: Alstom Group
Differential Revision: https://reviews.freebsd.org/D30226

[PowerPC64LE] Get XIVE up and running.

More endian conversion.

* Install TCEs correctly (i.e. in big endian)

* Convert to big endian and back when setting up queue pages and IRQs.

Sponsored by: Tag1 Consulting, Inc.

[PowerPC64LE] Endian fixes for opal_pci.c.

Since OPAL runs in big endian, any data being passed back and forth
via memory instead of registers needs to be byteswapped.

From my notes during development:

"A good way to find candidates is to look for vtophys() in opal_call()
parameters. The memory being passed will be written into in BE."

Sponsored by: Tag1 Consulting, Inc.

powerpc: clean up empty lines in .c and .h files

powerpc/powernv: Don't use the vmem quantum cache for OPAL PCI MSI allocations

vmem quantum cache is only needed when doing a lot of concurrent allocations,
which doesn't happen when allocating MSIs. This wastes memory for the cache
zones. Avoid this waste and don't use the quantum cache.

Reported by: markj

powerpc64/powernv: Use OPAL call for non-POWER8 PCI TCE reset

According to the OPAL documentation, only the POWER8 (PHB3) should use
the register write TCE reset method. All others should use the OPAL
call.

On POWER9 the call is semantically identical to the register write, with
a wait for completion.

powerpc/powernv: Fix OPAL cfgread/cfgwrite error handling

Freeze clearing needs to heppen any time OPAL reads return either an error
(except OPAL_HARDWARE), AND any time it returns 0xff for all bytes.

For cfgwrite, any error that's not OPAL_HARDWARE should be cleaned up.

powerpc/powernv: Only clear EEH freeze for some errors

Only clear an EEH freeze if an error occurs. However, if an OPAL_HARDWARE
error is returned, this indicates a hardware failure which cannot be
unfrozen, and instead needs a hardware reset. Attempting to unfreeze a
broken PCH will result in console spam for each attempt. To avoid the spam,
just don't do it.

Add NUMA support to powerpc

Summary:
Initial NUMA support:
- associate CPU with domain
- associate memory ranges with domain
- identify domain for devices
- limit device interrupt binding to appropriate domain

- Additionally fixes a bug in the setting of Maxmem which led to
only memory attached to the first socket being enabled for DMA

A pmap variant can opt in to numa support by by calling `numa_mem_regions`
at the end of pmap_bootstrap - registering the corresponding ranges with the
VM.

This yields a ~20% improvement in build times of llvm on dual socket POWER9
over non-NUMA.

Original patch by mmacy.

Differential Revision: https://reviews.freebsd.org/D17933

powerpc: Add opaque 'private data' to interrupt vectors

The XICS and XIVE need extra data beyond irq and vector. Rather than
performing a separate search, it's better for the general interrupt facility
to hold a private pointer, since the search already must be done anyway at
that level.

powerpc/powernv: Restrict the busdma tag to only POWER8

It seems this tag is causing problems on POWER9 systems. Since no POWER9 user
has encountered the problem fixed by r339589 just restrict it to POWER8 for now.
A better fix will likely be to update powerpc/busdma_machdep.c to handle the
window correctly.

Reported by: mmacy, others

ppc64: limited 32-bit DMA address range

Further investigation of issues with 32-bit DMA on PowerNV revealed that
its window is hardcoded by OPAL (at least in skiboot version 5.4.9) and
cannot be changed by the OS.
Thus, now jhb suggestion of limiting the range in PCI DMA tag seems
the best way to deal with it.

Reviewed by: jhibbits, nwhitehorn, sbruno
Approved by: jhibbits(mentor)
Differential Revision: https://reviews.freebsd.org/D17601

powerpc64/powernv:opal_pci: Fix the alignment of the TCE table

The TCE table need only be aligned to the size of the table, not the size of
the TCE segment.

powerpc64/powernv: Don't mask MSIs in OPAL

Summary:
Discussing with Benjamin Herrenschmidt, MSIs, and edge-triggered
interrupts in general, must not be masked in XICS and XIVE, else
subsequent interrupts may be ignored.

Testing locally on my Talos II (single CPU, 18-core POWER9), NVMe now
works with MSI, improving read throughput by ~70% (900MB/s -> 1.67GB/s,
with 64MB block size) over INTx interrupts, and snd_hda(4) now will
actually play music with MSI. Previously, snd_hda(4) would not receive
interrupts, timing out, and declaring the channels dead.

This has also been tested by Kevin Bowling, and others, with great
success. Kevin reported NVMe unusable on his Talos II prior to this
patch.

Reviewed by: nwhitehorn, kbowling
Approved by: re(rgrimes)
Differential Revision: https://reviews.freebsd.org/D17356

powerpc64/powernv: Avoid type promotion

There is a type promotion that transform count = -1 into a unsigned int causing
the default TCE SEG SIZE not being returned on a Boston POWER9 machine.

This machine does not have the 'ibm,supported-tce-sizes' entries, thus, count
is set to -1, and the function continue to execute instead of returning.

Reviewed by: jhibbits, wma
Approved by: jhibbits (mentor)
Differential Revision: https://reviews.freebsd.org/D15763

Remove a debug printf from opal_pci driver

Make opal_pci driver work with POWER9

Summary:
Coupled with r334365, this makes PCI work on POWER9. There is still more to
do to fully exploit the hardware capabilities, but this is sufficient to
enable USB and ethernet controllers on a POWER9 Talos II system.

Reviewed by: nwhitehorn, leitao
Differential Revision: https://reviews.freebsd.org/D15566

Cache the phandle of the PCI node in opal_pci_attach

Simple cleanup, no functional change. This is related to the fixups needed
for POWER9 support.

powerpc64: Add OPAL definitions

Summary:
Add additional OPAL PCI definitions and expand the code to use them in order to
ease the OPAL interface process for new comers.

These definitions came directly from the OPAL code and they are the same for
both PHB3 (POWER8) and PHB4 (POWER9).

Submitted by: Breno Leitao
Differential Revision: https://reviews.freebsd.org/D15432

Fix use of unitialized variables.

PowerNV: send MSI_EOI always after MSI unmask

MSI/MSI-x interrupts are edge-triggered. If an interrupt
arrives when IRQ line is masked, it will be lost and will
never recover. Perform MSI_EOI always after unmask to give
a chance for PHB/XICS to send an interrupt again if MSI/MSI-x
pending bit is set in MSI/MSI-x BAR space.

Submitted by: Wojciech Macek <wma@semihalf.org>
Obtained from: Semihalf
Sponsored by: IBM, QCM Technologies

PowerNV: make PowerNV PCIe working on a real hardware

Fixes:
- map all devices to PE0
- use 1:1 TCE mapping
- provide the same TCE mapping for all PEs (not only PE0)
- add TCE reset and alignment (required by OPAL)

Created by: Wojciech Macek <wma@semihalf.com>
Obtained from: Semihalf
Sponsored by: QCM Technologies

PowerNV: initial support for PCIe host controller

Provide initial support for PCIe host controller as
well as for IOMMU mapping. This commit allows proper
bus enumeration, but does not guarantee DMA operations
are working.

Created by: Nathan Whitehorn <nwhitehorn@freebsd.org>
Submitted by: Wojciech Macek <wma@semihalf.com>
Sponsored by: FreeBSD Foundation