MFC 302847, 302848, 302852, 302853:
Remove the remaining non-INTRNG support from the arm64 code.

Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Mark the ThunderX and generic PCI drivers as cache-coherent when we know
this to be the case. This will mean we don't try and handle the cache in
bus_dmamap_sync when it is not needed.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6605

Handle PCI_RES_BUS on the generic and ThunderX PCIe drivers. This has been
tested on the Pass 1.1 and 2.0 ThunderX machines in the Netperf cluster.

Reviewed by: jhb
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6453

Teach the ThunderX PCI PEM driver about intrng. This will be used later
when arm64 is supported by intrng.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Add a pcib interface for use by interrupt controllers that need to
translate the pci rid to a controller ID. The translation could be based
on the 'msi-map' OFW property, a similar ACPI option, or hard-coded for
hardware lacking the above options.

Reviewed by: wma
Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Push the logic to talk with the MSI/MSI-X interrupt controller to the FDT
attachment. This is where it will live when we import intrng as it will
need to look at either the msi-parent or msi-map FDT properties.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Group the ThunderX PCIe PEM newbus methods to help find them.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Pull out the MSI/MSI-X handling calls to simplify future intrng
integration.

Obtained from: ABT Systems Ltd
Sponsored by: The FreeBSD Foundation

Cleanup unnecessary semicolons from the kernel.

Found with devel/coccinelle.

Improve ThunderX PEM driver to work on pass2 revision

Things changed:
* do not allocate 4GB of SLI space, because it's the waste of
system resources. Allocate only small portions when needed.
* provide own implementation of activate_resource which performs
address translation between PCI bus and host PA address space.
This is temporary solution, should be replaced by bus_map_resource
once implemented.

Obtained from: Semihalf
Sponsored by: Cavium
Approved by: cognet (mentor)
Reviewed by: jhb
Differential revision: https://reviews.freebsd.org/D5294

Restore ThunderX Pass1.1 PCI changes removed by r295962

If Enhanced Allocation is not used, we can't allocate any random
range. All internal devices have hardcoded place where they can
be located within PCI address space. Fortunately, we can read
this value from BAR.

Obtained from: Semihalf
Sponsored by: Cavium
Approved by: cognet (mentor)
Reviewed by: zbb
Differential revision: https://reviews.freebsd.org/D5455

Make pci_host_generic and thunderx_pci common
* provided OFW interface for pci_host_generic (for handling devices which are present in DTS under the PCI node)
* removed support for internal PCI from arm64/cavium
* cleaned up and made most of the code common

Obtained from: Semihalf
Sponsored by: Cavium
Approved by: cognet (mentor)
Reviewed by: zbb
Differential revision: https://reviews.freebsd.org/D5261

Introduce a RMAN_IS_DEFAULT_RANGE() macro, and use it.

This simplifies checking for default resource range for bus_alloc_resource(),
and improves readability.

This is part of, and related to, the migration of rman_res_t from u_long to
uintmax_t.

Discussed with: jhb
Suggested by: marcel

Support PEM that is not a PCI endpoint on ThunderX

Some chip revisions don't have their external PCIe buses
behind the internal bridge. Add support for FDT-configurable
PEMs but keep ability for PCIe enumeration.

Reviewed by: andrew, wma
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5285

Fix VNIC enumeration after r294993 and r294990

ofw_bus_get_node() must be tested against negative values since
missing parent bus method will result in calling the default method
which simply returns (-1): sys/dev/ofw/ofw_bus_if.m
This was lost in the review process.

Obtained from: Semihalf
Sponsored by: Cavium

Add FDT bus capabilities to ThunderX PCI driver

New ThunderX firmware incorporates modified DTB that presents
different device hierarchy. In the new device tree, MDIO
devices are below two additional buses that oddly hang on
PCI bridge.

Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5069

Correct alloc_ and release_resource methods in thunder_pcie driver

- Avoid using BUS_ macros as bus_generic_ functions should be used instead.
- Fix mistaken device_t pointers in thunder_pcie_alloc_resource.
Should use dev->parent method and allocate resource for child device

Reviewed by: wma
Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5068

Divide ThunderX PCIe driver to general and FDT part

- Separate FDT and general PCIe driver parts
- Drop some irrelevant printfs that cannot be displayed in
FDT attach
- Move ranges parsing to FDT portion of PCIe code

Obtained from: Semihalf
Sponsored by: Cavium
Differential Revision: https://reviews.freebsd.org/D5067

Convert rman to use rman_res_t instead of u_long

Summary:
Migrate to using the semi-opaque type rman_res_t to specify rman resources. For
now, this is still compatible with u_long.

This is step one in migrating rman to use uintmax_t for resources instead of
u_long.

Going forward, this could feasibly be used to specify architecture-specific
definitions of resource ranges, rather than baking a specific integer type into
the API.

This change has been broken out to facilitate MFC'ing drivers back to 10 without
breaking ABI.

Reviewed By: jhb
Sponsored by: Alex Perez/Inertial Computing
Differential Revision: https://reviews.freebsd.org/D5075

Update ThunderX PCIe driver to fit new DTS layout

In recent EFI the DTS entries changed for PCIe controller.
This commit fixes internal PCIe, external is yet TBD.

Submitted by: Dominik Ermel <der@semihalf.com>
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D4976

Reduce the numbers of levels of indirection in arm64 pcib drivers by making
the MSI & MSI-X handler functions be directly callable from the driver
methods.

Sponsored by: ABT Systems Ltd

Add support for unspecified ranges on ThunderX system

When one tries to allocate a resource with unspecified range,
read already configured BAR values (by UEFI or whatever).
This is necessary to make VNIC VFs working and to allow them to be
properly allocated.

Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D3752

arm64 ThunderX PCIe workaround: enumerate only one slot for now

Otherwise an em(4) NIC is detected 32 times.

Submitted by: wma@semihalf.com
Obtained from: Semihalf
Differential Revision: https://reviews.freebsd.org/D3706

Add support for external PCIe (PEM) on Cavium's ThunderX

Reviewed by: jhb
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3257

Introduce support for internal PCIe for Cavium's ThunderX

This driver supports internal PCIe Root Complex on
Cavium ThunderX Pass 1.1 hardware.

Reviewed by: andrew, jhb
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3031