Partially revert r284034. In particular, revert the final change in this
MFC (281874). It broke suspend and resume on several Thinkpads (though not
all) in 10 even though it works fine on the same laptops in HEAD.

PR: 201239
Reported by: Kevin Oberman and several others

MFC 274633,274639,274663,277233-277235,281870,281871,281873,281874:
Various fixes for suspend and resume of PCI to PCI and PCI to Cardbus
bridges.

274633:

Remove stray empty comment. The code is adequately explained in the
block comment above, so there's nothing to add here.

274639:

Modernize comments about BIOSes being lame since in this detail they
aren't lame, the rules changed along the way. Catch up to 1999 or so
with the new rules.

274663:

Fix typo pointed out by avg@ and Joerg Sonnenberger. Add a clarifying
sentence too.

277233:

Suspend and resume were the only two functions not to follow the brdev
convention here, so fix that.

277234:

Move the suspsned and resume functions to the bus attachment. They
were accessing PCI config registers, which won't work for the ISA
version.

277235:

Always enable I/O, memory and dma cycles. Some BIOSes don't enable
them, sometimes they are reset for power state transitions or during
whatever happens while suspended. Also, it is good practice to always
do this.

281870:

Cosmetic change: use PCIR_SECLAT_2 rather than PCIR_SECLAT_1.

281871:

The minimim grant and maximum latency PCI config registers are only valid
for type 0 devices, not type 1 or 2 bridges. Don't read them for bridge
devices during bus scans and return an error when attempting to read them
as ivars for bridge devices.

281873:

Don't explicitly manage power states for PCI-PCI bridge devices in the
driver's suspend and resume routines. These have been redundant no-ops
since r214065 changed the PCI bus driver to manage power states for
all devices (including type 1/2 bridge devices) during suspend and resume.

281874:

Update the pci_cfg_save/restore routines to operate on bridge devices
(type 1 and type 2) as well as leaf devices (type 0). In particular,
this allows the existing PCI bus logic to save and restore capability
registers such as MSI and PCI-express work for bridge devices rather than
requiring that code to be duplicated in bridge drivers. It also means
that bridge drivers no longer need to save and restore basic registers
such as the PCI command register or BARs nor manage powerstates for the
bridge device.

While here, pci_setup_secbus() has been changed to initialize the 'sec'
and 'sub' fields in the 'secbus' structure instead of requiring the pcib
and pccbb drivers to do this in the NEW_PCIB + PCI_RES_BUS case.

MFC 261790:
Add support for managing PCI bus numbers. As with BARs and PCI-PCI bridge
I/O windows, the default is to preserve the firmware-assigned resources.
PCI bus numbers are only managed if NEW_PCIB is enabled and the architecture
defines a PCI_RES_BUS resource type.
- Add a helper API to create top-level PCI bus resource managers for each
PCI domain/segment. Host-PCI bridge drivers use this API to allocate
bus numbers from their associated domain.
- Change the PCI bus and CardBus drivers to allocate a bus resource for
their bus number from the parent PCI bridge device.
- Change the PCI-PCI and PCI-CardBus bridge drivers to allocate the
full range of bus numbers from secbus to subbus from their parent bridge.
The drivers also always program their primary bus register. The bridge
drivers also support growing their bus range by extending the bus resource
and updating subbus to match the larger range.
- Add support for managing PCI bus resources to the Host-PCI bridge drivers
used for amd64 and i386 (acpi_pcib, mptable_pcib, legacy_pcib, and qpi_pcib).
- Define a PCI_RES_BUS resource type for amd64 and i386.

PR: 197076

MFC r264007,r264008,r264009,r264011,r264012,r264013

MFC support for PCI Alternate RID Interpretation. ARI is an optional PCIe
feature that allows PCI devices to present up to 256 functions on a bus.
This is effectively a prerequisite for PCI SR-IOV support.

r264007:
Add a method to get the PCI RID for a device.

Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264008:
Re-implement the DMAR I/O MMU code in terms of PCI RIDs

Under the hood the VT-d spec is really implemented in terms of
PCI RIDs instead of bus/slot/function, even though the spec makes
pains to convert back to bus/slot/function in examples. However
working with bus/slot/function is not correct when PCI ARI is
in use, so convert to using RIDs in most cases. bus/slot/function
will only be used when reporting errors to a user.

Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264009:
Re-write bhyve's I/O MMU handling in terms of PCI RID.

Reviewed by: neel
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264011:
Add support for PCIe ARI

PCIe Alternate RID Interpretation (ARI) is an optional feature that
allows devices to have up to 256 different functions. It is
implemented by always setting the PCI slot number to 0 and
re-purposing the 5 bits used to encode the slot number to instead
contain the function number. Combined with the original 3 bits
allocated for the function number, this allows for 256 functions.

This is enabled by default, but it's expected to be a no-op on currently
supported hardware. It's a prerequisite for supporting PCI SR-IOV, and
I want the ARI support to go in early to help shake out any bugs in it.
ARI can be disabled by setting the tunable hw.pci.enable_ari=0.

Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264012:
Print status of ARI capability in pciconf -c

Teach pciconf how to print out the status (enabled/disabled) of the ARI
capability on PCI Root Complexes and Downstream Ports.

MFC after: 2 months
Sponsored by: Sandvine Inc.

r264013:
Add missing copyright date.

MFC after: 2 months

Partially revert r284034. In particular, revert the final change in this
MFC (281874). It broke suspend and resume on several Thinkpads (though not
all) in 10 even though it works fine on the same laptops in HEAD.

PR: 201239
Reported by: Kevin Oberman and several others

MFC 274633,274639,274663,277233-277235,281870,281871,281873,281874:
Various fixes for suspend and resume of PCI to PCI and PCI to Cardbus
bridges.

274633:

Remove stray empty comment. The code is adequately explained in the
block comment above, so there's nothing to add here.

274639:

Modernize comments about BIOSes being lame since in this detail they
aren't lame, the rules changed along the way. Catch up to 1999 or so
with the new rules.

274663:

Fix typo pointed out by avg@ and Joerg Sonnenberger. Add a clarifying
sentence too.

277233:

Suspend and resume were the only two functions not to follow the brdev
convention here, so fix that.

277234:

Move the suspsned and resume functions to the bus attachment. They
were accessing PCI config registers, which won't work for the ISA
version.

277235:

Always enable I/O, memory and dma cycles. Some BIOSes don't enable
them, sometimes they are reset for power state transitions or during
whatever happens while suspended. Also, it is good practice to always
do this.

281870:

Cosmetic change: use PCIR_SECLAT_2 rather than PCIR_SECLAT_1.

281871:

The minimim grant and maximum latency PCI config registers are only valid
for type 0 devices, not type 1 or 2 bridges. Don't read them for bridge
devices during bus scans and return an error when attempting to read them
as ivars for bridge devices.

281873:

Don't explicitly manage power states for PCI-PCI bridge devices in the
driver's suspend and resume routines. These have been redundant no-ops
since r214065 changed the PCI bus driver to manage power states for
all devices (including type 1/2 bridge devices) during suspend and resume.

281874:

Update the pci_cfg_save/restore routines to operate on bridge devices
(type 1 and type 2) as well as leaf devices (type 0). In particular,
this allows the existing PCI bus logic to save and restore capability
registers such as MSI and PCI-express work for bridge devices rather than
requiring that code to be duplicated in bridge drivers. It also means
that bridge drivers no longer need to save and restore basic registers
such as the PCI command register or BARs nor manage powerstates for the
bridge device.

While here, pci_setup_secbus() has been changed to initialize the 'sec'
and 'sub' fields in the 'secbus' structure instead of requiring the pcib
and pccbb drivers to do this in the NEW_PCIB + PCI_RES_BUS case.

MFC 261790:
Add support for managing PCI bus numbers. As with BARs and PCI-PCI bridge
I/O windows, the default is to preserve the firmware-assigned resources.
PCI bus numbers are only managed if NEW_PCIB is enabled and the architecture
defines a PCI_RES_BUS resource type.
- Add a helper API to create top-level PCI bus resource managers for each
PCI domain/segment. Host-PCI bridge drivers use this API to allocate
bus numbers from their associated domain.
- Change the PCI bus and CardBus drivers to allocate a bus resource for
their bus number from the parent PCI bridge device.
- Change the PCI-PCI and PCI-CardBus bridge drivers to allocate the
full range of bus numbers from secbus to subbus from their parent bridge.
The drivers also always program their primary bus register. The bridge
drivers also support growing their bus range by extending the bus resource
and updating subbus to match the larger range.
- Add support for managing PCI bus resources to the Host-PCI bridge drivers
used for amd64 and i386 (acpi_pcib, mptable_pcib, legacy_pcib, and qpi_pcib).
- Define a PCI_RES_BUS resource type for amd64 and i386.

PR: 197076

MFC r264007,r264008,r264009,r264011,r264012,r264013

MFC support for PCI Alternate RID Interpretation. ARI is an optional PCIe
feature that allows PCI devices to present up to 256 functions on a bus.
This is effectively a prerequisite for PCI SR-IOV support.

r264007:
Add a method to get the PCI RID for a device.

Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264008:
Re-implement the DMAR I/O MMU code in terms of PCI RIDs

Under the hood the VT-d spec is really implemented in terms of
PCI RIDs instead of bus/slot/function, even though the spec makes
pains to convert back to bus/slot/function in examples. However
working with bus/slot/function is not correct when PCI ARI is
in use, so convert to using RIDs in most cases. bus/slot/function
will only be used when reporting errors to a user.

Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264009:
Re-write bhyve's I/O MMU handling in terms of PCI RID.

Reviewed by: neel
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264011:
Add support for PCIe ARI

PCIe Alternate RID Interpretation (ARI) is an optional feature that
allows devices to have up to 256 different functions. It is
implemented by always setting the PCI slot number to 0 and
re-purposing the 5 bits used to encode the slot number to instead
contain the function number. Combined with the original 3 bits
allocated for the function number, this allows for 256 functions.

This is enabled by default, but it's expected to be a no-op on currently
supported hardware. It's a prerequisite for supporting PCI SR-IOV, and
I want the ARI support to go in early to help shake out any bugs in it.
ARI can be disabled by setting the tunable hw.pci.enable_ari=0.

Reviewed by: kib
MFC after: 2 months
Sponsored by: Sandvine Inc.

r264012:
Print status of ARI capability in pciconf -c

Teach pciconf how to print out the status (enabled/disabled) of the ARI
capability on PCI Root Complexes and Downstream Ports.

MFC after: 2 months
Sponsored by: Sandvine Inc.

r264013:
Add missing copyright date.

MFC after: 2 months