Copy head@r302406 to stable/11 as part of the 11.0-RELEASE cycle.
Prune svn:mergeinfo from the new branch, as nothing has been merged
here.

Additional commits post-branch will follow.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Add a new bus method to fetch device-specific CPU sets.

bus_get_cpus() returns a specified set of CPUs for a device. It accepts
an enum for the second parameter that indicates the type of cpuset to
request. Currently two valus are supported:

- LOCAL_CPUS (on x86 this returns all the CPUs in the package closest to
the device when DEVICE_NUMA is enabled)
- INTR_CPUS (like LOCAL_CPUS but only returns 1 SMT thread for each core)

For systems that do not support NUMA (or if it is not enabled in the kernel
config), LOCAL_CPUS fails with EINVAL. INTR_CPUS is mapped to 'all_cpus'
by default. The idea is that INTR_CPUS should always return a valid set.

Device drivers which want to use per-CPU interrupts should start using
INTR_CPUS instead of simply assigning interrupts to all available CPUs.
In the future we may wish to add tunables to control the policy of
INTR_CPUS (e.g. should it be local-only or global, should it ignore
SMT threads or not).

The x86 nexus driver exposes the internal set of interrupt CPUs from the
the x86 interrupt code via INTR_CPUS.

The ACPI bus driver and PCI bridge drivers use _PXM to return a suitable
LOCAL_CPUS set when _PXM exists and DEVICE_NUMA is enabled. They also and
the global INTR_CPUS set from the nexus driver with the per-domain set from
_PXM to generate a local INTR_CPUS set for child devices.

Compared to the r298933, this version uses 'struct _cpuset' in
<sys/bus.h> instead of 'cpuset_t' to avoid requiring <sys/param.h>
(<sys/_cpuset.h> still requires <sys/param.h> for MAXCPU even though
<sys/_bitset.h> does not after recent changes).

Revert bus_get_cpus() for now.

I really thought I had run this through the tinderbox before committing,
but many places need <sys/types.h> -> <sys/param.h> for <sys/bus.h> now.

Add a new bus method to fetch device-specific CPU sets.

bus_get_cpus() returns a specified set of CPUs for a device. It accepts
an enum for the second parameter that indicates the type of cpuset to
request. Currently two valus are supported:

- LOCAL_CPUS (on x86 this returns all the CPUs in the package closest to
the device when DEVICE_NUMA is enabled)
- INTR_CPUS (like LOCAL_CPUS but only returns 1 SMT thread for each core)

For systems that do not support NUMA (or if it is not enabled in the kernel
config), LOCAL_CPUS fails with EINVAL. INTR_CPUS is mapped to 'all_cpus'
by default. The idea is that INTR_CPUS should always return a valid set.

Device drivers which want to use per-CPU interrupts should start using
INTR_CPUS instead of simply assigning interrupts to all available CPUs.
In the future we may wish to add tunables to control the policy of
INTR_CPUS (e.g. should it be local-only or global, should it ignore
SMT threads or not).

The x86 nexus driver exposes the internal set of interrupt CPUs from the
the x86 interrupt code via INTR_CPUS.

The ACPI bus driver and PCI bridge drivers use _PXM to return a suitable
LOCAL_CPUS set when _PXM exists and DEVICE_NUMA is enabled. They also and
the global INTR_CPUS set from the nexus driver with the per-domain set from
_PXM to generate a local INTR_CPUS set for child devices.

Reviewed by: wblock (manpage)
Differential Revision: https://reviews.freebsd.org/D5519

Add a pcib_attach_child() method to manage adding the child "pci" device.

This allows the PCI-PCI bridge driver to save a reference to the child
device in its softc.

Note that this required moving the "pci" device creation out of
acpi_pcib_attach(). Instead, acpi_pcib_attach() is renamed to
acpi_pcib_fetch_prt() as it's sole action now is to fetch the PCI
interrupt routing table.

Differential Revision: https://reviews.freebsd.org/D6021

Add a new method to the PCI bridge interface, PCIB_POWER_FOR_SLEEP(). This
method is used by the PCI bus driver to query the power management system
to determine the proper device state to be used for a device during suspend
and resume. For the ACPI PCI bridge drivers this calls
acpi_device_pwr_for_sleep(). This removes ACPI-specific knowledge from
the PCI and PCI-PCI bridge drivers.

Reviewed by: jkim

- Retire acpi_pcib_resume(). It is has just been an alias for
bus_generic_resume() since the pci_link(4) driver was added.
- Change the ACPI PCI-PCI bridge driver to inherit most of its methods
from the generic PCI-PCI bridge driver. In particular, this will now
restore PCI config registers for ACPI PCI-PCI bridges.

Tested by: Oleg Sharoyko osharoiko of gmail

Protect against multiple includes and use _KERNEL to protect the PCI fns.

- Remove some no longer used constants.
- Sort function prototypes.

Rework the ACPI PCI link code.
- Use a new-bus device driver for the ACPI PCI link devices. The devices
are called pci_linkX. The driver includes suspend/resume support so that
the ACPI bridge drivers no longer have to poke the links to get them
to handle suspend/resume. Also, the code to handle which IRQs a link is
routed to and choosing an IRQ when a link is not already routed is all
contained in the link driver. The PCI bridge drivers now ask the link
driver which IRQ to use once they determine that a _PRT entry does not
use a hardwired interrupt number.
- The new link driver includes support for multiple IRQ resources per
link device as well as preserving any non-IRQ resources when adjusting
the IRQ that a link is routed to.
- The entire approach to routing when using a link device is now
link-centric rather than pci bus/device/pin specific. Thus, when
using a tunable to override the default IRQ settings, one now uses
a single tunable to route an entire link rather than routing a single
device that uses the link (which has great foot-shooting potential if
the user tries to route the same link to two different IRQs using two
different pci bus/device/pin hints). For example, to adjust the IRQ
that \_SB_.LNKA uses, one would set 'hw.pci.link.LNKA.irq=10' from the
loader.
- As a side effect of having the link driver, unused link devices will now
be disabled when they are probed.
- The algorithm for choosing an IRQ for a link that doesn't already have an
IRQ assigned is now much closer to the one used in $PIR routing. When a
link is routed via an ISA IRQ, only known-good IRQs that the BIOS has
already used are used for routing instead of using probabilities to
guess at which IRQs are probably not used by an ISA device. One change
from $PIR is that the SCI is always considered a viable ISA IRQ, so that
if the BIOS does not setup any IRQs the kernel will degenerate to routing
all interrupts over the SCI. For non ISA IRQs, interrupts are picked
from the possible pool using a simplistic weighting algorithm.

Tested by: ru, scottl, others on acpi@
Reviewed by: njl

Re-work ACPI PCI IRQ routing (_PRT, link devices). The old approach was
incomplete in that the PRT routing was not aware of link programming.
Fix this by doing all routing through the link devices. The new algorithm
for setting up links is:

1. Read _CRS to get current setting. If invalid (not in _PRS), then set
to 0.
2. Attempt to call _DIS on the link. If successful, mark the link as not
routed. Otherwise, assume it still is.

Then when a routing request occurs:

3. Update weights for all IRQs
4. Attempt to route the initial IRQ if valid
5. If that fails, walk through the sorted list, attempting to route IRQs.
6. Configure the trigger/polarity based on _PRS.

Other changes:
* Add acpi_pci_find_prt() to look up the PRT entry for a given device and
acpi_pci_link_route() to select/route the best IRQ for it.
* Remove duplicated code in acpi_pcib_route_interrupt() that picked the
first IRQ from _PRS.
* Remove unneeded arguments from acpi_pcib_resume() and friends.
* Ignore _STA on link devices but report if it seems strange.
* Add a prt_source handle to the PRT structure since the ACPI struct
ACPI_PCI_ROUTING_TABLE uses a fixed-size entry for it. We'll need to
dynamically size this object if we want to use it the same way ACPI-CA
does. Null-terminate the source.

Tested by: Luo Hong <luohong99_at_mails.tsinghua.edu.cn>,
Jeffrey Katcher <jmkatcher_at_yahoo.com>
Info from: jhb, Len Brown (Intel)

Add code for ACPI PCI link object manipulation.
This allocate the best IRQ to boot-disable devices (have IRQ 0).
Allocated IRQ will be used for PCI interrupt routing when ACPI is
enabled.

Note that verbose messaging enabled for the time being so that
people can easily notice the strange behavior if it happened.

Overhaul the ACPI PCI bridge driver a bit:
- Add an ACPI PCI-PCI bridge driver (the previous driver just handled
Host-PCI bridges) that is a PCI driver that is a subclass of the generic
PCI-PCI bridge driver. It overrides probe, attach, read_ivar, and
pci_route_interrupt.
- The probe routine only succeeds if our parent is an ACPI PCI bus which
we test for by seeing if we can read our ACPI_HANDLE as an ivar.
- The attach routine saves a copy of our handle and calls the new
acpi_pcib_attach_common() function described below.
- The read_ivar routine handles normal PCI-PCI bridge ivars and adds an
ivar to return the ACPI_HANDLE of the bus this bridge represents.
- The route_interrupt routine fetches the _PRT (PCI Interrupt Routing
Table) from the bridge device's softc and passes it off to
acpi_pcib_route_interrupt() to route the interrupt.
- Split the old ACPI Host-PCI bridge driver into two pieces. Part of
the attach routine and most of the route_interrupt routine remain in
acpi_pcib.c and are shared by both ACPI PCI bridge drivers.
- The attach routine verifies the PCI bridge is present, reads in
the _PRT for the bridge, and attaches the child PCI bus.
- The route_interrupt routine uses the passed in _PRT to route a PCI
interrupt.
The rest of the driver is the ACPI Host-PCI bridge specific bits that
live in acpi_pcib_acpi.c.
- We no longer duplicate pcib_maxslots but use it directly.
- The driver now uses the pcib devclass instead of its own devclass.
This means that PCI busses are now only children of pcib devices.
- Allow the ACPI_HANDLE for the child PCI bus to be read as an ivar
of the child bus.
- Fetch the _PRT for routing PCI interrupts directly from our softc
instead of walking the devclass to find ourself and then fetch our
own softc.

With this change and the new ACPI PCI bus driver, ACPI can now properly
route interrupts for devices behind PCI-PCI bridges. That is, the
Itanium2 with like 10 PCI busses can now boot ok and route all the PCI
interrupts. Hopefully this will also fix problems people are having with
CardBus bridges behind PCI-PCI bridges not properly routing interrupts
when ACPI is used.

Tested on: i386, ia64