s390/pci: Use dma-iommu layer

While s390 already has a standard IOMMU driver and previous changes have
added I/O TLB flushing operations this driver is currently only used for
user-space PCI access such as vfio-pci. For the DMA API s390 instead
utilizes its own implementation in arch/s390/pci/pci_dma.c which drives
the same hardware and shares some code but requires a complex and
fragile hand over between DMA API and IOMMU API use of a device and
despite code sharing still leads to significant duplication and
maintenance effort. Let's utilize the common code DMAP API
implementation from drivers/iommu/dma-iommu.c instead allowing us to
get rid of arch/s390/pci/pci_dma.c.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Link: https://lore.kernel.org/r/20230928-dma_iommu-v13-3-9e5fc4dacc36@linux.ibm.com
Signed-off-by: Joerg Roedel <jroedel@suse.de>

s390/pci: clean up left over special treatment for function zero

Prior to commit 960ac3626487 ("s390/pci: allow zPCI zbus without
a function zero") enabling and scanning a PCI function had to
potentially be postponed until the function with devfn zero on that bus
was plugged. While the commit removed the waiting itself extra code to
scan all functions on the PCI bus once function zero appeared was
missed. Remove that code and the outdated comments about waiting for
function zero.

Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Link: https://lore.kernel.org/r/20230306151014.60913-5-schnelle@linux.ibm.com
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: remove redundant pci_bus_add_devices() on new bus

The pci_bus_add_devices() call in zpci_bus_create_pci_bus() is without
function since at this point no device could have been added to the
freshly created PCI bus.

Suggested-by: Bjorn Helgaas <helgaas@kernel.org>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Link: https://lore.kernel.org/r/20230306151014.60913-4-schnelle@linux.ibm.com
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: only add specific device in zpci_bus_scan_device()

As the name suggests zpci_bus_scan_device() is used to scan a specific
device and thus pci_bus_add_device() for that device is sufficient.
Furthermore move this call inside the rescan/remove locking.

Suggested-by: Bjorn Helgaas <helgaas@kernel.org>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Link: https://lore.kernel.org/r/20230306151014.60913-3-schnelle@linux.ibm.com
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

PCI: s390: Fix use-after-free of PCI resources with per-function hotplug

On s390 PCI functions may be hotplugged individually even when they
belong to a multi-function device. In particular on an SR-IOV device VFs
may be removed and later re-added.

In commit a50297cf8235 ("s390/pci: separate zbus creation from
scanning") it was missed however that struct pci_bus and struct
zpci_bus's resource list retained a reference to the PCI functions MMIO
resources even though those resources are released and freed on
hot-unplug. These stale resources may subsequently be claimed when the
PCI function re-appears resulting in use-after-free.

One idea of fixing this use-after-free in s390 specific code that was
investigated was to simply keep resources around from the moment a PCI
function first appeared until the whole virtual PCI bus created for
a multi-function device disappears. The problem with this however is
that due to the requirement of artificial MMIO addreesses (address
cookies) extra logic is then needed to keep the address cookies
compatible on re-plug. At the same time the MMIO resources semantically
belong to the PCI function so tying their lifecycle to the function
seems more logical.

Instead a simpler approach is to remove the resources of an individually
hot-unplugged PCI function from the PCI bus's resource list while
keeping the resources of other PCI functions on the PCI bus untouched.

This is done by introducing pci_bus_remove_resource() to remove an
individual resource. Similarly the resource also needs to be removed
from the struct zpci_bus's resource list. It turns out however, that
there is really no need to add the MMIO resources to the struct
zpci_bus's resource list at all and instead we can simply use the
zpci_bar_struct's resource pointer directly.

Fixes: a50297cf8235 ("s390/pci: separate zbus creation from scanning")
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Link: https://lore.kernel.org/r/20230306151014.60913-2-schnelle@linux.ibm.com
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: allow zPCI zbus without a function zero

Currently the zPCI code blocks PCI bus creation and probing of a zPCI zbus
unless there is a PCI function with devfn 0. This is always the case for
the PCI functions with hidden RID, but may keep PCI functions from a
multi-function PCI device with RID information invisible until the function
0 becomes visible. Worse, as a PCI bus is necessary to even present a PCI
hotplug slot, even that remains invisible.

With the probing of these so-called isolated PCI functions enabled for s390
in common code, this restriction is no longer necessary. On network cards
with multiple ports and a PF per port this also allows using each port on
its own while still providing the physical PCI topology information in the
devfn needed to associate VFs with their parent PF.

Link: https://lore.kernel.org/r/20220628143100.3228092-6-schnelle@linux.ibm.com
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>

s390/pci: improve DMA translation init and exit

Currently zpci_dma_init_device()/zpci_dma_exit_device() is called as
part of zpci_enable_device()/zpci_disable_device() and errors for
zpci_dma_exit_device() are always ignored even if we could abort.

Improve upon this by moving zpci_dma_exit_device() out of
zpci_disable_device() and check for errors whenever we have a way to
abort the current operation. Note that for example in
zpci_event_hard_deconfigured() the device is expected to be gone so we
really can't abort and proceed even in case of error.

Similarly move the cc == 3 special case out of zpci_unregister_ioat()
and into the callers allowing to abort when finding an already disabled
devices precludes proceeding with the operation.

While we are at it log IOAT register/unregister errors in the s390
debugfs log,

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: reset zdev->zbus on registration failure

On failure to register a struct zpci_dev with a struct zpci_bus we left
a dangling pointer in zdev->zbus. As zpci_create_device() bails if
zpci_bus_device_register() fails this is of no consequence but still bad
practice.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: separate zbus registration from scanning

Now that the zbus can be created without being scanned we can go one
step further and make registering a device to a zbus independent from
scanning it. This way the zbus handling becomes much more natural
in that functions can be registered on the zbus to be scanned later more
closely resembling the handling of both real PCI hardware and other
virtual PCI busses like Hyper-V's virtual PCI bus (see for example
drivers/pci/controller/pci-hyperv.c:create_root_hv_pci_bus()).

Having zbus registration separate from scanning allows us to return
fully initialized but still disabled zdevs from zpci_create_device()
which can then be configured just as we would configure a zdev from
standby (minus the SCLP Configure already done by the platform). There
is still the exception that a PCI function with non-zero devfn can be
plugged before its PCI bus, which depends on the function with zero
devfn, is created. In this case the zdev returend from
zpci_create_device() is still missing its bus, hotplug slot, and
resources which need to be created later but at least it doesn't wait in
the enabled state and can otherwise be treated as initialized.

With this we also separate the initial PCI scan using CLP List PCI
Functions into two phases. In the CLP loop's callback we only register
each function with a virtual zbus creating the latter as needed. Then,
after we have built this virtual PCI topology based on our list of
zbusses, we can make use of the common code functionality to scan each
complete zbus as a separate child bus.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Acked-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: use mutex not spinlock for zbus list

In a later change we will first collect all PCI functions from the CLP
List PCI functions call, then register them to/creating the relevant
zbus. Then only after we've created our virtual bus structure will we
scan all zbusses iterating over the zbus list. Since scanning is
relatively slow a spinlock is a bad fit for protecting the
loop over the devices on the zbus. Furthermore doing the probing on the
bus we need to use pci_lock_rescan_remove() as devices are added to
the PCI subsystem and that is a mutex which can't be locked nested
inside a spinlock section. Note that the contention of this lock should
be very low either way as zbusses are only added/removed concurrently on
hotplug events.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: separate zbus creation from scanning

In the existing code the creation of the PCI bus and the scanning of
function zero all happens in zpci_scan_bus(). This in turn requires
functions to be enabled and their resources to be available before the
PCI bus is even created.

This not only means that functions are enabled long before they are
actually made available to the common PCI subsystem. In case of
functions with non-zero devfn which appeared before the function with
devfn zero they can wait arbitrarily long in this enabled but not
scanned state.

Fix this by separating the creation of the PCI bus from scanning it and
only prepare, that is enable and setup MMIO bus resources, functions
just before they are scanned. As they may be scanned multiple times
track if we already created resources in the zdev.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Acked-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: do more bus setup in zpci_bus_scan()

Pull setting the maximum bus speed and multifunction attribute into
zpci_bus_scan() in preparation for handling bus creation separately
from scanning the bus.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Acked-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: introduce zpci_bus_scan_device()

To match zpci_bus_scan_device() and the PCI common code terminology and
to remove some code duplication, we pull the multiple uses of
pci_scan_single_device() into a function. For now this has the side
effect of adding each device to the PCI bus separately and locking and
unlocking the rescan/remove lock for each instead of just once per bus.
This is clearly less efficient but provides a correct intermediate
behavior until a follow on change does both the adding and scanning only
once per bus.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Acked-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: move zpci_remove_device() to bus code

The zpci_remove_device() function removes the device from the PCI common
code core which is an operation dealing primarily with the zbus and PCI
bus code. With that and to match an upcoming refactoring of the
symmetric scanning part move it to the bus code.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: Mark all VFs as not implementing PCI_COMMAND_MEMORY

For s390 we can have VFs that are passed-through without the associated
PF. Firmware provides an emulation layer to allow these devices to
operate independently, but is missing emulation of the Memory Space
Enable bit. For these as well as linked VFs, set no_command_memory
which specifies these devices do not implement PCI_COMMAND_MEMORY.

Fixes: abafbc551fdd ("vfio-pci: Invalidate mmaps and block MMIO access on disabled memory")
Signed-off-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>

s390/pci: consolidate SR-IOV specific code

currently we have multiple #ifdef CONFIG_PCI_IOV blocks spread over
different compliation units and headers, all dealing with SR-IOV
specific behavior.
This violates the style guide which discourages conditionally compiled
code blocks and hinders maintainability by speading SR-IOV functionality
over many files.

Let's move all of this into a conditionally compiled pci_iov.c file and
local header and prefix SR-IOV specific functions with zpci_iov_*.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: fix PF/VF linking on hot plug

Currently there are four places in which a PCI function is scanned
and made available to drivers:
1. In pci_scan_root_bus() as part of the initial zbus
creation.
2. In zpci_bus_add_devices() when registering
a device in configured state on a zbus that has already been
scanned.
3. When a function is already known to zPCI (in reserved/standby state)
and configuration is triggered through firmware by PEC 0x301.
4. When a device is already known to zPCI (in standby/reserved state)
and configuration is triggered from within Linux using
enable_slot().

The PF/VF linking step and setting of pdev->is_virtfn introduced with
commit e5794cf1a270 ("s390/pci: create links between PFs and VFs") was
only triggered for the second case, which is where VFs created through
sriov_numvfs usually land. However unlike some other platforms but like
POWER VFs can be individually enabled/disabled through
/sys/bus/pci/slots.

Fix this by doing VF setup as part of pcibios_bus_add_device() which is
called in all of the above cases.

Finally to remove the PF/VF links call the common code
pci_iov_remove_virtfn() function to remove linked VFs.
This takes care of the necessary sysfs cleanup.

Fixes: e5794cf1a270 ("s390/pci: create links between PFs and VFs")
Cc: <stable@vger.kernel.org> # 5.8: 2f0230b2f2d5: s390/pci: re-introduce zpci_remove_device()
Cc: <stable@vger.kernel.org> # 5.8
Acked-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: fix zpci_bus_link_virtfn()

We were missing the pci_dev_put() for candidate PFs. Furhtermore in
discussion with upstream it turns out that somewhat counterintuitively
some common code, in particular the vfio-pci driver, assumes that
pdev->is_virtfn always implies that pdev->physfn is set, i.e. that VFs
are always linked.
While POWER does seem to set pdev->is_virtfn even for unlinked functions
(see comments in arch/powerpc/kernel/eeh.c:eeh_debugfs_break_device())
for now just be safe and only set pdev->is_virtfn on linking.
Also make sure that we only search for parent PFs if the zbus is
multifunction and we thus know the devfn values supplied by firmware
come from the RID.

Fixes: e5794cf1a270 ("s390/pci: create links between PFs and VFs")
Cc: <stable@vger.kernel.org> # 5.8
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: create links between PFs and VFs

On s390 PCI Virtual Functions (VFs) are scanned by firmware and are made
available to Linux via the hot-plug interface. As such the common code
path of doing the scan directly using the parent Physical Function (PF)
is not used and fenced off with the no_vf_scan attribute.

Even if the partition created the VFs itself e.g. using the sriov_numvfs
attribute of a PF, the PF/VF links thus need to be established after the
fact. To do this when a VF is plugged we scan through all functions on
the same zbus and test whether they are the parent PF in which case we
establish the necessary links.

With these links established there is now no more need to fence off
pci_iov_remove_virtfn() for pdev->no_vf_scan as the common code now
works fine.

Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Pierre Morel <pmorel@linux.ibm.com>
Link: https://lore.kernel.org/r/20200506154139.90609-3-schnelle@linux.ibm.com
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: removes wrong PCI multifunction assignment

The assignment of the PCI device multifunction attribute
is set during the PCI device probe.
There is no need to set it here.

Let's do it right and remove this assignment.

Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: Handling multifunctions

We allow multiple functions on a single bus.
We suppress the ZPCI_DEVFN definition and replace its
occurences with zpci->devfn.

We verify the number of device during the registration.

There can never be more domains in use than existing
devices, so we do not need to verify the count of domain
after having verified the count of devices.

Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: Adding bus resource

The current PCI implementation do not provide a bus resource.
This leads to a notice being print at boot.
Let's do it more nicely and provide the bus resource.

Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: create zPCI bus

The zPCI bus is in charge to handle common zPCI resources for
zPCI devices.

Creating the zPCI bus, the PCI bus, the zPCI devices and the
PCI devices and hotplug slots
done in a specific order:
- PCI hotplug slot creation needs a PCI bus
- PCI bus needs a PCI domain
which is reported by the pci_domain_nr() when setting up the
host bridge
- PCI domain is set from the zPCI with devfn 0
this is necessary to have a reproducible enumeration

Therefore we can not create devices or hotplug slots for any PCI
device associated with a zPCI device before having discovered
the function zero of the bus.

The discovery and initialization of devices can be done at several
points in the code:
- On Events, serialized in a thread context
- On initialization, in the kernel init thread context
- When powering on the hotplug slot, in a user thread context

The removal of devices and their parent bus may also be done on
events or for devices when powering down the slot.

To guarantee the existence of the bus and devices until they are
no more needed we use kref in zPCI bus and introduce a reference
count in the zPCI devices.

In this patch the zPCI bus still only accept a device with
a devfn 0.

Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>