PCI: s390: Fix use-after-free of PCI resources with per-function hotplug

On s390 PCI functions may be hotplugged individually even when they
belong to a multi-function device. In particular on an SR-IOV device VFs
may be removed and later re-added.

In commit a50297cf8235 ("s390/pci: separate zbus creation from
scanning") it was missed however that struct pci_bus and struct
zpci_bus's resource list retained a reference to the PCI functions MMIO
resources even though those resources are released and freed on
hot-unplug. These stale resources may subsequently be claimed when the
PCI function re-appears resulting in use-after-free.

One idea of fixing this use-after-free in s390 specific code that was
investigated was to simply keep resources around from the moment a PCI
function first appeared until the whole virtual PCI bus created for
a multi-function device disappears. The problem with this however is
that due to the requirement of artificial MMIO addreesses (address
cookies) extra logic is then needed to keep the address cookies
compatible on re-plug. At the same time the MMIO resources semantically
belong to the PCI function so tying their lifecycle to the function
seems more logical.

Instead a simpler approach is to remove the resources of an individually
hot-unplugged PCI function from the PCI bus's resource list while
keeping the resources of other PCI functions on the PCI bus untouched.

This is done by introducing pci_bus_remove_resource() to remove an
individual resource. Similarly the resource also needs to be removed
from the struct zpci_bus's resource list. It turns out however, that
there is really no need to add the MMIO resources to the struct
zpci_bus's resource list at all and instead we can simply use the
zpci_bar_struct's resource pointer directly.

Fixes: a50297cf8235 ("s390/pci: separate zbus creation from scanning")
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Link: https://lore.kernel.org/r/20230306151014.60913-2-schnelle@linux.ibm.com
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: rename get_zdev_by_bus() to zdev_from_bus()

Getting a zpci_dev via get_zdev_by_bus() uses the long lived reference
held in zbus->function[devfn]. This is accounted for in
pcibios_add_device() and pcibios_release_device().

Therefore there is no need to increment the reference count in
get_zdev_by_bus() as is done for get_zdev_by_fid(). Instead callers must
not access the device after pcibios_release_device() was called which is
necessary for common PCI code anyway. With this though the very similar
naming may be misleading so rename get_zdev_by_bus() to zdev_from_bus()
emphasizing that we are directly referencing the zdev via the bus.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: improve zpci_dev reference counting

Currently zpci_dev uses kref based reference counting but only accounts
for one original reference plus one reference from an added pci_dev to
its underlying zpci_dev. Counting just the original reference worked
until the pci_dev reference was added in commit 2a671f77ee49 ("s390/pci:
fix use after free of zpci_dev") because once a zpci_dev goes away, i.e.
enters the reserved state, it would immediately get released. However
with the pci_dev reference this is no longer the case and the zpci_dev
may still appear in multiple availability events indicating that it was
reserved. This was solved by detecting when the zpci_dev is already on
its way out but still hanging around. This has however shown some light
on how unusual our zpci_dev reference counting is.

Improve upon this by modelling zpci_dev reference counting on pci_dev.
Analogous to pci_get_slot() increment the reference count in
get_zdev_by_fid(). Thus all users of get_zdev_by_fid() must drop the
reference once they are done with the zpci_dev.

Similar to pci_scan_single_device(), zpci_create_device() returns the
device with an initial count of 1 and the device added to the zpci_list
(analogous to the PCI bus' device_list). In turn users of
zpci_create_device() must only drop the reference once the device is
gone from the point of view of the zPCI subsystem, it might still be
referenced by the common PCI subsystem though.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: fix use after free of zpci_dev

The struct pci_dev uses reference counting but zPCI assumed erroneously
that the last reference would always be the local reference after
calling pci_stop_and_remove_bus_device(). This is usually the case but
not how reference counting works and thus inherently fragile.

In fact one case where this causes a NULL pointer dereference when on an
SRIOV device the function 0 was hot unplugged before another function of
the same multi-function device. In this case the second function's
pdev->sriov->dev reference keeps the struct pci_dev of function 0 alive
even after the unplug. This bug was previously hidden by the fact that
we were leaking the struct pci_dev which in turn means that it always
outlived the struct zpci_dev. This was fixed in commit 0b13525c20fe
("s390/pci: fix leak of PCI device structure") exposing the broken
behavior.

Fix this by accounting for the long living reference a struct pci_dev
has to its underlying struct zpci_dev via the zbus->function[] array and
only release that in pcibios_release_device() ensuring that the struct
pci_dev is not left with a dangling reference. This is a minimal fix in
the future it would probably better to use fine grained reference
counting for struct zpci_dev.

Fixes: 05bc1be6db4b2 ("s390/pci: create zPCI bus")
Cc: stable@vger.kernel.org
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: separate zbus registration from scanning

Now that the zbus can be created without being scanned we can go one
step further and make registering a device to a zbus independent from
scanning it. This way the zbus handling becomes much more natural
in that functions can be registered on the zbus to be scanned later more
closely resembling the handling of both real PCI hardware and other
virtual PCI busses like Hyper-V's virtual PCI bus (see for example
drivers/pci/controller/pci-hyperv.c:create_root_hv_pci_bus()).

Having zbus registration separate from scanning allows us to return
fully initialized but still disabled zdevs from zpci_create_device()
which can then be configured just as we would configure a zdev from
standby (minus the SCLP Configure already done by the platform). There
is still the exception that a PCI function with non-zero devfn can be
plugged before its PCI bus, which depends on the function with zero
devfn, is created. In this case the zdev returend from
zpci_create_device() is still missing its bus, hotplug slot, and
resources which need to be created later but at least it doesn't wait in
the enabled state and can otherwise be treated as initialized.

With this we also separate the initial PCI scan using CLP List PCI
Functions into two phases. In the CLP loop's callback we only register
each function with a virtual zbus creating the latter as needed. Then,
after we have built this virtual PCI topology based on our list of
zbusses, we can make use of the common code functionality to scan each
complete zbus as a separate child bus.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Acked-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: separate zbus creation from scanning

In the existing code the creation of the PCI bus and the scanning of
function zero all happens in zpci_scan_bus(). This in turn requires
functions to be enabled and their resources to be available before the
PCI bus is even created.

This not only means that functions are enabled long before they are
actually made available to the common PCI subsystem. In case of
functions with non-zero devfn which appeared before the function with
devfn zero they can wait arbitrarily long in this enabled but not
scanned state.

Fix this by separating the creation of the PCI bus from scanning it and
only prepare, that is enable and setup MMIO bus resources, functions
just before they are scanned. As they may be scanned multiple times
track if we already created resources in the zdev.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Acked-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: introduce zpci_bus_scan_device()

To match zpci_bus_scan_device() and the PCI common code terminology and
to remove some code duplication, we pull the multiple uses of
pci_scan_single_device() into a function. For now this has the side
effect of adding each device to the PCI bus separately and locking and
unlocking the rescan/remove lock for each instead of just once per bus.
This is clearly less efficient but provides a correct intermediate
behavior until a follow on change does both the adding and scanning only
once per bus.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Acked-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: move zpci_remove_device() to bus code

The zpci_remove_device() function removes the device from the PCI common
code core which is an operation dealing primarily with the zbus and PCI
bus code. With that and to match an upcoming refactoring of the
symmetric scanning part move it to the bus code.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390: remove orphaned function declarations

arch/s390/pci/pci_bus.h: zpci_bus_init - only declaration left after
commit 05bc1be6db4b ("s390/pci: create zPCI bus")

arch/s390/include/asm/gmap.h: gmap_pte_notify - only declaration left
after commit 4be130a08420 ("s390/mm: add shadow gmap support")

arch/s390/include/asm/pgalloc.h: rcu_table_freelist_finish - only
declaration left after commit 36409f6353fc ("[S390] use generic RCU
page-table freeing code")

arch/s390/include/asm/tlbflush.h: smp_ptlb_all - only declaration left
after commit 5a79859ae0f3 ("s390: remove 31 bit support")

arch/s390/include/asm/vtimer.h: init_cpu_vtimer - only declaration left
after commit b5f87f15e200 ("s390/idle: consolidate idle functions and
definitions")

arch/s390/include/asm/pci.h: zpci_debug_info - only declaration left
after commit 386aa051fb4b ("s390/pci: remove per device debug attribute")

arch/s390/include/asm/vdso.h: vdso_alloc_boot_cpu - only declaration
left after commit 4bff8cb54502 ("s390: convert to GENERIC_VDSO")

arch/s390/include/asm/smp.h: smp_vcpu_scheduled - only declaration left
after commit 67626fadd269 ("s390: enforce CONFIG_SMP")

arch/s390/kernel/entry.h: restart_call_handler - only declaration left
after commit 8b646bd75908 ("[S390] rework smp code")

arch/s390/kernel/entry.h: startup_init_nobss - only declaration left
after commit 2e83e0eb85ca ("s390: clean .bss before running uncompressed
kernel")

arch/s390/kernel/entry.h: s390_early_resume - only declaration left after
commit 394216275c7d ("s390: remove broken hibernate / power management
support")

drivers/s390/char/raw3270.h: raw3270_request_alloc_bootmem - only
declaration left after commit 33403dcfcdfd ("[S390] 3270 console:
convert from bootmem to slab")

drivers/s390/cio/device.h: ccw_device_schedule_sch_unregister - only
declaration left after commit 37de53bb5290 ("[S390] cio: introduce ccw
device todos")

drivers/s390/char/tape.h: tape_hotplug_event - has only declaration
since recorded git history.

drivers/s390/char/tape.h: tape_oper_handler - has only declaration since
recorded git history.

drivers/s390/char/tape.h: tape_noper_handler - has only declaration
since recorded git history.

drivers/s390/char/tape_std.h: tape_std_check_locate - only declaration
left after commit 161beff8f40d ("s390/tape: remove tape block leftovers")

drivers/s390/char/tape_std.h: tape_std_default_handler - has only
declaration since recorded git history.

drivers/s390/char/tape_std.h: tape_std_unexpect_uchk_handler - has only
declaration since recorded git history.

drivers/s390/char/tape_std.h: tape_std_irq - has only declaration since
recorded git history.

drivers/s390/char/tape_std.h: tape_std_error_recovery - has only
declaration since recorded git history.

drivers/s390/char/tape_std.h: tape_std_error_recovery_has_failed -
has only declaration since recorded git history.

drivers/s390/char/tape_std.h: tape_std_error_recovery_succeded - has
only declaration since recorded git history.

drivers/s390/char/tape_std.h: tape_std_error_recovery_do_retry - has
only declaration since recorded git history.

drivers/s390/char/tape_std.h: tape_std_error_recovery_read_opposite -
has only declaration since recorded git history.

drivers/s390/char/tape_std.h: tape_std_error_recovery_HWBUG - has only
declaration since recorded git history.

Reviewed-by: Sven Schnelle <svens@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: consolidate SR-IOV specific code

currently we have multiple #ifdef CONFIG_PCI_IOV blocks spread over
different compliation units and headers, all dealing with SR-IOV
specific behavior.
This violates the style guide which discourages conditionally compiled
code blocks and hinders maintainability by speading SR-IOV functionality
over many files.

Let's move all of this into a conditionally compiled pci_iov.c file and
local header and prefix SR-IOV specific functions with zpci_iov_*.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: fix PF/VF linking on hot plug

Currently there are four places in which a PCI function is scanned
and made available to drivers:
1. In pci_scan_root_bus() as part of the initial zbus
creation.
2. In zpci_bus_add_devices() when registering
a device in configured state on a zbus that has already been
scanned.
3. When a function is already known to zPCI (in reserved/standby state)
and configuration is triggered through firmware by PEC 0x301.
4. When a device is already known to zPCI (in standby/reserved state)
and configuration is triggered from within Linux using
enable_slot().

The PF/VF linking step and setting of pdev->is_virtfn introduced with
commit e5794cf1a270 ("s390/pci: create links between PFs and VFs") was
only triggered for the second case, which is where VFs created through
sriov_numvfs usually land. However unlike some other platforms but like
POWER VFs can be individually enabled/disabled through
/sys/bus/pci/slots.

Fix this by doing VF setup as part of pcibios_bus_add_device() which is
called in all of the above cases.

Finally to remove the PF/VF links call the common code
pci_iov_remove_virtfn() function to remove linked VFs.
This takes care of the necessary sysfs cleanup.

Fixes: e5794cf1a270 ("s390/pci: create links between PFs and VFs")
Cc: <stable@vger.kernel.org> # 5.8: 2f0230b2f2d5: s390/pci: re-introduce zpci_remove_device()
Cc: <stable@vger.kernel.org> # 5.8
Acked-by: Pierre Morel <pmorel@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>

s390/pci: Handling multifunctions

We allow multiple functions on a single bus.
We suppress the ZPCI_DEVFN definition and replace its
occurences with zpci->devfn.

We verify the number of device during the registration.

There can never be more domains in use than existing
devices, so we do not need to verify the count of domain
after having verified the count of devices.

Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: create zPCI bus

The zPCI bus is in charge to handle common zPCI resources for
zPCI devices.

Creating the zPCI bus, the PCI bus, the zPCI devices and the
PCI devices and hotplug slots
done in a specific order:
- PCI hotplug slot creation needs a PCI bus
- PCI bus needs a PCI domain
which is reported by the pci_domain_nr() when setting up the
host bridge
- PCI domain is set from the zPCI with devfn 0
this is necessary to have a reproducible enumeration

Therefore we can not create devices or hotplug slots for any PCI
device associated with a zPCI device before having discovered
the function zero of the bus.

The discovery and initialization of devices can be done at several
points in the code:
- On Events, serialized in a thread context
- On initialization, in the kernel init thread context
- When powering on the hotplug slot, in a user thread context

The removal of devices and their parent bus may also be done on
events or for devices when powering down the slot.

To guarantee the existence of the bus and devices until they are
no more needed we use kref in zPCI bus and introduce a reference
count in the zPCI devices.

In this patch the zPCI bus still only accept a device with
a devfn 0.

Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>