s390/pci: Use dma-iommu layer

While s390 already has a standard IOMMU driver and previous changes have
added I/O TLB flushing operations this driver is currently only used for
user-space PCI access such as vfio-pci. For the DMA API s390 instead
utilizes its own implementation in arch/s390/pci/pci_dma.c which drives
the same hardware and shares some code but requires a complex and
fragile hand over between DMA API and IOMMU API use of a device and
despite code sharing still leads to significant duplication and
maintenance effort. Let's utilize the common code DMAP API
implementation from drivers/iommu/dma-iommu.c instead allowing us to
get rid of arch/s390/pci/pci_dma.c.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Link: https://lore.kernel.org/r/20230928-dma_iommu-v13-3-9e5fc4dacc36@linux.ibm.com
Signed-off-by: Joerg Roedel <jroedel@suse.de>

KVM: s390: pci: Hook to access KVM lowlevel from VFIO

We have a cross dependency between KVM and VFIO when using
s390 vfio_pci_zdev extensions for PCI passthrough
To be able to keep both subsystem modular we add a registering
hook inside the S390 core code.

This fixes a build problem when VFIO is built-in and KVM is built
as a module.

Reported-by: Randy Dunlap <rdunlap@infradead.org>
Reported-by: kernel test robot <lkp@intel.com>
Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Fixes: 09340b2fca007 ("KVM: s390: pci: add routines to start/stop interpretive execution")
Cc: <stable@vger.kernel.org>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Acked-by: Randy Dunlap <rdunlap@infradead.org> # build-tested
Link: https://lore.kernel.org/r/20220819122945.9309-1-pmorel@linux.ibm.com
Message-Id: <20220819122945.9309-1-pmorel@linux.ibm.com>
Signed-off-by: Janosch Frank <frankja@linux.ibm.com>

s390/pci: consolidate SR-IOV specific code

currently we have multiple #ifdef CONFIG_PCI_IOV blocks spread over
different compliation units and headers, all dealing with SR-IOV
specific behavior.
This violates the style guide which discourages conditionally compiled
code blocks and hinders maintainability by speading SR-IOV functionality
over many files.

Let's move all of this into a conditionally compiled pci_iov.c file and
local header and prefix SR-IOV specific functions with zpci_iov_*.

Reviewed-by: Matthew Rosato <mjrosato@linux.ibm.com>
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: create zPCI bus

The zPCI bus is in charge to handle common zPCI resources for
zPCI devices.

Creating the zPCI bus, the PCI bus, the zPCI devices and the
PCI devices and hotplug slots
done in a specific order:
- PCI hotplug slot creation needs a PCI bus
- PCI bus needs a PCI domain
which is reported by the pci_domain_nr() when setting up the
host bridge
- PCI domain is set from the zPCI with devfn 0
this is necessary to have a reproducible enumeration

Therefore we can not create devices or hotplug slots for any PCI
device associated with a zPCI device before having discovered
the function zero of the bus.

The discovery and initialization of devices can be done at several
points in the code:
- On Events, serialized in a thread context
- On initialization, in the kernel init thread context
- When powering on the hotplug slot, in a user thread context

The removal of devices and their parent bus may also be done on
events or for devices when powering down the slot.

To guarantee the existence of the bus and devices until they are
no more needed we use kref in zPCI bus and introduce a reference
count in the zPCI devices.

In this patch the zPCI bus still only accept a device with
a devfn 0.

Signed-off-by: Pierre Morel <pmorel@linux.ibm.com>
Reviewed-by: Niklas Schnelle <schnelle@linux.ibm.com>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>

s390/pci: move everything irq related to pci_irq.c

Move everything interrupt related from pci.c to pci_irq.c.
No functional change.

Signed-off-by: Sebastian Ott <sebott@linux.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>

s390: add a few more SPDX identifiers

Add the correct SPDX license to a few more files under arch/s390 and
drivers/s390 which have been missed to far.
The SPDX identifier is a legally binding shorthand, which can be used
instead of the full boiler plate text.

Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>

s390/kernel: add system calls for PCI memory access

Add the new __NR_s390_pci_mmio_write and __NR_s390_pci_mmio_read
system calls to allow user space applications to access device PCI I/O
memory pages on s390x platform.

[ Martin Schwidefsky: some code beautification ]

Signed-off-by: Alexey Ishchuk <aishchuk@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>

s390: convert interrupt handling to use generic hardirq

With the introduction of PCI it became apparent that s390 should
convert to generic hardirqs as too many drivers do not have the
correct dependency for GENERIC_HARDIRQS. On the architecture
level s390 does not have irq lines. It has external interrupts,
I/O interrupts and adapter interrupts. This patch hard-codes all
external interrupts as irq #1, all I/O interrupts as irq #2 and
all adapter interrupts as irq #3. The additional information from
the lowcore associated with the interrupt is stored in the
pt_regs of the interrupt frame, where the interrupt handler can
pick it up. For PCI/MSI interrupts the adapter interrupt handler
scans the relevant bit fields and calls generic_handle_irq with
the virtual irq number for the MSI interrupt.

Reviewed-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>

s390/pci: uninline instruction wrappers

Uninline pci related instruction wrappers to de-bloat the code:
add/remove: 15/0 grow/shrink: 2/24 up/down: 1326/-12628 (-11302)

This is especially useful for the inlined pci read and write functions
which are used all over the kernel. Also remove the unused __stpcifc
while at it.

Reviewed-by: Gerald Schaefer <gerald.schaefer@de.ibm.com>
Signed-off-by: Sebastian Ott <sebott@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>

s390/pci: performance statistics and debug infrastructure

Add support for reading the PCI function measurement block counters
provided by the hypervisor. Add two s390 debug features, one for
critical errors and one for tracing and provide wrappers to log data.

Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>

s390/pci: s390 specific PCI sysfs attributes

Add some s390 specific sysfs attributes to the PCI device directory.
The following attributes are introduced:
- function_id (PCI function ID)
- function_handle (PCI function handle)
- pchid (PCI channel ID)
- pfgid (PCI function group ID aka PCI root complex)

Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>

s390/pci: CHSC PCI support for error and availability events

Add CHSC store-event-information support for PCI (notfication type 2)
and report error and availability events to the PCI architecture layer.

Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>

s390/pci: DMA support

Add DMA IOMMU support using 4K page table entries. Implement dma_map_ops.

Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>

s390/pci: PCI adapter interrupts for MSI/MSI-X

Support PCI adapter interrupts using the Single-IRQ-mode. Single-IRQ-mode
disables an adapter IRQ automatically after delivering it until the SIC
instruction enables it again. This is used to reduce the number of IRQs
for streaming workloads.

Up to 64 MSI handlers can be registered per PCI function.
A hash table is used to map interrupt numbers to MSI descriptors.
The interrupt vector is scanned using the flogr instruction.
Only MSI/MSI-X interrupts are supported, no legacy INTs.

Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>

s390/pci: CLP interface

CLP instructions are used to query the firmware about detected PCI
functions, the attributes of those functions and to enable or disable
a PCI function. The CLP interface is the equivalent to a PCI bus scan.

Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>

s390/pci: base support

Add PCI support for s390, (only 64 bit mode is supported by hardware):
- PCI facility tests
- PCI instructions: pcilg, pcistg, pcistb, stpcifc, mpcifc, rpcit
- map readb/w/l/q and writeb/w/l/q to pcilg and pcistg instructions
- pci_iomap implementation
- memcpy_fromio/toio
- pci_root_ops using special pcilg/pcistg
- device, bus and domain allocation

Signed-off-by: Jan Glauber <jang@linux.vnet.ibm.com>
Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>