drm/nouveau/fifo: set gf100_fifo_nonstall_block_dump storage-class-specifier to static

gcc with W=1 reports
drivers/gpu/drm/nouveau/nvkm/engine/fifo/gf100.c:451:1: error:
no previous prototype for ‘gf100_fifo_nonstall_block’ [-Werror=missing-prototypes]
451 | gf100_fifo_nonstall_block(struct nvkm_event *event, int type, int index)
| ^~~~~~~~~~~~~~~~~~~~~~~~~

gf100_fifo_nonstall_block is only used in gf100.c, so it should be static

Signed-off-by: Tom Rix <trix@redhat.com>
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230303132731.1919329-1-trix@redhat.com

drm/nouveau/fifo: add new channel classes

Exposes a bunch of the new features that became possible as a result
of the earlier commits. DRM will build on this in the future to add
support for features such as SCG ("async compute") and multi-device
rendering, as part of the work necessary to be able to write a half-
decent vulkan driver - finally.

For the moment, this just crudely ports DRM to the API changes.

- channel class interfaces now the same for all HW classes
- channel group class exposed (SCG)
- channel runqueue selector exposed (SCG)
- channel sub-device id control exposed (multi-device rendering)
- channel names in logging will reflect creating process, not fd owner
- explicit USERD allocation required by VOLTA_CHANNEL_GPFIFO_A and newer
- drm is smarter about determining the appropriate channel class to use

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add new engine context handling

Builds on the context tracking that was added earlier.

- marks engine context PTEs as 'priv' where possible

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add RAMFC info to nvkm_chan_func

- adds support for specifying SUBDEVICE_ID for channel
- rounds non-power-of-two GPFIFO sizes down, rather than up

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add USERD info to nvkm_chan_func

And use it to cleanup multiple implementations of almost the same thing.

- prepares for non-polled / client-provided USERD
- only zeroes relevant "registers", rather than entire USERD

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add RAMIN info to nvkm_chan_func

Currently provided by {chan,dma,gpfifo}*.c, and those are going away.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add common runlist control

- less dependence on waiting for runlist updates, on GPUs that allow it
- supports runqueue selector in RAMRL entries
- completes switch to common runl/cgrp/chan topology info

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add common channel recovery

That sure was fun to untangle.

- handled per-runlist, rather than globally
- more straight-forward process in general
- various potential SW/HW races have been fixed
- fixes lockdep issues that were present in >=gk104's prior implementation
- volta recovery now actually stands a chance of working
- volta/turing waiting for PBDMA idle before engine reset
- turing using hw-provided TSG info for CTXSW_TIMEOUT

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: kill channel on a selection of PBDMA errors

A bunch of these can be handled in such a way that the channel can
continue, however, any of these are a pretty decent sign something
has gone horribly wrong, and the safest option is to disable the
channel.

This is a bit of a hack, we will want to handle these individually
and dump relevant debug info for each at some point.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add chan/cgrp preempt()

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add chan start()/stop()

- nvkm_chan_error() built on top, stops channel and sends 'killed' event
- removes an odd double-bashing of channel enable regs on kepler and up
- pokes doorbell on turing and up, after enabling channel

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add chan bind()/unbind()

- stops programming (non-existent) runl id field on bind(), from maxwell

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add runlist block()/allow()

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add runlist wait()

- adds g8x/turing registers, which were missing before
- switches fermi to polled wait, like later hw (see: 4f2fc25c0f8bc...)

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add new channel lookup interfaces

- supports per-runlist CHIDs
- channel group lock held across reference, rather than global lock

v2:
- remove unnecessary parenthesis

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: merge mmu fault handlers together

After updating GF100 implementation from the GK104/TU102 ones, and using
the new runlist/engine topology info, all three handlers become (almost)
identical.

- there's a temporary kludge to call through to the HW-specific recovery
- engine fault mapping info determined at load time, not on every fault

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: move PBDMA intr to runq

- merges gf100/gk104- NV_PFIFO_INTR_0_PBDMA and NV_PPBDMA_INTR_0 code

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: move PBDMA init to runq

- bumps pbdma timeout to value RM uses on newer HW
- bumps fb timeout to max from boot default
- one/both of these greatly improves stability on // piglit runs

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: program NV_PFIFO_FB_TIMEOUT on init

NVGPU and RM both program this value.

Fixes a bunch of random hangs running parallel piglit.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: tidy global PBDMA init

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: tidy up non-stall intr handling

- removes a layer of indirection in the intr handling
- prevents non-stall ctrl racing with unknown intrs

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: use explicit intr interfaces

More control, and shallower call-chain to get to the point.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: use runlist engine info to lookup engine classes

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add common runlist/engine topology

Creates an nvkm_runl for each runlist on the GPU, and an nvkm_engn for
each engine that is reachable from a runlist.

- basically what gk104- already does, but extended to all chips
- adds per-runlist CHID allocators (Ampere)
- splits g98/gt2xx out from g84 (different target engines)

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add runq

Creates an nvkm_runq for each PBDMA, these will be associated with the
relevant runlist(s) later.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add chid allocator

We need to be able to allocate TSG IDs as well as channel IDs, also,
Ampere has per-runlist channel IDs.

- holds per-ID private data, which will be used for/to protect lookup
- holds an nvkm_event which will be used for events tied to IDs
- not used yet beyond setup, and switching use of "fifo->nr - 1" for
channel ID mask to "chid->mask"

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: merge gk104_fifo_func into nvkm_host_func

This makes it easier to transition everything.

- a couple of function renames for collisions

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: pre-move some blocks of code around

- will make subsequent patches more obvious
- no code changes

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: unify handling of channel classes

Adds the basic skeleton for common channel (group) interfaces.

- common behaviour between <gk104 and >=gk104 impl's
- separates priv/user channel objects
- passthrough to existing object for now, kludges removed later

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add chid_nr()

- reads channel count from GPU from gm200 onwards
- removes gm20b/gp10b (they become identical to gm200/gp100)

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo/gf100-: switch static mmu fault id list to type+inst

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo/gf100: use fifo engine id for engine recovery mask

Instead of subdev index.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add id_engine hook

Will be used by common code in subsequent commits to lookup driver
engine state from HW engine ID.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: add engine_id hook

Will be used by common code in subsequent commits to replace arrays
indexed by subdev index.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: switch to instanced constructor

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/device: pass instance id when looking up a subdev/engine

This switches to using the subdev list for lookup, and otherwise should
be a no-op aside from switching the function signatures.

Callers will be transitioned to split type+inst individually.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/subdev: store full subdev name in struct

Much easier to store this to avoid having to reconstruct a string for a
specific subdev, taking into account whether it's instanced or not.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo: private mutex

nvkm_subdev.mutex is going away.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/fifo/gf1xx: convert to using nvkm_fault_data

Would like to be able to reuse gf100_fifo_intr_fault() for (some of) the
later chipsets too, as it's identical.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo/gf100-: call into BAR to reset BARs after MMU fault

This is needed for Turing, but we're supposed to wait for completion after
re-writing the value on older GPUs anyway.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo/gf100-: use new interfaces for vmm operations

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/core/memory: add reference counting

We need to be able to prevent memory from being freed while it's still
mapped in a GPU's address-space.

Will be used by upcoming MMU changes.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/core/memory: change map interface to support upcoming mmu changes

Map flags (access, kind, etc) are currently defined in either the VMA,
or the memory object, which turns out to not be ideal for things like
suballocated buffers, etc.

These will become per-map flags instead, so we need to support passing
these arguments in nvkm_memory_map().

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/bar: modify interface to bar1 vmm mapping

Upcoming changes will remove the nvkm_vmm pointer from nvkm_vma, instead
requiring it to be explicitly specified on each operation.

It's not currently possible to get this information for BAR1 mappings,
so let's fix that ahead of time.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo/gf100-: provide notification to user if channel is killed

There are instances (such as non-recoverable GPU page faults) where
NVKM decides that a channel's context is no longer viable, and will
be removed from the runlist.

This commit notifies the owner of the channel when this happens, so
it has the opportunity to take some kind of recovery action instead
of hanging.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/core/memory: distinguish between coherent/non-coherent targets

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo/gf100-: recover from host mmu faults

This has been on the TODO list for a while now, recovering from things
such as attempting to execute a push buffer or touch a semaphore in an
unmapped memory area.

The only thing required on the HW side here is that the offending
channel is removed from the runlist, and *not* a full reset of PFIFO.

This used to be a bit messier to handle before the rework to make use
of engine topology info, but is apparently now trivial.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo/gf100: fix certain engines not being recovered after a fault

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo/gf100: don't attempt recovery of unknown mmu engines

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo/gf100: identify fault-recovery members more clearly

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo/gf100: rename spooon to pbdma, and move detection to oneinit

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo/gf100: take runlist target into account

Bits 28:29 of RUNLIST_BASE specify the memory target of the runlist. Set
it to 0x3 (SYS_MEM_NONCOHERENT) if the runlist object resides in system
memory.

Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo/gf100: remove references to "daemon"

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo/gf100: fix race condition when updating engine runlists

Similar in spirit to the gk104 fix with a similar title.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/object: merge with handle

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/core: remove the remainder of the previous style

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: convert to new-style nvkm_engine

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/bar: convert to new-style nvkm_subdev

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: convert user classes to new-style nvkm_object

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: split user classes out from engine implementations

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/sw: remove dependence on namedb/engctx lookup

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: remove dependence on namedb/engctx lookup

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fb: remove dependence on namedb/engctx lookup

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/gpuobj: separate allocation from nvkm_object

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: directly use instmem for runlists and polling areas

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/imem: improve management of instance memory

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/device: prepare for new-style subdevs

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/subdev: implement support for new-style nvkm_subdev

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/object: implement support for new-style nvkm_object

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/nvif: replace pushbuf with vm in fermi/kepler gpfifo class args

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/nvif: assign internal class identifiers to sw classes

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/nvif: replace path-based object identification

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: switch to gpuobj accessor macros

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/core: remove last printks

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: switch to subdev printk macros

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: switch to new-style timer macros

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: switch to device pri macros

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: cosmetic changes

This is purely preparation for upcoming commits, there should be no
code changes here.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/fifo: namespace + nvidia gpu names (no binary change)

The namespace of NVKM is being changed to nvkm_ instead of nouveau_,
which will be used for the DRM part of the driver. This is being
done in order to make it very clear as to what part of the driver a
given symbol belongs to, and as a minor step towards splitting the
DRM driver out to be able to stand on its own (for virt).

Because there's already a large amount of churn here anyway, this is
as good a time as any to also switch to NVIDIA's device and chipset
naming to ease collaboration with them.

A comparison of objdump disassemblies proves no code changes.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>