drm/nouveau: nvkm/vmm: implement raw ops to manage uvmm

The new VM_BIND UAPI uses the DRM GPU VA manager to manage the VA space.
Hence, we a need a way to manipulate the MMUs page tables without going
through the internal range allocator implemented by nvkm/vmm.

This patch adds a raw interface for nvkm/vmm to pass the resposibility
for managing the address space and the corresponding map/unmap/sparse
operations to the upper layers.

Reviewed-by: Dave Airlie <airlied@redhat.com>
Signed-off-by: Danilo Krummrich <dakr@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20230804182406.5222-11-dakr@redhat.com

drm/nouveau/mmu: serialise mmu invalidations with private mutex

nvkm_subdev.mutex is going away.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Reviewed-by: Lyude Paul <lyude@redhat.com>

drm/nouveau/mmu: Add correct turing page kinds

Turing introduced a new simplified page kind
scheme, reducing the number of possible page
kinds from 256 to 16. It also is the first
NVIDIA GPU in which the highest possible page
kind value is not reserved as an "invalid" page
kind.

To address this, the invalid page kind is made
an explicit property of the MMU HAL, and a new
table of page kinds is added to the tu102 MMU
HAL.

One hardware change not addressed here is that
0x00 is technically no longer a supported page
kind, and pitch surfaces are instead intended to
share the block-linear generic page kind 0x06.
However, because that will be a rather invasive
change to nouveau and 0x00 still works fine in
practice on Turing hardware, addressing this new
behavior is deferred.

Signed-off-by: James Jones <jajones@nvidia.com>
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/mmu: support initialisation of client-managed address-spaces

NVKM is currently responsible for managing the allocation of a client's
GPU address-space, but there's various use-cases (ie. HMM address-space
mirroring) where giving a client more direct control is desirable.

This commit allows for a VMM to be created where the area allocated for
NVKM is limited to a client-specified window, the remainder of address-
space is controlled directly by the client.

Leaving a window is necessary to support various internal requirements,
but also to support existing allocation interfaces as not all of the HW
is capable of working with a HMM allocation.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/mmu/gf100-: virtualise setting pdb base address for invalidation

It appears that Pascal and newer need something different.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/mmu/gf100-: make mmu invalidate function more general

Will want to reuse this for fault replay/cancellation swmthds.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/mmu/gp100,gp10b: implement new vmm backend

Adds support for:
- 64KiB/2MiB big page sizes (128KiB not supported by HW with new PT layout).
- System-memory PTs.
- LPTE "invalid" state.
- (Tegra) Use of video memory aperture.
- Sparse PDEs/PTEs.
- Additional blocklinear kinds.
- 49-bit address-space.

GP100 supports an entirely new 5-level page table layout that provides
an expanded 49-bit address-space. It also supports the layout present
on previous generations, which we've been making do with until now.

This commit implements support for the new layout, and enables it by
default.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/mmu/gf100: implement new vmm backend

Adds support for:
- 64KiB big page size.
- System-memory PTs.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

drm/nouveau/mmu/gf100: implement vmm on top of new base

Adds support for:
- Selection of a 64KiB big page size (NvFbBigPage=16).
- System-memory PDs.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>