gpu/drm/radeon: fix two memleaks in radeon_vm_init

When radeon_bo_create and radeon_vm_clear_bo fail, the vm->page_tables
allocated before need to be freed. However, neither radeon_vm_init
itself nor its caller have done such deallocation.

Fixes: 6d2f2944e95e ("drm/radeon: use normal BOs for the page tables v4")
Signed-off-by: Zhipeng Lu <alexious@zju.edu.cn>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

dma-buf/drivers: make reserving a shared slot mandatory v4

Audit all the users of dma_resv_add_excl_fence() and make sure they
reserve a shared slot also when only trying to add an exclusive fence.

This is the next step towards handling the exclusive fence like a
shared one.

v2: fix missed case in amdgpu
v3: and two more radeon, rename function
v4: add one more case to TTM, fix i915 after rebase

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20220406075132.3263-2-christian.koenig@amd.com

gpu/drm/radeon: Fix typo in comments

Remove the repeated word 'the' from comments

Signed-off-by: Cai Huoqing <caihuoqing@baidu.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/radeon_vm: Fix function naming disparities

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/radeon_vm.c:61: warning: expecting prototype for radeon_vm_num_pde(). Prototype was for radeon_vm_num_pdes() instead
drivers/gpu/drm/radeon/radeon_vm.c:642: warning: expecting prototype for radeon_vm_update_pdes(). Prototype was for radeon_vm_update_page_directory() instead

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/radeon_vm: Fix some function parameter documentation

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/radeon_vm.c:131: warning: Function parameter or member 'rdev' not described in 'radeon_vm_get_bos'
drivers/gpu/drm/radeon/radeon_vm.c:643: warning: Excess function parameter 'start' description in 'radeon_vm_update_page_directory'
drivers/gpu/drm/radeon/radeon_vm.c:643: warning: Excess function parameter 'end' description in 'radeon_vm_update_page_directory'
drivers/gpu/drm/radeon/radeon_vm.c:819: warning: Function parameter or member 'ib' not described in 'radeon_vm_update_ptes'
drivers/gpu/drm/radeon/radeon_vm.c:915: warning: Function parameter or member 'bo_va' not described in 'radeon_vm_bo_update'
drivers/gpu/drm/radeon/radeon_vm.c:915: warning: Excess function parameter 'vm' description in 'radeon_vm_bo_update'
drivers/gpu/drm/radeon/radeon_vm.c:915: warning: Excess function parameter 'bo' description in 'radeon_vm_bo_update'
drivers/gpu/drm/radeon/radeon_vm.c:1155: warning: Excess function parameter 'vm' description in 'radeon_vm_bo_invalidate'

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

gpu/drm/radeon: fix spelling typo in comments

Modify the comment typo: "definately" -> "definitely".

Signed-off-by: Wang Qing <wangqing@vivo.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/ttm: move to driver binding/destroy functions. (v2)

Do agp decision in the driver, instead of special binding funcs

v2: use container_of, drop some {}.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200907204630.1406528-6-airlied@gmail.com

drm/ttm: rename ttm_mem_reg to ttm_resource.

This name better reflects what the object does. I didn't rename
all the pointers it seemed too messy.

Signed-off-by: Dave Airlie <airlied@redhat.com>
Acked-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Ben Skeggs <bskeggs@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20200804025632.3868079-60-airlied@gmail.com

drm/radeon: remove unnecessary braces around conditionals.

As single statement conditionals do not need to be wrapped around
braces, the unnecessary braces can be removed.

Signed-off-by: Wambui Karuga <wambui.karugax@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

dma-buf: rename reservation_object to dma_resv

Be more consistent with the naming of the other DMA-buf objects.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/323401/

drm/radeon: switch driver from bo->resv to bo->base.resv

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20190805140119.7337-12-kraxel@redhat.com

drm/radeon: drop use of drmP.h (1/2)

Drop use of drmP.h in all .c files named radeon*c.
To ease review a little drmP.h removal was divided in two commits.

Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: "David (ChunMing) Zhou" <David1.Zhou@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20190608080241.4958-7-sam@ravnborg.org

drm/ttm: allow reserving more than one shared slot v3

Let's support simultaneous submissions to multiple engines.

v2: rename the field to num_shared and fix up all users
v3: rebased

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm: radeon: fix overflow on 32bit systems

the type mem->start is unsigned long, so this can overflow on
32bit system, since the type addr is uint64_t.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Young Xiao <YangX92@hotmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

dma-buf: allow reserving more than one shared fence slot

Let's support simultaneous submissions to multiple engines.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Junwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: Huang Rui <ray.huang@amd.com>
Link: https://patchwork.kernel.org/patch/10626149/

drm/ttm: add operation ctx to ttm_bo_validate v2

Give moving a BO into place an operation context to work with.

v2: rebased

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Tested-by: Dieter Nützel <Dieter@nuetzel-hh.de>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

lib/interval_tree: fast overlap detection

Allow interval trees to quickly check for overlaps to avoid unnecesary
tree lookups in interval_tree_iter_first().

As of this patch, all interval tree flavors will require using a
'rb_root_cached' such that we can have the leftmost node easily
available. While most users will make use of this feature, those with
special functions (in addition to the generic insert, delete, search
calls) will avoid using the cached option as they can do funky things
with insertions -- for example, vma_interval_tree_insert_after().

[jglisse@redhat.com: fix deadlock from typo vm_lock_anon_vma()]
Link: http://lkml.kernel.org/r/20170808225719.20723-1-jglisse@redhat.com
Link: http://lkml.kernel.org/r/20170719014603.19029-12-dave@stgolabs.net
Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Acked-by: Christian König <christian.koenig@amd.com>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Doug Ledford <dledford@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Christian Benvenuti <benve@cisco.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

drm/radeon: Fix preferred typo

Change "prefered" to "preferred"

Signed-off-by: Kent Russell <kent.russell@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm: drop drm_[cm]alloc* helpers

Now that drm_[cm]alloc* helpers are simple one line wrappers around
kvmalloc_array and drm_free_large is just kvfree alias we can drop
them and replace by their native forms.

This shouldn't introduce any functional change.

Changes since v1
- fix typo in drivers/gpu//drm/etnaviv/etnaviv_gem.c - noticed by 0day
build robot

Suggested-by: Daniel Vetter <daniel@ffwll.ch>
Signed-off-by: Michal Hocko <mhocko@suse.com>drm: drop drm_[cm]alloc* helpers
[danvet: Fixup vgem which grew another user very recently.]
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Acked-by: Christian König <christian.koenig@amd.com>
Link: http://patchwork.freedesktop.org/patch/msgid/20170517122312.GK18247@dhcp22.suse.cz

drm/radeon: fix indentation.

I hate doing this but it hurts my eyes to go over code that does not
comply with indentation rules. Only thing that is not only space change
is in atom.c all other files are space indentation issues.

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Fix off-by-one errors in radeon_vm_bo_set_addr

eoffset is sometimes treated as the last address inside the address
range, and sometimes as the first address outside the range. This
was resulting in errors when a test filled up the entire address
space. Make it consistent to always be the last address within the
range. Also fixed related errors when checking the VA limit and in
radeon_vm_fence_pts.

Signed-off-by: Felix.Kuehling <Felix.Kuehling@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: check if BO_VA is set before adding it to the invalidation list

Otherwise we try to clear BO_VAs without an address.

Fixes: https://bugs.freedesktop.org/show_bug.cgi?id=91141

Signed-off-by: Christian König <christian.koenig@amd.com>
Test-by: hadack@gmx.de
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: allways add the VM clear duplicate

We need to allways add the VM clear duplicate of the BO_VA,
no matter what the old status was.

Signed-off-by: Christian König <christian.koenig@amd.com>
Test-by: hadack@gmx.de
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix adding all VAs to the freed list on remove v2

We only should do so when the BO_VA was actually mapped.
Otherwise we get a nice error message on the next CS.

v2: It actually doesn't matter if it was invalidated or not,
if it was mapped we need to clear the area where it was mapped.

Signed-off-by: Christian König <christian.koenig@amd.com>
Tested-by: Michel Dänzer <michel.daenzer@amd.com> (v1)
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Make sure radeon_vm_bo_set_addr always unreserves the BO

Some error paths didn't unreserve the BO. This resulted in a deadlock
down the road on the next attempt to reserve the (still reserved) BO.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=90873
Cc: stable@vger.kernel.org
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: stop using addr to check for BO move

It is theoretically possible that a swapped out BO gets the
same GTT address, but different backing pages while being swapped in.

Instead just use another VA state to note updated areas.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: check new address before removing old one

Otherwise the change isn't atomic.

Signed-off-by: Christian König <christian.koenig@amd.com>
CC: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: reset BOs address after clearing it.

Otherwise it is possible that we will have page table corruption
if we change a BOs address multiple times.

Signed-off-by: Christian König <christian.koenig@amd.com>
CC: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix lockup when BOs aren't part of the VM on release

If we unmap BOs before releasing them them the intervall tree locks
up because we try to remove an entry not inside the tree.

Based on a patch from Michel Dänzer.

Signed-off-by: Christian König <christian.koenig@amd.com>
CC: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: properly set vm fragment size for TN/RL

Should be the same as cayman. We don't use VM by default
on NI parts so this isn't critical.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: Remove rdev->gart.pages_addr array

radeon_vm_map_gart can use rdev->gart.pages_entry instead.

Also move the masking of the page address to radeon_vm_map_gart from its
callers.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add spinlock for BO_VA status protection (v2)

Stop using the VM mutex for this

v2: fix typo in comment

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fence PT updates as shared

The BO_VA contains everything necessary.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: rename radeon_cs_reloc to radeon_bo_list

Better match what it is actually doing.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: drop the handle from radeon_cs_reloc

It's only used for duplicate check and that
can be done on the original as well.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon drop gobj from radeon_cs_reloc

It's only used once after initializing and that
ptr can be calculated from the BO as well.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: sync PT updates as shared v2

Only invalidating PTEs needs to be executed synchronized to using the PT.

v2: fix sync to uses

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: sync PD updates as shared

We never invalidate PD entries and making them valid can
run with other users in parallel.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fence BO_VAs manually

This allows us to finally remove the VM fence and
so allow concurrent use of it from different engines.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use one VMID for each ring

Use multiple VMIDs for each VM, one for each ring. That allows
us to execute flushes separately on each ring, still not ideal
cause in a lot of cases rings can share IDs.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: track VM update fences separately

Note for each fence if it's a VM page table update or not. This allows
us to determine the last VM update in a sync object and so to figure
out if we need to flush the TLB or not.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fence PT updates manually v2

This allows us to add the real execution fence as shared.

v2: fix typo

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: split semaphore and sync object handling v2

Previously we just allocated space for four hardware semaphores
in each software semaphore object. Make software semaphore objects
represent only one hardware semaphore address again by splitting
the sync code into it's own object.

v2: fix typo in comment

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove unnecessary VM syncs

The PD/PTs reservation object now contains everything needed.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: stop re-reserving the BO in radeon_vm_bo_set_addr

That's useless when all callers drop the reservation
immediately after calling the function.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: rework vm_flush parameters

Use ring structure instead of index and provide vm_id and pd_addr separately.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Use drm_malloc_ab instead of kmalloc_array

Should avoid kmalloc failures due to large number of array entries.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=81991
Reviewed-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: export reservation_object from dmabuf to ttm

Adds an extra argument to radeon_bo_create, which is only used in radeon_prime.c.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: cope with foreign fences inside the reservation object

Not the whole world is a radeon! :-)

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add the infrastructure for concurrent buffer access

This allows us to specify if we want to sync to
the shared fences of a reservation object or not.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: allow fence to be added as shared

This patch adds a new flag to the ttm_validate_buffer list to
add the fence as shared to the reservation object.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/ttm: flip the switch, and convert to dma_fence

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>

drm/ttm: add interruptible parameter to ttm_eu_reserve_buffers

It seems some drivers really want this as a parameter,
like vmwgfx.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>

drm/radeon: re-enable selective GPUVM flushing

Now that the PFP and ME synchronization is fixed, we
can enable this again reliably.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>

drm/radeon: Only flush HDP cache for indirect buffers from userspace

It isn't necessary for command streams generated by the kernel (at least
not while we aren't storing ring or indirect buffers in VRAM).

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Always flush VM again on < CIK

Not doing this causes piglit hangs[0] on my Cape Verde card. No issues on
Bonaire and Kaveri though.

[0] Same symptoms as those fixed on CIK by 'drm/radeon: set VM base addr
using the PFP v2'.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add a check for allocation failure (v2)

We can easily return -ENOMEM here if kzalloc() fails.

v2: agd5f: drop the vm mutex

Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add userptr support v8

This patch adds an IOCTL for turning a pointer supplied by
userspace into a buffer object.

It imposes several restrictions upon the memory being mapped:

1. It must be page aligned (both start/end addresses, i.e ptr and size).

2. It must be normal system memory, not a pointer into another map of IO
space (e.g. it must not be a GTT mmapping of another object).

3. The BO is mapped into GTT, so the maximum amount of memory mapped at
all times is still the GTT limit.

4. The BO is only mapped readonly for now, so no write support.

5. List of backing pages is only acquired once, so they represent a
snapshot of the first use.

Exporting and sharing as well as mapping of buffer objects created by
this function is forbidden and results in an -EPERM.

v2: squash all previous changes into first public version
v3: fix tabs, map readonly, don't use MM callback any more
v4: set TTM_PAGE_FLAG_SG so that TTM never messes with the pages,
pin/unpin pages on bind/unbind instead of populate/unpopulate
v5: rebased on 3.17-wip, IOCTL renamed to userptr, reject any unknown
flags, better handle READONLY flag, improve permission check
v6: fix ptr cast warning, use set_page_dirty/mark_page_accessed on unpin
v7: add warning about it's availability in the API definition
v8: drop access_ok check, fix VM mapping bits

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com> (v4)
Reviewed-by: Jérôme Glisse <jglisse@redhat.com> (v4)
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: update IB size estimation for VM

That should allow us to allocate bigger BOs.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: split PT setup in more functions

Move the decision what to use into the common VM code.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: take a BO reference on VM cleanup

This closes a small window where the GPU might have accessed freed up memory.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: try to enable VM flushing once more

Let's try to fix bugs related to this instead of just disabling it.

Signed-off-by: Christian König <christian.koenig@amd.com>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use an intervall tree to manage the VMA v2

Scales much better than scanning the address range linearly.

v2: store pfn instead of address

Signed-off-by: Christian König <christian.koenig@amd.com>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: invalidate moved BOs in the VM (v2)

Don't wait for the BO to be used again, just
update the PT on the next VM use.

v2: remove stray semicolon.

Signed-off-by: Christian König <christian.koenig@amd.com>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add trace_radeon_vm_flush

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Allow write-combined CPU mappings of BOs in GTT (v2)

v2: fix rebase onto drm-fixes

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix error handling in radeon_vm_bo_set_addr

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix VM IB handling

Calling radeon_vm_bo_find on the IB BO during CS
is illegal and can lead to an crash.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix handling of radeon_vm_bo_rmv v3

v3: completely rewritten. We now just remember which areas
of the PT to clear and do so on the next command submission.

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=79980

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: page table BOs are kernel allocations

Userspace shouldn't be able to access them.

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: make vm_block_size a module parameter

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: rename alt_domain to allowed_domains

And also domain to prefered_domains. That matches better
what those values represent.

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: Marek Olšák <maraeo@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add proper support for RADEON_VM_BLOCK_SIZE v2

This patch makes it possible to decide how many address
bits are spend on the page directory vs the page tables.

v2: remove unintended change

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add large PTE support for NI, SI and CIK v5

This patch implements support for VRAM page table entry compression.
PTE construction is enhanced to identify physically contiguous page
ranges and mark them in the PTE fragment field. L1/L2 TLB support is
enabled for 64KB (SI/CIK) and 256KB (NI) PTE fragments, significantly
improving TLB utilization for VRAM allocations.

Linear store bandwidth is improved from 60GB/s to 125GB/s on Pitcairn.
Unigine Heaven 3.0 sees an average improvement from 24.7 to 27.7 FPS
on default settings at 1920x1200 resolution with vsync disabled.

See main comment in radeon_vm.c for a technical description.

v2 (chk): rebased and simplified.
v3 (chk): add missing hw setup
v4 (chk): rebased on current drm-fixes-3.15
v5 (chk): fix comments and commit text

Signed-off-by: Jay Cornwall <jay@jcornwall.me>
Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: sync page table updates

Only necessary if we don't use the same engine for buffer moves and table updates.

Signed-off-by: Christian König <christian.koenig@amd.com>

drm/radeon: fix vm buffer size estimation

Only relevant if we got VM_BLOCK_SIZE>9, but better save than sorry.

Signed-off-by: Christian König <christian.koenig@amd.com>

drm/radeon: only allocate necessary size for vm bo list

No need to always allocate the theoretical maximum here.

Signed-off-by: Christian König <christian.koenig@amd.com>

drm/radeon: fix page directory update size estimation

Take padding into account as well.

Fixes: https://bugzilla.kernel.org/show_bug.cgi?id=75651

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove struct radeon_bo_list

Just move all fields into radeon_cs_reloc, removing unused/duplicated fields.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove global vm lock

Not needed any more.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use normal BOs for the page tables v4

No need to make it more complicated than necessary,
just allocate the page tables as normal BO and
flush whenever the address change.

v2: update comments and function name
v3: squash bug fixes, page directory and tables patch
v4: rebased on Mareks changes

Signed-off-by: Christian König <christian.koenig@amd.com>

drm/radeon: further cleanup vm flushing & fencing

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: separate gart and vm functions

Both are complex enough on their own.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>