drm/radeon: Use RMW accessors for changing LNKCTL2

Convert open coded RMW accesses for LNKCTL2 to use
pcie_capability_clear_and_set_word() which makes its easier to
understand what the code tries to do.

LNKCTL2 is not really owned by any driver because it is a collection of
control bits that PCI core might need to touch. RMW accessors already
have support for proper locking for a selected set of registers
(LNKCTL2 is not yet among them but likely will be in the future) to
avoid losing concurrent updates.

Acked-by: Alex Deucher <alexander.deucher@amd.com>
Suggested-by: Lukas Wunner <lukas@wunner.de>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Use RMW accessors for changing LNKCTL

Don't assume that only the driver would be accessing LNKCTL. ASPM policy
changes can trigger write to LNKCTL outside of driver's control. And in
the case of upstream bridge, the driver does not even own the device it's
changing the registers for.

Use RMW capability accessors which do proper locking to avoid losing
concurrent updates to the register value.

Suggested-by: Lukas Wunner <lukas@wunner.de>
Fixes: 8a7cd27679d0 ("drm/radeon/cik: add support for pcie gen1/2/3 switching")
Fixes: b9d305dfb66c ("drm/radeon: implement pcie gen2/3 support for SI")
Link: https://lore.kernel.org/r/20230717120503.15276-7-ilpo.jarvinen@linux.intel.com
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>

drm: Spelling s/sempahore/semaphore/

Fix misspellings of "semaphore".

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
Reviewed-by: Hamza Mahfooz <hamza.mahfooz@amd.com>
Acked-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Javier Martinez Canillas <javierm@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/8b0542c12a2427f34a792c41ac2d2a2922874bfa.1689600102.git.geert+renesas@glider.be

drm/radeon: fix UVD suspend error

I met a bug recently and the kernel log:

[ 330.171875] radeon 0000:03:00.0: couldn't schedule ib
[ 330.175781] [drm:radeon_uvd_suspend [radeon]] *ERROR* Error destroying UVD (-22)!

In radeon drivers, using UVD suspend is as follows:

if (rdev->has_uvd) {
uvd_v1_0_fini(rdev);
radeon_uvd_suspend(rdev);
}

In radeon_ib_schedule function, we check the 'ring->ready' state,
but in uvd_v1_0_fini funciton, we've cleared the ready state.
So, just modify the suspend code flow to fix error.

Reviewed-by: Leo Liu <leo.liu@amd.com>
Signed-off-by: Qiang Ma <maqianga@uniontech.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: delete useless function return values & remove meaningless if(r) check code

Function radeon_fence_driver_init always returns success,
the function type maybe coule be changed to void.
This patch first delete the check of the return
value of the function call radeon_fence_driver_init, then,
optimise the function declaration and function to void type.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Bernard Zhao <bernard@vivo.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: Fix incorrectly named function 'cik_irq_suspend()'

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/cik.c:7450: warning: expecting prototype for cik_irq_disable(). Prototype was for cik_irq_suspend() instead

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: remove set but not used variables

The value of pipe_id and queue_id are not used under certain
circumstances, so just delete.

Signed-off-by: Tian Tao <tiantao6@hisilicon.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: Move 'vce_v2_0_enable_mgcg()'s prototype to shared header

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/vce_v2_0.c:111:6: warning: no previous prototype for ‘vce_v2_0_enable_mgcg’ [-Wmissing-prototypes]
111 | void vce_v2_0_enable_mgcg(struct radeon_device *rdev, bool enable)
| ^~~~~~~~~~~~~~~~~~~~

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: Move 'Move 'cik_sdma_*()'s prototypes to shared header

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/cik_sdma.c:331:6: warning: no previous prototype for ‘cik_sdma_enable’ [-Wmissing-prototypes]
331 | void cik_sdma_enable(struct radeon_device *rdev, bool enable)
| ^~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/cik_sdma.c:528:5: warning: no previous prototype for ‘cik_sdma_resume’ [-Wmissing-prototypes]
528 | int cik_sdma_resume(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/cik_sdma.c:557:6: warning: no previous prototype for ‘cik_sdma_fini’ [-Wmissing-prototypes]
557 | void cik_sdma_fini(struct radeon_device *rdev)
| ^~~~~~~~~~~~~

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: Move 'si_*()'s prototypes to shared header

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/si.c:4186:6: warning: no previous prototype for ‘si_vram_gtt_location’ [-Wmissing-prototypes]
4186 | void si_vram_gtt_location(struct radeon_device *rdev,
| ^~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/si.c:5186:6: warning: no previous prototype for ‘si_init_uvd_internal_cg’ [-Wmissing-prototypes]
5186 | void si_init_uvd_internal_cg(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/si.c:5801:6: warning: no previous prototype for ‘si_rlc_reset’ [-Wmissing-prototypes]
5801 | void si_rlc_reset(struct radeon_device *rdev)
| ^~~~~~~~~~~~

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/ci_dpm: Move 'ci_*()'s prototypes to shared header

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/cik.c:1868:5: warning: no previous prototype for ‘ci_mc_load_microcode’ [-Wmissing-prototypes]
1868 | int ci_mc_load_microcode(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/cik.c:5847:6: warning: no previous prototype for ‘cik_enter_rlc_safe_mode’ [-Wmissing-prototypes]
5847 | void cik_enter_rlc_safe_mode(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/cik.c:5868:6: warning: no previous prototype for ‘cik_exit_rlc_safe_mode’ [-Wmissing-prototypes]
5868 | void cik_exit_rlc_safe_mode(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/cik.c:6286:6: warning: no previous prototype for ‘cik_update_cg’ [-Wmissing-prototypes]
6286 | void cik_update_cg(struct radeon_device *rdev,
| ^~~~~~~~~~~~~

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: Move 'r600_ih_ring_{alloc, fini}()'s prototypes to shared header

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/r600.c:3480:5: warning: no previous prototype for ‘r600_ih_ring_alloc’ [-Wmissing-prototypes]
3480 | int r600_ih_ring_alloc(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/r600.c:3516:6: warning: no previous prototype for ‘r600_ih_ring_fini’ [-Wmissing-prototypes]
3516 | void r600_ih_ring_fini(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/evergreen: Move 'evergreen_*' and 'sumo_*' prototypes out to shared location

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/evergreen.c:1281:6: warning: no previous prototype for ‘evergreen_fix_pci_max_read_req_size’ [-Wmissing-prototypes]
1281 | void evergreen_fix_pci_max_read_req_size(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:2664:6: warning: no previous prototype for ‘evergreen_mc_stop’ [-Wmissing-prototypes]
2664 | void evergreen_mc_stop(struct radeon_device *rdev, struct evergreen_mc_save *save)
| ^~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:2762:6: warning: no previous prototype for ‘evergreen_mc_resume’ [-Wmissing-prototypes]
2762 | void evergreen_mc_resume(struct radeon_device *rdev, struct evergreen_mc_save *save)
| ^~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:2850:6: warning: no previous prototype for ‘evergreen_mc_program’ [-Wmissing-prototypes]
2850 | void evergreen_mc_program(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:3710:5: warning: no previous prototype for ‘evergreen_mc_init’ [-Wmissing-prototypes]
3710 | int evergreen_mc_init(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:3769:6: warning: no previous prototype for ‘evergreen_print_gpu_status_regs’ [-Wmissing-prototypes]
3769 | void evergreen_print_gpu_status_regs(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:3797:6: warning: no previous prototype for ‘evergreen_is_display_hung’ [-Wmissing-prototypes]
3797 | bool evergreen_is_display_hung(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:4009:6: warning: no previous prototype for ‘evergreen_gpu_pci_config_reset’ [-Wmissing-prototypes]
4009 | void evergreen_gpu_pci_config_reset(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:4110:6: warning: no previous prototype for ‘sumo_rlc_fini’ [-Wmissing-prototypes]
4110 | void sumo_rlc_fini(struct radeon_device *rdev)
| ^~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:4153:5: warning: no previous prototype for ‘sumo_rlc_init’ [-Wmissing-prototypes]
4153 | int sumo_rlc_init(struct radeon_device *rdev)
| ^~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:4381:5: warning: no previous prototype for ‘evergreen_rlc_resume’ [-Wmissing-prototypes]
4381 | int evergreen_rlc_resume(struct radeon_device *rdev)
| ^~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:4670:6: warning: no previous prototype for ‘evergreen_irq_suspend’ [-Wmissing-prototypes]
4670 | void evergreen_irq_suspend(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~

NB: Lots more of these - snipped for brevity

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: Remove set but unused variable 'mc_shared_chmap'

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/cik.c: In function ‘cik_gpu_init’:
drivers/gpu/drm/radeon/cik.c:3180:6: warning: variable ‘mc_shared_chmap’ set but not used [-Wunused-but-set-variable]

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: Fix a bunch of function parameter descriptions

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/cik.c:1869:5: warning: no previous prototype for ‘ci_mc_load_microcode’ [-Wmissing-prototypes]
drivers/gpu/drm/radeon/cik.c: In function ‘cik_gpu_init’:
drivers/gpu/drm/radeon/cik.c:3181:6: warning: variable ‘mc_shared_chmap’ set but not used [-Wunused-but-set-variable]
drivers/gpu/drm/radeon/cik.c: At top level:
drivers/gpu/drm/radeon/cik.c:4852:5: warning: no previous prototype for ‘cik_gpu_check_soft_reset’ [-Wmissing-prototypes]
drivers/gpu/drm/radeon/cik.c:5850:6: warning: no previous prototype for ‘cik_enter_rlc_safe_mode’ [-Wmissing-prototypes]
drivers/gpu/drm/radeon/cik.c:5871:6: warning: no previous prototype for ‘cik_exit_rlc_safe_mode’ [-Wmissing-prototypes]
drivers/gpu/drm/radeon/cik.c:6289:6: warning: no previous prototype for ‘cik_update_cg’ [-Wmissing-prototypes]
drivers/gpu/drm/radeon/cik.c:6421:6: warning: no previous prototype for ‘cik_init_cp_pg_table’ [-Wmissing-prototypes]
drivers/gpu/drm/radeon/cik.c:6678:5: warning: no previous prototype for ‘cik_get_csb_size’ [-Wmissing-prototypes]
drivers/gpu/drm/radeon/cik.c:6710:6: warning: no previous prototype for ‘cik_get_csb_buffer’ [-Wmissing-prototypes]
drivers/gpu/drm/radeon/cik.c:3084: warning: Function parameter or member 'max_rb_num_per_se' not described in 'cik_get_rb_disabled'
drivers/gpu/drm/radeon/cik.c:3084: warning: Excess function parameter 'max_rb_num' description in 'cik_get_rb_disabled'
drivers/gpu/drm/radeon/cik.c:3084: warning: Excess function parameter 'se_num' description in 'cik_get_rb_disabled'
drivers/gpu/drm/radeon/cik.c:3114: warning: Function parameter or member 'max_rb_num_per_se' not described in 'cik_setup_rb'
drivers/gpu/drm/radeon/cik.c:3114: warning: Excess function parameter 'max_rb_num' description in 'cik_setup_rb'
drivers/gpu/drm/radeon/cik.c:5662: warning: Function parameter or member 'mc_client' not described in 'cik_vm_decode_fault'
drivers/gpu/drm/radeon/cik.c:5690: warning: Function parameter or member 'ring' not described in 'cik_vm_flush'
drivers/gpu/drm/radeon/cik.c:5690: warning: Function parameter or member 'vm_id' not described in 'cik_vm_flush'
drivers/gpu/drm/radeon/cik.c:5690: warning: Function parameter or member 'pd_addr' not described in 'cik_vm_flush'

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use true,false for bool variable in cik.c

Fixes coccicheck warning:

drivers/gpu/drm/radeon/cik.c:8140:2-15: WARNING: Assignment of 0/1 to bool variable
drivers/gpu/drm/radeon/cik.c:8212:2-15: WARNING: Assignment of 0/1 to bool variable

Reported-by: Hulk Robot <hulkci@huawei.com>
Signed-off-by: zhengbin <zhengbin13@huawei.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Don't include <drm/drm_pci.h>

Including <drm/drm_pci.h> is unnecessary in most cases. Replace
these instances.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191203100406.9674-9-tzimmermann@suse.de

drm/radeon: fix bad DMA from INTERRUPT_CNTL2

The INTERRUPT_CNTL2 register expects a valid DMA address, but is
currently set with a GPU MC address. This can cause problems on
systems that detect the resulting DMA read from an invalid address
(found on a Power8 guest).

Instead, use the DMA address of the dummy page because it will always
be safe.

Fixes: d8f60cfc9345 ("drm/radeon/kms: Add support for interrupts on r6xx/r7xx chips (v3)")
Fixes: 25a857fbe973 ("drm/radeon/kms: add support for interrupts on SI")
Fixes: a59781bbe528 ("drm/radeon: add support for interrupts on CIK (v5)")
Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Prefer pcie_capability_read_word()

Commit 8c0d3a02c130 ("PCI: Add accessors for PCI Express Capability")
added accessors for the PCI Express Capability so that drivers didn't
need to be aware of differences between v1 and v2 of the PCI
Express Capability.

Replace pci_read_config_word() and pci_write_config_word() calls with
pcie_capability_read_word() and pcie_capability_write_word().

Link: https://lore.kernel.org/r/20191118003513.10852-1-fred@fredlawl.com
Signed-off-by: Frederick Lawler <fred@fredlawl.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Replace numbers with PCI_EXP_LNKCTL2 definitions

Replace hard-coded magic numbers with the descriptive PCI_EXP_LNKCTL2
definitions. No functional change intended.

Link: https://lore.kernel.org/r/20191112173503.176611-4-helgaas@kernel.org
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Correct Transmit Margin masks

Previously we masked PCIe Link Control 2 register values with "7 << 9",
which was apparently intended to be the Transmit Margin field, but instead
was the high order bit of Transmit Margin, the Enter Modified Compliance
bit, and the Compliance SOS bit.

Correct the mask to "7 << 7", which is the Transmit Margin field.

Link: https://lore.kernel.org/r/20191112173503.176611-3-helgaas@kernel.org
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix bad DMA from INTERRUPT_CNTL2

The INTERRUPT_CNTL2 register expects a valid DMA address, but is
currently set with a GPU MC address. This can cause problems on
systems that detect the resulting DMA read from an invalid address
(found on a Power8 guest).

Instead, use the DMA address of the dummy page because it will always
be safe.

Fixes: d8f60cfc9345 ("drm/radeon/kms: Add support for interrupts on r6xx/r7xx chips (v3)")
Fixes: 25a857fbe973 ("drm/radeon/kms: add support for interrupts on SI")
Fixes: a59781bbe528 ("drm/radeon: add support for interrupts on CIK (v5)")
Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove assignment for return value

Remove unnecessary assignment for return value and have the
function return the required value directly.
Issue found by coccinelle:
@@
local idexpression ret;
expression e;
@@

-ret =
+return
e;
-return ret;

Reviewed-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Wambui Karuga <wambui@karuga.xyz>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

dma-buf: rename reservation_object to dma_resv

Be more consistent with the naming of the other DMA-buf objects.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Link: https://patchwork.freedesktop.org/patch/323401/

drm/radeon: drop use of drmP.h (2/2)

Drop use of drmP.h in remaining .c files.
To ease review a little the drmP.h removal was divided in two commits.

Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: "David (ChunMing) Zhou" <David1.Zhou@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20190608080241.4958-8-sam@ravnborg.org

drm/radeon: drop dependency on drm_os_linux.h

The drm_os_linux.h header file is deprecated.
Drop all uses of symbols from this file.
All macros are just opencoded.
A few adjustments to include files was required as the obsolete
drm_os_linux.h included these headers.

Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: "David (ChunMing) Zhou" <David1.Zhou@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20190608080241.4958-4-sam@ravnborg.org

gpu: drm: radeon: cik: Replace mdelay() with msleep() in cik_pcie_gen3_enable()

cik_pcie_gen3_enable() is never called in atomic context.
It calls mdelay() to busily wait, which is not necessary.
mdelay() can be replaced with msleep().

This is found by a static analysis tool named DCNS written by myself.

Signed-off-by: Jia-Ju Bai <baijiaju1990@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use pcie functions for link width

This is the last user of drm_pcie_get_speed_cap_mask. Use the pci
version so we can drop drm_pcie_get_speed_cap_mask.

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix KV harvesting

Always set the graphics values to the max for the
asic type. E.g., some 1 RB chips are actually 1 RB chips,
others are actually harvested 2 RB chips.

Fixes: https://bugs.freedesktop.org/show_bug.cgi?id=99353
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix KV harvesting

Always set the graphics values to the max for the
asic type. E.g., some 1 RB chips are actually 1 RB chips,
others are actually harvested 2 RB chips.

Fixes: https://bugs.freedesktop.org/show_bug.cgi?id=99353
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

Revert "drm/radeon: reduce number of free VMIDs and pipes in KV"

This reverts the remaining changes of commit
62a7b7fbd08ef745bb51e8728e89125a0ba6327e, because radeon doesn't support
amdkfd anymore. The number of VMIDs was already changed back when amdkfd
support was removed.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Michel Dänzer <michel@daenzer.net>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

Revert "drm/radeon/cik: Don't touch int of pipes 1-7"

This reverts commit 28b57b856b635ea0d44f1281e2efdc963c100ea3. radeon
doesn't support amdkfd anymore, so the latter doesn't set up interrupts
for pipes 1-7.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Michel Dänzer <michel@daenzer.net>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

Revert "drm/radeon: adding synchronization for GRBM GFX"

This reverts commit 1c0a46255f8d7daf5b601668836e185fd1294e94. Not needed
anymore, since amdkfd is no longer supported with radeon.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Michel Dänzer <michel@daenzer.net>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove init of CIK VMIDs 8-16 for amdkfd

VMIDs 8-16 in Kaveri were reserved for use by the amdkfd driver.
Because we removed amdkfd support from radeon, those VMIDs are now
used by radeon and are initialized by radeon.

This patch removes the function that initialized those VMIDs for amdkfd
use.
This initialization overridden the radeon initialization and caused GPU
faults and GUI crashed.

Fixes: f4fa88ab28ab ("drm/radeon: deprecate and remove KFD interface")
Rported-by: Michel Dänzer <michel.daenzer@amd.com>
Acked-by: Christian König <christian.koenig@amd.com>
Reviewed-and-Tested-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: deprecate and remove KFD interface

To quote Felix: "For testing KV with current user mode stack, please use
amdgpu. I don't expect this to work with radeon and I'm not planning to
spend any effort on making radeon work with a current user mode stack."

Only compile tested, but should be straight forward.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@gmail.com>

drm/radeon: Fix overflow of watermark calcs at > 4k resolutions.

Commit e6b9a6c84b93
("drm/radeon: Make display watermark calculations more accurate")
made watermark calculations more accurate, but not for > 4k
resolutions on 32-Bit architectures, as it introduced an integer
overflow for those setups and resolutions.

Fix this by proper u64 casting and division.

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Reported-by: Ben Hutchings <ben.hutchings@codethink.co.uk>
Fixes: e6b9a6c84b93 ("drm/radeon: Make display watermark calculations more accurate")
Cc: Ben Hutchings <ben.hutchings@codethink.co.uk>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: take ownership of pipe initialization

Take ownership of pipe initialization away from KFD.

Note that hpd_eop_gpu_addr was already large enough to accomodate all
pipes.

Reviewed-by: Edward O'Callaghan <funfunctor@folklore1984.net>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Andres Rodriguez <andresx7@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Unbreak HPD handling for r600+

We end up reading the interrupt register for HPD5, and then writing it
to HPD6 which on systems without anything using HPD5 results in
permanently disabling hotplug on one of the display outputs after the
first time we acknowledge a hotplug interrupt from the GPU.

This code is really bad. But for now, let's just fix this. I will
hopefully have a large patch series to refactor all of this soon.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Lyude <lyude@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Unbreak HPD handling for r600+

We end up reading the interrupt register for HPD5, and then writing it
to HPD6 which on systems without anything using HPD5 results in
permanently disabling hotplug on one of the display outputs after the
first time we acknowledge a hotplug interrupt from the GPU.

This code is really bad. But for now, let's just fix this. I will
hopefully have a large patch series to refactor all of this soon.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Lyude <lyude@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix include notation and remove -Iinclude/drm flag

Include <drm/*.h> instead of relative path from include/drm, then
remove the -Iinclude/drm compiler flag.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: http://patchwork.freedesktop.org/patch/msgid/1493009447-31524-14-git-send-email-yamada.masahiro@socionext.com

drm/radeon: Make display watermark calculations more accurate

Avoid big roundoff errors in scanline/hactive durations for
high pixel clocks, especially for >= 500 Mhz, and thereby
program more accurate display fifo watermarks.

This is a port of the corresponding amdgpu patch.

Implemented for DCE 4,6,8.
Tested on Evergreen/DCE-4 with Radeon HD-5770.

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Avoid overflows/divide-by-zero in latency_watermark calculations.

At dot clocks > approx. 250 Mhz, some of these calcs will overflow and
cause miscalculation of latency watermarks, and for some overflows also
divide-by-zero driver crash. Make calcs more overflow resistant.

This is a direct port of the corresponding patch from amdgpu-kms,
copy-paste for cik from dce-8 and si from dce-6, with a slightly
simpler variant for evergreen dce-4/5.

Only tested on DCE-4 evergreen with a Radeon HD-5770.

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

gpu: drm: amd/radeon: Convert printk(KERN_<LEVEL> to pr_<level>

Use a more common logging style.

Miscellanea:

o Coalesce formats and realign arguments
o Neaten a few macros now using pr_<level>

Signed-off-by: Joe Perches <joe@perches.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: allow unaligned shader loads on CIK

Set alignment mode to unaligned on CIK to align with amdgpu. This is
needed for unaligned loads to work properly in mesa. The current setting
requires dword alignment.

Signed-off-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: squash lines for simple wrapper functions

Remove unneeded variables and assignments.

Reviewed-by: Gustavo Padovan <gustavo.padovan@collabora.co.uk>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Sean Paul <seanpaul@chromium.org>
Link: http://patchwork.freedesktop.org/patch/msgid/1473863952-7658-3-git-send-email-yamada.masahiro@socionext.com

drm/radeon: switch UVD code to use UVD_NO_OP for padding

Replace packet2's with packet0 writes to UVD_NO_OP. The
value written to UVD_NO_OP does not matter.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: remove dead code (v2)

In an if block for (running == 0) running cannot be non-zero.

v2: agd: remove unused variable

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: fix CP jump table size

Align to the jump table offset. May fix hangs on some
asics with GFX PG enabled.

Reviewed-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
Acked-by: Tom St Denis <tom.stdenis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/gfx7: expand cp jt size to handle GDS as well

The size needs to handle the CP JT and GDS.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: load different smc firmware on some CI variants

The power tables on some variants require different firmware.
This may fix stability issues on some newer CI parts.

bug:
https://bugs.freedesktop.org/show_bug.cgi?id=91880

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: allow to force hard GPU reset.

In some cases, like when freezing for hibernation, we need to be
able to force hard reset even if no engine are stuck. This patch
add a bool option to current asic reset callback to allow to force
hard reset on asic that supports it.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: consolidate cik vce initialization and startup code.

This match the exact same control flow as existing code. It just
use goto instead of multiple levels of if/else. It also clarify
early initialization failures by clearing rdev->has_vce doing so
does not change end result from hardware point of view, it only
avoids printing more error messages down the line and thus only
the original error is reported.

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: consolidate cik uvd initialization and startup code.

This match the exact same control flow as existing code. It just
use goto instead of multiple levels of if/else. It also clarify
early initialization failures by clearing rdev->has_uvd doing so
does not change end result from hardware point of view, it only
avoids printing more error messages down the line and thus only
the original error is reported.

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix indentation.

I hate doing this but it hurts my eyes to go over code that does not
comply with indentation rules. Only thing that is not only space change
is in atom.c all other files are space indentation issues.

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: refactor CIK tiling table initialization

Simplify the control flow of cik_tiling_mode_table_init() similar to how
it was done in gfx_v7_0.c and gfx_v8_0.c.

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Avoid double gpu reset by adding a timeout on IB ring tests.

When the radeon driver resets a gpu, it attempts to test whether all the
rings can successfully handle an IB. If these rings fail to respond, the
process will wait forever. Another gpu reset can't happen at this point,
as the current reset holds a lock required to do so. Instead, make all
the IB tests run with a timeout, so the system can attempt to recover
in this case.

While this doesn't fix the underlying issue with card resets failing, it
gives the system a higher chance of recovering. These timeouts have been
confirmed to help both a Tathi and Hawaii card recover after a gpu reset.

This also adds a new function, radeon_fence_wait_timeout, that behaves like
fence_wait_timeout. It is used instead of fence_wait_timeout as it continues
to work during a reset. radeon_fence_wait is changed to be implemented
using this function.

V2:
- Changed the timeout to 1s, as the default 10s from radeon_wait_timeout was
too long. A timeout of 100ms was tested and found to be too short.
- Changed radeon_fence_wait_timeout to behave more like fence_wait_timeout.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Matthew Dawson <matthew@mjdsystems.ca>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix typo in cik_ring_ib_execute documentation (v2)

v2: agd: clarify commit message, fix "an" as spotted by Michel.

Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Nicolai Hähnle <nicolai.haehnle@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Fixup hw vblank counter/ts for new drm_update_vblank_count() (v2)

commit 4dfd6486 "drm: Use vblank timestamps to guesstimate how many
vblanks were missed" introduced in Linux 4.4-rc1 makes the drm core
more fragile to drivers which don't update hw vblank counters and
vblank timestamps in sync with firing of the vblank irq and
essentially at leading edge of vblank.

This exposed a problem with radeon-kms/amdgpu-kms which do not
satisfy above requirements:

The vblank irq fires a few scanlines before start of vblank, but
programmed pageflips complete at start of vblank and
vblank timestamps update at start of vblank, whereas the
hw vblank counter increments only later, at start of vsync.

This leads to problems like off by one errors for vblank counter
updates, vblank counters apparently going backwards or vblank
timestamps apparently having time going backwards. The net result
is stuttering of graphics in games, or little hangs, as well as
total failure of timing sensitive applications.

See bug #93147 for an example of the regression on Linux 4.4-rc:

https://bugs.freedesktop.org/show_bug.cgi?id=93147

This patch tries to align all above events better from the
viewpoint of the drm core / of external callers to fix the problem:

1. The apparent start of vblank is shifted a few scanlines earlier,
so the vblank irq now always happens after start of this extended
vblank interval and thereby drm_update_vblank_count() always samples
the updated vblank count and timestamp of the new vblank interval.

To achieve this, the reporting of scanout positions by
radeon_get_crtc_scanoutpos() now operates as if the vblank starts
radeon_crtc->lb_vblank_lead_lines before the real start of the hw
vblank interval. This means that the vblank timestamps which are based
on these scanout positions will now update at this earlier start of
vblank.

2. The driver->get_vblank_counter() function will bump the returned
vblank count as read from the hw by +1 if the query happens after
the shifted earlier start of the vblank, but before the real hw increment
at start of vsync, so the counter appears to increment at start of vblank
in sync with the timestamp update.

3. Calls from vblank irq-context and regular non-irq calls are now
treated identical, always simulating the shifted vblank start, to
avoid inconsistent results for queries happening from vblank irq vs.
happening from drm_vblank_enable() or vblank_disable_fn().

4. The radeon_flip_work_func will delay mmio programming a pageflip until
the start of the real vblank iff it happens to execute inside the shifted
earlier start of the vblank, so pageflips now also appear to execute at
start of the shifted vblank, in sync with vblank counter and timestamp
updates. This to avoid some races between updates of vblank count and
timestamps that are used for swap scheduling and pageflip execution which
could cause pageflips to execute before the scheduled target vblank.

The lb_vblank_lead_lines "fudge" value is calculated as the size of
the display controllers line buffer in scanlines for the given video
mode: Vblank irq's are triggered by the line buffer logic when the line
buffer refill for a video frame ends, ie. when the line buffer source read
position enters the hw vblank. This means that a vblank irq could fire at
most as many scanlines before the current reported scanout position of the
crtc timing generator as the number of scanlines the line buffer can
maximally hold for a given video mode.

This patch has been successfully tested on a RV730 card with DCE-3 display
engine and on a evergreen card with DCE-4 display engine, in single-display
and dual-display configuration, with different video modes.

A similar patch is needed for amdgpu-kms to fix the same problem.

Limitations:

- Line buffer sizes in pixels are hard-coded on < DCE-4 to a value
i just guessed to be high enough to work ok, lacking info on the true
sizes atm.

Fixes: fdo#93147

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Michel Dänzer <michel.daenzer@amd.com>
Cc: Harry Wentland <Harry.Wentland@amd.com>
Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>

(v1) Tested-by: Dave Witbrodt <dawitbro@sbcglobal.net>

(v2) Refine radeon_flip_work_func() for better efficiency:

In radeon_flip_work_func, replace the busy waiting udelay(5)
with event lock held by a more performance and energy efficient
usleep_range() until at least predicted true start of hw vblank,
with some slack for scheduler happiness. Release the event lock
during waits to not delay other outputs in doing their stuff, as
the waiting can last up to 200 usecs in some cases.

Retested on DCE-3 and DCE-4 to verify it still works nicely.

(v2) Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

radeon/cik: Fix GFX IB test on Big-Endian

This patch makes the IB test on the GFX ring pass for CI-based cards
installed in Big-Endian machines.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@gmail.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Fixup hw vblank counter/ts for new drm_update_vblank_count() (v2)

commit 4dfd6486 "drm: Use vblank timestamps to guesstimate how many
vblanks were missed" introduced in Linux 4.4-rc1 makes the drm core
more fragile to drivers which don't update hw vblank counters and
vblank timestamps in sync with firing of the vblank irq and
essentially at leading edge of vblank.

This exposed a problem with radeon-kms/amdgpu-kms which do not
satisfy above requirements:

The vblank irq fires a few scanlines before start of vblank, but
programmed pageflips complete at start of vblank and
vblank timestamps update at start of vblank, whereas the
hw vblank counter increments only later, at start of vsync.

This leads to problems like off by one errors for vblank counter
updates, vblank counters apparently going backwards or vblank
timestamps apparently having time going backwards. The net result
is stuttering of graphics in games, or little hangs, as well as
total failure of timing sensitive applications.

See bug #93147 for an example of the regression on Linux 4.4-rc:

https://bugs.freedesktop.org/show_bug.cgi?id=93147

This patch tries to align all above events better from the
viewpoint of the drm core / of external callers to fix the problem:

1. The apparent start of vblank is shifted a few scanlines earlier,
so the vblank irq now always happens after start of this extended
vblank interval and thereby drm_update_vblank_count() always samples
the updated vblank count and timestamp of the new vblank interval.

To achieve this, the reporting of scanout positions by
radeon_get_crtc_scanoutpos() now operates as if the vblank starts
radeon_crtc->lb_vblank_lead_lines before the real start of the hw
vblank interval. This means that the vblank timestamps which are based
on these scanout positions will now update at this earlier start of
vblank.

2. The driver->get_vblank_counter() function will bump the returned
vblank count as read from the hw by +1 if the query happens after
the shifted earlier start of the vblank, but before the real hw increment
at start of vsync, so the counter appears to increment at start of vblank
in sync with the timestamp update.

3. Calls from vblank irq-context and regular non-irq calls are now
treated identical, always simulating the shifted vblank start, to
avoid inconsistent results for queries happening from vblank irq vs.
happening from drm_vblank_enable() or vblank_disable_fn().

4. The radeon_flip_work_func will delay mmio programming a pageflip until
the start of the real vblank iff it happens to execute inside the shifted
earlier start of the vblank, so pageflips now also appear to execute at
start of the shifted vblank, in sync with vblank counter and timestamp
updates. This to avoid some races between updates of vblank count and
timestamps that are used for swap scheduling and pageflip execution which
could cause pageflips to execute before the scheduled target vblank.

The lb_vblank_lead_lines "fudge" value is calculated as the size of
the display controllers line buffer in scanlines for the given video
mode: Vblank irq's are triggered by the line buffer logic when the line
buffer refill for a video frame ends, ie. when the line buffer source read
position enters the hw vblank. This means that a vblank irq could fire at
most as many scanlines before the current reported scanout position of the
crtc timing generator as the number of scanlines the line buffer can
maximally hold for a given video mode.

This patch has been successfully tested on a RV730 card with DCE-3 display
engine and on a evergreen card with DCE-4 display engine, in single-display
and dual-display configuration, with different video modes.

A similar patch is needed for amdgpu-kms to fix the same problem.

Limitations:

- Line buffer sizes in pixels are hard-coded on < DCE-4 to a value
i just guessed to be high enough to work ok, lacking info on the true
sizes atm.

Fixes: fdo#93147

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Michel Dänzer <michel.daenzer@amd.com>
Cc: Harry Wentland <Harry.Wentland@amd.com>
Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>

(v1) Tested-by: Dave Witbrodt <dawitbro@sbcglobal.net>

(v2) Refine radeon_flip_work_func() for better efficiency:

In radeon_flip_work_func, replace the busy waiting udelay(5)
with event lock held by a more performance and energy efficient
usleep_range() until at least predicted true start of hw vblank,
with some slack for scheduler happiness. Release the event lock
during waits to not delay other outputs in doing their stuff, as
the waiting can last up to 200 usecs in some cases.

Retested on DCE-3 and DCE-4 to verify it still works nicely.

(v2) Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Retry DDC probing on DVI on failure if we got an HPD interrupt

HPD signals on DVI ports can be fired off before the pins required for
DDC probing actually make contact, due to the pins for HPD making
contact first. This results in a HPD signal being asserted but DDC
probing failing, resulting in hotplugging occasionally failing.

This is somewhat rare on most cards (depending on what angle you plug
the DVI connector in), but on some cards it happens constantly. The
Radeon R5 on the machine used for testing this patch for instance, runs
into this issue just about every time I try to hotplug a DVI monitor and
as a result hotplugging almost never works.

Rescheduling the hotplug work for a second when we run into an HPD
signal with a failing DDC probe usually gives enough time for the rest
of the connector's pins to make contact, and fixes this issue.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Lyude <cpaul@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Handle irqs only based on irq ring, not irq status regs.

Trying to resolve issues with missed vblanks and impossible
values inside delivered kms pageflip completion events showed
that radeon's irq handling sometimes doesn't handle valid irqs,
but silently skips them. This was observed for vblank interrupts.

Although those irqs have corresponding events queued in the gpu's
irq ring at time of interrupt, and therefore the corresponding
handling code gets triggered by these events, the handling code
sometimes silently skipped processing the irq. The reason for those
skips is that the handling code double-checks for each irq event if
the corresponding irq status bits in the irq status registers
are set. Sometimes those bits are not set at time of check
for valid irqs, maybe due to some hardware race on some setups?

The problem only seems to happen on some machine + card combos
sometimes, e.g., never happened during my testing of different PC
cards of the DCE-2/3/4 generation a year ago, but happens consistently
now on two different Apple Mac cards (RV730, DCE-3, Apple iMac and
Evergreen JUNIPER, DCE-4 in a Apple MacPro). It also doesn't happen
at each interrupt but only occassionally every couple of
hundred or thousand vblank interrupts.

This results in XOrg warning messages like

"[ 7084.472] (WW) RADEON(0): radeon_dri2_flip_event_handler:
Pageflip completion event has impossible msc 420120 < target_msc 420121"

as well as skipped frames and problems for applications that
use kms pageflip events or vblank events, e.g., users of DRI2 and
DRI3/Present, Waylands Weston compositor, etc. See also

https://bugs.freedesktop.org/show_bug.cgi?id=85203

After some talking to Alex and Michel, we decided to fix this
by turning the double-check for asserted irq status bits into a
warning. Whenever a irq event is queued in the IH ring, always
execute the corresponding interrupt handler. Still check the irq
status bits, but only to log a DRM_DEBUG message on a mismatch.

This fixed the problems reliably on both previously failing
cards, RV-730 dual-head tested on both crtcs (pipes D1 and D2)
and a triple-output Juniper HD-5770 card tested on all three
available crtcs (D1/D2/D3). The r600 and evergreen irq handling
is therefore tested, but the cik an si handling is only compile
tested due to lack of hw.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
CC: Michel Dänzer <michel.daenzer@amd.com>
CC: Alex Deucher <alexander.deucher@amd.com>
CC: <stable@vger.kernel.org> # v3.16+
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: compute ring fix hibernation (CI GPU family) v2.

In order for hibernation to reliably work we need to cleanup more
thoroughly the compute ring. Hibernation is different from suspend
resume as when we resume from hibernation the hardware is first
fully initialize by regular kernel then freeze callback happens
(which correspond to a suspend inside the radeon kernel driver)
and turn off each of the block. It turns out we were not cleanly
shutting down the compute ring. This patch fix that.

Hibernation and suspend to ram were tested (several times) on :
Bonaire
Hawaii
Mullins
Kaveri
Kabini

Changed since v1:
- Factor the ring stop logic into a function taking ring as arg.

Cc: stable@vger.kernel.org
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

radeon: Deinline indirect register accessor functions

This patch deinlines indirect register accessor functions.

These functions perform two mmio accesses, framed by spin lock/unlock.
Spin lock/unlock by itself takes more than 50 cycles in ideal case
(if lock is exclusively cached on current CPU).

With this .config: http://busybox.net/~vda/kernel_config,
after uninlining these functions have sizes and callsite counts
as follows:

r600_uvd_ctx_rreg: 111 bytes, 4 callsites
r600_uvd_ctx_wreg: 113 bytes, 5 callsites
eg_pif_phy0_rreg: 106 bytes, 13 callsites
eg_pif_phy0_wreg: 108 bytes, 13 callsites
eg_pif_phy1_rreg: 107 bytes, 13 callsites
eg_pif_phy1_wreg: 108 bytes, 13 callsites
rv370_pcie_rreg: 111 bytes, 21 callsites
rv370_pcie_wreg: 113 bytes, 24 callsites
r600_rcu_rreg: 111 bytes, 16 callsites
r600_rcu_wreg: 113 bytes, 25 callsites
cik_didt_rreg: 106 bytes, 10 callsites
cik_didt_wreg: 107 bytes, 10 callsites
tn_smc_rreg: 106 bytes, 126 callsites
tn_smc_wreg: 107 bytes, 116 callsites
eg_cg_rreg: 107 bytes, 20 callsites
eg_cg_wreg: 108 bytes, 52 callsites

Functions r100_mm_rreg() and r100_mm_rreg() have a fast path and
a locked (slow) path. This patch deinlines only slow path.

r100_mm_rreg_slow: 78 bytes, 2083 callsites
r100_mm_wreg_slow: 81 bytes, 3570 callsites

Reduction in code size is more than 65,000 bytes:

text data bss dec hex filename
85740176 22294680 20627456 128662312 7ab3b28 vmlinux.before
85674192 22294776 20627456 128598664 7aa4288 vmlinux

Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: partially revert "fix VM_CONTEXT*_PAGE_TABLE_END_ADDR handling"

We have that bug for years and some users report side effects when fixing it on older hardware.

So revert it for VM_CONTEXT0_PAGE_TABLE_END_ADDR, but keep it for VM 1-15.

Signed-off-by: Christian König <christian.koenig@amd.com>
CC: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix VM_CONTEXT*_PAGE_TABLE_END_ADDR handling

The mapping range is inclusive between starting and ending addresses.

Signed-off-by: Christian König <christian.koenig@amd.com>
CC: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

radeon/cik: add support for short HPD irqs

This adds support to process short HPD irqs on CIK gpus.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add get_allowed_info_register for CIK

Registers that can be fetched from the info ioctl.

Tested-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: do a posting read in cik_set_irq

To make sure the writes go through the pci bridge.

bug:
https://bugzilla.kernel.org/show_bug.cgi?id=90741

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: enable SRBM timeout interrupt on CIK v2

v2: disable it on suspend

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: only enable kv/kb dpm interrupts once v3

Enable at init and disable on fini. Workaround for hardware problems.

v2 (chk): extend commit message
v3: add new function

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com> (v2)
Cc: stable@vger.kernel.org

drm/radeon: workaround for CP HW bug on CIK

Emit the EOP twice to avoid cache flushing problems.

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

radeon/audio: consolidate audio_fini() functions

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Slava Grigorev <slava.grigorev@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

radeon/audio: consolidate audio_init() functions

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Slava Grigorev <slava.grigorev@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix VM flush on CIK (v3)

We need to wait for the GPUVM flush to complete. There
was some confusion as to how this mechanism was supposed
to work. The operation is not atomic. For GPU initiated
invalidations you need to read back a VM register to
introduce enough latency for the update to complete.

v2: drop gart changes
v3: just read back rather than polling

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: use one VMID for each ring

Use multiple VMIDs for each VM, one for each ring. That allows
us to execute flushes separately on each ring, still not ideal
cause in a lot of cases rings can share IDs.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: split semaphore and sync object handling v2

Previously we just allocated space for four hardware semaphores
in each software semaphore object. Make software semaphore objects
represent only one hardware semaphore address again by splitting
the sync code into it's own object.

v2: fix typo in comment

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: rework vm_flush parameters

Use ring structure instead of index and provide vm_id and pd_addr separately.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: work around a hw bug in MGCG on CIK

Always need to set bit 0 of RLC_CGTT_MGCG_OVERRIDE
to avoid unreliable doorbell updates in some cases.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix for memory training on bonaire 0x6649

Workaround for memory link training on certain variants
of 0x6649.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: make sure mode init is complete in bandwidth_update

The power management code calls into the display code for
certain things. If certain power management sysfs attributes
are called before the driver has finished initializing all of
the hardware we can run into problems with uninitialized
modesetting state. Add a check to make sure modesetting
init has completed to the bandwidth update callbacks to
fix this. Can be triggered by the tlp and laptop start
up scripts depending on the timing.

bugs:
https://bugzilla.kernel.org/show_bug.cgi?id=83611
https://bugs.freedesktop.org/show_bug.cgi?id=85771

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: set correct CE ram size for CIK

CE ram size is 32k/0k/0k for GFX/CS0/CS1 with CIK

Ported from amdgpu driver.

Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: export reservation_object from dmabuf to ttm

Adds an extra argument to radeon_bo_create, which is only used in radeon_prime.c.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: cope with foreign fences inside the reservation object

Not the whole world is a radeon! :-)

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: write gfx ucode version to ucode addr reg

Helpful for debugging as the version shows up in a
register dump.

Cc: Jay Cornwall <jay.cornwall@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: use a separate counter for CP init timeout

Otherwise we may fail to init the second compute ring.

Noticed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: Update IH_RB_RPTR register after each processed interrupt

This might decrease the chance of IH ring buffer overflows.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Make IH ring overflow debugging output more useful

Use the same format for all ring indices, and fix the calculation of the
post-overflow RPTR.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Clear RB_OVERFLOW bit earlier

Otherwise the bit remains set in rdev->ih.rptr, so the wptr can never
match that and we still have an infinite loop.

This fix allows me to successfully recover from an IH ring buffer
overflow.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add the infrastructure for concurrent buffer access

This allows us to specify if we want to sync to
the shared fences of a reservation object or not.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

radeon: Test for PCI root bus before assuming bus->self

If we assign a Radeon device to a virtual machine, we can no longer
assume a fixed hardware topology, like the GPU having a parent device.
This patch simply adds a few pci_is_root_bus() tests to avoid passing
a NULL pointer to PCI access functions, allowing the radeon driver to
work in a QEMU 440FX machine with an assigned HD8570 on the emulated
PCI root bus.

Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: drop doing resets in a work item

Blocking completely innocent processes with a GPU reset is
a pretty bad idea. Just set needs_reset and let the next
command submission or fence wait do the job.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: save/restore the PD addr on suspend/resume

This fixes a problem with GPU resets and TLB flushes on SI/CIK.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add new KV pci id

bug:
https://bugs.freedesktop.org/show_bug.cgi?id=82912

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix active_cu mask on SI and CIK after re-init (v3)

Need to initialize the mask to 0 on init, otherwise it
keeps increasing.

bug:
https://bugzilla.kernel.org/show_bug.cgi?id=82581

v2: also fix cu count
v3: split count fix into separate patch

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix active cu count for SI and CIK

This fixes the CU count reported to userspace for
OpenCL.

bug:
https://bugzilla.kernel.org/show_bug.cgi?id=82581

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: Sync ME and PFP after CP semaphore waits v4

Fixes lockups due to CP read GPUVM faults when running piglit on Cape
Verde.

v2 (chk): apply the fix to R600+ as well, on CIK only the GFX CP has
a PFP, add more comments to R600 code, enable flushing again
v3: (agd5f): only apply to 7xx+. r6xx does not have the packet.
v4: (agd5f): split flush change into a separate patch, fix formatting

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Tested-by: Michel Dänzer <michel.daenzer@amd.com>

drm/radeon: Only flush HDP cache for indirect buffers from userspace

It isn't necessary for command streams generated by the kernel (at least
not while we aren't storing ring or indirect buffers in VRAM).

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Initialize compute vmid

This patch moves to radeon the initialization of compute vmid.

That initializations was done in kfd-->kgd interface, but doing it in radeon
as part of radeon's H/W initialization routines is more appropriate.

In addition, this simplifies the kfd-->kgd interface.

The patch removes the function from the interface file and from the interface
declaration file.

The function initializes memory apertures to fixed base/limit address and non
cached memory types.

Signed-off-by: Ben Goz <ben.goz@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use pfp for all vm_flush related updates

May fix hangs in some cases.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Use pflip irqs for pageflip completion if possible. (v2)

Skip the "manual" pageflip completion checks via polling and
guessing in the vblank handler radeon_crtc_handle_vblank() on
asics which are known to reliably support hw pageflip completion
irqs. Those pflip irqs are a more reliable and race-free method
of handling pageflip completion detection, whereas the "classic"
polling method has some small races in combination with dpm on,
and with the reworked pageflip implementation since Linux 3.16.

On old asics without pflip irqs, the classic method is used.

On asics with known good pflip irqs, only pflip irqs are used
by default, but a new module parameter "use_pflipirqs" allows to
override this in case we encounter asics in the wild with
unreliable or faulty pflip irqs. A module parameter of 0 allows
to use the classic method only in such a case. A parameter of 1
allows to use both classic method and pflip irqs as additional
band-aid to avoid some small races which could happen with the
classic method alone. The setting 1 gives Linux 3.16 behaviour.

Hw pflip irqs are available since R600.

Tested on DCE-4, AMD Cedar - FirePro 2270.

v2: agd5f: only enable pflip interrupts on DCE4+ as they are not
reliable on older asics.

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use packet3 for nop on hawaii with new firmware

Older firmware didn't support the new nop packet.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Andreas Boll <andreas.boll.dev@gmail.com>

drm/radeon: use packet2 for nop on hawaii with old firmware

Older firmware didn't support the new nop packet.

v2 (Andreas Boll):
- Drop usage of packet3 for new firmware

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com> (v1)
Signed-off-by: Andreas Boll <andreas.boll.dev@gmail.com>
Cc: stable@vger.kernel.org

drm/radeon: Always flush the HDP cache before submitting a CS to the GPU

This ensures the GPU sees all previous CPU writes to VRAM, which makes it
safe:

* For userspace to stream data from CPU to GPU via VRAM instead of GTT
* For IBs to be stored in VRAM instead of GTT
* For ring buffers to be stored in VRAM instead of GTT, if the HPD flush
is performed via MMIO

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: set VM base addr using the PFP v2

Seems to make VM flushes more stable on SI and CIK.

v2: only use the PFP on the GFX ring on CIK

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Allow write-combined CPU mappings of BOs in GTT (v2)

v2: fix rebase onto drm-fixes

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Remove radeon_gart_restore()

Doesn't seem necessary, the GART table memory should be persistent.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: Add support for new ucode format (v5)

This adds CIK support for the new ucode format.

v2: add size validation, integrate debug info
v3: add support for MEC2 on KV
v4: fix typos
v4: update to latest format

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix cut and paste issue for hawaii.

This is a halfway fix for hawaii acceleration. More fixes to come
but hopefully isolated to userspace.

Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon: fix irq ring buffer overflow handling

We must mask out the overflow bit as well, otherwise
the wptr will never match the rptr again and the interrupt
handler will loop forever.

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>

drm/radeon: Add radeon <--> amdkfd interface

This patch adds the interface between the radeon driver and the amdkfd driver.
The interface implementation is contained in radeon_kfd.c and radeon_kfd.h.

The interface itself is represented by a pointer to struct
kfd_dev. The pointer is located inside radeon_device structure.

All the register accesses that amdkfd need are done using this interface. This
allows us to avoid direct register accesses in amdkfd proper, while also
avoiding locking between amdkfd and radeon.

The single exception is the doorbells that are used in both of the drivers.
However, because they are located in separate pci bar pages, the danger of
sharing registers between the drivers is minimal.

Having said that, we are planning to move the doorbells as well to radeon.

v3:

Add interface for sa manager init and fini. The init function will allocate a
buffer on system memory and pin it to the GART address space via the radeon sa
manager.

All mappings of buffers to GART address space are done via the radeon sa
manager. The interface of allocate memory will use the radeon sa manager to sub
allocate from the single buffer that was allocated during the init function.

Change lower_32/upper_32 calls to use linux macros

Add documentation for the interface

v4:

Change ptr field type in kgd_mem from uint32_t* to void* to match to type that
is returned by radeon_sa_bo_cpu_addr

v5:

Change format of mqd structure to work with latest KV firmware
Add support for AQL queues creation to enable working with open-source HSA
runtime.
Move generic kfd-->kgd interface and other generic kgd definitions to a generic
header file that will be used by AMD's radeon and amdgpu drivers

Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>

drm/radeon: adding synchronization for GRBM GFX

Implementing a lock for selecting and accessing shader engines and arrays.
This lock will make sure that radeon and amdkfd are not colliding when
accessing shader engines and arrays with GRBM_GFX_INDEX register.

Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>

drm/radeon: only print meaningful VM faults

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add query for number of active CUs

Query to find out how many compute units on a GPU.
Useful for OpenCL usermode drivers.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: make vm_block_size a module parameter

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use lower_32_bits where appropriate

Replace occurrences of "v & 0xffffffff" with lower_32_bits(v)
when it's next to an upper_32_bits(v). Also remove unnecessary
"upper_32_bits(v) & 0xffffffff" code snippets.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: separate vblank and pflip crtc handling

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add proper support for RADEON_VM_BLOCK_SIZE v2

This patch makes it possible to decide how many address
bits are spend on the page directory vs the page tables.

v2: remove unintended change

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add large PTE support for NI, SI and CIK v5

This patch implements support for VRAM page table entry compression.
PTE construction is enhanced to identify physically contiguous page
ranges and mark them in the PTE fragment field. L1/L2 TLB support is
enabled for 64KB (SI/CIK) and 256KB (NI) PTE fragments, significantly
improving TLB utilization for VRAM allocations.

Linear store bandwidth is improved from 60GB/s to 125GB/s on Pitcairn.
Unigine Heaven 3.0 sees an average improvement from 24.7 to 27.7 FPS
on default settings at 1920x1200 resolution with vsync disabled.

See main comment in radeon_vm.c for a technical description.

v2 (chk): rebased and simplified.
v3 (chk): add missing hw setup
v4 (chk): rebased on current drm-fixes-3.15
v5 (chk): fix comments and commit text

Signed-off-by: Jay Cornwall <jay@jcornwall.me>
Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: update cik init for Mullins.

Also add golden registers, update firmware loading functions.

Signed-off-by: Samuel Li <samuel.li@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/radeon: drm/radeon: add missing radeon_semaphore_free to error path

It would appear this bug has been copy/pasted many times without being noticed.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use pflip irq on R600+ v2

Testing the update pending bit directly after issuing an
update is nonsense cause depending on the pixel clock the
CRTC needs a bit of time to execute the flip even when we
are in the VBLANK period.

This is just a non invasive patch to solve the problem at
hand, a more complete and cleaner solution should follow
in the next merge window.

Fixes: https://bugs.freedesktop.org/show_bug.cgi?id=76564

v2: fix source IDs for CRTC2-6

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org

drm/radeon/ci: make sure mc ucode is loaded before checking the size

Avoid a possible segfault.

Noticed-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: add support for newer mc ucode on CI (v2)

Fixes mclk stability on certain asics.

v2: print out mc firmware version used and size

bug:
https://bugs.freedesktop.org/show_bug.cgi?id=75992

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix typo in spectre_golden_registers

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix endian swap on hawaii clear state buffer setup

Need to swap on BE.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: set PIPE_CONFIG for 1D and linear tiling modes on CIK

This fixes fast color clear with 1D-tiled single-sample surfaces
and Hyper-Z corruption with 1D-tiled depth surfaces.

Even though it seems it is not needed for 1D tiling, CMASK and HTILE are
always 2D-tiled, thus the hw needs to know the actual pipe configuration
for CMASK and HTILE addressing no matter what the tiling mode of the surface
is.

Signed-off-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/radeon/cik: properly set compute ring status on disable

When we disable the rings, set the status properly. If
not other code pathes may try and use the rings which are
not functional at this point.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon/cik: fix typo in documentation

Copy-paste typo.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: resume old pm late

Moving the pm resume up in the init order to fix
dpm seems to have regressed somes cases with the old
pm code. Move it back to late resume.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: drop radeon_ring_force_activity

The reason for the false positives was fixed quite some time ago and since
most engines can still execute NOPs while being locked up it leads to false
negatives.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: drop drivers copy of the rptr

In all cases where it really matters we are using the read functions anyway.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: enable/disable vce cg when encoding v2

Some of the vce clocks are automatic, others need to
be manually enabled. For ease, just disable cg when
vce is active.

v2: rebased

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fill in set_vce_clocks for CIK asics

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: initial VCE support v4

Only VCE 2.0 support so far.

v2: squashing multiple patches into this one
v3: add IRQ support for CIK, major cleanups,
basic code documentation
v4: remove HAINAN from chipset list

Signed-off-by: Christian König <christian.koenig@amd.com>

drm/radeon: clean up active vram sizing

If we are not able to properly initialize one of the gpu
engines for buffer paging, we limit vram to the size of
the cpu visible aperture. We generally either use the gfx
or dma engine to do this. Clean up the size limiting code
to only adjust the size based on what ring is selected
for buffer paging rather than making assumptions about which
engine is selected for paging.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon/cik: Don't touch int of pipes 1-7

amdkfd should set interrupts for pipes 1-7.

Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>

drm/radeon/cik: use hw defaults for TC_CFG registers

Use the hw power up values rather than 0.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: use WAIT_REG_MEM special op for CP HDP flush

This is the preferred flushing method on CIK.

Note, this only works on the PFP so the engine bit must be
set.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: consolidate cp hdp flushing code for CIK

It's used in several places so move to a common shared
function.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

Revert "drm/radeon: disable CIK CP semaphores for now"

This reverts commit 99b4f25122f43210278cde17a9d100906235a074.

Semaphores work fine after further review and testing.
Cc: 3.13 <stable@vger.kernel.org> # 3.13

drm/radeon: reduce number of free VMIDs and pipes in KV

To support HSA on KV, we need to limit the number of vmids and pipes
that are available for radeon's use with KV.

This patch reserves VMIDs 8-15 for amdkfd (so radeon can only use VMIDs
0-7) and also makes radeon thinks that KV has only a single MEC with a single
pipe in it

v3: Use define for static vmid allocation in radeon

Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Oded Gabbay <oded.gabbay@amd.com>

drm/radeon: implement pci config reset for CIK (v3)

pci config reset is a low level reset that resets
the entire chip from the bus interface. It can
be more reliable if soft reset fails.

v2: fix rebase
v3: hide behind module parameter

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove generic rptr/wptr functions (v2)

Fill in asic family specific versions rather than
using the generic version. This lets us handle asic
specific differences more easily. In this case, we
disable sw swapping of the rtpr writeback value on
r6xx+ since the hw does it for us. Fixes bogus
rptr readback on BE systems.

v2: remove missed cpu_to_le32(), add comments

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/pm: move pm handling into the asic specific code

We need more control over the ordering of dpm init with
respect to the rest of the asic. Specifically, the SMC
has to be initialized before the rlc and cg/pg. The pm
code currently initializes late in the driver, but we need
it to happen much earlier so move pm handling into the asic
specific callbacks.

This makes dpm more reliable and makes clockgating work
properly on CIK parts and should help on SI parts as well.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: re-order firmware loading in preparation for dpm rework

We need to reorder the driver init sequence to better accomodate
dpm which needs to be loaded earlier in the init sequence. Move
fw init up so that it's available for dpm init.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: expose render backend mask to the userspace

This will allow userspace to correctly program the PA_SC_RASTER_CONFIG
register, so it can be considered a fix.

Signed-off-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix render backend setup for SI and CIK

Only the render backends of the first shader engine were enabled. The others
were erroneously disabled. Enabling the other render backends improves
performance a lot.

Unigine Sanctuary on Bonaire:
Before: 15 fps
After: 90 fps

Judging from the fan noise, the GPU was also underclocked when the other
render backends were disabled, resulting in horrible performance. The fan is
a lot noisy under load now.

Signed-off-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon/cik: Add macrotile mode array query

This is required to properly calculate the tiling parameters
in userspace.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use a single doorbell for cik kms compute

A single doorbell page is plenty for cik kms compute.
Use a single page and manage doorbell allocation by
individual doorbells rather than pages. Identify
doorbells by their index rather than byte offset.

Signed-off-by: Andrew Lewycky <Andrew.Lewycky@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: disable CIK CP semaphores for now

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: allow semaphore emission to fail

To workaround bugs and/or certain limits it's sometimes
useful to fall back to waiting on fences.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: update cik_get_csb_buffer for hawaii

Set the PA_SC_RASTER_CONFIG[_1] registers for hawaii.
The rest is the same as the other asics.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: update firmware loading for hawaii

This just updates the firmware loading functions
to look for the appropriate firmware files for
hawaii.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: update rb setup for hawaii

The formula needs to be adjusted since there are 4 RBs
per SH rather than 2 as on previous asics.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add golden register settings for hawaii

The golden register settings are optimal settings for
certain registers from the hardware team.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: update cik_tiling_mode_table_init() for hawaii

Hawaii uses a different tiling configuration. Add support
for it.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: minor updates to cik.c for hawaii

Skip programming a register that was removed and
adjust the mask of the VM client status.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: update cik_gpu_init() for hawaii

This adds the hawaii asic specific configuration
details.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: drop CP page table updates & cleanup v2

The DMA ring seems to be stable now.

v2: remove pt_ring_index as well

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: don't use PACKET2 on CIK

It is said to cause hangs.

Signed-off-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix endian handling in rlc buffer setup

The buffers needs to be in little endian format.

Noticed-by: Sylvain BERTRAND <sylware@legeek.net>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: implement blit copy callback for CIK

Uses the CP ring rather than the DMA ring. Useful
for debugging and benchmarking.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add a connector property for dither

Allows you to enable dither in the display hardware
when the monitor supports lower a lower bpc than the
current framebuffer format.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Add support for programming the FMT blocks

The FMT blocks control how data is sent from the backend
of the display pipe to to monitor. Proper set up of the
FMT blocks are required for 30bpp formats. Additionally,
dithering can be enabled on for better display with 18 and
24bpp displays. The exception is LVDS/eDP which atom
takes care of in the SelectCRTC_Source table. For now
just enable truncation until we test dithering more.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: make missing smc ucode non-fatal (CI)

Prevent driver load problems if the smc is missing.

bug:
https://bugzilla.kernel.org/show_bug.cgi?id=63011

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: stop the leaks in cik_ib_test

Stop leaking IB memory and scratch register space when the test fails.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: improve soft reset on CIK

Disable CG/PG before resetting.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: fix overflow in vram fetch

Missing ULL when calculating the amount of vram
leads to an overflow when the amount of vram is >= 4G.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon/cik: Fix encoding of number of banks in tiling configuration info

There are multiple valid values, not just 0 or 1. Required
to properly support 2D tiling in the userspace drivers.

Cc: stable@vger.kernel.org
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: Fix printing of client name on VM protection fault

The string is encoded from the MSB to the LSB of the register.

Cc: stable@vger.kernel.org
Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix typo in PG flags

s/CG/PG/ in the GFX powergating flag name.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: update gpu_init for an additional berlin gpu

Sets the right paramters for the new pci id.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: add spinlocks for indirect register accesss

This adds spinlocks to protect access to other
indirect register apertures. These indirect spaces are
used pretty infrequently and we haven't had an reported
problems, but better safe than sorry.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: properly handle internal cp ints

The internal cp interrupts need to be enabled and
disabled at specific times in order clockgating to
work properly. This patch changes the handling
of the CP_INT_CNTL register to respect the current
state of the internal CP interrupts when making
changes to the other interrupts in CP_INT_CNTL.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: enable UVD interrupts on CIK

The same as on evergreen.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reported-by: FrankR Huang <FrankR.Huang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix init ordering for r600+

The vram scratch buffer needs to be initialized
before the mc is programmed otherwise we program
0 as the GPU address of the default GPU fault
page. In most cases we put vram at zero anyway and
reserve a page for the legacy vga buffer so in practice
this shouldn't cause any problems, but better to make
it correct.

Was changed in:
6fab3febf6d949b0a12b1e4e73db38e4a177a79e

Reported-by: FrankR Huang <FrankR.Huang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: split out radeon_uvd_resume from uvd_v4_2_resume

For powergating, we just need to re-init the registers, there
is no need to restore the uvd BOs. This just adds needless
work when powergating uvd for playback while the system is
on. We only need to restore the uvd BOs on an actual resume
from suspend or when the driver loads.

This fixes multi-stream UVD playback on KB systems.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon: fill in gpu_init for berlin GPU cores

This fills in the GPU specific details for berlin
GPU cores so that the driver will work with them.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: update line buffer allocation for dce8

We need to allocate line buffer to each display when
setting up the watermarks. Failure to do so can lead
to a blank screen. This fixes blank screen problems
on dce8 asics.

Based on an initial fix from:
Jay Cornwall <jay.cornwall@amd.com>

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon/cik: properly set up the clearstate buffer for pg (v2)

The format of the clearstate buffer used for pg (powergating)
changed between NI and SI. This formats it properly for what
the hardware expects on SI+.

v2: fix addresses

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fixes for gfx clockgating on CIK

Clockgating requires signalling between the CP and the
RLC to work properly. Resetting the CP block in the
CP resume code messed up the internal coordination
between the blocks. Removing the reset allows gfx
clockgating to work properly. However, when gfx clock
gating is enabled, there is a strange interaction with
dpm which causes the chip to stay in the high performance
level all the time, so leave gfx clockgating disabled
for now.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: restructure cg/pg on cik (v2)

- use new cg/pg flags for finer grained clock and
powergating control
- restructure the cg/pg code so it can be called from
other components such as dpm

v2: fix build breakage from rebase

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add audio support for DCE6/8 GPUs (v12)

Similar to DCE4/5, but supports multiple audio pins
which can be assigned per afmt block.

v2: rework the driver to handle more than one audio
pin.
v3: try different dto reg
v4: properly program dto
v5 (ck): change dto programming order
v6: program speaker allocation block
v7: rebase
v8: rebase on Rafał's changes
v9: integrated Rafał's comments, update to latest
drm_edid_to_speaker_allocation API
v10: add missing line break in error message
v11: add back audio enabled messages
v12: fix copy paste typo in r600_audio_enable

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Rafał Miłecki <zajec5@gmail.com>

drm/radeon: separate DMA code

Similar to separating the UVD code, just put the DMA
functions into separate files.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: separate UVD code v3

Our different hardware blocks are actually completely
separated, so it doesn't make much sense any more to
structure the code by pure chipset generations.

Start restructuring the code by separating our the UVD block.

v2: updated commit message
v3: rebased and restructurized start/stop functions for kv dpm.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove special handling for the DMA ring

Now that we have callbacks for [rw]ptr handling we can
remove the special handling for the DMA rings and use
the callbacks instead.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: rework UVD writeback & [rw]ptr handling

The hardware just doesn't support this correctly.
Disable it before we accidentally write anywhere we shouldn't.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/dpm: implement UVD powergating for KB/KV

Powergate the UVD block when not in use to save power.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: restructure UVD code to handle UVD PG (v2)

When we PG (powergate) UVD, we need to re-initialize it
before we can use it again.

v2: rebase on UVD stop fixes

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add dpm support for CI dGPUs (v2)

This adds dpm support for btc asics. This includes:
- dynamic engine clock scaling
- dynamic memory clock scaling
- dynamic voltage scaling
- dynamic pcie gen switching

Set radeon.dpm=1 to enable.

v2: remove unused radeon_atombios.c changes,
make missing smc ucode non-fatal

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/kms: add dpm support for KB/KV

This adds dpm support for KB/KV asics. This includes:
- dynamic engine clock scaling
- dynamic voltage scaling
- power containment
- shader power scaling

Set radeon.dpm=1 to enable.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add get_temperature() callbacks for CIK (v2)

This added support for the on-chip thermal sensors on
CIK asics.

v2: fix register offset.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: add rlc helpers for DPM

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: implement clock and power gating for CIK (v3)

Only the APUs support power gating.

v2: disable cgcg for now
v3: workaround hw issue in mgcg

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: convert SI,CIK to use sumo_rlc functions

and remove duplicate si_rlc functions.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: restructure rlc setup

Restructure rlc setup to handle clock and power
gating.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add support for ASPM on CIK asics

Enables PCIE ASPM (Active State Power Management) on
CIK asics.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: add support for pcie gen1/2/3 switching

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: switch CIK to use radeon_ucode.h

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/kms: fix up dce8 display watermark calc for dpm

Calculate the low and high watermarks based on the low and high
clocks for the current power state. The dynamic pm hw will select
the appropriate watermark based on the internal dpm state.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: use a mutex to properly lock srbm instanced registers

We need proper locking in the driver when accessing instanced
registers on CIK.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: only save UVD bo when we have open handles

Otherwise just reinitialize from scratch on resume,
and so make it more likely to succeed.

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: always program the MC on startup

For r6xx+ asics. This mirrors the behavior of pre-r6xx
asics. We need to program the MC even if something
else in startup() fails. Failure to do so results in
an unusable GPU.

Based on a fix from: Mark Kettenis <kettenis@openbsd.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix halting UVD

Removing the clock/power or resetting the VCPU can cause
hangs if that happens in the middle of a register write.

Stall the memory and register bus before putting the VCPU
into reset. Keep it in reset when unloading the module or
suspending.

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: s/drm_order/order_base_2/

Last driver and pretty obviously a major user of this little function.

Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@gmail.com>

drm/radeon: never unpin UVD bo v3

Changing the UVD BOs offset on suspend/resume doesn't work because the VCPU
internally keeps pointers to it. Just keep it always pinned and save the
content manually.

Fixes: https://bugs.freedesktop.org/show_bug.cgi?id=66425

v2: fix compiler warning
v3: fix CIK support

Note: a version of this patch needs to go to stable.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add fault decode function for CIK

Helpful for debugging GPUVM errors as we can see what
hw block and page generated the fault in the log.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use radeon device for request firmware

Avoid creating temporary platform device that will lead to issue
when several radeon gpu are in same computer. Instead directly use
the radeon device for requesting firmware.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix typo in cik_select_se_sh()

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add cik tile mode array query

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: add support for golden register init

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: add support for compute interrupts

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix up ring functions for compute rings

The compute rings use RELEASE_MEM rather then EOP
packets for writing fences and there is no SYNC_PFP_ME
packet on the compute rings.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: switch to type3 nop packet for compute rings (v2)

Type 2 packets are deprecated on CIK MEC and we should use
type 3 nop packets. Setting the count field to the max value
(0x3fff) indicates that only one dword should be skipped
like a type 2 packet.

v2: add comment to code

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

drm/radeon/cik: Add support for compute queues (v4)

On CIK, the compute rings work slightly differently than
on previous asics, however the basic concepts are the same.

The main differences:
- New MEC engines for compute queues
- Multiple queues per MEC:
- CI/KB: 1 MEC, 4 pipes per MEC, 8 queues per pipe = 32 queues
- KV: 2 MEC, 4 pipes per MEC, 8 queues per pipe = 64 queues
- Queues can be allocated and scheduled by another queue
- New doorbell aperture allows you to assign space in the aperture
for the wptr which allows for userspace access to queues

v2: add wptr shadow, fix eop setup
v3: fix comment
v4: switch to new callback method

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

drm/radeon: implement simple doorbell page allocator

The doorbell aperture is a PCI BAR whose pages can be
mapped to compute resources for things like wptrs
for userspace queues.

This patch maps the BAR and sets up a simple allocator
to allocate pages from the BAR.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: add srbm_select function

Allows us to select instanced registers based on:
- ME (micro engine
- Pipe
- Queue
- VMID

Switch MC setup to use this new function.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add UVD support for CIK (v3)

v2: agd5f: fix clock dividers setup for bonaire
v3: agd5f: rebase

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: add pcie_port indirect register accessors

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add get_xclk() callback for CIK

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: update CIK soft reset

Update to the newer programming model.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add get_gpu_clock_counter() callback for cik

Used for GPU clock counter snapshots.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/dce8: add support for display watermark setup

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: fill in startup/shutdown callbacks (v5)

v2: update to latest driver changes
v3: properly tear down vm on suspend
v4: fix up irq init ordering
v5: remove outdated comment

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon/cik: add support for doing async VM pt updates (v5)

Async page table updates using the sDMA engine. sDMA has a
special packet for updating entries for contiguous pages
that reduces overhead.

v2: add support for and use the CP for now.
v3: update for 2 level PTs
v4: rebase, fix DMA packet
v5: switch to using an IB

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: implement async vm_flush for the sDMA (v6)

Update the page table base address and flush the
VM TLB using the sDMA.

V2: update for 2 level PTs
V3: update vm flush
V4: update SH_MEM* regs
V5: switch back to old style VM TLB invalidate
V6: fix packet formatting

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: add support for sDMA dma engines (v8)

CIK has new asynchronous DMA engines called sDMA
(system DMA). Each engine supports 1 ring buffer
for kernel and gfx and 2 userspace queues for compute.

TODO: fill in the compute setup.

v2: update to the latest reset code
v3: remove ib_parse
v4: fix copy_dma()
v5: drop WIP compute sDMA queues
v6: rebase
v7: endian fixes for IB
v8: cleanup for release

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: log and handle VM page fault interrupts

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add support for interrupts on CIK (v5)

Todo:
- handle interrupts for compute queues

v2: add documentation
v3: update to latest reset code
v4: update to latest illegal CP handling
v5: fix missing break in interrupt handler switch statement

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Add support for RLC init on CIK (v4)

RLC handles the interrupt controller and other tasks
on the GPU.

v2: add documentation
v3: update programming sequence
v4: additional setup

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: implement async vm_flush for the CP (v7)

Update the page table base address and flush the
VM TLB using the CP.

v2: update for 2 level PTs
v3: use new packet for invalidate
v4: update SH_MEM* regs when flushing the VM
v5: add pfp sync, go back to old style vm TLB invalidate
v6: fix hdp flush packet count
v7: use old style HDP flush

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add ring and IB tests for CIK (v3)

v2: add documenation
v3: update the latest ib changes

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add IB and fence dispatch functions for CIK gfx (v7)

For gfx ring only. Compute is still todo.

v2: add documentation
v3: update to latest reset changes, integrate emit update patch.
v4: fix count on wait_reg_mem for HDP flush
v5: use old hdp flush method for fence
v6: set valid bit for IB
v7: cleanup for release

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Add CP init for CIK (v7)

Sets up the GFX ring and loads ucode for GFX and Compute.

Todo:
- handle compute queue setup.

v2: add documentation
v3: integrate with latest reset changes
v4: additional init fixes
v5: scratch reg write back no longer supported on CIK
v6: properly set CP_RB0_BASE_HI
v7: rebase

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add support mc ucode loading on CIK (v2)

Load the GDDR5 ucode and train the links.

v2: update ucode

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add initial ucode loading for CIK (v5)

Currently the driver required 6 sets of ucode:
1. pfp - pre-fetch parser, part of the GFX CP
2. me - micro engine, part of the GFX CP
3. ce - constant engine, part of the GFX CP
4. rlc - interrupt, etc. controller
5. mc - memory controller (discrete cards only)
6. mec - compute engines, part of Compute CP

V2: add documentation
V3: update MC ucode
V4: rebase
V5: update mc ucode

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: stop page faults from hanging the system (v2)

Redirect invalid memory accesses to the default page
instead of locking up the memory controller.

v2: rebase on top of 2 level PTs

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add support for MC/VM setup on CIK (v6)

The vm callbacks are the same as the SI ones right now
(same regs and bits). We could share the SI variants, and
I may yet do that, but I figured I would add CIK specific
ones for now in case we need to change anything.

V2: add documentation, minor fixes.
V3: integrate vram offset fixes for APUs
V4: enable 2 level VM PTs
V5: index SH_MEM_* regs properly
V6: add ib_parse()

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Add support for CIK GPU reset (v2)

v2: split soft reset into compute and gfx. Still need
to make reset more fine grained, but this should be a
start.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add gpu init support for CIK (v9)

v2: tiling fixes
v3: more tiling fixes
v4: more tiling fixes
v5: additional register init
v6: rebase
v7: fix gb_addr_config for KV/KB
v8: drop wip KV bits for now, add missing config reg
v9: fix cu count on Bonaire

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>