drm/radeon: Use RMW accessors for changing LNKCTL2

Convert open coded RMW accesses for LNKCTL2 to use
pcie_capability_clear_and_set_word() which makes its easier to
understand what the code tries to do.

LNKCTL2 is not really owned by any driver because it is a collection of
control bits that PCI core might need to touch. RMW accessors already
have support for proper locking for a selected set of registers
(LNKCTL2 is not yet among them but likely will be in the future) to
avoid losing concurrent updates.

Acked-by: Alex Deucher <alexander.deucher@amd.com>
Suggested-by: Lukas Wunner <lukas@wunner.de>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Clean up errors in si.c

Fix the following errors reported by checkpatch:

ERROR: that open brace { should be on the previous line
ERROR: trailing statements should be on next lineo

Signed-off-by: XueBing Chen <chenxb_99091@126.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: check return value of radeon_ring_lock()

In the unlikely event of radeon_ring_lock() failing, its errno return
value should be processed. This patch checks said return value and
prints a debug message in case of an error.

Found by Linux Verification Center (linuxtesting.org) with static
analysis tool SVACE.

Fixes: 48c0c902e2e6 ("drm/radeon/kms: add support for CP setup on SI")
Signed-off-by: Nikita Zhandarovich <n.zhandarovich@fintech.ru>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Use RMW accessors for changing LNKCTL

Don't assume that only the driver would be accessing LNKCTL. ASPM policy
changes can trigger write to LNKCTL outside of driver's control. And in
the case of upstream bridge, the driver does not even own the device it's
changing the registers for.

Use RMW capability accessors which do proper locking to avoid losing
concurrent updates to the register value.

Suggested-by: Lukas Wunner <lukas@wunner.de>
Fixes: 8a7cd27679d0 ("drm/radeon/cik: add support for pcie gen1/2/3 switching")
Fixes: b9d305dfb66c ("drm/radeon: implement pcie gen2/3 support for SI")
Link: https://lore.kernel.org/r/20230717120503.15276-7-ilpo.jarvinen@linux.intel.com
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix UVD suspend error

I met a bug recently and the kernel log:

[ 330.171875] radeon 0000:03:00.0: couldn't schedule ib
[ 330.175781] [drm:radeon_uvd_suspend [radeon]] *ERROR* Error destroying UVD (-22)!

In radeon drivers, using UVD suspend is as follows:

if (rdev->has_uvd) {
uvd_v1_0_fini(rdev);
radeon_uvd_suspend(rdev);
}

In radeon_ib_schedule function, we check the 'ring->ready' state,
but in uvd_v1_0_fini funciton, we've cleared the ready state.
So, just modify the suspend code flow to fix error.

Reviewed-by: Leo Liu <leo.liu@amd.com>
Signed-off-by: Qiang Ma <maqianga@uniontech.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: delete useless function return values & remove meaningless if(r) check code

Function radeon_fence_driver_init always returns success,
the function type maybe coule be changed to void.
This patch first delete the check of the return
value of the function call radeon_fence_driver_init, then,
optimise the function declaration and function to void type.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Bernard Zhao <bernard@vivo.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: Fix inconsistent indenting

Kernel test robot throws below warning ->

smatch warnings:
drivers/gpu/drm/radeon/si.c:4514 si_vm_packet3_cp_dma_check() warn:
inconsistent indenting

Fixed the inconsistent indenting.

Reported-by: Abaci Robot <abaci@linux.alibaba.com>
Signed-off-by: Yang Li <yang.lee@linux.alibaba.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si_dpm: Move 'si_mc_load_microcode()'s prototype to shared header

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/si.c:1570:5: warning: no previous prototype for ‘si_mc_load_microcode’ [-Wmissing-prototypes]

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: Move 'r600_ih_ring_{alloc, fini}()'s prototypes to shared header

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/r600.c:3480:5: warning: no previous prototype for ‘r600_ih_ring_alloc’ [-Wmissing-prototypes]
3480 | int r600_ih_ring_alloc(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/r600.c:3516:6: warning: no previous prototype for ‘r600_ih_ring_fini’ [-Wmissing-prototypes]
3516 | void r600_ih_ring_fini(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: Sumit Semwal <sumit.semwal@linaro.org>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Cc: linux-media@vger.kernel.org
Cc: linaro-mm-sig@lists.linaro.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/evergreen: Move 'evergreen_*' and 'sumo_*' prototypes out to shared location

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/evergreen.c:1281:6: warning: no previous prototype for ‘evergreen_fix_pci_max_read_req_size’ [-Wmissing-prototypes]
1281 | void evergreen_fix_pci_max_read_req_size(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:2664:6: warning: no previous prototype for ‘evergreen_mc_stop’ [-Wmissing-prototypes]
2664 | void evergreen_mc_stop(struct radeon_device *rdev, struct evergreen_mc_save *save)
| ^~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:2762:6: warning: no previous prototype for ‘evergreen_mc_resume’ [-Wmissing-prototypes]
2762 | void evergreen_mc_resume(struct radeon_device *rdev, struct evergreen_mc_save *save)
| ^~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:2850:6: warning: no previous prototype for ‘evergreen_mc_program’ [-Wmissing-prototypes]
2850 | void evergreen_mc_program(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:3710:5: warning: no previous prototype for ‘evergreen_mc_init’ [-Wmissing-prototypes]
3710 | int evergreen_mc_init(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:3769:6: warning: no previous prototype for ‘evergreen_print_gpu_status_regs’ [-Wmissing-prototypes]
3769 | void evergreen_print_gpu_status_regs(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:3797:6: warning: no previous prototype for ‘evergreen_is_display_hung’ [-Wmissing-prototypes]
3797 | bool evergreen_is_display_hung(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:4009:6: warning: no previous prototype for ‘evergreen_gpu_pci_config_reset’ [-Wmissing-prototypes]
4009 | void evergreen_gpu_pci_config_reset(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:4110:6: warning: no previous prototype for ‘sumo_rlc_fini’ [-Wmissing-prototypes]
4110 | void sumo_rlc_fini(struct radeon_device *rdev)
| ^~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:4153:5: warning: no previous prototype for ‘sumo_rlc_init’ [-Wmissing-prototypes]
4153 | int sumo_rlc_init(struct radeon_device *rdev)
| ^~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:4381:5: warning: no previous prototype for ‘evergreen_rlc_resume’ [-Wmissing-prototypes]
4381 | int evergreen_rlc_resume(struct radeon_device *rdev)
| ^~~~~~~~~~~~~
drivers/gpu/drm/radeon/evergreen.c:4670:6: warning: no previous prototype for ‘evergreen_irq_suspend’ [-Wmissing-prototypes]
4670 | void evergreen_irq_suspend(struct radeon_device *rdev)
| ^~~~~~~~~~~~~~~~~~~~~

NB: Lots more of these - snipped for brevity

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: Remove set but unused variable 'mc_shared_chmap'

Fixes the following W=1 kernel build warning(s):

drivers/gpu/drm/radeon/si.c: In function ‘si_gpu_init’:
drivers/gpu/drm/radeon/si.c:3090:6: warning: variable ‘mc_shared_chmap’ set but not used [-Wunused-but-set-variable]

Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Cc: amd-gfx@lists.freedesktop.org
Cc: dri-devel@lists.freedesktop.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use true,false for bool variable in si.c

Fixes coccicheck warning:

drivers/gpu/drm/radeon/si.c:6475:2-15: WARNING: Assignment of 0/1 to bool variable
drivers/gpu/drm/radeon/si.c:6542:2-15: WARNING: Assignment of 0/1 to bool variable

Reported-by: Hulk Robot <hulkci@huawei.com>
Signed-off-by: zhengbin <zhengbin13@huawei.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Don't include <drm/drm_pci.h>

Including <drm/drm_pci.h> is unnecessary in most cases. Replace
these instances.

Signed-off-by: Thomas Zimmermann <tzimmermann@suse.de>
Reviewed-by: Emil Velikov <emil.velikov@collabora.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20191203100406.9674-9-tzimmermann@suse.de

drm/radeon: fix bad DMA from INTERRUPT_CNTL2

The INTERRUPT_CNTL2 register expects a valid DMA address, but is
currently set with a GPU MC address. This can cause problems on
systems that detect the resulting DMA read from an invalid address
(found on a Power8 guest).

Instead, use the DMA address of the dummy page because it will always
be safe.

Fixes: d8f60cfc9345 ("drm/radeon/kms: Add support for interrupts on r6xx/r7xx chips (v3)")
Fixes: 25a857fbe973 ("drm/radeon/kms: add support for interrupts on SI")
Fixes: a59781bbe528 ("drm/radeon: add support for interrupts on CIK (v5)")
Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Prefer pcie_capability_read_word()

Commit 8c0d3a02c130 ("PCI: Add accessors for PCI Express Capability")
added accessors for the PCI Express Capability so that drivers didn't
need to be aware of differences between v1 and v2 of the PCI
Express Capability.

Replace pci_read_config_word() and pci_write_config_word() calls with
pcie_capability_read_word() and pcie_capability_write_word().

Link: https://lore.kernel.org/r/20191118003513.10852-1-fred@fredlawl.com
Signed-off-by: Frederick Lawler <fred@fredlawl.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Replace numbers with PCI_EXP_LNKCTL2 definitions

Replace hard-coded magic numbers with the descriptive PCI_EXP_LNKCTL2
definitions. No functional change intended.

Link: https://lore.kernel.org/r/20191112173503.176611-4-helgaas@kernel.org
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Correct Transmit Margin masks

Previously we masked PCIe Link Control 2 register values with "7 << 9",
which was apparently intended to be the Transmit Margin field, but instead
was the high order bit of Transmit Margin, the Enter Modified Compliance
bit, and the Compliance SOS bit.

Correct the mask to "7 << 7", which is the Transmit Margin field.

Link: https://lore.kernel.org/r/20191112173503.176611-3-helgaas@kernel.org
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix bad DMA from INTERRUPT_CNTL2

The INTERRUPT_CNTL2 register expects a valid DMA address, but is
currently set with a GPU MC address. This can cause problems on
systems that detect the resulting DMA read from an invalid address
(found on a Power8 guest).

Instead, use the DMA address of the dummy page because it will always
be safe.

Fixes: d8f60cfc9345 ("drm/radeon/kms: Add support for interrupts on r6xx/r7xx chips (v3)")
Fixes: 25a857fbe973 ("drm/radeon/kms: add support for interrupts on SI")
Fixes: a59781bbe528 ("drm/radeon: add support for interrupts on CIK (v5)")
Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: drop use of drmP.h (2/2)

Drop use of drmP.h in remaining .c files.
To ease review a little the drmP.h removal was divided in two commits.

Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Cc: "David (ChunMing) Zhou" <David1.Zhou@amd.com>
Cc: David Airlie <airlied@linux.ie>
Cc: Daniel Vetter <daniel@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20190608080241.4958-8-sam@ravnborg.org

gpu: drm: radeon: si: Replace mdelay() with msleep() in si_pcie_gen3_enable()

si_pcie_gen3_enable() is never called in atomic context.
It calls mdelay() to busily wait, which is not necessary.
mdelay() can be replaced with msleep().

This is found by a static analysis tool named DCNS written by myself

Signed-off-by: Jia-Ju Bai <baijiaju1990@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use pcie functions for link width

This is the last user of drm_pcie_get_speed_cap_mask. Use the pci
version so we can drop drm_pcie_get_speed_cap_mask.

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Fix overflow of watermark calcs at > 4k resolutions.

Commit e6b9a6c84b93
("drm/radeon: Make display watermark calculations more accurate")
made watermark calculations more accurate, but not for > 4k
resolutions on 32-Bit architectures, as it introduced an integer
overflow for those setups and resolutions.

Fix this by proper u64 casting and division.

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Reported-by: Ben Hutchings <ben.hutchings@codethink.co.uk>
Fixes: e6b9a6c84b93 ("drm/radeon: Make display watermark calculations more accurate")
Cc: Ben Hutchings <ben.hutchings@codethink.co.uk>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Cleanup pageflipping IRQ handling for evergreen, si

Same as the previous patch, but for pageflipping now. This also lets us
clear up the copy paste for vblank/vline IRQs.

Changes since v1:
- Preserve the order all registers are written back

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Lyude <lyude@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Cleanup display interrupt handling for evergreen, si

The current code here is really, really bad. A huge amount of it looks
to be copy pasted, it has some weird hatred of arrays and code sharing,
switch cases everywhere for things that really don't need them, and it
makes the file seem immensely more complex then it actually is. This is
a pain for maintanence, and is vulnerable to more weird irq handling
bugs.

So, let's start cleaning this up a bit. Modify all of the IRQ handlers
for evergreen/si so that they just use for loops. As well, we add a
helper function radeon_irq_kms_set_irq_n_enabled(), whose purpose is
just to update the state of registers that enable/disable interrupts
while printing any changes to the set of enabled interrupts to the
kernel log.

Note in this commit, since vblank/vline irq acking is intertwined with
page flip irq acking, we can't cut out all of the copy paste in
evergreen/si_irq_ack() just yet.

Changes since v1:
- Preserve order we write back all registers

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Lyude <lyude@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Unbreak HPD handling for r600+

We end up reading the interrupt register for HPD5, and then writing it
to HPD6 which on systems without anything using HPD5 results in
permanently disabling hotplug on one of the display outputs after the
first time we acknowledge a hotplug interrupt from the GPU.

This code is really bad. But for now, let's just fix this. I will
hopefully have a large patch series to refactor all of this soon.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Lyude <lyude@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Unbreak HPD handling for r600+

We end up reading the interrupt register for HPD5, and then writing it
to HPD6 which on systems without anything using HPD5 results in
permanently disabling hotplug on one of the display outputs after the
first time we acknowledge a hotplug interrupt from the GPU.

This code is really bad. But for now, let's just fix this. I will
hopefully have a large patch series to refactor all of this soon.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Lyude <lyude@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Make display watermark calculations more accurate

Avoid big roundoff errors in scanline/hactive durations for
high pixel clocks, especially for >= 500 Mhz, and thereby
program more accurate display fifo watermarks.

This is a port of the corresponding amdgpu patch.

Implemented for DCE 4,6,8.
Tested on Evergreen/DCE-4 with Radeon HD-5770.

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Avoid overflows/divide-by-zero in latency_watermark calculations.

At dot clocks > approx. 250 Mhz, some of these calcs will overflow and
cause miscalculation of latency watermarks, and for some overflows also
divide-by-zero driver crash. Make calcs more overflow resistant.

This is a direct port of the corresponding patch from amdgpu-kms,
copy-paste for cik from dce-8 and si from dce-6, with a slightly
simpler variant for evergreen dce-4/5.

Only tested on DCE-4 evergreen with a Radeon HD-5770.

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

gpu: drm: amd/radeon: Convert printk(KERN_<LEVEL> to pr_<level>

Use a more common logging style.

Miscellanea:

o Coalesce formats and realign arguments
o Neaten a few macros now using pr_<level>

Signed-off-by: Joe Perches <joe@perches.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add support for new hainan variants

New hainan parts require updated smc firmware.

Cc: Sonny Jiang <sonny.jiang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: load special ucode for certain MC configs

Special MC ucode is required for these memory configurations.

Acked-by: Edward O'Callaghan <funfunctor@folklore1984.net>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: update smc firmware selection for SI

Use the appropriate smc firmware for each chip revision.
Using the wrong one can cause stability issues.

Acked-by: Edward O'Callaghan <funfunctor@folklore1984.net>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon/si: load the proper firmware on 0x87 oland boards

New variant.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: mark symbols static where possible

We get 4 warnings when building kernel with W=1:
drivers/gpu/drm/radeon/si.c:7850:5: warning: no previous prototype for 'si_vce_send_vcepll_ctlreq' [-Wmissing-prototypes]
drivers/gpu/drm/radeon/radeon_dp_mst.c:226:21: warning: no previous prototype for 'radeon_mst_best_encoder' [-Wmissing-prototypes]
drivers/gpu/drm/radeon/radeon_dp_mst.c:344:26: warning: no previous prototype for 'radeon_mst_find_connector' [-Wmissing-prototypes]
drivers/gpu/drm/radeon/radeon_dp_mst.c:600:6: warning: no previous prototype for 'radeon_dp_mst_encoder_destroy' [-Wmissing-prototypes]

In fact, these functions are only used in the file in which they are
declared and don't need a declaration, but can be made static.
So this patch marks these functions with 'static'.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Baoyou Xie <baoyou.xie@linaro.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: allow TA_CS_BC_BASE_ADDR on SI

Required for border colors in compute shaders.

Reviewed-by: Edward O'Callaghan <funfunctor@folklore1984.net>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: switch UVD code to use UVD_NO_OP for padding

Replace packet2's with packet0 writes to UVD_NO_OP. The
value written to UVD_NO_OP does not matter.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove dead code, si_mc_load_microcode (v2)

In an if block for (running == 0) running cannot be non-zero.

v2: agd: remove unused variable

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: load different smc firmware on some SI variants

The power tables on some variants require different firmware.
This may fix stability issues on some newer SI parts.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Allow setting shader registers using DMA/COPY packet3 on SI.

Mesa uses a COPY_DATA packet to copy the grid size for indirect dispatches
into COMPUTE_USER_DATA_*.

Setting those registers with a SET_SH_REG packet is allowed, not allowing
them with other packets seems like an oversight.

v2: Clarify commit message.

Signed-off-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: allow to force hard GPU reset.

In some cases, like when freezing for hibernation, we need to be
able to force hard reset even if no engine are stuck. This patch
add a bool option to current asic reset callback to allow to force
hard reset on asic that supports it.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: consolidate si vce initialization and startup code.

This match the exact same control flow as existing code. It just
use goto instead of multiple levels of if/else. It also clarify
early initialization failures by clearing rdev->has_vce doing so
does not change end result from hardware point of view, it only
avoids printing more error messages down the line and thus only
the original error is reported.

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: consolidate si uvd initialization and startup code.

This match the exact same control flow as existing code. It just
use goto instead of multiple levels of if/else. It also clarify
early initialization failures by clearing rdev->has_uvd doing so
does not change end result from hardware point of view, it only
avoids printing more error messages down the line and thus only
the original error is reported.

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix indentation.

I hate doing this but it hurts my eyes to go over code that does not
comply with indentation rules. Only thing that is not only space change
is in atom.c all other files are space indentation issues.

Acked-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: refactor SI tiling table initialization

Simplify the control flow of si_tiling_mode_table_init() similar to how
it was done in gfx_v7_0.c and gfx_v8_0.c.

Signed-off-by: Josh Poimboeuf <jpoimboe@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Fixup hw vblank counter/ts for new drm_update_vblank_count() (v2)

commit 4dfd6486 "drm: Use vblank timestamps to guesstimate how many
vblanks were missed" introduced in Linux 4.4-rc1 makes the drm core
more fragile to drivers which don't update hw vblank counters and
vblank timestamps in sync with firing of the vblank irq and
essentially at leading edge of vblank.

This exposed a problem with radeon-kms/amdgpu-kms which do not
satisfy above requirements:

The vblank irq fires a few scanlines before start of vblank, but
programmed pageflips complete at start of vblank and
vblank timestamps update at start of vblank, whereas the
hw vblank counter increments only later, at start of vsync.

This leads to problems like off by one errors for vblank counter
updates, vblank counters apparently going backwards or vblank
timestamps apparently having time going backwards. The net result
is stuttering of graphics in games, or little hangs, as well as
total failure of timing sensitive applications.

See bug #93147 for an example of the regression on Linux 4.4-rc:

https://bugs.freedesktop.org/show_bug.cgi?id=93147

This patch tries to align all above events better from the
viewpoint of the drm core / of external callers to fix the problem:

1. The apparent start of vblank is shifted a few scanlines earlier,
so the vblank irq now always happens after start of this extended
vblank interval and thereby drm_update_vblank_count() always samples
the updated vblank count and timestamp of the new vblank interval.

To achieve this, the reporting of scanout positions by
radeon_get_crtc_scanoutpos() now operates as if the vblank starts
radeon_crtc->lb_vblank_lead_lines before the real start of the hw
vblank interval. This means that the vblank timestamps which are based
on these scanout positions will now update at this earlier start of
vblank.

2. The driver->get_vblank_counter() function will bump the returned
vblank count as read from the hw by +1 if the query happens after
the shifted earlier start of the vblank, but before the real hw increment
at start of vsync, so the counter appears to increment at start of vblank
in sync with the timestamp update.

3. Calls from vblank irq-context and regular non-irq calls are now
treated identical, always simulating the shifted vblank start, to
avoid inconsistent results for queries happening from vblank irq vs.
happening from drm_vblank_enable() or vblank_disable_fn().

4. The radeon_flip_work_func will delay mmio programming a pageflip until
the start of the real vblank iff it happens to execute inside the shifted
earlier start of the vblank, so pageflips now also appear to execute at
start of the shifted vblank, in sync with vblank counter and timestamp
updates. This to avoid some races between updates of vblank count and
timestamps that are used for swap scheduling and pageflip execution which
could cause pageflips to execute before the scheduled target vblank.

The lb_vblank_lead_lines "fudge" value is calculated as the size of
the display controllers line buffer in scanlines for the given video
mode: Vblank irq's are triggered by the line buffer logic when the line
buffer refill for a video frame ends, ie. when the line buffer source read
position enters the hw vblank. This means that a vblank irq could fire at
most as many scanlines before the current reported scanout position of the
crtc timing generator as the number of scanlines the line buffer can
maximally hold for a given video mode.

This patch has been successfully tested on a RV730 card with DCE-3 display
engine and on a evergreen card with DCE-4 display engine, in single-display
and dual-display configuration, with different video modes.

A similar patch is needed for amdgpu-kms to fix the same problem.

Limitations:

- Line buffer sizes in pixels are hard-coded on < DCE-4 to a value
i just guessed to be high enough to work ok, lacking info on the true
sizes atm.

Fixes: fdo#93147

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Michel Dänzer <michel.daenzer@amd.com>
Cc: Harry Wentland <Harry.Wentland@amd.com>
Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>

(v1) Tested-by: Dave Witbrodt <dawitbro@sbcglobal.net>

(v2) Refine radeon_flip_work_func() for better efficiency:

In radeon_flip_work_func, replace the busy waiting udelay(5)
with event lock held by a more performance and energy efficient
usleep_range() until at least predicted true start of hw vblank,
with some slack for scheduler happiness. Release the event lock
during waits to not delay other outputs in doing their stuff, as
the waiting can last up to 200 usecs in some cases.

Retested on DCE-3 and DCE-4 to verify it still works nicely.

(v2) Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Fixup hw vblank counter/ts for new drm_update_vblank_count() (v2)

commit 4dfd6486 "drm: Use vblank timestamps to guesstimate how many
vblanks were missed" introduced in Linux 4.4-rc1 makes the drm core
more fragile to drivers which don't update hw vblank counters and
vblank timestamps in sync with firing of the vblank irq and
essentially at leading edge of vblank.

This exposed a problem with radeon-kms/amdgpu-kms which do not
satisfy above requirements:

The vblank irq fires a few scanlines before start of vblank, but
programmed pageflips complete at start of vblank and
vblank timestamps update at start of vblank, whereas the
hw vblank counter increments only later, at start of vsync.

This leads to problems like off by one errors for vblank counter
updates, vblank counters apparently going backwards or vblank
timestamps apparently having time going backwards. The net result
is stuttering of graphics in games, or little hangs, as well as
total failure of timing sensitive applications.

See bug #93147 for an example of the regression on Linux 4.4-rc:

https://bugs.freedesktop.org/show_bug.cgi?id=93147

This patch tries to align all above events better from the
viewpoint of the drm core / of external callers to fix the problem:

1. The apparent start of vblank is shifted a few scanlines earlier,
so the vblank irq now always happens after start of this extended
vblank interval and thereby drm_update_vblank_count() always samples
the updated vblank count and timestamp of the new vblank interval.

To achieve this, the reporting of scanout positions by
radeon_get_crtc_scanoutpos() now operates as if the vblank starts
radeon_crtc->lb_vblank_lead_lines before the real start of the hw
vblank interval. This means that the vblank timestamps which are based
on these scanout positions will now update at this earlier start of
vblank.

2. The driver->get_vblank_counter() function will bump the returned
vblank count as read from the hw by +1 if the query happens after
the shifted earlier start of the vblank, but before the real hw increment
at start of vsync, so the counter appears to increment at start of vblank
in sync with the timestamp update.

3. Calls from vblank irq-context and regular non-irq calls are now
treated identical, always simulating the shifted vblank start, to
avoid inconsistent results for queries happening from vblank irq vs.
happening from drm_vblank_enable() or vblank_disable_fn().

4. The radeon_flip_work_func will delay mmio programming a pageflip until
the start of the real vblank iff it happens to execute inside the shifted
earlier start of the vblank, so pageflips now also appear to execute at
start of the shifted vblank, in sync with vblank counter and timestamp
updates. This to avoid some races between updates of vblank count and
timestamps that are used for swap scheduling and pageflip execution which
could cause pageflips to execute before the scheduled target vblank.

The lb_vblank_lead_lines "fudge" value is calculated as the size of
the display controllers line buffer in scanlines for the given video
mode: Vblank irq's are triggered by the line buffer logic when the line
buffer refill for a video frame ends, ie. when the line buffer source read
position enters the hw vblank. This means that a vblank irq could fire at
most as many scanlines before the current reported scanout position of the
crtc timing generator as the number of scanlines the line buffer can
maximally hold for a given video mode.

This patch has been successfully tested on a RV730 card with DCE-3 display
engine and on a evergreen card with DCE-4 display engine, in single-display
and dual-display configuration, with different video modes.

A similar patch is needed for amdgpu-kms to fix the same problem.

Limitations:

- Line buffer sizes in pixels are hard-coded on < DCE-4 to a value
i just guessed to be high enough to work ok, lacking info on the true
sizes atm.

Fixes: fdo#93147

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: Michel Dänzer <michel.daenzer@amd.com>
Cc: Harry Wentland <Harry.Wentland@amd.com>
Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>

(v1) Tested-by: Dave Witbrodt <dawitbro@sbcglobal.net>

(v2) Refine radeon_flip_work_func() for better efficiency:

In radeon_flip_work_func, replace the busy waiting udelay(5)
with event lock held by a more performance and energy efficient
usleep_range() until at least predicted true start of hw vblank,
with some slack for scheduler happiness. Release the event lock
during waits to not delay other outputs in doing their stuff, as
the waiting can last up to 200 usecs in some cases.

Retested on DCE-3 and DCE-4 to verify it still works nicely.

(v2) Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Retry DDC probing on DVI on failure if we got an HPD interrupt

HPD signals on DVI ports can be fired off before the pins required for
DDC probing actually make contact, due to the pins for HPD making
contact first. This results in a HPD signal being asserted but DDC
probing failing, resulting in hotplugging occasionally failing.

This is somewhat rare on most cards (depending on what angle you plug
the DVI connector in), but on some cards it happens constantly. The
Radeon R5 on the machine used for testing this patch for instance, runs
into this issue just about every time I try to hotplug a DVI monitor and
as a result hotplugging almost never works.

Rescheduling the hotplug work for a second when we run into an HPD
signal with a failing DDC probe usually gives enough time for the rest
of the connector's pins to make contact, and fixes this issue.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Lyude <cpaul@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Handle irqs only based on irq ring, not irq status regs.

Trying to resolve issues with missed vblanks and impossible
values inside delivered kms pageflip completion events showed
that radeon's irq handling sometimes doesn't handle valid irqs,
but silently skips them. This was observed for vblank interrupts.

Although those irqs have corresponding events queued in the gpu's
irq ring at time of interrupt, and therefore the corresponding
handling code gets triggered by these events, the handling code
sometimes silently skipped processing the irq. The reason for those
skips is that the handling code double-checks for each irq event if
the corresponding irq status bits in the irq status registers
are set. Sometimes those bits are not set at time of check
for valid irqs, maybe due to some hardware race on some setups?

The problem only seems to happen on some machine + card combos
sometimes, e.g., never happened during my testing of different PC
cards of the DCE-2/3/4 generation a year ago, but happens consistently
now on two different Apple Mac cards (RV730, DCE-3, Apple iMac and
Evergreen JUNIPER, DCE-4 in a Apple MacPro). It also doesn't happen
at each interrupt but only occassionally every couple of
hundred or thousand vblank interrupts.

This results in XOrg warning messages like

"[ 7084.472] (WW) RADEON(0): radeon_dri2_flip_event_handler:
Pageflip completion event has impossible msc 420120 < target_msc 420121"

as well as skipped frames and problems for applications that
use kms pageflip events or vblank events, e.g., users of DRI2 and
DRI3/Present, Waylands Weston compositor, etc. See also

https://bugs.freedesktop.org/show_bug.cgi?id=85203

After some talking to Alex and Michel, we decided to fix this
by turning the double-check for asserted irq status bits into a
warning. Whenever a irq event is queued in the IH ring, always
execute the corresponding interrupt handler. Still check the irq
status bits, but only to log a DRM_DEBUG message on a mismatch.

This fixed the problems reliably on both previously failing
cards, RV-730 dual-head tested on both crtcs (pipes D1 and D2)
and a triple-output Juniper HD-5770 card tested on all three
available crtcs (D1/D2/D3). The r600 and evergreen irq handling
is therefore tested, but the cik an si handling is only compile
tested due to lack of hw.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
CC: Michel Dänzer <michel.daenzer@amd.com>
CC: Alex Deucher <alexander.deucher@amd.com>
CC: <stable@vger.kernel.org> # v3.16+
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: partially revert "fix VM_CONTEXT*_PAGE_TABLE_END_ADDR handling"

We have that bug for years and some users report side effects when fixing it on older hardware.

So revert it for VM_CONTEXT0_PAGE_TABLE_END_ADDR, but keep it for VM 1-15.

Signed-off-by: Christian König <christian.koenig@amd.com>
CC: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add VCE 1.0 support v4

Initial support for VCE 1.0 using newest firmware.

v2: rebased
v3: fix for TN
v4: fix FW size calculation

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: implement si_set_vce_clocks v2

For setting clocks with VCE v1.0

v2: (chk) rebased on current tree

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix VM_CONTEXT*_PAGE_TABLE_END_ADDR handling

The mapping range is inclusive between starting and ending addresses.

Signed-off-by: Christian König <christian.koenig@amd.com>
CC: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

radeon/si: add support for short HPD irqs

This adds support to process short HPD irqs on SI gpus.

Signed-off-by: Dave Airlie <airlied@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add get_allowed_info_register for SI

Registers that can be fetched from the info ioctl.

Tested-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: drop setting UPLL to sleep mode

Just keep it working, seems to fix some PLL problems.

Bug: https://bugs.freedesktop.org/show_bug.cgi?id=73378

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: do a posting read in si_set_irq

To make sure the writes go through the pci bridge.

bug:
https://bugzilla.kernel.org/show_bug.cgi?id=90741

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: enable SRBM timeout interrupt on SI

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: dump full IB if we hit a packet error

Dump the whole IB if we run into an invalid packet.
This makes things much easier to debug.

bug:
https://bugs.freedesktop.org/show_bug.cgi?id=89148

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

radeon/audio: consolidate audio_fini() functions

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Slava Grigorev <slava.grigorev@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

radeon/audio: consolidate audio_init() functions

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Slava Grigorev <slava.grigorev@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix VM flush on SI (v3)

We need to wait for the GPUVM flush to complete. There
was some confusion as to how this mechanism was supposed
to work. The operation is not atomic. For GPU initiated
invalidations you need to read back a VM register to
introduce enough latency for the update to complete.

v2: drop gart changes
v3: just read back rather than polling

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: use one VMID for each ring

Use multiple VMIDs for each VM, one for each ring. That allows
us to execute flushes separately on each ring, still not ideal
cause in a lot of cases rings can share IDs.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: rework vm_flush parameters

Use ring structure instead of index and provide vm_id and pd_addr separately.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: make sure mode init is complete in bandwidth_update

The power management code calls into the display code for
certain things. If certain power management sysfs attributes
are called before the driver has finished initializing all of
the hardware we can run into problems with uninitialized
modesetting state. Add a check to make sure modesetting
init has completed to the bandwidth update callbacks to
fix this. Can be triggered by the tlp and laptop start
up scripts depending on the timing.

bugs:
https://bugzilla.kernel.org/show_bug.cgi?id=83611
https://bugs.freedesktop.org/show_bug.cgi?id=85771

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon/si: print full CS when we hit a packet 0

We should never have a packet 0 in the command stream.
Dump the full command stream to help debug.

bug:
https://bugs.freedesktop.org/show_bug.cgi?id=84500

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Update IH_RB_RPTR register after each processed interrupt

This might decrease the chance of IH ring buffer overflows.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Make IH ring overflow debugging output more useful

Use the same format for all ring indices, and fix the calculation of the
post-overflow RPTR.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Clear RB_OVERFLOW bit earlier

Otherwise the bit remains set in rdev->ih.rptr, so the wptr can never
match that and we still have an infinite loop.

This fix allows me to successfully recover from an IH ring buffer
overflow.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

radeon: Test for PCI root bus before assuming bus->self

If we assign a Radeon device to a virtual machine, we can no longer
assume a fixed hardware topology, like the GPU having a parent device.
This patch simply adds a few pci_is_root_bus() tests to avoid passing
a NULL pointer to PCI access functions, allowing the radeon driver to
work in a QEMU 440FX machine with an assigned HD8570 on the emulated
PCI root bus.

Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: save/restore the PD addr on suspend/resume

This fixes a problem with GPU resets and TLB flushes on SI/CIK.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix active_cu mask on SI and CIK after re-init (v3)

Need to initialize the mask to 0 on init, otherwise it
keeps increasing.

bug:
https://bugzilla.kernel.org/show_bug.cgi?id=82581

v2: also fix cu count
v3: split count fix into separate patch

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix active cu count for SI and CIK

This fixes the CU count reported to userspace for
OpenCL.

bug:
https://bugzilla.kernel.org/show_bug.cgi?id=82581

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: Only flush HDP cache for indirect buffers from userspace

It isn't necessary for command streams generated by the kernel (at least
not while we aren't storing ring or indirect buffers in VRAM).

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use pfp for all vm_flush related updates

May fix hangs in some cases.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Use pflip irqs for pageflip completion if possible. (v2)

Skip the "manual" pageflip completion checks via polling and
guessing in the vblank handler radeon_crtc_handle_vblank() on
asics which are known to reliably support hw pageflip completion
irqs. Those pflip irqs are a more reliable and race-free method
of handling pageflip completion detection, whereas the "classic"
polling method has some small races in combination with dpm on,
and with the reworked pageflip implementation since Linux 3.16.

On old asics without pflip irqs, the classic method is used.

On asics with known good pflip irqs, only pflip irqs are used
by default, but a new module parameter "use_pflipirqs" allows to
override this in case we encounter asics in the wild with
unreliable or faulty pflip irqs. A module parameter of 0 allows
to use the classic method only in such a case. A parameter of 1
allows to use both classic method and pflip irqs as additional
band-aid to avoid some small races which could happen with the
classic method alone. The setting 1 gives Linux 3.16 behaviour.

Hw pflip irqs are available since R600.

Tested on DCE-4, AMD Cedar - FirePro 2270.

v2: agd5f: only enable pflip interrupts on DCE4+ as they are not
reliable on older asics.

Signed-off-by: Mario Kleiner <mario.kleiner.de@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: set VM base addr using the PFP v2

Seems to make VM flushes more stable on SI and CIK.

v2: only use the PFP on the GFX ring on CIK

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Remove radeon_gart_restore()

Doesn't seem necessary, the GART table memory should be persistent.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: Add support for new ucode format (v3)

This adds SI support for the new ucode format.

v2: add size validation, integrate debug info
v3: update to latest version

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix irq ring buffer overflow handling

We must mask out the overflow bit as well, otherwise
the wptr will never match the rptr again and the interrupt
handler will loop forever.

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>

drm/radeon: only print meaningful VM faults

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add query for number of active CUs

Query to find out how many compute units on a GPU.
Useful for OpenCL usermode drivers.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: make vm_block_size a module parameter

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use lower_32_bits where appropriate

Replace occurrences of "v & 0xffffffff" with lower_32_bits(v)
when it's next to an upper_32_bits(v). Also remove unnecessary
"upper_32_bits(v) & 0xffffffff" code snippets.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: separate vblank and pflip crtc handling

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add proper support for RADEON_VM_BLOCK_SIZE v2

This patch makes it possible to decide how many address
bits are spend on the page directory vs the page tables.

v2: remove unintended change

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add large PTE support for NI, SI and CIK v5

This patch implements support for VRAM page table entry compression.
PTE construction is enhanced to identify physically contiguous page
ranges and mark them in the PTE fragment field. L1/L2 TLB support is
enabled for 64KB (SI/CIK) and 256KB (NI) PTE fragments, significantly
improving TLB utilization for VRAM allocations.

Linear store bandwidth is improved from 60GB/s to 125GB/s on Pitcairn.
Unigine Heaven 3.0 sees an average improvement from 24.7 to 27.7 FPS
on default settings at 1920x1200 resolution with vsync disabled.

See main comment in radeon_vm.c for a technical description.

v2 (chk): rebased and simplified.
v3 (chk): add missing hw setup
v4 (chk): rebased on current drm-fixes-3.15
v5 (chk): fix comments and commit text

Signed-off-by: Jay Cornwall <jay@jcornwall.me>
Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use pflip irq on R600+ v2

Testing the update pending bit directly after issuing an
update is nonsense cause depending on the pixel clock the
CRTC needs a bit of time to execute the flip even when we
are in the VBLANK period.

This is just a non invasive patch to solve the problem at
hand, a more complete and cleaner solution should follow
in the next merge window.

Fixes: https://bugs.freedesktop.org/show_bug.cgi?id=76564

v2: fix source IDs for CRTC2-6

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org

drm/radeon/si: make sure mc ucode is loaded before checking the size

Avoid a possible segfault.

Noticed-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: add support for newer mc ucode on SI (v2)

May fix stability issues with some newer cards.

v2: print out mc firmware version used and size

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: resume old pm late

Moving the pm resume up in the init order to fix
dpm seems to have regressed somes cases with the old
pm code. Move it back to late resume.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: drop radeon_ring_force_activity

The reason for the false positives was fixed quite some time ago and since
most engines can still execute NOPs while being locked up it leads to false
negatives.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: drop drivers copy of the rptr

In all cases where it really matters we are using the read functions anyway.

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix UVD IRQ support on SI

Otherwise decoding isn't really useable.

bug:
https://bugs.freedesktop.org/show_bug.cgi?id=71448

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix DAC interrupt handling on DCE5+

DCE5 and newer hardware only has 1 DAC. Use the correct
offset. This may fix display problems on certain board
configurations.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: clean up active vram sizing

If we are not able to properly initialize one of the gpu
engines for buffer paging, we limit vram to the size of
the cpu visible aperture. We generally either use the gfx
or dma engine to do this. Clean up the size limiting code
to only adjust the size based on what ring is selected
for buffer paging rather than making assumptions about which
engine is selected for paging.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon: write gfx pg bases even when gfx pg is disabled

For consistency. These buffers aren't used when pg is
disabled.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: implement pci config reset for SI (v2)

pci config reset is a low level reset that resets
the entire chip from the bus interface. It can
be more reliable if soft reset fails.

v2: hide behind module parameter

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove generic rptr/wptr functions (v2)

Fill in asic family specific versions rather than
using the generic version. This lets us handle asic
specific differences more easily. In this case, we
disable sw swapping of the rtpr writeback value on
r6xx+ since the hw does it for us. Fixes bogus
rptr readback on BE systems.

v2: remove missed cpu_to_le32(), add comments

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/pm: move pm handling into the asic specific code

We need more control over the ordering of dpm init with
respect to the rest of the asic. Specifically, the SMC
has to be initialized before the rlc and cg/pg. The pm
code currently initializes late in the driver, but we need
it to happen much earlier so move pm handling into the asic
specific callbacks.

This makes dpm more reliable and makes clockgating work
properly on CIK parts and should help on SI parts as well.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: re-order firmware loading in preparation for dpm rework

We need to reorder the driver init sequence to better accomodate
dpm which needs to be loaded earlier in the init sequence. Move
fw init up so that it's available for dpm init.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: drop cg_update from dpm code

I'm not entirely sure this is required and it won't work
with the dpm restructing anyway.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: expose render backend mask to the userspace

This will allow userspace to correctly program the PA_SC_RASTER_CONFIG
register, so it can be considered a fix.

Signed-off-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix render backend setup for SI and CIK

Only the render backends of the first shader engine were enabled. The others
were erroneously disabled. Enabling the other render backends improves
performance a lot.

Unigine Sanctuary on Bonaire:
Before: 15 fps
After: 90 fps

Judging from the fan noise, the GPU was also underclocked when the other
render backends were disabled, resulting in horrible performance. The fan is
a lot noisy under load now.

Signed-off-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fixup bad vram size on SI

Some boards seem to have garbage in the upper
16 bits of the vram size register. Check for
this and clamp the size properly. Fixes
boards reporting bogus amounts of vram.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: drop CP page table updates & cleanup v2

The DMA ring seems to be stable now.

v2: remove pt_ring_index as well

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix endian handling in rlc buffer setup

The buffers needs to be in little endian format.

Noticed-by: Sylvain BERTRAND <sylware@legeek.net>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: make missing smc ucode non-fatal (r7xx-SI)

Prevent driver load problems if the smc is missing.

bug:
https://bugzilla.kernel.org/show_bug.cgi?id=63011

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Tested-by: Mikko Rapeli <mikko.rapeli@iki.fi>
Cc: stable@vger.kernel.org

drm/radeon: improve soft reset on SI

Disable CG/PG and stop the rlc before resetting.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix typo in PG flags

s/CG/PG/ in the GFX powergating flag name.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: properly handle internal cp ints

The internal cp interrupts need to be enabled and
disabled at specific times in order clockgating to
work properly. This patch changes the handling
of the CP_INT_CNTL register to respect the current
state of the internal CP interrupts when making
changes to the other interrupts in CP_INT_CNTL.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix init ordering for r600+

The vram scratch buffer needs to be initialized
before the mc is programmed otherwise we program
0 as the GPU address of the default GPU fault
page. In most cases we put vram at zero anyway and
reserve a page for the legacy vga buffer so in practice
this shouldn't cause any problems, but better to make
it correct.

Was changed in:
6fab3febf6d949b0a12b1e4e73db38e4a177a79e

Reported-by: FrankR Huang <FrankR.Huang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: update line buffer allocation for dce6

We need to allocate line buffer to each display when
setting up the watermarks. Failure to do so can lead
to a blank screen. This fixes blank screen problems
on dce6 asics.

Fixes:
https://bugs.freedesktop.org/show_bug.cgi?id=64850

Based on an initial fix from:
Jay Cornwall <jay.cornwall@amd.com>

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon/si: Add support for CP DMA to CS checker for compute v2

Also add a new RADEON_INFO query to check that CP DMA packets are
supported on the compute ring.

CP DMA has been supported since the 3.8 kernel, but due to an oversight
we forgot to teach the CS checker that the CP DMA packet was legal for
the compute ring on Southern Islands GPUs.

This patch fixes a bug where the radeon driver will incorrectly reject a legal
CP DMA packet from user space. I would like to have the patch
backported to stable so that we don't have to require Mesa users to use a
bleeding edge kernel in order to take advantage of this feature which
is already present in the stable kernels (3.8 and newer).

v2:
- Don't bump kms version, so this patch can be backported to stable
kernels.

Cc: stable@vger.kernel.org
Signed-off-by: Tom Stellard <thomas.stellard@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: enable DMA pg by default

Enable DMA powergating by default. The DMA engines
will be powergated when not in use.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: properly set up the clearstate buffer for pg (v2)

The format of the clearstate buffer used for pg (powergating)
changed between NI and SI. This formats it properly for what
the hardware expects on SI.

v2: fix addresses

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fixes for gfx clockgating on SI

Clockgating requires signalling between the CP and the
RLC to work properly. Resetting the CP block in the
CP resume code messed up the internal coordination
between the blocks. Removing the reset allows gfx
clockgating to work properly. However, when gfx clock
gating is enabled, there is a strange interaction with
dpm which causes the chip to stay in the high performance
level all the time, so leave gfx clockgating disabled
for now.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: restructure cg code (v3)

Resturcture clockgating code so that it can be
enabled/disabled from other components such as
dpm.

v2: make function static
v3: add fine grained cg controls

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use new cg/pg flags for SI

Allows us finer grained control over clock and
powergating on SI.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add audio support for DCE6/8 GPUs (v12)

Similar to DCE4/5, but supports multiple audio pins
which can be assigned per afmt block.

v2: rework the driver to handle more than one audio
pin.
v3: try different dto reg
v4: properly program dto
v5 (ck): change dto programming order
v6: program speaker allocation block
v7: rebase
v8: rebase on Rafał's changes
v9: integrated Rafał's comments, update to latest
drm_edid_to_speaker_allocation API
v10: add missing line break in error message
v11: add back audio enabled messages
v12: fix copy paste typo in r600_audio_enable

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Rafał Miłecki <zajec5@gmail.com>

drm/radeon: separate DMA code

Similar to separating the UVD code, just put the DMA
functions into separate files.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: separate UVD code v3

Our different hardware blocks are actually completely
separated, so it doesn't make much sense any more to
structure the code by pure chipset generations.

Start restructuring the code by separating our the UVD block.

v2: updated commit message
v3: rebased and restructurized start/stop functions for kv dpm.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove special handling for the DMA ring

Now that we have callbacks for [rw]ptr handling we can
remove the special handling for the DMA rings and use
the callbacks instead.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: rework UVD writeback & [rw]ptr handling

The hardware just doesn't support this correctly.
Disable it before we accidentally write anywhere we shouldn't.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: restructure UVD code to handle UVD PG (v2)

When we PG (powergate) UVD, we need to re-initialize it
before we can use it again.

v2: rebase on UVD stop fixes

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: implement clock and power gating for CIK (v3)

Only the APUs support power gating.

v2: disable cgcg for now
v3: workaround hw issue in mgcg

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: convert SI,CIK to use sumo_rlc functions

and remove duplicate si_rlc functions.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/cik: restructure rlc setup

Restructure rlc setup to handle clock and power
gating.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: make missing smc ucode non-fatal

The smc ucode is required for dpm (dynamic power
management), but if it's missing just skip dpm setup
and don't disable acceleration.

Should fix:
https://bugs.freedesktop.org/show_bug.cgi?id=67876

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: always program the MC on startup

For r6xx+ asics. This mirrors the behavior of pre-r6xx
asics. We need to program the MC even if something
else in startup() fails. Failure to do so results in
an unusable GPU.

Based on a fix from: Mark Kettenis <kettenis@openbsd.org>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix halting UVD

Removing the clock/power or resetting the VCPU can cause
hangs if that happens in the middle of a register write.

Stall the memory and register bus before putting the VCPU
into reset. Keep it in reset when unloading the module or
suspending.

Signed-off-by: Christian König <christian.koenig@amd.com>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: disable cgcg and pg for now

Coarse grain clockgating causes problems with reclocking on
some cards and powergating (verde only) causes problems with
ring initialization. The proper fix (restructuring the init
sequences) is too invasive for 3.11 so just disable them for
now.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: properly handle cg on asics without UVD

Don't try and enable clockgating if the asic doesn't have
UVD. Use rdev->has_uvd rather than using local checks.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: s/drm_order/order_base_2/

Last driver and pretty obviously a major user of this little function.

Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@gmail.com>

drm/radeon: add a module parameter to disable aspm

Can cause hangs when enabled in certain motherboards.
Set radeon.aspm=0 to disable aspm.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add fault decode function for SI (v2)

Helpful for debugging GPUVM errors as we can see what
hw block and page generated the fault in the log.

v2: simplify fault decoding

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon: use radeon device for request firmware

Avoid creating temporary platform device that will lead to issue
when several radeon gpu are in same computer. Instead directly use
the radeon device for requesting firmware.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: fix typo in function name

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/kms: add dpm support for SI (v7)

This adds dpm support for SI asics. This includes:
- dynamic engine clock scaling
- dynamic memory clock scaling
- dynamic voltage scaling
- dynamic pcie gen1/gen2/gen3 switching
- power containment
- shader power scaling

Set radeon.dpm=1 to enable.

v2: enable hainan support, rebase
v3: guard acpi stuff
v4: fix 64 bit math
v5: fix 64 bit div harder
v6: fix thermal interrupt check noticed by Jerome
v7: attempt fix state enable

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: switch SI to use radeon_ucode.h

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/dpm: save some display parameters for DPM

Required for SI.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: implement clock and power gating for SI

Only Cape Verde supports power gating.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add clearstate init for verde power gating

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: initialize save/restore buffer for pg on verde

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: update rlc programming sequence on SI

This is required for certain power management features.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add support for ASPM on SI asics (v2)

Enables PCIE ASPM (Active State Power Management) on
SI asics.

v2: fix typo

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: implement pcie gen2/3 support for SI

If both the motherboard and GPU support pcie gen2 or 3,
enable it. PCIE gen2 and 3 offer more bandwidth than
pcie gen1.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/kms: fix up dce6 display watermark calc for dpm

Calculate the low and high watermarks based on the low and high
clocks for the current power state. The dynamic pm hw will select
the appropriate watermark based on the internal dpm state.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add support for MC/VM setup on CIK (v6)

The vm callbacks are the same as the SI ones right now
(same regs and bits). We could share the SI variants, and
I may yet do that, but I figured I would add CIK specific
ones for now in case we need to change anything.

V2: add documentation, minor fixes.
V3: integrate vram offset fixes for APUs
V4: enable 2 level VM PTs
V5: index SH_MEM_* regs properly
V6: add ib_parse()

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

radeon: Fix system hang issue when using KMS with older cards

The current radeon driver initialization routines, when using KMS, are written
so that the IRQ installation routine is called before initializing the WB buffer
and the CP rings. With some ASICs, though, the IRQ routine tries to access the
GFX_INDEX ring causing a call to RREG32 with the value of -1 in
radeon_fence_read. This, in turn causes the system to completely hang with some
cards, requiring a hard reset.

A call stack that can cause such a hang looks like this (using rv515 ASIC for the
example here):
* rv515_init (rv515.c)
* radeon_irq_kms_init (radeon_irq_kms.c)
* drm_irq_install (drm_irq.c)
* radeon_driver_irq_preinstall_kms (radeon_irq_kms.c)
* rs600_irq_process (rs600.c)
* radeon_fence_process - due to SW interrupt (radeon_fence.c)
* radeon_fence_read (radeon_fence.c)
* hang due to RREG32(-1)

The patch moves the IRQ installation to the card startup routine, after the ring
has been initialized, but before the IRQ has been set. This fixes the issue, but
requires a check to see if the IRQ is already installed, as is the case in the
system resume codepath.
I have tested the patch on three machines using the rv515, the rv770 and the
evergreen ASIC. They worked without issues.

This seems to be a known issue and has been reported on several bug tracking
sites by various distributions (see links below). Most of reports recommend
booting the system with KMS disabled and then enabling KMS by reloading the
radeon module. For some reason, this was indeed a usable workaround, however,
UMS is now deprecated and disabled by default.

Bug reports:
https://bugzilla.redhat.com/show_bug.cgi?id=845745
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/561789
https://bbs.archlinux.org/viewtopic.php?id=156964

Signed-off-by: Adis Hamzić <adis@hamzadis.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: fix typo in cu_per_sh on verde

Should be 5 rather than 2.

Noticed by sroland and glisse on IRC.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: add golden register settings for Hainan (v2)

v2: fix typo

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon: sun/hainan chips do not have UVD (v2)

Skip UVD handling on them.

v2: split has_uvd tracking into separate patch

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fill in ucode loading support for Hainan

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon: don't touch DCE or VGA regs on Hainan (v3)

Hainan has no display hardware:
- no DCE (crtc, uniphy, dac, etc.)
- no VGA

v2: fix bios fetch
v3: fix interrupts

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon: fill in GPU init for Hainan (v2)

v2: fix gb_addr_config value

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Fix VRAM size calculation for VRAM >= 4GB

Add ULL prefix to avoid overflow.

Signed-off-by: Niels Ole Salscheider <niels_ole@salscheider-online.de>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: consolidate UVD clock programming

Instead of duplicating the code over and over again, just use a single
function to handle the clock calculations.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: add support for golden register init

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix typo in si_select_se_sh()

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: put UVD PLLs in bypass mode

Just power down the PLL when we get a VCLK or DCLK of zero.
Enabling the bypass mode early should also allow us to
switch UVD clocks on the fly.

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add si tile mode array query v3

Allow userspace to query for the tile mode array so userspace can properly
compute surface pitch and alignment requirement depending on tiling.

v2: Make strict aliasing safer by casting to char when copying
v3: merge fix from Christian

Signed-off-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove unused blit remnants from si.c

We use the DMA ring rather than the GFX ring for
bo moves. This code was never used and commented out.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add UVD tiling addr config v2

v2: set UVD tiling config for rv730

Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

drm/radeon: add set_uvd_clocks callback for SI

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: UVD bringup v8

Just everything needed to decode videos using UVD.

v6: just all the bugfixes and support for R7xx-SI merged in one patch
v7: UVD_CGC_GATE is a write only register, lockup detection fix
v8: split out VRAM fallback changes, remove support for RV770,
add support for HEMLOCK, add buffer sizes checks

Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/dce6: add missing display reg for tiling setup

A new tiling config register for the display blocks was
added on DCE6.

May fix:
https://bugs.freedesktop.org/show_bug.cgi?id=62889
https://bugs.freedesktop.org/show_bug.cgi?id=57919

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: clean up vram/gtt location handling

Add a per-asic MC (memory controller) mask which holds the
mak address mask the asic is capable of. Use this when
calculating the vram and gtt locations rather using asic
specific functions or limiting everything to 32 bits.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix S/R on VM systems (cayman/TN/SI)

We weren't properly tearing down the VM sub-alloctor
on suspend leading to bogus VM PTs on resume.

Fixes:
https://bugs.freedesktop.org/show_bug.cgi?id=60439

Reviewed-by: Christian König <christian.koenig@amd.com>
Tested-by: Dmitry Cherkasov <Dmitrii.Cherkasov@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

drm/radeon: skip MC reset as it's probably not hung

The MC is mostly likely busy (e.g., display requests), not hung
so no need to reset it. Doing an MC reset is tricky and not
particularly reliable. Fixes hangs in certain cases.

Reported-by: Josh Boyer <jwboyer@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: switch get_gpu_clock() to a callback (v2)

Cleans up the code for future asics

v2: rebase, fix some missing radeon_asic updates

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add a asic callback to get the xclk

This is required to get the reference clock used
by the gfx engine for things like timestamps. Fixes
support for GL extensions the use timestamps on
certain boards.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add ucode loading support for Oland

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fill in gpu init for Oland

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use IBs for VM page table updates v2

For very large page table updates, we can exceed the
size of the ring. To avoid this, use an IB to perform
the page table update.

v2(ck): cleanup the IB infrastructure and the use it instead
of filling the struct ourself.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>

drm/radeon: use the reset mask to determine if rings are hung

fetch the reset mask and check if the relevant ring flags
are set to determine whether the ring is hung or not.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: halt engines before disabling MC (si)

It's better to halt the engines before we disable the MC.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: use status regs to determine what to reset (si)

When we attempt the reset the GPU, look at the status registers
to determine what blocks need to be reset.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: rework GPU reset on cayman/TN

Update the code to better match the recommended
programming sequence for soft reset.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add a bios scratch asic hung helper

Used by all asic families from r600+.
Flag for the vbios and later instances of the driver
that the GPU is hung.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: consolidate redundant macros and constants

After refactoring the _cs logic, we ended up with many
macros and constants that #define the same thing.
Clean'em up.

Signed-off-by: Ilija Hadzic <ihadzic@research.bell-labs.com>
Reviewed-by: Marek Olšák <maraeo@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: clear reset flags if engines are idle

Fixes a hard lock in the gpu reset code after the
rework for DMA support (0ecebb9e0d14e9948e0b1529883a776758117d6f
"drm/radeon: switch to a finer grained reset for evergreen")
due to not bailing before the MC shutdown if the relevant engines
are idle.

Discussion:
http://lists.freedesktop.org/archives/dri-devel/2013-January/032985.html

Reported-by: Eldad Zack <eldad@fogrefinery.com>
Tested-by: Eldad Zack <eldad@fogrefinery.com>
Acked-by: Paul Menzel <paulepanter@users.sourceforge.net>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: switch to a finer grained reset for SI (v2)

No change in functionality as we currently set all the reset
flags.

v2: fix typo

Reviewed-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: reset dma engine on gpu reset (v2)

This try to reset the dma engine when performing gpu reset. Hopefully
bringing back the gpu dma engine in sane state.

v2: agd5f: fix dma reset on cayman/TN, add support for SI

Signed-off-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: print dma status reg on lockup (v2)

To help debug dma related lockup.

v2: agd5f: update SI as well

Signed-off-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/si: add VM CS checker support for CP DMA

Need to verify for copies involving registers.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: add dma engine support for vm pt updates on si (v2)

Async DMA has a special packet for contiguous pt updates
which saves overhead.

v2: rebase

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/kms: Add initial support for async DMA on SI

Pretty much the same as cayman. Some changes to the copy
packets.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: stop page faults from hanging the system (v2)

Redirect invalid memory accesses to the default page
instead of locking up the memory controller. Also
enable the invalid memory access interrupts and
start spamming system log with it.

v2 (agd5f): fix up against 2 level PT changes

Signed-off-by: Christian König <deathsimple@vodafone.de>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org

Fix misspellings of "whether" in comments.

"Whether" is misspelled in various comments across the tree; this
fixes them. No code changes.

Signed-off-by: Adam Buchbinder <adam.buchbinder@gmail.com>
Signed-off-by: Jiri Kosina <jkosina@suse.cz>

drm/radeon/si: add some missing regs to the VM reg checker

This register is needed for streamout to work properly.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>

drm/radeon: fix si_set_page v2

Handle requests that won't fit into a single packet.

v2: pe needs to increase as well.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix PFP sync in vm_flush

Otherwise the next IB might start reading commands
with the page table still invalid.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: update comments to clarify VM setup (v2)

The actual set up and assignment of VM page tables
is done on the fly in radeon_gart.c.

v2: update vm size comments

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon: add vm set_page() callback for SI

Use the new WRITE_DATA packet rather than the legacy
ME_WRITE packet.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: rework the vm_flush interface

Pass the vm and ring index rather than an IB. This allows
us to use the vm_flush interface for non-IB cases in the
future.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon: use WRITE_DATA packets for vm flush on SI

This is the preferred packet for writing data to memory
or registers on SI.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

UAPI: (Scripted) Convert #include "..." to #include <path/...> in drivers/gpu/

Convert #include "..." to #include <path/...> in drivers/gpu/.

Signed-off-by: David Howells <dhowells@redhat.com>
Acked-by: Dave Airlie <airlied@redhat.com>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: Dave Jones <davej@redhat.com>

drm/radeon: add 2-level VM pagetables support v9

PDE/PTE update code uses CP ring for memory writes.
All page table entries are preallocated for now in alloc_pt().

It is made as whole because it's hard to divide it to several patches
that compile and doesn't break anything being applied separately.

Tested on cayman card.

v2: rebased on top of "refactor set_page chipset interface v3",
code cleanups

v3: switched offsets calc macros to inline funcs where possible,
remove pd_addr from radeon_vm, switched RADEON_BLOCK_SIZE define,
to 9 (and PTE_COUNT to 1 << BLOCK_SIZE)

v4 (ck): move "incr" documentation to previous patch, cleanup and
document RADEON_VM_* constants, change commit message to
our usual format, simplify patch allot by removing
everything current not necessary, disable SI workaround.

v5: (agd5f): Fix typo in tables_size calculation in
radeon_vm_alloc_pt(). Second line should have been
'+=' rather than '='.

v6: fix npdes calculation. In scenario when pfns to be mapped overlap
two PDE spans:

+-----------+-------------+
| PDE span | PDE span |
+-----------+----+--------+
| |
+---------+
| pfns |
+---------+

the following npdes calculation gives incorrect result:

npdes = (nptes >> RADEON_VM_BLOCK_SIZE) + 1;

For the case above picture it should give npdes = 2, but gives one.

This patch corrects it by rounding last pfn up to 512 border,
first - down to 512 border and then subtracting and dividing by 512.

v7: Make npde calculation clearer, fix ndw calculation.

v8: (agd5f): reserve enough for 2 full VM PTs, add some
additional comments.

v9: fix typo in npde calculation

Signed-off-by: Dmitry Cherkasov <Dmitrii.Cherkasov@amd.com>
Signed-off-by: Christian König <deathsimple@vodafone.de>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: Mark all possible functions / structs as static

Let's allow GCC to optimize better.

This exposed some five unused functions, but this patch doesn't remove them.

Signed-off-by: Lauri Kasanen <cand@gmx.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: rework VMID handling

Move binding onto the ring, simplifying handling a bit.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

drm/radeon: make VM flushs a ring operation

Move flushing the VMs as function into the rings.
First step to make VM operations async.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

drm/radeon: remove vm_unbind

It actually isn't very useful.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

drm/radeon: cleanup VM id handling a bit

Store a reference to the VM into the IB structure, that
makes calculating the IBs address a bit less complicated.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

drm/radeon: remove gui_idle interrupt infrastructure

It was only used for dynpm, but has been replaced with
a better implementation using fences. Remove it.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon/kms: implement timestamp userspace query (v2)

Returns a snapshot of the GPU clock counter. Needed
for certain OpenGL extensions.

v2: agd5f
- address Jerome's comments
- add function documentation

Signed-off-by: Marek Olšák <maraeo@gmail.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix bank tiling parameters on SI

The sixteen bank case wasn't handled here, leading to GPU
crashes because of userspace miscalculation.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Cc: stable@vger.kernel.org
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: update rptr saving logic for memory buffers

Add support for using memory buffers rather than
scratch registers. Some rings may not be able to
write to scratch registers.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon: update ib_execute for SI (v2)

When submitting a CONST_IB, emit a SWITCH_BUFFER
packet before the CONST_IB. This isn't strictly necessary
(the driver will work fine without it), but is good practice
and allows for more flexible DE/CE sychronization options
in the future. Current userspace drivers do not take
advantage of the CE yet.

v2: - clean up code flow a bit
- no need to flush caches for CONST IB

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>

drm/radeon: record what is next valid wptr for each ring v4

Before emitting any indirect buffer, emit the offset of the next
valid ring content if any. This allow code that want to resume
ring to resume ring right after ib that caused GPU lockup.

v2: use scratch registers instead of storing it into memory
v3: skip over the surface sync for ni and si as well
v4: use SET_CONFIG_REG instead of PACKET0

Signed-off-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: move radeon_ib_ring_tests out of chipset code

Making it easier to control when it is executed.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove vm_manager start/suspend

Just restore the page table instead. Addressing three
problem with this change:

1. Calling vm_manager_suspend in the suspend path is
problematic cause it wants to wait for the VM use
to end, which in case of a lockup never happens.

2. In case of a locked up memory controller
unbinding the VM seems to make it even more
unstable, creating an unrecoverable lockup
in the end.

3. If we want to backup/restore the leftover ring
content we must not unbind VMs in between.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove r600_blit_suspend

Just reinitialize the shader content on resume instead.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove ip_pool start/suspend

The IB pool is in gart memory, so it is completely
superfluous to unpin / repin it on suspend / resume.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: remove FIXME comment from chipset suspend

For a normal suspend/resume we allready wait for
the rings to be empty, and for a suspend/reasume
in case of a lockup we REALLY don't want to wait
for anything.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix VM page table setup on SI

Cayman and trinity allow for variable sized VM page
tables, but SI requires that all page tables be the
same size. The current code assumes variablely sized
VM page tables so SI may end up with part of each page
table overlapping with other memory which could end
up being interpreted by the VM hw as garbage.

Change the code to better accomodate SI. Allocate enough
space for at least 2 full page tables and always set
last_pfn to max_pfn on SI so each VM is backed by a full
page table. This limits us to only 2 VMs active at any
given time on SI. This will be rectified and the code can
be reunified once we move to two level page tables.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>
Cc: stable@vger.kernel.org
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon: replace pflip and sw_int counters with atomics

So we can skip the locking. Also renames sw_int to
ring_int, cause that better matches its purpose.

Signed-off-by: Christian Koenig <christian.koenig@amd.com>

drm/radeon: apply Murphy's law to the kms irq code v3

1. It is really dangerous to have more than one
spinlock protecting the same information.

2. radeon_irq_set sometimes wasn't called with lock
protection, so it can happen that more than one
CPU would tamper with the irq regs at the same
time.

3. The pm.gui_idle variable was assuming that the 3D
engine wasn't becoming idle between testing the
register and setting the variable. So just remove
it and test the register directly.

v2: Also handle the hpd irq code the same way.
v3: Rename hpd parameter for clarification.

Signed-off-by: Christian Koenig <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

drm/radeon: fix & improve ih ring handling v3

The spinlock was actually there to protect the
rptr, but rptr was read outside of the locked area.

Also we don't really need a spinlock here, an
atomic should to quite fine since we only need to
prevent it from being reentrant.

v2: Keep the spinlock....
v3: Back to an atomic again after finding & fixing the real bug.

Signed-off-by: Christian Koenig <christian.koenig@amd.com>

drm/radeon: remove radeon_fence_create

It is completely unnecessary to create fences
before they are emitted, so remove it and a bunch
of checks if fences are emitted or not.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>

drm/radeon: fix gpu_init on si

- Properly set up the RBs
- Properly set up the SPI
- Properly set up gb_addr_config

This should fix rendering issues on certain cards.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon: add PRIME support (v2)

This adds prime->fd and fd->prime support to radeon.
It passes the sg object to ttm and then populates
the gart entries using it.

Compile tested only.

v2: stub kmap + use new helpers + add reimporting

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon: make use of radeon_gem_init() consistent

All radeon_gem_init() does is initialize the gem objects
list. radeon_device.c does this explicitly. r600+ calls
radeon_gem_init() so the list gets initialized twice. Older
asics don't call it at all and rely on the the init in
radeon_device.c. Just call radeon_gem_init() in radeon_device.c
and remove the explicit calls from all the newer asics.

All asics call radeon_gem_fini() in their fini pathes. That
could possibly be cleaned up too.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon: Also reset BCI on SI GPU reset.

Without this, e.g. egltri_screen looks scrambled after a GPU reset.

Signed-off-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon: simplify semaphore handling v2

Directly use the suballocator to get small chunks of memory.
It's equally fast and doesn't crash when we encounter a GPU reset.

v2: rebased on new SA interface.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Signed-off-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon: make forcing ring activity a common function

Nothing chipset or ring specific with it,
so also move it to radon_ring.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon: move lockup detection code into radeon_ring.c

It isn't chipset specific, so it makes no sense
to have that inside r100.c.

Signed-off-by: Christian König <deathsimple@vodafone.de>
Reviewed-by: Jerome Glisse <jglisse@redhat.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/si: add missing radeon_bo_unreserve in si_rlc_init() v2

Forget to unreserve after pinning. This can lead to problems in
soft reset and resume.

v2: rework patch as per Michel's suggestion.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon: enable pci bus mastering after card is initialised (v2)

This closes a race seen with kexec where we enable PCI bus mastering
but the card has been reinitialised fully yet.

This was previously fixed by a patch from Jerome, but this should
close the race completely.

v2: add SI support as suggested by Alex.

Reported-and-tested-by: Markus Trippelsdorf <markus@trippelsdorf.de>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: add support for ucode loading on trinity (v2)

v2: fix check for MC ucode from Tom.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: DCE6.1 watermark updates for TN

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: fill in startup/shutdown callbacks for SI

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: add support for interrupts on SI

This is mostly identical to evergreen/ni, however
there are some additional fields in the IV vector
for RINGID and VMID.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: Add support for RLC init on SI

RLC handles the interrupt controller and other tasks
on the GPU.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: add IB and fence dispatch functions for SI

Support both IBs (DE) and CONST IBs (CE).

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: add support for CP setup on SI

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: add support for MC ucode loading on SI

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: add ucode loading for SI

Currently the driver required 5 sets of ucode:
1. pfp - pre-fetch parser, part of the CP
2. me - micro engine, part of the CP
3. ce - constant engine, part of the CP
4. rlc - interrupt controller
5. mc - memory controller

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: add VM CS checker for SI

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: add support for MC/VM setup on SI

Sets up the VM and adds support for the new VM ioctls.

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: Add support for SI GPU reset

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: add gpu init support for SI

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: add support for internal thermal sensor on SI

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>

drm/radeon/kms: add initial DCE6 display watermark support

Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>