dmaengine: virt-dma: Add missing locking around list operations

All list operations are protected by &vc->lock. As vchan_vdesc_fini()
is called unlocked add the missing locking around the list operations.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
Reviewed-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
Tested-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
Link: https://lore.kernel.org/r/20191216105328.15198-6-s.hauer@pengutronix.de
Signed-off-by: Vinod Koul <vkoul@kernel.org>

dmaengine: virt-dma: Do not call desc_free() under a spin_lock

vchan_vdesc_fini() shouldn't be called under a spin_lock. This is done
in two places, once in vchan_terminate_vdesc() and once in
vchan_synchronize(). Instead of freeing the vdesc right away, collect
the aborted vdescs on a separate list and free them along with the other
vdescs. The terminated descs are also freed in vchan_synchronize as done
before this patch.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
Reviewed-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
Tested-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
Link: https://lore.kernel.org/r/20191216105328.15198-5-s.hauer@pengutronix.de
Signed-off-by: Vinod Koul <vkoul@kernel.org>

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500

Based on 2 normalized pattern(s):

this program is free software you can redistribute it and or modify
it under the terms of the gnu general public license version 2 as
published by the free software foundation

this program is free software you can redistribute it and or modify
it under the terms of the gnu general public license version 2 as
published by the free software foundation #

extracted by the scancode license scanner the SPDX license identifier

GPL-2.0-only

has been chosen to replace the boilerplate/reference in 4122 file(s).

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Enrico Weigelt <info@metux.net>
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Allison Randal <allison@lohutok.net>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190604081206.933168790@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

dmaengine: virt-dma: store result on dma descriptor

This allows each virtual channel to store information about each transfer
that completed, i.e. which transfer succeeded (or which failed) and if
there was any residue data on each (completed) transfer.

Signed-off-by: Alexandru Ardelean <alexandru.ardelean@analog.com>
Signed-off-by: Vinod Koul <vkoul@kernel.org>

dmaengine: virt-dma: Support for race free transfer termination

Even with the introduced vchan_synchronize() we can face race when
terminating a cyclic transfer.

If the terminate_all is called after the interrupt handler called
vchan_cyclic_callback(), but before the vchan_complete tasklet is called:
vc->cyclic is set to the cyclic descriptor, but the descriptor itself was
freed up in the driver's terminate_all() callback.
When the vhan_complete() is executed it will try to fetch the vc->cyclic
vdesc, but the pointer is pointing now to uninitialized memory leading to
(hard to reproduce) kernel crash.

In order to fix this, drivers should:
- call vchan_terminate_vdesc() from their terminate_all callback instead
calling their free_desc function to free up the descriptor.
- implement device_synchronize callback and call vchan_synchronize().

This way we can make sure that the descriptor is only going to be freed up
after the vchan_callback was executed in a safe manner.

Signed-off-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: virt-dma: Add helper to free/reuse a descriptor

The vchan_vdesc_fini() can be used to free or reuse a given descriptor
after it has been marked as completed.

Signed-off-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: virt-dma: move function declarations

We get 2 warnings when building kernel with W=1:
drivers/dma/virt-dma.c:22:14: warning: no previous prototype for 'vchan_tx_submit' [-Wmissing-prototypes]
drivers/dma/virt-dma.c:52:5: warning: no previous prototype for 'vchan_tx_desc_free' [-Wmissing-prototypes]

In fact, these two functions are incorrectly declared in a function.

So this patch moves function declarations out of this function.

Signed-off-by: Baoyou Xie <baoyou.xie@linaro.org>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: cleanup with list_first_entry_or_null()

The combo of list_empty() check and return list_first_entry()
can be replaced with list_first_entry_or_null().

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: virt-dma: don't always free descriptor upon completion

This patch attempts to enhance the case of a transfer submitted multiple
times, and where the cost of creating the descriptors chain is not
negligible.

This happens with big video buffers (several megabytes, ie. several
thousands of linked descriptors in one scatter-gather list). In these
cases, a video driver would want to do :
- tx = dmaengine_prep_slave_sg()
- dma_engine_submit(tx);
- dma_async_issue_pending()
- wait for video completion
- read video data (or not, skipping a frame is also possible)
- dma_engine_submit(tx)
=> here, the descriptors chain recalculation will take time
=> the dma coherent allocation over and over might create holes in
the dma pool, which is counter-productive.
- dma_async_issue_pending()
- etc ...

In order to cope with this case, virt-dma is modified to prevent freeing
the descriptors upon completion if DMA_CTRL_REUSE flag is set in the
transfer.

This patch is a respin of the former DMA_CTRL_ACK approach, which was
reverted due to a regression in audio drivers.

Signed-off-by: Robert Jarzmik <robert.jarzmik@free.fr>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: virt-dma: Add synchronization helper function

Add a synchronize helper function for the virt-dma library. The function
makes sure that any scheduled descriptor complete callbacks have finished
running before the function returns.

This needs to be called by drivers using virt-dma in their
device_synchronize() callback. Depending on the driver additional
operations might be necessary in addition to calling vchan_synchronize() to
ensure proper synchronization.

Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: virt-dma: Fix kernel-doc annotations

In kernel-doc annotations parameters need to start with a @ for them to be
properly recognized. Add those where missing for virt-dma.

Signed-off-by: Lars-Peter Clausen <lars@metafoo.de>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

Revert "dmaengine: virt-dma: don't always free descriptor upon completion"

This reverts commit b9855f03d560d351e95301b9de0bc3cad3b31fe9.
The patch break existing DMA usage case. For example, audio SOC
dmaengine never release channel and cause virt-dma to cache too
much memory in descriptor to exhaust system memory.

Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: virt-dma: don't always free descriptor upon completion

This patch attempts to enhance the case of a transfer submitted multiple
times, and where the cost of creating the descriptors chain is not
negligible.

This happens with big video buffers (several megabytes, ie. several
thousands of linked descriptors in one scatter-gather list). In these
cases, a video driver would want to do :
- tx = dmaengine_prep_slave_sg()
- dma_engine_submit(tx);
- dma_async_issue_pending()
- wait for video completion
- read video data (or not, skipping a frame is also possible)
- dma_engine_submit(tx)
=> here, the descriptors chain recalculation will take time
=> the dma coherent allocation over and over might create holes in
the dma pool, which is counter-productive.
- dma_async_issue_pending()
- etc ...

In order to cope with this case, virt-dma is modified to prevent freeing
the descriptors upon completion if DMA_CTRL_ACK flag is set in the
transfer.

Signed-off-by: Robert Jarzmik <robert.jarzmik@free.fr>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dma: fix vchan_cookie_complete() debug print

vd->tx.cookie is set zero on dma_cookie_complete(),
save to local before printing it.

Signed-off-by: Jonas Jensen <jonas.jensen@gmail.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: virt-dma: add support for cyclic DMA periodic callbacks

Add support for cyclic DMA's periodic callbacks. Drivers are expected
to call vchan_cyclic_callback() when a period has completed, which will
schedule the tasklet to make the callback into the driver.

As callbacks are made from tasklet context, it is important to realise
that we don't guarantee a callback for each completed period, but for
N completed periods where N may be greater than one.

Tested-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

dmaengine: virt-dma: vchan_find_desc()

Add a function to find a descriptor within the depths of the
virtualized DMA channel support. Needed for tx_status functionality.

Tested-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>

dmaengine: split out virtual channel DMA support from sa11x0 driver

Split the virtual slave channel DMA support from the sa11x0 driver so
this code can be shared with other slave DMA engine drivers.

Acked-by: Linus Walleij <linus.walleij@linaro.org>
Tested-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>