fsldma: fix very broken 32-bit ppc ioread64 functionality

Commit ef91bb196b0d ("kernel.h: Silence sparse warning in
lower_32_bits") caused new warnings to show in the fsldma driver, but
that commit was not to blame: it only exposed some very incorrect code
that tried to take the low 32 bits of an address.

That made no sense for multiple reasons, the most notable one being that
that code was intentionally limited to only 32-bit ppc builds, so "only
low 32 bits of an address" was completely nonsensical. There were no
high bits to mask off to begin with.

But even more importantly fropm a correctness standpoint, turning the
address into an integer then caused the subsequent address arithmetic to
be completely wrong too, and the "+1" actually incremented the address
by one, rather than by four.

Which again was incorrect, since the code was reading two 32-bit values
and trying to make a 64-bit end result of it all. Surprisingly, the
iowrite64() did not suffer from the same odd and incorrect model.

This code has never worked, but it's questionable whether anybody cared:
of the two users that actually read the 64-bit value (by way of some C
preprocessor hackery and eventually the 'get_cdar()' inline function),
one of them explicitly ignored the value, and the other one might just
happen to work despite the incorrect value being read.

This patch at least makes it not fail the build any more, and makes the
logic superficially sane. Whether it makes any difference to the code
_working_ or not shall remain a mystery.

Compile-tested-by: Guenter Roeck <linux@roeck-us.net>
Reviewed-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 40

Based on 1 normalized pattern(s):

this is free software you can redistribute it and or modify it under
the terms of the gnu general public license as published by the free
software foundation either version 2 of the license or at your
option any later version

extracted by the scancode license scanner the SPDX license identifier

GPL-2.0-or-later

has been chosen to replace the boilerplate/reference in 14 file(s).

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc)
Reviewed-by: Allison Randal <allison@lohutok.net>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190520170857.915677517@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

dmaengine: fsldma: Add 64-bit I/O accessors for powerpc64

Otherwise 64-bit PPC builds fail with undefined references
to these accessors.

Cc: Peng Ma <peng.ma@nxp.com>
Cc: Wen He <wen.he_1@nxp.com>
Fixes: 68997fff94afa (" dmaengine: fsldma: Adding macro FSL_DMA_IN/OUT implement for ARM platform")
Signed-off-by: Scott Wood <oss@buserror.net>
Signed-off-by: Vinod Koul <vkoul@kernel.org>

dmaengine: fsldma: Adding macro FSL_DMA_IN/OUT implement for ARM platform

This patch add the macro FSL_DMA_IN/OUT implement for ARM platform.

Signed-off-by: Wen He <wen.he_1@nxp.com>
Signed-off-by: Peng Ma <peng.ma@nxp.com>
Signed-off-by: Vinod Koul <vkoul@kernel.org>

dmaengine: fsldma: Replace DMA_IN/OUT by FSL_DMA_IN/OUT

This patch implement a standard macro call functions is
used to NXP dma drivers.

Signed-off-by: Wen He <wen.he_1@nxp.com>
Signed-off-by: Peng Ma <peng.ma@nxp.com>
Signed-off-by: Vinod Koul <vkoul@kernel.org>

dmaengine: fsldma: set BWC, DAHTS and SAHTS values correctly

The bits of BWC, DAHTS and SAHTS in the DMA mode register must be cleared
before a new value can be or-ed in.

Signed-off-by: Thomas Breitung <thomas.breitung@izt-labs.de>
Signed-off-by: Wolfgang Ocker <weo@reccoware.de>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: fsldma: declare slave capabilities for the generic code

Since commit ecc19d17868b ("dmaengine: Add a warning for drivers not
using the generic slave caps retrieval"), the dma drivers are required
to fill the caps infos in order to support generic slaves caps
retrieval. Otherwise we will get a warning like this:
WARNING: at drivers/dma/dmaengine.c:830
Modules linked in:
CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W 3.19.0-rc2-next-20150106-dirty #271
task: c0000001f70a0000 ti: c0000001f7044000 task.ti: c0000001f7044000
NIP: c00000000032b238 LR: c00000000032b234 CTR: c00000000001d258
REGS: c0000001f7047330 TRAP: 0700 Tainted: G W (3.19.0-rc2-next-20150106-dirty)
MSR: 0000000080029000 <CE,EE,ME> CR: 24adbe22 XER: 20000000
SOFTE: 1
GPR00: c00000000032b234 c0000001f70475b0 c0000000009b4848 0000000000000040
GPR04: 0000000000000001 0000000000000001 0000000000000000 000000000000000f
GPR08: 0000000000000000 c000000000902988 c000000000902988 00000000000052c8
GPR12: 0000000024adbe22 c00000000fff4000 c000000000002038 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR24: 0000000000000000 0000000000000000 c000000000972dc8 c0000000007e6fd0
GPR28: c0000001f76d1d30 c0000001f76d1c10 c0000001f76d1c00 0000000000000000
NIP [c00000000032b238] .dma_async_device_register+0x3f8/0x5b8
LR [c00000000032b234] .dma_async_device_register+0x3f4/0x5b8
Call Trace:
[c0000001f70475b0] [c00000000032b234] .dma_async_device_register+0x3f4/0x5b8 (unreliable)
[c0000001f70476a0] [c00000000032ca78] .fsldma_of_probe+0x298/0x438
[c0000001f7047750] [c00000000037080c] .platform_drv_probe+0x50/0x9c
[c0000001f70477d0] [c00000000036e74c] .really_probe+0xa4/0x29c
[c0000001f7047870] [c00000000036eae4] .__driver_attach+0x100/0x104
[c0000001f7047900] [c00000000036c1f0] .bus_for_each_dev+0x84/0xe4
[c0000001f70479a0] [c00000000036e164] .driver_attach+0x24/0x38
[c0000001f7047a10] [c00000000036dcc8] .bus_add_driver+0x1c8/0x2ac
[c0000001f7047ab0] [c00000000036f14c] .driver_register+0x8c/0x158
[c0000001f7047b30] [c0000000003707a8] .__platform_driver_register+0x6c/0x80
[c0000001f7047ba0] [c000000000898a3c] .fsldma_init+0x2c/0x40
[c0000001f7047c10] [c000000000001818] .do_one_initcall+0xb8/0x234
[c0000001f7047d00] [c000000000878e2c] .kernel_init_freeable+0x188/0x268
[c0000001f7047db0] [c000000000002054] .kernel_init+0x1c/0xfc8
[c0000001f7047e30] [c000000000000884] .ret_from_kernel_thread+0x58/0xd4
Instruction dump:
7fb9f840 3bffffe0 409effac 7f54d378 48000060 813d0050 2f890000 40befdd0
3c62ffe3 38632450 482f0aa9 60000000 <0fe00000> 4bfffdb8 7f03c378 482ed465

Signed-off-by: Kevin Hao <haokexin@gmail.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: Freescale: change descriptor release process for supporting async_tx

Fix the potential risk when enable config NET_DMA and ASYNC_TX. Async_tx is
lack of support in current release process of dma descriptor, all descriptors
will be released whatever is acked or no-acked by async_tx, so there is a
potential race condition when dma engine is uesd by others clients (e.g. when
enable NET_DMA to offload TCP).

In our case, a race condition which is raised when use both of talitos and
dmaengine to offload xor is because napi scheduler will sync all pending
requests in dma channels, it affects the process of raid operations due to
ack_tx is not checked in fsl dma. The no-acked descriptor is freed which is
submitted just now, as a dependent tx, this freed descriptor trigger
BUG_ON(async_tx_test_ack(depend_tx)) in async_tx_submit().

TASK = ee1a94a0[1390] 'md0_raid5' THREAD: ecf40000 CPU: 0
GPR00: 00000001 ecf41ca0 ee44/921a94a0 0000003f 00000001 c00593e4 00000000 00000001
GPR08: 00000000 a7a7a7a7 00000001 045/920000002 42028042 100a38d4 ed576d98 00000000
GPR16: ed5a11b0 00000000 2b162000 00000200 046/920000000 2d555000 ed3015e8 c15a7aa0
GPR24: 00000000 c155fc40 00000000 ecb63220 ecf41d28 e47/92f640bb0 ef640c30 ecf41ca0
NIP [c02b048c] async_tx_submit+0x6c/0x2b4
LR [c02b068c] async_tx_submit+0x26c/0x2b4
Call Trace:
[ecf41ca0] [c02b068c] async_tx_submit+0x26c/0x2b448/92 (unreliable)
[ecf41cd0] [c02b0a4c] async_memcpy+0x240/0x25c
[ecf41d20] [c0421064] async_copy_data+0xa0/0x17c
[ecf41d70] [c0421cf4] __raid_run_ops+0x874/0xe10
[ecf41df0] [c0426ee4] handle_stripe+0x820/0x25e8
[ecf41e90] [c0429080] raid5d+0x3d4/0x5b4
[ecf41f40] [c04329b8] md_thread+0x138/0x16c
[ecf41f90] [c008277c] kthread+0x8c/0x90
[ecf41ff0] [c0011630] kernel_thread+0x4c/0x68

Another modification in this patch is the change of completed descriptors,
there is a potential risk which caused by exception interrupt, all descriptors
in ld_running list are seemed completed when an interrupt raised, it works fine
under normal condition, but if there is an exception occured, it cannot work as
our excepted. Hardware should not be depend on s/w list, the right way is to
read current descriptor address register to find the last completed descriptor.
If an interrupt is raised by an error, all descriptors in ld_running should not
be seemed finished, or these unfinished descriptors in ld_running will be
released wrongly.

A simple way to reproduce:
Enable dmatest first, then insert some bad descriptors which can trigger
Programming Error interrupts before the good descriptors. Last, the good
descriptors will be freed before they are processsed because of the exception
intrerrupt.

Note: the bad descriptors are only for simulating an exception interrupt. This
case can illustrate the potential risk in current fsl-dma very well.

Signed-off-by: Hongbo Zhang <hongbo.zhang@freescale.com>
Signed-off-by: Qiang Liu <qiang.liu@freescale.com>
Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: Freescale: add suspend resume functions for DMA driver

This patch adds suspend and resume functions for Freescale DMA driver.

Signed-off-by: Hongbo Zhang <hongbo.zhang@freescale.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

DMA: Freescale: change BWC from 256 bytes to 1024 bytes

Freescale DMA has a feature of BandWidth Control (ab. BWC), which is currently
256 bytes and should be changed to 1024 bytes for best DMA throughput.
Changing BWC from 256 to 1024 will improve DMA performance much, in cases
whatever one channel is running or multi channels are running simultanously,
large or small buffers are copied. And this change doesn't impact memory
access performance remarkably, lmbench tests show that for some cases the
memory performance are decreased very slightly, while the others are even
better.
Tested on T4240.

Signed-off-by: Hongbo Zhang <hongbo.zhang@freescale.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

DMA: Freescale: update driver to support 8-channel DMA engine

This patch adds support to 8-channel DMA engine, thus the driver works for both
the new 8-channel and the legacy 4-channel DMA engines.

Signed-off-by: Hongbo Zhang <hongbo.zhang@freescale.com>
Signed-off-by: Vinod Koul <vinod.koul@intel.com>

dmaengine: move last completed cookie into generic dma_chan structure

Every DMA engine implementation declares a last completed dma cookie
in their private dma channel structures. This is pointless, and
forces driver specific code. Move this out into the common dma_chan
structure.

Signed-off-by: Russell King <rmk+kernel@arm.linux.org.uk>
Tested-by: Linus Walleij <linus.walleij@linaro.org>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Acked-by: Jassi Brar <jassisinghbrar@gmail.com>
[imx-sdma.c & mxs-dma.c]
Tested-by: Shawn Guo <shawn.guo@linaro.org>
Signed-off-by: Vinod Koul <vinod.koul@linux.intel.com>

fsldma: fix controller lockups

Enabling poisoning in the dmapool API quickly showed that the DMA
controller was fetching descriptors that should not have been in use.
This has caused intermittent controller lockups during testing.

I have been unable to figure out the exact set of conditions which cause
this to happen. However, I believe it is related to the driver using the
hardware registers to track whether the controller is busy or not. The
code can incorrectly decide that the hardware is idle due to lag between
register writes and the hardware actually becoming busy.

To fix this, the driver has been reworked to explicitly track the state
of the hardware, rather than try to guess what it is doing based on the
register values.

This has passed dmatest with 10 threads per channel, 100000 iterations
per thread several times without error. Previously, this would fail
within a few seconds.

Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: minor codingstyle and consistency fixes

This fixes some minor violations of the coding style. It also changes
the style of the device_prep_dma_*() function definitions so they are
identical.

Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: use channel name in printk output

This makes debugging the driver much easier when multiple channels are
running concurrently. In addition, you can see how much descriptor
memory each channel has allocated via the dmapool API in sysfs.

Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: fix issue of slow dma

Fixed fsl dma slow issue by initializing dma mode register with
bandwidth control. It boosts dma performance and should works
with 85xx board.

Signed-off-by: Forrest Shi <b29237@freescale.com>
Signed-off-by: Li Yang <leoli@freescale.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: major cleanups and fixes

Fix locking. Use two queues in the driver, one for pending transacions, and
one for transactions which are actually running on the hardware. Call
dma_run_dependencies() on descriptor cleanup so that the async_tx API works
correctly.

There are a number of places throughout the code where lists of descriptors
are freed in a loop. Create functions to handle this, and use them instead
of open-coding the loop each time.

Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: clean up the OF subsystem routines

This fixes some errors in the cleanup paths of the OF subsystem, including
missing checks for ioremap failing. Also, some variables were renamed for
brevity.

Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: rename dest to dst for uniformity

Most functions in the standard library use "dst" as a parameter, rather
than "dest". This renames all use of "dest" to "dst" to match the usual
convention.

Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: rename struct fsl_dma_chan to struct fsldma_chan

This is the beginning of a cleanup which will change all instances of
"fsl_dma" to "fsldma" to match the name of the driver itself.

Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: remove unused structure members

Remove some unused members from the fsldma data structures. A few trivial
uses of struct resource were converted to use the stack rather than keeping
the memory allocated for the lifetime of the driver.

Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: split apart external pause and request count features

When using the Freescale DMA controller in external control mode, both the
request count and external pause bits need to be setup correctly. This was
being done with the same function.

The 83xx controller lacks the external pause feature, but has a similar
feature called external start. This feature requires that the request count
bits be setup correctly.

Split the function into two parts, to make it possible to use the external
start feature on the 83xx controller.

Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: implement a private tx_list

Drop fsldma's use of tx_list from struct dma_async_tx_descriptor in
preparation for removal of this field.

Cc: Li Yang <leoli@freescale.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: use PCI Read Multiple command

By default, the Freescale 83xx DMA controller uses the PCI Read Line
command when reading data over the PCI bus. Setting the controller to use
the PCI Read Multiple command instead allows the controller to read much
larger bursts of data, which provides a drastic speed increase.

The slowdown due to using PCI Read Line was only observed when a PCI-to-PCI
bridge was between the devices trying to communicate.

A simple test driver showed an increase from 4MB/sec to 116MB/sec when
performing DMA over the PCI bus. Using DMA to transfer between blocks of
local SDRAM showed no change in performance with this patch. The dmatest
driver was also used to verify the correctness of the transfers, and showed
no errors.

Signed-off-by: Ira W. Snyder <iws@ovro.caltech.edu>
Acked-by: Timur Tabi <timur@freescale.com>
Acked-by: Kumar Gala <galak@kernel.crashing.org>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fsldma: allow Freescale Elo DMA driver to be compiled as a module

Modify the Freescale Elo / Elo Plus DMA driver so that it can be compiled as
a module.

The primary change is to stop treating the DMA controller as a bus, and the
DMA channels as devices on the bus. This is because the Open Firmware (OF)
kernel code does not allow busses to be removed, so although we can call
of_platform_bus_probe() to probe the DMA channels, there is no
of_platform_bus_remove(). Instead, the DMA channels are manually probed,
similar to what fsl_elbc_nand.c does.

Cc: Scott Wood <scottwood@freescale.com>
Acked-by: Li Yang <leoli@freescale.com>
Signed-off-by: Timur Tabi <timur@freescale.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

fix the broken annotations in fsldma

a) every bitwise declaration will give a unique type; use typedefs.

b) no need to bother with the stuff pointed to by iomem pointers,
unless it's accessed directly. noderef will force us to use helpers
anyway.

Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

fsldma: Fix the DMA halt when using DMA_INTERRUPT async_tx transfer.

The DMA_INTERRUPT async_tx is a NULL transfer, thus the BCR(count register)
is 0. When the transfer started with a byte count of zero, the DMA
controller will triger a PE(programming error) event and halt, not a normal
interrupt. I add special codes for PE event and DMA_INTERRUPT
async_tx testing.

Signed-off-by: Zhang Wei <wei.zhang@freescale.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>

dmaengine: add driver for Freescale MPC85xx DMA controller

The driver implements DMA engine API for Freescale MPC85xx DMA controller,
which could be used by devices in the silicon. The driver supports the
Basic mode of Freescale MPC85xx DMA controller. The MPC85xx processors
supported include MPC8540/60, MPC8555, MPC8548, MPC8641 and so on.

The MPC83xx(MPC8349, MPC8360) are also supported.

[kamalesh@linux.vnet.ibm.com: build fix]
[dan.j.williams@intel.com: merge mm fixes, rebase on async_tx-2.6.25]
Signed-off-by: Zhang Wei <wei.zhang@freescale.com>
Signed-off-by: Ebony Zhu <ebony.zhu@freescale.com>
Acked-by: Kumar Gala <galak@gate.crashing.org>
Cc: Shannon Nelson <shannon.nelson@intel.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>