soc: fsl: dpio: read the consumer index from the cache inhibited area

Once we added support in the dpaa2-eth for driver level software TSO we
observed the following situation: if the EQCR CI (consumer index) is
read from the cache-enabled area we sometimes end up with a computed
value of available enqueue entries bigger than the size of the ring.

This eventually will lead to the multiple enqueue of the same FD which
will determine the same FD to end up on the Tx confirmation path and the
same skb being freed twice.

Just read the consumer index from the cache inhibited area so that we
avoid this situation.

Signed-off-by: Ioana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

soc: fsl: dpio: use the combined functions to protect critical zone

In orininal code, use 2 function spin_lock() and local_irq_save() to
protect the critical zone. But when enable the kernel debug config,
there are below inconsistent lock state detected.
================================
WARNING: inconsistent lock state
5.10.63-yocto-standard #1 Not tainted
--------------------------------
inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-W} usage.
lock_torture_wr/226 [HC0[0]:SC1[5]:HE1:SE0] takes:
ffff002005b2dd80 (&p->access_spinlock){+.?.}-{3:3}, at: qbman_swp_enqueue_multiple_mem_back+0x44/0x270
{SOFTIRQ-ON-W} state was registered at:
lock_acquire.part.0+0xf8/0x250
lock_acquire+0x68/0x84
_raw_spin_lock+0x68/0x90
qbman_swp_enqueue_multiple_mem_back+0x44/0x270
......
cryptomgr_test+0x38/0x60
kthread+0x158/0x164
ret_from_fork+0x10/0x38
irq event stamp: 4498
hardirqs last enabled at (4498): [<ffff800010fcf980>] _raw_spin_unlock_irqrestore+0x90/0xb0
hardirqs last disabled at (4497): [<ffff800010fcffc4>] _raw_spin_lock_irqsave+0xd4/0xe0
softirqs last enabled at (4458): [<ffff8000100108c4>] __do_softirq+0x674/0x724
softirqs last disabled at (4465): [<ffff80001005b2a4>] __irq_exit_rcu+0x190/0x19c

other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&p->access_spinlock);
<Interrupt>
lock(&p->access_spinlock);
*** DEADLOCK ***

So, in order to avoid deadlock, use the combined functions
spin_lock_irqsave/spin_unlock_irqrestore() to protect critical zone.

Fixes: 3b2abda7d28c ("soc: fsl: dpio: Replace QMAN array mode with ring mode enqueue")
Cc: stable@vger.kernel.org
Signed-off-by: Meng Li <Meng.Li@windriver.com>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: rename the enqueue descriptor variable

The struct qbman_eq_desc 'd' variable declaration is covering one of the
function parameters. This has no functional impact since this function
parameter was not used after the new declaration.
Even so, rename the variable so that we make the code more readable.

Fixes: 3b2abda7d28c ("soc: fsl: dpio: Replace QMAN array mode with ring mode enqueue")
Signed-off-by: Youri Querry <youri.querry_1@nxp.com>
Signed-off-by: Ioana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: Unsigned compared against 0 in qbman_swp_set_irq_coalescing()

Coverity complains of unsigned compare against 0. There are 2 cases in
this function:

1821 itp = (irq_holdoff * 1000) / p->desc->qman_256_cycles_per_ns;

CID 121131 (#1 of 1): Unsigned compared against 0 (NO_EFFECT)
unsigned_compare: This less-than-zero comparison of an unsigned value is never true. itp < 0U.
1822 if (itp < 0 || itp > 4096) {
1823 max_holdoff = (p->desc->qman_256_cycles_per_ns * 4096) / 1000;
1824 pr_err("irq_holdoff must be between 0..%dus\n", max_holdoff);
1825 return -EINVAL;
1826 }
1827
unsigned_compare: This less-than-zero comparison of an unsigned value is never true. irq_threshold < 0U.
1828 if (irq_threshold >= p->dqrr.dqrr_size || irq_threshold < 0) {
1829 pr_err("irq_threshold must be between 0..%d\n",
1830 p->dqrr.dqrr_size - 1);
1831 return -EINVAL;
1832 }

Fix this by removing the comparisons altogether as they are incorrect. Zero is
a possible value in either case. Also fix a minor comment typo and update the
2 pr_err() calls to use %u formatting as well as be more precise regarding
the exact error.

Fixes: ed1d2143fee5 ("soc: fsl: dpio: add support for irq coalescing per software portal")
Cc: Ioana Ciornei <ioana.ciornei@nxp.com>
Cc: Roy Pledge <Roy.Pledge@nxp.com>
Cc: Li Yang <leoyang.li@nxp.com>
Cc: linux-kernel@vger.kernel.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: netdev@vger.kernel.org
Signed-off-by: Tim Gardner <tim.gardner@canonical.com>
Tested-by: Ioana Ciornei <ioana.ciornei@nxp.com>
Reviewed-by: Ioana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

soc: fsl: dpio: add support for irq coalescing per software portal

In DPAA2 based SoCs, the IRQ coalesing support per software portal has 2
configurable parameters:
- the IRQ timeout period (QBMAN_CINH_SWP_ITPR): how many 256 QBMAN
cycles need to pass until a dequeue interrupt is asserted.
- the IRQ threshold (QBMAN_CINH_SWP_DQRR_ITR): how many dequeue
responses in the DQRR ring would generate an IRQ.

Add support for setting up and querying these IRQ coalescing related
parameters.

Signed-off-by: Ioana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

soc: fsl: dpio: qbman-portal: Fix a bunch of kernel-doc misdemeanours

Fixes the following W=1 kernel build warning(s):

drivers/soc/fsl/dpio/qbman-portal.c:430: warning: Function parameter or member 'inhibit' not described in 'qbman_swp_interrupt_set_inhibit'
drivers/soc/fsl/dpio/qbman-portal.c:430: warning: Excess function parameter 'mask' description in 'qbman_swp_interrupt_set_inhibit'
drivers/soc/fsl/dpio/qbman-portal.c:518: warning: Function parameter or member 'd' not described in 'qbman_eq_desc_clear'
drivers/soc/fsl/dpio/qbman-portal.c:529: warning: Function parameter or member 'respond_success' not described in 'qbman_eq_desc_set_no_orp'
drivers/soc/fsl/dpio/qbman-portal.c:529: warning: Excess function parameter 'response_success' description in 'qbman_eq_desc_set_no_orp'
drivers/soc/fsl/dpio/qbman-portal.c:941: warning: Function parameter or member 's' not described in 'qbman_swp_push_get'
drivers/soc/fsl/dpio/qbman-portal.c:941: warning: Excess function parameter 'p' description in 'qbman_swp_push_get'
drivers/soc/fsl/dpio/qbman-portal.c:955: warning: Function parameter or member 's' not described in 'qbman_swp_push_set'
drivers/soc/fsl/dpio/qbman-portal.c:955: warning: Excess function parameter 'p' description in 'qbman_swp_push_set'
drivers/soc/fsl/dpio/qbman-portal.c:1052: warning: Function parameter or member 'd' not described in 'qbman_pull_desc_set_fq'
drivers/soc/fsl/dpio/qbman-portal.c:1065: warning: Function parameter or member 'd' not described in 'qbman_pull_desc_set_wq'
drivers/soc/fsl/dpio/qbman-portal.c:1079: warning: Function parameter or member 'd' not described in 'qbman_pull_desc_set_channel'
drivers/soc/fsl/dpio/qbman-portal.c:1403: warning: Function parameter or member 'd' not described in 'qbman_release_desc_clear'
drivers/soc/fsl/dpio/qbman-portal.c:1412: warning: Function parameter or member 'd' not described in 'qbman_release_desc_set_bpid'
drivers/soc/fsl/dpio/qbman-portal.c:1412: warning: Function parameter or member 'bpid' not described in 'qbman_release_desc_set_bpid'
drivers/soc/fsl/dpio/qbman-portal.c:1421: warning: Function parameter or member 'd' not described in 'qbman_release_desc_set_rcdi'
drivers/soc/fsl/dpio/qbman-portal.c:1421: warning: Function parameter or member 'enable' not described in 'qbman_release_desc_set_rcdi'

Cc: Roy Pledge <Roy.Pledge@nxp.com>
Cc: Li Yang <leoyang.li@nxp.com>
Cc: linuxppc-dev@lists.ozlabs.org
Signed-off-by: Lee Jones <lee.jones@linaro.org>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: remove set but not used 'addr_cena'

This addresses the following gcc warning with "make W=1":

drivers/soc/fsl/dpio/qbman-portal.c: In function
‘qbman_swp_enqueue_multiple_direct’:
drivers/soc/fsl/dpio/qbman-portal.c:650:11: warning: variable
‘addr_cena’ set but not used [-Wunused-but-set-variable]
650 | uint64_t addr_cena;
| ^~~~~~~~~

Reported-by: Hulk Robot <hulkci@huawei.com>
Signed-off-by: Jason Yan <yanaijie@huawei.com>
Reported-by: kernel test robot <lkp@intel.com>
Reviewed-by: Krzysztof Kozlowski <krzk@kernel.org>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: Remove unused inline function qbman_write_eqcr_am_rt_register

There's no callers in-tree anymore since commit
3b2abda7d28c ("soc: fsl: dpio: Replace QMAN array mode with ring mode enqueue")

Signed-off-by: YueHaibing <yuehaibing@huawei.com>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: properly compute the consumer index

Mask the consumer index before using it. Without this, we would be
writing frame descriptors beyond the ring size supported by the QBMAN
block.

Fixes: 3b2abda7d28c ("soc: fsl: dpio: Replace QMAN array mode with ring mode enqueue")
Signed-off-by: Ioana Ciornei <ioana.ciornei@nxp.com>
Acked-by: Li Yang <leoyang.li@nxp.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

soc: fsl: dpio: fix incorrect pointer conversions

Building dpio for 32 bit shows a new compiler warning from converting
a pointer to a u64:

drivers/soc/fsl/dpio/qbman-portal.c: In function 'qbman_swp_enqueue_multiple_desc_direct':
drivers/soc/fsl/dpio/qbman-portal.c:870:14: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
870 | addr_cena = (uint64_t)s->addr_cena;

The variable is not used anywhere, so removing the assignment seems
to be the correct workaround. After spotting what seemed to be
some confusion about address spaces, I ran the file through sparse,
which showed more warnings:

drivers/soc/fsl/dpio/qbman-portal.c:756:42: warning: incorrect type in argument 1 (different address spaces)
drivers/soc/fsl/dpio/qbman-portal.c:756:42: expected void const volatile [noderef] <asn:2> *addr
drivers/soc/fsl/dpio/qbman-portal.c:756:42: got unsigned int [usertype] *[assigned] p
drivers/soc/fsl/dpio/qbman-portal.c:902:42: warning: incorrect type in argument 1 (different address spaces)
drivers/soc/fsl/dpio/qbman-portal.c:902:42: expected void const volatile [noderef] <asn:2> *addr
drivers/soc/fsl/dpio/qbman-portal.c:902:42: got unsigned int [usertype] *[assigned] p

Here, the problem is passing a token from memremap() into __raw_readl(),
which is only defined to work on MMIO addresses but not RAM. Turning
this into a simple pointer dereference avoids this warning as well.

Link: https://lore.kernel.org/r/20200408185904.460563-1-arnd@arndb.de
Fixes: 3b2abda7d28c ("soc: fsl: dpio: Replace QMAN array mode with ring mode enqueue")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>

soc: fsl: dpio: fix dereference of pointer p before null check

Pointer p is currently being dereferenced before it is null
checked on a memory allocation failure check. Fix this by
checking if p is null before dereferencing it.

Addresses-Coverity: ("Dereference before null check")
Fixes: 3b2abda7d28c ("soc: fsl: dpio: Replace QMAN array mode with ring mode enqueue")
Signed-off-by: Colin Ian King <colin.king@canonical.com>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: Replace QMAN array mode with ring mode enqueue

This change of algorithm will enable faster bulk enqueue.
This will greatly benefit XDP bulk enqueue.

Signed-off-by: Youri Querry <youri.querry_1@nxp.com>
Acked-by: Roy Pledge <roy.pledge@nxp.com>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: QMAN performance improvement with function pointer indirection

We are making the access decision in the initialization and
setting the function pointers accordingly.

Signed-off-by: Youri Querry <youri.querry_1@nxp.com>
Acked-by: Roy Pledge <roy.pledge@nxp.com>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: Adding QMAN multiple enqueue interface

Update of QMAN the interface to enqueue frame. We now support multiple
enqueue (qbman_swp_enqueue_multiple) and multiple enqueue with
a table of descriptor (qbman_swp_enqueue_multiple_desc).

Signed-off-by: Youri Querry <youri.querry_1@nxp.com>
Acked-by: Roy Pledge <roy.pledge@nxp.com>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: Add support for memory backed QBMan portals

NXP devices with QBMan version 5 and above can enable software
portals that are memory backed. This allows the portal to be
mapped as cacheable/sharable (same as all normal memory) so
that portals can freely migrate between cores and clusters
in the SoC. The driver will enable this mode by default when
appropriate HW support is detected.

Signed-off-by: Youri Querry <youri.querry_1@nxp.com>
Signed-off-by: Roy Pledge <roy.pledge@nxp.com>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: enable frame data cache stashing per software portal

Enable cache stashing on the frame data dequeued using this software
portal. Also, enable dropping a stash request transaction when the
target request queue is almost full.

Signed-off-by: Ioana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: fix memory leak of a struct qbman on error exit path

Currently the error check for a null reg leaks a struct qbman
that was allocated earlier. Fix this by kfree'ing p on the error exit
path.

Signed-off-by: Colin Ian King <colin.king@canonical.com>
Signed-off-by: Li Yang <leoyang.li@nxp.com>

soc: fsl: dpio: Add BP and FQ query APIs

Add FQ (Frame Queue) and BP (Buffer Pool) query APIs that
users of QBMan can invoke to see the status of the queues
and pools that they are using.

Signed-off-by: Roy Pledge <roy.pledge@nxp.com>
Signed-off-by: Ioana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: Ioana Ciornei <ioana.ciornei@nxp.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

staging: fsl-mc: Move DPIO from staging to drivers/soc/fsl

Move the NXP DPIO (Datapath I/O Driver) out of the
drivers/staging directory and into the drivers/soc/fsl directory.

The DPIO driver enables access to Queue and Buffer Manager (QBMAN)
hardware on NXP DPAA2 devices. This is a prerequisite to moving the
DPAA2 Ethernet driver out of staging.

Signed-off-by: Roy Pledge <roy.pledge@nxp.com>
Reviewed-by: Horia Geantă <horia.geanta@nxp.com>
Reviewed-by: Ioana Radulescu <ruxandra.radulescu@nxp.com>
Signed-off-by: Li Yang <leoyang.li@nxp.com>