wifi: iwlwifi: refactor RX tracing

When there's not going to be any data in the data event, we
don't need to add it at all (unlike the TX version, it has
no data at all.)

Also combine the tracing into a separate inline so we only
call iwl_rx_trace_len() once, which also simplifies things,
and lets us have a single place to later add other checks.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Miri Korenblit <miriam.rachel.korenblit@intel.com>
Link: https://msgid.link/20231207044813.13325a4848d2.Ic9e7d794fc4aebfe5ac5136b539ee62789f210f3@changeid
Signed-off-by: Johannes Berg <johannes.berg@intel.com>

wifi: iwlwifi: pcie: don't synchronize IRQs from IRQ

On older devices (before unified image!) we can end up calling
stop_device from an rfkill interrupt. However, in stop_device
we attempt to synchronize IRQs, which then of course deadlocks.

Avoid this by checking the context, if running from the IRQ
thread then don't synchronize. This wouldn't be correct on a
new device since RSS is supported, but older devices only have
a single interrupt/queue.

Fixes: 37fb29bd1f90 ("wifi: iwlwifi: pcie: synchronize IRQs before NAPI")
Reviewed-by: Miri Korenblit <miriam.rachel.korenblit@intel.com>
Reviewed-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Kalle Valo <kvalo@kernel.org>
Link: https://msgid.link/20231215111335.59aab00baed7.Iadfe154d6248e7f9dfd69522e5429dbbd72925d7@changeid

wifi: iwlwifi: pcie: avoid a NULL pointer dereference

It possible that while the rx rb is being handled, the transport has
been stopped and re-started. In this case the tx queue pointer is not
yet initialized, which will lead to a NULL pointer dereference.
Fix it.

Signed-off-by: Avraham Stern <avraham.stern@intel.com>
Signed-off-by: Miri Korenblit <miriam.rachel.korenblit@intel.com>
Link: https://msgid.link/20231207044813.cd0898cafd89.I0b84daae753ba9612092bf383f5c6f761446e964@changeid
Signed-off-by: Johannes Berg <johannes.berg@intel.com>

netdev: use napi_schedule bool instead of napi_schedule_prep/__napi_schedule

Replace if condition of napi_schedule_prep/__napi_schedule and use bool
from napi_schedule directly where possible.

Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Link: https://lore.kernel.org/r/20231009133754.9834-5-ansuelsmth@gmail.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

wifi: iwlwifi: pcie: enable TOP fatal error interrupt

Enable the TOP (HW part) fatal error interrupt and add a
print when it happens. Currently FW always adds also the
SW error interrupt, but for >= Bz we'll need to do PLDR
in case this is asserted, so leave a TODO item already.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Gregory Greenman <gregory.greenman@intel.com>
Link: https://lore.kernel.org/r/20230913145231.127d914a4d0d.I41ea409df63474554ef727c49382d0b5bf15939e@changeid
Signed-off-by: Johannes Berg <johannes.berg@intel.com>

wifi: iwlwifi: pcie: fix RB status reading

On newer hardware, a queue's RB status / write pointer
can be bigger than 4095 (0xFFF), so we cannot mask the
value by 0xFFF unconditionally. Since anyway that's
only necessary on older hardware, move the masking to
the helper function and apply it only for older HW.
This also moves the endian conversion in to handle it
more easily.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Gregory Greenman <gregory.greenman@intel.com>
Link: https://lore.kernel.org/r/20230830112059.7be2a3fff6f4.I94f11dee314a4f7c1941d2d223936b1fa8aa9ee4@changeid
Signed-off-by: Johannes Berg <johannes.berg@intel.com>

wifi: iwlwifi: remove 'def_rx_queue' struct member

This is only ever initialized to zero, use a new define
for the default RX queue instead.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Gregory Greenman <gregory.greenman@intel.com>
Link: https://lore.kernel.org/r/20230816104355.e0c6fa57c162.I907bbb428cf99725f06a348c8dbce5d3dd877136@changeid
Signed-off-by: Johannes Berg <johannes.berg@intel.com>

wifi: iwlwifi: pcie: refactor RB status size calculation

We have three places doing this check, and even in
slightly different ways (with/without an intermediate).
Refactor that to a new small inline function.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Gregory Greenman <gregory.greenman@intel.com>
Link: https://lore.kernel.org/r/20230620125813.f3e87ddd5bce.Ifefba753043b68c394590a35bc6914a0f6497fd3@changeid
Signed-off-by: Johannes Berg <johannes.berg@intel.com>

wifi: iwlwifi: pcie: fix NULL pointer dereference in iwl_pcie_irq_rx_msix_handler()

rxq can be NULL only when trans_pcie->rxq is NULL and entry->entry
is zero. For the case when entry->entry is not equal to 0, rxq
won't be NULL even if trans_pcie->rxq is NULL. Modify checker to
check for trans_pcie->rxq.

Fixes: abc599efa67b ("iwlwifi: pcie: don't crash when rx queues aren't allocated in interrupt")
Signed-off-by: Anjaneyulu <pagadala.yesu.anjaneyulu@intel.com>
Signed-off-by: Gregory Greenman <gregory.greenman@intel.com>
Link: https://lore.kernel.org/r/20230614123446.5a5eb3889a4a.I375a1d58f16b48cd2044e7b7caddae512d7c86fd@changeid
Signed-off-by: Johannes Berg <johannes.berg@intel.com>

wifi: iwlwifi: unify checks for HW error values

The hardware, depending on which part fails or times out,
returns 0xA5A5A5A. or 0x5A5A5A5. with the lowest 4 bits
encoding some further reason/status. However, mostly we
don't really need to care about the exact reasons, so
unify the checks for this to avoid hardcoding those magic
values all over the driver.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Gregory Greenman <gregory.greenman@intel.com>
Link: https://lore.kernel.org/r/20230612184434.3e2959741a38.I1c297a53787b87e4e2b8f296c041921338573f4d@changeid
Signed-off-by: Johannes Berg <johannes.berg@intel.com>

wifi: iwlwifi: call napi_synchronize() before freeing rx/tx queues

When rx/tx queues are being freed, on a different CPU there could be
still rx flow running. Call napi_synchronize() to prevent such a race.

Signed-off-by: Gregory Greenman <gregory.greenman@intel.com>
Co-developed-by: Benjamin Berg <benjamin.berg@intel.com>
Signed-off-by: Benjamin Berg <benjamin.berg@intel.com>
Link: https://lore.kernel.org/r/20230416154301.5171ee44dcc1.Iff18718540da412e084e7d8266447d40730600ed@changeid
Signed-off-by: Johannes Berg <johannes.berg@intel.com>

net: drop the weight argument from netif_napi_add

We tell driver developers to always pass NAPI_POLL_WEIGHT
as the weight to netif_napi_add(). This may be confusing
to newcomers, drop the weight argument, those who really
need to tweak the weight can use netif_napi_add_weight().

Acked-by: Marc Kleine-Budde <mkl@pengutronix.de> # for CAN
Link: https://lore.kernel.org/r/20220927132753.750069-1-kuba@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

iwlwifi: Fix syntax errors in comments

Delete the redundant word 'the'.

Signed-off-by: Xiang wangx <wangxiang@cdjrlc.com>
Link: https://lore.kernel.org/r/20211216085756.11053-1-wangxiang@cdjrlc.com
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: don't dump_stack() when we get an unexpected interrupt

It is yet unclear if the WARNING really points to a real problem,
but for sure the stack dump doesn't help fixing it.
Just use a regular error print instead.

Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20220205112029.a79e733a12f7.I8189344294222be0589fa43cc70fdf38e3057045@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: Adapt rx queue write pointer for Bz family

Adapt rx queue write pointer for Bz family.
The register has moved to the same one as Tx.

Signed-off-by: Matti Gottlieb <matti.gottlieb@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20220204122220.57bd62a4365c.I873aa9b3d13abf5633a4963c55c3a09a833254f0@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: adjust to Bz completion descriptor

The Bz devices got a new completion descriptor again since
we only ever really used 4 out of 32 bytes anyway. Adjust
the code to deal with that. Note that the intention was to
reduce the size, but the hardware was implemented wrongly.

While at it, do some cleanups and remove the union to simplify
the code, clean up iwl_pcie_free_bd_size() to no longer need
an argument and add iwl_pcie_used_bd_size() with the logic to
selct completion descriptor size.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20220204122220.bef461a04110.I90c8885550fa54eb0aaa4363d322f50e301175a6@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: make sure iwl_rx_packet_payload_len() will not underflow

If the device is malfunctioning and reports too short rx descriptor
length, iwl_rx_packet_payload_len() will underflow, eventually resulting
in accessing memory out of bounds and other bad things. Prevent this.

Signed-off-by: Andrei Otcheretianski <andrei.otcheretianski@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20220130115024.ea00b52c6f25.I8b79b14f1af8b6f2f579f97b397b9e005fe446b1@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: yoyo: add IMR DRAM dump support

Support debug collection of the platform IMR memory region,
where data is copied by FW during d3 state

Signed-off-by: Mukesh Sisodiya <mukesh.sisodiya@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20220129105618.715f04ecc635.Ib89a6caa06c1324c1c0dd3f9f4cf7407f2857155@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: avoid void pointer arithmetic

Avoid void pointer arithmetic since it's technically
undefined and causes warnings in some places that use
our code.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20220128153014.e349104ecd94.Iadc937f475158b9437becdfefb361a97e7eaa934@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: prefer WIDE_ID() over iwl_cmd_id()

The order of arguments for iwl_cmd_id() is confusing, and the
version is always 0 and thus a useless argument. Prefer the
WIDE_ID() macro (which needs to be a macro due to use in switch
cases etc.) over the iwl_cmd_id() function.

Obviously done with spatch:

@@
expression G, C;
@@
-iwl_cmd_id(C, G, 0)
+WIDE_ID(G, C)

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20220128153014.cc4f9d1a2e9b.Ieb023cd773ea22e819d1ef1c37ae857ecc1a839d@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: make sure prph_info is set when treating wakeup IRQ

In some rare cases when the HW is in a bad state, we may get this
interrupt when prph_info is not set yet. Then we will try to
dereference it to check the sleep_notif element, which will cause an
oops.

Fix that by ignoring the interrupt if prph_info is not set yet.

Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20211219132536.0537aa562313.I183bb336345b9b3da196ba9e596a6f189fbcbd09@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: update sw error interrupt for BZ family

The cause for sw error in BZ device family was changed

Signed-off-by: Mike Golant <michael.golant@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20211024165252.f674cd409b8e.I519f554d0a22d4711077785ec2bd7c564997241f@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: dump error on FW reset handshake failures

If the firmware crashes while we're waiting for the reset
handshake then it cannot possibly make progress anymore,
and we will just time out the wait. That's pointless, so
just stop waiting at that point.

Additionally, if it never acknowledges the reset handshake,
something went wrong.

Dump an error in both of these cases, but we need to do it
synchronously here since the device will be turned off.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210802170640.8b6a33544b4b.I55f97f70f8efa64db064a9207177a094c60ac8f1@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: prepare for synchronous error dumps

In some cases it may be necessary to synchronously create
a firmware error report, add the necessary infrastructure
for this.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210802170640.481b6642f0fc.I7c9c958408a285e3d19aceed2a5a3341cfc08382@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: free RBs during configure

When switching op-modes, or more generally when reconfiguring,
we might switch the RB size. In _iwl_pcie_rx_init() we have a
comment saying we must free all RBs since we might switch the
size, but this is actually too late: the switch has been done
and we'll free the buffers with the wrong size.

Fix this by always freeing the buffers, if any, at the start
of configure, instead of only after the size may have changed.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210802170640.42d7c93279c4.I07f74e65aab0e3d965a81206fcb289dc92d74878@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: remove TR/CR tail allocations

The TR/CR tail data are meant to be per-queue-arrays, however,
we allocate them completely wrong (we have a separate allocation
per queue).

Looking at this more closely, it turns out that the hardware
never uses these - we have a separate free list per RX queue
and maintain a write pointer for that in a register, and the
RX itself is indicated in the RB status (rb_stts) DMA region.

Despite nothing using the tail pointers, the hardware will
unconditionally access them to write updates, even when we aren't
using CRs/TRs.

Give it dummy values that we never use/update so it can do that
without causing trouble.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210617110647.5f5764e04c46.I4d5de1929be048085767f1234a1e07b517ab6a2d@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: add ISR debug info for msix debug

The debug prints help in case we get timeout on waiting for
hw.

Signed-off-by: Mordechay Goodstein <mordechay.goodstein@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210411124417.306e2e56d3e8.I72e2977abbb1fddf23b8476bedf6a183fe969ff5@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: merge napi_poll_msix functions

The only difference between iwl_pcie_napi_poll_msix_shared() and
iwl_pcie_napi_poll_msix() is when we have a shared queue and nothing
in the rx queue. This case doesn't affect CPU performance, so we can
merge the two functions.

Signed-off-by: Mordechay Goodstein <mordechay.goodstein@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210411124417.9d1b61ef53a5.I60b33d5379cf7c12f1de30fc3fd4cefc38220141@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: clear only FH bits handle in the interrupt

For simplicity we assume that msix has 2 IRQ lines one used for rx data
called msix_non_share, and another used for one bit flags messages
(alive, hw error, sw error, rx data flag) called msix_share.

Every time the FW has data to send it puts it on the RX queue and HW
turns on the flags in msix_share (inta_fw) indicating about rx data,
and HW sends an interrupt a bit later to the msix_non_share _unless_
the msix_shared RX data bit was cleared.

Currently in the code every time we get an msix_shared we clear all bits
including rx data queue bits.

So we can have a race

----------------------------------------------------
DRIVER | HW | FW
----------------------------------------------------
- send host cmd to FW | |
| | - handle message
| | and put a response
| | on the RX queue
| - RX flag on |
| | - send alive msix
| - alive flag on |
| - interrupt |
| msix_share driver |
- handle msix_shared | |
and clear all flags | |
bits | |
| - don't send an |
| interrupt on |
| msix_non_shared |
| (driver cleared) |
- driver timeout on | |
waiting for host cmd | |
respond | |
| |
----------------------------------------------------

The change is to clear only the msi_shared flags that are handled in
the msix_shared flow, which will cause the hardware to send an interrupt
on the msix_non_share line as well, when it has data.

Signed-off-by: Mordechay Goodstein <mordechay.goodstein@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210330162204.a1cdda2fa270.I02a82312679f4541f30bb8db8747a797dbb70ee7@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())

We can't call netif_napi_add() with rxq-lock held, as there is a potential
for deadlock as spotted by lockdep (see below). rxq->lock is not
protecting anything over the netif_napi_add() codepath anyway, so let's
drop it just before calling into NAPI.

========================================================
WARNING: possible irq lock inversion dependency detected
5.12.0-rc1-00002-gbada49429032 #5 Not tainted
--------------------------------------------------------
irq/136-iwlwifi/565 just changed the state of lock:
ffff89f28433b0b0 (&rxq->lock){+.-.}-{2:2}, at: iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
but this lock took another, SOFTIRQ-unsafe lock in the past:
(napi_hash_lock){+.+.}-{2:2}

and interrupts could create inverse lock ordering between them.

other info that might help us debug this:
Possible interrupt unsafe locking scenario:

CPU0 CPU1
---- ----
lock(napi_hash_lock);
local_irq_disable();
lock(&rxq->lock);
lock(napi_hash_lock);
<Interrupt>
lock(&rxq->lock);

*** DEADLOCK ***

1 lock held by irq/136-iwlwifi/565:
#0: ffff89f2b1440170 (sync_cmd_lockdep_map){+.+.}-{0:0}, at: iwl_pcie_irq_handler+0x5/0xb30

the shortest dependencies between 2nd lock and 1st lock:
-> (napi_hash_lock){+.+.}-{2:2} {
HARDIRQ-ON-W at:
lock_acquire+0x277/0x3d0
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
e1000_probe+0x2fe/0xee0 [e1000e]
local_pci_probe+0x42/0x90
pci_device_probe+0x10b/0x1c0
really_probe+0xef/0x4b0
driver_probe_device+0xde/0x150
device_driver_attach+0x4f/0x60
__driver_attach+0x9c/0x140
bus_for_each_dev+0x79/0xc0
bus_add_driver+0x18d/0x220
driver_register+0x5b/0xf0
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
SOFTIRQ-ON-W at:
lock_acquire+0x277/0x3d0
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
e1000_probe+0x2fe/0xee0 [e1000e]
local_pci_probe+0x42/0x90
pci_device_probe+0x10b/0x1c0
really_probe+0xef/0x4b0
driver_probe_device+0xde/0x150
device_driver_attach+0x4f/0x60
__driver_attach+0x9c/0x140
bus_for_each_dev+0x79/0xc0
bus_add_driver+0x18d/0x220
driver_register+0x5b/0xf0
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
INITIAL USE at:
lock_acquire+0x277/0x3d0
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
e1000_probe+0x2fe/0xee0 [e1000e]
local_pci_probe+0x42/0x90
pci_device_probe+0x10b/0x1c0
really_probe+0xef/0x4b0
driver_probe_device+0xde/0x150
device_driver_attach+0x4f/0x60
__driver_attach+0x9c/0x140
bus_for_each_dev+0x79/0xc0
bus_add_driver+0x18d/0x220
driver_register+0x5b/0xf0
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
}
... key at: [<ffffffffae84ef38>] napi_hash_lock+0x18/0x40
... acquired at:
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
_iwl_pcie_rx_init+0x1f4/0x710 [iwlwifi]
iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
_iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
iwl_opmode_register+0x71/0xe0 [iwlwifi]
iwl_mvm_init+0x34/0x1000 [iwlmvm]
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae

[ ... lockdep output trimmed .... ]

Fixes: 25edc8f259c7106 ("iwlwifi: pcie: properly implement NAPI")
Signed-off-by: Jiri Kosina <jkosina@suse.cz>
Acked-by: Luca Coelho <luciano.coelho@intel.com>
Signed-off-by: Kalle Valo <kvalo@codeaurora.org>
Link: https://lore.kernel.org/r/nycvar.YFH.7.76.2103021134060.12405@cbobk.fhfr.pm

iwlwifi: pcie: don't crash when rx queues aren't allocated in interrupt

WARNING is better than crashing. Since this happened to me,
be on the safe side.

Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210210142629.d4651427fcda.I1bcecb73676d039e2521309c07fc6b6314a90546@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: remove flags argument for nic_access

Since we no longer save interrupts, we no longer need the flags
argument here, remove it throughout.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210210142629.8de8fe6f9fff.If040b056d0e8c771c65ac5c29230f939354a142b@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: NULLify pointers after free

Remember that those pointers have been freed by setting them
to NULL. Otherwise, we'd keep rxq pointing to random memory
which would prevent us from trying to re-allocate the Rx
resources if we call rx_alloc again.

Also, propagate the allocation failure to the caller of
iwl_pcie_nic_init so that we won't go further in the
start flow.

Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210210135352.996b400d2f1c.I630379c504644700322f57b259383ae0af8d1975@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: Disable softirqs during Rx queue init

When Rx queues are configured during module init, NAPI is enabled
while the Rx queue lock is held. However, since softirqs are not
disabled, it is possible that and IRQ would fire and call
iwl_pcie_rx_handle() which would also try to acquire the Rx lock.

Prevent this by disabling softirqs during Rx queue configuration,
as part of module init flow.

Signed-off-by: Ilan Peer <ilan.peer@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210205110447.d206ac428823.Ia19339efb09f9d80143f0d0e398a158180754cfa@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: tx: move handing sync/async host command to trans

Handling host commands in a sync way is not directly related to PCIe
transport, and can serve as common logic for any transport, so move
it to trans layer.

Signed-off-by: Mordechay Goodstein <mordechay.goodstein@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210117164916.fde99af4e0f7.I4cab95919eb35cc5bfb26d32dcf5e15419d0e0ef@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: properly implement NAPI

Instead of pretending to have NAPI and then relying entirely on
interrupts anyway, properly implement NAPI and schedule the poll
when we get an interrupt, re-enabling the interrupt only after
the poll completed.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20210117130510.a5951ac4fc06.I9c84a147288fcfb1b019572c6758f2d92949f5d7@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: support firmware reset handshake

There are some races in the hardware that can possibly lead to
a bus lockup later during a restart when we manage to kill the
firmware at a bad time (while it's accessing the bus).

To work around this, add support for a new handshake between
firmware and driver to ensure that the firmware is in a well-
known state before we kill it.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20201209231352.7756fcc9865c.I13de65e0ffcb4186dd4c1a465f66df2e98c9a947@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: clean up some rx code

We don't need the sequence/index/cmd_index unless we're doing
reclaim, they're not even valid in the other cases. Move the
variables and their assignments into the right if statement
and combine the two if statements into a single one as well.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20201209231352.6207fdcc91a9.Ia71e766ead7560262f4bc6ad3da6f1117c498cd6@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: use SPDX tags

Use SPDX tags instead of the long copyright notices. Also cleanup
some duplicate copyright notices and combine the years where possible.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20201210000603.481bcb512a6f.I8146abe5a637079e7336209f23cb26af98b12b31@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: remove MSIX_HW_INT_CAUSES_REG_IML handling

This is actually wrong, the bit used here by the image loader
is BIT(1), not BIT(2). The latter will be reused by the new
reset flow soon.

However, as we never had any complaints about not printing
the IML status or not handling the IML error interrupt (and
I suspect the code handling it was incorrectly anyway) just
remove the code for it.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20201209231352.9a323f4a3493.Ic7aee4dbbf4be42287c338c2fa1b111473724116@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: validate RX descriptor length

Validate the maximum RX descriptor length against the size
of the buffers we gave the device - if it doesn't fit then
the hardware messed up.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20201209231352.6378fb435cc0.Ib07485f3dc5999c74b03f21e7a808c50a05e353c@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: move all bus-independent TX functions to common code

After moving out all Tx fields not related to pcie-bus
it's time to move the code to a common place.

We also rename all pcie functions name to txq.

Signed-off-by: Mordechay Goodstein <mordechay.goodstein@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20200930161256.3947a5276003.I3fe1bec2b25a965a49532df288f47b8b59eb1500@changeid
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

mm, treewide: rename kzfree() to kfree_sensitive()

As said by Linus:

A symmetric naming is only helpful if it implies symmetries in use.
Otherwise it's actively misleading.

In "kzalloc()", the z is meaningful and an important part of what the
caller wants.

In "kzfree()", the z is actively detrimental, because maybe in the
future we really _might_ want to use that "memfill(0xdeadbeef)" or
something. The "zero" part of the interface isn't even _relevant_.

The main reason that kzfree() exists is to clear sensitive information
that should not be leaked to other future users of the same memory
objects.

Rename kzfree() to kfree_sensitive() to follow the example of the recently
added kvfree_sensitive() and make the intention of the API more explicit.
In addition, memzero_explicit() is used to clear the memory to make sure
that it won't get optimized away by the compiler.

The renaming is done by using the command sequence:

git grep -w --name-only kzfree |\
xargs sed -i 's/kzfree/kfree_sensitive/'

followed by some editing of the kfree_sensitive() kerneldoc and adding
a kzfree backward compatibility macro in slab.h.

[akpm@linux-foundation.org: fs/crypto/inline_crypt.c needs linux/slab.h]
[akpm@linux-foundation.org: fix fs/crypto/inline_crypt.c some more]

Suggested-by: Joe Perches <joe@perches.com>
Signed-off-by: Waiman Long <longman@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: David Howells <dhowells@redhat.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jarkko Sakkinen <jarkko.sakkinen@linux.intel.com>
Cc: James Morris <jmorris@namei.org>
Cc: "Serge E. Hallyn" <serge@hallyn.com>
Cc: Joe Perches <joe@perches.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Dan Carpenter <dan.carpenter@oracle.com>
Cc: "Jason A . Donenfeld" <Jason@zx2c4.com>
Link: http://lkml.kernel.org/r/20200616154311.12314-3-longman@redhat.com
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

iwlwifi: move txq-specific from trans_pcie to common trans

We don't want to have txq code in the PCIe transport code, so move all
the relevant elements to a new iwl_txq structure and store it in
iwl_trans.

spatch

@ replace_pcie @
struct iwl_trans_pcie *trans_pcie;
@@

(
-trans_pcie->queue_stopped
+trans->txqs.queue_stopped
|
-trans_pcie->queue_used
+trans->txqs.queue_used
|
-trans_pcie->txq
+trans->txqs.txq
|
-trans_pcie->txq
+trans->txqs.txq
|
-trans_pcie->cmd_queue
+trans->txqs.cmd.q_id
|
-trans_pcie->cmd_fifo
+trans->txqs.cmd.fifo
|
-trans_pcie->cmd_q_wdg_timeout
+trans->txqs.cmd.wdg_timeout
)

// clean all new unused variables
@ depends on replace_pcie @
type T;
identifier i;
expression E;
@@
- T i = E;
... when != i

Signed-off-by: Mordechay Goodstein <mordechay.goodstein@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20200529092401.a428d3c9d66f.Ie04ae55f33954636a39c98e7ae1e739c0507435b@changeid

iwlwifi: pcie: skip fragmented receive buffers

We don't really expect fragmented RBs, and don't seem to be seeing
them in practice since that would've caused a crash. Nevertheless,
we should be expecting the hardware to send them.

Parse the flag indicating a fragmented buffer, but then discard it
and any fragments thereof, at least for now. We need to do more
work in the higher layers to properly deal with this, since we may
not get "normal" firmware notifications that are fragmented, only
RX, and then we need to put it back together and add the necessary
API to report a chain of things to the higher layers, this doesn't
fit into the struct iwl_rx_cmd_buffer today.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Link: https://lore.kernel.org/r/iwlwifi.20200425130140.e78a59f70b1d.Ica656a98a4e4220d73edc97600edd680cbc97241@changeid

iwlwifi: move integrated, extra_phy and soc_latency to trans_cfg

These values are selected based on the PCI device ID, so the decision
to use them can be made early. By moving them to the trans_cfg, we
avoid duplicating the large cfg structs for small pieces of
data (sometimes a single boolean). This will also allow us to make
more decisions based on, for instance, the SoC type in used.

The trans_cfg concept changes a bit, because previously it was used
only to boot the device before reading further characteristics and now
it also contains more data that is associated with the device ID.

Change-Id: Ib71b07ea9e322eb74571dc5e8aa58f17eece5c9c
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

net: Fix packet reordering caused by GRO and listified RX cooperation

Commit 323ebb61e32b ("net: use listified RX for handling GRO_NORMAL
skbs") introduces batching of GRO_NORMAL packets in napi_frags_finish,
and commit 6570bc79c0df ("net: core: use listified Rx for GRO_NORMAL in
napi_gro_receive()") adds the same to napi_skb_finish. However,
dev_gro_receive (that is called just before napi_{frags,skb}_finish) can
also pass skbs to the networking stack: e.g., when the GRO session is
flushed, napi_gro_complete is called, which passes pp directly to
netif_receive_skb_internal, skipping napi->rx_list. It means that the
packet stored in pp will be handled by the stack earlier than the
packets that arrived before, but are still waiting in napi->rx_list. It
leads to TCP reorderings that can be observed in the TCPOFOQueue counter
in netstat.

This commit fixes the reordering issue by making napi_gro_complete also
use napi->rx_list, so that all packets going through GRO will keep their
order. In order to keep napi_gro_flush working properly, gro_normal_list
calls are moved after the flush to clear napi->rx_list.

iwlwifi calls napi_gro_flush directly and does the same thing that is
done by gro_normal_list, so the same change is applied there:
napi_gro_flush is moved to be before the flush of napi->rx_list.

A few other drivers also use napi_gro_flush (brocade/bna/bnad.c,
cortina/gemini.c, hisilicon/hns3/hns3_enet.c). The first two also use
napi_complete_done afterwards, which performs the gro_normal_list flush,
so they are fine. The latter calls napi_gro_receive right after
napi_gro_flush, so it can end up with non-empty napi->rx_list anyway.

Fixes: 323ebb61e32b ("net: use listified RX for handling GRO_NORMAL skbs")
Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Cc: Alexander Lobakin <alobakin@dlink.ru>
Cc: Edward Cree <ecree@solarflare.com>
Acked-by: Alexander Lobakin <alobakin@dlink.ru>
Acked-by: Saeed Mahameed <saeedm@mellanox.com>
Acked-by: Edward Cree <ecree@solarflare.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

iwlwifi: remove CSR registers abstraction

We needed this abstraction for some CSR registers for
IWL_DEVICE_22560, but that has been removed, so we don't need the
abstraction anymore. Remove it.

Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: use partial pages if applicable

If we have only 2k RBs like on the latest (AX210) hardware, then
even on x86 where PAGE_SIZE is 4k we currently waste half of the
memory.

If this is the case, return partial pages from the allocator and
track the offset in each RBD (to be able to find the data in them
and remap them later.)

This might also address other platforms with larger PAGE_SIZE by
putting more RBs into a single large page.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: map only used part of RX buffers

We don't need to map *everything* of the RX buffers, we won't use
that much, map only the part we're going to use. This save some
IOMMU space (if applicable and it can deal with that) and also
prepares a bit for mapping partial pages for 2K buffers later.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: allocate more receive buffers for HE devices

For HE-capable devices, we need to allocate more receive buffers as
there could be 256 frames aggregated into a single A-MPDU, and then
they might contain A-MSDUs as well. Until 22000 family, the devices
are able to put multiple frames into a single RB and the default RB
size is 4k, but starting from AX210 family this is no longer true.
On the other hand, those newer devices only use 2k receive buffers
(by default).

Modify the code and configuration to allocate an appropriate number
of RBs depending on the device capabilities:

* 4096 for AX210 HE devices, which use 2k buffers by default,
* 2048 for 22000 family devices which use 4k buffers by default,
* 512 for existing 9000 family devices, which doesn't really
change anything since that's the default before this patch,
* 512 also for AX210/22000 family devices that don't do HE.

Theoretically, for devices lower than AX210, we wouldn't have to
allocate that many RBs if the RB size was manually increased, but
to support that the code got more complex, and it didn't really
seem necessary as that's a use case for monitor mode only, where
hopefully the wasted memory isn't really much of a concern.

Note that AX210 devices actually support bigger than 12-bit VID,
which is required here as we want to allocate 4096 buffers plus
some for quick recycling, so adjust the code for that as well.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

net: wireless: intel: iwlwifi: fix GRO_NORMAL packet stalling

Commit 6570bc79c0df ("net: core: use listified Rx for GRO_NORMAL in
napi_gro_receive()") has applied batched GRO_NORMAL packets processing
to all napi_gro_receive() users, including mac80211-based drivers.

However, this change has led to a regression in iwlwifi driver [1][2] as
it is required for NAPI users to call napi_complete_done() or
napi_complete() and the end of every polling iteration, whilst iwlwifi
doesn't use NAPI scheduling at all and just calls napi_gro_flush().
In that particular case, packets which have not been already flushed
from napi->rx_list stall in it until at least next Rx cycle.

Fix this by adding a manual flushing of the list to iwlwifi driver right
before napi_gro_flush() call to mimic napi_complete() logics.

I prefer to open-code gro_normal_list() rather than exporting it for 2
reasons:
* to prevent from using it and napi_gro_flush() in any new drivers,
as it is the *really* bad way to use NAPI that should be avoided;
* to keep gro_normal_list() static and don't lose any CC optimizations.

I also don't add the "Fixes:" tag as the mentioned commit was only a
trigger that only exposed an improper usage of NAPI in this particular
driver.

[1] https://lore.kernel.org/netdev/PSXP216MB04388962C411CD0B17A86F47804A0@PSXP216MB0438.KORP216.PROD.OUTLOOK.COM
[2] https://bugzilla.kernel.org/show_bug.cgi?id=205647

Signed-off-by: Alexander Lobakin <alobakin@dlink.ru>
Acked-by: Luca Coelho <luciano.coelho@intel.com>
Reported-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Tested-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Reviewed-by: Edward Cree <ecree@solarflare.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

iwlwifi: pcie: rx: use rxq queue_size instead of constant

This is a little less efficient now as it's known to be a
multiqueue device in this function, but a future patch will
have to use a variable here anyway, so use rxq->queue_size
now instead to make it clearer.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: make some RX functions static

These aren't used outside the rx.c file, so make them static.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: remove IWL_DEVICE_22560/IWL_DEVICE_FAMILY_22560

This is dead code, nothing uses the IWL_DEVICE_22560 macro and
thus nothing every uses IWL_DEVICE_FAMILY_22560. Remove it all.

While at it, remove some code and definitions used only in this
case, and clean up some comments/names that still refer to it.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Signed-off-by: Kalle Valo <kvalo@codeaurora.org>

iwlwifi: add support for suspend-resume flow for new device generation

The new device generation has a slightly different suspend resume flow
Currently, the way the driver instruct the device to move to D3 is by
sending D3_CONFIG_CMD.
Instead of using the host command the indication is by writing to the
doorbell interrupt.
The FW will respond with interrupt to indicate transition completion.

Signed-off-by: Haim Dreyfuss <haim.dreyfuss@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: add a pointer to the trans_cfg directly in trans

Add a pointer to the iwl_trans structure and point it to the trans
part of the cfg. This is the first step in disassociating the trans
configuration from the rest of the configuration.

Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: separate elements from cfg that are needed by trans_alloc

In order to be able to select the cfg depending on the HW revision or
on the RF ID, we need to set up the trans before selecting the cfg.
To do so, move the elements from cfg that are needed by
iwl_trans_alloc() to a separate struct at the top of the cfg, so it
can be used by other cfg types as well, before selecting the rest of
the configuration.

Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: don't service an interrupt that was masked

Sometimes the register status can include interrupts that
were masked. We can, for example, get the RF-Kill bit set
in the interrupt status register although this interrupt
was masked. Then if we get the ALIVE interrupt (for example)
that was not masked, we need to *not* service the RF-Kill
interrupt.
Fix this in the MSI-X interrupt handler.

Cc: stable@vger.kernel.org
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: fix RF-Kill interrupt while FW load for gen2 devices

Newest devices have a new firmware load mechanism. This
mechanism is called the context info. It means that the
driver doesn't need to load the sections of the firmware.
The driver rather prepares a place in DRAM, with pointers
to the relevant sections of the firmware, and the firmware
loads itself.
At the end of the process, the firmware sends the ALIVE
interrupt. This is different from the previous scheme in
which the driver expected the FH_TX interrupt after each
section being transferred over the DMA.

In order to support this new flow, we enabled all the
interrupts. This broke the assumption that we have in the
code that the RF-Kill interrupt can't interrupt the firmware
load flow.

Change the context info flow to enable only the ALIVE
interrupt, and re-enable all the other interrupts only
after the firmware is alive. Then, we won't see the RF-Kill
interrupt until then. Getting the RF-Kill interrupt while
loading the firmware made us kill the firmware while it is
loading and we ended up dumping garbage instead of the firmware
state.

Re-enable the ALIVE | RX interrupts from the ISR when we
get the ALIVE interrupt to be able to get the RX interrupt
that comes immediately afterwards for the ALIVE
notification. This is needed for non MSI-X only.

Cc: stable@vger.kernel.org
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: fix ALIVE interrupt handling for gen2 devices w/o MSI-X

We added code to restock the buffer upon ALIVE interrupt
when MSI-X is disabled. This was added as part of the context
info code. This code was added only if the ISR debug level
is set which is very unlikely to be related.
Move this code to run even when the ISR debug level is not
set.

Note that gen2 devices work with MSI-X in most cases so that
this path is seldom used.

Cc: stable@vger.kernel.org
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: dbg: move trans debug fields to a separate struct

Unite iwl_trans debug related fields under iwl_trans_debug struct to
increase readability and keep iwl_trans clean.

Signed-off-by: Shahar S Matityahu <shahar.s.matityahu@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: remove stray character in iwl_pcie_rx_alloc_page()

We have a solitary and inconspicous ` in the middle of a comment in
this function, which should not be there. Remove it.

Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: don't crash on invalid RX interrupt

If for some reason the device gives us an RX interrupt before we're
ready for it, perhaps during device power-on with misconfigured IRQ
causes mapping or so, we can crash trying to access the queues.

Prevent that by checking that we actually have RXQs and that they
were properly allocated.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: dbg_ini: support HW error trigger

Differentiate between SW and HW error interrupts and support ini HW
error trigger.

Signed-off-by: Shahar S Matityahu <shahar.s.matityahu@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: switch to correct RBD/CD layout for 22560

The layout of the RBD (receive buffer descriptor) isn't quite right,
the hardware ended up being implemented differently. Switch to the
correct RBD layout. While at it, remove the now useless extra defines.

Also, switch the CD (completion descriptor) to the right format, which
is basically just a code cleanup because the only field we really used
(rbid) is still in the same place. We may need fragmentation later if
we ever want to use it.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: allocate rb_stts's for all queues in one place

AX210 devices assume that the (DRAM) addresses of the rb_stts's for
the different queues are continuous.
So allocate the rb_stts's for all the Rx queues in one place.

Signed-off-by: Shaul Triebitz <shaul.triebitz@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: align to new periphery address space for AX210 family

In AX210 family, UMAC periphery address space moved from
0xA00000 to 0xD00000.

Signed-off-by: Shaul Triebitz <shaul.triebitz@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: introduce device family AX210

Add new device family AX210.
Make the needed changes for this family.

Signed-off-by: Shaul Triebitz <shaul.triebitz@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: add TPT oriented prints

Currently there is no way to debug RX/TX paths using prints
without harming tpt. Add prints to debug RX allocation path.
We can still get 1.9 gbps with those on.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: fix emergency path

Allocator swaps the pending requests with 0 when it starts
working. This means that relying on it n RX path to decide if
to move to emergency is not always a good idea, since it may
be zero, but there are still a lot of unallocated RBs in the
system. Change allocator to decrement the pending requests on
real time. It is more expensive since it accesses the atomic
variable more times, but it gives the RX path a better idea
of the system's status.

Reported-by: Ilan Peer <ilan.peer@intel.com>
Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Fixes: 868a1e863f95 ("iwlwifi: pcie: avoid empty free RB queue")
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: align licensing to dual GPL/BSD

These files have a long history of code changes, but analysing
the remaining code leads to having only a few changes that are
not already owned by Intel, notably from
- Andy Lutomirski <luto@amacapital.net>
- Joonwoo Park <joonwpark81@gmail.com>
- Kirtika Ruchandani <kirtika@chromium.org>
- Rajat Jain <rajatja@google.com>
- Stanislaw Gruszka <sgruszka@redhat.com>
remaining in the code today.

Note that
- I myself was working for Intel and for any possibly code
that might be before my employment there give permission
- Wizery employees were working for Intel

More specifically, we identified the following commits that
(partially may) remain today:

25c03d8e8c13 Joonwoo Park <joonwpark81@gmail.com> ("iwlwifi: do not schedule tasklet when rcv unused irq")
f36d04abe684 Stanislaw Gruszka <sgruszka@redhat.com> ("iwlwifi: use dma_alloc_coherent")
387f3381f732 Stanislaw Gruszka <sgruszka@redhat.com> ("iwlwifi: fix dma mappings and skbs leak")
2624e96ce16b Stanislaw Gruszka <sgruszka@redhat.com> ("iwlwifi: fix possible data overwrite in hcmd callback")
bfe4b80e9f73 Stanislaw Gruszka <sgruszka@redhat.com> ("iwlwifi: always check if got h/w access before write")
d536c32b45d2 Andy Lutomirski <luto@amacapital.net> ("iwlwifi: pcie: log when waking the NIC for hcmd submission fails")
a6d24fad00d9 Rajat Jain <rajatja@google.com> ("iwlwifi: pcie: dump registers when HW becomes inaccessible")
fb12777ab59b Kirtika Ruchandani <kirtika@chromium.org> ("iwlwifi: Add more call-sites for pcie reg dumper")
3a73a30049f2 Stanislaw Gruszka <sgruszka@redhat.com> ("iwlwifi: cleanup/fix memory barriers")
aa5affbacb24 Stanislaw Gruszka <sgruszka@redhat.com> ("iwlwifi: dump stack when fail to gain access to the device")

Align the licenses with their permission to clean up and to
make it all identical.

CC: Joonwoo Park <joonwpark81@gmail.com>
CC: Stanislaw Gruszka <sgruszka@redhat.com>
CC: Andy Lutomirski <luto@amacapital.net>
CC: Rajat Jain <rajatja@google.com>
CC: Kirtika Ruchandani <kirtika@chromium.org>
Acked-by: Johannes Berg <johannes@sipsolutions.net>
Acked-by: Kirtika Ruchandani <kirtika@chromium.org>
Acked-by: Stanislaw Gruszka <sgruszka@redhat.com>
Acked-by: Joonwoo Park <joonwpark81@gmail.com>
Acked-by: Rajat Jain <rajatja@google.com>
Acked-by: Andy Lutomirski <luto@kernel.org>
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: add prints to track virtual ID

In case there are bugs in this area, this data can
help with debugging.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: remove unnecessary iwl_pcie_enable_rx_wake() function

This function was only used by 9000 A-step devices, which we don't
support anymore, so it can be removed.

Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

cross-tree: phase out dma_zalloc_coherent()

We already need to zero out memory for dma_alloc_coherent(), as such
using dma_zalloc_coherent() is superflous. Phase it out.

This change was generated with the following Coccinelle SmPL patch:

@ replace_dma_zalloc_coherent @
expression dev, size, data, handle, flags;
@@

-dma_zalloc_coherent(dev, size, handle, flags)
+dma_alloc_coherent(dev, size, handle, flags)

Suggested-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
[hch: re-ran the script on the latest tree]
Signed-off-by: Christoph Hellwig <hch@lst.de>

iwlwifi: pcie: avoid empty free RB queue

If all free RB queues are empty, the driver will never restock the
free RB queue. That's because the restocking happens in the Rx flow,
and if the free queue is empty there will be no Rx.

Although there's a background worker (a.k.a. allocator) allocating
memory for RBs so that the Rx handler can restock them, the worker may
run only after the free queue has become empty (and then it is too
late for restocking as explained above).

There is a solution for that called 'emergency': If the number of used
RB's reaches half the amount of all RB's, the Rx handler will not wait
for the allocator but immediately allocate memory for the used RB's
and restock the free queue.

But, since the used RB's is per queue, it may happen that the used
RB's are spread between the queues such that the emergency check will
fail for each of the queues
(and still run out of RBs, causing the above symptom).

To fix it, move to emergency mode if the sum of *all* used RBs (for
all Rx queues) reaches half the amount of all RB's

Signed-off-by: Shaul Triebitz <shaul.triebitz@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: remove all occurrences of the FSF address paragraph

The Free Software Foundation address is superfluous and causes
checkpatch to issue a warning when present. Remove all paragraphs
with FSF's address to prevent that.

Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: set interrupt coalescing also for gen2

We offloaded all the RX configuration of init to firmware. However,
the configuration of interrupt coalescing was left hanging - it wasn't
offloaded nor was it written by host.

This write to the CSR is allowed in gen2, so the host can do it.
Without it we have various issues with RX fullness.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: store the default rxq number

Store the default rxq number in a variable, so we won't need
to use the actual number in the code.

Signed-off-by: Golan Ben Ami <golan.ben.ami@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: make non-static hcmd and rx code

Allow other device generations to use the utilities that
are used to send and reclaim host commands and to allocate
rx, by making it non-static.

Signed-off-by: Golan Ben Ami <golan.ben.ami@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: drop packets with bad status in CD

We need to drop packets with errors (such as replay,
MIC, ICV, conversion, duplicate and so on).

Drop invalid packets, put the status bits in the metadata and
move the enum definition to the correct place (FW API header).

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: move some msix and rx functions to a common place

We would like to allow other utlities to init msix and rx.
Put their declarations in a place accessible to other utilities.

Signed-off-by: Golan Ben Ami <golan.ben.ami@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: move rxb retrieval to a helper function

This makes code less indented and more readable.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: unionize used bd and completion descriptor

This allows less "dummy" declarations and casting.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: support rx structures for 22560 devices

The rfh for 22560 devices has changed so it supports now
the same arch of using used and free lists, but different
structures to support the last.
Use the new structures, hw dependent, to manage the lists.

bd, the free list, uses the iwl_rx_transfer_desc,
in which the vid is stored in the structs' rbid
field, and the page address in the addr field.

used_bd, the used list, uses the iwl_rx_completion_desc
struct, in which the vid is stored in the structs' rbid
field.

rb_stts, the hw "write" pointer of rx is stored in a
__le16 array, in which each entry represents the write
pointer per queue.

Signed-off-by: Golan Ben Ami <golan.ben.ami@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: support 2k rx buffers

The smallest rb size supported today is 4k rx buffers.
22560 devices use 2k rxb's, so allow using 2k buffers.

Signed-off-by: Golan Ben Ami <golan.ben.ami@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: update registers changed for 22560 devices

In 22560 devices the firmware will do all the hw configurations,
but that's not ready yet.
Update the correct registers in the driver until the FW is ready
and does it by itself.

Signed-off-by: Golan Ben Ami <golan.ben.ami@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: update gen3 interrupts - sw error and image response

In 22560 devices the ROM sendis an interrupt to the host
once the IML reading is done.
Handle this interrupt, and indicate sw error in case the
value is fail.

Additionally, the cause for sw error in 22560 devices
have been changed, so update the cause list.

Signed-off-by: Golan Ben Ami <golan.ben.ami@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: allocate and free rx cr's and tr's tails

The hw now refers to two new blocks:
* rx tr tail - The Tail index on the free buffers queue TR,
which is update by the device after reading the free buffer
from the tr.
* rx cr tail - Updated by the driver when completing
processing a new completion descriptor in the cr.

Add these two new struct to the rxq, allocate and free them
when needed.

In addition, the register for rx write pointer had been changed
to HBUS_TARG_WRPTR. The way to differentiate tx from rx is the
queue number. TX range is 0-511, and RX's is 512-527.

Signed-off-by: Golan Ben Ami <golan.ben.ami@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: fix race in Rx buffer allocator

Make sure the rx_allocator worker is canceled before running the
rx_init routine. rx_init frees and re-allocates all rxb's pages. The
rx_allocator worker also allocates pages for the used rxb's. Running
rx_init and rx_allocator simultaniously causes a kernel panic. Fix
that by canceling the work in rx_init.

Signed-off-by: Shaul Triebitz <shaul.triebitz@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: allow different csr flags for different device families

Different device families may have different flag values
for passing a message to the fw (i.e. SW_RESET).
In order to keep the code readable, and avoid conditioning
upon the family, store a value for each flag, which indicates
the bit that needs to be enabled.

Signed-off-by: Golan Ben Ami <golan.ben.ami@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

locking/atomics: COCCINELLE/treewide: Convert trivial ACCESS_ONCE() patterns to READ_ONCE()/WRITE_ONCE()

Please do not apply this to mainline directly, instead please re-run the
coccinelle script shown below and apply its output.

For several reasons, it is desirable to use {READ,WRITE}_ONCE() in
preference to ACCESS_ONCE(), and new code is expected to use one of the
former. So far, there's been no reason to change most existing uses of
ACCESS_ONCE(), as these aren't harmful, and changing them results in
churn.

However, for some features, the read/write distinction is critical to
correct operation. To distinguish these cases, separate read/write
accessors must be used. This patch migrates (most) remaining
ACCESS_ONCE() instances to {READ,WRITE}_ONCE(), using the following
coccinelle script:

----
// Convert trivial ACCESS_ONCE() uses to equivalent READ_ONCE() and
// WRITE_ONCE()

// $ make coccicheck COCCI=/home/mark/once.cocci SPFLAGS="--include-headers" MODE=patch

virtual patch

@ depends on patch @
expression E1, E2;
@@

- ACCESS_ONCE(E1) = E2
+ WRITE_ONCE(E1, E2)

@ depends on patch @
expression E;
@@

- ACCESS_ONCE(E)
+ READ_ONCE(E)
----

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: davem@davemloft.net
Cc: linux-arch@vger.kernel.org
Cc: mpe@ellerman.id.au
Cc: shuah@kernel.org
Cc: snitzer@redhat.com
Cc: thor.thayer@linux.intel.com
Cc: tj@kernel.org
Cc: viro@zeniv.linux.org.uk
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/1508792849-3115-19-git-send-email-paulmck@linux.vnet.ibm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>

iwlwifi: pcie: move rx workqueue initialization to iwl_trans_pcie_alloc()

Work queues cannot be allocated when a mutex is held because the mutex
may be in use and that would make it sleep. Doing so generates the
following splat with 4.13+:

[ 19.513298] ======================================================
[ 19.513429] WARNING: possible circular locking dependency detected
[ 19.513557] 4.13.0-rc5+ #6 Not tainted
[ 19.513638] ------------------------------------------------------
[ 19.513767] cpuhp/0/12 is trying to acquire lock:
[ 19.513867] (&tz->lock){+.+.+.}, at: [<ffffffff924afebb>] thermal_zone_get_temp+0x5b/0xb0
[ 19.514047]
[ 19.514047] but task is already holding lock:
[ 19.514166] (cpuhp_state){+.+.+.}, at: [<ffffffff91cc4baa>] cpuhp_thread_fun+0x3a/0x210
[ 19.514338]
[ 19.514338] which lock already depends on the new lock.

This lock dependency already existed with previous kernel versions,
but it was not detected until commit 49dfe2a67797 ("cpuhotplug: Link
lock stacks for hotplug callbacks") was introduced.

Reported-by: David Weinehall <david.weinehall@intel.com>
Reported-by: Jiri Kosina <jikos@kernel.org>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>
Signed-off-by: Kalle Valo <kvalo@codeaurora.org>

iwlwifi: pcie: support short Tx queues for A000 device family

This allows to modify TFD_TX_CMD_SLOTS to a power of 2
which is smaller than 256.
Note that we still need to set values to wrap at 256
into the scheduler's write pointer, but all the rest of
the code can use shorter transmit queues.

Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: add MSI-X interrupt tracing

We have tracing for both pre-ICT and ICT interrupts, including all
the data read there. Extend the tracing to MSI-X interrupts.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: improve debug in iwl_pcie_rx_handle_rb()

Print the queue for the existing debug message and add a new
debug message indicating where the RB ended.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: improve "invalid queue" warning

Print out both queue IDs to be able to see what went wrong.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: only apply retention workaround on 9000-series A-step

Due to a hardware issue, certain power saving had to be
disabled. However, this issue was fixed in B-step, so the
workaround only needs to apply to A-step.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: delete the Tx queue timer earlier upon firmware crash

When the firmware crashes, the transmit queues can't make
any progress. This is why we stop the counter that monitor
the transmit queues' activity.
The call that notifies the error to the op_mode may take
a bit of time, so stop the timer of the transmit queues
earlier.

Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: don't disable bh when handling FW errors

When we started using threaded irqs, all the opmode calls were changed
to be called with local_bh disabled. The reason for this was it was
that mac80211 needs that. When we are handling FW errors, mac80211 is
not involved, so we don't need it.

Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: don't report RF-kill enabled while shutting down

When toggling the RF-kill pin quickly in succession, the driver can
get rather confused because it might be in the process of shutting
down, expecting all commands to go through quickly due to rfkill,
but the transport already thinks the device is accessible again,
even though it previously shut it down. This leads to bugs, and I
even observed a kernel panic.

Avoid this by making the PCIe code only report that the radio is
enabled again after the higher layers actually decided to shut it
off.

This also pulls out this common RF-kill checking code into a common
function called by both transport generations and also moves it to
the direct method - in the internal helper we don't really care
about the RF-kill status anymore since we won't report it up until
the stop anyway.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: add fake RF-kill to debugfs

In order to debug "hardware" RF-kill flows, add a low-level hook to
allow changing the "hardware" RF-kill from debugfs.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: pull out common rfkill IRQ handling code

There's no point in duplicating exactly the same code here
for legacy and MSI-X interrupts, so pull it out into a new
function to call in both places.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: use bitfield.h for some registers

Letting the preprocessor/compiler generate the shift/mask by itself
is a win for readability, so use bitfield.h for some registers.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: apply no-reclaim logic only to group 0

When applying no-reclaim logic to commands other than the group
zero for legacy commands, commands such as 0x1c (TX_CMD in group
0) can't be used in any other group. Fix that by applying this
logic only for group 0 - it's not and should never be needed for
any other groups.

Reported-by: Sharon Dvir <sharon.dvir@intel.com>
Reported-by: Shaul Triebitz <shaul.triebitz@intel.com>
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: alloc queues dynamically

Change queue allocation to be dynamic. On transport init only
the command queue is being allocated. Other queues are allocated
on demand.
This is due to the huge amount of queues we will soon enable (512)
and as a preparation for TX Virtual Queue Manager feature (TVQM),
where firmware will assign the actual queue number on demand.
This includes also allocation of the byte count table per queue
and not as a contiguous chunk of memory.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: prepare for dynamic queue allocation

In a000 transport we will allocate queues dynamically.
Right now queue are allocated as one big chunk of memory
and accessed as such.
The dynamic allocation of the queues will require accessing
the queues as pointers.
In order to keep simplicity of pre-a000 tx queues handling,
keep allocating and freeing the memory in the same style,
but move to access the queues in the various functions as
individual pointers.
Dynamic allocation for the a000 devices will be in a separate
patch.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: add context information support

Context information structure is going to be used in a000
devices for firmware self init.

The self init includes firmware self loading from DRAM by
ROM.
This means the TFH relevant firmware loading can be cleaned up.

The firmware loading includes the paging memory as well, so op
mode can stop initializing the paging and sending the DRAM_BLOCK_CMD.

Firmware is doing RFH, TFH and SCD configuration, while driver
only fills the required configurations and addresses in the
context information structure.

The only remaining access to RFH is the write pointer, which
is updated upon alive interrupt after FW configured the RFH.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: print less data upon firmware crash

We don't need to print so much data in the kernel log.
Limit the data to be printed to the queue that actually
got stuck in case of a TFD queue hang, and stop dumping
all the CSR and FH registers. Over the course of time, the
CSR and FH values haven't proven themselves to be really
useful for debugging, and they are now in the firmware dump
anyway.

This comes as a preparation to the addition of more data
required to be printed by the firwmare team.

Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: set STATUS_RFKILL immediately after interrupt

Currently, when getting a RFKILL interrupt, the transport enters a flow
in which it stops the device, disables other interrupts, etc. After
stopping the device, the transport resets the hw, and sleeps. During
the sleep, a context switch occurs and host commands are sent by upper
layers (e.g. mvm) to the fw. This is possible since the op_mode layer
and the transport layer hold different mutexes.

Since the STATUS_RFKILL bit isn't set, the transport layer doesn't
recognize that RFKILL was toggled on, and no commands can actually be
sent, so it enqueues the command to the tx queue and sets a timer on
the queue.

After switching context back to stopping the device, STATUS_RFKILL is
set, and then the transport can't send the command to the fw.
This eventually results in a queue hang.

Fix this by setting STATUS_RFKILL immediately when
the interrupt is fired.

Signed-off-by: Golan Ben-Ami <golan.ben.ami@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: fix another RF-kill race

When resuming, it's possible for the following scenario to occur:

* iwl_pci_resume() enables the RF-kill interrupt
* iwl_pci_resume() reads the RF-kill state (e.g. to 'radio enabled')
* RF_KILL interrupt triggers, and iwl_pcie_irq_handler() reads the
state, now 'radio disabled', and acquires the &trans_pcie->mutex.
* iwl_pcie_irq_handler() further calls iwl_trans_pcie_rf_kill() to
indicate to the higher layers that the radio is now disabled (and
stops the device while at it)
* iwl_pcie_irq_handler() drops the mutex
* iwl_pci_resume() continues, acquires the mutex and calls the higher
layers to indicate that the radio is enabled.

At this point, the device is stopped but the higher layers think it's
available, and can call deeply into the driver to try to enable it.
However, this will fail since the device is actually disabled.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: use LIST_HEAD() macro

There's no need to declare a list and then init it manually,
just use the LIST_HEAD() macro.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: log full command sequence

Log group as well. Remove 0x prefix to match TX logging.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: Configure shared interrupt vector in MSIX mode

In case the OS provides fewer interrupts than requested, different
causes will share the same interrupt vector as follow:
1.One interrupt less: non rx causes shared with FBQ.
2.Two interrupts less: non rx causes shared with FBQ and RSS.
3.More than two interrupts: we will use fewer RSS queues.

Also make the request depend on the number of online CPUs
instead of possible CPUs.

Signed-off-by: Haim Dreyfuss <haim.dreyfuss@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: merge iwl_queue and iwl_txq

The original intent was to have the general iwl_queue shared
between RX and TX queues, but it is not the actual status.
Since it is not shared with any struct but iwl_txq, it adds
unnecessary complexity. Merge those structs.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: centralize 64 bit HW registers write

Move the write_prph_64 of pcie to be transport agnostic.
Add direct write as well, as it is needed for a000 HW.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: track rxb status

In MQ environment and new architecture in early stages
we may encounter DMA issues. Track RXB status and bail
out in case we receive index to an RXB that was not
mapped and handed over to HW.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: fix a race in firmware loading flow

Upon firmware load interrupt (FH_TX), the ISR re-enables the
firmware load interrupt only to avoid races with other
flows as described in the commit below. When the firmware
is completely loaded, the thread that is loading the
firmware will enable all the interrupts to make sure that
the driver gets the ALIVE interrupt.
The problem with that is that the thread that is loading
the firmware is actually racing against the ISR and we can
get to the following situation:

CPU0 CPU1
iwl_pcie_load_given_ucode
...
iwl_pcie_load_firmware_chunk
wait_for_interrupt
<interrupt>
ISR handles CSR_INT_BIT_FH_TX
ISR wakes up the thread on CPU0
/* enable all the interrupts
* to get the ALIVE interrupt
*/
iwl_enable_interrupts
ISR re-enables CSR_INT_BIT_FH_TX only
/* start the firmware */
iwl_write32(trans, CSR_RESET, 0);

BUG! ALIVE interrupt will never arrive since it has been
masked by CPU1.

In order to fix that, change the ISR to first check if
STATUS_INT_ENABLED is set. If so, re-enable all the
interrupts. If STATUS_INT_ENABLED is clear, then we can
check what specific interrupt happened and re-enable only
that specific interrupt (RFKILL or FH_TX).

All the credit for the analysis goes to Kirtika who did the
actual debugging work.

Cc: <stable@vger.kernel.org> [4.5+]
Fixes: a6bd005fe92 ("iwlwifi: pcie: fix RF-Kill vs. firmware load race")
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: don't use vid 0

In cases of hardware or DMA error, the vid read from
a zeroed location will be 0, and we will access the rxb
at index 0 in the global table, while it may be NULL or
owned by hardware.
Invalidate vid 0 in order to detect the situation and
bail out.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: poll RFH for RX DMA stop

Somehow we ended up stopping RX using legacy RX registers
even for devices that support RFH. Fix it.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: unify restock calls on init

Currently code calls restock for mq devices during the init
function, unlike sq where restock is called after init.
This causes an harmless but alarming deadlock warning from
lockdep, to fix this - unify the init code.
Rename the restock functions while at it.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: make sure packet arrived to destined queue

Add a warning in case packet didn't end up in the HW
destined queue.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: allow more than one frame in RB for 9000 devices

We now have 9000 devices that support multiple frames in
a single RB. Enable it.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: set RB chunk size per bus

For 9000 devices we can have PCIe bus for discrete
devices and IOSF bus for integrated devices.
PCIe supports maximum transfer size of 128B while IOSF
bus supports maximum transfer size of 64B.
Configure RB size accordingly.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: workaround HW shadow registers bug

Integrated 9000 devices have a bug with shadow registers
value retention.
If driver writes RBD registers while MAC is asleep the
values are stored in shadow registers to be copied whenever
MAC wakes up.
However, in 9000 devices a MAC wakeup is not triggered
and when the bus powers down due to inactivity the shadow
values and dirty bits are lost.
Turn on the chicken-bits that cause MAC wakeup for RX-related
values as well when the device is in D0.
When the device is in low power mode turn the RX wakeup chicken
bits off since driver is idle and this W/A is not needed.
Remove previous W/A which was ineffective.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: grab NIC access only once on RX init

When initializing RX we grab NIC access for every read and
write. This is redundant - we can just grab access once.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: use shadow registers for updating write pointer

The RX queues have a shadow register for the write pointer
that enables updates without grabbing NIC access. Use them
instead of the periphery registers because accessing those
is much more expensive.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: don't wake up the NIC when writing CSRs in MSIX mode

CSR registers are always available even when the NIC is not awake, no
need to wake up the NIC before accessing them. This has a huge impact
when we re-enable an interrupt at the end of the ISR since waking up the
NIC can take some time.

Signed-off-by: Haim Dreyfuss <haim.dreyfuss@intel.com>
Signed-off-by: Luca Coelho <luciano.coelho@intel.com>

iwlwifi: pcie: remove duplicate assignment of variable isr_stats

isr_stats is written twice with the same value, remove one of the
redundant assignments to isr_stats.

Signed-off-by: Colin Ian King <colin.king@canonical.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: write to legacy register also in MQ

Due to hardware bug, upon any shadow free-queue register write
access, a legacy RBD shadow register must be written as well.
This is required in order to trigger a copy of the shadow registers
values after MAC exits sleep state.
Specifically, the driver has to write (any value) to the legacy RBD
register each time FRBDCB is accessed.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: fix global table size

My patch resized the pool size, but neglected to resize
the global table, which is obviously wrong since the global
table maps the pool's rxb to vid one to one. This results
in a panic in 9000 devices.
Add a build bug to avoid such a case in the future.

Fixes: 7b5424361ec9 ("iwlwifi: pcie: fine tune number of rxbs")
Reported-by: Haim Dreyfuss <haim.dreyfuss@intel.com>
Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: avoid restocks inside rx loop if not emergency

When trying to reach high Rx throughput of more than 500Mbps on
a device with a relatively weak CPU (Atom x5-Z8500), CPU utilization
may become a bottleneck. Analysis showed that we are looping in
iwl_pcie_rx_handle for very long periods which led to starvation
of other threads (iwl_pcie_rx_handle runs with _bh disabled).
We were handling Rx and allocating new buffers and the new buffers
were ready quickly enough to be available before we had finished
handling all the buffers available in the hardware. As a
consequence, we called iwl_pcie_rxq_restock to refill the hardware
with the new buffers, and start again handling new buffers without
exiting the function. Since we read the hardware pointer again when
we goto restart, new buffers were handled immediately instead of
exiting the function.

This patch avoids refilling RBs inside rx handling loop, unless an
emergency situation is reached. It also doesn't read the hardware
pointer again unless we are in an emergency (unlikely) case.
This significantly reduce the maximal time we spend in
iwl_pcie_rx_handle with _bh disabled.

Signed-off-by: Gregory Greenman <gregory.greenman@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: fine tune number of rxbs

We kick the allocator when we have 2 RBDs that don't have
attached RBs, and the allocator allocates 8 RBs meaning
that it needs another 6 RBDs to attach the RBs to.
The design is that allocator should always have enough RBDs
to fulfill requests, so we give in advance 6 RBDs to the
allocator so that when it is kicked, it gets additional 2 RBDs
and has enough RBDs.
These RBDs were taken from the Rx queue itself, meaning
that each Rx queue didn't have the maximal number of
RBDs, but MAX - 6.
Change initial number of RBDs in the system to include both
queue size and allocator reserves.
Note the multi-queue is always 511 instead of 512 to avoid a
full queue since we cannot detect this state easily enough in
the 9000 arch.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: set RB chunk size back to 64

128 byte chunk size is supported only on PCIe and not
on IOSF. For now, change it back to 64 byte.

Reported-by: Oren Givon <oren.givon@intel.com>
Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: refactor RXBs reclaiming code

Change the code to move rxbs directly from the allocator's
list to the queue's free list. This makes the code more
readable, saves the interim array and the double loop over
the free RBs.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: detect and workaround invalid write ptr behavior

In 9000 series A0 step the closed_rb_num is not wrapping around
properly. The queue is wrapping around as it should, so we can
W/A it by wrapping the closed_rb_num in the driver.
While at it, extend RX logging and add error handling of other
cases HW values may cause us to access invalid memory locations.
Add also a proper masking of vid value read from HW - this should
not have actual affect, but better to be on the safe side.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: configure more RFH settings

Fine tune RFH registers further:
* Set default queue explicitly
* Set RFH to drop frames exceeding RB size
* Set the maximum rx transfer size to DRAM to 128 instead of 64

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: Add new configuration to enable MSIX

Working with MSIX requires prior configuration.
This includes requesting interrupt vectors from the OS,
registering the vectors and mapping the optional causes to the
relevant interrupt. In addition add new interrupt handler
to handle MSIX interrupt.

Signed-off-by: Haim Dreyfuss <haim.dreyfuss@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: fix RF-Kill vs. firmware load race

When we load the firmware, we hold trans_pcie->mutex to
avoid nested flows. We also rely on the ISR to wake up the
thread when the DMA has finished copying a chunk. During
this flow, we enable the RF-Kill interrupt.

The problem is that the RF-Kill interrupt handler can take
the mutex and bring the device down. This means that if
we load the firmware while the RF-Kill switch is enabled
(which will happen when we load the INIT firmware to read
the device's capabilities and register to mac80211), we
may get an RF-Kill interrupt immediately and the ISR will
be waiting for the mutex held by the thread that is
currently loading the firmware. At this stage, the ISR
won't be able to service the DMA's interrupt needed to
wake up the thread that load the firmware. We are in a
deadlock situation which ends when the thread that loads
the firmware fails on timeout and releases the mutex.

To fix this, take the mutex later in the flow, disable
the interrupts and synchronize_irq() to give a chance to
the RF-Kill interrupt to run and complete.
After that, mask all the interrupts besides the DMA
interrupt and proceed with firmware load. Make sure to
check that there was no RF-Kill interrupt when the
interrupts were disabled.

This fixes https://bugzilla.kernel.org/show_bug.cgi?id=111361

Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: enable multi-queue rx path

Previous patches enabled new 9000 hardware DMA for one queue
only.
Enable the actual multi-queue path and configuration now.
This requires also per-queue NAPI struct.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: add 9000 series multi queue rx DMA support

The 9000 series introduces several changes in the device
DMA operation.
As the device now supports multi-queue rx, several DMA channels
should be configured.
The flows of providing the device with the allocated RBDs now
changes as well - the device maintains a separate table of used
and free table.

The hardware may use the free table to feed RBDs to any queue.
This requires maintaing a shared table to map returned RBDs to
the original RXB - for that purpose the VID is introduced - an
internal identifier of the RB placed in the lower 12 bits and
returned by HW in the used data.

Another change is the support of 64 bit DMA address.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: add infrastructure for multi-queue rx

The 9000 series devices will support multi rx queues.
Current code has one static rx queue - change it to allocate
a number of queues per the device capability (pre-9000 devices
have the number of rx queues set to one).

Subsequent generalizations are:

Change the code to access an explicit numbered rx queue only
when the queue number is known - when handling interrupt, when
accessing the default queue and when iterating the queues.
The rest of the functions will receive the rx queue as a pointer.

Generalize the warning in allocation failure to consider the
allocator status instead of a single rx queue status.

Move the rx initial pool of memory buffers to be shared among
all the queues and allocated to the default queue on init.

Signed-off-by: Sara Sharon <sara.sharon@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: update host command messages to new format

Host commands now have a group id, express this in printed messages.

Signed-off-by: Sharon Dvir <sharon.dvir@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: change the Intel Wireless email address

ilw@linux.intel.com is not available anymore.
linuxwifi@intel.com should be used instead.

Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: remove pointer from debug message

Since this pointer is not shown anywhere else, it's useless.
Remove it, just keeping the indexes.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: pcie: remove ICT allocation message

This message isn't very useful and presents a security risk
due to the use of %p - remove it.

Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: add support for 12K Receive Buffers

802.11ac allows A-MSDU that can be up to 12KB long. Since
an entire A-MSDU needs to fit into one single Receive
Buffer (RB), add support for big RBs.
Since this adds lots of pressure to the memory manager and
significantly increase the true_size of the RX buffers,
don't enable this by default.

Signed-off-by: Emmanuel Grumbach <emmanuel.grumbach@intel.com>

iwlwifi: move under intel vendor directory

Part of reorganising wireless drivers directory and Kconfig.

Signed-off-by: Kalle Valo <kvalo@codeaurora.org>