cpuidle, arch: Mark all ct_cpuidle_enter() callers __cpuidle

For all cpuidle drivers that use CPUIDLE_FLAG_RCU_IDLE, ensure that
all functions that call ct_cpuidle_enter() are marked __cpuidle.

( due to lack of noinstr validation on these platforms it is entirely
possible this isn't complete )

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Link: https://lore.kernel.org/r/20230112195542.274096325@infradead.org

cpuidle: Fix ct_idle_*() usage

The whole disable-RCU, enable-IRQS dance is very intricate since
changing IRQ state is traced, which depends on RCU.

Add two helpers for the cpuidle case that mirror the entry code:

ct_cpuidle_enter()
ct_cpuidle_exit()

And fix all the cases where the enter/exit dance was buggy.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Tested-by: Tony Lindgren <tony@atomide.com>
Tested-by: Ulf Hansson <ulf.hansson@linaro.org>
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Frederic Weisbecker <frederic@kernel.org>
Link: https://lore.kernel.org/r/20230112195540.130014793@infradead.org

context_tracking: Take idle eqs entrypoints over RCU

The RCU dynticks counter is going to be merged into the context tracking
subsystem. Start with moving the idle extended quiescent states
entrypoints to context tracking. For now those are dumb redirections to
existing RCU calls.

[ paulmck: Apply kernel test robot feedback. ]

Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
Cc: Uladzislau Rezki <uladzislau.rezki@sony.com>
Cc: Joel Fernandes <joel@joelfernandes.org>
Cc: Boqun Feng <boqun.feng@gmail.com>
Cc: Nicolas Saenz Julienne <nsaenz@kernel.org>
Cc: Marcelo Tosatti <mtosatti@redhat.com>
Cc: Xiongfeng Wang <wangxiongfeng2@huawei.com>
Cc: Yu Liao <liaoyu15@huawei.com>
Cc: Phil Auld <pauld@redhat.com>
Cc: Paul Gortmaker<paul.gortmaker@windriver.com>
Cc: Alex Belits <abelits@marvell.com>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Nicolas Saenz Julienne <nsaenzju@redhat.com>
Tested-by: Nicolas Saenz Julienne <nsaenzju@redhat.com>

ARM: imx6q: Fixup RCU usage for cpuidle

The commit eb1f00237aca ("lockdep,trace: Expose tracepoints"), started to
expose us for tracepoints. For imx6q cpuidle, this leads to an RCU splat
according to below.

[6.870684] [<c0db7690>] (_raw_spin_lock) from [<c011f6a4>] (imx6q_enter_wait+0x18/0x9c)
[6.878846] [<c011f6a4>] (imx6q_enter_wait) from [<c09abfb0>] (cpuidle_enter_state+0x168/0x5e4)

To fix the problem, let's assign the corresponding idlestate->flags the
CPUIDLE_FLAG_RCU_IDLE bit, which enables us to call rcu_idle_enter|exit()
at the proper point.

Reported-by: Dong Aisheng <aisheng.dong@nxp.com>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Ulf Hansson <ulf.hansson@linaro.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

cpuidle: Introduce cpuidle_driver_state_disabled() for driver quirks

Commit 99e98d3fb100 ("cpuidle: Consolidate disabled state checks")
overlooked the fact that the imx6q and tegra20 cpuidle drivers use
the "disabled" field in struct cpuidle_state for quirks which trigger
after the initialization of cpuidle, so reading the initial value of
that field is not sufficient for those drivers.

In order to allow them to implement the quirks without using the
"disabled" field in struct cpuidle_state, introduce a new helper
function and modify them to use it.

Fixes: 99e98d3fb100 ("cpuidle: Consolidate disabled state checks")
Reported-by: Len Brown <lenb@kernel.org>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500

Based on 2 normalized pattern(s):

this program is free software you can redistribute it and or modify
it under the terms of the gnu general public license version 2 as
published by the free software foundation

this program is free software you can redistribute it and or modify
it under the terms of the gnu general public license version 2 as
published by the free software foundation #

extracted by the scancode license scanner the SPDX license identifier

GPL-2.0-only

has been chosen to replace the boilerplate/reference in 4122 file(s).

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Enrico Weigelt <info@metux.net>
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Allison Randal <allison@lohutok.net>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190604081206.933168790@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

ARM: imx6: cpuidle: Use raw_spinlock_t

The idle call back is invoked with disabled interrupts and requires
raw_spinlock_t locks to work.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Reviewed-by: Fabio Estevam <festevam@gmail.com>
Signed-off-by: Shawn Guo <shawnguo@kernel.org>

ARM: imx6q: cpuidle: fix bug that CPU might not wake up at expected time

In the current cpuidle implementation for i.MX6q, the CPU that sets
'WAIT_UNCLOCKED' and the CPU that returns to 'WAIT_CLOCKED' are always
the same. While the CPU that sets 'WAIT_UNCLOCKED' is in IDLE state of
"WAIT", if the other CPU wakes up and enters IDLE state of "WFI"
istead of "WAIT", this CPU can not wake up at expired time.
Because, in the case of "WFI", the CPU must be waked up by the local
timer interrupt. But, while 'WAIT_UNCLOCKED' is set, the local timer
is stopped, when all CPUs execute "wfi" instruction. As a result, the
local timer interrupt is not fired.
In this situation, this CPU will wake up by IRQ different from local
timer. (e.g. broacast timer)

So, this fix changes CPU to return to 'WAIT_CLOCKED'.

Signed-off-by: Kohji Okuno <okuno.kohji@jp.panasonic.com>
Fixes: e5f9dec8ff5f ("ARM: imx6q: support WAIT mode using cpuidle")
Cc: <stable@vger.kernel.org>
Signed-off-by: Shawn Guo <shawnguo@kernel.org>

ARM: imx: rename imx6q_set_int_mem_clk_lpm() function

Let's rename the function imx6q_set_int_mem_clk_lpm()
to imx6_set_int_mem_clk_lpm() since it's actually
common for all i.MX6 SoCs.

Signed-off-by: Anson Huang <Anson.Huang@nxp.com>
Signed-off-by: Shawn Guo <shawnguo@kernel.org>

ARM: imx6: fix missing <soc/imx/cpuidle.h> in cpuidle-imx6q.c

The <soc/imx/cpuidle.h> file has declarations of two funcitons
exported from cpuidle-imx6q.c but it is not included. Fix the
following warnings by adding the include:

arch/arm/mach-imx/cpuidle-imx6q.c:71:6: warning: symbol 'imx6q_cpuidle_fec_irqs_used' was not declared. Should it be static?
arch/arm/mach-imx/cpuidle-imx6q.c:76:6: warning: symbol 'imx6q_cpuidle_fec_irqs_unused' was not declared. Should it be static?

Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
Signed-off-by: Shawn Guo <shawnguo@kernel.org>

ARM: imx6q: export cpuidle functions needed by fec driver

Export cpuidle functions needed by fec driver to fix the issue below
seen with fec module build.

ERROR: "imx6q_cpuidle_fec_irqs_unused" [drivers/net/ethernet/freescale/fec.ko] undefined!
ERROR: "imx6q_cpuidle_fec_irqs_used" [drivers/net/ethernet/freescale/fec.ko] undefined!

Reported-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Shawn Guo <shawnguo@kernel.org>

ARM: imx6: disable deeper idle states when FEC is active w/o HW workaround

The i.MX6 Q/DL has an erratum (ERR006687) that prevents the FEC from
waking the CPUs when they are in wait(unclocked) state. As the hardware
workaround isn't applicable to all boards, disable the deeper idle state
when the workaround isn't present and the FEC is in use.

This allows to safely run a kernel with CPUidle enabled on all i.MX6
boards.

Signed-off-by: Lucas Stach <l.stach@pengutronix.de>
Acked-by: David S. Miller <davem@davemloft.net> (for network changes)
Signed-off-by: Shawn Guo <shawnguo@kernel.org>

ARM: imx6: set initial power mode in pm function

Rather than setting initial low-power mode in every single i.MX6 clock
initialization function, we should really do that in pm code. Let's
move imx6q_set_lpm(WAIT_CLOCKED) call into imx6_pm_common_init().

While at it, let's rename the function to imx6_set_lpm() since it's
actually common for all i.MX6 SoCs.

Signed-off-by: Shawn Guo <shawn.guo@linaro.org>

ARM: cpuidle: Remove duplicate header inclusion

The cpu_do_idle() function is always used by the cpuidle drivers.

That led to have each driver including cpuidle.h and proc-fns.h, they are
always paired. That makes a lot of duplicate headers inclusion. Instead of
including both in each .c file, move the proc-fns.h header inclusion in the
cpuidle.h header file directly, so we can save some line of code.

Signed-off-by: Daniel Lezcano <daniel.lezcano@linaro.org>
Acked-by: Kevin Hilman <khilman@linaro.org>
Acked-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Tested-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>

cpuidle: Invert CPUIDLE_FLAG_TIME_VALID logic

The only place where the time is invalid is when the ACPI_CSTATE_FFH entry
method is not set. Otherwise for all the drivers, the time can be correctly
measured.

Instead of duplicating the CPUIDLE_FLAG_TIME_VALID flag in all the drivers
for all the states, just invert the logic by replacing it by the flag
CPUIDLE_FLAG_TIME_INVALID, hence we can set this flag only for the acpi idle
driver, remove the former flag from all the drivers and invert the logic with
this flag in the different governor.

Signed-off-by: Daniel Lezcano <daniel.lezcano@linaro.org>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

ARM: imx: remove imx_scu_standby_enable()

With commit c716483c3db1 ("ARM: 8122/1: smp_scu: enable SCU standby
support"), the STANDBY bit of SCU is handled by core function
scu_enable(). So imx_scu_standby_enable() can be removed now.

Signed-off-by: Shawn Guo <shawn.guo@freescale.com>

ARM: imx: mem bit must be cleared before entering DSM mode

According to hardware design, mem bit must be clear before
entering DSM mode, as ARM core will be power gated in DSM mode.

Signed-off-by: Anson Huang <b20788@freescale.com>
Signed-off-by: Shawn Guo <shawn.guo@freescale.com>

ARM: imx: add cpuidle support for i.mx6sx

Add cpuidle support for i.MX6SX, derive from i.MX6Q's
cpuidle, two levels supported:

1. WFI;
2. WAIT mode.

Signed-off-by: Anson Huang <b20788@freescale.com>
Signed-off-by: Shawn Guo <shawn.guo@freescale.com>

ARM: imx: Use INT_MEM_CLK_LPM as the bit name

Bit 17 of register CCM_CGPR is called INT_MEM_CLK_LPM as per the mx6
reference manual, so use this name instead.

Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
Signed-off-by: Shawn Guo <shawn.guo@linaro.org>

ARM: imx: cpuidle: use init/exit common routine

The code intializes the cpuidle driver at different places.
The cpuidle driver for :
* imx5 : is in the pm-imx5.c, the init function is in cpuidle.c
* imx6 : is in cpuidle-imx6q.c, the init function is in cpuidle.c
and cpuidle-imx6q.c

Instead of having the cpuidle code spread across different files,
let's create a driver for each SoC and use the common register function.

Signed-off-by: Daniel Lezcano <daniel.lezcano@linaro.org>
Acked-by: Shawn Guo <shawn.guo@linaro.org>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

cpuidle: remove en_core_tk_irqen flag

The en_core_tk_irqen flag is set in all the cpuidle driver which
means it is not necessary to specify this flag.

Remove the flag and the code related to it.

Signed-off-by: Daniel Lezcano <daniel.lezcano@linaro.org>
Acked-by: Kevin Hilman <khilman@linaro.org> # for mach-omap2/*
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

cpuidle: imx6: remove timer broadcast initialization

The initialization is done from the cpuidle framework.

Signed-off-by: Daniel Lezcano <daniel.lezcano@linaro.org>
Acked-by: Shawn Guo <shawn.guo@linaro.org>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

cpuidle / imx6 : use CPUIDLE_FLAG_TIMER_STOP flag

Use the CPUIDLE_FLAG_TIMER_STOP and let the cpuidle framework
to handle the CLOCK_EVT_NOTIFY_BROADCAST_ENTER/EXIT when entering
this state.

Signed-off-by: Daniel Lezcano <daniel.lezcano@linaro.org>
Acked-by: Santosh Shilimkar <santosh.shilimkar@ti.com>
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>

ARM: imx6q: support WAIT mode using cpuidle

Add WAIT mode (ARM core clock gating) support to imx6q cpuidle driver.
As WAIT mode is broken on imx6q TO 1.0 and 1.1, it only enables the
support for revision 1.2 with chicken bit set.

Signed-off-by: Shawn Guo <shawn.guo@linaro.org>

ARM: imx: move imx6q_cpuidle_driver into a separate file

Move imx6q_cpuidle_driver into a separate file as more codes will
be added when WAIT mode gets implemented as cpuidle.

Signed-off-by: Shawn Guo <shawn.guo@linaro.org>