scsi: ufs: core: Add config_scsi_dev vops comment

Add config_scsi_dev vops comment.

Signed-off-by: Peter Wang <peter.wang@mediatek.com>
Link: https://lore.kernel.org/r/20240301034610.24928-1-peter.wang@mediatek.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: mcq: Add definition for REG_UFS_MEM_CFG register

Instead of hardcoding the register field, add the proper definition. While
at it, let's also use ufshcd_rmwl() to simplify updating this register.

Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: ChanWoo Lee <cw9316.lee@samsung.com>
Link: https://lore.kernel.org/r/20240102014222.23351-1-cw9316.lee@samsung.com
Reviewed-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add CPU latency QoS support for UFS driver

Register UFS driver to CPU latency PM QoS framework to improve UFS device
random I/O performance.

PM QoS initialization will insert new QoS request into the CPU latency QoS
list with the maximum latency PM_QOS_DEFAULT_VALUE value.

The UFS driver will vote for performance mode on scale up and power save
mode for scale down.

If clock scaling feature is not enabled then voting will be based on clock
on or off condition. Also provide a sysfs interface to enable/disable PM
QoS feature.

tiotest benchmark tool I/O performance results on sm8550 platform:

1. Without PM QoS support
Type (Speed in) | Average of 18 iterations
Random Write(IPOS) | 41065.13
Random Read(IPOS) | 37101.3

2. With PM QoS support
Type (Speed in) | Average of 18 iterations
Random Write(IPOS) | 46784.9
Random Read(IPOS) | 42943.4

(Improvement with PM QoS = ~15%).

Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Reviewed-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
Co-developed-by: Nitin Rawat <quic_nitirawa@quicinc.com>
Signed-off-by: Nitin Rawat <quic_nitirawa@quicinc.com>
Co-developed-by: Naveen Kumar Goud Arepalli <quic_narepall@quicinc.com>
Signed-off-by: Naveen Kumar Goud Arepalli <quic_narepall@quicinc.com>
Signed-off-by: Maramaina Naresh <quic_mnaresh@quicinc.com>
Link: https://lore.kernel.org/r/20231219123706.6463-2-quic_mnaresh@quicinc.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Rename ufshcd_auto_hibern8_enable() and make it static

Rename ufshcd_auto_hibern8_enable() into ufshcd_configure_auto_hibern8()
since this function can enable or disable auto-hibernation. Since
ufshcd_auto_hibern8_enable() is only used inside the UFSHCI driver core,
declare it static. Additionally, move the definition of this function to
just before its first caller.

Suggested-by: Bao D. Nguyen <quic_nguyenb@quicinc.com>
Reviewed-by: Bao D. Nguyen <quic_nguyenb@quicinc.com>
Reviewed-by: Can Guo <quic_cang@quicinc.com>
Cc: Avri Altman <avri.altman@wdc.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20231214192416.3638077-2-bvanassche@acm.org
Reviewed-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add UFS RTC support

Add Real Time Clock (RTC) support for UFS device. This enhancement is
crucial for the internal maintenance operations of the UFS device. The
patch enables the device to handle both absolute and relative time
information. Furthermore, it includes periodic task to update the RTC in
accordance with the UFS Spec, ensuring the accuracy of RTC information for
the device's internal processes.

RTC and qTimestamp serve distinct purposes. The RTC provides a coarse level
of granularity with, at best, approximate single-second resolution. This
makes the RTC well-suited for the device to determine the approximate age
of programmed blocks after being updated by the host. On the other hand,
qTimestamp offers nanosecond granularity and is specifically designed for
synchronizing Device Error Log entries with corresponding host-side logs.

Given that the RTC has been a standard feature since UFS Spec 2.0, and
qTimestamp was introduced in UFS Spec 4.0, the majority of UFS devices
currently on the market rely on RTC. Therefore, it is advisable to continue
supporting RTC in the Linux kernel. This ensures compatibility with the
prevailing UFS device implementations and facilitates seamless integration
with existing hardware. By maintaining support for RTC, we ensure broad
compatibility and avoid potential issues arising from deviations in device
specifications across different UFS versions.

Signed-off-by: Bean Huo <beanhuo@micron.com>
Signed-off-by: Mike Bi <mikebi@micron.com>
Signed-off-by: Luca Porzio <lporzio@micron.com>
Link: https://lore.kernel.org/r/20231212220825.85255-3-beanhuo@iokpp.de
Acked-by: Avri Altman <avri.altman@wdc.com>
Reviewed-by: Thomas Weißschuh <linux@weissschuh.net>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: qcom: Export ufshcd_{enable/disable}_irq helpers and make use of them

Instead of duplicating the enable/disable IRQ part, let's export the
helpers available in ufshcd driver and make use of them. This also fixes
the possible redundant IRQ disable before asserting reset (when IRQ was
already disabled).

Fixes: 4a791574a0cc ("scsi: ufs: ufs-qcom: Disable interrupt in reset path")
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Link: https://lore.kernel.org/r/20231208065902.11006-7-manivannan.sadhasivam@linaro.org
Tested-by: Andrew Halaney <ahalaney@redhat.com> # sa8775p-ride
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Make fault injection dynamically configurable per HBA

The UFS driver has two driver-specific fault injection mechanisms
(trigger_eh and timeout). Each fault injection configuration can only be
specified by a module parameter and cannot be reconfigured without
reloading the driver. Also, each configuration is common to all HBAs.

This change adds the following subdirectories for each UFS HBA when
debugfs is enabled:

/sys/kernel/debug/ufshcd/<HBA>/timeout_inject
/sys/kernel/debug/ufshcd/<HBA>/trigger_eh_inject

Each fault injection attribute can be dynamically set per HBA by a
corresponding file in these directories.

This is tested with QEMU UFS devices.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Link: https://lore.kernel.org/r/20231118124443.1007116-1-akinobu.mita@gmail.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add support for parsing OPP

OPP framework can be used to scale the clocks along with other entities
such as regulators, performance state etc... So let's add support for
parsing OPP from devicetree. OPP support in devicetree is added through the
"operating-points-v2" property which accepts the OPP table defining clock
frequency, regulator voltage, power domain performance state etc...

Since the UFS controller requires multiple clocks to be controlled for
proper working, devm_pm_opp_set_config() has been used which supports
scaling multiple clocks through custom ufshcd_opp_config_clks() callback.

It should be noted that the OPP support is not compatible with the old
"freq-table-hz" property. So only one can be used at a time even though
the UFS core supports both.

Co-developed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Link: https://lore.kernel.org/r/20231012172129.65172-4-manivannan.sadhasivam@linaro.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add OPP support for scaling clocks and regulators

UFS core is only scaling the clocks during devfreq scaling and
initialization. But for an optimum power saving, regulators should also be
scaled along with the clocks.

So let's use the OPP framework which supports scaling clocks, regulators,
and performance state using OPP table defined in devicetree. For
accomodating the OPP support, the existing APIs (ufshcd_scale_clks,
ufshcd_is_devfreq_scaling_required and ufshcd_devfreq_scale) are modified
to accept "freq" as an argument which in turn used by the OPP helpers.

The OPP support is added along with the old freq-table based clock scaling
so that the existing platforms work as expected.

Co-developed-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Link: https://lore.kernel.org/r/20231012172129.65172-3-manivannan.sadhasivam@linaro.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Move the 4K alignment code into the Exynos driver

The DMA alignment for the Exynos controller follows directly from the PRDT
segment size configured in ufs-exynos.c. Hence, move the DMA alignment code
into the Exynos driver source code.

Cc: Alim Akhtar <alim.akhtar@samsung.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20230921192335.676924-3-bvanassche@acm.org
Reviewed-by: Alim Akhtar <alim.akhtar@samsung.com>
Tested-by: Alim Akhtar <alim.akhtar@samsung.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Export ufshcd_is_hba_active()

Export ufshcd_is_hba_active() to allow driver modules to check the state of
the host controller.

Signed-off-by: Nitin Rawat <quic_nitirawa@quicinc.com>
Link: https://lore.kernel.org/r/20230726134140.7180-2-quic_nitirawa@quicinc.com
Acked-by: Manivannan Sadhasivam <mani@kernel.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: Remove a member variable

Remove the 'response' member variable because no code reads its value.
Additionally, move the ufs_query_req and ufs_query_res data structure
definitions into include/ufs/ufshcd.h because these data structures are
related to the UFS host controller driver.

Reviewed-by: Avri Altman <avri.altman@wdc.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20230727194457.3152309-11-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: Improve type safety

Assign names to the enumeration types for UPIU types. Use these enumeration
types where appropriate.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20230727194457.3152309-8-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: Minimize #include directives

Only #include those header files that are needed.

Note: include/ufs/ufshcd.h needs <scsi/scsi_host.h> because of SG_ALL.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20230727194457.3152309-6-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Remove HPB support

Interest among UFS users in HPB has reduced significantly. I am not aware
of any current users of the HPB functionality. Hence remove HPB support
from the kernel.

A note: the work in JEDEC on a successor for HPB is nearing completion.
Zoned storage for UFS or ZUFS combines the UFS standard with ZBC-2.

Acked-by: Avri Altman <avri.altman@wdc.com>
Reviewed-by: Bean Huo <beanhuo@micron.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: ChanWoo Lee <cw9316.lee@samsung.com>
Cc: Daejun Park <daejun7.park@samsung.com>
Cc: Keoseong Park <keosung.park@samsung.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20230719165758.2787573-1-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Export symbols for MTK driver module

Export symbols for MediaTek UFS driver's PM flow and IRQ handler.

Signed-off-by: Po-Wen Kao <powen.kao@mediatek.com>
Link: https://lore.kernel.org/r/20230701124442.10489-2-powen.kao@mediatek.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add host quirk UFSHCD_QUIRK_MCQ_BROKEN_RTC

Some hosts do not implement SQ Run Time Command (SQRTC) register, thus we
need this quirk to skip the related flow.

Signed-off-by: Po-Wen Kao <powen.kao@mediatek.com>
Link: https://lore.kernel.org/r/20230612085817.12275-3-powen.kao@mediatek.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add host quirk UFSHCD_QUIRK_MCQ_BROKEN_INTR

Quirk UFSHCD_QUIRK_MCQ_BROKEN_INTR is introduced for hosts that implement a
different interrupt topology from the UFSHCI 4.0 spec. Some hosts raise
per hw queue interrupt in addition to CQES (traditional) when ESI is
disabled.

Enabling this quirk will disable CQES and use only per hw queue interrupt.

Signed-off-by: Po-Wen Kao <powen.kao@mediatek.com>
Link: https://lore.kernel.org/r/20230612085817.12275-2-powen.kao@mediatek.com
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: mcq: Fix the incorrect OCS value for the device command

In MCQ mode, when a device command uses a hardware queue shared with other
commands, a race condition may occur in the following scenario:

1. A device command is completed in CQx with CQE entry "e".

2. The interrupt handler copies the "cqe" pointer to "hba->dev_cmd.cqe"
and completes "hba->dev_cmd.complete".

3. The "ufshcd_wait_for_dev_cmd()" function is awakened and retrieves the
OCS value from "hba->dev_cmd.cqe".

However, there is a possibility that the CQE entry "e" will be overwritten
by newly completed commands in CQx, resulting in an incorrect OCS value
being received by "ufshcd_wait_for_dev_cmd()".

To avoid this race condition, the OCS value should be immediately copied to
the struct "lrb" of the device command. Then "ufshcd_wait_for_dev_cmd()"
can retrieve the OCS value from the struct "lrb".

Fixes: 57b1c0ef89ac ("scsi: ufs: core: mcq: Add support to allocate multiple queues")
Suggested-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Stanley Chu <stanley.chu@mediatek.com>
Link: https://lore.kernel.org/r/20230610021553.1213-2-powen.kao@mediatek.com
Tested-by: Po-Wen Kao <powen.kao@mediatek.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Simplify driver shutdown

All UFS host drivers call ufshcd_shutdown(). Hence, instead of calling
ufshcd_shutdown() from the host driver .shutdown() callback, inline that
function into ufshcd_wl_shutdown().

Reviewed-by: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20230524203659.1394307-5-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: mcq: Use ufshcd_mcq_poll_cqe_lock() in MCQ mode

In preparation for adding MCQ error handler support, update the MCQ code to
use the ufshcd_mcq_poll_cqe_lock() in interrupt context instead of using
ufshcd_mcq_poll_cqe_nolock(). This is to keep synchronization between MCQ
interrupt and error handler contexts because both need to access the MCQ
hardware in separate contexts.

Signed-off-by: Bao D. Nguyen <quic_nguyenb@quicinc.com>
Link: https://lore.kernel.org/r/6ae727ad2a4040469b8f0632b55e0577d80da11b.1685396241.git.quic_nguyenb@quicinc.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Tested-by: Stanley Chu <stanley.chu@mediatek.com>
Reviewed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: mcq: Add supporting functions for MCQ abort

Add supporting functions to handle UFS abort in MCQ mode.

Signed-off-by: Bao D. Nguyen <quic_nguyenb@quicinc.com>
Link: https://lore.kernel.org/r/d452c5ad62dc863cc067ec82daa0885ec98bd508.1685396241.git.quic_nguyenb@quicinc.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Tested-by: Stanley Chu <stanley.chu@mediatek.com>
Reviewed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: Ungate the clock synchronously

Ungating the clock asynchronously causes ufshcd_queuecommand() to return
SCSI_MLQUEUE_HOST_BUSY and hence causes commands to be requeued. This is
suboptimal. Allow ufshcd_queuecommand() to sleep such that clock ungating
does not trigger command requeuing. Remove the ufshcd_scsi_block_requests()
and ufshcd_scsi_unblock_requests() calls because these are no longer
needed. The flush_work(&hba->clk_gating.ungate_work) call is sufficient to
make the SCSI core wait for clock ungating to complete.

Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20230529202640.11883-6-bvanassche@acm.org
Reviewed-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Bao D. Nguyen <quic_nguyenb@quicinc.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Rename symbol sizeof_utp_transfer_cmd_desc()

Naming the functions after standard operators like sizeof() may cause
confusion. Rename it to ufshcd_get_ucd_size().

Signed-off-by: Po-Wen Kao <powen.kao@mediatek.com>
Link: https://lore.kernel.org/r/20230504154454.26654-3-powen.kao@mediatek.com
Suggested-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Reviewed-by: Ziqi Chen <quic_ziqichen@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: mcq: Use pointer arithmetic in ufshcd_send_command()

Make sqe_base_addr the UTRD pointer it is, instead of an opaque void *.

Signed-off-by: Avri Altman <avri.altman@wdc.com>
Link: https://lore.kernel.org/r/20230329101303.18377-3-avri.altman@wdc.com
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: core: Clean up struct ufs_saved_pwr_info

The "is_valid" field of the struct ufs_saved_pwr_info is no longer used,
and this struct can be replaced by struct ufs_pa_layer_attr without any
changes to the functionality.

Signed-off-by: Stanley Chu <stanley.chu@mediatek.com>
Link: https://lore.kernel.org/r/20230330012918.13748-1-stanley.chu@mediatek.com
Reviewed-by: Avri Altman <avri.altman@wdc.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: Revert "scsi: ufs: core: Initialize devfreq synchronously"

This reverts commit 7dafc3e007918384c8693ff8d70381b5c1e9c247.

This patch introduced a regression [1] where hba->pwr_info is used before
being initialized, which could create issues in ufshcd_scale_gear(). Revert
it until a better solution is found.

[1] https://lore.kernel.org/all/CAGaU9a_PMZhqv+YJ0r3w-hJMsR922oxW6Kg59vw+oen-NZ6Otw@mail.gmail.com

Signed-off-by: Adrien Thierry <athierry@redhat.com>
Link: https://lore.kernel.org/r/20230329205426.46393-1-athierry@redhat.com
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Initialize devfreq synchronously

During UFS initialization, devfreq initialization is asynchronous:
ufshcd_async_scan() calls ufshcd_add_lus(), which in turn initializes
devfreq for UFS. The simple ondemand governor is then loaded. If it is
built as a module, request_module() is called and throws a warning:

WARNING: CPU: 7 PID: 167 at kernel/kmod.c:136 __request_module+0x1e0/0x460
Modules linked in: crct10dif_ce llcc_qcom phy_qcom_qmp_usb ufs_qcom phy_qcom_snps_femto_v2 ufshcd_pltfrm phy_qcom_qmp_combo ufshcd_core phy_qcom_qmp_ufs qcom_wdt socinfo fuse ipv6
CPU: 7 PID: 167 Comm: kworker/u16:3 Not tainted 6.2.0-rc6-00009-g58706f7fb045 #1
Hardware name: Qualcomm SA8540P Ride (DT)
Workqueue: events_unbound async_run_entry_fn
pstate: 00400005 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : __request_module+0x1e0/0x460
lr : __request_module+0x1d8/0x460
sp : ffff800009323b90
x29: ffff800009323b90 x28: 0000000000000000 x27: 0000000000000000
x26: ffff800009323d50 x25: ffff7b9045f57810 x24: ffff7b9045f57830
x23: ffffdc5a83e426e8 x22: ffffdc5ae80a9818 x21: 0000000000000001
x20: ffffdc5ae7502f98 x19: ffff7b9045f57800 x18: ffffffffffffffff
x17: 312f716572667665 x16: 642f7366752e3030 x15: 0000000000000000
x14: 000000000000021c x13: 0000000000005400 x12: ffff7b9042ed7614
x11: ffff7b9042ed7600 x10: 00000000636c0890 x9 : 0000000000000038
x8 : ffff7b9045f2c880 x7 : ffff7b9045f57c68 x6 : 0000000000000080
x5 : 0000000000000000 x4 : 8000000000000000 x3 : 0000000000000000
x2 : 0000000000000000 x1 : ffffdc5ae5d382f0 x0 : 0000000000000001
Call trace:
__request_module+0x1e0/0x460
try_then_request_governor+0x7c/0x100
devfreq_add_device+0x4b0/0x5fc
ufshcd_async_scan+0x1d4/0x310 [ufshcd_core]
async_run_entry_fn+0x34/0xe0
process_one_work+0x1d0/0x320
worker_thread+0x14c/0x444
kthread+0x10c/0x110
ret_from_fork+0x10/0x20

This occurs because synchronous module loading from async is not
allowed. According to __request_module():

/*
* We don't allow synchronous module loading from async. Module
* init may invoke async_synchronize_full() which will end up
* waiting for this task which already is waiting for the module
* loading to complete, leading to a deadlock.
*/

Such a deadlock was experienced on the Qualcomm QDrive3/sa8540p-ride. With
DEVFREQ_GOV_SIMPLE_ONDEMAND=m, the boot hangs after the warning.

Fix both the warning and the deadlock by moving devfreq initialization out
of the async routine.

Tested on the sa8540p-ride by using fio to put the UFS under load, and
printing the trace generated by
/sys/kernel/tracing/events/ufs/ufshcd_clk_scaling events. The trace looks
similar with and without the change.

Link: https://lore.kernel.org/r/20230217194423.42553-1-athierry@redhat.com
Signed-off-by: Adrien Thierry <athierry@redhat.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add hibernation callbacks

Add freeze, thaw, and restore callbacks for hibernate and restore
functionality.

Link: https://lore.kernel.org/r/20230202161045.3956-2-quic_ahari@quicinc.com
Signed-off-by: Anjana Hari <quic_ahari@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: exynos: Fix DMA alignment for PAGE_SIZE != 4096

The Exynos UFS controller only supports scatter/gather list elements that
are aligned on a 4 KiB boundary. Fix DMA alignment in case PAGE_SIZE !=
4096. Rename UFSHCD_QUIRK_ALIGN_SG_WITH_PAGE_SIZE into
UFSHCD_QUIRK_4KB_DMA_ALIGNMENT.

Cc: Kiwoong Kim <kwmad.kim@samsung.com>
Fixes: 2b2bfc8aa519 ("scsi: ufs: Introduce a quirk to allow only page-aligned sg entries")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Alim Akhtar <alim.akhtar@samsung.com>
Tested-by: Alim Akhtar <alim.akhtar@samsung.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: mcq: Add Event Specific Interrupt enable and config functions

Add and export two functions to enable ESI and config ESI base addresses.
The calls to these exported functions will be added by the next patch in
this series.

Signed-off-by: Can Guo <quic_cang@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add Event Specific Interrupt configuration vendor specific ops

As Event Specific Interrupt message format is not defined in UFSHCI JEDEC
specs, and the ESI handling highly depends on how the format is designed,
hence add a vendor specific ops such that SoC vendors can configure their
own ESI handlers. If ESI vops is not provided or returning error, go with
the legacy (central) interrupt way.

Signed-off-by: Can Guo <quic_cang@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: mcq: Add completion support in poll

Complete CQE requests in poll. Assumption is that several poll completion
may happen in different CPUs for the same completion queue. Hence a spin
lock protection is added.

Co-developed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: mcq: Add completion support of a CQE

Add support for completing requests from Completion Queue. Some host
controllers support vendor specific registers that provide a bitmap of all
CQs which have at least one completed CQE. Add this support. The MCQ
specification doesn't provide the Task Tag or its equivalent in the
Completion Queue Entry. So use an indirect method to find the Task Tag
from the Completion Queue Entry.

Co-developed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Prepare ufshcd_send_command() for MCQ

Add support to send commands using multiple submission queues in MCQ mode.
Modify the functions that use ufshcd_send_command().

Co-developed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: mcq: Configure operation and runtime interface

Runtime and operation registers are defined per Submission and Completion
queue. The location of these registers is not defined in the spec; meaning
the offsets and stride may vary for different HC vendors. Establish the
stride, base address, and doorbell address offsets from vendor host driver
and program it.

Co-developed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: mcq: Allocate memory for MCQ mode

To read the bqueuedepth, the device descriptor is fetched in Single
Doorbell Mode. This allocated memory may not be enough for MCQ mode because
the number of tags supported in MCQ mode may be larger than in SDB mode.
Hence, release the memory allocated in SDB mode and allocate memory for MCQ
mode operation. Define the UFS hardware queue and Completion Queue Entry.

Co-developed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: mcq: Calculate queue depth

The UFS device defines the supported queuedepth by bqueuedepth which has a
max value of 256. The HC defines MAC (Max Active Commands) that defines
the max number of commands that in flight to the UFS device. Calculate and
configure the nutrs based on both these values.

Co-developed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: mcq: Configure resource regions

Define the MCQ resources and add support to ioremap the resource regions.

Co-developed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: mcq: Add support to allocate multiple queues

Multi-circular queue (MCQ) has been added in UFSHC v4.0 standard in
addition to the Single Doorbell mode. The MCQ mode supports multiple
submission and completion queues. Add support to allocate and configure
the queues. Add module parameters support to configure the queues.

Co-developed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Defer adding host to SCSI if MCQ is supported

If MCQ support is present, enabling it after MCQ support has been
configured would require reallocating tags and memory. It would also free
up the already allocated memory in Single Doorbell Mode. So defer invoking
scsi_add_host() until MCQ is configured.

Co-developed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Introduce multi-circular queue capability

Add support to check for MCQ capability in the UFSHC. Add a module
parameter to disable MCQ if needed.

Co-developed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Probe for EXT_IID support

Task Tag is limited to 8 bits and this restricts the number of active I/Os
to 255. In multi-circular queue mode, this may not be enough. The
specification provides EXT_IID which can be used to increase the number of
I/Os if the UFS device and UFSHC support it. This patch adds support to
probe for EXT_IID support in UFS device and UFSHC.

Co-developed-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Can Guo <quic_cang@quicinc.com>
Signed-off-by: Asutosh Das <quic_asutoshd@quicinc.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Avri Altman <avri.altman@wdc.com>
Reviewed-by: Manivannan Sadhasivam <mani@kernel.org>
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: core: Fix invisible definition compilation warning

In 'include/ufs/ufshcd.h' file, 'enum dma_data_direction' will be used,
which is defined in linux/dma-direction.h, however, this header file is not
included in ufshcd.h, thus causing the following compilation warning:

"warning: ‘enum dma_data_direction’ declared inside parameter list will not
be visible outside of this definition or declaration"

Fix this warning by including 'linux/dma-direction.h'.

Fixes: 6ff265fc5ef6 ("scsi: ufs: core: bsg: Add advanced RPMB support in ufs_bsg")
Reported-by: Xiaosen He <quic_xiaosenh@quicinc.com>
Reported-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add support for reinitializing the UFS device

Some platforms like Qcom, requires the UFS device to be reinitialized after
switching to maximum gear speed. So add support for that in UFS core by
introducing a new quirk (UFSHCD_CAP_REINIT_AFTER_MAX_GEAR_SWITCH) and doing
the reinitialization, if the quirk is enabled by the controller driver.

Suggested-by: Can Guo <quic_cang@quicinc.com>
Tested-by: Andrew Halaney <ahalaney@redhat.com> # Qdrive3/sa8540p-ride
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add reinit_notify() callback

reinit_notify() callback can be used by the UFS controller drivers to
perform changes required for UFSHCD reinit that can happen during max gear
switch.

Tested-by: Andrew Halaney <ahalaney@redhat.com> # Qdrive3/sa8540p-ride
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Remove redundant desc_size variable from hba

Always read the descriptor with QUERY_DESC_MAX_SIZE. According to the
spec, the device returns the actual size.

Signed-off-by: Arthur Simchaev <Arthur.Simchaev@wdc.com>
Reviewed-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Allow UFS host drivers to override the sg entry size

Modify the UFSHCD core to allow 'struct ufshcd_sg_entry' to be
variable-length. The default is the standard length, but variants can
override ufs_hba::sg_entry_size with a larger value if there are
vendor-specific fields following the standard ones.

This is needed to support inline encryption with ufs-exynos (FMP).

Cc: Eric Biggers <ebiggers@google.com>
Reviewed-by: Avri Altman <avri.altman@wdc.com>
Signed-off-by: Eric Biggers <ebiggers@google.com>
[ bvanassche: edited commit message and introduced CONFIG_SCSI_UFS_VARIABLE_SG_ENTRY_SIZE ]
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: bsg: Add advanced RPMB support in ufs_bsg

Add advanced RPMB support in ufs_bsg:

1. According to the UFS specification, only one RPMB operation can be
performed at any time. We can ensure this by using reserved slot and
its dev_cmd sync operation protection mechanism.

2. For Advanced RPMB, RPMB metadata is packaged in an EHS (Extra Header
Segment) of a command UPIU, and the corresponding reply EHS (from the
device) should also be returned to the user space. bsg_job->request
and bsg_job->reply allow us to pass and return EHS from/back to
userspace.

Compared to normal/legacy RPMB, the advantages of advanced RPMB are:

1. The data length in the Advanced RPMB data read/write command can be
larger than 4KB. For the legacy RPMB, the data length in a single RPMB
data transfer is 256 bytes.

2. All of the advanced RPMB operations will be a single command. For
legacy RPMB, take the read write-counter value as an example, you need
two commands (first SECURITY PROTOCOL OUT, then second SECURITY
PROTOCOL IN).

Signed-off-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Avri Altman <avri.altman@wdc.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Fix devfreq deadlocks

There is a lock inversion and rwsem read-lock recursion in the devfreq
target callback which can lead to deadlocks.

Specifically, ufshcd_devfreq_scale() already holds a clk_scaling_lock
read lock when toggling the write booster, which involves taking the
dev_cmd mutex before taking another clk_scaling_lock read lock.

This can lead to a deadlock if another thread:

1) tries to acquire the dev_cmd and clk_scaling locks in the correct
order, or

2) takes a clk_scaling write lock before the attempt to take the
clk_scaling read lock a second time.

Fix this by dropping the clk_scaling_lock before toggling the write booster
as was done before commit 0e9d4ca43ba8 ("scsi: ufs: Protect some contexts
from unexpected clock scaling").

While the devfreq callbacks are already serialised, add a second
serialising mutex to handle the unlikely case where a callback triggered
through the devfreq sysfs interface is racing with a request to disable
clock scaling through the UFS controller 'clkscale_enable' sysfs
attribute. This could otherwise lead to the write booster being left
disabled after having disabled clock scaling.

Also take the new mutex in ufshcd_clk_scaling_allow() to make sure that any
pending write booster update has completed on return.

Note that this currently only affects Qualcomm platforms since commit
87bd05016a64 ("scsi: ufs: core: Allow host driver to disable wb toggling
during clock scaling").

The lock inversion (i.e. 1 above) was reported by lockdep as:

======================================================
WARNING: possible circular locking dependency detected
6.1.0-next-20221216 #211 Not tainted
------------------------------------------------------
kworker/u16:2/71 is trying to acquire lock:
ffff076280ba98a0 (&hba->dev_cmd.lock){+.+.}-{3:3}, at: ufshcd_query_flag+0x50/0x1c0

but task is already holding lock:
ffff076280ba9cf0 (&hba->clk_scaling_lock){++++}-{3:3}, at: ufshcd_devfreq_scale+0x2b8/0x380

which lock already depends on the new lock.
[ +0.011606]
the existing dependency chain (in reverse order) is:

-> #1 (&hba->clk_scaling_lock){++++}-{3:3}:
lock_acquire+0x68/0x90
down_read+0x58/0x80
ufshcd_exec_dev_cmd+0x70/0x2c0
ufshcd_verify_dev_init+0x68/0x170
ufshcd_probe_hba+0x398/0x1180
ufshcd_async_scan+0x30/0x320
async_run_entry_fn+0x34/0x150
process_one_work+0x288/0x6c0
worker_thread+0x74/0x450
kthread+0x118/0x120
ret_from_fork+0x10/0x20

-> #0 (&hba->dev_cmd.lock){+.+.}-{3:3}:
__lock_acquire+0x12a0/0x2240
lock_acquire.part.0+0xcc/0x220
lock_acquire+0x68/0x90
__mutex_lock+0x98/0x430
mutex_lock_nested+0x2c/0x40
ufshcd_query_flag+0x50/0x1c0
ufshcd_query_flag_retry+0x64/0x100
ufshcd_wb_toggle+0x5c/0x120
ufshcd_devfreq_scale+0x2c4/0x380
ufshcd_devfreq_target+0xf4/0x230
devfreq_set_target+0x84/0x2f0
devfreq_update_target+0xc4/0xf0
devfreq_monitor+0x38/0x1f0
process_one_work+0x288/0x6c0
worker_thread+0x74/0x450
kthread+0x118/0x120
ret_from_fork+0x10/0x20

other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(&hba->clk_scaling_lock);
lock(&hba->dev_cmd.lock);
lock(&hba->clk_scaling_lock);
lock(&hba->dev_cmd.lock);

*** DEADLOCK ***

Fixes: 0e9d4ca43ba8 ("scsi: ufs: Protect some contexts from unexpected clock scaling")
Cc: stable@vger.kernel.org # 5.12
Cc: Can Guo <quic_cang@quicinc.com>
Tested-by: Andrew Halaney <ahalaney@redhat.com>
Signed-off-by: Johan Hovold <johan+linaro@kernel.org>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20230116161201.16923-1-johan+linaro@kernel.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Remove check_upiu_size() from ufshcd.h

Commit 68078d5cc1a5 ("[SCSI] ufs: Set fDeviceInit flag to initiate device
initialization") added check_upiu_size(), but no caller.

Cc: Dolev Raviv <draviv@codeaurora.org>
Link: https://lore.kernel.org/r/20221103055349epcms2p338f2550c2dd78d00231a83b24719a3d4@epcms2p3
Signed-off-by: Keoseong Park <keosung.park@samsung.com>
Reviewed-by: Avri Altman <avri.altman@wdc.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: Track system suspend / resume activity

Add a new boolean variable that tracks whether the system is suspending,
suspended or resuming. This information will be used in a later commit to
fix a deadlock between the SCSI error handler and the suspend code.

Reviewed-by: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Link: https://lore.kernel.org/r/20221018202958.1902564-9-bvanassche@acm.org
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Remove redundant function definitions from ufshcd.h

Remove Query-Request API function declarations from include/ufs/ufshcd.h
and move them to the ufs core private header.

Link: https://lore.kernel.org/r/1663761485-2532-1-git-send-email-Arthur.Simchaev@wdc.com
Reviewed-by: Bean Huo <beanhuo@micron.com>
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Arthur Simchaev <Arthur.Simchaev@wdc.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: wb: Add explicit flush sysfs attribute

There is the following quirk to bypass "WB Flush" in Write Booster.

- UFSHCI_QUIRK_SKIP_MANUAL_WB_FLUSH_CTRL

If this quirk is not set, there is no knob that can control "WB Flush".

There are three flags that control Write Booster Feature:

1. WB ON/OFF
2. WB Hibern Flush ON/OFF (implicitly)
3. WB Flush ON/OFF (explicit)

The sysfs attribute that controls the WB was implemented. (1)

In the case of "Hibern Flush", it is always good to turn on. Control may
not be required. (2)

Finally, "Flush" may be necessary because the Auto-Hibern8 is not supported
in a specific environment. So the sysfs attribute that controls this is
necessary. (3)

Link: https://lore.kernel.org/r/20220804075354epcms2p8c21c894b4e28840c5fc651875b7f435f@epcms2p8
Reviewed-by: Avri Altman <avri.altman@wdc.com>
Signed-off-by: Jinyoung Choi <j-young.choi@samsung.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Allow host driver to disable wb toggling during clock scaling

Mediatek UFS does not want to toggle write booster during clock scaling.
Permit host driver to disable wb toggling during clock scaling.

Introduce a flag UFSHCD_CAP_WB_WITH_CLK_SCALING to decouple WB and clock
scaling. UFSHCD_CAP_WB_WITH_CLK_SCALING is only valid when
UFSHCD_CAP_CLK_SCALING is set. Just like UFSHCD_CAP_HIBERN8_WITH_CLK_GATING
is valid only when UFSHCD_CAP_CLK_GATING set.

Set UFSHCD_CAP_WB_WITH_CLK_SCALING for qcom to compatible legacy design at
the same time.

Link: https://lore.kernel.org/r/20220804025422.18803-1-peter.wang@mediatek.com
Reviewed-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Peter Wang <peter.wang@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Use local_clock() for debugging timestamps

CLOCK_MONOTONIC is not advanced when the system is in suspend. This becomes
problematic when debugging issues related to suspend-resume: the timestamps
printed by ufshcd_print_trs can not be correlated with dmesg entries, which
are timestamped with local_clock().

Change the used clock to local_clock() for the informational timestamp
variables and adds mirroring *_local_clock instances for variables used in
subsequent derevations (to not change the semantics of those derevations).

Link: https://lore.kernel.org/r/20220804065019.v5.1.I699244ea7efbd326a34a6dfd9b5a31e78400cf68@changeid
Acked-by: Stanley Chu <stanley.chu@mediatek.com>
Acked-by: Avri Altman <avri.altman@wdc.com>
Signed-off-by: Daniil Lunev <dlunev@chromium.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: ufshcd: Constify pointed data

For code safety, constify arrays and pointers to data which is not
modified.

Link: https://lore.kernel.org/r/20220623102432.108059-4-krzysztof.kozlowski@linaro.org
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski@linaro.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: Export regulator functions

Export below regulator functions to allow vendors to
customize regulator configuration in their own platforms.

int ufshcd_populate_vreg(struct device *dev, const char *name,
struct ufs_vreg **out_vreg);
int ufshcd_get_vreg(struct device *dev, struct ufs_vreg *vreg);

Link: https://lore.kernel.org/r/20220616053725.5681-10-stanley.chu@mediatek.com
Signed-off-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: Export ufshcd_uic_change_pwr_mode()

Export ufshcd_uic_change_pwr_mode() to allow vendors to use it for
SoC-specific power mode change design limitations.

Link: https://lore.kernel.org/r/20220616053725.5681-2-stanley.chu@mediatek.com
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Stanley Chu <stanley.chu@mediatek.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add UFSHCD_QUIRK_HIBERN_FASTAUTO

Add UFSHCD_QUIRK_HIBERN_FASTAUTO quirk for host controllers which supports
auto-hibernate the capability but only FASTAUTO mode.

Link: https://lore.kernel.org/r/20220603110524.1997825-4-yoshihiro.shimoda.uh@renesas.com
Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: core: Add UFSHCD_QUIRK_BROKEN_64BIT_ADDRESS

Add UFSHCD_QUIRK_BROKEN_64BIT_ADDRESS for host controllers which do not
support 64-bit addressing.

Link: https://lore.kernel.org/r/20220603110524.1997825-3-yoshihiro.shimoda.uh@renesas.com
Signed-off-by: Yoshihiro Shimoda <yoshihiro.shimoda.uh@renesas.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>

scsi: ufs: Split the drivers/scsi/ufs directory

Split the drivers/scsi/ufs directory into 'core' and 'host' directories
under the drivers/ufs/ directory. Move shared header files into the
include/ufs/ directory. This separation makes it clear which header files
UFS drivers are allowed to include (include/ufs/*.h) and which header files
UFS drivers are not allowed to include (drivers/ufs/core/*.h).

Update the MAINTAINERS file. Add myself as a UFS reviewer.

Link: https://lore.kernel.org/r/20220511212552.655341-1-bvanassche@acm.org
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Avri Altman <avri.altman@wdc.com>
Cc: Bean Huo <beanhuo@micron.com>
Cc: Bjorn Andersson <bjorn.andersson@linaro.org>
Cc: Keoseong Park <keosung.park@samsung.com>
Tested-by: Bean Huo <beanhuo@micron.com>
Tested-by: Adrian Hunter <adrian.hunter@intel.com>
Reviewed-by: Bean Huo <beanhuo@micron.com>
Acked-by: Avri Altman <avri.altman@wdc.com>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>