[PATCH v2] scsi: ufs: Fix deadlock issue in ufshcd_wait_for_doorbell_clr()

[PATCH v2] scsi: ufs: Fix deadlock issue in ufshcd_wait_for_doorbell_clr()

[Bean Huo]

From: Bean Huo <beanhuo@micron.com>

Call shost_for_each_device() with holding host->host_lock will cause
a deadlock situation, which will cause the system to stall (the log
as follow). Fix this issue by using __shost_for_each_device() in
ufshcd_pending_cmds().

stalls on CPUs/tasks:
all trace:
__switch_to+0x120/0x170
0xffff800011643998
ask dump for CPU 5:
ask:kworker/u16:2   state:R  running task     stack:    0 pid:   80 ppid:     2 flags:0x0000000a
orkqueue: events_unbound async_run_entry_fn
all trace:
__switch_to+0x120/0x170
0x0
ask dump for CPU 6:
ask:kworker/u16:6   state:R  running task     stack:    0 pid:  164 ppid:     2 flags:0x0000000a
orkqueue: events_unbound async_run_entry_fn
all trace:
__switch_to+0x120/0x170
0xffff54e7c4429f80
ask dump for CPU 7:
ask:kworker/u16:4   state:R  running task     stack:    0 pid:  153 ppid:     2 flags:0x0000000a
orkqueue: events_unbound async_run_entry_fn
all trace:
__switch_to+0x120/0x170
blk_mq_run_hw_queue+0x34/0x110
blk_mq_sched_insert_request+0xb0/0x120
blk_execute_rq_nowait+0x68/0x88
blk_execute_rq+0x4c/0xd8
__scsi_execute+0xec/0x1d0
scsi_vpd_inquiry+0x84/0xf0
scsi_get_vpd_buf+0x34/0xb8
scsi_attach_vpd+0x34/0x140
scsi_probe_and_add_lun+0xa6c/0xab8
__scsi_scan_target+0x438/0x4f8
scsi_scan_channel+0x6c/0xa8
scsi_scan_host_selected+0xf0/0x150
do_scsi_scan_host+0x88/0x90
scsi_scan_host+0x1b4/0x1d0
ufshcd_async_scan+0x248/0x310
async_run_entry_fn+0x30/0x178
process_one_work+0x1e8/0x368
worker_thread+0x40/0x478
kthread+0x174/0x180
ret_from_fork+0x10/0x20

Fixes: 8d077ede48c1 ("scsi: ufs: Optimize the command queueing code")
Reported-by: YongQin Liu <yongqin.liu@linaro.org>
Reported-by: Amit Pundir <amit.pundir@linaro.org>
Tested-by: John Stultz <john.stultz@linaro.org>
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Signed-off-by: Bean Huo <beanhuo@micron.com>
---
 drivers/scsi/ufs/ufshcd.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index 6dd517267f1b..fbebb8309ef7 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -1082,7 +1082,8 @@ static u32 ufshcd_pending_cmds(struct ufs_hba *hba)
 	struct scsi_device *sdev;
 	u32 pending = 0;
 
-	shost_for_each_device(sdev, hba->host)
+	lockdep_assert_held(hba->host->host_lock);
+	__shost_for_each_device(sdev, hba->host)
 		pending += sbitmap_weight(&sdev->budget_map);
 
 	return pending;
-- 
2.25.1

Re: [PATCH v2] scsi: ufs: Fix deadlock issue in ufshcd_wait_for_doorbell_clr()

[Bjorn Andersson]

On Tue 14 Dec 04:05 PST 2021, Bean Huo wrote:

> From: Bean Huo <beanhuo@micron.com>
> 
> Call shost_for_each_device() with holding host->host_lock will cause
> a deadlock situation, which will cause the system to stall (the log
> as follow). Fix this issue by using __shost_for_each_device() in
> ufshcd_pending_cmds().
> 
> stalls on CPUs/tasks:
> all trace:
> __switch_to+0x120/0x170
> 0xffff800011643998
> ask dump for CPU 5:
> ask:kworker/u16:2   state:R  running task     stack:    0 pid:   80 ppid:     2 flags:0x0000000a
> orkqueue: events_unbound async_run_entry_fn
> all trace:
> __switch_to+0x120/0x170
> 0x0
> ask dump for CPU 6:
> ask:kworker/u16:6   state:R  running task     stack:    0 pid:  164 ppid:     2 flags:0x0000000a
> orkqueue: events_unbound async_run_entry_fn
> all trace:
> __switch_to+0x120/0x170
> 0xffff54e7c4429f80
> ask dump for CPU 7:
> ask:kworker/u16:4   state:R  running task     stack:    0 pid:  153 ppid:     2 flags:0x0000000a
> orkqueue: events_unbound async_run_entry_fn
> all trace:
> __switch_to+0x120/0x170
> blk_mq_run_hw_queue+0x34/0x110
> blk_mq_sched_insert_request+0xb0/0x120
> blk_execute_rq_nowait+0x68/0x88
> blk_execute_rq+0x4c/0xd8
> __scsi_execute+0xec/0x1d0
> scsi_vpd_inquiry+0x84/0xf0
> scsi_get_vpd_buf+0x34/0xb8
> scsi_attach_vpd+0x34/0x140
> scsi_probe_and_add_lun+0xa6c/0xab8
> __scsi_scan_target+0x438/0x4f8
> scsi_scan_channel+0x6c/0xa8
> scsi_scan_host_selected+0xf0/0x150
> do_scsi_scan_host+0x88/0x90
> scsi_scan_host+0x1b4/0x1d0
> ufshcd_async_scan+0x248/0x310
> async_run_entry_fn+0x30/0x178
> process_one_work+0x1e8/0x368
> worker_thread+0x40/0x478
> kthread+0x174/0x180
> ret_from_fork+0x10/0x20
> 
> Fixes: 8d077ede48c1 ("scsi: ufs: Optimize the command queueing code")
> Reported-by: YongQin Liu <yongqin.liu@linaro.org>
> Reported-by: Amit Pundir <amit.pundir@linaro.org>
> Tested-by: John Stultz <john.stultz@linaro.org>
> Signed-off-by: Bart Van Assche <bvanassche@acm.org>
> Signed-off-by: Bean Huo <beanhuo@micron.com>

Reviewed-by: Bjorn Andersson <bjorn.andersson@linaro.org>
Tested-by: Bjorn Andersson <bjorn.andersson@linaro.org>

Regards,
Bjorn

> ---
>  drivers/scsi/ufs/ufshcd.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
> index 6dd517267f1b..fbebb8309ef7 100644
> --- a/drivers/scsi/ufs/ufshcd.c
> +++ b/drivers/scsi/ufs/ufshcd.c
> @@ -1082,7 +1082,8 @@ static u32 ufshcd_pending_cmds(struct ufs_hba *hba)
>  	struct scsi_device *sdev;
>  	u32 pending = 0;
>  
> -	shost_for_each_device(sdev, hba->host)
> +	lockdep_assert_held(hba->host->host_lock);
> +	__shost_for_each_device(sdev, hba->host)
>  		pending += sbitmap_weight(&sdev->budget_map);
>  
>  	return pending;
> -- 
> 2.25.1
> 
> 

Re: [PATCH v2] scsi: ufs: Fix deadlock issue in ufshcd_wait_for_doorbell_clr()

[Martin K. Petersen]

Bean,

> Call shost_for_each_device() with holding host->host_lock will cause a
> deadlock situation, which will cause the system to stall (the log as
> follow). Fix this issue by using __shost_for_each_device() in
> ufshcd_pending_cmds().

Applied to 5.17/scsi-staging, thanks!

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH v2] scsi: ufs: Fix deadlock issue in ufshcd_wait_for_doorbell_clr()

[Martin K. Petersen]

On Tue, 14 Dec 2021 13:05:37 +0100, Bean Huo wrote:

> From: Bean Huo <beanhuo@micron.com>
> 
> Call shost_for_each_device() with holding host->host_lock will cause
> a deadlock situation, which will cause the system to stall (the log
> as follow). Fix this issue by using __shost_for_each_device() in
> ufshcd_pending_cmds().
> 
> [...]

Applied to 5.17/scsi-queue, thanks!

[1/1] scsi: ufs: Fix deadlock issue in ufshcd_wait_for_doorbell_clr()
      https://git.kernel.org/mkp/scsi/c/99c66a8868e3

-- 
Martin K. Petersen	Oracle Linux Engineering