[PATCH 0/4] Revert "Call blk_mq_free_tag_set() earlier"

[PATCH 0/4] Revert "Call blk_mq_free_tag_set() earlier"

[Bart Van Assche]

Hi Martin,

Since a device, target or host reference may be held when scsi_remove_host()
or scsi_remove_target() is called and since te patch series "Call
blk_mq_free_tag_set() earlier" makes these functions wait until all references
are gone, that patch series may trigger a deadlock. Hence this request to
revert the patch series "Call blk_mq_free_tag_set() earlier".

Thanks,

Bart.

Bart Van Assche (4):
  scsi: core: Revert "Call blk_mq_free_tag_set() earlier"
  scsi: core: Revert "Simplify LLD module reference counting"
  scsi: core: Revert "Make sure that hosts outlive targets"
  scsi: core: Revert "Make sure that targets outlive devices"

 drivers/scsi/hosts.c       | 18 +++++-------------
 drivers/scsi/scsi.c        |  9 +++------
 drivers/scsi/scsi_scan.c   |  9 ---------
 drivers/scsi/scsi_sysfs.c  | 29 ++++++++++++-----------------
 include/scsi/scsi_device.h |  2 --
 include/scsi/scsi_host.h   |  3 ---
 6 files changed, 20 insertions(+), 50 deletions(-)

[PATCH 1/4] scsi: core: Revert "Call blk_mq_free_tag_set() earlier"

[Bart Van Assche]

Revert the patch series "Call blk_mq_free_tag_set() earlier" because it
introduces a deadlock if the scsi_remove_host() caller holds a reference
on a device, target or host.

Reported-by: syzbot+bafeb834708b1bb750bc@syzkaller.appspotmail.com
Fixes: f323896fe6fa ("scsi: core: Call blk_mq_free_tag_set() earlier")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/scsi/hosts.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/scsi/hosts.c b/drivers/scsi/hosts.c
index 0738238ed6cc..20c1f5420ba6 100644
--- a/drivers/scsi/hosts.c
+++ b/drivers/scsi/hosts.c
@@ -197,8 +197,6 @@ void scsi_remove_host(struct Scsi_Host *shost)
 	 * the dependent SCSI targets and devices are gone before returning.
 	 */
 	wait_event(shost->targets_wq, atomic_read(&shost->target_count) == 0);
-
-	scsi_mq_destroy_tags(shost);
 }
 EXPORT_SYMBOL(scsi_remove_host);
 
@@ -309,8 +307,8 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,
 	return error;
 
 	/*
-	 * Any resources associated with the SCSI host in this function except
-	 * the tag set will be freed by scsi_host_dev_release().
+	 * Any host allocation in this function will be freed in
+	 * scsi_host_dev_release().
 	 */
  out_del_dev:
 	device_del(&shost->shost_dev);
@@ -326,7 +324,6 @@ int scsi_add_host_with_dma(struct Scsi_Host *shost, struct device *dev,
 	pm_runtime_disable(&shost->shost_gendev);
 	pm_runtime_set_suspended(&shost->shost_gendev);
 	pm_runtime_put_noidle(&shost->shost_gendev);
-	scsi_mq_destroy_tags(shost);
  fail:
 	return error;
 }
@@ -360,6 +357,9 @@ static void scsi_host_dev_release(struct device *dev)
 		kfree(dev_name(&shost->shost_dev));
 	}
 
+	if (shost->tag_set.tags)
+		scsi_mq_destroy_tags(shost);
+
 	kfree(shost->shost_data);
 
 	ida_free(&host_index_ida, shost->host_no);

[PATCH 2/4] scsi: core: Revert "Simplify LLD module reference counting"

[Bart Van Assche]

Revert the patch series "Call blk_mq_free_tag_set() earlier" because it
introduces a deadlock if the scsi_remove_host() caller holds a reference
on a device, target or host.

Reported-by: syzbot+bafeb834708b1bb750bc@syzkaller.appspotmail.com
Fixes: 1a9283782df2 ("scsi: core: Simplify LLD module reference counting")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/scsi/scsi.c       | 9 +++------
 drivers/scsi/scsi_sysfs.c | 9 +++++++++
 2 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/drivers/scsi/scsi.c b/drivers/scsi/scsi.c
index 086ec5b5862d..c59eac7a32f2 100644
--- a/drivers/scsi/scsi.c
+++ b/drivers/scsi/scsi.c
@@ -586,13 +586,10 @@ EXPORT_SYMBOL(scsi_device_get);
  */
 void scsi_device_put(struct scsi_device *sdev)
 {
-	/*
-	 * Decreasing the module reference count before the device reference
-	 * count is safe since scsi_remove_host() only returns after all
-	 * devices have been removed.
-	 */
-	module_put(sdev->host->hostt->module);
+	struct module *mod = sdev->host->hostt->module;
+
 	put_device(&sdev->sdev_gendev);
+	module_put(mod);
 }
 EXPORT_SYMBOL(scsi_device_put);
 
diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c
index 9dad2fd5297f..282b32781e8c 100644
--- a/drivers/scsi/scsi_sysfs.c
+++ b/drivers/scsi/scsi_sysfs.c
@@ -452,6 +452,9 @@ static void scsi_device_dev_release_usercontext(struct work_struct *work)
 	struct scsi_vpd *vpd_pg0 = NULL, *vpd_pg89 = NULL;
 	struct scsi_vpd *vpd_pgb0 = NULL, *vpd_pgb1 = NULL, *vpd_pgb2 = NULL;
 	unsigned long flags;
+	struct module *mod;
+
+	mod = sdev->host->hostt->module;
 
 	scsi_dh_release_device(sdev);
 
@@ -518,11 +521,17 @@ static void scsi_device_dev_release_usercontext(struct work_struct *work)
 
 	if (parent)
 		put_device(parent);
+	module_put(mod);
 }
 
 static void scsi_device_dev_release(struct device *dev)
 {
 	struct scsi_device *sdp = to_scsi_device(dev);
+
+	/* Set module pointer as NULL in case of module unloading */
+	if (!try_module_get(sdp->host->hostt->module))
+		sdp->host->hostt->module = NULL;
+
 	execute_in_process_context(scsi_device_dev_release_usercontext,
 				   &sdp->ew);
 }

[PATCH 3/4] scsi: core: Revert "Make sure that hosts outlive targets"

[Bart Van Assche]

Revert the patch series "Call blk_mq_free_tag_set() earlier" because it
introduces a deadlock if the scsi_remove_host() caller holds a reference
on a device, target or host.

Reported-by: syzbot+bafeb834708b1bb750bc@syzkaller.appspotmail.com
Fixes: 16728aaba62e ("scsi: core: Make sure that hosts outlive targets")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/scsi/hosts.c     | 8 --------
 drivers/scsi/scsi_scan.c | 7 -------
 include/scsi/scsi_host.h | 3 ---
 3 files changed, 18 deletions(-)

diff --git a/drivers/scsi/hosts.c b/drivers/scsi/hosts.c
index 20c1f5420ba6..26bf3b153595 100644
--- a/drivers/scsi/hosts.c
+++ b/drivers/scsi/hosts.c
@@ -190,13 +190,6 @@ void scsi_remove_host(struct Scsi_Host *shost)
 	transport_unregister_device(&shost->shost_gendev);
 	device_unregister(&shost->shost_dev);
 	device_del(&shost->shost_gendev);
-
-	/*
-	 * After scsi_remove_host() has returned the scsi LLD module can be
-	 * unloaded and/or the host resources can be released. Hence wait until
-	 * the dependent SCSI targets and devices are gone before returning.
-	 */
-	wait_event(shost->targets_wq, atomic_read(&shost->target_count) == 0);
 }
 EXPORT_SYMBOL(scsi_remove_host);
 
@@ -406,7 +399,6 @@ struct Scsi_Host *scsi_host_alloc(struct scsi_host_template *sht, int privsize)
 	INIT_LIST_HEAD(&shost->starved_list);
 	init_waitqueue_head(&shost->host_wait);
 	mutex_init(&shost->scan_mutex);
-	init_waitqueue_head(&shost->targets_wq);
 
 	index = ida_alloc(&host_index_ida, GFP_KERNEL);
 	if (index < 0) {
diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
index ac6059702d13..4c1efd6a3b0c 100644
--- a/drivers/scsi/scsi_scan.c
+++ b/drivers/scsi/scsi_scan.c
@@ -406,14 +406,9 @@ static void scsi_target_destroy(struct scsi_target *starget)
 static void scsi_target_dev_release(struct device *dev)
 {
 	struct device *parent = dev->parent;
-	struct Scsi_Host *shost = dev_to_shost(parent);
 	struct scsi_target *starget = to_scsi_target(dev);
 
 	kfree(starget);
-
-	if (atomic_dec_return(&shost->target_count) == 0)
-		wake_up(&shost->targets_wq);
-
 	put_device(parent);
 }
 
@@ -528,8 +523,6 @@ static struct scsi_target *scsi_alloc_target(struct device *parent,
 	starget->max_target_blocked = SCSI_DEFAULT_TARGET_BLOCKED;
 	init_waitqueue_head(&starget->sdev_wq);
 
-	atomic_inc(&shost->target_count);
-
  retry:
 	spin_lock_irqsave(shost->host_lock, flags);
 
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index aa7b7496c93a..b6e41ee3d566 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -690,9 +690,6 @@ struct Scsi_Host {
 	/* ldm bits */
 	struct device		shost_gendev, shost_dev;
 
-	atomic_t		target_count;
-	wait_queue_head_t	targets_wq;
-
 	/*
 	 * Points to the transport data (if any) which is allocated
 	 * separately

[PATCH 4/4] scsi: core: Revert "Make sure that targets outlive devices"

[Bart Van Assche]

Revert the patch series "Call blk_mq_free_tag_set() earlier" because it
introduces a deadlock if the scsi_remove_host() caller holds a reference
on a device, target or host.

Reported-by: syzbot+bafeb834708b1bb750bc@syzkaller.appspotmail.com
Fixes: fe442604199e ("scsi: core: Make sure that targets outlive devices")
Signed-off-by: Bart Van Assche <bvanassche@acm.org>
---
 drivers/scsi/scsi_scan.c   |  2 --
 drivers/scsi/scsi_sysfs.c  | 20 +++-----------------
 include/scsi/scsi_device.h |  2 --
 3 files changed, 3 insertions(+), 21 deletions(-)

diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
index 4c1efd6a3b0c..91ac901a6682 100644
--- a/drivers/scsi/scsi_scan.c
+++ b/drivers/scsi/scsi_scan.c
@@ -521,8 +521,6 @@ static struct scsi_target *scsi_alloc_target(struct device *parent,
 	starget->state = STARGET_CREATED;
 	starget->scsi_level = SCSI_2;
 	starget->max_target_blocked = SCSI_DEFAULT_TARGET_BLOCKED;
-	init_waitqueue_head(&starget->sdev_wq);
-
  retry:
 	spin_lock_irqsave(shost->host_lock, flags);
 
diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c
index 282b32781e8c..aa70d9282161 100644
--- a/drivers/scsi/scsi_sysfs.c
+++ b/drivers/scsi/scsi_sysfs.c
@@ -443,9 +443,7 @@ static void scsi_device_cls_release(struct device *class_dev)
 
 static void scsi_device_dev_release_usercontext(struct work_struct *work)
 {
-	struct scsi_device *sdev = container_of(work, struct scsi_device,
-						ew.work);
-	struct scsi_target *starget = sdev->sdev_target;
+	struct scsi_device *sdev;
 	struct device *parent;
 	struct list_head *this, *tmp;
 	struct scsi_vpd *vpd_pg80 = NULL, *vpd_pg83 = NULL;
@@ -454,6 +452,8 @@ static void scsi_device_dev_release_usercontext(struct work_struct *work)
 	unsigned long flags;
 	struct module *mod;
 
+	sdev = container_of(work, struct scsi_device, ew.work);
+
 	mod = sdev->host->hostt->module;
 
 	scsi_dh_release_device(sdev);
@@ -516,9 +516,6 @@ static void scsi_device_dev_release_usercontext(struct work_struct *work)
 	kfree(sdev->inquiry);
 	kfree(sdev);
 
-	if (starget && atomic_dec_return(&starget->sdev_count) == 0)
-		wake_up(&starget->sdev_wq);
-
 	if (parent)
 		put_device(parent);
 	module_put(mod);
@@ -1538,14 +1535,6 @@ static void __scsi_remove_target(struct scsi_target *starget)
 		goto restart;
 	}
 	spin_unlock_irqrestore(shost->host_lock, flags);
-
-	/*
-	 * After scsi_remove_target() returns its caller can remove resources
-	 * associated with @starget, e.g. an rport or session. Wait until all
-	 * devices associated with @starget have been removed to prevent that
-	 * a SCSI error handling callback function triggers a use-after-free.
-	 */
-	wait_event(starget->sdev_wq, atomic_read(&starget->sdev_count) == 0);
 }
 
 /**
@@ -1656,9 +1645,6 @@ void scsi_sysfs_device_initialize(struct scsi_device *sdev)
 	list_add_tail(&sdev->same_target_siblings, &starget->devices);
 	list_add_tail(&sdev->siblings, &shost->__devices);
 	spin_unlock_irqrestore(shost->host_lock, flags);
-
-	atomic_inc(&starget->sdev_count);
-
 	/*
 	 * device can now only be removed via __scsi_remove_device() so hold
 	 * the target.  Target will be held in CREATED state until something
diff --git a/include/scsi/scsi_device.h b/include/scsi/scsi_device.h
index 3113471ca375..2493bd65351a 100644
--- a/include/scsi/scsi_device.h
+++ b/include/scsi/scsi_device.h
@@ -309,8 +309,6 @@ struct scsi_target {
 	struct list_head	devices;
 	struct device		dev;
 	struct kref		reap_ref; /* last put renders target invisible */
-	atomic_t		sdev_count;
-	wait_queue_head_t	sdev_wq;
 	unsigned int		channel;
 	unsigned int		id; /* target id ... replace
 				     * scsi_device.id eventually */

Re: [PATCH 0/4] Revert "Call blk_mq_free_tag_set() earlier"

[Ming Lei]

Hi Bart,

On Mon, Aug 22, 2022 at 6:05 AM Bart Van Assche <bvanassche@acm.org> wrote:
>
> Hi Martin,
>
> Since a device, target or host reference may be held when scsi_remove_host()
> or scsi_remove_target() is called and since te patch series "Call
> blk_mq_free_tag_set() earlier" makes these functions wait until all references
> are gone, that patch series may trigger a deadlock. Hence this request to
> revert the patch series "Call blk_mq_free_tag_set() earlier".

Care to share the deadlock details? Such as dmesg log or theory behind.

Thanks,
Ming

Re: [PATCH 0/4] Revert "Call blk_mq_free_tag_set() earlier"

[Bart Van Assche]

On 8/29/22 02:14, Ming Lei wrote:
> On Mon, Aug 22, 2022 at 6:05 AM Bart Van Assche <bvanassche@acm.org> wrote:
>> Since a device, target or host reference may be held when scsi_remove_host()
>> or scsi_remove_target() is called and since te patch series "Call
>> blk_mq_free_tag_set() earlier" makes these functions wait until all references
>> are gone, that patch series may trigger a deadlock. Hence this request to
>> revert the patch series "Call blk_mq_free_tag_set() earlier".
> 
> Care to share the deadlock details? Such as dmesg log or theory behind.

Hi Ming,

Details of two different deadlock scenarios are available here:
* [syzbot] INFO: task hung in scsi_remove_host 
(https://lore.kernel.org/all/000000000000b5187d05e6c08086@google.com/).
* https://lore.kernel.org/all/Yv%2FMKymRC9O04Nqu@google.com/. The link 
[2] in this email includes a call trace and instructions for reproducing 
this issue. My root cause analysis of this deadlock is available here: 
https://lore.kernel.org/all/27d0dde8-344c-1dd0-cc26-0e10c4e1f296@acm.org/.

Thanks,

Bart.

Re: [PATCH 0/4] Revert "Call blk_mq_free_tag_set() earlier"

[Martin K. Petersen]

On Sun, 21 Aug 2022 15:04:58 -0700, Bart Van Assche wrote:

> Since a device, target or host reference may be held when scsi_remove_host()
> or scsi_remove_target() is called and since te patch series "Call
> blk_mq_free_tag_set() earlier" makes these functions wait until all references
> are gone, that patch series may trigger a deadlock. Hence this request to
> revert the patch series "Call blk_mq_free_tag_set() earlier".
> 
> Thanks,
> 
> [...]

Applied to 6.0/scsi-fixes, thanks!

[1/4] scsi: core: Revert "Call blk_mq_free_tag_set() earlier"
      https://git.kernel.org/mkp/scsi/c/2b36209ca818
[2/4] scsi: core: Revert "Simplify LLD module reference counting"
      https://git.kernel.org/mkp/scsi/c/70e8d057bef5
[3/4] scsi: core: Revert "Make sure that hosts outlive targets"
      https://git.kernel.org/mkp/scsi/c/d94b2d00f7bf
[4/4] scsi: core: Revert "Make sure that targets outlive devices"
      https://git.kernel.org/mkp/scsi/c/f782201ebc2b

-- 
Martin K. Petersen	Oracle Linux Engineering