* [PATCH v2 1/1] scsi: Fix racing between dev init and dev reset
@ 2022-03-08 2:07 Alice Chao
2022-03-08 23:23 ` Miles Chen
0 siblings, 1 reply; 4+ messages in thread
From: Alice Chao @ 2022-03-08 2:07 UTC (permalink / raw)
To: jejb, martin.petersen, matthias.bgg, linux-scsi, linux-kernel,
linux-arm-kernel, linux-mediatek
Cc: stanley.chu, peter.wang, chun-hung.wu, alice.chao, jonathan.hsu,
powen.kao, cc.chou, chaotian.jing, jiajie.hao, qilin.tan,
lin.gui, yanxu.wei, wsd_upstream
Device reset thread uses kobject_uevent_env() to get kobj.parent
after scsi_evt_emit(), and it races with device init thread which
calls device_add() to create kobj.parent before kobject_uevent_env().
Device reset call trace:
fill_kobj_path
kobject_get_path
kobject_uevent_env
scsi_evt_emit <- add wait_event()
scsi_evt_thread
Device init call trace:
fill_kobj_path
kobject_get_path
kobject_uevent_env
device_add <- create kobj.parent
scsi_target_add
scsi_sysfs_add_sdev
scsi_add_lun
scsi_probe_and_add_lun
These two jobs are scheduled asynchronously, we can't guaranteed that
kobj.parent will be created in device init thread before device reset
thread calls kobj_get_path().
To resolve the racing issue between device init thread and device
reset thread, we use wait_event() in scsi_evt_emit() to wait for
device_add() to complete the creation of kobj.parent.
Signed-off-by: Alice Chao <alice.chao@mediatek.com>
---
drivers/scsi/scsi_lib.c | 1 +
drivers/scsi/scsi_scan.c | 1 +
2 files changed, 2 insertions(+)
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 0a70aa763a96..abf9a71ed77c 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -2461,6 +2461,7 @@ static void scsi_evt_emit(struct scsi_device *sdev, struct scsi_event *evt)
break;
case SDEV_EVT_POWER_ON_RESET_OCCURRED:
envp[idx++] = "SDEV_UA=POWER_ON_RESET_OCCURRED";
+ wait_event(sdev->host->host_wait, sdev->sdev_gendev.kobj.parent != NULL);
break;
default:
/* do nothing */
diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
index f4e6c68ac99e..431f229ac435 100644
--- a/drivers/scsi/scsi_scan.c
+++ b/drivers/scsi/scsi_scan.c
@@ -1904,6 +1904,7 @@ static void do_scsi_scan_host(struct Scsi_Host *shost)
} else {
scsi_scan_host_selected(shost, SCAN_WILD_CARD, SCAN_WILD_CARD,
SCAN_WILD_CARD, 0);
+ wake_up(&shost->host_wait);
}
}
--
2.18.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH v2 1/1] scsi: Fix racing between dev init and dev reset
2022-03-08 2:07 [PATCH v2 1/1] scsi: Fix racing between dev init and dev reset Alice Chao
@ 2022-03-08 23:23 ` Miles Chen
2022-04-13 6:53 ` Alice Chao
0 siblings, 1 reply; 4+ messages in thread
From: Miles Chen @ 2022-03-08 23:23 UTC (permalink / raw)
To: alice.chao
Cc: cc.chou, chaotian.jing, chun-hung.wu, jejb, jiajie.hao,
jonathan.hsu, lin.gui, linux-arm-kernel, linux-kernel,
linux-mediatek, linux-scsi, martin.petersen, matthias.bgg,
peter.wang, powen.kao, qilin.tan, stanley.chu, wsd_upstream,
yanxu.wei
Hi Alice,
Thanks for your patch. I have some questions.
> Device reset thread uses kobject_uevent_env() to get kobj.parent
> after scsi_evt_emit(), and it races with device init thread which
> calls device_add() to create kobj.parent before kobject_uevent_env().
>
> Device reset call trace:
> fill_kobj_path
> kobject_get_path
> kobject_uevent_env
> scsi_evt_emit <- add wait_event()
> scsi_evt_thread
>
> Device init call trace:
> fill_kobj_path
> kobject_get_path
> kobject_uevent_env
> device_add <- create kobj.parent
> scsi_target_add
> scsi_sysfs_add_sdev
> scsi_add_lun
> scsi_probe_and_add_lun
These are backtraces, and the things above looks like your solution, not
the problem.
It's better to describe the problem first and solution later:
Device init call: Device reset call:
scsi_probe_and_add_lun() scsi_evt_thread()
scsi_add_lun()
scsi_sysfs_add_sdev() scsi_evt_emit()
scsi_target_add() kobject_uevent_env()
device_add() kobject_get_path() // something wrong?
fill_kobj_path()
get_device_parent() // I see a get, not create parent kobj, could you show me that point?
kobject_uevent()
kobject_uevent_env()
kobject_get_path()
fill_kobj_path()
>
> These two jobs are scheduled asynchronously, we can't guaranteed that
> kobj.parent will be created in device init thread before device reset
> thread calls kobj_get_path().
What happens if device reset thread calls kobj_get_path() before kobj.parent's
creation? a crash or something?
>
> To resolve the racing issue between device init thread and device
> reset thread, we use wait_event() in scsi_evt_emit() to wait for
> device_add() to complete the creation of kobj.parent.
>
> Signed-off-by: Alice Chao <alice.chao@mediatek.com>
When we send patch N+1, we have to put the change log between patch v1 to vN
under a "---", so others can understand the changes between patch N and N+1.
e.g.,
---
Change since v1
- remove Change-Id
> ---
> drivers/scsi/scsi_lib.c | 1 +
> drivers/scsi/scsi_scan.c | 1 +
> 2 files changed, 2 insertions(+)
>
> diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> index 0a70aa763a96..abf9a71ed77c 100644
> --- a/drivers/scsi/scsi_lib.c
> +++ b/drivers/scsi/scsi_lib.c
> @@ -2461,6 +2461,7 @@ static void scsi_evt_emit(struct scsi_device *sdev, struct scsi_event *evt)
> break;
> case SDEV_EVT_POWER_ON_RESET_OCCURRED:
> envp[idx++] = "SDEV_UA=POWER_ON_RESET_OCCURRED";
> + wait_event(sdev->host->host_wait, sdev->sdev_gendev.kobj.parent != NULL);
> break;
> default:
> /* do nothing */
> diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
> index f4e6c68ac99e..431f229ac435 100644
> --- a/drivers/scsi/scsi_scan.c
> +++ b/drivers/scsi/scsi_scan.c
> @@ -1904,6 +1904,7 @@ static void do_scsi_scan_host(struct Scsi_Host *shost)
> } else {
> scsi_scan_host_selected(shost, SCAN_WILD_CARD, SCAN_WILD_CARD,
> SCAN_WILD_CARD, 0);
> + wake_up(&shost->host_wait);
do_scsi_scan_host() is not in the call trace above, could you show the relationship
of do_scsi_scan_host in the call flow?
thanks,
Miles
> }
> }
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v2 1/1] scsi: Fix racing between dev init and dev reset
2022-03-08 23:23 ` Miles Chen
@ 2022-04-13 6:53 ` Alice Chao
2022-04-14 8:18 ` Miles Chen
0 siblings, 1 reply; 4+ messages in thread
From: Alice Chao @ 2022-04-13 6:53 UTC (permalink / raw)
To: Miles Chen
Cc: CC Chou (周志杰),
Chaotian Jing (井朝天),
Chun-Hung Wu (巫駿宏),
jejb, Jiajie Hao (郝加节),
jonathan.hsu, Lin Gui (桂林),
linux-arm-kernel, linux-kernel, linux-mediatek, linux-scsi,
martin.petersen, matthias.bgg,
Peter Wang (王信友),
Powen Kao (高伯文),
Qilin Tan (谭麒麟),
Stanley Chu (朱原陞),
wsd_upstream, Yanxu Wei (韦晏旭)
Hi Miles,
On Wed, 2022-03-09 at 07:23 +0800, Miles Chen wrote:
> Hi Alice,
>
> Thanks for your patch. I have some questions.
>
> > Device reset thread uses kobject_uevent_env() to get kobj.parent
> > after scsi_evt_emit(), and it races with device init thread which
> > calls device_add() to create kobj.parent before
> > kobject_uevent_env().
> >
> > Device reset call trace:
> > fill_kobj_path
> > kobject_get_path
> > kobject_uevent_env
> > scsi_evt_emit <- add wait_event()
> > scsi_evt_thread
> >
> > Device init call trace:
> > fill_kobj_path
> > kobject_get_path
> > kobject_uevent_env
> > device_add <- create kobj.parent
> > scsi_target_add
> > scsi_sysfs_add_sdev
> > scsi_add_lun
> > scsi_probe_and_add_lun
>
> These are backtraces, and the things above looks like your solution,
> not
> the problem.
> It's better to describe the problem first and solution later:
>
> Device init call: Device reset call:
> scsi_probe_and_add_lun() scsi_evt_thread()
> scsi_add_lun()
> scsi_sysfs_add_sdev() scsi_evt_emit()
> scsi_target_add() kobject_uevent_env
> ()
> device_add() kobject_get_path
> () // something wrong?
> fill_kobj_path()
> get_device_parent() // I see a get, not create parent kobj,
> could you show me that point?
> kobject_uevent()
> kobject_uevent_env()
> kobject_get_path()
> fill_kobj_path()
>
Device init call: Device reset call:
scsi_probe_and_add_lun() scsi_evt_thread()
scsi_add_lun() scsi_evt_emit()
scsi_sysfs_add_sdev() kobject_uevent_env() //get kobj.parent
scsi_target_add() kobject_get_path() //get wrong
kobj.parent device_add() // add kobj.parent fill_kobj_path()
kobject_uevent_env()
kobject_get_path()
fill_kobj_path()
Above backtrace describes the problem, device reset thread will get
wrong kobj.parent when device init thread didn’t add kobj.parent yet.
> >
> > These two jobs are scheduled asynchronously, we can't guaranteed
> > that
> > kobj.parent will be created in device init thread before device
> > reset
> > thread calls kobj_get_path().
>
> What happens if device reset thread calls kobj_get_path() before
> kobj.parent's
> creation? a crash or something?
>
It will cause slab-out-of-bounds in kobject_get_path and kernel panic
than crash.
> >
> > To resolve the racing issue between device init thread and device
> > reset thread, we use wait_event() in scsi_evt_emit() to wait for
> > device_add() to complete the creation of kobj.parent.
> >
> > Signed-off-by: Alice Chao <alice.chao@mediatek.com>
>
> When we send patch N+1, we have to put the change log between patch
> v1 to vN
> under a "---", so others can understand the changes between patch N
> and N+1.
> e.g.,
>
> ---
>
> Change since v1
> - remove Change-Id
>
> > ---
> > drivers/scsi/scsi_lib.c | 1 +
> > drivers/scsi/scsi_scan.c | 1 +
> > 2 files changed, 2 insertions(+)
> >
> > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
> > index 0a70aa763a96..abf9a71ed77c 100644
> > --- a/drivers/scsi/scsi_lib.c
> > +++ b/drivers/scsi/scsi_lib.c
> > @@ -2461,6 +2461,7 @@ static void scsi_evt_emit(struct scsi_device
> > *sdev, struct scsi_event *evt)
> > break;
> > case SDEV_EVT_POWER_ON_RESET_OCCURRED:
> > envp[idx++] = "SDEV_UA=POWER_ON_RESET_OCCURRED";
> > + wait_event(sdev->host->host_wait, sdev-
> > >sdev_gendev.kobj.parent != NULL);
> > break;
> > default:
> > /* do nothing */
> > diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
> > index f4e6c68ac99e..431f229ac435 100644
> > --- a/drivers/scsi/scsi_scan.c
> > +++ b/drivers/scsi/scsi_scan.c
> > @@ -1904,6 +1904,7 @@ static void do_scsi_scan_host(struct
> > Scsi_Host *shost)
> > } else {
> > scsi_scan_host_selected(shost, SCAN_WILD_CARD,
> > SCAN_WILD_CARD,
> > SCAN_WILD_CARD, 0);
> > + wake_up(&shost->host_wait);
>
> do_scsi_scan_host() is not in the call trace above, could you show
> the relationship
> of do_scsi_scan_host in the call flow?
>
ufshcd_async_scan()
scsi_scan_host()
do_scsi_scan_host() << here
scsi_scan_host_selected()
__scsi_scan_target()
scsi_probe_and_add_lun()
scsi_sysfs_add_sdev()
device_add()
kobject_uevent_env()
kobject_get_path()
fill_kobj_path()
After we add wake_up here, we can ensure that device reset thread will
get kobject after device init thread finishes adding parent.
> thanks,
> Miles
>
> > }
> > }
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v2 1/1] scsi: Fix racing between dev init and dev reset
2022-04-13 6:53 ` Alice Chao
@ 2022-04-14 8:18 ` Miles Chen
0 siblings, 0 replies; 4+ messages in thread
From: Miles Chen @ 2022-04-14 8:18 UTC (permalink / raw)
To: alice.chao
Cc: Chaotian.Jing, Chun-hung.Wu, Lin.Gui, Powen.Kao, Qilin.Tan,
Yanxu.Wei, cc.chou, jejb, jiajie.hao, jonathan.hsu,
linux-arm-kernel, linux-kernel, linux-mediatek, linux-scsi,
martin.petersen, matthias.bgg, miles.chen, peter.wang,
stanley.chu, wsd_upstream
Hi Alice,
>Hi Miles,
>
>On Wed, 2022-03-09 at 07:23 +0800, Miles Chen wrote:
>> Hi Alice,
>>
>> Thanks for your patch. I have some questions.
>>
>> > Device reset thread uses kobject_uevent_env() to get kobj.parent
>> > after scsi_evt_emit(), and it races with device init thread which
>> > calls device_add() to create kobj.parent before
>> > kobject_uevent_env().
>> >
>> > Device reset call trace:
>> > fill_kobj_path
>> > kobject_get_path
>> > kobject_uevent_env
>> > scsi_evt_emit <- add wait_event()
>> > scsi_evt_thread
>> >
>> > Device init call trace:
>> > fill_kobj_path
>> > kobject_get_path
>> > kobject_uevent_env
>> > device_add <- create kobj.parent
>> > scsi_target_add
>> > scsi_sysfs_add_sdev
>> > scsi_add_lun
>> > scsi_probe_and_add_lun
>>
>> These are backtraces, and the things above looks like your solution,
>> not
>> the problem.
>> It's better to describe the problem first and solution later:
>>
>> Device init call: Device reset call:
>> scsi_probe_and_add_lun() scsi_evt_thread()
>> scsi_add_lun()
>> scsi_sysfs_add_sdev() scsi_evt_emit()
>> scsi_target_add() kobject_uevent_env
>> ()
>> device_add() kobject_get_path
>> () // something wrong?
>> fill_kobj_path()
>> get_device_parent() // I see a get, not create parent kobj,
>> could you show me that point?
>> kobject_uevent()
>> kobject_uevent_env()
>> kobject_get_path()
>> fill_kobj_path()
>>
>
>Device init call: Device reset call:
>scsi_probe_and_add_lun() scsi_evt_thread()
> scsi_add_lun() scsi_evt_emit()
> scsi_sysfs_add_sdev() kobject_uevent_env() //get kobj.parent
> scsi_target_add() kobject_get_path() //get wrong
>kobj.parent device_add() // add kobj.parent fill_kobj_path()
> kobject_uevent_env()
> kobject_get_path()
> fill_kobj_path()
>Above backtrace describes the problem, device reset thread will get
>wrong kobj.parent when device init thread didn’t add kobj.parent yet.
Thanks for sharing the error log, I think it is clearer this way:
Device init call: Device reset call:
scsi_probe_and_add_lun() scsi_evt_thread()
scsi_add_lun() scsi_evt_emit()
scsi_sysfs_add_sdev() kobject_uevent_env() //get kobj.parent
scsi_target_add() kobject_get_path()
len = get_kobj_path_length () // len=1 because parent is not create yet
device_add() // add kobj.parent
kobject_uevent_env()
kobject_get_path() path = kzalloc()
fill_kobj_path() fill_kobj_path() // --length; length -= cur is a negative value
memcpy(path + length, kobject_name(parent), cur); // slab OOB!
Now I understood this slab OOB issue, please add the kasan log and the
analysis to the commit message so the maintainers can see it and help.
>
>> >
>> > These two jobs are scheduled asynchronously, we can't guaranteed
>> > that
>> > kobj.parent will be created in device init thread before device
>> > reset
>> > thread calls kobj_get_path().
>>
>> What happens if device reset thread calls kobj_get_path() before
>> kobj.parent's
>> creation? a crash or something?
>>
>
>It will cause slab-out-of-bounds in kobject_get_path and kernel panic
>than crash.
Okay, please add the KASAN slab-out-of-bounds error in the commit message.
Thanks,
Miles
>
>> >
>> > To resolve the racing issue between device init thread and device
>> > reset thread, we use wait_event() in scsi_evt_emit() to wait for
>> > device_add() to complete the creation of kobj.parent.
>> >
>> > Signed-off-by: Alice Chao <alice.chao@mediatek.com>
>>
>> When we send patch N+1, we have to put the change log between patch
>> v1 to vN
>> under a "---", so others can understand the changes between patch N
>> and N+1.
>> e.g.,
>>
>> ---
>>
>> Change since v1
>> - remove Change-Id
>>
>> > ---
>> > drivers/scsi/scsi_lib.c | 1 +
>> > drivers/scsi/scsi_scan.c | 1 +
>> > 2 files changed, 2 insertions(+)
>> >
>> > diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
>> > index 0a70aa763a96..abf9a71ed77c 100644
>> > --- a/drivers/scsi/scsi_lib.c
>> > +++ b/drivers/scsi/scsi_lib.c
>> > @@ -2461,6 +2461,7 @@ static void scsi_evt_emit(struct scsi_device
>> > *sdev, struct scsi_event *evt)
>> > break;
>> > case SDEV_EVT_POWER_ON_RESET_OCCURRED:
>> > envp[idx++] = "SDEV_UA=POWER_ON_RESET_OCCURRED";
>> > + wait_event(sdev->host->host_wait, sdev-
>> > >sdev_gendev.kobj.parent != NULL);
>> > break;
>> > default:
>> > /* do nothing */
>> > diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
>> > index f4e6c68ac99e..431f229ac435 100644
>> > --- a/drivers/scsi/scsi_scan.c
>> > +++ b/drivers/scsi/scsi_scan.c
>> > @@ -1904,6 +1904,7 @@ static void do_scsi_scan_host(struct
>> > Scsi_Host *shost)
>> > } else {
>> > scsi_scan_host_selected(shost, SCAN_WILD_CARD,
>> > SCAN_WILD_CARD,
>> > SCAN_WILD_CARD, 0);
>> > + wake_up(&shost->host_wait);
>>
>> do_scsi_scan_host() is not in the call trace above, could you show
>> the relationship
>> of do_scsi_scan_host in the call flow?
>>
>
>ufshcd_async_scan()
>scsi_scan_host()
>do_scsi_scan_host() << here
>scsi_scan_host_selected()
> __scsi_scan_target()
>scsi_probe_and_add_lun()
>scsi_sysfs_add_sdev()
>device_add()
>kobject_uevent_env()
>kobject_get_path()
>fill_kobj_path()
>
>After we add wake_up here, we can ensure that device reset thread will
>get kobject after device init thread finishes adding parent.
>
>> thanks,
>> Miles
>>
>> > }
>> > }
>
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-04-14 8:19 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-08 2:07 [PATCH v2 1/1] scsi: Fix racing between dev init and dev reset Alice Chao
2022-03-08 23:23 ` Miles Chen
2022-04-13 6:53 ` Alice Chao
2022-04-14 8:18 ` Miles Chen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).