* PROBLEM: a concurrency bug in drivers/md/md.c
@ 2020-11-06 18:58 Gong, Sishuai
2020-11-06 23:15 ` Song Liu
0 siblings, 1 reply; 4+ messages in thread
From: Gong, Sishuai @ 2020-11-06 18:58 UTC (permalink / raw)
To: song; +Cc: linux-raid
Hi,
We found a concurrency bug in linux 5.3.11 that we were able to reproduce in x86 under specific interleavings. This bug causes a warning message “WARNING: linux-5.3.11/drivers/md/md.c:7279 md_ioctl+0x9cd/0x1b02”.
This bug is triggered when two kernel threads run the md_ioctl() function on the same resource interleave with each other. The code sets the mddev->flags to indicate that the resource is being modified and resets it after the modification. However, the current code allows another thread to execute after the mddev->flags is set but before it is reset, resulting in the warning message.
------------------------------------------
Kernel console output
[ 140.524331] WARNING: CPU: 1 PID: 1815 at /tmp/tmp.B7zb7od2zE-5.3.11/extract/linux-5.3.11/drivers/md/md.c:7279 md_ioctl+0x9cd/0x1b02
[ 145.438749] Modules linked in:
[ 147.691130] CPU: 1 PID: 1815 Comm: ski-executor Not tainted 5.3.11 #1
[ 150.333839] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 153.712887] EIP: md_ioctl+0x9cd/0x1b02
[ 157.464368] Code: ff ff ff e8 0f ed 91 ff c6 45 84 01 e9 10 ff ff ff 8d 83 74 01 00 00 e8 75 33 24 00 c6 45 84 00 be f0 ff ff ff e9 3e f7 ff ff <0f> 0b eb bf b0 00 eb 02 b0 01 84 c0 0f 84 2c f7 ff ff 89 7c 24 0c
[ 168.813781] EAX: 00000002 EBX: f3df4800 ECX: f3df497c EDX: 00000002
[ 171.890615] ESI: 00000000 EDI: 00000932 EBP: e527be2c ESP: e527bd98
[ 175.465728] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000202
[ 179.394439] CR0: 80050033 CR2: 08572568 CR3: 25242000 CR4: 00000690
[ 183.140588] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 186.578976] DR6: 00000000 DR7: 00000000
------------------------------------------
Test input
The bug is triggered when the same kernel test program is executed concurrently by two different threads. In particular, it is triggered when the system call md_ioctl() interleaves with itself.
The test program is in Syzkaller’s format as follows:
r0 = openat$md(0xffffffffffffff9c, &(0x7f0000000000)='/dev/md0\x00', 0x0, 0x0)
ioctl$BLKTRACETEARDOWN(r0, 0x932, 0x0)
------------------------------------------
Interleaving
Our analysis revealed that the following interleaving can trigger this bug:
Thread 1 Thread 2
md_open()
-if (test_bit(MD_CLOSING, &mddev->flags)) {
mutex_unlock(&mddev->open_mutex);
err = -ENODEV;
goto out;
}
(condition is false)
-…
-mutex_unlock(&mddev->open_mutex);
-…
-return err;
(md_open finishes correctly)
md_open()
-if (test_bit(MD_CLOSING, &mddev->flags)) {
mutex_unlock(&mddev->open_mutex);
err = -ENODEV;
goto out;
}
(condition is false)
-...
-return err;
(md_open finishes correctly)
md_ioctl()
(drivers/md/md.c:7279)
-WARN_ON_ONCE(test_bit(MD_CLOSING, &mddev->flags));
-set_bit(MD_CLOSING, &mddev->flags);
-...
-mutex_unlock(&mddev->open_mutex);
md_ioctl()
(drivers/md/md.c:7279)
-WARN_ON_ONCE(test_bit(MD_CLOSING, &mddev->flags));
(warning message shows)
(drivers/md/md.c:7347)
-case STOP_ARRAY:
err = do_md_stop(mddev, 0, bdev);
goto unlock;
(mddev->flags will be cleared inside do_md_stop())
Thanks,
Sishuai
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: PROBLEM: a concurrency bug in drivers/md/md.c
2020-11-06 18:58 PROBLEM: a concurrency bug in drivers/md/md.c Gong, Sishuai
@ 2020-11-06 23:15 ` Song Liu
2020-11-07 1:06 ` Gong, Sishuai
0 siblings, 1 reply; 4+ messages in thread
From: Song Liu @ 2020-11-06 23:15 UTC (permalink / raw)
To: Gong, Sishuai; +Cc: linux-raid
On Fri, Nov 6, 2020 at 10:58 AM Gong, Sishuai <sishuai@purdue.edu> wrote:
>
>
> Hi,
>
> We found a concurrency bug in linux 5.3.11 that we were able to reproduce in x86 under specific interleavings. This bug causes a warning message “WARNING: linux-5.3.11/drivers/md/md.c:7279 md_ioctl+0x9cd/0x1b02”.
>
> This bug is triggered when two kernel threads run the md_ioctl() function on the same resource interleave with each other. The code sets the mddev->flags to indicate that the resource is being modified and resets it after the modification. However, the current code allows another thread to execute after the mddev->flags is set but before it is reset, resulting in the warning message.
>
> ------------------------------------------
> Kernel console output
> [ 140.524331] WARNING: CPU: 1 PID: 1815 at /tmp/tmp.B7zb7od2zE-5.3.11/extract/linux-5.3.11/drivers/md/md.c:7279 md_ioctl+0x9cd/0x1b02
> [ 145.438749] Modules linked in:
> [ 147.691130] CPU: 1 PID: 1815 Comm: ski-executor Not tainted 5.3.11 #1
> [ 150.333839] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
> [ 153.712887] EIP: md_ioctl+0x9cd/0x1b02
> [ 157.464368] Code: ff ff ff e8 0f ed 91 ff c6 45 84 01 e9 10 ff ff ff 8d 83 74 01 00 00 e8 75 33 24 00 c6 45 84 00 be f0 ff ff ff e9 3e f7 ff ff <0f> 0b eb bf b0 00 eb 02 b0 01 84 c0 0f 84 2c f7 ff ff 89 7c 24 0c
> [ 168.813781] EAX: 00000002 EBX: f3df4800 ECX: f3df497c EDX: 00000002
> [ 171.890615] ESI: 00000000 EDI: 00000932 EBP: e527be2c ESP: e527bd98
> [ 175.465728] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000202
> [ 179.394439] CR0: 80050033 CR2: 08572568 CR3: 25242000 CR4: 00000690
> [ 183.140588] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> [ 186.578976] DR6: 00000000 DR7: 00000000
>
> ------------------------------------------
> Test input
>
> The bug is triggered when the same kernel test program is executed concurrently by two different threads. In particular, it is triggered when the system call md_ioctl() interleaves with itself.
>
> The test program is in Syzkaller’s format as follows:
> r0 = openat$md(0xffffffffffffff9c, &(0x7f0000000000)='/dev/md0\x00', 0x0, 0x0)
> ioctl$BLKTRACETEARDOWN(r0, 0x932, 0x0)
>
>
>
> ------------------------------------------
> Interleaving
>
> Our analysis revealed that the following interleaving can trigger this bug:
>
> Thread 1 Thread 2
> md_open()
> -if (test_bit(MD_CLOSING, &mddev->flags)) {
> mutex_unlock(&mddev->open_mutex);
> err = -ENODEV;
> goto out;
> }
> (condition is false)
> -…
> -mutex_unlock(&mddev->open_mutex);
> -…
> -return err;
> (md_open finishes correctly)
> md_open()
> -if (test_bit(MD_CLOSING, &mddev->flags)) {
> mutex_unlock(&mddev->open_mutex);
> err = -ENODEV;
> goto out;
> }
> (condition is false)
> -...
> -return err;
> (md_open finishes correctly)
>
> md_ioctl()
> (drivers/md/md.c:7279)
> -WARN_ON_ONCE(test_bit(MD_CLOSING, &mddev->flags));
> -set_bit(MD_CLOSING, &mddev->flags);
> -...
> -mutex_unlock(&mddev->open_mutex);
> md_ioctl()
> (drivers/md/md.c:7279)
> -WARN_ON_ONCE(test_bit(MD_CLOSING, &mddev->flags));
> (warning message shows)
> (drivers/md/md.c:7347)
> -case STOP_ARRAY:
> err = do_md_stop(mddev, 0, bdev);
> goto unlock;
> (mddev->flags will be cleared inside do_md_stop())
>
Thanks for the report. Could you please verify whether this commit
address this issue:
https://git.kernel.org/pub/scm/linux/kernel/git/song/md.git/commit/?h=md-next&id=e7f1456b5ee4e97934ae724e7015d95f88984df0
Thanks,
Song
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: PROBLEM: a concurrency bug in drivers/md/md.c
2020-11-06 23:15 ` Song Liu
@ 2020-11-07 1:06 ` Gong, Sishuai
2020-11-09 16:49 ` Song Liu
0 siblings, 1 reply; 4+ messages in thread
From: Gong, Sishuai @ 2020-11-07 1:06 UTC (permalink / raw)
To: Song Liu; +Cc: linux-raid
Sorry, we didn’t notice this patch. It does fix the issue!
Thanks,
Sishuai
> On Nov 6, 2020, at 6:15 PM, Song Liu <song@kernel.org> wrote:
>
> On Fri, Nov 6, 2020 at 10:58 AM Gong, Sishuai <sishuai@purdue.edu> wrote:
>>
>>
>> Hi,
>>
>> We found a concurrency bug in linux 5.3.11 that we were able to reproduce in x86 under specific interleavings. This bug causes a warning message “WARNING: linux-5.3.11/drivers/md/md.c:7279 md_ioctl+0x9cd/0x1b02”.
>>
>> This bug is triggered when two kernel threads run the md_ioctl() function on the same resource interleave with each other. The code sets the mddev->flags to indicate that the resource is being modified and resets it after the modification. However, the current code allows another thread to execute after the mddev->flags is set but before it is reset, resulting in the warning message.
>>
>> ------------------------------------------
>> Kernel console output
>> [ 140.524331] WARNING: CPU: 1 PID: 1815 at /tmp/tmp.B7zb7od2zE-5.3.11/extract/linux-5.3.11/drivers/md/md.c:7279 md_ioctl+0x9cd/0x1b02
>> [ 145.438749] Modules linked in:
>> [ 147.691130] CPU: 1 PID: 1815 Comm: ski-executor Not tainted 5.3.11 #1
>> [ 150.333839] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
>> [ 153.712887] EIP: md_ioctl+0x9cd/0x1b02
>> [ 157.464368] Code: ff ff ff e8 0f ed 91 ff c6 45 84 01 e9 10 ff ff ff 8d 83 74 01 00 00 e8 75 33 24 00 c6 45 84 00 be f0 ff ff ff e9 3e f7 ff ff <0f> 0b eb bf b0 00 eb 02 b0 01 84 c0 0f 84 2c f7 ff ff 89 7c 24 0c
>> [ 168.813781] EAX: 00000002 EBX: f3df4800 ECX: f3df497c EDX: 00000002
>> [ 171.890615] ESI: 00000000 EDI: 00000932 EBP: e527be2c ESP: e527bd98
>> [ 175.465728] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000202
>> [ 179.394439] CR0: 80050033 CR2: 08572568 CR3: 25242000 CR4: 00000690
>> [ 183.140588] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
>> [ 186.578976] DR6: 00000000 DR7: 00000000
>>
>> ------------------------------------------
>> Test input
>>
>> The bug is triggered when the same kernel test program is executed concurrently by two different threads. In particular, it is triggered when the system call md_ioctl() interleaves with itself.
>>
>> The test program is in Syzkaller’s format as follows:
>> r0 = openat$md(0xffffffffffffff9c, &(0x7f0000000000)='/dev/md0\x00', 0x0, 0x0)
>> ioctl$BLKTRACETEARDOWN(r0, 0x932, 0x0)
>>
>>
>>
>> ------------------------------------------
>> Interleaving
>>
>> Our analysis revealed that the following interleaving can trigger this bug:
>>
>> Thread 1 Thread 2
>> md_open()
>> -if (test_bit(MD_CLOSING, &mddev->flags)) {
>> mutex_unlock(&mddev->open_mutex);
>> err = -ENODEV;
>> goto out;
>> }
>> (condition is false)
>> -…
>> -mutex_unlock(&mddev->open_mutex);
>> -…
>> -return err;
>> (md_open finishes correctly)
>> md_open()
>> -if (test_bit(MD_CLOSING, &mddev->flags)) {
>> mutex_unlock(&mddev->open_mutex);
>> err = -ENODEV;
>> goto out;
>> }
>> (condition is false)
>> -...
>> -return err;
>> (md_open finishes correctly)
>>
>> md_ioctl()
>> (drivers/md/md.c:7279)
>> -WARN_ON_ONCE(test_bit(MD_CLOSING, &mddev->flags));
>> -set_bit(MD_CLOSING, &mddev->flags);
>> -...
>> -mutex_unlock(&mddev->open_mutex);
>> md_ioctl()
>> (drivers/md/md.c:7279)
>> -WARN_ON_ONCE(test_bit(MD_CLOSING, &mddev->flags));
>> (warning message shows)
>> (drivers/md/md.c:7347)
>> -case STOP_ARRAY:
>> err = do_md_stop(mddev, 0, bdev);
>> goto unlock;
>> (mddev->flags will be cleared inside do_md_stop())
>>
>
> Thanks for the report. Could you please verify whether this commit
> address this issue:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/song/md.git/commit/?h=md-next&id=e7f1456b5ee4e97934ae724e7015d95f88984df0
>
> Thanks,
> Song
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: PROBLEM: a concurrency bug in drivers/md/md.c
2020-11-07 1:06 ` Gong, Sishuai
@ 2020-11-09 16:49 ` Song Liu
0 siblings, 0 replies; 4+ messages in thread
From: Song Liu @ 2020-11-09 16:49 UTC (permalink / raw)
To: Gong, Sishuai; +Cc: linux-raid
On Fri, Nov 6, 2020 at 5:06 PM Gong, Sishuai <sishuai@purdue.edu> wrote:
>
> Sorry, we didn’t notice this patch. It does fix the issue!
>
> Thanks,
> Sishuai
Thanks for confirming this fix! This patch will get in upstream soon.
Song
>
> > On Nov 6, 2020, at 6:15 PM, Song Liu <song@kernel.org> wrote:
> >
> > On Fri, Nov 6, 2020 at 10:58 AM Gong, Sishuai <sishuai@purdue.edu> wrote:
> >>
> >>
> >> Hi,
> >>
> >> We found a concurrency bug in linux 5.3.11 that we were able to reproduce in x86 under specific interleavings. This bug causes a warning message “WARNING: linux-5.3.11/drivers/md/md.c:7279 md_ioctl+0x9cd/0x1b02”.
> >>
> >> This bug is triggered when two kernel threads run the md_ioctl() function on the same resource interleave with each other. The code sets the mddev->flags to indicate that the resource is being modified and resets it after the modification. However, the current code allows another thread to execute after the mddev->flags is set but before it is reset, resulting in the warning message.
> >>
> >> ------------------------------------------
> >> Kernel console output
> >> [ 140.524331] WARNING: CPU: 1 PID: 1815 at /tmp/tmp.B7zb7od2zE-5.3.11/extract/linux-5.3.11/drivers/md/md.c:7279 md_ioctl+0x9cd/0x1b02
> >> [ 145.438749] Modules linked in:
> >> [ 147.691130] CPU: 1 PID: 1815 Comm: ski-executor Not tainted 5.3.11 #1
> >> [ 150.333839] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
> >> [ 153.712887] EIP: md_ioctl+0x9cd/0x1b02
> >> [ 157.464368] Code: ff ff ff e8 0f ed 91 ff c6 45 84 01 e9 10 ff ff ff 8d 83 74 01 00 00 e8 75 33 24 00 c6 45 84 00 be f0 ff ff ff e9 3e f7 ff ff <0f> 0b eb bf b0 00 eb 02 b0 01 84 c0 0f 84 2c f7 ff ff 89 7c 24 0c
> >> [ 168.813781] EAX: 00000002 EBX: f3df4800 ECX: f3df497c EDX: 00000002
> >> [ 171.890615] ESI: 00000000 EDI: 00000932 EBP: e527be2c ESP: e527bd98
> >> [ 175.465728] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000202
> >> [ 179.394439] CR0: 80050033 CR2: 08572568 CR3: 25242000 CR4: 00000690
> >> [ 183.140588] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
> >> [ 186.578976] DR6: 00000000 DR7: 00000000
> >>
> >> ------------------------------------------
> >> Test input
> >>
> >> The bug is triggered when the same kernel test program is executed concurrently by two different threads. In particular, it is triggered when the system call md_ioctl() interleaves with itself.
> >>
> >> The test program is in Syzkaller’s format as follows:
> >> r0 = openat$md(0xffffffffffffff9c, &(0x7f0000000000)='/dev/md0\x00', 0x0, 0x0)
> >> ioctl$BLKTRACETEARDOWN(r0, 0x932, 0x0)
> >>
> >>
> >>
> >> ------------------------------------------
> >> Interleaving
> >>
> >> Our analysis revealed that the following interleaving can trigger this bug:
> >>
> >> Thread 1 Thread 2
> >> md_open()
> >> -if (test_bit(MD_CLOSING, &mddev->flags)) {
> >> mutex_unlock(&mddev->open_mutex);
> >> err = -ENODEV;
> >> goto out;
> >> }
> >> (condition is false)
> >> -…
> >> -mutex_unlock(&mddev->open_mutex);
> >> -…
> >> -return err;
> >> (md_open finishes correctly)
> >> md_open()
> >> -if (test_bit(MD_CLOSING, &mddev->flags)) {
> >> mutex_unlock(&mddev->open_mutex);
> >> err = -ENODEV;
> >> goto out;
> >> }
> >> (condition is false)
> >> -...
> >> -return err;
> >> (md_open finishes correctly)
> >>
> >> md_ioctl()
> >> (drivers/md/md.c:7279)
> >> -WARN_ON_ONCE(test_bit(MD_CLOSING, &mddev->flags));
> >> -set_bit(MD_CLOSING, &mddev->flags);
> >> -...
> >> -mutex_unlock(&mddev->open_mutex);
> >> md_ioctl()
> >> (drivers/md/md.c:7279)
> >> -WARN_ON_ONCE(test_bit(MD_CLOSING, &mddev->flags));
> >> (warning message shows)
> >> (drivers/md/md.c:7347)
> >> -case STOP_ARRAY:
> >> err = do_md_stop(mddev, 0, bdev);
> >> goto unlock;
> >> (mddev->flags will be cleared inside do_md_stop())
> >>
> >
> > Thanks for the report. Could you please verify whether this commit
> > address this issue:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/song/md.git/commit/?h=md-next&id=e7f1456b5ee4e97934ae724e7015d95f88984df0
> >
> > Thanks,
> > Song
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-11-09 16:50 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-06 18:58 PROBLEM: a concurrency bug in drivers/md/md.c Gong, Sishuai
2020-11-06 23:15 ` Song Liu
2020-11-07 1:06 ` Gong, Sishuai
2020-11-09 16:49 ` Song Liu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).