From: "Gong, Sishuai" <sishuai@purdue.edu>
To: "song@kernel.org" <song@kernel.org>
Cc: "linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>
Subject: PROBLEM: a concurrency bug in drivers/md/md.c
Date: Fri, 6 Nov 2020 18:58:57 +0000 [thread overview]
Message-ID: <CF84A11A-D1E2-4B7D-825A-C54E2C82B28F@purdue.edu> (raw)
Hi,
We found a concurrency bug in linux 5.3.11 that we were able to reproduce in x86 under specific interleavings. This bug causes a warning message “WARNING: linux-5.3.11/drivers/md/md.c:7279 md_ioctl+0x9cd/0x1b02”.
This bug is triggered when two kernel threads run the md_ioctl() function on the same resource interleave with each other. The code sets the mddev->flags to indicate that the resource is being modified and resets it after the modification. However, the current code allows another thread to execute after the mddev->flags is set but before it is reset, resulting in the warning message.
------------------------------------------
Kernel console output
[ 140.524331] WARNING: CPU: 1 PID: 1815 at /tmp/tmp.B7zb7od2zE-5.3.11/extract/linux-5.3.11/drivers/md/md.c:7279 md_ioctl+0x9cd/0x1b02
[ 145.438749] Modules linked in:
[ 147.691130] CPU: 1 PID: 1815 Comm: ski-executor Not tainted 5.3.11 #1
[ 150.333839] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
[ 153.712887] EIP: md_ioctl+0x9cd/0x1b02
[ 157.464368] Code: ff ff ff e8 0f ed 91 ff c6 45 84 01 e9 10 ff ff ff 8d 83 74 01 00 00 e8 75 33 24 00 c6 45 84 00 be f0 ff ff ff e9 3e f7 ff ff <0f> 0b eb bf b0 00 eb 02 b0 01 84 c0 0f 84 2c f7 ff ff 89 7c 24 0c
[ 168.813781] EAX: 00000002 EBX: f3df4800 ECX: f3df497c EDX: 00000002
[ 171.890615] ESI: 00000000 EDI: 00000932 EBP: e527be2c ESP: e527bd98
[ 175.465728] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00000202
[ 179.394439] CR0: 80050033 CR2: 08572568 CR3: 25242000 CR4: 00000690
[ 183.140588] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 186.578976] DR6: 00000000 DR7: 00000000
------------------------------------------
Test input
The bug is triggered when the same kernel test program is executed concurrently by two different threads. In particular, it is triggered when the system call md_ioctl() interleaves with itself.
The test program is in Syzkaller’s format as follows:
r0 = openat$md(0xffffffffffffff9c, &(0x7f0000000000)='/dev/md0\x00', 0x0, 0x0)
ioctl$BLKTRACETEARDOWN(r0, 0x932, 0x0)
------------------------------------------
Interleaving
Our analysis revealed that the following interleaving can trigger this bug:
Thread 1 Thread 2
md_open()
-if (test_bit(MD_CLOSING, &mddev->flags)) {
mutex_unlock(&mddev->open_mutex);
err = -ENODEV;
goto out;
}
(condition is false)
-…
-mutex_unlock(&mddev->open_mutex);
-…
-return err;
(md_open finishes correctly)
md_open()
-if (test_bit(MD_CLOSING, &mddev->flags)) {
mutex_unlock(&mddev->open_mutex);
err = -ENODEV;
goto out;
}
(condition is false)
-...
-return err;
(md_open finishes correctly)
md_ioctl()
(drivers/md/md.c:7279)
-WARN_ON_ONCE(test_bit(MD_CLOSING, &mddev->flags));
-set_bit(MD_CLOSING, &mddev->flags);
-...
-mutex_unlock(&mddev->open_mutex);
md_ioctl()
(drivers/md/md.c:7279)
-WARN_ON_ONCE(test_bit(MD_CLOSING, &mddev->flags));
(warning message shows)
(drivers/md/md.c:7347)
-case STOP_ARRAY:
err = do_md_stop(mddev, 0, bdev);
goto unlock;
(mddev->flags will be cleared inside do_md_stop())
Thanks,
Sishuai
next reply other threads:[~2020-11-06 18:59 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-06 18:58 Gong, Sishuai [this message]
2020-11-06 23:15 ` PROBLEM: a concurrency bug in drivers/md/md.c Song Liu
2020-11-07 1:06 ` Gong, Sishuai
2020-11-09 16:49 ` Song Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CF84A11A-D1E2-4B7D-825A-C54E2C82B28F@purdue.edu \
--to=sishuai@purdue.edu \
--cc=linux-raid@vger.kernel.org \
--cc=song@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).