* [bug report] WARNING at block/mq-deadline.c:600 dd_exit_sched+0x1c6/0x260 triggered with blktests block/031
@ 2021-11-25 11:02 Yi Zhang
2021-11-26 2:27 ` Ming Lei
0 siblings, 1 reply; 6+ messages in thread
From: Yi Zhang @ 2021-11-25 11:02 UTC (permalink / raw)
To: linux-block; +Cc: Ming Lei
Hello
blktests block/031 triggered below WARNING with latest
linux-block/for-next[1], pls check it.
[1]
f0afafc21027 (HEAD, origin/for-next) Merge branch 'for-5.17/io_uring'
into for-next
[2]
[ 1801.516136] run blktests block/031 at 2021-11-25 05:51:22
[ 1801.691193] null_blk: module loaded
[ 1847.593539] ------------[ cut here ]------------
[ 1847.598906] statistics for priority 1: i 90452 m 0 d 182486 c 90326
[ 1847.606024] WARNING: CPU: 8 PID: 12958 at block/mq-deadline.c:600
dd_exit_sched+0x1c6/0x260
[ 1847.615439] Modules linked in: null_blk sr_mod cdrom
rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache
netfs rfkill sunrpc vfat fat dm_multipath intel_rapl_msr
intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp
coretemp kvm_intel kvm mgag200 i2c_algo_bit drm_kms_helper irqbypass
crct10dif_pclmul crc32_pclmul syscopyarea ghash_clmulni_intel
sysfillrect rapl sysimgblt intel_cstate fb_sys_fops iTCO_wdt
iTCO_vendor_support drm intel_uncore pcspkr cdc_ether mxm_wmi usbnet
mii lpc_ich ipmi_ssif mei_me mei ipmi_si ipmi_devintf ipmi_msghandler
acpi_power_meter acpi_pad xfs libcrc32c sd_mod sg nvme crc32c_intel
nvme_core t10_pi ahci libahci tg3 libata megaraid_sas wmi dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: null_blk]
[ 1847.691165] CPU: 8 PID: 12958 Comm: rmdir Tainted: G S
5.16.0-rc2+ #2
[ 1847.699888] Hardware name: Dell Inc. PowerEdge R730xd/ɲ\xdePow,
BIOS 2.13.0 05/14/2021
[ 1847.708633] RIP: 0010:dd_exit_sched+0x1c6/0x260
[ 1847.713782] Code: 34 48 89 f8 48 c1 e8 03 42 0f b6 04 38 84 c0 74
04 3c 03 7e 63 8b 55 30 41 89 d9 44 89 f6 48 c7 c7 a0 f3 47 a5 e8 6e
09 28 01 <0f> 0b e9 08 ff ff ff 0f 0b e9 e0 fe ff ff 48 89 d7 48 89 14
24 e8
[ 1847.734845] RSP: 0018:ffff88816917fb18 EFLAGS: 00010282
[ 1847.740763] RAX: 0000000000000000 RBX: 00000000000160d6 RCX: 0000000000000000
[ 1847.748810] RDX: 0000000000000001 RSI: ffffffffa548e4a0 RDI: ffffffffa89e2c40
[ 1847.756856] RBP: ffff8882b8584880 R08: ffffed103cb3fad1 R09: ffffed103cb3fad1
[ 1847.764901] R10: ffff8881e59fd687 R11: ffffed103cb3fad0 R12: ffff8882b8584800
[ 1847.772936] R13: ffff8882b8584948 R14: 0000000000000001 R15: dffffc0000000000
[ 1847.780974] FS: 00007f50bb899540(0000) GS:ffff8881e5800000(0000)
knlGS:0000000000000000
[ 1847.790078] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1847.796574] CR2: 000055bb317c00a8 CR3: 0000000139ecc003 CR4: 00000000001706e0
[ 1847.804608] Call Trace:
[ 1847.807418] <TASK>
[ 1847.810232] blk_mq_exit_sched+0x1db/0x2d0
[ 1847.814888] ? blk_release_queue+0xda/0x2d0
[ 1847.819674] elevator_exit+0x44/0x60
[ 1847.823752] blk_release_queue+0x13a/0x2d0
[ 1847.828416] kobject_release+0x10c/0x3a0
[ 1847.832893] disk_release+0x195/0x250
[ 1847.837066] device_release+0x9b/0x210
[ 1847.841333] kobject_release+0x10c/0x3a0
[ 1847.845799] null_del_dev.part.36+0x1e5/0x500 [null_blk]
[ 1847.851837] nullb_group_drop_item+0xa4/0xd0 [null_blk]
[ 1847.857754] configfs_rmdir+0x631/0x830
[ 1847.862185] ? configfs_unregister_subsystem+0x500/0x500
[ 1847.868226] vfs_rmdir+0x150/0x4c0
[ 1847.872124] do_rmdir+0x1bd/0x330
[ 1847.876260] ? __x64_sys_mkdir+0x80/0x80
[ 1847.880741] ? __check_object_size+0x272/0x330
[ 1847.885805] ? strncpy_from_user+0x66/0x340
[ 1847.890554] ? lockdep_hardirqs_on+0x79/0x100
[ 1847.895522] ? getname_flags+0xf8/0x510
[ 1847.899907] __x64_sys_rmdir+0x3e/0x50
[ 1847.904197] do_syscall_64+0x3a/0x80
[ 1847.908275] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 1847.913995] RIP: 0033:0x7f50bb39f1cb
[ 1847.918062] Code: 73 01 c3 48 8b 0d bd fc 2c 00 f7 d8 64 89 01 48
83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 54 00 00
00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8d fc 2c 00 f7 d8 64 89
01 48
[ 1847.939096] RSP: 002b:00007ffc5d7789c8 EFLAGS: 00000246 ORIG_RAX:
0000000000000054
[ 1847.947629] RAX: ffffffffffffffda RBX: 00007ffc5d779271 RCX: 00007f50bb39f1cb
[ 1847.955690] RDX: 00007f50bb6725c0 RSI: 0000000000000001 RDI: 00007ffc5d779271
[ 1847.963729] RBP: 00007ffc5d778af8 R08: 0000000000000000 R09: 0000000000000000
[ 1847.971768] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002
[ 1847.979804] R13: 00007f50bb66f374 R14: 0000000000000000 R15: 0000000000000000
[ 1847.987916] </TASK>
[ 1847.990429] irq event stamp: 25523
[ 1847.994295] hardirqs last enabled at (25533): [<ffffffffa2d97e02>]
__up_console_sem+0x52/0x60
[ 1848.003992] hardirqs last disabled at (25542): [<ffffffffa2d97de7>]
__up_console_sem+0x37/0x60
[ 1848.013675] softirqs last enabled at (25344): [<ffffffffa500064a>]
__do_softirq+0x64a/0xa4c
[ 1848.023200] softirqs last disabled at (25247): [<ffffffffa2c149ae>]
irq_exit_rcu+0x1ce/0x250
[ 1848.032702] ---[ end trace 622a93ec5416c956 ]---
--
Best Regards,
Yi Zhang
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [bug report] WARNING at block/mq-deadline.c:600 dd_exit_sched+0x1c6/0x260 triggered with blktests block/031
2021-11-25 11:02 [bug report] WARNING at block/mq-deadline.c:600 dd_exit_sched+0x1c6/0x260 triggered with blktests block/031 Yi Zhang
@ 2021-11-26 2:27 ` Ming Lei
[not found] ` <CAHj4cs8L0Q0oS2RgNOr+cC55NeALKCYW_BzeFxctWvcrub-iVQ@mail.gmail.com>
2021-11-26 16:24 ` Ming Lei
0 siblings, 2 replies; 6+ messages in thread
From: Ming Lei @ 2021-11-26 2:27 UTC (permalink / raw)
To: Yi Zhang; +Cc: linux-block
Hi Yi,
On Thu, Nov 25, 2021 at 07:02:43PM +0800, Yi Zhang wrote:
> Hello
>
> blktests block/031 triggered below WARNING with latest
> linux-block/for-next[1], pls check it.
>
> [1]
> f0afafc21027 (HEAD, origin/for-next) Merge branch 'for-5.17/io_uring'
> into for-next
After running block/031 for several times in today's linus tree, not
reproduce the issue:
[root@ktest-09 blktests]# uname -a
Linux ktest-09 5.16.0-rc2_up+ #47 SMP PREEMPT Thu Nov 25 21:14:38 EST 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@ktest-09 blktests]# ./check block/031
block/031 (do IO on null-blk with a host tag set) [passed]
runtime 30.302s ... 30.298s
Both for-5.17/block and for-5.17/io_uring have been rebased on v5.16-rc2,
maybe you can try the test again with latest block tree and see if it
can be reproduced.
Thanks,
Ming
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [bug report] WARNING at block/mq-deadline.c:600 dd_exit_sched+0x1c6/0x260 triggered with blktests block/031
[not found] ` <CAHj4cs8L0Q0oS2RgNOr+cC55NeALKCYW_BzeFxctWvcrub-iVQ@mail.gmail.com>
@ 2021-11-26 9:10 ` Ming Lei
0 siblings, 0 replies; 6+ messages in thread
From: Ming Lei @ 2021-11-26 9:10 UTC (permalink / raw)
To: Yi Zhang; +Cc: linux-block
Hi Yi,
On Fri, Nov 26, 2021 at 11:45:14AM +0800, Yi Zhang wrote:
> Hi MIng
> It still can be reproduced with the latest for-next[1].
> BTW, it was only reproduced when I first time run block/031, cannot be
> reproduced if I run it the second time.
Firstly can you reproduce it on v5.16-rc2?
Secondly, can you collect the bpftrace log via the attached script when
running block/031?
#!/usr/bin/bpftrace
#include <linux/blkdev.h>
#include <linux/blk-mq.h>
#include <linux/genhd.h>
kprobe:dd_insert_request
{
$rq = (struct request *)arg1;
if (!($rq->rq_flags & (1 << 12))) {
printf("%lu %16s %d %d: %s rq %lx/%x %s\n", nsecs / 1000, comm, pid, cpu,
ksym(reg("ip")), $rq->cmd_flags, $rq->rq_flags,
kstack);
}
}
Thanks,
Ming
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [bug report] WARNING at block/mq-deadline.c:600 dd_exit_sched+0x1c6/0x260 triggered with blktests block/031
2021-11-26 2:27 ` Ming Lei
[not found] ` <CAHj4cs8L0Q0oS2RgNOr+cC55NeALKCYW_BzeFxctWvcrub-iVQ@mail.gmail.com>
@ 2021-11-26 16:24 ` Ming Lei
2021-11-26 16:38 ` Jens Axboe
1 sibling, 1 reply; 6+ messages in thread
From: Ming Lei @ 2021-11-26 16:24 UTC (permalink / raw)
To: Yi Zhang; +Cc: linux-block
On Fri, Nov 26, 2021 at 10:27:47AM +0800, Ming Lei wrote:
> Hi Yi,
>
> On Thu, Nov 25, 2021 at 07:02:43PM +0800, Yi Zhang wrote:
> > Hello
> >
> > blktests block/031 triggered below WARNING with latest
> > linux-block/for-next[1], pls check it.
> >
> > [1]
> > f0afafc21027 (HEAD, origin/for-next) Merge branch 'for-5.17/io_uring'
> > into for-next
>
> After running block/031 for several times in today's linus tree, not
> reproduce the issue:
Yi, it should be one for-5.17/block only issue, please test the
following patch:
From 13b6abb5545f08bbe7dfea34a1bfc186e04932ac Mon Sep 17 00:00:00 2001
From: Ming Lei <ming.lei@redhat.com>
Date: Sat, 27 Nov 2021 00:19:43 +0800
Subject: [PATCH] blk-mq: use bio->bi_opf after bio is checked
bio->bi_opf isn't finalized before checking the bio, so use it after
submit_bio_checks() returns.
Fixes: f1880d26e517 ("blk-mq: cleanup request allocation")
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
block/blk-mq.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 82491ab676fb..b57c4373e59b 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2712,7 +2712,6 @@ static struct request *blk_mq_get_new_requests(struct request_queue *q,
struct blk_mq_alloc_data data = {
.q = q,
.nr_tags = 1,
- .cmd_flags = bio->bi_opf,
};
struct request *rq;
@@ -2725,6 +2724,8 @@ static struct request *blk_mq_get_new_requests(struct request_queue *q,
rq_qos_throttle(q, bio);
+ /* ->bi_opf is finalized after submit_bio_checks() returns */
+ data.cmd_flags = bio->bi_opf;
if (plug) {
data.nr_tags = plug->nr_ios;
plug->nr_ios = 1;
--
2.31.1
Thanks,
Ming
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [bug report] WARNING at block/mq-deadline.c:600 dd_exit_sched+0x1c6/0x260 triggered with blktests block/031
2021-11-26 16:24 ` Ming Lei
@ 2021-11-26 16:38 ` Jens Axboe
2021-11-27 2:19 ` Yi Zhang
0 siblings, 1 reply; 6+ messages in thread
From: Jens Axboe @ 2021-11-26 16:38 UTC (permalink / raw)
To: Ming Lei, Yi Zhang; +Cc: linux-block
On 11/26/21 9:24 AM, Ming Lei wrote:
> On Fri, Nov 26, 2021 at 10:27:47AM +0800, Ming Lei wrote:
>> Hi Yi,
>>
>> On Thu, Nov 25, 2021 at 07:02:43PM +0800, Yi Zhang wrote:
>>> Hello
>>>
>>> blktests block/031 triggered below WARNING with latest
>>> linux-block/for-next[1], pls check it.
>>>
>>> [1]
>>> f0afafc21027 (HEAD, origin/for-next) Merge branch 'for-5.17/io_uring'
>>> into for-next
>>
>> After running block/031 for several times in today's linus tree, not
>> reproduce the issue:
>
> Yi, it should be one for-5.17/block only issue, please test the
> following patch:
Good catch, again - can't believe this keeps biting us, guess we just
have to get used to this pattern. Though not expecting many changes
there in the future, so maybe not a huge issue.
--
Jens Axboe
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [bug report] WARNING at block/mq-deadline.c:600 dd_exit_sched+0x1c6/0x260 triggered with blktests block/031
2021-11-26 16:38 ` Jens Axboe
@ 2021-11-27 2:19 ` Yi Zhang
0 siblings, 0 replies; 6+ messages in thread
From: Yi Zhang @ 2021-11-27 2:19 UTC (permalink / raw)
To: Jens Axboe; +Cc: Ming Lei, linux-block
On Sat, Nov 27, 2021 at 12:38 AM Jens Axboe <axboe@kernel.dk> wrote:
>
> On 11/26/21 9:24 AM, Ming Lei wrote:
> > On Fri, Nov 26, 2021 at 10:27:47AM +0800, Ming Lei wrote:
> >> Hi Yi,
> >>
> >> On Thu, Nov 25, 2021 at 07:02:43PM +0800, Yi Zhang wrote:
> >>> Hello
> >>>
> >>> blktests block/031 triggered below WARNING with latest
> >>> linux-block/for-next[1], pls check it.
> >>>
> >>> [1]
> >>> f0afafc21027 (HEAD, origin/for-next) Merge branch 'for-5.17/io_uring'
> >>> into for-next
> >>
> >> After running block/031 for several times in today's linus tree, not
> >> reproduce the issue:
> >
> > Yi, it should be one for-5.17/block only issue, please test the
> > following patch:
Thanks Ming, the issue was fixed:
Tested-by: Yi Zhang <yi.zhang@redhat.com>
>
> Good catch, again - can't believe this keeps biting us, guess we just
> have to get used to this pattern. Though not expecting many changes
> there in the future, so maybe not a huge issue.
>
> --
> Jens Axboe
>
--
Best Regards,
Yi Zhang
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-11-27 2:21 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-25 11:02 [bug report] WARNING at block/mq-deadline.c:600 dd_exit_sched+0x1c6/0x260 triggered with blktests block/031 Yi Zhang
2021-11-26 2:27 ` Ming Lei
[not found] ` <CAHj4cs8L0Q0oS2RgNOr+cC55NeALKCYW_BzeFxctWvcrub-iVQ@mail.gmail.com>
2021-11-26 9:10 ` Ming Lei
2021-11-26 16:24 ` Ming Lei
2021-11-26 16:38 ` Jens Axboe
2021-11-27 2:19 ` Yi Zhang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).