5.17, WARNING: at block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0

5.17, WARNING: at block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0

[Chris Murphy]

Looks like a regression of some sort in BFQ, but I'm not immediately
aware of a manifestation in user space. I've also found four other
downstream BFQ related call traces with xfs and btrfs joining in. I'll
list those before the trace...

[   45.263999] kernel: ------------[ cut here ]------------
[   45.264006] kernel: WARNING: CPU: 4 PID: 73 at
block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0
[   45.264014] kernel: Modules linked in: uinput rfcomm snd_seq_dummy
snd_hrtimer nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast
nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink
qrtr bnep b43 snd_hda_codec_cirrus snd_hda_codec_generic btusb
uvcvideo cordic btrtl ledtrig_audio mac80211 btbcm videobuf2_vmalloc
snd_hda_intel btintel videobuf2_memops snd_intel_dspcfg videobuf2_v4l2
btmtk snd_intel_sdw_acpi libarc4 videobuf2_common snd_hda_codec
cfg80211 bluetooth intel_rapl_msr videodev intel_rapl_common
snd_hda_core ssb mc x86_pkg_temp_thermal intel_powerclamp coretemp
ecdh_generic snd_hwdep rfkill kvm_intel snd_seq bcm5974
apple_mfi_fastcharge snd_seq_device i915 joydev snd_pcm kvm snd_timer
irqbypass snd mei_pxp iTCO_wdt mei_hdcp intel_pmc_bxt bcma applesmc
at24 soundcore iTCO_vendor_support mei_me rapl i2c_i801 acpi_als
intel_cstate ttm
[   45.264066] kernel:  mei intel_uncore industrialio_triggered_buffer
i2c_smbus lpc_ich sbs kfifo_buf apple_gmux sbshc industrialio apple_bl
pcspkr zram isofs squashfs crct10dif_pclmul crc32_pclmul crc32c_intel
sdhci_pci cqhci ghash_clmulni_intel hid_appleir firewire_ohci sdhci
tg3 thunderbolt firewire_core mmc_core crc_itu_t hid_apple video uas
usb_storage sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i
cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp
libiscsi_tcp libiscsi scsi_transport_iscsi loop ip6_tables ip_tables
ipmi_devintf ipmi_msghandler fuse
[   45.264101] kernel: CPU: 4 PID: 73 Comm: kworker/u16:2 Not tainted
5.17.1-300.fc36.x86_64 #1
[   45.264104] kernel: Hardware name: Apple Inc.
MacBookPro8,2/Mac-94245A3940C91C80, BIOS
MBP81.88Z.0050.B00.1804101331 04/10/18
[   45.264106] kernel: Workqueue: loop0 loop_workfn [loop]
[   45.264112] kernel: RIP: 0010:bfqq_request_over_limit+0x122/0x3a0
[   45.264115] kernel: Code: 1e 48 8b 5b 60 8d 78 01 48 83 c6 08 48 85
db 0f 84 0e 02 00 00 89 f8 44 0f b6 63 18 45 84 e4 0f 84 f4 01 00 00
44 39 e8 7c d4 <0f> 0b 8d 58 ff 44 39 e8 0f 85 24 02 00 00 83 fb ff 0f
84 d8 01 00
[   45.264118] kernel: RSP: 0018:ffffa8e34034f7b8 EFLAGS: 00010046
[   45.264120] kernel: RAX: 0000000000000005 RBX: ffff92914c004098
RCX: 0000000000000000
[   45.264122] kernel: RDX: 0000000000000000 RSI: ffffa8e34034f800
RDI: 0000000000000005
[   45.264123] kernel: RBP: ffff9291435c5680 R08: 0000000000000800
R09: 0000000000008000
[   45.264125] kernel: R10: 0000000000000000 R11: ffffa8e34034f8e0
R12: 0000000000000001
[   45.264126] kernel: R13: 0000000000000005 R14: 0000000000000004
R15: 0000000000000002
[   45.264128] kernel: FS:  0000000000000000(0000)
GS:ffff9293a3b00000(0000) knlGS:0000000000000000
[   45.264130] kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   45.264131] kernel: CR2: 00007fa7a5755000 CR3: 000000010a9ec002
CR4: 00000000000606e0
[   45.264133] kernel: Call Trace:
[   45.264136] kernel:  <TASK>
[   45.264140] kernel:  ? mempool_alloc+0x4f/0x170
[   45.264145] kernel:  ? bvec_alloc+0x62/0xb0
[   45.264148] kernel:  ? kmem_cache_alloc+0x162/0x2c0
[   45.264152] kernel:  bfq_limit_depth+0xc3/0x220
[   45.264155] kernel:  __blk_mq_alloc_requests+0x237/0x2a0
[   45.264160] kernel:  blk_mq_submit_bio+0x3d3/0x620
[   45.264163] kernel:  submit_bio_noacct+0x1f3/0x2a0
[   45.264165] kernel:  mpage_readahead+0x133/0x180
[   45.264171] kernel:  ? isofs_get_blocks+0x210/0x210 [isofs]
[   45.264175] kernel:  read_pages+0x61/0x2a0
[   45.264178] kernel:  page_cache_ra_unbounded+0x1a8/0x200
[   45.264182] kernel:  filemap_get_pages+0x4ab/0x620
[   45.264185] kernel:  ? copy_page_to_iter+0x2bd/0x410
[   45.264189] kernel:  filemap_read+0xa8/0x2e0
[   45.264192] kernel:  ? avc_has_perm+0x7a/0x170
[   45.264196] kernel:  ? check_preempt_wakeup+0x125/0x2a0
[   45.264202] kernel:  do_iter_readv_writev+0x149/0x180
[   45.264206] kernel:  do_iter_read+0xde/0x1d0
[   45.264209] kernel:  loop_process_work+0x68f/0x8f0 [loop]
[   45.264214] kernel:  process_one_work+0x1c4/0x380
[   45.264218] kernel:  worker_thread+0x4d/0x380
[   45.264221] kernel:  ? process_one_work+0x380/0x380
[   45.264223] kernel:  kthread+0xe6/0x110
[   45.264225] kernel:  ? kthread_complete_and_exit+0x20/0x20
[   45.264227] kernel:  ret_from_fork+0x1f/0x30
[   45.264233] kernel:  </TASK>
[   45.264234] kernel: ---[ end trace 0000000000000000 ]---


https://bugzilla.redhat.com/show_bug.cgi?id=2068723
https://bugzilla.redhat.com/show_bug.cgi?id=2049004
https://bugzilla.redhat.com/show_bug.cgi?id=2066214
https://bugzilla.redhat.com/show_bug.cgi?id=2064732

-- 
Chris Murphy

Re: 5.17, WARNING: at block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0

[Jens Axboe]

Adding Paolo

On 4/2/22 12:39 AM, Chris Murphy wrote:
> Looks like a regression of some sort in BFQ, but I'm not immediately
> aware of a manifestation in user space. I've also found four other
> downstream BFQ related call traces with xfs and btrfs joining in. I'll
> list those before the trace...
> 
> [   45.263999] kernel: ------------[ cut here ]------------
> [   45.264006] kernel: WARNING: CPU: 4 PID: 73 at
> block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0
> [   45.264014] kernel: Modules linked in: uinput rfcomm snd_seq_dummy
> snd_hrtimer nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast
> nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
> nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink
> qrtr bnep b43 snd_hda_codec_cirrus snd_hda_codec_generic btusb
> uvcvideo cordic btrtl ledtrig_audio mac80211 btbcm videobuf2_vmalloc
> snd_hda_intel btintel videobuf2_memops snd_intel_dspcfg videobuf2_v4l2
> btmtk snd_intel_sdw_acpi libarc4 videobuf2_common snd_hda_codec
> cfg80211 bluetooth intel_rapl_msr videodev intel_rapl_common
> snd_hda_core ssb mc x86_pkg_temp_thermal intel_powerclamp coretemp
> ecdh_generic snd_hwdep rfkill kvm_intel snd_seq bcm5974
> apple_mfi_fastcharge snd_seq_device i915 joydev snd_pcm kvm snd_timer
> irqbypass snd mei_pxp iTCO_wdt mei_hdcp intel_pmc_bxt bcma applesmc
> at24 soundcore iTCO_vendor_support mei_me rapl i2c_i801 acpi_als
> intel_cstate ttm
> [   45.264066] kernel:  mei intel_uncore industrialio_triggered_buffer
> i2c_smbus lpc_ich sbs kfifo_buf apple_gmux sbshc industrialio apple_bl
> pcspkr zram isofs squashfs crct10dif_pclmul crc32_pclmul crc32c_intel
> sdhci_pci cqhci ghash_clmulni_intel hid_appleir firewire_ohci sdhci
> tg3 thunderbolt firewire_core mmc_core crc_itu_t hid_apple video uas
> usb_storage sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i
> cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp
> libiscsi_tcp libiscsi scsi_transport_iscsi loop ip6_tables ip_tables
> ipmi_devintf ipmi_msghandler fuse
> [   45.264101] kernel: CPU: 4 PID: 73 Comm: kworker/u16:2 Not tainted
> 5.17.1-300.fc36.x86_64 #1
> [   45.264104] kernel: Hardware name: Apple Inc.
> MacBookPro8,2/Mac-94245A3940C91C80, BIOS
> MBP81.88Z.0050.B00.1804101331 04/10/18
> [   45.264106] kernel: Workqueue: loop0 loop_workfn [loop]
> [   45.264112] kernel: RIP: 0010:bfqq_request_over_limit+0x122/0x3a0
> [   45.264115] kernel: Code: 1e 48 8b 5b 60 8d 78 01 48 83 c6 08 48 85
> db 0f 84 0e 02 00 00 89 f8 44 0f b6 63 18 45 84 e4 0f 84 f4 01 00 00
> 44 39 e8 7c d4 <0f> 0b 8d 58 ff 44 39 e8 0f 85 24 02 00 00 83 fb ff 0f
> 84 d8 01 00
> [   45.264118] kernel: RSP: 0018:ffffa8e34034f7b8 EFLAGS: 00010046
> [   45.264120] kernel: RAX: 0000000000000005 RBX: ffff92914c004098
> RCX: 0000000000000000
> [   45.264122] kernel: RDX: 0000000000000000 RSI: ffffa8e34034f800
> RDI: 0000000000000005
> [   45.264123] kernel: RBP: ffff9291435c5680 R08: 0000000000000800
> R09: 0000000000008000
> [   45.264125] kernel: R10: 0000000000000000 R11: ffffa8e34034f8e0
> R12: 0000000000000001
> [   45.264126] kernel: R13: 0000000000000005 R14: 0000000000000004
> R15: 0000000000000002
> [   45.264128] kernel: FS:  0000000000000000(0000)
> GS:ffff9293a3b00000(0000) knlGS:0000000000000000
> [   45.264130] kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   45.264131] kernel: CR2: 00007fa7a5755000 CR3: 000000010a9ec002
> CR4: 00000000000606e0
> [   45.264133] kernel: Call Trace:
> [   45.264136] kernel:  <TASK>
> [   45.264140] kernel:  ? mempool_alloc+0x4f/0x170
> [   45.264145] kernel:  ? bvec_alloc+0x62/0xb0
> [   45.264148] kernel:  ? kmem_cache_alloc+0x162/0x2c0
> [   45.264152] kernel:  bfq_limit_depth+0xc3/0x220
> [   45.264155] kernel:  __blk_mq_alloc_requests+0x237/0x2a0
> [   45.264160] kernel:  blk_mq_submit_bio+0x3d3/0x620
> [   45.264163] kernel:  submit_bio_noacct+0x1f3/0x2a0
> [   45.264165] kernel:  mpage_readahead+0x133/0x180
> [   45.264171] kernel:  ? isofs_get_blocks+0x210/0x210 [isofs]
> [   45.264175] kernel:  read_pages+0x61/0x2a0
> [   45.264178] kernel:  page_cache_ra_unbounded+0x1a8/0x200
> [   45.264182] kernel:  filemap_get_pages+0x4ab/0x620
> [   45.264185] kernel:  ? copy_page_to_iter+0x2bd/0x410
> [   45.264189] kernel:  filemap_read+0xa8/0x2e0
> [   45.264192] kernel:  ? avc_has_perm+0x7a/0x170
> [   45.264196] kernel:  ? check_preempt_wakeup+0x125/0x2a0
> [   45.264202] kernel:  do_iter_readv_writev+0x149/0x180
> [   45.264206] kernel:  do_iter_read+0xde/0x1d0
> [   45.264209] kernel:  loop_process_work+0x68f/0x8f0 [loop]
> [   45.264214] kernel:  process_one_work+0x1c4/0x380
> [   45.264218] kernel:  worker_thread+0x4d/0x380
> [   45.264221] kernel:  ? process_one_work+0x380/0x380
> [   45.264223] kernel:  kthread+0xe6/0x110
> [   45.264225] kernel:  ? kthread_complete_and_exit+0x20/0x20
> [   45.264227] kernel:  ret_from_fork+0x1f/0x30
> [   45.264233] kernel:  </TASK>
> [   45.264234] kernel: ---[ end trace 0000000000000000 ]---
> 
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=2068723
> https://bugzilla.redhat.com/show_bug.cgi?id=2049004
> https://bugzilla.redhat.com/show_bug.cgi?id=2066214
> https://bugzilla.redhat.com/show_bug.cgi?id=2064732
> 


-- 
Jens Axboe

Re: 5.17, WARNING: at block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0

[Thorsten Leemhuis]

Quick question, before the boilerplate:

Chris, I assumed this didn't happen with 5.16. Please correct me if
that's wrong.

Anyway, moving on:

[TLDR: I'm adding the regression report below to regzbot, the Linux
kernel regression tracking bot; all text you find below is compiled from
a few templates paragraphs you might have encountered already already
from similar mails.]

Hi, this is your Linux kernel regression tracker. Partly top-posting for
once, to make this easily accessible to everyone.

On 02.04.22 08:39, Chris Murphy wrote:
> Looks like a regression of some sort in BFQ, but I'm not immediately
> aware of a manifestation in user space. I've also found four other
> downstream BFQ related call traces with xfs and btrfs joining in. I'll
> list those before the trace...

Thanks for the report.

CCing the regression mailing list, as it should be in the loop for all
regressions, as explained here:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html

To be sure below issue doesn't fall through the cracks unnoticed, I'm
adding it to regzbot, my Linux kernel regression tracking bot:

#regzbot ^introduced v5.16..v5.17
#regzbot title block: bfq: WARNING: at block/bfq-iosched.c:602
bfqq_request_over_limit+0x122/0x3a0
#regzbot ignore-activity

If it turns out this isn't a regression, free free to remove it from the
tracking by sending a reply to this thread containing a paragraph like
"#regzbot invalid: reason why this is invalid" (without the quotes).

Reminder for developers: when fixing the issue, please add a 'Link:'
tags pointing to the report (the mail quoted above) using
lore.kernel.org/r/, as explained in
'Documentation/process/submitting-patches.rst' and
'Documentation/process/5.Posting.rst'. Regzbot needs them to
automatically connect reports with fixes, but they are useful in
general, too.

I'm sending this to everyone that got the initial report, to make
everyone aware of the tracking. I also hope that messages like this
motivate people to directly get at least the regression mailing list and
ideally even regzbot involved when dealing with regressions, as messages
like this wouldn't be needed then. And don't worry, if I need to send
other mails regarding this regression only relevant for regzbot I'll
send them to the regressions lists only (with a tag in the subject so
people can filter them away). With a bit of luck no such messages will
be needed anyway.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I'm getting a lot of
reports on my table. I can only look briefly into most of them and lack
knowledge about most of the areas they concern. I thus unfortunately
will sometimes get things wrong or miss something important. I hope
that's not the case here; if you think it is, don't hesitate to tell me
in a public reply, it's in everyone's interest to set the public record
straight.



> [   45.263999] kernel: ------------[ cut here ]------------
> [   45.264006] kernel: WARNING: CPU: 4 PID: 73 at
> block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0
> [   45.264014] kernel: Modules linked in: uinput rfcomm snd_seq_dummy
> snd_hrtimer nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast
> nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
> nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink
> qrtr bnep b43 snd_hda_codec_cirrus snd_hda_codec_generic btusb
> uvcvideo cordic btrtl ledtrig_audio mac80211 btbcm videobuf2_vmalloc
> snd_hda_intel btintel videobuf2_memops snd_intel_dspcfg videobuf2_v4l2
> btmtk snd_intel_sdw_acpi libarc4 videobuf2_common snd_hda_codec
> cfg80211 bluetooth intel_rapl_msr videodev intel_rapl_common
> snd_hda_core ssb mc x86_pkg_temp_thermal intel_powerclamp coretemp
> ecdh_generic snd_hwdep rfkill kvm_intel snd_seq bcm5974
> apple_mfi_fastcharge snd_seq_device i915 joydev snd_pcm kvm snd_timer
> irqbypass snd mei_pxp iTCO_wdt mei_hdcp intel_pmc_bxt bcma applesmc
> at24 soundcore iTCO_vendor_support mei_me rapl i2c_i801 acpi_als
> intel_cstate ttm
> [   45.264066] kernel:  mei intel_uncore industrialio_triggered_buffer
> i2c_smbus lpc_ich sbs kfifo_buf apple_gmux sbshc industrialio apple_bl
> pcspkr zram isofs squashfs crct10dif_pclmul crc32_pclmul crc32c_intel
> sdhci_pci cqhci ghash_clmulni_intel hid_appleir firewire_ohci sdhci
> tg3 thunderbolt firewire_core mmc_core crc_itu_t hid_apple video uas
> usb_storage sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i
> cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp
> libiscsi_tcp libiscsi scsi_transport_iscsi loop ip6_tables ip_tables
> ipmi_devintf ipmi_msghandler fuse
> [   45.264101] kernel: CPU: 4 PID: 73 Comm: kworker/u16:2 Not tainted
> 5.17.1-300.fc36.x86_64 #1
> [   45.264104] kernel: Hardware name: Apple Inc.
> MacBookPro8,2/Mac-94245A3940C91C80, BIOS
> MBP81.88Z.0050.B00.1804101331 04/10/18
> [   45.264106] kernel: Workqueue: loop0 loop_workfn [loop]
> [   45.264112] kernel: RIP: 0010:bfqq_request_over_limit+0x122/0x3a0
> [   45.264115] kernel: Code: 1e 48 8b 5b 60 8d 78 01 48 83 c6 08 48 85
> db 0f 84 0e 02 00 00 89 f8 44 0f b6 63 18 45 84 e4 0f 84 f4 01 00 00
> 44 39 e8 7c d4 <0f> 0b 8d 58 ff 44 39 e8 0f 85 24 02 00 00 83 fb ff 0f
> 84 d8 01 00
> [   45.264118] kernel: RSP: 0018:ffffa8e34034f7b8 EFLAGS: 00010046
> [   45.264120] kernel: RAX: 0000000000000005 RBX: ffff92914c004098
> RCX: 0000000000000000
> [   45.264122] kernel: RDX: 0000000000000000 RSI: ffffa8e34034f800
> RDI: 0000000000000005
> [   45.264123] kernel: RBP: ffff9291435c5680 R08: 0000000000000800
> R09: 0000000000008000
> [   45.264125] kernel: R10: 0000000000000000 R11: ffffa8e34034f8e0
> R12: 0000000000000001
> [   45.264126] kernel: R13: 0000000000000005 R14: 0000000000000004
> R15: 0000000000000002
> [   45.264128] kernel: FS:  0000000000000000(0000)
> GS:ffff9293a3b00000(0000) knlGS:0000000000000000
> [   45.264130] kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   45.264131] kernel: CR2: 00007fa7a5755000 CR3: 000000010a9ec002
> CR4: 00000000000606e0
> [   45.264133] kernel: Call Trace:
> [   45.264136] kernel:  <TASK>
> [   45.264140] kernel:  ? mempool_alloc+0x4f/0x170
> [   45.264145] kernel:  ? bvec_alloc+0x62/0xb0
> [   45.264148] kernel:  ? kmem_cache_alloc+0x162/0x2c0
> [   45.264152] kernel:  bfq_limit_depth+0xc3/0x220
> [   45.264155] kernel:  __blk_mq_alloc_requests+0x237/0x2a0
> [   45.264160] kernel:  blk_mq_submit_bio+0x3d3/0x620
> [   45.264163] kernel:  submit_bio_noacct+0x1f3/0x2a0
> [   45.264165] kernel:  mpage_readahead+0x133/0x180
> [   45.264171] kernel:  ? isofs_get_blocks+0x210/0x210 [isofs]
> [   45.264175] kernel:  read_pages+0x61/0x2a0
> [   45.264178] kernel:  page_cache_ra_unbounded+0x1a8/0x200
> [   45.264182] kernel:  filemap_get_pages+0x4ab/0x620
> [   45.264185] kernel:  ? copy_page_to_iter+0x2bd/0x410
> [   45.264189] kernel:  filemap_read+0xa8/0x2e0
> [   45.264192] kernel:  ? avc_has_perm+0x7a/0x170
> [   45.264196] kernel:  ? check_preempt_wakeup+0x125/0x2a0
> [   45.264202] kernel:  do_iter_readv_writev+0x149/0x180
> [   45.264206] kernel:  do_iter_read+0xde/0x1d0
> [   45.264209] kernel:  loop_process_work+0x68f/0x8f0 [loop]
> [   45.264214] kernel:  process_one_work+0x1c4/0x380
> [   45.264218] kernel:  worker_thread+0x4d/0x380
> [   45.264221] kernel:  ? process_one_work+0x380/0x380
> [   45.264223] kernel:  kthread+0xe6/0x110
> [   45.264225] kernel:  ? kthread_complete_and_exit+0x20/0x20
> [   45.264227] kernel:  ret_from_fork+0x1f/0x30
> [   45.264233] kernel:  </TASK>
> [   45.264234] kernel: ---[ end trace 0000000000000000 ]---
> 
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=2068723
> https://bugzilla.redhat.com/show_bug.cgi?id=2049004
> https://bugzilla.redhat.com/show_bug.cgi?id=2066214
> https://bugzilla.redhat.com/show_bug.cgi?id=2064732

-- 
Additional information about regzbot:

If you want to know more about regzbot, check out its web-interface, the
getting start guide, and the references documentation:

https://linux-regtracking.leemhuis.info/regzbot/
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md

The last two documents will explain how you can interact with regzbot
yourself if your want to.

Hint for reporters: when reporting a regression it's in your interest to
CC the regression list and tell regzbot about the issue, as that ensures
the regression makes it onto the radar of the Linux kernel's regression
tracker -- that's in your interest, as it ensures your report won't fall
through the cracks unnoticed.

Hint for developers: you normally don't need to care about regzbot once
it's involved. Fix the issue as you normally would, just remember to
include 'Link:' tag in the patch descriptions pointing to all reports
about the issue. This has been expected from developers even before
regzbot showed up for reasons explained in
'Documentation/process/submitting-patches.rst' and
'Documentation/process/5.Posting.rst'.
> 

Re: 5.17, WARNING: at block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0

[Paolo Valente]

This seems to have to do with Jan's patches on tag allocation. I'm CCing him too. Jan, I'm willing to provide my usual dev version for testing, if useful.

> Il giorno 2 apr 2022, alle ore 19:47, Jens Axboe <axboe@kernel.dk> ha scritto:
> 
> Adding Paolo
> 
> On 4/2/22 12:39 AM, Chris Murphy wrote:
>> Looks like a regression of some sort in BFQ, but I'm not immediately
>> aware of a manifestation in user space. I've also found four other
>> downstream BFQ related call traces with xfs and btrfs joining in. I'll
>> list those before the trace...
>> 
>> [   45.263999] kernel: ------------[ cut here ]------------
>> [   45.264006] kernel: WARNING: CPU: 4 PID: 73 at
>> block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0
>> [   45.264014] kernel: Modules linked in: uinput rfcomm snd_seq_dummy
>> snd_hrtimer nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast
>> nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink
>> qrtr bnep b43 snd_hda_codec_cirrus snd_hda_codec_generic btusb
>> uvcvideo cordic btrtl ledtrig_audio mac80211 btbcm videobuf2_vmalloc
>> snd_hda_intel btintel videobuf2_memops snd_intel_dspcfg videobuf2_v4l2
>> btmtk snd_intel_sdw_acpi libarc4 videobuf2_common snd_hda_codec
>> cfg80211 bluetooth intel_rapl_msr videodev intel_rapl_common
>> snd_hda_core ssb mc x86_pkg_temp_thermal intel_powerclamp coretemp
>> ecdh_generic snd_hwdep rfkill kvm_intel snd_seq bcm5974
>> apple_mfi_fastcharge snd_seq_device i915 joydev snd_pcm kvm snd_timer
>> irqbypass snd mei_pxp iTCO_wdt mei_hdcp intel_pmc_bxt bcma applesmc
>> at24 soundcore iTCO_vendor_support mei_me rapl i2c_i801 acpi_als
>> intel_cstate ttm
>> [   45.264066] kernel:  mei intel_uncore industrialio_triggered_buffer
>> i2c_smbus lpc_ich sbs kfifo_buf apple_gmux sbshc industrialio apple_bl
>> pcspkr zram isofs squashfs crct10dif_pclmul crc32_pclmul crc32c_intel
>> sdhci_pci cqhci ghash_clmulni_intel hid_appleir firewire_ohci sdhci
>> tg3 thunderbolt firewire_core mmc_core crc_itu_t hid_apple video uas
>> usb_storage sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i
>> cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp
>> libiscsi_tcp libiscsi scsi_transport_iscsi loop ip6_tables ip_tables
>> ipmi_devintf ipmi_msghandler fuse
>> [   45.264101] kernel: CPU: 4 PID: 73 Comm: kworker/u16:2 Not tainted
>> 5.17.1-300.fc36.x86_64 #1
>> [   45.264104] kernel: Hardware name: Apple Inc.
>> MacBookPro8,2/Mac-94245A3940C91C80, BIOS
>> MBP81.88Z.0050.B00.1804101331 04/10/18
>> [   45.264106] kernel: Workqueue: loop0 loop_workfn [loop]
>> [   45.264112] kernel: RIP: 0010:bfqq_request_over_limit+0x122/0x3a0
>> [   45.264115] kernel: Code: 1e 48 8b 5b 60 8d 78 01 48 83 c6 08 48 85
>> db 0f 84 0e 02 00 00 89 f8 44 0f b6 63 18 45 84 e4 0f 84 f4 01 00 00
>> 44 39 e8 7c d4 <0f> 0b 8d 58 ff 44 39 e8 0f 85 24 02 00 00 83 fb ff 0f
>> 84 d8 01 00
>> [   45.264118] kernel: RSP: 0018:ffffa8e34034f7b8 EFLAGS: 00010046
>> [   45.264120] kernel: RAX: 0000000000000005 RBX: ffff92914c004098
>> RCX: 0000000000000000
>> [   45.264122] kernel: RDX: 0000000000000000 RSI: ffffa8e34034f800
>> RDI: 0000000000000005
>> [   45.264123] kernel: RBP: ffff9291435c5680 R08: 0000000000000800
>> R09: 0000000000008000
>> [   45.264125] kernel: R10: 0000000000000000 R11: ffffa8e34034f8e0
>> R12: 0000000000000001
>> [   45.264126] kernel: R13: 0000000000000005 R14: 0000000000000004
>> R15: 0000000000000002
>> [   45.264128] kernel: FS:  0000000000000000(0000)
>> GS:ffff9293a3b00000(0000) knlGS:0000000000000000
>> [   45.264130] kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [   45.264131] kernel: CR2: 00007fa7a5755000 CR3: 000000010a9ec002
>> CR4: 00000000000606e0
>> [   45.264133] kernel: Call Trace:
>> [   45.264136] kernel:  <TASK>
>> [   45.264140] kernel:  ? mempool_alloc+0x4f/0x170
>> [   45.264145] kernel:  ? bvec_alloc+0x62/0xb0
>> [   45.264148] kernel:  ? kmem_cache_alloc+0x162/0x2c0
>> [   45.264152] kernel:  bfq_limit_depth+0xc3/0x220
>> [   45.264155] kernel:  __blk_mq_alloc_requests+0x237/0x2a0
>> [   45.264160] kernel:  blk_mq_submit_bio+0x3d3/0x620
>> [   45.264163] kernel:  submit_bio_noacct+0x1f3/0x2a0
>> [   45.264165] kernel:  mpage_readahead+0x133/0x180
>> [   45.264171] kernel:  ? isofs_get_blocks+0x210/0x210 [isofs]
>> [   45.264175] kernel:  read_pages+0x61/0x2a0
>> [   45.264178] kernel:  page_cache_ra_unbounded+0x1a8/0x200
>> [   45.264182] kernel:  filemap_get_pages+0x4ab/0x620
>> [   45.264185] kernel:  ? copy_page_to_iter+0x2bd/0x410
>> [   45.264189] kernel:  filemap_read+0xa8/0x2e0
>> [   45.264192] kernel:  ? avc_has_perm+0x7a/0x170
>> [   45.264196] kernel:  ? check_preempt_wakeup+0x125/0x2a0
>> [   45.264202] kernel:  do_iter_readv_writev+0x149/0x180
>> [   45.264206] kernel:  do_iter_read+0xde/0x1d0
>> [   45.264209] kernel:  loop_process_work+0x68f/0x8f0 [loop]
>> [   45.264214] kernel:  process_one_work+0x1c4/0x380
>> [   45.264218] kernel:  worker_thread+0x4d/0x380
>> [   45.264221] kernel:  ? process_one_work+0x380/0x380
>> [   45.264223] kernel:  kthread+0xe6/0x110
>> [   45.264225] kernel:  ? kthread_complete_and_exit+0x20/0x20
>> [   45.264227] kernel:  ret_from_fork+0x1f/0x30
>> [   45.264233] kernel:  </TASK>
>> [   45.264234] kernel: ---[ end trace 0000000000000000 ]---
>> 
>> 
>> https://bugzilla.redhat.com/show_bug.cgi?id=2068723
>> https://bugzilla.redhat.com/show_bug.cgi?id=2049004
>> https://bugzilla.redhat.com/show_bug.cgi?id=2066214
>> https://bugzilla.redhat.com/show_bug.cgi?id=2064732
>> 
> 
> 
> -- 
> Jens Axboe

Re: 5.17, WARNING: at block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0

[yukuai (C)]

在 2022/04/04 22:35, Paolo Valente 写道:
> This seems to have to do with Jan's patches on tag allocation. I'm CCing him too. Jan, I'm willing to provide my usual dev version for testing, if useful.

Hi, Paolo

I had observed this warning for a long time, and it's very to reporduce
by our reporducer for the problem that Jan fixed recently.

I thougt this warning is due to bfqq is associated with wrong cgroup,
which should be fixed by Jan's patch.

What do you think, Jan ?

> 
>> Il giorno 2 apr 2022, alle ore 19:47, Jens Axboe <axboe@kernel.dk> ha scritto:
>>
>> Adding Paolo
>>
>> On 4/2/22 12:39 AM, Chris Murphy wrote:
>>> Looks like a regression of some sort in BFQ, but I'm not immediately
>>> aware of a manifestation in user space. I've also found four other
>>> downstream BFQ related call traces with xfs and btrfs joining in. I'll
>>> list those before the trace...
>>>
>>> [   45.263999] kernel: ------------[ cut here ]------------
>>> [   45.264006] kernel: WARNING: CPU: 4 PID: 73 at
>>> block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0
>>> [   45.264014] kernel: Modules linked in: uinput rfcomm snd_seq_dummy
>>> snd_hrtimer nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast
>>> nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
>>> nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
>>> nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink
>>> qrtr bnep b43 snd_hda_codec_cirrus snd_hda_codec_generic btusb
>>> uvcvideo cordic btrtl ledtrig_audio mac80211 btbcm videobuf2_vmalloc
>>> snd_hda_intel btintel videobuf2_memops snd_intel_dspcfg videobuf2_v4l2
>>> btmtk snd_intel_sdw_acpi libarc4 videobuf2_common snd_hda_codec
>>> cfg80211 bluetooth intel_rapl_msr videodev intel_rapl_common
>>> snd_hda_core ssb mc x86_pkg_temp_thermal intel_powerclamp coretemp
>>> ecdh_generic snd_hwdep rfkill kvm_intel snd_seq bcm5974
>>> apple_mfi_fastcharge snd_seq_device i915 joydev snd_pcm kvm snd_timer
>>> irqbypass snd mei_pxp iTCO_wdt mei_hdcp intel_pmc_bxt bcma applesmc
>>> at24 soundcore iTCO_vendor_support mei_me rapl i2c_i801 acpi_als
>>> intel_cstate ttm
>>> [   45.264066] kernel:  mei intel_uncore industrialio_triggered_buffer
>>> i2c_smbus lpc_ich sbs kfifo_buf apple_gmux sbshc industrialio apple_bl
>>> pcspkr zram isofs squashfs crct10dif_pclmul crc32_pclmul crc32c_intel
>>> sdhci_pci cqhci ghash_clmulni_intel hid_appleir firewire_ohci sdhci
>>> tg3 thunderbolt firewire_core mmc_core crc_itu_t hid_apple video uas
>>> usb_storage sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i
>>> cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp
>>> libiscsi_tcp libiscsi scsi_transport_iscsi loop ip6_tables ip_tables
>>> ipmi_devintf ipmi_msghandler fuse
>>> [   45.264101] kernel: CPU: 4 PID: 73 Comm: kworker/u16:2 Not tainted
>>> 5.17.1-300.fc36.x86_64 #1
>>> [   45.264104] kernel: Hardware name: Apple Inc.
>>> MacBookPro8,2/Mac-94245A3940C91C80, BIOS
>>> MBP81.88Z.0050.B00.1804101331 04/10/18
>>> [   45.264106] kernel: Workqueue: loop0 loop_workfn [loop]
>>> [   45.264112] kernel: RIP: 0010:bfqq_request_over_limit+0x122/0x3a0
>>> [   45.264115] kernel: Code: 1e 48 8b 5b 60 8d 78 01 48 83 c6 08 48 85
>>> db 0f 84 0e 02 00 00 89 f8 44 0f b6 63 18 45 84 e4 0f 84 f4 01 00 00
>>> 44 39 e8 7c d4 <0f> 0b 8d 58 ff 44 39 e8 0f 85 24 02 00 00 83 fb ff 0f
>>> 84 d8 01 00
>>> [   45.264118] kernel: RSP: 0018:ffffa8e34034f7b8 EFLAGS: 00010046
>>> [   45.264120] kernel: RAX: 0000000000000005 RBX: ffff92914c004098
>>> RCX: 0000000000000000
>>> [   45.264122] kernel: RDX: 0000000000000000 RSI: ffffa8e34034f800
>>> RDI: 0000000000000005
>>> [   45.264123] kernel: RBP: ffff9291435c5680 R08: 0000000000000800
>>> R09: 0000000000008000
>>> [   45.264125] kernel: R10: 0000000000000000 R11: ffffa8e34034f8e0
>>> R12: 0000000000000001
>>> [   45.264126] kernel: R13: 0000000000000005 R14: 0000000000000004
>>> R15: 0000000000000002
>>> [   45.264128] kernel: FS:  0000000000000000(0000)
>>> GS:ffff9293a3b00000(0000) knlGS:0000000000000000
>>> [   45.264130] kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [   45.264131] kernel: CR2: 00007fa7a5755000 CR3: 000000010a9ec002
>>> CR4: 00000000000606e0
>>> [   45.264133] kernel: Call Trace:
>>> [   45.264136] kernel:  <TASK>
>>> [   45.264140] kernel:  ? mempool_alloc+0x4f/0x170
>>> [   45.264145] kernel:  ? bvec_alloc+0x62/0xb0
>>> [   45.264148] kernel:  ? kmem_cache_alloc+0x162/0x2c0
>>> [   45.264152] kernel:  bfq_limit_depth+0xc3/0x220
>>> [   45.264155] kernel:  __blk_mq_alloc_requests+0x237/0x2a0
>>> [   45.264160] kernel:  blk_mq_submit_bio+0x3d3/0x620
>>> [   45.264163] kernel:  submit_bio_noacct+0x1f3/0x2a0
>>> [   45.264165] kernel:  mpage_readahead+0x133/0x180
>>> [   45.264171] kernel:  ? isofs_get_blocks+0x210/0x210 [isofs]
>>> [   45.264175] kernel:  read_pages+0x61/0x2a0
>>> [   45.264178] kernel:  page_cache_ra_unbounded+0x1a8/0x200
>>> [   45.264182] kernel:  filemap_get_pages+0x4ab/0x620
>>> [   45.264185] kernel:  ? copy_page_to_iter+0x2bd/0x410
>>> [   45.264189] kernel:  filemap_read+0xa8/0x2e0
>>> [   45.264192] kernel:  ? avc_has_perm+0x7a/0x170
>>> [   45.264196] kernel:  ? check_preempt_wakeup+0x125/0x2a0
>>> [   45.264202] kernel:  do_iter_readv_writev+0x149/0x180
>>> [   45.264206] kernel:  do_iter_read+0xde/0x1d0
>>> [   45.264209] kernel:  loop_process_work+0x68f/0x8f0 [loop]
>>> [   45.264214] kernel:  process_one_work+0x1c4/0x380
>>> [   45.264218] kernel:  worker_thread+0x4d/0x380
>>> [   45.264221] kernel:  ? process_one_work+0x380/0x380
>>> [   45.264223] kernel:  kthread+0xe6/0x110
>>> [   45.264225] kernel:  ? kthread_complete_and_exit+0x20/0x20
>>> [   45.264227] kernel:  ret_from_fork+0x1f/0x30
>>> [   45.264233] kernel:  </TASK>
>>> [   45.264234] kernel: ---[ end trace 0000000000000000 ]---
>>>
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=2068723
>>> https://bugzilla.redhat.com/show_bug.cgi?id=2049004
>>> https://bugzilla.redhat.com/show_bug.cgi?id=2066214
>>> https://bugzilla.redhat.com/show_bug.cgi?id=2064732
>>>
>>
>>
>> -- 
>> Jens Axboe
> 
> .
> 

Re: 5.17, WARNING: at block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0

[Jan Kara]

Hello!

On Wed 06-04-22 09:44:55, yukuai (C) wrote:
> 在 2022/04/04 22:35, Paolo Valente 写道:
> > This seems to have to do with Jan's patches on tag allocation. I'm CCing him too. Jan, I'm willing to provide my usual dev version for testing, if useful.
> 
> I had observed this warning for a long time, and it's very to reporduce
> by our reporducer for the problem that Jan fixed recently.
> 
> I thougt this warning is due to bfqq is associated with wrong cgroup,
> which should be fixed by Jan's patch.
> 
> What do you think, Jan ?

Hum, the warning suggests that the blkcg's idea of cgroup hierarchy is not in
sync with BFQ's idea of cgroup hierarchy. One possibility how this could
happen is if bfqq is moved from one bfqg to another between the depth check
and the for_each_entity() loop. This should actually get fixed as a
side-effect of my patches because they change the logic so that alive bfqq
can change parent only if they contain a single process. But nevertheless
we should make the logic inside bfqq_request_over_limit() more robust in
place of process moving between cgroups. So I'll send a patch for that on
top of my patches.

								Honza


> > > Il giorno 2 apr 2022, alle ore 19:47, Jens Axboe <axboe@kernel.dk> ha scritto:
> > > 
> > > Adding Paolo
> > > 
> > > On 4/2/22 12:39 AM, Chris Murphy wrote:
> > > > Looks like a regression of some sort in BFQ, but I'm not immediately
> > > > aware of a manifestation in user space. I've also found four other
> > > > downstream BFQ related call traces with xfs and btrfs joining in. I'll
> > > > list those before the trace...
> > > > 
> > > > [   45.263999] kernel: ------------[ cut here ]------------
> > > > [   45.264006] kernel: WARNING: CPU: 4 PID: 73 at
> > > > block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0
> > > > [   45.264014] kernel: Modules linked in: uinput rfcomm snd_seq_dummy
> > > > snd_hrtimer nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast
> > > > nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet
> > > > nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat
> > > > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink
> > > > qrtr bnep b43 snd_hda_codec_cirrus snd_hda_codec_generic btusb
> > > > uvcvideo cordic btrtl ledtrig_audio mac80211 btbcm videobuf2_vmalloc
> > > > snd_hda_intel btintel videobuf2_memops snd_intel_dspcfg videobuf2_v4l2
> > > > btmtk snd_intel_sdw_acpi libarc4 videobuf2_common snd_hda_codec
> > > > cfg80211 bluetooth intel_rapl_msr videodev intel_rapl_common
> > > > snd_hda_core ssb mc x86_pkg_temp_thermal intel_powerclamp coretemp
> > > > ecdh_generic snd_hwdep rfkill kvm_intel snd_seq bcm5974
> > > > apple_mfi_fastcharge snd_seq_device i915 joydev snd_pcm kvm snd_timer
> > > > irqbypass snd mei_pxp iTCO_wdt mei_hdcp intel_pmc_bxt bcma applesmc
> > > > at24 soundcore iTCO_vendor_support mei_me rapl i2c_i801 acpi_als
> > > > intel_cstate ttm
> > > > [   45.264066] kernel:  mei intel_uncore industrialio_triggered_buffer
> > > > i2c_smbus lpc_ich sbs kfifo_buf apple_gmux sbshc industrialio apple_bl
> > > > pcspkr zram isofs squashfs crct10dif_pclmul crc32_pclmul crc32c_intel
> > > > sdhci_pci cqhci ghash_clmulni_intel hid_appleir firewire_ohci sdhci
> > > > tg3 thunderbolt firewire_core mmc_core crc_itu_t hid_apple video uas
> > > > usb_storage sunrpc be2iscsi bnx2i cnic uio cxgb4i cxgb4 tls cxgb3i
> > > > cxgb3 mdio libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp
> > > > libiscsi_tcp libiscsi scsi_transport_iscsi loop ip6_tables ip_tables
> > > > ipmi_devintf ipmi_msghandler fuse
> > > > [   45.264101] kernel: CPU: 4 PID: 73 Comm: kworker/u16:2 Not tainted
> > > > 5.17.1-300.fc36.x86_64 #1
> > > > [   45.264104] kernel: Hardware name: Apple Inc.
> > > > MacBookPro8,2/Mac-94245A3940C91C80, BIOS
> > > > MBP81.88Z.0050.B00.1804101331 04/10/18
> > > > [   45.264106] kernel: Workqueue: loop0 loop_workfn [loop]
> > > > [   45.264112] kernel: RIP: 0010:bfqq_request_over_limit+0x122/0x3a0
> > > > [   45.264115] kernel: Code: 1e 48 8b 5b 60 8d 78 01 48 83 c6 08 48 85
> > > > db 0f 84 0e 02 00 00 89 f8 44 0f b6 63 18 45 84 e4 0f 84 f4 01 00 00
> > > > 44 39 e8 7c d4 <0f> 0b 8d 58 ff 44 39 e8 0f 85 24 02 00 00 83 fb ff 0f
> > > > 84 d8 01 00
> > > > [   45.264118] kernel: RSP: 0018:ffffa8e34034f7b8 EFLAGS: 00010046
> > > > [   45.264120] kernel: RAX: 0000000000000005 RBX: ffff92914c004098
> > > > RCX: 0000000000000000
> > > > [   45.264122] kernel: RDX: 0000000000000000 RSI: ffffa8e34034f800
> > > > RDI: 0000000000000005
> > > > [   45.264123] kernel: RBP: ffff9291435c5680 R08: 0000000000000800
> > > > R09: 0000000000008000
> > > > [   45.264125] kernel: R10: 0000000000000000 R11: ffffa8e34034f8e0
> > > > R12: 0000000000000001
> > > > [   45.264126] kernel: R13: 0000000000000005 R14: 0000000000000004
> > > > R15: 0000000000000002
> > > > [   45.264128] kernel: FS:  0000000000000000(0000)
> > > > GS:ffff9293a3b00000(0000) knlGS:0000000000000000
> > > > [   45.264130] kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [   45.264131] kernel: CR2: 00007fa7a5755000 CR3: 000000010a9ec002
> > > > CR4: 00000000000606e0
> > > > [   45.264133] kernel: Call Trace:
> > > > [   45.264136] kernel:  <TASK>
> > > > [   45.264140] kernel:  ? mempool_alloc+0x4f/0x170
> > > > [   45.264145] kernel:  ? bvec_alloc+0x62/0xb0
> > > > [   45.264148] kernel:  ? kmem_cache_alloc+0x162/0x2c0
> > > > [   45.264152] kernel:  bfq_limit_depth+0xc3/0x220
> > > > [   45.264155] kernel:  __blk_mq_alloc_requests+0x237/0x2a0
> > > > [   45.264160] kernel:  blk_mq_submit_bio+0x3d3/0x620
> > > > [   45.264163] kernel:  submit_bio_noacct+0x1f3/0x2a0
> > > > [   45.264165] kernel:  mpage_readahead+0x133/0x180
> > > > [   45.264171] kernel:  ? isofs_get_blocks+0x210/0x210 [isofs]
> > > > [   45.264175] kernel:  read_pages+0x61/0x2a0
> > > > [   45.264178] kernel:  page_cache_ra_unbounded+0x1a8/0x200
> > > > [   45.264182] kernel:  filemap_get_pages+0x4ab/0x620
> > > > [   45.264185] kernel:  ? copy_page_to_iter+0x2bd/0x410
> > > > [   45.264189] kernel:  filemap_read+0xa8/0x2e0
> > > > [   45.264192] kernel:  ? avc_has_perm+0x7a/0x170
> > > > [   45.264196] kernel:  ? check_preempt_wakeup+0x125/0x2a0
> > > > [   45.264202] kernel:  do_iter_readv_writev+0x149/0x180
> > > > [   45.264206] kernel:  do_iter_read+0xde/0x1d0
> > > > [   45.264209] kernel:  loop_process_work+0x68f/0x8f0 [loop]
> > > > [   45.264214] kernel:  process_one_work+0x1c4/0x380
> > > > [   45.264218] kernel:  worker_thread+0x4d/0x380
> > > > [   45.264221] kernel:  ? process_one_work+0x380/0x380
> > > > [   45.264223] kernel:  kthread+0xe6/0x110
> > > > [   45.264225] kernel:  ? kthread_complete_and_exit+0x20/0x20
> > > > [   45.264227] kernel:  ret_from_fork+0x1f/0x30
> > > > [   45.264233] kernel:  </TASK>
> > > > [   45.264234] kernel: ---[ end trace 0000000000000000 ]---
> > > > 
> > > > 
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=2068723
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=2049004
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=2066214
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=2064732
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: 5.17, WARNING: at block/bfq-iosched.c:602 bfqq_request_over_limit+0x122/0x3a0

[Jan Kara]

On Wed 06-04-22 18:03:59, Jan Kara wrote:
> Hello!
> 
> On Wed 06-04-22 09:44:55, yukuai (C) wrote:
> > 在 2022/04/04 22:35, Paolo Valente 写道:
> > > This seems to have to do with Jan's patches on tag allocation. I'm CCing him too. Jan, I'm willing to provide my usual dev version for testing, if useful.
> > 
> > I had observed this warning for a long time, and it's very to reporduce
> > by our reporducer for the problem that Jan fixed recently.
> > 
> > I thougt this warning is due to bfqq is associated with wrong cgroup,
> > which should be fixed by Jan's patch.
> > 
> > What do you think, Jan ?
> 
> Hum, the warning suggests that the blkcg's idea of cgroup hierarchy is not in
> sync with BFQ's idea of cgroup hierarchy. One possibility how this could
> happen is if bfqq is moved from one bfqg to another between the depth check
> and the for_each_entity() loop. This should actually get fixed as a
> side-effect of my patches because they change the logic so that alive bfqq
> can change parent only if they contain a single process. But nevertheless
> we should make the logic inside bfqq_request_over_limit() more robust in
> place of process moving between cgroups. So I'll send a patch for that on
> top of my patches.

Posted: https://lore.kernel.org/all/20220407140738.9723-1-jack@suse.cz

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR