* kernel 5.11.x: BUG: sleeping function called from invalid context at kernel/locking/mutex.c:281
@ 2021-03-18 10:50 Stuart Shelton
2021-03-18 11:24 ` Filipe Manana
0 siblings, 1 reply; 2+ messages in thread
From: Stuart Shelton @ 2021-03-18 10:50 UTC (permalink / raw)
To: linux-btrfs
Hi all,
I recently migrated an existing ext4 fs using btrfs-convert (setting nodesize to 32k and enabling optional features `extref`, `skinny-metadata` and `no-holes` - the first two of which I believe are now the default in any case?), but I’m subsequently seeing very frequent BUGs being output by the kernel:
[ 821.843637] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:281
[ 821.843641] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 28214, name: podman
[ 821.843644] CPU: 3 PID: 28214 Comm: podman Tainted: G W 5.11.6 #15
[ 821.843646] Hardware name: Dell Inc. PowerEdge R330/084XW4, BIOS 2.11.0 12/08/2020
[ 821.843647] Call Trace:
[ 821.843650] dump_stack+0xa1/0xfb
[ 821.843656] ___might_sleep+0x144/0x160
[ 821.843659] mutex_lock+0x17/0x40
[ 821.843662] kernfs_remove_by_name_ns+0x1f/0x80
[ 821.843666] sysfs_remove_group+0x7d/0xe0
[ 821.843668] sysfs_remove_groups+0x28/0x40
[ 821.843670] kobject_del+0x2a/0x80
[ 821.843672] btrfs_sysfs_del_one_qgroup+0x2b/0x40 [btrfs]
[ 821.843685] __del_qgroup_rb+0x12/0x150 [btrfs]
[ 821.843696] btrfs_remove_qgroup+0x288/0x2a0 [btrfs]
[ 821.843707] btrfs_ioctl+0x3129/0x36a0 [btrfs]
[ 821.843717] ? __mod_lruvec_page_state+0x5e/0xb0
[ 821.843719] ? page_add_new_anon_rmap+0xbc/0x150
[ 821.843723] ? kfree+0x1b4/0x300
[ 821.843725] ? mntput_no_expire+0x55/0x330
[ 821.843728] __x64_sys_ioctl+0x5a/0xa0
[ 821.843731] do_syscall_64+0x33/0x70
[ 821.843733] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 821.843736] RIP: 0033:0x4cd3fb
[ 821.843739] Code: fa ff eb bd e8 86 8b fa ff e9 61 ff ff ff cc e8 fb 55 fa ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
[ 821.843741] RSP: 002b:000000c000906b20 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
[ 821.843744] RAX: ffffffffffffffda RBX: 000000c000050000 RCX: 00000000004cd3fb
[ 821.843745] RDX: 000000c000906b98 RSI: 000000004010942a RDI: 000000000000000f
[ 821.843747] RBP: 000000c000907cd0 R08: 000000c000622901 R09: 0000000000000000
[ 821.843748] R10: 000000c000d992c0 R11: 0000000000000206 R12: 000000000000012d
[ 821.843749] R13: 000000000000012c R14: 0000000000000200 R15: 0000000000000049
The system starts 24 containers on boot via `podman`, and by the time this process is complete there were (on the last power-cycle) 10 such BUG reports logged.
Is this a recognised issue?
Support information:
uname:
Linux dellr330 5.11.6 #15 SMP Wed Mar 17 15:18:52 GMT 2021 x86_64 Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz GenuineIntel GNU/Linux
version:
btrfs-progs v5.10.1
btrfs fi:
Label: 'space' uuid: 94cc0dca-4a1f-4d18-bdf8-943982d1b6ff
Total devices 1 FS bytes used 163.44GiB
devid 1 size 1.56TiB used 231.24GiB path /dev/mapper/storage-space
btrfs df:
Data, single: total=221.16GiB, used=154.74GiB
System, single: total=4.00MiB, used=384.00KiB
Metadata, single: total=10.08GiB, used=8.70GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
fstab entry:
LABEL=space /space btrfs noatime,compress-force=zstd:2,user_subvol_rm_allowed,nofail 0 2
Other dmesg entries:
[ 61.973985] Btrfs loaded, crc32c=crc32c-intel, zoned=yes
[ 63.310454] BTRFS: device label space devid 1 transid 24453 /dev/mapper/storage-space scanned by btrfs (6546)
[ 64.471111] BTRFS info (device dm-1): force zstd compression, level 2
[ 64.471126] BTRFS info (device dm-1): disk space caching is enabled
[ 64.471130] BTRFS info (device dm-1): has skinny extents
[ 81.247002] BTRFS info (device dm-1): checking UUID tree
[ 104.987371] BTRFS error (device dm-1): qgroup scan failed with -4
[ 106.615043] BTRFS error (device dm-1): qgroup scan failed with -4
[ 107.258435] BTRFS error (device dm-1): qgroup scan failed with -4
[ 107.962191] BTRFS error (device dm-1): qgroup scan failed with -4
[ 118.289293] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:281
[ 118.289296] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 9003, name: podman
[ 118.289298] CPU: 4 PID: 9003 Comm: podman Not tainted 5.11.6 #15
[ 118.289301] Hardware name: Dell Inc. PowerEdge R330/084XW4, BIOS 2.11.0 12/08/2020
[ 118.289301] Call Trace:
[ 118.289303] dump_stack+0xa1/0xfb
[ 118.289308] ___might_sleep+0x144/0x160
[ 118.289310] mutex_lock+0x17/0x40
[ 118.289313] kernfs_remove_by_name_ns+0x1f/0x80
[ 118.289317] sysfs_remove_group+0x7d/0xe0
[ 118.289319] sysfs_remove_groups+0x28/0x40
[ 118.289320] kobject_del+0x2a/0x80
[ 118.289322] btrfs_sysfs_del_one_qgroup+0x2b/0x40 [btrfs]
[ 118.289334] __del_qgroup_rb+0x12/0x150 [btrfs]
[ 118.289343] btrfs_remove_qgroup+0x288/0x2a0 [btrfs]
[ 118.289352] btrfs_ioctl+0x3129/0x36a0 [btrfs]
[ 118.289361] ? __mod_lruvec_page_state+0x5e/0xb0
[ 118.289363] ? page_add_new_anon_rmap+0xbc/0x150
[ 118.289366] ? kfree+0x1b4/0x300
[ 118.289368] ? mntput_no_expire+0x55/0x330
[ 118.289371] __x64_sys_ioctl+0x5a/0xa0
[ 118.289374] do_syscall_64+0x33/0x70
[ 118.289375] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 118.289378] RIP: 0033:0x4cd3fb
[ 118.289380] Code: fa ff eb bd e8 86 8b fa ff e9 61 ff ff ff cc e8 fb 55 fa ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
[ 118.289382] RSP: 002b:000000c0005e2b20 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
[ 118.289384] RAX: ffffffffffffffda RBX: 000000c000050000 RCX: 00000000004cd3fb
[ 118.289385] RDX: 000000c0005e2b98 RSI: 000000004010942a RDI: 0000000000000012
[ 118.289386] RBP: 000000c0005e3cd0 R08: 000000c000582c01 R09: 0000000000000000
[ 118.289387] R10: 000000c000708b70 R11: 0000000000000206 R12: 00000000000000b8
[ 118.289388] R13: 00000000000000b7 R14: 0000000000000200 R15: 0000000000000049
[ 498.003691] BTRFS info (device dm-1): qgroup scan completed (inconsistency flag cleared)
[ 499.522376] BTRFS error (device dm-1): qgroup scan failed with -4
[ 499.975886] BTRFS error (device dm-1): qgroup scan failed with -4
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: kernel 5.11.x: BUG: sleeping function called from invalid context at kernel/locking/mutex.c:281
2021-03-18 10:50 kernel 5.11.x: BUG: sleeping function called from invalid context at kernel/locking/mutex.c:281 Stuart Shelton
@ 2021-03-18 11:24 ` Filipe Manana
0 siblings, 0 replies; 2+ messages in thread
From: Filipe Manana @ 2021-03-18 11:24 UTC (permalink / raw)
To: Stuart Shelton; +Cc: linux-btrfs
On Thu, Mar 18, 2021 at 10:53 AM Stuart Shelton <srcshelton@gmail.com> wrote:
>
> Hi all,
>
> I recently migrated an existing ext4 fs using btrfs-convert (setting nodesize to 32k and enabling optional features `extref`, `skinny-metadata` and `no-holes` - the first two of which I believe are now the default in any case?), but I’m subsequently seeing very frequent BUGs being output by the kernel:
>
> [ 821.843637] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:281
> [ 821.843641] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 28214, name: podman
> [ 821.843644] CPU: 3 PID: 28214 Comm: podman Tainted: G W 5.11.6 #15
> [ 821.843646] Hardware name: Dell Inc. PowerEdge R330/084XW4, BIOS 2.11.0 12/08/2020
> [ 821.843647] Call Trace:
> [ 821.843650] dump_stack+0xa1/0xfb
> [ 821.843656] ___might_sleep+0x144/0x160
> [ 821.843659] mutex_lock+0x17/0x40
> [ 821.843662] kernfs_remove_by_name_ns+0x1f/0x80
> [ 821.843666] sysfs_remove_group+0x7d/0xe0
> [ 821.843668] sysfs_remove_groups+0x28/0x40
> [ 821.843670] kobject_del+0x2a/0x80
> [ 821.843672] btrfs_sysfs_del_one_qgroup+0x2b/0x40 [btrfs]
> [ 821.843685] __del_qgroup_rb+0x12/0x150 [btrfs]
> [ 821.843696] btrfs_remove_qgroup+0x288/0x2a0 [btrfs]
> [ 821.843707] btrfs_ioctl+0x3129/0x36a0 [btrfs]
> [ 821.843717] ? __mod_lruvec_page_state+0x5e/0xb0
> [ 821.843719] ? page_add_new_anon_rmap+0xbc/0x150
> [ 821.843723] ? kfree+0x1b4/0x300
> [ 821.843725] ? mntput_no_expire+0x55/0x330
> [ 821.843728] __x64_sys_ioctl+0x5a/0xa0
> [ 821.843731] do_syscall_64+0x33/0x70
> [ 821.843733] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 821.843736] RIP: 0033:0x4cd3fb
> [ 821.843739] Code: fa ff eb bd e8 86 8b fa ff e9 61 ff ff ff cc e8 fb 55 fa ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
> [ 821.843741] RSP: 002b:000000c000906b20 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
> [ 821.843744] RAX: ffffffffffffffda RBX: 000000c000050000 RCX: 00000000004cd3fb
> [ 821.843745] RDX: 000000c000906b98 RSI: 000000004010942a RDI: 000000000000000f
> [ 821.843747] RBP: 000000c000907cd0 R08: 000000c000622901 R09: 0000000000000000
> [ 821.843748] R10: 000000c000d992c0 R11: 0000000000000206 R12: 000000000000012d
> [ 821.843749] R13: 000000000000012c R14: 0000000000000200 R15: 0000000000000049
>
> The system starts 24 containers on boot via `podman`, and by the time this process is complete there were (on the last power-cycle) 10 such BUG reports logged.
>
> Is this a recognised issue?
Ah, it's taking a mutex while holding a spinlock.
I just sent a fix for this:
https://lore.kernel.org/linux-btrfs/206d121e2e2b609ffe31217e6d90bfabe1c4e121.1616066404.git.fdmanana@suse.com/
Thanks for the report.
>
>
> Support information:
>
> uname:
> Linux dellr330 5.11.6 #15 SMP Wed Mar 17 15:18:52 GMT 2021 x86_64 Intel(R) Xeon(R) CPU E3-1240L v5 @ 2.10GHz GenuineIntel GNU/Linux
>
> version:
> btrfs-progs v5.10.1
>
> btrfs fi:
> Label: 'space' uuid: 94cc0dca-4a1f-4d18-bdf8-943982d1b6ff
> Total devices 1 FS bytes used 163.44GiB
> devid 1 size 1.56TiB used 231.24GiB path /dev/mapper/storage-space
>
> btrfs df:
> Data, single: total=221.16GiB, used=154.74GiB
> System, single: total=4.00MiB, used=384.00KiB
> Metadata, single: total=10.08GiB, used=8.70GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> fstab entry:
> LABEL=space /space btrfs noatime,compress-force=zstd:2,user_subvol_rm_allowed,nofail 0 2
>
> Other dmesg entries:
> [ 61.973985] Btrfs loaded, crc32c=crc32c-intel, zoned=yes
> [ 63.310454] BTRFS: device label space devid 1 transid 24453 /dev/mapper/storage-space scanned by btrfs (6546)
> [ 64.471111] BTRFS info (device dm-1): force zstd compression, level 2
> [ 64.471126] BTRFS info (device dm-1): disk space caching is enabled
> [ 64.471130] BTRFS info (device dm-1): has skinny extents
> [ 81.247002] BTRFS info (device dm-1): checking UUID tree
> [ 104.987371] BTRFS error (device dm-1): qgroup scan failed with -4
> [ 106.615043] BTRFS error (device dm-1): qgroup scan failed with -4
> [ 107.258435] BTRFS error (device dm-1): qgroup scan failed with -4
> [ 107.962191] BTRFS error (device dm-1): qgroup scan failed with -4
> [ 118.289293] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:281
> [ 118.289296] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 9003, name: podman
> [ 118.289298] CPU: 4 PID: 9003 Comm: podman Not tainted 5.11.6 #15
> [ 118.289301] Hardware name: Dell Inc. PowerEdge R330/084XW4, BIOS 2.11.0 12/08/2020
> [ 118.289301] Call Trace:
> [ 118.289303] dump_stack+0xa1/0xfb
> [ 118.289308] ___might_sleep+0x144/0x160
> [ 118.289310] mutex_lock+0x17/0x40
> [ 118.289313] kernfs_remove_by_name_ns+0x1f/0x80
> [ 118.289317] sysfs_remove_group+0x7d/0xe0
> [ 118.289319] sysfs_remove_groups+0x28/0x40
> [ 118.289320] kobject_del+0x2a/0x80
> [ 118.289322] btrfs_sysfs_del_one_qgroup+0x2b/0x40 [btrfs]
> [ 118.289334] __del_qgroup_rb+0x12/0x150 [btrfs]
> [ 118.289343] btrfs_remove_qgroup+0x288/0x2a0 [btrfs]
> [ 118.289352] btrfs_ioctl+0x3129/0x36a0 [btrfs]
> [ 118.289361] ? __mod_lruvec_page_state+0x5e/0xb0
> [ 118.289363] ? page_add_new_anon_rmap+0xbc/0x150
> [ 118.289366] ? kfree+0x1b4/0x300
> [ 118.289368] ? mntput_no_expire+0x55/0x330
> [ 118.289371] __x64_sys_ioctl+0x5a/0xa0
> [ 118.289374] do_syscall_64+0x33/0x70
> [ 118.289375] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 118.289378] RIP: 0033:0x4cd3fb
> [ 118.289380] Code: fa ff eb bd e8 86 8b fa ff e9 61 ff ff ff cc e8 fb 55 fa ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
> [ 118.289382] RSP: 002b:000000c0005e2b20 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
> [ 118.289384] RAX: ffffffffffffffda RBX: 000000c000050000 RCX: 00000000004cd3fb
> [ 118.289385] RDX: 000000c0005e2b98 RSI: 000000004010942a RDI: 0000000000000012
> [ 118.289386] RBP: 000000c0005e3cd0 R08: 000000c000582c01 R09: 0000000000000000
> [ 118.289387] R10: 000000c000708b70 R11: 0000000000000206 R12: 00000000000000b8
> [ 118.289388] R13: 00000000000000b7 R14: 0000000000000200 R15: 0000000000000049
> [ 498.003691] BTRFS info (device dm-1): qgroup scan completed (inconsistency flag cleared)
> [ 499.522376] BTRFS error (device dm-1): qgroup scan failed with -4
> [ 499.975886] BTRFS error (device dm-1): qgroup scan failed with -4
>
>
--
Filipe David Manana,
“Whether you think you can, or you think you can't — you're right.”
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2021-03-18 11:25 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-18 10:50 kernel 5.11.x: BUG: sleeping function called from invalid context at kernel/locking/mutex.c:281 Stuart Shelton
2021-03-18 11:24 ` Filipe Manana
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).