* Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
@ 2021-07-14 3:21 Boyang Xue
2021-07-14 3:57 ` Boyang Xue
2021-07-14 4:11 ` Roman Gushchin
0 siblings, 2 replies; 21+ messages in thread
From: Boyang Xue @ 2021-07-14 3:21 UTC (permalink / raw)
To: linux-fsdevel; +Cc: guro
Hello,
I'm not sure if this is the right place to report this bug, please
correct me if I'm wrong.
I found kernel-5.14.0-rc1 (built from the Linus tree) crash when it's
running xfstests generic/256 on ext4 [1]. Looking at the call trace,
it looks like the bug had been introduced by the commit
c22d70a162d3 writeback, cgroup: release dying cgwbs by switching attached inodes
It only happens on aarch64, not on x86_64, ppc64le and s390x. Testing
was performed with the latest xfstests, and the bug can be reproduced
on ext{2, 3, 4} with {1k, 2k, 4k} block sizes.
Thanks,
Boyang
1. dmesg
```
[ 4366.380974] run fstests generic/256 at 2021-07-12 05:41:40
[ 4368.337078] EXT4-fs (vda3): mounted filesystem with ordered data
mode. Opts: . Quota mode: none.
[ 4371.275986] Unable to handle kernel NULL pointer dereference at
virtual address 0000000000000000
[ 4371.278210] Mem abort info:
[ 4371.278880] ESR = 0x96000005
[ 4371.279603] EC = 0x25: DABT (current EL), IL = 32 bits
[ 4371.280878] SET = 0, FnV = 0
[ 4371.281621] EA = 0, S1PTW = 0
[ 4371.282396] FSC = 0x05: level 1 translation fault
[ 4371.283635] Data abort info:
[ 4371.284333] ISV = 0, ISS = 0x00000005
[ 4371.285246] CM = 0, WnR = 0
[ 4371.285975] user pgtable: 64k pages, 48-bit VAs, pgdp=00000000b0502000
[ 4371.287640] [0000000000000000] pgd=0000000000000000,
p4d=0000000000000000, pud=0000000000000000
[ 4371.290016] Internal error: Oops: 96000005 [#1] SMP
[ 4371.291251] Modules linked in: dm_flakey dm_snapshot dm_bufio
dm_zero dm_mod loop tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver
nfs lockd grace fscache netfs rfkill sunrpc ext4 vfat fat mbcache jbd2
drm fuse xfs libcrc32c crct10dif_ce ghash_ce sha2_ce sha256_arm64
sha1_ce virtio_blk virtio_net net_failover virtio_console failover
virtio_mmio aes_neon_bs [last unloaded: scsi_debug]
[ 4371.300059] CPU: 0 PID: 408468 Comm: kworker/u8:5 Tainted: G
X --------- --- 5.14.0-0.rc1.15.bx.el9.aarch64 #1
[ 4371.303009] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 4371.304685] Workqueue: events_unbound cleanup_offline_cgwbs_workfn
[ 4371.306329] pstate: 004000c5 (nzcv daIF +PAN -UAO -TCO BTYPE=--)
[ 4371.307867] pc : cleanup_offline_cgwbs_workfn+0x320/0x394
[ 4371.309254] lr : cleanup_offline_cgwbs_workfn+0xe0/0x394
[ 4371.310597] sp : ffff80001554fd10
[ 4371.311443] x29: ffff80001554fd10 x28: 0000000000000000 x27: 0000000000000001
[ 4371.313320] x26: 0000000000000000 x25: 00000000000000e0 x24: ffffd2a2fbe671a8
[ 4371.315159] x23: ffff80001554fd88 x22: ffffd2a2fbe67198 x21: ffffd2a2fc25a730
[ 4371.316945] x20: ffff210412bc3000 x19: ffff210412bc3280 x18: 0000000000000000
[ 4371.318690] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[ 4371.320437] x14: 0000000000000000 x13: 0000000000000030 x12: 0000000000000040
[ 4371.322444] x11: ffff210481572238 x10: ffff21048157223a x9 : ffffd2a2fa276c60
[ 4371.324243] x8 : ffff210484106b60 x7 : 0000000000000000 x6 : 000000000007d18a
[ 4371.326049] x5 : ffff210416a86400 x4 : ffff210412bc0280 x3 : 0000000000000000
[ 4371.327898] x2 : ffff80001554fd88 x1 : ffff210412bc0280 x0 : 0000000000000003
[ 4371.329748] Call trace:
[ 4371.330372] cleanup_offline_cgwbs_workfn+0x320/0x394
[ 4371.331694] process_one_work+0x1f4/0x4b0
[ 4371.332767] worker_thread+0x184/0x540
[ 4371.333732] kthread+0x114/0x120
[ 4371.334535] ret_from_fork+0x10/0x18
[ 4371.335440] Code: d63f0020 97f99963 17ffffa6 f8588263 (f9400061)
[ 4371.337174] ---[ end trace e250fe289272792a ]---
[ 4371.338365] Kernel panic - not syncing: Oops: Fatal exception
[ 4371.339884] SMP: stopping secondary CPUs
[ 4372.424137] SMP: failed to stop secondary CPUs 0-2
[ 4372.436894] Kernel Offset: 0x52a2e9fa0000 from 0xffff800010000000
[ 4372.438408] PHYS_OFFSET: 0xfff0defca0000000
[ 4372.439496] CPU features: 0x00200251,23200840
[ 4372.440603] Memory Limit: none
[ 4372.441374] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---
```
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-14 3:21 Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash Boyang Xue
@ 2021-07-14 3:57 ` Boyang Xue
2021-07-14 4:11 ` Roman Gushchin
1 sibling, 0 replies; 21+ messages in thread
From: Boyang Xue @ 2021-07-14 3:57 UTC (permalink / raw)
To: linux-fsdevel; +Cc: guro
On Wed, Jul 14, 2021 at 11:21 AM Boyang Xue <bxue@redhat.com> wrote:
>
> Hello,
>
> I'm not sure if this is the right place to report this bug, please
> correct me if I'm wrong.
>
> I found kernel-5.14.0-rc1 (built from the Linus tree) crash when it's
> running xfstests generic/256 on ext4 [1]. Looking at the call trace,
> it looks like the bug had been introduced by the commit
>
> c22d70a162d3 writeback, cgroup: release dying cgwbs by switching attached inodes
>
> It only happens on aarch64, not on x86_64, ppc64le and s390x. Testing
Correction: It only happens on aarch64 and ppc64le, not on x86_64 and s390x.
> was performed with the latest xfstests, and the bug can be reproduced
> on ext{2, 3, 4} with {1k, 2k, 4k} block sizes.
>
> Thanks,
> Boyang
>
> 1. dmesg
> ```
> [ 4366.380974] run fstests generic/256 at 2021-07-12 05:41:40
> [ 4368.337078] EXT4-fs (vda3): mounted filesystem with ordered data
> mode. Opts: . Quota mode: none.
> [ 4371.275986] Unable to handle kernel NULL pointer dereference at
> virtual address 0000000000000000
> [ 4371.278210] Mem abort info:
> [ 4371.278880] ESR = 0x96000005
> [ 4371.279603] EC = 0x25: DABT (current EL), IL = 32 bits
> [ 4371.280878] SET = 0, FnV = 0
> [ 4371.281621] EA = 0, S1PTW = 0
> [ 4371.282396] FSC = 0x05: level 1 translation fault
> [ 4371.283635] Data abort info:
> [ 4371.284333] ISV = 0, ISS = 0x00000005
> [ 4371.285246] CM = 0, WnR = 0
> [ 4371.285975] user pgtable: 64k pages, 48-bit VAs, pgdp=00000000b0502000
> [ 4371.287640] [0000000000000000] pgd=0000000000000000,
> p4d=0000000000000000, pud=0000000000000000
> [ 4371.290016] Internal error: Oops: 96000005 [#1] SMP
> [ 4371.291251] Modules linked in: dm_flakey dm_snapshot dm_bufio
> dm_zero dm_mod loop tls rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver
> nfs lockd grace fscache netfs rfkill sunrpc ext4 vfat fat mbcache jbd2
> drm fuse xfs libcrc32c crct10dif_ce ghash_ce sha2_ce sha256_arm64
> sha1_ce virtio_blk virtio_net net_failover virtio_console failover
> virtio_mmio aes_neon_bs [last unloaded: scsi_debug]
> [ 4371.300059] CPU: 0 PID: 408468 Comm: kworker/u8:5 Tainted: G
> X --------- --- 5.14.0-0.rc1.15.bx.el9.aarch64 #1
> [ 4371.303009] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
> [ 4371.304685] Workqueue: events_unbound cleanup_offline_cgwbs_workfn
> [ 4371.306329] pstate: 004000c5 (nzcv daIF +PAN -UAO -TCO BTYPE=--)
> [ 4371.307867] pc : cleanup_offline_cgwbs_workfn+0x320/0x394
> [ 4371.309254] lr : cleanup_offline_cgwbs_workfn+0xe0/0x394
> [ 4371.310597] sp : ffff80001554fd10
> [ 4371.311443] x29: ffff80001554fd10 x28: 0000000000000000 x27: 0000000000000001
> [ 4371.313320] x26: 0000000000000000 x25: 00000000000000e0 x24: ffffd2a2fbe671a8
> [ 4371.315159] x23: ffff80001554fd88 x22: ffffd2a2fbe67198 x21: ffffd2a2fc25a730
> [ 4371.316945] x20: ffff210412bc3000 x19: ffff210412bc3280 x18: 0000000000000000
> [ 4371.318690] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> [ 4371.320437] x14: 0000000000000000 x13: 0000000000000030 x12: 0000000000000040
> [ 4371.322444] x11: ffff210481572238 x10: ffff21048157223a x9 : ffffd2a2fa276c60
> [ 4371.324243] x8 : ffff210484106b60 x7 : 0000000000000000 x6 : 000000000007d18a
> [ 4371.326049] x5 : ffff210416a86400 x4 : ffff210412bc0280 x3 : 0000000000000000
> [ 4371.327898] x2 : ffff80001554fd88 x1 : ffff210412bc0280 x0 : 0000000000000003
> [ 4371.329748] Call trace:
> [ 4371.330372] cleanup_offline_cgwbs_workfn+0x320/0x394
> [ 4371.331694] process_one_work+0x1f4/0x4b0
> [ 4371.332767] worker_thread+0x184/0x540
> [ 4371.333732] kthread+0x114/0x120
> [ 4371.334535] ret_from_fork+0x10/0x18
> [ 4371.335440] Code: d63f0020 97f99963 17ffffa6 f8588263 (f9400061)
> [ 4371.337174] ---[ end trace e250fe289272792a ]---
> [ 4371.338365] Kernel panic - not syncing: Oops: Fatal exception
> [ 4371.339884] SMP: stopping secondary CPUs
> [ 4372.424137] SMP: failed to stop secondary CPUs 0-2
> [ 4372.436894] Kernel Offset: 0x52a2e9fa0000 from 0xffff800010000000
> [ 4372.438408] PHYS_OFFSET: 0xfff0defca0000000
> [ 4372.439496] CPU features: 0x00200251,23200840
> [ 4372.440603] Memory Limit: none
> [ 4372.441374] ---[ end Kernel panic - not syncing: Oops: Fatal exception ]---
> ```
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-14 3:21 Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash Boyang Xue
2021-07-14 3:57 ` Boyang Xue
@ 2021-07-14 4:11 ` Roman Gushchin
2021-07-14 8:44 ` Boyang Xue
1 sibling, 1 reply; 21+ messages in thread
From: Roman Gushchin @ 2021-07-14 4:11 UTC (permalink / raw)
To: Boyang Xue; +Cc: linux-fsdevel, Jan Kara
On Wed, Jul 14, 2021 at 11:21:12AM +0800, Boyang Xue wrote:
> Hello,
>
> I'm not sure if this is the right place to report this bug, please
> correct me if I'm wrong.
>
> I found kernel-5.14.0-rc1 (built from the Linus tree) crash when it's
> running xfstests generic/256 on ext4 [1]. Looking at the call trace,
> it looks like the bug had been introduced by the commit
>
> c22d70a162d3 writeback, cgroup: release dying cgwbs by switching attached inodes
>
> It only happens on aarch64, not on x86_64, ppc64le and s390x. Testing
> was performed with the latest xfstests, and the bug can be reproduced
> on ext{2, 3, 4} with {1k, 2k, 4k} block sizes.
Hello Boyang,
thank you for the report!
Do you know on which line the oops happens?
I'll try to reproduce the problem. Do you mind sharing your .config, kvm options
and any other meaningful details?
Thank you!
Roman
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-14 4:11 ` Roman Gushchin
@ 2021-07-14 8:44 ` Boyang Xue
2021-07-14 9:26 ` Jan Kara
0 siblings, 1 reply; 21+ messages in thread
From: Boyang Xue @ 2021-07-14 8:44 UTC (permalink / raw)
To: Roman Gushchin; +Cc: linux-fsdevel, Jan Kara
Hi Roman,
On Wed, Jul 14, 2021 at 12:12 PM Roman Gushchin <guro@fb.com> wrote:
>
> On Wed, Jul 14, 2021 at 11:21:12AM +0800, Boyang Xue wrote:
> > Hello,
> >
> > I'm not sure if this is the right place to report this bug, please
> > correct me if I'm wrong.
> >
> > I found kernel-5.14.0-rc1 (built from the Linus tree) crash when it's
> > running xfstests generic/256 on ext4 [1]. Looking at the call trace,
> > it looks like the bug had been introduced by the commit
> >
> > c22d70a162d3 writeback, cgroup: release dying cgwbs by switching attached inodes
> >
> > It only happens on aarch64, not on x86_64, ppc64le and s390x. Testing
> > was performed with the latest xfstests, and the bug can be reproduced
> > on ext{2, 3, 4} with {1k, 2k, 4k} block sizes.
>
> Hello Boyang,
>
> thank you for the report!
>
> Do you know on which line the oops happens?
I was trying to inspect the vmcore with crash utility, but
unfortunately it doesn't work.
```
# crash /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux vmcore
...
crash: invalid structure member offset: task_struct_state
FILE: task.c LINE: 5929 FUNCTION: task_state()
[/usr/bin/crash] error trace: aaaae238b080 => aaaae238aff0 =>
aaaae23ff4e8 => aaaae23ff440
...
```
Could you suggest other ways to know "the line the oops happens"?
> I'll try to reproduce the problem. Do you mind sharing your .config, kvm options
> and any other meaningful details?
I can't access the VM host, so sorry I can't provide the kvm
configuration for now. Please check the following other info:
xfstests local.config
```
# cat local.config
FSTYP="ext4"
TEST_DIR="/test"
TEST_DEV="/dev/vda3"
SCRATCH_MNT="/scratch"
SCRATCH_DEV="/dev/vda4"
LOGWRITES_MNT="/logwrites"
LOGWRITES_DEV="/dev/vda6"
MKFS_OPTIONS="-b 4096"
MOUNT_OPTIONS="-o rw,relatime,seclabel"
TEST_FS_MOUNT_OPTS="-o rw,relatime,seclabel"
```
# lscpu
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Vendor ID: Cavium
BIOS Vendor ID: QEMU
Model name: ThunderX2 99xx
BIOS Model name: virt-rhel7.6.0
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 4
Socket(s): 4
Cluster(s): 1
Stepping: 0x1
BogoMIPS: 400.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32
atomics cpuid asimdrdm
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-3
Vulnerabilities:
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; __user pointer sanitization
Spectre v2: Mitigation; Branch predictor hardening
Srbds: Not affected
Tsx async abort: Not affected
# getconf PAGESIZE
65536
Please let me know if there's other useful info I can provide.
Thanks,
Boyang
>
> Thank you!
>
> Roman
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-14 8:44 ` Boyang Xue
@ 2021-07-14 9:26 ` Jan Kara
2021-07-14 16:22 ` Boyang Xue
0 siblings, 1 reply; 21+ messages in thread
From: Jan Kara @ 2021-07-14 9:26 UTC (permalink / raw)
To: Boyang Xue; +Cc: Roman Gushchin, linux-fsdevel, Jan Kara
On Wed 14-07-21 16:44:33, Boyang Xue wrote:
> Hi Roman,
>
> On Wed, Jul 14, 2021 at 12:12 PM Roman Gushchin <guro@fb.com> wrote:
> >
> > On Wed, Jul 14, 2021 at 11:21:12AM +0800, Boyang Xue wrote:
> > > Hello,
> > >
> > > I'm not sure if this is the right place to report this bug, please
> > > correct me if I'm wrong.
> > >
> > > I found kernel-5.14.0-rc1 (built from the Linus tree) crash when it's
> > > running xfstests generic/256 on ext4 [1]. Looking at the call trace,
> > > it looks like the bug had been introduced by the commit
> > >
> > > c22d70a162d3 writeback, cgroup: release dying cgwbs by switching attached inodes
> > >
> > > It only happens on aarch64, not on x86_64, ppc64le and s390x. Testing
> > > was performed with the latest xfstests, and the bug can be reproduced
> > > on ext{2, 3, 4} with {1k, 2k, 4k} block sizes.
> >
> > Hello Boyang,
> >
> > thank you for the report!
> >
> > Do you know on which line the oops happens?
>
> I was trying to inspect the vmcore with crash utility, but
> unfortunately it doesn't work.
Thanks for report! Have you tried addr2line utility? Looking at the oops I
can see:
[ 4371.307867] pc : cleanup_offline_cgwbs_workfn+0x320/0x394
Which means there's probably heavy inlining going on (do you use LTO by
any chance?) because I don't think cleanup_offline_cgwbs_workfn() itself
would compile into ~1k of code (but I don't have much experience with
aarch64). Anyway, add2line should tell us.
Also pasting oops into scripts/decodecode on aarch64 machine should tell
us more about where and why the kernel crashed.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-14 9:26 ` Jan Kara
@ 2021-07-14 16:22 ` Boyang Xue
2021-07-14 23:46 ` Roman Gushchin
2021-07-15 2:35 ` Matthew Wilcox
0 siblings, 2 replies; 21+ messages in thread
From: Boyang Xue @ 2021-07-14 16:22 UTC (permalink / raw)
To: Jan Kara; +Cc: Roman Gushchin, linux-fsdevel
Hi Jan,
On Wed, Jul 14, 2021 at 5:26 PM Jan Kara <jack@suse.cz> wrote:
>
> On Wed 14-07-21 16:44:33, Boyang Xue wrote:
> > Hi Roman,
> >
> > On Wed, Jul 14, 2021 at 12:12 PM Roman Gushchin <guro@fb.com> wrote:
> > >
> > > On Wed, Jul 14, 2021 at 11:21:12AM +0800, Boyang Xue wrote:
> > > > Hello,
> > > >
> > > > I'm not sure if this is the right place to report this bug, please
> > > > correct me if I'm wrong.
> > > >
> > > > I found kernel-5.14.0-rc1 (built from the Linus tree) crash when it's
> > > > running xfstests generic/256 on ext4 [1]. Looking at the call trace,
> > > > it looks like the bug had been introduced by the commit
> > > >
> > > > c22d70a162d3 writeback, cgroup: release dying cgwbs by switching attached inodes
> > > >
> > > > It only happens on aarch64, not on x86_64, ppc64le and s390x. Testing
> > > > was performed with the latest xfstests, and the bug can be reproduced
> > > > on ext{2, 3, 4} with {1k, 2k, 4k} block sizes.
> > >
> > > Hello Boyang,
> > >
> > > thank you for the report!
> > >
> > > Do you know on which line the oops happens?
> >
> > I was trying to inspect the vmcore with crash utility, but
> > unfortunately it doesn't work.
>
> Thanks for report! Have you tried addr2line utility? Looking at the oops I
> can see:
Thanks for the tips!
It's unclear to me that where to find the required address in the
addr2line command line, i.e.
addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
<what address here?>
But I have tried gdb like this,
# gdb /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
GNU gdb (GDB) Red Hat Enterprise Linux 10.1-14.el9
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from
/usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux...
(gdb) list *(cleanup_offline_cgwbs_workfn+0x320)
0xffff8000102d6ddc is in cleanup_offline_cgwbs_workfn
(./arch/arm64/include/asm/jump_label.h:38).
33 }
34
35 static __always_inline bool arch_static_branch_jump(struct
static_key *key,
36 bool branch)
37 {
38 asm_volatile_goto(
39 "1: b %l[l_yes] \n\t"
40 " .pushsection __jump_table, \"aw\" \n\t"
41 " .align 3 \n\t"
42 " .long 1b - ., %l[l_yes] - . \n\t"
(gdb)
I'm not sure is it meaningful?
>
> [ 4371.307867] pc : cleanup_offline_cgwbs_workfn+0x320/0x394
>
> Which means there's probably heavy inlining going on (do you use LTO by
> any chance?) because I don't think cleanup_offline_cgwbs_workfn() itself
> would compile into ~1k of code (but I don't have much experience with
> aarch64). Anyway, add2line should tell us.
Actually I built the kernel on an internal build service, so I don't
know much of the build details, like LTO.
>
> Also pasting oops into scripts/decodecode on aarch64 machine should tell
> us more about where and why the kernel crashed.
The output is:
# echo "Code: d63f0020 97f99963 17ffffa6 f8588263 (f9400061)" |
/usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/decodecode
Code: d63f0020 97f99963 17ffffa6 f8588263 (f9400061)
All code
========
0: d63f0020 blr x1
4: 97f99963 bl 0xffffffffffe66590
8: 17ffffa6 b 0xfffffffffffffea0
c: f8588263 ldur x3, [x19, #-120]
10:* f9400061 ldr x1, [x3] <-- trapping instruction
Code starting with the faulting instruction
===========================================
0: f9400061 ldr x1, [x3]
>
> Honza
>
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR
>
Thanks,
Boyang
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-14 16:22 ` Boyang Xue
@ 2021-07-14 23:46 ` Roman Gushchin
2021-07-15 1:42 ` Boyang Xue
2021-07-15 2:35 ` Matthew Wilcox
1 sibling, 1 reply; 21+ messages in thread
From: Roman Gushchin @ 2021-07-14 23:46 UTC (permalink / raw)
To: Boyang Xue; +Cc: Jan Kara, linux-fsdevel
On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> Hi Jan,
>
> On Wed, Jul 14, 2021 at 5:26 PM Jan Kara <jack@suse.cz> wrote:
> >
> > On Wed 14-07-21 16:44:33, Boyang Xue wrote:
> > > Hi Roman,
> > >
> > > On Wed, Jul 14, 2021 at 12:12 PM Roman Gushchin <guro@fb.com> wrote:
> > > >
> > > > On Wed, Jul 14, 2021 at 11:21:12AM +0800, Boyang Xue wrote:
> > > > > Hello,
> > > > >
> > > > > I'm not sure if this is the right place to report this bug, please
> > > > > correct me if I'm wrong.
> > > > >
> > > > > I found kernel-5.14.0-rc1 (built from the Linus tree) crash when it's
> > > > > running xfstests generic/256 on ext4 [1]. Looking at the call trace,
> > > > > it looks like the bug had been introduced by the commit
> > > > >
> > > > > c22d70a162d3 writeback, cgroup: release dying cgwbs by switching attached inodes
> > > > >
> > > > > It only happens on aarch64, not on x86_64, ppc64le and s390x. Testing
> > > > > was performed with the latest xfstests, and the bug can be reproduced
> > > > > on ext{2, 3, 4} with {1k, 2k, 4k} block sizes.
> > > >
> > > > Hello Boyang,
> > > >
> > > > thank you for the report!
> > > >
> > > > Do you know on which line the oops happens?
> > >
> > > I was trying to inspect the vmcore with crash utility, but
> > > unfortunately it doesn't work.
> >
> > Thanks for report! Have you tried addr2line utility? Looking at the oops I
> > can see:
>
> Thanks for the tips!
>
> It's unclear to me that where to find the required address in the
> addr2line command line, i.e.
>
> addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> <what address here?>
You can use $nm <vmlinux> to get an address of cleanup_offline_cgwbs_workfn()
and then add 0x320.
Alternatively, maybe you can put the image you're using somewhere?
I'm working on getting my arm64 setup and reproduce the problem, but it takes
time, and I'm not sure I'll be able to reproduce it in qemu running on top of x86.
Thanks!
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-14 23:46 ` Roman Gushchin
@ 2021-07-15 1:42 ` Boyang Xue
2021-07-15 9:31 ` Jan Kara
0 siblings, 1 reply; 21+ messages in thread
From: Boyang Xue @ 2021-07-15 1:42 UTC (permalink / raw)
To: Roman Gushchin; +Cc: Jan Kara, linux-fsdevel
On Thu, Jul 15, 2021 at 7:46 AM Roman Gushchin <guro@fb.com> wrote:
>
> On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > Hi Jan,
> >
> > On Wed, Jul 14, 2021 at 5:26 PM Jan Kara <jack@suse.cz> wrote:
> > >
> > > On Wed 14-07-21 16:44:33, Boyang Xue wrote:
> > > > Hi Roman,
> > > >
> > > > On Wed, Jul 14, 2021 at 12:12 PM Roman Gushchin <guro@fb.com> wrote:
> > > > >
> > > > > On Wed, Jul 14, 2021 at 11:21:12AM +0800, Boyang Xue wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I'm not sure if this is the right place to report this bug, please
> > > > > > correct me if I'm wrong.
> > > > > >
> > > > > > I found kernel-5.14.0-rc1 (built from the Linus tree) crash when it's
> > > > > > running xfstests generic/256 on ext4 [1]. Looking at the call trace,
> > > > > > it looks like the bug had been introduced by the commit
> > > > > >
> > > > > > c22d70a162d3 writeback, cgroup: release dying cgwbs by switching attached inodes
> > > > > >
> > > > > > It only happens on aarch64, not on x86_64, ppc64le and s390x. Testing
> > > > > > was performed with the latest xfstests, and the bug can be reproduced
> > > > > > on ext{2, 3, 4} with {1k, 2k, 4k} block sizes.
> > > > >
> > > > > Hello Boyang,
> > > > >
> > > > > thank you for the report!
> > > > >
> > > > > Do you know on which line the oops happens?
> > > >
> > > > I was trying to inspect the vmcore with crash utility, but
> > > > unfortunately it doesn't work.
> > >
> > > Thanks for report! Have you tried addr2line utility? Looking at the oops I
> > > can see:
> >
> > Thanks for the tips!
> >
> > It's unclear to me that where to find the required address in the
> > addr2line command line, i.e.
> >
> > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > <what address here?>
>
> You can use $nm <vmlinux> to get an address of cleanup_offline_cgwbs_workfn()
> and then add 0x320.
Thanks! Hope the following helps:
# grep cleanup_offline_cgwbs_workfn
/boot/System.map-5.14.0-0.rc1.15.bx.el9.aarch64
ffff8000102d6ab0 t cleanup_offline_cgwbs_workfn
## ffff8000102d6ab0+0x320=FFFF8000102D6DD0
# addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
FFFF8000102D6DD0
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265
# vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h
```
arch_atomic64_fetch_add_unless(atomic64_t *v, s64 a, s64 u)
{
s64 c = arch_atomic64_read(v); <=== line#2265
do {
if (unlikely(c == u))
break;
} while (!arch_atomic64_try_cmpxchg(v, &c, c + a));
return c;
}
```
# addr2line -i -e
/usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
FFFF8000102D6DD0
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679
# vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c
```
static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
{
struct bdi_writeback *wb;
LIST_HEAD(processed);
spin_lock_irq(&cgwb_lock);
while (!list_empty(&offline_cgwbs)) {
wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
offline_node);
list_move(&wb->offline_node, &processed);
/*
* If wb is dirty, cleaning up the writeback by switching
* attached inodes will result in an effective removal of any
* bandwidth restrictions, which isn't the goal. Instead,
* it can be postponed until the next time, when all io
* will be likely completed. If in the meantime some inodes
* will get re-dirtied, they should be eventually switched to
* a new cgwb.
*/
if (wb_has_dirty_io(wb))
continue;
if (!wb_tryget(wb)) <=== line#679
continue;
spin_unlock_irq(&cgwb_lock);
while (cleanup_offline_cgwb(wb))
cond_resched();
spin_lock_irq(&cgwb_lock);
wb_put(wb);
}
if (!list_empty(&processed))
list_splice_tail(&processed, &offline_cgwbs);
spin_unlock_irq(&cgwb_lock);
}
```
>
> Alternatively, maybe you can put the image you're using somewhere?
I put those rpms in the Google Drive
https://drive.google.com/drive/folders/1aw-WK2yWD11UWB059bJt6WKNW1OP_fex?usp=sharing
>
> I'm working on getting my arm64 setup and reproduce the problem, but it takes
> time, and I'm not sure I'll be able to reproduce it in qemu running on top of x86.
Thanks! It's only reproducible on aarch64 and ppc64le in my test. I'm
happy to help test patch, if it would help.
>
> Thanks!
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-14 16:22 ` Boyang Xue
2021-07-14 23:46 ` Roman Gushchin
@ 2021-07-15 2:35 ` Matthew Wilcox
2021-07-15 3:51 ` Boyang Xue
1 sibling, 1 reply; 21+ messages in thread
From: Matthew Wilcox @ 2021-07-15 2:35 UTC (permalink / raw)
To: Boyang Xue; +Cc: Jan Kara, Roman Gushchin, linux-fsdevel
On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> It's unclear to me that where to find the required address in the
> addr2line command line, i.e.
>
> addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> <what address here?>
./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-15 2:35 ` Matthew Wilcox
@ 2021-07-15 3:51 ` Boyang Xue
2021-07-15 17:10 ` Darrick J. Wong
0 siblings, 1 reply; 21+ messages in thread
From: Boyang Xue @ 2021-07-15 3:51 UTC (permalink / raw)
To: Matthew Wilcox; +Cc: Jan Kara, Roman Gushchin, linux-fsdevel
On Thu, Jul 15, 2021 at 10:36 AM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > It's unclear to me that where to find the required address in the
> > addr2line command line, i.e.
> >
> > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > <what address here?>
>
> ./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394
>
Thanks! The result is the same as the
addr2line -i -e
/usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
FFFF8000102D6DD0
But this script is very handy.
# /usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/faddr2line
/usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
cleanup_offlin
e_cgwbs_workfn+0x320/0x394
cleanup_offline_cgwbs_workfn+0x320/0x394:
arch_atomic64_fetch_add_unless at
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265
(inlined by) arch_atomic64_add_unless at
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290
(inlined by) atomic64_add_unless at
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149
(inlined by) atomic_long_add_unless at
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491
(inlined by) percpu_ref_tryget_many at
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247
(inlined by) percpu_ref_tryget at
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266
(inlined by) wb_tryget at
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227
(inlined by) wb_tryget at
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224
(inlined by) cleanup_offline_cgwbs_workfn at
/usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679
# vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c
```
static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
{
struct bdi_writeback *wb;
LIST_HEAD(processed);
spin_lock_irq(&cgwb_lock);
while (!list_empty(&offline_cgwbs)) {
wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
offline_node);
list_move(&wb->offline_node, &processed);
/*
* If wb is dirty, cleaning up the writeback by switching
* attached inodes will result in an effective removal of any
* bandwidth restrictions, which isn't the goal. Instead,
* it can be postponed until the next time, when all io
* will be likely completed. If in the meantime some inodes
* will get re-dirtied, they should be eventually switched to
* a new cgwb.
*/
if (wb_has_dirty_io(wb))
continue;
if (!wb_tryget(wb)) <=== line#679
continue;
spin_unlock_irq(&cgwb_lock);
while (cleanup_offline_cgwb(wb))
cond_resched();
spin_lock_irq(&cgwb_lock);
wb_put(wb);
}
if (!list_empty(&processed))
list_splice_tail(&processed, &offline_cgwbs);
spin_unlock_irq(&cgwb_lock);
}
```
BTW, this bug can be only reproduced on a non-debug production built
kernel (a.k.a kernel rpm package), it's not reproducible on a debug
build with various debug configuration enabled (a.k.a kernel-debug rpm
package)
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-15 1:42 ` Boyang Xue
@ 2021-07-15 9:31 ` Jan Kara
2021-07-15 16:04 ` Roman Gushchin
0 siblings, 1 reply; 21+ messages in thread
From: Jan Kara @ 2021-07-15 9:31 UTC (permalink / raw)
To: Boyang Xue; +Cc: Roman Gushchin, Jan Kara, linux-fsdevel
On Thu 15-07-21 09:42:06, Boyang Xue wrote:
> On Thu, Jul 15, 2021 at 7:46 AM Roman Gushchin <guro@fb.com> wrote:
> >
> > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > > Hi Jan,
> > >
> > > On Wed, Jul 14, 2021 at 5:26 PM Jan Kara <jack@suse.cz> wrote:
> > > >
> > > > On Wed 14-07-21 16:44:33, Boyang Xue wrote:
> > > > > Hi Roman,
> > > > >
> > > > > On Wed, Jul 14, 2021 at 12:12 PM Roman Gushchin <guro@fb.com> wrote:
> > > > > >
> > > > > > On Wed, Jul 14, 2021 at 11:21:12AM +0800, Boyang Xue wrote:
> > > > > > > Hello,
> > > > > > >
> > > > > > > I'm not sure if this is the right place to report this bug, please
> > > > > > > correct me if I'm wrong.
> > > > > > >
> > > > > > > I found kernel-5.14.0-rc1 (built from the Linus tree) crash when it's
> > > > > > > running xfstests generic/256 on ext4 [1]. Looking at the call trace,
> > > > > > > it looks like the bug had been introduced by the commit
> > > > > > >
> > > > > > > c22d70a162d3 writeback, cgroup: release dying cgwbs by switching attached inodes
> > > > > > >
> > > > > > > It only happens on aarch64, not on x86_64, ppc64le and s390x. Testing
> > > > > > > was performed with the latest xfstests, and the bug can be reproduced
> > > > > > > on ext{2, 3, 4} with {1k, 2k, 4k} block sizes.
> > > > > >
> > > > > > Hello Boyang,
> > > > > >
> > > > > > thank you for the report!
> > > > > >
> > > > > > Do you know on which line the oops happens?
> > > > >
> > > > > I was trying to inspect the vmcore with crash utility, but
> > > > > unfortunately it doesn't work.
> > > >
> > > > Thanks for report! Have you tried addr2line utility? Looking at the oops I
> > > > can see:
> > >
> > > Thanks for the tips!
> > >
> > > It's unclear to me that where to find the required address in the
> > > addr2line command line, i.e.
> > >
> > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > <what address here?>
> >
> > You can use $nm <vmlinux> to get an address of cleanup_offline_cgwbs_workfn()
> > and then add 0x320.
>
> Thanks! Hope the following helps:
Thanks for the data!
> static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
> {
> struct bdi_writeback *wb;
> LIST_HEAD(processed);
>
> spin_lock_irq(&cgwb_lock);
>
> while (!list_empty(&offline_cgwbs)) {
> wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
> offline_node);
> list_move(&wb->offline_node, &processed);
>
> /*
> * If wb is dirty, cleaning up the writeback by switching
> * attached inodes will result in an effective removal of any
> * bandwidth restrictions, which isn't the goal. Instead,
> * it can be postponed until the next time, when all io
> * will be likely completed. If in the meantime some inodes
> * will get re-dirtied, they should be eventually switched to
> * a new cgwb.
> */
> if (wb_has_dirty_io(wb))
> continue;
>
> if (!wb_tryget(wb)) <=== line#679
> continue;
Aha, interesting. So it seems we crashed trying to dereference
wb->refcnt->data. So it looks like cgwb_release_workfn() raced with
cleanup_offline_cgwbs_workfn() and percpu_ref_exit() got called from
cgwb_release_workfn() and then cleanup_offline_cgwbs_workfn() called
wb_tryget(). I think the proper fix is to move:
spin_lock_irq(&cgwb_lock);
list_del(&wb->offline_node);
spin_unlock_irq(&cgwb_lock);
in cgwb_release_workfn() to the beginning of that function so that we are
sure even cleanup_offline_cgwbs_workfn() cannot be working with the wb when
it is being released. Roman?
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-15 9:31 ` Jan Kara
@ 2021-07-15 16:04 ` Roman Gushchin
2021-07-16 1:37 ` Boyang Xue
0 siblings, 1 reply; 21+ messages in thread
From: Roman Gushchin @ 2021-07-15 16:04 UTC (permalink / raw)
To: Jan Kara, Boyang Xue; +Cc: linux-fsdevel
On Thu, Jul 15, 2021 at 11:31:17AM +0200, Jan Kara wrote:
> On Thu 15-07-21 09:42:06, Boyang Xue wrote:
> > On Thu, Jul 15, 2021 at 7:46 AM Roman Gushchin <guro@fb.com> wrote:
> > >
> > > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > > > Hi Jan,
> > > >
> > > > On Wed, Jul 14, 2021 at 5:26 PM Jan Kara <jack@suse.cz> wrote:
> > > > >
> > > > > On Wed 14-07-21 16:44:33, Boyang Xue wrote:
> > > > > > Hi Roman,
> > > > > >
> > > > > > On Wed, Jul 14, 2021 at 12:12 PM Roman Gushchin <guro@fb.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jul 14, 2021 at 11:21:12AM +0800, Boyang Xue wrote:
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I'm not sure if this is the right place to report this bug, please
> > > > > > > > correct me if I'm wrong.
> > > > > > > >
> > > > > > > > I found kernel-5.14.0-rc1 (built from the Linus tree) crash when it's
> > > > > > > > running xfstests generic/256 on ext4 [1]. Looking at the call trace,
> > > > > > > > it looks like the bug had been introduced by the commit
> > > > > > > >
> > > > > > > > c22d70a162d3 writeback, cgroup: release dying cgwbs by switching attached inodes
> > > > > > > >
> > > > > > > > It only happens on aarch64, not on x86_64, ppc64le and s390x. Testing
> > > > > > > > was performed with the latest xfstests, and the bug can be reproduced
> > > > > > > > on ext{2, 3, 4} with {1k, 2k, 4k} block sizes.
> > > > > > >
> > > > > > > Hello Boyang,
> > > > > > >
> > > > > > > thank you for the report!
> > > > > > >
> > > > > > > Do you know on which line the oops happens?
> > > > > >
> > > > > > I was trying to inspect the vmcore with crash utility, but
> > > > > > unfortunately it doesn't work.
> > > > >
> > > > > Thanks for report! Have you tried addr2line utility? Looking at the oops I
> > > > > can see:
> > > >
> > > > Thanks for the tips!
> > > >
> > > > It's unclear to me that where to find the required address in the
> > > > addr2line command line, i.e.
> > > >
> > > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > <what address here?>
> > >
> > > You can use $nm <vmlinux> to get an address of cleanup_offline_cgwbs_workfn()
> > > and then add 0x320.
> >
> > Thanks! Hope the following helps:
>
> Thanks for the data!
>
> > static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
> > {
> > struct bdi_writeback *wb;
> > LIST_HEAD(processed);
> >
> > spin_lock_irq(&cgwb_lock);
> >
> > while (!list_empty(&offline_cgwbs)) {
> > wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
> > offline_node);
> > list_move(&wb->offline_node, &processed);
> >
> > /*
> > * If wb is dirty, cleaning up the writeback by switching
> > * attached inodes will result in an effective removal of any
> > * bandwidth restrictions, which isn't the goal. Instead,
> > * it can be postponed until the next time, when all io
> > * will be likely completed. If in the meantime some inodes
> > * will get re-dirtied, they should be eventually switched to
> > * a new cgwb.
> > */
> > if (wb_has_dirty_io(wb))
> > continue;
> >
> > if (!wb_tryget(wb)) <=== line#679
> > continue;
>
> Aha, interesting. So it seems we crashed trying to dereference
> wb->refcnt->data. So it looks like cgwb_release_workfn() raced with
> cleanup_offline_cgwbs_workfn() and percpu_ref_exit() got called from
> cgwb_release_workfn() and then cleanup_offline_cgwbs_workfn() called
> wb_tryget(). I think the proper fix is to move:
>
> spin_lock_irq(&cgwb_lock);
> list_del(&wb->offline_node);
> spin_unlock_irq(&cgwb_lock);
>
> in cgwb_release_workfn() to the beginning of that function so that we are
> sure even cleanup_offline_cgwbs_workfn() cannot be working with the wb when
> it is being released. Roman?
Yes, it sounds like the most reasonable explanation.
Thank you!
Boyang, would you mind to test the following patch?
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 271f2ca862c8..f5561ea7d90a 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -398,12 +398,12 @@ static void cgwb_release_workfn(struct work_struct *work)
blkcg_unpin_online(blkcg);
fprop_local_destroy_percpu(&wb->memcg_completions);
- percpu_ref_exit(&wb->refcnt);
spin_lock_irq(&cgwb_lock);
list_del(&wb->offline_node);
spin_unlock_irq(&cgwb_lock);
+ percpu_ref_exit(&wb->refcnt);
wb_exit(wb);
WARN_ON_ONCE(!list_empty(&wb->b_attached));
kfree_rcu(wb, rcu);
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-15 3:51 ` Boyang Xue
@ 2021-07-15 17:10 ` Darrick J. Wong
2021-07-15 20:08 ` Roman Gushchin
0 siblings, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2021-07-15 17:10 UTC (permalink / raw)
To: Boyang Xue; +Cc: Matthew Wilcox, Jan Kara, Roman Gushchin, linux-fsdevel
On Thu, Jul 15, 2021 at 11:51:50AM +0800, Boyang Xue wrote:
> On Thu, Jul 15, 2021 at 10:36 AM Matthew Wilcox <willy@infradead.org> wrote:
> >
> > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > > It's unclear to me that where to find the required address in the
> > > addr2line command line, i.e.
> > >
> > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > <what address here?>
> >
> > ./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394
> >
>
> Thanks! The result is the same as the
>
> addr2line -i -e
> /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> FFFF8000102D6DD0
>
> But this script is very handy.
>
> # /usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/faddr2line
> /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> cleanup_offlin
> e_cgwbs_workfn+0x320/0x394
> cleanup_offline_cgwbs_workfn+0x320/0x394:
> arch_atomic64_fetch_add_unless at
> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265
> (inlined by) arch_atomic64_add_unless at
> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290
> (inlined by) atomic64_add_unless at
> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149
> (inlined by) atomic_long_add_unless at
> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491
> (inlined by) percpu_ref_tryget_many at
> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247
> (inlined by) percpu_ref_tryget at
> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266
> (inlined by) wb_tryget at
> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227
> (inlined by) wb_tryget at
> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224
> (inlined by) cleanup_offline_cgwbs_workfn at
> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679
>
> # vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c
> ```
> static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
> {
> struct bdi_writeback *wb;
> LIST_HEAD(processed);
>
> spin_lock_irq(&cgwb_lock);
>
> while (!list_empty(&offline_cgwbs)) {
> wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
> offline_node);
> list_move(&wb->offline_node, &processed);
>
> /*
> * If wb is dirty, cleaning up the writeback by switching
> * attached inodes will result in an effective removal of any
> * bandwidth restrictions, which isn't the goal. Instead,
> * it can be postponed until the next time, when all io
> * will be likely completed. If in the meantime some inodes
> * will get re-dirtied, they should be eventually switched to
> * a new cgwb.
> */
> if (wb_has_dirty_io(wb))
> continue;
>
> if (!wb_tryget(wb)) <=== line#679
> continue;
>
> spin_unlock_irq(&cgwb_lock);
> while (cleanup_offline_cgwb(wb))
> cond_resched();
> spin_lock_irq(&cgwb_lock);
>
> wb_put(wb);
> }
>
> if (!list_empty(&processed))
> list_splice_tail(&processed, &offline_cgwbs);
>
> spin_unlock_irq(&cgwb_lock);
> }
> ```
>
> BTW, this bug can be only reproduced on a non-debug production built
> kernel (a.k.a kernel rpm package), it's not reproducible on a debug
> build with various debug configuration enabled (a.k.a kernel-debug rpm
> package)
FWIW I've also seen this regularly on x86_64 kernels on ext4 with all
default mkfs settings when running generic/256.
# FSTYP=ext4 MOUNT_OPTIONS="-o acl,user_xattr," ./check
FSTYP -- ext4
PLATFORM -- Linux/x86_64 flax-mtr00 5.14.0-rc1-xfsx #rc1 SMP
PREEMPT Wed Jul 14 17:36:18 PDT 2021
MKFS_OPTIONS -- /dev/sdf
MOUNT_OPTIONS -- -o acl,user_xattr, /dev/sdf /opt
generic/256
Message from syslogd@flax-mtr00 at Jul 15 09:58:14 ...
kernel:[ 2508.987522] Dumping ftrace buffer:
And the dmesg looks like:
run fstests generic/256 at 2021-07-15 09:56:34
EXT4-fs (sdf): mounted filesystem with ordered data mode. Opts: acl,user_xattr. Quota mode: none.
BUG: kernel NULL pointer dereference, address: 0000000000000000
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 1 PID: 108604 Comm: u9:3 Not tainted 5.14.0-rc1-xfsx #rc1 486fb938eb99d57e79080268009b49f63f777aec
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1.1 04/01/2014
Workqueue: events_unbound cleanup_offline_cgwbs_workfn
RIP: 0010:cleanup_offline_cgwbs_workfn+0x1ef/0x220
Code: ff ff f0 48 83 28 01 0f 85 55 ff ff ff 48 8b 83 60 ff ff ff 48 8d bb 58 ff ff ff ff 50 08 e9 3f ff ff ff 48 8b 93 60 ff ff ff <48> 8b 02 48 85 c0 0f 84 2c ff ff ff 48 8d 48 01 f0 48 0f b1 0a 75
RSP: 0018:ffffc9000278be60 EFLAGS: 00010006
RAX: 0000000000000003 RBX: ffff888282dc0b30 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffc9000278be60 RDI: ffff888282dc0b30
RBP: ffff888282dc0800 R08: ffff88828006af30 R09: ffff88828006af30
R10: 000000000000000f R11: 000000000000000f R12: ffffc9000278be60
R13: ffff8881000d6800 R14: 0000000000000000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff888277d00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000102262003 CR4: 00000000001706a0
Call Trace:
process_one_work+0x1dd/0x3c0
worker_thread+0x53/0x3c0
? rescuer_thread+0x390/0x390
kthread+0x149/0x170
? set_kthread_struct+0x40/0x40
ret_from_fork+0x1f/0x30
Modules linked in: ext2 ext4 jbd2 dm_flakey mbcache xfs libcrc32c ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpudp ip_set_hash_ip ip_set_hash_net xt_set ip_set_hash_mac ip_set nfnetlink ip6table_filter ip6_tables bfq iptable_filter pvpanic_mmio pvpanic sch_fq_codel ip_tables x_tables overlay nfsv4 af_packet [last unloaded: jbd2]
Dumping ftrace buffer:
(ftrace buffer empty)
CR2: 0000000000000000
---[ end trace 242113b767739fb9 ]---
The faddr2line output points at the same line of code.
--D
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-15 17:10 ` Darrick J. Wong
@ 2021-07-15 20:08 ` Roman Gushchin
2021-07-15 22:28 ` Darrick J. Wong
0 siblings, 1 reply; 21+ messages in thread
From: Roman Gushchin @ 2021-07-15 20:08 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: Boyang Xue, Matthew Wilcox, Jan Kara, linux-fsdevel
On Thu, Jul 15, 2021 at 10:10:50AM -0700, Darrick J. Wong wrote:
> On Thu, Jul 15, 2021 at 11:51:50AM +0800, Boyang Xue wrote:
> > On Thu, Jul 15, 2021 at 10:36 AM Matthew Wilcox <willy@infradead.org> wrote:
> > >
> > > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > > > It's unclear to me that where to find the required address in the
> > > > addr2line command line, i.e.
> > > >
> > > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > <what address here?>
> > >
> > > ./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394
> > >
> >
> > Thanks! The result is the same as the
> >
> > addr2line -i -e
> > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > FFFF8000102D6DD0
> >
> > But this script is very handy.
> >
> > # /usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/faddr2line
> > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > cleanup_offlin
> > e_cgwbs_workfn+0x320/0x394
> > cleanup_offline_cgwbs_workfn+0x320/0x394:
> > arch_atomic64_fetch_add_unless at
> > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265
> > (inlined by) arch_atomic64_add_unless at
> > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290
> > (inlined by) atomic64_add_unless at
> > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149
> > (inlined by) atomic_long_add_unless at
> > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491
> > (inlined by) percpu_ref_tryget_many at
> > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247
> > (inlined by) percpu_ref_tryget at
> > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266
> > (inlined by) wb_tryget at
> > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227
> > (inlined by) wb_tryget at
> > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224
> > (inlined by) cleanup_offline_cgwbs_workfn at
> > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679
> >
> > # vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c
> > ```
> > static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
> > {
> > struct bdi_writeback *wb;
> > LIST_HEAD(processed);
> >
> > spin_lock_irq(&cgwb_lock);
> >
> > while (!list_empty(&offline_cgwbs)) {
> > wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
> > offline_node);
> > list_move(&wb->offline_node, &processed);
> >
> > /*
> > * If wb is dirty, cleaning up the writeback by switching
> > * attached inodes will result in an effective removal of any
> > * bandwidth restrictions, which isn't the goal. Instead,
> > * it can be postponed until the next time, when all io
> > * will be likely completed. If in the meantime some inodes
> > * will get re-dirtied, they should be eventually switched to
> > * a new cgwb.
> > */
> > if (wb_has_dirty_io(wb))
> > continue;
> >
> > if (!wb_tryget(wb)) <=== line#679
> > continue;
> >
> > spin_unlock_irq(&cgwb_lock);
> > while (cleanup_offline_cgwb(wb))
> > cond_resched();
> > spin_lock_irq(&cgwb_lock);
> >
> > wb_put(wb);
> > }
> >
> > if (!list_empty(&processed))
> > list_splice_tail(&processed, &offline_cgwbs);
> >
> > spin_unlock_irq(&cgwb_lock);
> > }
> > ```
> >
> > BTW, this bug can be only reproduced on a non-debug production built
> > kernel (a.k.a kernel rpm package), it's not reproducible on a debug
> > build with various debug configuration enabled (a.k.a kernel-debug rpm
> > package)
>
> FWIW I've also seen this regularly on x86_64 kernels on ext4 with all
> default mkfs settings when running generic/256.
Oh, that's a useful information, thank you!
Btw, would you mind to give a patch from an earlier message in the thread
a test? I'd highly appreciate it.
Thanks!
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-15 20:08 ` Roman Gushchin
@ 2021-07-15 22:28 ` Darrick J. Wong
2021-07-16 16:23 ` Darrick J. Wong
0 siblings, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2021-07-15 22:28 UTC (permalink / raw)
To: Roman Gushchin; +Cc: Boyang Xue, Matthew Wilcox, Jan Kara, linux-fsdevel
On Thu, Jul 15, 2021 at 01:08:15PM -0700, Roman Gushchin wrote:
> On Thu, Jul 15, 2021 at 10:10:50AM -0700, Darrick J. Wong wrote:
> > On Thu, Jul 15, 2021 at 11:51:50AM +0800, Boyang Xue wrote:
> > > On Thu, Jul 15, 2021 at 10:36 AM Matthew Wilcox <willy@infradead.org> wrote:
> > > >
> > > > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > > > > It's unclear to me that where to find the required address in the
> > > > > addr2line command line, i.e.
> > > > >
> > > > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > <what address here?>
> > > >
> > > > ./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394
> > > >
> > >
> > > Thanks! The result is the same as the
> > >
> > > addr2line -i -e
> > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > FFFF8000102D6DD0
> > >
> > > But this script is very handy.
> > >
> > > # /usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/faddr2line
> > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > cleanup_offlin
> > > e_cgwbs_workfn+0x320/0x394
> > > cleanup_offline_cgwbs_workfn+0x320/0x394:
> > > arch_atomic64_fetch_add_unless at
> > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265
> > > (inlined by) arch_atomic64_add_unless at
> > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290
> > > (inlined by) atomic64_add_unless at
> > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149
> > > (inlined by) atomic_long_add_unless at
> > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491
> > > (inlined by) percpu_ref_tryget_many at
> > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247
> > > (inlined by) percpu_ref_tryget at
> > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266
> > > (inlined by) wb_tryget at
> > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227
> > > (inlined by) wb_tryget at
> > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224
> > > (inlined by) cleanup_offline_cgwbs_workfn at
> > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679
> > >
> > > # vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c
> > > ```
> > > static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
> > > {
> > > struct bdi_writeback *wb;
> > > LIST_HEAD(processed);
> > >
> > > spin_lock_irq(&cgwb_lock);
> > >
> > > while (!list_empty(&offline_cgwbs)) {
> > > wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
> > > offline_node);
> > > list_move(&wb->offline_node, &processed);
> > >
> > > /*
> > > * If wb is dirty, cleaning up the writeback by switching
> > > * attached inodes will result in an effective removal of any
> > > * bandwidth restrictions, which isn't the goal. Instead,
> > > * it can be postponed until the next time, when all io
> > > * will be likely completed. If in the meantime some inodes
> > > * will get re-dirtied, they should be eventually switched to
> > > * a new cgwb.
> > > */
> > > if (wb_has_dirty_io(wb))
> > > continue;
> > >
> > > if (!wb_tryget(wb)) <=== line#679
> > > continue;
> > >
> > > spin_unlock_irq(&cgwb_lock);
> > > while (cleanup_offline_cgwb(wb))
> > > cond_resched();
> > > spin_lock_irq(&cgwb_lock);
> > >
> > > wb_put(wb);
> > > }
> > >
> > > if (!list_empty(&processed))
> > > list_splice_tail(&processed, &offline_cgwbs);
> > >
> > > spin_unlock_irq(&cgwb_lock);
> > > }
> > > ```
> > >
> > > BTW, this bug can be only reproduced on a non-debug production built
> > > kernel (a.k.a kernel rpm package), it's not reproducible on a debug
> > > build with various debug configuration enabled (a.k.a kernel-debug rpm
> > > package)
> >
> > FWIW I've also seen this regularly on x86_64 kernels on ext4 with all
> > default mkfs settings when running generic/256.
>
> Oh, that's a useful information, thank you!
>
> Btw, would you mind to give a patch from an earlier message in the thread
> a test? I'd highly appreciate it.
>
> Thanks!
Will do.
--D
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-15 16:04 ` Roman Gushchin
@ 2021-07-16 1:37 ` Boyang Xue
0 siblings, 0 replies; 21+ messages in thread
From: Boyang Xue @ 2021-07-16 1:37 UTC (permalink / raw)
To: Roman Gushchin; +Cc: Jan Kara, linux-fsdevel
On Fri, Jul 16, 2021 at 12:05 AM Roman Gushchin <guro@fb.com> wrote:
>
> On Thu, Jul 15, 2021 at 11:31:17AM +0200, Jan Kara wrote:
> > On Thu 15-07-21 09:42:06, Boyang Xue wrote:
> > > On Thu, Jul 15, 2021 at 7:46 AM Roman Gushchin <guro@fb.com> wrote:
> > > >
> > > > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > > > > Hi Jan,
> > > > >
> > > > > On Wed, Jul 14, 2021 at 5:26 PM Jan Kara <jack@suse.cz> wrote:
> > > > > >
> > > > > > On Wed 14-07-21 16:44:33, Boyang Xue wrote:
> > > > > > > Hi Roman,
> > > > > > >
> > > > > > > On Wed, Jul 14, 2021 at 12:12 PM Roman Gushchin <guro@fb.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jul 14, 2021 at 11:21:12AM +0800, Boyang Xue wrote:
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > I'm not sure if this is the right place to report this bug, please
> > > > > > > > > correct me if I'm wrong.
> > > > > > > > >
> > > > > > > > > I found kernel-5.14.0-rc1 (built from the Linus tree) crash when it's
> > > > > > > > > running xfstests generic/256 on ext4 [1]. Looking at the call trace,
> > > > > > > > > it looks like the bug had been introduced by the commit
> > > > > > > > >
> > > > > > > > > c22d70a162d3 writeback, cgroup: release dying cgwbs by switching attached inodes
> > > > > > > > >
> > > > > > > > > It only happens on aarch64, not on x86_64, ppc64le and s390x. Testing
> > > > > > > > > was performed with the latest xfstests, and the bug can be reproduced
> > > > > > > > > on ext{2, 3, 4} with {1k, 2k, 4k} block sizes.
> > > > > > > >
> > > > > > > > Hello Boyang,
> > > > > > > >
> > > > > > > > thank you for the report!
> > > > > > > >
> > > > > > > > Do you know on which line the oops happens?
> > > > > > >
> > > > > > > I was trying to inspect the vmcore with crash utility, but
> > > > > > > unfortunately it doesn't work.
> > > > > >
> > > > > > Thanks for report! Have you tried addr2line utility? Looking at the oops I
> > > > > > can see:
> > > > >
> > > > > Thanks for the tips!
> > > > >
> > > > > It's unclear to me that where to find the required address in the
> > > > > addr2line command line, i.e.
> > > > >
> > > > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > <what address here?>
> > > >
> > > > You can use $nm <vmlinux> to get an address of cleanup_offline_cgwbs_workfn()
> > > > and then add 0x320.
> > >
> > > Thanks! Hope the following helps:
> >
> > Thanks for the data!
> >
> > > static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
> > > {
> > > struct bdi_writeback *wb;
> > > LIST_HEAD(processed);
> > >
> > > spin_lock_irq(&cgwb_lock);
> > >
> > > while (!list_empty(&offline_cgwbs)) {
> > > wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
> > > offline_node);
> > > list_move(&wb->offline_node, &processed);
> > >
> > > /*
> > > * If wb is dirty, cleaning up the writeback by switching
> > > * attached inodes will result in an effective removal of any
> > > * bandwidth restrictions, which isn't the goal. Instead,
> > > * it can be postponed until the next time, when all io
> > > * will be likely completed. If in the meantime some inodes
> > > * will get re-dirtied, they should be eventually switched to
> > > * a new cgwb.
> > > */
> > > if (wb_has_dirty_io(wb))
> > > continue;
> > >
> > > if (!wb_tryget(wb)) <=== line#679
> > > continue;
> >
> > Aha, interesting. So it seems we crashed trying to dereference
> > wb->refcnt->data. So it looks like cgwb_release_workfn() raced with
> > cleanup_offline_cgwbs_workfn() and percpu_ref_exit() got called from
> > cgwb_release_workfn() and then cleanup_offline_cgwbs_workfn() called
> > wb_tryget(). I think the proper fix is to move:
> >
> > spin_lock_irq(&cgwb_lock);
> > list_del(&wb->offline_node);
> > spin_unlock_irq(&cgwb_lock);
> >
> > in cgwb_release_workfn() to the beginning of that function so that we are
> > sure even cleanup_offline_cgwbs_workfn() cannot be working with the wb when
> > it is being released. Roman?
>
> Yes, it sounds like the most reasonable explanation.
> Thank you!
>
> Boyang, would you mind to test the following patch?
No problem. I'm testing it. Thanks for the patch.
>
> diff --git a/mm/backing-dev.c b/mm/backing-dev.c
> index 271f2ca862c8..f5561ea7d90a 100644
> --- a/mm/backing-dev.c
> +++ b/mm/backing-dev.c
> @@ -398,12 +398,12 @@ static void cgwb_release_workfn(struct work_struct *work)
> blkcg_unpin_online(blkcg);
>
> fprop_local_destroy_percpu(&wb->memcg_completions);
> - percpu_ref_exit(&wb->refcnt);
>
> spin_lock_irq(&cgwb_lock);
> list_del(&wb->offline_node);
> spin_unlock_irq(&cgwb_lock);
>
> + percpu_ref_exit(&wb->refcnt);
> wb_exit(wb);
> WARN_ON_ONCE(!list_empty(&wb->b_attached));
> kfree_rcu(wb, rcu);
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-15 22:28 ` Darrick J. Wong
@ 2021-07-16 16:23 ` Darrick J. Wong
2021-07-16 20:03 ` Roman Gushchin
0 siblings, 1 reply; 21+ messages in thread
From: Darrick J. Wong @ 2021-07-16 16:23 UTC (permalink / raw)
To: Roman Gushchin; +Cc: Boyang Xue, Matthew Wilcox, Jan Kara, linux-fsdevel
On Thu, Jul 15, 2021 at 03:28:12PM -0700, Darrick J. Wong wrote:
> On Thu, Jul 15, 2021 at 01:08:15PM -0700, Roman Gushchin wrote:
> > On Thu, Jul 15, 2021 at 10:10:50AM -0700, Darrick J. Wong wrote:
> > > On Thu, Jul 15, 2021 at 11:51:50AM +0800, Boyang Xue wrote:
> > > > On Thu, Jul 15, 2021 at 10:36 AM Matthew Wilcox <willy@infradead.org> wrote:
> > > > >
> > > > > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > > > > > It's unclear to me that where to find the required address in the
> > > > > > addr2line command line, i.e.
> > > > > >
> > > > > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > > <what address here?>
> > > > >
> > > > > ./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394
> > > > >
> > > >
> > > > Thanks! The result is the same as the
> > > >
> > > > addr2line -i -e
> > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > FFFF8000102D6DD0
> > > >
> > > > But this script is very handy.
> > > >
> > > > # /usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/faddr2line
> > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > cleanup_offlin
> > > > e_cgwbs_workfn+0x320/0x394
> > > > cleanup_offline_cgwbs_workfn+0x320/0x394:
> > > > arch_atomic64_fetch_add_unless at
> > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265
> > > > (inlined by) arch_atomic64_add_unless at
> > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290
> > > > (inlined by) atomic64_add_unless at
> > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149
> > > > (inlined by) atomic_long_add_unless at
> > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491
> > > > (inlined by) percpu_ref_tryget_many at
> > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247
> > > > (inlined by) percpu_ref_tryget at
> > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266
> > > > (inlined by) wb_tryget at
> > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227
> > > > (inlined by) wb_tryget at
> > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224
> > > > (inlined by) cleanup_offline_cgwbs_workfn at
> > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679
> > > >
> > > > # vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c
> > > > ```
> > > > static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
> > > > {
> > > > struct bdi_writeback *wb;
> > > > LIST_HEAD(processed);
> > > >
> > > > spin_lock_irq(&cgwb_lock);
> > > >
> > > > while (!list_empty(&offline_cgwbs)) {
> > > > wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
> > > > offline_node);
> > > > list_move(&wb->offline_node, &processed);
> > > >
> > > > /*
> > > > * If wb is dirty, cleaning up the writeback by switching
> > > > * attached inodes will result in an effective removal of any
> > > > * bandwidth restrictions, which isn't the goal. Instead,
> > > > * it can be postponed until the next time, when all io
> > > > * will be likely completed. If in the meantime some inodes
> > > > * will get re-dirtied, they should be eventually switched to
> > > > * a new cgwb.
> > > > */
> > > > if (wb_has_dirty_io(wb))
> > > > continue;
> > > >
> > > > if (!wb_tryget(wb)) <=== line#679
> > > > continue;
> > > >
> > > > spin_unlock_irq(&cgwb_lock);
> > > > while (cleanup_offline_cgwb(wb))
> > > > cond_resched();
> > > > spin_lock_irq(&cgwb_lock);
> > > >
> > > > wb_put(wb);
> > > > }
> > > >
> > > > if (!list_empty(&processed))
> > > > list_splice_tail(&processed, &offline_cgwbs);
> > > >
> > > > spin_unlock_irq(&cgwb_lock);
> > > > }
> > > > ```
> > > >
> > > > BTW, this bug can be only reproduced on a non-debug production built
> > > > kernel (a.k.a kernel rpm package), it's not reproducible on a debug
> > > > build with various debug configuration enabled (a.k.a kernel-debug rpm
> > > > package)
> > >
> > > FWIW I've also seen this regularly on x86_64 kernels on ext4 with all
> > > default mkfs settings when running generic/256.
> >
> > Oh, that's a useful information, thank you!
> >
> > Btw, would you mind to give a patch from an earlier message in the thread
> > a test? I'd highly appreciate it.
> >
> > Thanks!
>
> Will do.
fstests passed here, so
Tested-by: Darrick J. Wong <djwong@kernel.org>
--D
>
> --D
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-16 16:23 ` Darrick J. Wong
@ 2021-07-16 20:03 ` Roman Gushchin
2021-07-17 12:00 ` Boyang Xue
0 siblings, 1 reply; 21+ messages in thread
From: Roman Gushchin @ 2021-07-16 20:03 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: Boyang Xue, Matthew Wilcox, Jan Kara, linux-fsdevel
On Fri, Jul 16, 2021 at 09:23:40AM -0700, Darrick J. Wong wrote:
> On Thu, Jul 15, 2021 at 03:28:12PM -0700, Darrick J. Wong wrote:
> > On Thu, Jul 15, 2021 at 01:08:15PM -0700, Roman Gushchin wrote:
> > > On Thu, Jul 15, 2021 at 10:10:50AM -0700, Darrick J. Wong wrote:
> > > > On Thu, Jul 15, 2021 at 11:51:50AM +0800, Boyang Xue wrote:
> > > > > On Thu, Jul 15, 2021 at 10:36 AM Matthew Wilcox <willy@infradead.org> wrote:
> > > > > >
> > > > > > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > > > > > > It's unclear to me that where to find the required address in the
> > > > > > > addr2line command line, i.e.
> > > > > > >
> > > > > > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > > > <what address here?>
> > > > > >
> > > > > > ./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394
> > > > > >
> > > > >
> > > > > Thanks! The result is the same as the
> > > > >
> > > > > addr2line -i -e
> > > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > FFFF8000102D6DD0
> > > > >
> > > > > But this script is very handy.
> > > > >
> > > > > # /usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/faddr2line
> > > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > cleanup_offlin
> > > > > e_cgwbs_workfn+0x320/0x394
> > > > > cleanup_offline_cgwbs_workfn+0x320/0x394:
> > > > > arch_atomic64_fetch_add_unless at
> > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265
> > > > > (inlined by) arch_atomic64_add_unless at
> > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290
> > > > > (inlined by) atomic64_add_unless at
> > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149
> > > > > (inlined by) atomic_long_add_unless at
> > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491
> > > > > (inlined by) percpu_ref_tryget_many at
> > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247
> > > > > (inlined by) percpu_ref_tryget at
> > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266
> > > > > (inlined by) wb_tryget at
> > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227
> > > > > (inlined by) wb_tryget at
> > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224
> > > > > (inlined by) cleanup_offline_cgwbs_workfn at
> > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679
> > > > >
> > > > > # vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c
> > > > > ```
> > > > > static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
> > > > > {
> > > > > struct bdi_writeback *wb;
> > > > > LIST_HEAD(processed);
> > > > >
> > > > > spin_lock_irq(&cgwb_lock);
> > > > >
> > > > > while (!list_empty(&offline_cgwbs)) {
> > > > > wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
> > > > > offline_node);
> > > > > list_move(&wb->offline_node, &processed);
> > > > >
> > > > > /*
> > > > > * If wb is dirty, cleaning up the writeback by switching
> > > > > * attached inodes will result in an effective removal of any
> > > > > * bandwidth restrictions, which isn't the goal. Instead,
> > > > > * it can be postponed until the next time, when all io
> > > > > * will be likely completed. If in the meantime some inodes
> > > > > * will get re-dirtied, they should be eventually switched to
> > > > > * a new cgwb.
> > > > > */
> > > > > if (wb_has_dirty_io(wb))
> > > > > continue;
> > > > >
> > > > > if (!wb_tryget(wb)) <=== line#679
> > > > > continue;
> > > > >
> > > > > spin_unlock_irq(&cgwb_lock);
> > > > > while (cleanup_offline_cgwb(wb))
> > > > > cond_resched();
> > > > > spin_lock_irq(&cgwb_lock);
> > > > >
> > > > > wb_put(wb);
> > > > > }
> > > > >
> > > > > if (!list_empty(&processed))
> > > > > list_splice_tail(&processed, &offline_cgwbs);
> > > > >
> > > > > spin_unlock_irq(&cgwb_lock);
> > > > > }
> > > > > ```
> > > > >
> > > > > BTW, this bug can be only reproduced on a non-debug production built
> > > > > kernel (a.k.a kernel rpm package), it's not reproducible on a debug
> > > > > build with various debug configuration enabled (a.k.a kernel-debug rpm
> > > > > package)
> > > >
> > > > FWIW I've also seen this regularly on x86_64 kernels on ext4 with all
> > > > default mkfs settings when running generic/256.
> > >
> > > Oh, that's a useful information, thank you!
> > >
> > > Btw, would you mind to give a patch from an earlier message in the thread
> > > a test? I'd highly appreciate it.
> > >
> > > Thanks!
> >
> > Will do.
>
> fstests passed here, so
>
> Tested-by: Darrick J. Wong <djwong@kernel.org>
Great, thank you!
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-16 20:03 ` Roman Gushchin
@ 2021-07-17 12:00 ` Boyang Xue
2021-07-22 5:29 ` Boyang Xue
0 siblings, 1 reply; 21+ messages in thread
From: Boyang Xue @ 2021-07-17 12:00 UTC (permalink / raw)
To: Roman Gushchin; +Cc: Darrick J. Wong, Matthew Wilcox, Jan Kara, linux-fsdevel
Testing fstests on aarch64, x86_64, s390x all passed. There's a
shortage of ppc64le systems, so I can't provide the ppc64le test
result for now, but I hope I can report the result next week.
Thanks,
Boyang
On Sat, Jul 17, 2021 at 4:04 AM Roman Gushchin <guro@fb.com> wrote:
>
> On Fri, Jul 16, 2021 at 09:23:40AM -0700, Darrick J. Wong wrote:
> > On Thu, Jul 15, 2021 at 03:28:12PM -0700, Darrick J. Wong wrote:
> > > On Thu, Jul 15, 2021 at 01:08:15PM -0700, Roman Gushchin wrote:
> > > > On Thu, Jul 15, 2021 at 10:10:50AM -0700, Darrick J. Wong wrote:
> > > > > On Thu, Jul 15, 2021 at 11:51:50AM +0800, Boyang Xue wrote:
> > > > > > On Thu, Jul 15, 2021 at 10:36 AM Matthew Wilcox <willy@infradead.org> wrote:
> > > > > > >
> > > > > > > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > > > > > > > It's unclear to me that where to find the required address in the
> > > > > > > > addr2line command line, i.e.
> > > > > > > >
> > > > > > > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > > > > <what address here?>
> > > > > > >
> > > > > > > ./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394
> > > > > > >
> > > > > >
> > > > > > Thanks! The result is the same as the
> > > > > >
> > > > > > addr2line -i -e
> > > > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > > FFFF8000102D6DD0
> > > > > >
> > > > > > But this script is very handy.
> > > > > >
> > > > > > # /usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/faddr2line
> > > > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > > cleanup_offlin
> > > > > > e_cgwbs_workfn+0x320/0x394
> > > > > > cleanup_offline_cgwbs_workfn+0x320/0x394:
> > > > > > arch_atomic64_fetch_add_unless at
> > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265
> > > > > > (inlined by) arch_atomic64_add_unless at
> > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290
> > > > > > (inlined by) atomic64_add_unless at
> > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149
> > > > > > (inlined by) atomic_long_add_unless at
> > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491
> > > > > > (inlined by) percpu_ref_tryget_many at
> > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247
> > > > > > (inlined by) percpu_ref_tryget at
> > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266
> > > > > > (inlined by) wb_tryget at
> > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227
> > > > > > (inlined by) wb_tryget at
> > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224
> > > > > > (inlined by) cleanup_offline_cgwbs_workfn at
> > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679
> > > > > >
> > > > > > # vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c
> > > > > > ```
> > > > > > static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
> > > > > > {
> > > > > > struct bdi_writeback *wb;
> > > > > > LIST_HEAD(processed);
> > > > > >
> > > > > > spin_lock_irq(&cgwb_lock);
> > > > > >
> > > > > > while (!list_empty(&offline_cgwbs)) {
> > > > > > wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
> > > > > > offline_node);
> > > > > > list_move(&wb->offline_node, &processed);
> > > > > >
> > > > > > /*
> > > > > > * If wb is dirty, cleaning up the writeback by switching
> > > > > > * attached inodes will result in an effective removal of any
> > > > > > * bandwidth restrictions, which isn't the goal. Instead,
> > > > > > * it can be postponed until the next time, when all io
> > > > > > * will be likely completed. If in the meantime some inodes
> > > > > > * will get re-dirtied, they should be eventually switched to
> > > > > > * a new cgwb.
> > > > > > */
> > > > > > if (wb_has_dirty_io(wb))
> > > > > > continue;
> > > > > >
> > > > > > if (!wb_tryget(wb)) <=== line#679
> > > > > > continue;
> > > > > >
> > > > > > spin_unlock_irq(&cgwb_lock);
> > > > > > while (cleanup_offline_cgwb(wb))
> > > > > > cond_resched();
> > > > > > spin_lock_irq(&cgwb_lock);
> > > > > >
> > > > > > wb_put(wb);
> > > > > > }
> > > > > >
> > > > > > if (!list_empty(&processed))
> > > > > > list_splice_tail(&processed, &offline_cgwbs);
> > > > > >
> > > > > > spin_unlock_irq(&cgwb_lock);
> > > > > > }
> > > > > > ```
> > > > > >
> > > > > > BTW, this bug can be only reproduced on a non-debug production built
> > > > > > kernel (a.k.a kernel rpm package), it's not reproducible on a debug
> > > > > > build with various debug configuration enabled (a.k.a kernel-debug rpm
> > > > > > package)
> > > > >
> > > > > FWIW I've also seen this regularly on x86_64 kernels on ext4 with all
> > > > > default mkfs settings when running generic/256.
> > > >
> > > > Oh, that's a useful information, thank you!
> > > >
> > > > Btw, would you mind to give a patch from an earlier message in the thread
> > > > a test? I'd highly appreciate it.
> > > >
> > > > Thanks!
> > >
> > > Will do.
> >
> > fstests passed here, so
> >
> > Tested-by: Darrick J. Wong <djwong@kernel.org>
>
> Great, thank you!
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-17 12:00 ` Boyang Xue
@ 2021-07-22 5:29 ` Boyang Xue
2021-07-22 5:41 ` Roman Gushchin
0 siblings, 1 reply; 21+ messages in thread
From: Boyang Xue @ 2021-07-22 5:29 UTC (permalink / raw)
To: Roman Gushchin; +Cc: Darrick J. Wong, Matthew Wilcox, Jan Kara, linux-fsdevel
Just FYI, the tests on ppc64le are done, no longer kernel panic, so my
tests on all arches are fine now.
On Sat, Jul 17, 2021 at 8:00 PM Boyang Xue <bxue@redhat.com> wrote:
>
> Testing fstests on aarch64, x86_64, s390x all passed. There's a
> shortage of ppc64le systems, so I can't provide the ppc64le test
> result for now, but I hope I can report the result next week.
>
> Thanks,
> Boyang
>
> On Sat, Jul 17, 2021 at 4:04 AM Roman Gushchin <guro@fb.com> wrote:
> >
> > On Fri, Jul 16, 2021 at 09:23:40AM -0700, Darrick J. Wong wrote:
> > > On Thu, Jul 15, 2021 at 03:28:12PM -0700, Darrick J. Wong wrote:
> > > > On Thu, Jul 15, 2021 at 01:08:15PM -0700, Roman Gushchin wrote:
> > > > > On Thu, Jul 15, 2021 at 10:10:50AM -0700, Darrick J. Wong wrote:
> > > > > > On Thu, Jul 15, 2021 at 11:51:50AM +0800, Boyang Xue wrote:
> > > > > > > On Thu, Jul 15, 2021 at 10:36 AM Matthew Wilcox <willy@infradead.org> wrote:
> > > > > > > >
> > > > > > > > On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
> > > > > > > > > It's unclear to me that where to find the required address in the
> > > > > > > > > addr2line command line, i.e.
> > > > > > > > >
> > > > > > > > > addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > > > > > <what address here?>
> > > > > > > >
> > > > > > > > ./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394
> > > > > > > >
> > > > > > >
> > > > > > > Thanks! The result is the same as the
> > > > > > >
> > > > > > > addr2line -i -e
> > > > > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > > > FFFF8000102D6DD0
> > > > > > >
> > > > > > > But this script is very handy.
> > > > > > >
> > > > > > > # /usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/faddr2line
> > > > > > > /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
> > > > > > > cleanup_offlin
> > > > > > > e_cgwbs_workfn+0x320/0x394
> > > > > > > cleanup_offline_cgwbs_workfn+0x320/0x394:
> > > > > > > arch_atomic64_fetch_add_unless at
> > > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265
> > > > > > > (inlined by) arch_atomic64_add_unless at
> > > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290
> > > > > > > (inlined by) atomic64_add_unless at
> > > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149
> > > > > > > (inlined by) atomic_long_add_unless at
> > > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491
> > > > > > > (inlined by) percpu_ref_tryget_many at
> > > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247
> > > > > > > (inlined by) percpu_ref_tryget at
> > > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266
> > > > > > > (inlined by) wb_tryget at
> > > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227
> > > > > > > (inlined by) wb_tryget at
> > > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224
> > > > > > > (inlined by) cleanup_offline_cgwbs_workfn at
> > > > > > > /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679
> > > > > > >
> > > > > > > # vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c
> > > > > > > ```
> > > > > > > static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
> > > > > > > {
> > > > > > > struct bdi_writeback *wb;
> > > > > > > LIST_HEAD(processed);
> > > > > > >
> > > > > > > spin_lock_irq(&cgwb_lock);
> > > > > > >
> > > > > > > while (!list_empty(&offline_cgwbs)) {
> > > > > > > wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
> > > > > > > offline_node);
> > > > > > > list_move(&wb->offline_node, &processed);
> > > > > > >
> > > > > > > /*
> > > > > > > * If wb is dirty, cleaning up the writeback by switching
> > > > > > > * attached inodes will result in an effective removal of any
> > > > > > > * bandwidth restrictions, which isn't the goal. Instead,
> > > > > > > * it can be postponed until the next time, when all io
> > > > > > > * will be likely completed. If in the meantime some inodes
> > > > > > > * will get re-dirtied, they should be eventually switched to
> > > > > > > * a new cgwb.
> > > > > > > */
> > > > > > > if (wb_has_dirty_io(wb))
> > > > > > > continue;
> > > > > > >
> > > > > > > if (!wb_tryget(wb)) <=== line#679
> > > > > > > continue;
> > > > > > >
> > > > > > > spin_unlock_irq(&cgwb_lock);
> > > > > > > while (cleanup_offline_cgwb(wb))
> > > > > > > cond_resched();
> > > > > > > spin_lock_irq(&cgwb_lock);
> > > > > > >
> > > > > > > wb_put(wb);
> > > > > > > }
> > > > > > >
> > > > > > > if (!list_empty(&processed))
> > > > > > > list_splice_tail(&processed, &offline_cgwbs);
> > > > > > >
> > > > > > > spin_unlock_irq(&cgwb_lock);
> > > > > > > }
> > > > > > > ```
> > > > > > >
> > > > > > > BTW, this bug can be only reproduced on a non-debug production built
> > > > > > > kernel (a.k.a kernel rpm package), it's not reproducible on a debug
> > > > > > > build with various debug configuration enabled (a.k.a kernel-debug rpm
> > > > > > > package)
> > > > > >
> > > > > > FWIW I've also seen this regularly on x86_64 kernels on ext4 with all
> > > > > > default mkfs settings when running generic/256.
> > > > >
> > > > > Oh, that's a useful information, thank you!
> > > > >
> > > > > Btw, would you mind to give a patch from an earlier message in the thread
> > > > > a test? I'd highly appreciate it.
> > > > >
> > > > > Thanks!
> > > >
> > > > Will do.
> > >
> > > fstests passed here, so
> > >
> > > Tested-by: Darrick J. Wong <djwong@kernel.org>
> >
> > Great, thank you!
> >
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash
2021-07-22 5:29 ` Boyang Xue
@ 2021-07-22 5:41 ` Roman Gushchin
0 siblings, 0 replies; 21+ messages in thread
From: Roman Gushchin @ 2021-07-22 5:41 UTC (permalink / raw)
To: Boyang Xue; +Cc: Darrick J. Wong, Matthew Wilcox, Jan Kara, linux-fsdevel
Thank you very much for testing it!
Sent from my iPhone
> On Jul 21, 2021, at 22:29, Boyang Xue <bxue@redhat.com> wrote:
>
> Just FYI, the tests on ppc64le are done, no longer kernel panic, so my
> tests on all arches are fine now.
>
>> On Sat, Jul 17, 2021 at 8:00 PM Boyang Xue <bxue@redhat.com> wrote:
>>
>> Testing fstests on aarch64, x86_64, s390x all passed. There's a
>> shortage of ppc64le systems, so I can't provide the ppc64le test
>> result for now, but I hope I can report the result next week.
>>
>> Thanks,
>> Boyang
>>
>>> On Sat, Jul 17, 2021 at 4:04 AM Roman Gushchin <guro@fb.com> wrote:
>>>
>>> On Fri, Jul 16, 2021 at 09:23:40AM -0700, Darrick J. Wong wrote:
>>>> On Thu, Jul 15, 2021 at 03:28:12PM -0700, Darrick J. Wong wrote:
>>>>> On Thu, Jul 15, 2021 at 01:08:15PM -0700, Roman Gushchin wrote:
>>>>>> On Thu, Jul 15, 2021 at 10:10:50AM -0700, Darrick J. Wong wrote:
>>>>>>> On Thu, Jul 15, 2021 at 11:51:50AM +0800, Boyang Xue wrote:
>>>>>>>> On Thu, Jul 15, 2021 at 10:36 AM Matthew Wilcox <willy@infradead.org> wrote:
>>>>>>>>>
>>>>>>>>> On Thu, Jul 15, 2021 at 12:22:28AM +0800, Boyang Xue wrote:
>>>>>>>>>> It's unclear to me that where to find the required address in the
>>>>>>>>>> addr2line command line, i.e.
>>>>>>>>>>
>>>>>>>>>> addr2line -e /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
>>>>>>>>>> <what address here?>
>>>>>>>>>
>>>>>>>>> ./scripts/faddr2line /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux cleanup_offline_cgwbs_workfn+0x320/0x394
>>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks! The result is the same as the
>>>>>>>>
>>>>>>>> addr2line -i -e
>>>>>>>> /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
>>>>>>>> FFFF8000102D6DD0
>>>>>>>>
>>>>>>>> But this script is very handy.
>>>>>>>>
>>>>>>>> # /usr/src/kernels/5.14.0-0.rc1.15.bx.el9.aarch64/scripts/faddr2line
>>>>>>>> /usr/lib/debug/lib/modules/5.14.0-0.rc1.15.bx.el9.aarch64/vmlinux
>>>>>>>> cleanup_offlin
>>>>>>>> e_cgwbs_workfn+0x320/0x394
>>>>>>>> cleanup_offline_cgwbs_workfn+0x320/0x394:
>>>>>>>> arch_atomic64_fetch_add_unless at
>>>>>>>> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2265
>>>>>>>> (inlined by) arch_atomic64_add_unless at
>>>>>>>> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/atomic-arch-fallback.h:2290
>>>>>>>> (inlined by) atomic64_add_unless at
>>>>>>>> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-instrumented.h:1149
>>>>>>>> (inlined by) atomic_long_add_unless at
>>>>>>>> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/asm-generic/atomic-long.h:491
>>>>>>>> (inlined by) percpu_ref_tryget_many at
>>>>>>>> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:247
>>>>>>>> (inlined by) percpu_ref_tryget at
>>>>>>>> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/percpu-refcount.h:266
>>>>>>>> (inlined by) wb_tryget at
>>>>>>>> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:227
>>>>>>>> (inlined by) wb_tryget at
>>>>>>>> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/./include/linux/backing-dev-defs.h:224
>>>>>>>> (inlined by) cleanup_offline_cgwbs_workfn at
>>>>>>>> /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c:679
>>>>>>>>
>>>>>>>> # vi /usr/src/debug/kernel-5.14.0-0.rc1.15.bx/linux-5.14.0-0.rc1.15.bx.el9.aarch64/mm/backing-dev.c
>>>>>>>> ```
>>>>>>>> static void cleanup_offline_cgwbs_workfn(struct work_struct *work)
>>>>>>>> {
>>>>>>>> struct bdi_writeback *wb;
>>>>>>>> LIST_HEAD(processed);
>>>>>>>>
>>>>>>>> spin_lock_irq(&cgwb_lock);
>>>>>>>>
>>>>>>>> while (!list_empty(&offline_cgwbs)) {
>>>>>>>> wb = list_first_entry(&offline_cgwbs, struct bdi_writeback,
>>>>>>>> offline_node);
>>>>>>>> list_move(&wb->offline_node, &processed);
>>>>>>>>
>>>>>>>> /*
>>>>>>>> * If wb is dirty, cleaning up the writeback by switching
>>>>>>>> * attached inodes will result in an effective removal of any
>>>>>>>> * bandwidth restrictions, which isn't the goal. Instead,
>>>>>>>> * it can be postponed until the next time, when all io
>>>>>>>> * will be likely completed. If in the meantime some inodes
>>>>>>>> * will get re-dirtied, they should be eventually switched to
>>>>>>>> * a new cgwb.
>>>>>>>> */
>>>>>>>> if (wb_has_dirty_io(wb))
>>>>>>>> continue;
>>>>>>>>
>>>>>>>> if (!wb_tryget(wb)) <=== line#679
>>>>>>>> continue;
>>>>>>>>
>>>>>>>> spin_unlock_irq(&cgwb_lock);
>>>>>>>> while (cleanup_offline_cgwb(wb))
>>>>>>>> cond_resched();
>>>>>>>> spin_lock_irq(&cgwb_lock);
>>>>>>>>
>>>>>>>> wb_put(wb);
>>>>>>>> }
>>>>>>>>
>>>>>>>> if (!list_empty(&processed))
>>>>>>>> list_splice_tail(&processed, &offline_cgwbs);
>>>>>>>>
>>>>>>>> spin_unlock_irq(&cgwb_lock);
>>>>>>>> }
>>>>>>>> ```
>>>>>>>>
>>>>>>>> BTW, this bug can be only reproduced on a non-debug production built
>>>>>>>> kernel (a.k.a kernel rpm package), it's not reproducible on a debug
>>>>>>>> build with various debug configuration enabled (a.k.a kernel-debug rpm
>>>>>>>> package)
>>>>>>>
>>>>>>> FWIW I've also seen this regularly on x86_64 kernels on ext4 with all
>>>>>>> default mkfs settings when running generic/256.
>>>>>>
>>>>>> Oh, that's a useful information, thank you!
>>>>>>
>>>>>> Btw, would you mind to give a patch from an earlier message in the thread
>>>>>> a test? I'd highly appreciate it.
>>>>>>
>>>>>> Thanks!
>>>>>
>>>>> Will do.
>>>>
>>>> fstests passed here, so
>>>>
>>>> Tested-by: Darrick J. Wong <djwong@kernel.org>
>>>
>>> Great, thank you!
>>>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2021-07-22 5:42 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-14 3:21 Patch 'writeback, cgroup: release dying cgwbs by switching attached inodes' leads to kernel crash Boyang Xue
2021-07-14 3:57 ` Boyang Xue
2021-07-14 4:11 ` Roman Gushchin
2021-07-14 8:44 ` Boyang Xue
2021-07-14 9:26 ` Jan Kara
2021-07-14 16:22 ` Boyang Xue
2021-07-14 23:46 ` Roman Gushchin
2021-07-15 1:42 ` Boyang Xue
2021-07-15 9:31 ` Jan Kara
2021-07-15 16:04 ` Roman Gushchin
2021-07-16 1:37 ` Boyang Xue
2021-07-15 2:35 ` Matthew Wilcox
2021-07-15 3:51 ` Boyang Xue
2021-07-15 17:10 ` Darrick J. Wong
2021-07-15 20:08 ` Roman Gushchin
2021-07-15 22:28 ` Darrick J. Wong
2021-07-16 16:23 ` Darrick J. Wong
2021-07-16 20:03 ` Roman Gushchin
2021-07-17 12:00 ` Boyang Xue
2021-07-22 5:29 ` Boyang Xue
2021-07-22 5:41 ` Roman Gushchin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.