Re: [PATCH v5 2/2] writeback, cgroup: release dying cgwbs by switching attached inodes

[Ming Lei <ming.lei@redhat.com>]

On Wed, May 26, 2021 at 03:25:57PM -0700, Roman Gushchin wrote:
> Asynchronously try to release dying cgwbs by switching clean attached
> inodes to the bdi's wb. It helps to get rid of per-cgroup writeback
> structures themselves and of pinned memory and block cgroups, which
> are way larger structures (mostly due to large per-cpu statistics
> data). It helps to prevent memory waste and different scalability
> problems caused by large piles of dying cgroups.
> 
> A cgwb cleanup operation can fail due to different reasons (e.g. the
> cgwb has in-glight/pending io, an attached inode is locked or isn't
> clean, etc). In this case the next scheduled cleanup will make a new
> attempt. An attempt is made each time a new cgwb is offlined (in other
> words a memcg and/or a blkcg is deleted by a user). In the future an
> additional attempt scheduled by a timer can be implemented.
> 
> Signed-off-by: Roman Gushchin <guro@fb.com>
> ---
>  fs/fs-writeback.c                | 35 ++++++++++++++++++
>  include/linux/backing-dev-defs.h |  1 +
>  include/linux/writeback.h        |  1 +
>  mm/backing-dev.c                 | 61 ++++++++++++++++++++++++++++++--
>  4 files changed, 96 insertions(+), 2 deletions(-)
> 

Hello Roman,

The following kernel panic is triggered by this patch:

[root@ktest-01 xfstests-dev]# ./check generic/563
[   47.186811] SGI XFS with ACLs, security attributes, realtime, verbose warnings, quota, no debug enabled
[   47.190152] XFS (sdb): Mounting V5 Filesystem
[   47.201551] XFS (sdb): Ending clean mount
[   47.205501] xfs filesystem being mounted at /mnt/test supports timestamps until 2038 (0x7fffffff)
FSTYP         -- xfs (non-debug)
PLATFORM      -- Linux/x86_64 ktest-01 5.13.0-rc3+ #294 SMP Fri May 28 10:51:02 CST 2021
MKFS_OPTIONS  -- -f -bsize=4096 /dev/sda
MOUNT_OPTIONS -- /dev/sda /mnt/scratch

[   47.431775] XFS (sda): Mounting V5 Filesystem
[   47.441731] XFS (sda): Ending clean mount
[   47.445080] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[   47.449189] XFS (sda): Unmounting Filesystem
[   47.473863] XFS (sdb): Unmounting Filesystem
[   47.614561] XFS (sdb): Mounting V5 Filesystem
[   47.628670] XFS (sdb): Ending clean mount
[   47.631904] xfs filesystem being mounted at /mnt/test supports timestamps until 2038 (0x7fffffff)
generic/563 1s ... [   47.661393] run fstests generic/563 at 2021-05-28 02:54:59
[   47.947414] loop0: detected capacity change from 0 to 16777216
[   48.034564] XFS (loop0): Mounting V5 Filesystem
[   48.069959] XFS (loop0): Ending clean mount
[   48.070726] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[   48.132314] XFS (loop0): Unmounting Filesystem
[   48.204548] XFS (loop0): Mounting V5 Filesystem
[   48.215500] XFS (loop0): Ending clean mount
[   48.219223] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[   48.534420] XFS (loop0): Unmounting Filesystem
[   48.535142] ------------[ cut here ]------------
[   48.535921] WARNING: CPU: 3 PID: 114 at mm/backing-dev.c:402 cgwb_release_workfn+0xa4/0xd8
[   48.537461] Modules linked in: xfs libcrc32c iTCO_wdt i2c_i801 iTCO_vendor_support nvme i2c_smbus lpc_ich usb_storage i2c_s
[   48.540613] CPU: 3 PID: 114 Comm: kworker/3:1 Not tainted 5.13.0-rc3+ #294
[   48.541927] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-1.fc33 04/01/2014
[   48.543439] Workqueue: cgwb_release cgwb_release_workfn
[   48.544365] RIP: 0010:cgwb_release_workfn+0xa4/0xd8
[   48.545185] Code: 00 00 00 48 85 db 75 d5 48 8d 7d 80 e8 98 71 20 00 48 8d bd 70 ff ff ff e8 36 7b 1d 00 48 8b 55 f0 48 8d4
[   48.548935] RSP: 0018:ffffc90001f47e88 EFLAGS: 00010202
[   48.549844] RAX: ffff88810321d280 RBX: ffffffff82f69ac0 RCX: 0000000080400011
[   48.552645] RDX: ffffffff82669f00 RSI: 0000000000210d00 RDI: ffff888100042500
[   48.553935] RBP: ffff88810321d290 R08: 0000000000000001 R09: ffffffff811c8754
[   48.555054] R10: 000000000000005e R11: 0000000000000046 R12: ffff88810321d000
[   48.556183] R13: ffff88815c4f2300 R14: 0000000000000000 R15: 0000000000000000
[   48.557116] FS:  0000000000000000(0000) GS:ffff88815c4c0000(0000) knlGS:0000000000000000
[   48.558131] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   48.558818] CR2: 00007f3aba17f9c0 CR3: 0000000108e86004 CR4: 0000000000370ee0
[   48.559950] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   48.561057] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   48.562075] Call Trace:
[   48.562421]  elfcorehdr_read+0xf/0xf
[   48.562920]  ? worker_thread+0x117/0x1b9
[   48.563443]  ? rescuer_thread+0x291/0x291
[   48.564001]  ? kthread+0xec/0xf4
[   48.564411]  ? kthread_create_worker_on_cpu+0x65/0x65
[   48.565086]  ? ret_from_fork+0x1f/0x30
[   48.565594] ---[ end trace bdeef00aa75cca5c ]---
[   48.601694] XFS (loop0): Mounting V5 Filesystem
[   48.605863] XFS (loop0): Ending clean mount
[   48.607129] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[   48.830734] general protection fault, probably for non-canonical address 0xffff11033f71f000: 0000 [#1] SMP NOPTI
[   48.832720] CPU: 10 PID: 234 Comm: kworker/10:1 Tainted: G        W         5.13.0-rc3+ #294
[   48.833932] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-1.fc33 04/01/2014
[   48.835146] Workqueue: events cleanup_offline_cgwbs_workfn
[   48.835952] RIP: 0010:percpu_ref_tryget_many.constprop.0+0x12/0x43
[   48.836849] Code: 48 8b 47 08 48 8b 40 08 ff d0 0f 1f 00 eb 04 65 48 ff 08 e9 25 fd ff ff 41 54 48 8b 07 a8 03 74 09 48 8b4
[   48.839494] RSP: 0018:ffffc90001383e58 EFLAGS: 00010046
[   48.840246] RAX: ffff88810321f000 RBX: ffff88810321d280 RCX: ffff8881016f9770
[   48.841165] RDX: ffffffff82669f00 RSI: 0000000000000280 RDI: ffff88810321d200
[   48.842050] RBP: ffffc90001383e68 R08: ffff88810006c8b0 R09: 000073746e657665
[   48.843224] R10: 8080808080808080 R11: fefefefefefefeff R12: ffff88810321d200
[   48.844133] R13: ffff88810321d000 R14: 0000000000000000 R15: 0000000000000000
[   48.845022] FS:  0000000000000000(0000) GS:ffff88823c500000(0000) knlGS:0000000000000000
[   48.845903] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   48.846495] CR2: 00007fb630166198 CR3: 0000000179a1e006 CR4: 0000000000370ee0
[   48.847414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   48.848405] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   48.849126] Call Trace:
[   48.849388]  cleanup_offline_cgwbs_workfn+0x8a/0x14c
[   48.849906]  process_one_work+0x15c/0x234
[   48.850346]  worker_thread+0x117/0x1b9
[   48.850706]  ? rescuer_thread+0x291/0x291
[   48.851065]  kthread+0xec/0xf4
[   48.851346]  ? kthread_create_worker_on_cpu+0x65/0x65
[   48.851815]  ret_from_fork+0x1f/0x30
[   48.852151] Modules linked in: xfs libcrc32c iTCO_wdt i2c_i801 iTCO_vendor_support nvme i2c_smbus lpc_ich usb_storage i2c_s
[   48.854166] Dumping ftrace buffer:
[   48.854546]    (ftrace buffer empty)
[   48.854909] ---[ end trace bdeef00aa75cca5d ]---



Thanks, 
Ming