From: Roman Gushchin <guro@fb.com> To: Jan Kara <jack@suse.cz>, Tejun Heo <tj@kernel.org> Cc: <linux-fsdevel@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>, Alexander Viro <viro@zeniv.linux.org.uk>, Dennis Zhou <dennis@kernel.org>, Dave Chinner <dchinner@redhat.com>, <cgroups@vger.kernel.org>, Roman Gushchin <guro@fb.com> Subject: [PATCH v8 0/8] cgroup, blkcg: prevent dirty inodes to pin dying memory cgroups Date: Mon, 7 Jun 2021 18:31:15 -0700 [thread overview] Message-ID: <20210608013123.1088882-1-guro@fb.com> (raw) When an inode is getting dirty for the first time it's associated with a wb structure (see __inode_attach_wb()). It can later be switched to another wb (if e.g. some other cgroup is writing a lot of data to the same inode), but otherwise stays attached to the original wb until being reclaimed. The problem is that the wb structure holds a reference to the original memory and blkcg cgroups. So if an inode has been dirty once and later is actively used in read-only mode, it has a good chance to pin down the original memory and blkcg cgroups forewer. This is often the case with services bringing data for other services, e.g. updating some rpm packages. In the real life it becomes a problem due to a large size of the memcg structure, which can easily be 1000x larger than an inode. Also a really large number of dying cgroups can raise different scalability issues, e.g. making the memory reclaim costly and less effective. To solve the problem inodes should be eventually detached from the corresponding writeback structure. It's inefficient to do it after every writeback completion. Instead it can be done whenever the original memory cgroup is offlined and writeback structure is getting killed. Scanning over a (potentially long) list of inodes and detach them from the writeback structure can take quite some time. To avoid scanning all inodes, attached inodes are kept on a new list (b_attached). To make it less noticeable to a user, the scanning and switching is performed from a work context. Big thanks to Jan Kara, Dennis Zhou, Hillf Danton and Tejun Heo for their ideas and contribution to this patchset. v8: - switch inodes to a nearest living ancestor wb instead of root wb - added two inodes switching fixes suggested by Jan Kara v7: - shared locking for multiple inode switching - introduced inode_prepare_wbs_switch() helper - extended the pre-switch inode check for I_WILL_FREE - added comments here and there v6: - extended and reused wbs switching functionality to switch inodes on cgwb cleanup - fixed offline_list handling - switched to the unbound_wq - other minor fixes v5: - switch inodes to bdi->wb instead of zeroing inode->i_wb - split the single patch into two - only cgwbs maintain lists of attached inodes - added cond_resched() - fixed !CONFIG_CGROUP_WRITEBACK handling - extended list of prohibited inodes flag - other small fixes Roman Gushchin (8): writeback, cgroup: do not switch inodes with I_WILL_FREE flag writeback, cgroup: add smp_mb() to cgroup_writeback_umount() writeback, cgroup: increment isw_nr_in_flight before grabbing an inode writeback, cgroup: switch to rcu_work API in inode_switch_wbs() writeback, cgroup: keep list of inodes attached to bdi_writeback writeback, cgroup: split out the functional part of inode_switch_wbs_work_fn() writeback, cgroup: support switching multiple inodes at once writeback, cgroup: release dying cgwbs by switching attached inodes fs/fs-writeback.c | 323 +++++++++++++++++++++---------- include/linux/backing-dev-defs.h | 20 +- include/linux/writeback.h | 1 + mm/backing-dev.c | 69 ++++++- 4 files changed, 312 insertions(+), 101 deletions(-) -- 2.31.1
WARNING: multiple messages have this Message-ID (diff)
From: Roman Gushchin <guro-b10kYP2dOMg@public.gmane.org> To: Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org>, Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> Cc: linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, Alexander Viro <viro-RmSDqhL/yNMiFSDQTTA3OLVCufUGDwFn@public.gmane.org>, Dennis Zhou <dennis-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>, Dave Chinner <dchinner-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Roman Gushchin <guro-b10kYP2dOMg@public.gmane.org> Subject: [PATCH v8 0/8] cgroup, blkcg: prevent dirty inodes to pin dying memory cgroups Date: Mon, 7 Jun 2021 18:31:15 -0700 [thread overview] Message-ID: <20210608013123.1088882-1-guro@fb.com> (raw) When an inode is getting dirty for the first time it's associated with a wb structure (see __inode_attach_wb()). It can later be switched to another wb (if e.g. some other cgroup is writing a lot of data to the same inode), but otherwise stays attached to the original wb until being reclaimed. The problem is that the wb structure holds a reference to the original memory and blkcg cgroups. So if an inode has been dirty once and later is actively used in read-only mode, it has a good chance to pin down the original memory and blkcg cgroups forewer. This is often the case with services bringing data for other services, e.g. updating some rpm packages. In the real life it becomes a problem due to a large size of the memcg structure, which can easily be 1000x larger than an inode. Also a really large number of dying cgroups can raise different scalability issues, e.g. making the memory reclaim costly and less effective. To solve the problem inodes should be eventually detached from the corresponding writeback structure. It's inefficient to do it after every writeback completion. Instead it can be done whenever the original memory cgroup is offlined and writeback structure is getting killed. Scanning over a (potentially long) list of inodes and detach them from the writeback structure can take quite some time. To avoid scanning all inodes, attached inodes are kept on a new list (b_attached). To make it less noticeable to a user, the scanning and switching is performed from a work context. Big thanks to Jan Kara, Dennis Zhou, Hillf Danton and Tejun Heo for their ideas and contribution to this patchset. v8: - switch inodes to a nearest living ancestor wb instead of root wb - added two inodes switching fixes suggested by Jan Kara v7: - shared locking for multiple inode switching - introduced inode_prepare_wbs_switch() helper - extended the pre-switch inode check for I_WILL_FREE - added comments here and there v6: - extended and reused wbs switching functionality to switch inodes on cgwb cleanup - fixed offline_list handling - switched to the unbound_wq - other minor fixes v5: - switch inodes to bdi->wb instead of zeroing inode->i_wb - split the single patch into two - only cgwbs maintain lists of attached inodes - added cond_resched() - fixed !CONFIG_CGROUP_WRITEBACK handling - extended list of prohibited inodes flag - other small fixes Roman Gushchin (8): writeback, cgroup: do not switch inodes with I_WILL_FREE flag writeback, cgroup: add smp_mb() to cgroup_writeback_umount() writeback, cgroup: increment isw_nr_in_flight before grabbing an inode writeback, cgroup: switch to rcu_work API in inode_switch_wbs() writeback, cgroup: keep list of inodes attached to bdi_writeback writeback, cgroup: split out the functional part of inode_switch_wbs_work_fn() writeback, cgroup: support switching multiple inodes at once writeback, cgroup: release dying cgwbs by switching attached inodes fs/fs-writeback.c | 323 +++++++++++++++++++++---------- include/linux/backing-dev-defs.h | 20 +- include/linux/writeback.h | 1 + mm/backing-dev.c | 69 ++++++- 4 files changed, 312 insertions(+), 101 deletions(-) -- 2.31.1
next reply other threads:[~2021-06-08 1:31 UTC|newest] Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-06-08 1:31 Roman Gushchin [this message] 2021-06-08 1:31 ` [PATCH v8 0/8] cgroup, blkcg: prevent dirty inodes to pin dying memory cgroups Roman Gushchin 2021-06-08 1:31 ` [PATCH v8 1/8] writeback, cgroup: do not switch inodes with I_WILL_FREE flag Roman Gushchin 2021-06-08 1:31 ` Roman Gushchin 2021-06-08 1:31 ` [PATCH v8 2/8] writeback, cgroup: add smp_mb() to cgroup_writeback_umount() Roman Gushchin 2021-06-08 1:31 ` Roman Gushchin 2021-06-08 8:43 ` Jan Kara 2021-06-08 1:31 ` [PATCH v8 3/8] writeback, cgroup: increment isw_nr_in_flight before grabbing an inode Roman Gushchin 2021-06-08 1:31 ` Roman Gushchin 2021-06-08 8:45 ` Jan Kara 2021-06-08 8:45 ` Jan Kara 2021-06-08 1:31 ` [PATCH v8 4/8] writeback, cgroup: switch to rcu_work API in inode_switch_wbs() Roman Gushchin 2021-06-08 1:31 ` Roman Gushchin 2021-06-08 1:31 ` [PATCH v8 5/8] writeback, cgroup: keep list of inodes attached to bdi_writeback Roman Gushchin 2021-06-08 1:31 ` Roman Gushchin 2021-06-08 1:31 ` [PATCH v8 6/8] writeback, cgroup: split out the functional part of inode_switch_wbs_work_fn() Roman Gushchin 2021-06-08 1:31 ` Roman Gushchin 2021-06-08 1:31 ` [PATCH v8 7/8] writeback, cgroup: support switching multiple inodes at once Roman Gushchin 2021-06-08 1:31 ` Roman Gushchin 2021-06-08 1:31 ` [PATCH v8 8/8] writeback, cgroup: release dying cgwbs by switching attached inodes Roman Gushchin 2021-06-08 1:31 ` Roman Gushchin 2021-06-08 8:54 ` Jan Kara 2021-06-08 16:08 ` Dennis Zhou 2021-06-08 16:08 ` Dennis Zhou 2021-06-08 22:37 ` Roman Gushchin 2021-06-08 22:37 ` Roman Gushchin
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210608013123.1088882-1-guro@fb.com \ --to=guro@fb.com \ --cc=cgroups@vger.kernel.org \ --cc=dchinner@redhat.com \ --cc=dennis@kernel.org \ --cc=jack@suse.cz \ --cc=linux-fsdevel@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=tj@kernel.org \ --cc=viro@zeniv.linux.org.uk \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.