From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC22AC47404 for ; Fri, 4 Oct 2019 22:11:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6588621D81 for ; Fri, 4 Oct 2019 22:11:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="W/qOGGD7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6588621D81 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CB6976B0003; Fri, 4 Oct 2019 18:11:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C41FA6B0005; Fri, 4 Oct 2019 18:11:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B072A6B0006; Fri, 4 Oct 2019 18:11:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0205.hostedemail.com [216.40.44.205]) by kanga.kvack.org (Postfix) with ESMTP id 89F4C6B0003 for ; Fri, 4 Oct 2019 18:11:14 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id 29A7C6135 for ; Fri, 4 Oct 2019 22:11:14 +0000 (UTC) X-FDA: 76007498868.24.camp24_abb657405127 X-HE-Tag: camp24_abb657405127 X-Filterd-Recvd-Size: 6801 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Fri, 4 Oct 2019 22:11:13 +0000 (UTC) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.16.0.42/8.16.0.42) with SMTP id x94M92mD024853 for ; Fri, 4 Oct 2019 15:11:10 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : mime-version : content-type; s=facebook; bh=xKwMmV5/lRZ5KOHFX9m6ZN94W9gYSkTsv8TGeCtH98A=; b=W/qOGGD7uKt5aA6bFogb3ZYuR04+jvI5XkukXG/hnsmGzKcpgKdOLKxZ1L+U2b1ELNDu RqoXJBbfbdBhsTR0ieHdUIdn/K0SHAvJcWBGqpNK2mkeatWf9V97FzE6BuhcW6i0kCnI CzdslRNn2Ph4HRUa57nDbADChKbbkg2cRVY= Received: from maileast.thefacebook.com ([163.114.130.16]) by m0089730.ppops.net with ESMTP id 2ve1ud3d55-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Fri, 04 Oct 2019 15:11:10 -0700 Received: from 2401:db00:30:6012:face:0:17:0 (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 4 Oct 2019 15:11:08 -0700 Received: by devvm2643.prn2.facebook.com (Postfix, from userid 111017) id 5E25F1841D558; Fri, 4 Oct 2019 15:11:07 -0700 (PDT) Smtp-Origin-Hostprefix: devvm From: Roman Gushchin Smtp-Origin-Hostname: devvm2643.prn2.facebook.com To: , CC: , , , Jan Kara , Roman Gushchin Smtp-Origin-Cluster: prn2c23 Subject: [PATCH] cgroup, blkcg: prevent dirty inodes to pin dying memory cgroups Date: Fri, 4 Oct 2019 15:11:04 -0700 Message-ID: <20191004221104.646711-1-guro@fb.com> X-Mailer: git-send-email 2.17.1 X-FB-Internal: Safe MIME-Version: 1.0 Content-Type: text/plain X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.95,1.0.8 definitions=2019-10-04_13:2019-10-03,2019-10-04 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 malwarescore=0 mlxlogscore=758 clxscore=1015 impostorscore=0 suspectscore=2 priorityscore=1501 spamscore=0 bulkscore=0 adultscore=0 mlxscore=0 lowpriorityscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1908290000 definitions=main-1910040185 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a RFC patch, which is not intended to be merged as is, but hopefully will start a discussion which can result in a good solution for the described problem. -- We've noticed that the number of dying cgroups on our production hosts tends to grow with the uptime. This time it's caused by the writeback code. An inode which is getting dirty for the first time is associated with the wb structure (look at __inode_attach_wb()). It can later be switched to another wb under some conditions (e.g. some other cgroup is writing a lot of data to the same inode), but generally stays associated up to the end of life of the inode structure. The problem is that the wb structure holds a reference to the original memory cgroup. So if the inode was dirty once, it has a good chance to pin down the original memory cgroup. An example from the real life: some service runs periodically and updates rpm packages. Each time in a new memory cgroup. Installed .so files are heavily used by other cgroups, so corresponding inodes tend to stay alive for a long. So do pinned memory cgroups. In production I've seen many hosts with 1-2 thousands of dying cgroups. This is not the first problem with the dying memory cgroups. As always, the problem is with their relative size: memory cgroups are large objects, easily 100x-1000x larger that inodes. So keeping a couple of thousands of dying cgroups in memory without a good reason (what we easily do with inodes) is quite costly (and is measured in tens and hundreds of Mb). One possible approach to this problem is to switch inodes associated with dying wbs to the root wb. Switching is a best effort operation which can fail silently, so unfortunately we can't run once over a list of associated inodes (even if we'd have such a list). So we really have to scan all inodes. In the proposed patch I schedule a work on each memory cgroup deletion, which is probably too often. Alternatively, we can do it periodically under some conditions (e.g. the number of dying memory cgroups is larger than X). So it's basically a gc run. I wonder if there are any better ideas? Signed-off-by: Roman Gushchin --- fs/fs-writeback.c | 29 +++++++++++++++++++++++++++++ mm/memcontrol.c | 5 +++++ 2 files changed, 34 insertions(+) diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 542b02d170f8..4bbc9a200b2c 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -545,6 +545,35 @@ static void inode_switch_wbs(struct inode *inode, int new_wb_id) up_read(&bdi->wb_switch_rwsem); } +static void reparent_dirty_inodes_one_sb(struct super_block *sb, void *arg) +{ + struct inode *inode, *next; + + spin_lock(&sb->s_inode_list_lock); + list_for_each_entry_safe(inode, next, &sb->s_inodes, i_sb_list) { + spin_lock(&inode->i_lock); + if (inode->i_state & (I_NEW | I_FREEING | I_WILL_FREE)) { + spin_unlock(&inode->i_lock); + continue; + } + + if (inode->i_wb && wb_dying(inode->i_wb)) { + spin_unlock(&inode->i_lock); + inode_switch_wbs(inode, root_mem_cgroup->css.id); + continue; + } + + spin_unlock(&inode->i_lock); + } + spin_unlock(&sb->s_inode_list_lock); + +} + +void reparent_dirty_inodes(struct work_struct *work) +{ + iterate_supers(reparent_dirty_inodes_one_sb, NULL); +} + /** * wbc_attach_and_unlock_inode - associate wbc with target inode and unlock it * @wbc: writeback_control of interest diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 9ec5e12486a7..ea8bc8d1403b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4911,6 +4911,9 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css) return 0; } +extern void reparent_dirty_inodes(struct work_struct *w); +static DECLARE_WORK(dirty_work, reparent_dirty_inodes); + static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); @@ -4934,6 +4937,8 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css) memcg_offline_kmem(memcg); wb_memcg_offline(memcg); + schedule_work(&dirty_work); + drain_all_stock(memcg); mem_cgroup_id_put(memcg); -- 2.21.0