From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F79EC433C1 for ; Tue, 30 Mar 2021 10:23:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 129A0619AD for ; Tue, 30 Mar 2021 10:23:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 129A0619AD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9086F6B00A0; Tue, 30 Mar 2021 06:22:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DED26B00A1; Tue, 30 Mar 2021 06:22:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 758B06B00A2; Tue, 30 Mar 2021 06:22:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 59A7A6B00A0 for ; Tue, 30 Mar 2021 06:22:59 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 1E5CC52CC for ; Tue, 30 Mar 2021 10:22:59 +0000 (UTC) X-FDA: 77976152478.22.292581F Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf30.hostedemail.com (Postfix) with ESMTP id D151AE0011C9 for ; Tue, 30 Mar 2021 10:22:52 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id f3so2182064pgv.0 for ; Tue, 30 Mar 2021 03:22:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LlVfK3tBgrU0BmUX2bWJJi+yNP+gPFz4okGwO5zSo8A=; b=npOuj/IXL32HPg+K23Sf+62ebOlixpsNhaNB6Q3mb/mp0VlX2286qJxPe43mPC206Q sKe0ASbkn8j9Y6Bu1nY89TM5lv4O7xe6RmLNinsrYRcvQoQkiDcrCvob1ltIglysLe1/ FJkYD9zLZVp7JaXlnxo/GoQDqb5523ZL6hvS6i5ihuVzIZ/dgZwqd7Vs253seeHV6wf6 SufLYUloFclKQVRLkUFN+8qZHerr5cE4Fi263QbIuTBVqNstFg6PYRMaZzV4gfsMOCjH MdPzLEykQ+MzbaHjAogbvIr1kHzzzthVUPvsf7zPTAY2Zxf2P0SjkwGjATdxL0Hp49kX 3glw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LlVfK3tBgrU0BmUX2bWJJi+yNP+gPFz4okGwO5zSo8A=; b=MITGOQ9TX8pkgKm7u5OJQdkjFShp3QknYbc4TYQM5Vaeq+5hrmNceuCekDdss2bXYx xAcz0CLQjK6fYnUjLdoo4eIO9kHbGQqfKR4+8mEZD4CVlOyRC+Jys8Gv2//5o8VhR7XB fKGSgx4tYkBnqd+KBktDaFtbNzVY5rkjueiOgIPt0djs4igg+JAcekJVZDXEicdoGFJ1 n3+5UpLeb2Xz1nJFZuGksLLvteYtng0htnIXiv6sJZtDlvp8Xl8WS+qN8hP1zTj+G1SE Nv77mWXJqWp9hKSqyL5Fl1VriChph7HWOsBQVHg/Ln0LO/UwIVxvgyvOIsdA7jaJrcj8 Ed7g== X-Gm-Message-State: AOAM530ab27caxoK5nw51UWEJ/VTv0IsbtXZZj28JKYsSnu772GutTRI DQwmaF8alnNOit86PqAQVp2xSw== X-Google-Smtp-Source: ABdhPJyOUnZ7tulMq0ZkReZZufRvoJjBeThdaJ2q71EIRLjtbmbmrs8OtM6B/E1tX6dkgSDD809a5w== X-Received: by 2002:a65:5308:: with SMTP id m8mr26351120pgq.266.1617099777573; Tue, 30 Mar 2021 03:22:57 -0700 (PDT) Received: from localhost.localdomain ([2408:8445:ad30:68d8:c87f:ca1b:dc00:4730]) by smtp.gmail.com with ESMTPSA id k10sm202259pfk.205.2021.03.30.03.22.48 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 30 Mar 2021 03:22:57 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [RFC PATCH 12/15] mm: memcontrol: introduce memcg_reparent_ops Date: Tue, 30 Mar 2021 18:15:28 +0800 Message-Id: <20210330101531.82752-13-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210330101531.82752-1-songmuchun@bytedance.com> References: <20210330101531.82752-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D151AE0011C9 X-Stat-Signature: rexnxe3qcwkiut9mctto6a4yn9pr6ip1 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=mail-pg1-f177.google.com; client-ip=209.85.215.177 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617099772-855463 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the previous patch, we know how to make the lruvec lock safe when the LRU pages reparented. We should do something like following. memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent me= mcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) do reparent // Move all the pages from the lruvec list to the parent lruvec l= ist. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); Apart from the page lruvec lock, the deferred split queue lock (THP only) also needs to do something similar. So we extracted the necessary 3 steps in the memcg_reparent_objcgs(). memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) reparent memcg_reparent_ops->reparent(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Now there are two different locks (e.g. lruvec lock and deferred split queue lock) need to use this infrastructure. In the next patch, we will use those APIs to make those locks safe when the LRU pages reparented. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 11 +++++++++++ mm/memcontrol.c | 49 ++++++++++++++++++++++++++++++++++++++++= ++++-- 2 files changed, 58 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 8944115ebf8e..c79770ce3c81 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -341,6 +341,17 @@ struct mem_cgroup { /* WARNING: nodeinfo must be the last member here */ }; =20 +struct memcg_reparent_ops { + struct list_head list; + + /* Irq is disabled before calling those functions. */ + void (*lock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*unlock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*reparent)(struct mem_cgroup *memcg, struct mem_cgroup *parent); +}; + +void __init register_memcg_repatent(struct memcg_reparent_ops *ops); + /* * size of first charge trial. "32" comes from vmscan.c's magic value. * TODO: maybe necessary to use big numbers in big irons. diff --git a/mm/memcontrol.c b/mm/memcontrol.c index cb650d089d9f..d5701117794a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -338,6 +338,41 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } =20 +static LIST_HEAD(reparent_ops_head); + +static void memcg_reparent_lock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + struct memcg_reparent_ops *ops; + + list_for_each_entry(ops, &reparent_ops_head, list) + ops->lock(memcg, parent); +} + +static void memcg_reparent_unlock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + struct memcg_reparent_ops *ops; + + list_for_each_entry(ops, &reparent_ops_head, list) + ops->unlock(memcg, parent); +} + +static void memcg_do_reparent(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + struct memcg_reparent_ops *ops; + + list_for_each_entry(ops, &reparent_ops_head, list) + ops->reparent(memcg, parent); +} + +void __init register_memcg_repatent(struct memcg_reparent_ops *ops) +{ + BUG_ON(!ops->lock || !ops->unlock || !ops->reparent); + list_add(&ops->list, &reparent_ops_head); +} + static void memcg_reparent_objcgs(struct mem_cgroup *memcg) { struct obj_cgroup *objcg, *iter; @@ -347,9 +382,13 @@ static void memcg_reparent_objcgs(struct mem_cgroup = *memcg) if (!parent) parent =3D root_mem_cgroup; =20 + local_irq_disable(); + + memcg_reparent_lock(memcg, parent); + objcg =3D rcu_replace_pointer(memcg->objcg, NULL, true); =20 - spin_lock_irq(&css_set_lock); + spin_lock(&css_set_lock); =20 /* 1) Ready to reparent active objcg. */ list_add(&objcg->list, &memcg->objcg_list); @@ -361,7 +400,13 @@ static void memcg_reparent_objcgs(struct mem_cgroup = *memcg) /* 3) Move already reparented objcgs to the parent's list */ list_splice(&memcg->objcg_list, &parent->objcg_list); =20 - spin_unlock_irq(&css_set_lock); + spin_unlock(&css_set_lock); + + memcg_do_reparent(memcg, parent); + + memcg_reparent_unlock(memcg, parent); + + local_irq_enable(); =20 percpu_ref_kill(&objcg->refcnt); } --=20 2.11.0