From: Michal Hocko <mhocko@suse.com>
To: Yosry Ahmed <yosryahmed@google.com>
Cc: "Tejun Heo" <tj@kernel.org>, "Josef Bacik" <josef@toxicpanda.com>,
"Jens Axboe" <axboe@kernel.dk>,
"Zefan Li" <lizefan.x@bytedance.com>,
"Johannes Weiner" <hannes@cmpxchg.org>,
"Roman Gushchin" <roman.gushchin@linux.dev>,
"Shakeel Butt" <shakeelb@google.com>,
"Muchun Song" <muchun.song@linux.dev>,
"Andrew Morton" <akpm@linux-foundation.org>,
"Michal Koutný" <mkoutny@suse.com>,
"Vasily Averin" <vasily.averin@linux.dev>,
cgroups@vger.kernel.org, linux-block@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
bpf@vger.kernel.org
Subject: Re: [PATCH v2 6/9] memcg: sleep during flushing stats in safe contexts
Date: Thu, 30 Mar 2023 09:35:56 +0200 [thread overview]
Message-ID: <ZCU73Ghje96Q0X7A@dhcp22.suse.cz> (raw)
In-Reply-To: <20230328221644.803272-7-yosryahmed@google.com>
On Tue 28-03-23 22:16:41, Yosry Ahmed wrote:
> Currently, all contexts that flush memcg stats do so with sleeping not
> allowed. Some of these contexts are perfectly safe to sleep in, such as
> reading cgroup files from userspace or the background periodic flusher.
>
> Refactor the code to make mem_cgroup_flush_stats() non-atomic (aka
> sleepable), and provide a separate atomic version. The atomic version is
> used in reclaim, refault, writeback, and in mem_cgroup_usage(). All
> other code paths are left to use the non-atomic version. This includes
> callbacks for userspace reads and the periodic flusher.
>
> Since refault is the only caller of mem_cgroup_flush_stats_ratelimited(),
> change it to mem_cgroup_flush_stats_atomic_ratelimited(). Reclaim and
> refault code paths are modified to do non-atomic flushing in separate
> later patches -- so it will eventually be changed back to
> mem_cgroup_flush_stats_ratelimited().
>
> Signed-off-by: Yosry Ahmed <yosryahmed@google.com>
> Acked-by: Shakeel Butt <shakeelb@google.com>
Acked-by: Michal Hocko <mhocko@suse.com>
> ---
> include/linux/memcontrol.h | 9 ++++++--
> mm/memcontrol.c | 45 ++++++++++++++++++++++++++++++--------
> mm/vmscan.c | 2 +-
> mm/workingset.c | 2 +-
> 4 files changed, 45 insertions(+), 13 deletions(-)
>
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index ac3f3b3a45e2..b424ba3ebd09 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -1037,7 +1037,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec,
> }
>
> void mem_cgroup_flush_stats(void);
> -void mem_cgroup_flush_stats_ratelimited(void);
> +void mem_cgroup_flush_stats_atomic(void);
> +void mem_cgroup_flush_stats_atomic_ratelimited(void);
>
> void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx,
> int val);
> @@ -1535,7 +1536,11 @@ static inline void mem_cgroup_flush_stats(void)
> {
> }
>
> -static inline void mem_cgroup_flush_stats_ratelimited(void)
> +static inline void mem_cgroup_flush_stats_atomic(void)
> +{
> +}
> +
> +static inline void mem_cgroup_flush_stats_atomic_ratelimited(void)
> {
> }
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 65750f8b8259..a2ce3aa10d94 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -634,7 +634,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val)
> }
> }
>
> -static void __mem_cgroup_flush_stats(void)
> +static void do_flush_stats(bool atomic)
> {
> /*
> * We always flush the entire tree, so concurrent flushers can just
> @@ -646,26 +646,46 @@ static void __mem_cgroup_flush_stats(void)
> return;
>
> WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME);
> - cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup);
> +
> + if (atomic)
> + cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup);
> + else
> + cgroup_rstat_flush(root_mem_cgroup->css.cgroup);
> +
> atomic_set(&stats_flush_threshold, 0);
> atomic_set(&stats_flush_ongoing, 0);
> }
>
> +static bool should_flush_stats(void)
> +{
> + return atomic_read(&stats_flush_threshold) > num_online_cpus();
> +}
> +
> void mem_cgroup_flush_stats(void)
> {
> - if (atomic_read(&stats_flush_threshold) > num_online_cpus())
> - __mem_cgroup_flush_stats();
> + if (should_flush_stats())
> + do_flush_stats(false);
> }
>
> -void mem_cgroup_flush_stats_ratelimited(void)
> +void mem_cgroup_flush_stats_atomic(void)
> +{
> + if (should_flush_stats())
> + do_flush_stats(true);
> +}
> +
> +void mem_cgroup_flush_stats_atomic_ratelimited(void)
> {
> if (time_after64(jiffies_64, READ_ONCE(flush_next_time)))
> - mem_cgroup_flush_stats();
> + mem_cgroup_flush_stats_atomic();
> }
>
> static void flush_memcg_stats_dwork(struct work_struct *w)
> {
> - __mem_cgroup_flush_stats();
> + /*
> + * Always flush here so that flushing in latency-sensitive paths is
> + * as cheap as possible.
> + */
> + do_flush_stats(false);
> queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME);
> }
>
> @@ -3685,9 +3705,12 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap)
> * done from irq context; use stale stats in this case.
> * Arguably, usage threshold events are not reliable on the root
> * memcg anyway since its usage is ill-defined.
> + *
> + * Additionally, other call paths through memcg_check_events()
> + * disable irqs, so make sure we are flushing stats atomically.
> */
> if (in_task())
> - mem_cgroup_flush_stats();
> + mem_cgroup_flush_stats_atomic();
> val = memcg_page_state(memcg, NR_FILE_PAGES) +
> memcg_page_state(memcg, NR_ANON_MAPPED);
> if (swap)
> @@ -4610,7 +4633,11 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages,
> struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css);
> struct mem_cgroup *parent;
>
> - mem_cgroup_flush_stats();
> + /*
> + * wb_writeback() takes a spinlock and calls
> + * wb_over_bg_thresh()->mem_cgroup_wb_stats(). Do not sleep.
> + */
> + mem_cgroup_flush_stats_atomic();
>
> *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY);
> *pwriteback = memcg_page_state(memcg, NR_WRITEBACK);
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 9c1c5e8b24b8..a9511ccb936f 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc)
> * Flush the memory cgroup stats, so that we read accurate per-memcg
> * lruvec stats for heuristics.
> */
> - mem_cgroup_flush_stats();
> + mem_cgroup_flush_stats_atomic();
>
> /*
> * Determine the scan balance between anon and file LRUs.
> diff --git a/mm/workingset.c b/mm/workingset.c
> index af862c6738c3..dab0c362b9e3 100644
> --- a/mm/workingset.c
> +++ b/mm/workingset.c
> @@ -462,7 +462,7 @@ void workingset_refault(struct folio *folio, void *shadow)
>
> mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr);
>
> - mem_cgroup_flush_stats_ratelimited();
> + mem_cgroup_flush_stats_atomic_ratelimited();
> /*
> * Compare the distance to the existing workingset size. We
> * don't activate pages that couldn't stay resident even if
> --
> 2.40.0.348.gf938b09366-goog
--
Michal Hocko
SUSE Labs
next prev parent reply other threads:[~2023-03-30 7:36 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-28 22:16 [PATCH v2 0/9] memcg: make rstat flushing irq and sleep Yosry Ahmed
2023-03-28 22:16 ` [PATCH v2 1/9] cgroup: rename cgroup_rstat_flush_"irqsafe" to "atomic" Yosry Ahmed
2023-03-28 22:16 ` [PATCH v2 2/9] memcg: rename mem_cgroup_flush_stats_"delayed" to "ratelimited" Yosry Ahmed
2023-03-29 11:56 ` Michal Hocko
2023-03-28 22:16 ` [PATCH v2 3/9] memcg: do not flush stats in irq context Yosry Ahmed
2023-03-29 11:58 ` Michal Hocko
2023-03-28 22:16 ` [PATCH v2 4/9] cgroup: rstat: add WARN_ON_ONCE() if flushing outside task context Yosry Ahmed
2023-03-29 11:22 ` Michal Hocko
2023-03-29 18:41 ` Yosry Ahmed
2023-03-29 19:20 ` Shakeel Butt
2023-03-30 7:06 ` Michal Hocko
2023-03-30 7:13 ` Yosry Ahmed
2023-03-30 7:49 ` Michal Hocko
2023-03-30 8:06 ` Yosry Ahmed
2023-03-30 8:14 ` Michal Hocko
2023-03-30 8:19 ` Yosry Ahmed
2023-03-30 8:39 ` Michal Hocko
2023-03-30 8:53 ` Yosry Ahmed
2023-03-31 11:02 ` Michal Hocko
2023-03-31 19:03 ` Yosry Ahmed
2023-04-03 8:38 ` Michal Hocko
2023-04-03 20:39 ` Yosry Ahmed
2023-03-28 22:16 ` [PATCH v2 5/9] memcg: replace stats_flush_lock with an atomic Yosry Ahmed
2023-03-28 22:22 ` Shakeel Butt
2023-03-29 15:58 ` Michal Hocko
2023-03-29 18:45 ` Yosry Ahmed
2023-03-28 22:16 ` [PATCH v2 6/9] memcg: sleep during flushing stats in safe contexts Yosry Ahmed
2023-03-30 7:35 ` Michal Hocko [this message]
2023-03-28 22:16 ` [PATCH v2 7/9] workingset: memcg: sleep when flushing stats in workingset_refault() Yosry Ahmed
2023-03-30 7:39 ` Michal Hocko
2023-03-30 7:42 ` Yosry Ahmed
2023-03-30 7:50 ` Michal Hocko
2023-03-30 7:55 ` Yosry Ahmed
2023-03-28 22:16 ` [PATCH v2 8/9] vmscan: memcg: sleep when flushing stats during reclaim Yosry Ahmed
2023-03-30 7:40 ` Michal Hocko
2023-03-30 7:44 ` Yosry Ahmed
2023-03-30 7:52 ` Michal Hocko
2023-03-30 7:54 ` Yosry Ahmed
2023-03-28 22:16 ` [PATCH v2 9/9] memcg: do not modify rstat tree for zero updates Yosry Ahmed
2023-03-30 7:43 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZCU73Ghje96Q0X7A@dhcp22.suse.cz \
--to=mhocko@suse.com \
--cc=akpm@linux-foundation.org \
--cc=axboe@kernel.dk \
--cc=bpf@vger.kernel.org \
--cc=cgroups@vger.kernel.org \
--cc=hannes@cmpxchg.org \
--cc=josef@toxicpanda.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=lizefan.x@bytedance.com \
--cc=mkoutny@suse.com \
--cc=muchun.song@linux.dev \
--cc=roman.gushchin@linux.dev \
--cc=shakeelb@google.com \
--cc=tj@kernel.org \
--cc=vasily.averin@linux.dev \
--cc=yosryahmed@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).