From: Daniel Jordan <daniel.m.jordan@oracle.com> To: Johannes Weiner <hannes@cmpxchg.org> Cc: Daniel Jordan <daniel.m.jordan@oracle.com>, Michal Hocko <mhocko@kernel.org>, Andrew Morton <akpm@linux-foundation.org>, Tejun Heo <tj@kernel.org>, Roman Gushchin <guro@fb.com>, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com, Peter Zijlstra <peterz@infradead.org> Subject: Re: [PATCH] mm: memcontrol: asynchronous reclaim for memory.high Date: Thu, 20 Feb 2020 10:45:24 -0500 [thread overview] Message-ID: <20200220154524.dql3i5brnjjwecft@ca-dmjordan1.us.oracle.com> (raw) In-Reply-To: <20200219220859.GF54486@cmpxchg.org> +Peter On Wed, Feb 19, 2020 at 05:08:59PM -0500, Johannes Weiner wrote: > On Wed, Feb 19, 2020 at 04:41:12PM -0500, Daniel Jordan wrote: > > On Wed, Feb 19, 2020 at 08:53:32PM +0100, Michal Hocko wrote: > > > On Wed 19-02-20 14:16:18, Johannes Weiner wrote: > > > > On Wed, Feb 19, 2020 at 07:37:31PM +0100, Michal Hocko wrote: > > > > > On Wed 19-02-20 13:12:19, Johannes Weiner wrote: > > > > > > This patch adds asynchronous reclaim to the memory.high cgroup limit > > > > > > while keeping direct reclaim as a fallback. In our testing, this > > > > > > eliminated all direct reclaim from the affected workload. > > > > > > > > > > Who is accounted for all the work? Unless I am missing something this > > > > > just gets hidden in the system activity and that might hurt the > > > > > isolation. I do see how moving the work to a different context is > > > > > desirable but this work has to be accounted properly when it is going to > > > > > become a normal mode of operation (rather than a rare exception like the > > > > > existing irq context handling). > > > > > > > > Yes, the plan is to account it to the cgroup on whose behalf we're > > > > doing the work. > > > > How are you planning to do that? > > > > I've been thinking about how to account a kernel thread's CPU usage to a cgroup > > on and off while working on the parallelizing Michal mentions below. A few > > approaches are described here: > > > > https://lore.kernel.org/linux-mm/20200212224731.kmss6o6agekkg3mw@ca-dmjordan1.us.oracle.com/ > > What we do for the IO controller is execute the work unthrottled but > charge the cgroup on whose behalf we are executing with whatever cost > or time or bandwith that was incurred. The cgroup will pay off this > debt when it requests more of that resource. > [snip code pointers] Thanks! Figuring out how the io controllers dealt with remote charging was on my list, this makes it easier. > The plan for the CPU controller is similar. When a remote execution > begins, flush the current runtime accumulated (update_curr) and > associate the current thread with another cgroup (similar to > current->active_memcg); when remote execution is done, flush the > runtime delta to that cgroup and unset the remote context. Ok, consistency with io and memory is one advantage to doing it that way. Creating kthreads in cgroups also seems viable so far, and it's unclear whether either approach is significantly simpler or more maintainable than the other, at least to me. Is someone on your side working on remote charging right now? I was planning to post an RFD comparing these soon and it would make sense to include them.
WARNING: multiple messages have this Message-ID (diff)
From: Daniel Jordan <daniel.m.jordan-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> To: Johannes Weiner <hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org> Cc: Daniel Jordan <daniel.m.jordan-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, Michal Hocko <mhocko-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>, Andrew Morton <akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>, Tejun Heo <tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>, Roman Gushchin <guro-b10kYP2dOMg@public.gmane.org>, linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, kernel-team-b10kYP2dOMg@public.gmane.org, Peter Zijlstra <peterz-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> Subject: Re: [PATCH] mm: memcontrol: asynchronous reclaim for memory.high Date: Thu, 20 Feb 2020 10:45:24 -0500 [thread overview] Message-ID: <20200220154524.dql3i5brnjjwecft@ca-dmjordan1.us.oracle.com> (raw) In-Reply-To: <20200219220859.GF54486-druUgvl0LCNAfugRpC6u6w@public.gmane.org> +Peter On Wed, Feb 19, 2020 at 05:08:59PM -0500, Johannes Weiner wrote: > On Wed, Feb 19, 2020 at 04:41:12PM -0500, Daniel Jordan wrote: > > On Wed, Feb 19, 2020 at 08:53:32PM +0100, Michal Hocko wrote: > > > On Wed 19-02-20 14:16:18, Johannes Weiner wrote: > > > > On Wed, Feb 19, 2020 at 07:37:31PM +0100, Michal Hocko wrote: > > > > > On Wed 19-02-20 13:12:19, Johannes Weiner wrote: > > > > > > This patch adds asynchronous reclaim to the memory.high cgroup limit > > > > > > while keeping direct reclaim as a fallback. In our testing, this > > > > > > eliminated all direct reclaim from the affected workload. > > > > > > > > > > Who is accounted for all the work? Unless I am missing something this > > > > > just gets hidden in the system activity and that might hurt the > > > > > isolation. I do see how moving the work to a different context is > > > > > desirable but this work has to be accounted properly when it is going to > > > > > become a normal mode of operation (rather than a rare exception like the > > > > > existing irq context handling). > > > > > > > > Yes, the plan is to account it to the cgroup on whose behalf we're > > > > doing the work. > > > > How are you planning to do that? > > > > I've been thinking about how to account a kernel thread's CPU usage to a cgroup > > on and off while working on the parallelizing Michal mentions below. A few > > approaches are described here: > > > > https://lore.kernel.org/linux-mm/20200212224731.kmss6o6agekkg3mw-S51bK0XF4qpuJJETbFA3a0B3C2bhBk7L0E9HWUfgJXw@public.gmane.org/ > > What we do for the IO controller is execute the work unthrottled but > charge the cgroup on whose behalf we are executing with whatever cost > or time or bandwith that was incurred. The cgroup will pay off this > debt when it requests more of that resource. > [snip code pointers] Thanks! Figuring out how the io controllers dealt with remote charging was on my list, this makes it easier. > The plan for the CPU controller is similar. When a remote execution > begins, flush the current runtime accumulated (update_curr) and > associate the current thread with another cgroup (similar to > current->active_memcg); when remote execution is done, flush the > runtime delta to that cgroup and unset the remote context. Ok, consistency with io and memory is one advantage to doing it that way. Creating kthreads in cgroups also seems viable so far, and it's unclear whether either approach is significantly simpler or more maintainable than the other, at least to me. Is someone on your side working on remote charging right now? I was planning to post an RFD comparing these soon and it would make sense to include them.
next prev parent reply other threads:[~2020-02-20 15:45 UTC|newest] Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-02-19 18:12 [PATCH] mm: memcontrol: asynchronous reclaim for memory.high Johannes Weiner 2020-02-19 18:12 ` Johannes Weiner 2020-02-19 18:37 ` Michal Hocko 2020-02-19 18:37 ` Michal Hocko 2020-02-19 19:16 ` Johannes Weiner 2020-02-19 19:16 ` Johannes Weiner 2020-02-19 19:53 ` Michal Hocko 2020-02-19 19:53 ` Michal Hocko 2020-02-19 21:17 ` Johannes Weiner 2020-02-20 9:46 ` Michal Hocko 2020-02-20 9:46 ` Michal Hocko 2020-02-20 14:41 ` Johannes Weiner 2020-02-20 14:41 ` Johannes Weiner 2020-02-19 21:41 ` Daniel Jordan 2020-02-19 21:41 ` Daniel Jordan 2020-02-19 22:08 ` Johannes Weiner 2020-02-19 22:08 ` Johannes Weiner 2020-02-20 15:45 ` Daniel Jordan [this message] 2020-02-20 15:45 ` Daniel Jordan 2020-02-20 15:56 ` Tejun Heo 2020-02-20 15:56 ` Tejun Heo 2020-02-20 18:23 ` Daniel Jordan 2020-02-20 18:23 ` Daniel Jordan 2020-02-20 18:45 ` Tejun Heo 2020-02-20 18:45 ` Tejun Heo 2020-02-20 19:55 ` Daniel Jordan 2020-02-20 19:55 ` Daniel Jordan 2020-02-20 20:54 ` Tejun Heo 2020-02-20 20:54 ` Tejun Heo 2020-02-19 19:17 ` Chris Down 2020-02-19 19:17 ` Chris Down 2020-02-19 19:31 ` Andrew Morton 2020-02-19 19:31 ` Andrew Morton 2020-02-19 21:33 ` Johannes Weiner 2020-02-26 20:25 ` Shakeel Butt 2020-02-26 20:25 ` Shakeel Butt 2020-02-26 20:25 ` Shakeel Butt 2020-02-26 22:26 ` Johannes Weiner 2020-02-26 22:26 ` Johannes Weiner 2020-02-26 23:36 ` Shakeel Butt 2020-02-26 23:36 ` Shakeel Butt 2020-02-26 23:36 ` Shakeel Butt 2020-02-26 23:46 ` Johannes Weiner 2020-02-27 0:12 ` Yang Shi 2020-02-27 0:12 ` Yang Shi 2020-02-27 2:42 ` Shakeel Butt 2020-02-27 2:42 ` Shakeel Butt 2020-02-27 2:42 ` Shakeel Butt 2020-02-27 9:58 ` Michal Hocko 2020-02-27 9:58 ` Michal Hocko 2020-02-27 12:50 ` Johannes Weiner 2020-02-27 12:50 ` Johannes Weiner 2020-02-26 23:59 ` Yang Shi 2020-02-26 23:59 ` Yang Shi 2020-02-27 2:36 ` Shakeel Butt 2020-02-27 2:36 ` Shakeel Butt
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200220154524.dql3i5brnjjwecft@ca-dmjordan1.us.oracle.com \ --to=daniel.m.jordan@oracle.com \ --cc=akpm@linux-foundation.org \ --cc=cgroups@vger.kernel.org \ --cc=guro@fb.com \ --cc=hannes@cmpxchg.org \ --cc=kernel-team@fb.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mhocko@kernel.org \ --cc=peterz@infradead.org \ --cc=tj@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.