From: Johannes Weiner <firstname.lastname@example.org> To: Michal Hocko <email@example.com> Cc: Tejun Heo <firstname.lastname@example.org>, Shakeel Butt <email@example.com>, Jakub Kicinski <firstname.lastname@example.org>, Andrew Morton <email@example.com>, Linux MM <firstname.lastname@example.org>, Kernel Team <email@example.com>, Chris Down <firstname.lastname@example.org>, Cgroups <email@example.com> Subject: Re: [PATCH 0/3] memcg: Slow down swap allocation as the available space gets depleted Date: Tue, 28 Apr 2020 10:24:32 -0400 [thread overview] Message-ID: <20200428142432.GA78561@cmpxchg.org> (raw) In-Reply-To: <20200424150510.GH11591@dhcp22.suse.cz> On Fri, Apr 24, 2020 at 05:05:10PM +0200, Michal Hocko wrote: > On Thu 23-04-20 11:00:15, Johannes Weiner wrote: > > On Wed, Apr 22, 2020 at 08:49:21PM +0200, Michal Hocko wrote: > > > On Wed 22-04-20 13:13:28, Johannes Weiner wrote: > > > > On Wed, Apr 22, 2020 at 05:43:18PM +0200, Michal Hocko wrote: > > > > > On Wed 22-04-20 10:15:14, Johannes Weiner wrote: > > > > > I am also missing some information about what the user can actually do > > > > > about this situation and call out explicitly that the throttling is > > > > > not going away until the swap usage is shrunk and the kernel is not > > > > > capable of doing that on its own without a help from the userspace. This > > > > > is really different from memory.high which has means to deal with the > > > > > excess and shrink it down in most cases. The following would clarify it > > > > > > > > I think we may be talking past each other. The user can do the same > > > > thing as in any OOM situation: wait for the kill. > > > > > > That assumes that reaching swap.high is going to converge to the OOM > > > eventually. And that is far from the general case. There might be a > > > lot of other reclaimable memory to reclaim and stay in the current > > > state. > > > > No, that's really the general case. And that's based on what users > > widely experience, including us at FB. When swap is full, it's over. > > Multiple parties have independently reached this conclusion. > > But we are talking about two things. You seem to be focusing on the full > swap (quota) while I am talking about swap.high which doesn't imply > that the quota/full swap is going to be reached soon. Hm, I'm not quite sure I understand. swap.high is supposed to set this quota. It's supposed to say: the workload has now shown such an appetite for swap that it's unlikely to survive for much longer - draw out its death just long enough for userspace OOM handling. Maybe this is our misunderstanding? It certainly doesn't make much sense to set swap.high to 0 or relatively low values. Should we add the above to the doc text? > > Once the workload expands its set of *used* pages past memory.high, we > > are talking about indefinite slowdowns / OOM situations. Because at > > that point, reclaim cannot push the workload back and everything will > > be okay: the pages it takes off mean refaults and continued reclaim, > > i.e. throttling. You get slowed down either way, and whether you > > reclaim or sleep() is - to the workload - an accounting difference. > > > > Reclaim does NOT have the power to help the workload get better. It > > can only do amputations to protect the rest of the system, but it > > cannot reduce the number of pages the workload is trying to access. > > Yes I do agree with you here and I believe this scenario wasn't really > what the dispute is about. As soon as the real working set doesn't > fit into the high limit and still growing then you are effectively > OOM and either you do handle that from the userspace or you have to > waaaaaaaaait for the kernel oom killer to trigger. > > But I believe this scenario is much easier to understand because the > memory consumption is growing. What I find largely unintuitive from the > user POV is that the throttling will remain in place without a userspace > intervention even when there is no runaway. > > Let me give you an example. Say you have a peak load which pushes > out a large part of an idle memory to swap. So much it fills up the > swap.high. The peak eventually finishes freeing up its resources. The > swap situation remains the same because that memory is not refaulted and > we do not pro-actively swap in memory (aka reclaim the swap space). You > are left with throttling even though the overall memcg consumption is > really low. Kernel is currently not able to do anything about that > and the userspace would need to be aware of the situation to fault in > swapped out memory back to get a normal behavior. Do you think this > is something so obvious that people would keep it in mind when using > swap.high? Okay, thanks for clarifying, I understand your concern now. This is not a scenario that swap.high is supposed to handle. It should *not* be set to an amount of memory that the workload can reasonably have sitting around idle. For example, if your memory allowance is 10G, it doesn't make sense to have swap.high at 200M or something. It should be set to "we don't expect healthy workloads to get here". And now I also understand what you mean by being different to memory.high. memory.high is definitely *expected* to get hit because of the cache trimming usecase. We just don't expect the *throttling* part to get into play unless the workload is truly unhealthy. But I can see how user expectations toward swap.high could be different. > Anyway, it seems that we are not making progress here. As I've said I > believe that swap.high might lead to a surprising behavior and therefore > I would appreciate more clarity in the documentation. If you see a > problem with that for some reason then I can live with that. This is not > a reason to nack. No, I agree we should document this. How about the following? memory.swap.high A read-write single value file which exists on non-root cgroups. The default is "max". Swap usage throttle limit. If a cgroup's swap usage exceeds this limit, all its further allocations will be throttled to allow userspace to implement custom out-of-memory procedures. This limit marks a point of no return for the cgroup. It is NOT designed to manage the amount of swapping a workload does during regular operation. Compare to memory.swap.max, which prohibits swapping past a set amount, but lets the cgroup continue unimpeded as long as other memory can be reclaimed.
next prev parent reply other threads:[~2020-04-28 14:24 UTC|newest] Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-04-17 1:06 Jakub Kicinski 2020-04-17 1:06 ` [PATCH 1/3] mm: prepare for swap over-high accounting and penalty calculation Jakub Kicinski 2020-04-17 1:06 ` [PATCH 2/3] mm: move penalty delay clamping out of calculate_high_delay() Jakub Kicinski 2020-04-17 1:06 ` [PATCH 3/3] mm: automatically penalize tasks with high swap use Jakub Kicinski 2020-04-17 7:37 ` Michal Hocko 2020-04-17 23:22 ` Jakub Kicinski 2020-04-17 16:11 ` [PATCH 0/3] memcg: Slow down swap allocation as the available space gets depleted Shakeel Butt 2020-04-17 16:23 ` Tejun Heo 2020-04-17 17:18 ` Shakeel Butt 2020-04-17 17:36 ` Tejun Heo 2020-04-17 17:51 ` Shakeel Butt 2020-04-17 19:35 ` Tejun Heo 2020-04-17 21:51 ` Shakeel Butt 2020-04-17 22:59 ` Tejun Heo 2020-04-20 16:12 ` Shakeel Butt 2020-04-20 16:47 ` Tejun Heo 2020-04-20 17:03 ` Michal Hocko 2020-04-20 17:06 ` Tejun Heo 2020-04-21 11:06 ` Michal Hocko 2020-04-21 14:27 ` Johannes Weiner 2020-04-21 16:11 ` Michal Hocko 2020-04-21 16:56 ` Johannes Weiner 2020-04-22 13:26 ` Michal Hocko 2020-04-22 14:15 ` Johannes Weiner 2020-04-22 15:43 ` Michal Hocko 2020-04-22 17:13 ` Johannes Weiner 2020-04-22 18:49 ` Michal Hocko 2020-04-23 15:00 ` Johannes Weiner 2020-04-24 15:05 ` Michal Hocko 2020-04-28 14:24 ` Johannes Weiner [this message] 2020-04-29 9:55 ` Michal Hocko 2020-04-21 19:09 ` Shakeel Butt 2020-04-21 21:59 ` Johannes Weiner 2020-04-21 22:39 ` Shakeel Butt 2020-04-21 15:20 ` Tejun Heo
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200428142432.GA78561@cmpxchg.org \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: [PATCH 0/3] memcg: Slow down swap allocation as the available space gets depleted' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).