linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: David Rientjes <rientjes@google.com>
To: Christoph Lameter <cl@linux-foundation.org>
Cc: Pekka Enberg <penberg@cs.helsinki.fi>,
	Andrew Morton <akpm@linux-foundation.org>,
	Paul Menage <menage@google.com>,
	Randy Dunlap <randy.dunlap@oracle.com>,
	linux-kernel@vger.kernel.org
Subject: Re: [patch 2/2] slub: enforce cpuset restrictions for cpu slabs
Date: Tue, 3 Mar 2009 09:19:07 -0800 (PST)	[thread overview]
Message-ID: <alpine.DEB.2.00.0903030900580.6643@chino.kir.corp.google.com> (raw)
In-Reply-To: <alpine.DEB.1.10.0903031136380.26454@qirst.com>

On Tue, 3 Mar 2009, Christoph Lameter wrote:

> > Slab allocations should respect cpuset hardwall restrictions.  Otherwise,
> > it is possible for tasks in a cpuset to fill slabs allocated on mems
> > assigned to a disjoint cpuset.
> 
> Not sure that I understand this correctly. If multiple tasks are running
> on the same processor that are part of disjoint cpusets and both taska are
> performing slab allocations without specifying a node then one task could
> allocate a page from the first cpuset, take one object from it and then
> the second task on the same cpu could consume the rest from a nodeset that
> it would otherwise not be allowed to access. On the other hand it is
> likely that the second task will also allocate memory from its allowed
> nodes that are then consumed by the first task. This is a tradeoff coming
> with the pushing of the enforcement of memory policy / cpuset stuff out of
> the slab allocator and relying for this on the page allocator.
> 

Yes, I agree that it's a significant optimization to allow the cpu slab to 
be used by tasks that are not allowed, either because of its mempolicy or 
cpuset restriction, to access the node on which it was allocated.  That's 
especially true for small object sizes or short-lived allocations where 
the hardwall infringment is acceptable for the speed-up.

Unfortunately, it also leads to a violation of the user imposed 
restriction on acceptable memory usage.  One of the important aspects of 
cpusets is to allow memory isolation from other siblings.  It should be 
possible to kill all tasks in a cpuset, for example, and expect its 
partial list to be emptied and not heavily fragmented by long-lived 
allocations that could prevent any partial slab freeing, which is possible 
when heavy slab users are allowed to allocate objects anywhere.

> > If an allocation is intended for a particular node that the task does not
> > have access to because of its cpuset, an allowed partial slab is used
> > instead of failing.
> 
> This would get us back to the slab allocator enforcing memory policies.
> 

Is that a problem?  get_any_partial() already enforces cpuset-aware memory 
policies when defragmenting remote partial slabs.

> > -static inline int node_match(struct kmem_cache_cpu *c, int node)
> > +static inline int node_match(struct kmem_cache_cpu *c, int node, gfp_t gfpflags)
> >  {
> >  #ifdef CONFIG_NUMA
> >  	if (node != -1 && c->node != node)
> >  		return 0;
> >  #endif
> > -	return 1;
> > +	return cpuset_node_allowed_hardwall(c->node, gfpflags);
> >  }
> 
> This is a hotpath function and doing an expensive function call here would
> significantly impact performance.
> 

It's not expensive.  It's a no-op for !CONFIG_CPUSETS configs and only a 
global variable read for machines running with a single cpuset.  When the 
machine has multiple cpusets, it indicates that memory restrictions are in 
place so checking current->mems_allowed is required and its performance 
impact should be assumed.

> It also will cause a reloading of the per cpu slab after each task switch
> in the scenario discussed above.
> 

There is no alternative solution to prevent egregious amounts of slab to 
be allocated in a disjoint cpuset that is supposedly mem_exclusive.

  reply	other threads:[~2009-03-03 17:19 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2009-03-03  4:50 [patch 1/2] cpusets: replace zone allowed functions with node allowed David Rientjes
2009-03-03  4:50 ` [patch 2/2] slub: enforce cpuset restrictions for cpu slabs David Rientjes
2009-03-03 16:47   ` Christoph Lameter
2009-03-03 17:19     ` David Rientjes [this message]
2009-03-03 18:46       ` Christoph Lameter
2009-03-03 19:05         ` David Rientjes
2009-03-03 20:14           ` Christoph Lameter
2009-03-03 21:35             ` David Rientjes
2009-03-03 21:55               ` Paul Menage
2009-03-03 22:29                 ` David Rientjes
2009-03-04  0:53                   ` Paul Menage
2009-03-04  1:14                     ` David Rientjes
2009-03-04 15:23                       ` Christoph Lameter
2009-03-04 18:06                         ` David Rientjes
2009-03-03 16:36 ` [patch 1/2] cpusets: replace zone allowed functions with node allowed Christoph Lameter

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=alpine.DEB.2.00.0903030900580.6643@chino.kir.corp.google.com \
    --to=rientjes@google.com \
    --cc=akpm@linux-foundation.org \
    --cc=cl@linux-foundation.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=menage@google.com \
    --cc=penberg@cs.helsinki.fi \
    --cc=randy.dunlap@oracle.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).