What can we do to get ready for memory controller merge in 2.6.25

What can we do to get ready for memory controller merge in 2.6.25

[Balbir Singh]

They say better strike when the iron is hot.

Since we have so many people discussing the memory controller, I would
like to access the readiness of the memory controller for mainline
merge. Given that we have some time until the merge window, I'd like to
set aside some time (from my other work items) to work on the memory
controller, fix review comments and defects.

In the past, we've received several useful comments from Rik Van Riel,
Lee Schermerhorn, Peter Zijlstra, Hugh Dickins, Nick Piggin, Paul Menage
and code contributions and bug fixes from Hugh Dickins, Pavel Emelianov,
Lee Schermerhorn, YAMAMOTO-San, Andrew Morton and KAMEZAWA-San. I
apologize if I missed out any other names or contributions

At the VM-Summit we decided to try the current double LRU approach for
memory control. At this juncture in the space-time continuum, I seek
your support, feedback, comments and help to move the memory controller

-- 
	Thanks,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: What can we do to get ready for memory controller merge in 2.6.25

[Rik van Riel]

On Thu, 29 Nov 2007 20:13:17 +0530
Balbir Singh <balbir@linux.vnet.ibm.com> wrote:

> They say better strike when the iron is hot.
> 
> Since we have so many people discussing the memory controller, I would
> like to access the readiness of the memory controller for mainline
> merge.

> At the VM-Summit we decided to try the current double LRU approach for
> memory control. At this juncture in the space-time continuum, I seek
> your support, feedback, comments and help to move the memory controller

The memory controller code currently in -mm seems fine to me,
especially with the changes that got committed over the last
days making reclaim more efficient.

I don't think there are any bugs left that can be found by
code inspection - only the kind of testing that the mainline
kernel gets might shake out more bugs.

I would like to see the memory controller code go into the
mainline kernel ASAP.

-- 
All Rights Reversed

Re: What can we do to get ready for memory controller merge in 2.6.25

[Balbir Singh]

Rik van Riel wrote:
> On Thu, 29 Nov 2007 20:13:17 +0530
> Balbir Singh <balbir@linux.vnet.ibm.com> wrote:
> 
>> They say better strike when the iron is hot.
>>
>> Since we have so many people discussing the memory controller, I would
>> like to access the readiness of the memory controller for mainline
>> merge.
> 
>> At the VM-Summit we decided to try the current double LRU approach for
>> memory control. At this juncture in the space-time continuum, I seek
>> your support, feedback, comments and help to move the memory controller
> 
> The memory controller code currently in -mm seems fine to me,
> especially with the changes that got committed over the last
> days making reclaim more efficient.
> 

Yes, I agree. Per zone reclaim and lists have helped make the code
better. Credit goes to KAMEZAWA-San for the per zone code and to
YAMAMOTO-San for background reclaim.

> I don't think there are any bugs left that can be found by
> code inspection - only the kind of testing that the mainline
> kernel gets might shake out more bugs.
> 
> I would like to see the memory controller code go into the
> mainline kernel ASAP.
> 

Excellent, thanks!


-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: What can we do to get ready for memory controller merge in 2.6.25

[Nick Piggin]

On Friday 30 November 2007 01:43, Balbir Singh wrote:
> They say better strike when the iron is hot.
>
> Since we have so many people discussing the memory controller, I would
> like to access the readiness of the memory controller for mainline
> merge. Given that we have some time until the merge window, I'd like to
> set aside some time (from my other work items) to work on the memory
> controller, fix review comments and defects.
>
> In the past, we've received several useful comments from Rik Van Riel,
> Lee Schermerhorn, Peter Zijlstra, Hugh Dickins, Nick Piggin, Paul Menage
> and code contributions and bug fixes from Hugh Dickins, Pavel Emelianov,
> Lee Schermerhorn, YAMAMOTO-San, Andrew Morton and KAMEZAWA-San. I
> apologize if I missed out any other names or contributions
>
> At the VM-Summit we decided to try the current double LRU approach for
> memory control. At this juncture in the space-time continuum, I seek
> your support, feedback, comments and help to move the memory controller

Do you have any test cases, performance numbers, etc.? And also some
results or even anecdotes of where this is going to be used would be
interesting...

Thanks,
Nick

Re: What can we do to get ready for memory controller merge in 2.6.25

[Balbir Singh]

Nick Piggin wrote:
> On Friday 30 November 2007 01:43, Balbir Singh wrote:
>> They say better strike when the iron is hot.
>>
>> Since we have so many people discussing the memory controller, I would
>> like to access the readiness of the memory controller for mainline
>> merge. Given that we have some time until the merge window, I'd like to
>> set aside some time (from my other work items) to work on the memory
>> controller, fix review comments and defects.
>>
>> In the past, we've received several useful comments from Rik Van Riel,
>> Lee Schermerhorn, Peter Zijlstra, Hugh Dickins, Nick Piggin, Paul Menage
>> and code contributions and bug fixes from Hugh Dickins, Pavel Emelianov,
>> Lee Schermerhorn, YAMAMOTO-San, Andrew Morton and KAMEZAWA-San. I
>> apologize if I missed out any other names or contributions
>>
>> At the VM-Summit we decided to try the current double LRU approach for
>> memory control. At this juncture in the space-time continuum, I seek
>> your support, feedback, comments and help to move the memory controller
> 
> Do you have any test cases, performance numbers, etc.? And also some
> results or even anecdotes of where this is going to be used would be
> interesting...
> 

Some test results were posted at

http://lkml.org/lkml/2007/8/17/69
http://lkml.org/lkml/2007/8/19/36
http://lwn.net/Articles/242554/

Some results for the RSS controller can be found in the OLS paper

https://ols2006.108.redhat.com/2007/Reprints/singh-Reprint.pdf

and at

http://lkml.org/lkml/2007/5/18/1

As far as test cases are concerned, I have a simple test case that I use
that allocates memory and touches all the allocated memory in a loop. I
can post that out if required. It uses various types of allocation

1. mmaped memory
2. anonymous memory
3. shared memory

I also run various benchmarks inside a control group, limited to 400 MB
of RAM.

One interesting that I noticed was that when I booted with mem=<some
memory> and created a container with the same <some value>. The swapout
test case ran much faster in the container (NOTE: This was prior to the
swap cache changes).

KAMEZAWA-San posted some test results on background reclaim and per zone
reclaim

http://forum.openvz.org/index.php?t=tree&th=4696&mid=23964&&rev=&reveal=

The simplest use cases that come to mind are

1. Memory control for containers/virtualization
2. Job Isolation


-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: What can we do to get ready for memory controller merge in 2.6.25

[KAMEZAWA Hiroyuki]

On Fri, 30 Nov 2007 08:43:35 +0530
Balbir Singh <balbir@linux.vnet.ibm.com> wrote:
> KAMEZAWA-San posted some test results on background reclaim and per zone
> reclaim
> 

I'd like to post some patches below in the next week.
  - throttling the number of callers of try_to_free_mem_cgroup_pages()
  - background reclaim and high/low watermark.
  - some cleanups.

And they are all patches I have now (for this window)

Thanks,
-Kame

Re: What can we do to get ready for memory controller merge in 2.6.25

[Paul Menage]

On Nov 29, 2007 6:11 PM, Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> And also some
> results or even anecdotes of where this is going to be used would be
> interesting...

We want to be able to run multiple isolated jobs on the same machine.
So being able to limit how much memory each job can consume, in terms
of anonymous memory and page cache, are useful. I've not had much time
to look at the patches in great detail, but they seem to provide a
sensible way to assign and enforce static limits on a bunch of jobs.

Some of our requirements are a bit beyond this, though:

In our experience, users are not good at figuring out how much memory
they really need. In general they tend to massively over-estimate
their requirements. So we want some way to determine how much of its
allocated memory a job is actively using, and how much could be thrown
away or swapped out without bothering the job too much.

Of course, the definition of "actve use" is tricky - one possibility
that we're looking at is "has been accessed within the last N
seconds", where N can be configured appropriately for different jobs
depending on the job's latency requirements. Active use should also be
reported for pages that can't be easily freed quickly, e.g. mlocked or
dirty pages, or anon pages on a swapless system. Inactive pages should
be easily freeable, and be the first ones to go in the event of memory
pressure. (From a scheduling point of view we can treat them as free
memory, and schedule more jobs on the machine)

The existing active/inactive distinction doesn't really capture this,
since it's relative rather than absolute.

We want to be able to overcommit a machine, so the sums of the cgroup
memory limits can add up to more than the total machine memory. So we
need control over what happens when there's global memory pressure,
and a way to ensure that the low-latency jobs don't get bogged down in
reclaim (or OOM) due to the activity of batch jobs.

Paul

Re: What can we do to get ready for memory controller merge in 2.6.25

[Balbir Singh]

Paul Menage wrote:
> On Nov 29, 2007 6:11 PM, Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>> And also some
>> results or even anecdotes of where this is going to be used would be
>> interesting...
> 
> We want to be able to run multiple isolated jobs on the same machine.
> So being able to limit how much memory each job can consume, in terms
> of anonymous memory and page cache, are useful. I've not had much time
> to look at the patches in great detail, but they seem to provide a
> sensible way to assign and enforce static limits on a bunch of jobs.
> 
> Some of our requirements are a bit beyond this, though:
> 
> In our experience, users are not good at figuring out how much memory
> they really need. In general they tend to massively over-estimate
> their requirements. So we want some way to determine how much of its
> allocated memory a job is actively using, and how much could be thrown
> away or swapped out without bothering the job too much.
> 

One would prefer the kernel provides the mechanism and user space
provides the policy. The algorithms to assign limits can exist in user
space and be supported by a good set of statistics.

> Of course, the definition of "actve use" is tricky - one possibility
> that we're looking at is "has been accessed within the last N
> seconds", where N can be configured appropriately for different jobs
> depending on the job's latency requirements. Active use should also be
> reported for pages that can't be easily freed quickly, e.g. mlocked or
> dirty pages, or anon pages on a swapless system. Inactive pages should
> be easily freeable, and be the first ones to go in the event of memory
> pressure. (From a scheduling point of view we can treat them as free
> memory, and schedule more jobs on the machine)
> 

This definition of active comes from the mainline kernel, which in-turn
is derived from our understanding of the working set.

> The existing active/inactive distinction doesn't really capture this,
> since it's relative rather than absolute.
> 

Not sure I understand why we need absolute use and not relative use.

> We want to be able to overcommit a machine, so the sums of the cgroup
> memory limits can add up to more than the total machine memory. So we
> need control over what happens when there's global memory pressure,
> and a way to ensure that the low-latency jobs don't get bogged down in
> reclaim (or OOM) due to the activity of batch jobs.
> 

I agree, well said. We need Job Isolation.

> Paul


-- 
	Warm Regards,
	Balbir Singh
	Linux Technology Center
	IBM, ISTL

Re: What can we do to get ready for memory controller merge in 2.6.25

[Rik van Riel]

On Sat, 01 Dec 2007 15:20:29 +0530
Balbir Singh <balbir@linux.vnet.ibm.com> wrote:

> > In our experience, users are not good at figuring out how much memory
> > they really need. In general they tend to massively over-estimate
> > their requirements. So we want some way to determine how much of its
> > allocated memory a job is actively using, and how much could be thrown
> > away or swapped out without bothering the job too much.
> 
> One would prefer the kernel provides the mechanism and user space
> provides the policy. The algorithms to assign limits can exist in user
> space and be supported by a good set of statistics.

With the /proc/refaults info, we can measure how much extra
memory each process group needs, if any.

As for how much memory a process group needs, at pageout time
we can check the fraction of pages that are accessed.  If 60%
of the pages were recently accessed at pageout time and this
process group is spending little or no time waiting for refaults,
40% of the pages are *not* recently accessed and we can probably
reduce the amount of memory assigned to this group.

Page cache that has only been accessed once can also be
counted as "not recently accessed", since streaming file
IO should not increase the working set of the process group.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: What can we do to get ready for memory controller merge in 2.6.25

[Paul Menage]

On Dec 1, 2007 10:36 AM, Rik van Riel <riel@redhat.com> wrote:
>
> With the /proc/refaults info, we can measure how much extra
> memory each process group needs, if any.

What's the status of that? It looks as though it would be better than
the "accessed in the last N seconds" metric that we've been playing
with, although it's possibly more intrusive?

Would it be practical to keep a non-resident set for each cgroup?

>
> As for how much memory a process group needs, at pageout time
> we can check the fraction of pages that are accessed.  If 60%
> of the pages were recently accessed at pageout time and this
> process group is spending little or no time waiting for refaults,
> 40% of the pages are *not* recently accessed and we can probably
> reduce the amount of memory assigned to this group.

It would probably be better to reduce its background-reclaim high
watermark than to reduce its limit. If you do the latter, you risk
triggering an OOM in the cgroup if it turns out that it did need all
that memory after all.

Paul

Re: What can we do to get ready for memory controller merge in 2.6.25

[Rik van Riel]

On Sat, 1 Dec 2007 11:02:32 -0800
"Paul Menage" <menage@google.com> wrote:

> On Dec 1, 2007 10:36 AM, Rik van Riel <riel@redhat.com> wrote:
> >
> > With the /proc/refaults info, we can measure how much extra
> > memory each process group needs, if any.
> 
> What's the status of that? It looks as though it would be better than
> the "accessed in the last N seconds" metric that we've been playing
> with, although it's possibly more intrusive?
> 
> Would it be practical to keep a non-resident set for each cgroup?

I have an implementation with a global array, but will have to
change it over to a per-radix tree implementation (not that
hard, with the slab reclaiming code) and per-cgroup reclaiming
information.

That way we can figure out per mapping, per cgroup or system
wide reclaim info (though not all at the same time).

> > As for how much memory a process group needs, at pageout time
> > we can check the fraction of pages that are accessed.  If 60%
> > of the pages were recently accessed at pageout time and this
> > process group is spending little or no time waiting for refaults,
> > 40% of the pages are *not* recently accessed and we can probably
> > reduce the amount of memory assigned to this group.
> 
> It would probably be better to reduce its background-reclaim high
> watermark than to reduce its limit. If you do the latter, you risk
> triggering an OOM in the cgroup if it turns out that it did need all
> that memory after all.

I did mean the RSS limit, not a virtual memory limit.

What exactly are all the terminologies you use that
I need to be aware of? :)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: What can we do to get ready for memory controller merge in 2.6.25

[KAMEZAWA Hiroyuki]

On Fri, 30 Nov 2007 19:11:14 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> I'd like to post some patches below in the next week.
>   - throttling the number of callers of try_to_free_mem_cgroup_pages()
>   - background reclaim and high/low watermark.
>   - some cleanups.
> 
I'd like to hold off "big changes" until the end of 2.6.25 merge window
and to make current memory controller be tested by many people.

I'll maintain new feature patches in countainer mailing list for a while.

Thanks,
-Kame