Re: OOM problems

[John Weekes <lists.xen@nuclearfallout.net>]

Performance is noticeably lower with aio on these bursty write 
workloads; I've been getting a number of complaints.

I see that 2.6.36 has some page_writeback changes:
http://www.kernel.org/diff/diffview.cgi?file=%2Fpub%2Flinux%2Fkernel%2Fv2.6%2Fpatch-2.6.36.bz2;z=8379 
. Any thoughts on whether these would make a difference for the problems 
with "file:"? I'm still trying to find a way to reproduce the issue in 
the lab, so I'd have to test the patch in production -- that's not a 
tantalizing prospect, unless there is a real chance that it will affect it.

-John

On 11/15/2010 9:59 AM, John Weekes wrote:
>
>> They are throttled, but the single control I'm aware of
>> is /proc/sys/vm/dirty_ratio (or dirty_bytes, nowadays). Which is only
>> per process, not a global limit. Could well be that's part of the
>> problem -- outwitting mm with just too many writers on too many cores?
>>
>> We had a bit of trouble when switching dom0 to 2.6.32, buffered writes
>> made it much easier than with e.g. 2.6.27 to drive everybody else into
>> costly reclaims.
>>
>> The Oom shown here reports about ~650M in dirty pages. The fact alone
>> that this counts as on oom condition doesn't sound quite right in
>> itself. That qemu might just have dared to ask at the wrong point in
>> time.
>>
>> Just to get an idea -- how many guests did this box carry?
>
> It carries about two dozen guests, with a mix of mostly HVMs (all 
> stubdom-based, some with PV-on-HVM drivers) and some PV.
>
> This problem occurred more often for me under 2.6.32 than 2.6.31, I 
> noticed. Since I made the switch to aio, I haven't seen a crash, but 
> it hasn't been long enough for that to mean much.
>
> Having extra caching in the dom0 is nice because it allows for domUs 
> to get away with having small amounts of free memory, while still 
> having very good (much faster than hardware) write performance. If you 
> have a large number of domUs that are all memory-constrained and use 
> the disk in infrequent, large bursts, this can work out pretty well, 
> since the big communal pool provides a better value proposition than 
> giving each domU a few more megabytes of RAM.
>
> If the OOM problem isn't something that can be fixed, it might be a 
> good idea to print out a warning to the user when a domain using 
> "file:" is started. Or, to go a step further and automatically run 
> "file" based domains as though "aio" was specified, possibly with a 
> warning and a way to override that behavior. It's not really intuitive 
> that "file" would cause crashes.
>
> -John
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel