RE: OOM problems

From: Daniel Stodden <daniel.stodden@citrix.com>
To: Jan Beulich <JBeulich@novell.com>
Cc: Ian Pratt <Ian.Pratt@eu.citrix.com>,
	"xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>,
	John Weekes <lists.xen@nuclearfallout.net>
Subject: RE: OOM problems
Date: Mon, 15 Nov 2010 01:40:37 -0800	[thread overview]
Message-ID: <1289814037.21694.22.camel@ramone> (raw)
In-Reply-To: <4CE1037402000078000222F0@vpn.id2.novell.com>

On Mon, 2010-11-15 at 03:55 -0500, Jan Beulich wrote:
> >>> On 13.11.10 at 10:13, Ian Pratt <Ian.Pratt@eu.citrix.com> wrote:
> >>   > What do the guests use for storage? (e.g. "blktap2 for VHD files on
> >> an iscsi mounted ext3 volume")
> >> 
> >> Simple sparse .img files on a local ext4 RAID volume, using "file:".
> > 
> > Ah, if you're using loop it may be that you're just filling memory with 
> > dirty pages. Older kernels certainly did this, not sure about newer ones.
> 
> Shouldn't this lead to the calling process being throttled, instead of
> the system running into OOM?

They are throttled, but the single control I'm aware of
is /proc/sys/vm/dirty_ratio (or dirty_bytes, nowadays). Which is only
per process, not a global limit. Could well be that's part of the
problem -- outwitting mm with just too many writers on too many cores?

We had a bit of trouble when switching dom0 to 2.6.32, buffered writes
made it much easier than with e.g. 2.6.27 to drive everybody else into
costly reclaims.

The Oom shown here reports about ~650M in dirty pages. The fact alone
that this counts as on oom condition doesn't sound quite right in
itself. That qemu might just have dared to ask at the wrong point in
time.

Just to get an idea -- how many guests did this box carry?

> Further, having got reports of similar problems lately, too, we have
> indications that using pv drivers also gets us around the issue,
> which makes me think that it's rather qemu-dm misbehaving (and
> not getting stopped doing so by the kernel for whatever reason -
> possibly just missing some non-infinite rlimit setting).
> 
> Not knowing much about the workings of stubdom, one thing I
> don't really understand is how qemu-dm in Dom0 would be
> heavily resource consuming here (actually I would have expected
> no qemu-dm in Dom0 at all in this case). Aren't the main I/O paths
> going from qemu-stubdom directly to the backends?
> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel