Re: User space out of memory approach

From: Andrea Arcangeli <andrea@suse.de>
To: Mauricio Lin <mauriciolin@gmail.com>
Cc: tglx@linutronix.de,
	Marcelo Tosatti <marcelo.tosatti@cyclades.com>,
	Edjard Souza Mota <edjard@gmail.com>,
	LKML <linux-kernel@vger.kernel.org>,
	Andrew Morton <akpm@osdl.org>
Subject: Re: User space out of memory approach
Date: Wed, 26 Jan 2005 01:49:01 +0100	[thread overview]
Message-ID: <20050126004901.GD7587@dualathlon.random> (raw)
In-Reply-To: <3f250c71050125161175234ef9@mail.gmail.com>

On Tue, Jan 25, 2005 at 08:11:19PM -0400, Mauricio Lin wrote:
> Sometimes the first application to be killed is XFree. AFAIK the

This makes more sense now. You need somebody trapping sigterm in order
to lockup and X sure traps it to recover the text console.

Can you replace this:

	if (cap_t(p->cap_effective) & CAP_TO_MASK(CAP_SYS_RAWIO)) {
		force_sig(SIGTERM, p);
	} else {
		force_sig(SIGKILL, p);
	}

with this?

	force_sig(SIGKILL, p);

in mm/oom_kill.c.

This should fix it. Problem is that SIGTERM is unsafe even if the app is
not malicious, there's not enough ram to pagein the userland sighander,
so the system lockups.

We need a sort of timeout where we fallback into SIGKILL if SIGTERM
didn't help.

Anyway this is not a new bug, I didn't touch a single bit in that code.
I'd really like to see current fixes merged, then we can take care of
root app getting killed reliably. In all my test I always run the
malicious app as non-root, and anyway I never trap sigterm (X is tiny in
my setup, so it never gets killed). Probably the GUI stuff you opened
has increased significantly X size for X to be killed.