All of lore.kernel.org
 help / color / mirror / Atom feed
From: Johannes Weiner <hannes@cmpxchg.org>
To: Michal Hocko <mhocko@suse.com>
Cc: Alan Cox <alan@llwyncelyn.cymru>, Christoph Hellwig <hch@lst.de>,
	Andrew Morton <akpm@linux-foundation.org>,
	linux-mm@kvack.org, linux-kernel@vger.kernel.org,
	kernel-team@fb.com
Subject: tty crash due to auto-failing vmalloc
Date: Tue, 3 Oct 2017 18:55:04 -0400	[thread overview]
Message-ID: <20171003225504.GA966@cmpxchg.org> (raw)

On some of our machines, we see this warning:

	/* switch the line discipline */
	tty->ldisc = ld;
	tty_set_termios_ldisc(tty, disc);
	retval = tty_ldisc_open(tty, tty->ldisc);
	if (retval) {
->		if (!WARN_ON(disc == N_TTY)) {
			tty_ldisc_put(tty->ldisc);
			tty->ldisc = NULL;
		}
	}

where the stack is

tty_ldisc_reinit
tty_ldisc_hangup
__tty_hangup
do_exit
do_signal
syscall

This is followed by a NULL pointer deref crash in n_tty_set_termios,
presumably when it tries to deref that unallocated tty->disc_data.

The only way n_tty_open() can fail is if the vmalloc in there fails.
struct n_tty_data isn't terribly big, but ever since the following
patch it doesn't even *try* the allocation:

commit 5d17a73a2ebeb8d1c6924b91e53ab2650fe86ffb
Author: Michal Hocko <mhocko@suse.com>
Date:   Fri Feb 24 14:58:53 2017 -0800

    vmalloc: back off when the current task is killed
    
    __vmalloc_area_node() allocates pages to cover the requested vmalloc
    size.  This can be a lot of memory.  If the current task is killed by
    the OOM killer, and thus has an unlimited access to memory reserves, it
    can consume all the memory theoretically.  Fix this by checking for
    fatal_signal_pending and back off early.
    
    Link: http://lkml.kernel.org/r/20170201092706.9966-4-mhocko@kernel.org
    Signed-off-by: Michal Hocko <mhocko@suse.com>
    Reviewed-by: Christoph Hellwig <hch@lst.de>
    Cc: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
    Cc: Al Viro <viro@zeniv.linux.org.uk>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

This talks about the oom killer and memory exhaustion, but most fatal
signals don't happen due to the OOM killer.

I think this patch should be reverted. If somebody is vmallocing crazy
amounts of memory in the exit path we should probably track them down
individually; the patch doesn't reference any real instances of that.
But we cannot start failing allocations that have never failed before.

That said, maybe we want Alan's N_NULL failover in the hangup path too?

WARNING: multiple messages have this Message-ID (diff)
From: Johannes Weiner <hannes@cmpxchg.org>
To: Michal Hocko <mhocko@suse.com>
Cc: Alan Cox <alan@llwyncelyn.cymru>, Christoph Hellwig <hch@lst.de>,
	Andrew Morton <akpm@linux-foundation.org>,
	linux-mm@kvack.org, linux-kernel@vger.kernel.org,
	kernel-team@fb.com
Subject: tty crash due to auto-failing vmalloc
Date: Tue, 3 Oct 2017 18:55:04 -0400	[thread overview]
Message-ID: <20171003225504.GA966@cmpxchg.org> (raw)

On some of our machines, we see this warning:

	/* switch the line discipline */
	tty->ldisc = ld;
	tty_set_termios_ldisc(tty, disc);
	retval = tty_ldisc_open(tty, tty->ldisc);
	if (retval) {
->		if (!WARN_ON(disc == N_TTY)) {
			tty_ldisc_put(tty->ldisc);
			tty->ldisc = NULL;
		}
	}

where the stack is

tty_ldisc_reinit
tty_ldisc_hangup
__tty_hangup
do_exit
do_signal
syscall

This is followed by a NULL pointer deref crash in n_tty_set_termios,
presumably when it tries to deref that unallocated tty->disc_data.

The only way n_tty_open() can fail is if the vmalloc in there fails.
struct n_tty_data isn't terribly big, but ever since the following
patch it doesn't even *try* the allocation:

commit 5d17a73a2ebeb8d1c6924b91e53ab2650fe86ffb
Author: Michal Hocko <mhocko@suse.com>
Date:   Fri Feb 24 14:58:53 2017 -0800

    vmalloc: back off when the current task is killed
    
    __vmalloc_area_node() allocates pages to cover the requested vmalloc
    size.  This can be a lot of memory.  If the current task is killed by
    the OOM killer, and thus has an unlimited access to memory reserves, it
    can consume all the memory theoretically.  Fix this by checking for
    fatal_signal_pending and back off early.
    
    Link: http://lkml.kernel.org/r/20170201092706.9966-4-mhocko@kernel.org
    Signed-off-by: Michal Hocko <mhocko@suse.com>
    Reviewed-by: Christoph Hellwig <hch@lst.de>
    Cc: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
    Cc: Al Viro <viro@zeniv.linux.org.uk>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

This talks about the oom killer and memory exhaustion, but most fatal
signals don't happen due to the OOM killer.

I think this patch should be reverted. If somebody is vmallocing crazy
amounts of memory in the exit path we should probably track them down
individually; the patch doesn't reference any real instances of that.
But we cannot start failing allocations that have never failed before.

That said, maybe we want Alan's N_NULL failover in the hangup path too?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

             reply	other threads:[~2017-10-03 22:55 UTC|newest]

Thread overview: 44+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-10-03 22:55 Johannes Weiner [this message]
2017-10-03 22:55 ` tty crash due to auto-failing vmalloc Johannes Weiner
2017-10-03 23:51 ` Alan Cox
2017-10-03 23:51   ` Alan Cox
2017-10-04  8:33 ` Michal Hocko
2017-10-04  8:33   ` Michal Hocko
2017-10-04 18:58 ` Johannes Weiner
2017-10-04 18:58   ` Johannes Weiner
2017-10-04 18:59   ` [PATCH 1/2] Revert "vmalloc: back off when the current task is killed" Johannes Weiner
2017-10-04 18:59     ` Johannes Weiner
2017-10-04 20:49     ` Tetsuo Handa
2017-10-04 20:49       ` Tetsuo Handa
2017-10-04 21:00       ` Johannes Weiner
2017-10-04 21:00         ` Johannes Weiner
2017-10-04 21:42         ` Tetsuo Handa
2017-10-04 21:42           ` Tetsuo Handa
2017-10-04 23:21           ` Johannes Weiner
2017-10-04 23:21             ` Johannes Weiner
2017-10-04 22:32     ` Andrew Morton
2017-10-04 22:32       ` Andrew Morton
2017-10-04 23:18       ` Johannes Weiner
2017-10-04 23:18         ` Johannes Weiner
2017-10-05  7:57         ` Michal Hocko
2017-10-05  7:57           ` Michal Hocko
2017-10-05 10:36           ` Tetsuo Handa
2017-10-05 10:36             ` Tetsuo Handa
2017-10-05 10:49             ` Michal Hocko
2017-10-05 10:49               ` Michal Hocko
2017-10-07  2:21             ` Tetsuo Handa
2017-10-07  2:21               ` Tetsuo Handa
2017-10-07  2:51               ` Johannes Weiner
2017-10-07  2:51                 ` Johannes Weiner
2017-10-07  4:05                 ` Tetsuo Handa
2017-10-07  4:05                   ` Tetsuo Handa
2017-10-07  7:59                   ` Michal Hocko
2017-10-07  7:59                     ` Michal Hocko
2017-10-07  9:57                     ` Tetsuo Handa
2017-10-07  9:57                       ` Tetsuo Handa
2017-10-05  6:49     ` Vlastimil Babka
2017-10-05  6:49       ` Vlastimil Babka
2017-10-05  7:54     ` Michal Hocko
2017-10-05  7:54       ` Michal Hocko
2017-10-04 18:59   ` [PATCH 2/2] tty: fall back to N_NULL if switching to N_TTY fails during hangup Johannes Weiner
2017-10-04 18:59     ` Johannes Weiner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20171003225504.GA966@cmpxchg.org \
    --to=hannes@cmpxchg.org \
    --cc=akpm@linux-foundation.org \
    --cc=alan@llwyncelyn.cymru \
    --cc=hch@lst.de \
    --cc=kernel-team@fb.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mhocko@suse.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.