From: Johannes Weiner <hannes@cmpxchg.org> To: Michal Hocko <mhocko@suse.com> Cc: Alan Cox <alan@llwyncelyn.cymru>, Christoph Hellwig <hch@lst.de>, Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: tty crash due to auto-failing vmalloc Date: Tue, 3 Oct 2017 18:55:04 -0400 [thread overview] Message-ID: <20171003225504.GA966@cmpxchg.org> (raw) On some of our machines, we see this warning: /* switch the line discipline */ tty->ldisc = ld; tty_set_termios_ldisc(tty, disc); retval = tty_ldisc_open(tty, tty->ldisc); if (retval) { -> if (!WARN_ON(disc == N_TTY)) { tty_ldisc_put(tty->ldisc); tty->ldisc = NULL; } } where the stack is tty_ldisc_reinit tty_ldisc_hangup __tty_hangup do_exit do_signal syscall This is followed by a NULL pointer deref crash in n_tty_set_termios, presumably when it tries to deref that unallocated tty->disc_data. The only way n_tty_open() can fail is if the vmalloc in there fails. struct n_tty_data isn't terribly big, but ever since the following patch it doesn't even *try* the allocation: commit 5d17a73a2ebeb8d1c6924b91e53ab2650fe86ffb Author: Michal Hocko <mhocko@suse.com> Date: Fri Feb 24 14:58:53 2017 -0800 vmalloc: back off when the current task is killed __vmalloc_area_node() allocates pages to cover the requested vmalloc size. This can be a lot of memory. If the current task is killed by the OOM killer, and thus has an unlimited access to memory reserves, it can consume all the memory theoretically. Fix this by checking for fatal_signal_pending and back off early. Link: http://lkml.kernel.org/r/20170201092706.9966-4-mhocko@kernel.org Signed-off-by: Michal Hocko <mhocko@suse.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Cc: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> Cc: Al Viro <viro@zeniv.linux.org.uk> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> This talks about the oom killer and memory exhaustion, but most fatal signals don't happen due to the OOM killer. I think this patch should be reverted. If somebody is vmallocing crazy amounts of memory in the exit path we should probably track them down individually; the patch doesn't reference any real instances of that. But we cannot start failing allocations that have never failed before. That said, maybe we want Alan's N_NULL failover in the hangup path too?
WARNING: multiple messages have this Message-ID (diff)
From: Johannes Weiner <hannes@cmpxchg.org> To: Michal Hocko <mhocko@suse.com> Cc: Alan Cox <alan@llwyncelyn.cymru>, Christoph Hellwig <hch@lst.de>, Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: tty crash due to auto-failing vmalloc Date: Tue, 3 Oct 2017 18:55:04 -0400 [thread overview] Message-ID: <20171003225504.GA966@cmpxchg.org> (raw) On some of our machines, we see this warning: /* switch the line discipline */ tty->ldisc = ld; tty_set_termios_ldisc(tty, disc); retval = tty_ldisc_open(tty, tty->ldisc); if (retval) { -> if (!WARN_ON(disc == N_TTY)) { tty_ldisc_put(tty->ldisc); tty->ldisc = NULL; } } where the stack is tty_ldisc_reinit tty_ldisc_hangup __tty_hangup do_exit do_signal syscall This is followed by a NULL pointer deref crash in n_tty_set_termios, presumably when it tries to deref that unallocated tty->disc_data. The only way n_tty_open() can fail is if the vmalloc in there fails. struct n_tty_data isn't terribly big, but ever since the following patch it doesn't even *try* the allocation: commit 5d17a73a2ebeb8d1c6924b91e53ab2650fe86ffb Author: Michal Hocko <mhocko@suse.com> Date: Fri Feb 24 14:58:53 2017 -0800 vmalloc: back off when the current task is killed __vmalloc_area_node() allocates pages to cover the requested vmalloc size. This can be a lot of memory. If the current task is killed by the OOM killer, and thus has an unlimited access to memory reserves, it can consume all the memory theoretically. Fix this by checking for fatal_signal_pending and back off early. Link: http://lkml.kernel.org/r/20170201092706.9966-4-mhocko@kernel.org Signed-off-by: Michal Hocko <mhocko@suse.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Cc: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> Cc: Al Viro <viro@zeniv.linux.org.uk> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> This talks about the oom killer and memory exhaustion, but most fatal signals don't happen due to the OOM killer. I think this patch should be reverted. If somebody is vmallocing crazy amounts of memory in the exit path we should probably track them down individually; the patch doesn't reference any real instances of that. But we cannot start failing allocations that have never failed before. That said, maybe we want Alan's N_NULL failover in the hangup path too? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next reply other threads:[~2017-10-03 22:55 UTC|newest] Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-10-03 22:55 Johannes Weiner [this message] 2017-10-03 22:55 ` tty crash due to auto-failing vmalloc Johannes Weiner 2017-10-03 23:51 ` Alan Cox 2017-10-03 23:51 ` Alan Cox 2017-10-04 8:33 ` Michal Hocko 2017-10-04 8:33 ` Michal Hocko 2017-10-04 18:58 ` Johannes Weiner 2017-10-04 18:58 ` Johannes Weiner 2017-10-04 18:59 ` [PATCH 1/2] Revert "vmalloc: back off when the current task is killed" Johannes Weiner 2017-10-04 18:59 ` Johannes Weiner 2017-10-04 20:49 ` Tetsuo Handa 2017-10-04 20:49 ` Tetsuo Handa 2017-10-04 21:00 ` Johannes Weiner 2017-10-04 21:00 ` Johannes Weiner 2017-10-04 21:42 ` Tetsuo Handa 2017-10-04 21:42 ` Tetsuo Handa 2017-10-04 23:21 ` Johannes Weiner 2017-10-04 23:21 ` Johannes Weiner 2017-10-04 22:32 ` Andrew Morton 2017-10-04 22:32 ` Andrew Morton 2017-10-04 23:18 ` Johannes Weiner 2017-10-04 23:18 ` Johannes Weiner 2017-10-05 7:57 ` Michal Hocko 2017-10-05 7:57 ` Michal Hocko 2017-10-05 10:36 ` Tetsuo Handa 2017-10-05 10:36 ` Tetsuo Handa 2017-10-05 10:49 ` Michal Hocko 2017-10-05 10:49 ` Michal Hocko 2017-10-07 2:21 ` Tetsuo Handa 2017-10-07 2:21 ` Tetsuo Handa 2017-10-07 2:51 ` Johannes Weiner 2017-10-07 2:51 ` Johannes Weiner 2017-10-07 4:05 ` Tetsuo Handa 2017-10-07 4:05 ` Tetsuo Handa 2017-10-07 7:59 ` Michal Hocko 2017-10-07 7:59 ` Michal Hocko 2017-10-07 9:57 ` Tetsuo Handa 2017-10-07 9:57 ` Tetsuo Handa 2017-10-05 6:49 ` Vlastimil Babka 2017-10-05 6:49 ` Vlastimil Babka 2017-10-05 7:54 ` Michal Hocko 2017-10-05 7:54 ` Michal Hocko 2017-10-04 18:59 ` [PATCH 2/2] tty: fall back to N_NULL if switching to N_TTY fails during hangup Johannes Weiner 2017-10-04 18:59 ` Johannes Weiner
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20171003225504.GA966@cmpxchg.org \ --to=hannes@cmpxchg.org \ --cc=akpm@linux-foundation.org \ --cc=alan@llwyncelyn.cymru \ --cc=hch@lst.de \ --cc=kernel-team@fb.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mhocko@suse.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.