From: Kees Cook <keescook@chromium.org>
To: Jann Horn <jannh@google.com>
Cc: Kees Cook <keescook@chromium.org>,
Greg KH <gregkh@linuxfoundation.org>,
Linus Torvalds <torvalds@linuxfoundation.org>,
Seth Jenkins <sethjenkins@google.com>,
Andy Lutomirski <luto@kernel.org>, Petr Mladek <pmladek@suse.com>,
Andrew Morton <akpm@linux-foundation.org>,
tangmeng <tangmeng@uniontech.com>,
"Guilherme G. Piccoli" <gpiccoli@igalia.com>,
Tiezhu Yang <yangtiezhu@loongson.cn>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
"Eric W. Biederman" <ebiederm@xmission.com>,
Arnd Bergmann <arnd@arndb.de>, Marco Elver <elver@google.com>,
Dmitry Vyukov <dvyukov@google.com>,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Valentin Schneider <vschneid@redhat.com>,
Andrey Ryabinin <ryabinin.a.a@gmail.com>,
Alexander Potapenko <glider@google.com>,
Andrey Konovalov <andreyknvl@gmail.com>,
Vincenzo Frascino <vincenzo.frascino@arm.com>,
Luis Chamberlain <mcgrof@kernel.org>,
David Gow <davidgow@google.com>,
"Paul E. McKenney" <paulmck@kernel.org>,
Jonathan Corbet <corbet@lwn.net>,
Baolin Wang <baolin.wang@linux.alibaba.com>,
"Jason A. Donenfeld" <Jason@zx2c4.com>,
Eric Biggers <ebiggers@google.com>,
Huang Ying <ying.huang@intel.com>,
Anton Vorontsov <anton@enomsg.org>,
Mauro Carvalho Chehab <mchehab+huawei@kernel.org>,
Laurent Dufour <ldufour@linux.ibm.com>,
Rob Herring <robh@kernel.org>,
linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com,
linux-mm@kvack.org, linux-doc@vger.kernel.org,
linux-hardening@vger.kernel.org
Subject: [PATCH v2 2/6] exit: Put an upper limit on how often we can oops
Date: Wed, 9 Nov 2022 12:00:45 -0800 [thread overview]
Message-ID: <20221109200050.3400857-2-keescook@chromium.org> (raw)
In-Reply-To: <20221109194404.gonna.558-kees@kernel.org>
From: Jann Horn <jannh@google.com>
Many Linux systems are configured to not panic on oops; but allowing an
attacker to oops the system **really** often can make even bugs that look
completely unexploitable exploitable (like NULL dereferences and such) if
each crash elevates a refcount by one or a lock is taken in read mode, and
this causes a counter to eventually overflow.
The most interesting counters for this are 32 bits wide (like open-coded
refcounts that don't use refcount_t). (The ldsem reader count on 32-bit
platforms is just 16 bits, but probably nobody cares about 32-bit platforms
that much nowadays.)
So let's panic the system if the kernel is constantly oopsing.
The speed of oopsing 2^32 times probably depends on several factors, like
how long the stack trace is and which unwinder you're using; an empirically
important one is whether your console is showing a graphical environment or
a text console that oopses will be printed to.
In a quick single-threaded benchmark, it looks like oopsing in a vfork()
child with a very short stack trace only takes ~510 microseconds per run
when a graphical console is active; but switching to a text console that
oopses are printed to slows it down around 87x, to ~45 milliseconds per
run.
(Adding more threads makes this faster, but the actual oops printing
happens under &die_lock on x86, so you can maybe speed this up by a factor
of around 2 and then any further improvement gets eaten up by lock
contention.)
It looks like it would take around 8-12 days to overflow a 32-bit counter
with repeated oopsing on a multi-core X86 system running a graphical
environment; both me (in an X86 VM) and Seth (with a distro kernel on
normal hardware in a standard configuration) got numbers in that ballpark.
12 days aren't *that* short on a desktop system, and you'd likely need much
longer on a typical server system (assuming that people don't run graphical
desktop environments on their servers), and this is a *very* noisy and
violent approach to exploiting the kernel; and it also seems to take orders
of magnitude longer on some machines, probably because stuff like EFI
pstore will slow it down a ton if that's active.
[Moved sysctl into kernel/exit.c -kees]
Signed-off-by: Jann Horn <jannh@google.com>
Signed-off-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20221107201317.324457-1-jannh@google.com
---
Documentation/admin-guide/sysctl/kernel.rst | 8 ++++
kernel/exit.c | 42 +++++++++++++++++++++
2 files changed, 50 insertions(+)
diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index 98d1b198b2b4..09f3fb2f8585 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -667,6 +667,14 @@ This is the default behavior.
an oops event is detected.
+oops_limit
+==========
+
+Number of kernel oopses after which the kernel should panic when
+``panic_on_oops`` is not set. Setting this to 0 or 1 has the same effect
+as setting ``panic_on_oops=1``.
+
+
osrelease, ostype & version
===========================
diff --git a/kernel/exit.c b/kernel/exit.c
index 35e0a31a0315..892f38aeb0a4 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -72,6 +72,33 @@
#include <asm/unistd.h>
#include <asm/mmu_context.h>
+/*
+ * The default value should be high enough to not crash a system that randomly
+ * crashes its kernel from time to time, but low enough to at least not permit
+ * overflowing 32-bit refcounts or the ldsem writer count.
+ */
+static unsigned int oops_limit = 10000;
+
+#if CONFIG_SYSCTL
+static struct ctl_table kern_exit_table[] = {
+ {
+ .procname = "oops_limit",
+ .data = &oops_limit,
+ .maxlen = sizeof(oops_limit),
+ .mode = 0644,
+ .proc_handler = proc_douintvec,
+ },
+ { }
+};
+
+static __init int kernel_exit_sysctls_init(void)
+{
+ register_sysctl_init("kernel", kern_exit_table);
+ return 0;
+}
+late_initcall(kernel_exit_sysctls_init);
+#endif
+
static void __unhash_process(struct task_struct *p, bool group_dead)
{
nr_threads--;
@@ -874,6 +901,8 @@ void __noreturn do_exit(long code)
void __noreturn make_task_dead(int signr)
{
+ static atomic_t oops_count = ATOMIC_INIT(0);
+
/*
* Take the task off the cpu after something catastrophic has
* happened.
@@ -897,6 +926,19 @@ void __noreturn make_task_dead(int signr)
preempt_count_set(PREEMPT_ENABLED);
}
+ /*
+ * Every time the system oopses, if the oops happens while a reference
+ * to an object was held, the reference leaks.
+ * If the oops doesn't also leak memory, repeated oopsing can cause
+ * reference counters to wrap around (if they're not using refcount_t).
+ * This means that repeated oopsing can make unexploitable-looking bugs
+ * exploitable through repeated oopsing.
+ * To make sure this can't happen, place an upper bound on how often the
+ * kernel may oops without panic().
+ */
+ if (atomic_inc_return(&oops_count) >= READ_ONCE(oops_limit))
+ panic("Oopsed too often (oops_limit is %d)", oops_limit);
+
/*
* We're taking recursive faults here in make_task_dead. Safest is to just
* leave this task alone and wait for reboot.
--
2.34.1
next prev parent reply other threads:[~2022-11-09 20:01 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-09 20:00 [PATCH next v2 0/6] exit: Put an upper limit on how often we can oops Kees Cook
2022-11-09 20:00 ` [PATCH v2 1/6] panic: Separate sysctl logic from CONFIG_SMP Kees Cook
2022-11-09 21:48 ` Bill Wendling
2022-11-10 0:02 ` Kees Cook
2022-11-09 20:00 ` Kees Cook [this message]
2022-11-09 20:00 ` [PATCH v2 3/6] exit: Expose "oops_count" to sysfs Kees Cook
2022-11-09 20:00 ` [PATCH v2 4/6] panic: Consolidate open-coded panic_on_warn checks Kees Cook
2022-11-14 9:57 ` Marco Elver
2022-11-17 23:28 ` Kees Cook
2022-11-09 20:00 ` [PATCH v2 5/6] panic: Introduce warn_limit Kees Cook
2022-11-14 9:48 ` Marco Elver
2022-11-17 23:27 ` Kees Cook
2022-11-09 20:00 ` [PATCH v2 6/6] panic: Expose "warn_count" to sysfs Kees Cook
2022-11-09 21:16 ` [PATCH next v2 0/6] exit: Put an upper limit on how often we can oops Luis Chamberlain
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221109200050.3400857-2-keescook@chromium.org \
--to=keescook@chromium.org \
--cc=Jason@zx2c4.com \
--cc=akpm@linux-foundation.org \
--cc=andreyknvl@gmail.com \
--cc=anton@enomsg.org \
--cc=arnd@arndb.de \
--cc=baolin.wang@linux.alibaba.com \
--cc=bigeasy@linutronix.de \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=corbet@lwn.net \
--cc=davidgow@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=dvyukov@google.com \
--cc=ebiederm@xmission.com \
--cc=ebiggers@google.com \
--cc=elver@google.com \
--cc=glider@google.com \
--cc=gpiccoli@igalia.com \
--cc=gregkh@linuxfoundation.org \
--cc=jannh@google.com \
--cc=juri.lelli@redhat.com \
--cc=kasan-dev@googlegroups.com \
--cc=ldufour@linux.ibm.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-hardening@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=luto@kernel.org \
--cc=mcgrof@kernel.org \
--cc=mchehab+huawei@kernel.org \
--cc=mgorman@suse.de \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=pmladek@suse.com \
--cc=robh@kernel.org \
--cc=rostedt@goodmis.org \
--cc=ryabinin.a.a@gmail.com \
--cc=sethjenkins@google.com \
--cc=tangmeng@uniontech.com \
--cc=torvalds@linuxfoundation.org \
--cc=vincent.guittot@linaro.org \
--cc=vincenzo.frascino@arm.com \
--cc=vschneid@redhat.com \
--cc=yangtiezhu@loongson.cn \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).