stable.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Eric Biggers <ebiggers@kernel.org>
To: stable@vger.kernel.org, Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Kees Cook <keescook@chromium.org>, SeongJae Park <sj@kernel.org>,
	Seth Jenkins <sethjenkins@google.com>,
	Jann Horn <jannh@google.com>,
	"Eric W . Biederman" <ebiederm@xmission.com>,
	linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org,
	Luis Chamberlain <mcgrof@kernel.org>
Subject: [PATCH 5.15 13/20] exit: Put an upper limit on how often we can oops
Date: Tue, 24 Jan 2023 10:51:03 -0800	[thread overview]
Message-ID: <20230124185110.143857-14-ebiggers@kernel.org> (raw)
In-Reply-To: <20230124185110.143857-1-ebiggers@kernel.org>

From: Jann Horn <jannh@google.com>

commit d4ccd54d28d3c8598e2354acc13e28c060961dbb upstream.

Many Linux systems are configured to not panic on oops; but allowing an
attacker to oops the system **really** often can make even bugs that look
completely unexploitable exploitable (like NULL dereferences and such) if
each crash elevates a refcount by one or a lock is taken in read mode, and
this causes a counter to eventually overflow.

The most interesting counters for this are 32 bits wide (like open-coded
refcounts that don't use refcount_t). (The ldsem reader count on 32-bit
platforms is just 16 bits, but probably nobody cares about 32-bit platforms
that much nowadays.)

So let's panic the system if the kernel is constantly oopsing.

The speed of oopsing 2^32 times probably depends on several factors, like
how long the stack trace is and which unwinder you're using; an empirically
important one is whether your console is showing a graphical environment or
a text console that oopses will be printed to.
In a quick single-threaded benchmark, it looks like oopsing in a vfork()
child with a very short stack trace only takes ~510 microseconds per run
when a graphical console is active; but switching to a text console that
oopses are printed to slows it down around 87x, to ~45 milliseconds per
run.
(Adding more threads makes this faster, but the actual oops printing
happens under &die_lock on x86, so you can maybe speed this up by a factor
of around 2 and then any further improvement gets eaten up by lock
contention.)

It looks like it would take around 8-12 days to overflow a 32-bit counter
with repeated oopsing on a multi-core X86 system running a graphical
environment; both me (in an X86 VM) and Seth (with a distro kernel on
normal hardware in a standard configuration) got numbers in that ballpark.

12 days aren't *that* short on a desktop system, and you'd likely need much
longer on a typical server system (assuming that people don't run graphical
desktop environments on their servers), and this is a *very* noisy and
violent approach to exploiting the kernel; and it also seems to take orders
of magnitude longer on some machines, probably because stuff like EFI
pstore will slow it down a ton if that's active.

Signed-off-by: Jann Horn <jannh@google.com>
Link: https://lore.kernel.org/r/20221107201317.324457-1-jannh@google.com
Reviewed-by: Luis Chamberlain <mcgrof@kernel.org>
Signed-off-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20221117234328.594699-2-keescook@chromium.org
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
 Documentation/admin-guide/sysctl/kernel.rst |  8 ++++
 kernel/exit.c                               | 43 +++++++++++++++++++++
 2 files changed, 51 insertions(+)

diff --git a/Documentation/admin-guide/sysctl/kernel.rst b/Documentation/admin-guide/sysctl/kernel.rst
index 609b891754081..b6e68d6f297e5 100644
--- a/Documentation/admin-guide/sysctl/kernel.rst
+++ b/Documentation/admin-guide/sysctl/kernel.rst
@@ -671,6 +671,14 @@ This is the default behavior.
 an oops event is detected.
 
 
+oops_limit
+==========
+
+Number of kernel oopses after which the kernel should panic when
+``panic_on_oops`` is not set. Setting this to 0 or 1 has the same effect
+as setting ``panic_on_oops=1``.
+
+
 osrelease, ostype & version
 ===========================
 
diff --git a/kernel/exit.c b/kernel/exit.c
index 5d1a507fd4bae..172d7f835f801 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -69,6 +69,33 @@
 #include <asm/unistd.h>
 #include <asm/mmu_context.h>
 
+/*
+ * The default value should be high enough to not crash a system that randomly
+ * crashes its kernel from time to time, but low enough to at least not permit
+ * overflowing 32-bit refcounts or the ldsem writer count.
+ */
+static unsigned int oops_limit = 10000;
+
+#ifdef CONFIG_SYSCTL
+static struct ctl_table kern_exit_table[] = {
+	{
+		.procname       = "oops_limit",
+		.data           = &oops_limit,
+		.maxlen         = sizeof(oops_limit),
+		.mode           = 0644,
+		.proc_handler   = proc_douintvec,
+	},
+	{ }
+};
+
+static __init int kernel_exit_sysctls_init(void)
+{
+	register_sysctl_init("kernel", kern_exit_table);
+	return 0;
+}
+late_initcall(kernel_exit_sysctls_init);
+#endif
+
 static void __unhash_process(struct task_struct *p, bool group_dead)
 {
 	nr_threads--;
@@ -879,10 +906,26 @@ EXPORT_SYMBOL_GPL(do_exit);
 
 void __noreturn make_task_dead(int signr)
 {
+	static atomic_t oops_count = ATOMIC_INIT(0);
+
 	/*
 	 * Take the task off the cpu after something catastrophic has
 	 * happened.
 	 */
+
+	/*
+	 * Every time the system oopses, if the oops happens while a reference
+	 * to an object was held, the reference leaks.
+	 * If the oops doesn't also leak memory, repeated oopsing can cause
+	 * reference counters to wrap around (if they're not using refcount_t).
+	 * This means that repeated oopsing can make unexploitable-looking bugs
+	 * exploitable through repeated oopsing.
+	 * To make sure this can't happen, place an upper bound on how often the
+	 * kernel may oops without panic().
+	 */
+	if (atomic_inc_return(&oops_count) >= READ_ONCE(oops_limit))
+		panic("Oopsed too often (kernel.oops_limit is %d)", oops_limit);
+
 	do_exit(signr);
 }
 
-- 
2.39.1


  parent reply	other threads:[~2023-01-24 18:52 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-01-24 18:50 [PATCH 5.15 00/20] Backport oops_limit to 5.15 Eric Biggers
2023-01-24 18:50 ` [PATCH 5.15 01/20] sysctl: add a new register_sysctl_init() interface Eric Biggers
2023-01-24 18:50 ` [PATCH 5.15 02/20] kernel/panic: move panic sysctls to its own file Eric Biggers
2023-01-24 18:50 ` [PATCH 5.15 03/20] panic: unset panic_on_warn inside panic() Eric Biggers
2023-01-24 18:50 ` [PATCH 5.15 04/20] ubsan: no need to unset panic_on_warn in ubsan_epilogue() Eric Biggers
2023-01-24 18:50 ` [PATCH 5.15 05/20] kasan: no need to unset panic_on_warn in end_report() Eric Biggers
2023-01-24 18:50 ` [PATCH 5.15 06/20] exit: Add and use make_task_dead Eric Biggers
2023-01-24 18:50 ` [PATCH 5.15 07/20] objtool: Add a missing comma to avoid string concatenation Eric Biggers
2023-01-24 18:50 ` [PATCH 5.15 08/20] hexagon: Fix function name in die() Eric Biggers
2023-01-24 18:50 ` [PATCH 5.15 09/20] h8300: Fix build errors from do_exit() to make_task_dead() transition Eric Biggers
2023-01-24 18:51 ` [PATCH 5.15 10/20] csky: Fix function name in csky_alignment() and die() Eric Biggers
2023-01-24 18:51 ` [PATCH 5.15 11/20] ia64: make IA64_MCA_RECOVERY bool instead of tristate Eric Biggers
2023-01-24 18:51 ` [PATCH 5.15 12/20] panic: Separate sysctl logic from CONFIG_SMP Eric Biggers
2023-01-24 18:51 ` Eric Biggers [this message]
2023-01-25 14:09   ` [PATCH 5.15 13/20] exit: Put an upper limit on how often we can oops Harshit Mogalapalli
2023-01-25 18:44     ` Eric Biggers
2023-01-25 18:59       ` Harshit Mogalapalli
2023-01-24 18:51 ` [PATCH 5.15 14/20] exit: Expose "oops_count" to sysfs Eric Biggers
2023-01-24 18:51 ` [PATCH 5.15 15/20] exit: Allow oops_limit to be disabled Eric Biggers
2023-01-24 18:51 ` [PATCH 5.15 16/20] panic: Consolidate open-coded panic_on_warn checks Eric Biggers
2023-01-24 18:51 ` [PATCH 5.15 17/20] panic: Introduce warn_limit Eric Biggers
2023-01-24 18:51 ` [PATCH 5.15 18/20] panic: Expose "warn_count" to sysfs Eric Biggers
2023-01-24 18:51 ` [PATCH 5.15 19/20] docs: Fix path paste-o for /sys/kernel/warn_count Eric Biggers
2023-01-24 18:51 ` [PATCH 5.15 20/20] exit: Use READ_ONCE() for all oops/warn limit reads Eric Biggers
2023-01-27  2:08 ` [PATCH 5.15 00/20] Backport oops_limit to 5.15 Sasha Levin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230124185110.143857-14-ebiggers@kernel.org \
    --to=ebiggers@kernel.org \
    --cc=ebiederm@xmission.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=jannh@google.com \
    --cc=keescook@chromium.org \
    --cc=linux-hardening@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mcgrof@kernel.org \
    --cc=sethjenkins@google.com \
    --cc=sj@kernel.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).