From: Josh Poimboeuf <jpoimboe@redhat.com>
To: Johannes Erdfelt <johannes@erdfelt.com>
Cc: Joe Lawrence <joe.lawrence@redhat.com>,
Jessica Yu <jeyu@kernel.org>, Jiri Kosina <jikos@kernel.org>,
Miroslav Benes <mbenes@suse.cz>,
Steven Rostedt <rostedt@goodmis.org>,
Ingo Molnar <mingo@redhat.com>,
live-patching@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: Oops caused by race between livepatch and ftrace
Date: Mon, 20 May 2019 16:19:31 -0500 [thread overview]
Message-ID: <20190520211931.vokbqxkx5kb6k2bz@treble> (raw)
In-Reply-To: <20190520210905.GC1646@sventech.com>
On Mon, May 20, 2019 at 02:09:05PM -0700, Johannes Erdfelt wrote:
> On Mon, May 20, 2019, Joe Lawrence <joe.lawrence@redhat.com> wrote:
> > [ fixed jeyu's email address ]
>
> Thank you, the bounce message made it seem like my mail server was
> blocked and not that the address didn't exist.
>
> I think MAINTAINERS needs an update since it still has the @redhat.com
> address.
I think you must have been looking at an old version.
[(v5.2-rc1)] ~/git/linux $ grep jeyu MAINTAINERS
M: Jessica Yu <jeyu@kernel.org>
> > On 5/20/19 3:49 PM, Johannes Erdfelt wrote:
> > > [ ... snip ... ]
> > >
> > > I have put together a test case that can reproduce the crash using
> > > KVM. The tarball includes a minimal kernel and initramfs, along with
> > > a script to run qemu and the .config used to build the kernel. By
> > > default it will attempt to reproduce by loading multiple livepatches
> > > at the same time. Passing 'test=ftrace' to the script will attempt to
> > > reproduce by racing with ftrace.
> > >
> > > My test setup reproduces the race and oops more reliably by loading
> > > multiple livepatches at the same time than with the ftrace method. It's
> > > not 100% reproducible, so the test case may need to be run multiple
> > > times.
> > >
> > > It can be found here (not attached because of its size):
> > > http://johannes.erdfelt.com/5.2.0-rc1-a188339ca5-livepatch-race.tar.gz
> >
> > Hi Johannes,
> >
> > This is cool way to distribute the repro kernel, modules, etc!
>
> This oops was common in our production environment and was particularly
> annoying since livepatches would load at boot and early enough to happen
> before networking and SSH were started.
>
> Unfortunately it was difficult to reproduce on other hardware (changing
> the timing just enough) and our production environment is very
> complicated.
>
> I spent more time than I'd like to admit trying to reproduce this fairly
> reliably. I knew that I needed to help make it as easy as possible to
> reproduce to root cause it and for others to take a look at it as well.
Can you try this patch (completely untested)?
diff --git a/kernel/livepatch/core.c b/kernel/livepatch/core.c
index 91cd519756d3..2d17e6e364b5 100644
--- a/kernel/livepatch/core.c
+++ b/kernel/livepatch/core.c
@@ -30,6 +30,7 @@
#include <linux/elf.h>
#include <linux/moduleloader.h>
#include <linux/completion.h>
+#include <linux/memory.h>
#include <asm/cacheflush.h>
#include "core.h"
#include "patch.h"
@@ -730,16 +731,21 @@ static int klp_init_object_loaded(struct klp_patch *patch,
struct klp_func *func;
int ret;
+ mutex_lock(&text_mutex);
+
module_disable_ro(patch->mod);
ret = klp_write_object_relocations(patch->mod, obj);
if (ret) {
module_enable_ro(patch->mod, true);
+ mutex_unlock(&text_mutex);
return ret;
}
arch_klp_init_object_loaded(patch, obj);
module_enable_ro(patch->mod, true);
+ mutex_unlock(&text_mutex);
+
klp_for_each_func(obj, func) {
ret = klp_find_object_symbol(obj->name, func->old_name,
func->old_sympos,
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index a12aff849c04..8259d4ba8b00 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -34,6 +34,7 @@
#include <linux/hash.h>
#include <linux/rcupdate.h>
#include <linux/kprobes.h>
+#include <linux/memory.h>
#include <trace/events/sched.h>
@@ -2610,10 +2611,12 @@ static void ftrace_run_update_code(int command)
{
int ret;
+ mutex_lock(&text_mutex);
+
ret = ftrace_arch_code_modify_prepare();
FTRACE_WARN_ON(ret);
if (ret)
- return;
+ goto out_unlock;
/*
* By default we use stop_machine() to modify the code.
@@ -2625,6 +2628,9 @@ static void ftrace_run_update_code(int command)
ret = ftrace_arch_code_modify_post_process();
FTRACE_WARN_ON(ret);
+
+out_unlock:
+ mutex_unlock(&text_mutex);
}
static void ftrace_run_modify_code(struct ftrace_ops *ops, int command,
@@ -5776,6 +5782,7 @@ void ftrace_module_enable(struct module *mod)
struct ftrace_page *pg;
mutex_lock(&ftrace_lock);
+ mutex_lock(&text_mutex);
if (ftrace_disabled)
goto out_unlock;
@@ -5837,6 +5844,7 @@ void ftrace_module_enable(struct module *mod)
ftrace_arch_code_modify_post_process();
out_unlock:
+ mutex_unlock(&text_mutex);
mutex_unlock(&ftrace_lock);
process_cached_mods(mod->name);
next prev parent reply other threads:[~2019-05-20 21:19 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-20 19:49 Oops caused by race between livepatch and ftrace Johannes Erdfelt
2019-05-20 20:46 ` Joe Lawrence
2019-05-20 21:09 ` Johannes Erdfelt
2019-05-20 21:19 ` Josh Poimboeuf [this message]
2019-05-20 21:39 ` Steven Rostedt
2019-05-21 14:16 ` Josh Poimboeuf
2019-05-21 14:42 ` Steven Rostedt
2019-05-21 16:42 ` Josh Poimboeuf
2019-05-21 16:53 ` Steven Rostedt
2019-05-29 11:17 ` Jiri Kosina
2019-05-29 12:06 ` Steven Rostedt
2019-05-29 12:30 ` Josh Poimboeuf
2019-05-22 13:00 ` Josh Poimboeuf
2019-05-29 17:29 ` Jessica Yu
2019-05-29 17:39 ` Josh Poimboeuf
2019-05-29 18:34 ` Jessica Yu
2019-05-20 21:48 ` Johannes Erdfelt
2019-05-20 21:19 ` Joe Lawrence
2019-05-21 19:27 ` Joe Lawrence
2019-05-21 21:00 ` Josh Poimboeuf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190520211931.vokbqxkx5kb6k2bz@treble \
--to=jpoimboe@redhat.com \
--cc=jeyu@kernel.org \
--cc=jikos@kernel.org \
--cc=joe.lawrence@redhat.com \
--cc=johannes@erdfelt.com \
--cc=linux-kernel@vger.kernel.org \
--cc=live-patching@vger.kernel.org \
--cc=mbenes@suse.cz \
--cc=mingo@redhat.com \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).