Re: Oops caused by race between livepatch and ftrace

From: Steven Rostedt <rostedt@goodmis.org>
To: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Johannes Erdfelt <johannes@erdfelt.com>,
	Joe Lawrence <joe.lawrence@redhat.com>,
	Jessica Yu <jeyu@kernel.org>, Jiri Kosina <jikos@kernel.org>,
	Miroslav Benes <mbenes@suse.cz>, Ingo Molnar <mingo@redhat.com>,
	live-patching@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: Oops caused by race between livepatch and ftrace
Date: Mon, 20 May 2019 17:39:10 -0400	[thread overview]
Message-ID: <20190520173910.6da9ddaf@gandalf.local.home> (raw)
In-Reply-To: <20190520211931.vokbqxkx5kb6k2bz@treble>

On Mon, 20 May 2019 16:19:31 -0500
Josh Poimboeuf <jpoimboe@redhat.com> wrote:

> diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
> index a12aff849c04..8259d4ba8b00 100644
> --- a/kernel/trace/ftrace.c
> +++ b/kernel/trace/ftrace.c
> @@ -34,6 +34,7 @@
>  #include <linux/hash.h>
>  #include <linux/rcupdate.h>
>  #include <linux/kprobes.h>
> +#include <linux/memory.h>
>  
>  #include <trace/events/sched.h>
>  
> @@ -2610,10 +2611,12 @@ static void ftrace_run_update_code(int command)
>  {
>  	int ret;
>  
> +	mutex_lock(&text_mutex);
> +

Hmm, this may blow up with lockdep, as I believe we already have a
locking dependency of:

 text_mutex -> ftrace_lock

And this will reverses it. (kprobes appears to take the locks in this
order).

Perhaps have live kernel patching grab ftrace_lock?

-- Steve

>  	ret = ftrace_arch_code_modify_prepare();
>  	FTRACE_WARN_ON(ret);
>  	if (ret)
> -		return;
> +		goto out_unlock;
>  
>  	/*
>  	 * By default we use stop_machine() to modify the code.
> @@ -2625,6 +2628,9 @@ static void ftrace_run_update_code(int command)
>  
>  	ret = ftrace_arch_code_modify_post_process();
>  	FTRACE_WARN_ON(ret);
> +
> +out_unlock:
> +	mutex_unlock(&text_mutex);
>  }
>  
>  static void ftrace_run_modify_code(struct ftrace_ops *ops, int command,
> @@ -5776,6 +5782,7 @@ void ftrace_module_enable(struct module *mod)
>  	struct ftrace_page *pg;
>  
>  	mutex_lock(&ftrace_lock);
> +	mutex_lock(&text_mutex);
>  
>  	if (ftrace_disabled)
>  		goto out_unlock;
> @@ -5837,6 +5844,7 @@ void ftrace_module_enable(struct module *mod)
>  		ftrace_arch_code_modify_post_process();
>  
>   out_unlock:
> +	mutex_unlock(&text_mutex);
>  	mutex_unlock(&ftrace_lock);
>  
>  	process_cached_mods(mod->name);