From: "Paul E. McKenney" <paulmck@kernel.org>
To: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
Peter Zijlstra <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>,
Uladzislau Rezki <urezki@gmail.com>,
LKML <linux-kernel@vger.kernel.org>, RCU <rcu@vger.kernel.org>,
Michael Ellerman <mpe@ellerman.id.au>,
Andrew Morton <akpm@linux-foundation.org>,
Daniel Axtens <dja@axtens.net>,
Frederic Weisbecker <frederic@kernel.org>,
Neeraj Upadhyay <neeraju@codeaurora.org>,
Joel Fernandes <joel@joelfernandes.org>,
Michal Hocko <mhocko@suse.com>,
"Theodore Y . Ts'o" <tytso@mit.edu>,
Oleksiy Avramchenko <oleksiy.avramchenko@sonymobile.com>,
rostedt@goodmis.org
Subject: Re: [PATCH 2/2] rcu-tasks: add RCU-tasks self tests
Date: Wed, 17 Feb 2021 10:17:38 -0800 [thread overview]
Message-ID: <20210217181738.GE2743@paulmck-ThinkPad-P72> (raw)
In-Reply-To: <20210217234759.b82e39049a2e99cf6358e1c2@kernel.org>
On Wed, Feb 17, 2021 at 11:47:59PM +0900, Masami Hiramatsu wrote:
> On Tue, 16 Feb 2021 09:30:03 -0800
> "Paul E. McKenney" <paulmck@kernel.org> wrote:
>
> > On Mon, Feb 15, 2021 at 12:28:26PM +0100, Sebastian Andrzej Siewior wrote:
> > > On 2021-02-13 08:45:54 [-0800], Paul E. McKenney wrote:
> > > > Glad you like it! But let's see which (if any) of these patches solves
> > > > the problem for Sebastian.
> > >
> > > Looking at that, is there any reason for doing this that can not be
> > > solved by moving the self-test a little later? Maybe once we reached at
> > > least SYSTEM_SCHEDULING?
> >
> > One problem is that ksoftirqd and the kprobes use are early_initcall(),
> > so we cannot count on ksoftirqd being spawned when kprobes first uses
> > synchronize_rcu_tasks(). Moving the selftest later won't fix this
> > problem, but rather just paper it over.
> >
> > > This happens now even before lockdep is up or the console is registered.
> > > So if something bad happens, you end up with a blank terminal.
> >
> > I was getting a splat, but I could easily believe that there are
> > configurations where the hang is totally silent. In other words, I do
> > agree that this needs a proper fix. All we need do is work out an
> > agreeable value of "proper". ;-)
> >
> > > There is nothing else that early in the boot process that requires
> > > working softirq. The only exception to this is wait_task_inactive()
> > > which is used while starting a new thread (including the ksoftirqd)
> > > which is why it was moved to schedule_hrtimeout().
> >
> > Moving kprobes initialization to early_initcall() [1] means that there
> > can be a call to synchronize_rcu_tasks() before the current spawning of
> > ksoftirqd. Because synchronize_rcu_tasks() needs timers to work, it needs
> > softirq to work. I know two straightforward ways to make that happen:
> >
> > 1. Spawn ksoftirqd earlier.
> >
> > 2. Suppress attempts to awaken ksoftirqd before it exists,
> > forcing all ksoftirq execution on the back of interrupts.
> >
> > Uladzislau and I each produced patches for #1, and I produced a patch
> > for #2.
> >
> > The only other option I know of is to push the call to init_kprobes()
> > later in the boot sequence, perhaps to its original subsys_initcall(),
> > or maybe only as late as core_initcall(). I added Masami and Steve on
> > CC for their thoughts on this.
> >
> > Is there some other proper fix that I am missing?
>
> Oh, I missed that the synchronize_rcu_tasks() will be involved the kprobes
> in early stage. Does the problem only exist in the synchronize_rcu_tasks()
> instead of synchronize_rcu()? If so I can just stop optimizer in early stage
> because I just want to enable kprobes in early stage, but not optprobes.
>
> Does the following patch help?
It does look to me like it would! I clearly should have asked you about
this a couple of months ago. ;-)
The proof of the pudding would be whether the powerpc guys can apply
this to v5.10-rc7 and have their kernel come up without hanging at boot.
Thanx, Paul
> >From e5fafcda3ff918cd52619f795a3f22fb95c72b11 Mon Sep 17 00:00:00 2001
> From: Masami Hiramatsu <mhiramat@kernel.org>
> Date: Wed, 17 Feb 2021 23:35:20 +0900
> Subject: [PATCH] kprobes: Fix to delay the kprobes jump optimization
>
> Since the kprobes jump optimization involves synchronize_rcu_tasks()
> which depends on the ksoftirqd, that can not be enabled at the
> early_initcall() boot stage. So this makes the kprobe optimization
> disabled in the early_initcall() and enables it in subsys_initcall().
>
> Note that non-optimized kprobes is still available after
> early_initcall(). Only jump optimization is delayed.
>
> Fixes: 36dadef23fcc ("kprobes: Init kprobes in early_initcall")
> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
> ---
> kernel/kprobes.c | 31 +++++++++++++++++++++----------
> 1 file changed, 21 insertions(+), 10 deletions(-)
>
> diff --git a/kernel/kprobes.c b/kernel/kprobes.c
> index d5a3eb74a657..779d8322e307 100644
> --- a/kernel/kprobes.c
> +++ b/kernel/kprobes.c
> @@ -861,7 +861,6 @@ static void try_to_optimize_kprobe(struct kprobe *p)
> cpus_read_unlock();
> }
>
> -#ifdef CONFIG_SYSCTL
> static void optimize_all_kprobes(void)
> {
> struct hlist_head *head;
> @@ -887,6 +886,7 @@ static void optimize_all_kprobes(void)
> mutex_unlock(&kprobe_mutex);
> }
>
> +#ifdef CONFIG_SYSCTL
> static void unoptimize_all_kprobes(void)
> {
> struct hlist_head *head;
> @@ -2497,18 +2497,14 @@ static int __init init_kprobes(void)
> }
> }
>
> -#if defined(CONFIG_OPTPROBES)
> -#if defined(__ARCH_WANT_KPROBES_INSN_SLOT)
> - /* Init kprobe_optinsn_slots */
> - kprobe_optinsn_slots.insn_size = MAX_OPTINSN_SIZE;
> -#endif
> - /* By default, kprobes can be optimized */
> - kprobes_allow_optimization = true;
> -#endif
> -
> /* By default, kprobes are armed */
> kprobes_all_disarmed = false;
>
> +#if defined(CONFIG_OPTPROBES) && defined(__ARCH_WANT_KPROBES_INSN_SLOT)
> + /* Init kprobe_optinsn_slots for allocation */
> + kprobe_optinsn_slots.insn_size = MAX_OPTINSN_SIZE;
> +#endif
> +
> err = arch_init_kprobes();
> if (!err)
> err = register_die_notifier(&kprobe_exceptions_nb);
> @@ -2523,6 +2519,21 @@ static int __init init_kprobes(void)
> }
> early_initcall(init_kprobes);
>
> +#if defined(CONFIG_OPTPROBES)
> +static int __init init_optprobes(void)
> +{
> + /*
> + * Enable kprobe optimization - this kicks the optimizer which
> + * depends on synchronize_rcu_tasks() and ksoftirqd, that is
> + * not spawned in early initcall. So delay the optimization.
> + */
> + optimize_all_kprobes();
> +
> + return 0;
> +}
> +subsys_initcall(init_optprobes);
> +#endif
> +
> #ifdef CONFIG_DEBUG_FS
> static void report_probe(struct seq_file *pi, struct kprobe *p,
> const char *sym, int offset, char *modname, struct kprobe *pp)
> --
> 2.25.1
>
>
> --
> Masami Hiramatsu <mhiramat@kernel.org>
next prev parent reply other threads:[~2021-02-17 18:18 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-09 20:27 [PATCH 1/2] rcu-tasks: move RCU-tasks initialization out of core_initcall() Uladzislau Rezki (Sony)
2020-12-09 20:27 ` [PATCH 2/2] rcu-tasks: add RCU-tasks self tests Uladzislau Rezki (Sony)
2020-12-16 15:49 ` Uladzislau Rezki
2020-12-16 23:29 ` Paul E. McKenney
2020-12-21 15:38 ` Uladzislau Rezki
2020-12-21 17:18 ` Paul E. McKenney
2020-12-21 18:45 ` Uladzislau Rezki
2020-12-21 19:29 ` Paul E. McKenney
2020-12-21 19:48 ` Uladzislau Rezki
2020-12-21 20:45 ` Paul E. McKenney
2020-12-21 21:28 ` Uladzislau Rezki
2021-02-12 19:20 ` Sebastian Andrzej Siewior
2021-02-12 21:12 ` Uladzislau Rezki
2021-02-12 23:48 ` Paul E. McKenney
2021-02-13 0:37 ` Paul E. McKenney
2021-02-13 0:43 ` Paul E. McKenney
2021-02-13 11:30 ` Uladzislau Rezki
2021-02-13 16:45 ` Paul E. McKenney
2021-02-13 20:00 ` Uladzislau Rezki
2021-02-15 11:28 ` Sebastian Andrzej Siewior
2021-02-16 17:30 ` Paul E. McKenney
2021-02-17 14:47 ` Masami Hiramatsu
2021-02-17 18:17 ` Paul E. McKenney [this message]
2021-02-18 5:03 ` Masami Hiramatsu
2021-02-18 8:36 ` Uladzislau Rezki
2021-02-18 14:29 ` Masami Hiramatsu
2020-12-09 20:37 ` [PATCH 1/2] rcu-tasks: move RCU-tasks initialization out of core_initcall() Uladzislau Rezki
2020-12-10 13:39 ` Daniel Axtens
2020-12-10 17:32 ` Paul E. McKenney
2020-12-10 18:17 ` Uladzislau Rezki
2020-12-10 3:26 ` Paul E. McKenney
2020-12-10 13:04 ` Uladzislau Rezki
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210217181738.GE2743@paulmck-ThinkPad-P72 \
--to=paulmck@kernel.org \
--cc=akpm@linux-foundation.org \
--cc=bigeasy@linutronix.de \
--cc=dja@axtens.net \
--cc=frederic@kernel.org \
--cc=joel@joelfernandes.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mhiramat@kernel.org \
--cc=mhocko@suse.com \
--cc=mpe@ellerman.id.au \
--cc=neeraju@codeaurora.org \
--cc=oleksiy.avramchenko@sonymobile.com \
--cc=peterz@infradead.org \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=tytso@mit.edu \
--cc=urezki@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).