From: Joel Fernandes <joel@joelfernandes.org>
To: Boqun Feng <boqun.feng@gmail.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
Florian Fainelli <f.fainelli@gmail.com>,
linux-kernel@vger.kernel.org, kernel-team@meta.com,
paulmck@kernel.org, mingo@kernel.org, tglx@linutronix.de,
rcu@vger.kernel.org, neeraj.upadhyay@amd.com, urezki@gmail.com,
qiang.zhang1211@gmail.com, frederic@kernel.org,
bigeasy@linutronix.de, anna-maria@linutronix.de,
chenzhongjin@huawei.com, yangjihong1@huawei.com,
rostedt@goodmis.org
Subject: Re: Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for v6.9]
Date: Wed, 13 Mar 2024 12:01:45 -0400 [thread overview]
Message-ID: <CAEXW_YRvz8xf-6hpwpYqS=YNa-xkn4CsuJzELJxOH_2FP+6ptQ@mail.gmail.com> (raw)
In-Reply-To: <ZfDptafiK0jns050@boqun-archlinux>
Hello Florian,
On Tue, Mar 12, 2024 at 7:48 PM Boqun Feng <boqun.feng@gmail.com> wrote:
>
> On Tue, Mar 12, 2024 at 02:44:14PM -0700, Linus Torvalds wrote:
> > On Tue, 12 Mar 2024 at 14:34, Florian Fainelli <f.fainelli@gmail.com> wrote:
> > >
> > > and here is a log where this fails:
> > >
> > > https://gist.github.com/ffainelli/ed08a2b3e853f59343786ebd20364fc8
> >
> > You could try the 'initcall_debug' kernel command line.
> >
>
> Right, that'll be helpful.
>
> Besides I took a look at the config Florian shared, no TASKS_RCU,
> RCU_LAZY or RCU nocb is enabled. So probably the only left changes in
> the PR are around RCU expedited. Florian, could you see if you can build
> and test with CONFIG_PROVE_RCU=y (you need to select
> CONFIG_PROVE_LOCKING for that)? That'll call synchronize_rcu() +
> synchronize_rcu_expedited() before and after we switch
> rcu_scheduler_active, and it may provide more information. Thanks!
Adding to everyone's suggestions, could you also try booting with
"rcupdate.rcu_normal=1" ? This will disable expedited RCU and help us
further confirm that it is indeed expedited RCU (and then we can look
into fixing that).
Also there are 2 additional users of expedited RCU in this release I noticed:
78c3253f27e5 ("net: use synchronize_rcu_expedited in cleanup_net()")
1ebb85f9c03d ("netfilter: conntrack: expedite rcu in
nf_conntrack_cleanup_net_list")
Could you also try reverting those patches as well, and see if the
issue goes away?
thanks,
- Joel
On Tue, Mar 12, 2024 at 7:48 PM Boqun Feng <boqun.feng@gmail.com> wrote:
>
> On Tue, Mar 12, 2024 at 02:44:14PM -0700, Linus Torvalds wrote:
> > On Tue, 12 Mar 2024 at 14:34, Florian Fainelli <f.fainelli@gmail.com> wrote:
> > >
> > > and here is a log where this fails:
> > >
> > > https://gist.github.com/ffainelli/ed08a2b3e853f59343786ebd20364fc8
> >
> > You could try the 'initcall_debug' kernel command line.
> >
>
> Right, that'll be helpful.
>
> Besides I took a look at the config Florian shared, no TASKS_RCU,
> RCU_LAZY or RCU nocb is enabled. So probably the only left changes in
> the PR are around RCU expedited. Florian, could you see if you can build
> and test with CONFIG_PROVE_RCU=y (you need to select
> CONFIG_PROVE_LOCKING for that)? That'll call synchronize_rcu() +
> synchronize_rcu_expedited() before and after we switch
> rcu_scheduler_active, and it may provide more information. Thanks!
>
> Regards,
> Boqun
>
> > It will make the above *much* noisier, but it might - thanks to all
> > the new noise - show exactly *what* is being crazy slow to initialize.
> >
> > Because right now it's just radio silence in between those
> >
> > [ 1.926435] bcmgenet f0480000.ethernet: GENET 5.0 EPHY: 0x0000
> > [ 162.148135] unimac-mdio unimac-mdio.0: Broadcom UniMAC MDIO bus
> >
> > things, and that's presumably because some random initcall there just
> > takes forever to time out.
> >
> > Linus
next prev parent reply other threads:[~2024-03-13 16:02 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-08 17:15 [GIT PULL] RCU changes for v6.9 Boqun Feng
2024-03-11 19:43 ` pr-tracker-bot
2024-03-12 20:32 ` Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for v6.9] Florian Fainelli
2024-03-12 21:01 ` Frederic Weisbecker
2024-03-12 21:15 ` Paul E. McKenney
2024-03-12 21:35 ` Florian Fainelli
2024-03-12 22:05 ` Florian Fainelli
2024-03-12 21:07 ` Boqun Feng
2024-03-12 21:34 ` Florian Fainelli
2024-03-12 21:44 ` Linus Torvalds
2024-03-12 23:48 ` Boqun Feng
2024-03-13 16:01 ` Joel Fernandes [this message]
2024-03-13 21:30 ` Florian Fainelli
2024-03-13 21:59 ` Russell King (Oracle)
2024-03-13 22:04 ` Florian Fainelli
2024-03-13 22:49 ` Russell King (Oracle)
2024-03-13 23:29 ` Florian Fainelli
2024-03-14 1:15 ` Linus Torvalds
2024-03-14 1:22 ` Florian Fainelli
2024-03-13 22:52 ` Frederic Weisbecker
2024-03-14 3:44 ` Florian Fainelli
2024-03-14 5:12 ` Boqun Feng
2024-03-14 6:33 ` Boqun Feng
2024-03-14 9:32 ` Thomas Gleixner
2024-03-14 9:11 ` Thomas Gleixner
2024-03-14 10:41 ` Frederic Weisbecker
2024-03-14 18:35 ` Florian Fainelli
2024-03-14 18:51 ` Boqun Feng
2024-03-14 19:09 ` Florian Fainelli
2024-03-14 20:45 ` Thomas Gleixner
2024-03-14 21:21 ` Thomas Gleixner
2024-03-14 21:53 ` Florian Fainelli
2024-03-14 22:51 ` Thomas Gleixner
2024-03-14 21:58 ` Thomas Gleixner
2024-03-14 22:05 ` Boqun Feng
2024-03-14 22:10 ` Boqun Feng
2024-03-15 1:14 ` [PATCH] timer/migration: Remove buggy early return on deactivation [was Re: Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for v6.9]] Frederic Weisbecker
2024-03-15 1:20 ` Frederic Weisbecker
2024-03-15 13:44 ` Florian Fainelli
2024-03-16 19:06 ` [tip: timers/urgent] timer/migration: Remove buggy early return on deactivation tip-bot2 for Frederic Weisbecker
2024-03-26 16:41 ` [PATCH] timer/migration: Remove buggy early return on deactivation [was Re: Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for v6.9]] Anna-Maria Behnsen
2024-03-26 17:18 ` Frederic Weisbecker
2024-04-04 16:50 ` [PATCH] timers/migration: Return early on deactivation Anna-Maria Behnsen
2024-04-04 22:19 ` Frederic Weisbecker
2024-04-05 8:53 ` [PATCH v2] " Anna-Maria Behnsen
2024-04-05 9:11 ` [tip: timers/urgent] " tip-bot2 for Anna-Maria Behnsen
2024-03-14 9:03 ` Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for v6.9] Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAEXW_YRvz8xf-6hpwpYqS=YNa-xkn4CsuJzELJxOH_2FP+6ptQ@mail.gmail.com' \
--to=joel@joelfernandes.org \
--cc=anna-maria@linutronix.de \
--cc=bigeasy@linutronix.de \
--cc=boqun.feng@gmail.com \
--cc=chenzhongjin@huawei.com \
--cc=f.fainelli@gmail.com \
--cc=frederic@kernel.org \
--cc=kernel-team@meta.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=neeraj.upadhyay@amd.com \
--cc=paulmck@kernel.org \
--cc=qiang.zhang1211@gmail.com \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=urezki@gmail.com \
--cc=yangjihong1@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).