From: Florian Fainelli <f.fainelli@gmail.com>
To: Boqun Feng <boqun.feng@gmail.com>
Cc: torvalds@linux-foundation.org, linux-kernel@vger.kernel.org,
kernel-team@meta.com, paulmck@kernel.org, mingo@kernel.org,
tglx@linutronix.de, rcu@vger.kernel.org, joel@joelfernandes.org,
neeraj.upadhyay@amd.com, urezki@gmail.com,
qiang.zhang1211@gmail.com, frederic@kernel.org,
bigeasy@linutronix.de, anna-maria@linutronix.de,
chenzhongjin@huawei.com, yangjihong1@huawei.com,
rostedt@goodmis.org
Subject: Re: Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for v6.9]
Date: Tue, 12 Mar 2024 14:34:29 -0700 [thread overview]
Message-ID: <c5f9c640-4c06-495e-9c7e-0c208b914fa7@gmail.com> (raw)
In-Reply-To: <ZfDEIs63EBIYBJIC@boqun-archlinux>
On 3/12/24 14:07, Boqun Feng wrote:
> On Tue, Mar 12, 2024 at 01:32:03PM -0700, Florian Fainelli wrote:
>> Hi Boqun,
>>
>> On 3/8/24 09:15, Boqun Feng wrote:
>>> Hi Linus,
>>>
>>> Please pull this for the RCU changes of v6.9:
>>>
>>> The following changes since commit 41bccc98fb7931d63d03f326a746ac4d429c1dd3:
>>>
>>> Linux 6.8-rc2 (2024-01-28 17:01:12 -0800)
>>>
>>> are available in the Git repository at:
>>>
>>> git://git.kernel.org/pub/scm/linux/kernel/git/boqun/linux.git tags/rcu.next.v6.9
>>>
>>> for you to fetch changes up to 3add00be5fe5810d7aa5ec3af8b6a245ef33144b:
>>>
>>> Merge branches 'rcu-doc.2024.02.14a', 'rcu-nocb.2024.02.14a', 'rcu-exp.2024.02.14a', 'rcu-tasks.2024.02.26a' and 'rcu-misc.2024.02.14a' into rcu.2024.02.26a (2024-02-26 17:37:25 -0800)
>>>
>>>
>>> Two merge conflicts were detected by linux-next:
>>>
>>> * https://lore.kernel.org/lkml/20240226135745.12ac854d@canb.auug.org.au/
>>> * https://lore.kernel.org/lkml/20240227125522.2bdbe6be@canb.auug.org.au/
>>>
>>> These conflict resolutions from linux-next look good to me, plus I made
>>> my own resolutions at branch merge/rcu.2024.02.27a for your reference.
>>>
>>>
>>> Some highlights of the changes:
>>>
>>> * Eliminates deadlocks involving do_exit() and RCU tasks, by Paul:
>>> Instead of SRCU read side critical sections, now a percpu list is used
>>> in do_exit() for scaning yet-to-exit tasks.
>>>
>>> * Fixes a deadlock due to the dependency between workqueue and RCU
>>> expedited grace period, reported by Anna-Maria Behnsen and Thomas
>>> Gleixner and fixed by Frederic: Now RCU expedited always uses its own
>>> kthread worker instead of a workqueue.
>>
>> At least one device in my test farm (ARM 32-bit) has consistently shown a
>> very long boot, and some others are intermittently affected. This
>> consistently looks like this on most of my devices:
>>
>> [ 2.450351] bcmgenet f0480000.ethernet: GENET 5.0 EPHY: 0x0000
>> [ 2.547562] ata1: SATA link down (SStatus 0 SControl 300)
>> [ 162.107264] unimac-mdio unimac-mdio.0: Broadcom UniMAC MDIO bus
>>
>> this gets flagged by my boot script as a boot failure since we exceeded the
>> 30 seconds timeout given to boot a kernel to a prompt.
>>
>> It has been somewhat difficult to get a reliable bisection going on, but
>> what I am sure of is that e5a3878c947ceef7b6ab68fdc093f3848059842c~1 does
>> not expose the problem for 10 consecutive boots, while I *might* see it at
>> e5a3878c947ceef7b6ab68fdc093f3848059842c and beyond.
>>
>> Any clues what is going on here?
>>
>
> Could you share the config file and your kernel parameters? Also could
> you share the whole log? Thanks!
Here is the configuration file:
https://gist.github.com/ffainelli/f3fd38752a186cee0cb8719089c0b6b8
and here is a log where this fails:
https://gist.github.com/ffainelli/ed08a2b3e853f59343786ebd20364fc8
--
Florian
next prev parent reply other threads:[~2024-03-12 21:34 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-08 17:15 [GIT PULL] RCU changes for v6.9 Boqun Feng
2024-03-11 19:43 ` pr-tracker-bot
2024-03-12 20:32 ` Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for v6.9] Florian Fainelli
2024-03-12 21:01 ` Frederic Weisbecker
2024-03-12 21:15 ` Paul E. McKenney
2024-03-12 21:35 ` Florian Fainelli
2024-03-12 22:05 ` Florian Fainelli
2024-03-12 21:07 ` Boqun Feng
2024-03-12 21:34 ` Florian Fainelli [this message]
2024-03-12 21:44 ` Linus Torvalds
2024-03-12 23:48 ` Boqun Feng
2024-03-13 16:01 ` Joel Fernandes
2024-03-13 21:30 ` Florian Fainelli
2024-03-13 21:59 ` Russell King (Oracle)
2024-03-13 22:04 ` Florian Fainelli
2024-03-13 22:49 ` Russell King (Oracle)
2024-03-13 23:29 ` Florian Fainelli
2024-03-14 1:15 ` Linus Torvalds
2024-03-14 1:22 ` Florian Fainelli
2024-03-13 22:52 ` Frederic Weisbecker
2024-03-14 3:44 ` Florian Fainelli
2024-03-14 5:12 ` Boqun Feng
2024-03-14 6:33 ` Boqun Feng
2024-03-14 9:32 ` Thomas Gleixner
2024-03-14 9:11 ` Thomas Gleixner
2024-03-14 10:41 ` Frederic Weisbecker
2024-03-14 18:35 ` Florian Fainelli
2024-03-14 18:51 ` Boqun Feng
2024-03-14 19:09 ` Florian Fainelli
2024-03-14 20:45 ` Thomas Gleixner
2024-03-14 21:21 ` Thomas Gleixner
2024-03-14 21:53 ` Florian Fainelli
2024-03-14 22:51 ` Thomas Gleixner
2024-03-14 21:58 ` Thomas Gleixner
2024-03-14 22:05 ` Boqun Feng
2024-03-14 22:10 ` Boqun Feng
2024-03-15 1:14 ` [PATCH] timer/migration: Remove buggy early return on deactivation [was Re: Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for v6.9]] Frederic Weisbecker
2024-03-15 1:20 ` Frederic Weisbecker
2024-03-15 13:44 ` Florian Fainelli
2024-03-16 19:06 ` [tip: timers/urgent] timer/migration: Remove buggy early return on deactivation tip-bot2 for Frederic Weisbecker
2024-03-26 16:41 ` [PATCH] timer/migration: Remove buggy early return on deactivation [was Re: Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for v6.9]] Anna-Maria Behnsen
2024-03-26 17:18 ` Frederic Weisbecker
2024-04-04 16:50 ` [PATCH] timers/migration: Return early on deactivation Anna-Maria Behnsen
2024-04-04 22:19 ` Frederic Weisbecker
2024-04-05 8:53 ` [PATCH v2] " Anna-Maria Behnsen
2024-04-05 9:11 ` [tip: timers/urgent] " tip-bot2 for Anna-Maria Behnsen
2024-03-14 9:03 ` Unexplained long boot delays [Was Re: [GIT PULL] RCU changes for v6.9] Thomas Gleixner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c5f9c640-4c06-495e-9c7e-0c208b914fa7@gmail.com \
--to=f.fainelli@gmail.com \
--cc=anna-maria@linutronix.de \
--cc=bigeasy@linutronix.de \
--cc=boqun.feng@gmail.com \
--cc=chenzhongjin@huawei.com \
--cc=frederic@kernel.org \
--cc=joel@joelfernandes.org \
--cc=kernel-team@meta.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=neeraj.upadhyay@amd.com \
--cc=paulmck@kernel.org \
--cc=qiang.zhang1211@gmail.com \
--cc=rcu@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=urezki@gmail.com \
--cc=yangjihong1@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).