From: Pierre Gondois <pierre.gondois@arm.com>
To: Mark Rutland <mark.rutland@arm.com>, Jan Kara <jack@suse.cz>
Cc: Waiman Long <longman@redhat.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
LKML <linux-kernel@vger.kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
Steven Rostedt <rostedt@goodmis.org>,
Mel Gorman <mgorman@suse.de>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>, Will Deacon <will@kernel.org>,
Catalin Marinas <catalin.marinas@arm.com>
Subject: Re: Crash with PREEMPT_RT on aarch64 machine
Date: Wed, 9 Nov 2022 11:11:46 +0100 [thread overview]
Message-ID: <fb95c436-c648-7459-b79e-c38afe11f1b5@arm.com> (raw)
In-Reply-To: <Y2t4+6MwVZEhoV5n@FVFF77S0Q05N.cambridge.arm.com>
On 11/9/22 10:55, Mark Rutland wrote:
> On Tue, Nov 08, 2022 at 06:45:29PM +0100, Jan Kara wrote:
>> On Tue 08-11-22 10:53:40, Mark Rutland wrote:
>>> On Mon, Nov 07, 2022 at 11:49:01AM -0500, Waiman Long wrote:
>>>> On 11/7/22 10:10, Sebastian Andrzej Siewior wrote:
>>>>> + locking, arm64
>>>>>
>>>>> On 2022-11-07 14:56:36 [+0100], Jan Kara wrote:
>>>>>>> spinlock_t and raw_spinlock_t differ slightly in terms of locking.
>>>>>>> rt_spin_lock() has the fast path via try_cmpxchg_acquire(). If you
>>>>>>> enable CONFIG_DEBUG_RT_MUTEXES then you would force the slow path which
>>>>>>> always acquires the rt_mutex_base::wait_lock (which is a raw_spinlock_t)
>>>>>>> while the actual lock is modified via cmpxchg.
>>>>>> So I've tried enabling CONFIG_DEBUG_RT_MUTEXES and indeed the corruption
>>>>>> stops happening as well. So do you suspect some bug in the CPU itself?
>>>>> If it is only enabling CONFIG_DEBUG_RT_MUTEXES (and not whole lockdep)
>>>>> then it looks very suspicious.
>>>>> CONFIG_DEBUG_RT_MUTEXES enables a few additional checks but the main
>>>>> part is that rt_mutex_cmpxchg_acquire() + rt_mutex_cmpxchg_release()
>>>>> always fail (and so the slowpath under a raw_spinlock_t is done).
>>>>>
>>>>> So if it is really the fast path (rt_mutex_cmpxchg_acquire()) then it
>>>>> somehow smells like the CPU is misbehaving.
>>>>>
>>>>> Could someone from the locking/arm64 department check if the locking in
>>>>> RT-mutex (rtlock_lock()) is correct?
>>>>>
>>>>> rtmutex locking uses try_cmpxchg_acquire(, ptr, ptr) for the fastpath
>>>>> (and try_cmpxchg_release(, ptr, ptr) for unlock).
>>>>> Now looking at it again, I don't see much difference compared to what
>>>>> queued_spin_trylock() does except the latter always operates on 32bit
>>>>> value instead a pointer.
>>>>
>>>> Both the fast path of queued spinlock and rt_spin_lock are using
>>>> try_cmpxchg_acquire(), the only difference I saw is the size of the data to
>>>> be cmpxchg'ed. qspinlock uses 32-bit integer whereas rt_spin_lock uses
>>>> 64-bit pointer. So I believe it is more on how the arm64 does cmpxchg. I
>>>> believe there are two different ways of doing it depending on whether LSE
>>>> atomics is available in the platform. So exactly what arm64 system is being
>>>> used here and what hardware capability does it have?
>>>
>>> From the /proc/cpuinfo output earlier, this is a Neoverse N1 system, with the
>>> LSE atomics. Assuming the kernel was built with support for atomics in-kernel
>>> (which is selected by default), it'll be using the LSE version.
>>
>> So I was able to reproduce the corruption both with LSE atomics enabled &
>> disabled in the kernel. It seems the problem takes considerably longer to
>> reproduce with LSE atomics enabled but it still does happen.
>>
>> BTW, I've tried to reproduced the problem on another aarch64 machine with
>> CPU from a different vendor:
>>
>> processor : 0
>> BogoMIPS : 200.00
>> Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm
>> CPU implementer : 0x48
>> CPU architecture: 8
>> CPU variant : 0x1
>> CPU part : 0xd01
>> CPU revision : 0
>>
>> And there the problem does not reproduce. So might it be a genuine bug in
>> the CPU implementation?
>
> Perhaps, though I suspect it's more likely that we have an ordering bug in the
> kernel code, and it shows up on CPUs with legitimate but more relaxed ordering.
> We've had a couple of those show up on Apple M1, so it might be worth trying on
> one of those.
>
> How easy is this to reproduce? What's necessary?
>
> Thanks,
> Mark.
It is possible to reproduce it on an Ampere Altra, which has the following cpuinfo:
processor : 0
BogoMIPS : 50.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x3
CPU part : 0xd0c
CPU revision : 1
Command used:
dbench 80 -t 300 --clients-per-process=8
With the diff [2] applied, I get [1]. So sb_clear_inode_writeback()
seems to be called on a CPU with a null i_count. Maybe:
- CPUx deletes the inode via iput().
- CPUy calls sb_clear_inode_writeback() to update inode->i_wb_list
at the same time.
[1]
[ 165.003036] ------------[ cut here ]------------
[ 165.003042] kernel BUG at fs/fs-writeback.c:1294!
[ 165.003047] Internal error: Oops - BUG: 0 [#1] PREEMPT_RT SMP
[ 165.003052] Modules linked in: [...]
[ 165.003131] CPU: 87 PID: 9 Comm: kworker/u320:0 Not tainted 6.0.5-rt14-[...] #92
[ 165.003137] Hardware name: WIWYNN Mt.Jade Server System B81.03001.0005/Mt.Jade Motherboard, BIOS 1.08.20220218 (SCP: 1.08.20220218) 2022/02/18
[ 165.003138] Workqueue: ext4-rsv-conversion ext4_end_io_rsv_work
[ 165.003148] pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 165.003151] pc : sb_clear_inode_writeback (fs/fs-writeback.c:1294 (discriminator 1))
[ 165.003159] lr : sb_clear_inode_writeback (./include/linux/atomic/atomic-instrumented.h:28 fs/fs-writeback.c:1292)
[...]
[ 165.003219] Call trace:
[ 165.003222] sb_clear_inode_writeback (fs/fs-writeback.c:1294 (discriminator 1))
[ 165.003226] __folio_end_writeback (./include/linux/spinlock_rt.h:123 mm/page-writeback.c:2942)
[ 165.003231] folio_end_writeback (mm/filemap.c:1620)
[ 165.003234] end_page_writeback (mm/folio-compat.c:27)
[ 165.003237] ext4_finish_bio (fs/ext4/page-io.c:145)
[ 165.003240] ext4_release_io_end (fs/ext4/page-io.c:161 (discriminator 3))
[ 165.003243] ext4_end_io_rsv_work (./include/linux/list.h:292 fs/ext4/page-io.c:254 fs/ext4/page-io.c:273)
[ 165.003246] process_one_work (kernel/workqueue.c:2294)
[ 165.003250] worker_thread (./include/linux/list.h:292 kernel/workqueue.c:2437)
[ 165.003252] kthread (kernel/kthread.c:376)
[ 165.003255] ret_from_fork (arch/arm64/kernel/entry.S:861)
[ 165.003260] Code: 54fff8a0 942cf1d4 17ffffc3 d4210000 (d4210000)
[...]
[ 165.245010] ------------[ cut here ]------------
[2]
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 443f83382b9b..0edb03eb43a4 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -1252,13 +1252,27 @@ void sb_mark_inode_writeback(struct inode *inode)
{
struct super_block *sb = inode->i_sb;
unsigned long flags;
+ int local_count;
if (list_empty(&inode->i_wb_list)) {
spin_lock_irqsave(&sb->s_inode_wblist_lock, flags);
+
+ local_count = atomic_read(&inode->count) + 1;
+ BUG_ON(local_count != atomic_inc_return(&inode->count));
+ BUG_ON(!atomic_read(&inode->i_count));
+
if (list_empty(&inode->i_wb_list)) {
+
+ BUG_ON(!atomic_read(&inode->i_count));
+
list_add_tail(&inode->i_wb_list, &sb->s_inodes_wb);
trace_sb_mark_inode_writeback(inode);
}
+
+ local_count = atomic_read(&inode->count) + 1;
+ BUG_ON(local_count != atomic_inc_return(&inode->count));
+ BUG_ON(!atomic_read(&inode->i_count));
+
spin_unlock_irqrestore(&sb->s_inode_wblist_lock, flags);
}
}
@@ -1270,13 +1284,27 @@ void sb_clear_inode_writeback(struct inode *inode)
{
struct super_block *sb = inode->i_sb;
unsigned long flags;
+ int local_count;
if (!list_empty(&inode->i_wb_list)) {
spin_lock_irqsave(&sb->s_inode_wblist_lock, flags);
+
+ local_count = atomic_read(&inode->count) - 1;
+ BUG_ON(local_count != atomic_dec_return(&inode->count));
+ BUG_ON(!atomic_read(&inode->i_count));
+
if (!list_empty(&inode->i_wb_list)) {
+
+ BUG_ON(!atomic_read(&inode->i_count));
+
list_del_init(&inode->i_wb_list);
trace_sb_clear_inode_writeback(inode);
}
+
+ local_count = atomic_read(&inode->count) - 1;
+ BUG_ON(local_count != atomic_dec_return(&inode->count));
+ BUG_ON(!atomic_read(&inode->i_count));
+
spin_unlock_irqrestore(&sb->s_inode_wblist_lock, flags);
}
}
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 56a4b4b02477..67027d4973a1 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -658,6 +658,7 @@ struct inode {
struct list_head i_lru; /* inode LRU list */
struct list_head i_sb_list;
struct list_head i_wb_list; /* backing dev writeback list */
+ atomic_t count;
union {
struct hlist_head i_dentry;
struct rcu_head i_rcu;
next prev parent reply other threads:[~2022-11-09 10:12 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-03 11:54 Crash with PREEMPT_RT on aarch64 machine Jan Kara
2022-11-04 16:30 ` Sebastian Andrzej Siewior
2022-11-07 13:56 ` Jan Kara
2022-11-07 15:10 ` Sebastian Andrzej Siewior
2022-11-07 16:30 ` Jan Kara
2022-11-07 17:12 ` Sebastian Andrzej Siewior
2022-11-07 16:49 ` Waiman Long
2022-11-08 10:53 ` Mark Rutland
2022-11-08 17:45 ` Jan Kara
2022-11-09 9:55 ` Mark Rutland
2022-11-09 10:11 ` Pierre Gondois [this message]
2022-11-09 10:54 ` Jan Kara
2022-11-09 11:01 ` Jan Kara
2022-11-09 13:52 ` Pierre Gondois
2022-11-09 14:21 ` Pierre Gondois
2022-11-09 12:57 ` Will Deacon
2022-11-09 15:40 ` Jan Kara
2022-11-11 14:27 ` Jan Kara
2022-11-14 12:41 ` Will Deacon
2022-11-28 15:58 ` Sebastian Andrzej Siewior
2022-11-28 20:30 ` kernel test robot
2022-11-28 21:11 ` kernel test robot
2022-11-29 5:16 ` kernel test robot
2022-11-29 5:26 ` kernel test robot
2022-11-29 6:48 ` kernel test robot
2022-11-29 7:39 ` kernel test robot
2022-11-30 17:20 ` Pierre Gondois
2022-12-01 12:37 ` Jan Kara
2022-11-30 20:22 ` Mel Gorman
2022-12-01 17:09 ` Mel Gorman
[not found] ` <20221104080637.626-1-hdanton@sina.com>
2022-11-07 12:41 ` Jan Kara
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=fb95c436-c648-7459-b79e-c38afe11f1b5@arm.com \
--to=pierre.gondois@arm.com \
--cc=bigeasy@linutronix.de \
--cc=catalin.marinas@arm.com \
--cc=jack@suse.cz \
--cc=linux-kernel@vger.kernel.org \
--cc=longman@redhat.com \
--cc=mark.rutland@arm.com \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).