From: Juri Lelli <juri.lelli@redhat.com>
To: linux-rt-users <linux-rt-users@vger.kernel.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
Thomas Gleixner <tglx@linutronix.de>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Clark Williams <williams@redhat.com>
Subject: [RT WARNING] DEBUG_LOCKS_WARN_ON(rt_mutex_owner(lock) != current) with fsfreeze (4.19.25-rt16)
Date: Tue, 26 Mar 2019 10:34:21 +0100 [thread overview]
Message-ID: <20190326093421.GA29508@localhost.localdomain> (raw)
Hi,
Running this reproducer on a 4.19.25-rt16 kernel (with lock debugging
turned on) produces warning below.
--->8---
# dd if=/dev/zero of=fsfreezetest count=999999
# mkfs -t xfs -q ./fsfreezetest
# mkdir testmount
# mount -t xfs -o loop ./fsfreezetest ./testmount
# for I in `seq 10`; do fsfreeze -f ./testmount; sleep 1; fsfreeze -u ./testmount; done
--->8---
------------[ cut here ]------------
DEBUG_LOCKS_WARN_ON(rt_mutex_owner(lock) != current)
WARNING: CPU: 10 PID: 1226 at kernel/locking/rtmutex-debug.c:145 debug_rt_mutex_unlock+0x9b/0xb0
Modules linked in: xfs [...]
CPU: 10 PID: 1226 Comm: fsfreeze Not tainted 4.19.25-rt16 #2
Hardware name: LENOVO 30B6S2F900/1030, BIOS S01KT61A 09/28/2018
RIP: 0010:debug_rt_mutex_unlock+0x9b/0xb0
Code: e8 aa af 3c 00 4c 8b 04 24 85 c0 74 a9 8b 05 3c 9c a6 02 85 c0 75 9f 48 c7 c6 b8 b4 2d 98 48 c7 c7 9b d2 2b 98 e8 d9 e5 f8 ff <0f> 0b 4c 8b 04 24 eb 84 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 c3
RSP: 0018:ffffa7efa60cbdd0 EFLAGS: 00010086
RAX: 0000000000000000 RBX: ffff991b72813920 RCX: 0000000000000000
RDX: 0000000000000007 RSI: ffffffff98318de2 RDI: 00000000ffffffff
RBP: 0000000000000246 R08: 0000000000000000 R09: 0000000000024200
R10: 0000000000000000 R11: 0000000000000000 R12: ffffa7efa60cbe08
R13: ffffa7efa60cbe18 R14: ffff991b72813478 R15: ffffffff9730718d
FS: 00007f19baf6a540(0000) GS:ffff991b9fb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f19bae87040 CR3: 000000103c6ee002 CR4: 00000000001606e0
Call Trace:
rt_mutex_slowunlock+0x24/0x70
__rt_mutex_unlock+0x45/0x80
percpu_up_write+0x4b/0x60
thaw_super_locked+0xdb/0x110
do_vfs_ioctl+0x647/0x6f0
ksys_ioctl+0x60/0x90
__x64_sys_ioctl+0x16/0x20
do_syscall_64+0x60/0x1f0
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7f19bae9704b
Code: 0f 1e fa 48 8b 05 3d be 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0d be 0c 00 f7 d8 64 89 01 48
RSP: 002b:00007ffc6d275358 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f19bae9704b
RDX: 0000000000000000 RSI: 00000000c0045878 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc6d275755
R13: 00007ffc6d275500 R14: 0000000000000000 R15: 0000000000000000
irq event stamp: 8002
hardirqs last enabled at (8001): [<ffffffff97a25981>] _raw_spin_unlock_irqrestore+0x81/0x90
hardirqs last disabled at (8002): [<ffffffff97a25aa0>] _raw_spin_lock_irqsave+0x20/0x60
softirqs last enabled at (0): [<ffffffff970c04ad>] copy_process.part.36+0x89d/0x2170
softirqs last disabled at (0): [<0000000000000000>] (null)
---[ end trace 0000000000000002 ]---
AFAIU, this is a legit warning, since
fsfreeze -f ./testmount grabs rt_mutexes embedded into
sb->s_writers.rw_sem[SB_FREEZE_LEVELS] (rt-rwsem) as part of executing
sb_wait_write() (for each FREEZE_LEVEL) in freeze_super().
We then return to userspace.
fsfreeze -u ./testmount unlocks the rt_mutexes while doing
sb_freeze_unlock() in thaw_super_locked(). This is a different process
w.r.t. the one that did the freeze above.
I noticed that a very similar problem was fixed (for !rt rwsem) by
5a817641f68a ("locking/percpu-rwsem: Annotate rwsem ownership transfer
by setting RWSEM_OWNER_UNKNOWN"). However, RT has of course to deal with
PI, so I wonder if there is an easy fix for this problem.
Suggestions?
Thanks,
- Juri
next reply other threads:[~2019-03-26 9:34 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-03-26 9:34 Juri Lelli [this message]
2019-03-28 10:17 ` [RT WARNING] DEBUG_LOCKS_WARN_ON(rt_mutex_owner(lock) != current) with fsfreeze (4.19.25-rt16) Sebastian Andrzej Siewior
2019-04-19 8:56 ` Juri Lelli
2019-04-30 12:51 ` Sebastian Andrzej Siewior
2019-04-30 13:28 ` Peter Zijlstra
2019-04-30 13:45 ` Sebastian Andrzej Siewior
2019-04-30 14:01 ` Peter Zijlstra
2019-04-30 14:15 ` Oleg Nesterov
2019-04-30 14:29 ` Peter Zijlstra
2019-04-30 14:42 ` Oleg Nesterov
2019-04-30 14:44 ` Peter Zijlstra
2019-04-30 14:53 ` Oleg Nesterov
2019-05-01 17:09 ` Peter Zijlstra
2019-05-01 17:26 ` Waiman Long
2019-05-01 18:54 ` Peter Zijlstra
2019-05-01 19:22 ` Davidlohr Bueso
2019-05-01 19:25 ` Peter Zijlstra
2019-05-02 10:09 ` Oleg Nesterov
2019-05-02 11:42 ` Oleg Nesterov
2019-05-03 14:50 ` Peter Zijlstra
2019-05-03 15:25 ` Peter Zijlstra
2019-05-06 16:50 ` Oleg Nesterov
2019-06-19 9:50 ` Peter Zijlstra
2019-05-03 14:16 ` Peter Zijlstra
2019-05-03 15:37 ` Oleg Nesterov
2019-05-03 15:46 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190326093421.GA29508@localhost.localdomain \
--to=juri.lelli@redhat.com \
--cc=bigeasy@linutronix.de \
--cc=bristot@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-users@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=williams@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).