From: Julia Cartwright <julia@ni.com>
To: linux-kernel@vger.kernel.org, linux-rt-users@vger.kernel.org
Cc: Thomas Gleixner <tglx@linutronix.de>,
Steven Rostedt <rostedt@goodmis.org>,
Carsten Emde <C.Emde@osadl.org>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
John Kacur <jkacur@redhat.com>,
Paul Gortmaker <paul.gortmaker@windriver.com>,
Daniel Wagner <daniel.wagner@siemens.com>,
tom.zanussi@linux.intel.com, Boqun Feng <boqun.feng@gmail.com>,
Peter Zijlstra <peterz@infradead.org>,
Lai Jiangshan <jiangshanlai@gmail.com>,
Josh Triplett <josh@joshtriplett.org>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
"Paul E . McKenney" <paulmck@linux.vnet.ibm.com>
Subject: [PATCH RT 05/22] rtmutex: Make rt_mutex_futex_unlock() safe for irq-off callsites
Date: Mon, 6 Aug 2018 04:17:21 -0500 [thread overview]
Message-ID: <bd21741bb5e918e5ed254b9fa869b5523a469b55.1533540554.git.julia@ni.com> (raw)
In-Reply-To: <cover.1533540554.git.julia@ni.com>
From: Boqun Feng <boqun.feng@gmail.com>
4.9.115-rt94-rc1 stable review patch.
If you have any objection to the inclusion of this patch, let me know.
--- 8< --- 8< --- 8< ---
[ Upstream commit 6b0ef92fee2a3189eba6d6b827b247cb4f6da7e9 ]
When running rcutorture with TREE03 config, CONFIG_PROVE_LOCKING=y, and
kernel cmdline argument "rcutorture.gp_exp=1", lockdep reports a
HARDIRQ-safe->HARDIRQ-unsafe deadlock:
=============================== WARNING: inconsistent lock state
4.16.0-rc4+ #1 Not tainted
--------------------------------
inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
takes:
__schedule+0xbe/0xaf0
{IN-HARDIRQ-W} state was registered at:
_raw_spin_lock+0x2a/0x40
scheduler_tick+0x47/0xf0
...
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0
----
lock(&rq->lock);
<Interrupt>
lock(&rq->lock);
*** DEADLOCK ***
1 lock held by rcu_torture_rea/724:
rcu_torture_read_lock+0x0/0x70
stack backtrace:
CPU: 2 PID: 724 Comm: rcu_torture_rea Not tainted 4.16.0-rc4+ #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-20171110_100015-anatol 04/01/2014
Call Trace:
lock_acquire+0x90/0x200
? __schedule+0xbe/0xaf0
_raw_spin_lock+0x2a/0x40
? __schedule+0xbe/0xaf0
__schedule+0xbe/0xaf0
preempt_schedule_irq+0x2f/0x60
retint_kernel+0x1b/0x2d
RIP: 0010:rcu_read_unlock_special+0x0/0x680
? rcu_torture_read_unlock+0x60/0x60
__rcu_read_unlock+0x64/0x70
rcu_torture_read_unlock+0x17/0x60
rcu_torture_reader+0x275/0x450
? rcutorture_booster_init+0x110/0x110
? rcu_torture_stall+0x230/0x230
? kthread+0x10e/0x130
kthread+0x10e/0x130
? kthread_create_worker_on_cpu+0x70/0x70
? call_usermodehelper_exec_async+0x11a/0x150
ret_from_fork+0x3a/0x50
This happens with the following even sequence:
preempt_schedule_irq();
local_irq_enable();
__schedule():
local_irq_disable(); // irq off
...
rcu_note_context_switch():
rcu_note_preempt_context_switch():
rcu_read_unlock_special():
local_irq_save(flags);
...
raw_spin_unlock_irqrestore(...,flags); // irq remains off
rt_mutex_futex_unlock():
raw_spin_lock_irq();
...
raw_spin_unlock_irq(); // accidentally set irq on
<return to __schedule()>
rq_lock():
raw_spin_lock(); // acquiring rq->lock with irq on
which means rq->lock becomes a HARDIRQ-unsafe lock, which can cause
deadlocks in scheduler code.
This problem was introduced by commit 02a7c234e540 ("rcu: Suppress
lockdep false-positive ->boost_mtx complaints"). That brought the user
of rt_mutex_futex_unlock() with irq off.
To fix this, replace the *lock_irq() in rt_mutex_futex_unlock() with
*lock_irq{save,restore}() to make it safe to call rt_mutex_futex_unlock()
with irq off.
Fixes: 02a7c234e540 ("rcu: Suppress lockdep false-positive ->boost_mtx complaints")
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Lai Jiangshan <jiangshanlai@gmail.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Josh Triplett <josh@joshtriplett.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: "Paul E . McKenney" <paulmck@linux.vnet.ibm.com>
Link: https://lkml.kernel.org/r/20180309065630.8283-1-boqun.feng@gmail.com
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Julia Cartwright <julia@ni.com>
---
kernel/locking/rtmutex.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 57361d631749..5e15f5c73637 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -2213,11 +2213,12 @@ void __sched rt_mutex_futex_unlock(struct rt_mutex *lock)
{
WAKE_Q(wake_q);
WAKE_Q(wake_sleeper_q);
+ unsigned long flags;
bool postunlock;
- raw_spin_lock_irq(&lock->wait_lock);
+ raw_spin_lock_irqsave(&lock->wait_lock, flags);
postunlock = __rt_mutex_futex_unlock(lock, &wake_q, &wake_sleeper_q);
- raw_spin_unlock_irq(&lock->wait_lock);
+ raw_spin_unlock_irqrestore(&lock->wait_lock, flags);
if (postunlock)
rt_mutex_postunlock(&wake_q, &wake_sleeper_q);
--
2.18.0
next prev parent reply other threads:[~2018-08-06 9:18 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-06 9:17 [PATCH RT 00/22] Linux 4.9.115-rt94-rc1 Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 01/22] futex: Fix pi_state->owner serialization Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 02/22] futex: Fix more put_pi_state() vs. exit_pi_state_list() races Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 03/22] futex: Avoid violating the 10th rule of futex Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 04/22] futex: Fix OWNER_DEAD fixup Julia Cartwright
2018-08-06 9:17 ` Julia Cartwright [this message]
2018-08-06 9:17 ` [PATCH RT 06/22] rcu: Do not include rtmutex_common.h unconditionally Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 07/22] rcu: Suppress lockdep false-positive ->boost_mtx complaints Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 08/22] sched, tracing: Fix trace_sched_pi_setprio() for deboosting Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 09/22] crypto: limit more FPU-enabled sections Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 10/22] arm*: disable NEON in kernel mode Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 11/22] mm/slub: close possible memory-leak in kmem_cache_alloc_bulk() Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 12/22] locking: add types.h Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 13/22] net: use task_struct instead of CPU number as the queue owner on -RT Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 14/22] Revert "rt,ntp: Move call to schedule_delayed_work() to helper thread" Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 15/22] Revert "block: blk-mq: Use swait" Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 16/22] block: blk-mq: move blk_queue_usage_counter_release() into process context Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 17/22] alarmtimer: Prevent live lock in alarm_cancel() Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 18/22] posix-timers: move the rcu head out of the union Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 19/22] locallock: provide {get,put}_locked_ptr() variants Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 20/22] squashfs: make use of local lock in multi_cpu decompressor Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 21/22] seqlock: provide the same ordering semantics as mainline Julia Cartwright
2018-08-06 9:17 ` [PATCH RT 22/22] Linux 4.9.115-rt94-rc1 Julia Cartwright
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bd21741bb5e918e5ed254b9fa869b5523a469b55.1533540554.git.julia@ni.com \
--to=julia@ni.com \
--cc=C.Emde@osadl.org \
--cc=bigeasy@linutronix.de \
--cc=boqun.feng@gmail.com \
--cc=daniel.wagner@siemens.com \
--cc=jiangshanlai@gmail.com \
--cc=jkacur@redhat.com \
--cc=josh@joshtriplett.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rt-users@vger.kernel.org \
--cc=mathieu.desnoyers@efficios.com \
--cc=paul.gortmaker@windriver.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=tglx@linutronix.de \
--cc=tom.zanussi@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).