From: Prateek Sood <prsood@codeaurora.org>
To: peterz@infradead.org, mingo@redhat.com
Cc: Prateek Sood <prsood@codeaurora.org>,
sramana@codeaurora.org, linux-kernel@vger.kernel.org
Subject: [PATCH] rwsem: fix missed wakeup due to reordering of load
Date: Wed, 23 Aug 2017 16:58:55 +0530 [thread overview]
Message-ID: <1503487735-4362-1-git-send-email-prsood@codeaurora.org> (raw)
If a spinner is present, there is a chance that the load of
rwsem_has_spinner() in rwsem_wake() can be reordered with
respect to decrement of rwsem count in __up_write() leading
to wakeup being missed.
spinning writer up_write caller
--------------- -----------------------
[S] osq_unlock() [L] osq
spin_lock(wait_lock)
sem->count=0xFFFFFFFF00000001
+0xFFFFFFFF00000000
count=sem->count
MB
sem->count=0xFFFFFFFE00000001
-0xFFFFFFFF00000001
RMB
spin_trylock(wait_lock)
return
rwsem_try_write_lock(count)
spin_unlock(wait_lock)
schedule()
Reordering of atomic_long_sub_return_release() in __up_write()
and rwsem_has_spinner() in rwsem_wake() can cause missing of
wakeup in up_write() context. In spinning writer, sem->count
and local variable count is 0XFFFFFFFE00000001. It would result
in rwsem_try_write_lock() failing to acquire rwsem and spinning
writer going to sleep in rwsem_down_write_failed().
The smp_rmb() will make sure that the spinner state is
consulted after sem->count is updated in up_write context.
Signed-off-by: Prateek Sood <prsood@codeaurora.org>
---
kernel/locking/rwsem-xadd.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 45 insertions(+)
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 34e727f..5c687f6 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -586,6 +586,51 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
DEFINE_WAKE_Q(wake_q);
/*
+ * If a spinner is present, there is a chance that the load of
+ * rwsem_has_spinner() in rwsem_wake() can be reordered with
+ * respect to decrement of rwsem count in __up_write() leading
+ * to wakeup being missed.
+ *
+ * spinning writer up_write caller
+ * --------------- -----------------------
+ * [S] osq_unlock() [L] osq
+ * spin_lock(wait_lock)
+ * sem->count=0xFFFFFFFF00000001
+ * +0xFFFFFFFF00000000
+ * count=sem->count
+ * MB
+ * sem->count=0xFFFFFFFE00000001
+ * -0xFFFFFFFF00000001
+ * RMB
+ * spin_trylock(wait_lock)
+ * return
+ * rwsem_try_write_lock(count)
+ * spin_unlock(wait_lock)
+ * schedule()
+ *
+ * Reordering of atomic_long_sub_return_release() in __up_write()
+ * and rwsem_has_spinner() in rwsem_wake() can cause missing of
+ * wakeup in up_write() context. In spinning writer, sem->count
+ * and local variable count is 0XFFFFFFFE00000001. It would result
+ * in rwsem_try_write_lock() failing to acquire rwsem and spinning
+ * writer going to sleep in rwsem_down_write_failed().
+ *
+ *
+ * The RMB in below example is to make sure that the spinner state is
+ * consulted after sem->count is updated in up_write context.
+ * This would guarantee trylock on rwsem is successful
+ * in rwsem_down_write_failed().
+ * spinning writer up_write caller
+ * --------------- -----------------------
+ * [S] osq_unlock() atomic_update(sem->count)
+ * RMB
+ * atomic_update(sem->count) [L] osq
+ * MB
+ * rwsem_try_write_lock(count)
+ */
+ smp_rmb();
+
+ /*
* If a spinner is present, it is not necessary to do the wakeup.
* Try to do wakeup only if the trylock succeeds to minimize
* spinlock contention which may introduce too much delay in the
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc.,
is a member of Code Aurora Forum, a Linux Foundation Collaborative Project.
next reply other threads:[~2017-08-23 11:29 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-08-23 11:28 Prateek Sood [this message]
2017-08-24 11:29 ` [PATCH] rwsem: fix missed wakeup due to reordering of load Peter Zijlstra
2017-08-24 12:33 ` Peter Zijlstra
2017-08-24 12:52 ` Peter Zijlstra
2017-09-07 14:08 ` Prateek Sood
-- strict thread matches above, loose matches on Subject: below --
2017-09-07 14:30 Prateek Sood
2017-09-19 14:05 ` Andrea Parri
2017-09-20 14:52 ` Davidlohr Bueso
2017-09-20 21:17 ` Andrea Parri
2017-09-27 21:20 ` Davidlohr Bueso
2017-09-26 18:37 ` Prateek Sood
2017-07-26 20:17 Prateek Sood
2017-07-27 15:48 ` Waiman Long
2017-07-27 16:59 ` Peter Zijlstra
2017-08-10 8:32 ` Andrea Parri
2017-08-10 10:41 ` Peter Zijlstra
2017-08-10 10:44 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1503487735-4362-1-git-send-email-prsood@codeaurora.org \
--to=prsood@codeaurora.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=sramana@codeaurora.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).