From: Sultan Alsawaf <sultan@kerneltoast.com>
To: unlisted-recipients:; (no To-header on input)
Cc: Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>, Will Deacon <will@kernel.org>,
linux-kernel@vger.kernel.org,
Sultan Alsawaf <sultan@kerneltoast.com>
Subject: [PATCH 2/2] locking/rwsem: Don't hog RCU read lock while optimistically spinning
Date: Fri, 7 Aug 2020 12:16:36 -0700 [thread overview]
Message-ID: <20200807191636.75045-2-sultan@kerneltoast.com> (raw)
In-Reply-To: <20200807191636.75045-1-sultan@kerneltoast.com>
From: Sultan Alsawaf <sultan@kerneltoast.com>
There's no reason to hold an RCU read lock the entire time while
optimistically spinning for a rwsem. This can needlessly lengthen RCU
grace periods and slow down synchronize_rcu() when it doesn't brute
force the RCU grace period via rcupdate.rcu_expedited=1.
Signed-off-by: Sultan Alsawaf <sultan@kerneltoast.com>
---
kernel/locking/rwsem.c | 27 +++++++++++++++++----------
1 file changed, 17 insertions(+), 10 deletions(-)
diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c
index f11b9bd3431d..a1e3ceb254d1 100644
--- a/kernel/locking/rwsem.c
+++ b/kernel/locking/rwsem.c
@@ -723,8 +723,10 @@ rwsem_spin_on_owner(struct rw_semaphore *sem, unsigned long nonspinnable)
if (state != OWNER_WRITER)
return state;
- rcu_read_lock();
for (;;) {
+ bool same_owner;
+
+ rcu_read_lock();
/*
* When a waiting writer set the handoff flag, it may spin
* on the owner as well. Once that writer acquires the lock,
@@ -732,27 +734,32 @@ rwsem_spin_on_owner(struct rw_semaphore *sem, unsigned long nonspinnable)
* handoff bit is set.
*/
new = rwsem_owner_flags(sem, &new_flags);
- if ((new != owner) || (new_flags != flags)) {
- state = rwsem_owner_state(new, new_flags, nonspinnable);
- break;
- }
/*
- * Ensure we emit the owner->on_cpu, dereference _after_
- * checking sem->owner still matches owner, if that fails,
+ * Ensure sem->owner still matches owner. If that fails,
* owner might point to free()d memory, if it still matches,
* the rcu_read_lock() ensures the memory stays valid.
*/
- barrier();
+ same_owner = new == owner && new_flags == flags;
+ if (same_owner && !owner_on_cpu(owner))
+ state = OWNER_NONSPINNABLE;
+ rcu_read_unlock();
- if (need_resched() || !owner_on_cpu(owner)) {
+ if (!same_owner) {
+ state = rwsem_owner_state(new, new_flags, nonspinnable);
+ break;
+ }
+
+ if (state == OWNER_NONSPINNABLE)
+ break;
+
+ if (need_resched()) {
state = OWNER_NONSPINNABLE;
break;
}
cpu_relax();
}
- rcu_read_unlock();
return state;
}
--
2.28.0
next prev parent reply other threads:[~2020-08-07 19:17 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-07 19:16 [PATCH 1/2] locking/mutex: Don't hog RCU read lock while optimistically spinning Sultan Alsawaf
2020-08-07 19:16 ` Sultan Alsawaf [this message]
2020-09-07 16:20 ` Will Deacon
2020-09-14 0:36 ` Sultan Alsawaf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200807191636.75045-2-sultan@kerneltoast.com \
--to=sultan@kerneltoast.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).