From: Waiman Long <longman@redhat.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>, Will Deacon <will@kernel.org>,
linux-kernel@vger.kernel.org, Davidlohr Bueso <dave@stgolabs.net>,
Phil Auld <pauld@redhat.com>
Subject: Re: [PATCH 3/5] locking/rwsem: Enable reader optimistic lock stealing
Date: Fri, 20 Nov 2020 12:26:58 -0500 [thread overview]
Message-ID: <d913b708-0206-2ba6-347f-ea57c2396a83@redhat.com> (raw)
In-Reply-To: <20201120143624.GD3040@hirez.programming.kicks-ass.net>
On 11/20/20 9:36 AM, Peter Zijlstra wrote:
> On Tue, Nov 17, 2020 at 10:04:27PM -0500, Waiman Long wrote:
>> diff --git a/kernel/locking/rwsem.c b/kernel/locking/rwsem.c
>> index ee374ae061c3..930dd4af3639 100644
>> --- a/kernel/locking/rwsem.c
>> +++ b/kernel/locking/rwsem.c
>> @@ -957,6 +957,12 @@ static inline bool rwsem_reader_phase_trylock(struct rw_semaphore *sem,
>> }
>> return false;
>> }
>> +
>> +static inline bool osq_is_empty(struct rw_semaphore *sem)
>> +{
>> + return !osq_is_locked(&sem->osq);
>> +}
>> +
>> #else
>> static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem,
>> unsigned long nonspinnable)
>> @@ -977,6 +983,10 @@ static inline bool rwsem_reader_phase_trylock(struct rw_semaphore *sem,
>> return false;
>> }
>>
>> +static inline bool osq_is_empty(sem)
>> +{
>> + return false;
>> +}
> Hurph, the naming seems to suggest this ought to be in osq_lock.h, but
> it really is part of rwsem, it captures the lack of osq member for this
> configuration.
>
> How about: rwsem_no_spinners() instead ?
Yes, sure. Will make the name change.
>
>> static inline int
>> rwsem_spin_on_owner(struct rw_semaphore *sem, unsigned long nonspinnable)
>> {
>> @@ -1007,6 +1017,22 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, int state, long count)
>> !(count & RWSEM_WRITER_LOCKED))
>> goto queue;
>>
>> + /*
>> + * Reader optimistic lock stealing
>> + *
>> + * We can take the read lock directly without doing
>> + * rwsem_optimistic_spin() if the conditions are right.
>> + * Also wake up other readers if it is the first reader.
>> + */
>> + if (!(count & (RWSEM_WRITER_LOCKED | RWSEM_FLAG_HANDOFF)) &&
>> + osq_is_empty(sem)) {
>> + rwsem_set_reader_owned(sem);
>> + lockevent_inc(rwsem_rlock_steal);
>> + if (rcnt == 1)
>> + goto wake_readers;
>> + return sem;
>> + }
> AFAICT this saves at least 3 atomic ops; how common is this case
> (you did add a counter but forgot to mention this).
>
Right, I should have mentioned the counter results.
Below is the relevant counter stats for a test system that have been up
for more than 21 hours:
rwsem_opt_rlock=11792583 (optmistically acquired read lock)
rwsem_rlock=193357272 (slowpath acquired read lock)
rwsem_rlock_steal=44795149 (lock stealing)
So lock stealing represents about 17.9% of the total read lock acquired
in non-fast path. I ran some microbenchmark test on the system before,
so it may skew a bit to the high side. Anyway, this is not an
insignificant amount.
Cheers,
Longman
next prev parent reply other threads:[~2020-11-20 17:27 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-18 3:04 [PATCH 0/5] locking/rwsem: Rework reader optimistic spinning Waiman Long
2020-11-18 3:04 ` [PATCH 1/5] locking/rwsem: Pass the current atomic count to rwsem_down_read_slowpath() Waiman Long
2020-11-18 3:04 ` [PATCH 2/5] locking/rwsem: Prevent potential lock starvation Waiman Long
2020-11-20 14:44 ` Peter Zijlstra
2020-11-20 17:27 ` Waiman Long
2020-11-18 3:04 ` [PATCH 3/5] locking/rwsem: Enable reader optimistic lock stealing Waiman Long
2020-11-20 14:36 ` Peter Zijlstra
2020-11-20 17:26 ` Waiman Long [this message]
2020-12-08 3:53 ` Davidlohr Bueso
2020-11-18 3:04 ` [PATCH 4/5] locking/rwsem: Wake up all waiting readers if RWSEM_WAKE_READ_OWNED Waiman Long
2020-11-18 4:53 ` Davidlohr Bueso
2020-11-19 18:37 ` Waiman Long
2020-11-18 3:04 ` [RFC PATCH 5/5] locking/rwsem: Remove reader optimistic spinning Waiman Long
2020-11-18 5:35 ` Davidlohr Bueso
2020-11-19 18:40 ` Waiman Long
2020-11-20 13:11 ` David Laight
2020-11-20 17:04 ` Waiman Long
2020-11-20 17:37 ` David Laight
2020-11-20 21:38 ` Davidlohr Bueso
2020-11-21 11:50 ` David Laight
2020-11-20 14:44 ` Peter Zijlstra
2020-11-20 22:39 ` Waiman Long
2020-11-20 14:42 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d913b708-0206-2ba6-347f-ea57c2396a83@redhat.com \
--to=longman@redhat.com \
--cc=dave@stgolabs.net \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=pauld@redhat.com \
--cc=peterz@infradead.org \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).