From: Waiman Long <Waiman.Long@hpe.com>
To: Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@redhat.com>
Cc: linux-kernel@vger.kernel.org, x86@kernel.org,
linux-alpha@vger.kernel.org, linux-ia64@vger.kernel.org,
linux-s390@vger.kernel.org, linux-arch@vger.kernel.org,
Davidlohr Bueso <dave@stgolabs.net>,
Jason Low <jason.low2@hp.com>, Dave Chinner <david@fromorbit.com>,
Scott J Norton <scott.norton@hpe.com>,
Douglas Hatch <doug.hatch@hpe.com>,
Waiman Long <Waiman.Long@hpe.com>
Subject: [RFC PATCH-tip v2 4/6] locking/rwsem: move down rwsem_down_read_failed function
Date: Tue, 14 Jun 2016 18:48:07 -0400 [thread overview]
Message-ID: <1465944489-43440-5-git-send-email-Waiman.Long@hpe.com> (raw)
In-Reply-To: <1465944489-43440-1-git-send-email-Waiman.Long@hpe.com>
Move the rwsem_down_read_failed() function down to below the optimistic
spinning section before enabling optimistic spinning for the readers.
There is no change in code.
Signed-off-by: Waiman Long <Waiman.Long@hpe.com>
---
kernel/locking/rwsem-xadd.c | 116 +++++++++++++++++++++---------------------
1 files changed, 58 insertions(+), 58 deletions(-)
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index adc2f44..c08c778 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -226,64 +226,6 @@ __rwsem_mark_wake(struct rw_semaphore *sem,
}
/*
- * Wait for the read lock to be granted
- */
-__visible
-struct rw_semaphore __sched *rwsem_down_read_failed(struct rw_semaphore *sem)
-{
- long count, adjustment = 0;
- struct rwsem_waiter waiter;
- struct task_struct *tsk = current;
- WAKE_Q(wake_q);
-
- /*
- * Undo read bias from down_read operation, stop active locking.
- * Doing that after taking the wait_lock may block writer lock
- * stealing for too long.
- */
- atomic_long_add(-RWSEM_ACTIVE_READ_BIAS, &sem->count);
-
- /* set up my own style of waitqueue */
- waiter.task = tsk;
- waiter.type = RWSEM_WAITING_FOR_READ;
-
- raw_spin_lock_irq(&sem->wait_lock);
- if (list_empty(&sem->wait_list))
- adjustment += RWSEM_WAITING_BIAS;
- list_add_tail(&waiter.list, &sem->wait_list);
-
- /* we're now waiting on the lock */
- if (adjustment)
- count = atomic_long_add_return(adjustment, &sem->count);
- else
- count = atomic_long_read(&sem->count);
-
- /* If there are no active locks, wake the front queued process(es).
- *
- * If there are no writers and we are first in the queue,
- * wake our own waiter to join the existing active readers !
- */
- if (count == RWSEM_WAITING_BIAS ||
- (count > RWSEM_WAITING_BIAS && adjustment))
- sem = __rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q);
-
- raw_spin_unlock_irq(&sem->wait_lock);
- wake_up_q(&wake_q);
-
- /* wait to be given the lock */
- while (true) {
- set_task_state(tsk, TASK_UNINTERRUPTIBLE);
- if (!waiter.task)
- break;
- schedule();
- }
-
- __set_task_state(tsk, TASK_RUNNING);
- return sem;
-}
-EXPORT_SYMBOL(rwsem_down_read_failed);
-
-/*
* This function must be called with the sem->wait_lock held to prevent
* race conditions between checking the rwsem wait list and setting the
* sem->count accordingly.
@@ -505,6 +447,64 @@ static inline bool reader_spinning_enabled(struct rw_semaphore *sem)
#endif
/*
+ * Wait for the read lock to be granted
+ */
+__visible
+struct rw_semaphore __sched * rwsem_down_read_failed(struct rw_semaphore *sem)
+{
+ long count, adjustment = 0;
+ struct rwsem_waiter waiter;
+ struct task_struct *tsk = current;
+ WAKE_Q(wake_q);
+
+ /*
+ * Undo read bias from down_read operation, stop active locking.
+ * Doing that after taking the wait_lock may block writer lock
+ * stealing for too long.
+ */
+ atomic_long_add(-RWSEM_ACTIVE_READ_BIAS, &sem->count);
+
+ /* set up my own style of waitqueue */
+ waiter.task = tsk;
+ waiter.type = RWSEM_WAITING_FOR_READ;
+
+ raw_spin_lock_irq(&sem->wait_lock);
+ if (list_empty(&sem->wait_list))
+ adjustment += RWSEM_WAITING_BIAS;
+ list_add_tail(&waiter.list, &sem->wait_list);
+
+ /* we're now waiting on the lock */
+ if (adjustment)
+ count = atomic_long_add_return(adjustment, &sem->count);
+ else
+ count = atomic_long_read(&sem->count);
+
+ /* If there are no active locks, wake the front queued process(es).
+ *
+ * If there are no writers and we are first in the queue,
+ * wake our own waiter to join the existing active readers !
+ */
+ if (count == RWSEM_WAITING_BIAS ||
+ (count > RWSEM_WAITING_BIAS && adjustment))
+ sem = __rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q);
+
+ raw_spin_unlock_irq(&sem->wait_lock);
+ wake_up_q(&wake_q);
+
+ /* wait to be given the lock */
+ while (true) {
+ set_task_state(tsk, TASK_UNINTERRUPTIBLE);
+ if (!waiter.task)
+ break;
+ schedule();
+ }
+
+ __set_task_state(tsk, TASK_RUNNING);
+ return sem;
+}
+EXPORT_SYMBOL(rwsem_down_read_failed);
+
+/*
* Wait until we successfully acquire the write lock
*/
static inline struct rw_semaphore *
--
1.7.1
next prev parent reply other threads:[~2016-06-14 22:49 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-14 22:48 [RFC PATCH-tip v2 0/6] locking/rwsem: Enable reader optimistic spinning Waiman Long
2016-06-14 22:48 ` [RFC PATCH-tip v2 1/6] locking/osq: Make lock/unlock proper acquire/release barrier Waiman Long
2016-06-15 8:04 ` Boqun Feng
2016-06-15 17:18 ` Peter Zijlstra
2016-06-15 19:01 ` Waiman Long
2016-06-16 2:19 ` Boqun Feng
2016-06-16 10:16 ` Will Deacon
2016-06-16 21:35 ` Waiman Long
2016-06-17 0:48 ` Boqun Feng
2016-06-17 15:26 ` Waiman Long
2016-06-17 15:45 ` Will Deacon
2016-06-17 18:17 ` Waiman Long
2016-06-18 8:46 ` Boqun Feng
2016-06-20 7:59 ` Will Deacon
2016-06-15 16:56 ` Davidlohr Bueso
2016-06-15 17:12 ` Peter Zijlstra
2016-06-15 18:27 ` Davidlohr Bueso
2016-06-15 18:40 ` Peter Zijlstra
2016-06-15 18:56 ` Davidlohr Bueso
2016-06-17 1:11 ` Davidlohr Bueso
2016-06-17 14:28 ` Waiman Long
2016-06-17 16:29 ` Davidlohr Bueso
2016-06-17 16:46 ` Davidlohr Bueso
2016-06-15 19:08 ` Waiman Long
2016-06-15 20:04 ` Waiman Long
2016-06-15 21:59 ` Peter Zijlstra
2016-06-14 22:48 ` [RFC PATCH-tip v2 2/6] locking/rwsem: Stop active read lock ASAP Waiman Long
2016-06-15 17:22 ` Peter Zijlstra
2016-06-15 19:17 ` Waiman Long
2016-06-16 2:14 ` Davidlohr Bueso
2016-06-16 21:25 ` Waiman Long
2016-06-14 22:48 ` [RFC PATCH-tip v2 3/6] locking/rwsem: Enable count-based spinning on reader Waiman Long
2016-06-15 17:38 ` Peter Zijlstra
2016-06-15 19:28 ` Waiman Long
2016-06-14 22:48 ` Waiman Long [this message]
2016-06-15 17:40 ` [RFC PATCH-tip v2 4/6] locking/rwsem: move down rwsem_down_read_failed function Peter Zijlstra
2016-06-15 19:21 ` Waiman Long
2016-06-14 22:48 ` [RFC PATCH-tip v2 5/6] locking/rwsem: Change RWSEM_WAITING_BIAS for better disambiguation Waiman Long
2016-06-15 17:43 ` Peter Zijlstra
2016-06-15 19:31 ` Waiman Long
2016-06-15 21:57 ` Peter Zijlstra
2016-06-15 17:45 ` Peter Zijlstra
2016-06-15 19:35 ` Waiman Long
2016-06-14 22:48 ` [RFC PATCH-tip v2 6/6] locking/rwsem: Enable spinning readers Waiman Long
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1465944489-43440-5-git-send-email-Waiman.Long@hpe.com \
--to=waiman.long@hpe.com \
--cc=dave@stgolabs.net \
--cc=david@fromorbit.com \
--cc=doug.hatch@hpe.com \
--cc=jason.low2@hp.com \
--cc=linux-alpha@vger.kernel.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-ia64@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-s390@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=scott.norton@hpe.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).