From: Waiman Long <Waiman.Long@hp.com>
To: Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, "H. Peter Anvin" <hpa@zytor.com>,
Peter Zijlstra <peterz@infradead.org>
Cc: linux-arch@vger.kernel.org, x86@kernel.org,
linux-kernel@vger.kernel.org,
virtualization@lists.linux-foundation.org,
xen-devel@lists.xenproject.org, kvm@vger.kernel.org,
Paolo Bonzini <paolo.bonzini@gmail.com>,
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>,
Boris Ostrovsky <boris.ostrovsky@oracle.com>,
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
Rik van Riel <riel@redhat.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>,
David Vrabel <david.vrabel@citrix.com>,
Oleg Nesterov <oleg@redhat.com>, Gleb Natapov <gleb@redhat.com>,
Scott J Norton <scott.norton@hp.com>,
Chegu Vinod <chegu_vinod@hp.com>,
Waiman Long <Waiman.Long@hp.com>
Subject: [PATCH v11 06/16] qspinlock: prolong the stay in the pending bit path
Date: Fri, 30 May 2014 11:43:52 -0400 [thread overview]
Message-ID: <1401464642-33890-7-git-send-email-Waiman.Long@hp.com> (raw)
In-Reply-To: <1401464642-33890-1-git-send-email-Waiman.Long@hp.com>
There is a problem in the current pending bit spinning code. When the
lock is free, but the pending bit holder hasn't grabbed the lock &
cleared the pending bit yet, the spinning code will not be run.
As a result, the regular queuing code path might be used most of
the time even when there is only 2 tasks contending for the lock.
Assuming that the pending bit holder is going to get the lock and
clear the pending bit soon, it is actually better to wait than to be
queued up which has a higher overhead.
The following tables show the before-patch execution time (in ms)
of a micro-benchmark where 5M iterations of the lock/unlock cycles
were run on a 10-core Westere-EX x86-64 CPU with 2 different types of
loads - standalone (lock and protected data in different cachelines)
and embedded (lock and protected data in the same cacheline).
[Standalone/Embedded - same node]
# of tasks Ticket lock Queue lock %Change
---------- ----------- ---------- -------
1 135/ 111 135/ 101 0%/ -9%
2 890/ 779 1885/1990 +112%/+156%
3 1932/1859 2333/2341 +21%/ +26%
4 2829/2726 2900/2923 +3%/ +7%
5 3834/3761 3655/3648 -5%/ -3%
6 4963/4976 4336/4326 -13%/ -13%
7 6299/6269 5057/5064 -20%/ -19%
8 7691/7569 5786/5798 -25%/ -23%
Of course, the results will varies depending on what kind of test
machine is used.
With 1 task per NUMA node, the execution times are:
[Standalone - different nodes]
# of nodes Ticket lock Queue lock %Change
---------- ----------- ---------- -------
1 135 135 0%
2 4604 5087 +10%
3 10940 12224 +12%
4 21555 10555 -51%
It can be seen that the queue spinlock is slower than the ticket
spinlock when there are 2 or 3 contending tasks. In all the other case,
the queue spinlock is either equal or faster than the ticket spinlock.
With this patch, the performance data for 2 contending tasks are:
[Standalone/Embedded]
# of tasks Ticket lock Queue lock %Change
---------- ----------- ---------- -------
2 890/779 984/871 +11%/+12%
[Standalone - different nodes]
# of nodes Ticket lock Queue lock %Change
---------- ----------- ---------- -------
2 4604 1364 -70%
It can be seen that the queue spinlock performance for 2 contending
tasks is now comparable to ticket spinlock on the same node, but much
faster when in different nodes. With 3 contending tasks, however,
the ticket spinlock is still quite a bit faster.
Signed-off-by: Waiman Long <Waiman.Long@hp.com>
---
kernel/locking/qspinlock.c | 18 ++++++++++++++++--
1 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c
index fc7fd8c..7f10758 100644
--- a/kernel/locking/qspinlock.c
+++ b/kernel/locking/qspinlock.c
@@ -233,11 +233,25 @@ void queue_spin_lock_slowpath(struct qspinlock *lock, u32 val)
*/
for (;;) {
/*
- * If we observe any contention; queue.
+ * If we observe that the queue is not empty or both
+ * the pending and lock bits are set, queue
*/
- if (val & ~_Q_LOCKED_MASK)
+ if ((val & _Q_TAIL_MASK) ||
+ (val == (_Q_LOCKED_VAL|_Q_PENDING_VAL)))
goto queue;
+ if (val == _Q_PENDING_VAL) {
+ /*
+ * Pending bit is set, but not the lock bit.
+ * Assuming that the pending bit holder is going to
+ * set the lock bit and clear the pending bit soon,
+ * it is better to wait than to exit at this point.
+ */
+ cpu_relax();
+ val = atomic_read(&lock->val);
+ continue;
+ }
+
new = _Q_LOCKED_VAL;
if (val == new)
new |= _Q_PENDING_VAL;
--
1.7.1
next prev parent reply other threads:[~2014-05-30 15:45 UTC|newest]
Thread overview: 102+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-05-30 15:43 [PATCH v11 00/16] qspinlock: a 4-byte queue spinlock with PV support Waiman Long
2014-05-30 15:43 ` [PATCH v11 01/16] qspinlock: A simple generic 4-byte queue spinlock Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` [PATCH v11 02/16] qspinlock, x86: Enable x86-64 to use " Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` [PATCH v11 03/16] qspinlock: Add pending bit Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` [PATCH v11 04/16] qspinlock: Extract out the exchange of tail code word Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` [PATCH v11 05/16] qspinlock: Optimize for smaller NR_CPUS Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long [this message]
2014-06-11 10:26 ` [PATCH v11 06/16] qspinlock: prolong the stay in the pending bit path Peter Zijlstra
2014-06-11 10:26 ` Peter Zijlstra
2014-06-11 21:22 ` Long, Wai Man
2014-06-11 21:22 ` Long, Wai Man
2014-06-11 21:22 ` Long, Wai Man
2014-06-12 6:00 ` Peter Zijlstra
2014-06-12 20:54 ` Waiman Long
2014-06-12 20:54 ` Waiman Long
2014-06-15 13:12 ` Peter Zijlstra
2014-06-15 13:12 ` Peter Zijlstra
2014-06-15 13:12 ` Peter Zijlstra
2014-06-12 20:54 ` Waiman Long
2014-06-12 6:00 ` Peter Zijlstra
2014-06-12 6:00 ` Peter Zijlstra
2014-06-11 10:26 ` Peter Zijlstra
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` [PATCH v11 07/16] qspinlock: Use a simple write to grab the lock, if applicable Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` [PATCH v11 08/16] qspinlock: Prepare for unfair lock support Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` [PATCH v11 09/16] qspinlock, x86: Allow unfair spinlock in a virtual guest Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-06-11 10:54 ` Peter Zijlstra
2014-06-11 10:54 ` Peter Zijlstra
2014-06-11 11:38 ` Peter Zijlstra
2014-06-11 11:38 ` Peter Zijlstra
2014-06-11 11:38 ` Peter Zijlstra
2014-06-12 1:37 ` Long, Wai Man
2014-06-12 1:37 ` Long, Wai Man
2014-06-12 1:37 ` Long, Wai Man
2014-06-12 5:50 ` Peter Zijlstra
2014-06-12 5:50 ` Peter Zijlstra
2014-06-12 21:08 ` Waiman Long
2014-06-12 21:08 ` Waiman Long
2014-06-15 13:14 ` Peter Zijlstra
2014-06-15 13:14 ` Peter Zijlstra
2014-06-15 13:14 ` Peter Zijlstra
2014-06-12 21:08 ` Waiman Long
2014-06-12 5:50 ` Peter Zijlstra
2014-06-11 10:54 ` Peter Zijlstra
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` [PATCH v11 10/16] qspinlock: Split the MCS queuing code into a separate slowerpath Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` [PATCH v11 11/16] pvqspinlock, x86: Rename paravirt_ticketlocks_enabled Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` [PATCH v11 12/16] pvqspinlock, x86: Add PV data structure & methods Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` [PATCH v11 13/16] pvqspinlock: Enable coexistence with the unfair lock Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:43 ` Waiman Long
2014-05-30 15:44 ` [PATCH v11 14/16] pvqspinlock: Add qspinlock para-virtualization support Waiman Long
2014-06-12 8:17 ` Peter Zijlstra
2014-06-12 8:17 ` Peter Zijlstra
2014-06-12 20:48 ` Waiman Long
2014-06-12 20:48 ` Waiman Long
2014-06-15 13:16 ` Peter Zijlstra
2014-06-15 13:16 ` Peter Zijlstra
2014-06-15 13:16 ` Peter Zijlstra
2014-06-17 20:59 ` Konrad Rzeszutek Wilk
2014-06-17 20:59 ` Konrad Rzeszutek Wilk
2014-06-17 20:59 ` Konrad Rzeszutek Wilk
2014-06-12 20:48 ` Waiman Long
2014-06-12 8:17 ` Peter Zijlstra
2014-05-30 15:44 ` Waiman Long
2014-05-30 15:44 ` Waiman Long
2014-05-30 15:44 ` [PATCH v11 15/16] pvqspinlock, x86: Enable PV qspinlock PV for KVM Waiman Long
2014-05-30 15:44 ` Waiman Long
2014-05-30 15:44 ` Waiman Long
2014-05-30 15:44 ` [PATCH v11 16/16] pvqspinlock, x86: Enable PV qspinlock for XEN Waiman Long
2014-05-30 15:44 ` Waiman Long
2014-05-30 15:44 ` Waiman Long
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1401464642-33890-7-git-send-email-Waiman.Long@hp.com \
--to=waiman.long@hp.com \
--cc=boris.ostrovsky@oracle.com \
--cc=chegu_vinod@hp.com \
--cc=david.vrabel@citrix.com \
--cc=gleb@redhat.com \
--cc=hpa@zytor.com \
--cc=konrad.wilk@oracle.com \
--cc=kvm@vger.kernel.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=oleg@redhat.com \
--cc=paolo.bonzini@gmail.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=raghavendra.kt@linux.vnet.ibm.com \
--cc=riel@redhat.com \
--cc=scott.norton@hp.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=virtualization@lists.linux-foundation.org \
--cc=x86@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.