From: Peter Zijlstra <peterz@infradead.org>
To: Waiman Long <Waiman.Long@hp.com>
Cc: Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, "H. Peter Anvin" <hpa@zytor.com>,
Arnd Bergmann <arnd@arndb.de>,
linux-arch@vger.kernel.org, x86@kernel.org,
linux-kernel@vger.kernel.org,
Steven Rostedt <rostedt@goodmis.org>,
Andrew Morton <akpm@linux-foundation.org>,
Michel Lespinasse <walken@google.com>,
Andi Kleen <andi@firstfloor.org>, Rik van Riel <riel@redhat.com>,
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>,
George Spelvin <linux@horizon.com>,
Tim Chen <tim.c.chen@linux.intel.com>,
"Aswin Chandramouleeswaran\"" <aswin@hp.com>,
Scott J Norton <scott.norton@hp.com>
Subject: Re: [PATCH v11 0/4] Introducing a queue read/write lock implementation
Date: Fri, 31 Jan 2014 10:26:16 +0100 [thread overview]
Message-ID: <20140131092616.GC5126@laptop.programming.kicks-ass.net> (raw)
In-Reply-To: <20140130151715.GA5126@laptop.programming.kicks-ass.net>
On Thu, Jan 30, 2014 at 04:17:15PM +0100, Peter Zijlstra wrote:
> The below is still small and actually works.
OK, so having actually worked through the thing; I realized we can
actually do a version without MCS lock and instead use a ticket lock for
the waitqueue.
This is both smaller (back to 8 bytes for the rwlock_t), and should be
faster under moderate contention for not having to touch extra
cachelines.
Completely untested and with a rather crude generic ticket lock
implementation to illustrate the concept:
---
--- a/include/asm-generic/qrwlock_types.h
+++ b/include/asm-generic/qrwlock_types.h
@@ -3,15 +3,13 @@
#include <linux/atomic.h>
-struct mcs_spinlock;
-
/*
* The queue read/write lock data structure
*/
struct qrwlock {
atomic_t cnts;
- struct mcs_spinlock *waitq;
+ atomic_t tickets;
};
#endif /* __ASM_GENERIC_QRWLOCK_TYPES_H */
--- a/kernel/locking/qrwlock.c
+++ b/kernel/locking/qrwlock.c
@@ -21,29 +21,32 @@
#include <linux/percpu.h>
#include <linux/hardirq.h>
#include <linux/mutex.h>
-#include <linux/mcs_spinlock.h>
#include <asm/qrwlock.h>
-/*
- * Compared with regular rwlock, the queue rwlock has has the following
- * advantages:
- * 1. Even though there is a slight chance of stealing the lock if come at
- * the right moment, the granting of the lock is mostly in FIFO order.
- * 2. It is usually faster in high contention situation.
- *
- * The only downside is that the lock is 4 bytes larger in 32-bit systems
- * and 12 bytes larger in 64-bit systems.
- *
- * There are two queues for writers. The writer field of the lock is a
- * one-slot wait queue. The writers that follow will have to wait in the
- * combined reader/writer queue (waitq).
- *
- * Compared with x86 ticket spinlock, the queue rwlock is faster in high
- * contention situation. The writer lock is also faster in single thread
- * operations. Therefore, queue rwlock can be considered as a replacement
- * for those spinlocks that are highly contended as long as an increase
- * in lock size is not an issue.
- */
+#define TICKET_MSB 0x8000
+#define TICKET_MASK 0x7FFF
+
+#define atomic_xadd(v, a) (atomic_add_return((v), (a)) - (v))
+
+static inline void ticket_spin_lock(atomic_t *tickets)
+{
+ u32 val = atomic_xadd(1 << 16, tickets);
+ u32 ticket = (val >> 16) & TICKET_MASK;
+
+ if (unlikely(val & TICKET_MSB))
+ atomic_clear_mask(TICKET_MSB, tickets);
+
+ if (unlikely((val & TICKET_MASK) != ticket)) {
+ while ((smp_load_acquire((u32 *)tickets) & TICKET_MASK) != ticket)
+ cpu_relax();
+ }
+}
+
+static inline void ticket_spin_unlock(atomic_t *tickets)
+{
+ smp_mb__before_atomic_inc();
+ atomic_inc(tickets);
+}
/**
* rspin_until_writer_unlock - inc reader count & spin until writer is gone
@@ -68,7 +71,6 @@ rspin_until_writer_unlock(struct qrwlock
*/
void queue_read_lock_slowpath(struct qrwlock *lock)
{
- struct mcs_spinlock node;
u32 cnts;
/*
@@ -88,7 +90,7 @@ void queue_read_lock_slowpath(struct qrw
/*
* Put the reader into the wait queue
*/
- mcs_spin_lock(&lock->waitq, &node);
+ ticket_spin_lock(&lock->tickets);
/*
* At the head of the wait queue now, wait until the writer state
@@ -100,13 +102,13 @@ void queue_read_lock_slowpath(struct qrw
while (atomic_read(&lock->cnts) & _QW_WMASK)
arch_mutex_cpu_relax();
- cnts = atomic_add_return(_QR_BIAS, &lock->cnts) - _QR_BIAS;
+ cnts = atomic_xadd(_QR_BIAS, &lock->cnts);
rspin_until_writer_unlock(lock, cnts);
/*
* Signal the next one in queue to become queue head
*/
- mcs_spin_unlock(&lock->waitq, &node);
+ ticket_spin_unlock(&lock->tickets);
}
EXPORT_SYMBOL(queue_read_lock_slowpath);
@@ -116,11 +118,10 @@ EXPORT_SYMBOL(queue_read_lock_slowpath);
*/
void queue_write_lock_slowpath(struct qrwlock *lock)
{
- struct mcs_spinlock node;
u32 cnts;
/* Put the writer into the wait queue */
- mcs_spin_lock(&lock->waitq, &node);
+ ticket_spin_lock(&lock->tickets);
/* Try to acquire the lock directly if no reader is present */
if (!atomic_read(&lock->cnts) &&
@@ -152,6 +153,6 @@ void queue_write_lock_slowpath(struct qr
arch_mutex_cpu_relax();
}
unlock:
- mcs_spin_unlock(&lock->waitq, &node);
+ ticket_spin_unlock(&lock->tickets);
}
EXPORT_SYMBOL(queue_write_lock_slowpath);
next prev parent reply other threads:[~2014-01-31 9:26 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-01-24 4:28 [PATCH v11 0/4] Introducing a queue read/write lock implementation Waiman Long
2014-01-24 4:28 ` [PATCH v11 1/4] qrwlock: A " Waiman Long
2014-01-24 8:25 ` Peter Zijlstra
2014-01-25 4:39 ` Waiman Long
2014-01-24 4:28 ` [PATCH v11 2/4] qrwlock, x86: Enable x86 to use queue read/write lock Waiman Long
2014-01-24 4:28 ` [PATCH v11 3/4] qrwlock, x86: Add char and short as atomic data type in x86 Waiman Long
2014-01-24 4:28 ` [PATCH v11 4/4] qrwlock: Use the mcs_spinlock helper functions for MCS queuing Waiman Long
2014-01-24 8:26 ` Peter Zijlstra
2014-01-25 4:30 ` Waiman Long
2014-01-30 13:04 ` [PATCH v11 0/4] Introducing a queue read/write lock implementation Peter Zijlstra
2014-01-30 15:17 ` Peter Zijlstra
2014-01-30 15:43 ` Waiman Long
2014-01-30 15:50 ` Waiman Long
2014-01-30 15:53 ` Peter Zijlstra
2014-01-30 15:44 ` Peter Zijlstra
2014-01-30 17:52 ` Will Deacon
2014-01-30 18:05 ` Peter Zijlstra
2014-01-30 18:11 ` Will Deacon
2014-01-30 18:16 ` Peter Zijlstra
2014-01-31 9:26 ` Peter Zijlstra [this message]
2014-01-31 10:03 ` George Spelvin
2014-01-31 10:17 ` Peter Zijlstra
2014-01-31 11:30 ` Peter Zijlstra
2014-02-01 23:22 ` Paul E. McKenney
2014-01-31 18:59 ` Waiman Long
2014-01-31 19:47 ` Peter Zijlstra
2014-02-01 10:38 ` George Spelvin
2014-01-31 20:14 ` Peter Zijlstra
2014-01-31 21:09 ` Waiman Long
2014-02-01 1:29 ` Davidlohr Bueso
2014-02-02 9:03 ` Ingo Molnar
2014-02-03 11:38 ` Peter Zijlstra
2014-02-06 3:08 ` Waiman Long
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140131092616.GC5126@laptop.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=Waiman.Long@hp.com \
--cc=akpm@linux-foundation.org \
--cc=andi@firstfloor.org \
--cc=arnd@arndb.de \
--cc=aswin@hp.com \
--cc=hpa@zytor.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux@horizon.com \
--cc=mingo@redhat.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=raghavendra.kt@linux.vnet.ibm.com \
--cc=riel@redhat.com \
--cc=rostedt@goodmis.org \
--cc=scott.norton@hp.com \
--cc=tglx@linutronix.de \
--cc=tim.c.chen@linux.intel.com \
--cc=torvalds@linux-foundation.org \
--cc=walken@google.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).