From: Waiman Long <longman@redhat.com>
To: Alex Kogan <alex.kogan@oracle.com>,
linux@armlinux.org.uk, peterz@infradead.org, mingo@redhat.com,
will.deacon@arm.com, arnd@arndb.de, linux-arch@vger.kernel.org,
linux-arm-kernel@lists.infradead.org,
linux-kernel@vger.kernel.org, tglx@linutronix.de, bp@alien8.de,
hpa@zytor.com, x86@kernel.org, guohanjun@huawei.com,
jglauber@marvell.com
Cc: steven.sistare@oracle.com, daniel.m.jordan@oracle.com,
dave.dice@oracle.com, rahul.x.yadav@oracle.com
Subject: Re: [PATCH v3 3/5] locking/qspinlock: Introduce CNA into the slow path of qspinlock
Date: Tue, 16 Jul 2019 22:16:29 -0400 [thread overview]
Message-ID: <9fa54e98-0b9b-0931-db32-c6bd6ccfe75b@redhat.com> (raw)
In-Reply-To: <20190715192536.104548-4-alex.kogan@oracle.com>
On 7/15/19 3:25 PM, Alex Kogan wrote:
> +/*
> + * Implement a NUMA-aware version of MCS (aka CNA, or compact NUMA-aware lock).
> + *
> + * In CNA, spinning threads are organized in two queues, a main queue for
> + * threads running on the same node as the current lock holder, and a
> + * secondary queue for threads running on other nodes. At the unlock time,
> + * the lock holder scans the main queue looking for a thread running on
> + * the same node. If found (call it thread T), all threads in the main queue
> + * between the current lock holder and T are moved to the end of the
> + * secondary queue, and the lock is passed to T. If such T is not found, the
> + * lock is passed to the first node in the secondary queue. Finally, if the
> + * secondary queue is empty, the lock is passed to the next thread in the
> + * main queue. To avoid starvation of threads in the secondary queue,
> + * those threads are moved back to the head of the main queue
> + * after a certain expected number of intra-node lock hand-offs.
> + *
> + * For more details, see https://arxiv.org/abs/1810.05600.
> + *
> + * Authors: Alex Kogan <alex.kogan@oracle.com>
> + * Dave Dice <dave.dice@oracle.com>
> + */
> +
> +struct cna_node {
> + struct mcs_spinlock mcs;
> + u32 numa_node;
> + u32 encoded_tail;
> + struct cna_node *tail; /* points to the secondary queue tail */
> +};
> +
> +#define CNA_NODE(ptr) ((struct cna_node *)(ptr))
> +
> +static void cna_init_node(struct mcs_spinlock *node)
> +{
> + struct cna_node *cn = CNA_NODE(node);
> + struct mcs_spinlock *base_node;
> + int cpuid;
> +
> + BUILD_BUG_ON(sizeof(struct cna_node) > sizeof(struct qnode));
> + /* we store a pointer in the node's @locked field */
> + BUILD_BUG_ON(sizeof(uintptr_t) > sizeof_field(struct mcs_spinlock, locked));
> +
> + cpuid = smp_processor_id();
> + cn->numa_node = cpu_to_node(cpuid);
> +
> + base_node = this_cpu_ptr(&qnodes[0].mcs);
> + cn->encoded_tail = encode_tail(cpuid, base_node->count - 1);
> +}
> +
> +/**
> + * find_successor - Scan the main waiting queue looking for the first
> + * thread running on the same node as the lock holder. If found (call it
> + * thread T), move all threads in the main queue between the lock holder
> + * and T to the end of the secondary queue and return T; otherwise, return NULL.
> + */
Here you talk about main and secondary queues. However, there is no
mention of what are those queues. As I am familiar with qspinlock queue,
I can figure out that the main queue is the mcs_node->next chain that
starts from the MCS lock holder. The secondary queue is a separate MCS
node chain with its head stored in the mcs_node->locked value of the MCS
lock holder and the tail stored in the cna->tail. More detail comments
on what and where they are will help to improve the readability of the
code. A simple graphic to illustrate those queues will help too, for example
/*
* MCS lock holder
* ===============
* mcs_node
* +--------+ +----+ +----+
* | next | ---> |next| -> ... |next| -> NULL [Main queue]
* | locked | -+ +----+ +----+
* +--------+ |
* | +----+ +----+
* +-> |next| -> ... |next| -> X [Secondary queue]
* cna_node +----+ +----+
* +--------* ^
* | tail | ----------------------+
* +--------*
*
* N.B. locked = 1 if secondary queue is absent.
*/
> +static struct cna_node *find_successor(struct mcs_spinlock *me)
> +{
> + struct cna_node *me_cna = CNA_NODE(me);
> + struct cna_node *head_other, *tail_other, *cur;
As you have talked about secondary queue, how about head_2nd, tail_2nd
instead of *_other?
Cheers,
Longman
next prev parent reply other threads:[~2019-07-17 2:16 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-15 19:25 [PATCH v3 0/5] Add NUMA-awareness to qspinlock Alex Kogan
2019-07-15 19:25 ` [PATCH v3 1/5] locking/qspinlock: Make arch_mcs_spin_unlock_contended more generic Alex Kogan
2019-07-16 10:23 ` Peter Zijlstra
2019-07-15 19:25 ` [PATCH v3 2/5] locking/qspinlock: Refactor the qspinlock slow path Alex Kogan
2019-07-16 10:20 ` Peter Zijlstra
2019-07-16 14:53 ` Alex Kogan
2019-07-16 15:58 ` Peter Zijlstra
2019-07-15 19:25 ` [PATCH v3 3/5] locking/qspinlock: Introduce CNA into the slow path of qspinlock Alex Kogan
2019-07-15 21:30 ` Waiman Long
2019-07-16 11:04 ` Peter Zijlstra
2019-07-16 14:26 ` Alex Kogan
2019-07-16 14:44 ` Waiman Long
[not found] ` <aa73b86d-902a-bb6f-d372-8645c8299a6d@redhat.com>
[not found] ` <C1C55A40-FDB1-43B5-B551-F9B8BE776DF8@oracle.com>
2019-07-16 14:50 ` Waiman Long
2019-07-17 17:44 ` Alex Kogan
2019-07-17 17:58 ` Waiman Long
2019-07-16 11:05 ` Peter Zijlstra
2019-07-16 14:30 ` Alex Kogan
2019-07-16 15:50 ` Peter Zijlstra
2019-07-16 17:19 ` Alex Kogan
2019-07-16 18:47 ` Peter Zijlstra
2019-07-17 8:39 ` Peter Zijlstra
2019-07-17 8:59 ` Peter Zijlstra
2019-07-17 14:52 ` Alex Kogan
[not found] ` <FFC2D45A-24B3-40E1-ABBB-1D696E830B23@oracle.com>
2019-07-17 15:09 ` Peter Zijlstra
2019-07-17 2:16 ` Waiman Long [this message]
2019-07-17 7:44 ` Peter Zijlstra
2019-07-17 13:35 ` Waiman Long
2019-07-17 14:42 ` Alex Kogan
2019-07-15 19:25 ` [PATCH v3 4/5] locking/qspinlock: Introduce starvation avoidance into CNA Alex Kogan
2019-07-16 15:59 ` Peter Zijlstra
2019-07-15 19:25 ` [PATCH v3 5/5] locking/qspinlock: Introduce the shuffle reduction optimization " Alex Kogan
2019-07-16 11:47 ` [PATCH v3 0/5] Add NUMA-awareness to qspinlock Nicholas Piggin
[not found] ` <7D29555E-8F72-4EDD-8A87-B1A59C3945A6@oracle.com>
2019-07-16 23:07 ` Nicholas Piggin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9fa54e98-0b9b-0931-db32-c6bd6ccfe75b@redhat.com \
--to=longman@redhat.com \
--cc=alex.kogan@oracle.com \
--cc=arnd@arndb.de \
--cc=bp@alien8.de \
--cc=daniel.m.jordan@oracle.com \
--cc=dave.dice@oracle.com \
--cc=guohanjun@huawei.com \
--cc=hpa@zytor.com \
--cc=jglauber@marvell.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux@armlinux.org.uk \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rahul.x.yadav@oracle.com \
--cc=steven.sistare@oracle.com \
--cc=tglx@linutronix.de \
--cc=will.deacon@arm.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).