From: Alex Kogan <alex.kogan@oracle.com> To: linux@armlinux.org.uk, peterz@infradead.org, mingo@redhat.com, will.deacon@arm.com, arnd@arndb.de, longman@redhat.com, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, tglx@linutronix.de, bp@alien8.de, hpa@zytor.com, x86@kernel.org, guohanjun@huawei.com, jglauber@marvell.com Cc: steven.sistare@oracle.com, daniel.m.jordan@oracle.com, alex.kogan@oracle.com, dave.dice@oracle.com Subject: [PATCH v12 5/5] locking/qspinlock: Avoid moving certain threads between waiting queues in CNA Date: Tue, 17 Nov 2020 18:13:23 -0500 [thread overview] Message-ID: <20201117231323.797104-6-alex.kogan@oracle.com> (raw) In-Reply-To: <20201117231323.797104-1-alex.kogan@oracle.com> Prohibit moving certain threads (e.g., in irq and nmi contexts) to the secondary queue. Those prioritized threads will always stay in the primary queue, and so will have a shorter wait time for the lock. Signed-off-by: Alex Kogan <alex.kogan@oracle.com> Reviewed-by: Steve Sistare <steven.sistare@oracle.com> Reviewed-by: Waiman Long <longman@redhat.com> --- kernel/locking/qspinlock_cna.h | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/kernel/locking/qspinlock_cna.h b/kernel/locking/qspinlock_cna.h index d3e2754..ac3109a 100644 --- a/kernel/locking/qspinlock_cna.h +++ b/kernel/locking/qspinlock_cna.h @@ -4,6 +4,7 @@ #endif #include <linux/topology.h> +#include <linux/sched/rt.h> /* * Implement a NUMA-aware version of MCS (aka CNA, or compact NUMA-aware lock). @@ -35,7 +36,8 @@ * running on the same NUMA node. If it is not, that waiter is detached from the * main queue and moved into the tail of the secondary queue. This way, we * gradually filter the primary queue, leaving only waiters running on the same - * preferred NUMA node. + * preferred NUMA node. Note that certain priortized waiters (e.g., in + * irq and nmi contexts) are excluded from being moved to the secondary queue. * * We change the NUMA node preference after a waiter at the head of the * secondary queue spins for a certain amount of time (10ms, by default). @@ -49,6 +51,8 @@ * Dave Dice <dave.dice@oracle.com> */ +#define CNA_PRIORITY_NODE 0xffff + struct cna_node { struct mcs_spinlock mcs; u16 numa_node; @@ -121,9 +125,10 @@ static int __init cna_init_nodes(void) static __always_inline void cna_init_node(struct mcs_spinlock *node) { + bool priority = !in_task() || irqs_disabled() || rt_task(current); struct cna_node *cn = (struct cna_node *)node; - cn->numa_node = cn->real_numa_node; + cn->numa_node = priority ? CNA_PRIORITY_NODE : cn->real_numa_node; cn->start_time = 0; } @@ -262,11 +267,13 @@ static u32 cna_order_queue(struct mcs_spinlock *node) next_numa_node = ((struct cna_node *)next)->numa_node; if (next_numa_node != numa_node) { - struct mcs_spinlock *nnext = READ_ONCE(next->next); + if (next_numa_node != CNA_PRIORITY_NODE) { + struct mcs_spinlock *nnext = READ_ONCE(next->next); - if (nnext) { - cna_splice_next(node, next, nnext); - next = nnext; + if (nnext) { + cna_splice_next(node, next, nnext); + next = nnext; + } } /* * Inherit NUMA node id of primary queue, to maintain the @@ -285,6 +292,13 @@ static __always_inline u32 cna_wait_head_or_lock(struct qspinlock *lock, if (!cn->start_time || !intra_node_threshold_reached(cn)) { /* + * We are at the head of the wait queue, no need to use + * the fake NUMA node ID. + */ + if (cn->numa_node == CNA_PRIORITY_NODE) + cn->numa_node = cn->real_numa_node; + + /* * Try and put the time otherwise spent spin waiting on * _Q_LOCKED_PENDING_MASK to use by sorting our lists. */ -- 2.7.4
WARNING: multiple messages have this Message-ID (diff)
From: Alex Kogan <alex.kogan@oracle.com> To: linux@armlinux.org.uk, peterz@infradead.org, mingo@redhat.com, will.deacon@arm.com, arnd@arndb.de, longman@redhat.com, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, tglx@linutronix.de, bp@alien8.de, hpa@zytor.com, x86@kernel.org, guohanjun@huawei.com, jglauber@marvell.com Cc: alex.kogan@oracle.com, dave.dice@oracle.com, steven.sistare@oracle.com, daniel.m.jordan@oracle.com Subject: [PATCH v12 5/5] locking/qspinlock: Avoid moving certain threads between waiting queues in CNA Date: Tue, 17 Nov 2020 18:13:23 -0500 [thread overview] Message-ID: <20201117231323.797104-6-alex.kogan@oracle.com> (raw) In-Reply-To: <20201117231323.797104-1-alex.kogan@oracle.com> Prohibit moving certain threads (e.g., in irq and nmi contexts) to the secondary queue. Those prioritized threads will always stay in the primary queue, and so will have a shorter wait time for the lock. Signed-off-by: Alex Kogan <alex.kogan@oracle.com> Reviewed-by: Steve Sistare <steven.sistare@oracle.com> Reviewed-by: Waiman Long <longman@redhat.com> --- kernel/locking/qspinlock_cna.h | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/kernel/locking/qspinlock_cna.h b/kernel/locking/qspinlock_cna.h index d3e2754..ac3109a 100644 --- a/kernel/locking/qspinlock_cna.h +++ b/kernel/locking/qspinlock_cna.h @@ -4,6 +4,7 @@ #endif #include <linux/topology.h> +#include <linux/sched/rt.h> /* * Implement a NUMA-aware version of MCS (aka CNA, or compact NUMA-aware lock). @@ -35,7 +36,8 @@ * running on the same NUMA node. If it is not, that waiter is detached from the * main queue and moved into the tail of the secondary queue. This way, we * gradually filter the primary queue, leaving only waiters running on the same - * preferred NUMA node. + * preferred NUMA node. Note that certain priortized waiters (e.g., in + * irq and nmi contexts) are excluded from being moved to the secondary queue. * * We change the NUMA node preference after a waiter at the head of the * secondary queue spins for a certain amount of time (10ms, by default). @@ -49,6 +51,8 @@ * Dave Dice <dave.dice@oracle.com> */ +#define CNA_PRIORITY_NODE 0xffff + struct cna_node { struct mcs_spinlock mcs; u16 numa_node; @@ -121,9 +125,10 @@ static int __init cna_init_nodes(void) static __always_inline void cna_init_node(struct mcs_spinlock *node) { + bool priority = !in_task() || irqs_disabled() || rt_task(current); struct cna_node *cn = (struct cna_node *)node; - cn->numa_node = cn->real_numa_node; + cn->numa_node = priority ? CNA_PRIORITY_NODE : cn->real_numa_node; cn->start_time = 0; } @@ -262,11 +267,13 @@ static u32 cna_order_queue(struct mcs_spinlock *node) next_numa_node = ((struct cna_node *)next)->numa_node; if (next_numa_node != numa_node) { - struct mcs_spinlock *nnext = READ_ONCE(next->next); + if (next_numa_node != CNA_PRIORITY_NODE) { + struct mcs_spinlock *nnext = READ_ONCE(next->next); - if (nnext) { - cna_splice_next(node, next, nnext); - next = nnext; + if (nnext) { + cna_splice_next(node, next, nnext); + next = nnext; + } } /* * Inherit NUMA node id of primary queue, to maintain the @@ -285,6 +292,13 @@ static __always_inline u32 cna_wait_head_or_lock(struct qspinlock *lock, if (!cn->start_time || !intra_node_threshold_reached(cn)) { /* + * We are at the head of the wait queue, no need to use + * the fake NUMA node ID. + */ + if (cn->numa_node == CNA_PRIORITY_NODE) + cn->numa_node = cn->real_numa_node; + + /* * Try and put the time otherwise spent spin waiting on * _Q_LOCKED_PENDING_MASK to use by sorting our lists. */ -- 2.7.4 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2020-11-17 23:21 UTC|newest] Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-11-17 23:13 [PATCH v12 0/5] Add NUMA-awareness to qspinlock Alex Kogan 2020-11-17 23:13 ` Alex Kogan 2020-11-17 23:13 ` [PATCH v12 1/5] locking/qspinlock: Rename mcs lock/unlock macros and make them more generic Alex Kogan 2020-11-17 23:13 ` Alex Kogan 2020-11-17 23:13 ` [PATCH v12 2/5] locking/qspinlock: Refactor the qspinlock slow path Alex Kogan 2020-11-17 23:13 ` Alex Kogan 2020-11-17 23:13 ` [PATCH v12 3/5] locking/qspinlock: Introduce CNA into the slow path of qspinlock Alex Kogan 2020-11-17 23:13 ` Alex Kogan 2020-11-22 7:38 ` [locking/qspinlock] 0dd6d5b8c0: vm-scalability.throughput 102.9% improvement kernel test robot 2020-11-17 23:13 ` [PATCH v12 4/5] locking/qspinlock: Introduce starvation avoidance into CNA Alex Kogan 2020-11-17 23:13 ` Alex Kogan 2020-11-17 23:13 ` Alex Kogan [this message] 2020-11-17 23:13 ` [PATCH v12 5/5] locking/qspinlock: Avoid moving certain threads between waiting queues in CNA Alex Kogan 2020-11-22 9:47 ` [locking/qspinlock] 6f9a39a437: unixbench.score -17.3% regression kernel test robot
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20201117231323.797104-6-alex.kogan@oracle.com \ --to=alex.kogan@oracle.com \ --cc=arnd@arndb.de \ --cc=bp@alien8.de \ --cc=daniel.m.jordan@oracle.com \ --cc=dave.dice@oracle.com \ --cc=guohanjun@huawei.com \ --cc=hpa@zytor.com \ --cc=jglauber@marvell.com \ --cc=linux-arch@vger.kernel.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux@armlinux.org.uk \ --cc=longman@redhat.com \ --cc=mingo@redhat.com \ --cc=peterz@infradead.org \ --cc=steven.sistare@oracle.com \ --cc=tglx@linutronix.de \ --cc=will.deacon@arm.com \ --cc=x86@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.