From: Alexander Gordeev <agordeev@redhat.com>
To: Christoph Hellwig <hch@lst.de>
Cc: tglx@linutronix.de, axboe@fb.com, linux-block@vger.kernel.org,
linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH 06/13] irq: add a helper spread an affinity mask for MSI/MSI-X vectors
Date: Sat, 25 Jun 2016 22:05:19 +0200 [thread overview]
Message-ID: <20160625200518.GA29251@dhcp-27-118.brq.redhat.com> (raw)
In-Reply-To: <1465934346-20648-7-git-send-email-hch@lst.de>
On Tue, Jun 14, 2016 at 09:58:59PM +0200, Christoph Hellwig wrote:
> This is lifted from the blk-mq code and adopted to use the affinity mask
> concept just intruced in the irq handling code.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> include/linux/interrupt.h | 11 +++++++++
> kernel/irq/Makefile | 1 +
> kernel/irq/affinity.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 72 insertions(+)
> create mode 100644 kernel/irq/affinity.c
>
> diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h
> index 9fcabeb..12003c0 100644
> --- a/include/linux/interrupt.h
> +++ b/include/linux/interrupt.h
> @@ -278,6 +278,9 @@ extern int irq_set_affinity_hint(unsigned int irq, const struct cpumask *m);
> extern int
> irq_set_affinity_notifier(unsigned int irq, struct irq_affinity_notify *notify);
>
> +int irq_create_affinity_mask(struct cpumask **affinity_mask,
> + unsigned int *nr_vecs);
> +
> #else /* CONFIG_SMP */
>
> static inline int irq_set_affinity(unsigned int irq, const struct cpumask *m)
> @@ -308,6 +311,14 @@ irq_set_affinity_notifier(unsigned int irq, struct irq_affinity_notify *notify)
> {
> return 0;
> }
> +
> +static inline int irq_create_affinity_mask(struct cpumask **affinity_mask,
> + unsigned int *nr_vecs)
> +{
> + *affinity_mask = NULL;
> + *nr_vecs = 1;
> + return 0;
> +}
> #endif /* CONFIG_SMP */
>
> /*
> diff --git a/kernel/irq/Makefile b/kernel/irq/Makefile
> index 2ee42e9..1d3ee31 100644
> --- a/kernel/irq/Makefile
> +++ b/kernel/irq/Makefile
> @@ -9,3 +9,4 @@ obj-$(CONFIG_GENERIC_IRQ_MIGRATION) += cpuhotplug.o
> obj-$(CONFIG_PM_SLEEP) += pm.o
> obj-$(CONFIG_GENERIC_MSI_IRQ) += msi.o
> obj-$(CONFIG_GENERIC_IRQ_IPI) += ipi.o
> +obj-$(CONFIG_SMP) += affinity.o
> diff --git a/kernel/irq/affinity.c b/kernel/irq/affinity.c
> new file mode 100644
> index 0000000..1daf8fb
> --- /dev/null
> +++ b/kernel/irq/affinity.c
> @@ -0,0 +1,60 @@
> +
> +#include <linux/interrupt.h>
> +#include <linux/kernel.h>
> +#include <linux/slab.h>
> +#include <linux/cpu.h>
> +
> +static int get_first_sibling(unsigned int cpu)
> +{
> + unsigned int ret;
> +
> + ret = cpumask_first(topology_sibling_cpumask(cpu));
> + if (ret < nr_cpu_ids)
> + return ret;
> + return cpu;
> +}
> +
> +/*
> + * Take a map of online CPUs and the number of available interrupt vectors
> + * and generate an output cpumask suitable for spreading MSI/MSI-X vectors
> + * so that they are distributed as good as possible around the CPUs. If
> + * more vectors than CPUs are available we'll map one to each CPU,
Unless I do not misinterpret a loop from msix_setup_entries() (patch 08/13),
the above is incorrect:
for (i = 0; i < nvec; i++) {
if (dev->irq_affinity) {
cpu = cpumask_next(cpu, dev->irq_affinity);
if (cpu >= nr_cpu_ids)
cpu = cpumask_first(dev->irq_affinity);
mask = cpumask_of(cpu);
}
...
entry->affinity = mask;
}
> + * otherwise we map one to the first sibling of each socket.
(*) I guess, in some topology configurations a total number of all
first siblings may be less than the number of vectors.
> + * If there are more vectors than CPUs we will still only have one bit
> + * set per CPU, but interrupt code will keep on assining the vectors from
> + * the start of the bitmap until we run out of vectors.
> + */
> +int irq_create_affinity_mask(struct cpumask **affinity_mask,
> + unsigned int *nr_vecs)
Both the callers of this function and the function itself IMHO would
read better if it simply returned the affinity mask. Or passed the
affinity mask pointer.
> +{
> + unsigned int vecs = 0;
In case (*nr_vecs >= num_online_cpus()) the contents of *nr_vecs
will be overwritten with 0.
> + if (*nr_vecs == 1) {
> + *affinity_mask = NULL;
> + return 0;
> + }
> +
> + *affinity_mask = kzalloc(cpumask_size(), GFP_KERNEL);
> + if (!*affinity_mask)
> + return -ENOMEM;
> +
> + if (*nr_vecs >= num_online_cpus()) {
> + cpumask_copy(*affinity_mask, cpu_online_mask);
> + } else {
> + unsigned int cpu;
> +
> + for_each_online_cpu(cpu) {
> + if (cpu == get_first_sibling(cpu)) {
> + cpumask_set_cpu(cpu, *affinity_mask);
> + vecs++;
> + }
> +
> + if (--(*nr_vecs) == 0)
> + break;
> + }
> + }
> +
> + *nr_vecs = vecs;
So considering (*) comment above the number of available vectors
might be unnecessarily shrunken here.
I think nr_vecs need not be an out-parameter since we always can
assign multiple vectors to a CPU. It is better than limiting number
of available vectors AFAIKT. Or you could pass one-per-cpu flag
explicitly.
> + return 0;
> +}
> --
> 2.1.4
>
next prev parent reply other threads:[~2016-06-25 20:05 UTC|newest]
Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-14 19:58 automatic interrupt affinity for MSI/MSI-X capable devices V2 Christoph Hellwig
2016-06-14 19:58 ` [PATCH 01/13] irq/msi: Remove unused MSI_FLAG_IDENTITY_MAP Christoph Hellwig
2016-06-16 9:05 ` Bart Van Assche
2016-06-14 19:58 ` [PATCH 02/13] irq: Introduce IRQD_AFFINITY_MANAGED flag Christoph Hellwig
2016-06-15 8:44 ` Bart Van Assche
2016-06-15 10:23 ` Christoph Hellwig
2016-06-15 10:42 ` Bart Van Assche
2016-06-15 15:14 ` Keith Busch
2016-06-15 15:28 ` Bart Van Assche
2016-06-15 16:03 ` Keith Busch
2016-06-15 19:36 ` Bart Van Assche
2016-06-15 20:06 ` Keith Busch
2016-06-15 20:12 ` Keith Busch
2016-06-15 20:50 ` Bart Van Assche
2016-06-16 15:19 ` Keith Busch
2016-06-22 11:56 ` Alexander Gordeev
2016-06-16 15:20 ` Christoph Hellwig
2016-06-16 15:39 ` Bart Van Assche
2016-06-20 12:22 ` Christoph Hellwig
2016-06-20 13:21 ` Bart Van Assche
2016-06-21 14:31 ` Christoph Hellwig
2016-06-16 9:08 ` Bart Van Assche
2016-06-14 19:58 ` [PATCH 03/13] irq: Add affinity hint to irq allocation Christoph Hellwig
2016-06-14 19:58 ` [PATCH 04/13] irq: Use affinity hint in irqdesc allocation Christoph Hellwig
2016-06-14 19:58 ` [PATCH 05/13] irq/msi: Make use of affinity aware allocations Christoph Hellwig
2016-06-14 19:58 ` [PATCH 06/13] irq: add a helper spread an affinity mask for MSI/MSI-X vectors Christoph Hellwig
2016-06-14 21:54 ` Guilherme G. Piccoli
2016-06-15 8:35 ` Bart Van Assche
2016-06-15 10:10 ` Christoph Hellwig
2016-06-15 13:09 ` Guilherme G. Piccoli
2016-06-16 15:16 ` Christoph Hellwig
2016-06-25 20:05 ` Alexander Gordeev [this message]
2016-06-30 17:48 ` Christoph Hellwig
2016-07-01 7:25 ` Alexander Gordeev
2016-06-14 19:59 ` [PATCH 07/13] pci: Provide sensible irq vector alloc/free routines Christoph Hellwig
2016-06-23 11:16 ` Alexander Gordeev
2016-06-30 16:54 ` Christoph Hellwig
2016-06-30 17:28 ` Alexander Gordeev
2016-06-30 17:35 ` Christoph Hellwig
2016-06-14 19:59 ` [PATCH 08/13] pci: spread interrupt vectors in pci_alloc_irq_vectors Christoph Hellwig
2016-06-25 20:22 ` Alexander Gordeev
2016-06-14 19:59 ` [PATCH 09/13] blk-mq: don't redistribute hardware queues on a CPU hotplug event Christoph Hellwig
2016-06-14 19:59 ` [PATCH 10/13] blk-mq: only allocate a single mq_map per tag_set Christoph Hellwig
2016-06-14 19:59 ` [PATCH 11/13] blk-mq: allow the driver to pass in an affinity mask Christoph Hellwig
2016-07-04 8:15 ` Alexander Gordeev
2016-07-04 8:38 ` Christoph Hellwig
2016-07-04 9:35 ` Alexander Gordeev
2016-07-10 3:41 ` Christoph Hellwig
2016-07-12 6:42 ` Alexander Gordeev
2016-06-14 19:59 ` [PATCH 12/13] nvme: switch to use pci_alloc_irq_vectors Christoph Hellwig
2016-06-14 19:59 ` [PATCH 13/13] nvme: remove the post_scan callout Christoph Hellwig
2016-06-16 9:45 ` automatic interrupt affinity for MSI/MSI-X capable devices V2 Bart Van Assche
2016-06-16 15:22 ` Christoph Hellwig
2016-06-26 19:40 ` Alexander Gordeev
2016-07-04 8:39 automatic interrupt affinity for MSI/MSI-X capable devices V3 Christoph Hellwig
2016-07-04 8:39 ` [PATCH 06/13] irq: add a helper spread an affinity mask for MSI/MSI-X vectors Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160625200518.GA29251@dhcp-27-118.brq.redhat.com \
--to=agordeev@redhat.com \
--cc=axboe@fb.com \
--cc=hch@lst.de \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).