From: Alexander Gordeev <agordeev@redhat.com>
To: Christoph Hellwig <hch@lst.de>
Cc: tglx@linutronix.de, axboe@fb.com, linux-block@vger.kernel.org,
linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH 08/13] pci: spread interrupt vectors in pci_alloc_irq_vectors
Date: Sat, 25 Jun 2016 22:22:49 +0200 [thread overview]
Message-ID: <20160625202249.GB29251@dhcp-27-118.brq.redhat.com> (raw)
In-Reply-To: <1465934346-20648-9-git-send-email-hch@lst.de>
On Tue, Jun 14, 2016 at 09:59:01PM +0200, Christoph Hellwig wrote:
> Set the affinity_mask before allocating vectors.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> drivers/pci/msi.c | 26 ++++++++++++++++++++++++--
> include/linux/pci.h | 1 +
> 2 files changed, 25 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> index a33adec..50d694c 100644
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -568,6 +568,7 @@ static struct msi_desc *msi_setup_entry(struct pci_dev *dev, int nvec)
> entry->msi_attrib.multi_cap = (control & PCI_MSI_FLAGS_QMASK) >> 1;
> entry->msi_attrib.multiple = ilog2(__roundup_pow_of_two(nvec));
> entry->nvec_used = nvec;
> + entry->affinity = dev->irq_affinity;
>
> if (control & PCI_MSI_FLAGS_64BIT)
> entry->mask_pos = dev->msi_cap + PCI_MSI_MASK_64;
> @@ -679,10 +680,18 @@ static void __iomem *msix_map_region(struct pci_dev *dev, unsigned nr_entries)
> static int msix_setup_entries(struct pci_dev *dev, void __iomem *base,
> struct msix_entry *entries, int nvec)
> {
> + const struct cpumask *mask = NULL;
> struct msi_desc *entry;
> - int i;
> + int cpu = -1, i;
>
> for (i = 0; i < nvec; i++) {
> + if (dev->irq_affinity) {
> + cpu = cpumask_next(cpu, dev->irq_affinity);
> + if (cpu >= nr_cpu_ids)
> + cpu = cpumask_first(dev->irq_affinity);
> + mask = cpumask_of(cpu);
> + }
> +
> entry = alloc_msi_entry(&dev->dev);
> if (!entry) {
> if (!i)
> @@ -699,6 +708,7 @@ static int msix_setup_entries(struct pci_dev *dev, void __iomem *base,
> entry->msi_attrib.default_irq = dev->irq;
> entry->mask_base = base;
> entry->nvec_used = 1;
> + entry->affinity = mask;
>
> list_add_tail(&entry->list, dev_to_msi_list(&dev->dev));
> }
> @@ -1176,12 +1186,20 @@ int pci_alloc_irq_vectors(struct pci_dev *dev, unsigned int min_vecs,
> {
> unsigned int vecs, i;
> u32 *irqs;
> + int ret;
>
> max_vecs = min(max_vecs, pci_nr_irq_vectors(dev));
>
> + ret = irq_create_affinity_mask(&dev->irq_affinity, &max_vecs);
dev->irq_affinity = irq_create_affinity_mask(&max_vecs); ?
> + if (ret)
> + return ret;
> + if (max_vecs < min_vecs)
> + return -ENOSPC;
irq_create_affinity_mask() should be called after MSI-X/MSI is enabled,
because we do not know number of vectors before the range functions
returned that number.
Since affinity masks is a function of number of vectors and CPU topology
the resulting masks might turn out suboptimal in general case (and
this code supposed to be general, right?).
I.e irq_create_affinity_mask() could decide "per-first-sibling" spreading
given number of available vectors, but only a subset of MSI vectors
were actually allocated. For that subset "per-core" affinity mask could
have been initialized, but we will still go with "per-first-sibling".
> + ret = -ENOMEM;
> irqs = kcalloc(max_vecs, sizeof(u32), GFP_KERNEL);
> if (!irqs)
> - return -ENOMEM;
> + goto out_free_affinity;
>
> if (!(flags & PCI_IRQ_NOMSIX)) {
> vecs = pci_enable_msix_range_wrapper(dev, irqs, min_vecs,
> @@ -1208,6 +1226,10 @@ int pci_alloc_irq_vectors(struct pci_dev *dev, unsigned int min_vecs,
> done:
> dev->irqs = irqs;
> return vecs;
> +out_free_affinity:
> + kfree(dev->irq_affinity);
> + dev->irq_affinity = NULL;
> + return ret;
> }
> EXPORT_SYMBOL(pci_alloc_irq_vectors);
>
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 84a20fc..f474611 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -321,6 +321,7 @@ struct pci_dev {
> */
> unsigned int irq;
> unsigned int *irqs;
> + struct cpumask *irq_affinity;
> struct resource resource[DEVICE_COUNT_RESOURCE]; /* I/O and memory regions + expansion ROMs */
>
> bool match_driver; /* Skip attaching driver */
> --
> 2.1.4
>
next prev parent reply other threads:[~2016-06-25 20:22 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-14 19:58 automatic interrupt affinity for MSI/MSI-X capable devices V2 Christoph Hellwig
2016-06-14 19:58 ` [PATCH 01/13] irq/msi: Remove unused MSI_FLAG_IDENTITY_MAP Christoph Hellwig
2016-06-16 9:05 ` Bart Van Assche
2016-06-14 19:58 ` [PATCH 02/13] irq: Introduce IRQD_AFFINITY_MANAGED flag Christoph Hellwig
2016-06-15 8:44 ` Bart Van Assche
2016-06-15 10:23 ` Christoph Hellwig
2016-06-15 10:42 ` Bart Van Assche
2016-06-15 15:14 ` Keith Busch
2016-06-15 15:28 ` Bart Van Assche
2016-06-15 16:03 ` Keith Busch
2016-06-15 19:36 ` Bart Van Assche
2016-06-15 20:06 ` Keith Busch
2016-06-15 20:12 ` Keith Busch
2016-06-15 20:50 ` Bart Van Assche
2016-06-16 15:19 ` Keith Busch
2016-06-22 11:56 ` Alexander Gordeev
2016-06-16 15:20 ` Christoph Hellwig
2016-06-16 15:39 ` Bart Van Assche
2016-06-20 12:22 ` Christoph Hellwig
2016-06-20 13:21 ` Bart Van Assche
2016-06-21 14:31 ` Christoph Hellwig
2016-06-16 9:08 ` Bart Van Assche
2016-06-14 19:58 ` [PATCH 03/13] irq: Add affinity hint to irq allocation Christoph Hellwig
2016-06-14 19:58 ` [PATCH 04/13] irq: Use affinity hint in irqdesc allocation Christoph Hellwig
2016-06-14 19:58 ` [PATCH 05/13] irq/msi: Make use of affinity aware allocations Christoph Hellwig
2016-06-14 19:58 ` [PATCH 06/13] irq: add a helper spread an affinity mask for MSI/MSI-X vectors Christoph Hellwig
2016-06-14 21:54 ` Guilherme G. Piccoli
2016-06-15 8:35 ` Bart Van Assche
2016-06-15 10:10 ` Christoph Hellwig
2016-06-15 13:09 ` Guilherme G. Piccoli
2016-06-16 15:16 ` Christoph Hellwig
2016-06-25 20:05 ` Alexander Gordeev
2016-06-30 17:48 ` Christoph Hellwig
2016-07-01 7:25 ` Alexander Gordeev
2016-06-14 19:59 ` [PATCH 07/13] pci: Provide sensible irq vector alloc/free routines Christoph Hellwig
2016-06-23 11:16 ` Alexander Gordeev
2016-06-30 16:54 ` Christoph Hellwig
2016-06-30 17:28 ` Alexander Gordeev
2016-06-30 17:35 ` Christoph Hellwig
2016-06-14 19:59 ` [PATCH 08/13] pci: spread interrupt vectors in pci_alloc_irq_vectors Christoph Hellwig
2016-06-25 20:22 ` Alexander Gordeev [this message]
2016-06-14 19:59 ` [PATCH 09/13] blk-mq: don't redistribute hardware queues on a CPU hotplug event Christoph Hellwig
2016-06-14 19:59 ` [PATCH 10/13] blk-mq: only allocate a single mq_map per tag_set Christoph Hellwig
2016-06-14 19:59 ` [PATCH 11/13] blk-mq: allow the driver to pass in an affinity mask Christoph Hellwig
2016-07-04 8:15 ` Alexander Gordeev
2016-07-04 8:38 ` Christoph Hellwig
2016-07-04 9:35 ` Alexander Gordeev
2016-07-10 3:41 ` Christoph Hellwig
2016-07-12 6:42 ` Alexander Gordeev
2016-06-14 19:59 ` [PATCH 12/13] nvme: switch to use pci_alloc_irq_vectors Christoph Hellwig
2016-06-14 19:59 ` [PATCH 13/13] nvme: remove the post_scan callout Christoph Hellwig
2016-06-16 9:45 ` automatic interrupt affinity for MSI/MSI-X capable devices V2 Bart Van Assche
2016-06-16 15:22 ` Christoph Hellwig
2016-06-26 19:40 ` Alexander Gordeev
2016-07-04 8:39 automatic interrupt affinity for MSI/MSI-X capable devices V3 Christoph Hellwig
2016-07-04 8:39 ` [PATCH 08/13] pci: spread interrupt vectors in pci_alloc_irq_vectors Christoph Hellwig
2016-07-07 11:05 ` Alexander Gordeev
2016-07-10 3:57 ` Christoph Hellwig
2016-07-12 6:49 ` Alexander Gordeev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160625202249.GB29251@dhcp-27-118.brq.redhat.com \
--to=agordeev@redhat.com \
--cc=axboe@fb.com \
--cc=hch@lst.de \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).