From: Dexuan Cui <decui@microsoft.com>
To: Thomas Gleixner <tglx@linutronix.de>,
Ming Lei <ming.lei@redhat.com>, Christoph Hellwig <hch@lst.de>,
Christian Borntraeger <borntraeger@de.ibm.com>,
Stefan Haberland <sth@linux.vnet.ibm.com>,
Jens Axboe <axboe@kernel.dk>, Marc Zyngier <marc.zyngier@arm.com>,
"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Cc: Long Li <longli@microsoft.com>,
Haiyang Zhang <haiyangz@microsoft.com>,
Michael Kelley <mikelley@microsoft.com>
Subject: RE: irq_build_affinity_masks() allocates improper affinity if num_possible_cpus() > num_present_cpus()?
Date: Wed, 7 Oct 2020 03:08:09 +0000 [thread overview]
Message-ID: <KU1P153MB0120B72473E50B1B723E25E6BF0A0@KU1P153MB0120.APCP153.PROD.OUTLOOK.COM> (raw)
In-Reply-To: <87lfgj6v30.fsf@nanos.tec.linutronix.de>
> From: Thomas Gleixner <tglx@linutronix.de>
> Sent: Tuesday, October 6, 2020 11:58 AM
> > ...
> > I pass through an MSI-X-capable PCI device to the Linux VM (which has
> > only 1 virtual CPU), and the below code does *not* report any error
> > (i.e. pci_alloc_irq_vectors_affinity() returns 2, and request_irq()
> > returns 0), but the code does not work: the second MSI-X interrupt is not
> > happening while the first interrupt does work fine.
> >
> > int nr_irqs = 2;
> > int i, nvec, irq;
> >
> > nvec = pci_alloc_irq_vectors_affinity(pdev, nr_irqs, nr_irqs,
> > PCI_IRQ_MSIX | PCI_IRQ_AFFINITY, NULL);
>
> Why should it return an error?
The above code returns -ENOSPC if num_possible_cpus() is also 1, and
returns 0 if num_possible_cpus() is 128. So it looks the above code is
not using the API correctly, and hence gets undefined results.
> > for (i = 0; i < nvec; i++) {
> > irq = pci_irq_vector(pdev, i);
> > err = request_irq(irq, test_intr, 0, "test_intr", &intr_cxt[i]);
> > }
>
> And why do you expect that the second interrupt works?
>
> This is about managed interrupts and the spreading code has two vectors
> to which it can spread the interrupts. One is assigned to one half of
> the possible CPUs and the other one to the other half. Now you have only
> one CPU online so only the interrupt with has the online CPU in the
> assigned affinity mask is started up.
>
> That's how managed interrupts work. If you don't want managed interrupts
> then don't use them.
>
> Thanks,
>
> tglx
Thanks for the clarification! It looks with PCI_IRQ_AFFINITY the kernel
guarantees that the allocated interrutps are 1:1 bound to CPUs, and the
userspace is unable to change the affinities. This is very useful to support
per-CPU I/O queues.
Thanks,
-- Dexuan
prev parent reply other threads:[~2020-10-07 3:08 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-06 6:47 irq_build_affinity_masks() allocates improper affinity if num_possible_cpus() > num_present_cpus()? Dexuan Cui
2020-10-06 8:37 ` David Woodhouse
2020-10-06 11:17 ` David Woodhouse
2020-10-06 19:00 ` Thomas Gleixner
2020-10-06 18:57 ` Thomas Gleixner
2020-10-07 3:08 ` Dexuan Cui [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=KU1P153MB0120B72473E50B1B723E25E6BF0A0@KU1P153MB0120.APCP153.PROD.OUTLOOK.COM \
--to=decui@microsoft.com \
--cc=axboe@kernel.dk \
--cc=borntraeger@de.ibm.com \
--cc=haiyangz@microsoft.com \
--cc=hch@lst.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=longli@microsoft.com \
--cc=marc.zyngier@arm.com \
--cc=mikelley@microsoft.com \
--cc=ming.lei@redhat.com \
--cc=sth@linux.vnet.ibm.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).