Re: [PATCH 2/2] nvme pci: try to allocate multiple irq vectors again in case of -EINVAL

Re: [PATCH 2/2] nvme pci: try to allocate multiple irq vectors again in case of -EINVAL

[Ming Lei]

On Wed, Dec 26, 2018 at 07:20:27PM +0100, Christoph Hellwig wrote:
> On Wed, Dec 26, 2018 at 06:37:55PM +0800, Ming Lei wrote:
> > It is observed on QEMU that pci_alloc_irq_vectors_affinity() may
> > returns -EINVAL when the requested number is too big(such as 64).
> 
> Which is not how this API is supposed to work and documented to work.
> 
> We need to fix pci_alloc_irq_vectors_affinity to not return a spurious
> error and just return the allocated number of vectors instead of
> hacking around that in drivers.

Yeah, you are right.

The issue is that QEMU nvme-pci is MSIX-capable only, and hasn't MSI
capability.

__pci_enable_msix_range() actually returns -ENOSPC, but __pci_enable_msi_range()
returns -EINVAL because dev->msi_cap is zero.

Maybe we need the following fix?

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 265ed3e4c920..b0bf260dc154 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -1186,7 +1186,7 @@ int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs,
                        return vecs;
        }
 
-       if (flags & PCI_IRQ_MSI) {
+       if ((flags & PCI_IRQ_MSI) && dev->msi_cap) {
                vecs = __pci_enable_msi_range(dev, min_vecs, max_vecs, affd);
                if (vecs > 0)
                        return vecs;


Thanks,
Ming

Re: [PATCH 2/2] nvme pci: try to allocate multiple irq vectors again in case of -EINVAL

[Christoph Hellwig]

On Thu, Dec 27, 2018 at 04:21:38PM +0800, Ming Lei wrote:
> On Wed, Dec 26, 2018 at 07:20:27PM +0100, Christoph Hellwig wrote:
> > On Wed, Dec 26, 2018 at 06:37:55PM +0800, Ming Lei wrote:
> > > It is observed on QEMU that pci_alloc_irq_vectors_affinity() may
> > > returns -EINVAL when the requested number is too big(such as 64).
> > 
> > Which is not how this API is supposed to work and documented to work.
> > 
> > We need to fix pci_alloc_irq_vectors_affinity to not return a spurious
> > error and just return the allocated number of vectors instead of
> > hacking around that in drivers.
> 
> Yeah, you are right.
> 
> The issue is that QEMU nvme-pci is MSIX-capable only, and hasn't MSI
> capability.
> 
> __pci_enable_msix_range() actually returns -ENOSPC, but __pci_enable_msi_range()
> returns -EINVAL because dev->msi_cap is zero.
> 
> Maybe we need the following fix?

Should it matter?  We still get a negative vecs back, and still fall
back to the next option.  Unless ther are no irqs available at all
for the selected types pci_alloc_irq_vectors_affinity should never
return an error.

Re: [PATCH 2/2] nvme pci: try to allocate multiple irq vectors again in case of -EINVAL

[Ming Lei]

On Thu, Dec 27, 2018 at 02:08:34PM +0100, Christoph Hellwig wrote:
> On Thu, Dec 27, 2018 at 04:21:38PM +0800, Ming Lei wrote:
> > On Wed, Dec 26, 2018 at 07:20:27PM +0100, Christoph Hellwig wrote:
> > > On Wed, Dec 26, 2018 at 06:37:55PM +0800, Ming Lei wrote:
> > > > It is observed on QEMU that pci_alloc_irq_vectors_affinity() may
> > > > returns -EINVAL when the requested number is too big(such as 64).
> > > 
> > > Which is not how this API is supposed to work and documented to work.
> > > 
> > > We need to fix pci_alloc_irq_vectors_affinity to not return a spurious
> > > error and just return the allocated number of vectors instead of
> > > hacking around that in drivers.
> > 
> > Yeah, you are right.
> > 
> > The issue is that QEMU nvme-pci is MSIX-capable only, and hasn't MSI
> > capability.
> > 
> > __pci_enable_msix_range() actually returns -ENOSPC, but __pci_enable_msi_range()
> > returns -EINVAL because dev->msi_cap is zero.
> > 
> > Maybe we need the following fix?
> 
> Should it matter?  We still get a negative vecs back, and still fall
> back to the next option.  Unless ther are no irqs available at all
> for the selected types pci_alloc_irq_vectors_affinity should never
> return an error.

The patch in last email does fix this issue.

In this case, the number of NVMe PCI's MSI-X table entries is 64, so
__pci_enable_msix_range() return -ENOSPC when we ask for 65.

However, the following __pci_enable_msi_range() returns -EINVAL because
the NVMe PCI isn't capable of MSI, then this error is returned from
pci_alloc_irq_vectors_affinity() finally to NVMe driver.

Of course, -EINVAL makes a difference because the current code only
tries to assign one irq vector in this case, and it shouldn't be
returned from pci_alloc_irq_vectors_affinity(), given there is enough
msix entries for fallback, right?

Thanks,
Ming

Re: [PATCH 2/2] nvme pci: try to allocate multiple irq vectors again in case of -EINVAL

[Bjorn Helgaas]

On Thu, Dec 27, 2018 at 02:08:34PM +0100, Christoph Hellwig wrote:
> On Thu, Dec 27, 2018 at 04:21:38PM +0800, Ming Lei wrote:
> > On Wed, Dec 26, 2018 at 07:20:27PM +0100, Christoph Hellwig wrote:
> > > On Wed, Dec 26, 2018 at 06:37:55PM +0800, Ming Lei wrote:
> > > > It is observed on QEMU that pci_alloc_irq_vectors_affinity() may
> > > > returns -EINVAL when the requested number is too big(such as 64).
> > > 
> > > Which is not how this API is supposed to work and documented to work.
> > > 
> > > We need to fix pci_alloc_irq_vectors_affinity to not return a spurious
> > > error and just return the allocated number of vectors instead of
> > > hacking around that in drivers.
> > 
> > Yeah, you are right.
> > 
> > The issue is that QEMU nvme-pci is MSIX-capable only, and hasn't MSI
> > capability.
> > 
> > __pci_enable_msix_range() actually returns -ENOSPC, but __pci_enable_msi_range()
> > returns -EINVAL because dev->msi_cap is zero.
> > 
> > Maybe we need the following fix?
> 
> Should it matter?  We still get a negative vecs back, and still fall
> back to the next option.  

I'm not sure how it matters either, since
pci_alloc_irq_vectors_affinity() will fail either way.  It *would* be
nice to return the correct error in case the caller uses it to emit a
message.

But if the caller wants to use -ENOSPC to reduce @min_vecs and try
again, that sounds like an incorrect use of the interface -- the
caller should have just used the smaller @min_vecs the first time
around.

> Unless ther are no irqs available at all
> for the selected types pci_alloc_irq_vectors_affinity should never
> return an error.

I don't quite understand this last sentence.  If @min_vecs == 5 and
the device only supports 4 MSI-X and 4 MSI vectors, the function
comment says we should fail with -ENOSPC.

Bjorn