[RFC] nvme-pci does not disable interrupt-source in threaded mode

[RFC] nvme-pci does not disable interrupt-source in threaded mode

[Sebastian Andrzej Siewior]

From drivers/nvme/host/pci.c:
| static int queue_request_irq(struct nvme_queue *nvmeq)
| {
|         struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
|         int nr = nvmeq->dev->ctrl.instance;
| 
|         if (use_threaded_interrupts) {
|                 return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
|                                 nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);

nvme_irq_check() checks if an interrupt is pending and does not disable
interrupt source if the interrupt is pending. It simply wakes the thread.
This works because the MSI-IRQ-controller is handled as edge and it does
not re-trigger. Should the IRQ fallback to non-MSI it would become LEVEL
and will keep the IRQ-line asserted so the threaded handler will no be reached.

One of these should be done:
- remove "use_threaded_interrupts" and rely simply on the "threadirqs"
  kernel command line.
- Let nvme_irq_check() disable the IRQ-source and nvme_irq() should
  enable it again once done.
- Pass IRQF_ONESHOT as irqflags.

Side note:
 msix_request_rcd_irq() (drivers/infiniband/hw/hfi1/msix.c) provides also
 both handlers and (according to the comment) receive_context_interrupt()
 disables the source before waking the thread. This one is fine and I
 did not find more users of pci_request_irq() providing both handlers.

Sebastian

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [RFC] nvme-pci does not disable interrupt-source in threaded mode

[Keith Busch]

On Tue, Nov 26, 2019 at 09:28:14PM +0100, Sebastian Andrzej Siewior wrote:
> One of these should be done:
> - remove "use_threaded_interrupts" and rely simply on the "threadirqs"
>   kernel command line.
> - Let nvme_irq_check() disable the IRQ-source and nvme_irq() should
>   enable it again once done.

There is an nvme specific way to mask interrupts really quickly, but it
only works for MSI. Can't use it for MSIx...

> - Pass IRQF_ONESHOT as irqflags.

IRQF_ONESHOT adds two non-posted transactions through pci_msi_mask_irq()
and pci_msi_unmask_irq() on each interupt. That's pretty slow.

Maybe we could have the nvme driver provide its own MSIx mask routine that
skips the readl(). That might mean local interrupts get renabled before
the device sees the MSIx mask set, but maybe that's ok?

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme