Linux-NVME Archive on lore.kernel.org
 help / color / Atom feed
* [RFC] nvme-pci does not disable interrupt-source in threaded mode
@ 2019-11-26 20:28 Sebastian Andrzej Siewior
  2019-11-26 22:28 ` Keith Busch
  0 siblings, 1 reply; 2+ messages in thread
From: Sebastian Andrzej Siewior @ 2019-11-26 20:28 UTC (permalink / raw)
  To: linux-nvme
  Cc: Keith Busch, Jens Axboe, Thomas Gleixner, Christoph Hellwig,
	Sagi Grimberg

From drivers/nvme/host/pci.c:
| static int queue_request_irq(struct nvme_queue *nvmeq)
| {
|         struct pci_dev *pdev = to_pci_dev(nvmeq->dev->dev);
|         int nr = nvmeq->dev->ctrl.instance;
| 
|         if (use_threaded_interrupts) {
|                 return pci_request_irq(pdev, nvmeq->cq_vector, nvme_irq_check,
|                                 nvme_irq, nvmeq, "nvme%dq%d", nr, nvmeq->qid);

nvme_irq_check() checks if an interrupt is pending and does not disable
interrupt source if the interrupt is pending. It simply wakes the thread.
This works because the MSI-IRQ-controller is handled as edge and it does
not re-trigger. Should the IRQ fallback to non-MSI it would become LEVEL
and will keep the IRQ-line asserted so the threaded handler will no be reached.

One of these should be done:
- remove "use_threaded_interrupts" and rely simply on the "threadirqs"
  kernel command line.
- Let nvme_irq_check() disable the IRQ-source and nvme_irq() should
  enable it again once done.
- Pass IRQF_ONESHOT as irqflags.

Side note:
 msix_request_rcd_irq() (drivers/infiniband/hw/hfi1/msix.c) provides also
 both handlers and (according to the comment) receive_context_interrupt()
 disables the source before waking the thread. This one is fine and I
 did not find more users of pci_request_irq() providing both handlers.

Sebastian

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [RFC] nvme-pci does not disable interrupt-source in threaded mode
  2019-11-26 20:28 [RFC] nvme-pci does not disable interrupt-source in threaded mode Sebastian Andrzej Siewior
@ 2019-11-26 22:28 ` Keith Busch
  0 siblings, 0 replies; 2+ messages in thread
From: Keith Busch @ 2019-11-26 22:28 UTC (permalink / raw)
  To: Sebastian Andrzej Siewior
  Cc: Jens Axboe, Thomas Gleixner, Christoph Hellwig, linux-nvme,
	Sagi Grimberg

On Tue, Nov 26, 2019 at 09:28:14PM +0100, Sebastian Andrzej Siewior wrote:
> One of these should be done:
> - remove "use_threaded_interrupts" and rely simply on the "threadirqs"
>   kernel command line.
> - Let nvme_irq_check() disable the IRQ-source and nvme_irq() should
>   enable it again once done.

There is an nvme specific way to mask interrupts really quickly, but it
only works for MSI. Can't use it for MSIx...

> - Pass IRQF_ONESHOT as irqflags.

IRQF_ONESHOT adds two non-posted transactions through pci_msi_mask_irq()
and pci_msi_unmask_irq() on each interupt. That's pretty slow.

Maybe we could have the nvme driver provide its own MSIx mask routine that
skips the readl(). That might mean local interrupts get renabled before
the device sees the MSIx mask set, but maybe that's ok?

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, back to index

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-26 20:28 [RFC] nvme-pci does not disable interrupt-source in threaded mode Sebastian Andrzej Siewior
2019-11-26 22:28 ` Keith Busch

Linux-NVME Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-nvme/0 linux-nvme/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nvme linux-nvme/ https://lore.kernel.org/linux-nvme \
		linux-nvme@lists.infradead.org
	public-inbox-index linux-nvme

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.infradead.lists.linux-nvme


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git