linux-nvme.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* [PATCHv3 0/4] nvme pci interrupt handling improvements
@ 2019-12-09 17:56 Keith Busch
  2019-12-09 17:56 ` [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler Keith Busch
                   ` (3 more replies)
  0 siblings, 4 replies; 29+ messages in thread
From: Keith Busch @ 2019-12-09 17:56 UTC (permalink / raw)
  To: linux-nvme; +Cc: sagi, bigeasy, ming.lei, Keith Busch, tglx, hch

Here's the next revision to nvme pci interrupt handling. The series is
attempting to address two issues that have been recently raised:

  1. The nvme driver makes it possible to soft lockup a CPU due to high
     utilization in irq context. This is most prevalent when multiple
     CPUs are driving a single hardware queue on one or more controllers.

  2. The threaded handler left interrupts unmasked, which breaks when
     used with level triggered interrupts, or unnecessarily runs
     in interrupt context when edge interrupts occur frequently.

Both issues are addressed by always configuring nvme interrupts to
run the threaded handler with interrupts disabled. A hybrid approch
to handling nvme completions in hard irq context and thread context is
introduced so that there should not be a performance impact from removing
the nvme.use_threaded_interrupts option.

The series appears to be a win or no impact on performance from what I
can test. I would be greatful to hear if others can confirm this with
other hardware.

I've dropped the fast MSIx masking. While I believe it's safe to skip
flushing the memory write, I think this series mitigates the impact of
the read back by ensuring the ratio of memory reads to IO is low enough
to be negligable (AFAICT on hardware available to me).

I've changed the exit condition for the polling nvme irq thread to
break out of the loop if we've wrapped the completion queue. Other irq
threads may be affinitized to the same CPU, so we need to schedule out
at some point, but I've been told multiple times that need_resched()
or cond_resched() won't work as desired from the thread's fifo priority.

Keith Busch (4):
  nvme/pci: Disable interrupts for threaded handler
  nvme/pci: Complete commands from primary handler
  nvme/pci: Remove use_threaded_interrupts
  nvme/pci: Poll threaded completions

 drivers/nvme/host/pci.c | 56 ++++++++++++++++++++++++++---------------
 1 file changed, 36 insertions(+), 20 deletions(-)

-- 
2.21.0


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2019-12-19  1:11 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-12-09 17:56 [PATCHv3 0/4] nvme pci interrupt handling improvements Keith Busch
2019-12-09 17:56 ` [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler Keith Busch
2019-12-10 15:12   ` Daniel Wagner
2019-12-10 15:28     ` Sebastian Andrzej Siewior
2019-12-10 15:54       ` Keith Busch
2019-12-10 16:44         ` Daniel Wagner
2019-12-10 16:57           ` Keith Busch
2019-12-10 17:11             ` Daniel Wagner
2019-12-12  9:09   ` Christoph Hellwig
2019-12-12 15:53     ` Keith Busch
2019-12-09 17:56 ` [PATCHv3 2/4] nvme/pci: Complete commands from primary handler Keith Busch
2019-12-10 20:00   ` Sagi Grimberg
2019-12-10 20:25     ` Keith Busch
2019-12-10 21:14       ` Sagi Grimberg
2019-12-11 17:35         ` Keith Busch
2019-12-12  0:40           ` Sagi Grimberg
2019-12-12  1:02             ` Keith Busch
2019-12-12 22:55               ` Ming Lei
2019-12-12 23:30                 ` Keith Busch
2019-12-13  0:52                   ` Ming Lei
2019-12-12  9:14   ` Christoph Hellwig
2019-12-09 17:56 ` [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts Keith Busch
2019-12-12  9:14   ` Christoph Hellwig
2019-12-12 15:45     ` Keith Busch
2019-12-18  7:29     ` Ming Lei
2019-12-18 15:50       ` Keith Busch
2019-12-19  1:10         ` Ming Lei
2019-12-09 17:56 ` [PATCHv3 4/4] nvme/pci: Poll threaded completions Keith Busch
2019-12-10 17:43   ` Daniel Wagner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).