From: Keith Busch <kbusch@kernel.org> To: linux-nvme@lists.infradead.org Cc: sagi@grimberg.me, bigeasy@linutronix.de, ming.lei@redhat.com, Keith Busch <kbusch@kernel.org>, tglx@linutronix.de, hch@lst.de Subject: [PATCHv3 0/4] nvme pci interrupt handling improvements Date: Tue, 10 Dec 2019 02:56:18 +0900 Message-ID: <20191209175622.1964-1-kbusch@kernel.org> (raw) Here's the next revision to nvme pci interrupt handling. The series is attempting to address two issues that have been recently raised: 1. The nvme driver makes it possible to soft lockup a CPU due to high utilization in irq context. This is most prevalent when multiple CPUs are driving a single hardware queue on one or more controllers. 2. The threaded handler left interrupts unmasked, which breaks when used with level triggered interrupts, or unnecessarily runs in interrupt context when edge interrupts occur frequently. Both issues are addressed by always configuring nvme interrupts to run the threaded handler with interrupts disabled. A hybrid approch to handling nvme completions in hard irq context and thread context is introduced so that there should not be a performance impact from removing the nvme.use_threaded_interrupts option. The series appears to be a win or no impact on performance from what I can test. I would be greatful to hear if others can confirm this with other hardware. I've dropped the fast MSIx masking. While I believe it's safe to skip flushing the memory write, I think this series mitigates the impact of the read back by ensuring the ratio of memory reads to IO is low enough to be negligable (AFAICT on hardware available to me). I've changed the exit condition for the polling nvme irq thread to break out of the loop if we've wrapped the completion queue. Other irq threads may be affinitized to the same CPU, so we need to schedule out at some point, but I've been told multiple times that need_resched() or cond_resched() won't work as desired from the thread's fifo priority. Keith Busch (4): nvme/pci: Disable interrupts for threaded handler nvme/pci: Complete commands from primary handler nvme/pci: Remove use_threaded_interrupts nvme/pci: Poll threaded completions drivers/nvme/host/pci.c | 56 ++++++++++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 20 deletions(-) -- 2.21.0 _______________________________________________ linux-nvme mailing list linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme
next reply index Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-12-09 17:56 Keith Busch [this message] 2019-12-09 17:56 ` [PATCHv3 1/4] nvme/pci: Disable interrupts for threaded handler Keith Busch 2019-12-10 15:12 ` Daniel Wagner 2019-12-10 15:28 ` Sebastian Andrzej Siewior 2019-12-10 15:54 ` Keith Busch 2019-12-10 16:44 ` Daniel Wagner 2019-12-10 16:57 ` Keith Busch 2019-12-10 17:11 ` Daniel Wagner 2019-12-12 9:09 ` Christoph Hellwig 2019-12-12 15:53 ` Keith Busch 2019-12-09 17:56 ` [PATCHv3 2/4] nvme/pci: Complete commands from primary handler Keith Busch 2019-12-10 20:00 ` Sagi Grimberg 2019-12-10 20:25 ` Keith Busch 2019-12-10 21:14 ` Sagi Grimberg 2019-12-11 17:35 ` Keith Busch 2019-12-12 0:40 ` Sagi Grimberg 2019-12-12 1:02 ` Keith Busch 2019-12-12 22:55 ` Ming Lei 2019-12-12 23:30 ` Keith Busch 2019-12-13 0:52 ` Ming Lei 2019-12-12 9:14 ` Christoph Hellwig 2019-12-09 17:56 ` [PATCHv3 3/4] nvme/pci: Remove use_threaded_interrupts Keith Busch 2019-12-12 9:14 ` Christoph Hellwig 2019-12-12 15:45 ` Keith Busch 2019-12-18 7:29 ` Ming Lei 2019-12-18 15:50 ` Keith Busch 2019-12-19 1:10 ` Ming Lei 2019-12-09 17:56 ` [PATCHv3 4/4] nvme/pci: Poll threaded completions Keith Busch 2019-12-10 17:43 ` Daniel Wagner
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20191209175622.1964-1-kbusch@kernel.org \ --to=kbusch@kernel.org \ --cc=bigeasy@linutronix.de \ --cc=hch@lst.de \ --cc=linux-nvme@lists.infradead.org \ --cc=ming.lei@redhat.com \ --cc=sagi@grimberg.me \ --cc=tglx@linutronix.de \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Linux-NVME Archive on lore.kernel.org Archives are clonable: git clone --mirror https://lore.kernel.org/linux-nvme/0 linux-nvme/git/0.git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V2 linux-nvme linux-nvme/ https://lore.kernel.org/linux-nvme \ linux-nvme@lists.infradead.org public-inbox-index linux-nvme Example config snippet for mirrors Newsgroup available over NNTP: nntp://nntp.lore.kernel.org/org.infradead.lists.linux-nvme AGPL code for this site: git clone https://public-inbox.org/public-inbox.git