From: Keith Busch <kbusch@kernel.org>
To: Bharat Kumar Gogada <bharatku@xilinx.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
Keith Busch <keith.busch@linux.intel.com>,
"keith.busch@intel.com" <keith.busch@intel.com>
Subject: Re: NVMe Poll CQ on timeout
Date: Thu, 19 Sep 2019 08:13:01 -0600 [thread overview]
Message-ID: <20190919141301.GA61660@C02WT3WMHTD6> (raw)
In-Reply-To: <MN2PR02MB633689DBBA6DE9DD7A34043FA5890@MN2PR02MB6336.namprd02.prod.outlook.com>
On Thu, Sep 19, 2019 at 01:47:50PM +0000, Bharat Kumar Gogada wrote:
> Hi All,
>
> We are testing NVMe cards on ARM64 platform, the card uses MSI-X interrupts.
> We are hitting following case in drivers/nvme/host/pci.c
> /*
> * Did we miss an interrupt?
> */
> if (__nvme_poll(nvmeq, req->tag)) {
> dev_warn(dev->ctrl.device,
> "I/O %d QID %d timeout, completion polled\n",
> req->tag, nvmeq->qid);
> return BLK_EH_DONE;
> }
>
> Can anyone tell when does nvme_timeout gets invoked ?
Timeout is invoked when the driver didn't see a completion to a
submitted command.
> In what cases we see this interrupt miss ?
That usually happens for one of two reasons:
1. The device didn't send any MSIx message for a CQE
2. The device sent the MSIx message before posting the CQE
I've also seen h/w errata where the MSIx and CQE are re-ordered, which
can also lead to this.
A hardware trace would provide the most detailed view of what's
happening. You might be able to infer if you carefully account for
commands sent, interrupts received, and spurious interrupts detected.
> We are seeing this issue only for reads with following fio command
> fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randread --bs=128k --direct=0 \
> --size=128M --numjobs=3 --group_reporting --filename=/dev/nvme0n1
>
> We are not seeing issue with --rw=randwrite for same size.
>
> Please let us know what can cause this issue.
next prev parent reply other threads:[~2019-09-19 14:13 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-09-19 13:47 NVMe Poll CQ on timeout Bharat Kumar Gogada
2019-09-19 14:13 ` Keith Busch [this message]
-- strict thread matches above, loose matches on Subject: below --
2018-04-26 14:59 Bharat Kumar Gogada
2018-05-07 6:57 ` Bharat Kumar Gogada
2018-05-07 16:02 ` Keith Busch
2018-05-15 13:58 ` Bharat Kumar Gogada
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190919141301.GA61660@C02WT3WMHTD6 \
--to=kbusch@kernel.org \
--cc=bharatku@xilinx.com \
--cc=keith.busch@intel.com \
--cc=keith.busch@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).