From: Keith Busch <kbusch@kernel.org>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: linux-nvme@lists.infradead.org, hch@lst.de
Subject: Re: nvme tcp receive errors
Date: Thu, 1 Apr 2021 05:49:58 +0900 [thread overview]
Message-ID: <20210331204958.GD23886@redsun51.ssa.fujisawa.hgst.com> (raw)
In-Reply-To: <0976ff40-751e-cb95-429a-04ffa229ebf0@grimberg.me>
On Wed, Mar 31, 2021 at 12:10:55PM -0700, Sagi Grimberg wrote:
> Hey Keith,
>
> > While running a read-write mixed workload, we are observing errors like:
> >
> > nvme nvme4: queue 2 no space in request 0x1
>
> This means that we get a data payload from a read request and
> we don't have a bio/bvec space to store it, which means we
> are probably not tracking the request iterator correctly if
> tcpdump shows that we are getting the right data length.
>
> > Based on tcpdump, all data for this queue is expected to satisfy the
> > command request. I'm not familiar enough with the tcp interfaces, so
> > could anyone provide pointers on how to debug this further?
>
> What was the size of the I/O that you were using? Is this easily
> reproducible?
>
> Do you have the below applied:
> ca1ff67d0fb1 ("nvme-tcp: fix possible data corruption with bio merges")
> 0dc9edaf80ea ("nvme-tcp: pass multipage bvec to request iov_iter")
>
> I'm assuming yes if you are using the latest nvme tree...
>
> Does the issue still happens when you revert 0dc9edaf80ea?
Thanks for the reply.
This was observed on the recent 5.12-rc4, so it has all the latest tcp
fixes. I'll check with reverting 0dc9edaf80ea and see if that makes a
difference. It is currently reproducible, though it can take over an
hour right now.
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2021-03-31 20:50 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-31 16:18 nvme tcp receive errors Keith Busch
2021-03-31 19:10 ` Sagi Grimberg
2021-03-31 20:49 ` Keith Busch [this message]
2021-03-31 22:16 ` Sagi Grimberg
2021-03-31 22:26 ` Keith Busch
2021-03-31 22:45 ` Sagi Grimberg
2021-04-02 17:11 ` Keith Busch
2021-04-02 17:27 ` Sagi Grimberg
2021-04-05 14:37 ` Keith Busch
2021-04-07 19:53 ` Keith Busch
2021-04-09 21:38 ` Sagi Grimberg
2021-04-27 23:39 ` Keith Busch
2021-04-27 23:55 ` Sagi Grimberg
2021-04-28 15:58 ` Keith Busch
2021-04-28 17:42 ` Sagi Grimberg
2021-04-28 18:01 ` Keith Busch
2021-04-28 23:06 ` Sagi Grimberg
2021-04-29 3:33 ` Keith Busch
2021-04-29 4:52 ` Sagi Grimberg
2021-05-03 18:51 ` Keith Busch
2021-05-03 19:58 ` Sagi Grimberg
2021-05-03 20:25 ` Keith Busch
2021-05-04 19:29 ` Sagi Grimberg
2021-04-09 18:04 ` Sagi Grimberg
2021-04-14 0:29 ` Keith Busch
2021-04-21 5:33 ` Sagi Grimberg
2021-04-21 14:28 ` Keith Busch
2021-04-21 16:59 ` Sagi Grimberg
2021-04-26 15:31 ` Keith Busch
2021-04-27 3:10 ` Sagi Grimberg
2021-04-27 18:12 ` Keith Busch
2021-04-27 23:58 ` Sagi Grimberg
2021-04-30 23:42 ` Sagi Grimberg
2021-05-03 14:28 ` Keith Busch
2021-05-03 19:36 ` Sagi Grimberg
2021-05-03 19:38 ` Sagi Grimberg
2021-05-03 19:44 ` Keith Busch
2021-05-03 20:00 ` Sagi Grimberg
2021-05-04 14:36 ` Keith Busch
2021-05-04 18:15 ` Sagi Grimberg
2021-05-04 19:14 ` Keith Busch
2021-05-10 18:06 ` Keith Busch
2021-05-10 18:18 ` Sagi Grimberg
2021-05-10 18:30 ` Keith Busch
2021-05-10 21:07 ` Sagi Grimberg
2021-05-11 3:00 ` Keith Busch
2021-05-11 17:17 ` Sagi Grimberg
2021-05-13 15:48 ` Keith Busch
2021-05-13 19:53 ` Sagi Grimberg
2021-05-17 20:48 ` Keith Busch
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210331204958.GD23886@redsun51.ssa.fujisawa.hgst.com \
--to=kbusch@kernel.org \
--cc=hch@lst.de \
--cc=linux-nvme@lists.infradead.org \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).