From: Sagi Grimberg <sagi@grimberg.me>
To: Potnuri Bharat Teja <bharat@chelsio.com>
Cc: Samuel Jones <sjones@kalrayinc.com>, "hch@lst.de" <hch@lst.de>,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>
Subject: Re: Request timeout seen with NVMEoF TCP
Date: Mon, 14 Dec 2020 09:51:28 -0800 [thread overview]
Message-ID: <dbb14893-da35-2f7e-0583-c33ce9f41243@grimberg.me> (raw)
In-Reply-To: <X9MfI872AfM+WMV/@chelsio.com>
> Hi Sagi,
> With above patch I still see the issue but less frequently. Without patch I was
> able to consistently reproduce the timouts with 4 target devices. With patch I
> see IO running fine for 4 targets. Tried the same test with 8 target devices
> and I see the below timeout. I've observed only one instance of timeout. So,
> I'll let it run for somemore time or rerun and update.
Hey Potnuri,
Have you observed this further?
I'd think that if the io_work reschedule itself when it races
with the direct send path this should not happen, but we may be
seeing a different race going on here, adding Samuel who saw
a similar phenomenon.
>
> Target dmesg:
> ---
> [ 1704.132366] nvmet: creating controller 1 for subsystem nvme-ram0 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.185987] nvmet: creating controller 2 for subsystem nvme-ram1 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.230065] nvmet: creating controller 3 for subsystem nvme-ram2 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.277712] nvmet: creating controller 4 for subsystem nvme-ram3 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.314457] nvmet: creating controller 5 for subsystem nvme-ram4 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.370124] nvmet: creating controller 6 for subsystem nvme-ram5 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.435581] nvmet: creating controller 7 for subsystem nvme-ram6 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 1704.501813] nvmet: creating controller 8 for subsystem nvme-ram7 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> [ 2103.965017] nvmet: creating controller 6 for subsystem nvme-ram5 for NQN nqn.2014-08.org.nvmexpress:uuid:77f6ffad-1c4a-4c0e-9f11-23cd4daf0216.
> ^^^^^^^^^^^^^^^
> ---
>
> Initiator dmesg:
> ---
> [ 1735.038634] EXT4-fs (nvme7n1): mounted filesystem with ordered data mode. Opts: (null)
> [ 2111.990419] nvme nvme5: queue 7: timeout request 0x57 type 4
> [ 2111.991835] nvme nvme5: starting error recovery
> [ 2111.998796] block nvme5n1: no usable path - requeuing I/O
> [ 2111.998816] nvme nvme5: Reconnecting in 10 seconds...
> [ 2122.253431] block nvme5n1: no usable path - requeuing I/O
> [ 2122.254732] nvme nvme5: creating 16 I/O queues.
> [ 2122.301169] nvme nvme5: mapped 16/0/0 default/read/poll queues.
> [ 2122.314229] nvme nvme5: Successfully reconnected (1 attempt)
> ---
>
>
> Thanks.
>
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2020-12-14 17:51 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-09 17:19 Request timeout seen with NVMEoF TCP Potnuri Bharat Teja
2020-12-09 18:02 ` Sagi Grimberg
2020-12-10 17:06 ` Potnuri Bharat Teja
2020-12-10 21:47 ` Sagi Grimberg
2020-12-11 7:26 ` Potnuri Bharat Teja
2020-12-14 17:51 ` Sagi Grimberg [this message]
2020-12-14 18:05 ` Potnuri Bharat Teja
2020-12-14 19:27 ` Samuel Jones
2020-12-14 20:13 ` Sagi Grimberg
2020-12-15 1:53 ` Sagi Grimberg
2020-12-15 18:30 ` Wunderlich, Mark
2020-12-19 1:04 ` Sagi Grimberg
2020-12-16 5:51 ` Potnuri Bharat Teja
2020-12-19 1:35 ` Sagi Grimberg
2020-12-21 7:50 ` Potnuri Bharat Teja
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=dbb14893-da35-2f7e-0583-c33ce9f41243@grimberg.me \
--to=sagi@grimberg.me \
--cc=bharat@chelsio.com \
--cc=hch@lst.de \
--cc=linux-nvme@lists.infradead.org \
--cc=sjones@kalrayinc.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.