Linux-NVME Archive on
 help / color / Atom feed
From: Victor Gladkov <>
To: James Smart <>,
	"" <>
Subject: RE: [PATCH] nvme-fabrics: reject I/O to offline device
Date: Tue, 3 Dec 2019 10:04:55 +0000
Message-ID: <> (raw)
In-Reply-To: <>

On 12/03/2019 00:47 AM, James Smart wrote:
> On 11/30/2019 11:59 PM, Victor Gladkov wrote:
> > Issue Description:
> > Commands get stuck while Host NVMe controller (TCP or RDMA) is in
> reconnect state.
> > NVMe controller enters into reconnect state when it loses connection with
> the target. It tries to reconnect every 10 seconds (default) until successful
> reconnection or until reconnect time-out is reached. The default reconnect
> time out is 10 minutes.
> > This behavior is different than ISCSI where Commands during reconnect
> state returns with the following error: "rejecting I/O to offline device"
> >
> > Fix Description:
> > Added a kernel module parameter "nvmef_reconnect_failfast" for nvme-
> fabrics module (default is true).
> > Interfere in the decision whether to queue IO command or retry IO
> command. The interface takes into account the controller reconnect state, in
> a way that during reconnect state, IO commands shall fail immediacy (default)
> or according to IO command timeout (depends on the module parameter
> value), and IO retry is prevented. As a result, commands do not get stuck in in
> reconnect state.
> This the patch seems incorrect at least as described. Multipathing inherently
> will "fastfail" and send to other paths. So the only way something is "stuck" is
> if it's last path. If last path, we definitely don't want to prematurely release
> i/o before we've given the subsystem every opportunity to reconnect.
> What I hear you saying is you don't like the kernel default controller-loss-
> timeout of 600s. What was designed, if you didn't like the kernel default, was
> to use the per-connection "--cltr-loss-tmo"
> option for "nvme connect".  The auto-connect scripts or the admin script that
> specifies the connection can set whatever value it likes.
> If that seems hard to do, perhaps it's time to implement the options that
> allow for a config file to specify new values to be used on all connections, or
> on connections to specific subsystems, and so on. But I don't think the kernel
> needs to change.
> -- james

James, thank you for the suggestion.
But let me explain it differently.

Applications are expecting commands to complete with success or error
within a certain timeout (30 seconds by default). 
The NVMe host is enforcing that timeout while it is connected, never the less, 
during reconnection, the timeout is not enforced and commands may get stuck for a long period or even forever.

The controller-loss-timeout should not affect IO timeout policy, these are two different policies.

linux-nvme mailing list

  reply index

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-01  7:59 Victor Gladkov
2019-12-02 22:26 ` Chaitanya Kulkarni
2019-12-02 22:47 ` James Smart
2019-12-03 10:04   ` Victor Gladkov [this message]
2019-12-03 16:19     ` James Smart
2019-12-04  8:28       ` Victor Gladkov
2019-12-06  0:38         ` James Smart
2019-12-06 22:18           ` Sagi Grimberg

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NVME Archive on

Archives are clonable:
	git clone --mirror linux-nvme/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nvme linux-nvme/ \
	public-inbox-index linux-nvme

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone