archive mirror
 help / color / mirror / Atom feed
From: Sagi Grimberg <>
To: Christoph Hellwig <>, Keith Busch <>,
	Jens Axboe <>
Cc: Chao Leng <>,,
Subject: Re: [PATCH 2/2] nvme-multipath: don't block on blk_queue_enter of the underlying device
Date: Mon, 22 Mar 2021 19:57:27 -0700	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

> When we reset/teardown a controller, we must freeze and quiesce the
> namespaces request queues to make sure that we safely stop inflight I/O
> submissions. Freeze is mandatory because if our hctx map changed between
> reconnects, blk_mq_update_nr_hw_queues will immediately attempt to freeze
> the queue, and if it still has pending submissions (that are still
> quiesced) it will hang.
> However, by freezing the namespaces request queues, and only unfreezing
> them when we successfully reconnect, inflight submissions that are
> running concurrently can now block grabbing the nshead srcu until either
> we successfully reconnect or ctrl_loss_tmo expired (or the user
> explicitly disconnected).
> This caused a deadlock when a different controller (different path on the
> same subsystem) became live (i.e. optimized/non-optimized). This is
> because nvme_mpath_set_live needs to synchronize the nshead srcu before
> requeueing I/O in order to make sure that current_path is visible to
> future (re-)submisions. However the srcu lock is taken by a blocked
> submission on a frozen request queue, and we have a deadlock.
> In order to fix this use the blk_mq_submit_bio_direct API to submit the
> bio to the low-level driver, which does not block on the queue free
> but instead allows nvme-multipath to pick another path or queue up the
> bio.


This still has the same issue but instead of blocking on
blk_queue_enter() it is blocked on blk_mq_get_tag():
  ? blk_bio_segment_split+0x235/0x2a0
  ? finish_wait+0x80/0x80
  nvme_ns_head_submit_bio+0xc3/0x2f0 [nvme_core]
  ? __io_complete_rw.constprop.0+0x150/0x150
  ? common_interrupt+0x14/0xa0
  ? update_load_avg+0x7a/0x5e0
  ? lock_timer_base+0x61/0x80
  ? io_worker_handle_work+0x330/0x330

  ? usleep_range+0x80/0x80
  ? usleep_range+0x80/0x80
  ? del_timer_sync+0x67/0xb0
  ? __prepare_to_swait+0x4b/0x70
  ? __bpf_trace_rcu_utilization+0x10/0x10
  nvme_mpath_set_live+0x61/0x130 [nvme_core]
  nvme_update_ana_state+0xd7/0x100 [nvme_core]
  nvme_parse_ana_log+0xa5/0x160 [nvme_core]
  ? nvme_mpath_set_live+0x130/0x130 [nvme_core]
  nvme_read_ana_log+0x7b/0xe0 [nvme_core]

If I were to always start the queues in nvme_tcp_teardown_ctrl
right after I cancel the tagset inflights like:
@@ -1934,8 +1934,7 @@ static void nvme_tcp_teardown_io_queues(struct 
nvme_ctrl *ctrl,
-       if (remove)
-               nvme_start_queues(ctrl);
+       nvme_start_queues(ctrl);
         nvme_tcp_destroy_io_queues(ctrl, remove);

then a simple reset during traffic bricks the host on infinite loop
because in the setup sequence we freeze the queue in
nvme_update_ns_info, so the queue is frozen but we still have an
available path (because the controller is back to live!) so nvme-mpath
keeps calling blk_mq_submit_bio_direct and fails, and
nvme_update_ns_info cannot properly freeze the queue..
-> deadlock.

So this is obviously incorrect.

Also, if we make nvme-mpath submit a REQ_NOWAIT we basically
will fail as soon as we run out of tags, even in the normal path...

So I'm not exactly sure what we should do to fix this...

Linux-nvme mailing list

  parent reply	other threads:[~2021-03-23  2:58 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-22  7:37 fix nvme-tcp and nvme-rdma controller reset hangs when using multipath Christoph Hellwig
2021-03-22  7:37 ` [PATCH 1/2] blk-mq: add a blk_mq_submit_bio_direct API Christoph Hellwig
2021-03-22 11:23   ` Hannes Reinecke
2021-03-22 15:30   ` Keith Busch
2021-03-22  7:37 ` [PATCH 2/2] nvme-multipath: don't block on blk_queue_enter of the underlying device Christoph Hellwig
2021-03-22 11:22   ` Hannes Reinecke
2021-03-22 15:31   ` Keith Busch
2021-03-23  2:57   ` Sagi Grimberg [this message]
2021-03-23  3:23     ` Sagi Grimberg
2021-03-23  7:04       ` Chao Leng
2021-03-23  7:36         ` Sagi Grimberg
2021-03-23  8:13           ` Chao Leng
2021-03-23 16:17             ` Christoph Hellwig
2021-03-23 16:15           ` Christoph Hellwig
2021-03-23 18:13             ` Sagi Grimberg
2021-03-23 18:22               ` Christoph Hellwig
2021-03-23 19:00                 ` Sagi Grimberg
2021-03-23 19:01                   ` Christoph Hellwig
2021-03-23 19:10                     ` Sagi Grimberg
2021-03-23  7:28     ` Hannes Reinecke
2021-03-23  7:31       ` Sagi Grimberg
2021-03-23  8:36         ` Hannes Reinecke
2021-03-23 14:53           ` Keith Busch
2021-03-23 16:19             ` Christoph Hellwig

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).