All of lore.kernel.org
 help / color / mirror / Atom feed
From: Christoph Hellwig <hch@lst.de>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: Chao Leng <lengchao@huawei.com>, Christoph Hellwig <hch@lst.de>,
	Keith Busch <kbusch@kernel.org>, Jens Axboe <axboe@kernel.dk>,
	linux-block@vger.kernel.org, linux-nvme@lists.infradead.org
Subject: Re: [PATCH 2/2] nvme-multipath: don't block on blk_queue_enter of the underlying device
Date: Tue, 23 Mar 2021 17:15:44 +0100	[thread overview]
Message-ID: <20210323161544.GA13402@lst.de> (raw)
In-Reply-To: <5d28226d-4619-74b6-1c73-c13ed57aa7ea@grimberg.me>

On Tue, Mar 23, 2021 at 12:36:40AM -0700, Sagi Grimberg wrote:
>> The process:
>> 1.nvme_ns_head_submit_bio call srcu_read_lock(&head->srcu).
>> 2.nvme_ns_head_submit_bio will add the bio to current->bio_list instead of 
>> waiting for the frozen queue.
>
> Nothing guarantees that you have a bio_list active at any point in time,
> in fact for a workload that submits one by one you will always drain
> that list directly in the submission...

It should always be active when ->submit_bio is called.

>
>> 3.nvme_ns_head_submit_bio call srcu_read_unlock(&head->srcu, srcu_idx).
>> So nvme_ns_head_submit_bio do not hold head->srcu long when the queue is 
>> frozen, can avoid deadlock.
>>
>> Sagi, suggest trying this patch.
>
> The above reproduces with the patch applied on upstream nvme code.

Weird.  I don't think the deadlock in your original report should
happen due to this.  Can you take a look at the callstacks in the
reproduced deadlock?  Either we're missing something obvious or it is a
a somewhat different deadlock.

WARNING: multiple messages have this Message-ID (diff)
From: Christoph Hellwig <hch@lst.de>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: Chao Leng <lengchao@huawei.com>, Christoph Hellwig <hch@lst.de>,
	Keith Busch <kbusch@kernel.org>, Jens Axboe <axboe@kernel.dk>,
	linux-block@vger.kernel.org, linux-nvme@lists.infradead.org
Subject: Re: [PATCH 2/2] nvme-multipath: don't block on blk_queue_enter of the underlying device
Date: Tue, 23 Mar 2021 17:15:44 +0100	[thread overview]
Message-ID: <20210323161544.GA13402@lst.de> (raw)
In-Reply-To: <5d28226d-4619-74b6-1c73-c13ed57aa7ea@grimberg.me>

On Tue, Mar 23, 2021 at 12:36:40AM -0700, Sagi Grimberg wrote:
>> The process:
>> 1.nvme_ns_head_submit_bio call srcu_read_lock(&head->srcu).
>> 2.nvme_ns_head_submit_bio will add the bio to current->bio_list instead of 
>> waiting for the frozen queue.
>
> Nothing guarantees that you have a bio_list active at any point in time,
> in fact for a workload that submits one by one you will always drain
> that list directly in the submission...

It should always be active when ->submit_bio is called.

>
>> 3.nvme_ns_head_submit_bio call srcu_read_unlock(&head->srcu, srcu_idx).
>> So nvme_ns_head_submit_bio do not hold head->srcu long when the queue is 
>> frozen, can avoid deadlock.
>>
>> Sagi, suggest trying this patch.
>
> The above reproduces with the patch applied on upstream nvme code.

Weird.  I don't think the deadlock in your original report should
happen due to this.  Can you take a look at the callstacks in the
reproduced deadlock?  Either we're missing something obvious or it is a
a somewhat different deadlock.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

  parent reply	other threads:[~2021-03-23 16:16 UTC|newest]

Thread overview: 48+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-22  7:37 fix nvme-tcp and nvme-rdma controller reset hangs when using multipath Christoph Hellwig
2021-03-22  7:37 ` Christoph Hellwig
2021-03-22  7:37 ` [PATCH 1/2] blk-mq: add a blk_mq_submit_bio_direct API Christoph Hellwig
2021-03-22  7:37   ` Christoph Hellwig
2021-03-22 11:23   ` Hannes Reinecke
2021-03-22 11:23     ` Hannes Reinecke
2021-03-22 15:30   ` Keith Busch
2021-03-22 15:30     ` Keith Busch
2021-03-22  7:37 ` [PATCH 2/2] nvme-multipath: don't block on blk_queue_enter of the underlying device Christoph Hellwig
2021-03-22  7:37   ` Christoph Hellwig
2021-03-22 11:22   ` Hannes Reinecke
2021-03-22 11:22     ` Hannes Reinecke
2021-03-22 15:31   ` Keith Busch
2021-03-22 15:31     ` Keith Busch
2021-03-23  2:57   ` Sagi Grimberg
2021-03-23  2:57     ` Sagi Grimberg
2021-03-23  3:23     ` Sagi Grimberg
2021-03-23  3:23       ` Sagi Grimberg
2021-03-23  7:04       ` Chao Leng
2021-03-23  7:04         ` Chao Leng
2021-03-23  7:36         ` Sagi Grimberg
2021-03-23  7:36           ` Sagi Grimberg
2021-03-23  8:13           ` Chao Leng
2021-03-23  8:13             ` Chao Leng
2021-03-23 16:17             ` Christoph Hellwig
2021-03-23 16:17               ` Christoph Hellwig
2021-03-23 16:15           ` Christoph Hellwig [this message]
2021-03-23 16:15             ` Christoph Hellwig
2021-03-23 18:13             ` Sagi Grimberg
2021-03-23 18:13               ` Sagi Grimberg
2021-03-23 18:22               ` Christoph Hellwig
2021-03-23 18:22                 ` Christoph Hellwig
2021-03-23 19:00                 ` Sagi Grimberg
2021-03-23 19:00                   ` Sagi Grimberg
2021-03-23 19:01                   ` Christoph Hellwig
2021-03-23 19:01                     ` Christoph Hellwig
2021-03-23 19:10                     ` Sagi Grimberg
2021-03-23 19:10                       ` Sagi Grimberg
2021-03-23  7:28     ` Hannes Reinecke
2021-03-23  7:28       ` Hannes Reinecke
2021-03-23  7:31       ` Sagi Grimberg
2021-03-23  7:31         ` Sagi Grimberg
2021-03-23  8:36         ` Hannes Reinecke
2021-03-23  8:36           ` Hannes Reinecke
2021-03-23 14:53           ` Keith Busch
2021-03-23 14:53             ` Keith Busch
2021-03-23 16:19             ` Christoph Hellwig
2021-03-23 16:19               ` Christoph Hellwig

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210323161544.GA13402@lst.de \
    --to=hch@lst.de \
    --cc=axboe@kernel.dk \
    --cc=kbusch@kernel.org \
    --cc=lengchao@huawei.com \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.