linux-nvme.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
From: Shinichiro Kawasaki <shinichiro.kawasaki@wdc.com>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
	"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
	"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>
Subject: Re: blktests failures with v6.4
Date: Thu, 13 Jul 2023 01:22:32 +0000	[thread overview]
Message-ID: <v3n4k4gk5uhbuh6ijl2pwaysvxzidzhrmjejourfnmobebwbzi@hejuqcryp4nc> (raw)
In-Reply-To: <165eceaa-55d4-f9c7-7e02-18115e6df6fe@grimberg.me>

On Jul 09, 2023 / 17:32, Sagi Grimberg wrote:
> 
> > #3: nvme/003 (fabrics transport)
> > 
> >     When nvme test group is run with trtype=rdma or tcp, the test case fails
> >     due to lockdep WARNING "possible circular locking dependency detected".
> >     Reported in May/2023. Bart suggested a fix for trytpe=rdma [4] but it
> >     needs more discussion.
> > 
> >     [4] https://lore.kernel.org/linux-nvme/20230511150321.103172-1-bvanassche@acm.org/
> 
> This patch is unfortunately incorrect and buggy.
> 
> This will likely make the issue go away, but adds another
> old issue where a client can DDOS a target by bombarding it
> with connect/disconnect. When releases are async and we don't
> have any back-pressure, it is likely to happen.
> --
> diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
> index 4597bca43a6d..8b4f4aa48206 100644
> --- a/drivers/nvme/target/rdma.c
> +++ b/drivers/nvme/target/rdma.c
> @@ -1582,11 +1582,6 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id
> *cm_id,
>                 goto put_device;
>         }
> 
> -       if (queue->host_qid == 0) {
> -               /* Let inflight controller teardown complete */
> -               flush_workqueue(nvmet_wq);
> -       }
> -
>         ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn);
>         if (ret) {
>                 /*
> diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
> index 868aa4de2e4c..c8cfa19e11c7 100644
> --- a/drivers/nvme/target/tcp.c
> +++ b/drivers/nvme/target/tcp.c
> @@ -1844,11 +1844,6 @@ static u16 nvmet_tcp_install_queue(struct nvmet_sq
> *sq)
>         struct nvmet_tcp_queue *queue =
>                 container_of(sq, struct nvmet_tcp_queue, nvme_sq);
> 
> -       if (sq->qid == 0) {
> -               /* Let inflight controller teardown complete */
> -               flush_workqueue(nvmet_wq);
> -       }
> -
>         queue->nr_cmds = sq->size * 2;
>         if (nvmet_tcp_alloc_cmds(queue))
>                 return NVME_SC_INTERNAL;
> --

Thanks Sagi, I tried the patch above and confirmed the lockdep WARN disappears
for both rdma and tcp. It indicates that the flush_workqueue(nvmet_wq)
introduced the circular lock dependency. I also found the two commits below
record why the flush_workqueue(nvmet_wq) was introduced.

 777dc82395de ("nvmet-rdma: occasionally flush ongoing controller teardown")
 8832cf922151 ("nvmet: use a private workqueue instead of the system workqueue")

The left question is how to avoid both the connect/disconnect bombarding DDOS
and the circular lock possibility related to the nvmet_wq completion.

  reply	other threads:[~2023-07-13  1:22 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-07-07  7:27 blktests failures with v6.4 Shinichiro Kawasaki
2023-07-09 14:32 ` Sagi Grimberg
2023-07-13  1:22   ` Shinichiro Kawasaki [this message]
2023-07-13  7:48     ` Sagi Grimberg
2023-07-13  8:41       ` Hannes Reinecke
2023-07-13 10:16         ` Sagi Grimberg

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=v3n4k4gk5uhbuh6ijl2pwaysvxzidzhrmjejourfnmobebwbzi@hejuqcryp4nc \
    --to=shinichiro.kawasaki@wdc.com \
    --cc=linux-block@vger.kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).