All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sagi Grimberg <sagi@grimberg.me>
To: Yi Zhang <yi.zhang@redhat.com>
Cc: linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org
Subject: Re: [bug report] NVMe/IB: reset_controller need more than 1min
Date: Wed, 23 Jun 2021 14:32:45 -0700	[thread overview]
Message-ID: <84208be5-a7a9-5261-398c-fa9bda3efbe3@grimberg.me> (raw)
In-Reply-To: <CAHj4cs_VZ7C7ciKy-q51a+Gc=uce0GDKRHNmUdoGOd7KSvURpA@mail.gmail.com>


> Hello
> 
> Gentle ping here, this issue still exists on latest 5.13-rc7
> 
> # time nvme reset /dev/nvme0
> 
> real 0m12.636s
> user 0m0.002s
> sys 0m0.005s
> # time nvme reset /dev/nvme0
> 
> real 0m12.641s
> user 0m0.000s
> sys 0m0.007s

Strange that even normal resets take so long...
What device are you using?

> # time nvme reset /dev/nvme0
> 
> real 1m16.133s
> user 0m0.000s
> sys 0m0.007s

There seems to be a spurious command timeout here, but maybe this
is due to the fact that the queues take so long to connect and
the target expires the keep-alive timer.

Does this patch help?
--
diff --git a/drivers/nvme/target/fabrics-cmd.c 
b/drivers/nvme/target/fabrics-cmd.c
index 7d0f3523fdab..f4a7db1ab3e5 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -142,6 +142,14 @@ static u16 nvmet_install_queue(struct nvmet_ctrl 
*ctrl, struct nvmet_req *req)
                 }
         }

+       /*
+        * Controller establishment flow may take some time, and the 
host may not
+        * send us keep-alive during this period, hence reset the
+        * traffic based keep-alive timer so we don't trigger a
+        * controller teardown as a result of a keep-alive expiration.
+        */
+       ctrl->reset_tbkas = true;
+
         return 0;

  err:
--

>> target:
>> [  934.306016] nvmet: creating controller 1 for subsystem testnqn for
>> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
>> [  944.875021] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
>> [  944.900051] nvmet: ctrl 1 fatal error occurred!
>> [ 1005.628340] nvmet: creating controller 1 for subsystem testnqn for
>> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
>>
>> client:
>> [  857.264029] nvme nvme0: resetting controller
>> [  864.115369] nvme nvme0: creating 40 I/O queues.
>> [  867.996746] nvme nvme0: mapped 40/0/0 default/read/poll queues.
>> [  868.001673] nvme nvme0: resetting controller
>> [  935.396789] nvme nvme0: I/O 9 QID 0 timeout
>> [  935.402036] nvme nvme0: Property Set error: 881, offset 0x14
>> [  935.438080] nvme nvme0: creating 40 I/O queues.
>> [  939.332125] nvme nvme0: mapped 40/0/0 default/read/poll queues.

WARNING: multiple messages have this Message-ID (diff)
From: Sagi Grimberg <sagi@grimberg.me>
To: Yi Zhang <yi.zhang@redhat.com>
Cc: linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org
Subject: Re: [bug report] NVMe/IB: reset_controller need more than 1min
Date: Wed, 23 Jun 2021 14:32:45 -0700	[thread overview]
Message-ID: <84208be5-a7a9-5261-398c-fa9bda3efbe3@grimberg.me> (raw)
In-Reply-To: <CAHj4cs_VZ7C7ciKy-q51a+Gc=uce0GDKRHNmUdoGOd7KSvURpA@mail.gmail.com>


> Hello
> 
> Gentle ping here, this issue still exists on latest 5.13-rc7
> 
> # time nvme reset /dev/nvme0
> 
> real 0m12.636s
> user 0m0.002s
> sys 0m0.005s
> # time nvme reset /dev/nvme0
> 
> real 0m12.641s
> user 0m0.000s
> sys 0m0.007s

Strange that even normal resets take so long...
What device are you using?

> # time nvme reset /dev/nvme0
> 
> real 1m16.133s
> user 0m0.000s
> sys 0m0.007s

There seems to be a spurious command timeout here, but maybe this
is due to the fact that the queues take so long to connect and
the target expires the keep-alive timer.

Does this patch help?
--
diff --git a/drivers/nvme/target/fabrics-cmd.c 
b/drivers/nvme/target/fabrics-cmd.c
index 7d0f3523fdab..f4a7db1ab3e5 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -142,6 +142,14 @@ static u16 nvmet_install_queue(struct nvmet_ctrl 
*ctrl, struct nvmet_req *req)
                 }
         }

+       /*
+        * Controller establishment flow may take some time, and the 
host may not
+        * send us keep-alive during this period, hence reset the
+        * traffic based keep-alive timer so we don't trigger a
+        * controller teardown as a result of a keep-alive expiration.
+        */
+       ctrl->reset_tbkas = true;
+
         return 0;

  err:
--

>> target:
>> [  934.306016] nvmet: creating controller 1 for subsystem testnqn for
>> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
>> [  944.875021] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
>> [  944.900051] nvmet: ctrl 1 fatal error occurred!
>> [ 1005.628340] nvmet: creating controller 1 for subsystem testnqn for
>> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
>>
>> client:
>> [  857.264029] nvme nvme0: resetting controller
>> [  864.115369] nvme nvme0: creating 40 I/O queues.
>> [  867.996746] nvme nvme0: mapped 40/0/0 default/read/poll queues.
>> [  868.001673] nvme nvme0: resetting controller
>> [  935.396789] nvme nvme0: I/O 9 QID 0 timeout
>> [  935.402036] nvme nvme0: Property Set error: 881, offset 0x14
>> [  935.438080] nvme nvme0: creating 40 I/O queues.
>> [  939.332125] nvme nvme0: mapped 40/0/0 default/read/poll queues.

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

  reply	other threads:[~2021-06-23 21:32 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-05-21 16:12 [bug report] NVMe/IB: reset_controller need more than 1min Yi Zhang
2021-05-21 16:12 ` Yi Zhang
2021-05-21 18:00 ` Sagi Grimberg
2021-05-21 18:00   ` Sagi Grimberg
2021-05-22  4:27   ` Yi Zhang
2021-05-22  4:27     ` Yi Zhang
2021-06-23 10:01     ` Yi Zhang
2021-06-23 10:01       ` Yi Zhang
2021-06-23 21:32       ` Sagi Grimberg [this message]
2021-06-23 21:32         ` Sagi Grimberg
2021-06-24 16:14         ` Yi Zhang
2021-06-24 16:14           ` Yi Zhang
2021-12-11  3:01           ` Yi Zhang
2021-12-12  9:45             ` Sagi Grimberg
2021-12-13  6:12               ` Yi Zhang
2021-12-13  9:04                 ` Sagi Grimberg
2021-12-13 17:05                   ` Yi Zhang
2021-12-14 10:39                     ` Sagi Grimberg
2021-12-14 12:00                       ` Max Gurtovoy
2021-12-15  1:15                         ` Yi Zhang
2021-12-15 12:10                           ` Max Gurtovoy
2021-12-16  2:18                             ` Yi Zhang
2021-12-16 13:21                               ` Max Gurtovoy
2021-12-16 16:32                                 ` Yi Zhang
2021-12-16 17:33                                   ` Haakon Bugge
2021-12-17  7:03                                     ` Yi Zhang
2021-12-17 11:19                                       ` Haakon Bugge
2022-02-14  9:47                                         ` Yi Zhang
2022-02-14 11:00                                           ` Chaitanya Kulkarni
2022-02-14 11:32                                           ` Sagi Grimberg
2022-02-14 12:11                                             ` Max Gurtovoy
2022-02-15 13:52                                               ` Yi Zhang
2022-02-15 14:30                                                 ` Max Gurtovoy
2022-02-21 10:00                                                   ` Yi Zhang
2022-02-23 10:04                                                     ` Max Gurtovoy
2022-02-23 10:30                                                       ` Sagi Grimberg
2022-02-23 11:20                                                         ` Max Gurtovoy
2022-03-01  0:06                                                       ` Yi Zhang
2022-03-16 15:16                                                         ` Sagi Grimberg
2022-03-19  7:29                                                           ` Yi Zhang
2022-03-20 10:50                                                             ` Max Gurtovoy
2022-03-20 13:03                                                               ` Sagi Grimberg
2022-03-20 15:11                                                                 ` Max Gurtovoy
2022-03-21  9:28                                                                   ` Sagi Grimberg
2022-03-21 12:11                                                                     ` Max Gurtovoy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=84208be5-a7a9-5261-398c-fa9bda3efbe3@grimberg.me \
    --to=sagi@grimberg.me \
    --cc=linux-nvme@lists.infradead.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=yi.zhang@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.