From: Yi Zhang <yi.zhang@redhat.com> To: Sagi Grimberg <sagi@grimberg.me> Cc: linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org Subject: Re: [bug report] NVMe/IB: reset_controller need more than 1min Date: Fri, 25 Jun 2021 00:14:42 +0800 [thread overview] Message-ID: <CAHj4cs8dgNNE5qcX3Y4ykuTYU8z_kea6=q64Pn_2vsdodgOJZQ@mail.gmail.com> (raw) In-Reply-To: <84208be5-a7a9-5261-398c-fa9bda3efbe3@grimberg.me> On Thu, Jun 24, 2021 at 5:32 AM Sagi Grimberg <sagi@grimberg.me> wrote: > > > > Hello > > > > Gentle ping here, this issue still exists on latest 5.13-rc7 > > > > # time nvme reset /dev/nvme0 > > > > real 0m12.636s > > user 0m0.002s > > sys 0m0.005s > > # time nvme reset /dev/nvme0 > > > > real 0m12.641s > > user 0m0.000s > > sys 0m0.007s > > Strange that even normal resets take so long... > What device are you using? Hi Sagi Here is the device info: Mellanox Technologies MT27700 Family [ConnectX-4] > > > # time nvme reset /dev/nvme0 > > > > real 1m16.133s > > user 0m0.000s > > sys 0m0.007s > > There seems to be a spurious command timeout here, but maybe this > is due to the fact that the queues take so long to connect and > the target expires the keep-alive timer. > > Does this patch help? The issue still exists, let me know if you need more testing for it. :) > -- > diff --git a/drivers/nvme/target/fabrics-cmd.c > b/drivers/nvme/target/fabrics-cmd.c > index 7d0f3523fdab..f4a7db1ab3e5 100644 > --- a/drivers/nvme/target/fabrics-cmd.c > +++ b/drivers/nvme/target/fabrics-cmd.c > @@ -142,6 +142,14 @@ static u16 nvmet_install_queue(struct nvmet_ctrl > *ctrl, struct nvmet_req *req) > } > } > > + /* > + * Controller establishment flow may take some time, and the > host may not > + * send us keep-alive during this period, hence reset the > + * traffic based keep-alive timer so we don't trigger a > + * controller teardown as a result of a keep-alive expiration. > + */ > + ctrl->reset_tbkas = true; > + > return 0; > > err: > -- > > >> target: > >> [ 934.306016] nvmet: creating controller 1 for subsystem testnqn for > >> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432. > >> [ 944.875021] nvmet: ctrl 1 keep-alive timer (5 seconds) expired! > >> [ 944.900051] nvmet: ctrl 1 fatal error occurred! > >> [ 1005.628340] nvmet: creating controller 1 for subsystem testnqn for > >> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432. > >> > >> client: > >> [ 857.264029] nvme nvme0: resetting controller > >> [ 864.115369] nvme nvme0: creating 40 I/O queues. > >> [ 867.996746] nvme nvme0: mapped 40/0/0 default/read/poll queues. > >> [ 868.001673] nvme nvme0: resetting controller > >> [ 935.396789] nvme nvme0: I/O 9 QID 0 timeout > >> [ 935.402036] nvme nvme0: Property Set error: 881, offset 0x14 > >> [ 935.438080] nvme nvme0: creating 40 I/O queues. > >> [ 939.332125] nvme nvme0: mapped 40/0/0 default/read/poll queues. > -- Best Regards, Yi Zhang
WARNING: multiple messages have this Message-ID (diff)
From: Yi Zhang <yi.zhang@redhat.com> To: Sagi Grimberg <sagi@grimberg.me> Cc: linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org Subject: Re: [bug report] NVMe/IB: reset_controller need more than 1min Date: Fri, 25 Jun 2021 00:14:42 +0800 [thread overview] Message-ID: <CAHj4cs8dgNNE5qcX3Y4ykuTYU8z_kea6=q64Pn_2vsdodgOJZQ@mail.gmail.com> (raw) In-Reply-To: <84208be5-a7a9-5261-398c-fa9bda3efbe3@grimberg.me> On Thu, Jun 24, 2021 at 5:32 AM Sagi Grimberg <sagi@grimberg.me> wrote: > > > > Hello > > > > Gentle ping here, this issue still exists on latest 5.13-rc7 > > > > # time nvme reset /dev/nvme0 > > > > real 0m12.636s > > user 0m0.002s > > sys 0m0.005s > > # time nvme reset /dev/nvme0 > > > > real 0m12.641s > > user 0m0.000s > > sys 0m0.007s > > Strange that even normal resets take so long... > What device are you using? Hi Sagi Here is the device info: Mellanox Technologies MT27700 Family [ConnectX-4] > > > # time nvme reset /dev/nvme0 > > > > real 1m16.133s > > user 0m0.000s > > sys 0m0.007s > > There seems to be a spurious command timeout here, but maybe this > is due to the fact that the queues take so long to connect and > the target expires the keep-alive timer. > > Does this patch help? The issue still exists, let me know if you need more testing for it. :) > -- > diff --git a/drivers/nvme/target/fabrics-cmd.c > b/drivers/nvme/target/fabrics-cmd.c > index 7d0f3523fdab..f4a7db1ab3e5 100644 > --- a/drivers/nvme/target/fabrics-cmd.c > +++ b/drivers/nvme/target/fabrics-cmd.c > @@ -142,6 +142,14 @@ static u16 nvmet_install_queue(struct nvmet_ctrl > *ctrl, struct nvmet_req *req) > } > } > > + /* > + * Controller establishment flow may take some time, and the > host may not > + * send us keep-alive during this period, hence reset the > + * traffic based keep-alive timer so we don't trigger a > + * controller teardown as a result of a keep-alive expiration. > + */ > + ctrl->reset_tbkas = true; > + > return 0; > > err: > -- > > >> target: > >> [ 934.306016] nvmet: creating controller 1 for subsystem testnqn for > >> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432. > >> [ 944.875021] nvmet: ctrl 1 keep-alive timer (5 seconds) expired! > >> [ 944.900051] nvmet: ctrl 1 fatal error occurred! > >> [ 1005.628340] nvmet: creating controller 1 for subsystem testnqn for > >> NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432. > >> > >> client: > >> [ 857.264029] nvme nvme0: resetting controller > >> [ 864.115369] nvme nvme0: creating 40 I/O queues. > >> [ 867.996746] nvme nvme0: mapped 40/0/0 default/read/poll queues. > >> [ 868.001673] nvme nvme0: resetting controller > >> [ 935.396789] nvme nvme0: I/O 9 QID 0 timeout > >> [ 935.402036] nvme nvme0: Property Set error: 881, offset 0x14 > >> [ 935.438080] nvme nvme0: creating 40 I/O queues. > >> [ 939.332125] nvme nvme0: mapped 40/0/0 default/read/poll queues. > -- Best Regards, Yi Zhang _______________________________________________ Linux-nvme mailing list Linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme
next prev parent reply other threads:[~2021-06-24 16:15 UTC|newest] Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-05-21 16:12 [bug report] NVMe/IB: reset_controller need more than 1min Yi Zhang 2021-05-21 16:12 ` Yi Zhang 2021-05-21 18:00 ` Sagi Grimberg 2021-05-21 18:00 ` Sagi Grimberg 2021-05-22 4:27 ` Yi Zhang 2021-05-22 4:27 ` Yi Zhang 2021-06-23 10:01 ` Yi Zhang 2021-06-23 10:01 ` Yi Zhang 2021-06-23 21:32 ` Sagi Grimberg 2021-06-23 21:32 ` Sagi Grimberg 2021-06-24 16:14 ` Yi Zhang [this message] 2021-06-24 16:14 ` Yi Zhang 2021-12-11 3:01 ` Yi Zhang 2021-12-12 9:45 ` Sagi Grimberg 2021-12-13 6:12 ` Yi Zhang 2021-12-13 9:04 ` Sagi Grimberg 2021-12-13 17:05 ` Yi Zhang 2021-12-14 10:39 ` Sagi Grimberg 2021-12-14 12:00 ` Max Gurtovoy 2021-12-15 1:15 ` Yi Zhang 2021-12-15 12:10 ` Max Gurtovoy 2021-12-16 2:18 ` Yi Zhang 2021-12-16 13:21 ` Max Gurtovoy 2021-12-16 16:32 ` Yi Zhang 2021-12-16 17:33 ` Haakon Bugge 2021-12-17 7:03 ` Yi Zhang 2021-12-17 11:19 ` Haakon Bugge 2022-02-14 9:47 ` Yi Zhang 2022-02-14 11:00 ` Chaitanya Kulkarni 2022-02-14 11:32 ` Sagi Grimberg 2022-02-14 12:11 ` Max Gurtovoy 2022-02-15 13:52 ` Yi Zhang 2022-02-15 14:30 ` Max Gurtovoy 2022-02-21 10:00 ` Yi Zhang 2022-02-23 10:04 ` Max Gurtovoy 2022-02-23 10:30 ` Sagi Grimberg 2022-02-23 11:20 ` Max Gurtovoy 2022-03-01 0:06 ` Yi Zhang 2022-03-16 15:16 ` Sagi Grimberg 2022-03-19 7:29 ` Yi Zhang 2022-03-20 10:50 ` Max Gurtovoy 2022-03-20 13:03 ` Sagi Grimberg 2022-03-20 15:11 ` Max Gurtovoy 2022-03-21 9:28 ` Sagi Grimberg 2022-03-21 12:11 ` Max Gurtovoy
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to='CAHj4cs8dgNNE5qcX3Y4ykuTYU8z_kea6=q64Pn_2vsdodgOJZQ@mail.gmail.com' \ --to=yi.zhang@redhat.com \ --cc=linux-nvme@lists.infradead.org \ --cc=linux-rdma@vger.kernel.org \ --cc=sagi@grimberg.me \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.