All of lore.kernel.org
 help / color / mirror / Atom feed
From: Yi Zhang <yi.zhang@redhat.com>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org
Subject: Re: [bug report] NVMe/IB: reset_controller need more than 1min
Date: Sat, 22 May 2021 12:27:36 +0800	[thread overview]
Message-ID: <CAHj4cs_3eLZd=vxRRrnBU2W4H38mqttcy0ZdSw6uw4KvbJWgeQ@mail.gmail.com> (raw)
In-Reply-To: <3c86dc88-97d9-5a71-20e1-a90279f47db5@grimberg.me>

On Sat, May 22, 2021 at 2:00 AM Sagi Grimberg <sagi@grimberg.me> wrote:
>
>
> > Hi
> > I found this issue on 5.13-rc2 with NVMe/IB environment, could anyone
> > help check it?
> > Thanks.
> >
> > $ time echo 1 >/sys/block/nvme0n1/device/reset_controller
> > real 0m10.678s
> > user 0m0.000s
> > sys 0m0.000s
> > $ time echo 1 >/sys/block/nvme0n1/device/reset_controller
> > real 1m11.530s
> > user 0m0.000s
> > sys 0m0.000s
> >
> > target:
> > $ dmesg | grep nvme
> > [  276.891454] nvmet: creating controller 1 for subsystem testnqn for
> > NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> > [  287.374412] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
> > [  287.399317] nvmet: ctrl 1 fatal error occurred!
> > [  348.412672] nvmet: creating controller 1 for subsystem testnqn for
> > NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> >
> > client:
> > $ dmesg | grep nvme
> > [  281.704475] nvme nvme0: creating 40 I/O queues.
> > [  285.557759] nvme nvme0: mapped 40/0/0 default/read/poll queues.
> > [  353.187809] nvme nvme0: I/O 8 QID 0 timeout
> > [  353.193100] nvme nvme0: Property Set error: 881, offset 0x14
> > [  353.226082] nvme nvme0: creating 40 I/O queues.
> > [  357.088266] nvme nvme0: mapped 40/0/0 default/read/poll queues.
>
> It appears that there is an admin timeout that is either triggered
> by the reset or unrelated.
>
> Can you run nvme reset /dev/nvme0 instead so we can see the "resetting
> controller" print?
>
Yes, here is the log:
------------------------0
+ nvme reset /dev/nvme0
real 0m10.737s
user 0m0.004s
sys 0m0.004s
------------------------1
+ nvme reset /dev/nvme0
real 1m11.335s
user 0m0.002s
sys 0m0.005s

target:
[  934.306016] nvmet: creating controller 1 for subsystem testnqn for
NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
[  944.875021] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
[  944.900051] nvmet: ctrl 1 fatal error occurred!
[ 1005.628340] nvmet: creating controller 1 for subsystem testnqn for
NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.

client:
[  857.264029] nvme nvme0: resetting controller
[  864.115369] nvme nvme0: creating 40 I/O queues.
[  867.996746] nvme nvme0: mapped 40/0/0 default/read/poll queues.
[  868.001673] nvme nvme0: resetting controller
[  935.396789] nvme nvme0: I/O 9 QID 0 timeout
[  935.402036] nvme nvme0: Property Set error: 881, offset 0x14
[  935.438080] nvme nvme0: creating 40 I/O queues.
[  939.332125] nvme nvme0: mapped 40/0/0 default/read/poll queues.


-- 
Best Regards,
  Yi Zhang


WARNING: multiple messages have this Message-ID (diff)
From: Yi Zhang <yi.zhang@redhat.com>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org
Subject: Re: [bug report] NVMe/IB: reset_controller need more than 1min
Date: Sat, 22 May 2021 12:27:36 +0800	[thread overview]
Message-ID: <CAHj4cs_3eLZd=vxRRrnBU2W4H38mqttcy0ZdSw6uw4KvbJWgeQ@mail.gmail.com> (raw)
In-Reply-To: <3c86dc88-97d9-5a71-20e1-a90279f47db5@grimberg.me>

On Sat, May 22, 2021 at 2:00 AM Sagi Grimberg <sagi@grimberg.me> wrote:
>
>
> > Hi
> > I found this issue on 5.13-rc2 with NVMe/IB environment, could anyone
> > help check it?
> > Thanks.
> >
> > $ time echo 1 >/sys/block/nvme0n1/device/reset_controller
> > real 0m10.678s
> > user 0m0.000s
> > sys 0m0.000s
> > $ time echo 1 >/sys/block/nvme0n1/device/reset_controller
> > real 1m11.530s
> > user 0m0.000s
> > sys 0m0.000s
> >
> > target:
> > $ dmesg | grep nvme
> > [  276.891454] nvmet: creating controller 1 for subsystem testnqn for
> > NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> > [  287.374412] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
> > [  287.399317] nvmet: ctrl 1 fatal error occurred!
> > [  348.412672] nvmet: creating controller 1 for subsystem testnqn for
> > NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
> >
> > client:
> > $ dmesg | grep nvme
> > [  281.704475] nvme nvme0: creating 40 I/O queues.
> > [  285.557759] nvme nvme0: mapped 40/0/0 default/read/poll queues.
> > [  353.187809] nvme nvme0: I/O 8 QID 0 timeout
> > [  353.193100] nvme nvme0: Property Set error: 881, offset 0x14
> > [  353.226082] nvme nvme0: creating 40 I/O queues.
> > [  357.088266] nvme nvme0: mapped 40/0/0 default/read/poll queues.
>
> It appears that there is an admin timeout that is either triggered
> by the reset or unrelated.
>
> Can you run nvme reset /dev/nvme0 instead so we can see the "resetting
> controller" print?
>
Yes, here is the log:
------------------------0
+ nvme reset /dev/nvme0
real 0m10.737s
user 0m0.004s
sys 0m0.004s
------------------------1
+ nvme reset /dev/nvme0
real 1m11.335s
user 0m0.002s
sys 0m0.005s

target:
[  934.306016] nvmet: creating controller 1 for subsystem testnqn for
NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.
[  944.875021] nvmet: ctrl 1 keep-alive timer (5 seconds) expired!
[  944.900051] nvmet: ctrl 1 fatal error occurred!
[ 1005.628340] nvmet: creating controller 1 for subsystem testnqn for
NQN nqn.2014-08.org.nvmexpress:uuid:4c4c4544-0056-4c10-8058-b7c04f383432.

client:
[  857.264029] nvme nvme0: resetting controller
[  864.115369] nvme nvme0: creating 40 I/O queues.
[  867.996746] nvme nvme0: mapped 40/0/0 default/read/poll queues.
[  868.001673] nvme nvme0: resetting controller
[  935.396789] nvme nvme0: I/O 9 QID 0 timeout
[  935.402036] nvme nvme0: Property Set error: 881, offset 0x14
[  935.438080] nvme nvme0: creating 40 I/O queues.
[  939.332125] nvme nvme0: mapped 40/0/0 default/read/poll queues.


-- 
Best Regards,
  Yi Zhang


_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

  reply	other threads:[~2021-05-22  4:27 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-05-21 16:12 [bug report] NVMe/IB: reset_controller need more than 1min Yi Zhang
2021-05-21 16:12 ` Yi Zhang
2021-05-21 18:00 ` Sagi Grimberg
2021-05-21 18:00   ` Sagi Grimberg
2021-05-22  4:27   ` Yi Zhang [this message]
2021-05-22  4:27     ` Yi Zhang
2021-06-23 10:01     ` Yi Zhang
2021-06-23 10:01       ` Yi Zhang
2021-06-23 21:32       ` Sagi Grimberg
2021-06-23 21:32         ` Sagi Grimberg
2021-06-24 16:14         ` Yi Zhang
2021-06-24 16:14           ` Yi Zhang
2021-12-11  3:01           ` Yi Zhang
2021-12-12  9:45             ` Sagi Grimberg
2021-12-13  6:12               ` Yi Zhang
2021-12-13  9:04                 ` Sagi Grimberg
2021-12-13 17:05                   ` Yi Zhang
2021-12-14 10:39                     ` Sagi Grimberg
2021-12-14 12:00                       ` Max Gurtovoy
2021-12-15  1:15                         ` Yi Zhang
2021-12-15 12:10                           ` Max Gurtovoy
2021-12-16  2:18                             ` Yi Zhang
2021-12-16 13:21                               ` Max Gurtovoy
2021-12-16 16:32                                 ` Yi Zhang
2021-12-16 17:33                                   ` Haakon Bugge
2021-12-17  7:03                                     ` Yi Zhang
2021-12-17 11:19                                       ` Haakon Bugge
2022-02-14  9:47                                         ` Yi Zhang
2022-02-14 11:00                                           ` Chaitanya Kulkarni
2022-02-14 11:32                                           ` Sagi Grimberg
2022-02-14 12:11                                             ` Max Gurtovoy
2022-02-15 13:52                                               ` Yi Zhang
2022-02-15 14:30                                                 ` Max Gurtovoy
2022-02-21 10:00                                                   ` Yi Zhang
2022-02-23 10:04                                                     ` Max Gurtovoy
2022-02-23 10:30                                                       ` Sagi Grimberg
2022-02-23 11:20                                                         ` Max Gurtovoy
2022-03-01  0:06                                                       ` Yi Zhang
2022-03-16 15:16                                                         ` Sagi Grimberg
2022-03-19  7:29                                                           ` Yi Zhang
2022-03-20 10:50                                                             ` Max Gurtovoy
2022-03-20 13:03                                                               ` Sagi Grimberg
2022-03-20 15:11                                                                 ` Max Gurtovoy
2022-03-21  9:28                                                                   ` Sagi Grimberg
2022-03-21 12:11                                                                     ` Max Gurtovoy

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAHj4cs_3eLZd=vxRRrnBU2W4H38mqttcy0ZdSw6uw4KvbJWgeQ@mail.gmail.com' \
    --to=yi.zhang@redhat.com \
    --cc=linux-nvme@lists.infradead.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.