From: Yi Zhang <yi.zhang@redhat.com>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: Max Gurtovoy <maxg@mellanox.com>, linux-nvme@lists.infradead.org
Subject: Re: NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background
Date: Fri, 27 Sep 2019 04:17:35 -0400 (EDT) [thread overview]
Message-ID: <1512861450.3461442.1569572255032.JavaMail.zimbra@redhat.com> (raw)
In-Reply-To: <0a8fffe2-f607-26ac-43f5-86182b1c782e@mellanox.com>
Hi Sagi
Confirmed the NULL pointer issue was fixed by this patch, and "I/O 1 QID 0 timeout" still exists, thanks.
<6>[ 5807.293577] nvme nvme2: creating 48 I/O queues.
<6>[ 5817.210168] nvme nvme2: Removing ctrl: NQN "testnqn"
<6>[ 5821.130348] nvme nvme2: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.186:4420
<6>[ 5821.139830] nvme nvme2: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
<6>[ 5821.185128] nvme nvme2: creating 48 I/O queues.
<6>[ 5821.925736] nvme nvme2: mapped 48/0/0 default/read/poll queues.
<6>[ 5821.950855] nvme nvme2: new ctrl: NQN "testnqn", addr 172.31.0.186:4420
<6>[ 5821.953152] nvme2n1: detected capacity change from 0 to 1600321314816
<4>[ 5826.546586] nvme_ns_head_make_request: 250644 callbacks suppressed
<4>[ 5826.546589] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546591] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546606] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546609] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546624] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546627] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546629] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546633] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546635] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546636] block nvme2n1: no usable path - requeuing I/O
<4>[ 5837.481054] hfi1_opa0.8024: P_Key 0x8024 is not found
<4>[ 5837.486199] hfi1_opa0.8022: P_Key 0x8022 is not found
<6>[ 5837.503278] IPv6: ADDRCONF(NETDEV_CHANGE): hfi1_opa0: link becomes ready
<4>[ 5882.465388] hfi1_opa0.8024: P_Key 0x8024 is not found
<4>[ 5882.470520] hfi1_opa0.8022: P_Key 0x8022 is not found
<6>[ 5882.487647] IPv6: ADDRCONF(NETDEV_CHANGE): hfi1_opa0: link becomes ready
<4>[ 5888.515395] nvme nvme2: I/O 1 QID 0 timeout
<3>[ 5888.533361] nvme nvme2: Connect command failed, error wo/DNR bit: 7
<6>[ 5888.539645] nvme nvme2: failed to connect queue: 0 ret=7
<6>[ 5888.544994] nvme nvme2: Reconnecting in 10 seconds...
<6>[ 5898.774955] nvme nvme2: creating 48 I/O queues.
<6>[ 5899.570053] nvme nvme2: Successfully reconnected (2 attempts)
<4>[ 5927.818466] hfi1_opa0.8024: P_Key 0x8024 is not found
<4>[ 5927.823550] hfi1_opa0.8022: P_Key 0x8022 is not found
<6>[ 5927.843972] IPv6: ADDRCONF(NETDEV_CHANGE): hfi1_opa0: link becomes ready
<6>[ 6004.970479] nvme nvme2: Removing ctrl: NQN "testnqn"
Best Regards,
Yi Zhang
----- Original Message -----
From: "Max Gurtovoy" <maxg@mellanox.com>
To: "Sagi Grimberg" <sagi@grimberg.me>, "Yi Zhang" <yi.zhang@redhat.com>, linux-nvme@lists.infradead.org
Sent: Monday, September 23, 2019 11:25:31 PM
Subject: Re: NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background
Any update Yi ?
we must fix this issue..
On 9/20/2019 7:58 PM, Sagi Grimberg wrote:
> Thanks for reporting Yi,
>
> Does this fix your issue?
>
> --
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index dfa07bb9dfeb..981da9ce3cfc 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -614,7 +614,8 @@ static int nvme_rdma_start_queue(struct
> nvme_rdma_ctrl *ctrl, int idx)
> if (!ret) {
> set_bit(NVME_RDMA_Q_LIVE, &queue->flags);
> } else {
> - __nvme_rdma_stop_queue(queue);
> + if (test_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
> + __nvme_rdma_stop_queue(queue);
> dev_info(ctrl->ctrl.device,
> "failed to connect queue: %d ret=%d\n", idx,
> ret);
> }
> --
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
prev parent reply other threads:[~2019-09-27 8:17 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <1823328454.445263.1568796846850.JavaMail.zimbra@redhat.com>
2019-09-18 9:13 ` NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background Yi Zhang
2019-09-18 14:21 ` Max Gurtovoy
2019-09-20 3:37 ` Yi Zhang
2019-09-20 16:58 ` Sagi Grimberg
2019-09-23 15:25 ` Max Gurtovoy
2019-09-24 4:52 ` Yi Zhang
2019-09-27 8:17 ` Yi Zhang [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1512861450.3461442.1569572255032.JavaMail.zimbra@redhat.com \
--to=yi.zhang@redhat.com \
--cc=linux-nvme@lists.infradead.org \
--cc=maxg@mellanox.com \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).