[SPDK] SPDK NVMF target hangs when shutting down

[SPDK] SPDK NVMF target hangs when shutting down

[Wenhua Liu]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

Hi,


I've been using SPDK NVMF target over a month and it worked fine. Yesterday, I checked out the latest SPDK source code from https://github.com/spdk/spdk up to commit 5ee4728d0cc05e506fba2b12445d93d088188fe3. This time, when SPDK NVMF target is running, if I press ctrl-C, SPDK does not shutdown as before, it just prints some message and hangs there. Here is the message:


^C
=========================
   NVMF shutdown signal
=========================
[nvmf] subsystem.c: 224:spdk_nvmf_delete_subsystem: subsystem is 0x559509c81f00
[nvmf] subsystem.c: 224:spdk_nvmf_delete_subsystem: subsystem is 0x559509c82c80
p@?

Has anyone seen this? Is this something wrong in SPDK or some configuration issue? If it's configuration issue, how should I fix it?


Thanks,

-Wenhua


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2137 bytes --]

Re: [SPDK] SPDK NVMF target hangs when shutting down

[Wenhua Liu]

[-- Attachment #1: Type: text/plain, Size: 4769 bytes --]

You're right, it's rdma_destroy_id being blocked. Here is the stack trace from gdb:


(gdb) bt
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fd4ff8fe2c3 in rdma_destroy_id () from /usr/lib/x86_64-linux-gnu/librdmacm.so.1
#2  0x0000559508ad9315 in spdk_nvmf_rdma_conn_destroy (rdma_conn=0x559509cca430) at rdma.c:213
#3  0x0000559508adc9da in spdk_nvmf_rdma_close_conn (conn=0x559509cca430) at rdma.c:1346
#4  0x0000559508ad592e in spdk_nvmf_session_destruct (session=0x559509c84320) at session.c:179
#5  0x0000559508ad3197 in spdk_nvmf_delete_subsystem (subsystem=0x559509c82c80) at subsystem.c:243
#6  0x0000559508abae24 in subsystem_delete_event (arg1=0x559509c821f0, arg2=0x0) at nvmf_tgt.c:75
#7  0x0000559508ae01ee in spdk_event_queue_run_batch (lcore=0) at reactor.c:201
#8  0x0000559508ae04b8 in _spdk_reactor_run (arg=0x559508e2f7c0 <g_reactors>) at reactor.c:315
#9  0x0000559508ae08de in spdk_reactor_start (reactor=0x559508e2f7c0 <g_reactors>) at reactor.c:419
#10 0x0000559508ae0c96 in spdk_reactors_start () at reactor.c:538
#11 0x0000559508ade738 in spdk_app_start (start_fn=0x559508abb61c <spdk_nvmf_startup>, arg1=0x0, arg2=0x0) at app.c:449
#12 0x0000559508abb9ee in main (argc=5, argv=0x7ffe82144598) at nvmf_tgt.c:471
(gdb)


One more thing I want to mention is in my test, after receving the response for that wrong format read request, my initiator crashes. That means if rdma_destroy_id expects response from initiator, it will not get it.


Thanks,
-Wenhua
________________________________
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Wenhua Liu <liuw(a)vmware.com>
Sent: Wednesday, January 25, 2017 11:16 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] SPDK NVMF target hangs when shutting down


It's pretty simple scenario. My initiator connects to the controller, creates 2 IO queues, discovers 4 namespaces, does a read at LBA 0 on first namespace, then does another read with wrong parameters which causes failure. Then I press ctlr+C and hit the problem.


Here is the diff between my nvmf.conf and the original nvmf.conf.in

vmware(a)proma-1s-dhcp84:~/spdk-new/etc/spdk$ diff nvmf.conf nvmf.conf.in
102,108c102,108
< #[Subsystem1]
< #  NQN nqn.2016-06.io.spdk:cnode1
< #  Core 0
< #  Mode Direct
< #  Listen RDMA 12.12.12.100:4420
< #  Host nqn.2016-06.io.spdk:init
< #  NVMe 0000:00:00.0
---
> [Subsystem1]
>   NQN nqn.2016-06.io.spdk:cnode1
>   Core 0
>   Mode Direct
>   Listen RDMA 15.15.15.2:4420
>   Host nqn.2016-06.io.spdk:init
>   NVMe 0000:00:00.0
116,117c116,117
<   Listen RDMA 12.12.12.100:4420
< #  Host nqn.2016-06.io.spdk:init
---
>   Listen RDMA 192.168.2.21:4420
>   Host nqn.2016-06.io.spdk:init
121,122d120
<   Namespace Malloc2
<   Namespace Malloc3
vmware(a)proma-1s-dhcp84:~/spdk-new/etc/spdk$

Thanks,

-Wenhua


________________________________
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Yang, Ziye <ziye.yang(a)intel.com>
Sent: Wednesday, January 25, 2017 10:32 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] SPDK NVMF target hangs when shutting down


Hi,



Could you provide more details, e.g., configuration and usage scenario ( For example, ctrl+c failed after IO test). I think that mostly likely, it may block by rdma_destroy_id.  Could you reproduce it and use gdb to attach nvmf target process, and use “bt” dump the stack.



Thanks.



From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wenhua Liu
Sent: Thursday, January 26, 2017 9:15 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK NVMF target hangs when shutting down



Hi,



I've been using SPDK NVMF target over a month and it worked fine. Yesterday, I checked out the latest SPDK source code from https://github.com/spdk/spdk<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_spdk_spdk&d=DwMF3g&c=uilaK90D4TOVoH58JNXRgQ&r=-xGUcQt5607zrir8TikGRQ&m=7aDI5JisX7WsKSKos0Z4hHWm_FvwmXxN3VAQ92qmyd8&s=n_Uh3JuG_1DMcRgxsfw91OYS9glh6s42vQ_PYtL-ovo&e=> up to commit 5ee4728d0cc05e506fba2b12445d93d088188fe3. This time, when SPDK NVMF target is running, if I press ctrl-C, SPDK does not shutdown as before, it just prints some message and hangs there. Here is the message:



^C

=========================

   NVMF shutdown signal

=========================

[nvmf] subsystem.c: 224:spdk_nvmf_delete_subsystem: subsystem is 0x559509c81f00

[nvmf] subsystem.c: 224:spdk_nvmf_delete_subsystem: subsystem is 0x559509c82c80

p@¦



Has anyone seen this? Is this something wrong in SPDK or some configuration issue? If it's configuration issue, how should I fix it?



Thanks,

-Wenhua



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 11838 bytes --]

Re: [SPDK] SPDK NVMF target hangs when shutting down

[Wenhua Liu]

[-- Attachment #1: Type: text/plain, Size: 2923 bytes --]

It's pretty simple scenario. My initiator connects to the controller, creates 2 IO queues, discovers 4 namespaces, does a read at LBA 0 on first namespace, then does another read with wrong parameters which causes failure. Then I press ctlr+C and hit the problem.


Here is the diff between my nvmf.conf and the original nvmf.conf.in

vmware(a)proma-1s-dhcp84:~/spdk-new/etc/spdk$ diff nvmf.conf nvmf.conf.in
102,108c102,108
< #[Subsystem1]
< #  NQN nqn.2016-06.io.spdk:cnode1
< #  Core 0
< #  Mode Direct
< #  Listen RDMA 12.12.12.100:4420
< #  Host nqn.2016-06.io.spdk:init
< #  NVMe 0000:00:00.0
---
> [Subsystem1]
>   NQN nqn.2016-06.io.spdk:cnode1
>   Core 0
>   Mode Direct
>   Listen RDMA 15.15.15.2:4420
>   Host nqn.2016-06.io.spdk:init
>   NVMe 0000:00:00.0
116,117c116,117
<   Listen RDMA 12.12.12.100:4420
< #  Host nqn.2016-06.io.spdk:init
---
>   Listen RDMA 192.168.2.21:4420
>   Host nqn.2016-06.io.spdk:init
121,122d120
<   Namespace Malloc2
<   Namespace Malloc3
vmware(a)proma-1s-dhcp84:~/spdk-new/etc/spdk$

Thanks,

-Wenhua


________________________________
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Yang, Ziye <ziye.yang(a)intel.com>
Sent: Wednesday, January 25, 2017 10:32 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] SPDK NVMF target hangs when shutting down


Hi,



Could you provide more details, e.g., configuration and usage scenario ( For example, ctrl+c failed after IO test). I think that mostly likely, it may block by rdma_destroy_id.  Could you reproduce it and use gdb to attach nvmf target process, and use "bt" dump the stack.



Thanks.



From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wenhua Liu
Sent: Thursday, January 26, 2017 9:15 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK NVMF target hangs when shutting down



Hi,



I've been using SPDK NVMF target over a month and it worked fine. Yesterday, I checked out the latest SPDK source code from https://github.com/spdk/spdk<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_spdk_spdk&d=DwMF3g&c=uilaK90D4TOVoH58JNXRgQ&r=-xGUcQt5607zrir8TikGRQ&m=7aDI5JisX7WsKSKos0Z4hHWm_FvwmXxN3VAQ92qmyd8&s=n_Uh3JuG_1DMcRgxsfw91OYS9glh6s42vQ_PYtL-ovo&e=> up to commit 5ee4728d0cc05e506fba2b12445d93d088188fe3. This time, when SPDK NVMF target is running, if I press ctrl-C, SPDK does not shutdown as before, it just prints some message and hangs there. Here is the message:



^C

=========================

   NVMF shutdown signal

=========================

[nvmf] subsystem.c: 224:spdk_nvmf_delete_subsystem: subsystem is 0x559509c81f00

[nvmf] subsystem.c: 224:spdk_nvmf_delete_subsystem: subsystem is 0x559509c82c80

p@?



Has anyone seen this? Is this something wrong in SPDK or some configuration issue? If it's configuration issue, how should I fix it?



Thanks,

-Wenhua



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9422 bytes --]

Re: [SPDK] SPDK NVMF target hangs when shutting down

[Yang, Ziye]

[-- Attachment #1: Type: text/plain, Size: 1303 bytes --]

Hi,

Could you provide more details, e.g., configuration and usage scenario ( For example, ctrl+c failed after IO test). I think that mostly likely, it may block by rdma_destroy_id.  Could you reproduce it and use gdb to attach nvmf target process, and use "bt" dump the stack.

Thanks.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wenhua Liu
Sent: Thursday, January 26, 2017 9:15 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK NVMF target hangs when shutting down


Hi,



I've been using SPDK NVMF target over a month and it worked fine. Yesterday, I checked out the latest SPDK source code from https://github.com/spdk/spdk up to commit 5ee4728d0cc05e506fba2b12445d93d088188fe3. This time, when SPDK NVMF target is running, if I press ctrl-C, SPDK does not shutdown as before, it just prints some message and hangs there. Here is the message:


^C
=========================
   NVMF shutdown signal
=========================
[nvmf] subsystem.c: 224:spdk_nvmf_delete_subsystem: subsystem is 0x559509c81f00
[nvmf] subsystem.c: 224:spdk_nvmf_delete_subsystem: subsystem is 0x559509c82c80
p@▒

Has anyone seen this? Is this something wrong in SPDK or some configuration issue? If it's configuration issue, how should I fix it?



Thanks,

-Wenhua



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7315 bytes --]

Re: [SPDK] SPDK NVMF target hangs when shutting down

[Liu, Changpeng]

[-- Attachment #1: Type: text/plain, Size: 1085 bytes --]

Thank you Wenhua.

We'll look at the issue you reported, and fix it.


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Wenhua Liu
Sent: Thursday, January 26, 2017 9:15 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK NVMF target hangs when shutting down


Hi,



I've been using SPDK NVMF target over a month and it worked fine. Yesterday, I checked out the latest SPDK source code from https://github.com/spdk/spdk up to commit 5ee4728d0cc05e506fba2b12445d93d088188fe3. This time, when SPDK NVMF target is running, if I press ctrl-C, SPDK does not shutdown as before, it just prints some message and hangs there. Here is the message:


^C
=========================
   NVMF shutdown signal
=========================
[nvmf] subsystem.c: 224:spdk_nvmf_delete_subsystem: subsystem is 0x559509c81f00
[nvmf] subsystem.c: 224:spdk_nvmf_delete_subsystem: subsystem is 0x559509c82c80
p@▒

Has anyone seen this? Is this something wrong in SPDK or some configuration issue? If it's configuration issue, how should I fix it?



Thanks,

-Wenhua



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 45156 bytes --]