"nvme disconnect" for mounted volumes - stuck thread

"nvme disconnect" for mounted volumes - stuck thread

[Szymon Scharmach]

Hi,

In the deployment scenario i am using we use around 200 NvmeOF volumes
that are being connected to different initiators and after data have
been saved - reconnected to different ones.
nvme cli allows to disconnect volumes that are either mounted or
umount is in progress which leads threads to end up in uninterruptible
sleep. (D state)

root      69368  0.0  0.0   6704   640 ?        D    09:29   0:00 nvme
disconnect --nqn=pvc-4f6b7502-cb08-11e9-a2bf-3cfdfeb878d0
root      69375  0.0  0.0   5920   900 ?        D    09:29   0:00
umount /var/lib/origin/openshift.local.volumes/pods/4f6caee2-cb08-11e9-a2bf-3cfdfe-b878d0/
volumes/kubernetes.io~csi/pvc-4f6b7502-cb08-11e9-a2bf-3c

Those threads are stuck in D forever and whole block stack on the node
becomes unusable.
Is it the responsibility of nvme-cli or kernel module (nvme_fabrics
AFAIK) to block 'disconnect' in that case - or is it expected behavior
(nvme disconnect with force flag maybe) ?


BR\
Szymon Scharmach

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: "nvme disconnect" for mounted volumes - stuck thread

[Sagi Grimberg]

> Hi,
> 
> In the deployment scenario i am using we use around 200 NvmeOF volumes
> that are being connected to different initiators and after data have
> been saved - reconnected to different ones.
> nvme cli allows to disconnect volumes that are either mounted or
> umount is in progress which leads threads to end up in uninterruptible
> sleep. (D state)
> 
> root      69368  0.0  0.0   6704   640 ?        D    09:29   0:00 nvme
> disconnect --nqn=pvc-4f6b7502-cb08-11e9-a2bf-3cfdfeb878d0
> root      69375  0.0  0.0   5920   900 ?        D    09:29   0:00
> umount /var/lib/origin/openshift.local.volumes/pods/4f6caee2-cb08-11e9-a2bf-3cfdfe-b878d0/
> volumes/kubernetes.io~csi/pvc-4f6b7502-cb08-11e9-a2bf-3c

Do you have the watchdog hand detector trace to share?

> 
> Those threads are stuck in D forever and whole block stack on the node
> becomes unusable.
> Is it the responsibility of nvme-cli or kernel module (nvme_fabrics
> AFAIK) to block 'disconnect' in that case - or is it expected behavior
> (nvme disconnect with force flag maybe) ?

None, I guess it should be something like _netdev fstab entry...

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme