CIFS mounts hanging after CIFS server restart

CIFS mounts hanging after CIFS server restart

[Robert Smith]

Hi,

We have seen an issue where a CIFS server restarting has caused client processes to hang.

Server:
Dell Isilon NAS (accessed via DFS)

Client: 
Flatcar Container Linux by Kinvolk 2605.4.0 (Oklo)
Kernel: 5.4.65-flatcar

On the client, we run "touch --no-create /mnt/cfs/.cifskeepalive" regularly to keep the CIFS connections alive (workaround for another issue where idle connections die). These touch commands started hanging (along with all other users of CIFS). I sampled /proc/*/stack for several of the touch cmds, all gave the stacktrace below. We ended up with around 700 hung touch processes.

[<0>] d_alloc_parallel+0x420/0x480
[<0>] __lookup_slow+0x6e/0x150
[<0>] lookup_slow+0x35/0x50
[<0>] walk_component+0x1bf/0x330
[<0>] path_lookupat.isra.50+0x6d/0x220
[<0>] filename_lookup.part.66+0xa0/0x170
[<0>] do_utimes+0xd9/0x160
[<0>] __x64_sys_utimensat+0x7a/0xc0
[<0>] do_syscall_64+0x4e/0x120
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

Also sampled a few cifs processes, these were all in

Robert Smith  1 day ago
# cat /proc/2477052/stack
[<0>] iterate_supers_type+0x6d/0xf0
[<0>] cifs_reconnect+0x8e/0xfd0 [cifs]
[<0>] cifs_reconnect+0xeed/0xfd0 [cifs]
[<0>] cifs_read_from_socket+0x4a/0x70 [cifs]
[<0>] cifs_handle_standard+0x298/0xd60 [cifs]
[<0>] kthread+0x112/0x130
[<0>] ret_from_fork+0x35/0x40

Sampling a few minutes apart showed voluntary_ctxt_switches increasing for a few of the 700 touch processes, so may not be totally hung.

Any ideas what's causing this?

Thanks for your help,

Rob


--------------
G-RESEARCH believes the information provided herein is reliable. While every care has been taken to ensure accuracy, the information is furnished to the recipients with no warranty as to the completeness and accuracy of its contents and on condition that any errors or omissions shall not be made the basis of any claim, demand or cause of action.
The information in this email is intended only for the named recipient.  If you are not the intended recipient please notify us immediately and do not copy, distribute or take action based on this e-mail.
All messages sent to and from this e-mail address will be logged by G-RESEARCH and are subject to archival storage, monitoring, review and disclosure. For information about how G-RESEARCH uses your personal data, please refer to our Privacy Policy at https://www.gresearch.co.uk/privacy-policy/.
G-RESEARCH is the trading name of Trenchant Limited, 5th Floor, Whittington House, 19-30 Alfred Place, London WC1E 7EA.
Trenchant Limited is a company registered in England with company number 08127121.
--------------