Stability problem with 70+ simultaneous CIFS mounts

Stability problem with 70+ simultaneous CIFS mounts

[SNSY (Sune Stjerneby)]

Dear Linux CIFS list,

We are experiencing an issue when the number of simultaneous mounts reaches about 70.

The users experience that they cannot access their mounted shares, though only periodically.

The shares are permanently mounted in the users homedirs by root (using a suid wrapper and script).

Example error messages seen from the user:

[user@myclient]$ ls hdrive
ls: cannot access 'hdrive': Permission denied
[user@myclient]$ ls projstat
ls: cannot access 'projstat': Resource temporarily unavailable

After some retries, they regain access. It can be very sporadic, for instance an "ls <mount>" will fail, but moments later a "cd <mount>" will succeed.

Sometimes it is necessary to unmount and mount the shares in order to regain access.

In /var/log/messages we find messages such as these, correlating with times when the users report the issue:

Nov 2 15:27:31 myclient kernel: cifs_setup_session: 40 callbacks suppressed
Nov 2 15:27:31 myclient kernel: CIFS VFS: \\myserver Send error in SessSetup = -13
Nov 2 15:27:31 myclient kernel: Status code returned 0xc000006d STATUS_LOGON_FAILURE

The messages repeat several times.

The client-specific log (/var/log/samba/log.myclient) on the Samba server does not show any messages that would correlate in time with the issues on the client.

We have two identical Linux clients that are not experiencing the issue. The number of mounts on those clients is much lower.

In addition, we have a large number of Windows 10 and Windows 2016 clients that do not experience any issue at all with the Samba server.

The environment:

Client:
Red Hat Enterprise Linux release 8.2 (Ootpa)
Kernel version:                            Linux myclient 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Wed Aug 26 15:29:02 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
mount.cifs version:               6.8

Server:
HP-UX B.11.31
HPE CIFS Server B.04.05.15.00, based on Samba 4.5.15

Any suggestions on how to proceed? I suppose the next step for us is to enable additional debug, and collect tcpdump captures.

_____________________________________________ 
Sune Stjerneby
Senior Operations Specialist (UNIX)
662.01 IT Platform & Technology Services

NNIT A/S
Østmarken 3A
DK-2860 Søborg
Denmark
+45 3079 9461 (mobile)
snsy@nnit.com

This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.


_____________________________________________ 
Sune Stjerneby
Senior Operations Specialist (UNIX)
662.01 IT Platform & Technology Services

NNIT A/S
Østmarken 3A
DK-2860 Søborg
Denmark
+45 3079 9461 (mobile)
snsy@nnit.com

This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.

Re: Stability problem with 70+ simultaneous CIFS mounts

[Aurélien Aptel]

Hi,

"SNSY (Sune Stjerneby)" <SNSY@nnit.com> writes:
> Nov 2 15:27:31 myclient kernel: cifs_setup_session: 40 callbacks suppressed
> Nov 2 15:27:31 myclient kernel: CIFS VFS: \\myserver Send error in SessSetup = -13
> Nov 2 15:27:31 myclient kernel: Status code returned 0xc000006d STATUS_LOGON_FAILURE

It looks like login failures. Are all mount points connecting to a share
on the same server?

> Any suggestions on how to proceed? I suppose the next step for us is to enable additional debug, and collect tcpdump captures.

Yes. You could try more recent kernels as well.

Cheers,
-- 
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

RE: Stability problem with 70+ simultaneous CIFS mounts

[SNSY (Sune Stjerneby)]

> It looks like login failures. Are all mount points connecting to a share on the same server?

Yes, all mount against the same server. Each user mounts a common share, as well as a personal share.

Extract from output of 'mount | grep myserver':

//myserver/projstat on /homedirs/myuser1/projstat type cifs (rw,relatime,vers=3.1.1,cache=strict,username=myuser1,domain=MYDOMAIN,uid=589733577,forceuid,gid=1432000513,forcegid,addr=10.0.0.1,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
//myserver/myuser1 on /homedirs/myuser1/hdrive type cifs (rw,relatime,vers=3.1.1,cache=strict,username=myuser1,domain=MYDOMAIN,uid=589733577,forceuid,gid=1432000513,forcegid,addr=10.0.0.1,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)

//myserver/projstat on /homedirs/myuser2/projstat type cifs (rw,relatime,vers=3.1.1,cache=strict,username=myuser2,domain=MYDOMAIN,uid=409183706,forceuid,gid=1432000513,forcegid,addr=10.0.0.1,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)
//myserver/myuser3 on /homedirs/myuser2/hdrive type cifs (rw,relatime,vers=3.1.1,cache=strict,username=myuser2,domain=MYDOMAIN,uid=409183706,forceuid,gid=1432000513,forcegid,addr=10.0.0.1,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)

(...)

> Yes. You could try more recent kernels as well.

We'll try that. Thanks.

//Sune

-----Original Message-----
From: Aurélien Aptel <aaptel@suse.com>
Sent: 9. november 2020 18:21
To: SNSY (Sune Stjerneby) <SNSY@nnit.com>; linux-cifs@vger.kernel.org
Subject: Re: Stability problem with 70+ simultaneous CIFS mounts

Hi,

"SNSY (Sune Stjerneby)" <SNSY@nnit.com> writes:
> Nov 2 15:27:31 myclient kernel: cifs_setup_session: 40 callbacks suppressed
> Nov 2 15:27:31 myclient kernel: CIFS VFS: \\myserver Send error in SessSetup = -13
> Nov 2 15:27:31 myclient kernel: Status code returned 0xc000006d STATUS_LOGON_FAILURE

It looks like login failures. Are all mount points connecting to a share
on the same server?

> Any suggestions on how to proceed? I suppose the next step for us is to enable additional debug, and collect tcpdump captures.

Yes. You could try more recent kernels as well.

Cheers,
--
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97  8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)

This e-mail (including any attachments) is intended for the addressee(s) stated above only and may contain confidential information protected by law. You are hereby notified that any unauthorized reading, disclosure, copying or distribution of this e-mail or use of information contained herein is strictly prohibited and may violate rights to proprietary information. If you are not an intended recipient, please return this e-mail to the sender and delete it immediately hereafter. Thank you.