Frequent reconnections / session startups?

[James Wettenhall <james.wettenhall@monash.edu>]

Hi,

We run a Django / Celery application which makes heavy use of CIFS
mounts.  We are experiencing frequent reconnections / session startups
and would like to understand how to avoid hammering the CIFS server
and/or the authentication server.  We've had multiple reports of
DoS-like hammering from server admins, causing frequent
re-authentication attempts and in one case causing core dumps on the
CIFS server.

Our CIFS client VMs have the following:

OS: Ubuntu 18.04.3
Kernel: 4.15.0-58-generic
mount.cifs: 6.8

Current mount options:
rw,relatime,vers=3.0,sec=ntlmssp,cache=strict,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1

We don't run the CIFS server, but we can request any information
required to diagnose the issue.

Over the past 10 hours, one of our virtual machine's kernel log has accumulated:

8453 kern.log messages including "CIFS"

To break that down, we have:

8305 "Free previous auth_key.response" messages
111 "validate protocol negotiate failed: -11" messages
26 "Close unmatched open" messages
7 "has not responded in 120 seconds" messages
4  "cifs_mount failed w/return code = -11" messages

The server is an HSM (Hierarchical Storage Management) system, so it
can be slow to respond if our application requests a file which is
only available on tape, not on disk.

The most common operation our application is performing on the
CIFS-mounted files is calculating MD5 checksums - with many Celery
worker processes running concurrently.

We would appreciate any advice on how to investigate further.

Thanks,
James