Re: more problems with NFS. sort of repeatable problem with vmplayer

From: bfields@fieldses.org (J. Bruce Fields)
To: Charles Hedrick <hedrick@cs.rutgers.edu>
Cc: Charles Hedrick <hedrick@rutgers.edu>,
	Chuck Lever III <chuck.lever@oracle.com>,
	Benjamin Coddington <bcodding@redhat.com>,
	Patrick Goetz <pgoetz@math.utexas.edu>,
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: more problems with NFS. sort of repeatable problem with vmplayer
Date: Mon, 11 Oct 2021 10:45:52 -0400	[thread overview]
Message-ID: <20211011144552.GD22387@fieldses.org> (raw)
In-Reply-To: <2C0DB856-23E5-41A2-A9F5-6E64F5A9FCB6@cs.rutgers.edu>

On Tue, Oct 05, 2021 at 03:46:21PM -0400, Charles Hedrick wrote:
> We just found a nearly repeatable problem. If you run vmplayer (a desktop VM system from VMware). with its vm storage on NFS, the system eventually locks up. Some of the time. It happens consistently for one user, and I just saw it.

Could you explain what you mean by "locks up"?  Is one application
hanging, or is the whole machine unresponsive until rebooted?

> When we told the user for which it is consistent to move his vm’s to local storage, the problem went away.
> 
> It tried running vmplayer. Shortly after starting to create a new VM, vmplayer hung. I had another window with a shell. I went into the directory with the vm files and did “ls -ltrc”. It didn’t quite hang, but look about a minute to finish I also saw log entries from VMware complaining that disk operations took several seconds.

Any kernel messages in the log around that time?

> We saw this problem last semester consistentl, though I didn’t realize
> a connection with vmplayer (if it existed). We fixed it by forcing
> mounts to use NFS 4.0. Since delegations are now disabled on our
> server, I’m assuming that the problem is locking. We don’t normally
> use locking a lot, but I believe that VMware uses it extensively. 

So you're normally using NFSv4.2?

--b.

> 
> The problem occurs on Ubuntu 20.04 with both the normal (5.4) and HWE
> (5.11) kernels.
> 
> Any thoughts? At the moment I’m tempted to force 4.0, but I’d like to
> be able to use 4.2 at some point. Since it still happens with 5.11 it
> doesn’t look good. I’m willing to try a more recent kernel if it’s
> likely to help.
> 
> We’re probably an unusual installation. We’re a CS department, with
> researchers and also a large time-sharing environment for students
> (spread across many machines, with a graphical interface using Xrdb,
> etc). Our people use every piece of software under the sun.
> 
> Client and server are both Ubuntu 20.04. Server is on ZFS with NVMe
> storage.