Re: rsize,wsize=1M causes severe lags in 10/100 Mbps

[Daniel Forrest <dforrest@wisc.edu>]

On Thu, Sep 19, 2019 at 05:42:26PM -0400, Trond Myklebust wrote:
> On Thu, 2019-09-19 at 16:19 -0500, Daniel Forrest wrote:
> > On Thu, Sep 19, 2019 at 08:40:41PM +0000, Trond Myklebust wrote:
> > > On Thu, 2019-09-19 at 23:20 +0300, Alkis Georgopoulos wrote:
> > > > On 9/19/19 11:05 PM, Trond Myklebust wrote:
> > > > > There are plenty of operations that can take longer than 700 ms
> > > > > to
> > > > > complete. Synchronous writes to disk are one, but COMMIT (i.e.
> > > > > the
> > > > > NFS
> > > > > equivalent of fsync()) can often take much longer even though
> > > > > it
> > > > > has no
> > > > > payload.
> > > > > 
> > > > > So the problem is not the size of the WRITE payload. The real
> > > > > problem
> > > > > is the timeout.
> > > > > 
> > > > > The bottom line is that if you want to keep timeo=7 as a mount
> > > > > option
> > > > > for TCP, then you are on your own.
> > > > > 
> > > > 
> > > > The problem isn't timeo at all.
> > > > If I understand it correctly, when I try to launch firefox over
> > > > nfsroot, 
> > > > NFS will wait until it fills 1M before "replying" to the
> > > > application.
> > > > Thus the applications will launch a lot slower, as they get
> > > > "disk 
> > > > feedback" in larger chunks and not "snappy".
> > > > 
> > > > In numbers:
> > > > timeo=600,rsize=1M => firefox opens in 30 secs
> > > > timeo=600,rsize=32k => firefox opens in 20 secs
> > > > 
> > > 
> > > That's a different problem, and is most likely due to readahead
> > > causing
> > > your client to read more data than it needs to. It is also true
> > > that
> > > the maximum readahead size is proportional to the rsize and that
> > > maybe
> > > it shouldn't be.
> > > However the VM layer is supposed to ensure that the kernel doesn't
> > > try
> > > to read ahead more than necessary. It is bounded by the maximum we
> > > set
> > > in the NFS layer, but it isn't supposed to hit that maximum unless
> > > the
> > > readahead heuristics show that the application may need it.
> > > 
> > > -- 
> > > Trond Myklebust
> > > Linux NFS client maintainer, Hammerspace
> > > trond.myklebust@hammerspace.com
> > 
> > What may be happening here is something I have noticed with glibc.
> > 
> > - statfs reports the rsize/wsize as the block size of the filesystem.
> > 
> > - glibc uses the block size as the default buffer size for
> > fread/fwrite.
> > 
> > If an application is using fread/fwrite on an NFS mounted file with
> > an rsize/wsize of 1M it will try to fill a 1MB buffer.
> > 
> > I have often changed mounts to use rsize/wsize=64K to alleviate this.
> > 
> 
> That sounds like an abuse of the filesystem block size. There is
> nothing in the POSIX definition of either fread() or fwrite() that
> requires glibc to do this: 
> https://pubs.opengroup.org/onlinepubs/9699919799/functions/fread.html
> https://pubs.opengroup.org/onlinepubs/9699919799/functions/fwrite.html
> 
> 
> -- 
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> trond.myklebust@hammerspace.com

It looks like this was fixed in glibc 2.25:

https://sourceware.org/bugzilla/show_bug.cgi?id=4099

But this version is not on the CentOS 6/7 systems I use.

-- 
Dan Forrest
Space Science and Engineering Center University of Wisconsin, Madison
dforrest@wisc.edu