Linux-NFS Archive on lore.kernel.org
 help / color / Atom feed
From: Trond Myklebust <trondmy@gmail.com>
To: Daniel Forrest <dforrest@wisc.edu>
Cc: "alkisg@gmail.com" <alkisg@gmail.com>,
	"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: rsize,wsize=1M causes severe lags in 10/100 Mbps
Date: Thu, 19 Sep 2019 17:42:26 -0400
Message-ID: <503c22ad34b3f3a15015b7384bcad469b2899cb4.camel@hammerspace.com> (raw)
In-Reply-To: <20190919211912.GA21865@cosmos.ssec.wisc.edu>

On Thu, 2019-09-19 at 16:19 -0500, Daniel Forrest wrote:
> On Thu, Sep 19, 2019 at 08:40:41PM +0000, Trond Myklebust wrote:
> > On Thu, 2019-09-19 at 23:20 +0300, Alkis Georgopoulos wrote:
> > > On 9/19/19 11:05 PM, Trond Myklebust wrote:
> > > > There are plenty of operations that can take longer than 700 ms
> > > > to
> > > > complete. Synchronous writes to disk are one, but COMMIT (i.e.
> > > > the
> > > > NFS
> > > > equivalent of fsync()) can often take much longer even though
> > > > it
> > > > has no
> > > > payload.
> > > > 
> > > > So the problem is not the size of the WRITE payload. The real
> > > > problem
> > > > is the timeout.
> > > > 
> > > > The bottom line is that if you want to keep timeo=7 as a mount
> > > > option
> > > > for TCP, then you are on your own.
> > > > 
> > > 
> > > The problem isn't timeo at all.
> > > If I understand it correctly, when I try to launch firefox over
> > > nfsroot, 
> > > NFS will wait until it fills 1M before "replying" to the
> > > application.
> > > Thus the applications will launch a lot slower, as they get
> > > "disk 
> > > feedback" in larger chunks and not "snappy".
> > > 
> > > In numbers:
> > > timeo=600,rsize=1M => firefox opens in 30 secs
> > > timeo=600,rsize=32k => firefox opens in 20 secs
> > > 
> > 
> > That's a different problem, and is most likely due to readahead
> > causing
> > your client to read more data than it needs to. It is also true
> > that
> > the maximum readahead size is proportional to the rsize and that
> > maybe
> > it shouldn't be.
> > However the VM layer is supposed to ensure that the kernel doesn't
> > try
> > to read ahead more than necessary. It is bounded by the maximum we
> > set
> > in the NFS layer, but it isn't supposed to hit that maximum unless
> > the
> > readahead heuristics show that the application may need it.
> > 
> > -- 
> > Trond Myklebust
> > Linux NFS client maintainer, Hammerspace
> > trond.myklebust@hammerspace.com
> 
> What may be happening here is something I have noticed with glibc.
> 
> - statfs reports the rsize/wsize as the block size of the filesystem.
> 
> - glibc uses the block size as the default buffer size for
> fread/fwrite.
> 
> If an application is using fread/fwrite on an NFS mounted file with
> an
> rsize/wsize of 1M it will try to fill a 1MB buffer.
> 
> I have often changed mounts to use rsize/wsize=64K to alleviate this.
> 

That sounds like an abuse of the filesystem block size. There is
nothing in the POSIX definition of either fread() or fwrite() that
requires glibc to do this: 
https://pubs.opengroup.org/onlinepubs/9699919799/functions/fread.html
https://pubs.opengroup.org/onlinepubs/9699919799/functions/fwrite.html


-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com




  reply index

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-19  7:29 rsize,wsize=1M causes severe lags in 10/100 Mbps, what sets those defaults? Alkis Georgopoulos
2019-09-19 15:08 ` Trond Myklebust
2019-09-19 15:58   ` rsize,wsize=1M causes severe lags in 10/100 Mbps Alkis Georgopoulos
2019-09-19 16:11     ` Trond Myklebust
2019-09-19 19:21       ` Alkis Georgopoulos
2019-09-19 19:51         ` Trond Myklebust
2019-09-19 19:57           ` Alkis Georgopoulos
2019-09-19 20:05             ` Trond Myklebust
2019-09-19 20:20               ` Alkis Georgopoulos
2019-09-19 20:40                 ` Trond Myklebust
2019-09-19 21:19                   ` Daniel Forrest
2019-09-19 21:42                     ` Trond Myklebust [this message]
2019-09-19 22:16                       ` Daniel Forrest
2019-09-20  9:25                         ` Alkis Georgopoulos
2019-09-20  9:48                           ` Alkis Georgopoulos
2019-09-20 10:04                             ` Alkis Georgopoulos
2019-09-21  7:52                               ` Alkis Georgopoulos
2019-09-21  7:59                                 ` Alkis Georgopoulos
2019-09-21 11:02                                   ` Alkis Georgopoulos

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=503c22ad34b3f3a15015b7384bcad469b2899cb4.camel@hammerspace.com \
    --to=trondmy@gmail.com \
    --cc=alkisg@gmail.com \
    --cc=dforrest@wisc.edu \
    --cc=linux-nfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-nfs/0 linux-nfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nfs linux-nfs/ https://lore.kernel.org/linux-nfs \
		linux-nfs@vger.kernel.org
	public-inbox-index linux-nfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-nfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git