Linux-NFS Archive on
 help / color / Atom feed
From: Alkis Georgopoulos <>
To: Trond Myklebust <>,
	"" <>
Subject: Re: rsize,wsize=1M causes severe lags in 10/100 Mbps
Date: Fri, 20 Sep 2019 12:25:16 +0300
Message-ID: <> (raw)
In-Reply-To: <>

On 9/20/19 1:16 AM, Daniel Forrest wrote:
>>> What may be happening here is something I have noticed with glibc.
>>> - statfs reports the rsize/wsize as the block size of the filesystem.
>>> - glibc uses the block size as the default buffer size for
>>> fread/fwrite.
>>> If an application is using fread/fwrite on an NFS mounted file with
>>> an rsize/wsize of 1M it will try to fill a 1MB buffer.
>>> I have often changed mounts to use rsize/wsize=64K to alleviate this.
>> That sounds like an abuse of the filesystem block size. There is
>> nothing in the POSIX definition of either fread() or fwrite() that
>> requires glibc to do this:
> It looks like this was fixed in glibc 2.25:

This is likely not the exact issue I'm experiencing, as I'm testing e.g.
with glibc 2.27-3ubuntu1 on Ubuntu 18.04 and kernel 5.0.

New benchmark, measuring the boot time of a netbooted client,
from right after the kernel is loaded to the display manager screen:

1) On 10 Mbps:
a) tcp,timeo=600,rsize=32K: 304 secs
b) tcp,timeo=600,rsize=1M: 618 secs

2) On 100 Mbps:
a) tcp,timeo=600,rsize=32K: 40 secs
b) tcp,timeo=600,rsize=1M: 84 secs

3) On 1000 Mbps:
a) tcp,timeo=600,rsize=32K: 20 secs
b) tcp,timeo=600,rsize=1M: 24 secs

32K is always faster, even on full gigabit.
Disk access on gigabit was *significantly* faster to result in 4 seconds 
lower boot time. In the 10/100 cases, rsize=1M is pretty much unusable.
There are no writes involved, they go in a local tmpfs/overlayfs.
Would it make sense for me to measure the *boot bandwidth* in each case, 
to see if more things (readahead) are downloaded with rsize=1M?

I can do whatever benchmarks and test whatever parameters you tell me 
to, but I do not know the NFS/kernel internals to be able to explain why 
this happens.

The reason I investigated this is because I developed the new version of (GPL netbooting software), where we switched from 
squashfs-over-NBD to squashfs-over-NFS, and netbooting was extremely 
slow until I lowered rsize to 32K, so I thought I'd share my findings in 
case it makes a better default for everyone (or reveals a problem 
With rsize=32K, squashfs-over-NFS is as speedy as squashfs-over-NBD, but 
a lot more stable.

Of course the same rsize findings apply for NFS /home too (without 
nfsmount), or for just transferring large or small files, not just for /.

says the kernel nfsroot defaults are timeo=7,rsize=4096,wsize=4096. This 
is about the internal kernel netbooting support, not using klibc 
nfsmount; but I haven't tested it as it would involve compiling a kernel 
with my NIC driver.

Thank you,
Alkis Georgopoulos
LTSP developer

  reply index

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-19  7:29 rsize,wsize=1M causes severe lags in 10/100 Mbps, what sets those defaults? Alkis Georgopoulos
2019-09-19 15:08 ` Trond Myklebust
2019-09-19 15:58   ` rsize,wsize=1M causes severe lags in 10/100 Mbps Alkis Georgopoulos
2019-09-19 16:11     ` Trond Myklebust
2019-09-19 19:21       ` Alkis Georgopoulos
2019-09-19 19:51         ` Trond Myklebust
2019-09-19 19:57           ` Alkis Georgopoulos
2019-09-19 20:05             ` Trond Myklebust
2019-09-19 20:20               ` Alkis Georgopoulos
2019-09-19 20:40                 ` Trond Myklebust
2019-09-19 21:19                   ` Daniel Forrest
2019-09-19 21:42                     ` Trond Myklebust
2019-09-19 22:16                       ` Daniel Forrest
2019-09-20  9:25                         ` Alkis Georgopoulos [this message]
2019-09-20  9:48                           ` Alkis Georgopoulos
2019-09-20 10:04                             ` Alkis Georgopoulos
2019-09-21  7:52                               ` Alkis Georgopoulos
2019-09-21  7:59                                 ` Alkis Georgopoulos
2019-09-21 11:02                                   ` Alkis Georgopoulos

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NFS Archive on

Archives are clonable:
	git clone --mirror linux-nfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nfs linux-nfs/ \
	public-inbox-index linux-nfs

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone