Linux-NFS Archive on lore.kernel.org
 help / color / Atom feed
From: Trond Myklebust <trondmy@hammerspace.com>
To: "alkisg@gmail.com" <alkisg@gmail.com>,
	"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: rsize,wsize=1M causes severe lags in 10/100 Mbps
Date: Thu, 19 Sep 2019 19:51:46 +0000
Message-ID: <e51876b8c2540521c8141ba11b11556d22bde20b.camel@hammerspace.com> (raw)
In-Reply-To: <7afc5770-abfa-99bb-dae9-7d11680875fd@gmail.com>

On Thu, 2019-09-19 at 22:21 +0300, Alkis Georgopoulos wrote:
> On 9/19/19 7:11 PM, Trond Myklebust wrote:
> > No. It is not a problem, because nfs-utils defaults to using TCP
> > mounts. Fragmentation is only a problem with UDP, and we stopped
> > defaulting to that almost 2 decades ago.
> > 
> > However it may well be that klibc is still defaulting to using UDP,
> > in
> > which case it should be fixed. There are major Linux distros out
> > there
> > today that don't even compile in support for NFS over UDP any more.
> 
> I haven't tested with UDP at all; the problem was with TCP.
> I saw the problem in klibc nfsmount with TCP + NFS 3,
> and in `mount -t nfs -o timeo=7 server:/share /mnt` with TCP + NFS
> 4.2.
> 
> Steps to reproduce:
> 1) Connect server <=> client at 10 or 100 Mbps.
> Gigabit is also "less snappy" but it's less obvious there.
> For reliable results, I made sure that server/client/network didn't
> have 
> any other load at all.
> 
> 2) Server:
> echo '/srv *(ro,async,no_subtree_check)' >> /etc/exports
> exportfs -ra
> truncate -s 10G /srv/10G.file
> The sparse file ensures that disk IO bandwidth isn't an issue.
> 
> 3) Client:
> mount -t nfs -o timeo=7 192.168.1.112:/srv /mnt
> dd if=/mnt/10G.file of=/dev/null status=progress
> 
> 4) Result:
> dd there starts with 11.2 MB/sec, which is fine/expected,
> and it slowly drops to 2 MB/sec after a while,
> it lags, omitting some seconds in its output line,
> e.g. 507510784 bytes (508 MB, 484 MiB) copied, 186 s, 2,7 MB/s^C,
> at which point "Ctrl+C" needs 30+ seconds to stop dd,
> because of IO waiting etc.
> 
> In another terminal tab, `dmesg -w` is full of these:
> [  316.404250] nfs: server 192.168.1.112 not responding, still trying
> [  316.759512] nfs: server 192.168.1.112 OK
> 
> 5) Remarks:
> With timeo=600, there are no errors in dmesg.
> The fact that timeo=7 (the nfsmount default) causes errors, proves
> that 
> some packets need more than 0.7 secs to arrive.
> Which in turn explains why all the applications open extremely
> slowly 
> and feel sluggish on netroot = 100 Mbps, NFS, TCP.
> 
> Lowering rsize,wsize from 1M to 32K solves all those issues without
> any 
> negative side effects that I can see. Even on gigabit, 32K makes 
> applications a lot more snappy so it's better even there.
> On 10 Mbps, rsize=1M is completely unusable.
> 
> So I'm not sure where rsize=1M is a better default. Is it only for
> 10G+ 
> connections?
> 

I don't understand why klibc would default to supplying a timeo=7
argument at all. It would be MUCH better if it just let the kernel set
the default, which in the case of TCP is timeo=600.

I agree with your argument that replaying requests every 0.7 seconds is
just going to cause congestion. TCP provides for reliable delivery of
RPC messages to the server, which is why the kernel default is a full
minute.

So please ask the klibc developers to change libmount to let the kernel
decide the default mount options. Their current setting is just plain
wrong.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com



  reply index

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-19  7:29 rsize,wsize=1M causes severe lags in 10/100 Mbps, what sets those defaults? Alkis Georgopoulos
2019-09-19 15:08 ` Trond Myklebust
2019-09-19 15:58   ` rsize,wsize=1M causes severe lags in 10/100 Mbps Alkis Georgopoulos
2019-09-19 16:11     ` Trond Myklebust
2019-09-19 19:21       ` Alkis Georgopoulos
2019-09-19 19:51         ` Trond Myklebust [this message]
2019-09-19 19:57           ` Alkis Georgopoulos
2019-09-19 20:05             ` Trond Myklebust
2019-09-19 20:20               ` Alkis Georgopoulos
2019-09-19 20:40                 ` Trond Myklebust
2019-09-19 21:19                   ` Daniel Forrest
2019-09-19 21:42                     ` Trond Myklebust
2019-09-19 22:16                       ` Daniel Forrest
2019-09-20  9:25                         ` Alkis Georgopoulos
2019-09-20  9:48                           ` Alkis Georgopoulos
2019-09-20 10:04                             ` Alkis Georgopoulos
2019-09-21  7:52                               ` Alkis Georgopoulos
2019-09-21  7:59                                 ` Alkis Georgopoulos
2019-09-21 11:02                                   ` Alkis Georgopoulos

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e51876b8c2540521c8141ba11b11556d22bde20b.camel@hammerspace.com \
    --to=trondmy@hammerspace.com \
    --cc=alkisg@gmail.com \
    --cc=linux-nfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-nfs/0 linux-nfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nfs linux-nfs/ https://lore.kernel.org/linux-nfs \
		linux-nfs@vger.kernel.org
	public-inbox-index linux-nfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-nfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git