Linux-NFS Archive on lore.kernel.org
 help / color / Atom feed
From: Alkis Georgopoulos <alkisg@gmail.com>
To: "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: rsize,wsize=1M causes severe lags in 10/100 Mbps
Date: Sat, 21 Sep 2019 10:59:55 +0300
Message-ID: <675724ca-bdb4-324b-b3ec-2fba608b8358@gmail.com> (raw)
In-Reply-To: <ddf47f3c-d27e-972b-fd38-1fdfeaa105a3@gmail.com>

On 9/21/19 10:52 AM, Alkis Georgopoulos wrote:
> I think it's caused by the kernel readahead, not glibc readahead.
> TL;DR: This solves the problem:
> echo 4 > /sys/devices/virtual/bdi/0:58/read_ahead_kb
> 
> Question: how to configure NFS/kernel to automatically set that?
> 
> Long version:
> Doing step (4) below results in tremendous speedup:
> 
> 1) mount -t nfs -o tcp,timeo=600,rsize=1048576,wsize=1048576 
> 10.161.254.11:/srv/ltsp /mnt
> 
> 2) cat /proc/fs/nfsfs/volumes
> We see the DEV number from there, e.g. 0:58
> 
> 3) cat /sys/devices/virtual/bdi/0:58/read_ahead_kb
> 15360
> I assume that this means the kernel will try to read ahead up to 15 MB 
> for each accessed file. *THIS IS THE PROBLEM*. For non-NFS devices, this 
> value is 128 (KB).
> 
> 4) echo 4 > /sys/devices/virtual/bdi/0:58/read_ahead_kb
> 
> 5) Test. Traffic now should be a *lot* less, and speed a *lot* more.
> E.g. my NFS booting tests:
>   - read_ahead_kb=15360 (the default) => 1160 MB traffic to boot
>   - read_ahead_kb=128 => 324MB traffic
>   - read_ahead_kb=4 => 223MB traffic
> 
> So the question that remains, is how to properly configure either NFS or 
> the kernel, to use small readahead values for NFS.
> 
> I'm currently doing it with this workaround:
> for f in $(awk '/^v[0-9]/ { print $4 }' < /proc/fs/nfsfs/volumes); do 
> echo 4 > /sys/devices/virtual/bdi/$f/read_ahead_kb; done
> 
> Thanks,
> Alkis



Quoting https://lkml.org/lkml/2010/2/26/48
 > nfs: use 2*rsize readahead size
 > With default rsize=512k and NFS_MAX_READAHEAD=15, the current NFS
 > readahead size 512k*15=7680k is too large than necessary for typical
 > clients.

I.e. the problem probably is that when NFS_MAX_READAHEAD=15 was 
implemented, rsize was 512k; now that rsize=1M, this results in 
readaheads of 15M, which cause all the traffic and lags.

  reply index

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-19  7:29 rsize,wsize=1M causes severe lags in 10/100 Mbps, what sets those defaults? Alkis Georgopoulos
2019-09-19 15:08 ` Trond Myklebust
2019-09-19 15:58   ` rsize,wsize=1M causes severe lags in 10/100 Mbps Alkis Georgopoulos
2019-09-19 16:11     ` Trond Myklebust
2019-09-19 19:21       ` Alkis Georgopoulos
2019-09-19 19:51         ` Trond Myklebust
2019-09-19 19:57           ` Alkis Georgopoulos
2019-09-19 20:05             ` Trond Myklebust
2019-09-19 20:20               ` Alkis Georgopoulos
2019-09-19 20:40                 ` Trond Myklebust
2019-09-19 21:19                   ` Daniel Forrest
2019-09-19 21:42                     ` Trond Myklebust
2019-09-19 22:16                       ` Daniel Forrest
2019-09-20  9:25                         ` Alkis Georgopoulos
2019-09-20  9:48                           ` Alkis Georgopoulos
2019-09-20 10:04                             ` Alkis Georgopoulos
2019-09-21  7:52                               ` Alkis Georgopoulos
2019-09-21  7:59                                 ` Alkis Georgopoulos [this message]
2019-09-21 11:02                                   ` Alkis Georgopoulos

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=675724ca-bdb4-324b-b3ec-2fba608b8358@gmail.com \
    --to=alkisg@gmail.com \
    --cc=linux-nfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-nfs/0 linux-nfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nfs linux-nfs/ https://lore.kernel.org/linux-nfs \
		linux-nfs@vger.kernel.org
	public-inbox-index linux-nfs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-nfs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git