Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Justin Piszcz]

Now that I have 74GB raptors in both of my Linux boxes, I thought I would 
compare throughput between FTP and NFS over a gigabit network.

I am using the same kernel versions and same motherboard on both machines 
and even the same raptor hdd model.

Here are my results:

NFS, COPY 700MB FILE FROM 1 RAPTOR TO ANOTHER RAPTOR VIA GIGABIT ETHERNET:

$ cp file /remote/dst
0.02user 1.86system 0:38.07elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+196minor)pagefaults 0swaps

FTP, SAME

lftp> put file
733045488 bytes transferred in 10 seconds (67.38M/s)

What is wrong with NFS?

NFS options used: rw,bg,hard,intr,nfsvers=3
Is it doing some kind of weird caching?
I am using NFSv3 & XFS as the filesystem, any ideas?

I suppose I should try NFS with TCP, yes?

Thanks!

Justin.

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Phil Oester]

On Mon, Jan 16, 2006 at 08:07:02PM -0500, Justin Piszcz wrote:
> I suppose I should try NFS with TCP, yes?

Precisely.

Phil

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Justin Piszcz]

Also, some people mentioned tuning, I used 8192 as the w/r size it then 
took 15 seconds, with 65535 it took 28 seconds.

I wonder how much faster NFS over TCP would be, or if NFS in the kernel is 
the problem itself?

Will try later thanks.

FTP seems to be the winner for now:

<--- 226 8.927 seconds (measured here), 78.31 Mbytes per second
733045488 bytes transferred in 9 seconds (77.08M/s)


On Mon, 16 Jan 2006, Phil Oester wrote:

> On Mon, Jan 16, 2006 at 08:07:02PM -0500, Justin Piszcz wrote:
>> I suppose I should try NFS with TCP, yes?
>
> Precisely.
>
> Phil
>

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Tomasz Kłoczko]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 590 bytes --]

On Mon, 16 Jan 2006, Justin Piszcz wrote:

> Also, some people mentioned tuning, I used 8192 as the w/r size it then took 
> 15 seconds, with 65535 it took 28 seconds.
>
> I wonder how much faster NFS over TCP would be, or if NFS in the kernel is 
> the problem itself?

On Linux NFS over TCP is slower than over UDP ~10%.

kloczek
-- 
-----------------------------------------------------------
*Ludzie nie mają problemów, tylko sobie sami je stwarzają*
-----------------------------------------------------------
Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: kloczek@rudy.mif.pg.gda.pl*

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Alan Cox]

On Maw, 2006-01-17 at 18:48 +0100, Tomasz Kłoczko wrote:
> > I wonder how much faster NFS over TCP would be, or if NFS in the kernel is 
> > the problem itself?
> 
> On Linux NFS over TCP is slower than over UDP ~10%.

For the specific case you measured. Its never quite that simple because
behaviour over different networks and error patterns varies a lot and
TCP can be a big win on loaded networks or under error conditions,
especially packet loss, where fragmentation losses kill throughput on
UDP.

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 742 bytes --]

Alan, is it normal for FTP to be 2x as fast as NFS?
With 100mbps, I never seemed to have any issues, but with GIGABIT I 
definitely see all sorts of weird issues.


On Tue, 17 Jan 2006, Alan Cox wrote:

> On Maw, 2006-01-17 at 18:48 +0100, Tomasz Kłoczko wrote:
>>> I wonder how much faster NFS over TCP would be, or if NFS in the kernel is
>>> the problem itself?
>>
>> On Linux NFS over TCP is slower than over UDP ~10%.
>
> For the specific case you measured. Its never quite that simple because
> behaviour over different networks and error patterns varies a lot and
> TCP can be a big win on loaded networks or under error conditions,
> especially packet loss, where fragmentation losses kill throughput on
> UDP.
>
>

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Alan Cox]

On Maw, 2006-01-17 at 13:24 -0500, Justin Piszcz wrote:
> Alan, is it normal for FTP to be 2x as fast as NFS?
> With 100mbps, I never seemed to have any issues, but with GIGABIT I 
> definitely see all sorts of weird issues.

NFS performance is limited by the fact it is a file system so sees only
what the file system can tell it. It also takes a hit because it has
strict rules on committing data to disk before acknowledging it (so data
is not lost over a crash). That makes NFS a bigger user of CPU resources
and more disk dependant than FTP which simply throws the entire file
down the pipe when in binary mode, does no processing and makes no
guarantee about restarts or what hits disk

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Trond Myklebust]

On Tue, 2006-01-17 at 13:24 -0500, Justin Piszcz wrote:
> Alan, is it normal for FTP to be 2x as fast as NFS?
> With 100mbps, I never seemed to have any issues, but with GIGABIT I 
> definitely see all sorts of weird issues.

Reading or writing?

The readahead algorithm has been borken in 2.6.x for at least the past 6
months. It leads to NFS collapsing down to 4k reads on the wire instead
of doing 32k or 64k.
An effort was made to look at fixing this, but it turns out that nobody
really understands the current messy implementation, and so progress has
been slow.

Cheers,
  Trond

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Justin Piszcz]

Writing from SRC(A) -> DST(B).
I have not tested reading, but as I recall there were similar speed issues 
going the other way too, although I have not tested it recently.

Justin.

On Tue, 17 Jan 2006, Trond Myklebust wrote:

> On Tue, 2006-01-17 at 13:24 -0500, Justin Piszcz wrote:
>> Alan, is it normal for FTP to be 2x as fast as NFS?
>> With 100mbps, I never seemed to have any issues, but with GIGABIT I
>> definitely see all sorts of weird issues.
>
> Reading or writing?
>
> The readahead algorithm has been borken in 2.6.x for at least the past 6
> months. It leads to NFS collapsing down to 4k reads on the wire instead
> of doing 32k or 64k.
> An effort was made to look at fixing this, but it turns out that nobody
> really understands the current messy implementation, and so progress has
> been slow.
>
> Cheers,
>  Trond
>

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Trond Myklebust]

On Tue, 2006-01-17 at 13:38 -0500, Justin Piszcz wrote:
> Writing from SRC(A) -> DST(B).
> I have not tested reading, but as I recall there were similar speed issues 
> going the other way too, although I have not tested it recently.

How were you testing it? I'm not sure that ftp will actually sync your
file to disk (whereas that is pretty much mandatory for an NFS server),
so unless you are transferring very large files, you would expect to see
a speed difference due to caching of writes by the server.

Cheers,
  Trond

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Justin Piszcz]

Did you get my other e-mail?

$ cp file /nfs/destination
$ lftp> put file


On Tue, 17 Jan 2006, Trond Myklebust wrote:

> On Tue, 2006-01-17 at 13:38 -0500, Justin Piszcz wrote:
>> Writing from SRC(A) -> DST(B).
>> I have not tested reading, but as I recall there were similar speed issues
>> going the other way too, although I have not tested it recently.
>
> How were you testing it? I'm not sure that ftp will actually sync your
> file to disk (whereas that is pretty much mandatory for an NFS server),
> so unless you are transferring very large files, you would expect to see
> a speed difference due to caching of writes by the server.
>
> Cheers,
>  Trond
>

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Trond Myklebust]

On Tue, 2006-01-17 at 13:55 -0500, Justin Piszcz wrote:
> Did you get my other e-mail?
> 
> $ cp file /nfs/destination
> $ lftp> put file


Yes, but how big a file is this? Is it significantly larger than the
amount of cache memory on the server? As I said, if ftp is failing to
sync the file to disk, then you may be comparing apples and oranges.

Cheers,
  Trond

> On Tue, 17 Jan 2006, Trond Myklebust wrote:
> 
> > On Tue, 2006-01-17 at 13:38 -0500, Justin Piszcz wrote:
> >> Writing from SRC(A) -> DST(B).
> >> I have not tested reading, but as I recall there were similar speed issues
> >> going the other way too, although I have not tested it recently.
> >
> > How were you testing it? I'm not sure that ftp will actually sync your
> > file to disk (whereas that is pretty much mandatory for an NFS server),
> > so unless you are transferring very large files, you would expect to see
> > a speed difference due to caching of writes by the server.
> >
> > Cheers,
> >  Trond
> >

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Justin Piszcz]

The file is 700MB.

Machine A (src) has 2GB of RAM / 2GB of swap
Machine B (dst) has 1GB of RAM / 2GB of swap.

Justin.

On Tue, 17 Jan 2006, Trond Myklebust wrote:

> On Tue, 2006-01-17 at 13:55 -0500, Justin Piszcz wrote:
>> Did you get my other e-mail?
>>
>> $ cp file /nfs/destination
>> $ lftp> put file
>
>
> Yes, but how big a file is this? Is it significantly larger than the
> amount of cache memory on the server? As I said, if ftp is failing to
> sync the file to disk, then you may be comparing apples and oranges.
>
> Cheers,
>  Trond
>
>> On Tue, 17 Jan 2006, Trond Myklebust wrote:
>>
>>> On Tue, 2006-01-17 at 13:38 -0500, Justin Piszcz wrote:
>>>> Writing from SRC(A) -> DST(B).
>>>> I have not tested reading, but as I recall there were similar speed issues
>>>> going the other way too, although I have not tested it recently.
>>>
>>> How were you testing it? I'm not sure that ftp will actually sync your
>>> file to disk (whereas that is pretty much mandatory for an NFS server),
>>> so unless you are transferring very large files, you would expect to see
>>> a speed difference due to caching of writes by the server.
>>>
>>> Cheers,
>>>  Trond
>>>
>

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Jan Engelhardt]

>> Did you get my other e-mail?
>> 
>> $ cp file /nfs/destination
>> $ lftp> put file
>
>
>Yes, but how big a file is this? Is it significantly larger than the
>amount of cache memory on the server? As I said, if ftp is failing to
>sync the file to disk, then you may be comparing apples and oranges.


Ok, so what happens if you use NFS with the async option, does it go a 
little faster?



Jan Engelhardt
-- 

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Justin Piszcz]

               auto   Can be mounted with the -a option.

               defaults
                      Use default options: rw, suid, dev, exec,  auto,
                      nouser, and async.

The default is async, no?

On Tue, 17 Jan 2006, Jan Engelhardt wrote:

>>> Did you get my other e-mail?
>>>
>>> $ cp file /nfs/destination
>>> $ lftp> put file
>>
>>
>> Yes, but how big a file is this? Is it significantly larger than the
>> amount of cache memory on the server? As I said, if ftp is failing to
>> sync the file to disk, then you may be comparing apples and oranges.
>
>
> Ok, so what happens if you use NFS with the async option, does it go a
> little faster?
>
>
>
> Jan Engelhardt
> -- 
>

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Jan Engelhardt]

> auto   Can be mounted with the -a option.
>
> defaults
> Use default options: rw, suid, dev, exec,  auto,
> nouser, and async.
>
> The default is async, no?

The server side also needs to specify async in exports. You even get a 
warning if you do not specify sync or async, because the default had 
been changed once.



Jan Engelhardt
-- 

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Lee Revell]

On Tue, 2006-01-17 at 23:07 +0100, Jan Engelhardt wrote:
> > auto   Can be mounted with the -a option.
> >
> > defaults
> > Use default options: rw, suid, dev, exec,  auto,
> > nouser, and async.
> >
> > The default is async, no?
> 
> The server side also needs to specify async in exports. You even get a 
> warning if you do not specify sync or async, because the default had 
> been changed once.
> 

What is the date on the above man page?  Looks like the docs need to be
updated.

I believe the default was originally async, which violates the NFS spec
and is dangerous, and changed to sync at some point.

Lee

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Justin Piszcz]

man mount

On Tue, 17 Jan 2006, Lee Revell wrote:

> On Tue, 2006-01-17 at 23:07 +0100, Jan Engelhardt wrote:
>>> auto   Can be mounted with the -a option.
>>>
>>> defaults
>>> Use default options: rw, suid, dev, exec,  auto,
>>> nouser, and async.
>>>
>>> The default is async, no?
>>
>> The server side also needs to specify async in exports. You even get a
>> warning if you do not specify sync or async, because the default had
>> been changed once.
>>
>
> What is the date on the above man page?  Looks like the docs need to be
> updated.
>
> I believe the default was originally async, which violates the NFS spec
> and is dangerous, and changed to sync at some point.
>
> Lee
>

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Lee Revell]

On Tue, 2006-01-17 at 18:19 -0500, Justin Piszcz wrote:
> man mount
> 

async is the default for most filesystems but the NFS standard requires
writes to be synchronous.

Lee

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Trond Myklebust]

On Tue, 2006-01-17 at 18:39 -0500, Lee Revell wrote:
> On Tue, 2006-01-17 at 18:19 -0500, Justin Piszcz wrote:
> > man mount
> > 
> 
> async is the default for most filesystems but the NFS standard requires
> writes to be synchronous.

On the server side, note. Not the client side. Justin appears to be
looking at the client, whereas you are referring to an export option on
the server.

The client only guarantees that writes must have been committed to disk
on the server when either fsync() or close() have been called.

Cheers,
  Trond

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Justin Piszcz]

NFS is still twice as slow as FTP, but best with a r/w size of 8192.

DEFAULT, NO OPTIONS
# mount p34:/ /nfs/p34
$ /usr/bin/time cp 700mb.img  /p34/x/d
0.01user 1.64system 0:34.23elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+196minor)pagefaults 0swaps

TCP, NO CACHING
# mount p34:/ /nfs/p34 -o nfsvers=3,tcp,noac
$ /usr/bin/time cp 700mb.img  /p34/x/a
0.02user 5.25system 0:58.43elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+197minor)pagefaults 0swaps

UDP, NO CACHING
# mount p34:/ /nfs/p34 -o nfsvers=3,noac
$ /usr/bin/time cp 700mb.img  /p34/x/b
0.02user 5.54system 1:00.34elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+196minor)pagefaults 0swaps

UDP, NO CACHING (w/65535 r/w size)
# mount p34:/ /nfs/p34 -o nfsvers=3,noac,rsize=65535,wsize=65535
$ /usr/bin/time cp 700mb.img  /p34/x/c
0.01user 5.75system 0:59.89elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+196minor)pagefaults 0swaps

# mount p34:/ /nfs/p34 -o nfsvers=3,rsize=8192,wsize=8192
0.04user 1.78system 0:14.16elapsed 12%CPU (0avgtext+0avgdata 
0maxresident)k
0inputs+0outputs (0major+190minor)pagefaults 0swaps

UDP, NFSV3 + (w/8192 r/w size)
$ /usr/bin/time cp 700mb.img  /p34/x/g
0.04user 1.78system 0:14.16elapsed 12%CPU (0avgtext+0avgdata 
0maxresident)k
0inputs+0outputs (0major+190minor)pagefaults 0swaps

TCP, NFSV3 + (w/8192 r/w size)
0.03user 1.81system 0:14.98elapsed 12%CPU (0avgtext+0avgdata 
0maxresident)k
0inputs+0outputs (0major+190minor)pagefaults 0swaps

UDP, NFSV3 + (w/16384 r/w size)
# mount p34:/ /nfs/p34 -o nfsvers=3,rsize=16834,wsize=16384
$ /usr/bin/time cp 700mb.img  /p34/x/e
0.03user 1.75system 0:20.20elapsed 8%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+192minor)pagefaults 0swaps

UDP, NFSV3 + (w/32768 r/w size)
# mount p34:/ /nfs/p34 -o nfsvers=3,rsize=32768,wsize=32768
  /usr/bin/time cp 700mb.img  /p34/x/f
0.01user 1.59system 0:32.87elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+196minor)pagefaults 0swaps






On Mon, 16 Jan 2006, Phil Oester wrote:

> On Mon, Jan 16, 2006 at 08:07:02PM -0500, Justin Piszcz wrote:
>> I suppose I should try NFS with TCP, yes?
>
> Precisely.
>
> Phil
>

Re: Kernel 2.6.15.1 + NFS is 4 times slower than FTP!?

[Jan Engelhardt]

> NFS is still twice as slow as FTP, but best with a r/w size of 8192.

Screams for a kftpd ;)


Jan Engelhardt
-- 
| Alphagate Systems, http://alphagate.hopto.org/
| jengelh's site, http://jengelh.hopto.org/