NFS Client and SMP

NFS Client and SMP

[Oliver Paukstadt]

HY HY

I have massive problems using client nfs on SMP boxes.
I can reproduce it 2.4.[0-7] on s390 and s390x and with 2.4.[0-8] on IA32.

Try to reproduce starting massive IO on an nfs mounted volume, eg. tar it
to /dev/null.
I tested it against verious servers, eg Slowlaris, HP-UX, DEC, Linux 2.2,
Linux 2.4, no tar survived.
using NFS v2 or v3 caused no differences.
One Intel we have to identical machines with identical setup and only the
box locks up running nfs client (we switched the roles of the boxes)

On S390 it took 5 to 30 minutes to lock the system, on Intel sometimes it
took up to 3 hours.

Running the system with only one cpu caused no hangs, all tars finished.

BYtE Oli

+++LINUX++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++Manchmal stehe ich sogar nachts auf und installiere mir eins....+++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 
 

Re: NFS Client and SMP

[Trond Myklebust]

     > HY HY I have massive problems using client nfs on SMP boxes.  I
     > can reproduce it 2.4.[0-7] on s390 and s390x and with 2.4.[0-8]
     > on IA32.

What do you mean by 'hang' in this context? Does the entire machine
die, or is it just the nfs mount?

Also, can you reproduce it with the patch

  http://www.fys.uio.no/~trondmy/src/2.4.9/linux-2.4.9-rpc_smpfixes.dif

(the same patch applies fine to 2.4.[678] if you for some reason don't
like 2.4.9)

Cheers,
  Trond

Re: NFS Client and SMP

[Trond Myklebust]

     > HY HY I have massive problems using client nfs on SMP boxes.  I
     > can reproduce it 2.4.[0-7] on s390 and s390x and with 2.4.[0-8]
     > on IA32.

One other thing. If you're running on a Gigabit network, try turning
off jumbo frames - there seems to be some problems still with getting
that to work properly, and it's been known to cause NFS hangs.

Cheers,
  Trond

Re: NFS Client and SMP

[Tom Sightler]

> One other thing. If you're running on a Gigabit network, try turning
> off jumbo frames - there seems to be some problems still with getting
> that to work properly, and it's been known to cause NFS hangs.
>

Is this problem specific to the 2.4 series or is the 2.2 NFS implementation
suspect to this as well.  I am curious because we have a good number of SMP
servers running SuSE's 2.2.19 kernel connected to a NetApp filer via GigE
with jumbo frames and have not had any problems during development.
However, we are about 1 month from going production and load will increase
tremendously then (even though we've attempted to stress the system in
development, real production always introduces new loads).

Anyway, If this is a known problem with 2.2 as well I may reconsider leaving
jumbo frames enabled.

Thanks,
Tom

Re: NFS Client and SMP

[Trond Myklebust]

>>>>> " " == Tom Sightler <ttsig@tuxyturvy.com> writes:

     > Is this problem specific to the 2.4 series or is the 2.2 NFS
     > implementation suspect to this as well.  I am curious because
     > we have a good number of SMP servers running SuSE's 2.2.19
     > kernel connected to a NetApp filer via GigE with jumbo frames
     > and have not had any problems during development.  However, we
     > are about 1 month from going production and load will increase
     > tremendously then (even though we've attempted to stress the
     > system in development, real production always introduces new
     > loads).

I don't have acces to a test setup for Gigabit, so I have to rely on
other people's reports. I can only remember hearing of problems in
2.4.x, however you might want to check the archives on the NFS
discussion list on Sourceforge.

Cheers,
  Trond

RE: NFS Client and SMP

[Elgar, Jeremy]

Ill force a freeze tonight and check the logs just to check, but as you say
im inclined to suspect some problem with the server.

Cheers.
Jeremy


> -----Original Message-----
> From: Alan Cox [mailto:alan@lxorguk.ukuu.org.uk]
> Sent: 28 August 2001 17:48
> To: JElgar@ndsuk.com
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: NFS Client and SMP
> 
> 
> > Copying a large (n>20) number of file from local disk to an 
> nfs share (on
> > the BSD box)
> > causes the server to totally freeze (have to reboot) 
> normally have to bring
> > the local machines nic up and down to get anything back. 
> kill's on the cp's
> > wont do anything
> 
> Whichever end froze is the buggy one. NFS clients are 
> supposed to be robust
> so if Linux was doing something bad the openbsd box should 
> have errored it
> and vice versa. Both may indeed be buggy but the freeze is th efirst
> target.
> 
> Alan
> 


 
===============================================================
Information contained in this email message is intended only for
use of the individual or entity named above. If the reader of this
message is not the intended recipient, or the employee or agent
responsible to deliver it to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
communication is strictly prohibited. If you have received this
communication in error, please immediately notify us by email
to postmaster@ndsuk.com and destroy the original message. 

Re: NFS Client and SMP

[Alan Cox]

> Copying a large (n>20) number of file from local disk to an nfs share (on
> the BSD box)
> causes the server to totally freeze (have to reboot) normally have to bring
> the local machines nic up and down to get anything back. kill's on the cp's
> wont do anything

Whichever end froze is the buggy one. NFS clients are supposed to be robust
so if Linux was doing something bad the openbsd box should have errored it
and vice versa. Both may indeed be buggy but the freeze is th efirst
target.

Alan

RE: NFS Client and SMP

[Elgar, Jeremy]

I am having a similar problem that I had been putting down to a problem with
an OpenBSD box,

Copying a large (n>20) number of file from local disk to an nfs share (on
the BSD box)
causes the server to totally freeze (have to reboot) normally have to bring
the local machines nic up and down to get anything back. kill's on the cp's
wont do anything

happened with each 2.4.x from 4 onwards, on an SMP Dual PIII 933 2Gb Ram

Oddly enough its only on a 'cp -r dir' or 'cp dir/*' it seems to hang, (so
much so that I have been doing a for each script hack, upto now.

All other machines (a mixture of Debian and open BSD boxes are okay)

I can supply / test if it turns out to be a linux problem.

Jeremy

  

> -----Original Message-----
> From: Oliver Paukstadt [mailto:pstadt@stud.fh-heilbronn.de]
> Sent: 28 August 2001 17:17
> To: Linux-Kernel
> Subject: NFS Client and SMP
> 
> 
> HY HY
> 
> I have massive problems using client nfs on SMP boxes.
> I can reproduce it 2.4.[0-7] on s390 and s390x and with 
> 2.4.[0-8] on IA32.
> 
> Try to reproduce starting massive IO on an nfs mounted 
> volume, eg. tar it
> to /dev/null.
> I tested it against verious servers, eg Slowlaris, HP-UX, 
> DEC, Linux 2.2,
> Linux 2.4, no tar survived.
> using NFS v2 or v3 caused no differences.
> One Intel we have to identical machines with identical setup 
> and only the
> box locks up running nfs client (we switched the roles of the boxes)
> 
> On S390 it took 5 to 30 minutes to lock the system, on Intel 
> sometimes it
> took up to 3 hours.
> 
> Running the system with only one cpu caused no hangs, all 
> tars finished.
> 
> BYtE Oli
> 
> +++LINUX++++++++++++++++++++++++++++++++++++++++++++++++++++++
> ++++++++++++
> +++Manchmal stehe ich sogar nachts auf und installiere mir 
> eins....+++++++
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> ++++++++++++ 
>  
> 
> -
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


 
===============================================================
Information contained in this email message is intended only for
use of the individual or entity named above. If the reader of this
message is not the intended recipient, or the employee or agent
responsible to deliver it to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
communication is strictly prohibited. If you have received this
communication in error, please immediately notify us by email
to postmaster@ndsuk.com and destroy the original message.