Re: NFS/MOUNT/sunrpc problem?

Re: NFS/MOUNT/sunrpc problem?

[Marco Bertoncin - Sun Microsystems UK - Platform OS Development Engineer]

> Have you tried a more recent kernel? 2.4.18 was released more than one
> and a half years ago...

This was a 2.4.18 kernel. But I even tried a RH9.0 ... still the problem 
happens.
Marco

Re: NFS/MOUNT/sunrpc problem?

[Trond Myklebust]

     > This was a 2.4.18 kernel. But I even tried a RH9.0 ... still
     > the problem happens.  Marco

The latest RedHat 9.0 errata kernels are (I believe) 2.4.20
based. That's still more than 9 months old. Please could you check if
the fixes that went into 2.4.22 have any effect on the problem.

Cheers,
  Trond

Re: NFS/MOUNT/sunrpc problem?

[Trond Myklebust]

>>>>> " " == Marco Bertoncin <- Sun Microsystems UK - Platform OS Development Engineer <Marco.Bertoncin@Sun.COM>> writes:

     > This is a MOUNT req, not NFS, so we are using userland rpc?

Correct. Unless you are doing NFSroot (which was what I thought you
might be using)...
For ordinary NFSv2 and NFSv3 mounts, the 'mount' program talks
directly to the server, then passes the resulting filehandle down to
the kernel.

Cheers,
  Trond

Re: NFS/MOUNT/sunrpc problem?

[Marco Bertoncin - Sun Microsystems UK - Platform OS Development Engineer]

An update on this.
After numerous experiments, I now suspect some code in userland. I managed to 
find the traceback involved in the storm and every packet generated in the storm 
repeats the whole of the patern:

...
tg3_start_xmit
qdisc_restart
dev_queue_xmit
ip_finish_output2
ip_build_xmit
udp_sendmsg
inet_sendmsg
sock_sendmsg
sys_sendto
sys_socketcall
...

This is a MOUNT req, not NFS, so we are using userland rpc?

Now, just as a conformation, if I put a delay in sys_socketcall after each group 
of 40 packets sent, my storm recovers after 40 packets!

I was sidetracked by my ignorance of the fact that the MOUNT call does not go 
through the kernel module nfs/sunrpc, which made me go off instrumenting, 
debugging and tracing those modules that do not even get used :-((((((((!


Marco

Re: NFS/MOUNT/sunrpc problem?

[Trond Myklebust]

>>>>> " " == Marco Bertoncin <- Sun Microsystems UK - Platform OS Development Engineer <Marco.Bertoncin@Sun.COM>> writes:


     > switch.  How to reproduce
     > - During install a MOUNT (V2) request is sent to the install
     >   server
     > - the ACK is dropped
     > Symptom
     > - the blade, after 3 seconds, starts a storm of retransmit
     >   (MOUNT reqs) that
     > won't stop, unless an ACK (one of the several ACKS sent for
     > each retransmitted requests) has the chance to get
     > through. This is sometimes after a few hundreds packets,
     > sometimes after a lot more, causing an apparent hang of the
     > installation process, and what's even worse, bringing to a
     > grinding halt the server (bombarded by near 1Gbit/sec packets).

Have you tried a more recent kernel? 2.4.18 was released more than one
and a half years ago...

Cheers,
  Trond

Re: NFS/MOUNT/sunrpc problem?

[Jeff Garzik]

On Wed, Sep 10, 2003 at 04:41:31PM +0100, Alan Cox wrote:
> On Mer, 2003-09-10 at 15:37, Marco Bertoncin - Sun Microsystems UK -
> Platform OS Development Engineer wrote:
> > - PXE booting x86 'headless' blades (2.0 Ghz 2P Xeon) to install RedHat 8.0 
> > (kernel 2.4.18).
> 
> Update the kernel once installed, the 2.4.18- kernels are obsoleted by
> other security fixes
> 
> > - the blade, after 3 seconds, starts a storm of retransmit (MOUNT reqs) that 
> > won't stop, unless an ACK (one of the several ACKS sent for each retransmitted 
> > requests) has the chance to get through. This is sometimes after a few hundreds 
> > packets, sometimes after a lot more, causing an apparent hang of the 
> > installation process, and what's even worse, bringing to a grinding halt the  
> > server (bombarded by near 1Gbit/sec packets).
> 
> I've seen one other report of this (with a via chip),

FWIW, on my Via EPIA (pre-Nehemiah C3) Wal-Mart box, I see NFS (?) bugs
as well.  I can mount just fine and do an 'ls' of a remote dir... but
any attempt to actually transfer data causes the entire mount to hang in
disk wait.

I swapped out NICs several times (and verified the NICs in other boxes)
but can reproduce this behavior quite easily.

ssh, ftp, and other services work flawlessly... it's just NFS.

I even tried tcp instead of udp, to no avail.

	Jeff, inevitably lacking time to track things down

Re: NFS/MOUNT/sunrpc problem?

[Marco Bertoncin - Sun Microsystems UK - Platform OS Development Engineer]

I just realized I sent my reply only to Alan ... sorry!


------------- Begin Forwarded Message -------------

Date: Wed, 10 Sep 2003 17:02:24 +0100 (BST)
From: Marco Bertoncin - Sun Microsystems UK - Platform OS Development Engineer 
<mb144209@ms-ebrk02-02.uk>
Subject: Re: NFS/MOUNT/sunrpc problem?
To: alan@lxorguk.ukuu.org.uk



Alan,
thanks for replying so quickly!

> Update the kernel once installed, the 2.4.18- kernels are obsoleted by
> other security fixes

Yes, we are working with several versions (rh-8.0, rh-9.0, as2.1, el3.0a2), and 
would like to support all of them. If this was for internal use, we would just 
go to the most up-to-date version, but sometimes customers are reluctant to 
upgrade (don't ask, I am not a marketing person :-))))

> 
> I've seen one other report of this (with a via chip),
>

This is with Intel82801CA

 
> Are you using NFS root or just NFS mounts ?

Just NFS mounts. This is how we do PXE install: download vmlinuz and an initial 
initrd.img image. vmlinuz uncompresses, allocates a RAM disk and mounts 
initrd.img where it finds the modules necessary to carry on (tg3 driver amongst 
them), then it nfs mount a directory from the install server (something like 
/tftp/install/this_release/images/ where it would find configurastion files and 
images for the install) and it is this MOUNT request that, if the reply gets 
lost, starts the storm.
You said you already saw a similar report of this. Is this logged somewhere?

Again, Thanks very much.
Marco


------------- End Forwarded Message -------------

Re: NFS/MOUNT/sunrpc problem?

[Alan Cox]

On Mer, 2003-09-10 at 15:37, Marco Bertoncin - Sun Microsystems UK -
Platform OS Development Engineer wrote:
> - PXE booting x86 'headless' blades (2.0 Ghz 2P Xeon) to install RedHat 8.0 
> (kernel 2.4.18).

Update the kernel once installed, the 2.4.18- kernels are obsoleted by
other security fixes

> - the blade, after 3 seconds, starts a storm of retransmit (MOUNT reqs) that 
> won't stop, unless an ACK (one of the several ACKS sent for each retransmitted 
> requests) has the chance to get through. This is sometimes after a few hundreds 
> packets, sometimes after a lot more, causing an apparent hang of the 
> installation process, and what's even worse, bringing to a grinding halt the  
> server (bombarded by near 1Gbit/sec packets).

I've seen one other report of this (with a via chip),

> Ah, one last experiment I did was to try and reproduce the problem on an 
> installed blade (same version of the kernel). No chance. I noticed, though that 
> the MOUNT request sent by the 'installed linux' (it would be a proper i686-smp 
> build instead of a up i386) is V3, whilst that during installation is V2. 
> Thinking this might have been a hunch, I tried "mount -o nfsvers=2 
> server:/export /mnt": I saw the requests, the dropped ack (on the server side, 
> of course) but no storm ...!).

Are you using NFS root or just NFS mounts ?

NFS/MOUNT/sunrpc problem?

[Marco Bertoncin - Sun Microsystems UK - Platform OS Development Engineer]

Hello all,

I work for Sun Microsystems and we are currently doing some work to put Linux on 
Sun x86 Blades, but we have got a problem with PXE booting I am hoping someone 
could help us with:


Environment
- PXE booting x86 'headless' blades (2.0 Ghz 2P Xeon) to install RedHat 8.0 
(kernel 2.4.18).
- vmlinuz and modules in initrd.img were compiled for i386
- install server is a Sun box
- link is a BCM5704 (broadcom) 1Gbit chipset (tg3 driver) connected to a Gigabit 
switch.
How to reproduce
- During install a MOUNT (V2) request is sent to the install server
- the ACK is dropped
Symptom
- the blade, after 3 seconds, starts a storm of retransmit (MOUNT reqs) that 
won't stop, unless an ACK (one of the several ACKS sent for each retransmitted 
requests) has the chance to get through. This is sometimes after a few hundreds 
packets, sometimes after a lot more, causing an apparent hang of the 
installation process, and what's even worse, bringing to a grinding halt the  
server (bombarded by near 1Gbit/sec packets).

How I think I ruled out the h/w, the switch or any component other than the 
Linux network stack (I may be mistaken, in which case, please correct me):

I made a 'modified' version of the tg3 driver (driver for the BCM570x family of 
1Gbit controllers) that would:

- drop the ack (as a quick hack, it would actually just drop 1 every 2 packets, 
so I had 50% probability to reproduce the problem at every attempt)

- udelay 100th of a second every 100 packets sent. Also a hack, as my previous 
attempt at measuring the transmit frequency using a simple counter and the 
jiffies variable failed. This, by the way, might be indicative of the fact that 
the 'storm' is somehow generated in a tight loop where clock interrupts are 
disabled at least for a good part of the time?

Whilst with the unmodified version of the tg3 driver when the storm started it 
could last a very long time, with this version I could see the ACK go through 
after exactly 100 packets sent by the x86 blade, which would then recover and 
stop retransmitting the request ( I have a few snoop output files if any one 
wants to have a look).

I have looked in bugzilla and searched the linux kernel archive for a couple of 
days, but I did not notice anything related to this issue.

Ah, one last experiment I did was to try and reproduce the problem on an 
installed blade (same version of the kernel). No chance. I noticed, though that 
the MOUNT request sent by the 'installed linux' (it would be a proper i686-smp 
build instead of a up i386) is V3, whilst that during installation is V2. 
Thinking this might have been a hunch, I tried "mount -o nfsvers=2 
server:/export /mnt": I saw the requests, the dropped ack (on the server side, 
of course) but no storm ...!).

So, my question is: has anyome experienced a similar problem? Is anyone aware of 
any issue in the nfs/sunrpc retry mechanism that could lead to issue the retries 
at near wire speed instead of after the normal nfs timeout?

Thanks everyone in advance for you time.

Marco