Stale file not being refreshed automatically?

Stale file not being refreshed automatically?

[Nathan March]

Hi All,

Having a strange issue, I've reproduced this both on nfs 3 and 4. 
Currently using v4 with:

exports:
/var/home/voyage 
10.2.1.0/24(rw,fsid=0,async,wdelay,secure,no_subtree_check,anonuid=1000,anongid=1006)

fstab:
10.2.1.1:/ /mnt/voyage nfs4 
rsize=32768,wsize=32768,noatime,nosuid,rw,bg,hard,intr 0 0

Can be produced using these two commands on 2 machines:

perl -le '$| = 1 ; while (1) { sleep 1; $res = -e "/home/voyage/test"; 
if ($res) { print "-e : $res"; } else { print "-e : $res ($!)"; } }'
-e : 1

perl -le '$| = 1 ; while (1) { sleep 5; open(F, ">", 
"/home/voyage/test.foo"); close F; my $res = 
rename("/home/voyage/test.foo", "/home/voyage/test"); print "rename 
($res)"; }'

The first command will occasionally print out: -e :  (Stale NFS file handle)

Now according to http://nfs.sourceforge.net/#faq_a10 as of 2.6.12 the 
vfs should automatically retry when it gets a estale during path 
resolution. This doesn't appear to be happening though based on the 
errors I'm seeing.

Anyone have any thoughts on what might be happening here? Or am I 
misinterpreting that faq entry? (This is on ext3 if it matters)

- Nathan

Re: Stale file not being refreshed automatically?

[Nathan March]

Just to add, this is 2.6.32.27 w/ grsecurity.

- Nathan

On 1/25/2011 2:11 PM, Nathan March wrote:
> Hi All,
>
> Having a strange issue, I've reproduced this both on nfs 3 and 4. 
> Currently using v4 with:
>
> exports:
> /var/home/voyage 
> 10.2.1.0/24(rw,fsid=0,async,wdelay,secure,no_subtree_check,anonuid=1000,anongid=1006)
>
> fstab:
> 10.2.1.1:/ /mnt/voyage nfs4 
> rsize=32768,wsize=32768,noatime,nosuid,rw,bg,hard,intr 0 0
>
> Can be produced using these two commands on 2 machines:
>
> perl -le '$| = 1 ; while (1) { sleep 1; $res = -e "/home/voyage/test"; 
> if ($res) { print "-e : $res"; } else { print "-e : $res ($!)"; } }'
> -e : 1
>
> perl -le '$| = 1 ; while (1) { sleep 5; open(F, ">", 
> "/home/voyage/test.foo"); close F; my $res = 
> rename("/home/voyage/test.foo", "/home/voyage/test"); print "rename 
> ($res)"; }'
>
> The first command will occasionally print out: -e :  (Stale NFS file 
> handle)
>
> Now according to http://nfs.sourceforge.net/#faq_a10 as of 2.6.12 the 
> vfs should automatically retry when it gets a estale during path 
> resolution. This doesn't appear to be happening though based on the 
> errors I'm seeing.
>
> Anyone have any thoughts on what might be happening here? Or am I 
> misinterpreting that faq entry? (This is on ext3 if it matters)
>
> - Nathan
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Stale file not being refreshed automatically?

[Trond Myklebust]

On Tue, 2011-01-25 at 14:11 -0800, Nathan March wrote: 
> Hi All,
> 
> Having a strange issue, I've reproduced this both on nfs 3 and 4. 
> Currently using v4 with:
> 
> exports:
> /var/home/voyage 
> 10.2.1.0/24(rw,fsid=0,async,wdelay,secure,no_subtree_check,anonuid=1000,anongid=1006)
> 
> fstab:
> 10.2.1.1:/ /mnt/voyage nfs4 
> rsize=32768,wsize=32768,noatime,nosuid,rw,bg,hard,intr 0 0
> 
> Can be produced using these two commands on 2 machines:
> 
> perl -le '$| = 1 ; while (1) { sleep 1; $res = -e "/home/voyage/test"; 
> if ($res) { print "-e : $res"; } else { print "-e : $res ($!)"; } }'
> -e : 1
> 
> perl -le '$| = 1 ; while (1) { sleep 5; open(F, ">", 
> "/home/voyage/test.foo"); close F; my $res = 
> rename("/home/voyage/test.foo", "/home/voyage/test"); print "rename 
> ($res)"; }'
> 
> The first command will occasionally print out: -e :  (Stale NFS file handle)
> 
> Now according to http://nfs.sourceforge.net/#faq_a10 as of 2.6.12 the 
> vfs should automatically retry when it gets a estale during path 
> resolution. This doesn't appear to be happening though based on the 
> errors I'm seeing.
> 
> Anyone have any thoughts on what might be happening here? Or am I 
> misinterpreting that faq entry? (This is on ext3 if it matters)

Does '-e' in perl only do a lookup, or does it result in a 'stat()'
call? If the latter, then your test is flawed: a stat() is a lookup+a
getattr, and the latter can definitely return ESTALE when you are
constantly replacing the file.

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: Stale file not being refreshed automatically?

[Nathan March]

Ah ha, yes, it's using a stat call.

What would be the proper C call to check the file exists without 
incurring the stat()?

Thanks!

- Nathan

On 1/25/2011 2:28 PM, Trond Myklebust wrote:
> On Tue, 2011-01-25 at 14:11 -0800, Nathan March wrote:
>> Hi All,
>>
>> Having a strange issue, I've reproduced this both on nfs 3 and 4.
>> Currently using v4 with:
>>
>> exports:
>> /var/home/voyage
>> 10.2.1.0/24(rw,fsid=0,async,wdelay,secure,no_subtree_check,anonuid=1000,anongid=1006)
>>
>> fstab:
>> 10.2.1.1:/ /mnt/voyage nfs4
>> rsize=32768,wsize=32768,noatime,nosuid,rw,bg,hard,intr 0 0
>>
>> Can be produced using these two commands on 2 machines:
>>
>> perl -le '$| = 1 ; while (1) { sleep 1; $res = -e "/home/voyage/test";
>> if ($res) { print "-e : $res"; } else { print "-e : $res ($!)"; } }'
>> -e : 1
>>
>> perl -le '$| = 1 ; while (1) { sleep 5; open(F, ">",
>> "/home/voyage/test.foo"); close F; my $res =
>> rename("/home/voyage/test.foo", "/home/voyage/test"); print "rename
>> ($res)"; }'
>>
>> The first command will occasionally print out: -e :  (Stale NFS file handle)
>>
>> Now according to http://nfs.sourceforge.net/#faq_a10 as of 2.6.12 the
>> vfs should automatically retry when it gets a estale during path
>> resolution. This doesn't appear to be happening though based on the
>> errors I'm seeing.
>>
>> Anyone have any thoughts on what might be happening here? Or am I
>> misinterpreting that faq entry? (This is on ext3 if it matters)
> Does '-e' in perl only do a lookup, or does it result in a 'stat()'
> call? If the latter, then your test is flawed: a stat() is a lookup+a
> getattr, and the latter can definitely return ESTALE when you are
> constantly replacing the file.
>
> Cheers
>    Trond

Re: Stale file not being refreshed automatically?

[Trond Myklebust]

On Tue, 2011-01-25 at 14:38 -0800, Nathan March wrote: 
> Ah ha, yes, it's using a stat call.
> 
> What would be the proper C call to check the file exists without 
> incurring the stat()?

access("file", F_OK) should do it.

Cheers
  Trond

> On 1/25/2011 2:28 PM, Trond Myklebust wrote:
> > On Tue, 2011-01-25 at 14:11 -0800, Nathan March wrote:
> >> Hi All,
> >>
> >> Having a strange issue, I've reproduced this both on nfs 3 and 4.
> >> Currently using v4 with:
> >>
> >> exports:
> >> /var/home/voyage
> >> 10.2.1.0/24(rw,fsid=0,async,wdelay,secure,no_subtree_check,anonuid=1000,anongid=1006)
> >>
> >> fstab:
> >> 10.2.1.1:/ /mnt/voyage nfs4
> >> rsize=32768,wsize=32768,noatime,nosuid,rw,bg,hard,intr 0 0
> >>
> >> Can be produced using these two commands on 2 machines:
> >>
> >> perl -le '$| = 1 ; while (1) { sleep 1; $res = -e "/home/voyage/test";
> >> if ($res) { print "-e : $res"; } else { print "-e : $res ($!)"; } }'
> >> -e : 1
> >>
> >> perl -le '$| = 1 ; while (1) { sleep 5; open(F, ">",
> >> "/home/voyage/test.foo"); close F; my $res =
> >> rename("/home/voyage/test.foo", "/home/voyage/test"); print "rename
> >> ($res)"; }'
> >>
> >> The first command will occasionally print out: -e :  (Stale NFS file handle)
> >>
> >> Now according to http://nfs.sourceforge.net/#faq_a10 as of 2.6.12 the
> >> vfs should automatically retry when it gets a estale during path
> >> resolution. This doesn't appear to be happening though based on the
> >> errors I'm seeing.
> >>
> >> Anyone have any thoughts on what might be happening here? Or am I
> >> misinterpreting that faq entry? (This is on ext3 if it matters)
> > Does '-e' in perl only do a lookup, or does it result in a 'stat()'
> > call? If the latter, then your test is flawed: a stat() is a lookup+a
> > getattr, and the latter can definitely return ESTALE when you are
> > constantly replacing the file.
> >
> > Cheers
> >    Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: Stale file not being refreshed automatically?

[Nathan March]

Awesome, I'll see about a perl equiv. Thanks again!

- Nathan

On 1/25/2011 2:48 PM, Trond Myklebust wrote:
> On Tue, 2011-01-25 at 14:38 -0800, Nathan March wrote:
>> Ah ha, yes, it's using a stat call.
>>
>> What would be the proper C call to check the file exists without
>> incurring the stat()?
> access("file", F_OK) should do it.
>
> Cheers
>    Trond
>
>> On 1/25/2011 2:28 PM, Trond Myklebust wrote:
>>> On Tue, 2011-01-25 at 14:11 -0800, Nathan March wrote:
>>>> Hi All,
>>>>
>>>> Having a strange issue, I've reproduced this both on nfs 3 and 4.
>>>> Currently using v4 with:
>>>>
>>>> exports:
>>>> /var/home/voyage
>>>> 10.2.1.0/24(rw,fsid=0,async,wdelay,secure,no_subtree_check,anonuid=1000,anongid=1006)
>>>>
>>>> fstab:
>>>> 10.2.1.1:/ /mnt/voyage nfs4
>>>> rsize=32768,wsize=32768,noatime,nosuid,rw,bg,hard,intr 0 0
>>>>
>>>> Can be produced using these two commands on 2 machines:
>>>>
>>>> perl -le '$| = 1 ; while (1) { sleep 1; $res = -e "/home/voyage/test";
>>>> if ($res) { print "-e : $res"; } else { print "-e : $res ($!)"; } }'
>>>> -e : 1
>>>>
>>>> perl -le '$| = 1 ; while (1) { sleep 5; open(F, ">",
>>>> "/home/voyage/test.foo"); close F; my $res =
>>>> rename("/home/voyage/test.foo", "/home/voyage/test"); print "rename
>>>> ($res)"; }'
>>>>
>>>> The first command will occasionally print out: -e :  (Stale NFS file handle)
>>>>
>>>> Now according to http://nfs.sourceforge.net/#faq_a10 as of 2.6.12 the
>>>> vfs should automatically retry when it gets a estale during path
>>>> resolution. This doesn't appear to be happening though based on the
>>>> errors I'm seeing.
>>>>
>>>> Anyone have any thoughts on what might be happening here? Or am I
>>>> misinterpreting that faq entry? (This is on ext3 if it matters)
>>> Does '-e' in perl only do a lookup, or does it result in a 'stat()'
>>> call? If the latter, then your test is flawed: a stat() is a lookup+a
>>> getattr, and the latter can definitely return ESTALE when you are
>>> constantly replacing the file.
>>>
>>> Cheers
>>>     Trond

Re: Stale file not being refreshed automatically?

[Nathan March]

On 1/25/2011 2:28 PM, Trond Myklebust wrote:
> On Tue, 2011-01-25 at 14:
> Does '-e' in perl only do a lookup, or does it result in a 'stat()'
> call? If the latter, then your test is flawed: a stat() is a lookup+a
> getattr, and the latter can definitely return ESTALE when you are
> constantly replacing the file.
>
> Cheers
>    Trond

Hi Trond,

Has any behavior here changed between 2.6.19 and 2.6.32 that you know of?

Testing that perl one liner out with a variety of nfs setups we run:

2.6.11 client vs 2.6.32 server - no error
2.6.11 client vs netapp - no error
2.6.19 client vs 2.6.19 server - no error
2.6.19 client vs netapp - no error
2.6.32 client vs 2.6.32 server - stale file handles
2.6.32 client vs netapp - stale file handles

So it seems to be a client specific change sometime between 2.6.19 and 
2.6.32....

- Nathan

Re: Stale file not being refreshed automatically?

[Nathan March]

On 1/25/2011 2:28 PM, Trond Myklebust wrote:
> On Tue, 2011-01-25 at 14:
> Does '-e' in perl only do a lookup, or does it result in a 'stat()'
> call? If the latter, then your test is flawed: a stat() is a lookup+a
> getattr, and the latter can definitely return ESTALE when you are
> constantly replacing the file.
>
> Cheers
>    Trond

Hi Trond,

Has any behavior here changed between 2.6.19 and 2.6.32 that you know of?

Testing that perl one liner out with a variety of nfs setups we run:

2.6.11 client vs 2.6.32 server - no error
2.6.11 client vs netapp - no error
2.6.19 client vs 2.6.19 server - no error
2.6.19 client vs netapp - no error
2.6.32 client vs 2.6.32 server - stale file handles
2.6.32 client vs netapp - stale file handles

So it seems to be a client specific change sometime between 2.6.19 and 
2.6.32....

- Nathan

Re: Stale file not being refreshed automatically?

[Trond Myklebust]

On Tue, 2011-01-25 at 15:29 -0800, Nathan March wrote: 
> On 1/25/2011 2:28 PM, Trond Myklebust wrote:
> > On Tue, 2011-01-25 at 14:
> > Does '-e' in perl only do a lookup, or does it result in a 'stat()'
> > call? If the latter, then your test is flawed: a stat() is a lookup+a
> > getattr, and the latter can definitely return ESTALE when you are
> > constantly replacing the file.
> >
> > Cheers
> >    Trond
> 
> Hi Trond,
> 
> Has any behavior here changed between 2.6.19 and 2.6.32 that you know of?
> 
> Testing that perl one liner out with a variety of nfs setups we run:
> 
> 2.6.11 client vs 2.6.32 server - no error
> 2.6.11 client vs netapp - no error
> 2.6.19 client vs 2.6.19 server - no error
> 2.6.19 client vs netapp - no error
> 2.6.32 client vs 2.6.32 server - stale file handles
> 2.6.32 client vs netapp - stale file handles
> 
> So it seems to be a client specific change sometime between 2.6.19 and 
> 2.6.32....

You are probably seeing an effect due to attribute caching. Newer
kernels have tighter restrictions on when they are allowed to cache.

IOW: I suspect the above is just a timing artefact that would probably
disappear on a slower machine.

Cheers
  Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com