NFS v3 cached directory content out of sync

NFS v3 cached directory content out of sync

[Stefan Egli]

Hi there,

I'm experiencing a problem with NFS v3. After a rush of data written
to a particular NFS partition (to/from NetApp) I have noticed
directory listings between three different NFS v3 clients (doing a'ls
in the same directory) to see different content. The 'rush of data'
included approx 400GB (and took a few hours to do so). The
inconsistency that the three clients experienced was still the case 4
hours after the mentioned data rush.

I would 'somehow understand' that such a 'data storm' could maybe
overwhelm the NFS client's caches - and thus would 'accept' (although
not liking it) a delay between the client's cache updates. But that
such an inconsistency is still existing 4 hours afterwards is IMHO a
plain bug.

=A0Question 1: Would you agree that this is a bug or is it 'NFS as desi=
gned' ?

What I did having seen this is creating a new file in that directory -
and voila, all clients immediately got to see the right directory
content. This again calmed my nerves as in 'NFS still works'.

Having said that - I now did some more testing with this NFS cache
delay and noticed that file content updates between the three NFS
clients easily takes a few seconds - up to 10-15 seconds.

=A0Question 2: Is such a file content delay in NFS 'as designed' - I'm
assuming that fiddling with NFS mount parameters could put a 'defined
maximum' to such a delay? Or is there no such maximum and under 'bad
luck' situations it can go infinitely high (which would be Q1) ?

=A0Question 3: What is a suggested best-practice NFS mount parameter se=
t
for complying with the following requirements:
=A0=A0 * lots of reads - tons of files - reads often from different fil=
es
=A0=A0 * few writes - but if written it should propagate to all NFS
clients 'immediately'
=A0=A0 * high load situations (as with the 400GB read/write stuff above=
) -
and after or even during this doing a 'ls' in a directory should
produce consistent results on different attached NFS clients

=A0Question 4: If we'd somehow manually detected such a directory
content inconsistency - would there be something like a 'hey NFS
client, flush all NFS caches NOW' thing?

=A0Question 5: any of this related to commit 37d9d76d8b3a2ac5817e1fa326=
3cfe
0fdb439e51: NFS: flush cached directory information slightly more readi=
ly. ?

Thanks for answers to any or all of above! :)

MUCH appreciated!

Regards,
Stefan

Re: NFS v3 cached directory content out of sync

[Trond Myklebust]

On Fri, 2009-08-21 at 20:22 +0200, Stefan Egli wrote:
>  Question 1: Would you agree that this is a bug or is it 'NFS as designed' ?

It sounds like a bug. You don't mention which client you are using.

>  Question 2: Is such a file content delay in NFS 'as designed' - I'm
> assuming that fiddling with NFS mount parameters could put a 'defined
> maximum' to such a delay? Or is there no such maximum and under 'bad
> luck' situations it can go infinitely high (which would be Q1) ?

You can tune the amount of time the client caches information by using
the 'ac*' mount options. In this case, you will want to adjust the
values of 'acdirmin' and 'acdirmax', probably setting them to zero.

'man 5 nfs' should provide you with more information.

>  Question 3: What is a suggested best-practice NFS mount parameter set
> for complying with the following requirements:
>    * lots of reads - tons of files - reads often from different files
>    * few writes - but if written it should propagate to all NFS
> clients 'immediately'
>    * high load situations (as with the 400GB read/write stuff above) -
> and after or even during this doing a 'ls' in a directory should
> produce consistent results on different attached NFS clients

See above

>  Question 4: If we'd somehow manually detected such a directory
> content inconsistency - would there be something like a 'hey NFS
> client, flush all NFS caches NOW' thing?

No.

>  Question 5: any of this related to commit 37d9d76d8b3a2ac5817e1fa3263cfe
> 0fdb439e51: NFS: flush cached directory information slightly more readily. ?

You client should be seeing mtime changes when you are creating new
files, so it shouldn't need to look at the ctime.

The only time when ctime changes are relevant are if you use 'rsync' to
copy the files without specifying --omit-dir-times.
IOW: if something is explicitly setting the mtime on the directory.

Cheers
  Trond

Re: NFS v3 cached directory content out of sync

[Stefan Egli]

Hi Trond,

> It sounds like a bug. You don't mention which client you are using.

from uname:
Linux (Ubuntu) 2.6.24-24-generic #1 SMP Tue Jun 30 19:54:36 UTC 2009
x86_64 GNU/Linux

from mount:
type nfs (ro,tcp,rsize=3D8192,wsize=3D8192,nfsvers=3D3,addr=3D192.168.X=
X.YY)

> You can tune the amount of time the client caches information by usin=
g
> the 'ac*' mount options. In this case, you will want to adjust the
> values of 'acdirmin' and 'acdirmax', probably setting them to zero.
>
> 'man 5 nfs' should provide you with more information.

Ok, so setting acdirmin=3D0 and acdirmax=3D0 would mean no directory
content caching, right?

>> =A0Question 4: If we'd somehow manually detected such a directory
>> content inconsistency - would there be something like a 'hey NFS
>> client, flush all NFS caches NOW' thing?
>
> No.

unmount / mount would - but that's obviously not feasible. bugger
there's nothing for that...

>> =A0Question 5: any of this related to commit 37d9d76d8b3a2ac5817e1fa=
3263cfe
>> 0fdb439e51: NFS: flush cached directory information slightly more re=
adily. ?
>
> You client should be seeing mtime changes when you are creating new
> files, so it shouldn't need to look at the ctime.
>
> The only time when ctime changes are relevant are if you use 'rsync' =
to
> copy the files without specifying --omit-dir-times.
> IOW: if something is explicitly setting the mtime on the directory.

Would have to check if that applies in our case. We're doing
backup/restore from tivoli (tsm) - maybe that guy is to be blamed
for not correctly updating mtime/ctimes ?

Cheers,
Stefan

Re: NFS v3 cached directory content out of sync

[Stefan Egli]

> from uname:
> Linux (Ubuntu) 2.6.24-24-generic #1 SMP Tue Jun 30 19:54:36 UTC 2009
> x86_64 GNU/Linux
>
> from mount:
> type nfs (ro,tcp,rsize=8192,wsize=8192,nfsvers=3,addr=192.168.XX.YY)

sorry, that was wrong - it should read:
(rw,tcp,rsize=8192,wsize=8192,nfsvers=3,addr=192.168.XX.YY)

Cheers,
Stefan

Re: NFS v3 cached directory content out of sync

[Simon Kirby]

On Fri, Aug 21, 2009 at 09:08:01PM +0200, Stefan Egli wrote:

> >> ?Question 4: If we'd somehow manually detected such a directory
> >> content inconsistency - would there be something like a 'hey NFS
> >> client, flush all NFS caches NOW' thing?
> >
> > No.
> 
> unmount / mount would - but that's obviously not feasible. bugger
> there's nothing for that...

Wouldn't (admittedly sledgehammer) echo 3 > /proc/sys/vm/drop_caches
accomplish this?

Simon-

Re: NFS v3 cached directory content out of sync

[Trond Myklebust]

On Fri, 2009-08-21 at 12:15 -0700, Simon Kirby wrote:
> On Fri, Aug 21, 2009 at 09:08:01PM +0200, Stefan Egli wrote:
> 
> > >> ?Question 4: If we'd somehow manually detected such a directory
> > >> content inconsistency - would there be something like a 'hey NFS
> > >> client, flush all NFS caches NOW' thing?
> > >
> > > No.
> > 
> > unmount / mount would - but that's obviously not feasible. bugger
> > there's nothing for that...
> 
> Wouldn't (admittedly sledgehammer) echo 3 > /proc/sys/vm/drop_caches
> accomplish this?

Only if you are certain that there are no processes actually using that
directory or any of its subdirs/files.

Cheers
  Trond

Re: NFS v3 cached directory content out of sync

[Trond Myklebust]

On Fri, 2009-08-21 at 21:08 +0200, Stefan Egli wrote:
> Hi Trond,
> 
> > It sounds like a bug. You don't mention which client you are using.
> 
> from uname:
> Linux (Ubuntu) 2.6.24-24-generic #1 SMP Tue Jun 30 19:54:36 UTC 2009
> x86_64 GNU/Linux
> 
> from mount:
> type nfs (ro,tcp,rsize=8192,wsize=8192,nfsvers=3,addr=192.168.XX.YY)
> 
> > You can tune the amount of time the client caches information by using
> > the 'ac*' mount options. In this case, you will want to adjust the
> > values of 'acdirmin' and 'acdirmax', probably setting them to zero.
> >
> > 'man 5 nfs' should provide you with more information.
> 
> Ok, so setting acdirmin=0 and acdirmax=0 would mean no directory
> content caching, right?

No directory _attribute_ caching, which again should mean that we check
if the parent directory mtime has changed on each lookup.

> >>  Question 5: any of this related to commit 37d9d76d8b3a2ac5817e1fa3263cfe
> >> 0fdb439e51: NFS: flush cached directory information slightly more readily. ?
> >
> > You client should be seeing mtime changes when you are creating new
> > files, so it shouldn't need to look at the ctime.
> >
> > The only time when ctime changes are relevant are if you use 'rsync' to
> > copy the files without specifying --omit-dir-times.
> > IOW: if something is explicitly setting the mtime on the directory.
> 
> Would have to check if that applies in our case. We're doing
> backup/restore from tivoli (tsm) - maybe that guy is to be blamed
> for not correctly updating mtime/ctimes ?

The restore will probably mess with the mtime, in which case you should
probably apply the above commit (applications cannot change the ctime).

Cheers
  Trond

Re: NFS v3 cached directory content out of sync

[Stefan Egli]

> No directory _attribute_ caching, which again should mean that we check
> if the parent directory mtime has changed on each lookup.

Ok, sorry for not doing a complete RTFM but is there an attribute then for
disabling directory caching?

> The restore will probably mess with the mtime, in which case you should
> probably apply the above commit (applications cannot change the ctime).

The odd thing is, that we're seeing sync delays of up to 4-5 hours when
doing a restore onto NFS, but also 1-2 hours when doing a backup from
NFS (which is then only reads) - the latter is rather odd as in making cache
syncs that slow, isn't it? Looks like an overload situation maybe?

Cheers,
Stefan

Re: NFS v3 cached directory content out of sync

[Trond Myklebust]

On Mon, 2009-08-24 at 12:32 +0200, Stefan Egli wrote:
> Hi,
> 
> It's me again. I'm catching up on 'man 5 nfs' and reading all the input I found
> wrt the NFS directory cache issue so far. So this is hopefully a condensed
> set of findings and questions ;)
> 
> 1) As I understand, the Linux version we're using (2.6.24-24-generic) does not
>    contain the patch 37d9d76d8b3a2ac5817e1fa3263cfe0fdb439e51 - which
>    only made it to the kernel 2.6.30, right?

That would be a question for the Debian/Ubuntu maintainers. I have no
idea what they have included in 2.6.24-24-generic

> 2) I'm still not sure I understand that patch correctly though
>    (37d9d76d8b3a2ac5817e1fa3263cfe0fdb439e5): Would the tivoli restore
>    be able to mess with mtimes and *therefore* cause the 4-5 hour cache
>    inconsistency I'm seeing? If yes, would the patch then magically fix this?

If tivoli is messing with the mtime, then it _might_ cause the
condition, in a restore situation by changing the directory contents but
not the mtime (which is what tells the NFS client and applications that
the directory contents have changed).
By looking at the ctime too, the client can always tell that something
has changed, and thus clear its cache.

> 3) The other part of my issue which I still don't understand is, why a backup
>    (not restore) could cause completely unrelated files (which we update every
>    5 minutes using some simple raw shell scripts) to remain out-of-sync
>    with other NFS clients - that is, client A changes the content every 5 min
>    but client B and C see it only after 1-2 hours. Seems somewhat new and
>    unrelated to the restore case of above - as restore does change directory
>    content (and that's where the problem lies) but backup merely changes
>    the file's mtime probably (but not even sure it does that - maybe leaves it
>    unchanged)

I've no idea why the other clients aren't seeing a change in this case.
I certainly cannot reproduce this with my own setup.

With the default acdirmin/max settings, the clients should not be
caching the mtime for longer than 1 minute. While they might be blind to
the directory changes during that time, they should at least see it
after the minute expires.

> 4) If I see this correctly, the acdirmin/acdirmax parameters could not prevent
>   a situation 2) of above?

No.

> 5) Would rdirplus be of any help here at all?

No.

> 6) Regarding the echo 3 > /proc/sys/vm/drop_caches issue:
> 
>   > Only if you are certain that there are no processes actually using that
>   > directory or any of its subdirs/files.
> 
>   What happens if there are processes still having files open in subdirs?
>   Would probably create .nfs files - but would it otherwise work - or corrupt
>   read/written data?

The VFS clears the cache by dropping unreferenced dentries. By holding a
reference to the directory or a subfile/subdir, your process would
simply cause the VFS to be unable to drop that dentry.

Cheers
  Trond

Re: NFS v3 cached directory content out of sync

[Stefan Egli]

Thanks Trond for the input! I've got just one more puzzle (s.below)

On Mon, Aug 24, 2009 at 2:11 PM, Trond Myklebust wrote:
>
> On Mon, 2009-08-24 at 12:32 +0200, Stefan Egli wrote:
>
> > 2) I'm still not sure I understand that patch correctly though
> > =A0 =A0(37d9d76d8b3a2ac5817e1fa3263cfe0fdb439e5): Would the tivoli =
restore
> > =A0 =A0be able to mess with mtimes and *therefore* cause the 4-5 ho=
ur cache
> > =A0 =A0inconsistency I'm seeing? If yes, would the patch then magic=
ally fix this?
>
> If tivoli is messing with the mtime, then it _might_ cause the
> condition, in a restore situation by changing the directory contents =
but
> not the mtime (which is what tells the NFS client and applications th=
at
> the directory contents have changed).
> By looking at the ctime too, the client can always tell that somethin=
g
> has changed, and thus clear its cache.

I suspect, the problem I'm seeing could be similar to this one:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/269172

What puzzles me with that 269172 bug is that it requires that you
first do a 'ls' on machine A - then do that script mentioned in the
bug on machine B -
and then do another 'ls' on machine A. This looks like with the first '=
ls'
machine A fills its attribute-cache - which then later on becomes out-o=
f-sync
with what the NFS server says.

In my situation though, I'm not doing any ls in the subdirectories affe=
cted -
which would mean it's not 100% explainable with the above bug... bugger=
=2E..

or is it ?

>
> > =A0 =A0(not restore) could cause completely unrelated files (which =
we update every
> > =A0 =A05 minutes using some simple raw shell scripts) to remain out=
-of-sync
> > =A0 =A0with other NFS clients - that is, client A changes the conte=
nt every 5 min
> > =A0 =A0but client B and C see it only after 1-2 hours. Seems somewh=
at new and
> > =A0 =A0unrelated to the restore case of above - as restore does cha=
nge directory
> > =A0 =A0content (and that's where the problem lies) but backup merel=
y changes
> > =A0 =A0the file's mtime probably (but not even sure it does that - =
maybe leaves it
> > =A0 =A0unchanged)
>
> > 3) The other part of my issue which I still don't understand is, wh=
y a backup
> I've no idea why the other clients aren't seeing a change in this cas=
e.
> I certainly cannot reproduce this with my own setup.
>
> With the default acdirmin/max settings, the clients should not be
> caching the mtime for longer than 1 minute. While they might be blind=
 to
> the directory changes during that time, they should at least see it
> after the minute expires.

=46orget this issue - I found the problem - not related to NFS


Cheers,
Stefan

Re: NFS v3 cached directory content out of sync

[Trond Myklebust]

On Tue, 2009-08-25 at 18:01 +0200, Stefan Egli wrote:
> Thanks Trond for the input! I've got just one more puzzle (s.below)
> 
> On Mon, Aug 24, 2009 at 2:11 PM, Trond Myklebust wrote:
> >
> > On Mon, 2009-08-24 at 12:32 +0200, Stefan Egli wrote:
> >
> > > 2) I'm still not sure I understand that patch correctly though
> > >    (37d9d76d8b3a2ac5817e1fa3263cfe0fdb439e5): Would the tivoli restore
> > >    be able to mess with mtimes and *therefore* cause the 4-5 hour cache
> > >    inconsistency I'm seeing? If yes, would the patch then magically fix this?
> >
> > If tivoli is messing with the mtime, then it _might_ cause the
> > condition, in a restore situation by changing the directory contents but
> > not the mtime (which is what tells the NFS client and applications that
> > the directory contents have changed).
> > By looking at the ctime too, the client can always tell that something
> > has changed, and thus clear its cache.
> 
> I suspect, the problem I'm seeing could be similar to this one:
> 
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/269172
> 
> What puzzles me with that 269172 bug is that it requires that you
> first do a 'ls' on machine A - then do that script mentioned in the
> bug on machine B -
> and then do another 'ls' on machine A. This looks like with the first 'ls'
> machine A fills its attribute-cache - which then later on becomes out-of-sync
> with what the NFS server says.
> 
> In my situation though, I'm not doing any ls in the subdirectories affected -
> which would mean it's not 100% explainable with the above bug... bugger...
> 
> or is it ?

Yes. The NFS client will cache mtime whether or not you do an 'ls'. I
suspect the only reason for the 'ls' in that launchpad bug report, is to
force an mtime update (and to fill the readdir cache).

Cheers
  Trond

Re: NFS v3 cached directory content out of sync

[Stefan Egli]

On Tue, Aug 25, 2009 at 9:00 PM, Trond Myklebust wrote:
> Yes. The NFS client will cache mtime whether or not you do an 'ls'. I
> suspect the only reason for the 'ls' in that launchpad bug report, is to
> force an mtime update (and to fill the readdir cache).

Ok, interesting. I guess though that 'something' needs to read at least
a file in that particular directory for the directory content to be
cached, right? (i.e. either an 'ls' or some create/delete of a file?)

Cheers,
Stefan

Re: NFS v3 cached directory content out of sync

[Trond Myklebust]

On Tue, 2009-08-25 at 21:06 +0200, Stefan Egli wrote:
> On Tue, Aug 25, 2009 at 9:00 PM, Trond Myklebust wrote:
> > Yes. The NFS client will cache mtime whether or not you do an 'ls'. I
> > suspect the only reason for the 'ls' in that launchpad bug report, is to
> > force an mtime update (and to fill the readdir cache).
> 
> Ok, interesting. I guess though that 'something' needs to read at least
> a file in that particular directory for the directory content to be
> cached, right? (i.e. either an 'ls' or some create/delete of a file?)

Every time you use a filename, the act of looking up that name is
cached. The parent directory's mtime is also cached. If it changes, then
the cached lookup is invalidated. If not, then the cached lookup is
assumed still valid (since the directory contents are not supposed to
have changed).

Re: NFS v3 cached directory content out of sync

[Stefan Egli]

On Tue, Aug 25, 2009 at 9:12 PM, Trond Myklebust wrote:
> On Tue, 2009-08-25 at 21:06 +0200, Stefan Egli wrote:
>> On Tue, Aug 25, 2009 at 9:00 PM, Trond Myklebust wrote:
>> > Yes. The NFS client will cache mtime whether or not you do an 'ls'. I
>> > suspect the only reason for the 'ls' in that launchpad bug report, is to
>> > force an mtime update (and to fill the readdir cache).
>>
>> Ok, interesting. I guess though that 'something' needs to read at least
>> a file in that particular directory for the directory content to be
>> cached, right? (i.e. either an 'ls' or some create/delete of a file?)
>
> Every time you use a filename, the act of looking up that name is
> cached. The parent directory's mtime is also cached. If it changes, then
> the cached lookup is invalidated. If not, then the cached lookup is
> assumed still valid (since the directory contents are not supposed to
> have changed).

Great thanks, got that!

Is there some docu about this level of NFS detail you know of?
Esp the attribute and the data cache?

Cheers,
Stefan

Re: NFS v3 cached directory content out of sync

[Trond Myklebust]

On Tue, 2009-08-25 at 21:19 +0200, Stefan Egli wrote:
> On Tue, Aug 25, 2009 at 9:12 PM, Trond Myklebust wrote:
> > On Tue, 2009-08-25 at 21:06 +0200, Stefan Egli wrote:
> >> On Tue, Aug 25, 2009 at 9:00 PM, Trond Myklebust wrote:
> >> > Yes. The NFS client will cache mtime whether or not you do an 'ls'. I
> >> > suspect the only reason for the 'ls' in that launchpad bug report, is to
> >> > force an mtime update (and to fill the readdir cache).
> >>
> >> Ok, interesting. I guess though that 'something' needs to read at least
> >> a file in that particular directory for the directory content to be
> >> cached, right? (i.e. either an 'ls' or some create/delete of a file?)
> >
> > Every time you use a filename, the act of looking up that name is
> > cached. The parent directory's mtime is also cached. If it changes, then
> > the cached lookup is invalidated. If not, then the cached lookup is
> > assumed still valid (since the directory contents are not supposed to
> > have changed).
> 
> Great thanks, got that!
> 
> Is there some docu about this level of NFS detail you know of?
> Esp the attribute and the data cache?

There should be some information (perhaps a bit out of date, but most of
it good) in the old FAQ here: http://nfs.sourceforge.net/

Cheers
  Trond

Re: NFS v3 cached directory content out of sync

[Chuck Lever]

On Aug 25, 2009, at 3:25 PM, Trond Myklebust wrote:
> On Tue, 2009-08-25 at 21:19 +0200, Stefan Egli wrote:
>> On Tue, Aug 25, 2009 at 9:12 PM, Trond Myklebust wrote:
>>> On Tue, 2009-08-25 at 21:06 +0200, Stefan Egli wrote:
>>>> On Tue, Aug 25, 2009 at 9:00 PM, Trond Myklebust wrote:
>>>>> Yes. The NFS client will cache mtime whether or not you do an  
>>>>> 'ls'. I
>>>>> suspect the only reason for the 'ls' in that launchpad bug  
>>>>> report, is to
>>>>> force an mtime update (and to fill the readdir cache).
>>>>
>>>> Ok, interesting. I guess though that 'something' needs to read at  
>>>> least
>>>> a file in that particular directory for the directory content to be
>>>> cached, right? (i.e. either an 'ls' or some create/delete of a  
>>>> file?)
>>>
>>> Every time you use a filename, the act of looking up that name is
>>> cached. The parent directory's mtime is also cached. If it  
>>> changes, then
>>> the cached lookup is invalidated. If not, then the cached lookup is
>>> assumed still valid (since the directory contents are not supposed  
>>> to
>>> have changed).
>>
>> Great thanks, got that!
>>
>> Is there some docu about this level of NFS detail you know of?
>> Esp the attribute and the data cache?
>
> There should be some information (perhaps a bit out of date, but  
> most of
> it good) in the old FAQ here: http://nfs.sourceforge.net/

nfs(5) has an extensive addendum on metadata caching.  An upcoming  
update will document the lookupcache behavior.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: NFS v3 cached directory content out of sync

[Stefan Egli]

Thanks Mates! I've seen those manuals but the bug I'm chasing doesn't
seem to be documented ;)

Just now that I got your attention :) one final clarification question:
 * the attribute cache (==metadata cache) is part of the nfs client code
 * the date (=file content) cache would be the normal linux caching

Am I on the right track there?

Cheers,
Stefan

Re: NFS v3 cached directory content out of sync

[Trond Myklebust]

On Tue, 2009-08-25 at 21:45 +0200, Stefan Egli wrote:
> Thanks Mates! I've seen those manuals but the bug I'm chasing doesn't
> seem to be documented ;)
> 
> Just now that I got your attention :) one final clarification question:
>  * the attribute cache (==metadata cache) is part of the nfs client code
>  * the date (=file content) cache would be the normal linux caching

Yes.
     data == file contents.
 metadata == file attributes (i.e. stat() info, acls,...)

Both will be cached by the client, but the rules for how to cache them
differ.

Re: NFS v3 cached directory content out of sync

[Chuck Lever]

On Aug 25, 2009, at 3:45 PM, Stefan Egli wrote:
> Thanks Mates! I've seen those manuals but the bug I'm chasing doesn't
> seem to be documented ;)
>
> Just now that I got your attention :) one final clarification  
> question:
> * the attribute cache (==metadata cache) is part of the nfs client  
> code
> * the date (=file content) cache would be the normal linux caching

A file's mtime/ctime/atime is stored in the generic in-core inode.   
The inode cache is "normal linux caching".  File content is cached in  
the kernel's regular page cache, also considered "normal linux  
caching".  The difference for NFS files is how this information is  
updated and flushed back to persistent storage.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: NFS v3 cached directory content out of sync

[Stefan Egli]

FYI: For the reference a quick summary of this issue:

Linux Kernels older than 2.6.30 (or at least in 2.6.26 which we're
using) have this bug which can be reproduced as follows:

 * have two hosts: host A and host B - both having the same NFS
directory mounted and being in the same subdirectory on that NFS

 * host A: mkdir testdir; touch -m --date "2009-08-25 11:35" testdir
 * host B: ls testdir
 * host A: touch testdir/aNewFile; touch -m --date "2009-08-25 11:35" testdir
 * host B: ls testdir
   * shows an empty directory - even though 'aNewFile' just got
created. This remains so for hours.

The commit-set ID which is probably related (as of now I couldn't test
the above on 2.6.30):

37d9d76d8b3a2ac5817e1fa3263cfe0fdb439e51

Cheers,
Stefan