possible client stale filehandle bug?

possible client stale filehandle bug?

[Garrick Staples]

[-- Attachment #1: Type: text/plain, Size: 3095 bytes --]

Hi all,
   I have lots of storage in a large Solaris samfs environment that is NFS
shared to a large number of Solaris and RHEL3 clients.  Under some conditions,
linux apps have been getting stale filehandles during the normal course of
their activity.  Various file handling syscalls like read() or open() might
error.  Lots of renames and setattrs calls seem to trigger the problem.  
'ci' and 'cvs commit' are particularly good at this.

It seems that the Solaris clients never report any such errors, only the Linux
clients.  However, watching 'snoop' on the Solaris NFS server, I see that it IS
returning stale file handles to both OSes, but Solaris clients seem to retry
the request several times; and the Linux clients immediately pass the error up
to the application.

Is there some condition that the 2.4 kernel is handling incorrectly?


Sample snippet from the 'snoop' on the Solaris server with a Solaris client
waiting...

rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log
almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE
rcf102.usc.edu -> almaak.usc.edu TCP D=2049 S=610     Ack=3071279992 Seq=337022612 Len=0 Win=64240
rcf102.usc.edu -> almaak.usc.edu NFS C ACCESS3 FH=7BFE (read,modify,extend,execute)
almaak.usc.edu -> rcf102.usc.edu TCP D=610 S=2049     Ack=337022752 Seq=3071279992 Len=0 Win=64240
almaak.usc.edu -> rcf102.usc.edu NFS R ACCESS3 Stale NFS file handle
rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log
almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE
rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log
almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE
rcf102.usc.edu -> almaak.usc.edu TCP D=2049 S=610     Ack=3071280516 Seq=337023056 Len=0 Win=64240
rcf102.usc.edu -> almaak.usc.edu NFS C ACCESS3 FH=7BFE (read,modify,extend,execute)
almaak.usc.edu -> rcf102.usc.edu TCP D=610 S=2049     Ack=337023196 Seq=3071280516 Len=0 Win=64240
almaak.usc.edu -> rcf102.usc.edu NFS R ACCESS3 Stale NFS file handle
rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log
almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE
rcf102.usc.edu -> almaak.usc.edu TCP D=2049 S=610     Ack=3071280796 Seq=337023348 Len=0 Win=64240
rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log
almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE
rcf102.usc.edu -> almaak.usc.edu TCP D=2049 S=610     Ack=3071281040 Seq=337023500 Len=0 Win=64240
rcf102.usc.edu -> almaak.usc.edu NFS C ACCESS3 FH=7BFE (read,modify,extend,execute)
almaak.usc.edu -> rcf102.usc.edu TCP D=610 S=2049     Ack=337023640 Seq=3071281040 Len=0 Win=64240
almaak.usc.edu -> rcf102.usc.edu NFS R ACCESS3 Stale NFS file handle
rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log
almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE
rcf102.usc.edu -> almaak.usc.edu NFS C LOOKUP3 FH=B41B Entries.Log
almaak.usc.edu -> rcf102.usc.edu NFS R LOOKUP3 OK FH=7BFE

-- 
Garrick Staples, Linux/HPCC Administrator
University of Southern California

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: possible client stale filehandle bug?

[Trond Myklebust]

ty den 25.01.2005 Klokka 09:39 (-0800) skreiv Garrick Staples:
> Hi all,
>    I have lots of storage in a large Solaris samfs environment that is NFS
> shared to a large number of Solaris and RHEL3 clients.  Under some conditions,
> linux apps have been getting stale filehandles during the normal course of
> their activity.  Various file handling syscalls like read() or open() might
> error.  Lots of renames and setattrs calls seem to trigger the problem.  
> 'ci' and 'cvs commit' are particularly good at this.

ESTALE is usually a sign that someone is deleting a file on the server
that is in use by the client. It is a sign that you are doing something
that violates the caching rules of NFS.

> It seems that the Solaris clients never report any such errors, only the Linux
> clients.  However, watching 'snoop' on the Solaris NFS server, I see that it IS
> returning stale file handles to both OSes, but Solaris clients seem to retry
> the request several times; and the Linux clients immediately pass the error up
> to the application.
> 
> Is there some condition that the 2.4 kernel is handling incorrectly?

I do not believe that Solaris redrives ESTALE on read, but they may do
it on open(). Linux does not redrive either case. See the many
discussions in the NFS list archives for why.

Cheers,
  Trond

 
-- 
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: possible client stale filehandle bug?

[Garrick Staples]

[-- Attachment #1: Type: text/plain, Size: 2419 bytes --]

On Tue, Jan 25, 2005 at 10:06:27PM -0800, Trond Myklebust alleged:
> ty den 25.01.2005 Klokka 09:39 (-0800) skreiv Garrick Staples:
> > Hi all,
> >    I have lots of storage in a large Solaris samfs environment that is NFS
> > shared to a large number of Solaris and RHEL3 clients.  Under some conditions,
> > linux apps have been getting stale filehandles during the normal course of
> > their activity.  Various file handling syscalls like read() or open() might
> > error.  Lots of renames and setattrs calls seem to trigger the problem.  
> > 'ci' and 'cvs commit' are particularly good at this.
> 
> ESTALE is usually a sign that someone is deleting a file on the server
> that is in use by the client. It is a sign that you are doing something
> that violates the caching rules of NFS.

Nothing of the kind is happening here.  I've tested this a thousand times over
the last few days trying to find a solution.  In this case, Sun's samfs
filesystem is definitely at fault and doing the wrong thing.  Backline
engineers at Sun confirm this and are working on a fix.  

The reason for _this_ email isn't because of the ESTALEs, it's regarding the
handling of the ESTALEs.  Right now I need the Solaris client behaviour to
deal with this particular buggy server.

Incidentally, 2.6.10 never has a problem.  It's behaviour never creates ESTALEs in
the first place.

 
> > It seems that the Solaris clients never report any such errors, only the Linux
> > clients.  However, watching 'snoop' on the Solaris NFS server, I see that it IS
> > returning stale file handles to both OSes, but Solaris clients seem to retry
> > the request several times; and the Linux clients immediately pass the error up
> > to the application.
> > 
> > Is there some condition that the 2.4 kernel is handling incorrectly?
> 
> I do not believe that Solaris redrives ESTALE on read, but they may do
> it on open(). Linux does not redrive either case. See the many
> discussions in the NFS list archives for why.

Did you look at the 'snoop' bits in the previous email?  During that time, the
process on the Solaris client is hanging in a write() call.

I'd be very happy to see any patches lieing around that might do this
behaviour.  It would get me through the short term until Sun fixes this bug in
samfs.

-- 
Garrick Staples, Linux/HPCC Administrator
University of Southern California

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: possible client stale filehandle bug?

[Neil Horman]

Trond Myklebust wrote:
> ty den 25.01.2005 Klokka 09:39 (-0800) skreiv Garrick Staples:
> 
>>Hi all,
>>   I have lots of storage in a large Solaris samfs environment that is NFS
>>shared to a large number of Solaris and RHEL3 clients.  Under some conditions,
>>linux apps have been getting stale filehandles during the normal course of
>>their activity.  Various file handling syscalls like read() or open() might
>>error.  Lots of renames and setattrs calls seem to trigger the problem.  
>>'ci' and 'cvs commit' are particularly good at this.
> 
> 
> ESTALE is usually a sign that someone is deleting a file on the server
> that is in use by the client. It is a sign that you are doing something
> that violates the caching rules of NFS.
> 
> 
>>It seems that the Solaris clients never report any such errors, only the Linux
>>clients.  However, watching 'snoop' on the Solaris NFS server, I see that it IS
>>returning stale file handles to both OSes, but Solaris clients seem to retry
>>the request several times; and the Linux clients immediately pass the error up
>>to the application.
>>
>>Is there some condition that the 2.4 kernel is handling incorrectly?
> 
> 
> I do not believe that Solaris redrives ESTALE on read, but they may do
> it on open(). Linux does not redrive either case. See the many
> discussions in the NFS list archives for why.
> 

Solaris does in fact retry on operations on ESTALE errors, definately on 
open, and I think on read/readdir/stat/etc. as well.  We had some 
discussion about tht here recently.


-- 
/***************************************************
  *Neil Horman
  *Software Engineer
  *Red Hat, Inc.
  *nhorman@redhat.com
  *gpg keyid: 1024D / 0x92A74FA1
  *http://pgp.mit.edu
  ***************************************************/


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: possible client stale filehandle bug?

[Neil Horman]

[-- Attachment #1: Type: text/plain, Size: 2949 bytes --]

Garrick Staples wrote:
> On Tue, Jan 25, 2005 at 10:06:27PM -0800, Trond Myklebust alleged:
> 
>>ty den 25.01.2005 Klokka 09:39 (-0800) skreiv Garrick Staples:
>>
>>>Hi all,
>>>   I have lots of storage in a large Solaris samfs environment that is NFS
>>>shared to a large number of Solaris and RHEL3 clients.  Under some conditions,
>>>linux apps have been getting stale filehandles during the normal course of
>>>their activity.  Various file handling syscalls like read() or open() might
>>>error.  Lots of renames and setattrs calls seem to trigger the problem.  
>>>'ci' and 'cvs commit' are particularly good at this.
>>
>>ESTALE is usually a sign that someone is deleting a file on the server
>>that is in use by the client. It is a sign that you are doing something
>>that violates the caching rules of NFS.
> 
> 
> Nothing of the kind is happening here.  I've tested this a thousand times over
> the last few days trying to find a solution.  In this case, Sun's samfs
> filesystem is definitely at fault and doing the wrong thing.  Backline
> engineers at Sun confirm this and are working on a fix.  
> 
> The reason for _this_ email isn't because of the ESTALEs, it's regarding the
> handling of the ESTALEs.  Right now I need the Solaris client behaviour to
> deal with this particular buggy server.
> 
> Incidentally, 2.6.10 never has a problem.  It's behaviour never creates ESTALEs in
> the first place.
> 
>  
> 
>>>It seems that the Solaris clients never report any such errors, only the Linux
>>>clients.  However, watching 'snoop' on the Solaris NFS server, I see that it IS
>>>returning stale file handles to both OSes, but Solaris clients seem to retry
>>>the request several times; and the Linux clients immediately pass the error up
>>>to the application.
>>>
>>>Is there some condition that the 2.4 kernel is handling incorrectly?
>>
>>I do not believe that Solaris redrives ESTALE on read, but they may do
>>it on open(). Linux does not redrive either case. See the many
>>discussions in the NFS list archives for why.
> 
> 
> Did you look at the 'snoop' bits in the previous email?  During that time, the
> process on the Solaris client is hanging in a write() call.
> 
> I'd be very happy to see any patches lieing around that might do this
> behaviour.  It would get me through the short term until Sun fixes this bug in
> samfs.
> 

The easiest, cleanest thing to do is retry the operation in the sys_* 
function.  Red Hat's latest kernel does this to prevent ESTALE.  I've 
attached the patch for your reference.  Mind you, due to the reasons 
you'll find in the list archives that Trond referenced, you'll not see 
this get into upstream kernels, so caveat emptor.

HTH
Neil


-- 
/***************************************************
  *Neil Horman
  *Software Engineer
  *Red Hat, Inc.
  *nhorman@redhat.com
  *gpg keyid: 1024D / 0x92A74FA1
  *http://pgp.mit.edu
  ***************************************************/

[-- Attachment #2: linux-2.4.21-nfs-estale.patch --]
[-- Type: text/plain, Size: 5410 bytes --]

diff -urNp linux-6090/fs/nfs/inode.c linux-6091/fs/nfs/inode.c
--- linux-6090/fs/nfs/inode.c
+++ linux-6091/fs/nfs/inode.c
@@ -1022,7 +1022,15 @@ int
 nfs_revalidate(struct dentry *dentry)
 {
 	struct inode *inode = dentry->d_inode;
-	return nfs_revalidate_inode(NFS_SERVER(inode), inode);
+	int error;
+
+	error = nfs_revalidate_inode(NFS_SERVER(inode), inode);
+	if (error == -ESTALE) {
+		nfs_zap_caches(dentry->d_parent->d_inode);
+		d_drop(dentry);
+	}
+
+	return error;
 }
 
 /*
diff -urNp linux-6090/fs/nfs/xattr.c linux-6091/fs/nfs/xattr.c
--- linux-6090/fs/nfs/xattr.c
+++ linux-6091/fs/nfs/xattr.c
@@ -53,8 +53,13 @@ nfs_getxattr(struct dentry *dentry, cons
 	acl = ERR_PTR(-EOPNOTSUPP);
 	if (NFS_PROTO(inode)->version == 3 && NFS_PROTO(inode)->getacl)
 		acl = NFS_PROTO(inode)->getacl(inode, type);
-	if (IS_ERR(acl))
+	if (IS_ERR(acl)) {
+		if (PTR_ERR(acl) == -ESTALE) {
+			nfs_zap_caches(dentry->d_parent->d_inode);
+			d_drop(dentry);
+		}
 		return PTR_ERR(acl);
+	}
 	else if (acl) {
 		if (type == ACL_TYPE_ACCESS && acl->a_count == 0)
 			error = -ENODATA;
diff -urNp linux-6090/fs/open.c linux-6091/fs/open.c
--- linux-6090/fs/open.c
+++ linux-6091/fs/open.c
@@ -807,6 +807,30 @@ asmlinkage long sys_open(const char * fi
 		if (fd >= 0) {
 			struct file *f = filp_open(tmp, flags, mode);
 			error = PTR_ERR(f);
+
+			/*
+			 * ESTALE errors can be a pain.  On some
+			 * filesystems (e.g. NFS), ESTALE can often
+			 * be resolved by retry, as the ESTALE resulted
+			 * in a cache invalidation.  We perform this
+			 * retry here, once for every directory element
+			 * in the path to avoid the case where the removal
+			 * of the nth parent directory of the file we're
+			 * trying to open results in n ESTALE errors.
+			 */
+			if (error == -ESTALE) {
+				int nretries = 1;
+				char *cp;
+
+				for (cp = tmp; *cp; cp++) {
+					if (*cp == '/')
+						nretries++;
+				}
+				do {
+					f = filp_open(tmp, flags, mode);
+					error = PTR_ERR(f);
+				} while (error == -ESTALE && --nretries > 0);
+			}
 			if (IS_ERR(f))
 				goto out_error;
 			fd_install(fd, f);
diff -urNp linux-6090/fs/stat.c linux-6091/fs/stat.c
--- linux-6090/fs/stat.c
+++ linux-6091/fs/stat.c
@@ -143,8 +143,9 @@ static int cp_new_stat(struct inode * in
 asmlinkage long sys_stat(char * filename, struct __old_kernel_stat * statbuf)
 {
 	struct nameidata nd;
-	int error;
+	int error, errcnt = 0;
 
+again:
 	error = user_path_walk(filename, &nd);
 	if (!error) {
 		error = do_revalidate(nd.dentry);
@@ -152,6 +153,10 @@ asmlinkage long sys_stat(char * filename
 			error = cp_old_stat(nd.dentry->d_inode, statbuf);
 		path_release(&nd);
 	}
+	if (error == -ESTALE && !errcnt) {
+		errcnt++;
+		goto again;
+	}
 	return error;
 }
 #endif
@@ -159,8 +164,9 @@ asmlinkage long sys_stat(char * filename
 asmlinkage long sys_newstat(char * filename, struct stat * statbuf)
 {
 	struct nameidata nd;
-	int error;
+	int error, errcnt = 0;
 
+again:
 	error = user_path_walk(filename, &nd);
 	if (!error) {
 		error = do_revalidate(nd.dentry);
@@ -168,6 +174,11 @@ asmlinkage long sys_newstat(char * filen
 			error = cp_new_stat(nd.dentry->d_inode, statbuf);
 		path_release(&nd);
 	}
+	if (error == -ESTALE && !errcnt) {
+		errcnt++;
+		goto again;
+	}
+
 	return error;
 }
 
@@ -180,8 +191,9 @@ asmlinkage long sys_newstat(char * filen
 asmlinkage long sys_lstat(char * filename, struct __old_kernel_stat * statbuf)
 {
 	struct nameidata nd;
-	int error;
+	int error, errcnt = 0;
 
+again:
 	error = user_path_walk_link(filename, &nd);
 	if (!error) {
 		error = do_revalidate(nd.dentry);
@@ -189,6 +201,11 @@ asmlinkage long sys_lstat(char * filenam
 			error = cp_old_stat(nd.dentry->d_inode, statbuf);
 		path_release(&nd);
 	}
+	if (error == -ESTALE && !errcnt) {
+		errcnt++;
+		goto again;
+	}
+
 	return error;
 }
 
@@ -197,8 +214,9 @@ asmlinkage long sys_lstat(char * filenam
 asmlinkage long sys_newlstat(char * filename, struct stat * statbuf)
 {
 	struct nameidata nd;
-	int error;
+	int error, errcnt = 0;
 
+again:
 	error = user_path_walk_link(filename, &nd);
 	if (!error) {
 		error = do_revalidate(nd.dentry);
@@ -206,6 +224,12 @@ asmlinkage long sys_newlstat(char * file
 			error = cp_new_stat(nd.dentry->d_inode, statbuf);
 		path_release(&nd);
 	}
+
+	if (error == -ESTALE && !errcnt) {
+		errcnt++;
+		goto again;
+	}
+
 	return error;
 }
 
@@ -340,8 +364,9 @@ static long cp_new_stat64(struct inode *
 asmlinkage long sys_stat64(char * filename, struct stat64 * statbuf, long flags)
 {
 	struct nameidata nd;
-	int error;
+	int error, errcnt = 0;
 
+again:
 	error = user_path_walk(filename, &nd);
 	if (!error) {
 		error = do_revalidate(nd.dentry);
@@ -349,14 +374,20 @@ asmlinkage long sys_stat64(char * filena
 			error = cp_new_stat64(nd.dentry->d_inode, statbuf);
 		path_release(&nd);
 	}
+	if (error == -ESTALE && !errcnt) {
+		errcnt++;
+		goto again;
+	}
+
 	return error;
 }
 
 asmlinkage long sys_lstat64(char * filename, struct stat64 * statbuf, long flags)
 {
 	struct nameidata nd;
-	int error;
+	int error, errcnt = 0;
 
+again:
 	error = user_path_walk_link(filename, &nd);
 	if (!error) {
 		error = do_revalidate(nd.dentry);
@@ -364,6 +395,11 @@ asmlinkage long sys_lstat64(char * filen
 			error = cp_new_stat64(nd.dentry->d_inode, statbuf);
 		path_release(&nd);
 	}
+	if (error == -ESTALE && !errcnt) {
+		errcnt++;
+		goto again;
+	}
+
 	return error;
 }
 

Re: possible client stale filehandle bug?

[raven]

On Tue, 25 Jan 2005, Garrick Staples wrote:

>
> I'd be very happy to see any patches lieing around that might do this
> behaviour.  It would get me through the short term until Sun fixes this bug in
> samfs.

Don't hold your breath waiting!

Ian



-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: possible client stale filehandle bug?

[Garrick Staples]

[-- Attachment #1: Type: text/plain, Size: 455 bytes --]

On Wed, Jan 26, 2005 at 10:31:57PM +0800, raven@themaw.net alleged:
> On Tue, 25 Jan 2005, Garrick Staples wrote:
> 
> >
> >I'd be very happy to see any patches lieing around that might do this
> >behaviour.  It would get me through the short term until Sun fixes this 
> >bug in
> >samfs.
> 
> Don't hold your breath waiting!

Now you are just being cruel!

-- 
Garrick Staples, Linux/HPCC Administrator
University of Southern California

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: possible client stale filehandle bug?

[Ian Kent]

On Wed, 26 Jan 2005, Garrick Staples wrote:

> On Wed, Jan 26, 2005 at 10:31:57PM +0800, raven@themaw.net alleged:
> > On Tue, 25 Jan 2005, Garrick Staples wrote:
> > 
> > >
> > >I'd be very happy to see any patches lieing around that might do this
> > >behaviour.  It would get me through the short term until Sun fixes this 
> > >bug in
> > >samfs.
> > 
> > Don't hold your breath waiting!
> 
> Now you are just being cruel!

We don't use SAMfs in the standard configuration either!

Oh joy, oh bliss!

Sun purchasing LSCI was a real bummer for us.

Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: possible client stale filehandle bug?

[nhorman]

On Thu, Feb 24, 2005 at 11:33:43AM -0800, Trond Myklebust wrote:
> on den 16.02.2005 Klokka 16:23 (-0500) skreiv Neil Horman:
> > I agree, it probably doesn't re-drive on any operation that doesn't walk 
> > a path, which is in line with what RHEL is doing currently.  I didn't 
> > mean to imply that solaris retired ESTALE in all occurances of the 
> > event.  Anywho, Can you point me to your patches?  I'd be interested to 
> > know how you managed to implement retry on ESTALE without leaking into 
> > the VFS, which I think you will recall was the big sticking point that 
> > we were debating here.
> 
> It does leak into the VFS, but in a way that is non-intrusive. The VFS
> basically sets a new flag, LOOKUP_REVAL, in the struct nameidata flags
> (which are passed down to d_op->d_revalidate() on Linux 2.4.x).
> The NFS code takes that as an command to expire the caches, and force a
> revalidation.
> 
> The exact same trick is performed on Linux-2.6.x: see the patch
> "linux-2.6.11-31-stale_retry.dif"  in
>      http://client.linux-nfs.org/Linux-2.6.x/2.6.11-rc5/
> 
> Cheers,
>   Trond
> 
> -- 
> Trond Myklebust <trond.myklebust@fys.uio.no>
> 

Thanks for the pointer!
Neil

-- 
/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *nhorman@redhat.com
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: possible client stale filehandle bug?

[Trond Myklebust]

on den 16.02.2005 Klokka 16:23 (-0500) skreiv Neil Horman:
> I agree, it probably doesn't re-drive on any operation that doesn't walk 
> a path, which is in line with what RHEL is doing currently.  I didn't 
> mean to imply that solaris retired ESTALE in all occurances of the 
> event.  Anywho, Can you point me to your patches?  I'd be interested to 
> know how you managed to implement retry on ESTALE without leaking into 
> the VFS, which I think you will recall was the big sticking point that 
> we were debating here.

It does leak into the VFS, but in a way that is non-intrusive. The VFS
basically sets a new flag, LOOKUP_REVAL, in the struct nameidata flags
(which are passed down to d_op->d_revalidate() on Linux 2.4.x).
The NFS code takes that as an command to expire the caches, and force a
revalidation.

The exact same trick is performed on Linux-2.6.x: see the patch
"linux-2.6.11-31-stale_retry.dif"  in
     http://client.linux-nfs.org/Linux-2.6.x/2.6.11-rc5/

Cheers,
  Trond

-- 
Trond Myklebust <trond.myklebust@fys.uio.no>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: possible client stale filehandle bug?

[Neil Horman]

Lever, Charles wrote:
>>I agree, it probably doesn't re-drive on any operation that 
>>doesn't walk 
>>a path, which is in line with what RHEL is doing currently.  I didn't 
>>mean to imply that solaris retired ESTALE in all occurances of the 
>>event.  Anywho, Can you point me to your patches?  I'd be 
>>interested to 
>>know how you managed to implement retry on ESTALE without 
>>leaking into 
>>the VFS, which I think you will recall was the big sticking 
>>point that 
>>we were debating here.
> 
> 
> the patches do touch fs/namei.c (it was al viro's suggestion) with a
> pretty simple change.  and i think they are KABI friendly enough to be
> included in RHEL 3, once we are satisfied that the solution is
> effective.
> 
> the cto-lookup-revalidate patch adds just enough of the 2.6
> lookup-intent logic to the 2.4 VFS layer to allow us to support NFS
> close-to-open in nfs_lookup_revalidate instead of in nfs_open.  this
> resolves one of the most common ESTALE failure modes, where just the
> object at the end of the pathname has been replaced.
> 
> the second patch applies on top of this.  it adds logic to redrive
> pathname resolution if an ESTALE is encountered anywhere during a
> pathname lookup.  it redrives it once from the top, asserting a flag
> that causes the VFS layer to abandon the dcache and use only real
> lookups for this resolution request.  if the redriven resolution fails,
> we give up.  this resolves the other typical ESTALE failure mode, where
> some or all of the path has been replaced, while avoiding retrying an
> unbounded number of times.
Fantastic, Thanks!
Neil

-- 
/***************************************************
  *Neil Horman
  *Software Engineer
  *Red Hat, Inc.
  *nhorman@redhat.com
  *gpg keyid: 1024D / 0x92A74FA1
  *http://pgp.mit.edu
  ***************************************************/


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: possible client stale filehandle bug?

[Lever, Charles]

[-- Attachment #1: Type: text/plain, Size: 1555 bytes --]

> I agree, it probably doesn't re-drive on any operation that 
> doesn't walk 
> a path, which is in line with what RHEL is doing currently.  I didn't 
> mean to imply that solaris retired ESTALE in all occurances of the 
> event.  Anywho, Can you point me to your patches?  I'd be 
> interested to 
> know how you managed to implement retry on ESTALE without 
> leaking into 
> the VFS, which I think you will recall was the big sticking 
> point that 
> we were debating here.

the patches do touch fs/namei.c (it was al viro's suggestion) with a
pretty simple change.  and i think they are KABI friendly enough to be
included in RHEL 3, once we are satisfied that the solution is
effective.

the cto-lookup-revalidate patch adds just enough of the 2.6
lookup-intent logic to the 2.4 VFS layer to allow us to support NFS
close-to-open in nfs_lookup_revalidate instead of in nfs_open.  this
resolves one of the most common ESTALE failure modes, where just the
object at the end of the pathname has been replaced.

the second patch applies on top of this.  it adds logic to redrive
pathname resolution if an ESTALE is encountered anywhere during a
pathname lookup.  it redrives it once from the top, asserting a flag
that causes the VFS layer to abandon the dcache and use only real
lookups for this resolution request.  if the redriven resolution fails,
we give up.  this resolves the other typical ESTALE failure mode, where
some or all of the path has been replaced, while avoiding retrying an
unbounded number of times.

[-- Attachment #2: linux-2.4.21-nfs-cto-lookup-revalidate.patch --]
[-- Type: application/octet-stream, Size: 4728 bytes --]

diff -X /home/cel/src/linux/dont-diff -Naurp linux-2.4.21/fs/exec.c new/fs/exec.c
--- linux-2.4.21/fs/exec.c	2004-12-23 12:28:49.582148000 -0500
+++ new/fs/exec.c	2004-12-23 12:29:53.852334000 -0500
@@ -402,7 +402,7 @@ struct file *open_exec(const char *name)
 	struct file *file;
 	int err = 0;
 
-	err = path_lookup(name, LOOKUP_FOLLOW|LOOKUP_POSITIVE, &nd);
+	err = path_lookup(name, LOOKUP_FOLLOW|LOOKUP_POSITIVE|LOOKUP_OPEN, &nd);
 	file = ERR_PTR(err);
 	if (!err) {
 		inode = nd.dentry->d_inode;
diff -X /home/cel/src/linux/dont-diff -Naurp linux-2.4.21/fs/namei.c new/fs/namei.c
--- linux-2.4.21/fs/namei.c	2004-12-23 12:28:38.884159000 -0500
+++ new/fs/namei.c	2004-12-23 12:29:53.857334000 -0500
@@ -645,7 +645,7 @@ return_reval:
 		dentry = nd->dentry;
 		if (dentry && dentry->d_op && dentry->d_op->d_revalidate) {
 			err = -ESTALE;
-			if (!dentry->d_op->d_revalidate(dentry, 0)) {
+			if (!dentry->d_op->d_revalidate(dentry, nd->flags & LOOKUP_OPEN)) {
 				d_invalidate(dentry);
 				break;
 			}
@@ -951,7 +951,7 @@ static inline int may_create(struct inod
  */
 static inline int lookup_flags(unsigned int f)
 {
-	unsigned long retval = LOOKUP_FOLLOW;
+	unsigned long retval = LOOKUP_FOLLOW | LOOKUP_OPEN;
 
 	if (f & O_NOFOLLOW)
 		retval &= ~LOOKUP_FOLLOW;
@@ -1032,7 +1032,7 @@ int open_namei(const char * pathname, in
 	/*
 	 * Create - we need to know the parent.
 	 */
-	error = path_lookup(pathname, LOOKUP_PARENT, nd);
+	error = path_lookup(pathname, LOOKUP_PARENT | LOOKUP_OPEN, nd);
 	if (error)
 		return error;
 
diff -X /home/cel/src/linux/dont-diff -Naurp linux-2.4.21/fs/nfs/dir.c new/fs/nfs/dir.c
--- linux-2.4.21/fs/nfs/dir.c	2004-12-23 12:28:39.850868000 -0500
+++ new/fs/nfs/dir.c	2004-12-23 12:29:53.861335000 -0500
@@ -505,10 +505,20 @@ static inline void nfs_renew_times(struc
 	dentry->d_time = jiffies;
 }
 
-static inline
-int nfs_lookup_verify_inode(struct inode *inode)
+/*
+ * Refresh the inode if the attribute cache has expired.
+ *
+ * During an open(2), force an inode update in order to maintain close-
+ * to-open cache coherency, unless the "nocto" mount option is set.
+ */
+static inline int nfs_lookup_verify_inode(struct inode *inode, int flags)
 {
-	return nfs_revalidate_inode(NFS_SERVER(inode), inode);
+	if (!(flags & LOOKUP_OPEN) ||
+	    (NFS_SERVER(inode)->flags & NFS_MOUNT_NOCTO))
+		return nfs_revalidate_inode(NFS_SERVER(inode), inode);
+	dfprintk(VFS, "NFS: %s: inode (%x/%Ld) revalidate on open\n",
+		__FUNCTION__, inode->i_dev, (long long)NFS_FILEID(inode));
+	return __nfs_revalidate_inode(NFS_SERVER(inode),inode);
 }
 
 /*
@@ -562,7 +572,7 @@ static int nfs_lookup_revalidate(struct 
 
 	/* Force a full look up iff the parent directory has changed */
 	if (nfs_check_verifier(dir, dentry)) {
-		if (nfs_lookup_verify_inode(inode))
+		if (nfs_lookup_verify_inode(inode, flags))
 			goto out_bad;
 		goto out_valid;
 	}
@@ -571,7 +581,7 @@ static int nfs_lookup_revalidate(struct 
 	if (!error) {
 		if (memcmp(NFS_FH(inode), &fhandle, sizeof(struct nfs_fh))!= 0)
 			goto out_bad;
-		if (nfs_lookup_verify_inode(inode))
+		if (nfs_lookup_verify_inode(inode, flags))
 			goto out_bad;
 		goto out_valid_renew;
 	}
diff -X /home/cel/src/linux/dont-diff -Naurp linux-2.4.21/fs/nfs/inode.c new/fs/nfs/inode.c
--- linux-2.4.21/fs/nfs/inode.c	2004-12-23 12:28:50.419958000 -0500
+++ new/fs/nfs/inode.c	2004-12-23 12:31:15.610732000 -0500
@@ -1005,23 +1005,18 @@ int nfs_open(struct inode *inode, struct
 {
 	struct rpc_auth *auth;
 	struct rpc_cred *cred;
-	int err = 0;
+
+	dfprintk(VFS, "NFS: nfs_open (%x/%Ld)\n",
+		inode->i_dev, (long long)NFS_FILEID(inode));
 
 	lock_kernel();
-	/* Ensure that we revalidate the data cache */
-	if (! (NFS_SERVER(inode)->flags & NFS_MOUNT_NOCTO)) {
-		err = __nfs_revalidate_inode(NFS_SERVER(inode),inode);
-		if (err)
-			goto out;
-	}
 	auth = NFS_CLIENT(inode)->cl_auth;
 	cred = rpcauth_lookupcred(auth, 0);
 	filp->private_data = cred;
 	if (filp->f_mode & FMODE_WRITE)
 		nfs_set_mmcred(inode, cred);
-out:
 	unlock_kernel();
-	return err;
+	return 0;
 }
 
 int nfs_release(struct inode *inode, struct file *filp)
diff -X /home/cel/src/linux/dont-diff -Naurp linux-2.4.21/include/linux/fs.h new/include/linux/fs.h
--- linux-2.4.21/include/linux/fs.h	2004-12-23 12:28:50.382958000 -0500
+++ new/include/linux/fs.h	2004-12-23 12:29:53.871334000 -0500
@@ -1407,6 +1407,7 @@ static inline long IS_ERR(const void *pt
 #define LOOKUP_PARENT		(16)
 #define LOOKUP_NOALT		(32)
 #define LOOKUP_ATOMIC		(64)
+#define LOOKUP_OPEN		(128)
 /*
  * Type of the last component on LOOKUP_PARENT
  */

[-- Attachment #3: linux-2.4.21-pathname-retry.patch --]
[-- Type: application/octet-stream, Size: 5935 bytes --]

diff -X /home/cel/src/linux/dont-diff -Naurp linux-2.4.21/fs/namei.c new/fs/namei.c
--- linux-2.4.21/fs/namei.c	2005-02-09 09:13:30.000000000 -0500
+++ new/fs/namei.c	2005-02-14 15:27:27.000000000 -0500
@@ -447,7 +447,7 @@ static inline void follow_dotdot(struct 
  *
  * We expect 'base' to be positive and a directory.
  */
-int link_path_walk(const char * name, struct nameidata *nd)
+static int __link_path_walk(const char *name, struct nameidata *nd)
 {
 	struct dentry *dentry;
 	struct inode *inode;
@@ -645,7 +645,7 @@ return_reval:
 		dentry = nd->dentry;
 		if (dentry && dentry->d_op && dentry->d_op->d_revalidate) {
 			err = -ESTALE;
-			if (!dentry->d_op->d_revalidate(dentry, nd->flags & LOOKUP_OPEN)) {
+			if (!dentry->d_op->d_revalidate(dentry, nd->flags)) {
 				d_invalidate(dentry);
 				break;
 			}
@@ -661,6 +661,37 @@ return_err:
 	return err;
 }
 
+/*
+ * Wrapper to retry pathname resolution whenever the underlying
+ * file system returns an ESTALE.
+ *
+ * Retry the whole path once, forcing real lookup requests
+ * instead of relying on the dcache.
+ */
+int link_path_walk(const char *name, struct nameidata *nd)
+{
+	struct nameidata save = *nd;
+	int result;
+
+	/* make sure the stuff we saved doesn't go away */
+	dget(save.dentry);
+	mntget(save.mnt);
+
+	result = __link_path_walk(name, nd);
+	if (result == -ESTALE) {
+		*nd = save;
+		dget(nd->dentry);
+		mntget(nd->mnt);
+		nd->flags |= LOOKUP_REVAL;
+		result = __link_path_walk(name, nd);
+	}
+
+	dput(save.dentry);
+	mntput(save.mnt);
+
+	return result;
+}
+
 int path_walk(const char * name, struct nameidata *nd)
 {
 	current->total_link_count = 0;
diff -X /home/cel/src/linux/dont-diff -Naurp linux-2.4.21/fs/nfs/dir.c new/fs/nfs/dir.c
--- linux-2.4.21/fs/nfs/dir.c	2005-02-09 09:13:30.000000000 -0500
+++ new/fs/nfs/dir.c	2005-02-14 15:36:37.000000000 -0500
@@ -513,11 +513,22 @@ static inline void nfs_renew_times(struc
  */
 static inline int nfs_lookup_verify_inode(struct inode *inode, int flags)
 {
-	if (!(flags & LOOKUP_OPEN) ||
-	    (NFS_SERVER(inode)->flags & NFS_MOUNT_NOCTO))
-		return nfs_revalidate_inode(NFS_SERVER(inode), inode);
-	dfprintk(VFS, "NFS: %s: inode (%x/%Ld) revalidate on open\n",
-		__FUNCTION__, inode->i_dev, (long long)NFS_FILEID(inode));
+	if (flags & LOOKUP_REVAL) {
+		dfprintk(VFS, "NFS: %s: inode (%x/%Ld) revalidate on estale\n",
+			__FUNCTION__, inode->i_dev, (long long)NFS_FILEID(inode));
+		goto force_revalidate;
+	}
+
+	if ((flags & LOOKUP_OPEN) &&
+	    !(NFS_SERVER(inode)->flags & NFS_MOUNT_NOCTO)) {
+		dfprintk(VFS, "NFS: %s: inode (%x/%Ld) revalidate on open\n",
+			__FUNCTION__, inode->i_dev, (long long)NFS_FILEID(inode));
+		goto force_revalidate;
+	}
+
+	return nfs_revalidate_inode(NFS_SERVER(inode), inode);
+
+force_revalidate:
 	return __nfs_revalidate_inode(NFS_SERVER(inode),inode);
 }
 
diff -X /home/cel/src/linux/dont-diff -Naurp linux-2.4.21/fs/nfs/nfs3proc.c new/fs/nfs/nfs3proc.c
--- linux-2.4.21/fs/nfs/nfs3proc.c	2005-02-09 09:13:16.000000000 -0500
+++ new/fs/nfs/nfs3proc.c	2005-02-09 09:23:09.000000000 -0500
@@ -61,7 +61,7 @@ nfs3_proc_get_root(struct nfs_server *se
 	dprintk("NFS call  getroot\n");
 	fattr->valid = 0;
 	status = rpc_call(server->client, NFS3PROC_GETATTR, fhandle, fattr, 0);
-	dprintk("NFS reply getroot\n");
+	dprintk("NFS reply getroot: %d\n", status);
 	return status;
 }
 
@@ -77,7 +77,7 @@ nfs3_proc_getattr(struct inode *inode, s
 	fattr->valid = 0;
 	status = rpc_call(NFS_CLIENT(inode), NFS3PROC_GETATTR,
 			  NFS_FH(inode), fattr, 0);
-	dprintk("NFS reply getattr\n");
+	dprintk("NFS reply getattr: %d\n", status);
 	return status;
 }
 
@@ -91,7 +91,7 @@ nfs3_proc_setattr(struct inode *inode, s
 	dprintk("NFS call  setattr\n");
 	fattr->valid = 0;
 	status = rpc_call(NFS_CLIENT(inode), NFS3PROC_SETATTR, &arg, fattr, 0);
-	dprintk("NFS reply setattr\n");
+	dprintk("NFS reply setattr: %d\n", status);
 	return status;
 }
 
@@ -162,7 +162,7 @@ nfs3_proc_access(struct inode *inode, st
 
 	status = rpc_call_sync(NFS_CLIENT(inode), &msg, 0);
 	nfs_refresh_inode(inode, &fattr);
-	dprintk("NFS reply access\n");
+	dprintk("NFS reply access: %d\n", status);
 
 	if (status == 0 && (access & res.access) != access)
 		status = -EACCES;
diff -X /home/cel/src/linux/dont-diff -Naurp linux-2.4.21/fs/nfs/proc.c new/fs/nfs/proc.c
--- linux-2.4.21/fs/nfs/proc.c	2005-02-09 09:12:47.000000000 -0500
+++ new/fs/nfs/proc.c	2005-02-09 09:23:09.000000000 -0500
@@ -56,7 +56,7 @@ nfs_proc_get_root(struct nfs_server *ser
 	dprintk("NFS call  getroot\n");
 	fattr->valid = 0;
 	status = rpc_call(server->client, NFSPROC_GETATTR, fhandle, fattr, 0);
-	dprintk("NFS reply getroot\n");
+	dprintk("NFS reply getroot: %d\n", status);
 	return status;
 }
 
@@ -72,7 +72,7 @@ nfs_proc_getattr(struct inode *inode, st
 	fattr->valid = 0;
 	status = rpc_call(NFS_CLIENT(inode), NFSPROC_GETATTR,
 				NFS_FH(inode), fattr, 0);
-	dprintk("NFS reply getattr\n");
+	dprintk("NFS reply getattr: %d\n", status);
 	return status;
 }
 
@@ -86,7 +86,7 @@ nfs_proc_setattr(struct inode *inode, st
 	dprintk("NFS call  setattr\n");
 	fattr->valid = 0;
 	status = rpc_call(NFS_CLIENT(inode), NFSPROC_SETATTR, &arg, fattr, 0);
-	dprintk("NFS reply setattr\n");
+	dprintk("NFS reply setattr: %d\n", status);
 	return status;
 }
 
diff -X /home/cel/src/linux/dont-diff -Naurp linux-2.4.21/include/linux/fs.h new/include/linux/fs.h
--- linux-2.4.21/include/linux/fs.h	2005-02-09 09:13:30.000000000 -0500
+++ new/include/linux/fs.h	2005-02-14 15:29:26.000000000 -0500
@@ -1408,6 +1408,7 @@ static inline long IS_ERR(const void *pt
 #define LOOKUP_NOALT		(32)
 #define LOOKUP_ATOMIC		(64)
 #define LOOKUP_OPEN		(128)
+#define LOOKUP_REVAL		(256)
 /*
  * Type of the last component on LOOKUP_PARENT
  */

Re: possible client stale filehandle bug?

[Neil Horman]

Lever, Charles wrote:
> hi neil-
> 
> 
>>>>It seems that the Solaris clients never report any such 
>>
>>errors, only the Linux
>>
>>>>clients.  However, watching 'snoop' on the Solaris NFS 
>>
>>server, I see that it IS
>>
>>>>returning stale file handles to both OSes, but Solaris 
>>
>>clients seem to retry
>>
>>>>the request several times; and the Linux clients 
>>
>>immediately pass the error up
>>
>>>>to the application.
>>>>
>>>>Is there some condition that the 2.4 kernel is handling incorrectly?
>>>
>>>
>>>I do not believe that Solaris redrives ESTALE on read, but 
>>
>>they may do
>>
>>>it on open(). Linux does not redrive either case. See the many
>>>discussions in the NFS list archives for why.
>>>
>>
>>Solaris does in fact retry on operations on ESTALE errors, 
>>definately on 
>>open, and I think on read/readdir/stat/etc. as well.  We had some 
>>discussion about tht here recently.
> 
> 
> as far as i know Solaris doesn't redrive on read or write, but only
> during pathname resolution.  redriving a read or write will only work in
> the case where the server has taken the export offline temporarily; if
> the file handle really is bad, then redriving a read or write is
> probably safe, but won't accomplish anything.
> 
> i have a patch that adds support for pathname resolution retry to 2.6
> (now in Trond's NFS_ALL for 2.6.11-rc4) and a pair of patches that
> implement this for RHEL 3.0 that i've sent to steve and al viro for
> review.

I agree, it probably doesn't re-drive on any operation that doesn't walk 
a path, which is in line with what RHEL is doing currently.  I didn't 
mean to imply that solaris retired ESTALE in all occurances of the 
event.  Anywho, Can you point me to your patches?  I'd be interested to 
know how you managed to implement retry on ESTALE without leaking into 
the VFS, which I think you will recall was the big sticking point that 
we were debating here.

Thanks! :)
Neil

-- 
/***************************************************
  *Neil Horman
  *Software Engineer
  *Red Hat, Inc.
  *nhorman@redhat.com
  *gpg keyid: 1024D / 0x92A74FA1
  *http://pgp.mit.edu
  ***************************************************/


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: possible client stale filehandle bug?

[Lever, Charles]

hi neil-

> >>It seems that the Solaris clients never report any such=20
> errors, only the Linux
> >>clients.  However, watching 'snoop' on the Solaris NFS=20
> server, I see that it IS
> >>returning stale file handles to both OSes, but Solaris=20
> clients seem to retry
> >>the request several times; and the Linux clients=20
> immediately pass the error up
> >>to the application.
> >>
> >>Is there some condition that the 2.4 kernel is handling incorrectly?
> >=20
> >=20
> > I do not believe that Solaris redrives ESTALE on read, but=20
> they may do
> > it on open(). Linux does not redrive either case. See the many
> > discussions in the NFS list archives for why.
> >=20
>=20
> Solaris does in fact retry on operations on ESTALE errors,=20
> definately on=20
> open, and I think on read/readdir/stat/etc. as well.  We had some=20
> discussion about tht here recently.

as far as i know Solaris doesn't redrive on read or write, but only
during pathname resolution.  redriving a read or write will only work in
the case where the server has taken the export offline temporarily; if
the file handle really is bad, then redriving a read or write is
probably safe, but won't accomplish anything.

i have a patch that adds support for pathname resolution retry to 2.6
(now in Trond's NFS_ALL for 2.6.11-rc4) and a pair of patches that
implement this for RHEL 3.0 that i've sent to steve and al viro for
review.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs