nfsdv4 leaks file descriptors

nfsdv4 leaks file descriptors

[Gerhard Heift]

[-- Attachment #1.1: Type: text/plain, Size: 959 bytes --]

Hello,

I have a linux kernel (2.6.38.2, armv5tel) running which exports a
directory with following options:

/data ip/32(rw,async,crossmnt,insecure,root_squash,no_subtree_check,fsid=0)

On the client (Debian unstable, kernel 2.6.38-2-amd64) I mounted the
directory under /media(nfs4import with

# mount -t nfs4 ip:/ /media/nfs4import

After a lot running fdupes on this folder the server kernel starts
throwing messages like this one:

  VFS: file-max limit [number] reached

After a reboot I recognised that the number of used file descriptors
increases during the run of fdupes (see /proc/sys/fs/file-nr). Even
after unmounting the exported filesystem on the client, unexport the
file system on the server and stopping the nfs-server the file
descriptors are still allocated and unmounting the filesystem is not
possible.

After a little bit of research I could create a small program, which
increases the used file descriptors by one.

Regards,
  Gerhard

[-- Attachment #1.2: leak-test.c --]
[-- Type: text/x-csrc, Size: 664 bytes --]

#include <stdio.h>

#define DIR "/media/nfs4import/"

int main(void) {
	FILE *f1;
	FILE *f2;
	char b[1024];
	int i;

	// create file1
	f1 = fopen(DIR "file1.txt", "w");
	fwrite("test\n", 5, 1, f1);
	close(f1);

	// open file1 in read mode
	f1 = fopen(DIR "file1.txt", "r");

	for (i = 0; i < 1; i++) {
		// open file2 in write mode and close it
		f2 = fopen(DIR "file2.txt", "w");
		fclose(f2);
	
		// open file2 in read mode and close it
		// this causes the leak of a file descriptor on the server
		// just watch /proc/sys/fs/file-nr on the server
		f2 = fopen(DIR "file2.txt", "r");
		fclose(f2);
	}
	fclose(f1);

	return 0;
}

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: nfsdv4 leaks file descriptors

[J. Bruce Fields]

On Tue, Apr 12, 2011 at 04:01:50PM +0200, Gerhard Heift wrote:
> Hello,
> 
> I have a linux kernel (2.6.38.2, armv5tel) running which exports a
> directory with following options:

Should be fixed in more recent stable kernel; let me know if not.

--b.

> 
> /data ip/32(rw,async,crossmnt,insecure,root_squash,no_subtree_check,fsid=0)
> 
> On the client (Debian unstable, kernel 2.6.38-2-amd64) I mounted the
> directory under /media(nfs4import with
> 
> # mount -t nfs4 ip:/ /media/nfs4import
> 
> After a lot running fdupes on this folder the server kernel starts
> throwing messages like this one:
> 
>   VFS: file-max limit [number] reached
> 
> After a reboot I recognised that the number of used file descriptors
> increases during the run of fdupes (see /proc/sys/fs/file-nr). Even
> after unmounting the exported filesystem on the client, unexport the
> file system on the server and stopping the nfs-server the file
> descriptors are still allocated and unmounting the filesystem is not
> possible.
> 
> After a little bit of research I could create a small program, which
> increases the used file descriptors by one.
> 
> Regards,
>   Gerhard

> #include <stdio.h>
> 
> #define DIR "/media/nfs4import/"
> 
> int main(void) {
> 	FILE *f1;
> 	FILE *f2;
> 	char b[1024];
> 	int i;
> 
> 	// create file1
> 	f1 = fopen(DIR "file1.txt", "w");
> 	fwrite("test\n", 5, 1, f1);
> 	close(f1);
> 
> 	// open file1 in read mode
> 	f1 = fopen(DIR "file1.txt", "r");
> 
> 	for (i = 0; i < 1; i++) {
> 		// open file2 in write mode and close it
> 		f2 = fopen(DIR "file2.txt", "w");
> 		fclose(f2);
> 	
> 		// open file2 in read mode and close it
> 		// this causes the leak of a file descriptor on the server
> 		// just watch /proc/sys/fs/file-nr on the server
> 		f2 = fopen(DIR "file2.txt", "r");
> 		fclose(f2);
> 	}
> 	fclose(f1);
> 
> 	return 0;
> }

Re: nfsdv4 leaks file descriptors

[Gerhard Heift]

On Wed, Apr 13, 2011 at 05:12:11PM -0400, J. Bruce Fields wrote:
> On Tue, Apr 12, 2011 at 04:01:50PM +0200, Gerhard Heift wrote:
> > Hello,
> > 
> > I have a linux kernel (2.6.38.2, armv5tel) running which exports a
> > directory with following options:
> 
> Should be fixed in more recent stable kernel; let me know if not.

An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and
there the leak happens too.

> --b.

Regards,
  Gerhard

> > /data ip/32(rw,async,crossmnt,insecure,root_squash,no_subtree_check,fsid=0)
> > 
> > On the client (Debian unstable, kernel 2.6.38-2-amd64) I mounted the
> > directory under /media(nfs4import with
> > 
> > # mount -t nfs4 ip:/ /media/nfs4import
> > 
> > After a lot running fdupes on this folder the server kernel starts
> > throwing messages like this one:
> > 
> >   VFS: file-max limit [number] reached
> > 
> > After a reboot I recognised that the number of used file descriptors
> > increases during the run of fdupes (see /proc/sys/fs/file-nr). Even
> > after unmounting the exported filesystem on the client, unexport the
> > file system on the server and stopping the nfs-server the file
> > descriptors are still allocated and unmounting the filesystem is not
> > possible.
> > 
> > After a little bit of research I could create a small program, which
> > increases the used file descriptors by one.
> > 
> > Regards,
> >   Gerhard
> 
> > #include <stdio.h>
> > 
> > #define DIR "/media/nfs4import/"
> > 
> > int main(void) {
> > 	FILE *f1;
> > 	FILE *f2;
> > 	char b[1024];
> > 	int i;
> > 
> > 	// create file1
> > 	f1 = fopen(DIR "file1.txt", "w");
> > 	fwrite("test\n", 5, 1, f1);
> > 	close(f1);
> > 
> > 	// open file1 in read mode
> > 	f1 = fopen(DIR "file1.txt", "r");
> > 
> > 	for (i = 0; i < 1; i++) {
> > 		// open file2 in write mode and close it
> > 		f2 = fopen(DIR "file2.txt", "w");
> > 		fclose(f2);
> > 	
> > 		// open file2 in read mode and close it
> > 		// this causes the leak of a file descriptor on the server
> > 		// just watch /proc/sys/fs/file-nr on the server
> > 		f2 = fopen(DIR "file2.txt", "r");
> > 		fclose(f2);
> > 	}
> > 	fclose(f1);
> > 
> > 	return 0;
> > }
> 
> 
> 

Re: nfsdv4 leaks file descriptors

[J. Bruce Fields]

On Thu, Apr 14, 2011 at 07:48:45AM +0200, Gerhard Heift wrote:
> On Wed, Apr 13, 2011 at 05:12:11PM -0400, J. Bruce Fields wrote:
> > On Tue, Apr 12, 2011 at 04:01:50PM +0200, Gerhard Heift wrote:
> > > Hello,
> > > 
> > > I have a linux kernel (2.6.38.2, armv5tel) running which exports a
> > > directory with following options:
> > 
> > Should be fixed in more recent stable kernel; let me know if not.
> 
> An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and
> there the leak happens too.

Ugh.  OK, investigating.

--b.

> 
> > --b.
> 
> Regards,
>   Gerhard
> 
> > > /data ip/32(rw,async,crossmnt,insecure,root_squash,no_subtree_check,fsid=0)
> > > 
> > > On the client (Debian unstable, kernel 2.6.38-2-amd64) I mounted the
> > > directory under /media(nfs4import with
> > > 
> > > # mount -t nfs4 ip:/ /media/nfs4import
> > > 
> > > After a lot running fdupes on this folder the server kernel starts
> > > throwing messages like this one:
> > > 
> > >   VFS: file-max limit [number] reached
> > > 
> > > After a reboot I recognised that the number of used file descriptors
> > > increases during the run of fdupes (see /proc/sys/fs/file-nr). Even
> > > after unmounting the exported filesystem on the client, unexport the
> > > file system on the server and stopping the nfs-server the file
> > > descriptors are still allocated and unmounting the filesystem is not
> > > possible.
> > > 
> > > After a little bit of research I could create a small program, which
> > > increases the used file descriptors by one.
> > > 
> > > Regards,
> > >   Gerhard
> > 
> > > #include <stdio.h>
> > > 
> > > #define DIR "/media/nfs4import/"
> > > 
> > > int main(void) {
> > > 	FILE *f1;
> > > 	FILE *f2;
> > > 	char b[1024];
> > > 	int i;
> > > 
> > > 	// create file1
> > > 	f1 = fopen(DIR "file1.txt", "w");
> > > 	fwrite("test\n", 5, 1, f1);
> > > 	close(f1);
> > > 
> > > 	// open file1 in read mode
> > > 	f1 = fopen(DIR "file1.txt", "r");
> > > 
> > > 	for (i = 0; i < 1; i++) {
> > > 		// open file2 in write mode and close it
> > > 		f2 = fopen(DIR "file2.txt", "w");
> > > 		fclose(f2);
> > > 	
> > > 		// open file2 in read mode and close it
> > > 		// this causes the leak of a file descriptor on the server
> > > 		// just watch /proc/sys/fs/file-nr on the server
> > > 		f2 = fopen(DIR "file2.txt", "r");
> > > 		fclose(f2);
> > > 	}
> > > 	fclose(f1);
> > > 
> > > 	return 0;
> > > }
> > 
> > 
> > 

Re: nfsdv4 leaks file descriptors

[Gerhard Heift]

[-- Attachment #1: Type: text/plain, Size: 808 bytes --]

On Thu, Apr 14, 2011 at 03:10:06PM -0400, J. Bruce Fields wrote:
> On Thu, Apr 14, 2011 at 07:48:45AM +0200, Gerhard Heift wrote:
> > On Wed, Apr 13, 2011 at 05:12:11PM -0400, J. Bruce Fields wrote:
> > > On Tue, Apr 12, 2011 at 04:01:50PM +0200, Gerhard Heift wrote:
> > > > Hello,
> > > > 
> > > > I have a linux kernel (2.6.38.2, armv5tel) running which exports a
> > > > directory with following options:
> > > 
> > > Should be fixed in more recent stable kernel; let me know if not.
> > 
> > An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and
> > there the leak happens too.
> 
> Ugh.  OK, investigating.

It took me a bit of time, but I located the commit which introduced this
behavior: acfdf5c383b38f7f4dddae41b97c97f1ae058f49

> --b.

Regards,
  Gerhard

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: nfsdv4 leaks file descriptors

[J. Bruce Fields]

On Fri, Apr 15, 2011 at 03:20:30PM +0200, Gerhard Heift wrote:
> On Thu, Apr 14, 2011 at 03:10:06PM -0400, J. Bruce Fields wrote:
> > On Thu, Apr 14, 2011 at 07:48:45AM +0200, Gerhard Heift wrote:
> > > On Wed, Apr 13, 2011 at 05:12:11PM -0400, J. Bruce Fields wrote:
> > > > On Tue, Apr 12, 2011 at 04:01:50PM +0200, Gerhard Heift wrote:
> > > > > Hello,
> > > > > 
> > > > > I have a linux kernel (2.6.38.2, armv5tel) running which exports a
> > > > > directory with following options:
> > > > 
> > > > Should be fixed in more recent stable kernel; let me know if not.
> > > 
> > > An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and
> > > there the leak happens too.
> > 
> > Ugh.  OK, investigating.
> 
> It took me a bit of time, but I located the commit which introduced this
> behavior: acfdf5c383b38f7f4dddae41b97c97f1ae058f49

Doh--probably we just need this (lightly tested).

Thanks for tracking this down.

--b.

commit 21cd8aaf5efb086361cf12f9245c14a3b19fbafd
Author: J. Bruce Fields <bfields@redhat.com>
Date:   Fri Apr 15 18:08:26 2011 -0400

    nfsd4: fix struct file leak on delegation
    
    Introduced by acfdf5c383b38f7f4dddae41b97c97f1ae058f49.
    
    Reported-by: Gerhard Heift <ml-nfs-linux-20110412-ef47@gheift.de>
    Signed-off-by: J. Bruce Fields <bfields@redhat.com>

diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index aa309aa..c79a983 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -258,6 +258,7 @@ static void nfs4_put_deleg_lease(struct nfs4_file *fp)
 	if (atomic_dec_and_test(&fp->fi_delegees)) {
 		vfs_setlease(fp->fi_deleg_file, F_UNLCK, &fp->fi_lease);
 		fp->fi_lease = NULL;
+		fput(fp->fi_deleg_file);
 		fp->fi_deleg_file = NULL;
 	}
 }

Re: nfsdv4 leaks file descriptors

[Gerhard Heift]

[-- Attachment #1: Type: text/plain, Size: 2015 bytes --]

On Fri, Apr 15, 2011 at 06:36:21PM -0400, J. Bruce Fields wrote:
> On Fri, Apr 15, 2011 at 03:20:30PM +0200, Gerhard Heift wrote:
> > On Thu, Apr 14, 2011 at 03:10:06PM -0400, J. Bruce Fields wrote:
> > > On Thu, Apr 14, 2011 at 07:48:45AM +0200, Gerhard Heift wrote:
> > > > On Wed, Apr 13, 2011 at 05:12:11PM -0400, J. Bruce Fields wrote:
> > > > > On Tue, Apr 12, 2011 at 04:01:50PM +0200, Gerhard Heift wrote:
> > > > > > Hello,
> > > > > > 
> > > > > > I have a linux kernel (2.6.38.2, armv5tel) running which exports a
> > > > > > directory with following options:
> > > > > 
> > > > > Should be fixed in more recent stable kernel; let me know if not.
> > > > 
> > > > An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and
> > > > there the leak happens too.
> > > 
> > > Ugh.  OK, investigating.
> > 
> > It took me a bit of time, but I located the commit which introduced this
> > behavior: acfdf5c383b38f7f4dddae41b97c97f1ae058f49
> 
> Doh--probably we just need this (lightly tested).
> 
> Thanks for tracking this down.

This patch works for me. Thanks for fixing it.

> --b.

Regards,
  Gerhard

> commit 21cd8aaf5efb086361cf12f9245c14a3b19fbafd
> Author: J. Bruce Fields <bfields@redhat.com>
> Date:   Fri Apr 15 18:08:26 2011 -0400
> 
>     nfsd4: fix struct file leak on delegation
>     
>     Introduced by acfdf5c383b38f7f4dddae41b97c97f1ae058f49.
>     
>     Reported-by: Gerhard Heift <ml-nfs-linux-20110412-ef47@gheift.de>
>     Signed-off-by: J. Bruce Fields <bfields@redhat.com>
> 
> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> index aa309aa..c79a983 100644
> --- a/fs/nfsd/nfs4state.c
> +++ b/fs/nfsd/nfs4state.c
> @@ -258,6 +258,7 @@ static void nfs4_put_deleg_lease(struct nfs4_file *fp)
>  	if (atomic_dec_and_test(&fp->fi_delegees)) {
>  		vfs_setlease(fp->fi_deleg_file, F_UNLCK, &fp->fi_lease);
>  		fp->fi_lease = NULL;
> +		fput(fp->fi_deleg_file);
>  		fp->fi_deleg_file = NULL;
>  	}
>  }
> 

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: nfsdv4 leaks file descriptors

[J. Bruce Fields]

On Wed, Apr 20, 2011 at 01:08:36PM +0200, Gerhard Heift wrote:
> On Fri, Apr 15, 2011 at 06:36:21PM -0400, J. Bruce Fields wrote:
> > On Fri, Apr 15, 2011 at 03:20:30PM +0200, Gerhard Heift wrote:
> > > On Thu, Apr 14, 2011 at 03:10:06PM -0400, J. Bruce Fields wrote:
> > > > On Thu, Apr 14, 2011 at 07:48:45AM +0200, Gerhard Heift wrote:
> > > > > On Wed, Apr 13, 2011 at 05:12:11PM -0400, J. Bruce Fields wrote:
> > > > > > On Tue, Apr 12, 2011 at 04:01:50PM +0200, Gerhard Heift wrote:
> > > > > > > Hello,
> > > > > > > 
> > > > > > > I have a linux kernel (2.6.38.2, armv5tel) running which exports a
> > > > > > > directory with following options:
> > > > > > 
> > > > > > Should be fixed in more recent stable kernel; let me know if not.
> > > > > 
> > > > > An even more recent stable than 2.6.38.2? I just compiled 2.6.39-rc3 and
> > > > > there the leak happens too.
> > > > 
> > > > Ugh.  OK, investigating.
> > > 
> > > It took me a bit of time, but I located the commit which introduced this
> > > behavior: acfdf5c383b38f7f4dddae41b97c97f1ae058f49
> > 
> > Doh--probably we just need this (lightly tested).
> > 
> > Thanks for tracking this down.
> 
> This patch works for me. Thanks for fixing it.

Thanks for the confirmation.  I'll pass the patch along soon.

--b.