exportfs -u hangs on SMP

exportfs -u hangs on SMP

[Ulrich Gemkow]

Hello,

we have (for security reasons) implemented a "session layer" above the usual
NFS, where paths are exported on a NFS-Server for a host when an user logs in
and are un-exported on logout.

>From this results a rather high rate of calls to "exportfs" and "exportfs -u"
for individual hosts/paths on the server during normal work (not only on
server-startup/shutdown as usual).

Now sometimes we have hangs of "exportfs -u". The process hangs and is not
killable. The server works normal except that it is no longer responding to 
NFS requests. Only a reboot of the server helps.

The "hang rate" is about once per day with 20 users and 5 directories exported
on login for each user. The server has medium to high NFS-load.

The hangs only happen when the server is running a SMP-enabled kernel. When
using a non-SMP-kernel on the same hardware, the hangs are not happening.

Environment is Linux 2.4.21 (vanilla), nfs-utils 1.0.1 (vanilla, xlog-patch
applied), glibc 2.3 (RedHat 9 based) on a 2-processor PIII hardware.

Please tell me if I can be of any help diagnosing and solving the problem.

Thank you and greetings

Ulrich

-- 
|-----------------------------------------------------------------------
| Ulrich Gemkow
| University of Stuttgart, Germany
| Institute of Communication Networks and Computer Engineering (IKR)
|-----------------------------------------------------------------------


-------------------------------------------------------
This SF.net email is sponsored by: VM Ware
With VMware you can run multiple operating systems on a single machine.
WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the
same time. Free trial click here: http://www.vmware.com/wl/offer/345/0
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: exportfs -u hangs on SMP

[Neil Brown]

On Sunday July 20, gemkow@ikr.uni-stuttgart.de wrote:
> 
> Now sometimes we have hangs of "exportfs -u". The process hangs and is not
> killable. The server works normal except that it is no longer responding to 
> NFS requests. Only a reboot of the server helps.
> 
> The "hang rate" is about once per day with 20 users and 5 directories exported
> on login for each user. The server has medium to high NFS-load.
> 
> The hangs only happen when the server is running a SMP-enabled kernel. When
> using a non-SMP-kernel on the same hardware, the hangs are not happening.
> 
> Environment is Linux 2.4.21 (vanilla), nfs-utils 1.0.1 (vanilla, xlog-patch
> applied), glibc 2.3 (RedHat 9 based) on a 2-processor PIII hardware.

Could you try this patch please?
Most of the nfsd is under the BigKernelLock in 2.4, so SMP or not
shouldn't have much effect.  However I just found a bit that isn't
under the lock, and it could be causing your problem.

If you do test it, please let me know how it goes.

Thanks,
NeilBrown


diff ./fs/nfsd/export.c~current~ ./fs/nfsd/export.c
--- ./fs/nfsd/export.c~current~	2003-07-21 09:12:01.000000000 +1000
+++ ./fs/nfsd/export.c	2003-07-21 09:12:16.000000000 +1000
@@ -19,6 +19,7 @@
 #include <linux/stat.h>
 #include <linux/in.h>
 #include <linux/seq_file.h>
+#include <linux/smp_lock.h>
 
 #include <linux/sunrpc/svc.h>
 #include <linux/nfsd/nfsd.h>
@@ -571,6 +572,7 @@ static void *e_start(struct seq_file *m,
 	svc_client *clp;
 	struct list_head *p;
 	
+	lock_kernel();
 	exp_readlock();
 	if (!n--)
 		return (void *)1;
@@ -622,6 +624,7 @@ static void *e_next(struct seq_file *m, 
 static void e_stop(struct seq_file *m, void *p)
 {
 	exp_unlock();
+	unlock_kernel();
 }
 
 struct flags {


-------------------------------------------------------
This SF.net email is sponsored by: VM Ware
With VMware you can run multiple operating systems on a single machine.
WITHOUT REBOOTING! Mix Linux / Windows / Novell virtual machines at the
same time. Free trial click here: http://www.vmware.com/wl/offer/345/0
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: exportfs -u hangs on SMP

[Ulrich Gemkow]

Hello Neil,

I tested the patch and our server survived for almost one week now - much
longer than without the patch. So it seems the patch solves the problem
and would be helpful in mainline too.

Thank you again for your quick help and greetings

Ulrich

> On Sunday July 20, gemkow@ikr.uni-stuttgart.de wrote:
> > Now sometimes we have hangs of "exportfs -u". The process hangs and is
> > not killable. The server works normal except that it is no longer
> > responding to NFS requests. Only a reboot of the server helps.
> >
> > The "hang rate" is about once per day with 20 users and 5 directories
> > exported on login for each user. The server has medium to high NFS-load.
> >
> > The hangs only happen when the server is running a SMP-enabled kernel.
> > When using a non-SMP-kernel on the same hardware, the hangs are not
> > happening.
> >
> > Environment is Linux 2.4.21 (vanilla), nfs-utils 1.0.1 (vanilla,
> > xlog-patch applied), glibc 2.3 (RedHat 9 based) on a 2-processor PIII
> > hardware.
>
> Could you try this patch please?
> Most of the nfsd is under the BigKernelLock in 2.4, so SMP or not
> shouldn't have much effect.  However I just found a bit that isn't
> under the lock, and it could be causing your problem.
>
> If you do test it, please let me know how it goes.
>
> Thanks,
> NeilBrown
>
>
> diff ./fs/nfsd/export.c~current~ ./fs/nfsd/export.c
> --- ./fs/nfsd/export.c~current~	2003-07-21 09:12:01.000000000 +1000
> +++ ./fs/nfsd/export.c	2003-07-21 09:12:16.000000000 +1000
> @@ -19,6 +19,7 @@
>  #include <linux/stat.h>
>  #include <linux/in.h>
>  #include <linux/seq_file.h>
> +#include <linux/smp_lock.h>
>
>  #include <linux/sunrpc/svc.h>
>  #include <linux/nfsd/nfsd.h>
> @@ -571,6 +572,7 @@ static void *e_start(struct seq_file *m,
>  	svc_client *clp;
>  	struct list_head *p;
>
> +	lock_kernel();
>  	exp_readlock();
>  	if (!n--)
>  		return (void *)1;
> @@ -622,6 +624,7 @@ static void *e_next(struct seq_file *m,
>  static void e_stop(struct seq_file *m, void *p)
>  {
>  	exp_unlock();
> +	unlock_kernel();
>  }
>
>  struct flags {

-- 
|-----------------------------------------------------------------------
| Ulrich Gemkow
| University of Stuttgart
| Institute of Communication Networks and Computer Engineering (IKR)
|-----------------------------------------------------------------------



-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: exportfs -u hangs on SMP

[Neil Brown]

On Saturday July 26, gemkow@ikr.uni-stuttgart.de wrote:
> Hello Neil,
> 
> I tested the patch and our server survived for almost one week now - much
> longer than without the patch. So it seems the patch solves the problem
> and would be helpful in mainline too.

Thanks for letting me know.  The patch will be in 2.4.22

NeilBrown


-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: exportfs -u hangs on SMP

[Donavan Pantke]

	WE've been seeing similar wierd locking problems (nfsd gets in a DW state), 
and this may or may not be linked. When would this section of code get 
executed? Only by exportfs, or could there be other times this may occur?

	Thanks!
	Donavan Pantke
	Network Engineer
	Appriss, Inc.


On Monday 28 July 2003 22:38, Neil Brown wrote:
> On Saturday July 26, gemkow@ikr.uni-stuttgart.de wrote:
> > Hello Neil,
> >
> > I tested the patch and our server survived for almost one week now - much
> > longer than without the patch. So it seems the patch solves the problem
> > and would be helpful in mainline too.
>
> Thanks for letting me know.  The patch will be in 2.4.22
>
> NeilBrown
>
>
> -------------------------------------------------------
> This SF.Net email sponsored by: Free pre-built ASP.NET sites including
> Data Reports, E-commerce, Portals, and Forums are available now.
> Download today and enter to win an XBOX or Visual Studio .NET.
> http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs



-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: exportfs -u hangs on SMP

[Neil Brown]

On Tuesday July 29, avatar@dcr.net wrote:
> 	WE've been seeing similar wierd locking problems (nfsd gets in a DW state), 
> and this may or may not be linked. When would this section of code get 
> executed? Only by exportfs, or could there be other times this may occur?

The section of code gets run whenever /proc/fs/nfs/exports is
accessed.
The only standard program that accesses this that I know of if
exportfs.   Ofcourse a script could equally easily access it.

NeilBrown


-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs