dentry/inode cache invalidation when using nfsd filesystem on 2.6

dentry/inode cache invalidation when using nfsd filesystem on 2.6

[Jose R. Santos]

Hi,

I have come across a interesting behavior when using NFS server on machines
with large amounts of memory(16/32Gb).  If I have exported filesystems with
lots of data and many directories that were access and had dentry/inode cache
entries cached on the server, the clients took a long time(sometimes minutes)
to mount the NFS exported directories.

While the client was mostly IDLE, the servers rpc.mountd thread was consuming
100% of one CPU on the system.  After some digging, I found that the problem
lies on the new nfsd filesystem code that talks to nfsctl.  Every time I issue
a mount on the client, the server invalidates both inode and dentry caches.

Inode cache get invalidated on sys_nfsservctl(fs/nfsctl.c) on the fput call
which eventually calls invalidate_inodes. Dentry caches get invalidated on
do_open(fs/nfsctl.c) when calling do_kernel_mount which eventually calls
shrink_dcache_sb.

The inode cache invalidation is easy to work around by just mounting the nfsd
to something like /tmp/nfsd, but I haven't been able(except with a very
dangerous and stupid patch) to work around the dentry cache invalidation.

This invalidation seem unnecessary and it cause by the interaction with this
new interface for nfsctl.

Any help would be appreciated. :)

Thanks - JRS


-------------------------------------------------------
This SF.net email is sponsored by OSDN developer relations
Here's your chance to show off your extensive product knowledge
We want to know what you know. Tell us and you have a chance to win $100
http://www.zoomerang.com/survey.zgi?HRPT1X3RYQNC5V4MLNSV3E54
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: dentry/inode cache invalidation when using nfsd filesystem on 2.6

[Neil Brown]

On Tuesday October 21, jrsantos@austin.ibm.com wrote:
> Hi,
> 
> I have come across a interesting behavior when using NFS server on machines
> with large amounts of memory(16/32Gb).  If I have exported filesystems with
> lots of data and many directories that were access and had dentry/inode cache
> entries cached on the server, the clients took a long time(sometimes minutes)
> to mount the NFS exported directories.
> 
> While the client was mostly IDLE, the servers rpc.mountd thread was consuming
> 100% of one CPU on the system.  After some digging, I found that the problem
> lies on the new nfsd filesystem code that talks to nfsctl.  Every time I issue
> a mount on the client, the server invalidates both inode and dentry caches.
> 
> Inode cache get invalidated on sys_nfsservctl(fs/nfsctl.c) on the fput call
> which eventually calls invalidate_inodes. Dentry caches get invalidated on
> do_open(fs/nfsctl.c) when calling do_kernel_mount which eventually calls
> shrink_dcache_sb.
> 
> The inode cache invalidation is easy to work around by just mounting the nfsd
> to something like /tmp/nfsd, but I haven't been able(except with a very
> dangerous and stupid patch) to work around the dentry cache invalidation.
> 
> This invalidation seem unnecessary and it cause by the interaction with this
> new interface for nfsctl.
> 
> Any help would be appreciated. :)
> 
> Thanks - JRS

The problem is NOT that the inode or dentry caches are being flushed.

The problem is that the inode and dentry caches *for*the*(tiny)*nfsd*filesystem*
are being flushed, and this requires iterating through each cache under a
spinlock, and this can take a while.

Basically, "mount" and "unmount" are slow operations.

As you note, "unmount" can be avoided by mounting the nfsd filesystem
somewhere.  I recommend /proc/fs/nfsd.
The "mount" can be avoided by not using the nfsservctl systemcall at all. 
This is achived by mounting nfsd at /proc/fs/nfsd, and running the latest
release of nfs-utils.

*fixing* the problem requires optimising the mount and unmount process,
which is SEP  (Somebody Elses Problem).

NeilBrown


-------------------------------------------------------
This SF.net email is sponsored by: The SF.net Donation Program.
Do you like what SourceForge.net is doing for the Open
Source Community?  Make a contribution, and help us add new
features and functionality. Click here: http://sourceforge.net/donate/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: dentry/inode cache invalidation when using nfsd filesystem on 2.6

[Jose R. Santos]

On 10/22/03 22:57:50, Neil Brown wrote:
> 
> The problem is NOT that the inode or dentry caches are being flushed.
> 
> The problem is that the inode and dentry caches
> *for*the*(tiny)*nfsd*filesystem*
> are being flushed, and this requires iterating through each cache under a
> spinlock, and this can take a while.

I saw some weirdness that led me to believe kernel was invalidating more than  
it should.  I will investigate further to make sure this is not happening.
 
> Basically, "mount" and "unmount" are slow operations.

The only problem is that it seems to get worst as the memory of the system get  
bigger, but I'll agree with you on the SEP. So, I'll just drop it. :)
 
> As you note, "unmount" can be avoided by mounting the nfsd filesystem
> somewhere.  I recommend /proc/fs/nfsd.
> The "mount" can be avoided by not using the nfsservctl systemcall at all.
> This is achived by mounting nfsd at /proc/fs/nfsd, and running the latest
> release of nfs-utils.

>From what you say here, it seems that the old nfsservctl systemcall method is  
deprecated and the new method is the one to use.  If this a fact? Would be  
nice to have this documented in the post-halloween document.  I could drop a  
note to Dave Jones to have this change.

BTW:  What version of nfs-utils introduce this new interface for /proc/fs/ 
nfsd?

> *fixing* the problem requires optimising the mount and unmount process,
> which is SEP  (Somebody Elses Problem).

:)

Thanks - JRS


-------------------------------------------------------
This SF.net email is sponsored by: The SF.net Donation Program.
Do you like what SourceForge.net is doing for the Open
Source Community?  Make a contribution, and help us add new
features and functionality. Click here: http://sourceforge.net/donate/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: dentry/inode cache invalidation when using nfsd filesystem on 2.6

[Neil Brown]

On Thursday October 23, jrsantos@austin.ibm.com wrote:
> On 10/22/03 22:57:50, Neil Brown wrote:
> > 
> > The problem is NOT that the inode or dentry caches are being flushed.
> > 
> > The problem is that the inode and dentry caches
> > *for*the*(tiny)*nfsd*filesystem*
> > are being flushed, and this requires iterating through each cache under a
> > spinlock, and this can take a while.
> 
> I saw some weirdness that led me to believe kernel was invalidating more than  
> it should.  I will investigate further to make sure this is not happening.
>  
> > Basically, "mount" and "unmount" are slow operations.
> 
> The only problem is that it seems to get worst as the memory of the system get  
> bigger, but I'll agree with you on the SEP. So, I'll just drop
> it. :)

The more memory, the bigger the caches, the longer it takes to hunt
through them for inodes/dentries on the target filesystem.

>  
> > As you note, "unmount" can be avoided by mounting the nfsd filesystem
> > somewhere.  I recommend /proc/fs/nfsd.
> > The "mount" can be avoided by not using the nfsservctl systemcall at all.
> > This is achived by mounting nfsd at /proc/fs/nfsd, and running the latest
> > release of nfs-utils.
> 
> >From what you say here, it seems that the old nfsservctl systemcall method is  
> deprecated and the new method is the one to use.  If this a fact? Would be  
> nice to have this documented in the post-halloween document.  I could drop a  
> note to Dave Jones to have this change.

I guess.... I really wanted the syscall and the nfsd filesystem to
both work equally side by side in 2.6, and the syscall would be gone
in 2.8.  So I didn't really want to 'deprecate' it.  Just make it
disappear after everyone had a very good likelyhood of upgrading.

But it might be worth saying something to that effect in the
post-halloween document.



> 
> BTW:  What version of nfs-utils introduce this new interface for /proc/fs/ 
> nfsd?

About 1.0.4 I think, but it was buggy until 1.0.6 which is current.

NeilBrown


-------------------------------------------------------
This SF.net email is sponsored by: The SF.net Donation Program.
Do you like what SourceForge.net is doing for the Open
Source Community?  Make a contribution, and help us add new
features and functionality. Click here: http://sourceforge.net/donate/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

dentry/inode cache invalidation when using nfsd filesystem on 2.6

[Jose R. Santos]

Hi,

I have come across a interesting behavior when using NFS server on machines  
with large amounts of memory(16/32Gb).  If I have exported filesystems with  
lots of data and many directories that were access and had dentry/inode cache  
entries cached on the server, the clients took a long time(sometimes minutes)  
to mount the NFS exported directories.

While the client was mostly IDLE, the servers rpc.mountd thread was consuming  
100% of one CPU on the system.  After some digging, I found that the problem  
lies on the new nfsd filesystem code that talks to nfsctl.  Every time I issue  
a mount on the client, the server invalidates both inode and dentry caches.

Inode cache get invalidated on sys_nfsservctl(fs/nfsctl.c) on the fput call  
which eventually calls invalidate_inodes. Dentry caches get invalidated on  
do_open(fs/nfsctl.c) when calling do_kernel_mount which eventually calls  
shrink_dcache_sb.

The inode cache invalidation is easy to work around by just mounting the nfsd  
to something like /tmp/nfsd, but I haven't been able(except with a very  
dangerous and stupid patch) to work around the dentry cache invalidation.

This invalidation seem unnecessary and it cause by the interaction with this  
new interface for nfsctl.

Any help would be appreciated. :)

Thanks - JRS


-------------------------------------------------------
This SF.net email is sponsored by OSDN developer relations
Here's your chance to show off your extensive product knowledge
We want to know what you know. Tell us and you have a chance to win $100
http://www.zoomerang.com/survey.zgi?HRPT1X3RYQNC5V4MLNSV3E54
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs