2.6.38: Quota over NFS4

2.6.38: Quota over NFS4

[Adam Lackorzynski]

Hello,

I'm seeing a problem with quotas in a system where the server running
2.6.38 exports an XFS filesystem via NFS4 to a client. The client kernel
version does not seem to play a role, checked with 2.6.38, 2.6.37 and
2.6.36. The following script and output show the problem:

#! /bin/sh

quota | grep home
du
cp /bin/ls x1
du
cat x1 > /dev/null
rm x1
du
quota | grep home

Output:

   homes:/home/ 8194720  9072000 9174400          403670  500000  550000        
0       .
96      .
0       .
   homes:/home/ 8194816  9072000 9174400          403671  500000  550000        


As can be seen the 96 kb are still accounted on the quota of the user.
Removing the 'cat' command from the script makes the quota be ok again
(original value). Also mounting via nfs3 does not exhibit it, same for running
the script on the nfs-server directly.

Client mount options are:
homes:/home/ /home nfs4 rw,nosuid,nodev,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=1.2.3.3,minorversion=0,local_lock=none,addr=1.2.3.4 0 0

Server mount options are:
/dev/mapper/vg-homefs /home xfs rw,nosuid,nodev,relatime,attr2,usrquota 0 0




Thanks,
Adam
-- 
Adam                 adam@os.inf.tu-dresden.de
  Lackorzynski         http://os.inf.tu-dresden.de/~adam/

Re: 2.6.38: Quota over NFS4

[J. Bruce Fields]

On Thu, Mar 17, 2011 at 02:32:47PM +0100, Adam Lackorzynski wrote:
> Hello,
> 
> I'm seeing a problem with quotas in a system where the server running
> 2.6.38 exports an XFS filesystem via NFS4 to a client. The client kernel
> version does not seem to play a role, checked with 2.6.38, 2.6.37 and
> 2.6.36. The following script and output show the problem:
> 
> #! /bin/sh
> 
> quota | grep home
> du
> cp /bin/ls x1
> du
> cat x1 > /dev/null
> rm x1
> du
> quota | grep home
> 
> Output:
> 
>    homes:/home/ 8194720  9072000 9174400          403670  500000  550000        
> 0       .
> 96      .
> 0       .
>    homes:/home/ 8194816  9072000 9174400          403671  500000  550000        
> 
> 
> As can be seen the 96 kb are still accounted on the quota of the user.
> Removing the 'cat' command from the script makes the quota be ok again
> (original value). Also mounting via nfs3 does not exhibit it, same for running
> the script on the nfs-server directly.

Does "df" show the same problem?  And does unmounting/remounting on the
client clear the problem?  (Or that, in combination with stopping the
server, unmounting the xfs export, remounting it, and restarting?)

Was there an earlier server version that didn't exhibit this problem?

--b.

> 
> Client mount options are:
> homes:/home/ /home nfs4 rw,nosuid,nodev,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=1.2.3.3,minorversion=0,local_lock=none,addr=1.2.3.4 0 0
> 
> Server mount options are:
> /dev/mapper/vg-homefs /home xfs rw,nosuid,nodev,relatime,attr2,usrquota 0 0
> 
> 
> 
> 
> Thanks,
> Adam
> -- 
> Adam                 adam@os.inf.tu-dresden.de
>   Lackorzynski         http://os.inf.tu-dresden.de/~adam/
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 2.6.38: Quota over NFS4

[Adam Lackorzynski]

On Thu Mar 17, 2011 at 13:38:05 -0400, J. Bruce Fields wrote:
> On Thu, Mar 17, 2011 at 02:32:47PM +0100, Adam Lackorzynski wrote:
> > Hello,
> > 
> > I'm seeing a problem with quotas in a system where the server running
> > 2.6.38 exports an XFS filesystem via NFS4 to a client. The client kernel
> > version does not seem to play a role, checked with 2.6.38, 2.6.37 and
> > 2.6.36. The following script and output show the problem:
> > 
> > #! /bin/sh
> > 
> > quota | grep home
> > du
> > cp /bin/ls x1
> > du
> > cat x1 > /dev/null
> > rm x1
> > du
> > quota | grep home
> > 
> > Output:
> > 
> >    homes:/home/ 8194720  9072000 9174400          403670  500000  550000        
> > 0       .
> > 96      .
> > 0       .
> >    homes:/home/ 8194816  9072000 9174400          403671  500000  550000        
> > 
> > 
> > As can be seen the 96 kb are still accounted on the quota of the user.
> > Removing the 'cat' command from the script makes the quota be ok again
> > (original value). Also mounting via nfs3 does not exhibit it, same for running
> > the script on the nfs-server directly.
> 
> Does "df" show the same problem?

With '/bin/ls' it does not change at all, so I took a bigger binary
which yields to:

   homes:/home/ 8203780  9072000 9174400          403688  500000  550000        
0       .
Filesystem           1K-blocks      Used Available Use% Mounted on
  homes:/home        513671168 335251456 178419712  66% /tmp/xx
4592    .
Filesystem           1K-blocks      Used Available Use% Mounted on
  homes:/home        513671168 335256576 178414592  66% /tmp/xx
0       .
Filesystem           1K-blocks      Used Available Use% Mounted on
  homes:/home        513671168 335256576 178414592  66% /tmp/xx
   homes:/home/ 8208372  9072000 9174400          403689  500000  550000        

So yes, it seems to be there as well.

> And does unmounting/remounting on the
> client clear the problem?

No.

> (Or that, in combination with stopping the
> server, unmounting the xfs export, remounting it, and restarting?)

I rebooted once, got a recovery and then the quotas were ok again (and
supposedly the used blocks as well). I assume a unmount/mount would show
the same behaviour but requires a bit of preparation to try out.

> Was there an earlier server version that didn't exhibit this problem?

Had 2.6.37 on the server before and that was fine regarding this.

> > Client mount options are:
> > homes:/home/ /home nfs4 rw,nosuid,nodev,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=1.2.3.3,minorversion=0,local_lock=none,addr=1.2.3.4 0 0
> > 
> > Server mount options are:
> > /dev/mapper/vg-homefs /home xfs rw,nosuid,nodev,relatime,attr2,usrquota 0 0

Adam
-- 
Adam                 adam@os.inf.tu-dresden.de
  Lackorzynski         http://os.inf.tu-dresden.de/~adam/

Re: 2.6.38: Quota over NFS4

[J. Bruce Fields]

On Thu, Mar 17, 2011 at 10:33:03PM +0100, Adam Lackorzynski wrote:
> 
> On Thu Mar 17, 2011 at 13:38:05 -0400, J. Bruce Fields wrote:
> > On Thu, Mar 17, 2011 at 02:32:47PM +0100, Adam Lackorzynski wrote:
> > > Hello,
> > > 
> > > I'm seeing a problem with quotas in a system where the server running
> > > 2.6.38 exports an XFS filesystem via NFS4 to a client. The client kernel
> > > version does not seem to play a role, checked with 2.6.38, 2.6.37 and
> > > 2.6.36. The following script and output show the problem:
> > > 
> > > #! /bin/sh
> > > 
> > > quota | grep home
> > > du
> > > cp /bin/ls x1
> > > du
> > > cat x1 > /dev/null
> > > rm x1
> > > du
> > > quota | grep home
> > > 
> > > Output:
> > > 
> > >    homes:/home/ 8194720  9072000 9174400          403670  500000  550000        
> > > 0       .
> > > 96      .
> > > 0       .
> > >    homes:/home/ 8194816  9072000 9174400          403671  500000  550000        
> > > 
> > > 
> > > As can be seen the 96 kb are still accounted on the quota of the user.
> > > Removing the 'cat' command from the script makes the quota be ok again
> > > (original value). Also mounting via nfs3 does not exhibit it, same for running
> > > the script on the nfs-server directly.
> > 
> > Does "df" show the same problem?
> 
> With '/bin/ls' it does not change at all, so I took a bigger binary
> which yields to:
> 
>    homes:/home/ 8203780  9072000 9174400          403688  500000  550000        
> 0       .
> Filesystem           1K-blocks      Used Available Use% Mounted on
>   homes:/home        513671168 335251456 178419712  66% /tmp/xx
> 4592    .
> Filesystem           1K-blocks      Used Available Use% Mounted on
>   homes:/home        513671168 335256576 178414592  66% /tmp/xx
> 0       .
> Filesystem           1K-blocks      Used Available Use% Mounted on
>   homes:/home        513671168 335256576 178414592  66% /tmp/xx
>    homes:/home/ 8208372  9072000 9174400          403689  500000  550000        
> 
> So yes, it seems to be there as well.

It might be easier to see with "df -i" (assuming we're leaking an
inode).

> > And does unmounting/remounting on the
> > client clear the problem?
> 
> No.
> 
> > (Or that, in combination with stopping the
> > server, unmounting the xfs export, remounting it, and restarting?)
> 
> I rebooted once, got a recovery and then the quotas were ok again (and
> supposedly the used blocks as well). I assume a unmount/mount would show
> the same behaviour but requires a bit of preparation to try out.
> 
> > Was there an earlier server version that didn't exhibit this problem?
> 
> Had 2.6.37 on the server before and that was fine regarding this.

OK, thanks.  Sounds like a server bug, but I'm not managing to reproduce
it here yet....

--b.

Re: 2.6.38: Quota over NFS4

[Adam Lackorzynski]

On Thu Mar 17, 2011 at 18:27:32 -0400, J. Bruce Fields wrote:
> On Thu, Mar 17, 2011 at 10:33:03PM +0100, Adam Lackorzynski wrote:
> > 
> > On Thu Mar 17, 2011 at 13:38:05 -0400, J. Bruce Fields wrote:
> > > On Thu, Mar 17, 2011 at 02:32:47PM +0100, Adam Lackorzynski wrote:
> > > > Hello,
> > > > 
> > > > I'm seeing a problem with quotas in a system where the server running
> > > > 2.6.38 exports an XFS filesystem via NFS4 to a client. The client kernel
> > > > version does not seem to play a role, checked with 2.6.38, 2.6.37 and
> > > > 2.6.36. The following script and output show the problem:
> > > > 
> > > > #! /bin/sh
> > > > 
> > > > quota | grep home
> > > > du
> > > > cp /bin/ls x1
> > > > du
> > > > cat x1 > /dev/null
> > > > rm x1
> > > > du
> > > > quota | grep home
> > > > 
> > > > Output:
> > > > 
> > > >    homes:/home/ 8194720  9072000 9174400          403670  500000  550000        
> > > > 0       .
> > > > 96      .
> > > > 0       .
> > > >    homes:/home/ 8194816  9072000 9174400          403671  500000  550000        
> > > > 
> > > > 
> > > > As can be seen the 96 kb are still accounted on the quota of the user.
> > > > Removing the 'cat' command from the script makes the quota be ok again
> > > > (original value). Also mounting via nfs3 does not exhibit it, same for running
> > > > the script on the nfs-server directly.
> > > 
> > > Does "df" show the same problem?
> > 
> > With '/bin/ls' it does not change at all, so I took a bigger binary
> > which yields to:
> > 
> >    homes:/home/ 8203780  9072000 9174400          403688  500000  550000        
> > 0       .
> > Filesystem           1K-blocks      Used Available Use% Mounted on
> >   homes:/home        513671168 335251456 178419712  66% /tmp/xx
> > 4592    .
> > Filesystem           1K-blocks      Used Available Use% Mounted on
> >   homes:/home        513671168 335256576 178414592  66% /tmp/xx
> > 0       .
> > Filesystem           1K-blocks      Used Available Use% Mounted on
> >   homes:/home        513671168 335256576 178414592  66% /tmp/xx
> >    homes:/home/ 8208372  9072000 9174400          403689  500000  550000        
> > 
> > So yes, it seems to be there as well.
> 
> It might be easier to see with "df -i" (assuming we're leaking an
> inode).

Result is as expected, inode goes one up and not down again.
 
> > > And does unmounting/remounting on the
> > > client clear the problem?
> > 
> > No.
> > 
> > > (Or that, in combination with stopping the
> > > server, unmounting the xfs export, remounting it, and restarting?)
> > 
> > I rebooted once, got a recovery and then the quotas were ok again (and
> > supposedly the used blocks as well). I assume a unmount/mount would show
> > the same behaviour but requires a bit of preparation to try out.
> > 
> > > Was there an earlier server version that didn't exhibit this problem?
> > 
> > Had 2.6.37 on the server before and that was fine regarding this.
> 
> OK, thanks.  Sounds like a server bug, but I'm not managing to reproduce
> it here yet....


Adam
-- 
Adam                 adam@os.inf.tu-dresden.de
  Lackorzynski         http://os.inf.tu-dresden.de/~adam/

Re: 2.6.38: Quota over NFS4

[J. Bruce Fields]

On Thu, Mar 17, 2011 at 11:59:08PM +0100, Adam Lackorzynski wrote:
> 
> On Thu Mar 17, 2011 at 18:27:32 -0400, J. Bruce Fields wrote:
> > On Thu, Mar 17, 2011 at 10:33:03PM +0100, Adam Lackorzynski wrote:
> > > 
> > > On Thu Mar 17, 2011 at 13:38:05 -0400, J. Bruce Fields wrote:
> > > > On Thu, Mar 17, 2011 at 02:32:47PM +0100, Adam Lackorzynski wrote:
> > > > > Hello,
> > > > > 
> > > > > I'm seeing a problem with quotas in a system where the server running
> > > > > 2.6.38 exports an XFS filesystem via NFS4 to a client. The client kernel
> > > > > version does not seem to play a role, checked with 2.6.38, 2.6.37 and
> > > > > 2.6.36. The following script and output show the problem:
> > > > > 
> > > > > #! /bin/sh
> > > > > 
> > > > > quota | grep home
> > > > > du
> > > > > cp /bin/ls x1
> > > > > du
> > > > > cat x1 > /dev/null
> > > > > rm x1
> > > > > du
> > > > > quota | grep home
> > > > > 
> > > > > Output:
> > > > > 
> > > > >    homes:/home/ 8194720  9072000 9174400          403670  500000  550000        
> > > > > 0       .
> > > > > 96      .
> > > > > 0       .
> > > > >    homes:/home/ 8194816  9072000 9174400          403671  500000  550000        
> > > > > 
> > > > > 
> > > > > As can be seen the 96 kb are still accounted on the quota of the user.
> > > > > Removing the 'cat' command from the script makes the quota be ok again
> > > > > (original value). Also mounting via nfs3 does not exhibit it, same for running
> > > > > the script on the nfs-server directly.
> > > > 
> > > > Does "df" show the same problem?
> > > 
> > > With '/bin/ls' it does not change at all, so I took a bigger binary
> > > which yields to:
> > > 
> > >    homes:/home/ 8203780  9072000 9174400          403688  500000  550000        
> > > 0       .
> > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > >   homes:/home        513671168 335251456 178419712  66% /tmp/xx
> > > 4592    .
> > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > >   homes:/home        513671168 335256576 178414592  66% /tmp/xx
> > > 0       .
> > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > >   homes:/home        513671168 335256576 178414592  66% /tmp/xx
> > >    homes:/home/ 8208372  9072000 9174400          403689  500000  550000        
> > > 
> > > So yes, it seems to be there as well.
> > 
> > It might be easier to see with "df -i" (assuming we're leaking an
> > inode).
> 
> Result is as expected, inode goes one up and not down again.

Is this something special about binaries?  If you copy something other
than a binary, do you not see the bug?

--b.

Re: 2.6.38: Quota over NFS4

[Adam Lackorzynski]

On Thu Mar 17, 2011 at 19:03:15 -0400, J. Bruce Fields wrote:
> On Thu, Mar 17, 2011 at 11:59:08PM +0100, Adam Lackorzynski wrote:
> > 
> > On Thu Mar 17, 2011 at 18:27:32 -0400, J. Bruce Fields wrote:
> > > On Thu, Mar 17, 2011 at 10:33:03PM +0100, Adam Lackorzynski wrote:
> > > > 
> > > > On Thu Mar 17, 2011 at 13:38:05 -0400, J. Bruce Fields wrote:
> > > > > On Thu, Mar 17, 2011 at 02:32:47PM +0100, Adam Lackorzynski wrote:
> > > > > > Hello,
> > > > > > 
> > > > > > I'm seeing a problem with quotas in a system where the server running
> > > > > > 2.6.38 exports an XFS filesystem via NFS4 to a client. The client kernel
> > > > > > version does not seem to play a role, checked with 2.6.38, 2.6.37 and
> > > > > > 2.6.36. The following script and output show the problem:
> > > > > > 
> > > > > > #! /bin/sh
> > > > > > 
> > > > > > quota | grep home
> > > > > > du
> > > > > > cp /bin/ls x1
> > > > > > du
> > > > > > cat x1 > /dev/null
> > > > > > rm x1
> > > > > > du
> > > > > > quota | grep home
> > > > > > 
> > > > > > Output:
> > > > > > 
> > > > > >    homes:/home/ 8194720  9072000 9174400          403670  500000  550000        
> > > > > > 0       .
> > > > > > 96      .
> > > > > > 0       .
> > > > > >    homes:/home/ 8194816  9072000 9174400          403671  500000  550000        
> > > > > > 
> > > > > > 
> > > > > > As can be seen the 96 kb are still accounted on the quota of the user.
> > > > > > Removing the 'cat' command from the script makes the quota be ok again
> > > > > > (original value). Also mounting via nfs3 does not exhibit it, same for running
> > > > > > the script on the nfs-server directly.
> > > > > 
> > > > > Does "df" show the same problem?
> > > > 
> > > > With '/bin/ls' it does not change at all, so I took a bigger binary
> > > > which yields to:
> > > > 
> > > >    homes:/home/ 8203780  9072000 9174400          403688  500000  550000        
> > > > 0       .
> > > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > > >   homes:/home        513671168 335251456 178419712  66% /tmp/xx
> > > > 4592    .
> > > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > > >   homes:/home        513671168 335256576 178414592  66% /tmp/xx
> > > > 0       .
> > > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > > >   homes:/home        513671168 335256576 178414592  66% /tmp/xx
> > > >    homes:/home/ 8208372  9072000 9174400          403689  500000  550000        
> > > > 
> > > > So yes, it seems to be there as well.
> > > 
> > > It might be easier to see with "df -i" (assuming we're leaking an
> > > inode).
> > 
> > Result is as expected, inode goes one up and not down again.
> 
> Is this something special about binaries?  If you copy something other
> than a binary, do you not see the bug?

No change when using a plain text file instead of a binary.


Adam
-- 
Adam                 adam@os.inf.tu-dresden.de
  Lackorzynski         http://os.inf.tu-dresden.de/~adam/

Re: 2.6.38: Quota over NFS4

[Maciej Rutecki]

On czwartek, 17 marca 2011 o 14:32:47 Adam Lackorzynski wrote:
> Hello,
> 
> I'm seeing a problem with quotas in a system where the server running
> 2.6.38 exports an XFS filesystem via NFS4 to a client. The client kernel
> version does not seem to play a role, checked with 2.6.38, 2.6.37 and
> 2.6.36. The following script and output show the problem:
> 
> #! /bin/sh
> 
> quota | grep home
> du
> cp /bin/ls x1
> du
> cat x1 > /dev/null
> rm x1
> du
> quota | grep home
> 
> Output:
> 
>    homes:/home/ 8194720  9072000 9174400          403670  500000  550000
> 0       .
> 96      .
> 0       .
>    homes:/home/ 8194816  9072000 9174400          403671  500000  550000
> 
> 
> As can be seen the 96 kb are still accounted on the quota of the user.
> Removing the 'cat' command from the script makes the quota be ok again
> (original value). Also mounting via nfs3 does not exhibit it, same for
> running the script on the nfs-server directly.
> 
> Client mount options are:
> homes:/home/ /home nfs4
> rw,nosuid,nodev,relatime,vers=4,rsize=1048576,wsize=1048576,namlen=255,har
> d,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=1.2.3.3,minorvers
> ion=0,local_lock=none,addr=1.2.3.4 0 0
> 
> Server mount options are:
> /dev/mapper/vg-homefs /home xfs rw,nosuid,nodev,relatime,attr2,usrquota 0 0
> 
>
I created a Bugzilla entry at 
https://bugzilla.kernel.org/show_bug.cgi?id=31532
for your bug report, please add your address to the CC list in there, thanks!

-- 
Maciej Rutecki
http://www.maciek.unixy.pl

Re: 2.6.38: Quota over NFS4

[J. Bruce Fields]

On Wed, Mar 23, 2011 at 06:40:52PM +0100, Adam Lackorzynski wrote:
> Hi,
> 
> On Wed Mar 23, 2011 at 11:03:28 -0400, J. Bruce Fields wrote:
> > (How'd linux-nfs get dropped off the cc:?)
> 
> I must have accidently done single reply only, sorry.

OK, assuming you won't mind my re-adding it.

> 
> > On Tue, Mar 22, 2011 at 11:13:06PM +0100, Adam Lackorzynski wrote:
> > > 
> > > On Mon Mar 21, 2011 at 18:23:01 -0400, J. Bruce Fields wrote:
> > > > 
> > > > Oh, right, you'd want to unmount on the sender before copying.
> > > > 
> > > > You *may* still get some kind of usable result if you fsck the fs you
> > > > get on the receiver, but who knows.
> > > 
> > > Ok, good news. After some fight I got it mounted in the VM. Reproducible
> > > with the original 2.6.38 kernel (as released) and also reproducible with
> > > latest git as of today. What now?
> > 
> > OK, so now you can reproduce the problem on one xfs filesystem, but not
> > the other.
> > 
> > Can we find any differences between the two filesystems?  Was one
> > created with different features, for example?
> >
> > Might also be worth asking xfs developers for ideas.
> 
> Bisecting shows this commit:

That's an odd place for it to end up.  You're positive the problem is
reliably reproduceable after this commit and not in the commit
immediately previous?

--b.

> 5d926e8c2f46dc09f4ddde86644a5f1d0726a470 is the first bad commit
> commit 5d926e8c2f46dc09f4ddde86644a5f1d0726a470
> Author: J. Bruce Fields <bfields@redhat.com>
> Date:   Mon Feb 7 16:53:46 2011 -0500
> 
>     nfsd4: modify fi_delegations under recall_lock
>     
>     Modify fi_delegations only under the recall_lock, allowing us to use
>     that list on lease breaks.
>     
>     Also some trivial cleanup to simplify later changes.
>     
>     Signed-off-by: J. Bruce Fields <bfields@redhat.com>
> 
> :040000 040000 fa71771d862119d1918c40a0ea25815f2f8b16bf 5bf94b91c7c696586f0aaf8b5df1cd9c4aa70ac0 M      fs
> 
> 
> Adam
> -- 
> Adam                 adam@os.inf.tu-dresden.de
>   Lackorzynski         http://os.inf.tu-dresden.de/~adam/

Re: 2.6.38: Quota over NFS4

[Adam Lackorzynski]

On Wed Mar 23, 2011 at 15:06:32 -0400, J. Bruce Fields wrote:
> On Wed, Mar 23, 2011 at 06:40:52PM +0100, Adam Lackorzynski wrote:
> > > On Tue, Mar 22, 2011 at 11:13:06PM +0100, Adam Lackorzynski wrote:
> > > > 
> > > > On Mon Mar 21, 2011 at 18:23:01 -0400, J. Bruce Fields wrote:
> > > > > 
> > > > > Oh, right, you'd want to unmount on the sender before copying.
> > > > > 
> > > > > You *may* still get some kind of usable result if you fsck the fs you
> > > > > get on the receiver, but who knows.
> > > > 
> > > > Ok, good news. After some fight I got it mounted in the VM. Reproducible
> > > > with the original 2.6.38 kernel (as released) and also reproducible with
> > > > latest git as of today. What now?
> > > 
> > > OK, so now you can reproduce the problem on one xfs filesystem, but not
> > > the other.
> > > 
> > > Can we find any differences between the two filesystems?  Was one
> > > created with different features, for example?
> > >
> > > Might also be worth asking xfs developers for ideas.
> > 
> > Bisecting shows this commit:
> 
> That's an odd place for it to end up.  You're positive the problem is
> reliably reproduceable after this commit and not in the commit
> immediately previous?

I probably mistyped something and thus bisected again:

acfdf5c383b38f7f4dddae41b97c97f1ae058f49 is the first bad commit
commit acfdf5c383b38f7f4dddae41b97c97f1ae058f49
Author: J. Bruce Fields <bfields@redhat.com>
Date:   Mon Jan 31 19:20:39 2011 -0500

    nfsd4: acquire only one lease per file
    
    Instead of acquiring one lease each time another client opens a file,
    nfsd can acquire just one lease to represent all of them, and reference
    count it to determine when to release it.
    
    This fixes a regression introduced by
    c45821d263a8a5109d69a9e8942b8d65bcd5f31a "locks: eliminate fl_mylease
    callback": after that patch, only the struct file * is used to determine
    who owns a given lease.  But since we recently converted the server to
    share a single struct file per open, if we acquire multiple leases on
    the same file from nfsd, it then becomes impossible on unlocking a lease
    to determine which of those leases (all of whom share the same struct
    file *) we meant to remove.
    
    Thanks to Takashi Iwai <tiwai@suse.de> for catching a bug in a previous
    version of this patch.
    
    Tested-by: Takashi Iwai <tiwai@suse.de>
    Signed-off-by: J. Bruce Fields <bfields@redhat.com>

:040000 040000 5bf94b91c7c696586f0aaf8b5df1cd9c4aa70ac0 4ee1f39f455ee7f648fccedc1ddba4da6a94cec7 M      fs

 


Adam
-- 
Adam                 adam@os.inf.tu-dresden.de
  Lackorzynski         http://os.inf.tu-dresden.de/~adam/

Re: 2.6.38: Quota over NFS4

[Christoph Hellwig]

On Wed, Mar 23, 2011 at 11:30:17PM +0100, Adam Lackorzynski wrote:
> I probably mistyped something and thus bisected again:

maybe this is just a symptom.  When was the nfs4 code converted
to just use a single open struct file, and what are the lifetime
rules for it?  It would easily be that a long alive file might
make XFS keep persistent preallocations longer or similar issues.

Adam, can you create a loop filesystem with ext3 or something else
on it and see if the problem is XFS-specific?

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2.6.38: Quota over NFS4

[Christoph Hellwig]

On Wed, Mar 23, 2011 at 11:30:17PM +0100, Adam Lackorzynski wrote:
> I probably mistyped something and thus bisected again:

maybe this is just a symptom.  When was the nfs4 code converted
to just use a single open struct file, and what are the lifetime
rules for it?  It would easily be that a long alive file might
make XFS keep persistent preallocations longer or similar issues.

Adam, can you create a loop filesystem with ext3 or something else
on it and see if the problem is XFS-specific?

Re: 2.6.38: Quota over NFS4

[J. Bruce Fields]

On Thu, Mar 24, 2011 at 01:17:05PM -0400, Christoph Hellwig wrote:
> On Wed, Mar 23, 2011 at 11:30:17PM +0100, Adam Lackorzynski wrote:
> > I probably mistyped something and thus bisected again:
> 
> maybe this is just a symptom.  When was the nfs4 code converted
> to just use a single open struct file, and what are the lifetime
> rules for it?  It would easily be that a long alive file might
> make XFS keep persistent preallocations longer or similar issues.

Definitely, the commit he's bisect to there seems much more likely to be
a culprit.  I'll take a closer look....

> Adam, can you create a loop filesystem with ext3 or something else
> on it and see if the problem is XFS-specific?

Originally it sounded like he was able to reproduce this only on one
specific filesystem so I wondered whether there was something particular
to that filesystem.

--b.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2.6.38: Quota over NFS4

[J. Bruce Fields]

On Thu, Mar 24, 2011 at 01:17:05PM -0400, Christoph Hellwig wrote:
> On Wed, Mar 23, 2011 at 11:30:17PM +0100, Adam Lackorzynski wrote:
> > I probably mistyped something and thus bisected again:
> 
> maybe this is just a symptom.  When was the nfs4 code converted
> to just use a single open struct file, and what are the lifetime
> rules for it?  It would easily be that a long alive file might
> make XFS keep persistent preallocations longer or similar issues.

Definitely, the commit he's bisect to there seems much more likely to be
a culprit.  I'll take a closer look....

> Adam, can you create a loop filesystem with ext3 or something else
> on it and see if the problem is XFS-specific?

Originally it sounded like he was able to reproduce this only on one
specific filesystem so I wondered whether there was something particular
to that filesystem.

--b.

Re: 2.6.38: Quota over NFS4

[Adam Lackorzynski]

On Thu Mar 24, 2011 at 13:51:06 -0400, J. Bruce Fields wrote:
> On Thu, Mar 24, 2011 at 01:17:05PM -0400, Christoph Hellwig wrote:
> > On Wed, Mar 23, 2011 at 11:30:17PM +0100, Adam Lackorzynski wrote:
> > > I probably mistyped something and thus bisected again:
> > 
> > maybe this is just a symptom.  When was the nfs4 code converted
> > to just use a single open struct file, and what are the lifetime
> > rules for it?  It would easily be that a long alive file might
> > make XFS keep persistent preallocations longer or similar issues.
> 
> Definitely, the commit he's bisect to there seems much more likely to be
> a culprit.  I'll take a closer look....
> 
> > Adam, can you create a loop filesystem with ext3 or something else
> > on it and see if the problem is XFS-specific?
> 
> Originally it sounded like he was able to reproduce this only on one
> specific filesystem so I wondered whether there was something particular
> to that filesystem.

I just found out it happens also with a newly created xfs and also with
ext3 and ext4. The reason that I did not see that initially is that the
script I'm using needs to be on the same fs which wasn't the case when I
initially tried it with a new fs. The sequence I'm using is:

# mount x.x.x:/home /tmp/st
# su - foo
$ cd /tmp/st/foo/x
$ ../test-script
foo@host:/tmp/st/foo/x$ ../test-script 
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
x.x.x:/home    39321600      15 39321585    1% /tmp/st
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
x.x.x:/home    39321600      16 39321584    1% /tmp/st
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
x.x.x:/home    39321600      16 39321584    1% /tmp/st



test-script:
#! /bin/sh

df -i .
cp /bin/ls x1
df -i .
cat x1 > /dev/null
rm x1
df -i .






Adam
-- 
Adam                 adam@os.inf.tu-dresden.de
  Lackorzynski         http://os.inf.tu-dresden.de/~adam/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2.6.38: Quota over NFS4

[Adam Lackorzynski]

On Thu Mar 24, 2011 at 13:51:06 -0400, J. Bruce Fields wrote:
> On Thu, Mar 24, 2011 at 01:17:05PM -0400, Christoph Hellwig wrote:
> > On Wed, Mar 23, 2011 at 11:30:17PM +0100, Adam Lackorzynski wrote:
> > > I probably mistyped something and thus bisected again:
> > 
> > maybe this is just a symptom.  When was the nfs4 code converted
> > to just use a single open struct file, and what are the lifetime
> > rules for it?  It would easily be that a long alive file might
> > make XFS keep persistent preallocations longer or similar issues.
> 
> Definitely, the commit he's bisect to there seems much more likely to be
> a culprit.  I'll take a closer look....
> 
> > Adam, can you create a loop filesystem with ext3 or something else
> > on it and see if the problem is XFS-specific?
> 
> Originally it sounded like he was able to reproduce this only on one
> specific filesystem so I wondered whether there was something particular
> to that filesystem.

I just found out it happens also with a newly created xfs and also with
ext3 and ext4. The reason that I did not see that initially is that the
script I'm using needs to be on the same fs which wasn't the case when I
initially tried it with a new fs. The sequence I'm using is:

# mount x.x.x:/home /tmp/st
# su - foo
$ cd /tmp/st/foo/x
$ ../test-script
foo@host:/tmp/st/foo/x$ ../test-script 
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
x.x.x:/home    39321600      15 39321585    1% /tmp/st
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
x.x.x:/home    39321600      16 39321584    1% /tmp/st
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
x.x.x:/home    39321600      16 39321584    1% /tmp/st



test-script:
#! /bin/sh

df -i .
cp /bin/ls x1
df -i .
cat x1 > /dev/null
rm x1
df -i .






Adam
-- 
Adam                 adam@os.inf.tu-dresden.de
  Lackorzynski         http://os.inf.tu-dresden.de/~adam/

Re: 2.6.38: Quota over NFS4

[J. Bruce Fields]

On Thu, Mar 24, 2011 at 11:28:07PM +0100, Adam Lackorzynski wrote:
> 
> On Thu Mar 24, 2011 at 13:51:06 -0400, J. Bruce Fields wrote:
> > On Thu, Mar 24, 2011 at 01:17:05PM -0400, Christoph Hellwig wrote:
> > > On Wed, Mar 23, 2011 at 11:30:17PM +0100, Adam Lackorzynski wrote:
> > > > I probably mistyped something and thus bisected again:
> > > 
> > > maybe this is just a symptom.  When was the nfs4 code converted
> > > to just use a single open struct file, and what are the lifetime
> > > rules for it?  It would easily be that a long alive file might
> > > make XFS keep persistent preallocations longer or similar issues.
> > 
> > Definitely, the commit he's bisect to there seems much more likely to be
> > a culprit.  I'll take a closer look....
> > 
> > > Adam, can you create a loop filesystem with ext3 or something else
> > > on it and see if the problem is XFS-specific?
> > 
> > Originally it sounded like he was able to reproduce this only on one
> > specific filesystem so I wondered whether there was something particular
> > to that filesystem.
> 
> I just found out it happens also with a newly created xfs and also with
> ext3 and ext4. The reason that I did not see that initially is that the
> script I'm using needs to be on the same fs which wasn't the case when I
> initially tried it with a new fs. The sequence I'm using is:

Ah-hah, got it.  So the leak's probably of a delegation on the file
containing the script itself.  OK, I should be able to fix that, thanks!

--b.

> 
> # mount x.x.x:/home /tmp/st
> # su - foo
> $ cd /tmp/st/foo/x
> $ ../test-script
> foo@host:/tmp/st/foo/x$ ../test-script 
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> x.x.x:/home    39321600      15 39321585    1% /tmp/st
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> x.x.x:/home    39321600      16 39321584    1% /tmp/st
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> x.x.x:/home    39321600      16 39321584    1% /tmp/st
> 
> 
> 
> test-script:
> #! /bin/sh
> 
> df -i .
> cp /bin/ls x1
> df -i .
> cat x1 > /dev/null
> rm x1
> df -i .
> 
> 
> 
> 
> 
> 
> Adam
> -- 
> Adam                 adam@os.inf.tu-dresden.de
>   Lackorzynski         http://os.inf.tu-dresden.de/~adam/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2.6.38: Quota over NFS4

[J. Bruce Fields]

On Thu, Mar 24, 2011 at 11:28:07PM +0100, Adam Lackorzynski wrote:
> 
> On Thu Mar 24, 2011 at 13:51:06 -0400, J. Bruce Fields wrote:
> > On Thu, Mar 24, 2011 at 01:17:05PM -0400, Christoph Hellwig wrote:
> > > On Wed, Mar 23, 2011 at 11:30:17PM +0100, Adam Lackorzynski wrote:
> > > > I probably mistyped something and thus bisected again:
> > > 
> > > maybe this is just a symptom.  When was the nfs4 code converted
> > > to just use a single open struct file, and what are the lifetime
> > > rules for it?  It would easily be that a long alive file might
> > > make XFS keep persistent preallocations longer or similar issues.
> > 
> > Definitely, the commit he's bisect to there seems much more likely to be
> > a culprit.  I'll take a closer look....
> > 
> > > Adam, can you create a loop filesystem with ext3 or something else
> > > on it and see if the problem is XFS-specific?
> > 
> > Originally it sounded like he was able to reproduce this only on one
> > specific filesystem so I wondered whether there was something particular
> > to that filesystem.
> 
> I just found out it happens also with a newly created xfs and also with
> ext3 and ext4. The reason that I did not see that initially is that the
> script I'm using needs to be on the same fs which wasn't the case when I
> initially tried it with a new fs. The sequence I'm using is:

Ah-hah, got it.  So the leak's probably of a delegation on the file
containing the script itself.  OK, I should be able to fix that, thanks!

--b.

> 
> # mount x.x.x:/home /tmp/st
> # su - foo
> $ cd /tmp/st/foo/x
> $ ../test-script
> foo@host:/tmp/st/foo/x$ ../test-script 
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> x.x.x:/home    39321600      15 39321585    1% /tmp/st
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> x.x.x:/home    39321600      16 39321584    1% /tmp/st
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> x.x.x:/home    39321600      16 39321584    1% /tmp/st
> 
> 
> 
> test-script:
> #! /bin/sh
> 
> df -i .
> cp /bin/ls x1
> df -i .
> cat x1 > /dev/null
> rm x1
> df -i .
> 
> 
> 
> 
> 
> 
> Adam
> -- 
> Adam                 adam@os.inf.tu-dresden.de
>   Lackorzynski         http://os.inf.tu-dresden.de/~adam/

Re: 2.6.38: Quota over NFS4

[J. Bruce Fields]

On Thu, Mar 24, 2011 at 08:03:10PM -0400, J. Bruce Fields wrote:
> On Thu, Mar 24, 2011 at 11:28:07PM +0100, Adam Lackorzynski wrote:
> > 
> > On Thu Mar 24, 2011 at 13:51:06 -0400, J. Bruce Fields wrote:
> > > On Thu, Mar 24, 2011 at 01:17:05PM -0400, Christoph Hellwig wrote:
> > > > On Wed, Mar 23, 2011 at 11:30:17PM +0100, Adam Lackorzynski wrote:
> > > > > I probably mistyped something and thus bisected again:
> > > > 
> > > > maybe this is just a symptom.  When was the nfs4 code converted
> > > > to just use a single open struct file, and what are the lifetime
> > > > rules for it?  It would easily be that a long alive file might
> > > > make XFS keep persistent preallocations longer or similar issues.
> > > 
> > > Definitely, the commit he's bisect to there seems much more likely to be
> > > a culprit.  I'll take a closer look....
> > > 
> > > > Adam, can you create a loop filesystem with ext3 or something else
> > > > on it and see if the problem is XFS-specific?
> > > 
> > > Originally it sounded like he was able to reproduce this only on one
> > > specific filesystem so I wondered whether there was something particular
> > > to that filesystem.
> > 
> > I just found out it happens also with a newly created xfs and also with
> > ext3 and ext4. The reason that I did not see that initially is that the
> > script I'm using needs to be on the same fs which wasn't the case when I
> > initially tried it with a new fs. The sequence I'm using is:
> 
> Ah-hah, got it.  So the leak's probably of a delegation on the file
> containing the script itself.  OK, I should be able to fix that, thanks!

This should be fixed in the latest upstream (so, in 39-rc5, when that
comes out) and soon in stable.  Let me know if not.

--b.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: 2.6.38: Quota over NFS4

[J. Bruce Fields]

On Thu, Mar 24, 2011 at 08:03:10PM -0400, J. Bruce Fields wrote:
> On Thu, Mar 24, 2011 at 11:28:07PM +0100, Adam Lackorzynski wrote:
> > 
> > On Thu Mar 24, 2011 at 13:51:06 -0400, J. Bruce Fields wrote:
> > > On Thu, Mar 24, 2011 at 01:17:05PM -0400, Christoph Hellwig wrote:
> > > > On Wed, Mar 23, 2011 at 11:30:17PM +0100, Adam Lackorzynski wrote:
> > > > > I probably mistyped something and thus bisected again:
> > > > 
> > > > maybe this is just a symptom.  When was the nfs4 code converted
> > > > to just use a single open struct file, and what are the lifetime
> > > > rules for it?  It would easily be that a long alive file might
> > > > make XFS keep persistent preallocations longer or similar issues.
> > > 
> > > Definitely, the commit he's bisect to there seems much more likely to be
> > > a culprit.  I'll take a closer look....
> > > 
> > > > Adam, can you create a loop filesystem with ext3 or something else
> > > > on it and see if the problem is XFS-specific?
> > > 
> > > Originally it sounded like he was able to reproduce this only on one
> > > specific filesystem so I wondered whether there was something particular
> > > to that filesystem.
> > 
> > I just found out it happens also with a newly created xfs and also with
> > ext3 and ext4. The reason that I did not see that initially is that the
> > script I'm using needs to be on the same fs which wasn't the case when I
> > initially tried it with a new fs. The sequence I'm using is:
> 
> Ah-hah, got it.  So the leak's probably of a delegation on the file
> containing the script itself.  OK, I should be able to fix that, thanks!

This should be fixed in the latest upstream (so, in 39-rc5, when that
comes out) and soon in stable.  Let me know if not.

--b.