Strange case of deleting lots of files and "df" not recognizing it....(unless I umount/mount) the disk...

Strange case of deleting lots of files and "df" not recognizing it....(unless I umount/mount) the disk...

[Linda Walsh]

This isn't that big of a deal, as I'm not close to the limit,
but I pruned a backup disk that was at 87% full (1.5T free), down
by 3.8T.  Indeed, doing a 'du' of the disk shows space is ~6-7T (/11T)
but df still shows 9.5T usage.

I've done sync a few times, but the files were deleted almost a half an hour
ago.

So when does it actually synchronize w/o me forcing it?  (I.e. umount/mount)?

Have seen this before, and that's what brought it into 'sync'....

none of the xfs_centitimes are greater than 3000 or 30 seconds.  Not a big
problem that I know of, hardly a problem at all, cept that it is weird.

Curious?
-linda

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange case of deleting lots of files and "df" not recognizing it....(unless I umount/mount) the disk...

[Emmanuel Florac]

Le Sun, 24 Jun 2012 22:58:12 -0700 vous écriviez:

> So when does it actually synchronize w/o me forcing it?  (I.e.
> umount/mount)?
> 

Yes, it happens sometimes and a umount/mount is needed to fix it.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange case of deleting lots of files and "df" not recognizing it....(unless I umount/mount) the disk...

[Ben Myers]

On Mon, Jun 25, 2012 at 09:02:11AM +0200, Emmanuel Florac wrote:
> Le Sun, 24 Jun 2012 22:58:12 -0700 vous écriviez:
> 
> > So when does it actually synchronize w/o me forcing it?  (I.e.
> > umount/mount)?
> > 
> 
> Yes, it happens sometimes and a umount/mount is needed to fix it.

You may find that this will get the job done:
echo 3 > /proc/sys/vm/drop_caches

-Ben

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange case of deleting lots of files and "df" not recognizing it....(unless I umount/mount) the disk...

[Linda Walsh]

Ben Myers wrote:
> On Mon, Jun 25, 2012 at 09:02:11AM +0200, Emmanuel Florac wrote:
>> Le Sun, 24 Jun 2012 22:58:12 -0700 vous écriviez:
>>
>>> So when does it actually synchronize w/o me forcing it?  (I.e.
>>> umount/mount)?
>>>
>> Yes, it happens sometimes and a umount/mount is needed to fix it.
> 
> You may find that this will get the job done:
> echo 3 > /proc/sys/vm/drop_caches
> 
> -Ben
> 
Yup...it went away for a LONG time...maybe 30 seconds...

Does that mean there was that much unsynchronized data being held in memory
that wasn't being written out??

It DID fix the space allocation issue.

When I read the value before changing it, it said '0'.

when it finished, I tried to set it back to '0', but got an invalid argument.???
Was it really '2' or something else?


BTW -- it had been in the weird state for 13 hours before I issued the
order to drop_caches... seems like a long time not to cache data...?


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange case of deleting lots of files and "df" not recognizing it....(unless I umount/mount) the disk...

[Ben Myers]

Hey Linda,

On Mon, Jun 25, 2012 at 03:08:15PM -0700, Linda Walsh wrote:
> Ben Myers wrote:
> >On Mon, Jun 25, 2012 at 09:02:11AM +0200, Emmanuel Florac wrote:
> >>Le Sun, 24 Jun 2012 22:58:12 -0700 vous écriviez:
> >>
> >>>So when does it actually synchronize w/o me forcing it?  (I.e.
> >>>umount/mount)?
> >>>
> >>Yes, it happens sometimes and a umount/mount is needed to fix it.
> >
> >You may find that this will get the job done:
> >echo 3 > /proc/sys/vm/drop_caches
> >
> >-Ben
> 
> Yup...it went away for a LONG time...maybe 30 seconds...
> 
> Does that mean there was that much unsynchronized data being held in memory
> that wasn't being written out??
>
> It DID fix the space allocation issue.

XFS doesn't clean up the blocks for a deleted file until the last reference on
the inode goes away.  Dropping caches just forces the issue.

> When I read the value before changing it, it said '0'.
> 
> when it finished, I tried to set it back to '0', but got an invalid argument.???
> Was it really '2' or something else?

Don't worry about the current value of /proc/sys/vm/drop_caches... It just
retains the last value written to it but doesn't do anything with it.  The only
time it takes an action is when you write to it.

> BTW -- it had been in the weird state for 13 hours before I issued the
> order to drop_caches... seems like a long time not to cache data...?

Yeah.  13 hours is a long time, esp for something you just removed.  ;)

Fortunately it's not the data that is cached, it's probably just a pesky dentry
hanging around on the lru with a single reference on the inode keeping it
active, preventing xfs from freeing up the blocks.

I've had some luck on v3.0 with the following patch... it's not yet tested on
3.5 so YMMV.  Are you running an NFS server?  Something that often creates
anonymous dentries?

Regards,
Ben

---
 fs/xfs/xfs_vnodeops.c |   41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

Index: xfs/fs/xfs/xfs_vnodeops.c
===================================================================
--- xfs.orig/fs/xfs/xfs_vnodeops.c
+++ xfs/fs/xfs/xfs_vnodeops.c
@@ -1203,6 +1203,41 @@ xfs_lock_two_inodes(
 	}
 }
 
+/*
+ * Prune any disconnected dentries from the inode.  This will help to ensure
+ * timely teardown of the inode by unhashing all disconnected (anonymous)
+ * dentries that may have been added by an interface that uses filehandles like
+ * NFS.
+ *
+ * Here we know that there must currently be a dentry with a name on this inode
+ * because we're in an unlink/rmdir path, so we do not run the risk of inode
+ * reclaim here because we only unhash disconnected dentries.
+ */
+static void
+xfs_d_prune_disconnected(
+	struct xfs_inode	*ip)
+{
+	struct inode		*inode = VFS_I(ip);
+	struct dentry		*alias;
+
+restart:
+	spin_lock(&inode->i_lock);
+	list_for_each_entry(alias, &inode->i_dentry, d_alias) {
+		spin_lock(&alias->d_lock);
+		if (alias->d_flags & DCACHE_DISCONNECTED &&
+		    !d_unhashed(alias)) {
+			dget_dlock(alias);
+			__d_drop(alias);
+			spin_unlock(&alias->d_lock);
+			spin_unlock(&inode->i_lock);
+			dput(alias);
+			goto restart;
+		}
+		spin_unlock(&alias->d_lock);
+	}
+	spin_unlock(&inode->i_lock);
+}
+
 int
 xfs_remove(
 	xfs_inode_t             *dp,
@@ -1347,6 +1382,9 @@ xfs_remove(
 	if (error)
 		goto std_return;
 
+	if (link_zero)
+		xfs_d_prune_disconnected(ip);
+
 	/*
 	 * If we are using filestreams, kill the stream association.
 	 * If the file is still open it may get a new one but that

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange case of deleting lots of files and "df" not recognizing it....(unless I umount/mount) the disk...

[Linda Walsh]

Ben Myers wrote:

> 
> Yeah.  13 hours is a long time, esp for something you just removed.  ;)
> 
> Fortunately it's not the data that is cached, it's probably just a pesky dentry
> hanging around on the lru with a single reference on the inode keeping it
> active, preventing xfs from freeing up the blocks.
> 
> I've had some luck on v3.0 with the following patch... it's not yet tested on
> 3.5 so YMMV.  Are you running an NFS server?  Something that often creates
> anonymous dentries?
----
nfs server is running but no clients (usage=rare)
samba server is running and has clients (usage=24/7)...

I wasn't even able to compile the last kernel I tried...my release has been
a bit screwey...

finally got perl to compile, that was another thorn in my side...

(when it rains it pours....)

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange case of deleting lots of files and "df" not recognizing it....(unless I umount/mount) the disk...

[Dave Chinner]

On Sun, Jun 24, 2012 at 10:58:12PM -0700, Linda Walsh wrote:
> This isn't that big of a deal, as I'm not close to the limit,
> but I pruned a backup disk that was at 87% full (1.5T free), down
> by 3.8T.  Indeed, doing a 'du' of the disk shows space is ~6-7T (/11T)
> but df still shows 9.5T usage.
> 
> I've done sync a few times, but the files were deleted almost a half an hour
> ago.
> 
> So when does it actually synchronize w/o me forcing it?  (I.e. umount/mount)?

As ben said, the last reference has to go away before the space is
freed. lsof is your friend.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange case of deleting lots of files and "df" not recognizing it....(unless I umount/mount) the disk...

[Emmanuel Florac]

Le Tue, 26 Jun 2012 12:33:02 +1000 vous écriviez:

> As ben said, the last reference has to go away before the space is
> freed. lsof is your friend.

I just had the same problem recently, but lsof reported zero open files
on the volume. umount/mount worked though, I didn't thought the cache
trick would be enough.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange case of deleting lots of files and "df" not recognizing it....(unless I umount/mount) the disk...

[Dave Chinner]

On Tue, Jun 26, 2012 at 09:33:23AM +0200, Emmanuel Florac wrote:
> Le Tue, 26 Jun 2012 12:33:02 +1000 vous écriviez:
> 
> > As ben said, the last reference has to go away before the space is
> > freed. lsof is your friend.
> 
> I just had the same problem recently, but lsof reported zero open files
> on the volume. umount/mount worked though, I didn't thought the cache
> trick would be enough.

I think that we need a reproducable test case for this issue before
we'll be able to make any progress on it, then....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Strange case of deleting lots of files and "df" not recognizing it....(unless I umount/mount) the disk...

[Linda Walsh]

Well, in my case, may be a while, it is usually every few months when I
clear out about 30% of a large (~11T) volume on my backup disk...

It only grows slowly as I make backups on it...then I delete maybe ...
60-100 files worth of backups (xfsdumps)....but for me dropping
the cache did work so it may not be the same problem.



Dave Chinner wrote:
> On Tue, Jun 26, 2012 at 09:33:23AM +0200, Emmanuel Florac wrote:
>> Le Tue, 26 Jun 2012 12:33:02 +1000 vous écriviez:
>>
>>> As ben said, the last reference has to go away before the space is
>>> freed. lsof is your friend.
>> I just had the same problem recently, but lsof reported zero open files
>> on the volume. umount/mount worked though, I didn't thought the cache
>> trick would be enough.
> 
> I think that we need a reproducable test case for this issue before
> we'll be able to make any progress on it, then....
> 
> Cheers,
> 
> Dave.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs