Counters in df, du

Counters in df, du

[Alexey Vlasov]

[-- Attachment #1: Type: text/plain, Size: 804 bytes --]

Hi.

Periodically there is an overflow on /tmp on my boxes, and I can't find
the reaason why. May be it isn't a bug in tmpfs and you can help me find
the reason, and if it is an error, I hope you will help me to eliminate
it.

I know that df, du can show different values if some files were deleted,
but the processes which opened these files are still acting.

Well now I see this picture on one of my boxes. Below I post the results
of how I was trying to find what was /tmp full of.

# df -h /tmp
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 4.1G  3.3G  762M  82% /tmp

# du -hs /tmp
518M    /tmp

# lsof /tmp | awk '{if ($7 != 0) print$0}'
(in attach)

I still don't see any files taking 3G. Now I know only one solution
mount/remount /tmp.

-- 
BRGDS. Alexey Vlasov.

[-- Attachment #2: tmp.lsof --]
[-- Type: text/plain, Size: 1176 bytes --]

# lsof /tmp | awk '{if ($7 != 0) print$0}'
COMMAND     PID                           USER   FD   TYPE             DEVICE  SIZE      NODE NAME
authdaemo  6555                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo  6558                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo  6559                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo  6560                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo  6561                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
authdaemo  6562                           root    5u  unix 0xffff81030b8c2840       281375180 /var/lib/courier/authdaemon/socket.tmp
mysqld    22905                          mysql    6u   REG               0,13    20 663938322 /tmp/ib1oZTSZ (deleted)
mysqld    22905                          mysql    8u   REG               0,13 84773 663938424 /tmp/ibRcarbD (deleted)

Re: Counters in df, du

[Andi Kleen]

Alexey Vlasov <renton@renton.name> writes:
>
> I know that df, du can show different values if some files were deleted,
> but the processes which opened these files are still acting.

You can check for that by doing ls -l /proc/*/fd | grep deleted

It won't tell you the file size though.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: Counters in df, du

[Alexey Vlasov]

On Mon, Jul 13, 2009 at 11:09:29AM +0200, Andi Kleen wrote:
> Alexey Vlasov <renton@renton.name> writes:
> >
> > I know that df, du can show different values if some files were deleted,
> > but the processes which opened these files are still acting.
> 
> You can check for that by doing ls -l /proc/*/fd | grep deleted
> 
> It won't tell you the file size though.

There comes a big list, like this:

lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
[cut]
lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)

But you can see the file date, these are new files and by next search
you won't see them any more.

The most interesting is that the /tmp size constantly grows.

-- 
BRGDS. Alexey Vlasov.

[PATCH] Report file size for deleted files; was Re: Counters in df, du

[Andi Kleen]

> But you can see the file date, these are new files and by next search

I appended a patch that reports the size.

-Andi

---

VFS: Report file size for deleted files in /proc

It's useful to know how big deleted files are, in case a file system
is filled up unexpectedly and the admin wants to track down which
process is the culprit. This patch reports the file size 
in addition to (deleted) for /proc/*/fd links.

This modifies a public interface, but it's reasonable to assume
that noone relies on the particular format here.

Signed-off-by: Andi Kleen <ak@linux.intel.com>

---
 fs/dcache.c |   14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

Index: linux-2.6.31-rc1-ak/fs/dcache.c
===================================================================
--- linux-2.6.31-rc1-ak.orig/fs/dcache.c
+++ linux-2.6.31-rc1-ak/fs/dcache.c
@@ -1910,9 +1910,19 @@ char *__d_path(const struct path *path,
 
 	spin_lock(&vfsmount_lock);
 	prepend(&end, &buflen, "\0", 1);
-	if (d_unlinked(dentry) &&
-		(prepend(&end, &buflen, " (deleted)", 10) != 0))
+	if (d_unlinked(dentry)) {
+		char buf[40];
+		int n;
+		if (dentry->d_inode) { /* check needed? */
+			n = snprintf(buf, sizeof buf, " (deleted; size %lluK)",
+					dentry->d_inode->i_size >> 10);
+		} else {
+			strcpy(buf, " (deleted)");
+			n = 10;
+		}
+		if (prepend(&end, &buflen, buf, n) != 0)
 			goto Elong;
+	}
 
 	if (buflen < 1)
 		goto Elong;

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [PATCH] Report file size for deleted files; was Re: Counters in df, du

[Andreas Schwab]

Andi Kleen <andi@firstfloor.org> writes:

>> But you can see the file date, these are new files and by next search
>
> I appended a patch that reports the size.

You can get the size already by using stat instead of lstat, ie. use ls -lL.

Andreas.

-- 
Andreas Schwab, schwab@redhat.com
GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84  5EC7 45C6 250E 6F00 984E
"And now for something completely different."

Re: [PATCH] Report file size for deleted files; was Re: Counters in df, du

[Andi Kleen]

On Mon, Jul 13, 2009 at 03:13:09PM +0200, Andreas Schwab wrote:
> Andi Kleen <andi@firstfloor.org> writes:
> 
> >> But you can see the file date, these are new files and by next search
> >
> > I appended a patch that reports the size.
> 
> You can get the size already by using stat instead of lstat, ie. use ls -lL.

True, on the other hand you need two passes because that doesn't give
you file names. So the patch would be still more convenient.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: Counters in df, du

[Bryan Donlan]

2009/7/13 Alexey Vlasov <renton@renton.name>:
> On Mon, Jul 13, 2009 at 11:09:29AM +0200, Andi Kleen wrote:
>> Alexey Vlasov <renton@renton.name> writes:
>> >
>> > I know that df, du can show different values if some files were deleted,
>> > but the processes which opened these files are still acting.
>>
>> You can check for that by doing ls -l /proc/*/fd | grep deleted
>>
>> It won't tell you the file size though.
>
> There comes a big list, like this:
>
> lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
> lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
> lrwx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
> [cut]
> lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
> lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
> lrwx------ 1 root root 64 Jul 13 15:29 317 -> /tmp/ZCUDXiOHCk (deleted)
>
> But you can see the file date, these are new files and by next search
> you won't see them any more.
>
> The most interesting is that the /tmp size constantly grows.

Try:

lsof -n | grep deleted

This will show the process holding them more easily, and the file size
(in the seventh column). It won't show the date they were opened
though.

Re: Counters in df, du

[Alexey Vlasov]

On Mon, Jul 13, 2009 at 09:54:22AM -0400, Bryan Donlan wrote:
> >> Alexey Vlasov <renton@renton.name> writes:
> >> >
> >> > I know that df, du can show different values if some files were deleted,
> >> > but the processes which opened these files are still acting.
> >>
> >> You can check for that by doing ls -l /proc/*/fd | grep deleted
> >>
> >> It won't tell you the file size though.
> >
> > There comes a big list, like this:
> >
> > lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
> >
> > The most interesting is that the /tmp size constantly grows.
> 
> lsof -n | grep deleted

# lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'

mysqld    22905                          mysql    6u   REG
0,13    20 663938322 /tmp/ib1oZTSZ (deleted)
mysqld    22905                          mysql    8u   REG
0,13 84773 663938424 /tmp/ibRcarbD (deleted)

# df -h /tmp
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 4.1G  3.4G  630M  85% /tmp

It will soon be up to 100% full and I'll have to remount.

-- 
BRGDS. Alexey Vlasov.

Re: Counters in df, du

[Bryan Donlan]

2009/7/13 Alexey Vlasov <renton@renton.name>:
> On Mon, Jul 13, 2009 at 09:54:22AM -0400, Bryan Donlan wrote:
>> >> Alexey Vlasov <renton@renton.name> writes:
>> >> >
>> >> > I know that df, du can show different values if some files were deleted,
>> >> > but the processes which opened these files are still acting.
>> >>
>> >> You can check for that by doing ls -l /proc/*/fd | grep deleted
>> >>
>> >> It won't tell you the file size though.
>> >
>> > There comes a big list, like this:
>> >
>> > lowx------ 1 root root 64 Jul 13 14:58 1763 -> /tmp/ZCUDXqiMTp (deleted)
>> >
>> > The most interesting is that the /tmp size constantly grows.
>>
>> lsof -n | grep deleted
>
> # lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'
>
> mysqld    22905                          mysql    6u   REG
> 0,13    20 663938322 /tmp/ib1oZTSZ (deleted)
> mysqld    22905                          mysql    8u   REG
> 0,13 84773 663938424 /tmp/ibRcarbD (deleted)
>
> # df -h /tmp
> Filesystem            Size  Used Avail Use% Mounted on
> tmpfs                 4.1G  3.4G  630M  85% /tmp
>
> It will soon be up to 100% full and I'll have to remount.

Are those the only two results? That should only be adding up to ~84kb
of space...

Re: Counters in df, du

[Alexey Vlasov]

On Mon, Jul 13, 2009 at 03:58:36PM -0400, Bryan Donlan wrote:
> 2009/7/13 Alexey Vlasov <renton@renton.name>:
> > # lsof -n /tmp | awk '/deleted/ {if ($7 != 0) print$0}'
> >
> > mysqld    22905                          mysql    6u   REG
> > 0,13    20 663938322 /tmp/ib1oZTSZ (deleted)
> > mysqld    22905                          mysql    8u   REG
> > 0,13 84773 663938424 /tmp/ibRcarbD (deleted)
> >
> > # df -h /tmp
> > Filesystem            Size  Used Avail Use% Mounted on
> > tmpfs                 4.1G  3.4G  630M  85% /tmp
> 
> Are those the only two results? That should only be adding up to ~84kb
> of space...

Yes, only two lines where the 7-th column isn't equal to 0

-- 
BRGDS. Alexey Vlasov.

Re: [PATCH] Report file size for deleted files; was Re: Counters in df, du

[Alexey Vlasov]

Hi Andi Kleen.

Patch can hardly hellp, because there is another problem. Proc shows
neither the size of deleted files, nor what he's full with /tmp.

On Mon, Jul 13, 2009 at 01:59:17PM +0200, Andi Kleen wrote:
> > But you can see the file date, these are new files and by next search
> 
> I appended a patch that reports the size.
> 
> -Andi
> 
> ---
> 
> VFS: Report file size for deleted files in /proc
> 
> It's useful to know how big deleted files are, in case a file system
> is filled up unexpectedly and the admin wants to track down which
> process is the culprit. This patch reports the file size 
> in addition to (deleted) for /proc/*/fd links.
> 
> This modifies a public interface, but it's reasonable to assume
> that noone relies on the particular format here.
> 
> Signed-off-by: Andi Kleen <ak@linux.intel.com>
> 
> ---
>  fs/dcache.c |   14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6.31-rc1-ak/fs/dcache.c
> ===================================================================
> --- linux-2.6.31-rc1-ak.orig/fs/dcache.c
> +++ linux-2.6.31-rc1-ak/fs/dcache.c
> @@ -1910,9 +1910,19 @@ char *__d_path(const struct path *path,
>  
>  	spin_lock(&vfsmount_lock);
>  	prepend(&end, &buflen, "\0", 1);
> -	if (d_unlinked(dentry) &&
> -		(prepend(&end, &buflen, " (deleted)", 10) != 0))
> +	if (d_unlinked(dentry)) {
> +		char buf[40];
> +		int n;
> +		if (dentry->d_inode) { /* check needed? */
> +			n = snprintf(buf, sizeof buf, " (deleted; size %lluK)",
> +					dentry->d_inode->i_size >> 10);
> +		} else {
> +			strcpy(buf, " (deleted)");
> +			n = 10;
> +		}
> +		if (prepend(&end, &buflen, buf, n) != 0)
>  			goto Elong;
> +	}
>  
>  	if (buflen < 1)
>  		goto Elong;
> 
> -- 
> ak@linux.intel.com -- Speaking for myself only.

-- 
BRGDS. Alexey Vlasov.