fsck memory leak

fsck memory leak

[U.Mutlu]

Hi,
when I as root do "touch /forcefsck" and reboot, then fsck will be done.
But afterwards one has less free memory available than normal.
Example:
used mem immediately after login:
  without fsck during boot:  98 MB (this the normal level here)
  with fsck during boot   : 139 MB
So, there is a memory leak of about 41 MB.

fsck from util-linux 2.25.2
Filesys: ext4, 500 MB SSD
OS: Debian 8 X86_64, uptodate

-- 
U.Mutlu

Re: fsck memory leak

[U.Mutlu]

U.Mutlu wrote on 12/04/2015 08:15 AM:
> Hi,
> when I as root do "touch /forcefsck" and reboot, then fsck will be done.
> But afterwards one has less free memory available than normal.
> Example:
> used mem immediately after login:
>   without fsck during boot:  98 MB (this the normal level here)
>   with fsck during boot   : 139 MB
> So, there is a memory leak of about 41 MB.

This discrepancy is reproducible here, ie. happens always.

Not sure, but it could also be a kernel issue, because
the longer the system runs the more memory gets bound.
The source of the culprit is not easily detectable.
I guess it gets eaten by the kernel, probably by the ext4-driver.

> fsck from util-linux 2.25.2
> Filesys: ext4, 500 MB SSD

fix: of course 500 GB was meant

> OS: Debian 8 X86_64, uptodate

Re: fsck memory leak

[Karel Zak]

On Fri, Dec 04, 2015 at 08:49:34AM +0100, U.Mutlu wrote:
> U.Mutlu wrote on 12/04/2015 08:15 AM:
> >Hi,
> >when I as root do "touch /forcefsck" and reboot, then fsck will be done.
> >But afterwards one has less free memory available than normal.
> >Example:
> >used mem immediately after login:
> >  without fsck during boot:  98 MB (this the normal level here)
> >  with fsck during boot   : 139 MB
> >So, there is a memory leak of about 41 MB.
> 
> This discrepancy is reproducible here, ie. happens always.
> 
> Not sure, but it could also be a kernel issue, because
> the longer the system runs the more memory gets bound.
> The source of the culprit is not easily detectable.
> I guess it gets eaten by the kernel, probably by the ext4-driver.
> 
> >fsck from util-linux 2.25.2
> >Filesys: ext4, 500 MB SSD
> 
> fix: of course 500 GB was meant
> 
> >OS: Debian 8 X86_64, uptodate

And are you talking about fsck (wrapper) or fsck.ext4 (from
e2fsprogs)?

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: fsck memory leak

[Theodore Ts'o]

On Fri, Dec 04, 2015 at 08:15:14AM +0100, U.Mutlu wrote:
> Hi,
> when I as root do "touch /forcefsck" and reboot, then fsck will be done.
> But afterwards one has less free memory available than normal.
> Example:
> used mem immediately after login:
>  without fsck during boot:  98 MB (this the normal level here)
>  with fsck during boot   : 139 MB
> So, there is a memory leak of about 41 MB.

Please send the output of cat /proc/meminfo (a) before running fsck,
(b) after running fsck, and then (c) after running "echo 3 >
/proc/sys/vm/drop_caches".

Regards,

						- Ted

Re: fsck memory leak

[U.Mutlu]

Theodore Ts'o wrote on 12/04/2015 04:19 PM:
> On Fri, Dec 04, 2015 at 08:15:14AM +0100, U.Mutlu wrote:
>> Hi,
>> when I as root do "touch /forcefsck" and reboot, then fsck will be done.
>> But afterwards one has less free memory available than normal.
>> Example:
>> used mem immediately after login:
>>   without fsck during boot:  98 MB (this the normal level here)
>>   with fsck during boot   : 139 MB
>> So, there is a memory leak of about 41 MB.
>
> Please send the output of cat /proc/meminfo (a) before running fsck,
> (b) after running fsck, and then (c) after running "echo 3 >
> /proc/sys/vm/drop_caches".

Just tried it out.
Doing "echo 3 > /proc/sys/vm/drop_caches" solves the problem.

> Regards,
>
> 						- Ted

Re: fsck memory leak

[U.Mutlu]

Karel Zak wrote on 12/04/2015 11:21 AM:
> On Fri, Dec 04, 2015 at 08:49:34AM +0100, U.Mutlu wrote:
>> U.Mutlu wrote on 12/04/2015 08:15 AM:
>>> Hi,
>>> when I as root do "touch /forcefsck" and reboot, then fsck will be done.
>>> But afterwards one has less free memory available than normal.
>>> Example:
>>> used mem immediately after login:
>>>   without fsck during boot:  98 MB (this the normal level here)
>>>   with fsck during boot   : 139 MB
>>> So, there is a memory leak of about 41 MB.
>>
>> This discrepancy is reproducible here, ie. happens always.
>>
>> Not sure, but it could also be a kernel issue, because
>> the longer the system runs the more memory gets bound.
>> The source of the culprit is not easily detectable.
>> I guess it gets eaten by the kernel, probably by the ext4-driver.
>>
>>> fsck from util-linux 2.25.2
>>> Filesys: ext4, 500 MB SSD
>>
>> fix: of course 500 GB was meant
>>
>>> OS: Debian 8 X86_64, uptodate
>
> And are you talking about fsck (wrapper) or fsck.ext4 (from
> e2fsprogs)?

I don't know which one gets used in the above scenario because
it is done automatically by the system.

But the problem is solved now by setting /proc/sys/vm/drop_caches
to the value 3 as was written in the other postings.

>
>      Karel

Re: fsck memory leak

[U.Mutlu]

U.Mutlu wrote on 12/04/2015 06:53 PM:
> Theodore Ts'o wrote on 12/04/2015 04:19 PM:
>> On Fri, Dec 04, 2015 at 08:15:14AM +0100, U.Mutlu wrote:
>>> Hi,
>>> when I as root do "touch /forcefsck" and reboot, then fsck will be done.
>>> But afterwards one has less free memory available than normal.
>>> Example:
>>> used mem immediately after login:
>>>   without fsck during boot:  98 MB (this the normal level here)
>>>   with fsck during boot   : 139 MB
>>> So, there is a memory leak of about 41 MB.
>>
>> Please send the output of cat /proc/meminfo (a) before running fsck,
>> (b) after running fsck, and then (c) after running "echo 3 >
>> /proc/sys/vm/drop_caches".
>
> Just tried it out.
> Doing "echo 3 > /proc/sys/vm/drop_caches" solves the problem.

But thinking twice about this leads me to the conclusion that
there still must be something fishy in the system, because
I had read the used mem from the "-/+ buffers/cache" line of the "free -lh" 
command, here an example output:

$ free -lh
              total       used       free     shared    buffers     cached
Mem:          6.9G       1.3G       5.6G        17M        70M       882M
Low:          6.9G       1.3G       5.6G
High:           0B         0B         0B
-/+ buffers/cache:       404M       6.5G
Swap:         8.0G         0B       8.0G

So, it should have already counted-in the cached memory, and IMO it does.
But then it's mysterious why the said discrepancy of 41 MB shows up
if /proc/sys/vm/drop_caches has the default value of 0.

Re: fsck memory leak

[Theodore Ts'o]

On Fri, Dec 04, 2015 at 06:53:53PM +0100, U.Mutlu wrote:
> >Please send the output of cat /proc/meminfo (a) before running fsck,
> >(b) after running fsck, and then (c) after running "echo 3 >
> >/proc/sys/vm/drop_caches".
> 
> Just tried it out.
> Doing "echo 3 > /proc/sys/vm/drop_caches" solves the problem.

There's no problem then --- except in your understanding of how
Linux's memory management system works.

Linux keeps data that was read from disk in its buffer and page cache,
because there is always a chance that data could be needed again.  So
although it looks like free space is being consumed by the buffer and
page caches, if those pages are clean (not dirty; if they were
modified, we've gotten around to writing the modified data back to
disk) and inactive (not installed in some process's page table, such
that the page is visible in some process's virtual address space),
even though those pages aren't counted as "free" memory, it can be
instantly reused for something else.

So doing something like "echo 3 > /proc/sys/vm/drop_caches" is
actually a _bad_ thing, since it means you are throwing away data that
could be used to avoid blocking waiting for disk I/O to complete.
However, if it helps satisfy some system administrator's insecurity
because they are looking at this field:

          total      used      free    shared  buff/cache   available
Mem:        15G      4.4G  ==> 4.5G <==  392M        6.4G         10G
Swap:        0B        0B        0B

when they should be looking at *this* field:

          total      used      free    shared  buff/cache   available
Mem:        15G      4.4G      4.5G      392M        6.4G    ===> 10G <===
Swap:        0B        0B        0B

It's just being silly.

Yes, after doing the "echo 3 > /proc/sys/vm/drop_caches", you would
see this instead:

          total      used      free    shared  buff/cache   available
Mem:        15G      4.1G       10G      390M        927M         10G
Swap:        0B        0B        0B

But reducing the size of buff/cache is not a win!  If you suddenly
need the memory because you suddently started some memory hog such as
Eclipse, the system will automatically reclaim the clean and inactive
pages for use by the new memory user.  But until then, why not keep
the cache pages around just in case they are needed?

Regards,

						- Ted

Re: fsck memory leak

[U.Mutlu]

Theodore Ts'o wrote on 12/04/2015 08:58 PM:
> On Fri, Dec 04, 2015 at 06:53:53PM +0100, U.Mutlu wrote:
>>> Please send the output of cat /proc/meminfo (a) before running fsck,
>>> (b) after running fsck, and then (c) after running "echo 3 >
>>> /proc/sys/vm/drop_caches".
>>
>> Just tried it out.
>> Doing "echo 3 > /proc/sys/vm/drop_caches" solves the problem.
>
> There's no problem then --- except in your understanding of how
> Linux's memory management system works.
>
> Linux keeps data that was read from disk in its buffer and page cache,
> because there is always a chance that data could be needed again.  So
> although it looks like free space is being consumed by the buffer and
> page caches, if those pages are clean (not dirty; if they were
> modified, we've gotten around to writing the modified data back to
> disk) and inactive (not installed in some process's page table, such
> that the page is visible in some process's virtual address space),
> even though those pages aren't counted as "free" memory, it can be
> instantly reused for something else.
>
> So doing something like "echo 3 > /proc/sys/vm/drop_caches" is
> actually a _bad_ thing, since it means you are throwing away data that
> could be used to avoid blocking waiting for disk I/O to complete.
> However, if it helps satisfy some system administrator's insecurity
> because they are looking at this field:
>
>            total      used      free    shared  buff/cache   available
> Mem:        15G      4.4G  ==> 4.5G <==  392M        6.4G         10G
> Swap:        0B        0B        0B
>
> when they should be looking at *this* field:
>
>            total      used      free    shared  buff/cache   available
> Mem:        15G      4.4G      4.5G      392M        6.4G    ===> 10G <===
> Swap:        0B        0B        0B
>
> It's just being silly.
>
> Yes, after doing the "echo 3 > /proc/sys/vm/drop_caches", you would
> see this instead:
>
>            total      used      free    shared  buff/cache   available
> Mem:        15G      4.1G       10G      390M        927M         10G
> Swap:        0B        0B        0B
>
> But reducing the size of buff/cache is not a win!  If you suddenly
> need the memory because you suddently started some memory hog such as
> Eclipse, the system will automatically reclaim the clean and inactive
> pages for use by the new memory user.  But until then, why not keep
> the cache pages around just in case they are needed?
>
> Regards,
>
> 						- Ted


I think it's a double-edged sword: if user has less memory then
the integrated caching will IMO degrade the performance.

Btw, why does my "free" command not have the "available" column?
Or did you use a different tool for the above outputs?

Re: fsck memory leak

[Ruediger Meier]

On Friday 04 December 2015, U.Mutlu wrote:
> I think it's a double-edged sword: if user has less memory then
> the integrated caching will IMO degrade the performance.

It will use as much memory as available (not more). Ideally Linux would 
use always 100% memory. You've spent money for memory ... why you 
wouldn't want to use it?

After ...
$ echo 3 > /proc/sys/vm/drop_caches

... my memory looks like this:
$ free -h
             total    used    free       shared    buffers     cached
Mem:          7.7G    1.8G    6.0G         230M       4.4M       487M
-/+ buffers/cache:    1.3G    6.5G
Swap:         1.7G     68M    1.6G

Then after ...
$ dd if=/dev/sda of=/dev/null count=8K bs=1M

... cache/buffer is filled
$ free -h
             total    used    free       shared    buffers     cached
Mem:          7.7G    7.6G    168M <=(1)   230M       5.6G       665M
-/+ buffers/cache:    1.3G    6.5G <=(2)
Swap:         1.7G     68M    1.6G

... and this should not change until reboot.

(1) shows that almost 100% memory is "in use"
(2) shows that it's just buffer or cache

sorun yapma ;)
Rudi

Re: fsck memory leak

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 360 bytes --]

On 04 Dec 2015 14:58, Theodore Ts'o wrote:
> So doing something like "echo 3 > /proc/sys/vm/drop_caches" is
> actually a _bad_ thing, since it means you are throwing away data that
> could be used to avoid blocking waiting for disk I/O to complete.

just to clarify for other readers: "bad" here means "bad for performance",
not "bad" as in "data loss".
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: fsck memory leak

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

On 04 Dec 2015 21:31, U.Mutlu wrote:
> I think it's a double-edged sword: if user has less memory then
> the integrated caching will IMO degrade the performance.

it is not.  the user doesn't have less memory -- if the user requests pages
and there aren't enough free, then the caches will automatically be trimmed
to satisfy the request.  (obviously i'm glossing over details, but i think
this is sufficient based on the quested generalization.)
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: fsck memory leak

[Theodore Ts'o]

On Fri, Dec 04, 2015 at 09:31:36PM +0100, U.Mutlu wrote:
> 
> I think it's a double-edged sword: if user has less memory then
> the integrated caching will IMO degrade the performance.

You're wrong.

> Btw, why does my "free" command not have the "available" column?
> Or did you use a different tool for the above outputs?

Probably because you're using an older version of procps.  The
difference is between 2:3.3.9-9 and 2:3.3.10-2 (which recently entered
Debian testing).

The older version of free has a "-/+ buffers/cache" line, and the Free
entry in the "-/+ buffer/cache" line is equivalent to the available
line.  So you're version of free looks like this:

             total       used       free     shared    buffers     cached
Mem:           15G       7.1G       8.2G       421M       303M       1.6G
-/+ buffers/cache:       5.2G   ===> 10G <===
Swap:           0B         0B         0B

							- Ted
















;



> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe util-linux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: fsck memory leak

[U.Mutlu]

Ruediger Meier wrote on 12/04/2015 10:40 PM:
> On Friday 04 December 2015, U.Mutlu wrote:
>> I think it's a double-edged sword: if user has less memory then
>> the integrated caching will IMO degrade the performance.
>
> It will use as much memory as available (not more). Ideally Linux would
> use always 100% memory. You've spent money for memory ... why you
> wouldn't want to use it?
>
> After ...
> $ echo 3 > /proc/sys/vm/drop_caches
>
> ... my memory looks like this:
> $ free -h
>               total    used    free       shared    buffers     cached
> Mem:          7.7G    1.8G    6.0G         230M       4.4M       487M
> -/+ buffers/cache:    1.3G    6.5G
> Swap:         1.7G     68M    1.6G
>
> Then after ...
> $ dd if=/dev/sda of=/dev/null count=8K bs=1M
>
> ... cache/buffer is filled
> $ free -h
>               total    used    free       shared    buffers     cached
> Mem:          7.7G    7.6G    168M <=(1)   230M       5.6G       665M
> -/+ buffers/cache:    1.3G    6.5G <=(2)
> Swap:         1.7G     68M    1.6G
>
> ... and this should not change until reboot.
>
> (1) shows that almost 100% memory is "in use"
> (2) shows that it's just buffer or cache

Try the test with fsck at boot with drop_caches=0, and you will get an
illogical result as shown in my initial posting.

I'm not a friend of such default integrated system caching, it reminds
me of Windows idiocy. This is nothing but a diskcache in ram, but then
the admin should have the the freedom to set the size of the cache via
a config file in etc, for example /etc/default/cache or in /etc/default/tmpfs.

> sorun yapma ;)

:-)

> Rudi

Re: fsck memory leak

[Dave Rutherford]

On Fri, Dec 4, 2015 at 5:26 PM, U.Mutlu <for-gmane@mutluit.com> wrote:
> I'm not a friend of such default integrated system caching, it reminds
> me of Windows idiocy. This is nothing but a diskcache in ram,

... made up strictly of memory that nothing else wants.
This is different than a ramdisk, which itself is something that wants 
RAM.

> but then
> the admin should have the the freedom to set the size of the cache via
> a config file in etc, for example /etc/default/cache or in 
/etc/default/tmpfs.

Why? It's memory that *nothing else wants*. It's given up instantly
the moment something else needs to use it. So there's no downside,
unlike with that other "operating system" you mention.

> :-)

:-(

-- 
Prius I, Pontiff, Schenectadian Catholic Church