Why is Shmem included in Cached in /proc/meminfo?

Why is Shmem included in Cached in /proc/meminfo?

[Mikko Rantalainen]

It's not immediately obvious from fs/proc/meminfo.c function
meminfo_proc_show() but the output of Cached: field seems to always
include all of Shmem: field, too.

Is this intentional? Usually cache is something that can be discarded if
needed but shared memory (e.g. used to contain files in tmpfs) cannot be
discarded without a data-loss. As such, I'd argue that it shouldn't be
included in the Cached: output.

A simple fix could be

-	cached = global_node_page_state(NR_FILE_PAGES) -
-			total_swapcache_pages() - i.bufferram;
+	cached = global_node_page_state(NR_FILE_PAGES) -
+			total_swapcache_pages()
+			- i.bufferram - i.sharedram;

-- 
Mikko

Re: Why is Shmem included in Cached in /proc/meminfo?

[Randy Dunlap]

[add linux-mm mailing list]

On 8/30/21 12:44 AM, Mikko Rantalainen wrote:
> It's not immediately obvious from fs/proc/meminfo.c function
> meminfo_proc_show() but the output of Cached: field seems to always
> include all of Shmem: field, too.
> 
> Is this intentional? Usually cache is something that can be discarded if
> needed but shared memory (e.g. used to contain files in tmpfs) cannot be
> discarded without a data-loss. As such, I'd argue that it shouldn't be
> included in the Cached: output.
> 
> A simple fix could be
> 
> -	cached = global_node_page_state(NR_FILE_PAGES) -
> -			total_swapcache_pages() - i.bufferram;
> +	cached = global_node_page_state(NR_FILE_PAGES) -
> +			total_swapcache_pages()
> +			- i.bufferram - i.sharedram;
> 


-- 
~Randy

Re: Why is Shmem included in Cached in /proc/meminfo?

[Matthew Wilcox]

On Mon, Aug 30, 2021 at 07:34:38AM -0700, Randy Dunlap wrote:
> [add linux-mm mailing list]
> 
> On 8/30/21 12:44 AM, Mikko Rantalainen wrote:
> > It's not immediately obvious from fs/proc/meminfo.c function
> > meminfo_proc_show() but the output of Cached: field seems to always
> > include all of Shmem: field, too.
> > 
> > Is this intentional? Usually cache is something that can be discarded if
> > needed but shared memory (e.g. used to contain files in tmpfs) cannot be
> > discarded without a data-loss. As such, I'd argue that it shouldn't be
> > included in the Cached: output.

That's a reasonable position to take.

Another point of view is that everything in tmpfs is part of the page
cache and can be written out to swap, so keeping it as part of Cached
is not misleading.

I can see it both ways, and personally, I'd lean towards clarifying
the documentation about how shmem is accounted rather than changing
how the memory usage is reported.

Re: Why is Shmem included in Cached in /proc/meminfo?

[Vlastimil Babka]

On 8/30/21 16:41, Matthew Wilcox wrote:
> On Mon, Aug 30, 2021 at 07:34:38AM -0700, Randy Dunlap wrote:
>> [add linux-mm mailing list]
>> 
>> On 8/30/21 12:44 AM, Mikko Rantalainen wrote:
>> > It's not immediately obvious from fs/proc/meminfo.c function
>> > meminfo_proc_show() but the output of Cached: field seems to always
>> > include all of Shmem: field, too.
>> > 
>> > Is this intentional? Usually cache is something that can be discarded if
>> > needed but shared memory (e.g. used to contain files in tmpfs) cannot be
>> > discarded without a data-loss. As such, I'd argue that it shouldn't be
>> > included in the Cached: output.
> 
> That's a reasonable position to take.
> 
> Another point of view is that everything in tmpfs is part of the page
> cache and can be written out to swap, so keeping it as part of Cached
> is not misleading.

Yeah, but with that view, anonymous memory can be also written out to swap. But
it's non-intuitive that something called "Cached" will contain something that
(if not dirty) can't be just dropped. I've had to point this Shmem oddity out a
number of times to someone, so I would say that it would be much better if it
was not part of Cached.

However, if we change it now, we might create even larger confusion. People
looking at the output for the first time (and IIRC also the 'free' command uses
it) on a new kernel wouldn't be misled anymore. But people working with both old
and new kernels will now have to take in account that it changed at some
point... not good.

> I can see it both ways, and personally, I'd lean towards clarifying
> the documentation about how shmem is accounted rather than changing
> how the memory usage is reported.
> 

Re: Why is Shmem included in Cached in /proc/meminfo?

[Matthew Wilcox]

On Mon, Aug 30, 2021 at 06:05:58PM +0200, Vlastimil Babka wrote:
> On 8/30/21 16:41, Matthew Wilcox wrote:
> > On Mon, Aug 30, 2021 at 07:34:38AM -0700, Randy Dunlap wrote:
> >> [add linux-mm mailing list]
> >> 
> >> On 8/30/21 12:44 AM, Mikko Rantalainen wrote:
> >> > It's not immediately obvious from fs/proc/meminfo.c function
> >> > meminfo_proc_show() but the output of Cached: field seems to always
> >> > include all of Shmem: field, too.
> >> > 
> >> > Is this intentional? Usually cache is something that can be discarded if
> >> > needed but shared memory (e.g. used to contain files in tmpfs) cannot be
> >> > discarded without a data-loss. As such, I'd argue that it shouldn't be
> >> > included in the Cached: output.
> > 
> > That's a reasonable position to take.
> > 
> > Another point of view is that everything in tmpfs is part of the page
> > cache and can be written out to swap, so keeping it as part of Cached
> > is not misleading.
> 
> Yeah, but with that view, anonymous memory can be also written out to swap. But
> it's non-intuitive that something called "Cached" will contain something that
> (if not dirty) can't be just dropped.

That's equally true for normal filesystems & shmem though.  Consider shmem
written to swap, then brought back in by a read.  Now it can be dropped
without being swapped out.  Or even a file on shmem ftruncated to a
large size, then only read.  The pages will be clean and full of zeroes.
They can be dropped under memory pressure without being written out.

> I've had to point this Shmem oddity out a
> number of times to someone, so I would say that it would be much better if it
> was not part of Cached.
> However, if we change it now, we might create even larger confusion. People
> looking at the output for the first time (and IIRC also the 'free' command uses
> it) on a new kernel wouldn't be misled anymore. But people working with both old
> and new kernels will now have to take in account that it changed at some
> point... not good.

Another good point.

Re: Why is Shmem included in Cached in /proc/meminfo?

[Mikko Rantalainen]

Matthew Wilcox (2021-08-30 19:17 Europe/Helsinki):
> On Mon, Aug 30, 2021 at 06:05:58PM +0200, Vlastimil Babka wrote:
>> On 8/30/21 16:41, Matthew Wilcox wrote:
>>> Another point of view is that everything in tmpfs is part of the page
>>> cache and can be written out to swap, so keeping it as part of Cached
>>> is not misleading.
>>
>> Yeah, but with that view, anonymous memory can be also written out to swap. But
>> it's non-intuitive that something called "Cached" will contain something that
>> (if not dirty) can't be just dropped.
> 
> That's equally true for normal filesystems & shmem though.  Consider shmem
> written to swap, then brought back in by a read.  Now it can be dropped
> without being swapped out.  Or even a file on shmem ftruncated to a
> large size, then only read.  The pages will be clean and full of zeroes.
> They can be dropped under memory pressure without being written out.
> 
>> I've had to point this Shmem oddity out a
>> number of times to someone, so I would say that it would be much better if it
>> was not part of Cached.
>> However, if we change it now, we might create even larger confusion. People
>> looking at the output for the first time (and IIRC also the 'free' command uses
>> it) on a new kernel wouldn't be misled anymore. But people working with both old
>> and new kernels will now have to take in account that it changed at some
>> point... not good.
> 
> Another good point.

I agree that backwards compatibility is a huge point to consider.

That said, I always assumed that Shmem was in *addition* to Cached and I
was happy to see lots of Cached on all my systems. However, when I
started to run into OOM situations with 10+ GB of Cached I guessed that
something was not working correctly with memory reclaim - it turned out
that the problem was that over 95% of my "Cached" was actually Shmem.

Do you know any existing software where it's important to have the
behavior of Cached and Shmem that the current kernel has?

Without backwards compatibility issues I'd prefer following fields:

- Dirty: total amount of RAM used to buffer data to be written on
permanent storage (dirty). Gets converted to Cached when write is
complete. (Actually I would call this "Buffers" but Dirty is okay, too.)
- Cached: total amount of RAM used to improve *performance* that can be
*immediately dropped* without any data-loss – note that this includes
all untouched RAM backed by swap.
- Shared: total amount of RAM shared between multiple process that
cannot be freed even if any single process gets killed. (If this is even
possible to know - note that this would *only* contain COW pages in
practice. We already have Committed_AS which is about as good for real
world heuristics.)
- RamDrive: total amount of RAM used for tmpfs storage and similar. Note
that this would tend towards zero when tmpfs storage is written to swap.
- Committed_RamDrive: virtual amount of space used for tmpfs (e.g. large
truncated files that, if written, will require to be backed for real).
- Swapped_RamDrive: the amount of swap backing tmpfs and similar.
- LegacyIPCS: all stuff reported by ipcs.

That is, I would declare "Cached" as true cache and split Shmem into
separate parts that are easier to understand. The Shmem currently seems
to be catch-all class for more and more stuff. Obviously the Shmem field
could be kept as precomputed sum of the separate parts for backwards
compatibility. Does Shmem nowadays include other big memory users but
ipcs and tmpfs?

My point is that it's currently too hard to understand (at least) tmpfs
and ipcs load on RAM and swap.

I guess the important thing to decide is if "Cached" should represent
actual cache size or is backwards compatibility more important? My guess
is that the backwards compability would affect statistics and heuristics
only but that's obviously just a guess.

Of course one possible solution is to keep "Cached" as is and introduce
"Cache" with the real cache semantics (that is, it includes sum of
(Cached - Shmem) and memory backed RAM). That way system administrators
would at least see two different fields with unique values and look for
the documentation.

-- 
Mikko

Re: Why is Shmem included in Cached in /proc/meminfo?

[Khalid Aziz]

On Mon, 2021-08-30 at 20:26 +0300, Mikko Rantalainen wrote:
> ... deleted ...
> Of course one possible solution is to keep "Cached" as is and
> introduce
> "Cache" with the real cache semantics (that is, it includes sum of
> (Cached - Shmem) and memory backed RAM). That way system
> administrators
> would at least see two different fields with unique values and look
> for
> the documentation.
> 

I would recommend adding new field. There is likely to be a good number
of tools/scripts out there that already interpret the data from
/proc/meminfo and possily take actions based upon that data. Those
tools will break if we change the sense of existing data. A new field
has the down side of expanding the output further but it also doesn't
break existing tols.

--
Khalid

Re: Why is Shmem included in Cached in /proc/meminfo?

[Mikko Rantalainen]

Mikko Rantalainen (2021-08-30 20:26 Europe/Helsinki):
> Of course one possible solution is to keep "Cached" as is and introduce
> "Cache" with the real cache semantics (that is, it includes sum of
> (Cached - Shmem) and memory backed RAM).

That should have been "swap backed RAM", of course.