Is anonymous memory part of the page cache on Linux?

Is anonymous memory part of the page cache on Linux?

[Peter Weber]

Hello!
Hopefully I'm asking here in the right place and don't disturb.

Is anonymous memory - i.e. program heap and stack - part of the page 
cache on Linux? The documentation[1] of the kernel does not state that. 
But the Wikipedia entry about page cache contains a graphic[2] (look at 
the top right) which gives me the impression that 'malloc()' allocates 
dynamic memory within the page cache.
Is it possible that the page cache serves as general foundation for 
memory management on Linux? Taking into account that `mmap()` also 
allows for anonymous mappings this could fit together?

I've asked this originally on Stackoverflow[3].

Thank you
Peter

[1] 
https://www.kernel.org/doc/html/latest/admin-guide/mm/concepts.html#anonymous-memory
[2] 
https://en.wikipedia.org/wiki/Page_cache#/media/File:The_Linux_Storage_Stack_Diagram.svg
[3] https://stackoverflow.com/q/66587345/1054324

Re: Is anonymous memory part of the page cache on Linux?

[Matthew Wilcox]

On Fri, Mar 12, 2021 at 03:43:33PM +0100, Peter Weber wrote:
> Hello!
> Hopefully I'm asking here in the right place and don't disturb.
> 
> Is anonymous memory - i.e. program heap and stack - part of the page cache
> on Linux? The documentation[1] of the kernel does not state that. But the
> Wikipedia entry about page cache contains a graphic[2] (look at the top
> right) which gives me the impression that 'malloc()' allocates dynamic
> memory within the page cache.

The wikipedia diagram is wrong.  Anonymous memory is not handled by the
page cache.  Anonymous pages enter the storage stack via swap; they are
found in the page tables, sent to the swap cache and then written to
swap devices or swap files.  Filesystems may get involved at that point,
but not always.

There are other weird things in the wikipedia diagram, like Direct I/O
being seemingly detached from applications, and not appearing to pass
through the VFS.

Re: Is anonymous memory part of the page cache on Linux?

[Peter Weber]

Thank you Matthew!


Am 2021-03-12 16:15, schrieb Matthew Wilcox:
> The wikipedia diagram is wrong.  Anonymous memory is not handled by the
> page cache.

Is it roughly right to say, that the virtual memory uses page tables to 
handle anonymous memory?

> Anonymous pages enter the storage stack via swap; they are
> found in the page tables, sent to the swap cache and then written to
> swap devices or swap files.  Filesystems may get involved at that 
> point,
> but not always.

And the page cache doesn't handle anonymous memory - even when it is 
swapped?

> There are other weird things in the wikipedia diagram, like Direct I/O
> being seemingly detached from applications, and not appearing to pass
> through the VFS.

I'm not an expert. Maybe I should add at least note about that on 
Wikipedia?

Re: Is anonymous memory part of the page cache on Linux?

[Matthew Wilcox]

On Fri, Mar 12, 2021 at 04:41:55PM +0100, Peter Weber wrote:
> Thank you Matthew!
> 
> 
> Am 2021-03-12 16:15, schrieb Matthew Wilcox:
> > The wikipedia diagram is wrong.  Anonymous memory is not handled by the
> > page cache.
> 
> Is it roughly right to say, that the virtual memory uses page tables to
> handle anonymous memory?

I tend to deal with page cache and not anonymous memory, so I'm not sure
exactly how someone who's an expert would phrase it.  Anonymous pages are
"handled" in a number of different ways -- they can be found on LRU lists
and they can be found through the page tables.  It's all a bit ad-hoc as
far as I can tell ;-)

> > Anonymous pages enter the storage stack via swap; they are
> > found in the page tables, sent to the swap cache and then written to
> > swap devices or swap files.  Filesystems may get involved at that point,
> > but not always.
> 
> And the page cache doesn't handle anonymous memory - even when it is
> swapped?

Right.  There's a swap cache, but that's not the same thing as the page
cache.

> > There are other weird things in the wikipedia diagram, like Direct I/O
> > being seemingly detached from applications, and not appearing to pass
> > through the VFS.
> 
> I'm not an expert. Maybe I should add at least note about that on Wikipedia?

Maybe!

Re: Is anonymous memory part of the page cache on Linux?

[David Hildenbrand]

On 12.03.21 23:45, Matthew Wilcox wrote:
> On Fri, Mar 12, 2021 at 04:41:55PM +0100, Peter Weber wrote:
>> Thank you Matthew!
>>
>>
>> Am 2021-03-12 16:15, schrieb Matthew Wilcox:
>>> The wikipedia diagram is wrong.  Anonymous memory is not handled by the
>>> page cache.
>>
>> Is it roughly right to say, that the virtual memory uses page tables to
>> handle anonymous memory?
> 

You'll have to distinguish between private and shared anonymous memory.


"private anonymous memory" -- mmap(MAP_ANONYMOUS | MAP_PRIVATE) -- is 
usually completely managed using the process page tables. The only way 
to get such pages mapped into another process is via fork(), whereby 
page table are copied and COW (Copy On Write) applies.

Once dereferenced from all page tables, there are no other references 
anymore; memory is handed back to the buddy as free memory, from where 
it can be reused for other purposes. Of course, there are case where 
there might be other references being taken (i.e., GUP, direct I/O ...), 
but let's ignore that for now.

The essence for "private anonymous memory" is, that there is no other 
way to get access to that memory besides the page tables. If you 
MADV_DONTNEED virtual memory regions to zap the page table entries and 
later reaccess the same virtual memory locations, always fresh memory 
will be populated.


"shared anonymous memory" -- mmap(MAP_ANONYMOUS | MAP_SHARED) -- is a 
little different and involves the page cache (it's pretty much shmem 
without a user-visible fd). The only way to get such pages mapped into 
another process is similarly via fork() (AFAIU), whereby page table are 
copied and COW does _not_ apply.

Once dereferenced from all page tables, there is still a reference to 
the page from the page cache. Only once evicted from the page cache, the 
memory can be freed up.

The essence for "shared anonymous memory" is, that there are ways to get 
access to that memory besides the page tables. If you MADV_DONTNEED such 
virtual memory locations to zap the page table entries and later 
reaccess the same virtual memory location again, the previous page will 
be re-instantiated from the page cache. To evict the pages from the page 
cache you would need MADV_REMOVE.

-- 
Thanks,

David / dhildenb

Re: Is anonymous memory part of the page cache on Linux?

[Wxz76]

[-- Attachment #1: Type: text/plain, Size: 2050 bytes --]

Hi Matthew and Peter,

I had a few questions to clarify my understanding of the page/swap cache.

> There's a swap cache, but that's not the same thing as the page cache.

My understanding of the swap cache comes from:

1) Understanding the Linux Kernel by Bovet and Cesati:

“The swap cache is implemented by the page cache data structures and procedures ” and

“Pages in the swap cache are stored as every other page in the page cache, with the following special treatment:

• The mapping field of the page descriptor is set to NULL.

• The PG_swapcache flag of the page descriptor is set.

• The private field stores the swapped-out page identifier associated with the page”

2) Understanding the Linux Virtual Memory Manager by Mel Gorman:

“The swap cache is purely conceptual because it is simply a specialization of the page cache. The first principal difference between pages in the swap cache rather than the page cache is that pages in the swap

cache always use swapper space as their address space in page→mapping. The second difference is that pages are added to the swap cache with add to swap cache(), shown in Figure 11.3, instead of add to page cache().”

I understand that those books are more than ten years old, but is what they write no longer the case?

Is the swap cache mechanism not a specialization of the page cache, and, if not, how are they different?

> Anonymous memory is not handled by the page cache.

> Anonymous pages enter the storage stack via swap; they are

> found in the page tables, sent to the swap cache and then written to

> swap devices or swap files.

This is for the case of swapping out anonymous memory, but what about anonymous memory that is allocated dynamically with malloc/mmap: where is this memory coming from?

When mmap opens files, it maps a process address spaces to a region in the page cache for the file, does it not?

Is this behavior not the same for allocating anonymous memory (minus dealing with a file)?

I appreciate the help in clarifying this for me.

wxz

[-- Attachment #2: Type: text/html, Size: 4843 bytes --]

Re: Is anonymous memory part of the page cache on Linux?

[Matthew Wilcox]

On Sun, Mar 14, 2021 at 11:06:12PM +0000, Wxz76@protonmail.com wrote:
> Hi Matthew and Peter,
> 
> I had a few questions to clarify my understanding of the page/swap cache.

These are good questions to ask.  I see the confusion, and I'm not
entirely sure how to sort it out for you, but let me try.

> > There's a swap cache, but that's not the same thing as the page cache.
> 
> My understanding of the swap cache comes from:
> 
> 1) Understanding the Linux Kernel by Bovet and Cesati:
> 
> “The swap cache is implemented by the page cache data structures and procedures ” and
> 
> “Pages in the swap cache are stored as every other page in the page cache, with the following special treatment:
> 
> • The mapping field of the page descriptor is set to NULL.
> 
> • The PG_swapcache flag of the page descriptor is set.
> 
> • The private field stores the swapped-out page identifier associated with the page”

They're not wrong, but may be misleading.  The swap cache reuses
(repurposes?)  many of the same data structures used by the page cache.
In particular the address_space.  It's still considered to be separate
from the page cache.

> 2) Understanding the Linux Virtual Memory Manager by Mel Gorman:
> 
> “The swap cache is purely conceptual because it is simply a specialization of the page cache. The first principal difference between pages in the swap cache rather than the page cache is that pages in the swap
> 
> cache always use swapper space as their address space in page→mapping. The second difference is that pages are added to the swap cache with add to swap cache(), shown in Figure 11.3, instead of add to page cache().”
> 
> I understand that those books are more than ten years old, but is what they write no longer the case?

That one, being a little more specific, is now a little more out of date.

> Is the swap cache mechanism not a specialization of the page cache, and, if not, how are they different?
> 
> > Anonymous memory is not handled by the page cache.
> 
> > Anonymous pages enter the storage stack via swap; they are
> 
> > found in the page tables, sent to the swap cache and then written to
> 
> > swap devices or swap files.
> 
> This is for the case of swapping out anonymous memory, but what about anonymous memory that is allocated dynamically with malloc/mmap: where is this memory coming from?

When the kernel needs to allocate a page for stack or malloc (whether it
be implemented through MAP_PRIVATE or brk()), it gets it from its pool of
free pages.  It sets up the process's page tables to refer to that page,
and it adds the page to the LRU list (so it can be swapped out if the
pool of free pages runs low).

> When mmap opens files, it maps a process address spaces to a region in the page cache for the file, does it not?

Yes.  This is the vm_area_struct which records the mapping from the
process address spce to the region of the file.

> Is this behavior not the same for allocating anonymous memory (minus dealing with a file)?

It's also a vm_area_struct, but several of the fields in the
vm_area_struct are used differently by an anonymous mapping than they
are by a file-based mapping.

The particularly interesting case that you didn't ask about is what
happens for mmap(MAP_PRIVATE) of a file.  In that case, we set up for
a file-based mapping, but on a write fault, we allocate a new page,
copy from the page cache into the new page, set up the process page
table to point to this new page and put this new page into the LRU list,
so it can be swapped out if needed.

> I appreciate the help in clarifying this for me.

You're welcome!  I hope this is useful.