Memory not shared for files opened from different subvolumes

Memory not shared for files opened from different subvolumes

[Saikrishna Arcot]

Hi,

I was looking at the page map allocations for some process (mainly to 
see why a certain application was taking so much memory), and noticed 
that for some files that were mapped into memory (such as shared 
libraries), the share count for those memory pages weren't as high as I 
expected it to be, or they were not shared at all. For example, I have 
Docker running with the btrfs storage driver instead of the default 
overlay2 storage driver. With multiple instances of a container running 
at the same time with the same processes, the libraries that they had 
open were using their own physical memory pages, instead of sharing the 
physical memory pages (since it's the same file anyways).

As a smaller example, I have two subvolume snapshots of my root 
partition, and if I mount both of them and mmap a file from both of them 
that is the same in both of the subvolume snapshots, then the physical 
memory pages will be different. IIRC, with Docker using the overlay2 
storage driver, the physical memory pages were indeed being shared for 
the same file being mapped into memory.

Here's a small test case showing two subvolume snapshots being mounted, 
a small file that is the same in both snapshots being picked, and the 
physical pages used for the files being different (page mapping table 
was read from /proc/self/pagemap).

$ sha1sum mount/subvol{1,2}/lib/x86_64-linux-gnu/libdl.so.2
e483a364675ffaee4ebace1e116aca0db322a7d5 
mount/subvol1/lib/x86_64-linux-gnu/libdl.so.2
e483a364675ffaee4ebace1e116aca0db322a7d5 
mount/subvol2/lib/x86_64-linux-gnu/libdl.so.2
$ stat mount/subvol{1,2}/lib/x86_64-linux-gnu/libdl.so.2
   File: mount/subvol1/lib/x86_64-linux-gnu/libdl.so.2
   Size: 14432           Blocks: 32         IO Block: 4096   regular file
Device: 77h/119d        Inode: 77161762    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-05-21 11:55:04.530135877 -0700
Modify: 2022-03-03 18:54:17.000000000 -0800
Change: 2022-05-21 11:55:04.494136093 -0700
  Birth: 2022-05-21 11:55:04.418136551 -0700
   File: mount/subvol2/lib/x86_64-linux-gnu/libdl.so.2
   Size: 14432           Blocks: 32         IO Block: 4096   regular file
Device: 78h/120d        Inode: 77161762    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-05-21 11:55:04.530135877 -0700
Modify: 2022-03-03 18:54:17.000000000 -0800
Change: 2022-05-21 11:55:04.494136093 -0700
  Birth: 2022-05-21 11:55:04.418136551 -0700
$ sudo ./a.out mount/subvol{1,2}/lib/x86_64-linux-gnu/libdl.so.2
File 1:
0x00007fdeb50cb000: 0xa1000000006eda02
0x00007fdeb50cc000: 0xa100000000795de1
0x00007fdeb50cd000: 0xa10000000013867b
0x00007fdeb50ce000: 0xa1000000004e4a58

File 2:
0x00007fdeb50c7000: 0xa10000000036ad44
0x00007fdeb50c8000: 0xa1000000003b08e0
0x00007fdeb50c9000: 0xa100000000143c7e
0x00007fdeb50ca000: 0xa1000000001c3d45

This was on the 5.17.5 kernel, but I've observed this on 5.13 as well.

My question is, is it possible to share the physical pages for the same 
file from two different mount points being opened? This would reduce the 
physical memory used in larger cases.

-- 
Saikrishna Arcot

Re: Memory not shared for files opened from different subvolumes

[Holger Hoffstätte]

On 2022-06-23 23:36, Saikrishna Arcot wrote:

[snip]

> My question is, is it possible to share the physical pages for the
> same file from two different mount points being opened? This would
> reduce the physical memory used in larger cases.

Not an immediate solution but more background info on the problem and
some ongoing research: https://lwn.net/Articles/895907/
Hope this helps.

-h

Re: Memory not shared for files opened from different subvolumes

[Saikrishna Arcot]

On 6/24/22 09:03, Holger Hoffstätte wrote:
> On 2022-06-23 23:36, Saikrishna Arcot wrote:
> 
> [snip]
> 
>> My question is, is it possible to share the physical pages for the
>> same file from two different mount points being opened? This would
>> reduce the physical memory used in larger cases.
> 
> Not an immediate solution but more background info on the problem and
> some ongoing research: https://lwn.net/Articles/895907/
> Hope this helps.
> 
> -h

Thanks, that's extremely helpful! I was primarily wondering if this was 
being looked at, and I'm glad to see there's some work being done.

-- 
Saikrishna Arcot