Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified image filesystem

[Gao Xiang <hsiangkao@linux.alibaba.com>]

On 2023/1/24 21:10, Alexander Larsson wrote:
> On Tue, 2023-01-24 at 05:24 +0200, Amir Goldstein wrote:
>> On Mon, Jan 23, 2023 at 7:56 PM Alexander Larsson <alexl@redhat.com>

...

>>
>> No it is not overlayfs, it is overlayfs+squashfs, please stick to
>> facts.
>> As Gao wrote, squashfs does not optimize directory lookup.
>> You can run a test with ext4 for POC as Gao suggested.
>> I am sure that mkfs.erofs sparse file support can be added if needed.
> 
> New measurements follow, they now include also erofs over loopback,
> although that isn't strictly fair, because that image is much larger
> due to the fact that it didn't store the files sparsely. It also
> includes a version where the topmost lower is directly on the backing
> xfs (i.e. not via loopback). I attached the scripts used to create the
> images and do the profiling in case anyone wants to reproduce.
> 
> Here are the results (on x86-64, xfs base fs):
> 
> overlayfs + loopback squashfs - uncached
> Benchmark 1: ls -lR mnt-ovl
>    Time (mean ± σ):      2.483 s ±  0.029 s    [User: 0.167 s, System: 1.656 s]
>    Range (min … max):    2.427 s …  2.530 s    10 runs
>   
> overlayfs + loopback squashfs - cached
> Benchmark 1: ls -lR mnt-ovl
>    Time (mean ± σ):     429.2 ms ±   4.6 ms    [User: 123.6 ms, System: 295.0 ms]
>    Range (min … max):   421.2 ms … 435.3 ms    10 runs
>   
> overlayfs + loopback ext4 - uncached
> Benchmark 1: ls -lR mnt-ovl
>    Time (mean ± σ):      4.332 s ±  0.060 s    [User: 0.204 s, System: 3.150 s]
>    Range (min … max):    4.261 s …  4.442 s    10 runs
>   
> overlayfs + loopback ext4 - cached
> Benchmark 1: ls -lR mnt-ovl
>    Time (mean ± σ):     528.3 ms ±   4.0 ms    [User: 143.4 ms, System: 381.2 ms]
>    Range (min … max):   521.1 ms … 536.4 ms    10 runs
>   
> overlayfs + loopback erofs - uncached
> Benchmark 1: ls -lR mnt-ovl
>    Time (mean ± σ):      3.045 s ±  0.127 s    [User: 0.198 s, System: 1.129 s]
>    Range (min … max):    2.926 s …  3.338 s    10 runs
>   
> overlayfs + loopback erofs - cached
> Benchmark 1: ls -lR mnt-ovl
>    Time (mean ± σ):     516.9 ms ±   5.7 ms    [User: 139.4 ms, System: 374.0 ms]
>    Range (min … max):   503.6 ms … 521.9 ms    10 runs
>   
> overlayfs + direct - uncached
> Benchmark 1: ls -lR mnt-ovl
>    Time (mean ± σ):      2.562 s ±  0.028 s    [User: 0.199 s, System: 1.129 s]
>    Range (min … max):    2.497 s …  2.585 s    10 runs
>   
> overlayfs + direct - cached
> Benchmark 1: ls -lR mnt-ovl
>    Time (mean ± σ):     524.5 ms ±   1.6 ms    [User: 148.7 ms, System: 372.2 ms]
>    Range (min … max):   522.8 ms … 527.8 ms    10 runs
>   
> composefs - uncached
> Benchmark 1: ls -lR mnt-fs
>    Time (mean ± σ):     681.4 ms ±  14.1 ms    [User: 154.4 ms, System: 369.9 ms]
>    Range (min … max):   652.5 ms … 703.2 ms    10 runs
>   
> composefs - cached
> Benchmark 1: ls -lR mnt-fs
>    Time (mean ± σ):     390.8 ms ±   4.7 ms    [User: 144.7 ms, System: 243.7 ms]
>    Range (min … max):   382.8 ms … 399.1 ms    10 runs
> 
> For the uncached case, composefs is still almost four times faster than
> the fastest overlay combo (squashfs), and the non-squashfs versions are
> strictly slower. For the cached case the difference is less (10%) but
> with similar order of performance.
> 
> For size comparison, here are the resulting images:
> 
> 8.6M large.composefs
> 2.5G large.erofs
> 200M large.ext4
> 2.6M large.squashfs
Ok, I have to say I'm a bit surprised by these results. Just a wild guess,
`ls -lR` is a seq-like access, so that compressed data (assumed that you
use it) is benefited from it.  I cannot think of a proper cause before
looking into more.  EROFS is impacted since EROFS on-disk inodes are not
arranged together with the current mkfs.erofs implemenetation (it's just
a userspace implementation details, if people really care about it, I
will refine the implementation), and I will also implement such sparse
files later so that all on-disk inodes won't be impacted as well (I'm on
vacation, but I will try my best).

 From the overall results, I don't really know what's the most bottleneck
point honestly:
   maybe just like what you said -- due to overlayfs overhead;
   or maybe a bottleneck of loopback device.

   so it's much better to show some results of "ls -lR" without overlayfs
stacked too.

IMHO, Amir's main point is always [1]
"w.r.t overlayfs, I am not even sure that anything needs to be modified
  in the driver.
  overlayfs already supports "metacopy" feature which means that an upper
  layer could be composed in a way that the file content would be read
  from an arbitrary path in lower fs, e.g. objects/cc/XXX. "

I think there is nothing wrong with it (except for fsverity). From the
results, such functionality indeed can already be achieved by overlayfs
+ some localfs with some user-space adaption. And it was not mentioned
in RFC and v2.

So without fs-verity requirement, currently your proposal is mainly
resolving a performance issue of an exist in-kernel approach (except for
unprivileged mounts).  It's much better to describe in the cover letter
-- The original problem, why overlayfs + (localfs or FUSE for metadata)
doesn't meet the requirements.  That makes much sense compared with the
current cover letter.

Thanks,
Gao Xiang

[1] https://lore.kernel.org/r/CAOQ4uxh34udueT-+Toef6TmTtyLjFUnSJs=882DH=HxADX8pKw@mail.gmail.com/