From: Jingbo Xu <jefflexu@linux.alibaba.com>
To: Alexander Larsson <alexl@redhat.com>,
Gao Xiang <hsiangkao@linux.alibaba.com>,
Amir Goldstein <amir73il@gmail.com>,
gscrivan@redhat.com, brauner@kernel.org
Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
david@fromorbit.com, viro@zeniv.linux.org.uk,
Vivek Goyal <vgoyal@redhat.com>,
Miklos Szeredi <miklos@szeredi.hu>
Subject: Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified image filesystem
Date: Thu, 2 Feb 2023 12:57:14 +0800 [thread overview]
Message-ID: <64348226-0be8-4fb4-8f1e-1f118511bdc7@linux.alibaba.com> (raw)
In-Reply-To: <071074ad149b189661681aada453995741f75039.camel@redhat.com>
On 2/1/23 5:46 PM, Alexander Larsson wrote:
> On Wed, 2023-02-01 at 12:28 +0800, Jingbo Xu wrote:
>> Hi all,
>>
>> There are some updated performance statistics with different
>> combinations on my test environment if you are interested.
>>
>>
>> On 1/27/23 6:24 PM, Gao Xiang wrote:
>>> ...
>>>
>>> I've made a version and did some test, it can be fetched from:
>>> git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git
>>> -b
>>> experimental
>>>
>>
>> Setup
>> ======
>> CPU: x86_64 Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
>> Disk: 6800 IOPS upper limit
>> OS: Linux v6.2 (with composefs v3 patchset)
>
> For the record, what was the filesystem backing the basedir files?
>
>> I build erofs/squashfs images following the scripts attached on [1],
>> with each file in the rootfs tagged with "metacopy" and "redirect"
>> xattr.
>>
>> The source rootfs is from the docker image of tensorflow [2].
>>
>> The erofs images are built with mkfs.erofs with support for sparse
>> file
>> added [3].
>>
>> [1]
>> https://lore.kernel.org/linux-fsdevel/5fb32a1297821040edd8c19ce796fc0540101653.camel@redhat.com/
>> [2]
>> https://hub.docker.com/layers/tensorflow/tensorflow/2.10.0/images/sha256-7f9f23ce2473eb52d17fe1b465c79c3a3604047343e23acc036296f512071bc9?context=explore
>> [3]
>> https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/commit/?h=experimental&id=7c49e8b195ad90f6ca9dfccce9f6e3e39a8676f6
>>
>>
>>
>> Image size
>> ===========
>> 6.4M large.composefs
>> 5.7M large.composefs.w/o.digest (w/o --compute-digest)
>> 6.2M large.erofs
>> 5.2M large.erofs.T0 (with -T0, i.e. w/o nanosecond timestamp)
>> 1.7M large.squashfs
>> 5.8M large.squashfs.uncompressed (with -noI -noD -noF -noX)
>>
>> (large.erofs.T0 is built without nanosecond timestamp, so that we get
>> smaller disk inode size (same with squashfs).)
>>
>>
>> Runtime Perf
>> =============
>>
>> The "uncached" column is tested with:
>> hyperfine -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR $MNTPOINT"
>>
>>
>> While the "cached" column is tested with:
>> hyperfine -w 1 "ls -lR $MNTPOINT"
>>
>>
>> erofs and squashfs are mounted with loopback device.
>>
>>
>> | uncached(ms)| cached(ms)
>> ----------------------------------|-------------|-----------
>> composefs (with digest) | 326 | 135
>> erofs (w/o -T0) | 264 | 172
>> erofs (w/o -T0) + overlayfs | 651 | 238
>> squashfs (compressed) | 538 | 211
>> squashfs (compressed) + overlayfs | 968 | 302
>
>
> Clearly erofs with sparse files is the best fs now for the ro-fs +
> overlay case. But still, we can see that the additional cost of the
> overlayfs layer is not negligible.
>
> According to amir this could be helped by a special composefs-like mode
> in overlayfs, but its unclear what performance that would reach, and
> we're then talking net new development that further complicates the
> overlayfs codebase. Its not clear to me which alternative is easier to
> develop/maintain.
>
> Also, the difference between cached and uncached here is less than in
> my tests. Probably because my test image was larger. With the test
> image I use, the results are:
>
> | uncached(ms)| cached(ms)
> ----------------------------------|-------------|-----------
> composefs (with digest) | 681 | 390
> erofs (w/o -T0) + overlayfs | 1788 | 532
> squashfs (compressed) + overlayfs | 2547 | 443
>
>
> I gotta say it is weird though that squashfs performed better than
> erofs in the cached case. May be worth looking into. The test data I'm
> using is available here:
>
> https://my.owndrive.com/index.php/s/irHJXRpZHtT3a5i
>
>
Hi,
I also tested upon the rootfs you given.
Setup
======
CPU: x86_64 Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz
Disk: 11800 IOPS upper limit
OS: Linux v6.2 (with composefs v3 patchset)
FS of backing objects: xfs
Image size
===========
8.6M large.composefs (with --compute-digest)
7.6M large.composefs.wo.digest (w/o --compute-digest)
8.9M large.erofs
7.4M large.erofs.T0 (with -T0, i.e. w/o nanosecond timestamp)
2.6M large.squashfs.compressed
8.2M large.squashfs.uncompressed (with -noI -noD -noF -noX)
Runtime Perf
=============
The "uncached" column is tested with:
hyperfine -p "echo 3 > /proc/sys/vm/drop_caches" "ls -lR $MNTPOINT"
While the "cached" column is tested with:
hyperfine -w 1 "ls -lR $MNTPOINT"
erofs and squashfs are mounted with loopback device.
| uncached(ms)| cached(ms)
----------------------------------|-------------|-----------
composefs | 408 | 176
erofs | 308 | 190
erofs + overlayfs | 1097 | 294
erofs.hack | 298 | 187
erofs.hack + overlayfs | 524 | 283
squashfs (compressed) | 770 | 265
squashfs (compressed) + overlayfs | 1600 | 372
squashfs (uncompressed) | 646 | 223
squashfs (uncompressed)+overlayfs | 1480 | 330
- all erofs mounted with "noacl"
- composefs: using large.composefs
- erofs: using large.erofs
- erofs.hack: using large.erofs.hack where each file in the erofs layer
redirecting to the same lower block, e.g.
"/objects/00/02bef8682cac782594e542d1ec6e031b9f7ac40edcfa6a1eb6d15d3b1ab126",
to evaluate the potential optimization of composefs like "lazy lookup"
in overlayfs
- squashfs (compressed): using large.squashfs.compressed
- squashfs (uncompressed): using large.squashfs.uncompressed
--
Thanks,
Jingbo
next prev parent reply other threads:[~2023-02-02 4:57 UTC|newest]
Thread overview: 80+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-20 15:23 [PATCH v3 0/6] Composefs: an opportunistically sharing verified image filesystem Alexander Larsson
2023-01-20 15:23 ` [PATCH v3 1/6] fsverity: Export fsverity_get_digest Alexander Larsson
2023-01-20 15:23 ` [PATCH v3 2/6] composefs: Add on-disk layout header Alexander Larsson
2023-01-20 15:23 ` [PATCH v3 3/6] composefs: Add descriptor parsing code Alexander Larsson
2023-01-20 15:23 ` [PATCH v3 4/6] composefs: Add filesystem implementation Alexander Larsson
2023-01-20 15:23 ` [PATCH v3 5/6] composefs: Add documentation Alexander Larsson
2023-01-21 2:19 ` Bagas Sanjaya
2023-01-20 15:23 ` [PATCH v3 6/6] composefs: Add kconfig and build support Alexander Larsson
2023-01-20 19:44 ` [PATCH v3 0/6] Composefs: an opportunistically sharing verified image filesystem Amir Goldstein
2023-01-20 22:18 ` Giuseppe Scrivano
2023-01-21 3:08 ` Gao Xiang
2023-01-21 16:19 ` Giuseppe Scrivano
2023-01-21 17:15 ` Gao Xiang
2023-01-21 22:34 ` Giuseppe Scrivano
2023-01-22 0:39 ` Gao Xiang
2023-01-22 9:01 ` Giuseppe Scrivano
2023-01-22 9:32 ` Giuseppe Scrivano
2023-01-24 0:08 ` Gao Xiang
2023-01-21 10:57 ` Amir Goldstein
2023-01-21 15:01 ` Giuseppe Scrivano
2023-01-21 15:54 ` Amir Goldstein
2023-01-21 16:26 ` Gao Xiang
2023-01-23 17:56 ` Alexander Larsson
2023-01-23 23:59 ` Gao Xiang
2023-01-24 3:24 ` Amir Goldstein
2023-01-24 13:10 ` Alexander Larsson
2023-01-24 14:40 ` Gao Xiang
2023-01-24 19:06 ` Amir Goldstein
2023-01-25 4:18 ` Dave Chinner
2023-01-25 8:32 ` Amir Goldstein
2023-01-25 10:08 ` Alexander Larsson
2023-01-25 10:43 ` Amir Goldstein
2023-01-25 10:39 ` Giuseppe Scrivano
2023-01-25 11:17 ` Amir Goldstein
2023-01-25 12:30 ` Giuseppe Scrivano
2023-01-25 12:46 ` Amir Goldstein
2023-01-25 13:10 ` Giuseppe Scrivano
2023-01-25 18:07 ` Amir Goldstein
2023-01-25 19:45 ` Giuseppe Scrivano
2023-01-25 20:23 ` Amir Goldstein
2023-01-25 20:29 ` Amir Goldstein
2023-01-27 15:57 ` Vivek Goyal
2023-01-25 15:24 ` Christian Brauner
2023-01-25 16:05 ` Giuseppe Scrivano
2023-01-25 9:37 ` Alexander Larsson
2023-01-25 10:05 ` Gao Xiang
2023-01-25 10:15 ` Alexander Larsson
2023-01-27 10:24 ` Gao Xiang
2023-02-01 4:28 ` Jingbo Xu
2023-02-01 7:44 ` Amir Goldstein
2023-02-01 8:59 ` Jingbo Xu
2023-02-01 9:52 ` Alexander Larsson
2023-02-01 12:39 ` Jingbo Xu
2023-02-01 9:46 ` Alexander Larsson
2023-02-01 10:01 ` Gao Xiang
2023-02-01 11:22 ` Gao Xiang
2023-02-02 6:37 ` Amir Goldstein
2023-02-02 7:17 ` Gao Xiang
2023-02-02 7:37 ` Gao Xiang
2023-02-03 11:32 ` Alexander Larsson
2023-02-03 12:46 ` Amir Goldstein
2023-02-03 15:09 ` Gao Xiang
2023-02-05 19:06 ` Amir Goldstein
2023-02-06 7:59 ` Amir Goldstein
2023-02-06 10:35 ` Miklos Szeredi
2023-02-06 13:30 ` Amir Goldstein
2023-02-06 16:34 ` Miklos Szeredi
2023-02-06 17:16 ` Amir Goldstein
2023-02-06 18:17 ` Amir Goldstein
2023-02-06 19:32 ` Miklos Szeredi
2023-02-06 20:06 ` Amir Goldstein
2023-02-07 8:12 ` Alexander Larsson
2023-02-06 12:51 ` Alexander Larsson
2023-02-07 8:12 ` Jingbo Xu
2023-02-06 12:43 ` Alexander Larsson
2023-02-06 13:27 ` Gao Xiang
2023-02-06 15:31 ` Alexander Larsson
2023-02-01 12:06 ` Jingbo Xu
2023-02-02 4:57 ` Jingbo Xu [this message]
2023-02-02 4:59 ` Jingbo Xu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=64348226-0be8-4fb4-8f1e-1f118511bdc7@linux.alibaba.com \
--to=jefflexu@linux.alibaba.com \
--cc=alexl@redhat.com \
--cc=amir73il@gmail.com \
--cc=brauner@kernel.org \
--cc=david@fromorbit.com \
--cc=gscrivan@redhat.com \
--cc=hsiangkao@linux.alibaba.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=miklos@szeredi.hu \
--cc=vgoyal@redhat.com \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).