From: Gao Xiang <gaoxiang25@huawei.com> To: <lsf-pc@lists.linux-foundation.org> Cc: <linux-fsdevel@vger.kernel.org>, "linux-erofs@lists.ozlabs.org" <linux-erofs@lists.ozlabs.org>, Chao Yu <yuchao0@huawei.com>, Miao Xie <miaoxie@huawei.com>, <lizefan@huawei.com>, "fangwei (I)" <fangwei1@huawei.com>, "liguifu (C)" <bluce.liguifu@huawei.com>, "Duwei (Device OS)" <weidu.du@huawei.com>, "Wangzhigang (Brooke)" <brooke.wangzhigang@hisilicon.com> Subject: [LSF/MM TOPIC] EROFS, our smartphone practice and the next Date: Wed, 23 Jan 2019 14:55:32 +0800 [thread overview] Message-ID: <f44b1696-2f73-3637-9964-d73e3d5832b7@huawei.com> (raw) Hi, Our team would like to give a topic about EROFS real practice in our HUAWEI smartphone in the past 2018 year. Actually, we attempted to apply data compression technology on our consumer electronics with very limited memory, but to provide high performance in order to keep great user experience. We tried squashfs 3 years ago with no luck since it causes serious memory issue on our internal heavy workload and internal beta test (ex, Camera opening), it still behaves poor after turning down its blocksize to reduce its read amplification. After some research on squashfs, we found that its fundamental on-disk format could be better and more flexable since its (meta)data cannot be designed inline and its directory structure cannot perform well in random name lookup. And we also observed some efforts to tune squashfs from Google and they ran into a stone wall [1]. Therefore, instead of keeping the historical burden, we planned to design a more lightweight filesystem than squashfs with modern features and high performance to build better products and serve our consumers, EROFS. Actually EROFS is designed as a better filesystem solution for the following scenarios: - read-only storage media or - part of a fully trusted read-only solution, which means it needs to be immutable and bit-for-bit identical to the official golden image for their releases due to security and other considerations and - hope to save some extra storage space with guaranteed end-to-end performance by using reduced metadata and transparent file compression, especially for those embedded devices with limited memory (ex, smartphone); and the document (first release) is also available at [2]. A rather simple on-disk format, fixed-output compression and in-place decompression are introduced in EROFS by design. In this topic, we would like to share the issue and experience of real-time decompression in these consumer embeded devices, EROFS detailed design, benchmark, testdata when landing EROFS to our smartphones, and comparsion with exist squashfs, compressed btrfs, etc... These all can be shown months after. We also would like to discuss our EROFS future roadmap including but not limited to: - Decompression tuning; - Data encryption (used as a part of tamper resistance); - Data deduplication; - More compression algorithms other than lz4; - (Selectable) bootloader & FUSE support only for compatibility and other OSs; - ... and HUAWEI kernel storage team will keep actively contributing and expend it to more widely scenarios. We also hope that more people could join us to make EROFS better since there are still a lot of stuffs to do. Thanks, [1] https://lkml.org/lkml/2017/9/22/814 [2] https://lists.ozlabs.org/pipermail/linux-erofs/2019-January/001242.html
WARNING: multiple messages have this Message-ID (diff)
From: gaoxiang25@huawei.com (Gao Xiang) Subject: [LSF/MM TOPIC] EROFS, our smartphone practice and the next Date: Wed, 23 Jan 2019 14:55:32 +0800 [thread overview] Message-ID: <f44b1696-2f73-3637-9964-d73e3d5832b7@huawei.com> (raw) Hi, Our team would like to give a topic about EROFS real practice in our HUAWEI smartphone in the past 2018 year. Actually, we attempted to apply data compression technology on our consumer electronics with very limited memory, but to provide high performance in order to keep great user experience. We tried squashfs 3 years ago with no luck since it causes serious memory issue on our internal heavy workload and internal beta test (ex, Camera opening), it still behaves poor after turning down its blocksize to reduce its read amplification. After some research on squashfs, we found that its fundamental on-disk format could be better and more flexable since its (meta)data cannot be designed inline and its directory structure cannot perform well in random name lookup. And we also observed some efforts to tune squashfs from Google and they ran into a stone wall [1]. Therefore, instead of keeping the historical burden, we planned to design a more lightweight filesystem than squashfs with modern features and high performance to build better products and serve our consumers, EROFS. Actually EROFS is designed as a better filesystem solution for the following scenarios: - read-only storage media or - part of a fully trusted read-only solution, which means it needs to be immutable and bit-for-bit identical to the official golden image for their releases due to security and other considerations and - hope to save some extra storage space with guaranteed end-to-end performance by using reduced metadata and transparent file compression, especially for those embedded devices with limited memory (ex, smartphone); and the document (first release) is also available at [2]. A rather simple on-disk format, fixed-output compression and in-place decompression are introduced in EROFS by design. In this topic, we would like to share the issue and experience of real-time decompression in these consumer embeded devices, EROFS detailed design, benchmark, testdata when landing EROFS to our smartphones, and comparsion with exist squashfs, compressed btrfs, etc... These all can be shown months after. We also would like to discuss our EROFS future roadmap including but not limited to: - Decompression tuning; - Data encryption (used as a part of tamper resistance); - Data deduplication; - More compression algorithms other than lz4; - (Selectable) bootloader & FUSE support only for compatibility and other OSs; - ... and HUAWEI kernel storage team will keep actively contributing and expend it to more widely scenarios. We also hope that more people could join us to make EROFS better since there are still a lot of stuffs to do. Thanks, [1] https://lkml.org/lkml/2017/9/22/814 [2] https://lists.ozlabs.org/pipermail/linux-erofs/2019-January/001242.html
next reply other threads:[~2019-01-23 6:55 UTC|newest] Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-01-23 6:55 Gao Xiang [this message] 2019-01-23 6:55 ` [LSF/MM TOPIC] EROFS, our smartphone practice and the next Gao Xiang
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=f44b1696-2f73-3637-9964-d73e3d5832b7@huawei.com \ --to=gaoxiang25@huawei.com \ --cc=bluce.liguifu@huawei.com \ --cc=brooke.wangzhigang@hisilicon.com \ --cc=fangwei1@huawei.com \ --cc=linux-erofs@lists.ozlabs.org \ --cc=linux-fsdevel@vger.kernel.org \ --cc=lizefan@huawei.com \ --cc=lsf-pc@lists.linux-foundation.org \ --cc=miaoxie@huawei.com \ --cc=weidu.du@huawei.com \ --cc=yuchao0@huawei.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.