driverdev-devel.linuxdriverproject.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v6 00/24] erofs: promote erofs from staging
@ 2019-08-02 12:53 Gao Xiang
  2019-08-02 12:53 ` [PATCH v6 01/24] erofs: add on-disk layout Gao Xiang
                   ` (24 more replies)
  0 siblings, 25 replies; 141+ messages in thread
From: Gao Xiang @ 2019-08-02 12:53 UTC (permalink / raw)


Hi folks,

We'd like to hear if there are potential suggestions or
maybe objections of EROFS upstreaming stuffs. If any,
we still have time and we can improve it in this round.

As related materials mentioned before, the goal of EROFS
filesystem is to save extra storage space with guaranteed
end-to-end performance for read-only files, which has better
performance over exist Linux compression filesystems based
on fixed-sized output compression and inplace decompression.
It even has better performance in a large compression ratio
range compared with generic uncompressed filesystems with
proper CPU-storage combinations. And we think this direction
is right and we have a dedicated kernel team working on it,
enough testers and beta / end users using it.

EROFS has been applied to 10+ million HUAWEI mobile phones 
(Yes, the number is still increasing by time) and it seems
like a success. It can be used in more wider scenarios.
We think it's useful for Linux / Android OS community and
it's the time moving out of staging.

EROFS is a self-contained filesystem driver. Although there
are still some TODOs, we will keep on developping / tuning
EROFS with the evolution of Linux kernel as the other
in-kernel filesystems.

Kindly share your comments about EROFS. Thank you in advance!

Thanks,
Gao Xiang

Changelog from v5:
 o keep up with "[PATCH v2] staging: erofs: updates according to erofs-outofstaging v4"
    https://lore.kernel.org/lkml/20190731155752.210602-1-gaoxiang25 at huawei.com/
   which mainly addresses review comments from Chao:
  - keep the marco EROFS_IO_MAX_RETRIES_NOFAIL in internal.h;
  - kill a redundant NULL check in "__stagingpage_alloc";
  - add some descriptions in document about "use_vmap";
  - rearrange erofs_vmap of "staging: erofs: kill CONFIG_EROFS_FS_USE_VM_MAP_RAM";

 o all changes have been merged into staging tree, which are under staging-testing:
    https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/log/?h=staging-testing

It can also be found in git at tag "erofs_2019-08-02" at:
 https://git.kernel.org/pub/scm/linux/kernel/git/xiang/linux.git/

and the latest fs code is available at:
 https://git.kernel.org/pub/scm/linux/kernel/git/xiang/linux.git/tree/fs/erofs?h=erofs-outofstaging

Changelog from v4:
 o rebase on Linus 5.3-rc1;

 o keep up with "staging: erofs: updates according to erofs-outofstaging v4"
   in order to get main code bit-for-bit identical with staging tree:
    https://lore.kernel.org/lkml/20190729065159.62378-1-gaoxiang25 at huawei.com/

Changelog from v3:
 o use GPL-2.0-only for SPDX-License-Identifier suggested by Stephen;

 o kill all kconfig cache strategies and turn them into mount options
   "cache_strategy={disable|readahead|readaround}" suggested by Ted.
   As the first step, cached pages can still be usable after cache is
   disabled by remounting, and these pages will be fallen out over
   time, which can be refined in the later version if some requirement
   is needed. Update related document as well;

 o turn on CONFIG_EROFS_FS_SECURITY by default suggested by David;

 o kill CONFIG_EROFS_FS_IO_MAX_RETRIES and fold it into code; turn
   EROFS_FS_USE_VM_MAP_RAM into a module parameter ("use_vmap")
   suggested by David.

Changelog from v2:
 o kill sbi->dev_name and clean up all failure handling in
   fill_super() suggested by Al.
   Note that the initialzation of managed_cache is now moved
   after s_root is assigned since it's more preferred to iput()
   in .put_super() and all inodes should be evicted before
   the end of generic_shutdown_super(sb);

 o fold in the following staging patches (and thanks):
   staging: erofs:converting all 'unsigned' to 'unsigned int'
   staging: erofs: Remove function erofs_kill_sb()
    - However it was revoked due to erofs_kill_sb reused...
   staging: erofs: avoid opened loop codes
   staging: erofs: support bmap

 o move EROFS_SUPER_MAGIC_V1 from linux/fs/erofs/erofs_fs.h to
   include/uapi/linux/magic.h for userspace utilities.

Changelog from v1:
 o resend the whole filesystem into a patchset suggested by Greg;
 o code is more cleaner, especially for decompression frontend.

--8<----(original cover)----

Hi,

EROFS file system has been in Linux-staging for about a year.
It has been proved to be stable enough to move out of staging
by 10+ millions of HUAWEI Android mobile phones on the market
from EMUI 9.0.1, and it was promoted as one of the key features
of EMUI 9.1 [1], including P30(pro).

EROFS is a read-only file system designed to save extra storage
space with guaranteed end-to-end performance by applying
fixed-size output compression, inplace I/O and decompression
inplace technologies [2] to Linux filesystem.

In our observation, EROFS is one of the fastest Linux compression
filesystem using buffered I/O in the world. It will support
direct I/O in the future if needed. EROFS even has better read
performance in a large CR range compared with generic uncompressed
file systems with proper CPU-storage combination, which is
a reason why EROFS can be landed to speed up mobile phone
performance, and which can be probably used for other use cases
such as LiveCD and Docker image as well.

Currently EROFS supports 4k LZ4 fixed-size output compression
since LZ4 is the fastest widely-used decompression solution in
the world and 4k leads to unnoticable read amplification for
the worst case. More compression algorithms and cluster sizes
could be added later, which depends on the real requirement.

More information about EROFS itself are available at:
 Documentation/filesystems/erofs.txt
 https://kccncosschn19eng.sched.com/event/Nru2/erofs-an-introduction-and-our-smartphone-practice-xiang-gao-huawei

erofs-utils (mainly mkfs.erofs now) is available at
git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git

Preliminary iomap support has been pending in EROFS mailing
list by Chao Yu. The key issue is that current iomap doesn't
support tail-end packing inline data yet, it should be
resolved later.

Thanks to many contributors in the last year, the code is more
clean and improved. We hope EROFS can be used in wider use cases
so let's promote erofs out of staging and enhance it more actively.

Kindly share comments about EROFS! We think EROFS is useful to
community as a part of Linux upstream.

Thank you very much,
Gao Xiang

[1] http://web.archive.org/web/20190627021241/https://consumer.huawei.com/en/emui/
[2] https://lore.kernel.org/lkml/20190624072258.28362-1-hsiangkao at aol.com/

Cc: Greg Kroah-Hartman <gregkh at linuxfoundation.org>
Cc: Alexander Viro <viro at zeniv.linux.org.uk>
Cc: Andrew Morton <akpm at linux-foundation.org>
Cc: Stephen Rothwell <sfr at canb.auug.org.au>
Cc: Theodore Ts'o <tytso at mit.edu>
Cc: Pavel Machek <pavel at denx.de>
Cc: David Sterba <dsterba at suse.cz>
Cc: Amir Goldstein <amir73il at gmail.com>
Cc: Christoph Hellwig <hch at infradead.org>
Cc: Darrick J . Wong <darrick.wong at oracle.com>
Cc: Dave Chinner <david at fromorbit.com>
Cc: Jaegeuk Kim <jaegeuk at kernel.org>
Cc: Jan Kara <jack at suse.cz> 
Cc: Chao Yu <yuchao0 at huawei.com>
Cc: Miao Xie <miaoxie at huawei.com>
Cc: Li Guifu <bluce.liguifu at huawei.com>
Cc: Fang Wei <fangwei1 at huawei.com>
Signed-off-by: Gao Xiang <gaoxiang25 at huawei.com>

Gao Xiang (24):
  erofs: add on-disk layout
  erofs: add erofs in-memory stuffs
  erofs: add super block operations
  erofs: add raw address_space operations
  erofs: add inode operations
  erofs: support special inode
  erofs: add directory operations
  erofs: add namei functions
  erofs: support tracepoint
  erofs: update Kconfig and Makefile
  erofs: introduce xattr & posixacl support
  erofs: introduce tagged pointer
  erofs: add compression indexes support
  erofs: introduce superblock registration
  erofs: introduce erofs shrinker
  erofs: introduce workstation for decompression
  erofs: introduce per-CPU buffers implementation
  erofs: introduce pagevec for decompression subsystem
  erofs: add erofs_allocpage()
  erofs: introduce generic decompression backend
  erofs: introduce LZ4 decompression inplace
  erofs: introduce the decompression frontend
  erofs: introduce cached decompression
  erofs: add document

 Documentation/filesystems/erofs.txt |  225 +++++
 fs/Kconfig                          |    1 +
 fs/Makefile                         |    1 +
 fs/erofs/Kconfig                    |   98 ++
 fs/erofs/Makefile                   |   11 +
 fs/erofs/compress.h                 |   62 ++
 fs/erofs/data.c                     |  429 ++++++++
 fs/erofs/decompressor.c             |  360 +++++++
 fs/erofs/dir.c                      |  147 +++
 fs/erofs/erofs_fs.h                 |  316 ++++++
 fs/erofs/inode.c                    |  326 +++++++
 fs/erofs/internal.h                 |  554 +++++++++++
 fs/erofs/namei.c                    |  251 +++++
 fs/erofs/super.c                    |  666 +++++++++++++
 fs/erofs/tagptr.h                   |  110 +++
 fs/erofs/utils.c                    |  335 +++++++
 fs/erofs/xattr.c                    |  700 +++++++++++++
 fs/erofs/xattr.h                    |   94 ++
 fs/erofs/zdata.c                    | 1405 +++++++++++++++++++++++++++
 fs/erofs/zdata.h                    |  195 ++++
 fs/erofs/zmap.c                     |  462 +++++++++
 fs/erofs/zpvec.h                    |  159 +++
 include/trace/events/erofs.h        |  256 +++++
 include/uapi/linux/magic.h          |    1 +
 24 files changed, 7164 insertions(+)
 create mode 100644 Documentation/filesystems/erofs.txt
 create mode 100644 fs/erofs/Kconfig
 create mode 100644 fs/erofs/Makefile
 create mode 100644 fs/erofs/compress.h
 create mode 100644 fs/erofs/data.c
 create mode 100644 fs/erofs/decompressor.c
 create mode 100644 fs/erofs/dir.c
 create mode 100644 fs/erofs/erofs_fs.h
 create mode 100644 fs/erofs/inode.c
 create mode 100644 fs/erofs/internal.h
 create mode 100644 fs/erofs/namei.c
 create mode 100644 fs/erofs/super.c
 create mode 100644 fs/erofs/tagptr.h
 create mode 100644 fs/erofs/utils.c
 create mode 100644 fs/erofs/xattr.c
 create mode 100644 fs/erofs/xattr.h
 create mode 100644 fs/erofs/zdata.c
 create mode 100644 fs/erofs/zdata.h
 create mode 100644 fs/erofs/zmap.c
 create mode 100644 fs/erofs/zpvec.h
 create mode 100644 include/trace/events/erofs.h

-- 
2.17.1

^ permalink raw reply	[flat|nested] 141+ messages in thread

end of thread, other threads:[~2019-09-05 11:30 UTC | newest]

Thread overview: 141+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-02 12:53 [PATCH v6 00/24] erofs: promote erofs from staging Gao Xiang
2019-08-02 12:53 ` [PATCH v6 01/24] erofs: add on-disk layout Gao Xiang
     [not found]   ` <20190829095954.GB20598@infradead.org>
2019-09-01  7:54     ` Gao Xiang
2019-09-02 12:45       ` Christoph Hellwig
2019-09-02 13:02         ` Gao Xiang
2019-09-02  8:40     ` Pavel Machek
2019-09-02 10:35       ` Gao Xiang
     [not found]     ` <20190829103252.GA64893@architecture4>
     [not found]       ` <67d6efbbc9ac6db23215660cb970b7ef29dc0c1d.camel@perches.com>
     [not found]         ` <20190830120714.GN2752@twin.jikos.cz>
2019-09-02  8:43           ` Pavel Machek
2019-09-02 14:07             ` David Sterba
2019-09-03 11:27               ` Pavel Machek
2019-08-02 12:53 ` [PATCH v6 02/24] erofs: add erofs in-memory stuffs Gao Xiang
2019-08-02 12:53 ` [PATCH v6 03/24] erofs: add super block operations Gao Xiang
     [not found]   ` <20190829101545.GC20598@infradead.org>
     [not found]     ` <20190829105048.GB64893@architecture4>
2019-08-30 16:39       ` Christoph Hellwig
2019-08-30 17:15         ` Gao Xiang
2019-08-31  0:54           ` Gao Xiang
2019-08-31  6:34           ` Amir Goldstein
2019-08-31  6:48             ` Gao Xiang
2019-09-01  8:54     ` Gao Xiang
2019-09-02 12:51       ` Christoph Hellwig
2019-09-02 14:43         ` Gao Xiang
2019-09-02 15:19           ` Christoph Hellwig
2019-09-02 15:24             ` Gao Xiang
2019-08-02 12:53 ` [PATCH v6 04/24] erofs: add raw address_space operations Gao Xiang
     [not found]   ` <20190829101721.GD20598@infradead.org>
     [not found]     ` <20190829114610.GF64893@architecture4>
2019-08-30 16:40       ` Christoph Hellwig
2019-08-30 17:23         ` Gao Xiang
2019-08-02 12:53 ` [PATCH v6 05/24] erofs: add inode operations Gao Xiang
     [not found]   ` <20190829102426.GE20598@infradead.org>
     [not found]     ` <20190829115922.GG64893@architecture4>
2019-08-30 16:42       ` Christoph Hellwig
2019-08-30 18:46         ` Gao Xiang
2019-09-01  9:34     ` Gao Xiang
2019-09-02 12:53       ` Christoph Hellwig
2019-09-02 13:43       ` David Sterba
2019-09-02 13:55         ` Gao Xiang
2019-08-02 12:53 ` [PATCH v6 06/24] erofs: support special inode Gao Xiang
     [not found]   ` <20190829102503.GF20598@infradead.org>
2019-09-01  9:39     ` Gao Xiang
2019-08-02 12:53 ` [PATCH v6 07/24] erofs: add directory operations Gao Xiang
2019-08-02 12:53 ` [PATCH v6 08/24] erofs: add namei functions Gao Xiang
2019-08-02 12:53 ` [PATCH v6 09/24] erofs: support tracepoint Gao Xiang
2019-08-02 12:53 ` [PATCH v6 10/24] erofs: update Kconfig and Makefile Gao Xiang
2019-08-02 12:53 ` [PATCH v6 11/24] erofs: introduce xattr & posixacl support Gao Xiang
2019-08-02 12:53 ` [PATCH v6 12/24] erofs: introduce tagged pointer Gao Xiang
2019-08-02 12:53 ` [PATCH v6 13/24] erofs: add compression indexes support Gao Xiang
2019-08-02 12:53 ` [PATCH v6 14/24] erofs: introduce superblock registration Gao Xiang
2019-08-02 12:53 ` [PATCH v6 15/24] erofs: introduce erofs shrinker Gao Xiang
2019-08-02 12:53 ` [PATCH v6 16/24] erofs: introduce workstation for decompression Gao Xiang
2019-08-02 12:53 ` [PATCH v6 17/24] erofs: introduce per-CPU buffers implementation Gao Xiang
2019-08-02 12:53 ` [PATCH v6 18/24] erofs: introduce pagevec for decompression subsystem Gao Xiang
2019-08-02 12:53 ` [PATCH v6 19/24] erofs: add erofs_allocpage() Gao Xiang
2019-08-02 12:53 ` [PATCH v6 20/24] erofs: introduce generic decompression backend Gao Xiang
2019-08-02 12:53 ` [PATCH v6 21/24] erofs: introduce LZ4 decompression inplace Gao Xiang
2019-08-02 12:53 ` [PATCH v6 22/24] erofs: introduce the decompression frontend Gao Xiang
2019-08-02 12:53 ` [PATCH v6 23/24] erofs: introduce cached decompression Gao Xiang
2019-08-02 12:53 ` [PATCH v6 24/24] erofs: add document Gao Xiang
2019-09-01  5:51 ` [PATCH 00/21] erofs: patchset addressing Christoph's comments Gao Xiang
2019-09-01  5:51   ` [PATCH 01/21] erofs: remove all the byte offset comments Gao Xiang
2019-09-02 12:05     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 02/21] erofs: on-disk format should have explicitly assigned numbers Gao Xiang
2019-09-02 12:05     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 03/21] erofs: some macros are much more readable as a function Gao Xiang
2019-09-02 12:06     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 04/21] erofs: kill __packed for on-disk structures Gao Xiang
2019-09-02 12:06     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 05/21] erofs: update erofs_inode_is_data_compressed helper Gao Xiang
2019-09-02 12:07     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 06/21] erofs: kill erofs_{init,exit}_inode_cache Gao Xiang
2019-09-02 12:09     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 07/21] erofs: use erofs_inode naming Gao Xiang
2019-09-02 12:10     ` Christoph Hellwig
2019-09-02 12:13       ` Gao Xiang
2019-09-02 12:47         ` Christoph Hellwig
2019-09-02 13:33           ` Gao Xiang
2019-09-01  5:51   ` [PATCH 08/21] erofs: update comments in inode.c Gao Xiang
2019-09-01  5:51   ` [PATCH 09/21] erofs: update erofs symlink stuffs Gao Xiang
2019-09-02 12:11     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 10/21] erofs: kill is_inode_layout_compression() Gao Xiang
2019-09-02 12:11     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 11/21] erofs: use dsb instead of layout for ondisk super_block Gao Xiang
2019-09-02 12:12     ` Christoph Hellwig
2019-09-02 12:15       ` Gao Xiang
2019-09-01  5:51   ` [PATCH 12/21] erofs: kill verbose debug info in erofs_fill_super Gao Xiang
2019-09-02 12:14     ` Christoph Hellwig
2019-09-02 12:18       ` Gao Xiang
2019-09-01  5:51   ` [PATCH 13/21] erofs: simplify erofs_grab_bio() since bio_alloc() never fail Gao Xiang
2019-09-02 12:20     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 14/21] erofs: kill prio and nofail of erofs_get_meta_page() Gao Xiang
2019-09-02 12:21     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 15/21] erofs: kill __submit_bio() Gao Xiang
2019-09-01  5:51   ` [PATCH 16/21] erofs: kill magic underscores Gao Xiang
2019-09-02 12:26     ` Christoph Hellwig
2019-09-02 12:39       ` Gao Xiang
2019-09-02 12:54         ` Christoph Hellwig
2019-09-02 13:38           ` Gao Xiang
2019-09-01  5:51   ` [PATCH 17/21] erofs: use a switch statement when dealing with the file modes Gao Xiang
2019-09-02 12:27     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 18/21] erofs: add "erofs_" prefix for common and short functions Gao Xiang
2019-09-02 12:28     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 19/21] erofs: kill all erofs specific fault injection Gao Xiang
2019-09-02 12:28     ` Christoph Hellwig
2019-09-01  5:51   ` [PATCH 20/21] erofs: kill use_vmap module parameter Gao Xiang
2019-09-02 12:31     ` Christoph Hellwig
2019-09-02 12:43       ` Gao Xiang
2019-09-01  5:51   ` [PATCH 21/21] erofs: save one level of indentation Gao Xiang
2019-09-02 12:31     ` Christoph Hellwig
2019-09-02 12:46   ` [PATCH 00/21] erofs: patchset addressing Christoph's comments Christoph Hellwig
2019-09-02 14:24     ` Gao Xiang
2019-09-02 15:23       ` Christoph Hellwig
2019-09-02 15:50         ` Gao Xiang
2019-09-03  6:58           ` Christoph Hellwig
2019-09-03  8:17             ` Gao Xiang
2019-09-03 15:37               ` Christoph Hellwig
2019-09-03 15:43                 ` Gao Xiang
2019-09-04  2:08   ` [PATCH v2 00/25] " Gao Xiang
2019-09-04  2:08     ` [PATCH v2 01/25] erofs: remove all the byte offset comments Gao Xiang
2019-09-04  2:08     ` [PATCH v2 02/25] erofs: on-disk format should have explicitly assigned numbers Gao Xiang
2019-09-04  2:08     ` [PATCH v2 03/25] erofs: some macros are much more readable as a function Gao Xiang
2019-09-04  2:08     ` [PATCH v2 04/25] erofs: kill __packed for on-disk structures Gao Xiang
2019-09-04  2:08     ` [PATCH v2 05/25] erofs: update erofs_inode_is_data_compressed helper Gao Xiang
2019-09-04  2:08     ` [PATCH v2 06/25] erofs: use feature_incompat rather than requirements Gao Xiang
2019-09-04  2:08     ` [PATCH v2 07/25] erofs: better naming for erofs inode related stuffs Gao Xiang
2019-09-04  2:08     ` [PATCH v2 08/25] erofs: kill erofs_{init,exit}_inode_cache Gao Xiang
2019-09-04  2:08     ` [PATCH v2 09/25] erofs: use erofs_inode naming Gao Xiang
2019-09-04  2:08     ` [PATCH v2 10/25] erofs: update erofs_fs.h comments Gao Xiang
2019-09-04  2:08     ` [PATCH v2 11/25] erofs: update comments in inode.c Gao Xiang
2019-09-04  2:08     ` [PATCH v2 12/25] erofs: better erofs symlink stuffs Gao Xiang
2019-09-04  2:09     ` [PATCH v2 13/25] erofs: use dsb instead of layout for ondisk super_block Gao Xiang
2019-09-04  2:09     ` [PATCH v2 14/25] erofs: kill verbose debug info in erofs_fill_super Gao Xiang
2019-09-04  2:09     ` [PATCH v2 15/25] erofs: localize erofs_grab_bio() Gao Xiang
2019-09-04  2:09     ` [PATCH v2 16/25] erofs: kill prio and nofail of erofs_get_meta_page() Gao Xiang
2019-09-04  2:09     ` [PATCH v2 17/25] erofs: kill __submit_bio() Gao Xiang
2019-09-04  2:09     ` [PATCH v2 18/25] erofs: add "erofs_" prefix for common and short functions Gao Xiang
2019-09-04  2:09     ` [PATCH v2 19/25] erofs: kill all erofs specific fault injection Gao Xiang
2019-09-04  2:09     ` [PATCH v2 20/25] erofs: kill use_vmap module parameter Gao Xiang
2019-09-04  2:09     ` [PATCH v2 21/25] erofs: save one level of indentation Gao Xiang
2019-09-04  2:09     ` [PATCH v2 22/25] erofs: rename errln/infoln/debugln to erofs_{err, info, dbg} Gao Xiang
2019-09-04  2:09     ` [PATCH v2 23/25] erofs: use read_mapping_page instead of sb_bread Gao Xiang
2019-09-04  2:09     ` [PATCH v2 24/25] erofs: always use iget5_locked Gao Xiang
2019-09-04  2:09     ` [PATCH v2 25/25] erofs: use read_cache_page_gfp for erofs_get_meta_page Gao Xiang
2019-09-04  3:27     ` [PATCH v2 00/25] erofs: patchset addressing Christoph's comments Chao Yu
2019-09-05  1:03       ` Gao Xiang
2019-09-05 11:30         ` Christoph Hellwig
2019-09-04  5:16     ` Christoph Hellwig
2019-09-04  6:08       ` Gao Xiang

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).