* [PATCH 1/2] erofs-utils: drop known issue in README [not found] <20201116125527.3644-1-hsiangkao.ref@aol.com> @ 2020-11-16 12:55 ` Gao Xiang via Linux-erofs 2020-11-16 12:55 ` [PATCH 2/2] erofs-utils: update README Gao Xiang via Linux-erofs 2020-11-21 6:31 ` [PATCH 1/2] erofs-utils: drop known issue in README Li GuiFu via Linux-erofs 0 siblings, 2 replies; 4+ messages in thread From: Gao Xiang via Linux-erofs @ 2020-11-16 12:55 UTC (permalink / raw) To: linux-erofs From: Gao Xiang <hsiangkao@aol.com> Since lz4-1.9.3 has been released, https://github.com/lz4/lz4/releases/tag/v1.9.3 Move this lz4hc issue (lz4 <= 1.8.2) to "Comments" instead. Signed-off-by: Gao Xiang <hsiangkao@aol.com> --- README | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/README b/README index 5addd6b80e04..7a7ac5d5cb6f 100644 --- a/README +++ b/README @@ -19,7 +19,7 @@ It can create 2 primary kinds of erofs images: (un)compressed. Dependencies ~~~~~~~~~~~~ - lz4-1.8.0+ for lz4 enabled [2], lz4-1.9.0+ recommended + lz4-1.8.0+ for lz4 enabled [2], lz4-1.9.3+ recommended [4]. How to build for lz4-1.9.0 or above ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -48,7 +48,7 @@ On Fedora, static lz4 can be installed using: yum install lz4-static.x86_64 However, it's not recommended to use those versions since there were bugs -in these compressors, see [2] [3] as well. +in these compressors, see [2] [3] [4] as well. How to generate erofs images ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -70,19 +70,6 @@ add "-E legacy-compress" to the command line, e.g. $ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/ - -Known issues -~~~~~~~~~~~~ - -1. LZ4HC cannot compress long zeroed buffer properly with - LZ4_compress_HC_destSize() - https://github.com/lz4/lz4/issues/784 - - which has been resolved in - https://github.com/lz4/lz4/commit/e7fe105ac6ed02019d34731d2ba3aceb11b51bb1 - - and will be included in lz4-1.9.3 if all goes well. - Obsoleted erofs.mkfs ~~~~~~~~~~~~~~~~~~~~ @@ -149,3 +136,13 @@ Comments version as well) or backport all stable bugfixes to old stable versions, e.g. our unoffical lz4 fork: https://github.com/erofs/lz4 +[4] LZ4HC didn't compress long zeroed buffer properly with + LZ4_compress_HC_destSize() + https://github.com/lz4/lz4/issues/784 + + which has been resolved in + https://github.com/lz4/lz4/commit/e7fe105ac6ed02019d34731d2ba3aceb11b51bb1 + + and already included in lz4-1.9.3, see: + https://github.com/lz4/lz4/releases/tag/v1.9.3 + -- 2.24.0 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH 2/2] erofs-utils: update README 2020-11-16 12:55 ` [PATCH 1/2] erofs-utils: drop known issue in README Gao Xiang via Linux-erofs @ 2020-11-16 12:55 ` Gao Xiang via Linux-erofs 2020-11-16 14:51 ` [PATCH v2 " Gao Xiang via Linux-erofs 2020-11-21 6:31 ` [PATCH 1/2] erofs-utils: drop known issue in README Li GuiFu via Linux-erofs 1 sibling, 1 reply; 4+ messages in thread From: Gao Xiang via Linux-erofs @ 2020-11-16 12:55 UTC (permalink / raw) To: linux-erofs From: Gao Xiang <hsiangkao@aol.com> Make it easier to understand... Signed-off-by: Gao Xiang <hsiangkao@aol.com> --- README | 63 ++++++++++++++++++++++++++++++---------------------------- 1 file changed, 33 insertions(+), 30 deletions(-) diff --git a/README b/README index 7a7ac5d5cb6f..a69607a32e57 100644 --- a/README +++ b/README @@ -1,30 +1,32 @@ erofs-utils =========== -erofs-utils includes user-space tools for erofs filesystem images. +erofs-utils includes user-space tools for erofs filesystem. Currently only mkfs.erofs is available. mkfs.erofs ---------- -It can create 2 primary kinds of erofs images: (un)compressed. +It can generate 2 primary kinds of erofs images: (un)compressed. - - For compressed images, it's able to use several compression - algorithms, but lz4(hc) are only supported due to the current - linux kernel implementation. + - For uncompressed images, there will be none of compression + files in these images. However, it can decide whether the tail + block of a file should be inlined or not properly [1]. - - For uncompressed images, it can decide whether the last page of - a file should be inlined or not properly [1]. + - For compressed images, it will try to use lz4(hc) algorithm + first for each regular file and see if storage space can be + saved with compression. If not, fallback to an uncompressed + file. Dependencies ~~~~~~~~~~~~ - lz4-1.8.0+ for lz4 enabled [2], lz4-1.9.3+ recommended [4]. + lz4-1.8.0+ for lz4 enabled [2], lz4-1.9.3+ highly recommended [4]. How to build for lz4-1.9.0 or above ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To build you can run the following commands in order: +To build, you can run the following commands in order: :: @@ -32,23 +34,25 @@ To build you can run the following commands in order: $ ./configure $ make -mkfs.erofs binary will be generated under mkfs folder. There are still -some issues which affect the stability of LZ4_compress_destSize() -* they have impacts on lz4 only rather than lz4HC * [3]. +mkfs.erofs binary will be generated under mkfs folder. + +* For lz4 < 1.9.2, there were some stability issues about + LZ4_compress_destSize(). (lz4hc aren't impacted) [3]. How to build for lz4-1.8.0~1.8.3 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -For these old lz4 versions, lz4hc algorithm cannot be supported without -lz4 static libary due to LZ4_compress_HC_destSize unstable api usage, -which means only lz4 algrithm is available if lz4 static library isn't found. +For these old lz4 versions, lz4hc algorithm cannot be supported +without lz4-static due to LZ4_compress_HC_destSize() unstable +api usage, which means only lz4 algrithm is available if lz4-static +isn't found. -On Fedora, static lz4 can be installed using: +On Fedora, lz4-static can be installed by using: yum install lz4-static.x86_64 -However, it's not recommended to use those versions since there were bugs -in these compressors, see [2] [3] [4] as well. +However, it's not recommended to use those versions directly since +there were serious bugs in these compressors, see [2] [3] [4] as well. How to generate erofs images ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -78,14 +82,12 @@ which was replaced by the new erofs-utils implementation. git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs -It may still be useful since new erofs-utils has not been widely used in -commercial products. However, if that happens, please report bug to us -as well. +PLEASE NOTE: This version is highly _not recommended_ now. Contribution ------------ -erofs-utils is a GPLv2+ project as a part of erofs file system, +erofs-utils is under GPLv2+ as a part of erofs project, feel free to send patches or feedback to us. To: @@ -101,19 +103,20 @@ Cc: Comments -------- -[1] According to the erofs on-disk format, the last page of files could - be inlined aggressively with its metadata in order to reduce the I/O - overhead and save the storage space. +[1] According to the erofs on-disk format, the tail block of files + could be inlined aggressively with its metadata in order to reduce + the I/O overhead and save the storage space (called tail-packing). -[2] There was a bug until lz4-1.8.3, which can crash erofs-utils randomly. - Fortunately bugfix by our colleague Qiuyang Sun was merged in lz4-1.9.0. +[2] There was a bug until lz4-1.8.3, which can crash erofs-utils + randomly. Fortunately bugfix by our colleague Qiuyang Sun was + merged in lz4-1.9.0. For more details, please refer to https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82 -[3] There are many crash fixes merged to lz4 1.9.2 for LZ4_compress_destSize(), - and I once ran into some crashs due to those issues. - * Again lz4HC is not effected for this section. * +[3] There were many bugfixes merged into lz4-1.9.2 for + LZ4_compress_destSize(), and I once ran into some crashs due to + those issues. * Again lz4hc is not affected. * [LZ4_compress_destSize] Allow 2 more bytes of match length https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda -- 2.24.0 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH v2 2/2] erofs-utils: update README 2020-11-16 12:55 ` [PATCH 2/2] erofs-utils: update README Gao Xiang via Linux-erofs @ 2020-11-16 14:51 ` Gao Xiang via Linux-erofs 0 siblings, 0 replies; 4+ messages in thread From: Gao Xiang via Linux-erofs @ 2020-11-16 14:51 UTC (permalink / raw) To: linux-erofs Make it easier to understand... Signed-off-by: Gao Xiang <hsiangkao@aol.com> --- changes since v1: - fix more typos and descriptions. README | 70 +++++++++++++++++++++++++++++++--------------------------- 1 file changed, 37 insertions(+), 33 deletions(-) diff --git a/README b/README index 7a7ac5d5cb6f..88c45a25ace9 100644 --- a/README +++ b/README @@ -1,30 +1,32 @@ erofs-utils =========== -erofs-utils includes user-space tools for erofs filesystem images. +erofs-utils includes user-space tools for erofs filesystem. Currently only mkfs.erofs is available. mkfs.erofs ---------- -It can create 2 primary kinds of erofs images: (un)compressed. +It can generate 2 primary kinds of erofs images: (un)compressed. - - For compressed images, it's able to use several compression - algorithms, but lz4(hc) are only supported due to the current - linux kernel implementation. + - For uncompressed images, there will be none of compression + files in these images. However, it can decide whether the tail + block of a file should be inlined or not properly [1]. - - For uncompressed images, it can decide whether the last page of - a file should be inlined or not properly [1]. + - For compressed images, it will try to use lz4(hc) algorithm + first for each regular file and see if storage space can be + saved with compression. If not, fallback to an uncompressed + file. Dependencies ~~~~~~~~~~~~ - lz4-1.8.0+ for lz4 enabled [2], lz4-1.9.3+ recommended [4]. + lz4-1.8.0+ for lz4 enabled [2], lz4-1.9.3+ highly recommended [4]. How to build for lz4-1.9.0 or above ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To build you can run the following commands in order: +To build, you can run the following commands in order: :: @@ -32,23 +34,26 @@ To build you can run the following commands in order: $ ./configure $ make -mkfs.erofs binary will be generated under mkfs folder. There are still -some issues which affect the stability of LZ4_compress_destSize() -* they have impacts on lz4 only rather than lz4HC * [3]. +mkfs.erofs binary will be generated under mkfs folder. + +* For lz4 < 1.9.2, there are some stability issues about + LZ4_compress_destSize(). (lz4hc isn't impacted) [3]. How to build for lz4-1.8.0~1.8.3 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -For these old lz4 versions, lz4hc algorithm cannot be supported without -lz4 static libary due to LZ4_compress_HC_destSize unstable api usage, -which means only lz4 algrithm is available if lz4 static library isn't found. +For these old lz4 versions, lz4hc algorithm cannot be supported +without lz4-static installed due to LZ4_compress_HC_destSize() +unstable api usage, which means only lz4 algrithm will be available +if lz4-static isn't found. -On Fedora, static lz4 can be installed using: +On Fedora, lz4-static can be installed by using: yum install lz4-static.x86_64 -However, it's not recommended to use those versions since there were bugs -in these compressors, see [2] [3] [4] as well. +However, it's still not recommended to use those versions directly +since there are serious bugs in these compressors, see [2] [3] [4] +as well. How to generate erofs images ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -56,17 +61,17 @@ How to generate erofs images Currently lz4 and lz4hc are available for compression, e.g. $ mkfs.erofs -zlz4hc foo.erofs.img foo/ -Or leave all files uncompressed as a option: +Or leave all files uncompressed as an option: $ mkfs.erofs foo.erofs.img foo/ How to generate legacy erofs images ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Decompression inplace and compacted indexes have been introduced in -linux-5.3, which are not backward-compatible with older kernels. +linux-5.3, which are not forward-compatible with older kernels. In order to generate _legacy_ erofs images for old kernels, -add "-E legacy-compress" to the command line, e.g. +consider adding "-E legacy-compress" to the command line, e.g. $ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/ @@ -78,14 +83,12 @@ which was replaced by the new erofs-utils implementation. git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs -It may still be useful since new erofs-utils has not been widely used in -commercial products. However, if that happens, please report bug to us -as well. +PLEASE NOTE: This version is highly _NOT recommended_ now. Contribution ------------ -erofs-utils is a GPLv2+ project as a part of erofs file system, +erofs-utils is under GPLv2+ as a part of erofs project, feel free to send patches or feedback to us. To: @@ -101,19 +104,20 @@ Cc: Comments -------- -[1] According to the erofs on-disk format, the last page of files could - be inlined aggressively with its metadata in order to reduce the I/O - overhead and save the storage space. +[1] According to the erofs on-disk format, the tail block of files + could be inlined aggressively with its metadata in order to reduce + the I/O overhead and save the storage space (called tail-packing). -[2] There was a bug until lz4-1.8.3, which can crash erofs-utils randomly. - Fortunately bugfix by our colleague Qiuyang Sun was merged in lz4-1.9.0. +[2] There was a bug until lz4-1.8.3, which can crash erofs-utils + randomly. Fortunately bugfix by our colleague Qiuyang Sun was + merged in lz4-1.9.0. For more details, please refer to https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82 -[3] There are many crash fixes merged to lz4 1.9.2 for LZ4_compress_destSize(), - and I once ran into some crashs due to those issues. - * Again lz4HC is not effected for this section. * +[3] There were many bugfixes merged into lz4-1.9.2 for + LZ4_compress_destSize(), and I once ran into some crashs due to + those issues. * Again lz4hc is not affected. * [LZ4_compress_destSize] Allow 2 more bytes of match length https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda -- 2.24.0 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH 1/2] erofs-utils: drop known issue in README 2020-11-16 12:55 ` [PATCH 1/2] erofs-utils: drop known issue in README Gao Xiang via Linux-erofs 2020-11-16 12:55 ` [PATCH 2/2] erofs-utils: update README Gao Xiang via Linux-erofs @ 2020-11-21 6:31 ` Li GuiFu via Linux-erofs 1 sibling, 0 replies; 4+ messages in thread From: Li GuiFu via Linux-erofs @ 2020-11-21 6:31 UTC (permalink / raw) To: Gao Xiang, linux-erofs On 2020/11/16 20:55, Gao Xiang via Linux-erofs wrote: > From: Gao Xiang <hsiangkao@aol.com> > > Since lz4-1.9.3 has been released, > https://github.com/lz4/lz4/releases/tag/v1.9.3 > > Move this lz4hc issue (lz4 <= 1.8.2) to "Comments" instead. > > Signed-off-by: Gao Xiang <hsiangkao@aol.com> It looks good Reviewed-by: Li Guifu <bluce.lee@aliyun.com> Thanks, ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-11-21 6:32 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <20201116125527.3644-1-hsiangkao.ref@aol.com> 2020-11-16 12:55 ` [PATCH 1/2] erofs-utils: drop known issue in README Gao Xiang via Linux-erofs 2020-11-16 12:55 ` [PATCH 2/2] erofs-utils: update README Gao Xiang via Linux-erofs 2020-11-16 14:51 ` [PATCH v2 " Gao Xiang via Linux-erofs 2020-11-21 6:31 ` [PATCH 1/2] erofs-utils: drop known issue in README Li GuiFu via Linux-erofs
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).