[PATCH v4 1/5] erofs-utils: sync up with in-kernel erofs_fs.h

[PATCH v4 1/5] erofs-utils: sync up with in-kernel erofs_fs.h

[Gao Xiang]

This matches the latest in-kernel version.

Signed-off-by: Gao Xiang <xiang@kernel.org>
---
changes since v3:
 - fix typos in commit messages of 4/5 and 5/5;

Note that I'll apply them to dev branch directly tomorrow if still
no response from Guifu since more further work needs stable dev
branch like more pclustersize strategy for different data pattern
and LZMA compression.

In addition, we're quite close to release erofs-utils v1.3 as well
and these patches are needed.

 include/erofs_fs.h | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/include/erofs_fs.h b/include/erofs_fs.h
index 52da7abaac92..18fc1820c58c 100644
--- a/include/erofs_fs.h
+++ b/include/erofs_fs.h
@@ -27,13 +27,15 @@
 	 EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \
 	 EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER)
 
-/* 128-byte erofs on-disk super block */
+#define EROFS_SB_EXTSLOT_SIZE	16
+
+/* erofs on-disk super block (currently 128 bytes) */
 struct erofs_super_block {
 	__le32 magic;           /* file system magic number */
 	__le32 checksum;        /* crc32c(super_block) */
 	__le32 feature_compat;
 	__u8 blkszbits;         /* support block_size == PAGE_SIZE only */
-	__u8 reserved;
+	__u8 sb_extslots;	/* superblock size = 128 + sb_extslots * 16 */
 
 	__le16 root_nid;	/* nid of root directory */
 	__le64 inos;            /* total valid ino # (== f_files - f_favail) */
@@ -49,8 +51,9 @@ struct erofs_super_block {
 	union {
 		/* bitmap for available compression algorithms */
 		__le16 available_compr_algs;
+		/* customized sliding window size instead of 64k by default */
 		__le16 lz4_max_distance;
-	} u1;
+	} __packed u1;
 	__u8 reserved2[42];
 };
 
@@ -87,6 +90,9 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
 #define EROFS_I_VERSION_BIT             0
 #define EROFS_I_DATALAYOUT_BIT          1
 
+#define EROFS_I_ALL	\
+	((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
+
 /* 32-byte reduced form of an ondisk inode */
 struct erofs_inode_compact {
 	__le16 i_format;	/* inode format hints */
@@ -209,6 +215,7 @@ enum {
 	Z_EROFS_COMPRESSION_LZ4	= 0,
 	Z_EROFS_COMPRESSION_MAX
 };
+#define Z_EROFS_ALL_COMPR_ALGS		(1 << (Z_EROFS_COMPRESSION_MAX - 1))
 
 /* 14 bytes (+ length field = 16 bytes) */
 struct z_erofs_lz4_cfgs {
@@ -238,9 +245,7 @@ struct z_erofs_map_header {
 	__u8	h_algorithmtype;
 	/*
 	 * bit 0-2 : logical cluster bits - 12, e.g. 0 for 4096;
-	 * bit 3-4 : (physical - logical) cluster bits of head 1:
-	 *       For example, if logical clustersize = 4096, 1 for 8192.
-	 * bit 5-7 : (physical - logical) cluster bits of head 2.
+	 * bit 3-7 : reserved.
 	 */
 	__u8	h_clusterbits;
 };
-- 
2.20.1

[PATCH v4 2/5] erofs-utils: warn out experimental big pcluster

[Gao Xiang]

It's still an experimental feature for now. Also set the default
logging level to 2 in order to print warn messages.

Signed-off-by: Gao Xiang <xiang@kernel.org>
---
 lib/compress.c   | 2 ++
 lib/config.c     | 2 +-
 man/mkfs.erofs.1 | 3 ++-
 3 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/compress.c b/lib/compress.c
index 654286d3f33e..b8bb89e7ae9d 100644
--- a/lib/compress.c
+++ b/lib/compress.c
@@ -619,6 +619,8 @@ int z_erofs_compress_init(struct erofs_buffer_head *sb_bh)
 		mapheader.h_advise |= Z_EROFS_ADVISE_BIG_PCLUSTER_1;
 		if (!cfg.c_legacy_compress)
 			mapheader.h_advise |= Z_EROFS_ADVISE_BIG_PCLUSTER_2;
+
+		erofs_warn("EXPERIMENTAL big pcluster feature in use. Use at your own risk!");
 	}
 	mapheader.h_algorithmtype = algorithmtype[1] << 4 |
 					  algorithmtype[0];
diff --git a/lib/config.c b/lib/config.c
index d733cc366794..bc0afa284807 100644
--- a/lib/config.c
+++ b/lib/config.c
@@ -17,7 +17,7 @@ void erofs_init_configure(void)
 {
 	memset(&cfg, 0, sizeof(cfg));
 
-	cfg.c_dbg_lvl  = 0;
+	cfg.c_dbg_lvl  = 2;
 	cfg.c_version  = PACKAGE_VERSION;
 	cfg.c_dry_run  = false;
 	cfg.c_compr_level_master = -1;
diff --git a/man/mkfs.erofs.1 b/man/mkfs.erofs.1
index 254c3ec4de41..2520b6a08974 100644
--- a/man/mkfs.erofs.1
+++ b/man/mkfs.erofs.1
@@ -25,7 +25,8 @@ Set an algorithm for file compression, which can be set with an optional
 compression level separated by a comma.
 .TP
 .BI "\-d " #
-Specify the level of debugging messages. The default is 0.
+Specify the level of debugging messages. The default is 2, which shows basic
+warning messages.
 .TP
 .BI "\-x " #
 Specify the upper limit of an xattr which is still inlined. The default is 2.
-- 
2.20.1

[PATCH v4 3/5] erofs-utils: manpage: add missing -C option for big pcluster

[Gao Xiang]

Update the manpage as well.

Signed-off-by: Gao Xiang <xiang@kernel.org>
---
 man/mkfs.erofs.1 | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/man/mkfs.erofs.1 b/man/mkfs.erofs.1
index 2520b6a08974..4f2e43565799 100644
--- a/man/mkfs.erofs.1
+++ b/man/mkfs.erofs.1
@@ -24,6 +24,10 @@ from \fISOURCE\fR directory.
 Set an algorithm for file compression, which can be set with an optional
 compression level separated by a comma.
 .TP
+.BI "\-C " max-pcluster-size
+Specify the maximum size of compress physical cluster in bytes. It may enable
+big pcluster feature if needed (Linux v5.13+).
+.TP
 .BI "\-d " #
 Specify the level of debugging messages. The default is 2, which shows basic
 warning messages.
-- 
2.20.1

[PATCH v4 4/5] erofs-utils: zero out garbage trailing data for non-0padding cases

[Gao Xiang]

When "-E legacy-compress" is used, lz4 0padding feature will be
disabled by default in order to support old kernels (< Linux v5.3).

In that case, the current mkfs leaves previous garbage data after
valid compressed data if the length becomes shorter. This doesn't
matter for kernels >= v5.0 since LZ4_decompress_safe_partial() is used.

However, for staging erofs v4.19, it used an in-house customized
lz4 implemention due to LZ4_decompress_safe_partial() didn't work
as expected at that time, yet it doesn't allow trailing random
data in practice or decompression failure could happen.

I don't think it really matters since "obsoleted_mkfs" works perfectly
for such old staging versions (v4.19). Anyway, trailing garbage data
sounds unreasonable, so let's zero out it now.

Fixes: 66653ef10a7f ("erofs-utils: introduce compression for regular files")
Signed-off-by: Gao Xiang <xiang@kernel.org>
---
 lib/compress.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/lib/compress.c b/lib/compress.c
index b8bb89e7ae9d..deef6a2c8ae3 100644
--- a/lib/compress.c
+++ b/lib/compress.c
@@ -189,18 +189,22 @@ nocompression:
 			ctx->compressedblks = 1;
 			raw = true;
 		} else {
-			const unsigned int used = ret & (EROFS_BLKSIZ - 1);
-			const unsigned int margin =
-				erofs_sb_has_lz4_0padding() && used ?
-					EROFS_BLKSIZ - used : 0;
+			const unsigned int tailused = ret & (EROFS_BLKSIZ - 1);
+			const unsigned int padding =
+				erofs_sb_has_lz4_0padding() && tailused ?
+					EROFS_BLKSIZ - tailused : 0;
 
 			ctx->compressedblks = DIV_ROUND_UP(ret, EROFS_BLKSIZ);
+			/* zero out garbage trailing data for non-0padding */
+			if (!erofs_sb_has_lz4_0padding())
+				memset(dst + ret, 0,
+				       roundup(ret, EROFS_BLKSIZ) - ret);
 
 			/* write compressed data */
 			erofs_dbg("Writing %u compressed data to %u of %u blocks",
 				  count, ctx->blkaddr, ctx->compressedblks);
 
-			ret = blk_write(dst - margin, ctx->blkaddr,
+			ret = blk_write(dst - padding, ctx->blkaddr,
 					ctx->compressedblks);
 			if (ret)
 				return ret;
-- 
2.20.1

[PATCH v4 5/5] erofs-utils: manpage: add manual for erofsfuse

[Gao Xiang]

This patch adds missing erofsfuse manpage.

Signed-off-by: Gao Xiang <xiang@kernel.org>
---
 man/erofsfuse.1 | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 man/erofsfuse.1

diff --git a/man/erofsfuse.1 b/man/erofsfuse.1
new file mode 100644
index 000000000000..6bd48b0460bd
--- /dev/null
+++ b/man/erofsfuse.1
@@ -0,0 +1,44 @@
+.\" Copyright (c) 2021 Gao Xiang <xiang@kernel.org>
+.\"
+.TH EROFSFUSE 1
+.SH NAME
+erofsfuse \- FUSE file system client for erofs file system
+.SH SYNOPSIS
+\fBerofsfuse\fR [\fIOPTIONS\fR] \fIDEVICE\fR \fIMOUNTPOINT\fR
+.SH DESCRIPTION
+.B erofsfuse
+is a FUSE file system client that supports reading from devices or image files
+containing erofs file system.
+.SH OPTIONS
+.SS "general options:"
+.TP
+\fB\-o\fR opt,[opt...]
+mount options
+.TP
+\fB\-h\fR   \fB\-\-help\fR
+display help and exit
+.SS "erofsfuse options:"
+.TP
+.BI "\-\-dbglevel=" #
+Specify the level of debugging messages. The default is 2, which shows basic
+warning messages.
+.SS "FUSE options:"
+.TP
+\fB-d -o\fR debug
+enable debug output (implies -f)
+.TP
+\fB-f\fR
+foreground operation
+.TP
+\fB-s\fR
+disable multi-threaded operation
+.P
+For other FUSE options please see
+.BR mount.fuse (8)
+or see the output of
+.I erofsfuse \-\-help
+.SH AVAILABILITY
+\fBerofsfuse\fR is part of erofs-utils package and is available from
+git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git.
+.SH SEE ALSO
+.BR mount.fuse (8)
-- 
2.20.1

Re: [PATCH v4 2/5] erofs-utils: warn out experimental big pcluster

[Li GuiFu via Linux-erofs]

On 2021/4/30 12:03, Gao Xiang wrote:
> It's still an experimental feature for now. Also set the default
> logging level to 2 in order to print warn messages.
> 
> Signed-off-by: Gao Xiang <xiang@kernel.org>
It looks good
Reviewed-by: Li Guifu <bluce.lee@aliyun.com>

Thanks,

Re: [PATCH v4 3/5] erofs-utils: manpage: add missing -C option for big pcluster

[Li GuiFu via Linux-erofs]

On 2021/4/30 12:03, Gao Xiang wrote:
> Update the manpage as well.
> 
> Signed-off-by: Gao Xiang <xiang@kernel.org>
It looks good
Reviewed-by: Li Guifu <bluce.lee@aliyun.com>

Thanks,

Re: [PATCH v4 4/5] erofs-utils: zero out garbage trailing data for non-0padding cases

[Li GuiFu via Linux-erofs]

On 2021/4/30 12:03, Gao Xiang wrote:
> When "-E legacy-compress" is used, lz4 0padding feature will be
> disabled by default in order to support old kernels (< Linux v5.3).
> 
> In that case, the current mkfs leaves previous garbage data after
> valid compressed data if the length becomes shorter. This doesn't
> matter for kernels >= v5.0 since LZ4_decompress_safe_partial() is used.
> 
> However, for staging erofs v4.19, it used an in-house customized
> lz4 implemention due to LZ4_decompress_safe_partial() didn't work
> as expected at that time, yet it doesn't allow trailing random
> data in practice or decompression failure could happen.
> 
> I don't think it really matters since "obsoleted_mkfs" works perfectly
> for such old staging versions (v4.19). Anyway, trailing garbage data
> sounds unreasonable, so let's zero out it now.
> 
> Fixes: 66653ef10a7f ("erofs-utils: introduce compression for regular files")
> Signed-off-by: Gao Xiang <xiang@kernel.org>
> ---
>  lib/compress.c | 14 +++++++++-----
>  1 file changed, 9 insertions(+), 5 deletions(-)
> 
It looks good
Reviewed-by: Li Guifu <bluce.lee@aliyun.com>

Thanks,

Re: [PATCH v4 5/5] erofs-utils: manpage: add manual for erofsfuse

[Li GuiFu via Linux-erofs]

On 2021/4/30 12:03, Gao Xiang wrote:
> This patch adds missing erofsfuse manpage.
> 
> Signed-off-by: Gao Xiang <xiang@kernel.org>
> ---
>  man/erofsfuse.1 | 44 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 44 insertions(+)
>  create mode 100644 man/erofsfuse.1
> 
It looks good
Reviewed-by: Li Guifu <bluce.lee@aliyun.com>

Thanks,