linux-erofs.lists.ozlabs.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v4 1/5] erofs-utils: sync up with in-kernel erofs_fs.h
@ 2021-04-30  4:03 Gao Xiang
  2021-04-30  4:03 ` [PATCH v4 2/5] erofs-utils: warn out experimental big pcluster Gao Xiang
                   ` (3 more replies)
  0 siblings, 4 replies; 9+ messages in thread
From: Gao Xiang @ 2021-04-30  4:03 UTC (permalink / raw)
  To: linux-erofs, Li Guifu; +Cc: Gao Xiang

This matches the latest in-kernel version.

Signed-off-by: Gao Xiang <xiang@kernel.org>
---
changes since v3:
 - fix typos in commit messages of 4/5 and 5/5;

Note that I'll apply them to dev branch directly tomorrow if still
no response from Guifu since more further work needs stable dev
branch like more pclustersize strategy for different data pattern
and LZMA compression.

In addition, we're quite close to release erofs-utils v1.3 as well
and these patches are needed.

 include/erofs_fs.h | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/include/erofs_fs.h b/include/erofs_fs.h
index 52da7abaac92..18fc1820c58c 100644
--- a/include/erofs_fs.h
+++ b/include/erofs_fs.h
@@ -27,13 +27,15 @@
 	 EROFS_FEATURE_INCOMPAT_COMPR_CFGS | \
 	 EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER)
 
-/* 128-byte erofs on-disk super block */
+#define EROFS_SB_EXTSLOT_SIZE	16
+
+/* erofs on-disk super block (currently 128 bytes) */
 struct erofs_super_block {
 	__le32 magic;           /* file system magic number */
 	__le32 checksum;        /* crc32c(super_block) */
 	__le32 feature_compat;
 	__u8 blkszbits;         /* support block_size == PAGE_SIZE only */
-	__u8 reserved;
+	__u8 sb_extslots;	/* superblock size = 128 + sb_extslots * 16 */
 
 	__le16 root_nid;	/* nid of root directory */
 	__le64 inos;            /* total valid ino # (== f_files - f_favail) */
@@ -49,8 +51,9 @@ struct erofs_super_block {
 	union {
 		/* bitmap for available compression algorithms */
 		__le16 available_compr_algs;
+		/* customized sliding window size instead of 64k by default */
 		__le16 lz4_max_distance;
-	} u1;
+	} __packed u1;
 	__u8 reserved2[42];
 };
 
@@ -87,6 +90,9 @@ static inline bool erofs_inode_is_data_compressed(unsigned int datamode)
 #define EROFS_I_VERSION_BIT             0
 #define EROFS_I_DATALAYOUT_BIT          1
 
+#define EROFS_I_ALL	\
+	((1 << (EROFS_I_DATALAYOUT_BIT + EROFS_I_DATALAYOUT_BITS)) - 1)
+
 /* 32-byte reduced form of an ondisk inode */
 struct erofs_inode_compact {
 	__le16 i_format;	/* inode format hints */
@@ -209,6 +215,7 @@ enum {
 	Z_EROFS_COMPRESSION_LZ4	= 0,
 	Z_EROFS_COMPRESSION_MAX
 };
+#define Z_EROFS_ALL_COMPR_ALGS		(1 << (Z_EROFS_COMPRESSION_MAX - 1))
 
 /* 14 bytes (+ length field = 16 bytes) */
 struct z_erofs_lz4_cfgs {
@@ -238,9 +245,7 @@ struct z_erofs_map_header {
 	__u8	h_algorithmtype;
 	/*
 	 * bit 0-2 : logical cluster bits - 12, e.g. 0 for 4096;
-	 * bit 3-4 : (physical - logical) cluster bits of head 1:
-	 *       For example, if logical clustersize = 4096, 1 for 8192.
-	 * bit 5-7 : (physical - logical) cluster bits of head 2.
+	 * bit 3-7 : reserved.
 	 */
 	__u8	h_clusterbits;
 };
-- 
2.20.1


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v4 2/5] erofs-utils: warn out experimental big pcluster
  2021-04-30  4:03 [PATCH v4 1/5] erofs-utils: sync up with in-kernel erofs_fs.h Gao Xiang
@ 2021-04-30  4:03 ` Gao Xiang
  2021-04-30 16:16   ` Li GuiFu via Linux-erofs
  2021-04-30  4:03 ` [PATCH v4 3/5] erofs-utils: manpage: add missing -C option for " Gao Xiang
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 9+ messages in thread
From: Gao Xiang @ 2021-04-30  4:03 UTC (permalink / raw)
  To: linux-erofs, Li Guifu; +Cc: Gao Xiang

It's still an experimental feature for now. Also set the default
logging level to 2 in order to print warn messages.

Signed-off-by: Gao Xiang <xiang@kernel.org>
---
 lib/compress.c   | 2 ++
 lib/config.c     | 2 +-
 man/mkfs.erofs.1 | 3 ++-
 3 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/compress.c b/lib/compress.c
index 654286d3f33e..b8bb89e7ae9d 100644
--- a/lib/compress.c
+++ b/lib/compress.c
@@ -619,6 +619,8 @@ int z_erofs_compress_init(struct erofs_buffer_head *sb_bh)
 		mapheader.h_advise |= Z_EROFS_ADVISE_BIG_PCLUSTER_1;
 		if (!cfg.c_legacy_compress)
 			mapheader.h_advise |= Z_EROFS_ADVISE_BIG_PCLUSTER_2;
+
+		erofs_warn("EXPERIMENTAL big pcluster feature in use. Use at your own risk!");
 	}
 	mapheader.h_algorithmtype = algorithmtype[1] << 4 |
 					  algorithmtype[0];
diff --git a/lib/config.c b/lib/config.c
index d733cc366794..bc0afa284807 100644
--- a/lib/config.c
+++ b/lib/config.c
@@ -17,7 +17,7 @@ void erofs_init_configure(void)
 {
 	memset(&cfg, 0, sizeof(cfg));
 
-	cfg.c_dbg_lvl  = 0;
+	cfg.c_dbg_lvl  = 2;
 	cfg.c_version  = PACKAGE_VERSION;
 	cfg.c_dry_run  = false;
 	cfg.c_compr_level_master = -1;
diff --git a/man/mkfs.erofs.1 b/man/mkfs.erofs.1
index 254c3ec4de41..2520b6a08974 100644
--- a/man/mkfs.erofs.1
+++ b/man/mkfs.erofs.1
@@ -25,7 +25,8 @@ Set an algorithm for file compression, which can be set with an optional
 compression level separated by a comma.
 .TP
 .BI "\-d " #
-Specify the level of debugging messages. The default is 0.
+Specify the level of debugging messages. The default is 2, which shows basic
+warning messages.
 .TP
 .BI "\-x " #
 Specify the upper limit of an xattr which is still inlined. The default is 2.
-- 
2.20.1


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v4 3/5] erofs-utils: manpage: add missing -C option for big pcluster
  2021-04-30  4:03 [PATCH v4 1/5] erofs-utils: sync up with in-kernel erofs_fs.h Gao Xiang
  2021-04-30  4:03 ` [PATCH v4 2/5] erofs-utils: warn out experimental big pcluster Gao Xiang
@ 2021-04-30  4:03 ` Gao Xiang
  2021-04-30 16:16   ` Li GuiFu via Linux-erofs
  2021-04-30  4:03 ` [PATCH v4 4/5] erofs-utils: zero out garbage trailing data for non-0padding cases Gao Xiang
  2021-04-30  4:03 ` [PATCH v4 5/5] erofs-utils: manpage: add manual for erofsfuse Gao Xiang
  3 siblings, 1 reply; 9+ messages in thread
From: Gao Xiang @ 2021-04-30  4:03 UTC (permalink / raw)
  To: linux-erofs, Li Guifu; +Cc: Gao Xiang

Update the manpage as well.

Signed-off-by: Gao Xiang <xiang@kernel.org>
---
 man/mkfs.erofs.1 | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/man/mkfs.erofs.1 b/man/mkfs.erofs.1
index 2520b6a08974..4f2e43565799 100644
--- a/man/mkfs.erofs.1
+++ b/man/mkfs.erofs.1
@@ -24,6 +24,10 @@ from \fISOURCE\fR directory.
 Set an algorithm for file compression, which can be set with an optional
 compression level separated by a comma.
 .TP
+.BI "\-C " max-pcluster-size
+Specify the maximum size of compress physical cluster in bytes. It may enable
+big pcluster feature if needed (Linux v5.13+).
+.TP
 .BI "\-d " #
 Specify the level of debugging messages. The default is 2, which shows basic
 warning messages.
-- 
2.20.1


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v4 4/5] erofs-utils: zero out garbage trailing data for non-0padding cases
  2021-04-30  4:03 [PATCH v4 1/5] erofs-utils: sync up with in-kernel erofs_fs.h Gao Xiang
  2021-04-30  4:03 ` [PATCH v4 2/5] erofs-utils: warn out experimental big pcluster Gao Xiang
  2021-04-30  4:03 ` [PATCH v4 3/5] erofs-utils: manpage: add missing -C option for " Gao Xiang
@ 2021-04-30  4:03 ` Gao Xiang
  2021-04-30 16:17   ` Li GuiFu via Linux-erofs
  2021-04-30  4:03 ` [PATCH v4 5/5] erofs-utils: manpage: add manual for erofsfuse Gao Xiang
  3 siblings, 1 reply; 9+ messages in thread
From: Gao Xiang @ 2021-04-30  4:03 UTC (permalink / raw)
  To: linux-erofs, Li Guifu; +Cc: Gao Xiang

When "-E legacy-compress" is used, lz4 0padding feature will be
disabled by default in order to support old kernels (< Linux v5.3).

In that case, the current mkfs leaves previous garbage data after
valid compressed data if the length becomes shorter. This doesn't
matter for kernels >= v5.0 since LZ4_decompress_safe_partial() is used.

However, for staging erofs v4.19, it used an in-house customized
lz4 implemention due to LZ4_decompress_safe_partial() didn't work
as expected at that time, yet it doesn't allow trailing random
data in practice or decompression failure could happen.

I don't think it really matters since "obsoleted_mkfs" works perfectly
for such old staging versions (v4.19). Anyway, trailing garbage data
sounds unreasonable, so let's zero out it now.

Fixes: 66653ef10a7f ("erofs-utils: introduce compression for regular files")
Signed-off-by: Gao Xiang <xiang@kernel.org>
---
 lib/compress.c | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/lib/compress.c b/lib/compress.c
index b8bb89e7ae9d..deef6a2c8ae3 100644
--- a/lib/compress.c
+++ b/lib/compress.c
@@ -189,18 +189,22 @@ nocompression:
 			ctx->compressedblks = 1;
 			raw = true;
 		} else {
-			const unsigned int used = ret & (EROFS_BLKSIZ - 1);
-			const unsigned int margin =
-				erofs_sb_has_lz4_0padding() && used ?
-					EROFS_BLKSIZ - used : 0;
+			const unsigned int tailused = ret & (EROFS_BLKSIZ - 1);
+			const unsigned int padding =
+				erofs_sb_has_lz4_0padding() && tailused ?
+					EROFS_BLKSIZ - tailused : 0;
 
 			ctx->compressedblks = DIV_ROUND_UP(ret, EROFS_BLKSIZ);
+			/* zero out garbage trailing data for non-0padding */
+			if (!erofs_sb_has_lz4_0padding())
+				memset(dst + ret, 0,
+				       roundup(ret, EROFS_BLKSIZ) - ret);
 
 			/* write compressed data */
 			erofs_dbg("Writing %u compressed data to %u of %u blocks",
 				  count, ctx->blkaddr, ctx->compressedblks);
 
-			ret = blk_write(dst - margin, ctx->blkaddr,
+			ret = blk_write(dst - padding, ctx->blkaddr,
 					ctx->compressedblks);
 			if (ret)
 				return ret;
-- 
2.20.1


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v4 5/5] erofs-utils: manpage: add manual for erofsfuse
  2021-04-30  4:03 [PATCH v4 1/5] erofs-utils: sync up with in-kernel erofs_fs.h Gao Xiang
                   ` (2 preceding siblings ...)
  2021-04-30  4:03 ` [PATCH v4 4/5] erofs-utils: zero out garbage trailing data for non-0padding cases Gao Xiang
@ 2021-04-30  4:03 ` Gao Xiang
  2021-04-30 16:17   ` Li GuiFu via Linux-erofs
  3 siblings, 1 reply; 9+ messages in thread
From: Gao Xiang @ 2021-04-30  4:03 UTC (permalink / raw)
  To: linux-erofs, Li Guifu; +Cc: Gao Xiang

This patch adds missing erofsfuse manpage.

Signed-off-by: Gao Xiang <xiang@kernel.org>
---
 man/erofsfuse.1 | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 man/erofsfuse.1

diff --git a/man/erofsfuse.1 b/man/erofsfuse.1
new file mode 100644
index 000000000000..6bd48b0460bd
--- /dev/null
+++ b/man/erofsfuse.1
@@ -0,0 +1,44 @@
+.\" Copyright (c) 2021 Gao Xiang <xiang@kernel.org>
+.\"
+.TH EROFSFUSE 1
+.SH NAME
+erofsfuse \- FUSE file system client for erofs file system
+.SH SYNOPSIS
+\fBerofsfuse\fR [\fIOPTIONS\fR] \fIDEVICE\fR \fIMOUNTPOINT\fR
+.SH DESCRIPTION
+.B erofsfuse
+is a FUSE file system client that supports reading from devices or image files
+containing erofs file system.
+.SH OPTIONS
+.SS "general options:"
+.TP
+\fB\-o\fR opt,[opt...]
+mount options
+.TP
+\fB\-h\fR   \fB\-\-help\fR
+display help and exit
+.SS "erofsfuse options:"
+.TP
+.BI "\-\-dbglevel=" #
+Specify the level of debugging messages. The default is 2, which shows basic
+warning messages.
+.SS "FUSE options:"
+.TP
+\fB-d -o\fR debug
+enable debug output (implies -f)
+.TP
+\fB-f\fR
+foreground operation
+.TP
+\fB-s\fR
+disable multi-threaded operation
+.P
+For other FUSE options please see
+.BR mount.fuse (8)
+or see the output of
+.I erofsfuse \-\-help
+.SH AVAILABILITY
+\fBerofsfuse\fR is part of erofs-utils package and is available from
+git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git.
+.SH SEE ALSO
+.BR mount.fuse (8)
-- 
2.20.1


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v4 2/5] erofs-utils: warn out experimental big pcluster
  2021-04-30  4:03 ` [PATCH v4 2/5] erofs-utils: warn out experimental big pcluster Gao Xiang
@ 2021-04-30 16:16   ` Li GuiFu via Linux-erofs
  0 siblings, 0 replies; 9+ messages in thread
From: Li GuiFu via Linux-erofs @ 2021-04-30 16:16 UTC (permalink / raw)
  To: Gao Xiang, linux-erofs, Li Guifu



On 2021/4/30 12:03, Gao Xiang wrote:
> It's still an experimental feature for now. Also set the default
> logging level to 2 in order to print warn messages.
> 
> Signed-off-by: Gao Xiang <xiang@kernel.org>
It looks good
Reviewed-by: Li Guifu <bluce.lee@aliyun.com>

Thanks,

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v4 3/5] erofs-utils: manpage: add missing -C option for big pcluster
  2021-04-30  4:03 ` [PATCH v4 3/5] erofs-utils: manpage: add missing -C option for " Gao Xiang
@ 2021-04-30 16:16   ` Li GuiFu via Linux-erofs
  0 siblings, 0 replies; 9+ messages in thread
From: Li GuiFu via Linux-erofs @ 2021-04-30 16:16 UTC (permalink / raw)
  To: Gao Xiang, linux-erofs, Li Guifu



On 2021/4/30 12:03, Gao Xiang wrote:
> Update the manpage as well.
> 
> Signed-off-by: Gao Xiang <xiang@kernel.org>
It looks good
Reviewed-by: Li Guifu <bluce.lee@aliyun.com>

Thanks,


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v4 4/5] erofs-utils: zero out garbage trailing data for non-0padding cases
  2021-04-30  4:03 ` [PATCH v4 4/5] erofs-utils: zero out garbage trailing data for non-0padding cases Gao Xiang
@ 2021-04-30 16:17   ` Li GuiFu via Linux-erofs
  0 siblings, 0 replies; 9+ messages in thread
From: Li GuiFu via Linux-erofs @ 2021-04-30 16:17 UTC (permalink / raw)
  To: Gao Xiang, linux-erofs, Li Guifu



On 2021/4/30 12:03, Gao Xiang wrote:
> When "-E legacy-compress" is used, lz4 0padding feature will be
> disabled by default in order to support old kernels (< Linux v5.3).
> 
> In that case, the current mkfs leaves previous garbage data after
> valid compressed data if the length becomes shorter. This doesn't
> matter for kernels >= v5.0 since LZ4_decompress_safe_partial() is used.
> 
> However, for staging erofs v4.19, it used an in-house customized
> lz4 implemention due to LZ4_decompress_safe_partial() didn't work
> as expected at that time, yet it doesn't allow trailing random
> data in practice or decompression failure could happen.
> 
> I don't think it really matters since "obsoleted_mkfs" works perfectly
> for such old staging versions (v4.19). Anyway, trailing garbage data
> sounds unreasonable, so let's zero out it now.
> 
> Fixes: 66653ef10a7f ("erofs-utils: introduce compression for regular files")
> Signed-off-by: Gao Xiang <xiang@kernel.org>
> ---
>  lib/compress.c | 14 +++++++++-----
>  1 file changed, 9 insertions(+), 5 deletions(-)
> 
It looks good
Reviewed-by: Li Guifu <bluce.lee@aliyun.com>

Thanks,

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v4 5/5] erofs-utils: manpage: add manual for erofsfuse
  2021-04-30  4:03 ` [PATCH v4 5/5] erofs-utils: manpage: add manual for erofsfuse Gao Xiang
@ 2021-04-30 16:17   ` Li GuiFu via Linux-erofs
  0 siblings, 0 replies; 9+ messages in thread
From: Li GuiFu via Linux-erofs @ 2021-04-30 16:17 UTC (permalink / raw)
  To: Gao Xiang, linux-erofs, Li Guifu



On 2021/4/30 12:03, Gao Xiang wrote:
> This patch adds missing erofsfuse manpage.
> 
> Signed-off-by: Gao Xiang <xiang@kernel.org>
> ---
>  man/erofsfuse.1 | 44 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 44 insertions(+)
>  create mode 100644 man/erofsfuse.1
> 
It looks good
Reviewed-by: Li Guifu <bluce.lee@aliyun.com>

Thanks,

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-04-30 16:17 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-30  4:03 [PATCH v4 1/5] erofs-utils: sync up with in-kernel erofs_fs.h Gao Xiang
2021-04-30  4:03 ` [PATCH v4 2/5] erofs-utils: warn out experimental big pcluster Gao Xiang
2021-04-30 16:16   ` Li GuiFu via Linux-erofs
2021-04-30  4:03 ` [PATCH v4 3/5] erofs-utils: manpage: add missing -C option for " Gao Xiang
2021-04-30 16:16   ` Li GuiFu via Linux-erofs
2021-04-30  4:03 ` [PATCH v4 4/5] erofs-utils: zero out garbage trailing data for non-0padding cases Gao Xiang
2021-04-30 16:17   ` Li GuiFu via Linux-erofs
2021-04-30  4:03 ` [PATCH v4 5/5] erofs-utils: manpage: add manual for erofsfuse Gao Xiang
2021-04-30 16:17   ` Li GuiFu via Linux-erofs

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).