linux-erofs.lists.ozlabs.org archive mirror
 help / color / mirror / Atom feed
From: Gao Xiang <xiang@kernel.org>
To: linux-erofs@lists.ozlabs.org
Cc: Gao Xiang <xiang@kernel.org>, Guo Weichao <guoweichao@oppo.com>
Subject: [PATCH v1.1 1/8] erofs-utils: support adjust lz4 history window size
Date: Thu, 15 Apr 2021 11:27:05 +0800	[thread overview]
Message-ID: <20210415032705.27515-1-xiang@kernel.org> (raw)
In-Reply-To: <20210411034844.12673-1-xiang@kernel.org>

From: Huang Jianan <huangjianan@oppo.com>

lz4 uses LZ4_DISTANCE_MAX to record history preservation. When
using rolling decompression, a block with a higher compression
ratio will cause a larger memory allocation (up to 64k). It may
cause a large resource burden in extreme cases on devices with
small memory and a large number of concurrent IOs. So appropriately
reducing this value can improve performance.

Decreasing this value will reduce the compression ratio (except
when input_size <LZ4_DISTANCE_MAX). But considering that erofs
currently only supports 4k output, reducing this value will not
significantly reduce the compression benefits.

The maximum value of LZ4_DISTANCE_MAX defined by lz4 is 64k, and
we can only reduce this value. For the old kernel, it just can't
reduce the memory allocation during rolling decompression without
affecting the decompression result.

Signed-off-by: Huang Jianan <huangjianan@oppo.com>
Signed-off-by: Guo Weichao <guoweichao@oppo.com>
Signed-off-by: Gao Xiang <xiang@kernel.org>
---
changes since v1:
 - fix missing LZ4_DISTANCE_MAX definition for lz4hc reported
   by travis CI.

 include/erofs/internal.h | 1 +
 include/erofs_fs.h       | 3 ++-
 lib/compressor_lz4.c     | 5 +++++
 lib/compressor_lz4hc.c   | 6 ++++++
 mkfs/main.c              | 1 +
 5 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/include/erofs/internal.h b/include/erofs/internal.h
index ac5b270329e2..3849980d8eab 100644
--- a/include/erofs/internal.h
+++ b/include/erofs/internal.h
@@ -79,6 +79,7 @@ struct erofs_sb_info {
 	u64 inos;
 
 	u8 uuid[16];
+	u16 lz4_max_distance;
 };
 
 /* global sbi */
diff --git a/include/erofs_fs.h b/include/erofs_fs.h
index a69f179a51a5..ae2305c1eb79 100644
--- a/include/erofs_fs.h
+++ b/include/erofs_fs.h
@@ -41,7 +41,8 @@ struct erofs_super_block {
 	__u8 uuid[16];          /* 128-bit uuid for volume */
 	__u8 volume_name[16];   /* volume name */
 	__le32 feature_incompat;
-	__u8 reserved2[44];
+	__le16 lz4_max_distance;
+	__u8 reserved2[42];
 };
 
 /*
diff --git a/lib/compressor_lz4.c b/lib/compressor_lz4.c
index 8540a0d01cbb..292d0f27fe0e 100644
--- a/lib/compressor_lz4.c
+++ b/lib/compressor_lz4.c
@@ -10,6 +10,10 @@
 #include "erofs/internal.h"
 #include "compressor.h"
 
+#ifndef LZ4_DISTANCE_MAX	/* history window size */
+#define LZ4_DISTANCE_MAX 65535	/* set to maximum value by default */
+#endif
+
 static int lz4_compress_destsize(struct erofs_compress *c,
 				 int compression_level,
 				 void *src, unsigned int *srcsize,
@@ -32,6 +36,7 @@ static int compressor_lz4_exit(struct erofs_compress *c)
 static int compressor_lz4_init(struct erofs_compress *c)
 {
 	c->alg = &erofs_compressor_lz4;
+	sbi.lz4_max_distance = LZ4_DISTANCE_MAX;
 	return 0;
 }
 
diff --git a/lib/compressor_lz4hc.c b/lib/compressor_lz4hc.c
index 6680563986c3..14c3a71b0a80 100644
--- a/lib/compressor_lz4hc.c
+++ b/lib/compressor_lz4hc.c
@@ -11,6 +11,10 @@
 #include "erofs/internal.h"
 #include "compressor.h"
 
+#ifndef LZ4_DISTANCE_MAX	/* history window size */
+#define LZ4_DISTANCE_MAX 65535	/* set to maximum value by default */
+#endif
+
 static int lz4hc_compress_destsize(struct erofs_compress *c,
 				   int compression_level,
 				   void *src,
@@ -44,6 +48,8 @@ static int compressor_lz4hc_init(struct erofs_compress *c)
 	c->private_data = LZ4_createStreamHC();
 	if (!c->private_data)
 		return -ENOMEM;
+
+	sbi.lz4_max_distance = LZ4_DISTANCE_MAX;
 	return 0;
 }
 
diff --git a/mkfs/main.c b/mkfs/main.c
index 6bc179db8dc3..8a9611f0f3d0 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -322,6 +322,7 @@ int erofs_mkfs_update_super_block(struct erofs_buffer_head *bh,
 		.feature_incompat = cpu_to_le32(sbi.feature_incompat),
 		.feature_compat = cpu_to_le32(sbi.feature_compat &
 					      ~EROFS_FEATURE_COMPAT_SB_CHKSUM),
+		.lz4_max_distance = cpu_to_le16(sbi.lz4_max_distance),
 	};
 	const unsigned int sb_blksize =
 		round_up(EROFS_SUPER_END, EROFS_BLKSIZ);
-- 
2.20.1


      parent reply	other threads:[~2021-04-15  3:27 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-11  3:48 [PATCH 0/8] erofs-utils: support big pcluster compression Gao Xiang
2021-04-11  3:48 ` [PATCH 1/8] erofs-utils: support adjust lz4 history window size Gao Xiang
2021-04-11  3:48 ` [PATCH 2/8] erofs-utils: introduce ondisk compression cfgs Gao Xiang
2021-04-11  3:48 ` [PATCH 3/8] erofs-utils: add -C# for the maximum size of pclusters Gao Xiang
2021-04-11  3:48 ` [PATCH 4/8] erofs-utils: add big physical cluster definition Gao Xiang
2021-04-11  3:48 ` [PATCH 5/8] erofs-utils: fuse: support multiple block compression Gao Xiang
2021-04-11  3:48 ` [PATCH 6/8] erofs-utils: mkfs: " Gao Xiang
2021-04-11  3:48 ` [PATCH 7/8] erofs-utils: fuse: support compact indexes for bigpcluster Gao Xiang
2021-04-11  3:48 ` [PATCH 8/8] erofs-utils: mkfs: " Gao Xiang
2021-04-15  3:27 ` Gao Xiang [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210415032705.27515-1-xiang@kernel.org \
    --to=xiang@kernel.org \
    --cc=guoweichao@oppo.com \
    --cc=linux-erofs@lists.ozlabs.org \
    --subject='Re: [PATCH v1.1 1/8] erofs-utils: support adjust lz4 history window size' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).