Linux-EROFS Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH v3] erofs-utils: introduce segment size to limit the max input stream
@ 2020-06-10 17:17 Li Guifu via Linux-erofs
  2020-06-15 14:43 ` Gao Xiang via Linux-erofs
  0 siblings, 1 reply; 2+ messages in thread
From: Li Guifu via Linux-erofs @ 2020-06-10 17:17 UTC (permalink / raw)
  To: linux-erofs; +Cc: Li Guifu

Limit the max input stream size by adding segment compression
(e.g. 4M segment size), it will benefits:
 - more friendly to block diff (and more details about this);
 - it can also be used for parallel compression in the same file.

Signed-off-by: Li Guifu <bluce.lee@aliyun.com>
---
 include/erofs/config.h |  1 +
 lib/compress.c         | 11 +++++++++++
 lib/config.c           |  1 +
 3 files changed, 13 insertions(+)

diff --git a/include/erofs/config.h b/include/erofs/config.h
index 2f09749..9125c1e 100644
--- a/include/erofs/config.h
+++ b/include/erofs/config.h
@@ -36,6 +36,7 @@ struct erofs_configure {
 	char *c_src_path;
 	char *c_compr_alg_master;
 	int c_compr_level_master;
+	unsigned int c_compr_seg_size;	/* max segment compress size */
 	int c_force_inodeversion;
 	/* < 0, xattr disabled and INT_MAX, always use inline xattrs */
 	int c_inline_xattr_tolerance;
diff --git a/lib/compress.c b/lib/compress.c
index 6cc68ed..8fdbfb2 100644
--- a/lib/compress.c
+++ b/lib/compress.c
@@ -32,6 +32,8 @@ struct z_erofs_vle_compress_ctx {
 
 	erofs_blk_t blkaddr;	/* pointing to the next blkaddr */
 	u16 clusterofs;
+	unsigned int comprlimits;
+	unsigned int comr_seg_size;
 };
 
 #define Z_EROFS_LEGACY_MAP_HEADER_SIZE	\
@@ -158,6 +160,12 @@ static int vle_compress_one(struct erofs_inode *inode,
 	while (len) {
 		bool raw;
 
+		if (ctx->comprlimits >= ctx->comr_seg_size ||
+			ctx->comprlimits + EROFS_BLKSIZ >= ctx->comr_seg_size) {
+			ctx->comprlimits = 0;
+			goto nocompression;
+		}
+
 		if (len <= EROFS_BLKSIZ) {
 			if (final)
 				goto nocompression;
@@ -202,6 +210,7 @@ nocompression:
 
 		++ctx->blkaddr;
 		len -= count;
+		ctx->comprlimits += count;
 
 		if (!final && ctx->head >= EROFS_CONFIG_COMPR_MAX_SZ) {
 			const unsigned int qh_aligned =
@@ -422,6 +431,8 @@ int erofs_write_compressed_file(struct erofs_inode *inode)
 	ctx.head = ctx.tail = 0;
 	ctx.clusterofs = 0;
 	remaining = inode->i_size;
+	ctx.comprlimits = 0;
+	ctx.comr_seg_size = cfg.c_compr_seg_size;
 
 	while (remaining) {
 		const u64 readcount = min_t(u64, remaining,
diff --git a/lib/config.c b/lib/config.c
index da0c260..1c39403 100644
--- a/lib/config.c
+++ b/lib/config.c
@@ -23,6 +23,7 @@ void erofs_init_configure(void)
 	cfg.c_force_inodeversion = 0;
 	cfg.c_inline_xattr_tolerance = 2;
 	cfg.c_unix_timestamp = -1;
+	cfg.c_compr_seg_size = 1024U * EROFS_BLKSIZ;
 }
 
 void erofs_show_config(void)
-- 
2.17.1


^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [PATCH v3] erofs-utils: introduce segment size to limit the max input stream
  2020-06-10 17:17 [PATCH v3] erofs-utils: introduce segment size to limit the max input stream Li Guifu via Linux-erofs
@ 2020-06-15 14:43 ` Gao Xiang via Linux-erofs
  0 siblings, 0 replies; 2+ messages in thread
From: Gao Xiang via Linux-erofs @ 2020-06-15 14:43 UTC (permalink / raw)
  To: Li Guifu; +Cc: linux-erofs

Hi Guifu,

On Thu, Jun 11, 2020 at 01:17:28AM +0800, Li Guifu via Linux-erofs wrote:
> Limit the max input stream size by adding segment compression
> (e.g. 4M segment size), it will benefits:
>  - more friendly to block diff (and more details about this);
>  - it can also be used for parallel compression in the same file.

Subject: erofs-utils: introduce segment compression

Support segment compression which seperates files in several logic
units (segments) and each segment is compressed independently.

Advantages:
 - more friendly for data differencing;
 - it can also be used for parallel compression in the same file later.

> 
> Signed-off-by: Li Guifu <bluce.lee@aliyun.com>
> ---
>  include/erofs/config.h |  1 +
>  lib/compress.c         | 11 +++++++++++
>  lib/config.c           |  1 +
>  3 files changed, 13 insertions(+)
> 
> diff --git a/include/erofs/config.h b/include/erofs/config.h
> index 2f09749..9125c1e 100644
> --- a/include/erofs/config.h
> +++ b/include/erofs/config.h
> @@ -36,6 +36,7 @@ struct erofs_configure {
>  	char *c_src_path;
>  	char *c_compr_alg_master;
>  	int c_compr_level_master;
> +	unsigned int c_compr_seg_size;	/* max segment compress size */

Add a command line argument "-S segment size" for this, and
update the manpage as well.

>  	int c_force_inodeversion;
>  	/* < 0, xattr disabled and INT_MAX, always use inline xattrs */
>  	int c_inline_xattr_tolerance;
> diff --git a/lib/compress.c b/lib/compress.c
> index 6cc68ed..8fdbfb2 100644
> --- a/lib/compress.c
> +++ b/lib/compress.c
> @@ -32,6 +32,8 @@ struct z_erofs_vle_compress_ctx {
>  
>  	erofs_blk_t blkaddr;	/* pointing to the next blkaddr */
>  	u16 clusterofs;
> +	unsigned int comprlimits;
> +	unsigned int comr_seg_size;

Let's update the whole logic as what we discussed on phone just now.

Thanks,
Gao Xiang

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, back to index

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-10 17:17 [PATCH v3] erofs-utils: introduce segment size to limit the max input stream Li Guifu via Linux-erofs
2020-06-15 14:43 ` Gao Xiang via Linux-erofs

Linux-EROFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-erofs/0 linux-erofs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-erofs linux-erofs/ https://lore.kernel.org/linux-erofs \
		linux-erofs@lists.ozlabs.org linux-erofs@ozlabs.org
	public-inbox-index linux-erofs

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.ozlabs.lists.linux-erofs


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git