[RFC][PATCH 0/4] zram: add zlib compression bckend support

[RFC][PATCH 0/4] zram: add zlib compression bckend support

[Sergey Senozhatsky]

Hello,

RFC

I'll just post this series as a separate thread, I guess, sorry if it makes
any inconvenience. Joonsoo will resend his patch series, so discussions
will `relocate' anyway.

This patchset uses a different, let's say traditional, zram/zcomp approach.
it defines a new zlib compression backend same way as lzo ad lz4 are defined.

The key difference is that zlib requires zstream for both compression and
decompression. zram has stream-less decompression path for lzo and lz4, and
it works perfectly fast. In order to support zlib we need decompression
path to *optionally require* zstream. I want to make ZCOMP_NEED_READ_ZSTRM
flag (backend requires zstream for decompression) backend dependent; so we
still will have fastest lzo/lz4 possible.

This is one of the reasons I didn't implement it using crypto api -- crypto
api requires tfm for compression and decompression. Which implies that read
now either
a) has to share idle streams list with write path, thus reads and writes will
become slower
b) has to define its own idle stream list. but it does
   1) limit the number of concurrently executed read operations (to the number
      of stremas in the list)
   2) increase memory usage by the module (each streams occupies pages for
      workspace buffers, etc.)

For the time being, crypto API does not provide stream-less decompression
functions, to the best of my knowledge.


I, frankly, tempted to rewrite zram to use crypto several times. But each
time I couldn't find a real reason. Yes, it *in theory* will give people
HUGE possibilities to select compression algorithms. But the question
is -- zram has been around for quite some years, so does anybody need this
flexibility? I can easily picture people selecting between

       ratio                         speed                       alg
  OK compression ratio             very fast                   LZO/LZ4
and
  very good comp ratio      eh... but good comp ratio           zlib


But anything in the middle is just anything in the middle, IMHO. I can't
convince myself that people really want to have
      "eh... comp ration" + "eh.. speed"
comp algorithm, for example.


>From https://code.google.com/p/lz4/ it seems that lzo+lz4+zlib is quite a
good package.

And zram obviously was missing the `other side' algorithm -- zlib, when IO speed
is not SO important.



I did some zlib backend testing. A copy paste from patch 0003:


Copy dir with the linux kernel to a zram device (du -sh 2.3G) and check
memory usage stats.

mm_stat fields:
        orig_data_size
        compr_data_size
        mem_used_total
        mem_limit
        mem_used_max
        zero_pages
        num_migrated

zlib
cat /sys/block/zram0/mm_stat
2522685440 1210486447 1230729216        0 1230729216     5461        0

lzo
cat /sys/block/zram0/mm_stat
2525872128 1713351248 1738387456        0 1738387456     4682        0

ZLIB uses 484+MiB less memory in the test.



Sergey Senozhatsky (4):
  zram: introduce zcomp_backend flags callback
  zram: extend zcomp_backend decompress callback
  zram: add zlib backend
  zram: enable zlib backend support

 drivers/block/zram/Kconfig      |  14 ++++-
 drivers/block/zram/Makefile     |   1 +
 drivers/block/zram/zcomp.c      |  30 +++++++++-
 drivers/block/zram/zcomp.h      |  12 +++-
 drivers/block/zram/zcomp_lz4.c  |   8 ++-
 drivers/block/zram/zcomp_lzo.c  |   8 ++-
 drivers/block/zram/zcomp_zlib.c | 120 ++++++++++++++++++++++++++++++++++++++++
 drivers/block/zram/zcomp_zlib.h |  17 ++++++
 drivers/block/zram/zram_drv.c   |  23 ++++++--
 9 files changed, 222 insertions(+), 11 deletions(-)
 create mode 100644 drivers/block/zram/zcomp_zlib.c
 create mode 100644 drivers/block/zram/zcomp_zlib.h

-- 
2.5.0

[RFC][PATCH 1/4] zram: introduce zcomp_backend flags callback

[Sergey Senozhatsky]

Add ->flags() callback to struct zcomp_backend and zcomp_flags()
function. Compression backends can define own flags and let
compression frontend/zram know them. This is a preparation
patch to enable ZLIB compression backend support.

This also adds two new functions to zcomp: zcomp_decompress_begin()
and zcomp_decompress_end(). These functions are here to hide
backend specific ->flags from zram. For backends that don't
require zstrm for read path zcomp_decompress_begin() will return
NULL, passing it to zcomp_decompress_end() will do no harm.
For backends that do require zstrm for decompression it will
provide opaque pointer to zstrm.

IOW, it goes like this:

  void *decomp_data = zcomp_decompress_begin(zram->comp);
  ret = zram_decompress_page(zram, decomp_data, uncmem, index);
  zcomp_decompress_end(zram->comp, decomp_data);

For backends that require decompress zstrm, zcomp_decompress_begin()
calls zcomp_strm_find(), so it can sleep.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
---
 drivers/block/zram/zcomp.c     | 9 +++++++++
 drivers/block/zram/zcomp.h     | 5 +++++
 drivers/block/zram/zcomp_lz4.c | 6 ++++++
 drivers/block/zram/zcomp_lzo.c | 6 ++++++
 4 files changed, 26 insertions(+)

diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 965d1af..61e06b4 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -325,6 +325,15 @@ void zcomp_destroy(struct zcomp *comp)
 	kfree(comp);
 }
 
+void *zcomp_decompress_begin(struct zcomp *comp)
+{
+	return NULL;
+}
+
+void zcomp_decompress_end(struct zcomp *comp, void *private)
+{
+}
+
 /*
  * search available compressors for requested algorithm.
  * allocate new zcomp and initialize it. return compressing
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index 46e2b9f..b8cf0b0 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -33,6 +33,8 @@ struct zcomp_backend {
 	int (*decompress)(const unsigned char *src, size_t src_len,
 			unsigned char *dst);
 
+	int (*flags)(void);
+
 	void *(*create)(void);
 	void (*destroy)(void *private);
 
@@ -56,6 +58,9 @@ bool zcomp_available_algorithm(const char *comp);
 struct zcomp *zcomp_create(const char *comp, int max_strm);
 void zcomp_destroy(struct zcomp *comp);
 
+void *zcomp_decompress_begin(struct zcomp *comp);
+void zcomp_decompress_end(struct zcomp *comp, void *private);
+
 struct zcomp_strm *zcomp_strm_find(struct zcomp *comp);
 void zcomp_strm_release(struct zcomp *comp, struct zcomp_strm *zstrm);
 
diff --git a/drivers/block/zram/zcomp_lz4.c b/drivers/block/zram/zcomp_lz4.c
index f2afb7e..99ed995 100644
--- a/drivers/block/zram/zcomp_lz4.c
+++ b/drivers/block/zram/zcomp_lz4.c
@@ -23,6 +23,11 @@ static void zcomp_lz4_destroy(void *private)
 	kfree(private);
 }
 
+static int zcomp_lz4_flags(void)
+{
+	return 0;
+}
+
 static int zcomp_lz4_compress(const unsigned char *src, unsigned char *dst,
 		size_t *dst_len, void *private)
 {
@@ -43,5 +48,6 @@ struct zcomp_backend zcomp_lz4 = {
 	.decompress = zcomp_lz4_decompress,
 	.create = zcomp_lz4_create,
 	.destroy = zcomp_lz4_destroy,
+	.flags = zcomp_lz4_flags,
 	.name = "lz4",
 };
diff --git a/drivers/block/zram/zcomp_lzo.c b/drivers/block/zram/zcomp_lzo.c
index da1bc47..7572d92 100644
--- a/drivers/block/zram/zcomp_lzo.c
+++ b/drivers/block/zram/zcomp_lzo.c
@@ -23,6 +23,11 @@ static void lzo_destroy(void *private)
 	kfree(private);
 }
 
+static int lzo_flags(void)
+{
+	return 0;
+}
+
 static int lzo_compress(const unsigned char *src, unsigned char *dst,
 		size_t *dst_len, void *private)
 {
@@ -43,5 +48,6 @@ struct zcomp_backend zcomp_lzo = {
 	.decompress = lzo_decompress,
 	.create = lzo_create,
 	.destroy = lzo_destroy,
+	.flags = lzo_flags,
 	.name = "lzo",
 };
-- 
2.5.0

[RFC][PATCH 2/4] zram: extend zcomp_backend decompress callback

[Sergey Senozhatsky]

This is a preparation patch to make ZLIB compression backend
support possible. ZLIB requires zstrm both for compress and
decompress operations, which is not needed for LZO and LZ4
backends. Extend zcomp_decompress() and zcomp_backend decompress
callback to optionally accept zstrm pointer. Pass NULL as
a zstrm pointer to LZO and LZ4 backends.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
---
 drivers/block/zram/zcomp.c     | 5 +++--
 drivers/block/zram/zcomp.h     | 5 +++--
 drivers/block/zram/zcomp_lz4.c | 2 +-
 drivers/block/zram/zcomp_lzo.c | 2 +-
 drivers/block/zram/zram_drv.c  | 2 +-
 5 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 61e06b4..a1c67be 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -313,10 +313,11 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
 			zstrm->private);
 }
 
-int zcomp_decompress(struct zcomp *comp, const unsigned char *src,
+int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,
+		const unsigned char *src,
 		size_t src_len, unsigned char *dst)
 {
-	return comp->backend->decompress(src, src_len, dst);
+	return comp->backend->decompress(src, src_len, dst, NULL);
 }
 
 void zcomp_destroy(struct zcomp *comp)
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index b8cf0b0..5cb9a0b 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -31,7 +31,7 @@ struct zcomp_backend {
 			size_t *dst_len, void *private);
 
 	int (*decompress)(const unsigned char *src, size_t src_len,
-			unsigned char *dst);
+			unsigned char *dst, void *private);
 
 	int (*flags)(void);
 
@@ -67,7 +67,8 @@ void zcomp_strm_release(struct zcomp *comp, struct zcomp_strm *zstrm);
 int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
 		const unsigned char *src, size_t *dst_len);
 
-int zcomp_decompress(struct zcomp *comp, const unsigned char *src,
+int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,
+		const unsigned char *src,
 		size_t src_len, unsigned char *dst);
 
 bool zcomp_set_max_streams(struct zcomp *comp, int num_strm);
diff --git a/drivers/block/zram/zcomp_lz4.c b/drivers/block/zram/zcomp_lz4.c
index 99ed995..09bf30c 100644
--- a/drivers/block/zram/zcomp_lz4.c
+++ b/drivers/block/zram/zcomp_lz4.c
@@ -36,7 +36,7 @@ static int zcomp_lz4_compress(const unsigned char *src, unsigned char *dst,
 }
 
 static int zcomp_lz4_decompress(const unsigned char *src, size_t src_len,
-		unsigned char *dst)
+		unsigned char *dst, void *private)
 {
 	size_t dst_len = PAGE_SIZE;
 	/* return  : Success if return 0 */
diff --git a/drivers/block/zram/zcomp_lzo.c b/drivers/block/zram/zcomp_lzo.c
index 7572d92..9cc2fa4 100644
--- a/drivers/block/zram/zcomp_lzo.c
+++ b/drivers/block/zram/zcomp_lzo.c
@@ -36,7 +36,7 @@ static int lzo_compress(const unsigned char *src, unsigned char *dst,
 }
 
 static int lzo_decompress(const unsigned char *src, size_t src_len,
-		unsigned char *dst)
+		unsigned char *dst, void *private)
 {
 	size_t dst_len = PAGE_SIZE;
 	int ret = lzo1x_decompress_safe(src, src_len, dst, &dst_len);
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 509023f..0aec4ce 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -582,7 +582,7 @@ static int zram_decompress_page(struct zram *zram, char *mem, u32 index)
 	if (size == PAGE_SIZE)
 		copy_page(mem, cmem);
 	else
-		ret = zcomp_decompress(zram->comp, cmem, size, mem);
+		ret = zcomp_decompress(zram->comp, NULL, cmem, size, mem);
 	zs_unmap_object(meta->mem_pool, handle);
 	bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value);
 
-- 
2.5.0

[RFC][PATCH 3/4] zram: add zlib backend

[Sergey Senozhatsky]

Add ZLIB backend (disabled yet, will be enalbed in later patch).

======================================================================
TEST

Copy dir with the linux kernel to a zram device (du -sh 2.3G) and check
memory usage stats.

mm_stat fields:
        orig_data_size
        compr_data_size
        mem_used_total
        mem_limit
        mem_used_max
        zero_pages
        num_migrated

zlib
cat /sys/block/zram0/mm_stat
2522685440 1210486447 1230729216        0 1230729216     5461        0

lzo
cat /sys/block/zram0/mm_stat
2525872128 1713351248 1738387456        0 1738387456     4682        0

ZLIB uses 484+MiB less memory in the test.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
---
 drivers/block/zram/Makefile     |   1 +
 drivers/block/zram/zcomp.c      |  11 ++++
 drivers/block/zram/zcomp.h      |   2 +
 drivers/block/zram/zcomp_zlib.c | 120 ++++++++++++++++++++++++++++++++++++++++
 drivers/block/zram/zcomp_zlib.h |  17 ++++++
 5 files changed, 151 insertions(+)
 create mode 100644 drivers/block/zram/zcomp_zlib.c
 create mode 100644 drivers/block/zram/zcomp_zlib.h

diff --git a/drivers/block/zram/Makefile b/drivers/block/zram/Makefile
index be0763f..0922f54 100644
--- a/drivers/block/zram/Makefile
+++ b/drivers/block/zram/Makefile
@@ -1,5 +1,6 @@
 zram-y	:=	zcomp_lzo.o zcomp.o zram_drv.o
 
 zram-$(CONFIG_ZRAM_LZ4_COMPRESS) += zcomp_lz4.o
+zram-$(CONFIG_ZRAM_ZLIB_COMPRESS) += zcomp_zlib.o
 
 obj-$(CONFIG_ZRAM)	+=	zram.o
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index a1c67be..a0cef0b 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -19,6 +19,9 @@
 #ifdef CONFIG_ZRAM_LZ4_COMPRESS
 #include "zcomp_lz4.h"
 #endif
+#ifdef CONFIG_ZRAM_ZLIB_COMPRESS
+#include "zcomp_zlib.h"
+#endif
 
 /*
  * single zcomp_strm backend
@@ -48,6 +51,9 @@ static struct zcomp_backend *backends[] = {
 #ifdef CONFIG_ZRAM_LZ4_COMPRESS
 	&zcomp_lz4,
 #endif
+#ifdef CONFIG_ZRAM_ZLIB_COMPRESS
+	&zcomp_zlib,
+#endif
 	NULL
 };
 
@@ -328,11 +334,16 @@ void zcomp_destroy(struct zcomp *comp)
 
 void *zcomp_decompress_begin(struct zcomp *comp)
 {
+	if (unlikely(comp->backend->flags() & ZCOMP_NEED_READ_ZSTRM))
+		return zcomp_strm_find(comp);
+
 	return NULL;
 }
 
 void zcomp_decompress_end(struct zcomp *comp, void *private)
 {
+	if (unlikely(private))
+		zcomp_strm_release(comp, private);
 }
 
 /*
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index 5cb9a0b..dbb7f1f 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -12,6 +12,8 @@
 
 #include <linux/mutex.h>
 
+#define ZCOMP_NEED_READ_ZSTRM	(1 << 0)
+
 struct zcomp_strm {
 	/* compression/decompression buffer */
 	void *buffer;
diff --git a/drivers/block/zram/zcomp_zlib.c b/drivers/block/zram/zcomp_zlib.c
new file mode 100644
index 0000000..711eddd
--- /dev/null
+++ b/drivers/block/zram/zcomp_zlib.c
@@ -0,0 +1,120 @@
+/*
+ * Copyright (C) 2015 Sergey Senozhatsky.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+#include <linux/zlib.h>
+
+#include "zcomp_zlib.h"
+
+#define ZLIB_COMPRESSION_LEVEL	3
+
+static void *zlib_create(void)
+{
+	z_stream *stream;
+	size_t size;
+
+	stream = kzalloc(sizeof(*stream), GFP_KERNEL);
+	if (!stream)
+		return NULL;
+
+	size = max(zlib_deflate_workspacesize(MAX_WBITS, MAX_MEM_LEVEL),
+			zlib_inflate_workspacesize());
+	stream->workspace = vmalloc(size);
+	if (!stream->workspace) {
+		kfree(stream);
+		stream = NULL;
+	}
+
+	return stream;
+}
+
+static void zlib_destroy(void *private)
+{
+	z_stream *stream = private;
+
+	vfree(stream->workspace);
+	kfree(stream);
+}
+
+static int zlib_flags(void)
+{
+	return ZCOMP_NEED_READ_ZSTRM;
+}
+
+static int zlib_compress(const unsigned char *src, unsigned char *dst,
+		size_t *dst_len, void *private)
+{
+	z_stream *stream = private;
+	int err;
+
+	err = zlib_deflateInit(stream, ZLIB_COMPRESSION_LEVEL);
+	if (err != Z_OK)
+		goto out;
+
+	stream->next_in = src;
+	stream->avail_in = PAGE_SIZE;
+	stream->total_in = 0;
+	stream->next_out = dst;
+	stream->avail_out = PAGE_SIZE;
+	stream->total_out = 0;
+
+	err = zlib_deflate(stream, Z_FINISH);
+	if (err != Z_STREAM_END)
+		goto out;
+
+	err = zlib_deflateEnd(stream);
+	if (err != Z_OK)
+		goto out;
+
+	if (stream->total_out >= stream->total_in)
+		goto out;
+
+	*dst_len = stream->total_out;
+out:
+	return err == Z_OK ? 0 : err;
+}
+
+static int zlib_decompress(const unsigned char *src, size_t src_len,
+		unsigned char *dst, void *private)
+{
+	z_stream *stream = private;
+	int err;
+
+	err = zlib_inflateInit(stream);
+	if (err != Z_OK)
+		goto out;
+
+	stream->next_in = src;
+	stream->avail_in = src_len;
+	stream->total_in = 0;
+	stream->next_out = dst;
+	stream->avail_out = PAGE_SIZE;
+	stream->total_out = 0;
+
+	err = zlib_inflate(stream, Z_FINISH);
+	if (err != Z_STREAM_END)
+		goto out;
+
+	err = zlib_inflateEnd(stream);
+	if (err != Z_OK)
+		goto out;
+out:
+	return err == Z_OK ? 0 : err;
+}
+
+struct zcomp_backend zcomp_zlib = {
+	.compress = zlib_compress,
+	.decompress = zlib_decompress,
+	.create = zlib_create,
+	.destroy = zlib_destroy,
+	.flags = zlib_flags,
+	.name = "zlib",
+};
diff --git a/drivers/block/zram/zcomp_zlib.h b/drivers/block/zram/zcomp_zlib.h
new file mode 100644
index 0000000..d0e4fa0
--- /dev/null
+++ b/drivers/block/zram/zcomp_zlib.h
@@ -0,0 +1,17 @@
+/*
+ * Copyright (C) 2015 Sergey Senozhatsky.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+
+#ifndef _ZCOMP_ZLIB_H_
+#define _ZCOMP_ZLIB_H_
+
+#include "zcomp.h"
+
+extern struct zcomp_backend zcomp_zlib;
+
+#endif /* _ZCOMP_ZLIB_H_ */
-- 
2.5.0

[RFC][PATCH 4/4] zram: enable zlib backend support

[Sergey Senozhatsky]

Update zram and zcomp read (decompress) path to provide a
zstrm to backends that set ZCOMP_NEED_READ_ZSTRM bit in flags,
so now we can enable ZLIB compression backend in Kconfig.

Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
---
 drivers/block/zram/Kconfig    | 14 +++++++++++++-
 drivers/block/zram/zcomp.c    |  7 ++++++-
 drivers/block/zram/zram_drv.c | 23 +++++++++++++++++++----
 3 files changed, 38 insertions(+), 6 deletions(-)

diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig
index 386ba3d..1858762 100644
--- a/drivers/block/zram/Kconfig
+++ b/drivers/block/zram/Kconfig
@@ -23,4 +23,16 @@ config ZRAM_LZ4_COMPRESS
 	default n
 	help
 	  This option enables LZ4 compression algorithm support. Compression
-	  algorithm can be changed using `comp_algorithm' device attribute.
\ No newline at end of file
+	  algorithm can be changed using `comp_algorithm' device attribute.
+
+config ZRAM_ZLIB_COMPRESS
+	bool "Enable ZLIB algorithm support"
+	depends on ZRAM
+	select ZLIB_INFLATE
+	select ZLIB_DEFLATE
+	default n
+	help
+	  This option enables ZLIB compression algorithm support. ZLIB gives
+	  a significantly better compression ratio, at a price of significantly
+	  worse performance. Compression algorithm can be changed using
+	  `comp_algorithm' device attribute.
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index a0cef0b..8f24820 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -323,7 +323,12 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,
 		const unsigned char *src,
 		size_t src_len, unsigned char *dst)
 {
-	return comp->backend->decompress(src, src_len, dst, NULL);
+	void *private = NULL;
+
+	if (unlikely(zstrm))
+		private = zstrm->private;
+
+	return comp->backend->decompress(src, src_len, dst, private);
 }
 
 void zcomp_destroy(struct zcomp *comp)
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 0aec4ce..5c52400 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -560,7 +560,8 @@ static void zram_free_page(struct zram *zram, size_t index)
 	zram_set_obj_size(meta, index, 0);
 }
 
-static int zram_decompress_page(struct zram *zram, char *mem, u32 index)
+static int zram_decompress_page(struct zram *zram, struct zcomp_strm *zstrm,
+		char *mem, u32 index)
 {
 	int ret = 0;
 	unsigned char *cmem;
@@ -582,7 +583,7 @@ static int zram_decompress_page(struct zram *zram, char *mem, u32 index)
 	if (size == PAGE_SIZE)
 		copy_page(mem, cmem);
 	else
-		ret = zcomp_decompress(zram->comp, NULL, cmem, size, mem);
+		ret = zcomp_decompress(zram->comp, zstrm, cmem, size, mem);
 	zs_unmap_object(meta->mem_pool, handle);
 	bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value);
 
@@ -602,6 +603,7 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
 	struct page *page;
 	unsigned char *user_mem, *uncmem = NULL;
 	struct zram_meta *meta = zram->meta;
+	void *dzstrm;
 	page = bvec->bv_page;
 
 	bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value);
@@ -617,6 +619,7 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
 		/* Use  a temporary buffer to decompress the page */
 		uncmem = kmalloc(PAGE_SIZE, GFP_NOIO);
 
+	dzstrm = zcomp_decompress_begin(zram->comp);
 	user_mem = kmap_atomic(page);
 	if (!is_partial_io(bvec))
 		uncmem = user_mem;
@@ -627,7 +630,11 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
 		goto out_cleanup;
 	}
 
-	ret = zram_decompress_page(zram, uncmem, index);
+	ret = zram_decompress_page(zram, dzstrm, uncmem, index);
+
+	zcomp_decompress_end(zram->comp, dzstrm);
+	dzstrm = NULL;
+
 	/* Should NEVER happen. Return bio error if it does. */
 	if (unlikely(ret))
 		goto out_cleanup;
@@ -638,10 +645,13 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
 
 	flush_dcache_page(page);
 	ret = 0;
+
 out_cleanup:
 	kunmap_atomic(user_mem);
+	zcomp_decompress_end(zram->comp, dzstrm);
 	if (is_partial_io(bvec))
 		kfree(uncmem);
+
 	return ret;
 }
 
@@ -659,6 +669,8 @@ static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, u32 index,
 
 	page = bvec->bv_page;
 	if (is_partial_io(bvec)) {
+		void *dzstrm;
+
 		/*
 		 * This is a partial IO. We need to read the full page
 		 * before to write the changes.
@@ -668,7 +680,10 @@ static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, u32 index,
 			ret = -ENOMEM;
 			goto out;
 		}
-		ret = zram_decompress_page(zram, uncmem, index);
+
+		dzstrm = zcomp_decompress_begin(zram->comp);
+		ret = zram_decompress_page(zram, dzstrm, uncmem, index);
+		zcomp_decompress_end(zram->comp, dzstrm);
 		if (ret)
 			goto out;
 	}
-- 
2.5.0

Re: [RFC][PATCH 0/4] zram: add zlib compression bckend support

[Minchan Kim]

Hello Sergey,

I talked with Joonsoo today and he has no time to support it at the moment
and I can't wait zlib support for zram until crypto work is merged.
So, I want to merge your work.
If you have an interest, still, could you mind resending the work
rebased on recent zram?

Thanks.

On Thu, Aug 13, 2015 at 10:55:19PM +0900, Sergey Senozhatsky wrote:
> Hello,
> 
> RFC
> 
> I'll just post this series as a separate thread, I guess, sorry if it makes
> any inconvenience. Joonsoo will resend his patch series, so discussions
> will `relocate' anyway.
> 
> This patchset uses a different, let's say traditional, zram/zcomp approach.
> it defines a new zlib compression backend same way as lzo ad lz4 are defined.
> 
> The key difference is that zlib requires zstream for both compression and
> decompression. zram has stream-less decompression path for lzo and lz4, and
> it works perfectly fast. In order to support zlib we need decompression
> path to *optionally require* zstream. I want to make ZCOMP_NEED_READ_ZSTRM
> flag (backend requires zstream for decompression) backend dependent; so we
> still will have fastest lzo/lz4 possible.
> 
> This is one of the reasons I didn't implement it using crypto api -- crypto
> api requires tfm for compression and decompression. Which implies that read
> now either
> a) has to share idle streams list with write path, thus reads and writes will
> become slower
> b) has to define its own idle stream list. but it does
>    1) limit the number of concurrently executed read operations (to the number
>       of stremas in the list)
>    2) increase memory usage by the module (each streams occupies pages for
>       workspace buffers, etc.)
> 
> For the time being, crypto API does not provide stream-less decompression
> functions, to the best of my knowledge.
> 
> 
> I, frankly, tempted to rewrite zram to use crypto several times. But each
> time I couldn't find a real reason. Yes, it *in theory* will give people
> HUGE possibilities to select compression algorithms. But the question
> is -- zram has been around for quite some years, so does anybody need this
> flexibility? I can easily picture people selecting between
> 
>        ratio                         speed                       alg
>   OK compression ratio             very fast                   LZO/LZ4
> and
>   very good comp ratio      eh... but good comp ratio           zlib
> 
> 
> But anything in the middle is just anything in the middle, IMHO. I can't
> convince myself that people really want to have
>       "eh... comp ration" + "eh.. speed"
> comp algorithm, for example.
> 
> 
> From https://code.google.com/p/lz4/ it seems that lzo+lz4+zlib is quite a
> good package.
> 
> And zram obviously was missing the `other side' algorithm -- zlib, when IO speed
> is not SO important.
> 
> 
> 
> I did some zlib backend testing. A copy paste from patch 0003:
> 
> 
> Copy dir with the linux kernel to a zram device (du -sh 2.3G) and check
> memory usage stats.
> 
> mm_stat fields:
>         orig_data_size
>         compr_data_size
>         mem_used_total
>         mem_limit
>         mem_used_max
>         zero_pages
>         num_migrated
> 
> zlib
> cat /sys/block/zram0/mm_stat
> 2522685440 1210486447 1230729216        0 1230729216     5461        0
> 
> lzo
> cat /sys/block/zram0/mm_stat
> 2525872128 1713351248 1738387456        0 1738387456     4682        0
> 
> ZLIB uses 484+MiB less memory in the test.
> 
> 
> 
> Sergey Senozhatsky (4):
>   zram: introduce zcomp_backend flags callback
>   zram: extend zcomp_backend decompress callback
>   zram: add zlib backend
>   zram: enable zlib backend support
> 
>  drivers/block/zram/Kconfig      |  14 ++++-
>  drivers/block/zram/Makefile     |   1 +
>  drivers/block/zram/zcomp.c      |  30 +++++++++-
>  drivers/block/zram/zcomp.h      |  12 +++-
>  drivers/block/zram/zcomp_lz4.c  |   8 ++-
>  drivers/block/zram/zcomp_lzo.c  |   8 ++-
>  drivers/block/zram/zcomp_zlib.c | 120 ++++++++++++++++++++++++++++++++++++++++
>  drivers/block/zram/zcomp_zlib.h |  17 ++++++
>  drivers/block/zram/zram_drv.c   |  23 ++++++--
>  9 files changed, 222 insertions(+), 11 deletions(-)
>  create mode 100644 drivers/block/zram/zcomp_zlib.c
>  create mode 100644 drivers/block/zram/zcomp_zlib.h
> 
> -- 
> 2.5.0
> 

Re: [RFC][PATCH 0/4] zram: add zlib compression bckend support

[Sergey Senozhatsky]

Hello Minchan,

On (05/23/16 14:53), Minchan Kim wrote:
> Hello Sergey,
> 
> I talked with Joonsoo today and he has no time to support it at the moment
> and I can't wait zlib support for zram until crypto work is merged.
> So, I want to merge your work.

oh, I see.

> If you have an interest, still, could you mind resending the work
> rebased on recent zram?

ok, sure. will re-visit, re-test and resend.

> Thanks.

Thanks.

	-ss