[PATCH 0/8] makedumpfile: Add Snappy Compression Support

[PATCH 0/8] makedumpfile: Add Snappy Compression Support

[HATAYAMA Daisuke]

This patch series add snappy compression support, applied on top of
v1.4.4. snappy is fast compressoin algorhythm like lzo, but it's more
optimized than lzo on x86 machines and some others.

A lot of benchmark is available on the web. Here I only point at my
benchmark I posted a few weeks ago,

- http://lists.infradead.org/pipermail/kexec/2012-June/006425.html

  where snappy shows mostly as good performance as simple copying on
  the worst case that input data increases its data size during
  compression; this means we can use snappy with mostly NO risk.

How to get snappy libraries:

  1) Use yum framework to get snappy and snappy-devel packages, or

  2) Visit official website: http://code.google.com/p/snappy/,
    download snappy-<version>.tar.gz and then build it.

How to build makedumpfile with snappy support:

  Do make as follows:

  $ make USESNAPPY=on

---

HATAYAMA Daisuke (8):
      Add manual description
      Add help message
      Add uncompression processing
      Add compression processing
      Notify snappy unsupporting when disabled
      Add snappy build support
      Add command-line processing for snappy
      Add dump header for snappy


 Makefile       |    5 +++++
 diskdump_mod.h |    2 ++
 makedumpfile.8 |    9 +++++---
 makedumpfile.c |   60 ++++++++++++++++++++++++++++++++++++++++++++++++++------
 makedumpfile.h |    3 +++
 print_info.c   |   16 ++++++++++-----
 6 files changed, 80 insertions(+), 15 deletions(-)

-- 

Thanks.
HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH 1/8] Add dump header for snappy

[HATAYAMA Daisuke]

Signed-off-by: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com>
---

 diskdump_mod.h |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/diskdump_mod.h b/diskdump_mod.h
index e672485..af060b6 100644
--- a/diskdump_mod.h
+++ b/diskdump_mod.h
@@ -80,6 +80,8 @@ struct kdump_sub_header {
 /* page flags */
 #define DUMP_DH_COMPRESSED_ZLIB	0x1	/* page is compressed with zlib */
 #define DUMP_DH_COMPRESSED_LZO	0x2	/* paged is compressed with lzo */
+#define DUMP_DH_COMPRESSED_SNAPPY	0x4
+					/* paged is compressed with snappy */
 
 /* descriptor of each page for vmcore */
 typedef struct page_desc {


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH 2/8] Add command-line processing for snappy

[HATAYAMA Daisuke]

Specify -p option to compress vmcore in snappy.

Signed-off-by: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com>
---

 makedumpfile.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index d024e95..14fd9f6 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -6968,7 +6968,7 @@ main(int argc, char *argv[])
 
 	info->block_order = DEFAULT_ORDER;
 	message_level = DEFAULT_MSG_LEVEL;
-	while ((opt = getopt_long(argc, argv, "b:cDd:EFfg:hi:lMRrsvXx:", longopts,
+	while ((opt = getopt_long(argc, argv, "b:cDd:EFfg:hi:lMpRrsvXx:", longopts,
 	    NULL)) != -1) {
 		switch (opt) {
 		case 'b':
@@ -7026,6 +7026,9 @@ main(int argc, char *argv[])
 		case 'M':
 			info->flag_dmesg = 1;
 			break;
+		case 'p':
+			info->flag_compress = DUMP_DH_COMPRESSED_SNAPPY;
+			break;
 		case 'P':
 			info->xen_phys_start = strtoul(optarg, NULL, 0);
 			break;


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH 3/8] Add snappy build support

[HATAYAMA Daisuke]

Signed-off-by: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com>
---

 Makefile |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/Makefile b/Makefile
index 7337c58..8e5d72d 100644
--- a/Makefile
+++ b/Makefile
@@ -55,6 +55,11 @@ LIBS := -llzo2 $(LIBS)
 CFLAGS += -DUSELZO
 endif
 
+ifeq ($(USESNAPPY), on)
+LIBS := -lsnappy $(LIBS)
+CFLAGS += -DUSESNAPPY
+endif
+
 all: makedumpfile
 
 $(OBJ_PART): $(SRC_PART)


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH 4/8] Notify snappy unsupporting when disabled

[HATAYAMA Daisuke]

Signed-off-by: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com>
---

 makedumpfile.c |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index 14fd9f6..a47a3ec 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -2538,6 +2538,15 @@ initial(void)
 	}
 #endif
 
+#ifndef USESNAPPY
+	if (info->flag_compress == DUMP_DH_COMPRESSED_SNAPPY) {
+		MSG("'-p' option is disabled, ");
+		MSG("because this binary doesn't support snappy "
+		    "compression.\n");
+		MSG("Try `make USESNAPPY=on` when building.\n");
+	}
+#endif
+
 	if (!is_xen_memory() && info->flag_exclude_xen_dom) {
 		MSG("'-X' option is disable,");
 		MSG("because %s is not Xen's memory core image.\n", info->name_memory);


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH 5/8] Add compression processing

[HATAYAMA Daisuke]

Signed-off-by: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com>
---

 makedumpfile.c |   30 +++++++++++++++++++++++++-----
 makedumpfile.h |    3 +++
 2 files changed, 28 insertions(+), 5 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index a47a3ec..1525b30 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -4718,6 +4718,7 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 	struct dump_bitmap bitmap2;
 	struct timeval tv_start;
 	const off_t failed = (off_t)-1;
+	unsigned long len_buf_out_zlib, len_buf_out_lzo, len_buf_out_snappy;
 
 	int ret = FALSE;
 
@@ -4726,8 +4727,9 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 
 	initialize_2nd_bitmap(&bitmap2);
 
+	len_buf_out_zlib = len_buf_out_lzo = len_buf_out_snappy = 0;
+
 #ifdef USELZO
-	unsigned long len_buf_out_zlib, len_buf_out_lzo;
 	lzo_bytep wrkmem;
 
 	if ((wrkmem = malloc(LZO1X_1_MEM_COMPRESS)) == NULL) {
@@ -4736,13 +4738,19 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 		goto out;
 	}
 
-	len_buf_out_zlib = compressBound(info->page_size);
 	len_buf_out_lzo = info->page_size + info->page_size / 16 + 64 + 3;
-	len_buf_out = MAX(len_buf_out_zlib, len_buf_out_lzo);
-#else
-	len_buf_out = compressBound(info->page_size);
 #endif
 
+#ifdef USESNAPPY
+	len_buf_out_snappy = snappy_max_compressed_length(info->page_size);
+#endif
+
+	len_buf_out_zlib = compressBound(info->page_size);
+	
+	len_buf_out = MAX(len_buf_out_zlib,
+			  MAX(len_buf_out_lzo,
+			      len_buf_out_snappy));
+
 	if ((buf_out = malloc(len_buf_out)) == NULL) {
 		ERRMSG("Can't allocate memory for the compression buffer. %s\n",
 		    strerror(errno));
@@ -4843,6 +4851,18 @@ write_kdump_pages(struct cache_data *cd_header, struct cache_data *cd_page)
 			pd.size  = size_out;
 			memcpy(buf, buf_out, pd.size);
 #endif
+#ifdef USESNAPPY
+		} else if ((info->flag_compress & DUMP_DH_COMPRESSED_SNAPPY)
+			   && ((size_out = len_buf_out_snappy),
+			       snappy_compress((char *)buf, info->page_size,
+					       (char *)buf_out,
+					       (size_t *)&size_out)
+			       == SNAPPY_OK)
+			   && (size_out < info->page_size)) {
+			pd.flags = DUMP_DH_COMPRESSED_SNAPPY;
+			pd.size  = size_out;
+			memcpy(buf, buf_out, pd.size);
+#endif
 		} else {
 			pd.flags = 0;
 			pd.size  = info->page_size;
diff --git a/makedumpfile.h b/makedumpfile.h
index 6f5489d..72d8327 100644
--- a/makedumpfile.h
+++ b/makedumpfile.h
@@ -34,6 +34,9 @@
 #ifdef USELZO
 #include <lzo/lzo1x.h>
 #endif
+#ifdef USESNAPPY
+#include <snappy-c.h>
+#endif
 #include "common.h"
 #include "dwarf_info.h"
 #include "diskdump_mod.h"


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH 6/8] Add uncompression processing

[HATAYAMA Daisuke]

Signed-off-by: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com>
---

 makedumpfile.c |   16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/makedumpfile.c b/makedumpfile.c
index 1525b30..e42b07d 100644
--- a/makedumpfile.c
+++ b/makedumpfile.c
@@ -279,6 +279,22 @@ readpmem_kdump_compressed(unsigned long long paddr, void *bufptr, size_t size)
 		}
 		memcpy(bufptr, buf2 + page_offset, size);
 #endif
+#ifdef USESNAPPY
+	} else if ((pd.flags & DUMP_DH_COMPRESSED_SNAPPY)) {
+
+		ret = snappy_uncompressed_length(buf, pd.size, &retlen);
+		if (ret != SNAPPY_OK) {
+			ERRMSG("Uncompress failed: %d\n", ret);
+			goto error;
+		}
+
+		ret = snappy_uncompress(buf, pd.size, buf2, &retlen);
+		if ((ret != SNAPPY_OK) || (retlen != info->page_size)) {
+			ERRMSG("Uncompress failed: %d\n", ret);
+			goto error;
+		}
+		memcpy(bufptr, buf2 + page_offset, size);
+#endif
 	} else
 		memcpy(bufptr, buf + page_offset, size);
 


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH 7/8] Add help message

[HATAYAMA Daisuke]

Signed-off-by: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com>
---

 print_info.c |   16 +++++++++++-----
 1 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/print_info.c b/print_info.c
index 61cafed..1b455b2 100644
--- a/print_info.c
+++ b/print_info.c
@@ -39,6 +39,12 @@ print_usage(void)
 #else
 	MSG("  disabled ('-l' option will be ignored.)\n");
 #endif
+	MSG("snappy support:\n");
+#ifdef USESNAPPY
+	MSG("  enabled\n");
+#else
+	MSG("  disabled ('-p' option will be ignored.)\n");
+#endif
 	MSG("\n");
 	MSG("Usage:\n");
 	MSG("  Creating DUMPFILE:\n");
@@ -83,10 +89,10 @@ print_usage(void)
 	MSG("\n");
 	MSG("\n");
 	MSG("Available options:\n");
-	MSG("  [-c|-l]:\n");
-	MSG("      Compress dump data by each page using zlib for -c option and lzo for\n");
-	MSG("      -l option. A user cannot specify either of these options with -E option,\n");
-	MSG("      because the ELF format does not support compressed data.\n");
+	MSG("  [-c|-l|-p]:\n");
+	MSG("      Compress dump data by each page using zlib for -c option, lzo for -l option\n");
+	MSG("      or snappy for -p option. A user cannot specify either of these options with\n");
+	MSG("      -E option, because the ELF format does not support compressed data.\n");
 	MSG("      THIS IS ONLY FOR THE CRASH UTILITY.\n");
 	MSG("\n");
 	MSG("  [-d DL]:\n");
@@ -222,7 +228,7 @@ print_usage(void)
 	MSG("      Overwrite DUMPFILE even if it already exists.\n");
 	MSG("\n");
 	MSG("  [-h]:\n");
-	MSG("      Show help message and LZO support status (enabled/disabled).\n");
+	MSG("      Show help message and LZO/snappy support status (enabled/disabled).\n");
 	MSG("\n");
 	MSG("  [-b <order>]\n");
 	MSG("      Specify the cache 2^order pages in ram when generating vmcore info\n");


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

[PATCH 8/8] Add manual description

[HATAYAMA Daisuke]

Signed-off-by: HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com>
---

 makedumpfile.8 |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/makedumpfile.8 b/makedumpfile.8
index 1fb9756..e9d4cbe 100644
--- a/makedumpfile.8
+++ b/makedumpfile.8
@@ -121,9 +121,10 @@ configuration, you need to use --diskset option.
 .SH OPTIONS
 
 .TP
-\fB\-c,\-l\fR
-Compress dump data by each page using zlib for -c option or lzo for -l option.
-(-l option needs USELZO=on when building.)
+\fB\-c,\-l,\-p\fR
+Compress dump data by each page using zlib for -c option, lzo for -l
+option or snappy for -p option.
+(-l option needs USELZO=on and -p option needs USESNAPPY=on when building)
 .br
 A user cannot specify this option with \-E option, because the ELF format does
 not support compressed data.
@@ -492,7 +493,7 @@ Print debugging message.
 
 .TP
 \fB\-h\fR
-Show help message and LZO support status (enabled/disabled).
+Show help message and LZO/snappy support status (enabled/disabled).
 
 .TP
 \fB\-v\fR


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/8] makedumpfile: Add Snappy Compression Support

[Atsushi Kumagai]

Hello HATAYAMA-san,

On Tue, 03 Jul 2012 12:07:27 +0900
HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com> wrote:

> This patch series add snappy compression support, applied on top of
> v1.4.4. snappy is fast compressoin algorhythm like lzo, but it's more
> optimized than lzo on x86 machines and some others.
> 
> A lot of benchmark is available on the web. Here I only point at my
> benchmark I posted a few weeks ago,
> 
> - http://lists.infradead.org/pipermail/kexec/2012-June/006425.html
> 
>   where snappy shows mostly as good performance as simple copying on
>   the worst case that input data increases its data size during
>   compression; this means we can use snappy with mostly NO risk.
> 
> How to get snappy libraries:
> 
>   1) Use yum framework to get snappy and snappy-devel packages, or
> 
>   2) Visit official website: http://code.google.com/p/snappy/,
>     download snappy-<version>.tar.gz and then build it.
> 
> How to build makedumpfile with snappy support:
> 
>   Do make as follows:
> 
>   $ make USESNAPPY=on

Thank you for your work.

I will merge your patches into the next version with small addition:

diff --git a/README b/README
index ae986d1..638b111 100644
--- a/README
+++ b/README
@@ -45,6 +45,9 @@
   6.Build with lzo support:
     # make USELZO=on ; make install
     The user has to prepare lzo library.
+  7.Build with snappy support:
+    # make USESNAPPY=on ; make install
+    The user has to prepare snappy library.

 * SUPPORTED KERNELS
   This makedumpfile supports the following kernels.


In addition, I did brief performance test with your patches:
  - The source data is a vmcore saved on the disk, it might be sparse data.
  - makedumpfile writes dumpfile to the same disk as the source data.
  - execution time is average of 5 times.

                        |    source     |    zlib     |     LZO     |   snappy  
------------------------+---------------+-------------+-------------+-------------
       size (byte)      | 5,107,498,116 | 242,398,239 | 309,549,499 | 487,542,710 
  compression  ratio(%) |      ---      |     4.75    |     6.06    |    9.55
   execution time (sec) |      ---      |      143    |     49.6    |    51.2


It seems that LZO still has worth in the case that good compression ratio is expected,
as you said in your benchmark report.


Thanks
Atsushi Kumagai

> ---
> 
> HATAYAMA Daisuke (8):
>       Add manual description
>       Add help message
>       Add uncompression processing
>       Add compression processing
>       Notify snappy unsupporting when disabled
>       Add snappy build support
>       Add command-line processing for snappy
>       Add dump header for snappy
> 
> 
>  Makefile       |    5 +++++
>  diskdump_mod.h |    2 ++
>  makedumpfile.8 |    9 +++++---
>  makedumpfile.c |   60 ++++++++++++++++++++++++++++++++++++++++++++++++++------
>  makedumpfile.h |    3 +++
>  print_info.c   |   16 ++++++++++-----
>  6 files changed, 80 insertions(+), 15 deletions(-)
> 
> -- 
> 
> Thanks.
> HATAYAMA, Daisuke

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: [PATCH 0/8] makedumpfile: Add Snappy Compression Support

[HATAYAMA Daisuke]

From: Atsushi Kumagai <kumagai-atsushi@mxc.nes.nec.co.jp>
Subject: Re: [PATCH 0/8] makedumpfile: Add Snappy Compression Support
Date: Thu, 5 Jul 2012 14:05:59 +0900

> On Tue, 03 Jul 2012 12:07:27 +0900
> HATAYAMA Daisuke <d.hatayama@jp.fujitsu.com> wrote:

> 
> Thank you for your work.
> 
> I will merge your patches into the next version with small addition:
> 
> diff --git a/README b/README
> index ae986d1..638b111 100644
> --- a/README
> +++ b/README
> @@ -45,6 +45,9 @@
>    6.Build with lzo support:
>      # make USELZO=on ; make install
>      The user has to prepare lzo library.
> +  7.Build with snappy support:
> +    # make USESNAPPY=on ; make install
> +    The user has to prepare snappy library.
> 
>  * SUPPORTED KERNELS
>    This makedumpfile supports the following kernels.
> 

Thanks. I overlooked README file...

> 
> In addition, I did brief performance test with your patches:
>   - The source data is a vmcore saved on the disk, it might be sparse data.
>   - makedumpfile writes dumpfile to the same disk as the source data.
>   - execution time is average of 5 times.
> 
>                         |    source     |    zlib     |     LZO     |   snappy  
> ------------------------+---------------+-------------+-------------+-------------
>        size (byte)      | 5,107,498,116 | 242,398,239 | 309,549,499 | 487,542,710 
>   compression  ratio(%) |      ---      |     4.75    |     6.06    |    9.55
>    execution time (sec) |      ---      |      143    |     49.6    |    51.2
> 
> 
> It seems that LZO still has worth in the case that good compression ratio is expected,
> as you said in your benchmark report.
> 

Yes, this is just as I'm assuming. I think it's better to use snappy
for the machine with hundreads of gigabytes or more. For the machine
with less than that, it's still reasonable to use LZO because risk at
the worst case is small.

Also, snappy is well optimized for little endian, 64-bit
machines. Though I don't see this actually to be honest, this might
show considerably bad performance on other archs except for x86_64.

Thanks.
HATAYAMA, Daisuke


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec