* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-26 6:24 ` Kyungsik Lee
0 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-26 6:24 UTC (permalink / raw)
To: Andrew Morton, Russell King, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev
Cc: Nicolas Pitre, Nitin Gupta, Markus F.X.J. Oberhumer,
Richard Purdie, Josh Triplett, Joe Millenbach, David Sterba,
Richard Cochran, Albin Tonnerre, Egon Alter, hyojun.im,
chan.jeong, raphael.andy.lee, Kyungsik Lee
Hi,
First of all, Thank you for the comments and emails from the community.
Here is the second version of support for LZ4-compressed kernel.
In this version, lz4_decompress() has been added. In case of knowing
the uncompressed data size, this function can be used to decompress
more faster.
Through the benchmark, it was found that -Os Compiler flag for
decompress.o brought better decompression performance in most of cases
(ex, different compiler and hardware spec.) in ARM architecture.
Lastly, CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is not always the best
option even though it is supported. The decompression speed can be
slightly slower in some cases.
This patchset is based on 3.8.
Any comments are appreciated.
Thanks,
Kyungsik
Benchmark Results(PATCH v2)
Compiler: Linaro ARM gcc 4.6.2
1. ARMv7, 1.5GHz based board
Kernel: linux 3.4
Uncompressed Kernel Size: 14MB
Compressed Size Decompression Speed
LZO 6.7MB 21.1MB/s
LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA)
2. ARMv7, 1.7GHz based board
Kernel: linux 3.7
Uncompressed Kernel Size: 14MB
Compressed Size Decompression Speed
LZO 6.0MB 34.1MB/s
LZ4 6.5MB 86.7MB/s
UA: Unaligned memory Access support
Change log: v2
- Clean up code
- Enable unaligned access for ARM v6 and above with
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
- Add lz4_decompress() for faster decompression with
uncompressed output size
- Use lz4_decompress() for LZ4-compressed kernel during
boot-process
- Apply -Os to decompress.o to improve decompress
performance during boot-up process
Kyungsik Lee (4):
decompressor: Add LZ4 decompressor module
lib: Add support for LZ4-compressed kernel
arm: Add support for LZ4-compressed kernel
x86: Add support for LZ4-compressed kernel
arch/arm/Kconfig | 1 +
arch/arm/boot/compressed/.gitignore | 1 +
arch/arm/boot/compressed/Makefile | 6 +-
arch/arm/boot/compressed/decompress.c | 4 +
arch/arm/boot/compressed/piggy.lz4.S | 6 +
arch/x86/Kconfig | 1 +
arch/x86/boot/compressed/Makefile | 5 +-
arch/x86/boot/compressed/misc.c | 4 +
include/linux/decompress/unlz4.h | 10 +
include/linux/lz4.h | 48 +++++
init/Kconfig | 13 +-
lib/Kconfig | 7 +
lib/Makefile | 2 +
lib/decompress.c | 5 +
lib/decompress_unlz4.c | 190 +++++++++++++++++++
lib/lz4/Makefile | 1 +
lib/lz4/lz4_decompress.c | 331 ++++++++++++++++++++++++++++++++++
lib/lz4/lz4defs.h | 93 ++++++++++
scripts/Makefile.lib | 5 +
usr/Kconfig | 9 +
20 files changed, 739 insertions(+), 3 deletions(-)
create mode 100644 arch/arm/boot/compressed/piggy.lz4.S
create mode 100644 include/linux/decompress/unlz4.h
create mode 100644 include/linux/lz4.h
create mode 100644 lib/decompress_unlz4.c
create mode 100644 lib/lz4/Makefile
create mode 100644 lib/lz4/lz4_decompress.c
create mode 100644 lib/lz4/lz4defs.h
--
1.8.1.1
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-26 6:24 ` Kyungsik Lee
0 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-26 6:24 UTC (permalink / raw)
To: linux-arm-kernel
Hi,
First of all, Thank you for the comments and emails from the community.
Here is the second version of support for LZ4-compressed kernel.
In this version, lz4_decompress() has been added. In case of knowing
the uncompressed data size, this function can be used to decompress
more faster.
Through the benchmark, it was found that -Os Compiler flag for
decompress.o brought better decompression performance in most of cases
(ex, different compiler and hardware spec.) in ARM architecture.
Lastly, CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is not always the best
option even though it is supported. The decompression speed can be
slightly slower in some cases.
This patchset is based on 3.8.
Any comments are appreciated.
Thanks,
Kyungsik
Benchmark Results(PATCH v2)
Compiler: Linaro ARM gcc 4.6.2
1. ARMv7, 1.5GHz based board
Kernel: linux 3.4
Uncompressed Kernel Size: 14MB
Compressed Size Decompression Speed
LZO 6.7MB 21.1MB/s
LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA)
2. ARMv7, 1.7GHz based board
Kernel: linux 3.7
Uncompressed Kernel Size: 14MB
Compressed Size Decompression Speed
LZO 6.0MB 34.1MB/s
LZ4 6.5MB 86.7MB/s
UA: Unaligned memory Access support
Change log: v2
- Clean up code
- Enable unaligned access for ARM v6 and above with
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
- Add lz4_decompress() for faster decompression with
uncompressed output size
- Use lz4_decompress() for LZ4-compressed kernel during
boot-process
- Apply -Os to decompress.o to improve decompress
performance during boot-up process
Kyungsik Lee (4):
decompressor: Add LZ4 decompressor module
lib: Add support for LZ4-compressed kernel
arm: Add support for LZ4-compressed kernel
x86: Add support for LZ4-compressed kernel
arch/arm/Kconfig | 1 +
arch/arm/boot/compressed/.gitignore | 1 +
arch/arm/boot/compressed/Makefile | 6 +-
arch/arm/boot/compressed/decompress.c | 4 +
arch/arm/boot/compressed/piggy.lz4.S | 6 +
arch/x86/Kconfig | 1 +
arch/x86/boot/compressed/Makefile | 5 +-
arch/x86/boot/compressed/misc.c | 4 +
include/linux/decompress/unlz4.h | 10 +
include/linux/lz4.h | 48 +++++
init/Kconfig | 13 +-
lib/Kconfig | 7 +
lib/Makefile | 2 +
lib/decompress.c | 5 +
lib/decompress_unlz4.c | 190 +++++++++++++++++++
lib/lz4/Makefile | 1 +
lib/lz4/lz4_decompress.c | 331 ++++++++++++++++++++++++++++++++++
lib/lz4/lz4defs.h | 93 ++++++++++
scripts/Makefile.lib | 5 +
usr/Kconfig | 9 +
20 files changed, 739 insertions(+), 3 deletions(-)
create mode 100644 arch/arm/boot/compressed/piggy.lz4.S
create mode 100644 include/linux/decompress/unlz4.h
create mode 100644 include/linux/lz4.h
create mode 100644 lib/decompress_unlz4.c
create mode 100644 lib/lz4/Makefile
create mode 100644 lib/lz4/lz4_decompress.c
create mode 100644 lib/lz4/lz4defs.h
--
1.8.1.1
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 1/4] decompressor: Add LZ4 decompressor module
2013-02-26 6:24 ` Kyungsik Lee
@ 2013-02-26 6:24 ` Kyungsik Lee
-1 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-26 6:24 UTC (permalink / raw)
To: Andrew Morton, Russell King, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev
Cc: Nicolas Pitre, Nitin Gupta, Markus F.X.J. Oberhumer,
Richard Purdie, Josh Triplett, Joe Millenbach, David Sterba,
Richard Cochran, Albin Tonnerre, Egon Alter, hyojun.im,
chan.jeong, raphael.andy.lee, Kyungsik Lee
This patch adds support for LZ4 decompression in the Linux Kernel.
LZ4 Decompression APIs for kernel are based on LZ4 implementation
by Yann Collet.
LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
LZ4 source repository : http://code.google.com/p/lz4/
Signed-off-by: Kyungsik Lee <kyungsik.lee@lge.com>
v2:
- Clean up code
- Enable unaligned access for ARM v6 and above with
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
- Add lz4_decompress() for faster decompression with
uncompressed output size
---
include/linux/lz4.h | 48 +++++++
lib/lz4/lz4_decompress.c | 331 +++++++++++++++++++++++++++++++++++++++++++++++
lib/lz4/lz4defs.h | 93 +++++++++++++
3 files changed, 472 insertions(+)
create mode 100644 include/linux/lz4.h
create mode 100644 lib/lz4/lz4_decompress.c
create mode 100644 lib/lz4/lz4defs.h
diff --git a/include/linux/lz4.h b/include/linux/lz4.h
new file mode 100644
index 0000000..66b504c
--- /dev/null
+++ b/include/linux/lz4.h
@@ -0,0 +1,48 @@
+#ifndef __LZ4_H__
+#define __LZ4_H__
+/*
+ * LZ4 Kernel Interface
+ *
+ * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+/*
+ * LZ4_COMPRESSBOUND()
+ * Provides the maximum size that LZ4 may output in a "worst case" scenario
+ * (input data not compressible)
+ */
+#define LZ4_COMPRESSBOUND(isize) (isize + ((isize)/255) + 16)
+
+/*
+ * lz4_decompress()
+ * src : source address of the compressed data
+ * src_len : is the input size, whcih is returned after decompress done
+ * dest : output buffer address of the decompressed data
+ * actual_dest_len: is the size of uncompressed data, supposing it's known
+ * return : Success if return 0
+ * Error if return (< 0)
+ * note : Destination buffer must be already allocated.
+ * a bit faster than lz4_decompress_unknownoutputsize()
+ */
+int lz4_decompress(const char *src, size_t *src_len, char *dest,
+ size_t actual_dest_len);
+
+/*
+ * lz4_decompress_unknownoutputsize()
+ * src : source address of the compressed data
+ * src_len : is the input size, therefore the compressed size
+ * dest : output buffer address of the decompressed data
+ * dest_len: is the max size of the destination buffer, which is
+ * returned with actual size of decompressed data after
+ * decompress done
+ * return : Success if return 0
+ * Error if return (< 0)
+ * note : Destination buffer must be already allocated.
+ */
+int lz4_decompress_unknownoutputsize(const char *src, size_t src_len,
+ char *dest, size_t *dest_len);
+#endif
diff --git a/lib/lz4/lz4_decompress.c b/lib/lz4/lz4_decompress.c
new file mode 100644
index 0000000..1998d7a
--- /dev/null
+++ b/lib/lz4/lz4_decompress.c
@@ -0,0 +1,331 @@
+/*
+ * LZ4 Decompressor for Linux kernel
+ *
+ * Copyright (C) 2013 LG Electronics Co., Ltd. (http://www.lge.com/)
+ *
+ * Based on LZ4 implementation by Yann Collet.
+ *
+ * LZ4 - Fast LZ compression algorithm
+ * Copyright (C) 2011-2012, Yann Collet.
+ * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following disclaimer
+ * in the documentation and/or other materials provided with the
+ * distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * You can contact the author at :
+ * - LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
+ * - LZ4 source repository : http://code.google.com/p/lz4/
+ */
+
+#ifndef STATIC
+#include <linux/module.h>
+#include <linux/kernel.h>
+#endif
+#include <linux/lz4.h>
+
+#include <asm/unaligned.h>
+
+#include "lz4defs.h"
+
+static int lz4_uncompress(const char *source, char *dest, int osize)
+{
+ const BYTE *ip = (const BYTE *) source;
+ const BYTE *ref;
+
+ BYTE *op = (BYTE *) dest;
+ BYTE * const oend = op + osize;
+ BYTE *cpy;
+
+ unsigned token;
+
+ size_t length;
+ size_t dec32table[] = {0, 3, 2, 3, 0, 0, 0, 0};
+#if LZ4_ARCH64
+ size_t dec64table[] = {0, 0, 0, -1, 0, 1, 2, 3};
+#endif
+
+ while (1) {
+
+ /* get runlength */
+ token = *ip++;
+ length = (token >> ML_BITS);
+ if (length == RUN_MASK) {
+ size_t len;
+
+ /* for (; (len = *ip++) == 255; length += 255){} */
+ len = *ip++;
+ for (; len == 255; length += 255)
+ len = *ip++;
+ length += len;
+ }
+
+ /* copy literals */
+ cpy = op + length;
+ if (unlikely(cpy > oend - COPYLENGTH)) {
+
+ /*
+ * Error: not enough place for another match
+ * (min 4) + 5 literals
+ */
+ if (cpy != oend)
+ goto _output_error;
+
+ memcpy(op, ip, length);
+ ip += length;
+ break; /* EOF */
+ }
+ LZ4_WILDCOPY(ip, op, cpy);
+ ip -= (op - cpy);
+ op = cpy;
+
+ /* get offset */
+ LZ4_READ_LITTLEENDIAN_16(ref, cpy, ip);
+ ip += 2;
+
+ /* Error: offset create reference outside destination buffer */
+ if (unlikely(ref < (BYTE *const)dest))
+ goto _output_error;
+
+ /* get matchlength */
+ length = token & ML_MASK;
+ if (length == ML_MASK) {
+ for (; *ip == 255; length += 255)
+ ip++;
+ length += *ip++;
+ }
+
+ /* copy repeated sequence */
+ if (unlikely((op - ref) < STEPSIZE)) {
+#if LZ4_ARCH64
+ size_t dec64 = dec64table[op - ref];
+#else
+ const int dec64 = 0;
+#endif
+ op[0] = ref[0];
+ op[1] = ref[1];
+ op[2] = ref[2];
+ op[3] = ref[3];
+ op += 4;
+ ref += 4;
+ ref -= dec32table[op-ref];
+ PUT4(ref, op);
+ op += STEPSIZE - 4;
+ ref -= dec64;
+ } else {
+ LZ4_COPYSTEP(ref, op);
+ }
+ cpy = op + length - (STEPSIZE - 4);
+ if (cpy > oend - COPYLENGTH) {
+
+ /* Error: request to write beyond destination buffer */
+ if (cpy > oend)
+ goto _output_error;
+ LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
+ while (op < cpy)
+ *op++ = *ref++;
+ op = cpy;
+ /*
+ * Check EOF (should never happen, since last 5 bytes
+ * are supposed to be literals)
+ */
+ if (op == oend)
+ goto _output_error;
+ continue;
+ }
+ LZ4_SECURECOPY(ref, op, cpy);
+ op = cpy; /* correction */
+ }
+ /* end of decoding */
+ return (int) (((char *)ip) - source);
+
+ /* write overflow error detected */
+_output_error:
+ return (int) (-(((char *)ip) - source));
+}
+
+static int lz4_uncompress_unknownoutputsize(const char *source, char *dest,
+ int isize, size_t maxoutputsize)
+{
+ const BYTE *ip = (const BYTE *) source;
+ const BYTE *const iend = ip + isize;
+ const BYTE *ref;
+
+
+ BYTE *op = (BYTE *) dest;
+ BYTE * const oend = op + maxoutputsize;
+ BYTE *cpy;
+
+ size_t dec32table[] = {0, 3, 2, 3, 0, 0, 0, 0};
+#if LZ4_ARCH64
+ size_t dec64table[] = {0, 0, 0, -1, 0, 1, 2, 3};
+#endif
+
+ /* Main Loop */
+ while (ip < iend) {
+
+ unsigned token;
+ size_t length;
+
+ /* get runlength */
+ token = *ip++;
+ length = (token >> ML_BITS);
+ if (length == RUN_MASK) {
+ int s = 255;
+ while ((ip < iend) && (s == 255)) {
+ s = *ip++;
+ length += s;
+ }
+ }
+ /* copy literals */
+ cpy = op + length;
+ if ((cpy > oend - COPYLENGTH) ||
+ (ip + length > iend - COPYLENGTH)) {
+
+ if (cpy > oend)
+ goto _output_error;/* writes beyond buffer */
+
+ if (ip + length != iend)
+ goto _output_error;/*
+ * Error: LZ4 format requires
+ * to consume all input
+ * at this stage
+ */
+ memcpy(op, ip, length);
+ op += length;
+ break;/* Necessarily EOF, due to parsing restrictions */
+ }
+ LZ4_WILDCOPY(ip, op, cpy);
+ ip -= (op - cpy);
+ op = cpy;
+
+ /* get offset */
+ LZ4_READ_LITTLEENDIAN_16(ref, cpy, ip);
+ ip += 2;
+ if (ref < (BYTE * const)dest)
+ goto _output_error;
+ /*
+ * Error : offset creates reference
+ * outside of destination buffer
+ */
+
+ /* get matchlength */
+ length = (token & ML_MASK);
+ if (length == ML_MASK) {
+ while (ip < iend) {
+ int s = *ip++;
+ length += s;
+ if (s == 255)
+ continue;
+ break;
+ }
+ }
+
+ /* copy repeated sequence */
+ if (unlikely(op - ref < STEPSIZE)) {
+#if LZ4_ARCH64
+ size_t dec64 = dec64table[op - ref];
+#else
+ const int dec64 = 0;
+#endif
+ op[0] = ref[0];
+ op[1] = ref[1];
+ op[2] = ref[2];
+ op[3] = ref[3];
+ op += 4;
+ ref += 4;
+ ref -= dec32table[op - ref];
+ PUT4(ref, op);
+ op += STEPSIZE - 4;
+ ref -= dec64;
+ } else {
+ LZ4_COPYSTEP(ref, op);
+ }
+ cpy = op + length - (STEPSIZE-4);
+ if (cpy > oend - COPYLENGTH) {
+ if (cpy > oend)
+ goto _output_error; /* write outside of buf */
+
+ LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
+ while (op < cpy)
+ *op++ = *ref++;
+ op = cpy;
+ /*
+ * Check EOF (should never happen, since last 5 bytes
+ * are supposed to be literals)
+ */
+ if (op == oend)
+ goto _output_error;
+ continue;
+ }
+ LZ4_SECURECOPY(ref, op, cpy);
+ op = cpy; /* correction */
+ }
+ /* end of decoding */
+ return (int) (((char *)op) - dest);
+
+ /* write overflow error detected */
+_output_error:
+ return (int) (-(((char *)ip) - source));
+}
+
+int lz4_decompress(const char *src, size_t *src_len, char *dest,
+ size_t actual_dest_len)
+{
+ int ret = -1;
+ int input_len = 0;
+
+ input_len = lz4_uncompress(src, dest, actual_dest_len);
+ if (input_len < 0)
+ goto exit_0;
+ *src_len = input_len;
+
+ return 0;
+exit_0:
+ return ret;
+}
+#ifndef STATIC
+EXPORT_SYMBOL_GPL(lz4_decompress);
+#endif
+
+int lz4_decompress_unknownoutputsize(const char *src, size_t src_len,
+ char *dest, size_t *dest_len)
+{
+ int ret = -1;
+ int out_len = 0;
+
+ out_len = lz4_uncompress_unknownoutputsize(src, dest, src_len,
+ *dest_len);
+ if (out_len < 0)
+ goto exit_0;
+ *dest_len = out_len;
+
+ return 0;
+exit_0:
+ return ret;
+}
+#ifndef STATIC
+EXPORT_SYMBOL_GPL(lz4_decompress_unknownoutputsize);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("LZ4 Decompressor");
+#endif
diff --git a/lib/lz4/lz4defs.h b/lib/lz4/lz4defs.h
new file mode 100644
index 0000000..fde76e6
--- /dev/null
+++ b/lib/lz4/lz4defs.h
@@ -0,0 +1,93 @@
+/*
+ * lz4defs.h -- architecture specific defines
+ *
+ * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+/*
+ * Detects 64 bits mode
+ */
+#if (defined(__x86_64__) || defined(__x86_64) || defined(__amd64__) \
+ || defined(__ppc64__) || defined(__LP64__))
+#define LZ4_ARCH64 1
+#else
+#define LZ4_ARCH64 0
+#endif
+
+/*
+ * Architecture-specific macros
+ */
+#define BYTE u8
+#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) || defined(CONFIG_ARM) \
+ && __LINUX_ARM_ARCH__ >= 6 \
+ && defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)
+typedef struct _U32_S { u32 v; } U32_S;
+typedef struct _U64_S { u64 v; } U64_S;
+
+#define A32(x) (((U32_S *)(x))->v)
+#define A64(x) (((U64_S *)(x))->v)
+
+#define PUT4(s, d) (A32(d) = A32(s))
+#define PUT8(s, d) (A64(d) = A64(s))
+#else /* CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS */
+
+#define PUT4(s, d) \
+ put_unaligned(get_unaligned((const u32 *) s), (u32 *) d)
+#define PUT8(s, d) \
+ put_unaligned(get_unaligned((const u64 *) s), (u64 *) d)
+#endif
+
+#define COPYLENGTH 8
+#define ML_BITS 4
+#define ML_MASK ((1U << ML_BITS) - 1)
+#define RUN_BITS (8 - ML_BITS)
+#define RUN_MASK ((1U << RUN_BITS) - 1)
+
+#if LZ4_ARCH64/* 64-bit */
+#define STEPSIZE 8
+
+#define LZ4_COPYSTEP(s, d) \
+ do { \
+ PUT8(s, d); \
+ d += 8; \
+ s += 8; \
+ } while (0)
+
+#define LZ4_COPYPACKET(s, d) LZ4_COPYSTEP(s, d)
+
+#define LZ4_SECURECOPY(s, d, e) \
+ do { \
+ if (d < e) { \
+ LZ4_WILDCOPY(s, d, e); \
+ } \
+ } while (0)
+
+#else /* 32-bit */
+#define STEPSIZE 4
+
+#define LZ4_COPYSTEP(s, d) \
+ do { \
+ PUT4(s, d); \
+ d += 4; \
+ s += 4; \
+ } while (0)
+
+#define LZ4_COPYPACKET(s, d) \
+ do { \
+ LZ4_COPYSTEP(s, d); \
+ LZ4_COPYSTEP(s, d); \
+ } while (0)
+
+#define LZ4_SECURECOPY LZ4_WILDCOPY
+#endif
+
+#define LZ4_READ_LITTLEENDIAN_16(d, s, p) \
+ (d = s - get_unaligned_le16(p))
+#define LZ4_WILDCOPY(s, d, e) \
+ do { \
+ LZ4_COPYPACKET(s, d); \
+ } while (d < e)
--
1.8.1.1
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [RFC PATCH v2 1/4] decompressor: Add LZ4 decompressor module
@ 2013-02-26 6:24 ` Kyungsik Lee
0 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-26 6:24 UTC (permalink / raw)
To: linux-arm-kernel
This patch adds support for LZ4 decompression in the Linux Kernel.
LZ4 Decompression APIs for kernel are based on LZ4 implementation
by Yann Collet.
LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
LZ4 source repository : http://code.google.com/p/lz4/
Signed-off-by: Kyungsik Lee <kyungsik.lee@lge.com>
v2:
- Clean up code
- Enable unaligned access for ARM v6 and above with
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
- Add lz4_decompress() for faster decompression with
uncompressed output size
---
include/linux/lz4.h | 48 +++++++
lib/lz4/lz4_decompress.c | 331 +++++++++++++++++++++++++++++++++++++++++++++++
lib/lz4/lz4defs.h | 93 +++++++++++++
3 files changed, 472 insertions(+)
create mode 100644 include/linux/lz4.h
create mode 100644 lib/lz4/lz4_decompress.c
create mode 100644 lib/lz4/lz4defs.h
diff --git a/include/linux/lz4.h b/include/linux/lz4.h
new file mode 100644
index 0000000..66b504c
--- /dev/null
+++ b/include/linux/lz4.h
@@ -0,0 +1,48 @@
+#ifndef __LZ4_H__
+#define __LZ4_H__
+/*
+ * LZ4 Kernel Interface
+ *
+ * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+/*
+ * LZ4_COMPRESSBOUND()
+ * Provides the maximum size that LZ4 may output in a "worst case" scenario
+ * (input data not compressible)
+ */
+#define LZ4_COMPRESSBOUND(isize) (isize + ((isize)/255) + 16)
+
+/*
+ * lz4_decompress()
+ * src : source address of the compressed data
+ * src_len : is the input size, whcih is returned after decompress done
+ * dest : output buffer address of the decompressed data
+ * actual_dest_len: is the size of uncompressed data, supposing it's known
+ * return : Success if return 0
+ * Error if return (< 0)
+ * note : Destination buffer must be already allocated.
+ * a bit faster than lz4_decompress_unknownoutputsize()
+ */
+int lz4_decompress(const char *src, size_t *src_len, char *dest,
+ size_t actual_dest_len);
+
+/*
+ * lz4_decompress_unknownoutputsize()
+ * src : source address of the compressed data
+ * src_len : is the input size, therefore the compressed size
+ * dest : output buffer address of the decompressed data
+ * dest_len: is the max size of the destination buffer, which is
+ * returned with actual size of decompressed data after
+ * decompress done
+ * return : Success if return 0
+ * Error if return (< 0)
+ * note : Destination buffer must be already allocated.
+ */
+int lz4_decompress_unknownoutputsize(const char *src, size_t src_len,
+ char *dest, size_t *dest_len);
+#endif
diff --git a/lib/lz4/lz4_decompress.c b/lib/lz4/lz4_decompress.c
new file mode 100644
index 0000000..1998d7a
--- /dev/null
+++ b/lib/lz4/lz4_decompress.c
@@ -0,0 +1,331 @@
+/*
+ * LZ4 Decompressor for Linux kernel
+ *
+ * Copyright (C) 2013 LG Electronics Co., Ltd. (http://www.lge.com/)
+ *
+ * Based on LZ4 implementation by Yann Collet.
+ *
+ * LZ4 - Fast LZ compression algorithm
+ * Copyright (C) 2011-2012, Yann Collet.
+ * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following disclaimer
+ * in the documentation and/or other materials provided with the
+ * distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ *
+ * You can contact the author@:
+ * - LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
+ * - LZ4 source repository : http://code.google.com/p/lz4/
+ */
+
+#ifndef STATIC
+#include <linux/module.h>
+#include <linux/kernel.h>
+#endif
+#include <linux/lz4.h>
+
+#include <asm/unaligned.h>
+
+#include "lz4defs.h"
+
+static int lz4_uncompress(const char *source, char *dest, int osize)
+{
+ const BYTE *ip = (const BYTE *) source;
+ const BYTE *ref;
+
+ BYTE *op = (BYTE *) dest;
+ BYTE * const oend = op + osize;
+ BYTE *cpy;
+
+ unsigned token;
+
+ size_t length;
+ size_t dec32table[] = {0, 3, 2, 3, 0, 0, 0, 0};
+#if LZ4_ARCH64
+ size_t dec64table[] = {0, 0, 0, -1, 0, 1, 2, 3};
+#endif
+
+ while (1) {
+
+ /* get runlength */
+ token = *ip++;
+ length = (token >> ML_BITS);
+ if (length == RUN_MASK) {
+ size_t len;
+
+ /* for (; (len = *ip++) == 255; length += 255){} */
+ len = *ip++;
+ for (; len == 255; length += 255)
+ len = *ip++;
+ length += len;
+ }
+
+ /* copy literals */
+ cpy = op + length;
+ if (unlikely(cpy > oend - COPYLENGTH)) {
+
+ /*
+ * Error: not enough place for another match
+ * (min 4) + 5 literals
+ */
+ if (cpy != oend)
+ goto _output_error;
+
+ memcpy(op, ip, length);
+ ip += length;
+ break; /* EOF */
+ }
+ LZ4_WILDCOPY(ip, op, cpy);
+ ip -= (op - cpy);
+ op = cpy;
+
+ /* get offset */
+ LZ4_READ_LITTLEENDIAN_16(ref, cpy, ip);
+ ip += 2;
+
+ /* Error: offset create reference outside destination buffer */
+ if (unlikely(ref < (BYTE *const)dest))
+ goto _output_error;
+
+ /* get matchlength */
+ length = token & ML_MASK;
+ if (length == ML_MASK) {
+ for (; *ip == 255; length += 255)
+ ip++;
+ length += *ip++;
+ }
+
+ /* copy repeated sequence */
+ if (unlikely((op - ref) < STEPSIZE)) {
+#if LZ4_ARCH64
+ size_t dec64 = dec64table[op - ref];
+#else
+ const int dec64 = 0;
+#endif
+ op[0] = ref[0];
+ op[1] = ref[1];
+ op[2] = ref[2];
+ op[3] = ref[3];
+ op += 4;
+ ref += 4;
+ ref -= dec32table[op-ref];
+ PUT4(ref, op);
+ op += STEPSIZE - 4;
+ ref -= dec64;
+ } else {
+ LZ4_COPYSTEP(ref, op);
+ }
+ cpy = op + length - (STEPSIZE - 4);
+ if (cpy > oend - COPYLENGTH) {
+
+ /* Error: request to write beyond destination buffer */
+ if (cpy > oend)
+ goto _output_error;
+ LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
+ while (op < cpy)
+ *op++ = *ref++;
+ op = cpy;
+ /*
+ * Check EOF (should never happen, since last 5 bytes
+ * are supposed to be literals)
+ */
+ if (op == oend)
+ goto _output_error;
+ continue;
+ }
+ LZ4_SECURECOPY(ref, op, cpy);
+ op = cpy; /* correction */
+ }
+ /* end of decoding */
+ return (int) (((char *)ip) - source);
+
+ /* write overflow error detected */
+_output_error:
+ return (int) (-(((char *)ip) - source));
+}
+
+static int lz4_uncompress_unknownoutputsize(const char *source, char *dest,
+ int isize, size_t maxoutputsize)
+{
+ const BYTE *ip = (const BYTE *) source;
+ const BYTE *const iend = ip + isize;
+ const BYTE *ref;
+
+
+ BYTE *op = (BYTE *) dest;
+ BYTE * const oend = op + maxoutputsize;
+ BYTE *cpy;
+
+ size_t dec32table[] = {0, 3, 2, 3, 0, 0, 0, 0};
+#if LZ4_ARCH64
+ size_t dec64table[] = {0, 0, 0, -1, 0, 1, 2, 3};
+#endif
+
+ /* Main Loop */
+ while (ip < iend) {
+
+ unsigned token;
+ size_t length;
+
+ /* get runlength */
+ token = *ip++;
+ length = (token >> ML_BITS);
+ if (length == RUN_MASK) {
+ int s = 255;
+ while ((ip < iend) && (s == 255)) {
+ s = *ip++;
+ length += s;
+ }
+ }
+ /* copy literals */
+ cpy = op + length;
+ if ((cpy > oend - COPYLENGTH) ||
+ (ip + length > iend - COPYLENGTH)) {
+
+ if (cpy > oend)
+ goto _output_error;/* writes beyond buffer */
+
+ if (ip + length != iend)
+ goto _output_error;/*
+ * Error: LZ4 format requires
+ * to consume all input
+ * at this stage
+ */
+ memcpy(op, ip, length);
+ op += length;
+ break;/* Necessarily EOF, due to parsing restrictions */
+ }
+ LZ4_WILDCOPY(ip, op, cpy);
+ ip -= (op - cpy);
+ op = cpy;
+
+ /* get offset */
+ LZ4_READ_LITTLEENDIAN_16(ref, cpy, ip);
+ ip += 2;
+ if (ref < (BYTE * const)dest)
+ goto _output_error;
+ /*
+ * Error : offset creates reference
+ * outside of destination buffer
+ */
+
+ /* get matchlength */
+ length = (token & ML_MASK);
+ if (length == ML_MASK) {
+ while (ip < iend) {
+ int s = *ip++;
+ length += s;
+ if (s == 255)
+ continue;
+ break;
+ }
+ }
+
+ /* copy repeated sequence */
+ if (unlikely(op - ref < STEPSIZE)) {
+#if LZ4_ARCH64
+ size_t dec64 = dec64table[op - ref];
+#else
+ const int dec64 = 0;
+#endif
+ op[0] = ref[0];
+ op[1] = ref[1];
+ op[2] = ref[2];
+ op[3] = ref[3];
+ op += 4;
+ ref += 4;
+ ref -= dec32table[op - ref];
+ PUT4(ref, op);
+ op += STEPSIZE - 4;
+ ref -= dec64;
+ } else {
+ LZ4_COPYSTEP(ref, op);
+ }
+ cpy = op + length - (STEPSIZE-4);
+ if (cpy > oend - COPYLENGTH) {
+ if (cpy > oend)
+ goto _output_error; /* write outside of buf */
+
+ LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
+ while (op < cpy)
+ *op++ = *ref++;
+ op = cpy;
+ /*
+ * Check EOF (should never happen, since last 5 bytes
+ * are supposed to be literals)
+ */
+ if (op == oend)
+ goto _output_error;
+ continue;
+ }
+ LZ4_SECURECOPY(ref, op, cpy);
+ op = cpy; /* correction */
+ }
+ /* end of decoding */
+ return (int) (((char *)op) - dest);
+
+ /* write overflow error detected */
+_output_error:
+ return (int) (-(((char *)ip) - source));
+}
+
+int lz4_decompress(const char *src, size_t *src_len, char *dest,
+ size_t actual_dest_len)
+{
+ int ret = -1;
+ int input_len = 0;
+
+ input_len = lz4_uncompress(src, dest, actual_dest_len);
+ if (input_len < 0)
+ goto exit_0;
+ *src_len = input_len;
+
+ return 0;
+exit_0:
+ return ret;
+}
+#ifndef STATIC
+EXPORT_SYMBOL_GPL(lz4_decompress);
+#endif
+
+int lz4_decompress_unknownoutputsize(const char *src, size_t src_len,
+ char *dest, size_t *dest_len)
+{
+ int ret = -1;
+ int out_len = 0;
+
+ out_len = lz4_uncompress_unknownoutputsize(src, dest, src_len,
+ *dest_len);
+ if (out_len < 0)
+ goto exit_0;
+ *dest_len = out_len;
+
+ return 0;
+exit_0:
+ return ret;
+}
+#ifndef STATIC
+EXPORT_SYMBOL_GPL(lz4_decompress_unknownoutputsize);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("LZ4 Decompressor");
+#endif
diff --git a/lib/lz4/lz4defs.h b/lib/lz4/lz4defs.h
new file mode 100644
index 0000000..fde76e6
--- /dev/null
+++ b/lib/lz4/lz4defs.h
@@ -0,0 +1,93 @@
+/*
+ * lz4defs.h -- architecture specific defines
+ *
+ * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+/*
+ * Detects 64 bits mode
+ */
+#if (defined(__x86_64__) || defined(__x86_64) || defined(__amd64__) \
+ || defined(__ppc64__) || defined(__LP64__))
+#define LZ4_ARCH64 1
+#else
+#define LZ4_ARCH64 0
+#endif
+
+/*
+ * Architecture-specific macros
+ */
+#define BYTE u8
+#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) || defined(CONFIG_ARM) \
+ && __LINUX_ARM_ARCH__ >= 6 \
+ && defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)
+typedef struct _U32_S { u32 v; } U32_S;
+typedef struct _U64_S { u64 v; } U64_S;
+
+#define A32(x) (((U32_S *)(x))->v)
+#define A64(x) (((U64_S *)(x))->v)
+
+#define PUT4(s, d) (A32(d) = A32(s))
+#define PUT8(s, d) (A64(d) = A64(s))
+#else /* CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS */
+
+#define PUT4(s, d) \
+ put_unaligned(get_unaligned((const u32 *) s), (u32 *) d)
+#define PUT8(s, d) \
+ put_unaligned(get_unaligned((const u64 *) s), (u64 *) d)
+#endif
+
+#define COPYLENGTH 8
+#define ML_BITS 4
+#define ML_MASK ((1U << ML_BITS) - 1)
+#define RUN_BITS (8 - ML_BITS)
+#define RUN_MASK ((1U << RUN_BITS) - 1)
+
+#if LZ4_ARCH64/* 64-bit */
+#define STEPSIZE 8
+
+#define LZ4_COPYSTEP(s, d) \
+ do { \
+ PUT8(s, d); \
+ d += 8; \
+ s += 8; \
+ } while (0)
+
+#define LZ4_COPYPACKET(s, d) LZ4_COPYSTEP(s, d)
+
+#define LZ4_SECURECOPY(s, d, e) \
+ do { \
+ if (d < e) { \
+ LZ4_WILDCOPY(s, d, e); \
+ } \
+ } while (0)
+
+#else /* 32-bit */
+#define STEPSIZE 4
+
+#define LZ4_COPYSTEP(s, d) \
+ do { \
+ PUT4(s, d); \
+ d += 4; \
+ s += 4; \
+ } while (0)
+
+#define LZ4_COPYPACKET(s, d) \
+ do { \
+ LZ4_COPYSTEP(s, d); \
+ LZ4_COPYSTEP(s, d); \
+ } while (0)
+
+#define LZ4_SECURECOPY LZ4_WILDCOPY
+#endif
+
+#define LZ4_READ_LITTLEENDIAN_16(d, s, p) \
+ (d = s - get_unaligned_le16(p))
+#define LZ4_WILDCOPY(s, d, e) \
+ do { \
+ LZ4_COPYPACKET(s, d); \
+ } while (d < e)
--
1.8.1.1
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [RFC PATCH v2 2/4] lib: Add support for LZ4-compressed kernel
2013-02-26 6:24 ` Kyungsik Lee
@ 2013-02-26 6:24 ` Kyungsik Lee
-1 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-26 6:24 UTC (permalink / raw)
To: Andrew Morton, Russell King, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev
Cc: Nicolas Pitre, Nitin Gupta, Markus F.X.J. Oberhumer,
Richard Purdie, Josh Triplett, Joe Millenbach, David Sterba,
Richard Cochran, Albin Tonnerre, Egon Alter, hyojun.im,
chan.jeong, raphael.andy.lee, Kyungsik Lee
This patch adds support for extracting LZ4-compressed kernel images,
as well as LZ4-compressed ramdisk images in the kernel boot process.
This depends on the patch below
decompressor: Add LZ4 decompressor module
Signed-off-by: Kyungsik Lee <kyungsik.lee@lge.com>
v2:
- Clean up code
- Use lz4_decompress() for LZ4-compressed kernel during boot-process
---
include/linux/decompress/unlz4.h | 10 +++
init/Kconfig | 13 ++-
lib/Kconfig | 7 ++
lib/Makefile | 2 +
lib/decompress.c | 5 ++
lib/decompress_unlz4.c | 190 +++++++++++++++++++++++++++++++++++++++
lib/lz4/Makefile | 1 +
lib/lz4/lz4_decompress.c | 2 +-
scripts/Makefile.lib | 5 ++
usr/Kconfig | 9 ++
10 files changed, 242 insertions(+), 2 deletions(-)
create mode 100644 include/linux/decompress/unlz4.h
create mode 100644 lib/decompress_unlz4.c
create mode 100644 lib/lz4/Makefile
diff --git a/include/linux/decompress/unlz4.h b/include/linux/decompress/unlz4.h
new file mode 100644
index 0000000..d5b68bf
--- /dev/null
+++ b/include/linux/decompress/unlz4.h
@@ -0,0 +1,10 @@
+#ifndef DECOMPRESS_UNLZ4_H
+#define DECOMPRESS_UNLZ4_H
+
+int unlz4(unsigned char *inbuf, int len,
+ int(*fill)(void*, unsigned int),
+ int(*flush)(void*, unsigned int),
+ unsigned char *output,
+ int *pos,
+ void(*error)(char *x));
+#endif
diff --git a/init/Kconfig b/init/Kconfig
index be8b7f5..86dc67c 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -133,10 +133,13 @@ config HAVE_KERNEL_XZ
config HAVE_KERNEL_LZO
bool
+config HAVE_KERNEL_LZ4
+ bool
+
choice
prompt "Kernel compression mode"
default KERNEL_GZIP
- depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO
+ depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4
help
The linux kernel is a kind of self-extracting executable.
Several compression algorithms are available, which differ
@@ -203,6 +206,14 @@ config KERNEL_LZO
size is about 10% bigger than gzip; however its speed
(both compression and decompression) is the fastest.
+config KERNEL_LZ4
+ bool "LZ4"
+ depends on HAVE_KERNEL_LZ4
+ help
+ Its compression ratio is worse than LZO. The size of the kernel
+ is about 8% bigger than LZO. But the decompression speed is
+ faster than LZO.
+
endchoice
config DEFAULT_HOSTNAME
diff --git a/lib/Kconfig b/lib/Kconfig
index 75cdb77..b108047 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -189,6 +189,9 @@ config LZO_COMPRESS
config LZO_DECOMPRESS
tristate
+config LZ4_DECOMPRESS
+ tristate
+
source "lib/xz/Kconfig"
#
@@ -213,6 +216,10 @@ config DECOMPRESS_LZO
select LZO_DECOMPRESS
tristate
+config DECOMPRESS_LZ4
+ select LZ4_DECOMPRESS
+ tristate
+
#
# Generic allocator support is selected if needed
#
diff --git a/lib/Makefile b/lib/Makefile
index 02ed6c0..c2073bf 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -72,6 +72,7 @@ obj-$(CONFIG_REED_SOLOMON) += reed_solomon/
obj-$(CONFIG_BCH) += bch.o
obj-$(CONFIG_LZO_COMPRESS) += lzo/
obj-$(CONFIG_LZO_DECOMPRESS) += lzo/
+obj-$(CONFIG_LZ4_DECOMPRESS) += lz4/
obj-$(CONFIG_XZ_DEC) += xz/
obj-$(CONFIG_RAID6_PQ) += raid6/
@@ -80,6 +81,7 @@ lib-$(CONFIG_DECOMPRESS_BZIP2) += decompress_bunzip2.o
lib-$(CONFIG_DECOMPRESS_LZMA) += decompress_unlzma.o
lib-$(CONFIG_DECOMPRESS_XZ) += decompress_unxz.o
lib-$(CONFIG_DECOMPRESS_LZO) += decompress_unlzo.o
+lib-$(CONFIG_DECOMPRESS_LZ4) += decompress_unlz4.o
obj-$(CONFIG_TEXTSEARCH) += textsearch.o
obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o
diff --git a/lib/decompress.c b/lib/decompress.c
index 31a8042..c70810e 100644
--- a/lib/decompress.c
+++ b/lib/decompress.c
@@ -11,6 +11,7 @@
#include <linux/decompress/unxz.h>
#include <linux/decompress/inflate.h>
#include <linux/decompress/unlzo.h>
+#include <linux/decompress/unlz4.h>
#include <linux/types.h>
#include <linux/string.h>
@@ -31,6 +32,9 @@
#ifndef CONFIG_DECOMPRESS_LZO
# define unlzo NULL
#endif
+#ifndef CONFIG_DECOMPRESS_LZ4
+# define unlz4 NULL
+#endif
struct compress_format {
unsigned char magic[2];
@@ -45,6 +49,7 @@ static const struct compress_format compressed_formats[] __initdata = {
{ {0x5d, 0x00}, "lzma", unlzma },
{ {0xfd, 0x37}, "xz", unxz },
{ {0x89, 0x4c}, "lzo", unlzo },
+ { {0x02, 0x21}, "lz4", unlz4 },
{ {0, 0}, NULL, NULL }
};
diff --git a/lib/decompress_unlz4.c b/lib/decompress_unlz4.c
new file mode 100644
index 0000000..84346c4
--- /dev/null
+++ b/lib/decompress_unlz4.c
@@ -0,0 +1,190 @@
+/*
+ * Wrapper for decompressing LZ4-compressed kernel, initramfs, and initrd
+ *
+ * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ */
+
+#ifdef STATIC
+#define PREBOOT
+#include "lz4/lz4_decompress.c"
+#else
+#include <linux/decompress/unlz4.h>
+#endif
+#include <linux/types.h>
+#include <linux/lz4.h>
+#include <linux/decompress/mm.h>
+#include <linux/compiler.h>
+
+#include <asm/unaligned.h>
+
+
+#define LZ4_CHUNK_SIZE (8<<20)
+#define ARCHIVE_MAGICNUMBER 0x184C2102
+
+STATIC inline int INIT unlz4(u8 *input, int in_len,
+ int (*fill) (void *, unsigned int),
+ int (*flush) (void *, unsigned int),
+ u8 *output, int *posp,
+ void (*error) (char *x))
+{
+ int ret = -1;
+ size_t chunksize = 0;
+ u8 *inp;
+ u8 *inp_start;
+ u8 *outp;
+ int size = in_len;
+#ifdef PREBOOT
+ size_t out_len = get_unaligned_le32(input + in_len);
+#endif
+ size_t dest_len;
+
+
+ if (output) {
+ outp = output;
+ } else if (!flush) {
+ error("NULL output pointer and no flush function provided");
+ goto exit_0;
+ } else {
+ outp = large_malloc(LZ4_CHUNK_SIZE);
+ if (!outp) {
+ error("Could not allocate output buffer");
+ goto exit_0;
+ }
+ }
+
+ if (input && fill) {
+ error("Both input pointer and fill function provided,");
+ goto exit_1;
+ } else if (input) {
+ inp = input;
+ } else if (!fill) {
+ error("NULL input pointer and missing fill function");
+ goto exit_1;
+ } else {
+ inp = large_malloc(LZ4_COMPRESSBOUND(LZ4_CHUNK_SIZE));
+ if (!inp) {
+ error("Could not allocate input buffer");
+ goto exit_1;
+ }
+ }
+ inp_start = inp;
+
+ if (posp)
+ *posp = 0;
+
+ if (fill)
+ fill(inp, 4);
+
+ chunksize = get_unaligned_le32(inp);
+ if (chunksize == ARCHIVE_MAGICNUMBER) {
+ inp += 4;
+ size -= 4;
+ } else {
+ error("invalid header");
+ goto exit_2;
+ }
+
+ if (posp)
+ *posp += 4;
+
+ for (;;) {
+
+ if (fill)
+ fill(inp, 4);
+
+ chunksize = get_unaligned_le32(inp);
+ if (chunksize == ARCHIVE_MAGICNUMBER) {
+ inp += 4;
+ size -= 4;
+ if (posp)
+ *posp += 4;
+ continue;
+ }
+ inp += 4;
+ size -= 4;
+
+ if (posp)
+ *posp += 4;
+
+ if (fill) {
+ if (chunksize > LZ4_COMPRESSBOUND(LZ4_CHUNK_SIZE)) {
+ error("chunk length is longer than allocated");
+ goto exit_2;
+ }
+ fill(inp, chunksize);
+ }
+#ifdef PREBOOT
+ if (out_len >= LZ4_CHUNK_SIZE) {
+ dest_len = LZ4_CHUNK_SIZE;
+ out_len -= dest_len;
+ } else
+ dest_len = out_len;
+ ret = lz4_decompress(inp, &chunksize, outp, dest_len);
+#else
+ dest_len = LZ4_CHUNK_SIZE;
+ ret = lz4_decompress_unknownoutputsize(inp, chunksize, outp,
+ &dest_len);
+#endif
+ if (ret < 0) {
+ error("Decoding failed");
+ goto exit_2;
+ }
+
+ if (flush && flush(outp, dest_len) != dest_len)
+ goto exit_2;
+ if (output)
+ outp += dest_len;
+ if (posp)
+ *posp += chunksize;
+
+ size -= chunksize;
+
+ if (size == 0)
+ break;
+ else if (size < 0) {
+ error("data corrupted");
+ goto exit_2;
+ }
+
+ inp += chunksize;
+ if (fill)
+ inp = inp_start;
+ }
+
+ ret = 0;
+exit_2:
+ if (!input)
+ large_free(inp_start);
+exit_1:
+ if (!output)
+ large_free(outp);
+exit_0:
+ return ret;
+}
+
+#ifdef PREBOOT
+STATIC int INIT decompress(unsigned char *buf, int in_len,
+ int(*fill)(void*, unsigned int),
+ int(*flush)(void*, unsigned int),
+ unsigned char *output,
+ int *posp,
+ void(*error)(char *x)
+ )
+{
+ return unlz4(buf, in_len - 4, fill, flush, output, posp, error);
+}
+#endif
diff --git a/lib/lz4/Makefile b/lib/lz4/Makefile
new file mode 100644
index 0000000..7f548c6
--- /dev/null
+++ b/lib/lz4/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_LZ4_DECOMPRESS) += lz4_decompress.o
diff --git a/lib/lz4/lz4_decompress.c b/lib/lz4/lz4_decompress.c
index 1998d7a..f58eaca 100644
--- a/lib/lz4/lz4_decompress.c
+++ b/lib/lz4/lz4_decompress.c
@@ -1,7 +1,7 @@
/*
* LZ4 Decompressor for Linux kernel
*
- * Copyright (C) 2013 LG Electronics Co., Ltd. (http://www.lge.com/)
+ * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
*
* Based on LZ4 implementation by Yann Collet.
*
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index bdf42fd..9293ca1 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -307,6 +307,11 @@ cmd_lzo = (cat $(filter-out FORCE,$^) | \
lzop -9 && $(call size_append, $(filter-out FORCE,$^))) > $@ || \
(rm -f $@ ; false)
+quiet_cmd_lz4 = LZ4 $@
+cmd_lz4 = (cat $(filter-out FORCE,$^) | \
+ lz4 -c1 stdin stdout && $(call size_append, $(filter-out FORCE,$^))) > $@ || \
+ (rm -f $@ ; false)
+
# U-Boot mkimage
# ---------------------------------------------------------------------------
diff --git a/usr/Kconfig b/usr/Kconfig
index 085872b..642f503 100644
--- a/usr/Kconfig
+++ b/usr/Kconfig
@@ -90,6 +90,15 @@ config RD_LZO
Support loading of a LZO encoded initial ramdisk or cpio buffer
If unsure, say N.
+config RD_LZ4
+ bool "Support initial ramdisks compressed using LZ4" if EXPERT
+ default !EXPERT
+ depends on BLK_DEV_INITRD
+ select DECOMPRESS_LZ4
+ help
+ Support loading of a LZ4 encoded initial ramdisk or cpio buffer
+ If unsure, say N.
+
choice
prompt "Built-in initramfs compression mode" if INITRAMFS_SOURCE!=""
help
--
1.8.1.1
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [RFC PATCH v2 2/4] lib: Add support for LZ4-compressed kernel
@ 2013-02-26 6:24 ` Kyungsik Lee
0 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-26 6:24 UTC (permalink / raw)
To: linux-arm-kernel
This patch adds support for extracting LZ4-compressed kernel images,
as well as LZ4-compressed ramdisk images in the kernel boot process.
This depends on the patch below
decompressor: Add LZ4 decompressor module
Signed-off-by: Kyungsik Lee <kyungsik.lee@lge.com>
v2:
- Clean up code
- Use lz4_decompress() for LZ4-compressed kernel during boot-process
---
include/linux/decompress/unlz4.h | 10 +++
init/Kconfig | 13 ++-
lib/Kconfig | 7 ++
lib/Makefile | 2 +
lib/decompress.c | 5 ++
lib/decompress_unlz4.c | 190 +++++++++++++++++++++++++++++++++++++++
lib/lz4/Makefile | 1 +
lib/lz4/lz4_decompress.c | 2 +-
scripts/Makefile.lib | 5 ++
usr/Kconfig | 9 ++
10 files changed, 242 insertions(+), 2 deletions(-)
create mode 100644 include/linux/decompress/unlz4.h
create mode 100644 lib/decompress_unlz4.c
create mode 100644 lib/lz4/Makefile
diff --git a/include/linux/decompress/unlz4.h b/include/linux/decompress/unlz4.h
new file mode 100644
index 0000000..d5b68bf
--- /dev/null
+++ b/include/linux/decompress/unlz4.h
@@ -0,0 +1,10 @@
+#ifndef DECOMPRESS_UNLZ4_H
+#define DECOMPRESS_UNLZ4_H
+
+int unlz4(unsigned char *inbuf, int len,
+ int(*fill)(void*, unsigned int),
+ int(*flush)(void*, unsigned int),
+ unsigned char *output,
+ int *pos,
+ void(*error)(char *x));
+#endif
diff --git a/init/Kconfig b/init/Kconfig
index be8b7f5..86dc67c 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -133,10 +133,13 @@ config HAVE_KERNEL_XZ
config HAVE_KERNEL_LZO
bool
+config HAVE_KERNEL_LZ4
+ bool
+
choice
prompt "Kernel compression mode"
default KERNEL_GZIP
- depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO
+ depends on HAVE_KERNEL_GZIP || HAVE_KERNEL_BZIP2 || HAVE_KERNEL_LZMA || HAVE_KERNEL_XZ || HAVE_KERNEL_LZO || HAVE_KERNEL_LZ4
help
The linux kernel is a kind of self-extracting executable.
Several compression algorithms are available, which differ
@@ -203,6 +206,14 @@ config KERNEL_LZO
size is about 10% bigger than gzip; however its speed
(both compression and decompression) is the fastest.
+config KERNEL_LZ4
+ bool "LZ4"
+ depends on HAVE_KERNEL_LZ4
+ help
+ Its compression ratio is worse than LZO. The size of the kernel
+ is about 8% bigger than LZO. But the decompression speed is
+ faster than LZO.
+
endchoice
config DEFAULT_HOSTNAME
diff --git a/lib/Kconfig b/lib/Kconfig
index 75cdb77..b108047 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -189,6 +189,9 @@ config LZO_COMPRESS
config LZO_DECOMPRESS
tristate
+config LZ4_DECOMPRESS
+ tristate
+
source "lib/xz/Kconfig"
#
@@ -213,6 +216,10 @@ config DECOMPRESS_LZO
select LZO_DECOMPRESS
tristate
+config DECOMPRESS_LZ4
+ select LZ4_DECOMPRESS
+ tristate
+
#
# Generic allocator support is selected if needed
#
diff --git a/lib/Makefile b/lib/Makefile
index 02ed6c0..c2073bf 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -72,6 +72,7 @@ obj-$(CONFIG_REED_SOLOMON) += reed_solomon/
obj-$(CONFIG_BCH) += bch.o
obj-$(CONFIG_LZO_COMPRESS) += lzo/
obj-$(CONFIG_LZO_DECOMPRESS) += lzo/
+obj-$(CONFIG_LZ4_DECOMPRESS) += lz4/
obj-$(CONFIG_XZ_DEC) += xz/
obj-$(CONFIG_RAID6_PQ) += raid6/
@@ -80,6 +81,7 @@ lib-$(CONFIG_DECOMPRESS_BZIP2) += decompress_bunzip2.o
lib-$(CONFIG_DECOMPRESS_LZMA) += decompress_unlzma.o
lib-$(CONFIG_DECOMPRESS_XZ) += decompress_unxz.o
lib-$(CONFIG_DECOMPRESS_LZO) += decompress_unlzo.o
+lib-$(CONFIG_DECOMPRESS_LZ4) += decompress_unlz4.o
obj-$(CONFIG_TEXTSEARCH) += textsearch.o
obj-$(CONFIG_TEXTSEARCH_KMP) += ts_kmp.o
diff --git a/lib/decompress.c b/lib/decompress.c
index 31a8042..c70810e 100644
--- a/lib/decompress.c
+++ b/lib/decompress.c
@@ -11,6 +11,7 @@
#include <linux/decompress/unxz.h>
#include <linux/decompress/inflate.h>
#include <linux/decompress/unlzo.h>
+#include <linux/decompress/unlz4.h>
#include <linux/types.h>
#include <linux/string.h>
@@ -31,6 +32,9 @@
#ifndef CONFIG_DECOMPRESS_LZO
# define unlzo NULL
#endif
+#ifndef CONFIG_DECOMPRESS_LZ4
+# define unlz4 NULL
+#endif
struct compress_format {
unsigned char magic[2];
@@ -45,6 +49,7 @@ static const struct compress_format compressed_formats[] __initdata = {
{ {0x5d, 0x00}, "lzma", unlzma },
{ {0xfd, 0x37}, "xz", unxz },
{ {0x89, 0x4c}, "lzo", unlzo },
+ { {0x02, 0x21}, "lz4", unlz4 },
{ {0, 0}, NULL, NULL }
};
diff --git a/lib/decompress_unlz4.c b/lib/decompress_unlz4.c
new file mode 100644
index 0000000..84346c4
--- /dev/null
+++ b/lib/decompress_unlz4.c
@@ -0,0 +1,190 @@
+/*
+ * Wrapper for decompressing LZ4-compressed kernel, initramfs, and initrd
+ *
+ * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ */
+
+#ifdef STATIC
+#define PREBOOT
+#include "lz4/lz4_decompress.c"
+#else
+#include <linux/decompress/unlz4.h>
+#endif
+#include <linux/types.h>
+#include <linux/lz4.h>
+#include <linux/decompress/mm.h>
+#include <linux/compiler.h>
+
+#include <asm/unaligned.h>
+
+
+#define LZ4_CHUNK_SIZE (8<<20)
+#define ARCHIVE_MAGICNUMBER 0x184C2102
+
+STATIC inline int INIT unlz4(u8 *input, int in_len,
+ int (*fill) (void *, unsigned int),
+ int (*flush) (void *, unsigned int),
+ u8 *output, int *posp,
+ void (*error) (char *x))
+{
+ int ret = -1;
+ size_t chunksize = 0;
+ u8 *inp;
+ u8 *inp_start;
+ u8 *outp;
+ int size = in_len;
+#ifdef PREBOOT
+ size_t out_len = get_unaligned_le32(input + in_len);
+#endif
+ size_t dest_len;
+
+
+ if (output) {
+ outp = output;
+ } else if (!flush) {
+ error("NULL output pointer and no flush function provided");
+ goto exit_0;
+ } else {
+ outp = large_malloc(LZ4_CHUNK_SIZE);
+ if (!outp) {
+ error("Could not allocate output buffer");
+ goto exit_0;
+ }
+ }
+
+ if (input && fill) {
+ error("Both input pointer and fill function provided,");
+ goto exit_1;
+ } else if (input) {
+ inp = input;
+ } else if (!fill) {
+ error("NULL input pointer and missing fill function");
+ goto exit_1;
+ } else {
+ inp = large_malloc(LZ4_COMPRESSBOUND(LZ4_CHUNK_SIZE));
+ if (!inp) {
+ error("Could not allocate input buffer");
+ goto exit_1;
+ }
+ }
+ inp_start = inp;
+
+ if (posp)
+ *posp = 0;
+
+ if (fill)
+ fill(inp, 4);
+
+ chunksize = get_unaligned_le32(inp);
+ if (chunksize == ARCHIVE_MAGICNUMBER) {
+ inp += 4;
+ size -= 4;
+ } else {
+ error("invalid header");
+ goto exit_2;
+ }
+
+ if (posp)
+ *posp += 4;
+
+ for (;;) {
+
+ if (fill)
+ fill(inp, 4);
+
+ chunksize = get_unaligned_le32(inp);
+ if (chunksize == ARCHIVE_MAGICNUMBER) {
+ inp += 4;
+ size -= 4;
+ if (posp)
+ *posp += 4;
+ continue;
+ }
+ inp += 4;
+ size -= 4;
+
+ if (posp)
+ *posp += 4;
+
+ if (fill) {
+ if (chunksize > LZ4_COMPRESSBOUND(LZ4_CHUNK_SIZE)) {
+ error("chunk length is longer than allocated");
+ goto exit_2;
+ }
+ fill(inp, chunksize);
+ }
+#ifdef PREBOOT
+ if (out_len >= LZ4_CHUNK_SIZE) {
+ dest_len = LZ4_CHUNK_SIZE;
+ out_len -= dest_len;
+ } else
+ dest_len = out_len;
+ ret = lz4_decompress(inp, &chunksize, outp, dest_len);
+#else
+ dest_len = LZ4_CHUNK_SIZE;
+ ret = lz4_decompress_unknownoutputsize(inp, chunksize, outp,
+ &dest_len);
+#endif
+ if (ret < 0) {
+ error("Decoding failed");
+ goto exit_2;
+ }
+
+ if (flush && flush(outp, dest_len) != dest_len)
+ goto exit_2;
+ if (output)
+ outp += dest_len;
+ if (posp)
+ *posp += chunksize;
+
+ size -= chunksize;
+
+ if (size == 0)
+ break;
+ else if (size < 0) {
+ error("data corrupted");
+ goto exit_2;
+ }
+
+ inp += chunksize;
+ if (fill)
+ inp = inp_start;
+ }
+
+ ret = 0;
+exit_2:
+ if (!input)
+ large_free(inp_start);
+exit_1:
+ if (!output)
+ large_free(outp);
+exit_0:
+ return ret;
+}
+
+#ifdef PREBOOT
+STATIC int INIT decompress(unsigned char *buf, int in_len,
+ int(*fill)(void*, unsigned int),
+ int(*flush)(void*, unsigned int),
+ unsigned char *output,
+ int *posp,
+ void(*error)(char *x)
+ )
+{
+ return unlz4(buf, in_len - 4, fill, flush, output, posp, error);
+}
+#endif
diff --git a/lib/lz4/Makefile b/lib/lz4/Makefile
new file mode 100644
index 0000000..7f548c6
--- /dev/null
+++ b/lib/lz4/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_LZ4_DECOMPRESS) += lz4_decompress.o
diff --git a/lib/lz4/lz4_decompress.c b/lib/lz4/lz4_decompress.c
index 1998d7a..f58eaca 100644
--- a/lib/lz4/lz4_decompress.c
+++ b/lib/lz4/lz4_decompress.c
@@ -1,7 +1,7 @@
/*
* LZ4 Decompressor for Linux kernel
*
- * Copyright (C) 2013 LG Electronics Co., Ltd. (http://www.lge.com/)
+ * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
*
* Based on LZ4 implementation by Yann Collet.
*
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index bdf42fd..9293ca1 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -307,6 +307,11 @@ cmd_lzo = (cat $(filter-out FORCE,$^) | \
lzop -9 && $(call size_append, $(filter-out FORCE,$^))) > $@ || \
(rm -f $@ ; false)
+quiet_cmd_lz4 = LZ4 $@
+cmd_lz4 = (cat $(filter-out FORCE,$^) | \
+ lz4 -c1 stdin stdout && $(call size_append, $(filter-out FORCE,$^))) > $@ || \
+ (rm -f $@ ; false)
+
# U-Boot mkimage
# ---------------------------------------------------------------------------
diff --git a/usr/Kconfig b/usr/Kconfig
index 085872b..642f503 100644
--- a/usr/Kconfig
+++ b/usr/Kconfig
@@ -90,6 +90,15 @@ config RD_LZO
Support loading of a LZO encoded initial ramdisk or cpio buffer
If unsure, say N.
+config RD_LZ4
+ bool "Support initial ramdisks compressed using LZ4" if EXPERT
+ default !EXPERT
+ depends on BLK_DEV_INITRD
+ select DECOMPRESS_LZ4
+ help
+ Support loading of a LZ4 encoded initial ramdisk or cpio buffer
+ If unsure, say N.
+
choice
prompt "Built-in initramfs compression mode" if INITRAMFS_SOURCE!=""
help
--
1.8.1.1
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [RFC PATCH v2 3/4] arm: Add support for LZ4-compressed kernel
2013-02-26 6:24 ` Kyungsik Lee
@ 2013-02-26 6:24 ` Kyungsik Lee
-1 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-26 6:24 UTC (permalink / raw)
To: Andrew Morton, Russell King, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev
Cc: Nicolas Pitre, Nitin Gupta, Markus F.X.J. Oberhumer,
Richard Purdie, Josh Triplett, Joe Millenbach, David Sterba,
Richard Cochran, Albin Tonnerre, Egon Alter, hyojun.im,
chan.jeong, raphael.andy.lee, Kyungsik Lee
This patch integrates the LZ4 decompression code to the arm pre-boot code.
And it depends on two patchs below
lib: Add support for LZ4-compressed kernel
decompressor: Add LZ4 decompressor module
Signed-off-by: Kyungsik Lee <kyungsik.lee@lge.com>
v2:
- Apply CFLAGS, -Os to decompress.o to improve decompress
performance during boot-up process
---
arch/arm/Kconfig | 1 +
arch/arm/boot/compressed/.gitignore | 1 +
arch/arm/boot/compressed/Makefile | 6 +++++-
arch/arm/boot/compressed/decompress.c | 4 ++++
arch/arm/boot/compressed/piggy.lz4.S | 6 ++++++
5 files changed, 17 insertions(+), 1 deletion(-)
create mode 100644 arch/arm/boot/compressed/piggy.lz4.S
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 67874b8..0f9b363 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -38,6 +38,7 @@ config ARM
select HAVE_IDE if PCI || ISA || PCMCIA
select HAVE_IRQ_WORK
select HAVE_KERNEL_GZIP
+ select HAVE_KERNEL_LZ4
select HAVE_KERNEL_LZMA
select HAVE_KERNEL_LZO
select HAVE_KERNEL_XZ
diff --git a/arch/arm/boot/compressed/.gitignore b/arch/arm/boot/compressed/.gitignore
index f79a08e..47279aa 100644
--- a/arch/arm/boot/compressed/.gitignore
+++ b/arch/arm/boot/compressed/.gitignore
@@ -6,6 +6,7 @@ piggy.gzip
piggy.lzo
piggy.lzma
piggy.xzkern
+piggy.lz4
vmlinux
vmlinux.lds
diff --git a/arch/arm/boot/compressed/Makefile b/arch/arm/boot/compressed/Makefile
index 5cad8a6..2249d52 100644
--- a/arch/arm/boot/compressed/Makefile
+++ b/arch/arm/boot/compressed/Makefile
@@ -24,6 +24,9 @@ endif
AFLAGS_head.o += -DTEXT_OFFSET=$(TEXT_OFFSET)
HEAD = head.o
OBJS += misc.o decompress.o
+ifeq ($(CONFIG_KERNEL_LZ4),y)
+CFLAGS_decompress.o := -Os
+endif
FONTC = $(srctree)/drivers/video/console/font_acorn_8x8.c
# string library code (-Os is enforced to keep it much smaller)
@@ -88,6 +91,7 @@ suffix_$(CONFIG_KERNEL_GZIP) = gzip
suffix_$(CONFIG_KERNEL_LZO) = lzo
suffix_$(CONFIG_KERNEL_LZMA) = lzma
suffix_$(CONFIG_KERNEL_XZ) = xzkern
+suffix_$(CONFIG_KERNEL_LZ4) = lz4
# Borrowed libfdt files for the ATAG compatibility mode
@@ -112,7 +116,7 @@ targets := vmlinux vmlinux.lds \
font.o font.c head.o misc.o $(OBJS)
# Make sure files are removed during clean
-extra-y += piggy.gzip piggy.lzo piggy.lzma piggy.xzkern \
+extra-y += piggy.gzip piggy.lzo piggy.lzma piggy.xzkern piggy.lz4 \
lib1funcs.S ashldi3.S $(libfdt) $(libfdt_hdrs)
ifeq ($(CONFIG_FUNCTION_TRACER),y)
diff --git a/arch/arm/boot/compressed/decompress.c b/arch/arm/boot/compressed/decompress.c
index 9deb56a..a95f071 100644
--- a/arch/arm/boot/compressed/decompress.c
+++ b/arch/arm/boot/compressed/decompress.c
@@ -53,6 +53,10 @@ extern char * strstr(const char * s1, const char *s2);
#include "../../../../lib/decompress_unxz.c"
#endif
+#ifdef CONFIG_KERNEL_LZ4
+#include "../../../../lib/decompress_unlz4.c"
+#endif
+
int do_decompress(u8 *input, int len, u8 *output, void (*error)(char *x))
{
return decompress(input, len, NULL, NULL, output, NULL, error);
diff --git a/arch/arm/boot/compressed/piggy.lz4.S b/arch/arm/boot/compressed/piggy.lz4.S
new file mode 100644
index 0000000..3d9a575
--- /dev/null
+++ b/arch/arm/boot/compressed/piggy.lz4.S
@@ -0,0 +1,6 @@
+ .section .piggydata,#alloc
+ .globl input_data
+input_data:
+ .incbin "arch/arm/boot/compressed/piggy.lz4"
+ .globl input_data_end
+input_data_end:
--
1.8.1.1
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [RFC PATCH v2 3/4] arm: Add support for LZ4-compressed kernel
@ 2013-02-26 6:24 ` Kyungsik Lee
0 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-26 6:24 UTC (permalink / raw)
To: linux-arm-kernel
This patch integrates the LZ4 decompression code to the arm pre-boot code.
And it depends on two patchs below
lib: Add support for LZ4-compressed kernel
decompressor: Add LZ4 decompressor module
Signed-off-by: Kyungsik Lee <kyungsik.lee@lge.com>
v2:
- Apply CFLAGS, -Os to decompress.o to improve decompress
performance during boot-up process
---
arch/arm/Kconfig | 1 +
arch/arm/boot/compressed/.gitignore | 1 +
arch/arm/boot/compressed/Makefile | 6 +++++-
arch/arm/boot/compressed/decompress.c | 4 ++++
arch/arm/boot/compressed/piggy.lz4.S | 6 ++++++
5 files changed, 17 insertions(+), 1 deletion(-)
create mode 100644 arch/arm/boot/compressed/piggy.lz4.S
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 67874b8..0f9b363 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -38,6 +38,7 @@ config ARM
select HAVE_IDE if PCI || ISA || PCMCIA
select HAVE_IRQ_WORK
select HAVE_KERNEL_GZIP
+ select HAVE_KERNEL_LZ4
select HAVE_KERNEL_LZMA
select HAVE_KERNEL_LZO
select HAVE_KERNEL_XZ
diff --git a/arch/arm/boot/compressed/.gitignore b/arch/arm/boot/compressed/.gitignore
index f79a08e..47279aa 100644
--- a/arch/arm/boot/compressed/.gitignore
+++ b/arch/arm/boot/compressed/.gitignore
@@ -6,6 +6,7 @@ piggy.gzip
piggy.lzo
piggy.lzma
piggy.xzkern
+piggy.lz4
vmlinux
vmlinux.lds
diff --git a/arch/arm/boot/compressed/Makefile b/arch/arm/boot/compressed/Makefile
index 5cad8a6..2249d52 100644
--- a/arch/arm/boot/compressed/Makefile
+++ b/arch/arm/boot/compressed/Makefile
@@ -24,6 +24,9 @@ endif
AFLAGS_head.o += -DTEXT_OFFSET=$(TEXT_OFFSET)
HEAD = head.o
OBJS += misc.o decompress.o
+ifeq ($(CONFIG_KERNEL_LZ4),y)
+CFLAGS_decompress.o := -Os
+endif
FONTC = $(srctree)/drivers/video/console/font_acorn_8x8.c
# string library code (-Os is enforced to keep it much smaller)
@@ -88,6 +91,7 @@ suffix_$(CONFIG_KERNEL_GZIP) = gzip
suffix_$(CONFIG_KERNEL_LZO) = lzo
suffix_$(CONFIG_KERNEL_LZMA) = lzma
suffix_$(CONFIG_KERNEL_XZ) = xzkern
+suffix_$(CONFIG_KERNEL_LZ4) = lz4
# Borrowed libfdt files for the ATAG compatibility mode
@@ -112,7 +116,7 @@ targets := vmlinux vmlinux.lds \
font.o font.c head.o misc.o $(OBJS)
# Make sure files are removed during clean
-extra-y += piggy.gzip piggy.lzo piggy.lzma piggy.xzkern \
+extra-y += piggy.gzip piggy.lzo piggy.lzma piggy.xzkern piggy.lz4 \
lib1funcs.S ashldi3.S $(libfdt) $(libfdt_hdrs)
ifeq ($(CONFIG_FUNCTION_TRACER),y)
diff --git a/arch/arm/boot/compressed/decompress.c b/arch/arm/boot/compressed/decompress.c
index 9deb56a..a95f071 100644
--- a/arch/arm/boot/compressed/decompress.c
+++ b/arch/arm/boot/compressed/decompress.c
@@ -53,6 +53,10 @@ extern char * strstr(const char * s1, const char *s2);
#include "../../../../lib/decompress_unxz.c"
#endif
+#ifdef CONFIG_KERNEL_LZ4
+#include "../../../../lib/decompress_unlz4.c"
+#endif
+
int do_decompress(u8 *input, int len, u8 *output, void (*error)(char *x))
{
return decompress(input, len, NULL, NULL, output, NULL, error);
diff --git a/arch/arm/boot/compressed/piggy.lz4.S b/arch/arm/boot/compressed/piggy.lz4.S
new file mode 100644
index 0000000..3d9a575
--- /dev/null
+++ b/arch/arm/boot/compressed/piggy.lz4.S
@@ -0,0 +1,6 @@
+ .section .piggydata,#alloc
+ .globl input_data
+input_data:
+ .incbin "arch/arm/boot/compressed/piggy.lz4"
+ .globl input_data_end
+input_data_end:
--
1.8.1.1
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [RFC PATCH v2 4/4] x86: Add support for LZ4-compressed kernel
2013-02-26 6:24 ` Kyungsik Lee
@ 2013-02-26 6:24 ` Kyungsik Lee
-1 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-26 6:24 UTC (permalink / raw)
To: Andrew Morton, Russell King, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev
Cc: Nicolas Pitre, Nitin Gupta, Markus F.X.J. Oberhumer,
Richard Purdie, Josh Triplett, Joe Millenbach, David Sterba,
Richard Cochran, Albin Tonnerre, Egon Alter, hyojun.im,
chan.jeong, raphael.andy.lee, Kyungsik Lee
This patch integrates the LZ4 decompression code to the x86 pre-boot code.
And it depends on two patchs below
lib: Add support for LZ4-compressed kernel
decompressor: Add LZ4 decompressor module
Signed-off-by: Kyungsik Lee <kyungsik.lee@lge.com>
---
arch/x86/Kconfig | 1 +
arch/x86/boot/compressed/Makefile | 5 ++++-
arch/x86/boot/compressed/misc.c | 4 ++++
3 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 225543b..ab916fd 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -63,6 +63,7 @@ config X86
select HAVE_KERNEL_LZMA
select HAVE_KERNEL_XZ
select HAVE_KERNEL_LZO
+ select HAVE_KERNEL_LZ4
select HAVE_HW_BREAKPOINT
select HAVE_MIXED_BREAKPOINTS_REGS
select PERF_EVENTS
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 8a84501..c275db5 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -4,7 +4,7 @@
# create a compressed vmlinux image from the original vmlinux
#
-targets := vmlinux.lds vmlinux vmlinux.bin vmlinux.bin.gz vmlinux.bin.bz2 vmlinux.bin.lzma vmlinux.bin.xz vmlinux.bin.lzo head_$(BITS).o misc.o string.o cmdline.o early_serial_console.o piggy.o
+targets := vmlinux.lds vmlinux vmlinux.bin vmlinux.bin.gz vmlinux.bin.bz2 vmlinux.bin.lzma vmlinux.bin.xz vmlinux.bin.lzo vmlinux.bin.lz4 head_$(BITS).o misc.o string.o cmdline.o early_serial_console.o piggy.o
KBUILD_CFLAGS := -m$(BITS) -D__KERNEL__ $(LINUX_INCLUDE) -O2
KBUILD_CFLAGS += -fno-strict-aliasing -fPIC
@@ -64,12 +64,15 @@ $(obj)/vmlinux.bin.xz: $(vmlinux.bin.all-y) FORCE
$(call if_changed,xzkern)
$(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
$(call if_changed,lzo)
+$(obj)/vmlinux.bin.lz4: $(vmlinux.bin.all-y) FORCE
+ $(call if_changed,lz4)
suffix-$(CONFIG_KERNEL_GZIP) := gz
suffix-$(CONFIG_KERNEL_BZIP2) := bz2
suffix-$(CONFIG_KERNEL_LZMA) := lzma
suffix-$(CONFIG_KERNEL_XZ) := xz
suffix-$(CONFIG_KERNEL_LZO) := lzo
+suffix-$(CONFIG_KERNEL_LZ4) := lz4
quiet_cmd_mkpiggy = MKPIGGY $@
cmd_mkpiggy = $(obj)/mkpiggy $< > $@ || ( rm -f $@ ; false )
diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c
index 88f7ff6..166a0a8 100644
--- a/arch/x86/boot/compressed/misc.c
+++ b/arch/x86/boot/compressed/misc.c
@@ -145,6 +145,10 @@ static int lines, cols;
#include "../../../../lib/decompress_unlzo.c"
#endif
+#ifdef CONFIG_KERNEL_LZ4
+#include "../../../../lib/decompress_unlz4.c"
+#endif
+
static void scroll(void)
{
int i;
--
1.8.1.1
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [RFC PATCH v2 4/4] x86: Add support for LZ4-compressed kernel
@ 2013-02-26 6:24 ` Kyungsik Lee
0 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-26 6:24 UTC (permalink / raw)
To: linux-arm-kernel
This patch integrates the LZ4 decompression code to the x86 pre-boot code.
And it depends on two patchs below
lib: Add support for LZ4-compressed kernel
decompressor: Add LZ4 decompressor module
Signed-off-by: Kyungsik Lee <kyungsik.lee@lge.com>
---
arch/x86/Kconfig | 1 +
arch/x86/boot/compressed/Makefile | 5 ++++-
arch/x86/boot/compressed/misc.c | 4 ++++
3 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 225543b..ab916fd 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -63,6 +63,7 @@ config X86
select HAVE_KERNEL_LZMA
select HAVE_KERNEL_XZ
select HAVE_KERNEL_LZO
+ select HAVE_KERNEL_LZ4
select HAVE_HW_BREAKPOINT
select HAVE_MIXED_BREAKPOINTS_REGS
select PERF_EVENTS
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 8a84501..c275db5 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -4,7 +4,7 @@
# create a compressed vmlinux image from the original vmlinux
#
-targets := vmlinux.lds vmlinux vmlinux.bin vmlinux.bin.gz vmlinux.bin.bz2 vmlinux.bin.lzma vmlinux.bin.xz vmlinux.bin.lzo head_$(BITS).o misc.o string.o cmdline.o early_serial_console.o piggy.o
+targets := vmlinux.lds vmlinux vmlinux.bin vmlinux.bin.gz vmlinux.bin.bz2 vmlinux.bin.lzma vmlinux.bin.xz vmlinux.bin.lzo vmlinux.bin.lz4 head_$(BITS).o misc.o string.o cmdline.o early_serial_console.o piggy.o
KBUILD_CFLAGS := -m$(BITS) -D__KERNEL__ $(LINUX_INCLUDE) -O2
KBUILD_CFLAGS += -fno-strict-aliasing -fPIC
@@ -64,12 +64,15 @@ $(obj)/vmlinux.bin.xz: $(vmlinux.bin.all-y) FORCE
$(call if_changed,xzkern)
$(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
$(call if_changed,lzo)
+$(obj)/vmlinux.bin.lz4: $(vmlinux.bin.all-y) FORCE
+ $(call if_changed,lz4)
suffix-$(CONFIG_KERNEL_GZIP) := gz
suffix-$(CONFIG_KERNEL_BZIP2) := bz2
suffix-$(CONFIG_KERNEL_LZMA) := lzma
suffix-$(CONFIG_KERNEL_XZ) := xz
suffix-$(CONFIG_KERNEL_LZO) := lzo
+suffix-$(CONFIG_KERNEL_LZ4) := lz4
quiet_cmd_mkpiggy = MKPIGGY $@
cmd_mkpiggy = $(obj)/mkpiggy $< > $@ || ( rm -f $@ ; false )
diff --git a/arch/x86/boot/compressed/misc.c b/arch/x86/boot/compressed/misc.c
index 88f7ff6..166a0a8 100644
--- a/arch/x86/boot/compressed/misc.c
+++ b/arch/x86/boot/compressed/misc.c
@@ -145,6 +145,10 @@ static int lines, cols;
#include "../../../../lib/decompress_unlzo.c"
#endif
+#ifdef CONFIG_KERNEL_LZ4
+#include "../../../../lib/decompress_unlz4.c"
+#endif
+
static void scroll(void)
{
int i;
--
1.8.1.1
^ permalink raw reply related [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 1/4] decompressor: Add LZ4 decompressor module
2013-02-26 6:24 ` Kyungsik Lee
@ 2013-02-26 13:12 ` David Sterba
-1 siblings, 0 replies; 67+ messages in thread
From: David Sterba @ 2013-02-26 13:12 UTC (permalink / raw)
To: Kyungsik Lee
Cc: Andrew Morton, Russell King, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nicolas Pitre, Nitin Gupta,
Markus F.X.J. Oberhumer, Richard Purdie, Josh Triplett,
Joe Millenbach, David Sterba, Richard Cochran, Albin Tonnerre,
Egon Alter, hyojun.im, chan.jeong, raphael.andy.lee
On Tue, Feb 26, 2013 at 03:24:27PM +0900, Kyungsik Lee wrote:
> This patch adds support for LZ4 decompression in the Linux Kernel.
> LZ4 Decompression APIs for kernel are based on LZ4 implementation
> by Yann Collet.
>
> LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
> LZ4 source repository : http://code.google.com/p/lz4/
What SVN version did you use?
> --- /dev/null
> +++ b/include/linux/lz4.h
> @@ -0,0 +1,48 @@
> +#ifndef __LZ4_H__
> +#define __LZ4_H__
> +/*
> + * LZ4 Kernel Interface
> + *
> + * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +/*
> + * LZ4_COMPRESSBOUND()
> + * Provides the maximum size that LZ4 may output in a "worst case" scenario
> + * (input data not compressible)
> + */
> +#define LZ4_COMPRESSBOUND(isize) (isize + ((isize)/255) + 16)
For safety reasons I suggest to add a temporary variable to avoid double
evaluation of isize.
> --- /dev/null
> +++ b/lib/lz4/lz4_decompress.c
> + }
> + cpy = op + length - (STEPSIZE - 4);
> + if (cpy > oend - COPYLENGTH) {
> +
> + /* Error: request to write beyond destination buffer */
> + if (cpy > oend)
> + goto _output_error;
> + LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
> + while (op < cpy)
> + *op++ = *ref++;
> + op = cpy;
> + /*
> + * Check EOF (should never happen, since last 5 bytes
> + * are supposed to be literals)
> + */
> + if (op == oend)
> + goto _output_error;
> + continue;
> + }
> + LZ4_SECURECOPY(ref, op, cpy);
> + op = cpy; /* correction */
> + }
Does this compile? The } is an extra one, and does not match the
original sources.
> + /* end of decoding */
> + return (int) (((char *)ip) - source);
> +
> + /* write overflow error detected */
> +_output_error:
> + return (int) (-(((char *)ip) - source));
> +}
> +
> --- /dev/null
> +++ b/lib/lz4/lz4defs.h
> @@ -0,0 +1,93 @@
> +#define LZ4_COPYSTEP(s, d) \
> + do { \
> + PUT8(s, d); \
> + d += 8; \
> + s += 8; \
> + } while (0)
> +
> +#define LZ4_COPYPACKET(s, d) LZ4_COPYSTEP(s, d)
> +
> +#define LZ4_SECURECOPY(s, d, e) \
> + do { \
> + if (d < e) { \
> + LZ4_WILDCOPY(s, d, e); \
> + } \
> + } while (0)
> +
> +#else /* 32-bit */
> +#define STEPSIZE 4
> +
> +#define LZ4_COPYSTEP(s, d) \
> + do { \
> + PUT4(s, d); \
> + d += 4; \
> + s += 4; \
> + } while (0)
> +
> +#define LZ4_COPYPACKET(s, d) \
> + do { \
> + LZ4_COPYSTEP(s, d); \
> + LZ4_COPYSTEP(s, d); \
> + } while (0)
> +
> +#define LZ4_SECURECOPY LZ4_WILDCOPY
> +#endif
> +
> +#define LZ4_READ_LITTLEENDIAN_16(d, s, p) \
> + (d = s - get_unaligned_le16(p))
> +#define LZ4_WILDCOPY(s, d, e) \
> + do { \
> + LZ4_COPYPACKET(s, d); \
> + } while (d < e)
All the \ at the ends of lines would look better aligned in one column.
david
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 1/4] decompressor: Add LZ4 decompressor module
@ 2013-02-26 13:12 ` David Sterba
0 siblings, 0 replies; 67+ messages in thread
From: David Sterba @ 2013-02-26 13:12 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, Feb 26, 2013 at 03:24:27PM +0900, Kyungsik Lee wrote:
> This patch adds support for LZ4 decompression in the Linux Kernel.
> LZ4 Decompression APIs for kernel are based on LZ4 implementation
> by Yann Collet.
>
> LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
> LZ4 source repository : http://code.google.com/p/lz4/
What SVN version did you use?
> --- /dev/null
> +++ b/include/linux/lz4.h
> @@ -0,0 +1,48 @@
> +#ifndef __LZ4_H__
> +#define __LZ4_H__
> +/*
> + * LZ4 Kernel Interface
> + *
> + * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + */
> +
> +/*
> + * LZ4_COMPRESSBOUND()
> + * Provides the maximum size that LZ4 may output in a "worst case" scenario
> + * (input data not compressible)
> + */
> +#define LZ4_COMPRESSBOUND(isize) (isize + ((isize)/255) + 16)
For safety reasons I suggest to add a temporary variable to avoid double
evaluation of isize.
> --- /dev/null
> +++ b/lib/lz4/lz4_decompress.c
> + }
> + cpy = op + length - (STEPSIZE - 4);
> + if (cpy > oend - COPYLENGTH) {
> +
> + /* Error: request to write beyond destination buffer */
> + if (cpy > oend)
> + goto _output_error;
> + LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
> + while (op < cpy)
> + *op++ = *ref++;
> + op = cpy;
> + /*
> + * Check EOF (should never happen, since last 5 bytes
> + * are supposed to be literals)
> + */
> + if (op == oend)
> + goto _output_error;
> + continue;
> + }
> + LZ4_SECURECOPY(ref, op, cpy);
> + op = cpy; /* correction */
> + }
Does this compile? The } is an extra one, and does not match the
original sources.
> + /* end of decoding */
> + return (int) (((char *)ip) - source);
> +
> + /* write overflow error detected */
> +_output_error:
> + return (int) (-(((char *)ip) - source));
> +}
> +
> --- /dev/null
> +++ b/lib/lz4/lz4defs.h
> @@ -0,0 +1,93 @@
> +#define LZ4_COPYSTEP(s, d) \
> + do { \
> + PUT8(s, d); \
> + d += 8; \
> + s += 8; \
> + } while (0)
> +
> +#define LZ4_COPYPACKET(s, d) LZ4_COPYSTEP(s, d)
> +
> +#define LZ4_SECURECOPY(s, d, e) \
> + do { \
> + if (d < e) { \
> + LZ4_WILDCOPY(s, d, e); \
> + } \
> + } while (0)
> +
> +#else /* 32-bit */
> +#define STEPSIZE 4
> +
> +#define LZ4_COPYSTEP(s, d) \
> + do { \
> + PUT4(s, d); \
> + d += 4; \
> + s += 4; \
> + } while (0)
> +
> +#define LZ4_COPYPACKET(s, d) \
> + do { \
> + LZ4_COPYSTEP(s, d); \
> + LZ4_COPYSTEP(s, d); \
> + } while (0)
> +
> +#define LZ4_SECURECOPY LZ4_WILDCOPY
> +#endif
> +
> +#define LZ4_READ_LITTLEENDIAN_16(d, s, p) \
> + (d = s - get_unaligned_le16(p))
> +#define LZ4_WILDCOPY(s, d, e) \
> + do { \
> + LZ4_COPYPACKET(s, d); \
> + } while (d < e)
All the \@the ends of lines would look better aligned in one column.
david
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 2/4] lib: Add support for LZ4-compressed kernel
2013-02-26 6:24 ` Kyungsik Lee
@ 2013-02-26 14:00 ` David Sterba
-1 siblings, 0 replies; 67+ messages in thread
From: David Sterba @ 2013-02-26 14:00 UTC (permalink / raw)
To: Kyungsik Lee
Cc: Andrew Morton, Russell King, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nicolas Pitre, Nitin Gupta,
Markus F.X.J. Oberhumer, Richard Purdie, Josh Triplett,
Joe Millenbach, David Sterba, Richard Cochran, Albin Tonnerre,
Egon Alter, hyojun.im, chan.jeong, raphael.andy.lee
On Tue, Feb 26, 2013 at 03:24:28PM +0900, Kyungsik Lee wrote:
> +config KERNEL_LZ4
> + bool "LZ4"
> + depends on HAVE_KERNEL_LZ4
> + help
> + Its compression ratio is worse than LZO. The size of the kernel
> + is about 8% bigger than LZO. But the decompression speed is
> + faster than LZO.
> +
Can you please add a sentence what lz4 actually is before you start
comparing it with the current competitor(s)?
> --- /dev/null
> +++ b/lib/decompress_unlz4.c
> @@ -0,0 +1,190 @@
> +#define LZ4_CHUNK_SIZE (8<<20)
Please use a less cryptic way of representing a value of 8 MB. Also,
this is a hardcoded value that must be used on the compressor side. It's
an upper limit, so anything below 8MB does not break decompresssion, but
this must be somehow checked or saved along in the binary stream. You
seem to use the lz4demo.c on the userspace side for compression, but
this is not a standard tool nor the output format is well-defined or
stabilized.
For proper use I would like to see a commandline tool similar to
gzip/bzip2/lzop that can be packaged and shipped by distros, and the
output format defintion.
Yann has some ideas for the format
http://fastcompression.blogspot.cz/2012/04/file-container-format-for-lz4.html
For kernel, the minimum of meta information is total compressed length,
total uncompressed length and chunk size. I don't know if the first two
aren't stored elsewhere in the generic kernel image headers, but chunk
size must be specified.
david
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 2/4] lib: Add support for LZ4-compressed kernel
@ 2013-02-26 14:00 ` David Sterba
0 siblings, 0 replies; 67+ messages in thread
From: David Sterba @ 2013-02-26 14:00 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, Feb 26, 2013 at 03:24:28PM +0900, Kyungsik Lee wrote:
> +config KERNEL_LZ4
> + bool "LZ4"
> + depends on HAVE_KERNEL_LZ4
> + help
> + Its compression ratio is worse than LZO. The size of the kernel
> + is about 8% bigger than LZO. But the decompression speed is
> + faster than LZO.
> +
Can you please add a sentence what lz4 actually is before you start
comparing it with the current competitor(s)?
> --- /dev/null
> +++ b/lib/decompress_unlz4.c
> @@ -0,0 +1,190 @@
> +#define LZ4_CHUNK_SIZE (8<<20)
Please use a less cryptic way of representing a value of 8 MB. Also,
this is a hardcoded value that must be used on the compressor side. It's
an upper limit, so anything below 8MB does not break decompresssion, but
this must be somehow checked or saved along in the binary stream. You
seem to use the lz4demo.c on the userspace side for compression, but
this is not a standard tool nor the output format is well-defined or
stabilized.
For proper use I would like to see a commandline tool similar to
gzip/bzip2/lzop that can be packaged and shipped by distros, and the
output format defintion.
Yann has some ideas for the format
http://fastcompression.blogspot.cz/2012/04/file-container-format-for-lz4.html
For kernel, the minimum of meta information is total compressed length,
total uncompressed length and chunk size. I don't know if the first two
aren't stored elsewhere in the generic kernel image headers, but chunk
size must be specified.
david
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-26 6:24 ` Kyungsik Lee
@ 2013-02-26 20:33 ` Markus F.X.J. Oberhumer
-1 siblings, 0 replies; 67+ messages in thread
From: Markus F.X.J. Oberhumer @ 2013-02-26 20:33 UTC (permalink / raw)
To: Kyungsik Lee
Cc: Andrew Morton, Russell King, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nicolas Pitre, Nitin Gupta,
Richard Purdie, Josh Triplett, Joe Millenbach, David Sterba,
Richard Cochran, Albin Tonnerre, Egon Alter, hyojun.im,
chan.jeong, raphael.andy.lee
[-- Attachment #1: Type: text/plain, Size: 3866 bytes --]
On 2013-02-26 07:24, Kyungsik Lee wrote:
> Hi,
>
> [...]
>
> Through the benchmark, it was found that -Os Compiler flag for
> decompress.o brought better decompression performance in most of cases
> (ex, different compiler and hardware spec.) in ARM architecture.
>
> Lastly, CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is not always the best
> option even though it is supported. The decompression speed can be
> slightly slower in some cases.
>
> This patchset is based on 3.8.
>
> Any comments are appreciated.
Did you actually *try* the new LZO version and the patch (which is attached
once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
Because the new LZO version is faster than LZ4 in my testing, at least
when comparing apples with apples and enabling unaligned access in
BOTH versions:
armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
compression speed decompression speed
LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
~Markus
>
> Thanks,
> Kyungsik
>
>
> Benchmark Results(PATCH v2)
> Compiler: Linaro ARM gcc 4.6.2
> 1. ARMv7, 1.5GHz based board
> Kernel: linux 3.4
> Uncompressed Kernel Size: 14MB
> Compressed Size Decompression Speed
> LZO 6.7MB 21.1MB/s
> LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA)
> 2. ARMv7, 1.7GHz based board
> Kernel: linux 3.7
> Uncompressed Kernel Size: 14MB
> Compressed Size Decompression Speed
> LZO 6.0MB 34.1MB/s
> LZ4 6.5MB 86.7MB/s
> UA: Unaligned memory Access support
>
>
> Change log: v2
> - Clean up code
> - Enable unaligned access for ARM v6 and above with
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
> - Add lz4_decompress() for faster decompression with
> uncompressed output size
> - Use lz4_decompress() for LZ4-compressed kernel during
> boot-process
> - Apply -Os to decompress.o to improve decompress
> performance during boot-up process
>
>
> Kyungsik Lee (4):
> decompressor: Add LZ4 decompressor module
> lib: Add support for LZ4-compressed kernel
> arm: Add support for LZ4-compressed kernel
> x86: Add support for LZ4-compressed kernel
>
> arch/arm/Kconfig | 1 +
> arch/arm/boot/compressed/.gitignore | 1 +
> arch/arm/boot/compressed/Makefile | 6 +-
> arch/arm/boot/compressed/decompress.c | 4 +
> arch/arm/boot/compressed/piggy.lz4.S | 6 +
> arch/x86/Kconfig | 1 +
> arch/x86/boot/compressed/Makefile | 5 +-
> arch/x86/boot/compressed/misc.c | 4 +
> include/linux/decompress/unlz4.h | 10 +
> include/linux/lz4.h | 48 +++++
> init/Kconfig | 13 +-
> lib/Kconfig | 7 +
> lib/Makefile | 2 +
> lib/decompress.c | 5 +
> lib/decompress_unlz4.c | 190 +++++++++++++++++++
> lib/lz4/Makefile | 1 +
> lib/lz4/lz4_decompress.c | 331 ++++++++++++++++++++++++++++++++++
> lib/lz4/lz4defs.h | 93 ++++++++++
> scripts/Makefile.lib | 5 +
> usr/Kconfig | 9 +
> 20 files changed, 739 insertions(+), 3 deletions(-)
> create mode 100644 arch/arm/boot/compressed/piggy.lz4.S
> create mode 100644 include/linux/decompress/unlz4.h
> create mode 100644 include/linux/lz4.h
> create mode 100644 lib/decompress_unlz4.c
> create mode 100644 lib/lz4/Makefile
> create mode 100644 lib/lz4/lz4_decompress.c
> create mode 100644 lib/lz4/lz4defs.h
>
--
Markus Oberhumer, <markus@oberhumer.com>, http://www.oberhumer.com/
[-- Attachment #2: lib-lzo-huge-LZO-decompression-speedup-on-ARM.patch --]
[-- Type: text/x-patch, Size: 1584 bytes --]
commit 8745b927fcfcd6953ada9bd1220a73083db5948a
Author: Markus F.X.J. Oberhumer <markus@oberhumer.com>
Date: Mon Feb 4 02:26:14 2013 +0100
lib/lzo: huge LZO decompression speedup on ARM by using unaligned access
Signed-off-by: Markus F.X.J. Oberhumer <markus@oberhumer.com>
diff --git a/lib/lzo/lzo1x_decompress_safe.c b/lib/lzo/lzo1x_decompress_safe.c
index 569985d..e3edc5f 100644
--- a/lib/lzo/lzo1x_decompress_safe.c
+++ b/lib/lzo/lzo1x_decompress_safe.c
@@ -72,9 +72,11 @@ copy_literal_run:
COPY8(op, ip);
op += 8;
ip += 8;
+# if !defined(__arm__)
COPY8(op, ip);
op += 8;
ip += 8;
+# endif
} while (ip < ie);
ip = ie;
op = oe;
@@ -159,9 +161,11 @@ copy_literal_run:
COPY8(op, m_pos);
op += 8;
m_pos += 8;
+# if !defined(__arm__)
COPY8(op, m_pos);
op += 8;
m_pos += 8;
+# endif
} while (op < oe);
op = oe;
if (HAVE_IP(6)) {
diff --git a/lib/lzo/lzodefs.h b/lib/lzo/lzodefs.h
index 5a4beb2..b230601 100644
--- a/lib/lzo/lzodefs.h
+++ b/lib/lzo/lzodefs.h
@@ -12,8 +12,14 @@
*/
+#if 1 && defined(__arm__) && ((__LINUX_ARM_ARCH__ >= 6) || defined(__ARM_FEATURE_UNALIGNED))
+#define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS 1
+#define COPY4(dst, src) \
+ * (u32 *) (void *) (dst) = * (const u32 *) (const void *) (src)
+#else
#define COPY4(dst, src) \
put_unaligned(get_unaligned((const u32 *)(src)), (u32 *)(dst))
+#endif
#if defined(__x86_64__)
#define COPY8(dst, src) \
put_unaligned(get_unaligned((const u64 *)(src)), (u64 *)(dst))
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-26 20:33 ` Markus F.X.J. Oberhumer
0 siblings, 0 replies; 67+ messages in thread
From: Markus F.X.J. Oberhumer @ 2013-02-26 20:33 UTC (permalink / raw)
To: linux-arm-kernel
On 2013-02-26 07:24, Kyungsik Lee wrote:
> Hi,
>
> [...]
>
> Through the benchmark, it was found that -Os Compiler flag for
> decompress.o brought better decompression performance in most of cases
> (ex, different compiler and hardware spec.) in ARM architecture.
>
> Lastly, CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is not always the best
> option even though it is supported. The decompression speed can be
> slightly slower in some cases.
>
> This patchset is based on 3.8.
>
> Any comments are appreciated.
Did you actually *try* the new LZO version and the patch (which is attached
once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
Because the new LZO version is faster than LZ4 in my testing, at least
when comparing apples with apples and enabling unaligned access in
BOTH versions:
armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
compression speed decompression speed
LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
~Markus
>
> Thanks,
> Kyungsik
>
>
> Benchmark Results(PATCH v2)
> Compiler: Linaro ARM gcc 4.6.2
> 1. ARMv7, 1.5GHz based board
> Kernel: linux 3.4
> Uncompressed Kernel Size: 14MB
> Compressed Size Decompression Speed
> LZO 6.7MB 21.1MB/s
> LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA)
> 2. ARMv7, 1.7GHz based board
> Kernel: linux 3.7
> Uncompressed Kernel Size: 14MB
> Compressed Size Decompression Speed
> LZO 6.0MB 34.1MB/s
> LZ4 6.5MB 86.7MB/s
> UA: Unaligned memory Access support
>
>
> Change log: v2
> - Clean up code
> - Enable unaligned access for ARM v6 and above with
> CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
> - Add lz4_decompress() for faster decompression with
> uncompressed output size
> - Use lz4_decompress() for LZ4-compressed kernel during
> boot-process
> - Apply -Os to decompress.o to improve decompress
> performance during boot-up process
>
>
> Kyungsik Lee (4):
> decompressor: Add LZ4 decompressor module
> lib: Add support for LZ4-compressed kernel
> arm: Add support for LZ4-compressed kernel
> x86: Add support for LZ4-compressed kernel
>
> arch/arm/Kconfig | 1 +
> arch/arm/boot/compressed/.gitignore | 1 +
> arch/arm/boot/compressed/Makefile | 6 +-
> arch/arm/boot/compressed/decompress.c | 4 +
> arch/arm/boot/compressed/piggy.lz4.S | 6 +
> arch/x86/Kconfig | 1 +
> arch/x86/boot/compressed/Makefile | 5 +-
> arch/x86/boot/compressed/misc.c | 4 +
> include/linux/decompress/unlz4.h | 10 +
> include/linux/lz4.h | 48 +++++
> init/Kconfig | 13 +-
> lib/Kconfig | 7 +
> lib/Makefile | 2 +
> lib/decompress.c | 5 +
> lib/decompress_unlz4.c | 190 +++++++++++++++++++
> lib/lz4/Makefile | 1 +
> lib/lz4/lz4_decompress.c | 331 ++++++++++++++++++++++++++++++++++
> lib/lz4/lz4defs.h | 93 ++++++++++
> scripts/Makefile.lib | 5 +
> usr/Kconfig | 9 +
> 20 files changed, 739 insertions(+), 3 deletions(-)
> create mode 100644 arch/arm/boot/compressed/piggy.lz4.S
> create mode 100644 include/linux/decompress/unlz4.h
> create mode 100644 include/linux/lz4.h
> create mode 100644 lib/decompress_unlz4.c
> create mode 100644 lib/lz4/Makefile
> create mode 100644 lib/lz4/lz4_decompress.c
> create mode 100644 lib/lz4/lz4defs.h
>
--
Markus Oberhumer, <markus@oberhumer.com>, http://www.oberhumer.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lib-lzo-huge-LZO-decompression-speedup-on-ARM.patch
Type: text/x-patch
Size: 1584 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130226/266cf234/attachment.bin>
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-26 20:33 ` Markus F.X.J. Oberhumer
@ 2013-02-26 20:59 ` Nicolas Pitre
-1 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-26 20:59 UTC (permalink / raw)
To: Markus F.X.J. Oberhumer
Cc: Kyungsik Lee, Andrew Morton, Russell King, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Michal Marek, linux-arm-kernel,
linux-kernel, linux-kbuild, x86, celinux-dev, Nitin Gupta,
Richard Purdie, Josh Triplett, Joe Millenbach, David Sterba,
Richard Cochran, Albin Tonnerre, Egon Alter, hyojun.im,
chan.jeong, raphael.andy.lee
On Tue, 26 Feb 2013, Markus F.X.J. Oberhumer wrote:
> On 2013-02-26 07:24, Kyungsik Lee wrote:
> > Hi,
> >
> > [...]
> >
> > Through the benchmark, it was found that -Os Compiler flag for
> > decompress.o brought better decompression performance in most of cases
> > (ex, different compiler and hardware spec.) in ARM architecture.
> >
> > Lastly, CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is not always the best
> > option even though it is supported. The decompression speed can be
> > slightly slower in some cases.
> >
> > This patchset is based on 3.8.
> >
> > Any comments are appreciated.
>
> Did you actually *try* the new LZO version and the patch (which is attached
> once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
>
> Because the new LZO version is faster than LZ4 in my testing, at least
> when comparing apples with apples and enabling unaligned access in
> BOTH versions:
>
> armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
>
> compression speed decompression speed
>
> LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
> LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
> LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
To be fair, you should also take into account the compressed size of a
typical ARM kernel. Sometimes a slightly slower decompressor may be
faster overall if the compressed image to work on is smaller.
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-26 20:59 ` Nicolas Pitre
0 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-26 20:59 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, 26 Feb 2013, Markus F.X.J. Oberhumer wrote:
> On 2013-02-26 07:24, Kyungsik Lee wrote:
> > Hi,
> >
> > [...]
> >
> > Through the benchmark, it was found that -Os Compiler flag for
> > decompress.o brought better decompression performance in most of cases
> > (ex, different compiler and hardware spec.) in ARM architecture.
> >
> > Lastly, CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is not always the best
> > option even though it is supported. The decompression speed can be
> > slightly slower in some cases.
> >
> > This patchset is based on 3.8.
> >
> > Any comments are appreciated.
>
> Did you actually *try* the new LZO version and the patch (which is attached
> once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
>
> Because the new LZO version is faster than LZ4 in my testing, at least
> when comparing apples with apples and enabling unaligned access in
> BOTH versions:
>
> armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
>
> compression speed decompression speed
>
> LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
> LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
> LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
To be fair, you should also take into account the compressed size of a
typical ARM kernel. Sometimes a slightly slower decompressor may be
faster overall if the compressed image to work on is smaller.
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-26 20:59 ` Nicolas Pitre
@ 2013-02-26 21:58 ` Peter Korsgaard
-1 siblings, 0 replies; 67+ messages in thread
From: Peter Korsgaard @ 2013-02-26 21:58 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Markus F.X.J. Oberhumer, Kyungsik Lee, Andrew Morton,
Russell King, Thomas Gleixner, Ingo Molnar, H. Peter Anvin,
Michal Marek, linux-arm-kernel, linux-kernel, linux-kbuild, x86,
celinux-dev, Nitin Gupta, Richard Purdie, Josh Triplett,
Joe Millenbach, David Sterba, Richard Cochran, Albin Tonnerre,
Egon Alter, hyojun.im, chan.jeong, raphael.andy.lee
>>>>> "Nicolas" == Nicolas Pitre <nico@fluxnic.net> writes:
Hi,
>> Did you actually *try* the new LZO version and the patch (which is attached
>> once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
>>
>> Because the new LZO version is faster than LZ4 in my testing, at least
>> when comparing apples with apples and enabling unaligned access in
>> BOTH versions:
>>
>> armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
>>
>> compression speed decompression speed
>>
>> LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
>> LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
>> LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
Nicolas> To be fair, you should also take into account the compressed
Nicolas> size of a typical ARM kernel. Sometimes a slightly slower
Nicolas> decompressor may be faster overall if the compressed image to
Nicolas> work on is smaller.
Yes, but notice that lzo compressed BETTER than lz4 - E.G. from the
introduction mail:
1. ARMv7, 1.5GHz based board
Kernel: linux 3.4
Uncompressed Kernel Size: 14MB
Compressed Size Decompression Speed
LZO 6.7MB 21.1MB/s
LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA)
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-26 21:58 ` Peter Korsgaard
0 siblings, 0 replies; 67+ messages in thread
From: Peter Korsgaard @ 2013-02-26 21:58 UTC (permalink / raw)
To: linux-arm-kernel
>>>>> "Nicolas" == Nicolas Pitre <nico@fluxnic.net> writes:
Hi,
>> Did you actually *try* the new LZO version and the patch (which is attached
>> once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
>>
>> Because the new LZO version is faster than LZ4 in my testing, at least
>> when comparing apples with apples and enabling unaligned access in
>> BOTH versions:
>>
>> armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
>>
>> compression speed decompression speed
>>
>> LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
>> LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
>> LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
Nicolas> To be fair, you should also take into account the compressed
Nicolas> size of a typical ARM kernel. Sometimes a slightly slower
Nicolas> decompressor may be faster overall if the compressed image to
Nicolas> work on is smaller.
Yes, but notice that lzo compressed BETTER than lz4 - E.G. from the
introduction mail:
1. ARMv7, 1.5GHz based board
Kernel: linux 3.4
Uncompressed Kernel Size: 14MB
Compressed Size Decompression Speed
LZO 6.7MB 21.1MB/s
LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA)
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-26 21:58 ` Peter Korsgaard
@ 2013-02-26 22:09 ` Nicolas Pitre
-1 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-26 22:09 UTC (permalink / raw)
To: Peter Korsgaard
Cc: Markus F.X.J. Oberhumer, Kyungsik Lee, Andrew Morton,
Russell King, Thomas Gleixner, Ingo Molnar, H. Peter Anvin,
Michal Marek, linux-arm-kernel, linux-kernel, linux-kbuild, x86,
celinux-dev, Nitin Gupta, Richard Purdie, Josh Triplett,
Joe Millenbach, David Sterba, Richard Cochran, Albin Tonnerre,
Egon Alter, hyojun.im, chan.jeong, raphael.andy.lee
On Tue, 26 Feb 2013, Peter Korsgaard wrote:
> >>>>> "Nicolas" == Nicolas Pitre <nico@fluxnic.net> writes:
>
> Hi,
>
> >> Did you actually *try* the new LZO version and the patch (which is attached
> >> once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
> >>
> >> Because the new LZO version is faster than LZ4 in my testing, at least
> >> when comparing apples with apples and enabling unaligned access in
> >> BOTH versions:
> >>
> >> armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
> >>
> >> compression speed decompression speed
> >>
> >> LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
> >> LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
> >> LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
>
> Nicolas> To be fair, you should also take into account the compressed
> Nicolas> size of a typical ARM kernel. Sometimes a slightly slower
> Nicolas> decompressor may be faster overall if the compressed image to
> Nicolas> work on is smaller.
>
> Yes, but notice that lzo compressed BETTER than lz4 - E.G. from the
> introduction mail:
>
> 1. ARMv7, 1.5GHz based board
> Kernel: linux 3.4
> Uncompressed Kernel Size: 14MB
> Compressed Size Decompression Speed
> LZO 6.7MB 21.1MB/s
> LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA)
OK. If LZO is now faster than LZ4 while still compressing more then I
have no argument.
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-26 22:09 ` Nicolas Pitre
0 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-26 22:09 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, 26 Feb 2013, Peter Korsgaard wrote:
> >>>>> "Nicolas" == Nicolas Pitre <nico@fluxnic.net> writes:
>
> Hi,
>
> >> Did you actually *try* the new LZO version and the patch (which is attached
> >> once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
> >>
> >> Because the new LZO version is faster than LZ4 in my testing, at least
> >> when comparing apples with apples and enabling unaligned access in
> >> BOTH versions:
> >>
> >> armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
> >>
> >> compression speed decompression speed
> >>
> >> LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
> >> LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
> >> LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
>
> Nicolas> To be fair, you should also take into account the compressed
> Nicolas> size of a typical ARM kernel. Sometimes a slightly slower
> Nicolas> decompressor may be faster overall if the compressed image to
> Nicolas> work on is smaller.
>
> Yes, but notice that lzo compressed BETTER than lz4 - E.G. from the
> introduction mail:
>
> 1. ARMv7, 1.5GHz based board
> Kernel: linux 3.4
> Uncompressed Kernel Size: 14MB
> Compressed Size Decompression Speed
> LZO 6.7MB 21.1MB/s
> LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA)
OK. If LZO is now faster than LZ4 while still compressing more then I
have no argument.
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-26 21:58 ` Peter Korsgaard
@ 2013-02-26 22:10 ` Russell King - ARM Linux
-1 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-26 22:10 UTC (permalink / raw)
To: Peter Korsgaard
Cc: Nicolas Pitre, Markus F.X.J. Oberhumer, Kyungsik Lee,
Andrew Morton, Thomas Gleixner, Ingo Molnar, H. Peter Anvin,
Michal Marek, linux-arm-kernel, linux-kernel, linux-kbuild, x86,
celinux-dev, Nitin Gupta, Richard Purdie, Josh Triplett,
Joe Millenbach, David Sterba, Richard Cochran, Albin Tonnerre,
Egon Alter, hyojun.im, chan.jeong, raphael.andy.lee
On Tue, Feb 26, 2013 at 10:58:02PM +0100, Peter Korsgaard wrote:
> >>>>> "Nicolas" == Nicolas Pitre <nico@fluxnic.net> writes:
>
> Hi,
>
> >> Did you actually *try* the new LZO version and the patch (which is attached
> >> once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
> >>
> >> Because the new LZO version is faster than LZ4 in my testing, at least
> >> when comparing apples with apples and enabling unaligned access in
> >> BOTH versions:
> >>
> >> armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
> >>
> >> compression speed decompression speed
> >>
> >> LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
> >> LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
> >> LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
>
> Nicolas> To be fair, you should also take into account the compressed
> Nicolas> size of a typical ARM kernel. Sometimes a slightly slower
> Nicolas> decompressor may be faster overall if the compressed image to
> Nicolas> work on is smaller.
>
> Yes, but notice that lzo compressed BETTER than lz4 - E.G. from the
> introduction mail:
>
> 1. ARMv7, 1.5GHz based board
> Kernel: linux 3.4
> Uncompressed Kernel Size: 14MB
> Compressed Size Decompression Speed
> LZO 6.7MB 21.1MB/s
> LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA)
Well, until someone can put all the pieces together so that a reasonably
meaningful test between:
- The new LZO code
- The new LZ4 code
then you're all comparing different things. TBH, I'm disappointed that
all the comments about this from the previous posting of LZ4 have been
totally ignored, and we _still_ don't really have this information. It
seems like replying to these threads is a waste of time.
So... for a selected kernel version of a particular size, can we please
have a comparison between the new LZO code and this LZ4 code, so that
we can see whether it's worth updating the LZO code or replacing the
LZO code with LZ4?
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-26 22:10 ` Russell King - ARM Linux
0 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-26 22:10 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, Feb 26, 2013 at 10:58:02PM +0100, Peter Korsgaard wrote:
> >>>>> "Nicolas" == Nicolas Pitre <nico@fluxnic.net> writes:
>
> Hi,
>
> >> Did you actually *try* the new LZO version and the patch (which is attached
> >> once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
> >>
> >> Because the new LZO version is faster than LZ4 in my testing, at least
> >> when comparing apples with apples and enabling unaligned access in
> >> BOTH versions:
> >>
> >> armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
> >>
> >> compression speed decompression speed
> >>
> >> LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
> >> LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
> >> LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
>
> Nicolas> To be fair, you should also take into account the compressed
> Nicolas> size of a typical ARM kernel. Sometimes a slightly slower
> Nicolas> decompressor may be faster overall if the compressed image to
> Nicolas> work on is smaller.
>
> Yes, but notice that lzo compressed BETTER than lz4 - E.G. from the
> introduction mail:
>
> 1. ARMv7, 1.5GHz based board
> Kernel: linux 3.4
> Uncompressed Kernel Size: 14MB
> Compressed Size Decompression Speed
> LZO 6.7MB 21.1MB/s
> LZ4 7.3MB 29.1MB/s, 45.6MB/s(UA)
Well, until someone can put all the pieces together so that a reasonably
meaningful test between:
- The new LZO code
- The new LZ4 code
then you're all comparing different things. TBH, I'm disappointed that
all the comments about this from the previous posting of LZ4 have been
totally ignored, and we _still_ don't really have this information. It
seems like replying to these threads is a waste of time.
So... for a selected kernel version of a particular size, can we please
have a comparison between the new LZO code and this LZ4 code, so that
we can see whether it's worth updating the LZO code or replacing the
LZO code with LZ4?
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-26 22:10 ` Russell King - ARM Linux
@ 2013-02-27 1:40 ` Joe Perches
-1 siblings, 0 replies; 67+ messages in thread
From: Joe Perches @ 2013-02-27 1:40 UTC (permalink / raw)
To: Russell King - ARM Linux
Cc: Peter Korsgaard, Nicolas Pitre, Markus F.X.J. Oberhumer,
Kyungsik Lee, Andrew Morton, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nitin Gupta, Richard Purdie,
Josh Triplett, Joe Millenbach, David Sterba, Richard Cochran,
Albin Tonnerre, Egon Alter, hyojun.im, chan.jeong,
raphael.andy.lee
On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> So... for a selected kernel version of a particular size, can we please
> have a comparison between the new LZO code and this LZ4 code, so that
> we can see whether it's worth updating the LZO code or replacing the
> LZO code with LZ4?
How could it be questionable that it's worth updating the LZO code?
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 1:40 ` Joe Perches
0 siblings, 0 replies; 67+ messages in thread
From: Joe Perches @ 2013-02-27 1:40 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> So... for a selected kernel version of a particular size, can we please
> have a comparison between the new LZO code and this LZ4 code, so that
> we can see whether it's worth updating the LZO code or replacing the
> LZO code with LZ4?
How could it be questionable that it's worth updating the LZO code?
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 1/4] decompressor: Add LZ4 decompressor module
2013-02-26 13:12 ` David Sterba
@ 2013-02-27 4:38 ` Kyungsik Lee
-1 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-27 4:38 UTC (permalink / raw)
To: David Sterba
Cc: Andrew Morton, Russell King, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nicolas Pitre, Nitin Gupta,
Markus F.X.J. Oberhumer, Richard Purdie, Josh Triplett,
Joe Millenbach, Richard Cochran, Albin Tonnerre, Egon Alter,
hyojun.im, chan.jeong, raphael.andy.lee
On Tue, Feb 26, 2013 at 02:12:06PM +0100, David Sterba wrote:
> On Tue, Feb 26, 2013 at 03:24:27PM +0900, Kyungsik Lee wrote:
> > This patch adds support for LZ4 decompression in the Linux Kernel.
> > LZ4 Decompression APIs for kernel are based on LZ4 implementation
> > by Yann Collet.
> >
> > LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
> > LZ4 source repository : http://code.google.com/p/lz4/
>
> What SVN version did you use?
It's based on r88.
> > +/*
> > + * LZ4_COMPRESSBOUND()
> > + * Provides the maximum size that LZ4 may output in a "worst case" scenario
> > + * (input data not compressible)
> > + */
> > +#define LZ4_COMPRESSBOUND(isize) (isize + ((isize)/255) + 16)
>
> For safety reasons I suggest to add a temporary variable to avoid double
> evaluation of isize.
Yes, Good point.
> > --- /dev/null
> > +++ b/lib/lz4/lz4_decompress.c
> > + }
> > + cpy = op + length - (STEPSIZE - 4);
> > + if (cpy > oend - COPYLENGTH) {
> > +
> > + /* Error: request to write beyond destination buffer */
> > + if (cpy > oend)
> > + goto _output_error;
> > + LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
> > + while (op < cpy)
> > + *op++ = *ref++;
> > + op = cpy;
> > + /*
> > + * Check EOF (should never happen, since last 5 bytes
> > + * are supposed to be literals)
> > + */
> > + if (op == oend)
> > + goto _output_error;
> > + continue;
> > + }
> > + LZ4_SECURECOPY(ref, op, cpy);
> > + op = cpy; /* correction */
> > + }
>
> Does this compile? The } is an extra one, and does not match the
> original sources.
>
It's about indent errors. The } is a pair of braces regarding
"while (1) {". However it compiled. It will be fixed.
> > +#define LZ4_COPYPACKET(s, d) \
> > + do { \
> > + LZ4_COPYSTEP(s, d); \
> > + LZ4_COPYSTEP(s, d); \
> > + } while (0)
> > +
> > +#define LZ4_SECURECOPY LZ4_WILDCOPY
> > +#endif
> > +
> > +#define LZ4_READ_LITTLEENDIAN_16(d, s, p) \
> > + (d = s - get_unaligned_le16(p))
> > +#define LZ4_WILDCOPY(s, d, e) \
> > + do { \
> > + LZ4_COPYPACKET(s, d); \
> > + } while (d < e)
>
> All the \ at the ends of lines would look better aligned in one column.
>
Yes, right. It looks better if it's aligned.
Thanks,
Kyungsik
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 1/4] decompressor: Add LZ4 decompressor module
@ 2013-02-27 4:38 ` Kyungsik Lee
0 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-27 4:38 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, Feb 26, 2013 at 02:12:06PM +0100, David Sterba wrote:
> On Tue, Feb 26, 2013 at 03:24:27PM +0900, Kyungsik Lee wrote:
> > This patch adds support for LZ4 decompression in the Linux Kernel.
> > LZ4 Decompression APIs for kernel are based on LZ4 implementation
> > by Yann Collet.
> >
> > LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
> > LZ4 source repository : http://code.google.com/p/lz4/
>
> What SVN version did you use?
It's based on r88.
> > +/*
> > + * LZ4_COMPRESSBOUND()
> > + * Provides the maximum size that LZ4 may output in a "worst case" scenario
> > + * (input data not compressible)
> > + */
> > +#define LZ4_COMPRESSBOUND(isize) (isize + ((isize)/255) + 16)
>
> For safety reasons I suggest to add a temporary variable to avoid double
> evaluation of isize.
Yes, Good point.
> > --- /dev/null
> > +++ b/lib/lz4/lz4_decompress.c
> > + }
> > + cpy = op + length - (STEPSIZE - 4);
> > + if (cpy > oend - COPYLENGTH) {
> > +
> > + /* Error: request to write beyond destination buffer */
> > + if (cpy > oend)
> > + goto _output_error;
> > + LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
> > + while (op < cpy)
> > + *op++ = *ref++;
> > + op = cpy;
> > + /*
> > + * Check EOF (should never happen, since last 5 bytes
> > + * are supposed to be literals)
> > + */
> > + if (op == oend)
> > + goto _output_error;
> > + continue;
> > + }
> > + LZ4_SECURECOPY(ref, op, cpy);
> > + op = cpy; /* correction */
> > + }
>
> Does this compile? The } is an extra one, and does not match the
> original sources.
>
It's about indent errors. The } is a pair of braces regarding
"while (1) {". However it compiled. It will be fixed.
> > +#define LZ4_COPYPACKET(s, d) \
> > + do { \
> > + LZ4_COPYSTEP(s, d); \
> > + LZ4_COPYSTEP(s, d); \
> > + } while (0)
> > +
> > +#define LZ4_SECURECOPY LZ4_WILDCOPY
> > +#endif
> > +
> > +#define LZ4_READ_LITTLEENDIAN_16(d, s, p) \
> > + (d = s - get_unaligned_le16(p))
> > +#define LZ4_WILDCOPY(s, d, e) \
> > + do { \
> > + LZ4_COPYPACKET(s, d); \
> > + } while (d < e)
>
> All the \ at the ends of lines would look better aligned in one column.
>
Yes, right. It looks better if it's aligned.
Thanks,
Kyungsik
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-26 20:33 ` Markus F.X.J. Oberhumer
@ 2013-02-27 7:36 ` Kyungsik Lee
-1 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-27 7:36 UTC (permalink / raw)
To: Markus F.X.J. Oberhumer
Cc: Andrew Morton, Russell King, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nicolas Pitre, Nitin Gupta,
Richard Purdie, Josh Triplett, Joe Millenbach, David Sterba,
Richard Cochran, Albin Tonnerre, Egon Alter, hyojun.im,
chan.jeong, raphael.andy.lee
On Tue, Feb 26, 2013 at 09:33:22PM +0100, Markus F.X.J. Oberhumer wrote:
> On 2013-02-26 07:24, Kyungsik Lee wrote:
> > Hi,
> >
> > [...]
> >
> > Through the benchmark, it was found that -Os Compiler flag for
> > decompress.o brought better decompression performance in most of cases
> > (ex, different compiler and hardware spec.) in ARM architecture.
> >
> > Lastly, CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is not always the best
> > option even though it is supported. The decompression speed can be
> > slightly slower in some cases.
> >
> > This patchset is based on 3.8.
> >
> > Any comments are appreciated.
>
> Did you actually *try* the new LZO version and the patch (which is attached
> once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
>
> Because the new LZO version is faster than LZ4 in my testing, at least
> when comparing apples with apples and enabling unaligned access in
> BOTH versions:
>
> armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
>
> compression speed decompression speed
>
> LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
> LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
> LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
>
I agree that the new LZO version provided shows better decompression
speed than 3.7 based. It is much improved especially for UA.
Compiler: Linaro ARM gcc 4.6.2
2. ARMv7, 1.7GHz based board
Kernel: linux 3.7
Uncompressed Kernel Size: 14MB
Compressed Size Decompression Speed
LZO 6.0MB 34.1MB/s Old
----------------------------------------
6.0MB 34.7MB/s New
6.0MB 52.2MB/s(UA)
=============================================
LZ4 6.5MB 86.7MB/s
UA: Unaligned memory Access support
One thing I can say that the code you may have used, guessing
"lz4demo" is not the same code provided in this patch.
It has been ported for the kernel and uses different function
not like the "lz4demo".
Thanks,
Kyungsik
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 7:36 ` Kyungsik Lee
0 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-27 7:36 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, Feb 26, 2013 at 09:33:22PM +0100, Markus F.X.J. Oberhumer wrote:
> On 2013-02-26 07:24, Kyungsik Lee wrote:
> > Hi,
> >
> > [...]
> >
> > Through the benchmark, it was found that -Os Compiler flag for
> > decompress.o brought better decompression performance in most of cases
> > (ex, different compiler and hardware spec.) in ARM architecture.
> >
> > Lastly, CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is not always the best
> > option even though it is supported. The decompression speed can be
> > slightly slower in some cases.
> >
> > This patchset is based on 3.8.
> >
> > Any comments are appreciated.
>
> Did you actually *try* the new LZO version and the patch (which is attached
> once again) as explained in https://lkml.org/lkml/2013/2/3/367 ?
>
> Because the new LZO version is faster than LZ4 in my testing, at least
> when comparing apples with apples and enabling unaligned access in
> BOTH versions:
>
> armv7 (Cortex-A9), Linaro gcc-4.6 -O3, Silesia test corpus, 256 kB block-size:
>
> compression speed decompression speed
>
> LZO-2012 : 44 MB/sec 117 MB/sec no unaligned access
> LZO-2013-UA : 47 MB/sec 167 MB/sec Unaligned Access
> LZ4 r88 UA : 46 MB/sec 154 MB/sec Unaligned Access
>
I agree that the new LZO version provided shows better decompression
speed than 3.7 based. It is much improved especially for UA.
Compiler: Linaro ARM gcc 4.6.2
2. ARMv7, 1.7GHz based board
Kernel: linux 3.7
Uncompressed Kernel Size: 14MB
Compressed Size Decompression Speed
LZO 6.0MB 34.1MB/s Old
----------------------------------------
6.0MB 34.7MB/s New
6.0MB 52.2MB/s(UA)
=============================================
LZ4 6.5MB 86.7MB/s
UA: Unaligned memory Access support
One thing I can say that the code you may have used, guessing
"lz4demo" is not the same code provided in this patch.
It has been ported for the kernel and uses different function
not like the "lz4demo".
Thanks,
Kyungsik
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 7:36 ` Kyungsik Lee
@ 2013-02-27 9:51 ` Russell King - ARM Linux
-1 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-27 9:51 UTC (permalink / raw)
To: Kyungsik Lee, Andrew Morton
Cc: Markus F.X.J. Oberhumer, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nicolas Pitre, Nitin Gupta,
Richard Purdie, Josh Triplett, Joe Millenbach, David Sterba,
Richard Cochran, Albin Tonnerre, Egon Alter, hyojun.im,
chan.jeong, raphael.andy.lee
On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:
> Compiler: Linaro ARM gcc 4.6.2
> 2. ARMv7, 1.7GHz based board
> Kernel: linux 3.7
> Uncompressed Kernel Size: 14MB
> Compressed Size Decompression Speed
> LZO 6.0MB 34.1MB/s Old
> ----------------------------------------
> 6.0MB 34.7MB/s New
> 6.0MB 52.2MB/s(UA)
> =============================================
> LZ4 6.5MB 86.7MB/s
> UA: Unaligned memory Access support
That is pretty conclusive - it shows an 8% increase in image size vs a
66% increase in decompression speed. It will take a _lot_ to offset
that increase in decompression speed.
So, what I think is that yes, we should accept LZ4 and drop LZO from
the kernel - the "fast but may not be small" compression title has
clearly been taken by LZ4.
Akpm - what's your thoughts?
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 9:51 ` Russell King - ARM Linux
0 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-27 9:51 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:
> Compiler: Linaro ARM gcc 4.6.2
> 2. ARMv7, 1.7GHz based board
> Kernel: linux 3.7
> Uncompressed Kernel Size: 14MB
> Compressed Size Decompression Speed
> LZO 6.0MB 34.1MB/s Old
> ----------------------------------------
> 6.0MB 34.7MB/s New
> 6.0MB 52.2MB/s(UA)
> =============================================
> LZ4 6.5MB 86.7MB/s
> UA: Unaligned memory Access support
That is pretty conclusive - it shows an 8% increase in image size vs a
66% increase in decompression speed. It will take a _lot_ to offset
that increase in decompression speed.
So, what I think is that yes, we should accept LZ4 and drop LZO from
the kernel - the "fast but may not be small" compression title has
clearly been taken by LZ4.
Akpm - what's your thoughts?
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 1:40 ` Joe Perches
@ 2013-02-27 9:56 ` Russell King - ARM Linux
-1 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-27 9:56 UTC (permalink / raw)
To: Joe Perches
Cc: Peter Korsgaard, Nicolas Pitre, Markus F.X.J. Oberhumer,
Kyungsik Lee, Andrew Morton, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nitin Gupta, Richard Purdie,
Josh Triplett, Joe Millenbach, David Sterba, Richard Cochran,
Albin Tonnerre, Egon Alter, hyojun.im, chan.jeong,
raphael.andy.lee
On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > So... for a selected kernel version of a particular size, can we please
> > have a comparison between the new LZO code and this LZ4 code, so that
> > we can see whether it's worth updating the LZO code or replacing the
> > LZO code with LZ4?
>
> How could it be questionable that it's worth updating the LZO code?
Please read the comments against the previous posting of these patches
where I first stated this argument - and with agreement from those
following the thread. The thread started on 26 Jan 2013. Thanks.
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 9:56 ` Russell King - ARM Linux
0 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-27 9:56 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > So... for a selected kernel version of a particular size, can we please
> > have a comparison between the new LZO code and this LZ4 code, so that
> > we can see whether it's worth updating the LZO code or replacing the
> > LZO code with LZ4?
>
> How could it be questionable that it's worth updating the LZO code?
Please read the comments against the previous posting of these patches
where I first stated this argument - and with agreement from those
following the thread. The thread started on 26 Jan 2013. Thanks.
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 9:51 ` Russell King - ARM Linux
@ 2013-02-27 10:20 ` Johannes Stezenbach
-1 siblings, 0 replies; 67+ messages in thread
From: Johannes Stezenbach @ 2013-02-27 10:20 UTC (permalink / raw)
To: Russell King - ARM Linux
Cc: Kyungsik Lee, Andrew Morton, Markus F.X.J. Oberhumer,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Michal Marek,
linux-arm-kernel, linux-kernel, linux-kbuild, x86, celinux-dev,
Nicolas Pitre, Nitin Gupta, Richard Purdie, Josh Triplett,
Joe Millenbach, David Sterba, Richard Cochran, Albin Tonnerre,
Egon Alter, hyojun.im, chan.jeong, raphael.andy.lee
On Wed, Feb 27, 2013 at 09:51:39AM +0000, Russell King - ARM Linux wrote:
> On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:
> > Compiler: Linaro ARM gcc 4.6.2
> > 2. ARMv7, 1.7GHz based board
> > Kernel: linux 3.7
> > Uncompressed Kernel Size: 14MB
> > Compressed Size Decompression Speed
> > LZO 6.0MB 34.1MB/s Old
> > ----------------------------------------
> > 6.0MB 34.7MB/s New
> > 6.0MB 52.2MB/s(UA)
> > =============================================
> > LZ4 6.5MB 86.7MB/s
> > UA: Unaligned memory Access support
>
> That is pretty conclusive - it shows an 8% increase in image size vs a
> 66% increase in decompression speed. It will take a _lot_ to offset
> that increase in decompression speed.
>
> So, what I think is that yes, we should accept LZ4 and drop LZO from
> the kernel - the "fast but may not be small" compression title has
> clearly been taken by LZ4.
I think LZO may be used by squashfs, jffs2 and btrfs, thus you
cannot drop it without breaking on disk storage formats.
Johannes
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 10:20 ` Johannes Stezenbach
0 siblings, 0 replies; 67+ messages in thread
From: Johannes Stezenbach @ 2013-02-27 10:20 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, Feb 27, 2013 at 09:51:39AM +0000, Russell King - ARM Linux wrote:
> On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:
> > Compiler: Linaro ARM gcc 4.6.2
> > 2. ARMv7, 1.7GHz based board
> > Kernel: linux 3.7
> > Uncompressed Kernel Size: 14MB
> > Compressed Size Decompression Speed
> > LZO 6.0MB 34.1MB/s Old
> > ----------------------------------------
> > 6.0MB 34.7MB/s New
> > 6.0MB 52.2MB/s(UA)
> > =============================================
> > LZ4 6.5MB 86.7MB/s
> > UA: Unaligned memory Access support
>
> That is pretty conclusive - it shows an 8% increase in image size vs a
> 66% increase in decompression speed. It will take a _lot_ to offset
> that increase in decompression speed.
>
> So, what I think is that yes, we should accept LZ4 and drop LZO from
> the kernel - the "fast but may not be small" compression title has
> clearly been taken by LZ4.
I think LZO may be used by squashfs, jffs2 and btrfs, thus you
cannot drop it without breaking on disk storage formats.
Johannes
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 9:51 ` Russell King - ARM Linux
@ 2013-02-27 13:23 ` Kyungsik Lee
-1 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-27 13:23 UTC (permalink / raw)
To: Russell King - ARM Linux, Markus F.X.J. Oberhumer, Andrew Morton
Cc: Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Michal Marek,
linux-arm-kernel, linux-kernel, linux-kbuild, x86, celinux-dev,
Nicolas Pitre, Nitin Gupta, Richard Purdie, Josh Triplett,
Joe Millenbach, David Sterba, Richard Cochran, Albin Tonnerre,
Egon Alter, hyojun.im, chan.jeong, raphael.andy.lee
On Wed, Feb 27, 2013 at 09:51:39AM +0000, Russell King - ARM Linux wrote:
> On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:
> > Compiler: Linaro ARM gcc 4.6.2
> > 2. ARMv7, 1.7GHz based board
> > Kernel: linux 3.7
> > Uncompressed Kernel Size: 14MB
> > Compressed Size Decompression Speed
> > LZO 6.0MB 34.1MB/s Old
> > ----------------------------------------
> > 6.0MB 34.7MB/s New
> > 6.0MB 52.2MB/s(UA)
> > =============================================
> > LZ4 6.5MB 86.7MB/s
> > UA: Unaligned memory Access support
>
> That is pretty conclusive - it shows an 8% increase in image size vs a
> 66% increase in decompression speed. It will take a _lot_ to offset
> that increase in decompression speed.
>
> So, what I think is that yes, we should accept LZ4 and drop LZO from
> the kernel - the "fast but may not be small" compression title has
> clearly been taken by LZ4.
I have read the comments regarding how many compressors the kernel
should support and understand that it can not support all the
compressors available.
However, I don't think that LZO can be replaced by LZ4 in all the
cases. The benchmark above shows only about improved decompression
speed.
Thanks,
Kyungsik
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 13:23 ` Kyungsik Lee
0 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-27 13:23 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, Feb 27, 2013 at 09:51:39AM +0000, Russell King - ARM Linux wrote:
> On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:
> > Compiler: Linaro ARM gcc 4.6.2
> > 2. ARMv7, 1.7GHz based board
> > Kernel: linux 3.7
> > Uncompressed Kernel Size: 14MB
> > Compressed Size Decompression Speed
> > LZO 6.0MB 34.1MB/s Old
> > ----------------------------------------
> > 6.0MB 34.7MB/s New
> > 6.0MB 52.2MB/s(UA)
> > =============================================
> > LZ4 6.5MB 86.7MB/s
> > UA: Unaligned memory Access support
>
> That is pretty conclusive - it shows an 8% increase in image size vs a
> 66% increase in decompression speed. It will take a _lot_ to offset
> that increase in decompression speed.
>
> So, what I think is that yes, we should accept LZ4 and drop LZO from
> the kernel - the "fast but may not be small" compression title has
> clearly been taken by LZ4.
I have read the comments regarding how many compressors the kernel
should support and understand that it can not support all the
compressors available.
However, I don't think that LZO can be replaced by LZ4 in all the
cases. The benchmark above shows only about improved decompression
speed.
Thanks,
Kyungsik
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 10:20 ` Johannes Stezenbach
@ 2013-02-27 15:35 ` Nicolas Pitre
-1 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-27 15:35 UTC (permalink / raw)
To: Johannes Stezenbach
Cc: Russell King - ARM Linux, Kyungsik Lee, Andrew Morton,
Markus F.X.J. Oberhumer, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nitin Gupta, Richard Purdie,
Josh Triplett, Joe Millenbach, David Sterba, Richard Cochran,
Albin Tonnerre, Egon Alter, hyojun.im, chan.jeong,
raphael.andy.lee
On Wed, 27 Feb 2013, Johannes Stezenbach wrote:
> On Wed, Feb 27, 2013 at 09:51:39AM +0000, Russell King - ARM Linux wrote:
> > On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:
> > > Compiler: Linaro ARM gcc 4.6.2
> > > 2. ARMv7, 1.7GHz based board
> > > Kernel: linux 3.7
> > > Uncompressed Kernel Size: 14MB
> > > Compressed Size Decompression Speed
> > > LZO 6.0MB 34.1MB/s Old
> > > ----------------------------------------
> > > 6.0MB 34.7MB/s New
> > > 6.0MB 52.2MB/s(UA)
> > > =============================================
> > > LZ4 6.5MB 86.7MB/s
> > > UA: Unaligned memory Access support
> >
> > That is pretty conclusive - it shows an 8% increase in image size vs a
> > 66% increase in decompression speed. It will take a _lot_ to offset
> > that increase in decompression speed.
> >
> > So, what I think is that yes, we should accept LZ4 and drop LZO from
> > the kernel - the "fast but may not be small" compression title has
> > clearly been taken by LZ4.
>
> I think LZO may be used by squashfs, jffs2 and btrfs, thus you
> cannot drop it without breaking on disk storage formats.
It is not about dropping LZO from the kernel entirely. It's about
removing support for compressing zImage using LZO (and some others).
There is no compatibility issue as zImage embeds its own decompression
code.
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 15:35 ` Nicolas Pitre
0 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-27 15:35 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, 27 Feb 2013, Johannes Stezenbach wrote:
> On Wed, Feb 27, 2013 at 09:51:39AM +0000, Russell King - ARM Linux wrote:
> > On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:
> > > Compiler: Linaro ARM gcc 4.6.2
> > > 2. ARMv7, 1.7GHz based board
> > > Kernel: linux 3.7
> > > Uncompressed Kernel Size: 14MB
> > > Compressed Size Decompression Speed
> > > LZO 6.0MB 34.1MB/s Old
> > > ----------------------------------------
> > > 6.0MB 34.7MB/s New
> > > 6.0MB 52.2MB/s(UA)
> > > =============================================
> > > LZ4 6.5MB 86.7MB/s
> > > UA: Unaligned memory Access support
> >
> > That is pretty conclusive - it shows an 8% increase in image size vs a
> > 66% increase in decompression speed. It will take a _lot_ to offset
> > that increase in decompression speed.
> >
> > So, what I think is that yes, we should accept LZ4 and drop LZO from
> > the kernel - the "fast but may not be small" compression title has
> > clearly been taken by LZ4.
>
> I think LZO may be used by squashfs, jffs2 and btrfs, thus you
> cannot drop it without breaking on disk storage formats.
It is not about dropping LZO from the kernel entirely. It's about
removing support for compressing zImage using LZO (and some others).
There is no compatibility issue as zImage embeds its own decompression
code.
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 9:56 ` Russell King - ARM Linux
@ 2013-02-27 15:49 ` Joe Perches
-1 siblings, 0 replies; 67+ messages in thread
From: Joe Perches @ 2013-02-27 15:49 UTC (permalink / raw)
To: Russell King - ARM Linux
Cc: Peter Korsgaard, Nicolas Pitre, Markus F.X.J. Oberhumer,
Kyungsik Lee, Andrew Morton, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nitin Gupta, Richard Purdie,
Josh Triplett, Joe Millenbach, David Sterba, Richard Cochran,
Albin Tonnerre, Egon Alter, hyojun.im, chan.jeong,
raphael.andy.lee
On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > So... for a selected kernel version of a particular size, can we please
> > > have a comparison between the new LZO code and this LZ4 code, so that
> > > we can see whether it's worth updating the LZO code or replacing the
> > > LZO code with LZ4?
> >
> > How could it be questionable that it's worth updating the LZO code?
>
> Please read the comments against the previous posting of these patches
> where I first stated this argument - and with agreement from those
> following the thread. The thread started on 26 Jan 2013. Thanks.
https://lkml.org/lkml/2013/1/29/145
I did not and do not see significant value in
adding LZ4 given Markus' LZO improvements.
I asked about LZO.
Why would the LZO code not be updated?
The new LZO code is faster than ever and it's
a standalone improvement.
Markus has posted what seems a clean git pull
request. It was not cc'd to arm or linux-arch.
http://linux-kernel.2935.n7.nabble.com/GIT-PULL-Update-LZO-compression-code-for-v3-9-td605184.html
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 15:49 ` Joe Perches
0 siblings, 0 replies; 67+ messages in thread
From: Joe Perches @ 2013-02-27 15:49 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > So... for a selected kernel version of a particular size, can we please
> > > have a comparison between the new LZO code and this LZ4 code, so that
> > > we can see whether it's worth updating the LZO code or replacing the
> > > LZO code with LZ4?
> >
> > How could it be questionable that it's worth updating the LZO code?
>
> Please read the comments against the previous posting of these patches
> where I first stated this argument - and with agreement from those
> following the thread. The thread started on 26 Jan 2013. Thanks.
https://lkml.org/lkml/2013/1/29/145
I did not and do not see significant value in
adding LZ4 given Markus' LZO improvements.
I asked about LZO.
Why would the LZO code not be updated?
The new LZO code is faster than ever and it's
a standalone improvement.
Markus has posted what seems a clean git pull
request. It was not cc'd to arm or linux-arch.
http://linux-kernel.2935.n7.nabble.com/GIT-PULL-Update-LZO-compression-code-for-v3-9-td605184.html
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 15:49 ` Joe Perches
(?)
@ 2013-02-27 16:08 ` Nicolas Pitre
-1 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-27 16:08 UTC (permalink / raw)
To: Joe Perches
Cc: Russell King - ARM Linux, Peter Korsgaard,
Markus F.X.J. Oberhumer, Kyungsik Lee, Andrew Morton,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Michal Marek,
linux-arm-kernel, linux-kernel, linux-kbuild, x86, celinux-dev,
Nitin Gupta, Richard Purdie, Josh Triplett, Joe Millenbach,
David Sterba, Richard Cochran, Albin Tonnerre, Egon Alter,
hyojun.im, chan.jeong, raphael.andy.lee
On Wed, 27 Feb 2013, Joe Perches wrote:
> https://lkml.org/lkml/2013/1/29/145
Connecting to lkml.org (lkml.org)|87.253.128.182|:443... connected.
HTTP request sent, awaiting response... 500 Server Error
> I did not and do not see significant value in
> adding LZ4 given Markus' LZO improvements.
Please someone post a comprehensive comparison with all the results in
the same email.
> The new LZO code is faster than ever and it's
> a standalone improvement.
>
> Why would the LZO code not be updated?
It is used by filesystems, etc. So of course it needs to be updated to
faster code.
> Markus has posted what seems a clean git pull
> request. It was not cc'd to arm or linux-arch.
>
> http://linux-kernel.2935.n7.nabble.com/GIT-PULL-Update-LZO-compression-code-for-v3-9-td605184.html
Maybe a reminder should be sent to Linus about this.
>From the above we can see this:
**LZO-2013-UA : updated LZO version available in linux-next plus
experimental ARM Unaligned Access patch. This needs approval from
some ARM maintainer ist NOT YET INCLUDED.
What is that experimental patch in need of approval?
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 16:08 ` Nicolas Pitre
0 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-27 16:08 UTC (permalink / raw)
To: Joe Perches
Cc: Russell King - ARM Linux, Peter Korsgaard,
Markus F.X.J. Oberhumer, Kyungsik Lee, Andrew Morton,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Michal Marek,
linux-arm-kernel, linux-kernel, linux-kbuild, x86, celinux-dev,
Nitin Gupta, Richard Purdie, Josh Triplett, Joe Millenbach,
David Sterba, Richard Cochran, Albin Tonnerre, Egon Alter,
hyojun.im, chan.jeong, raphael.andy.lee
On Wed, 27 Feb 2013, Joe Perches wrote:
> https://lkml.org/lkml/2013/1/29/145
Connecting to lkml.org (lkml.org)|87.253.128.182|:443... connected.
HTTP request sent, awaiting response... 500 Server Error
> I did not and do not see significant value in
> adding LZ4 given Markus' LZO improvements.
Please someone post a comprehensive comparison with all the results in
the same email.
> The new LZO code is faster than ever and it's
> a standalone improvement.
>
> Why would the LZO code not be updated?
It is used by filesystems, etc. So of course it needs to be updated to
faster code.
> Markus has posted what seems a clean git pull
> request. It was not cc'd to arm or linux-arch.
>
> http://linux-kernel.2935.n7.nabble.com/GIT-PULL-Update-LZO-compression-code-for-v3-9-td605184.html
Maybe a reminder should be sent to Linus about this.
From the above we can see this:
**LZO-2013-UA : updated LZO version available in linux-next plus
experimental ARM Unaligned Access patch. This needs approval from
some ARM maintainer ist NOT YET INCLUDED.
What is that experimental patch in need of approval?
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 16:08 ` Nicolas Pitre
0 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-27 16:08 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, 27 Feb 2013, Joe Perches wrote:
> https://lkml.org/lkml/2013/1/29/145
Connecting to lkml.org (lkml.org)|87.253.128.182|:443... connected.
HTTP request sent, awaiting response... 500 Server Error
> I did not and do not see significant value in
> adding LZ4 given Markus' LZO improvements.
Please someone post a comprehensive comparison with all the results in
the same email.
> The new LZO code is faster than ever and it's
> a standalone improvement.
>
> Why would the LZO code not be updated?
It is used by filesystems, etc. So of course it needs to be updated to
faster code.
> Markus has posted what seems a clean git pull
> request. It was not cc'd to arm or linux-arch.
>
> http://linux-kernel.2935.n7.nabble.com/GIT-PULL-Update-LZO-compression-code-for-v3-9-td605184.html
Maybe a reminder should be sent to Linus about this.
>From the above we can see this:
**LZO-2013-UA : updated LZO version available in linux-next plus
experimental ARM Unaligned Access patch. This needs approval from
some ARM maintainer ist NOT YET INCLUDED.
What is that experimental patch in need of approval?
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 15:49 ` Joe Perches
@ 2013-02-27 16:31 ` Russell King - ARM Linux
-1 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-27 16:31 UTC (permalink / raw)
To: Joe Perches
Cc: Peter Korsgaard, Nicolas Pitre, Markus F.X.J. Oberhumer,
Kyungsik Lee, Andrew Morton, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nitin Gupta, Richard Purdie,
Josh Triplett, Joe Millenbach, David Sterba, Richard Cochran,
Albin Tonnerre, Egon Alter, hyojun.im, chan.jeong,
raphael.andy.lee
On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > So... for a selected kernel version of a particular size, can we please
> > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > we can see whether it's worth updating the LZO code or replacing the
> > > > LZO code with LZ4?
> > >
> > > How could it be questionable that it's worth updating the LZO code?
> >
> > Please read the comments against the previous posting of these patches
> > where I first stated this argument - and with agreement from those
> > following the thread. The thread started on 26 Jan 2013. Thanks.
>
> https://lkml.org/lkml/2013/1/29/145
>
> I did not and do not see significant value in
> adding LZ4 given Markus' LZO improvements.
Sorry, a 66% increase in decompression speed over the updated LZO code
isn't "significant value" ?
I'm curious - what in your mind qualifies "significant value" ?
Maybe "significant value" is a patch which buggily involves converting
all those "<n>" printk format strings in assembly files to KERN_* macros,
thereby breaking those strings because you've not paid attention to what
.asciz means? (Yes, I've just cleaned that crap up after you...)
> Why would the LZO code not be updated?
I'm not saying that the LZO code should not be updated. I'm saying that
the kernel boot time decompressor is not a play ground for an ever
increasing number of "my favourite compression method" crap. We don't
need four, five or even six compression methods there. We just need
three - a "fast but large", "small but slow" and "all round popular
medium".
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 16:31 ` Russell King - ARM Linux
0 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-27 16:31 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > So... for a selected kernel version of a particular size, can we please
> > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > we can see whether it's worth updating the LZO code or replacing the
> > > > LZO code with LZ4?
> > >
> > > How could it be questionable that it's worth updating the LZO code?
> >
> > Please read the comments against the previous posting of these patches
> > where I first stated this argument - and with agreement from those
> > following the thread. The thread started on 26 Jan 2013. Thanks.
>
> https://lkml.org/lkml/2013/1/29/145
>
> I did not and do not see significant value in
> adding LZ4 given Markus' LZO improvements.
Sorry, a 66% increase in decompression speed over the updated LZO code
isn't "significant value" ?
I'm curious - what in your mind qualifies "significant value" ?
Maybe "significant value" is a patch which buggily involves converting
all those "<n>" printk format strings in assembly files to KERN_* macros,
thereby breaking those strings because you've not paid attention to what
.asciz means? (Yes, I've just cleaned that crap up after you...)
> Why would the LZO code not be updated?
I'm not saying that the LZO code should not be updated. I'm saying that
the kernel boot time decompressor is not a play ground for an ever
increasing number of "my favourite compression method" crap. We don't
need four, five or even six compression methods there. We just need
three - a "fast but large", "small but slow" and "all round popular
medium".
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 16:31 ` Russell King - ARM Linux
@ 2013-02-27 16:53 ` Borislav Petkov
-1 siblings, 0 replies; 67+ messages in thread
From: Borislav Petkov @ 2013-02-27 16:53 UTC (permalink / raw)
To: Russell King - ARM Linux
Cc: Joe Perches, Peter Korsgaard, Nicolas Pitre,
Markus F.X.J. Oberhumer, Kyungsik Lee, Andrew Morton,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Michal Marek,
linux-arm-kernel, linux-kernel, linux-kbuild, x86, celinux-dev,
Nitin Gupta, Richard Purdie, Josh Triplett, Joe Millenbach,
David Sterba, Richard Cochran, Albin Tonnerre, Egon Alter,
hyojun.im, chan.jeong, raphael.andy.lee
On Wed, Feb 27, 2013 at 04:31:18PM +0000, Russell King - ARM Linux wrote:
> I'm not saying that the LZO code should not be updated. I'm saying
> that the kernel boot time decompressor is not a play ground for an
> ever increasing number of "my favourite compression method" crap.
> We don't need four, five or even six compression methods there. We
> just need three - a "fast but large", "small but slow" and "all round
> popular medium".
Hell yeah!
--
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 16:53 ` Borislav Petkov
0 siblings, 0 replies; 67+ messages in thread
From: Borislav Petkov @ 2013-02-27 16:53 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, Feb 27, 2013 at 04:31:18PM +0000, Russell King - ARM Linux wrote:
> I'm not saying that the LZO code should not be updated. I'm saying
> that the kernel boot time decompressor is not a play ground for an
> ever increasing number of "my favourite compression method" crap.
> We don't need four, five or even six compression methods there. We
> just need three - a "fast but large", "small but slow" and "all round
> popular medium".
Hell yeah!
--
Regards/Gruss,
Boris.
Sent from a fat crate under my desk. Formatting is fine.
--
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 16:31 ` Russell King - ARM Linux
@ 2013-02-27 17:04 ` Joe Perches
-1 siblings, 0 replies; 67+ messages in thread
From: Joe Perches @ 2013-02-27 17:04 UTC (permalink / raw)
To: Russell King - ARM Linux
Cc: Peter Korsgaard, Nicolas Pitre, Markus F.X.J. Oberhumer,
Kyungsik Lee, Andrew Morton, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nitin Gupta, Richard Purdie,
Josh Triplett, Joe Millenbach, David Sterba, Richard Cochran,
Albin Tonnerre, Egon Alter, hyojun.im, chan.jeong,
raphael.andy.lee
On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:
> On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> > On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > > So... for a selected kernel version of a particular size, can we please
> > > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > > we can see whether it's worth updating the LZO code or replacing the
> > > > > LZO code with LZ4?
> > > >
> > > > How could it be questionable that it's worth updating the LZO code?
> > >
> > > Please read the comments against the previous posting of these patches
> > > where I first stated this argument - and with agreement from those
> > > following the thread. The thread started on 26 Jan 2013. Thanks.
> >
> > https://lkml.org/lkml/2013/1/29/145
> >
> > I did not and do not see significant value in
> > adding LZ4 given Markus' LZO improvements.
>
> Sorry, a 66% increase in decompression speed over the updated LZO code
> isn't "significant value" ?
We disagree.
> I'm curious - what in your mind qualifies "significant value" ?
faster boot time. smaller, faster overall code.
> Maybe "significant value" is a patch which buggily involves converting
> all those "<n>" printk format strings in assembly files to KERN_* macros,
> thereby breaking those strings because you've not paid attention to what
> .asciz means? (Yes, I've just cleaned that crap up after you...)
If you mean commit 0cc41e4a21d43, perhaps you could clarify with an
example. I don't see any relevant changes by you in -next, but
maybe I'm not looking in the right spot.
The change did enable reducing code size.
> > Why would the LZO code not be updated?
> I'm not saying that the LZO code should not be updated.
You said:
> > > > > so that we can see whether it's worth updating the LZO code
Sounded as if you were doubtful to me.
> I'm saying that
> the kernel boot time decompressor is not a play ground for an ever
> increasing number of "my favourite compression method" crap.
Completely agree.
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 17:04 ` Joe Perches
0 siblings, 0 replies; 67+ messages in thread
From: Joe Perches @ 2013-02-27 17:04 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:
> On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> > On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > > So... for a selected kernel version of a particular size, can we please
> > > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > > we can see whether it's worth updating the LZO code or replacing the
> > > > > LZO code with LZ4?
> > > >
> > > > How could it be questionable that it's worth updating the LZO code?
> > >
> > > Please read the comments against the previous posting of these patches
> > > where I first stated this argument - and with agreement from those
> > > following the thread. The thread started on 26 Jan 2013. Thanks.
> >
> > https://lkml.org/lkml/2013/1/29/145
> >
> > I did not and do not see significant value in
> > adding LZ4 given Markus' LZO improvements.
>
> Sorry, a 66% increase in decompression speed over the updated LZO code
> isn't "significant value" ?
We disagree.
> I'm curious - what in your mind qualifies "significant value" ?
faster boot time. smaller, faster overall code.
> Maybe "significant value" is a patch which buggily involves converting
> all those "<n>" printk format strings in assembly files to KERN_* macros,
> thereby breaking those strings because you've not paid attention to what
> .asciz means? (Yes, I've just cleaned that crap up after you...)
If you mean commit 0cc41e4a21d43, perhaps you could clarify with an
example. I don't see any relevant changes by you in -next, but
maybe I'm not looking in the right spot.
The change did enable reducing code size.
> > Why would the LZO code not be updated?
> I'm not saying that the LZO code should not be updated.
You said:
> > > > > so that we can see whether it's worth updating the LZO code
Sounded as if you were doubtful to me.
> I'm saying that
> the kernel boot time decompressor is not a play ground for an ever
> increasing number of "my favourite compression method" crap.
Completely agree.
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 17:04 ` Joe Perches
@ 2013-02-27 17:16 ` Nicolas Pitre
-1 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-27 17:16 UTC (permalink / raw)
To: Joe Perches
Cc: Russell King - ARM Linux, Peter Korsgaard,
Markus F.X.J. Oberhumer, Kyungsik Lee, Andrew Morton,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Michal Marek,
linux-arm-kernel, linux-kernel, linux-kbuild, x86, celinux-dev,
Nitin Gupta, Richard Purdie, Josh Triplett, Joe Millenbach,
David Sterba, Richard Cochran, Albin Tonnerre, Egon Alter,
hyojun.im, chan.jeong, raphael.andy.lee
On Wed, 27 Feb 2013, Joe Perches wrote:
> On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:
> > On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> > > On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > > > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > > > So... for a selected kernel version of a particular size, can we please
> > > > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > > > we can see whether it's worth updating the LZO code or replacing the
> > > > > > LZO code with LZ4?
> > > > >
> > > > > How could it be questionable that it's worth updating the LZO code?
> > > >
> > > > Please read the comments against the previous posting of these patches
> > > > where I first stated this argument - and with agreement from those
> > > > following the thread. The thread started on 26 Jan 2013. Thanks.
> > >
> > > https://lkml.org/lkml/2013/1/29/145
> > >
> > > I did not and do not see significant value in
> > > adding LZ4 given Markus' LZO improvements.
> >
> > Sorry, a 66% increase in decompression speed over the updated LZO code
> > isn't "significant value" ?
>
> We disagree.
>
> > I'm curious - what in your mind qualifies "significant value" ?
>
> faster boot time. smaller, faster overall code.
Sorry, but you certainly successfully got me confused, and probably
others as well.
RMK says that "66% increase in decompression speed over LZO" is
significant. You apparently disagree with that.
Then you say that faster boot time is significant.
Again, can you (or anyone else) provide comprehensive test results in a
single email with both compression methods?
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 17:16 ` Nicolas Pitre
0 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-27 17:16 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, 27 Feb 2013, Joe Perches wrote:
> On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:
> > On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> > > On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > > > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > > > So... for a selected kernel version of a particular size, can we please
> > > > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > > > we can see whether it's worth updating the LZO code or replacing the
> > > > > > LZO code with LZ4?
> > > > >
> > > > > How could it be questionable that it's worth updating the LZO code?
> > > >
> > > > Please read the comments against the previous posting of these patches
> > > > where I first stated this argument - and with agreement from those
> > > > following the thread. The thread started on 26 Jan 2013. Thanks.
> > >
> > > https://lkml.org/lkml/2013/1/29/145
> > >
> > > I did not and do not see significant value in
> > > adding LZ4 given Markus' LZO improvements.
> >
> > Sorry, a 66% increase in decompression speed over the updated LZO code
> > isn't "significant value" ?
>
> We disagree.
>
> > I'm curious - what in your mind qualifies "significant value" ?
>
> faster boot time. smaller, faster overall code.
Sorry, but you certainly successfully got me confused, and probably
others as well.
RMK says that "66% increase in decompression speed over LZO" is
significant. You apparently disagree with that.
Then you say that faster boot time is significant.
Again, can you (or anyone else) provide comprehensive test results in a
single email with both compression methods?
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 17:04 ` Joe Perches
@ 2013-02-27 17:36 ` Russell King - ARM Linux
-1 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-27 17:36 UTC (permalink / raw)
To: Joe Perches
Cc: Peter Korsgaard, Nicolas Pitre, Markus F.X.J. Oberhumer,
Kyungsik Lee, Andrew Morton, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nitin Gupta, Richard Purdie,
Josh Triplett, Joe Millenbach, David Sterba, Richard Cochran,
Albin Tonnerre, Egon Alter, hyojun.im, chan.jeong,
raphael.andy.lee
On Wed, Feb 27, 2013 at 09:04:48AM -0800, Joe Perches wrote:
> On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:
> > On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> > > On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > > > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > > > So... for a selected kernel version of a particular size, can we please
> > > > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > > > we can see whether it's worth updating the LZO code or replacing the
> > > > > > LZO code with LZ4?
> > > > >
> > > > > How could it be questionable that it's worth updating the LZO code?
> > > >
> > > > Please read the comments against the previous posting of these patches
> > > > where I first stated this argument - and with agreement from those
> > > > following the thread. The thread started on 26 Jan 2013. Thanks.
> > >
> > > https://lkml.org/lkml/2013/1/29/145
> > >
> > > I did not and do not see significant value in
> > > adding LZ4 given Markus' LZO improvements.
> >
> > Sorry, a 66% increase in decompression speed over the updated LZO code
> > isn't "significant value" ?
>
> We disagree.
ROTFL.
> > I'm curious - what in your mind qualifies "significant value" ?
>
> faster boot time. smaller, faster overall code.
ROTFL again! Because you've just disagreed with your above statement.
"66% increase in decompression speed" as far as I know _is_ "faster
boot time" !
> > Maybe "significant value" is a patch which buggily involves converting
> > all those "<n>" printk format strings in assembly files to KERN_* macros,
> > thereby breaking those strings because you've not paid attention to what
> > .asciz means? (Yes, I've just cleaned that crap up after you...)
>
> If you mean commit 0cc41e4a21d43, perhaps you could clarify with an
> example. I don't see any relevant changes by you in -next, but
> maybe I'm not looking in the right spot.
While recently asking someone to enable VFP debugging, so I could help
sort out a problem they had reported, this is the debug output I was
greeted by thanks to your meddling:
[ 927.235546] \x01\x01\x01\x01\x01\x01\x01\x01
...
[ 927.241505] \x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01
Yes, really useful debug output isn't it? You can really see what's
going on there. These are coming from ultimately two commits - the
one you refer to above, which on its own would've changed the printk
string to be merely "<7>" - and the follow on commit changing the
way printk levels are dealt with.
The above output is produced by:
#define KERN_SOH "\001" /* ASCII Start Of Header */
#define KERN_DEBUG KERN_SOH "7" /* debug-level messages */
.asciz KERN_DEBUG "VFP: \str\n"
7.6 `.asciz "STRING"'...
========================
`.asciz' is just like `.ascii', but each string is followed by a zero
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
byte. The "z" in `.asciz' stands for "zero".
^^^^^
0000 01003700 5646503a 20696e73 74722025 ..7.VFP: instr %
^^ ^^
0010 30387820 70632025 30387820 73746174 08x pc %08x stat
0020 65202570 0a000100 37005646 503a2066 e %p....7.VFP: f
^^
...
That is: \x01 \x00 7 \x00 VFP: instr %08x pc %08x state %p \x00
See - three separately terminated strings because you changed:
.asciz "<7>VFP: \str\n"
to:
.asciz "<7>" "VFP: \str\n"
which turned it into _two_ separately NUL-terminated strings, and then
the follow-on changes to printk kern levels changed this to:
.asciz "\001" "7" "VFP: \str\n"
producing _three_ separately NUL-terminated strings.
The commit is not in mainline, nor linux-next, but in my tree as of
yesterday (e36815e2e), ready to be pushed out when I've finished working
on fixing other problems with VFP - or when I decide to push it out ready
for submission during this merge window.
> The change did enable reducing code size.
??? Yea, right, meanwhile breaking the ability of stuff to produce
kernel messages.
> > > Why would the LZO code not be updated?
> > I'm not saying that the LZO code should not be updated.
>
> You said:
>
> > > > > > so that we can see whether it's worth updating the LZO code
>
> Sounded as if you were doubtful to me.
_In_ the decompressor. We're talking about the _decompressor_ in
this thread.
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 17:36 ` Russell King - ARM Linux
0 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-27 17:36 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, Feb 27, 2013 at 09:04:48AM -0800, Joe Perches wrote:
> On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:
> > On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> > > On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > > > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > > > So... for a selected kernel version of a particular size, can we please
> > > > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > > > we can see whether it's worth updating the LZO code or replacing the
> > > > > > LZO code with LZ4?
> > > > >
> > > > > How could it be questionable that it's worth updating the LZO code?
> > > >
> > > > Please read the comments against the previous posting of these patches
> > > > where I first stated this argument - and with agreement from those
> > > > following the thread. The thread started on 26 Jan 2013. Thanks.
> > >
> > > https://lkml.org/lkml/2013/1/29/145
> > >
> > > I did not and do not see significant value in
> > > adding LZ4 given Markus' LZO improvements.
> >
> > Sorry, a 66% increase in decompression speed over the updated LZO code
> > isn't "significant value" ?
>
> We disagree.
ROTFL.
> > I'm curious - what in your mind qualifies "significant value" ?
>
> faster boot time. smaller, faster overall code.
ROTFL again! Because you've just disagreed with your above statement.
"66% increase in decompression speed" as far as I know _is_ "faster
boot time" !
> > Maybe "significant value" is a patch which buggily involves converting
> > all those "<n>" printk format strings in assembly files to KERN_* macros,
> > thereby breaking those strings because you've not paid attention to what
> > .asciz means? (Yes, I've just cleaned that crap up after you...)
>
> If you mean commit 0cc41e4a21d43, perhaps you could clarify with an
> example. I don't see any relevant changes by you in -next, but
> maybe I'm not looking in the right spot.
While recently asking someone to enable VFP debugging, so I could help
sort out a problem they had reported, this is the debug output I was
greeted by thanks to your meddling:
[ 927.235546] \x01\x01\x01\x01\x01\x01\x01\x01
...
[ 927.241505] \x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01
Yes, really useful debug output isn't it? You can really see what's
going on there. These are coming from ultimately two commits - the
one you refer to above, which on its own would've changed the printk
string to be merely "<7>" - and the follow on commit changing the
way printk levels are dealt with.
The above output is produced by:
#define KERN_SOH "\001" /* ASCII Start Of Header */
#define KERN_DEBUG KERN_SOH "7" /* debug-level messages */
.asciz KERN_DEBUG "VFP: \str\n"
7.6 `.asciz "STRING"'...
========================
`.asciz' is just like `.ascii', but each string is followed by a zero
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
byte. The "z" in `.asciz' stands for "zero".
^^^^^
0000 01003700 5646503a 20696e73 74722025 ..7.VFP: instr %
^^ ^^
0010 30387820 70632025 30387820 73746174 08x pc %08x stat
0020 65202570 0a000100 37005646 503a2066 e %p....7.VFP: f
^^
...
That is: \x01 \x00 7 \x00 VFP: instr %08x pc %08x state %p \x00
See - three separately terminated strings because you changed:
.asciz "<7>VFP: \str\n"
to:
.asciz "<7>" "VFP: \str\n"
which turned it into _two_ separately NUL-terminated strings, and then
the follow-on changes to printk kern levels changed this to:
.asciz "\001" "7" "VFP: \str\n"
producing _three_ separately NUL-terminated strings.
The commit is not in mainline, nor linux-next, but in my tree as of
yesterday (e36815e2e), ready to be pushed out when I've finished working
on fixing other problems with VFP - or when I decide to push it out ready
for submission during this merge window.
> The change did enable reducing code size.
??? Yea, right, meanwhile breaking the ability of stuff to produce
kernel messages.
> > > Why would the LZO code not be updated?
> > I'm not saying that the LZO code should not be updated.
>
> You said:
>
> > > > > > so that we can see whether it's worth updating the LZO code
>
> Sounded as if you were doubtful to me.
_In_ the decompressor. We're talking about the _decompressor_ in
this thread.
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 17:16 ` Nicolas Pitre
@ 2013-02-27 17:39 ` Joe Perches
-1 siblings, 0 replies; 67+ messages in thread
From: Joe Perches @ 2013-02-27 17:39 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Russell King - ARM Linux, Peter Korsgaard,
Markus F.X.J. Oberhumer, Kyungsik Lee, Andrew Morton,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Michal Marek,
linux-arm-kernel, linux-kernel, linux-kbuild, x86, celinux-dev,
Nitin Gupta, Richard Purdie, Josh Triplett, Joe Millenbach,
David Sterba, Richard Cochran, Albin Tonnerre, Egon Alter,
hyojun.im, chan.jeong, raphael.andy.lee
On Wed, 2013-02-27 at 12:16 -0500, Nicolas Pitre wrote:
> On Wed, 27 Feb 2013, Joe Perches wrote:
> > On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:
> > > On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> > > > On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > > > > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > > > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > > > > So... for a selected kernel version of a particular size, can we please
> > > > > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > > > > we can see whether it's worth updating the LZO code or replacing the
> > > > > > > LZO code with LZ4?
> > > > > >
> > > > > > How could it be questionable that it's worth updating the LZO code?
> > > > >
> > > > > Please read the comments against the previous posting of these patches
> > > > > where I first stated this argument - and with agreement from those
> > > > > following the thread. The thread started on 26 Jan 2013. Thanks.
> > > >
> > > > https://lkml.org/lkml/2013/1/29/145
> > > >
> > > > I did not and do not see significant value in
> > > > adding LZ4 given Markus' LZO improvements.
> > >
> > > Sorry, a 66% increase in decompression speed over the updated LZO code
> > > isn't "significant value" ?
> >
> > We disagree.
> >
> > > I'm curious - what in your mind qualifies "significant value" ?
> >
> > faster boot time. smaller, faster overall code.
>
> Sorry, but you certainly successfully got me confused, and probably
> others as well.
>
> RMK says that "66% increase in decompression speed over LZO" is
> significant. You apparently disagree with that.
Yeah, I can see how that can be interpreted.
I'm referring only to the new LZO.
I guess Russell has not reviewed the new LZO.
There is apparently no speed increase for LZ4 over
the new LZO.
I believe Markus has shown comparison testing in
this very thread.
https://patchwork.kernel.org/patch/2187441/
> Then you say that faster boot time is significant.
Increasing speed in incumbent code without adding
defects is always useful no?. Replacing incumbent
code with new code should be debated for utility.
I still think there's not much value in adding LZ4.
LZ4 is not not faster than LZO, it's just more code.
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 17:39 ` Joe Perches
0 siblings, 0 replies; 67+ messages in thread
From: Joe Perches @ 2013-02-27 17:39 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, 2013-02-27 at 12:16 -0500, Nicolas Pitre wrote:
> On Wed, 27 Feb 2013, Joe Perches wrote:
> > On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:
> > > On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> > > > On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > > > > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > > > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > > > > So... for a selected kernel version of a particular size, can we please
> > > > > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > > > > we can see whether it's worth updating the LZO code or replacing the
> > > > > > > LZO code with LZ4?
> > > > > >
> > > > > > How could it be questionable that it's worth updating the LZO code?
> > > > >
> > > > > Please read the comments against the previous posting of these patches
> > > > > where I first stated this argument - and with agreement from those
> > > > > following the thread. The thread started on 26 Jan 2013. Thanks.
> > > >
> > > > https://lkml.org/lkml/2013/1/29/145
> > > >
> > > > I did not and do not see significant value in
> > > > adding LZ4 given Markus' LZO improvements.
> > >
> > > Sorry, a 66% increase in decompression speed over the updated LZO code
> > > isn't "significant value" ?
> >
> > We disagree.
> >
> > > I'm curious - what in your mind qualifies "significant value" ?
> >
> > faster boot time. smaller, faster overall code.
>
> Sorry, but you certainly successfully got me confused, and probably
> others as well.
>
> RMK says that "66% increase in decompression speed over LZO" is
> significant. You apparently disagree with that.
Yeah, I can see how that can be interpreted.
I'm referring only to the new LZO.
I guess Russell has not reviewed the new LZO.
There is apparently no speed increase for LZ4 over
the new LZO.
I believe Markus has shown comparison testing in
this very thread.
https://patchwork.kernel.org/patch/2187441/
> Then you say that faster boot time is significant.
Increasing speed in incumbent code without adding
defects is always useful no?. Replacing incumbent
code with new code should be debated for utility.
I still think there's not much value in adding LZ4.
LZ4 is not not faster than LZO, it's just more code.
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 17:39 ` Joe Perches
@ 2013-02-27 17:52 ` Nicolas Pitre
-1 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-27 17:52 UTC (permalink / raw)
To: Joe Perches
Cc: Russell King - ARM Linux, Peter Korsgaard,
Markus F.X.J. Oberhumer, Kyungsik Lee, Andrew Morton,
Thomas Gleixner, Ingo Molnar, H. Peter Anvin, Michal Marek,
linux-arm-kernel, linux-kernel, linux-kbuild, x86, celinux-dev,
Nitin Gupta, Richard Purdie, Josh Triplett, Joe Millenbach,
David Sterba, Richard Cochran, Albin Tonnerre, Egon Alter,
hyojun.im, chan.jeong, raphael.andy.lee
On Wed, 27 Feb 2013, Joe Perches wrote:
> On Wed, 2013-02-27 at 12:16 -0500, Nicolas Pitre wrote:
> > RMK says that "66% increase in decompression speed over LZO" is
> > significant. You apparently disagree with that.
>
> Yeah, I can see how that can be interpreted.
> I'm referring only to the new LZO.
>
> I guess Russell has not reviewed the new LZO.
>
> There is apparently no speed increase for LZ4 over
> the new LZO.
>
> I believe Markus has shown comparison testing in
> this very thread.
>
> https://patchwork.kernel.org/patch/2187441/
Right.
Can the new LZO code be merged by Linus now? It has been sitting in
linux-next for quite some time. Afterwards we could revisit lz4
worthiness without all the present confusion.
BTW, I still wonder what that patch requiring ARM people approval is.
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 17:52 ` Nicolas Pitre
0 siblings, 0 replies; 67+ messages in thread
From: Nicolas Pitre @ 2013-02-27 17:52 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, 27 Feb 2013, Joe Perches wrote:
> On Wed, 2013-02-27 at 12:16 -0500, Nicolas Pitre wrote:
> > RMK says that "66% increase in decompression speed over LZO" is
> > significant. You apparently disagree with that.
>
> Yeah, I can see how that can be interpreted.
> I'm referring only to the new LZO.
>
> I guess Russell has not reviewed the new LZO.
>
> There is apparently no speed increase for LZ4 over
> the new LZO.
>
> I believe Markus has shown comparison testing in
> this very thread.
>
> https://patchwork.kernel.org/patch/2187441/
Right.
Can the new LZO code be merged by Linus now? It has been sitting in
linux-next for quite some time. Afterwards we could revisit lz4
worthiness without all the present confusion.
BTW, I still wonder what that patch requiring ARM people approval is.
Nicolas
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 17:39 ` Joe Perches
@ 2013-02-27 17:57 ` Russell King - ARM Linux
-1 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-27 17:57 UTC (permalink / raw)
To: Joe Perches
Cc: Nicolas Pitre, Peter Korsgaard, Markus F.X.J. Oberhumer,
Kyungsik Lee, Andrew Morton, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nitin Gupta, Richard Purdie,
Josh Triplett, Joe Millenbach, David Sterba, Richard Cochran,
Albin Tonnerre, Egon Alter, hyojun.im, chan.jeong,
raphael.andy.lee
On Wed, Feb 27, 2013 at 09:39:47AM -0800, Joe Perches wrote:
> On Wed, 2013-02-27 at 12:16 -0500, Nicolas Pitre wrote:
> > On Wed, 27 Feb 2013, Joe Perches wrote:
> > > On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:
> > > > On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> > > > > On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > > > > > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > > > > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > > > > > So... for a selected kernel version of a particular size, can we please
> > > > > > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > > > > > we can see whether it's worth updating the LZO code or replacing the
> > > > > > > > LZO code with LZ4?
> > > > > > >
> > > > > > > How could it be questionable that it's worth updating the LZO code?
> > > > > >
> > > > > > Please read the comments against the previous posting of these patches
> > > > > > where I first stated this argument - and with agreement from those
> > > > > > following the thread. The thread started on 26 Jan 2013. Thanks.
> > > > >
> > > > > https://lkml.org/lkml/2013/1/29/145
> > > > >
> > > > > I did not and do not see significant value in
> > > > > adding LZ4 given Markus' LZO improvements.
> > > >
> > > > Sorry, a 66% increase in decompression speed over the updated LZO code
> > > > isn't "significant value" ?
> > >
> > > We disagree.
> > >
> > > > I'm curious - what in your mind qualifies "significant value" ?
> > >
> > > faster boot time. smaller, faster overall code.
> >
> > Sorry, but you certainly successfully got me confused, and probably
> > others as well.
> >
> > RMK says that "66% increase in decompression speed over LZO" is
> > significant. You apparently disagree with that.
>
> Yeah, I can see how that can be interpreted.
> I'm referring only to the new LZO.
>
> I guess Russell has not reviewed the new LZO.
>
> There is apparently no speed increase for LZ4 over
> the new LZO.
Total claptrap. I've no idea where you're getting your data from, but
it's franky wrong and you're now being totally misleading to anyone
else reading this thread.
I explicitly asked for a comparison of the _new_ LZO vs the LZ4 code,
and this is what I received from Kyungsik Lee in this thread:
Compiler: Linaro ARM gcc 4.6.2
2. ARMv7, 1.7GHz based board
Kernel: linux 3.7
Uncompressed Kernel Size: 14MB
Compressed Size Decompression Speed
LZO 6.0MB 34.1MB/s Old
----------------------------------------
6.0MB 34.7MB/s New
6.0MB 52.2MB/s(UA)
=============================================
LZ4 6.5MB 86.7MB/s
UA: Unaligned memory Access support
And my statement of a "66% increase in speed" of LZ4 is comparing the
_new_ LZO code with unaligned access with the LZ4 code.
Now, you refer to Markus' results - but Markus' results do not say what
they're comparing - they don't say what the size of the compressed image
is, nor what the size of the uncompressed image was.
Now, Markus' results show a 42% increase in speed between the LZO-2012
and LZO-2013-UA versions (do the calculation yourself - I'm sure you're
capable of that? If not, we can turn this into a maths lesson too).
The above shows a 53% increase in speed between the existing LZO code
and the new LZO code with unaligned accesses.
_But_ the above shows an additional 66% increase between the new LZO
code with unaligned accesses and LZ4. Or, a whopping 150% increase
in speed over the _existing_ LZO code.
So please, stop stating what I have and have not reviewed. Unlike you,
I _have_ been following everything that's been said in this thread, and
- unlike you - I have analysed the figures put forward and drawn
conclusions which are fully supported by the published data from them,
and stated them - now many times.
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 17:57 ` Russell King - ARM Linux
0 siblings, 0 replies; 67+ messages in thread
From: Russell King - ARM Linux @ 2013-02-27 17:57 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, Feb 27, 2013 at 09:39:47AM -0800, Joe Perches wrote:
> On Wed, 2013-02-27 at 12:16 -0500, Nicolas Pitre wrote:
> > On Wed, 27 Feb 2013, Joe Perches wrote:
> > > On Wed, 2013-02-27 at 16:31 +0000, Russell King - ARM Linux wrote:
> > > > On Wed, Feb 27, 2013 at 07:49:12AM -0800, Joe Perches wrote:
> > > > > On Wed, 2013-02-27 at 09:56 +0000, Russell King - ARM Linux wrote:
> > > > > > On Tue, Feb 26, 2013 at 05:40:34PM -0800, Joe Perches wrote:
> > > > > > > On Tue, 2013-02-26 at 22:10 +0000, Russell King - ARM Linux wrote:
> > > > > > > > So... for a selected kernel version of a particular size, can we please
> > > > > > > > have a comparison between the new LZO code and this LZ4 code, so that
> > > > > > > > we can see whether it's worth updating the LZO code or replacing the
> > > > > > > > LZO code with LZ4?
> > > > > > >
> > > > > > > How could it be questionable that it's worth updating the LZO code?
> > > > > >
> > > > > > Please read the comments against the previous posting of these patches
> > > > > > where I first stated this argument - and with agreement from those
> > > > > > following the thread. The thread started on 26 Jan 2013. Thanks.
> > > > >
> > > > > https://lkml.org/lkml/2013/1/29/145
> > > > >
> > > > > I did not and do not see significant value in
> > > > > adding LZ4 given Markus' LZO improvements.
> > > >
> > > > Sorry, a 66% increase in decompression speed over the updated LZO code
> > > > isn't "significant value" ?
> > >
> > > We disagree.
> > >
> > > > I'm curious - what in your mind qualifies "significant value" ?
> > >
> > > faster boot time. smaller, faster overall code.
> >
> > Sorry, but you certainly successfully got me confused, and probably
> > others as well.
> >
> > RMK says that "66% increase in decompression speed over LZO" is
> > significant. You apparently disagree with that.
>
> Yeah, I can see how that can be interpreted.
> I'm referring only to the new LZO.
>
> I guess Russell has not reviewed the new LZO.
>
> There is apparently no speed increase for LZ4 over
> the new LZO.
Total claptrap. I've no idea where you're getting your data from, but
it's franky wrong and you're now being totally misleading to anyone
else reading this thread.
I explicitly asked for a comparison of the _new_ LZO vs the LZ4 code,
and this is what I received from Kyungsik Lee in this thread:
Compiler: Linaro ARM gcc 4.6.2
2. ARMv7, 1.7GHz based board
Kernel: linux 3.7
Uncompressed Kernel Size: 14MB
Compressed Size Decompression Speed
LZO 6.0MB 34.1MB/s Old
----------------------------------------
6.0MB 34.7MB/s New
6.0MB 52.2MB/s(UA)
=============================================
LZ4 6.5MB 86.7MB/s
UA: Unaligned memory Access support
And my statement of a "66% increase in speed" of LZ4 is comparing the
_new_ LZO code with unaligned access with the LZ4 code.
Now, you refer to Markus' results - but Markus' results do not say what
they're comparing - they don't say what the size of the compressed image
is, nor what the size of the uncompressed image was.
Now, Markus' results show a 42% increase in speed between the LZO-2012
and LZO-2013-UA versions (do the calculation yourself - I'm sure you're
capable of that? If not, we can turn this into a maths lesson too).
The above shows a 53% increase in speed between the existing LZO code
and the new LZO code with unaligned accesses.
_But_ the above shows an additional 66% increase between the new LZO
code with unaligned accesses and LZ4. Or, a whopping 150% increase
in speed over the _existing_ LZO code.
So please, stop stating what I have and have not reviewed. Unlike you,
I _have_ been following everything that's been said in this thread, and
- unlike you - I have analysed the figures put forward and drawn
conclusions which are fully supported by the published data from them,
and stated them - now many times.
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 9:51 ` Russell King - ARM Linux
@ 2013-02-27 22:21 ` Andrew Morton
-1 siblings, 0 replies; 67+ messages in thread
From: Andrew Morton @ 2013-02-27 22:21 UTC (permalink / raw)
To: Russell King - ARM Linux
Cc: Kyungsik Lee, Markus F.X.J. Oberhumer, Thomas Gleixner,
Ingo Molnar, H. Peter Anvin, Michal Marek, linux-arm-kernel,
linux-kernel, linux-kbuild, x86, celinux-dev, Nicolas Pitre,
Nitin Gupta, Richard Purdie, Josh Triplett, Joe Millenbach,
David Sterba, Richard Cochran, Albin Tonnerre, Egon Alter,
hyojun.im, chan.jeong, raphael.andy.lee
On Wed, 27 Feb 2013 09:51:39 +0000
Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:
> > Compiler: Linaro ARM gcc 4.6.2
> > 2. ARMv7, 1.7GHz based board
> > Kernel: linux 3.7
> > Uncompressed Kernel Size: 14MB
> > Compressed Size Decompression Speed
> > LZO 6.0MB 34.1MB/s Old
> > ----------------------------------------
> > 6.0MB 34.7MB/s New
> > 6.0MB 52.2MB/s(UA)
> > =============================================
> > LZ4 6.5MB 86.7MB/s
> > UA: Unaligned memory Access support
>
> That is pretty conclusive - it shows an 8% increase in image size vs a
> 66% increase in decompression speed. It will take a _lot_ to offset
> that increase in decompression speed.
>
> So, what I think is that yes, we should accept LZ4 and drop LZO from
> the kernel - the "fast but may not be small" compression title has
> clearly been taken by LZ4.
>
> Akpm - what's your thoughts?
It sounds like we should merge both.
I've sent Linus a little reminder for Markus's 3.9 pull request. Let's
get down and review and test this new code?
David's review comments were useful.
I'd like to also see a Kconfig patch which makes x86 and arm kernels
default to the new LZ4 code. Then I can sneak that patch into
linux-next so the new code will get some testing. If we don't do that,
very few people will run it.
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-27 22:21 ` Andrew Morton
0 siblings, 0 replies; 67+ messages in thread
From: Andrew Morton @ 2013-02-27 22:21 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, 27 Feb 2013 09:51:39 +0000
Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> On Wed, Feb 27, 2013 at 04:36:47PM +0900, Kyungsik Lee wrote:
> > Compiler: Linaro ARM gcc 4.6.2
> > 2. ARMv7, 1.7GHz based board
> > Kernel: linux 3.7
> > Uncompressed Kernel Size: 14MB
> > Compressed Size Decompression Speed
> > LZO 6.0MB 34.1MB/s Old
> > ----------------------------------------
> > 6.0MB 34.7MB/s New
> > 6.0MB 52.2MB/s(UA)
> > =============================================
> > LZ4 6.5MB 86.7MB/s
> > UA: Unaligned memory Access support
>
> That is pretty conclusive - it shows an 8% increase in image size vs a
> 66% increase in decompression speed. It will take a _lot_ to offset
> that increase in decompression speed.
>
> So, what I think is that yes, we should accept LZ4 and drop LZO from
> the kernel - the "fast but may not be small" compression title has
> clearly been taken by LZ4.
>
> Akpm - what's your thoughts?
It sounds like we should merge both.
I've sent Linus a little reminder for Markus's 3.9 pull request. Let's
get down and review and test this new code?
David's review comments were useful.
I'd like to also see a Kconfig patch which makes x86 and arm kernels
default to the new LZ4 code. Then I can sneak that patch into
linux-next so the new code will get some testing. If we don't do that,
very few people will run it.
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
2013-02-27 17:36 ` Russell King - ARM Linux
@ 2013-02-28 4:22 ` Joe Perches
-1 siblings, 0 replies; 67+ messages in thread
From: Joe Perches @ 2013-02-28 4:22 UTC (permalink / raw)
To: Russell King - ARM Linux
Cc: Peter Korsgaard, Nicolas Pitre, Markus F.X.J. Oberhumer,
Kyungsik Lee, Andrew Morton, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nitin Gupta, Josh Triplett,
Joe Millenbach, David Sterba, Richard Cochran, Egon Alter,
hyojun.im, chan.jeong, raphael.andy.lee
(removed Richard Purdie and Albin Tonnerre as their email addresses
seem to be bounding)
> While recently asking someone to enable VFP debugging, so I could help
> sort out a problem they had reported, this is the debug output I was
> greeted by thanks to your meddling:
:) Meddling... You sound like one of those nameless
villains on Scooby Doo. If only I had a cool nickname
like Dave "Shaggy" Kliekamp. I guess you'd have to call
me Velma.
> [ 927.235546] \x01\x01\x01\x01\x01\x01\x01\x01
> ...
> [ 927.241505] \x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01
>
> 7.6 `.asciz "STRING"'...
> ========================
>
> `.asciz' is just like `.ascii', but each string is followed by a zero
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> byte. The "z" in `.asciz' stands for "zero".
> ^^^^^
Yeah, sorry. I thought that the zero was after any
concatenation like .c. Learned something. 'preciate
that. Would have appreciated a polite "you broke it"
email too.
> ??? Yea, right, meanwhile breaking the ability of stuff to produce
> kernel messages.
Fortunately, that's the only .S instance.
> > > > > > > so that we can see whether it's worth updating the LZO code
> > Sounded as if you were doubtful to me.
> _In_ the decompressor. We're talking about the _decompressor_ in
> this thread.
My opinion is it's useful to update LZO.
cheers, Joe (aka: Velma)
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel
@ 2013-02-28 4:22 ` Joe Perches
0 siblings, 0 replies; 67+ messages in thread
From: Joe Perches @ 2013-02-28 4:22 UTC (permalink / raw)
To: linux-arm-kernel
(removed Richard Purdie and Albin Tonnerre as their email addresses
seem to be bounding)
> While recently asking someone to enable VFP debugging, so I could help
> sort out a problem they had reported, this is the debug output I was
> greeted by thanks to your meddling:
:) Meddling... You sound like one of those nameless
villains on Scooby Doo. If only I had a cool nickname
like Dave "Shaggy" Kliekamp. I guess you'd have to call
me Velma.
> [ 927.235546] \x01\x01\x01\x01\x01\x01\x01\x01
> ...
> [ 927.241505] \x01\x01\x01\x01\x01\x01\x01\x01\x01\x01\x01
>
> 7.6 `.asciz "STRING"'...
> ========================
>
> `.asciz' is just like `.ascii', but each string is followed by a zero
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> byte. The "z" in `.asciz' stands for "zero".
> ^^^^^
Yeah, sorry. I thought that the zero was after any
concatenation like .c. Learned something. 'preciate
that. Would have appreciated a polite "you broke it"
email too.
> ??? Yea, right, meanwhile breaking the ability of stuff to produce
> kernel messages.
Fortunately, that's the only .S instance.
> > > > > > > so that we can see whether it's worth updating the LZO code
> > Sounded as if you were doubtful to me.
> _In_ the decompressor. We're talking about the _decompressor_ in
> this thread.
My opinion is it's useful to update LZO.
cheers, Joe (aka: Velma)
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [RFC PATCH v2 2/4] lib: Add support for LZ4-compressed kernel
2013-02-26 14:00 ` David Sterba
@ 2013-02-28 5:22 ` Kyungsik Lee
-1 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-28 5:22 UTC (permalink / raw)
To: David Sterba
Cc: Andrew Morton, Russell King, Thomas Gleixner, Ingo Molnar,
H. Peter Anvin, Michal Marek, linux-arm-kernel, linux-kernel,
linux-kbuild, x86, celinux-dev, Nicolas Pitre, Nitin Gupta,
Markus F.X.J. Oberhumer, Josh Triplett, Joe Millenbach,
Richard Cochran, Egon Alter, hyojun.im, chan.jeong,
raphael.andy.lee
On Tue, Feb 26, 2013 at 03:00:51PM +0100, David Sterba wrote:
> On Tue, Feb 26, 2013 at 03:24:28PM +0900, Kyungsik Lee wrote:
> > +config KERNEL_LZ4
> > + bool "LZ4"
> > + depends on HAVE_KERNEL_LZ4
> > + help
> > + Its compression ratio is worse than LZO. The size of the kernel
> > + is about 8% bigger than LZO. But the decompression speed is
> > + faster than LZO.
> > +
>
> Can you please add a sentence what lz4 actually is before you start
> comparing it with the current competitor(s)?
Yes, I will update it.
> > --- /dev/null
> > +++ b/lib/decompress_unlz4.c
> > @@ -0,0 +1,190 @@
> > +#define LZ4_CHUNK_SIZE (8<<20)
>
> Please use a less cryptic way of representing a value of 8 MB. Also,
> this is a hardcoded value that must be used on the compressor side. It's
> an upper limit, so anything below 8MB does not break decompresssion, but
> this must be somehow checked or saved along in the binary stream. You
> seem to use the lz4demo.c on the userspace side for compression, but
> this is not a standard tool nor the output format is well-defined or
> stabilized.
Yes, It will be easy to parse and extract the meta information required
for the kernel decompressor if output format is well-defined and
standardized.
> For proper use I would like to see a commandline tool similar to
> gzip/bzip2/lzop that can be packaged and shipped by distros, and the
> output format defintion.
>
> Yann has some ideas for the format
> http://fastcompression.blogspot.cz/2012/04/file-container-format-for-lz4.html
Actually, I would like to use a packaged tool by distros too.
lz4demo is what I use for compressing the kernel so far.
I name it lz4 in the patch.
> For kernel, the minimum of meta information is total compressed length,
> total uncompressed length and chunk size. I don't know if the first two
> aren't stored elsewhere in the generic kernel image headers, but chunk
> size must be specified.
Right, Currently the chunk size used in the kernel decompressor is the same
as defined in lz4demo but it should have been provided as meta
information.
Thanks,
Kyungsik
^ permalink raw reply [flat|nested] 67+ messages in thread
* [RFC PATCH v2 2/4] lib: Add support for LZ4-compressed kernel
@ 2013-02-28 5:22 ` Kyungsik Lee
0 siblings, 0 replies; 67+ messages in thread
From: Kyungsik Lee @ 2013-02-28 5:22 UTC (permalink / raw)
To: linux-arm-kernel
On Tue, Feb 26, 2013 at 03:00:51PM +0100, David Sterba wrote:
> On Tue, Feb 26, 2013 at 03:24:28PM +0900, Kyungsik Lee wrote:
> > +config KERNEL_LZ4
> > + bool "LZ4"
> > + depends on HAVE_KERNEL_LZ4
> > + help
> > + Its compression ratio is worse than LZO. The size of the kernel
> > + is about 8% bigger than LZO. But the decompression speed is
> > + faster than LZO.
> > +
>
> Can you please add a sentence what lz4 actually is before you start
> comparing it with the current competitor(s)?
Yes, I will update it.
> > --- /dev/null
> > +++ b/lib/decompress_unlz4.c
> > @@ -0,0 +1,190 @@
> > +#define LZ4_CHUNK_SIZE (8<<20)
>
> Please use a less cryptic way of representing a value of 8 MB. Also,
> this is a hardcoded value that must be used on the compressor side. It's
> an upper limit, so anything below 8MB does not break decompresssion, but
> this must be somehow checked or saved along in the binary stream. You
> seem to use the lz4demo.c on the userspace side for compression, but
> this is not a standard tool nor the output format is well-defined or
> stabilized.
Yes, It will be easy to parse and extract the meta information required
for the kernel decompressor if output format is well-defined and
standardized.
> For proper use I would like to see a commandline tool similar to
> gzip/bzip2/lzop that can be packaged and shipped by distros, and the
> output format defintion.
>
> Yann has some ideas for the format
> http://fastcompression.blogspot.cz/2012/04/file-container-format-for-lz4.html
Actually, I would like to use a packaged tool by distros too.
lz4demo is what I use for compressing the kernel so far.
I name it lz4 in the patch.
> For kernel, the minimum of meta information is total compressed length,
> total uncompressed length and chunk size. I don't know if the first two
> aren't stored elsewhere in the generic kernel image headers, but chunk
> size must be specified.
Right, Currently the chunk size used in the kernel decompressor is the same
as defined in lz4demo but it should have been provided as meta
information.
Thanks,
Kyungsik
^ permalink raw reply [flat|nested] 67+ messages in thread
end of thread, other threads:[~2013-02-28 5:22 UTC | newest]
Thread overview: 67+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-02-26 6:24 [RFC PATCH v2 0/4] Add support for LZ4-compressed kernel Kyungsik Lee
2013-02-26 6:24 ` Kyungsik Lee
2013-02-26 6:24 ` [RFC PATCH v2 1/4] decompressor: Add LZ4 decompressor module Kyungsik Lee
2013-02-26 6:24 ` Kyungsik Lee
2013-02-26 13:12 ` David Sterba
2013-02-26 13:12 ` David Sterba
2013-02-27 4:38 ` Kyungsik Lee
2013-02-27 4:38 ` Kyungsik Lee
2013-02-26 6:24 ` [RFC PATCH v2 2/4] lib: Add support for LZ4-compressed kernel Kyungsik Lee
2013-02-26 6:24 ` Kyungsik Lee
2013-02-26 14:00 ` David Sterba
2013-02-26 14:00 ` David Sterba
2013-02-28 5:22 ` Kyungsik Lee
2013-02-28 5:22 ` Kyungsik Lee
2013-02-26 6:24 ` [RFC PATCH v2 3/4] arm: " Kyungsik Lee
2013-02-26 6:24 ` Kyungsik Lee
2013-02-26 6:24 ` [RFC PATCH v2 4/4] x86: " Kyungsik Lee
2013-02-26 6:24 ` Kyungsik Lee
2013-02-26 20:33 ` [RFC PATCH v2 0/4] " Markus F.X.J. Oberhumer
2013-02-26 20:33 ` Markus F.X.J. Oberhumer
2013-02-26 20:59 ` Nicolas Pitre
2013-02-26 20:59 ` Nicolas Pitre
2013-02-26 21:58 ` Peter Korsgaard
2013-02-26 21:58 ` Peter Korsgaard
2013-02-26 22:09 ` Nicolas Pitre
2013-02-26 22:09 ` Nicolas Pitre
2013-02-26 22:10 ` Russell King - ARM Linux
2013-02-26 22:10 ` Russell King - ARM Linux
2013-02-27 1:40 ` Joe Perches
2013-02-27 1:40 ` Joe Perches
2013-02-27 9:56 ` Russell King - ARM Linux
2013-02-27 9:56 ` Russell King - ARM Linux
2013-02-27 15:49 ` Joe Perches
2013-02-27 15:49 ` Joe Perches
2013-02-27 16:08 ` Nicolas Pitre
2013-02-27 16:08 ` Nicolas Pitre
2013-02-27 16:08 ` Nicolas Pitre
2013-02-27 16:31 ` Russell King - ARM Linux
2013-02-27 16:31 ` Russell King - ARM Linux
2013-02-27 16:53 ` Borislav Petkov
2013-02-27 16:53 ` Borislav Petkov
2013-02-27 17:04 ` Joe Perches
2013-02-27 17:04 ` Joe Perches
2013-02-27 17:16 ` Nicolas Pitre
2013-02-27 17:16 ` Nicolas Pitre
2013-02-27 17:39 ` Joe Perches
2013-02-27 17:39 ` Joe Perches
2013-02-27 17:52 ` Nicolas Pitre
2013-02-27 17:52 ` Nicolas Pitre
2013-02-27 17:57 ` Russell King - ARM Linux
2013-02-27 17:57 ` Russell King - ARM Linux
2013-02-27 17:36 ` Russell King - ARM Linux
2013-02-27 17:36 ` Russell King - ARM Linux
2013-02-28 4:22 ` Joe Perches
2013-02-28 4:22 ` Joe Perches
2013-02-27 7:36 ` Kyungsik Lee
2013-02-27 7:36 ` Kyungsik Lee
2013-02-27 9:51 ` Russell King - ARM Linux
2013-02-27 9:51 ` Russell King - ARM Linux
2013-02-27 10:20 ` Johannes Stezenbach
2013-02-27 10:20 ` Johannes Stezenbach
2013-02-27 15:35 ` Nicolas Pitre
2013-02-27 15:35 ` Nicolas Pitre
2013-02-27 13:23 ` Kyungsik Lee
2013-02-27 13:23 ` Kyungsik Lee
2013-02-27 22:21 ` Andrew Morton
2013-02-27 22:21 ` Andrew Morton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.