From: Andrew Morton <akpm@linux-foundation.org>
To: 4sschmid@informatik.uni-hamburg.de, akpm@linux-foundation.org,
gaoxiang25@huawei.com, gregkh@linuxfoundation.org,
linux-mm@kvack.org, mingo@kernel.org, mm-commits@vger.kernel.org,
nivedita@alum.mit.edu, terrelln@fb.com,
torvalds@linux-foundation.org, yann.collet.73@gmail.com
Subject: [patch 03/39] lz4: fix kernel decompression speed
Date: Fri, 14 Aug 2020 17:30:10 -0700 [thread overview]
Message-ID: <20200815003010.RsbhdyhBK%akpm@linux-foundation.org> (raw)
In-Reply-To: <20200814172939.55d6d80b6e21e4241f1ee1f3@linux-foundation.org>
From: Nick Terrell <terrelln@fb.com>
Subject: lz4: fix kernel decompression speed
This patch replaces all memcpy() calls with LZ4_memcpy() which calls
__builtin_memcpy() so the compiler can inline it.
LZ4 relies heavily on memcpy() with a constant size being inlined. In x86
and i386 pre-boot environments memcpy() cannot be inlined because memcpy()
doesn't get defined as __builtin_memcpy().
An equivalent patch has been applied upstream so that the next import
won't lose this change [1].
I've measured the kernel decompression speed using QEMU before and after
this patch for the x86_64 and i386 architectures. The speed-up is about
10x as shown below.
Code Arch Kernel Size Time Speed
v5.8 x86_64 11504832 B 148 ms 79 MB/s
patch x86_64 11503872 B 13 ms 885 MB/s
v5.8 i386 9621216 B 91 ms 106 MB/s
patch i386 9620224 B 10 ms 962 MB/s
I also measured the time to decompress the initramfs on x86_64, i386, and
arm. All three show the same decompression speed before and after, as
expected.
[1] https://github.com/lz4/lz4/pull/890
Link: http://lkml.kernel.org/r/20200803194022.2966806-1-nickrterrell@gmail.com
Signed-off-by: Nick Terrell <terrelln@fb.com>
Cc: Yann Collet <yann.collet.73@gmail.com>
Cc: Gao Xiang <gaoxiang25@huawei.com>
Cc: Sven Schmidt <4sschmid@informatik.uni-hamburg.de>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Arvind Sankar <nivedita@alum.mit.edu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
lib/lz4/lz4_compress.c | 4 ++--
lib/lz4/lz4_decompress.c | 18 +++++++++---------
lib/lz4/lz4defs.h | 10 ++++++++++
lib/lz4/lz4hc_compress.c | 2 +-
4 files changed, 22 insertions(+), 12 deletions(-)
--- a/lib/lz4/lz4_compress.c~lz4-fix-kernel-decompression-speed
+++ a/lib/lz4/lz4_compress.c
@@ -446,7 +446,7 @@ _last_literals:
*op++ = (BYTE)(lastRun << ML_BITS);
}
- memcpy(op, anchor, lastRun);
+ LZ4_memcpy(op, anchor, lastRun);
op += lastRun;
}
@@ -708,7 +708,7 @@ _last_literals:
} else {
*op++ = (BYTE)(lastRunSize<<ML_BITS);
}
- memcpy(op, anchor, lastRunSize);
+ LZ4_memcpy(op, anchor, lastRunSize);
op += lastRunSize;
}
--- a/lib/lz4/lz4_decompress.c~lz4-fix-kernel-decompression-speed
+++ a/lib/lz4/lz4_decompress.c
@@ -153,7 +153,7 @@ static FORCE_INLINE int LZ4_decompress_g
&& likely((endOnInput ? ip < shortiend : 1) &
(op <= shortoend))) {
/* Copy the literals */
- memcpy(op, ip, endOnInput ? 16 : 8);
+ LZ4_memcpy(op, ip, endOnInput ? 16 : 8);
op += length; ip += length;
/*
@@ -172,9 +172,9 @@ static FORCE_INLINE int LZ4_decompress_g
(offset >= 8) &&
(dict == withPrefix64k || match >= lowPrefix)) {
/* Copy the match. */
- memcpy(op + 0, match + 0, 8);
- memcpy(op + 8, match + 8, 8);
- memcpy(op + 16, match + 16, 2);
+ LZ4_memcpy(op + 0, match + 0, 8);
+ LZ4_memcpy(op + 8, match + 8, 8);
+ LZ4_memcpy(op + 16, match + 16, 2);
op += length + MINMATCH;
/* Both stages worked, load the next token. */
continue;
@@ -263,7 +263,7 @@ static FORCE_INLINE int LZ4_decompress_g
}
}
- memcpy(op, ip, length);
+ LZ4_memcpy(op, ip, length);
ip += length;
op += length;
@@ -350,7 +350,7 @@ _copy_match:
size_t const copySize = (size_t)(lowPrefix - match);
size_t const restSize = length - copySize;
- memcpy(op, dictEnd - copySize, copySize);
+ LZ4_memcpy(op, dictEnd - copySize, copySize);
op += copySize;
if (restSize > (size_t)(op - lowPrefix)) {
/* overlap copy */
@@ -360,7 +360,7 @@ _copy_match:
while (op < endOfMatch)
*op++ = *copyFrom++;
} else {
- memcpy(op, lowPrefix, restSize);
+ LZ4_memcpy(op, lowPrefix, restSize);
op += restSize;
}
}
@@ -386,7 +386,7 @@ _copy_match:
while (op < copyEnd)
*op++ = *match++;
} else {
- memcpy(op, match, mlen);
+ LZ4_memcpy(op, match, mlen);
}
op = copyEnd;
if (op == oend)
@@ -400,7 +400,7 @@ _copy_match:
op[2] = match[2];
op[3] = match[3];
match += inc32table[offset];
- memcpy(op + 4, match, 4);
+ LZ4_memcpy(op + 4, match, 4);
match -= dec64table[offset];
} else {
LZ4_copy8(op, match);
--- a/lib/lz4/lz4defs.h~lz4-fix-kernel-decompression-speed
+++ a/lib/lz4/lz4defs.h
@@ -137,6 +137,16 @@ static FORCE_INLINE void LZ4_writeLE16(v
return put_unaligned_le16(value, memPtr);
}
+/*
+ * LZ4 relies on memcpy with a constant size being inlined. In freestanding
+ * environments, the compiler can't assume the implementation of memcpy() is
+ * standard compliant, so apply its specialized memcpy() inlining logic. When
+ * possible, use __builtin_memcpy() to tell the compiler to analyze memcpy()
+ * as-if it were standard compliant, so it can inline it in freestanding
+ * environments. This is needed when decompressing the Linux Kernel, for example.
+ */
+#define LZ4_memcpy(dst, src, size) __builtin_memcpy(dst, src, size)
+
static FORCE_INLINE void LZ4_copy8(void *dst, const void *src)
{
#if LZ4_ARCH64
--- a/lib/lz4/lz4hc_compress.c~lz4-fix-kernel-decompression-speed
+++ a/lib/lz4/lz4hc_compress.c
@@ -570,7 +570,7 @@ _Search3:
*op++ = (BYTE) lastRun;
} else
*op++ = (BYTE)(lastRun<<ML_BITS);
- memcpy(op, anchor, iend - anchor);
+ LZ4_memcpy(op, anchor, iend - anchor);
op += iend - anchor;
}
_
next prev parent reply other threads:[~2020-08-15 0:30 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-15 0:29 incoming Andrew Morton
2020-08-15 0:30 ` [patch 01/39] asm-generic: pgalloc.h: use correct #ifdef to enable pud_alloc_one() Andrew Morton
2020-08-15 0:30 ` [patch 02/39] Revert "mm/vmstat.c: do not show lowmem reserve protection information of empty zone" Andrew Morton
2020-08-15 0:30 ` Andrew Morton [this message]
2020-08-15 0:30 ` [patch 04/39] exec: restore EACCES of S_ISDIR execve() Andrew Morton
2020-08-15 0:30 ` [patch 05/39] selftests/exec: add file type errno tests Andrew Morton
2020-08-15 0:30 ` [patch 06/39] mailmap: add entry for Greg Kurz Andrew Morton
2020-08-15 0:30 ` [patch 07/39] mm: store compound_nr as well as compound_order Andrew Morton
2020-08-15 0:30 ` [patch 08/39] mm: move page-flags include to top of file Andrew Morton
2020-08-15 0:30 ` [patch 09/39] mm: add thp_order Andrew Morton
2020-08-15 0:30 ` [patch 10/39] mm: add thp_size Andrew Morton
2020-08-15 0:30 ` [patch 11/39] mm: replace hpage_nr_pages with thp_nr_pages Andrew Morton
2020-08-15 0:30 ` [patch 12/39] mm: add thp_head Andrew Morton
2020-08-15 0:30 ` [patch 13/39] mm: introduce offset_in_thp Andrew Morton
2020-08-15 0:30 ` [patch 14/39] fs: autofs: delete repeated words in comments Andrew Morton
2020-08-15 0:30 ` [patch 15/39] mm/madvise: pass task and mm to do_madvise Andrew Morton
2020-08-15 0:30 ` [patch 16/39] pid: move pidfd_get_pid() to pid.c Andrew Morton
2020-08-15 0:30 ` [patch 17/39] mm/madvise: introduce process_madvise() syscall: an external memory hinting API Andrew Morton
2020-08-16 8:12 ` Christian Brauner
2020-08-17 15:10 ` Minchan Kim
2020-08-15 0:31 ` [patch 18/39] mm/madvise: check fatal signal pending of target process Andrew Morton
2020-08-15 2:53 ` Linus Torvalds
2020-08-15 4:59 ` Minchan Kim
2020-08-15 14:57 ` Linus Torvalds
2020-08-15 18:34 ` Minchan Kim
2020-08-16 1:43 ` Linus Torvalds
2020-08-16 5:58 ` Minchan Kim
2020-08-15 0:31 ` [patch 19/39] all arch: remove system call sys_sysctl Andrew Morton
2020-08-15 0:31 ` [patch 20/39] mm/kmemleak: silence KCSAN splats in checksum Andrew Morton
2020-08-15 0:31 ` [patch 21/39] mm/frontswap: mark various intentional data races Andrew Morton
2020-08-15 0:31 ` [patch 22/39] mm/page_io: " Andrew Morton
2020-08-15 0:31 ` [patch 23/39] mm/swap_state: " Andrew Morton
2020-08-15 0:31 ` [patch 24/39] mm/filemap.c: fix a data race in filemap_fault() Andrew Morton
2020-08-15 0:31 ` [patch 25/39] mm/swapfile: fix and annotate various data races Andrew Morton
2020-08-15 0:31 ` [patch 26/39] mm/page_counter: fix various data races at memsw Andrew Morton
2020-08-15 0:31 ` [patch 27/39] mm/memcontrol: fix a data race in scan count Andrew Morton
2020-08-15 0:31 ` [patch 28/39] mm/list_lru: fix a data race in list_lru_count_one Andrew Morton
2020-08-15 0:31 ` [patch 29/39] mm/mempool: fix a data race in mempool_free() Andrew Morton
2020-08-15 0:31 ` [patch 30/39] mm/rmap: annotate a data race at tlb_flush_batched Andrew Morton
2020-08-15 0:31 ` [patch 31/39] mm/swap.c: annotate data races for lru_rotate_pvecs Andrew Morton
2020-08-15 0:31 ` [patch 32/39] mm: annotate a data race in page_zonenum() Andrew Morton
2020-08-15 0:31 ` [patch 33/39] include/asm-generic/vmlinux.lds.h: align ro_after_init Andrew Morton
2020-08-15 0:32 ` [patch 34/39] sh: clkfwk: remove r8/r16/r32 Andrew Morton
2020-08-15 0:32 ` [patch 35/39] sh: use generic strncpy() Andrew Morton
2020-08-15 0:32 ` [patch 36/39] iomap: constify ioreadX() iomem argument (as in generic implementation) Andrew Morton
2020-08-15 0:32 ` [patch 37/39] rtl818x: " Andrew Morton
2020-08-15 0:32 ` [patch 38/39] ntb: intel: " Andrew Morton
2020-08-15 0:32 ` [patch 39/39] virtio: pci: " Andrew Morton
2020-08-19 23:09 ` mmotm 2020-08-19-16-09 uploaded Andrew Morton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200815003010.RsbhdyhBK%akpm@linux-foundation.org \
--to=akpm@linux-foundation.org \
--cc=4sschmid@informatik.uni-hamburg.de \
--cc=gaoxiang25@huawei.com \
--cc=gregkh@linuxfoundation.org \
--cc=linux-mm@kvack.org \
--cc=mingo@kernel.org \
--cc=mm-commits@vger.kernel.org \
--cc=nivedita@alum.mit.edu \
--cc=terrelln@fb.com \
--cc=torvalds@linux-foundation.org \
--cc=yann.collet.73@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).