From: Nick Terrell <nickrterrell@gmail.com>
To: Nick Terrell <terrelln@fb.com>
Cc: linux-kernel@vger.kernel.org, Chris Mason <clm@fb.com>,
linux-kbuild@vger.kernel.org, x86@kernel.org,
gregkh@linuxfoundation.org, Petr Malat <oss@malat.biz>,
Kees Cook <keescook@chromium.org>,
Kernel Team <Kernel-team@fb.com>,
Nick Terrell <nickrterrell@gmail.com>,
Adam Borowski <kilobyte@angband.pl>,
Patrick Williams <patrickw3@fb.com>,
rmikey@fb.com, mingo@kernel.org,
Patrick Williams <patrick@stwcx.xyz>
Subject: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs
Date: Wed, 25 Mar 2020 12:58:41 -0700 [thread overview]
Message-ID: <20200325195849.407900-1-nickrterrell@gmail.com> (raw)
From: Nick Terrell <terrelln@fb.com>
Hi all,
I attempted to send this patch on March 19, but it got rejected by the
mailing lists for some reason, so I'm resending it.
This patch set adds support for a ZSTD-compressed kernel, ramdisk, and
initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs
are supported on all architectures. The ZSTD-compressed kernel is only
hooked up to x86 in this patch set.
Zstandard requires slightly more memory during the kernel decompression
on x86 (192 KB vs 64 KB), and the memory usage is independent of the
window size.
Zstandard requires memory proprortional to the window size used during
compression for decompressing the ramdisk image, since streaming mode is
used. Newer versions of zstd (1.3.2+) list the window size of a file
with `zstd -lv <file>'. The absolute maximum amount of memory required
is just over 8 MB, but it can be controlled at compression time.
This patch set has been boot tested with buildroot and QEMU based off
of linux-5.6-rc6.
On i386 and x86_64 I have tested the following configurations:
* zstd compressed kernel and a separate zstd compressed initramfs
* zstd compressed kernel and a built-in zstd compressed initramfs
* gzip compressed kernel and a separate gzip compressed initramfs
* gzip compressed kernel and a built-in gzip compressed initramfs
On arm and aarch64 I tested the same configurations, except that the kernel is
always gzip compressed.
Facebook has been using v1 of these patches on x86_64 devices for more than 6
months. When we switched from a xz compressed initramfs to a zstd compressed
initramfs decompression time shrunk from 12 seconds to 3 seconds. When we
switched from a xz compressed kernel to a zstd compressed kernel we saved 2
seconds of boot time.
Facebook has been using v2 of these patches on aarch64 devices for a few weeks.
When we switched from an lzma compressed initramfs to a zstd compressed initramfs
decompression time shrunk from 27 seconds to 8 seconds.
The zstd compressed kernel is smaller than the gzip compressed kernel but larger
than the xz or lzma compressed kernels, and it decompresses faster than
everything except lz4. See the table below for the measurement of an x86_64
kernel ordered by compressed size:
algo size
xz 6,509,792
lzma 6,856,576
zstd 7,399,157
gzip 8,522,527
bzip 8,629,603
lzo 9,808,035
lz4 10,705,570
none 32,565,672
v1 -> v2:
- Rebase
- usr/Makefile and init/Kconfig were changed so the patches were updated
- No functional changes except to rebase
- Split the patches up into smaller chunks
v2 -> v3:
- Add *.zst to the .gitignore in patch 8
- Style nits in patch 3
- Rename the PREBOOT macro to ZSTD_PREBOOT and XXH_PREBOOT in patches
1 through 3
Best,
Nick Terrell
Adam Borowski (1):
.gitignore: add ZSTD-compressed files
Nick Terrell (7):
lib: prepare zstd for preboot environment
lib: prepare xxhash for preboot environment
lib: add zstd support to decompress
init: add support for zstd compressed kernel
usr: add support for zstd compressed initramfs
x86: bump ZO_z_extra_bytes margin for zstd
x86: Add support for ZSTD compressed kernel
.gitignore | 1 +
Documentation/x86/boot.rst | 6 +-
arch/x86/Kconfig | 1 +
arch/x86/boot/compressed/Makefile | 5 +-
arch/x86/boot/compressed/misc.c | 4 +
arch/x86/boot/header.S | 8 +-
arch/x86/include/asm/boot.h | 6 +-
include/linux/decompress/unzstd.h | 11 +
init/Kconfig | 15 +-
lib/Kconfig | 4 +
lib/Makefile | 1 +
lib/decompress.c | 5 +
lib/decompress_unzstd.c | 338 ++++++++++++++++++++++++++++++
lib/xxhash.c | 21 +-
lib/zstd/decompress.c | 2 +
lib/zstd/fse_decompress.c | 9 +-
lib/zstd/zstd_internal.h | 14 +-
scripts/Makefile.lib | 15 ++
usr/Kconfig | 20 ++
usr/Makefile | 1 +
20 files changed, 460 insertions(+), 27 deletions(-)
create mode 100644 include/linux/decompress/unzstd.h
create mode 100644 lib/decompress_unzstd.c
--
2.25.1
next reply other threads:[~2020-03-25 19:58 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-03-25 19:58 Nick Terrell [this message]
2020-03-25 19:58 ` [PATCH v3 1/8] lib: prepare zstd for preboot environment Nick Terrell
2020-03-26 15:40 ` Petr Malat
2020-03-26 18:50 ` Nick Terrell
2020-03-25 19:58 ` [PATCH v3 2/8] lib: prepare xxhash " Nick Terrell
2020-03-25 19:58 ` [PATCH v3 3/8] lib: add zstd support to decompress Nick Terrell
2020-03-26 16:47 ` Petr Malat
2020-03-26 19:03 ` Nick Terrell
2020-03-26 20:16 ` Petr Malat
2020-03-26 21:13 ` Nick Terrell
2020-03-26 21:44 ` Petr Malat
2020-03-26 21:58 ` Nick Terrell
2020-04-01 2:46 ` Nick Terrell
2020-03-25 19:58 ` [PATCH v3 4/8] init: add support for zstd compressed kernel Nick Terrell
2020-03-25 19:58 ` [PATCH v3 5/8] usr: add support for zstd compressed initramfs Nick Terrell
2020-03-25 19:58 ` [PATCH v3 6/8] x86: bump ZO_z_extra_bytes margin for zstd Nick Terrell
2020-03-25 19:58 ` [PATCH v3 7/8] x86: Add support for ZSTD compressed kernel Nick Terrell
2020-03-25 19:58 ` [PATCH v3 8/8] .gitignore: add ZSTD-compressed files Nick Terrell
2020-03-25 20:10 ` [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs Kees Cook
-- strict thread matches above, loose matches on Subject: below --
2020-03-29 15:45 Sedat Dilek
2020-03-31 4:15 ` Nick Terrell
2020-03-31 15:11 ` Sedat Dilek
2020-04-01 2:52 ` Nick Terrell
2020-04-01 7:27 ` Adam Borowski
2020-04-01 10:00 ` Sedat Dilek
2020-04-01 9:50 ` Sedat Dilek
2020-04-01 19:00 ` Nick Terrell
2020-04-02 14:37 ` Sedat Dilek
2020-04-01 7:18 ` Adam Borowski
[not found] <20200319195934.1101679-1-terrelln@fb.com>
2020-03-20 18:10 ` Kees Cook
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200325195849.407900-1-nickrterrell@gmail.com \
--to=nickrterrell@gmail.com \
--cc=Kernel-team@fb.com \
--cc=clm@fb.com \
--cc=gregkh@linuxfoundation.org \
--cc=keescook@chromium.org \
--cc=kilobyte@angband.pl \
--cc=linux-kbuild@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=oss@malat.biz \
--cc=patrick@stwcx.xyz \
--cc=patrickw3@fb.com \
--cc=rmikey@fb.com \
--cc=terrelln@fb.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).