linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Kees Cook <keescook@chromium.org>
To: Nick Terrell <nickrterrell@gmail.com>
Cc: Nick Terrell <terrelln@fb.com>,
	linux-kernel@vger.kernel.org, Chris Mason <clm@fb.com>,
	linux-kbuild@vger.kernel.org, x86@kernel.org,
	gregkh@linuxfoundation.org, Petr Malat <oss@malat.biz>,
	Kernel Team <Kernel-team@fb.com>,
	Adam Borowski <kilobyte@angband.pl>,
	Patrick Williams <patrickw3@fb.com>,
	rmikey@fb.com, mingo@kernel.org,
	Patrick Williams <patrick@stwcx.xyz>
Subject: Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs
Date: Wed, 25 Mar 2020 13:10:32 -0700	[thread overview]
Message-ID: <202003251309.349484C8E@keescook> (raw)
In-Reply-To: <20200325195849.407900-1-nickrterrell@gmail.com>

On Wed, Mar 25, 2020 at 12:58:41PM -0700, Nick Terrell wrote:
> From: Nick Terrell <terrelln@fb.com>
> 
> Hi all,
> 
> I attempted to send this patch on March 19, but it got rejected by the
> mailing lists for some reason, so I'm resending it.
> 
> This patch set adds support for a ZSTD-compressed kernel, ramdisk, and
> initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs
> are supported on all architectures. The ZSTD-compressed kernel is only
> hooked up to x86 in this patch set.
> 
> Zstandard requires slightly more memory during the kernel decompression
> on x86 (192 KB vs 64 KB), and the memory usage is independent of the
> window size.
> 
> Zstandard requires memory proprortional to the window size used during
> compression for decompressing the ramdisk image, since streaming mode is
> used. Newer versions of zstd (1.3.2+) list the window size of a file
> with `zstd -lv <file>'. The absolute maximum amount of memory required
> is just over 8 MB, but it can be controlled at compression time.
> 
> This patch set has been boot tested with buildroot and QEMU based off
> of linux-5.6-rc6.
> 
> On i386 and x86_64 I have tested the following configurations:
> * zstd compressed kernel and a separate zstd compressed initramfs
> * zstd compressed kernel and a built-in zstd compressed initramfs
> * gzip compressed kernel and a separate gzip compressed initramfs
> * gzip compressed kernel and a built-in gzip compressed initramfs
> 
> On arm and aarch64 I tested the same configurations, except that the kernel is
> always gzip compressed.
> 
> Facebook has been using v1 of these patches on x86_64 devices for more than 6
> months. When we switched from a xz compressed initramfs to a zstd compressed
> initramfs decompression time shrunk from 12 seconds to 3 seconds. When we
> switched from a xz compressed kernel to a zstd compressed kernel we saved 2
> seconds of boot time.
> 
> Facebook has been using v2 of these patches on aarch64 devices for a few weeks.
> When we switched from an lzma compressed initramfs to a zstd compressed initramfs
> decompression time shrunk from 27 seconds to 8 seconds.
> 
> The zstd compressed kernel is smaller than the gzip compressed kernel but larger
> than the xz or lzma compressed kernels, and it decompresses faster than
> everything except lz4. See the table below for the measurement of an x86_64
> kernel ordered by compressed size:
> 
> algo	size
> xz  	 6,509,792
> lzma	 6,856,576
> zstd	 7,399,157
> gzip	 8,522,527
> bzip	 8,629,603
> lzo 	 9,808,035
> lz4 	10,705,570
> none	32,565,672
> 
> v1 -> v2:
> - Rebase
>   - usr/Makefile and init/Kconfig were changed so the patches were updated
> - No functional changes except to rebase
> - Split the patches up into smaller chunks
> 
> v2 -> v3:
> - Add *.zst to the .gitignore in patch 8
> - Style nits in patch 3
> - Rename the PREBOOT macro to ZSTD_PREBOOT and XXH_PREBOOT in patches
>   1 through 3
> 
> Best,
> Nick Terrell
> 

Hi! Just resending Reviewed-by, since it looks like the earlier v3
didn't make it to the mailing list, as Nick just pointed out to me:

https://lore.kernel.org/lkml/202003201109.00FDE311A7@keescook/

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

> Adam Borowski (1):
>   .gitignore: add ZSTD-compressed files
> 
> Nick Terrell (7):
>   lib: prepare zstd for preboot environment
>   lib: prepare xxhash for preboot environment
>   lib: add zstd support to decompress
>   init: add support for zstd compressed kernel
>   usr: add support for zstd compressed initramfs
>   x86: bump ZO_z_extra_bytes margin for zstd
>   x86: Add support for ZSTD compressed kernel
> 
>  .gitignore                        |   1 +
>  Documentation/x86/boot.rst        |   6 +-
>  arch/x86/Kconfig                  |   1 +
>  arch/x86/boot/compressed/Makefile |   5 +-
>  arch/x86/boot/compressed/misc.c   |   4 +
>  arch/x86/boot/header.S            |   8 +-
>  arch/x86/include/asm/boot.h       |   6 +-
>  include/linux/decompress/unzstd.h |  11 +
>  init/Kconfig                      |  15 +-
>  lib/Kconfig                       |   4 +
>  lib/Makefile                      |   1 +
>  lib/decompress.c                  |   5 +
>  lib/decompress_unzstd.c           | 338 ++++++++++++++++++++++++++++++
>  lib/xxhash.c                      |  21 +-
>  lib/zstd/decompress.c             |   2 +
>  lib/zstd/fse_decompress.c         |   9 +-
>  lib/zstd/zstd_internal.h          |  14 +-
>  scripts/Makefile.lib              |  15 ++
>  usr/Kconfig                       |  20 ++
>  usr/Makefile                      |   1 +
>  20 files changed, 460 insertions(+), 27 deletions(-)
>  create mode 100644 include/linux/decompress/unzstd.h
>  create mode 100644 lib/decompress_unzstd.c
> 
> -- 
> 2.25.1
> 

-- 
Kees Cook

  parent reply	other threads:[~2020-03-25 20:10 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-25 19:58 [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs Nick Terrell
2020-03-25 19:58 ` [PATCH v3 1/8] lib: prepare zstd for preboot environment Nick Terrell
2020-03-26 15:40   ` Petr Malat
2020-03-26 18:50     ` Nick Terrell
2020-03-25 19:58 ` [PATCH v3 2/8] lib: prepare xxhash " Nick Terrell
2020-03-25 19:58 ` [PATCH v3 3/8] lib: add zstd support to decompress Nick Terrell
2020-03-26 16:47   ` Petr Malat
2020-03-26 19:03     ` Nick Terrell
2020-03-26 20:16       ` Petr Malat
2020-03-26 21:13         ` Nick Terrell
2020-03-26 21:44           ` Petr Malat
2020-03-26 21:58             ` Nick Terrell
2020-04-01  2:46         ` Nick Terrell
2020-03-25 19:58 ` [PATCH v3 4/8] init: add support for zstd compressed kernel Nick Terrell
2020-03-25 19:58 ` [PATCH v3 5/8] usr: add support for zstd compressed initramfs Nick Terrell
2020-03-25 19:58 ` [PATCH v3 6/8] x86: bump ZO_z_extra_bytes margin for zstd Nick Terrell
2020-03-25 19:58 ` [PATCH v3 7/8] x86: Add support for ZSTD compressed kernel Nick Terrell
2020-03-25 19:58 ` [PATCH v3 8/8] .gitignore: add ZSTD-compressed files Nick Terrell
2020-03-25 20:10 ` Kees Cook [this message]
  -- strict thread matches above, loose matches on Subject: below --
2020-03-29 15:45 [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs Sedat Dilek
2020-03-31  4:15 ` Nick Terrell
2020-03-31 15:11   ` Sedat Dilek
2020-04-01  2:52     ` Nick Terrell
2020-04-01  7:27       ` Adam Borowski
2020-04-01 10:00         ` Sedat Dilek
2020-04-01  9:50       ` Sedat Dilek
2020-04-01 19:00         ` Nick Terrell
2020-04-02 14:37           ` Sedat Dilek
2020-04-01  7:18     ` Adam Borowski
     [not found] <20200319195934.1101679-1-terrelln@fb.com>
2020-03-20 18:10 ` Kees Cook

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=202003251309.349484C8E@keescook \
    --to=keescook@chromium.org \
    --cc=Kernel-team@fb.com \
    --cc=clm@fb.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=kilobyte@angband.pl \
    --cc=linux-kbuild@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=nickrterrell@gmail.com \
    --cc=oss@malat.biz \
    --cc=patrick@stwcx.xyz \
    --cc=patrickw3@fb.com \
    --cc=rmikey@fb.com \
    --cc=terrelln@fb.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).