Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

From: Nick Terrell <terrelln@fb.com>
To: "sedat.dilek@gmail.com" <sedat.dilek@gmail.com>
Cc: Nick Terrell <nickrterrell@gmail.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Chris Mason <clm@fb.com>,
	"linux-kbuild@vger.kernel.org" <linux-kbuild@vger.kernel.org>,
	"x86@kernel.org" <x86@kernel.org>,
	"gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
	Petr Malat <oss@malat.biz>, Kees Cook <keescook@chromium.org>,
	Kernel Team <Kernel-team@fb.com>,
	Adam Borowski <kilobyte@angband.pl>,
	Patrick Williams <patrickw3@fb.com>,
	"Michael van der Westhuizen" <rmikey@fb.com>,
	"mingo@kernel.org" <mingo@kernel.org>,
	"Patrick Williams" <patrick@stwcx.xyz>,
	Nick Desaulniers <ndesaulniers@google.com>
Subject: Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs
Date: Wed, 1 Apr 2020 02:52:06 +0000	[thread overview]
Message-ID: <5F9D2C8D-B1DE-4F95-992E-524E02851CF4@fb.com> (raw)
In-Reply-To: <CA+icZUWBJz6765Szg65HcOfhDh9tyoarJTnZ_kmahqvB5QKU=g@mail.gmail.com>

> On Mar 31, 2020, at 8:11 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> 
> On Tue, Mar 31, 2020 at 6:15 AM Nick Terrell <terrelln@fb.com> wrote:
> 
> [...]
> 
>>> Do you plan to push this for Linux v5.7?
>> 
>> I hope that it will be accepted. From my point of view this patch set is
>> ready for merge, except for the maximum window size increase
>> requested by Petr.
>> 
> 
> Hi Nick,
> 
> thanks for your answer.
> 
> Did you send out a pull-request already?

I haven’t sent out a pull request. If that is something that I need to
do, or if it makes the process easier for a maintainer to merge then
I can do that.

> Can you point me to that request or patch of Petr?
> Is it relevant or optional for the pull-request?

The patches from Petr are a parallel implementation of initramfs
decompression and are available here [0]. They support initramfs
compression well, but are missing some pieces for correct and
performant zstd kernel decompression. Namely the equivalent of
patch 6 in this series to increase ZO_z_extra_bytes, the
performance fix in patch 1 that switches memcpy() to
__builtin_memcpy(), and the decompress_single() function in patch 3.

Without those changes a very compressible kernel may fail to
decompress successfully, and will be corrupted. And the performance
is about 4x worse on x86_64. My patches took 70 ms to decompress vs
318 ms with Petr’s. See the detailed comparison here [1].

>>> Feel free to add credits for the whole series:
>>> 
>>>  Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
>> 
>> Thanks for testing the patch set Sedat!
>> 
> 
> I have re-tested zstd-v3 patchset with Linux version 5.6 final and
> Clang/LLD version 10.0.0 final (from Debian/unstable repository).
> 
> Is it possible to mention that there might distro-specific changes
> needed to initramfs-handling?
> For Debian you are welcome to include below Link [1].
> Not sure I will send/ask to/on the debian-kernel mailing list in this topic.

I don’t expect any distro specific changes are required to continue operating
as-is. However, if a distro wanted to switch to a zstd compressed initramfs 
they would need to update their toolchain to compress with zstd.

> Thanks and bonne chance.
> 
> Regards,
> - Sedat -
> 
> [1] LINK: https://lore.kernel.org/lkml/CA+icZUXCn2an9aNDrm+-eneSAOyGibz0W1xYhwkA5k3B3U-5vQ@mail.gmail.com/

[0] https://lkml.org/lkml/2020/3/16/461
[1] https://lkml.org/lkml/2020/3/31/1573