Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[Adam Borowski]

On Wed, Mar 21, 2018 at 06:29:41PM -0700, Nick Terrell wrote:
> This patch set adds support for a ZSTD-compressed kernel and ramdisk
> images in the kernel boot process. It only integrates the support with
> x86, though the first patch is generic to all architectures.

I'm running this patch set since October on amd64 armhf arm64, no
explosions -- besides the obvious when it's not applied yet I forget to
reconfigure initramfs-tools.  Which is getting tedious, so let's merge this
already. :)

I've tested initrd on all of these archs; here's a debug patch to check if
you're actually using it.

As for compressing the kernel itself: the second patch works as submitted
(ie, x86 only).  Porting it to other architectures is straightforward, but
eg. on arm, most boards use u-boot which insists on decompressing the kernel
by itself instead of passing control.  EFI/grub should work, but I haven't
tested that yet.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄⠀⠀⠀⠀ A master species delegates.

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[Adam Borowski]

On Thu, Mar 22, 2018 at 12:09:45PM +0100, René Rebe wrote:
> Should this currently just work without any arch change on e.g.
> ppc64, sparc64 et al.? I could do a test build and boot if that is
> of any value, ...

Initrd: no reason it wouldn't work, although for anything related to the
boot process testing is a very good idea.  If you test, you'd really want to
apply that printk patch I posted, it's too easy to accidentally get a silent
fallback to gzip or uncompressed.  I wonder, perhaps such a printk could be
permanently added, at a low message priority?

Kernel itself: needs per-arch porting, but it's not advertised in kconfig
outside x86 so that's not a problem.  

If you are knowledgeful about bootloaders on ppc64, sparc64, etc, such
information would be great.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ 
⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can.
⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener.
⠈⠳⣄⠀⠀⠀⠀ A master species delegates.

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[René Rebe]

Hi,

On 22 Mar 2018, at 13:35, Adam Borowski <kilobyte@angband.pl> wrote:

> On Thu, Mar 22, 2018 at 12:09:45PM +0100, René Rebe wrote:
>> Should this currently just work without any arch change on e.g.
>> ppc64, sparc64 et al.? I could do a test build and boot if that is
>> of any value, ...
> 
> Initrd: no reason it wouldn't work, although for anything related to the
> boot process testing is a very good idea.  If you test, you'd really want to
> apply that printk patch I posted, it's too easy to accidentally get a silent
> fallback to gzip or uncompressed.  I wonder, perhaps such a printk could be
> permanently added, at a low message priority?

zstd initrd worked on sparc64/sun4u - Ultra 30:

	https://www.youtube.com/watch?v=10q2OxHAzQ4

Any chance we could get this initrd zstd support upstream?

> Kernel itself: needs per-arch porting, but it's not advertised in kconfig
> outside x86 so that's not a problem.  
> 
> If you are knowledgeful about bootloaders on ppc64, sparc64, etc, such
> information would be great.


-- 
 ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
 http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[Nick Terrell]

> On Mar 21, 2018, at 6:29 PM, Nick Terrell <terrelln@fb.com> wrote:
> This patch set adds support for a ZSTD-compressed kernel and ramdisk
> images in the kernel boot process. It only integrates the support with
> x86, though the first patch is generic to all architectures.

Hi all,

Is there anything blocking this from getting merged?
Please let me know if there is anything I can do to help move this patch forward.

Thanks,
Nick

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[René Rebe]

Hi Nick,

On 09 Jul 2018, at 20:04, Nick Terrell <terrelln@fb.com> wrote:

>> On Mar 21, 2018, at 6:29 PM, Nick Terrell <terrelln@fb.com> wrote:
>> This patch set adds support for a ZSTD-compressed kernel and ramdisk
>> images in the kernel boot process. It only integrates the support with
>> x86, though the first patch is generic to all architectures.
> 
> Hi all,
> 
> Is there anything blocking this from getting merged?
> Please let me know if there is anything I can do to help move this patch forward.

works fine on my side, initrd test booted on i486, x86-64, ppc, ppc64 and
sparc64. For testing on mips64 I would first need to re-base the Octane^1
patch-set ;-)

Greetings,
	René

^1) https://www.youtube.com/watch?v=AU_RV8uoTIo

-- 
 ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
 http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[René Rebe]

Hey,

is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk

	René

On 10 Jul 2018, at 00:13, René Rebe <rene@exactcode.com> wrote:

> Hi Nick,
> 
> On 09 Jul 2018, at 20:04, Nick Terrell <terrelln@fb.com> wrote:
> 
>>> On Mar 21, 2018, at 6:29 PM, Nick Terrell <terrelln@fb.com> wrote:
>>> This patch set adds support for a ZSTD-compressed kernel and ramdisk
>>> images in the kernel boot process. It only integrates the support with
>>> x86, though the first patch is generic to all architectures.
>> 
>> Hi all,
>> 
>> Is there anything blocking this from getting merged?
>> Please let me know if there is anything I can do to help move this patch forward.
> 
> works fine on my side, initrd test booted on i486, x86-64, ppc, ppc64 and
> sparc64. For testing on mips64 I would first need to re-base the Octane^1
> patch-set ;-)
> 
> Greetings,
> 	René
> 
> ^1) https://www.youtube.com/watch?v=AU_RV8uoTIo
> 
> -- 
> ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
> http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
> 

-- 
 ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
 DE Legal: Amtsgericht Berlin (Charlottenburg) HRB 105123B, Tax-ID#: DE251602478
 Managing Director: René Rebe
 http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[Andi Kleen]

On Fri, Aug 17, 2018 at 06:15:45PM +0200, René Rebe wrote:
> Hey,
> 
> is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk

Can you remove some other compressor for the kernel image if you merge this?

The "favourite compressor" seems to roughly change every year, so if
we keep adding new ones things will get more and more convoluted.

Surely zstd is universally better than some existing compressor
the kernel already uses for itself for a particular sweet spot?
If it isn't it's likely not needed.

-Andi

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[René Rebe]

Hi,

On 17 Aug 2018, at 18:54, Andi Kleen <ak@linux.intel.com> wrote:

> On Fri, Aug 17, 2018 at 06:15:45PM +0200, René Rebe wrote:
>> Hey,
>> 
>> is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk
> 
> Can you remove some other compressor for the kernel image if you merge this?
> 
> The "favourite compressor" seems to roughly change every year, so if
> we keep adding new ones things will get more and more convoluted.

Well, we (t2 formerly known as rocklinux) only changed our pkg archive compression from bzip2 to zstd once in 20 years ,-)

> Surely zstd is universally better than some existing compressor
> the kernel already uses for itself for a particular sweet spot?
> If it isn't it's likely not needed.

Well, IMHO this is not that much code, but if you want to remove something I guess bzip2 or LZMA may be candidates.

	http://zstd.net

-- 
 ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
 http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[Adam Borowski]

On Fri, Aug 17, 2018 at 09:54:03AM -0700, Andi Kleen wrote:
> On Fri, Aug 17, 2018 at 06:15:45PM +0200, René Rebe wrote:
> > Hey,
> > 
> > is there any mainline future for this zstd support?
> > Currently my most favourite compressor for this, and for what it’s worth
> > zstd/initrd now even tested on #t2sde / hp/pa-risc
> 
> Can you remove some other compressor for the kernel image if you merge this?

Funny you say this.  I want to post this after the merge window is over:
https://github.com/kilobyte/linux/commits/nobz2-next-20180731

Sorry for a tree atop next, but there are conflicts that needed to be
adressed.  The patchset is also undertested (works for me, but not even
compile-tested on archs I don't have a toolchain for at hand).

> The "favourite compressor" seems to roughly change every year, so if
> we keep adding new ones things will get more and more convoluted.

The above patchset drops just bzip2.  It is the only one that's strictly
beaten in every way (ratio, time, memory usage), there are also no other
uses of bzip2 anywhere in the kernel so we'd get to drop its code
completely: 900 lines of Linus' happiness.

Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead),
but those are used elsewhere thus there's hardly any gain.  If you want them
gone, please say so -- I'll include their droppage.

> Surely zstd is universally better than some existing compressor
> the kernel already uses for itself for a particular sweet spot?
> If it isn't it's likely not needed.

zstd is better in the balanced niche: unless you want extreme speed (lz4) or
best compression (xz), zstd handily beats algorithms in the middle.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition:
⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17]
⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37]
⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[Andi Kleen]

On Fri, Aug 17, 2018 at 07:57:46PM +0200, Adam Borowski wrote:
> > The "favourite compressor" seems to roughly change every year, so if
> > we keep adding new ones things will get more and more convoluted.
> 
> The above patchset drops just bzip2.  It is the only one that's strictly
> beaten in every way (ratio, time, memory usage), there are also no other

Does time include build time? I've been reverting back to gzip recently
because I care very much about that.

> uses of bzip2 anywhere in the kernel so we'd get to drop its code
> completely: 900 lines of Linus' happiness.

Great!

> 
> Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead),
> but those are used elsewhere thus there's hardly any gain.  If you want them
> gone, please say so -- I'll include their droppage.

Yes would be good to remove the kernel image support for those too
just to simplify the config process, even if it doesn't save much code.

-Andi

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[Adam Borowski]

On Fri, Aug 17, 2018 at 12:22:44PM -0700, Andi Kleen wrote:
> On Fri, Aug 17, 2018 at 07:57:46PM +0200, Adam Borowski wrote:
> > > The "favourite compressor" seems to roughly change every year, so if
> > > we keep adding new ones things will get more and more convoluted.
> > 
> > The above patchset drops just bzip2.  It is the only one that's strictly
> > beaten in every way (ratio, time, memory usage), there are also no other
> 
> Does time include build time? I've been reverting back to gzip recently
> because I care very much about that.

Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while
ago), here's copypasta of a random 16824672 byte executable, in userspace,
with default level setting:

	comp	decomp	size
xz	8.038s	0.356s	4320292
bz2	2.265s	0.730s	5234516
zst	0.274s	0.102s	5657626
gz	0.880s	0.152s	6515505
Z	0.499s	0.133s	8932459
lzo	0.100s	0.095s	9198874

As you can see, zstd's compression time is drastically better than gzip,
while ratio is better.  The default level is very low (-3 on -1..-22 scale)
but you can crank it up for stronger compression.

The defaults fit your use case.

> > uses of bzip2 anywhere in the kernel so we'd get to drop its code
> > completely: 900 lines of Linus' happiness.
> 
> Great!
> 
> > Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead),
> > but those are used elsewhere thus there's hardly any gain.  If you want them
> > gone, please say so -- I'll include their droppage.
> 
> Yes would be good to remove the kernel image support for those too
> just to simplify the config process, even if it doesn't save much code.

There's one caveat: fast choices are quite new:
* lz4 userspace tools are not even in current Debian stable (just unstable)
* uncompressed kernel got in only this merge window
* zstd has userspace tools in Debian stable but is not merged into the
  kernel yet
(other dists are probably similar)

Thus, it might be a good idea to keep lzo for a while longer.

Bare lzma can probably go -- xz filters are nice for binaries of archs it
knows (disabled otherwise), and lack of header requires hacks to find out
the payload's size.

So it's up to you guys: do you want me to drop lzo and/or lzma?
We can also drop them just for vmlinuz but not initrd.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition:
⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17]
⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37]
⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[Nick Terrell]

On Aug 17, 2018, at 1:07 PM, Adam Borowski <kilobyte@angband.pl> wrote:
> On Fri, Aug 17, 2018 at 12:22:44PM -0700, Andi Kleen wrote:
>> On Fri, Aug 17, 2018 at 07:57:46PM +0200, Adam Borowski wrote:
>>>> The "favourite compressor" seems to roughly change every year, so if
>>>> we keep adding new ones things will get more and more convoluted.
>>> 
>>> The above patchset drops just bzip2.  It is the only one that's strictly
>>> beaten in every way (ratio, time, memory usage), there are also no other
>> 
>> Does time include build time? I've been reverting back to gzip recently
>> because I care very much about that.
> 
> Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while
> ago), here's copypasta of a random 16824672 byte executable, in userspace,
> with default level setting:
> 
> 	comp	decomp	size
> xz	8.038s	0.356s	4320292
> bz2	2.265s	0.730s	5234516
> zst	0.274s	0.102s	5657626
> gz	0.880s	0.152s	6515505
> Z	0.499s	0.133s	8932459
> lzo	0.100s	0.095s	9198874
> 
> As you can see, zstd's compression time is drastically better than gzip,
> while ratio is better.  The default level is very low (-3 on -1..-22 scale)
> but you can crank it up for stronger compression.
> 
> The defaults fit your use case.
> 
>>> uses of bzip2 anywhere in the kernel so we'd get to drop its code
>>> completely: 900 lines of Linus' happiness.
>> 
>> Great!
>> 
>>> Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead),
>>> but those are used elsewhere thus there's hardly any gain.  If you want them
>>> gone, please say so -- I'll include their droppage.
>> 
>> Yes would be good to remove the kernel image support for those too
>> just to simplify the config process, even if it doesn't save much code.
> 
> There's one caveat: fast choices are quite new:
> * lz4 userspace tools are not even in current Debian stable (just unstable)
> * uncompressed kernel got in only this merge window
> * zstd has userspace tools in Debian stable but is not merged into the
>  kernel yet
> (other dists are probably similar)
> 
> Thus, it might be a good idea to keep lzo for a while longer.
> 
> Bare lzma can probably go -- xz filters are nice for binaries of archs it
> knows (disabled otherwise), and lack of header requires hacks to find out
> the payload's size.

Personally, I'd be very happy to see LZMA go. It is a custom implementation
that doesn't use the lib/xz/ library. When I was implementing decompress_zstd.c
I fuzzed all of the kernel decompressors, and unlzma() will crash on invalid input.
There is no reason, other than not breaking compatibility, to use LZMA over XZ.

> So it's up to you guys: do you want me to drop lzo and/or lzma?
> We can also drop them just for vmlinuz but not initrd.
> 
> 
> Meow!
> -- 
> ⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition:
> ⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17]
> ⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37]
> ⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[Nick Terrell]

> On Jun 5, 2019, at 7:43 AM, René Rebe <rene@exactcode.com> wrote:
> 
> Hey there,
> 
> just wanted to check about the status of zstd-compressed kernel.
> It works great for us, and the initrd part I tested on PPC, PPC64, SPARC64,
> HPPA, MIPS64 and probably other random things that I may forgot about in the
> meantime.
> 
> Would be great if something like this could be mainlined.

We'd love to get this mainlined as well!

We're using these patches internally as well. We're seeing an improvement on an
Intel Atom N3710, where boot time is reduced by one second over using an xz
compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel,
but zstd is likely a better trade off, because it compresses much better and still has
excellent decompression speed.

Since its been nearly a year since I sent these out, I will take some time to rebase
and retest the patches in case anything changed, and then then resend the patches
in the next weeks.

Nick

> [...]

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[Adam Borowski]

On Fri, Jun 07, 2019 at 07:20:46PM +0000, Nick Terrell wrote:
> We'd love to get this mainlined as well!
> 
> We're using these patches internally as well. We're seeing an improvement on an
> Intel Atom N3710, where boot time is reduced by one second over using an xz
> compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel,
> but zstd is likely a better trade off, because it compresses much better and still has
> excellent decompression speed.
> 
> Since its been nearly a year since I sent these out, I will take some time to rebase
> and retest the patches in case anything changed, and then then resend the patches
> in the next weeks.

Hi!
After the ping, I intended to resend the patch-set (with removals included)
after I return from miniDebconf Hamburg, but you 1. are the author of the
non-trivial part, 2. you have a better test machinery, and 3. I have a
deeply seated preference for effort to be done by people who are not me.

A rebased and working version is at https://github.com/kilobyte/linux/tree/nobz2-v3
but there are no real improvements beyond rebases, a typo fix, and Paul Burton's
ACK for mips.

There's an unaddressed comment by Ingo Molnar
https://lore.kernel.org/lkml/20181112042200.GA96061@gmail.com/
for your part of the code.

So what do you suggest?


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ Latin:   meow 4 characters, 4 columns,  4 bytes
⣾⠁⢠⠒⠀⣿⡁ Greek:   μεου 4 characters, 4 columns,  8 bytes
⢿⡄⠘⠷⠚⠋  Runes:   ᛗᛖᛟᚹ 4 characters, 4 columns, 12 bytes
⠈⠳⣄⠀⠀⠀⠀ Chinese: 喵   1 character,  2 columns,  3 bytes <-- best!

Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel

[Nick Terrell]

> On Jun 7, 2019, at 4:06 PM, Adam Borowski <kilobyte@angband.pl> wrote:
> 
> On Fri, Jun 07, 2019 at 07:20:46PM +0000, Nick Terrell wrote:
>> We'd love to get this mainlined as well!
>> 
>> We're using these patches internally as well. We're seeing an improvement on an
>> Intel Atom N3710, where boot time is reduced by one second over using an xz
>> compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel,
>> but zstd is likely a better trade off, because it compresses much better and still has
>> excellent decompression speed.
>> 
>> Since its been nearly a year since I sent these out, I will take some time to rebase
>> and retest the patches in case anything changed, and then then resend the patches
>> in the next weeks.
> 
> Hi!
> After the ping, I intended to resend the patch-set (with removals included)
> after I return from miniDebconf Hamburg, but you 1. are the author of the
> non-trivial part, 2. you have a better test machinery, and 3. I have a
> deeply seated preference for effort to be done by people who are not me.

If its okay with you I will resend the patch set with your removals included by the end
of next week at the latest. I'll aim for this week.

-Nick

> A rebased and working version is at https://github.com/kilobyte/linux/tree/nobz2-v3
> but there are no real improvements beyond rebases, a typo fix, and Paul Burton's
> ACK for mips.
> 
> There's an unaddressed comment by Ingo Molnar
> https://lore.kernel.org/lkml/20181112042200.GA96061@gmail.com/
> for your part of the code.
> 
> So what do you suggest?
> 
> 
> Meow!
> -- 
> ⢀⣴⠾⠻⢶⣦⠀ Latin:   meow 4 characters, 4 columns,  4 bytes
> ⣾⠁⢠⠒⠀⣿⡁ Greek:   μεου 4 characters, 4 columns,  8 bytes
> ⢿⡄⠘⠷⠚⠋  Runes:   ᛗᛖᛟᚹ 4 characters, 4 columns, 12 bytes
> ⠈⠳⣄⠀⠀⠀⠀ Chinese: 喵   1 character,  2 columns,  3 bytes <-- best!

[PATCH] .gitignore: add ZSTD-compressed files

[Adam Borowski]

For now, that's arch/x86/boot/compressed/vmlinux.bin.zst but probably more
will come, thus let's be consistent with all other compressors.

Signed-off-by: Adam Borowski <kilobyte@angband.pl>
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 97ba6b79834c..0d09cf1c053c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -42,6 +42,7 @@
 *.tab.[ch]
 *.tar
 *.xz
+*.zst
 Module.symvers
 modules.builtin
 
-- 
2.18.0