* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel [not found] <20180322012943.4145794-1-terrelln@fb.com> @ 2018-03-22 2:58 ` Adam Borowski [not found] ` <C0384AAE-FF89-4BAA-8EFC-FB3B0A8F02AB@exactcode.com> 2018-07-09 18:04 ` Nick Terrell 2 siblings, 0 replies; 16+ messages in thread From: Adam Borowski @ 2018-03-22 2:58 UTC (permalink / raw) To: Nick Terrell Cc: Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, kernel-team, Chris Mason, Rene Rebe, Julian Andres Klode, Andi Kleen On Wed, Mar 21, 2018 at 06:29:41PM -0700, Nick Terrell wrote: > This patch set adds support for a ZSTD-compressed kernel and ramdisk > images in the kernel boot process. It only integrates the support with > x86, though the first patch is generic to all architectures. I'm running this patch set since October on amd64 armhf arm64, no explosions -- besides the obvious when it's not applied yet I forget to reconfigure initramfs-tools. Which is getting tedious, so let's merge this already. :) I've tested initrd on all of these archs; here's a debug patch to check if you're actually using it. As for compressing the kernel itself: the second patch works as submitted (ie, x86 only). Porting it to other architectures is straightforward, but eg. on arm, most boards use u-boot which insists on decompressing the kernel by itself instead of passing control. EFI/grub should work, but I haven't tested that yet. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can. ⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener. ⠈⠳⣄⠀⠀⠀⠀ A master species delegates. ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <C0384AAE-FF89-4BAA-8EFC-FB3B0A8F02AB@exactcode.com>]
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel [not found] ` <C0384AAE-FF89-4BAA-8EFC-FB3B0A8F02AB@exactcode.com> @ 2018-03-22 12:35 ` Adam Borowski 2018-04-23 21:17 ` René Rebe 0 siblings, 1 reply; 16+ messages in thread From: Adam Borowski @ 2018-03-22 12:35 UTC (permalink / raw) To: René Rebe Cc: Nick Terrell, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, kernel-team, Chris Mason, Julian Andres Klode, Andi Kleen On Thu, Mar 22, 2018 at 12:09:45PM +0100, René Rebe wrote: > Should this currently just work without any arch change on e.g. > ppc64, sparc64 et al.? I could do a test build and boot if that is > of any value, ... Initrd: no reason it wouldn't work, although for anything related to the boot process testing is a very good idea. If you test, you'd really want to apply that printk patch I posted, it's too easy to accidentally get a silent fallback to gzip or uncompressed. I wonder, perhaps such a printk could be permanently added, at a low message priority? Kernel itself: needs per-arch porting, but it's not advertised in kconfig outside x86 so that's not a problem. If you are knowledgeful about bootloaders on ppc64, sparc64, etc, such information would be great. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ A dumb species has no way to open a tuna can. ⢿⡄⠘⠷⠚⠋⠀ A smart species invents a can opener. ⠈⠳⣄⠀⠀⠀⠀ A master species delegates. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel 2018-03-22 12:35 ` Adam Borowski @ 2018-04-23 21:17 ` René Rebe 0 siblings, 0 replies; 16+ messages in thread From: René Rebe @ 2018-04-23 21:17 UTC (permalink / raw) To: Adam Borowski Cc: Nick Terrell, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, kernel-team, Chris Mason, Julian Andres Klode, Andi Kleen Hi, On 22 Mar 2018, at 13:35, Adam Borowski <kilobyte@angband.pl> wrote: > On Thu, Mar 22, 2018 at 12:09:45PM +0100, René Rebe wrote: >> Should this currently just work without any arch change on e.g. >> ppc64, sparc64 et al.? I could do a test build and boot if that is >> of any value, ... > > Initrd: no reason it wouldn't work, although for anything related to the > boot process testing is a very good idea. If you test, you'd really want to > apply that printk patch I posted, it's too easy to accidentally get a silent > fallback to gzip or uncompressed. I wonder, perhaps such a printk could be > permanently added, at a low message priority? zstd initrd worked on sparc64/sun4u - Ultra 30: https://www.youtube.com/watch?v=10q2OxHAzQ4 Any chance we could get this initrd zstd support upstream? > Kernel itself: needs per-arch porting, but it's not advertised in kconfig > outside x86 so that's not a problem. > > If you are knowledgeful about bootloaders on ppc64, sparc64, etc, such > information would be great. -- ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel [not found] <20180322012943.4145794-1-terrelln@fb.com> 2018-03-22 2:58 ` [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel Adam Borowski [not found] ` <C0384AAE-FF89-4BAA-8EFC-FB3B0A8F02AB@exactcode.com> @ 2018-07-09 18:04 ` Nick Terrell 2018-07-09 22:13 ` René Rebe 2018-07-13 17:42 ` [PATCH] .gitignore: add ZSTD-compressed files Adam Borowski 2 siblings, 2 replies; 16+ messages in thread From: Nick Terrell @ 2018-07-09 18:04 UTC (permalink / raw) To: Nick Terrell Cc: Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Rene Rebe, Adam Borowski, Julian Andres Klode, Andi Kleen > On Mar 21, 2018, at 6:29 PM, Nick Terrell <terrelln@fb.com> wrote: > This patch set adds support for a ZSTD-compressed kernel and ramdisk > images in the kernel boot process. It only integrates the support with > x86, though the first patch is generic to all architectures. Hi all, Is there anything blocking this from getting merged? Please let me know if there is anything I can do to help move this patch forward. Thanks, Nick ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel 2018-07-09 18:04 ` Nick Terrell @ 2018-07-09 22:13 ` René Rebe 2018-08-17 16:15 ` René Rebe 2018-07-13 17:42 ` [PATCH] .gitignore: add ZSTD-compressed files Adam Borowski 1 sibling, 1 reply; 16+ messages in thread From: René Rebe @ 2018-07-09 22:13 UTC (permalink / raw) To: Nick Terrell Cc: Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Adam Borowski, Julian Andres Klode, Andi Kleen Hi Nick, On 09 Jul 2018, at 20:04, Nick Terrell <terrelln@fb.com> wrote: >> On Mar 21, 2018, at 6:29 PM, Nick Terrell <terrelln@fb.com> wrote: >> This patch set adds support for a ZSTD-compressed kernel and ramdisk >> images in the kernel boot process. It only integrates the support with >> x86, though the first patch is generic to all architectures. > > Hi all, > > Is there anything blocking this from getting merged? > Please let me know if there is anything I can do to help move this patch forward. works fine on my side, initrd test booted on i486, x86-64, ppc, ppc64 and sparc64. For testing on mips64 I would first need to re-base the Octane^1 patch-set ;-) Greetings, René ^1) https://www.youtube.com/watch?v=AU_RV8uoTIo -- ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel 2018-07-09 22:13 ` René Rebe @ 2018-08-17 16:15 ` René Rebe 2018-08-17 16:54 ` Andi Kleen 0 siblings, 1 reply; 16+ messages in thread From: René Rebe @ 2018-08-17 16:15 UTC (permalink / raw) To: Nick Terrell Cc: Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Adam Borowski, Julian Andres Klode, Andi Kleen Hey, is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk René On 10 Jul 2018, at 00:13, René Rebe <rene@exactcode.com> wrote: > Hi Nick, > > On 09 Jul 2018, at 20:04, Nick Terrell <terrelln@fb.com> wrote: > >>> On Mar 21, 2018, at 6:29 PM, Nick Terrell <terrelln@fb.com> wrote: >>> This patch set adds support for a ZSTD-compressed kernel and ramdisk >>> images in the kernel boot process. It only integrates the support with >>> x86, though the first patch is generic to all architectures. >> >> Hi all, >> >> Is there anything blocking this from getting merged? >> Please let me know if there is anything I can do to help move this patch forward. > > works fine on my side, initrd test booted on i486, x86-64, ppc, ppc64 and > sparc64. For testing on mips64 I would first need to re-base the Octane^1 > patch-set ;-) > > Greetings, > René > > ^1) https://www.youtube.com/watch?v=AU_RV8uoTIo > > -- > ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin > http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de > -- ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin DE Legal: Amtsgericht Berlin (Charlottenburg) HRB 105123B, Tax-ID#: DE251602478 Managing Director: René Rebe http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel 2018-08-17 16:15 ` René Rebe @ 2018-08-17 16:54 ` Andi Kleen 2018-08-17 17:15 ` René Rebe 2018-08-17 17:57 ` Adam Borowski 0 siblings, 2 replies; 16+ messages in thread From: Andi Kleen @ 2018-08-17 16:54 UTC (permalink / raw) To: René Rebe Cc: Nick Terrell, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Adam Borowski, Julian Andres Klode On Fri, Aug 17, 2018 at 06:15:45PM +0200, René Rebe wrote: > Hey, > > is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk Can you remove some other compressor for the kernel image if you merge this? The "favourite compressor" seems to roughly change every year, so if we keep adding new ones things will get more and more convoluted. Surely zstd is universally better than some existing compressor the kernel already uses for itself for a particular sweet spot? If it isn't it's likely not needed. -Andi ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel 2018-08-17 16:54 ` Andi Kleen @ 2018-08-17 17:15 ` René Rebe 2018-08-17 17:57 ` Adam Borowski 1 sibling, 0 replies; 16+ messages in thread From: René Rebe @ 2018-08-17 17:15 UTC (permalink / raw) To: Andi Kleen Cc: Nick Terrell, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Adam Borowski, Julian Andres Klode Hi, On 17 Aug 2018, at 18:54, Andi Kleen <ak@linux.intel.com> wrote: > On Fri, Aug 17, 2018 at 06:15:45PM +0200, René Rebe wrote: >> Hey, >> >> is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk > > Can you remove some other compressor for the kernel image if you merge this? > > The "favourite compressor" seems to roughly change every year, so if > we keep adding new ones things will get more and more convoluted. Well, we (t2 formerly known as rocklinux) only changed our pkg archive compression from bzip2 to zstd once in 20 years ,-) > Surely zstd is universally better than some existing compressor > the kernel already uses for itself for a particular sweet spot? > If it isn't it's likely not needed. Well, IMHO this is not that much code, but if you want to remove something I guess bzip2 or LZMA may be candidates. http://zstd.net -- ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel 2018-08-17 16:54 ` Andi Kleen 2018-08-17 17:15 ` René Rebe @ 2018-08-17 17:57 ` Adam Borowski 2018-08-17 19:22 ` Andi Kleen 1 sibling, 1 reply; 16+ messages in thread From: Adam Borowski @ 2018-08-17 17:57 UTC (permalink / raw) To: Andi Kleen Cc: René Rebe, Nick Terrell, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Julian Andres Klode On Fri, Aug 17, 2018 at 09:54:03AM -0700, Andi Kleen wrote: > On Fri, Aug 17, 2018 at 06:15:45PM +0200, René Rebe wrote: > > Hey, > > > > is there any mainline future for this zstd support? > > Currently my most favourite compressor for this, and for what it’s worth > > zstd/initrd now even tested on #t2sde / hp/pa-risc > > Can you remove some other compressor for the kernel image if you merge this? Funny you say this. I want to post this after the merge window is over: https://github.com/kilobyte/linux/commits/nobz2-next-20180731 Sorry for a tree atop next, but there are conflicts that needed to be adressed. The patchset is also undertested (works for me, but not even compile-tested on archs I don't have a toolchain for at hand). > The "favourite compressor" seems to roughly change every year, so if > we keep adding new ones things will get more and more convoluted. The above patchset drops just bzip2. It is the only one that's strictly beaten in every way (ratio, time, memory usage), there are also no other uses of bzip2 anywhere in the kernel so we'd get to drop its code completely: 900 lines of Linus' happiness. Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), but those are used elsewhere thus there's hardly any gain. If you want them gone, please say so -- I'll include their droppage. > Surely zstd is universally better than some existing compressor > the kernel already uses for itself for a particular sweet spot? > If it isn't it's likely not needed. zstd is better in the balanced niche: unless you want extreme speed (lz4) or best compression (xz), zstd handily beats algorithms in the middle. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition: ⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17] ⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37] ⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel 2018-08-17 17:57 ` Adam Borowski @ 2018-08-17 19:22 ` Andi Kleen 2018-08-17 20:07 ` Adam Borowski 0 siblings, 1 reply; 16+ messages in thread From: Andi Kleen @ 2018-08-17 19:22 UTC (permalink / raw) To: Adam Borowski Cc: René Rebe, Nick Terrell, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Julian Andres Klode On Fri, Aug 17, 2018 at 07:57:46PM +0200, Adam Borowski wrote: > > The "favourite compressor" seems to roughly change every year, so if > > we keep adding new ones things will get more and more convoluted. > > The above patchset drops just bzip2. It is the only one that's strictly > beaten in every way (ratio, time, memory usage), there are also no other Does time include build time? I've been reverting back to gzip recently because I care very much about that. > uses of bzip2 anywhere in the kernel so we'd get to drop its code > completely: 900 lines of Linus' happiness. Great! > > Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), > but those are used elsewhere thus there's hardly any gain. If you want them > gone, please say so -- I'll include their droppage. Yes would be good to remove the kernel image support for those too just to simplify the config process, even if it doesn't save much code. -Andi ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel 2018-08-17 19:22 ` Andi Kleen @ 2018-08-17 20:07 ` Adam Borowski 2018-08-28 2:36 ` Nick Terrell 0 siblings, 1 reply; 16+ messages in thread From: Adam Borowski @ 2018-08-17 20:07 UTC (permalink / raw) To: Andi Kleen Cc: René Rebe, Nick Terrell, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Julian Andres Klode On Fri, Aug 17, 2018 at 12:22:44PM -0700, Andi Kleen wrote: > On Fri, Aug 17, 2018 at 07:57:46PM +0200, Adam Borowski wrote: > > > The "favourite compressor" seems to roughly change every year, so if > > > we keep adding new ones things will get more and more convoluted. > > > > The above patchset drops just bzip2. It is the only one that's strictly > > beaten in every way (ratio, time, memory usage), there are also no other > > Does time include build time? I've been reverting back to gzip recently > because I care very much about that. Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while ago), here's copypasta of a random 16824672 byte executable, in userspace, with default level setting: comp decomp size xz 8.038s 0.356s 4320292 bz2 2.265s 0.730s 5234516 zst 0.274s 0.102s 5657626 gz 0.880s 0.152s 6515505 Z 0.499s 0.133s 8932459 lzo 0.100s 0.095s 9198874 As you can see, zstd's compression time is drastically better than gzip, while ratio is better. The default level is very low (-3 on -1..-22 scale) but you can crank it up for stronger compression. The defaults fit your use case. > > uses of bzip2 anywhere in the kernel so we'd get to drop its code > > completely: 900 lines of Linus' happiness. > > Great! > > > Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), > > but those are used elsewhere thus there's hardly any gain. If you want them > > gone, please say so -- I'll include their droppage. > > Yes would be good to remove the kernel image support for those too > just to simplify the config process, even if it doesn't save much code. There's one caveat: fast choices are quite new: * lz4 userspace tools are not even in current Debian stable (just unstable) * uncompressed kernel got in only this merge window * zstd has userspace tools in Debian stable but is not merged into the kernel yet (other dists are probably similar) Thus, it might be a good idea to keep lzo for a while longer. Bare lzma can probably go -- xz filters are nice for binaries of archs it knows (disabled otherwise), and lack of header requires hacks to find out the payload's size. So it's up to you guys: do you want me to drop lzo and/or lzma? We can also drop them just for vmlinuz but not initrd. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition: ⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17] ⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37] ⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel 2018-08-17 20:07 ` Adam Borowski @ 2018-08-28 2:36 ` Nick Terrell [not found] ` <0ABE55AB-D29B-4E12-8E12-A9AFD6E39382@exactcode.com> 0 siblings, 1 reply; 16+ messages in thread From: Nick Terrell @ 2018-08-28 2:36 UTC (permalink / raw) To: Adam Borowski Cc: Andi Kleen, René Rebe, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Julian Andres Klode On Aug 17, 2018, at 1:07 PM, Adam Borowski <kilobyte@angband.pl> wrote: > On Fri, Aug 17, 2018 at 12:22:44PM -0700, Andi Kleen wrote: >> On Fri, Aug 17, 2018 at 07:57:46PM +0200, Adam Borowski wrote: >>>> The "favourite compressor" seems to roughly change every year, so if >>>> we keep adding new ones things will get more and more convoluted. >>> >>> The above patchset drops just bzip2. It is the only one that's strictly >>> beaten in every way (ratio, time, memory usage), there are also no other >> >> Does time include build time? I've been reverting back to gzip recently >> because I care very much about that. > > Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while > ago), here's copypasta of a random 16824672 byte executable, in userspace, > with default level setting: > > comp decomp size > xz 8.038s 0.356s 4320292 > bz2 2.265s 0.730s 5234516 > zst 0.274s 0.102s 5657626 > gz 0.880s 0.152s 6515505 > Z 0.499s 0.133s 8932459 > lzo 0.100s 0.095s 9198874 > > As you can see, zstd's compression time is drastically better than gzip, > while ratio is better. The default level is very low (-3 on -1..-22 scale) > but you can crank it up for stronger compression. > > The defaults fit your use case. > >>> uses of bzip2 anywhere in the kernel so we'd get to drop its code >>> completely: 900 lines of Linus' happiness. >> >> Great! >> >>> Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead), >>> but those are used elsewhere thus there's hardly any gain. If you want them >>> gone, please say so -- I'll include their droppage. >> >> Yes would be good to remove the kernel image support for those too >> just to simplify the config process, even if it doesn't save much code. > > There's one caveat: fast choices are quite new: > * lz4 userspace tools are not even in current Debian stable (just unstable) > * uncompressed kernel got in only this merge window > * zstd has userspace tools in Debian stable but is not merged into the > kernel yet > (other dists are probably similar) > > Thus, it might be a good idea to keep lzo for a while longer. > > Bare lzma can probably go -- xz filters are nice for binaries of archs it > knows (disabled otherwise), and lack of header requires hacks to find out > the payload's size. Personally, I'd be very happy to see LZMA go. It is a custom implementation that doesn't use the lib/xz/ library. When I was implementing decompress_zstd.c I fuzzed all of the kernel decompressors, and unlzma() will crash on invalid input. There is no reason, other than not breaking compatibility, to use LZMA over XZ. > So it's up to you guys: do you want me to drop lzo and/or lzma? > We can also drop them just for vmlinuz but not initrd. > > > Meow! > -- > ⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition: > ⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17] > ⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37] > ⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26] ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <0ABE55AB-D29B-4E12-8E12-A9AFD6E39382@exactcode.com>]
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel [not found] ` <0ABE55AB-D29B-4E12-8E12-A9AFD6E39382@exactcode.com> @ 2019-06-07 19:20 ` Nick Terrell 2019-06-07 23:06 ` Adam Borowski 0 siblings, 1 reply; 16+ messages in thread From: Nick Terrell @ 2019-06-07 19:20 UTC (permalink / raw) To: René Rebe Cc: Adam Borowski, Andi Kleen, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Julian Andres Klode > On Jun 5, 2019, at 7:43 AM, René Rebe <rene@exactcode.com> wrote: > > Hey there, > > just wanted to check about the status of zstd-compressed kernel. > It works great for us, and the initrd part I tested on PPC, PPC64, SPARC64, > HPPA, MIPS64 and probably other random things that I may forgot about in the > meantime. > > Would be great if something like this could be mainlined. We'd love to get this mainlined as well! We're using these patches internally as well. We're seeing an improvement on an Intel Atom N3710, where boot time is reduced by one second over using an xz compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel, but zstd is likely a better trade off, because it compresses much better and still has excellent decompression speed. Since its been nearly a year since I sent these out, I will take some time to rebase and retest the patches in case anything changed, and then then resend the patches in the next weeks. Nick > [...] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel 2019-06-07 19:20 ` Nick Terrell @ 2019-06-07 23:06 ` Adam Borowski 2019-06-10 18:40 ` Nick Terrell 0 siblings, 1 reply; 16+ messages in thread From: Adam Borowski @ 2019-06-07 23:06 UTC (permalink / raw) To: Nick Terrell Cc: René Rebe, Andi Kleen, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Julian Andres Klode On Fri, Jun 07, 2019 at 07:20:46PM +0000, Nick Terrell wrote: > We'd love to get this mainlined as well! > > We're using these patches internally as well. We're seeing an improvement on an > Intel Atom N3710, where boot time is reduced by one second over using an xz > compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel, > but zstd is likely a better trade off, because it compresses much better and still has > excellent decompression speed. > > Since its been nearly a year since I sent these out, I will take some time to rebase > and retest the patches in case anything changed, and then then resend the patches > in the next weeks. Hi! After the ping, I intended to resend the patch-set (with removals included) after I return from miniDebconf Hamburg, but you 1. are the author of the non-trivial part, 2. you have a better test machinery, and 3. I have a deeply seated preference for effort to be done by people who are not me. A rebased and working version is at https://github.com/kilobyte/linux/tree/nobz2-v3 but there are no real improvements beyond rebases, a typo fix, and Paul Burton's ACK for mips. There's an unaddressed comment by Ingo Molnar https://lore.kernel.org/lkml/20181112042200.GA96061@gmail.com/ for your part of the code. So what do you suggest? Meow! -- ⢀⣴⠾⠻⢶⣦⠀ Latin: meow 4 characters, 4 columns, 4 bytes ⣾⠁⢠⠒⠀⣿⡁ Greek: μεου 4 characters, 4 columns, 8 bytes ⢿⡄⠘⠷⠚⠋ Runes: ᛗᛖᛟᚹ 4 characters, 4 columns, 12 bytes ⠈⠳⣄⠀⠀⠀⠀ Chinese: 喵 1 character, 2 columns, 3 bytes <-- best! ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel 2019-06-07 23:06 ` Adam Borowski @ 2019-06-10 18:40 ` Nick Terrell 0 siblings, 0 replies; 16+ messages in thread From: Nick Terrell @ 2019-06-10 18:40 UTC (permalink / raw) To: Adam Borowski Cc: René Rebe, Andi Kleen, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Julian Andres Klode > On Jun 7, 2019, at 4:06 PM, Adam Borowski <kilobyte@angband.pl> wrote: > > On Fri, Jun 07, 2019 at 07:20:46PM +0000, Nick Terrell wrote: >> We'd love to get this mainlined as well! >> >> We're using these patches internally as well. We're seeing an improvement on an >> Intel Atom N3710, where boot time is reduced by one second over using an xz >> compressed kernel. It looks like Ubuntu just switched to a lz4 compressed kernel, >> but zstd is likely a better trade off, because it compresses much better and still has >> excellent decompression speed. >> >> Since its been nearly a year since I sent these out, I will take some time to rebase >> and retest the patches in case anything changed, and then then resend the patches >> in the next weeks. > > Hi! > After the ping, I intended to resend the patch-set (with removals included) > after I return from miniDebconf Hamburg, but you 1. are the author of the > non-trivial part, 2. you have a better test machinery, and 3. I have a > deeply seated preference for effort to be done by people who are not me. If its okay with you I will resend the patch set with your removals included by the end of next week at the latest. I'll aim for this week. -Nick > A rebased and working version is at https://github.com/kilobyte/linux/tree/nobz2-v3 > but there are no real improvements beyond rebases, a typo fix, and Paul Burton's > ACK for mips. > > There's an unaddressed comment by Ingo Molnar > https://lore.kernel.org/lkml/20181112042200.GA96061@gmail.com/ > for your part of the code. > > So what do you suggest? > > > Meow! > -- > ⢀⣴⠾⠻⢶⣦⠀ Latin: meow 4 characters, 4 columns, 4 bytes > ⣾⠁⢠⠒⠀⣿⡁ Greek: μεου 4 characters, 4 columns, 8 bytes > ⢿⡄⠘⠷⠚⠋ Runes: ᛗᛖᛟᚹ 4 characters, 4 columns, 12 bytes > ⠈⠳⣄⠀⠀⠀⠀ Chinese: 喵 1 character, 2 columns, 3 bytes <-- best! ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH] .gitignore: add ZSTD-compressed files 2018-07-09 18:04 ` Nick Terrell 2018-07-09 22:13 ` René Rebe @ 2018-07-13 17:42 ` Adam Borowski 1 sibling, 0 replies; 16+ messages in thread From: Adam Borowski @ 2018-07-13 17:42 UTC (permalink / raw) To: Nick Terrell, Thomas Gleixner, Ingo Molnar, H. Peter Anvin, linux-kernel, x86, Kernel Team, Chris Mason, Rene Rebe, Adam Borowski, Julian Andres Klode, Andi Kleen For now, that's arch/x86/boot/compressed/vmlinux.bin.zst but probably more will come, thus let's be consistent with all other compressors. Signed-off-by: Adam Borowski <kilobyte@angband.pl> --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 97ba6b79834c..0d09cf1c053c 100644 --- a/.gitignore +++ b/.gitignore @@ -42,6 +42,7 @@ *.tab.[ch] *.tar *.xz +*.zst Module.symvers modules.builtin -- 2.18.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
end of thread, other threads:[~2019-06-10 18:41 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <20180322012943.4145794-1-terrelln@fb.com> 2018-03-22 2:58 ` [RESEND PATCH v2 0/2] Add support for ZSTD-compressed kernel Adam Borowski [not found] ` <C0384AAE-FF89-4BAA-8EFC-FB3B0A8F02AB@exactcode.com> 2018-03-22 12:35 ` Adam Borowski 2018-04-23 21:17 ` René Rebe 2018-07-09 18:04 ` Nick Terrell 2018-07-09 22:13 ` René Rebe 2018-08-17 16:15 ` René Rebe 2018-08-17 16:54 ` Andi Kleen 2018-08-17 17:15 ` René Rebe 2018-08-17 17:57 ` Adam Borowski 2018-08-17 19:22 ` Andi Kleen 2018-08-17 20:07 ` Adam Borowski 2018-08-28 2:36 ` Nick Terrell [not found] ` <0ABE55AB-D29B-4E12-8E12-A9AFD6E39382@exactcode.com> 2019-06-07 19:20 ` Nick Terrell 2019-06-07 23:06 ` Adam Borowski 2019-06-10 18:40 ` Nick Terrell 2018-07-13 17:42 ` [PATCH] .gitignore: add ZSTD-compressed files Adam Borowski
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).