* [PATCH] kbuild: add variables for compression tools
@ 2020-06-04 13:05 Sedat Dilek
2020-06-04 13:55 ` Denis Efremov
0 siblings, 1 reply; 6+ messages in thread
From: Sedat Dilek @ 2020-06-04 13:05 UTC (permalink / raw)
To: Denis Efremov; +Cc: linux-kbuild, Masahiro Yamada
Hi Denis,
is it possible to add ZSTD compression support with the possibility to
add - for example multithreading - options?
Quote from your patch:
> As a sidenote, for multi-threaded lzma, xz compression one can use:
> $ export XZ_OPT="--threads=0"
man zstd says:
-T#, --threads=#
Compress using # working threads (default: 1). If # is
0, attempt to detect and use the number of physical CPU cores. In all
cases, the nb of threads is capped to ZST‐
DMT_NBTHREADS_MAX==200. This modifier does nothing if
zstd is compiled without multithread support.
Of course, I can help with testing.
For the documentation - which I have not checked - are the tools and
libs mentioned you will need to have installed for certain compression
tools?
For ZSTD on Debian - these are zstd and libzstd1:amd64 packages.
Thanks.
Regards,
- Sedat -
[1] https://patchwork.kernel.org/patch/11585381/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git/commit/?h=kbuild&id=1312a1e434c1816e3bbcd4f806aa862dc735dec0
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] kbuild: add variables for compression tools 2020-06-04 13:05 [PATCH] kbuild: add variables for compression tools Sedat Dilek @ 2020-06-04 13:55 ` Denis Efremov 2020-06-04 16:07 ` Sedat Dilek 0 siblings, 1 reply; 6+ messages in thread From: Denis Efremov @ 2020-06-04 13:55 UTC (permalink / raw) To: sedat.dilek; +Cc: linux-kbuild, Masahiro Yamada Hi, On 6/4/20 4:05 PM, Sedat Dilek wrote: > Hi Denis, > > is it possible to add ZSTD compression support with the possibility to > add - for example multithreading - options? Where do you want to use zstd in kernel? For example, to compress headers xz is used, to compress /proc/config.gz gzip is used. It could be relatively easy patched to use zstd for it and it could be easy to add tarzstd-pkg target for building the kernel as a zstd compressed tarball. However, I doubt that someone really needs it. xz is best for headers because of compression ratio, gzip is best for config because gzip is installed by default on most systems. To compress modules, ramdisk or kernel image with zstd one needs to implement zstd decompression algorithm in kernel. Given that there are already gzip,bzip2,lzma,xz,lzo,lz4 I really doubt that it will be useful. There should be a good comparison showing that kernel image with zstd, for example, beats gzip and couple of others in comression ratio and decompression speed to properly position the zstd algo. > For the documentation - which I have not checked - are the tools and > libs mentioned you will need to have installed for certain compression > tools? gzip, bzip2 are already installed on most systems. It's highly likely that lzo, lzma, xz will require you to install them. Thanks, Denis ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] kbuild: add variables for compression tools 2020-06-04 13:55 ` Denis Efremov @ 2020-06-04 16:07 ` Sedat Dilek 2020-06-04 17:01 ` Denis Efremov 0 siblings, 1 reply; 6+ messages in thread From: Sedat Dilek @ 2020-06-04 16:07 UTC (permalink / raw) To: efremov; +Cc: linux-kbuild, Masahiro Yamada, Nick Terrell On Thu, Jun 4, 2020 at 3:55 PM Denis Efremov <efremov@linux.com> wrote: > > Hi, > > On 6/4/20 4:05 PM, Sedat Dilek wrote: > > Hi Denis, > > > > is it possible to add ZSTD compression support with the possibility to > > add - for example multithreading - options? > > Where do you want to use zstd in kernel? > > For example, to compress headers xz is used, to compress /proc/config.gz > gzip is used. It could be relatively easy patched to use zstd for it and > it could be easy to add tarzstd-pkg target for building the kernel > as a zstd compressed tarball. However, I doubt that someone really needs it. > xz is best for headers because of compression ratio, gzip is best for config > because gzip is installed by default on most systems. > To compress modules, ramdisk or kernel image with zstd one needs > to implement zstd decompression algorithm in kernel. Given that there > are already gzip,bzip2,lzma,xz,lzo,lz4 I really doubt that it will be > useful. There should be a good comparison showing that kernel image > with zstd, for example, beats gzip and couple of others in comression ratio > and decompression speed to properly position the zstd algo. > [ CC Nick ] Hi, me and other people use "support for ZSTD-compressed kernel and initramfs" for a long time successfully. I have modified Debian's initramfs-tools package to support creating such an initrd.img and boot it successfully. See Debian Bug #955469. There is a pull-request sent out to integrate into Linus tree. Some numbers from Nick Terell in his pull-request: " The zstd compressed kernel is smaller than the gzip compressed kernel but larger than the xz or lzma compressed kernels, and it decompresses faster than everything except lz4. See the table below for the measurement of an x86_64 kernel ordered by compressed size: algo size xz 6,509,792 lzma 6,856,576 zstd 7,399,157 gzip 8,522,527 bzip 8,629,603 lzo 9,808,035 lz4 10,705,570 none 32,565,672 " More numbers in [0]. Hope this helps you to position the zstd algo. Maybe, Nick can give you some more details. Thanks. Regards, - Sedat - [0] https://lwn.net/Articles/817134/ [1] https://github.com/terrelln/linux/commits/zstd-v5 [2] https://lkml.org/lkml/2020/6/1/1590 [3] https://bugs.debian.org/955469 > > For the documentation - which I have not checked - are the tools and > > libs mentioned you will need to have installed for certain compression > > tools? > > gzip, bzip2 are already installed on most systems. It's highly likely that > lzo, lzma, xz will require you to install them. > > Thanks, > Denis ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] kbuild: add variables for compression tools 2020-06-04 16:07 ` Sedat Dilek @ 2020-06-04 17:01 ` Denis Efremov 2020-06-04 17:05 ` Sedat Dilek 0 siblings, 1 reply; 6+ messages in thread From: Denis Efremov @ 2020-06-04 17:01 UTC (permalink / raw) To: sedat.dilek; +Cc: linux-kbuild, Masahiro Yamada, Nick Terrell > > me and other people use "support for ZSTD-compressed kernel and > initramfs" for a long time successfully. > > I have modified Debian's initramfs-tools package to support creating > such an initrd.img and boot it successfully. > See Debian Bug #955469. > > There is a pull-request sent out to integrate into Linus tree. > > Some numbers from Nick Terell in his pull-request: > " > The zstd compressed kernel is smaller than the gzip compressed kernel but larger > than the xz or lzma compressed kernels, and it decompresses faster than > everything except lz4. See the table below for the measurement of an x86_64 > kernel ordered by compressed size: > > algo size > xz 6,509,792 > lzma 6,856,576 > zstd 7,399,157 > gzip 8,522,527 > bzip 8,629,603 > lzo 9,808,035 > lz4 10,705,570 > none 32,565,672 > " > > More numbers in [0]. > > Hope this helps you to position the zstd algo. Well, I didn't know that it's already implemented. My patch simply adds variables like GZIP, BZIP2, etc to the top makefile. Unfortunately there is no ZSTD in kernel now. Maybe it's in the linux-next? I can't see it. If my patch will be accepted to the mainline then it will be reasonable to add ZSTD var in makefiles as soon as zstd compression will be merged. Thanks, Denis ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] kbuild: add variables for compression tools 2020-06-04 17:01 ` Denis Efremov @ 2020-06-04 17:05 ` Sedat Dilek 2020-06-04 17:18 ` Sedat Dilek 0 siblings, 1 reply; 6+ messages in thread From: Sedat Dilek @ 2020-06-04 17:05 UTC (permalink / raw) To: efremov; +Cc: linux-kbuild, Masahiro Yamada, Nick Terrell On Thu, Jun 4, 2020 at 7:01 PM Denis Efremov <efremov@linux.com> wrote: > > > > > me and other people use "support for ZSTD-compressed kernel and > > initramfs" for a long time successfully. > > > > I have modified Debian's initramfs-tools package to support creating > > such an initrd.img and boot it successfully. > > See Debian Bug #955469. > > > > There is a pull-request sent out to integrate into Linus tree. > > > > Some numbers from Nick Terell in his pull-request: > > " > > The zstd compressed kernel is smaller than the gzip compressed kernel but larger > > than the xz or lzma compressed kernels, and it decompresses faster than > > everything except lz4. See the table below for the measurement of an x86_64 > > kernel ordered by compressed size: > > > > algo size > > xz 6,509,792 > > lzma 6,856,576 > > zstd 7,399,157 > > gzip 8,522,527 > > bzip 8,629,603 > > lzo 9,808,035 > > lz4 10,705,570 > > none 32,565,672 > > " > > > > More numbers in [0]. > > > > Hope this helps you to position the zstd algo. > > Well, I didn't know that it's already implemented. > > My patch simply adds variables like GZIP, BZIP2, etc to the top makefile. > Unfortunately there is no ZSTD in kernel now. Maybe it's in the linux-next? > I can't see it. If my patch will be accepted to the mainline then it will > be reasonable to add ZSTD var in makefiles as soon as zstd compression > will be merged. > No, it was not merged anywhere AFAICS. If you want to test please pull from [1]. I am using it on top of Linux v5.7. - Sedat - [1] https://github.com/terrelln/linux/commits/zstd-v5 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] kbuild: add variables for compression tools 2020-06-04 17:05 ` Sedat Dilek @ 2020-06-04 17:18 ` Sedat Dilek 0 siblings, 0 replies; 6+ messages in thread From: Sedat Dilek @ 2020-06-04 17:18 UTC (permalink / raw) To: efremov; +Cc: linux-kbuild, Masahiro Yamada, Nick Terrell On Thu, Jun 4, 2020 at 7:05 PM Sedat Dilek <sedat.dilek@gmail.com> wrote: > > On Thu, Jun 4, 2020 at 7:01 PM Denis Efremov <efremov@linux.com> wrote: > > > > > > > > me and other people use "support for ZSTD-compressed kernel and > > > initramfs" for a long time successfully. > > > > > > I have modified Debian's initramfs-tools package to support creating > > > such an initrd.img and boot it successfully. > > > See Debian Bug #955469. > > > > > > There is a pull-request sent out to integrate into Linus tree. > > > > > > Some numbers from Nick Terell in his pull-request: > > > " > > > The zstd compressed kernel is smaller than the gzip compressed kernel but larger > > > than the xz or lzma compressed kernels, and it decompresses faster than > > > everything except lz4. See the table below for the measurement of an x86_64 > > > kernel ordered by compressed size: > > > > > > algo size > > > xz 6,509,792 > > > lzma 6,856,576 > > > zstd 7,399,157 > > > gzip 8,522,527 > > > bzip 8,629,603 > > > lzo 9,808,035 > > > lz4 10,705,570 > > > none 32,565,672 > > > " > > > > > > More numbers in [0]. > > > > > > Hope this helps you to position the zstd algo. > > > > Well, I didn't know that it's already implemented. > > > > My patch simply adds variables like GZIP, BZIP2, etc to the top makefile. > > Unfortunately there is no ZSTD in kernel now. Maybe it's in the linux-next? > > I can't see it. If my patch will be accepted to the mainline then it will > > be reasonable to add ZSTD var in makefiles as soon as zstd compression > > will be merged. > > > > No, it was not merged anywhere AFAICS. > > If you want to test please pull from [1]. > I am using it on top of Linux v5.7. > > - Sedat - > > [1] https://github.com/terrelln/linux/commits/zstd-v5 v5 series overview at patchwork: https://lore.kernel.org/patchwork/project/lkml/list/?series=437934 v5 single mbox file (cleanly applied here): https://lore.kernel.org/patchwork/series/437934/mbox/ - Sedat - ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-06-04 17:18 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-06-04 13:05 [PATCH] kbuild: add variables for compression tools Sedat Dilek 2020-06-04 13:55 ` Denis Efremov 2020-06-04 16:07 ` Sedat Dilek 2020-06-04 17:01 ` Denis Efremov 2020-06-04 17:05 ` Sedat Dilek 2020-06-04 17:18 ` Sedat Dilek
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.