[PATCH] kbuild: add variables for compression tools

[PATCH] kbuild: add variables for compression tools

[Sedat Dilek]

Hi Denis,

is it possible to add ZSTD compression support with the possibility to
add - for example multithreading - options?

Quote from your patch:

> As a sidenote, for multi-threaded lzma, xz compression one can use:
> $ export XZ_OPT="--threads=0"

man zstd says:

       -T#, --threads=#
              Compress using # working threads (default: 1). If # is
0, attempt to detect and use the number of physical CPU cores. In all
cases, the nb  of  threads  is  capped  to  ZST‐
              DMT_NBTHREADS_MAX==200. This modifier does nothing if
zstd is compiled without multithread support.

Of course, I can help with testing.

For the documentation - which I have not checked - are the tools and
libs mentioned you will need to have installed for certain compression
tools?
For ZSTD on Debian - these are zstd and libzstd1:amd64 packages.

Thanks.

Regards,
- Sedat -


[1] https://patchwork.kernel.org/patch/11585381/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git/commit/?h=kbuild&id=1312a1e434c1816e3bbcd4f806aa862dc735dec0

Re: [PATCH] kbuild: add variables for compression tools

[Denis Efremov]

Hi,

On 6/4/20 4:05 PM, Sedat Dilek wrote:
> Hi Denis,
> 
> is it possible to add ZSTD compression support with the possibility to
> add - for example multithreading - options?

Where do you want to use zstd in kernel?

For example, to compress headers xz is used, to compress /proc/config.gz
gzip is used. It could be relatively easy patched to use zstd for it and
it could be easy to add tarzstd-pkg target for building the kernel
as a zstd compressed tarball. However, I doubt that someone really needs it.
xz is best for headers because of compression ratio, gzip is best for config
because gzip is installed by default on most systems.
To compress modules, ramdisk or kernel image with zstd one needs
to implement zstd decompression algorithm in kernel. Given that there
are already gzip,bzip2,lzma,xz,lzo,lz4 I really doubt that it will be
useful. There should be a good comparison showing that kernel image
with zstd, for example, beats gzip and couple of others in comression ratio
and decompression speed to properly position the zstd algo.

> For the documentation - which I have not checked - are the tools and
> libs mentioned you will need to have installed for certain compression
> tools?

gzip, bzip2 are already installed on most systems. It's highly likely that
lzo, lzma, xz will require you to install them.

Thanks,
Denis

Re: [PATCH] kbuild: add variables for compression tools

[Sedat Dilek]

On Thu, Jun 4, 2020 at 3:55 PM Denis Efremov <efremov@linux.com> wrote:
>
> Hi,
>
> On 6/4/20 4:05 PM, Sedat Dilek wrote:
> > Hi Denis,
> >
> > is it possible to add ZSTD compression support with the possibility to
> > add - for example multithreading - options?
>
> Where do you want to use zstd in kernel?
>
> For example, to compress headers xz is used, to compress /proc/config.gz
> gzip is used. It could be relatively easy patched to use zstd for it and
> it could be easy to add tarzstd-pkg target for building the kernel
> as a zstd compressed tarball. However, I doubt that someone really needs it.
> xz is best for headers because of compression ratio, gzip is best for config
> because gzip is installed by default on most systems.
> To compress modules, ramdisk or kernel image with zstd one needs
> to implement zstd decompression algorithm in kernel. Given that there
> are already gzip,bzip2,lzma,xz,lzo,lz4 I really doubt that it will be
> useful. There should be a good comparison showing that kernel image
> with zstd, for example, beats gzip and couple of others in comression ratio
> and decompression speed to properly position the zstd algo.
>

[ CC Nick ]

Hi,

me and other people use "support for ZSTD-compressed kernel and
initramfs" for a long time successfully.

I have modified Debian's initramfs-tools package to support creating
such an initrd.img and boot it successfully.
See Debian Bug #955469.

There is a pull-request sent out to integrate into Linus tree.

Some numbers from Nick Terell in his pull-request:
"
The zstd compressed kernel is smaller than the gzip compressed kernel but larger
than the xz or lzma compressed kernels, and it decompresses faster than
everything except lz4. See the table below for the measurement of an x86_64
kernel ordered by compressed size:

algo size
xz   6,509,792
lzma 6,856,576
zstd 7,399,157
gzip 8,522,527
bzip 8,629,603
lzo 9,808,035
lz4 10,705,570
none 32,565,672
"

More numbers in [0].

Hope this helps you to position the zstd algo.

Maybe, Nick can give you some more details.

Thanks.

Regards,
- Sedat -

[0] https://lwn.net/Articles/817134/
[1] https://github.com/terrelln/linux/commits/zstd-v5
[2] https://lkml.org/lkml/2020/6/1/1590
[3] https://bugs.debian.org/955469

> > For the documentation - which I have not checked - are the tools and
> > libs mentioned you will need to have installed for certain compression
> > tools?
>
> gzip, bzip2 are already installed on most systems. It's highly likely that
> lzo, lzma, xz will require you to install them.
>
> Thanks,
> Denis

Re: [PATCH] kbuild: add variables for compression tools

[Denis Efremov]

> 
> me and other people use "support for ZSTD-compressed kernel and
> initramfs" for a long time successfully.
> 
> I have modified Debian's initramfs-tools package to support creating
> such an initrd.img and boot it successfully.
> See Debian Bug #955469.
> 
> There is a pull-request sent out to integrate into Linus tree.
> 
> Some numbers from Nick Terell in his pull-request:
> "
> The zstd compressed kernel is smaller than the gzip compressed kernel but larger
> than the xz or lzma compressed kernels, and it decompresses faster than
> everything except lz4. See the table below for the measurement of an x86_64
> kernel ordered by compressed size:
> 
> algo size
> xz   6,509,792
> lzma 6,856,576
> zstd 7,399,157
> gzip 8,522,527
> bzip 8,629,603
> lzo 9,808,035
> lz4 10,705,570
> none 32,565,672
> "
> 
> More numbers in [0].
> 
> Hope this helps you to position the zstd algo.

Well, I didn't know that it's already implemented.

My patch simply adds variables like GZIP, BZIP2, etc to the top makefile.
Unfortunately there is no ZSTD in kernel now. Maybe it's in the linux-next?
I can't see it. If my patch will be accepted to the mainline then it will
be reasonable to add ZSTD var in makefiles as soon as zstd compression
will be merged.

Thanks,
Denis

Re: [PATCH] kbuild: add variables for compression tools

[Sedat Dilek]

On Thu, Jun 4, 2020 at 7:01 PM Denis Efremov <efremov@linux.com> wrote:
>
> >
> > me and other people use "support for ZSTD-compressed kernel and
> > initramfs" for a long time successfully.
> >
> > I have modified Debian's initramfs-tools package to support creating
> > such an initrd.img and boot it successfully.
> > See Debian Bug #955469.
> >
> > There is a pull-request sent out to integrate into Linus tree.
> >
> > Some numbers from Nick Terell in his pull-request:
> > "
> > The zstd compressed kernel is smaller than the gzip compressed kernel but larger
> > than the xz or lzma compressed kernels, and it decompresses faster than
> > everything except lz4. See the table below for the measurement of an x86_64
> > kernel ordered by compressed size:
> >
> > algo size
> > xz   6,509,792
> > lzma 6,856,576
> > zstd 7,399,157
> > gzip 8,522,527
> > bzip 8,629,603
> > lzo 9,808,035
> > lz4 10,705,570
> > none 32,565,672
> > "
> >
> > More numbers in [0].
> >
> > Hope this helps you to position the zstd algo.
>
> Well, I didn't know that it's already implemented.
>
> My patch simply adds variables like GZIP, BZIP2, etc to the top makefile.
> Unfortunately there is no ZSTD in kernel now. Maybe it's in the linux-next?
> I can't see it. If my patch will be accepted to the mainline then it will
> be reasonable to add ZSTD var in makefiles as soon as zstd compression
> will be merged.
>

No, it was not merged anywhere AFAICS.

If you want to test please pull from [1].
I am using it on top of Linux v5.7.

- Sedat -

[1] https://github.com/terrelln/linux/commits/zstd-v5

Re: [PATCH] kbuild: add variables for compression tools

[Sedat Dilek]

On Thu, Jun 4, 2020 at 7:05 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Thu, Jun 4, 2020 at 7:01 PM Denis Efremov <efremov@linux.com> wrote:
> >
> > >
> > > me and other people use "support for ZSTD-compressed kernel and
> > > initramfs" for a long time successfully.
> > >
> > > I have modified Debian's initramfs-tools package to support creating
> > > such an initrd.img and boot it successfully.
> > > See Debian Bug #955469.
> > >
> > > There is a pull-request sent out to integrate into Linus tree.
> > >
> > > Some numbers from Nick Terell in his pull-request:
> > > "
> > > The zstd compressed kernel is smaller than the gzip compressed kernel but larger
> > > than the xz or lzma compressed kernels, and it decompresses faster than
> > > everything except lz4. See the table below for the measurement of an x86_64
> > > kernel ordered by compressed size:
> > >
> > > algo size
> > > xz   6,509,792
> > > lzma 6,856,576
> > > zstd 7,399,157
> > > gzip 8,522,527
> > > bzip 8,629,603
> > > lzo 9,808,035
> > > lz4 10,705,570
> > > none 32,565,672
> > > "
> > >
> > > More numbers in [0].
> > >
> > > Hope this helps you to position the zstd algo.
> >
> > Well, I didn't know that it's already implemented.
> >
> > My patch simply adds variables like GZIP, BZIP2, etc to the top makefile.
> > Unfortunately there is no ZSTD in kernel now. Maybe it's in the linux-next?
> > I can't see it. If my patch will be accepted to the mainline then it will
> > be reasonable to add ZSTD var in makefiles as soon as zstd compression
> > will be merged.
> >
>
> No, it was not merged anywhere AFAICS.
>
> If you want to test please pull from [1].
> I am using it on top of Linux v5.7.
>
> - Sedat -
>
> [1] https://github.com/terrelln/linux/commits/zstd-v5

v5 series overview at patchwork:
https://lore.kernel.org/patchwork/project/lkml/list/?series=437934

v5 single mbox file (cleanly applied here):
https://lore.kernel.org/patchwork/series/437934/mbox/

- Sedat -