* [PATCH] init: add support for zstd compressed modules @ 2021-01-13 14:50 > 0 siblings, 0 replies; 12+ messages in thread From: > @ 2021-01-13 14:50 UTC (permalink / raw) To: linux-kbuild; +Cc: Piotr Gorski From: Piotr Gorski <lucjan.lucjanov@gmail.com> kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> --- Makefile | 7 +++++-- init/Kconfig | 9 ++++++--- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index cf54d33ba863..90986a3f5897 100644 --- a/Makefile +++ b/Makefile @@ -1049,8 +1049,8 @@ endif # INSTALL_MOD_STRIP export mod_strip_cmd # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP -# or CONFIG_MODULE_COMPRESS_XZ. +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP, +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD. mod_compress_cmd = true ifdef CONFIG_MODULE_COMPRESS @@ -1060,6 +1060,9 @@ ifdef CONFIG_MODULE_COMPRESS ifdef CONFIG_MODULE_COMPRESS_XZ mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f endif # CONFIG_MODULE_COMPRESS_XZ + ifdef CONFIG_MODULE_COMPRESS_ZSTD + mod_compress_cmd = $(ZSTD) -T0 -9 --rm -f + endif # CONFIG_MODULE_COMPRESS_ZSTD endif # CONFIG_MODULE_COMPRESS export mod_compress_cmd diff --git a/init/Kconfig b/init/Kconfig index b77c60f8b963..11d7dfea98b6 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -2216,8 +2216,8 @@ config MODULE_COMPRESS bool "Compress modules on installation" help - Compresses kernel modules when 'make modules_install' is run; gzip or - xz depending on "Compression algorithm" below. + Compresses kernel modules when 'make modules_install' is run; gzip, + xz, or zstd depending on "Compression algorithm" below. module-init-tools MAY support gzip, and kmod MAY support gzip and xz. @@ -2239,7 +2239,7 @@ choice This determines which sort of compression will be used during 'make modules_install'. - GZIP (default) and XZ are supported. + GZIP (default), XZ, and ZSTD are supported. config MODULE_COMPRESS_GZIP bool "GZIP" @@ -2247,6 +2247,9 @@ config MODULE_COMPRESS_GZIP config MODULE_COMPRESS_XZ bool "XZ" +config MODULE_COMPRESS_ZSTD + bool "ZSTD" + endchoice config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS -- 2.30.0.81.g72c4083ddf ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH] init: add support for zstd compressed modules @ 2021-03-30 11:32 Piotr Gorski 2021-03-30 11:50 ` Oleksandr Natalenko 2021-04-07 14:32 ` Masahiro Yamada 0 siblings, 2 replies; 12+ messages in thread From: Piotr Gorski @ 2021-03-30 11:32 UTC (permalink / raw) To: linux-kernel Cc: linux-kbuild, Oleksandr Natalenko, Masahiro Yamada, Piotr Gorski kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> --- Makefile | 7 +++++-- init/Kconfig | 9 ++++++--- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 5160ff8903c1..82f4f4cc2955 100644 --- a/Makefile +++ b/Makefile @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP export mod_strip_cmd # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP -# or CONFIG_MODULE_COMPRESS_XZ. +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP, +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD. mod_compress_cmd = true ifdef CONFIG_MODULE_COMPRESS @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS ifdef CONFIG_MODULE_COMPRESS_XZ mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f endif # CONFIG_MODULE_COMPRESS_XZ + ifdef CONFIG_MODULE_COMPRESS_ZSTD + mod_compress_cmd = $(ZSTD) -T0 --rm -f -q + endif # CONFIG_MODULE_COMPRESS_ZSTD endif # CONFIG_MODULE_COMPRESS export mod_compress_cmd diff --git a/init/Kconfig b/init/Kconfig index 8c2cfd88f6ef..86a452bc2747 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS bool "Compress modules on installation" help - Compresses kernel modules when 'make modules_install' is run; gzip or - xz depending on "Compression algorithm" below. + Compresses kernel modules when 'make modules_install' is run; gzip, + xz, or zstd depending on "Compression algorithm" below. module-init-tools MAY support gzip, and kmod MAY support gzip and xz. @@ -2273,7 +2273,7 @@ choice This determines which sort of compression will be used during 'make modules_install'. - GZIP (default) and XZ are supported. + GZIP (default), XZ, and ZSTD are supported. config MODULE_COMPRESS_GZIP bool "GZIP" @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP config MODULE_COMPRESS_XZ bool "XZ" +config MODULE_COMPRESS_ZSTD + bool "ZSTD" + endchoice config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS -- 2.31.0.97.g1424303384 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH] init: add support for zstd compressed modules 2021-03-30 11:32 Piotr Gorski @ 2021-03-30 11:50 ` Oleksandr Natalenko 2021-03-31 17:39 ` Nick Terrell 2021-04-07 14:32 ` Masahiro Yamada 1 sibling, 1 reply; 12+ messages in thread From: Oleksandr Natalenko @ 2021-03-30 11:50 UTC (permalink / raw) To: Piotr Gorski Cc: linux-kernel, linux-kbuild, Masahiro Yamada, Michal Marek, Andrew Morton, Kees Cook, Nathan Chancellor, Daniel Borkmann, Valentin Schneider, Jan Alexander Steffens (heftig), Nick Terrell, David Howells, Johannes Weiner On Tue, Mar 30, 2021 at 01:32:35PM +0200, Piotr Gorski wrote: > kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. > > Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> > --- > Makefile | 7 +++++-- > init/Kconfig | 9 ++++++--- > 2 files changed, 11 insertions(+), 5 deletions(-) > > diff --git a/Makefile b/Makefile > index 5160ff8903c1..82f4f4cc2955 100644 > --- a/Makefile > +++ b/Makefile > @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP > export mod_strip_cmd > > # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed > -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP > -# or CONFIG_MODULE_COMPRESS_XZ. > +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP, > +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD. > > mod_compress_cmd = true > ifdef CONFIG_MODULE_COMPRESS > @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS > ifdef CONFIG_MODULE_COMPRESS_XZ > mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f > endif # CONFIG_MODULE_COMPRESS_XZ > + ifdef CONFIG_MODULE_COMPRESS_ZSTD > + mod_compress_cmd = $(ZSTD) -T0 --rm -f -q > + endif # CONFIG_MODULE_COMPRESS_ZSTD > endif # CONFIG_MODULE_COMPRESS > export mod_compress_cmd > > diff --git a/init/Kconfig b/init/Kconfig > index 8c2cfd88f6ef..86a452bc2747 100644 > --- a/init/Kconfig > +++ b/init/Kconfig > @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS > bool "Compress modules on installation" > help > > - Compresses kernel modules when 'make modules_install' is run; gzip or > - xz depending on "Compression algorithm" below. > + Compresses kernel modules when 'make modules_install' is run; gzip, > + xz, or zstd depending on "Compression algorithm" below. > > module-init-tools MAY support gzip, and kmod MAY support gzip and xz. > > @@ -2273,7 +2273,7 @@ choice > This determines which sort of compression will be used during > 'make modules_install'. > > - GZIP (default) and XZ are supported. > + GZIP (default), XZ, and ZSTD are supported. > > config MODULE_COMPRESS_GZIP > bool "GZIP" > @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP > config MODULE_COMPRESS_XZ > bool "XZ" > > +config MODULE_COMPRESS_ZSTD > + bool "ZSTD" > + > endchoice > > config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS > -- > 2.31.0.97.g1424303384 > Great! Reviewed-by: Oleksandr Natalenko <oleksandr@natalenko.name> This works perfectly fine in Arch Linux if accompanied by the following mkinitcpio amendment: [1]. I'm also Cc'ing other people from get_maintainers output just to make this submission more visible. Thanks. [1] https://github.com/archlinux/mkinitcpio/pull/43 -- Oleksandr Natalenko (post-factum) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] init: add support for zstd compressed modules 2021-03-30 11:50 ` Oleksandr Natalenko @ 2021-03-31 17:39 ` Nick Terrell 2021-03-31 17:48 ` Oleksandr Natalenko 0 siblings, 1 reply; 12+ messages in thread From: Nick Terrell @ 2021-03-31 17:39 UTC (permalink / raw) To: Oleksandr Natalenko Cc: Piotr Gorski, LKML, Linux Kbuild mailing list, Masahiro Yamada, Michal Marek, Andrew Morton, Kees Cook, Nathan Chancellor, Daniel Borkmann, Valentin Schneider, Jan Alexander Steffens (heftig), David Howells, Johannes Weiner > On Mar 30, 2021, at 4:50 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote: > > On Tue, Mar 30, 2021 at 01:32:35PM +0200, Piotr Gorski wrote: >> kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. >> >> Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> >> --- >> Makefile | 7 +++++-- >> init/Kconfig | 9 ++++++--- >> 2 files changed, 11 insertions(+), 5 deletions(-) >> >> diff --git a/Makefile b/Makefile >> index 5160ff8903c1..82f4f4cc2955 100644 >> --- a/Makefile >> +++ b/Makefile >> @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP >> export mod_strip_cmd >> >> # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed >> -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP >> -# or CONFIG_MODULE_COMPRESS_XZ. >> +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP, >> +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD. >> >> mod_compress_cmd = true >> ifdef CONFIG_MODULE_COMPRESS >> @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS >> ifdef CONFIG_MODULE_COMPRESS_XZ >> mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f >> endif # CONFIG_MODULE_COMPRESS_XZ >> + ifdef CONFIG_MODULE_COMPRESS_ZSTD >> + mod_compress_cmd = $(ZSTD) -T0 --rm -f -q This will use the default zstd level, level 3. I think it would make more sense to use a high compression level. Level 19 would probably be a good choice. That will choose a window size of up to 8MB, meaning the decompressor needs to allocate that much memory. If that is unacceptable, you could use `zstd -T0 --rm -f -q -19 --zstd=wlog=21`, which will use a window size of up to 2MB, to match the XZ command. Note that if the file is smaller than the window size, it will be shrunk to the smallest power of two at least as large as the file. Best, Nick Terrell >> + endif # CONFIG_MODULE_COMPRESS_ZSTD >> endif # CONFIG_MODULE_COMPRESS >> export mod_compress_cmd >> >> diff --git a/init/Kconfig b/init/Kconfig >> index 8c2cfd88f6ef..86a452bc2747 100644 >> --- a/init/Kconfig >> +++ b/init/Kconfig >> @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS >> bool "Compress modules on installation" >> help >> >> - Compresses kernel modules when 'make modules_install' is run; gzip or >> - xz depending on "Compression algorithm" below. >> + Compresses kernel modules when 'make modules_install' is run; gzip, >> + xz, or zstd depending on "Compression algorithm" below. >> >> module-init-tools MAY support gzip, and kmod MAY support gzip and xz. >> >> @@ -2273,7 +2273,7 @@ choice >> This determines which sort of compression will be used during >> 'make modules_install'. >> >> - GZIP (default) and XZ are supported. >> + GZIP (default), XZ, and ZSTD are supported. >> >> config MODULE_COMPRESS_GZIP >> bool "GZIP" >> @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP >> config MODULE_COMPRESS_XZ >> bool "XZ" >> >> +config MODULE_COMPRESS_ZSTD >> + bool "ZSTD" >> + >> endchoice >> >> config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS >> -- >> 2.31.0.97.g1424303384 >> > > Great! > > Reviewed-by: Oleksandr Natalenko <oleksandr@natalenko.name> > > This works perfectly fine in Arch Linux if accompanied by the > following mkinitcpio amendment: [1]. > > I'm also Cc'ing other people from get_maintainers output just > to make this submission more visible. > > Thanks. > > [1] https://github.com/archlinux/mkinitcpio/pull/43 > > -- > Oleksandr Natalenko (post-factum) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] init: add support for zstd compressed modules 2021-03-31 17:39 ` Nick Terrell @ 2021-03-31 17:48 ` Oleksandr Natalenko 2021-03-31 19:21 ` Nick Terrell 0 siblings, 1 reply; 12+ messages in thread From: Oleksandr Natalenko @ 2021-03-31 17:48 UTC (permalink / raw) To: Nick Terrell Cc: Piotr Gorski, LKML, Linux Kbuild mailing list, Masahiro Yamada, Michal Marek, Andrew Morton, Kees Cook, Nathan Chancellor, Daniel Borkmann, Valentin Schneider, Jan Alexander Steffens (heftig), David Howells, Johannes Weiner Hello. On Wed, Mar 31, 2021 at 05:39:25PM +0000, Nick Terrell wrote: > > > > On Mar 30, 2021, at 4:50 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote: > > > > On Tue, Mar 30, 2021 at 01:32:35PM +0200, Piotr Gorski wrote: > >> kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. > >> > >> Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> > >> --- > >> Makefile | 7 +++++-- > >> init/Kconfig | 9 ++++++--- > >> 2 files changed, 11 insertions(+), 5 deletions(-) > >> > >> diff --git a/Makefile b/Makefile > >> index 5160ff8903c1..82f4f4cc2955 100644 > >> --- a/Makefile > >> +++ b/Makefile > >> @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP > >> export mod_strip_cmd > >> > >> # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed > >> -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP > >> -# or CONFIG_MODULE_COMPRESS_XZ. > >> +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP, > >> +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD. > >> > >> mod_compress_cmd = true > >> ifdef CONFIG_MODULE_COMPRESS > >> @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS > >> ifdef CONFIG_MODULE_COMPRESS_XZ > >> mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f > >> endif # CONFIG_MODULE_COMPRESS_XZ > >> + ifdef CONFIG_MODULE_COMPRESS_ZSTD > >> + mod_compress_cmd = $(ZSTD) -T0 --rm -f -q > > This will use the default zstd level, level 3. I think it would make more sense to use a high > compression level. Level 19 would probably be a good choice. That will choose a window > size of up to 8MB, meaning the decompressor needs to allocate that much memory. If that > is unacceptable, you could use `zstd -T0 --rm -f -q -19 --zstd=wlog=21`, which will use a > window size of up to 2MB, to match the XZ command. Note that if the file is smaller than > the window size, it will be shrunk to the smallest power of two at least as large as the file. Please no. We've already done that with initramfs in Arch, and it increased the time to generate it enormously. I understand that building a kernel is a more rare operation than regenerating initramfs, but still I'd go against hard-coding the level. And if it should be specified anyway, I'd opt in for an explicit configuration option. Remember, not all the kernel are built on build farms... FWIW, Piotr originally used level 9 which worked okay, but I insisted on sending the patch initially without specifying level at all like it is done for other compressors. If this is a wrong approach, then oh meh, mea culpa ;). Whatever default non-standard compression level you choose, I'm fine as long as I can change it without editing Makefile. Thanks! > > Best, > Nick Terrell > > >> + endif # CONFIG_MODULE_COMPRESS_ZSTD > >> endif # CONFIG_MODULE_COMPRESS > >> export mod_compress_cmd > >> > >> diff --git a/init/Kconfig b/init/Kconfig > >> index 8c2cfd88f6ef..86a452bc2747 100644 > >> --- a/init/Kconfig > >> +++ b/init/Kconfig > >> @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS > >> bool "Compress modules on installation" > >> help > >> > >> - Compresses kernel modules when 'make modules_install' is run; gzip or > >> - xz depending on "Compression algorithm" below. > >> + Compresses kernel modules when 'make modules_install' is run; gzip, > >> + xz, or zstd depending on "Compression algorithm" below. > >> > >> module-init-tools MAY support gzip, and kmod MAY support gzip and xz. > >> > >> @@ -2273,7 +2273,7 @@ choice > >> This determines which sort of compression will be used during > >> 'make modules_install'. > >> > >> - GZIP (default) and XZ are supported. > >> + GZIP (default), XZ, and ZSTD are supported. > >> > >> config MODULE_COMPRESS_GZIP > >> bool "GZIP" > >> @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP > >> config MODULE_COMPRESS_XZ > >> bool "XZ" > >> > >> +config MODULE_COMPRESS_ZSTD > >> + bool "ZSTD" > >> + > >> endchoice > >> > >> config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS > >> -- > >> 2.31.0.97.g1424303384 > >> > > > > Great! > > > > Reviewed-by: Oleksandr Natalenko <oleksandr@natalenko.name> > > > > This works perfectly fine in Arch Linux if accompanied by the > > following mkinitcpio amendment: [1]. > > > > I'm also Cc'ing other people from get_maintainers output just > > to make this submission more visible. > > > > Thanks. > > > > [1] https://github.com/archlinux/mkinitcpio/pull/43 > > > > -- > > Oleksandr Natalenko (post-factum) > -- Oleksandr Natalenko (post-factum) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] init: add support for zstd compressed modules 2021-03-31 17:48 ` Oleksandr Natalenko @ 2021-03-31 19:21 ` Nick Terrell 2021-03-31 20:28 ` Oleksandr Natalenko 2021-04-07 13:53 ` Masahiro Yamada 0 siblings, 2 replies; 12+ messages in thread From: Nick Terrell @ 2021-03-31 19:21 UTC (permalink / raw) To: Oleksandr Natalenko Cc: Piotr Gorski, LKML, Linux Kbuild mailing list, Masahiro Yamada, Michal Marek, Andrew Morton, Kees Cook, Nathan Chancellor, Daniel Borkmann, Valentin Schneider, Jan Alexander Steffens (heftig), David Howells, Johannes Weiner > On Mar 31, 2021, at 10:48 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote: > > Hello. > > On Wed, Mar 31, 2021 at 05:39:25PM +0000, Nick Terrell wrote: >> >> >>> On Mar 30, 2021, at 4:50 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote: >>> >>> On Tue, Mar 30, 2021 at 01:32:35PM +0200, Piotr Gorski wrote: >>>> kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. >>>> >>>> Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> >>>> --- >>>> Makefile | 7 +++++-- >>>> init/Kconfig | 9 ++++++--- >>>> 2 files changed, 11 insertions(+), 5 deletions(-) >>>> >>>> diff --git a/Makefile b/Makefile >>>> index 5160ff8903c1..82f4f4cc2955 100644 >>>> --- a/Makefile >>>> +++ b/Makefile >>>> @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP >>>> export mod_strip_cmd >>>> >>>> # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed >>>> -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP >>>> -# or CONFIG_MODULE_COMPRESS_XZ. >>>> +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP, >>>> +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD. >>>> >>>> mod_compress_cmd = true >>>> ifdef CONFIG_MODULE_COMPRESS >>>> @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS >>>> ifdef CONFIG_MODULE_COMPRESS_XZ >>>> mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f >>>> endif # CONFIG_MODULE_COMPRESS_XZ >>>> + ifdef CONFIG_MODULE_COMPRESS_ZSTD >>>> + mod_compress_cmd = $(ZSTD) -T0 --rm -f -q >> >> This will use the default zstd level, level 3. I think it would make more sense to use a high >> compression level. Level 19 would probably be a good choice. That will choose a window >> size of up to 8MB, meaning the decompressor needs to allocate that much memory. If that >> is unacceptable, you could use `zstd -T0 --rm -f -q -19 --zstd=wlog=21`, which will use a >> window size of up to 2MB, to match the XZ command. Note that if the file is smaller than >> the window size, it will be shrunk to the smallest power of two at least as large as the file. > > Please no. We've already done that with initramfs in Arch, and it > increased the time to generate it enormously. > > I understand that building a kernel is a more rare operation than > regenerating initramfs, but still I'd go against hard-coding the level. > And if it should be specified anyway, I'd opt in for an explicit > configuration option. Remember, not all the kernel are built on > build farms... > > FWIW, Piotr originally used level 9 which worked okay, but I insisted > on sending the patch initially without specifying level at all like it is > done for other compressors. If this is a wrong approach, then oh meh, > mea culpa ;). > > Whatever default non-standard compression level you choose, I'm fine > as long as I can change it without editing Makefile. That makes sense to me. I have a deep seated need to compress files as efficiently as possible for widely distributed packages. But, I understand that slow compression significantly impacts build times for quick iteration. I’d be happy with a compression level parameter that defaults to a happy middle. I’m also fine with taking this patch as-is if it is easier, and I can put up another patch that adds a compression level parameter, since I don’t want to block merging this. Best, Nick Terrell > Thanks! > >> >> Best, >> Nick Terrell >> >>>> + endif # CONFIG_MODULE_COMPRESS_ZSTD >>>> endif # CONFIG_MODULE_COMPRESS >>>> export mod_compress_cmd >>>> >>>> diff --git a/init/Kconfig b/init/Kconfig >>>> index 8c2cfd88f6ef..86a452bc2747 100644 >>>> --- a/init/Kconfig >>>> +++ b/init/Kconfig >>>> @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS >>>> bool "Compress modules on installation" >>>> help >>>> >>>> - Compresses kernel modules when 'make modules_install' is run; gzip or >>>> - xz depending on "Compression algorithm" below. >>>> + Compresses kernel modules when 'make modules_install' is run; gzip, >>>> + xz, or zstd depending on "Compression algorithm" below. >>>> >>>> module-init-tools MAY support gzip, and kmod MAY support gzip and xz. >>>> >>>> @@ -2273,7 +2273,7 @@ choice >>>> This determines which sort of compression will be used during >>>> 'make modules_install'. >>>> >>>> - GZIP (default) and XZ are supported. >>>> + GZIP (default), XZ, and ZSTD are supported. >>>> >>>> config MODULE_COMPRESS_GZIP >>>> bool "GZIP" >>>> @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP >>>> config MODULE_COMPRESS_XZ >>>> bool "XZ" >>>> >>>> +config MODULE_COMPRESS_ZSTD >>>> + bool "ZSTD" >>>> + >>>> endchoice >>>> >>>> config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS >>>> -- >>>> 2.31.0.97.g1424303384 >>>> >>> >>> Great! >>> >>> Reviewed-by: Oleksandr Natalenko <oleksandr@natalenko.name> >>> >>> This works perfectly fine in Arch Linux if accompanied by the >>> following mkinitcpio amendment: [1]. >>> >>> I'm also Cc'ing other people from get_maintainers output just >>> to make this submission more visible. >>> >>> Thanks. >>> >>> [1] https://github.com/archlinux/mkinitcpio/pull/43 >>> >>> -- >>> Oleksandr Natalenko (post-factum) >> > > -- > Oleksandr Natalenko (post-factum) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] init: add support for zstd compressed modules 2021-03-31 19:21 ` Nick Terrell @ 2021-03-31 20:28 ` Oleksandr Natalenko 2021-04-07 13:53 ` Masahiro Yamada 1 sibling, 0 replies; 12+ messages in thread From: Oleksandr Natalenko @ 2021-03-31 20:28 UTC (permalink / raw) To: Nick Terrell Cc: Piotr Gorski, LKML, Linux Kbuild mailing list, Masahiro Yamada, Michal Marek, Andrew Morton, Kees Cook, Nathan Chancellor, Daniel Borkmann, Valentin Schneider, Jan Alexander Steffens (heftig), David Howells, Johannes Weiner Hello. On Wed, Mar 31, 2021 at 07:21:07PM +0000, Nick Terrell wrote: > > > > On Mar 31, 2021, at 10:48 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote: > > > > Hello. > > > > On Wed, Mar 31, 2021 at 05:39:25PM +0000, Nick Terrell wrote: > >> > >> > >>> On Mar 30, 2021, at 4:50 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote: > >>> > >>> On Tue, Mar 30, 2021 at 01:32:35PM +0200, Piotr Gorski wrote: > >>>> kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. > >>>> > >>>> Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> > >>>> --- > >>>> Makefile | 7 +++++-- > >>>> init/Kconfig | 9 ++++++--- > >>>> 2 files changed, 11 insertions(+), 5 deletions(-) > >>>> > >>>> diff --git a/Makefile b/Makefile > >>>> index 5160ff8903c1..82f4f4cc2955 100644 > >>>> --- a/Makefile > >>>> +++ b/Makefile > >>>> @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP > >>>> export mod_strip_cmd > >>>> > >>>> # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed > >>>> -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP > >>>> -# or CONFIG_MODULE_COMPRESS_XZ. > >>>> +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP, > >>>> +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD. > >>>> > >>>> mod_compress_cmd = true > >>>> ifdef CONFIG_MODULE_COMPRESS > >>>> @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS > >>>> ifdef CONFIG_MODULE_COMPRESS_XZ > >>>> mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f > >>>> endif # CONFIG_MODULE_COMPRESS_XZ > >>>> + ifdef CONFIG_MODULE_COMPRESS_ZSTD > >>>> + mod_compress_cmd = $(ZSTD) -T0 --rm -f -q > >> > >> This will use the default zstd level, level 3. I think it would make more sense to use a high > >> compression level. Level 19 would probably be a good choice. That will choose a window > >> size of up to 8MB, meaning the decompressor needs to allocate that much memory. If that > >> is unacceptable, you could use `zstd -T0 --rm -f -q -19 --zstd=wlog=21`, which will use a > >> window size of up to 2MB, to match the XZ command. Note that if the file is smaller than > >> the window size, it will be shrunk to the smallest power of two at least as large as the file. > > > > Please no. We've already done that with initramfs in Arch, and it > > increased the time to generate it enormously. > > > > I understand that building a kernel is a more rare operation than > > regenerating initramfs, but still I'd go against hard-coding the level. > > And if it should be specified anyway, I'd opt in for an explicit > > configuration option. Remember, not all the kernel are built on > > build farms... > > > > FWIW, Piotr originally used level 9 which worked okay, but I insisted > > on sending the patch initially without specifying level at all like it is > > done for other compressors. If this is a wrong approach, then oh meh, > > mea culpa ;). > > > > Whatever default non-standard compression level you choose, I'm fine > > as long as I can change it without editing Makefile. > > That makes sense to me. I have a deep seated need to compress files as > efficiently as possible for widely distributed packages. But, I understand that > slow compression significantly impacts build times for quick iteration. I’d be > happy with a compression level parameter that defaults to a happy middle. > > I’m also fine with taking this patch as-is if it is easier, and I can put up another > patch that adds a compression level parameter, since I don’t want to block > merging this. Well, it seems Andrew already took this into his tree, so feel free to drop another one on top of that! > > Best, > Nick Terrell > > > Thanks! > > > >> > >> Best, > >> Nick Terrell > >> > >>>> + endif # CONFIG_MODULE_COMPRESS_ZSTD > >>>> endif # CONFIG_MODULE_COMPRESS > >>>> export mod_compress_cmd > >>>> > >>>> diff --git a/init/Kconfig b/init/Kconfig > >>>> index 8c2cfd88f6ef..86a452bc2747 100644 > >>>> --- a/init/Kconfig > >>>> +++ b/init/Kconfig > >>>> @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS > >>>> bool "Compress modules on installation" > >>>> help > >>>> > >>>> - Compresses kernel modules when 'make modules_install' is run; gzip or > >>>> - xz depending on "Compression algorithm" below. > >>>> + Compresses kernel modules when 'make modules_install' is run; gzip, > >>>> + xz, or zstd depending on "Compression algorithm" below. > >>>> > >>>> module-init-tools MAY support gzip, and kmod MAY support gzip and xz. > >>>> > >>>> @@ -2273,7 +2273,7 @@ choice > >>>> This determines which sort of compression will be used during > >>>> 'make modules_install'. > >>>> > >>>> - GZIP (default) and XZ are supported. > >>>> + GZIP (default), XZ, and ZSTD are supported. > >>>> > >>>> config MODULE_COMPRESS_GZIP > >>>> bool "GZIP" > >>>> @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP > >>>> config MODULE_COMPRESS_XZ > >>>> bool "XZ" > >>>> > >>>> +config MODULE_COMPRESS_ZSTD > >>>> + bool "ZSTD" > >>>> + > >>>> endchoice > >>>> > >>>> config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS > >>>> -- > >>>> 2.31.0.97.g1424303384 > >>>> > >>> > >>> Great! > >>> > >>> Reviewed-by: Oleksandr Natalenko <oleksandr@natalenko.name> > >>> > >>> This works perfectly fine in Arch Linux if accompanied by the > >>> following mkinitcpio amendment: [1]. > >>> > >>> I'm also Cc'ing other people from get_maintainers output just > >>> to make this submission more visible. > >>> > >>> Thanks. > >>> > >>> [1] https://github.com/archlinux/mkinitcpio/pull/43 > >>> > >>> -- > >>> Oleksandr Natalenko (post-factum) > >> > > > > -- > > Oleksandr Natalenko (post-factum) > -- Oleksandr Natalenko (post-factum) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] init: add support for zstd compressed modules 2021-03-31 19:21 ` Nick Terrell 2021-03-31 20:28 ` Oleksandr Natalenko @ 2021-04-07 13:53 ` Masahiro Yamada 2021-04-07 17:45 ` Nick Terrell 1 sibling, 1 reply; 12+ messages in thread From: Masahiro Yamada @ 2021-04-07 13:53 UTC (permalink / raw) To: Nick Terrell Cc: Oleksandr Natalenko, Piotr Gorski, LKML, Linux Kbuild mailing list, Michal Marek, Andrew Morton, Kees Cook, Nathan Chancellor, Daniel Borkmann, Valentin Schneider, Jan Alexander Steffens (heftig), David Howells, Johannes Weiner On Thu, Apr 1, 2021 at 4:21 AM Nick Terrell <terrelln@fb.com> wrote: > > > > > On Mar 31, 2021, at 10:48 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote: > > > > Hello. > > > > On Wed, Mar 31, 2021 at 05:39:25PM +0000, Nick Terrell wrote: > >> > >> > >>> On Mar 30, 2021, at 4:50 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote: > >>> > >>> On Tue, Mar 30, 2021 at 01:32:35PM +0200, Piotr Gorski wrote: > >>>> kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. > >>>> > >>>> Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> > >>>> --- > >>>> Makefile | 7 +++++-- > >>>> init/Kconfig | 9 ++++++--- > >>>> 2 files changed, 11 insertions(+), 5 deletions(-) > >>>> > >>>> diff --git a/Makefile b/Makefile > >>>> index 5160ff8903c1..82f4f4cc2955 100644 > >>>> --- a/Makefile > >>>> +++ b/Makefile > >>>> @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP > >>>> export mod_strip_cmd > >>>> > >>>> # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed > >>>> -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP > >>>> -# or CONFIG_MODULE_COMPRESS_XZ. > >>>> +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP, > >>>> +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD. > >>>> > >>>> mod_compress_cmd = true > >>>> ifdef CONFIG_MODULE_COMPRESS > >>>> @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS > >>>> ifdef CONFIG_MODULE_COMPRESS_XZ > >>>> mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f > >>>> endif # CONFIG_MODULE_COMPRESS_XZ > >>>> + ifdef CONFIG_MODULE_COMPRESS_ZSTD > >>>> + mod_compress_cmd = $(ZSTD) -T0 --rm -f -q > >> > >> This will use the default zstd level, level 3. I think it would make more sense to use a high > >> compression level. Level 19 would probably be a good choice. That will choose a window > >> size of up to 8MB, meaning the decompressor needs to allocate that much memory. If that > >> is unacceptable, you could use `zstd -T0 --rm -f -q -19 --zstd=wlog=21`, which will use a > >> window size of up to 2MB, to match the XZ command. Note that if the file is smaller than > >> the window size, it will be shrunk to the smallest power of two at least as large as the file. > > > > Please no. We've already done that with initramfs in Arch, and it > > increased the time to generate it enormously. > > > > I understand that building a kernel is a more rare operation than > > regenerating initramfs, but still I'd go against hard-coding the level. > > And if it should be specified anyway, I'd opt in for an explicit > > configuration option. Remember, not all the kernel are built on > > build farms... > > > > FWIW, Piotr originally used level 9 which worked okay, but I insisted > > on sending the patch initially without specifying level at all like it is > > done for other compressors. If this is a wrong approach, then oh meh, > > mea culpa ;). > > > > Whatever default non-standard compression level you choose, I'm fine > > as long as I can change it without editing Makefile. > > That makes sense to me. I have a deep seated need to compress files as > efficiently as possible for widely distributed packages. But, I understand that > slow compression significantly impacts build times for quick iteration. I’d be > happy with a compression level parameter that defaults to a happy middle. > > I’m also fine with taking this patch as-is if it is easier, and I can put up another > patch that adds a compression level parameter, since I don’t want to block > merging this. I do not want to take such a patch. Meeking everyone's requirement results in a bad project for everyone. Does this work for you? make modules_install ZSTD="zstd -19" > Best, > Nick Terrell > > > Thanks! > > > >> > >> Best, > >> Nick Terrell > >> > >>>> + endif # CONFIG_MODULE_COMPRESS_ZSTD > >>>> endif # CONFIG_MODULE_COMPRESS > >>>> export mod_compress_cmd > >>>> > >>>> diff --git a/init/Kconfig b/init/Kconfig > >>>> index 8c2cfd88f6ef..86a452bc2747 100644 > >>>> --- a/init/Kconfig > >>>> +++ b/init/Kconfig > >>>> @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS > >>>> bool "Compress modules on installation" > >>>> help > >>>> > >>>> - Compresses kernel modules when 'make modules_install' is run; gzip or > >>>> - xz depending on "Compression algorithm" below. > >>>> + Compresses kernel modules when 'make modules_install' is run; gzip, > >>>> + xz, or zstd depending on "Compression algorithm" below. > >>>> > >>>> module-init-tools MAY support gzip, and kmod MAY support gzip and xz. > >>>> > >>>> @@ -2273,7 +2273,7 @@ choice > >>>> This determines which sort of compression will be used during > >>>> 'make modules_install'. > >>>> > >>>> - GZIP (default) and XZ are supported. > >>>> + GZIP (default), XZ, and ZSTD are supported. > >>>> > >>>> config MODULE_COMPRESS_GZIP > >>>> bool "GZIP" > >>>> @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP > >>>> config MODULE_COMPRESS_XZ > >>>> bool "XZ" > >>>> > >>>> +config MODULE_COMPRESS_ZSTD > >>>> + bool "ZSTD" > >>>> + > >>>> endchoice > >>>> > >>>> config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS > >>>> -- > >>>> 2.31.0.97.g1424303384 > >>>> > >>> > >>> Great! > >>> > >>> Reviewed-by: Oleksandr Natalenko <oleksandr@natalenko.name> > >>> > >>> This works perfectly fine in Arch Linux if accompanied by the > >>> following mkinitcpio amendment: [1]. > >>> > >>> I'm also Cc'ing other people from get_maintainers output just > >>> to make this submission more visible. > >>> > >>> Thanks. > >>> > >>> [1] https://github.com/archlinux/mkinitcpio/pull/43 > >>> > >>> -- > >>> Oleksandr Natalenko (post-factum) > >> > > > > -- > > Oleksandr Natalenko (post-factum) > -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] init: add support for zstd compressed modules 2021-04-07 13:53 ` Masahiro Yamada @ 2021-04-07 17:45 ` Nick Terrell 0 siblings, 0 replies; 12+ messages in thread From: Nick Terrell @ 2021-04-07 17:45 UTC (permalink / raw) To: Masahiro Yamada Cc: Oleksandr Natalenko, Piotr Gorski, LKML, Linux Kbuild mailing list, Michal Marek, Andrew Morton, Kees Cook, Nathan Chancellor, Daniel Borkmann, Valentin Schneider, Jan Alexander Steffens (heftig), David Howells, Johannes Weiner > On Apr 7, 2021, at 6:53 AM, Masahiro Yamada <masahiroy@kernel.org> wrote: > > On Thu, Apr 1, 2021 at 4:21 AM Nick Terrell <terrelln@fb.com> wrote: >> >> >> >>> On Mar 31, 2021, at 10:48 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote: >>> >>> Hello. >>> >>> On Wed, Mar 31, 2021 at 05:39:25PM +0000, Nick Terrell wrote: >>>> >>>> >>>>> On Mar 30, 2021, at 4:50 AM, Oleksandr Natalenko <oleksandr@natalenko.name> wrote: >>>>> >>>>> On Tue, Mar 30, 2021 at 01:32:35PM +0200, Piotr Gorski wrote: >>>>>> kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. >>>>>> >>>>>> Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> >>>>>> --- >>>>>> Makefile | 7 +++++-- >>>>>> init/Kconfig | 9 ++++++--- >>>>>> 2 files changed, 11 insertions(+), 5 deletions(-) >>>>>> >>>>>> diff --git a/Makefile b/Makefile >>>>>> index 5160ff8903c1..82f4f4cc2955 100644 >>>>>> --- a/Makefile >>>>>> +++ b/Makefile >>>>>> @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP >>>>>> export mod_strip_cmd >>>>>> >>>>>> # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed >>>>>> -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP >>>>>> -# or CONFIG_MODULE_COMPRESS_XZ. >>>>>> +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP, >>>>>> +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD. >>>>>> >>>>>> mod_compress_cmd = true >>>>>> ifdef CONFIG_MODULE_COMPRESS >>>>>> @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS >>>>>> ifdef CONFIG_MODULE_COMPRESS_XZ >>>>>> mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f >>>>>> endif # CONFIG_MODULE_COMPRESS_XZ >>>>>> + ifdef CONFIG_MODULE_COMPRESS_ZSTD >>>>>> + mod_compress_cmd = $(ZSTD) -T0 --rm -f -q >>>> >>>> This will use the default zstd level, level 3. I think it would make more sense to use a high >>>> compression level. Level 19 would probably be a good choice. That will choose a window >>>> size of up to 8MB, meaning the decompressor needs to allocate that much memory. If that >>>> is unacceptable, you could use `zstd -T0 --rm -f -q -19 --zstd=wlog=21`, which will use a >>>> window size of up to 2MB, to match the XZ command. Note that if the file is smaller than >>>> the window size, it will be shrunk to the smallest power of two at least as large as the file. >>> >>> Please no. We've already done that with initramfs in Arch, and it >>> increased the time to generate it enormously. >>> >>> I understand that building a kernel is a more rare operation than >>> regenerating initramfs, but still I'd go against hard-coding the level. >>> And if it should be specified anyway, I'd opt in for an explicit >>> configuration option. Remember, not all the kernel are built on >>> build farms... >>> >>> FWIW, Piotr originally used level 9 which worked okay, but I insisted >>> on sending the patch initially without specifying level at all like it is >>> done for other compressors. If this is a wrong approach, then oh meh, >>> mea culpa ;). >>> >>> Whatever default non-standard compression level you choose, I'm fine >>> as long as I can change it without editing Makefile. >> >> That makes sense to me. I have a deep seated need to compress files as >> efficiently as possible for widely distributed packages. But, I understand that >> slow compression significantly impacts build times for quick iteration. I’d be >> happy with a compression level parameter that defaults to a happy middle. >> >> I’m also fine with taking this patch as-is if it is easier, and I can put up another >> patch that adds a compression level parameter, since I don’t want to block >> merging this. > > > I do not want to take such a patch. > Meeking everyone's requirement > results in a bad project for everyone. > > > Does this work for you? > > make modules_install ZSTD="zstd -19" Yeah, that’s perfect. Do you think it is worth mentioning in the docs for `MODULE_COMPRESS_ZSTD`? Best, Nick >> Best, >> Nick Terrell >> >>> Thanks! >>> >>>> >>>> Best, >>>> Nick Terrell >>>> >>>>>> + endif # CONFIG_MODULE_COMPRESS_ZSTD >>>>>> endif # CONFIG_MODULE_COMPRESS >>>>>> export mod_compress_cmd >>>>>> >>>>>> diff --git a/init/Kconfig b/init/Kconfig >>>>>> index 8c2cfd88f6ef..86a452bc2747 100644 >>>>>> --- a/init/Kconfig >>>>>> +++ b/init/Kconfig >>>>>> @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS >>>>>> bool "Compress modules on installation" >>>>>> help >>>>>> >>>>>> - Compresses kernel modules when 'make modules_install' is run; gzip or >>>>>> - xz depending on "Compression algorithm" below. >>>>>> + Compresses kernel modules when 'make modules_install' is run; gzip, >>>>>> + xz, or zstd depending on "Compression algorithm" below. >>>>>> >>>>>> module-init-tools MAY support gzip, and kmod MAY support gzip and xz. >>>>>> >>>>>> @@ -2273,7 +2273,7 @@ choice >>>>>> This determines which sort of compression will be used during >>>>>> 'make modules_install'. >>>>>> >>>>>> - GZIP (default) and XZ are supported. >>>>>> + GZIP (default), XZ, and ZSTD are supported. >>>>>> >>>>>> config MODULE_COMPRESS_GZIP >>>>>> bool "GZIP" >>>>>> @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP >>>>>> config MODULE_COMPRESS_XZ >>>>>> bool "XZ" >>>>>> >>>>>> +config MODULE_COMPRESS_ZSTD >>>>>> + bool "ZSTD" >>>>>> + >>>>>> endchoice >>>>>> >>>>>> config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS >>>>>> -- >>>>>> 2.31.0.97.g1424303384 >>>>>> >>>>> >>>>> Great! >>>>> >>>>> Reviewed-by: Oleksandr Natalenko <oleksandr@natalenko.name> >>>>> >>>>> This works perfectly fine in Arch Linux if accompanied by the >>>>> following mkinitcpio amendment: [1]. >>>>> >>>>> I'm also Cc'ing other people from get_maintainers output just >>>>> to make this submission more visible. >>>>> >>>>> Thanks. >>>>> >>>>> [1] https://github.com/archlinux/mkinitcpio/pull/43 >>>>> >>>>> -- >>>>> Oleksandr Natalenko (post-factum) >>>> >>> >>> -- >>> Oleksandr Natalenko (post-factum) >> > > > -- > Best Regards > Masahiro Yamada ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] init: add support for zstd compressed modules 2021-03-30 11:32 Piotr Gorski 2021-03-30 11:50 ` Oleksandr Natalenko @ 2021-04-07 14:32 ` Masahiro Yamada 1 sibling, 0 replies; 12+ messages in thread From: Masahiro Yamada @ 2021-04-07 14:32 UTC (permalink / raw) To: Piotr Gorski Cc: Linux Kernel Mailing List, Linux Kbuild mailing list, Oleksandr Natalenko On Tue, Mar 30, 2021 at 8:33 PM Piotr Gorski <lucjan.lucjanov@gmail.com> wrote: > > kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel. > > Signed-off-by: Piotr Gorski <lucjan.lucjanov@gmail.com> > --- > Makefile | 7 +++++-- > init/Kconfig | 9 ++++++--- > 2 files changed, 11 insertions(+), 5 deletions(-) Piort, sorry for bothering you, but could you rebase on top of this clean-up patch set? https://patchwork.kernel.org/project/linux-kbuild/list/?series=458809 or git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git kbuild The module compression code should not be placed in the top Makefile. > diff --git a/Makefile b/Makefile > index 5160ff8903c1..82f4f4cc2955 100644 > --- a/Makefile > +++ b/Makefile > @@ -1156,8 +1156,8 @@ endif # INSTALL_MOD_STRIP > export mod_strip_cmd > > # CONFIG_MODULE_COMPRESS, if defined, will cause module to be compressed > -# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP > -# or CONFIG_MODULE_COMPRESS_XZ. > +# after they are installed in agreement with CONFIG_MODULE_COMPRESS_GZIP, > +# CONFIG_MODULE_COMPRESS_XZ, or CONFIG_MODULE_COMPRESS_ZSTD. > > mod_compress_cmd = true > ifdef CONFIG_MODULE_COMPRESS > @@ -1167,6 +1167,9 @@ ifdef CONFIG_MODULE_COMPRESS > ifdef CONFIG_MODULE_COMPRESS_XZ > mod_compress_cmd = $(XZ) --lzma2=dict=2MiB -f > endif # CONFIG_MODULE_COMPRESS_XZ > + ifdef CONFIG_MODULE_COMPRESS_ZSTD > + mod_compress_cmd = $(ZSTD) -T0 --rm -f -q > + endif # CONFIG_MODULE_COMPRESS_ZSTD > endif # CONFIG_MODULE_COMPRESS > export mod_compress_cmd > > diff --git a/init/Kconfig b/init/Kconfig > index 8c2cfd88f6ef..86a452bc2747 100644 > --- a/init/Kconfig > +++ b/init/Kconfig > @@ -2250,8 +2250,8 @@ config MODULE_COMPRESS > bool "Compress modules on installation" > help > > - Compresses kernel modules when 'make modules_install' is run; gzip or > - xz depending on "Compression algorithm" below. > + Compresses kernel modules when 'make modules_install' is run; gzip, > + xz, or zstd depending on "Compression algorithm" below. > > module-init-tools MAY support gzip, and kmod MAY support gzip and xz. > > @@ -2273,7 +2273,7 @@ choice > This determines which sort of compression will be used during > 'make modules_install'. > > - GZIP (default) and XZ are supported. > + GZIP (default), XZ, and ZSTD are supported. > > config MODULE_COMPRESS_GZIP > bool "GZIP" > @@ -2281,6 +2281,9 @@ config MODULE_COMPRESS_GZIP > config MODULE_COMPRESS_XZ > bool "XZ" > > +config MODULE_COMPRESS_ZSTD > + bool "ZSTD" > + > endchoice > > config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS > -- > 2.31.0.97.g1424303384 > -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <722602000.5829.1617263646814@office.mailbox.org>]
* Re: [PATCH] init: add support for zstd compressed modules [not found] <722602000.5829.1617263646814@office.mailbox.org> @ 2021-04-01 8:01 ` torvic9 2021-04-01 19:18 ` Nick Terrell 1 sibling, 0 replies; 12+ messages in thread From: torvic9 @ 2021-04-01 8:01 UTC (permalink / raw) To: Piotr Górski Cc: linux-kernel, linux-kbuild, masahiroy, oleksandr, terrelln, michal.lkml Thanks Piotr, good work! Question: Is `-T0` really faster in this particular case than the default `-T1`? Are modules installed sequentially? I also saw that Masahiro did some work on modules_install, moving MODULE_COMPRESS from the base Makefile to scripts/Makefile.modinst, so perhaps this should also be moved there at a later point. (sorry for double post) Tor Vic ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] init: add support for zstd compressed modules [not found] <722602000.5829.1617263646814@office.mailbox.org> 2021-04-01 8:01 ` torvic9 @ 2021-04-01 19:18 ` Nick Terrell 1 sibling, 0 replies; 12+ messages in thread From: Nick Terrell @ 2021-04-01 19:18 UTC (permalink / raw) To: torvic9 Cc: Piotr Górski, linux-kernel, linux-kbuild, masahiroy, oleksandr, michal.lkml > On Apr 1, 2021, at 12:54 AM, torvic9@mailbox.org wrote: > > Thanks Piotr, good work! > Question: Is `-T0` really faster in this particular case than the default `-T1`? Are modules installed sequentially? The zstd CLI produces deterministic output regardless of the number of threads used. `-T1` (or not specifying `-T`) will produce the same output as `-T0`. `-T0` will be faster for large files (at the default level, multiple jobs will be spawned for files > 8MB), and be just as fast as `-T1` for smaller files. Best, Nick > I also saw that Masahiro did some work on modules_install, moving MODULE_COMPRESS from the base Makefile to scripts/Makefile.modinst, so perhaps this should also be moved there at a later point. > > Tor Vic ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2021-04-07 17:46 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-13 14:50 [PATCH] init: add support for zstd compressed modules >
2021-03-30 11:32 Piotr Gorski
2021-03-30 11:50 ` Oleksandr Natalenko
2021-03-31 17:39 ` Nick Terrell
2021-03-31 17:48 ` Oleksandr Natalenko
2021-03-31 19:21 ` Nick Terrell
2021-03-31 20:28 ` Oleksandr Natalenko
2021-04-07 13:53 ` Masahiro Yamada
2021-04-07 17:45 ` Nick Terrell
2021-04-07 14:32 ` Masahiro Yamada
[not found] <722602000.5829.1617263646814@office.mailbox.org>
2021-04-01 8:01 ` torvic9
2021-04-01 19:18 ` Nick Terrell
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.