* [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support
@ 2021-01-27 15:49 Takashi Iwai
2021-01-27 15:49 ` [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files Takashi Iwai
` (4 more replies)
0 siblings, 5 replies; 19+ messages in thread
From: Takashi Iwai @ 2021-01-27 15:49 UTC (permalink / raw)
To: Luis Chamberlain; +Cc: Greg Kroah-Hartman, Rafael J . Wysocki, linux-kernel
Hi,
it seems that ZSTD format is getting popular, and I've been asked
about the firmware loader support. So I took a quick glance, and it
turned out that it's fairly easy thanks to the existing ZSTD API.
Now high time to submit something.
The first patch adds a new Kconfig CONFIG_FW_LOADER_COMPRESS_ZSTD and
the corresponding decompression function to the firmware loader code.
For the already supported XZ-compression, CONFIG_FW_LOADER_COMPRESS_XZ
is added to make it selectable explicitly, too.
The rest three patches are for selftest: a cleanup, a fix and the
additional support of ZSTD format.
Currently, I have no idea whether any distro would use ZSTD files for
firmware files in near future, though. That's the reason of this
patch set being an RFC for now.
thanks,
Takashi
===
Takashi Iwai (4):
firmware: Add the support for ZSTD-compressed firmware files
selftests: firmware: Simplify test patterns
selftest: firmware: Fix the request_firmware_into_buf() test for XZ
format
selftest: firmware: Add ZSTD compressed file tests
drivers/base/firmware_loader/Kconfig | 21 ++-
drivers/base/firmware_loader/main.c | 74 +++++++-
.../selftests/firmware/fw_filesystem.sh | 167 +++++++++---------
tools/testing/selftests/firmware/fw_lib.sh | 12 +-
4 files changed, 175 insertions(+), 99 deletions(-)
--
2.26.2
^ permalink raw reply [flat|nested] 19+ messages in thread* [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files 2021-01-27 15:49 [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support Takashi Iwai @ 2021-01-27 15:49 ` Takashi Iwai 2021-02-17 13:16 ` Luis Chamberlain 2021-02-17 13:24 ` Luis Chamberlain 2021-01-27 15:49 ` [PATCH RFC 2/4] selftests: firmware: Simplify test patterns Takashi Iwai ` (3 subsequent siblings) 4 siblings, 2 replies; 19+ messages in thread From: Takashi Iwai @ 2021-01-27 15:49 UTC (permalink / raw) To: Luis Chamberlain; +Cc: Greg Kroah-Hartman, Rafael J . Wysocki, linux-kernel Due to the popular demands on ZSTD, here is a patch to add a support of ZSTD-compressed firmware files via the direct firmware loader. It's just like XZ-compressed file support, providing a decompressor with ZSTD. Since ZSTD API can give the decompression size beforehand, the code is even simpler than XZ. Signed-off-by: Takashi Iwai <tiwai@suse.de> --- drivers/base/firmware_loader/Kconfig | 21 ++++++-- drivers/base/firmware_loader/main.c | 74 ++++++++++++++++++++++++++-- 2 files changed, 87 insertions(+), 8 deletions(-) diff --git a/drivers/base/firmware_loader/Kconfig b/drivers/base/firmware_loader/Kconfig index 5b24f3959255..f5307978927c 100644 --- a/drivers/base/firmware_loader/Kconfig +++ b/drivers/base/firmware_loader/Kconfig @@ -157,17 +157,28 @@ config FW_LOADER_USER_HELPER_FALLBACK config FW_LOADER_COMPRESS bool "Enable compressed firmware support" - select FW_LOADER_PAGED_BUF - select XZ_DEC help This option enables the support for loading compressed firmware files. The caller of firmware API receives the decompressed file content. The compressed file is loaded as a fallback, only after loading the raw file failed at first. - Currently only XZ-compressed files are supported, and they have to - be compressed with either none or crc32 integrity check type (pass - "-C crc32" option to xz command). +if FW_LOADER_COMPRESS +config FW_LOADER_COMPRESS_XZ + bool "Enable XZ-compressed firmware support" + select FW_LOADER_PAGED_BUF + select XZ_DEC + help + This option adds the support for XZ-compressed files. + The files have to be compressed with either none or crc32 + integrity check type (pass "-C crc32" option to xz command). + +config FW_LOADER_COMPRESS_ZSTD + bool "Enable ZSTD-compressed firmware support" + select ZSTD_DECOMPRESS + help + This option adds the support for ZSTD-compressed files. +endif # FW_LOADER_COMPRESS config FW_CACHE bool "Enable firmware caching during suspend" diff --git a/drivers/base/firmware_loader/main.c b/drivers/base/firmware_loader/main.c index 78355095e00d..71332ed4959d 100644 --- a/drivers/base/firmware_loader/main.c +++ b/drivers/base/firmware_loader/main.c @@ -34,6 +34,7 @@ #include <linux/syscore_ops.h> #include <linux/reboot.h> #include <linux/security.h> +#include <linux/zstd.h> #include <linux/xz.h> #include <generated/utsrelease.h> @@ -362,10 +363,72 @@ int fw_map_paged_buf(struct fw_priv *fw_priv) } #endif +/* + * ZSTD-compressed firmware support + */ +#ifdef CONFIG_FW_LOADER_COMPRESS_ZSTD +static int fw_decompress_zstd(struct device *dev, struct fw_priv *fw_priv, + size_t in_size, const void *in_buffer) +{ + size_t len, out_size, workspace_size; + void *workspace, *out_buf; + ZSTD_DCtx *ctx; + int err; + + if (fw_priv->data) { + out_size = fw_priv->allocated_size; + out_buf = fw_priv->data; + } else { + out_size = ZSTD_findDecompressedSize(in_buffer, in_size); + if (out_size == ZSTD_CONTENTSIZE_UNKNOWN || + out_size == ZSTD_CONTENTSIZE_ERROR) { + dev_dbg(dev, "%s: invalid decompression size\n", __func__); + return -EINVAL; + } + out_buf = vzalloc(out_size); + if (!out_buf) + return -ENOMEM; + } + + workspace_size = ZSTD_DCtxWorkspaceBound(); + workspace = kvzalloc(workspace_size, GFP_KERNEL); + if (!workspace) { + err = -ENOMEM; + goto error; + } + + ctx = ZSTD_initDCtx(workspace, workspace_size); + if (!ctx) { + dev_dbg(dev, "%s: failed to initialize context\n", __func__); + err = -EINVAL; + goto error; + } + + len = ZSTD_decompressDCtx(ctx, out_buf, out_size, in_buffer, in_size); + if (ZSTD_isError(len)) { + dev_dbg(dev, "%s: failed to decompress: %d\n", __func__, + ZSTD_getErrorCode(len)); + err = -EINVAL; + goto error; + } + + fw_priv->size = len; + if (!fw_priv->data) + fw_priv->data = out_buf; + err = 0; + + error: + kvfree(workspace); + if (!fw_priv->data) + vfree(out_buf); + return err; +} +#endif /* CONFIG_FW_LOADER_COMPRESS_ZSTD */ + /* * XZ-compressed firmware support */ -#ifdef CONFIG_FW_LOADER_COMPRESS +#ifdef CONFIG_FW_LOADER_COMPRESS_XZ /* show an error and return the standard error code */ static int fw_decompress_xz_error(struct device *dev, enum xz_ret xz_ret) { @@ -459,7 +522,7 @@ static int fw_decompress_xz(struct device *dev, struct fw_priv *fw_priv, else return fw_decompress_xz_pages(dev, fw_priv, in_size, in_buffer); } -#endif /* CONFIG_FW_LOADER_COMPRESS */ +#endif /* CONFIG_FW_LOADER_COMPRESS_XZ */ /* direct firmware loading support */ static char fw_path_para[256]; @@ -814,7 +877,12 @@ _request_firmware(const struct firmware **firmware_p, const char *name, if (!(opt_flags & FW_OPT_PARTIAL)) nondirect = true; -#ifdef CONFIG_FW_LOADER_COMPRESS +#ifdef CONFIG_FW_LOADER_COMPRESS_ZSTD + if (ret == -ENOENT && nondirect) + ret = fw_get_filesystem_firmware(device, fw->priv, ".zst", + fw_decompress_zstd); +#endif +#ifdef CONFIG_FW_LOADER_COMPRESS_XZ if (ret == -ENOENT && nondirect) ret = fw_get_filesystem_firmware(device, fw->priv, ".xz", fw_decompress_xz); -- 2.26.2 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files 2021-01-27 15:49 ` [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files Takashi Iwai @ 2021-02-17 13:16 ` Luis Chamberlain 2021-02-17 13:22 ` Takashi Iwai 2021-02-17 13:24 ` Luis Chamberlain 1 sibling, 1 reply; 19+ messages in thread From: Luis Chamberlain @ 2021-02-17 13:16 UTC (permalink / raw) To: Takashi Iwai; +Cc: Greg Kroah-Hartman, Rafael J . Wysocki, linux-kernel On Wed, Jan 27, 2021 at 04:49:36PM +0100, Takashi Iwai wrote: > Due to the popular demands on ZSTD, here is a patch to add a support > of ZSTD-compressed firmware files via the direct firmware loader. > It's just like XZ-compressed file support, providing a decompressor > with ZSTD. Since ZSTD API can give the decompression size beforehand, > the code is even simpler than XZ. > > Signed-off-by: Takashi Iwai <tiwai@suse.de> > --- > drivers/base/firmware_loader/Kconfig | 21 ++++++-- > drivers/base/firmware_loader/main.c | 74 ++++++++++++++++++++++++++-- > 2 files changed, 87 insertions(+), 8 deletions(-) > > diff --git a/drivers/base/firmware_loader/Kconfig b/drivers/base/firmware_loader/Kconfig > index 5b24f3959255..f5307978927c 100644 > --- a/drivers/base/firmware_loader/Kconfig > +++ b/drivers/base/firmware_loader/Kconfig > @@ -157,17 +157,28 @@ config FW_LOADER_USER_HELPER_FALLBACK > > config FW_LOADER_COMPRESS > bool "Enable compressed firmware support" > - select FW_LOADER_PAGED_BUF > - select XZ_DEC > help > This option enables the support for loading compressed firmware > files. The caller of firmware API receives the decompressed file > content. The compressed file is loaded as a fallback, only after > loading the raw file failed at first. > > - Currently only XZ-compressed files are supported, and they have to > - be compressed with either none or crc32 integrity check type (pass > - "-C crc32" option to xz command). > +if FW_LOADER_COMPRESS > +config FW_LOADER_COMPRESS_XZ > + bool "Enable XZ-compressed firmware support" > + select FW_LOADER_PAGED_BUF > + select XZ_DEC > + help > + This option adds the support for XZ-compressed files. > + The files have to be compressed with either none or crc32 > + integrity check type (pass "-C crc32" option to xz command). > + > +config FW_LOADER_COMPRESS_ZSTD > + bool "Enable ZSTD-compressed firmware support" > + select ZSTD_DECOMPRESS > + help > + This option adds the support for ZSTD-compressed files. > +endif # FW_LOADER_COMPRESS > > config FW_CACHE > bool "Enable firmware caching during suspend" > diff --git a/drivers/base/firmware_loader/main.c b/drivers/base/firmware_loader/main.c > index 78355095e00d..71332ed4959d 100644 > --- a/drivers/base/firmware_loader/main.c > +++ b/drivers/base/firmware_loader/main.c > @@ -34,6 +34,7 @@ > #include <linux/syscore_ops.h> > #include <linux/reboot.h> > #include <linux/security.h> > +#include <linux/zstd.h> > #include <linux/xz.h> > > #include <generated/utsrelease.h> > @@ -362,10 +363,72 @@ int fw_map_paged_buf(struct fw_priv *fw_priv) > } > #endif > > +/* > + * ZSTD-compressed firmware support > + */ > +#ifdef CONFIG_FW_LOADER_COMPRESS_ZSTD > +static int fw_decompress_zstd(struct device *dev, struct fw_priv *fw_priv, > + size_t in_size, const void *in_buffer) > +{ > + size_t len, out_size, workspace_size; > + void *workspace, *out_buf; > + ZSTD_DCtx *ctx; > + int err; > + > + if (fw_priv->data) { > + out_size = fw_priv->allocated_size; > + out_buf = fw_priv->data; > + } else { > + out_size = ZSTD_findDecompressedSize(in_buffer, in_size); > + if (out_size == ZSTD_CONTENTSIZE_UNKNOWN || > + out_size == ZSTD_CONTENTSIZE_ERROR) { > + dev_dbg(dev, "%s: invalid decompression size\n", __func__); > + return -EINVAL; > + } > + out_buf = vzalloc(out_size); > + if (!out_buf) > + return -ENOMEM; > + } > + > + workspace_size = ZSTD_DCtxWorkspaceBound(); > + workspace = kvzalloc(workspace_size, GFP_KERNEL); > + if (!workspace) { > + err = -ENOMEM; > + goto error; > + } > + > + ctx = ZSTD_initDCtx(workspace, workspace_size); > + if (!ctx) { > + dev_dbg(dev, "%s: failed to initialize context\n", __func__); > + err = -EINVAL; > + goto error; > + } > + > + len = ZSTD_decompressDCtx(ctx, out_buf, out_size, in_buffer, in_size); > + if (ZSTD_isError(len)) { > + dev_dbg(dev, "%s: failed to decompress: %d\n", __func__, > + ZSTD_getErrorCode(len)); > + err = -EINVAL; > + goto error; > + } > + > + fw_priv->size = len; > + if (!fw_priv->data) > + fw_priv->data = out_buf; > + err = 0; > + > + error: > + kvfree(workspace); > + if (!fw_priv->data) > + vfree(out_buf); > + return err; > +} > +#endif /* CONFIG_FW_LOADER_COMPRESS_ZSTD */ > + > /* > * XZ-compressed firmware support > */ > -#ifdef CONFIG_FW_LOADER_COMPRESS > +#ifdef CONFIG_FW_LOADER_COMPRESS_XZ > /* show an error and return the standard error code */ > static int fw_decompress_xz_error(struct device *dev, enum xz_ret xz_ret) > { > @@ -459,7 +522,7 @@ static int fw_decompress_xz(struct device *dev, struct fw_priv *fw_priv, > else > return fw_decompress_xz_pages(dev, fw_priv, in_size, in_buffer); > } > -#endif /* CONFIG_FW_LOADER_COMPRESS */ > +#endif /* CONFIG_FW_LOADER_COMPRESS_XZ */ > > /* direct firmware loading support */ > static char fw_path_para[256]; > @@ -814,7 +877,12 @@ _request_firmware(const struct firmware **firmware_p, const char *name, > if (!(opt_flags & FW_OPT_PARTIAL)) > nondirect = true; > > -#ifdef CONFIG_FW_LOADER_COMPRESS > +#ifdef CONFIG_FW_LOADER_COMPRESS_ZSTD > + if (ret == -ENOENT && nondirect) > + ret = fw_get_filesystem_firmware(device, fw->priv, ".zst", > + fw_decompress_zstd); > +#endif > +#ifdef CONFIG_FW_LOADER_COMPRESS_XZ > if (ret == -ENOENT && nondirect) > ret = fw_get_filesystem_firmware(device, fw->priv, ".xz", > fw_decompress_xz); Since this is growing, it would be nicer if we avoid the #ifdef mess and simply have code iterate over supported compressions, and this is observed here in one line. Luis ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files 2021-02-17 13:16 ` Luis Chamberlain @ 2021-02-17 13:22 ` Takashi Iwai 0 siblings, 0 replies; 19+ messages in thread From: Takashi Iwai @ 2021-02-17 13:22 UTC (permalink / raw) To: Luis Chamberlain; +Cc: Greg Kroah-Hartman, Rafael J . Wysocki, linux-kernel On Wed, 17 Feb 2021 14:16:44 +0100, Luis Chamberlain wrote: > > On Wed, Jan 27, 2021 at 04:49:36PM +0100, Takashi Iwai wrote: > > Due to the popular demands on ZSTD, here is a patch to add a support > > of ZSTD-compressed firmware files via the direct firmware loader. > > It's just like XZ-compressed file support, providing a decompressor > > with ZSTD. Since ZSTD API can give the decompression size beforehand, > > the code is even simpler than XZ. > > > > Signed-off-by: Takashi Iwai <tiwai@suse.de> > > --- > > drivers/base/firmware_loader/Kconfig | 21 ++++++-- > > drivers/base/firmware_loader/main.c | 74 ++++++++++++++++++++++++++-- > > 2 files changed, 87 insertions(+), 8 deletions(-) > > > > diff --git a/drivers/base/firmware_loader/Kconfig b/drivers/base/firmware_loader/Kconfig > > index 5b24f3959255..f5307978927c 100644 > > --- a/drivers/base/firmware_loader/Kconfig > > +++ b/drivers/base/firmware_loader/Kconfig > > @@ -157,17 +157,28 @@ config FW_LOADER_USER_HELPER_FALLBACK > > > > config FW_LOADER_COMPRESS > > bool "Enable compressed firmware support" > > - select FW_LOADER_PAGED_BUF > > - select XZ_DEC > > help > > This option enables the support for loading compressed firmware > > files. The caller of firmware API receives the decompressed file > > content. The compressed file is loaded as a fallback, only after > > loading the raw file failed at first. > > > > - Currently only XZ-compressed files are supported, and they have to > > - be compressed with either none or crc32 integrity check type (pass > > - "-C crc32" option to xz command). > > +if FW_LOADER_COMPRESS > > +config FW_LOADER_COMPRESS_XZ > > + bool "Enable XZ-compressed firmware support" > > + select FW_LOADER_PAGED_BUF > > + select XZ_DEC > > + help > > + This option adds the support for XZ-compressed files. > > + The files have to be compressed with either none or crc32 > > + integrity check type (pass "-C crc32" option to xz command). > > + > > +config FW_LOADER_COMPRESS_ZSTD > > + bool "Enable ZSTD-compressed firmware support" > > + select ZSTD_DECOMPRESS > > + help > > + This option adds the support for ZSTD-compressed files. > > +endif # FW_LOADER_COMPRESS > > > > config FW_CACHE > > bool "Enable firmware caching during suspend" > > diff --git a/drivers/base/firmware_loader/main.c b/drivers/base/firmware_loader/main.c > > index 78355095e00d..71332ed4959d 100644 > > --- a/drivers/base/firmware_loader/main.c > > +++ b/drivers/base/firmware_loader/main.c > > @@ -34,6 +34,7 @@ > > #include <linux/syscore_ops.h> > > #include <linux/reboot.h> > > #include <linux/security.h> > > +#include <linux/zstd.h> > > #include <linux/xz.h> > > > > #include <generated/utsrelease.h> > > @@ -362,10 +363,72 @@ int fw_map_paged_buf(struct fw_priv *fw_priv) > > } > > #endif > > > > +/* > > + * ZSTD-compressed firmware support > > + */ > > +#ifdef CONFIG_FW_LOADER_COMPRESS_ZSTD > > +static int fw_decompress_zstd(struct device *dev, struct fw_priv *fw_priv, > > + size_t in_size, const void *in_buffer) > > +{ > > + size_t len, out_size, workspace_size; > > + void *workspace, *out_buf; > > + ZSTD_DCtx *ctx; > > + int err; > > + > > + if (fw_priv->data) { > > + out_size = fw_priv->allocated_size; > > + out_buf = fw_priv->data; > > + } else { > > + out_size = ZSTD_findDecompressedSize(in_buffer, in_size); > > + if (out_size == ZSTD_CONTENTSIZE_UNKNOWN || > > + out_size == ZSTD_CONTENTSIZE_ERROR) { > > + dev_dbg(dev, "%s: invalid decompression size\n", __func__); > > + return -EINVAL; > > + } > > + out_buf = vzalloc(out_size); > > + if (!out_buf) > > + return -ENOMEM; > > + } > > + > > + workspace_size = ZSTD_DCtxWorkspaceBound(); > > + workspace = kvzalloc(workspace_size, GFP_KERNEL); > > + if (!workspace) { > > + err = -ENOMEM; > > + goto error; > > + } > > + > > + ctx = ZSTD_initDCtx(workspace, workspace_size); > > + if (!ctx) { > > + dev_dbg(dev, "%s: failed to initialize context\n", __func__); > > + err = -EINVAL; > > + goto error; > > + } > > + > > + len = ZSTD_decompressDCtx(ctx, out_buf, out_size, in_buffer, in_size); > > + if (ZSTD_isError(len)) { > > + dev_dbg(dev, "%s: failed to decompress: %d\n", __func__, > > + ZSTD_getErrorCode(len)); > > + err = -EINVAL; > > + goto error; > > + } > > + > > + fw_priv->size = len; > > + if (!fw_priv->data) > > + fw_priv->data = out_buf; > > + err = 0; > > + > > + error: > > + kvfree(workspace); > > + if (!fw_priv->data) > > + vfree(out_buf); > > + return err; > > +} > > +#endif /* CONFIG_FW_LOADER_COMPRESS_ZSTD */ > > + > > /* > > * XZ-compressed firmware support > > */ > > -#ifdef CONFIG_FW_LOADER_COMPRESS > > +#ifdef CONFIG_FW_LOADER_COMPRESS_XZ > > /* show an error and return the standard error code */ > > static int fw_decompress_xz_error(struct device *dev, enum xz_ret xz_ret) > > { > > @@ -459,7 +522,7 @@ static int fw_decompress_xz(struct device *dev, struct fw_priv *fw_priv, > > else > > return fw_decompress_xz_pages(dev, fw_priv, in_size, in_buffer); > > } > > -#endif /* CONFIG_FW_LOADER_COMPRESS */ > > +#endif /* CONFIG_FW_LOADER_COMPRESS_XZ */ > > > > /* direct firmware loading support */ > > static char fw_path_para[256]; > > @@ -814,7 +877,12 @@ _request_firmware(const struct firmware **firmware_p, const char *name, > > if (!(opt_flags & FW_OPT_PARTIAL)) > > nondirect = true; > > > > -#ifdef CONFIG_FW_LOADER_COMPRESS > > +#ifdef CONFIG_FW_LOADER_COMPRESS_ZSTD > > + if (ret == -ENOENT && nondirect) > > + ret = fw_get_filesystem_firmware(device, fw->priv, ".zst", > > + fw_decompress_zstd); > > +#endif > > +#ifdef CONFIG_FW_LOADER_COMPRESS_XZ > > if (ret == -ENOENT && nondirect) > > ret = fw_get_filesystem_firmware(device, fw->priv, ".xz", > > fw_decompress_xz); > > Since this is growing, it would be nicer if we avoid the #ifdef mess and > simply have code iterate over supported compressions, and this is > observed here in one line. If it ever grows further, yeah, but I guess this won't happen so often :) As of now, I chose the open-code as there are only two. thanks, Takashi ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files 2021-01-27 15:49 ` [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files Takashi Iwai 2021-02-17 13:16 ` Luis Chamberlain @ 2021-02-17 13:24 ` Luis Chamberlain 2021-02-17 13:34 ` Takashi Iwai 1 sibling, 1 reply; 19+ messages in thread From: Luis Chamberlain @ 2021-02-17 13:24 UTC (permalink / raw) To: Takashi Iwai; +Cc: Greg Kroah-Hartman, Rafael J . Wysocki, linux-kernel On Wed, Jan 27, 2021 at 04:49:36PM +0100, Takashi Iwai wrote: > Due to the popular demands on ZSTD, here is a patch to add a support > of ZSTD-compressed firmware files via the direct firmware loader. > It's just like XZ-compressed file support, providing a decompressor > with ZSTD. Since ZSTD API can give the decompression size beforehand, > the code is even simpler than XZ. > > Signed-off-by: Takashi Iwai <tiwai@suse.de> It also occurs to me that having a simple like #define HAVE_FIRMWARE_COMPRESS_ZSTD on include/linux/firmware.h would enable userspace to be aware (if they have kernel sources) to determine if the kernels supports this format. Luis ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files 2021-02-17 13:24 ` Luis Chamberlain @ 2021-02-17 13:34 ` Takashi Iwai 2021-02-17 14:17 ` Greg Kroah-Hartman 0 siblings, 1 reply; 19+ messages in thread From: Takashi Iwai @ 2021-02-17 13:34 UTC (permalink / raw) To: Luis Chamberlain; +Cc: Greg Kroah-Hartman, Rafael J . Wysocki, linux-kernel On Wed, 17 Feb 2021 14:24:19 +0100, Luis Chamberlain wrote: > > On Wed, Jan 27, 2021 at 04:49:36PM +0100, Takashi Iwai wrote: > > Due to the popular demands on ZSTD, here is a patch to add a support > > of ZSTD-compressed firmware files via the direct firmware loader. > > It's just like XZ-compressed file support, providing a decompressor > > with ZSTD. Since ZSTD API can give the decompression size beforehand, > > the code is even simpler than XZ. > > > > Signed-off-by: Takashi Iwai <tiwai@suse.de> > > It also occurs to me that having a simple like #define HAVE_FIRMWARE_COMPRESS_ZSTD > on include/linux/firmware.h would enable userspace to be aware (if they > have kernel sources) to determine if the kernels supports this format. Extending that idea, we might want to have a sysfs entry showing the supported formats instead? This will allow to judge dynamically. thanks, Takashi ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files 2021-02-17 13:34 ` Takashi Iwai @ 2021-02-17 14:17 ` Greg Kroah-Hartman 2021-02-17 14:21 ` Takashi Iwai 0 siblings, 1 reply; 19+ messages in thread From: Greg Kroah-Hartman @ 2021-02-17 14:17 UTC (permalink / raw) To: Takashi Iwai; +Cc: Luis Chamberlain, Rafael J . Wysocki, linux-kernel On Wed, Feb 17, 2021 at 02:34:34PM +0100, Takashi Iwai wrote: > On Wed, 17 Feb 2021 14:24:19 +0100, > Luis Chamberlain wrote: > > > > On Wed, Jan 27, 2021 at 04:49:36PM +0100, Takashi Iwai wrote: > > > Due to the popular demands on ZSTD, here is a patch to add a support > > > of ZSTD-compressed firmware files via the direct firmware loader. > > > It's just like XZ-compressed file support, providing a decompressor > > > with ZSTD. Since ZSTD API can give the decompression size beforehand, > > > the code is even simpler than XZ. > > > > > > Signed-off-by: Takashi Iwai <tiwai@suse.de> > > > > It also occurs to me that having a simple like #define HAVE_FIRMWARE_COMPRESS_ZSTD > > on include/linux/firmware.h would enable userspace to be aware (if they > > have kernel sources) to determine if the kernels supports this format. > > Extending that idea, we might want to have a sysfs entry showing the > supported formats instead? This will allow to judge dynamically. What could userspace do with that information? ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files 2021-02-17 14:17 ` Greg Kroah-Hartman @ 2021-02-17 14:21 ` Takashi Iwai 0 siblings, 0 replies; 19+ messages in thread From: Takashi Iwai @ 2021-02-17 14:21 UTC (permalink / raw) To: Greg Kroah-Hartman; +Cc: Luis Chamberlain, Rafael J . Wysocki, linux-kernel On Wed, 17 Feb 2021 15:17:52 +0100, Greg Kroah-Hartman wrote: > > On Wed, Feb 17, 2021 at 02:34:34PM +0100, Takashi Iwai wrote: > > On Wed, 17 Feb 2021 14:24:19 +0100, > > Luis Chamberlain wrote: > > > > > > On Wed, Jan 27, 2021 at 04:49:36PM +0100, Takashi Iwai wrote: > > > > Due to the popular demands on ZSTD, here is a patch to add a support > > > > of ZSTD-compressed firmware files via the direct firmware loader. > > > > It's just like XZ-compressed file support, providing a decompressor > > > > with ZSTD. Since ZSTD API can give the decompression size beforehand, > > > > the code is even simpler than XZ. > > > > > > > > Signed-off-by: Takashi Iwai <tiwai@suse.de> > > > > > > It also occurs to me that having a simple like #define HAVE_FIRMWARE_COMPRESS_ZSTD > > > on include/linux/firmware.h would enable userspace to be aware (if they > > > have kernel sources) to determine if the kernels supports this format. > > > > Extending that idea, we might want to have a sysfs entry showing the > > supported formats instead? This will allow to judge dynamically. > > What could userspace do with that information? Decide whether to store the firmware files on the system in the supported compression format or not. But admittedly it can be little help, as that's rather information you want not for the running kernel but for the system to be installed... Takashi ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH RFC 2/4] selftests: firmware: Simplify test patterns 2021-01-27 15:49 [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support Takashi Iwai 2021-01-27 15:49 ` [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files Takashi Iwai @ 2021-01-27 15:49 ` Takashi Iwai 2021-01-27 15:49 ` [PATCH RFC 3/4] selftest: firmware: Fix the request_firmware_into_buf() test for XZ format Takashi Iwai ` (2 subsequent siblings) 4 siblings, 0 replies; 19+ messages in thread From: Takashi Iwai @ 2021-01-27 15:49 UTC (permalink / raw) To: Luis Chamberlain; +Cc: Greg Kroah-Hartman, Rafael J . Wysocki, linux-kernel The test patterns are almost same in three sequential tests. Make the unified helper function for improving the readability. Signed-off-by: Takashi Iwai <tiwai@suse.de> --- .../selftests/firmware/fw_filesystem.sh | 106 +++++------------- 1 file changed, 30 insertions(+), 76 deletions(-) diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh index c2a2a100114b..2424a97da65b 100755 --- a/tools/testing/selftests/firmware/fw_filesystem.sh +++ b/tools/testing/selftests/firmware/fw_filesystem.sh @@ -435,6 +435,32 @@ test_request_partial_firmware_into_buf() echo "OK" } +do_tests () +{ + mode="$1" + suffix="$2" + + for i in $(seq 1 5); do + test_batched_request_firmware$suffix $i $mode + done + + for i in $(seq 1 5); do + test_batched_request_firmware_into_buf$suffix $i $mode + done + + for i in $(seq 1 5); do + test_batched_request_firmware_direct$suffix $i $mode + done + + for i in $(seq 1 5); do + test_request_firmware_nowait_uevent$suffix $i $mode + done + + for i in $(seq 1 5); do + test_request_firmware_nowait_custom$suffix $i $mode + done +} + # Only continue if batched request triggers are present on the # test-firmware driver test_config_present @@ -442,25 +468,7 @@ test_config_present # test with the file present echo echo "Testing with the file present..." -for i in $(seq 1 5); do - test_batched_request_firmware $i normal -done - -for i in $(seq 1 5); do - test_batched_request_firmware_into_buf $i normal -done - -for i in $(seq 1 5); do - test_batched_request_firmware_direct $i normal -done - -for i in $(seq 1 5); do - test_request_firmware_nowait_uevent $i normal -done - -for i in $(seq 1 5); do - test_request_firmware_nowait_custom $i normal -done +do_tests normal # Partial loads cannot use fallback, so do not repeat tests. test_request_partial_firmware_into_buf 0 10 @@ -472,25 +480,7 @@ test_request_partial_firmware_into_buf 2 10 # a hung task, which would require a hard reset. echo echo "Testing with the file missing..." -for i in $(seq 1 5); do - test_batched_request_firmware_nofile $i -done - -for i in $(seq 1 5); do - test_batched_request_firmware_into_buf_nofile $i -done - -for i in $(seq 1 5); do - test_batched_request_firmware_direct_nofile $i -done - -for i in $(seq 1 5); do - test_request_firmware_nowait_uevent_nofile $i -done - -for i in $(seq 1 5); do - test_request_firmware_nowait_custom_nofile $i -done +do_tests nofile _nofile # Partial loads cannot use fallback, so do not repeat tests. test_request_partial_firmware_into_buf_nofile 0 10 @@ -505,48 +495,12 @@ xz -9 -C crc32 -k $FW config_set_name $NAME echo echo "Testing with both plain and xz files present..." -for i in $(seq 1 5); do - test_batched_request_firmware $i both -done - -for i in $(seq 1 5); do - test_batched_request_firmware_into_buf $i both -done - -for i in $(seq 1 5); do - test_batched_request_firmware_direct $i both -done - -for i in $(seq 1 5); do - test_request_firmware_nowait_uevent $i both -done - -for i in $(seq 1 5); do - test_request_firmware_nowait_custom $i both -done +do_tests both # test with only xz file present mv "$FW" "${FW}-orig" echo echo "Testing with only xz file present..." -for i in $(seq 1 5); do - test_batched_request_firmware $i xzonly -done - -for i in $(seq 1 5); do - test_batched_request_firmware_into_buf $i xzonly -done - -for i in $(seq 1 5); do - test_batched_request_firmware_direct $i xzonly -done - -for i in $(seq 1 5); do - test_request_firmware_nowait_uevent $i xzonly -done - -for i in $(seq 1 5); do - test_request_firmware_nowait_custom $i xzonly -done +do_tests xzonly exit 0 -- 2.26.2 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH RFC 3/4] selftest: firmware: Fix the request_firmware_into_buf() test for XZ format 2021-01-27 15:49 [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support Takashi Iwai 2021-01-27 15:49 ` [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files Takashi Iwai 2021-01-27 15:49 ` [PATCH RFC 2/4] selftests: firmware: Simplify test patterns Takashi Iwai @ 2021-01-27 15:49 ` Takashi Iwai 2021-01-27 15:49 ` [PATCH RFC 4/4] selftest: firmware: Add ZSTD compressed file tests Takashi Iwai 2021-01-27 17:09 ` [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support Greg Kroah-Hartman 4 siblings, 0 replies; 19+ messages in thread From: Takashi Iwai @ 2021-01-27 15:49 UTC (permalink / raw) To: Luis Chamberlain; +Cc: Greg Kroah-Hartman, Rafael J . Wysocki, linux-kernel The test uses a different firmware name, and we forgot to adapt for the XZ compressed file tests. Signed-off-by: Takashi Iwai <tiwai@suse.de> --- tools/testing/selftests/firmware/fw_filesystem.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh index 2424a97da65b..f1976e650672 100755 --- a/tools/testing/selftests/firmware/fw_filesystem.sh +++ b/tools/testing/selftests/firmware/fw_filesystem.sh @@ -492,6 +492,7 @@ test "$HAS_FW_LOADER_COMPRESS" != "yes" && exit 0 # test with both files present xz -9 -C crc32 -k $FW +xz -9 -C crc32 -k $FW_INTO_BUF config_set_name $NAME echo echo "Testing with both plain and xz files present..." @@ -499,6 +500,7 @@ do_tests both # test with only xz file present mv "$FW" "${FW}-orig" +mv "$FW_INTO_BUF" "${FW_INTO_BUF}-orig" echo echo "Testing with only xz file present..." do_tests xzonly -- 2.26.2 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH RFC 4/4] selftest: firmware: Add ZSTD compressed file tests 2021-01-27 15:49 [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support Takashi Iwai ` (2 preceding siblings ...) 2021-01-27 15:49 ` [PATCH RFC 3/4] selftest: firmware: Fix the request_firmware_into_buf() test for XZ format Takashi Iwai @ 2021-01-27 15:49 ` Takashi Iwai 2021-01-27 17:09 ` [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support Greg Kroah-Hartman 4 siblings, 0 replies; 19+ messages in thread From: Takashi Iwai @ 2021-01-27 15:49 UTC (permalink / raw) To: Luis Chamberlain; +Cc: Greg Kroah-Hartman, Rafael J . Wysocki, linux-kernel It's similar like XZ compressed files. For the simplicity, both XZ and ZSTD tests are done in a single function. The format is specified via $COMPRESS_FORMAT and the compression function is pre-defined. Signed-off-by: Takashi Iwai <tiwai@suse.de> --- .../selftests/firmware/fw_filesystem.sh | 75 ++++++++++++++----- tools/testing/selftests/firmware/fw_lib.sh | 12 ++- 2 files changed, 64 insertions(+), 23 deletions(-) diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh index f1976e650672..abaceac83d6e 100755 --- a/tools/testing/selftests/firmware/fw_filesystem.sh +++ b/tools/testing/selftests/firmware/fw_filesystem.sh @@ -211,7 +211,7 @@ read_firmwares() else fwfile="$FW" fi - if [ "$1" = "xzonly" ]; then + if [ "$1" = "componly" ]; then fwfile="${fwfile}-orig" fi for i in $(seq 0 3); do @@ -235,7 +235,7 @@ read_partial_firmwares() fwfile="${FW}" fi - if [ "$1" = "xzonly" ]; then + if [ "$1" = "componly" ]; then fwfile="${fwfile}-orig" fi @@ -409,10 +409,8 @@ test_request_firmware_nowait_custom() config_unset_uevent RANDOM_FILE_PATH=$(setup_random_file) RANDOM_FILE="$(basename $RANDOM_FILE_PATH)" - if [ "$2" = "both" ]; then - xz -9 -C crc32 -k $RANDOM_FILE_PATH - elif [ "$2" = "xzonly" ]; then - xz -9 -C crc32 $RANDOM_FILE_PATH + if [ -n "$2" -a "$2" != "normal" ]; then + compress-"$2"-$COMPRESS_FORMAT $RANDOM_FILE_PATH fi config_set_name $RANDOM_FILE config_trigger_async @@ -488,21 +486,58 @@ test_request_partial_firmware_into_buf_nofile 0 5 test_request_partial_firmware_into_buf_nofile 1 6 test_request_partial_firmware_into_buf_nofile 2 10 -test "$HAS_FW_LOADER_COMPRESS" != "yes" && exit 0 +test_request_firmware_compressed () +{ + export COMPRESS_FORMAT="$1" -# test with both files present -xz -9 -C crc32 -k $FW -xz -9 -C crc32 -k $FW_INTO_BUF -config_set_name $NAME -echo -echo "Testing with both plain and xz files present..." -do_tests both + # test with both files present + compress-both-$COMPRESS_FORMAT $FW + compress-both-$COMPRESS_FORMAT $FW_INTO_BUF -# test with only xz file present -mv "$FW" "${FW}-orig" -mv "$FW_INTO_BUF" "${FW_INTO_BUF}-orig" -echo -echo "Testing with only xz file present..." -do_tests xzonly + config_set_name $NAME + echo + echo "Testing with both plain and $COMPRESS_FORMAT files present..." + do_tests both + + # test with only compressed file present + mv "$FW" "${FW}-orig" + mv "$FW_INTO_BUF" "${FW_INTO_BUF}-orig" + + config_set_name $NAME + echo + echo "Testing with only $COMPRESS_FORMAT file present..." + do_tests componly + + mv "${FW}-orig" "$FW" + mv "${FW_INTO_BUF}-orig" "$FW_INTO_BUF" +} + +compress-both-XZ () +{ + xz -k -9 -C crc32 "$@" +} + +compress-componly-XZ () +{ + xz -9 -C crc32 "$@" +} + +compress-both-ZSTD () +{ + zstd -q -k "$@" +} + +compress-componly-ZSTD () +{ + zstd -q --rm "$@" +} + +if test "$HAS_FW_LOADER_COMPRESS_XZ" = "yes"; then + test_request_firmware_compressed XZ +fi + +if test "$HAS_FW_LOADER_COMPRESS_ZSTD" = "yes"; then + test_request_firmware_compressed ZSTD +fi exit 0 diff --git a/tools/testing/selftests/firmware/fw_lib.sh b/tools/testing/selftests/firmware/fw_lib.sh index 5b8c0fedee76..3fa8282b053b 100755 --- a/tools/testing/selftests/firmware/fw_lib.sh +++ b/tools/testing/selftests/firmware/fw_lib.sh @@ -62,7 +62,8 @@ check_setup() { HAS_FW_LOADER_USER_HELPER="$(kconfig_has CONFIG_FW_LOADER_USER_HELPER=y)" HAS_FW_LOADER_USER_HELPER_FALLBACK="$(kconfig_has CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y)" - HAS_FW_LOADER_COMPRESS="$(kconfig_has CONFIG_FW_LOADER_COMPRESS=y)" + HAS_FW_LOADER_COMPRESS_XZ="$(kconfig_has CONFIG_FW_LOADER_COMPRESS_XZ=y)" + HAS_FW_LOADER_COMPRESS_ZSTD="$(kconfig_has CONFIG_FW_LOADER_COMPRESS_ZSTD=y)" PROC_FW_IGNORE_SYSFS_FALLBACK="0" PROC_FW_FORCE_SYSFS_FALLBACK="0" @@ -98,9 +99,14 @@ check_setup() OLD_FWPATH="$(cat /sys/module/firmware_class/parameters/path)" - if [ "$HAS_FW_LOADER_COMPRESS" = "yes" ]; then + if [ "$HAS_FW_LOADER_COMPRESS_XZ" = "yes" ]; then if ! which xz 2> /dev/null > /dev/null; then - HAS_FW_LOADER_COMPRESS="" + HAS_FW_LOADER_COMPRESS_XZ="" + fi + fi + if [ "$HAS_FW_LOADER_COMPRESS_ZSTD" = "yes" ]; then + if ! which zstd 2> /dev/null > /dev/null; then + HAS_FW_LOADER_COMPRESS_ZSTD="" fi fi } -- 2.26.2 ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support 2021-01-27 15:49 [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support Takashi Iwai ` (3 preceding siblings ...) 2021-01-27 15:49 ` [PATCH RFC 4/4] selftest: firmware: Add ZSTD compressed file tests Takashi Iwai @ 2021-01-27 17:09 ` Greg Kroah-Hartman 2022-01-27 18:22 ` Hideki Yamane 4 siblings, 1 reply; 19+ messages in thread From: Greg Kroah-Hartman @ 2021-01-27 17:09 UTC (permalink / raw) To: Takashi Iwai; +Cc: Luis Chamberlain, Rafael J . Wysocki, linux-kernel On Wed, Jan 27, 2021 at 04:49:35PM +0100, Takashi Iwai wrote: > Hi, > > it seems that ZSTD format is getting popular, and I've been asked > about the firmware loader support. So I took a quick glance, and it > turned out that it's fairly easy thanks to the existing ZSTD API. > Now high time to submit something. > > The first patch adds a new Kconfig CONFIG_FW_LOADER_COMPRESS_ZSTD and > the corresponding decompression function to the firmware loader code. > For the already supported XZ-compression, CONFIG_FW_LOADER_COMPRESS_XZ > is added to make it selectable explicitly, too. > > The rest three patches are for selftest: a cleanup, a fix and the > additional support of ZSTD format. > > Currently, I have no idea whether any distro would use ZSTD files for > firmware files in near future, though. That's the reason of this > patch set being an RFC for now. Looks sane enough to me, if we have a real user, I see no reason why to not merge this. thanks, greg k-h ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support 2021-01-27 17:09 ` [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support Greg Kroah-Hartman @ 2022-01-27 18:22 ` Hideki Yamane 2022-01-27 18:41 ` Greg Kroah-Hartman 0 siblings, 1 reply; 19+ messages in thread From: Hideki Yamane @ 2022-01-27 18:22 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: Takashi Iwai, Luis Chamberlain, Rafael J . Wysocki, linux-kernel On Wed, 27 Jan 2021 18:09:48 +0100 Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > > > Currently, I have no idea whether any distro would use ZSTD files for > > firmware files in near future, though. That's the reason of this > > patch set being an RFC for now. > > Looks sane enough to me, if we have a real user, I see no reason why to > not merge this. Just curious, any progress for this? -- Hideki Yamane <henrich@iijmio-mail.jp> ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support 2022-01-27 18:22 ` Hideki Yamane @ 2022-01-27 18:41 ` Greg Kroah-Hartman 2022-01-28 0:33 ` Hideki Yamane 0 siblings, 1 reply; 19+ messages in thread From: Greg Kroah-Hartman @ 2022-01-27 18:41 UTC (permalink / raw) To: Hideki Yamane Cc: Takashi Iwai, Luis Chamberlain, Rafael J . Wysocki, linux-kernel On Fri, Jan 28, 2022 at 03:22:13AM +0900, Hideki Yamane wrote: > On Wed, 27 Jan 2021 18:09:48 +0100 > Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > > > > Currently, I have no idea whether any distro would use ZSTD files for > > > firmware files in near future, though. That's the reason of this > > > patch set being an RFC for now. > > > > Looks sane enough to me, if we have a real user, I see no reason why to > > not merge this. > > Just curious, any progress for this? What is "this"? You are responding to a year-old email message :) ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support 2022-01-27 18:41 ` Greg Kroah-Hartman @ 2022-01-28 0:33 ` Hideki Yamane 2022-01-28 6:54 ` Greg Kroah-Hartman 0 siblings, 1 reply; 19+ messages in thread From: Hideki Yamane @ 2022-01-28 0:33 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: Takashi Iwai, Luis Chamberlain, Rafael J . Wysocki, linux-kernel On Thu, 27 Jan 2022 19:41:18 +0100 Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > What is "this"? You are responding to a year-old email message :) Oh, yes, sorry. What's the status for adding "ZSTD-compressed file support" for loading firmware feature config FW_LOADER_COMPRESS in drivers/base/firmware_loader/Kconfig has still XZ_DEC only, where should I check to know its progress? -- Hideki Yamane <henrich@iijmio-mail.jp> ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support 2022-01-28 0:33 ` Hideki Yamane @ 2022-01-28 6:54 ` Greg Kroah-Hartman 2022-01-28 8:13 ` Takashi Iwai 0 siblings, 1 reply; 19+ messages in thread From: Greg Kroah-Hartman @ 2022-01-28 6:54 UTC (permalink / raw) To: Hideki Yamane Cc: Takashi Iwai, Luis Chamberlain, Rafael J . Wysocki, linux-kernel On Fri, Jan 28, 2022 at 09:33:35AM +0900, Hideki Yamane wrote: > On Thu, 27 Jan 2022 19:41:18 +0100 > Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > > What is "this"? You are responding to a year-old email message :) > > Oh, yes, sorry. > > What's the status for adding "ZSTD-compressed file support" for loading > firmware feature config FW_LOADER_COMPRESS in drivers/base/firmware_loader/Kconfig > has still XZ_DEC only, where should I check to know its progress? If you need this feature, take the patches and rebase them to the latest kernel tree and submit them for inclusion. thanks, greg k-h ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support 2022-01-28 6:54 ` Greg Kroah-Hartman @ 2022-01-28 8:13 ` Takashi Iwai 2022-02-01 17:36 ` Hideki Yamane 0 siblings, 1 reply; 19+ messages in thread From: Takashi Iwai @ 2022-01-28 8:13 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: Hideki Yamane, Takashi Iwai, Luis Chamberlain, Rafael J . Wysocki, linux-kernel On Fri, 28 Jan 2022 07:54:22 +0100, Greg Kroah-Hartman wrote: > > On Fri, Jan 28, 2022 at 09:33:35AM +0900, Hideki Yamane wrote: > > On Thu, 27 Jan 2022 19:41:18 +0100 > > Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > > > What is "this"? You are responding to a year-old email message :) > > > > Oh, yes, sorry. > > > > What's the status for adding "ZSTD-compressed file support" for loading > > firmware feature config FW_LOADER_COMPRESS in drivers/base/firmware_loader/Kconfig > > has still XZ_DEC only, where should I check to know its progress? > > If you need this feature, take the patches and rebase them to the latest > kernel tree and submit them for inclusion. Yes, it's a kind of chicken-and-egg problem, and we want that the patches have actually users before merging to the mainline. I rebased the patches to 5.17-rc1 with the adaption for the latest ZSTD API and pushed to topic/fw-loader-zstd-5.17 branch of my sound.git tree. It's totally untested, and it'd be appreciated if anyone can confirm that it's working and used. Then we can happily re-submit the patches for the merge. thanks, Takashi ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support 2022-01-28 8:13 ` Takashi Iwai @ 2022-02-01 17:36 ` Hideki Yamane 2022-02-02 10:56 ` Takashi Iwai 0 siblings, 1 reply; 19+ messages in thread From: Hideki Yamane @ 2022-02-01 17:36 UTC (permalink / raw) To: Takashi Iwai Cc: Greg Kroah-Hartman, Luis Chamberlain, Rafael J . Wysocki, linux-kernel, henrich Hi, On Fri, 28 Jan 2022 09:13:36 +0100 Takashi Iwai <tiwai@suse.de> wrote: > It's totally untested, and it'd be appreciated if > anyone can confirm that it's working and used. Then we can happily > re-submit the patches for the merge. I've built it with 5.16.4 based one in Debian experimental repository and it seems to work (note: build failure with 5.15.15 in Debian unstable). How to check: 1. compress firmware files under /lib/firmware with zstd 2. boot with distro kernel that is not enable this feature 3. Wifi that needs firmware doesn't work as expected (*1) 4. boot with patched kernel 5. Wifi works :) (*2) logs: (*1) 2月 02 02:09:43 elitebook830 kernel: Intel(R) Wireless WiFi driver for Linux 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: enabling device (0000 -> 0002) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-36.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: firmware_class: See https://wiki.debian.org/Firmware for information about missing firmware 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-36.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-35.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-35.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-34.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-34.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-33.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-33.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-32.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-32.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-31.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-31.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-30.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-30.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-29.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-29.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-28.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-28.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-27.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-27.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-26.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-26.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-25.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-25.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-24.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-24.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-23.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-23.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-22.ucode (-2) 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: Direct firmware load for iwlwifi-8265-22.ucode failed with error -2 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: minimum version required: iwlwifi-8265-22 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: maximum version supported: iwlwifi-8265-36 2月 02 02:09:43 elitebook830 kernel: iwlwifi 0000:01:00.0: check git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git (*2) 2月 02 02:12:35 elitebook830 kernel: Intel(R) Wireless WiFi driver for Linux 2月 02 02:12:35 elitebook830 kernel: iwlwifi 0000:01:00.0: enabling device (0000 -> 0002) 2月 02 02:12:35 elitebook830 kernel: usb 3-1.3.2.1.4.2: FTDI USB Serial Device converter now attached to ttyUSB0 2月 02 02:12:35 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: failed to load iwlwifi-8265-36.ucode (-2) 2月 02 02:12:35 elitebook830 kernel: firmware_class: See https://wiki.debian.org/Firmware for information about missing firmware 2月 02 02:12:35 elitebook830 kernel: iwlwifi 0000:01:00.0: firmware: direct-loading firmware iwlwifi-8265-36.ucode 2月 02 02:12:35 elitebook830 kernel: iwlwifi 0000:01:00.0: loaded firmware version 36.ca7b901d.0 8265-36.ucode op_mode iwlmvm -- Hideki Yamane <henrich@iijmio-mail.jp> ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support 2022-02-01 17:36 ` Hideki Yamane @ 2022-02-02 10:56 ` Takashi Iwai 0 siblings, 0 replies; 19+ messages in thread From: Takashi Iwai @ 2022-02-02 10:56 UTC (permalink / raw) To: Hideki Yamane Cc: Takashi Iwai, Greg Kroah-Hartman, Luis Chamberlain, Rafael J . Wysocki, linux-kernel On Tue, 01 Feb 2022 18:36:18 +0100, Hideki Yamane wrote: > > Hi, > > On Fri, 28 Jan 2022 09:13:36 +0100 > Takashi Iwai <tiwai@suse.de> wrote: > > It's totally untested, and it'd be appreciated if > > anyone can confirm that it's working and used. Then we can happily > > re-submit the patches for the merge. > > I've built it with 5.16.4 based one in Debian experimental repository > and it seems to work (note: build failure with 5.15.15 in Debian unstable). > > > How to check: > 1. compress firmware files under /lib/firmware with zstd > 2. boot with distro kernel that is not enable this feature > 3. Wifi that needs firmware doesn't work as expected (*1) > 4. boot with patched kernel > 5. Wifi works :) (*2) Good to know that it's working. (BTW, you can see more details when you enable the debug, e.g. "firmware_class.dyndbg=+p" boot option.) The next step is actually using this in some distro (or privately). If the feature is confirmed to be needed, I'll resubmit the latest patches for the merge request. thanks, Takashi ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2022-02-02 10:57 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-27 15:49 [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support Takashi Iwai 2021-01-27 15:49 ` [PATCH RFC 1/4] firmware: Add the support for ZSTD-compressed firmware files Takashi Iwai 2021-02-17 13:16 ` Luis Chamberlain 2021-02-17 13:22 ` Takashi Iwai 2021-02-17 13:24 ` Luis Chamberlain 2021-02-17 13:34 ` Takashi Iwai 2021-02-17 14:17 ` Greg Kroah-Hartman 2021-02-17 14:21 ` Takashi Iwai 2021-01-27 15:49 ` [PATCH RFC 2/4] selftests: firmware: Simplify test patterns Takashi Iwai 2021-01-27 15:49 ` [PATCH RFC 3/4] selftest: firmware: Fix the request_firmware_into_buf() test for XZ format Takashi Iwai 2021-01-27 15:49 ` [PATCH RFC 4/4] selftest: firmware: Add ZSTD compressed file tests Takashi Iwai 2021-01-27 17:09 ` [PATCH RFC 0/4] firmware: Add ZSTD-compressed file support Greg Kroah-Hartman 2022-01-27 18:22 ` Hideki Yamane 2022-01-27 18:41 ` Greg Kroah-Hartman 2022-01-28 0:33 ` Hideki Yamane 2022-01-28 6:54 ` Greg Kroah-Hartman 2022-01-28 8:13 ` Takashi Iwai 2022-02-01 17:36 ` Hideki Yamane 2022-02-02 10:56 ` Takashi Iwai
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).