[PATCH] image_types: declare support for wic.zst

[PATCH] image_types: declare support for wic.zst

[Diego]

Declare images in wic.zst format as supported, as bmaptool now supports
zstd:
https://github.com/intel/bmap-tools/commit/1b8437d58447d3796dc11fd0f2c62bec5746e5d6
Given the very fast decompression speed of zstd over, for example, gzip,
writing images with bmaptool can provide big write speed improvements
when decompression speed is the limiting factor (especially with very
sparse images).

Signed-off-by: Diego Rondini <diego.rondini@kynetics.com>
---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index f82f1d8862..ab05cc90ff 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -273,7 +273,7 @@ IMAGE_TYPES = " \
     ubi ubifs multiubi \
     tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
     cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
-    wic wic.gz wic.bz2 wic.lzma \
+    wic wic.gz wic.bz2 wic.lzma wic.zst \
     container \
     f2fs \
 "
-- 
2.21.3

Re: [OE-core] [PATCH] image_types: declare support for wic.zst

[Richard Purdie]

On Thu, 2020-05-14 at 15:19 +0200, Diego wrote:
> Declare images in wic.zst format as supported, as bmaptool now supports
> zstd:
> https://github.com/intel/bmap-tools/commit/1b8437d58447d3796dc11fd0f2c62bec5746e5d6
> Given the very fast decompression speed of zstd over, for example, gzip,
> writing images with bmaptool can provide big write speed improvements
> when decompression speed is the limiting factor (especially with very
> sparse images).
> 
> Signed-off-by: Diego Rondini <diego.rondini@kynetics.com>
> ---
>  meta/classes/image_types.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> index f82f1d8862..ab05cc90ff 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -273,7 +273,7 @@ IMAGE_TYPES = " \
>      ubi ubifs multiubi \
>      tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
>      cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
> -    wic wic.gz wic.bz2 wic.lzma \
> +    wic wic.gz wic.bz2 wic.lzma wic.zst \
>      container \
>      f2fs \
>  "

Causes selftest failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/931

(oe-selftest -r imagefeatures.ImageFeatures.test_image_fstypes)

Cheers,

Richard

Re: [OE-core] [PATCH] image_types: declare support for wic.zst

[Diego]

[-- Attachment #1: Type: text/plain, Size: 2251 bytes --]

Hi Richard,

On Thu, May 14, 2020 at 11:21 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Thu, 2020-05-14 at 15:19 +0200, Diego wrote:
> > Declare images in wic.zst format as supported, as bmaptool now supports
> > zstd:
> >
> https://github.com/intel/bmap-tools/commit/1b8437d58447d3796dc11fd0f2c62bec5746e5d6
> > Given the very fast decompression speed of zstd over, for example, gzip,
> > writing images with bmaptool can provide big write speed improvements
> > when decompression speed is the limiting factor (especially with very
> > sparse images).
> >
> > Signed-off-by: Diego Rondini <diego.rondini@kynetics.com>
> > ---
> >  meta/classes/image_types.bbclass | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/classes/image_types.bbclass
> b/meta/classes/image_types.bbclass
> > index f82f1d8862..ab05cc90ff 100644
> > --- a/meta/classes/image_types.bbclass
> > +++ b/meta/classes/image_types.bbclass
> > @@ -273,7 +273,7 @@ IMAGE_TYPES = " \
> >      ubi ubifs multiubi \
> >      tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
> >      cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
> > -    wic wic.gz wic.bz2 wic.lzma \
> > +    wic wic.gz wic.bz2 wic.lzma wic.zst \
> >      container \
> >      f2fs \
> >  "
>
> Causes selftest failures:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/931
>
> (oe-selftest -r imagefeatures.ImageFeatures.test_image_fstypes)
>
>
I'm wondering how this didn't pop up earlier. The following patch
introduced zstd support:
http://git.openembedded.org/openembedded-core/commit/meta/classes/image_types.bbclass?id=77e0f502020085875bbc74da66815c89f19a9351
of course specifying a DEPENDS on zstd-native:
http://git.openembedded.org/openembedded-core/tree/meta/classes/image_types.bbclass?h=yocto-3.1#n316
but... zstd is NOT in oe-core, but in meta-oe!
http://layers.openembedded.org/layerindex/recipe/116635/

I guess the solution for master is to move zstd to oe-core, right?

But what about dunfell? It has the zstd support in image_types.bbclass
patch, but we can't add zstd. Shall we revert the patch?

Regards,
Diego Rondini
Sr. Embedded Engineer

Kynetics
www.kynetics.com

[-- Attachment #2: Type: text/html, Size: 3657 bytes --]

Re: [OE-core] [PATCH] image_types: declare support for wic.zst

[Richard Purdie]

On Thu, 2020-05-14 at 23:53 +0200, Diego Rondini wrote:
> Hi Richard,
> 
> On Thu, May 14, 2020 at 11:21 PM Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> > On Thu, 2020-05-14 at 15:19 +0200, Diego wrote:
> > > Declare images in wic.zst format as supported, as bmaptool now supports
> > > zstd:
> > > https://github.com/intel/bmap-tools/commit/1b8437d58447d3796dc11fd0f2c62bec5746e5d6
> > > Given the very fast decompression speed of zstd over, for example, gzip,
> > > writing images with bmaptool can provide big write speed improvements
> > > when decompression speed is the limiting factor (especially with very
> > > sparse images).
> > > 
> > > Signed-off-by: Diego Rondini <diego.rondini@kynetics.com>
> > > ---
> > >  meta/classes/image_types.bbclass | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> > > index f82f1d8862..ab05cc90ff 100644
> > > --- a/meta/classes/image_types.bbclass
> > > +++ b/meta/classes/image_types.bbclass
> > > @@ -273,7 +273,7 @@ IMAGE_TYPES = " \
> > >      ubi ubifs multiubi \
> > >      tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
> > >      cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
> > > -    wic wic.gz wic.bz2 wic.lzma \
> > > +    wic wic.gz wic.bz2 wic.lzma wic.zst \
> > >      container \
> > >      f2fs \
> > >  "
> > 
> > Causes selftest failures:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/931
> > 
> > (oe-selftest -r imagefeatures.ImageFeatures.test_image_fstypes)
> > 
> 
> I'm wondering how this didn't pop up earlier. The following patch introduced zstd support:
> http://git.openembedded.org/openembedded-core/commit/meta/classes/image_types.bbclass?id=77e0f502020085875bbc74da66815c89f19a9351
> of course specifying a DEPENDS on zstd-native:
> http://git.openembedded.org/openembedded-core/tree/meta/classes/image_types.bbclass?h=yocto-3.1#n316
> but... zstd is NOT in oe-core, but in meta-oe!
> http://layers.openembedded.org/layerindex/recipe/116635/
> 
> I guess the solution for master is to move zstd to oe-core, right?
> 
> But what about dunfell? It has the zstd support in image_types.bbclass patch, but we can't add zstd. Shall we revert the patch?

I think:

meta/lib/oeqa/selftest/cases/imagefeatures.py:        blacklist = set(('container', 'elf', 'f2fs', 'multiubi', 'tar.zst'))

may be the clue to fixing this.

Cheers,

Richard

Re: [OE-core] [PATCH] image_types: declare support for wic.zst

[Diego]

[-- Attachment #1: Type: text/plain, Size: 2994 bytes --]

Hi Richard,

On Thu, May 14, 2020 at 11:59 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Thu, 2020-05-14 at 23:53 +0200, Diego Rondini wrote:
> > Hi Richard,
> >
> > On Thu, May 14, 2020 at 11:21 PM Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> > > On Thu, 2020-05-14 at 15:19 +0200, Diego wrote:
> > > > Declare images in wic.zst format as supported, as bmaptool now
> supports
> > > > zstd:
> > > >
> https://github.com/intel/bmap-tools/commit/1b8437d58447d3796dc11fd0f2c62bec5746e5d6
> > > > Given the very fast decompression speed of zstd over, for example,
> gzip,
> > > > writing images with bmaptool can provide big write speed improvements
> > > > when decompression speed is the limiting factor (especially with very
> > > > sparse images).
> > > >
> > > > Signed-off-by: Diego Rondini <diego.rondini@kynetics.com>
> > > > ---
> > > >  meta/classes/image_types.bbclass | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/meta/classes/image_types.bbclass
> b/meta/classes/image_types.bbclass
> > > > index f82f1d8862..ab05cc90ff 100644
> > > > --- a/meta/classes/image_types.bbclass
> > > > +++ b/meta/classes/image_types.bbclass
> > > > @@ -273,7 +273,7 @@ IMAGE_TYPES = " \
> > > >      ubi ubifs multiubi \
> > > >      tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
> > > >      cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
> > > > -    wic wic.gz wic.bz2 wic.lzma \
> > > > +    wic wic.gz wic.bz2 wic.lzma wic.zst \
> > > >      container \
> > > >      f2fs \
> > > >  "
> > >
> > > Causes selftest failures:
> > >
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/931
> > >
> > > (oe-selftest -r imagefeatures.ImageFeatures.test_image_fstypes)
> > >
> >
> > I'm wondering how this didn't pop up earlier. The following patch
> introduced zstd support:
> >
> http://git.openembedded.org/openembedded-core/commit/meta/classes/image_types.bbclass?id=77e0f502020085875bbc74da66815c89f19a9351
> > of course specifying a DEPENDS on zstd-native:
> >
> http://git.openembedded.org/openembedded-core/tree/meta/classes/image_types.bbclass?h=yocto-3.1#n316
> > but... zstd is NOT in oe-core, but in meta-oe!
> > http://layers.openembedded.org/layerindex/recipe/116635/
> >
> > I guess the solution for master is to move zstd to oe-core, right?
> >
> > But what about dunfell? It has the zstd support in image_types.bbclass
> patch, but we can't add zstd. Shall we revert the patch?
>
> I think:
>
> meta/lib/oeqa/selftest/cases/imagefeatures.py:        blacklist =
> set(('container', 'elf', 'f2fs', 'multiubi', 'tar.zst'))
>
> may be the clue to fixing this.
>

Ah, ok, so it is legit to support a compression format not available in
oe-core itself.

I'll resend the patch with the test disabled then.


> Cheers,
>
> Richard
>
>

Regards,
Diego Rondini
Sr. Embedded Engineer

Kynetics
www.kynetics.com

[-- Attachment #2: Type: text/html, Size: 5010 bytes --]