* Re: [OE-core] [PATCH] opkg: enable zstd support
[not found] ` <CANOoYsMVb9YpfwjTNHv4kSARM15_3T34CA-Ekr=t7P5F0XO4bA@mail.gmail.com>
@ 2022-09-13 20:24 ` Alex Feinman
2022-09-13 20:34 ` Khem Raj
2022-09-13 21:57 ` Alex Stewart
2 siblings, 0 replies; 12+ messages in thread
From: Alex Feinman @ 2022-09-13 20:24 UTC (permalink / raw)
To: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 6011 bytes --]
On Tue, Sep 13, 2022 at 1:20 PM Alex Feinman <afeinman@snap.com> wrote:
> I do have some numbers. When I was selling this change internally, I did a
> comparison on our internal build.
>
> Combined write IPK times (Σ t do_package_write_ipk)
>
> xz 162m 35s
>
> gz 52m 13s
>
> zstd 33m 49s
>
> Compression rate for zstd was closer to xz than to gz but not as good as xz. For systems that have to cache packages on the device with limited storage xz might be a better option, but for the bulk of projects zstd is the best choice
>
> Additionally, zstd offers much faster decompression than xz so the rootfs build step that includes unpacking all of the ipks, takes 3m 58s with xz and 2m 44s with zstd.
>
> One other thing of note - if your build includes debug packages, some may be quite large. E.g. one of our components produces a 2.2 GB debug package (uncompressed). On large files xz requires a disproportionally large amount of time resulting in 15 minutes needed to simply write ipk for the abovementioned packages, whereas zstd took about 45 sec. For frequent tasks like bitbaking a single package this translates in a lot of saved time.
>
> Bottom line - I think making xz a default package compressor was not entirely thought through. gzip or zstd is what the default should be.
>
> One final note: I could not find a reasonable explanation for why opkg-tools require code changes to support a different compressor. BSD tar and GNU tar both can easily accept compressors that they have no idea about (via -I option) because all of them provide a unified command line interface for use in pipes. If this were done similar to tar, we could have used any compressor we wanted, including the multithreaded versions (zstdmt)
>
>
> On Tue, Sep 13, 2022 at 12:43 PM Khem Raj <raj.khem@gmail.com> wrote:
>
>> On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart <alex.stewart@ni.com>
>> wrote:
>> >
>> > ACK from me - apart from enabling zstd by default.
>> >
>> > On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org wrote:
>> > > This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
>> > > OPKGBUILDCMD = "opkg-build -Z zstd"
>> > >
>> > > Signed-off-by: Alex Feinman <afeinman@snap.com>
>> > > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
>> > > ---
>> > > meta/recipes-devtools/opkg/opkg_0.6.0.bb | 3 ++-
>> > > 1 file changed, 2 insertions(+), 1 deletion(-)
>> > >
>> > > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > > index 7b351e8123..e38d9d6f3f 100644
>> > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
>> > > target_localstatedir := "${localstatedir}"
>> > > OPKGLIBDIR ??= "${target_localstatedir}/lib"
>> > >
>> > > -PACKAGECONFIG ??= "libsolv"
>> > > +PACKAGECONFIG ??= "libsolv zstd"
>> >
>> > Building in zstd support by default is a little suspect to me.
>> >
>> > Unless I'm mistaken, OE-core will only build xz-compressed IPKs by
>> > default. So zstd support would be unnecessary for a distro integrator
>> > who just uses upstream OE-core.
>> >
>> > For distros which use zstd compression in their packages, I think it
>> > would be more appropriate to overwrite the opkg PACKAGECONFIG in a
>> > .bbappend.
>> >
>>
>> This is perhaps fine. I do wonder if there is some performance
>> comparison data between xz and zstd compressed ipks
>> with opkg, it might help users on making this choice and also if we
>> should consider using
>> zstd by default at some point or not.
>>
>> > Is there something I'm not considering here?
>> >
>> > >
>> > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
>> > > gnupg gpgme libgpg-error,\
>> > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
>> > > PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
>> > > PACKAGECONFIG[ssl-curl] =
>> "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
>> > > PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
>> > > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
>> > > PACKAGECONFIG[libsolv] = "--with-libsolv,--without-libsolv,libsolv"
>> > >
>> > > EXTRA_OECONF:class-native =
>> "--localstatedir=/${@os.path.relpath('${localstatedir}',
>> '${STAGING_DIR_NATIVE}')} --sysconfdir=/${@os.path.relpath('${sysconfdir}',
>> '${STAGING_DIR_NATIVE}')}"
>> > >
>> > >
>> > >
>> >
>> > --
>> > Alex Stewart
>> > Software Engineer - NI Real-Time OS
>> > NI (National Instruments)
>> >
>> > alex.stewart@ni.com
>> >
>> >
>> > -=-=-=-=-=-=-=-=-=-=-=-
>> > Links: You receive all messages sent to this group.
>> > View/Reply Online (#170608):
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
>>
>> > Mute This Topic:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
>>
>> > Group Owner: openembedded-core+owner@lists.openembedded.org
>> > Unsubscribe:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
>> [raj.khem@gmail.com]
>> > -=-=-=-=-=-=-=-=-=-=-=-
>> >
>>
>
[-- Attachment #2: Type: text/html, Size: 11245 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [OE-core] [PATCH] opkg: enable zstd support
[not found] ` <CANOoYsMVb9YpfwjTNHv4kSARM15_3T34CA-Ekr=t7P5F0XO4bA@mail.gmail.com>
2022-09-13 20:24 ` Alex Feinman
@ 2022-09-13 20:34 ` Khem Raj
2022-09-13 21:57 ` Alex Stewart
2 siblings, 0 replies; 12+ messages in thread
From: Khem Raj @ 2022-09-13 20:34 UTC (permalink / raw)
To: Alex Feinman
Cc: Alex Stewart, ecordonnier, openembedded-core, Etienne Cordonnier
Thanks for sharing, really appreciate it. How close were compression
rates in numbers ? I think some users might be sensitive to the
archive sizes, if its something
with in acceptable range, I do agree, zstd would be a better default these days.
On Tue, Sep 13, 2022 at 1:20 PM Alex Feinman <afeinman@snap.com> wrote:
>
> I do have some numbers. When I was selling this change internally, I did a comparison on our internal build.
>
> Combined write IPK times (Σ t do_package_write_ipk)
>
> xz 162m 35s
>
> gz 52m 13s
>
> zstd 33m 49s
>
> Compression rate for zstd was closer to xz than to gz but not as good as xz. For systems that have to cache packages on the device with limited storage xz might be a better option, but for the bulk of projects zstd is the best choice
>
> Additionally, zstd offers much faster decompression than xz so the rootfs build step that includes unpacking all of the ipks, takes 3m 58s with xz and 2m 44s with zstd.
>
> One other thing of note - if your build includes debug packages, some may be quite large. E.g. one of our components produces a 2.2 GB debug package (uncompressed). On large files xz requires a disproportionally large amount of time resulting in 15 minutes needed to simply write ipk for the abovementioned packages, whereas zstd took about 45 sec. For frequent tasks like bitbaking a single package this translates in a lot of saved time.
>
> Bottom line - I think making xz a default package compressor was not entirely thought through. gzip or zstd is what the default should be.
>
> One final note: I could not find a reasonable explanation for why opkg-tools require code changes to support a different compressor. BSD tar and GNU tar both can easily accept compressors that they have no idea about (via -I option) because all of them provide a unified command line interface for use in pipes. If this were done similar to tar, we could have used any compressor we wanted, including the multithreaded versions (zstdmt)
>
>
> On Tue, Sep 13, 2022 at 12:43 PM Khem Raj <raj.khem@gmail.com> wrote:
>>
>> On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart <alex.stewart@ni.com> wrote:
>> >
>> > ACK from me - apart from enabling zstd by default.
>> >
>> > On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org wrote:
>> > > This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
>> > > OPKGBUILDCMD = "opkg-build -Z zstd"
>> > >
>> > > Signed-off-by: Alex Feinman <afeinman@snap.com>
>> > > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
>> > > ---
>> > > meta/recipes-devtools/opkg/opkg_0.6.0.bb | 3 ++-
>> > > 1 file changed, 2 insertions(+), 1 deletion(-)
>> > >
>> > > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > > index 7b351e8123..e38d9d6f3f 100644
>> > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
>> > > target_localstatedir := "${localstatedir}"
>> > > OPKGLIBDIR ??= "${target_localstatedir}/lib"
>> > >
>> > > -PACKAGECONFIG ??= "libsolv"
>> > > +PACKAGECONFIG ??= "libsolv zstd"
>> >
>> > Building in zstd support by default is a little suspect to me.
>> >
>> > Unless I'm mistaken, OE-core will only build xz-compressed IPKs by
>> > default. So zstd support would be unnecessary for a distro integrator
>> > who just uses upstream OE-core.
>> >
>> > For distros which use zstd compression in their packages, I think it
>> > would be more appropriate to overwrite the opkg PACKAGECONFIG in a
>> > .bbappend.
>> >
>>
>> This is perhaps fine. I do wonder if there is some performance
>> comparison data between xz and zstd compressed ipks
>> with opkg, it might help users on making this choice and also if we
>> should consider using
>> zstd by default at some point or not.
>>
>> > Is there something I'm not considering here?
>> >
>> > >
>> > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
>> > > gnupg gpgme libgpg-error,\
>> > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
>> > > PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
>> > > PACKAGECONFIG[ssl-curl] = "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
>> > > PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
>> > > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
>> > > PACKAGECONFIG[libsolv] = "--with-libsolv,--without-libsolv,libsolv"
>> > >
>> > > EXTRA_OECONF:class-native = "--localstatedir=/${@os.path.relpath('${localstatedir}', '${STAGING_DIR_NATIVE}')} --sysconfdir=/${@os.path.relpath('${sysconfdir}', '${STAGING_DIR_NATIVE}')}"
>> > >
>> > >
>> > >
>> >
>> > --
>> > Alex Stewart
>> > Software Engineer - NI Real-Time OS
>> > NI (National Instruments)
>> >
>> > alex.stewart@ni.com
>> >
>> >
>> > -=-=-=-=-=-=-=-=-=-=-=-
>> > Links: You receive all messages sent to this group.
>> > View/Reply Online (#170608): https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
>> > Mute This Topic: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
>> > Group Owner: openembedded-core+owner@lists.openembedded.org
>> > Unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e= [raj.khem@gmail.com]
>> > -=-=-=-=-=-=-=-=-=-=-=-
>> >
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Re: [OE-core] [PATCH] opkg: enable zstd support
[not found] ` <CANOoYsMVb9YpfwjTNHv4kSARM15_3T34CA-Ekr=t7P5F0XO4bA@mail.gmail.com>
2022-09-13 20:24 ` Alex Feinman
2022-09-13 20:34 ` Khem Raj
@ 2022-09-13 21:57 ` Alex Stewart
2022-09-14 9:58 ` Etienne Cordonnier
2 siblings, 1 reply; 12+ messages in thread
From: Alex Stewart @ 2022-09-13 21:57 UTC (permalink / raw)
To: Alex Feinman, Khem Raj; +Cc: ecordonnier, openembedded-core, Etienne Cordonnier
On 9/13/22 15:20, Alex Feinman wrote:
> I do have some numbers. When I was selling this change internally, I
> did a comparison on our internal build.
> Combined write IPK times (Σ t do_package_write_ipk)
> xz 162m 35s
> gz 52m 13s
> zstd 33m 49s
> Compression rate for zstd was closer to xz than to gz but not as good
> as xz. For systems that have to cache packages on the device with
> limited storage xz might be a better option, but for the bulk of
> projects zstd is the best choice
> Additionally, zstd offers much faster decompression than xz so the
> rootfs build step that includes unpacking all of the ipks, takes 3m
> 58s with xz and 2m 44s with zstd.
> One other thing of note - if your build includes debug packages, some
> may be quite large. E.g. one of our components produces a 2.2 GB debug
> package (uncompressed). On large files xz requires a disproportionally
> large amount of time resulting in 15 minutes needed to simply write
> ipk for the abovementioned packages, whereas zstd took about 45 sec.
> For frequent tasks like bitbaking a single package this translates in
> a lot of saved time.
Those are certainly compelling performance improvements. Assuming that
the final data-segment size is within 5%-ish of xz, then I would agree
with the rest of the thread that it should probably be the contemporary
default.
And if we make it the default compressor for OE IPKs, then obviously my
criticism in the original PR is satisfied.
> Bottom line - I think making xz a default package compressor was not
> entirely thought through. gzip or zstd is what the default should be.
ZStandard support was only added to opkg last September [1]. Before
that, xz was the new hotness that replaced gzip. :)
[1]
https://git.yoctoproject.org/opkg/commit/?id=5dead419e94bce2e6b743ad786c1daec0e1aa294
> One final note: I could not find a reasonable explanation for why
> opkg-tools require code changes to support a different compressor. BSD
> tar and GNU tar both can easily accept compressors that they have no
> idea about (via -I option) because all of them provide a unified
> command line interface for use in pipes. If this were done similar to
> tar, we could have used any compressor we wanted, including the
> multithreaded versions (zstdmt)
Well, presumably IPK creation tools can only support the matrix of
compression algorithms which your opkg binary can decompress. I suppose
someone could try to implement a plugable compression module system for
opkg. But given that nearly everyone uses opkg in an embedded context,
I'm not sure it would get much use.
>
> On Tue, Sep 13, 2022 at 12:43 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
> <alex.stewart@ni.com> wrote:
> >
> > ACK from me - apart from enabling zstd by default.
> >
> > On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org
> <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$>
> wrote:
> > > This allows the use of zstd for opkg packages by using
> OPKGBUILDCMD:
> > > OPKGBUILDCMD = "opkg-build -Z zstd"
> > >
> > > Signed-off-by: Alex Feinman <afeinman@snap.com>
> > > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
> > > ---
> > > meta/recipes-devtools/opkg/opkg_0.6.0.bb
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> | 3 ++-
> > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > index 7b351e8123..e38d9d6f3f 100644
> > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
> > > target_localstatedir := "${localstatedir}"
> > > OPKGLIBDIR ??= "${target_localstatedir}/lib"
> > >
> > > -PACKAGECONFIG ??= "libsolv"
> > > +PACKAGECONFIG ??= "libsolv zstd"
> >
> > Building in zstd support by default is a little suspect to me.
> >
> > Unless I'm mistaken, OE-core will only build xz-compressed IPKs by
> > default. So zstd support would be unnecessary for a distro
> integrator
> > who just uses upstream OE-core.
> >
> > For distros which use zstd compression in their packages, I think it
> > would be more appropriate to overwrite the opkg PACKAGECONFIG in a
> > .bbappend.
> >
>
> This is perhaps fine. I do wonder if there is some performance
> comparison data between xz and zstd compressed ipks
> with opkg, it might help users on making this choice and also if we
> should consider using
> zstd by default at some point or not.
>
> > Is there something I'm not considering here?
> >
> > >
> > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
> > > gnupg gpgme libgpg-error,\
> > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
> "--enable-gpg,--disable-gpg,\
> > > PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
> > > PACKAGECONFIG[ssl-curl] =
> "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
> > > PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
> > > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
> > > PACKAGECONFIG[libsolv] =
> "--with-libsolv,--without-libsolv,libsolv"
> > >
> > > EXTRA_OECONF:class-native =
> "--localstatedir=/${@os.path.relpath('${localstatedir}',
> '${STAGING_DIR_NATIVE}')}
> --sysconfdir=/${@os.path.relpath('${sysconfdir}',
> '${STAGING_DIR_NATIVE}')}"
> > >
> > >
> > >
> >
> > --
> > Alex Stewart
> > Software Engineer - NI Real-Time OS
> > NI (National Instruments)
> >
> > alex.stewart@ni.com
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#170608):
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>
>
> > Mute This Topic:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>
>
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> <mailto:openembedded-core%2Bowner@lists.openembedded.org>
> > Unsubscribe:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>
> [raj.khem@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)
alex.stewart@ni.com
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Re: [OE-core] [PATCH] opkg: enable zstd support
2022-09-13 21:57 ` Alex Stewart
@ 2022-09-14 9:58 ` Etienne Cordonnier
2022-09-14 10:08 ` Etienne Cordonnier
0 siblings, 1 reply; 12+ messages in thread
From: Etienne Cordonnier @ 2022-09-14 9:58 UTC (permalink / raw)
To: Alex Stewart
Cc: Alex Feinman, Khem Raj, openembedded-core, Etienne Cordonnier
[-- Attachment #1: Type: text/plain, Size: 12165 bytes --]
I ran a build of core-image-full-cmdline using xz and zstd, using
pre-populated downloads and sstate-cache directories but with empty tmp
directory. Here are the numbers:
zstd:
bitbake core-image-full-cmdline took 2m52.768s (real), the resulting
directory tmp/deploy/ipk is 1.6GB big.
xz:
bitbake core-image-full-cmdline took 4m14.214s (real), the resulting
directory tmp/deploy/ipk/ is 996M big
So the build with xz is 47% slower (254/172) than with zstd for this
"incremental build" use-case.
See the 5 biggest packages, the difference in compression-ratio increases
with big files:
build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
total 1.6G
-rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39
binutils-dbg_2.39-r0_core2-64.ipk
build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
total 963M
-rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
binutils-dbg_2.39-r0_core2-64.ipk
...
I think for use-cases where the size of the ipk packages matters, xz is
better. For use-cases where it does not matter (ipk packages not deployed),
build-time matters more than compression-ratio and zstd is better.
Regarding the enablement of zstd in opkg per default, I'll send a new
version of the patch without this line.
My thinking for enabling zstd per default in opkg was that zstd is already
enabled per default in libarchive's PACKAGECONFIG, and disabling zstd in
opkg's PACKAGECONFIG removes only a few lines of code from opkg (opkg uses
libarchive for doing the actual compression/decompression).
On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart <alex.stewart@ni.com> wrote:
>
>
> On 9/13/22 15:20, Alex Feinman wrote:
> > I do have some numbers. When I was selling this change internally, I
> > did a comparison on our internal build.
> > Combined write IPK times (Σ t do_package_write_ipk)
> > xz 162m 35s
> > gz 52m 13s
> > zstd 33m 49s
> > Compression rate for zstd was closer to xz than to gz but not as good
> > as xz. For systems that have to cache packages on the device with
> > limited storage xz might be a better option, but for the bulk of
> > projects zstd is the best choice
> > Additionally, zstd offers much faster decompression than xz so the
> > rootfs build step that includes unpacking all of the ipks, takes 3m
> > 58s with xz and 2m 44s with zstd.
> > One other thing of note - if your build includes debug packages, some
> > may be quite large. E.g. one of our components produces a 2.2 GB debug
> > package (uncompressed). On large files xz requires a disproportionally
> > large amount of time resulting in 15 minutes needed to simply write
> > ipk for the abovementioned packages, whereas zstd took about 45 sec.
> > For frequent tasks like bitbaking a single package this translates in
> > a lot of saved time.
>
> Those are certainly compelling performance improvements. Assuming that
> the final data-segment size is within 5%-ish of xz, then I would agree
> with the rest of the thread that it should probably be the contemporary
> default.
>
> And if we make it the default compressor for OE IPKs, then obviously my
> criticism in the original PR is satisfied.
>
> > Bottom line - I think making xz a default package compressor was not
> > entirely thought through. gzip or zstd is what the default should be.
>
> ZStandard support was only added to opkg last September [1]. Before
> that, xz was the new hotness that replaced gzip. :)
>
> [1]
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
>
>
> > One final note: I could not find a reasonable explanation for why
> > opkg-tools require code changes to support a different compressor. BSD
> > tar and GNU tar both can easily accept compressors that they have no
> > idea about (via -I option) because all of them provide a unified
> > command line interface for use in pipes. If this were done similar to
> > tar, we could have used any compressor we wanted, including the
> > multithreaded versions (zstdmt)
>
> Well, presumably IPK creation tools can only support the matrix of
> compression algorithms which your opkg binary can decompress. I suppose
> someone could try to implement a plugable compression module system for
> opkg. But given that nearly everyone uses opkg in an embedded context,
> I'm not sure it would get much use.
>
> >
> > On Tue, Sep 13, 2022 at 12:43 PM Khem Raj <raj.khem@gmail.com> wrote:
> >
> > On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
> > <alex.stewart@ni.com> wrote:
> > >
> > > ACK from me - apart from enabling zstd by default.
> > >
> > > On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org
> > <
> https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$
> >
> > wrote:
> > > > This allows the use of zstd for opkg packages by using
> > OPKGBUILDCMD:
> > > > OPKGBUILDCMD = "opkg-build -Z zstd"
> > > >
> > > > Signed-off-by: Alex Feinman <afeinman@snap.com>
> > > > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
> > > > ---
> > > > meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > | 3 ++-
> > > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > index 7b351e8123..e38d9d6f3f 100644
> > > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
> > > > target_localstatedir := "${localstatedir}"
> > > > OPKGLIBDIR ??= "${target_localstatedir}/lib"
> > > >
> > > > -PACKAGECONFIG ??= "libsolv"
> > > > +PACKAGECONFIG ??= "libsolv zstd"
> > >
> > > Building in zstd support by default is a little suspect to me.
> > >
> > > Unless I'm mistaken, OE-core will only build xz-compressed IPKs by
> > > default. So zstd support would be unnecessary for a distro
> > integrator
> > > who just uses upstream OE-core.
> > >
> > > For distros which use zstd compression in their packages, I think
> it
> > > would be more appropriate to overwrite the opkg PACKAGECONFIG in a
> > > .bbappend.
> > >
> >
> > This is perhaps fine. I do wonder if there is some performance
> > comparison data between xz and zstd compressed ipks
> > with opkg, it might help users on making this choice and also if we
> > should consider using
> > zstd by default at some point or not.
> >
> > > Is there something I'm not considering here?
> > >
> > > >
> > > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
> > > > gnupg gpgme libgpg-error,\
> > > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
> > "--enable-gpg,--disable-gpg,\
> > > > PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
> > > > PACKAGECONFIG[ssl-curl] =
> > "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
> > > > PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
> > > > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
> > > > PACKAGECONFIG[libsolv] =
> > "--with-libsolv,--without-libsolv,libsolv"
> > > >
> > > > EXTRA_OECONF:class-native =
> > "--localstatedir=/${@os.path.relpath('${localstatedir}',
> > '${STAGING_DIR_NATIVE}')}
> > --sysconfdir=/${@os.path.relpath('${sysconfdir}',
> > '${STAGING_DIR_NATIVE}')}"
> > > >
> > > >
> > > >
> > >
> > > --
> > > Alex Stewart
> > > Software Engineer - NI Real-Time OS
> > > NI (National Instruments)
> > >
> > > alex.stewart@ni.com
> > >
> > >
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > > Links: You receive all messages sent to this group.
> > > View/Reply Online (#170608):
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
> > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
> >
> >
> > > Mute This Topic:
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
> > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
> >
> >
> > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > <mailto:openembedded-core%2Bowner@lists.openembedded.org>
> > > Unsubscribe:
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
> > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
> >
> > [raj.khem@gmail.com]
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > >
> >
>
> --
> Alex Stewart
> Software Engineer - NI Real-Time OS
> NI (National Instruments)
>
> alex.stewart@ni.com
>
>
[-- Attachment #2: Type: text/html, Size: 19212 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Re: [OE-core] [PATCH] opkg: enable zstd support
2022-09-14 9:58 ` Etienne Cordonnier
@ 2022-09-14 10:08 ` Etienne Cordonnier
2022-09-14 15:37 ` Alex Stewart
0 siblings, 1 reply; 12+ messages in thread
From: Etienne Cordonnier @ 2022-09-14 10:08 UTC (permalink / raw)
To: Alex Stewart
Cc: Alex Feinman, Khem Raj, openembedded-core, Etienne Cordonnier
[-- Attachment #1: Type: text/plain, Size: 12680 bytes --]
Also note that zstd's default compression level is 3 per default (from a 1
to 19 scale). I did not test other compression levels.
On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier <
ecordonnier@snapchat.com> wrote:
> I ran a build of core-image-full-cmdline using xz and zstd, using
> pre-populated downloads and sstate-cache directories but with empty tmp
> directory. Here are the numbers:
> zstd:
> bitbake core-image-full-cmdline took 2m52.768s (real), the resulting
> directory tmp/deploy/ipk is 1.6GB big.
> xz:
> bitbake core-image-full-cmdline took 4m14.214s (real), the resulting
> directory tmp/deploy/ipk/ is 996M big
>
> So the build with xz is 47% slower (254/172) than with zstd for this
> "incremental build" use-case.
>
> See the 5 biggest packages, the difference in compression-ratio increases
> with big files:
> build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
> total 1.6G
> -rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39
> binutils-dbg_2.39-r0_core2-64.ipk
> build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
> total 963M
> -rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
> binutils-dbg_2.39-r0_core2-64.ipk
> ...
>
> I think for use-cases where the size of the ipk packages matters, xz is
> better. For use-cases where it does not matter (ipk packages not deployed),
> build-time matters more than compression-ratio and zstd is better.
>
> Regarding the enablement of zstd in opkg per default, I'll send a new
> version of the patch without this line.
> My thinking for enabling zstd per default in opkg was that zstd is already
> enabled per default in libarchive's PACKAGECONFIG, and disabling zstd in
> opkg's PACKAGECONFIG removes only a few lines of code from opkg (opkg uses
> libarchive for doing the actual compression/decompression).
>
> On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart <alex.stewart@ni.com> wrote:
>
>>
>>
>> On 9/13/22 15:20, Alex Feinman wrote:
>> > I do have some numbers. When I was selling this change internally, I
>> > did a comparison on our internal build.
>> > Combined write IPK times (Σ t do_package_write_ipk)
>> > xz 162m 35s
>> > gz 52m 13s
>> > zstd 33m 49s
>> > Compression rate for zstd was closer to xz than to gz but not as good
>> > as xz. For systems that have to cache packages on the device with
>> > limited storage xz might be a better option, but for the bulk of
>> > projects zstd is the best choice
>> > Additionally, zstd offers much faster decompression than xz so the
>> > rootfs build step that includes unpacking all of the ipks, takes 3m
>> > 58s with xz and 2m 44s with zstd.
>> > One other thing of note - if your build includes debug packages, some
>> > may be quite large. E.g. one of our components produces a 2.2 GB debug
>> > package (uncompressed). On large files xz requires a disproportionally
>> > large amount of time resulting in 15 minutes needed to simply write
>> > ipk for the abovementioned packages, whereas zstd took about 45 sec.
>> > For frequent tasks like bitbaking a single package this translates in
>> > a lot of saved time.
>>
>> Those are certainly compelling performance improvements. Assuming that
>> the final data-segment size is within 5%-ish of xz, then I would agree
>> with the rest of the thread that it should probably be the contemporary
>> default.
>>
>> And if we make it the default compressor for OE IPKs, then obviously my
>> criticism in the original PR is satisfied.
>>
>> > Bottom line - I think making xz a default package compressor was not
>> > entirely thought through. gzip or zstd is what the default should be.
>>
>> ZStandard support was only added to opkg last September [1]. Before
>> that, xz was the new hotness that replaced gzip. :)
>>
>> [1]
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
>>
>>
>> > One final note: I could not find a reasonable explanation for why
>> > opkg-tools require code changes to support a different compressor. BSD
>> > tar and GNU tar both can easily accept compressors that they have no
>> > idea about (via -I option) because all of them provide a unified
>> > command line interface for use in pipes. If this were done similar to
>> > tar, we could have used any compressor we wanted, including the
>> > multithreaded versions (zstdmt)
>>
>> Well, presumably IPK creation tools can only support the matrix of
>> compression algorithms which your opkg binary can decompress. I suppose
>> someone could try to implement a plugable compression module system for
>> opkg. But given that nearly everyone uses opkg in an embedded context,
>> I'm not sure it would get much use.
>>
>> >
>> > On Tue, Sep 13, 2022 at 12:43 PM Khem Raj <raj.khem@gmail.com> wrote:
>> >
>> > On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
>> > <alex.stewart@ni.com> wrote:
>> > >
>> > > ACK from me - apart from enabling zstd by default.
>> > >
>> > > On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org
>> > <
>> https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$
>> >
>> > wrote:
>> > > > This allows the use of zstd for opkg packages by using
>> > OPKGBUILDCMD:
>> > > > OPKGBUILDCMD = "opkg-build -Z zstd"
>> > > >
>> > > > Signed-off-by: Alex Feinman <afeinman@snap.com>
>> > > > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
>> > > > ---
>> > > > meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > <
>> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
>> >
>> > | 3 ++-
>> > > > 1 file changed, 2 insertions(+), 1 deletion(-)
>> > > >
>> > > > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > <
>> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
>> >
>> > b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > <
>> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
>> >
>> > > > index 7b351e8123..e38d9d6f3f 100644
>> > > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > <
>> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
>> >
>> > > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>> > <
>> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
>> >
>> > > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
>> > > > target_localstatedir := "${localstatedir}"
>> > > > OPKGLIBDIR ??= "${target_localstatedir}/lib"
>> > > >
>> > > > -PACKAGECONFIG ??= "libsolv"
>> > > > +PACKAGECONFIG ??= "libsolv zstd"
>> > >
>> > > Building in zstd support by default is a little suspect to me.
>> > >
>> > > Unless I'm mistaken, OE-core will only build xz-compressed IPKs by
>> > > default. So zstd support would be unnecessary for a distro
>> > integrator
>> > > who just uses upstream OE-core.
>> > >
>> > > For distros which use zstd compression in their packages, I think
>> it
>> > > would be more appropriate to overwrite the opkg PACKAGECONFIG in a
>> > > .bbappend.
>> > >
>> >
>> > This is perhaps fine. I do wonder if there is some performance
>> > comparison data between xz and zstd compressed ipks
>> > with opkg, it might help users on making this choice and also if we
>> > should consider using
>> > zstd by default at some point or not.
>> >
>> > > Is there something I'm not considering here?
>> > >
>> > > >
>> > > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
>> > > > gnupg gpgme libgpg-error,\
>> > > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
>> > "--enable-gpg,--disable-gpg,\
>> > > > PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
>> > > > PACKAGECONFIG[ssl-curl] =
>> > "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
>> > > > PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
>> > > > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
>> > > > PACKAGECONFIG[libsolv] =
>> > "--with-libsolv,--without-libsolv,libsolv"
>> > > >
>> > > > EXTRA_OECONF:class-native =
>> > "--localstatedir=/${@os.path.relpath('${localstatedir}',
>> > '${STAGING_DIR_NATIVE}')}
>> > --sysconfdir=/${@os.path.relpath('${sysconfdir}',
>> > '${STAGING_DIR_NATIVE}')}"
>> > > >
>> > > >
>> > > >
>> > >
>> > > --
>> > > Alex Stewart
>> > > Software Engineer - NI Real-Time OS
>> > > NI (National Instruments)
>> > >
>> > > alex.stewart@ni.com
>> > >
>> > >
>> > > -=-=-=-=-=-=-=-=-=-=-=-
>> > > Links: You receive all messages sent to this group.
>> > > View/Reply Online (#170608):
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
>> > <
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
>> >
>> >
>> > > Mute This Topic:
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
>> > <
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
>> >
>> >
>> > > Group Owner: openembedded-core+owner@lists.openembedded.org
>> > <mailto:openembedded-core%2Bowner@lists.openembedded.org>
>> > > Unsubscribe:
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
>> > <
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
>> >
>> > [raj.khem@gmail.com]
>> > > -=-=-=-=-=-=-=-=-=-=-=-
>> > >
>> >
>>
>> --
>> Alex Stewart
>> Software Engineer - NI Real-Time OS
>> NI (National Instruments)
>>
>> alex.stewart@ni.com
>>
>>
[-- Attachment #2: Type: text/html, Size: 19734 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Re: Re: [OE-core] [PATCH] opkg: enable zstd support
2022-09-14 10:08 ` Etienne Cordonnier
@ 2022-09-14 15:37 ` Alex Stewart
2022-09-14 15:41 ` Khem Raj
0 siblings, 1 reply; 12+ messages in thread
From: Alex Stewart @ 2022-09-14 15:37 UTC (permalink / raw)
To: Etienne Cordonnier
Cc: Alex Feinman, Khem Raj, openembedded-core, Etienne Cordonnier
Thanks for checking.
I'd be interested to know if setting a higher compression level for zstd
can get us to a similar compression ratio to xz. If so, then I think it
could be some real value to distro maintainers to be able to *tune*
their compression.
That's not blocking for your new PR though.
On 9/14/22 05:08, Etienne Cordonnier wrote:
> Also note that zstd's default compression level is 3 per default (from
> a 1 to 19 scale). I did not test other compression levels.
>
> On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier
> <ecordonnier@snapchat.com> wrote:
>
> I ran a build of core-image-full-cmdline using xz and zstd, using
> pre-populated downloads and sstate-cache directories but with
> empty tmp directory. Here are the numbers:
> zstd:
> bitbake core-image-full-cmdline took 2m52.768s (real), the
> resulting directory tmp/deploy/ipk is 1.6GB big.
> xz:
> bitbake core-image-full-cmdline took 4m14.214s (real), the
> resulting directory tmp/deploy/ipk/ is 996M big
>
> So the build with xz is 47% slower (254/172) than with zstd for
> this "incremental build" use-case.
>
> See the 5 biggest packages, the difference in compression-ratio
> increases with big files:
> build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
> total 1.6G
> -rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39
> binutils-dbg_2.39-r0_core2-64.ipk
> build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
> total 963M
> -rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
> binutils-dbg_2.39-r0_core2-64.ipk
> ...
>
> I think for use-cases where the size of the ipk packages matters,
> xz is better. For use-cases where it does not matter (ipk packages
> not deployed), build-time matters more than compression-ratio and
> zstd is better.
>
> Regarding the enablement of zstd in opkg per default, I'll send a
> new version of the patch without this line.
> My thinking for enabling zstd per default in opkg was that zstd is
> already enabled per default in libarchive's PACKAGECONFIG, and
> disabling zstd in opkg's PACKAGECONFIG removes only a few lines of
> code from opkg (opkg uses libarchive for doing the actual
> compression/decompression).
>
> On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart
> <alex.stewart@ni.com> wrote:
>
>
>
> On 9/13/22 15:20, Alex Feinman wrote:
> > I do have some numbers. When I was selling this change
> internally, I
> > did a comparison on our internal build.
> > Combined write IPK times (Σ t do_package_write_ipk)
> > xz 162m 35s
> > gz 52m 13s
> > zstd 33m 49s
> > Compression rate for zstd was closer to xz than to gz but
> not as good
> > as xz. For systems that have to cache packages on the device
> with
> > limited storage xz might be a better option, but for the
> bulk of
> > projects zstd is the best choice
> > Additionally, zstd offers much faster decompression than xz
> so the
> > rootfs build step that includes unpacking all of the ipks,
> takes 3m
> > 58s with xz and 2m 44s with zstd.
> > One other thing of note - if your build includes debug
> packages, some
> > may be quite large. E.g. one of our components produces a
> 2.2 GB debug
> > package (uncompressed). On large files xz requires a
> disproportionally
> > large amount of time resulting in 15 minutes needed to
> simply write
> > ipk for the abovementioned packages, whereas zstd took about
> 45 sec.
> > For frequent tasks like bitbaking a single package this
> translates in
> > a lot of saved time.
>
> Those are certainly compelling performance improvements.
> Assuming that
> the final data-segment size is within 5%-ish of xz, then I
> would agree
> with the rest of the thread that it should probably be the
> contemporary
> default.
>
> And if we make it the default compressor for OE IPKs, then
> obviously my
> criticism in the original PR is satisfied.
>
> > Bottom line - I think making xz a default package compressor
> was not
> > entirely thought through. gzip or zstd is what the default
> should be.
>
> ZStandard support was only added to opkg last September [1].
> Before
> that, xz was the new hotness that replaced gzip. :)
>
> [1]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=>
>
>
> > One final note: I could not find a reasonable explanation
> for why
> > opkg-tools require code changes to support a different
> compressor. BSD
> > tar and GNU tar both can easily accept compressors that they
> have no
> > idea about (via -I option) because all of them provide a
> unified
> > command line interface for use in pipes. If this were done
> similar to
> > tar, we could have used any compressor we wanted, including the
> > multithreaded versions (zstdmt)
>
> Well, presumably IPK creation tools can only support the
> matrix of
> compression algorithms which your opkg binary can decompress.
> I suppose
> someone could try to implement a plugable compression module
> system for
> opkg. But given that nearly everyone uses opkg in an embedded
> context,
> I'm not sure it would get much use.
>
> >
> > On Tue, Sep 13, 2022 at 12:43 PM Khem Raj
> <raj.khem@gmail.com> wrote:
> >
> > On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
> > <alex.stewart@ni.com> wrote:
> > >
> > > ACK from me - apart from enabling zstd by default.
> > >
> > > On 9/13/22 07:37, Etienne Cordonnier via
> lists.openembedded.org
> <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFhOF1_vg$>
> >
> <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$>
> > wrote:
> > > > This allows the use of zstd for opkg packages by using
> > OPKGBUILDCMD:
> > > > OPKGBUILDCMD = "opkg-build -Z zstd"
> > > >
> > > > Signed-off-by: Alex Feinman <afeinman@snap.com>
> > > > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
> > > > ---
> > > > meta/recipes-devtools/opkg/opkg_0.6.0.bb
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
> >
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > | 3 ++-
> > > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git
> a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
> >
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
> >
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > > index 7b351e8123..e38d9d6f3f 100644
> > > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
> >
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
> >
> <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
> > > > target_localstatedir := "${localstatedir}"
> > > > OPKGLIBDIR ??= "${target_localstatedir}/lib"
> > > >
> > > > -PACKAGECONFIG ??= "libsolv"
> > > > +PACKAGECONFIG ??= "libsolv zstd"
> > >
> > > Building in zstd support by default is a little
> suspect to me.
> > >
> > > Unless I'm mistaken, OE-core will only build
> xz-compressed IPKs by
> > > default. So zstd support would be unnecessary for a distro
> > integrator
> > > who just uses upstream OE-core.
> > >
> > > For distros which use zstd compression in their
> packages, I think it
> > > would be more appropriate to overwrite the opkg
> PACKAGECONFIG in a
> > > .bbappend.
> > >
> >
> > This is perhaps fine. I do wonder if there is some
> performance
> > comparison data between xz and zstd compressed ipks
> > with opkg, it might help users on making this choice and
> also if we
> > should consider using
> > zstd by default at some point or not.
> >
> > > Is there something I'm not considering here?
> > >
> > > >
> > > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
> > > > gnupg gpgme libgpg-error,\
> > > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
> > "--enable-gpg,--disable-gpg,\
> > > > PACKAGECONFIG[curl] =
> "--enable-curl,--disable-curl,curl"
> > > > PACKAGECONFIG[ssl-curl] =
> > "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
> > > > PACKAGECONFIG[sha256] =
> "--enable-sha256,--disable-sha256"
> > > > +PACKAGECONFIG[zstd] =
> "--enable-zstd,--disable-zstd,zstd"
> > > > PACKAGECONFIG[libsolv] =
> > "--with-libsolv,--without-libsolv,libsolv"
> > > >
> > > > EXTRA_OECONF:class-native =
> > "--localstatedir=/${@os.path.relpath('${localstatedir}',
> > '${STAGING_DIR_NATIVE}')}
> > --sysconfdir=/${@os.path.relpath('${sysconfdir}',
> > '${STAGING_DIR_NATIVE}')}"
> > > >
> > > >
> > > >
> > >
> > > --
> > > Alex Stewart
> > > Software Engineer - NI Real-Time OS
> > > NI (National Instruments)
> > >
> > > alex.stewart@ni.com
> > >
> > >
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > > Links: You receive all messages sent to this group.
> > > View/Reply Online (#170608):
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>>
> >
> > > Mute This Topic:
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>>
> >
> > > Group Owner:
> openembedded-core+owner@lists.openembedded.org
> <mailto:openembedded-core%2Bowner@lists.openembedded.org>
> > <mailto:openembedded-core%2Bowner@lists.openembedded.org
> <mailto:openembedded-core%252Bowner@lists.openembedded.org>>
> > > Unsubscribe:
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>
> >
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>>
> > [raj.khem@gmail.com]
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > >
> >
>
> --
> Alex Stewart
> Software Engineer - NI Real-Time OS
> NI (National Instruments)
>
> alex.stewart@ni.com
>
--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)
alex.stewart@ni.com
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Re: Re: [OE-core] [PATCH] opkg: enable zstd support
2022-09-14 15:37 ` Alex Stewart
@ 2022-09-14 15:41 ` Khem Raj
2022-09-28 16:50 ` Etienne Cordonnier
0 siblings, 1 reply; 12+ messages in thread
From: Khem Raj @ 2022-09-14 15:41 UTC (permalink / raw)
To: Alex Stewart
Cc: Etienne Cordonnier, Alex Feinman, openembedded-core, Etienne Cordonnier
On Wed, Sep 14, 2022 at 8:37 AM Alex Stewart <alex.stewart@ni.com> wrote:
>
> Thanks for checking.
>
> I'd be interested to know if setting a higher compression level for zstd
> can get us to a similar compression ratio to xz. If so, then I think it
> could be some real value to distro maintainers to be able to *tune*
> their compression.
yeah it will be interesting to say try something like level 9 but I think times
might regress with that but it might be good to know the balance and perhaps
suggest size mode and performance mode of zstd instead of xz
>
> That's not blocking for your new PR though.
>
>
> On 9/14/22 05:08, Etienne Cordonnier wrote:
> > Also note that zstd's default compression level is 3 per default (from
> > a 1 to 19 scale). I did not test other compression levels.
> >
> > On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier
> > <ecordonnier@snapchat.com> wrote:
> >
> > I ran a build of core-image-full-cmdline using xz and zstd, using
> > pre-populated downloads and sstate-cache directories but with
> > empty tmp directory. Here are the numbers:
> > zstd:
> > bitbake core-image-full-cmdline took 2m52.768s (real), the
> > resulting directory tmp/deploy/ipk is 1.6GB big.
> > xz:
> > bitbake core-image-full-cmdline took 4m14.214s (real), the
> > resulting directory tmp/deploy/ipk/ is 996M big
> >
> > So the build with xz is 47% slower (254/172) than with zstd for
> > this "incremental build" use-case.
> >
> > See the 5 biggest packages, the difference in compression-ratio
> > increases with big files:
> > build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
> > total 1.6G
> > -rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39
> > gcc-dbg_12.2.0-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39
> > openssl-dbg_3.0.5-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39
> > openssl-ptest_3.0.5-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39
> > binutils-dbg_2.39-r0_core2-64.ipk
> > build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
> > total 963M
> > -rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44
> > gcc-dbg_12.2.0-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44
> > openssl-dbg_3.0.5-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
> > openssl-ptest_3.0.5-r0_core2-64.ipk
> > -rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
> > binutils-dbg_2.39-r0_core2-64.ipk
> > ...
> >
> > I think for use-cases where the size of the ipk packages matters,
> > xz is better. For use-cases where it does not matter (ipk packages
> > not deployed), build-time matters more than compression-ratio and
> > zstd is better.
> >
> > Regarding the enablement of zstd in opkg per default, I'll send a
> > new version of the patch without this line.
> > My thinking for enabling zstd per default in opkg was that zstd is
> > already enabled per default in libarchive's PACKAGECONFIG, and
> > disabling zstd in opkg's PACKAGECONFIG removes only a few lines of
> > code from opkg (opkg uses libarchive for doing the actual
> > compression/decompression).
> >
> > On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart
> > <alex.stewart@ni.com> wrote:
> >
> >
> >
> > On 9/13/22 15:20, Alex Feinman wrote:
> > > I do have some numbers. When I was selling this change
> > internally, I
> > > did a comparison on our internal build.
> > > Combined write IPK times (Σ t do_package_write_ipk)
> > > xz 162m 35s
> > > gz 52m 13s
> > > zstd 33m 49s
> > > Compression rate for zstd was closer to xz than to gz but
> > not as good
> > > as xz. For systems that have to cache packages on the device
> > with
> > > limited storage xz might be a better option, but for the
> > bulk of
> > > projects zstd is the best choice
> > > Additionally, zstd offers much faster decompression than xz
> > so the
> > > rootfs build step that includes unpacking all of the ipks,
> > takes 3m
> > > 58s with xz and 2m 44s with zstd.
> > > One other thing of note - if your build includes debug
> > packages, some
> > > may be quite large. E.g. one of our components produces a
> > 2.2 GB debug
> > > package (uncompressed). On large files xz requires a
> > disproportionally
> > > large amount of time resulting in 15 minutes needed to
> > simply write
> > > ipk for the abovementioned packages, whereas zstd took about
> > 45 sec.
> > > For frequent tasks like bitbaking a single package this
> > translates in
> > > a lot of saved time.
> >
> > Those are certainly compelling performance improvements.
> > Assuming that
> > the final data-segment size is within 5%-ish of xz, then I
> > would agree
> > with the rest of the thread that it should probably be the
> > contemporary
> > default.
> >
> > And if we make it the default compressor for OE IPKs, then
> > obviously my
> > criticism in the original PR is satisfied.
> >
> > > Bottom line - I think making xz a default package compressor
> > was not
> > > entirely thought through. gzip or zstd is what the default
> > should be.
> >
> > ZStandard support was only added to opkg last September [1].
> > Before
> > that, xz was the new hotness that replaced gzip. :)
> >
> > [1]
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=>
> >
> >
> > > One final note: I could not find a reasonable explanation
> > for why
> > > opkg-tools require code changes to support a different
> > compressor. BSD
> > > tar and GNU tar both can easily accept compressors that they
> > have no
> > > idea about (via -I option) because all of them provide a
> > unified
> > > command line interface for use in pipes. If this were done
> > similar to
> > > tar, we could have used any compressor we wanted, including the
> > > multithreaded versions (zstdmt)
> >
> > Well, presumably IPK creation tools can only support the
> > matrix of
> > compression algorithms which your opkg binary can decompress.
> > I suppose
> > someone could try to implement a plugable compression module
> > system for
> > opkg. But given that nearly everyone uses opkg in an embedded
> > context,
> > I'm not sure it would get much use.
> >
> > >
> > > On Tue, Sep 13, 2022 at 12:43 PM Khem Raj
> > <raj.khem@gmail.com> wrote:
> > >
> > > On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
> > > <alex.stewart@ni.com> wrote:
> > > >
> > > > ACK from me - apart from enabling zstd by default.
> > > >
> > > > On 9/13/22 07:37, Etienne Cordonnier via
> > lists.openembedded.org
> > <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFhOF1_vg$>
> > >
> > <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$>
> > > wrote:
> > > > > This allows the use of zstd for opkg packages by using
> > > OPKGBUILDCMD:
> > > > > OPKGBUILDCMD = "opkg-build -Z zstd"
> > > > >
> > > > > Signed-off-by: Alex Feinman <afeinman@snap.com>
> > > > > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
> > > > > ---
> > > > > meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
> > >
> > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > | 3 ++-
> > > > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > > > >
> > > > > diff --git
> > a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
> > >
> > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
> > >
> > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > > > index 7b351e8123..e38d9d6f3f 100644
> > > > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
> > >
> > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
> > >
> > <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
> > > > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
> > > > > target_localstatedir := "${localstatedir}"
> > > > > OPKGLIBDIR ??= "${target_localstatedir}/lib"
> > > > >
> > > > > -PACKAGECONFIG ??= "libsolv"
> > > > > +PACKAGECONFIG ??= "libsolv zstd"
> > > >
> > > > Building in zstd support by default is a little
> > suspect to me.
> > > >
> > > > Unless I'm mistaken, OE-core will only build
> > xz-compressed IPKs by
> > > > default. So zstd support would be unnecessary for a distro
> > > integrator
> > > > who just uses upstream OE-core.
> > > >
> > > > For distros which use zstd compression in their
> > packages, I think it
> > > > would be more appropriate to overwrite the opkg
> > PACKAGECONFIG in a
> > > > .bbappend.
> > > >
> > >
> > > This is perhaps fine. I do wonder if there is some
> > performance
> > > comparison data between xz and zstd compressed ipks
> > > with opkg, it might help users on making this choice and
> > also if we
> > > should consider using
> > > zstd by default at some point or not.
> > >
> > > > Is there something I'm not considering here?
> > > >
> > > > >
> > > > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
> > > > > gnupg gpgme libgpg-error,\
> > > > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
> > > "--enable-gpg,--disable-gpg,\
> > > > > PACKAGECONFIG[curl] =
> > "--enable-curl,--disable-curl,curl"
> > > > > PACKAGECONFIG[ssl-curl] =
> > > "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
> > > > > PACKAGECONFIG[sha256] =
> > "--enable-sha256,--disable-sha256"
> > > > > +PACKAGECONFIG[zstd] =
> > "--enable-zstd,--disable-zstd,zstd"
> > > > > PACKAGECONFIG[libsolv] =
> > > "--with-libsolv,--without-libsolv,libsolv"
> > > > >
> > > > > EXTRA_OECONF:class-native =
> > > "--localstatedir=/${@os.path.relpath('${localstatedir}',
> > > '${STAGING_DIR_NATIVE}')}
> > > --sysconfdir=/${@os.path.relpath('${sysconfdir}',
> > > '${STAGING_DIR_NATIVE}')}"
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Alex Stewart
> > > > Software Engineer - NI Real-Time OS
> > > > NI (National Instruments)
> > > >
> > > > alex.stewart@ni.com
> > > >
> > > >
> > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > Links: You receive all messages sent to this group.
> > > > View/Reply Online (#170608):
> > >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>
> > >
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>>
> > >
> > > > Mute This Topic:
> > >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>
> > >
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>>
> > >
> > > > Group Owner:
> > openembedded-core+owner@lists.openembedded.org
> > <mailto:openembedded-core%2Bowner@lists.openembedded.org>
> > > <mailto:openembedded-core%2Bowner@lists.openembedded.org
> > <mailto:openembedded-core%252Bowner@lists.openembedded.org>>
> > > > Unsubscribe:
> > >
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>
> > >
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
> > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>>
> > > [raj.khem@gmail.com]
> > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > >
> > >
> >
> > --
> > Alex Stewart
> > Software Engineer - NI Real-Time OS
> > NI (National Instruments)
> >
> > alex.stewart@ni.com
> >
>
> --
> Alex Stewart
> Software Engineer - NI Real-Time OS
> NI (National Instruments)
>
> alex.stewart@ni.com
>
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Re: Re: [OE-core] [PATCH] opkg: enable zstd support
2022-09-14 15:41 ` Khem Raj
@ 2022-09-28 16:50 ` Etienne Cordonnier
2022-09-28 18:03 ` Alex Stewart
0 siblings, 1 reply; 12+ messages in thread
From: Etienne Cordonnier @ 2022-09-28 16:50 UTC (permalink / raw)
To: Khem Raj
Cc: Alex Stewart, Alex Feinman, openembedded-core, Etienne Cordonnier
[-- Attachment #1: Type: text/plain, Size: 23478 bytes --]
I tested it a bit more today. I used the standard poky's local.conf and
added those lines. You can see that at zstd level 9 there is still a
significant difference in the compression ratio with xz for gcc-dbg which
is a big file. At zstd level 19, gcc-dbg is 241MB big instead of 214MB with
xz (12.6% more).
PACKAGE_CLASSES = "package_ipk"
ZSTD_DEFAULTS = "-T0 -19"
PACKAGECONFIG:append:pn-opkg-native = " zstd"
OPKGBUILDCMD = "opkg-build -Z zstd -a ${ZSTD_DEFAULTS}"
Zstd at level 9:
build$ ls -lh tmp-zstd-level9/deploy/ipk/core2-64/ --sort=size | head -n10
total 1.5G
-rw-r--r-- 3 ecordonnier ecordonnier 302M Sep 28 18:33
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 249M Sep 28 18:33
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 106M Sep 28 18:33
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 44M Sep 28 18:33
binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 26M Sep 28 18:33
perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 23M Sep 28 18:33
gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 15M Sep 28 18:32
elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 15M Sep 28 18:33
gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 13M Sep 28 18:33
coreutils-dbg_9.1-r0_core2-64.ipk
zstd at level 19:
build$ ls -lh tmp-zstd-level19/deploy/ipk/core2-64/ --sort=size | head -n10
total 1.1G
-rw-r--r-- 3 ecordonnier ecordonnier 241M Sep 28 18:04
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 213M Sep 28 18:03
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 36M Sep 28 17:58
binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 30M Sep 28 18:00
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 22M Sep 28 17:59
perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 20M Sep 28 17:58
gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 14M Sep 28 17:57
elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 28 17:58
gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 28 17:57
g++_12.2.0-r0_core2-64.ipk
xz at level 9 (the default):
build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n10
total 963M
-rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44
gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44
openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 20M Sep 14 10:44
perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 19M Sep 14 10:44
gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 13M Sep 14 10:44
elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 14 10:44
gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 11M Sep 14 10:44
g++_12.2.0-r0_core2-64.ipk
On Wed, Sep 14, 2022 at 5:42 PM Khem Raj <raj.khem@gmail.com> wrote:
> On Wed, Sep 14, 2022 at 8:37 AM Alex Stewart <alex.stewart@ni.com> wrote:
> >
> > Thanks for checking.
> >
> > I'd be interested to know if setting a higher compression level for zstd
> > can get us to a similar compression ratio to xz. If so, then I think it
> > could be some real value to distro maintainers to be able to *tune*
> > their compression.
>
> yeah it will be interesting to say try something like level 9 but I think
> times
> might regress with that but it might be good to know the balance and
> perhaps
> suggest size mode and performance mode of zstd instead of xz
>
> >
> > That's not blocking for your new PR though.
> >
> >
> > On 9/14/22 05:08, Etienne Cordonnier wrote:
> > > Also note that zstd's default compression level is 3 per default (from
> > > a 1 to 19 scale). I did not test other compression levels.
> > >
> > > On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier
> > > <ecordonnier@snapchat.com> wrote:
> > >
> > > I ran a build of core-image-full-cmdline using xz and zstd, using
> > > pre-populated downloads and sstate-cache directories but with
> > > empty tmp directory. Here are the numbers:
> > > zstd:
> > > bitbake core-image-full-cmdline took 2m52.768s (real), the
> > > resulting directory tmp/deploy/ipk is 1.6GB big.
> > > xz:
> > > bitbake core-image-full-cmdline took 4m14.214s (real), the
> > > resulting directory tmp/deploy/ipk/ is 996M big
> > >
> > > So the build with xz is 47% slower (254/172) than with zstd for
> > > this "incremental build" use-case.
> > >
> > > See the 5 biggest packages, the difference in compression-ratio
> > > increases with big files:
> > > build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
> > > total 1.6G
> > > -rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39
> > > gcc-dbg_12.2.0-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39
> > > openssl-dbg_3.0.5-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39
> > > openssl-ptest_3.0.5-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39
> > > binutils-dbg_2.39-r0_core2-64.ipk
> > > build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
> > > total 963M
> > > -rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44
> > > gcc-dbg_12.2.0-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44
> > > openssl-dbg_3.0.5-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
> > > openssl-ptest_3.0.5-r0_core2-64.ipk
> > > -rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
> > > binutils-dbg_2.39-r0_core2-64.ipk
> > > ...
> > >
> > > I think for use-cases where the size of the ipk packages matters,
> > > xz is better. For use-cases where it does not matter (ipk packages
> > > not deployed), build-time matters more than compression-ratio and
> > > zstd is better.
> > >
> > > Regarding the enablement of zstd in opkg per default, I'll send a
> > > new version of the patch without this line.
> > > My thinking for enabling zstd per default in opkg was that zstd is
> > > already enabled per default in libarchive's PACKAGECONFIG, and
> > > disabling zstd in opkg's PACKAGECONFIG removes only a few lines of
> > > code from opkg (opkg uses libarchive for doing the actual
> > > compression/decompression).
> > >
> > > On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart
> > > <alex.stewart@ni.com> wrote:
> > >
> > >
> > >
> > > On 9/13/22 15:20, Alex Feinman wrote:
> > > > I do have some numbers. When I was selling this change
> > > internally, I
> > > > did a comparison on our internal build.
> > > > Combined write IPK times (Σ t
> do_package_write_ipk)
> > > > xz 162m 35s
> > > > gz 52m 13s
> > > > zstd 33m 49s
> > > > Compression rate for zstd was closer to xz than to gz but
> > > not as good
> > > > as xz. For systems that have to cache packages on the device
> > > with
> > > > limited storage xz might be a better option, but for the
> > > bulk of
> > > > projects zstd is the best choice
> > > > Additionally, zstd offers much faster decompression than xz
> > > so the
> > > > rootfs build step that includes unpacking all of the ipks,
> > > takes 3m
> > > > 58s with xz and 2m 44s with zstd.
> > > > One other thing of note - if your build includes debug
> > > packages, some
> > > > may be quite large. E.g. one of our components produces a
> > > 2.2 GB debug
> > > > package (uncompressed). On large files xz requires a
> > > disproportionally
> > > > large amount of time resulting in 15 minutes needed to
> > > simply write
> > > > ipk for the abovementioned packages, whereas zstd took about
> > > 45 sec.
> > > > For frequent tasks like bitbaking a single package this
> > > translates in
> > > > a lot of saved time.
> > >
> > > Those are certainly compelling performance improvements.
> > > Assuming that
> > > the final data-segment size is within 5%-ish of xz, then I
> > > would agree
> > > with the rest of the thread that it should probably be the
> > > contemporary
> > > default.
> > >
> > > And if we make it the default compressor for OE IPKs, then
> > > obviously my
> > > criticism in the original PR is satisfied.
> > >
> > > > Bottom line - I think making xz a default package compressor
> > > was not
> > > > entirely thought through. gzip or zstd is what the default
> > > should be.
> > >
> > > ZStandard support was only added to opkg last September [1].
> > > Before
> > > that, xz was the new hotness that replaced gzip. :)
> > >
> > > [1]
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
> >
> > >
> > >
> > > > One final note: I could not find a reasonable explanation
> > > for why
> > > > opkg-tools require code changes to support a different
> > > compressor. BSD
> > > > tar and GNU tar both can easily accept compressors that they
> > > have no
> > > > idea about (via -I option) because all of them provide a
> > > unified
> > > > command line interface for use in pipes. If this were done
> > > similar to
> > > > tar, we could have used any compressor we wanted, including
> the
> > > > multithreaded versions (zstdmt)
> > >
> > > Well, presumably IPK creation tools can only support the
> > > matrix of
> > > compression algorithms which your opkg binary can decompress.
> > > I suppose
> > > someone could try to implement a plugable compression module
> > > system for
> > > opkg. But given that nearly everyone uses opkg in an embedded
> > > context,
> > > I'm not sure it would get much use.
> > >
> > > >
> > > > On Tue, Sep 13, 2022 at 12:43 PM Khem Raj
> > > <raj.khem@gmail.com> wrote:
> > > >
> > > > On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
> > > > <alex.stewart@ni.com> wrote:
> > > > >
> > > > > ACK from me - apart from enabling zstd by default.
> > > > >
> > > > > On 9/13/22 07:37, Etienne Cordonnier via
> > > lists.openembedded.org
> > > <
> https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFhOF1_vg$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$
> >
> > > > wrote:
> > > > > > This allows the use of zstd for opkg packages by
> using
> > > > OPKGBUILDCMD:
> > > > > > OPKGBUILDCMD = "opkg-build -Z zstd"
> > > > > >
> > > > > > Signed-off-by: Alex Feinman <afeinman@snap.com>
> > > > > > Signed-off-by: Etienne Cordonnier <
> ecordonnier@snap.com>
> > > > > > ---
> > > > > > meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > | 3 ++-
> > > > > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git
> > > a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > > > index 7b351e8123..e38d9d6f3f 100644
> > > > > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$
> >
> > > >
> > > <
> https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$
> >
> > > > > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
> > > > > > target_localstatedir := "${localstatedir}"
> > > > > > OPKGLIBDIR ??= "${target_localstatedir}/lib"
> > > > > >
> > > > > > -PACKAGECONFIG ??= "libsolv"
> > > > > > +PACKAGECONFIG ??= "libsolv zstd"
> > > > >
> > > > > Building in zstd support by default is a little
> > > suspect to me.
> > > > >
> > > > > Unless I'm mistaken, OE-core will only build
> > > xz-compressed IPKs by
> > > > > default. So zstd support would be unnecessary for a
> distro
> > > > integrator
> > > > > who just uses upstream OE-core.
> > > > >
> > > > > For distros which use zstd compression in their
> > > packages, I think it
> > > > > would be more appropriate to overwrite the opkg
> > > PACKAGECONFIG in a
> > > > > .bbappend.
> > > > >
> > > >
> > > > This is perhaps fine. I do wonder if there is some
> > > performance
> > > > comparison data between xz and zstd compressed ipks
> > > > with opkg, it might help users on making this choice and
> > > also if we
> > > > should consider using
> > > > zstd by default at some point or not.
> > > >
> > > > > Is there something I'm not considering here?
> > > > >
> > > > > >
> > > > > > PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
> > > > > > gnupg gpgme libgpg-error,\
> > > > > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
> > > > "--enable-gpg,--disable-gpg,\
> > > > > > PACKAGECONFIG[curl] =
> > > "--enable-curl,--disable-curl,curl"
> > > > > > PACKAGECONFIG[ssl-curl] =
> > > > "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
> > > > > > PACKAGECONFIG[sha256] =
> > > "--enable-sha256,--disable-sha256"
> > > > > > +PACKAGECONFIG[zstd] =
> > > "--enable-zstd,--disable-zstd,zstd"
> > > > > > PACKAGECONFIG[libsolv] =
> > > > "--with-libsolv,--without-libsolv,libsolv"
> > > > > >
> > > > > > EXTRA_OECONF:class-native =
> > > > "--localstatedir=/${@os.path.relpath('${localstatedir}',
> > > > '${STAGING_DIR_NATIVE}')}
> > > > --sysconfdir=/${@os.path.relpath('${sysconfdir}',
> > > > '${STAGING_DIR_NATIVE}')}"
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Alex Stewart
> > > > > Software Engineer - NI Real-Time OS
> > > > > NI (National Instruments)
> > > > >
> > > > > alex.stewart@ni.com
> > > > >
> > > > >
> > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > > Links: You receive all messages sent to this group.
> > > > > View/Reply Online (#170608):
> > > >
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
> >
> > > >
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
> >>
> > > >
> > > > > Mute This Topic:
> > > >
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
> >
> > > >
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
> >>
> > > >
> > > > > Group Owner:
> > > openembedded-core+owner@lists.openembedded.org
> > > <mailto:openembedded-core%2Bowner@lists.openembedded.org>
> > > > <mailto:openembedded-core%2Bowner@lists.openembedded.org
> > > <mailto:openembedded-core%252Bowner@lists.openembedded.org>>
> > > > > Unsubscribe:
> > > >
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
> >
> > > >
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
> > > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
> >>
> > > > [raj.khem@gmail.com]
> > > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > >
> > > >
> > >
> > > --
> > > Alex Stewart
> > > Software Engineer - NI Real-Time OS
> > > NI (National Instruments)
> > >
> > > alex.stewart@ni.com
> > >
> >
> > --
> > Alex Stewart
> > Software Engineer - NI Real-Time OS
> > NI (National Instruments)
> >
> > alex.stewart@ni.com
> >
>
[-- Attachment #2: Type: text/html, Size: 38571 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Re: Re: Re: [OE-core] [PATCH] opkg: enable zstd support
2022-09-28 16:50 ` Etienne Cordonnier
@ 2022-09-28 18:03 ` Alex Stewart
0 siblings, 0 replies; 12+ messages in thread
From: Alex Stewart @ 2022-09-28 18:03 UTC (permalink / raw)
To: Etienne Cordonnier, Khem Raj
Cc: Alex Feinman, openembedded-core, Etienne Cordonnier
Thanks for doing the research.
What was the build time on the zstd L19 compression vs. xz? Was there
still an improvement?
On 9/28/22 11:50, Etienne Cordonnier wrote:
> I tested it a bit more today. I used the standard poky's local.conf
> and added those lines. You can see that at zstd level 9 there is still
> a significant difference in the compression ratio with xz for gcc-dbg
> which is a big file. At zstd level 19, gcc-dbg is 241MB big instead of
> 214MB with xz (12.6% more).
>
> PACKAGE_CLASSES = "package_ipk"
> ZSTD_DEFAULTS = "-T0 -19"
> PACKAGECONFIG:append:pn-opkg-native = " zstd"
> OPKGBUILDCMD = "opkg-build -Z zstd -a ${ZSTD_DEFAULTS}"
>
> Zstd at level 9:
>
> build$ ls -lh tmp-zstd-level9/deploy/ipk/core2-64/ --sort=size | head
> -n10
> total 1.5G
> -rw-r--r-- 3 ecordonnier ecordonnier 302M Sep 28 18:33
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 249M Sep 28 18:33
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 106M Sep 28 18:33
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 44M Sep 28 18:33
> binutils-dbg_2.39-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 26M Sep 28 18:33
> perl-ptest_5.36.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 23M Sep 28 18:33
> gcc_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 15M Sep 28 18:32
> elfutils-ptest_0.187-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 15M Sep 28 18:33
> gcc-src_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 13M Sep 28 18:33
> coreutils-dbg_9.1-r0_core2-64.ipk
>
> zstd at level 19:
>
> build$ ls -lh tmp-zstd-level19/deploy/ipk/core2-64/ --sort=size |
> head -n10
> total 1.1G
> -rw-r--r-- 3 ecordonnier ecordonnier 241M Sep 28 18:04
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 213M Sep 28 18:03
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 36M Sep 28 17:58
> binutils-dbg_2.39-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 30M Sep 28 18:00
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 22M Sep 28 17:59
> perl-ptest_5.36.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 20M Sep 28 17:58
> gcc_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 14M Sep 28 17:57
> elfutils-ptest_0.187-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 28 17:58
> gcc-src_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 28 17:57
> g++_12.2.0-r0_core2-64.ipk
>
> xz at level 9 (the default):
>
> build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n10
> total 963M
> -rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
> binutils-dbg_2.39-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 20M Sep 14 10:44
> perl-ptest_5.36.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 19M Sep 14 10:44
> gcc_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 13M Sep 14 10:44
> elfutils-ptest_0.187-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 14 10:44
> gcc-src_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier 11M Sep 14 10:44
> g++_12.2.0-r0_core2-64.ipk
>
> On Wed, Sep 14, 2022 at 5:42 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> On Wed, Sep 14, 2022 at 8:37 AM Alex Stewart <alex.stewart@ni.com>
> wrote:
> >
> > Thanks for checking.
> >
> > I'd be interested to know if setting a higher compression level
> for zstd
> > can get us to a similar compression ratio to xz. If so, then I
> think it
> > could be some real value to distro maintainers to be able to *tune*
> > their compression.
>
> yeah it will be interesting to say try something like level 9 but
> I think times
> might regress with that but it might be good to know the balance
> and perhaps
> suggest size mode and performance mode of zstd instead of xz
>
--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)
alex.stewart@ni.com
^ permalink raw reply [flat|nested] 12+ messages in thread