Re: [poky] [PATCH] sstate: Add ZStandard compressor support

["Henry Kleynhans" <hkleynhans@fb.com>]

[-- Attachment #1: Type: text/plain, Size: 3142 bytes --]

Thank you all for the feedback!  I like the idea of switching completely to zstd.  I will update the patch accordingly and resubmit.

Kind regards, Henry
________________________________
From: Joshua Watt <jpewhacker@gmail.com>
Sent: 01 October 2021 14:44
To: Richard Purdie <richard.purdie@linuxfoundation.org>
Cc: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; Henry Kleynhans <hkleynhans@fb.com>; poky@lists.yoctoproject.org <poky@lists.yoctoproject.org>; Michael van der Westhuizen <rmikey@fb.com>
Subject: Re: [poky] [PATCH] sstate: Add ZStandard compressor support

On Fri, Oct 1, 2021 at 8:16 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Fri, 2021-10-01 at 13:13 +0000, Peter Kjellerstedt wrote:
> > > -----Original Message-----
> > > From: poky@lists.yoctoproject.org <poky@lists.yoctoproject.org> On Behalf
> > > Of Henry Kleynhans via lists.yoctoproject.org
> > > Sent: den 1 oktober 2021 12:11
> > > To: poky@lists.yoctoproject.org
> > > Cc: hkleynhans@fb.com; rmikey@fb.com
> > > Subject: [poky] [PATCH] sstate: Add ZStandard compressor support
> > >
> > > This patch adds support to optionally use the Zstandard compressor for
> > > ssate cache files.
> > >
> > > Zstandard compression provides a significant improvement in
> > > decompression speed as well as improvement in compression speed and disk
> > > usage over the 'tgz' format in use.  Furthermore, its configurable
> > > compression level offers a trade-off between time spent compressing
> > > sstate cache files and disk space used by those files.  The reduced disk
> > > usage also contributes to saving network traffic for those sharing their
> > > sstate cache with others.
> > >
> > > Zstandard should therefore be a good choice when:
> > > * disk space is at a premium
> > > * network speed / resources are limited
> > > * the CI server can sstate packages can be created at high compression
> > > * less CPU on the build server should be used for sstate decompression
> > >
> > > Signed-off-by: Henry Kleynhans <hkleynhans@fb.com>
> > > ---
> > >  meta/classes/sstate.bbclass        | 49 +++++++++++++++++++++++-------
> > >  scripts/sstate-cache-management.sh | 40 ++++++++++++------------
> > >  2 files changed, 58 insertions(+), 31 deletions(-)
> > >
> > > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> > > index 92a73114bb..a73d631679 100644
> > > --- a/meta/classes/sstate.bbclass
> > > +++ b/meta/classes/sstate.bbclass
> > > @@ -1,17 +1,30 @@
> > >  SSTATE_VERSION = "3"
> > >
> > > +SSTATE_USE_ZSTD ?= "0"
> > > +SSTATE_ZSTD_CLEVEL ?= "3"
> > > +SSTATE_ZSTD_NTHREADS ?= "0"
> >
> > Do we really need to make this configurable? Can't we just decide
> > to use zstd and be done with it?
>
> I have to admit I was also wondering that. We started requiring zstd on the host
> system so it may just be better to switch over?

Since zstd is required and a clear winner in compression/decompression
time, I think requiring it and simplifying the sstate code to only use
zstd would be fine.

>
> Cheers,
>
> Richard
>
>
>
>
> 
>

[-- Attachment #2: Type: text/html, Size: 4720 bytes --]