Re: [poky] [PATCH] sstate: Add ZStandard compressor support

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Fri, 2021-10-01 at 13:13 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: poky@lists.yoctoproject.org <poky@lists.yoctoproject.org> On Behalf
> > Of Henry Kleynhans via lists.yoctoproject.org
> > Sent: den 1 oktober 2021 12:11
> > To: poky@lists.yoctoproject.org
> > Cc: hkleynhans@fb.com; rmikey@fb.com
> > Subject: [poky] [PATCH] sstate: Add ZStandard compressor support
> > 
> > This patch adds support to optionally use the Zstandard compressor for
> > ssate cache files.
> > 
> > Zstandard compression provides a significant improvement in
> > decompression speed as well as improvement in compression speed and disk
> > usage over the 'tgz' format in use.  Furthermore, its configurable
> > compression level offers a trade-off between time spent compressing
> > sstate cache files and disk space used by those files.  The reduced disk
> > usage also contributes to saving network traffic for those sharing their
> > sstate cache with others.
> > 
> > Zstandard should therefore be a good choice when:
> > * disk space is at a premium
> > * network speed / resources are limited
> > * the CI server can sstate packages can be created at high compression
> > * less CPU on the build server should be used for sstate decompression
> > 
> > Signed-off-by: Henry Kleynhans <hkleynhans@fb.com>
> > ---
> >  meta/classes/sstate.bbclass        | 49 +++++++++++++++++++++++-------
> >  scripts/sstate-cache-management.sh | 40 ++++++++++++------------
> >  2 files changed, 58 insertions(+), 31 deletions(-)
> > 
> > diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> > index 92a73114bb..a73d631679 100644
> > --- a/meta/classes/sstate.bbclass
> > +++ b/meta/classes/sstate.bbclass
> > @@ -1,17 +1,30 @@
> >  SSTATE_VERSION = "3"
> > 
> > +SSTATE_USE_ZSTD ?= "0"
> > +SSTATE_ZSTD_CLEVEL ?= "3"
> > +SSTATE_ZSTD_NTHREADS ?= "0"
> 
> Do we really need to make this configurable? Can't we just decide 
> to use zstd and be done with it?

I have to admit I was also wondering that. We started requiring zstd on the host
system so it may just be better to switch over?

Cheers,

Richard