Sharing sstate

Sharing sstate

[Gary Thomas]

I'd really like to be able to share sstate effectively.  One problem I have with
it is the size - for a build of my system (something similar to core-image-sato
on BeagleBoard), the sstate cache for a build-from-scratch is nearly 5GB!  Not
exactly simple to share across sites.

It seems to me that maybe I don't need to share everything.  For example, the
webkit-gtk package:

$ ls -l sstate-cache/*webkit-gtk*
-rw-rw-r-- 1 gary gary 331221793 Sep 23 16:21 
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz
-rw-rw-r-- 1 gary gary     12740 Sep 23 16:21 
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz.siginfo
-rw-r--r-- 1 gary gary 326376172 Sep 23 16:15 
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz
-rw-r--r-- 1 gary gary     21652 Sep 23 16:15 
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz.siginfo
-rw-r--r-- 1 gary gary 665832323 Sep 23 16:19 
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz
-rw-r--r-- 1 gary gary     98746 Sep 23 16:19 
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz.siginfo
-rw-rw-r-- 1 gary gary      3117 Sep 23 15:38 
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz
-rw-rw-r-- 1 gary gary      8359 Sep 23 15:38 
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz.siginfo

Would it be possible to only share part of this, maybe the "package" file?
It seems to me that this could make the needed (shared) cache much smaller...

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Sharing sstate

[Richard Purdie]

On Tue, 2011-09-27 at 08:22 -0600, Gary Thomas wrote:
> I'd really like to be able to share sstate effectively.  One problem I have with
> it is the size - for a build of my system (something similar to core-image-sato
> on BeagleBoard), the sstate cache for a build-from-scratch is nearly 5GB!  Not
> exactly simple to share across sites.
> 
> It seems to me that maybe I don't need to share everything.  For example, the
> webkit-gtk package:
> 
> $ ls -l sstate-cache/*webkit-gtk*
> -rw-rw-r-- 1 gary gary 331221793 Sep 23 16:21 
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz
> -rw-rw-r-- 1 gary gary     12740 Sep 23 16:21 
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz.siginfo
> -rw-r--r-- 1 gary gary 326376172 Sep 23 16:15 
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz
> -rw-r--r-- 1 gary gary     21652 Sep 23 16:15 
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz.siginfo
> -rw-r--r-- 1 gary gary 665832323 Sep 23 16:19 
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz
> -rw-r--r-- 1 gary gary     98746 Sep 23 16:19 
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz.siginfo
> -rw-rw-r-- 1 gary gary      3117 Sep 23 15:38 
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz
> -rw-rw-r-- 1 gary gary      8359 Sep 23 15:38 
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz.siginfo
> 
> Would it be possible to only share part of this, maybe the "package" file?
> It seems to me that this could make the needed (shared) cache much smaller...

The build will only use the pieces it needs. If your build only needs
the deploy-ipk pieces and doesn't need to write new packages files it
can survive without the package part. If it doesn't need to compile
something against webkit, it won't use the populate-sysroot file.

So what I'm saying is you likely could get away with not sharing the
*_package files at least. The only downside is if you turn on rpm
packaging it would have to rebuild.

Cheers,

Richard

Re: Sharing sstate

[Gary Thomas]

On 2011-09-27 10:17, Richard Purdie wrote:
> On Tue, 2011-09-27 at 08:22 -0600, Gary Thomas wrote:
>> I'd really like to be able to share sstate effectively.  One problem I have with
>> it is the size - for a build of my system (something similar to core-image-sato
>> on BeagleBoard), the sstate cache for a build-from-scratch is nearly 5GB!  Not
>> exactly simple to share across sites.
>>
>> It seems to me that maybe I don't need to share everything.  For example, the
>> webkit-gtk package:
>>
>> $ ls -l sstate-cache/*webkit-gtk*
>> -rw-rw-r-- 1 gary gary 331221793 Sep 23 16:21
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz
>> -rw-rw-r-- 1 gary gary     12740 Sep 23 16:21
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz.siginfo
>> -rw-r--r-- 1 gary gary 326376172 Sep 23 16:15
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz
>> -rw-r--r-- 1 gary gary     21652 Sep 23 16:15
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz.siginfo
>> -rw-r--r-- 1 gary gary 665832323 Sep 23 16:19
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz
>> -rw-r--r-- 1 gary gary     98746 Sep 23 16:19
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz.siginfo
>> -rw-rw-r-- 1 gary gary      3117 Sep 23 15:38
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz
>> -rw-rw-r-- 1 gary gary      8359 Sep 23 15:38
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz.siginfo
>>
>> Would it be possible to only share part of this, maybe the "package" file?
>> It seems to me that this could make the needed (shared) cache much smaller...
>
> The build will only use the pieces it needs. If your build only needs
> the deploy-ipk pieces and doesn't need to write new packages files it
> can survive without the package part. If it doesn't need to compile
> something against webkit, it won't use the populate-sysroot file.
>
> So what I'm saying is you likely could get away with not sharing the
> *_package files at least. The only downside is if you turn on rpm
> packaging it would have to rebuild.

I tried removing the *package* files and the build ended up rebuilding
most things, in particular the toolchain bits.  That's the part I really
want to be able to package the sstate for :-(

So then I tried having just a small subset of packages in the sstate
cache, but including all the pieces of each, like this:

rsync -auv sstate-cache/*eglibc-locale* \
   sstate-cache/*autoconf-native* \
   sstate-cache/*automake-native* \
   sstate-cache/*bzip2-native* \
   sstate-cache/*gettext-native* \
   sstate-cache/*gnu-config-native* \
   sstate-cache/*help2man-native* \
   sstate-cache/*libtool-native* \
   sstate-cache/*libxml2-native* \
   sstate-cache/*m4-native* \
   sstate-cache/*ncurses-native* \
   sstate-cache/*openssl-native* \
   sstate-cache/*pkgconfig-native* \
   sstate-cache/*pseudo-native* \
   sstate-cache/*python-native* \
   sstate-cache/*quilt-native* \
   sstate-cache/*readline-native* \
   sstate-cache/*sqlite3-native* \
   sstate-cache/*tar-replacement-native* \
   sstate-cache/*zlib-native* \
   sstate-cache/*gcc* \
   sstate-cache/*binutils* \
   sstate-cache/*eglibc* \
   /tmp/sstate-test

This also fails, but differently:

ERROR: Task 756 (/home/local/poky-multi/meta/recipes-devtools/libtool/libtool-cross_2.4.bb, do_configure) failed with exit code '1'

looking at the config.log:

/home/local/p60_sstate_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-amltd-linux-gnueabi/../../libexec/armv7a-vfp
-neon-amltd-linux-gnueabi/gcc/arm-amltd-linux-gnueabi/4.6.1/cc1: error while loading shared libraries: libmpc.so.2: cann
ot open shared object file: No such file or directory

Filed as http://bugzilla.pokylinux.org/show_bug.cgi?id=1536

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Sharing sstate

[Mike Looijmans]

On 26-03-15 14:44, Gary Thomas wrote:
> I have two development servers, one running an older version
> of Fedora (x86 *), the other runs a recent Ubuntu (x86_64).
> Both are the same CPU core (Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz)
>
> I've just tried building the same target, using the same version
> of Poky/Yocto, sharing the sstate cache (fully populated) from
> the x86_64 machine with the x86 machine.  This was done using
> HTTP.  Sadly, there was not a single file found in the cache
> that could be used.
>
> Should this not be expected to work?  Does sstate cache only
> work across like systems (32 vs 64 bit hosts)?

The hosts need to be similar for the build tools to be recycled. If they have 
different OS versions, or even different CFLAGS you'll see a rebuild of 
"native" packages.

It should however be able to recycle things for the target, because these 
should not depend on the build host's configuration.

We currently have a "server" with ubuntu 12 on it, while most "clients" run 
14. Typically the clients build their own tools, but they do re-use all target 
packages (like FPGA bitstream and ARM software packages).


> Also, Mark Hatle mentioned a few days ago that there are ways
> to "lock" versions of the sstate cache.  I still haven't found
> any info/documentation on this.  Can someone please explain more?
>
> Thanks
>
> (*) I can't update my x86 machine to a more recent (x86_64)
> distribution as I have many older "projects" that only run
> on that hardware and no time/opportunity to upgrade, hence
> my desire to share in this manner.
>



Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: Sharing sstate

[Gary Thomas]

On 2015-03-26 07:44, Gary Thomas wrote:
> I have two development servers, one running an older version
> of Fedora (x86 *), the other runs a recent Ubuntu (x86_64).
> Both are the same CPU core (Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz)
>
> I've just tried building the same target, using the same version
> of Poky/Yocto, sharing the sstate cache (fully populated) from
> the x86_64 machine with the x86 machine.  This was done using
> HTTP.  Sadly, there was not a single file found in the cache
> that could be used.
>
> Should this not be expected to work?  Does sstate cache only
> work across like systems (32 vs 64 bit hosts)?
>
> Also, Mark Hatle mentioned a few days ago that there are ways
> to "lock" versions of the sstate cache.  I still haven't found
> any info/documentation on this.  Can someone please explain more?
>
> Thanks
>
> (*) I can't update my x86 machine to a more recent (x86_64)
> distribution as I have many older "projects" that only run
> on that hardware and no time/opportunity to upgrade, hence
> my desire to share in this manner.
>

I've just verified that if I move my build tree from the x86
machine to the x86_64 (really just build/conf), then sstate
works as expected.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Sharing sstate

[Gary Thomas]

I have two development servers, one running an older version
of Fedora (x86 *), the other runs a recent Ubuntu (x86_64).
Both are the same CPU core (Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz)

I've just tried building the same target, using the same version
of Poky/Yocto, sharing the sstate cache (fully populated) from
the x86_64 machine with the x86 machine.  This was done using
HTTP.  Sadly, there was not a single file found in the cache
that could be used.

Should this not be expected to work?  Does sstate cache only
work across like systems (32 vs 64 bit hosts)?

Also, Mark Hatle mentioned a few days ago that there are ways
to "lock" versions of the sstate cache.  I still haven't found
any info/documentation on this.  Can someone please explain more?

Thanks

(*) I can't update my x86 machine to a more recent (x86_64)
distribution as I have many older "projects" that only run
on that hardware and no time/opportunity to upgrade, hence
my desire to share in this manner.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------