Adding S3 as option for sstate

Adding S3 as option for sstate

[Alexander Nassian]

Hi together,

We would like to extend poky to be able to write and read sstate packages not only from HTTP and FILE but also S3 buckets which would fit our CI setup much better than our current „zip everything and upload to S3“ in terms of incremental speed up builds. I already did some research if there is something read-to-use but didn’t find anything.

My two questions are: 
1. Do you have any hint where to hook into? I already went through runqueue.py but I’m just a extensive user of poky, never worked on itself.
2. Would that change have any chance to be accepted upstream, or is this something too special for the project?

Best regards,
Alexander
-- 













—


bitshift dynamics GmbH
Neudorfer Str. 1, 79541 Lörrach
Registergericht: Amtsgericht Freiburg i. Breisgau, HRB 713747
Geschäftsführer: Alexander Nassian, Markus Pfaffinger


http://www.bitshift-dynamics.de <http://www.bitshift-dynamics.de/>

Zentrale: +49 762158673 - 0
Fax: +49 7621 58673 - 90


Allgemeine Anfragen: 
info@bitshift-dynamics.com <mailto:info@bitshift-dynamics.com>
Technischer 
Support: support@bitshift-dynamics.com 
<mailto:support@bitshift-dynamics.com>
Buchhaltung: 
invoice@bitshift-dynamics.com <mailto:invoice@bitshift-dynamics.com>

Re: [poky] Adding S3 as option for sstate

[Matt Madison]

On Fri, Nov 13, 2020 at 11:50 PM Alexander Nassian
<nassian@bitshift-dynamics.com> wrote:
>
> Hi together,
>
> We would like to extend poky to be able to write and read sstate packages not only from HTTP and FILE but also S3 buckets which would fit our CI setup much better than our current „zip everything and upload to S3“ in terms of incremental speed up builds. I already did some research if there is something read-to-use but didn’t find anything.
>
> My two questions are:
> 1. Do you have any hint where to hook into? I already went through runqueue.py but I’m just a extensive user of poky, never worked on itself.
> 2. Would that change have any chance to be accepted upstream, or is this something too special for the project?

I have a CI setup that I host in EC2 using an S3 bucket for mirroring
and have implemented some hooks that make it work better for me.  The
basic parts consist of a replacement for the built-in S3 fetcher and a
bbclass to assist with populating the S3-hosted mirror.  I have a
writeup on it at
https://github.com/madisongh/tegra-test-distro/wiki/Using-AWS-S3-for-sstate-and-downloads-mirrors
if you're interested.  It may not be the right answer for you, but it
may give you some ideas.

Regards,
-Matt

>
> Best regards,
> Alexander
> --
>
>
>
>
>
>
>
>
>
>
>
>
>
> —
>
>
> bitshift dynamics GmbH
> Neudorfer Str. 1, 79541 Lörrach
> Registergericht: Amtsgericht Freiburg i. Breisgau, HRB 713747
> Geschäftsführer: Alexander Nassian, Markus Pfaffinger
>
>
> http://www.bitshift-dynamics.de <http://www.bitshift-dynamics.de/>
>
> Zentrale: +49 762158673 - 0
> Fax: +49 7621 58673 - 90
>
>
> Allgemeine Anfragen:
> info@bitshift-dynamics.com <mailto:info@bitshift-dynamics.com>
> Technischer
> Support: support@bitshift-dynamics.com
> <mailto:support@bitshift-dynamics.com>
> Buchhaltung:
> invoice@bitshift-dynamics.com <mailto:invoice@bitshift-dynamics.com>
>
> 
>

Re: [poky] Adding S3 as option for sstate

[Alexander Nassian]

[-- Attachment #1: Type: text/plain, Size: 2359 bytes --]

Thank you very much for your feedback and sharing your solution! I will definetly have a look into it, but had a very busy online training week.

If I see it right your class enables pulling sstate packages from S3 but not uploading them on the go like the filesystem module, right?

Beste Grüße / Best regards,

Alexander Nassian, Geschäftsführer / CEO
nassian@bitshift-dynamics.com

> Am 14.11.2020 um 15:30 schrieb Matt Madison <matt@madison.systems>:
> 
> On Fri, Nov 13, 2020 at 11:50 PM Alexander Nassian
> <nassian@bitshift-dynamics.com> wrote:
>> 
>> Hi together,
>> 
>> We would like to extend poky to be able to write and read sstate packages not only from HTTP and FILE but also S3 buckets which would fit our CI setup much better than our current „zip everything and upload to S3“ in terms of incremental speed up builds. I already did some research if there is something read-to-use but didn’t find anything.
>> 
>> My two questions are:
>> 1. Do you have any hint where to hook into? I already went through runqueue.py but I’m just a extensive user of poky, never worked on itself.
>> 2. Would that change have any chance to be accepted upstream, or is this something too special for the project?
> 
> I have a CI setup that I host in EC2 using an S3 bucket for mirroring
> and have implemented some hooks that make it work better for me.  The
> basic parts consist of a replacement for the built-in S3 fetcher and a
> bbclass to assist with populating the S3-hosted mirror.  I have a
> writeup on it at
> https://github.com/madisongh/tegra-test-distro/wiki/Using-AWS-S3-for-sstate-and-downloads-mirrors
> if you're interested.  It may not be the right answer for you, but it
> may give you some ideas.
> 
> Regards,
> -Matt


-- 










—


bitshift dynamics GmbH
Neudorfer Str. 1, 79541 Lörrach
Registergericht: Amtsgericht Freiburg i. Breisgau, HRB 713747
Geschäftsführer: Alexander Nassian, Markus Pfaffinger


https://bitshift-dynamics.com <https://bitshift-dynamics.com>

Phone: +49 
762158673 - 0
General support: info@bitshift-dynamics.com 
<mailto:info@bitshift-dynamics.com>
Technical support: 
support@bitshift-dynamics.com <mailto:support@bitshift-dynamics.com>
Accounting: invoice@bitshift-dynamics.com 
<mailto:invoice@bitshift-dynamics.com>

[-- Attachment #2: Type: text/html, Size: 5512 bytes --]

Re: [poky] Adding S3 as option for sstate

[Matt Madison]

On Fri, Nov 27, 2020 at 10:34 PM Alexander Nassian
<nassian@bitshift-dynamics.com> wrote:
>
> Thank you very much for your feedback and sharing your solution! I will definetly have a look into it, but had a very busy online training week.
>
> If I see it right your class enables pulling sstate packages from S3 but not uploading them on the go like the filesystem module, right?

No, it does handle uploading both downloaded source packages as well
as sstate packages.

Regards,
-Matt

>
> Beste Grüße / Best regards,
>
> Alexander Nassian, Geschäftsführer / CEO
> nassian@bitshift-dynamics.com
>
> Am 14.11.2020 um 15:30 schrieb Matt Madison <matt@madison.systems>:
>
> On Fri, Nov 13, 2020 at 11:50 PM Alexander Nassian
> <nassian@bitshift-dynamics.com> wrote:
>
>
> Hi together,
>
> We would like to extend poky to be able to write and read sstate packages not only from HTTP and FILE but also S3 buckets which would fit our CI setup much better than our current „zip everything and upload to S3“ in terms of incremental speed up builds. I already did some research if there is something read-to-use but didn’t find anything.
>
> My two questions are:
> 1. Do you have any hint where to hook into? I already went through runqueue.py but I’m just a extensive user of poky, never worked on itself.
> 2. Would that change have any chance to be accepted upstream, or is this something too special for the project?
>
>
> I have a CI setup that I host in EC2 using an S3 bucket for mirroring
> and have implemented some hooks that make it work better for me.  The
> basic parts consist of a replacement for the built-in S3 fetcher and a
> bbclass to assist with populating the S3-hosted mirror.  I have a
> writeup on it at
> https://github.com/madisongh/tegra-test-distro/wiki/Using-AWS-S3-for-sstate-and-downloads-mirrors
> if you're interested.  It may not be the right answer for you, but it
> may give you some ideas.
>
> Regards,
> -Matt
>
>
>
> —
>
> bitshift dynamics GmbH
> Neudorfer Str. 1, 79541 Lörrach
> Registergericht: Amtsgericht Freiburg i. Breisgau, HRB 713747
> Geschäftsführer: Alexander Nassian, Markus Pfaffinger
>
> https://bitshift-dynamics.com
>
> Phone: +49 762158673 - 0
> General support: info@bitshift-dynamics.com
> Technical support: support@bitshift-dynamics.com
> Accounting: invoice@bitshift-dynamics.com