Re: Storing Sstate in S3 success stories?

From: Timothy Froehlich <tfroehlich@archsys.io>
To: Erik Hoogeveen <erik.hoogeveen@outlook.com>
Cc: Yocto discussion list <yocto@yoctoproject.org>
Subject: Re: Storing Sstate in S3 success stories?
Date: Tue, 26 Feb 2019 10:35:40 -0800	[thread overview]
Message-ID: <CAHSUxjQ21WkhJehKd9B_e2uqqf5RA4QizA+JYq0rA9Ujzjr7rw@mail.gmail.com> (raw)
In-Reply-To: <DB6PR0701MB2629CC6C1069B27F94037DF7957B0@DB6PR0701MB2629.eurprd07.prod.outlook.com>

[-- Attachment #1: Type: text/plain, Size: 3155 bytes --]

Well, based on the responses above I did some more research and it didn't
seem like the file sizes should be causing problems on the scale that I was
seeing so I investigated further. I realized that despite my build/sstate
directory getting slowly larger, it wasn't actually getting the files and
was leaving empty files behind. I tried just changing the https in my
SSTATE_MIRRORs line just http and it worked perfectly, pulling down a half
gig of sstate before I could tell it was even working. So there was likely
some other misconfiguration in our AWS account that caused the https to
fail (despite being able to wget individual files using https). Thanks for
the responses!

On Tue, Feb 26, 2019 at 12:19 AM Erik Hoogeveen <erik.hoogeveen@outlook.com>
wrote:

> Hi Timothy,
>
> The S3 protocol is HTTP(S) based, the overhead per object is quite
> significant. This is not much of a problem for large files but the
> sstate_cache contains mostly lots of really small files. I think in this
> case you’re better of storing the cache on a secondary EBS volume that you
> can attach as a regular block device to the EC2 instance. You can swtich on
> deletion protection to make it survive EC2 termination.
>
> Since EBS volumes are quite a bit more expensive than S3 buckets  you
> could make snapshots to transfer the state between build runs in stead then
> you can destroy the EBS volume when nothing is running.
>
> Alls the documentation about EBS is here
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html
>
> Cheers,
> Erik
> On 26 Feb 2019, 02:45 +0100, Timothy Froehlich <tfroehlich@archsys.io>,
> wrote:
>
> I've been spending a bit too long this past week trying to build up a
> reproducable build infrastructure in AWS and I've got very little
> experience with cloud infrastucture and I'm wondering if I'm going in the
> wrong direction. I'm attempting to host my sstate_cache as a mirror in a
> private S3 bucket, and I believe I have everything configured properly,
> including exposing the bucket to http requests, since I can wget files that
> I've previously synced up to the bucket. However if I add in the
> SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM)
> and barely seems to get anything. The EC2 instance is in the same region as
> the S3 bucket, roles have been configured properly to allow access, etc.
>
> I'm not looking for help debugging this, I just want to know whether I'm
> right that hosting my sstate in an S3 bucket should work. I've only been
> able to find one mention of it being done with no reproduction hints.
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.yoctoproject.org%2Flistinfo%2Fyocto&amp;data=02%7C01%7C%7Cdc66c2ab9da44272220b08d69b8c2262%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636867423506178146&amp;sdata=TjwcnaFaZQpk9iuxL16%2FosjuqEH2S7aXB16JjBIDGko%3D&amp;reserved=0
>
>

-- 
Tim Froehlich
Embedded Linux Engineer
tfroehlich@archsys.io
215-218-8955

[-- Attachment #2: Type: text/html, Size: 4856 bytes --]