[cip-dev] [ANNOUNCE] Introducing gitlab-cloud-ci. Perform ISAR-based CI builds in the cloud.

[cip-dev] [ANNOUNCE] Introducing gitlab-cloud-ci. Perform ISAR-based CI builds in the cloud.

[Adler, Michael]

Hi everyone,

I have created a little Python tool called gitlab-cloud-ci [1] which allows you to set up an AWS-based infrastructure to perform ISAR CI-builds.
It does:

* Create a Kubernetes-cluster on AWS from scratch
* Scale horizontally: run an (almost) arbitrary number of CI jobs in parallel
* Scale dynamically: scale up and down based on the current workload situation
* Support running privileged containers for CI pipelines (unlike the default Gitlab runners) which is required for Isar builds

The project gitlab-cloud-ci is still in a very early stage (the public release was just yesterday).

We (Siemens) would like to propose this tool for creating an AWS-based build infrastructure in CIP. As a first step, we
could create a test cluster on AWS and demonstrate building cip-core (deby and isar-based), then discuss further steps.

Kind regards,
Michael

[1] https://gitlab.com/cip-playground/gitlab-cloud-ci

-- 
Michael Adler
Siemens AG, Corporate Technology, CT RDA IOT SES-DE, Otto-Hahn-Ring 6, 81739 Munich, Germany

Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard Cromme; Managing Board: Joe Kaeser, Chairman, President and Chief Executive Officer; Roland Busch, Lisa Davis, Klaus Helmrich, Janina Kugel, Siegfried Russwurm, Ralf P. Thomas; Registered offices: Berlin and Munich, Germany; Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322

[cip-dev] [ANNOUNCE] Introducing gitlab-cloud-ci. Perform ISAR-based CI builds in the cloud.

[daniel.sangorrin at toshiba.co.jp]

Hi Michael,

> From: Adler, Michael
> I have created a little Python tool called gitlab-cloud-ci [1] which allows you to set up an AWS-based
> infrastructure to perform ISAR CI-builds.
> It does:
> 
> * Create a Kubernetes-cluster on AWS from scratch
> * Scale horizontally: run an (almost) arbitrary number of CI jobs in parallel
> * Scale dynamically: scale up and down based on the current workload situation
> * Support running privileged containers for CI pipelines (unlike the default Gitlab runners) which is required for
> Isar builds

Great work!!
# Looking at "share/k8s/setup-host-binfmt/daemonset.yaml" I see that you must have run into some difficult problems.

> The project gitlab-cloud-ci is still in a very early stage (the public release was just yesterday).
> 
> We (Siemens) would like to propose this tool for creating an AWS-based build infrastructure in CIP. As a first step,
> we
> could create a test cluster on AWS and demonstrate building cip-core (deby and isar-based), then discuss further
> steps.

That sounds good to me.
You can ask Jan for a user account on CIP AWS account.
If you are not able to allocate resources (e.g.: EC2, S3, Route53) for the test cluster please let us know.
Once you finish the initial test, we should be able to estimate the cost of using the cluster and probably ask the CIP board members to approve.

Thanks,
Daniel

> 
> Kind regards,
> Michael
> 
> [1] https://gitlab.com/cip-playground/gitlab-cloud-ci
> 
> --
> Michael Adler
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE, Otto-Hahn-Ring 6, 81739 Munich, Germany
> 
> Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard Cromme; Managing Board: Joe Kaeser,
> Chairman, President and Chief Executive Officer; Roland Busch, Lisa Davis, Klaus Helmrich, Janina Kugel,
> Siegfried Russwurm, Ralf P. Thomas; Registered offices: Berlin and Munich, Germany; Commercial registries:
> Berlin Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322
> _______________________________________________
> cip-dev mailing list
> cip-dev at lists.cip-project.org
> https://lists.cip-project.org/mailman/listinfo/cip-dev

[cip-dev] [ANNOUNCE] Introducing gitlab-cloud-ci. Perform ISAR-based CI builds in the cloud.

[Adler, Michael]

> Great work!!

Thanks :)

> # Looking at "share/k8s/setup-host-binfmt/daemonset.yaml" I see that you must have run into some difficult problems.

Good catch! That was indeed one of the trickier parts.

> You can ask Jan for a user account on CIP AWS account.
> Once you finish the initial test, we should be able to estimate the cost of using the cluster and probably ask the CIP board members to approve.

Alright, we'll probably give it a try on Monday and see how it goes.

Have a nice weekend,
Michael

-- 
Michael Adler
Siemens AG, Corporate Technology, CT RDA IOT SES-DE, Otto-Hahn-Ring 6, 81739 Munich, Germany

Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard Cromme; Managing Board: Joe Kaeser, Chairman, President and Chief Executive Officer; Roland Busch, Lisa Davis, Klaus Helmrich, Janina Kugel, Siegfried Russwurm, Ralf P. Thomas; Registered offices: Berlin and Munich, Germany; Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322

[cip-dev] [ANNOUNCE] Introducing gitlab-cloud-ci. Perform ISAR-based CI builds in the cloud.

[Dan Rue]

On Tue, May 21, 2019 at 01:09:19PM +0000, Adler, Michael wrote:
> Hi everyone,

Hi! I just lurk here (I primarily work on LKFT and kernelci), but this
is really interesting! Thanks for sharing, I'll be following this
closely.

One question, based on something I read in the README: "By default,
unneeded nodes are removed after being idle for 55 minutes. This is
because AWS billing is usually done on an hourly usage."

I thought they billed per-second now days, but looking into it, it seems
that may only be for amazon linux and ubuntu AMIs? I'm not sure what
distro you're using, but it may be a good reason to favor one of those
two. If it is in fact billed hourly, 55 minutes may only cause you to
buy an extra hour most of the time.

Do your builds take advantage of any caching, and if so, how are you
making it persistent?

I'm also really curious about your experience with k8s, but mostly I'm
interested to find out how it works for this project over time in terms
of stability, cost, complexity, etc.

Thanks again! If there is any place else I can follow information about
this let me know.

Dan

> 
> I have created a little Python tool called gitlab-cloud-ci [1] which allows you to set up an AWS-based infrastructure to perform ISAR CI-builds.
> It does:
> 
> * Create a Kubernetes-cluster on AWS from scratch
> * Scale horizontally: run an (almost) arbitrary number of CI jobs in parallel
> * Scale dynamically: scale up and down based on the current workload situation
> * Support running privileged containers for CI pipelines (unlike the default Gitlab runners) which is required for Isar builds
> 
> The project gitlab-cloud-ci is still in a very early stage (the public release was just yesterday).
> 
> We (Siemens) would like to propose this tool for creating an AWS-based build infrastructure in CIP. As a first step, we
> could create a test cluster on AWS and demonstrate building cip-core (deby and isar-based), then discuss further steps.
> 
> Kind regards,
> Michael
> 
> [1] https://gitlab.com/cip-playground/gitlab-cloud-ci
> 
> -- 
> Michael Adler
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE, Otto-Hahn-Ring 6, 81739 Munich, Germany
> 
> Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard Cromme; Managing Board: Joe Kaeser, Chairman, President and Chief Executive Officer; Roland Busch, Lisa Davis, Klaus Helmrich, Janina Kugel, Siegfried Russwurm, Ralf P. Thomas; Registered offices: Berlin and Munich, Germany; Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322
> _______________________________________________
> cip-dev mailing list
> cip-dev at lists.cip-project.org
> https://lists.cip-project.org/mailman/listinfo/cip-dev

-- 
Linaro - Kernel Validation

[cip-dev] [ANNOUNCE] Introducing gitlab-cloud-ci. Perform ISAR-based CI builds in the cloud.

[Adler, Michael]

Hi Dan,

> I thought they billed per-second now days, but looking into it, it seems that may only be for amazon linux and ubuntu
> AMIs?

you are right, it looks like they've changed it quite a while back [1] - at least for AMIs outside of the AWS
Marketplace, if I understood that part correctly.  Currently, I'm just using the kops default (Debian Jessie, with a 4.4
kernel), but Amazon Linux is supposed to work as well [2].

So yes, I think we can reduce the timeout value dramatically (maybe even down to 1 minute), possibly saving quite some
money over time. Thanks for the input, it's much appreciated!

> Do your builds take advantage of any caching, and if so, how are you making it persistent?

Caching is supported in general and up to the user (or rather their .gitlab-ci.yml [3]). IIRC, Gitlab Runner just
creates a zip file of the files/directories you've specified and uploads it to S3.

> I'm also really curious about your experience with k8s, but mostly I'm
> interested to find out how it works for this project over time in terms
> of stability, cost, complexity, etc.

Well, so am I :)

> Thanks again! If there is any place else I can follow information about
> this let me know.

For now, I can advise following the git repo and this mailing list. Code contributions are welcome as well, of course.

Best regards,
Michael

[1] https://aws.amazon.com/blogs/aws/new-per-second-billing-for-ec2-instances-and-ebs-volumes/
[2] https://github.com/kubernetes/kops/blob/release-1.11/docs/images.md
[3] https://docs.gitlab.com/ee/ci/caching/

-- 
Michael Adler
Siemens AG, Corporate Technology, CT RDA IOT SES-DE, Otto-Hahn-Ring 6, 81739 Munich, Germany

Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard Cromme; Managing Board: Joe Kaeser, Chairman, President and Chief Executive Officer; Roland Busch, Lisa Davis, Klaus Helmrich, Janina Kugel, Siegfried Russwurm, Ralf P. Thomas; Registered offices: Berlin and Munich, Germany; Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322

[cip-dev] [ANNOUNCE] Introducing gitlab-cloud-ci. Perform ISAR-based CI builds in the cloud.

[Adler, Michael]

> Alright, we'll probably give it a try on Monday and see how it goes.

Today we performed the initial tests successfully [1]. It took 24 minutes to build the master branch, spending 3 minutes
waiting for the node (4 cores/16GB RAM/150GB NVMe SSD) to come up.

> Once you finish the initial test, we should be able to estimate the cost of using the cluster and probably ask the CIP board members to approve.

The base cost (no CI pipelines included) is $39.24 per month for the Kubernetes master node (t2.medium), which runs 24/7
and spins the slaves up and down, depending on the current workload situation.
The actual cost depends on the utilization (the number of CI pipelines triggered) and the configured node size. If we
take the node size from our example (m5d.xlarge), the AWS cost calculator [2] says:

* An average utilization of 2 hours/day (~4 builds) per month costs $16.60 per month

This scales linearly up and down, i.e. 4 hours/day per month would result in $33.20.  So in total you have the base cost
of $39.24 plus the node cost.  All numbers are for the region Frankfurt (just because it's my default, but you are free
to choose any region you like; e.g. North Virginia is only $13.79 instead of $16.60).

Best regards,
  Michael

[1] https://gitlab.com/cip-project/cip-core/isar-cip-core/pipelines/63516863
[2] https://calculator.s3.amazonaws.com/index.html

-- 
Michael Adler
Siemens AG, Corporate Technology, CT RDA IOT SES-DE, Otto-Hahn-Ring 6, 81739 Munich, Germany

Siemens Aktiengesellschaft: Chairman of the Supervisory Board: Gerhard Cromme; Managing Board: Joe Kaeser, Chairman, President and Chief Executive Officer; Roland Busch, Lisa Davis, Klaus Helmrich, Janina Kugel, Siegfried Russwurm, Ralf P. Thomas; Registered offices: Berlin and Munich, Germany; Commercial registries: Berlin Charlottenburg, HRB 12300, Munich, HRB 6684; WEEE-Reg.-No. DE 23691322