gitlab-ci helper scripts for OpenEmbedded builds

gitlab-ci helper scripts for OpenEmbedded builds

[Yann Dirson]

Hi all,

We released our scripts to help in setting up continuous integration
of a yocto-based project
using Gitlab-CI.  You'll find the repository at
https://github.com/BladeGroup/gitlab-oe

Although we're using this in production, it still has a couple of
rough edges and may need
tuning for different usages.  We'd love to hear how well it fits (or
doesn't fit ;) with other use
cases, and will gladly consider evolutions to make it more generally usable.

Best regards,
-- 
Yann Dirson <yann@blade-group.com>
Blade / Shadow -- http://shadow.tech

Re: gitlab-ci helper scripts for OpenEmbedded builds

[Thomas Goodwin]

[-- Attachment #1: Type: text/plain, Size: 3137 bytes --]

Hi Yann,

Thanks for sharing! We're working through something similar using a tweak
to the CROPS docker containers and GitLab-CI (we started with autobuilder
2, so we've actually merged quite a bit of that experience with our GitLab
setup).

Using docker runners as our cluster, we setup volume mounts to the host
system for caching the shared state and downloads directories (as well as
other artifacts upon successful build).  As part of the CI setup, we then
insert a build/conf/auto.conf file that enforces the usage of that volume
(DL_DIR and SSTATE_DIR).  At the deploy stage (package feeds, etc. like
you're doing), we did the same thing -- copy out to that path so that it
ends up on the network share rather than the container.

Also like you, we're using the "include" and "extends" instructions
throughout so that we have a top-level "common" set of CI YAML includes
alongside a growing set of repositories for each machine-specific build.
Each of those repositories follows the Yocto release branching style so
that we can trace build success to newer releases by simply creating a
branch and using the branch name for the initial clone of the layers.

One thing that came directly from our autobuilder 2 experience is a way to
inject extra variables into the autoconf.  AB2 called this *EXTRAARGS*,
which was simply a list of variables to add.  For our common CI "setup"
stage, we check for a file of the same name (extraargs.conf) and append it
to the auto.conf if it exists.  In this way our downstream projects can
simply include that file if necessary and version control it as new release
branches are added.

One of the drawbacks we've seen in going this route is that every
repository has its own build timeout limit, which is always laughably
small.  There's a backlog issue on GitLab for a global definition of this
value, but it's slated for 12.7 (I think).

I'm hoping to put a write-up out soon with examples for how this all worked
together, but I want to hold off until we can get the CROPS changes we
needed upstreamed in a way that doesn't break those containers for everyone
else (something about the entrypoint doesn't appreciate the runner's
bash/shell detection script).

Cheers,

Thomas

Geon Technologies, LLC


On Mon, Sep 23, 2019 at 5:50 PM Yann Dirson <yann.dirson@blade-group.com>
wrote:

> Hi all,
>
> We released our scripts to help in setting up continuous integration
> of a yocto-based project
> using Gitlab-CI.  You'll find the repository at
> https://github.com/BladeGroup/gitlab-oe
>
> Although we're using this in production, it still has a couple of
> rough edges and may need
> tuning for different usages.  We'd love to hear how well it fits (or
> doesn't fit ;) with other use
> cases, and will gladly consider evolutions to make it more generally
> usable.
>
> Best regards,
> --
> Yann Dirson <yann@blade-group.com>
> Blade / Shadow -- http://shadow.tech
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 4719 bytes --]

Re: gitlab-ci helper scripts for OpenEmbedded builds

[Robert Berger@yocto.user]

Hi,

On 23/09/2019 23:48, Yann Dirson wrote:
> Hi all,
> 
> We released our scripts to help in setting up continuous integration
> of a yocto-based project
> using Gitlab-CI.  You'll find the repository at
> https://github.com/BladeGroup/gitlab-oe

I am currently not too far away from cern.ch ;)

Thanks!

Regards,

Robert

Re: gitlab-ci helper scripts for OpenEmbedded builds

[Robert Berger@yocto.user]

Hi Thomas,

Please see my comments below.

On 24/09/2019 14:36, Thomas Goodwin wrote:
> Hi Yann,
> 
> Thanks for sharing! We're working through something similar using a 
> tweak to the CROPS docker containers and GitLab-CI (we started with 
> autobuilder 2, so we've actually merged quite a bit of that experience 
> with our GitLab setup).
> 

I really would like to discuss details about your setup.
Seems quite interesting!

> 
> Cheers,
> 
> Thomas
> 
> Geon Technologies, LLC
> 

Regards,

Robert

Re: gitlab-ci helper scripts for OpenEmbedded builds

[Yann Dirson]

[-- Attachment #1: Type: text/plain, Size: 3556 bytes --]

Hi Thomas,

Le mar. 24 sept. 2019 à 14:36, Thomas Goodwin <btgoodwin@geontech.com> a
écrit :

> Hi Yann,
>
> Thanks for sharing! We're working through something similar using a tweak
> to the CROPS docker containers and GitLab-CI (we started with autobuilder
> 2, so we've actually merged quite a bit of that experience with our GitLab
> setup).
>

Thanks for pointing me to CROPS, that will help bringing Docker support
here :)


> Using docker runners as our cluster, we setup volume mounts to the host
> system for caching the shared state and downloads directories (as well as
> other artifacts upon successful build).  As part of the CI setup, we then
> insert a build/conf/auto.conf file that enforces the usage of that volume
> (DL_DIR and SSTATE_DIR).  At the deploy stage (package feeds, etc. like
> you're doing), we did the same thing -- copy out to that path so that it
> ends up on the network share rather than the container.
>
> Also like you, we're using the "include" and "extends" instructions
> throughout so that we have a top-level "common" set of CI YAML includes
> alongside a growing set of repositories for each machine-specific build.
> Each of those repositories follows the Yocto release branching style so
> that we can trace build success to newer releases by simply creating a
> branch and using the branch name for the initial clone of the layers.
>
> One thing that came directly from our autobuilder 2 experience is a way to
> inject extra variables into the autoconf.  AB2 called this *EXTRAARGS*,
> which was simply a list of variables to add.  For our common CI "setup"
> stage, we check for a file of the same name (extraargs.conf) and append it
> to the auto.conf if it exists.  In this way our downstream projects can
> simply include that file if necessary and version control it as new release
> branches are added.
>
> One of the drawbacks we've seen in going this route is that every
> repository has its own build timeout limit, which is always laughably
> small.  There's a backlog issue on GitLab for a global definition of this
> value, but it's slated for 12.7 (I think).
>
> I'm hoping to put a write-up out soon with examples for how this all
> worked together, but I want to hold off until we can get the CROPS changes
> we needed upstreamed in a way that doesn't break those containers for
> everyone else (something about the entrypoint doesn't appreciate the
> runner's bash/shell detection script).
>

Looking forward to this!


> Cheers,
>
> Thomas
>
> Geon Technologies, LLC
>
>
> On Mon, Sep 23, 2019 at 5:50 PM Yann Dirson <yann.dirson@blade-group.com>
> wrote:
>
>> Hi all,
>>
>> We released our scripts to help in setting up continuous integration
>> of a yocto-based project
>> using Gitlab-CI.  You'll find the repository at
>> https://github.com/BladeGroup/gitlab-oe
>>
>> Although we're using this in production, it still has a couple of
>> rough edges and may need
>> tuning for different usages.  We'd love to hear how well it fits (or
>> doesn't fit ;) with other use
>> cases, and will gladly consider evolutions to make it more generally
>> usable.
>>
>> Best regards,
>> --
>> Yann Dirson <yann@blade-group.com>
>> Blade / Shadow -- http://shadow.tech
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>>
>

-- 
Yann Dirson <yann@blade-group.com>
Blade / Shadow -- http://shadow.tech

[-- Attachment #2: Type: text/html, Size: 5938 bytes --]

Re: gitlab-ci helper scripts for OpenEmbedded builds

[Thomas Goodwin]

[-- Attachment #1: Type: text/plain, Size: 4984 bytes --]

The CROPS changes are all upstream now, so I've put up an example gist
here: https://gist.github.com/btgoodwin/028c3a63315a27cae3c5989455058eef.

There are comments throughout the files, and I'm still hammering out the
"deploy" stage.  There are ongoing quirks like in the case of deploying the
SDK, despite running the populate_sdk task at the start of the stage, the
related sdk directory doesn't exist or is empty as if it was not unpacked
from the shared state cache (I could not replicate this behavior at my
desk).  Another odd behavior was getting a bitbake worker halt error as if
the container ran out of space or memory, but the server doesn't agree with
that information.

One improvement I would suggest for anyone going this route would be to
have 2 separate definitions for artifacts.  The files only get touched in
the setup stage, so that should be the long-running artifact to collect
whereas all others could be trimmed back to perhaps a few minutes of keep
time.  This would prevent a copy of those artifacts being present at every
job and retained for a week (which would be a good GB for a whole pipeline
depending on how many layers and the size of the layer history).

Cheers,

Thomas


On Wed, Sep 25, 2019 at 5:17 AM Yann Dirson <yann.dirson@blade-group.com>
wrote:

> Hi Thomas,
>
> Le mar. 24 sept. 2019 à 14:36, Thomas Goodwin <btgoodwin@geontech.com> a
> écrit :
>
>> Hi Yann,
>>
>> Thanks for sharing! We're working through something similar using a tweak
>> to the CROPS docker containers and GitLab-CI (we started with autobuilder
>> 2, so we've actually merged quite a bit of that experience with our GitLab
>> setup).
>>
>
> Thanks for pointing me to CROPS, that will help bringing Docker support
> here :)
>
>
>> Using docker runners as our cluster, we setup volume mounts to the host
>> system for caching the shared state and downloads directories (as well as
>> other artifacts upon successful build).  As part of the CI setup, we then
>> insert a build/conf/auto.conf file that enforces the usage of that volume
>> (DL_DIR and SSTATE_DIR).  At the deploy stage (package feeds, etc. like
>> you're doing), we did the same thing -- copy out to that path so that it
>> ends up on the network share rather than the container.
>>
>> Also like you, we're using the "include" and "extends" instructions
>> throughout so that we have a top-level "common" set of CI YAML includes
>> alongside a growing set of repositories for each machine-specific build.
>> Each of those repositories follows the Yocto release branching style so
>> that we can trace build success to newer releases by simply creating a
>> branch and using the branch name for the initial clone of the layers.
>>
>> One thing that came directly from our autobuilder 2 experience is a way
>> to inject extra variables into the autoconf.  AB2 called this *EXTRAARGS*,
>> which was simply a list of variables to add.  For our common CI "setup"
>> stage, we check for a file of the same name (extraargs.conf) and append it
>> to the auto.conf if it exists.  In this way our downstream projects can
>> simply include that file if necessary and version control it as new release
>> branches are added.
>>
>> One of the drawbacks we've seen in going this route is that every
>> repository has its own build timeout limit, which is always laughably
>> small.  There's a backlog issue on GitLab for a global definition of this
>> value, but it's slated for 12.7 (I think).
>>
>> I'm hoping to put a write-up out soon with examples for how this all
>> worked together, but I want to hold off until we can get the CROPS changes
>> we needed upstreamed in a way that doesn't break those containers for
>> everyone else (something about the entrypoint doesn't appreciate the
>> runner's bash/shell detection script).
>>
>
> Looking forward to this!
>
>
>> Cheers,
>>
>> Thomas
>>
>> Geon Technologies, LLC
>>
>>
>> On Mon, Sep 23, 2019 at 5:50 PM Yann Dirson <yann.dirson@blade-group.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> We released our scripts to help in setting up continuous integration
>>> of a yocto-based project
>>> using Gitlab-CI.  You'll find the repository at
>>> https://github.com/BladeGroup/gitlab-oe
>>>
>>> Although we're using this in production, it still has a couple of
>>> rough edges and may need
>>> tuning for different usages.  We'd love to hear how well it fits (or
>>> doesn't fit ;) with other use
>>> cases, and will gladly consider evolutions to make it more generally
>>> usable.
>>>
>>> Best regards,
>>> --
>>> Yann Dirson <yann@blade-group.com>
>>> Blade / Shadow -- http://shadow.tech
>>> --
>>> _______________________________________________
>>> yocto mailing list
>>> yocto@yoctoproject.org
>>> https://lists.yoctoproject.org/listinfo/yocto
>>>
>>
>
> --
> Yann Dirson <yann@blade-group.com>
> Blade / Shadow -- http://shadow.tech
>
>

[-- Attachment #2: Type: text/html, Size: 7771 bytes --]