bitbake of individual repositories

bitbake of individual repositories

[Andrew Geissler]

Assuming we get some more compute resources for our community CI, the next
logical step would be to enable bitbake on some individual code repositories.
bmcweb would be my first repo for this because we find the most
regressions there
later in the CI cycle (meta-phosphor) with the Redfish validation tests.

I know we've done some brainstorming in the past on how we could do bitbake of
code repositories easily and was interested in others thoughts/ideas.

- Modify the recipe to use the nobranch=1 option and have it pull the commit
out of github (all gerrit commits are mirrored to github so you can update a
recipe to pull that from github)

This option is nice in that it's just a two line change to a recipe and then
the normal bitbake process. It's challenging in that we'll probably have to
hard code the association between a repository, its recipe, and its URI/SRCREV
info. I just don't think we've been consistent enough with our recipe names
to ensure this could always be dynamically figured out?

- Use devtool to create a local repo of the repo you want to test and then
copy in the code that triggered the jenkins job.

This option is nice because you already have the code you want to test from the
gerrit trigger in Jenkins. It's challenging in that you'd have to ensure the
devtool modify maps to the repo name (which may not always be the case?) and
you'd have to parse the command line feedback from devtool to know where to
copy the files. devtool is something you'd have to run within the bitbake
environement which adds a bit of complexity since our current design is to
set up the src directory and then just have the docker container build it.

Other ideas out there?

Andrew

Re: bitbake of individual repositories

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]

On Tue, Dec 17, 2019 at 04:02:12PM -0600, Andrew Geissler wrote:
> Other ideas out there?

I've mentioned repo before as it can assist with this.  Bitbake has a
feature where you can set "SRCREV=${AUTOREV}" and it will automatically
pick up what is in the git repo.  What I've done on another project is
this:

   1. Use `repo` to pull down all the source repositories in a consistent
      tree (and ensure it is in the Docker build image, such as in
      /workdir).

   2. Run a small script to fetch the GERRIT_REFSPEC and/or GERRIT_TOPIC
      on top of the checked out `repo` location.

   3. Add a small .bbclass that replaces "SRCREV=${AUTOREV}" when an
      environment variable is set and update the recipes to inherit
      this class.

   4. Set the environment variable prior to running `bitbake` and add a
      source mirror to point at the /workdir location instead of Gerrit
      directly (eg. 1 line in local.conf).

Now when bitbake runs you're getting the full bitbake to run against the
commit (or commits in the case of a GERRIT_TOPIC) that you've extracted.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: bitbake of individual repositories

[Andrew Geissler]

> On Dec 18, 2019, at 12:02 PM, Patrick Williams <patrick@stwcx.xyz> wrote:
> 
> On Tue, Dec 17, 2019 at 04:02:12PM -0600, Andrew Geissler wrote:
>> Other ideas out there?
> 
> I've mentioned repo before as it can assist with this.  Bitbake has a
> feature where you can set "SRCREV=${AUTOREV}" and it will automatically
> pick up what is in the git repo.  What I've done on another project is
> this:

Does this mean all recipes will need to change to do this? I think we’ve
discussed this in the past and stated we like to have more control over
when code is picked up in the meta-* layers.

> 
>   1. Use `repo` to pull down all the source repositories in a consistent
>      tree (and ensure it is in the Docker build image, such as in
>      /workdir).

I’m not familiar with repo. Will this require any changes to our upstream
repositories or is this just a tool we can use from within our CI scripts?

>   2. Run a small script to fetch the GERRIT_REFSPEC and/or GERRIT_TOPIC
>      on top of the checked out `repo` location.
> 
>   3. Add a small .bbclass that replaces "SRCREV=${AUTOREV}" when an
>      environment variable is set and update the recipes to inherit
>      this class.
> 
>   4. Set the environment variable prior to running `bitbake` and add a
>      source mirror to point at the /workdir location instead of Gerrit
>      directly (eg. 1 line in local.conf).

Your option seems even more complicated then the one’s I had above :)
It does come with the added feature of the TOPIC support but our general
statement has been to just avoid requisites.

As with anything though, if you’re willing to put the work in to help make
this happen, and assuming it requires minimal changes to existing
repository layouts and recipes, I’d be interested.

> 
> Now when bitbake runs you're getting the full bitbake to run against the
> commit (or commits in the case of a GERRIT_TOPIC) that you've extracted.
> 
> -- 
> Patrick Williams

Re: bitbake of individual repositories

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 3345 bytes --]

On Thu, Jan 02, 2020 at 03:57:05PM -0600, Andrew Geissler wrote:
> > On Dec 18, 2019, at 12:02 PM, Patrick Williams <patrick@stwcx.xyz> wrote:
> > On Tue, Dec 17, 2019 at 04:02:12PM -0600, Andrew Geissler wrote:
> >> Other ideas out there?
> > 
> > I've mentioned repo before as it can assist with this.  Bitbake has a
> > feature where you can set "SRCREV=${AUTOREV}" and it will automatically
> > pick up what is in the git repo.  What I've done on another project is
> > this:
> 
> Does this mean all recipes will need to change to do this? I think we’ve
> discussed this in the past and stated we like to have more control over
> when code is picked up in the meta-* layers.

It is a pretty simple bbclass that you include in each recipe.  That's
all.

I want to be clear, that you still have the exact same control on what
code is picked up as you have today.  In general the SRCREV is set by
the recipe and is an explicit commit number (or tag), just like today.
Only in the case that a special environment variable is set does SRCREV
become AUTOREV and in that case you use 'repo' to populate exactly the
code you want.  There is never a case where you do not have control on
what code is populated or where you cannot reproduce a build.  All that
is happening is you are delegating responsibility of the SRCREV to repo.

> >   1. Use `repo` to pull down all the source repositories in a consistent
> >      tree (and ensure it is in the Docker build image, such as in
> >      /workdir).
> 
> I’m not familiar with repo. Will this require any changes to our upstream
> repositories or is this just a tool we can use from within our CI scripts?

I have another email chain with Brad where I think repo might be a
reasonable alternative to "merging all the metas together" problem, but
otherwise, they are not related.  You could use repo just for CI.  In
fact, I proposed that as a first step in that email chain (you were CC'd
I believe).

> >   2. Run a small script to fetch the GERRIT_REFSPEC and/or GERRIT_TOPIC
> >      on top of the checked out `repo` location.
> > 
> >   3. Add a small .bbclass that replaces "SRCREV=${AUTOREV}" when an
> >      environment variable is set and update the recipes to inherit
> >      this class.
> > 
> >   4. Set the environment variable prior to running `bitbake` and add a
> >      source mirror to point at the /workdir location instead of Gerrit
> >      directly (eg. 1 line in local.conf).
> 
> Your option seems even more complicated then the one’s I had above :)
> It does come with the added feature of the TOPIC support but our general
> statement has been to just avoid requisites.

I'd probably argue that it is just as complicated as trying to use
devtool to formulate a code package of sorts. ;)

> As with anything though, if you’re willing to put the work in to help make
> this happen, and assuming it requires minimal changes to existing
> repository layouts and recipes, I’d be interested.

I've written it once before but that wasn't open source.  Probably
wouldn't be too hard to recreate something similar.  I should have some
cycles to put towards on OpenBMC CI work the next two months.  I've been
wanting to prototype a `repo` example anyhow per that other email chain.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]