Incrementally building constantly changing application source in a workflow-friendly way with #yocto

Incrementally building constantly changing application source in a workflow-friendly way with #yocto

[Pieter Smith]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

I am defining a meta-layer with our business logic (applications), one or two recipes and an image definition. I would like to maintain the C/C++ sources with this meta-layer in a single repository, because they are tightly coupled from a lifecycle perspective.

Is there a way to always build the current state of the source-code from a path relative to the recipe.bb file in a workflow friendly way? After scouring the web and mailing lists for days, I have no satisfactory answer. With OpenWRT, Buildroot 2 and ptxdist this was extremely simple to achieve.

My requirements are:
* Treat the meta-layer, it's recipes and images as well as package source-code as a monolithic whole with a single lifecycle within a single git repository,
* Have the current checked out state of git repository built whenever I bake an image,
* Not have things fall apart when I check-out in another location.
I do not mind having to do a commit before each build so that the state can be tracked, but having to push and edit the recipe with the new commit ID so every change can be built deployed and tested is driving me nuts, especially since some changes are cross application, requiring an atomic update of both applications.

[-- Attachment #2: Type: text/html, Size: 1284 bytes --]

Re: [yocto] Incrementally building constantly changing application source in a workflow-friendly way with #yocto

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1719 bytes --]

You can specify a directory relative to the one the recipe is in in
SRC_URI. Bitbake will checksum all files in it (so will trigger a re-build
if they change), and will copy the directory into the $WORKDIR. I have
never actually tried this, but I'd start there, maybe with a hello world
type project for proof of concept.

Alex

On Thu, 13 Feb 2020 at 17:29, Pieter Smith <smipi01@gmail.com> wrote:

> I am defining a meta-layer with our business logic (applications), one or
> two recipes and an image definition. I would like to maintain the C/C++
> sources with this meta-layer in a single repository, because they are
> tightly coupled from a lifecycle perspective.
>
> Is there a way to always build the current state of the source-code from a
> path relative to the recipe.bb file in a workflow friendly way? After
> scouring the web and mailing lists for days, I have no satisfactory answer.
> With OpenWRT, Buildroot 2 and ptxdist this was extremely simple to achieve.
>
> My requirements are:
>
>    1. Treat the meta-layer, it's recipes and images as well as package
>    source-code as a monolithic whole with a single lifecycle within a single
>    git repository,
>    2. Have the current checked out state of git repository built whenever
>    I bake an image,
>    3. Not have things fall apart when I check-out in another location.
>
> I do not mind having to do a commit before each build so that the state
> can be tracked, but having to push and edit the recipe with the new commit
> ID so every change can be built deployed and tested is driving me nuts,
> especially since some changes are cross application, requiring an atomic
> update of both applications. 
>

[-- Attachment #2: Type: text/html, Size: 2074 bytes --]

Re: Incrementally building constantly changing application source in a workflow-friendly way with #yocto

[Pieter Smith]

[-- Attachment #1: Type: text/plain, Size: 743 bytes --]

Thanks Alex,

I tried this, but still cannot get it set-up so that the sources are fetched. I have the following directory structure in my repo:

├── conf
│   └── layer.conf
├── COPYING.MIT
├── README
├── recipes-images
│   └── my-image
│       └── my-image.bb
├── recipes-my-application
│   └── my-application
│       └── my-application_1.0.bb
└── src
├── my-application.c
├── secrets.c
└── secrets.h

None of the following in my-application_1.0.bb succeeds in fetching the source:

SRC_URI = "file://../src/*"
SRC_URI = "file://../src/"
SRC_URI = "file://src/*"
SRC_URI = "file://src/"

What am I missing?

[-- Attachment #2: Type: text/html, Size: 2176 bytes --]

Re: [yocto] Incrementally building constantly changing application source in a workflow-friendly way with #yocto

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

Try placing the sources into recipes-my-application/files/src, then SRC_URI
= "file://src/" should work.

Alex

On Fri, 14 Feb 2020 at 10:52, Pieter Smith <smipi01@gmail.com> wrote:

> Thanks Alex,
>
> I tried this, but still cannot get it set-up so that the sources are
> fetched. I have the following directory structure in my repo:
> ├── conf
> │   └── layer.conf
> ├── COPYING.MIT
> ├── README
> ├── recipes-images
> │   └── my-image
> │       └── my-image.bb
> ├── recipes-my-application
> │   └── my-application
> │       └── my-application_1.0.bb
> └── src
>     ├── my-application.c
>     ├── secrets.c
>     └── secrets.h
>
> None of the following in my-application_1.0.bb succeeds in fetching the
> source:
>
> SRC_URI = "file://../src/*"
> SRC_URI = "file://../src/"
> SRC_URI = "file://src/*"
> SRC_URI = "file://src/"
>
> What am I missing?
>
> 
>

[-- Attachment #2: Type: text/html, Size: 2724 bytes --]

Re: Incrementally building constantly changing application source in a workflow-friendly way with #yocto

[Pieter Smith]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

Thanks Alex!

This also works if files/src is a symlink to the actual sources. I just need to take into account that the sources land in $WORKDIR/src.

I still need to do the following though to get the behaviour that I actually need:

bitbake -f -c clean my-application && \
bitbake -f -c fetch my-application && \
bitbake ...

Can you suggest a more elegant way to achieve this?

Pieter

[-- Attachment #2: Type: text/html, Size: 472 bytes --]

Re: [yocto] Incrementally building constantly changing application source in a workflow-friendly way with #yocto

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

This should simply work; what exactly doesn't work for you?

Do wipe tmp/ before proceeding further, as -f 'taints' the build.

Alex

On Fri, 14 Feb 2020 at 11:45, Pieter Smith <smipi01@gmail.com> wrote:

> Thanks Alex!
>
> This also works if files/src is a symlink to the actual sources. I just
> need to take into account that the sources land in $WORKDIR/src.
>
> I still need to do the following though to get the behaviour that I
> actually need:
>
> bitbake -f -c clean my-application && \
> bitbake -f -c fetch my-application && \
> bitbake ...
>
> Can you suggest a more elegant way to achieve this?
>
> Pieter
> 
>

[-- Attachment #2: Type: text/html, Size: 1042 bytes --]

Re: Incrementally building constantly changing application source in a workflow-friendly way with #yocto

[Pieter Smith]

[-- Attachment #1: Type: text/plain, Size: 183 bytes --]

Once `my-application` is built, changes to the contents of `src` does not result in a rebuild for `bitbake my-application`. The contents of `src` also isn't synched to the $WORKDIR.

[-- Attachment #2: Type: text/html, Size: 183 bytes --]

Re: [yocto] Incrementally building constantly changing application source in a workflow-friendly way with #yocto

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 563 bytes --]

Can you issue a 'bitbake-dumpsig -t fetch recipename'

(writing from memory, so exact syntax may vary).

This should show what goes into the task signature for the fetcher, and all
individual files in src should be in it. I'm fairly sure I saw this on the
recent master branch.

Alex

On Fri, 14 Feb 2020 at 14:17, Pieter Smith <smipi01@gmail.com> wrote:

> Once `my-application` is built, changes to the contents of `src` does not
> result in a rebuild for `bitbake my-application`. The contents of `src`
> also isn't synched to the $WORKDIR. 
>

[-- Attachment #2: Type: text/html, Size: 954 bytes --]

Re: Incrementally building constantly changing application source in a workflow-friendly way with #yocto

[Pieter Smith]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

Hi Alexander,

Based on your and Oliver Westermann's help, I came up with something that works for our project requirements:

My application bitbake file looks like this:

EXTERNALSRC = "${@os.path.abspath(os.path.join(os.path.dirname(d.getVar('FILE')), '..', '..', 'src'))}"
inherit externalsrc

S = "${EXTERNALSRC}"
inherit cmake

externalsrc.bb expects EXTERNALSRC to contain an absolute path. We have a relocatable workspace based on a git tree with submodules, so I calculate the absolute path of the source based on the bb file location. Our src directory is a cmake project, so I only have to convince cmake to use the EXTERNALSRC location for an out-of-tree build in WORKDIR by setting S.

The workflow for our project developers then becomes:

bitbake -c cleansstate <application-name> && bitbake ...

I can hide this in a wrapper Makefile.

Am I missing anything?

Regards,
Pieter Smith

[-- Attachment #2: Type: text/html, Size: 1149 bytes --]