recipe dependency on externally built binaries?

recipe dependency on externally built binaries?

[Brian Karcz]

[-- Attachment #1: Type: text/plain, Size: 1482 bytes --]

Hi,

I'm curious if there is a way to tell a package recipe that its resulting package output is dependent on the pieces it's assembling. The scenario I have is as follows. I created a series of recipes that assemble an externally built executable, and some configurations files that are local to the recipe:

xyz-app.bb

    SRC_URI = "file://<path to xyz-app executable> \
               File://xyz-app.config<File:///\\xyz-app.config>"

    do_install () {
        <make the required directories>
        <put the files in their places>
    }

abc-image.bb

    POKY_EXTRA_INSTALL += "xyz-app"

This setup works for assembling all the pieces I need, but unless the app recipe revision gets bumped, it's not able to tell when the xyz-app executable or xyz-app.config get changed and create a new xyz-app package file. Is there a way to tell the recipe that there is a dependency for its output package on the contents of the SRC_URI line without having to bump the revision?

These used to be assembled in an image preprocess command, but I wanted to break out some pieces, that are common to multiple images, into their own recipes in a common layer. In the prior model, all the apps and their config files (that were local to the image recipe) were assembled every time the image was bitbaked. Now that they are in their own recipes, some sort of dependency is required to re-create the package file that the image is bringing in.

Thanks,
Brian





[-- Attachment #2: Type: text/html, Size: 6774 bytes --]

Re: recipe dependency on externally built binaries?

[Paul Eggleton]

Hi Brian,

On Monday 29 July 2013 19:05:04 Brian Karcz wrote:
> I'm curious if there is a way to tell a package recipe that its resulting
> package output is dependent on the pieces it's assembling. The scenario I
> have is as follows. I created a series of recipes that assemble an
> externally built executable, and some configurations files that are local
> to the recipe:
> 
> xyz-app.bb
> 
>     SRC_URI = "file://<path to xyz-app executable> \
>                File://xyz-app.config<File:///\\xyz-app.config>"
> 
>     do_install () {
>         <make the required directories>
>         <put the files in their places>
>     }
> 
> abc-image.bb
> 
>     POKY_EXTRA_INSTALL += "xyz-app"
> 
> This setup works for assembling all the pieces I need, but unless the app
> recipe revision gets bumped, it's not able to tell when the xyz-app
> executable or xyz-app.config get changed and create a new xyz-app package
> file. Is there a way to tell the recipe that there is a dependency for its
> output package on the contents of the SRC_URI line without having to bump
> the revision?

It depends on which version of the build system you are using. In danny (1.3) 
and later a checksum is performed on every local file referred to in SRC_URI 
(and the checksum will be refreshed if the file's mtime changes); if the 
checksum is different then do_fetch for the recipe and all tasks that depend 
upon it should be re-executed.

However, the fact that you refer to POKY_EXTRA_INSTALL suggests you might be 
using an older version of the build system - the current equivalent variable 
is called CORE_IMAGE_EXTRA_INSTALL. Which version are you currently working 
with?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: recipe dependency on externally built binaries?

[Brian Karcz]

Hi Paul,

That's a better answer than I was fearing. I was starting to think that Yocto/Poky was only intended to build/assemble versioned or released code.

We are still using Edison. I started going through the cycles of trying to upgrade us to Dylan a month or so ago, but ran into an assortment of issues regarding udev (at runtime) and some issues with do_fetch/do_unpack not executing on an image re-build like they do in Edison. My priority list got adjusted a bit, and I had to abandon that effort for a while. This information definitely adds a little more to the benefit column of moving us forward.

Are you saying that if I were to move our build system forward to danny or Dylan that the dependency behavior I'm looking for in these recipes would start to work as intended? Would this behavior survive the use of the rm_work directive?

I found this reply in my SPAM this morning. While I was in there, I found your response to the fetch/unpack issue that you sent a couple weeks back. I'll be replying to that shortly.

Thanks,
Brian



-----Original Message-----
From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com] 
Sent: Wednesday, July 31, 2013 5:52 AM
To: Brian Karcz
Cc: yocto@yoctoproject.org
Subject: Re: [yocto] recipe dependency on externally built binaries?

Hi Brian,

On Monday 29 July 2013 19:05:04 Brian Karcz wrote:
> I'm curious if there is a way to tell a package recipe that its 
> resulting package output is dependent on the pieces it's assembling. 
> The scenario I have is as follows. I created a series of recipes that 
> assemble an externally built executable, and some configurations files 
> that are local to the recipe:
> 
> xyz-app.bb
> 
>     SRC_URI = "file://<path to xyz-app executable> \
>                File://xyz-app.config<File:///\\xyz-app.config>"
> 
>     do_install () {
>         <make the required directories>
>         <put the files in their places>
>     }
> 
> abc-image.bb
> 
>     POKY_EXTRA_INSTALL += "xyz-app"
> 
> This setup works for assembling all the pieces I need, but unless the 
> app recipe revision gets bumped, it's not able to tell when the 
> xyz-app executable or xyz-app.config get changed and create a new 
> xyz-app package file. Is there a way to tell the recipe that there is 
> a dependency for its output package on the contents of the SRC_URI 
> line without having to bump the revision?

It depends on which version of the build system you are using. In danny (1.3) and later a checksum is performed on every local file referred to in SRC_URI (and the checksum will be refreshed if the file's mtime changes); if the checksum is different then do_fetch for the recipe and all tasks that depend upon it should be re-executed.

However, the fact that you refer to POKY_EXTRA_INSTALL suggests you might be using an older version of the build system - the current equivalent variable is called CORE_IMAGE_EXTRA_INSTALL. Which version are you currently working with?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: recipe dependency on externally built binaries?

[Paul Eggleton]

On Wednesday 31 July 2013 14:04:05 Brian Karcz wrote:
> That's a better answer than I was fearing. I was starting to think that
> Yocto/Poky was only intended to build/assemble versioned or released code.

Most of the time that's the way the system operates, but we're not limited to 
it. The main purpose of the checksums is to allow you to update patches or 
configuration files referred to in SRC_URI, but it works for this case as well.

The only thing I'd advise is to take care that you have mechanisms in place 
that will allow you to easily reproduce older branches. You may already have 
taken care of this; it's just that it's easy to overlook when you point to a 
local file that is not under the control of the build system.

> We are still using Edison. I started going through the cycles of trying to
> upgrade us to Dylan a month or so ago, but ran into an assortment of issues
> regarding udev (at runtime) and some issues with do_fetch/do_unpack not
> executing on an image re-build like they do in Edison. My priority list got
> adjusted a bit, and I had to abandon that effort for a while. This
> information definitely adds a little more to the benefit column of moving
> us forward.

Right. FYI, the traditional way of working with edison and older was just to 
bump PR each time you want the recipe to rebuild.

> Are you saying that if I were to move our build system forward to danny or
> Dylan that the dependency behavior I'm looking for in these recipes would
> start to work as intended? 

Yes, that's correct. If you enable the PR server you can even have PR 
incremented automatically each time a change is made, assuming this helps i.e. 
you have package management enabled on the target.

> Would this behavior survive the use of the rm_work directive?

Yes, it should interact just fine with rm_work.
 
Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre