Trouble capturing layer revision as variable in recipe

Trouble capturing layer revision as variable in recipe

[Chris Laplante]

[-- Attachment #1: Type: text/plain, Size: 1777 bytes --]

Hello all,

We are on Yocto 'zeus'.

Our u-boot recipe generates a versions.txt file containing the git commit of the U-Boot source tree. It was pointed out that since recipes can patch the source, just the source commit ID isn't quite enough. So we wanted to also include the git commit of the layer housing the recipe. Call this layer "meta-our-layer".

This is the implementation we tried:

def get_layer_path(d, name):
    layer_list = (d.getVar("BBLAYERS") or "").split()
    return [os.path.abspath(l) for l in layer_list if l.endswith(name)][0]

def get_meta_our_layer_commit(d):
    import subprocess
    output = subprocess.check_output(["git", "rev-parse", "HEAD"], universal_newlines=True, cwd=d.getVar("OUR_LAYER_PATH"))
    return output.strip()

OUR_LAYER_PATH := "${@get_layer_path(d, "meta-our-layer")}"

OUR_LAYER_COMMIT = "${@get_meta_our_layer_commit(d)}"
OUR_LAYER_COMMIT[vardepvalue] = "${@get_meta_our_layer_commit(d)}"

do_deploy_append() {
    echo "meta-our-layer:${OUR_LAYER_COMMIT}" >> ${DEPLOYDIR}/versions.txt
}


However there are two problems:

  1.  Making an empty commit to meta-our-layer doesn't cause the recipe to be re-run. What's weird is that when I use 'bitbake u-boot -e', I can see that the value of  OUR_LAYER_COMMIT is indeed changing. And with bitbake-dumpsig I can see that OUR_LAYER_COMMIT is in the task deps for do_deploy.
  2.  If I build 'u-boot', then make an empty commit to meta-our-layer, and then run "bitbake-whatchanged u-boot", I get a "The metadata is not deterministic and this needs to be fixed." error.

Am I misunderstanding the purpose of vardepvalue? I thought it could be used to here to help bitbake 'see' through values that otherwise change unexpectedly.

Thanks,
Chris

[-- Attachment #2: Type: text/html, Size: 6658 bytes --]

Re: [bitbake-devel] Trouble capturing layer revision as variable in recipe

[Richard Purdie]

On Wed, 2021-08-18 at 19:21 +0000, Chris Laplante via lists.openembedded.org wrote:
> Hello all,
>  
> We are on Yocto ‘zeus’.
>  
> Our u-boot recipe generates a versions.txt file containing the git commit of 
> the U-Boot source tree. It was pointed out that since recipes can patch the 
> source, just the source commit ID isn’t quite enough. So we wanted to also 
> include the git commit of the layer housing the recipe. Call this layer 
> “meta-our-layer”.
>  
> This is the implementation we tried:
>  
> def get_layer_path(d, name):
>     layer_list = (d.getVar("BBLAYERS") or "").split()
>     return [os.path.abspath(l) for l in layer_list if l.endswith(name)][0]
>  
> def get_meta_our_layer_commit(d):
>     import subprocess
>     output = subprocess.check_output(["git", "rev-parse", "HEAD"], universal_newlines=True,
> cwd=d.getVar("OUR_LAYER_PATH"))
>     return output.strip()
>  
> OUR_LAYER_PATH := "${@get_layer_path(d, "meta-our-layer")}"
>  
> OUR_LAYER_COMMIT = "${@get_meta_our_layer_commit(d)}"
> OUR_LAYER_COMMIT[vardepvalue] = "${@get_meta_our_layer_commit(d)}"
>  
> do_deploy_append() {
>     echo “meta-our-layer:${OUR_LAYER_COMMIT}” >> ${DEPLOYDIR}/versions.txt
> }
>  
>  
> However there are two problems:
>    1. Making an empty commit to meta-our-layer doesn’t cause the recipe to be 
> re-run. What’s weird is that when I use ‘bitbake u-boot –e’, I can see that 
> the value of  OUR_LAYER_COMMIT is indeed changing. And with bitbake-dumpsig
>  I can see that OUR_LAYER_COMMIT is in the task deps for do_deploy.
>    2. If I build ‘u-boot’, then make an empty commit to meta-our-layer, and 
> then run “bitbake-whatchanged u-boot”, I get a “The metadata is not deterministic
>  and this needs to be fixed.” error.
>  
> Am I misunderstanding the purpose of vardepvalue? I thought it could be used to 
> here to help bitbake ‘see’ through values that otherwise change unexpectedly.

I suspect you need to have bitbake always reparse the recipe too. Maybe
BB_DONT_CACHE = "1" ?

Cheers,

Richard

Re: [bitbake-devel] Trouble capturing layer revision as variable in recipe

[Quentin Schulz]

Hi Chris,

On Wed, Aug 18, 2021 at 07:21:05PM +0000, Chris Laplante via lists.openembedded.org wrote:
> Hello all,
> 
> We are on Yocto 'zeus'.
> 
> Our u-boot recipe generates a versions.txt file containing the git commit of the U-Boot source tree. It was pointed out that since recipes can patch the source, just the source commit ID isn't quite enough. So we wanted to also include the git commit of the layer housing the recipe. Call this layer "meta-our-layer".
> 
> This is the implementation we tried:
> 
> def get_layer_path(d, name):
>     layer_list = (d.getVar("BBLAYERS") or "").split()
>     return [os.path.abspath(l) for l in layer_list if l.endswith(name)][0]
> 
> def get_meta_our_layer_commit(d):
>     import subprocess
>     output = subprocess.check_output(["git", "rev-parse", "HEAD"], universal_newlines=True, cwd=d.getVar("OUR_LAYER_PATH"))
>     return output.strip()
> 
> OUR_LAYER_PATH := "${@get_layer_path(d, "meta-our-layer")}"
> 

Maybe LAYERDIR works?

But for sure THISDIR would work.

So just replace cwd with d.getVar("THISDIR") instead of the complex
handling since git rev-parse can run in a dir different from the root.

> OUR_LAYER_COMMIT = "${@get_meta_our_layer_commit(d)}"
> OUR_LAYER_COMMIT[vardepvalue] = "${@get_meta_our_layer_commit(d)}"
> 
> do_deploy_append() {
>     echo "meta-our-layer:${OUR_LAYER_COMMIT}" >> ${DEPLOYDIR}/versions.txt
> }
> 

If you have logic around this versions.txt make sure this logic depends
on the do_deploy of u-boot recipe (DEPENDS/RDEPENDS is *NOT* enough, it
is susceptible to race conditions).

Cheers,
Quentin

Re: [bitbake-devel] Trouble capturing layer revision as variable in recipe

[Enrico Scholz]

"Chris Laplante via lists.openembedded.org"
<chris.laplante=agilent.com@lists.openembedded.org> writes:

> OUR_LAYER_PATH := "${@get_layer_path(d, "meta-our-layer")}"

fwiw, we capture this location by 'LAYERDIR_<layer> := "${LAYERDIR}"'
within our layer.conf[1].

It would be nice, when bitbake adds such a variable automatically...


Enrico

Footnotes:
[1]  https://github.com/sigma-embedded/meta-de.sigma-chemnitz/blob/5c4f79148e72d5c009be322e5cc32256ffb58bb4/conf/layer.conf#L17

Re: [bitbake-devel] Trouble capturing layer revision as variable in recipe

[Chris Laplante]

> > However there are two problems:
> >    1. Making an empty commit to meta-our-layer doesn’t cause the
> > recipe to be re-run. What’s weird is that when I use ‘bitbake u-boot
> > –e’, I can see that the value of  OUR_LAYER_COMMIT is indeed changing.
> > And with bitbake-dumpsig  I can see that OUR_LAYER_COMMIT is in the
> task deps for do_deploy.
> >    2. If I build ‘u-boot’, then make an empty commit to
> > meta-our-layer, and then run “bitbake-whatchanged u-boot”, I get a
> > “The metadata is not deterministic  and this needs to be fixed.” error.
> >
> > Am I misunderstanding the purpose of vardepvalue? I thought it could
> > be used to here to help bitbake ‘see’ through values that otherwise change
> unexpectedly.
> 
> I suspect you need to have bitbake always reparse the recipe too. Maybe
> BB_DONT_CACHE = "1" ?

Thanks for the suggestion. I guess this is because unless BitBake sees that the recipe file(s) changed (or they use SRCREV = AUTOREV), it won't reparse stuff in general?

Thanks,
Chris

 

Re: [bitbake-devel] Trouble capturing layer revision as variable in recipe

[Chris Laplante]

> > OUR_LAYER_PATH := "${@get_layer_path(d, "meta-our-layer")}"
> >
> 
> Maybe LAYERDIR works?
> 
> But for sure THISDIR would work.
> 
> So just replace cwd with d.getVar("THISDIR") instead of the complex
> handling since git rev-parse can run in a dir different from the root.

Thanks, but one detail I left out is that this is a bbappended recipe. The bbappend lives in meta-our-layer, so LAYERDIR/THISDIR won't work here.

Chris

Re: [bitbake-devel] Trouble capturing layer revision as variable in recipe

[Richard Purdie]

On Wed, 2021-08-25 at 15:26 +0000, chris.laplante@agilent.com wrote:
> > > However there are two problems:
> > >    1. Making an empty commit to meta-our-layer doesn’t cause the
> > > recipe to be re-run. What’s weird is that when I use ‘bitbake u-boot
> > > –e’, I can see that the value of  OUR_LAYER_COMMIT is indeed changing.
> > > And with bitbake-dumpsig  I can see that OUR_LAYER_COMMIT is in the
> > task deps for do_deploy.
> > >    2. If I build ‘u-boot’, then make an empty commit to
> > > meta-our-layer, and then run “bitbake-whatchanged u-boot”, I get a
> > > “The metadata is not deterministic  and this needs to be fixed.” error.
> > > 
> > > Am I misunderstanding the purpose of vardepvalue? I thought it could
> > > be used to here to help bitbake ‘see’ through values that otherwise change
> > unexpectedly.
> > 
> > I suspect you need to have bitbake always reparse the recipe too. Maybe
> > BB_DONT_CACHE = "1" ?
> 
> Thanks for the suggestion. I guess this is because unless BitBake sees that
> the recipe file(s) changed (or they use SRCREV = AUTOREV), it won't reparse
> stuff in general?

Correct, it doesn't reparse at each invocation as in general that would upset
users! It doesn't know it needs to reparse in this case.

Cheers,

Richard