Best way to mask bbappends based on Poky version to have a layer support multiple versions of Poky?

Best way to mask bbappends based on Poky version to have a layer support multiple versions of Poky?

[Matt Campbell]

[-- Attachment #1: Type: text/plain, Size: 1728 bytes --]

HI All,

We have a layer where we want to concurrently support two releases of Poky.
There is an issue when we have bbappnds against recipes that have
different versions in the two poky releases. for instance, imagine recipe
foo that is version 1.0 in Zeus and 1.2 in Dunfell. If we had a bbappend in
our layer `foo_1.0.bbappend` and tried to use our layer with Dunfell,
bitbake will error out saying that `foo_1.0.bbappend` has no base recipe.

The question is: how do we dynamically hide that bbappend only from
Dunfell? Or, why is what we are trying to do not really the right

I found BBFILES_DYNAMIC which is _alomst_ what we want, but only allows
filtering based on the presence of another layer, not the version of Poky.

Right now we've got a solution using Python inline variable substitution to
set BBMASK conditionally based on the poky version. (Side note: it looks
like you can't do anonymous Python in the layer.conf file which I couldn't
find any documentation about).

# Things to mask from zeus
BBMASK += "${@ ' '.join([ \

'meta-izo-modus/recipes-devtools/libedit/libedit_20180525-3.1.bbappend', \
        'meta-izo-modus/recipes-devtools/lua/lua_5.3.4.bbappend', \

'meta-izo-modus/recipes-extended/logrotate/logrotate_3.14.0.bbappend', \
        'meta-izo-modus/recipes-kernel/linux/linux-yocto_4.18.bbappend', \
    ]) if d.getVar('LAYERSERIES_CORENAMES') == 'zeus' else ''}"

This isn't the cleanest, but it does work.

I'm really curious if there Is there a more canonical and clean way of
achieving what we are going for here.

Thanks in advance,
~Matt
-- 
Matthew Campbell
Senior Embedded Systems Engineer
mcampbell@izotope.com

iZotope, Inc.
www.izotope.com

[-- Attachment #2: Type: text/html, Size: 2347 bytes --]

Re: [yocto] Best way to mask bbappends based on Poky version to have a layer support multiple versions of Poky?

[rpjday]

On Thu, 26 Mar 2020, Matt Campbell wrote:

> HI All,

> We have a layer where we want to concurrently support two releases
> of Poky. There is an issue when we have bbappnds against recipes
> that have different versions in the two poky releases. for instance,
> imagine recipe foo that is version 1.0 in Zeus and 1.2 in Dunfell.
> If we had a bbappend in our layer `foo_1.0.bbappend` and tried to
> use our layer with Dunfell, bitbake will error out saying that
> `foo_1.0.bbappend` has no base recipe.

  not sure if this really solves the underlying issue, but you can
always turn those errors into warnings with:

  BB_DANGLINGAPPENDS_WARNONLY = "1"

in your local.conf, although i'm still skeptical as to whether that's
really the problem you're trying to solve.

rday

Re: [yocto] Best way to mask bbappends based on Poky version to have a layer support multiple versions of Poky?

[Matt Campbell]

[-- Attachment #1: Type: text/plain, Size: 2589 bytes --]

I didn't know about  BB_DANGLINGAPPENDS_WARNONLY. That would mask the
problem, but doesn't feel like a great solution. Either way I do
appreciate you sharing that.

Further implementation and discussion with our team brought up another
possible solution. We could wildcard all bbappends (_%.bbappend) and use
some anonymous python inside our bbappend files that will error out if the
package version isn't in a supported list. We could also easily roll this
up into a bbclass to prevent the need to duplicate this everywhere.

python () {
    package_version = d.getVar("PV")
    if  package_version is not in ['3.14", "3.15"]:
        bb.error("This bbappend file isn't compatible with the version {}.
You will need to add support to this bbappend for that
version.".format(package_version))
}

This still seems more like we are fighting bitbake rather than working with
it. Does anyone have any thoughts or suggestions on this?

Other upstreams seem to maintain different branches for different Poky
releases. That is a road we would rather avoid if possible. Our goal is to
be able to have an extra CI build against the version of Poky under
development so we can continuously fix the upgrade issues as they come up
rather than as a landslide when we upgrade. Making a separate branch for
this would mean we would need to merge all active development into each
branch to get the benefits of a poky next canary build plan. That said, I'd
love to hear about a solution that lets us have our cake and eat it too.

~Matt

On Thu, Mar 26, 2020 at 9:55 AM Robert P. J. Day <rpjday@crashcourse.ca>
wrote:

> On Thu, 26 Mar 2020, Matt Campbell wrote:
>
> > HI All,
>
> > We have a layer where we want to concurrently support two releases
> > of Poky. There is an issue when we have bbappnds against recipes
> > that have different versions in the two poky releases. for instance,
> > imagine recipe foo that is version 1.0 in Zeus and 1.2 in Dunfell.
> > If we had a bbappend in our layer `foo_1.0.bbappend` and tried to
> > use our layer with Dunfell, bitbake will error out saying that
> > `foo_1.0.bbappend` has no base recipe.
>
>   not sure if this really solves the underlying issue, but you can
> always turn those errors into warnings with:
>
>   BB_DANGLINGAPPENDS_WARNONLY = "1"
>
> in your local.conf, although i'm still skeptical as to whether that's
> really the problem you're trying to solve.
>
> rday
>


-- 
Matthew Campbell
Senior Embedded Systems Engineer
mcampbell@izotope.com

iZotope, Inc.
www.izotope.com

[-- Attachment #2: Type: text/html, Size: 3415 bytes --]

Re: [yocto] Best way to mask bbappends based on Poky version to have a layer support multiple versions of Poky?

[Konrad Weihmann]

[-- Attachment #1: Type: text/plain, Size: 3464 bytes --]

Hi,

I'll get your point.
Maybe this could be a solution to your problem 
https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#var-BBVERSIONS.
Instead of having different bbappends have one and pick the right steps 
inside of the append.

If that is not working for you, you could also fake the behavior of 
BBFILES_DYNAMIC with putting the bbappend into separate subfolder, which 
than are
referenced by something like this

BBFILES += "${LAYERDIR}/dynamic-recipes/${BB_VERSION}/*.bbappend"

Regards
Konrad

On 26.03.20 20:56, Matt Campbell wrote:
> I didn't know about BB_DANGLINGAPPENDS_WARNONLY. That would mask the 
> problem, but doesn't feel like a great solution. Either way I do 
> appreciate you sharing that.
>
> Further implementation and discussion with our team brought up another 
> possible solution. We could wildcard all bbappends (_%.bbappend) and 
> use some anonymous python inside our bbappend files that will error 
> out if the package version isn't in a supported list. We could also 
> easily roll this up into a bbclass to prevent the need to duplicate 
> this everywhere.
>
> python () {
>     package_version = d.getVar("PV")
>     if  package_version is not in ['3.14", "3.15"]:
>         bb.error("This bbappend file isn't compatible with the version 
> {}. You will need to add support to this bbappend for that 
> version.".format(package_version))
> }
>
> This still seems more like we are fighting bitbake rather than working 
> with it. Does anyone have any thoughts or suggestions on this?
>
> Other upstreams seem to maintain different branches for different Poky 
> releases. That is a road we would rather avoid if possible. Our goal 
> is to be able to have an extra CI build against the version of Poky 
> under development so we can continuously fix the upgrade issues as 
> they come up rather than as a landslide when we upgrade. Making a 
> separate branch for this would mean we would need to merge all active 
> development into each branch to get the benefits of a poky next canary 
> build plan. That said, I'd love to hear about a solution that lets us 
> have our cake and eat it too.
>
> ~Matt
>
> On Thu, Mar 26, 2020 at 9:55 AM Robert P. J. Day 
> <rpjday@crashcourse.ca <mailto:rpjday@crashcourse.ca>> wrote:
>
>     On Thu, 26 Mar 2020, Matt Campbell wrote:
>
>     > HI All,
>
>     > We have a layer where we want to concurrently support two releases
>     > of Poky. There is an issue when we have bbappnds against recipes
>     > that have different versions in the two poky releases. for instance,
>     > imagine recipe foo that is version 1.0 in Zeus and 1.2 in Dunfell.
>     > If we had a bbappend in our layer `foo_1.0.bbappend` and tried to
>     > use our layer with Dunfell, bitbake will error out saying that
>     > `foo_1.0.bbappend` has no base recipe.
>
>       not sure if this really solves the underlying issue, but you can
>     always turn those errors into warnings with:
>
>       BB_DANGLINGAPPENDS_WARNONLY = "1"
>
>     in your local.conf, although i'm still skeptical as to whether that's
>     really the problem you're trying to solve.
>
>     rday
>
>
>
> -- 
> Matthew Campbell
> Senior Embedded Systems Engineer
> mcampbell@izotope.com <mailto:mcampbell@izotope.com>
>
> iZotope, Inc.
> www.izotope.com <http://www.izotope.com>
>
> 

[-- Attachment #2: Type: text/html, Size: 5801 bytes --]

Re: [yocto] Best way to mask bbappends based on Poky version to have a layer support multiple versions of Poky?

[Matt Campbell]

[-- Attachment #1: Type: text/plain, Size: 5201 bytes --]

Thanks for the info. I think both of those will run into issues where the
bbappend won't apply to the original recipe as that is fully done by path
and file name and both of these methods mangle those. Please correct me if
I'm wrong as there might be some cheese down that tunnel for me.

I've been thinking more about this over the weekend and have rethought the
problem a bit. I think what I really want is to be able to conditionally
skip a bbappend in recipe using python rather than having to do this
centralized in the layer.conf with BBMASK. There are two reasons for this:

1) This puts the logic that skips the bbappend in the file itself which
makes it more obvious what is going on when browsing/maintaining meta-data.
The BBMASK method is centralized, but is hidden when you are just working
with the metadata.
2) This allows for arbitrary and more complex logic for skipping bbappends.

I'm not super familiar with the internals of bitbake, but I think what I
want is something similar to the bb.parse.SkipRecipe() exception. For
example

file: test_%.bbappend
python() {
    # Skip based on poky version
    if d.getVar("DISTRO_CODENAME") != 'zeus':
        raise bb.parse.SkipBbappend("This bbappend only supports Zeus.
Skipping this bbappend")

    # Skip based on package version
    if d.getVar("PV") != '1.0':
        raise bb.parse.SkipBbappend("This bbappend only supports version
1.0 of test. Skipping bbappend")
}

1) Does this seem like a good solution to the problem of dangling appends
in layers that want to support two versions of Poky?
2) Any thoughts on how to go about implementing this?

~Matt

On Thu, Mar 26, 2020 at 4:09 PM Konrad Weihmann <kweihmann@outlook.com>
wrote:

> Hi,
>
> I'll get your point.
> Maybe this could be a solution to your problem
> https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#var-BBVERSIONS
> .
> Instead of having different bbappends have one and pick the right steps
> inside of the append.
>
> If that is not working for you, you could also fake the behavior of
> BBFILES_DYNAMIC with putting the bbappend into separate subfolder, which
> than are
> referenced by something like this
>
> BBFILES += "${LAYERDIR}/dynamic-recipes/${BB_VERSION}/*.bbappend"
>
> Regards
> Konrad
> On 26.03.20 20:56, Matt Campbell wrote:
>
> I didn't know about  BB_DANGLINGAPPENDS_WARNONLY. That would mask the
> problem, but doesn't feel like a great solution. Either way I do
> appreciate you sharing that.
>
> Further implementation and discussion with our team brought up another
> possible solution. We could wildcard all bbappends (_%.bbappend) and use
> some anonymous python inside our bbappend files that will error out if the
> package version isn't in a supported list. We could also easily roll this
> up into a bbclass to prevent the need to duplicate this everywhere.
>
> python () {
>     package_version = d.getVar("PV")
>     if  package_version is not in ['3.14", "3.15"]:
>         bb.error("This bbappend file isn't compatible with the version {}.
> You will need to add support to this bbappend for that
> version.".format(package_version))
> }
>
> This still seems more like we are fighting bitbake rather than working
> with it. Does anyone have any thoughts or suggestions on this?
>
> Other upstreams seem to maintain different branches for different Poky
> releases. That is a road we would rather avoid if possible. Our goal is to
> be able to have an extra CI build against the version of Poky under
> development so we can continuously fix the upgrade issues as they come up
> rather than as a landslide when we upgrade. Making a separate branch for
> this would mean we would need to merge all active development into each
> branch to get the benefits of a poky next canary build plan. That said, I'd
> love to hear about a solution that lets us have our cake and eat it too.
>
> ~Matt
>
> On Thu, Mar 26, 2020 at 9:55 AM Robert P. J. Day <rpjday@crashcourse.ca>
> wrote:
>
>> On Thu, 26 Mar 2020, Matt Campbell wrote:
>>
>> > HI All,
>>
>> > We have a layer where we want to concurrently support two releases
>> > of Poky. There is an issue when we have bbappnds against recipes
>> > that have different versions in the two poky releases. for instance,
>> > imagine recipe foo that is version 1.0 in Zeus and 1.2 in Dunfell.
>> > If we had a bbappend in our layer `foo_1.0.bbappend` and tried to
>> > use our layer with Dunfell, bitbake will error out saying that
>> > `foo_1.0.bbappend` has no base recipe.
>>
>>   not sure if this really solves the underlying issue, but you can
>> always turn those errors into warnings with:
>>
>>   BB_DANGLINGAPPENDS_WARNONLY = "1"
>>
>> in your local.conf, although i'm still skeptical as to whether that's
>> really the problem you're trying to solve.
>>
>> rday
>>
>
>
> --
> Matthew Campbell
> Senior Embedded Systems Engineer
> mcampbell@izotope.com
>
> iZotope, Inc.
> www.izotope.com
>
> 
>
>

-- 
Matthew Campbell
Senior Embedded Systems Engineer
mcampbell@izotope.com

iZotope, Inc.
www.izotope.com

[-- Attachment #2: Type: text/html, Size: 8126 bytes --]