Idea for bbappend / layer organization and naming convention

Idea for bbappend / layer organization and naming convention

[Gunnar Andersson]

Yocto community,

Triggered by the previous email request to yocto@ about best practices for
layer organization I'm finally firing off this email for (hopefully) some
feedback on an idea we first kicked around, discussed for rough consensus
and then decided to implement in the Yocto based GDP project [1].  You can
see it as a trial, anything can be changed, but so far so good...

We adopted a somewhat novel (but actually not really unique, see inside)
naming convention [2] for all modifications to components that belong to
other layers.

This convention shows what layers are being modified by the .bbappends by
actually naming the layer it is (intending to**) modify.  The initiative
also aims to highlight and separate .bbappends (modifications) from uniquely
added components in our project (new .bb files) in GDP.

**Note that we are well aware this does NOT control bitbake behavior (as
neither does the recipe-xxx/ directory convention), and is therefore just a
guide.  If the project is misconfigured, bitbake could do something
different than the naming convention implies and it could even be
misleading, but at least the *programmer intent* is shown clearly.

I think it also makes .bbappends more explicitly visible and might trigger
the idea:   Do we really need this?  Are we really modifying *that* adopted
layer, and if so, why?

To Russel who wrote the previous request for guidance - feel free to
consider, but I won't recommend this directly since it's by no means an
agreed best practice in all of the community.  I'm rather asking for other
experienced OE developers to consider and comment on the idea.  

For oldtimers it's probably easy to find this odd and just be negative
against it since well, "standard practice", inertia, NIH and all that, but
I'll stick my neck out anyway because it seems to solve some issues of
organization and understanding at least for us (and I'm a sucker for some
flamebait ;)

If the intro is too formal and long, just look at the final examples.  I
think you'll get it.  Feel free to ask.


- Gunnar

-- 
Gunnar Andersson <gandersson@genivi.org>
Development Lead
GENIVI Alliance

[1] https://github.com/GENIVI/genivi-dev-platform
[2] Naming strategy for bitbake extension layers:			
https://at.projects.genivi.org/wiki/x/w4Xk

Re: Idea for bbappend / layer organization and naming convention

[Bruce Ashfield]

On 2017-08-14 3:36 PM, Gunnar Andersson wrote:
> 
> Yocto community,
> 
> Triggered by the previous email request to yocto@ about best practices for
> layer organization I'm finally firing off this email for (hopefully) some
> feedback on an idea we first kicked around, discussed for rough consensus
> and then decided to implement in the Yocto based GDP project [1].  You can
> see it as a trial, anything can be changed, but so far so good...
> 
> We adopted a somewhat novel (but actually not really unique, see inside)
> naming convention [2] for all modifications to components that belong to
> other layers.
> 

I've been using a similar naming/sorting mechanism in layers that
I maintain for several years now. When you have a single layer that
is carrying bbappends to many other layers, I find that it really
helps keep everything separated and aids finding the original
recipe.

(that being said, recent work with layer index, etc, make it
easier to locate the original recipe and any bbappends .. but that
doesn't preclude keeping things organized in a layer).

Cheers,

Bruce

> This convention shows what layers are being modified by the .bbappends by
> actually naming the layer it is (intending to**) modify.  The initiative
> also aims to highlight and separate .bbappends (modifications) from uniquely
> added components in our project (new .bb files) in GDP.
> 
> **Note that we are well aware this does NOT control bitbake behavior (as
> neither does the recipe-xxx/ directory convention), and is therefore just a
> guide.  If the project is misconfigured, bitbake could do something
> different than the naming convention implies and it could even be
> misleading, but at least the *programmer intent* is shown clearly.
> 
> I think it also makes .bbappends more explicitly visible and might trigger
> the idea:   Do we really need this?  Are we really modifying *that* adopted
> layer, and if so, why?
> 
> To Russel who wrote the previous request for guidance - feel free to
> consider, but I won't recommend this directly since it's by no means an
> agreed best practice in all of the community.  I'm rather asking for other
> experienced OE developers to consider and comment on the idea.
> 
> For oldtimers it's probably easy to find this odd and just be negative
> against it since well, "standard practice", inertia, NIH and all that, but
> I'll stick my neck out anyway because it seems to solve some issues of
> organization and understanding at least for us (and I'm a sucker for some
> flamebait ;)
> 
> If the intro is too formal and long, just look at the final examples.  I
> think you'll get it.  Feel free to ask.
> 
> 
> - Gunnar
> 
> -- 
> Gunnar Andersson <gandersson@genivi.org>
> Development Lead
> GENIVI Alliance
> 
> [1] https://github.com/GENIVI/genivi-dev-platform
> [2] Naming strategy for bitbake extension layers:			
> https://at.projects.genivi.org/wiki/x/w4Xk
> 
> 
> 

Re: Idea for bbappend / layer organization and naming convention

[Gunnar Andersson]

On Mon, 2017-08-14 at 23:02 -0400, Bruce Ashfield wrote:
> On 2017-08-14 3:36 PM, Gunnar Andersson wrote:
> > 
> > Yocto community,
> > 
> > Triggered by the previous email request to yocto@ about best practices
> > for layer organization I'm finally firing off this email for (hopefully)
> > some feedback on an idea we first kicked around, discussed for rough
> > consensus and then decided to implement in the Yocto based GDP project
> > [1].  You can see it as a trial, anything can be changed, but so far so
> > good...
> > 
> > We adopted a somewhat novel (but actually not really unique, see inside)
> > naming convention [2] for all modifications to components that belong to
> > other layers.
> > 
> 
> I've been using a similar naming/sorting mechanism in layers that
> I maintain for several years now. 

Great.  Is it similar, or /exactly/ like this?  I'm only asking because I
think it would be useful to have some written down rules and if there are
additional tweaks that will improve this proposal, I'd like to hear them.

I suspected it might actually be somewhat of a common practice.

> When you have a single layer that is carrying bbappends to many other
> layers, I find that it really helps keep everything separated and aids
> finding the original recipe.

Yes, that is the idea.

> 
> (that being said, recent work with layer index, etc, make it
> easier to locate the original recipe and any bbappends .. but that
> doesn't preclude keeping things organized in a layer).

... but I didn't get any more feedback than yours, so I'll just leave it
where it is for now.  Maybe this is not something the other Yocto community
cares about, or similar practices are actually done in practice, but not
written down.  Or maybe everyone is OK with the state of mixing .bb and
.bbappend files within the layers...

So far I think the initial experience on our side is positive.  It's
something that needs to be looked after a bit (we have a few mistakes to
fix).  But since some directories will only have .bb files, and others only
.bbappend files, it's therefore easy to script a warning system for anything
that is out of place.

Best Regards
- Gunnar

> Cheers,
> 
> Bruce
> 
[trimmed]

> > -- 
> > Gunnar Andersson <gandersson@genivi.org>
> > Development Lead
> > GENIVI Alliance
> > 
> > [1] https://github.com/GENIVI/genivi-dev-platform
> > [2] Naming strategy for bitbake extension layers:			
> > https://at.projects.genivi.org/wiki/x/w4Xk

Re: Idea for bbappend / layer organization and naming convention

[Bruce Ashfield]

On 08/18/2017 10:46 AM, Gunnar Andersson wrote:
> 
> On Mon, 2017-08-14 at 23:02 -0400, Bruce Ashfield wrote:
>> On 2017-08-14 3:36 PM, Gunnar Andersson wrote:
>>>
>>> Yocto community,
>>>
>>> Triggered by the previous email request to yocto@ about best practices
>>> for layer organization I'm finally firing off this email for (hopefully)
>>> some feedback on an idea we first kicked around, discussed for rough
>>> consensus and then decided to implement in the Yocto based GDP project
>>> [1].  You can see it as a trial, anything can be changed, but so far so
>>> good...
>>>
>>> We adopted a somewhat novel (but actually not really unique, see inside)
>>> naming convention [2] for all modifications to components that belong to
>>> other layers.
>>>
>>
>> I've been using a similar naming/sorting mechanism in layers that
>> I maintain for several years now.
> 
> Great.  Is it similar, or /exactly/ like this?  I'm only asking because I
> think it would be useful to have some written down rules and if there are
> additional tweaks that will improve this proposal, I'd like to hear them.
> 

Exactly. The layer name and recipe-* structure is nested in a layer
that is carrying bbappends. That way there's zero confusion about where
the main recipe can be found.

Bruce

> I suspected it might actually be somewhat of a common practice.
> 
>> When you have a single layer that is carrying bbappends to many other
>> layers, I find that it really helps keep everything separated and aids
>> finding the original recipe.
> 
> Yes, that is the idea.
> 
>>
>> (that being said, recent work with layer index, etc, make it
>> easier to locate the original recipe and any bbappends .. but that
>> doesn't preclude keeping things organized in a layer).
> 
> ... but I didn't get any more feedback than yours, so I'll just leave it
> where it is for now.  Maybe this is not something the other Yocto community
> cares about, or similar practices are actually done in practice, but not
> written down.  Or maybe everyone is OK with the state of mixing .bb and
> .bbappend files within the layers...
> 
> So far I think the initial experience on our side is positive.  It's
> something that needs to be looked after a bit (we have a few mistakes to
> fix).  But since some directories will only have .bb files, and others only
> .bbappend files, it's therefore easy to script a warning system for anything
> that is out of place.
> 
> Best Regards
> - Gunnar
> 
>> Cheers,
>>
>> Bruce
>>
> [trimmed]
> 
>>> -- 
>>> Gunnar Andersson <gandersson@genivi.org>
>>> Development Lead
>>> GENIVI Alliance
>>>
>>> [1] https://github.com/GENIVI/genivi-dev-platform
>>> [2] Naming strategy for bitbake extension layers:			
>>> https://at.projects.genivi.org/wiki/x/w4Xk
> 
> 

Re: Idea for bbappend / layer organization and naming convention

[Mark Hatle]

On 8/18/17 1:47 PM, Bruce Ashfield wrote:
> On 08/18/2017 10:46 AM, Gunnar Andersson wrote:
>>
>> On Mon, 2017-08-14 at 23:02 -0400, Bruce Ashfield wrote:
>>> On 2017-08-14 3:36 PM, Gunnar Andersson wrote:
>>>>
>>>> Yocto community,
>>>>
>>>> Triggered by the previous email request to yocto@ about best practices
>>>> for layer organization I'm finally firing off this email for (hopefully)
>>>> some feedback on an idea we first kicked around, discussed for rough
>>>> consensus and then decided to implement in the Yocto based GDP project
>>>> [1].  You can see it as a trial, anything can be changed, but so far so
>>>> good...
>>>>
>>>> We adopted a somewhat novel (but actually not really unique, see inside)
>>>> naming convention [2] for all modifications to components that belong to
>>>> other layers.
>>>>
>>>
>>> I've been using a similar naming/sorting mechanism in layers that
>>> I maintain for several years now.
>>
>> Great.  Is it similar, or /exactly/ like this?  I'm only asking because I
>> think it would be useful to have some written down rules and if there are
>> additional tweaks that will improve this proposal, I'd like to hear them.
>>

I suspect suggestions for best practices along with examples are what people
would like to see.  This may be reasonable to do with both the Yocto Project and
OpenEmbedded communities.

Obviously there are currently a large number of existing layers.  So showing how
this works, and examples of how it makes it easier to maintain and manage the
work over the longer term is a really good idea.

(Keep in mind, many of the current layers were originally designed/updated
before there was a large ecosystem of layers.  So a lot of poor examples, of
this kind of maintenance, may exist.  It is a good idea to steer people away
from using these as examples, and even a way to potentially help the maintainers
of those layers evolve into a better and more efficient format.)

--Mark

> Exactly. The layer name and recipe-* structure is nested in a layer
> that is carrying bbappends. That way there's zero confusion about where
> the main recipe can be found.
> 
> Bruce
> 
>> I suspected it might actually be somewhat of a common practice.
>>
>>> When you have a single layer that is carrying bbappends to many other
>>> layers, I find that it really helps keep everything separated and aids
>>> finding the original recipe.
>>
>> Yes, that is the idea.
>>
>>>
>>> (that being said, recent work with layer index, etc, make it
>>> easier to locate the original recipe and any bbappends .. but that
>>> doesn't preclude keeping things organized in a layer).
>>
>> ... but I didn't get any more feedback than yours, so I'll just leave it
>> where it is for now.  Maybe this is not something the other Yocto community
>> cares about, or similar practices are actually done in practice, but not
>> written down.  Or maybe everyone is OK with the state of mixing .bb and
>> .bbappend files within the layers...
>>
>> So far I think the initial experience on our side is positive.  It's
>> something that needs to be looked after a bit (we have a few mistakes to
>> fix).  But since some directories will only have .bb files, and others only
>> .bbappend files, it's therefore easy to script a warning system for anything
>> that is out of place.
>>
>> Best Regards
>> - Gunnar
>>
>>> Cheers,
>>>
>>> Bruce
>>>
>> [trimmed]
>>
>>>> -- 
>>>> Gunnar Andersson <gandersson@genivi.org>
>>>> Development Lead
>>>> GENIVI Alliance
>>>>
>>>> [1] https://github.com/GENIVI/genivi-dev-platform
>>>> [2] Naming strategy for bitbake extension layers:			
>>>> https://at.projects.genivi.org/wiki/x/w4Xk
>>
>>
> 

Re: Idea for bbappend / layer organization and naming convention

[Stephane Desneux]

Hi all,

I've followed this thread with interest. I'll try to bring here some ideas we
had in parallel on Automotive Grade Linux (AGL) side.

What we plan to do with backports on AGL is still WIP (see
https://jira.automotivelinux.org/browse/SPEC-802).

It's still cooking, but we're going to adopt a solution close to the one
proposed by Gunnar in GENIVI: I mean: we agree on the same goal, but what we end
up with differs slightly.

The scheme we're thinking about would be typically something like this:

meta-agl/meta-agl-backports/<meta-origin>/<recipes-origin>/<recipe>/<feature|reason>/<recipe>.bbappend

Where:
* the first meta-agl is our main git repo (not a meta in fact): the real meta is
meta-agl/meta-agl :)
* meta-agl/meta-agl-backports is a layer
* <meta-origin> is the name of the layer on which the backport is applied
* <recipes-origin> is the origin folder name for recipes in the origin layer
* <recipe> is the name of the recipe which is backported/amended
* <feature|reason> is something that defines why we have a backport here (bug
id, quick desc...)

And finally, in the folder where we put the bbappend, we want to add a
README.backport in YAML, similar to this one:

—
Origin: <name>
Origin-Url: <source repo url>
Origin-Ref: <rev/tag>
Destination: <AGL version name: dab, eel,...>
Expiration: <when> (if known)
Bug-AGL: <reference to bug ID>
Upstream-Status: <the same as OE's Upstream-Status tag>
Reason: <comment
   on
   multiple
   lines>
—

With that model, we expect to smooth the workflow and make the alignment on
Yocto/OE/Poky versions easier.

HTH

Best regards,
---
Stephane Desneux - CTO - IoT.bzh
stephane.desneux@iot.bzh - www.iot.bzh

On 18/08/17 21:35, Mark Hatle wrote:
> On 8/18/17 1:47 PM, Bruce Ashfield wrote:
>> On 08/18/2017 10:46 AM, Gunnar Andersson wrote:
>>>
>>> On Mon, 2017-08-14 at 23:02 -0400, Bruce Ashfield wrote:
>>>> On 2017-08-14 3:36 PM, Gunnar Andersson wrote:
>>>>>
>>>>> Yocto community,
>>>>>
>>>>> Triggered by the previous email request to yocto@ about best practices
>>>>> for layer organization I'm finally firing off this email for (hopefully)
>>>>> some feedback on an idea we first kicked around, discussed for rough
>>>>> consensus and then decided to implement in the Yocto based GDP project
>>>>> [1].  You can see it as a trial, anything can be changed, but so far so
>>>>> good...
>>>>>
>>>>> We adopted a somewhat novel (but actually not really unique, see inside)
>>>>> naming convention [2] for all modifications to components that belong to
>>>>> other layers.
>>>>>
>>>>
>>>> I've been using a similar naming/sorting mechanism in layers that
>>>> I maintain for several years now.
>>>
>>> Great.  Is it similar, or /exactly/ like this?  I'm only asking because I
>>> think it would be useful to have some written down rules and if there are
>>> additional tweaks that will improve this proposal, I'd like to hear them.
>>>
> 
> I suspect suggestions for best practices along with examples are what people
> would like to see.  This may be reasonable to do with both the Yocto Project and
> OpenEmbedded communities.
> 
> Obviously there are currently a large number of existing layers.  So showing how
> this works, and examples of how it makes it easier to maintain and manage the
> work over the longer term is a really good idea.
> 
> (Keep in mind, many of the current layers were originally designed/updated
> before there was a large ecosystem of layers.  So a lot of poor examples, of
> this kind of maintenance, may exist.  It is a good idea to steer people away
> from using these as examples, and even a way to potentially help the maintainers
> of those layers evolve into a better and more efficient format.)
> 
> --Mark
> 
>> Exactly. The layer name and recipe-* structure is nested in a layer
>> that is carrying bbappends. That way there's zero confusion about where
>> the main recipe can be found.
>>
>> Bruce
>>
>>> I suspected it might actually be somewhat of a common practice.
>>>
>>>> When you have a single layer that is carrying bbappends to many other
>>>> layers, I find that it really helps keep everything separated and aids
>>>> finding the original recipe.
>>>
>>> Yes, that is the idea.
>>>
>>>>
>>>> (that being said, recent work with layer index, etc, make it
>>>> easier to locate the original recipe and any bbappends .. but that
>>>> doesn't preclude keeping things organized in a layer).
>>>
>>> ... but I didn't get any more feedback than yours, so I'll just leave it
>>> where it is for now.  Maybe this is not something the other Yocto community
>>> cares about, or similar practices are actually done in practice, but not
>>> written down.  Or maybe everyone is OK with the state of mixing .bb and
>>> .bbappend files within the layers...
>>>
>>> So far I think the initial experience on our side is positive.  It's
>>> something that needs to be looked after a bit (we have a few mistakes to
>>> fix).  But since some directories will only have .bb files, and others only
>>> .bbappend files, it's therefore easy to script a warning system for anything
>>> that is out of place.
>>>
>>> Best Regards
>>> - Gunnar
>>>
>>>> Cheers,
>>>>
>>>> Bruce
>>>>
>>> [trimmed]
>>>
>>>>> -- 
>>>>> Gunnar Andersson <gandersson@genivi.org>
>>>>> Development Lead
>>>>> GENIVI Alliance
>>>>>
>>>>> [1] https://github.com/GENIVI/genivi-dev-platform
>>>>> [2] Naming strategy for bitbake extension layers:			
>>>>> https://at.projects.genivi.org/wiki/x/w4Xk
>>>
>>>
>>
>