Splitting meta-oe?

Splitting meta-oe?

[Burton, Ross]

Hi,

Is now a good time to talk about splitting up meta-oe?  Some layers are
actively developed and maintained (one example: meta-python), others are
basically bitrotting and only get touched when something else causes them
to break world builds (one example: meta-gnome).  I've long felt that
meta-oe should be split up and the high quality layers managed in their own
repositories so patches to them don't get held up by breakage in other
sub-layers.

Another advantage of splitting out the high quality layers is that we'd
like to look at running more community layers through the Yocto
autobuilder, and granular layers make that easier to manage.

Comments?

Ross

Re: Splitting meta-oe?

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 2936 bytes --]

[[oe] Splitting meta-oe?] On 18.02.20 (Tue 10:45) Burton, Ross wrote:

> Hi,
> 
> Is now a good time to talk about splitting up meta-oe?  Some layers are
> actively developed and maintained (one example: meta-python), others are
> basically bitrotting and only get touched when something else causes them
> to break world builds (one example: meta-gnome).  I've long felt that
> meta-oe should be split up and the high quality layers managed in their own
> repositories so patches to them don't get held up by breakage in other
> sub-layers.

Way back when I first proposed creating meta-networking I pitched it as
a separate layer managed outside the meta-oe umbrella.  At the time I
thought it made sense for my use case and for the use case I anticipated
from consumers of meta-networking packages both for the reasons you
state below and because my intent was to create a layer primarily useful
for creating network-centric devices (bridges, firewalls, switches, APs,
etc.) with an absolute minimum of other layer dependencies.  I think I
actually proposed the highly ambitious goal of oe-core + meta-networking
as a functional set.

Anyway, in the intervening time there's been enough situations crop up
that now the minimal set is something like oe-core + meta-oe +
meta-perl(we're still open on that, I think) + meta-python +
meta-networking.

So my original goal of keeping meta-networking largely independent of
other layers isn't practical (wasn't from get-go, really) and I'm okay
with that.  I also think there's considerable value in being able to say
"we don't need that recipe in this layer, it's already maintained by
someone else in a common area in the same repo, just a different layer".
I wouldn't want to start down a path that leads to multiple incompatible
versions of recipes being maintained in layers purely out of convenience
to the layer maintainer.

I can see a lot of value in removing old recipes / layers that aren't
being used and / or have fallen far out of date, though.  If you've got
a list of those you'd like to discuss, I'm sure that's a useful
discussion to have.

I'll also take the opportunity to plug my old patch for flagging recipes
as deprecated (https://patchwork.openembedded.org/series/5489/).  It's
never gotten any traction, but I think it's a good early-warning for
consumers of recipes that are quietly using them with reasonable success
as-is despite any issues the maintainers may have with them.  Could be a
good way to determine who actually cares about these problem child
recipes before they become a big enough headache that we just need to
abandon them.

-J.

> Another advantage of splitting out the high quality layers is that we'd
> like to look at running more community layers through the Yocto
> autobuilder, and granular layers make that easier to manage.
> 
> Comments?
> 
> Ross
-- 
-Joe MacDonald.
:wq

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: Splitting meta-oe?

[akuster808]

On 02/20/2018 02:45 AM, Burton, Ross wrote:
> Hi,
>
> Is now a good time to talk about splitting up meta-oe?  Some layers are
> actively developed and maintained (one example: meta-python), others are
> basically bitrotting and only get touched when something else causes them
> to break world builds (one example: meta-gnome).  I've long felt that
> meta-oe should be split up and the high quality layers managed in their own
> repositories so patches to them don't get held up by breakage in other
> sub-layers.
You make it sound like meta-oe is not a high quality layer.  I could
make the same claim about oe-core master.

I don't see the connection in patches being held up due to breakage in
other sub layers. This only happens if the dependency fail to build.

You lose control over the quality in current layers that reside in
meta-openbedded just like you have no control over all the other layers
residing in the community. It makes maintaining stable versions very
difficult. Well, unless The Yocto Project takes over them.. I guess that
would work then.

>
> Another advantage of splitting out the high quality layers is that we'd
> like to look at running more community layers through the Yocto
> autobuilder, and granular layers make that easier to manage.
 I thought not including layers in bblayers.conf was easy enough. 
>
> Comments?

What problem do you thing you are trying to solve here?

- armin
>
> Ross

Re: Splitting meta-oe?

[Bruce Ashfield]

On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com> wrote:
>
>
> On 02/20/2018 02:45 AM, Burton, Ross wrote:
>> Hi,
>>
>> Is now a good time to talk about splitting up meta-oe?  Some layers are
>> actively developed and maintained (one example: meta-python), others are
>> basically bitrotting and only get touched when something else causes them
>> to break world builds (one example: meta-gnome).  I've long felt that
>> meta-oe should be split up and the high quality layers managed in their own
>> repositories so patches to them don't get held up by breakage in other
>> sub-layers.
> You make it sound like meta-oe is not a high quality layer.  I could
> make the same claim about oe-core master.
>
> I don't see the connection in patches being held up due to breakage in
> other sub layers. This only happens if the dependency fail to build.
>
> You lose control over the quality in current layers that reside in
> meta-openbedded just like you have no control over all the other layers
> residing in the community. It makes maintaining stable versions very
> difficult. Well, unless The Yocto Project takes over them.. I guess that
> would work then.
>
>>
>> Another advantage of splitting out the high quality layers is that we'd
>> like to look at running more community layers through the Yocto
>> autobuilder, and granular layers make that easier to manage.
>  I thought not including layers in bblayers.conf was easy enough.
>>
>> Comments?
>
> What problem do you thing you are trying to solve here?

My unrelated issues are that I can't update one layer, without getting
all of the updates.
.. but that is both a good thing (i.e. they are all tested together,
so you know that the
single SRCREV update is good for all layers), and a bad thing (when
you just want a
new python recipe update from meta-python, but don't want other changes).

It is very likely that splitting the layer would help one issue, but
make the other worse.

So no solution from me, just an observation about potential issue.

Cheers,

Bruce

>
> - armin
>>
>> Ross
>
>
> --
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Splitting meta-oe?

[Khem Raj]

On 2/20/18 9:13 AM, Bruce Ashfield wrote:
> On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com> wrote:
>>
>>
>> On 02/20/2018 02:45 AM, Burton, Ross wrote:
>>> Hi,
>>>
>>> Is now a good time to talk about splitting up meta-oe?  Some layers are
>>> actively developed and maintained (one example: meta-python), others are
>>> basically bitrotting and only get touched when something else causes them
>>> to break world builds (one example: meta-gnome).  I've long felt that
>>> meta-oe should be split up and the high quality layers managed in their own
>>> repositories so patches to them don't get held up by breakage in other
>>> sub-layers.
>> You make it sound like meta-oe is not a high quality layer.  I could
>> make the same claim about oe-core master.
>>
>> I don't see the connection in patches being held up due to breakage in
>> other sub layers. This only happens if the dependency fail to build.
>>
>> You lose control over the quality in current layers that reside in
>> meta-openbedded just like you have no control over all the other layers
>> residing in the community. It makes maintaining stable versions very
>> difficult. Well, unless The Yocto Project takes over them.. I guess that
>> would work then.
>>
>>>
>>> Another advantage of splitting out the high quality layers is that we'd
>>> like to look at running more community layers through the Yocto
>>> autobuilder, and granular layers make that easier to manage.
>>  I thought not including layers in bblayers.conf was easy enough.
>>>
>>> Comments?
>>
>> What problem do you thing you are trying to solve here?
> 
> My unrelated issues are that I can't update one layer, without getting
> all of the updates.
> .. but that is both a good thing (i.e. they are all tested together,
> so you know that the
> single SRCREV update is good for all layers), and a bad thing (when
> you just want a
> new python recipe update from meta-python, but don't want other changes).
> 

if you dont include the layers in your BBLAYERS and they become
effectively non existent, unless you are on metered internet connection,
where downloading unused stuff would save you bandwidth, it should be
ok.  No ?

> It is very likely that splitting the layer would help one issue, but
> make the other worse.
> 
> So no solution from me, just an observation about potential issue.
> 
> Cheers,
> 
> Bruce
> 
>>
>> - armin
>>>
>>> Ross
>>
>>
>> --
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> 
> 
> 

Re: Splitting meta-oe?

[Bruce Ashfield]

On Tue, Feb 20, 2018 at 1:52 PM, Khem Raj <raj.khem@gmail.com> wrote:
> On 2/20/18 9:13 AM, Bruce Ashfield wrote:
>> On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com> wrote:
>>>
>>>
>>> On 02/20/2018 02:45 AM, Burton, Ross wrote:
>>>> Hi,
>>>>
>>>> Is now a good time to talk about splitting up meta-oe?  Some layers are
>>>> actively developed and maintained (one example: meta-python), others are
>>>> basically bitrotting and only get touched when something else causes them
>>>> to break world builds (one example: meta-gnome).  I've long felt that
>>>> meta-oe should be split up and the high quality layers managed in their own
>>>> repositories so patches to them don't get held up by breakage in other
>>>> sub-layers.
>>> You make it sound like meta-oe is not a high quality layer.  I could
>>> make the same claim about oe-core master.
>>>
>>> I don't see the connection in patches being held up due to breakage in
>>> other sub layers. This only happens if the dependency fail to build.
>>>
>>> You lose control over the quality in current layers that reside in
>>> meta-openbedded just like you have no control over all the other layers
>>> residing in the community. It makes maintaining stable versions very
>>> difficult. Well, unless The Yocto Project takes over them.. I guess that
>>> would work then.
>>>
>>>>
>>>> Another advantage of splitting out the high quality layers is that we'd
>>>> like to look at running more community layers through the Yocto
>>>> autobuilder, and granular layers make that easier to manage.
>>>  I thought not including layers in bblayers.conf was easy enough.
>>>>
>>>> Comments?
>>>
>>> What problem do you thing you are trying to solve here?
>>
>> My unrelated issues are that I can't update one layer, without getting
>> all of the updates.
>> .. but that is both a good thing (i.e. they are all tested together,
>> so you know that the
>> single SRCREV update is good for all layers), and a bad thing (when
>> you just want a
>> new python recipe update from meta-python, but don't want other changes).
>>
>
> if you dont include the layers in your BBLAYERS and they become
> effectively non existent, unless you are on metered internet connection,
> where downloading unused stuff would save you bandwidth, it should be
> ok.  No ?

Its not that.

I *am* building the different layers, but say I have a stable set of packages
and working images .. but for whatever reason, I need an updated python-<foo>
package for an unrelated package, or some other layer that needs a newer
version, etc.

How do I get that, without taking updates to all the layers ? .. and layers that
I really didn't want to update. I have to do some sort of combo-layer, carry
my own copy of the recipe, etc.

So there are definitely ways to do it, I'm just pointing out that I
end up taking
some build failures/issues from time to time on packages I didn't really
need to update.

The flip side of that argument is that all of the layers and sub layers have
gone through some sort of global build, and hence I know that they all have
worked together for someone. If I can update pieces individually, I break
that .. and I own the broken bits after that .. which again, goes to
my point that
fixing one workflow/issue can break another :D

Bruce

>
>> It is very likely that splitting the layer would help one issue, but
>> make the other worse.
>>
>> So no solution from me, just an observation about potential issue.
>>
>> Cheers,
>>
>> Bruce
>>
>>>
>>> - armin
>>>>
>>>> Ross
>>>
>>>
>>> --
>>> _______________________________________________
>>> Openembedded-devel mailing list
>>> Openembedded-devel@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>>
>>
>>
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Splitting meta-oe?

[Martin Jansa]

> I need an updated python-<foo> package for an unrelated package

And how far will you go?

If you want just newer python-<foo> and nothing else, will you take other
changes to other python-* recipes from meta-python layer? There is a lot of
recipes there, if you're so picky about updates, then you shouldn't update
whole oe-core as well.

On Wed, Feb 21, 2018 at 1:51 AM, Bruce Ashfield <bruce.ashfield@gmail.com>
wrote:

> On Tue, Feb 20, 2018 at 1:52 PM, Khem Raj <raj.khem@gmail.com> wrote:
> > On 2/20/18 9:13 AM, Bruce Ashfield wrote:
> >> On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com>
> wrote:
> >>>
> >>>
> >>> On 02/20/2018 02:45 AM, Burton, Ross wrote:
> >>>> Hi,
> >>>>
> >>>> Is now a good time to talk about splitting up meta-oe?  Some layers
> are
> >>>> actively developed and maintained (one example: meta-python), others
> are
> >>>> basically bitrotting and only get touched when something else causes
> them
> >>>> to break world builds (one example: meta-gnome).  I've long felt that
> >>>> meta-oe should be split up and the high quality layers managed in
> their own
> >>>> repositories so patches to them don't get held up by breakage in other
> >>>> sub-layers.
> >>> You make it sound like meta-oe is not a high quality layer.  I could
> >>> make the same claim about oe-core master.
> >>>
> >>> I don't see the connection in patches being held up due to breakage in
> >>> other sub layers. This only happens if the dependency fail to build.
> >>>
> >>> You lose control over the quality in current layers that reside in
> >>> meta-openbedded just like you have no control over all the other layers
> >>> residing in the community. It makes maintaining stable versions very
> >>> difficult. Well, unless The Yocto Project takes over them.. I guess
> that
> >>> would work then.
> >>>
> >>>>
> >>>> Another advantage of splitting out the high quality layers is that
> we'd
> >>>> like to look at running more community layers through the Yocto
> >>>> autobuilder, and granular layers make that easier to manage.
> >>>  I thought not including layers in bblayers.conf was easy enough.
> >>>>
> >>>> Comments?
> >>>
> >>> What problem do you thing you are trying to solve here?
> >>
> >> My unrelated issues are that I can't update one layer, without getting
> >> all of the updates.
> >> .. but that is both a good thing (i.e. they are all tested together,
> >> so you know that the
> >> single SRCREV update is good for all layers), and a bad thing (when
> >> you just want a
> >> new python recipe update from meta-python, but don't want other
> changes).
> >>
> >
> > if you dont include the layers in your BBLAYERS and they become
> > effectively non existent, unless you are on metered internet connection,
> > where downloading unused stuff would save you bandwidth, it should be
> > ok.  No ?
>
> Its not that.
>
> I *am* building the different layers, but say I have a stable set of
> packages
> and working images .. but for whatever reason, I need an updated
> python-<foo>
> package for an unrelated package, or some other layer that needs a newer
> version, etc.
>
> How do I get that, without taking updates to all the layers ? .. and
> layers that
> I really didn't want to update. I have to do some sort of combo-layer,
> carry
> my own copy of the recipe, etc.
>
> So there are definitely ways to do it, I'm just pointing out that I
> end up taking
> some build failures/issues from time to time on packages I didn't really
> need to update.
>
> The flip side of that argument is that all of the layers and sub layers
> have
> gone through some sort of global build, and hence I know that they all have
> worked together for someone. If I can update pieces individually, I break
> that .. and I own the broken bits after that .. which again, goes to
> my point that
> fixing one workflow/issue can break another :D
>
> Bruce
>
> >
> >> It is very likely that splitting the layer would help one issue, but
> >> make the other worse.
> >>
> >> So no solution from me, just an observation about potential issue.
> >>
> >> Cheers,
> >>
> >> Bruce
> >>
> >>>
> >>> - armin
> >>>>
> >>>> Ross
> >>>
> >>>
> >>> --
> >>> _______________________________________________
> >>> Openembedded-devel mailing list
> >>> Openembedded-devel@lists.openembedded.org
> >>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> >>
> >>
> >>
> >
>
>
>
> --
> "Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end"
> --
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>

Re: Splitting meta-oe?

[Martin Jansa]

> I'm actually very worried about these (re)tired maintainers. If the
layers were more independent it would allow some of the patch handling
responsibilities and testing responsibilities to move to other people,
reducing the load on those maintainers.

Armin can update his own view, but for me the main issue wasn't the "patch
handling" part, but actually the lack of patches for the new issues (often
caused by new changes in oe-core).

Yes, the "patch handling" part was the most boring piece, but patches from
regular contributors or co-maintainers were always easy to handle and
rarely caused any extra work.

The untested drive-by contributions from people who don't even reply to
review comments are completely different category, but separate git
repositories won't drive those away (until it gets so confusing that these
people will even fail to find correct repository and just gave up trying to
contribute anything at all). Nothing is blocking more people to collect,
fix, test these drive-by contributions and then send consolidated
pull-request which will be easily merged to shared repository - but I never
seen this happen and I don't know how separate git repositories will help
with this.

In the end after spending a lot of own time doing this boring part, I
always felt responsible for fixing as many build failures detected in world
build as I could, but that's never-ending uphill battle where very few
people ever help (big thanks to those exceptions who sometimes help). And
what you get for that extra afford to get it building as much as possible
in spare time? Comments about how garbage meta-oe layers are.

We need patches not more git repos!

On Wed, Feb 21, 2018 at 9:49 AM, Martin Jansa <martin.jansa@gmail.com>
wrote:

> > I need an updated python-<foo> package for an unrelated package
>
> And how far will you go?
>
> If you want just newer python-<foo> and nothing else, will you take other
> changes to other python-* recipes from meta-python layer? There is a lot of
> recipes there, if you're so picky about updates, then you shouldn't update
> whole oe-core as well.
>
> On Wed, Feb 21, 2018 at 1:51 AM, Bruce Ashfield <bruce.ashfield@gmail.com>
> wrote:
>
>> On Tue, Feb 20, 2018 at 1:52 PM, Khem Raj <raj.khem@gmail.com> wrote:
>> > On 2/20/18 9:13 AM, Bruce Ashfield wrote:
>> >> On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com>
>> wrote:
>> >>>
>> >>>
>> >>> On 02/20/2018 02:45 AM, Burton, Ross wrote:
>> >>>> Hi,
>> >>>>
>> >>>> Is now a good time to talk about splitting up meta-oe?  Some layers
>> are
>> >>>> actively developed and maintained (one example: meta-python), others
>> are
>> >>>> basically bitrotting and only get touched when something else causes
>> them
>> >>>> to break world builds (one example: meta-gnome).  I've long felt that
>> >>>> meta-oe should be split up and the high quality layers managed in
>> their own
>> >>>> repositories so patches to them don't get held up by breakage in
>> other
>> >>>> sub-layers.
>> >>> You make it sound like meta-oe is not a high quality layer.  I could
>> >>> make the same claim about oe-core master.
>> >>>
>> >>> I don't see the connection in patches being held up due to breakage in
>> >>> other sub layers. This only happens if the dependency fail to build.
>> >>>
>> >>> You lose control over the quality in current layers that reside in
>> >>> meta-openbedded just like you have no control over all the other
>> layers
>> >>> residing in the community. It makes maintaining stable versions very
>> >>> difficult. Well, unless The Yocto Project takes over them.. I guess
>> that
>> >>> would work then.
>> >>>
>> >>>>
>> >>>> Another advantage of splitting out the high quality layers is that
>> we'd
>> >>>> like to look at running more community layers through the Yocto
>> >>>> autobuilder, and granular layers make that easier to manage.
>> >>>  I thought not including layers in bblayers.conf was easy enough.
>> >>>>
>> >>>> Comments?
>> >>>
>> >>> What problem do you thing you are trying to solve here?
>> >>
>> >> My unrelated issues are that I can't update one layer, without getting
>> >> all of the updates.
>> >> .. but that is both a good thing (i.e. they are all tested together,
>> >> so you know that the
>> >> single SRCREV update is good for all layers), and a bad thing (when
>> >> you just want a
>> >> new python recipe update from meta-python, but don't want other
>> changes).
>> >>
>> >
>> > if you dont include the layers in your BBLAYERS and they become
>> > effectively non existent, unless you are on metered internet connection,
>> > where downloading unused stuff would save you bandwidth, it should be
>> > ok.  No ?
>>
>> Its not that.
>>
>> I *am* building the different layers, but say I have a stable set of
>> packages
>> and working images .. but for whatever reason, I need an updated
>> python-<foo>
>> package for an unrelated package, or some other layer that needs a newer
>> version, etc.
>>
>> How do I get that, without taking updates to all the layers ? .. and
>> layers that
>> I really didn't want to update. I have to do some sort of combo-layer,
>> carry
>> my own copy of the recipe, etc.
>>
>> So there are definitely ways to do it, I'm just pointing out that I
>> end up taking
>> some build failures/issues from time to time on packages I didn't really
>> need to update.
>>
>> The flip side of that argument is that all of the layers and sub layers
>> have
>> gone through some sort of global build, and hence I know that they all
>> have
>> worked together for someone. If I can update pieces individually, I break
>> that .. and I own the broken bits after that .. which again, goes to
>> my point that
>> fixing one workflow/issue can break another :D
>>
>> Bruce
>>
>> >
>> >> It is very likely that splitting the layer would help one issue, but
>> >> make the other worse.
>> >>
>> >> So no solution from me, just an observation about potential issue.
>> >>
>> >> Cheers,
>> >>
>> >> Bruce
>> >>
>> >>>
>> >>> - armin
>> >>>>
>> >>>> Ross
>> >>>
>> >>>
>> >>> --
>> >>> _______________________________________________
>> >>> Openembedded-devel mailing list
>> >>> Openembedded-devel@lists.openembedded.org
>> >>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>> >>
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> "Thou shalt not follow the NULL pointer, for chaos and madness await
>> thee at its end"
>> --
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>>
>
>

Re: Splitting meta-oe?

[Andrea Adami]

On Wed, Feb 21, 2018 at 10:06 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
>> I'm actually very worried about these (re)tired maintainers. If the
> layers were more independent it would allow some of the patch handling
> responsibilities and testing responsibilities to move to other people,
> reducing the load on those maintainers.
>
> Armin can update his own view, but for me the main issue wasn't the "patch
> handling" part, but actually the lack of patches for the new issues (often
> caused by new changes in oe-core).
>

That's one point.
If I may add my experience as maintainer of two minor layers
(meta-handheld and meta-initramfs), I have to say there have been many
cases where an upgrade to oe-core did break one of these layers.
Just a variable rename for example...this fall-outs could have been
avoided if the oe-core dev would have spent a minute grepping the
repository,
Heh..now we have to define what is the public repository, which layers
have to be checked...

And about layer interdependencies...well..it is very delicate.
For example meta-handheld, a BSP layer, must depend since some time on
meta-oe. Why? Because tslib was removed from oe-core.
This is ugly...here a more separated layering structure would help

So I somehow feel we should refine what is in the repository but
absolutely not split it around.
Even, I think we should host all public layers (definition lacks
today) in the OpenEmbedded Git Repository .

My 2 cents
Andrea

> Yes, the "patch handling" part was the most boring piece, but patches from
> regular contributors or co-maintainers were always easy to handle and
> rarely caused any extra work.
>
> The untested drive-by contributions from people who don't even reply to
> review comments are completely different category, but separate git
> repositories won't drive those away (until it gets so confusing that these
> people will even fail to find correct repository and just gave up trying to
> contribute anything at all). Nothing is blocking more people to collect,
> fix, test these drive-by contributions and then send consolidated
> pull-request which will be easily merged to shared repository - but I never
> seen this happen and I don't know how separate git repositories will help
> with this.
>
> In the end after spending a lot of own time doing this boring part, I
> always felt responsible for fixing as many build failures detected in world
> build as I could, but that's never-ending uphill battle where very few
> people ever help (big thanks to those exceptions who sometimes help). And
> what you get for that extra afford to get it building as much as possible
> in spare time? Comments about how garbage meta-oe layers are.
>
> We need patches not more git repos!
>
> On Wed, Feb 21, 2018 at 9:49 AM, Martin Jansa <martin.jansa@gmail.com>
> wrote:
>
>> > I need an updated python-<foo> package for an unrelated package
>>
>> And how far will you go?
>>
>> If you want just newer python-<foo> and nothing else, will you take other
>> changes to other python-* recipes from meta-python layer? There is a lot of
>> recipes there, if you're so picky about updates, then you shouldn't update
>> whole oe-core as well.
>>
>> On Wed, Feb 21, 2018 at 1:51 AM, Bruce Ashfield <bruce.ashfield@gmail.com>
>> wrote:
>>
>>> On Tue, Feb 20, 2018 at 1:52 PM, Khem Raj <raj.khem@gmail.com> wrote:
>>> > On 2/20/18 9:13 AM, Bruce Ashfield wrote:
>>> >> On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com>
>>> wrote:
>>> >>>
>>> >>>
>>> >>> On 02/20/2018 02:45 AM, Burton, Ross wrote:
>>> >>>> Hi,
>>> >>>>
>>> >>>> Is now a good time to talk about splitting up meta-oe?  Some layers
>>> are
>>> >>>> actively developed and maintained (one example: meta-python), others
>>> are
>>> >>>> basically bitrotting and only get touched when something else causes
>>> them
>>> >>>> to break world builds (one example: meta-gnome).  I've long felt that
>>> >>>> meta-oe should be split up and the high quality layers managed in
>>> their own
>>> >>>> repositories so patches to them don't get held up by breakage in
>>> other
>>> >>>> sub-layers.
>>> >>> You make it sound like meta-oe is not a high quality layer.  I could
>>> >>> make the same claim about oe-core master.
>>> >>>
>>> >>> I don't see the connection in patches being held up due to breakage in
>>> >>> other sub layers. This only happens if the dependency fail to build.
>>> >>>
>>> >>> You lose control over the quality in current layers that reside in
>>> >>> meta-openbedded just like you have no control over all the other
>>> layers
>>> >>> residing in the community. It makes maintaining stable versions very
>>> >>> difficult. Well, unless The Yocto Project takes over them.. I guess
>>> that
>>> >>> would work then.
>>> >>>
>>> >>>>
>>> >>>> Another advantage of splitting out the high quality layers is that
>>> we'd
>>> >>>> like to look at running more community layers through the Yocto
>>> >>>> autobuilder, and granular layers make that easier to manage.
>>> >>>  I thought not including layers in bblayers.conf was easy enough.
>>> >>>>
>>> >>>> Comments?
>>> >>>
>>> >>> What problem do you thing you are trying to solve here?
>>> >>
>>> >> My unrelated issues are that I can't update one layer, without getting
>>> >> all of the updates.
>>> >> .. but that is both a good thing (i.e. they are all tested together,
>>> >> so you know that the
>>> >> single SRCREV update is good for all layers), and a bad thing (when
>>> >> you just want a
>>> >> new python recipe update from meta-python, but don't want other
>>> changes).
>>> >>
>>> >
>>> > if you dont include the layers in your BBLAYERS and they become
>>> > effectively non existent, unless you are on metered internet connection,
>>> > where downloading unused stuff would save you bandwidth, it should be
>>> > ok.  No ?
>>>
>>> Its not that.
>>>
>>> I *am* building the different layers, but say I have a stable set of
>>> packages
>>> and working images .. but for whatever reason, I need an updated
>>> python-<foo>
>>> package for an unrelated package, or some other layer that needs a newer
>>> version, etc.
>>>
>>> How do I get that, without taking updates to all the layers ? .. and
>>> layers that
>>> I really didn't want to update. I have to do some sort of combo-layer,
>>> carry
>>> my own copy of the recipe, etc.
>>>
>>> So there are definitely ways to do it, I'm just pointing out that I
>>> end up taking
>>> some build failures/issues from time to time on packages I didn't really
>>> need to update.
>>>
>>> The flip side of that argument is that all of the layers and sub layers
>>> have
>>> gone through some sort of global build, and hence I know that they all
>>> have
>>> worked together for someone. If I can update pieces individually, I break
>>> that .. and I own the broken bits after that .. which again, goes to
>>> my point that
>>> fixing one workflow/issue can break another :D
>>>
>>> Bruce
>>>
>>> >
>>> >> It is very likely that splitting the layer would help one issue, but
>>> >> make the other worse.
>>> >>
>>> >> So no solution from me, just an observation about potential issue.
>>> >>
>>> >> Cheers,
>>> >>
>>> >> Bruce
>>> >>
>>> >>>
>>> >>> - armin
>>> >>>>
>>> >>>> Ross
>>> >>>
>>> >>>
>>> >>> --
>>> >>> _______________________________________________
>>> >>> Openembedded-devel mailing list
>>> >>> Openembedded-devel@lists.openembedded.org
>>> >>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>>> >>
>>> >>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> "Thou shalt not follow the NULL pointer, for chaos and madness await
>>> thee at its end"
>>> --
>>> _______________________________________________
>>> Openembedded-devel mailing list
>>> Openembedded-devel@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>>>
>>
>>
> --
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Re: Splitting meta-oe?

[Martin Jansa]

There is good example of inter-layer dependencies from real world:
http://lists.openembedded.org/pipermail/openembedded-devel/2017-February/111447.html

Do you want
A) new git repository meta-libio-socket-ssl-perl so that meta-networking
will depend on this on instead of whole meta-perl
B) meta-ddclient which will probably depend on both meta-perl and
meta-networking
C) ddclient and its dependencies in meta-perl
D) libio-socket-ssl-perl moved to oe-core, so that next time we can say
that oe-core is just like old oe-classic just with a bit less stuff in it

Neither of these options is ideal, but meta-networking getting meta-perl
dependency is the one which causes fewest issues to OE users. As mentioned
before by few people, most builds for "real world products" are already
including many layers, adding meta-perl to BBLAYERS (it's probably there
already for other recipes) without the need to update other build setup
like checking additional git repositories is simple and I don't see how it
makes meta-oe to be like oe-classic. The layers still separate stuff and I
don't need to include layers from where I don't need anything at all.

Smallest set of upstream layers we use at LGE already contains half of
meta-oe repository:
oe-core meta-gplv2 meta-oe meta-multimedia meta-networking meta-python
meta-filesystems meta-qt5
and around 8 internal layers, so it's far from oe-classic days and
different builds for different products have different subset of layers.

For internal layers we also use only a few git repositories 12 layers in
the main one, because it makes it easier to keep them all compatible with
each other at all times (e.g. by triggering the CI for all products from
each gerrit review for this single repository).

I'm not advocating for putting _everything_ into single git repository or
even the use of combo-layer tool, but keeping the layers maintained by the
same "organization" with the same update cycle makes things easier to
maintain and easier to consume by others.

Regards,

On Wed, Feb 21, 2018 at 10:48 AM, Andrea Adami <andrea.adami@gmail.com>
wrote:

> On Wed, Feb 21, 2018 at 10:06 AM, Martin Jansa <martin.jansa@gmail.com>
> wrote:
> >> I'm actually very worried about these (re)tired maintainers. If the
> > layers were more independent it would allow some of the patch handling
> > responsibilities and testing responsibilities to move to other people,
> > reducing the load on those maintainers.
> >
> > Armin can update his own view, but for me the main issue wasn't the
> "patch
> > handling" part, but actually the lack of patches for the new issues
> (often
> > caused by new changes in oe-core).
> >
>
> That's one point.
> If I may add my experience as maintainer of two minor layers
> (meta-handheld and meta-initramfs), I have to say there have been many
> cases where an upgrade to oe-core did break one of these layers.
> Just a variable rename for example...this fall-outs could have been
> avoided if the oe-core dev would have spent a minute grepping the
> repository,
> Heh..now we have to define what is the public repository, which layers
> have to be checked...
>
> And about layer interdependencies...well..it is very delicate.
> For example meta-handheld, a BSP layer, must depend since some time on
> meta-oe. Why? Because tslib was removed from oe-core.
> This is ugly...here a more separated layering structure would help
>
> So I somehow feel we should refine what is in the repository but
> absolutely not split it around.
> Even, I think we should host all public layers (definition lacks
> today) in the OpenEmbedded Git Repository .
>
> My 2 cents
> Andrea
>
> > Yes, the "patch handling" part was the most boring piece, but patches
> from
> > regular contributors or co-maintainers were always easy to handle and
> > rarely caused any extra work.
> >
> > The untested drive-by contributions from people who don't even reply to
> > review comments are completely different category, but separate git
> > repositories won't drive those away (until it gets so confusing that
> these
> > people will even fail to find correct repository and just gave up trying
> to
> > contribute anything at all). Nothing is blocking more people to collect,
> > fix, test these drive-by contributions and then send consolidated
> > pull-request which will be easily merged to shared repository - but I
> never
> > seen this happen and I don't know how separate git repositories will help
> > with this.
> >
> > In the end after spending a lot of own time doing this boring part, I
> > always felt responsible for fixing as many build failures detected in
> world
> > build as I could, but that's never-ending uphill battle where very few
> > people ever help (big thanks to those exceptions who sometimes help). And
> > what you get for that extra afford to get it building as much as possible
> > in spare time? Comments about how garbage meta-oe layers are.
> >
> > We need patches not more git repos!
> >
> > On Wed, Feb 21, 2018 at 9:49 AM, Martin Jansa <martin.jansa@gmail.com>
> > wrote:
> >
> >> > I need an updated python-<foo> package for an unrelated package
> >>
> >> And how far will you go?
> >>
> >> If you want just newer python-<foo> and nothing else, will you take
> other
> >> changes to other python-* recipes from meta-python layer? There is a
> lot of
> >> recipes there, if you're so picky about updates, then you shouldn't
> update
> >> whole oe-core as well.
> >>
> >> On Wed, Feb 21, 2018 at 1:51 AM, Bruce Ashfield <
> bruce.ashfield@gmail.com>
> >> wrote:
> >>
> >>> On Tue, Feb 20, 2018 at 1:52 PM, Khem Raj <raj.khem@gmail.com> wrote:
> >>> > On 2/20/18 9:13 AM, Bruce Ashfield wrote:
> >>> >> On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com>
> >>> wrote:
> >>> >>>
> >>> >>>
> >>> >>> On 02/20/2018 02:45 AM, Burton, Ross wrote:
> >>> >>>> Hi,
> >>> >>>>
> >>> >>>> Is now a good time to talk about splitting up meta-oe?  Some
> layers
> >>> are
> >>> >>>> actively developed and maintained (one example: meta-python),
> others
> >>> are
> >>> >>>> basically bitrotting and only get touched when something else
> causes
> >>> them
> >>> >>>> to break world builds (one example: meta-gnome).  I've long felt
> that
> >>> >>>> meta-oe should be split up and the high quality layers managed in
> >>> their own
> >>> >>>> repositories so patches to them don't get held up by breakage in
> >>> other
> >>> >>>> sub-layers.
> >>> >>> You make it sound like meta-oe is not a high quality layer.  I
> could
> >>> >>> make the same claim about oe-core master.
> >>> >>>
> >>> >>> I don't see the connection in patches being held up due to
> breakage in
> >>> >>> other sub layers. This only happens if the dependency fail to
> build.
> >>> >>>
> >>> >>> You lose control over the quality in current layers that reside in
> >>> >>> meta-openbedded just like you have no control over all the other
> >>> layers
> >>> >>> residing in the community. It makes maintaining stable versions
> very
> >>> >>> difficult. Well, unless The Yocto Project takes over them.. I guess
> >>> that
> >>> >>> would work then.
> >>> >>>
> >>> >>>>
> >>> >>>> Another advantage of splitting out the high quality layers is that
> >>> we'd
> >>> >>>> like to look at running more community layers through the Yocto
> >>> >>>> autobuilder, and granular layers make that easier to manage.
> >>> >>>  I thought not including layers in bblayers.conf was easy enough.
> >>> >>>>
> >>> >>>> Comments?
> >>> >>>
> >>> >>> What problem do you thing you are trying to solve here?
> >>> >>
> >>> >> My unrelated issues are that I can't update one layer, without
> getting
> >>> >> all of the updates.
> >>> >> .. but that is both a good thing (i.e. they are all tested together,
> >>> >> so you know that the
> >>> >> single SRCREV update is good for all layers), and a bad thing (when
> >>> >> you just want a
> >>> >> new python recipe update from meta-python, but don't want other
> >>> changes).
> >>> >>
> >>> >
> >>> > if you dont include the layers in your BBLAYERS and they become
> >>> > effectively non existent, unless you are on metered internet
> connection,
> >>> > where downloading unused stuff would save you bandwidth, it should be
> >>> > ok.  No ?
> >>>
> >>> Its not that.
> >>>
> >>> I *am* building the different layers, but say I have a stable set of
> >>> packages
> >>> and working images .. but for whatever reason, I need an updated
> >>> python-<foo>
> >>> package for an unrelated package, or some other layer that needs a
> newer
> >>> version, etc.
> >>>
> >>> How do I get that, without taking updates to all the layers ? .. and
> >>> layers that
> >>> I really didn't want to update. I have to do some sort of combo-layer,
> >>> carry
> >>> my own copy of the recipe, etc.
> >>>
> >>> So there are definitely ways to do it, I'm just pointing out that I
> >>> end up taking
> >>> some build failures/issues from time to time on packages I didn't
> really
> >>> need to update.
> >>>
> >>> The flip side of that argument is that all of the layers and sub layers
> >>> have
> >>> gone through some sort of global build, and hence I know that they all
> >>> have
> >>> worked together for someone. If I can update pieces individually, I
> break
> >>> that .. and I own the broken bits after that .. which again, goes to
> >>> my point that
> >>> fixing one workflow/issue can break another :D
> >>>
> >>> Bruce
> >>>
> >>> >
> >>> >> It is very likely that splitting the layer would help one issue, but
> >>> >> make the other worse.
> >>> >>
> >>> >> So no solution from me, just an observation about potential issue.
> >>> >>
> >>> >> Cheers,
> >>> >>
> >>> >> Bruce
> >>> >>
> >>> >>>
> >>> >>> - armin
> >>> >>>>
> >>> >>>> Ross
> >>> >>>
> >>> >>>
> >>> >>> --
> >>> >>> _______________________________________________
> >>> >>> Openembedded-devel mailing list
> >>> >>> Openembedded-devel@lists.openembedded.org
> >>> >>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> >>> >>
> >>> >>
> >>> >>
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> "Thou shalt not follow the NULL pointer, for chaos and madness await
> >>> thee at its end"
> >>> --
> >>> _______________________________________________
> >>> Openembedded-devel mailing list
> >>> Openembedded-devel@lists.openembedded.org
> >>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> >>>
> >>
> >>
> > --
> > _______________________________________________
> > Openembedded-devel mailing list
> > Openembedded-devel@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>

Re: Splitting meta-oe?

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 12782 bytes --]

[Re: [oe] Splitting meta-oe?] On 18.02.21 (Wed 11:22) Martin Jansa wrote:

> There is good example of inter-layer dependencies from real world:
> http://lists.openembedded.org/pipermail/openembedded-devel/2017-February/111447.html
> 
> Do you want
> A) new git repository meta-libio-socket-ssl-perl so that meta-networking
> will depend on this on instead of whole meta-perl
> B) meta-ddclient which will probably depend on both meta-perl and
> meta-networking
> C) ddclient and its dependencies in meta-perl
> D) libio-socket-ssl-perl moved to oe-core, so that next time we can say
> that oe-core is just like old oe-classic just with a bit less stuff in it
> 
> Neither of these options is ideal, but meta-networking getting meta-perl
> dependency is the one which causes fewest issues to OE users.

Yeah, I've been thinking about this and trying to decide what is the
"right" thing to do here.  Because I already have added layer
dependencies in meta-networking that I didn't originally envision, so
what's one more that is, as you say, almost certainly in the majority of
consumers' projects anyway?  But it does force a new layer dependency on
consumers of the meta-networking layer who may not care about ddclient,
and I'd like to avoid that if possible.

Honestly, now that I'm back from my vacation, I think the right thing is
to add the dependency and then start thinking about a way to specify
layer dependencies with greater granularity than on a meta-layer basis.
Like, there's no question meta-networking depends on core.  It's
nonsense to think of it without that dependency.  But it'd be nice to be
able to specify a layer dependency that only exists if your project
includes specific recipes out of that layer.

But that kind of mechanism seems highly prone to breakage and likely to
be highly contentious even if it was shown to be reliable, so it may not
get beyond a "that'd be nice" thing for me.

Unless someone else has already implemented it and I'm just not aware of
how to use it?  :-)

-J.

> As mentioned
> before by few people, most builds for "real world products" are already
> including many layers, adding meta-perl to BBLAYERS (it's probably there
> already for other recipes) without the need to update other build setup
> like checking additional git repositories is simple and I don't see how it
> makes meta-oe to be like oe-classic. The layers still separate stuff and I
> don't need to include layers from where I don't need anything at all.
> 
> Smallest set of upstream layers we use at LGE already contains half of
> meta-oe repository:
> oe-core meta-gplv2 meta-oe meta-multimedia meta-networking meta-python
> meta-filesystems meta-qt5
> and around 8 internal layers, so it's far from oe-classic days and
> different builds for different products have different subset of layers.
> 
> For internal layers we also use only a few git repositories 12 layers in
> the main one, because it makes it easier to keep them all compatible with
> each other at all times (e.g. by triggering the CI for all products from
> each gerrit review for this single repository).
> 
> I'm not advocating for putting _everything_ into single git repository or
> even the use of combo-layer tool, but keeping the layers maintained by the
> same "organization" with the same update cycle makes things easier to
> maintain and easier to consume by others.
> 
> Regards,
> 
> On Wed, Feb 21, 2018 at 10:48 AM, Andrea Adami <andrea.adami@gmail.com>
> wrote:
> 
> > On Wed, Feb 21, 2018 at 10:06 AM, Martin Jansa <martin.jansa@gmail.com>
> > wrote:
> > >> I'm actually very worried about these (re)tired maintainers. If the
> > > layers were more independent it would allow some of the patch handling
> > > responsibilities and testing responsibilities to move to other people,
> > > reducing the load on those maintainers.
> > >
> > > Armin can update his own view, but for me the main issue wasn't the
> > "patch
> > > handling" part, but actually the lack of patches for the new issues
> > (often
> > > caused by new changes in oe-core).
> > >
> >
> > That's one point.
> > If I may add my experience as maintainer of two minor layers
> > (meta-handheld and meta-initramfs), I have to say there have been many
> > cases where an upgrade to oe-core did break one of these layers.
> > Just a variable rename for example...this fall-outs could have been
> > avoided if the oe-core dev would have spent a minute grepping the
> > repository,
> > Heh..now we have to define what is the public repository, which layers
> > have to be checked...
> >
> > And about layer interdependencies...well..it is very delicate.
> > For example meta-handheld, a BSP layer, must depend since some time on
> > meta-oe. Why? Because tslib was removed from oe-core.
> > This is ugly...here a more separated layering structure would help
> >
> > So I somehow feel we should refine what is in the repository but
> > absolutely not split it around.
> > Even, I think we should host all public layers (definition lacks
> > today) in the OpenEmbedded Git Repository .
> >
> > My 2 cents
> > Andrea
> >
> > > Yes, the "patch handling" part was the most boring piece, but patches
> > from
> > > regular contributors or co-maintainers were always easy to handle and
> > > rarely caused any extra work.
> > >
> > > The untested drive-by contributions from people who don't even reply to
> > > review comments are completely different category, but separate git
> > > repositories won't drive those away (until it gets so confusing that
> > these
> > > people will even fail to find correct repository and just gave up trying
> > to
> > > contribute anything at all). Nothing is blocking more people to collect,
> > > fix, test these drive-by contributions and then send consolidated
> > > pull-request which will be easily merged to shared repository - but I
> > never
> > > seen this happen and I don't know how separate git repositories will help
> > > with this.
> > >
> > > In the end after spending a lot of own time doing this boring part, I
> > > always felt responsible for fixing as many build failures detected in
> > world
> > > build as I could, but that's never-ending uphill battle where very few
> > > people ever help (big thanks to those exceptions who sometimes help). And
> > > what you get for that extra afford to get it building as much as possible
> > > in spare time? Comments about how garbage meta-oe layers are.
> > >
> > > We need patches not more git repos!
> > >
> > > On Wed, Feb 21, 2018 at 9:49 AM, Martin Jansa <martin.jansa@gmail.com>
> > > wrote:
> > >
> > >> > I need an updated python-<foo> package for an unrelated package
> > >>
> > >> And how far will you go?
> > >>
> > >> If you want just newer python-<foo> and nothing else, will you take
> > other
> > >> changes to other python-* recipes from meta-python layer? There is a
> > lot of
> > >> recipes there, if you're so picky about updates, then you shouldn't
> > update
> > >> whole oe-core as well.
> > >>
> > >> On Wed, Feb 21, 2018 at 1:51 AM, Bruce Ashfield <
> > bruce.ashfield@gmail.com>
> > >> wrote:
> > >>
> > >>> On Tue, Feb 20, 2018 at 1:52 PM, Khem Raj <raj.khem@gmail.com> wrote:
> > >>> > On 2/20/18 9:13 AM, Bruce Ashfield wrote:
> > >>> >> On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com>
> > >>> wrote:
> > >>> >>>
> > >>> >>>
> > >>> >>> On 02/20/2018 02:45 AM, Burton, Ross wrote:
> > >>> >>>> Hi,
> > >>> >>>>
> > >>> >>>> Is now a good time to talk about splitting up meta-oe?  Some
> > layers
> > >>> are
> > >>> >>>> actively developed and maintained (one example: meta-python),
> > others
> > >>> are
> > >>> >>>> basically bitrotting and only get touched when something else
> > causes
> > >>> them
> > >>> >>>> to break world builds (one example: meta-gnome).  I've long felt
> > that
> > >>> >>>> meta-oe should be split up and the high quality layers managed in
> > >>> their own
> > >>> >>>> repositories so patches to them don't get held up by breakage in
> > >>> other
> > >>> >>>> sub-layers.
> > >>> >>> You make it sound like meta-oe is not a high quality layer.  I
> > could
> > >>> >>> make the same claim about oe-core master.
> > >>> >>>
> > >>> >>> I don't see the connection in patches being held up due to
> > breakage in
> > >>> >>> other sub layers. This only happens if the dependency fail to
> > build.
> > >>> >>>
> > >>> >>> You lose control over the quality in current layers that reside in
> > >>> >>> meta-openbedded just like you have no control over all the other
> > >>> layers
> > >>> >>> residing in the community. It makes maintaining stable versions
> > very
> > >>> >>> difficult. Well, unless The Yocto Project takes over them.. I guess
> > >>> that
> > >>> >>> would work then.
> > >>> >>>
> > >>> >>>>
> > >>> >>>> Another advantage of splitting out the high quality layers is that
> > >>> we'd
> > >>> >>>> like to look at running more community layers through the Yocto
> > >>> >>>> autobuilder, and granular layers make that easier to manage.
> > >>> >>>  I thought not including layers in bblayers.conf was easy enough.
> > >>> >>>>
> > >>> >>>> Comments?
> > >>> >>>
> > >>> >>> What problem do you thing you are trying to solve here?
> > >>> >>
> > >>> >> My unrelated issues are that I can't update one layer, without
> > getting
> > >>> >> all of the updates.
> > >>> >> .. but that is both a good thing (i.e. they are all tested together,
> > >>> >> so you know that the
> > >>> >> single SRCREV update is good for all layers), and a bad thing (when
> > >>> >> you just want a
> > >>> >> new python recipe update from meta-python, but don't want other
> > >>> changes).
> > >>> >>
> > >>> >
> > >>> > if you dont include the layers in your BBLAYERS and they become
> > >>> > effectively non existent, unless you are on metered internet
> > connection,
> > >>> > where downloading unused stuff would save you bandwidth, it should be
> > >>> > ok.  No ?
> > >>>
> > >>> Its not that.
> > >>>
> > >>> I *am* building the different layers, but say I have a stable set of
> > >>> packages
> > >>> and working images .. but for whatever reason, I need an updated
> > >>> python-<foo>
> > >>> package for an unrelated package, or some other layer that needs a
> > newer
> > >>> version, etc.
> > >>>
> > >>> How do I get that, without taking updates to all the layers ? .. and
> > >>> layers that
> > >>> I really didn't want to update. I have to do some sort of combo-layer,
> > >>> carry
> > >>> my own copy of the recipe, etc.
> > >>>
> > >>> So there are definitely ways to do it, I'm just pointing out that I
> > >>> end up taking
> > >>> some build failures/issues from time to time on packages I didn't
> > really
> > >>> need to update.
> > >>>
> > >>> The flip side of that argument is that all of the layers and sub layers
> > >>> have
> > >>> gone through some sort of global build, and hence I know that they all
> > >>> have
> > >>> worked together for someone. If I can update pieces individually, I
> > break
> > >>> that .. and I own the broken bits after that .. which again, goes to
> > >>> my point that
> > >>> fixing one workflow/issue can break another :D
> > >>>
> > >>> Bruce
> > >>>
> > >>> >
> > >>> >> It is very likely that splitting the layer would help one issue, but
> > >>> >> make the other worse.
> > >>> >>
> > >>> >> So no solution from me, just an observation about potential issue.
> > >>> >>
> > >>> >> Cheers,
> > >>> >>
> > >>> >> Bruce
> > >>> >>
> > >>> >>>
> > >>> >>> - armin
> > >>> >>>>
> > >>> >>>> Ross
> > >>> >>>
> > >>> >>>
> > >>> >>> --
> > >>> >>> _______________________________________________
> > >>> >>> Openembedded-devel mailing list
> > >>> >>> Openembedded-devel@lists.openembedded.org
> > >>> >>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> > >>> >>
> > >>> >>
> > >>> >>
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> "Thou shalt not follow the NULL pointer, for chaos and madness await
> > >>> thee at its end"
> > >>> --
> > >>> _______________________________________________
> > >>> Openembedded-devel mailing list
> > >>> Openembedded-devel@lists.openembedded.org
> > >>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> > >>>
> > >>
> > >>
> > > --
> > > _______________________________________________
> > > Openembedded-devel mailing list
> > > Openembedded-devel@lists.openembedded.org
> > > http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> >
-- 
-Joe MacDonald.
:wq

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: Splitting meta-oe?

[Burton, Ross]

On 21 February 2018 at 14:02, Joe MacDonald <Joe_MacDonald@mentor.com>
wrote:

> Honestly, now that I'm back from my vacation, I think the right thing is
> to add the dependency and then start thinking about a way to specify
> layer dependencies with greater granularity than on a meta-layer basis.
> Like, there's no question meta-networking depends on core.  It's
> nonsense to think of it without that dependency.  But it'd be nice to be
> able to specify a layer dependency that only exists if your project
> includes specific recipes out of that layer.
>
> But that kind of mechanism seems highly prone to breakage and likely to
> be highly contentious even if it was shown to be reliable, so it may not
> get beyond a "that'd be nice" thing for me.
>
> Unless someone else has already implemented it and I'm just not aware of
> how to use it?  :-)
>

meta-freescale (iirc) does this, conditionally adds sub-layers to BBLAYERS
based on what other layers are already enabled.

Ross

Re: Splitting meta-oe?

[Patrick Ohly]

On Wed, 2018-02-21 at 14:14 +0000, Burton, Ross wrote:
> > But that kind of mechanism seems highly prone to breakage and
> > likely to
> > be highly contentious even if it was shown to be reliable, so it
> > may not
> > get beyond a "that'd be nice" thing for me.
> > 
> > Unless someone else has already implemented it and I'm just not
> > aware of
> > how to use it?  :-)
> > 
> 
> meta-freescale (iirc) does this, conditionally adds
> sub-layers to BBLAYERS based on what other layers are already
> enabled.

The approach used by meta-freescale works with older releases.
BBFILES_DYNAMIC [1] (supported by bitbake 1.36/Yocto 2.4) is a bit more
robust.

[1] https://patchwork.openembedded.org/patch/140532/

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Splitting meta-oe?

[Otavio Salvador]

On Wed, Feb 21, 2018 at 11:58 AM, Patrick Ohly <patrick.ohly@intel.com> wrote:
> On Wed, 2018-02-21 at 14:14 +0000, Burton, Ross wrote:
>> > But that kind of mechanism seems highly prone to breakage and
>> > likely to
>> > be highly contentious even if it was shown to be reliable, so it
>> > may not
>> > get beyond a "that'd be nice" thing for me.
>> >
>> > Unless someone else has already implemented it and I'm just not
>> > aware of
>> > how to use it?  :-)
>> >
>>
>> meta-freescale (iirc) does this, conditionally adds
>> sub-layers to BBLAYERS based on what other layers are already
>> enabled.
>
> The approach used by meta-freescale works with older releases.
> BBFILES_DYNAMIC [1] (supported by bitbake 1.36/Yocto 2.4) is a bit more
> robust.
>
> [1] https://patchwork.openembedded.org/patch/140532/

That's nice; I will update our use for rocko and beyond! :-)

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Splitting meta-oe?

[Andreas Oberritter]

On Wed, 21 Feb 2018 15:58:17 +0100
Patrick Ohly <patrick.ohly@intel.com> wrote:

> On Wed, 2018-02-21 at 14:14 +0000, Burton, Ross wrote:
> > > But that kind of mechanism seems highly prone to breakage and
> > > likely to
> > > be highly contentious even if it was shown to be reliable, so it
> > > may not
> > > get beyond a "that'd be nice" thing for me.
> > > 
> > > Unless someone else has already implemented it and I'm just not
> > > aware of
> > > how to use it?  :-)
> > >   
> > 
> > meta-freescale (iirc) does this, conditionally adds
> > sub-layers to BBLAYERS based on what other layers are already
> > enabled.  
> 
> The approach used by meta-freescale works with older releases.
> BBFILES_DYNAMIC [1] (supported by bitbake 1.36/Yocto 2.4) is a bit more
> robust.
> 
> [1] https://patchwork.openembedded.org/patch/140532/
> 

That's interesting. Is it possible to express dependencies on more than one
layer for a group of recipes with this mechanism?

Regards,
Andreas

Re: Splitting meta-oe?

[Patrick Ohly]

On Wed, 2018-02-21 at 20:33 +0100, Andreas Oberritter wrote:
> On Wed, 21 Feb 2018 15:58:17 +0100
> Patrick Ohly <patrick.ohly@intel.com> wrote:
> > The approach used by meta-freescale works with older releases.
> > BBFILES_DYNAMIC [1] (supported by bitbake 1.36/Yocto 2.4) is a bit
> > more
> > robust.
> > 
> > [1] https://patchwork.openembedded.org/patch/140532/
> > 
> 
> That's interesting. Is it possible to express dependencies on more
> than one layer for a group of recipes with this mechanism?

No, not at the moment.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Splitting meta-oe?

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2895 bytes --]

On Wed, Feb 21, 2018 at 09:02:53AM -0500, Joe MacDonald wrote:
> [Re: [oe] Splitting meta-oe?] On 18.02.21 (Wed 11:22) Martin Jansa wrote:
> 
> > There is good example of inter-layer dependencies from real world:
> > http://lists.openembedded.org/pipermail/openembedded-devel/2017-February/111447.html
> > 
> > Do you want
> > A) new git repository meta-libio-socket-ssl-perl so that meta-networking
> > will depend on this on instead of whole meta-perl
> > B) meta-ddclient which will probably depend on both meta-perl and
> > meta-networking
> > C) ddclient and its dependencies in meta-perl
> > D) libio-socket-ssl-perl moved to oe-core, so that next time we can say
> > that oe-core is just like old oe-classic just with a bit less stuff in it
> > 
> > Neither of these options is ideal, but meta-networking getting meta-perl
> > dependency is the one which causes fewest issues to OE users.
> 
> Yeah, I've been thinking about this and trying to decide what is the
> "right" thing to do here.  Because I already have added layer
> dependencies in meta-networking that I didn't originally envision, so
> what's one more that is, as you say, almost certainly in the majority of
> consumers' projects anyway?  But it does force a new layer dependency on
> consumers of the meta-networking layer who may not care about ddclient,
> and I'd like to avoid that if possible.
> 
> Honestly, now that I'm back from my vacation, I think the right thing is
> to add the dependency and then start thinking about a way to specify
> layer dependencies with greater granularity than on a meta-layer basis.
> Like, there's no question meta-networking depends on core.  It's
> nonsense to think of it without that dependency.  But it'd be nice to be
> able to specify a layer dependency that only exists if your project
> includes specific recipes out of that layer.
> 
> But that kind of mechanism seems highly prone to breakage and likely to
> be highly contentious even if it was shown to be reliable, so it may not
> get beyond a "that'd be nice" thing for me.
> 
> Unless someone else has already implemented it and I'm just not aware of
> how to use it?  :-)

One thing I've been thinking about is that some layers need more
sub-layers.  Taking a very tiny peek into meta-networking, perhaps
re-organize into meta-networking-core, meta-networking-iscsi,
meta-networking-vpn, etc.  Or maybe that won't help with dependencies.
But the need for layer X for a single recipe can sometimes I think be
solved by re-thinking the layer.

All that said, it might be better instead to add something like
RECIPE_LAYERDEPENDS so that for the one-or-two offs, the recipe will
fail to build (and implement that similar to the logic in
image-container.bbclass? so that you only get a failure on building that
recipe rather than anything in the layer) ?

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Splitting meta-oe?

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 3445 bytes --]

[Re: [oe] Splitting meta-oe?] On 18.02.21 (Wed 09:20) Tom Rini wrote:

> On Wed, Feb 21, 2018 at 09:02:53AM -0500, Joe MacDonald wrote:
> > [Re: [oe] Splitting meta-oe?] On 18.02.21 (Wed 11:22) Martin Jansa wrote:
> > 
> > > There is good example of inter-layer dependencies from real world:
> > > http://lists.openembedded.org/pipermail/openembedded-devel/2017-February/111447.html
> > > 
> > > Do you want
> > > A) new git repository meta-libio-socket-ssl-perl so that meta-networking
> > > will depend on this on instead of whole meta-perl
> > > B) meta-ddclient which will probably depend on both meta-perl and
> > > meta-networking
> > > C) ddclient and its dependencies in meta-perl
> > > D) libio-socket-ssl-perl moved to oe-core, so that next time we can say
> > > that oe-core is just like old oe-classic just with a bit less stuff in it
> > > 
> > > Neither of these options is ideal, but meta-networking getting meta-perl
> > > dependency is the one which causes fewest issues to OE users.
> > 
> > Yeah, I've been thinking about this and trying to decide what is the
> > "right" thing to do here.  Because I already have added layer
> > dependencies in meta-networking that I didn't originally envision, so
> > what's one more that is, as you say, almost certainly in the majority of
> > consumers' projects anyway?  But it does force a new layer dependency on
> > consumers of the meta-networking layer who may not care about ddclient,
> > and I'd like to avoid that if possible.
> > 
> > Honestly, now that I'm back from my vacation, I think the right thing is
> > to add the dependency and then start thinking about a way to specify
> > layer dependencies with greater granularity than on a meta-layer basis.
> > Like, there's no question meta-networking depends on core.  It's
> > nonsense to think of it without that dependency.  But it'd be nice to be
> > able to specify a layer dependency that only exists if your project
> > includes specific recipes out of that layer.
> > 
> > But that kind of mechanism seems highly prone to breakage and likely to
> > be highly contentious even if it was shown to be reliable, so it may not
> > get beyond a "that'd be nice" thing for me.
> > 
> > Unless someone else has already implemented it and I'm just not aware of
> > how to use it?  :-)
> 
> One thing I've been thinking about is that some layers need more
> sub-layers.  Taking a very tiny peek into meta-networking, perhaps
> re-organize into meta-networking-core, meta-networking-iscsi,
> meta-networking-vpn, etc.  Or maybe that won't help with dependencies.
> But the need for layer X for a single recipe can sometimes I think be
> solved by re-thinking the layer.

I really like that idea.  We effectively started that with the netkit
stuff when Armin first submitted it, so making it more formal isn't a
big step anyway.

> All that said, it might be better instead to add something like
> RECIPE_LAYERDEPENDS so that for the one-or-two offs, the recipe will
> fail to build (and implement that similar to the logic in
> image-container.bbclass? so that you only get a failure on building that
> recipe rather than anything in the layer) ?

Yeah, that's kind of what I'd been thinking.  In the short term, though,
the reorg you suggested above sounds *very* appealing to me at first
glance.  Might be a "let's do both" situation.

-- 
-Joe MacDonald.
:wq

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: Splitting meta-oe?

[Bruce Ashfield]

On Wed, Feb 21, 2018 at 3:49 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
>> I need an updated python-<foo> package for an unrelated package
>
> And how far will you go?
>

pretty far. I work with a lot of deep stacks that have a lot of specific
dependencies as well as compatibility issues.

> If you want just newer python-<foo> and nothing else, will you take other
> changes to other python-* recipes from meta-python layer? There is a lot of
> recipes there, if you're so picky about updates, then you shouldn't update
> whole oe-core as well.

I actually don't always, but generally speaking, I haven't had many
problems with package updates from oe-core. I end up with breakage
due to core build system changes, and that impacts oe-core and any
layer either.

But as I pointed out, I'm not looking to have my problem 'solved', I'm
just pointing out that it is a valid concern .. whether or not others
agree.

Cheers,

Bruce

>
> On Wed, Feb 21, 2018 at 1:51 AM, Bruce Ashfield <bruce.ashfield@gmail.com>
> wrote:
>>
>> On Tue, Feb 20, 2018 at 1:52 PM, Khem Raj <raj.khem@gmail.com> wrote:
>> > On 2/20/18 9:13 AM, Bruce Ashfield wrote:
>> >> On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com>
>> >> wrote:
>> >>>
>> >>>
>> >>> On 02/20/2018 02:45 AM, Burton, Ross wrote:
>> >>>> Hi,
>> >>>>
>> >>>> Is now a good time to talk about splitting up meta-oe?  Some layers
>> >>>> are
>> >>>> actively developed and maintained (one example: meta-python), others
>> >>>> are
>> >>>> basically bitrotting and only get touched when something else causes
>> >>>> them
>> >>>> to break world builds (one example: meta-gnome).  I've long felt that
>> >>>> meta-oe should be split up and the high quality layers managed in
>> >>>> their own
>> >>>> repositories so patches to them don't get held up by breakage in
>> >>>> other
>> >>>> sub-layers.
>> >>> You make it sound like meta-oe is not a high quality layer.  I could
>> >>> make the same claim about oe-core master.
>> >>>
>> >>> I don't see the connection in patches being held up due to breakage in
>> >>> other sub layers. This only happens if the dependency fail to build.
>> >>>
>> >>> You lose control over the quality in current layers that reside in
>> >>> meta-openbedded just like you have no control over all the other
>> >>> layers
>> >>> residing in the community. It makes maintaining stable versions very
>> >>> difficult. Well, unless The Yocto Project takes over them.. I guess
>> >>> that
>> >>> would work then.
>> >>>
>> >>>>
>> >>>> Another advantage of splitting out the high quality layers is that
>> >>>> we'd
>> >>>> like to look at running more community layers through the Yocto
>> >>>> autobuilder, and granular layers make that easier to manage.
>> >>>  I thought not including layers in bblayers.conf was easy enough.
>> >>>>
>> >>>> Comments?
>> >>>
>> >>> What problem do you thing you are trying to solve here?
>> >>
>> >> My unrelated issues are that I can't update one layer, without getting
>> >> all of the updates.
>> >> .. but that is both a good thing (i.e. they are all tested together,
>> >> so you know that the
>> >> single SRCREV update is good for all layers), and a bad thing (when
>> >> you just want a
>> >> new python recipe update from meta-python, but don't want other
>> >> changes).
>> >>
>> >
>> > if you dont include the layers in your BBLAYERS and they become
>> > effectively non existent, unless you are on metered internet connection,
>> > where downloading unused stuff would save you bandwidth, it should be
>> > ok.  No ?
>>
>> Its not that.
>>
>> I *am* building the different layers, but say I have a stable set of
>> packages
>> and working images .. but for whatever reason, I need an updated
>> python-<foo>
>> package for an unrelated package, or some other layer that needs a newer
>> version, etc.
>>
>> How do I get that, without taking updates to all the layers ? .. and
>> layers that
>> I really didn't want to update. I have to do some sort of combo-layer,
>> carry
>> my own copy of the recipe, etc.
>>
>> So there are definitely ways to do it, I'm just pointing out that I
>> end up taking
>> some build failures/issues from time to time on packages I didn't really
>> need to update.
>>
>> The flip side of that argument is that all of the layers and sub layers
>> have
>> gone through some sort of global build, and hence I know that they all
>> have
>> worked together for someone. If I can update pieces individually, I break
>> that .. and I own the broken bits after that .. which again, goes to
>> my point that
>> fixing one workflow/issue can break another :D
>>
>> Bruce
>>
>> >
>> >> It is very likely that splitting the layer would help one issue, but
>> >> make the other worse.
>> >>
>> >> So no solution from me, just an observation about potential issue.
>> >>
>> >> Cheers,
>> >>
>> >> Bruce
>> >>
>> >>>
>> >>> - armin
>> >>>>
>> >>>> Ross
>> >>>
>> >>>
>> >>> --
>> >>> _______________________________________________
>> >>> Openembedded-devel mailing list
>> >>> Openembedded-devel@lists.openembedded.org
>> >>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>> >>
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> "Thou shalt not follow the NULL pointer, for chaos and madness await
>> thee at its end"
>> --
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Splitting meta-oe?

[Bruce Ashfield]

On Wed, Feb 21, 2018 at 8:34 AM, Bruce Ashfield
<bruce.ashfield@gmail.com> wrote:
> On Wed, Feb 21, 2018 at 3:49 AM, Martin Jansa <martin.jansa@gmail.com> wrote:
>>> I need an updated python-<foo> package for an unrelated package
>>
>> And how far will you go?
>>
>
> pretty far. I work with a lot of deep stacks that have a lot of specific
> dependencies as well as compatibility issues.
>
>> If you want just newer python-<foo> and nothing else, will you take other
>> changes to other python-* recipes from meta-python layer? There is a lot of
>> recipes there, if you're so picky about updates, then you shouldn't update
>> whole oe-core as well.
>
> I actually don't always, but generally speaking, I haven't had many
> problems with package updates from oe-core. I end up with breakage
> due to core build system changes, and that impacts oe-core and any
> layer either.

oh, and I'd add that this point is somewhat contrived (few package
breakages), since that is really just a fallout of the oe-core packages
being fairly ... core .. (and boring), so really there aren't any hard or
strange dependencies that cause issues and that's a fall out of the
content, not the workflow or model (or git repo split, or not!).

Cheers,

Bruce

>
> But as I pointed out, I'm not looking to have my problem 'solved', I'm
> just pointing out that it is a valid concern .. whether or not others
> agree.
>
> Cheers,
>
> Bruce
>
>>
>> On Wed, Feb 21, 2018 at 1:51 AM, Bruce Ashfield <bruce.ashfield@gmail.com>
>> wrote:
>>>
>>> On Tue, Feb 20, 2018 at 1:52 PM, Khem Raj <raj.khem@gmail.com> wrote:
>>> > On 2/20/18 9:13 AM, Bruce Ashfield wrote:
>>> >> On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>>
>>> >>> On 02/20/2018 02:45 AM, Burton, Ross wrote:
>>> >>>> Hi,
>>> >>>>
>>> >>>> Is now a good time to talk about splitting up meta-oe?  Some layers
>>> >>>> are
>>> >>>> actively developed and maintained (one example: meta-python), others
>>> >>>> are
>>> >>>> basically bitrotting and only get touched when something else causes
>>> >>>> them
>>> >>>> to break world builds (one example: meta-gnome).  I've long felt that
>>> >>>> meta-oe should be split up and the high quality layers managed in
>>> >>>> their own
>>> >>>> repositories so patches to them don't get held up by breakage in
>>> >>>> other
>>> >>>> sub-layers.
>>> >>> You make it sound like meta-oe is not a high quality layer.  I could
>>> >>> make the same claim about oe-core master.
>>> >>>
>>> >>> I don't see the connection in patches being held up due to breakage in
>>> >>> other sub layers. This only happens if the dependency fail to build.
>>> >>>
>>> >>> You lose control over the quality in current layers that reside in
>>> >>> meta-openbedded just like you have no control over all the other
>>> >>> layers
>>> >>> residing in the community. It makes maintaining stable versions very
>>> >>> difficult. Well, unless The Yocto Project takes over them.. I guess
>>> >>> that
>>> >>> would work then.
>>> >>>
>>> >>>>
>>> >>>> Another advantage of splitting out the high quality layers is that
>>> >>>> we'd
>>> >>>> like to look at running more community layers through the Yocto
>>> >>>> autobuilder, and granular layers make that easier to manage.
>>> >>>  I thought not including layers in bblayers.conf was easy enough.
>>> >>>>
>>> >>>> Comments?
>>> >>>
>>> >>> What problem do you thing you are trying to solve here?
>>> >>
>>> >> My unrelated issues are that I can't update one layer, without getting
>>> >> all of the updates.
>>> >> .. but that is both a good thing (i.e. they are all tested together,
>>> >> so you know that the
>>> >> single SRCREV update is good for all layers), and a bad thing (when
>>> >> you just want a
>>> >> new python recipe update from meta-python, but don't want other
>>> >> changes).
>>> >>
>>> >
>>> > if you dont include the layers in your BBLAYERS and they become
>>> > effectively non existent, unless you are on metered internet connection,
>>> > where downloading unused stuff would save you bandwidth, it should be
>>> > ok.  No ?
>>>
>>> Its not that.
>>>
>>> I *am* building the different layers, but say I have a stable set of
>>> packages
>>> and working images .. but for whatever reason, I need an updated
>>> python-<foo>
>>> package for an unrelated package, or some other layer that needs a newer
>>> version, etc.
>>>
>>> How do I get that, without taking updates to all the layers ? .. and
>>> layers that
>>> I really didn't want to update. I have to do some sort of combo-layer,
>>> carry
>>> my own copy of the recipe, etc.
>>>
>>> So there are definitely ways to do it, I'm just pointing out that I
>>> end up taking
>>> some build failures/issues from time to time on packages I didn't really
>>> need to update.
>>>
>>> The flip side of that argument is that all of the layers and sub layers
>>> have
>>> gone through some sort of global build, and hence I know that they all
>>> have
>>> worked together for someone. If I can update pieces individually, I break
>>> that .. and I own the broken bits after that .. which again, goes to
>>> my point that
>>> fixing one workflow/issue can break another :D
>>>
>>> Bruce
>>>
>>> >
>>> >> It is very likely that splitting the layer would help one issue, but
>>> >> make the other worse.
>>> >>
>>> >> So no solution from me, just an observation about potential issue.
>>> >>
>>> >> Cheers,
>>> >>
>>> >> Bruce
>>> >>
>>> >>>
>>> >>> - armin
>>> >>>>
>>> >>>> Ross
>>> >>>
>>> >>>
>>> >>> --
>>> >>> _______________________________________________
>>> >>> Openembedded-devel mailing list
>>> >>> Openembedded-devel@lists.openembedded.org
>>> >>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>>> >>
>>> >>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> "Thou shalt not follow the NULL pointer, for chaos and madness await
>>> thee at its end"
>>> --
>>> _______________________________________________
>>> Openembedded-devel mailing list
>>> Openembedded-devel@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>>
>>
>
>
>
> --
> "Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end"



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Splitting meta-oe?

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 5794 bytes --]

[Re: [oe] Splitting meta-oe?] On 18.02.21 (Wed 09:49) Martin Jansa wrote:

> > I need an updated python-<foo> package for an unrelated package
> 
> And how far will you go?
> 
> If you want just newer python-<foo> and nothing else, will you take other
> changes to other python-* recipes from meta-python layer? There is a lot of
> recipes there, if you're so picky about updates, then you shouldn't update
> whole oe-core as well.

It seems like this part is already settled, but it would seem to me that
this scenario, if I understand it correctly, is "I'm using oe-core and
see up-stream meta-oe/meta-somethingorother has an updated or new recipe
for something I want in my image but I don't want to do a pull on all of
meta-oe and potentially cause a huge rebuild of stuff I don't really
care about".

That sounds like a problem better solved with 'git branch', 'git fetch'
and 'git cherry-pick' on the developer's side than breaking up meta-oe
across existing meta-boundaries.

-J.

> 
> On Wed, Feb 21, 2018 at 1:51 AM, Bruce Ashfield <bruce.ashfield@gmail.com>
> wrote:
> 
> > On Tue, Feb 20, 2018 at 1:52 PM, Khem Raj <raj.khem@gmail.com> wrote:
> > > On 2/20/18 9:13 AM, Bruce Ashfield wrote:
> > >> On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com>
> > wrote:
> > >>>
> > >>>
> > >>> On 02/20/2018 02:45 AM, Burton, Ross wrote:
> > >>>> Hi,
> > >>>>
> > >>>> Is now a good time to talk about splitting up meta-oe?  Some layers
> > are
> > >>>> actively developed and maintained (one example: meta-python), others
> > are
> > >>>> basically bitrotting and only get touched when something else causes
> > them
> > >>>> to break world builds (one example: meta-gnome).  I've long felt that
> > >>>> meta-oe should be split up and the high quality layers managed in
> > their own
> > >>>> repositories so patches to them don't get held up by breakage in other
> > >>>> sub-layers.
> > >>> You make it sound like meta-oe is not a high quality layer.  I could
> > >>> make the same claim about oe-core master.
> > >>>
> > >>> I don't see the connection in patches being held up due to breakage in
> > >>> other sub layers. This only happens if the dependency fail to build.
> > >>>
> > >>> You lose control over the quality in current layers that reside in
> > >>> meta-openbedded just like you have no control over all the other layers
> > >>> residing in the community. It makes maintaining stable versions very
> > >>> difficult. Well, unless The Yocto Project takes over them.. I guess
> > that
> > >>> would work then.
> > >>>
> > >>>>
> > >>>> Another advantage of splitting out the high quality layers is that
> > we'd
> > >>>> like to look at running more community layers through the Yocto
> > >>>> autobuilder, and granular layers make that easier to manage.
> > >>>  I thought not including layers in bblayers.conf was easy enough.
> > >>>>
> > >>>> Comments?
> > >>>
> > >>> What problem do you thing you are trying to solve here?
> > >>
> > >> My unrelated issues are that I can't update one layer, without getting
> > >> all of the updates.
> > >> .. but that is both a good thing (i.e. they are all tested together,
> > >> so you know that the
> > >> single SRCREV update is good for all layers), and a bad thing (when
> > >> you just want a
> > >> new python recipe update from meta-python, but don't want other
> > changes).
> > >>
> > >
> > > if you dont include the layers in your BBLAYERS and they become
> > > effectively non existent, unless you are on metered internet connection,
> > > where downloading unused stuff would save you bandwidth, it should be
> > > ok.  No ?
> >
> > Its not that.
> >
> > I *am* building the different layers, but say I have a stable set of
> > packages
> > and working images .. but for whatever reason, I need an updated
> > python-<foo>
> > package for an unrelated package, or some other layer that needs a newer
> > version, etc.
> >
> > How do I get that, without taking updates to all the layers ? .. and
> > layers that
> > I really didn't want to update. I have to do some sort of combo-layer,
> > carry
> > my own copy of the recipe, etc.
> >
> > So there are definitely ways to do it, I'm just pointing out that I
> > end up taking
> > some build failures/issues from time to time on packages I didn't really
> > need to update.
> >
> > The flip side of that argument is that all of the layers and sub layers
> > have
> > gone through some sort of global build, and hence I know that they all have
> > worked together for someone. If I can update pieces individually, I break
> > that .. and I own the broken bits after that .. which again, goes to
> > my point that
> > fixing one workflow/issue can break another :D
> >
> > Bruce
> >
> > >
> > >> It is very likely that splitting the layer would help one issue, but
> > >> make the other worse.
> > >>
> > >> So no solution from me, just an observation about potential issue.
> > >>
> > >> Cheers,
> > >>
> > >> Bruce
> > >>
> > >>>
> > >>> - armin
> > >>>>
> > >>>> Ross
> > >>>
> > >>>
> > >>> --
> > >>> _______________________________________________
> > >>> Openembedded-devel mailing list
> > >>> Openembedded-devel@lists.openembedded.org
> > >>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> > >>
> > >>
> > >>
> > >
> >
> >
> >
> > --
> > "Thou shalt not follow the NULL pointer, for chaos and madness await
> > thee at its end"
> > --
> > _______________________________________________
> > Openembedded-devel mailing list
> > Openembedded-devel@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> >
-- 
-Joe MacDonald.
:wq

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 499 bytes --]

Re: Splitting meta-oe?

[Bruce Ashfield]

On Wed, Feb 21, 2018 at 8:45 AM, Joe MacDonald <Joe_MacDonald@mentor.com> wrote:
> [Re: [oe] Splitting meta-oe?] On 18.02.21 (Wed 09:49) Martin Jansa wrote:
>
>> > I need an updated python-<foo> package for an unrelated package
>>
>> And how far will you go?
>>
>> If you want just newer python-<foo> and nothing else, will you take other
>> changes to other python-* recipes from meta-python layer? There is a lot of
>> recipes there, if you're so picky about updates, then you shouldn't update
>> whole oe-core as well.
>
> It seems like this part is already settled, but it would seem to me that
> this scenario, if I understand it correctly, is "I'm using oe-core and
> see up-stream meta-oe/meta-somethingorother has an updated or new recipe
> for something I want in my image but I don't want to do a pull on all of
> meta-oe and potentially cause a huge rebuild of stuff I don't really
> care about".
>
> That sounds like a problem better solved with 'git branch', 'git fetch'
> and 'git cherry-pick' on the developer's side than breaking up meta-oe
> across existing meta-boundaries.

Agreed. I was just probing to see if anyone had ideas on how that could
be managed by infrastructure, versus me mucking about.

Maybe RP's thoughts on layer tooling or setup possibilities would help,
but either way, I'll keep chugging along :D

Cheers,

Bruce

>
> -J.
>
>>
>> On Wed, Feb 21, 2018 at 1:51 AM, Bruce Ashfield <bruce.ashfield@gmail.com>
>> wrote:
>>
>> > On Tue, Feb 20, 2018 at 1:52 PM, Khem Raj <raj.khem@gmail.com> wrote:
>> > > On 2/20/18 9:13 AM, Bruce Ashfield wrote:
>> > >> On Tue, Feb 20, 2018 at 11:50 AM, akuster808 <akuster808@gmail.com>
>> > wrote:
>> > >>>
>> > >>>
>> > >>> On 02/20/2018 02:45 AM, Burton, Ross wrote:
>> > >>>> Hi,
>> > >>>>
>> > >>>> Is now a good time to talk about splitting up meta-oe?  Some layers
>> > are
>> > >>>> actively developed and maintained (one example: meta-python), others
>> > are
>> > >>>> basically bitrotting and only get touched when something else causes
>> > them
>> > >>>> to break world builds (one example: meta-gnome).  I've long felt that
>> > >>>> meta-oe should be split up and the high quality layers managed in
>> > their own
>> > >>>> repositories so patches to them don't get held up by breakage in other
>> > >>>> sub-layers.
>> > >>> You make it sound like meta-oe is not a high quality layer.  I could
>> > >>> make the same claim about oe-core master.
>> > >>>
>> > >>> I don't see the connection in patches being held up due to breakage in
>> > >>> other sub layers. This only happens if the dependency fail to build.
>> > >>>
>> > >>> You lose control over the quality in current layers that reside in
>> > >>> meta-openbedded just like you have no control over all the other layers
>> > >>> residing in the community. It makes maintaining stable versions very
>> > >>> difficult. Well, unless The Yocto Project takes over them.. I guess
>> > that
>> > >>> would work then.
>> > >>>
>> > >>>>
>> > >>>> Another advantage of splitting out the high quality layers is that
>> > we'd
>> > >>>> like to look at running more community layers through the Yocto
>> > >>>> autobuilder, and granular layers make that easier to manage.
>> > >>>  I thought not including layers in bblayers.conf was easy enough.
>> > >>>>
>> > >>>> Comments?
>> > >>>
>> > >>> What problem do you thing you are trying to solve here?
>> > >>
>> > >> My unrelated issues are that I can't update one layer, without getting
>> > >> all of the updates.
>> > >> .. but that is both a good thing (i.e. they are all tested together,
>> > >> so you know that the
>> > >> single SRCREV update is good for all layers), and a bad thing (when
>> > >> you just want a
>> > >> new python recipe update from meta-python, but don't want other
>> > changes).
>> > >>
>> > >
>> > > if you dont include the layers in your BBLAYERS and they become
>> > > effectively non existent, unless you are on metered internet connection,
>> > > where downloading unused stuff would save you bandwidth, it should be
>> > > ok.  No ?
>> >
>> > Its not that.
>> >
>> > I *am* building the different layers, but say I have a stable set of
>> > packages
>> > and working images .. but for whatever reason, I need an updated
>> > python-<foo>
>> > package for an unrelated package, or some other layer that needs a newer
>> > version, etc.
>> >
>> > How do I get that, without taking updates to all the layers ? .. and
>> > layers that
>> > I really didn't want to update. I have to do some sort of combo-layer,
>> > carry
>> > my own copy of the recipe, etc.
>> >
>> > So there are definitely ways to do it, I'm just pointing out that I
>> > end up taking
>> > some build failures/issues from time to time on packages I didn't really
>> > need to update.
>> >
>> > The flip side of that argument is that all of the layers and sub layers
>> > have
>> > gone through some sort of global build, and hence I know that they all have
>> > worked together for someone. If I can update pieces individually, I break
>> > that .. and I own the broken bits after that .. which again, goes to
>> > my point that
>> > fixing one workflow/issue can break another :D
>> >
>> > Bruce
>> >
>> > >
>> > >> It is very likely that splitting the layer would help one issue, but
>> > >> make the other worse.
>> > >>
>> > >> So no solution from me, just an observation about potential issue.
>> > >>
>> > >> Cheers,
>> > >>
>> > >> Bruce
>> > >>
>> > >>>
>> > >>> - armin
>> > >>>>
>> > >>>> Ross
>> > >>>
>> > >>>
>> > >>> --
>> > >>> _______________________________________________
>> > >>> Openembedded-devel mailing list
>> > >>> Openembedded-devel@lists.openembedded.org
>> > >>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>> > >>
>> > >>
>> > >>
>> > >
>> >
>> >
>> >
>> > --
>> > "Thou shalt not follow the NULL pointer, for chaos and madness await
>> > thee at its end"
>> > --
>> > _______________________________________________
>> > Openembedded-devel mailing list
>> > Openembedded-devel@lists.openembedded.org
>> > http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>> >
> --
> -Joe MacDonald.
> :wq



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Splitting meta-oe?

[Otavio Salvador]

On Wed, Feb 21, 2018 at 10:55 AM, Bruce Ashfield
<bruce.ashfield@gmail.com> wrote:
> On Wed, Feb 21, 2018 at 8:45 AM, Joe MacDonald <Joe_MacDonald@mentor.com> wrote:
>> [Re: [oe] Splitting meta-oe?] On 18.02.21 (Wed 09:49) Martin Jansa wrote:
>>
>>> > I need an updated python-<foo> package for an unrelated package
>>>
>>> And how far will you go?
>>>
>>> If you want just newer python-<foo> and nothing else, will you take other
>>> changes to other python-* recipes from meta-python layer? There is a lot of
>>> recipes there, if you're so picky about updates, then you shouldn't update
>>> whole oe-core as well.
>>
>> It seems like this part is already settled, but it would seem to me that
>> this scenario, if I understand it correctly, is "I'm using oe-core and
>> see up-stream meta-oe/meta-somethingorother has an updated or new recipe
>> for something I want in my image but I don't want to do a pull on all of
>> meta-oe and potentially cause a huge rebuild of stuff I don't really
>> care about".
>>
>> That sounds like a problem better solved with 'git branch', 'git fetch'
>> and 'git cherry-pick' on the developer's side than breaking up meta-oe
>> across existing meta-boundaries.
>
> Agreed. I was just probing to see if anyone had ideas on how that could
> be managed by infrastructure, versus me mucking about.
>
> Maybe RP's thoughts on layer tooling or setup possibilities would help,
> but either way, I'll keep chugging along :D

I think this is a valid thing to discuss but not in this thread (as
well as for tooling and etc.).

Here I think we ought to focus in:

 Should we split meta-oe?

My vote is no! Yocto Project as a whole would benefit of people inside
of Yocto Project QA sending fixes for parse errors introduced by their
changes in OE-Core  and so I'd prefer to keep it as is.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Splitting meta-oe?

[Richard Purdie]

Repositories have reputations. OE-Core is fairly heavily curated and
tested and has certain "guarantees" about what people can expect to
work. The meta-oe repo is a little trickier as the different pieces do
have different 'SLA's (service level agreements). Some pieces like
meta-networking/meta-python likely build well, other pieces like some
of the bitrotting pieces of the meta-oe layer probably don't build in
as many different configurations/architectures and on balance may be
more likely to hit issues.

I think the world may need to change a bit. We should probably change:

* The way people get started with the Yocto Project and Poky
  to move away from the combo repo and into specific layers.
* to have YP testing more layers as part of its builds and testing
* to make it easier for people to establish an SLA type reputation for 
  a layer.

Part of this is a need for a more universal "using OE" later setup
tooling. I've wanted to work on that for a while and I will get there
but if I spend time on it, I want to get it right. I've not had enough
time to get it right (as yet).

Part of this means redefining poky, the YP autobuilders and so on.
Pieces of the autobuilder work are in process and will lead to wider
layer testing.

I have strong influence over the above so I can likely make those
happen, time constraints aside. One piece I need the help of others
with is meta-oe (the repo).

Even just splitting meta-oe out from meta-oe might be enough to
establish the SLA differences, I don't know.

There is a question about what problem would this solve. The move from
OE-Classic was partly about defining maintainership and processes for
recipes which we do with layers. The pieces in the separate layers have
now become the things we set out to create but the catchall of meta-oe
(the layer) remains. I do think we should still be aiming to filter
meta-oe into distinct layers with distinct maintainership. There will
always be some "leftovers" in a catchall but I think it does have a
different personality to the other well separated components and we
need to show to users that they do have different expectations from
them.

I hope I'm managing to convey this concept ok, I'm really trying to
take a step back here and think what will help OE users and the project
and where we need to go with things. I'm not saying we should/shouldn't
do this for 100% certain, I do want people to think about the
alternatives though and what options may be possible.

Cheers,

Richard

Re: Splitting meta-oe?

[Tim Orling]

The recent improvements to the Auto Upgrade Helper have made maintenance of
meta-perl less effort and therefore you have seen an uptick in my updates
to recipes. I also plan to mass-add ptest to all perl module recipes
"sometime soon", which will further simplify my upgrade workflow and add to
the confidence of said recipes. My upgrade workflow touches all recipes in
oe-core, meta-oe and meta-perl that fit a lib*perl regex.

I will note, however, that we are revisiting meta-cpan and that may
ultimately be the preferred perl layer. If you really heavily on perl
modules, please comment on whether meta-cpan would fit your needs.

Derek and others have been doing a great job keeping meta-python up to
date. I mostly jump in when there are trickier failures that need more
concentrated effort. I would like to see increased testing (more packages
with ptest), but that is trickier with python recipes as there are many
more approaches to testing than there are in perl. I am happy with the
current efforts to keep recipes up to date as a priority. I would like to
see complete coverage of python3 for all recipes that are capable in
meta-python.

I am open to discussion about what direction we go. Individual layers that
are curated and built together by YP auto builders sounds like an
intriguing path. If this was coupled with increased ptest or testimage
usage, our confidence in layer quality would go up dramatically.

Cheers,

--Tim

Re: Splitting meta-oe?

[Martin Jansa]

I still fail to see how splitting the layers into separate repositories
will magically help with the quality of recipes stored there.

meta-python, meta-perl, meta-filesystems, meta-gnome, meta-networking,
meta-webserver, meta-xfce all depend on meta-oe layer.

Build testing fewer layers in "world" builds will probably lower the
overall quality of the layers (and their compatibility with each other).

Building the same set of layers in "world" builds just from different
repositories, won't make the build failures go away, it will just cause
more work for Armin or whoever will be willing to continue running those
builds.

Testing some layers like meta-python and meta-perl on YP auto builder would
be nice, but why do they need to be in separate repository for that?

Making meta-networking, meta-python and meta-perl not to depend on meta-oe
is also nice goal, but again not blocked by all these layers sharing the
same git repository. Creating new
meta-shared-stuff-nobody-knows-how-to-call-this-layer as common dependency
for these 3 is also possible in meta-oe git repository (even easier to
switch to then creating new repository for each of them).

Armin is still merging patches e.g. to meta-python even when last clean
"bitbake world" status was on 2017-05-03. So don't make it sound like
meta-gnome is blocking meta-python progress. FWIW: There is only one
failing recipe in meta-gnome and at least 7 failing in meta-networking (to
be fair 5 of them are caused by very recent kernel upgrade).

If the git repository has the reputation (not the layers you're using from
there), then I can say that oe-core git repository is quite bad, because it
has meta-skeleton which doesn't have anything useful for me and has only 4
commits in last 2 years.

Regards,

On Tue, Feb 20, 2018 at 7:00 PM, Tim Orling <ticotimo@gmail.com> wrote:

> The recent improvements to the Auto Upgrade Helper have made maintenance of
> meta-perl less effort and therefore you have seen an uptick in my updates
> to recipes. I also plan to mass-add ptest to all perl module recipes
> "sometime soon", which will further simplify my upgrade workflow and add to
> the confidence of said recipes. My upgrade workflow touches all recipes in
> oe-core, meta-oe and meta-perl that fit a lib*perl regex.
>
> I will note, however, that we are revisiting meta-cpan and that may
> ultimately be the preferred perl layer. If you really heavily on perl
> modules, please comment on whether meta-cpan would fit your needs.
>
> Derek and others have been doing a great job keeping meta-python up to
> date. I mostly jump in when there are trickier failures that need more
> concentrated effort. I would like to see increased testing (more packages
> with ptest), but that is trickier with python recipes as there are many
> more approaches to testing than there are in perl. I am happy with the
> current efforts to keep recipes up to date as a priority. I would like to
> see complete coverage of python3 for all recipes that are capable in
> meta-python.
>
> I am open to discussion about what direction we go. Individual layers that
> are curated and built together by YP auto builders sounds like an
> intriguing path. If this was coupled with increased ptest or testimage
> usage, our confidence in layer quality would go up dramatically.
>
> Cheers,
>
> --Tim
> --
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>

Re: Splitting meta-oe?

[Khem Raj]

On 2/20/18 10:00 AM, Tim Orling wrote:
> The recent improvements to the Auto Upgrade Helper have made maintenance of
> meta-perl less effort and therefore you have seen an uptick in my updates
> to recipes. I also plan to mass-add ptest to all perl module recipes
> "sometime soon", which will further simplify my upgrade workflow and add to
> the confidence of said recipes. My upgrade workflow touches all recipes in
> oe-core, meta-oe and meta-perl that fit a lib*perl regex.
> 
> I will note, however, that we are revisiting meta-cpan and that may
> ultimately be the preferred perl layer. If you really heavily on perl
> modules, please comment on whether meta-cpan would fit your needs.
> 
> Derek and others have been doing a great job keeping meta-python up to
> date. I mostly jump in when there are trickier failures that need more
> concentrated effort. I would like to see increased testing (more packages
> with ptest), but that is trickier with python recipes as there are many
> more approaches to testing than there are in perl. I am happy with the
> current efforts to keep recipes up to date as a priority. I would like to
> see complete coverage of python3 for all recipes that are capable in
> meta-python.
> 
> I am open to discussion about what direction we go. Individual layers that
> are curated and built together by YP auto builders sounds like an
> intriguing path. If this was coupled with increased ptest or testimage
> usage, our confidence in layer quality would go up dramatically.
> 

I would like to understand whats stopping YP autobuilders to build
layers under meta-openembedded repo and contribute changes as needed.
All changes with test improvements etc. above are a good change and
should be adopted across layers. However splitting layers is least of
the problem as of now.

> Cheers,
> 
> --Tim
> 

Re: Splitting meta-oe?

[Richard Purdie]

On Tue, 2018-02-20 at 10:40 -0800, Khem Raj wrote:
> On 2/20/18 10:00 AM, Tim Orling wrote:
> > I am open to discussion about what direction we go. Individual
> > layers that
> > are curated and built together by YP auto builders sounds like an
> > intriguing path. If this was coupled with increased ptest or
> > testimage
> > usage, our confidence in layer quality would go up dramatically.
> > 
> I would like to understand whats stopping YP autobuilders to build
> layers under meta-openembedded repo and contribute changes as needed.
> All changes with test improvements etc. above are a good change and
> should be adopted across layers. However splitting layers is least of
> the problem as of now.

The Yocto Project cannot commit to building everything in the meta-oe
repository. There are some specific layers we would consider building
but right now I think there are inter-dependencies that cause problems.

Sure, we can look into those and try and fix them in some way and that
would be one less hurdle. I do appreciate we have autobuilder issues
right now which also causes us problems, I've already committed to
working through those.

Even once we do that, we (as in YP) can't send out a clear message
about what we're testing and users will clone meta-oe and expect
everything to work. So right now I do have problems trying to get to a
point where YP can use meta-oe effectively.

I could combo-layer pieces of meta-oe into poky but I'd imagine that
would create more problems than it would solve too and given the
general dislike of combo-layer, I think ultimately better layer tooling
would be a better answer and more acceptable to everyone.

Cheers,

Richard

Re: Splitting meta-oe?

[Khem Raj]

On 2/20/18 10:52 AM, Richard Purdie wrote:
> On Tue, 2018-02-20 at 10:40 -0800, Khem Raj wrote:
>> On 2/20/18 10:00 AM, Tim Orling wrote:
>>> I am open to discussion about what direction we go. Individual
>>> layers that
>>> are curated and built together by YP auto builders sounds like an
>>> intriguing path. If this was coupled with increased ptest or
>>> testimage
>>> usage, our confidence in layer quality would go up dramatically.
>>>
>> I would like to understand whats stopping YP autobuilders to build
>> layers under meta-openembedded repo and contribute changes as needed.
>> All changes with test improvements etc. above are a good change and
>> should be adopted across layers. However splitting layers is least of
>> the problem as of now.
> 
> The Yocto Project cannot commit to building everything in the meta-oe
> repository. There are some specific layers we would consider building
> but right now I think there are inter-dependencies that cause problems.
> 

1 is better than zero so go for it :) As you see fit, send patches to
subsume the dependencies into other layers and make it meta-oe layer
free, this seems a good step forward

> Sure, we can look into those and try and fix them in some way and that
> would be one less hurdle. I do appreciate we have autobuilder issues
> right now which also causes us problems, I've already committed to
> working through those.
> 
> Even once we do that, we (as in YP) can't send out a clear message
> about what we're testing and users will clone meta-oe and expect
> everything to work. So right now I do have problems trying to get to a
> point where YP can use meta-oe effectively.

poky is a different git repo and it has its own way of subsuming subset
of oe layers ( oe-core bitbake meta-poky ..),  meta-oe is just another
one to deal with fhere. it seems more like a distro problem here to me.

> 
> I could combo-layer pieces of meta-oe into poky but I'd imagine that
> would create more problems than it would solve too and given the
> general dislike of combo-layer, I think ultimately better layer tooling
> would be a better answer and more acceptable to everyone.
> 

I think monolithic nature of poky repo is your problem here. if it was
using layers independently it would have been better.

> Cheers,
> 
> Richard
> 
> 
> 
> 

Re: Splitting meta-oe?

[Richard Purdie]

On Tue, 2018-02-20 at 11:15 -0800, Khem Raj wrote:
> On 2/20/18 10:52 AM, Richard Purdie wrote:
> > 
> > On Tue, 2018-02-20 at 10:40 -0800, Khem Raj wrote:
> > > 
> > > On 2/20/18 10:00 AM, Tim Orling wrote:
> > > > 
> > > > I am open to discussion about what direction we go. Individual
> > > > layers that
> > > > are curated and built together by YP auto builders sounds like
> > > > an
> > > > intriguing path. If this was coupled with increased ptest or
> > > > testimage
> > > > usage, our confidence in layer quality would go up
> > > > dramatically.
> > > > 
> > > I would like to understand whats stopping YP autobuilders to
> > > build
> > > layers under meta-openembedded repo and contribute changes as
> > > needed.
> > > All changes with test improvements etc. above are a good change
> > > and
> > > should be adopted across layers. However splitting layers is
> > > least of
> > > the problem as of now.
> > The Yocto Project cannot commit to building everything in the meta-
> > oe
> > repository. There are some specific layers we would consider
> > building
> > but right now I think there are inter-dependencies that cause
> > problems.
> > 
> 1 is better than zero so go for it :) As you see fit, send patches to
> subsume the dependencies into other layers and make it meta-oe layer
> free, this seems a good step forward

Sure, should I do that before or after I fix the autobuilder, sort the
stable releases and write the layer tool? ;-)

Seriously, I'd love to but I'm probably not going to get there any time
soon personally, much as I wish it were otherwise.

> > Sure, we can look into those and try and fix them in some way and
> > that
> > would be one less hurdle. I do appreciate we have autobuilder
> > issues
> > right now which also causes us problems, I've already committed to
> > working through those.
> > 
> > Even once we do that, we (as in YP) can't send out a clear message
> > about what we're testing and users will clone meta-oe and expect
> > everything to work. So right now I do have problems trying to get
> > to a
> > point where YP can use meta-oe effectively.
> poky is a different git repo and it has its own way of subsuming
> subset of oe layers ( oe-core bitbake meta-poky ..),  meta-oe is just
> another one to deal with fhere. it seems more like a distro problem
> here to me.

Think about this a different way. The message meta-oe is sending to
users is that monolithic repositories of many different layers, lumping
all recipes together in ways which mean they can't be separated out is
"the right way" to do things. In that sense its little progress over
oe-classic!

If our tooling is so bad we can't work with layers, perhaps we should
abandon that idea? Or perhaps we need to fix the tooling and allow
things to be split up?

I agree we have some challenges with testing, with automated upgrades,
with tooling and so on, but I think meta-oe does need to evolve (as do
other parts of the system including things like poky).

Cheers,

Richard

Re: Splitting meta-oe?

[Martin Jansa]

Richard, I agree with all you said.

But I still don't see how replacing meta-oe git repository with 10
different git repositories helps with anything.

It won't give you more time, it would only cause more work to already
exhausted and (re)tired meta-oe maintainers.

It won't make the layers independent.

It won't improve existing tooling.

It won't improve automated upgrades or testing.

Yes meta-oe needs to improve and it would be great to have layers as those
fancy colorful boxes from Yocto documentation, which you can willy-nilly
add to your BBLAYERS without any dependencies. But that's not how it is now
and splitting the repository alone won't help with that. If there are
people willing to make it work better, then they can improve it in existing
git repository and when it works well and there are really independent
layers, then we can re-evaluate splitting the independent ones to separate
repositories for easier consumption.

Regards,




On Tue, Feb 20, 2018 at 10:55 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Tue, 2018-02-20 at 11:15 -0800, Khem Raj wrote:
> > On 2/20/18 10:52 AM, Richard Purdie wrote:
> > >
> > > On Tue, 2018-02-20 at 10:40 -0800, Khem Raj wrote:
> > > >
> > > > On 2/20/18 10:00 AM, Tim Orling wrote:
> > > > >
> > > > > I am open to discussion about what direction we go. Individual
> > > > > layers that
> > > > > are curated and built together by YP auto builders sounds like
> > > > > an
> > > > > intriguing path. If this was coupled with increased ptest or
> > > > > testimage
> > > > > usage, our confidence in layer quality would go up
> > > > > dramatically.
> > > > >
> > > > I would like to understand whats stopping YP autobuilders to
> > > > build
> > > > layers under meta-openembedded repo and contribute changes as
> > > > needed.
> > > > All changes with test improvements etc. above are a good change
> > > > and
> > > > should be adopted across layers. However splitting layers is
> > > > least of
> > > > the problem as of now.
> > > The Yocto Project cannot commit to building everything in the meta-
> > > oe
> > > repository. There are some specific layers we would consider
> > > building
> > > but right now I think there are inter-dependencies that cause
> > > problems.
> > >
> > 1 is better than zero so go for it :) As you see fit, send patches to
> > subsume the dependencies into other layers and make it meta-oe layer
> > free, this seems a good step forward
>
> Sure, should I do that before or after I fix the autobuilder, sort the
> stable releases and write the layer tool? ;-)
>
> Seriously, I'd love to but I'm probably not going to get there any time
> soon personally, much as I wish it were otherwise.
>
> > > Sure, we can look into those and try and fix them in some way and
> > > that
> > > would be one less hurdle. I do appreciate we have autobuilder
> > > issues
> > > right now which also causes us problems, I've already committed to
> > > working through those.
> > >
> > > Even once we do that, we (as in YP) can't send out a clear message
> > > about what we're testing and users will clone meta-oe and expect
> > > everything to work. So right now I do have problems trying to get
> > > to a
> > > point where YP can use meta-oe effectively.
> > poky is a different git repo and it has its own way of subsuming
> > subset of oe layers ( oe-core bitbake meta-poky ..),  meta-oe is just
> > another one to deal with fhere. it seems more like a distro problem
> > here to me.
>
> Think about this a different way. The message meta-oe is sending to
> users is that monolithic repositories of many different layers, lumping
> all recipes together in ways which mean they can't be separated out is
> "the right way" to do things. In that sense its little progress over
> oe-classic!
>
> If our tooling is so bad we can't work with layers, perhaps we should
> abandon that idea? Or perhaps we need to fix the tooling and allow
> things to be split up?
>
> I agree we have some challenges with testing, with automated upgrades,
> with tooling and so on, but I think meta-oe does need to evolve (as do
> other parts of the system including things like poky).
>
> Cheers,
>
> Richard
>
>
> --
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>

Re: Splitting meta-oe?

[Andreas Müller]

On Tue, Feb 20, 2018 at 11:27 PM, Martin Jansa <martin.jansa@gmail.com> wrote:
> Richard, I agree with all you said.
>
> But I still don't see how replacing meta-oe git repository with 10
> different git repositories helps with anything.
>
> It won't give you more time, it would only cause more work to already
> exhausted and (re)tired meta-oe maintainers.
>
> It won't make the layers independent.
>
> It won't improve existing tooling.
>
> It won't improve automated upgrades or testing.
>
> Yes meta-oe needs to improve and it would be great to have layers as those
> fancy colorful boxes from Yocto documentation, which you can willy-nilly
> add to your BBLAYERS without any dependencies. But that's not how it is now
> and splitting the repository alone won't help with that. If there are
> people willing to make it work better, then they can improve it in existing
> git repository and when it works well and there are really independent
> layers, then we can re-evaluate splitting the independent ones to separate
> repositories for easier consumption.
>
> Regards,
>
>
And it won't reduce efforts in sending patches / creating releases
(and this in times where resources are limited).

Reading all this, the only intention I can see is to kick out what
core-boys call bit rotted - regardless if working perfectly fine - and
hoping it will die due to extra efforts. Ah and don't forget to create
a layer oe-core-trashbin.

Andreas

Re: Splitting meta-oe?

[Richard Purdie]

On Tue, 2018-02-20 at 23:27 +0100, Martin Jansa wrote:
> Richard, I agree with all you said.
> 
> But I still don't see how replacing meta-oe git repository with 10
> different git repositories helps with anything.
> 
> It won't give you more time, it would only cause more work to already
> exhausted and (re)tired meta-oe maintainers.

I'm actually very worried about these (re)tired maintainers. If the
layers were more independent it would allow some of the patch handling
responsibilities and testing responsibilities to move to other people,
reducing the load on those maintainers.

Obviously this can't happen with everything all at once but I think
some pieces of the current repo may not be far away or that difficult
to separate out.

Therefore raising the question doesn't seem unreasonable.

The separate layers and maintainership is the way we designed the new
layered approach to OE to scale.

> It won't make the layers independent.

It won't, but it would give people more of a reason to fix it.

> It won't improve existing tooling.
> 
> It won't improve automated upgrades or testing.
> 
> Yes meta-oe needs to improve and it would be great to have layers as
> those fancy colorful boxes from Yocto documentation, which you can
> willy-nilly add to your BBLAYERS without any dependencies. But that's
> not how it is now and splitting the repository alone won't help with
> that. If there are people willing to make it work better, then they
> can improve it in existing git repository and when it works well and
> there are really independent layers, then we can re-evaluate
> splitting the independent ones to separate repositories for easier
> consumption.

It alone will not change things but it *might* just trigger more people
to get involved, help and take some of the workload from those
maintainers who are overloaded right now. Its a chicken and egg
problem, our difference is we disagree on which step might help. We've
tried the monolithic repo for quite a while, it might be time to try a
different approach maybe with one piece of meta-oe?

Cheers,

Richard
[away tomorrow so further replies may be delayed]

Re: Splitting meta-oe?

[Trevor Woerner]

On Tue 2018-02-20 @ 11:41:15 PM, Richard Purdie wrote:
> The separate layers and maintainership is the way we designed the new
> layered approach to OE to scale.

I only became an active user of OE after the OE-classic split; I had used it a
couple times during the OE-classic era, but not very deeply. Therefore: my
apologies for not having a fuller understanding of the issue(s).

Would anyone like to make an honest, unbiased attempt at answering:
1. What problem(s) was the post-OE-classic split attempting to solve?
2. Did it work? Can it be said that the problem(s) the OE-split was attempting
   to solve, have actually been solved by the split? (and, if new problems
   arose as a result of this split, were they small an manageable relative to
   the pre-split problems?)

The experiment to decide whether we need more splits or consolidation has
already been done. Have the results been sufficiently understood?

Re: Splitting meta-oe?

[Philip Balister]

On 03/16/2018 08:50 PM, Trevor Woerner wrote:
> On Tue 2018-02-20 @ 11:41:15 PM, Richard Purdie wrote:
>> The separate layers and maintainership is the way we designed the new
>> layered approach to OE to scale.
> 
> I only became an active user of OE after the OE-classic split; I had used it a
> couple times during the OE-classic era, but not very deeply. Therefore: my
> apologies for not having a fuller understanding of the issue(s).
> 
> Would anyone like to make an honest, unbiased attempt at answering:
> 1. What problem(s) was the post-OE-classic split attempting to solve?

The all in one approach is untestable.

> 2. Did it work? Can it be said that the problem(s) the OE-split was attempting
>    to solve, have actually been solved by the split? (and, if new problems
>    arose as a result of this split, were they small an manageable relative to
>    the pre-split problems?)
> 

OE-core is well tested, and that has taken a lot of resources. There are
not dedicated resources to test much beyond this. The real problem is
how to get resources to test layers beyond oe-core. And maintain testing
of oe-core.

Philip


> The experiment to decide whether we need more splits or consolidation has
> already been done. Have the results been sufficiently understood?
> 

Re: Splitting meta-oe?

[Trevor Woerner]

On Sat 2018-03-17 @ 07:23:52 AM, Philip Balister wrote:
> > Would anyone like to make an honest, unbiased attempt at answering:
> > 1. What problem(s) was the post-OE-classic split attempting to solve?
> 
> The all in one approach is untestable.

Thank you.

I was able to find the old OE-classic[1]:

	$ cd OE-classic
	$ find . -name "*bb" -print | wc -l
	7853

	$ cd meta-openembedded
	$ find . -name "*bb" -print | wc -l
	1477

	$ cd openembedded-core
	$ find . -name "*bb" -print | wc -l
	837

Wow!

> > 2. Did it work? Can it be said that the problem(s) the OE-split was attempting
> >    to solve, have actually been solved by the split? (and, if new problems
> >    arose as a result of this split, were they small an manageable relative to
> >    the pre-split problems?)
> > 
> 
> OE-core is well tested, and that has taken a lot of resources. There are
> not dedicated resources to test much beyond this. The real problem is
> how to get resources to test layers beyond oe-core. And maintain testing
> of oe-core.

It sounds to me as though the goal-posts of what to include in a layer are not
set by what logically hangs together, but rather by how much can be tested. It
sounds like the "capacity that can be tested" is the metric that will be used.
So we should stop making passionate arguments about what should be included
together in a layer, and start talking concretely about resources and capacity.

If we determine that X number of recipes can be tested in a hour, and that we
only want to test for Y number of hours, then no layer should have more than
Y*X recipes.

And if that means that it will now take 30 layers to build anything useful,
then so be it. The problem is, you might be the first and only person to ever
try to put together those specific 30 layers in a given way. So although the
project will be able to say "these 30 layers are tested amazingly well...
individually" it's anyone's guess what you'll end up with when you put all of
them together.

The argument has been that smaller layers can be more easily tested, which
gives us the feeling that the quality is going up. And that might be true, on
an individual layer's basis. But it's also been my experience that as the
number of layers needed to build a specific project go up, the quality tends
to go down (and complication increases).

Why not look to other projects for inspiration? Take the Linux kernel: it's
a huge project! Does anyone promise that every possible combination of
configurations is tested and verified before each release (or that they even
compile cleanly)? Although the parts of the kernel have their own trees and
mailing lists, at the end of the day, it's shipped as one kernel in one
repository.


[1] which isn't easy, thanks to https://www.oeclassic.com/

Re: Splitting meta-oe?

[Otavio Salvador]

On Tue, Feb 20, 2018 at 3:52 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> I could combo-layer pieces of meta-oe into poky but I'd imagine that
> would create more problems than it would solve too and given the
> general dislike of combo-layer, I think ultimately better layer tooling
> would be a better answer and more acceptable to everyone.

Poky creates more problems then it solves
>
> Cheers,
>
> Richard
>
>
>
>
> --
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel



-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Splitting meta-oe?

[Otavio Salvador]

On Tue, Feb 20, 2018 at 9:07 PM, Otavio Salvador
<otavio@ossystems.com.br> wrote:
> On Tue, Feb 20, 2018 at 3:52 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>> I could combo-layer pieces of meta-oe into poky but I'd imagine that
>> would create more problems than it would solve too and given the
>> general dislike of combo-layer, I think ultimately better layer tooling
>> would be a better answer and more acceptable to everyone.
>
> Poky creates more problems then it solves

... send was too soon ...

Poky creates more problems then it solves.

 - it causes confusion
 - it avoids the urgency in adopting a setup script
 - it does not use the layers as we market as being a good thing

So adding more things to it, just makes it worse.

The setup script is more urgent to be discussed then splitting meta-oe.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Splitting meta-oe?

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1450 bytes --]

On Tue, Feb 20, 2018 at 09:10:25PM -0300, Otavio Salvador wrote:
> On Tue, Feb 20, 2018 at 9:07 PM, Otavio Salvador
> <otavio@ossystems.com.br> wrote:
> > On Tue, Feb 20, 2018 at 3:52 PM, Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> >> I could combo-layer pieces of meta-oe into poky but I'd imagine that
> >> would create more problems than it would solve too and given the
> >> general dislike of combo-layer, I think ultimately better layer tooling
> >> would be a better answer and more acceptable to everyone.
> >
> > Poky creates more problems then it solves
> 
> ... send was too soon ...
> 
> Poky creates more problems then it solves.
> 
>  - it causes confusion
>  - it avoids the urgency in adopting a setup script
>  - it does not use the layers as we market as being a good thing
> 
> So adding more things to it, just makes it worse.
> 
> The setup script is more urgent to be discussed then splitting meta-oe.

I agree that a setup script of some sort (off the top of my head,
something that takes layer-names as input, checks vs a list,
fetches/clones, creates a wrapper around bitbake-layers to always add
them) should be a high priority.  I don't have a problem telling my
customers to clone meta-openembedded and then use the layers that are
needed in that specific project.  But it's painful to have a shell
for-loop in the docs we provide so they can setup a build.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Splitting meta-oe?

[Otavio Salvador]

On Wed, Feb 21, 2018 at 10:57 AM, Tom Rini <trini@konsulko.com> wrote:
> On Tue, Feb 20, 2018 at 09:10:25PM -0300, Otavio Salvador wrote:
>> On Tue, Feb 20, 2018 at 9:07 PM, Otavio Salvador
>> <otavio@ossystems.com.br> wrote:
>> > On Tue, Feb 20, 2018 at 3:52 PM, Richard Purdie
>> > <richard.purdie@linuxfoundation.org> wrote:
>> >> I could combo-layer pieces of meta-oe into poky but I'd imagine that
>> >> would create more problems than it would solve too and given the
>> >> general dislike of combo-layer, I think ultimately better layer tooling
>> >> would be a better answer and more acceptable to everyone.
>> >
>> > Poky creates more problems then it solves
>>
>> ... send was too soon ...
>>
>> Poky creates more problems then it solves.
>>
>>  - it causes confusion
>>  - it avoids the urgency in adopting a setup script
>>  - it does not use the layers as we market as being a good thing
>>
>> So adding more things to it, just makes it worse.
>>
>> The setup script is more urgent to be discussed then splitting meta-oe.
>
> I agree that a setup script of some sort (off the top of my head,
> something that takes layer-names as input, checks vs a list,
> fetches/clones, creates a wrapper around bitbake-layers to always add
> them) should be a high priority.  I don't have a problem telling my
> customers to clone meta-openembedded and then use the layers that are
> needed in that specific project.  But it's painful to have a shell
> for-loop in the docs we provide so they can setup a build.

I think we ought to start a thread about the tooling, but let's focus
on meta-oe split here.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Splitting meta-oe?

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1862 bytes --]

On Wed, Feb 21, 2018 at 11:00:27AM -0300, Otavio Salvador wrote:
> On Wed, Feb 21, 2018 at 10:57 AM, Tom Rini <trini@konsulko.com> wrote:
> > On Tue, Feb 20, 2018 at 09:10:25PM -0300, Otavio Salvador wrote:
> >> On Tue, Feb 20, 2018 at 9:07 PM, Otavio Salvador
> >> <otavio@ossystems.com.br> wrote:
> >> > On Tue, Feb 20, 2018 at 3:52 PM, Richard Purdie
> >> > <richard.purdie@linuxfoundation.org> wrote:
> >> >> I could combo-layer pieces of meta-oe into poky but I'd imagine that
> >> >> would create more problems than it would solve too and given the
> >> >> general dislike of combo-layer, I think ultimately better layer tooling
> >> >> would be a better answer and more acceptable to everyone.
> >> >
> >> > Poky creates more problems then it solves
> >>
> >> ... send was too soon ...
> >>
> >> Poky creates more problems then it solves.
> >>
> >>  - it causes confusion
> >>  - it avoids the urgency in adopting a setup script
> >>  - it does not use the layers as we market as being a good thing
> >>
> >> So adding more things to it, just makes it worse.
> >>
> >> The setup script is more urgent to be discussed then splitting meta-oe.
> >
> > I agree that a setup script of some sort (off the top of my head,
> > something that takes layer-names as input, checks vs a list,
> > fetches/clones, creates a wrapper around bitbake-layers to always add
> > them) should be a high priority.  I don't have a problem telling my
> > customers to clone meta-openembedded and then use the layers that are
> > needed in that specific project.  But it's painful to have a shell
> > for-loop in the docs we provide so they can setup a build.
> 
> I think we ought to start a thread about the tooling, but let's focus
> on meta-oe split here.

Agreed.  And to be clear, I'm another vote against splitting meta-oe.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Splitting meta-oe?

[Martin Hundebøll]

On 2018-02-21 14:57, Tom Rini wrote:
> On Tue, Feb 20, 2018 at 09:10:25PM -0300, Otavio Salvador wrote:
>> On Tue, Feb 20, 2018 at 9:07 PM, Otavio Salvador
>> <otavio@ossystems.com.br> wrote:
>>> On Tue, Feb 20, 2018 at 3:52 PM, Richard Purdie
>>> <richard.purdie@linuxfoundation.org> wrote:
>>>> I could combo-layer pieces of meta-oe into poky but I'd imagine that
>>>> would create more problems than it would solve too and given the
>>>> general dislike of combo-layer, I think ultimately better layer tooling
>>>> would be a better answer and more acceptable to everyone.
>>>
>>> Poky creates more problems then it solves
>>
>> ... send was too soon ...
>>
>> Poky creates more problems then it solves.
>>
>>   - it causes confusion
>>   - it avoids the urgency in adopting a setup script
>>   - it does not use the layers as we market as being a good thing
>>
>> So adding more things to it, just makes it worse.
>>
>> The setup script is more urgent to be discussed then splitting meta-oe.
> 
> I agree that a setup script of some sort (off the top of my head,
> something that takes layer-names as input, checks vs a list,
> fetches/clones, creates a wrapper around bitbake-layers to always add
> them) should be a high priority.  I don't have a problem telling my
> customers to clone meta-openembedded and then use the layers that are
> needed in that specific project.  But it's painful to have a shell
> for-loop in the docs we provide so they can setup a build.

Now that the discussion branched out a bit...

We would like better support for this too. Our setup uses a "manifest" 
repository with git submodules to setup the layers:

 > yocto/
 >       meta-poky/
 >       meta-qt5/
 >       meta-foo/
 >       meta-bar/
 >       conf/
 >            bblayers.conf
 >            local.conf
 >       .gitmodules

With this setup, customers simply need to clone our yocto repo 
recursively, run `yocto/meta-poky/oe-init-build-env yocto` and then 
`bitbake image-recipe`.

But this is rather inflexible, as it requires the "yocto" folder to be 
the build folder to activate the config files...

We looked into putting the configs in "meta-foo/conf/*.conf.sample" and 
using TEMPLATECONF, but the "oe-init-build-env" script is rather picky 
about poky being the "top" directory.

I guess the oe-init-build-env script can be changed to look for 
.templateconf in any parent folder?

// Martin

Re: Splitting meta-oe?

[Jonas Bonn]

On 21 February 2018 at 15:09, Martin Hundebøll <mnhu@prevas.dk> wrote:

> Now that the discussion branched out a bit...
>
> We would like better support for this too. Our setup uses a "manifest"
> repository with git submodules to setup the layers:
>
> > yocto/
> >       meta-poky/
> >       meta-qt5/
> >       meta-foo/
> >       meta-bar/
> >       conf/
> >            bblayers.conf
> >            local.conf
> >       .gitmodules
>
> With this setup, customers simply need to clone our yocto repo
> recursively, run `yocto/meta-poky/oe-init-build-env yocto` and then
> `bitbake image-recipe`.
>
> But this is rather inflexible, as it requires the "yocto" folder to be the
> build folder to activate the config files...
>
> We looked into putting the configs in "meta-foo/conf/*.conf.sample" and
> using TEMPLATECONF, but the "oe-init-build-env" script is rather picky
> about poky being the "top" directory.
>
> I guess the oe-init-build-env script can be changed to look for
> .templateconf in any parent folder?


Putting together a deliverable setup that's easy for the customer to get
started with is a bit tricky.  Here's the approach that's worked well for
me:

/myproject
    /env        <-- script
    /build
    /meta-myproject
    /bitbake
    /oe-core
    /meta-layer1
    /meta-layer2

env, build, meta-myproject are part of the myproject repo, everything else
is a submodule.

"env" is a script containing just the following:
. ./oe-core/oe-init-build-env build/ bitbake/

meta-myproject is the "local layer" that contains project specific recipes
and appends.

The build directory contains:
build/conf/bblayers.conf.sample
build/conf/conf-notes.txt
build/conf/local.conf.sample
build/conf/templateconf.cfg

The interesting parts here are:
bblayers.conf.sample contains something like this:
BBLAYERS ?= " \
  ##OEROOT##/meta \
  ##OEROOT##/../meta-layer1 \
  ##OEROOT##/../meta-layer2 \
  ##OEROOT##/../meta-myproject \
  "

and templateconf.cfg contains this magic:
../build/conf

With this setup, the customer does:
git clone ..../myproject.git
git submodule --update init
./env
bitbake customer-image

The 'env' command nicely prints the contents of conf-notes which reminds
the customer of the names of the images (customer-image,
customer-dev-image, customer-special-image, etc) that are avaiable to be
built so that they don't need to search the repo for the image recipes
because they can't remember what they were called.

The only additional tweak that I recommand be made manually beyond the
above is to point build/sstate-cache to somewhere outside of the cloned
repo.  That way, the entire repo can be deleted, re-cloned, and quickly
rebuilt when people get themselves lost in the labyrinth...

Advantages of the above:
i)  oe-core is just another layer and therefore just another submodule...
no changes are made to it locally
ii)  git log on the myproject repo ties submodule updates and modifications
to the "local layer" together so it's clear why recipes were changed
iii)  since meta-mylayer is not an external layer/repo, I'm working in only
one git repo when doing updates (assuming the submodules aren't changing)

Anyway, just wanted to share that, in case it's useful for someone.

/Jonas

Re: Splitting meta-oe?

[Patrick Ohly]

On Thu, 2018-02-22 at 07:53 +0100, Jonas Bonn wrote:
> On 21 February 2018 at 15:09, Martin Hundebøll <mnhu@prevas.dk>
> wrote:
> 
> > Now that the discussion branched out a bit...
> > 
> > We would like better support for this too. Our setup uses a
> > "manifest"
> > repository with git submodules to setup the layers:
> > 
> > > yocto/
> > >       meta-poky/
> > >       meta-qt5/
> > >       meta-foo/
> > >       meta-bar/
> > >       conf/
> > >            bblayers.conf
> > >            local.conf
> > >       .gitmodules
> > 
> > With this setup, customers simply need to clone our yocto repo
> > recursively, run `yocto/meta-poky/oe-init-build-env yocto` and then
> > `bitbake image-recipe`.
> > 
> > But this is rather inflexible, as it requires the "yocto" folder to
> > be the
> > build folder to activate the config files...
> > 
> > We looked into putting the configs in "meta-foo/conf/*.conf.sample" 
> > and
> > using TEMPLATECONF, but the "oe-init-build-env" script is rather
> > picky
> > about poky being the "top" directory.
> > 
> > I guess the oe-init-build-env script can be changed to look for
> > .templateconf in any parent folder?
> 
> 
> Putting together a deliverable setup that's easy for the customer to
> get
> started with is a bit tricky.  Here's the approach that's worked well
> for
> me:
> 
> /myproject
>     /env        <-- script
>     /build
>     /meta-myproject
>     /bitbake
>     /oe-core
>     /meta-layer1
>     /meta-layer2
> 
> env, build, meta-myproject are part of the myproject repo, everything
> else is a submodule.

refkit used the same approach. One thing that I would prefer to do
differently is the location of the submodule: having them in their own
directory would make it more transparent which code is "external" and
which is "internal".

> "env" is a script containing just the following:
> . ./oe-core/oe-init-build-env build/ bitbake/

We ended up with a top-level "oe-init-build-env" wrapper script around
the actual oe-core/oe-init-build-env. That way the repo could be used
the same way as poky. The script sets TEMPLATECONF, so the usual local
build setup happens based on refkit sample files.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Splitting meta-oe?

[Otavio Salvador]

On Thu, Feb 22, 2018 at 6:27 AM, Patrick Ohly <patrick.ohly@intel.com> wrote:
> On Thu, 2018-02-22 at 07:53 +0100, Jonas Bonn wrote:
>> On 21 February 2018 at 15:09, Martin Hundebøll <mnhu@prevas.dk>
>> wrote:
>>
>> > Now that the discussion branched out a bit...
>> >
>> > We would like better support for this too. Our setup uses a
>> > "manifest"
>> > repository with git submodules to setup the layers:
>> >
>> > > yocto/
>> > >       meta-poky/
>> > >       meta-qt5/
>> > >       meta-foo/
>> > >       meta-bar/
>> > >       conf/
>> > >            bblayers.conf
>> > >            local.conf
>> > >       .gitmodules
>> >
>> > With this setup, customers simply need to clone our yocto repo
>> > recursively, run `yocto/meta-poky/oe-init-build-env yocto` and then
>> > `bitbake image-recipe`.
>> >
>> > But this is rather inflexible, as it requires the "yocto" folder to
>> > be the
>> > build folder to activate the config files...
>> >
>> > We looked into putting the configs in "meta-foo/conf/*.conf.sample"
>> > and
>> > using TEMPLATECONF, but the "oe-init-build-env" script is rather
>> > picky
>> > about poky being the "top" directory.
>> >
>> > I guess the oe-init-build-env script can be changed to look for
>> > .templateconf in any parent folder?
>>
>>
>> Putting together a deliverable setup that's easy for the customer to
>> get
>> started with is a bit tricky.  Here's the approach that's worked well
>> for
>> me:
>>
>> /myproject
>>     /env        <-- script
>>     /build
>>     /meta-myproject
>>     /bitbake
>>     /oe-core
>>     /meta-layer1
>>     /meta-layer2
>>
>> env, build, meta-myproject are part of the myproject repo, everything
>> else is a submodule.
>
> refkit used the same approach. One thing that I would prefer to do
> differently is the location of the submodule: having them in their own
> directory would make it more transparent which code is "external" and
> which is "internal".
>
>> "env" is a script containing just the following:
>> . ./oe-core/oe-init-build-env build/ bitbake/
>
> We ended up with a top-level "oe-init-build-env" wrapper script around
> the actual oe-core/oe-init-build-env. That way the repo could be used
> the same way as poky. The script sets TEMPLATECONF, so the usual local
> build setup happens based on refkit sample files.

We have a script set and a document which describes how we do it:

http://doc.ossystems.com.br/managing-platforms.html

The script sources can be seen at:

https://code.ossystems.com.br/gitweb?p=ossystems-yocto-base-scripts.git

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: Splitting meta-oe?

[Patrick Ohly]

On Tue, 2018-02-20 at 18:52 +0000, Richard Purdie wrote:
> Even once we do that, we (as in YP) can't send out a clear message
> about what we're testing and users will clone meta-oe and expect
> everything to work. So right now I do have problems trying to get to
> a point where YP can use meta-oe effectively.

We had the same issue in refkit: the bblayers.conf.sample enabled a
large amount of layers, but the distro itself only needed and could
test only a subset of the recipes in those layers.

We solved this with supported-recipes.bbclass [1] and an explicit list
of recipes that were considered part of the distro [2] and thus got
tested. A "bitbake world" only builds those recipes. Users of the
distro could enable additional recipes, but then knew that they were on
their own regarding those.

[1] https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit-core/classes/supported-recipes.bbclass
[2] https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit/conf/distro/include/refkit-supported-recipes.txt

Note that this mechanism also allowed us to support only a subset of,
for example, OE-core: we settled on systemd as the only supported init
system, so sysvinit wasn't listed as supported. This is something that
cannot realistically be achieved by splitting up layers and/or repos
containing layers.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Splitting meta-oe?

[akuster808]

On 02/20/2018 10:40 AM, Khem Raj wrote:
> On 2/20/18 10:00 AM, Tim Orling wrote:
>> The recent improvements to the Auto Upgrade Helper have made maintenance of
>> meta-perl less effort and therefore you have seen an uptick in my updates
>> to recipes. I also plan to mass-add ptest to all perl module recipes
>> "sometime soon", which will further simplify my upgrade workflow and add to
>> the confidence of said recipes. My upgrade workflow touches all recipes in
>> oe-core, meta-oe and meta-perl that fit a lib*perl regex.
>>
>> I will note, however, that we are revisiting meta-cpan and that may
>> ultimately be the preferred perl layer. If you really heavily on perl
>> modules, please comment on whether meta-cpan would fit your needs.
>>
>> Derek and others have been doing a great job keeping meta-python up to
>> date. I mostly jump in when there are trickier failures that need more
>> concentrated effort. I would like to see increased testing (more packages
>> with ptest), but that is trickier with python recipes as there are many
>> more approaches to testing than there are in perl. I am happy with the
>> current efforts to keep recipes up to date as a priority. I would like to
>> see complete coverage of python3 for all recipes that are capable in
>> meta-python.
>>
>> I am open to discussion about what direction we go. Individual layers that
>> are curated and built together by YP auto builders sounds like an
>> intriguing path. If this was coupled with increased ptest or testimage
>> usage, our confidence in layer quality would go up dramatically.
>>
> I would like to understand whats stopping YP autobuilders to build
> layers under meta-openembedded repo and contribute changes as needed.
> All changes with test improvements etc. above are a good change and
> should be adopted across layers. 
It tends to be the same Yocto Project members carrying the brunt of the
work load for that work.  Its the same issue meta-openembedded suffers
from.. Resources..

- Armin
> However splitting layers is least of
> the problem as of now.
>
>> Cheers,
>>
>> --Tim
>>

Re: Splitting meta-oe?

[Khem Raj]

On 2/20/18 9:46 AM, Richard Purdie wrote:
> Repositories have reputations. OE-Core is fairly heavily curated and
> tested and has certain "guarantees" about what people can expect to
> work. The meta-oe repo is a little trickier as the different pieces do
> have different 'SLA's (service level agreements). Some pieces like
> meta-networking/meta-python likely build well, other pieces like some
> of the bitrotting pieces of the meta-oe layer probably don't build in
> as many different configurations/architectures and on balance may be
> more likely to hit issues.
> 
> I think the world may need to change a bit. We should probably change:
> 
> * The way people get started with the Yocto Project and Poky
>   to move away from the combo repo and into specific layers.
> * to have YP testing more layers as part of its builds and testing
> * to make it easier for people to establish an SLA type reputation for 
>   a layer.
> 
> Part of this is a need for a more universal "using OE" later setup
> tooling. I've wanted to work on that for a while and I will get there
> but if I spend time on it, I want to get it right. I've not had enough
> time to get it right (as yet).
> 
> Part of this means redefining poky, the YP autobuilders and so on.
> Pieces of the autobuilder work are in process and will lead to wider
> layer testing.
> 
> I have strong influence over the above so I can likely make those
> happen, time constraints aside. One piece I need the help of others
> with is meta-oe (the repo).
> 
> Even just splitting meta-oe out from meta-oe might be enough to
> establish the SLA differences, I don't know.

this is short term panacea, few years later we will again talk about
meta-oe in same sense because it would have caught the packages which
belong to no other specific layers and cant be in oe-core

unless there is a solution for meta-oe layer, it seems to be like we are
just changing maintainer control structure for repos.

> 
> There is a question about what problem would this solve. The move from
> OE-Classic was partly about defining maintainership and processes for
> recipes which we do with layers. The pieces in the separate layers have
> now become the things we set out to create but the catchall of meta-oe
> (the layer) remains. I do think we should still be aiming to filter
> meta-oe into distinct layers with distinct maintainership. There will
> always be some "leftovers" in a catchall but I think it does have a
> different personality to the other well separated components and we
> need to show to users that they do have different expectations from
> them.

in practical downstream projects for real products, we include almost
all layers, so from that perspective it just is another hassle for
integrating.

> 
> I hope I'm managing to convey this concept ok, I'm really trying to
> take a step back here and think what will help OE users and the project
> and where we need to go with things. I'm not saying we should/shouldn't
> do this for 100% certain, I do want people to think about the
> alternatives though and what options may be possible.
> 

I dont disagee here. but it would be nicer if we had a restructuring
plan for layers within repo

> Cheers,
> 
> Richard
> 
> 
> 

Re: Splitting meta-oe?

[Vesa Jääskeläinen]

On 20/02/2018 12.45, Burton, Ross wrote:
> Hi,
>
> Is now a good time to talk about splitting up meta-oe?  Some layers are
> actively developed and maintained (one example: meta-python), others are
> basically bitrotting and only get touched when something else causes them
> to break world builds (one example: meta-gnome).  I've long felt that
> meta-oe should be split up and the high quality layers managed in their own
> repositories so patches to them don't get held up by breakage in other
> sub-layers.
>
> Another advantage of splitting out the high quality layers is that we'd
> like to look at running more community layers through the Yocto
> autobuilder, and granular layers make that easier to manage.
>
> Comments?
>
> Ross
Other related problem to meta layer management is that should there be 
recommended scheme for meta layer priority usage. Now they are a bit 
randomly used in different layers as there is no guidance what priority 
should be used for what purpose.

Eg. what should in-house layers use so that they fit properly in scheme 
and does not conflict with layer X that can be found from github.

Eg. we had to once re-base our layer priorities due to conflicts in 
priorities.

Thanks,
Vesa Jääskeläinen

Re: Splitting meta-oe?

[Khem Raj]

On 2/20/18 2:45 AM, Burton, Ross wrote:
> Hi,
> 
> Is now a good time to talk about splitting up meta-oe?  Some layers are
> actively developed and maintained (one example: meta-python), others are
> basically bitrotting and only get touched when something else causes them
> to break world builds (one example: meta-gnome).  I've long felt that
> meta-oe should be split up and the high quality layers managed in their own
> repositories so patches to them don't get held up by breakage in other
> sub-layers.

lets improve them and add them to YP, you might mask the ones you don't
like, feed the results back to community and lets see what then
maintainers think about split etc. IMO split is least of issues. patches
get held up for oe-core too because they break meta-gpl2 :) its just
nature of stuff.

split should not be take out these layers because some folks are now
interested in them and leave others, because that will create another
meta-oe in few years. If split has to happen it has to happen by
completely retiring meta-openembedded.


> 
> Another advantage of splitting out the high quality layers is that we'd
> like to look at running more community layers through the Yocto
> autobuilder, and granular layers make that easier to manage.

why do we have to split layers for them to be usable with YP
autobuilder, I fail to understand.

> 
> Comments?
> 
> Ross
> 

Re: Splitting meta-oe?

[Alexander Kanavin]

On 02/20/2018 12:45 PM, Burton, Ross wrote:
> Is now a good time to talk about splitting up meta-oe?  Some layers are
> actively developed and maintained (one example: meta-python), others are
> basically bitrotting and only get touched when something else causes them
> to break world builds (one example: meta-gnome).  I've long felt that
> meta-oe should be split up and the high quality layers managed in their own
> repositories so patches to them don't get held up by breakage in other
> sub-layers.
> 
> Another advantage of splitting out the high quality layers is that we'd
> like to look at running more community layers through the Yocto
> autobuilder, and granular layers make that easier to manage.
> 
> Comments?

I just read the whole discussion (been on holiday), and while it seems 
that splitting layers is seen as too heavy-handed, there is still 
something that I believe should be done: a policy for blacklisting and 
removing unmaintained recipes. Such a policy should be:

a) clearly defined;
b) consistently applied.

If that is in place, I, as an oe-core maintainer, would be a lot more 
willing to contribute to meta-oe, knowing that recipes I contribute to 
(for example by fixing issues caused by changes in oe-core) are, in 
fact, used and taken care of otherwise. However, me endlessly fixing 
well obsolete gnome2 stuff is just a fast track to not caring anymore.

So, which recipes are unmaintained?

1. Badly out of date compared to upstream development. Say, one year or 
more between version provided by meta-oe master, and latest version 
released by upstream.
2. Recipes which fail to build, and the situation hasn't been addressed, 
in, say, six months.

Once either of these is established, the recipe enters a grace period 
before it is removed. Any objection to such removal should come with a 
patch that addresses the reason for it.


Alex

Re: Splitting meta-oe?

[Andreas Müller]

On Wed, Feb 28, 2018 at 6:17 PM, Alexander Kanavin
<alexander.kanavin@linux.intel.com> wrote:
> On 02/20/2018 12:45 PM, Burton, Ross wrote:
>>
>> Is now a good time to talk about splitting up meta-oe?  Some layers are
>> actively developed and maintained (one example: meta-python), others are
>> basically bitrotting and only get touched when something else causes them
>> to break world builds (one example: meta-gnome).  I've long felt that
>> meta-oe should be split up and the high quality layers managed in their
>> own
>> repositories so patches to them don't get held up by breakage in other
>> sub-layers.
>>
>> Another advantage of splitting out the high quality layers is that we'd
>> like to look at running more community layers through the Yocto
>> autobuilder, and granular layers make that easier to manage.
>>
>> Comments?
>
>
> I just read the whole discussion (been on holiday), and while it seems that
> splitting layers is seen as too heavy-handed, there is still something that
> I believe should be done: a policy for blacklisting and removing
> unmaintained recipes. Such a policy should be:
>
> a) clearly defined;
> b) consistently applied.
>
> If that is in place, I, as an oe-core maintainer, would be a lot more
> willing to contribute to meta-oe, knowing that recipes I contribute to (for
> example by fixing issues caused by changes in oe-core) are, in fact, used
> and taken care of otherwise. However, me endlessly fixing well obsolete
> gnome2 stuff is just a fast track to not caring anymore.
Isn't there somebody outside willing/capable/having enough time to
move to gnome 3? This would be the best way to end the gnome2 bit-rott
discussion and we'd have a desktop which is commonly used and
addresses touch input. I tried that many years ago but the blocker at
that time was gobject-introspection. These times are over for long. I
wanted to start this during christmas-holiday but then my
music-machine turned into more efforts than expected... For those
still wanting what's left over from gnome2 currently (for me gedit - I
don't like gedit3 search..) there is at least the mate-project an
alternative.
>
> So, which recipes are unmaintained?
>
> 1. Badly out of date compared to upstream development. Say, one year or more
> between version provided by meta-oe master, and latest version released by
> upstream.
For example: What if something changes upstream that makes it very
difficult or impossible for us to follow? Have no recent example for
that but I think if gobject-introspection would have worked for us
years ago, gnome2 would not be an issue anymore. You mention below
that in this case a patch has to be sent out. A comment in the recipe
would be good enough?
> 2. Recipes which fail to build, and the situation hasn't been addressed, in,
> say, six months.
Yes: recipes not building are useless. Martin has done blacklisting in
the late phase of his maintainer-ship. The only question I have here:
Does not build mean for all archs or for just .. how many?
>
> Once either of these is established, the recipe enters a grace period before
> it is removed. Any objection to such removal should come with a patch that
> addresses the reason for it.
>
I remember the times of mass blacklisting (if I remember correct last
was recipe-specific-sysroot) this was

* kind of alarm for me for those recipes I need / take care of
* easy grep'able for those recipes I thought 'why not - I have some
spare cycles left over'

So my opinion:
* Recipes not building for a time to be defined: 1. blacklist and
after x month -> remove
* Recipes outdated: same 1.blacklist (undo if somebody complains with
patch commenting in the recipe why not to remove) 2. remove after a
time period to be defined

Problem/Challenge:
The way this was handled in the past was an extra duty put on
maintainer's shoulders. If somebody could create (or is there
something already?) a script for that and append it to world builds...

Andreas

Re: Splitting meta-oe?

[akuster808]

On 02/28/2018 01:33 PM, Andreas Müller wrote:
> On Wed, Feb 28, 2018 at 6:17 PM, Alexander Kanavin
> <alexander.kanavin@linux.intel.com> wrote:
>> On 02/20/2018 12:45 PM, Burton, Ross wrote:
>>> Is now a good time to talk about splitting up meta-oe?  Some layers are
>>> actively developed and maintained (one example: meta-python), others are
>>> basically bitrotting and only get touched when something else causes them
>>> to break world builds (one example: meta-gnome).  I've long felt that
>>> meta-oe should be split up and the high quality layers managed in their
>>> own
>>> repositories so patches to them don't get held up by breakage in other
>>> sub-layers.
>>>
>>> Another advantage of splitting out the high quality layers is that we'd
>>> like to look at running more community layers through the Yocto
>>> autobuilder, and granular layers make that easier to manage.
>>>
>>> Comments?
>>
>> I just read the whole discussion (been on holiday), and while it seems that
>> splitting layers is seen as too heavy-handed, there is still something that
>> I believe should be done: a policy for blacklisting and removing
>> unmaintained recipes. Such a policy should be:
>>
>> a) clearly defined;
>> b) consistently applied.
>>
>> If that is in place, I, as an oe-core maintainer, would be a lot more
>> willing to contribute to meta-oe, knowing that recipes I contribute to (for
>> example by fixing issues caused by changes in oe-core) are, in fact, used
>> and taken care of otherwise. However, me endlessly fixing well obsolete
>> gnome2 stuff is just a fast track to not caring anymore.
> Isn't there somebody outside willing/capable/having enough time to
> move to gnome 3? This would be the best way to end the gnome2 bit-rott
> discussion and we'd have a desktop which is commonly used and
> addresses touch input. I tried that many years ago but the blocker at
> that time was gobject-introspection. These times are over for long. I
> wanted to start this during christmas-holiday but then my
> music-machine turned into more efforts than expected... For those
> still wanting what's left over from gnome2 currently (for me gedit - I
> don't like gedit3 search..) there is at least the mate-project an
> alternative.
>> So, which recipes are unmaintained?
>>
>> 1. Badly out of date compared to upstream development. Say, one year or more
>> between version provided by meta-oe master, and latest version released by
>> upstream.
> For example: What if something changes upstream that makes it very
> difficult or impossible for us to follow? Have no recent example for
> that but I think if gobject-introspection would have worked for us
> years ago, gnome2 would not be an issue anymore. You mention below
> that in this case a patch has to be sent out. A comment in the recipe
> would be good enough?
>> 2. Recipes which fail to build, and the situation hasn't been addressed, in,
>> say, six months.
> Yes: recipes not building are useless. Martin has done blacklisting in
> the late phase of his maintainer-ship. The only question I have here:
> Does not build mean for all archs or for just .. how many?

See the "State of the world" emails. It included the Arches and packages
failing.

- armin
>> Once either of these is established, the recipe enters a grace period before
>> it is removed. Any objection to such removal should come with a patch that
>> addresses the reason for it.
>>
> I remember the times of mass blacklisting (if I remember correct last
> was recipe-specific-sysroot) this was
>
> * kind of alarm for me for those recipes I need / take care of
> * easy grep'able for those recipes I thought 'why not - I have some
> spare cycles left over'
>
> So my opinion:
> * Recipes not building for a time to be defined: 1. blacklist and
> after x month -> remove
> * Recipes outdated: same 1.blacklist (undo if somebody complains with
> patch commenting in the recipe why not to remove) 2. remove after a
> time period to be defined
>
> Problem/Challenge:
> The way this was handled in the past was an extra duty put on
> maintainer's shoulders. If somebody could create (or is there
> something already?) a script for that and append it to world builds...
>
> Andreas

Re: Splitting meta-oe?

[Alexander Kanavin]

On 02/28/2018 11:33 PM, Andreas Müller wrote:
> Isn't there somebody outside willing/capable/having enough time to
> move to gnome 3? This would be the best way to end the gnome2 bit-rott
> discussion and we'd have a desktop which is commonly used and
> addresses touch input. I tried that many years ago but the blocker at
> that time was gobject-introspection. These times are over for long. I
> wanted to start this during christmas-holiday but then my
> music-machine turned into more efforts than expected... For those
> still wanting what's left over from gnome2 currently (for me gedit - I
> don't like gedit3 search..) there is at least the mate-project an
> alternative.

Looks like no one is interested enough in gnome 3 to actually package 
it. It is not a small undertaking. I'd say you need a company with a 
product to sell, and in that space, Qt has won.

If you find time to look into it, please do, but I'd say gnome2 stuff 
just needs to disappear, regardless of whether it is replaced by modern 
gnome.

> For example: What if something changes upstream that makes it very
> difficult or impossible for us to follow? Have no recent example for
> that but I think if gobject-introspection would have worked for us
> years ago, gnome2 would not be an issue anymore. You mention below
> that in this case a patch has to be sent out. A comment in the recipe
> would be good enough?

Yes, absolutely. In fact there is an established practice in oe-core for 
this: RECIPE_NO_UPDATE_REASON, which also makes auto-upgrade-helper skip 
the recipe. Latest case where it was used is librsvg, which is being 
rewritten in Rust. So if upstream adds unworkable dependencies or 
serious architectural issues, it's totally fine to freeze the recipe 
using that variable, with a clear reason. But do note that the reason 
cannot be "the upgrade will break some dependent layer/recipe, or some 
specific product".


>> 2. Recipes which fail to build, and the situation hasn't been addressed, in,
>> say, six months.
> Yes: recipes not building are useless. Martin has done blacklisting in
> the late phase of his maintainer-ship. The only question I have here:
> Does not build mean for all archs or for just .. how many?

All I guess? It is not reasonable to remove a recipe because it is 
failing in specific cases, but works generally. In oe-core there is a 
stricter criteria: if a patch causes even one autobuilder arch to fail, 
it is rejected.

Alex

Re: Splitting meta-oe?

[akuster808]

On 02/28/2018 09:17 AM, Alexander Kanavin wrote:
> On 02/20/2018 12:45 PM, Burton, Ross wrote:
>> Is now a good time to talk about splitting up meta-oe?  Some layers are
>> actively developed and maintained (one example: meta-python), others are
>> basically bitrotting and only get touched when something else causes
>> them
>> to break world builds (one example: meta-gnome).  I've long felt that
>> meta-oe should be split up and the high quality layers managed in
>> their own
>> repositories so patches to them don't get held up by breakage in other
>> sub-layers.
>>
>> Another advantage of splitting out the high quality layers is that we'd
>> like to look at running more community layers through the Yocto
>> autobuilder, and granular layers make that easier to manage.
>>
>> Comments?
>
> I just read the whole discussion (been on holiday), and while it seems
> that splitting layers is seen as too heavy-handed, there is still
> something that I believe should be done: a policy for blacklisting and
> removing unmaintained recipes. Such a policy should be:
>
> a) clearly defined;
> b) consistently applied.

Thank for the input. Its more of a mater of someone willing to sit down
and documenting it. Having that said, people tend not to follow process
or even read them. I thought Martin sent out an email regarding this
Blacklisting process.

>
> If that is in place, I, as an oe-core maintainer, would be a lot more
> willing to contribute to meta-oe, knowing that recipes I contribute to
> (for example by fixing issues caused by changes in oe-core) are, in
> fact, used and taken care of otherwise. However, me endlessly fixing
> well obsolete gnome2 stuff is just a fast track to not caring anymore.

We had/have a situation with the Yocto 4.12 kernel that broke wireguard
in meta-networking. Their are two patches that don't exist in K.O. which
are causing the problem.
Meta-openembedded can't fix that, do I blacklist wireguard? Wireguard
builds fine with K.O 4.12.

We are at the mercy of oe-core; kernel, toolchain and other core
changes. Most of the failures now are do to the 4.15 kernel update.

>
> So, which recipes are unmaintained?
>
> 1. Badly out of date compared to upstream development. Say, one year
> or more between version provided by meta-oe master, and latest version
> released by upstream.
> 2. Recipes which fail to build, and the situation hasn't been
> addressed, in, say, six months.
>
> Once either of these is established, the recipe enters a grace period
> before it is removed. Any objection to such removal should come with a
> patch that addresses the reason for it.

Thanks for your input.

regards,
Armin
>
>
> Alex

Re: Splitting meta-oe?

[Alexander Kanavin]

On 03/01/2018 03:17 AM, akuster808 wrote:
> We had/have a situation with the Yocto 4.12 kernel that broke wireguard
> in meta-networking. Their are two patches that don't exist in K.O. which
> are causing the problem.
> Meta-openembedded can't fix that, do I blacklist wireguard? Wireguard
> builds fine with K.O 4.12.

I'm not sure what is K.O., but if the breakage can be resolved by 
changing things in oe-core, then by all means do let us know.

Alex

Re: Splitting meta-oe?

[akuster808]

On 03/01/2018 01:04 AM, Alexander Kanavin wrote:
> On 03/01/2018 03:17 AM, akuster808 wrote:
>> We had/have a situation with the Yocto 4.12 kernel that broke wireguard
>> in meta-networking. Their are two patches that don't exist in K.O. which
>> are causing the problem.
>> Meta-openembedded can't fix that, do I blacklist wireguard? Wireguard
>> builds fine with K.O 4.12.
>
> I'm not sure what is K.O., 
Kernel.org. Upstream linux-4.12.y branch in this case.

> but if the breakage can be resolved by changing things in oe-core,
> then by all means do let us know.

These two commits introduced an ABI change. 

http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-4.12/commit/include/linux/random.h?h=standard/base&id=6e4990d8d226e1861a5a2b632cf93bc70feab3af
http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-4.12/commit/include/linux/random.h?h=standard/base&id=20543a488913755be70f54be52b32f727e8179d8

This makes linux-kernel_4.12 different than linux 4.12.y.

- Armin
>
> Alex

Re: Splitting meta-oe?

[Alexander Kanavin]

On 03/01/2018 08:44 PM, akuster808 wrote:
> 
> These two commits introduced an ABI change.
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-4.12/commit/include/linux/random.h?h=standard/base&id=6e4990d8d226e1861a5a2b632cf93bc70feab3af
> http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-4.12/commit/include/linux/random.h?h=standard/base&id=20543a488913755be70f54be52b32f727e8179d8
> 
> This makes linux-kernel_4.12 different than linux 4.12.y.

This should probably be taken to the linux-yocto mailing list?

Alex

Re: Splitting meta-oe?

[Bruce Ashfield]

On Thu, Mar 1, 2018 at 1:44 PM, akuster808 <akuster808@gmail.com> wrote:
>
>
> On 03/01/2018 01:04 AM, Alexander Kanavin wrote:
>> On 03/01/2018 03:17 AM, akuster808 wrote:
>>> We had/have a situation with the Yocto 4.12 kernel that broke wireguard
>>> in meta-networking. Their are two patches that don't exist in K.O. which
>>> are causing the problem.
>>> Meta-openembedded can't fix that, do I blacklist wireguard? Wireguard
>>> builds fine with K.O 4.12.
>>
>> I'm not sure what is K.O.,
> Kernel.org. Upstream linux-4.12.y branch in this case.
>
>> but if the breakage can be resolved by changing things in oe-core,
>> then by all means do let us know.
>
> These two commits introduced an ABI change.
>
> http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-4.12/commit/include/linux/random.h?h=standard/base&id=6e4990d8d226e1861a5a2b632cf93bc70feab3af
> http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-4.12/commit/include/linux/random.h?h=standard/base&id=20543a488913755be70f54be52b32f727e8179d8
>
> This makes linux-kernel_4.12 different than linux 4.12.y.

To be clear, the work that PaulG is doing is the korg 4.12.y. He's
doing -stable off the end of where Greg EOL'd it, so it doesn't differ
.. it continues on from where the other left off.

Those commits are from the 4.x -stable queue from Greg's other
kernels, so they are consistent with the other stable trees and
mainline kernel.

But yah, if you have questions on the process, email Paul, cc' the
linux-yocto list and it can get sorted out. Paul documents everything
he does in a patch queue, a tree and in his pull requests, so the info
is all floating around.

Cheers,

Bruce

>
> - Armin
>>
>> Alex
>
>
> --
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Splitting meta-oe?

[Burton, Ross]

The recent storm of breakage in meta-oe caused by recipe specific sysroots
was quite dramatic:

ross@flashheart ~/Yocto/meta-oe ((044e518...))
$ git grep PNBLACKLIST| wc -l
320

Is it time to talk about splitting meta-oe into smaller real repositories
so they can be maintained at their own pace by more maintainers?

Ross

Re: Splitting meta-oe?

[Andreas Müller]

On Fri, Feb 17, 2017 at 6:07 PM, Burton, Ross <ross.burton@intel.com> wrote:
> The recent storm of breakage in meta-oe caused by recipe specific sysroots
> was quite dramatic:
>
> ross@flashheart ~/Yocto/meta-oe ((044e518...))
> $ git grep PNBLACKLIST| wc -l
> 320
>
> Is it time to talk about splitting meta-oe into smaller real repositories
> so they can be maintained at their own pace by more maintainers?
>
> Ross
What exactly gets better by splitting?

Splitting = Rotting to death due to unmanageable maintenance burden
caused by oe? Don't misunderstand me recipe specific sysroot is
basically a good thing. But I am tired of this game. As you can see in
meta-qt5-extra: I am interested in having cool images not in a cool
build system. Long live core-image-sato!

Andreas

Re: Splitting meta-oe?

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]

On Fri, Feb 17, 2017 at 06:24:09PM +0100, Andreas Müller wrote:
> On Fri, Feb 17, 2017 at 6:07 PM, Burton, Ross <ross.burton@intel.com> wrote:
> > The recent storm of breakage in meta-oe caused by recipe specific sysroots
> > was quite dramatic:
> >
> > ross@flashheart ~/Yocto/meta-oe ((044e518...))
> > $ git grep PNBLACKLIST| wc -l
> > 320
> >
> > Is it time to talk about splitting meta-oe into smaller real repositories
> > so they can be maintained at their own pace by more maintainers?
> >
> > Ross
> What exactly gets better by splitting?

I agree with Andreas.

1) RSS is good thing.

2) The breakage wasn't caused by lack of maintainers (at least I don't
   think that I or Joe were the bottleneck for integrating the fixes).

3) More maintainers doesn't mean more contributions from people actually
   using now broken components, it's actually easier to just send a fix
   than to be a maintainer of some layer just to be able to also merge
   your fix yourself.

4) It doesn't look so dramatic if it turns out that 200 of those
   blacklisted recipes weren't actually used by anyone still active in
   OE ecosystem.

5) If someone wants to replace me as meta-oe maintainer, go ahead, it
   stopped being fun for me long time ago, now it's just slightly annoying
   routine which takes my free time I would rather invest in something
   cooler

Regards,

> Splitting = Rotting to death due to unmanageable maintenance burden
> caused by oe? Don't misunderstand me recipe specific sysroot is
> basically a good thing. But I am tired of this game. As you can see in
> meta-qt5-extra: I am interested in having cool images not in a cool
> build system. Long live core-image-sato!
> 
> Andreas
> -- 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: Splitting meta-oe?

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 3351 bytes --]

[Re: [oe] Splitting meta-oe?] On 17.02.17 (Fri 19:02) Martin Jansa wrote:

> On Fri, Feb 17, 2017 at 06:24:09PM +0100, Andreas Müller wrote:
> > On Fri, Feb 17, 2017 at 6:07 PM, Burton, Ross <ross.burton@intel.com> wrote:
> > > The recent storm of breakage in meta-oe caused by recipe specific sysroots
> > > was quite dramatic:
> > >
> > > ross@flashheart ~/Yocto/meta-oe ((044e518...))
> > > $ git grep PNBLACKLIST| wc -l
> > > 320
> > >
> > > Is it time to talk about splitting meta-oe into smaller real repositories
> > > so they can be maintained at their own pace by more maintainers?
> > >
> > > Ross
> > What exactly gets better by splitting?
> 
> I agree with Andreas.

And I agree with both of you.

> 1) RSS is good thing.
> 
> 2) The breakage wasn't caused by lack of maintainers (at least I don't
>    think that I or Joe were the bottleneck for integrating the fixes).

My schedule is highly variable but I do try to be responsive to
breakages and I do my best to watch and digest the state of the world
messages, those are extremely valuable IMO.

> 3) More maintainers doesn't mean more contributions from people actually
>    using now broken components, it's actually easier to just send a fix
>    than to be a maintainer of some layer just to be able to also merge
>    your fix yourself.
> 
> 4) It doesn't look so dramatic if it turns out that 200 of those
>    blacklisted recipes weren't actually used by anyone still active in
>    OE ecosystem.

Agreed.  And if those blacklisted recipes are something that someone
in the ecosystem cares about, they really should be sending fixes back
upstream.  Breaking meta-oe up into smaller parts doesn't seem likely to
make them more likely to send patches, AFAICT.  Probably won't make them
less-so, either, but I don't see changing the structure to be a
significant win.

And do note that when we first created meta-networking I was proposing
it be separate from meta-oe, I'm now completely on the other side of
that argument, for what that's worth.

> 5) If someone wants to replace me as meta-oe maintainer, go ahead, it
>    stopped being fun for me long time ago, now it's just slightly annoying
>    routine which takes my free time I would rather invest in something
>    cooler

I frankly think that'd be a loss for everyone, but it's understandable.
That's a big, largely thankless job you've taken on.  Obviously I can't
offer to take on more of the job than I already have with
meta-networking or I would, but maybe someone else with a similarly "big
picture" view will be able to share some of the workload.

-J.

> 
> Regards,
> 
> > Splitting = Rotting to death due to unmanageable maintenance burden
> > caused by oe? Don't misunderstand me recipe specific sysroot is
> > basically a good thing. But I am tired of this game. As you can see in
> > meta-qt5-extra: I am interested in having cool images not in a cool
> > build system. Long live core-image-sato!
> > 
> > Andreas
> > -- 
> > _______________________________________________
> > Openembedded-devel mailing list
> > Openembedded-devel@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-devel
> 
> -- 
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com



-- 
-Joe MacDonald.
:wq

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 484 bytes --]

Re: Splitting meta-oe?

[Philip Balister]

[-- Attachment #1.1: Type: text/plain, Size: 4218 bytes --]

On 02/17/2017 01:28 PM, Joe MacDonald wrote:
> [Re: [oe] Splitting meta-oe?] On 17.02.17 (Fri 19:02) Martin Jansa wrote:
> 
>> On Fri, Feb 17, 2017 at 06:24:09PM +0100, Andreas Müller wrote:
>>> On Fri, Feb 17, 2017 at 6:07 PM, Burton, Ross <ross.burton@intel.com> wrote:
>>>> The recent storm of breakage in meta-oe caused by recipe specific sysroots
>>>> was quite dramatic:
>>>>
>>>> ross@flashheart ~/Yocto/meta-oe ((044e518...))
>>>> $ git grep PNBLACKLIST| wc -l
>>>> 320
>>>>
>>>> Is it time to talk about splitting meta-oe into smaller real repositories
>>>> so they can be maintained at their own pace by more maintainers?
>>>>
>>>> Ross
>>> What exactly gets better by splitting?
>>
>> I agree with Andreas.
> 
> And I agree with both of you.
> 
>> 1) RSS is good thing.
>>
>> 2) The breakage wasn't caused by lack of maintainers (at least I don't
>>    think that I or Joe were the bottleneck for integrating the fixes).
> 
> My schedule is highly variable but I do try to be responsive to
> breakages and I do my best to watch and digest the state of the world
> messages, those are extremely valuable IMO.
> 
>> 3) More maintainers doesn't mean more contributions from people actually
>>    using now broken components, it's actually easier to just send a fix
>>    than to be a maintainer of some layer just to be able to also merge
>>    your fix yourself.
>>
>> 4) It doesn't look so dramatic if it turns out that 200 of those
>>    blacklisted recipes weren't actually used by anyone still active in
>>    OE ecosystem.
> 
> Agreed.  And if those blacklisted recipes are something that someone
> in the ecosystem cares about, they really should be sending fixes back
> upstream.  Breaking meta-oe up into smaller parts doesn't seem likely to
> make them more likely to send patches, AFAICT.  Probably won't make them
> less-so, either, but I don't see changing the structure to be a
> significant win.
> 
> And do note that when we first created meta-networking I was proposing
> it be separate from meta-oe, I'm now completely on the other side of
> that argument, for what that's worth.
> 
>> 5) If someone wants to replace me as meta-oe maintainer, go ahead, it
>>    stopped being fun for me long time ago, now it's just slightly annoying
>>    routine which takes my free time I would rather invest in something
>>    cooler
> 
> I frankly think that'd be a loss for everyone, but it's understandable.
> That's a big, largely thankless job you've taken on.  Obviously I can't
> offer to take on more of the job than I already have with
> meta-networking or I would, but maybe someone else with a similarly "big
> picture" view will be able to share some of the workload.

And I'm with these gyus. Splitting the git repository doesn't solve any
underlying problems. The real problem from my point of view is very few
of use are actually paid to maintain the layers we maintain.

Employers want to pay things they profit from, and that is not paying
someone to maintain "core infrastructure".

Layer maintainers interests change over time, and you burn out
supporting people who get to do all the cool stuff with the layers you
maintain. In the end, you get all the crap and non of the glory. Within
this list, most people appreciate your work. Outside the community,
people completely underestimate the amount of work required to keep the
ecosystem running.

Yeah, add my name to the list of cranky people.

Philip

> 
> -J.
> 
>>
>> Regards,
>>
>>> Splitting = Rotting to death due to unmanageable maintenance burden
>>> caused by oe? Don't misunderstand me recipe specific sysroot is
>>> basically a good thing. But I am tired of this game. As you can see in
>>> meta-qt5-extra: I am interested in having cool images not in a cool
>>> build system. Long live core-image-sato!
>>>
>>> Andreas
>>> -- 
>>> _______________________________________________
>>> Openembedded-devel mailing list
>>> Openembedded-devel@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>>
>> -- 
>> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com
> 
> 
> 
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 514 bytes --]

Re: Splitting meta-oe?

[Richard Purdie]

On Fri, 2017-02-17 at 14:45 -0500, Philip Balister wrote:
> And I'm with these gyus. Splitting the git repository doesn't solve
> any underlying problems. The real problem from my point of view is
> very few of use are actually paid to maintain the layers we maintain.
> 
> Employers want to pay things they profit from, and that is not paying
> someone to maintain "core infrastructure".
> 
> Layer maintainers interests change over time, and you burn out
> supporting people who get to do all the cool stuff with the layers
> you maintain. In the end, you get all the crap and non of the glory.
> Within this list, most people appreciate your work. Outside the
> community, people completely underestimate the amount of work
> required to keep the ecosystem running.
> 
> Yeah, add my name to the list of cranky people.

I do think this is a valid question that Ross asks and that whilst the
first quick reaction is "no", its worth thinking about the pros/cons.

The pros to me would be about better test time on patches and in theory
more specialist knowledge. This isn't to say Martin/Joe don't do a bad
job but the size of meta-oe does mean there are limits.

The cons are more around finding suitable layer maintainers, which as
we all know are hard to find.

I'd probably suggest that:

a) We need to encourage/empower more people to maintain layers
b) Having better infrastructure, tools and processes that help a) would
   therefore be desirable.
c) We need to be willing to separate out pieces for people to maintain
   in such layers. It might not always work out but we should be 
   willing to try.

As for the comments about core changes, I really do try hard not to
make them in many ways. The ones we do make, I'd hope are for the right
reasons.

No easy answers but don't shoot Ross for asking what I think is a
reasonable question.

Cheers,

Richard

Re: Splitting meta-oe?

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 4851 bytes --]

[Re: [oe] Splitting meta-oe?] On 17.02.19 (Sun 19:31) Richard Purdie wrote:

> On Fri, 2017-02-17 at 14:45 -0500, Philip Balister wrote:
> > And I'm with these gyus. Splitting the git repository doesn't solve
> > any underlying problems. The real problem from my point of view is
> > very few of use are actually paid to maintain the layers we maintain.
> > 
> > Employers want to pay things they profit from, and that is not paying
> > someone to maintain "core infrastructure".
> > 
> > Layer maintainers interests change over time, and you burn out
> > supporting people who get to do all the cool stuff with the layers
> > you maintain. In the end, you get all the crap and non of the glory.
> > Within this list, most people appreciate your work. Outside the
> > community, people completely underestimate the amount of work
> > required to keep the ecosystem running.
> > 
> > Yeah, add my name to the list of cranky people.
> 
> I do think this is a valid question that Ross asks and that whilst the
> first quick reaction is "no", its worth thinking about the pros/cons.

Absolutely, and that's why I brought up earlier that my initial proposal
for meta-networking was that it be a separate tree from the rest of
meta-oe.  When I first looked at this and what my initial goals were for
meta-networking, it made sense to me to be at arms' length.  Still does,
to be honest, but being several years into the project now I find myself
with a different mindset on the costs/benefits.

> The pros to me would be about better test time on patches and in
> theory more specialist knowledge. This isn't to say Martin/Joe don't
> do a bad job but the size of meta-oe does mean there are limits.

This is the thing that I'd like to get more of your point of view on,
because it doesn't seem clear to me.  Today, a quick survey of meta-oe
shows we have 14 top-level layers in meta-oe (yeah, does seem like a
lot) and of those I maintain one (sharing ownership with Armin when it
comes to anything netkit-related, as noted in the MAINTAINERS file) of
the others only meta-gnome seems to have only Martin listed as a
maintainer (though a few, for example gpe, multimedia and oe, have only
Martin and Koen, so there's certainly a case to be made there for there
being a larger burden on them).

Maybe we're talking specifically about meta-openembedded/meta-oe, in
which case I apologize for derailing the discussion.  I certain tried to
help that situation with creating meta-networking and moving some of the
stuff in meta-oe out into the new layer when we created it, but meta-oe
still is a really big layer (664 recipes if 'find' can be trusted), no
question.

> The cons are more around finding suitable layer maintainers, which as
> we all know are hard to find.
> 
> I'd probably suggest that:
> 
> a) We need to encourage/empower more people to maintain layers

Yes, without question.

> b) Having better infrastructure, tools and processes that help a) would
>    therefore be desirable.

The updated patchwork has seriously made my life a lot easier.  And I
don't want to go off too much on an tangent, but I'd really love to
bring meta-selinux into that same fold.

> c) We need to be willing to separate out pieces for people to maintain
>    in such layers. It might not always work out but we should be 
>    willing to try.

This is the part I'm still struggling with.  For good or ill
meta-networking got created, I've tried to maintain it reasonably
carefully and along the way the only thing that stands out for me as an
issue while remaining part of the overall meta-openembedded project was
a brief hiccup following the hardware upgrade at the end of December
where I lost the ability to push commits.  But since Martin hadn't, I
just asked him to merge the meta-networking queue I was trying to merge
and that's all sorted out now.

I certainly don't object to having another meta-something in the overall
meta-openembedded project and I don't object to having additional hands
to carry the workload, but I'm not clear how separate git repositories
addresses the current challenge.

> As for the comments about core changes, I really do try hard not to
> make them in many ways. The ones we do make, I'd hope are for the right
> reasons.
> 
> No easy answers but don't shoot Ross for asking what I think is a
> reasonable question.

I really, honestly did not mean for anything I said to sound like that.
It's good to have this kind of discussion from time to time.  It's very
easy to get stuck in the "this is the way it's always been done" rut, so
I hope Ross didn't feel like anyone was shouting him down.

Wish I was going to be around Wilsonville this week, I'd like to talk
more about this with you all.

-- 
-Joe MacDonald.
:wq

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 484 bytes --]

Re: Splitting meta-oe?

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2982 bytes --]

On Sun, Feb 19, 2017 at 07:31:03PM -0800, Richard Purdie wrote:
> On Fri, 2017-02-17 at 14:45 -0500, Philip Balister wrote:
> > And I'm with these gyus. Splitting the git repository doesn't solve
> > any underlying problems. The real problem from my point of view is
> > very few of use are actually paid to maintain the layers we maintain.
> > 
> > Employers want to pay things they profit from, and that is not paying
> > someone to maintain "core infrastructure".
> > 
> > Layer maintainers interests change over time, and you burn out
> > supporting people who get to do all the cool stuff with the layers
> > you maintain. In the end, you get all the crap and non of the glory.
> > Within this list, most people appreciate your work. Outside the
> > community, people completely underestimate the amount of work
> > required to keep the ecosystem running.
> > 
> > Yeah, add my name to the list of cranky people.
> 
> I do think this is a valid question that Ross asks and that whilst the
> first quick reaction is "no", its worth thinking about the pros/cons.
> 
> The pros to me would be about better test time on patches and in theory
> more specialist knowledge. This isn't to say Martin/Joe don't do a bad
> job but the size of meta-oe does mean there are limits.

If I continue to do the same "bitbake world" builds to test as many
layers as possible, then the test time will be exactly the same even if
we split meta-oe repository into 10 smaller repositories, probably a bit
longer for fetching all those small repos.

> The cons are more around finding suitable layer maintainers, which as
> we all know are hard to find.
> 
> I'd probably suggest that:
> 
> a) We need to encourage/empower more people to maintain layers

I think we lack people willing to contribute patches for recipes they
use, not people willing to merge them into corresponding repository.

> b) Having better infrastructure, tools and processes that help a) would
>    therefore be desirable.
> c) We need to be willing to separate out pieces for people to maintain
>    in such layers. It might not always work out but we should be 
>    willing to try.
> 
> As for the comments about core changes, I really do try hard not to
> make them in many ways. The ones we do make, I'd hope are for the right
> reasons.

Yes everybody agreed that RSS is good change and worth breaking unused
recipes.

> No easy answers but don't shoot Ross for asking what I think is a
> reasonable question.

I wasn't trying to shoot him, but I still don't see how more
repositories solve the issue of unused recipes and lack of people
contributing to fix those still in use somewhere.

And I still think it's easier to send a patch to fix something instead
of volunteering to be maintainer of the layer with the one recipe you're
interested in fixing to merge your fix yourself.

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: Splitting meta-oe?

[akuster808]

On 02/20/2017 03:18 AM, Martin Jansa wrote:
> On Sun, Feb 19, 2017 at 07:31:03PM -0800, Richard Purdie wrote:
>> On Fri, 2017-02-17 at 14:45 -0500, Philip Balister wrote:
>>> And I'm with these gyus. Splitting the git repository doesn't solve
>>> any underlying problems. The real problem from my point of view is
>>> very few of use are actually paid to maintain the layers we maintain.
>>>
>>> Employers want to pay things they profit from, and that is not paying
>>> someone to maintain "core infrastructure".
>>>
>>> Layer maintainers interests change over time, and you burn out
>>> supporting people who get to do all the cool stuff with the layers
>>> you maintain. In the end, you get all the crap and non of the glory.
>>> Within this list, most people appreciate your work. Outside the
>>> community, people completely underestimate the amount of work
>>> required to keep the ecosystem running.
>>>
>>> Yeah, add my name to the list of cranky people.
>> I do think this is a valid question that Ross asks and that whilst the
>> first quick reaction is "no", its worth thinking about the pros/cons.
>>
>> The pros to me would be about better test time on patches and in theory
>> more specialist knowledge. This isn't to say Martin/Joe don't do a bad
>> job but the size of meta-oe does mean there are limits.
> If I continue to do the same "bitbake world" builds to test as many
> layers as possible, then the test time will be exactly the same even if
> we split meta-oe repository into 10 smaller repositories, probably a bit
> longer for fetching all those small repos.
>
>> The cons are more around finding suitable layer maintainers, which as
>> we all know are hard to find.
>>
>> I'd probably suggest that:
>>
>> a) We need to encourage/empower more people to maintain layers
> I think we lack people willing to contribute patches for recipes they
> use, not people willing to merge them into corresponding repository.

Correct. This does not solve the underlining issue.

We do need to think what to do about black listed recipes. Suspect its a 
separate thread.
>
>> b) Having better infrastructure, tools and processes that help a) would
>>     therefore be desirable.
Are you saying we have to make it easier for people to participate?

Please Donate to OE so we can improve the infrastructure.

If you split out each layer, each maintainer will have there own process 
that works best for them.

>> c) We need to be willing to separate out pieces for people to maintain
>>     in such layers. It might not always work out but we should be
>>     willing to try.
so a layer per recipe would be the best in this case. This wont bring in 
more help.

>>
>> As for the comments about core changes, I really do try hard not to
>> make them in many ways. The ones we do make, I'd hope are for the right
>> reasons.
> Yes everybody agreed that RSS is good change and worth breaking unused
> recipes.
>
>> No easy answers but don't shoot Ross for asking what I think is a
>> reasonable question.
> I wasn't trying to shoot him, but I still don't see how more
> repositories solve the issue of unused recipes and lack of people
> contributing to fix those still in use somewhere.

There are other implications to think about after the break-up.

1) What happens to meta-openembedded repo and or layer?
2) Do we adopt the K.O model where the sub-layers are maintained 
independently and then merge to meta-openemedded ( this could offload 
work and improve quality).
3) How do we sync branching? if we do #2, that is easier.
4) Since the maintainers will have write permissions for their own 
layers, does this mean we no longer have overall architects (a.k.a 
Martin & Koen)
5) How do we ensure the same level of support master currently benefits 
from Martin's world if those responsibilities transfer to other layer 
maintainers?
6) How do we enforce dup recipes?

If we decide to stop having a meta-openembedded layer;
7) How long do you keep it around?
8) Will the stable branches prior to the breakup only be in the aging 
meta-openembedded layer and not new scheme?
9) Do we have a meta-openembedded-classic?



>
> And I still think it's easier to send a patch to fix something instead
> of volunteering to be maintainer of the layer with the one recipe you're
> interested in fixing to merge your fix yourself.
Agree

- armin
>
> Cheers,
>
>
>

Re: Splitting meta-oe?

[Andreas Müller]

On Fri, Feb 17, 2017 at 7:28 PM, Joe MacDonald <Joe_MacDonald@mentor.com> wrote:
> [Re: [oe] Splitting meta-oe?] On 17.02.17 (Fri 19:02) Martin Jansa wrote:
>
>> On Fri, Feb 17, 2017 at 06:24:09PM +0100, Andreas Müller wrote:
>> > On Fri, Feb 17, 2017 at 6:07 PM, Burton, Ross <ross.burton@intel.com> wrote:
>> > > The recent storm of breakage in meta-oe caused by recipe specific sysroots
>> > > was quite dramatic:
>> > >
>> > > ross@flashheart ~/Yocto/meta-oe ((044e518...))
>> > > $ git grep PNBLACKLIST| wc -l
>> > > 320
>> > >
>> > > Is it time to talk about splitting meta-oe into smaller real repositories
>> > > so they can be maintained at their own pace by more maintainers?
>> > >
>> > > Ross
>> > What exactly gets better by splitting?
>>
>> I agree with Andreas.
>
> And I agree with both of you.
>
>> 1) RSS is good thing.
>>
>> 2) The breakage wasn't caused by lack of maintainers (at least I don't
>>    think that I or Joe were the bottleneck for integrating the fixes).
>
> My schedule is highly variable but I do try to be responsive to
> breakages and I do my best to watch and digest the state of the world
> messages, those are extremely valuable IMO.
>
>> 3) More maintainers doesn't mean more contributions from people actually
>>    using now broken components, it's actually easier to just send a fix
>>    than to be a maintainer of some layer just to be able to also merge
>>    your fix yourself.
>>
>> 4) It doesn't look so dramatic if it turns out that 200 of those
>>    blacklisted recipes weren't actually used by anyone still active in
>>    OE ecosystem.
>
> Agreed.  And if those blacklisted recipes are something that someone
> in the ecosystem cares about, they really should be sending fixes back
> upstream.  Breaking meta-oe up into smaller parts doesn't seem likely to
> make them more likely to send patches, AFAICT.  Probably won't make them
> less-so, either, but I don't see changing the structure to be a
> significant win.
>
> And do note that when we first created meta-networking I was proposing
> it be separate from meta-oe, I'm now completely on the other side of
> that argument, for what that's worth.
>
>> 5) If someone wants to replace me as meta-oe maintainer, go ahead, it
>>    stopped being fun for me long time ago, now it's just slightly annoying
>>    routine which takes my free time I would rather invest in something
>>    cooler
>
> I frankly think that'd be a loss for everyone, but it's understandable.
> That's a big, largely thankless job you've taken on.  Obviously I can't
> offer to take on more of the job than I already have with
> meta-networking or I would, but maybe someone else with a similarly "big
> picture" view will be able to share some of the workload.
Agreed. Martin you are doing a great job and without you projects
would break into 'private' layers. This cannot be the target. Coming
to blacklisting: This is the only way for you to force people start
fixing things. Nobody expects you fixing things out of your interest.

From my personal perspective as 'maintainer' of e.g
meta-xfce/meta-office/meta-qt5-extra:

* there is only very limited community - I take care for these layers
more or less myself (ok meta-xfce has seen more than the others)

* due to many oddities as
    * Cmake is cross mess
    * Qt-people write small C++ tools for each and every build-job
    * libreoffice has more or less tailored build system and builds take ages
  caused me to use dirty hacks. Most of those are terribly broken now
- I need to rethink a lot and that takes time.

Coming back to Ross' suggestion: Why panic and break things apart? As
soon as somebody needs an environment requiring current oe-core and
blacklisted recipes in meta-openembedded that person will (hopefully)
send patches. If that lasts - so be it. If there is some pressing for
release cycle - feel free to send patches.

One final note on splitting - has anybody taken a look on layer index
in recent past? I think that shows what will happen if we split.
People come 'yeah I have a great layer', publish that in layer index
and leave after a while. How shall a newbie get into that?

Andreas

Re: Splitting meta-oe?

[Burton, Ross]

On 17 February 2017 at 18:02, Martin Jansa <martin.jansa@gmail.com> wrote:

> 1) RSS is good thing.
>
> 2) The breakage wasn't caused by lack of maintainers (at least I don't
>    think that I or Joe were the bottleneck for integrating the fixes).
>
> 3) More maintainers doesn't mean more contributions from people actually
>    using now broken components, it's actually easier to just send a fix
>    than to be a maintainer of some layer just to be able to also merge
>    your fix yourself.
>
> 4) It doesn't look so dramatic if it turns out that 200 of those
>    blacklisted recipes weren't actually used by anyone still active in
>    OE ecosystem.
>
> 5) If someone wants to replace me as meta-oe maintainer, go ahead, it
>    stopped being fun for me long time ago, now it's just slightly annoying
>    routine which takes my free time I would rather invest in something
>    cooler
>

I'll probably reply to this properly when I'm not rushing between various
things, but I want to make it clear that I think the work you (JaMa) has
been doing on meta-oe is beyond legendary and I've no problem at all with
that.  I just wonder if it would be easier to deal with stale layers that
nobody is actually using if they were separate repositories instead of
being part of the meta-oe umbrella.

But they're not my layers so I have no say, and obviously everyone else
disagrees! :)

Ross

Re: Splitting meta-oe?

[akuster808]

On 02/17/2017 09:07 AM, Burton, Ross wrote:
> The recent storm of breakage in meta-oe caused by recipe specific sysroots
> was quite dramatic:
>
> ross@flashheart ~/Yocto/meta-oe ((044e518...))
> $ git grep PNBLACKLIST| wc -l
> 320

If I recall correctly we gave Martin the thumbs up to use "blacklisting" 
to  get to a point where master builds cleanly and not expect the 
meta-openbedded layer maintainer's to do that work. The hope was if a 
person needed a blacklisted recipe, they would jump in a correct the 
issue. I think that has worked to some success.
> Is it time to talk about splitting meta-oe into smaller real repositories
> so they can be maintained at their own pace by more maintainers?
If I am not mistaken, each sub meta layer has its own maintainer, if 
someone want to be a package maintainer, go for it.  I would prefer that 
"We" the community help the maintainers.

- armin
>
> Ross

Re: Splitting meta-oe?

[Khem Raj]

On Fri, Feb 17, 2017 at 9:07 AM, Burton, Ross <ross.burton@intel.com> wrote:
> The recent storm of breakage in meta-oe caused by recipe specific sysroots
> was quite dramatic:
>
> ross@flashheart ~/Yocto/meta-oe ((044e518...))
> $ git grep PNBLACKLIST| wc -l
> 320
>
> Is it time to talk about splitting meta-oe into smaller real repositories
> so they can be maintained at their own pace by more maintainers?
>

I think thats a fine idea, however, it needs maintainers to sign up
first before any of this can be sustained. I think it was forewarned
that such a change is coming. recipe maintainers or users should have
risen up and sent patches. This can not be solved even if you divide
the layer into smaller layers. Problems still remains. Whats required
is to look other way, where we have
faster infrastructure to test meta-openemebedded layers and ease out
maintainer's job by testing and working though the changes. Blacklist
is infact the better way to disable the recipes instead of shuffling
layers. One still has a good base to start and fix it.