LAYERSERIES

LAYERSERIES

[Richard Purdie]

I finally found 5 mins to sit and look at where I'd gotten to with the
LAYERSERIES variable handling. I sent out discussion about this a while
ago but as a reminder:

We have a problem where people expect master branch of meta-X to work
with OE-Core master. If meta-X hasn't been maintained for a long time,
this may give all kinds of weird errors. This creates a large support
burden and means we struggle to spot unmaintained/un-updated layers.
We'd like to fix this.

In answer to this I added LAYERSERIES_CORENAMES to OE-Core and strongly
suggest layers then set:

LAYERSERIES_COMPAT_xxx = "sumo"

in their layer.conf where xxx is the collection name (used elsewhere in
layer.conf) to indicate which versions of OE-Core they're expected to
work against.

The code already exists to validate this with a fatal error for
mismatches. What was missing was:

a) to print a warning if LAYERSERIES_COMPAT_xxx is unset. I've a patch 
   queued for bitbake to add this.
b) add a check to yocto-check-layer and require this to be set for YP 
   Compatible v2. I've added
   https://bugzilla.yoctoproject.org/show_bug.cgi?id=12661
   to track this.

Yes, its late in the release and the warnings will be annoying but I
think its worth fixing this now so I'm pushing ahead to resolve this.

Cheers,

Richard

Re: [Openembedded-architecture] LAYERSERIES

[Trevor Woerner]

A) Some layers only switch to an official branch name when the find a reason
to. E.g. branch "sumo" is created on openembedded-core but meta-A
keeps working on master unless an incompatible change is created in
openembedded-core that forces meta-A to create a "sumo" branch.

B) Other layers create official branches the moment they exist. E.g. branch
"sumo" is created on openembedded-core and meta-B instantly creates a "sumo"
branch to mark this point in time, and continues working on master. If
meta-B's "sumo" branch fails to build against openembedded-core's "sumo"
branch because an incompatible change is made to openembedded-core's sumo
branch, meta-B fixes the issue on the sumo branch.


I can see how the change you've implemented will be very useful for the A)
cases. Will it be needed for the B) cases? In other words, does the code
you're adding implicitly assume:

	LAYERSERIES_COMPAT_<...> = "layer"

for any given "layer"?

Re: [Openembedded-architecture] LAYERSERIES

[Richard Purdie]

On Fri, 2018-04-06 at 10:16 -0400, Trevor Woerner wrote:
> A) Some layers only switch to an official branch name when the find a
> reason to. E.g. branch "sumo" is created on openembedded-core but
> meta-A keeps working on master unless an incompatible change is
> created in openembedded-core that forces meta-A to create a "sumo"
> branch.
> 
> B) Other layers create official branches the moment they exist. E.g.
> branch "sumo" is created on openembedded-core and meta-B instantly
> creates a "sumo" branch to mark this point in time, and continues
> working on master. If meta-B's "sumo" branch fails to build against
> openembedded-core's "sumo" branch because an incompatible change is
> made to openembedded-core's sumo branch, meta-B fixes the issue on
> the sumo branch.
> 
> 
> I can see how the change you've implemented will be very useful for
> the A)
> cases. Will it be needed for the B) cases? In other words, does the
> code
> you're adding implicitly assume:
> 
> 	LAYERSERIES_COMPAT_<...> = "layer"
> 
> for any given "layer"?

No, there is no implicit assumption.

In both A) and B) cases the maintainer adds the new "codename" to the
list of compatible layer series in LAYERSERIES_COMPAT_<layer> for their
layer. They can list multiple layers in there, e.g. "rocko sumo".

The one annoying thing about all this is the layer maintainers do have
to update layer.conf each time a new release codename comes out. I
think that is a reasonable compromise to be able to give users a much
better idea of which layers are compatible or incomaptible with their
setup though. It means someone has looked at it and believes it will
work with a given release series.

Just to be clear, "layer" would never be a valid value there, it will
always be the release/branch codenames.

Cheers,

Richard

Re: [Openembedded-architecture] LAYERSERIES

[Scott Rifenbark]

[-- Attachment #1: Type: text/plain, Size: 2593 bytes --]

We have this variable appearing in a couple manuals BTW.  If anyone has any
feedback on what is being said here I can update.

Thanks,
Scott

https://www.yoctoproject.org/docs/2.5/ref-manual/ref-manual.html#var-LAYERSERIES_COMPAT
https://www.yoctoproject.org/docs/2.5/dev-manual/dev-manual.html#creating-your-own-layer



On Fri, Apr 6, 2018 at 8:29 AM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Fri, 2018-04-06 at 10:16 -0400, Trevor Woerner wrote:
> > A) Some layers only switch to an official branch name when the find a
> > reason to. E.g. branch "sumo" is created on openembedded-core but
> > meta-A keeps working on master unless an incompatible change is
> > created in openembedded-core that forces meta-A to create a "sumo"
> > branch.
> >
> > B) Other layers create official branches the moment they exist. E.g.
> > branch "sumo" is created on openembedded-core and meta-B instantly
> > creates a "sumo" branch to mark this point in time, and continues
> > working on master. If meta-B's "sumo" branch fails to build against
> > openembedded-core's "sumo" branch because an incompatible change is
> > made to openembedded-core's sumo branch, meta-B fixes the issue on
> > the sumo branch.
> >
> >
> > I can see how the change you've implemented will be very useful for
> > the A)
> > cases. Will it be needed for the B) cases? In other words, does the
> > code
> > you're adding implicitly assume:
> >
> >       LAYERSERIES_COMPAT_<...> = "layer"
> >
> > for any given "layer"?
>
> No, there is no implicit assumption.
>
> In both A) and B) cases the maintainer adds the new "codename" to the
> list of compatible layer series in LAYERSERIES_COMPAT_<layer> for their
> layer. They can list multiple layers in there, e.g. "rocko sumo".
>
> The one annoying thing about all this is the layer maintainers do have
> to update layer.conf each time a new release codename comes out. I
> think that is a reasonable compromise to be able to give users a much
> better idea of which layers are compatible or incomaptible with their
> setup though. It means someone has looked at it and believes it will
> work with a given release series.
>
> Just to be clear, "layer" would never be a valid value there, it will
> always be the release/branch codenames.
>
> Cheers,
>
> Richard
>
>
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
>

[-- Attachment #2: Type: text/html, Size: 3749 bytes --]

Re: [Openembedded-architecture] LAYERSERIES

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3774 bytes --]

FWIW: I've updated some layers with this LAYERSERIES_COMPAT_<layer> today
and I think it's good idea.

It probably won't be enough to stop questions like this:
http://lists.openembedded.org/pipermail/openembedded-core/2018-April/149696.html

but it's still better to show warning first and then some often
harder-to-read exception or error log.

Updating layer.conf in meta-qt5 was a bit more tricky as people tend to mix
the meta-qt5 branches with different branches of other layers, just because
their requirements for Qt are often very different than for other layers
(e.g. many people still using meta-qt5/krogoth with Qt 5.6 because of the
license, even with much newer morty, pyro, rocko, sumo, master branches of
other layers and vice versa that someone is stuck with krogoth release for
whatever reason, but needs newer LTS Qt 5.9 from meta-qt5/rocko). So for
meta-qt5 I've used multiple values for LAYERSERIES_COMPAT_<layer> for
combinations I use somewhere (so that I cover at least some testing with
such uncommon combination) and I'm willing to accept more values to it if
someone really uses another combination regularly. The rest of combination
(like meta-qt5/krogoth with dizzy, which is usable as described in meta-qt5
commits which introduced incompatiblitites, will eventually trigger the
warning which is fine, because people should be aware that they are doing
something less tested and that they might to implement some work arounds to
make it usable).

Regards,

On Fri, Apr 6, 2018 at 5:29 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Fri, 2018-04-06 at 10:16 -0400, Trevor Woerner wrote:
> > A) Some layers only switch to an official branch name when the find a
> > reason to. E.g. branch "sumo" is created on openembedded-core but
> > meta-A keeps working on master unless an incompatible change is
> > created in openembedded-core that forces meta-A to create a "sumo"
> > branch.
> >
> > B) Other layers create official branches the moment they exist. E.g.
> > branch "sumo" is created on openembedded-core and meta-B instantly
> > creates a "sumo" branch to mark this point in time, and continues
> > working on master. If meta-B's "sumo" branch fails to build against
> > openembedded-core's "sumo" branch because an incompatible change is
> > made to openembedded-core's sumo branch, meta-B fixes the issue on
> > the sumo branch.
> >
> >
> > I can see how the change you've implemented will be very useful for
> > the A)
> > cases. Will it be needed for the B) cases? In other words, does the
> > code
> > you're adding implicitly assume:
> >
> >       LAYERSERIES_COMPAT_<...> = "layer"
> >
> > for any given "layer"?
>
> No, there is no implicit assumption.
>
> In both A) and B) cases the maintainer adds the new "codename" to the
> list of compatible layer series in LAYERSERIES_COMPAT_<layer> for their
> layer. They can list multiple layers in there, e.g. "rocko sumo".
>
> The one annoying thing about all this is the layer maintainers do have
> to update layer.conf each time a new release codename comes out. I
> think that is a reasonable compromise to be able to give users a much
> better idea of which layers are compatible or incomaptible with their
> setup though. It means someone has looked at it and believes it will
> work with a given release series.
>
> Just to be clear, "layer" would never be a valid value there, it will
> always be the release/branch codenames.
>
> Cheers,
>
> Richard
>
>
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
>

[-- Attachment #2: Type: text/html, Size: 7978 bytes --]

Re: [Openembedded-architecture] LAYERSERIES

[Denys Dmytriyenko]

I understand the need to reduce the support burden. But there are some valid 
cases when you want to use components/layers from a different set of releases. 

It can sometimes be an older component in otherwise newer setup (i.e. one 
layer hasn't been updated yet from rocko, but it still works fine with 
everything else from sumo).

Alternatively, sometimes you need to bring a newer version of the component to 
your older release - as long as it works for me, I don't want this to be 
artificially restricted.

In other words, the intention is good, but it needs to be either more flexible 
or allow overriding this from distro config or local.conf...

-- 
Denys


On Fri, Apr 06, 2018 at 11:37:01AM +0100, Richard Purdie wrote:
> I finally found 5 mins to sit and look at where I'd gotten to with the
> LAYERSERIES variable handling. I sent out discussion about this a while
> ago but as a reminder:
> 
> We have a problem where people expect master branch of meta-X to work
> with OE-Core master. If meta-X hasn't been maintained for a long time,
> this may give all kinds of weird errors. This creates a large support
> burden and means we struggle to spot unmaintained/un-updated layers.
> We'd like to fix this.
> 
> In answer to this I added LAYERSERIES_CORENAMES to OE-Core and strongly
> suggest layers then set:
> 
> LAYERSERIES_COMPAT_xxx = "sumo"
> 
> in their layer.conf where xxx is the collection name (used elsewhere in
> layer.conf) to indicate which versions of OE-Core they're expected to
> work against.
> 
> The code already exists to validate this with a fatal error for
> mismatches. What was missing was:
> 
> a) to print a warning if LAYERSERIES_COMPAT_xxx is unset. I've a patch 
>    queued for bitbake to add this.
> b) add a check to yocto-check-layer and require this to be set for YP 
>    Compatible v2. I've added
>    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12661
>    to track this.
> 
> Yes, its late in the release and the warnings will be annoying but I
> think its worth fixing this now so I'm pushing ahead to resolve this.
> 
> Cheers,
> 
> Richard
> 
> 
> 
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture

Re: [Openembedded-architecture] LAYERSERIES

[akuster808]

Shouldn't Master get all these changes first then we switch to stable
branch when we have a new release?

My German genes are raging about "Rules" ; )

- armin


On 04/06/2018 08:29 AM, Richard Purdie wrote:
> On Fri, 2018-04-06 at 10:16 -0400, Trevor Woerner wrote:
>> A) Some layers only switch to an official branch name when the find a
>> reason to. E.g. branch "sumo" is created on openembedded-core but
>> meta-A keeps working on master unless an incompatible change is
>> created in openembedded-core that forces meta-A to create a "sumo"
>> branch.
>>
>> B) Other layers create official branches the moment they exist. E.g.
>> branch "sumo" is created on openembedded-core and meta-B instantly
>> creates a "sumo" branch to mark this point in time, and continues
>> working on master. If meta-B's "sumo" branch fails to build against
>> openembedded-core's "sumo" branch because an incompatible change is
>> made to openembedded-core's sumo branch, meta-B fixes the issue on
>> the sumo branch.
>>
>>
>> I can see how the change you've implemented will be very useful for
>> the A)
>> cases. Will it be needed for the B) cases? In other words, does the
>> code
>> you're adding implicitly assume:
>>
>> 	LAYERSERIES_COMPAT_<...> = "layer"
>>
>> for any given "layer"?
> No, there is no implicit assumption.
>
> In both A) and B) cases the maintainer adds the new "codename" to the
> list of compatible layer series in LAYERSERIES_COMPAT_<layer> for their
> layer. They can list multiple layers in there, e.g. "rocko sumo".
>
> The one annoying thing about all this is the layer maintainers do have
> to update layer.conf each time a new release codename comes out. I
> think that is a reasonable compromise to be able to give users a much
> better idea of which layers are compatible or incomaptible with their
> setup though. It means someone has looked at it and believes it will
> work with a given release series.
>
> Just to be clear, "layer" would never be a valid value there, it will
> always be the release/branch codenames.
>
> Cheers,
>
> Richard
>
>
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture

Re: [Openembedded-architecture] LAYERSERIES

[Richard Purdie]

On Fri, 2018-04-06 at 14:58 -0400, Denys Dmytriyenko wrote:
> I understand the need to reduce the support burden. But there are
> some valid 
> cases when you want to use components/layers from a different set of
> releases. 
> 
> It can sometimes be an older component in otherwise newer setup (i.e.
> one 
> layer hasn't been updated yet from rocko, but it still works fine
> with 
> everything else from sumo).
> 
> Alternatively, sometimes you need to bring a newer version of the
> component to 
> your older release - as long as it works for me, I don't want this to
> be 
> artificially restricted.
> 
> In other words, the intention is good, but it needs to be either more
> flexible or allow overriding this from distro config or local.conf...

Whilst I will admit I've not extensively tested it, I think that should
be possible to override things as this only gets validated after all
the layer.conf files are pulled in. It would need to be from the
distro's layer.conf though, or a layer.conf in the build directory as
local.conf comes in much later. Basically its possible but not easy,
which I think is probably the right thing to do here.

Cheers,

Richard

Re: [Openembedded-architecture] LAYERSERIES

[Richard Purdie]

On Fri, 2018-04-06 at 13:46 -0700, akuster808 wrote:
> Shouldn't Master get all these changes first then we switch to stable
> branch when we have a new release?
> 
> My German genes are raging about "Rules" ; )

master did get them?

To be really clear, "master" is never a value that would appear in
LAYERSERIES. That would let everyone be lazy and master would match
everywhere, then this would all be pointless.

Early in the next cycle I'll add "thud" to the list in OE-Core master
branch. Later in the cycle, when we make more invasive changes which
break compatibility, I'll drop "sumo". At that point people who haven't
added "thud" will start to see errors but this should all work out
earlier in the cycle than happened this time around.

It was a choice of doing this now, or wait another six months to get
started and since we have had the variables for a while I've opted to
move us forward on this. I'd already had comments that "rocko" in
layer.conf was incorrect, which it was and changing that was always
going to cause some pain.

Cheers,

Richard

Re: [Openembedded-architecture] LAYERSERIES

[Richard Purdie]

On Fri, 2018-04-06 at 20:25 +0200, Martin Jansa wrote:
> FWIW: I've updated some layers with this LAYERSERIES_COMPAT_<layer>
> today and I think it's good idea.

Thanks. I know its late in the cycle but I think it will help.

> It probably won't be enough to stop questions like this:
> http://lists.openembedded.org/pipermail/openembedded-core/2018-April/
> 149696.html

If the layers did have the settings I think it would have fatally
errored out before that. The way this was designed, it will not parse
further that the layer.conf files if mismatch is detected.

> but it's still better to show warning first and then some often
> harder-to-read exception or error log.

Agreed.

> Updating layer.conf in meta-qt5 was a bit more tricky as people tend
> to mix the meta-qt5 branches with different branches of other layers,
> just because their requirements for Qt are often very different than
> for other layers (e.g. many people still using meta-qt5/krogoth with
> Qt 5.6 because of the license, even with much newer morty, pyro,
> rocko, sumo, master branches of other layers and vice versa that
> someone is stuck with krogoth release for whatever reason, but needs
> newer LTS Qt 5.9 from meta-qt5/rocko). So for meta-qt5 I've used
> multiple values for LAYERSERIES_COMPAT_<layer> for combinations I use
> somewhere (so that I cover at least some testing with such uncommon
> combination) and I'm willing to accept more values to it if someone
> really uses another combination regularly. The rest of combination
> (like meta-qt5/krogoth with dizzy, which is usable as described in
> meta-qt5 commits which introduced incompatiblitites, will eventually
> trigger the warning which is fine, because people should be aware
> that they are doing something less tested and that they might to
> implement some work arounds to make it usable).

That sounds good. The system was flexible enough to allow "mismatch",
*if* the layer maintainer supports it. I'm also not expecting people to
backport this to older releases, just to start doing it now so things
improve going forward.

Cheers,

Richard

Re: [Openembedded-architecture] LAYERSERIES

[Richard Purdie]

On Fri, 2018-04-06 at 11:16 -0700, Scott Rifenbark wrote:
> We have this variable appearing in a couple manuals BTW.  If anyone
> has any feedback on what is being said here I can update.

Some small suggested tweaks:

"Using the LAYERSERIES_COMPAT variable makes it clear when a given
layer is unmaintained and untested with new releases of OE-Core. "
->
"Using the LAYERSERIES_COMPAT variable allows the layer maintainer to
indicate which combinations of the layer and OE-Core may be expected to
work. It gives the system a way to detect when a layer has not been
tested with new releases of OE-Core, for example if it had become
unmaintained."

"The OpenEmbedded build system, however, does not produce an error if
the variable is not set."
->
"The OpenEmbedded build system will warn if the variable is not set for
any given layer"

Cheers,

Richard

Re: [Openembedded-architecture] LAYERSERIES

[Scott Rifenbark]

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

Thanks Richard....

https://www.yoctoproject.org/docs/2.5/ref-manual/ref-manual.html#var-LAYERSERIES_COMPAT

Scott

On Fri, Apr 6, 2018 at 3:25 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Fri, 2018-04-06 at 11:16 -0700, Scott Rifenbark wrote:
> > We have this variable appearing in a couple manuals BTW.  If anyone
> > has any feedback on what is being said here I can update.
>
> Some small suggested tweaks:
>
> "Using the LAYERSERIES_COMPAT variable makes it clear when a given
> layer is unmaintained and untested with new releases of OE-Core. "
> ->
> "Using the LAYERSERIES_COMPAT variable allows the layer maintainer to
> indicate which combinations of the layer and OE-Core may be expected to
> work. It gives the system a way to detect when a layer has not been
> tested with new releases of OE-Core, for example if it had become
> unmaintained."
>
> "The OpenEmbedded build system, however, does not produce an error if
> the variable is not set."
> ->
> "The OpenEmbedded build system will warn if the variable is not set for
> any given layer"
>
> Cheers,
>
> Richard
>

[-- Attachment #2: Type: text/html, Size: 1673 bytes --]

Re: [Openembedded-architecture] LAYERSERIES

[Richard Purdie]

On Fri, 2018-04-06 at 15:44 -0700, Scott Rifenbark wrote:
> Thanks Richard....
> 
> https://www.yoctoproject.org/docs/2.5/ref-manual/ref-manual.html#var-
> LAYERSERIES_COMPAT

Can you tweak:

"Lists the versions of the OpenEmbedded Core project"
->
"Lists the versions of OpenEmbedded-Core"

we would never say "OE-Core Project". Also:

"To specify the OE-Core projects"
->
"To specify the OE-Core versions" 

I'd go as far as to say any reference to "OE-Core project" in the
manual probably needs tweaking.

Cheers,

Richard

Re: [Openembedded-architecture] LAYERSERIES

[Trevor Woerner]

On Fri 2018-04-06 @ 10:56:53 PM, Richard Purdie wrote:
> To be really clear, "master" is never a value that would appear in
> LAYERSERIES. That would let everyone be lazy and master would match
> everywhere, then this would all be pointless.

So what happens when the master branch of a BSP layer doesn't work with the
last release, but bitbake hasn't been updated with the next release's name?

I.e. the sumo branch of meta-bsp has

	LAYERSERIES_COMPAT_bsp = "sumo"

but what am I supposed to put as the value when meta-bsp's master branch
becomes incompatible with sumo? "master" is not allowed, and "thud" fails too.

At some point between one release and the next something might/will break. In
which case the only thing a BSP's master branch will work with is
openembedded-core's master.

Re: [Openembedded-architecture] LAYERSERIES

[Richard Purdie]

On Sat, 2018-04-14 at 17:48 -0400, Trevor Woerner wrote:
> On Fri 2018-04-06 @ 10:56:53 PM, Richard Purdie wrote:
> > 
> > To be really clear, "master" is never a value that would appear in
> > LAYERSERIES. That would let everyone be lazy and master would match
> > everywhere, then this would all be pointless.
> So what happens when the master branch of a BSP layer doesn't work
> with the
> last release, but bitbake hasn't been updated with the next release's
> name?
> 
> I.e. the sumo branch of meta-bsp has
> 
> 	LAYERSERIES_COMPAT_bsp = "sumo"
> 
> but what am I supposed to put as the value when meta-bsp's master
> branch
> becomes incompatible with sumo? "master" is not allowed, and "thud"
> fails too.
> 
> At some point between one release and the next something might/will
> break. In
> which case the only thing a BSP's master branch will work with is
> openembedded-core's master.

Post release of sumo we'll add thud as a possible option to OE-Core and
things should be ok?

Cheers,

Richard