Fwd: Maintaining ABI Compatibility for LTS branch

Fwd: Maintaining ABI Compatibility for LTS branch

[Richard Purdie]

[-- Attachment #1: Type: text/plain, Size: 84 bytes --]

Forwarding to the correct list address and cc the LTS maintainer.

Cheers,

Richard

[-- Attachment #2: Forwarded message — Maintaining ABI Compatibility for LTS branch --]
[-- Type: message/rfc822, Size: 6930 bytes --]

From: Sinan Kaya <okaya@kernel.org>
To: Richard Purdie <richard.purdie@linuxfoundation.org>, Paul Eggleton <bluelightning@bluelightning.org>, Yocto list discussion <yocto@yoctoproject.org>
Subject: Maintaining ABI Compatibility for LTS branch
Date: Sun, 6 Feb 2022 20:14:37 -0500
Message-ID: <90135b4d-6a37-f5d7-dbba-7d0ef47aa778@kernel.org>

Hi Everyone,

One of the limitations of Yocto LTS branch is that there is no 
guaranteed backwards compatibility. Therefore, any time we move a branch 
forward to move to latest dunfell release, we are taking a risk of 
breaking our customers.

Yocto reserves the right to move a package version forward if a
security fix cannot be applied properly as an example.

This promise is being held true on the kernel by running kernel API
tests etc. and running test benches across different CI environments.

I was curious about how everyone is approaching this problem.
There was some attempt to bring ABI checking functionality in the past 
but this has never been merged.

Is everyone rolling their own solution? or never moving forward?

Sinan

Re: [yocto] Maintaining ABI Compatibility for LTS branch

[Ross Burton]

On Tue, 8 Feb 2022 at 23:07, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> I was curious about how everyone is approaching this problem.
> There was some attempt to bring ABI checking functionality in the past
> but this has never been merged.
>
> Is everyone rolling their own solution? or never moving forward?

Relatedly, somewhere I have a branch that uses libabigail to extract
the library ABIs in an build.  The next step was doing the comparison,
but I never got that far.  I wouldn't be surprised if someone has done
the same but actually finished the effort.

Ross

Re: Maintaining ABI Compatibility for LTS branch

[Steve Sakoman]

On Tue, Feb 8, 2022 at 1:07 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> Forwarding to the correct list address and cc the LTS maintainer.
>
> Cheers,
>
> Richard
>
>
>
> ---------- Forwarded message ----------
> From: Sinan Kaya <okaya@kernel.org>
> To: Richard Purdie <richard.purdie@linuxfoundation.org>, Paul Eggleton <bluelightning@bluelightning.org>, Yocto list discussion <yocto@yoctoproject.org>
> Cc:
> Bcc:
> Date: Sun, 6 Feb 2022 20:14:37 -0500
> Subject: Maintaining ABI Compatibility for LTS branch
> Hi Everyone,
>
> One of the limitations of Yocto LTS branch is that there is no
> guaranteed backwards compatibility. Therefore, any time we move a branch
> forward to move to latest dunfell release, we are taking a risk of
> breaking our customers.

I'd be interested in hearing about any cases where we've broken things!

In general I don't take version upgrades (other than bug fix/cve fix
only dot releases)

> Yocto reserves the right to move a package version forward if a
> security fix cannot be applied properly as an example.

These cases should be extremely rare, and the community/technical
steering committee is given an opportunity to review this before it
happens.

I'm certainly open to suggestions on how we can do better.

I'd love to see some tooling to do ABI checking!

Steve

> This promise is being held true on the kernel by running kernel API
> tests etc. and running test benches across different CI environments.
>
> I was curious about how everyone is approaching this problem.
> There was some attempt to bring ABI checking functionality in the past
> but this has never been merged.
>
> Is everyone rolling their own solution? or never moving forward?
>
> Sinan
>

Re: [yocto] Maintaining ABI Compatibility for LTS branch

[Khem Raj]

On Wed, Feb 9, 2022 at 8:37 AM Steve Sakoman <steve@sakoman.com> wrote:
>
> On Tue, Feb 8, 2022 at 1:07 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > Forwarding to the correct list address and cc the LTS maintainer.
> >
> > Cheers,
> >
> > Richard
> >
> >
> >
> > ---------- Forwarded message ----------
> > From: Sinan Kaya <okaya@kernel.org>
> > To: Richard Purdie <richard.purdie@linuxfoundation.org>, Paul Eggleton <bluelightning@bluelightning.org>, Yocto list discussion <yocto@yoctoproject.org>
> > Cc:
> > Bcc:
> > Date: Sun, 6 Feb 2022 20:14:37 -0500
> > Subject: Maintaining ABI Compatibility for LTS branch
> > Hi Everyone,
> >
> > One of the limitations of Yocto LTS branch is that there is no
> > guaranteed backwards compatibility. Therefore, any time we move a branch
> > forward to move to latest dunfell release, we are taking a risk of
> > breaking our customers.
>
> I'd be interested in hearing about any cases where we've broken things!
>
> In general I don't take version upgrades (other than bug fix/cve fix
> only dot releases)
>
> > Yocto reserves the right to move a package version forward if a
> > security fix cannot be applied properly as an example.
>
> These cases should be extremely rare, and the community/technical
> steering committee is given an opportunity to review this before it
> happens.
>
> I'm certainly open to suggestions on how we can do better.
>
> I'd love to see some tooling to do ABI checking!

I think its not that LTS itself but also upgrade from LTS to LTS or other
newer versions, it wouuld be good to have this tool and prerhaps a list
of API/ABI changes to help migration/upgrade to newer LTS releases.

>
> Steve
>
> > This promise is being held true on the kernel by running kernel API
> > tests etc. and running test benches across different CI environments.
> >
> > I was curious about how everyone is approaching this problem.
> > There was some attempt to bring ABI checking functionality in the past
> > but this has never been merged.
> >
> > Is everyone rolling their own solution? or never moving forward?
> >
> > Sinan
> >
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> You automatically follow any topics you start or reply to.
> View/Reply Online (#56121): https://lists.yoctoproject.org/g/yocto/message/56121
> Mute This Topic: https://lists.yoctoproject.org/mt/89009568/1997914
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: Maintaining ABI Compatibility for LTS branch

[Sinan Kaya]

On 2/9/2022 11:42 AM, Richard Purdie wrote:
> There are two reasons people are interested:
> 
> a) for release stability as you mention
> b) for performance as it could be tied into the hash equivalence mechanism for
> artefact reuse - if A hasn't changed ABI, B dependning on it needn't rebuild.
> 
> There was a proof of concept of b) here:
> https://lists.yoctoproject.org/g/yocto/message/52650
> 
> There are lots of levels it could be implemented at but it is something someone
> would need to pick up and drive forward with a long term view to helping with
> issues etc.

What would be the minimum acceptable solution with the least investment?
in other words, do we have a list of requirements?

Our team has posted a solution. BMW folks posted a solution.
None of them got merged.

Could we take the version from BMW folks, merge and have the next person
add new features where it doesn't satisfy requirements?

or vice versa? or as Ross said some other work?

or none of the solutions are acceptable?

Re: Maintaining ABI Compatibility for LTS branch

[Richard Purdie]

On Wed, 2022-02-09 at 13:15 -0500, Sinan Kaya wrote:
> On 2/9/2022 11:42 AM, Richard Purdie wrote:
> > There are two reasons people are interested:
> > 
> > a) for release stability as you mention
> > b) for performance as it could be tied into the hash equivalence mechanism for
> > artefact reuse - if A hasn't changed ABI, B dependning on it needn't rebuild.
> > 
> > There was a proof of concept of b) here:
> > https://lists.yoctoproject.org/g/yocto/message/52650
> > 
> > There are lots of levels it could be implemented at but it is something someone
> > would need to pick up and drive forward with a long term view to helping with
> > issues etc.
> 
> What would be the minimum acceptable solution with the least investment?
> in other words, do we have a list of requirements?

You're after our LTS to maintain ABI. In order to do that we need help, not just
with patches generating some output, but in day to day running of the project,
i.e. help comparing output before and after changes. Whenever a patch is
submitted which breaks this, it will need attention and help some someone to
explain to that submitter what the issue, why it is important and hints on how
to fix it.

The idea of "least investment" sends shivers down my spine since it sounds like
you want to do the absolute bare minimum to have this happen, rather than a more
community oriented approach.

Anyway, my point is there is more to this than just a patch. We have various
kinds of build output already and reports on test regressions - nobody helps
with them. I need some kind of a sign that ABI would be different and there are
people able to help with review on an ongoing basis, else it will just be
something else I and a small number of others become overloaded with.

> Our team has posted a solution. BMW folks posted a solution.
> None of them got merged.

Can you remind me of your team's please?

The BMW one is about hash equivalence so wouldn't help your ABI output problem
with the LTS. From what I remember, it predates hash equivalence and the project
needed a generic equivalance mechanism complete with server done at the runqueue
level in bitbake. This has now happened so we could revisit the idea of what is
in that layer but translating it to a hash equivalence plugin.

I'd also add that even with hash equivalence done well like we ended up with, we
have only two people interested/able to work on bugs with it, the author and
myself. For a key component of the system, this worries me a lot. Adding more
complexity without maintainer support isn't going to help anyone.

> Could we take the version from BMW folks, merge and have the next person
> add new features where it doesn't satisfy requirements?

See above on the BMW version. I'm a little worried you're suggesting merging
something which doesn't actually do what you need/want :(.

> or vice versa? or as Ross said some other work?
> or none of the solutions are acceptable?

I have to admit I can't remember what the conclusion was on your team's version
but if you remind me of the patches I can try and remember. This is a bigger
problem than just patches though sadly.

Cheers,

Richard

Re: Maintaining ABI Compatibility for LTS branch

[Sinan Kaya]

On 2/9/2022 1:41 PM, Richard Purdie wrote:
>>> There are lots of levels it could be implemented at but it is something someone
>>> would need to pick up and drive forward with a long term view to helping with
>>> issues etc.
>> What would be the minimum acceptable solution with the least investment?
>> in other words, do we have a list of requirements?
> You're after our LTS to maintain ABI. In order to do that we need help, not just
> with patches generating some output, but in day to day running of the project,
> i.e. help comparing output before and after changes. Whenever a patch is
> submitted which breaks this, it will need attention and help some someone to
> explain to that submitter what the issue, why it is important and hints on how
> to fix it.
> 

That's true, this will require engagement from the community. Tool could
take few iterations to perfect. Until then, I expect tool owner to be
responsible for fixing these bugs. Once stability is reached, it becomes
community maintained.

If the tool owner doesn't maintain and community has no interest, tool
dies and gets reverted. It is as simple as any open source engagement.

When stability is established, each code contributor to LTS would be
subject to addressing issues found before they get merged.

> The idea of "least investment" sends shivers down my spine since it sounds like
> you want to do the absolute bare minimum to have this happen, rather than a more
> community oriented approach.

It depends on your taste. I believe in smaller improvements
as opposed to throwing a big project to you that no-one will use it.

Everyone has different needs. We need to find the common ones.
That's why, I'm asking if there is an existing tool that works for
large part of the community accepting that there will be some folks
that won't have their needs addressed.

I'm interested in revisiting the tooling discussion and have these gaps
addressed for the biggest audience so that we can have something to
build upon.

> 
> Anyway, my point is there is more to this than just a patch. We have various
> kinds of build output already and reports on test regressions - nobody helps
> with them. I need some kind of a sign that ABI would be different and there are
> people able to help with review on an ongoing basis, else it will just be
> something else I and a small number of others become overloaded with.
> 

Noted. Hopefully, things will be not that volatile for the LTS branch
and tool would actually help the maintainer.

In an ideal world, change needs to be stopped before that happens and
have the patch author address it similar to how you monitor build
pipelines.

>> Our team has posted a solution. BMW folks posted a solution.
>> None of them got merged.
> Can you remind me of your team's please?
> 

https://lists.yoctoproject.org/g/yocto/topic/85279259?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,160,85279259

This was an intern project from last summer that we are interested
in expanding coverage.

> The BMW one is about hash equivalence so wouldn't help your ABI output problem
> with the LTS. From what I remember, it predates hash equivalence and the project
> needed a generic equivalance mechanism complete with server done at the runqueue
> level in bitbake. This has now happened so we could revisit the idea of what is
> in that layer but translating it to a hash equivalence plugin.
> 
> I'd also add that even with hash equivalence done well like we ended up with, we
> have only two people interested/able to work on bugs with it, the author and
> myself. For a key component of the system, this worries me a lot. Adding more
> complexity without maintainer support isn't going to help anyone.
> 

OK, I didn't know the story behind the change.

>> Could we take the version from BMW folks, merge and have the next person
>> add new features where it doesn't satisfy requirements?
> See above on the BMW version. I'm a little worried you're suggesting merging
> something which doesn't actually do what you need/want:(.
> 

Got it.

>> or vice versa? or as Ross said some other work?
>> or none of the solutions are acceptable?
> I have to admit I can't remember what the conclusion was on your team's version
> but if you remind me of the patches I can try and remember. This is a bigger
> problem than just patches though sadly.

Sure, let's find out what everyone is doing.

Re: Maintaining ABI Compatibility for LTS branch

[Richard Purdie]

On Wed, 2022-02-09 at 18:41 +0000, Richard Purdie wrote:
> 
> The BMW one is about hash equivalence so wouldn't help your ABI output problem
> with the LTS. From what I remember, it predates hash equivalence and the project
> needed a generic equivalance mechanism complete with server done at the runqueue
> level in bitbake. This has now happened so we could revisit the idea of what is
> in that layer but translating it to a hash equivalence plugin.
> 
> I'd also add that even with hash equivalence done well like we ended up with, we
> have only two people interested/able to work on bugs with it, the author and
> myself. For a key component of the system, this worries me a lot. Adding more
> complexity without maintainer support isn't going to help anyone.

Sorry, I'm getting confused here with earlier work Michael Ho did at BMW. The
links here:

https://lists.yoctoproject.org/g/yocto/message/52650
https://github.com/bmwcarit/meta-abicompat

are the revised version from last year which *does* hook into hash equivalence.
I'm getting two things confused, sorry.

The nice thing about the layer above is that it is a standalone layer, we don't
have to merge it in order to use it. This shows the power of the new hash
equivalence code as it is a plugin to it. We may consider merging it at some
point but there is less of a pressing need and we need time to experiment with
it.

At this point it is a proof of concept and doesn't solve the ABI problem you're
describing in the original email. Sorry about any confusion.

The abidw recipe could be useful to your ABI issue.

Cheers,

Richard

Re: Maintaining ABI Compatibility for LTS branch

[Richard Purdie]

On Wed, 2022-02-09 at 14:13 -0500, Sinan Kaya wrote:
> On 2/9/2022 1:41 PM, Richard Purdie wrote:
> > > > There are lots of levels it could be implemented at but it is something someone
> > > > would need to pick up and drive forward with a long term view to helping with
> > > > issues etc.
> > > What would be the minimum acceptable solution with the least investment?
> > > in other words, do we have a list of requirements?
> > You're after our LTS to maintain ABI. In order to do that we need help, not just
> > with patches generating some output, but in day to day running of the project,
> > i.e. help comparing output before and after changes. Whenever a patch is
> > submitted which breaks this, it will need attention and help some someone to
> > explain to that submitter what the issue, why it is important and hints on how
> > to fix it.
> > 
> 
> That's true, this will require engagement from the community. Tool could
> take few iterations to perfect. Until then, I expect tool owner to be
> responsible for fixing these bugs. Once stability is reached, it becomes
> community maintained.
> 
> If the tool owner doesn't maintain and community has no interest, tool
> dies and gets reverted. It is as simple as any open source engagement.
> 
> When stability is established, each code contributor to LTS would be
> subject to addressing issues found before they get merged.

I agree, this either needs input from the community in order to drive it or a
sponsor. It will be interesting to see if people are interested in doing this
and I guess we can gauge that from the replies to this thread to see if people
do want to do it. I can tell you first hand how badly the existing maintainers
are burning out with the current load so we need new people.

> > The idea of "least investment" sends shivers down my spine since it sounds like
> > you want to do the absolute bare minimum to have this happen, rather than a more
> > community oriented approach.
> 
> It depends on your taste. I believe in smaller improvements
> as opposed to throwing a big project to you that no-one will use it.

We both agree on incremental improvements and I am fine with that. I don't want
any patch acceptance taken as a sign we're goging to add a significant process
burden though. I'd prefer we have a broad agreement of what the end objective is
architecture wise too.

> Everyone has different needs. We need to find the common ones.
> That's why, I'm asking if there is an existing tool that works for
> large part of the community accepting that there will be some folks
> that won't have their needs addressed.
> 
> I'm interested in revisiting the tooling discussion and have these gaps
> addressed for the biggest audience so that we can have something to
> build upon.
> 
> > 
> > Anyway, my point is there is more to this than just a patch. We have various
> > kinds of build output already and reports on test regressions - nobody helps
> > with them. I need some kind of a sign that ABI would be different and there are
> > people able to help with review on an ongoing basis, else it will just be
> > something else I and a small number of others become overloaded with.
> > 
> 
> Noted. Hopefully, things will be not that volatile for the LTS branch
> and tool would actually help the maintainer.
> 
> In an ideal world, change needs to be stopped before that happens and
> have the patch author address it similar to how you monitor build
> pipelines.
> 
> > > Our team has posted a solution. BMW folks posted a solution.
> > > None of them got merged.
> > Can you remind me of your team's please?
> > 
> 
> https://lists.yoctoproject.org/g/yocto/topic/85279259?p=,,,20,0,0,0::recentpostdate/sticky,,,20,2,160,85279259
> 
> This was an intern project from last summer that we are interested
> in expanding coverage.

"None of them got merged" isn't particularly fair here. There was a brief thread
on the yocto list, no patches were proposed or reviewed and the implementation
is a standalone layer so doesn't need to merge anyway, people could use it
already if they wanted.

If the question is whether something would be accepted into core, the answer is
possibly, it would depends. The quality of the code in that layer needs
improving for core and I'm not sure about the approach. Hooking it against
buildhistory is "easy" but as I mentioned in separate discussions earlier today,
buildhistory is getting pulled in different directions (such as a SBOM) and I'm
worried about some of the extensions to it. Certainly, the ABI saving shouldn't
really be buried in a do_install postfunc and perhaps is more of a build history
item that some others that are being proposed.

This probably does need a discussion on the architecture list and we need some
discussion and decisions about where/what buildhistory could/should do. Adding
this to buildhistory is all well and good but we don't have a meaningful
integration/monitoring of existing buildhistory issues in our
autobuilder/workflow today even before adding new things.

> 
> > I have to admit I can't remember what the conclusion was on your team's version
> > but if you remind me of the patches I can try and remember. This is a bigger
> > problem than just patches though sadly.
> 
> Sure, let's find out what everyone is doing.

Ok.

Cheers,

Richard

Re: Maintaining ABI Compatibility for LTS branch

[Sinan Kaya]

On 2/9/2022 4:38 PM, Richard Purdie wrote:
> This probably does need a discussion on the architecture list and we need some
> discussion and decisions about where/what buildhistory could/should do. Adding
> this to buildhistory is all well and good but we don't have a meaningful
> integration/monitoring of existing buildhistory issues in our
> autobuilder/workflow today even before adding new things.

I was hoping for free cycles. I didn't get one. This will be an intern
project.

The way I'm thinking is to have the ABI compat XML be part of the state
cache tgz file and come up with a CVE check kind of hook maybe called
"ABI check" that will start flagging problems.

Would this be a better architecture?

Re: Maintaining ABI Compatibility for LTS branch

[Richard Purdie]

On Mon, 2022-05-02 at 17:44 -0400, Sinan Kaya wrote:
> On 2/9/2022 4:38 PM, Richard Purdie wrote:
> > This probably does need a discussion on the architecture list and we need some
> > discussion and decisions about where/what buildhistory could/should do. Adding
> > this to buildhistory is all well and good but we don't have a meaningful
> > integration/monitoring of existing buildhistory issues in our
> > autobuilder/workflow today even before adding new things.
> 
> I was hoping for free cycles. I didn't get one. This will be an intern
> project.
> 
> The way I'm thinking is to have the ABI compat XML be part of the state
> cache tgz file and come up with a CVE check kind of hook maybe called
> "ABI check" that will start flagging problems.
> 
> Would this be a better architecture?

I'm not sure it would.

Shared state works by computing inputs into a checksum and then using that
checksum to access the cache. Finding a "previous version" or a history to
compare to therefore isn't a good fit for it.

Buildhistory does fit much better but isn't scaling in a controlled way and I
think we need the architecture discussion and an agreed plan rather than trying
to tack things onto it. That does mean someone stepping back and considering the
various needs users have rather than just an isolated use case.

Cheers,

Richard