Stable ABI checking (take 2)

Stable ABI checking (take 2)

[Andrew Cooper]

Hello,

Staging is now capable of writing out an ABI description when the
appropriate tool (abi-dumper) is available.

We now have to possible courses of action for ABI checking in builds.

1) Publish the ABI descriptions on xenbits, update all downstream test
systems to invoke abi-compliance-checker manually.

2) Commit/update the ABI descriptions when RELEASE-$X.$Y.0 is tagged,
update the main build to use abi-compliance-checker when available.


Pros/Cons:

The ABI descriptions claim to be sensitive to toolchain in use.  I don't
know how true this is in practice.

Publishing on xenbits involves obtaining even more misc artefacts during
the build, which is going to be firm -2 from downstreams.

Committing the ABI descriptions lets abi checking work in developer
builds (with suitable tools installed).  It also means we get checking
"for free" in Gitlab CI and OSSTest without custom logic.


Thoughts on which approach is better?  I'm leaning in favour of option 2
because it allows for consumption by developers and test systems.

If we do go with route 2, I was thinking of adding a `make check`
hierarchy.  Longer term, this can be used to queue up other unit tests
which can be run from within the build tree.

~Andrew

Re: Stable ABI checking (take 2)

[Jan Beulich]

On 22.02.2021 15:03, Andrew Cooper wrote:
> Hello,
> 
> Staging is now capable of writing out an ABI description when the
> appropriate tool (abi-dumper) is available.
> 
> We now have to possible courses of action for ABI checking in builds.
> 
> 1) Publish the ABI descriptions on xenbits, update all downstream test
> systems to invoke abi-compliance-checker manually.
> 
> 2) Commit/update the ABI descriptions when RELEASE-$X.$Y.0 is tagged,
> update the main build to use abi-compliance-checker when available.
> 
> 
> Pros/Cons:
> 
> The ABI descriptions claim to be sensitive to toolchain in use.  I don't
> know how true this is in practice.
> 
> Publishing on xenbits involves obtaining even more misc artefacts during
> the build, which is going to be firm -2 from downstreams.
> 
> Committing the ABI descriptions lets abi checking work in developer
> builds (with suitable tools installed).  It also means we get checking
> "for free" in Gitlab CI and OSSTest without custom logic.
> 
> 
> Thoughts on which approach is better?  I'm leaning in favour of option 2
> because it allows for consumption by developers and test systems.

+1 for option 2, fwiw.

> If we do go with route 2, I was thinking of adding a `make check`
> hierarchy.  Longer term, this can be used to queue up other unit tests
> which can be run from within the build tree.

Is there a reason the normal build process can't be made fail in
case verification fails? Besides "make check" typically meaning to
invoke a functional testsuite rather than (just) some compatibility
checking, I'd also be worried of no-one (likely including me) to
remember to separately run "make check" at appropriate times.

Jan

Re: Stable ABI checking (take 2)

[Andrew Cooper]

On 22/02/2021 14:37, Jan Beulich wrote:
> On 22.02.2021 15:03, Andrew Cooper wrote:
>> Hello,
>>
>> Staging is now capable of writing out an ABI description when the
>> appropriate tool (abi-dumper) is available.
>>
>> We now have to possible courses of action for ABI checking in builds.
>>
>> 1) Publish the ABI descriptions on xenbits, update all downstream test
>> systems to invoke abi-compliance-checker manually.
>>
>> 2) Commit/update the ABI descriptions when RELEASE-$X.$Y.0 is tagged,
>> update the main build to use abi-compliance-checker when available.
>>
>>
>> Pros/Cons:
>>
>> The ABI descriptions claim to be sensitive to toolchain in use.  I don't
>> know how true this is in practice.
>>
>> Publishing on xenbits involves obtaining even more misc artefacts during
>> the build, which is going to be firm -2 from downstreams.
>>
>> Committing the ABI descriptions lets abi checking work in developer
>> builds (with suitable tools installed).  It also means we get checking
>> "for free" in Gitlab CI and OSSTest without custom logic.
>>
>>
>> Thoughts on which approach is better?  I'm leaning in favour of option 2
>> because it allows for consumption by developers and test systems.
> +1 for option 2, fwiw.
>
>> If we do go with route 2, I was thinking of adding a `make check`
>> hierarchy.  Longer term, this can be used to queue up other unit tests
>> which can be run from within the build tree.
> Is there a reason the normal build process can't be made fail in
> case verification fails? Besides "make check" typically meaning to
> invoke a functional testsuite rather than (just) some compatibility
> checking, I'd also be worried of no-one (likely including me) to
> remember to separately run "make check" at appropriate times.

As far as RPM is concerned, splitting the two is important, as %build
and %check are explicitly separate steps.  I have no idea what the deb
policy/organisation is here.

Merging some of check into build would be a layering violation, and even
if we did so, where do you draw the line?

~Andrew

Re: Stable ABI checking (take 2)

[Jan Beulich]

On 22.02.2021 17:00, Andrew Cooper wrote:
> On 22/02/2021 14:37, Jan Beulich wrote:
>> On 22.02.2021 15:03, Andrew Cooper wrote:
>>> Hello,
>>>
>>> Staging is now capable of writing out an ABI description when the
>>> appropriate tool (abi-dumper) is available.
>>>
>>> We now have to possible courses of action for ABI checking in builds.
>>>
>>> 1) Publish the ABI descriptions on xenbits, update all downstream test
>>> systems to invoke abi-compliance-checker manually.
>>>
>>> 2) Commit/update the ABI descriptions when RELEASE-$X.$Y.0 is tagged,
>>> update the main build to use abi-compliance-checker when available.
>>>
>>>
>>> Pros/Cons:
>>>
>>> The ABI descriptions claim to be sensitive to toolchain in use.  I don't
>>> know how true this is in practice.
>>>
>>> Publishing on xenbits involves obtaining even more misc artefacts during
>>> the build, which is going to be firm -2 from downstreams.
>>>
>>> Committing the ABI descriptions lets abi checking work in developer
>>> builds (with suitable tools installed).  It also means we get checking
>>> "for free" in Gitlab CI and OSSTest without custom logic.
>>>
>>>
>>> Thoughts on which approach is better?  I'm leaning in favour of option 2
>>> because it allows for consumption by developers and test systems.
>> +1 for option 2, fwiw.
>>
>>> If we do go with route 2, I was thinking of adding a `make check`
>>> hierarchy.  Longer term, this can be used to queue up other unit tests
>>> which can be run from within the build tree.
>> Is there a reason the normal build process can't be made fail in
>> case verification fails? Besides "make check" typically meaning to
>> invoke a functional testsuite rather than (just) some compatibility
>> checking, I'd also be worried of no-one (likely including me) to
>> remember to separately run "make check" at appropriate times.
> 
> As far as RPM is concerned, splitting the two is important, as %build
> and %check are explicitly separate steps.  I have no idea what the deb
> policy/organisation is here.
> 
> Merging some of check into build would be a layering violation, and even
> if we did so, where do you draw the line?

Well, building a shared object that won't load is as bad as building
a shared object that won't work because of violating expected
guarantees. The closest similarity I can think of right away would be
the linker error you ought to get when a to-be-exported symbol can't
be resolved. The line imo would be drawn between things detectable at
build time vs those only detectable by actually using the generated
binaries.

Jan

Re: Stable ABI checking (take 2)

[Ian Jackson]

Jan Beulich writes ("Re: Stable ABI checking (take 2)"):
> On 22.02.2021 15:03, Andrew Cooper wrote:
> +1 for option 2, fwiw.

I'm in favour of option 2.

Andrew Cooper writes ("Re: Stable ABI checking (take 2)"):
> As far as RPM is concerned, splitting the two is important, as %build
> and %check are explicitly separate steps.  I have no idea what the deb
> policy/organisation is here.

The reason why distro build systems like to distinguish "build" from
"check" (run tests) is that often the tests are time-consuming (or
have intrusive dependencies or other practical problems).

IMO if the ABI check is very fast there is no reason not to run it by
default.  (We have configure to deal with the dependency issue.)

Ian.

Re: Stable ABI checking (take 2)

[Andrew Cooper]

On 22/02/2021 17:21, Ian Jackson wrote:
> Jan Beulich writes ("Re: Stable ABI checking (take 2)"):
>> On 22.02.2021 15:03, Andrew Cooper wrote:
>> +1 for option 2, fwiw.
> I'm in favour of option 2.

Option 2 it is then.

>
> Andrew Cooper writes ("Re: Stable ABI checking (take 2)"):
>> As far as RPM is concerned, splitting the two is important, as %build
>> and %check are explicitly separate steps.  I have no idea what the deb
>> policy/organisation is here.
> The reason why distro build systems like to distinguish "build" from
> "check" (run tests) is that often the tests are time-consuming (or
> have intrusive dependencies or other practical problems).
>
> IMO if the ABI check is very fast there is no reason not to run it by
> default.  (We have configure to deal with the dependency issue.)

It turns out that libxl causes abi-dumper to churn for ~4s or so, which
isn't ideal.  All the other libraries are in the noise.

If we are going for a check-by-default policy, then we obviously need to
exclude the non-stable libraries by default.

However, to fix problems pertaining to the unstable libraries,
downstreams do need to be able to invoke checking on the other libraries
as well.

I'll have a think as to how best to make that happen.

~Andrew

Re: Stable ABI checking (take 2)

[Ian Jackson]

Andrew Cooper writes ("Re: Stable ABI checking (take 2)"):
> It turns out that libxl causes abi-dumper to churn for ~4s or so, which
> isn't ideal.  All the other libraries are in the noise.

I think that means making it part of "make check" or something.

> However, to fix problems pertaining to the unstable libraries,
> downstreams do need to be able to invoke checking on the other libraries
> as well.

Fine by me.

Ian