Help with intermittent autobuilder exposed bugs

Help with intermittent autobuilder exposed bugs

[Richard Purdie]

We have a situation brewing on the autobuilder. Each run there are some
bugs which sometimes can appear. Law of averages says one or more of
the issues below will now appear as we have so many of them. The
project's standard processes aren't helping to capture the issues or
track them down. 

With the release mostly sorted out from my perspective I can look at
some of the other health aspects of the project.

In theory SWAT should have filed bugs for these. In most cases they
haven't. If they did, we'd discuss them at triage, decide they looked
hard and then there wouldn't be anyone to "assign" to them so they'd
most likely end up on my plate anyway. I find it hard to know what to
do with these. I already have a load in my bug backlog which I've not
gotten sorted out so adding more probably won't help.

To summarise the issues [just from the last few builds]:

https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/756
oe-selftest recipetool.RecipetoolTests.test_recipetool_load_plugin
RP has a long standing bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13070
https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1177
qa-extras2 step2c testimage logrotate.LogrotateTest.test_2_logrotate
for core-image-sato
[no open bug?]

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/441
oe-selftest devtool.DevtoolExtractTests.test_devtool_deploy_target
[no open bug, unhelpful backtrace from tinfoil]

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/440
oe-sefltest signing.LockedSignatures.test_locked_signatures
[no open bug or any idea why it failed]

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/440
oe-selftest distrodata.Distrodata.test_maintainer
[no open bug, same unhelpful backtrace from tinfoil]

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/440
oe-selftest reproducible.ReproducibleTests.test_reproducible_builds
[no open bug?, fedora specific perldoc reproducibility issue]

https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/52
qemuarm systemd timeout parselogs error
"""Central error: Oct 19 11:21:48 qemuarm login[228]:
pam_systemd(login:session): Failed to create session: Connection timed
out"""
[no open bug]

Some of these are with master-next but I don't believe are related to
any issue specific to -next, its just because -next has had more test
runs.

I'm not sure what we do about this. The SWAT process really needs to
pay more attention, equally the people are volunteers and therefore I
try to be grateful for any help.

These issues are becoming more frequent and mean we lose confidence in
the builds which delays patch merging. People often ask me how they can
help, ideas on how we can move forward with this would be welcome...

I also know Armin has a difficult to debug selftest issue with warrior
which is blocking its release build right now.

Cheers,

Richard

Re: Help with intermittent autobuilder exposed bugs

[akuster808]

On 10/20/19 5:54 AM, Richard Purdie wrote:
> We have a situation brewing on the autobuilder. Each run there are some
> bugs which sometimes can appear. Law of averages says one or more of
> the issues below will now appear as we have so many of them. The
> project's standard processes aren't helping to capture the issues or
> track them down. 
>
> With the release mostly sorted out from my perspective I can look at
> some of the other health aspects of the project.
>
> In theory SWAT should have filed bugs for these. In most cases they
> haven't. If they did, we'd discuss them at triage, decide they looked
> hard and then there wouldn't be anyone to "assign" to them so they'd
> most likely end up on my plate anyway. I find it hard to know what to
> do with these. I already have a load in my bug backlog which I've not
> gotten sorted out so adding more probably won't help.
>
> To summarise the issues [just from the last few builds]:

I'll get started logging bugs
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/756
> oe-selftest recipetool.RecipetoolTests.test_recipetool_load_plugin
> RP has a long standing bug:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13070
> https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1177
> qa-extras2 step2c testimage logrotate.LogrotateTest.test_2_logrotate
> for core-image-sato
> [no open bug?]
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13602


>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/441
> oe-selftest devtool.DevtoolExtractTests.test_devtool_deploy_target
> [no open bug, unhelpful backtrace from tinfoil]

Bug #13601, seen with stable/warrior-nmut and on same host - fedora30-ty-1



> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/440
> oe-sefltest signing.LockedSignatures.test_locked_signatures
> [no open bug or any idea why it failed]
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13605
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/440
> oe-selftest distrodata.Distrodata.test_maintainer
> [no open bug, same unhelpful backtrace from tinfoil]
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13604
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/440
> oe-selftest reproducible.ReproducibleTests.test_reproducible_builds
> [no open bug?, fedora specific perldoc reproducibility issue]
Think its the same as
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13602
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/52
> qemuarm systemd timeout parselogs error
> """Central error: Oct 19 11:21:48 qemuarm login[228]:I have 
> pam_systemd(login:session): Failed to create session: Connection timed
> out"""
> [no open bug]
I am seeing this on one of my arm machines at home when it boots so I
will take this one.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13603

>
> Some of these are with master-next but I don't believe are related to
> any issue specific to -next, its just because -next has had more test
> runs.
>
> I'm not sure what we do about this. The SWAT process really needs to
> pay more attention, equally the people are volunteers and therefore I
> try to be grateful for any help.
>
> These issues are becoming more frequent and mean we lose confidence in
> the builds which delays patch merging. People often ask me how they can
> help, ideas on how we can move forward with this would be welcome...
>
> I also know Armin has a difficult to debug selftest issue with warrior
> which is blocking its release build right now.
>
> Cheers,
>
> Richard
>

Re: Help with intermittent autobuilder exposed bugs

[Ross Burton]

On 20/10/2019 16:54, akuster808 wrote:
> 
> 
> On 10/20/19 5:54 AM, Richard Purdie wrote:
>> We have a situation brewing on the autobuilder. Each run there are some
>> bugs which sometimes can appear. Law of averages says one or more of
>> the issues below will now appear as we have so many of them. The
>> project's standard processes aren't helping to capture the issues or
>> track them down.
>>
>> With the release mostly sorted out from my perspective I can look at
>> some of the other health aspects of the project.
>>
>> In theory SWAT should have filed bugs for these. In most cases they
>> haven't. If they did, we'd discuss them at triage, decide they looked
>> hard and then there wouldn't be anyone to "assign" to them so they'd
>> most likely end up on my plate anyway. I find it hard to know what to
>> do with these. I already have a load in my bug backlog which I've not
>> gotten sorted out so adding more probably won't help.
>>
>> To summarise the issues [just from the last few builds]:
> 
> I'll get started logging bugs
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/56/builds/756
>> oe-selftest recipetool.RecipetoolTests.test_recipetool_load_plugin
>> RP has a long standing bug:
>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13070
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1177
>> qa-extras2 step2c testimage logrotate.LogrotateTest.test_2_logrotate
>> for core-image-sato
>> [no open bug?]
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13602
> 
> 
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/441
>> oe-selftest devtool.DevtoolExtractTests.test_devtool_deploy_target
>> [no open bug, unhelpful backtrace from tinfoil]
> 
> Bug #13601, seen with stable/warrior-nmut and on same host - fedora30-ty-1
> 
> 
> 
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/440
>> oe-sefltest signing.LockedSignatures.test_locked_signatures
>> [no open bug or any idea why it failed]
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13605
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/440
>> oe-selftest distrodata.Distrodata.test_maintainer
>> [no open bug, same unhelpful backtrace from tinfoil]
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13604
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/440
>> oe-selftest reproducible.ReproducibleTests.test_reproducible_builds
>> [no open bug?, fedora specific perldoc reproducibility issue]
> Think its the same as
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13602

I think you got the wrong number?

If you meant the bug where pod2man writes its version number into the 
manpages then that isn't Fedora-specific, nor impacts these files:

perl-dbg_5.30.0-r0_amd64.deb
perl-module-config-heavy_5.30.0-r0_amd64.deb
perl-module-digest-md5_5.30.0-r0_amd64.deb
perl-module-encode-byte_5.30.0-r0_amd64.deb
perl-module-encode-cn_5.30.0-r0_amd64.deb
perl-module-encode-ebcdic_5.30.0-r0_amd64.deb
perl-module-encode-jp_5.30.0-r0_amd64.deb
perl-module-encode-kr_5.30.0-r0_amd64.deb
perl-module-encode-symbol_5.30.0-r0_amd64.deb
perl-module-encode-tw_5.30.0-r0_amd64.deb
perl-module-encode_5.30.0-r0_amd64.deb
perl-module-tie-hash-namedcapture_5.30.0-r0_amd64.deb
perl-ptest_5.30.0-r0_amd64.deb
perl-src_5.30.0-r0_amd64.deb

I don't actually have a Fedora machine to hand right now but if anyone 
does, this should be fairly trivial to replicate/isolate.

Ross

Re: Help with intermittent autobuilder exposed bugs

[Ross Burton]

On 20/10/2019 16:54, akuster808 wrote:
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/440
>> oe-selftest reproducible.ReproducibleTests.test_reproducible_builds
>> [no open bug?, fedora specific perldoc reproducibility issue]
> Think its the same as
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13602

Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=13606.

Ross

Re: Help with intermittent autobuilder exposed bugs

[Joshua Watt]

On Mon, Oct 21, 2019 at 11:50 AM Ross Burton <ross.burton@intel.com> wrote:
>
> On 20/10/2019 16:54, akuster808 wrote:
> >> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/440
> >> oe-selftest reproducible.ReproducibleTests.test_reproducible_builds
> >> [no open bug?, fedora specific perldoc reproducibility issue]
> > Think its the same as
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=13602
>
> Filed as https://bugzilla.yoctoproject.org/show_bug.cgi?id=13606.

I'll look into that one

>
> Ross
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core