QA notification for completed autobuilder build (yocto-4.3.rc1)

QA notification for completed autobuilder build (yocto-4.3.rc1)

[Pokybuild User]

    A build flagged for QA (yocto-4.3.rc1) was completed on the autobuilder and is available at:


        https://autobuilder.yocto.io/pub/releases/yocto-4.3.rc1


    Build URL: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6062

    Build hash information: 

    bitbake: 5419a8473d6d4cd1d01537de68ad8d72cf5be0b2
meta-agl: 4063b4f9a712e32337c1d9678b2f2481dde2a346
meta-arm: 3ed13d25a065f29bd46ee725c708d12ebc3f175a
meta-aws: a30a2b66f1447dc5abdbca6c5de743e39c08b99b
meta-intel: 1bca60610c597571769edc4a057a04bfdbd2f994
meta-mingw: 65ef95a74f6ae815f63f636ed53e140a26a014ce
meta-openembedded: 35bcd8c6ddfb6bc8729d0006dab887afcc772ec9
meta-virtualization: 827092c2ec925ea3a024dcda9ccfd738e351e6ba
oecore: 4f84537670020a8d902248479efa9f062089c0d3
poky: f65f100bc5379c3153ee00b2aa62ea5c9a66ec79


    
This is an automated message from the Yocto Project Autobuilder
Git: git://git.yoctoproject.org/yocto-autobuilder2
Email: richard.purdie@linuxfoundation.org


     

Re: [qa-build-notification] QA notification for completed autobuilder build (yocto-4.3.rc1)

[Richard Purdie]

On Wed, 2023-10-18 at 06:16 +0000, Pokybuild User wrote:
>     A build flagged for QA (yocto-4.3.rc1) was completed on the autobuilder and is available at:
> 
> 
>         https://autobuilder.yocto.io/pub/releases/yocto-4.3.rc1
> 
> 
>     Build URL: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6062

There was one failure in the build, a serial login issue on ttyS1, so
an occurrence of our annoying 6.5 issue. This is the first time we've
seen it with the workaround applied.

The question is to proceed with rc1 in testing, or apply the upstream
fixes and try an rc2 with that? I'm torn...

Cheers,

Richard

Re: [tsc] [qa-build-notification] QA notification for completed autobuilder build (yocto-4.3.rc1)

[Ross Burton]

On 18 Oct 2023, at 07:29, Richard Purdie via lists.yoctoproject.org <richard.purdie=linuxfoundation.org@lists.yoctoproject.org> wrote:
> 
> On Wed, 2023-10-18 at 06:16 +0000, Pokybuild User wrote:
>>    A build flagged for QA (yocto-4.3.rc1) was completed on the autobuilder and is available at:
>> 
>> 
>>        https://autobuilder.yocto.io/pub/releases/yocto-4.3.rc1
>> 
>> 
>>    Build URL: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6062
> 
> There was one failure in the build, a serial login issue on ttyS1, so
> an occurrence of our annoying 6.5 issue. This is the first time we've
> seen it with the workaround applied.
> 
> The question is to proceed with rc1 in testing, or apply the upstream
> fixes and try an rc2 with that? I'm torn…

I’d say we should proceed with RC1 whilst testing upstream fixes.

If they manage to shake out all of the failures then we can decide whether to spin another RC with the fixes in, or just release with what with have - with copious release notes obviously. 

Ross

Re: [tsc] [qa-build-notification] QA notification for completed autobuilder build (yocto-4.3.rc1)

[Richard Purdie]

On Wed, 2023-10-18 at 15:32 +0000, Ross Burton wrote:
> On 18 Oct 2023, at 07:29, Richard Purdie via lists.yoctoproject.org <richard.purdie=linuxfoundation.org@lists.yoctoproject.org> wrote:
> > 
> > On Wed, 2023-10-18 at 06:16 +0000, Pokybuild User wrote:
> > >    A build flagged for QA (yocto-4.3.rc1) was completed on the autobuilder and is available at:
> > > 
> > > 
> > >        https://autobuilder.yocto.io/pub/releases/yocto-4.3.rc1
> > > 
> > > 
> > >    Build URL: https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6062
> > 
> > There was one failure in the build, a serial login issue on ttyS1, so
> > an occurrence of our annoying 6.5 issue. This is the first time we've
> > seen it with the workaround applied.
> > 
> > The question is to proceed with rc1 in testing, or apply the upstream
> > fixes and try an rc2 with that? I'm torn…
> 
> I’d say we should proceed with RC1 whilst testing upstream fixes.
> 
> If they manage to shake out all of the failures then we can decide
> whether to spin another RC with the fixes in, or just release with
> what with have - with copious release notes obviously. 

Lets run with this approach, I agree. I am running test builds in the
background and we'll see how reliable the upstream fix is. I think
moving forward with rc1 as is makes sense.

Cheers,

Richard

Re: [yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)

[Alexis Lothoré]

Hello,

On 10/18/23 08:16, Pokybuild User wrote:
> 
>     A build flagged for QA (yocto-4.3.rc1) was completed on the autobuilder and is available at:
> 
> 
>         https://autobuilder.yocto.io/pub/releases/yocto-4.3.rc1

The regression report looks worryingly empty. It has been generated with
yocto-4.2 as comparison base. When taking a look at yocto-testresults content
for it, I only find a few test results (11, while it should be something around
300). I am not sure yet if we did some mistakes while re-uploading manually
missing tests results during the 4.2 cycle, or if tests results are indeed
suffering some issues for release builds (this is repeatable on different 4.2.x
releases).

So in the mean time, here is a manual regression report between 4.3_M3 and
yocto-4.3.rc1:

https://pastebin.com/6QbfGstR

The report has a "rate limit" applied for noisy regression (patch incoming)

Kind regards,
Alexis

-- 
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)

[Ross Burton]

Hi Alexis,

> On 19 Oct 2023, at 09:36, Alexis Lothoré via lists.yoctoproject.org <alexis.lothore=bootlin.com@lists.yoctoproject.org> wrote:
> The regression report looks worryingly empty. It has been generated with
> yocto-4.2 as comparison base. When taking a look at yocto-testresults content
> for it, I only find a few test results (11, while it should be something around
> 300). I am not sure yet if we did some mistakes while re-uploading manually
> missing tests results during the 4.2 cycle, or if tests results are indeed
> suffering some issues for release builds (this is repeatable on different 4.2.x
> releases).
> 
> So in the mean time, here is a manual regression report between 4.3_M3 and
> yocto-4.3.rc1:
> 
> https://pastebin.com/6QbfGstR
> 
> The report has a "rate limit" applied for noisy regression (patch incoming)

Thanks for that, much appreciated.

I just skimmed your report and have some feedback to hopefully make it easier to read in the future.

I’d suggest sorting the output in order of importance.  For example, this is a section that I really don’t care about:

Match:       sdk_core-image-sato_x86_64_fvp-base_20230910083055
             sdk_core-image-sato_x86_64_fvp-base_20231017222150

Put those at the bottom, or even better collate them into a single section where there have been no changes.

Similarly:

Match:       runtime_core-image-sato_qemux86_20230911011430
             runtime_core-image-sato_qemux86_20231017223736
    Additionally, 1 new test(s) is/are present

I guess marginally more important than identical results, but “there are new tests that passed” isn’t very interesting.

Now on to the regressions.

Regression: oeselftest_ubuntu-22.04_qemux86-64_20230911011940
    oeselftest_almalinux-9.2_qemux86-64_20231017221342

Should they have matched? The host distro doesn’t match and this matters for some of the tests, as some distros don’t support some of the selftests. In this case specifically, there are seven regressions and six of them are specific to the host changing, which has the side-effect of hiding the one actual regression.

The report then lists the first however many regressions before announcing the summary:

 (In total, 7134 regressions/status changes detected)
    Additionally, 7 previously failing test(s) is/are now passing
    Additionally, 4622 new test(s) is/are present
 
The headline figure of 7134 regressions should be first, as that’s the most important data point in a skim of the report.  List the summary first, and then the breakdown.

Grouping the results would be interesting, because the list got truncated I can’t see easily if all 7134 regressions were in ptestresult.gcc-g++-user.* or if that was just the first 100 and the rest were other components.  Breaking the ptest results up by the second level component would be interesting, if it said something like this then we’d be able to get a feel for what components have broken from the report.

7134 regressions detected.
    ptestresult.gcc-libstdc++-v3-user.30_threads/thread/native_handle/cancel.cc execution test: PASS -> FAIL
    [ say 10 results per component ]
    And 6123 more in ptestresult.gcc-libstdc++v3
    ptestresult.gcc-libgomp.libgomp.c++/ctor-10.C: UNSUPPORTED -> UNRESOLVED


This one also caught my eye:
Regression:  runtime_core-image-sato_qemuppc_20230910082140
             runtime_core-image-sato_qemuppc_20231017222112
    systemd.SystemdJournalTests.test_systemd_boot_time: PASSED -> SKIPPED
    Additionally, 1 new test(s) is/are present

Is that comparing a systemd test run with a sysvinit test run?

Thanks for the work on the tool so far, this is a lot easier to read than the full reports!

Ross

Re: [yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)

[Richard Purdie]

On Thu, 2023-10-19 at 10:36 +0200, Alexis Lothoré wrote:
> Hello,
> 
> On 10/18/23 08:16, Pokybuild User wrote:
> > 
> >     A build flagged for QA (yocto-4.3.rc1) was completed on the autobuilder and is available at:
> > 
> > 
> >         https://autobuilder.yocto.io/pub/releases/yocto-4.3.rc1
> 
> The regression report looks worryingly empty. It has been generated with
> yocto-4.2 as comparison base. When taking a look at yocto-testresults content
> for it, I only find a few test results (11, while it should be something around
> 300). I am not sure yet if we did some mistakes while re-uploading manually
> missing tests results during the 4.2 cycle, or if tests results are indeed
> suffering some issues for release builds (this is repeatable on different 4.2.x
> releases).

I'd noticed that too, not sure what happened but it wasn't what I'd
expected. It would be good to understand what went wrong.

> So in the mean time, here is a manual regression report between 4.3_M3 and
> yocto-4.3.rc1:
> 
> https://pastebin.com/6QbfGstR
> 
> The report has a "rate limit" applied for noisy regression (patch incoming)

I was thinking about ideas like printing the regressions first, then
the matches. I think Ross has some good feedback about how we can
improve things.

Cheers,

Richard

Re: [yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)

[Alexis Lothoré]

Hi Ross,
On 10/19/23 13:04, Ross Burton wrote:
>> On 19 Oct 2023, at 09:36, Alexis Lothoré via lists.yoctoproject.org <alexis.lothore=bootlin.com@lists.yoctoproject.org> wrote:
> I just skimmed your report and have some feedback to hopefully make it easier to read in the future.
> 
> I’d suggest sorting the output in order of importance.  For example, this is a section that I really don’t care about:
> 
> Match:       sdk_core-image-sato_x86_64_fvp-base_20230910083055
>              sdk_core-image-sato_x86_64_fvp-base_20231017222150
> 
> Put those at the bottom, or even better collate them into a single section where there have been no changes.
> 
> Similarly:
> 
> Match:       runtime_core-image-sato_qemux86_20230911011430
>              runtime_core-image-sato_qemux86_20231017223736
>     Additionally, 1 new test(s) is/are present
> 
> I guess marginally more important than identical results, but “there are new tests that passed” isn’t very interesting.

Indeed, not so useful and hiding the real content at the bottom

> Regression: oeselftest_ubuntu-22.04_qemux86-64_20230911011940
>     oeselftest_almalinux-9.2_qemux86-64_20231017221342
> 
> Should they have matched? The host distro doesn’t match and this matters for some of the tests, as some distros don’t support some of the selftests. In this case specifically, there are seven regressions and six of them are specific to the host changing, which has the side-effect of hiding the one actual regression.

Yeah, that's a point I have been struggling with when starting to update those
tools. The initial assumption I have started working with, after discussing the
matter with Richard (see [1]), is the following:
1. MACHINE _must_ match between base and target
2. Different HOSTS _can_ be cross-checked

But the issue you are pointing tends to show it does not work well in some
cases. I will have to do some tests to see if dropping 2. reduce this noise
without loosing valuable data, or if we need to find something smarter

> 
> The report then lists the first however many regressions before announcing the summary:
> 
>  (In total, 7134 regressions/status changes detected)
>     Additionally, 7 previously failing test(s) is/are now passing
>     Additionally, 4622 new test(s) is/are present
>  
> The headline figure of 7134 regressions should be first, as that’s the most important data point in a skim of the report.  List the summary first, and then the breakdown.

ACK

> Grouping the results would be interesting, because the list got truncated I can’t see easily if all 7134 regressions were in ptestresult.gcc-g++-user.* or if that was just the first 100 and the rest were other components.  Breaking the ptest results up by the second level component would be interesting, if it said something like this then we’d be able to get a feel for what components have broken from the report.
> 
> 7134 regressions detected.
>     ptestresult.gcc-libstdc++-v3-user.30_threads/thread/native_handle/cancel.cc execution test: PASS -> FAIL
>     [ say 10 results per component ]
>     And 6123 more in ptestresult.gcc-libstdc++v3
>     ptestresult.gcc-libgomp.libgomp.c++/ctor-10.C: UNSUPPORTED -> UNRESOLVED

Makes sense. I have posted this morning the series introducing the display limit
([2]), I can work on a v2 implementing the ptest specific display limit

> This one also caught my eye:
> Regression:  runtime_core-image-sato_qemuppc_20230910082140
>              runtime_core-image-sato_qemuppc_20231017222112
>     systemd.SystemdJournalTests.test_systemd_boot_time: PASSED -> SKIPPED
>     Additionally, 1 new test(s) is/are present
> 
> Is that comparing a systemd test run with a sysvinit test run?

I think the comparison is relevant, both are bout systemd system (many systemd
tests are present and OK in runtime_core-image-sato_qemuppc_20231017222112
results). However I do not get why it is marked as skipped and not failed:

            "systemd.SystemdJournalTests.test_systemd_boot_time": {
                "duration": 2.3783957958221436,
                "log": "Error when parsing time from boot string",
                "status": "SKIPPED"
            }

Maybe an issue in the corresponding runner ?

> Thanks for the work on the tool so far, this is a lot easier to read than the full reports!
> 
> Ross

Thank you for having taken time to give some feedback !

[1] https://lists.yoctoproject.org/g/automated-testing/message/1216
[2]
https://lore.kernel.org/openembedded-core/20231019095352.25923-1-alexis.lothore@bootlin.com/

-- 
Alexis Lothoré, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [yocto] QA notification for completed autobuilder build (yocto-4.3.rc1)

[Richard Purdie]

On Thu, 2023-10-19 at 14:28 +0200, Alexis Lothoré wrote:
> Hi Ross,
> On 10/19/23 13:04, Ross Burton wrote:
> > > On 19 Oct 2023, at 09:36, Alexis Lothoré via lists.yoctoproject.org <alexis.lothore=bootlin.com@lists.yoctoproject.org> wrote:
> > I just skimmed your report and have some feedback to hopefully make it easier to read in the future.
> > 
> > I’d suggest sorting the output in order of importance.  For example, this is a section that I really don’t care about:
> > 
> > Match:       sdk_core-image-sato_x86_64_fvp-base_20230910083055
> >              sdk_core-image-sato_x86_64_fvp-base_20231017222150
> > 
> > Put those at the bottom, or even better collate them into a single section where there have been no changes.
> > 
> > Similarly:
> > 
> > Match:       runtime_core-image-sato_qemux86_20230911011430
> >              runtime_core-image-sato_qemux86_20231017223736
> >     Additionally, 1 new test(s) is/are present
> > 
> > I guess marginally more important than identical results, but “there are new tests that passed” isn’t very interesting.
> 
> Indeed, not so useful and hiding the real content at the bottom
> 
> > Regression: oeselftest_ubuntu-22.04_qemux86-64_20230911011940
> >     oeselftest_almalinux-9.2_qemux86-64_20231017221342
> > 
> > Should they have matched? The host distro doesn’t match and this matters for some of the tests, as some distros don’t support some of the selftests. In this case specifically, there are seven regressions and six of them are specific to the host changing, which has the side-effect of hiding the one actual regression.
> 
> Yeah, that's a point I have been struggling with when starting to update those
> tools. The initial assumption I have started working with, after discussing the
> matter with Richard (see [1]), is the following:
> 1. MACHINE _must_ match between base and target
> 2. Different HOSTS _can_ be cross-checked
> 
> But the issue you are pointing tends to show it does not work well in some
> cases. I will have to do some tests to see if dropping 2. reduce this noise
> without loosing valuable data, or if we need to find something smarter

What the autobuilder does is either runs one selftest on a random host
for q-quick, or for a-full it will run five selftests, one for
"centos", one for "arm", one for "ubuntu", one for "fedora" and one for
"debian".

Ideally we'd therefore compare debian to debian if we have any choice.
It is a question of finding the closest matches. That is hard in code
though whilst keeping it generic.

Cheers,

Richard

RE: [qa-build-notification] QA notification for completed autobuilder build (yocto-4.3.rc1)

[Tham, Jing Hui]

Hi all,
 
Intel and WR YP QA is planning for QA execution for YP build yocto-4.3.rc1. We are planning to execute following tests for this cycle:
 
OEQA-manual tests for following module:
1. OE-Core
2. BSP-hw
 
Runtime auto test for following platforms:
	1. MinnowBoard Turbot - 32bit
	2. Kaby Lake (7th Generation Intel(r) Core(tm) Processors)
	3. Tiger Lake (11th Generation Intel(r) Core(tm) Processors)
	4. Alder Lake-S (12th Generation Intel(r) Core(tm) Processors)
	5. Raptor Lake-P (13th Generation Intel(r) Core(tm) Processors)
	6. Beaglebone

 
ETA for completion next Tuesday, October 24.
 
Best regards,
Jing Hui



> -----Original Message-----
> From: qa-build-notification@lists.yoctoproject.org <qa-build-
> notification@lists.yoctoproject.org> On Behalf Of Pokybuild User
> Sent: Wednesday, October 18, 2023 2:16 PM
> To: yocto@lists.yoctoproject.org
> Cc: qa-build-notification@lists.yoctoproject.org
> Subject: [qa-build-notification] QA notification for completed autobuilder
> build (yocto-4.3.rc1)
> 
> 
>     A build flagged for QA (yocto-4.3.rc1) was completed on the autobuilder
> and is available at:
> 
> 
>         https://autobuilder.yocto.io/pub/releases/yocto-4.3.rc1
> 
> 
>     Build URL:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6062
> 
>     Build hash information:
> 
>     bitbake: 5419a8473d6d4cd1d01537de68ad8d72cf5be0b2
> meta-agl: 4063b4f9a712e32337c1d9678b2f2481dde2a346
> meta-arm: 3ed13d25a065f29bd46ee725c708d12ebc3f175a
> meta-aws: a30a2b66f1447dc5abdbca6c5de743e39c08b99b
> meta-intel: 1bca60610c597571769edc4a057a04bfdbd2f994
> meta-mingw: 65ef95a74f6ae815f63f636ed53e140a26a014ce
> meta-openembedded: 35bcd8c6ddfb6bc8729d0006dab887afcc772ec9
> meta-virtualization: 827092c2ec925ea3a024dcda9ccfd738e351e6ba
> oecore: 4f84537670020a8d902248479efa9f062089c0d3
> poky: f65f100bc5379c3153ee00b2aa62ea5c9a66ec79
> 
> 
> 
> This is an automated message from the Yocto Project Autobuilder
> Git: git://git.yoctoproject.org/yocto-autobuilder2
> Email: richard.purdie@linuxfoundation.org
> 
> 
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#321): https://lists.yoctoproject.org/g/qa-build-
> notification/message/321
> Mute This Topic: https://lists.yoctoproject.org/mt/102034503/6424044
> Group Owner: qa-build-notification+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/qa-build-
> notification/leave/10762429/6424044/1746813897/xyzzy
> [jing.hui.tham@intel.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [qa-build-notification] QA notification for completed autobuilder build (yocto-4.3.rc1)

[Richard Purdie]

On Thu, 2023-10-19 at 14:06 +0000, Jing Hui Tham wrote:
> Hi all,
>  
> Intel and WR YP QA is planning for QA execution for YP build yocto-4.3.rc1. We are planning to execute following tests for this cycle:
>  
> OEQA-manual tests for following module:
> 1. OE-Core
> 2. BSP-hw
>  
> Runtime auto test for following platforms:
> 	1. MinnowBoard Turbot - 32bit
> 	2. Kaby Lake (7th Generation Intel(r) Core(tm) Processors)
> 	3. Tiger Lake (11th Generation Intel(r) Core(tm) Processors)
> 	4. Alder Lake-S (12th Generation Intel(r) Core(tm) Processors)
> 	5. Raptor Lake-P (13th Generation Intel(r) Core(tm) Processors)
> 	6. Beaglebone
> 
>  
> ETA for completion next Tuesday, October 24.

We've realised there is a nasty bug in rc1 related to patchtest
deleting local changes. Given the build failure in the rc1 build and
other security issues we're thinking we should make an rc2 build and
abandon rc1.

Sorry for the churn. The new build should be ready soon with any luck.

Cheers,

Richard