Re: QA notification for completed autobuilder build (yocto-2.8_M1.rc2)

Re: QA notification for completed autobuilder build (yocto-2.8_M1.rc2)

[richard.purdie]

On Tue, 2019-06-11 at 03:32 +0000, pokybuild@debian9-ty-1.yocto.io
wrote:
> A build flagged for QA (yocto-2.8_M1.rc2) was completed on the
> autobuilder and is available at:
> 
> 
>     https://autobuilder.yocto.io/pub/releases/yocto-2.8_M1.rc2
>

Some notes for QA:

There was a failure in this build in qemux86 oe-selftest. It looks to
be a race issue where qemu was shutting down. This needs a bug and
looking into but is a minor issue which wouldn't block the milestone
release or further QA.

There are three timeouts in ptest, acl, bluez5 (#13366) and mdadm
(#13368). These are also not going to block the milestone release. The
bluez5 issue is gcc9 related, mdadm was exposed by wider testing in
master. The acl issue needs a bug opening and investigation as that is
a regression.

There is a lot more testing happening under automation than in previous
builds, particularly on arm hosts/targets.

Cheers,

Richard

Re: QA notification for completed autobuilder build (yocto-2.8_M1.rc2)

[Jain, Sangeeta]

>-----Original Message-----
>From: richard.purdie@linuxfoundation.org <richard.purdie@linuxfoundation.org>
>Sent: Tuesday, 11 June, 2019 4:51 PM
>To: pokybuild@debian9-ty-1.yocto.io; yocto@yoctoproject.org
>Cc: otavio@ossystems.com.br; yi.zhao@windriver.com; Sangal, Apoorv
><apoorv.sangal@intel.com>; Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; Chan,
>Aaron Chun Yew <aaron.chun.yew.chan@intel.com>; Ang, Chin Huat
><chin.huat.ang@intel.com>; akuster808@gmail.com; sjolley.yp.pm@gmail.com;
>Jain, Sangeeta <sangeeta.jain@intel.com>
>Subject: Re: QA notification for completed autobuilder build (yocto-2.8_M1.rc2)
>
>On Tue, 2019-06-11 at 03:32 +0000, pokybuild@debian9-ty-1.yocto.io
>wrote:
>> A build flagged for QA (yocto-2.8_M1.rc2) was completed on the
>> autobuilder and is available at:
>>
>>
>>     https://autobuilder.yocto.io/pub/releases/yocto-2.8_M1.rc2
>>
>
>Some notes for QA:
Thanks Richard for sharing this.
>
>There was a failure in this build in qemux86 oe-selftest. It looks to be a race issue
>where qemu was shutting down. This needs a bug and looking into but is a minor
>issue which wouldn't block the milestone release or further QA.

We are analysing the failures and will file the bugs as suggested.
>
>There are three timeouts in ptest, acl, bluez5 (#13366) and mdadm (#13368).
>These are also not going to block the milestone release. The
>bluez5 issue is gcc9 related, mdadm was exposed by wider testing in master. The
>acl issue needs a bug opening and investigation as that is a regression.

Looks like some disconnect here. In test result report, no timeout issue reported for acl ptest.
We are referring to test results at yocto-testresults under tag " master/54573-g9910a3631ce953b8dd8f3d57d6e122c7fd8cb462/1"
>
>There is a lot more testing happening under automation than in previous builds,
>particularly on arm hosts/targets.
Thanks for increased coverage.

>
>Cheers,
>
>Richard

Thanks & Regards,
Sangeeta Jain

Re: QA notification for completed autobuilder build (yocto-2.8_M1.rc2)

[richard.purdie]

On Wed, 2019-06-12 at 00:51 +0000, Jain, Sangeeta wrote:
> On Tue, 2019-06-11 at 03:32 +0000, pokybuild@debian9-ty-1.yocto.io
> > wrote:
> > > A build flagged for QA (yocto-2.8_M1.rc2) was completed on the
> > > autobuilder and is available at:
> > > 
> > > 
> > >     https://autobuilder.yocto.io/pub/releases/yocto-2.8_M1.rc2
> > > 
> > 
> > Some notes for QA:
> Thanks Richard for sharing this.
> > There was a failure in this build in qemux86 oe-selftest. It looks
> > to be a race issue where qemu was shutting down. This needs a bug
> > and looking into but is a minor issue which wouldn't block the
> > milestone release or further QA.
> 
> We are analysing the failures and will file the bugs as suggested.

Just for reference I did file some of the bugs.

> > There are three timeouts in ptest, acl, bluez5 (#13366) and mdadm
> > (#13368).
> > These are also not going to block the milestone release. The
> > bluez5 issue is gcc9 related, mdadm was exposed by wider testing in
> > master. The
> > acl issue needs a bug opening and investigation as that is a
> > regression.
> 
> Looks like some disconnect here. In test result report, no timeout
> issue reported for acl ptest.

https://autobuilder.yocto.io/pub/releases/yocto-2.8_M1.rc2/testresults/testresult-report.txt

acl                          | 184         | 9        | 0         | 302 T

I hadn't spotted that dbus-test also has a timeout :(.

This timeout is also in the testresults file:

ERROR: Exit status is 9\nDURATION: 302\nTIMEOUT: /usr/lib/acl/ptest\n

in:

https://autobuilder.yocto.io/pub/releases/yocto-2.8_M1.rc2/testresults/qemux86-64-ptest/testresults.json

> We are referring to test results at yocto-testresults under tag "
> master/54573-g9910a3631ce953b8dd8f3d57d6e122c7fd8cb462/1"

You are right, there is no mention of this in the log at:

http://git.yoctoproject.org/cgit.cgi/yocto-testresults/tree/runtime/poky/qemux86-64/core-image-sato-sdk-ptest/ptest-acl.log?id=56b1caa6044ce30e92a1895eae48d3935562b627
or
http://git.yoctoproject.org/cgit.cgi/yocto-testresults/commit/runtime/poky/qemux86-64/core-image-sato-sdk-ptest/ptest-acl.log?id=ac623e13e3bc3a348b60fe3b884e6ae567b8230a

I suspect this is due to the information being filtered out as part of
the log processing. We probably shouldn't do that.

Looking at the individual logs in https://autobuilder.yocto.io/pub/releases/yocto-2.8_M1.rc2/testresults/qemux86-64-ptest/testresults.json, the timeout information looks to be truncated off there too.

I do however note that:

            "ptestresult.sections": {
                "acl": {
                    "duration": "302",
                    "exitcode": "9",
                    "timeout": true
                },

is in http://git.yoctoproject.org/cgit.cgi/yocto-testresults/tree/runtime/poky/qemux86-64/core-image-sato-sdk-ptest/testresults.json?id=56b1caa6044ce30e92a1895eae48d3935562b627
so the timeout is recorded in the results.

Regards,

Richard

Re: QA notification for completed autobuilder build (yocto-2.8_M1.rc2)

[Jain, Sangeeta]

>-----Original Message-----
>From: richard.purdie@linuxfoundation.org <richard.purdie@linuxfoundation.org>
>Sent: Wednesday, 12 June, 2019 4:05 PM
>To: Jain, Sangeeta <sangeeta.jain@intel.com>; pokybuild@debian9-ty-
>1.yocto.io; yocto@yoctoproject.org
>Cc: otavio@ossystems.com.br; yi.zhao@windriver.com; Sangal, Apoorv
><apoorv.sangal@intel.com>; Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; Chan,
>Aaron Chun Yew <aaron.chun.yew.chan@intel.com>; Ang, Chin Huat
><chin.huat.ang@intel.com>; akuster808@gmail.com; sjolley.yp.pm@gmail.com
>Subject: Re: QA notification for completed autobuilder build (yocto-2.8_M1.rc2)
>
>On Wed, 2019-06-12 at 00:51 +0000, Jain, Sangeeta wrote:
>> On Tue, 2019-06-11 at 03:32 +0000, pokybuild@debian9-ty-1.yocto.io
>> > wrote:
>> > > A build flagged for QA (yocto-2.8_M1.rc2) was completed on the
>> > > autobuilder and is available at:
>> > >
>> > >
>> > >     https://autobuilder.yocto.io/pub/releases/yocto-2.8_M1.rc2
>> > >
>> >
>> > Some notes for QA:
>> Thanks Richard for sharing this.
>> > There was a failure in this build in qemux86 oe-selftest. It looks
>> > to be a race issue where qemu was shutting down. This needs a bug
>> > and looking into but is a minor issue which wouldn't block the
>> > milestone release or further QA.
>>
>> We are analysing the failures and will file the bugs as suggested.
>
>Just for reference I did file some of the bugs.
>
>> > There are three timeouts in ptest, acl, bluez5 (#13366) and mdadm
>> > (#13368).
>> > These are also not going to block the milestone release. The
>> > bluez5 issue is gcc9 related, mdadm was exposed by wider testing in
>> > master. The acl issue needs a bug opening and investigation as that
>> > is a regression.
>>
>> Looks like some disconnect here. In test result report, no timeout
>> issue reported for acl ptest.
>
>https://autobuilder.yocto.io/pub/releases/yocto-
>2.8_M1.rc2/testresults/testresult-report.txt
>
>acl                          | 184         | 9        | 0         | 302 T
>
>I hadn't spotted that dbus-test also has a timeout :(.
>
>This timeout is also in the testresults file:
>
>ERROR: Exit status is 9\nDURATION: 302\nTIMEOUT: /usr/lib/acl/ptest\n
>
>in:
>
>https://autobuilder.yocto.io/pub/releases/yocto-
>2.8_M1.rc2/testresults/qemux86-64-ptest/testresults.json
>
>> We are referring to test results at yocto-testresults under tag "
>> master/54573-g9910a3631ce953b8dd8f3d57d6e122c7fd8cb462/1"
>
>You are right, there is no mention of this in the log at:
>
>http://git.yoctoproject.org/cgit.cgi/yocto-
>testresults/tree/runtime/poky/qemux86-64/core-image-sato-sdk-ptest/ptest-
>acl.log?id=56b1caa6044ce30e92a1895eae48d3935562b627
>or
>http://git.yoctoproject.org/cgit.cgi/yocto-
>testresults/commit/runtime/poky/qemux86-64/core-image-sato-sdk-ptest/ptest-
>acl.log?id=ac623e13e3bc3a348b60fe3b884e6ae567b8230a
>
>I suspect this is due to the information being filtered out as part of the log
>processing. We probably shouldn't do that.
>
>Looking at the individual logs in https://autobuilder.yocto.io/pub/releases/yocto-
>2.8_M1.rc2/testresults/qemux86-64-ptest/testresults.json, the timeout
>information looks to be truncated off there too.
>
>I do however note that:
>
>            "ptestresult.sections": {
>                "acl": {
>                    "duration": "302",
>                    "exitcode": "9",
>                    "timeout": true
>                },
>
>is in http://git.yoctoproject.org/cgit.cgi/yocto-
>testresults/tree/runtime/poky/qemux86-64/core-image-sato-sdk-
>ptest/testresults.json?id=56b1caa6044ce30e92a1895eae48d3935562b627
>so the timeout is recorded in the results.

Thanks. Able to trace all time-out issues, including dbus-test !
>
>Regards,
>
>Richard
>

Thanks & Regards,
Sangeeta Jain

Re: QA notification for completed autobuilder build (yocto-2.8_M1.rc2)

[Jain, Sangeeta]

Hello All,

Intel and WR YP QA is now working on QA execution for YP build 2.8 M1 RC2.
We are planning to execute following tests for this cycle:

OEQA-manual tests for following module:

1. OE-Core
2. BSP-hw
3. BSP-Qemu

Runtime auto test for following platforms:

1. MinnowTurbot 32-bit
2. Coffee Lake
3. NUC 7
4. NUC 6
5. Edgerouter
6. MPC8315e-rdb
7. Beaglebone

ETA for completion is Friday, 15 June.


Thanks & Regards,
Sangeeta Jain

>-----Original Message-----
>From: richard.purdie@linuxfoundation.org <richard.purdie@linuxfoundation.org>
>Sent: Tuesday, 11 June, 2019 4:51 PM
>To: pokybuild@debian9-ty-1.yocto.io; yocto@yoctoproject.org
>Cc: otavio@ossystems.com.br; yi.zhao@windriver.com; Sangal, Apoorv
><apoorv.sangal@intel.com>; Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; Chan,
>Aaron Chun Yew <aaron.chun.yew.chan@intel.com>; Ang, Chin Huat
><chin.huat.ang@intel.com>; akuster808@gmail.com; sjolley.yp.pm@gmail.com;
>Jain, Sangeeta <sangeeta.jain@intel.com>
>Subject: Re: QA notification for completed autobuilder build (yocto-2.8_M1.rc2)
>
>On Tue, 2019-06-11 at 03:32 +0000, pokybuild@debian9-ty-1.yocto.io
>wrote:
>> A build flagged for QA (yocto-2.8_M1.rc2) was completed on the
>> autobuilder and is available at:
>>
>>
>>     https://autobuilder.yocto.io/pub/releases/yocto-2.8_M1.rc2
>>
>
>Some notes for QA:
>
>There was a failure in this build in qemux86 oe-selftest. It looks to be a race issue
>where qemu was shutting down. This needs a bug and looking into but is a minor
>issue which wouldn't block the milestone release or further QA.
>
>There are three timeouts in ptest, acl, bluez5 (#13366) and mdadm (#13368).
>These are also not going to block the milestone release. The
>bluez5 issue is gcc9 related, mdadm was exposed by wider testing in master. The
>acl issue needs a bug opening and investigation as that is a regression.
>
>There is a lot more testing happening under automation than in previous builds,
>particularly on arm hosts/targets.
>
>Cheers,
>
>Richard

Re: QA notification for completed autobuilder build (yocto-2.8_M1.rc2)

[Jain, Sangeeta]

QA cycle report for 2.8 M1 RC2:

	1. No high milestone defects.  
	2. Test results are available at following location:
 		• For results of all automated tests, please refer to results at public AB [1]
		• For other test results, refer to git repo " yocto-testresults-contrib" [2]
		• For test report for test cases run by Intel and WR team, refer to git repo " yocto-testresults-contrib" [2]
            3. 3 new defects are found in this cycle, BSP HW - parselog issue [3], BSP HW - audio issue [4], qemux86 oe-selftest failure on AB [5] 
            4. ptests failing in this release which were passing in previous release - acl [6]. 3 timeout issues observed in ptests - acl [7], bluez5 [8]  and mdadm [9].
	Richard's comments on ptest failures:  There are three timeouts in ptest, acl, bluez5 (#13366) and mdadm (#13368). These are also not going to block the milestone release. The
                                                                                     bluez5 issue is gcc9 related, mdadm was exposed by wider testing in master. The acl issue needs a bug opening and investigation as that is a regression.
 	Summary of ptest in this build: 
	qemuarm64-ptest:
		Total : 435597
		Pass : 433344
 		 Fail :  65
		Skip : 2188
		Timeout issues in bluez5 and mdadm
            qemux86-64-ptest
            		Total : 45346
            		Pass : 43235
           		 Fail :  56
            		Skip : 2055
            		Timeout issues in acl, bluez5 and mdadm

        5. Summary of ltp test run in this build:
	qemuarm64-ltp
		Total : 1886
		Pass : 1759
 		 Fail :  127
              qemux86-64-ltp
		Total : 1879
                             Pass : 1825
                             Fail :  54

======= Links ========

[1] - https://autobuilder.yocto.io/pub/releases/yocto-2.8_M1.rc2/testresults/

[2] - git clone git@push.yoctoproject.org:yocto-testresults-contrib -b zeus

[3] - [QA 2.8 M1 RC1][BSP HW] parselogs.ParseLogsTest.test_parselogs failure on coffeelake
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13396

[4] - [QA 2.8 M1 RC1][BSP HW] audio is not playing on coffeelake and nuc7
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13397

[5] - [QA 2.8 M1 RC1] test_boot_machine_slirp is showing error for qemux86
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13399

[6] - [QA 2.8 M1 RC1] acl ptest failure
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13395

[7] - acl ptest timeout due to perl update
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13391

[8] - bluez5 ptest hangs with gcc 9
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13366

[9] - mdadm ptest times out
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13368


Thanks & Regards,
Sangeeta Jain

QA notification for completed autobuilder build (yocto-2.8_M1.rc2)

[pokybuild]

A build flagged for QA (yocto-2.8_M1.rc2) was completed on the autobuilder and is available at:


    https://autobuilder.yocto.io/pub/releases/yocto-2.8_M1.rc2


Build hash information: 

bitbake: c4d90890547af642e99cc541af3415df3559563e
meta-gplv2: 168a5070bdf3bc45edb5bf2a1add9b7c081f5b64
meta-intel: c941d896c08d6d83694ba0af1fdf4f4dcb625e9f
meta-mingw: d17724b0eac76792a49e56213cba74e32edba48f
oecore: 86266dfcd70b8e3435d267538c3e6e4d69be829e
poky: 9910a3631ce953b8dd8f3d57d6e122c7fd8cb462



This is an automated message from the Yocto Project Autobuilder
Git: git://git.yoctoproject.org/yocto-autobuilder2
Email: richard.purdie@linuxfoundation.org