Resend QA cycle report for 2.4.2 RC2

Resend QA cycle report for 2.4.2 RC2

[Yeoh, Ee Peng]

Hello All,
Resend the full report for 2.4.2 RC2 with updated information:  
https://wiki.yoctoproject.org/wiki/WW10_-_2018-03-05-_Full_Test_Cycle_-_2.4.2_rc2

======= Summary ========

The QA cycle for release 2.4.2 RC2 was completed including the performance test.  Team had discovered 3 new bugs (oeselftest.sstatetests.test_sstate_sametune_samesigs failed, x11perf failed on minnowmax and nuc, toaster run again button does not behaved as expected).

For testrun# 9206-9212, testopia testcases not 100% completed as there are some outdated testcases that were no longer available in oeqa/selftest testsuite, pending testopia data cleanup. 

Performance tests were executed for both 2.4.2 RC2 and 2.4.1 RC1 (as regression and benchmark) on these new machines in linux foundation. 

======= QA-Hints========

QA shows no major bug.

======= Bugs ========

New Bugs
[1] 12587 - [2.4.2 rc2] test_sstate_sametune_samesigs (sstatetests.SStateTests) [2] 12590 - [2.4.2 rc2] No. of Chars/sec For Yocto Image is very Low To that of Ubuntu [3] 12591 - [2.4.2 rc2] Run again button from all builds page must run the specified task

======== Links =========
1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12587
2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12590
3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12591

Regards
Ee Peng 

Re: Resend QA cycle report for 2.4.2 RC2

[Jolley, Stephen K]

Why are we showing 62 (2%) of the test cases as Idle.  (Not Run).  Were they blocked? What happened to them?

Thanks,

Stephen

-----Original Message-----
From: Yeoh, Ee Peng 
Sent: Thursday, March 08, 2018 1:41 AM
To: 'yocto@yoctoproject.org' <yocto@yoctoproject.org>; Purdie, Richard <richard.purdie@intel.com>; Jolley, Stephen K <stephen.k.jolley@intel.com>; Jordan, Robin L <robin.l.jordan@intel.com>; Zhang, Jessica <jessica.zhang@intel.com>; Lock, Joshua G <joshua.g.lock@intel.com>; Eggleton, Paul <paul.eggleton@intel.com>
Cc: Sangal, Apoorv <apoorv.sangal@intel.com>
Subject: Resend QA cycle report for 2.4.2 RC2

Hello All,
Resend the full report for 2.4.2 RC2 with updated information:  
https://wiki.yoctoproject.org/wiki/WW10_-_2018-03-05-_Full_Test_Cycle_-_2.4.2_rc2

======= Summary ========

The QA cycle for release 2.4.2 RC2 was completed including the performance test.  Team had discovered 3 new bugs (oeselftest.sstatetests.test_sstate_sametune_samesigs failed, x11perf failed on minnowmax and nuc, toaster run again button does not behaved as expected).

For testrun# 9206-9212, testopia testcases not 100% completed as there are some outdated testcases that were no longer available in oeqa/selftest testsuite, pending testopia data cleanup. 

Performance tests were executed for both 2.4.2 RC2 and 2.4.1 RC1 (as regression and benchmark) on these new machines in linux foundation. 

======= QA-Hints========

QA shows no major bug.

======= Bugs ========

New Bugs
[1] 12587 - [2.4.2 rc2] test_sstate_sametune_samesigs (sstatetests.SStateTests) [2] 12590 - [2.4.2 rc2] No. of Chars/sec For Yocto Image is very Low To that of Ubuntu [3] 12591 - [2.4.2 rc2] Run again button from all builds page must run the specified task

======== Links =========
1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12587
2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12590
3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12591

Regards
Ee Peng 

Re: Resend QA cycle report for 2.4.2 RC2

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 502 bytes --]

On 8 March 2018 at 09:41, Yeoh, Ee Peng <ee.peng.yeoh@intel.com> wrote:

> [1] 12587 - [2.4.2 rc2] test_sstate_sametune_samesigs
> (sstatetests.SStateTests)


Can't replicate this reliably with rm_work disabled.  Does the QA
autobuilder run with rm_work enabled?

$ oe-selftest -r sstatetests
...
2018-03-08 20:35:41,863 - oe-selftest - INFO - oe-selftest () - Ran 20
tests in 4308.314s
2018-03-08 20:35:41,863 - oe-selftest - INFO - oe-selftest - OK - All
required tests passed

Ross

[-- Attachment #2: Type: text/html, Size: 973 bytes --]

Re: Resend QA cycle report for 2.4.2 RC2

[Yeoh, Ee Peng]

Hi Stephen,

Those idle test cases are actually belongs to the outdated testcases inside Testopia, GDC had opened a Bugzilla ticket to remove and replace outdated and duplicated test cases. We are in progress of removing ducplicated test cases, next step is to remove outdated test cases using automated script that utilize testopia python api. 

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12435

For testrun# 9206-9212, testopia testcases not 100% completed as there are some outdated testcases that were no longer available in oeqa/selftest testsuite, pending testopia data cleanup.

Please let me know if any more question.

Thanks,
Ee Peng 

-----Original Message-----
From: Jolley, Stephen K 
Sent: Friday, March 9, 2018 12:48 AM
To: Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; 'yocto@yoctoproject.org' <yocto@yoctoproject.org>; Purdie, Richard <richard.purdie@intel.com>; Jordan, Robin L <robin.l.jordan@intel.com>; Zhang, Jessica <jessica.zhang@intel.com>; Lock, Joshua G <joshua.g.lock@intel.com>; Eggleton, Paul <paul.eggleton@intel.com>
Cc: Sangal, Apoorv <apoorv.sangal@intel.com>
Subject: RE: Resend QA cycle report for 2.4.2 RC2

Why are we showing 62 (2%) of the test cases as Idle.  (Not Run).  Were they blocked? What happened to them?

Thanks,

Stephen

-----Original Message-----
From: Yeoh, Ee Peng 
Sent: Thursday, March 08, 2018 1:41 AM
To: 'yocto@yoctoproject.org' <yocto@yoctoproject.org>; Purdie, Richard <richard.purdie@intel.com>; Jolley, Stephen K <stephen.k.jolley@intel.com>; Jordan, Robin L <robin.l.jordan@intel.com>; Zhang, Jessica <jessica.zhang@intel.com>; Lock, Joshua G <joshua.g.lock@intel.com>; Eggleton, Paul <paul.eggleton@intel.com>
Cc: Sangal, Apoorv <apoorv.sangal@intel.com>
Subject: Resend QA cycle report for 2.4.2 RC2

Hello All,
Resend the full report for 2.4.2 RC2 with updated information:  
https://wiki.yoctoproject.org/wiki/WW10_-_2018-03-05-_Full_Test_Cycle_-_2.4.2_rc2

======= Summary ========

The QA cycle for release 2.4.2 RC2 was completed including the performance test.  Team had discovered 3 new bugs (oeselftest.sstatetests.test_sstate_sametune_samesigs failed, x11perf failed on minnowmax and nuc, toaster run again button does not behaved as expected).

For testrun# 9206-9212, testopia testcases not 100% completed as there are some outdated testcases that were no longer available in oeqa/selftest testsuite, pending testopia data cleanup. 

Performance tests were executed for both 2.4.2 RC2 and 2.4.1 RC1 (as regression and benchmark) on these new machines in linux foundation. 

======= QA-Hints========

QA shows no major bug.

======= Bugs ========

New Bugs
[1] 12587 - [2.4.2 rc2] test_sstate_sametune_samesigs (sstatetests.SStateTests) [2] 12590 - [2.4.2 rc2] No. of Chars/sec For Yocto Image is very Low To that of Ubuntu [3] 12591 - [2.4.2 rc2] Run again button from all builds page must run the specified task

======== Links =========
1. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12587
2. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12590
3. https://bugzilla.yoctoproject.org/show_bug.cgi?id=12591

Regards
Ee Peng 

Re: Resend QA cycle report for 2.4.2 RC2

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

On 8 March 2018 at 20:39, Burton, Ross <ross.burton@intel.com> wrote:

> On 8 March 2018 at 09:41, Yeoh, Ee Peng <ee.peng.yeoh@intel.com> wrote:
>
>> [1] 12587 - [2.4.2 rc2] test_sstate_sametune_samesigs
>> (sstatetests.SStateTests)
>
>
> Can't replicate this reliably with rm_work disabled.  Does the QA
> autobuilder run with rm_work enabled?
>
> $ oe-selftest -r sstatetests
> ...
> 2018-03-08 20:35:41,863 - oe-selftest - INFO - oe-selftest () - Ran 20
> tests in 4308.314s
> 2018-03-08 20:35:41,863 - oe-selftest - INFO - oe-selftest - OK - All
> required tests passed
>

We now understand what is happening.  We've had this fixed in master for a
while[1], Richard is backporting it to the stable branches now.

I don't consider this a release blocker.

Ross

[1]
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=58e05020e09971f05effb79f8f1eb2012df4f0db

[-- Attachment #2: Type: text/html, Size: 1846 bytes --]