QA cycle report for 2.6 M4 RC1

QA cycle report for 2.6 M4 RC1

[Jain, Sangeeta]

[-- Attachment #1: Type: text/plain, Size: 2342 bytes --]



Hello All,



This is the full report for 2.6 M4 RC1:

https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1





Summary



All planned tests were executed.



Total Test Executed - 3339

Passed Test - 3322

Failed Test - 7

Blocked Test - 4



There were zero high priority defect.  Team had found 4 new defects.

ptest for 1 module failed in current release but passed in previous 2.6 M3 rc1.  For Openssl no test was executed in current release as well as previous release. Present status of bugs for respective modules is as  follows:



ModuleName  -  BugId  -  Present Status



gstreamer        - 12990     - New



Note: For busybox, pass rate is lower than previous release, but no test cases which passes in previous release failed in current release. Lower pass rate is due to new test cases added in this release. No bug filed.



Performance test



 rootfs performance on ubuntu1604 upgraded by  13.04% in 2.6 M3 RC1 w.r.t. 2.6 M2 RC1



QA-Hints



For performance test, in this release, QA team has performed analysis on results from "yocto-perf" mailing list, rather than on results from Linux foundation machines.
We observed two major difference between machines used to run Performance test by "yocto-perf" mailing list and Linux Foundation machines:

1) Mailing List data points was not constant, sometime more, sometime less
2) Mailing List used machine with different spec



New Bugs



[1] Bug 12974 -  [2.6 M4 RC1] [OE-Core] Crosstap doesn't work on 2.6 M4 RC1

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12974



 [2] Bug 129<https://bugzilla.yoctoproject.org/show_bug.cgi?id=12952>79 - [2.6 M4 rc1] test_recipetool_create_cmake failed on Fedora 27

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12979


[3] Bug 129<https://bugzilla.yoctoproject.org/show_bug.cgi?id=12946>91 - [2.6 M4 RC1][Build-Appliance] Bitbake build-appliance-image getting failed during building image due to webkitgtk package
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12991

[4] Bug 129<https://bugzilla.yoctoproject.org/show_bug.cgi?id=12954>92 - [2.6 M4 rc1] test_devtool_add_fetch_git failed on Fedora 27
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12992

Thanks & Regards,
Sangeeta Jain


[-- Attachment #2: Type: text/html, Size: 13200 bytes --]

Re: QA cycle report for 2.6 M4 RC1

[akuster808]

[-- Attachment #1: Type: text/plain, Size: 2810 bytes --]


On 11/9/18 1:24 AM, Jain, Sangeeta wrote:
>
>  
>
> Hello All,
>
>  
>
Thank you to the Intel and Wind QA teams for performing these tasks.


Will the Build appliance failure keep it from functioning ?


- armin


> This is the full report for 2.6 M4 RC1: 
>
> https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
>
>  
>
>  
>
> *Summary*
>
>  
>
> All planned tests were executed.
>
>  
>
> Total Test Executed – 3339
>
> Passed Test – 3322
>
> Failed Test – 7
>
> Blocked Test - 4
>
>  
>
> There were zero high priority defect.  Team had found 4 new defects.
> ptest for 1modulefailed in current release but passed in previous 2.6
> M3 rc1.  For Openssl no test was executed in current release as well
> as previous release. Present status of bugs for respective modules is
> as  follows:
>  
> ModuleName  –  BugId  –  Present Status
>  
> gstreamer       – 12990     - New
>  
> Note: For busybox, pass rate is lower than previous release, but no
> test cases which passes in previous release failed in current release.
> Lower pass rate is due to new test cases added in this release. No bug
> filed.
>
>  
>
> *Performance test*
>
>  
>
>  rootfs performance on ubuntu1604 upgraded by  13.04% in 2.6 M3 RC1
> w.r.t. 2.6 M2 RC1
>
>  
>
> *QA-Hints *
>
> * *
>
> For performance test, in this release, QA team has performed analysis
> on results from “yocto-perf” mailing list, rather than on results from
> Linux foundation machines.
>
> Weobserved two major difference between machines used to run
> Performance test by “yocto-perf” mailing list and Linux Foundation
> machines:
>
>  
>
> 1) Mailing List data points was not constant, sometime more, sometime less
>
> 2) Mailing List used machine with different spec
>
> * *
>
> *New Bugs*
>
> * *
>
> [1] Bug 12974 -  [2.6 M4 RC1] [OE-Core] Crosstap doesn't work on 2.6
> M4 RC1
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12974
>
>  
>
>  [2] Bug 129
> <https://bugzilla.yoctoproject.org/show_bug.cgi?id=12952>79 - [2.6 M4
> rc1] test_recipetool_create_cmake failed on Fedora 27
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12979
>
> * *
>
> [3] Bug 129
> <https://bugzilla.yoctoproject.org/show_bug.cgi?id=12946>91 - [2.6 M4
> RC1][Build-Appliance] Bitbake build-appliance-image getting failed
> during building image due to webkitgtk package
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12991
>
>  
>
> [4] Bug 129
> <https://bugzilla.yoctoproject.org/show_bug.cgi?id=12954>92 - [2.6 M4
> rc1] test_devtool_add_fetch_git failed on Fedora 27
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=12992
>
>  
>
> Thanks & Regards,
>
> Sangeeta Jain
>
>  
>
>

[-- Attachment #2: Type: text/html, Size: 16167 bytes --]

Re: QA cycle report for 2.6 M4 RC1

[richard.purdie]

On Fri, 2018-11-09 at 09:24 +0000, Jain, Sangeeta wrote:
> 
> This is the full report for 2.6 M4 RC1: 
> https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1

Thanks Sangeeta and team!

Now we have the QA report for YP 2.6 M4 rc1 (Final 2.6) we need to make
a release go or nogo decision. To do this we have the following:
 
QA Report: https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
Release Criteria: https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.6_Status#Milestone_4.2FFinal_-_Target_Oct._26.2C_2018

We'd be happy to take representations from members and the community to
help reach that decision.

My personal view is that whilst there are a number of issues present in
rc1, we should release it, collect up fixes on the thud branch (aleady
happening) and plan on a 2.6.1 as soon as it looks like we have enough
critical mass behind those as opposed to an rc2 and further delays to
the release.

Cheers,

Richard

Re: QA cycle report for 2.6 M4 RC1

[Randy MacLeod]

On 11/10/18 11:25 AM, richard.purdie@linuxfoundation.org wrote:
> On Fri, 2018-11-09 at 09:24 +0000, Jain, Sangeeta wrote:
>>
>> This is the full report for 2.6 M4 RC1:
>> https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
> 
> Thanks Sangeeta and team!
> 
> Now we have the QA report for YP 2.6 M4 rc1 (Final 2.6) we need to make
> a release go or nogo decision. To do this we have the following:
>   
> QA Report: https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
> Release Criteria: https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.6_Status#Milestone_4.2FFinal_-_Target_Oct._26.2C_2018
> 
> We'd be happy to take representations from members and the community to
> help reach that decision.
> 
> My personal view is that whilst there are a number of issues present in
> rc1, we should release it, collect up fixes on the thud branch (aleady
> happening) and plan on a 2.6.1 as soon as it looks like we have enough
> critical mass behind those as opposed to an rc2 and further delays to
> the release.


Despite the fact that there are a few release criteria that are not
in the 'Done' state yet, I approve of releasing YP-2.6M4 on the
condition that the Docs and Web site criteria are taken care of
before release.

I have reviewed the open bugs. Several are resolved or will be soon and
the ones that remain appear to be either limited in impact such
as 12974/systemtap or are likely due to builder problems such
as 12991/webkitgtk on the build appliance.

Build time tests have crept up somewhat for the rootfs and eSDK
tests but not so dramatically that I would suggest blocking GA.
We haven't had anyone investigate the root cause yet AFAIK but
that can happen post-release and noted in the release notes.

The package update status is unknown due to the tracker being down
but in M3 we were at 81% done so we're still in a good albeit
unquantified state.

What are the plans for the Documentation checks and Wiki/Web
site update? That needs to be 'Done' but I expect it will
be taken care of in the coming days.

../Randy


> 
> Cheers,
> 
> Richard
> 
> 
> 
> 
> 


-- 
# Randy MacLeod
# Wind River Linux

Re: QA cycle report for 2.6 M4 RC1

[akuster808]

[-- Attachment #1: Type: text/plain, Size: 1502 bytes --]


On 11/10/18 8:25 AM, richard.purdie@linuxfoundation.org wrote:
> On Fri, 2018-11-09 at 09:24 +0000, Jain, Sangeeta wrote:
>> This is the full report for 2.6 M4 RC1: 
>> https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
> Thanks Sangeeta and team!
>
> Now we have the QA report for YP 2.6 M4 rc1 (Final 2.6) we need to make
> a release go or nogo decision. To do this we have the following:
>  
> QA Report: https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
> Release Criteria: https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.6_Status#Milestone_4.2FFinal_-_Target_Oct._26.2C_2018
>
> We'd be happy to take representations from members and the community to
> help reach that decision.

Regarding.

*Bug 12991* <https://bugzilla.yoctoproject.org/show_bug.cgi?id=12991>
-[2.6 M4 RC1][Build-Appliance] Bitbake build-appliance-image getting
failed during building image due to webkitgtk package

Does it mean the Build-Appliance is non functioning ?  It was broken at
the Sumo release time as well. Should it be dropped as the release criteria?

- Armin

> My personal view is that whilst there are a number of issues present in
> rc1, we should release it, collect up fixes on the thud branch (aleady
> happening) and plan on a 2.6.1 as soon as it looks like we have enough
> critical mass behind those as opposed to an rc2 and further delays to
> the release.
>
> Cheers,
>
> Richard
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 3153 bytes --]

Re: QA cycle report for 2.6 M4 RC1

[richard.purdie]

On Sat, 2018-11-10 at 10:02 -0800, akuster808 wrote:
> 
> On 11/10/18 8:25 AM, richard.purdie@linuxfoundation.org wrote:
> > On Fri, 2018-11-09 at 09:24 +0000, Jain, Sangeeta wrote:
> > > This is the full report for 2.6 M4 RC1: 
> > > https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
> > 
> > Thanks Sangeeta and team!
> > 
> > Now we have the QA report for YP 2.6 M4 rc1 (Final 2.6) we need to make
> > a release go or nogo decision. To do this we have the following:
> >  
> > QA Report: https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
> > Release Criteria: https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.6_Status#Milestone_4.2FFinal_-_Target_Oct._26.2C_2018
> > 
> > We'd be happy to take representations from members and the community to
> > help reach that decision.
> 
> Regarding. 
> Bug 12991 - [2.6 M4 RC1][Build-Appliance] Bitbake build-appliance-
> image getting failed during building image due to webkitgtk package
> Does it mean the Build-Appliance is non functioning ?  It was broken
> at the Sumo release time as well. Should it be dropped as the release
> criteria?

Build-appliance is a tricky test as it tests multiple things, roughly:

* vmdk images under vmware
* the web browser
* toaster
* whether we can self host (build poky within poky)

The fact the webkit recipe failed to build may be due to several
reasons:

* random race type condition
* lack of memory in the VM
* phase of the moon
* other things

I'm not convinced its a release blocker, or that it invalidates the b-a 
test, or that it would even reproduce. If it does reproduce that would
be more interesting and easier to debug.

I do think we're going to have to split up the b-a test in 2.7 so that
even if its not manual QA'd, we can automatically test some pieces of
what it covers.

Cheers,

Richard

Re: QA cycle report for 2.6 M4 RC1

[akuster808]

On 11/11/18 2:21 AM, richard.purdie@linuxfoundation.org wrote:
> On Sat, 2018-11-10 at 10:02 -0800, akuster808 wrote:
>> On 11/10/18 8:25 AM, richard.purdie@linuxfoundation.org wrote:
>>> On Fri, 2018-11-09 at 09:24 +0000, Jain, Sangeeta wrote:
>>>> This is the full report for 2.6 M4 RC1: 
>>>> https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
>>> Thanks Sangeeta and team!
>>>
>>> Now we have the QA report for YP 2.6 M4 rc1 (Final 2.6) we need to make
>>> a release go or nogo decision. To do this we have the following:
>>>  
>>> QA Report: https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
>>> Release Criteria: https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.6_Status#Milestone_4.2FFinal_-_Target_Oct._26.2C_2018
>>>
>>> We'd be happy to take representations from members and the community to
>>> help reach that decision.
>> Regarding. 
>> Bug 12991 - [2.6 M4 RC1][Build-Appliance] Bitbake build-appliance-
>> image getting failed during building image due to webkitgtk package
>> Does it mean the Build-Appliance is non functioning ?  It was broken
>> at the Sumo release time as well. Should it be dropped as the release
>> criteria?
> Build-appliance is a tricky test as it tests multiple things, roughly:
>
> * vmdk images under vmware
> * the web browser
> * toaster
> * whether we can self host (build poky within poky)
>
> The fact the webkit recipe failed to build may be due to several
> reasons:
>
> * random race type condition
> * lack of memory in the VM
> * phase of the moon
> * other things
>
> I'm not convinced its a release blocker, or that it invalidates the b-a 
> test, or that it would even reproduce. If it does reproduce that would
> be more interesting and easier to debug.

ok, but is the b-a functional  in 2.6?

- armin

>
> I do think we're going to have to split up the b-a test in 2.7 so that
> even if its not manual QA'd, we can automatically test some pieces of
> what it covers.
>
> Cheers,
>
> Richard
>
>

Re: QA cycle report for 2.6 M4 RC1

[richard.purdie]

On Sun, 2018-11-11 at 13:18 -0800, akuster808 wrote:
> On 11/11/18 2:21 AM, richard.purdie@linuxfoundation.org wrote:
> > On Sat, 2018-11-10 at 10:02 -0800, akuster808 wrote:
> > > On 11/10/18 8:25 AM, richard.purdie@linuxfoundation.org wrote:
> > > > On Fri, 2018-11-09 at 09:24 +0000, Jain, Sangeeta wrote:
> > > > > This is the full report for 2.6 M4 RC1: 
> > > > > 
https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
> > > > 
> > > > Thanks Sangeeta and team!
> > > > 
> > > > Now we have the QA report for YP 2.6 M4 rc1 (Final 2.6) we need
> > > > to make
> > > > a release go or nogo decision. To do this we have the
> > > > following:
> > > >  
> > > > QA Report: 
> > > > https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
> > > > Release Criteria: 
> > > > https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.6_Status#Milestone_4.2FFinal_-_Target_Oct._26.2C_2018
> > > > 
> > > > We'd be happy to take representations from members and the
> > > > community to
> > > > help reach that decision.
> > > 
> > > Regarding. 
> > > Bug 12991 - [2.6 M4 RC1][Build-Appliance] Bitbake build-
> > > appliance-
> > > image getting failed during building image due to webkitgtk
> > > package
> > > Does it mean the Build-Appliance is non functioning ?  It was
> > > broken
> > > at the Sumo release time as well. Should it be dropped as the
> > > release
> > > criteria?
> > 
> > Build-appliance is a tricky test as it tests multiple things,
> > roughly:
> > 
> > * vmdk images under vmware
> > * the web browser
> > * toaster
> > * whether we can self host (build poky within poky)
> > 
> > The fact the webkit recipe failed to build may be due to several
> > reasons:
> > 
> > * random race type condition
> > * lack of memory in the VM
> > * phase of the moon
> > * other things
> > 
> > I'm not convinced its a release blocker, or that it invalidates the
> > b-a 
> > test, or that it would even reproduce. If it does reproduce that
> > would
> > be more interesting and easier to debug.
> 
> ok, but is the b-a functional  in 2.6?

It booted under VMWare, I believe the web browser was functional and it
managed a build to a point somewhere in webkitgtk which was the only
failure in the build.

I suspect but have little evidence that the failure was a race or
system resource issue rather than a functionality problem with b-a.
Does that make b-a functional? In my view, yes, your view may vary.

Cheers,

Richard

Re: QA cycle report for 2.6 M4 RC1

[akuster808]

On 11/11/18 1:24 PM, richard.purdie@linuxfoundation.org wrote:
> On Sun, 2018-11-11 at 13:18 -0800, akuster808 wrote:
>> On 11/11/18 2:21 AM, richard.purdie@linuxfoundation.org wrote:
>>> On Sat, 2018-11-10 at 10:02 -0800, akuster808 wrote:
>>>> On 11/10/18 8:25 AM, richard.purdie@linuxfoundation.org wrote:
>>>>> On Fri, 2018-11-09 at 09:24 +0000, Jain, Sangeeta wrote:
>>>>>> This is the full report for 2.6 M4 RC1: 
>>>>>>
> https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
>>>>> Thanks Sangeeta and team!
>>>>>
>>>>> Now we have the QA report for YP 2.6 M4 rc1 (Final 2.6) we need
>>>>> to make
>>>>> a release go or nogo decision. To do this we have the
>>>>> following:
>>>>>  
>>>>> QA Report: 
>>>>> https://wiki.yoctoproject.org/wiki/WW44_-_2018-10-30_-_Full_Test_Cycle_2.6_M4_RC1
>>>>> Release Criteria: 
>>>>> https://wiki.yoctoproject.org/wiki/Yocto_Project_v2.6_Status#Milestone_4.2FFinal_-_Target_Oct._26.2C_2018
>>>>>
>>>>> We'd be happy to take representations from members and the
>>>>> community to
>>>>> help reach that decision.
>>>> Regarding. 
>>>> Bug 12991 - [2.6 M4 RC1][Build-Appliance] Bitbake build-
>>>> appliance-
>>>> image getting failed during building image due to webkitgtk
>>>> package
>>>> Does it mean the Build-Appliance is non functioning ?  It was
>>>> broken
>>>> at the Sumo release time as well. Should it be dropped as the
>>>> release
>>>> criteria?
>>> Build-appliance is a tricky test as it tests multiple things,
>>> roughly:
>>>
>>> * vmdk images under vmware
>>> * the web browser
>>> * toaster
>>> * whether we can self host (build poky within poky)
>>>
>>> The fact the webkit recipe failed to build may be due to several
>>> reasons:
>>>
>>> * random race type condition
>>> * lack of memory in the VM
>>> * phase of the moon
>>> * other things
>>>
>>> I'm not convinced its a release blocker, or that it invalidates the
>>> b-a 
>>> test, or that it would even reproduce. If it does reproduce that
>>> would
>>> be more interesting and easier to debug.
>> ok, but is the b-a functional  in 2.6?
> It booted under VMWare, I believe the web browser was functional and it
> managed a build to a point somewhere in webkitgtk which was the only
> failure in the build.
>
> I suspect but have little evidence that the failure was a race or
> system resource issue rather than a functionality problem with b-a.
> Does that make b-a functional? In my view, yes, your view may vary.

Nope. I am good. If it is functional I am good with it.  We can address
the other issues later.

thanks,

Armin

>
> Cheers,
>
> Richard
>
>

Re: QA cycle report for 2.6 M4 RC1

[Burton, Ross]

On Sat, 10 Nov 2018 at 16:26, <richard.purdie@linuxfoundation.org> wrote:
> My personal view is that whilst there are a number of issues present in
> rc1, we should release it, collect up fixes on the thud branch (aleady
> happening) and plan on a 2.6.1 as soon as it looks like we have enough
> critical mass behind those as opposed to an rc2 and further delays to
> the release.

I'd suggest we schedule 2.6.1 for about month after release, there's
quite a queue of security fixes already.

With that caveat, I'm fine with this plan.

Ross

Re: QA cycle report for 2.6 M4 RC1

[akuster808]

On 11/12/18 8:12 AM, Burton, Ross wrote:
> On Sat, 10 Nov 2018 at 16:26, <richard.purdie@linuxfoundation.org> wrote:
>> My personal view is that whilst there are a number of issues present in
>> rc1, we should release it, collect up fixes on the thud branch (aleady
>> happening) and plan on a 2.6.1 as soon as it looks like we have enough
>> critical mass behind those as opposed to an rc2 and further delays to
>> the release.
> I'd suggest we schedule 2.6.1 for about month after release, there's
> quite a queue of security fixes already.

Maybe shoot for before the Dec Holidays instead of a fixed period after
2.6 releases, a nice way to end the year : )

- armin

>
> With that caveat, I'm fine with this plan.
>
> Ross

Re: QA cycle report for 2.6 M4 RC1

[Burton, Ross]

On Mon, 12 Nov 2018 at 16:20, akuster808 <akuster808@gmail.com> wrote:
> Maybe shoot for before the Dec Holidays instead of a fixed period after
> 2.6 releases, a nice way to end the year : )

*Before* the holidays, I don't plan on taking a laptop with me and RP
works enough outside of Mon/Fri 9-5 as it is! :)

Ross