Re: QA cycle report for 2.4.4 RC1

[richard.purdie@linuxfoundation.org]

On Fri, 2018-11-23 at 03:23 +0000, Jain, Sangeeta wrote:
>  
> Hello All,
>  
> This is the full report for 2.4.4 RC1: 
> 
https://wiki.yoctoproject.org/wiki/WW46_-_2018-11-14_-_Full_Test_Cycle_2.4.4_RC1
>  
>  
> Summary
>  
> All planned tests were executed.
>  
> Total Test Executed – 3330
> Passed Test – 3318
> Failed Test – 8
> Blocked Test - 4
>  
> There were zero high priority defect.  Team had found 2 new defects. 
>  
> New Bugs
>  
> [1] Bug 13033 -  [2.4.4RC1] [Bitbake] [Case 142] PR number is not
> getting increased with remote server/local client mode
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13033
>  
>  [2]Bug 13038 - [2.4.4RC1][Package Management][Getting Error Message
> "Failed to synchronize cache for repo 'repository'"]
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13038

Thanks for running the QA for this. I do have some
questions/observations.

Firstly, this report is a little misleading as there are only two bugs
mentioned but 8 failures. The full report shows other bugs which were
for example reopened or already open.

Taking the above bugs first, I believe they're both manual versions of
tests which are already automated. I believe the automated tests have
passed and there appears to be some kind of problem with the manual
execution such as the wrong process being documented. I'm not quite
sure why these are being run manually? Was the list of manual tests we
received from Intel incorrect?

Looking at other bugs in the QA report:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12670

It was reopened as it had been fixed in sumo onwards but not rocko.
I've backported the fix to the rocko branch. I don't believe its a
release blocker.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12991

This is webkitgtk failing to build in the build-appliance and looks
entirely related to resources in the VM. How this could have worked in
the past yet fail now makes me wonder about whether it was in fact
tested previously, whether the VM configuration changed or quite what
happened. I don't believe its a blocking bug as it does seem to be a VM
resource problem, not a real failure or problem with build-appliance.
Its hard to tell for sure with the limited data we have though.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12240

A reopened lsb testing bug but its not really showing problems with the
actual image, just testing difficulties.


Based on this, I think 2.4.4 is good to be released, I don't see any
blocking issues. I would like to understand what tests are actually
being run and why though.

Cheers,

Richard