All of lore.kernel.org
 help / color / mirror / Atom feed
* QA cycle report for 2.4 M2 rc2
@ 2017-08-03  0:55 Cruz, Libertad
  2017-08-03 10:19 ` Richard Purdie
  0 siblings, 1 reply; 2+ messages in thread
From: Cruz, Libertad @ 2017-08-03  0:55 UTC (permalink / raw)
  To: yocto, Purdie, Richard, Eggleton, Paul, Jolley, Stephen K, Zhang,
	Jessica
  Cc: Cruz Alcaraz, Juan M

[-- Attachment #1: Type: text/plain, Size: 7206 bytes --]

Hello All,



Here is the report for the 2.4 M2 rc2 full point release test cycle.

Full Report : https://wiki.yoctoproject.org/wiki/WW31_-_2017-07-31_-_Full_Test_Cycle_2.4_M2_rc2



======= Summary ========

The QA cycle for release 2.4 M2 rc2 is complete.


Performance
Measurements were mostly the same levels as of 2.4 M1 rc1, there was a little improvement of 5% on rootfs build time for Fedora machine.


Ubuntu    Test       2.4 M1 rc1    2.4 M2 rc2     %
                 sato         1:12:11          1:13:58          2.47
                 rootfs      2:24                2:31               4.86
                 rmwork  1:08:27          1:07:22         -1.58
                 kernel     5:26                5:39                3.99
                 eSDK       2:30                2:30                0.00


Fedora    Test      2.4 M1 rc1     2.4 M2 rc2     %
               sato         1:13:49           1:13:52         0.07
               rootfs      2:38                 2:29              -5.70
               rmwork  1:08:22           1:08:42         0.49
               kernel     6:10                 6:04              -1.62
               eSDK       2:36                 2:34              -1.28





ptest

There was an improvement of the pass rate for gdk-pixbuf of 5% , an improvement of 1.30% on glib-2.0, an improvement of 2.9% on openssh, an improvement of 15.2% on util-linux. There was a regression on the pass rate for libxml2 of 4.20% .



Compliance

For posix test there is no mayor change in the number of failures on 2.4m2 rc2 and 2.4 m1 rc1.



                    Failures            2.4 M2 RC2      2.4 M1 RC1

                  Generic86-64           46                      45

                  Beaglebone              45                      45

                  Edgerouter               46                      46

                  Mpc8315e-rdb        45                      45



Results are found here: https://wiki.yoctoproject.org/wiki/Posix_result

Detail analysis found here:  https://wiki.yoctoproject.org/wiki/POSIX-results



For LTP testing we had 2 additional failures in comparison to 2.4 M1 rc1



                                                                Genericx86-64            mpc8315e-rdb

2.4 M2 rc2                Total                        1384                              1348

                                     Failures                    24                                   30



                                                                Genericx86-64            mpc8315e-rdb

2.4 M1 rc1                Total                        1384                              1348

                                     Failures                    22                                   30





The LPT results are found: https://wiki.yoctoproject.org/wiki/LTP_result .





======= QA-Hints========



QA does not approve that  the workarounds provided  for 2.4 m2 rc2  were on different commits. Testing was not limited to 2.4 m2 rc2 commit "8e15e9b6e478f6368034519b2a8fd3c7ea71d23b" it was performed by following each commit on the provided workarounds described below.



      --11850[1] was opened to track the fact that Eclipse plugins were not publish under 2.4 m2 rc2 so, the plugins were taken from previous build 2.4 m2 rc1. These files need to be copied in to 2.4m2 rc2 if and before release is approved.

      --11834[10] describes that if you add a bad layer to bblayers bitbake server will hang in connecting and  although is resolved & verified this bug was linked to this release candidate because the commit is not integrated to 2.4 m2 rc2. By linking this issue in the test report users can then find the correct commit that will fix it.

      --11857[9] was opened to track that there are some problems creating wic images on CentOS is also resolved & verified, but it is still linked to 2.4 m2 rc2 because the commit is not integrated to this release candidate. Users can then find the correct commit that will fix it.

      --11855[4]  this bug reports the fact that libgcc-initial can't build on fedora i686, but again when trying with latest master commit it passes. Users can then know which commit to work on.

      --11889[11] Toaster does not build on 2.4 m2 rc2 but when changing to latest master commit it works fine.





This testing cycle report does not reflects the results  exclusively of 2.4 m2 rc2 but of all the other commits that were applied so that eclipse, toaster, bitbake and  ADT on Fedora i686 would not be blocked. We recommend a 2.4 m2 rc3 where all the commits are integrated.



On the wiki a  new section was created for Compliance results. For LTP testing there is currently an issue with the hardware infrastructure of Wind River hence  beaglebone and  edgerouter have not been tested.



======= Bugs ========



       New Bugs
            -11850[1]  [Autobuilder] Eclipse-plugin files are not found in autobuilder
            -11873[2] These TCs need to be added to testopia Distrodata.test_checkpkg and Wic.test_mkfs_extraopts
            -11879[3] Testimage: multiple "uvesafb" errors present in log in minnowmax on genericx86/genericx86-64
            -11855[4] Failed to build libgcc-initial on Fedora26 i686
            -11859[5] [Test Case 1883] TestImage.test_testimage_dnf Centos7
            -11868[6] wic.Wic.test_mkfs_extraopts - Testcase -1 opensuse422
            -11869[7] FAILED runqemu.RunqemuTests.test_boot_deploy - Testcase 2007: Ubuntu1604
            -11871[8] distrodata.Distrodata.test_checkpkg - Testcase -1: ubuntu1604
            -11857[9] FAIL [23.106s]: test_mkfs_extraopts (wic.Wic) centos7

            -11834[10] if you have a bad layer in bblayers, bitbake server connection hangs





Full Bug Report : https://wiki.yoctoproject.org/wiki/WW31_-_2017-07-31_-_Full_Test_Cycle_2.4_M2_rc2#Bugs_Found_during_QA_Test





======== Links =========



    1.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11850 [1]

    2.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11873 <https://bugzilla.yoctoproject.org/show_bug.cgi?id=10477%20> [2]

    3.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11879 <https://bugzilla.yoctoproject.org/show_bug.cgi?id=10832%20> [3]

    4.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11855 <https://bugzilla.yoctoproject.org/show_bug.cgi?id=11811%20> [4]

    5.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11859 <https://bugzilla.yoctoproject.org/show_bug.cgi?id=11812%20> [5]

    6.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11868<https://bugzilla.yoctoproject.org/show_bug.cgi?id=11813> [6]

    7.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11869 [7]

    8.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11871 [8]

    9.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11857 [9]

    10.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11834 [10]

    11.    https://bugzilla.yoctoproject.org/show_bug.cgi?id=11889 [11]





Regards

Libertad G.

[-- Attachment #2: Type: text/html, Size: 32993 bytes --]

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: QA cycle report for 2.4 M2 rc2
  2017-08-03  0:55 QA cycle report for 2.4 M2 rc2 Cruz, Libertad
@ 2017-08-03 10:19 ` Richard Purdie
  0 siblings, 0 replies; 2+ messages in thread
From: Richard Purdie @ 2017-08-03 10:19 UTC (permalink / raw)
  To: Cruz, Libertad, yocto, Eggleton, Paul, Jolley, Stephen K, Zhang, Jessica
  Cc: Cruz Alcaraz, Juan M

On Wed, 2017-08-02 at 17:55 -0700, Cruz, Libertad wrote:
> QA does not approve that  the workarounds provided  for 2.4 m2 rc2
>  were on different commits. Testing was not limited to 2.4 m2 rc2
> commit “8e15e9b6e478f6368034519b2a8fd3c7ea71d23b” it was performed by
> following each commit on the provided workarounds described below.  

Thanks Libertad.

The different commits wasn't intentional, there were bugs which were
reported and fixed after rc2 was built. It made sense to resolve those
bugs as fixed in master but obviously they were still present in M2.

Given the high volume of issues and other things that went on with rc2
I've decided to build an rc3. We'll have to take current master as
there are some many fixes it only makes sense to use that rather than
create any special branch. I've set that build away.

Cheers,

Richard


^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2017-08-03 10:19 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-03  0:55 QA cycle report for 2.4 M2 rc2 Cruz, Libertad
2017-08-03 10:19 ` Richard Purdie

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.