* ptest with BBCLASSEXTEND
@ 2019-05-16 10:54 Richard Purdie
2019-05-16 12:36 ` Richard Purdie
2019-05-17 16:22 ` Jonathan Rajotte-Julien
0 siblings, 2 replies; 11+ messages in thread
From: Richard Purdie @ 2019-05-16 10:54 UTC (permalink / raw)
To: openembedded-core
We have a need to check the dependencies of our ptest packages. The
best way to do that is to take a minimal image and run them under that
in isolation. That would be a bit tedious and boring.
It got me thinking about creative ways to do that and I thought I'd
share the result.
I created an image recipe:
$ cat p-t-image.bb
require recipes-core/images/core-image-minimal.bb
require conf/distro/include/ptest-packagelists.inc
PTESTS = "${PTESTS_SLOW} ${PTESTS_FAST}"
ALL_PTESTS = "${@' '.join(['p-t-image-'+x for x in d.getVar('PTESTS').split()])}"
IMAGE_INSTALL_append = " ${MCNAME} openssh"
BBCLASSEXTEND = "${@' '.join(['mcextend:'+x for x in d.getVar('PTESTS').split()])}"
IMAGE_OVERHEAD_FACTOR = "1.0"
IMAGE_ROOTFS_EXTRA_SPACE = "1324288"
# ptests need more memory than standard to avoid the OOM killer
QB_MEM = "-m 1024"
Its not fast to parse this creates about 59 variants of the image. Its
(ab)using the mcextend class to an image and add the ptest to that
image for each ptest in the PTESTS_SLOW and PTESTS_FAST variables. It
also adds openssh since we need an ssh client in the images for
testimage to work. I also had to add extra space and memory to the
image config so all the tests could work.
You can get a list of the target images with:
bitbake p-t-image-elfutils-ptest -e | grep ALL_PTESTS=
then run:
bitbake <targets>
and then
bitbake <targets> -c testimage
which batch build and then runs all the images in parallel.
The results also get aggregated automatically into the json file so if
you delete it before starting, the json file is equavalient from a
ptest perspective to if they'd all been run in the same image.
The results I got from this are:
--------------------------------------------------------------------------------------------------------------
Recipe | Passed | Failed | Skipped | Time(s)
--------------------------------------------------------------------------------------------------------------
acl | 1 | 1 | 0 | 0
apr | 0 | 0 | 0 | -
apr-util | 4 | 0 | 0 | -
attr | 0 | 0 | 0 | 0
bash | 36 | 8 | 0 | 107 T
bluez5 | 23 | 2 | 0 | 5
busybox | 528 | 4 | 76 | 4
bzip2 | 0 | 0 | 0 | 1
dbus-test | 0 | 0 | 0 | 91 T
diffstat | 439 | 0 | 0 | 4
diffutils | 19 | 0 | 2 | 2
e2fsprogs | 0 | 0 | 0 | 78
elfutils | 172 | 24 | 7 | 21
ethtool | 2 | 0 | 0 | 0
flex | 0 | 0 | 0 | 1
gawk | 342 | 0 | 0 | 2
gdbm | 24 | 0 | 0 | 1
gdk-pixbuf | 20 | 0 | 0 | 39
gettext | 0 | 0 | 0 | 1
glib-2.0 | 242 | 3 | 0 | 344
gstreamer1.0 | 103 | 1 | 0 | 105
gzip | 0 | 0 | 0 | 1
json-glib | 14 | 0 | 0 | 1
kbd | 5 | 2 | 0 | 0
libconvert-asn1-perl | 0 | 22 | 0 | 0
liberror-perl | 0 | 15 | 0 | 1
libevent | 1 | 0 | 0 | 86
libpcre | 0 | 0 | 0 | 0
libtest-needs-perl | 0 | 2 | 0 | 0
libtimedate-perl | 3 | 2 | 0 | 0
liburi-perl | 0 | 39 | 2 | 0
libusb1 | 4 | 0 | 0 | 6
libxml-namespacesupport-perl | 0 | 1 | 0 | 0
libxml-parser-perl | 0 | 15 | 0 | 0
libxml-perl | 0 | 6 | 0 | 0
libxml-sax-base-perl | 0 | 19 | 0 | 1
libxml-sax-perl | 0 | 14 | 0 | 0
libxml-simple-perl | 0 | 14 | 0 | 0
libxml2 | 134 | 0 | 0 | 16
lttng-tools | 3397 | 626 | 1 | 348 T
lzo | 5 | 0 | 0 | 2
m4 | 0 | 0 | 0 | 1
nettle | 97 | 0 | 2 | 10
openssh | 0 | 0 | 1 | 1
openssl | 146 | 1 | 8 | 78
opkg | 0 | 0 | 0 | 11
pango | 12 | 0 | 0 | 1
parted | 0 | 0 | 0 | 0
perl | 2423 | 19 | 238 | 703
python | 302 | 5 | 41 | 330
python3 | 30277 | 2 | 1043 | 1005
quilt | 0 | 57 | 0 | 1
sed | 86 | 0 | 0 | 4
slang | 0 | 0 | 0 | 1
strace | 466 | 18 | 147 | 179
tcl | 149 | 0 | 0 | 183
util-linux | 336 | 2 | 80 | 40
valgrind | 149 | 1 | 9 | 537
zlib | 1 | 0 | 0 | 1
which can be compared against:
https://autobuilder.yocto.io/pub/non-release/20190514-14/testresults/testresult-report.txt
which means we have a fair number of ptest recipes with dependency problems:
--------------------------------------------------------------------------------------------------------------
Recipe | Passed | Failed | Skipped | Time(s)
--------------------------------------------------------------------------------------------------------------
acl | 1 | 1 | 0 | 0
apr | 0 | 0 | 0 | -
apr-util | 4 | 0 | 0 | -
attr | 0 | 0 | 0 | 0
bash | 36 | 8 | 0 | 107 T
bluez5 | 23 | 2 | 0 | 5
bzip2 | 0 | 0 | 0 | 1
dbus-test | 0 | 0 | 0 | 91 T
diffutils | 19 | 0 | 2 | 2
e2fsprogs | 0 | 0 | 0 | 78
elfutils | 172 | 24 | 7 | 21
ethtool | 2 | 0 | 0 | 0
flex | 0 | 0 | 0 | 1
gettext | 0 | 0 | 0 | 1
glib-2.0 | 242 | 3 | 0 | 344
gzip | 0 | 0 | 0 | 1
kbd | 5 | 2 | 0 | 0
libconvert-asn1-perl | 0 | 22 | 0 | 0
liberror-perl | 0 | 15 | 0 | 1
libpcre | 0 | 0 | 0 | 0
libtest-needs-perl | 0 | 2 | 0 | 0
libtimedate-perl | 3 | 2 | 0 | 0
liburi-perl | 0 | 39 | 2 | 0
libxml-namespacesupport-perl | 0 | 1 | 0 | 0
libxml-parser-perl | 0 | 15 | 0 | 0
libxml-perl | 0 | 6 | 0 | 0
libxml-sax-base-perl | 0 | 19 | 0 | 1
libxml-sax-perl | 0 | 14 | 0 | 0
libxml-simple-perl | 0 | 14 | 0 | 0
lttng-tools | 3397 | 626 | 1 | 348 T
m4 | 0 | 0 | 0 | 1
openssh | 0 | 0 | 1 | 1
opkg | 0 | 0 | 0 | 11
parted | 0 | 0 | 0 | 0
python | 302 | 5 | 41 | 330
python3 | 30277 | 2 | 1043 | 1005
quilt | 0 | 57 | 0 | 1
slang | 0 | 0 | 0 | 1
util-linux | 336 | 2 | 80 | 40
The logs from this are available here:
https://www.rpsys.net/wp/rp/testresults.json.gz
(can be extracted with resulttool log)
Anyhow, I thought I'd share this slightly crazy but IMO fairly neat
trick :)
Cheers,
Richard
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: ptest with BBCLASSEXTEND 2019-05-16 10:54 ptest with BBCLASSEXTEND Richard Purdie @ 2019-05-16 12:36 ` Richard Purdie 2019-05-16 21:13 ` Randy MacLeod 2019-05-17 16:22 ` Jonathan Rajotte-Julien 1 sibling, 1 reply; 11+ messages in thread From: Richard Purdie @ 2019-05-16 12:36 UTC (permalink / raw) To: openembedded-core On Thu, 2019-05-16 at 11:54 +0100, Richard Purdie wrote: > https://autobuilder.yocto.io/pub/non-release/20190514-14/testresults/testresult-report.txt A couple of patches (on the list) gets us to the results below which is better. The perl module failures and quilt are from missing perl module dependencies. -------------------------------------------------------------------------------------------------------------- Recipe | Passed | Failed | Skipped | Time(s) -------------------------------------------------------------------------------------------------------------- acl | 3 | 0 | 0 | 0 apr | 44 | 0 | 0 | 58 apr-util | 19 | 0 | 0 | 8 attr | 0 | 0 | 0 | 0 bash | 36 | 8 | 0 | 108 T bluez5 | 23 | 2 | 0 | 5 busybox | 528 | 4 | 76 | 4 bzip2 | 6 | 0 | 0 | 1 dbus-test | 0 | 0 | 0 | 91 T diffstat | 439 | 0 | 0 | 3 diffutils | 19 | 0 | 2 | 2 e2fsprogs | 0 | 0 | 0 | 108 T elfutils | 172 | 24 | 7 | 20 ethtool | 2 | 0 | 0 | 1 flex | 114 | 0 | 0 | 2 gawk | 342 | 0 | 0 | 2 gdbm | 24 | 0 | 0 | 2 gdk-pixbuf | 20 | 0 | 0 | 39 gettext | 354 | 21 | 37 | 33 glib-2.0 | 242 | 3 | 0 | 347 gstreamer1.0 | 103 | 1 | 0 | 104 gzip | 19 | 0 | 3 | 4 json-glib | 14 | 0 | 0 | 0 kbd | 5 | 2 | 0 | 1 libconvert-asn1-perl | 0 | 22 | 0 | 0 liberror-perl | 0 | 15 | 0 | 0 libevent | 1 | 0 | 0 | 86 libpcre | 0 | 0 | 0 | 1 libtest-needs-perl | 0 | 2 | 0 | 0 libtimedate-perl | 3 | 2 | 0 | 0 liburi-perl | 0 | 39 | 2 | 0 libusb1 | 4 | 0 | 0 | 6 libxml-namespacesupport-perl | 0 | 1 | 0 | 0 libxml-parser-perl | 0 | 15 | 0 | 0 libxml-perl | 0 | 6 | 0 | 0 libxml-sax-base-perl | 0 | 19 | 0 | 0 libxml-sax-perl | 0 | 14 | 0 | 0 libxml-simple-perl | 0 | 14 | 0 | 0 libxml2 | 134 | 0 | 0 | 17 lttng-tools | 3397 | 626 | 1 | 350 T lzo | 5 | 0 | 0 | 2 m4 | 144 | 8 | 18 | 5 nettle | 97 | 0 | 2 | 9 openssh | 0 | 0 | 1 | 1 openssl | 146 | 1 | 8 | 91 opkg | 0 | 0 | 0 | 11 pango | 12 | 0 | 0 | 1 parted | 41 | 1 | 22 | 5 perl | 2423 | 19 | 238 | 713 python | 301 | 6 | 41 | 325 python3 | 30277 | 2 | 1043 | 990 quilt | 0 | 57 | 0 | 0 sed | 86 | 0 | 0 | 4 slang | 97 | 0 | 0 | 22 strace | 466 | 18 | 147 | 176 tcl | 149 | 0 | 0 | 179 util-linux | 336 | 2 | 80 | 41 valgrind | 149 | 1 | 9 | 544 zlib | 1 | 0 | 0 | 1 Cheers, Richard ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ptest with BBCLASSEXTEND 2019-05-16 12:36 ` Richard Purdie @ 2019-05-16 21:13 ` Randy MacLeod 2019-05-16 22:28 ` richard.purdie 2019-05-17 1:58 ` Changqing Li 0 siblings, 2 replies; 11+ messages in thread From: Randy MacLeod @ 2019-05-16 21:13 UTC (permalink / raw) To: Richard Purdie, openembedded-core, Li, Changqing, Sakib Sajal On 5/16/19 8:36 AM, Richard Purdie wrote: > On Thu, 2019-05-16 at 11:54 +0100, Richard Purdie wrote: >> https://autobuilder.yocto.io/pub/non-release/20190514-14/testresults/testresult-report.txt > > A couple of patches (on the list) gets us to the results below which is > better. The perl module failures and quilt are from missing perl module > dependencies. > > -------------------------------------------------------------------------------------------------------------- > Recipe | Passed | Failed | Skipped | Time(s) > -------------------------------------------------------------------------------------------------------------- > acl | 3 | 0 | 0 | 0 > apr | 44 | 0 | 0 | 58 > apr-util | 19 | 0 | 0 | 8 > attr | 0 | 0 | 0 | 0 > bash | 36 | 8 | 0 | 108 T > bluez5 | 23 | 2 | 0 | 5 > busybox | 528 | 4 | 76 | 4 > bzip2 | 6 | 0 | 0 | 1 > dbus-test | 0 | 0 | 0 | 91 T > diffstat | 439 | 0 | 0 | 3 > diffutils | 19 | 0 | 2 | 2 > e2fsprogs | 0 | 0 | 0 | 108 T > elfutils | 172 | 24 | 7 | 20 > ethtool | 2 | 0 | 0 | 1 > flex | 114 | 0 | 0 | 2 > gawk | 342 | 0 | 0 | 2 > gdbm | 24 | 0 | 0 | 2 > gdk-pixbuf | 20 | 0 | 0 | 39 > gettext | 354 | 21 | 37 | 33 > glib-2.0 | 242 | 3 | 0 | 347 > gstreamer1.0 | 103 | 1 | 0 | 104 > gzip | 19 | 0 | 3 | 4 > json-glib | 14 | 0 | 0 | 0 > kbd | 5 | 2 | 0 | 1 > libconvert-asn1-perl | 0 | 22 | 0 | 0 > liberror-perl | 0 | 15 | 0 | 0 > libevent | 1 | 0 | 0 | 86 > libpcre | 0 | 0 | 0 | 1 > libtest-needs-perl | 0 | 2 | 0 | 0 > libtimedate-perl | 3 | 2 | 0 | 0 > liburi-perl | 0 | 39 | 2 | 0 > libusb1 | 4 | 0 | 0 | 6 > libxml-namespacesupport-perl | 0 | 1 | 0 | 0 > libxml-parser-perl | 0 | 15 | 0 | 0 > libxml-perl | 0 | 6 | 0 | 0 > libxml-sax-base-perl | 0 | 19 | 0 | 0 > libxml-sax-perl | 0 | 14 | 0 | 0 > libxml-simple-perl | 0 | 14 | 0 | 0 > libxml2 | 134 | 0 | 0 | 17 > lttng-tools | 3397 | 626 | 1 | 350 T > lzo | 5 | 0 | 0 | 2 > m4 | 144 | 8 | 18 | 5 > nettle | 97 | 0 | 2 | 9 > openssh | 0 | 0 | 1 | 1 > openssl | 146 | 1 | 8 | 91 > opkg | 0 | 0 | 0 | 11 > pango | 12 | 0 | 0 | 1 > parted | 41 | 1 | 22 | 5 > perl | 2423 | 19 | 238 | 713 > python | 301 | 6 | 41 | 325 > python3 | 30277 | 2 | 1043 | 990 > quilt | 0 | 57 | 0 | 0 > sed | 86 | 0 | 0 | 4 > slang | 97 | 0 | 0 | 22 > strace | 466 | 18 | 147 | 176 > tcl | 149 | 0 | 0 | 179 > util-linux | 336 | 2 | 80 | 41 > valgrind | 149 | 1 | 9 | 544 > zlib | 1 | 0 | 0 | 1 > > Cheers, > > Richard > During the weekly YP bug review meeting, we discussed creating a defect per package for each of these ptest failures and marking them each as NEWCOMER bugs but we really need to keep working on ptest improvements. I'd like Sandy (Changqing), Sajal and perhaps others at Wind River (or elsewhere!) to split up the list above to fix any missing dependencies. Sajal is a new student working at working from Ottawa. He has some experience with Yocto from a previous student placement. I've asked him to work on the bash package ptest errors and I'll add quilt and strace as well. For the rest of the list, Sandy can pick a few and see if there's a pattern to the failures or not. Once we have that info we can decide how many people we need to deal with the rest of the packages for fix most of the dependency issues over the coming weeks. It would be nice to have most of them wrapped up for 2.8-M1. -- # Randy MacLeod # Wind River Linux ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ptest with BBCLASSEXTEND 2019-05-16 21:13 ` Randy MacLeod @ 2019-05-16 22:28 ` richard.purdie 2019-05-21 22:31 ` Richard Purdie 2019-05-22 22:28 ` Jonathan Rajotte-Julien 2019-05-17 1:58 ` Changqing Li 1 sibling, 2 replies; 11+ messages in thread From: richard.purdie @ 2019-05-16 22:28 UTC (permalink / raw) To: Randy MacLeod, openembedded-core, Li, Changqing, Sakib Sajal On Thu, 2019-05-16 at 17:13 -0400, Randy MacLeod wrote: > On 5/16/19 8:36 AM, Richard Purdie wrote: > > On Thu, 2019-05-16 at 11:54 +0100, Richard Purdie wrote: > > > https://autobuilder.yocto.io/pub/non-release/20190514-14/testresults/testresult-report.txt > > > > A couple of patches (on the list) gets us to the results below which is > > better. The perl module failures and quilt are from missing perl module > > dependencies. > > > > -------------------------------------------------------------------------------------------------------------- > > Recipe | Passed | Failed | Skipped | Time(s) > > -------------------------------------------------------------------------------------------------------------- > > acl | 3 | 0 | 0 | 0 > > apr | 44 | 0 | 0 | 58 > > apr-util | 19 | 0 | 0 | 8 > > attr | 0 | 0 | 0 | 0 > > bash | 36 | 8 | 0 | 108 T > > bluez5 | 23 | 2 | 0 | 5 > > busybox | 528 | 4 | 76 | 4 > > bzip2 | 6 | 0 | 0 | 1 > > dbus-test | 0 | 0 | 0 | 91 T > > diffstat | 439 | 0 | 0 | 3 > > diffutils | 19 | 0 | 2 | 2 > > e2fsprogs | 0 | 0 | 0 | 108 T > > elfutils | 172 | 24 | 7 | 20 > > ethtool | 2 | 0 | 0 | 1 > > flex | 114 | 0 | 0 | 2 > > gawk | 342 | 0 | 0 | 2 > > gdbm | 24 | 0 | 0 | 2 > > gdk-pixbuf | 20 | 0 | 0 | 39 > > gettext | 354 | 21 | 37 | 33 > > glib-2.0 | 242 | 3 | 0 | 347 > > gstreamer1.0 | 103 | 1 | 0 | 104 > > gzip | 19 | 0 | 3 | 4 > > json-glib | 14 | 0 | 0 | 0 > > kbd | 5 | 2 | 0 | 1 > > libconvert-asn1-perl | 0 | 22 | 0 | 0 > > liberror-perl | 0 | 15 | 0 | 0 > > libevent | 1 | 0 | 0 | 86 > > libpcre | 0 | 0 | 0 | 1 > > libtest-needs-perl | 0 | 2 | 0 | 0 > > libtimedate-perl | 3 | 2 | 0 | 0 > > liburi-perl | 0 | 39 | 2 | 0 > > libusb1 | 4 | 0 | 0 | 6 > > libxml-namespacesupport-perl | 0 | 1 | 0 | 0 > > libxml-parser-perl | 0 | 15 | 0 | 0 > > libxml-perl | 0 | 6 | 0 | 0 > > libxml-sax-base-perl | 0 | 19 | 0 | 0 > > libxml-sax-perl | 0 | 14 | 0 | 0 > > libxml-simple-perl | 0 | 14 | 0 | 0 > > libxml2 | 134 | 0 | 0 | 17 > > lttng-tools | 3397 | 626 | 1 | 350 T > > lzo | 5 | 0 | 0 | 2 > > m4 | 144 | 8 | 18 | 5 > > nettle | 97 | 0 | 2 | 9 > > openssh | 0 | 0 | 1 | 1 > > openssl | 146 | 1 | 8 | 91 > > opkg | 0 | 0 | 0 | 11 > > pango | 12 | 0 | 0 | 1 > > parted | 41 | 1 | 22 | 5 > > perl | 2423 | 19 | 238 | 713 > > python | 301 | 6 | 41 | 325 > > python3 | 30277 | 2 | 1043 | 990 > > quilt | 0 | 57 | 0 | 0 > > sed | 86 | 0 | 0 | 4 > > slang | 97 | 0 | 0 | 22 > > strace | 466 | 18 | 147 | 176 > > tcl | 149 | 0 | 0 | 179 > > util-linux | 336 | 2 | 80 | 41 > > valgrind | 149 | 1 | 9 | 544 > > zlib | 1 | 0 | 0 | 1 > > > > Cheers, > > > > Richard > > > > During the weekly YP bug review meeting, we discussed creating a > defect per package for each of these ptest failures and marking them > each as NEWCOMER bugs but we really need to keep working on ptest > improvements. I'd like Sandy (Changqing), Sajal and perhaps others > at Wind River (or elsewhere!) to split up the list above to fix > any missing dependencies. > > Sajal is a new student working at working from Ottawa. He has some > experience with Yocto from a previous student placement. I've asked him > to work on the bash package ptest errors and I'll add quilt and strace > as well. > > For the rest of the list, Sandy can pick a few and see if there's > a pattern to the failures or not. Once we have that info we can > decide how many people we need to deal with the rest of the > packages for fix most of the dependency issues over the coming > weeks. It would be nice to have most of them wrapped up for 2.8-M1. Thanks Randy! Just to update from my perspective: quilt and lib*-perl all look like missing perl module dependencies. The place those dependencies are missing may be for the ptest or may be the actual PN package itself, depending on where the module dependency is. Ross looks like he's looking at the glib-2.0 issues. I think Jonathan Rajotte might be able to help us with lttng-tools. I have a local patch for acl which looks like a missing e2fsprogs- mke2fs and make dependencies but I've not tested it yet. The failures with timeouts (T next to them) are the ones which worry me most of all. Cheers, Richard ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ptest with BBCLASSEXTEND 2019-05-16 22:28 ` richard.purdie @ 2019-05-21 22:31 ` Richard Purdie 2019-06-03 3:10 ` Changqing Li 2019-05-22 22:28 ` Jonathan Rajotte-Julien 1 sibling, 1 reply; 11+ messages in thread From: Richard Purdie @ 2019-05-21 22:31 UTC (permalink / raw) To: Randy MacLeod, openembedded-core, Li, Changqing, Sakib Sajal On Thu, 2019-05-16 at 23:28 +0100, richard.purdie@linuxfoundation.org wrote: > Just to update from my perspective: > > quilt and lib*-perl all look like missing perl module dependencies. > The place those dependencies are missing may be for the ptest or may > be the actual PN package itself, depending on where the module > dependency is. I've sent out a further patch series which resolves some of the issues. Some quilt tests now pass, some still fail. The perl modules fail in different ways to before, there are clearly more missing dependencies to be added. A quick summary of roughly the recipes still with problems (not 100% accurately filtered): ============================================================================================================== PTest Result Summary ============================================================================================================== -------------------------------------------------------------------------------------------------------------- Recipe | Passed | Failed | Skipped | Time(s) -------------------------------------------------------------------------------------------------------------- bash | 41 | 3 | 0 | 107 T bluez5 | 23 | 2 | 0 | 5 dbus-test | 0 | 0 | 0 | 91 T diffutils | 19 | 0 | 2 | 1 e2fsprogs | 0 | 0 | 0 | 108 T elfutils | 172 | 24 | 7 | 19 gettext | 354 | 21 | 37 | 33 glib-2.0 | 245 | 0 | 0 | 367 gstreamer1.0 | 103 | 1 | 0 | 103 gzip | 19 | 0 | 3 | 3 kbd | 5 | 2 | 0 | 0 libconvert-asn1-perl | 0 | 22 | 0 | 0 liberror-perl | 2 | 13 | 0 | 0 libpcre | 3 | 0 | 0 | 2 libtest-needs-perl | 0 | 2 | 0 | 0 libtimedate-perl | 3 | 2 | 0 | 0 liburi-perl | 0 | 39 | 2 | 0 libxml-namespacesupport-perl | 0 | 1 | 0 | 0 libxml-parser-perl | 0 | 15 | 0 | 0 libxml-perl | 0 | 6 | 0 | 0 libxml-sax-base-perl | 2 | 17 | 0 | 1 libxml-sax-perl | 0 | 14 | 0 | 0 libxml-simple-perl | 0 | 14 | 0 | 0 lttng-tools | 3397 | 626 | 1 | 350 T m4 | 144 | 8 | 18 | 5 nettle | 97 | 0 | 2 | 8 openssh | 15 | 1 | 1 | 138 openssl | 146 | 1 | 8 | 87 opkg | 0 | 0 | 0 | 9 parted | 41 | 1 | 22 | 4 perl | 2423 | 19 | 238 | 727 python | 301 | 6 | 41 | 325 python3 | 30277 | 2 | 1043 | 1005 quilt | 10 | 47 | 0 | 24 tcl | 149 | 0 | 0 | 177 valgrind | 149 | 1 | 9 | 552 -------------------------------------------------------------------------------------------------------------- Cheers, Richard ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ptest with BBCLASSEXTEND 2019-05-21 22:31 ` Richard Purdie @ 2019-06-03 3:10 ` Changqing Li 2019-06-03 13:02 ` richard.purdie 0 siblings, 1 reply; 11+ messages in thread From: Changqing Li @ 2019-06-03 3:10 UTC (permalink / raw) To: Richard Purdie, Randy MacLeod, openembedded-core, Sakib Sajal On 5/22/19 6:31 AM, Richard Purdie wrote: > On Thu, 2019-05-16 at 23:28 +0100, richard.purdie@linuxfoundation.org > wrote: >> Just to update from my perspective: >> >> quilt and lib*-perl all look like missing perl module dependencies. >> The place those dependencies are missing may be for the ptest or may >> be the actual PN package itself, depending on where the module >> dependency is. > I've sent out a further patch series which resolves some of the issues. > Some quilt tests now pass, some still fail. The perl modules fail in > different ways to before, there are clearly more missing dependencies > to be added. > > A quick summary of roughly the recipes still with problems (not 100% > accurately filtered): > > ============================================================================================================== > PTest Result Summary > ============================================================================================================== > -------------------------------------------------------------------------------------------------------------- > Recipe | Passed | Failed | Skipped | Time(s) > -------------------------------------------------------------------------------------------------------------- > bash | 41 | 3 | 0 | 107 T > bluez5 | 23 | 2 | 0 | 5 > dbus-test | 0 | 0 | 0 | 91 T > diffutils | 19 | 0 | 2 | 1 > e2fsprogs | 0 | 0 | 0 | 108 T > elfutils | 172 | 24 | 7 | 19 > gettext | 354 | 21 | 37 | 33 > glib-2.0 | 245 | 0 | 0 | 367 > gstreamer1.0 | 103 | 1 | 0 | 103 > gzip | 19 | 0 | 3 | 3 > kbd | 5 | 2 | 0 | 0 > libconvert-asn1-perl | 0 | 22 | 0 | 0 > liberror-perl | 2 | 13 | 0 | 0 > libpcre | 3 | 0 | 0 | 2 > libtest-needs-perl | 0 | 2 | 0 | 0 > libtimedate-perl | 3 | 2 | 0 | 0 > liburi-perl | 0 | 39 | 2 | 0 > libxml-namespacesupport-perl | 0 | 1 | 0 | 0 > libxml-parser-perl | 0 | 15 | 0 | 0 > libxml-perl | 0 | 6 | 0 | 0 > libxml-sax-base-perl | 2 | 17 | 0 | 1 > libxml-sax-perl | 0 | 14 | 0 | 0 > libxml-simple-perl | 0 | 14 | 0 | 0 > lttng-tools | 3397 | 626 | 1 | 350 T > m4 | 144 | 8 | 18 | 5 > nettle | 97 | 0 | 2 | 8 > openssh | 15 | 1 | 1 | 138 > openssl | 146 | 1 | 8 | 87 > opkg | 0 | 0 | 0 | 9 > parted | 41 | 1 | 22 | 4 > perl | 2423 | 19 | 238 | 727 > python | 301 | 6 | 41 | 325 > python3 | 30277 | 2 | 1043 | 1005 > quilt | 10 | 47 | 0 | 24 > tcl | 149 | 0 | 0 | 177 > valgrind | 149 | 1 | 9 | 552 > -------------------------------------------------------------------------------------------------------------- > > Cheers, > > Richard > > Randy, Based on this list, except lib*perl and packages that are dealing with by others, I picked some for your reference to evaluate the workload. If more info is needed, feel free to tell me. diffutils depend on perl elfutils: mingli have work on it and send patch to oe-core mail list gettext: depend on certain locales for some tests depend on autoconf for some test gzip: have tests depend on diff, while diff in busybox incompatible with POSIX have tests depend on grep, while grep in busybox don't support context options have tests depend on perl kbd: have tests depend on readlink, busybox's readlink don't support -e option m4: one test depend on perl, some test skip since depend on certain locale/codeset, need to check one by one -- BRs Sandy(Li Changqing) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ptest with BBCLASSEXTEND 2019-06-03 3:10 ` Changqing Li @ 2019-06-03 13:02 ` richard.purdie 0 siblings, 0 replies; 11+ messages in thread From: richard.purdie @ 2019-06-03 13:02 UTC (permalink / raw) To: Changqing Li, Randy MacLeod, openembedded-core, Sakib Sajal On Mon, 2019-06-03 at 11:10 +0800, Changqing Li wrote: > Based on this list, except lib*perl and packages that are dealing > with by others, > > I picked some for your reference to evaluate the workload. If more > info is needed, > > feel free to tell me. If it helps, I reran the minimal image comparison and found the following issues: Recipes where we have problems (differences) in minimal images: bash | 76 | 5 | 0 | 163 diffutils | 19 | 0 | 2 | 1 e2fsprogs | 0 | 0 | 0 | 111 elfutils | 194 | 0 | 9 | 32 gettext | 354 | 21 | 37 | 32 gzip | 19 | 0 | 3 | 3 kbd | 5 | 2 | 0 | 0 m4 | 145 | 7 | 18 | 5 openssh | 15 | 1 | 1 | 145 parted | 41 | 1 | 22 | 5 perl | 2413 | 18 | 245 | 861 python | 301 | 6 | 41 | 327 python3 | 30277 | 2 | 1043 | 1012 Others: acl tests are timing out? gawk shows failures in sdk image but minimal: gawk | 407 | 24 | 0 | 20 gstreamer shows two fewer failures in minimal image: gstreamer1.0 | 104 | 1 | 0 | 104 (I'm using https://autobuilder.yocto.io/pub/non-release/20190601-2/testresults/testresult-report.txt for comparison) There are also some open ptest bugs in bugzilla which I didn't duplicate here. The above list is much smaller than the original particularly now we have the perl module issues resolved. Cheers, Richard > diffutils depend on perl > elfutils: mingli have work on it and send patch to oe-core mail > list > gettext: depend on certain locales for some tests > > depend on autoconf for some test > gzip: have tests depend on diff, while diff in busybox > incompatible with POSIX > have tests depend on grep, while grep in busybox > don't > support context options > have tests depend on perl > kbd: have tests depend on readlink, busybox's readlink don't > support -e option > m4: one test depend on perl, some test skip since depend on > certain locale/codeset, need to check one by one > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ptest with BBCLASSEXTEND 2019-05-16 22:28 ` richard.purdie 2019-05-21 22:31 ` Richard Purdie @ 2019-05-22 22:28 ` Jonathan Rajotte-Julien 2019-05-22 22:33 ` richard.purdie 1 sibling, 1 reply; 11+ messages in thread From: Jonathan Rajotte-Julien @ 2019-05-22 22:28 UTC (permalink / raw) To: richard.purdie; +Cc: openembedded-core Hi, > I think Jonathan Rajotte might be able to help us with lttng-tools. Finally had some time for this. See [1][2]. [2] does bring the matter of forcing lttng-ust support of lttng-tools when ptest is deployed. "lttng-ust" is already set in PACKAGECONFIG by default but is there any way to "enforce" the lttng-ust config & dependency when lttng-tools-ptest is used? [1] http://lists.openembedded.org/pipermail/openembedded-core/2019-May/282733.html [2] http://lists.openembedded.org/pipermail/openembedded-core/2019-May/282734.html Cheers -- Jonathan Rajotte-Julien EfficiOS ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ptest with BBCLASSEXTEND 2019-05-22 22:28 ` Jonathan Rajotte-Julien @ 2019-05-22 22:33 ` richard.purdie 0 siblings, 0 replies; 11+ messages in thread From: richard.purdie @ 2019-05-22 22:33 UTC (permalink / raw) To: Jonathan Rajotte-Julien; +Cc: openembedded-core On Wed, 2019-05-22 at 18:28 -0400, Jonathan Rajotte-Julien wrote: > Hi, > > > I think Jonathan Rajotte might be able to help us with lttng-tools. > > Finally had some time for this. > > See [1][2]. > > [2] does bring the matter of forcing lttng-ust support of lttng-tools > when ptest > is deployed. "lttng-ust" is already set in PACKAGECONFIG by default > but is there any > way to "enforce" the lttng-ust config & dependency when lttng-tools- > ptest is used? > > [1] http://lists.openembedded.org/pipermail/openembedded-core/2019-May/282733.html > [2] http://lists.openembedded.org/pipermail/openembedded-core/2019-May/282734.html Thanks for looking at this. It sounds like we need some runtime dependencies as well as the PACKAGECONFIG option. I think the lttng- tool ust option should be changed: -PACKAGECONFIG[lttng-ust] = "--with-lttng-ust, --without-lttng-ust, lttng-ust" +PACKAGECONFIG[lttng-ust] = "--with-lttng-ust, --without-lttng-ust, lttng-ust,lttng-ust" i.e. force the dependency at runtime as well as buildtime. Cheers, Richard ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ptest with BBCLASSEXTEND 2019-05-16 21:13 ` Randy MacLeod 2019-05-16 22:28 ` richard.purdie @ 2019-05-17 1:58 ` Changqing Li 1 sibling, 0 replies; 11+ messages in thread From: Changqing Li @ 2019-05-17 1:58 UTC (permalink / raw) To: Randy MacLeod, Richard Purdie, openembedded-core, Sakib Sajal On 5/17/19 5:13 AM, Randy MacLeod wrote: > On 5/16/19 8:36 AM, Richard Purdie wrote: >> On Thu, 2019-05-16 at 11:54 +0100, Richard Purdie wrote: >>> https://autobuilder.yocto.io/pub/non-release/20190514-14/testresults/testresult-report.txt >>> >> >> A couple of patches (on the list) gets us to the results below which is >> better. The perl module failures and quilt are from missing perl module >> dependencies. >> >> -------------------------------------------------------------------------------------------------------------- >> >> Recipe | Passed | Failed | Skipped | >> Time(s) >> -------------------------------------------------------------------------------------------------------------- >> >> acl | 3 | 0 | 0 | 0 >> apr | 44 | 0 | 0 | 58 >> apr-util | 19 | 0 | 0 | 8 >> attr | 0 | 0 | 0 | 0 >> bash | 36 | 8 | 0 | >> 108 T >> bluez5 | 23 | 2 | 0 | 5 >> busybox | 528 | 4 | 76 | 4 >> bzip2 | 6 | 0 | 0 | 1 >> dbus-test | 0 | 0 | 0 | >> 91 T >> diffstat | 439 | 0 | 0 | 3 >> diffutils | 19 | 0 | 2 | 2 >> e2fsprogs | 0 | 0 | 0 | >> 108 T >> elfutils | 172 | 24 | 7 | 20 >> ethtool | 2 | 0 | 0 | 1 >> flex | 114 | 0 | 0 | 2 >> gawk | 342 | 0 | 0 | 2 >> gdbm | 24 | 0 | 0 | 2 >> gdk-pixbuf | 20 | 0 | 0 | 39 >> gettext | 354 | 21 | 37 | 33 >> glib-2.0 | 242 | 3 | 0 | 347 >> gstreamer1.0 | 103 | 1 | 0 | 104 >> gzip | 19 | 0 | 3 | 4 >> json-glib | 14 | 0 | 0 | 0 >> kbd | 5 | 2 | 0 | 1 >> libconvert-asn1-perl | 0 | 22 | 0 | 0 >> liberror-perl | 0 | 15 | 0 | 0 >> libevent | 1 | 0 | 0 | 86 >> libpcre | 0 | 0 | 0 | 1 >> libtest-needs-perl | 0 | 2 | 0 | 0 >> libtimedate-perl | 3 | 2 | 0 | 0 >> liburi-perl | 0 | 39 | 2 | 0 >> libusb1 | 4 | 0 | 0 | 6 >> libxml-namespacesupport-perl | 0 | 1 | 0 | 0 >> libxml-parser-perl | 0 | 15 | 0 | 0 >> libxml-perl | 0 | 6 | 0 | 0 >> libxml-sax-base-perl | 0 | 19 | 0 | 0 >> libxml-sax-perl | 0 | 14 | 0 | 0 >> libxml-simple-perl | 0 | 14 | 0 | 0 >> libxml2 | 134 | 0 | 0 | 17 >> lttng-tools | 3397 | 626 | 1 | >> 350 T >> lzo | 5 | 0 | 0 | 2 >> m4 | 144 | 8 | 18 | 5 >> nettle | 97 | 0 | 2 | 9 >> openssh | 0 | 0 | 1 | 1 >> openssl | 146 | 1 | 8 | 91 >> opkg | 0 | 0 | 0 | 11 >> pango | 12 | 0 | 0 | 1 >> parted | 41 | 1 | 22 | 5 >> perl | 2423 | 19 | 238 | 713 >> python | 301 | 6 | 41 | 325 >> python3 | 30277 | 2 | 1043 | 990 >> quilt | 0 | 57 | 0 | 0 >> sed | 86 | 0 | 0 | 4 >> slang | 97 | 0 | 0 | 22 >> strace | 466 | 18 | 147 | 176 >> tcl | 149 | 0 | 0 | 179 >> util-linux | 336 | 2 | 80 | 41 >> valgrind | 149 | 1 | 9 | 544 >> zlib | 1 | 0 | 0 | 1 >> >> Cheers, >> >> Richard >> > > During the weekly YP bug review meeting, we discussed creating a > defect per package for each of these ptest failures and marking them > each as NEWCOMER bugs but we really need to keep working on ptest > improvements. I'd like Sandy (Changqing), Sajal and perhaps others > at Wind River (or elsewhere!) to split up the list above to fix > any missing dependencies. > > Sajal is a new student working at working from Ottawa. He has some > experience with Yocto from a previous student placement. I've asked him > to work on the bash package ptest errors and I'll add quilt and strace > as well. > > For the rest of the list, Sandy can pick a few and see if there's > a pattern to the failures or not. Once we have that info we can > decide how many people we need to deal with the rest of the > packages for fix most of the dependency issues over the coming > weeks. It would be nice to have most of them wrapped up for 2.8-M1. > OK, I will work on this after completing current one. -- BRs Sandy(Li Changqing) ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ptest with BBCLASSEXTEND 2019-05-16 10:54 ptest with BBCLASSEXTEND Richard Purdie 2019-05-16 12:36 ` Richard Purdie @ 2019-05-17 16:22 ` Jonathan Rajotte-Julien 1 sibling, 0 replies; 11+ messages in thread From: Jonathan Rajotte-Julien @ 2019-05-17 16:22 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core > Its not fast to parse this creates about 59 variants of the image. Its > (ab)using the mcextend class to an image and add the ptest to that > image for each ptest in the PTESTS_SLOW and PTESTS_FAST variables. It > also adds openssh since we need an ssh client in the images for > testimage to work. I also had to add extra space and memory to the > image config so all the tests could work. > > You can get a list of the target images with: > For the non-initiated, you will need to have the following in your local.conf for this to work. DISTRO_FEATURES_append = " ptest" That said, I should be able to take a look at the problem related to lttng-tools. Thanks -- Jonathan Rajotte-Julien EfficiOS ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2019-06-03 13:02 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-05-16 10:54 ptest with BBCLASSEXTEND Richard Purdie 2019-05-16 12:36 ` Richard Purdie 2019-05-16 21:13 ` Randy MacLeod 2019-05-16 22:28 ` richard.purdie 2019-05-21 22:31 ` Richard Purdie 2019-06-03 3:10 ` Changqing Li 2019-06-03 13:02 ` richard.purdie 2019-05-22 22:28 ` Jonathan Rajotte-Julien 2019-05-22 22:33 ` richard.purdie 2019-05-17 1:58 ` Changqing Li 2019-05-17 16:22 ` Jonathan Rajotte-Julien
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.