[OE-core][PATCH] python3: parallelize ptests, add test_cppext dependencies

[OE-core][PATCH] python3: parallelize ptests, add test_cppext dependencies

[Trevor Gamblin]

This is based on a previous patch from Alex Kanavin (thanks Alex),
but it retains our use of sed to parse the test output into a usable
form. The intent was to use the test module's --junit-xml feature and
parse the resulting log instead of using sed, but various errors were
encountered when testing this method, so that will need further
investigation before we can consider adopting it. Two missing ptest
dependencies on python3-cgitb and python3-zipapp were already merged
in a previous patch, so only gcc, g++, and binutils were left to add
as ptest RDEPENDS.

ptest output and runtime with this change:

== Tests result: SUCCESS ==

405 tests OK.

29 tests skipped:
    test_asdl_parser test_check_c_globals test_clinic test_curses
    test_devpoll test_gdb test_idle test_ioctl test_kqueue
    test_launcher test_msilib test_ossaudiodev test_readline
    test_smtpnet test_socketserver test_startfile test_tcl test_tix
    test_tk test_ttk_guionly test_ttk_textonly test_turtle
    test_urllib2net test_urllibnet test_winconsoleio test_winreg
    test_winsound test_xmlrpc_net test_zipfile64

Total duration: 5 min 3 sec

[YOCTO #13298]

Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
---
 meta/recipes-core/images/core-image-ptest.bb   | 1 +
 meta/recipes-devtools/python/python3/run-ptest | 2 +-
 meta/recipes-devtools/python/python3_3.11.4.bb | 2 +-
 3 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/images/core-image-ptest.bb b/meta/recipes-core/images/core-image-ptest.bb
index ddc56c8f9f..74cf933b72 100644
--- a/meta/recipes-core/images/core-image-ptest.bb
+++ b/meta/recipes-core/images/core-image-ptest.bb
@@ -26,6 +26,7 @@ IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-lttng-tools = "1524288"
 # ptests need more memory than standard to avoid the OOM killer
 QB_MEM = "-m 1024"
 QB_MEM:virtclass-mcextend-lttng-tools = "-m 4096"
+QB_MEM:virtclass-mcextend-python3 = "-m 2048"
 QB_MEM:virtclass-mcextend-python3-cryptography = "-m 4096"
 
 TEST_SUITES = "ping ssh parselogs ptest"
diff --git a/meta/recipes-devtools/python/python3/run-ptest b/meta/recipes-devtools/python/python3/run-ptest
index 05396e91ab..ee1130284b 100644
--- a/meta/recipes-devtools/python/python3/run-ptest
+++ b/meta/recipes-devtools/python/python3/run-ptest
@@ -1,3 +1,3 @@
 #!/bin/sh
 
-SETUPTOOLS_USE_DISTUTILS=nonlocal python3 -m test -v | sed -u -e '/\.\.\. ok/ s/^/PASS: /g' -r -e '/\.\.\. (ERROR|FAIL)/ s/^/FAIL: /g' -e '/\.\.\. skipped/ s/^/SKIP: /g' -e 's/ \.\.\. ok//g' -e 's/ \.\.\. ERROR//g' -e 's/ \.\.\. FAIL//g' -e 's/ \.\.\. skipped//g'
+SETUPTOOLS_USE_DISTUTILS=nonlocal python3 -m test -v -j 4 | sed -u -e '/\.\.\. ok/ s/^/PASS: /g' -r -e '/\.\.\. (ERROR|FAIL)/ s/^/FAIL: /g' -e '/\.\.\. skipped/ s/^/SKIP: /g' -e 's/ \.\.\. ok//g' -e 's/ \.\.\. ERROR//g' -e 's/ \.\.\. FAIL//g' -e 's/ \.\.\. skipped//g'
diff --git a/meta/recipes-devtools/python/python3_3.11.4.bb b/meta/recipes-devtools/python/python3_3.11.4.bb
index 86ecc4e297..2d7b1bac2c 100644
--- a/meta/recipes-devtools/python/python3_3.11.4.bb
+++ b/meta/recipes-devtools/python/python3_3.11.4.bb
@@ -426,7 +426,7 @@ FILES:${PN}-man = "${datadir}/man"
 # See https://bugs.python.org/issue18748 and https://bugs.python.org/issue37395
 RDEPENDS:libpython3:append:libc-glibc = " libgcc"
 RDEPENDS:${PN}-ctypes:append:libc-glibc = " ${MLPREFIX}ldconfig"
-RDEPENDS:${PN}-ptest = "${PN}-modules ${PN}-tests ${PN}-dev ${PN}-cgitb ${PN}-zipapp unzip bzip2 libgcc tzdata coreutils sed"
+RDEPENDS:${PN}-ptest = "${PN}-modules ${PN}-tests ${PN}-dev ${PN}-cgitb ${PN}-zipapp unzip bzip2 libgcc tzdata coreutils sed gcc g++ binutils"
 RDEPENDS:${PN}-ptest:append:libc-glibc = " locale-base-fr-fr locale-base-en-us locale-base-tr-tr locale-base-de-de"
 RDEPENDS:${PN}-tkinter += "${@bb.utils.contains('PACKAGECONFIG', 'tk', '${MLPREFIX}tk ${MLPREFIX}tk-lib', '', d)}"
 RDEPENDS:${PN}-idle += "${@bb.utils.contains('PACKAGECONFIG', 'tk', '${PN}-tkinter ${MLPREFIX}tcl', '', d)}"
-- 
2.41.0

Re: [OE-core][PATCH] python3: parallelize ptests, add test_cppext dependencies

[Alexander Kanavin]

I would suggest just one more tweak here: insert
|| echo "FAIL: python3"
after python3 but before sed. That way it'll catch failures that sed
does not (that silent regressing did just happen with
zipapp/cgitb/cppext).

Alex

On Tue, 11 Jul 2023 at 19:25, Trevor Gamblin <tgamblin@baylibre.com> wrote:
>
> This is based on a previous patch from Alex Kanavin (thanks Alex),
> but it retains our use of sed to parse the test output into a usable
> form. The intent was to use the test module's --junit-xml feature and
> parse the resulting log instead of using sed, but various errors were
> encountered when testing this method, so that will need further
> investigation before we can consider adopting it. Two missing ptest
> dependencies on python3-cgitb and python3-zipapp were already merged
> in a previous patch, so only gcc, g++, and binutils were left to add
> as ptest RDEPENDS.
>
> ptest output and runtime with this change:
>
> == Tests result: SUCCESS ==
>
> 405 tests OK.
>
> 29 tests skipped:
>     test_asdl_parser test_check_c_globals test_clinic test_curses
>     test_devpoll test_gdb test_idle test_ioctl test_kqueue
>     test_launcher test_msilib test_ossaudiodev test_readline
>     test_smtpnet test_socketserver test_startfile test_tcl test_tix
>     test_tk test_ttk_guionly test_ttk_textonly test_turtle
>     test_urllib2net test_urllibnet test_winconsoleio test_winreg
>     test_winsound test_xmlrpc_net test_zipfile64
>
> Total duration: 5 min 3 sec
>
> [YOCTO #13298]
>
> Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
> ---
>  meta/recipes-core/images/core-image-ptest.bb   | 1 +
>  meta/recipes-devtools/python/python3/run-ptest | 2 +-
>  meta/recipes-devtools/python/python3_3.11.4.bb | 2 +-
>  3 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/meta/recipes-core/images/core-image-ptest.bb b/meta/recipes-core/images/core-image-ptest.bb
> index ddc56c8f9f..74cf933b72 100644
> --- a/meta/recipes-core/images/core-image-ptest.bb
> +++ b/meta/recipes-core/images/core-image-ptest.bb
> @@ -26,6 +26,7 @@ IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-lttng-tools = "1524288"
>  # ptests need more memory than standard to avoid the OOM killer
>  QB_MEM = "-m 1024"
>  QB_MEM:virtclass-mcextend-lttng-tools = "-m 4096"
> +QB_MEM:virtclass-mcextend-python3 = "-m 2048"
>  QB_MEM:virtclass-mcextend-python3-cryptography = "-m 4096"
>
>  TEST_SUITES = "ping ssh parselogs ptest"
> diff --git a/meta/recipes-devtools/python/python3/run-ptest b/meta/recipes-devtools/python/python3/run-ptest
> index 05396e91ab..ee1130284b 100644
> --- a/meta/recipes-devtools/python/python3/run-ptest
> +++ b/meta/recipes-devtools/python/python3/run-ptest
> @@ -1,3 +1,3 @@
>  #!/bin/sh
>
> -SETUPTOOLS_USE_DISTUTILS=nonlocal python3 -m test -v | sed -u -e '/\.\.\. ok/ s/^/PASS: /g' -r -e '/\.\.\. (ERROR|FAIL)/ s/^/FAIL: /g' -e '/\.\.\. skipped/ s/^/SKIP: /g' -e 's/ \.\.\. ok//g' -e 's/ \.\.\. ERROR//g' -e 's/ \.\.\. FAIL//g' -e 's/ \.\.\. skipped//g'
> +SETUPTOOLS_USE_DISTUTILS=nonlocal python3 -m test -v -j 4 | sed -u -e '/\.\.\. ok/ s/^/PASS: /g' -r -e '/\.\.\. (ERROR|FAIL)/ s/^/FAIL: /g' -e '/\.\.\. skipped/ s/^/SKIP: /g' -e 's/ \.\.\. ok//g' -e 's/ \.\.\. ERROR//g' -e 's/ \.\.\. FAIL//g' -e 's/ \.\.\. skipped//g'
> diff --git a/meta/recipes-devtools/python/python3_3.11.4.bb b/meta/recipes-devtools/python/python3_3.11.4.bb
> index 86ecc4e297..2d7b1bac2c 100644
> --- a/meta/recipes-devtools/python/python3_3.11.4.bb
> +++ b/meta/recipes-devtools/python/python3_3.11.4.bb
> @@ -426,7 +426,7 @@ FILES:${PN}-man = "${datadir}/man"
>  # See https://bugs.python.org/issue18748 and https://bugs.python.org/issue37395
>  RDEPENDS:libpython3:append:libc-glibc = " libgcc"
>  RDEPENDS:${PN}-ctypes:append:libc-glibc = " ${MLPREFIX}ldconfig"
> -RDEPENDS:${PN}-ptest = "${PN}-modules ${PN}-tests ${PN}-dev ${PN}-cgitb ${PN}-zipapp unzip bzip2 libgcc tzdata coreutils sed"
> +RDEPENDS:${PN}-ptest = "${PN}-modules ${PN}-tests ${PN}-dev ${PN}-cgitb ${PN}-zipapp unzip bzip2 libgcc tzdata coreutils sed gcc g++ binutils"
>  RDEPENDS:${PN}-ptest:append:libc-glibc = " locale-base-fr-fr locale-base-en-us locale-base-tr-tr locale-base-de-de"
>  RDEPENDS:${PN}-tkinter += "${@bb.utils.contains('PACKAGECONFIG', 'tk', '${MLPREFIX}tk ${MLPREFIX}tk-lib', '', d)}"
>  RDEPENDS:${PN}-idle += "${@bb.utils.contains('PACKAGECONFIG', 'tk', '${PN}-tkinter ${MLPREFIX}tcl', '', d)}"
> --
> 2.41.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#184149): https://lists.openembedded.org/g/openembedded-core/message/184149
> Mute This Topic: https://lists.openembedded.org/mt/100083497/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core][PATCH] python3: parallelize ptests, add test_cppext dependencies

[Trevor Gamblin]

On 2023-07-12 14:27, Alexander Kanavin wrote:
> I would suggest just one more tweak here: insert
> || echo "FAIL: python3"
> after python3 but before sed. That way it'll catch failures that sed
> does not (that silent regressing did just happen with
> zipapp/cgitb/cppext).

Sorry for not responding sooner. Trying this out now - it works fine 
with manual tests but doesn't report correctly with testimage:

WARNING: core-image-ptest-python3-1.0-r0 do_testimage: There were 
failing ptests.
Traceback (most recent call last):
   File 
"/workspace/yocto/manual/poky/meta/lib/oeqa/core/decorator/__init__.py", 
line 35, in wrapped_f
     return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
   File 
"/workspace/yocto/manual/poky/meta/lib/oeqa/core/decorator/__init__.py", 
line 35, in wrapped_f
     return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
   File 
"/workspace/yocto/manual/poky/meta/lib/oeqa/core/decorator/__init__.py", 
line 35, in wrapped_f
     return func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
   File 
"/workspace/yocto/manual/poky/meta/lib/oeqa/runtime/cases/ptest.py", 
line 27, in test_ptestrunner_expectfail
     self.do_ptestrunner()
   File 
"/workspace/yocto/manual/poky/meta/lib/oeqa/runtime/cases/ptest.py", 
line 120, in do_ptestrunner
     self.fail(failmsg)
AssertionError:
ptests which had no test results:
['python3']

If I add the echo after sed then it's OK, but that might not be what we 
want.

>
> Alex
>
> On Tue, 11 Jul 2023 at 19:25, Trevor Gamblin <tgamblin@baylibre.com> wrote:
>> This is based on a previous patch from Alex Kanavin (thanks Alex),
>> but it retains our use of sed to parse the test output into a usable
>> form. The intent was to use the test module's --junit-xml feature and
>> parse the resulting log instead of using sed, but various errors were
>> encountered when testing this method, so that will need further
>> investigation before we can consider adopting it. Two missing ptest
>> dependencies on python3-cgitb and python3-zipapp were already merged
>> in a previous patch, so only gcc, g++, and binutils were left to add
>> as ptest RDEPENDS.
>>
>> ptest output and runtime with this change:
>>
>> == Tests result: SUCCESS ==
>>
>> 405 tests OK.
>>
>> 29 tests skipped:
>>      test_asdl_parser test_check_c_globals test_clinic test_curses
>>      test_devpoll test_gdb test_idle test_ioctl test_kqueue
>>      test_launcher test_msilib test_ossaudiodev test_readline
>>      test_smtpnet test_socketserver test_startfile test_tcl test_tix
>>      test_tk test_ttk_guionly test_ttk_textonly test_turtle
>>      test_urllib2net test_urllibnet test_winconsoleio test_winreg
>>      test_winsound test_xmlrpc_net test_zipfile64
>>
>> Total duration: 5 min 3 sec
>>
>> [YOCTO #13298]
>>
>> Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
>> ---
>>   meta/recipes-core/images/core-image-ptest.bb   | 1 +
>>   meta/recipes-devtools/python/python3/run-ptest | 2 +-
>>   meta/recipes-devtools/python/python3_3.11.4.bb | 2 +-
>>   3 files changed, 3 insertions(+), 2 deletions(-)
>>
>> diff --git a/meta/recipes-core/images/core-image-ptest.bb b/meta/recipes-core/images/core-image-ptest.bb
>> index ddc56c8f9f..74cf933b72 100644
>> --- a/meta/recipes-core/images/core-image-ptest.bb
>> +++ b/meta/recipes-core/images/core-image-ptest.bb
>> @@ -26,6 +26,7 @@ IMAGE_ROOTFS_EXTRA_SPACE:virtclass-mcextend-lttng-tools = "1524288"
>>   # ptests need more memory than standard to avoid the OOM killer
>>   QB_MEM = "-m 1024"
>>   QB_MEM:virtclass-mcextend-lttng-tools = "-m 4096"
>> +QB_MEM:virtclass-mcextend-python3 = "-m 2048"
>>   QB_MEM:virtclass-mcextend-python3-cryptography = "-m 4096"
>>
>>   TEST_SUITES = "ping ssh parselogs ptest"
>> diff --git a/meta/recipes-devtools/python/python3/run-ptest b/meta/recipes-devtools/python/python3/run-ptest
>> index 05396e91ab..ee1130284b 100644
>> --- a/meta/recipes-devtools/python/python3/run-ptest
>> +++ b/meta/recipes-devtools/python/python3/run-ptest
>> @@ -1,3 +1,3 @@
>>   #!/bin/sh
>>
>> -SETUPTOOLS_USE_DISTUTILS=nonlocal python3 -m test -v | sed -u -e '/\.\.\. ok/ s/^/PASS: /g' -r -e '/\.\.\. (ERROR|FAIL)/ s/^/FAIL: /g' -e '/\.\.\. skipped/ s/^/SKIP: /g' -e 's/ \.\.\. ok//g' -e 's/ \.\.\. ERROR//g' -e 's/ \.\.\. FAIL//g' -e 's/ \.\.\. skipped//g'
>> +SETUPTOOLS_USE_DISTUTILS=nonlocal python3 -m test -v -j 4 | sed -u -e '/\.\.\. ok/ s/^/PASS: /g' -r -e '/\.\.\. (ERROR|FAIL)/ s/^/FAIL: /g' -e '/\.\.\. skipped/ s/^/SKIP: /g' -e 's/ \.\.\. ok//g' -e 's/ \.\.\. ERROR//g' -e 's/ \.\.\. FAIL//g' -e 's/ \.\.\. skipped//g'
>> diff --git a/meta/recipes-devtools/python/python3_3.11.4.bb b/meta/recipes-devtools/python/python3_3.11.4.bb
>> index 86ecc4e297..2d7b1bac2c 100644
>> --- a/meta/recipes-devtools/python/python3_3.11.4.bb
>> +++ b/meta/recipes-devtools/python/python3_3.11.4.bb
>> @@ -426,7 +426,7 @@ FILES:${PN}-man = "${datadir}/man"
>>   # See https://bugs.python.org/issue18748 and https://bugs.python.org/issue37395
>>   RDEPENDS:libpython3:append:libc-glibc = " libgcc"
>>   RDEPENDS:${PN}-ctypes:append:libc-glibc = " ${MLPREFIX}ldconfig"
>> -RDEPENDS:${PN}-ptest = "${PN}-modules ${PN}-tests ${PN}-dev ${PN}-cgitb ${PN}-zipapp unzip bzip2 libgcc tzdata coreutils sed"
>> +RDEPENDS:${PN}-ptest = "${PN}-modules ${PN}-tests ${PN}-dev ${PN}-cgitb ${PN}-zipapp unzip bzip2 libgcc tzdata coreutils sed gcc g++ binutils"
>>   RDEPENDS:${PN}-ptest:append:libc-glibc = " locale-base-fr-fr locale-base-en-us locale-base-tr-tr locale-base-de-de"
>>   RDEPENDS:${PN}-tkinter += "${@bb.utils.contains('PACKAGECONFIG', 'tk', '${MLPREFIX}tk ${MLPREFIX}tk-lib', '', d)}"
>>   RDEPENDS:${PN}-idle += "${@bb.utils.contains('PACKAGECONFIG', 'tk', '${PN}-tkinter ${MLPREFIX}tcl', '', d)}"
>> --
>> 2.41.0
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#184149): https://lists.openembedded.org/g/openembedded-core/message/184149
>> Mute This Topic: https://lists.openembedded.org/mt/100083497/1686489
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

Re: [OE-core][PATCH] python3: parallelize ptests, add test_cppext dependencies

[Alexander Kanavin]

On Mon, 17 Jul 2023 at 15:06, Trevor Gamblin <tgamblin@baylibre.com> wrote:
> AssertionError:
> ptests which had no test results:
> ['python3']

This happens when the test prints no PASS: and no FAIL:, this is used
to catch regressed ptests that don't test anything but don't return an
error either.

Can you run run-ptest manually and check what it prints? There should
be at least one PASS:

Alex

Re: [OE-core][PATCH] python3: parallelize ptests, add test_cppext dependencies

[Trevor Gamblin]

On 2023-07-17 16:16, Alexander Kanavin wrote:
> On Mon, 17 Jul 2023 at 15:06, Trevor Gamblin <tgamblin@baylibre.com> wrote:
>> AssertionError:
>> ptests which had no test results:
>> ['python3']
> This happens when the test prints no PASS: and no FAIL:, this is used
> to catch regressed ptests that don't test anything but don't return an
> error either.
>
> Can you run run-ptest manually and check what it prints? There should
> be at least one PASS:
Fixed it. Needed to have curly brackets around the OR statement for it 
to print. Since the patch is already merged, I've sent a new one to add 
this in.
>
> Alex