[PATCH 0/1] python3: skip tests requiring tools-sdk

[PATCH 0/1] python3: skip tests requiring tools-sdk

[Tim Orling]

When running python3-ptest for any release since Python 3.8.8 with a minimal
target (e.g. python3-ptest), two test cases will fail:

https://github.com/python/cpython/blob/main/Lib/ctypes/test/test_find.py#L116
https://github.com/python/cpython/blob/main/Lib/ctypes/test/test_find.py#L120

For the autobuilder, these tests have always been running on an image
which set IMAGE_FEATURES to include 'tools-sdk', which includes the
required packagegroup-core-buildessential. Here we add a patch to test
for 'tools-sdk' and skip the tests if it is not present.

The following changes since commit b8723c114b3d9b58d2da1a86d728d176dcbeb4b1:

  nfs-utils: upgrade 2.5.3 -> 2.5.4 (2021-06-17 22:18:41 +0100)

are available in the Git repository at:

  git://push.openembedded.org/openembedded-core-contrib timo/python3-skip-test-find

Tim Orling (1):
  python3: skip tests requiring tools-sdk

 ...pes.test_find-skip-without-tools-sdk.patch | 33 +++++++++++++++++++
 meta/recipes-devtools/python/python3_3.9.5.bb |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch

-- 
2.30.2

[PATCH 1/1] python3: skip tests requiring tools-sdk

[Tim Orling]

Conditionally skip test_ctypes.test_find unless
IMAGE_FEATURES contains 'tools-sdk' as these test
cases require full packagegroup-core-buildessential

Fixes:
  AssertionError: Failed ptests:
  {'python3': ['test_find_library_with_gcc', 'test_find_library_with_ld']}

Signed-off-by: Tim Orling <timothy.t.orling@intel.com>
---
 ...pes.test_find-skip-without-tools-sdk.patch | 33 +++++++++++++++++++
 meta/recipes-devtools/python/python3_3.9.5.bb |  1 +
 2 files changed, 34 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch

diff --git a/meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch b/meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch
new file mode 100644
index 00000000000..7d684b3c43c
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch
@@ -0,0 +1,33 @@
+From 7a2bddfa437be633bb6945d0e6b7d6f27da870ad Mon Sep 17 00:00:00 2001
+From: Tim Orling <timothy.t.orling@intel.com>
+Date: Fri, 18 Jun 2021 11:56:50 -0700
+Subject: [PATCH] test_ctypes.test_find: skip without tools-sdk
+
+These tests need full packagegroup-core-buildessential, the
+easiest way to dynamically check for that is looking for
+'tools-sdk' in IMAGE_FEATURES.
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Tim Orling <timothy.t.orlign@intel.com>
+---
+ Lib/ctypes/test/test_find.py | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Lib/ctypes/test/test_find.py b/Lib/ctypes/test/test_find.py
+index 92ac184..0d009d1 100644
+--- a/Lib/ctypes/test/test_find.py
++++ b/Lib/ctypes/test/test_find.py
+@@ -112,10 +112,12 @@ class FindLibraryLinux(unittest.TestCase):
+                 # LD_LIBRARY_PATH)
+                 self.assertEqual(find_library(libname), 'lib%s.so' % libname)
+ 
++    @unittest.skip("Needs IMAGE_FEATURE += \"tools-sdk\"")
+     def test_find_library_with_gcc(self):
+         with unittest.mock.patch("ctypes.util._findSoname_ldconfig", lambda *args: None):
+             self.assertNotEqual(find_library('c'), None)
+ 
++    @unittest.skip("Needs IMAGE_FEATURE += \"tools-sdk\"")
+     def test_find_library_with_ld(self):
+         with unittest.mock.patch("ctypes.util._findSoname_ldconfig", lambda *args: None), \
+              unittest.mock.patch("ctypes.util._findLib_gcc", lambda *args: None):
diff --git a/meta/recipes-devtools/python/python3_3.9.5.bb b/meta/recipes-devtools/python/python3_3.9.5.bb
index f4002f2516f..1717ff737d4 100644
--- a/meta/recipes-devtools/python/python3_3.9.5.bb
+++ b/meta/recipes-devtools/python/python3_3.9.5.bb
@@ -31,6 +31,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
            file://0020-configure.ac-setup.py-do-not-add-a-curses-include-pa.patch \
            file://0001-Lib-sysconfig.py-use-libdir-values-from-configuratio.patch \
            file://0001-Skip-failing-tests-due-to-load-variability-on-YP-AB.patch \
+           ${@bb.utils.contains('IMAGE_FEATURES', 'tools-sdk', '', 'file://0001-test_ctypes.test_find-skip-without-tools-sdk.patch', d)} \
            "
 
 SRC_URI_append_class-native = " \
-- 
2.30.2

Re: [OE-core] [PATCH 1/1] python3: skip tests requiring tools-sdk

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 3710 bytes --]

Wait. How does this work, if IMAGE_FEATURES are set from image recipes, and
not globally?

I'm afraid this may have to be reverted, as it just made it to master.

Alex

On Sat, 19 Jun 2021 at 02:00, Tim Orling <ticotimo@gmail.com> wrote:

> Conditionally skip test_ctypes.test_find unless
> IMAGE_FEATURES contains 'tools-sdk' as these test
> cases require full packagegroup-core-buildessential
>
> Fixes:
>   AssertionError: Failed ptests:
>   {'python3': ['test_find_library_with_gcc', 'test_find_library_with_ld']}
>
> Signed-off-by: Tim Orling <timothy.t.orling@intel.com>
> ---
>  ...pes.test_find-skip-without-tools-sdk.patch | 33 +++++++++++++++++++
>  meta/recipes-devtools/python/python3_3.9.5.bb |  1 +
>  2 files changed, 34 insertions(+)
>  create mode 100644
> meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch
>
> diff --git
> a/meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch
> b/meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch
> new file mode 100644
> index 00000000000..7d684b3c43c
> --- /dev/null
> +++
> b/meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch
> @@ -0,0 +1,33 @@
> +From 7a2bddfa437be633bb6945d0e6b7d6f27da870ad Mon Sep 17 00:00:00 2001
> +From: Tim Orling <timothy.t.orling@intel.com>
> +Date: Fri, 18 Jun 2021 11:56:50 -0700
> +Subject: [PATCH] test_ctypes.test_find: skip without tools-sdk
> +
> +These tests need full packagegroup-core-buildessential, the
> +easiest way to dynamically check for that is looking for
> +'tools-sdk' in IMAGE_FEATURES.
> +
> +Upstream-Status: Inappropriate [oe-specific]
> +
> +Signed-off-by: Tim Orling <timothy.t.orlign@intel.com>
> +---
> + Lib/ctypes/test/test_find.py | 2 ++
> + 1 file changed, 2 insertions(+)
> +
> +diff --git a/Lib/ctypes/test/test_find.py b/Lib/ctypes/test/test_find.py
> +index 92ac184..0d009d1 100644
> +--- a/Lib/ctypes/test/test_find.py
> ++++ b/Lib/ctypes/test/test_find.py
> +@@ -112,10 +112,12 @@ class FindLibraryLinux(unittest.TestCase):
> +                 # LD_LIBRARY_PATH)
> +                 self.assertEqual(find_library(libname), 'lib%s.so' %
> libname)
> +
> ++    @unittest.skip("Needs IMAGE_FEATURE += \"tools-sdk\"")
> +     def test_find_library_with_gcc(self):
> +         with unittest.mock.patch("ctypes.util._findSoname_ldconfig",
> lambda *args: None):
> +             self.assertNotEqual(find_library('c'), None)
> +
> ++    @unittest.skip("Needs IMAGE_FEATURE += \"tools-sdk\"")
> +     def test_find_library_with_ld(self):
> +         with unittest.mock.patch("ctypes.util._findSoname_ldconfig",
> lambda *args: None), \
> +              unittest.mock.patch("ctypes.util._findLib_gcc", lambda
> *args: None):
> diff --git a/meta/recipes-devtools/python/python3_3.9.5.bb
> b/meta/recipes-devtools/python/python3_3.9.5.bb
> index f4002f2516f..1717ff737d4 100644
> --- a/meta/recipes-devtools/python/python3_3.9.5.bb
> +++ b/meta/recipes-devtools/python/python3_3.9.5.bb
> @@ -31,6 +31,7 @@ SRC_URI = "
> http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
>
> file://0020-configure.ac-setup.py-do-not-add-a-curses-include-pa.patch \
>
> file://0001-Lib-sysconfig.py-use-libdir-values-from-configuratio.patch \
>
> file://0001-Skip-failing-tests-due-to-load-variability-on-YP-AB.patch \
> +           ${@bb.utils.contains('IMAGE_FEATURES', 'tools-sdk', '',
> 'file://0001-test_ctypes.test_find-skip-without-tools-sdk.patch', d)} \
>             "
>
>  SRC_URI_append_class-native = " \
> --
> 2.30.2
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 5136 bytes --]

Re: [OE-core] [PATCH 1/1] python3: skip tests requiring tools-sdk

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 4632 bytes --]

On Mon, Jun 21, 2021 at 5:00 AM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> Wait. How does this work, if IMAGE_FEATURES are set from image recipes,
> and not globally?
>
> I'm afraid this may have to be reverted, as it just made it to master.
>
>
These tests have only been passing (since.python3 3.9.0 upgrade) because
the AB ptest images in one way or another have included
packagegroup-core-buildessential.

I asked RP about adding packagegroup-core-buildessential as RDEPENDS for
python3-ptest and he was concerned about build performance. I tried to
narrow the scope to gcc-symlinks and binutils-symlinks, but that was not
enough to make the tests pass.

As I see it, we have two options, this patch which skips the tests
conditionally, or we UNCONDITIONALLY skip the tests.
If you run python3-ptest (and nothing else) on core-image-minimal, these
test cases will fail, which means we do not have proper RDEPENDS for
python3-ptest.

Open to other ideas.


> Alex
>
> On Sat, 19 Jun 2021 at 02:00, Tim Orling <ticotimo@gmail.com> wrote:
>
>> Conditionally skip test_ctypes.test_find unless
>> IMAGE_FEATURES contains 'tools-sdk' as these test
>> cases require full packagegroup-core-buildessential
>>
>> Fixes:
>>   AssertionError: Failed ptests:
>>   {'python3': ['test_find_library_with_gcc', 'test_find_library_with_ld']}
>>
>> Signed-off-by: Tim Orling <timothy.t.orling@intel.com>
>> ---
>>  ...pes.test_find-skip-without-tools-sdk.patch | 33 +++++++++++++++++++
>>  meta/recipes-devtools/python/python3_3.9.5.bb |  1 +
>>  2 files changed, 34 insertions(+)
>>  create mode 100644
>> meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch
>>
>> diff --git
>> a/meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch
>> b/meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch
>> new file mode 100644
>> index 00000000000..7d684b3c43c
>> --- /dev/null
>> +++
>> b/meta/recipes-devtools/python/python3/0001-test_ctypes.test_find-skip-without-tools-sdk.patch
>> @@ -0,0 +1,33 @@
>> +From 7a2bddfa437be633bb6945d0e6b7d6f27da870ad Mon Sep 17 00:00:00 2001
>> +From: Tim Orling <timothy.t.orling@intel.com>
>> +Date: Fri, 18 Jun 2021 11:56:50 -0700
>> +Subject: [PATCH] test_ctypes.test_find: skip without tools-sdk
>> +
>> +These tests need full packagegroup-core-buildessential, the
>> +easiest way to dynamically check for that is looking for
>> +'tools-sdk' in IMAGE_FEATURES.
>> +
>> +Upstream-Status: Inappropriate [oe-specific]
>> +
>> +Signed-off-by: Tim Orling <timothy.t.orlign@intel.com>
>> +---
>> + Lib/ctypes/test/test_find.py | 2 ++
>> + 1 file changed, 2 insertions(+)
>> +
>> +diff --git a/Lib/ctypes/test/test_find.py b/Lib/ctypes/test/test_find.py
>> +index 92ac184..0d009d1 100644
>> +--- a/Lib/ctypes/test/test_find.py
>> ++++ b/Lib/ctypes/test/test_find.py
>> +@@ -112,10 +112,12 @@ class FindLibraryLinux(unittest.TestCase):
>> +                 # LD_LIBRARY_PATH)
>> +                 self.assertEqual(find_library(libname), 'lib%s.so' %
>> libname)
>> +
>> ++    @unittest.skip("Needs IMAGE_FEATURE += \"tools-sdk\"")
>> +     def test_find_library_with_gcc(self):
>> +         with unittest.mock.patch("ctypes.util._findSoname_ldconfig",
>> lambda *args: None):
>> +             self.assertNotEqual(find_library('c'), None)
>> +
>> ++    @unittest.skip("Needs IMAGE_FEATURE += \"tools-sdk\"")
>> +     def test_find_library_with_ld(self):
>> +         with unittest.mock.patch("ctypes.util._findSoname_ldconfig",
>> lambda *args: None), \
>> +              unittest.mock.patch("ctypes.util._findLib_gcc", lambda
>> *args: None):
>> diff --git a/meta/recipes-devtools/python/python3_3.9.5.bb
>> b/meta/recipes-devtools/python/python3_3.9.5.bb
>> index f4002f2516f..1717ff737d4 100644
>> --- a/meta/recipes-devtools/python/python3_3.9.5.bb
>> +++ b/meta/recipes-devtools/python/python3_3.9.5.bb
>> @@ -31,6 +31,7 @@ SRC_URI = "
>> http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
>>
>> file://0020-configure.ac-setup.py-do-not-add-a-curses-include-pa.patch \
>>
>> file://0001-Lib-sysconfig.py-use-libdir-values-from-configuratio.patch \
>>
>> file://0001-Skip-failing-tests-due-to-load-variability-on-YP-AB.patch \
>> +           ${@bb.utils.contains('IMAGE_FEATURES', 'tools-sdk', '',
>> 'file://0001-test_ctypes.test_find-skip-without-tools-sdk.patch', d)} \
>>             "
>>
>>  SRC_URI_append_class-native = " \
>> --
>> 2.30.2
>>
>>
>> 
>>
>>

[-- Attachment #2: Type: text/html, Size: 6543 bytes --]

Re: [OE-core] [PATCH 1/1] python3: skip tests requiring tools-sdk

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1588 bytes --]

On Mon, 21 Jun 2021 at 19:30, Tim Orling <ticotimo@gmail.com> wrote:

>
>
> On Mon, Jun 21, 2021 at 5:00 AM Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
>
>> Wait. How does this work, if IMAGE_FEATURES are set from image recipes,
>> and not globally?
>>
>> I'm afraid this may have to be reverted, as it just made it to master.
>>
>>
> These tests have only been passing (since.python3 3.9.0 upgrade) because
> the AB ptest images in one way or another have included
> packagegroup-core-buildessential.
>
> I asked RP about adding packagegroup-core-buildessential as RDEPENDS for
> python3-ptest and he was concerned about build performance. I tried to
> narrow the scope to gcc-symlinks and binutils-symlinks, but that was not
> enough to make the tests pass.
>
> As I see it, we have two options, this patch which skips the tests
> conditionally, or we UNCONDITIONALLY skip the tests.
> If you run python3-ptest (and nothing else) on core-image-minimal, these
> test cases will fail, which means we do not have proper RDEPENDS for
> python3-ptest.
>

I think the right way out of this is to modify the tests so that they check
for the presence of items that are used by the tests and skip themselfes if
those items (gcc, etc) are not present (turn a hard dependency into a soft
one). Then simply apply the patch without any conditions in the recipe.

There are many other tests which self-skip in a similar way already, we do
not necessarily maintain the full soft dependency set for the fullest ptest
coverage, not should we try to.

Alex

[-- Attachment #2: Type: text/html, Size: 2462 bytes --]

Re: [OE-core] [PATCH 1/1] python3: skip tests requiring tools-sdk

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 1950 bytes --]

On Mon, Jun 21, 2021 at 1:16 PM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> On Mon, 21 Jun 2021 at 19:30, Tim Orling <ticotimo@gmail.com> wrote:
>
>>
>>
>> On Mon, Jun 21, 2021 at 5:00 AM Alexander Kanavin <alex.kanavin@gmail.com>
>> wrote:
>>
>>> Wait. How does this work, if IMAGE_FEATURES are set from image recipes,
>>> and not globally?
>>>
>>> I'm afraid this may have to be reverted, as it just made it to master.
>>>
>>>
>> These tests have only been passing (since.python3 3.9.0 upgrade) because
>> the AB ptest images in one way or another have included
>> packagegroup-core-buildessential.
>>
>> I asked RP about adding packagegroup-core-buildessential as RDEPENDS for
>> python3-ptest and he was concerned about build performance. I tried to
>> narrow the scope to gcc-symlinks and binutils-symlinks, but that was not
>> enough to make the tests pass.
>>
>> As I see it, we have two options, this patch which skips the tests
>> conditionally, or we UNCONDITIONALLY skip the tests.
>> If you run python3-ptest (and nothing else) on core-image-minimal, these
>> test cases will fail, which means we do not have proper RDEPENDS for
>> python3-ptest.
>>
>
> I think the right way out of this is to modify the tests so that they
> check for the presence of items that are used by the tests and skip
> themselfes if those items (gcc, etc) are not present (turn a hard
> dependency into a soft one). Then simply apply the patch without any
> conditions in the recipe.
>
>
I don't disagree, but a simple check for gcc or ld (and ldconfig) is not
enough to pass the tests. I proved this by adding gcc-symlinks and
binutils-symlinks to the RDEPENDS.
It's a bit convoluted in the code base.

There are many other tests which self-skip in a similar way already, we do
> not necessarily maintain the full soft dependency set for the fullest ptest
> coverage, not should we try to.
>
> Alex
>

[-- Attachment #2: Type: text/html, Size: 3329 bytes --]

Re: [OE-core] [PATCH 1/1] python3: skip tests requiring tools-sdk

[Richard Purdie]

On Mon, 2021-06-21 at 10:30 -0700, Tim Orling wrote:
> 
> 
> On Mon, Jun 21, 2021 at 5:00 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> > Wait. How does this work, if IMAGE_FEATURES are set from image recipes, and not globally?
> > 
> > I'm afraid this may have to be reverted, as it just made it to master.
> > 
> > 
> 
> 
> These tests have only been passing (since.python3 3.9.0 upgrade) because the AB ptest images in one way or
> another have included packagegroup-core-buildessential.
> 
> I asked RP about adding packagegroup-core-buildessential as RDEPENDS for python3-ptest and he was concerned
> about build performance. I tried to narrow the scope to gcc-symlinks and binutils-symlinks, but that was not
> enough to make the tests pass.
> 
> As I see it, we have two options, this patch which skips the tests conditionally, or we UNCONDITIONALLY skip
> the tests.
> If you run python3-ptest (and nothing else) on core-image-minimal, these test cases will fail, which means
> we do not have proper RDEPENDS for python3-ptest.
> 
> Open to other ideas.

I think we're all fine with disabling these tests. The issue is that 
IMAGE_FEATURES isn't available in a recipe. As such we may as well 
unconditionally disable them as that is effectively what the current
change does.

Cheers,

Richard

Re: [OE-core] [PATCH 1/1] python3: skip tests requiring tools-sdk

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

On Mon, 21 Jun 2021 at 22:51, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> I think we're all fine with disabling these tests. The issue is that
> IMAGE_FEATURES isn't available in a recipe. As such we may as well
> unconditionally disable them as that is effectively what the current
> change does.
>

I'm ok with this too, although I'd prefer a check with a fallback to a skip
in the test.

Alex

[-- Attachment #2: Type: text/html, Size: 787 bytes --]