[RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[Taras Kondratiuk]

'devtool modify' fails for packages that inherit python3native (e.g.
gpgme or systemd, via meson class):
| Exception: ModuleNotFoundError: No module named '_sysconfigdata'

After commit 02714c105426 ("python3: upgrade to 3.7.2") python3native
class exports _PYTHON_SYSCONFIGDATA_NAME = '_sysconfigdata'. I think the
expectation is that it will find
usr/lib/python-sysconfigdata/_sysconfigdata.py prepared by the same
commit. But some places seems to still use host python3 which doesn't
have _sysconfigdata.py. This leads to "No module named '_sysconfigdata'"
errors. E.g. commit 4b26eaf7152f ("prservice.py: fix do_package with
newer Python in Ubuntu 20.04") recently tried to workaround it for PR
service. And now we see similar issue in devtool. To unblock devtool
apply similar workaround here.

But looks like there is some deeper issue here. I haven't fully grasped
the purpose of _sysconfigdata.py yet. It is copied from
_sysconfigdata_m_linux_x86_64-linux-gnu.py and then some variables are
modified. But new values seems to be wrong. For example:
In _sysconfigdata_m_linux_x86_64-linux-gnu.py:
'INCLUDEPY': 'FIXMESTAGINGDIRHOST/usr/include/python3.7m',
While in _sysconfigdata.py:
'INCLUDEPY': 'FIXMESTAGINGDIRHOSTFIXMESTAGINGDIRHOST/usr/include/python3.7m',

When FIXMESTAGINGDIRHOST is replaces by staging.bbclass, the path in
_sysconfigdata.py points to nowhere while
_sysconfigdata_m_linux_x86_64-linux-gnu.py has a correct path.

Signed-off-by: Taras Kondratiuk <takondra@cisco.com>
---
 meta/classes/devtool-source.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/devtool-source.bbclass b/meta/classes/devtool-source.bbclass
index 280d6009f3c2..4a469bfacad7 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -65,6 +65,11 @@ python() {
 
 
 python devtool_post_unpack() {
+    # Otherwise this fails when called from recipes which e.g. inherit
+    # python3native (which sets _PYTHON_SYSCONFIGDATA_NAME) with: No module
+    # named '_sysconfigdata'
+    if '_PYTHON_SYSCONFIGDATA_NAME' in os.environ:
+        del os.environ['_PYTHON_SYSCONFIGDATA_NAME']
     import oe.recipeutils
     import shutil
     sys.path.insert(0, os.path.join(d.getVar('COREBASE'), 'scripts', 'lib'))
-- 
2.25.1

Re: [RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 3012 bytes --]

I am able to do 'devtool modify gpgme' without any errors. In what setting
does it fail for you?

The purpose of _sysconfigdata is to substitute configuration values that
are specific for target python when running native python in cross builds
with requests from target component build systems to supply those values.
For that to work, the recipe also needs to have target python3 in its
DEPENDS. It

I feel the patch is fixing the symptom rather than the issue somehow?


Alex

On Fri, 10 Jul 2020 at 21:08, Taras Kondratiuk <takondra@cisco.com> wrote:

> 'devtool modify' fails for packages that inherit python3native (e.g.
> gpgme or systemd, via meson class):
> | Exception: ModuleNotFoundError: No module named '_sysconfigdata'
>
> After commit 02714c105426 ("python3: upgrade to 3.7.2") python3native
> class exports _PYTHON_SYSCONFIGDATA_NAME = '_sysconfigdata'. I think the
> expectation is that it will find
> usr/lib/python-sysconfigdata/_sysconfigdata.py prepared by the same
> commit. But some places seems to still use host python3 which doesn't
> have _sysconfigdata.py. This leads to "No module named '_sysconfigdata'"
> errors. E.g. commit 4b26eaf7152f ("prservice.py: fix do_package with
> newer Python in Ubuntu 20.04") recently tried to workaround it for PR
> service. And now we see similar issue in devtool. To unblock devtool
> apply similar workaround here.
>
> But looks like there is some deeper issue here. I haven't fully grasped
> the purpose of _sysconfigdata.py yet. It is copied from
> _sysconfigdata_m_linux_x86_64-linux-gnu.py and then some variables are
> modified. But new values seems to be wrong. For example:
> In _sysconfigdata_m_linux_x86_64-linux-gnu.py:
> 'INCLUDEPY': 'FIXMESTAGINGDIRHOST/usr/include/python3.7m',
> While in _sysconfigdata.py:
> 'INCLUDEPY':
> 'FIXMESTAGINGDIRHOSTFIXMESTAGINGDIRHOST/usr/include/python3.7m',
>
> When FIXMESTAGINGDIRHOST is replaces by staging.bbclass, the path in
> _sysconfigdata.py points to nowhere while
> _sysconfigdata_m_linux_x86_64-linux-gnu.py has a correct path.
>
> Signed-off-by: Taras Kondratiuk <takondra@cisco.com>
> ---
>  meta/classes/devtool-source.bbclass | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/meta/classes/devtool-source.bbclass
> b/meta/classes/devtool-source.bbclass
> index 280d6009f3c2..4a469bfacad7 100644
> --- a/meta/classes/devtool-source.bbclass
> +++ b/meta/classes/devtool-source.bbclass
> @@ -65,6 +65,11 @@ python() {
>
>
>  python devtool_post_unpack() {
> +    # Otherwise this fails when called from recipes which e.g. inherit
> +    # python3native (which sets _PYTHON_SYSCONFIGDATA_NAME) with: No
> module
> +    # named '_sysconfigdata'
> +    if '_PYTHON_SYSCONFIGDATA_NAME' in os.environ:
> +        del os.environ['_PYTHON_SYSCONFIGDATA_NAME']
>      import oe.recipeutils
>      import shutil
>      sys.path.insert(0, os.path.join(d.getVar('COREBASE'), 'scripts',
> 'lib'))
> --
> 2.25.1
>
>

[-- Attachment #2: Type: text/html, Size: 3726 bytes --]

Re: [OE-core] [RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[Richard Purdie]

On Fri, 2020-07-10 at 22:41 +0200, Alexander Kanavin wrote:
> I am able to do 'devtool modify gpgme' without any errors. In what
> setting does it fail for you?
> 
> The purpose of _sysconfigdata is to substitute configuration values
> that are specific for target python when running native python in
> cross builds with requests from target component build systems to
> supply those values. For that to work, the recipe also needs to have
> target python3 in its DEPENDS. It 
> 
> I feel the patch is fixing the symptom rather than the issue somehow?

We've had other patches like this and were playing "whack-a-mole" since
as soon as one merged, an issue appeared somewhere else. In the end I
stopped taking them and asked the issue be debugged and fixed properly.
There should be more info in the archives about what I suggested/asked
for to move forward with this...

Cheers,

Richard

Re: [OE-core] [RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[Taras Kondratiuk]

On Fri, 2020-07-10 at 22:41 +0200, Alexander Kanavin wrote:
> I am able to do 'devtool modify gpgme' without any errors. In what
> setting does it fail for you?

Forgot to mention: it is reproducible with host Python 3.6+. The Python
should have the patch that handles _PYTHON_SYSCONFIGDATA_NAME:
https://github.com/python/cpython/commit/92dec548ff14

I'm currently reproducing this on the latest Poky on Ubuntu 20.04 (has
Python 3.8.2). This is also reproducible on previous branches, like
Zeus.

> The purpose of _sysconfigdata is to substitute configuration values
> that are specific for target python when running native python in
> cross builds with requests from target component build systems to
> supply those values. For that to work, the recipe also needs to have
> target python3 in its DEPENDS. It 

What configuration does it aim to replace? When I compare default
_sysconfigdata__linux_x86_64-linux-gnu.py with _sysconfigdata.py, then
I see this:

% diff -u ./usr/lib/python3.8/_sysconfigdata__linux_x86_64-linux-gnu.py ./usr/lib/python-sysconfigdata/_sysconfigdata.py
--- ./usr/lib/python3.8/_sysconfigdata__linux_x86_64-linux-gnu.py       2020-07-10 01:33:47.193574344 -0700
+++ ./usr/lib/python-sysconfigdata/_sysconfigdata.py    2020-07-10 01:33:47.197574384 -0700
@@ -577,12 +577,12 @@
  'HAVE_ZLIB_COPY': 1,
  'HAVE__GETPTY': 0,
  'HOST_GNU_TYPE': 'x86_64-pc-linux-gnu',
- 'INCLDIRSTOMAKE': 'FIXMESTAGINGDIRHOST/usr/include '
-                   'FIXMESTAGINGDIRHOST/usr/include '
+ 'INCLDIRSTOMAKE': 'FIXMESTAGINGDIRHOSTFIXMESTAGINGDIRHOST/usr/include '
+                   'FIXMESTAGINGDIRHOSTFIXMESTAGINGDIRHOST/usr/include '
                    'FIXMESTAGINGDIRHOST/usr/include/python3.8 '
                    'FIXMESTAGINGDIRHOST/usr/include/python3.8',
  'INCLUDEDIR': 'FIXMESTAGINGDIRHOST/usr/include',
- 'INCLUDEPY': 'FIXMESTAGINGDIRHOST/usr/include/python3.8',
+ 'INCLUDEPY': 'FIXMESTAGINGDIRHOSTFIXMESTAGINGDIRHOST/usr/include/python3.8',
  'INSTALL': 'FIXME_HOSTTOOLS_DIR/install -c',
  'INSTALL_DATA': 'FIXME_HOSTTOOLS_DIR/install -c -m '
                  '644',

The only replaced parameters seems to be replaced with wrong values. Also
FIXMESTAGINGDIRHOST will point to recipe-sysroot-native which is not
target-specific. What am I missing?

> I feel the patch is fixing the symptom rather than the issue somehow

Agree, it is a workaround, hence I've posted it as RFC. Note, there is
similar workaround already merged for PR server 4b26eaf7152f
("prservice.py: fix do_package with newer Python in Ubuntu 20.04").

Re: [OE-core] [RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 471 bytes --]

On Sat, 11 Jul 2020 at 00:03, Taras Kondratiuk (takondra) <
takondra@cisco.com> wrote:

>
> The only replaced parameters seems to be replaced with wrong values. Also
> FIXMESTAGINGDIRHOST will point to recipe-sysroot-native which is not
> target-specific. What am I missing?
>

You are comparing with _sysconfigdata from native sysroot. You need to take
_sysconfigdata from target sysroot, and then you will see quite a few
arch-specific differences.

Alex

[-- Attachment #2: Type: text/html, Size: 774 bytes --]

Re: [OE-core] [RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[Taras Kondratiuk]

On Sat, 2020-07-11 at 00:41 +0200, Alexander Kanavin wrote:
> On Sat, 11 Jul 2020 at 00:03, Taras Kondratiuk (takondra) <
> takondra@cisco.com> wrote:
> > The only replaced parameters seems to be replaced with wrong
> > values. Also
> > FIXMESTAGINGDIRHOST will point to recipe-sysroot-native which is
> > not
> > target-specific. What am I missing?
> 
> You are comparing with _sysconfigdata from native sysroot. You need
> to take _sysconfigdata from target sysroot, and then you will see
> quite a few arch-specific differences.

Oh, thanks. Now it makes a bit more sense. I didn't realize that
python3native class uses target-specific configuration and target
sysroot. Shouldn't it be called something like python3cross then?

Now need to figure out how to prevent _PYTHON_SYSCONFIGDATA_NAME from
python3native bleeding into other tools.

Re: [OE-core] [RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[Taras Kondratiuk]

On Fri, 2020-07-10 at 22:16 +0100, Richard Purdie wrote:
> On Fri, 2020-07-10 at 22:41 +0200, Alexander Kanavin wrote:
> > I am able to do 'devtool modify gpgme' without any errors. In what
> > setting does it fail for you?
> > 
> > The purpose of _sysconfigdata is to substitute configuration values
> > that are specific for target python when running native python in
> > cross builds with requests from target component build systems to
> > supply those values. For that to work, the recipe also needs to
> > have
> > target python3 in its DEPENDS. It 
> > 
> > I feel the patch is fixing the symptom rather than the issue
> > somehow?
> 
> We've had other patches like this and were playing "whack-a-mole"
> since
> as soon as one merged, an issue appeared somewhere else. In the end I
> stopped taking them and asked the issue be debugged and fixed
> properly.
> There should be more info in the archives about what I
> suggested/asked
> for to move forward with this...

Is this the one you are referring to?
https://lists.yoctoproject.org/g/yocto/topic/74637733#49571

I like Chris' approach. I'll try to look how to make it work with glib
and a few other packages that unset _PYTHON_SYSCONFIGDATA_NAME.

Re: [OE-core] [RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1259 bytes --]

On Sat, 11 Jul 2020 at 02:58, Taras Kondratiuk (takondra) <
takondra@cisco.com> wrote:

> Oh, thanks. Now it makes a bit more sense. I didn't realize that
> python3native class uses target-specific configuration and target
> sysroot. Shouldn't it be called something like python3cross then?
>

Nope - the purpose of python3native itself is to provide native python to
recipes for which host python may not be good enough (for example when they
need 3rd party modules, or exact version supplied by oe-core).

If they also poke into target configuration, then that is provided by
adding a DEPENDS on *target* python3, and having this patch that makes
native python load the target sysconfigdata:
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/python/python3/0001-distutils-sysconfig-append-STAGING_LIBDIR-python-sys.patch

I understand it is not intuitive, but this is one of those legacy oe-core
things that nobody has time to clean up. You are welcome to do so. I didn't
invent this approach, I just replicated it when I rewrote the python3
recipe from scratch, throwing away all the other baggage that nobody
remembered the purpose of, so that python3 could be upgraded again without
going insane :)

Alex

[-- Attachment #2: Type: text/html, Size: 1788 bytes --]

Re: [OE-core] [RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[Taras Kondratiuk]

On Sat, 2020-07-11 at 09:42 +0200, Alexander Kanavin wrote:
> On Sat, 11 Jul 2020 at 02:58, Taras Kondratiuk (takondra) <
> takondra@cisco.com> wrote:
> > Oh, thanks. Now it makes a bit more sense. I didn't realize that
> > python3native class uses target-specific configuration and target
> > sysroot. Shouldn't it be called something like python3cross then?
> 
> Nope - the purpose of python3native itself is to provide native
> python to recipes for which host python may not be good enough (for
> example when they need 3rd party modules, or exact version supplied
> by oe-core).

IIUC python3native by itself is currently broken. See case #2 below.

> If they also poke into target configuration, then that is provided by
> adding a DEPENDS on *target* python3, and having this patch that
> makes native python load the target sysconfigdata:
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-devtools/python/python3/0001-distutils-sysconfig-append-STAGING_LIBDIR-python-sys.patch
> 
> I understand it is not intuitive, but this is one of those legacy oe-
> core things that nobody has time to clean up. You are welcome to do
> so. I didn't invent this approach, I just replicated it when I
> rewrote the python3 recipe from scratch, throwing away all the other
> baggage that nobody remembered the purpose of, so that python3 could
> be upgraded again without going insane :)

There seems to be at least three distinct use-cases and exporting
_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata breaks two of them:

1. If a recipe needs python3-native and needs to work with python
target configuration it DEPENDS on python3 (to get python3 and
_sysconfigdata.py in recipe-sysroot) and inherits python3native. This
works as intended.

2. If a recipe needs python3-native only, then it inherits
python3native which exports _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata.
This doesn't work, because python3-native will fail to
find _sysconfigdata.py in recipe-sysroot (because there is no python3
in DEPENDS). As a workaround several recipes started to unset
_PYTHON_SYSCONFIGDATA_NAME (glib [1], mesa [2], scons class [3]).

3. Some tools use host python under recipe's environment. Host python
fails when those recipes inherit python3native with exported
_PYTHON_SYSCONFIGDATA_NAME. Some tools started to workaround it also by
unsetting _PYTHON_SYSCONFIGDATA_NAME (git-make-shallow [4], prservice
[5]).

Chris' approach [6] would address case #3, but not #2. I'm not sure yet
what is the best way to address #2. Maybe export
_PYTHON_SYSCONFIGDATA_NAME only if DEPENDS has python3?

[1] 
https://git.openembedded.org/openembedded-core/tree/meta/recipes-core/glib-2.0/glib.inc?id=e185235dd97510bfdc621cef9c18d8d13b16006d#n40
[2] 
https://git.openembedded.org/openembedded-core/commit/?id=8fc255f7d63c53167c771cd6b78895784f37d33f
[3] 
https://git.openembedded.org/openembedded-core/commit/?id=44f303ba9fb193a985e8e4b7c6962883ae3970d1
[4] 
https://git.openembedded.org/bitbake/commit/?id=d94ccd506d04aff182ab48f501f6f366d5dd14f5
[5] 
https://git.openembedded.org/openembedded-core/commit/?id=4b26eaf7152fb712aba47a0c746333578f58ee8d
[6] 
https://github.com/openembedded/openembedded-core/compare/master...kergoth:python-sysconfig

Re: [OE-core] [RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1382 bytes --]

On Tue, 14 Jul 2020 at 03:47, Taras Kondratiuk (takondra) <
takondra@cisco.com> wrote:

> 2. If a recipe needs python3-native only, then it inherits
> python3native which exports _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata.
> This doesn't work, because python3-native will fail to
> find _sysconfigdata.py in recipe-sysroot (because there is no python3
> in DEPENDS). As a workaround several recipes started to unset
> _PYTHON_SYSCONFIGDATA_NAME (glib [1], mesa [2], scons class [3]).
>

This is not the case. There are plenty of recipes which inherit
python3native, and do not need to unset anything.
The reason glib or mesa have to do it is that they use meson build system
to find the python3 binary (only to run
some scripts; target configuration is not used), and meson always probes
into target config, even when that config is
simply thrown away. Adding target python3 to DEPENDS is undesirable as that
will lengthen the build dependency chain
without any actual need for the target python3.

Chris' approach [6] would address case #3, but not #2. I'm not sure yet
> what is the best way to address #2. Maybe export
> _PYTHON_SYSCONFIGDATA_NAME only if DEPENDS has python3?
>

I am not sure either. I do not have a clear idea how to sort this mess.
Maybe we should talk to python upstream and ask
how cross-compilation is supposed to work.

Alex

[-- Attachment #2: Type: text/html, Size: 1913 bytes --]

Re: [OE-core] [RFC PATCH] devtool: remove _PYTHON_SYSCONFIGDATA_NAME to fix do_unpack

[Taras Kondratiuk]

On Tue, 2020-07-14 at 10:35 +0200, Alexander Kanavin wrote:
> On Tue, 14 Jul 2020 at 03:47, Taras Kondratiuk (takondra) <
> takondra@cisco.com> wrote:
> > 2. If a recipe needs python3-native only, then it inherits
> > python3native which exports
> > _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata.
> > This doesn't work, because python3-native will fail to
> > find _sysconfigdata.py in recipe-sysroot (because there is no
> > python3
> > in DEPENDS). As a workaround several recipes started to unset
> > _PYTHON_SYSCONFIGDATA_NAME (glib [1], mesa [2], scons class [3]).
> 
> This is not the case. There are plenty of recipes which inherit
> python3native, and do not need to unset anything.
> The reason glib or mesa have to do it is that they use meson build
> system to find the python3 binary (only to run
> some scripts; target configuration is not used), and meson always
> probes into target config, even when that config is
> simply thrown away. Adding target python3 to DEPENDS is undesirable
> as that will lengthen the build dependency chain
> without any actual need for the target python3.

Right..., my testcase had sysconfig invocation. So the correct
statement would be: sysconfig in python3native doesn't work unless
python3 is in DEPENDS.

> 
> > Chris' approach [6] would address case #3, but not #2. I'm not sure
> > yet
> > what is the best way to address #2. Maybe export
> > _PYTHON_SYSCONFIGDATA_NAME only if DEPENDS has python3?
> 
> I am not sure either. I do not have a clear idea how to sort this
> mess. Maybe we should talk to python upstream and ask
> how cross-compilation is supposed to work.

Thanks, I'll try to look into this deeper.