[PATCH] libxcb: Add inherit python3native

[PATCH] libxcb: Add inherit python3native

[krellan]

During compilation, the local Python environment leaks into
the Bitbake build. Adding "inherit python3native" fixes this.

Signed-off-by: Josh Lehan <krellan@google.com>
---
 meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
index f0a223a19c..40c7b1cc3c 100644
--- a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
+++ b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
@@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
 FILES_${PN} = "${libdir}/libxcb.so.*"
 
 inherit autotools pkgconfig features_check
+inherit python3native
 
 # The libxau and others requires x11 in DISTRO_FEATURES
 REQUIRED_DISTRO_FEATURES = "x11"
-- 
2.31.1.295.g9ea45b61b8-goog

Re: [OE-core] [PATCH] libxcb: Add inherit python3native

[Richard Purdie]

On Tue, 2021-04-13 at 02:24 -0700, Josh Lehan via lists.openembedded.org wrote:
> During compilation, the local Python environment leaks into
> the Bitbake build. Adding "inherit python3native" fixes this.
> 
> Signed-off-by: Josh Lehan <krellan@google.com>
> ---
>  meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> index f0a223a19c..40c7b1cc3c 100644
> --- a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> +++ b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> @@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
>  FILES_${PN} = "${libdir}/libxcb.so.*"
> 
>  inherit autotools pkgconfig features_check
> +inherit python3native
>  


Could you provide a little more information about how this is breaking things?

Whilst python is used by the libxcb build, I believe it shouldn't change the 
output, it doesn't in our reproducibity tests.

Adding a python-native dependency does slow the build down a bit so if we 
don't need it, it would be good to avoid it.

Cheers,

Richard

Re: [OE-core] [PATCH] libxcb: Add inherit python3native

[Josh Lehan]

[-- Attachment #1: Type: text/plain, Size: 3764 bytes --]

Hi there, thanks. I'm not entirely sure what "python3native" does exactly,
but this is my observation. The problem we are seeing is that, without
"inherit python3native", the build machine's local Python version (whatever
it happens to be) takes precedence over the Yocto-included Python version.
In our case, the build machine is running Python 3.9, and evidently
something changed between Python 3.8 and Python 3.9 to be incompatible with
libxcb. Here's the error message:

Failed to load the xcbgen Python package!
Make sure that xcb/proto installed it on your Python path.
If not, you will need to create a .pth file or define $PYTHONPATH
to extend the path.
Refer to the README file in xcb/proto for more info.

Traceback (most recent call last):
  File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
line 3338, in <module>
    from xcbgen.state import Module
  File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py",
line 7, in <module>
    from xcbgen import matcher
  File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py",
line 12, in <module>
    from xcbgen.xtypes import *
  File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py",
line 5, in <module>
    from xcbgen.align import Alignment, AlignmentLog
  File
"/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py",
line 5, in <module>
    from fractions import gcd
ImportError: cannot import name 'gcd' from 'fractions'
(/usr/lib/python3.9/fractions.py)

The above error text repeats for many of the files it tries to compile
within libxcb.

In general, it seems the safest approach is to standardize on a Python
version that's known to work, without depending on whatever version of
Python the developer has installed locally on their build machine (which
can vary due to distribution, developer's preference, and so on).

Josh

Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941
<(650)%20733-8941>



On Tue, Apr 13, 2021 at 2:45 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Tue, 2021-04-13 at 02:24 -0700, Josh Lehan via lists.openembedded.org
> wrote:
> > During compilation, the local Python environment leaks into
> > the Bitbake build. Adding "inherit python3native" fixes this.
> >
> > Signed-off-by: Josh Lehan <krellan@google.com>
> > ---
> >  meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> > index f0a223a19c..40c7b1cc3c 100644
> > --- a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> > +++ b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> > @@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
> >  FILES_${PN} = "${libdir}/libxcb.so.*"
> >
> >  inherit autotools pkgconfig features_check
> > +inherit python3native
> >
>
>
> Could you provide a little more information about how this is breaking
> things?
>
> Whilst python is used by the libxcb build, I believe it shouldn't change
> the
> output, it doesn't in our reproducibity tests.
>
> Adding a python-native dependency does slow the build down a bit so if we
> don't need it, it would be good to avoid it.
>
> Cheers,
>
> Richard
>
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 5868 bytes --]

Re: [OE-core] [PATCH] libxcb: Add inherit python3native

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 4201 bytes --]

I think you need to dig a little deeper. Why 'from fractions import gcd' is
a problem with host python3.9 but not a problem with native python 3.9?

Alex

On Tue, 13 Apr 2021 at 20:58, Josh Lehan via lists.openembedded.org
<krellan=google.com@lists.openembedded.org> wrote:

> Hi there, thanks. I'm not entirely sure what "python3native" does exactly,
> but this is my observation. The problem we are seeing is that, without
> "inherit python3native", the build machine's local Python version (whatever
> it happens to be) takes precedence over the Yocto-included Python version.
> In our case, the build machine is running Python 3.9, and evidently
> something changed between Python 3.8 and Python 3.9 to be incompatible with
> libxcb. Here's the error message:
>
> Failed to load the xcbgen Python package!
> Make sure that xcb/proto installed it on your Python path.
> If not, you will need to create a .pth file or define $PYTHONPATH
> to extend the path.
> Refer to the README file in xcb/proto for more info.
>
> Traceback (most recent call last):
>   File
> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
> line 3338, in <module>
>     from xcbgen.state import Module
>   File
> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py",
> line 7, in <module>
>     from xcbgen import matcher
>   File
> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py",
> line 12, in <module>
>     from xcbgen.xtypes import *
>   File
> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py",
> line 5, in <module>
>     from xcbgen.align import Alignment, AlignmentLog
>   File
> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py",
> line 5, in <module>
>     from fractions import gcd
> ImportError: cannot import name 'gcd' from 'fractions'
> (/usr/lib/python3.9/fractions.py)
>
> The above error text repeats for many of the files it tries to compile
> within libxcb.
>
> In general, it seems the safest approach is to standardize on a Python
> version that's known to work, without depending on whatever version of
> Python the developer has installed locally on their build machine (which
> can vary due to distribution, developer's preference, and so on).
>
> Josh
>
> Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941
> <(650)%20733-8941>
>
>
>
> On Tue, Apr 13, 2021 at 2:45 AM Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> On Tue, 2021-04-13 at 02:24 -0700, Josh Lehan via lists.openembedded.org
>> wrote:
>> > During compilation, the local Python environment leaks into
>> > the Bitbake build. Adding "inherit python3native" fixes this.
>> >
>> > Signed-off-by: Josh Lehan <krellan@google.com>
>> > ---
>> >  meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
>> >  1 file changed, 1 insertion(+)
>> >
>> > diff --git a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>> b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>> > index f0a223a19c..40c7b1cc3c 100644
>> > --- a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>> > +++ b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>> > @@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
>> >  FILES_${PN} = "${libdir}/libxcb.so.*"
>> >
>> >  inherit autotools pkgconfig features_check
>> > +inherit python3native
>> >
>>
>>
>> Could you provide a little more information about how this is breaking
>> things?
>>
>> Whilst python is used by the libxcb build, I believe it shouldn't change
>> the
>> output, it doesn't in our reproducibity tests.
>>
>> Adding a python-native dependency does slow the build down a bit so if we
>> don't need it, it would be good to avoid it.
>>
>> Cheers,
>>
>> Richard
>>
>>
>>
>>
>>
>>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 6512 bytes --]

Re: [OE-core] [PATCH] libxcb: Add inherit python3native

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 4560 bytes --]

Also note: oe-core master already has 3.9, so you need to find out why
'3.8' shows up in install paths as well. Something doesn't compute here.

Alex

On Tue, 13 Apr 2021 at 22:07, Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> I think you need to dig a little deeper. Why 'from fractions import gcd'
> is a problem with host python3.9 but not a problem with native python 3.9?
>
> Alex
>
> On Tue, 13 Apr 2021 at 20:58, Josh Lehan via lists.openembedded.org
> <krellan=google.com@lists.openembedded.org> wrote:
>
>> Hi there, thanks. I'm not entirely sure what "python3native" does
>> exactly, but this is my observation. The problem we are seeing is that,
>> without "inherit python3native", the build machine's local Python version
>> (whatever it happens to be) takes precedence over the Yocto-included Python
>> version. In our case, the build machine is running Python 3.9, and
>> evidently something changed between Python 3.8 and Python 3.9 to be
>> incompatible with libxcb. Here's the error message:
>>
>> Failed to load the xcbgen Python package!
>> Make sure that xcb/proto installed it on your Python path.
>> If not, you will need to create a .pth file or define $PYTHONPATH
>> to extend the path.
>> Refer to the README file in xcb/proto for more info.
>>
>> Traceback (most recent call last):
>>   File
>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
>> line 3338, in <module>
>>     from xcbgen.state import Module
>>   File
>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py",
>> line 7, in <module>
>>     from xcbgen import matcher
>>   File
>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py",
>> line 12, in <module>
>>     from xcbgen.xtypes import *
>>   File
>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py",
>> line 5, in <module>
>>     from xcbgen.align import Alignment, AlignmentLog
>>   File
>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py",
>> line 5, in <module>
>>     from fractions import gcd
>> ImportError: cannot import name 'gcd' from 'fractions'
>> (/usr/lib/python3.9/fractions.py)
>>
>> The above error text repeats for many of the files it tries to compile
>> within libxcb.
>>
>> In general, it seems the safest approach is to standardize on a Python
>> version that's known to work, without depending on whatever version of
>> Python the developer has installed locally on their build machine (which
>> can vary due to distribution, developer's preference, and so on).
>>
>> Josh
>>
>> Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941
>> <(650)%20733-8941>
>>
>>
>>
>> On Tue, Apr 13, 2021 at 2:45 AM Richard Purdie <
>> richard.purdie@linuxfoundation.org> wrote:
>>
>>> On Tue, 2021-04-13 at 02:24 -0700, Josh Lehan via lists.openembedded.org
>>> wrote:
>>> > During compilation, the local Python environment leaks into
>>> > the Bitbake build. Adding "inherit python3native" fixes this.
>>> >
>>> > Signed-off-by: Josh Lehan <krellan@google.com>
>>> > ---
>>> >  meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
>>> >  1 file changed, 1 insertion(+)
>>> >
>>> > diff --git a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>> b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>> > index f0a223a19c..40c7b1cc3c 100644
>>> > --- a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>> > +++ b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>> > @@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
>>> >  FILES_${PN} = "${libdir}/libxcb.so.*"
>>> >
>>> >  inherit autotools pkgconfig features_check
>>> > +inherit python3native
>>> >
>>>
>>>
>>> Could you provide a little more information about how this is breaking
>>> things?
>>>
>>> Whilst python is used by the libxcb build, I believe it shouldn't change
>>> the
>>> output, it doesn't in our reproducibity tests.
>>>
>>> Adding a python-native dependency does slow the build down a bit so if
>>> we
>>> don't need it, it would be good to avoid it.
>>>
>>> Cheers,
>>>
>>> Richard
>>>
>>>
>>>
>>>
>>>
>>>
>> 
>>
>>

[-- Attachment #2: Type: text/html, Size: 7109 bytes --]

Re: [OE-core] [PATCH] libxcb: Add inherit python3native

[Josh Lehan]

[-- Attachment #1: Type: text/plain, Size: 6258 bytes --]

Thanks. It's true, my local build is a little bit outdated. I'm running
OpenBMC and it's slightly behind the latest upstream master. For now, it is
still compiling with Python 3.8.

However, there is the same underlying problem, even if I upgraded. It looks
like the version of Python on the build machine is wrongly leaking into the
build, and it's being used instead of the oe-core preferred version of
Python. Ideally, the locally-installed Python wouldn't influence the
outcome of the build, as oe-core has its own Python that it should prefer
to use instead, as it does for other packages. This is what "inherit
python3native" enables, right? Otherwise, the problem will happen again in
the future, whenever somebody wants to try out a new Python version on
their machine. The idea is to obtain some separation, for safety, and to
help ensure a clean reproducible build. It's a good practice, I think, to
minimize the opportunities for local variance to creep into the build.

As for "from fractions import gcd" becoming an error in 3.9, there was this
change: https://bugs.python.org/issue39350

It appears fractions.gcd was deprecated for a while. It was moved to
math.gcd a while ago, and deprecated, but not deleted entirely until Python
3.9. So, that's why it works in 3.8 (what my old oe-core build is running),
but not in 3.9 (what my local distribution is running).

Josh

Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941



On Tue, Apr 13, 2021 at 1:08 PM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> Also note: oe-core master already has 3.9, so you need to find out why
> '3.8' shows up in install paths as well. Something doesn't compute here.
>
> Alex
>
> On Tue, 13 Apr 2021 at 22:07, Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
>
>> I think you need to dig a little deeper. Why 'from fractions import gcd'
>> is a problem with host python3.9 but not a problem with native python 3.9?
>>
>> Alex
>>
>> On Tue, 13 Apr 2021 at 20:58, Josh Lehan via lists.openembedded.org
>> <krellan=google.com@lists.openembedded.org> wrote:
>>
>>> Hi there, thanks. I'm not entirely sure what "python3native" does
>>> exactly, but this is my observation. The problem we are seeing is that,
>>> without "inherit python3native", the build machine's local Python version
>>> (whatever it happens to be) takes precedence over the Yocto-included Python
>>> version. In our case, the build machine is running Python 3.9, and
>>> evidently something changed between Python 3.8 and Python 3.9 to be
>>> incompatible with libxcb. Here's the error message:
>>>
>>> Failed to load the xcbgen Python package!
>>> Make sure that xcb/proto installed it on your Python path.
>>> If not, you will need to create a .pth file or define $PYTHONPATH
>>> to extend the path.
>>> Refer to the README file in xcb/proto for more info.
>>>
>>> Traceback (most recent call last):
>>>   File
>>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
>>> line 3338, in <module>
>>>     from xcbgen.state import Module
>>>   File
>>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py",
>>> line 7, in <module>
>>>     from xcbgen import matcher
>>>   File
>>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py",
>>> line 12, in <module>
>>>     from xcbgen.xtypes import *
>>>   File
>>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py",
>>> line 5, in <module>
>>>     from xcbgen.align import Alignment, AlignmentLog
>>>   File
>>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py",
>>> line 5, in <module>
>>>     from fractions import gcd
>>> ImportError: cannot import name 'gcd' from 'fractions'
>>> (/usr/lib/python3.9/fractions.py)
>>>
>>> The above error text repeats for many of the files it tries to compile
>>> within libxcb.
>>>
>>> In general, it seems the safest approach is to standardize on a Python
>>> version that's known to work, without depending on whatever version of
>>> Python the developer has installed locally on their build machine (which
>>> can vary due to distribution, developer's preference, and so on).
>>>
>>> Josh
>>>
>>> Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941
>>> <(650)%20733-8941>
>>>
>>>
>>>
>>> On Tue, Apr 13, 2021 at 2:45 AM Richard Purdie <
>>> richard.purdie@linuxfoundation.org> wrote:
>>>
>>>> On Tue, 2021-04-13 at 02:24 -0700, Josh Lehan via
>>>> lists.openembedded.org wrote:
>>>> > During compilation, the local Python environment leaks into
>>>> > the Bitbake build. Adding "inherit python3native" fixes this.
>>>> >
>>>> > Signed-off-by: Josh Lehan <krellan@google.com>
>>>> > ---
>>>> >  meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
>>>> >  1 file changed, 1 insertion(+)
>>>> >
>>>> > diff --git a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>>> b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>>> > index f0a223a19c..40c7b1cc3c 100644
>>>> > --- a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>>> > +++ b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>>> > @@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
>>>> >  FILES_${PN} = "${libdir}/libxcb.so.*"
>>>> >
>>>> >  inherit autotools pkgconfig features_check
>>>> > +inherit python3native
>>>> >
>>>>
>>>>
>>>> Could you provide a little more information about how this is breaking
>>>> things?
>>>>
>>>> Whilst python is used by the libxcb build, I believe it shouldn't
>>>> change the
>>>> output, it doesn't in our reproducibity tests.
>>>>
>>>> Adding a python-native dependency does slow the build down a bit so if
>>>> we
>>>> don't need it, it would be good to avoid it.
>>>>
>>>> Cheers,
>>>>
>>>> Richard
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>> 
>>>
>>>

[-- Attachment #2: Type: text/html, Size: 10004 bytes --]

Re: [OE-core] [PATCH] libxcb: Add inherit python3native

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 7257 bytes --]

As a matter of policy, yocto recipes default to using python from the host,
and are only switched to native python when there host python does not come
with needed 3rd party modules (which are then pulled in via -native
DEPENDS), or there is some other issue that cannot be resolved with host
python. Native python dependency lengthens the dependency chains, which is
especially critical for -native recipes.

In this case, the problem is that libxcb in your build is too old for the
latest python 3.9, so you need to either update libxcb or backport an
upstream fix to make it work with 3.9 again. Your patch would not resolve
the issue, as native python in latest oe-core  master is also 3.9, and the
failure would be the same.

Alex

On Wed, 14 Apr 2021 at 07:15, Josh Lehan <krellan@google.com> wrote:

> Thanks. It's true, my local build is a little bit outdated. I'm running
> OpenBMC and it's slightly behind the latest upstream master. For now, it is
> still compiling with Python 3.8.
>
> However, there is the same underlying problem, even if I upgraded. It
> looks like the version of Python on the build machine is wrongly leaking
> into the build, and it's being used instead of the oe-core preferred
> version of Python. Ideally, the locally-installed Python wouldn't influence
> the outcome of the build, as oe-core has its own Python that it should
> prefer to use instead, as it does for other packages. This is what "inherit
> python3native" enables, right? Otherwise, the problem will happen again in
> the future, whenever somebody wants to try out a new Python version on
> their machine. The idea is to obtain some separation, for safety, and to
> help ensure a clean reproducible build. It's a good practice, I think, to
> minimize the opportunities for local variance to creep into the build.
>
> As for "from fractions import gcd" becoming an error in 3.9, there was
> this change: https://bugs.python.org/issue39350
>
> It appears fractions.gcd was deprecated for a while. It was moved to
> math.gcd a while ago, and deprecated, but not deleted entirely until Python
> 3.9. So, that's why it works in 3.8 (what my old oe-core build is running),
> but not in 3.9 (what my local distribution is running).
>
> Josh
>
> Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941
>
>
>
> On Tue, Apr 13, 2021 at 1:08 PM Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
>
>> Also note: oe-core master already has 3.9, so you need to find out why
>> '3.8' shows up in install paths as well. Something doesn't compute here.
>>
>> Alex
>>
>> On Tue, 13 Apr 2021 at 22:07, Alexander Kanavin <alex.kanavin@gmail.com>
>> wrote:
>>
>>> I think you need to dig a little deeper. Why 'from fractions import gcd'
>>> is a problem with host python3.9 but not a problem with native python 3.9?
>>>
>>> Alex
>>>
>>> On Tue, 13 Apr 2021 at 20:58, Josh Lehan via lists.openembedded.org
>>> <krellan=google.com@lists.openembedded.org> wrote:
>>>
>>>> Hi there, thanks. I'm not entirely sure what "python3native" does
>>>> exactly, but this is my observation. The problem we are seeing is that,
>>>> without "inherit python3native", the build machine's local Python version
>>>> (whatever it happens to be) takes precedence over the Yocto-included Python
>>>> version. In our case, the build machine is running Python 3.9, and
>>>> evidently something changed between Python 3.8 and Python 3.9 to be
>>>> incompatible with libxcb. Here's the error message:
>>>>
>>>> Failed to load the xcbgen Python package!
>>>> Make sure that xcb/proto installed it on your Python path.
>>>> If not, you will need to create a .pth file or define $PYTHONPATH
>>>> to extend the path.
>>>> Refer to the README file in xcb/proto for more info.
>>>>
>>>> Traceback (most recent call last):
>>>>   File
>>>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
>>>> line 3338, in <module>
>>>>     from xcbgen.state import Module
>>>>   File
>>>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py",
>>>> line 7, in <module>
>>>>     from xcbgen import matcher
>>>>   File
>>>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py",
>>>> line 12, in <module>
>>>>     from xcbgen.xtypes import *
>>>>   File
>>>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py",
>>>> line 5, in <module>
>>>>     from xcbgen.align import Alignment, AlignmentLog
>>>>   File
>>>> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py",
>>>> line 5, in <module>
>>>>     from fractions import gcd
>>>> ImportError: cannot import name 'gcd' from 'fractions'
>>>> (/usr/lib/python3.9/fractions.py)
>>>>
>>>> The above error text repeats for many of the files it tries to compile
>>>> within libxcb.
>>>>
>>>> In general, it seems the safest approach is to standardize on a Python
>>>> version that's known to work, without depending on whatever version of
>>>> Python the developer has installed locally on their build machine (which
>>>> can vary due to distribution, developer's preference, and so on).
>>>>
>>>> Josh
>>>>
>>>> Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941
>>>> <(650)%20733-8941>
>>>>
>>>>
>>>>
>>>> On Tue, Apr 13, 2021 at 2:45 AM Richard Purdie <
>>>> richard.purdie@linuxfoundation.org> wrote:
>>>>
>>>>> On Tue, 2021-04-13 at 02:24 -0700, Josh Lehan via
>>>>> lists.openembedded.org wrote:
>>>>> > During compilation, the local Python environment leaks into
>>>>> > the Bitbake build. Adding "inherit python3native" fixes this.
>>>>> >
>>>>> > Signed-off-by: Josh Lehan <krellan@google.com>
>>>>> > ---
>>>>> >  meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
>>>>> >  1 file changed, 1 insertion(+)
>>>>> >
>>>>> > diff --git a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>>>> b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>>>> > index f0a223a19c..40c7b1cc3c 100644
>>>>> > --- a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>>>> > +++ b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>>>> > @@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
>>>>> >  FILES_${PN} = "${libdir}/libxcb.so.*"
>>>>> >
>>>>> >  inherit autotools pkgconfig features_check
>>>>> > +inherit python3native
>>>>> >
>>>>>
>>>>>
>>>>> Could you provide a little more information about how this is breaking
>>>>> things?
>>>>>
>>>>> Whilst python is used by the libxcb build, I believe it shouldn't
>>>>> change the
>>>>> output, it doesn't in our reproducibity tests.
>>>>>
>>>>> Adding a python-native dependency does slow the build down a bit so if
>>>>> we
>>>>> don't need it, it would be good to avoid it.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Richard
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> 
>>>>
>>>>

[-- Attachment #2: Type: text/html, Size: 11132 bytes --]

Re: [OE-core] [PATCH] libxcb: Add inherit python3native

[William A. Kennington III]

It's correct because the package it depends on (xcb-proto-native) is
already `inherit python3native`. This package's xcbgen invocation is
broken because it is using a combination of files generated for native
python but executed through host python (look at the site-packages
version). It can't bloat our dependency tree any more as it already
implicitly depends on python3-native.

On Wed, Apr 14, 2021 at 12:25 AM Alexander Kanavin
<alex.kanavin@gmail.com> wrote:
>
> As a matter of policy, yocto recipes default to using python from the host, and are only switched to native python when there host python does not come with needed 3rd party modules (which are then pulled in via -native DEPENDS), or there is some other issue that cannot be resolved with host python. Native python dependency lengthens the dependency chains, which is especially critical for -native recipes.
>
> In this case, the problem is that libxcb in your build is too old for the latest python 3.9, so you need to either update libxcb or backport an upstream fix to make it work with 3.9 again. Your patch would not resolve the issue, as native python in latest oe-core  master is also 3.9, and the failure would be the same.
>
> Alex
>
> On Wed, 14 Apr 2021 at 07:15, Josh Lehan <krellan@google.com> wrote:
>>
>> Thanks. It's true, my local build is a little bit outdated. I'm running OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.
>>
>> However, there is the same underlying problem, even if I upgraded. It looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.
>>
>> As for "from fractions import gcd" becoming an error in 3.9, there was this change: https://bugs.python.org/issue39350
>>
>> It appears fractions.gcd was deprecated for a while. It was moved to math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).
>>
>> Josh
>>
>> Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941
>>
>>
>>
>> On Tue, Apr 13, 2021 at 1:08 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>>>
>>> Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.
>>>
>>> Alex
>>>
>>> On Tue, 13 Apr 2021 at 22:07, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>>>>
>>>> I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?
>>>>
>>>> Alex
>>>>
>>>> On Tue, 13 Apr 2021 at 20:58, Josh Lehan via lists.openembedded.org <krellan=google.com@lists.openembedded.org> wrote:
>>>>>
>>>>> Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:
>>>>>
>>>>> Failed to load the xcbgen Python package!
>>>>> Make sure that xcb/proto installed it on your Python path.
>>>>> If not, you will need to create a .pth file or define $PYTHONPATH
>>>>> to extend the path.
>>>>> Refer to the README file in xcb/proto for more info.
>>>>>
>>>>> Traceback (most recent call last):
>>>>>   File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py", line 3338, in <module>
>>>>>     from xcbgen.state import Module
>>>>>   File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
>>>>>     from xcbgen import matcher
>>>>>   File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
>>>>>     from xcbgen.xtypes import *
>>>>>   File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
>>>>>     from xcbgen.align import Alignment, AlignmentLog
>>>>>   File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
>>>>>     from fractions import gcd
>>>>> ImportError: cannot import name 'gcd' from 'fractions' (/usr/lib/python3.9/fractions.py)
>>>>>
>>>>> The above error text repeats for many of the files it tries to compile within libxcb.
>>>>>
>>>>> In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).
>>>>>
>>>>> Josh
>>>>>
>>>>> Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 13, 2021 at 2:45 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
>>>>>>
>>>>>> On Tue, 2021-04-13 at 02:24 -0700, Josh Lehan via lists.openembedded.org wrote:
>>>>>> > During compilation, the local Python environment leaks into
>>>>>> > the Bitbake build. Adding "inherit python3native" fixes this.
>>>>>> >
>>>>>> > Signed-off-by: Josh Lehan <krellan@google.com>
>>>>>> > ---
>>>>>> >  meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
>>>>>> >  1 file changed, 1 insertion(+)
>>>>>> >
>>>>>> > diff --git a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>>>>> > index f0a223a19c..40c7b1cc3c 100644
>>>>>> > --- a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>>>>> > +++ b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
>>>>>> > @@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
>>>>>> >  FILES_${PN} = "${libdir}/libxcb.so.*"
>>>>>> >
>>>>>> >  inherit autotools pkgconfig features_check
>>>>>> > +inherit python3native
>>>>>> >
>>>>>>
>>>>>>
>>>>>> Could you provide a little more information about how this is breaking things?
>>>>>>
>>>>>> Whilst python is used by the libxcb build, I believe it shouldn't change the
>>>>>> output, it doesn't in our reproducibity tests.
>>>>>>
>>>>>> Adding a python-native dependency does slow the build down a bit so if we
>>>>>> don't need it, it would be good to avoid it.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Richard
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>
> 
>

Re: [OE-core] [PATCH] libxcb: Add inherit python3native

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 8273 bytes --]

That seems correct, but if we force the native python 3.9 to be used, is
the current libxcb/xcbgen in master compatible with it (e.g. not anymore
using the removed feautre)?

Alex

On Wed, 14 Apr 2021 at 11:55, William Kennington <wak@google.com> wrote:

> It's correct because the package it depends on (xcb-proto-native) is
> already `inherit python3native`. This package's xcbgen invocation is
> broken because it is using a combination of files generated for native
> python but executed through host python (look at the site-packages
> version). It can't bloat our dependency tree any more as it already
> implicitly depends on python3-native.
>
> On Wed, Apr 14, 2021 at 12:25 AM Alexander Kanavin
> <alex.kanavin@gmail.com> wrote:
> >
> > As a matter of policy, yocto recipes default to using python from the
> host, and are only switched to native python when there host python does
> not come with needed 3rd party modules (which are then pulled in via
> -native DEPENDS), or there is some other issue that cannot be resolved with
> host python. Native python dependency lengthens the dependency chains,
> which is especially critical for -native recipes.
> >
> > In this case, the problem is that libxcb in your build is too old for
> the latest python 3.9, so you need to either update libxcb or backport an
> upstream fix to make it work with 3.9 again. Your patch would not resolve
> the issue, as native python in latest oe-core  master is also 3.9, and the
> failure would be the same.
> >
> > Alex
> >
> > On Wed, 14 Apr 2021 at 07:15, Josh Lehan <krellan@google.com> wrote:
> >>
> >> Thanks. It's true, my local build is a little bit outdated. I'm running
> OpenBMC and it's slightly behind the latest upstream master. For now, it is
> still compiling with Python 3.8.
> >>
> >> However, there is the same underlying problem, even if I upgraded. It
> looks like the version of Python on the build machine is wrongly leaking
> into the build, and it's being used instead of the oe-core preferred
> version of Python. Ideally, the locally-installed Python wouldn't influence
> the outcome of the build, as oe-core has its own Python that it should
> prefer to use instead, as it does for other packages. This is what "inherit
> python3native" enables, right? Otherwise, the problem will happen again in
> the future, whenever somebody wants to try out a new Python version on
> their machine. The idea is to obtain some separation, for safety, and to
> help ensure a clean reproducible build. It's a good practice, I think, to
> minimize the opportunities for local variance to creep into the build.
> >>
> >> As for "from fractions import gcd" becoming an error in 3.9, there was
> this change: https://bugs.python.org/issue39350
> >>
> >> It appears fractions.gcd was deprecated for a while. It was moved to
> math.gcd a while ago, and deprecated, but not deleted entirely until Python
> 3.9. So, that's why it works in 3.8 (what my old oe-core build is running),
> but not in 3.9 (what my local distribution is running).
> >>
> >> Josh
> >>
> >> Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941
> >>
> >>
> >>
> >> On Tue, Apr 13, 2021 at 1:08 PM Alexander Kanavin <
> alex.kanavin@gmail.com> wrote:
> >>>
> >>> Also note: oe-core master already has 3.9, so you need to find out why
> '3.8' shows up in install paths as well. Something doesn't compute here.
> >>>
> >>> Alex
> >>>
> >>> On Tue, 13 Apr 2021 at 22:07, Alexander Kanavin <
> alex.kanavin@gmail.com> wrote:
> >>>>
> >>>> I think you need to dig a little deeper. Why 'from fractions import
> gcd' is a problem with host python3.9 but not a problem with native python
> 3.9?
> >>>>
> >>>> Alex
> >>>>
> >>>> On Tue, 13 Apr 2021 at 20:58, Josh Lehan via lists.openembedded.org
> <krellan=google.com@lists.openembedded.org> wrote:
> >>>>>
> >>>>> Hi there, thanks. I'm not entirely sure what "python3native" does
> exactly, but this is my observation. The problem we are seeing is that,
> without "inherit python3native", the build machine's local Python version
> (whatever it happens to be) takes precedence over the Yocto-included Python
> version. In our case, the build machine is running Python 3.9, and
> evidently something changed between Python 3.8 and Python 3.9 to be
> incompatible with libxcb. Here's the error message:
> >>>>>
> >>>>> Failed to load the xcbgen Python package!
> >>>>> Make sure that xcb/proto installed it on your Python path.
> >>>>> If not, you will need to create a .pth file or define $PYTHONPATH
> >>>>> to extend the path.
> >>>>> Refer to the README file in xcb/proto for more info.
> >>>>>
> >>>>> Traceback (most recent call last):
> >>>>>   File
> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py",
> line 3338, in <module>
> >>>>>     from xcbgen.state import Module
> >>>>>   File
> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py",
> line 7, in <module>
> >>>>>     from xcbgen import matcher
> >>>>>   File
> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py",
> line 12, in <module>
> >>>>>     from xcbgen.xtypes import *
> >>>>>   File
> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py",
> line 5, in <module>
> >>>>>     from xcbgen.align import Alignment, AlignmentLog
> >>>>>   File
> "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py",
> line 5, in <module>
> >>>>>     from fractions import gcd
> >>>>> ImportError: cannot import name 'gcd' from 'fractions'
> (/usr/lib/python3.9/fractions.py)
> >>>>>
> >>>>> The above error text repeats for many of the files it tries to
> compile within libxcb.
> >>>>>
> >>>>> In general, it seems the safest approach is to standardize on a
> Python version that's known to work, without depending on whatever version
> of Python the developer has installed locally on their build machine (which
> can vary due to distribution, developer's preference, and so on).
> >>>>>
> >>>>> Josh
> >>>>>
> >>>>> Josh Lehan | Software Engineer | krellan@google.com | +1
> 650-733-8941
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 13, 2021 at 2:45 AM Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> >>>>>>
> >>>>>> On Tue, 2021-04-13 at 02:24 -0700, Josh Lehan via
> lists.openembedded.org wrote:
> >>>>>> > During compilation, the local Python environment leaks into
> >>>>>> > the Bitbake build. Adding "inherit python3native" fixes this.
> >>>>>> >
> >>>>>> > Signed-off-by: Josh Lehan <krellan@google.com>
> >>>>>> > ---
> >>>>>> >  meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
> >>>>>> >  1 file changed, 1 insertion(+)
> >>>>>> >
> >>>>>> > diff --git a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> >>>>>> > index f0a223a19c..40c7b1cc3c 100644
> >>>>>> > --- a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> >>>>>> > +++ b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> >>>>>> > @@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
> >>>>>> >  FILES_${PN} = "${libdir}/libxcb.so.*"
> >>>>>> >
> >>>>>> >  inherit autotools pkgconfig features_check
> >>>>>> > +inherit python3native
> >>>>>> >
> >>>>>>
> >>>>>>
> >>>>>> Could you provide a little more information about how this is
> breaking things?
> >>>>>>
> >>>>>> Whilst python is used by the libxcb build, I believe it shouldn't
> change the
> >>>>>> output, it doesn't in our reproducibity tests.
> >>>>>>
> >>>>>> Adding a python-native dependency does slow the build down a bit so
> if we
> >>>>>> don't need it, it would be good to avoid it.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Richard
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >
> > 
> >
>

[-- Attachment #2: Type: text/html, Size: 11631 bytes --]

Re: [OE-core] [PATCH] libxcb: Add inherit python3native

[Khem Raj]

On Wed, Apr 14, 2021 at 2:55 AM William A. Kennington III via
lists.openembedded.org <wak=google.com@lists.openembedded.org> wrote:
>
> It's correct because the package it depends on (xcb-proto-native) is
> already `inherit python3native`. This package's xcbgen invocation is
> broken because it is using a combination of files generated for native
> python but executed through host python (look at the site-packages
> version). It can't bloat our dependency tree any more as it already
> implicitly depends on python3-native.

I tend to agree. Although this makes me think if we have more such mix
and match happening elsewhere too
in that case, would it be better to always use a captured version of
python3 on native build hosts and ignore distro
version of python3 always something like buildtools-tarball should
then become mandatory.

>
> On Wed, Apr 14, 2021 at 12:25 AM Alexander Kanavin
> <alex.kanavin@gmail.com> wrote:
> >
> > As a matter of policy, yocto recipes default to using python from the host, and are only switched to native python when there host python does not come with needed 3rd party modules (which are then pulled in via -native DEPENDS), or there is some other issue that cannot be resolved with host python. Native python dependency lengthens the dependency chains, which is especially critical for -native recipes.
> >
> > In this case, the problem is that libxcb in your build is too old for the latest python 3.9, so you need to either update libxcb or backport an upstream fix to make it work with 3.9 again. Your patch would not resolve the issue, as native python in latest oe-core  master is also 3.9, and the failure would be the same.
> >
> > Alex
> >
> > On Wed, 14 Apr 2021 at 07:15, Josh Lehan <krellan@google.com> wrote:
> >>
> >> Thanks. It's true, my local build is a little bit outdated. I'm running OpenBMC and it's slightly behind the latest upstream master. For now, it is still compiling with Python 3.8.
> >>
> >> However, there is the same underlying problem, even if I upgraded. It looks like the version of Python on the build machine is wrongly leaking into the build, and it's being used instead of the oe-core preferred version of Python. Ideally, the locally-installed Python wouldn't influence the outcome of the build, as oe-core has its own Python that it should prefer to use instead, as it does for other packages. This is what "inherit python3native" enables, right? Otherwise, the problem will happen again in the future, whenever somebody wants to try out a new Python version on their machine. The idea is to obtain some separation, for safety, and to help ensure a clean reproducible build. It's a good practice, I think, to minimize the opportunities for local variance to creep into the build.
> >>
> >> As for "from fractions import gcd" becoming an error in 3.9, there was this change: https://bugs.python.org/issue39350
> >>
> >> It appears fractions.gcd was deprecated for a while. It was moved to math.gcd a while ago, and deprecated, but not deleted entirely until Python 3.9. So, that's why it works in 3.8 (what my old oe-core build is running), but not in 3.9 (what my local distribution is running).
> >>
> >> Josh
> >>
> >> Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941
> >>
> >>
> >>
> >> On Tue, Apr 13, 2021 at 1:08 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> >>>
> >>> Also note: oe-core master already has 3.9, so you need to find out why '3.8' shows up in install paths as well. Something doesn't compute here.
> >>>
> >>> Alex
> >>>
> >>> On Tue, 13 Apr 2021 at 22:07, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> >>>>
> >>>> I think you need to dig a little deeper. Why 'from fractions import gcd' is a problem with host python3.9 but not a problem with native python 3.9?
> >>>>
> >>>> Alex
> >>>>
> >>>> On Tue, 13 Apr 2021 at 20:58, Josh Lehan via lists.openembedded.org <krellan=google.com@lists.openembedded.org> wrote:
> >>>>>
> >>>>> Hi there, thanks. I'm not entirely sure what "python3native" does exactly, but this is my observation. The problem we are seeing is that, without "inherit python3native", the build machine's local Python version (whatever it happens to be) takes precedence over the Yocto-included Python version. In our case, the build machine is running Python 3.9, and evidently something changed between Python 3.8 and Python 3.9 to be incompatible with libxcb. Here's the error message:
> >>>>>
> >>>>> Failed to load the xcbgen Python package!
> >>>>> Make sure that xcb/proto installed it on your Python path.
> >>>>> If not, you will need to create a .pth file or define $PYTHONPATH
> >>>>> to extend the path.
> >>>>> Refer to the README file in xcb/proto for more info.
> >>>>>
> >>>>> Traceback (most recent call last):
> >>>>>   File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/build/src/../../libxcb-1.14/src/c_client.py", line 3338, in <module>
> >>>>>     from xcbgen.state import Module
> >>>>>   File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/state.py", line 7, in <module>
> >>>>>     from xcbgen import matcher
> >>>>>   File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/matcher.py", line 12, in <module>
> >>>>>     from xcbgen.xtypes import *
> >>>>>   File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/xtypes.py", line 5, in <module>
> >>>>>     from xcbgen.align import Alignment, AlignmentLog
> >>>>>   File "/OpenBMC/build/tmp/work/x86_64-linux/libxcb-native/1.14-r0/recipe-sysroot-native/usr/lib/pkgconfig/../../../usr/lib/python3.8/site-packages/xcbgen/align.py", line 5, in <module>
> >>>>>     from fractions import gcd
> >>>>> ImportError: cannot import name 'gcd' from 'fractions' (/usr/lib/python3.9/fractions.py)
> >>>>>
> >>>>> The above error text repeats for many of the files it tries to compile within libxcb.
> >>>>>
> >>>>> In general, it seems the safest approach is to standardize on a Python version that's known to work, without depending on whatever version of Python the developer has installed locally on their build machine (which can vary due to distribution, developer's preference, and so on).
> >>>>>
> >>>>> Josh
> >>>>>
> >>>>> Josh Lehan | Software Engineer | krellan@google.com | +1 650-733-8941
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Tue, Apr 13, 2021 at 2:45 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> >>>>>>
> >>>>>> On Tue, 2021-04-13 at 02:24 -0700, Josh Lehan via lists.openembedded.org wrote:
> >>>>>> > During compilation, the local Python environment leaks into
> >>>>>> > the Bitbake build. Adding "inherit python3native" fixes this.
> >>>>>> >
> >>>>>> > Signed-off-by: Josh Lehan <krellan@google.com>
> >>>>>> > ---
> >>>>>> >  meta/recipes-graphics/xorg-lib/libxcb_1.14.bb | 1 +
> >>>>>> >  1 file changed, 1 insertion(+)
> >>>>>> >
> >>>>>> > diff --git a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> >>>>>> > index f0a223a19c..40c7b1cc3c 100644
> >>>>>> > --- a/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> >>>>>> > +++ b/meta/recipes-graphics/xorg-lib/libxcb_1.14.bb
> >>>>>> > @@ -24,6 +24,7 @@ PACKAGES_DYNAMIC = "^${PN}-.*"
> >>>>>> >  FILES_${PN} = "${libdir}/libxcb.so.*"
> >>>>>> >
> >>>>>> >  inherit autotools pkgconfig features_check
> >>>>>> > +inherit python3native
> >>>>>> >
> >>>>>>
> >>>>>>
> >>>>>> Could you provide a little more information about how this is breaking things?
> >>>>>>
> >>>>>> Whilst python is used by the libxcb build, I believe it shouldn't change the
> >>>>>> output, it doesn't in our reproducibity tests.
> >>>>>>
> >>>>>> Adding a python-native dependency does slow the build down a bit so if we
> >>>>>> don't need it, it would be good to avoid it.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Richard
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >
> >
> >
>
> 
>