From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7837EC433EF for ; Thu, 24 Feb 2022 04:50:49 +0000 (UTC) Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com [209.85.218.52]) by mx.groups.io with SMTP id smtpd.web10.6614.1645678248130622743 for ; Wed, 23 Feb 2022 20:50:48 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@gmail.com header.s=20210112 header.b=GRV0Y3hd; spf=pass (domain: gmail.com, ip: 209.85.218.52, mailfrom: ticotimo@gmail.com) Received: by mail-ej1-f52.google.com with SMTP id hw13so1803243ejc.9 for ; Wed, 23 Feb 2022 20:50:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qoRhcvRxPi+mKgRaLbcLpiBmDDEQ2fjpMp6uXtNtBU4=; b=GRV0Y3hdkaYUKhQo8cCCL/77SJPb55uB0QFL+VknLDdsLjHaNlOOMAwmtHsX3rwdFI tdcyjLJOdxlduVflZnG/o8ACpgn0TJp9PUmJi05rAfldM5UeQNQR0gD/95WDuuWC2jS3 VvLAhKGza5spXTVUqR626gzOjRhehspc1ipnZz5s5Q8YnKgyBlwMT4Fg6IrpRgzGNyo+ a9bZ1nI+m/EjBn+W1FgeiGNRJ9kd8kQkeNFKasICWV/1E0wglgM4FRTqxKmG6GdrFvoS etYIkLagdZuBgm+jbczhP/14hlMc2u/PfbVZLnv6YMClaiPfWK0ckmunbHXMOdfjKfay VUGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qoRhcvRxPi+mKgRaLbcLpiBmDDEQ2fjpMp6uXtNtBU4=; b=YGW/u5VGhyTXYk19E2/YVisPD4PqRBuURWMQWG5AmctlhVEP4BkSQj428oq5uSkLud atEBQM6kWaSZW6FuOhSEdwoxaMGVGPhBhHUhxTR8pt0u0vq6wmNgPwV+K3PUBRGJjEuw HazLas5wQmzQWbGU2SMepMMkP8zkpaJpdWLQCQSGegtDucmoCbi6e77JzgUQ1fcHDITh EtJZOW10OYRJ8g8vhDyI5xKa3a2s0UiVWjBOL4qnkDlDtEvdTi77yfy596H/PeLFBwFM G6dx6y0JbJwdfKn0s2LgZhpnBUVzwAH7VZUnq8tkuCHm9olpHXMLN0PROD9fEbZNAJ70 zbRQ== X-Gm-Message-State: AOAM532PjxmFOtLKR2iZe/0Op2buwVZaOBBmaX6e1zl4o7axGy+6oFIB k+oDGuZ792zPbNaXyADmtTsS8fHlKBatUlJ0SYU= X-Google-Smtp-Source: ABdhPJzb7o6xu0/K+t4P9izpfSOXW/XrU8sC+qpsL6RGj6PNfbLFQ0X+h9osOrws3ADqu4gsZzzj/XHXtn/NOu3xx/s= X-Received: by 2002:a17:907:6e10:b0:6d1:4447:95bc with SMTP id sd16-20020a1709076e1000b006d1444795bcmr767103ejc.610.1645678246351; Wed, 23 Feb 2022 20:50:46 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Tim Orling Date: Wed, 23 Feb 2022 20:50:35 -0800 Message-ID: Subject: Re: [OE-core] [PATCH v2 00/32] Python PEP-517: build wheels and install with pip To: Khem Raj Cc: Patches and discussions about the oe-core layer Content-Type: multipart/alternative; boundary="00000000000049d8a805d8bc5374" List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 24 Feb 2022 04:50:49 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/162292 --00000000000049d8a805d8bc5374 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Feb 23, 2022 at 8:07 PM Khem Raj wrote: > Right now approx 150 out of 470 recipes [1] in meta-python are directly > failing, > perhaps there will be more once these are fixed then there are other laye= rs > which might be carrying some impacted python modules. The issue has mostly been detection of the wheel filename and this should be improved by: https://lore.kernel.org/openembedded-core/20220224041200.2645054-1-tim.orli= ng@konsulko.com/T/#t Also YP compatibility > is broken due to python3-nose removal. We can try to run 2to3 on the python3-nose plugins or remove them surgically to reintroduce cleanly Python 3 only python3-nose. I propose that we move it to meta-python even so because nothing in oe-core depends on nose and the code has been completely bit-rotting since 2016. It could not be built anymore in oe-core and something had to give. > If we are planning > to include this in the upcoming release,we would need some commitment > and strategy to fix these and other ripple effects throughout the ecosyst= em > especially since this is an LTS release. I wish this changeset was propos= ed > in a release prior to LTS release, so it could have given enough time to > adjust for the ecosystem. This is will take some effort in stabilizing > This has been months of work on my part to attempt to get us to where upstream Python Packaging Authority has been pushing since PEP-517 was announced in 2015. We are very far behind the curve of upstream Python packaging. While I too would love to have had this ready sooner, that was not possible for many reasons. But no matter what, this would have been painful until recipes could be migrated, even if had been available at M1. > I also wonder, what do we lose if we do not put this into the LTS > release? We already have recipes in oe-core such as python3-jsonschema which can no longer be upgraded without moving to pyproject.toml based package builds. I'm sure there are others in meta-python and other layers in a similar story. We once again are in a situation that is a bad image for the project because we are using the now deprecated "setup.py install" and still building eggs. To step into an LTS release with the python packaging already entirely deprecated is just not a good impression. I understand your frustration with the build failures, but it has been a very solid week of getting oe-core stabilized (see https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3280). I am running a build now for meta-python with the improved wheel filename patch as well as a patch to check for whether the current recipe already has pyproject.toml back-end declared: https://lore.kernel.org/openembedded-core/20220224041200.2645054-2-tim.orli= ng@konsulko.com/T/#u > [1] https://autobuilder.yoctoproject.org/typhoon/#/builders/88/builds/1460 > > On Tue, Feb 22, 2022 at 11:23 AM Tim Orling wrote: > > > > > > > > On Tue, Feb 22, 2022 at 11:18 AM Khem Raj wrote: > >> > >> On Tue, Feb 22, 2022 at 10:40 AM Tim Orling wrote= : > >> > > >> > > >> > > >> > On Tue, Feb 22, 2022 at 10:24 AM Khem Raj wrote= : > >> >> > >> >> On Tue, Feb 22, 2022 at 10:21 AM Tim Orling > wrote: > >> >> > > >> >> > > >> >> > > >> >> > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj > wrote: > >> >> >> > >> >> >> this is going into dep loops Try > >> >> >> > >> >> >> bitbake python3-wheel-native > >> >> > > >> >> > > >> >> > It shouldn=E2=80=99t > >> >> > > https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/pyth= on/python3-wheel_0.37.1.bb?h=3Dtimo/bootstrap-wheels > >> >> > > >> >> > >> >> I have been trying whats on master-next > >> >> > >> > > >> > I can't reproduce. Can you point me at an error log please? > >> > It might have something to do with sstate cache, but that's just a > hunch. > >> > > >> > >> This is because python3-wheel was moved to core and modified but not > >> deleted from meta-python at same time. I have sent a patch for that > >> > > Already sent one: > > > https://patchwork.yoctoproject.org/project/oe/patch/20220222184920.231039= 1-1-tim.orling@konsulko.com/ > > > >> > >> >> > >> >> > DEPENDS:remove:class-native =3D "python3-pip-native" > >> >> > > >> >> > do_install:class-native () { > >> >> > # We need to bootstrap python3-wheel-native > >> >> > install -d ${D}${PYTHON_SITEPACKAGES_DIR} > >> >> > PYPA_WHEEL=3D"${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" > >> >> > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ > >> >> > bbfatal_log "Failed to install" > >> >> > } > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling > wrote: > >> >> >> > > >> >> >> > Upstream Python is moving to pyproject.toml rather than > setup.py and > >> >> >> > wheels rather than eggs. The main place this is documented is = in > >> >> >> > PEP-517[1]. Installing packages with setup.py install is > deprecated. > >> >> >> > > >> >> >> > We have a choice with the 'kirkstone' LTS release of > introducing this > >> >> >> > series (and fixing any fall out during the stabilization > period) or > >> >> >> > living with deprecated Python packaging for the duration of th= e > LTS (at > >> >> >> > a minimum 2 years). This seems to be a compelling enough reaso= n > to bring > >> >> >> > in this series, even if it results in a slight slip of the > 'feature > >> >> >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. > >> >> >> > > >> >> >> > This series bootstraps python3-setuptools-native, > >> >> >> > python3-flit-core-native, python3-wheel-native and > python3-pip-native, > >> >> >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This > allows all > >> >> >> > other recipes to be installed with pip, as intended by upstrea= m. > >> >> >> > > >> >> >> > Three new classes are introduced: > >> >> >> > (1) flit_core which builds PEP-517 compliant packages which us= e > >> >> >> > flit_core.buildapi as their build-backend. > >> >> >> > (2) setuptools_build_meta which builds PEP-517 compliant > packages which use > >> >> >> > setuptools_build_meta as their build-backend. > >> >> >> > (3) pip_install_wheel, which simply installs wheels built by > other > >> >> >> > classes. > >> >> >> > > >> >> >> > The original setuptools3.bbclass do_compile is modified to > >> >> >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than > the deprecated > >> >> >> > 'setup.py build' mechanism. > >> >> >> > > >> >> >> > The original setuptools3.bbclass do_install is modified to > >> >> >> > use the pip_install_wheel.bbclass to install wheels with pip > rather than > >> >> >> > the deprecated 'setup.py install' mechanism. > >> >> >> > > >> >> >> > For pip install, we disable dependency checking (--no-deps) > because it > >> >> >> > proves to be brittle in our environment without providing full > wheel > >> >> >> > cache that seems like overhead that we do not need. We also > disable > >> >> >> > fetching wheels from pypi (--no-index). This is no different > than our > >> >> >> > normal workflow of requiring dependencies to be defined in a > recipe. > >> >> >> > > >> >> >> > Missing from this series is documentation of the new features, > classes > >> >> >> > and variables (this will be done under separate cover). > >> >> >> > > >> >> >> > Also desired and previously discussed in YP meetings is a help= er > >> >> >> > function to warn that a recipe which currently inherits > setuptools3 > >> >> >> > but has a pyproject.toml and defines a > build-system.build-backend should > >> >> >> > instead use of the new PEP-517 classes. This function is still > in progress. > >> >> >> > > >> >> >> > During testing, recipes which use python3-setuptools-scm are > failing to > >> >> >> > determine the version of the python package, For several > recipes, the fix > >> >> >> > was to use pyproject.toml and PEP-517 build backends. To be > sure, other recipes > >> >> >> > are still going to fail for the short term, until the root > cause can be > >> >> >> > determined. This was not seen again in v2 builds. > >> >> >> > > >> >> >> > Another PEP-517 build-backend, poetry.core.masonry.api has bee= n > >> >> >> > identified in recipes in meta-python, but not in oe-core. The > >> >> >> > python3-poetry-core recipe is WIP, but brings in a number of > >> >> >> > dependencies that are perhaps not worth it to bring into > oe-core at this > >> >> >> > time. Rather we can support these recipes in meta-python? > >> >> >> > > >> >> >> > Fixes in v2 include passing --root=3D${D} to pip install (rath= er > than --target) > >> >> >> > and also passing --prefix so that scripts are (usually) > installed in the proper > >> >> >> > location. pip install also 'helpfully' overwrites python3 > intepreter with nativepython3 > >> >> >> > in #!, so add a fix to return the values to what is actually > correct (similar to what is > >> >> >> > done in setuptools3.bbclass). Also several recipes still neede= d > PIP_INSTALL_PACKAGE defined. > >> >> >> > > >> >> >> > Also added self as maintainer of new recipes. > >> >> >> > > >> >> >> > [YOCTO #14638] > >> >> >> > > >> >> >> > [1] https://www.python.org/dev/peps/pep-0517/ > >> >> >> > > >> >> >> > The following changes since commit > 41ae26ecab731f2fb6593df143476c94183be209: > >> >> >> > > >> >> >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000= ) > >> >> >> > > >> >> >> > are available in the Git repository at: > >> >> >> > > >> >> >> > git://git.yoctoproject.org/git/poky-contrib > timo/bootstrap-wheels > >> >> >> > > http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=3Dtimo/boots= trap-wheels > >> >> >> > > >> >> >> > Tim Orling (32): > >> >> >> > python3-wheel: move 0.37.0 from meta-python > >> >> >> > python3-flit-core: add recipe for 3.6.0 > >> >> >> > flit_core.bbclass: add helper for newer python packaging > >> >> >> > python3-wheel: inherit flit_core > >> >> >> > pip_install_wheel.bbclass: add helper class > >> >> >> > setuptools_build_meta.bbclass: add helper class > >> >> >> > python3-pip: inherit setuptools_build_meta > >> >> >> > python3-attrs: inherit setuptools_build_meta > >> >> >> > python3-git: inherit setuptools_build_meta > >> >> >> > python3-pytest: inherit setuptools_build_meta > >> >> >> > python3-setuptools-scm: inherit setuptools_build_meta > >> >> >> > python3-zipp: inherit setuptools_build_meta > >> >> >> > python3-iniconfig: inherit setuptools_build_meta > >> >> >> > python3-py: inherit setuptools_build_meta > >> >> >> > python3-pluggy: inherit setuptools_build_meta > >> >> >> > python3-setuptools: inherit setuptools_base_meta > >> >> >> > setuptools3.bbclass: refactor for wheels > >> >> >> > python3-more-itertools: set PIP_INSTALL_PACKAGE > >> >> >> > meson: inherit setuptools_build_meta > >> >> >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE > >> >> >> > python3-smartypants: patch hash bang to python3 > >> >> >> > python3-scons: merge -native recipe > >> >> >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE > >> >> >> > python3-magic: set PIP_INSTALL_PACKAGE > >> >> >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER > >> >> >> > asciidoc: set PIP_INSTALL_PACKAGE > >> >> >> > gi-docgen: set PIP_INSTALL_PACKAGE > >> >> >> > numpy: set PIP_INSTALL_PACKAGE > >> >> >> > python3-dbusmock: set PIP_INSTALL_PACKAGE > >> >> >> > python3-mako: inherit setuptools_build_meta > >> >> >> > python3-packaging: inherit setuptools_build_meta > >> >> >> > python3-nose: drop recipe > >> >> >> > > >> >> >> > meta/classes/flit_core.bbclass | 16 +++ > >> >> >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ > >> >> >> > meta/classes/setuptools3.bbclass | 20 ++-- > >> >> >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ > >> >> >> > meta/conf/distro/include/maintainers.inc | 3 +- > >> >> >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- > >> >> >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- > >> >> >> > .../python/python-subunit.inc | 14 --- > >> >> >> > .../python/python3-attrs_21.4.0.bb | 2 +- > >> >> >> > .../python/python3-dbusmock_0.25.0.bb | 1 + > >> >> >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ > >> >> >> > .../python/python3-git_3.1.26.bb | 2 +- > >> >> >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- > >> >> >> > .../python/python3-libarchive-c_4.0.bb | 2 + > >> >> >> > .../python/python3-magic_0.4.25.bb | 1 + > >> >> >> > .../python/python3-mako_1.1.6.bb | 2 +- > >> >> >> > .../python/python3-more-itertools_8.12.0.bb | 2 + > >> >> >> > .../python/python3-nose_1.3.7.bb | 2 - > >> >> >> > .../python/python3-numpy_1.22.2.bb | 1 + > >> >> >> > .../python/python3-packaging_21.3.bb | 2 +- > >> >> >> > .../python/python3-pip_22.0.3.bb | 19 +++- > >> >> >> > .../python/python3-pluggy_1.0.0.bb | 2 +- > >> >> >> > .../python/python3-py_1.11.0.bb | 2 +- > >> >> >> > .../python/python3-pytest_7.0.1.bb | 4 +- > >> >> >> > .../python/python3-scons-native_4.3.0.bb | 7 -- > >> >> >> > .../python/python3-scons_4.3.0.bb | 13 ++- > >> >> >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- > >> >> >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- > >> >> >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ > >> >> >> > .../python/python3-smartypants_2.0.0.bb | 1 + > >> >> >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ > >> >> >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- > >> >> >> > .../python/python3-subunit_1.4.0.bb | 17 ++- > >> >> >> > ...roject.toml-from-flit-backend-branch.patch | 100 > ++++++++++++++++++ > >> >> >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ > >> >> >> > .../python/python3-zipp_3.7.0.bb | 2 +- > >> >> >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + > >> >> >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + > >> >> >> > .../{bmap-tools_3.6.bb =3D> bmap-tools_git.bb} | 6 +- > >> >> >> > 39 files changed, 412 insertions(+), 78 deletions(-) > >> >> >> > create mode 100644 meta/classes/flit_core.bbclass > >> >> >> > create mode 100644 meta/classes/pip_install_wheel.bbclass > >> >> >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass > >> >> >> > delete mode 100644 meta/recipes-devtools/python/python-nose.i= nc > >> >> >> > delete mode 100644 > meta/recipes-devtools/python/python-subunit.inc > >> >> >> > create mode 100644 meta/recipes-devtools/python/ > python3-flit-core_3.6.0.bb > >> >> >> > delete mode 100644 meta/recipes-devtools/python/ > python3-nose_1.3.7.bb > >> >> >> > delete mode 100644 meta/recipes-devtools/python/ > python3-scons-native_4.3.0.bb > >> >> >> > create mode 100644 > meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to= -python3.patch > >> >> >> > create mode 100644 > meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-ven= dored-_distutils.patch > >> >> >> > create mode 100644 > meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-f= rom-flit-backend-branch.patch > >> >> >> > create mode 100644 meta/recipes-devtools/python/ > python3-wheel_0.37.1.bb > >> >> >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb =3D= > > bmap-tools_git.bb} (86%) > >> >> >> > > >> >> >> > -- > >> >> >> > 2.30.2 > >> >> >> > > >> >> >> > > >> >> >> > -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- > >> >> >> > Links: You receive all messages sent to this group. > >> >> >> > View/Reply Online (#162076): > https://lists.openembedded.org/g/openembedded-core/message/162076 > >> >> >> > Mute This Topic: > https://lists.openembedded.org/mt/89304007/1997914 > >> >> >> > Group Owner: openembedded-core+owner@lists.openembedded.org > >> >> >> > Unsubscribe: > https://lists.openembedded.org/g/openembedded-core/unsub [ > raj.khem@gmail.com] > >> >> >> > -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- > >> >> >> > > --00000000000049d8a805d8bc5374 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Wed, Feb 23, 2022 at 8:07 PM Khem = Raj <raj.khem@gmail.com> wr= ote:
Right now a= pprox 150 out of 470 recipes [1] in meta-python are directly failing,
perhaps there will be more once these are fixed then there are other layers=
which might be carrying some impacted python modules.

=
The issue has mostly been detection of the wheel filename and th= is should be improved by:

Also YP compatibility
is broken due to python3-nose removal.

We = can try to run 2to3 on the python3-nose plugins or remove them surgically t= o reintroduce cleanly Python 3 only
python3-nose. I propose that = we move it to meta-python even so because nothing in oe-core depends on nos= e and the
code has been completely bit-rotting since 2016. It cou= ld not be built anymore in oe-core and something had to give.
=C2= =A0
If we are planni= ng
to include this in the upcoming release,we would need some commitment
and strategy to fix these and other ripple effects throughout the ecosystem=
especially since this is an LTS release. I wish this changeset was proposed=
in a release prior to LTS release, so it could have given enough time to adjust for the ecosystem. This is will take some effort in stabilizing
<= /blockquote>

This has been months of work on my part to = attempt to get us to where upstream
Python Packaging Authority ha= s been pushing since PEP-517 was announced in 2015.
We are very f= ar behind the curve of upstream Python packaging.

= While I too would love to have had this ready sooner, that was not possible= for many reasons.
But no matter what, this would have been painf= ul until recipes could be migrated, even if had
been available at= M1.


I also wonder, what do we lose if we do not put this into the LTS
release?

We already have recipes in oe-core= such as python3-jsonschema which can no longer be upgraded
witho= ut moving to pyproject.toml based package builds. I'm sure there are ot= hers in meta-python and
other layers in a similar story. We once = again are in a situation that is a bad image for the project because
<= div>we are using the now deprecated "setup.py install" and still = building eggs. To step into an LTS release
with the python packag= ing already entirely deprecated is just not a good impression.
I understand your frustration with the build failures, but it = has been a very solid week of getting oe-core

I am running a build now for me= ta-python with the improved wheel filename patch as well as a patch to
check for whether the current recipe already has pyproject.toml back-= end declared:

=C2=A0
=C2=A0[1] https://autobuilder.= yoctoproject.org/typhoon/#/builders/88/builds/1460

On Tue, Feb 22, 2022 at 11:23 AM Tim Orling <ticotimo@gmail.com> wrote:
>
>
>
> On Tue, Feb 22, 2022 at 11:18 AM Khem Raj <raj.khem@gmail.com> wrote:
>>
>> On Tue, Feb 22, 2022 at 10:40 AM Tim Orling <ticotimo@gmail.com> wrote:
>> >
>> >
>> >
>> > On Tue, Feb 22, 2022 at 10:24 AM Khem Raj <raj.khem@gmail.com> wrote:<= br> >> >>
>> >> On Tue, Feb 22, 2022 at 10:21 AM Tim Orling <ticotimo@gmail.com>= wrote:
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj <raj.khem@gmail.com>= wrote:
>> >> >>
>> >> >> this is going into dep loops Try
>> >> >>
>> >> >> bitbake python3-wheel-native
>> >> >
>> >> >
>> >> > It shouldn=E2=80=99t
>> >> > https://git.yoctoproject.or= g/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h= =3Dtimo/bootstrap-wheels
>> >> >
>> >>
>> >> I have been trying whats on master-next
>> >>
>> >
>> > I can't reproduce. Can you point me at an error log pleas= e?
>> > It might have something to do with sstate cache, but that'= ;s just a hunch.
>> >
>>
>> This is because python3-wheel was moved to core and modified but n= ot
>> deleted from meta-python at same time. I have sent a patch for tha= t
>>
> Already sent one:
> https://patchwork.yoctoproject.org/project/oe/patch/20220222184920.23= 10391-1-tim.orling@konsulko.com/
>
>>
>> >>
>> >> > DEPENDS:remove:class-native =3D "python3-pip-na= tive"
>> >> >
>> >> > do_install:class-native () {
>> >> >=C2=A0 =C2=A0 =C2=A0# We need to bootstrap python3-wh= eel-native
>> >> >=C2=A0 =C2=A0 =C2=A0install -d ${D}${PYTHON_SITEPACKA= GES_DIR}
>> >> >=C2=A0 =C2=A0 =C2=A0PYPA_WHEEL=3D"${B}/dist/${PY= PI_PACKAGE}-${PV}-*.whl"
>> >> >=C2=A0 =C2=A0 =C2=A0unzip -d ${D}${PYTHON_SITEPACKAGE= S_DIR} ${PYPA_WHEEL} || \
>> >> >=C2=A0 =C2=A0 =C2=A0bbfatal_log "Failed to insta= ll"
>> >> > }
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling <<= a href=3D"mailto:ticotimo@gmail.com" target=3D"_blank">ticotimo@gmail.com> wrote:
>> >> >> >
>> >> >> > Upstream Python is moving to pyproject.toml= rather than setup.py and
>> >> >> > wheels rather than eggs. The main place thi= s is documented is in
>> >> >> > PEP-517[1]. Installing packages with setup.= py install is deprecated.
>> >> >> >
>> >> >> > We have a choice with the 'kirkstone= 9; LTS release of introducing this
>> >> >> > series (and fixing any fall out during the = stabilization period) or
>> >> >> > living with deprecated Python packaging for= the duration of the LTS (at
>> >> >> > a minimum 2 years). This seems to be a comp= elling enough reason to bring
>> >> >> > in this series, even if it results in a sli= ght slip of the 'feature
>> >> >> > freeze', which is officially tomorrow, = Monday 21 Feb, 2022.
>> >> >> >
>> >> >> > This series bootstraps python3-setuptools-n= ative,
>> >> >> > python3-flit-core-native, python3-wheel-nat= ive and python3-pip-native,
>> >> >> > simply unzipping the wheel into PYTHON_SITE= PACKAGES_DIR. This allows all
>> >> >> > other recipes to be installed with pip, as = intended by upstream.
>> >> >> >
>> >> >> > Three new classes are introduced:
>> >> >> > (1) flit_core which builds PEP-517 complian= t packages which use
>> >> >> >=C2=A0 =C2=A0 =C2=A0flit_core.buildapi as th= eir build-backend.
>> >> >> > (2) setuptools_build_meta which builds PEP-= 517 compliant packages which use
>> >> >> >=C2=A0 =C2=A0 =C2=A0setuptools_build_meta=C2= =A0 as their build-backend.
>> >> >> > (3) pip_install_wheel, which simply install= s wheels built by other
>> >> >> >=C2=A0 =C2=A0 =C2=A0classes.
>> >> >> >
>> >> >> > The original setuptools3.bbclass do_compile= is modified to
>> >> >> > 'setup.py bdist_wheel' (from python= 3-wheel-native) rather than the deprecated
>> >> >> > 'setup.py build' mechanism.
>> >> >> >
>> >> >> > The original setuptools3.bbclass do_install= is modified to
>> >> >> > use the pip_install_wheel.bbclass to instal= l wheels with pip rather than
>> >> >> > the deprecated 'setup.py install' m= echanism.
>> >> >> >
>> >> >> > For pip install, we disable dependency chec= king (--no-deps) because it
>> >> >> > proves to be brittle in our environment wit= hout providing full wheel
>> >> >> > cache that seems like overhead that we do n= ot need. We also disable
>> >> >> > fetching wheels from pypi (--no-index). Thi= s is no different than our
>> >> >> > normal workflow of requiring dependencies t= o be defined in a recipe.
>> >> >> >
>> >> >> > Missing from this series is documentation o= f the new features, classes
>> >> >> > and variables (this will be done under sepa= rate cover).
>> >> >> >
>> >> >> > Also desired and previously discussed in YP= meetings is a helper
>> >> >> > function to warn that a recipe which curren= tly inherits setuptools3
>> >> >> > but has a pyproject.toml and defines a buil= d-system.build-backend should
>> >> >> > instead use of the new PEP-517 classes. Thi= s function is still in progress.
>> >> >> >
>> >> >> > During testing, recipes which use python3-s= etuptools-scm are failing to
>> >> >> > determine the version of the python package= , For several recipes, the fix
>> >> >> > was to use pyproject.toml and PEP-517 build= backends. To be sure, other recipes
>> >> >> > are still going to fail for the short term,= until the root cause can be
>> >> >> > determined. This was not seen again in v2 b= uilds.
>> >> >> >
>> >> >> > Another PEP-517 build-backend, poetry.core.= masonry.api has been
>> >> >> > identified in recipes in meta-python, but n= ot in oe-core. The
>> >> >> > python3-poetry-core recipe is WIP, but brin= gs in a number of
>> >> >> > dependencies that are perhaps not worth it = to bring into oe-core at this
>> >> >> > time. Rather we can support these recipes i= n meta-python?
>> >> >> >
>> >> >> > Fixes in v2 include passing --root=3D${D} t= o pip install (rather than --target)
>> >> >> > and also passing --prefix so that scripts a= re (usually) installed in the proper
>> >> >> > location. pip install also 'helpfully&#= 39; overwrites python3 intepreter with nativepython3
>> >> >> > in #!, so add a fix to return the values to= what is actually correct (similar to what is
>> >> >> > done in setuptools3.bbclass). Also several = recipes still needed PIP_INSTALL_PACKAGE defined.
>> >> >> >
>> >> >> > Also added self as maintainer of new recipe= s.
>> >> >> >
>> >> >> > [YOCTO #14638]
>> >> >> >
>> >> >> > [1] https://www.python.org/= dev/peps/pep-0517/
>> >> >> >
>> >> >> > The following changes since commit 41ae26ec= ab731f2fb6593df143476c94183be209:
>> >> >> >
>> >> >> >=C2=A0 =C2=A0releases: update to include 3.4= .2 (2022-02-21 16:46:22 +0000)
>> >> >> >
>> >> >> > are available in the Git repository at:
>> >> >> >
>> >> >> >=C2=A0 =C2=A0git://git.yo= ctoproject.org/git/poky-contrib timo/bootstrap-wheels
>> >> >> >=C2=A0 =C2=A0http://git.yoctoproject.org/cgit.cgi/git/poky-c= ontrib/log/?h=3Dtimo/bootstrap-wheels
>> >> >> >
>> >> >> > Tim Orling (32):
>> >> >> >=C2=A0 =C2=A0python3-wheel: move 0.37.0 from= meta-python
>> >> >> >=C2=A0 =C2=A0python3-flit-core: add recipe f= or 3.6.0
>> >> >> >=C2=A0 =C2=A0flit_core.bbclass: add helper f= or newer python packaging
>> >> >> >=C2=A0 =C2=A0python3-wheel: inherit flit_cor= e
>> >> >> >=C2=A0 =C2=A0pip_install_wheel.bbclass: add = helper class
>> >> >> >=C2=A0 =C2=A0setuptools_build_meta.bbclass: = add helper class
>> >> >> >=C2=A0 =C2=A0python3-pip: inherit setuptools= _build_meta
>> >> >> >=C2=A0 =C2=A0python3-attrs: inherit setuptoo= ls_build_meta
>> >> >> >=C2=A0 =C2=A0python3-git: inherit setuptools= _build_meta
>> >> >> >=C2=A0 =C2=A0python3-pytest: inherit setupto= ols_build_meta
>> >> >> >=C2=A0 =C2=A0python3-setuptools-scm: inherit= setuptools_build_meta
>> >> >> >=C2=A0 =C2=A0python3-zipp: inherit setuptool= s_build_meta
>> >> >> >=C2=A0 =C2=A0python3-iniconfig: inherit setu= ptools_build_meta
>> >> >> >=C2=A0 =C2=A0python3-py: inherit setuptools_= build_meta
>> >> >> >=C2=A0 =C2=A0python3-pluggy: inherit setupto= ols_build_meta
>> >> >> >=C2=A0 =C2=A0python3-setuptools: inherit set= uptools_base_meta
>> >> >> >=C2=A0 =C2=A0setuptools3.bbclass: refactor f= or wheels
>> >> >> >=C2=A0 =C2=A0python3-more-itertools: set PIP= _INSTALL_PACKAGE
>> >> >> >=C2=A0 =C2=A0meson: inherit setuptools_build= _meta
>> >> >> >=C2=A0 =C2=A0python3-libarchive-d: set PIP_I= NSTALL_PACKAGE
>> >> >> >=C2=A0 =C2=A0python3-smartypants: patch hash= bang to python3
>> >> >> >=C2=A0 =C2=A0python3-scons: merge -native re= cipe
>> >> >> >=C2=A0 =C2=A0python3-subunit: merge inc; set= PIP_INSTALL_PACKAGE
>> >> >> >=C2=A0 =C2=A0python3-magic: set PIP_INSTALL_= PACKAGE
>> >> >> >=C2=A0 =C2=A0bmap-tools: set PIP_INSTALL_PAC= KAGE, BASEVER
>> >> >> >=C2=A0 =C2=A0asciidoc: set PIP_INSTALL_PACKA= GE
>> >> >> >=C2=A0 =C2=A0gi-docgen: set PIP_INSTALL_PACK= AGE
>> >> >> >=C2=A0 =C2=A0numpy: set PIP_INSTALL_PACKAGE<= br> >> >> >> >=C2=A0 =C2=A0python3-dbusmock: set PIP_INSTA= LL_PACKAGE
>> >> >> >=C2=A0 =C2=A0python3-mako: inherit setuptool= s_build_meta
>> >> >> >=C2=A0 =C2=A0python3-packaging: inherit setu= ptools_build_meta
>> >> >> >=C2=A0 =C2=A0python3-nose: drop recipe
>> >> >> >
>> >> >> >=C2=A0 meta/classes/flit_core.bbclass=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 16 +++
>> >> >> >=C2=A0 meta/classes/pip_install_wheel.bbclas= s=C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 37 +++++++
>> >> >> >=C2=A0 meta/classes/setuptools3.bbclass=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 20 ++--
>> >> >> >=C2=A0 meta/classes/setuptools_build_meta.bb= class=C2=A0 =C2=A0 |=C2=A0 18 ++++
>> >> >> >=C2=A0 meta/conf/distro/include/maintainers.= inc=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A03 +-
>> >> >> >=C2=A0 meta/recipes-devtools/meson/meson_0.61= .2.bb=C2=A0 =C2=A0|=C2=A0 =C2=A04 +-
>> >> >> >=C2=A0 meta/recipes-devtools/python/python-n= ose.inc=C2=A0 |=C2=A0 18 ----
>> >> >> >=C2=A0 .../python/python-subunit.inc=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 14 ---
>> >> >> >=C2=A0 .../python/python3-attrs_21.4.0.= bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A02 +-
>> >> >> >=C2=A0 .../python/python3-dbusmock_0= .25.0.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A01 +
>> >> >> >=C2=A0 .../python/python3-flit-core_= 3.6.0.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 41 +++++++
>> >> >> >=C2=A0 .../python/python3-git_3.1.26.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A02 +-
>> >> >> >=C2=A0 .../python/python3-iniconfig_= 1.1.1.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A04 +-
>> >> >> >=C2=A0 .../python/python3-libarchiv= e-c_4.0.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A02 +
>> >> >> >=C2=A0 .../python/python3-magic_0.4.25.= bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A01 +
>> >> >> >=C2=A0 .../python/python3-mako_1.1.6.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A02 +-
>> >> >> >=C2=A0 .../python/python3-more= -itertools_8.12.0.bb=C2=A0 =C2=A0|=C2=A0 =C2=A02 +
>> >> >> >=C2=A0 .../python/python3-nose_1.3.7.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A02 -
>> >> >> >=C2=A0 .../python/python3-numpy_1.22.2.= bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A01 +
>> >> >> >=C2=A0 .../python/python3-packaging_2= 1.3.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A02 +-
>> >> >> >=C2=A0 .../python/python3-pip_22.0.3.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 19 +++-
>> >> >> >=C2=A0 .../python/python3-pluggy_1.0.0.= bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A02 +-
>> >> >> >=C2=A0 .../python/python3-py_1.11.0.bb= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A02 +- >> >> >> >=C2=A0 .../python/python3-pytest_7.0.1.= bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A04 +-
>> >> >> >=C2=A0 .../python/python3-scons-n= ative_4.3.0.bb=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A07 --
>> >> >> >=C2=A0 .../python/python3-scons_4.3.0.bb= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 13 ++-
>> >> >> >=C2=A0 .../python/python3-setup= tools-scm_6.4.2.bb=C2=A0 =C2=A0 |=C2=A0 =C2=A02 +-
>> >> >> >=C2=A0 .../python/python3-setuptoo= ls_59.5.0.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 19 +++-
>> >> >> >=C2=A0 .../0001-Change-hash-bang-to-python3.= patch=C2=A0 =C2=A0 |=C2=A0 47 ++++++++
>> >> >> >=C2=A0 .../python/python3-smartypa= nts_2.0.0.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A01 +
>> >> >> >=C2=A0 ...001-setup.py-use-vendored-_distuti= ls.patch |=C2=A0 27 +++++
>> >> >> >=C2=A0 .../python/python3-strict-= rfc3339_0.7.bb=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A04 +-
>> >> >> >=C2=A0 .../python/python3-subunit_1.4.= 0.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 17 ++-
>> >> >> >=C2=A0 ...roject.toml-from-flit-backend-bran= ch.patch | 100 ++++++++++++++++++
>> >> >> >=C2=A0 .../python/python3-wheel_0.37.1.= bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 24 +++++
>> >> >> >=C2=A0 .../python/python3-zipp_3.7.0.bb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A02 +-
>> >> >> >=C2=A0 .../asciidoc/asciidoc_10.1.1.bb= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A01 + >> >> >> >=C2=A0 meta/recipes-gnome/gi-docgen/gi-docgen= _git.bb |=C2=A0 =C2=A01 +
>> >> >> >=C2=A0 .../{bmap-tools_3.6.bb =3D> bmap= -tools_git.bb}=C2=A0 |=C2=A0 =C2=A06 +-
>> >> >> >=C2=A0 39 files changed, 412 insertions(+), = 78 deletions(-)
>> >> >> >=C2=A0 create mode 100644 meta/classes/flit_= core.bbclass
>> >> >> >=C2=A0 create mode 100644 meta/classes/pip_i= nstall_wheel.bbclass
>> >> >> >=C2=A0 create mode 100644 meta/classes/setup= tools_build_meta.bbclass
>> >> >> >=C2=A0 delete mode 100644 meta/recipes-devto= ols/python/python-nose.inc
>> >> >> >=C2=A0 delete mode 100644 meta/recipes-devto= ols/python/python-subunit.inc
>> >> >> >=C2=A0 create mode 100644 meta/recipes-devto= ols/python/python3-flit-core_3.6.0.bb
>> >> >> >=C2=A0 delete mode 100644 meta/recipes-devto= ols/python/python3-nose_1.3.7.bb
>> >> >> >=C2=A0 delete mode 100644 meta/recipes-devto= ols/python/python3-scons-native_4.3.0.bb
>> >> >> >=C2=A0 create mode 100644 meta/recipes-devto= ols/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch
>> >> >> >=C2=A0 create mode 100644 meta/recipes-devto= ols/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.pat= ch
>> >> >> >=C2=A0 create mode 100644 meta/recipes-devto= ols/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-bra= nch.patch
>> >> >> >=C2=A0 create mode 100644 meta/recipes-devto= ols/python/python3-wheel_0.37.1.bb
>> >> >> >=C2=A0 rename meta/recipes-support/bmap-tool= s/{bmap-tools_3.6.bb =3D> bmap-tools_git.bb} (86%)
>> >> >> >
>> >> >> > --
>> >> >> > 2.30.2
>> >> >> >
>> >> >> >
>> >> >> > -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-= =3D-
>> >> >> > Links: You receive all messages sent to thi= s group.
>> >> >> > View/Reply Online (#162076): https://lists.openembedded.org/g/openembedded-cor= e/message/162076
>> >> >> > Mute This Topic: h= ttps://lists.openembedded.org/mt/89304007/1997914
>> >> >> > Group Owner: openembedded-core+o= wner@lists.openembedded.org
>> >> >> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
>> >> >> > -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-= =3D-
>> >> >> >
--00000000000049d8a805d8bc5374--