On Wed, Feb 23, 2022 at 8:07 PM Khem Raj wrote: > Right now approx 150 out of 470 recipes [1] in meta-python are directly > failing, > perhaps there will be more once these are fixed then there are other layers > which might be carrying some impacted python modules. The issue has mostly been detection of the wheel filename and this should be improved by: https://lore.kernel.org/openembedded-core/20220224041200.2645054-1-tim.orling@konsulko.com/T/#t Also YP compatibility > is broken due to python3-nose removal. We can try to run 2to3 on the python3-nose plugins or remove them surgically to reintroduce cleanly Python 3 only python3-nose. I propose that we move it to meta-python even so because nothing in oe-core depends on nose and the code has been completely bit-rotting since 2016. It could not be built anymore in oe-core and something had to give. > If we are planning > to include this in the upcoming release,we would need some commitment > and strategy to fix these and other ripple effects throughout the ecosystem > especially since this is an LTS release. I wish this changeset was proposed > in a release prior to LTS release, so it could have given enough time to > adjust for the ecosystem. This is will take some effort in stabilizing > This has been months of work on my part to attempt to get us to where upstream Python Packaging Authority has been pushing since PEP-517 was announced in 2015. We are very far behind the curve of upstream Python packaging. While I too would love to have had this ready sooner, that was not possible for many reasons. But no matter what, this would have been painful until recipes could be migrated, even if had been available at M1. > I also wonder, what do we lose if we do not put this into the LTS > release? We already have recipes in oe-core such as python3-jsonschema which can no longer be upgraded without moving to pyproject.toml based package builds. I'm sure there are others in meta-python and other layers in a similar story. We once again are in a situation that is a bad image for the project because we are using the now deprecated "setup.py install" and still building eggs. To step into an LTS release with the python packaging already entirely deprecated is just not a good impression. I understand your frustration with the build failures, but it has been a very solid week of getting oe-core stabilized (see https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3280). I am running a build now for meta-python with the improved wheel filename patch as well as a patch to check for whether the current recipe already has pyproject.toml back-end declared: https://lore.kernel.org/openembedded-core/20220224041200.2645054-2-tim.orling@konsulko.com/T/#u > [1] https://autobuilder.yoctoproject.org/typhoon/#/builders/88/builds/1460 > > On Tue, Feb 22, 2022 at 11:23 AM Tim Orling wrote: > > > > > > > > On Tue, Feb 22, 2022 at 11:18 AM Khem Raj wrote: > >> > >> On Tue, Feb 22, 2022 at 10:40 AM Tim Orling wrote: > >> > > >> > > >> > > >> > On Tue, Feb 22, 2022 at 10:24 AM Khem Raj wrote: > >> >> > >> >> On Tue, Feb 22, 2022 at 10:21 AM Tim Orling > wrote: > >> >> > > >> >> > > >> >> > > >> >> > On Tue, Feb 22, 2022 at 9:22 AM Khem Raj > wrote: > >> >> >> > >> >> >> this is going into dep loops Try > >> >> >> > >> >> >> bitbake python3-wheel-native > >> >> > > >> >> > > >> >> > It shouldn’t > >> >> > > https://git.yoctoproject.org/poky-contrib/tree/meta/recipes-devtools/python/python3-wheel_0.37.1.bb?h=timo/bootstrap-wheels > >> >> > > >> >> > >> >> I have been trying whats on master-next > >> >> > >> > > >> > I can't reproduce. Can you point me at an error log please? > >> > It might have something to do with sstate cache, but that's just a > hunch. > >> > > >> > >> This is because python3-wheel was moved to core and modified but not > >> deleted from meta-python at same time. I have sent a patch for that > >> > > Already sent one: > > > https://patchwork.yoctoproject.org/project/oe/patch/20220222184920.2310391-1-tim.orling@konsulko.com/ > > > >> > >> >> > >> >> > DEPENDS:remove:class-native = "python3-pip-native" > >> >> > > >> >> > do_install:class-native () { > >> >> > # We need to bootstrap python3-wheel-native > >> >> > install -d ${D}${PYTHON_SITEPACKAGES_DIR} > >> >> > PYPA_WHEEL="${B}/dist/${PYPI_PACKAGE}-${PV}-*.whl" > >> >> > unzip -d ${D}${PYTHON_SITEPACKAGES_DIR} ${PYPA_WHEEL} || \ > >> >> > bbfatal_log "Failed to install" > >> >> > } > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> On Mon, Feb 21, 2022 at 1:23 PM Tim Orling > wrote: > >> >> >> > > >> >> >> > Upstream Python is moving to pyproject.toml rather than > setup.py and > >> >> >> > wheels rather than eggs. The main place this is documented is in > >> >> >> > PEP-517[1]. Installing packages with setup.py install is > deprecated. > >> >> >> > > >> >> >> > We have a choice with the 'kirkstone' LTS release of > introducing this > >> >> >> > series (and fixing any fall out during the stabilization > period) or > >> >> >> > living with deprecated Python packaging for the duration of the > LTS (at > >> >> >> > a minimum 2 years). This seems to be a compelling enough reason > to bring > >> >> >> > in this series, even if it results in a slight slip of the > 'feature > >> >> >> > freeze', which is officially tomorrow, Monday 21 Feb, 2022. > >> >> >> > > >> >> >> > This series bootstraps python3-setuptools-native, > >> >> >> > python3-flit-core-native, python3-wheel-native and > python3-pip-native, > >> >> >> > simply unzipping the wheel into PYTHON_SITEPACKAGES_DIR. This > allows all > >> >> >> > other recipes to be installed with pip, as intended by upstream. > >> >> >> > > >> >> >> > Three new classes are introduced: > >> >> >> > (1) flit_core which builds PEP-517 compliant packages which use > >> >> >> > flit_core.buildapi as their build-backend. > >> >> >> > (2) setuptools_build_meta which builds PEP-517 compliant > packages which use > >> >> >> > setuptools_build_meta as their build-backend. > >> >> >> > (3) pip_install_wheel, which simply installs wheels built by > other > >> >> >> > classes. > >> >> >> > > >> >> >> > The original setuptools3.bbclass do_compile is modified to > >> >> >> > 'setup.py bdist_wheel' (from python3-wheel-native) rather than > the deprecated > >> >> >> > 'setup.py build' mechanism. > >> >> >> > > >> >> >> > The original setuptools3.bbclass do_install is modified to > >> >> >> > use the pip_install_wheel.bbclass to install wheels with pip > rather than > >> >> >> > the deprecated 'setup.py install' mechanism. > >> >> >> > > >> >> >> > For pip install, we disable dependency checking (--no-deps) > because it > >> >> >> > proves to be brittle in our environment without providing full > wheel > >> >> >> > cache that seems like overhead that we do not need. We also > disable > >> >> >> > fetching wheels from pypi (--no-index). This is no different > than our > >> >> >> > normal workflow of requiring dependencies to be defined in a > recipe. > >> >> >> > > >> >> >> > Missing from this series is documentation of the new features, > classes > >> >> >> > and variables (this will be done under separate cover). > >> >> >> > > >> >> >> > Also desired and previously discussed in YP meetings is a helper > >> >> >> > function to warn that a recipe which currently inherits > setuptools3 > >> >> >> > but has a pyproject.toml and defines a > build-system.build-backend should > >> >> >> > instead use of the new PEP-517 classes. This function is still > in progress. > >> >> >> > > >> >> >> > During testing, recipes which use python3-setuptools-scm are > failing to > >> >> >> > determine the version of the python package, For several > recipes, the fix > >> >> >> > was to use pyproject.toml and PEP-517 build backends. To be > sure, other recipes > >> >> >> > are still going to fail for the short term, until the root > cause can be > >> >> >> > determined. This was not seen again in v2 builds. > >> >> >> > > >> >> >> > Another PEP-517 build-backend, poetry.core.masonry.api has been > >> >> >> > identified in recipes in meta-python, but not in oe-core. The > >> >> >> > python3-poetry-core recipe is WIP, but brings in a number of > >> >> >> > dependencies that are perhaps not worth it to bring into > oe-core at this > >> >> >> > time. Rather we can support these recipes in meta-python? > >> >> >> > > >> >> >> > Fixes in v2 include passing --root=${D} to pip install (rather > than --target) > >> >> >> > and also passing --prefix so that scripts are (usually) > installed in the proper > >> >> >> > location. pip install also 'helpfully' overwrites python3 > intepreter with nativepython3 > >> >> >> > in #!, so add a fix to return the values to what is actually > correct (similar to what is > >> >> >> > done in setuptools3.bbclass). Also several recipes still needed > PIP_INSTALL_PACKAGE defined. > >> >> >> > > >> >> >> > Also added self as maintainer of new recipes. > >> >> >> > > >> >> >> > [YOCTO #14638] > >> >> >> > > >> >> >> > [1] https://www.python.org/dev/peps/pep-0517/ > >> >> >> > > >> >> >> > The following changes since commit > 41ae26ecab731f2fb6593df143476c94183be209: > >> >> >> > > >> >> >> > releases: update to include 3.4.2 (2022-02-21 16:46:22 +0000) > >> >> >> > > >> >> >> > are available in the Git repository at: > >> >> >> > > >> >> >> > git://git.yoctoproject.org/git/poky-contrib > timo/bootstrap-wheels > >> >> >> > > http://git.yoctoproject.org/cgit.cgi/git/poky-contrib/log/?h=timo/bootstrap-wheels > >> >> >> > > >> >> >> > Tim Orling (32): > >> >> >> > python3-wheel: move 0.37.0 from meta-python > >> >> >> > python3-flit-core: add recipe for 3.6.0 > >> >> >> > flit_core.bbclass: add helper for newer python packaging > >> >> >> > python3-wheel: inherit flit_core > >> >> >> > pip_install_wheel.bbclass: add helper class > >> >> >> > setuptools_build_meta.bbclass: add helper class > >> >> >> > python3-pip: inherit setuptools_build_meta > >> >> >> > python3-attrs: inherit setuptools_build_meta > >> >> >> > python3-git: inherit setuptools_build_meta > >> >> >> > python3-pytest: inherit setuptools_build_meta > >> >> >> > python3-setuptools-scm: inherit setuptools_build_meta > >> >> >> > python3-zipp: inherit setuptools_build_meta > >> >> >> > python3-iniconfig: inherit setuptools_build_meta > >> >> >> > python3-py: inherit setuptools_build_meta > >> >> >> > python3-pluggy: inherit setuptools_build_meta > >> >> >> > python3-setuptools: inherit setuptools_base_meta > >> >> >> > setuptools3.bbclass: refactor for wheels > >> >> >> > python3-more-itertools: set PIP_INSTALL_PACKAGE > >> >> >> > meson: inherit setuptools_build_meta > >> >> >> > python3-libarchive-d: set PIP_INSTALL_PACKAGE > >> >> >> > python3-smartypants: patch hash bang to python3 > >> >> >> > python3-scons: merge -native recipe > >> >> >> > python3-subunit: merge inc; set PIP_INSTALL_PACKAGE > >> >> >> > python3-magic: set PIP_INSTALL_PACKAGE > >> >> >> > bmap-tools: set PIP_INSTALL_PACKAGE, BASEVER > >> >> >> > asciidoc: set PIP_INSTALL_PACKAGE > >> >> >> > gi-docgen: set PIP_INSTALL_PACKAGE > >> >> >> > numpy: set PIP_INSTALL_PACKAGE > >> >> >> > python3-dbusmock: set PIP_INSTALL_PACKAGE > >> >> >> > python3-mako: inherit setuptools_build_meta > >> >> >> > python3-packaging: inherit setuptools_build_meta > >> >> >> > python3-nose: drop recipe > >> >> >> > > >> >> >> > meta/classes/flit_core.bbclass | 16 +++ > >> >> >> > meta/classes/pip_install_wheel.bbclass | 37 +++++++ > >> >> >> > meta/classes/setuptools3.bbclass | 20 ++-- > >> >> >> > meta/classes/setuptools_build_meta.bbclass | 18 ++++ > >> >> >> > meta/conf/distro/include/maintainers.inc | 3 +- > >> >> >> > meta/recipes-devtools/meson/meson_0.61.2.bb | 4 +- > >> >> >> > meta/recipes-devtools/python/python-nose.inc | 18 ---- > >> >> >> > .../python/python-subunit.inc | 14 --- > >> >> >> > .../python/python3-attrs_21.4.0.bb | 2 +- > >> >> >> > .../python/python3-dbusmock_0.25.0.bb | 1 + > >> >> >> > .../python/python3-flit-core_3.6.0.bb | 41 +++++++ > >> >> >> > .../python/python3-git_3.1.26.bb | 2 +- > >> >> >> > .../python/python3-iniconfig_1.1.1.bb | 4 +- > >> >> >> > .../python/python3-libarchive-c_4.0.bb | 2 + > >> >> >> > .../python/python3-magic_0.4.25.bb | 1 + > >> >> >> > .../python/python3-mako_1.1.6.bb | 2 +- > >> >> >> > .../python/python3-more-itertools_8.12.0.bb | 2 + > >> >> >> > .../python/python3-nose_1.3.7.bb | 2 - > >> >> >> > .../python/python3-numpy_1.22.2.bb | 1 + > >> >> >> > .../python/python3-packaging_21.3.bb | 2 +- > >> >> >> > .../python/python3-pip_22.0.3.bb | 19 +++- > >> >> >> > .../python/python3-pluggy_1.0.0.bb | 2 +- > >> >> >> > .../python/python3-py_1.11.0.bb | 2 +- > >> >> >> > .../python/python3-pytest_7.0.1.bb | 4 +- > >> >> >> > .../python/python3-scons-native_4.3.0.bb | 7 -- > >> >> >> > .../python/python3-scons_4.3.0.bb | 13 ++- > >> >> >> > .../python/python3-setuptools-scm_6.4.2.bb | 2 +- > >> >> >> > .../python/python3-setuptools_59.5.0.bb | 19 +++- > >> >> >> > .../0001-Change-hash-bang-to-python3.patch | 47 ++++++++ > >> >> >> > .../python/python3-smartypants_2.0.0.bb | 1 + > >> >> >> > ...001-setup.py-use-vendored-_distutils.patch | 27 +++++ > >> >> >> > .../python/python3-strict-rfc3339_0.7.bb | 4 +- > >> >> >> > .../python/python3-subunit_1.4.0.bb | 17 ++- > >> >> >> > ...roject.toml-from-flit-backend-branch.patch | 100 > ++++++++++++++++++ > >> >> >> > .../python/python3-wheel_0.37.1.bb | 24 +++++ > >> >> >> > .../python/python3-zipp_3.7.0.bb | 2 +- > >> >> >> > .../asciidoc/asciidoc_10.1.1.bb | 1 + > >> >> >> > meta/recipes-gnome/gi-docgen/gi-docgen_git.bb | 1 + > >> >> >> > .../{bmap-tools_3.6.bb => bmap-tools_git.bb} | 6 +- > >> >> >> > 39 files changed, 412 insertions(+), 78 deletions(-) > >> >> >> > create mode 100644 meta/classes/flit_core.bbclass > >> >> >> > create mode 100644 meta/classes/pip_install_wheel.bbclass > >> >> >> > create mode 100644 meta/classes/setuptools_build_meta.bbclass > >> >> >> > delete mode 100644 meta/recipes-devtools/python/python-nose.inc > >> >> >> > delete mode 100644 > meta/recipes-devtools/python/python-subunit.inc > >> >> >> > create mode 100644 meta/recipes-devtools/python/ > python3-flit-core_3.6.0.bb > >> >> >> > delete mode 100644 meta/recipes-devtools/python/ > python3-nose_1.3.7.bb > >> >> >> > delete mode 100644 meta/recipes-devtools/python/ > python3-scons-native_4.3.0.bb > >> >> >> > create mode 100644 > meta/recipes-devtools/python/python3-smartypants/0001-Change-hash-bang-to-python3.patch > >> >> >> > create mode 100644 > meta/recipes-devtools/python/python3-strict-rfc3339/0001-setup.py-use-vendored-_distutils.patch > >> >> >> > create mode 100644 > meta/recipes-devtools/python/python3-wheel/0001-Backport-pyproject.toml-from-flit-backend-branch.patch > >> >> >> > create mode 100644 meta/recipes-devtools/python/ > python3-wheel_0.37.1.bb > >> >> >> > rename meta/recipes-support/bmap-tools/{bmap-tools_3.6.bb => > bmap-tools_git.bb} (86%) > >> >> >> > > >> >> >> > -- > >> >> >> > 2.30.2 > >> >> >> > > >> >> >> > > >> >> >> > -=-=-=-=-=-=-=-=-=-=-=- > >> >> >> > Links: You receive all messages sent to this group. > >> >> >> > View/Reply Online (#162076): > https://lists.openembedded.org/g/openembedded-core/message/162076 > >> >> >> > Mute This Topic: > https://lists.openembedded.org/mt/89304007/1997914 > >> >> >> > Group Owner: openembedded-core+owner@lists.openembedded.org > >> >> >> > Unsubscribe: > https://lists.openembedded.org/g/openembedded-core/unsub [ > raj.khem@gmail.com] > >> >> >> > -=-=-=-=-=-=-=-=-=-=-=- > >> >> >> > >