[RFC PATCH 00/10] Change Python package building to use picobuild

[RFC PATCH 00/10] Change Python package building to use picobuild

[Ross Burton]

Hi,

Currently we build modern Python packages by calling the PEP-517 API directly
(see python_pep517.bbclass, do_compile()). This mostly works, but sometimes
doesn't. For example we don't verify build dependencies, which led to the cbor2
ugprade silently failing to actually package anything. We should use a proper
build frontend which actually validates dependencies and does builds in the
correct manner.

The standard frontend is pypa/build, but for source-based distributions
that can be annoying to build as it depends on the following packages:

- tomli
- pep517
- packaging
- pyparsing

Manually bootstrapping those recipes is possible, but tedious.

Picobuild is another frontend (written by myself) which is designed
explicitly to be used programatically by source-based distributions: it
doesn't support builds inside virtual environments as we're building
distribution packages, and it vendors the dependencies for bootstrapping
if they're not available.

Over time more packages are expected to move to using Flit to build which makes
the bootstrapping process slightly easier, and tomli will be integrated into
Python 3.11, so it's possible that in the future we drop picobuild and switch to
build.  But for now, this works better than calling the API directly, and can be
fairly simply swapped out in the future if needed.

The impact is pretty minimal: the PEP517_BUILD_API variable is obsolete so can
be removed everywhere, and some recipes now fail because they have missing build
dependencies.  I've already fixed up the bulk of meta-oe in
meta-oe-contrib:ross/picobuild.  Most recipes don't need changes and just build
a little better now.

This works for me but could do with further testing, but this is primarily a RFC
because I'm about to leave on holiday for a week and then have a conference, so
I don't recommend merging this straight away!

Cheers,
Ross


Ross Burton (10):
  python3-pluggy: add BBCLASSEXTEND for native/nativesdk
  setuptools3: clean up class
  python3-flit-core: bootstrap explicitly
  python3-installer: bootstrap by installing installer with installer
  python3-pep517: add new recipe
  python3-setuptools-scm: DEPEND on python3-packaging
  python3-picobuild: add new recipe
  python_pep517: use picobuild instead of manually calling the API
  classes: remove obsolete PEP517_BUILD_API
  documentation: remove obsolete PEP517_BUILD_API

 documentation/ref-manual/classes.rst          |  4 +--
 documentation/ref-manual/variables.rst        |  5 ----
 meta/classes/python_flit_core.bbclass         |  5 +++-
 meta/classes/python_pep517.bbclass            |  8 ++----
 meta/classes/python_poetry_core.bbclass       |  2 --
 .../python_setuptools_build_meta.bbclass      |  2 --
 meta/classes/setuptools3.bbclass              |  5 ++--
 .../python/python3-flit-core_3.7.1.bb         |  6 ++++-
 .../python/python3-installer_0.5.1.bb         | 14 +++++++---
 .../python/python3-pep517_0.12.0.bb           | 10 +++++++
 .../python/python3-picobuild_0.1.bb           | 27 +++++++++++++++++++
 .../python/python3-pluggy_1.0.0.bb            |  2 ++
 .../python/python3-setuptools-scm_6.4.2.bb    |  2 +-
 13 files changed, 64 insertions(+), 28 deletions(-)
 create mode 100644 meta/recipes-devtools/python/python3-pep517_0.12.0.bb
 create mode 100644 meta/recipes-devtools/python/python3-picobuild_0.1.bb

-- 
2.25.1

[RFC PATCH 01/10] python3-pluggy: add BBCLASSEXTEND for native/nativesdk

[Ross Burton]

This is a dependency of Hatchling, another Python build tool, so we need
it natively.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/recipes-devtools/python/python3-pluggy_1.0.0.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-devtools/python/python3-pluggy_1.0.0.bb b/meta/recipes-devtools/python/python3-pluggy_1.0.0.bb
index 3f2895a061b..2935acf95fa 100644
--- a/meta/recipes-devtools/python/python3-pluggy_1.0.0.bb
+++ b/meta/recipes-devtools/python/python3-pluggy_1.0.0.bb
@@ -24,3 +24,5 @@ do_install_ptest() {
 	install -d ${D}${PTEST_PATH}/testing
 	cp -rf ${S}/testing/* ${D}${PTEST_PATH}/testing/
 }
+
+BBCLASSEXTEND = "native nativesdk"
-- 
2.25.1

[RFC PATCH 02/10] setuptools3: clean up class

[Ross Burton]

Remove a commented-out B, re-order DEPENDS< and add comments.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes/setuptools3.bbclass | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/classes/setuptools3.bbclass b/meta/classes/setuptools3.bbclass
index 556bc801af7..66c94660d8e 100644
--- a/meta/classes/setuptools3.bbclass
+++ b/meta/classes/setuptools3.bbclass
@@ -1,7 +1,6 @@
 inherit setuptools3-base python_pep517
 
-# bdist_wheel builds in ./dist
-#B = "${WORKDIR}/build"
+DEPENDS += "python3-setuptools-native python3-wheel-native"
 
 SETUPTOOLS_BUILD_ARGS ?= ""
 
@@ -23,6 +22,7 @@ setuptools3_do_compile() {
 setuptools3_do_compile[vardepsexclude] = "MACHINE"
 do_compile[cleandirs] += "${PEP517_WHEEL_PATH}"
 
+# This could be removed in the future but some recipes in meta-oe still use it
 setuptools3_do_install() {
         python_pep517_do_install
 }
@@ -30,4 +30,3 @@ setuptools3_do_install() {
 EXPORT_FUNCTIONS do_configure do_compile do_install
 
 export LDSHARED="${CCLD} -shared"
-DEPENDS += "python3-setuptools-native python3-wheel-native"
-- 
2.25.1

[RFC PATCH 03/10] python3-flit-core: bootstrap explicitly

[Ross Burton]

Add a method to python_flit_core.bbclass that does a manual build with
flit explicitly, and use that to bootstrap python3-flit-core-native which
can build itself.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes/python_flit_core.bbclass                   | 5 +++++
 meta/recipes-devtools/python/python3-flit-core_3.7.1.bb | 4 ++++
 2 files changed, 9 insertions(+)

diff --git a/meta/classes/python_flit_core.bbclass b/meta/classes/python_flit_core.bbclass
index 96652aa2043..a900f7b5579 100644
--- a/meta/classes/python_flit_core.bbclass
+++ b/meta/classes/python_flit_core.bbclass
@@ -3,3 +3,8 @@ inherit python_pep517 python3native python3-dir setuptools3-base
 DEPENDS += "python3 python3-flit-core-native"
 
 PEP517_BUILD_API = "flit_core.buildapi"
+
+python_flit_core_do_manual_build () {
+    cd ${PEP517_SOURCE_PATH}
+    nativepython3 -m flit_core.wheel --outdir ${PEP517_WHEEL_PATH} .
+}
diff --git a/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb b/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb
index 8d107384a6d..abe620374c0 100644
--- a/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb
+++ b/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb
@@ -20,6 +20,10 @@ DEPENDS:append:class-native = " unzip-native"
 PYPI_PACKAGE = "flit"
 PEP517_SOURCE_PATH = "${S}/flit_core"
 
+do_compile:class-native () {
+    python_flit_core_do_manual_build
+}
+
 do_install:class-native () {
     python_pep517_do_bootstrap_install
 }
-- 
2.25.1

[RFC PATCH 04/10] python3-installer: bootstrap by installing installer with installer

[Ross Burton]

The installer can install itself by simply adding src/ to PYTHONPATH.

As this previously used unzip, we need to disable bytecode as otherwise
the newly packaged .pyc files conflict with the any generated .pyc files
in the sysroot.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/recipes-devtools/python/python3-installer_0.5.1.bb | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-devtools/python/python3-installer_0.5.1.bb b/meta/recipes-devtools/python/python3-installer_0.5.1.bb
index f4f9e1bde64..b7780702d6b 100644
--- a/meta/recipes-devtools/python/python3-installer_0.5.1.bb
+++ b/meta/recipes-devtools/python/python3-installer_0.5.1.bb
@@ -12,11 +12,13 @@ SRC_URI[sha256sum] = "f970995ec2bb815e2fdaf7977b26b2091e1e386f0f42eafd5ac811953d
 
 inherit pypi python_flit_core
 
+# Bootstrap the native install by installing with ourself
 DEPENDS:remove:class-native = "python3-installer-native"
-DEPENDS:append:class-native = " unzip-native"
 
-do_install:class-native () {
-    python_pep517_do_bootstrap_install
+INSTALL_WHEEL_COMPILE_BYTECODE:class-native = "--no-compile-bytecode"
+
+do_install:prepend:class-native() {
+    export PYTHONPATH="${S}/src"
 }
 
 BBCLASSEXTEND = "native nativesdk"
-- 
2.25.1

[RFC PATCH 05/10] python3-pep517: add new recipe

[Ross Burton]

This is a runtime dependency of most Python package build frontends.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/recipes-devtools/python/python3-pep517_0.12.0.bb | 10 ++++++++++
 1 file changed, 10 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-pep517_0.12.0.bb

diff --git a/meta/recipes-devtools/python/python3-pep517_0.12.0.bb b/meta/recipes-devtools/python/python3-pep517_0.12.0.bb
new file mode 100644
index 00000000000..43ac35a3f79
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-pep517_0.12.0.bb
@@ -0,0 +1,10 @@
+SUMMARY = "Wrappers to build Python packages using PEP 517 hooks"
+HOMEPAGE = "https://github.com/pypa/pep517"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=aad69c93f605003e3342b174d9b0708c"
+
+SRC_URI[sha256sum] = "931378d93d11b298cf511dd634cf5ea4cb249a28ef84160b3247ee9afb4e8ab0"
+
+inherit pypi python_flit_core
+
+BBCLASSEXTEND = "native nativesdk"
\ No newline at end of file
-- 
2.25.1

[RFC PATCH 06/10] python3-setuptools-scm: DEPEND on python3-packaging

[Ross Burton]

This is a build dependency that isn't flagged when we call the build
API directly.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/recipes-devtools/python/python3-setuptools-scm_6.4.2.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/python/python3-setuptools-scm_6.4.2.bb b/meta/recipes-devtools/python/python3-setuptools-scm_6.4.2.bb
index 9aaae071d31..212aa8a005a 100644
--- a/meta/recipes-devtools/python/python3-setuptools-scm_6.4.2.bb
+++ b/meta/recipes-devtools/python/python3-setuptools-scm_6.4.2.bb
@@ -11,7 +11,7 @@ inherit pypi python_setuptools_build_meta
 
 UPSTREAM_CHECK_REGEX = "setuptools_scm-(?P<pver>.*)\.tar"
 
-DEPENDS += "python3-tomli-native"
+DEPENDS += "python3-tomli-native python3-packaging-native"
 
 RDEPENDS:${PN} = "\
     ${PYTHON_PN}-packaging \
-- 
2.25.1

[RFC PATCH 07/10] python3-picobuild: add new recipe

[Ross Burton]

Picobuild is a pico-scale Python PEP517 build frontend, designed to have
minimal dependencies (via vendoring) to be well suited for building
source-based distributions, such as OpenEmbedded.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 .../python/python3-picobuild_0.1.bb           | 25 +++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-picobuild_0.1.bb

diff --git a/meta/recipes-devtools/python/python3-picobuild_0.1.bb b/meta/recipes-devtools/python/python3-picobuild_0.1.bb
new file mode 100644
index 00000000000..bbd3636eea8
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-picobuild_0.1.bb
@@ -0,0 +1,25 @@
+SUMMARY = "Library and tool for installing Python wheels"
+
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=6c811a9fbdf5641ff0b0d43fbbb440e5"
+# TODO vendorered license files
+
+SRC_URI = "git://gitlab.com/rossburton/picobuild.git;protocol=https;branch=main"
+SRCREV = "f08fe0375e2b93e3363416f210534d1ac3f8fc68"
+
+S = "${WORKDIR}/git"
+
+inherit python_flit_core
+
+# For target builds we can deleted the vendored libraries and RDEPEND on them
+# instead. Use an explicit list to be sure we delete what we RDEPEND on.
+do_install:append:class-target() {
+    rm -rf ${D}${PYTHON_SITEPACKAGES_DIR}/picobuild/vendored/packaging
+    rm -rf ${D}${PYTHON_SITEPACKAGES_DIR}/picobuild/vendored/pep517*
+    rm -rf ${D}${PYTHON_SITEPACKAGES_DIR}/picobuild/vendored/pyparsing
+    rm -rf ${D}${PYTHON_SITEPACKAGES_DIR}/picobuild/vendored/tomli
+}
+
+RDEPENDS:${PN}:append:class-target = " python3-tomli python3-pep517 python3-packaging python3-pyparsing"
+
+BBCLASSEXTEND = "native nativesdk"
-- 
2.25.1

[RFC PATCH 08/10] python_pep517: use picobuild instead of manually calling the API

[Ross Burton]

Calling the PEP-517 API directly mostly works, but sometimes doesn't.
For example we don't verify build dependencies, which led to the cbor2
ugprade silently failing to actually package anything.

The standard frontend is pypa/build, but for source-based distributions
that can be annoying to build as it depends on the following packages:

- tomli
- pep517
- packaging
- pyparsing

Manually bootstrapping those recipes is possible, but tedious.

Picobuild is another frontend (written by myself) which is designed
explicitly to be used programatically by source-based distributions: it
doesn't support builds inside virtual environments as we're building
distribution packages, and it vendors the dependencies for bootstrapping
if they're not available.

Over time more packages are expected to move to using Flit to build
which makes the bootstrapping process slightly easier, and tomli will be
integrated into Python 3.11, so it's possible that in the future we
drop picobuild and switch to build.

This change means the PEP517_BUILD_API variable is obsolete, so remove
it.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes/python_pep517.bbclass                      | 8 ++------
 meta/recipes-devtools/python/python3-flit-core_3.7.1.bb | 2 +-
 meta/recipes-devtools/python/python3-installer_0.5.1.bb | 8 ++++++--
 meta/recipes-devtools/python/python3-picobuild_0.1.bb   | 2 ++
 4 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/meta/classes/python_pep517.bbclass b/meta/classes/python_pep517.bbclass
index 34ffdc9c0db..9c37db6be60 100644
--- a/meta/classes/python_pep517.bbclass
+++ b/meta/classes/python_pep517.bbclass
@@ -4,14 +4,11 @@
 # This class will build a wheel in do_compile, and use pypa/installer to install
 # it in do_install.
 
-DEPENDS:append = " python3-installer-native"
+DEPENDS:append = " python3-picobuild-native python3-installer-native"
 
 # Where to execute the build process from
 PEP517_SOURCE_PATH ?= "${S}"
 
-# The PEP517 build API entry point
-PEP517_BUILD_API ?= "unset"
-
 # The directory where wheels will be written
 PEP517_WHEEL_PATH ?= "${WORKDIR}/dist"
 
@@ -31,8 +28,7 @@ python_pep517_do_configure () {
 # When we have Python 3.11 we can parse pyproject.toml to determine the build
 # API entry point directly
 python_pep517_do_compile () {
-    cd ${PEP517_SOURCE_PATH}
-    nativepython3 -c "import ${PEP517_BUILD_API} as api; api.build_wheel('${PEP517_WHEEL_PATH}')"
+    nativepython3 -m picobuild --wheel --source ${PEP517_SOURCE_PATH} --dest ${PEP517_WHEEL_PATH}
 }
 do_compile[cleandirs] += "${PEP517_WHEEL_PATH}"
 
diff --git a/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb b/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb
index abe620374c0..fe6b16ac540 100644
--- a/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb
+++ b/meta/recipes-devtools/python/python3-flit-core_3.7.1.bb
@@ -13,7 +13,7 @@ SRC_URI[sha256sum] = "3c9bd9c140515bfe62dd938c6610d10d6efb9e35cc647fc614fe5fb3a5
 inherit pypi python_flit_core
 
 # Need to install by hand as there's a dependency loop
-DEPENDS:remove:class-native = " python3-installer-native"
+DEPENDS:remove:class-native = " python3-picobuild-native python3-installer-native"
 DEPENDS:append:class-native = " unzip-native"
 
 # We need the full flit tarball
diff --git a/meta/recipes-devtools/python/python3-installer_0.5.1.bb b/meta/recipes-devtools/python/python3-installer_0.5.1.bb
index b7780702d6b..07bbafa81fb 100644
--- a/meta/recipes-devtools/python/python3-installer_0.5.1.bb
+++ b/meta/recipes-devtools/python/python3-installer_0.5.1.bb
@@ -12,11 +12,15 @@ SRC_URI[sha256sum] = "f970995ec2bb815e2fdaf7977b26b2091e1e386f0f42eafd5ac811953d
 
 inherit pypi python_flit_core
 
-# Bootstrap the native install by installing with ourself
-DEPENDS:remove:class-native = "python3-installer-native"
+# Bootstrap the native build
+DEPENDS:remove:class-native = "python3-picobuild-native python3-installer-native"
 
 INSTALL_WHEEL_COMPILE_BYTECODE:class-native = "--no-compile-bytecode"
 
+do_compile:class-native () {
+    python_flit_core_do_manual_build
+}
+
 do_install:prepend:class-native() {
     export PYTHONPATH="${S}/src"
 }
diff --git a/meta/recipes-devtools/python/python3-picobuild_0.1.bb b/meta/recipes-devtools/python/python3-picobuild_0.1.bb
index bbd3636eea8..49aebf81032 100644
--- a/meta/recipes-devtools/python/python3-picobuild_0.1.bb
+++ b/meta/recipes-devtools/python/python3-picobuild_0.1.bb
@@ -11,6 +11,8 @@ S = "${WORKDIR}/git"
 
 inherit python_flit_core
 
+DEPENDS:remove:class-native = "python3-picobuild-native"
+
 # For target builds we can deleted the vendored libraries and RDEPEND on them
 # instead. Use an explicit list to be sure we delete what we RDEPEND on.
 do_install:append:class-target() {
-- 
2.25.1

[RFC PATCH 09/10] classes: remove obsolete PEP517_BUILD_API

[Ross Burton]

This variable is no longer used, so remove it from the python_* classes.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/classes/python_flit_core.bbclass             | 2 --
 meta/classes/python_poetry_core.bbclass           | 2 --
 meta/classes/python_setuptools_build_meta.bbclass | 2 --
 3 files changed, 6 deletions(-)

diff --git a/meta/classes/python_flit_core.bbclass b/meta/classes/python_flit_core.bbclass
index a900f7b5579..7109307de5b 100644
--- a/meta/classes/python_flit_core.bbclass
+++ b/meta/classes/python_flit_core.bbclass
@@ -2,8 +2,6 @@ inherit python_pep517 python3native python3-dir setuptools3-base
 
 DEPENDS += "python3 python3-flit-core-native"
 
-PEP517_BUILD_API = "flit_core.buildapi"
-
 python_flit_core_do_manual_build () {
     cd ${PEP517_SOURCE_PATH}
     nativepython3 -m flit_core.wheel --outdir ${PEP517_WHEEL_PATH} .
diff --git a/meta/classes/python_poetry_core.bbclass b/meta/classes/python_poetry_core.bbclass
index 577663b8f17..0aaf66b1944 100644
--- a/meta/classes/python_poetry_core.bbclass
+++ b/meta/classes/python_poetry_core.bbclass
@@ -1,5 +1,3 @@
 inherit python_pep517 python3native setuptools3-base
 
 DEPENDS += "python3-poetry-core-native"
-
-PEP517_BUILD_API = "poetry.core.masonry.api"
diff --git a/meta/classes/python_setuptools_build_meta.bbclass b/meta/classes/python_setuptools_build_meta.bbclass
index b2bba35a0bc..974054fe5ac 100644
--- a/meta/classes/python_setuptools_build_meta.bbclass
+++ b/meta/classes/python_setuptools_build_meta.bbclass
@@ -1,5 +1,3 @@
 inherit setuptools3-base python_pep517
 
 DEPENDS += "python3-setuptools-native python3-wheel-native"
-
-PEP517_BUILD_API = "setuptools.build_meta"
-- 
2.25.1

[RFC PATCH 10/10] documentation: remove obsolete PEP517_BUILD_API

[Ross Burton]

This variable has been obsoleted, so remove it from the documentation.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 documentation/ref-manual/classes.rst   | 4 +---
 documentation/ref-manual/variables.rst | 5 -----
 2 files changed, 1 insertion(+), 8 deletions(-)

diff --git a/documentation/ref-manual/classes.rst b/documentation/ref-manual/classes.rst
index 729aa259e02..e170d31cd9d 100644
--- a/documentation/ref-manual/classes.rst
+++ b/documentation/ref-manual/classes.rst
@@ -2000,9 +2000,7 @@ The ``python_pep517`` class builds and installs a Python ``wheel`` binary
 archive (see `PEP-517 <https://peps.python.org/pep-0517/>`__).
 
 Recipes wouldn't inherit this directly, instead typically another class will
-inherit this, add the relevant native dependencies, and set
-:term:`PEP517_BUILD_API` to the Python class which implements the PEP-517 build
-API.
+inherit this and add the relevant native dependencies.
 
 Examples of classes which do this are :ref:`python_flit_core
 <ref-classes-python_flit_core>`, :ref:`python_setuptools_build_meta
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 367b4674e2c..da9b8577842 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -5657,11 +5657,6 @@ system and gives an overview of their function and contents.
 
       :term:`PE` is the default value of the :term:`PKGE` variable.
 
-   :term:`PEP517_BUILD_API`
-      When used by recipes that inherit the :ref:`python_pep517
-      <ref-classes-python_pep517>` class, specifies the entry point to the
-      PEP-517 compliant build API (such as ``flit_core.buildapi``).
-
    :term:`PEP517_WHEEL_PATH`
       When used by recipes that inherit the
       :ref:`python_pep517 <ref-classes-python_pep517>` class,
-- 
2.25.1