Re: [Buildroot] [PATCH v7 1/5] package/pkg-python: migrate flit to new bootstrapping sequence

[Arnout Vandecappelle <arnout@mind.be>]

On 24/04/2022 23:39, James Hilliard wrote:
> There are a number of flit toolchain dependencies currently in the
> process of deprecating distutils based fallbacks.
> 
> This will be needed in order to update tomli.
> 
> We need to migrate these to use a new bootstrap based build+install
> sequence which relies on flit's bootstrap wheel build+install
> features to build and install host-python-pypa-build and
> host-python-installer which gives us a full pep517 toolchain.
> 
> Note that one can run host-python-flit-core commands for building
> and installing itself since the package build directory is the cwd.
> 
> We need to add a special flit-bootstrap SETUP_TYPE for dependencies
> of host-python-pypa-build and host-python-installer which can not
> use the normal flit SETUP_TYPE which would cause a circular dependency
> issue.
> 
> We need to special case dependency exclusions for
> host-python-flit-core and host-python-installer to avoid circular
> dependencies in the flit-bootstrap SETUP_TYPE.
> 
> We also need to special case the installation command for
> host-python-flit-core since it can not depend on host-python-installer
> due to host-python-installer requiring host-python-flit-core.
> 
> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> ---
> Changes v3 -> v4:
>    - rebase
> Changes v2 -> v3:
>    - add special flit-bootstrap SETUP_TYPE
>    - don't change package SETUP_TYPE's yet
> Changes v1 -> v2:
>    - formatting/cleanup
>    - add comments
> ---
>   package/pkg-python.mk | 26 +++++++++++++++++++++++++-
>   1 file changed, 25 insertions(+), 1 deletion(-)
> 
> diff --git a/package/pkg-python.mk b/package/pkg-python.mk
> index 867341fc7b..2e7704a0a9 100644
> --- a/package/pkg-python.mk
> +++ b/package/pkg-python.mk
> @@ -154,6 +154,9 @@ HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \
>   	--scripts=$(HOST_DIR)/bin \
>   	--data=$(HOST_DIR)
>   
> +HOST_PKG_PYTHON_PEP517_BOOTSTRAP_INSTALL_OPTS = \
> +	--installdir=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages
> +
>   ################################################################################
>   # inner-python-package -- defines how the configuration, compilation
>   # and installation of a Python package should be done, implements a
> @@ -203,7 +206,7 @@ $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_SETUPTOOLS_ENV)
>   $(2)_BASE_BUILD_CMD = setup.py build
>   $(2)_BASE_INSTALL_CMD = setup.py install $$(HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS)
>   endif
> -else ifneq ($$(filter flit pep517,$$($(2)_SETUP_TYPE)),)
> +else ifneq ($$(filter flit flit-bootstrap pep517,$$($(2)_SETUP_TYPE)),)
>   ifeq ($(4),target)
>   $(2)_BASE_ENV = $$(PKG_PYTHON_PEP517_ENV)
>   $(2)_BASE_BUILD_CMD = -m build -n -w
> @@ -211,9 +214,24 @@ $(2)_BASE_INSTALL_TARGET_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $
>   $(2)_BASE_INSTALL_STAGING_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS)
>   else
>   $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV)
> +# Use flit built in wheel builder for packages that are flit-bootstrap packages.
> +# This is needed to avoid a circular with host-python-pypa-build and those dependencies.
> +#
> +ifeq ($$($(2)_SETUP_TYPE),flit-bootstrap)
> +$(2)_BASE_BUILD_CMD = -m flit_core.wheel
> +ifeq ($(1),host-python-flit-core)
> +# Use flit built in bootstrap_install for installing host-python-flit-core.
> +# This is due to host-python-installer depending on host-python-flit-core.
> +#
> +$(2)_BASE_INSTALL_CMD = -m bootstrap_install dist/* $$(HOST_PKG_PYTHON_PEP517_BOOTSTRAP_INSTALL_OPTS)
> +else
> +$(2)_BASE_INSTALL_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS)
> +endif

  I still think this is too complicated. And I'm stubborn, so I changed it and 
committed to master [1]. I've basically done two things:

- Treat flit-bootstrap as a completely separate setup type. Only the _BASE_ENV 
is exactly the same as in the other cases; the conditional flow becomes a lot 
easier to understand by making it explicit. This also allows an $(error ...) 
when it's used for a target package.

  The special-casing for flit-core I solved by using ?= and overriding it in 
python-flit-core.mk

> +else
>   $(2)_BASE_BUILD_CMD = -m build -n -w
>   $(2)_BASE_INSTALL_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS)
>   endif
> +endif
>   else
>   $$(error "Invalid $(2)_SETUP_TYPE. Valid options are 'distutils', 'setuptools', 'pep517' or 'flit'.")
>   endif
> @@ -239,6 +257,12 @@ $(2)_DEPENDENCIES += host-python-pypa-build host-python-installer
>   ifeq ($$($(2)_SETUP_TYPE),flit)
>   $(2)_DEPENDENCIES += host-python-flit-core
>   endif
> +else ifeq ($$($(2)_SETUP_TYPE),flit-bootstrap)
> +ifneq ($$(filter host-python-flit-core host-python-installer,$(1)),)
> +$(2)_DEPENDENCIES += $$(if $$(filter host-python-flit-core,$(1)),,host-python-flit-core)

  Again, too complicated for me. I just removed this conditional branch. So for 
host-python-flit-core and host-python-installer, no dependencies are added. 
host-python-flit-core in fact doesn't need any. host-python-installer has to add 
it manually.

  Regards,
  Arnout

[1] 
https://git.buildroot.org/buildroot/commit/?id=8b4f831f36cd653504f38023f909f76912730534


> +else
> +$(2)_DEPENDENCIES += host-python-flit-core host-python-installer
> +endif
>   endif # SETUP_TYPE
>   
>   # Python interpreter to use for building the package.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot