[Buildroot] [PATCH 0/3] Prepare Python support for top-level parallel build

[Buildroot] [PATCH 0/3] Prepare Python support for top-level parallel build

[Thomas Petazzoni]

Hello,

This series prepares our Python support for top-level parallel
build. To understand it, you need to read it from the last patch to
the first patch, because PATCH 1 is a consequence of PATCH 2, which is
itself a consequence of PATCH 3.

So, the problem started with the fact that when we install host Python
modules based on setuptools, we use the "regular" Python installation
process based on Python Eggs. This installation process has the
not-so-nice property of registering all installed Python Eggs in a
single common file:
$(HOST_DIR)/lib/python2.7/site-packages/easy-install.pth. Clearly,
this doesn't work well with per-package directory: if you have package
A that depends on host-python-B and host-python-C, with no dependency
relationship between host-python-B and host-python-C, when the
per-package directory for "A" gets created, it gets a version of the
easy-install.pth file either from host-python-B or host-python-C... so
it will not "see" one of them.

To fix this, we want to use the same installation process we use for
host Python modules than the one we use for target Python modules:
avoid Python Eggs, using the --single-version-externally-managed
setuptools option. This is the purpose of PATCH 3/3, which is simple
enough.

However, the story started becoming more complicated when we tried to
build host-meson. Indeed, host-meson needs host-python3 and its
installation process is based on setuptools. The issue is that when
Python 2.x is enabled for the target, or when no Python at all is
selected for the target, the default host Python version is 2.x, and
therefore host-python-setuptools is installed for Python 2.x, not
Python 3.x. This makes python-setuptools not usable for the
installation of host-meson. This was not visible until now, because
host-meson "transparently" falls back to distutils for its
installation process. *But* once you pass
--single-version-externally-managed, you have a problem: this option
is only supported by setuptools, not by distutils, so PATCH 3/3 breaks
the build of host-meson.

To solve this, we do something that we have been discussing for quite
a while: make sure python-setuptools can be available for host Python
3.x, even if Python 3.x is not the default host Python version. To
achieve this:

 - PATCH 1/3 adds a host-python3-setuptools package.

 - PATCH 2/3 changes the Python package infrastructure to use
   host-python3-setuptools or host-python-setuptools depending on the
   situation. host-python-setuptools is changed to always install for
   Python 2.x, regardless of what is the default host Python version
   so that host-python-setuptools and host-python3-setuptools never
   step on each other.

Hopefully this cover letter helps explain the overall situation
clearly enough. Let me know if additional details are needed. This
code has survived some extensive autobuilder testing, hopefully I
haven't missed any special case.

Best regards,

Thomas

Thomas Petazzoni (3):
  package/python3-setuptools: new package
  package/pkg-python: use host-python3-setuptools when needed
  package/pkg-python: use --single-version-externally-managed for host
    setuptools

 package/lirc-tools/lirc-tools.mk              |  2 +-
 package/pkg-python.mk                         | 39 +++++++---
 .../python-setuptools/python-setuptools.mk    |  1 +
 .../0001-add-executable.patch                 | 72 +++++++++++++++++++
 .../python3-setuptools.hash                   |  1 +
 .../python3-setuptools/python3-setuptools.mk  | 24 +++++++
 6 files changed, 129 insertions(+), 10 deletions(-)
 create mode 100644 package/python3-setuptools/0001-add-executable.patch
 create mode 120000 package/python3-setuptools/python3-setuptools.hash
 create mode 100644 package/python3-setuptools/python3-setuptools.mk

-- 
2.20.1

[Buildroot] [PATCH 1/3] package/python3-setuptools: new package

[Thomas Petazzoni]

This new package allows to install the host version of setuptools for
python3, regardless of what the default Python version is. It will be
used by the python-package infrastructure to make sure that host
Python packages that need python3 and setuptools have setuptools
installed for Python 3.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 .../0001-add-executable.patch                 | 72 +++++++++++++++++++
 .../python3-setuptools.hash                   |  1 +
 .../python3-setuptools/python3-setuptools.mk  | 24 +++++++
 3 files changed, 97 insertions(+)
 create mode 100644 package/python3-setuptools/0001-add-executable.patch
 create mode 120000 package/python3-setuptools/python3-setuptools.hash
 create mode 100644 package/python3-setuptools/python3-setuptools.mk

diff --git a/package/python3-setuptools/0001-add-executable.patch b/package/python3-setuptools/0001-add-executable.patch
new file mode 100644
index 0000000000..1f8714eaeb
--- /dev/null
+++ b/package/python3-setuptools/0001-add-executable.patch
@@ -0,0 +1,72 @@
+From 9b3d307f8f6a1af88f3f810f5a6cf0835830e1e8 Mon Sep 17 00:00:00 2001
+From: =?UTF-8?q?J=C3=B6rg=20Krause?= <joerg.krause@embedded.rocks>
+Date: Mon, 7 Dec 2015 01:14:33 +0100
+Subject: [PATCH 1/1] add executable
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Add a new --executable option to distribute so that we can
+force the shebang line in installed python scripts.
+
+[Thomas: refresh for setuptools 5.8.]
+[J?rg: refresh for setuptools 18.7.1]
+
+Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
+Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+Signed-off-by: J?rg Krause <joerg.krause@embedded.rocks>
+---
+ setuptools/command/install.py         | 2 ++
+ setuptools/command/install_scripts.py | 9 +++++++++
+ 2 files changed, 11 insertions(+)
+
+diff --git a/setuptools/command/install.py b/setuptools/command/install.py
+index d2bca2e..b60ef19 100644
+--- a/setuptools/command/install.py
++++ b/setuptools/command/install.py
+@@ -16,6 +16,7 @@ class install(orig.install):
+     """Use easy_install to install the package, w/dependencies"""
+ 
+     user_options = orig.install.user_options + [
++        ('executable=', 'e', "specify final destination interpreter path"),
+         ('old-and-unmanageable', None, "Try not to use this!"),
+         ('single-version-externally-managed', None,
+          "used by system package builders to create 'flat' eggs"),
+@@ -31,6 +32,7 @@ class install(orig.install):
+ 
+     def initialize_options(self):
+         orig.install.initialize_options(self)
++        self.executable = None
+         self.old_and_unmanageable = None
+         self.single_version_externally_managed = None
+ 
+diff --git a/setuptools/command/install_scripts.py b/setuptools/command/install_scripts.py
+index be66cb2..07afaf8 100755
+--- a/setuptools/command/install_scripts.py
++++ b/setuptools/command/install_scripts.py
+@@ -11,6 +11,13 @@ class install_scripts(orig.install_scripts):
+     def initialize_options(self):
+         orig.install_scripts.initialize_options(self)
+         self.no_ep = False
++        self.executable = None
++
++    def finalize_options(self):
++        orig.install_scripts.finalize_options(self)
++        self.set_undefined_options('install',
++                ('executable','executable')
++        )
+ 
+     def run(self):
+         import setuptools.command.easy_install as ei
+@@ -31,6 +38,8 @@ class install_scripts(orig.install_scripts):
+         )
+         bs_cmd = self.get_finalized_command('build_scripts')
+         exec_param = getattr(bs_cmd, 'executable', None)
++        if self.executable is not None:
++            exec_param = self.executable
+         bw_cmd = self.get_finalized_command("bdist_wininst")
+         is_wininst = getattr(bw_cmd, '_is_running', False)
+         writer = ei.ScriptWriter
+-- 
+2.6.3
+
diff --git a/package/python3-setuptools/python3-setuptools.hash b/package/python3-setuptools/python3-setuptools.hash
new file mode 120000
index 0000000000..a3af7f0fb1
--- /dev/null
+++ b/package/python3-setuptools/python3-setuptools.hash
@@ -0,0 +1 @@
+../python-setuptools/python-setuptools.hash
\ No newline at end of file
diff --git a/package/python3-setuptools/python3-setuptools.mk b/package/python3-setuptools/python3-setuptools.mk
new file mode 100644
index 0000000000..b6de5d4b25
--- /dev/null
+++ b/package/python3-setuptools/python3-setuptools.mk
@@ -0,0 +1,24 @@
+################################################################################
+#
+# python3-setuptools
+#
+################################################################################
+
+# Please keep in sync with
+# package/python-setuptools/python-setuptools.mk
+PYTHON3_SETUPTOOLS_VERSION = 40.6.3
+PYTHON3_SETUPTOOLS_SOURCE = setuptools-$(PYTHON3_SETUPTOOLS_VERSION).zip
+PYTHON3_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861991beeade31473868463dad0e58b1978c209de27384ae541b0b
+PYTHON3_SETUPTOOLS_LICENSE = MIT
+PYTHON3_SETUPTOOLS_LICENSE_FILES = LICENSE
+PYTHON3_SETUPTOOLS_SETUP_TYPE = setuptools
+HOST_PYTHON3_SETUPTOOLS_DL_SUBDIR = python-setuptools
+HOST_PYTHON3_SETUPTOOLS_NEEDS_HOST_PYTHON = python3
+
+define HOST_PYTHON3_SETUPTOOLS_EXTRACT_CMDS
+	$(UNZIP) -d $(@D) $(HOST_PYTHON3_SETUPTOOLS_DL_DIR)/$(PYTHON3_SETUPTOOLS_SOURCE)
+	mv $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION)/* $(@D)
+	$(RM) -r $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION)
+endef
+
+$(eval $(host-python-package))
-- 
2.20.1

[Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed

[Thomas Petazzoni]

When a package uses "setuptools" as its <pkg>_SETUP_TYPE, we currently
add a dependency on host-python-setuptools. This means that:

 (1) When BR2_PACKAGE_PYTHON=y, the default host Python version is
     Python 2.x, and host-python-setuptools is installed for
     host-python.

 (2) When BR2_PACKAGE_PYTHON3=y, the default host Python version is
     Python 3.x, and host-python-setuptools is installed for
     host-python3.

 (3) When no target Python interpreter is selected, the default host
     Python version is Python 2.x, and host-python-setuptools is
     installed for host-python.

Situations (1) and (3) are problematic for host Python packages that
need Python 3.x. Such packages use <pkg>_NEEDS_HOST_PYTHON = python3,
but if they use setuptools as their setup type, they will not find
setuptools installed for host-python3 in situations (1) and (3)
described above.

We currently have a single package that sets <pkg>_NEEDS_HOST_PYTHON =
python3: host-meson. host-meson generally works because if setuptools
is not found, it falls back to distutils, which is part of the
standard Python library. However, if there is a setuptools version
installed system-wide, it may be picked up, but may not necessarily be
the same version as Buildroot setuptools, potentially causing
problems.

This commit makes the necessary change to the python-package
infrastructure to fix this behavior, by identifying the following
cases:

 - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python3,
   then we know it wants setuptools installed for host-python3, so we
   use host-python3-setuptools.

 - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python2,
   then we known it wants setuptools installed for host-python, so we
   use host-python-setuptools.

 - When BR2_PACKAGE_PYTHON3=y, and we have a target package, or a host
   package with no NEEDS_HOST_PYTHON option, then we want setuptools
   installed for host-python3, so we use host-python3-setuptools.

 - When BR2_PACKAGE_PYTHON=y or no target interpreter is enabled at
   all, and we have a target package, or a host package with no
   NEEDS_HOST_PYTHON option, then we want setuptools for host-python,
   so we use host-python-setuptools.

To make this happen, we use host-python3-setuptools introduced in a
previous commit, but we also change host-python-setuptools to force
its installation for host-python. The latter is needed if you build
with BR2_PACKAGE_PYTHON3=y but want to install a Python-based package
that has NEEDS_HOST_PYTHON=python2.

There is one single package that needs be adjusted following this:
lirc-tools, because it is not using the python-package
infrastructure. It directly depends on host-python-setuptools, which
no longer works because host-python-setuptools now only installs for
Python 2.x, while lirc-tools Python binding only supports Python
3.x. Switching to host-python3-setuptools solves this problem.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/lirc-tools/lirc-tools.mk              |  2 +-
 package/pkg-python.mk                         | 35 ++++++++++++++-----
 .../python-setuptools/python-setuptools.mk    |  1 +
 3 files changed, 29 insertions(+), 9 deletions(-)

diff --git a/package/lirc-tools/lirc-tools.mk b/package/lirc-tools/lirc-tools.mk
index a5165ad438..e468f2a25d 100644
--- a/package/lirc-tools/lirc-tools.mk
+++ b/package/lirc-tools/lirc-tools.mk
@@ -46,7 +46,7 @@ LIRC_TOOLS_DEPENDENCIES += libftdi1
 endif
 
 ifeq ($(BR2_PACKAGE_PYTHON3),y)
-LIRC_TOOLS_DEPENDENCIES += python3 host-python-setuptools
+LIRC_TOOLS_DEPENDENCIES += python3 host-python3-setuptools
 LIRC_TOOLS_MAKE_ENV += SETUPTOOLS_ENV="$(PKG_PYTHON_SETUPTOOLS_ENV)"
 endif
 
diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index 3dc8a5d10f..b641e3f865 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -188,16 +188,35 @@ endif
 endif # ($$($(2)_NEEDS_HOST_PYTHON),)
 endif # ($(4),target)
 
-# Setuptools based packages will need host-python-setuptools (both
-# host and target). We need to have a special exclusion for the
-# host-setuptools package itself: it is setuptools-based, but
-# shouldn't depend on host-setuptools (because it would otherwise
-# depend on itself!).
+# Setuptools based packages will need setuptools for the host Python
+# interpreter (both host and target).
+#
+# If we have a host package that says "I need Python 3", we install
+# setuptools for python3.
+#
+# If we have a host packge that says "I need Python 2", we install
+# setuptools for python2.
+#
+# If we have a target package, or a host package that doesn't have any
+# <pkg>_NEEDS_HOST_PYTHON, and BR2_PACKAGE_PYTHON3 is used, then
+# Python 3.x is the default Python interpreter, so we install
+# setuptools for python3.
+#
+# In all other cases, we install setuptools for python2. Those other
+# cases are: a target package or host package with
+# BR2_PACKAGE_PYTHON=y, or a host-package with neither
+# BR2_PACKAGE_PYTHON3=y or BR2_PACKAGE_PYTHON=y.
 ifeq ($$($(2)_SETUP_TYPE),setuptools)
-ifneq ($(2),HOST_PYTHON_SETUPTOOLS)
-$(2)_DEPENDENCIES += host-python-setuptools
-endif
+ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python3)
+$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools)
+else ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python2)
+$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools)
+else ifeq ($$(BR2_PACKAGE_PYTHON3),y)
+$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools)
+else
+$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools)
 endif
+endif # SETUP_TYPE
 
 # Python interpreter to use for building the package.
 #
diff --git a/package/python-setuptools/python-setuptools.mk b/package/python-setuptools/python-setuptools.mk
index cc6f991cdd..d915449c5d 100644
--- a/package/python-setuptools/python-setuptools.mk
+++ b/package/python-setuptools/python-setuptools.mk
@@ -10,6 +10,7 @@ PYTHON_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861
 PYTHON_SETUPTOOLS_LICENSE = MIT
 PYTHON_SETUPTOOLS_LICENSE_FILES = LICENSE
 PYTHON_SETUPTOOLS_SETUP_TYPE = setuptools
+HOST_PYTHON_SETUPTOOLS_NEEDS_HOST_PYTHON = python2
 
 define PYTHON_SETUPTOOLS_EXTRACT_CMDS
 	$(UNZIP) -d $(@D) $(PYTHON_SETUPTOOLS_DL_DIR)/$(PYTHON_SETUPTOOLS_SOURCE)
-- 
2.20.1

[Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools

[Thomas Petazzoni]

Currently, host Python packages that use setuptools are installed as
Python Eggs, i.e they are installed in
$(HOST_DIR)/lib/python2.7/site-packages/Something-<version>.egg. Once
installed, each Python Egg is registered to a file called
$(HOST_DIR)/lib/python2.7/site-packages/easy-install.pth. This file is
read by the Python interpreter so that the installation location of
each Egg is added to the Python path, and can be found by the Python
interpreter.

However, the fact that the installation of different Python modules
need to update a common file is clearly not compatible with
per-package directories and top-level parallel build.

To fix this, we avoid using Python Eggs using the same
--single-version-externally-managed option that we use for target
modules. This option is normally meant for distributions packaging
Python modules, and can therefore only be used if either --record (to
record the list of files being installed) or --root is
passed. --root=/ works fine and was suggested by
https://stackoverflow.com/questions/6301003/stopping-setup-py-from-installing-as-egg/33791008#33791008.

With this change, host Python modules installed by setuptools are now
installed in the "regular" way, i.e directly in
$(HOST_DIR)/lib/python2.7/site-packages/mako for host-python-mako.

This makes the installation of host Python modules more similar to the
one of target modules, and makes it compatible with per-package
directory support and top-level parallel build.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 package/pkg-python.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index b641e3f865..4f42e2efc2 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -85,7 +85,9 @@ HOST_PKG_PYTHON_SETUPTOOLS_ENV = \
 	PYTHONNOUSERSITE=1
 
 HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \
-	--prefix=$(HOST_DIR)
+	--prefix=$(HOST_DIR) \
+	--root=/ \
+	--single-version-externally-managed
 
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define PKG_PYTHON_FIXUP_SYSCONFIGDATA
-- 
2.20.1

[Buildroot] [PATCH 1/3] package/python3-setuptools: new package

[Asaf Kahlon]

Hello,

On Fri, Dec 28, 2018 at 7:01 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> This new package allows to install the host version of setuptools for
> python3, regardless of what the default Python version is. It will be
> used by the python-package infrastructure to make sure that host
> Python packages that need python3 and setuptools have setuptools
> installed for Python 3.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  .../0001-add-executable.patch                 | 72 +++++++++++++++++++
>  .../python3-setuptools.hash                   |  1 +
>  .../python3-setuptools/python3-setuptools.mk  | 24 +++++++
>  3 files changed, 97 insertions(+)
>  create mode 100644 package/python3-setuptools/0001-add-executable.patch
>  create mode 120000 package/python3-setuptools/python3-setuptools.hash
>  create mode 100644 package/python3-setuptools/python3-setuptools.mk
>
> diff --git a/package/python3-setuptools/0001-add-executable.patch b/package/python3-setuptools/0001-add-executable.patch
> new file mode 100644
> index 0000000000..1f8714eaeb
> --- /dev/null
> +++ b/package/python3-setuptools/0001-add-executable.patch
> @@ -0,0 +1,72 @@
> +From 9b3d307f8f6a1af88f3f810f5a6cf0835830e1e8 Mon Sep 17 00:00:00 2001
> +From: =?UTF-8?q?J=C3=B6rg=20Krause?= <joerg.krause@embedded.rocks>
> +Date: Mon, 7 Dec 2015 01:14:33 +0100
> +Subject: [PATCH 1/1] add executable
> +MIME-Version: 1.0
> +Content-Type: text/plain; charset=UTF-8
> +Content-Transfer-Encoding: 8bit
> +
> +Add a new --executable option to distribute so that we can
> +force the shebang line in installed python scripts.
> +
> +[Thomas: refresh for setuptools 5.8.]
> +[J?rg: refresh for setuptools 18.7.1]
> +
> +Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
> +Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> +Signed-off-by: J?rg Krause <joerg.krause@embedded.rocks>
> +---
> + setuptools/command/install.py         | 2 ++
> + setuptools/command/install_scripts.py | 9 +++++++++
> + 2 files changed, 11 insertions(+)
> +
> +diff --git a/setuptools/command/install.py b/setuptools/command/install.py
> +index d2bca2e..b60ef19 100644
> +--- a/setuptools/command/install.py
> ++++ b/setuptools/command/install.py
> +@@ -16,6 +16,7 @@ class install(orig.install):
> +     """Use easy_install to install the package, w/dependencies"""
> +
> +     user_options = orig.install.user_options + [
> ++        ('executable=', 'e', "specify final destination interpreter path"),
> +         ('old-and-unmanageable', None, "Try not to use this!"),
> +         ('single-version-externally-managed', None,
> +          "used by system package builders to create 'flat' eggs"),
> +@@ -31,6 +32,7 @@ class install(orig.install):
> +
> +     def initialize_options(self):
> +         orig.install.initialize_options(self)
> ++        self.executable = None
> +         self.old_and_unmanageable = None
> +         self.single_version_externally_managed = None
> +
> +diff --git a/setuptools/command/install_scripts.py b/setuptools/command/install_scripts.py
> +index be66cb2..07afaf8 100755
> +--- a/setuptools/command/install_scripts.py
> ++++ b/setuptools/command/install_scripts.py
> +@@ -11,6 +11,13 @@ class install_scripts(orig.install_scripts):
> +     def initialize_options(self):
> +         orig.install_scripts.initialize_options(self)
> +         self.no_ep = False
> ++        self.executable = None
> ++
> ++    def finalize_options(self):
> ++        orig.install_scripts.finalize_options(self)
> ++        self.set_undefined_options('install',
> ++                ('executable','executable')
> ++        )
> +
> +     def run(self):
> +         import setuptools.command.easy_install as ei
> +@@ -31,6 +38,8 @@ class install_scripts(orig.install_scripts):
> +         )
> +         bs_cmd = self.get_finalized_command('build_scripts')
> +         exec_param = getattr(bs_cmd, 'executable', None)
> ++        if self.executable is not None:
> ++            exec_param = self.executable
> +         bw_cmd = self.get_finalized_command("bdist_wininst")
> +         is_wininst = getattr(bw_cmd, '_is_running', False)
> +         writer = ei.ScriptWriter
> +--
> +2.6.3
> +
Is there any reason to copy the exact patch file from
python-setuptools, instead of creating a link (like you did for the
hash file)?
I don't think the patch will be different between host and target
compilations...

> diff --git a/package/python3-setuptools/python3-setuptools.hash b/package/python3-setuptools/python3-setuptools.hash
> new file mode 120000
> index 0000000000..a3af7f0fb1
> --- /dev/null
> +++ b/package/python3-setuptools/python3-setuptools.hash
> @@ -0,0 +1 @@
> +../python-setuptools/python-setuptools.hash
> \ No newline at end of file
> diff --git a/package/python3-setuptools/python3-setuptools.mk b/package/python3-setuptools/python3-setuptools.mk
> new file mode 100644
> index 0000000000..b6de5d4b25
> --- /dev/null
> +++ b/package/python3-setuptools/python3-setuptools.mk
> @@ -0,0 +1,24 @@
> +################################################################################
> +#
> +# python3-setuptools
> +#
> +################################################################################
> +
> +# Please keep in sync with
> +# package/python-setuptools/python-setuptools.mk
> +PYTHON3_SETUPTOOLS_VERSION = 40.6.3
> +PYTHON3_SETUPTOOLS_SOURCE = setuptools-$(PYTHON3_SETUPTOOLS_VERSION).zip
> +PYTHON3_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861991beeade31473868463dad0e58b1978c209de27384ae541b0b
> +PYTHON3_SETUPTOOLS_LICENSE = MIT
> +PYTHON3_SETUPTOOLS_LICENSE_FILES = LICENSE
> +PYTHON3_SETUPTOOLS_SETUP_TYPE = setuptools
> +HOST_PYTHON3_SETUPTOOLS_DL_SUBDIR = python-setuptools
> +HOST_PYTHON3_SETUPTOOLS_NEEDS_HOST_PYTHON = python3
> +
> +define HOST_PYTHON3_SETUPTOOLS_EXTRACT_CMDS
> +       $(UNZIP) -d $(@D) $(HOST_PYTHON3_SETUPTOOLS_DL_DIR)/$(PYTHON3_SETUPTOOLS_SOURCE)
> +       mv $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION)/* $(@D)
> +       $(RM) -r $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION)
> +endef
> +
> +$(eval $(host-python-package))
> --
> 2.20.1
>

Regards,
Asaf.

[Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed

[Asaf Kahlon]

Hello,

On Fri, Dec 28, 2018 at 7:01 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> When a package uses "setuptools" as its <pkg>_SETUP_TYPE, we currently
> add a dependency on host-python-setuptools. This means that:
>
>  (1) When BR2_PACKAGE_PYTHON=y, the default host Python version is
>      Python 2.x, and host-python-setuptools is installed for
>      host-python.
>
>  (2) When BR2_PACKAGE_PYTHON3=y, the default host Python version is
>      Python 3.x, and host-python-setuptools is installed for
>      host-python3.
>
>  (3) When no target Python interpreter is selected, the default host
>      Python version is Python 2.x, and host-python-setuptools is
>      installed for host-python.
>
> Situations (1) and (3) are problematic for host Python packages that
> need Python 3.x. Such packages use <pkg>_NEEDS_HOST_PYTHON = python3,
> but if they use setuptools as their setup type, they will not find
> setuptools installed for host-python3 in situations (1) and (3)
> described above.
>
> We currently have a single package that sets <pkg>_NEEDS_HOST_PYTHON =
> python3: host-meson. host-meson generally works because if setuptools
> is not found, it falls back to distutils, which is part of the
> standard Python library. However, if there is a setuptools version
> installed system-wide, it may be picked up, but may not necessarily be
> the same version as Buildroot setuptools, potentially causing
> problems.
>
> This commit makes the necessary change to the python-package
> infrastructure to fix this behavior, by identifying the following
> cases:
>
>  - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python3,
>    then we know it wants setuptools installed for host-python3, so we
>    use host-python3-setuptools.
>
>  - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python2,
>    then we known it wants setuptools installed for host-python, so we
>    use host-python-setuptools.
>
>  - When BR2_PACKAGE_PYTHON3=y, and we have a target package, or a host
>    package with no NEEDS_HOST_PYTHON option, then we want setuptools
>    installed for host-python3, so we use host-python3-setuptools.
>
>  - When BR2_PACKAGE_PYTHON=y or no target interpreter is enabled at
>    all, and we have a target package, or a host package with no
>    NEEDS_HOST_PYTHON option, then we want setuptools for host-python,
>    so we use host-python-setuptools.
>
> To make this happen, we use host-python3-setuptools introduced in a
> previous commit, but we also change host-python-setuptools to force
> its installation for host-python. The latter is needed if you build
> with BR2_PACKAGE_PYTHON3=y but want to install a Python-based package
> that has NEEDS_HOST_PYTHON=python2.
>
> There is one single package that needs be adjusted following this:
> lirc-tools, because it is not using the python-package
> infrastructure. It directly depends on host-python-setuptools, which
> no longer works because host-python-setuptools now only installs for
> Python 2.x, while lirc-tools Python binding only supports Python
> 3.x. Switching to host-python3-setuptools solves this problem.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/lirc-tools/lirc-tools.mk              |  2 +-
>  package/pkg-python.mk                         | 35 ++++++++++++++-----
>  .../python-setuptools/python-setuptools.mk    |  1 +
>  3 files changed, 29 insertions(+), 9 deletions(-)
>
> diff --git a/package/lirc-tools/lirc-tools.mk b/package/lirc-tools/lirc-tools.mk
> index a5165ad438..e468f2a25d 100644
> --- a/package/lirc-tools/lirc-tools.mk
> +++ b/package/lirc-tools/lirc-tools.mk
> @@ -46,7 +46,7 @@ LIRC_TOOLS_DEPENDENCIES += libftdi1
>  endif
>
>  ifeq ($(BR2_PACKAGE_PYTHON3),y)
> -LIRC_TOOLS_DEPENDENCIES += python3 host-python-setuptools
> +LIRC_TOOLS_DEPENDENCIES += python3 host-python3-setuptools
>  LIRC_TOOLS_MAKE_ENV += SETUPTOOLS_ENV="$(PKG_PYTHON_SETUPTOOLS_ENV)"
>  endif
>
> diff --git a/package/pkg-python.mk b/package/pkg-python.mk
> index 3dc8a5d10f..b641e3f865 100644
> --- a/package/pkg-python.mk
> +++ b/package/pkg-python.mk
> @@ -188,16 +188,35 @@ endif
>  endif # ($$($(2)_NEEDS_HOST_PYTHON),)
>  endif # ($(4),target)
>
> -# Setuptools based packages will need host-python-setuptools (both
> -# host and target). We need to have a special exclusion for the
> -# host-setuptools package itself: it is setuptools-based, but
> -# shouldn't depend on host-setuptools (because it would otherwise
> -# depend on itself!).
> +# Setuptools based packages will need setuptools for the host Python
> +# interpreter (both host and target).
> +#
> +# If we have a host package that says "I need Python 3", we install
> +# setuptools for python3.
> +#
> +# If we have a host packge that says "I need Python 2", we install
> +# setuptools for python2.
> +#
> +# If we have a target package, or a host package that doesn't have any
> +# <pkg>_NEEDS_HOST_PYTHON, and BR2_PACKAGE_PYTHON3 is used, then
> +# Python 3.x is the default Python interpreter, so we install
> +# setuptools for python3.
> +#
> +# In all other cases, we install setuptools for python2. Those other
> +# cases are: a target package or host package with
> +# BR2_PACKAGE_PYTHON=y, or a host-package with neither
> +# BR2_PACKAGE_PYTHON3=y or BR2_PACKAGE_PYTHON=y.
>  ifeq ($$($(2)_SETUP_TYPE),setuptools)
> -ifneq ($(2),HOST_PYTHON_SETUPTOOLS)
> -$(2)_DEPENDENCIES += host-python-setuptools
> -endif
> +ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python3)
> +$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools)
> +else ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python2)
> +$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools)
> +else ifeq ($$(BR2_PACKAGE_PYTHON3),y)
> +$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools)
> +else
> +$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools)
>  endif
> +endif # SETUP_TYPE
>
>  # Python interpreter to use for building the package.
>  #
> diff --git a/package/python-setuptools/python-setuptools.mk b/package/python-setuptools/python-setuptools.mk
> index cc6f991cdd..d915449c5d 100644
> --- a/package/python-setuptools/python-setuptools.mk
> +++ b/package/python-setuptools/python-setuptools.mk
> @@ -10,6 +10,7 @@ PYTHON_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861
>  PYTHON_SETUPTOOLS_LICENSE = MIT
>  PYTHON_SETUPTOOLS_LICENSE_FILES = LICENSE
>  PYTHON_SETUPTOOLS_SETUP_TYPE = setuptools
> +HOST_PYTHON_SETUPTOOLS_NEEDS_HOST_PYTHON = python2
>
>  define PYTHON_SETUPTOOLS_EXTRACT_CMDS
>         $(UNZIP) -d $(@D) $(PYTHON_SETUPTOOLS_DL_DIR)/$(PYTHON_SETUPTOOLS_SOURCE)
> --
> 2.20.1
>

Reviewed-by: Asaf Kahlon <asafka7@gmail.com>

[Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools

[Asaf Kahlon]

Hello,

On Fri, Dec 28, 2018 at 7:01 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Currently, host Python packages that use setuptools are installed as
> Python Eggs, i.e they are installed in
> $(HOST_DIR)/lib/python2.7/site-packages/Something-<version>.egg. Once
> installed, each Python Egg is registered to a file called
> $(HOST_DIR)/lib/python2.7/site-packages/easy-install.pth. This file is
> read by the Python interpreter so that the installation location of
> each Egg is added to the Python path, and can be found by the Python
> interpreter.
>
> However, the fact that the installation of different Python modules
> need to update a common file is clearly not compatible with
> per-package directories and top-level parallel build.
>
> To fix this, we avoid using Python Eggs using the same
> --single-version-externally-managed option that we use for target
> modules. This option is normally meant for distributions packaging
> Python modules, and can therefore only be used if either --record (to
> record the list of files being installed) or --root is
> passed. --root=/ works fine and was suggested by
> https://stackoverflow.com/questions/6301003/stopping-setup-py-from-installing-as-egg/33791008#33791008.
>
> With this change, host Python modules installed by setuptools are now
> installed in the "regular" way, i.e directly in
> $(HOST_DIR)/lib/python2.7/site-packages/mako for host-python-mako.
>
> This makes the installation of host Python modules more similar to the
> one of target modules, and makes it compatible with per-package
> directory support and top-level parallel build.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/pkg-python.mk | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/package/pkg-python.mk b/package/pkg-python.mk
> index b641e3f865..4f42e2efc2 100644
> --- a/package/pkg-python.mk
> +++ b/package/pkg-python.mk
> @@ -85,7 +85,9 @@ HOST_PKG_PYTHON_SETUPTOOLS_ENV = \
>         PYTHONNOUSERSITE=1
>
>  HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \
> -       --prefix=$(HOST_DIR)
> +       --prefix=$(HOST_DIR) \
> +       --root=/ \
> +       --single-version-externally-managed
>
>  ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
>  define PKG_PYTHON_FIXUP_SYSCONFIGDATA
> --
> 2.20.1
>

Reviewed-by: Asaf Kahlon <asafka7@gmail.com>

[Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed

[Yegor Yefremov]

On Sat, Dec 29, 2018 at 4:15 PM Asaf Kahlon <asafka7@gmail.com> wrote:
>
> Hello,
>
> On Fri, Dec 28, 2018 at 7:01 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> >
> > When a package uses "setuptools" as its <pkg>_SETUP_TYPE, we currently
> > add a dependency on host-python-setuptools. This means that:
> >
> >  (1) When BR2_PACKAGE_PYTHON=y, the default host Python version is
> >      Python 2.x, and host-python-setuptools is installed for
> >      host-python.
> >
> >  (2) When BR2_PACKAGE_PYTHON3=y, the default host Python version is
> >      Python 3.x, and host-python-setuptools is installed for
> >      host-python3.
> >
> >  (3) When no target Python interpreter is selected, the default host
> >      Python version is Python 2.x, and host-python-setuptools is
> >      installed for host-python.
> >
> > Situations (1) and (3) are problematic for host Python packages that
> > need Python 3.x. Such packages use <pkg>_NEEDS_HOST_PYTHON = python3,
> > but if they use setuptools as their setup type, they will not find
> > setuptools installed for host-python3 in situations (1) and (3)
> > described above.
> >
> > We currently have a single package that sets <pkg>_NEEDS_HOST_PYTHON =
> > python3: host-meson. host-meson generally works because if setuptools
> > is not found, it falls back to distutils, which is part of the
> > standard Python library. However, if there is a setuptools version
> > installed system-wide, it may be picked up, but may not necessarily be
> > the same version as Buildroot setuptools, potentially causing
> > problems.
> >
> > This commit makes the necessary change to the python-package
> > infrastructure to fix this behavior, by identifying the following
> > cases:
> >
> >  - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python3,
> >    then we know it wants setuptools installed for host-python3, so we
> >    use host-python3-setuptools.
> >
> >  - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python2,
> >    then we known it wants setuptools installed for host-python, so we
> >    use host-python-setuptools.
> >
> >  - When BR2_PACKAGE_PYTHON3=y, and we have a target package, or a host
> >    package with no NEEDS_HOST_PYTHON option, then we want setuptools
> >    installed for host-python3, so we use host-python3-setuptools.
> >
> >  - When BR2_PACKAGE_PYTHON=y or no target interpreter is enabled at
> >    all, and we have a target package, or a host package with no
> >    NEEDS_HOST_PYTHON option, then we want setuptools for host-python,
> >    so we use host-python-setuptools.
> >
> > To make this happen, we use host-python3-setuptools introduced in a
> > previous commit, but we also change host-python-setuptools to force
> > its installation for host-python. The latter is needed if you build
> > with BR2_PACKAGE_PYTHON3=y but want to install a Python-based package
> > that has NEEDS_HOST_PYTHON=python2.
> >
> > There is one single package that needs be adjusted following this:
> > lirc-tools, because it is not using the python-package
> > infrastructure. It directly depends on host-python-setuptools, which
> > no longer works because host-python-setuptools now only installs for
> > Python 2.x, while lirc-tools Python binding only supports Python
> > 3.x. Switching to host-python3-setuptools solves this problem.
> >
> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > ---
> >  package/lirc-tools/lirc-tools.mk              |  2 +-
> >  package/pkg-python.mk                         | 35 ++++++++++++++-----
> >  .../python-setuptools/python-setuptools.mk    |  1 +
> >  3 files changed, 29 insertions(+), 9 deletions(-)
> >
> > diff --git a/package/lirc-tools/lirc-tools.mk b/package/lirc-tools/lirc-tools.mk
> > index a5165ad438..e468f2a25d 100644
> > --- a/package/lirc-tools/lirc-tools.mk
> > +++ b/package/lirc-tools/lirc-tools.mk
> > @@ -46,7 +46,7 @@ LIRC_TOOLS_DEPENDENCIES += libftdi1
> >  endif
> >
> >  ifeq ($(BR2_PACKAGE_PYTHON3),y)
> > -LIRC_TOOLS_DEPENDENCIES += python3 host-python-setuptools
> > +LIRC_TOOLS_DEPENDENCIES += python3 host-python3-setuptools
> >  LIRC_TOOLS_MAKE_ENV += SETUPTOOLS_ENV="$(PKG_PYTHON_SETUPTOOLS_ENV)"
> >  endif
> >
> > diff --git a/package/pkg-python.mk b/package/pkg-python.mk
> > index 3dc8a5d10f..b641e3f865 100644
> > --- a/package/pkg-python.mk
> > +++ b/package/pkg-python.mk
> > @@ -188,16 +188,35 @@ endif
> >  endif # ($$($(2)_NEEDS_HOST_PYTHON),)
> >  endif # ($(4),target)
> >
> > -# Setuptools based packages will need host-python-setuptools (both
> > -# host and target). We need to have a special exclusion for the
> > -# host-setuptools package itself: it is setuptools-based, but
> > -# shouldn't depend on host-setuptools (because it would otherwise
> > -# depend on itself!).
> > +# Setuptools based packages will need setuptools for the host Python
> > +# interpreter (both host and target).
> > +#
> > +# If we have a host package that says "I need Python 3", we install
> > +# setuptools for python3.
> > +#
> > +# If we have a host packge that says "I need Python 2", we install
> > +# setuptools for python2.
> > +#
> > +# If we have a target package, or a host package that doesn't have any
> > +# <pkg>_NEEDS_HOST_PYTHON, and BR2_PACKAGE_PYTHON3 is used, then
> > +# Python 3.x is the default Python interpreter, so we install
> > +# setuptools for python3.
> > +#
> > +# In all other cases, we install setuptools for python2. Those other
> > +# cases are: a target package or host package with
> > +# BR2_PACKAGE_PYTHON=y, or a host-package with neither
> > +# BR2_PACKAGE_PYTHON3=y or BR2_PACKAGE_PYTHON=y.
> >  ifeq ($$($(2)_SETUP_TYPE),setuptools)
> > -ifneq ($(2),HOST_PYTHON_SETUPTOOLS)
> > -$(2)_DEPENDENCIES += host-python-setuptools
> > -endif
> > +ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python3)
> > +$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools)
> > +else ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python2)
> > +$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools)
> > +else ifeq ($$(BR2_PACKAGE_PYTHON3),y)
> > +$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools)
> > +else
> > +$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools)
> >  endif
> > +endif # SETUP_TYPE
> >
> >  # Python interpreter to use for building the package.
> >  #
> > diff --git a/package/python-setuptools/python-setuptools.mk b/package/python-setuptools/python-setuptools.mk
> > index cc6f991cdd..d915449c5d 100644
> > --- a/package/python-setuptools/python-setuptools.mk
> > +++ b/package/python-setuptools/python-setuptools.mk
> > @@ -10,6 +10,7 @@ PYTHON_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861
> >  PYTHON_SETUPTOOLS_LICENSE = MIT
> >  PYTHON_SETUPTOOLS_LICENSE_FILES = LICENSE
> >  PYTHON_SETUPTOOLS_SETUP_TYPE = setuptools
> > +HOST_PYTHON_SETUPTOOLS_NEEDS_HOST_PYTHON = python2
> >
> >  define PYTHON_SETUPTOOLS_EXTRACT_CMDS
> >         $(UNZIP) -d $(@D) $(PYTHON_SETUPTOOLS_DL_DIR)/$(PYTHON_SETUPTOOLS_SOURCE)
> > --
> > 2.20.1
> >
>
> Reviewed-by: Asaf Kahlon <asafka7@gmail.com>

Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>

[Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools

[Yegor Yefremov]

On Sat, Dec 29, 2018 at 4:16 PM Asaf Kahlon <asafka7@gmail.com> wrote:
>
> Hello,
>
> On Fri, Dec 28, 2018 at 7:01 PM Thomas Petazzoni
> <thomas.petazzoni@bootlin.com> wrote:
> >
> > Currently, host Python packages that use setuptools are installed as
> > Python Eggs, i.e they are installed in
> > $(HOST_DIR)/lib/python2.7/site-packages/Something-<version>.egg. Once
> > installed, each Python Egg is registered to a file called
> > $(HOST_DIR)/lib/python2.7/site-packages/easy-install.pth. This file is
> > read by the Python interpreter so that the installation location of
> > each Egg is added to the Python path, and can be found by the Python
> > interpreter.
> >
> > However, the fact that the installation of different Python modules
> > need to update a common file is clearly not compatible with
> > per-package directories and top-level parallel build.
> >
> > To fix this, we avoid using Python Eggs using the same
> > --single-version-externally-managed option that we use for target
> > modules. This option is normally meant for distributions packaging
> > Python modules, and can therefore only be used if either --record (to
> > record the list of files being installed) or --root is
> > passed. --root=/ works fine and was suggested by
> > https://stackoverflow.com/questions/6301003/stopping-setup-py-from-installing-as-egg/33791008#33791008.
> >
> > With this change, host Python modules installed by setuptools are now
> > installed in the "regular" way, i.e directly in
> > $(HOST_DIR)/lib/python2.7/site-packages/mako for host-python-mako.
> >
> > This makes the installation of host Python modules more similar to the
> > one of target modules, and makes it compatible with per-package
> > directory support and top-level parallel build.
> >
> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > ---
> >  package/pkg-python.mk | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/package/pkg-python.mk b/package/pkg-python.mk
> > index b641e3f865..4f42e2efc2 100644
> > --- a/package/pkg-python.mk
> > +++ b/package/pkg-python.mk
> > @@ -85,7 +85,9 @@ HOST_PKG_PYTHON_SETUPTOOLS_ENV = \
> >         PYTHONNOUSERSITE=1
> >
> >  HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \
> > -       --prefix=$(HOST_DIR)
> > +       --prefix=$(HOST_DIR) \
> > +       --root=/ \
> > +       --single-version-externally-managed
> >
> >  ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
> >  define PKG_PYTHON_FIXUP_SYSCONFIGDATA
> > --
> > 2.20.1
> >
>
> Reviewed-by: Asaf Kahlon <asafka7@gmail.com>

Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>

[Buildroot] [PATCH 1/3] package/python3-setuptools: new package

[Yegor Yefremov]

On Fri, Dec 28, 2018 at 6:01 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> This new package allows to install the host version of setuptools for
> python3, regardless of what the default Python version is. It will be
> used by the python-package infrastructure to make sure that host
> Python packages that need python3 and setuptools have setuptools
> installed for Python 3.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>

> ---
>  .../0001-add-executable.patch                 | 72 +++++++++++++++++++
>  .../python3-setuptools.hash                   |  1 +
>  .../python3-setuptools/python3-setuptools.mk  | 24 +++++++
>  3 files changed, 97 insertions(+)
>  create mode 100644 package/python3-setuptools/0001-add-executable.patch
>  create mode 120000 package/python3-setuptools/python3-setuptools.hash
>  create mode 100644 package/python3-setuptools/python3-setuptools.mk
>
> diff --git a/package/python3-setuptools/0001-add-executable.patch b/package/python3-setuptools/0001-add-executable.patch
> new file mode 100644
> index 0000000000..1f8714eaeb
> --- /dev/null
> +++ b/package/python3-setuptools/0001-add-executable.patch
> @@ -0,0 +1,72 @@
> +From 9b3d307f8f6a1af88f3f810f5a6cf0835830e1e8 Mon Sep 17 00:00:00 2001
> +From: =?UTF-8?q?J=C3=B6rg=20Krause?= <joerg.krause@embedded.rocks>
> +Date: Mon, 7 Dec 2015 01:14:33 +0100
> +Subject: [PATCH 1/1] add executable
> +MIME-Version: 1.0
> +Content-Type: text/plain; charset=UTF-8
> +Content-Transfer-Encoding: 8bit
> +
> +Add a new --executable option to distribute so that we can
> +force the shebang line in installed python scripts.
> +
> +[Thomas: refresh for setuptools 5.8.]
> +[J?rg: refresh for setuptools 18.7.1]
> +
> +Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
> +Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> +Signed-off-by: J?rg Krause <joerg.krause@embedded.rocks>
> +---
> + setuptools/command/install.py         | 2 ++
> + setuptools/command/install_scripts.py | 9 +++++++++
> + 2 files changed, 11 insertions(+)
> +
> +diff --git a/setuptools/command/install.py b/setuptools/command/install.py
> +index d2bca2e..b60ef19 100644
> +--- a/setuptools/command/install.py
> ++++ b/setuptools/command/install.py
> +@@ -16,6 +16,7 @@ class install(orig.install):
> +     """Use easy_install to install the package, w/dependencies"""
> +
> +     user_options = orig.install.user_options + [
> ++        ('executable=', 'e', "specify final destination interpreter path"),
> +         ('old-and-unmanageable', None, "Try not to use this!"),
> +         ('single-version-externally-managed', None,
> +          "used by system package builders to create 'flat' eggs"),
> +@@ -31,6 +32,7 @@ class install(orig.install):
> +
> +     def initialize_options(self):
> +         orig.install.initialize_options(self)
> ++        self.executable = None
> +         self.old_and_unmanageable = None
> +         self.single_version_externally_managed = None
> +
> +diff --git a/setuptools/command/install_scripts.py b/setuptools/command/install_scripts.py
> +index be66cb2..07afaf8 100755
> +--- a/setuptools/command/install_scripts.py
> ++++ b/setuptools/command/install_scripts.py
> +@@ -11,6 +11,13 @@ class install_scripts(orig.install_scripts):
> +     def initialize_options(self):
> +         orig.install_scripts.initialize_options(self)
> +         self.no_ep = False
> ++        self.executable = None
> ++
> ++    def finalize_options(self):
> ++        orig.install_scripts.finalize_options(self)
> ++        self.set_undefined_options('install',
> ++                ('executable','executable')
> ++        )
> +
> +     def run(self):
> +         import setuptools.command.easy_install as ei
> +@@ -31,6 +38,8 @@ class install_scripts(orig.install_scripts):
> +         )
> +         bs_cmd = self.get_finalized_command('build_scripts')
> +         exec_param = getattr(bs_cmd, 'executable', None)
> ++        if self.executable is not None:
> ++            exec_param = self.executable
> +         bw_cmd = self.get_finalized_command("bdist_wininst")
> +         is_wininst = getattr(bw_cmd, '_is_running', False)
> +         writer = ei.ScriptWriter
> +--
> +2.6.3
> +
> diff --git a/package/python3-setuptools/python3-setuptools.hash b/package/python3-setuptools/python3-setuptools.hash
> new file mode 120000
> index 0000000000..a3af7f0fb1
> --- /dev/null
> +++ b/package/python3-setuptools/python3-setuptools.hash
> @@ -0,0 +1 @@
> +../python-setuptools/python-setuptools.hash
> \ No newline at end of file
> diff --git a/package/python3-setuptools/python3-setuptools.mk b/package/python3-setuptools/python3-setuptools.mk
> new file mode 100644
> index 0000000000..b6de5d4b25
> --- /dev/null
> +++ b/package/python3-setuptools/python3-setuptools.mk
> @@ -0,0 +1,24 @@
> +################################################################################
> +#
> +# python3-setuptools
> +#
> +################################################################################
> +
> +# Please keep in sync with
> +# package/python-setuptools/python-setuptools.mk
> +PYTHON3_SETUPTOOLS_VERSION = 40.6.3
> +PYTHON3_SETUPTOOLS_SOURCE = setuptools-$(PYTHON3_SETUPTOOLS_VERSION).zip
> +PYTHON3_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861991beeade31473868463dad0e58b1978c209de27384ae541b0b
> +PYTHON3_SETUPTOOLS_LICENSE = MIT
> +PYTHON3_SETUPTOOLS_LICENSE_FILES = LICENSE
> +PYTHON3_SETUPTOOLS_SETUP_TYPE = setuptools
> +HOST_PYTHON3_SETUPTOOLS_DL_SUBDIR = python-setuptools
> +HOST_PYTHON3_SETUPTOOLS_NEEDS_HOST_PYTHON = python3
> +
> +define HOST_PYTHON3_SETUPTOOLS_EXTRACT_CMDS
> +       $(UNZIP) -d $(@D) $(HOST_PYTHON3_SETUPTOOLS_DL_DIR)/$(PYTHON3_SETUPTOOLS_SOURCE)
> +       mv $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION)/* $(@D)
> +       $(RM) -r $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION)
> +endef
> +
> +$(eval $(host-python-package))
> --
> 2.20.1
>

[Buildroot] [PATCH 1/3] package/python3-setuptools: new package

[Thomas Petazzoni]

Hello,

On Fri, 28 Dec 2018 18:01:29 +0100, Thomas Petazzoni wrote:
> This new package allows to install the host version of setuptools for
> python3, regardless of what the default Python version is. It will be
> used by the python-package infrastructure to make sure that host
> Python packages that need python3 and setuptools have setuptools
> installed for Python 3.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  .../0001-add-executable.patch                 | 72 +++++++++++++++++++
>  .../python3-setuptools.hash                   |  1 +
>  .../python3-setuptools/python3-setuptools.mk  | 24 +++++++
>  3 files changed, 97 insertions(+)
>  create mode 100644 package/python3-setuptools/0001-add-executable.patch
>  create mode 120000 package/python3-setuptools/python3-setuptools.hash
>  create mode 100644 package/python3-setuptools/python3-setuptools.mk

Applied to master after changing the duplicated patch by a symlink to
the same patch in package/python-setuptools/, as suggested by Asaf.
Thanks Asaf and Yegor for the review!

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed

[Thomas Petazzoni]

Hello,

On Fri, 28 Dec 2018 18:01:30 +0100, Thomas Petazzoni wrote:
> When a package uses "setuptools" as its <pkg>_SETUP_TYPE, we currently
> add a dependency on host-python-setuptools. This means that:
> 
>  (1) When BR2_PACKAGE_PYTHON=y, the default host Python version is
>      Python 2.x, and host-python-setuptools is installed for
>      host-python.
> 
>  (2) When BR2_PACKAGE_PYTHON3=y, the default host Python version is
>      Python 3.x, and host-python-setuptools is installed for
>      host-python3.
> 
>  (3) When no target Python interpreter is selected, the default host
>      Python version is Python 2.x, and host-python-setuptools is
>      installed for host-python.
> 
> Situations (1) and (3) are problematic for host Python packages that
> need Python 3.x. Such packages use <pkg>_NEEDS_HOST_PYTHON = python3,
> but if they use setuptools as their setup type, they will not find
> setuptools installed for host-python3 in situations (1) and (3)
> described above.
> 
> We currently have a single package that sets <pkg>_NEEDS_HOST_PYTHON =
> python3: host-meson. host-meson generally works because if setuptools
> is not found, it falls back to distutils, which is part of the
> standard Python library. However, if there is a setuptools version
> installed system-wide, it may be picked up, but may not necessarily be
> the same version as Buildroot setuptools, potentially causing
> problems.
> 
> This commit makes the necessary change to the python-package
> infrastructure to fix this behavior, by identifying the following
> cases:
> 
>  - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python3,
>    then we know it wants setuptools installed for host-python3, so we
>    use host-python3-setuptools.
> 
>  - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python2,
>    then we known it wants setuptools installed for host-python, so we
>    use host-python-setuptools.
> 
>  - When BR2_PACKAGE_PYTHON3=y, and we have a target package, or a host
>    package with no NEEDS_HOST_PYTHON option, then we want setuptools
>    installed for host-python3, so we use host-python3-setuptools.
> 
>  - When BR2_PACKAGE_PYTHON=y or no target interpreter is enabled at
>    all, and we have a target package, or a host package with no
>    NEEDS_HOST_PYTHON option, then we want setuptools for host-python,
>    so we use host-python-setuptools.
> 
> To make this happen, we use host-python3-setuptools introduced in a
> previous commit, but we also change host-python-setuptools to force
> its installation for host-python. The latter is needed if you build
> with BR2_PACKAGE_PYTHON3=y but want to install a Python-based package
> that has NEEDS_HOST_PYTHON=python2.
> 
> There is one single package that needs be adjusted following this:
> lirc-tools, because it is not using the python-package
> infrastructure. It directly depends on host-python-setuptools, which
> no longer works because host-python-setuptools now only installs for
> Python 2.x, while lirc-tools Python binding only supports Python
> 3.x. Switching to host-python3-setuptools solves this problem.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/lirc-tools/lirc-tools.mk              |  2 +-
>  package/pkg-python.mk                         | 35 ++++++++++++++-----
>  .../python-setuptools/python-setuptools.mk    |  1 +
>  3 files changed, 29 insertions(+), 9 deletions(-)

Applied to master, thanks. Thanks Asaf and Yegor for the review!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools

[Thomas Petazzoni]

Hello,

On Fri, 28 Dec 2018 18:01:31 +0100, Thomas Petazzoni wrote:
> Currently, host Python packages that use setuptools are installed as
> Python Eggs, i.e they are installed in
> $(HOST_DIR)/lib/python2.7/site-packages/Something-<version>.egg. Once
> installed, each Python Egg is registered to a file called
> $(HOST_DIR)/lib/python2.7/site-packages/easy-install.pth. This file is
> read by the Python interpreter so that the installation location of
> each Egg is added to the Python path, and can be found by the Python
> interpreter.
> 
> However, the fact that the installation of different Python modules
> need to update a common file is clearly not compatible with
> per-package directories and top-level parallel build.
> 
> To fix this, we avoid using Python Eggs using the same
> --single-version-externally-managed option that we use for target
> modules. This option is normally meant for distributions packaging
> Python modules, and can therefore only be used if either --record (to
> record the list of files being installed) or --root is
> passed. --root=/ works fine and was suggested by
> https://stackoverflow.com/questions/6301003/stopping-setup-py-from-installing-as-egg/33791008#33791008.
> 
> With this change, host Python modules installed by setuptools are now
> installed in the "regular" way, i.e directly in
> $(HOST_DIR)/lib/python2.7/site-packages/mako for host-python-mako.
> 
> This makes the installation of host Python modules more similar to the
> one of target modules, and makes it compatible with per-package
> directory support and top-level parallel build.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  package/pkg-python.mk | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 1/3] package/python3-setuptools: new package

[Arnout Vandecappelle]

On 28/12/2018 18:01, Thomas Petazzoni wrote:
> +# Please keep in sync with
> +# package/python-setuptools/python-setuptools.mk

 It is in fact more important to have this comment in python-setuptools.mk,
because that is the one that most likely will be bumped...

 Note that, because you symlinked the hash file, autobuilders would eventually
discover such an issue anyway.

 Regards,
 Arnout

[Buildroot] [PATCH 1/3] package/python3-setuptools: new package

[Thomas Petazzoni]

Hello,

On Mon, 14 Jan 2019 19:31:05 +0100, Arnout Vandecappelle wrote:
> On 28/12/2018 18:01, Thomas Petazzoni wrote:
> > +# Please keep in sync with
> > +# package/python-setuptools/python-setuptools.mk  
> 
>  It is in fact more important to have this comment in python-setuptools.mk,
> because that is the one that most likely will be bumped...

I just submitted a patch doing this, you're Cc'ed. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com