* [Buildroot] [PATCH 0/3] Prepare Python support for top-level parallel build
@ 2018-12-28 17:01 Thomas Petazzoni
2018-12-28 17:01 ` [Buildroot] [PATCH 1/3] package/python3-setuptools: new package Thomas Petazzoni
` (2 more replies)
0 siblings, 3 replies; 15+ messages in thread
From: Thomas Petazzoni @ 2018-12-28 17:01 UTC (permalink / raw)
To: buildroot
Hello,
This series prepares our Python support for top-level parallel
build. To understand it, you need to read it from the last patch to
the first patch, because PATCH 1 is a consequence of PATCH 2, which is
itself a consequence of PATCH 3.
So, the problem started with the fact that when we install host Python
modules based on setuptools, we use the "regular" Python installation
process based on Python Eggs. This installation process has the
not-so-nice property of registering all installed Python Eggs in a
single common file:
$(HOST_DIR)/lib/python2.7/site-packages/easy-install.pth. Clearly,
this doesn't work well with per-package directory: if you have package
A that depends on host-python-B and host-python-C, with no dependency
relationship between host-python-B and host-python-C, when the
per-package directory for "A" gets created, it gets a version of the
easy-install.pth file either from host-python-B or host-python-C... so
it will not "see" one of them.
To fix this, we want to use the same installation process we use for
host Python modules than the one we use for target Python modules:
avoid Python Eggs, using the --single-version-externally-managed
setuptools option. This is the purpose of PATCH 3/3, which is simple
enough.
However, the story started becoming more complicated when we tried to
build host-meson. Indeed, host-meson needs host-python3 and its
installation process is based on setuptools. The issue is that when
Python 2.x is enabled for the target, or when no Python at all is
selected for the target, the default host Python version is 2.x, and
therefore host-python-setuptools is installed for Python 2.x, not
Python 3.x. This makes python-setuptools not usable for the
installation of host-meson. This was not visible until now, because
host-meson "transparently" falls back to distutils for its
installation process. *But* once you pass
--single-version-externally-managed, you have a problem: this option
is only supported by setuptools, not by distutils, so PATCH 3/3 breaks
the build of host-meson.
To solve this, we do something that we have been discussing for quite
a while: make sure python-setuptools can be available for host Python
3.x, even if Python 3.x is not the default host Python version. To
achieve this:
- PATCH 1/3 adds a host-python3-setuptools package.
- PATCH 2/3 changes the Python package infrastructure to use
host-python3-setuptools or host-python-setuptools depending on the
situation. host-python-setuptools is changed to always install for
Python 2.x, regardless of what is the default host Python version
so that host-python-setuptools and host-python3-setuptools never
step on each other.
Hopefully this cover letter helps explain the overall situation
clearly enough. Let me know if additional details are needed. This
code has survived some extensive autobuilder testing, hopefully I
haven't missed any special case.
Best regards,
Thomas
Thomas Petazzoni (3):
package/python3-setuptools: new package
package/pkg-python: use host-python3-setuptools when needed
package/pkg-python: use --single-version-externally-managed for host
setuptools
package/lirc-tools/lirc-tools.mk | 2 +-
package/pkg-python.mk | 39 +++++++---
.../python-setuptools/python-setuptools.mk | 1 +
.../0001-add-executable.patch | 72 +++++++++++++++++++
.../python3-setuptools.hash | 1 +
.../python3-setuptools/python3-setuptools.mk | 24 +++++++
6 files changed, 129 insertions(+), 10 deletions(-)
create mode 100644 package/python3-setuptools/0001-add-executable.patch
create mode 120000 package/python3-setuptools/python3-setuptools.hash
create mode 100644 package/python3-setuptools/python3-setuptools.mk
--
2.20.1
^ permalink raw reply [flat|nested] 15+ messages in thread* [Buildroot] [PATCH 1/3] package/python3-setuptools: new package 2018-12-28 17:01 [Buildroot] [PATCH 0/3] Prepare Python support for top-level parallel build Thomas Petazzoni @ 2018-12-28 17:01 ` Thomas Petazzoni 2018-12-29 15:13 ` Asaf Kahlon ` (3 more replies) 2018-12-28 17:01 ` [Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed Thomas Petazzoni 2018-12-28 17:01 ` [Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools Thomas Petazzoni 2 siblings, 4 replies; 15+ messages in thread From: Thomas Petazzoni @ 2018-12-28 17:01 UTC (permalink / raw) To: buildroot This new package allows to install the host version of setuptools for python3, regardless of what the default Python version is. It will be used by the python-package infrastructure to make sure that host Python packages that need python3 and setuptools have setuptools installed for Python 3. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- .../0001-add-executable.patch | 72 +++++++++++++++++++ .../python3-setuptools.hash | 1 + .../python3-setuptools/python3-setuptools.mk | 24 +++++++ 3 files changed, 97 insertions(+) create mode 100644 package/python3-setuptools/0001-add-executable.patch create mode 120000 package/python3-setuptools/python3-setuptools.hash create mode 100644 package/python3-setuptools/python3-setuptools.mk diff --git a/package/python3-setuptools/0001-add-executable.patch b/package/python3-setuptools/0001-add-executable.patch new file mode 100644 index 0000000000..1f8714eaeb --- /dev/null +++ b/package/python3-setuptools/0001-add-executable.patch @@ -0,0 +1,72 @@ +From 9b3d307f8f6a1af88f3f810f5a6cf0835830e1e8 Mon Sep 17 00:00:00 2001 +From: =?UTF-8?q?J=C3=B6rg=20Krause?= <joerg.krause@embedded.rocks> +Date: Mon, 7 Dec 2015 01:14:33 +0100 +Subject: [PATCH 1/1] add executable +MIME-Version: 1.0 +Content-Type: text/plain; charset=UTF-8 +Content-Transfer-Encoding: 8bit + +Add a new --executable option to distribute so that we can +force the shebang line in installed python scripts. + +[Thomas: refresh for setuptools 5.8.] +[J?rg: refresh for setuptools 18.7.1] + +Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar> +Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> +Signed-off-by: J?rg Krause <joerg.krause@embedded.rocks> +--- + setuptools/command/install.py | 2 ++ + setuptools/command/install_scripts.py | 9 +++++++++ + 2 files changed, 11 insertions(+) + +diff --git a/setuptools/command/install.py b/setuptools/command/install.py +index d2bca2e..b60ef19 100644 +--- a/setuptools/command/install.py ++++ b/setuptools/command/install.py +@@ -16,6 +16,7 @@ class install(orig.install): + """Use easy_install to install the package, w/dependencies""" + + user_options = orig.install.user_options + [ ++ ('executable=', 'e', "specify final destination interpreter path"), + ('old-and-unmanageable', None, "Try not to use this!"), + ('single-version-externally-managed', None, + "used by system package builders to create 'flat' eggs"), +@@ -31,6 +32,7 @@ class install(orig.install): + + def initialize_options(self): + orig.install.initialize_options(self) ++ self.executable = None + self.old_and_unmanageable = None + self.single_version_externally_managed = None + +diff --git a/setuptools/command/install_scripts.py b/setuptools/command/install_scripts.py +index be66cb2..07afaf8 100755 +--- a/setuptools/command/install_scripts.py ++++ b/setuptools/command/install_scripts.py +@@ -11,6 +11,13 @@ class install_scripts(orig.install_scripts): + def initialize_options(self): + orig.install_scripts.initialize_options(self) + self.no_ep = False ++ self.executable = None ++ ++ def finalize_options(self): ++ orig.install_scripts.finalize_options(self) ++ self.set_undefined_options('install', ++ ('executable','executable') ++ ) + + def run(self): + import setuptools.command.easy_install as ei +@@ -31,6 +38,8 @@ class install_scripts(orig.install_scripts): + ) + bs_cmd = self.get_finalized_command('build_scripts') + exec_param = getattr(bs_cmd, 'executable', None) ++ if self.executable is not None: ++ exec_param = self.executable + bw_cmd = self.get_finalized_command("bdist_wininst") + is_wininst = getattr(bw_cmd, '_is_running', False) + writer = ei.ScriptWriter +-- +2.6.3 + diff --git a/package/python3-setuptools/python3-setuptools.hash b/package/python3-setuptools/python3-setuptools.hash new file mode 120000 index 0000000000..a3af7f0fb1 --- /dev/null +++ b/package/python3-setuptools/python3-setuptools.hash @@ -0,0 +1 @@ +../python-setuptools/python-setuptools.hash \ No newline at end of file diff --git a/package/python3-setuptools/python3-setuptools.mk b/package/python3-setuptools/python3-setuptools.mk new file mode 100644 index 0000000000..b6de5d4b25 --- /dev/null +++ b/package/python3-setuptools/python3-setuptools.mk @@ -0,0 +1,24 @@ +################################################################################ +# +# python3-setuptools +# +################################################################################ + +# Please keep in sync with +# package/python-setuptools/python-setuptools.mk +PYTHON3_SETUPTOOLS_VERSION = 40.6.3 +PYTHON3_SETUPTOOLS_SOURCE = setuptools-$(PYTHON3_SETUPTOOLS_VERSION).zip +PYTHON3_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861991beeade31473868463dad0e58b1978c209de27384ae541b0b +PYTHON3_SETUPTOOLS_LICENSE = MIT +PYTHON3_SETUPTOOLS_LICENSE_FILES = LICENSE +PYTHON3_SETUPTOOLS_SETUP_TYPE = setuptools +HOST_PYTHON3_SETUPTOOLS_DL_SUBDIR = python-setuptools +HOST_PYTHON3_SETUPTOOLS_NEEDS_HOST_PYTHON = python3 + +define HOST_PYTHON3_SETUPTOOLS_EXTRACT_CMDS + $(UNZIP) -d $(@D) $(HOST_PYTHON3_SETUPTOOLS_DL_DIR)/$(PYTHON3_SETUPTOOLS_SOURCE) + mv $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION)/* $(@D) + $(RM) -r $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION) +endef + +$(eval $(host-python-package)) -- 2.20.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 1/3] package/python3-setuptools: new package 2018-12-28 17:01 ` [Buildroot] [PATCH 1/3] package/python3-setuptools: new package Thomas Petazzoni @ 2018-12-29 15:13 ` Asaf Kahlon 2018-12-31 18:32 ` Yegor Yefremov ` (2 subsequent siblings) 3 siblings, 0 replies; 15+ messages in thread From: Asaf Kahlon @ 2018-12-29 15:13 UTC (permalink / raw) To: buildroot Hello, On Fri, Dec 28, 2018 at 7:01 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > This new package allows to install the host version of setuptools for > python3, regardless of what the default Python version is. It will be > used by the python-package infrastructure to make sure that host > Python packages that need python3 and setuptools have setuptools > installed for Python 3. > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > .../0001-add-executable.patch | 72 +++++++++++++++++++ > .../python3-setuptools.hash | 1 + > .../python3-setuptools/python3-setuptools.mk | 24 +++++++ > 3 files changed, 97 insertions(+) > create mode 100644 package/python3-setuptools/0001-add-executable.patch > create mode 120000 package/python3-setuptools/python3-setuptools.hash > create mode 100644 package/python3-setuptools/python3-setuptools.mk > > diff --git a/package/python3-setuptools/0001-add-executable.patch b/package/python3-setuptools/0001-add-executable.patch > new file mode 100644 > index 0000000000..1f8714eaeb > --- /dev/null > +++ b/package/python3-setuptools/0001-add-executable.patch > @@ -0,0 +1,72 @@ > +From 9b3d307f8f6a1af88f3f810f5a6cf0835830e1e8 Mon Sep 17 00:00:00 2001 > +From: =?UTF-8?q?J=C3=B6rg=20Krause?= <joerg.krause@embedded.rocks> > +Date: Mon, 7 Dec 2015 01:14:33 +0100 > +Subject: [PATCH 1/1] add executable > +MIME-Version: 1.0 > +Content-Type: text/plain; charset=UTF-8 > +Content-Transfer-Encoding: 8bit > + > +Add a new --executable option to distribute so that we can > +force the shebang line in installed python scripts. > + > +[Thomas: refresh for setuptools 5.8.] > +[J?rg: refresh for setuptools 18.7.1] > + > +Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar> > +Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> > +Signed-off-by: J?rg Krause <joerg.krause@embedded.rocks> > +--- > + setuptools/command/install.py | 2 ++ > + setuptools/command/install_scripts.py | 9 +++++++++ > + 2 files changed, 11 insertions(+) > + > +diff --git a/setuptools/command/install.py b/setuptools/command/install.py > +index d2bca2e..b60ef19 100644 > +--- a/setuptools/command/install.py > ++++ b/setuptools/command/install.py > +@@ -16,6 +16,7 @@ class install(orig.install): > + """Use easy_install to install the package, w/dependencies""" > + > + user_options = orig.install.user_options + [ > ++ ('executable=', 'e', "specify final destination interpreter path"), > + ('old-and-unmanageable', None, "Try not to use this!"), > + ('single-version-externally-managed', None, > + "used by system package builders to create 'flat' eggs"), > +@@ -31,6 +32,7 @@ class install(orig.install): > + > + def initialize_options(self): > + orig.install.initialize_options(self) > ++ self.executable = None > + self.old_and_unmanageable = None > + self.single_version_externally_managed = None > + > +diff --git a/setuptools/command/install_scripts.py b/setuptools/command/install_scripts.py > +index be66cb2..07afaf8 100755 > +--- a/setuptools/command/install_scripts.py > ++++ b/setuptools/command/install_scripts.py > +@@ -11,6 +11,13 @@ class install_scripts(orig.install_scripts): > + def initialize_options(self): > + orig.install_scripts.initialize_options(self) > + self.no_ep = False > ++ self.executable = None > ++ > ++ def finalize_options(self): > ++ orig.install_scripts.finalize_options(self) > ++ self.set_undefined_options('install', > ++ ('executable','executable') > ++ ) > + > + def run(self): > + import setuptools.command.easy_install as ei > +@@ -31,6 +38,8 @@ class install_scripts(orig.install_scripts): > + ) > + bs_cmd = self.get_finalized_command('build_scripts') > + exec_param = getattr(bs_cmd, 'executable', None) > ++ if self.executable is not None: > ++ exec_param = self.executable > + bw_cmd = self.get_finalized_command("bdist_wininst") > + is_wininst = getattr(bw_cmd, '_is_running', False) > + writer = ei.ScriptWriter > +-- > +2.6.3 > + Is there any reason to copy the exact patch file from python-setuptools, instead of creating a link (like you did for the hash file)? I don't think the patch will be different between host and target compilations... > diff --git a/package/python3-setuptools/python3-setuptools.hash b/package/python3-setuptools/python3-setuptools.hash > new file mode 120000 > index 0000000000..a3af7f0fb1 > --- /dev/null > +++ b/package/python3-setuptools/python3-setuptools.hash > @@ -0,0 +1 @@ > +../python-setuptools/python-setuptools.hash > \ No newline at end of file > diff --git a/package/python3-setuptools/python3-setuptools.mk b/package/python3-setuptools/python3-setuptools.mk > new file mode 100644 > index 0000000000..b6de5d4b25 > --- /dev/null > +++ b/package/python3-setuptools/python3-setuptools.mk > @@ -0,0 +1,24 @@ > +################################################################################ > +# > +# python3-setuptools > +# > +################################################################################ > + > +# Please keep in sync with > +# package/python-setuptools/python-setuptools.mk > +PYTHON3_SETUPTOOLS_VERSION = 40.6.3 > +PYTHON3_SETUPTOOLS_SOURCE = setuptools-$(PYTHON3_SETUPTOOLS_VERSION).zip > +PYTHON3_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861991beeade31473868463dad0e58b1978c209de27384ae541b0b > +PYTHON3_SETUPTOOLS_LICENSE = MIT > +PYTHON3_SETUPTOOLS_LICENSE_FILES = LICENSE > +PYTHON3_SETUPTOOLS_SETUP_TYPE = setuptools > +HOST_PYTHON3_SETUPTOOLS_DL_SUBDIR = python-setuptools > +HOST_PYTHON3_SETUPTOOLS_NEEDS_HOST_PYTHON = python3 > + > +define HOST_PYTHON3_SETUPTOOLS_EXTRACT_CMDS > + $(UNZIP) -d $(@D) $(HOST_PYTHON3_SETUPTOOLS_DL_DIR)/$(PYTHON3_SETUPTOOLS_SOURCE) > + mv $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION)/* $(@D) > + $(RM) -r $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION) > +endef > + > +$(eval $(host-python-package)) > -- > 2.20.1 > Regards, Asaf. ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 1/3] package/python3-setuptools: new package 2018-12-28 17:01 ` [Buildroot] [PATCH 1/3] package/python3-setuptools: new package Thomas Petazzoni 2018-12-29 15:13 ` Asaf Kahlon @ 2018-12-31 18:32 ` Yegor Yefremov 2019-01-01 10:31 ` Thomas Petazzoni 2019-01-14 18:31 ` Arnout Vandecappelle 3 siblings, 0 replies; 15+ messages in thread From: Yegor Yefremov @ 2018-12-31 18:32 UTC (permalink / raw) To: buildroot On Fri, Dec 28, 2018 at 6:01 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > This new package allows to install the host version of setuptools for > python3, regardless of what the default Python version is. It will be > used by the python-package infrastructure to make sure that host > Python packages that need python3 and setuptools have setuptools > installed for Python 3. > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> > --- > .../0001-add-executable.patch | 72 +++++++++++++++++++ > .../python3-setuptools.hash | 1 + > .../python3-setuptools/python3-setuptools.mk | 24 +++++++ > 3 files changed, 97 insertions(+) > create mode 100644 package/python3-setuptools/0001-add-executable.patch > create mode 120000 package/python3-setuptools/python3-setuptools.hash > create mode 100644 package/python3-setuptools/python3-setuptools.mk > > diff --git a/package/python3-setuptools/0001-add-executable.patch b/package/python3-setuptools/0001-add-executable.patch > new file mode 100644 > index 0000000000..1f8714eaeb > --- /dev/null > +++ b/package/python3-setuptools/0001-add-executable.patch > @@ -0,0 +1,72 @@ > +From 9b3d307f8f6a1af88f3f810f5a6cf0835830e1e8 Mon Sep 17 00:00:00 2001 > +From: =?UTF-8?q?J=C3=B6rg=20Krause?= <joerg.krause@embedded.rocks> > +Date: Mon, 7 Dec 2015 01:14:33 +0100 > +Subject: [PATCH 1/1] add executable > +MIME-Version: 1.0 > +Content-Type: text/plain; charset=UTF-8 > +Content-Transfer-Encoding: 8bit > + > +Add a new --executable option to distribute so that we can > +force the shebang line in installed python scripts. > + > +[Thomas: refresh for setuptools 5.8.] > +[J?rg: refresh for setuptools 18.7.1] > + > +Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar> > +Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> > +Signed-off-by: J?rg Krause <joerg.krause@embedded.rocks> > +--- > + setuptools/command/install.py | 2 ++ > + setuptools/command/install_scripts.py | 9 +++++++++ > + 2 files changed, 11 insertions(+) > + > +diff --git a/setuptools/command/install.py b/setuptools/command/install.py > +index d2bca2e..b60ef19 100644 > +--- a/setuptools/command/install.py > ++++ b/setuptools/command/install.py > +@@ -16,6 +16,7 @@ class install(orig.install): > + """Use easy_install to install the package, w/dependencies""" > + > + user_options = orig.install.user_options + [ > ++ ('executable=', 'e', "specify final destination interpreter path"), > + ('old-and-unmanageable', None, "Try not to use this!"), > + ('single-version-externally-managed', None, > + "used by system package builders to create 'flat' eggs"), > +@@ -31,6 +32,7 @@ class install(orig.install): > + > + def initialize_options(self): > + orig.install.initialize_options(self) > ++ self.executable = None > + self.old_and_unmanageable = None > + self.single_version_externally_managed = None > + > +diff --git a/setuptools/command/install_scripts.py b/setuptools/command/install_scripts.py > +index be66cb2..07afaf8 100755 > +--- a/setuptools/command/install_scripts.py > ++++ b/setuptools/command/install_scripts.py > +@@ -11,6 +11,13 @@ class install_scripts(orig.install_scripts): > + def initialize_options(self): > + orig.install_scripts.initialize_options(self) > + self.no_ep = False > ++ self.executable = None > ++ > ++ def finalize_options(self): > ++ orig.install_scripts.finalize_options(self) > ++ self.set_undefined_options('install', > ++ ('executable','executable') > ++ ) > + > + def run(self): > + import setuptools.command.easy_install as ei > +@@ -31,6 +38,8 @@ class install_scripts(orig.install_scripts): > + ) > + bs_cmd = self.get_finalized_command('build_scripts') > + exec_param = getattr(bs_cmd, 'executable', None) > ++ if self.executable is not None: > ++ exec_param = self.executable > + bw_cmd = self.get_finalized_command("bdist_wininst") > + is_wininst = getattr(bw_cmd, '_is_running', False) > + writer = ei.ScriptWriter > +-- > +2.6.3 > + > diff --git a/package/python3-setuptools/python3-setuptools.hash b/package/python3-setuptools/python3-setuptools.hash > new file mode 120000 > index 0000000000..a3af7f0fb1 > --- /dev/null > +++ b/package/python3-setuptools/python3-setuptools.hash > @@ -0,0 +1 @@ > +../python-setuptools/python-setuptools.hash > \ No newline at end of file > diff --git a/package/python3-setuptools/python3-setuptools.mk b/package/python3-setuptools/python3-setuptools.mk > new file mode 100644 > index 0000000000..b6de5d4b25 > --- /dev/null > +++ b/package/python3-setuptools/python3-setuptools.mk > @@ -0,0 +1,24 @@ > +################################################################################ > +# > +# python3-setuptools > +# > +################################################################################ > + > +# Please keep in sync with > +# package/python-setuptools/python-setuptools.mk > +PYTHON3_SETUPTOOLS_VERSION = 40.6.3 > +PYTHON3_SETUPTOOLS_SOURCE = setuptools-$(PYTHON3_SETUPTOOLS_VERSION).zip > +PYTHON3_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861991beeade31473868463dad0e58b1978c209de27384ae541b0b > +PYTHON3_SETUPTOOLS_LICENSE = MIT > +PYTHON3_SETUPTOOLS_LICENSE_FILES = LICENSE > +PYTHON3_SETUPTOOLS_SETUP_TYPE = setuptools > +HOST_PYTHON3_SETUPTOOLS_DL_SUBDIR = python-setuptools > +HOST_PYTHON3_SETUPTOOLS_NEEDS_HOST_PYTHON = python3 > + > +define HOST_PYTHON3_SETUPTOOLS_EXTRACT_CMDS > + $(UNZIP) -d $(@D) $(HOST_PYTHON3_SETUPTOOLS_DL_DIR)/$(PYTHON3_SETUPTOOLS_SOURCE) > + mv $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION)/* $(@D) > + $(RM) -r $(@D)/setuptools-$(PYTHON3_SETUPTOOLS_VERSION) > +endef > + > +$(eval $(host-python-package)) > -- > 2.20.1 > ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 1/3] package/python3-setuptools: new package 2018-12-28 17:01 ` [Buildroot] [PATCH 1/3] package/python3-setuptools: new package Thomas Petazzoni 2018-12-29 15:13 ` Asaf Kahlon 2018-12-31 18:32 ` Yegor Yefremov @ 2019-01-01 10:31 ` Thomas Petazzoni 2019-01-14 18:31 ` Arnout Vandecappelle 3 siblings, 0 replies; 15+ messages in thread From: Thomas Petazzoni @ 2019-01-01 10:31 UTC (permalink / raw) To: buildroot Hello, On Fri, 28 Dec 2018 18:01:29 +0100, Thomas Petazzoni wrote: > This new package allows to install the host version of setuptools for > python3, regardless of what the default Python version is. It will be > used by the python-package infrastructure to make sure that host > Python packages that need python3 and setuptools have setuptools > installed for Python 3. > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > .../0001-add-executable.patch | 72 +++++++++++++++++++ > .../python3-setuptools.hash | 1 + > .../python3-setuptools/python3-setuptools.mk | 24 +++++++ > 3 files changed, 97 insertions(+) > create mode 100644 package/python3-setuptools/0001-add-executable.patch > create mode 120000 package/python3-setuptools/python3-setuptools.hash > create mode 100644 package/python3-setuptools/python3-setuptools.mk Applied to master after changing the duplicated patch by a symlink to the same patch in package/python-setuptools/, as suggested by Asaf. Thanks Asaf and Yegor for the review! Best regards, Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 1/3] package/python3-setuptools: new package 2018-12-28 17:01 ` [Buildroot] [PATCH 1/3] package/python3-setuptools: new package Thomas Petazzoni ` (2 preceding siblings ...) 2019-01-01 10:31 ` Thomas Petazzoni @ 2019-01-14 18:31 ` Arnout Vandecappelle 2019-01-14 19:59 ` Thomas Petazzoni 3 siblings, 1 reply; 15+ messages in thread From: Arnout Vandecappelle @ 2019-01-14 18:31 UTC (permalink / raw) To: buildroot On 28/12/2018 18:01, Thomas Petazzoni wrote: > +# Please keep in sync with > +# package/python-setuptools/python-setuptools.mk It is in fact more important to have this comment in python-setuptools.mk, because that is the one that most likely will be bumped... Note that, because you symlinked the hash file, autobuilders would eventually discover such an issue anyway. Regards, Arnout ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 1/3] package/python3-setuptools: new package 2019-01-14 18:31 ` Arnout Vandecappelle @ 2019-01-14 19:59 ` Thomas Petazzoni 0 siblings, 0 replies; 15+ messages in thread From: Thomas Petazzoni @ 2019-01-14 19:59 UTC (permalink / raw) To: buildroot Hello, On Mon, 14 Jan 2019 19:31:05 +0100, Arnout Vandecappelle wrote: > On 28/12/2018 18:01, Thomas Petazzoni wrote: > > +# Please keep in sync with > > +# package/python-setuptools/python-setuptools.mk > > It is in fact more important to have this comment in python-setuptools.mk, > because that is the one that most likely will be bumped... I just submitted a patch doing this, you're Cc'ed. Thanks! Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed 2018-12-28 17:01 [Buildroot] [PATCH 0/3] Prepare Python support for top-level parallel build Thomas Petazzoni 2018-12-28 17:01 ` [Buildroot] [PATCH 1/3] package/python3-setuptools: new package Thomas Petazzoni @ 2018-12-28 17:01 ` Thomas Petazzoni 2018-12-29 15:15 ` Asaf Kahlon 2019-01-01 10:31 ` Thomas Petazzoni 2018-12-28 17:01 ` [Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools Thomas Petazzoni 2 siblings, 2 replies; 15+ messages in thread From: Thomas Petazzoni @ 2018-12-28 17:01 UTC (permalink / raw) To: buildroot When a package uses "setuptools" as its <pkg>_SETUP_TYPE, we currently add a dependency on host-python-setuptools. This means that: (1) When BR2_PACKAGE_PYTHON=y, the default host Python version is Python 2.x, and host-python-setuptools is installed for host-python. (2) When BR2_PACKAGE_PYTHON3=y, the default host Python version is Python 3.x, and host-python-setuptools is installed for host-python3. (3) When no target Python interpreter is selected, the default host Python version is Python 2.x, and host-python-setuptools is installed for host-python. Situations (1) and (3) are problematic for host Python packages that need Python 3.x. Such packages use <pkg>_NEEDS_HOST_PYTHON = python3, but if they use setuptools as their setup type, they will not find setuptools installed for host-python3 in situations (1) and (3) described above. We currently have a single package that sets <pkg>_NEEDS_HOST_PYTHON = python3: host-meson. host-meson generally works because if setuptools is not found, it falls back to distutils, which is part of the standard Python library. However, if there is a setuptools version installed system-wide, it may be picked up, but may not necessarily be the same version as Buildroot setuptools, potentially causing problems. This commit makes the necessary change to the python-package infrastructure to fix this behavior, by identifying the following cases: - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python3, then we know it wants setuptools installed for host-python3, so we use host-python3-setuptools. - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python2, then we known it wants setuptools installed for host-python, so we use host-python-setuptools. - When BR2_PACKAGE_PYTHON3=y, and we have a target package, or a host package with no NEEDS_HOST_PYTHON option, then we want setuptools installed for host-python3, so we use host-python3-setuptools. - When BR2_PACKAGE_PYTHON=y or no target interpreter is enabled at all, and we have a target package, or a host package with no NEEDS_HOST_PYTHON option, then we want setuptools for host-python, so we use host-python-setuptools. To make this happen, we use host-python3-setuptools introduced in a previous commit, but we also change host-python-setuptools to force its installation for host-python. The latter is needed if you build with BR2_PACKAGE_PYTHON3=y but want to install a Python-based package that has NEEDS_HOST_PYTHON=python2. There is one single package that needs be adjusted following this: lirc-tools, because it is not using the python-package infrastructure. It directly depends on host-python-setuptools, which no longer works because host-python-setuptools now only installs for Python 2.x, while lirc-tools Python binding only supports Python 3.x. Switching to host-python3-setuptools solves this problem. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- package/lirc-tools/lirc-tools.mk | 2 +- package/pkg-python.mk | 35 ++++++++++++++----- .../python-setuptools/python-setuptools.mk | 1 + 3 files changed, 29 insertions(+), 9 deletions(-) diff --git a/package/lirc-tools/lirc-tools.mk b/package/lirc-tools/lirc-tools.mk index a5165ad438..e468f2a25d 100644 --- a/package/lirc-tools/lirc-tools.mk +++ b/package/lirc-tools/lirc-tools.mk @@ -46,7 +46,7 @@ LIRC_TOOLS_DEPENDENCIES += libftdi1 endif ifeq ($(BR2_PACKAGE_PYTHON3),y) -LIRC_TOOLS_DEPENDENCIES += python3 host-python-setuptools +LIRC_TOOLS_DEPENDENCIES += python3 host-python3-setuptools LIRC_TOOLS_MAKE_ENV += SETUPTOOLS_ENV="$(PKG_PYTHON_SETUPTOOLS_ENV)" endif diff --git a/package/pkg-python.mk b/package/pkg-python.mk index 3dc8a5d10f..b641e3f865 100644 --- a/package/pkg-python.mk +++ b/package/pkg-python.mk @@ -188,16 +188,35 @@ endif endif # ($$($(2)_NEEDS_HOST_PYTHON),) endif # ($(4),target) -# Setuptools based packages will need host-python-setuptools (both -# host and target). We need to have a special exclusion for the -# host-setuptools package itself: it is setuptools-based, but -# shouldn't depend on host-setuptools (because it would otherwise -# depend on itself!). +# Setuptools based packages will need setuptools for the host Python +# interpreter (both host and target). +# +# If we have a host package that says "I need Python 3", we install +# setuptools for python3. +# +# If we have a host packge that says "I need Python 2", we install +# setuptools for python2. +# +# If we have a target package, or a host package that doesn't have any +# <pkg>_NEEDS_HOST_PYTHON, and BR2_PACKAGE_PYTHON3 is used, then +# Python 3.x is the default Python interpreter, so we install +# setuptools for python3. +# +# In all other cases, we install setuptools for python2. Those other +# cases are: a target package or host package with +# BR2_PACKAGE_PYTHON=y, or a host-package with neither +# BR2_PACKAGE_PYTHON3=y or BR2_PACKAGE_PYTHON=y. ifeq ($$($(2)_SETUP_TYPE),setuptools) -ifneq ($(2),HOST_PYTHON_SETUPTOOLS) -$(2)_DEPENDENCIES += host-python-setuptools -endif +ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python3) +$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools) +else ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python2) +$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools) +else ifeq ($$(BR2_PACKAGE_PYTHON3),y) +$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools) +else +$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools) endif +endif # SETUP_TYPE # Python interpreter to use for building the package. # diff --git a/package/python-setuptools/python-setuptools.mk b/package/python-setuptools/python-setuptools.mk index cc6f991cdd..d915449c5d 100644 --- a/package/python-setuptools/python-setuptools.mk +++ b/package/python-setuptools/python-setuptools.mk @@ -10,6 +10,7 @@ PYTHON_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861 PYTHON_SETUPTOOLS_LICENSE = MIT PYTHON_SETUPTOOLS_LICENSE_FILES = LICENSE PYTHON_SETUPTOOLS_SETUP_TYPE = setuptools +HOST_PYTHON_SETUPTOOLS_NEEDS_HOST_PYTHON = python2 define PYTHON_SETUPTOOLS_EXTRACT_CMDS $(UNZIP) -d $(@D) $(PYTHON_SETUPTOOLS_DL_DIR)/$(PYTHON_SETUPTOOLS_SOURCE) -- 2.20.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed 2018-12-28 17:01 ` [Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed Thomas Petazzoni @ 2018-12-29 15:15 ` Asaf Kahlon 2018-12-31 18:29 ` Yegor Yefremov 2019-01-01 10:31 ` Thomas Petazzoni 1 sibling, 1 reply; 15+ messages in thread From: Asaf Kahlon @ 2018-12-29 15:15 UTC (permalink / raw) To: buildroot Hello, On Fri, Dec 28, 2018 at 7:01 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > When a package uses "setuptools" as its <pkg>_SETUP_TYPE, we currently > add a dependency on host-python-setuptools. This means that: > > (1) When BR2_PACKAGE_PYTHON=y, the default host Python version is > Python 2.x, and host-python-setuptools is installed for > host-python. > > (2) When BR2_PACKAGE_PYTHON3=y, the default host Python version is > Python 3.x, and host-python-setuptools is installed for > host-python3. > > (3) When no target Python interpreter is selected, the default host > Python version is Python 2.x, and host-python-setuptools is > installed for host-python. > > Situations (1) and (3) are problematic for host Python packages that > need Python 3.x. Such packages use <pkg>_NEEDS_HOST_PYTHON = python3, > but if they use setuptools as their setup type, they will not find > setuptools installed for host-python3 in situations (1) and (3) > described above. > > We currently have a single package that sets <pkg>_NEEDS_HOST_PYTHON = > python3: host-meson. host-meson generally works because if setuptools > is not found, it falls back to distutils, which is part of the > standard Python library. However, if there is a setuptools version > installed system-wide, it may be picked up, but may not necessarily be > the same version as Buildroot setuptools, potentially causing > problems. > > This commit makes the necessary change to the python-package > infrastructure to fix this behavior, by identifying the following > cases: > > - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python3, > then we know it wants setuptools installed for host-python3, so we > use host-python3-setuptools. > > - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python2, > then we known it wants setuptools installed for host-python, so we > use host-python-setuptools. > > - When BR2_PACKAGE_PYTHON3=y, and we have a target package, or a host > package with no NEEDS_HOST_PYTHON option, then we want setuptools > installed for host-python3, so we use host-python3-setuptools. > > - When BR2_PACKAGE_PYTHON=y or no target interpreter is enabled at > all, and we have a target package, or a host package with no > NEEDS_HOST_PYTHON option, then we want setuptools for host-python, > so we use host-python-setuptools. > > To make this happen, we use host-python3-setuptools introduced in a > previous commit, but we also change host-python-setuptools to force > its installation for host-python. The latter is needed if you build > with BR2_PACKAGE_PYTHON3=y but want to install a Python-based package > that has NEEDS_HOST_PYTHON=python2. > > There is one single package that needs be adjusted following this: > lirc-tools, because it is not using the python-package > infrastructure. It directly depends on host-python-setuptools, which > no longer works because host-python-setuptools now only installs for > Python 2.x, while lirc-tools Python binding only supports Python > 3.x. Switching to host-python3-setuptools solves this problem. > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > package/lirc-tools/lirc-tools.mk | 2 +- > package/pkg-python.mk | 35 ++++++++++++++----- > .../python-setuptools/python-setuptools.mk | 1 + > 3 files changed, 29 insertions(+), 9 deletions(-) > > diff --git a/package/lirc-tools/lirc-tools.mk b/package/lirc-tools/lirc-tools.mk > index a5165ad438..e468f2a25d 100644 > --- a/package/lirc-tools/lirc-tools.mk > +++ b/package/lirc-tools/lirc-tools.mk > @@ -46,7 +46,7 @@ LIRC_TOOLS_DEPENDENCIES += libftdi1 > endif > > ifeq ($(BR2_PACKAGE_PYTHON3),y) > -LIRC_TOOLS_DEPENDENCIES += python3 host-python-setuptools > +LIRC_TOOLS_DEPENDENCIES += python3 host-python3-setuptools > LIRC_TOOLS_MAKE_ENV += SETUPTOOLS_ENV="$(PKG_PYTHON_SETUPTOOLS_ENV)" > endif > > diff --git a/package/pkg-python.mk b/package/pkg-python.mk > index 3dc8a5d10f..b641e3f865 100644 > --- a/package/pkg-python.mk > +++ b/package/pkg-python.mk > @@ -188,16 +188,35 @@ endif > endif # ($$($(2)_NEEDS_HOST_PYTHON),) > endif # ($(4),target) > > -# Setuptools based packages will need host-python-setuptools (both > -# host and target). We need to have a special exclusion for the > -# host-setuptools package itself: it is setuptools-based, but > -# shouldn't depend on host-setuptools (because it would otherwise > -# depend on itself!). > +# Setuptools based packages will need setuptools for the host Python > +# interpreter (both host and target). > +# > +# If we have a host package that says "I need Python 3", we install > +# setuptools for python3. > +# > +# If we have a host packge that says "I need Python 2", we install > +# setuptools for python2. > +# > +# If we have a target package, or a host package that doesn't have any > +# <pkg>_NEEDS_HOST_PYTHON, and BR2_PACKAGE_PYTHON3 is used, then > +# Python 3.x is the default Python interpreter, so we install > +# setuptools for python3. > +# > +# In all other cases, we install setuptools for python2. Those other > +# cases are: a target package or host package with > +# BR2_PACKAGE_PYTHON=y, or a host-package with neither > +# BR2_PACKAGE_PYTHON3=y or BR2_PACKAGE_PYTHON=y. > ifeq ($$($(2)_SETUP_TYPE),setuptools) > -ifneq ($(2),HOST_PYTHON_SETUPTOOLS) > -$(2)_DEPENDENCIES += host-python-setuptools > -endif > +ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python3) > +$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools) > +else ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python2) > +$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools) > +else ifeq ($$(BR2_PACKAGE_PYTHON3),y) > +$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools) > +else > +$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools) > endif > +endif # SETUP_TYPE > > # Python interpreter to use for building the package. > # > diff --git a/package/python-setuptools/python-setuptools.mk b/package/python-setuptools/python-setuptools.mk > index cc6f991cdd..d915449c5d 100644 > --- a/package/python-setuptools/python-setuptools.mk > +++ b/package/python-setuptools/python-setuptools.mk > @@ -10,6 +10,7 @@ PYTHON_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861 > PYTHON_SETUPTOOLS_LICENSE = MIT > PYTHON_SETUPTOOLS_LICENSE_FILES = LICENSE > PYTHON_SETUPTOOLS_SETUP_TYPE = setuptools > +HOST_PYTHON_SETUPTOOLS_NEEDS_HOST_PYTHON = python2 > > define PYTHON_SETUPTOOLS_EXTRACT_CMDS > $(UNZIP) -d $(@D) $(PYTHON_SETUPTOOLS_DL_DIR)/$(PYTHON_SETUPTOOLS_SOURCE) > -- > 2.20.1 > Reviewed-by: Asaf Kahlon <asafka7@gmail.com> ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed 2018-12-29 15:15 ` Asaf Kahlon @ 2018-12-31 18:29 ` Yegor Yefremov 0 siblings, 0 replies; 15+ messages in thread From: Yegor Yefremov @ 2018-12-31 18:29 UTC (permalink / raw) To: buildroot On Sat, Dec 29, 2018 at 4:15 PM Asaf Kahlon <asafka7@gmail.com> wrote: > > Hello, > > On Fri, Dec 28, 2018 at 7:01 PM Thomas Petazzoni > <thomas.petazzoni@bootlin.com> wrote: > > > > When a package uses "setuptools" as its <pkg>_SETUP_TYPE, we currently > > add a dependency on host-python-setuptools. This means that: > > > > (1) When BR2_PACKAGE_PYTHON=y, the default host Python version is > > Python 2.x, and host-python-setuptools is installed for > > host-python. > > > > (2) When BR2_PACKAGE_PYTHON3=y, the default host Python version is > > Python 3.x, and host-python-setuptools is installed for > > host-python3. > > > > (3) When no target Python interpreter is selected, the default host > > Python version is Python 2.x, and host-python-setuptools is > > installed for host-python. > > > > Situations (1) and (3) are problematic for host Python packages that > > need Python 3.x. Such packages use <pkg>_NEEDS_HOST_PYTHON = python3, > > but if they use setuptools as their setup type, they will not find > > setuptools installed for host-python3 in situations (1) and (3) > > described above. > > > > We currently have a single package that sets <pkg>_NEEDS_HOST_PYTHON = > > python3: host-meson. host-meson generally works because if setuptools > > is not found, it falls back to distutils, which is part of the > > standard Python library. However, if there is a setuptools version > > installed system-wide, it may be picked up, but may not necessarily be > > the same version as Buildroot setuptools, potentially causing > > problems. > > > > This commit makes the necessary change to the python-package > > infrastructure to fix this behavior, by identifying the following > > cases: > > > > - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python3, > > then we know it wants setuptools installed for host-python3, so we > > use host-python3-setuptools. > > > > - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python2, > > then we known it wants setuptools installed for host-python, so we > > use host-python-setuptools. > > > > - When BR2_PACKAGE_PYTHON3=y, and we have a target package, or a host > > package with no NEEDS_HOST_PYTHON option, then we want setuptools > > installed for host-python3, so we use host-python3-setuptools. > > > > - When BR2_PACKAGE_PYTHON=y or no target interpreter is enabled at > > all, and we have a target package, or a host package with no > > NEEDS_HOST_PYTHON option, then we want setuptools for host-python, > > so we use host-python-setuptools. > > > > To make this happen, we use host-python3-setuptools introduced in a > > previous commit, but we also change host-python-setuptools to force > > its installation for host-python. The latter is needed if you build > > with BR2_PACKAGE_PYTHON3=y but want to install a Python-based package > > that has NEEDS_HOST_PYTHON=python2. > > > > There is one single package that needs be adjusted following this: > > lirc-tools, because it is not using the python-package > > infrastructure. It directly depends on host-python-setuptools, which > > no longer works because host-python-setuptools now only installs for > > Python 2.x, while lirc-tools Python binding only supports Python > > 3.x. Switching to host-python3-setuptools solves this problem. > > > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > > --- > > package/lirc-tools/lirc-tools.mk | 2 +- > > package/pkg-python.mk | 35 ++++++++++++++----- > > .../python-setuptools/python-setuptools.mk | 1 + > > 3 files changed, 29 insertions(+), 9 deletions(-) > > > > diff --git a/package/lirc-tools/lirc-tools.mk b/package/lirc-tools/lirc-tools.mk > > index a5165ad438..e468f2a25d 100644 > > --- a/package/lirc-tools/lirc-tools.mk > > +++ b/package/lirc-tools/lirc-tools.mk > > @@ -46,7 +46,7 @@ LIRC_TOOLS_DEPENDENCIES += libftdi1 > > endif > > > > ifeq ($(BR2_PACKAGE_PYTHON3),y) > > -LIRC_TOOLS_DEPENDENCIES += python3 host-python-setuptools > > +LIRC_TOOLS_DEPENDENCIES += python3 host-python3-setuptools > > LIRC_TOOLS_MAKE_ENV += SETUPTOOLS_ENV="$(PKG_PYTHON_SETUPTOOLS_ENV)" > > endif > > > > diff --git a/package/pkg-python.mk b/package/pkg-python.mk > > index 3dc8a5d10f..b641e3f865 100644 > > --- a/package/pkg-python.mk > > +++ b/package/pkg-python.mk > > @@ -188,16 +188,35 @@ endif > > endif # ($$($(2)_NEEDS_HOST_PYTHON),) > > endif # ($(4),target) > > > > -# Setuptools based packages will need host-python-setuptools (both > > -# host and target). We need to have a special exclusion for the > > -# host-setuptools package itself: it is setuptools-based, but > > -# shouldn't depend on host-setuptools (because it would otherwise > > -# depend on itself!). > > +# Setuptools based packages will need setuptools for the host Python > > +# interpreter (both host and target). > > +# > > +# If we have a host package that says "I need Python 3", we install > > +# setuptools for python3. > > +# > > +# If we have a host packge that says "I need Python 2", we install > > +# setuptools for python2. > > +# > > +# If we have a target package, or a host package that doesn't have any > > +# <pkg>_NEEDS_HOST_PYTHON, and BR2_PACKAGE_PYTHON3 is used, then > > +# Python 3.x is the default Python interpreter, so we install > > +# setuptools for python3. > > +# > > +# In all other cases, we install setuptools for python2. Those other > > +# cases are: a target package or host package with > > +# BR2_PACKAGE_PYTHON=y, or a host-package with neither > > +# BR2_PACKAGE_PYTHON3=y or BR2_PACKAGE_PYTHON=y. > > ifeq ($$($(2)_SETUP_TYPE),setuptools) > > -ifneq ($(2),HOST_PYTHON_SETUPTOOLS) > > -$(2)_DEPENDENCIES += host-python-setuptools > > -endif > > +ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python3) > > +$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools) > > +else ifeq ($(4):$$($(2)_NEEDS_HOST_PYTHON),host:python2) > > +$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools) > > +else ifeq ($$(BR2_PACKAGE_PYTHON3),y) > > +$(2)_DEPENDENCIES += $$(if $$(filter host-python3-setuptools,$(1)),,host-python3-setuptools) > > +else > > +$(2)_DEPENDENCIES += $$(if $$(filter host-python-setuptools,$(1)),,host-python-setuptools) > > endif > > +endif # SETUP_TYPE > > > > # Python interpreter to use for building the package. > > # > > diff --git a/package/python-setuptools/python-setuptools.mk b/package/python-setuptools/python-setuptools.mk > > index cc6f991cdd..d915449c5d 100644 > > --- a/package/python-setuptools/python-setuptools.mk > > +++ b/package/python-setuptools/python-setuptools.mk > > @@ -10,6 +10,7 @@ PYTHON_SETUPTOOLS_SITE = https://files.pythonhosted.org/packages/37/1b/b25507861 > > PYTHON_SETUPTOOLS_LICENSE = MIT > > PYTHON_SETUPTOOLS_LICENSE_FILES = LICENSE > > PYTHON_SETUPTOOLS_SETUP_TYPE = setuptools > > +HOST_PYTHON_SETUPTOOLS_NEEDS_HOST_PYTHON = python2 > > > > define PYTHON_SETUPTOOLS_EXTRACT_CMDS > > $(UNZIP) -d $(@D) $(PYTHON_SETUPTOOLS_DL_DIR)/$(PYTHON_SETUPTOOLS_SOURCE) > > -- > > 2.20.1 > > > > Reviewed-by: Asaf Kahlon <asafka7@gmail.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed 2018-12-28 17:01 ` [Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed Thomas Petazzoni 2018-12-29 15:15 ` Asaf Kahlon @ 2019-01-01 10:31 ` Thomas Petazzoni 1 sibling, 0 replies; 15+ messages in thread From: Thomas Petazzoni @ 2019-01-01 10:31 UTC (permalink / raw) To: buildroot Hello, On Fri, 28 Dec 2018 18:01:30 +0100, Thomas Petazzoni wrote: > When a package uses "setuptools" as its <pkg>_SETUP_TYPE, we currently > add a dependency on host-python-setuptools. This means that: > > (1) When BR2_PACKAGE_PYTHON=y, the default host Python version is > Python 2.x, and host-python-setuptools is installed for > host-python. > > (2) When BR2_PACKAGE_PYTHON3=y, the default host Python version is > Python 3.x, and host-python-setuptools is installed for > host-python3. > > (3) When no target Python interpreter is selected, the default host > Python version is Python 2.x, and host-python-setuptools is > installed for host-python. > > Situations (1) and (3) are problematic for host Python packages that > need Python 3.x. Such packages use <pkg>_NEEDS_HOST_PYTHON = python3, > but if they use setuptools as their setup type, they will not find > setuptools installed for host-python3 in situations (1) and (3) > described above. > > We currently have a single package that sets <pkg>_NEEDS_HOST_PYTHON = > python3: host-meson. host-meson generally works because if setuptools > is not found, it falls back to distutils, which is part of the > standard Python library. However, if there is a setuptools version > installed system-wide, it may be picked up, but may not necessarily be > the same version as Buildroot setuptools, potentially causing > problems. > > This commit makes the necessary change to the python-package > infrastructure to fix this behavior, by identifying the following > cases: > > - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python3, > then we know it wants setuptools installed for host-python3, so we > use host-python3-setuptools. > > - When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python2, > then we known it wants setuptools installed for host-python, so we > use host-python-setuptools. > > - When BR2_PACKAGE_PYTHON3=y, and we have a target package, or a host > package with no NEEDS_HOST_PYTHON option, then we want setuptools > installed for host-python3, so we use host-python3-setuptools. > > - When BR2_PACKAGE_PYTHON=y or no target interpreter is enabled at > all, and we have a target package, or a host package with no > NEEDS_HOST_PYTHON option, then we want setuptools for host-python, > so we use host-python-setuptools. > > To make this happen, we use host-python3-setuptools introduced in a > previous commit, but we also change host-python-setuptools to force > its installation for host-python. The latter is needed if you build > with BR2_PACKAGE_PYTHON3=y but want to install a Python-based package > that has NEEDS_HOST_PYTHON=python2. > > There is one single package that needs be adjusted following this: > lirc-tools, because it is not using the python-package > infrastructure. It directly depends on host-python-setuptools, which > no longer works because host-python-setuptools now only installs for > Python 2.x, while lirc-tools Python binding only supports Python > 3.x. Switching to host-python3-setuptools solves this problem. > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > package/lirc-tools/lirc-tools.mk | 2 +- > package/pkg-python.mk | 35 ++++++++++++++----- > .../python-setuptools/python-setuptools.mk | 1 + > 3 files changed, 29 insertions(+), 9 deletions(-) Applied to master, thanks. Thanks Asaf and Yegor for the review! Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools 2018-12-28 17:01 [Buildroot] [PATCH 0/3] Prepare Python support for top-level parallel build Thomas Petazzoni 2018-12-28 17:01 ` [Buildroot] [PATCH 1/3] package/python3-setuptools: new package Thomas Petazzoni 2018-12-28 17:01 ` [Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed Thomas Petazzoni @ 2018-12-28 17:01 ` Thomas Petazzoni 2018-12-29 15:16 ` Asaf Kahlon 2019-01-01 10:31 ` Thomas Petazzoni 2 siblings, 2 replies; 15+ messages in thread From: Thomas Petazzoni @ 2018-12-28 17:01 UTC (permalink / raw) To: buildroot Currently, host Python packages that use setuptools are installed as Python Eggs, i.e they are installed in $(HOST_DIR)/lib/python2.7/site-packages/Something-<version>.egg. Once installed, each Python Egg is registered to a file called $(HOST_DIR)/lib/python2.7/site-packages/easy-install.pth. This file is read by the Python interpreter so that the installation location of each Egg is added to the Python path, and can be found by the Python interpreter. However, the fact that the installation of different Python modules need to update a common file is clearly not compatible with per-package directories and top-level parallel build. To fix this, we avoid using Python Eggs using the same --single-version-externally-managed option that we use for target modules. This option is normally meant for distributions packaging Python modules, and can therefore only be used if either --record (to record the list of files being installed) or --root is passed. --root=/ works fine and was suggested by https://stackoverflow.com/questions/6301003/stopping-setup-py-from-installing-as-egg/33791008#33791008. With this change, host Python modules installed by setuptools are now installed in the "regular" way, i.e directly in $(HOST_DIR)/lib/python2.7/site-packages/mako for host-python-mako. This makes the installation of host Python modules more similar to the one of target modules, and makes it compatible with per-package directory support and top-level parallel build. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- package/pkg-python.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/package/pkg-python.mk b/package/pkg-python.mk index b641e3f865..4f42e2efc2 100644 --- a/package/pkg-python.mk +++ b/package/pkg-python.mk @@ -85,7 +85,9 @@ HOST_PKG_PYTHON_SETUPTOOLS_ENV = \ PYTHONNOUSERSITE=1 HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \ - --prefix=$(HOST_DIR) + --prefix=$(HOST_DIR) \ + --root=/ \ + --single-version-externally-managed ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define PKG_PYTHON_FIXUP_SYSCONFIGDATA -- 2.20.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools 2018-12-28 17:01 ` [Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools Thomas Petazzoni @ 2018-12-29 15:16 ` Asaf Kahlon 2018-12-31 18:30 ` Yegor Yefremov 2019-01-01 10:31 ` Thomas Petazzoni 1 sibling, 1 reply; 15+ messages in thread From: Asaf Kahlon @ 2018-12-29 15:16 UTC (permalink / raw) To: buildroot Hello, On Fri, Dec 28, 2018 at 7:01 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > Currently, host Python packages that use setuptools are installed as > Python Eggs, i.e they are installed in > $(HOST_DIR)/lib/python2.7/site-packages/Something-<version>.egg. Once > installed, each Python Egg is registered to a file called > $(HOST_DIR)/lib/python2.7/site-packages/easy-install.pth. This file is > read by the Python interpreter so that the installation location of > each Egg is added to the Python path, and can be found by the Python > interpreter. > > However, the fact that the installation of different Python modules > need to update a common file is clearly not compatible with > per-package directories and top-level parallel build. > > To fix this, we avoid using Python Eggs using the same > --single-version-externally-managed option that we use for target > modules. This option is normally meant for distributions packaging > Python modules, and can therefore only be used if either --record (to > record the list of files being installed) or --root is > passed. --root=/ works fine and was suggested by > https://stackoverflow.com/questions/6301003/stopping-setup-py-from-installing-as-egg/33791008#33791008. > > With this change, host Python modules installed by setuptools are now > installed in the "regular" way, i.e directly in > $(HOST_DIR)/lib/python2.7/site-packages/mako for host-python-mako. > > This makes the installation of host Python modules more similar to the > one of target modules, and makes it compatible with per-package > directory support and top-level parallel build. > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > package/pkg-python.mk | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/package/pkg-python.mk b/package/pkg-python.mk > index b641e3f865..4f42e2efc2 100644 > --- a/package/pkg-python.mk > +++ b/package/pkg-python.mk > @@ -85,7 +85,9 @@ HOST_PKG_PYTHON_SETUPTOOLS_ENV = \ > PYTHONNOUSERSITE=1 > > HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \ > - --prefix=$(HOST_DIR) > + --prefix=$(HOST_DIR) \ > + --root=/ \ > + --single-version-externally-managed > > ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) > define PKG_PYTHON_FIXUP_SYSCONFIGDATA > -- > 2.20.1 > Reviewed-by: Asaf Kahlon <asafka7@gmail.com> ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools 2018-12-29 15:16 ` Asaf Kahlon @ 2018-12-31 18:30 ` Yegor Yefremov 0 siblings, 0 replies; 15+ messages in thread From: Yegor Yefremov @ 2018-12-31 18:30 UTC (permalink / raw) To: buildroot On Sat, Dec 29, 2018 at 4:16 PM Asaf Kahlon <asafka7@gmail.com> wrote: > > Hello, > > On Fri, Dec 28, 2018 at 7:01 PM Thomas Petazzoni > <thomas.petazzoni@bootlin.com> wrote: > > > > Currently, host Python packages that use setuptools are installed as > > Python Eggs, i.e they are installed in > > $(HOST_DIR)/lib/python2.7/site-packages/Something-<version>.egg. Once > > installed, each Python Egg is registered to a file called > > $(HOST_DIR)/lib/python2.7/site-packages/easy-install.pth. This file is > > read by the Python interpreter so that the installation location of > > each Egg is added to the Python path, and can be found by the Python > > interpreter. > > > > However, the fact that the installation of different Python modules > > need to update a common file is clearly not compatible with > > per-package directories and top-level parallel build. > > > > To fix this, we avoid using Python Eggs using the same > > --single-version-externally-managed option that we use for target > > modules. This option is normally meant for distributions packaging > > Python modules, and can therefore only be used if either --record (to > > record the list of files being installed) or --root is > > passed. --root=/ works fine and was suggested by > > https://stackoverflow.com/questions/6301003/stopping-setup-py-from-installing-as-egg/33791008#33791008. > > > > With this change, host Python modules installed by setuptools are now > > installed in the "regular" way, i.e directly in > > $(HOST_DIR)/lib/python2.7/site-packages/mako for host-python-mako. > > > > This makes the installation of host Python modules more similar to the > > one of target modules, and makes it compatible with per-package > > directory support and top-level parallel build. > > > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > > --- > > package/pkg-python.mk | 4 +++- > > 1 file changed, 3 insertions(+), 1 deletion(-) > > > > diff --git a/package/pkg-python.mk b/package/pkg-python.mk > > index b641e3f865..4f42e2efc2 100644 > > --- a/package/pkg-python.mk > > +++ b/package/pkg-python.mk > > @@ -85,7 +85,9 @@ HOST_PKG_PYTHON_SETUPTOOLS_ENV = \ > > PYTHONNOUSERSITE=1 > > > > HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \ > > - --prefix=$(HOST_DIR) > > + --prefix=$(HOST_DIR) \ > > + --root=/ \ > > + --single-version-externally-managed > > > > ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) > > define PKG_PYTHON_FIXUP_SYSCONFIGDATA > > -- > > 2.20.1 > > > > Reviewed-by: Asaf Kahlon <asafka7@gmail.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools 2018-12-28 17:01 ` [Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools Thomas Petazzoni 2018-12-29 15:16 ` Asaf Kahlon @ 2019-01-01 10:31 ` Thomas Petazzoni 1 sibling, 0 replies; 15+ messages in thread From: Thomas Petazzoni @ 2019-01-01 10:31 UTC (permalink / raw) To: buildroot Hello, On Fri, 28 Dec 2018 18:01:31 +0100, Thomas Petazzoni wrote: > Currently, host Python packages that use setuptools are installed as > Python Eggs, i.e they are installed in > $(HOST_DIR)/lib/python2.7/site-packages/Something-<version>.egg. Once > installed, each Python Egg is registered to a file called > $(HOST_DIR)/lib/python2.7/site-packages/easy-install.pth. This file is > read by the Python interpreter so that the installation location of > each Egg is added to the Python path, and can be found by the Python > interpreter. > > However, the fact that the installation of different Python modules > need to update a common file is clearly not compatible with > per-package directories and top-level parallel build. > > To fix this, we avoid using Python Eggs using the same > --single-version-externally-managed option that we use for target > modules. This option is normally meant for distributions packaging > Python modules, and can therefore only be used if either --record (to > record the list of files being installed) or --root is > passed. --root=/ works fine and was suggested by > https://stackoverflow.com/questions/6301003/stopping-setup-py-from-installing-as-egg/33791008#33791008. > > With this change, host Python modules installed by setuptools are now > installed in the "regular" way, i.e directly in > $(HOST_DIR)/lib/python2.7/site-packages/mako for host-python-mako. > > This makes the installation of host Python modules more similar to the > one of target modules, and makes it compatible with per-package > directory support and top-level parallel build. > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > package/pkg-python.mk | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) Applied to master, thanks. Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2019-01-14 19:59 UTC | newest] Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-12-28 17:01 [Buildroot] [PATCH 0/3] Prepare Python support for top-level parallel build Thomas Petazzoni 2018-12-28 17:01 ` [Buildroot] [PATCH 1/3] package/python3-setuptools: new package Thomas Petazzoni 2018-12-29 15:13 ` Asaf Kahlon 2018-12-31 18:32 ` Yegor Yefremov 2019-01-01 10:31 ` Thomas Petazzoni 2019-01-14 18:31 ` Arnout Vandecappelle 2019-01-14 19:59 ` Thomas Petazzoni 2018-12-28 17:01 ` [Buildroot] [PATCH 2/3] package/pkg-python: use host-python3-setuptools when needed Thomas Petazzoni 2018-12-29 15:15 ` Asaf Kahlon 2018-12-31 18:29 ` Yegor Yefremov 2019-01-01 10:31 ` Thomas Petazzoni 2018-12-28 17:01 ` [Buildroot] [PATCH 3/3] package/pkg-python: use --single-version-externally-managed for host setuptools Thomas Petazzoni 2018-12-29 15:16 ` Asaf Kahlon 2018-12-31 18:30 ` Yegor Yefremov 2019-01-01 10:31 ` Thomas Petazzoni
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.