[Buildroot] [PATCH v2 1/2] package/ninja: invoke python3 explicitly to configure the package

[Buildroot] [PATCH v2 1/2] package/ninja: invoke python3 explicitly to configure the package

[Avi Shukron]

ninja depends on python3 specifically, but the configure.py file
simply uses "env python". Where no python is selected for the target
you simply won't get a python symlink in $(HOST_DIR)/usr/bin, so the
configure.py script fails to run since it can't find "python".

Notice that in order to reproduce the issue, you must not have
python2 installed on your host machine.

Signed-off-by: Avi Shukron <avraham.shukron@gmail.com>
---
 package/ninja/ninja.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/ninja/ninja.mk b/package/ninja/ninja.mk
index 4897012150..281cd70412 100644
--- a/package/ninja/ninja.mk
+++ b/package/ninja/ninja.mk
@@ -17,7 +17,7 @@ NINJA_LICENSE_FILES = COPYING
 HOST_NINJA_DEPENDENCIES = host-python3
 
 define HOST_NINJA_BUILD_CMDS
-	(cd $(@D); ./configure.py --bootstrap)
+	(cd $(@D); $(HOST_DIR)/bin/python3 configure.py --bootstrap)
 endef
 
 define HOST_NINJA_INSTALL_CMDS
-- 
2.17.1

[Buildroot] [PATCH v2 2/2] package/qemu: add host-python as an explicit dependency

[Avi Shukron]

qemu requires python in its configre script. Yet host-python was
not listed as one of the package's dependencies. If no other package
requested host-python, then configuring this package will fail since
it won't find any executable named python in the host dir.

In order to reproduce this issue you must not have python2 installed
on your host machine.

Signed-off-by: Avi Shukron <avraham.shukron@gmail.com>
---
 package/qemu/qemu.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/qemu/qemu.mk b/package/qemu/qemu.mk
index a91f9d3eeb..00d413b68d 100644
--- a/package/qemu/qemu.mk
+++ b/package/qemu/qemu.mk
@@ -21,7 +21,7 @@ QEMU_LICENSE_FILES = COPYING COPYING.LIB
 #-------------------------------------------------------------
 # Target-qemu
 
-QEMU_DEPENDENCIES = host-pkgconf libglib2 zlib pixman
+QEMU_DEPENDENCIES = host-pkgconf libglib2 zlib pixman host-python
 
 # Need the LIBS variable because librt and libm are
 # not automatically pulled. :-(
@@ -182,7 +182,7 @@ $(eval $(generic-package))
 #-------------------------------------------------------------
 # Host-qemu
 
-HOST_QEMU_DEPENDENCIES = host-pkgconf host-zlib host-libglib2 host-pixman
+HOST_QEMU_DEPENDENCIES = host-pkgconf host-zlib host-libglib2 host-pixman host-python
 
 #       BR ARCH         qemu
 #       -------         ----
-- 
2.17.1

[Buildroot] [PATCH v2 1/2] package/ninja: invoke python3 explicitly to configure the package

[Thomas Petazzoni]

On Mon, 11 Nov 2019 23:31:45 +0200
Avi Shukron <avraham.shukron@gmail.com> wrote:

> ninja depends on python3 specifically, but the configure.py file
> simply uses "env python". Where no python is selected for the target
> you simply won't get a python symlink in $(HOST_DIR)/usr/bin, so the
> configure.py script fails to run since it can't find "python".
> 
> Notice that in order to reproduce the issue, you must not have
> python2 installed on your host machine.
> 
> Signed-off-by: Avi Shukron <avraham.shukron@gmail.com>
> ---
>  package/ninja/ninja.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v2 2/2] package/qemu: add host-python as an explicit dependency

[Thomas Petazzoni]

On Mon, 11 Nov 2019 23:31:46 +0200
Avi Shukron <avraham.shukron@gmail.com> wrote:

> qemu requires python in its configre script. Yet host-python was
> not listed as one of the package's dependencies. If no other package
> requested host-python, then configuring this package will fail since
> it won't find any executable named python in the host dir.
> 
> In order to reproduce this issue you must not have python2 installed
> on your host machine.
> 
> Signed-off-by: Avi Shukron <avraham.shukron@gmail.com>

qemu needs python 2 *or* python 3, so we can do something like this:

QEMU_DEPENDENCIES = host-pkgconf libglib2 zlib pixman $(if $(BR2_PACKAGE_PYTHON3),host-python3,host-python)

This way, if you have Python 3.x enabled for the target (and therefore
you are already building host-python3), it will use host-python3.

Could you test this in a configuration that has BR2_PACKAGE_QEMU=y and
BR2_PACKAGE_PYTHON3=y, and check it works properly ?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v2 2/2] package/qemu: add host-python as an explicit dependency

[Avraham Shukron]

On Tue, Nov 12, 2019 at 11:05 PM Thomas Petazzoni <
thomas.petazzoni@bootlin.com> wrote:
>
>
> qemu needs python 2 *or* python 3, so we can do something like this:
>
> QEMU_DEPENDENCIES = host-pkgconf libglib2 zlib pixman $(if
$(BR2_PACKAGE_PYTHON3),host-python3,host-python)
>
> This way, if you have Python 3.x enabled for the target (and therefore
> you are already building host-python3), it will use host-python3.
>
> Could you test this in a configuration that has BR2_PACKAGE_QEMU=y and
> BR2_PACKAGE_PYTHON3=y, and check it works properly ?

Your suggestion works. If python3 is selected - host-python3 is used.
BUT - host-qemu also depends on host-libglib2 which depends on host-meson
which depends on python3,
so even if you **don't** select python3 you end up with host-python3
because of meson, and also
host-python because now host-qemu directly depends on it. Attached a
dependency graph depicting
the scenario.

So - since host-python3 will be used anyway - I suggest we just depend on
it exclusively.
What do you think?

BTW - I'm pretty sure that there aren't really so many packages that truly
depends on **host**-python2 (except maybe python2 itself)
So in general I think it could be a good idea to just replace **all**
host-python with host-python3 (again - except python2 itself) and
just see what happens.
It might just work.

Avi.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20191113/a331bcf6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: host-qemu-graph-depends.png
Type: image/png
Size: 203002 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20191113/a331bcf6/attachment-0001.png>

[Buildroot] [PATCH v3] package/qemu: add host-python as an explicit dependency

[Avi Shukron]

qemu requires python in its configre script. Yet host-python was
not listed as one of the package's dependencies. If no other package
requested host-python, then configuring this package will fail since
it won't find any executable named python in the host dir.

In order to reproduce this issue you must not have python2 installed
on your host machine.

Signed-off-by: Avi Shukron <avraham.shukron@gmail.com>
---
Changes v2->v3:
- Depend directly on host-python3, since it is already a transitive
  dependency of qemu
---
 package/qemu/qemu.mk | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/package/qemu/qemu.mk b/package/qemu/qemu.mk
index a91f9d3eeb..c46e41919f 100644
--- a/package/qemu/qemu.mk
+++ b/package/qemu/qemu.mk
@@ -20,8 +20,7 @@ QEMU_LICENSE_FILES = COPYING COPYING.LIB
 
 #-------------------------------------------------------------
 # Target-qemu
-
-QEMU_DEPENDENCIES = host-pkgconf libglib2 zlib pixman
+QEMU_DEPENDENCIES = host-pkgconf libglib2 zlib pixman host-python3
 
 # Need the LIBS variable because librt and libm are
 # not automatically pulled. :-(
@@ -182,7 +181,7 @@ $(eval $(generic-package))
 #-------------------------------------------------------------
 # Host-qemu
 
-HOST_QEMU_DEPENDENCIES = host-pkgconf host-zlib host-libglib2 host-pixman
+HOST_QEMU_DEPENDENCIES = host-pkgconf host-zlib host-libglib2 host-pixman host-python3
 
 #       BR ARCH         qemu
 #       -------         ----
-- 
2.17.1

[Buildroot] [PATCH v2 2/2] package/qemu: add host-python as an explicit dependency

[Thomas Petazzoni]

Hello,

On Wed, 13 Nov 2019 22:22:00 +0200
Avraham Shukron <avraham.shukron@gmail.com> wrote:

> Your suggestion works. If python3 is selected - host-python3 is used.
> BUT - host-qemu also depends on host-libglib2 which depends on host-meson
> which depends on python3,

True. But then, how did you get build issues when building qemu ? I
mean, your v3 in fact doesn't do anything really: host-python3 was
already built as one of the dependencies of qemu, through libglib2. So
how did you get issues ?

> BTW - I'm pretty sure that there aren't really so many packages that truly
> depends on **host**-python2 (except maybe python2 itself)
> So in general I think it could be a good idea to just replace **all**
> host-python with host-python3 (again - except python2 itself) and
> just see what happens.

We discussed this at the latest Buildroot Developers meeting, you can
see some notes in "Python 2 deprecation" at
https://bimestriel.framapad.org/p/buildroot-elce-2019-meeting. We
welcome contributions in this area.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v2 2/2] package/qemu: add host-python as an explicit dependency

[Avraham Shukron]

On Fri, Nov 15, 2019 at 11:08 PM Thomas Petazzoni <
thomas.petazzoni@bootlin.com> wrote:
> True. But then, how did you get build issues when building qemu ? I
> mean, your v3 in fact doesn't do anything really: host-python3 was
> already built as one of the dependencies of qemu, through libglib2. So
> how did you get issues ?

Damn, you are right, host-python3 won't work.
I've managed to build it only because I had host-python(2) still installed
in my host dir during my experiments.
A clean build still fails because there is no "python" executable in the
host dir unless you build any python for
the target.

So we only left with your suggestion, that in the best case (python3 is
selected for the target)
will only build host-python3.
Any other case will force both host-python and host-python3 to be built.

So I'll re-submit with your $(if..) solution soon.

> We discussed this at the latest Buildroot Developers meeting, you can
> see some notes in "Python 2 deprecation" at
> https://bimestriel.framapad.org/p/buildroot-elce-2019-meeting. We
> welcome contributions in this area.

Great, I'll try to put  some work into that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20191116/34ead835/attachment.html>

[Buildroot] [PATCH v2 2/2] package/qemu: add host-python as an explicit dependency

[Thomas Petazzoni]

Hello,

On Sat, 16 Nov 2019 00:04:59 +0200
Avraham Shukron <avraham.shukron@gmail.com> wrote:

> Damn, you are right, host-python3 won't work.
> I've managed to build it only because I had host-python(2) still installed
> in my host dir during my experiments.
> A clean build still fails because there is no "python" executable in the
> host dir unless you build any python for
> the target.
> 
> So we only left with your suggestion, that in the best case (python3 is
> selected for the target)
> will only build host-python3.
> Any other case will force both host-python and host-python3 to be built.
> 
> So I'll re-submit with your $(if..) solution soon.

So, the qemu build system requires having an interpreter called "python" ?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH v2 2/2] package/qemu: add host-python as an explicit dependency

[Avraham Shukron]

On Sat, Nov 16, 2019 at 12:51 AM Thomas Petazzoni <
thomas.petazzoni@bootlin.com> wrote:

> So, the qemu build system requires having an interpreter called "python" ?
>

Yes.
But now I see that I can pass --python to the configure script with
the path to the interpreter, so I can simply pass $(HOST_DIR)/bin/python3
and
be done with it...
One less package depending on host-python(2).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20191116/9e191f39/attachment.html>

[Buildroot] [PATCH v4] package/qemu: add host-python3 as an explicit dependency

[Avi Shukron]

qemu requires python in its configre script. Yet host-python was
not listed as one of the package's dependencies. If no other package
requested host-python, then configuring this package will fail since
it won't find any executable named python in the host dir.

In order to reproduce this issue you must not have python2 installed
on your host machine.

Signed-off-by: Avi Shukron <avraham.shukron@gmail.com>
---
Changes v2->v3:
 - Depend directly on host-python3, since it is already a transitive
   dependency of qemu

Changes v3->v4:
 - Pass python3 explicitly to the configure script
---
 package/qemu/qemu.mk | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/package/qemu/qemu.mk b/package/qemu/qemu.mk
index a91f9d3eeb..283577d749 100644
--- a/package/qemu/qemu.mk
+++ b/package/qemu/qemu.mk
@@ -20,8 +20,7 @@ QEMU_LICENSE_FILES = COPYING COPYING.LIB
 
 #-------------------------------------------------------------
 # Target-qemu
-
-QEMU_DEPENDENCIES = host-pkgconf libglib2 zlib pixman
+QEMU_DEPENDENCIES = host-pkgconf libglib2 zlib pixman host-python3
 
 # Need the LIBS variable because librt and libm are
 # not automatically pulled. :-(
@@ -132,6 +131,7 @@ define QEMU_CONFIGURE_CMDS
 			--prefix=/usr \
 			--cross-prefix=$(TARGET_CROSS) \
 			--audio-drv-list= \
+			--python=$(HOST_DIR)/bin/python3 \
 			--enable-kvm \
 			--enable-attr \
 			--enable-vhost-net \
@@ -182,7 +182,7 @@ $(eval $(generic-package))
 #-------------------------------------------------------------
 # Host-qemu
 
-HOST_QEMU_DEPENDENCIES = host-pkgconf host-zlib host-libglib2 host-pixman
+HOST_QEMU_DEPENDENCIES = host-pkgconf host-zlib host-libglib2 host-pixman host-python3
 
 #       BR ARCH         qemu
 #       -------         ----
@@ -299,6 +299,7 @@ define HOST_QEMU_CONFIGURE_CMDS
 		--host-cc="$(HOSTCC)" \
 		--extra-cflags="$(HOST_CFLAGS)" \
 		--extra-ldflags="$(HOST_LDFLAGS)" \
+		--python=$(HOST_DIR)/bin/python3 \
 		$(HOST_QEMU_OPTS)
 endef
 
-- 
2.17.1

[Buildroot] [PATCH v2 1/2] package/ninja: invoke python3 explicitly to configure the package

[Peter Korsgaard]

>>>>> "Avi" == Avi Shukron <avraham.shukron@gmail.com> writes:

 > ninja depends on python3 specifically, but the configure.py file
 > simply uses "env python". Where no python is selected for the target
 > you simply won't get a python symlink in $(HOST_DIR)/usr/bin, so the
 > configure.py script fails to run since it can't find "python".

 > Notice that in order to reproduce the issue, you must not have
 > python2 installed on your host machine.

 > Signed-off-by: Avi Shukron <avraham.shukron@gmail.com>

Committed to 2019.08.x, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v3] package/qemu: add host-python as an explicit dependency

[Thomas Petazzoni]

On Fri, 15 Nov 2019 22:19:00 +0200
Avi Shukron <avraham.shukron@gmail.com> wrote:

> qemu requires python in its configre script. Yet host-python was
> not listed as one of the package's dependencies. If no other package
> requested host-python, then configuring this package will fail since
> it won't find any executable named python in the host dir.
> 
> In order to reproduce this issue you must not have python2 installed
> on your host machine.
> 
> Signed-off-by: Avi Shukron <avraham.shukron@gmail.com>
> ---
> Changes v2->v3:
> - Depend directly on host-python3, since it is already a transitive
>   dependency of qemu
> ---
>  package/qemu/qemu.mk | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com