[Buildroot] [PATCH v2] Python 3, simple, external package support

[Buildroot] [PATCH v2] Python 3, simple, external package support

[Patrick Gerber]

This patch series refactor python support in buildroot to enable the use of
from line 'This patch series refactor python support in buildroot to enable
the use of external package with both python2 and python3. The packages 
python-bottle and python-serial have adapted as examples.

To support others external package more work is needed as they are based on
python setuptools. It's no more supported by python3 and should be replaced
by "Distribute". This should be done in another patch series.

v2: Add short patch title

[Buildroot] [PATCH 1/5] Python menu reorganization

[Patrick Gerber]

In order to have a better support for python external package make the python
menu clearer. Python2 and Python3 are now mutually exclusive

Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
---
 package/Config.in                |   19 +------------
 package/python-generic/Config.in |   56 ++++++++++++++++++++++++++++++++++++++
 package/python/Config.in         |   14 ----------
 package/python3/Config.in        |   14 ----------
 4 files changed, 57 insertions(+), 46 deletions(-)
 create mode 100644 package/python-generic/Config.in

diff --git a/package/Config.in b/package/Config.in
index 2fad94d..b7da22f 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -315,24 +315,7 @@ endmenu
 endif
 source "package/microperl/Config.in"
 source "package/php/Config.in"
-source "package/python/Config.in"
-source "package/python3/Config.in"
-if BR2_PACKAGE_PYTHON
-menu "external python modules"
-source "package/python-bottle/Config.in"
-source "package/python-dpkt/Config.in"
-source "package/python-id3/Config.in"
-source "package/python-mad/Config.in"
-source "package/python-meld3/Config.in"
-source "package/python-netifaces/Config.in"
-source "package/python-nfc/Config.in"
-source "package/python-protobuf/Config.in"
-source "package/python-pygame/Config.in"
-source "package/python-pyparsing/Config.in"
-source "package/python-serial/Config.in"
-source "package/python-setuptools/Config.in"
-endmenu
-endif
+source "package/python-generic/Config.in"
 source "package/ruby/Config.in"
 source "package/tcl/Config.in"
 if BR2_PACKAGE_TCL
diff --git a/package/python-generic/Config.in b/package/python-generic/Config.in
new file mode 100644
index 0000000..4acd79b
--- /dev/null
+++ b/package/python-generic/Config.in
@@ -0,0 +1,56 @@
+config BR2_PACKAGE_PYTHON_GENERIC
+	bool "Python"
+	depends on BR2_USE_WCHAR
+	# uses fork()
+	depends on BR2_USE_MMU
+	help
+	  The python language interpreter.
+
+	  http://www.python.org/
+
+comment "python3 requires a toolchain with WCHAR support"
+	depends on !BR2_USE_WCHAR
+
+choice
+	depends on BR2_PACKAGE_PYTHON_GENERIC
+	prompt "Python interpreter"
+	help
+	  Select which version of Python you would like to use.
+
+config BR2_PACKAGE_PYTHON
+	bool "python2"
+	select BR2_PACKAGE_LIBFFI
+	help
+	  Use Python 2.x 
+
+
+config BR2_PACKAGE_PYTHON3
+	bool "python3"
+	# uses fork()
+	depends on BR2_USE_MMU
+	select BR2_PACKAGE_LIBFFI
+	help
+	  Use Python 3.x 
+
+endchoice
+
+source "package/python/Config.in"
+source "package/python3/Config.in"
+
+if (BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3)
+menu "external python modules"
+source "package/python-bottle/Config.in"
+source "package/python-dpkt/Config.in"
+source "package/python-id3/Config.in"
+source "package/python-mad/Config.in"
+source "package/python-meld3/Config.in"
+source "package/python-netifaces/Config.in"
+source "package/python-nfc/Config.in"
+source "package/python-protobuf/Config.in"
+source "package/python-pygame/Config.in"
+source "package/python-pyparsing/Config.in"
+source "package/python-serial/Config.in"
+source "package/python-setuptools/Config.in"
+endmenu
+endif
+
diff --git a/package/python/Config.in b/package/python/Config.in
index 4089cce..b4fd6c2 100644
--- a/package/python/Config.in
+++ b/package/python/Config.in
@@ -1,17 +1,3 @@
-config BR2_PACKAGE_PYTHON
-	bool "python"
-	depends on BR2_USE_WCHAR
-	# uses fork()
-	depends on BR2_USE_MMU
-	select BR2_PACKAGE_LIBFFI
-	help
-	  The python language interpreter.
-
-	  http://www.python.org/
-
-comment "python requires a toolchain with WCHAR support"
-	depends on !BR2_USE_WCHAR
-
 if BR2_PACKAGE_PYTHON
 
 choice
diff --git a/package/python3/Config.in b/package/python3/Config.in
index 5959d31..279f31b 100644
--- a/package/python3/Config.in
+++ b/package/python3/Config.in
@@ -1,17 +1,3 @@
-config BR2_PACKAGE_PYTHON3
-	bool "python3"
-	depends on BR2_USE_WCHAR
-	# uses fork()
-	depends on BR2_USE_MMU
-	select BR2_PACKAGE_LIBFFI
-	help
-	  The python language interpreter.
-
-	  http://www.python.org/
-
-comment "python3 requires a toolchain with WCHAR support"
-	depends on !BR2_USE_WCHAR
-
 if BR2_PACKAGE_PYTHON3
 
 choice
-- 
1.7.10.4

[Buildroot] [PATCH 2/5] Add python3 to python simlinks in staging

[Patrick Gerber]

When using python3 a symlink "python" is added in staging and point to
python3. This is needed to call python setup for external package. A
second simlink is added to alias the config-3.3m directory to config

Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
---
 package/python3/python3.mk |    8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/package/python3/python3.mk b/package/python3/python3.mk
index 7551a7b..5974824 100644
--- a/package/python3/python3.mk
+++ b/package/python3/python3.mk
@@ -160,6 +160,14 @@ ifneq ($(BR2_PACKAGE_PYTHON),y)
 PYTHON3_POST_INSTALL_TARGET_HOOKS += PYTHON3_INSTALL_SYMLINK
 endif
 
+define HOST_PYTHON3_INSTALL_SYMLINK
+	ln -fs python3 $(HOST_DIR)/usr/bin/python
+	ln -fs config-3.3m $(HOST_DIR)/usr/lib/python3.3/config
+endef
+
+HOST_PYTHON3_POST_INSTALL_HOOKS += HOST_PYTHON3_INSTALL_SYMLINK
+
+
 ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y)
 define PYTHON3_REMOVE_MODULES_FILES
 	for i in `find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) \
-- 
1.7.10.4

[Buildroot] [PATCH 2/5] Add python3 to python simlinks in staging

[Samuel Martin]

Hi Patrick,

2013/2/12 Patrick Gerber <kpa_info@yahoo.fr>:
> When using python3 a symlink "python" is added in staging and point to
s/staging/host directory/

> python3. This is needed to call python setup for external package. A
> second simlink is added to alias the config-3.3m directory to config
s/simlink/symlink/

>
> Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
> ---
>  package/python3/python3.mk |    8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/package/python3/python3.mk b/package/python3/python3.mk
> index 7551a7b..5974824 100644
> --- a/package/python3/python3.mk
> +++ b/package/python3/python3.mk
> @@ -160,6 +160,14 @@ ifneq ($(BR2_PACKAGE_PYTHON),y)
>  PYTHON3_POST_INSTALL_TARGET_HOOKS += PYTHON3_INSTALL_SYMLINK
>  endif
>
> +define HOST_PYTHON3_INSTALL_SYMLINK
> +       ln -fs python3 $(HOST_DIR)/usr/bin/python
> +       ln -fs config-3.3m $(HOST_DIR)/usr/lib/python3.3/config
> +endef
> +
> +HOST_PYTHON3_POST_INSTALL_HOOKS += HOST_PYTHON3_INSTALL_SYMLINK
BTW, I'm a bit concerned about the consequencies this may have...

From my experience, having python poiting to python3 in the PATH has
some bad effect,
and some patch have been posted some time ago to fix this issue:
http://patchwork.ozlabs.org/patch/208803/

Also, I'm dubious about having python poiting to python3.
Though there is nothing really stated, there is usually a consensus:
"python" refers to the
python-2 interpreter if it exists on the system, otherwise it refers
to the python-3 one (if existing).

What worries me is that, python (i.e. python 2.6 or 2.7) is currently
a mandatory for Buildroot
(see: http://buildroot.org/downloads/manual/manual.html#requirement-mandatory),
and here,
you are overriding it with python3 in the PATH inside Buildroot.

I undersatnd that, when BR2_PACKAGE_PYTHON3_PY_ONLY is enabled, you
want to build
all modules using python3, but meanwhile some other build scripts may
use 'python' referring
to python2 and may badly failed because they will trigger python3...

I may have missed something... so, enlighten me please.


Regards,

-- 
Samuel

[Buildroot] [PATCH 3/5] Add common declaration to ease the support of both version

[Patrick Gerber]

Define a new variable called PYTHON_GENERIC_DEPENDENCIES either to
"python" or "python3" depending of the selected Python version. This
is needed to easily set dependencies for packages that could be used
either with Python 2 or Python 3.

Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
---
 package/python-generic/python-generic.mk |   16 ++++++++++++++++
 1 file changed, 16 insertions(+)
 create mode 100644 package/python-generic/python-generic.mk

diff --git a/package/python-generic/python-generic.mk b/package/python-generic/python-generic.mk
new file mode 100644
index 0000000..0cf0e9d
--- /dev/null
+++ b/package/python-generic/python-generic.mk
@@ -0,0 +1,16 @@
+#############################################################
+#
+# python-generic
+#
+#############################################################
+# PYTHON_GENERIC_DEPENDENCIES is defined to the selected python 
+# version. 
+
+ifeq ($(BR2_PACKAGE_PYTHON),y)
+	PYTHON_GENERIC_DEPENDENCIES = python
+endif
+
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+	PYTHON_GENERIC_DEPENDENCIES = python3
+endif
+
-- 
1.7.10.4

[Buildroot] [PATCH 4/5] python-bottle: Convert package to be used with python2 or python3

[Patrick Gerber]

Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
---
 package/python-bottle/Config.in        |    1 +
 package/python-bottle/python-bottle.mk |    2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/python-bottle/Config.in b/package/python-bottle/Config.in
index f9b60ca..e6f2f3b 100644
--- a/package/python-bottle/Config.in
+++ b/package/python-bottle/Config.in
@@ -1,5 +1,6 @@
 config BR2_PACKAGE_PYTHON_BOTTLE
 	bool "python-bottle"
+	depends on (BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3)
 	help
 	  Bottle is a fast, simple and lightweight WSGI micro web-framework
 	  for Python. It is distributed as a single file module and has no
diff --git a/package/python-bottle/python-bottle.mk b/package/python-bottle/python-bottle.mk
index 370b31b..d24e0ec 100644
--- a/package/python-bottle/python-bottle.mk
+++ b/package/python-bottle/python-bottle.mk
@@ -7,7 +7,7 @@
 PYTHON_BOTTLE_VERSION = 0.11.4
 PYTHON_BOTTLE_SOURCE = bottle-$(PYTHON_BOTTLE_VERSION).tar.gz
 PYTHON_BOTTLE_SITE = http://pypi.python.org/packages/source/b/bottle
-PYTHON_BOTTLE_DEPENDENCIES = python
+PYTHON_BOTTLE_DEPENDENCIES = $(PYTHON_GENERIC_DEPENDENCIES)
 PYTHON_BOTTLE_LICENSE = MIT
 # README.rst refers to the file "LICENSE" but it's not included
 
-- 
1.7.10.4

[Buildroot] [PATCH 4/5] python-bottle: Convert package to be used with python2 or python3

[Thomas Petazzoni]

Dear Patrick Gerber,

On Tue, 12 Feb 2013 15:14:00 +0100, Patrick Gerber wrote:
> 
> Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
> ---
>  package/python-bottle/Config.in        |    1 +
>  package/python-bottle/python-bottle.mk |    2 +-
>  2 files changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/package/python-bottle/Config.in b/package/python-bottle/Config.in
> index f9b60ca..e6f2f3b 100644
> --- a/package/python-bottle/Config.in
> +++ b/package/python-bottle/Config.in
> @@ -1,5 +1,6 @@
>  config BR2_PACKAGE_PYTHON_BOTTLE
>  	bool "python-bottle"
> +	depends on (BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3)

This change is useless, since the python-bottle package is only
available when either BR2_PACKAGE_PYTHON or BR2_PACKAGE_PYTHON3 are
enabled:

+if (BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3)
+menu "external python modules"
+source "package/python-bottle/Config.in"

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH 5/5] python-serial: Convert package to be used with python2 or python3

[Patrick Gerber]

Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
---
 package/python-serial/Config.in        |    2 +-
 package/python-serial/python-serial.mk |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/python-serial/Config.in b/package/python-serial/Config.in
index ea090bd..db6a4b1 100644
--- a/package/python-serial/Config.in
+++ b/package/python-serial/Config.in
@@ -1,6 +1,6 @@
 config BR2_PACKAGE_PYTHON_SERIAL
 	bool "python-serial"
-	depends on BR2_PACKAGE_PYTHON
+	depends on (BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3)
 	help
 	  python-serial is a Python library to access serial ports.
 
diff --git a/package/python-serial/python-serial.mk b/package/python-serial/python-serial.mk
index fb6b31c..6f2bb85 100644
--- a/package/python-serial/python-serial.mk
+++ b/package/python-serial/python-serial.mk
@@ -8,7 +8,7 @@ PYTHON_SERIAL_VERSION = 2.6
 PYTHON_SERIAL_SOURCE  = pyserial-$(PYTHON_SERIAL_VERSION).tar.gz
 PYTHON_SERIAL_SITE    = http://pypi.python.org/packages/source/p/pyserial/
 
-PYTHON_SERIAL_DEPENDENCIES = python
+PYTHON_SERIAL_DEPENDENCIES = $(PYTHON_GENERIC_DEPENDENCIES)
 
 define PYTHON_SERIAL_BUILD_CMDS
 	(cd $(@D); $(HOST_DIR)/usr/bin/python setup.py build)
-- 
1.7.10.4

[Buildroot] [PATCH v2] Python 3, simple, external package support

[Thomas Petazzoni]

Dear Patrick Gerber,

On Tue, 12 Feb 2013 15:13:56 +0100, Patrick Gerber wrote:
> This patch series refactor python support in buildroot to enable the use of
> from line 'This patch series refactor python support in buildroot to enable
> the use of external package with both python2 and python3. The packages 
> python-bottle and python-serial have adapted as examples.
> 
> To support others external package more work is needed as they are based on
> python setuptools. It's no more supported by python3 and should be replaced
> by "Distribute". This should be done in another patch series.

Thanks, but it still doesn't answer the question:

"""
As Samuel pointed it out, this unfortunately may break a number of
things in Buildroot. They are packages that depend on host-python, and
would only be capable of using an Python 2.x host-python.

Or at least, I would like to have the confirmation that all packages
that depend on host-python continue to build normally when host-python
is Python 3.

That said, some packages are not really clear about this. For example,
Scons uses the Python provided with the distribution, and not the
host-python built by Buildroot.

As you can see, this is not a trivial topic.
"""

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH v2] Python 3, simple, external package support

[Patrick]

On 02/12/2013 03:21 PM, Thomas Petazzoni wrote:
> Dear Patrick Gerber,
>
> On Tue, 12 Feb 2013 15:13:56 +0100, Patrick Gerber wrote:
>> This patch series refactor python support in buildroot to enable the use of
>> from line 'This patch series refactor python support in buildroot to enable
>> the use of external package with both python2 and python3. The packages
>> python-bottle and python-serial have adapted as examples.
>>
>> To support others external package more work is needed as they are based on
>> python setuptools. It's no more supported by python3 and should be replaced
>> by "Distribute". This should be done in another patch series.
>
> Thanks, but it still doesn't answer the question:
>
> """
> As Samuel pointed it out, this unfortunately may break a number of
> things in Buildroot. They are packages that depend on host-python, and
> would only be capable of using an Python 2.x host-python.
>
> Or at least, I would like to have the confirmation that all packages
> that depend on host-python continue to build normally when host-python
> is Python 3.
>
> That said, some packages are not really clear about this. For example,
> Scons uses the Python provided with the distribution, and not the
> host-python built by Buildroot.
>
> As you can see, this is not a trivial topic.
> """
>
> Best regards,
>
> Thomas
>

Related with host-python, the only difference is that now the patch add 
a symlink in  HOST_DIR. I agree that this could be a problem.

I could edit the external python package to use "python" or "python3" 
depending of the selected version. This will not impact host-python at 
all. Does it seems good to you ?

Patrick

[Buildroot] [PATCH v2] Python 3, simple, external package support

[Thomas Petazzoni]

Dear Patrick,

On Tue, 12 Feb 2013 15:42:54 +0100, Patrick wrote:

> Related with host-python, the only difference is that now the patch add 
> a symlink in  HOST_DIR. I agree that this could be a problem.
> 
> I could edit the external python package to use "python" or "python3" 
> depending of the selected version. This will not impact host-python at 
> all. Does it seems good to you ?

No, it doesn't solve the problem of packages that currently depend on
host-python, and that expect a Python 2 to be installed.

You seem to be focused only on Python external modules, but you
completely forget that there are packages in Buildroot that depend on
host-python as their build dependencies, even though those packages are
not directly Python related.

For example, are you sure that mesa3d and libxcb will built properly
once host-python is Python 3 ?

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH v2] Python 3, simple, external package support

[Patrick]

On 02/12/2013 03:53 PM, Thomas Petazzoni wrote:
> Dear Patrick,
>
> On Tue, 12 Feb 2013 15:42:54 +0100, Patrick wrote:
>
>> Related with host-python, the only difference is that now the patch add
>> a symlink in  HOST_DIR. I agree that this could be a problem.
>>
>> I could edit the external python package to use "python" or "python3"
>> depending of the selected version. This will not impact host-python at
>> all. Does it seems good to you ?
>
> No, it doesn't solve the problem of packages that currently depend on
> host-python, and that expect a Python 2 to be installed.
>
> You seem to be focused only on Python external modules, but you
> completely forget that there are packages in Buildroot that depend on
> host-python as their build dependencies, even though those packages are
> not directly Python related.
>
> For example, are you sure that mesa3d and libxcb will built properly
> once host-python is Python 3 ?
>
> Thomas
>
Thomas, I am aware that there is others packages that depends on 
host-python, mostly x11. But I have not changed the way host-python nor 
host-python3 are build and used. I don't understand how the changes I 
have made could break this compared as previously ? If I'm not mistaken, 
host-python is selected directly by the package (mesa3d for example) 
when needed, and this is independent of the selected python package, right ?

Patrick

[Buildroot] [PATCH v2] Python 3, simple, external package support

[Thomas Petazzoni]

Dear Patrick,

On Tue, 12 Feb 2013 16:20:53 +0100, Patrick wrote:

> Thomas, I am aware that there is others packages that depends on 
> host-python, mostly x11. But I have not changed the way host-python nor 
> host-python3 are build and used. I don't understand how the changes I 
> have made could break this compared as previously ? If I'm not mistaken, 
> host-python is selected directly by the package (mesa3d for example) 
> when needed, and this is independent of the selected python package, right ?

True.

So we might end up with two versions of Python installed in
$(HOST_DIR): python2 and python3. Hopefully, it should work.

Regarding the patch set, I think I would prefer to rename the current
python package to python2, and have the 'python' package be the virtual
package that provides either python2 or python3. That's indeed more
work, but I find the name "python-generic" not so great.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH v2] Python 3, simple, external package support

[Patrick]

Dear Thomas,

On 02/12/2013 04:29 PM, Thomas Petazzoni wrote:
> Dear Patrick,
>
> On Tue, 12 Feb 2013 16:20:53 +0100, Patrick wrote:
>
>> Thomas, I am aware that there is others packages that depends on
>> host-python, mostly x11. But I have not changed the way host-python nor
>> host-python3 are build and used. I don't understand how the changes I
>> have made could break this compared as previously ? If I'm not mistaken,
>> host-python is selected directly by the package (mesa3d for example)
>> when needed, and this is independent of the selected python package, right ?
>
> True.
>
> So we might end up with two versions of Python installed in
> $(HOST_DIR): python2 and python3. Hopefully, it should work.
>

This is already the case if I enable python3, no ?

> Regarding the patch set, I think I would prefer to rename the current
> python package to python2, and have the 'python' package be the virtual
> package that provides either python2 or python3. That's indeed more
> work, but I find the name "python-generic" not so great.

I have tried like you suggest, but then I have to rename python.mk to 
python2.mk and then any occurrence of python or host-python to python2 
and host-python2 in many package dependencies, no ?

But I could rename python-generic to python-common or something like that ?