[master][RFC] Adding back wrapper and using OEPYTHON3HOME variable for python3

[master][RFC] Adding back wrapper and using OEPYTHON3HOME variable for python3

[Jaewon Lee]

Adding back the python wrapper and adding a patch to use OEPYTHON3HOME
instead of PYTHONHOME if set, for python3.

If we add back the wrapper as is, we would see the following error that
we also see in Thud:

ImportError: No module named site
OpenEmbedded requires 'python' to be python v2 (>= 2.7.3), not python
v3.
Please upgrade your python v2

This is because python3 would've set PYTHONHOME to use nativesdk
python3 libraries but when the oe-buildenv-internal script tries to call
python2 for the py_v27_check, there will be no python2 libraries in the
PYTHONHOME directory.
In other words, bitbake needs host python2 and the env variable set from
the wrapper contaminates the env and host python2 won't be able to find
its libraries

Creating another variable OEPYTHON3HOME and using this in the python3
wrapper to allow for a way to set a different paths for python3 and
python2

[YOCTO #13208]

Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
 ...EPYTHON3HOME-is-set-use-instead-of-PYTHON.patch | 35 ++++++++++++++++++++++
 meta/recipes-devtools/python/python3_3.7.3.bb      |  7 +++++
 2 files changed, 42 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3/0001-main.c-if-OEPYTHON3HOME-is-set-use-instead-of-PYTHON.patch

diff --git a/meta/recipes-devtools/python/python3/0001-main.c-if-OEPYTHON3HOME-is-set-use-instead-of-PYTHON.patch b/meta/recipes-devtools/python/python3/0001-main.c-if-OEPYTHON3HOME-is-set-use-instead-of-PYTHON.patch
new file mode 100644
index 0000000..12aeab9
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-main.c-if-OEPYTHON3HOME-is-set-use-instead-of-PYTHON.patch
@@ -0,0 +1,35 @@
+From e4363ca1d84b4014184a79a847fb2affb3dfe86e Mon Sep 17 00:00:00 2001
+From: Jaewon Lee <jaewon.lee@xilinx.com>
+Date: Tue, 23 Apr 2019 17:01:08 -0700
+Subject: [PATCH] main.c: if OEPYTHON3HOME is set use instead of PYTHONHOME
+
+There is one variable PYTHONHOME to determine where libraries are coming
+from for both python2 and python3. This becomes an issue if only one has
+libraries in the specified PYTHONHOME path, but they are using the same
+PYTHONHOME. Creating another variable OEPYTHON3HOME to allow for a way
+to set a different path for python3
+
+Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com>
+---
+ Modules/main.c | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/Modules/main.c b/Modules/main.c
+index a745381..25ca435 100644
+--- a/Modules/main.c
++++ b/Modules/main.c
+@@ -1857,6 +1857,11 @@ config_init_home(_PyCoreConfig *config)
+     }
+ 
+     int res = config_get_env_var_dup(&home, L"PYTHONHOME", "PYTHONHOME");
++
++    const char *oepython3home = config_get_env_var("OEPYTHON3HOME");
++    if (oepython3home) {
++        res = config_get_env_var_dup(&home, L"OEPYTHON3HOME", "OEPYTHON3HOME");
++    }
+     if (res < 0) {
+         return DECODE_LOCALE_ERR("PYTHONHOME", res);
+     }
+-- 
+2.7.4
+
diff --git a/meta/recipes-devtools/python/python3_3.7.3.bb b/meta/recipes-devtools/python/python3_3.7.3.bb
index ea46b05..af7ede1 100644
--- a/meta/recipes-devtools/python/python3_3.7.3.bb
+++ b/meta/recipes-devtools/python/python3_3.7.3.bb
@@ -28,6 +28,9 @@ SRC_URI_append_class-native = " \
            file://0001-distutils-sysconfig-append-STAGING_LIBDIR-python-sys.patch \
            file://12-distutils-prefix-is-inside-staging-area.patch \
            "
+SRC_URI_append_class-nativesdk = " \
+           file://0001-main.c-if-OEPYTHON3HOME-is-set-use-instead-of-PYTHON.patch \
+           "
 
 SRC_URI[md5sum] = "93df27aec0cd18d6d42173e601ffbbfd"
 SRC_URI[sha256sum] = "da60b54064d4cfcd9c26576f6df2690e62085123826cff2e667e72a91952d318"
@@ -131,6 +134,10 @@ do_install_append() {
                 ${D}${libdir}/python-sysconfigdata/_sysconfigdata.py
 }
 
+do_install_append_class-nativesdk () {
+    create_wrapper ${D}${bindir}/python${PYTHON_MAJMIN} OEPYTHON3HOME='${prefix}' TERMINFO_DIRS='${sysconfdir}/terminfo:/etc/terminfo:/usr/share/terminfo:/usr/share/misc/terminfo:/lib/terminfo' PYTHONNOUSERSITE='1'
+}
+
 SSTATE_SCAN_FILES += "Makefile _sysconfigdata.py"
 PACKAGE_PREPROCESS_FUNCS += "py_package_preprocess"
 
-- 
2.7.4

Re: [master][RFC] Adding back wrapper and using OEPYTHON3HOME variable for python3

[richard.purdie]

On Thu, 2019-04-25 at 12:56 -0700, Jaewon Lee wrote:
> +diff --git a/Modules/main.c b/Modules/main.c
> +index a745381..25ca435 100644
> +--- a/Modules/main.c
> ++++ b/Modules/main.c
> +@@ -1857,6 +1857,11 @@ config_init_home(_PyCoreConfig *config)
> +     }
> + 
> +     int res = config_get_env_var_dup(&home, L"PYTHONHOME", "PYTHONHOME");
> ++
> ++    const char *oepython3home = config_get_env_var("OEPYTHON3HOME");
> ++    if (oepython3home) {
> ++        res = config_get_env_var_dup(&home, L"OEPYTHON3HOME", "OEPYTHON3HOME");
> ++    }
> +     if (res < 0) {
> +         return DECODE_LOCALE_ERR("PYTHONHOME", res);
> +     }
> +-- 
> +2.7.4

I think the above will leak memory.

Instead I think the code should be something like:

int res;
const char *oepython3home = config_get_env_var("OEPYTHON3HOME");
if (oepython3home) {
    res = config_get_env_var_dup(&home, L"OEPYTHON3HOME", "OEPYTHON3HOME");
    if (res < 0)
        return DECODE_LOCALE_ERR("OEPYTHON3HOME", res);
} else {
    res = config_get_env_var_dup(&home, L"PYTHONHOME", "PYTHONHOME");
    if (res < 0)
        return DECODE_LOCALE_ERR("PYTHONHOME", res);
}

and then a copy of PYTHONHOME isn't created in the OEPYTHON3HOME case.

Cheers,

Richard

Re: [master][RFC] Adding back wrapper and using OEPYTHON3HOME variable for python3

[Jaewon Lee]

Hi Richard,

Agreed, will send a v2

Thanks,
Jaewon

-----Original Message-----
From: richard.purdie@linuxfoundation.org <richard.purdie@linuxfoundation.org> 
Sent: Thursday, April 25, 2019 1:50 PM
To: Jaewon Lee <JAEWON@xilinx.com>; openembedded-core@lists.openembedded.org; Manjukumar Harthikote Matha <MANJUKUM@xilinx.com>; Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Subject: Re: [OE-core][master][RFC] Adding back wrapper and using OEPYTHON3HOME variable for python3

On Thu, 2019-04-25 at 12:56 -0700, Jaewon Lee wrote:
> +diff --git a/Modules/main.c b/Modules/main.c index a745381..25ca435 
> +100644
> +--- a/Modules/main.c
> ++++ b/Modules/main.c
> +@@ -1857,6 +1857,11 @@ config_init_home(_PyCoreConfig *config)
> +     }
> + 
> +     int res = config_get_env_var_dup(&home, L"PYTHONHOME", 
> + "PYTHONHOME");
> ++
> ++    const char *oepython3home = config_get_env_var("OEPYTHON3HOME");
> ++    if (oepython3home) {
> ++        res = config_get_env_var_dup(&home, L"OEPYTHON3HOME", "OEPYTHON3HOME");
> ++    }
> +     if (res < 0) {
> +         return DECODE_LOCALE_ERR("PYTHONHOME", res);
> +     }
> +--
> +2.7.4

I think the above will leak memory.

Instead I think the code should be something like:

int res;
const char *oepython3home = config_get_env_var("OEPYTHON3HOME");
if (oepython3home) {
    res = config_get_env_var_dup(&home, L"OEPYTHON3HOME", "OEPYTHON3HOME");
    if (res < 0)
        return DECODE_LOCALE_ERR("OEPYTHON3HOME", res); } else {
    res = config_get_env_var_dup(&home, L"PYTHONHOME", "PYTHONHOME");
    if (res < 0)
        return DECODE_LOCALE_ERR("PYTHONHOME", res); }

and then a copy of PYTHONHOME isn't created in the OEPYTHON3HOME case.

Cheers,

Richard