[PATCH] distutils: Run python from the PATH in the -native case as well

All of lore.kernel.org
 help / color / mirror / Atom feed

* [PATCH] distutils: Run python from the PATH in the -native case as well
@ 2019-04-30  2:34 Douglas Royds
  2019-04-30  8:37 ` Richard Purdie
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Douglas Royds @ 2019-04-30  2:34 UTC (permalink / raw)
  To: openembedded-core

The python distutils generate a python wrapper script for each package.
These python scripts have shebang lines pointing to the python executable.
In our case, this is a fully-qualified path to python-native in the
recipe-sysroot-native.

Ubuntu 18.04 restricts the useful length of the shebang line to 125
characters, and Ubuntu 16.04 restricts it to 77. In both cases, the
staged python script fails to run due to the length of the path to
the python-native executable.

Replace the shebang line with:

    #!/usr/bin/env python

We were already doing this for on-target distutils components.
This change applies the sed-line to -native distutils components as
well.

The python executable must be in the PATH.
Client components must inherit pythonnative.

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/distutils.bbclass | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.bbclass
index 9862731493..e7d79271e3 100644
--- a/meta/classes/distutils.bbclass
+++ b/meta/classes/distutils.bbclass
@@ -53,18 +53,14 @@ distutils_do_install() {
 
         if test -e ${D}${bindir} ; then	
             for i in ${D}${bindir}/* ; do \
-                if [ ${PN} != "${BPN}-native" ]; then
-			sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
-		fi
+                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
                 sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
             done
         fi
 
         if [ -e ${D}${sbindir} ]; then
             for i in ${D}${sbindir}/* ; do \
-                if [ ${PN} != "${BPN}-native" ]; then
-			sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
-		fi
+                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
                 sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
             done
         fi
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [PATCH] distutils: Run python from the PATH in the -native case as well
  2019-04-30  2:34 [PATCH] distutils: Run python from the PATH in the -native case as well Douglas Royds
@ 2019-04-30  8:37 ` Richard Purdie
  2019-05-03  2:23 ` [PATCH v2] " Douglas Royds
  2019-05-06  6:47 ` [PATCH v3] " Douglas Royds
  2 siblings, 0 replies; 4+ messages in thread
From: Richard Purdie @ 2019-04-30  8:37 UTC (permalink / raw)
  To: Douglas Royds, openembedded-core

On Tue, 2019-04-30 at 14:34 +1200, Douglas Royds via Openembedded-core wrote:
> The python distutils generate a python wrapper script for each package.
> These python scripts have shebang lines pointing to the python executable.
> In our case, this is a fully-qualified path to python-native in the
> recipe-sysroot-native.
> 
> Ubuntu 18.04 restricts the useful length of the shebang line to 125
> characters, and Ubuntu 16.04 restricts it to 77. In both cases, the
> staged python script fails to run due to the length of the path to
> the python-native executable.
> 
> Replace the shebang line with:
> 
>     #!/usr/bin/env python
> 
> We were already doing this for on-target distutils components.
> This change applies the sed-line to -native distutils components as
> well.
> 
> The python executable must be in the PATH.
> Client components must inherit pythonnative.
> 
> Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
> ---
>  meta/classes/distutils.bbclass | 8 ++------
>  1 file changed, 2 insertions(+), 6 deletions(-)
> 
> diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.bbclass
> index 9862731493..e7d79271e3 100644
> --- a/meta/classes/distutils.bbclass
> +++ b/meta/classes/distutils.bbclass
> @@ -53,18 +53,14 @@ distutils_do_install() {
>  
>          if test -e ${D}${bindir} ; then	
>              for i in ${D}${bindir}/* ; do \
> -                if [ ${PN} != "${BPN}-native" ]; then
> -			sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
> -		fi
> +                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
>                  sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
>              done
>          fi
>  
>          if [ -e ${D}${sbindir} ]; then
>              for i in ${D}${sbindir}/* ; do \
> -                if [ ${PN} != "${BPN}-native" ]; then
> -			sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
> -		fi
> +                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
>                  sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
>              done
>          fi

I have a bad feeling that there is a reason we use the full path. I
have memories of cases where the "python-native" component isn't in
PATH but we obviously still want it to use our python since that is
what its been built against.

One fix would be to apply this, then ensure all those locations inherit
the pythonnative bbclass however I think that becomes annoying as
anything using the recipe as a dependency also has to inherit the
class.

I *think* we might have therefore created a nativepython symlink of
some kind to ./python-native/python. The right fix might therefore be
to use "${USRBINPATH}/env\ nativepython"?

You'll have to check the code for the correct names as I'm going from
memory.

Cheers,

Richard





^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH v2] distutils: Run python from the PATH in the -native case as well
  2019-04-30  2:34 [PATCH] distutils: Run python from the PATH in the -native case as well Douglas Royds
  2019-04-30  8:37 ` Richard Purdie
@ 2019-05-03  2:23 ` Douglas Royds
  2019-05-06  6:47 ` [PATCH v3] " Douglas Royds
  2 siblings, 0 replies; 4+ messages in thread
From: Douglas Royds @ 2019-05-03  2:23 UTC (permalink / raw)
  To: openembedded-core

The python distutils generate a python wrapper script for each package,
containing shebang lines pointing to the python executable.
In our case, this is a fully-qualified path to python-native in the
recipe-sysroot-native.

Ubuntu 18.04 restricts the useful length of the shebang line to 125
characters, and Ubuntu 16.04 restricts it to 77. In both cases, the
staged python script fails to run due to the length of the path to
the python-native executable.

Replace the shebang line with nativepython or nativepython3 as appropriate.
The nativepython symlink is installed by the python-native recipe:

    #!/usr/bin/env nativepython

We were already doing this for on-target distutils components.
This change applies the sed-line to -native distutils components as well.
In this way, -native clients of these components can invoke the wrapper scripts
directly, without themselves needing to inherit pythonnative.

This works around a known setuptools issue:
https://github.com/pypa/setuptools/issues/494
Even once this issue has been resolved upstream,
we will still need to replace `python` with `nativepython`

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/distutils.bbclass  | 11 +++++------
 meta/classes/distutils3.bbclass |  7 +++++--
 2 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.bbclass
index 9862731493..8d258daea4 100644
--- a/meta/classes/distutils.bbclass
+++ b/meta/classes/distutils.bbclass
@@ -9,6 +9,9 @@ DISTUTILS_INSTALL_ARGS ?= "--root=${D} \
     --install-lib=${PYTHON_SITEPACKAGES_DIR} \
     --install-data=${datadir}"
 
+DISTUTILS_PYTHON = "${PYTHON_PN}"
+DISTUTILS_PYTHON_class-native = "native${PYTHON_PN}"
+
 distutils_do_configure() {
         if [ "${CLEANBROKEN}" != "1" ] ; then
                 NO_FETCH_BUILD=1 \
@@ -53,18 +56,14 @@ distutils_do_install() {
 
         if test -e ${D}${bindir} ; then	
             for i in ${D}${bindir}/* ; do \
-                if [ ${PN} != "${BPN}-native" ]; then
-			sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
-		fi
+                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ ${DISTUTILS_PYTHON}:g $i
                 sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
             done
         fi
 
         if [ -e ${D}${sbindir} ]; then
             for i in ${D}${sbindir}/* ; do \
-                if [ ${PN} != "${BPN}-native" ]; then
-			sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
-		fi
+                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ ${DISTUTILS_PYTHON}:g $i
                 sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
             done
         fi
diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3.bbclass
index 834e322474..bae536d343 100644
--- a/meta/classes/distutils3.bbclass
+++ b/meta/classes/distutils3.bbclass
@@ -10,6 +10,9 @@ DISTUTILS_INSTALL_ARGS ?= "--root=${D} \
     --install-lib=${PYTHON_SITEPACKAGES_DIR} \
     --install-data=${datadir}"
 
+DISTUTILS_PYTHON = "${PYTHON_PN}"
+DISTUTILS_PYTHON_class-native = "native${PYTHON_PN}"
+
 distutils3_do_configure() {
 	if [ "${CLEANBROKEN}" != "1" ] ; then
 		NO_FETCH_BUILD=1 \
@@ -57,14 +60,14 @@ distutils3_do_install() {
 
         if test -e ${D}${bindir} ; then	
             for i in ${D}${bindir}/* ; do \
-                sed -i -e s:${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}:${USRBINPATH}/env\ ${PYTHON_PN}:g $i
+                sed -i -e s:${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}:${USRBINPATH}/env\ ${DISTUTILS_PYTHON}:g $i
                 sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
             done
         fi
 
         if test -e ${D}${sbindir}; then
             for i in ${D}${sbindir}/* ; do \
-                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-${PYTHON_PN}/${PYTHON_PN}:${USRBINPATH}/env\ ${PYTHON_PN}:g $i
+                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-${PYTHON_PN}/${PYTHON_PN}:${USRBINPATH}/env\ ${DISTUTILS_PYTHON}:g $i
                 sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
             done
         fi
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 4+ messages in thread

* [PATCH v3] distutils: Run python from the PATH in the -native case as well
  2019-04-30  2:34 [PATCH] distutils: Run python from the PATH in the -native case as well Douglas Royds
  2019-04-30  8:37 ` Richard Purdie
  2019-05-03  2:23 ` [PATCH v2] " Douglas Royds
@ 2019-05-06  6:47 ` Douglas Royds
  2 siblings, 0 replies; 4+ messages in thread
From: Douglas Royds @ 2019-05-06  6:47 UTC (permalink / raw)
  To: openembedded-core

The python distutils generate a python wrapper script for each package,
containing shebang lines pointing to the python executable.
In our case, this is a fully-qualified path to python-native in the
recipe-sysroot-native.

Ubuntu 18.04 restricts the useful length of the shebang line to 125
characters, and Ubuntu 16.04 restricts it to 77. In both cases, the
staged python script fails to run due to the length of the path to
the python-native executable.

Replace the shebang line with nativepython or nativepython3 as appropriate.
The nativepython symlink is installed by the python-native recipe:

    #!/usr/bin/env nativepython

We were already doing this for on-target distutils components.
This change applies the sed-line to -native distutils components as well.
In this way, -native clients of these components can invoke the wrapper scripts
directly, without themselves needing to inherit pythonnative.

This works around a known setuptools issue:
https://github.com/pypa/setuptools/issues/494
Even once this issue has been resolved upstream,
we will still need to replace `python` with `nativepython`

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/distutils.bbclass  | 11 +++++------
 meta/classes/distutils3.bbclass |  7 +++++--
 2 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.bbclass
index 9862731493..b5c9c2fbbd 100644
--- a/meta/classes/distutils.bbclass
+++ b/meta/classes/distutils.bbclass
@@ -9,6 +9,9 @@ DISTUTILS_INSTALL_ARGS ?= "--root=${D} \
     --install-lib=${PYTHON_SITEPACKAGES_DIR} \
     --install-data=${datadir}"
 
+DISTUTILS_PYTHON = "python"
+DISTUTILS_PYTHON_class-native = "nativepython"
+
 distutils_do_configure() {
         if [ "${CLEANBROKEN}" != "1" ] ; then
                 NO_FETCH_BUILD=1 \
@@ -53,18 +56,14 @@ distutils_do_install() {
 
         if test -e ${D}${bindir} ; then	
             for i in ${D}${bindir}/* ; do \
-                if [ ${PN} != "${BPN}-native" ]; then
-			sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
-		fi
+                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ ${DISTUTILS_PYTHON}:g $i
                 sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
             done
         fi
 
         if [ -e ${D}${sbindir} ]; then
             for i in ${D}${sbindir}/* ; do \
-                if [ ${PN} != "${BPN}-native" ]; then
-			sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ python:g $i
-		fi
+                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-native/python:${USRBINPATH}/env\ ${DISTUTILS_PYTHON}:g $i
                 sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
             done
         fi
diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3.bbclass
index 834e322474..42e7f5ae53 100644
--- a/meta/classes/distutils3.bbclass
+++ b/meta/classes/distutils3.bbclass
@@ -10,6 +10,9 @@ DISTUTILS_INSTALL_ARGS ?= "--root=${D} \
     --install-lib=${PYTHON_SITEPACKAGES_DIR} \
     --install-data=${datadir}"
 
+DISTUTILS_PYTHON = "python3"
+DISTUTILS_PYTHON_class-native = "nativepython3"
+
 distutils3_do_configure() {
 	if [ "${CLEANBROKEN}" != "1" ] ; then
 		NO_FETCH_BUILD=1 \
@@ -57,14 +60,14 @@ distutils3_do_install() {
 
         if test -e ${D}${bindir} ; then	
             for i in ${D}${bindir}/* ; do \
-                sed -i -e s:${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}:${USRBINPATH}/env\ ${PYTHON_PN}:g $i
+                sed -i -e s:${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}:${USRBINPATH}/env\ ${DISTUTILS_PYTHON}:g $i
                 sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
             done
         fi
 
         if test -e ${D}${sbindir}; then
             for i in ${D}${sbindir}/* ; do \
-                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-${PYTHON_PN}/${PYTHON_PN}:${USRBINPATH}/env\ ${PYTHON_PN}:g $i
+                sed -i -e s:${STAGING_BINDIR_NATIVE}/python-${PYTHON_PN}/${PYTHON_PN}:${USRBINPATH}/env\ ${DISTUTILS_PYTHON}:g $i
                 sed -i -e s:${STAGING_BINDIR_NATIVE}:${bindir}:g $i
             done
         fi
-- 
2.17.1



^ permalink raw reply related	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2019-05-06  6:47 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-04-30  2:34 [PATCH] distutils: Run python from the PATH in the -native case as well Douglas Royds
2019-04-30  8:37 ` Richard Purdie
2019-05-03  2:23 ` [PATCH v2] " Douglas Royds
2019-05-06  6:47 ` [PATCH v3] " Douglas Royds

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.