[PATCH 0/2] distutils/setuptools:fix recipes may try to fetch code during do_compile

[PATCH 0/2] distutils/setuptools:fix recipes may try to fetch code during do_compile

[Hongxu Jia]

The following changes since commit 05c32d2de1ee4681cc78cb107a158e9ab22c9619:

  oeqa/selftest/wic: Split tests into two class groups (2018-07-16 13:21:35 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib hongxu/setuptools-fetch
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=hongxu/setuptools-fetch

Hongxu Jia (2):
  distutils/setuptools-native: do not try to fetch code during
    do_compile
  distutils3/setuptools3-native: do not try to fetch code during
    do_compile

 meta/classes/distutils.bbclass                     |  2 ++
 meta/classes/distutils3.bbclass                    |  2 ++
 ...ionally-do-not-fetch-code-by-easy_install.patch | 35 ++++++++++++++++++++++
 .../python/python-setuptools_40.0.0.bb             |  2 ++
 ...ionally-do-not-fetch-code-by-easy_install.patch | 35 ++++++++++++++++++++++
 .../python/python3-setuptools_40.0.0.bb            |  2 ++
 6 files changed, 78 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch
 create mode 100644 meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch

-- 
2.8.1

[PATCH 1/2] distutils/setuptools-native: do not try to fetch code during do_compile

[Hongxu Jia]

If a python recipe is using setuptools and the setup_requires argument, where
setuptools will use easy_install to fetch the module if it isn't
present.

The build failed on a machine where a proxy was required, but succeeded on a
machine which had direct access to the internet

Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not
allow to fetch code from internet during do_compile.

Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS.
Traceback (most recent call last):
  File "setup.py", line 56, in <module>
...

The improvement is flexible for test_requirements argument (used at
`setup.py test'), where use easy_install also.

[YOCTO #12084]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/distutils.bbclass                     |  2 ++
 ...ionally-do-not-fetch-code-by-easy_install.patch | 35 ++++++++++++++++++++++
 .../python/python-setuptools_40.0.0.bb             |  2 ++
 3 files changed, 39 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch

diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.bbclass
index 1930c35..872f4e3 100644
--- a/meta/classes/distutils.bbclass
+++ b/meta/classes/distutils.bbclass
@@ -8,6 +8,8 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
     --install-data=${D}/${datadir}"
 
 distutils_do_compile() {
+         NO_FETCH_BUILD=1 \
+         NO_FETCH_BUILD_HELP="Please add its native recipe to DEPENDS." \
          STAGING_INCDIR=${STAGING_INCDIR} \
          STAGING_LIBDIR=${STAGING_LIBDIR} \
          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS} || \
diff --git a/meta/recipes-devtools/python/python-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch b/meta/recipes-devtools/python/python-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch
new file mode 100644
index 0000000..441af16
--- /dev/null
+++ b/meta/recipes-devtools/python/python-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch
@@ -0,0 +1,35 @@
+From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Tue, 17 Jul 2018 10:13:38 +0800
+Subject: [PATCH] conditionally do not fetch code by easy_install
+
+If var-NO_FETCH_BUILD is set, do not allow to fetch code from
+internet by easy_install, and log the help message from
+var-NO_FETCH_BUILD_HELP
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+ setuptools/command/easy_install.py | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
+index 85ee40f..7eb9d2a 100755
+--- a/setuptools/command/easy_install.py
++++ b/setuptools/command/easy_install.py
+@@ -637,6 +637,11 @@ class easy_install(Command):
+             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
+ 
+     def easy_install(self, spec, deps=False):
++        if os.environ.get('NO_FETCH_BUILD', None):
++            log.error("ERROR: Do not try to fetch `%s' for building. %s" %
++                             (spec,  os.environ.get('NO_FETCH_BUILD_HELP', '')))
++            return None
++
+         if not self.editable:
+             self.install_site_py()
+ 
+-- 
+2.7.4
+
diff --git a/meta/recipes-devtools/python/python-setuptools_40.0.0.bb b/meta/recipes-devtools/python/python-setuptools_40.0.0.bb
index cf94404..9d9a093 100644
--- a/meta/recipes-devtools/python/python-setuptools_40.0.0.bb
+++ b/meta/recipes-devtools/python/python-setuptools_40.0.0.bb
@@ -1,5 +1,7 @@
 require python-setuptools.inc
 
+SRC_URI_append_class-native = " file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"
+
 PROVIDES = "python-distribute"
 
 inherit setuptools
-- 
2.8.1

[PATCH 2/2] distutils3/setuptools3-native: do not try to fetch code during do_compile

[Hongxu Jia]

If a python recipe is using setuptools and the setup_requires argument, where
setuptools will use easy_install to fetch the module if it isn't
present.

The build failed on a machine where a proxy was required, but succeeded on a
machine which had direct access to the internet

Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not
allow to fetch code from internet during do_compile.

Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS.
Traceback (most recent call last):
File "setup.py", line 56, in <module>
...

The improvement is flexible for test_requirements argument (used at
`setup.py test'), where use easy_install also.

[YOCTO #12084]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/distutils3.bbclass                    |  2 ++
 ...ionally-do-not-fetch-code-by-easy_install.patch | 35 ++++++++++++++++++++++
 .../python/python3-setuptools_40.0.0.bb            |  2 ++
 3 files changed, 39 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch

diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3.bbclass
index d4b92a3..fea7368 100644
--- a/meta/classes/distutils3.bbclass
+++ b/meta/classes/distutils3.bbclass
@@ -15,6 +15,8 @@ distutils3_do_configure() {
 }
 
 distutils3_do_compile() {
+        NO_FETCH_BUILD=1 \
+        NO_FETCH_BUILD_HELP="Please add its native recipe to DEPENDS." \
         STAGING_INCDIR=${STAGING_INCDIR} \
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \
diff --git a/meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch b/meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch
new file mode 100644
index 0000000..441af16
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-setuptools/0001-conditionally-do-not-fetch-code-by-easy_install.patch
@@ -0,0 +1,35 @@
+From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Tue, 17 Jul 2018 10:13:38 +0800
+Subject: [PATCH] conditionally do not fetch code by easy_install
+
+If var-NO_FETCH_BUILD is set, do not allow to fetch code from
+internet by easy_install, and log the help message from
+var-NO_FETCH_BUILD_HELP
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+ setuptools/command/easy_install.py | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
+index 85ee40f..7eb9d2a 100755
+--- a/setuptools/command/easy_install.py
++++ b/setuptools/command/easy_install.py
+@@ -637,6 +637,11 @@ class easy_install(Command):
+             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
+ 
+     def easy_install(self, spec, deps=False):
++        if os.environ.get('NO_FETCH_BUILD', None):
++            log.error("ERROR: Do not try to fetch `%s' for building. %s" %
++                             (spec,  os.environ.get('NO_FETCH_BUILD_HELP', '')))
++            return None
++
+         if not self.editable:
+             self.install_site_py()
+ 
+-- 
+2.7.4
+
diff --git a/meta/recipes-devtools/python/python3-setuptools_40.0.0.bb b/meta/recipes-devtools/python/python3-setuptools_40.0.0.bb
index 0dc1ed8..713a1b5 100644
--- a/meta/recipes-devtools/python/python3-setuptools_40.0.0.bb
+++ b/meta/recipes-devtools/python/python3-setuptools_40.0.0.bb
@@ -1,6 +1,8 @@
 require python-setuptools.inc
 inherit setuptools3
 
+SRC_URI_append_class-native = " file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"
+
 do_install_append() {
     mv ${D}${bindir}/easy_install ${D}${bindir}/easy3_install
 }
-- 
2.8.1

Re: [PATCH 2/2] distutils3/setuptools3-native: do not try to fetch code during do_compile

[Burton, Ross]

On 17 July 2018 at 04:34, Hongxu Jia <hongxu.jia@windriver.com> wrote:
>  distutils3_do_compile() {
> +        NO_FETCH_BUILD=1 \
> +        NO_FETCH_BUILD_HELP="Please add its native recipe to DEPENDS." \

...

> +     def easy_install(self, spec, deps=False):
> ++        if os.environ.get('NO_FETCH_BUILD', None):
> ++            log.error("ERROR: Do not try to fetch `%s' for building. %s" %
> ++                             (spec,  os.environ.get('NO_FETCH_BUILD_HELP', '')))

Passing the help to the patch seems overkill considering we're
patching in the message in the first place.  Why not just put this in
the patch itself?

> +SRC_URI_append_class-native = " file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"

Do this in setuptools.inc and you won't need to patch both recipes.

Ross

[PATCH V2] distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile

[Hongxu Jia]

If a python recipe is using setuptools and the setup_requires argument, where
setuptools will use easy_install to fetch the module if it isn't
present.

The build failed on a machine where a proxy was required, but succeeded on a
machine which had direct access to the internet

Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not
allow to fetch code from internet during do_compile.

Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS.
Traceback (most recent call last):
  File "setup.py", line 56, in <module>
...

The improvement is flexible for test_requirements argument (used at
`setup.py test'), where use easy_install also.

[YOCTO #12084]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/distutils.bbclass                     |  1 +
 meta/classes/distutils3.bbclass                    |  1 +
 ...ionally-do-not-fetch-code-by-easy_install.patch | 34 ++++++++++++++++++++++
 meta/recipes-devtools/python/python-setuptools.inc |  2 ++
 4 files changed, 38 insertions(+)
 create mode 100644 meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch

diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.bbclass
index 1930c35..c2f7403 100644
--- a/meta/classes/distutils.bbclass
+++ b/meta/classes/distutils.bbclass
@@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
     --install-data=${D}/${datadir}"
 
 distutils_do_compile() {
+         NO_FETCH_BUILD=1 \
          STAGING_INCDIR=${STAGING_INCDIR} \
          STAGING_LIBDIR=${STAGING_LIBDIR} \
          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS} || \
diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3.bbclass
index d4b92a3..f8aca5e 100644
--- a/meta/classes/distutils3.bbclass
+++ b/meta/classes/distutils3.bbclass
@@ -15,6 +15,7 @@ distutils3_do_configure() {
 }
 
 distutils3_do_compile() {
+        NO_FETCH_BUILD=1 \
         STAGING_INCDIR=${STAGING_INCDIR} \
         STAGING_LIBDIR=${STAGING_LIBDIR} \
         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \
diff --git a/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
new file mode 100644
index 0000000..4b17921
--- /dev/null
+++ b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
@@ -0,0 +1,34 @@
+From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Tue, 17 Jul 2018 10:13:38 +0800
+Subject: [PATCH] conditionally do not fetch code by easy_install
+
+If var-NO_FETCH_BUILD is set, do not allow to fetch code from
+internet by easy_install.
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+ setuptools/command/easy_install.py | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/setuptools/command/easy_install.py b/setuptools/command/easy_install.py
+index 85ee40f..7eb9d2a 100755
+--- a/setuptools/command/easy_install.py
++++ b/setuptools/command/easy_install.py
+@@ -637,6 +637,11 @@ class easy_install(Command):
+             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
+ 
+     def easy_install(self, spec, deps=False):
++        if os.environ.get('NO_FETCH_BUILD', None):
++            log.error("ERROR: Do not try to fetch `%s' for building. "
++                      "Please add its native recipe to DEPENDS." % spec)
++            return None
++
+         if not self.editable:
+             self.install_site_py()
+ 
+-- 
+2.7.4
+
diff --git a/meta/recipes-devtools/python/python-setuptools.inc b/meta/recipes-devtools/python/python-setuptools.inc
index 25be446..24b3ac4 100644
--- a/meta/recipes-devtools/python/python-setuptools.inc
+++ b/meta/recipes-devtools/python/python-setuptools.inc
@@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT = "zip"
 
 inherit pypi
 
+SRC_URI_append_class-native = " file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"
+
 SRC_URI[md5sum] = "260630ae1a64bafa39dcc53404d63829"
 SRC_URI[sha256sum] = "012adb8e25fbfd64c652e99e7bab58799a3aaf05d39ab38561f69190a909015f"
 
-- 
2.7.4

Re: [PATCH V2] distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 5820 bytes --]

I like this change, some recipes in meta-oe seems to be broken now, is
someone already working on a fix?

I'm seeing following 5 failures in my world builds:

python-stevedore-1.28.0-r0 do_compile and python3-stevedore-1.28.0-r0
do_compile
pkg_resources.DistributionNotFound: The 'pbr>=2.0.0' distribution was not
found and is required by the application

python-dateutil-2.7.3-r0 do_compile and python3-dateutil-2.7.3-r0 do_compile
pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution was
not found and is required by the application

python-daemon-2.1.2-r0 do_compile
pkg_resources.DistributionNotFound: The 'docutils' distribution was not
found and is required by the application

I've found a fix for python-dateutils from Derek:
https://patchwork.openembedded.org/patch/153002/

On Tue, Jul 17, 2018 at 3:27 PM Hongxu Jia <hongxu.jia@windriver.com> wrote:

> If a python recipe is using setuptools and the setup_requires argument,
> where
> setuptools will use easy_install to fetch the module if it isn't
> present.
>
> The build failed on a machine where a proxy was required, but succeeded on
> a
> machine which had direct access to the internet
>
> Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not
> allow to fetch code from internet during do_compile.
>
> Example result:
> ...
> ERROR: Do not try to fetch `pytest-runner1' for building. Please add its
> native recipe to DEPENDS.
> Traceback (most recent call last):
>   File "setup.py", line 56, in <module>
> ...
>
> The improvement is flexible for test_requirements argument (used at
> `setup.py test'), where use easy_install also.
>
> [YOCTO #12084]
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
>  meta/classes/distutils.bbclass                     |  1 +
>  meta/classes/distutils3.bbclass                    |  1 +
>  ...ionally-do-not-fetch-code-by-easy_install.patch | 34
> ++++++++++++++++++++++
>  meta/recipes-devtools/python/python-setuptools.inc |  2 ++
>  4 files changed, 38 insertions(+)
>  create mode 100644
> meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
>
> diff --git a/meta/classes/distutils.bbclass
> b/meta/classes/distutils.bbclass
> index 1930c35..c2f7403 100644
> --- a/meta/classes/distutils.bbclass
> +++ b/meta/classes/distutils.bbclass
> @@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
>      --install-data=${D}/${datadir}"
>
>  distutils_do_compile() {
> +         NO_FETCH_BUILD=1 \
>           STAGING_INCDIR=${STAGING_INCDIR} \
>           STAGING_LIBDIR=${STAGING_LIBDIR} \
>           ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
> setup.py build ${DISTUTILS_BUILD_ARGS} || \
> diff --git a/meta/classes/distutils3.bbclass
> b/meta/classes/distutils3.bbclass
> index d4b92a3..f8aca5e 100644
> --- a/meta/classes/distutils3.bbclass
> +++ b/meta/classes/distutils3.bbclass
> @@ -15,6 +15,7 @@ distutils3_do_configure() {
>  }
>
>  distutils3_do_compile() {
> +        NO_FETCH_BUILD=1 \
>          STAGING_INCDIR=${STAGING_INCDIR} \
>          STAGING_LIBDIR=${STAGING_LIBDIR} \
>          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
> setup.py \
> diff --git
> a/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
> b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
> new file mode 100644
> index 0000000..4b17921
> --- /dev/null
> +++
> b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
> @@ -0,0 +1,34 @@
> +From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
> +From: Hongxu Jia <hongxu.jia@windriver.com>
> +Date: Tue, 17 Jul 2018 10:13:38 +0800
> +Subject: [PATCH] conditionally do not fetch code by easy_install
> +
> +If var-NO_FETCH_BUILD is set, do not allow to fetch code from
> +internet by easy_install.
> +
> +Upstream-Status: Inappropriate [oe specific]
> +
> +Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> +---
> + setuptools/command/easy_install.py | 5 +++++
> + 1 file changed, 5 insertions(+)
> +
> +diff --git a/setuptools/command/easy_install.py
> b/setuptools/command/easy_install.py
> +index 85ee40f..7eb9d2a 100755
> +--- a/setuptools/command/easy_install.py
> ++++ b/setuptools/command/easy_install.py
> +@@ -637,6 +637,11 @@ class easy_install(Command):
> +             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
> +
> +     def easy_install(self, spec, deps=False):
> ++        if os.environ.get('NO_FETCH_BUILD', None):
> ++            log.error("ERROR: Do not try to fetch `%s' for building. "
> ++                      "Please add its native recipe to DEPENDS." % spec)
> ++            return None
> ++
> +         if not self.editable:
> +             self.install_site_py()
> +
> +--
> +2.7.4
> +
> diff --git a/meta/recipes-devtools/python/python-setuptools.inc
> b/meta/recipes-devtools/python/python-setuptools.inc
> index 25be446..24b3ac4 100644
> --- a/meta/recipes-devtools/python/python-setuptools.inc
> +++ b/meta/recipes-devtools/python/python-setuptools.inc
> @@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT = "zip"
>
>  inherit pypi
>
> +SRC_URI_append_class-native = "
> file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"
> +
>  SRC_URI[md5sum] = "260630ae1a64bafa39dcc53404d63829"
>  SRC_URI[sha256sum] =
> "012adb8e25fbfd64c652e99e7bab58799a3aaf05d39ab38561f69190a909015f"
>
> --
> 2.7.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 7159 bytes --]

Re: [PATCH V2] distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile

[Derek Straka]

[-- Attachment #1: Type: text/plain, Size: 6138 bytes --]

Yes.  I was just waiting for my master build to wrap up.

On Fri, Jul 27, 2018, 7:39 AM Martin Jansa <martin.jansa@gmail.com> wrote:

> I like this change, some recipes in meta-oe seems to be broken now, is
> someone already working on a fix?
>
> I'm seeing following 5 failures in my world builds:
>
> python-stevedore-1.28.0-r0 do_compile and python3-stevedore-1.28.0-r0
> do_compile
> pkg_resources.DistributionNotFound: The 'pbr>=2.0.0' distribution was not
> found and is required by the application
>
> python-dateutil-2.7.3-r0 do_compile and python3-dateutil-2.7.3-r0
> do_compile
> pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution was
> not found and is required by the application
>
> python-daemon-2.1.2-r0 do_compile
> pkg_resources.DistributionNotFound: The 'docutils' distribution was not
> found and is required by the application
>
> I've found a fix for python-dateutils from Derek:
> https://patchwork.openembedded.org/patch/153002/
>
> On Tue, Jul 17, 2018 at 3:27 PM Hongxu Jia <hongxu.jia@windriver.com>
> wrote:
>
>> If a python recipe is using setuptools and the setup_requires argument,
>> where
>> setuptools will use easy_install to fetch the module if it isn't
>> present.
>>
>> The build failed on a machine where a proxy was required, but succeeded
>> on a
>> machine which had direct access to the internet
>>
>> Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not
>> allow to fetch code from internet during do_compile.
>>
>> Example result:
>> ...
>> ERROR: Do not try to fetch `pytest-runner1' for building. Please add its
>> native recipe to DEPENDS.
>> Traceback (most recent call last):
>>   File "setup.py", line 56, in <module>
>> ...
>>
>> The improvement is flexible for test_requirements argument (used at
>> `setup.py test'), where use easy_install also.
>>
>> [YOCTO #12084]
>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>> ---
>>  meta/classes/distutils.bbclass                     |  1 +
>>  meta/classes/distutils3.bbclass                    |  1 +
>>  ...ionally-do-not-fetch-code-by-easy_install.patch | 34
>> ++++++++++++++++++++++
>>  meta/recipes-devtools/python/python-setuptools.inc |  2 ++
>>  4 files changed, 38 insertions(+)
>>  create mode 100644
>> meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
>>
>> diff --git a/meta/classes/distutils.bbclass
>> b/meta/classes/distutils.bbclass
>> index 1930c35..c2f7403 100644
>> --- a/meta/classes/distutils.bbclass
>> +++ b/meta/classes/distutils.bbclass
>> @@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
>>      --install-data=${D}/${datadir}"
>>
>>  distutils_do_compile() {
>> +         NO_FETCH_BUILD=1 \
>>           STAGING_INCDIR=${STAGING_INCDIR} \
>>           STAGING_LIBDIR=${STAGING_LIBDIR} \
>>           ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
>> setup.py build ${DISTUTILS_BUILD_ARGS} || \
>> diff --git a/meta/classes/distutils3.bbclass
>> b/meta/classes/distutils3.bbclass
>> index d4b92a3..f8aca5e 100644
>> --- a/meta/classes/distutils3.bbclass
>> +++ b/meta/classes/distutils3.bbclass
>> @@ -15,6 +15,7 @@ distutils3_do_configure() {
>>  }
>>
>>  distutils3_do_compile() {
>> +        NO_FETCH_BUILD=1 \
>>          STAGING_INCDIR=${STAGING_INCDIR} \
>>          STAGING_LIBDIR=${STAGING_LIBDIR} \
>>          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
>> setup.py \
>> diff --git
>> a/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
>> b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
>> new file mode 100644
>> index 0000000..4b17921
>> --- /dev/null
>> +++
>> b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
>> @@ -0,0 +1,34 @@
>> +From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
>> +From: Hongxu Jia <hongxu.jia@windriver.com>
>> +Date: Tue, 17 Jul 2018 10:13:38 +0800
>> +Subject: [PATCH] conditionally do not fetch code by easy_install
>> +
>> +If var-NO_FETCH_BUILD is set, do not allow to fetch code from
>> +internet by easy_install.
>> +
>> +Upstream-Status: Inappropriate [oe specific]
>> +
>> +Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>> +---
>> + setuptools/command/easy_install.py | 5 +++++
>> + 1 file changed, 5 insertions(+)
>> +
>> +diff --git a/setuptools/command/easy_install.py
>> b/setuptools/command/easy_install.py
>> +index 85ee40f..7eb9d2a 100755
>> +--- a/setuptools/command/easy_install.py
>> ++++ b/setuptools/command/easy_install.py
>> +@@ -637,6 +637,11 @@ class easy_install(Command):
>> +             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
>> +
>> +     def easy_install(self, spec, deps=False):
>> ++        if os.environ.get('NO_FETCH_BUILD', None):
>> ++            log.error("ERROR: Do not try to fetch `%s' for building. "
>> ++                      "Please add its native recipe to DEPENDS." % spec)
>> ++            return None
>> ++
>> +         if not self.editable:
>> +             self.install_site_py()
>> +
>> +--
>> +2.7.4
>> +
>> diff --git a/meta/recipes-devtools/python/python-setuptools.inc
>> b/meta/recipes-devtools/python/python-setuptools.inc
>> index 25be446..24b3ac4 100644
>> --- a/meta/recipes-devtools/python/python-setuptools.inc
>> +++ b/meta/recipes-devtools/python/python-setuptools.inc
>> @@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT = "zip"
>>
>>  inherit pypi
>>
>> +SRC_URI_append_class-native = "
>> file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"
>> +
>>  SRC_URI[md5sum] = "260630ae1a64bafa39dcc53404d63829"
>>  SRC_URI[sha256sum] =
>> "012adb8e25fbfd64c652e99e7bab58799a3aaf05d39ab38561f69190a909015f"
>>
>> --
>> 2.7.4
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>

[-- Attachment #2: Type: text/html, Size: 7688 bytes --]

Re: [PATCH V2] distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile

[Derek Straka]

[-- Attachment #1: Type: text/plain, Size: 6795 bytes --]

python-daemon - https://patchwork.openembedded.org/patch/153034/

python-stevedore - https://patchwork.openembedded.org/patch/153032/

I think that's everything though my world python build is still going.
Thanks for pointing the failures out and let me know if there are others.
I'm guessing this will ripple through the python heavy layers; I found a
couple yesterday in meta-iot-cloud.

On Fri, Jul 27, 2018 at 7:41 AM, Derek Straka <derek@asterius.io> wrote:

> Yes.  I was just waiting for my master build to wrap up.
>
> On Fri, Jul 27, 2018, 7:39 AM Martin Jansa <martin.jansa@gmail.com> wrote:
>
>> I like this change, some recipes in meta-oe seems to be broken now, is
>> someone already working on a fix?
>>
>> I'm seeing following 5 failures in my world builds:
>>
>> python-stevedore-1.28.0-r0 do_compile and python3-stevedore-1.28.0-r0
>> do_compile
>> pkg_resources.DistributionNotFound: The 'pbr>=2.0.0' distribution was
>> not found and is required by the application
>>
>> python-dateutil-2.7.3-r0 do_compile and python3-dateutil-2.7.3-r0
>> do_compile
>> pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution
>> was not found and is required by the application
>>
>> python-daemon-2.1.2-r0 do_compile
>> pkg_resources.DistributionNotFound: The 'docutils' distribution was not
>> found and is required by the application
>>
>> I've found a fix for python-dateutils from Derek:
>> https://patchwork.openembedded.org/patch/153002/
>>
>> On Tue, Jul 17, 2018 at 3:27 PM Hongxu Jia <hongxu.jia@windriver.com>
>> wrote:
>>
>>> If a python recipe is using setuptools and the setup_requires argument,
>>> where
>>> setuptools will use easy_install to fetch the module if it isn't
>>> present.
>>>
>>> The build failed on a machine where a proxy was required, but succeeded
>>> on a
>>> machine which had direct access to the internet
>>>
>>> Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not
>>> allow to fetch code from internet during do_compile.
>>>
>>> Example result:
>>> ...
>>> ERROR: Do not try to fetch `pytest-runner1' for building. Please add its
>>> native recipe to DEPENDS.
>>> Traceback (most recent call last):
>>>   File "setup.py", line 56, in <module>
>>> ...
>>>
>>> The improvement is flexible for test_requirements argument (used at
>>> `setup.py test'), where use easy_install also.
>>>
>>> [YOCTO #12084]
>>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>>> ---
>>>  meta/classes/distutils.bbclass                     |  1 +
>>>  meta/classes/distutils3.bbclass                    |  1 +
>>>  ...ionally-do-not-fetch-code-by-easy_install.patch | 34
>>> ++++++++++++++++++++++
>>>  meta/recipes-devtools/python/python-setuptools.inc |  2 ++
>>>  4 files changed, 38 insertions(+)
>>>  create mode 100644 meta/recipes-devtools/python/
>>> files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
>>>
>>> diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.
>>> bbclass
>>> index 1930c35..c2f7403 100644
>>> --- a/meta/classes/distutils.bbclass
>>> +++ b/meta/classes/distutils.bbclass
>>> @@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
>>>      --install-data=${D}/${datadir}"
>>>
>>>  distutils_do_compile() {
>>> +         NO_FETCH_BUILD=1 \
>>>           STAGING_INCDIR=${STAGING_INCDIR} \
>>>           STAGING_LIBDIR=${STAGING_LIBDIR} \
>>>           ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
>>> setup.py build ${DISTUTILS_BUILD_ARGS} || \
>>> diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3.
>>> bbclass
>>> index d4b92a3..f8aca5e 100644
>>> --- a/meta/classes/distutils3.bbclass
>>> +++ b/meta/classes/distutils3.bbclass
>>> @@ -15,6 +15,7 @@ distutils3_do_configure() {
>>>  }
>>>
>>>  distutils3_do_compile() {
>>> +        NO_FETCH_BUILD=1 \
>>>          STAGING_INCDIR=${STAGING_INCDIR} \
>>>          STAGING_LIBDIR=${STAGING_LIBDIR} \
>>>          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
>>> setup.py \
>>> diff --git a/meta/recipes-devtools/python/files/0001-
>>> conditionally-do-not-fetch-code-by-easy_install.patch
>>> b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-
>>> code-by-easy_install.patch
>>> new file mode 100644
>>> index 0000000..4b17921
>>> --- /dev/null
>>> +++ b/meta/recipes-devtools/python/files/0001-
>>> conditionally-do-not-fetch-code-by-easy_install.patch
>>> @@ -0,0 +1,34 @@
>>> +From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
>>> +From: Hongxu Jia <hongxu.jia@windriver.com>
>>> +Date: Tue, 17 Jul 2018 10:13:38 +0800
>>> +Subject: [PATCH] conditionally do not fetch code by easy_install
>>> +
>>> +If var-NO_FETCH_BUILD is set, do not allow to fetch code from
>>> +internet by easy_install.
>>> +
>>> +Upstream-Status: Inappropriate [oe specific]
>>> +
>>> +Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>>> +---
>>> + setuptools/command/easy_install.py | 5 +++++
>>> + 1 file changed, 5 insertions(+)
>>> +
>>> +diff --git a/setuptools/command/easy_install.py
>>> b/setuptools/command/easy_install.py
>>> +index 85ee40f..7eb9d2a 100755
>>> +--- a/setuptools/command/easy_install.py
>>> ++++ b/setuptools/command/easy_install.py
>>> +@@ -637,6 +637,11 @@ class easy_install(Command):
>>> +             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
>>> +
>>> +     def easy_install(self, spec, deps=False):
>>> ++        if os.environ.get('NO_FETCH_BUILD', None):
>>> ++            log.error("ERROR: Do not try to fetch `%s' for building. "
>>> ++                      "Please add its native recipe to DEPENDS." %
>>> spec)
>>> ++            return None
>>> ++
>>> +         if not self.editable:
>>> +             self.install_site_py()
>>> +
>>> +--
>>> +2.7.4
>>> +
>>> diff --git a/meta/recipes-devtools/python/python-setuptools.inc
>>> b/meta/recipes-devtools/python/python-setuptools.inc
>>> index 25be446..24b3ac4 100644
>>> --- a/meta/recipes-devtools/python/python-setuptools.inc
>>> +++ b/meta/recipes-devtools/python/python-setuptools.inc
>>> @@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT = "zip"
>>>
>>>  inherit pypi
>>>
>>> +SRC_URI_append_class-native = " file://0001-conditionally-do-
>>> not-fetch-code-by-easy_install.patch"
>>> +
>>>  SRC_URI[md5sum] = "260630ae1a64bafa39dcc53404d63829"
>>>  SRC_URI[sha256sum] = "012adb8e25fbfd64c652e99e7bab58
>>> 799a3aaf05d39ab38561f69190a909015f"
>>>
>>> --
>>> 2.7.4
>>>
>>> --
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>
>>

[-- Attachment #2: Type: text/html, Size: 8994 bytes --]

Re: [PATCH V2] distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 7068 bytes --]

Thanks a lot.

On Fri, Jul 27, 2018 at 2:43 PM Derek Straka <derek@asterius.io> wrote:

> python-daemon - https://patchwork.openembedded.org/patch/153034/
>
> python-stevedore - https://patchwork.openembedded.org/patch/153032/
>
> I think that's everything though my world python build is still going.
> Thanks for pointing the failures out and let me know if there are others.
> I'm guessing this will ripple through the python heavy layers; I found a
> couple yesterday in meta-iot-cloud.
>
> On Fri, Jul 27, 2018 at 7:41 AM, Derek Straka <derek@asterius.io> wrote:
>
>> Yes.  I was just waiting for my master build to wrap up.
>>
>> On Fri, Jul 27, 2018, 7:39 AM Martin Jansa <martin.jansa@gmail.com>
>> wrote:
>>
>>> I like this change, some recipes in meta-oe seems to be broken now, is
>>> someone already working on a fix?
>>>
>>> I'm seeing following 5 failures in my world builds:
>>>
>>> python-stevedore-1.28.0-r0 do_compile and python3-stevedore-1.28.0-r0
>>> do_compile
>>> pkg_resources.DistributionNotFound: The 'pbr>=2.0.0' distribution was
>>> not found and is required by the application
>>>
>>> python-dateutil-2.7.3-r0 do_compile and python3-dateutil-2.7.3-r0
>>> do_compile
>>> pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution
>>> was not found and is required by the application
>>>
>>> python-daemon-2.1.2-r0 do_compile
>>> pkg_resources.DistributionNotFound: The 'docutils' distribution was not
>>> found and is required by the application
>>>
>>> I've found a fix for python-dateutils from Derek:
>>> https://patchwork.openembedded.org/patch/153002/
>>>
>>> On Tue, Jul 17, 2018 at 3:27 PM Hongxu Jia <hongxu.jia@windriver.com>
>>> wrote:
>>>
>>>> If a python recipe is using setuptools and the setup_requires argument,
>>>> where
>>>> setuptools will use easy_install to fetch the module if it isn't
>>>> present.
>>>>
>>>> The build failed on a machine where a proxy was required, but succeeded
>>>> on a
>>>> machine which had direct access to the internet
>>>>
>>>> Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does
>>>> not
>>>> allow to fetch code from internet during do_compile.
>>>>
>>>> Example result:
>>>> ...
>>>> ERROR: Do not try to fetch `pytest-runner1' for building. Please add
>>>> its native recipe to DEPENDS.
>>>> Traceback (most recent call last):
>>>>   File "setup.py", line 56, in <module>
>>>> ...
>>>>
>>>> The improvement is flexible for test_requirements argument (used at
>>>> `setup.py test'), where use easy_install also.
>>>>
>>>> [YOCTO #12084]
>>>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>>>> ---
>>>>  meta/classes/distutils.bbclass                     |  1 +
>>>>  meta/classes/distutils3.bbclass                    |  1 +
>>>>  ...ionally-do-not-fetch-code-by-easy_install.patch | 34
>>>> ++++++++++++++++++++++
>>>>  meta/recipes-devtools/python/python-setuptools.inc |  2 ++
>>>>  4 files changed, 38 insertions(+)
>>>>  create mode 100644
>>>> meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
>>>>
>>>> diff --git a/meta/classes/distutils.bbclass
>>>> b/meta/classes/distutils.bbclass
>>>> index 1930c35..c2f7403 100644
>>>> --- a/meta/classes/distutils.bbclass
>>>> +++ b/meta/classes/distutils.bbclass
>>>> @@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \
>>>>      --install-data=${D}/${datadir}"
>>>>
>>>>  distutils_do_compile() {
>>>> +         NO_FETCH_BUILD=1 \
>>>>           STAGING_INCDIR=${STAGING_INCDIR} \
>>>>           STAGING_LIBDIR=${STAGING_LIBDIR} \
>>>>           ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
>>>> setup.py build ${DISTUTILS_BUILD_ARGS} || \
>>>> diff --git a/meta/classes/distutils3.bbclass
>>>> b/meta/classes/distutils3.bbclass
>>>> index d4b92a3..f8aca5e 100644
>>>> --- a/meta/classes/distutils3.bbclass
>>>> +++ b/meta/classes/distutils3.bbclass
>>>> @@ -15,6 +15,7 @@ distutils3_do_configure() {
>>>>  }
>>>>
>>>>  distutils3_do_compile() {
>>>> +        NO_FETCH_BUILD=1 \
>>>>          STAGING_INCDIR=${STAGING_INCDIR} \
>>>>          STAGING_LIBDIR=${STAGING_LIBDIR} \
>>>>          ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
>>>> setup.py \
>>>> diff --git
>>>> a/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
>>>> b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
>>>> new file mode 100644
>>>> index 0000000..4b17921
>>>> --- /dev/null
>>>> +++
>>>> b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_install.patch
>>>> @@ -0,0 +1,34 @@
>>>> +From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001
>>>> +From: Hongxu Jia <hongxu.jia@windriver.com>
>>>> +Date: Tue, 17 Jul 2018 10:13:38 +0800
>>>> +Subject: [PATCH] conditionally do not fetch code by easy_install
>>>> +
>>>> +If var-NO_FETCH_BUILD is set, do not allow to fetch code from
>>>> +internet by easy_install.
>>>> +
>>>> +Upstream-Status: Inappropriate [oe specific]
>>>> +
>>>> +Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>>>> +---
>>>> + setuptools/command/easy_install.py | 5 +++++
>>>> + 1 file changed, 5 insertions(+)
>>>> +
>>>> +diff --git a/setuptools/command/easy_install.py
>>>> b/setuptools/command/easy_install.py
>>>> +index 85ee40f..7eb9d2a 100755
>>>> +--- a/setuptools/command/easy_install.py
>>>> ++++ b/setuptools/command/easy_install.py
>>>> +@@ -637,6 +637,11 @@ class easy_install(Command):
>>>> +             os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir))
>>>> +
>>>> +     def easy_install(self, spec, deps=False):
>>>> ++        if os.environ.get('NO_FETCH_BUILD', None):
>>>> ++            log.error("ERROR: Do not try to fetch `%s' for building. "
>>>> ++                      "Please add its native recipe to DEPENDS." %
>>>> spec)
>>>> ++            return None
>>>> ++
>>>> +         if not self.editable:
>>>> +             self.install_site_py()
>>>> +
>>>> +--
>>>> +2.7.4
>>>> +
>>>> diff --git a/meta/recipes-devtools/python/python-setuptools.inc
>>>> b/meta/recipes-devtools/python/python-setuptools.inc
>>>> index 25be446..24b3ac4 100644
>>>> --- a/meta/recipes-devtools/python/python-setuptools.inc
>>>> +++ b/meta/recipes-devtools/python/python-setuptools.inc
>>>> @@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT = "zip"
>>>>
>>>>  inherit pypi
>>>>
>>>> +SRC_URI_append_class-native = "
>>>> file://0001-conditionally-do-not-fetch-code-by-easy_install.patch"
>>>> +
>>>>  SRC_URI[md5sum] = "260630ae1a64bafa39dcc53404d63829"
>>>>  SRC_URI[sha256sum] =
>>>> "012adb8e25fbfd64c652e99e7bab58799a3aaf05d39ab38561f69190a909015f"
>>>>
>>>> --
>>>> 2.7.4
>>>>
>>>> --
>>>> _______________________________________________
>>>> Openembedded-core mailing list
>>>> Openembedded-core@lists.openembedded.org
>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>>
>>>
>

[-- Attachment #2: Type: text/html, Size: 9107 bytes --]