From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail4.static.mailgun.info (mail4.static.mailgun.info [104.130.122.4]) by mail.openembedded.org (Postfix) with ESMTP id 685B778CC0 for ; Fri, 27 Jul 2018 12:43:20 +0000 (UTC) DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=asterius.io; q=dns/txt; s=mailo; t=1532695401; h=Content-Type: Cc: To: Subject: Message-ID: Date: From: References: In-Reply-To: MIME-Version: Sender; bh=+JwZDQWhg3FvW53gQtSVxO+Pd7ekEmxYJnThUO5U7jM=; b=SSGuV+hCCH4GqWoRC8w8X+EYIhSkpeWvZt9Kf6pVIESIM+uN+NJyZDCqWGo4yQcV4SLFv9Sl 1TzXvcy2AAyZeKpTT81+Kx3B9C+lSKa18DKwKmexV2msWc/mWTZzPe+OuRrmhQ7Xl9ue3L3/ flp+ZdLateWIN1T0BQDE6O4dAqw= X-Mailgun-Sending-Ip: 104.130.122.4 X-Mailgun-Sid: WyJiMTQwMSIsICJvcGVuZW1iZWRkZWQtY29yZUBsaXN0cy5vcGVuZW1iZWRkZWQub3JnIiwgIjNhNTY4NSJd Sender: derek@asterius.io Received: from mail-it0-f47.google.com (mail-it0-f47.google.com [209.85.214.47]) by mxa.mailgun.org with ESMTP id 5b5b1368.7f9d2196dc30-smtp-out-n03; Fri, 27 Jul 2018 12:43:20 -0000 (UTC) Received: by mail-it0-f47.google.com with SMTP id d9-v6so7105008itf.2 for ; Fri, 27 Jul 2018 05:43:20 -0700 (PDT) X-Gm-Message-State: AOUpUlGIAjTiLvXiG8Zgm3QQ+Yzy/acjLyljhkJxgg5PzGIAQWr9pWTC aZ0aO/7bFB4KTEoPV5bOeEVL3ToR3kpFm9Z41WY= X-Google-Smtp-Source: AAOMgpemkjMMHEpYZnT8XtPmK4jBzSRGIkAog3papjg4vWN7P6uZnBNgX9AOdX+B1Ikwf4FHjNWOTWA272UCmQJSv1U= X-Received: by 2002:a24:3b0e:: with SMTP id c14-v6mr5676011ita.42.1532695400355; Fri, 27 Jul 2018 05:43:20 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a6b:8c4e:0:0:0:0:0 with HTTP; Fri, 27 Jul 2018 05:43:19 -0700 (PDT) In-Reply-To: References: <1531834595-4814-1-git-send-email-hongxu.jia@windriver.com> From: Derek Straka Date: Fri, 27 Jul 2018 08:43:19 -0400 X-Gmail-Original-Message-ID: Message-ID: To: Martin Jansa Cc: Paul Eggleton , openembedded-core Subject: Re: [PATCH V2] distutils/setuptools-native, distutils3/setuptools3-native: do not try to fetch code during do_compile X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2018 12:43:20 -0000 Content-Type: multipart/alternative; boundary="000000000000e320170571fa7485" --000000000000e320170571fa7485 Content-Type: text/plain; charset="UTF-8" python-daemon - https://patchwork.openembedded.org/patch/153034/ python-stevedore - https://patchwork.openembedded.org/patch/153032/ I think that's everything though my world python build is still going. Thanks for pointing the failures out and let me know if there are others. I'm guessing this will ripple through the python heavy layers; I found a couple yesterday in meta-iot-cloud. On Fri, Jul 27, 2018 at 7:41 AM, Derek Straka wrote: > Yes. I was just waiting for my master build to wrap up. > > On Fri, Jul 27, 2018, 7:39 AM Martin Jansa wrote: > >> I like this change, some recipes in meta-oe seems to be broken now, is >> someone already working on a fix? >> >> I'm seeing following 5 failures in my world builds: >> >> python-stevedore-1.28.0-r0 do_compile and python3-stevedore-1.28.0-r0 >> do_compile >> pkg_resources.DistributionNotFound: The 'pbr>=2.0.0' distribution was >> not found and is required by the application >> >> python-dateutil-2.7.3-r0 do_compile and python3-dateutil-2.7.3-r0 >> do_compile >> pkg_resources.DistributionNotFound: The 'setuptools_scm' distribution >> was not found and is required by the application >> >> python-daemon-2.1.2-r0 do_compile >> pkg_resources.DistributionNotFound: The 'docutils' distribution was not >> found and is required by the application >> >> I've found a fix for python-dateutils from Derek: >> https://patchwork.openembedded.org/patch/153002/ >> >> On Tue, Jul 17, 2018 at 3:27 PM Hongxu Jia >> wrote: >> >>> If a python recipe is using setuptools and the setup_requires argument, >>> where >>> setuptools will use easy_install to fetch the module if it isn't >>> present. >>> >>> The build failed on a machine where a proxy was required, but succeeded >>> on a >>> machine which had direct access to the internet >>> >>> Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not >>> allow to fetch code from internet during do_compile. >>> >>> Example result: >>> ... >>> ERROR: Do not try to fetch `pytest-runner1' for building. Please add its >>> native recipe to DEPENDS. >>> Traceback (most recent call last): >>> File "setup.py", line 56, in >>> ... >>> >>> The improvement is flexible for test_requirements argument (used at >>> `setup.py test'), where use easy_install also. >>> >>> [YOCTO #12084] >>> Signed-off-by: Hongxu Jia >>> --- >>> meta/classes/distutils.bbclass | 1 + >>> meta/classes/distutils3.bbclass | 1 + >>> ...ionally-do-not-fetch-code-by-easy_install.patch | 34 >>> ++++++++++++++++++++++ >>> meta/recipes-devtools/python/python-setuptools.inc | 2 ++ >>> 4 files changed, 38 insertions(+) >>> create mode 100644 meta/recipes-devtools/python/ >>> files/0001-conditionally-do-not-fetch-code-by-easy_install.patch >>> >>> diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils. >>> bbclass >>> index 1930c35..c2f7403 100644 >>> --- a/meta/classes/distutils.bbclass >>> +++ b/meta/classes/distutils.bbclass >>> @@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?= "--prefix=${D}/${prefix} \ >>> --install-data=${D}/${datadir}" >>> >>> distutils_do_compile() { >>> + NO_FETCH_BUILD=1 \ >>> STAGING_INCDIR=${STAGING_INCDIR} \ >>> STAGING_LIBDIR=${STAGING_LIBDIR} \ >>> ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} >>> setup.py build ${DISTUTILS_BUILD_ARGS} || \ >>> diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3. >>> bbclass >>> index d4b92a3..f8aca5e 100644 >>> --- a/meta/classes/distutils3.bbclass >>> +++ b/meta/classes/distutils3.bbclass >>> @@ -15,6 +15,7 @@ distutils3_do_configure() { >>> } >>> >>> distutils3_do_compile() { >>> + NO_FETCH_BUILD=1 \ >>> STAGING_INCDIR=${STAGING_INCDIR} \ >>> STAGING_LIBDIR=${STAGING_LIBDIR} \ >>> ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} >>> setup.py \ >>> diff --git a/meta/recipes-devtools/python/files/0001- >>> conditionally-do-not-fetch-code-by-easy_install.patch >>> b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch- >>> code-by-easy_install.patch >>> new file mode 100644 >>> index 0000000..4b17921 >>> --- /dev/null >>> +++ b/meta/recipes-devtools/python/files/0001- >>> conditionally-do-not-fetch-code-by-easy_install.patch >>> @@ -0,0 +1,34 @@ >>> +From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 2001 >>> +From: Hongxu Jia >>> +Date: Tue, 17 Jul 2018 10:13:38 +0800 >>> +Subject: [PATCH] conditionally do not fetch code by easy_install >>> + >>> +If var-NO_FETCH_BUILD is set, do not allow to fetch code from >>> +internet by easy_install. >>> + >>> +Upstream-Status: Inappropriate [oe specific] >>> + >>> +Signed-off-by: Hongxu Jia >>> +--- >>> + setuptools/command/easy_install.py | 5 +++++ >>> + 1 file changed, 5 insertions(+) >>> + >>> +diff --git a/setuptools/command/easy_install.py >>> b/setuptools/command/easy_install.py >>> +index 85ee40f..7eb9d2a 100755 >>> +--- a/setuptools/command/easy_install.py >>> ++++ b/setuptools/command/easy_install.py >>> +@@ -637,6 +637,11 @@ class easy_install(Command): >>> + os.path.exists(tmpdir) and rmtree(rmtree_safe(tmpdir)) >>> + >>> + def easy_install(self, spec, deps=False): >>> ++ if os.environ.get('NO_FETCH_BUILD', None): >>> ++ log.error("ERROR: Do not try to fetch `%s' for building. " >>> ++ "Please add its native recipe to DEPENDS." % >>> spec) >>> ++ return None >>> ++ >>> + if not self.editable: >>> + self.install_site_py() >>> + >>> +-- >>> +2.7.4 >>> + >>> diff --git a/meta/recipes-devtools/python/python-setuptools.inc >>> b/meta/recipes-devtools/python/python-setuptools.inc >>> index 25be446..24b3ac4 100644 >>> --- a/meta/recipes-devtools/python/python-setuptools.inc >>> +++ b/meta/recipes-devtools/python/python-setuptools.inc >>> @@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT = "zip" >>> >>> inherit pypi >>> >>> +SRC_URI_append_class-native = " file://0001-conditionally-do- >>> not-fetch-code-by-easy_install.patch" >>> + >>> SRC_URI[md5sum] = "260630ae1a64bafa39dcc53404d63829" >>> SRC_URI[sha256sum] = "012adb8e25fbfd64c652e99e7bab58 >>> 799a3aaf05d39ab38561f69190a909015f" >>> >>> -- >>> 2.7.4 >>> >>> -- >>> _______________________________________________ >>> Openembedded-core mailing list >>> Openembedded-core@lists.openembedded.org >>> http://lists.openembedded.org/mailman/listinfo/openembedded-core >>> >> --000000000000e320170571fa7485 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
python-daemon -=C2=A0https://patchwork.openembedded.org/patch/153034/


I think that's everything tho= ugh my world python build is still going.=C2=A0 Thanks for pointing the fai= lures out and let me know if there are others.=C2=A0 I'm guessing this = will ripple through the python heavy layers; I found a couple yesterday in = meta-iot-cloud.

On Fri, Jul 27, 2018 at 7:41 AM, Derek Straka &= lt;derek@asterius.io= > wrote:
= Yes.=C2=A0 I was just waiting for my master build to wrap up.=C2=A0=C2=A0

=
On Fri, Jul 27, 2018, 7:39 AM Martin Jansa <martin.jansa@gmail.com= > wrote:
I like= this change, some recipes in meta-oe seems to be broken now, is someone al= ready working on a fix?

I'm seeing following 5 failu= res in my world builds:

python-stevedore-1.28.0-r0= do_compile and=C2=A0python3-stevedore-1.28.0-r0 do_compile
<= div>pkg_resources.DistributionNotFound: The 'pbr>=3D2.0.0' = distribution was not found and is required by the application

python-dateutil-2.7.3-r0 do_compile and=C2=A0python3-dateut= il-2.7.3-r0 do_compile
pkg_resources.DistributionNotFoun= d: The 'setuptools_scm' distribution was not found and is required = by the application

python-daemon-2.1.2-r0 do_c= ompile
pkg_resources.DistributionNotFound: The 'docu= tils' distribution was not found and is required by the application
=

I've found a fix for python-dateutils from De= rek:

On Tue, Jul 17, 2018 at 3:27 PM Hongxu Jia <hongxu.jia@wi= ndriver.com> wrote:
If a pyt= hon recipe is using setuptools and the setup_requires argument, where
setuptools will use easy_install to fetch the module if it isn't
present.

The build failed on a machine where a proxy was required, but succeeded on = a
machine which had direct access to the internet

Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not allow to fetch code from internet during do_compile.

Example result:
...
ERROR: Do not try to fetch `pytest-runner1' for building. Please add it= s native recipe to DEPENDS.
Traceback (most recent call last):
=C2=A0 File "setup.py", line 56, in <module>
...

The improvement is flexible for test_requirements argument (used at
`setup.py test'), where use easy_install also.

[YOCTO #12084]
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
=C2=A0meta/classes/distutils.bbclass=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 1 +
=C2=A0meta/classes/distutils3.bbclass=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 1 +
=C2=A0...ionally-do-not-fetch-code-by-easy_install.patch | 34 ++++++++= ++++++++++++++
=C2=A0meta/recipes-devtools/python/python-setuptools.inc |=C2=A0 2 ++<= br> =C2=A04 files changed, 38 insertions(+)
=C2=A0create mode 100644 meta/recipes-devtools/python/files/0001-condi= tionally-do-not-fetch-code-by-easy_install.patch

diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.<= wbr>bbclass
index 1930c35..c2f7403 100644
--- a/meta/classes/distutils.bbclass
+++ b/meta/classes/distutils.bbclass
@@ -8,6 +8,7 @@ DISTUTILS_INSTALL_ARGS ?=3D "--prefix=3D${D}/${prefix}= \
=C2=A0 =C2=A0 =C2=A0--install-data=3D${D}/${datadir}"

=C2=A0distutils_do_compile() {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0NO_FETCH_BUILD=3D1 \
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 STAGING_INCDIR=3D${STAGING_INCDIR} = \
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 STAGING_LIBDIR=3D${STAGING_LIBDIR} = \
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ${STAGING_BINDIR_NATIVE}/${PYTHON_P= N}-native/${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS} || \
diff --git a/meta/classes/distutils3.bbclass b/meta/classes/distutils3= .bbclass
index d4b92a3..f8aca5e 100644
--- a/meta/classes/distutils3.bbclass
+++ b/meta/classes/distutils3.bbclass
@@ -15,6 +15,7 @@ distutils3_do_configure() {
=C2=A0}

=C2=A0distutils3_do_compile() {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 NO_FETCH_BUILD=3D1 \
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0STAGING_INCDIR=3D${STAGING_INCDIR} \=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0STAGING_LIBDIR=3D${STAGING_LIBDIR} \=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0${STAGING_BINDIR_NATIVE}/${PYTHON_PN= }-native/${PYTHON_PN} setup.py \
diff --git a/meta/recipes-devtools/python/files/0001-conditionall= y-do-not-fetch-code-by-easy_install.patch b/meta/recipes-devtools/python/files/0001-conditionally-do-not-fetch-code-by-easy_instal= l.patch
new file mode 100644
index 0000000..4b17921
--- /dev/null
+++ b/meta/recipes-devtools/python/files/0001-conditionally-do-no= t-fetch-code-by-easy_install.patch
@@ -0,0 +1,34 @@
+From 019422d19c6401fa8f45b1311ecd31bb087c809d Mon Sep 17 00:00:00 200= 1
+From: Hongxu Jia <hongxu.jia@windriver.com>
+Date: Tue, 17 Jul 2018 10:13:38 +0800
+Subject: [PATCH] conditionally do not fetch code by easy_install
+
+If var-NO_FETCH_BUILD is set, do not allow to fetch code from
+internet by easy_install.
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
+---
+ setuptools/command/easy_install.py | 5 +++++
+ 1 file changed, 5 insertions(+)
+
+diff --git a/setuptools/command/easy_install.py b/setuptools/command/= easy_install.py
+index 85ee40f..7eb9d2a 100755
+--- a/setuptools/command/easy_install.py
++++ b/setuptools/command/easy_install.py
+@@ -637,6 +637,11 @@ class easy_install(Command):
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0os.path.exists(tmpdir) and= rmtree(rmtree_safe(tmpdir))
+
+=C2=A0 =C2=A0 =C2=A0def easy_install(self, spec, deps=3DFalse):
++=C2=A0 =C2=A0 =C2=A0 =C2=A0 if os.environ.get('NO_FETCH_BUILD= 9;, None):
++=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 log.error("ERROR: Do not t= ry to fetch `%s' for building. "
++=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 "Please add its native recipe to DEPENDS." % spec)
++=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return None
++
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if not self.editable:
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0self.install_site_py()
+
+--
+2.7.4
+
diff --git a/meta/recipes-devtools/python/python-setuptools.inc b/meta= /recipes-devtools/python/python-setuptools.inc
index 25be446..24b3ac4 100644
--- a/meta/recipes-devtools/python/python-setuptools.inc
+++ b/meta/recipes-devtools/python/python-setuptools.inc
@@ -9,6 +9,8 @@ PYPI_PACKAGE_EXT =3D "zip"

=C2=A0inherit pypi

+SRC_URI_append_class-native =3D " file://0001-conditionally-do-n= ot-fetch-code-by-easy_install.patch"
+
=C2=A0SRC_URI[md5sum] =3D "260630ae1a64bafa39dcc53404d63829&= quot;
=C2=A0SRC_URI[sha256sum] =3D "012adb8e25fbfd64c652e99e7bab58= 799a3aaf05d39ab38561f69190a909015f"

--
2.7.4

--
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded= .org/mailman/listinfo/openembedded-core

--000000000000e320170571fa7485--