Help (python-jsonschema do_compile Failed)

Help (python-jsonschema do_compile Failed)

[Li, Xin]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

Hi,all

When I bitbake python-jsonschema,the error is occurred in the step of do_compile as following:

------------------------------
DEBUG: Executing shell function do_compile
Download error on https://pypi.python.org/simple/vcversioner/: [Errno -5] No address associated with hostname -- Some packages may not be found!
Couldn't find index page for 'vcversioner' (maybe misspelled?)
Download error on https://pypi.python.org/simple/: [Errno -5] No address associated with hostname -- Some packages may not be found!
No local packages or download links found for vcversioner
Traceback (most recent call last):
  File "setup.py", line 44, in <module>
vcversioner={"version_module_paths" : ["jsonschema/_version.py"]},
-------------------------------
If you know how to fix it, Please tell me.

Regards,
li




[-- Attachment #2: Type: text/html, Size: 4034 bytes --]

Re: Help (python-jsonschema do_compile Failed)

[Mark Asselstine]

On Thu, Dec 3, 2015 at 12:50 AM, Li, Xin <lixin.fnst@cn.fujitsu.com> wrote:
> Hi,all
>
>
>
> When I bitbake python-jsonschema,the error is occurred in the step of
> do_compile as following:
>
>
>
> ------------------------------
>
> DEBUG: Executing shell function do_compile
>
> Download error on https://pypi.python.org/simple/vcversioner/: [Errno -5] No
> address associated with hostname -- Some packages may not be found!
>
> Couldn't find index page for 'vcversioner' (maybe misspelled?)
>
> Download error on https://pypi.python.org/simple/: [Errno -5] No address
> associated with hostname -- Some packages may not be found!
>
> No local packages or download links found for vcversioner
>
> Traceback (most recent call last):
>
>   File "setup.py", line 44, in <module>
>
> vcversioner={"version_module_paths" : ["jsonschema/_version.py"]},
>
> -------------------------------
>
> If you know how to fix it, Please tell me.

A download error during do_compile(). My guess is that we have a
missing dependency. With python, Ruby and some other packages we have
to be careful as their builds will actually cause downloads to happen
if a dependency is missing (python will fetch eggs, ruby will fetch
gems...). When I have time I usually do builds with IP tables rules in
place to force errors when these fetches happen so that I can catch
them and fix them. Downloads should only be done by the fetcher and
not bundler, pdb etc. If you see an error about downloading and you
are not in do_fetch() something is wrong.

Start by adding python-vcversioner as a DEPENDS in python-jsonschema
and you should be good to go.

Mark

>
>
>
> Regards,
>
> li
>
>
>
>
> --
> _______________________________________________
> meta-virtualization mailing list
> meta-virtualization@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/meta-virtualization
>

Re: Help (python-jsonschema do_compile Failed)

[Li, Xin]

Hi, Mark and all,

Thanks a lot for your help.

>> When I bitbake python-jsonschema,the error is occurred in the step of
>> do_compile as following:
>> ------------------------------
>>
>> DEBUG: Executing shell function do_compile
>>
>> Download error on https://pypi.python.org/simple/vcversioner/: [Errno
>> -5] No address associated with hostname -- Some packages may not be found!
>>
>> Couldn't find index page for 'vcversioner' (maybe misspelled?)
>>
>> Download error on https://pypi.python.org/simple/: [Errno -5] No
>> address associated with hostname -- Some packages may not be found!
>>
>> No local packages or download links found for vcversioner
>>
>> Traceback (most recent call last):
>>
>>   File "setup.py", line 44, in <module>
>>
>> vcversioner={"version_module_paths" : ["jsonschema/_version.py"]},
>>
>> -------------------------------
>>
>> If you know how to fix it, Please tell me.
>
>A download error during do_compile(). My guess is that we have a missing
>dependency. With python, Ruby and some other packages we have to be careful
>as their builds will actually cause downloads to happen if a dependency is missing
>(python will fetch eggs, ruby will fetch gems...). When I have time I usually do
>builds with IP tables rules in place to force errors when these fetches happen so
>that I can catch them and fix them. Downloads should only be done by the fetcher
>and not bundler, pdb etc. If you see an error about downloading and you are not
>in do_fetch() something is wrong.
>
>Start by adding python-vcversioner as a DEPENDS in python-jsonschema and you
>should be good to go.
>

The python-vcversioner is already as a DEPENDS in python-jsonschema, and
It is still failed.

Perhaps it is the problem about proxy set.

When I bitbake python-jsonschema, the error occurs in the step of do_compile,
But when I into directory "python-jsonschema/2.5.1-r0/jsonschema-2.5.1/ ", and run.do_complie, no errors occur.
----------------------------- 
$ cd build-x86_64/tmp/work/x86_64 -linux/python-jsonschema/2.5.1-r0/jsonschema-2.5.1/
$ ../temp/run.do_compile
zip_safe flag not set; analyzing archive contents...

Installed ***/work/x86_64 -linux/python-jsonschema/2.5.1-r0/jsonschema-2.5.1/.eggs/vcversioner-2.14.0.0-py2.7.egg
running build
running build_py
creating build
creating build/lib
creating build/lib/jsonschema
......
$ echo $?
0
--------------------------------- 

And I have proxy set in ~/.wgetrc like this:
http_proxy = http://rep.proxy.nic.fujitsu.com:8080/
ftp_proxy = http://rep.proxy.nic.fujitsu.com:8080/
https_proxy=https://rep.proxy.nic.fujitsu.com:8080/

If you know how to fix it, please help me.

Regards,
Li

Re: Help (python-jsonschema do_compile Failed)

[Mark Asselstine]

On Sun, Dec 13, 2015 at 8:46 PM, Li, Xin <lixin.fnst@cn.fujitsu.com> wrote:
> Hi, Mark and all,
>
> Thanks a lot for your help.
>
>>> When I bitbake python-jsonschema,the error is occurred in the step of
>>> do_compile as following:
>>> ------------------------------
>>>
>>> DEBUG: Executing shell function do_compile
>>>
>>> Download error on https://pypi.python.org/simple/vcversioner/: [Errno
>>> -5] No address associated with hostname -- Some packages may not be found!
>>>
>>> Couldn't find index page for 'vcversioner' (maybe misspelled?)
>>>
>>> Download error on https://pypi.python.org/simple/: [Errno -5] No
>>> address associated with hostname -- Some packages may not be found!
>>>
>>> No local packages or download links found for vcversioner
>>>
>>> Traceback (most recent call last):
>>>
>>>   File "setup.py", line 44, in <module>
>>>
>>> vcversioner={"version_module_paths" : ["jsonschema/_version.py"]},
>>>
>>> -------------------------------
>>>
>>> If you know how to fix it, Please tell me.
>>
>>A download error during do_compile(). My guess is that we have a missing
>>dependency. With python, Ruby and some other packages we have to be careful
>>as their builds will actually cause downloads to happen if a dependency is missing
>>(python will fetch eggs, ruby will fetch gems...). When I have time I usually do
>>builds with IP tables rules in place to force errors when these fetches happen so
>>that I can catch them and fix them. Downloads should only be done by the fetcher
>>and not bundler, pdb etc. If you see an error about downloading and you are not
>>in do_fetch() something is wrong.
>>
>>Start by adding python-vcversioner as a DEPENDS in python-jsonschema and you
>>should be good to go.
>>
>
> The python-vcversioner is already as a DEPENDS in python-jsonschema, and
> It is still failed.
>
> Perhaps it is the problem about proxy set.

No, again this would just be band-aid for a real problem. I did have
time to look at this today, but not enough to find a suitable
solution.

The issue is "setup_requires" in setup.py for python-jsonschema. This
is a rarely used directive that has some unexpected consequences. See:

https://github.com/pypa/pip/issues/410
https://pip.pypa.io/en/latest/user_guide/#installation-bundles
(warning at bottom of section)
https://pip.pypa.io/en/latest/reference/pip_install/#controlling-setup-requires

I can reproduce your issue by using iptables rules to DROP packets on
port 80 and 443.

The solution is one of 3.
1) We can cleanup the setup.py to not require vcversioner. This is
most likely the simplest but may have bad side effects.
2) We build vcversioner .egg file in an addition to do_compile() and
add it to the -dev pkg to ensure it is in the sysroot. We then copy
the egg from the sysroot to our build as a step before
python-jsonschema do_compile(). This is most likely the best fix.
3) We build the .egg as in (2) but build an index file in the sysroot,
allowing us to use find_links

I really believe (2) is the best approach. Building the egg and having
it in the sysroot is very straightforward, the copy shouldn't be that
difficult either. I can attempt this but most likely won't have a fix
until tomorrow. For now manually download the egg for vcversioner and
place it in tmp/work/core2-64-poky-linux/python-jsonschema/2.5.1-r0/jsonschema-2.5.1/.eggs/vcversioner-2.14.0.0-py2.7.egg
and you should be able to at least continue to build.

Mark


>
> When I bitbake python-jsonschema, the error occurs in the step of do_compile,
> But when I into directory "python-jsonschema/2.5.1-r0/jsonschema-2.5.1/ ", and run.do_complie, no errors occur.
> -----------------------------
> $ cd build-x86_64/tmp/work/x86_64 -linux/python-jsonschema/2.5.1-r0/jsonschema-2.5.1/
> $ ../temp/run.do_compile
> zip_safe flag not set; analyzing archive contents...
>
> Installed ***/work/x86_64 -linux/python-jsonschema/2.5.1-r0/jsonschema-2.5.1/.eggs/vcversioner-2.14.0.0-py2.7.egg
> running build
> running build_py
> creating build
> creating build/lib
> creating build/lib/jsonschema
> ......
> $ echo $?
> 0
> ---------------------------------
>
> And I have proxy set in ~/.wgetrc like this:
> http_proxy = http://rep.proxy.nic.fujitsu.com:8080/
> ftp_proxy = http://rep.proxy.nic.fujitsu.com:8080/
> https_proxy=https://rep.proxy.nic.fujitsu.com:8080/
>
> If you know how to fix it, please help me.
>
> Regards,
> Li
>
>
> --
> _______________________________________________
> meta-virtualization mailing list
> meta-virtualization@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/meta-virtualization

Re: Help (python-jsonschema do_compile Failed)

[Li, Xin]

Hi, Mark

Thank you very much for your kindly reply.

>>>
>>>A download error during do_compile(). My guess is that we have a
>>>missing dependency. With python, Ruby and some other packages we have
>>>to be careful as their builds will actually cause downloads to happen
>>>if a dependency is missing (python will fetch eggs, ruby will fetch
>>>gems...). When I have time I usually do builds with IP tables rules in
>>>place to force errors when these fetches happen so that I can catch
>>>them and fix them. Downloads should only be done by the fetcher and
>>>not bundler, pdb etc. If you see an error about downloading and you are not in
>do_fetch() something is wrong.
>>>
>>>Start by adding python-vcversioner as a DEPENDS in python-jsonschema
>>>and you should be good to go.
>>>
>>
>> The python-vcversioner is already as a DEPENDS in python-jsonschema,
>> and It is still failed.
>>
>> Perhaps it is the problem about proxy set.
>
>No, again this would just be band-aid for a real problem. I did have time to look at
>this today, but not enough to find a suitable solution.
>
>The issue is "setup_requires" in setup.py for python-jsonschema. This is a rarely
>used directive that has some unexpected consequences. See:
>
>https://github.com/pypa/pip/issues/410
>https://pip.pypa.io/en/latest/user_guide/#installation-bundles
>(warning at bottom of section)
>https://pip.pypa.io/en/latest/reference/pip_install/#controlling-setup-requires
>
>I can reproduce your issue by using iptables rules to DROP packets on port 80 and
>443.
>
>The solution is one of 3.
>1) We can cleanup the setup.py to not require vcversioner. This is most likely the
>simplest but may have bad side effects.
>2) We build vcversioner .egg file in an addition to do_compile() and add it to the
>-dev pkg to ensure it is in the sysroot. We then copy the egg from the sysroot to
>our build as a step before python-jsonschema do_compile(). This is most likely the
>best fix.
>3) We build the .egg as in (2) but build an index file in the sysroot, allowing us to
>use find_links
>
>I really believe (2) is the best approach. Building the egg and having it in the
>sysroot is very straightforward, the copy shouldn't be that difficult either. I can
>attempt this but most likely won't have a fix until tomorrow. For now manually
>download the egg for vcversioner and place it in
>tmp/work/core2-64-poky-linux/python-jsonschema/2.5.1-r0/jsonschema-2.5.1/.e
>ggs/vcversioner-2.14.0.0-py2.7.egg
>and you should be able to at least continue to build.

I chose (2) solution. And it is build with no errors, so how about this Modify?

1. Modify python-vcversioner_2.14.0.0.bb, and create vcversioner-2.14.0.0-py2.7.egg by do_compile_append() .
And then install it. 
-------------------------------------
diff --git a/meta-openstack/recipes-devtools/python/python-vcversioner_2.14.0.0.bb b/meta-openstack/recipes-devtools/p
index 2fe96ab..b8131b1 100644
......
-inherit setuptools
+inherit setuptools pythonnative
+
+do_compile_append() {
+    export BUILD_SYS=${BUILD_SYS} HOST_SYS=${HOST_SYS}
+    ${PYTHON} setup.py -q bdist_egg --dist-dir ./
+}
+
+do_install_append() {
+    install -m 0644 ${S}/vcversioner-2.14.0.0-py2.7.egg ${D}/${PYTHON_SITEPACKAGES_DIR}/
+}
--------------------------------------

2. Modify python-jsonschema_2.5.1.bb, copy vcversioner-2.14.0.0-py2.7.egg from sysroot to ${S}/.eggs/
-------------------------------------- 
$ git diff meta-openstack/recipes-devtools/python/python-jsonschema_2.5.1.bb
diff --git a/meta-openstack/recipes-devtools/python/python-jsonschema_2.5.1.bb b/meta-openstack/recipes-devtools/pytho
index a3a7ec0..a2055c8 100644
--- a/meta-openstack/recipes-devtools/python/python-jsonschema_2.5.1.bb
+++ b/meta-openstack/recipes-devtools/python/python-jsonschema_2.5.1.bb
@@ -16,4 +16,9 @@ S = "${WORKDIR}/${SRCNAME}-${PV}"

 inherit setuptools

-DEPENDS += "python-vcversioner"
+DEPENDS += "python-vcversioner python-pip"
+
+do_compile_prepend() {
+    install -d ${S}/.eggs
+    install -m 0644 ${STAGING_DIR_HOST}/${PYTHON_SITEPACKAGES_DIR}/vcversioner-2.14.0.0-py2.7.egg ${S}/.eggs/
+}
-------------------------------------- 

Thanks a lot for your help.

Regards,
Li

>
>>
>> When I bitbake python-jsonschema, the error occurs in the step of
>> do_compile, But when I into directory
>"python-jsonschema/2.5.1-r0/jsonschema-2.5.1/ ", and run.do_complie, no
>errors occur.
>> -----------------------------
>> $ cd build-x86_64/tmp/work/x86_64
>> -linux/python-jsonschema/2.5.1-r0/jsonschema-2.5.1/
>> $ ../temp/run.do_compile
>> zip_safe flag not set; analyzing archive contents...
>>
>> Installed ***/work/x86_64
>> -linux/python-jsonschema/2.5.1-r0/jsonschema-2.5.1/.eggs/vcversioner-2
>> .14.0.0-py2.7.egg
>> running build
>> running build_py
>> creating build
>> creating build/lib
>> creating build/lib/jsonschema
>> ......
>> $ echo $?
>> 0
>> ---------------------------------
>>
>> And I have proxy set in ~/.wgetrc like this:
>> http_proxy = http://rep.proxy.nic.fujitsu.com:8080/
>> ftp_proxy = http://rep.proxy.nic.fujitsu.com:8080/
>> https_proxy=https://rep.proxy.nic.fujitsu.com:8080/
>>
>> If you know how to fix it, please help me.
>>
>> Regards,
>> Li
>>
>>
>> --
>> _______________________________________________
>> meta-virtualization mailing list
>> meta-virtualization@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/meta-virtualization
>