[Buildroot] Python standard library problems

[Buildroot] Python standard library problems

[Pedro Sanchez]

Hello,

I'm testing Python 2.7.1 as available in current BR and I find that I 
can't load several standard modules that otherwise seem to be present.

# python
Python 2.7.1 (r271:86832, Aug  4 2011, 16:19:27)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> import socket
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/socket.py", line 47, in <module>
ImportError: No module named _socket

 >>> import fcntl
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
ImportError: No module named fcntl

 >>> from subprocess import Popen, PIPE
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/subprocess.py", line 428, in <module>
ImportError: No module named select


Any ideas what's going on? This is on BR from git.

-- 
Pedro

[Buildroot] Python standard library problems

[Dominic Newton]

Hi,

What version of gcc are you using and what is your target? I have python 
fully functional on x86 and arm platforms with latest buildroot and use 
all 3 modules you are having problems with.

One thing that can help is the setup.py file in the python build 
directory manages the build of the compiled modules like _socket, and 
the output from this file may give a clue on any missing dependencies or 
include files.

Regards

Dominic Newton
In 2 Technology Ltd.

On 10/08/2011 20:18, Pedro Sanchez wrote:
> Hello,
>
> I'm testing Python 2.7.1 as available in current BR and I find that I
> can't load several standard modules that otherwise seem to be present.
>
> # python
> Python 2.7.1 (r271:86832, Aug 4 2011, 16:19:27)
> [GCC 4.5.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import socket
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/lib/python2.7/socket.py", line 47, in <module>
> ImportError: No module named _socket
>
>  >>> import fcntl
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> ImportError: No module named fcntl
>
>  >>> from subprocess import Popen, PIPE
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/lib/python2.7/subprocess.py", line 428, in <module>
> ImportError: No module named select
>
>
> Any ideas what's going on? This is on BR from git.
>

[Buildroot] Python standard library problems

[Pedro Sanchez]

My build is done with the CodeSourcery ARM 2011.03 (gcc 4.5) external 
toolchain which is downloaded automatically by BR.

As for the build logs, can you tell me which file to look for? The only 
log file I see is output/build/python-2.7.1/config.log, a large file 
that doesn't seem to report any errors.

Does your BR use the native buildroot toolchain? I'll run a build with 
this toolchain to see it it makes any difference.

-- 
Pedro


On 08/10/2011 05:48 PM, Dominic Newton wrote:
> Hi,
>
> What version of gcc are you using and what is your target? I have python
> fully functional on x86 and arm platforms with latest buildroot and use
> all 3 modules you are having problems with.
>
> One thing that can help is the setup.py file in the python build
> directory manages the build of the compiled modules like _socket, and
> the output from this file may give a clue on any missing dependencies or
> include files.
>
> Regards
>
> Dominic Newton
> In 2 Technology Ltd.
>
> On 10/08/2011 20:18, Pedro Sanchez wrote:
>> Hello,
>>
>> I'm testing Python 2.7.1 as available in current BR and I find that I
>> can't load several standard modules that otherwise seem to be present.
>>
>> # python
>> Python 2.7.1 (r271:86832, Aug 4 2011, 16:19:27)
>> [GCC 4.5.2] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import socket
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File "/usr/lib/python2.7/socket.py", line 47, in <module>
>> ImportError: No module named _socket
>>
>> >>> import fcntl
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> ImportError: No module named fcntl
>>
>> >>> from subprocess import Popen, PIPE
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File "/usr/lib/python2.7/subprocess.py", line 428, in <module>
>> ImportError: No module named select
>>
>>
>> Any ideas what's going on? This is on BR from git.
>>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] Python standard library problems

[Dominic Newton]

Hi

Yes I'm using the native toolchain. One thing to check might be any 
dependencies the modules you are trying to build - for example have you 
got appropriate libraries being built before python for networking which 
will get _socket built.

The easiest way to look at the setup.py output is to redirect the stdout 
to a file and then inspect it, any messages regarding module builds will 
be there. If python cannot build a module it won't stop, but just sends 
some output to the console. It is possible to edit setup.py to output 
results direct to a log file, but this can take a while to sort out.

Not sure if this is much help as this has (so far) always worked for me 
out of the box for a number of targets.

Dominic Newton
In 2 Technology Ltd.

[Buildroot] Python standard library problems

[Pedro Sanchez]

Well, I'm not trying to build specific Python modules. I'm just 
selecting the Python interpreter from BR's config menu, that's all. I 
was expecting to get a full interpreter with support for the Python 
standard library.

I'm trying to build BR now using the crosstool-ng toolchain ... If I run 
into similar problems then I'll see how to hook up into setup.py's 
output to see what's going on.

-- 
Pedro


On 08/11/2011 11:37 AM, Dominic Newton wrote:
> Hi
>
> Yes I'm using the native toolchain. One thing to check might be any
> dependencies the modules you are trying to build - for example have you
> got appropriate libraries being built before python for networking which
> will get _socket built.
>
> The easiest way to look at the setup.py output is to redirect the stdout
> to a file and then inspect it, any messages regarding module builds will
> be there. If python cannot build a module it won't stop, but just sends
> some output to the console. It is possible to edit setup.py to output
> results direct to a log file, but this can take a while to sort out.
>
> Not sure if this is much help as this has (so far) always worked for me
> out of the box for a number of targets.
>
> Dominic Newton
> In 2 Technology Ltd.
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] Python standard library problems

[Pedro Sanchez]

I built BR with crosstool-ng and with the native toolchain (uClibc) and 
I get the same results. I cloned the git repo three times to start from 
scratch each time, just changing the toolchain selection.

In my view building Python in current's BR is broken  :-(

-- 
Pedro


On 08/11/2011 02:23 PM, Pedro Sanchez wrote:
> Well, I'm not trying to build specific Python modules. I'm just
> selecting the Python interpreter from BR's config menu, that's all. I
> was expecting to get a full interpreter with support for the Python
> standard library.
>
> I'm trying to build BR now using the crosstool-ng toolchain ... If I run
> into similar problems then I'll see how to hook up into setup.py's
> output to see what's going on.
>

[Buildroot] Python standard library problems

[Maxime Ripard]

Hi all,

With the following config, I have the same problem that Pedro encountered.

When you look closer to the build output, when building python, at the
end of the process, you have :

Failed to build these modules:
_bisect            _codecs_iso2022    _collections
_csv               _ctypes            _ctypes_test
_elementtree       _functools         _heapq
_hotshot           _io                _json
_locale            _lsprof            _md5
_multibytecodec    _multiprocessing   _random
_sha               _sha256            _sha512
_socket            _struct            _testcapi
array              audioop            binascii
cmath              cPickle            crypt
cStringIO          datetime           dl
fcntl              future_builtins    grp
imageop            itertools          linuxaudiodev
math               mmap               operator
ossaudiodev        parser             resource
select             spwd               strop
syslog             termios            time
unicodedata

Which is a *lot* of modules. It might be "normal" for some of them, like
ossaudiodev which is likely to have an unmet dependency, but note that
all three modules Pedro have mentionned are here.
Some of them are deprecated too, like imageop.

It would be ideal if we could manage to drop the number of failed build
modules to 0.

I'll try to look into it.

Maxime

-- 
Maxime Ripard, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: .config
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20110812/d0a68980/attachment-0001.ksh>

[Buildroot] Python standard library problems

[Maxime Ripard]

Ok,

A little update on that.

From my tests, this brokenness occurs only when using a 64-bits system.
32 bits systems are fine, which explains why some have a working python
and other don't.

To complete a bit what I said earlier, in the build log, you have a
bunch of
Include/pyport.h:849:2: error: #error "LONG_BIT definition appears wrong
for platform (bad gcc/glibc config?)."

One for every module that fails to build.

This error comes from the code below in pyport.h

#ifndef LONG_BIT
#define LONG_BIT (8 * SIZEOF_LONG)
#endif

#if LONG_BIT != 8 * SIZEOF_LONG
/* 04-Oct-2000 LONG_BIT is apparently (mis)defined as 64 on some recent
 * 32-bit platforms using gcc.  We try to catch that here at compile-time
 * rather than waiting for integer multiplication to trigger bogus
 * overflows.
 */
#error "LONG_BIT definition appears wrong for platform (bad gcc/glibc
config?)."
#endif

LONG_BIT seems correctly defined to 32 in bits/xopen_lim.h of the toolchain.

I'm a bit confused on why this is happening...

On 12/08/2011 12:01, Maxime Ripard wrote:
> Hi all,
> 
> With the following config, I have the same problem that Pedro encountered.
> 
> When you look closer to the build output, when building python, at the
> end of the process, you have :
> 
> Failed to build these modules:
> _bisect            _codecs_iso2022    _collections
> _csv               _ctypes            _ctypes_test
> _elementtree       _functools         _heapq
> _hotshot           _io                _json
> _locale            _lsprof            _md5
> _multibytecodec    _multiprocessing   _random
> _sha               _sha256            _sha512
> _socket            _struct            _testcapi
> array              audioop            binascii
> cmath              cPickle            crypt
> cStringIO          datetime           dl
> fcntl              future_builtins    grp
> imageop            itertools          linuxaudiodev
> math               mmap               operator
> ossaudiodev        parser             resource
> select             spwd               strop
> syslog             termios            time
> unicodedata
> 
> Which is a *lot* of modules. It might be "normal" for some of them, like
> ossaudiodev which is likely to have an unmet dependency, but note that
> all three modules Pedro have mentionned are here.
> Some of them are deprecated too, like imageop.
> 
> It would be ideal if we could manage to drop the number of failed build
> modules to 0.
> 
> I'll try to look into it.
> 
> Maxime
> 


-- 
Maxime Ripard, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Python standard library problems

[Pedro Sanchez]

Thanks Maxime for looking at this. Indeed, my workstation is running 
Linux 64 bits. I'm glad to know that cross-compiling Python goes well on 
a 32-bit OS. I'll have to fire up a VM just to run BR :-|

On the specific problem we have, I don't have any insight yet. All I can 
tell you is that I did the exercise of building BR w/Python using three 
different toolchains (CodeSourcery, Crosstool-ng, and uClibC) and I got 
exactly the same disappointing results**.

Maybe this can help?

http://blog.devork.be/2009/02/compiling-32-bit-python-on-amd64.html

-- 
Pedro

** BTW, uClibC fails to generate the toolchain when the target is arm 
Cortex-A8, just in case you try by chance.



On 08/12/2011 11:44 AM, Maxime Ripard wrote:
> Ok,
>
> A little update on that.
>
>  From my tests, this brokenness occurs only when using a 64-bits system.
> 32 bits systems are fine, which explains why some have a working python
> and other don't.
>
> To complete a bit what I said earlier, in the build log, you have a
> bunch of
> Include/pyport.h:849:2: error: #error "LONG_BIT definition appears wrong
> for platform (bad gcc/glibc config?)."
>
> One for every module that fails to build.
>
> This error comes from the code below in pyport.h
>
> #ifndef LONG_BIT
> #define LONG_BIT (8 * SIZEOF_LONG)
> #endif
>
> #if LONG_BIT != 8 * SIZEOF_LONG
> /* 04-Oct-2000 LONG_BIT is apparently (mis)defined as 64 on some recent
>   * 32-bit platforms using gcc.  We try to catch that here at compile-time
>   * rather than waiting for integer multiplication to trigger bogus
>   * overflows.
>   */
> #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc
> config?)."
> #endif
>
> LONG_BIT seems correctly defined to 32 in bits/xopen_lim.h of the toolchain.
>
> I'm a bit confused on why this is happening...
>
> On 12/08/2011 12:01, Maxime Ripard wrote:
>> Hi all,
>>
>> With the following config, I have the same problem that Pedro encountered.
>>
>> When you look closer to the build output, when building python, at the
>> end of the process, you have :
>>
>> Failed to build these modules:
>> _bisect            _codecs_iso2022    _collections
>> _csv               _ctypes            _ctypes_test
>> _elementtree       _functools         _heapq
>> _hotshot           _io                _json
>> _locale            _lsprof            _md5
>> _multibytecodec    _multiprocessing   _random
>> _sha               _sha256            _sha512
>> _socket            _struct            _testcapi
>> array              audioop            binascii
>> cmath              cPickle            crypt
>> cStringIO          datetime           dl
>> fcntl              future_builtins    grp
>> imageop            itertools          linuxaudiodev
>> math               mmap               operator
>> ossaudiodev        parser             resource
>> select             spwd               strop
>> syslog             termios            time
>> unicodedata
>>
>> Which is a *lot* of modules. It might be "normal" for some of them, like
>> ossaudiodev which is likely to have an unmet dependency, but note that
>> all three modules Pedro have mentionned are here.
>> Some of them are deprecated too, like imageop.
>>
>> It would be ideal if we could manage to drop the number of failed build
>> modules to 0.
>>
>> I'll try to look into it.
>>
>> Maxime
>>
>
>


-- 
Pedro

[Buildroot] Python standard library problems

[Maxime Ripard]

Hi,

On 12/08/2011 19:47, Pedro Sanchez wrote:
> Thanks Maxime for looking at this. Indeed, my workstation is running
> Linux 64 bits. I'm glad to know that cross-compiling Python goes well on
> a 32-bit OS. I'll have to fire up a VM just to run BR :-|

Or maybe we can just fix this bug :)

> On the specific problem we have, I don't have any insight yet. All I can
> tell you is that I did the exercise of building BR w/Python using three
> different toolchains (CodeSourcery, Crosstool-ng, and uClibC) and I got
> exactly the same disappointing results**.

Neither am I. I don't get why a cross-toolchain which runs fine could
fail to build a specific module. After all, on 32 and 64 bits, we use
the exact same toolchains (at least for the Code Sourcery one.).

After all, the host-python should be compiled for 64 bits using the
native toolchain, and the target one compiled (in our case at least) for
32 bits, with the cross-toolchain. There shouldn't be any 64-to-32 bits
compilation at all.

> Maybe this can help?
> 
> http://blog.devork.be/2009/02/compiling-32-bit-python-on-amd64.html

Yep, I ran into that trick last week too. I'm not sure this is a good
one though. This is a good quick fix, but I wonder what will happen if
your target is a 64bits architecture ? Moreover, this adds the
dependency to gcc-multilib.

Peter, what do you think of it ?

Maxime

-- 
Maxime Ripard, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Python standard library problems

[Maxime Ripard]

Hi Pedro,

I've just sent a patch for this issue, could you test it ?

Maxime

On 16/08/2011 10:22, Maxime Ripard wrote:
> Hi,
> 
> On 12/08/2011 19:47, Pedro Sanchez wrote:
>> Thanks Maxime for looking at this. Indeed, my workstation is running
>> Linux 64 bits. I'm glad to know that cross-compiling Python goes well on
>> a 32-bit OS. I'll have to fire up a VM just to run BR :-|
> 
> Or maybe we can just fix this bug :)
> 
>> On the specific problem we have, I don't have any insight yet. All I can
>> tell you is that I did the exercise of building BR w/Python using three
>> different toolchains (CodeSourcery, Crosstool-ng, and uClibC) and I got
>> exactly the same disappointing results**.
> 
> Neither am I. I don't get why a cross-toolchain which runs fine could
> fail to build a specific module. After all, on 32 and 64 bits, we use
> the exact same toolchains (at least for the Code Sourcery one.).
> 
> After all, the host-python should be compiled for 64 bits using the
> native toolchain, and the target one compiled (in our case at least) for
> 32 bits, with the cross-toolchain. There shouldn't be any 64-to-32 bits
> compilation at all.
> 
>> Maybe this can help?
>>
>> http://blog.devork.be/2009/02/compiling-32-bit-python-on-amd64.html
> 
> Yep, I ran into that trick last week too. I'm not sure this is a good
> one though. This is a good quick fix, but I wonder what will happen if
> your target is a 64bits architecture ? Moreover, this adds the
> dependency to gcc-multilib.
> 
> Peter, what do you think of it ?
> 
> Maxime
> 


-- 
Maxime Ripard, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Python standard library problems

[Pedro Sanchez]

Thanks for the patch. I think it's been tested since I see it's 
committed already. But I'll give a try anyway and will let you know.

Thanks!,

-- 
Pedro
On 08/29/2011 11:58 AM, Maxime Ripard wrote:
> Hi Pedro,
>
> I've just sent a patch for this issue, could you test it ?
>
> Maxime
>
> On 16/08/2011 10:22, Maxime Ripard wrote:
>> Hi,
>>
>> On 12/08/2011 19:47, Pedro Sanchez wrote:
>>> Thanks Maxime for looking at this. Indeed, my workstation is running
>>> Linux 64 bits. I'm glad to know that cross-compiling Python goes well on
>>> a 32-bit OS. I'll have to fire up a VM just to run BR :-|
>>
>> Or maybe we can just fix this bug :)
>>
>>> On the specific problem we have, I don't have any insight yet. All I can
>>> tell you is that I did the exercise of building BR w/Python using three
>>> different toolchains (CodeSourcery, Crosstool-ng, and uClibC) and I got
>>> exactly the same disappointing results**.
>>
>> Neither am I. I don't get why a cross-toolchain which runs fine could
>> fail to build a specific module. After all, on 32 and 64 bits, we use
>> the exact same toolchains (at least for the Code Sourcery one.).
>>
>> After all, the host-python should be compiled for 64 bits using the
>> native toolchain, and the target one compiled (in our case at least) for
>> 32 bits, with the cross-toolchain. There shouldn't be any 64-to-32 bits
>> compilation at all.
>>
>>> Maybe this can help?
>>>
>>> http://blog.devork.be/2009/02/compiling-32-bit-python-on-amd64.html
>>
>> Yep, I ran into that trick last week too. I'm not sure this is a good
>> one though. This is a good quick fix, but I wonder what will happen if
>> your target is a 64bits architecture ? Moreover, this adds the
>> dependency to gcc-multilib.
>>
>> Peter, what do you think of it ?
>>
>> Maxime
>>
>
>


-- 
Pedro

[Buildroot] Python standard library problems

[Pedro Sanchez]

On 08/30/2011 10:36 AM, Pedro Sanchez wrote:
> Thanks for the patch. I think it's been tested since I see it's
> committed already. But I'll give a try anyway and will let you know.
>
> Thanks!,
>

Buildroot (fresh from git repo) built with the CodeSourcery compiler on 
my Ubuntu 10.04 LTS 64-bit worstation.

Tested loading the Python modules of my application. Only one module 
fails to load now.

$ python
Python 2.7.1 (r271:86832, Aug 31 2011, 09:38:18)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

 >>> import fcntl
 >>> import os
 >>> import sys
 >>> import signal
 >>> import socket
 >>> import time
 >>> import datetime
 >>> from collections import deque
 >>> from subprocess import Popen, PIPE
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/subprocess.py", line 432, in <module>
   File "/usr/lib/python2.7/pickle.py", line 1266, in <module>
ImportError: No module named binascii


-- 
Pedro

[Buildroot] Python standard library problems

[Thomas Petazzoni]

Le Wed, 31 Aug 2011 10:17:23 -0400,
Pedro Sanchez <psanchez@fosstel.com> a ?crit :

> Tested loading the Python modules of my application. Only one module 
> fails to load now.
> 
> $ python
> Python 2.7.1 (r271:86832, Aug 31 2011, 09:38:18)
> [GCC 4.5.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> 
>  >>> import fcntl
>  >>> import os
>  >>> import sys
>  >>> import signal
>  >>> import socket
>  >>> import time
>  >>> import datetime
>  >>> from collections import deque
>  >>> from subprocess import Popen, PIPE
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "/usr/lib/python2.7/subprocess.py", line 432, in <module>
>    File "/usr/lib/python2.7/pickle.py", line 1266, in <module>
> ImportError: No module named binascii

Strange, I've built Python for ARM with the latest git, and importing
subprocess just works fine.

Could you post the complete build log somewhere (not to the list but on
some pastebin site) :

make clean
make 2>&1 | tee logfile

and then post logfile.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Python standard library problems

[Pedro Sanchez]

On 09/01/2011 03:50 AM, Thomas Petazzoni wrote:
> Le Wed, 31 Aug 2011 10:17:23 -0400,
> Pedro Sanchez<psanchez@fosstel.com>  a ?crit :
>
>> Tested loading the Python modules of my application. Only one module
>> fails to load now.
>>
>> $ python
>> Python 2.7.1 (r271:86832, Aug 31 2011, 09:38:18)
>> [GCC 4.5.2] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>
>>   >>>  import fcntl
>>   >>>  import os
>>   >>>  import sys
>>   >>>  import signal
>>   >>>  import socket
>>   >>>  import time
>>   >>>  import datetime
>>   >>>  from collections import deque
>>   >>>  from subprocess import Popen, PIPE
>> Traceback (most recent call last):
>>     File "<stdin>", line 1, in<module>
>>     File "/usr/lib/python2.7/subprocess.py", line 432, in<module>
>>     File "/usr/lib/python2.7/pickle.py", line 1266, in<module>
>> ImportError: No module named binascii
>
> Strange, I've built Python for ARM with the latest git, and importing
> subprocess just works fine.
>
> Could you post the complete build log somewhere (not to the list but on
> some pastebin site) :
>
> make clean
> make 2>&1 | tee logfile
>
> and then post logfile.
>
> Thomas

Hi,

The log says:

Failed to build these modules:
binascii           readline

And when building binascii it says:

Include/pyport.h:849:2: error: #error "LONG_BIT definition appears wrong 
for platform (bad gcc/glibc config?)."


The fact that also readline fails to build explains why I can't use the 
cursor keys when using the Python interpreter on my ARM board.

The full log is here (copy and paste to a local file):

http://pastebin.me/746c47fed3b257dbced48473b01ab27c


-- 
Pedro