Building Custom Python 3 Packages

Building Custom Python 3 Packages

[Seilis, Aaron]

Hi,

I'm building a couple of custom Python 3 scripts+modules as part of the -native tools required to build a number of my embedded components. I'm having difficulties getting my recipe to build when I've run `devtool modify` but the build works fine in a regular build. The Python tools are stored in various Git repositories and use the typical setuptools infrastructure.

My recipe is roughly as follows:

> inherit setuptools3
>
> SUMMARY = "My tool"
> DESCRIPTION = "Description"
>
> SECTION = "devel/scripts"
> LICENSE = "CLOSED"
>
> SRC_URI = "my-URL"
> SRCREV = "git hash"
>
> S = "${WORKDIR}/git/"
> BBCLASSEXTEND = "native"

This builds fine when running Bitbake normally; the tool is built and can be used to build other components. However, when I run `devtool modify`, the ${S} variable gets set to my alternate source location but the ${B} variable remains under ${WORKDIR}. This causes the build to fail with the error:

> ERROR: mytool-native-1.3.5-r0 do_install: python3 setup.py install execution failed.
> ERROR: mytool-native-1.3.5-r0 do_install: Function failed: do_install (log file is located at /.../build/tmp/work/x86_64-linux/mytool-native/1.3.5-r0/temp/log.do_install.39172)
> ERROR: Logfile of failure stored in: /.../build/tmp/work/x86_64-linux/mytool-native/1.3.5-r0/temp/log.do_install.39172
> Log data follows:
> | DEBUG: Executing shell function do_install
> | /.../build/tmp/sysroots/x86_64-linux/usr/bin/python3-native/python3: can't open file 'setup.py': [Errno 2] No such file or directory
> | ERROR: python3 setup.py install execution failed.
> ERROR: Function failed: do_install (log file is located at /.../build/tmp/work/x86_64-linux/mytool-native/1.3.5-r0/temp/log.do_install.39172)
> ERROR: Task 0 (virtual:native:/.../sources/meta-myproject/recipes-tools/mytool/mytool_1.3.5.bb, do_install) failed with exit code '1'

Looking at the build recipes for Python, shows that the do_compile() step for Python modules (with setuptools3) is:

>distutils3_do_compile() {
>        if [ x86_64-linux != aarch64-WaveserverOS-linux ]; then
>                SYS=wcs
>        else
>                SYS=aarch64-WaveserverOS-linux
>        fi
>        STAGING_INCDIR=/localdata/projects/waveserver/build/tmp/sysroots/wcs/usr/include \
>        STAGING_LIBDIR=/localdata/projects/waveserver/build/tmp/sysroots/wcs/usr/lib \
>        BUILD_SYS=x86_64-linux HOST_SYS=${SYS} \
>        /localdata/projects/waveserver/build/tmp/sysroots/x86_64-linux/usr/bin/python3-native/python3 setup.py \
>        build  || \
>        bbfatal_log "python3 setup.py build_ext execution failed."
>}

This clearly indicates that the issue is that the build is looking for setup.py in the ${B} location, but it is only present in the ${S} location when `devtool modify` has been run. I have tried setting ${B} to ${S} explicitly in the recipe, but this doesn't result in ${B} being changed when I run `bitbake -e mytool`. I could always copy ${S} to ${B} in the recipe, but that seems a bit hack-ish.

Did I miss something or is there another way that Python builds are intended to work?

Thanks,
Aaron

Re: Building Custom Python 3 Packages

[Alexander Kanavin]

On 08/31/2017 04:54 PM, Seilis, Aaron wrote:
> This clearly indicates that the issue is that the build is looking
> for setup.py in the ${B} location, but it is only present in the ${S}
> location when `devtool modify` has been run. I have tried setting
> ${B} to ${S} explicitly in the recipe, but this doesn't result in
> ${B} being changed when I run `bitbake -e mytool`. I could always
> copy ${S} to ${B} in the recipe, but that seems a bit hack-ish.
> 
> Did I miss something or is there another way that Python builds are
> intended to work?

I think this might be a limitation of setuptools: they do not support 
out-of-tree builds (which is a must for devtool modify'). If you can 
figure out how to solve it, that would be nice!

Alex

Re: Building Custom Python 3 Packages

[Paul Eggleton]

On Saturday, 2 September 2017 12:42:09 AM NZST Alexander Kanavin wrote:
> On 08/31/2017 04:54 PM, Seilis, Aaron wrote:
> > This clearly indicates that the issue is that the build is looking
> > for setup.py in the ${B} location, but it is only present in the ${S}
> > location when `devtool modify` has been run. I have tried setting
> > ${B} to ${S} explicitly in the recipe, but this doesn't result in
> > ${B} being changed when I run `bitbake -e mytool`. I could always
> > copy ${S} to ${B} in the recipe, but that seems a bit hack-ish.
> > 
> > Did I miss something or is there another way that Python builds are
> > intended to work?
> 
> I think this might be a limitation of setuptools: they do not support 
> out-of-tree builds (which is a must for devtool modify'). If you can 
> figure out how to solve it, that would be nice!

devtool modify does not require out-of-tree builds - I think in this instance 
it's assuming that's OK, and it's supposed to be following what the recipe 
originally used so that may be a bug. The workaround is to tell it to use the 
same directory with the -s/--same-dir option.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre