Python module and build_ext args

Python module and build_ext args

[Craig McQueen]

I'm trying to make a recipe for python3-uvloop, using setuptools3.

The Python 3 uvloop module depends on libuv. It bundles a version of libuv, and setup.py tries to build it, but it doesn't work well for cross-compilation. However, it also provides a build_ext parameter "--use-system-libuv", which seems to work when I try running it manually in devshell. (I have made a suitable recipe for libuv and added libuv to DEPENDS.)

How can I specify the "--use-system-libuv" parameter for build_ext in the python3-uvloop recipe? I see a reference to DISTUTILS_BUILD_EXT_ARGS, but it doesn't seem to be functional.

-- 
Craig McQueen

Re: Python module and build_ext args

[Craig McQueen]

I wrote:
> I'm trying to make a recipe for python3-uvloop, using setuptools3.
> 
> The Python 3 uvloop module depends on libuv. It bundles a version of libuv,
> and setup.py tries to build it, but it doesn't work well for cross-compilation.
> However, it also provides a build_ext parameter "--use-system-libuv", which
> seems to work when I try running it manually in devshell. (I have made a
> suitable recipe for libuv and added libuv to DEPENDS.)
> 
> How can I specify the "--use-system-libuv" parameter for build_ext in the
> python3-uvloop recipe? I see a reference to DISTUTILS_BUILD_EXT_ARGS,
> but it doesn't seem to be functional.


I see DISTUTILS_BUILD_EXT_ARGS was submitted in this patch:
https://patchwork.openembedded.org/patch/66071/

However, in the Yocto poky repository, I see commit 0221af0f4ee9e8bfb8796841bdf806e38bc600c6 which appears to be a broken version of the above patch with the separate build_ext step not actually executed with the DISTUTILS_BUILD_EXT_ARGS parameters.

-- 
Craig McQueen

Re: Python module and build_ext args

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1398 bytes --]

On Thu, Aug 24, 2017 at 6:48 PM Craig McQueen <craig.mcqueen@innerrange.com>
wrote:

> I wrote:
> > I'm trying to make a recipe for python3-uvloop, using setuptools3.
> >
> > The Python 3 uvloop module depends on libuv. It bundles a version of
> libuv,
> > and setup.py tries to build it, but it doesn't work well for
> cross-compilation.
> > However, it also provides a build_ext parameter "--use-system-libuv",
> which
> > seems to work when I try running it manually in devshell. (I have made a
> > suitable recipe for libuv and added libuv to DEPENDS.)
> >
> > How can I specify the "--use-system-libuv" parameter for build_ext in the
> > python3-uvloop recipe? I see a reference to DISTUTILS_BUILD_EXT_ARGS,
> > but it doesn't seem to be functional.
>
>
> I see DISTUTILS_BUILD_EXT_ARGS was submitted in this patch:
> https://patchwork.openembedded.org/patch/66071/
>
> However, in the Yocto poky repository, I see commit
> 0221af0f4ee9e8bfb8796841bdf806e38bc600c6 which appears to be a broken
> version of the above patch with the separate build_ext step not actually
> executed with the DISTUTILS_BUILD_EXT_ARGS parameters.


You did not explain broken in which sense ?

>
>
> --
> Craig McQueen
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 2141 bytes --]

Re: Python module and build_ext args

[Craig McQueen]

[-- Attachment #1: Type: text/plain, Size: 2272 bytes --]

Khem Raj wrote:

On Thu, Aug 24, 2017 at 6:48 PM Craig McQueen <craig.mcqueen@innerrange.com<mailto:craig.mcqueen@innerrange.com>> wrote:
I wrote:
> I'm trying to make a recipe for python3-uvloop, using setuptools3.
>
> The Python 3 uvloop module depends on libuv. It bundles a version of libuv,
> and setup.py tries to build it, but it doesn't work well for cross-compilation.
> However, it also provides a build_ext parameter "--use-system-libuv", which
> seems to work when I try running it manually in devshell. (I have made a
> suitable recipe for libuv and added libuv to DEPENDS.)
>
> How can I specify the "--use-system-libuv" parameter for build_ext in the
> python3-uvloop recipe? I see a reference to DISTUTILS_BUILD_EXT_ARGS,
> but it doesn't seem to be functional.


I see DISTUTILS_BUILD_EXT_ARGS was submitted in this patch:
https://patchwork.openembedded.org/patch/66071/

However, in the Yocto poky repository, I see commit 0221af0f4ee9e8bfb8796841bdf806e38bc600c6 which appears to be a broken version of the above patch with the separate build_ext step not actually executed with the DISTUTILS_BUILD_EXT_ARGS parameters.

You did not explain broken in which sense ?

It is broken in the sense that: The separate build_ext step is not actually executed with the DISTUTILS_BUILD_EXT_ARGS parameters. The original patch submission contained:

          STAGING_INCDIR=${STAGING_INCDIR} \
          STAGING_LIBDIR=${STAGING_LIBDIR} \
          BUILD_SYS=${BUILD_SYS} HOST_SYS=${HOST_SYS} \
-         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS} || \
+         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py \
+         build_ext --include-dirs ${STAGING_INCDIR}/${PYTHON_DIR}${PYTHON_ABI} \
+         --library-dirs ${STAGING_LIBCDIR}/${PYTHON_DIR} \
+         ${DISTUTILS_BUILD_EXT_ARGS} \
+         build ${DISTUTILS_BUILD_ARGS} || \
+         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py build_ext ${DISTUTILS_BUILD_ARGS} || \
          bbfatal "${PYTHON_PN} setup.py build_ext execution failed."
}

However that part of the patch is not present in commit 0221af0f4ee9e8bfb8796841bdf806e38bc600c6.

--
Craig McQueen


[-- Attachment #2: Type: text/html, Size: 7898 bytes --]

Re: Python module and build_ext args

[Khem Raj]

On Thu, Aug 24, 2017 at 9:42 PM, Craig McQueen
<craig.mcqueen@innerrange.com> wrote:
> Khem Raj wrote:
>
>
>
> On Thu, Aug 24, 2017 at 6:48 PM Craig McQueen <craig.mcqueen@innerrange.com>
> wrote:
>
> I wrote:
>> I'm trying to make a recipe for python3-uvloop, using setuptools3.
>>
>> The Python 3 uvloop module depends on libuv. It bundles a version of
>> libuv,
>> and setup.py tries to build it, but it doesn't work well for
>> cross-compilation.
>> However, it also provides a build_ext parameter "--use-system-libuv",
>> which
>> seems to work when I try running it manually in devshell. (I have made a
>> suitable recipe for libuv and added libuv to DEPENDS.)
>>
>> How can I specify the "--use-system-libuv" parameter for build_ext in the
>> python3-uvloop recipe? I see a reference to DISTUTILS_BUILD_EXT_ARGS,
>> but it doesn't seem to be functional.
>
>
> I see DISTUTILS_BUILD_EXT_ARGS was submitted in this patch:
> https://patchwork.openembedded.org/patch/66071/
>
> However, in the Yocto poky repository, I see commit
> 0221af0f4ee9e8bfb8796841bdf806e38bc600c6 which appears to be a broken
> version of the above patch with the separate build_ext step not actually
> executed with the DISTUTILS_BUILD_EXT_ARGS parameters.
>
>
>
> You did not explain broken in which sense ?
>
>
>
> It is broken in the sense that: The separate build_ext step is not actually
> executed with the DISTUTILS_BUILD_EXT_ARGS parameters. The original patch
> submission contained:
>
>
>
>           STAGING_INCDIR=${STAGING_INCDIR} \
>
>           STAGING_LIBDIR=${STAGING_LIBDIR} \
>
>           BUILD_SYS=${BUILD_SYS} HOST_SYS=${HOST_SYS} \
>
> -         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py
> build ${DISTUTILS_BUILD_ARGS} || \
>
> +         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py
> \
>
> +         build_ext --include-dirs
> ${STAGING_INCDIR}/${PYTHON_DIR}${PYTHON_ABI} \
>
> +         --library-dirs ${STAGING_LIBCDIR}/${PYTHON_DIR} \
>
> +         ${DISTUTILS_BUILD_EXT_ARGS} \
>
> +         build ${DISTUTILS_BUILD_ARGS} || \
>
> +         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py
> build_ext ${DISTUTILS_BUILD_ARGS} || \
>
>           bbfatal "${PYTHON_PN} setup.py build_ext execution failed."
>
> }
>
>
>
> However that part of the patch is not present in commit
> 0221af0f4ee9e8bfb8796841bdf806e38bc600c6.
>

IIRC there were some breakages with this but it was long time ago. Can
you appy this change and test it out ?

Re: Python module and build_ext args

[Craig McQueen]

Khem Raj wrote:
> On Thu, Aug 24, 2017 at 9:42 PM, Craig McQueen
> <craig.mcqueen@innerrange.com> wrote:
> > Khem Raj wrote:
> > On Thu, Aug 24, 2017 at 6:48 PM Craig McQueen
> > <craig.mcqueen@innerrange.com>
> > wrote:
> >
> > I wrote:
> >> I'm trying to make a recipe for python3-uvloop, using setuptools3.
> >>
> >> The Python 3 uvloop module depends on libuv. It bundles a version of
> >> libuv, and setup.py tries to build it, but it doesn't work well for
> >> cross-compilation.
> >> However, it also provides a build_ext parameter "--use-system-libuv",
> >> which seems to work when I try running it manually in devshell. (I
> >> have made a suitable recipe for libuv and added libuv to DEPENDS.)
> >>
> >> How can I specify the "--use-system-libuv" parameter for build_ext in
> >> the python3-uvloop recipe? I see a reference to
> >> DISTUTILS_BUILD_EXT_ARGS, but it doesn't seem to be functional.
> >
> >
> > I see DISTUTILS_BUILD_EXT_ARGS was submitted in this patch:
> > https://patchwork.openembedded.org/patch/66071/
> >
> > However, in the Yocto poky repository, I see commit
> > 0221af0f4ee9e8bfb8796841bdf806e38bc600c6 which appears to be a
> broken
> > version of the above patch with the separate build_ext step not
> > actually executed with the DISTUTILS_BUILD_EXT_ARGS parameters.
> >
> >
> >
> > You did not explain broken in which sense ?
> >
> >
> >
> > It is broken in the sense that: The separate build_ext step is not
> > actually executed with the DISTUTILS_BUILD_EXT_ARGS parameters. The
> > original patch submission contained:
> >
> >
> >
> >           STAGING_INCDIR=${STAGING_INCDIR} \
> >
> >           STAGING_LIBDIR=${STAGING_LIBDIR} \
> >
> >           BUILD_SYS=${BUILD_SYS} HOST_SYS=${HOST_SYS} \
> >
> > -         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
> setup.py
> > build ${DISTUTILS_BUILD_ARGS} || \
> >
> > +         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
> > + setup.py
> > \
> >
> > +         build_ext --include-dirs
> > ${STAGING_INCDIR}/${PYTHON_DIR}${PYTHON_ABI} \
> >
> > +         --library-dirs ${STAGING_LIBCDIR}/${PYTHON_DIR} \
> >
> > +         ${DISTUTILS_BUILD_EXT_ARGS} \
> >
> > +         build ${DISTUTILS_BUILD_ARGS} || \
> >
> > +         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
> > + setup.py
> > build_ext ${DISTUTILS_BUILD_ARGS} || \
> >
> >           bbfatal "${PYTHON_PN} setup.py build_ext execution failed."
> >
> > }
> >
> >
> >
> > However that part of the patch is not present in commit
> > 0221af0f4ee9e8bfb8796841bdf806e38bc600c6.
> >
> 
> IIRC there were some breakages with this but it was long time ago. Can you
> appy this change and test it out ?

I tried applying the change manually. The Yocto do_compile step succeeded. But then the do_install step still failed. It seems that the do_install step (which runs setup.py install) still runs the setup.py build_ext step a second time, without the extra option I was trying to use.

I'm not familiar enough with the working of python3 setup.py to know why the install step re-runs the build_ext step a second time.

-- 
Craig McQueen

Re: Python module and build_ext args

[Craig McQueen]

I wrote:
> Khem Raj wrote:
> > On Thu, Aug 24, 2017 at 9:42 PM, Craig McQueen
> > <craig.mcqueen@innerrange.com> wrote:
> > > Khem Raj wrote:
> > > On Thu, Aug 24, 2017 at 6:48 PM Craig McQueen
> > > <craig.mcqueen@innerrange.com>
> > > wrote:
> > >
> > > I wrote:
> > >> I'm trying to make a recipe for python3-uvloop, using setuptools3.
> > >>
> > >> The Python 3 uvloop module depends on libuv. It bundles a version
> > >> of libuv, and setup.py tries to build it, but it doesn't work well
> > >> for cross-compilation.
> > >> However, it also provides a build_ext parameter
> > >> "--use-system-libuv", which seems to work when I try running it
> > >> manually in devshell. (I have made a suitable recipe for libuv and
> > >> added libuv to DEPENDS.)
> > >>
> > >> How can I specify the "--use-system-libuv" parameter for build_ext
> > >> in the python3-uvloop recipe? I see a reference to
> > >> DISTUTILS_BUILD_EXT_ARGS, but it doesn't seem to be functional.
> > >
> > >
> > > I see DISTUTILS_BUILD_EXT_ARGS was submitted in this patch:
> > > https://patchwork.openembedded.org/patch/66071/
> > >
> > > However, in the Yocto poky repository, I see commit
> > > 0221af0f4ee9e8bfb8796841bdf806e38bc600c6 which appears to be a
> > broken
> > > version of the above patch with the separate build_ext step not
> > > actually executed with the DISTUTILS_BUILD_EXT_ARGS parameters.
> > >
> > >
> > >
> > > You did not explain broken in which sense ?
> > >
> > >
> > >
> > > It is broken in the sense that: The separate build_ext step is not
> > > actually executed with the DISTUTILS_BUILD_EXT_ARGS parameters. The
> > > original patch submission contained:
> > >
> > >
> > >
> > >           STAGING_INCDIR=${STAGING_INCDIR} \
> > >
> > >           STAGING_LIBDIR=${STAGING_LIBDIR} \
> > >
> > >           BUILD_SYS=${BUILD_SYS} HOST_SYS=${HOST_SYS} \
> > >
> > > -         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-
> native/${PYTHON_PN}
> > setup.py
> > > build ${DISTUTILS_BUILD_ARGS} || \
> > >
> > > +         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-
> native/${PYTHON_PN}
> > > + setup.py
> > > \
> > >
> > > +         build_ext --include-dirs
> > > ${STAGING_INCDIR}/${PYTHON_DIR}${PYTHON_ABI} \
> > >
> > > +         --library-dirs ${STAGING_LIBCDIR}/${PYTHON_DIR} \
> > >
> > > +         ${DISTUTILS_BUILD_EXT_ARGS} \
> > >
> > > +         build ${DISTUTILS_BUILD_ARGS} || \
> > >
> > > +         ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-
> native/${PYTHON_PN}
> > > + setup.py
> > > build_ext ${DISTUTILS_BUILD_ARGS} || \
> > >
> > >           bbfatal "${PYTHON_PN} setup.py build_ext execution failed."
> > >
> > > }
> > >
> > >
> > >
> > > However that part of the patch is not present in commit
> > > 0221af0f4ee9e8bfb8796841bdf806e38bc600c6.
> > >
> >
> > IIRC there were some breakages with this but it was long time ago. Can
> > you appy this change and test it out ?
> 
> I tried applying the change manually. The Yocto do_compile step succeeded.
> But then the do_install step still failed. It seems that the do_install step
> (which runs setup.py install) still runs the setup.py build_ext step a second
> time, without the extra option I was trying to use.
> 
> I'm not familiar enough with the working of python3 setup.py to know why
> the install step re-runs the build_ext step a second time.

I wasn't able to find an elegant solution for DISTUTILS_BUILD_EXT_ARGS. I ended up getting a working python3-uvloop recipe a different way, using two patches to the setup.py.

I have posted to the uvloop project regarding this:

https://github.com/MagicStack/uvloop/issues/104

-- 
Craig McQueen