* [PATCH] distutils: pass along parallel make flags to setup.py build
@ 2019-10-22 23:59 Nick Owens
2019-10-23 10:56 ` Ross Burton
2019-10-23 17:59 ` [PATCH v2] " Nick Owens
0 siblings, 2 replies; 7+ messages in thread
From: Nick Owens @ 2019-10-22 23:59 UTC (permalink / raw)
To: openembedded-core
parallel builds for native code in python modules was added about 5
years ago. distutils understands '-j N', so just pass along
${PARALLEL_MAKE} to setup.py build.
Signed-off-by: Nick Owens <nick.owens@eero.com>
---
meta/classes/distutils.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.bbclass
index b5c9c2fbbd..d60f7083ef 100644
--- a/meta/classes/distutils.bbclass
+++ b/meta/classes/distutils.bbclass
@@ -23,7 +23,7 @@ distutils_do_compile() {
NO_FETCH_BUILD=1 \
STAGING_INCDIR=${STAGING_INCDIR} \
STAGING_LIBDIR=${STAGING_LIBDIR} \
- ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS} || \
+ ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py build ${PARALLEL_MAKE} ${DISTUTILS_BUILD_ARGS} || \
bbfatal_log "'${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS}' execution failed."
}
--
2.17.1
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH] distutils: pass along parallel make flags to setup.py build
2019-10-22 23:59 [PATCH] distutils: pass along parallel make flags to setup.py build Nick Owens
@ 2019-10-23 10:56 ` Ross Burton
2019-10-23 17:59 ` [PATCH v2] " Nick Owens
1 sibling, 0 replies; 7+ messages in thread
From: Ross Burton @ 2019-10-23 10:56 UTC (permalink / raw)
To: openembedded-core
On 23/10/2019 00:59, Nick Owens wrote:
> parallel builds for native code in python modules was added about 5
> years ago. distutils understands '-j N', so just pass along
> ${PARALLEL_MAKE} to setup.py build.
Does distutils *only* understand -j? It's not unusual to set
PARALLEL_MAKE to something like '-j10 -l5' to factor in the load average.
If distutils doesn't handle -l then it's safer to use a function in
oe.utils instead:
... ${@oe.utils.parallel_make_argument(d, "-j %d")} ...
Ross
^ permalink raw reply [flat|nested] 7+ messages in thread* [PATCH v2] distutils: pass along parallel make flags to setup.py build
2019-10-22 23:59 [PATCH] distutils: pass along parallel make flags to setup.py build Nick Owens
2019-10-23 10:56 ` Ross Burton
@ 2019-10-23 17:59 ` Nick Owens
2019-10-23 17:59 ` [PATCH] " Nick Owens
1 sibling, 1 reply; 7+ messages in thread
From: Nick Owens @ 2019-10-23 17:59 UTC (permalink / raw)
To: openembedded-core
thanks ross, i've updated my patch.
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH] distutils: pass along parallel make flags to setup.py build
2019-10-23 17:59 ` [PATCH v2] " Nick Owens
@ 2019-10-23 17:59 ` Nick Owens
2019-10-24 8:21 ` Richard Purdie
0 siblings, 1 reply; 7+ messages in thread
From: Nick Owens @ 2019-10-23 17:59 UTC (permalink / raw)
To: openembedded-core
parallel builds for native code in python modules was added about 5
years ago. distutils understands '-j N', so just pass along
the right argument to setup.py build.
Signed-off-by: Nick Owens <nick.owens@eero.com>
---
meta/classes/distutils.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/distutils.bbclass b/meta/classes/distutils.bbclass
index b5c9c2fbbd..ea479f552e 100644
--- a/meta/classes/distutils.bbclass
+++ b/meta/classes/distutils.bbclass
@@ -23,7 +23,7 @@ distutils_do_compile() {
NO_FETCH_BUILD=1 \
STAGING_INCDIR=${STAGING_INCDIR} \
STAGING_LIBDIR=${STAGING_LIBDIR} \
- ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS} || \
+ ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py build ${@oe.utils.parallel_make_argument(d, "-j %d")} ${DISTUTILS_BUILD_ARGS} || \
bbfatal_log "'${PYTHON_PN} setup.py build ${DISTUTILS_BUILD_ARGS}' execution failed."
}
--
2.17.1
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH] distutils: pass along parallel make flags to setup.py build
2019-10-23 17:59 ` [PATCH] " Nick Owens
@ 2019-10-24 8:21 ` Richard Purdie
2019-10-24 18:25 ` Nick Owens
0 siblings, 1 reply; 7+ messages in thread
From: Richard Purdie @ 2019-10-24 8:21 UTC (permalink / raw)
To: Nick Owens, openembedded-core
On Wed, 2019-10-23 at 10:59 -0700, Nick Owens wrote:
> parallel builds for native code in python modules was added about 5
> years ago. distutils understands '-j N', so just pass along
> the right argument to setup.py build.
>
> Signed-off-by: Nick Owens <nick.owens@eero.com>
> ---
> meta/classes/distutils.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/distutils.bbclass
> b/meta/classes/distutils.bbclass
> index b5c9c2fbbd..ea479f552e 100644
> --- a/meta/classes/distutils.bbclass
> +++ b/meta/classes/distutils.bbclass
> @@ -23,7 +23,7 @@ distutils_do_compile() {
> NO_FETCH_BUILD=1 \
> STAGING_INCDIR=${STAGING_INCDIR} \
> STAGING_LIBDIR=${STAGING_LIBDIR} \
> - ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
> setup.py build ${DISTUTILS_BUILD_ARGS} || \
> + ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
> setup.py build ${@oe.utils.parallel_make_argument(d, "-j %d")}
> ${DISTUTILS_BUILD_ARGS} || \
> bbfatal_log "'${PYTHON_PN} setup.py build
> ${DISTUTILS_BUILD_ARGS}' execution failed."
> }
Fails on the autobuilder in testing (step 3b):
https://autobuilder.yoctoproject.org/typhoon/#/builders/69/builds/1164
ERROR: python-setuptools-native-41.4.0-r0 do_compile: 'python setup.py build ' execution failed.
ERROR: python-setuptools-native-41.4.0-r0 do_compile: Execution of '/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/run.do_compile.5035' failed with exit code 1:
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: option -j not recognized
WARNING: /home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/run.do_compile.5035:1 exit 1 from 'exit 1'
ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/log.do_compile.5035
Log data follows:
| DEBUG: Executing shell function do_compile
| usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
| or: setup.py --help [cmd1 cmd2 ...]
| or: setup.py --help-commands
| or: setup.py cmd --help
|
| error: option -j not recognized
| ERROR: 'python setup.py build ' execution failed.
| WARNING: /home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/run.do_compile.5035:1 exit 1 from 'exit 1'
| ERROR: Execution of '/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/run.do_compile.5035' failed with exit code 1:
| usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
| or: setup.py --help [cmd1 cmd2 ...]
| or: setup.py --help-commands
| or: setup.py cmd --help
|
| error: option -j not recognized
| WARNING: /home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/run.do_compile.5035:1 exit 1 from 'exit 1'
|
NOTE: recipe python-setuptools-native-41.4.0-r0: task do_compile: Failed
ERROR: Task (virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-devtools/python/python-setuptools_41.4.0.bb:do_compile) failed with exit code '1'
Also:
https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/1164
https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/1166
https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/1163
Cheers,
Richard
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] distutils: pass along parallel make flags to setup.py build
2019-10-24 8:21 ` Richard Purdie
@ 2019-10-24 18:25 ` Nick Owens
2019-10-28 21:31 ` Ross Burton
0 siblings, 1 reply; 7+ messages in thread
From: Nick Owens @ 2019-10-24 18:25 UTC (permalink / raw)
To: Richard Purdie; +Cc: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 4173 bytes --]
i see, it looks like i should apply this to distutils3.bbclass and not
distutils.bbclass, since python2 never got support for --parallel. does
that seem right?
On Thu, Oct 24, 2019 at 1:21 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> On Wed, 2019-10-23 at 10:59 -0700, Nick Owens wrote:
> > parallel builds for native code in python modules was added about 5
> > years ago. distutils understands '-j N', so just pass along
> > the right argument to setup.py build.
> >
> > Signed-off-by: Nick Owens <nick.owens@eero.com>
> > ---
> > meta/classes/distutils.bbclass | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/classes/distutils.bbclass
> > b/meta/classes/distutils.bbclass
> > index b5c9c2fbbd..ea479f552e 100644
> > --- a/meta/classes/distutils.bbclass
> > +++ b/meta/classes/distutils.bbclass
> > @@ -23,7 +23,7 @@ distutils_do_compile() {
> > NO_FETCH_BUILD=1 \
> > STAGING_INCDIR=${STAGING_INCDIR} \
> > STAGING_LIBDIR=${STAGING_LIBDIR} \
> > - ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
> > setup.py build ${DISTUTILS_BUILD_ARGS} || \
> > + ${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN}
> > setup.py build ${@oe.utils.parallel_make_argument(d, "-j %d")}
> > ${DISTUTILS_BUILD_ARGS} || \
> > bbfatal_log "'${PYTHON_PN} setup.py build
> > ${DISTUTILS_BUILD_ARGS}' execution failed."
> > }
>
> Fails on the autobuilder in testing (step 3b):
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/69/builds/1164
>
> ERROR: python-setuptools-native-41.4.0-r0 do_compile: 'python setup.py
> build ' execution failed.
> ERROR: python-setuptools-native-41.4.0-r0 do_compile: Execution of
> '/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/run.do_compile.5035'
> failed with exit code 1:
> usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
> or: setup.py --help [cmd1 cmd2 ...]
> or: setup.py --help-commands
> or: setup.py cmd --help
>
> error: option -j not recognized
> WARNING:
> /home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/run.do_compile.5035:1
> exit 1 from 'exit 1'
>
> ERROR: Logfile of failure stored in:
> /home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/log.do_compile.5035
> Log data follows:
> | DEBUG: Executing shell function do_compile
> | usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
> | or: setup.py --help [cmd1 cmd2 ...]
> | or: setup.py --help-commands
> | or: setup.py cmd --help
> |
> | error: option -j not recognized
> | ERROR: 'python setup.py build ' execution failed.
> | WARNING:
> /home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/run.do_compile.5035:1
> exit 1 from 'exit 1'
> | ERROR: Execution of
> '/home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/run.do_compile.5035'
> failed with exit code 1:
> | usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
> | or: setup.py --help [cmd1 cmd2 ...]
> | or: setup.py --help-commands
> | or: setup.py cmd --help
> |
> | error: option -j not recognized
> | WARNING:
> /home/pokybuild/yocto-worker/qa-extras/build/build/tmp/work/x86_64-linux/python-setuptools-native/41.4.0-r0/temp/run.do_compile.5035:1
> exit 1 from 'exit 1'
> |
> NOTE: recipe python-setuptools-native-41.4.0-r0: task do_compile: Failed
> ERROR: Task
> (virtual:native:/home/pokybuild/yocto-worker/qa-extras/build/meta/recipes-devtools/python/python-setuptools_41.4.0.bb:do_compile)
> failed with exit code '1'
>
> Also:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/64/builds/1164
> https://autobuilder.yoctoproject.org/typhoon/#/builders/45/builds/1166
> https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/1163
>
> Cheers,
>
> Richard
>
>
>
[-- Attachment #2: Type: text/html, Size: 5354 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-10-28 21:31 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-22 23:59 [PATCH] distutils: pass along parallel make flags to setup.py build Nick Owens
2019-10-23 10:56 ` Ross Burton
2019-10-23 17:59 ` [PATCH v2] " Nick Owens
2019-10-23 17:59 ` [PATCH] " Nick Owens
2019-10-24 8:21 ` Richard Purdie
2019-10-24 18:25 ` Nick Owens
2019-10-28 21:31 ` Ross Burton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.