All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 1/3] ptest.bbclass: Honor PARALLEL_MAKE, PARALLEL_MAKEINST
@ 2022-06-22 19:16 Niko Mauno
  2022-06-22 19:16 ` [PATCH 2/3] strace: Drop redundant oe_runmake parameter Niko Mauno
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Niko Mauno @ 2022-06-22 19:16 UTC (permalink / raw)
  To: openembedded-core; +Cc: Niko Mauno

Since oe_runmake wrapper doesn't by default involve PARALLEL_MAKE
outside do_compile() nor PARALLEL_MAKEINST outside do_install(),
enable parallellized make by default when oe_runmake is invoked from
do_compile_ptest() or do_install_ptest() by declaring wrapper task
specific EXTRA_OEMAKE overrides in fashion similar to do_compile and
do_install overrides in meta/conf/bitbake.conf.

Parallel make can still be disabled by resetting bbclass specific
PTEST_PARALLEL_MAKE and PTEST_PARALLEL_MAKEINST variables in recipe
e.g. if a race issue needs to be avoided without modifying source code.

Tested by issuing following command sequence on a 32-core build host:

  $ bitbake -c clean util-linux && bitbake --skip-setscene -c compile util-linux && time bitbake --skip-setscene -c compile_ptest_base util-linux

and found that before this change the result was

  real	0m34.684s
  user	0m0.753s
  sys	0m0.131s

and after this change

  real	0m9.868s
  user	0m0.749s
  sys	0m0.150s

Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
---
 meta/classes/ptest.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/ptest.bbclass b/meta/classes/ptest.bbclass
index 1ec23c0923..c162f5d934 100644
--- a/meta/classes/ptest.bbclass
+++ b/meta/classes/ptest.bbclass
@@ -5,6 +5,10 @@ This package contains a test directory ${PTEST_PATH} for package test purposes."
 PTEST_PATH ?= "${libdir}/${BPN}/ptest"
 PTEST_BUILD_HOST_FILES ?= "Makefile"
 PTEST_BUILD_HOST_PATTERN ?= ""
+PTEST_PARALLEL_MAKE ?= "${PARALLEL_MAKE}"
+PTEST_PARALLEL_MAKEINST ?= "${PARALLEL_MAKEINST}"
+EXTRA_OEMAKE:prepend:task-compile-ptest-base = "${PTEST_PARALLEL_MAKE} "
+EXTRA_OEMAKE:prepend:task-install-ptest-base = "${PTEST_PARALLEL_MAKEINST} "
 
 FILES:${PN}-ptest += "${PTEST_PATH}"
 SECTION:${PN}-ptest = "devel"
-- 
2.35.1



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 2/3] strace: Drop redundant oe_runmake parameter
  2022-06-22 19:16 [PATCH 1/3] ptest.bbclass: Honor PARALLEL_MAKE, PARALLEL_MAKEINST Niko Mauno
@ 2022-06-22 19:16 ` Niko Mauno
  2022-06-23  9:30   ` [OE-core] " Luca Ceresoli
  2022-06-22 19:16 ` [PATCH 3/3] valgrind: " Niko Mauno
  2022-06-25 16:55 ` [OE-core] [PATCH 1/3] ptest.bbclass: Honor PARALLEL_MAKE, PARALLEL_MAKEINST Richard Purdie
  2 siblings, 1 reply; 7+ messages in thread
From: Niko Mauno @ 2022-06-22 19:16 UTC (permalink / raw)
  To: openembedded-core; +Cc: Niko Mauno

PARALLEL_MAKE is now honored in ptest.bbclass, so drop redundant
parameter from oe_runmake call.

Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
---
 meta/recipes-devtools/strace/strace_5.17.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/strace/strace_5.17.bb b/meta/recipes-devtools/strace/strace_5.17.bb
index c7fd9edf40..0542780c00 100644
--- a/meta/recipes-devtools/strace/strace_5.17.bb
+++ b/meta/recipes-devtools/strace/strace_5.17.bb
@@ -34,7 +34,7 @@ TESTDIR = "tests"
 PTEST_BUILD_HOST_PATTERN = "^(DEB_CHANGELOGTIME|RPM_CHANGELOGTIME|WARN_CFLAGS_FOR_BUILD|LDFLAGS_FOR_BUILD)"
 
 do_compile_ptest() {
-	oe_runmake ${PARALLEL_MAKE} -C ${TESTDIR} buildtest-TESTS
+	oe_runmake -C ${TESTDIR} buildtest-TESTS
 }
 
 do_install_ptest() {
-- 
2.35.1



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* [PATCH 3/3] valgrind: Drop redundant oe_runmake parameter
  2022-06-22 19:16 [PATCH 1/3] ptest.bbclass: Honor PARALLEL_MAKE, PARALLEL_MAKEINST Niko Mauno
  2022-06-22 19:16 ` [PATCH 2/3] strace: Drop redundant oe_runmake parameter Niko Mauno
@ 2022-06-22 19:16 ` Niko Mauno
  2022-06-25 16:55 ` [OE-core] [PATCH 1/3] ptest.bbclass: Honor PARALLEL_MAKE, PARALLEL_MAKEINST Richard Purdie
  2 siblings, 0 replies; 7+ messages in thread
From: Niko Mauno @ 2022-06-22 19:16 UTC (permalink / raw)
  To: openembedded-core; +Cc: Niko Mauno

PARALLEL_MAKE is now honored in ptest.bbclass, so drop redundant
parameter from oe_runmake call.

Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
---
 meta/recipes-devtools/valgrind/valgrind_3.19.0.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/valgrind/valgrind_3.19.0.bb b/meta/recipes-devtools/valgrind/valgrind_3.19.0.bb
index 6e3234be6b..69915de505 100644
--- a/meta/recipes-devtools/valgrind/valgrind_3.19.0.bb
+++ b/meta/recipes-devtools/valgrind/valgrind_3.19.0.bb
@@ -141,7 +141,7 @@ SKIP_FILEDEPS:${PN}-ptest = '1'
 INSANE_SKIP:${PN}-ptest = "debug-deps"
 
 do_compile_ptest() {
-    oe_runmake ${PARALLEL_MAKE} check
+    oe_runmake check
 }
 
 
-- 
2.35.1



^ permalink raw reply related	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH 2/3] strace: Drop redundant oe_runmake parameter
  2022-06-22 19:16 ` [PATCH 2/3] strace: Drop redundant oe_runmake parameter Niko Mauno
@ 2022-06-23  9:30   ` Luca Ceresoli
  2022-06-23 16:53     ` Niko Mauno
  0 siblings, 1 reply; 7+ messages in thread
From: Luca Ceresoli @ 2022-06-23  9:30 UTC (permalink / raw)
  To: niko.mauno@vaisala.com via lists.openembedded.org
  Cc: niko.mauno, openembedded-core

Hi Niko,

On Wed, 22 Jun 2022 22:16:14 +0300
"niko.mauno@vaisala.com via lists.openembedded.org"
<niko.mauno=vaisala.com@lists.openembedded.org> wrote:

> PARALLEL_MAKE is now honored in ptest.bbclass, so drop redundant
> parameter from oe_runmake call.
> 
> Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
> ---
>  meta/recipes-devtools/strace/strace_5.17.bb | 2 +-

Thank you for your patches!

Unfortunately this patch does not apply as strace has been bumped to
5.18 by a recent commit. Could you send a v2 series with this updated
please?

-- 
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH 2/3] strace: Drop redundant oe_runmake parameter
  2022-06-23  9:30   ` [OE-core] " Luca Ceresoli
@ 2022-06-23 16:53     ` Niko Mauno
  2022-06-24 10:30       ` Luca Ceresoli
  0 siblings, 1 reply; 7+ messages in thread
From: Niko Mauno @ 2022-06-23 16:53 UTC (permalink / raw)
  To: openembedded-core; +Cc: luca.ceresoli


On 6/23/22 12:30, Luca Ceresoli via lists.openembedded.org wrote:
> Unfortunately this patch does not apply as strace has been bumped to
> 5.18 by a recent commit. Could you send a v2 series with this updated
> please?

Hi Luca, new v2 series sent with updated strace version: 
https://patchwork.yoctoproject.org/project/oe-core/list/?series=5032

-Niko


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH 2/3] strace: Drop redundant oe_runmake parameter
  2022-06-23 16:53     ` Niko Mauno
@ 2022-06-24 10:30       ` Luca Ceresoli
  0 siblings, 0 replies; 7+ messages in thread
From: Luca Ceresoli @ 2022-06-24 10:30 UTC (permalink / raw)
  To: niko.mauno@vaisala.com via lists.openembedded.org
  Cc: niko.mauno, openembedded-core

Hi Niko,

On Thu, 23 Jun 2022 19:53:58 +0300
"niko.mauno@vaisala.com via lists.openembedded.org"
<niko.mauno=vaisala.com@lists.openembedded.org> wrote:

> On 6/23/22 12:30, Luca Ceresoli via lists.openembedded.org wrote:
> > Unfortunately this patch does not apply as strace has been bumped to
> > 5.18 by a recent commit. Could you send a v2 series with this updated
> > please?  
> 
> Hi Luca, new v2 series sent with updated strace version: 
> https://patchwork.yoctoproject.org/project/oe-core/list/?series=5032

Thank you. Yesterday I had already picked patches 1 and 3 for testing
on the autobuilders and all builds succeeded. I'm taking your v2 for my
next testing run today.

-- 
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [OE-core] [PATCH 1/3] ptest.bbclass: Honor PARALLEL_MAKE, PARALLEL_MAKEINST
  2022-06-22 19:16 [PATCH 1/3] ptest.bbclass: Honor PARALLEL_MAKE, PARALLEL_MAKEINST Niko Mauno
  2022-06-22 19:16 ` [PATCH 2/3] strace: Drop redundant oe_runmake parameter Niko Mauno
  2022-06-22 19:16 ` [PATCH 3/3] valgrind: " Niko Mauno
@ 2022-06-25 16:55 ` Richard Purdie
  2 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2022-06-25 16:55 UTC (permalink / raw)
  To: niko.mauno, openembedded-core

On Wed, 2022-06-22 at 22:16 +0300, niko.mauno@vaisala.com via
lists.openembedded.org wrote:
> Since oe_runmake wrapper doesn't by default involve PARALLEL_MAKE
> outside do_compile() nor PARALLEL_MAKEINST outside do_install(),
> enable parallellized make by default when oe_runmake is invoked from
> do_compile_ptest() or do_install_ptest() by declaring wrapper task
> specific EXTRA_OEMAKE overrides in fashion similar to do_compile and
> do_install overrides in meta/conf/bitbake.conf.
> 
> Parallel make can still be disabled by resetting bbclass specific
> PTEST_PARALLEL_MAKE and PTEST_PARALLEL_MAKEINST variables in recipe
> e.g. if a race issue needs to be avoided without modifying source code.
> 
> Tested by issuing following command sequence on a 32-core build host:
> 
>   $ bitbake -c clean util-linux && bitbake --skip-setscene -c compile util-linux && time bitbake --skip-setscene -c compile_ptest_base util-linux
> 
> and found that before this change the result was
> 
>   real	0m34.684s
>   user	0m0.753s
>   sys	0m0.131s
> 
> and after this change
> 
>   real	0m9.868s
>   user	0m0.749s
>   sys	0m0.150s
> 
> Signed-off-by: Niko Mauno <niko.mauno@vaisala.com>
> ---
>  meta/classes/ptest.bbclass | 4 ++++
>  1 file changed, 4 insertions(+)

This has merged but is causing a few problems. So far, elfutils and
flex both ran into make races with a ton of failed builds as a result.
I've a patch to disable paralell compile for those two recipes in the
ptest compile. Ideally we'd look into and get this issues fixed
upstream.

Some random build failure links (there are many more in that build):

https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/3045
https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5446
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5417
https://autobuilder.yoctoproject.org/typhoon/#/builders/95/builds/3428

Cheers,

Richard




^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2022-06-25 16:55 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-06-22 19:16 [PATCH 1/3] ptest.bbclass: Honor PARALLEL_MAKE, PARALLEL_MAKEINST Niko Mauno
2022-06-22 19:16 ` [PATCH 2/3] strace: Drop redundant oe_runmake parameter Niko Mauno
2022-06-23  9:30   ` [OE-core] " Luca Ceresoli
2022-06-23 16:53     ` Niko Mauno
2022-06-24 10:30       ` Luca Ceresoli
2022-06-22 19:16 ` [PATCH 3/3] valgrind: " Niko Mauno
2022-06-25 16:55 ` [OE-core] [PATCH 1/3] ptest.bbclass: Honor PARALLEL_MAKE, PARALLEL_MAKEINST Richard Purdie

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.