[PATCH] perf: disable parallelism for 'make clean'

[PATCH] perf: disable parallelism for 'make clean'

[Rasmus Villemoes]

The Yocto build system does a 'make clean' when rebuilding due to
changed dependencies, and that consistently fails for me (causing the
whole BSP build to fail) with errors such as

| find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
| find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
| find: find: '[...]/perf/1.0-r9/perf-1.0/libtraceevent.a''[...]/perf/1.0-r9/perf-1.0/libtraceevent.a': No such file or directory: No such file or directory
|
[...]
| find: cannot delete '/mnt/xfs/devel/pil/yocto/tmp-glibc/work/wandboard-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/util/.pstack.o.cmd': No such file or directory

Apparently (despite the comment), 'make clean' ends up launching
multiple sub-makes that all want to remove the same things - perhaps
this only happens in combination with a O=... parameter. In any case, we
don't lose much by explicitly disabling the parallelism for the clean
target, and it makes automated builds much more reliable.

Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
---
Another option is to add -ignore_readdir_race to all find commands,
but that's a lot more invasive and only works properly together with
-delete with a very recent GNU find
(https://savannah.gnu.org/bugs/?52981).

 tools/perf/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/perf/Makefile b/tools/perf/Makefile
index 225454416ed5..7902a5681fc8 100644
--- a/tools/perf/Makefile
+++ b/tools/perf/Makefile
@@ -84,10 +84,10 @@ endif # has_clean
 endif # MAKECMDGOALS
 
 #
-# The clean target is not really parallel, don't print the jobs info:
+# Explicitly disable parallelism for the clean target.
 #
 clean:
-	$(make)
+	$(make) -j1
 
 #
 # The build-test target is not really parallel, don't print the jobs info,
-- 
2.16.4

Re: [PATCH] perf: disable parallelism for 'make clean'

[Jiri Olsa]

On Thu, Jul 05, 2018 at 03:15:27PM +0200, Rasmus Villemoes wrote:
> The Yocto build system does a 'make clean' when rebuilding due to
> changed dependencies, and that consistently fails for me (causing the
> whole BSP build to fail) with errors such as
> 
> | find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
> | find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
> | find: find: '[...]/perf/1.0-r9/perf-1.0/libtraceevent.a''[...]/perf/1.0-r9/perf-1.0/libtraceevent.a': No such file or directory: No such file or directory
> |
> [...]
> | find: cannot delete '/mnt/xfs/devel/pil/yocto/tmp-glibc/work/wandboard-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/util/.pstack.o.cmd': No such file or directory
> 
> Apparently (despite the comment), 'make clean' ends up launching
> multiple sub-makes that all want to remove the same things - perhaps
> this only happens in combination with a O=... parameter. In any case, we
> don't lose much by explicitly disabling the parallelism for the clean
> target, and it makes automated builds much more reliable.
> 
> Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>

Acked-by: Jiri Olsa <jolsa@kernel.org>

thanks,
jirka

Re: [PATCH] perf: disable parallelism for 'make clean'

[Rasmus Villemoes]

On 2018-07-05 15:49, Jiri Olsa wrote:
> On Thu, Jul 05, 2018 at 03:15:27PM +0200, Rasmus Villemoes wrote:

>> this only happens in combination with a O=... parameter. In any case, we
>> don't lose much by explicitly disabling the parallelism for the clean
>> target, and it makes automated builds much more reliable.
>>
>> Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
> 
> Acked-by: Jiri Olsa <jolsa@kernel.org>

Was this picked up by anyone? I'm not seeing it in -next?

Rasmus

Re: [PATCH] perf: disable parallelism for 'make clean'

[Arnaldo Carvalho de Melo]

Em Wed, Aug 15, 2018 at 09:30:39AM +0200, Rasmus Villemoes escreveu:
> On 2018-07-05 15:49, Jiri Olsa wrote:
> > On Thu, Jul 05, 2018 at 03:15:27PM +0200, Rasmus Villemoes wrote:
> 
> >> this only happens in combination with a O=... parameter. In any case, we
> >> don't lose much by explicitly disabling the parallelism for the clean
> >> target, and it makes automated builds much more reliable.
> >>
> >> Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
> > 
> > Acked-by: Jiri Olsa <jolsa@kernel.org>
> 
> Was this picked up by anyone? I'm not seeing it in -next?

Thanks, applied.

- Arnaldo

[tip:perf/urgent] perf tools: Disable parallelism for 'make clean'

[tip-bot for Rasmus Villemoes]

Commit-ID:  da15fc2fa9c07b23db8f5e479bd8a9f0d741ca07
Gitweb:     https://git.kernel.org/tip/da15fc2fa9c07b23db8f5e479bd8a9f0d741ca07
Author:     Rasmus Villemoes <linux@rasmusvillemoes.dk>
AuthorDate: Thu, 5 Jul 2018 15:15:27 +0200
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Mon, 20 Aug 2018 08:54:58 -0300

perf tools: Disable parallelism for 'make clean'

The Yocto build system does a 'make clean' when rebuilding due to
changed dependencies, and that consistently fails for me (causing the
whole BSP build to fail) with errors such as

| find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
| find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
| find: find: '[...]/perf/1.0-r9/perf-1.0/libtraceevent.a''[...]/perf/1.0-r9/perf-1.0/libtraceevent.a': No such file or directory: No such file or directory
|
[...]
| find: cannot delete '/mnt/xfs/devel/pil/yocto/tmp-glibc/work/wandboard-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/util/.pstack.o.cmd': No such file or directory

Apparently (despite the comment), 'make clean' ends up launching
multiple sub-makes that all want to remove the same things - perhaps
this only happens in combination with a O=... parameter. In any case, we
don't lose much by explicitly disabling the parallelism for the clean
target, and it makes automated builds much more reliable.

Signed-off-by: Rasmus Villemoes <linux@rasmusvillemoes.dk>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20180705131527.19749-1-linux@rasmusvillemoes.dk
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/perf/Makefile b/tools/perf/Makefile
index 225454416ed5..7902a5681fc8 100644
--- a/tools/perf/Makefile
+++ b/tools/perf/Makefile
@@ -84,10 +84,10 @@ endif # has_clean
 endif # MAKECMDGOALS
 
 #
-# The clean target is not really parallel, don't print the jobs info:
+# Explicitly disable parallelism for the clean target.
 #
 clean:
-	$(make)
+	$(make) -j1
 
 #
 # The build-test target is not really parallel, don't print the jobs info,

Re: [PATCH] perf: disable parallelism for 'make clean'

[Rasmus Villemoes]

On 2018-07-06 00:18, Andre McCurdy wrote:
> On Thu, Jul 5, 2018 at 1:55 PM, Rasmus Villemoes
> <rasmus.villemoes@prevas.dk> wrote:
>> Whenever perf got rebuilt, I was consistently getting errors such as
>>
>> | find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
>> | find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
>> | find: find: '[...]/perf/1.0-r9/perf-1.0/libtraceevent.a''[...]/perf/1.0-r9/perf-1.0/libtraceevent.a': No such file or directory: No such file or directory
>> |
>> [...]
>> | find: cannot delete '/mnt/xfs/devel/pil/yocto/tmp-glibc/work/wandboard-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/util/.pstack.o.cmd': No such file or directory
>>
>> breaking the whole build. The root cause seems to be that the implicit
>> 'make clean' done during do_configure ends up running in parallel, and
>> thus multiple find commands attempt to stat and/or delete the same
>> file.
>>
>> This patch of tools/perf/Makefile has been ack'ed
>> upstream (lkml.kernel.org/r/20180705134955.GB3686@krava). Since the sed
>> script matches exactly the lines
>>
>> clean:
>>         $(make)
>>
>> it should be harmless to have this even with a fixed kernel, and it can
>> be removed if and when all relevant -stable kernels have it.
> 
> If it works, then something like this might be a cleaner solution:
> 
>   EXTRA_OEMAKE_append_task-configure = " -j1"

tools/perf/Makefile explicitly ignores a -j argument, and computes its
own value. But one can pass JOBS=1 instead. I'll try that and resend.

Rasmus

Re: [PATCH] perf: disable parallelism for 'make clean'

[Andre McCurdy]

On Thu, Jul 5, 2018 at 1:55 PM, Rasmus Villemoes
<rasmus.villemoes@prevas.dk> wrote:
> Whenever perf got rebuilt, I was consistently getting errors such as
>
> | find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
> | find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
> | find: find: '[...]/perf/1.0-r9/perf-1.0/libtraceevent.a''[...]/perf/1.0-r9/perf-1.0/libtraceevent.a': No such file or directory: No such file or directory
> |
> [...]
> | find: cannot delete '/mnt/xfs/devel/pil/yocto/tmp-glibc/work/wandboard-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/util/.pstack.o.cmd': No such file or directory
>
> breaking the whole build. The root cause seems to be that the implicit
> 'make clean' done during do_configure ends up running in parallel, and
> thus multiple find commands attempt to stat and/or delete the same
> file.
>
> This patch of tools/perf/Makefile has been ack'ed
> upstream (lkml.kernel.org/r/20180705134955.GB3686@krava). Since the sed
> script matches exactly the lines
>
> clean:
>         $(make)
>
> it should be harmless to have this even with a fixed kernel, and it can
> be removed if and when all relevant -stable kernels have it.

If it works, then something like this might be a cleaner solution:

  EXTRA_OEMAKE_append_task-configure = " -j1"

(and if it doesn't work, it would be good to understand why).

> Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
> ---
>  meta/recipes-kernel/perf/perf.bb | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb
> index 51f5597eb9..1cf2666ee5 100644
> --- a/meta/recipes-kernel/perf/perf.bb
> +++ b/meta/recipes-kernel/perf/perf.bb
> @@ -145,6 +145,11 @@ python copy_perf_source_from_kernel() {
>              bb.utils.copyfile(src, dest)
>  }
>
> +do_configure[prefuncs] += "disable_clean_parallelism"
> +disable_clean_parallelism () {
> +       sed -i '/^clean:$/{n;s/^\t\$(make)$/\t\$(make) -j1/}' ${S}/tools/perf/Makefile
> +}
> +
>  do_configure_prepend () {
>      # If building a multlib based perf, the incorrect library path will be
>      # detected by perf, since it triggers via: ifeq ($(ARCH),x86_64). In a 32 bit
> --
> 2.16.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

[PATCH] perf: disable parallelism for 'make clean'

[Rasmus Villemoes]

Whenever perf got rebuilt, I was consistently getting errors such as

| find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
| find: '[...]/perf/1.0-r9/perf-1.0/plugin_mac80211.so': No such file or directory
| find: find: '[...]/perf/1.0-r9/perf-1.0/libtraceevent.a''[...]/perf/1.0-r9/perf-1.0/libtraceevent.a': No such file or directory: No such file or directory
|
[...]
| find: cannot delete '/mnt/xfs/devel/pil/yocto/tmp-glibc/work/wandboard-oe-linux-gnueabi/perf/1.0-r9/perf-1.0/util/.pstack.o.cmd': No such file or directory

breaking the whole build. The root cause seems to be that the implicit
'make clean' done during do_configure ends up running in parallel, and
thus multiple find commands attempt to stat and/or delete the same
file.

This patch of tools/perf/Makefile has been ack'ed
upstream (lkml.kernel.org/r/20180705134955.GB3686@krava). Since the sed
script matches exactly the lines

clean:
	$(make)

it should be harmless to have this even with a fixed kernel, and it can
be removed if and when all relevant -stable kernels have it.

Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
---
 meta/recipes-kernel/perf/perf.bb | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/recipes-kernel/perf/perf.bb b/meta/recipes-kernel/perf/perf.bb
index 51f5597eb9..1cf2666ee5 100644
--- a/meta/recipes-kernel/perf/perf.bb
+++ b/meta/recipes-kernel/perf/perf.bb
@@ -145,6 +145,11 @@ python copy_perf_source_from_kernel() {
             bb.utils.copyfile(src, dest)
 }
 
+do_configure[prefuncs] += "disable_clean_parallelism"
+disable_clean_parallelism () {
+	sed -i '/^clean:$/{n;s/^\t\$(make)$/\t\$(make) -j1/}' ${S}/tools/perf/Makefile
+}
+
 do_configure_prepend () {
     # If building a multlib based perf, the incorrect library path will be
     # detected by perf, since it triggers via: ifeq ($(ARCH),x86_64). In a 32 bit
-- 
2.16.4