From: Julia Lawall <julia.lawall@lip6.fr>
To: Masahiro Yamada <yamada.masahiro@socionext.com>
Cc: linux-kbuild@vger.kernel.org, cocci@systeme.lip6.fr,
Nicolas Palix <nicolas.palix@imag.fr>,
Gilles Muller <Gilles.Muller@lip6.fr>,
linux-kernel@vger.kernel.org,
Michal Marek <michal.lkml@markovi.net>
Subject: Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck
Date: Fri, 10 Nov 2017 16:42:25 +0100 (CET) [thread overview]
Message-ID: <alpine.DEB.2.20.1711101634225.3187@hadrien> (raw)
In-Reply-To: <1510210850-28866-1-git-send-email-yamada.masahiro@socionext.com>
On Thu, 9 Nov 2017, Masahiro Yamada wrote:
> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> "coccicheck failed" error messages.
The question is where parallelism should be specified. Currently, make
coccicheck picks up the number of cores on the machine and passes that to
Coccinelle.
OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
On my 80 core machine with hyperthreading, this runs 160 jobs in parallel,
while in practice that degrades the performance as compared to 40 or 80
cores.
On the other hand, if we use the make command line argument (-j), then we
will only get parallelism up to the number of semantic patches. Since
some finish quickly, there will be a lot of wasted cycles.
The best would be that the user knows what works well for his machine, and
specifies it on the command line, and then that value gets propagated to
Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in
parallel but instead would cause Coccinelle to run one semantic patch on 8
files in parallel. But I don't know if that can be done.
julia
>
> I do not know the coccinelle internals, but I guess --jobs does not
> work well if spatch is invoked from Make running in parallel.
> Disable --jobs in this case.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Changes in v2:
> - Grep '-j' instead of '--jobserver-auth'.
> '--jobserver-*' is not a stable option flag.
> Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
> - Add -q option to grep
>
> scripts/coccicheck | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 040a8b1..8bab11e 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
> # Take only the last argument, which is the C file to test
> shift $(( $# - 1 ))
> OPTIONS="$COCCIINCLUDE $1"
> +
> + # --jobs does not work if Make is running in parallel
> + echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
> else
> ONLINE=0
> if [ "$KBUILD_EXTMOD" = "" ] ; then
> --
> 2.7.4
>
>
next prev parent reply other threads:[~2017-11-10 15:42 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-11-09 7:00 [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck Masahiro Yamada
2017-11-10 15:42 ` Julia Lawall [this message]
2017-11-11 7:30 ` [Cocci] " Julia Lawall
2017-11-13 16:31 ` Masahiro Yamada
2017-11-13 16:44 ` Julia Lawall
2017-11-13 15:30 ` Julia Lawall
2017-11-13 16:32 ` Masahiro Yamada
2017-11-13 16:45 ` Julia Lawall
2017-11-14 3:51 ` Masahiro Yamada
2017-11-14 6:44 ` Julia Lawall
2017-11-14 9:08 ` Masahiro Yamada
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.2.20.1711101634225.3187@hadrien \
--to=julia.lawall@lip6.fr \
--cc=Gilles.Muller@lip6.fr \
--cc=cocci@systeme.lip6.fr \
--cc=linux-kbuild@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=michal.lkml@markovi.net \
--cc=nicolas.palix@imag.fr \
--cc=yamada.masahiro@socionext.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).