From: Masahiro Yamada <yamada.masahiro@socionext.com> To: linux-kbuild@vger.kernel.org Cc: cocci@systeme.lip6.fr, Julia Lawall <Julia.Lawall@lip6.fr>, Nicolas Palix <nicolas.palix@imag.fr>, Gilles Muller <Gilles.Muller@lip6.fr>, Masahiro Yamada <yamada.masahiro@socionext.com>, linux-kernel@vger.kernel.org, Michal Marek <michal.lkml@markovi.net> Subject: [PATCH v4] coccinelle: fix parallel build with CHECK=scripts/coccicheck Date: Tue, 14 Nov 2017 20:38:07 +0900 [thread overview] Message-ID: <1510659487-16361-1-git-send-email-yamada.masahiro@socionext.com> (raw) The command "make -j8 C=1 CHECK=scripts/coccicheck COCCI=..." produces lots of "coccicheck failed" error messages. Julia Lawall explained the Coccinelle behavior as follows: "The problem on the Coccinelle side is that it uses a subdirectory with the name of the semantic patch to store standard output and standard error for the different threads. I didn't want to use a name with the pid, so that one could easily find this information while Coccinelle is running. Normally the subdirectory is cleaned up when Coccinelle completes, so there is only one of them at a time. Maybe it is best to just add the pid. There is the risk that these subdirectories will accumulate if Coccinelle crashes in a way such that they don't get cleaned up, but Coccinelle could print a warning if it detects this case, rather than failing." When scripts/coccicheck is used as CHECK tool and -j option is given to Make, the whole of build process runs in parallel. So, multiple processes try to get access to the same subdirectory. I notice spatch creates the subdirectory only when it runs in parallel (i.e. --jobs <N> is given and <N> is greater than 1). Setting NPROC=1 is a reasonable solution; spatch does not create the subdirectory. Besides, ONLINE=1 mode takes a single file input for each spatch invocation, so there is no reason to parallelize it in the first place. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> --- Changes in v4: - Remove unnecessary comments. One line comment will be enough. Changes in v3: - Set NPROC=1 because this is a more sensible solution given that there is no reason to run coccinelle in parallel for ONLINE=1 mode. - Move J= option handling to a proper place. - Add more detailed explanation Changes in v2: - Grep '-j' instead of '--jobserver-auth'. '--jobserver-*' is not a stable option flag. Make 4.2 change '--jobserver-fds' into '--jobserver-auth' - Add -q option to grep scripts/coccicheck | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index 040a8b1..fefda40 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -30,12 +30,6 @@ else VERBOSE=0 fi -if [ -z "$J" ]; then - NPROC=$(getconf _NPROCESSORS_ONLN) -else - NPROC="$J" -fi - FLAGS="--very-quiet" # You can use SPFLAGS to append extra arguments to coccicheck or override any @@ -70,6 +64,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then # Take only the last argument, which is the C file to test shift $(( $# - 1 )) OPTIONS="$COCCIINCLUDE $1" + + # No need to parallelize Coccinelle since this mode takes one input file. + NPROC=1 else ONLINE=0 if [ "$KBUILD_EXTMOD" = "" ] ; then @@ -77,6 +74,12 @@ else else OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" fi + + if [ -z "$J" ]; then + NPROC=$(getconf _NPROCESSORS_ONLN) + else + NPROC="$J" + fi fi if [ "$KBUILD_EXTMOD" != "" ] ; then -- 2.7.4
WARNING: multiple messages have this Message-ID (diff)
From: yamada.masahiro@socionext.com (Masahiro Yamada) To: cocci@systeme.lip6.fr Subject: [Cocci] [PATCH v4] coccinelle: fix parallel build with CHECK=scripts/coccicheck Date: Tue, 14 Nov 2017 20:38:07 +0900 [thread overview] Message-ID: <1510659487-16361-1-git-send-email-yamada.masahiro@socionext.com> (raw) The command "make -j8 C=1 CHECK=scripts/coccicheck COCCI=..." produces lots of "coccicheck failed" error messages. Julia Lawall explained the Coccinelle behavior as follows: "The problem on the Coccinelle side is that it uses a subdirectory with the name of the semantic patch to store standard output and standard error for the different threads. I didn't want to use a name with the pid, so that one could easily find this information while Coccinelle is running. Normally the subdirectory is cleaned up when Coccinelle completes, so there is only one of them at a time. Maybe it is best to just add the pid. There is the risk that these subdirectories will accumulate if Coccinelle crashes in a way such that they don't get cleaned up, but Coccinelle could print a warning if it detects this case, rather than failing." When scripts/coccicheck is used as CHECK tool and -j option is given to Make, the whole of build process runs in parallel. So, multiple processes try to get access to the same subdirectory. I notice spatch creates the subdirectory only when it runs in parallel (i.e. --jobs <N> is given and <N> is greater than 1). Setting NPROC=1 is a reasonable solution; spatch does not create the subdirectory. Besides, ONLINE=1 mode takes a single file input for each spatch invocation, so there is no reason to parallelize it in the first place. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> --- Changes in v4: - Remove unnecessary comments. One line comment will be enough. Changes in v3: - Set NPROC=1 because this is a more sensible solution given that there is no reason to run coccinelle in parallel for ONLINE=1 mode. - Move J= option handling to a proper place. - Add more detailed explanation Changes in v2: - Grep '-j' instead of '--jobserver-auth'. '--jobserver-*' is not a stable option flag. Make 4.2 change '--jobserver-fds' into '--jobserver-auth' - Add -q option to grep scripts/coccicheck | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index 040a8b1..fefda40 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -30,12 +30,6 @@ else VERBOSE=0 fi -if [ -z "$J" ]; then - NPROC=$(getconf _NPROCESSORS_ONLN) -else - NPROC="$J" -fi - FLAGS="--very-quiet" # You can use SPFLAGS to append extra arguments to coccicheck or override any @@ -70,6 +64,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then # Take only the last argument, which is the C file to test shift $(( $# - 1 )) OPTIONS="$COCCIINCLUDE $1" + + # No need to parallelize Coccinelle since this mode takes one input file. + NPROC=1 else ONLINE=0 if [ "$KBUILD_EXTMOD" = "" ] ; then @@ -77,6 +74,12 @@ else else OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE" fi + + if [ -z "$J" ]; then + NPROC=$(getconf _NPROCESSORS_ONLN) + else + NPROC="$J" + fi fi if [ "$KBUILD_EXTMOD" != "" ] ; then -- 2.7.4
next reply other threads:[~2017-11-14 11:39 UTC|newest] Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-11-14 11:38 Masahiro Yamada [this message] 2017-11-14 11:38 ` [Cocci] [PATCH v4] coccinelle: fix parallel build with CHECK=scripts/coccicheck Masahiro Yamada 2017-11-14 12:35 ` Julia Lawall 2017-11-14 12:35 ` [Cocci] " Julia Lawall 2017-11-23 14:18 ` Masahiro Yamada 2017-11-23 14:18 ` [Cocci] " Masahiro Yamada 2017-11-14 13:32 ` [Cocci] [v4] Coccinelle: " SF Markus Elfring
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1510659487-16361-1-git-send-email-yamada.masahiro@socionext.com \ --to=yamada.masahiro@socionext.com \ --cc=Gilles.Muller@lip6.fr \ --cc=Julia.Lawall@lip6.fr \ --cc=cocci@systeme.lip6.fr \ --cc=linux-kbuild@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=michal.lkml@markovi.net \ --cc=nicolas.palix@imag.fr \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.