[PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

The command "make -j8 C=1 CHECK=scripts/coccicheck COCCI=..." produces
lots of "coccicheck failed" error messages.

Julia Lawall explained the coccinelle behavior as follows:
"The problem on the Coccinelle side is that it uses a subdirectory
with the name of the semantic patch to store standard output and
standard error for the different threads.  I didn't want to use a
name with the pid, so that one could easily find this information
while Coccinelle is running.  Normally the subdirectory is cleaned
up when Coccinelle completes, so there is only one of them at a time.
Maybe it is best to just add the pid.  There is the risk that these
subdirectories will accumulate if Coccinelle crashes in a way such
that they don't get cleaned up, but Coccinelle could print a warning
if it detects this case, rather than failing."

When scripts/coccicheck is used as CHECK tool and -j option is given
to Make, the whole of build process runs in parallel.  So, multiple
processes try to get access to the same subdirectory.

I notice spatch creates the subdirectory only when it runs in parallel
(i.e. --jobs <N> is given and <N> is greater than 1).

Setting NPROC=1 is a sensible solution; spatch does not create the
subdirectory.  Besides, ONLINE=1 mode takes a single file input for
each spatch invocation, so there is no reason to parallelize it in
the first place.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v3:
  - Set NPROC=1 because this is a more sensible solution
    given that there is no reason to run coccinelle in parallel
    for ONLINE=1 mode.
  - Move J= option handling to a proper place.
  - Add more detailed explanation

Changes in v2:
  - Grep '-j' instead of '--jobserver-auth'.
    '--jobserver-*' is not a stable option flag.
    Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
  - Add -q option to grep

 scripts/coccicheck | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 040a8b1..7da82a1 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -30,12 +30,6 @@ else
 	VERBOSE=0
 fi
 
-if [ -z "$J" ]; then
-	NPROC=$(getconf _NPROCESSORS_ONLN)
-else
-	NPROC="$J"
-fi
-
 FLAGS="--very-quiet"
 
 # You can use SPFLAGS to append extra arguments to coccicheck or override any
@@ -70,6 +64,13 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
     # Take only the last argument, which is the C file to test
     shift $(( $# - 1 ))
     OPTIONS="$COCCIINCLUDE $1"
+
+    # If -j option is given to Make, scripts/coccicheck runs in parallel.
+    # If coccinelle also runs in parallel, it fails because multiple processes
+    # try to get access to the same subdirectory that stores stdout/stderr.
+    # No need to parallelize coccinelle in this case - this mode takes only
+    # one file input.
+    NPROC=1
 else
     ONLINE=0
     if [ "$KBUILD_EXTMOD" = "" ] ; then
@@ -77,6 +78,12 @@ else
     else
         OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE"
     fi
+
+    if [ -z "$J" ]; then
+        NPROC=$(getconf _NPROCESSORS_ONLN)
+    else
+        NPROC="$J"
+    fi
 fi
 
 if [ "$KBUILD_EXTMOD" != "" ] ; then
-- 
2.7.4

[Cocci] [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

The command "make -j8 C=1 CHECK=scripts/coccicheck COCCI=..." produces
lots of "coccicheck failed" error messages.

Julia Lawall explained the coccinelle behavior as follows:
"The problem on the Coccinelle side is that it uses a subdirectory
with the name of the semantic patch to store standard output and
standard error for the different threads.  I didn't want to use a
name with the pid, so that one could easily find this information
while Coccinelle is running.  Normally the subdirectory is cleaned
up when Coccinelle completes, so there is only one of them at a time.
Maybe it is best to just add the pid.  There is the risk that these
subdirectories will accumulate if Coccinelle crashes in a way such
that they don't get cleaned up, but Coccinelle could print a warning
if it detects this case, rather than failing."

When scripts/coccicheck is used as CHECK tool and -j option is given
to Make, the whole of build process runs in parallel.  So, multiple
processes try to get access to the same subdirectory.

I notice spatch creates the subdirectory only when it runs in parallel
(i.e. --jobs <N> is given and <N> is greater than 1).

Setting NPROC=1 is a sensible solution; spatch does not create the
subdirectory.  Besides, ONLINE=1 mode takes a single file input for
each spatch invocation, so there is no reason to parallelize it in
the first place.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v3:
  - Set NPROC=1 because this is a more sensible solution
    given that there is no reason to run coccinelle in parallel
    for ONLINE=1 mode.
  - Move J= option handling to a proper place.
  - Add more detailed explanation

Changes in v2:
  - Grep '-j' instead of '--jobserver-auth'.
    '--jobserver-*' is not a stable option flag.
    Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
  - Add -q option to grep

 scripts/coccicheck | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 040a8b1..7da82a1 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -30,12 +30,6 @@ else
 	VERBOSE=0
 fi
 
-if [ -z "$J" ]; then
-	NPROC=$(getconf _NPROCESSORS_ONLN)
-else
-	NPROC="$J"
-fi
-
 FLAGS="--very-quiet"
 
 # You can use SPFLAGS to append extra arguments to coccicheck or override any
@@ -70,6 +64,13 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
     # Take only the last argument, which is the C file to test
     shift $(( $# - 1 ))
     OPTIONS="$COCCIINCLUDE $1"
+
+    # If -j option is given to Make, scripts/coccicheck runs in parallel.
+    # If coccinelle also runs in parallel, it fails because multiple processes
+    # try to get access to the same subdirectory that stores stdout/stderr.
+    # No need to parallelize coccinelle in this case - this mode takes only
+    # one file input.
+    NPROC=1
 else
     ONLINE=0
     if [ "$KBUILD_EXTMOD" = "" ] ; then
@@ -77,6 +78,12 @@ else
     else
         OPTIONS="--dir $KBUILD_EXTMOD $COCCIINCLUDE"
     fi
+
+    if [ -z "$J" ]; then
+        NPROC=$(getconf _NPROCESSORS_ONLN)
+    else
+        NPROC="$J"
+    fi
 fi
 
 if [ "$KBUILD_EXTMOD" != "" ] ; then
-- 
2.7.4

Re: [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

> +
> +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
> +    # If coccinelle also runs in parallel, it fails because multiple processes
> +    # try to get access to the same subdirectory that stores stdout/stderr.
> +    # No need to parallelize coccinelle in this case - this mode takes only
> +    # one file input.
> +    NPROC=1

Since I am also changing Coccinelle to avoid the problem, maybe it would
be better to just remove the explanation sentence (If coccinelle also runs
in parallel,...).

julia

[Cocci] [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

> +
> +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
> +    # If coccinelle also runs in parallel, it fails because multiple processes
> +    # try to get access to the same subdirectory that stores stdout/stderr.
> +    # No need to parallelize coccinelle in this case - this mode takes only
> +    # one file input.
> +    NPROC=1

Since I am also changing Coccinelle to avoid the problem, maybe it would
be better to just remove the explanation sentence (If coccinelle also runs
in parallel,...).

julia

Re: [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

Hi Julia,


2017-11-14 18:49 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>> +
>> +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
>> +    # If coccinelle also runs in parallel, it fails because multiple processes
>> +    # try to get access to the same subdirectory that stores stdout/stderr.
>> +    # No need to parallelize coccinelle in this case - this mode takes only
>> +    # one file input.
>> +    NPROC=1
>
> Since I am also changing Coccinelle to avoid the problem, maybe it would
> be better to just remove the explanation sentence (If coccinelle also runs
> in parallel,...).
>
> julia

OK.  Which lines are unneeded?

Is it OK to remove all the comments, then just add "NPROC=1"?


-- 
Best Regards
Masahiro Yamada

[Cocci] [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

Hi Julia,


2017-11-14 18:49 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>> +
>> +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
>> +    # If coccinelle also runs in parallel, it fails because multiple processes
>> +    # try to get access to the same subdirectory that stores stdout/stderr.
>> +    # No need to parallelize coccinelle in this case - this mode takes only
>> +    # one file input.
>> +    NPROC=1
>
> Since I am also changing Coccinelle to avoid the problem, maybe it would
> be better to just remove the explanation sentence (If coccinelle also runs
> in parallel,...).
>
> julia

OK.  Which lines are unneeded?

Is it OK to remove all the comments, then just add "NPROC=1"?


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, Masahiro Yamada wrote:

> Hi Julia,
>
>
> 2017-11-14 18:49 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
> >> +
> >> +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
> >> +    # If coccinelle also runs in parallel, it fails because multiple processes
> >> +    # try to get access to the same subdirectory that stores stdout/stderr.
> >> +    # No need to parallelize coccinelle in this case - this mode takes only
> >> +    # one file input.
> >> +    NPROC=1
> >
> > Since I am also changing Coccinelle to avoid the problem, maybe it would
> > be better to just remove the explanation sentence (If coccinelle also runs
> > in parallel,...).
> >
> > julia
>
> OK.  Which lines are unneeded?
>
> Is it OK to remove all the comments, then just add "NPROC=1"?

How about keeping "# No need to parallelize coccinelle in this case - this
mode takes only one file input."?

But if you think it is obvious, it would be ok to remove it also.  Someone
can always do git blame and read the commit message if they have
questions.

julia

[Cocci] [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, Masahiro Yamada wrote:

> Hi Julia,
>
>
> 2017-11-14 18:49 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
> >> +
> >> +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
> >> +    # If coccinelle also runs in parallel, it fails because multiple processes
> >> +    # try to get access to the same subdirectory that stores stdout/stderr.
> >> +    # No need to parallelize coccinelle in this case - this mode takes only
> >> +    # one file input.
> >> +    NPROC=1
> >
> > Since I am also changing Coccinelle to avoid the problem, maybe it would
> > be better to just remove the explanation sentence (If coccinelle also runs
> > in parallel,...).
> >
> > julia
>
> OK.  Which lines are unneeded?
>
> Is it OK to remove all the comments, then just add "NPROC=1"?

How about keeping "# No need to parallelize coccinelle in this case - this
mode takes only one file input."?

But if you think it is obvious, it would be ok to remove it also.  Someone
can always do git blame and read the commit message if they have
questions.

julia

Re: [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

2017-11-14 20:09 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>
>
> On Tue, 14 Nov 2017, Masahiro Yamada wrote:
>
>> Hi Julia,
>>
>>
>> 2017-11-14 18:49 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>> >> +
>> >> +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
>> >> +    # If coccinelle also runs in parallel, it fails because multiple processes
>> >> +    # try to get access to the same subdirectory that stores stdout/stderr.
>> >> +    # No need to parallelize coccinelle in this case - this mode takes only
>> >> +    # one file input.
>> >> +    NPROC=1
>> >
>> > Since I am also changing Coccinelle to avoid the problem, maybe it would
>> > be better to just remove the explanation sentence (If coccinelle also runs
>> > in parallel,...).
>> >
>> > julia
>>
>> OK.  Which lines are unneeded?
>>
>> Is it OK to remove all the comments, then just add "NPROC=1"?
>
> How about keeping "# No need to parallelize coccinelle in this case - this
> mode takes only one file input."?

Fair enough.

I will send v4 shortly.



> But if you think it is obvious, it would be ok to remove it also.  Someone
> can always do git blame and read the commit message if they have
> questions.
>
> julia
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

[Cocci] [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

2017-11-14 20:09 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>
>
> On Tue, 14 Nov 2017, Masahiro Yamada wrote:
>
>> Hi Julia,
>>
>>
>> 2017-11-14 18:49 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>> >> +
>> >> +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
>> >> +    # If coccinelle also runs in parallel, it fails because multiple processes
>> >> +    # try to get access to the same subdirectory that stores stdout/stderr.
>> >> +    # No need to parallelize coccinelle in this case - this mode takes only
>> >> +    # one file input.
>> >> +    NPROC=1
>> >
>> > Since I am also changing Coccinelle to avoid the problem, maybe it would
>> > be better to just remove the explanation sentence (If coccinelle also runs
>> > in parallel,...).
>> >
>> > julia
>>
>> OK.  Which lines are unneeded?
>>
>> Is it OK to remove all the comments, then just add "NPROC=1"?
>
> How about keeping "# No need to parallelize coccinelle in this case - this
> mode takes only one file input."?

Fair enough.

I will send v4 shortly.



> But if you think it is obvious, it would be ok to remove it also.  Someone
> can always do git blame and read the commit message if they have
> questions.
>
> julia
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

Re: [Cocci] [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Luis R. Rodriguez]

On Tue, Nov 14, 2017 at 06:04:49PM +0900, Masahiro Yamada wrote:
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 040a8b1..7da82a1 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -70,6 +64,13 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
>      # Take only the last argument, which is the C file to test
>      shift $(( $# - 1 ))
>      OPTIONS="$COCCIINCLUDE $1"
> +
> +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
> +    # If coccinelle also runs in parallel, it fails because multiple processes
> +    # try to get access to the same subdirectory that stores stdout/stderr.
> +    # No need to parallelize coccinelle in this case - this mode takes only
> +    # one file input.
> +    NPROC=1

Shouldn't this also warn to the user, and recommend to use a proper form to
parallelize coccinelle? Otherwise a user might get the impression they are
parallelizing coccinelle where they really did not.

  Luis

>  else
>      ONLINE=0
>      if [ "$KBUILD_EXTMOD" = "" ] ; then

[Cocci] [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Luis R. Rodriguez]

On Tue, Nov 14, 2017 at 06:04:49PM +0900, Masahiro Yamada wrote:
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 040a8b1..7da82a1 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -70,6 +64,13 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
>      # Take only the last argument, which is the C file to test
>      shift $(( $# - 1 ))
>      OPTIONS="$COCCIINCLUDE $1"
> +
> +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
> +    # If coccinelle also runs in parallel, it fails because multiple processes
> +    # try to get access to the same subdirectory that stores stdout/stderr.
> +    # No need to parallelize coccinelle in this case - this mode takes only
> +    # one file input.
> +    NPROC=1

Shouldn't this also warn to the user, and recommend to use a proper form to
parallelize coccinelle? Otherwise a user might get the impression they are
parallelizing coccinelle where they really did not.

  Luis

>  else
>      ONLINE=0
>      if [ "$KBUILD_EXTMOD" = "" ] ; then

Re: [Cocci] [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, Luis R. Rodriguez wrote:

> On Tue, Nov 14, 2017 at 06:04:49PM +0900, Masahiro Yamada wrote:
> > diff --git a/scripts/coccicheck b/scripts/coccicheck
> > index 040a8b1..7da82a1 100755
> > --- a/scripts/coccicheck
> > +++ b/scripts/coccicheck
> > @@ -70,6 +64,13 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
> >      # Take only the last argument, which is the C file to test
> >      shift $(( $# - 1 ))
> >      OPTIONS="$COCCIINCLUDE $1"
> > +
> > +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
> > +    # If coccinelle also runs in parallel, it fails because multiple processes
> > +    # try to get access to the same subdirectory that stores stdout/stderr.
> > +    # No need to parallelize coccinelle in this case - this mode takes only
> > +    # one file input.
> > +    NPROC=1
>
> Shouldn't this also warn to the user, and recommend to use a proper form to
> parallelize coccinelle? Otherwise a user might get the impression they are
> parallelizing coccinelle where they really did not.

Coccinelle sees a file at a time when using C=1 or C=2.  There is nothing
to parallelize at the Coccinelle level, if you want to use these options.

julia


>
>   Luis
>
> >  else
> >      ONLINE=0
> >      if [ "$KBUILD_EXTMOD" = "" ] ; then
> _______________________________________________
> Cocci mailing list
> Cocci@systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>

[Cocci] [PATCH v3] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, Luis R. Rodriguez wrote:

> On Tue, Nov 14, 2017 at 06:04:49PM +0900, Masahiro Yamada wrote:
> > diff --git a/scripts/coccicheck b/scripts/coccicheck
> > index 040a8b1..7da82a1 100755
> > --- a/scripts/coccicheck
> > +++ b/scripts/coccicheck
> > @@ -70,6 +64,13 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
> >      # Take only the last argument, which is the C file to test
> >      shift $(( $# - 1 ))
> >      OPTIONS="$COCCIINCLUDE $1"
> > +
> > +    # If -j option is given to Make, scripts/coccicheck runs in parallel.
> > +    # If coccinelle also runs in parallel, it fails because multiple processes
> > +    # try to get access to the same subdirectory that stores stdout/stderr.
> > +    # No need to parallelize coccinelle in this case - this mode takes only
> > +    # one file input.
> > +    NPROC=1
>
> Shouldn't this also warn to the user, and recommend to use a proper form to
> parallelize coccinelle? Otherwise a user might get the impression they are
> parallelizing coccinelle where they really did not.

Coccinelle sees a file at a time when using C=1 or C=2.  There is nothing
to parallelize at the Coccinelle level, if you want to use these options.

julia


>
>   Luis
>
> >  else
> >      ONLINE=0
> >      if [ "$KBUILD_EXTMOD" = "" ] ; then
> _______________________________________________
> Cocci mailing list
> Cocci at systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>