[PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
"coccicheck failed" error messages.

I do not know the coccinelle internals, but I guess --jobs does not
work well if spatch is invoked from Make running in parallel.
Disable --jobs in this case.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Grep '-j' instead of '--jobserver-auth'.
    '--jobserver-*' is not a stable option flag.
    Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
  - Add -q option to grep

 scripts/coccicheck | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 040a8b1..8bab11e 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
     # Take only the last argument, which is the C file to test
     shift $(( $# - 1 ))
     OPTIONS="$COCCIINCLUDE $1"
+
+    # --jobs does not work if Make is running in parallel
+    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
 else
     ONLINE=0
     if [ "$KBUILD_EXTMOD" = "" ] ; then
-- 
2.7.4

[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
"coccicheck failed" error messages.

I do not know the coccinelle internals, but I guess --jobs does not
work well if spatch is invoked from Make running in parallel.
Disable --jobs in this case.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

Changes in v2:
  - Grep '-j' instead of '--jobserver-auth'.
    '--jobserver-*' is not a stable option flag.
    Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
  - Add -q option to grep

 scripts/coccicheck | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 040a8b1..8bab11e 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
     # Take only the last argument, which is the C file to test
     shift $(( $# - 1 ))
     OPTIONS="$COCCIINCLUDE $1"
+
+    # --jobs does not work if Make is running in parallel
+    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
 else
     ONLINE=0
     if [ "$KBUILD_EXTMOD" = "" ] ; then
-- 
2.7.4

Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Thu, 9 Nov 2017, Masahiro Yamada wrote:

> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> "coccicheck failed" error messages.

The question is where parallelism should be specified.  Currently, make
coccicheck picks up the number of cores on the machine and passes that to
Coccinelle.

OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"

On my 80 core machine with hyperthreading, this runs 160 jobs in parallel,
while in practice that degrades the performance as compared to 40 or 80
cores.

On the other hand, if we use the make command line argument (-j), then we
will only get parallelism up to the number of semantic patches.  Since
some finish quickly, there will be a lot of wasted cycles.

The best would be that the user knows what works well for his machine, and
specifies it on the command line, and then that value gets propagated to
Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in
parallel but instead would cause Coccinelle to run one semantic patch on 8
files in parallel.  But I don't know if that can be done.

julia

>
> I do not know the coccinelle internals, but I guess --jobs does not
> work well if spatch is invoked from Make running in parallel.
> Disable --jobs in this case.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Changes in v2:
>   - Grep '-j' instead of '--jobserver-auth'.
>     '--jobserver-*' is not a stable option flag.
>     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
>   - Add -q option to grep
>
>  scripts/coccicheck | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 040a8b1..8bab11e 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
>      # Take only the last argument, which is the C file to test
>      shift $(( $# - 1 ))
>      OPTIONS="$COCCIINCLUDE $1"
> +
> +    # --jobs does not work if Make is running in parallel
> +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
>  else
>      ONLINE=0
>      if [ "$KBUILD_EXTMOD" = "" ] ; then
> --
> 2.7.4
>
>

[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Thu, 9 Nov 2017, Masahiro Yamada wrote:

> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> "coccicheck failed" error messages.

The question is where parallelism should be specified.  Currently, make
coccicheck picks up the number of cores on the machine and passes that to
Coccinelle.

OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"

On my 80 core machine with hyperthreading, this runs 160 jobs in parallel,
while in practice that degrades the performance as compared to 40 or 80
cores.

On the other hand, if we use the make command line argument (-j), then we
will only get parallelism up to the number of semantic patches.  Since
some finish quickly, there will be a lot of wasted cycles.

The best would be that the user knows what works well for his machine, and
specifies it on the command line, and then that value gets propagated to
Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in
parallel but instead would cause Coccinelle to run one semantic patch on 8
files in parallel.  But I don't know if that can be done.

julia

>
> I do not know the coccinelle internals, but I guess --jobs does not
> work well if spatch is invoked from Make running in parallel.
> Disable --jobs in this case.
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Changes in v2:
>   - Grep '-j' instead of '--jobserver-auth'.
>     '--jobserver-*' is not a stable option flag.
>     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
>   - Add -q option to grep
>
>  scripts/coccicheck | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 040a8b1..8bab11e 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
>      # Take only the last argument, which is the C file to test
>      shift $(( $# - 1 ))
>      OPTIONS="$COCCIINCLUDE $1"
> +
> +    # --jobs does not work if Make is running in parallel
> +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
>  else
>      ONLINE=0
>      if [ "$KBUILD_EXTMOD" = "" ] ; then
> --
> 2.7.4
>
>

Re: [Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Fri, 10 Nov 2017, Julia Lawall wrote:

>
>
> On Thu, 9 Nov 2017, Masahiro Yamada wrote:
>
> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> > "coccicheck failed" error messages.
>
> The question is where parallelism should be specified.  Currently, make
> coccicheck picks up the number of cores on the machine and passes that to
> Coccinelle.
>
> OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
>
> On my 80 core machine with hyperthreading, this runs 160 jobs in parallel,
> while in practice that degrades the performance as compared to 40 or 80
> cores.
>
> On the other hand, if we use the make command line argument (-j), then we
> will only get parallelism up to the number of semantic patches.  Since
> some finish quickly, there will be a lot of wasted cycles.
>
> The best would be that the user knows what works well for his machine, and
> specifies it on the command line, and then that value gets propagated to
> Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in
> parallel but instead would cause Coccinelle to run one semantic patch on 8
> files in parallel.  But I don't know if that can be done.

Sorry for these fairly nonsensical comments.  make -j is going to consider
every file, then parse and run every semantic patch on that file.  If the
parallelism is pushed down into Coccinelle, each semantic patch will be
parsed only once, and then Coccinelle will choose the files for which it
is relevant.  If indexing is used (idutils, glimpse), then for semantic
patches that focus on specific keywords, Coccinelle will efficiently
ignore files that are not relevant.  I don't think there would be many
cases where make -j would win.  Perhaps it would be possible to detect
its used and abort with an appopriate message?

julia


>
> julia
>
> >
> > I do not know the coccinelle internals, but I guess --jobs does not
> > work well if spatch is invoked from Make running in parallel.
> > Disable --jobs in this case.
> >
> > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> > ---
> >
> > Changes in v2:
> >   - Grep '-j' instead of '--jobserver-auth'.
> >     '--jobserver-*' is not a stable option flag.
> >     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
> >   - Add -q option to grep
> >
> >  scripts/coccicheck | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/scripts/coccicheck b/scripts/coccicheck
> > index 040a8b1..8bab11e 100755
> > --- a/scripts/coccicheck
> > +++ b/scripts/coccicheck
> > @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
> >      # Take only the last argument, which is the C file to test
> >      shift $(( $# - 1 ))
> >      OPTIONS="$COCCIINCLUDE $1"
> > +
> > +    # --jobs does not work if Make is running in parallel
> > +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
> >  else
> >      ONLINE=0
> >      if [ "$KBUILD_EXTMOD" = "" ] ; then
> > --
> > 2.7.4
> >
> >
> _______________________________________________
> Cocci mailing list
> Cocci@systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>

[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Fri, 10 Nov 2017, Julia Lawall wrote:

>
>
> On Thu, 9 Nov 2017, Masahiro Yamada wrote:
>
> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> > "coccicheck failed" error messages.
>
> The question is where parallelism should be specified.  Currently, make
> coccicheck picks up the number of cores on the machine and passes that to
> Coccinelle.
>
> OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
>
> On my 80 core machine with hyperthreading, this runs 160 jobs in parallel,
> while in practice that degrades the performance as compared to 40 or 80
> cores.
>
> On the other hand, if we use the make command line argument (-j), then we
> will only get parallelism up to the number of semantic patches.  Since
> some finish quickly, there will be a lot of wasted cycles.
>
> The best would be that the user knows what works well for his machine, and
> specifies it on the command line, and then that value gets propagated to
> Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in
> parallel but instead would cause Coccinelle to run one semantic patch on 8
> files in parallel.  But I don't know if that can be done.

Sorry for these fairly nonsensical comments.  make -j is going to consider
every file, then parse and run every semantic patch on that file.  If the
parallelism is pushed down into Coccinelle, each semantic patch will be
parsed only once, and then Coccinelle will choose the files for which it
is relevant.  If indexing is used (idutils, glimpse), then for semantic
patches that focus on specific keywords, Coccinelle will efficiently
ignore files that are not relevant.  I don't think there would be many
cases where make -j would win.  Perhaps it would be possible to detect
its used and abort with an appopriate message?

julia


>
> julia
>
> >
> > I do not know the coccinelle internals, but I guess --jobs does not
> > work well if spatch is invoked from Make running in parallel.
> > Disable --jobs in this case.
> >
> > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> > ---
> >
> > Changes in v2:
> >   - Grep '-j' instead of '--jobserver-auth'.
> >     '--jobserver-*' is not a stable option flag.
> >     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
> >   - Add -q option to grep
> >
> >  scripts/coccicheck | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/scripts/coccicheck b/scripts/coccicheck
> > index 040a8b1..8bab11e 100755
> > --- a/scripts/coccicheck
> > +++ b/scripts/coccicheck
> > @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
> >      # Take only the last argument, which is the C file to test
> >      shift $(( $# - 1 ))
> >      OPTIONS="$COCCIINCLUDE $1"
> > +
> > +    # --jobs does not work if Make is running in parallel
> > +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
> >  else
> >      ONLINE=0
> >      if [ "$KBUILD_EXTMOD" = "" ] ; then
> > --
> > 2.7.4
> >
> >
> _______________________________________________
> Cocci mailing list
> Cocci at systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
>

Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Thu, 9 Nov 2017, Masahiro Yamada wrote:

> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> "coccicheck failed" error messages.
>
> I do not know the coccinelle internals, but I guess --jobs does not
> work well if spatch is invoked from Make running in parallel.
> Disable --jobs in this case.

Why is this change under:

if [ "$C" = "1" -o "$C" = "2" ];

The coccicheck failed messages come also if one runs Coccinelle on the
entire kernel.

julia


>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Changes in v2:
>   - Grep '-j' instead of '--jobserver-auth'.
>     '--jobserver-*' is not a stable option flag.
>     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
>   - Add -q option to grep
>
>  scripts/coccicheck | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 040a8b1..8bab11e 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
>      # Take only the last argument, which is the C file to test
>      shift $(( $# - 1 ))
>      OPTIONS="$COCCIINCLUDE $1"
> +
> +    # --jobs does not work if Make is running in parallel
> +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
>  else
>      ONLINE=0
>      if [ "$KBUILD_EXTMOD" = "" ] ; then
> --
> 2.7.4
>
>

[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Thu, 9 Nov 2017, Masahiro Yamada wrote:

> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> "coccicheck failed" error messages.
>
> I do not know the coccinelle internals, but I guess --jobs does not
> work well if spatch is invoked from Make running in parallel.
> Disable --jobs in this case.

Why is this change under:

if [ "$C" = "1" -o "$C" = "2" ];

The coccicheck failed messages come also if one runs Coccinelle on the
entire kernel.

julia


>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
>
> Changes in v2:
>   - Grep '-j' instead of '--jobserver-auth'.
>     '--jobserver-*' is not a stable option flag.
>     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
>   - Add -q option to grep
>
>  scripts/coccicheck | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 040a8b1..8bab11e 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
>      # Take only the last argument, which is the C file to test
>      shift $(( $# - 1 ))
>      OPTIONS="$COCCIINCLUDE $1"
> +
> +    # --jobs does not work if Make is running in parallel
> +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
>  else
>      ONLINE=0
>      if [ "$KBUILD_EXTMOD" = "" ] ; then
> --
> 2.7.4
>
>

Re: [Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

Hi Julia,


2017-11-11 16:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>
>
> On Fri, 10 Nov 2017, Julia Lawall wrote:
>
>>
>>
>> On Thu, 9 Nov 2017, Masahiro Yamada wrote:
>>
>> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
>> > "coccicheck failed" error messages.
>>
>> The question is where parallelism should be specified.  Currently, make
>> coccicheck picks up the number of cores on the machine and passes that to
>> Coccinelle.
>>
>> OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
>>
>> On my 80 core machine with hyperthreading, this runs 160 jobs in parallel,
>> while in practice that degrades the performance as compared to 40 or 80
>> cores.
>>
>> On the other hand, if we use the make command line argument (-j), then we
>> will only get parallelism up to the number of semantic patches.  Since
>> some finish quickly, there will be a lot of wasted cycles.
>>
>> The best would be that the user knows what works well for his machine, and
>> specifies it on the command line, and then that value gets propagated to
>> Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in
>> parallel but instead would cause Coccinelle to run one semantic patch on 8
>> files in parallel.  But I don't know if that can be done.
>
> Sorry for these fairly nonsensical comments.  make -j is going to consider
> every file, then parse and run every semantic patch on that file.  If the
> parallelism is pushed down into Coccinelle, each semantic patch will be
> parsed only once, and then Coccinelle will choose the files for which it
> is relevant.  If indexing is used (idutils, glimpse), then for semantic
> patches that focus on specific keywords, Coccinelle will efficiently
> ignore files that are not relevant.  I don't think there would be many
> cases where make -j would win.  Perhaps it would be possible to detect
> its used and abort with an appopriate message?


I am afraid you and I are talking different things.


For a usual usage of coccicheck, only one thread runs scripts/coccicheck
even if -j is passed from the command line.

coccicheck provides "J" to specify parallelism.

if [ -z "$J" ]; then
        NPROC=$(getconf _NPROCESSORS_ONLN)
else
        NPROC="$J"
fi


If you are unhappy with 160 threading, you can give J=40 from the command line.



My patch addresses a problem where coccicheck is used as CHECK.
The default of CHECK is "sparse", but you can use any checker tool.

In CHECK=scripts/coccicheck case, if -j is passed, all tasks run in parallel
under control of GNU Make, so scripts/coccicheck is also invoked from
multiple threads.
Passing --jobs to spatch is not sensible because it checks only one file.




> julia
>
>
>>
>> julia
>>
>> >
>> > I do not know the coccinelle internals, but I guess --jobs does not
>> > work well if spatch is invoked from Make running in parallel.
>> > Disable --jobs in this case.
>> >
>> > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> > ---
>> >
>> > Changes in v2:
>> >   - Grep '-j' instead of '--jobserver-auth'.
>> >     '--jobserver-*' is not a stable option flag.
>> >     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
>> >   - Add -q option to grep
>> >
>> >  scripts/coccicheck | 3 +++
>> >  1 file changed, 3 insertions(+)
>> >
>> > diff --git a/scripts/coccicheck b/scripts/coccicheck
>> > index 040a8b1..8bab11e 100755
>> > --- a/scripts/coccicheck
>> > +++ b/scripts/coccicheck
>> > @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
>> >      # Take only the last argument, which is the C file to test
>> >      shift $(( $# - 1 ))
>> >      OPTIONS="$COCCIINCLUDE $1"
>> > +
>> > +    # --jobs does not work if Make is running in parallel
>> > +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
>> >  else
>> >      ONLINE=0
>> >      if [ "$KBUILD_EXTMOD" = "" ] ; then
>> > --
>> > 2.7.4
>> >
>> >
>> _______________________________________________
>> Cocci mailing list
>> Cocci@systeme.lip6.fr
>> https://systeme.lip6.fr/mailman/listinfo/cocci
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

Hi Julia,


2017-11-11 16:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>
>
> On Fri, 10 Nov 2017, Julia Lawall wrote:
>
>>
>>
>> On Thu, 9 Nov 2017, Masahiro Yamada wrote:
>>
>> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
>> > "coccicheck failed" error messages.
>>
>> The question is where parallelism should be specified.  Currently, make
>> coccicheck picks up the number of cores on the machine and passes that to
>> Coccinelle.
>>
>> OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
>>
>> On my 80 core machine with hyperthreading, this runs 160 jobs in parallel,
>> while in practice that degrades the performance as compared to 40 or 80
>> cores.
>>
>> On the other hand, if we use the make command line argument (-j), then we
>> will only get parallelism up to the number of semantic patches.  Since
>> some finish quickly, there will be a lot of wasted cycles.
>>
>> The best would be that the user knows what works well for his machine, and
>> specifies it on the command line, and then that value gets propagated to
>> Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in
>> parallel but instead would cause Coccinelle to run one semantic patch on 8
>> files in parallel.  But I don't know if that can be done.
>
> Sorry for these fairly nonsensical comments.  make -j is going to consider
> every file, then parse and run every semantic patch on that file.  If the
> parallelism is pushed down into Coccinelle, each semantic patch will be
> parsed only once, and then Coccinelle will choose the files for which it
> is relevant.  If indexing is used (idutils, glimpse), then for semantic
> patches that focus on specific keywords, Coccinelle will efficiently
> ignore files that are not relevant.  I don't think there would be many
> cases where make -j would win.  Perhaps it would be possible to detect
> its used and abort with an appopriate message?


I am afraid you and I are talking different things.


For a usual usage of coccicheck, only one thread runs scripts/coccicheck
even if -j is passed from the command line.

coccicheck provides "J" to specify parallelism.

if [ -z "$J" ]; then
        NPROC=$(getconf _NPROCESSORS_ONLN)
else
        NPROC="$J"
fi


If you are unhappy with 160 threading, you can give J=40 from the command line.



My patch addresses a problem where coccicheck is used as CHECK.
The default of CHECK is "sparse", but you can use any checker tool.

In CHECK=scripts/coccicheck case, if -j is passed, all tasks run in parallel
under control of GNU Make, so scripts/coccicheck is also invoked from
multiple threads.
Passing --jobs to spatch is not sensible because it checks only one file.




> julia
>
>
>>
>> julia
>>
>> >
>> > I do not know the coccinelle internals, but I guess --jobs does not
>> > work well if spatch is invoked from Make running in parallel.
>> > Disable --jobs in this case.
>> >
>> > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> > ---
>> >
>> > Changes in v2:
>> >   - Grep '-j' instead of '--jobserver-auth'.
>> >     '--jobserver-*' is not a stable option flag.
>> >     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
>> >   - Add -q option to grep
>> >
>> >  scripts/coccicheck | 3 +++
>> >  1 file changed, 3 insertions(+)
>> >
>> > diff --git a/scripts/coccicheck b/scripts/coccicheck
>> > index 040a8b1..8bab11e 100755
>> > --- a/scripts/coccicheck
>> > +++ b/scripts/coccicheck
>> > @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
>> >      # Take only the last argument, which is the C file to test
>> >      shift $(( $# - 1 ))
>> >      OPTIONS="$COCCIINCLUDE $1"
>> > +
>> > +    # --jobs does not work if Make is running in parallel
>> > +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
>> >  else
>> >      ONLINE=0
>> >      if [ "$KBUILD_EXTMOD" = "" ] ; then
>> > --
>> > 2.7.4
>> >
>> >
>> _______________________________________________
>> Cocci mailing list
>> Cocci at systeme.lip6.fr
>> https://systeme.lip6.fr/mailman/listinfo/cocci
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

Hi Julia,


2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>
>
> On Thu, 9 Nov 2017, Masahiro Yamada wrote:
>
>> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
>> "coccicheck failed" error messages.
>>
>> I do not know the coccinelle internals, but I guess --jobs does not
>> work well if spatch is invoked from Make running in parallel.
>> Disable --jobs in this case.
>
> Why is this change under:
>
> if [ "$C" = "1" -o "$C" = "2" ];
>
> The coccicheck failed messages come also if one runs Coccinelle on the
> entire kernel.

As far as I tested, "coccicheck failed" error only happens
when ONLINE=1.


make -j8 C=1 CHECK=scripts/coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci

emits lots of errors.


make -j8 coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci

is fine.


Have you tested it?
Do you mean you got a different result from mine?



> julia
>
>
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> Changes in v2:
>>   - Grep '-j' instead of '--jobserver-auth'.
>>     '--jobserver-*' is not a stable option flag.
>>     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
>>   - Add -q option to grep
>>
>>  scripts/coccicheck | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/scripts/coccicheck b/scripts/coccicheck
>> index 040a8b1..8bab11e 100755
>> --- a/scripts/coccicheck
>> +++ b/scripts/coccicheck
>> @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
>>      # Take only the last argument, which is the C file to test
>>      shift $(( $# - 1 ))
>>      OPTIONS="$COCCIINCLUDE $1"
>> +
>> +    # --jobs does not work if Make is running in parallel
>> +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
>>  else
>>      ONLINE=0
>>      if [ "$KBUILD_EXTMOD" = "" ] ; then
>> --
>> 2.7.4
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

Hi Julia,


2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>
>
> On Thu, 9 Nov 2017, Masahiro Yamada wrote:
>
>> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
>> "coccicheck failed" error messages.
>>
>> I do not know the coccinelle internals, but I guess --jobs does not
>> work well if spatch is invoked from Make running in parallel.
>> Disable --jobs in this case.
>
> Why is this change under:
>
> if [ "$C" = "1" -o "$C" = "2" ];
>
> The coccicheck failed messages come also if one runs Coccinelle on the
> entire kernel.

As far as I tested, "coccicheck failed" error only happens
when ONLINE=1.


make -j8 C=1 CHECK=scripts/coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci

emits lots of errors.


make -j8 coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci

is fine.


Have you tested it?
Do you mean you got a different result from mine?



> julia
>
>
>>
>> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
>> ---
>>
>> Changes in v2:
>>   - Grep '-j' instead of '--jobserver-auth'.
>>     '--jobserver-*' is not a stable option flag.
>>     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
>>   - Add -q option to grep
>>
>>  scripts/coccicheck | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/scripts/coccicheck b/scripts/coccicheck
>> index 040a8b1..8bab11e 100755
>> --- a/scripts/coccicheck
>> +++ b/scripts/coccicheck
>> @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
>>      # Take only the last argument, which is the C file to test
>>      shift $(( $# - 1 ))
>>      OPTIONS="$COCCIINCLUDE $1"
>> +
>> +    # --jobs does not work if Make is running in parallel
>> +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
>>  else
>>      ONLINE=0
>>      if [ "$KBUILD_EXTMOD" = "" ] ; then
>> --
>> 2.7.4
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Best Regards
Masahiro Yamada

Re: [Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, Masahiro Yamada wrote:

> Hi Julia,
>
>
> 2017-11-11 16:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
> >
> >
> > On Fri, 10 Nov 2017, Julia Lawall wrote:
> >
> >>
> >>
> >> On Thu, 9 Nov 2017, Masahiro Yamada wrote:
> >>
> >> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> >> > "coccicheck failed" error messages.
> >>
> >> The question is where parallelism should be specified.  Currently, make
> >> coccicheck picks up the number of cores on the machine and passes that to
> >> Coccinelle.
> >>
> >> OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
> >>
> >> On my 80 core machine with hyperthreading, this runs 160 jobs in parallel,
> >> while in practice that degrades the performance as compared to 40 or 80
> >> cores.
> >>
> >> On the other hand, if we use the make command line argument (-j), then we
> >> will only get parallelism up to the number of semantic patches.  Since
> >> some finish quickly, there will be a lot of wasted cycles.
> >>
> >> The best would be that the user knows what works well for his machine, and
> >> specifies it on the command line, and then that value gets propagated to
> >> Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in
> >> parallel but instead would cause Coccinelle to run one semantic patch on 8
> >> files in parallel.  But I don't know if that can be done.
> >
> > Sorry for these fairly nonsensical comments.  make -j is going to consider
> > every file, then parse and run every semantic patch on that file.  If the
> > parallelism is pushed down into Coccinelle, each semantic patch will be
> > parsed only once, and then Coccinelle will choose the files for which it
> > is relevant.  If indexing is used (idutils, glimpse), then for semantic
> > patches that focus on specific keywords, Coccinelle will efficiently
> > ignore files that are not relevant.  I don't think there would be many
> > cases where make -j would win.  Perhaps it would be possible to detect
> > its used and abort with an appopriate message?
>
>
> I am afraid you and I are talking different things.
>
>
> For a usual usage of coccicheck, only one thread runs scripts/coccicheck
> even if -j is passed from the command line.
>
> coccicheck provides "J" to specify parallelism.
>
> if [ -z "$J" ]; then
>         NPROC=$(getconf _NPROCESSORS_ONLN)
> else
>         NPROC="$J"
> fi

Even if J is not specified, then it still runs with the maximum number of
threads:

Coccinelle parallelization
---------------------------

By default, coccicheck tries to run as parallel as possible.

Indeed, J= does set the number of threads speficied.

> My patch addresses a problem where coccicheck is used as CHECK.
> The default of CHECK is "sparse", but you can use any checker tool.
>
> In CHECK=scripts/coccicheck case, if -j is passed, all tasks run in parallel
> under control of GNU Make, so scripts/coccicheck is also invoked from
> multiple threads.
> Passing --jobs to spatch is not sensible because it checks only one file.

OK.  I tried a simple make coccicheck -j4 and indeed it does not seem to
be complaining.  The number of spatch processes goes over 160 though.

julia

[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, Masahiro Yamada wrote:

> Hi Julia,
>
>
> 2017-11-11 16:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
> >
> >
> > On Fri, 10 Nov 2017, Julia Lawall wrote:
> >
> >>
> >>
> >> On Thu, 9 Nov 2017, Masahiro Yamada wrote:
> >>
> >> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> >> > "coccicheck failed" error messages.
> >>
> >> The question is where parallelism should be specified.  Currently, make
> >> coccicheck picks up the number of cores on the machine and passes that to
> >> Coccinelle.
> >>
> >> OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
> >>
> >> On my 80 core machine with hyperthreading, this runs 160 jobs in parallel,
> >> while in practice that degrades the performance as compared to 40 or 80
> >> cores.
> >>
> >> On the other hand, if we use the make command line argument (-j), then we
> >> will only get parallelism up to the number of semantic patches.  Since
> >> some finish quickly, there will be a lot of wasted cycles.
> >>
> >> The best would be that the user knows what works well for his machine, and
> >> specifies it on the command line, and then that value gets propagated to
> >> Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in
> >> parallel but instead would cause Coccinelle to run one semantic patch on 8
> >> files in parallel.  But I don't know if that can be done.
> >
> > Sorry for these fairly nonsensical comments.  make -j is going to consider
> > every file, then parse and run every semantic patch on that file.  If the
> > parallelism is pushed down into Coccinelle, each semantic patch will be
> > parsed only once, and then Coccinelle will choose the files for which it
> > is relevant.  If indexing is used (idutils, glimpse), then for semantic
> > patches that focus on specific keywords, Coccinelle will efficiently
> > ignore files that are not relevant.  I don't think there would be many
> > cases where make -j would win.  Perhaps it would be possible to detect
> > its used and abort with an appopriate message?
>
>
> I am afraid you and I are talking different things.
>
>
> For a usual usage of coccicheck, only one thread runs scripts/coccicheck
> even if -j is passed from the command line.
>
> coccicheck provides "J" to specify parallelism.
>
> if [ -z "$J" ]; then
>         NPROC=$(getconf _NPROCESSORS_ONLN)
> else
>         NPROC="$J"
> fi

Even if J is not specified, then it still runs with the maximum number of
threads:

Coccinelle parallelization
---------------------------

By default, coccicheck tries to run as parallel as possible.

Indeed, J= does set the number of threads speficied.

> My patch addresses a problem where coccicheck is used as CHECK.
> The default of CHECK is "sparse", but you can use any checker tool.
>
> In CHECK=scripts/coccicheck case, if -j is passed, all tasks run in parallel
> under control of GNU Make, so scripts/coccicheck is also invoked from
> multiple threads.
> Passing --jobs to spatch is not sensible because it checks only one file.

OK.  I tried a simple make coccicheck -j4 and indeed it does not seem to
be complaining.  The number of spatch processes goes over 160 though.

julia

Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, Masahiro Yamada wrote:

> Hi Julia,
>
>
> 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
> >
> >
> > On Thu, 9 Nov 2017, Masahiro Yamada wrote:
> >
> >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> >> "coccicheck failed" error messages.
> >>
> >> I do not know the coccinelle internals, but I guess --jobs does not
> >> work well if spatch is invoked from Make running in parallel.
> >> Disable --jobs in this case.
> >
> > Why is this change under:
> >
> > if [ "$C" = "1" -o "$C" = "2" ];
> >
> > The coccicheck failed messages come also if one runs Coccinelle on the
> > entire kernel.
>
> As far as I tested, "coccicheck failed" error only happens
> when ONLINE=1.
>
>
> make -j8 C=1 CHECK=scripts/coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>
> emits lots of errors.
>
>
> make -j8 coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>
> is fine.
>
>
> Have you tested it?
> Do you mean you got a different result from mine?

I agree with your results, with respect to the number of errors.

julia

>
>
>
> > julia
> >
> >
> >>
> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> >> ---
> >>
> >> Changes in v2:
> >>   - Grep '-j' instead of '--jobserver-auth'.
> >>     '--jobserver-*' is not a stable option flag.
> >>     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
> >>   - Add -q option to grep
> >>
> >>  scripts/coccicheck | 3 +++
> >>  1 file changed, 3 insertions(+)
> >>
> >> diff --git a/scripts/coccicheck b/scripts/coccicheck
> >> index 040a8b1..8bab11e 100755
> >> --- a/scripts/coccicheck
> >> +++ b/scripts/coccicheck
> >> @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
> >>      # Take only the last argument, which is the C file to test
> >>      shift $(( $# - 1 ))
> >>      OPTIONS="$COCCIINCLUDE $1"
> >> +
> >> +    # --jobs does not work if Make is running in parallel
> >> +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
> >>  else
> >>      ONLINE=0
> >>      if [ "$KBUILD_EXTMOD" = "" ] ; then
> >> --
> >> 2.7.4
> >>
> >>
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Best Regards
> Masahiro Yamada
>

[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, Masahiro Yamada wrote:

> Hi Julia,
>
>
> 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
> >
> >
> > On Thu, 9 Nov 2017, Masahiro Yamada wrote:
> >
> >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> >> "coccicheck failed" error messages.
> >>
> >> I do not know the coccinelle internals, but I guess --jobs does not
> >> work well if spatch is invoked from Make running in parallel.
> >> Disable --jobs in this case.
> >
> > Why is this change under:
> >
> > if [ "$C" = "1" -o "$C" = "2" ];
> >
> > The coccicheck failed messages come also if one runs Coccinelle on the
> > entire kernel.
>
> As far as I tested, "coccicheck failed" error only happens
> when ONLINE=1.
>
>
> make -j8 C=1 CHECK=scripts/coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>
> emits lots of errors.
>
>
> make -j8 coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>
> is fine.
>
>
> Have you tested it?
> Do you mean you got a different result from mine?

I agree with your results, with respect to the number of errors.

julia

>
>
>
> > julia
> >
> >
> >>
> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> >> ---
> >>
> >> Changes in v2:
> >>   - Grep '-j' instead of '--jobserver-auth'.
> >>     '--jobserver-*' is not a stable option flag.
> >>     Make 4.2 change '--jobserver-fds' into '--jobserver-auth'
> >>   - Add -q option to grep
> >>
> >>  scripts/coccicheck | 3 +++
> >>  1 file changed, 3 insertions(+)
> >>
> >> diff --git a/scripts/coccicheck b/scripts/coccicheck
> >> index 040a8b1..8bab11e 100755
> >> --- a/scripts/coccicheck
> >> +++ b/scripts/coccicheck
> >> @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then
> >>      # Take only the last argument, which is the C file to test
> >>      shift $(( $# - 1 ))
> >>      OPTIONS="$COCCIINCLUDE $1"
> >> +
> >> +    # --jobs does not work if Make is running in parallel
> >> +    echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no"
> >>  else
> >>      ONLINE=0
> >>      if [ "$KBUILD_EXTMOD" = "" ] ; then
> >> --
> >> 2.7.4
> >>
> >>
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> > the body of a message to majordomo at vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Best Regards
> Masahiro Yamada
>

[Cocci] Coccinelle: fix parallel build with CHECK=scripts/coccicheck

[yamada.masahiro at socionext.com]

Hi Markus,

I will add more description in the commit log.
Thanks

> -----Original Message-----
> From: SF Markus Elfring [mailto:elfring at users.sourceforge.net]
> Sent: Saturday, November 11, 2017 1:27 AM
> To: Yamada, Masahiro/?? ?? <yamada.masahiro@socionext.com>
> Cc: cocci at systeme.lip6.fr
> Subject: Re: Coccinelle: fix parallel build with CHECK=scripts/coccicheck
> 
> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> > "coccicheck failed" error messages.
> 
> * Which software versions did you try out for these source file checks?
> 
> * Can the software situation be clarified any more?
> 
> 
> > I do not know the coccinelle internals,
> 
> Would you like to extend your knowledge in this area?
> 
> 
> > but I guess --jobs does not work well if spatch is invoked from Make
> > running in parallel.
> 
> The desired parallelisation support can evolve into additional directions,
> can't it?
> 
> 
> > Disable --jobs in this case.
> 
> How do you think about further possibilities for separation of work between
> Coccinelle and Make processes?
> 
> Regards,
> Markus

Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

Hi Julia,

2017-11-14 1:45 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>
>
> On Tue, 14 Nov 2017, Masahiro Yamada wrote:
>
>> Hi Julia,
>>
>>
>> 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>> >
>> >
>> > On Thu, 9 Nov 2017, Masahiro Yamada wrote:
>> >
>> >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
>> >> "coccicheck failed" error messages.
>> >>
>> >> I do not know the coccinelle internals, but I guess --jobs does not
>> >> work well if spatch is invoked from Make running in parallel.
>> >> Disable --jobs in this case.
>> >
>> > Why is this change under:
>> >
>> > if [ "$C" = "1" -o "$C" = "2" ];
>> >
>> > The coccicheck failed messages come also if one runs Coccinelle on the
>> > entire kernel.
>>
>> As far as I tested, "coccicheck failed" error only happens
>> when ONLINE=1.
>>
>>
>> make -j8 C=1 CHECK=scripts/coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>>
>> emits lots of errors.
>>
>>
>> make -j8 coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>>
>> is fine.
>>
>>
>> Have you tested it?
>> Do you mean you got a different result from mine?
>
> I agree with your results, with respect to the number of errors.
>
> julia
>

So, what shall we do?

If you do not like to fix it (or you can fix coccinelle itself),
I can take back this patch.

I am not a coccinelle developer, so
setting USE_JOBS="no" is the best I can do.




-- 
Best Regards
Masahiro Yamada

[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

Hi Julia,

2017-11-14 1:45 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>
>
> On Tue, 14 Nov 2017, Masahiro Yamada wrote:
>
>> Hi Julia,
>>
>>
>> 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>> >
>> >
>> > On Thu, 9 Nov 2017, Masahiro Yamada wrote:
>> >
>> >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
>> >> "coccicheck failed" error messages.
>> >>
>> >> I do not know the coccinelle internals, but I guess --jobs does not
>> >> work well if spatch is invoked from Make running in parallel.
>> >> Disable --jobs in this case.
>> >
>> > Why is this change under:
>> >
>> > if [ "$C" = "1" -o "$C" = "2" ];
>> >
>> > The coccicheck failed messages come also if one runs Coccinelle on the
>> > entire kernel.
>>
>> As far as I tested, "coccicheck failed" error only happens
>> when ONLINE=1.
>>
>>
>> make -j8 C=1 CHECK=scripts/coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>>
>> emits lots of errors.
>>
>>
>> make -j8 coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>>
>> is fine.
>>
>>
>> Have you tested it?
>> Do you mean you got a different result from mine?
>
> I agree with your results, with respect to the number of errors.
>
> julia
>

So, what shall we do?

If you do not like to fix it (or you can fix coccinelle itself),
I can take back this patch.

I am not a coccinelle developer, so
setting USE_JOBS="no" is the best I can do.




-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, Masahiro Yamada wrote:

> Hi Julia,
>
> 2017-11-14 1:45 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
> >
> >
> > On Tue, 14 Nov 2017, Masahiro Yamada wrote:
> >
> >> Hi Julia,
> >>
> >>
> >> 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
> >> >
> >> >
> >> > On Thu, 9 Nov 2017, Masahiro Yamada wrote:
> >> >
> >> >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> >> >> "coccicheck failed" error messages.
> >> >>
> >> >> I do not know the coccinelle internals, but I guess --jobs does not
> >> >> work well if spatch is invoked from Make running in parallel.
> >> >> Disable --jobs in this case.
> >> >
> >> > Why is this change under:
> >> >
> >> > if [ "$C" = "1" -o "$C" = "2" ];
> >> >
> >> > The coccicheck failed messages come also if one runs Coccinelle on the
> >> > entire kernel.
> >>
> >> As far as I tested, "coccicheck failed" error only happens
> >> when ONLINE=1.
> >>
> >>
> >> make -j8 C=1 CHECK=scripts/coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
> >>
> >> emits lots of errors.
> >>
> >>
> >> make -j8 coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
> >>
> >> is fine.
> >>
> >>
> >> Have you tested it?
> >> Do you mean you got a different result from mine?
> >
> > I agree with your results, with respect to the number of errors.
> >
> > julia
> >
>
> So, what shall we do?
>
> If you do not like to fix it (or you can fix coccinelle itself),
> I can take back this patch.

I'm OK with your fix.  I will check and ack it today.

> I am not a coccinelle developer, so
> setting USE_JOBS="no" is the best I can do.

The problem on the Coccinelle side is that it uses a subdirectory with the
name of the semantic patch to store standard output and standard error for
the different threads.  I didn't want to use a name with the pid, so that
one could easily find this information while Coccinelle is running.
Normally the subdirectory is cleaned up when Coccinelle completes, so
there is only one of them at a time.  Maybe it is best to just add the
pid.  There is the risk that these subdirectories will accumulate if
Coccinelle crashes in a way such that they don't get cleaned up, but
Coccinelle could print a warning if it detects this case, rather than
failing.

Still I think it is useful to do something on the make coccicheck side,
because there is no need for the double layer of parallelism.

julia

[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, Masahiro Yamada wrote:

> Hi Julia,
>
> 2017-11-14 1:45 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
> >
> >
> > On Tue, 14 Nov 2017, Masahiro Yamada wrote:
> >
> >> Hi Julia,
> >>
> >>
> >> 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
> >> >
> >> >
> >> > On Thu, 9 Nov 2017, Masahiro Yamada wrote:
> >> >
> >> >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
> >> >> "coccicheck failed" error messages.
> >> >>
> >> >> I do not know the coccinelle internals, but I guess --jobs does not
> >> >> work well if spatch is invoked from Make running in parallel.
> >> >> Disable --jobs in this case.
> >> >
> >> > Why is this change under:
> >> >
> >> > if [ "$C" = "1" -o "$C" = "2" ];
> >> >
> >> > The coccicheck failed messages come also if one runs Coccinelle on the
> >> > entire kernel.
> >>
> >> As far as I tested, "coccicheck failed" error only happens
> >> when ONLINE=1.
> >>
> >>
> >> make -j8 C=1 CHECK=scripts/coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
> >>
> >> emits lots of errors.
> >>
> >>
> >> make -j8 coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
> >>
> >> is fine.
> >>
> >>
> >> Have you tested it?
> >> Do you mean you got a different result from mine?
> >
> > I agree with your results, with respect to the number of errors.
> >
> > julia
> >
>
> So, what shall we do?
>
> If you do not like to fix it (or you can fix coccinelle itself),
> I can take back this patch.

I'm OK with your fix.  I will check and ack it today.

> I am not a coccinelle developer, so
> setting USE_JOBS="no" is the best I can do.

The problem on the Coccinelle side is that it uses a subdirectory with the
name of the semantic patch to store standard output and standard error for
the different threads.  I didn't want to use a name with the pid, so that
one could easily find this information while Coccinelle is running.
Normally the subdirectory is cleaned up when Coccinelle completes, so
there is only one of them at a time.  Maybe it is best to just add the
pid.  There is the risk that these subdirectories will accumulate if
Coccinelle crashes in a way such that they don't get cleaned up, but
Coccinelle could print a warning if it detects this case, rather than
failing.

Still I think it is useful to do something on the make coccicheck side,
because there is no need for the double layer of parallelism.

julia

Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

Hi Julia,


2017-11-14 15:44 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>
>
> On Tue, 14 Nov 2017, Masahiro Yamada wrote:
>
>> Hi Julia,
>>
>> 2017-11-14 1:45 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>> >
>> >
>> > On Tue, 14 Nov 2017, Masahiro Yamada wrote:
>> >
>> >> Hi Julia,
>> >>
>> >>
>> >> 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>> >> >
>> >> >
>> >> > On Thu, 9 Nov 2017, Masahiro Yamada wrote:
>> >> >
>> >> >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
>> >> >> "coccicheck failed" error messages.
>> >> >>
>> >> >> I do not know the coccinelle internals, but I guess --jobs does not
>> >> >> work well if spatch is invoked from Make running in parallel.
>> >> >> Disable --jobs in this case.
>> >> >
>> >> > Why is this change under:
>> >> >
>> >> > if [ "$C" = "1" -o "$C" = "2" ];
>> >> >
>> >> > The coccicheck failed messages come also if one runs Coccinelle on the
>> >> > entire kernel.
>> >>
>> >> As far as I tested, "coccicheck failed" error only happens
>> >> when ONLINE=1.
>> >>
>> >>
>> >> make -j8 C=1 CHECK=scripts/coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>> >>
>> >> emits lots of errors.
>> >>
>> >>
>> >> make -j8 coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>> >>
>> >> is fine.
>> >>
>> >>
>> >> Have you tested it?
>> >> Do you mean you got a different result from mine?
>> >
>> > I agree with your results, with respect to the number of errors.
>> >
>> > julia
>> >
>>
>> So, what shall we do?
>>
>> If you do not like to fix it (or you can fix coccinelle itself),
>> I can take back this patch.
>
> I'm OK with your fix.  I will check and ack it today.
>
>> I am not a coccinelle developer, so
>> setting USE_JOBS="no" is the best I can do.
>
> The problem on the Coccinelle side is that it uses a subdirectory with the
> name of the semantic patch to store standard output and standard error for
> the different threads.  I didn't want to use a name with the pid, so that
> one could easily find this information while Coccinelle is running.
> Normally the subdirectory is cleaned up when Coccinelle completes, so
> there is only one of them at a time.  Maybe it is best to just add the
> pid.  There is the risk that these subdirectories will accumulate if
> Coccinelle crashes in a way such that they don't get cleaned up, but
> Coccinelle could print a warning if it detects this case, rather than
> failing.
>
> Still I think it is useful to do something on the make coccicheck side,
> because there is no need for the double layer of parallelism.
>


Thanks a lot for detailed explanation!

I brushed up my patch.

Could you check v3, please?


-- 
Best Regards
Masahiro Yamada

[Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Masahiro Yamada]

Hi Julia,


2017-11-14 15:44 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>
>
> On Tue, 14 Nov 2017, Masahiro Yamada wrote:
>
>> Hi Julia,
>>
>> 2017-11-14 1:45 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>> >
>> >
>> > On Tue, 14 Nov 2017, Masahiro Yamada wrote:
>> >
>> >> Hi Julia,
>> >>
>> >>
>> >> 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>:
>> >> >
>> >> >
>> >> > On Thu, 9 Nov 2017, Masahiro Yamada wrote:
>> >> >
>> >> >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of
>> >> >> "coccicheck failed" error messages.
>> >> >>
>> >> >> I do not know the coccinelle internals, but I guess --jobs does not
>> >> >> work well if spatch is invoked from Make running in parallel.
>> >> >> Disable --jobs in this case.
>> >> >
>> >> > Why is this change under:
>> >> >
>> >> > if [ "$C" = "1" -o "$C" = "2" ];
>> >> >
>> >> > The coccicheck failed messages come also if one runs Coccinelle on the
>> >> > entire kernel.
>> >>
>> >> As far as I tested, "coccicheck failed" error only happens
>> >> when ONLINE=1.
>> >>
>> >>
>> >> make -j8 C=1 CHECK=scripts/coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>> >>
>> >> emits lots of errors.
>> >>
>> >>
>> >> make -j8 coccicheck  COCCI=scripts/coccinelle/misc/bugon.cocci
>> >>
>> >> is fine.
>> >>
>> >>
>> >> Have you tested it?
>> >> Do you mean you got a different result from mine?
>> >
>> > I agree with your results, with respect to the number of errors.
>> >
>> > julia
>> >
>>
>> So, what shall we do?
>>
>> If you do not like to fix it (or you can fix coccinelle itself),
>> I can take back this patch.
>
> I'm OK with your fix.  I will check and ack it today.
>
>> I am not a coccinelle developer, so
>> setting USE_JOBS="no" is the best I can do.
>
> The problem on the Coccinelle side is that it uses a subdirectory with the
> name of the semantic patch to store standard output and standard error for
> the different threads.  I didn't want to use a name with the pid, so that
> one could easily find this information while Coccinelle is running.
> Normally the subdirectory is cleaned up when Coccinelle completes, so
> there is only one of them at a time.  Maybe it is best to just add the
> pid.  There is the risk that these subdirectories will accumulate if
> Coccinelle crashes in a way such that they don't get cleaned up, but
> Coccinelle could print a warning if it detects this case, rather than
> failing.
>
> Still I think it is useful to do something on the make coccicheck side,
> because there is no need for the double layer of parallelism.
>


Thanks a lot for detailed explanation!

I brushed up my patch.

Could you check v3, please?


-- 
Best Regards
Masahiro Yamada

[Cocci] Coccinelle: fix parallel build with CHECK=scripts/coccicheck

[SF Markus Elfring]

> The problem on the Coccinelle side is that it uses a subdirectory with the
> name of the semantic patch to store standard output and standard error for
> the different threads.

It is occasionally good to know such background information.


> I didn't want to use a name with the pid, so that one could easily find
> this information while Coccinelle is running.

Do you mark these data as ?hidden? in the file system?


> Normally the subdirectory is cleaned up when Coccinelle completes,
> so there is only one of them at a time.

I find that this can be insufficient occasionally.


> Maybe it is best to just add the pid.

I imagine that the storage algorithm could become better configurable
besides the parameters ?--temp-files? and ?--tmp-dir?.


> There is the risk that these subdirectories will accumulate if
> Coccinelle crashes in a way such that they don't get cleaned up,

I noticed such a situation a few times.


> but Coccinelle could print a warning if it detects this case,
> rather than failing.

I would appreciate such a software adjustment.


> Still I think it is useful to do something on the make coccicheck side,
> because there is no need for the double layer of parallelism.

Would you like to improve the corresponding documentation in any ways?

Regards,
Markus

[Cocci] Coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, SF Markus Elfring wrote:

> > The problem on the Coccinelle side is that it uses a subdirectory with the
> > name of the semantic patch to store standard output and standard error for
> > the different threads.
>
> It is occasionally good to know such background information.
>
>
> > I didn't want to use a name with the pid, so that one could easily find
> > this information while Coccinelle is running.
>
> Do you mark these data as ?hidden? in the file system?

I don't know what this means.

>
>
> > Normally the subdirectory is cleaned up when Coccinelle completes,
> > so there is only one of them at a time.
>
> I find that this can be insufficient occasionally.
>
>
> > Maybe it is best to just add the pid.
>
> I imagine that the storage algorithm could become better configurable
> besides the parameters ?--temp-files? and ?--tmp-dir?.

The problem is that running multiple coccinelles each with parallelism is
fairly pointless, especially on the same semantic patch.  So the only
people who do it will be those who don't understand how Coccinelle works.
Those people won't realize that they should use such extra options either.

julia

>
>
> > There is the risk that these subdirectories will accumulate if
> > Coccinelle crashes in a way such that they don't get cleaned up,
>
> I noticed such a situation a few times.
>
>
> > but Coccinelle could print a warning if it detects this case,
> > rather than failing.
>
> I would appreciate such a software adjustment.
>
>
> > Still I think it is useful to do something on the make coccicheck side,
> > because there is no need for the double layer of parallelism.
>
> Would you like to improve the corresponding documentation in any ways?
>
> Regards,
> Markus
>

[Cocci] Coccinelle: fix parallel build with CHECK=scripts/coccicheck

[SF Markus Elfring]

>>> I didn't want to use a name with the pid, so that one could easily find
>>> this information while Coccinelle is running.
>>
>> Do you mark these data as ?hidden? in the file system?
> 
> I don't know what this means.

Do you fiddle with file attributes or use a dot as the first character
in the file name?


>> I imagine that the storage algorithm could become better configurable
>> besides the parameters ?--temp-files? and ?--tmp-dir?.
> 
> The problem is that running multiple coccinelles each with parallelism is
> fairly pointless,

The situation might be different if the execution environment provides
enough computation resources for processing with special data distributions.


> especially on the same semantic patch.

There can be restrictions usually.


> So the only people who do it will be those who don't understand
> how Coccinelle works.

This might be. - Or do they compare different software versions (in parallel)?


> Those people won't realize that they should use such extra options either.

Some time is needed to become more aware about advanced parameters.

Regards,
Markus

[Cocci] Coccinelle: fix parallel build with CHECK=scripts/coccicheck

[Julia Lawall]

On Tue, 14 Nov 2017, SF Markus Elfring wrote:

> >>> I didn't want to use a name with the pid, so that one could easily find
> >>> this information while Coccinelle is running.
> >>
> >> Do you mark these data as ?hidden? in the file system?
> >
> > I don't know what this means.
>
> Do you fiddle with file attributes or use a dot as the first character
> in the file name?

The goal is that the user can easily find the stdou and stderr information
while the semantic patch is running.  This is useful for long running
semantic patches to see if things are going well or not.

I decided on creating a directory that contains the name of the semantic
patch, then the date, then the pid.  It should be easy for the user to see
which is the most recent, in the case that previous instances are
remaining.

> This might be. - Or do they compare different software versions (in parallel)?

It's only an issue when they have the same name and are run from the same
directory.

> > Those people won't realize that they should use such extra options either.
>
> Some time is needed to become more aware about advanced parameters.

The goal is to avoid getting a mysterious error from make coccicheck, not
to meet the needs of advanced users.

julia

[Cocci] Coccinelle: fix parallel build with CHECK=scripts/coccicheck

[SF Markus Elfring]

> The goal is that the user can easily find the stdou and stderr information
> while the semantic patch is running.

It matters when the tool ?spatch? tries to apply its own parallelisation strategy.


> This is useful for long running semantic patches to see
> if things are going well or not.

Yes, of course.


> I decided on creating a directory that contains the name of the semantic
> patch, then the date, then the pid.

I imagine that time stamps and process identifiers can become fragile.


> It should be easy for the user to see which is the most recent,
> in the case that previous instances are remaining.

This is usually fine.


> The goal is to avoid getting a mysterious error from make coccicheck,
> not to meet the needs of advanced users.

* How advanced are they if they pass the parameter ?CHECK=scripts/coccicheck??

* The Linux make script will be improved so that extra parallelisation
  will be usually avoided by the Coccinelle software.

Regards,
Markus