* [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck @ 2017-11-09 7:00 Masahiro Yamada 2017-11-10 15:42 ` Julia Lawall 2017-11-13 15:30 ` Julia Lawall 0 siblings, 2 replies; 11+ messages in thread From: Masahiro Yamada @ 2017-11-09 7:00 UTC (permalink / raw) To: linux-kbuild Cc: cocci, Julia Lawall, Nicolas Palix, Gilles Muller, Masahiro Yamada, linux-kernel, Michal Marek The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of "coccicheck failed" error messages. I do not know the coccinelle internals, but I guess --jobs does not work well if spatch is invoked from Make running in parallel. Disable --jobs in this case. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> --- Changes in v2: - Grep '-j' instead of '--jobserver-auth'. '--jobserver-*' is not a stable option flag. Make 4.2 change '--jobserver-fds' into '--jobserver-auth' - Add -q option to grep scripts/coccicheck | 3 +++ 1 file changed, 3 insertions(+) diff --git a/scripts/coccicheck b/scripts/coccicheck index 040a8b1..8bab11e 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then # Take only the last argument, which is the C file to test shift $(( $# - 1 )) OPTIONS="$COCCIINCLUDE $1" + + # --jobs does not work if Make is running in parallel + echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no" else ONLINE=0 if [ "$KBUILD_EXTMOD" = "" ] ; then -- 2.7.4 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck 2017-11-09 7:00 [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck Masahiro Yamada @ 2017-11-10 15:42 ` Julia Lawall 2017-11-11 7:30 ` [Cocci] " Julia Lawall 2017-11-13 15:30 ` Julia Lawall 1 sibling, 1 reply; 11+ messages in thread From: Julia Lawall @ 2017-11-10 15:42 UTC (permalink / raw) To: Masahiro Yamada Cc: linux-kbuild, cocci, Nicolas Palix, Gilles Muller, linux-kernel, Michal Marek On Thu, 9 Nov 2017, Masahiro Yamada wrote: > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of > "coccicheck failed" error messages. The question is where parallelism should be specified. Currently, make coccicheck picks up the number of cores on the machine and passes that to Coccinelle. OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1" On my 80 core machine with hyperthreading, this runs 160 jobs in parallel, while in practice that degrades the performance as compared to 40 or 80 cores. On the other hand, if we use the make command line argument (-j), then we will only get parallelism up to the number of semantic patches. Since some finish quickly, there will be a lot of wasted cycles. The best would be that the user knows what works well for his machine, and specifies it on the command line, and then that value gets propagated to Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in parallel but instead would cause Coccinelle to run one semantic patch on 8 files in parallel. But I don't know if that can be done. julia > > I do not know the coccinelle internals, but I guess --jobs does not > work well if spatch is invoked from Make running in parallel. > Disable --jobs in this case. > > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> > --- > > Changes in v2: > - Grep '-j' instead of '--jobserver-auth'. > '--jobserver-*' is not a stable option flag. > Make 4.2 change '--jobserver-fds' into '--jobserver-auth' > - Add -q option to grep > > scripts/coccicheck | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/scripts/coccicheck b/scripts/coccicheck > index 040a8b1..8bab11e 100755 > --- a/scripts/coccicheck > +++ b/scripts/coccicheck > @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then > # Take only the last argument, which is the C file to test > shift $(( $# - 1 )) > OPTIONS="$COCCIINCLUDE $1" > + > + # --jobs does not work if Make is running in parallel > + echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no" > else > ONLINE=0 > if [ "$KBUILD_EXTMOD" = "" ] ; then > -- > 2.7.4 > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck 2017-11-10 15:42 ` Julia Lawall @ 2017-11-11 7:30 ` Julia Lawall 2017-11-13 16:31 ` Masahiro Yamada 0 siblings, 1 reply; 11+ messages in thread From: Julia Lawall @ 2017-11-11 7:30 UTC (permalink / raw) To: Masahiro Yamada Cc: Michal Marek, linux-kbuild, Nicolas Palix, linux-kernel, cocci On Fri, 10 Nov 2017, Julia Lawall wrote: > > > On Thu, 9 Nov 2017, Masahiro Yamada wrote: > > > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of > > "coccicheck failed" error messages. > > The question is where parallelism should be specified. Currently, make > coccicheck picks up the number of cores on the machine and passes that to > Coccinelle. > > OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1" > > On my 80 core machine with hyperthreading, this runs 160 jobs in parallel, > while in practice that degrades the performance as compared to 40 or 80 > cores. > > On the other hand, if we use the make command line argument (-j), then we > will only get parallelism up to the number of semantic patches. Since > some finish quickly, there will be a lot of wasted cycles. > > The best would be that the user knows what works well for his machine, and > specifies it on the command line, and then that value gets propagated to > Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in > parallel but instead would cause Coccinelle to run one semantic patch on 8 > files in parallel. But I don't know if that can be done. Sorry for these fairly nonsensical comments. make -j is going to consider every file, then parse and run every semantic patch on that file. If the parallelism is pushed down into Coccinelle, each semantic patch will be parsed only once, and then Coccinelle will choose the files for which it is relevant. If indexing is used (idutils, glimpse), then for semantic patches that focus on specific keywords, Coccinelle will efficiently ignore files that are not relevant. I don't think there would be many cases where make -j would win. Perhaps it would be possible to detect its used and abort with an appopriate message? julia > > julia > > > > > I do not know the coccinelle internals, but I guess --jobs does not > > work well if spatch is invoked from Make running in parallel. > > Disable --jobs in this case. > > > > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> > > --- > > > > Changes in v2: > > - Grep '-j' instead of '--jobserver-auth'. > > '--jobserver-*' is not a stable option flag. > > Make 4.2 change '--jobserver-fds' into '--jobserver-auth' > > - Add -q option to grep > > > > scripts/coccicheck | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/scripts/coccicheck b/scripts/coccicheck > > index 040a8b1..8bab11e 100755 > > --- a/scripts/coccicheck > > +++ b/scripts/coccicheck > > @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then > > # Take only the last argument, which is the C file to test > > shift $(( $# - 1 )) > > OPTIONS="$COCCIINCLUDE $1" > > + > > + # --jobs does not work if Make is running in parallel > > + echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no" > > else > > ONLINE=0 > > if [ "$KBUILD_EXTMOD" = "" ] ; then > > -- > > 2.7.4 > > > > > _______________________________________________ > Cocci mailing list > Cocci@systeme.lip6.fr > https://systeme.lip6.fr/mailman/listinfo/cocci > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck 2017-11-11 7:30 ` [Cocci] " Julia Lawall @ 2017-11-13 16:31 ` Masahiro Yamada 2017-11-13 16:44 ` Julia Lawall 0 siblings, 1 reply; 11+ messages in thread From: Masahiro Yamada @ 2017-11-13 16:31 UTC (permalink / raw) To: Julia Lawall Cc: Michal Marek, Linux Kbuild mailing list, Nicolas Palix, Linux Kernel Mailing List, cocci Hi Julia, 2017-11-11 16:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>: > > > On Fri, 10 Nov 2017, Julia Lawall wrote: > >> >> >> On Thu, 9 Nov 2017, Masahiro Yamada wrote: >> >> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of >> > "coccicheck failed" error messages. >> >> The question is where parallelism should be specified. Currently, make >> coccicheck picks up the number of cores on the machine and passes that to >> Coccinelle. >> >> OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1" >> >> On my 80 core machine with hyperthreading, this runs 160 jobs in parallel, >> while in practice that degrades the performance as compared to 40 or 80 >> cores. >> >> On the other hand, if we use the make command line argument (-j), then we >> will only get parallelism up to the number of semantic patches. Since >> some finish quickly, there will be a lot of wasted cycles. >> >> The best would be that the user knows what works well for his machine, and >> specifies it on the command line, and then that value gets propagated to >> Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in >> parallel but instead would cause Coccinelle to run one semantic patch on 8 >> files in parallel. But I don't know if that can be done. > > Sorry for these fairly nonsensical comments. make -j is going to consider > every file, then parse and run every semantic patch on that file. If the > parallelism is pushed down into Coccinelle, each semantic patch will be > parsed only once, and then Coccinelle will choose the files for which it > is relevant. If indexing is used (idutils, glimpse), then for semantic > patches that focus on specific keywords, Coccinelle will efficiently > ignore files that are not relevant. I don't think there would be many > cases where make -j would win. Perhaps it would be possible to detect > its used and abort with an appopriate message? I am afraid you and I are talking different things. For a usual usage of coccicheck, only one thread runs scripts/coccicheck even if -j is passed from the command line. coccicheck provides "J" to specify parallelism. if [ -z "$J" ]; then NPROC=$(getconf _NPROCESSORS_ONLN) else NPROC="$J" fi If you are unhappy with 160 threading, you can give J=40 from the command line. My patch addresses a problem where coccicheck is used as CHECK. The default of CHECK is "sparse", but you can use any checker tool. In CHECK=scripts/coccicheck case, if -j is passed, all tasks run in parallel under control of GNU Make, so scripts/coccicheck is also invoked from multiple threads. Passing --jobs to spatch is not sensible because it checks only one file. > julia > > >> >> julia >> >> > >> > I do not know the coccinelle internals, but I guess --jobs does not >> > work well if spatch is invoked from Make running in parallel. >> > Disable --jobs in this case. >> > >> > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> >> > --- >> > >> > Changes in v2: >> > - Grep '-j' instead of '--jobserver-auth'. >> > '--jobserver-*' is not a stable option flag. >> > Make 4.2 change '--jobserver-fds' into '--jobserver-auth' >> > - Add -q option to grep >> > >> > scripts/coccicheck | 3 +++ >> > 1 file changed, 3 insertions(+) >> > >> > diff --git a/scripts/coccicheck b/scripts/coccicheck >> > index 040a8b1..8bab11e 100755 >> > --- a/scripts/coccicheck >> > +++ b/scripts/coccicheck >> > @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then >> > # Take only the last argument, which is the C file to test >> > shift $(( $# - 1 )) >> > OPTIONS="$COCCIINCLUDE $1" >> > + >> > + # --jobs does not work if Make is running in parallel >> > + echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no" >> > else >> > ONLINE=0 >> > if [ "$KBUILD_EXTMOD" = "" ] ; then >> > -- >> > 2.7.4 >> > >> > >> _______________________________________________ >> Cocci mailing list >> Cocci@systeme.lip6.fr >> https://systeme.lip6.fr/mailman/listinfo/cocci >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Cocci] [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck 2017-11-13 16:31 ` Masahiro Yamada @ 2017-11-13 16:44 ` Julia Lawall 0 siblings, 0 replies; 11+ messages in thread From: Julia Lawall @ 2017-11-13 16:44 UTC (permalink / raw) To: Masahiro Yamada Cc: Michal Marek, Linux Kbuild mailing list, Nicolas Palix, Linux Kernel Mailing List, cocci On Tue, 14 Nov 2017, Masahiro Yamada wrote: > Hi Julia, > > > 2017-11-11 16:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>: > > > > > > On Fri, 10 Nov 2017, Julia Lawall wrote: > > > >> > >> > >> On Thu, 9 Nov 2017, Masahiro Yamada wrote: > >> > >> > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of > >> > "coccicheck failed" error messages. > >> > >> The question is where parallelism should be specified. Currently, make > >> coccicheck picks up the number of cores on the machine and passes that to > >> Coccinelle. > >> > >> OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1" > >> > >> On my 80 core machine with hyperthreading, this runs 160 jobs in parallel, > >> while in practice that degrades the performance as compared to 40 or 80 > >> cores. > >> > >> On the other hand, if we use the make command line argument (-j), then we > >> will only get parallelism up to the number of semantic patches. Since > >> some finish quickly, there will be a lot of wasted cycles. > >> > >> The best would be that the user knows what works well for his machine, and > >> specifies it on the command line, and then that value gets propagated to > >> Coccinelle, eg so that -j8 would cause not 8 semantic patches to run in > >> parallel but instead would cause Coccinelle to run one semantic patch on 8 > >> files in parallel. But I don't know if that can be done. > > > > Sorry for these fairly nonsensical comments. make -j is going to consider > > every file, then parse and run every semantic patch on that file. If the > > parallelism is pushed down into Coccinelle, each semantic patch will be > > parsed only once, and then Coccinelle will choose the files for which it > > is relevant. If indexing is used (idutils, glimpse), then for semantic > > patches that focus on specific keywords, Coccinelle will efficiently > > ignore files that are not relevant. I don't think there would be many > > cases where make -j would win. Perhaps it would be possible to detect > > its used and abort with an appopriate message? > > > I am afraid you and I are talking different things. > > > For a usual usage of coccicheck, only one thread runs scripts/coccicheck > even if -j is passed from the command line. > > coccicheck provides "J" to specify parallelism. > > if [ -z "$J" ]; then > NPROC=$(getconf _NPROCESSORS_ONLN) > else > NPROC="$J" > fi Even if J is not specified, then it still runs with the maximum number of threads: Coccinelle parallelization --------------------------- By default, coccicheck tries to run as parallel as possible. Indeed, J= does set the number of threads speficied. > My patch addresses a problem where coccicheck is used as CHECK. > The default of CHECK is "sparse", but you can use any checker tool. > > In CHECK=scripts/coccicheck case, if -j is passed, all tasks run in parallel > under control of GNU Make, so scripts/coccicheck is also invoked from > multiple threads. > Passing --jobs to spatch is not sensible because it checks only one file. OK. I tried a simple make coccicheck -j4 and indeed it does not seem to be complaining. The number of spatch processes goes over 160 though. julia ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck 2017-11-09 7:00 [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck Masahiro Yamada 2017-11-10 15:42 ` Julia Lawall @ 2017-11-13 15:30 ` Julia Lawall 2017-11-13 16:32 ` Masahiro Yamada 1 sibling, 1 reply; 11+ messages in thread From: Julia Lawall @ 2017-11-13 15:30 UTC (permalink / raw) To: Masahiro Yamada Cc: linux-kbuild, cocci, Nicolas Palix, Gilles Muller, linux-kernel, Michal Marek On Thu, 9 Nov 2017, Masahiro Yamada wrote: > The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of > "coccicheck failed" error messages. > > I do not know the coccinelle internals, but I guess --jobs does not > work well if spatch is invoked from Make running in parallel. > Disable --jobs in this case. Why is this change under: if [ "$C" = "1" -o "$C" = "2" ]; The coccicheck failed messages come also if one runs Coccinelle on the entire kernel. julia > > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> > --- > > Changes in v2: > - Grep '-j' instead of '--jobserver-auth'. > '--jobserver-*' is not a stable option flag. > Make 4.2 change '--jobserver-fds' into '--jobserver-auth' > - Add -q option to grep > > scripts/coccicheck | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/scripts/coccicheck b/scripts/coccicheck > index 040a8b1..8bab11e 100755 > --- a/scripts/coccicheck > +++ b/scripts/coccicheck > @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then > # Take only the last argument, which is the C file to test > shift $(( $# - 1 )) > OPTIONS="$COCCIINCLUDE $1" > + > + # --jobs does not work if Make is running in parallel > + echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no" > else > ONLINE=0 > if [ "$KBUILD_EXTMOD" = "" ] ; then > -- > 2.7.4 > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck 2017-11-13 15:30 ` Julia Lawall @ 2017-11-13 16:32 ` Masahiro Yamada 2017-11-13 16:45 ` Julia Lawall 0 siblings, 1 reply; 11+ messages in thread From: Masahiro Yamada @ 2017-11-13 16:32 UTC (permalink / raw) To: Julia Lawall Cc: Linux Kbuild mailing list, cocci, Nicolas Palix, Gilles Muller, Linux Kernel Mailing List, Michal Marek Hi Julia, 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>: > > > On Thu, 9 Nov 2017, Masahiro Yamada wrote: > >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of >> "coccicheck failed" error messages. >> >> I do not know the coccinelle internals, but I guess --jobs does not >> work well if spatch is invoked from Make running in parallel. >> Disable --jobs in this case. > > Why is this change under: > > if [ "$C" = "1" -o "$C" = "2" ]; > > The coccicheck failed messages come also if one runs Coccinelle on the > entire kernel. As far as I tested, "coccicheck failed" error only happens when ONLINE=1. make -j8 C=1 CHECK=scripts/coccicheck COCCI=scripts/coccinelle/misc/bugon.cocci emits lots of errors. make -j8 coccicheck COCCI=scripts/coccinelle/misc/bugon.cocci is fine. Have you tested it? Do you mean you got a different result from mine? > julia > > >> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> >> --- >> >> Changes in v2: >> - Grep '-j' instead of '--jobserver-auth'. >> '--jobserver-*' is not a stable option flag. >> Make 4.2 change '--jobserver-fds' into '--jobserver-auth' >> - Add -q option to grep >> >> scripts/coccicheck | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/scripts/coccicheck b/scripts/coccicheck >> index 040a8b1..8bab11e 100755 >> --- a/scripts/coccicheck >> +++ b/scripts/coccicheck >> @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then >> # Take only the last argument, which is the C file to test >> shift $(( $# - 1 )) >> OPTIONS="$COCCIINCLUDE $1" >> + >> + # --jobs does not work if Make is running in parallel >> + echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no" >> else >> ONLINE=0 >> if [ "$KBUILD_EXTMOD" = "" ] ; then >> -- >> 2.7.4 >> >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck 2017-11-13 16:32 ` Masahiro Yamada @ 2017-11-13 16:45 ` Julia Lawall 2017-11-14 3:51 ` Masahiro Yamada 0 siblings, 1 reply; 11+ messages in thread From: Julia Lawall @ 2017-11-13 16:45 UTC (permalink / raw) To: Masahiro Yamada Cc: Linux Kbuild mailing list, cocci, Nicolas Palix, Gilles Muller, Linux Kernel Mailing List, Michal Marek On Tue, 14 Nov 2017, Masahiro Yamada wrote: > Hi Julia, > > > 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>: > > > > > > On Thu, 9 Nov 2017, Masahiro Yamada wrote: > > > >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of > >> "coccicheck failed" error messages. > >> > >> I do not know the coccinelle internals, but I guess --jobs does not > >> work well if spatch is invoked from Make running in parallel. > >> Disable --jobs in this case. > > > > Why is this change under: > > > > if [ "$C" = "1" -o "$C" = "2" ]; > > > > The coccicheck failed messages come also if one runs Coccinelle on the > > entire kernel. > > As far as I tested, "coccicheck failed" error only happens > when ONLINE=1. > > > make -j8 C=1 CHECK=scripts/coccicheck COCCI=scripts/coccinelle/misc/bugon.cocci > > emits lots of errors. > > > make -j8 coccicheck COCCI=scripts/coccinelle/misc/bugon.cocci > > is fine. > > > Have you tested it? > Do you mean you got a different result from mine? I agree with your results, with respect to the number of errors. julia > > > > > julia > > > > > >> > >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> > >> --- > >> > >> Changes in v2: > >> - Grep '-j' instead of '--jobserver-auth'. > >> '--jobserver-*' is not a stable option flag. > >> Make 4.2 change '--jobserver-fds' into '--jobserver-auth' > >> - Add -q option to grep > >> > >> scripts/coccicheck | 3 +++ > >> 1 file changed, 3 insertions(+) > >> > >> diff --git a/scripts/coccicheck b/scripts/coccicheck > >> index 040a8b1..8bab11e 100755 > >> --- a/scripts/coccicheck > >> +++ b/scripts/coccicheck > >> @@ -70,6 +70,9 @@ if [ "$C" = "1" -o "$C" = "2" ]; then > >> # Take only the last argument, which is the C file to test > >> shift $(( $# - 1 )) > >> OPTIONS="$COCCIINCLUDE $1" > >> + > >> + # --jobs does not work if Make is running in parallel > >> + echo $MAKEFLAGS | grep -q -E '(^| )-j' && USE_JOBS="no" > >> else > >> ONLINE=0 > >> if [ "$KBUILD_EXTMOD" = "" ] ; then > >> -- > >> 2.7.4 > >> > >> > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- > Best Regards > Masahiro Yamada > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck 2017-11-13 16:45 ` Julia Lawall @ 2017-11-14 3:51 ` Masahiro Yamada 2017-11-14 6:44 ` Julia Lawall 0 siblings, 1 reply; 11+ messages in thread From: Masahiro Yamada @ 2017-11-14 3:51 UTC (permalink / raw) To: Julia Lawall Cc: Linux Kbuild mailing list, cocci, Nicolas Palix, Gilles Muller, Linux Kernel Mailing List, Michal Marek Hi Julia, 2017-11-14 1:45 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>: > > > On Tue, 14 Nov 2017, Masahiro Yamada wrote: > >> Hi Julia, >> >> >> 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>: >> > >> > >> > On Thu, 9 Nov 2017, Masahiro Yamada wrote: >> > >> >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of >> >> "coccicheck failed" error messages. >> >> >> >> I do not know the coccinelle internals, but I guess --jobs does not >> >> work well if spatch is invoked from Make running in parallel. >> >> Disable --jobs in this case. >> > >> > Why is this change under: >> > >> > if [ "$C" = "1" -o "$C" = "2" ]; >> > >> > The coccicheck failed messages come also if one runs Coccinelle on the >> > entire kernel. >> >> As far as I tested, "coccicheck failed" error only happens >> when ONLINE=1. >> >> >> make -j8 C=1 CHECK=scripts/coccicheck COCCI=scripts/coccinelle/misc/bugon.cocci >> >> emits lots of errors. >> >> >> make -j8 coccicheck COCCI=scripts/coccinelle/misc/bugon.cocci >> >> is fine. >> >> >> Have you tested it? >> Do you mean you got a different result from mine? > > I agree with your results, with respect to the number of errors. > > julia > So, what shall we do? If you do not like to fix it (or you can fix coccinelle itself), I can take back this patch. I am not a coccinelle developer, so setting USE_JOBS="no" is the best I can do. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck 2017-11-14 3:51 ` Masahiro Yamada @ 2017-11-14 6:44 ` Julia Lawall 2017-11-14 9:08 ` Masahiro Yamada 0 siblings, 1 reply; 11+ messages in thread From: Julia Lawall @ 2017-11-14 6:44 UTC (permalink / raw) To: Masahiro Yamada Cc: Linux Kbuild mailing list, cocci, Nicolas Palix, Gilles Muller, Linux Kernel Mailing List, Michal Marek On Tue, 14 Nov 2017, Masahiro Yamada wrote: > Hi Julia, > > 2017-11-14 1:45 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>: > > > > > > On Tue, 14 Nov 2017, Masahiro Yamada wrote: > > > >> Hi Julia, > >> > >> > >> 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>: > >> > > >> > > >> > On Thu, 9 Nov 2017, Masahiro Yamada wrote: > >> > > >> >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of > >> >> "coccicheck failed" error messages. > >> >> > >> >> I do not know the coccinelle internals, but I guess --jobs does not > >> >> work well if spatch is invoked from Make running in parallel. > >> >> Disable --jobs in this case. > >> > > >> > Why is this change under: > >> > > >> > if [ "$C" = "1" -o "$C" = "2" ]; > >> > > >> > The coccicheck failed messages come also if one runs Coccinelle on the > >> > entire kernel. > >> > >> As far as I tested, "coccicheck failed" error only happens > >> when ONLINE=1. > >> > >> > >> make -j8 C=1 CHECK=scripts/coccicheck COCCI=scripts/coccinelle/misc/bugon.cocci > >> > >> emits lots of errors. > >> > >> > >> make -j8 coccicheck COCCI=scripts/coccinelle/misc/bugon.cocci > >> > >> is fine. > >> > >> > >> Have you tested it? > >> Do you mean you got a different result from mine? > > > > I agree with your results, with respect to the number of errors. > > > > julia > > > > So, what shall we do? > > If you do not like to fix it (or you can fix coccinelle itself), > I can take back this patch. I'm OK with your fix. I will check and ack it today. > I am not a coccinelle developer, so > setting USE_JOBS="no" is the best I can do. The problem on the Coccinelle side is that it uses a subdirectory with the name of the semantic patch to store standard output and standard error for the different threads. I didn't want to use a name with the pid, so that one could easily find this information while Coccinelle is running. Normally the subdirectory is cleaned up when Coccinelle completes, so there is only one of them at a time. Maybe it is best to just add the pid. There is the risk that these subdirectories will accumulate if Coccinelle crashes in a way such that they don't get cleaned up, but Coccinelle could print a warning if it detects this case, rather than failing. Still I think it is useful to do something on the make coccicheck side, because there is no need for the double layer of parallelism. julia ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck 2017-11-14 6:44 ` Julia Lawall @ 2017-11-14 9:08 ` Masahiro Yamada 0 siblings, 0 replies; 11+ messages in thread From: Masahiro Yamada @ 2017-11-14 9:08 UTC (permalink / raw) To: Julia Lawall Cc: Linux Kbuild mailing list, cocci, Nicolas Palix, Gilles Muller, Linux Kernel Mailing List, Michal Marek Hi Julia, 2017-11-14 15:44 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>: > > > On Tue, 14 Nov 2017, Masahiro Yamada wrote: > >> Hi Julia, >> >> 2017-11-14 1:45 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>: >> > >> > >> > On Tue, 14 Nov 2017, Masahiro Yamada wrote: >> > >> >> Hi Julia, >> >> >> >> >> >> 2017-11-14 0:30 GMT+09:00 Julia Lawall <julia.lawall@lip6.fr>: >> >> > >> >> > >> >> > On Thu, 9 Nov 2017, Masahiro Yamada wrote: >> >> > >> >> >> The command "make -j8 C=1 CHECK=scripts/coccicheck" produces lots of >> >> >> "coccicheck failed" error messages. >> >> >> >> >> >> I do not know the coccinelle internals, but I guess --jobs does not >> >> >> work well if spatch is invoked from Make running in parallel. >> >> >> Disable --jobs in this case. >> >> > >> >> > Why is this change under: >> >> > >> >> > if [ "$C" = "1" -o "$C" = "2" ]; >> >> > >> >> > The coccicheck failed messages come also if one runs Coccinelle on the >> >> > entire kernel. >> >> >> >> As far as I tested, "coccicheck failed" error only happens >> >> when ONLINE=1. >> >> >> >> >> >> make -j8 C=1 CHECK=scripts/coccicheck COCCI=scripts/coccinelle/misc/bugon.cocci >> >> >> >> emits lots of errors. >> >> >> >> >> >> make -j8 coccicheck COCCI=scripts/coccinelle/misc/bugon.cocci >> >> >> >> is fine. >> >> >> >> >> >> Have you tested it? >> >> Do you mean you got a different result from mine? >> > >> > I agree with your results, with respect to the number of errors. >> > >> > julia >> > >> >> So, what shall we do? >> >> If you do not like to fix it (or you can fix coccinelle itself), >> I can take back this patch. > > I'm OK with your fix. I will check and ack it today. > >> I am not a coccinelle developer, so >> setting USE_JOBS="no" is the best I can do. > > The problem on the Coccinelle side is that it uses a subdirectory with the > name of the semantic patch to store standard output and standard error for > the different threads. I didn't want to use a name with the pid, so that > one could easily find this information while Coccinelle is running. > Normally the subdirectory is cleaned up when Coccinelle completes, so > there is only one of them at a time. Maybe it is best to just add the > pid. There is the risk that these subdirectories will accumulate if > Coccinelle crashes in a way such that they don't get cleaned up, but > Coccinelle could print a warning if it detects this case, rather than > failing. > > Still I think it is useful to do something on the make coccicheck side, > because there is no need for the double layer of parallelism. > Thanks a lot for detailed explanation! I brushed up my patch. Could you check v3, please? -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2017-11-14 9:09 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-11-09 7:00 [PATCH v2] coccinelle: fix parallel build with CHECK=scripts/coccicheck Masahiro Yamada 2017-11-10 15:42 ` Julia Lawall 2017-11-11 7:30 ` [Cocci] " Julia Lawall 2017-11-13 16:31 ` Masahiro Yamada 2017-11-13 16:44 ` Julia Lawall 2017-11-13 15:30 ` Julia Lawall 2017-11-13 16:32 ` Masahiro Yamada 2017-11-13 16:45 ` Julia Lawall 2017-11-14 3:51 ` Masahiro Yamada 2017-11-14 6:44 ` Julia Lawall 2017-11-14 9:08 ` Masahiro Yamada
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).