* [PATCH 0/4] scripts/coccicheck: add paramap and indexing options @ 2016-06-10 20:42 Luis R. Rodriguez 2016-06-10 20:42 ` [PATCH 1/4] coccicheck: move spatch binary check up Luis R. Rodriguez ` (4 more replies) 0 siblings, 5 replies; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-10 20:42 UTC (permalink / raw) To: Julia.Lawall, Gilles.Muller, nicolas.palix, mmarek Cc: linux-kernel, cocci, Luis R. Rodriguez coccicheck hasn't been updated for a while. The backports project has been using some features for a while now that we should be able to also take advantage of with coccicheck, the most important one is paramap support. glimpseindex stuff wasn't even building but today I decided to go tackle and fix that, the public open source release is is now working and you can optionally use that. Note that using git performs just as well, glimpseindex just shaves off a bit of time, however if since we can support it now we do it. Luis R. Rodriguez (4): coccicheck: move spatch binary check up coccicheck: enable paramap support scripts: add glimpse.sh for indexing the kernel coccicheck: add indexing enhancement options scripts/coccicheck | 62 +++++++++++++++++++++++++++++++++++++++++++++++------- scripts/glimpse.sh | 12 +++++++++++ 2 files changed, 66 insertions(+), 8 deletions(-) create mode 100755 scripts/glimpse.sh -- 2.8.2 ^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH 1/4] coccicheck: move spatch binary check up 2016-06-10 20:42 [PATCH 0/4] scripts/coccicheck: add paramap and indexing options Luis R. Rodriguez @ 2016-06-10 20:42 ` Luis R. Rodriguez 2016-06-10 20:42 ` [PATCH 2/4] coccicheck: enable paramap support Luis R. Rodriguez ` (3 subsequent siblings) 4 siblings, 0 replies; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-10 20:42 UTC (permalink / raw) To: Julia.Lawall, Gilles.Muller, nicolas.palix, mmarek Cc: linux-kernel, cocci, Luis R. Rodriguez This has no functional changes. This is being done to enable us to later use spatch binary for some flag checking for certain features early on. Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> --- scripts/coccicheck | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index dd85a455b2ba..aa5e78fba270 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -7,6 +7,11 @@ SPATCH="`which ${SPATCH:=spatch}`" +if [ ! -x "$SPATCH" ]; then + echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/' + exit 1 +fi + trap kill_running SIGTERM SIGINT declare -a SPATCH_PID @@ -51,11 +56,6 @@ if [ "$KBUILD_EXTMOD" != "" ] ; then OPTIONS="--patch $srctree $OPTIONS" fi -if [ ! -x "$SPATCH" ]; then - echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/' - exit 1 -fi - if [ "$MODE" = "" ] ; then if [ "$ONLINE" = "0" ] ; then echo 'You have not explicitly specified the mode to use. Using default "report" mode.' -- 2.8.2 ^ permalink raw reply related [flat|nested] 45+ messages in thread
* [PATCH 2/4] coccicheck: enable paramap support 2016-06-10 20:42 [PATCH 0/4] scripts/coccicheck: add paramap and indexing options Luis R. Rodriguez 2016-06-10 20:42 ` [PATCH 1/4] coccicheck: move spatch binary check up Luis R. Rodriguez @ 2016-06-10 20:42 ` Luis R. Rodriguez 2016-06-11 5:45 ` [Cocci] " SF Markus Elfring 2016-06-10 20:42 ` [PATCH 3/4] scripts: add glimpse.sh for indexing the kernel Luis R. Rodriguez ` (2 subsequent siblings) 4 siblings, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-10 20:42 UTC (permalink / raw) To: Julia.Lawall, Gilles.Muller, nicolas.palix, mmarek Cc: linux-kernel, cocci, Luis R. Rodriguez Coccinelle has had paramap support since 1.0.2, this means it supports --jobs, enabling built-in multithreaded functionality, instead of needing one to script it out. Just look for --jobs in the help output to determine if this is supported. Also enable the load balancing to be dynamic, so that if a thread finishes early we keep feeding it. If --jobs is not supported we fallback to the old mechanism. As a small example, prior to this change, on an 8-core system: Before: $ export COCCI=scripts/coccinelle/free/kfree.cocci $ time make coccicheck MODE=report ... real 29m14.912s user 103m1.796s sys 0m4.464s After: real 16m22.435s user 128m30.060s sys 0m2.712s Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> --- scripts/coccicheck | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index aa5e78fba270..eeb5fdc142ca 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then exit 1 fi -trap kill_running SIGTERM SIGINT -declare -a SPATCH_PID +USE_JOBS="no" +$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes" # The verbosity may be set by the environmental parameter V= # as for example with 'make V=1 coccicheck' @@ -82,7 +82,21 @@ if [ "$ONLINE" = "0" ] ; then echo '' fi -run_cmd() { +if [ "$USE_JOBS" = "no" ]; then + trap kill_running SIGTERM SIGINT + declare -a SPATCH_PID +else + OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1" +fi + +run_cmd_paramap() { + if [ $VERBOSE -ne 0 ] ; then + echo "Running ($NPROC in parallel): $@" + fi + $@ +} + +run_cmd_old() { local i if [ $VERBOSE -ne 0 ] ; then echo "Running ($NPROC in parallel): $@" @@ -97,6 +111,14 @@ run_cmd() { wait } +run_cmd() { + if [ "$USE_JOBS" = "yes" ]; then + run_cmd_paramap $@ + else + run_cmd_old $@ + fi +} + kill_running() { for i in $(seq 0 $(( NPROC - 1 )) ); do if [ $VERBOSE -eq 2 ] ; then -- 2.8.2 ^ permalink raw reply related [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 2/4] coccicheck: enable paramap support 2016-06-10 20:42 ` [PATCH 2/4] coccicheck: enable paramap support Luis R. Rodriguez @ 2016-06-11 5:45 ` SF Markus Elfring 2016-06-11 5:55 ` Julia Lawall 0 siblings, 1 reply; 45+ messages in thread From: SF Markus Elfring @ 2016-06-11 5:45 UTC (permalink / raw) To: Luis R. Rodriguez Cc: cocci, linux-kernel, Gilles Muller, Michal Marek, Nicolas Palix > Also enable the load balancing to be dynamic, so that > if a thread finishes early we keep feeding it. Is this functionality influenced by the parameter "chunksize"? Regards, Markus ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 2/4] coccicheck: enable paramap support 2016-06-11 5:45 ` [Cocci] " SF Markus Elfring @ 2016-06-11 5:55 ` Julia Lawall 0 siblings, 0 replies; 45+ messages in thread From: Julia Lawall @ 2016-06-11 5:55 UTC (permalink / raw) To: SF Markus Elfring; +Cc: Luis R. Rodriguez, cocci, Michal Marek, linux-kernel On Sat, 11 Jun 2016, SF Markus Elfring wrote: > > Also enable the load balancing to be dynamic, so that > > if a thread finishes early we keep feeding it. > > Is this functionality influenced by the parameter "chunksize"? Yes, without chunksize the distribution of work to processes is static. julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH 3/4] scripts: add glimpse.sh for indexing the kernel 2016-06-10 20:42 [PATCH 0/4] scripts/coccicheck: add paramap and indexing options Luis R. Rodriguez 2016-06-10 20:42 ` [PATCH 1/4] coccicheck: move spatch binary check up Luis R. Rodriguez 2016-06-10 20:42 ` [PATCH 2/4] coccicheck: enable paramap support Luis R. Rodriguez @ 2016-06-10 20:42 ` Luis R. Rodriguez 2016-06-11 17:09 ` [Cocci] " SF Markus Elfring 2016-06-10 20:42 ` [PATCH 4/4] coccicheck: add indexing enhancement options Luis R. Rodriguez 2016-06-11 5:27 ` [Cocci] [PATCH 0/4] scripts/coccicheck: add paramap and indexing options SF Markus Elfring 4 siblings, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-10 20:42 UTC (permalink / raw) To: Julia.Lawall, Gilles.Muller, nicolas.palix, mmarek Cc: linux-kernel, cocci, Luis R. Rodriguez Glimpse is a tool you can use to index the kernel. The tool was recently open sourced under the ISC license and can be obtained at: https://github.com/mcgrof/glimpse.git Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> --- scripts/glimpse.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100755 scripts/glimpse.sh diff --git a/scripts/glimpse.sh b/scripts/glimpse.sh new file mode 100755 index 000000000000..5fd24e49f31b --- /dev/null +++ b/scripts/glimpse.sh @@ -0,0 +1,12 @@ +#!/bin/bash + +DIR=$(dirname $(readlink -f $0)) +DIR="${DIR}/../" + +GLIMPSEINDEX="`which ${GLIMPSEINDEX:=glimpseindex}`" +if [ ! -x "$GLIMPSEINDEX" ]; then + echo 'glimpseindex can be obtained at https://github.com/mcgrof/glimpse.git' + exit 1 +fi + +find $DIR/* -name "*.[ch]" | $GLIMPSEINDEX -o -H . -F -- 2.8.2 ^ permalink raw reply related [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 3/4] scripts: add glimpse.sh for indexing the kernel 2016-06-10 20:42 ` [PATCH 3/4] scripts: add glimpse.sh for indexing the kernel Luis R. Rodriguez @ 2016-06-11 17:09 ` SF Markus Elfring 0 siblings, 0 replies; 45+ messages in thread From: SF Markus Elfring @ 2016-06-11 17:09 UTC (permalink / raw) To: Luis R. Rodriguez Cc: cocci, linux-kernel, Gilles Muller, Nicolas Palix, Michal Marek > Glimpse is a tool you can use to index the kernel. The tool > was recently open sourced under the ISC license and can be > obtained at: How do you think about to mention the script addition also directly in the commit message? > @@ -0,0 +1,12 @@ > +#!/bin/bash > + > +DIR=$(dirname $(readlink -f $0)) > +DIR="${DIR}/../" Would you like to use the following variable assignment (instead of two before)? +DIR="$(dirname $(readlink -f $0))/../" By the way: How are the chances to achieve further software improvements? https://github.com/gvelez17/glimpse/issues Regards, Markus ^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 20:42 [PATCH 0/4] scripts/coccicheck: add paramap and indexing options Luis R. Rodriguez ` (2 preceding siblings ...) 2016-06-10 20:42 ` [PATCH 3/4] scripts: add glimpse.sh for indexing the kernel Luis R. Rodriguez @ 2016-06-10 20:42 ` Luis R. Rodriguez 2016-06-10 21:02 ` Julia Lawall 2016-06-11 5:55 ` [Cocci] " SF Markus Elfring 2016-06-11 5:27 ` [Cocci] [PATCH 0/4] scripts/coccicheck: add paramap and indexing options SF Markus Elfring 4 siblings, 2 replies; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-10 20:42 UTC (permalink / raw) To: Julia.Lawall, Gilles.Muller, nicolas.palix, mmarek Cc: linux-kernel, cocci, Luis R. Rodriguez Enable indexing optimizations heuristics. Coccinelle has support to make use of its own enhanced "grep" mechanisms instead of using regular grep for searching code 'coccigrep', in practice though this seems to not perform better than regular grep however its expected to help with some use cases so we use that if you have no other indexing options in place available. Since git has its own index, support for using 'git grep' has been added to Coccinelle, that should on average perform better than using the internal cocci grep, and regular grep. Lastly, Coccinelle has had support for glimpseindex for a long while, however the tool was previously closed source, its now open sourced, and provides the best performance, so support that if we can detect you have a glimpse index. These tests have been run on an 8 core system: Before: $ export COCCI=scripts/coccinelle/free/kfree.cocci $ time make coccicheck MODE=report Before this patch with no indexing or anything: real 16m22.435s user 128m30.060s sys 0m2.712s Using coccigrep (after this patch if you have no .git): real 16m27.650s user 128m47.904s sys 0m2.176s If you have .git and therefore use gitgrep: real 16m21.220s user 129m30.940s sys 0m2.060s And if you have a .glimpse_index: real 16m14.794s user 128m42.356s sys 0m1.880s Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> --- scripts/coccicheck | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/scripts/coccicheck b/scripts/coccicheck index eeb5fdc142ca..f31c9a152559 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -5,6 +5,8 @@ # version 1.0.0-rc11. # +DIR=$(dirname $(readlink -f $0)) +DIR="${DIR}/../" SPATCH="`which ${SPATCH:=spatch}`" if [ ! -x "$SPATCH" ]; then @@ -15,6 +17,20 @@ fi USE_JOBS="no" $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes" +# 0. --use-glimpse currently outperforms all. Refer +# to scripts/glimpse.sh for details. +# 1. Second best is --use-gitgrep, this is very comparable to --use-glimpse +# 2. Use --use-coccigrep if no indexing options are available and your +# version of coccinelle supports it +USE_GLIMPSE="no" +$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f $DIR/.glimpse_index ] && USE_GLIMPSE="yes" + +USE_GITGREP="no" +$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && USE_GITGREP="yes" + +USE_COCCIGREP="no" +$SPATCH --help | grep "\-\-use\-coccigrep" > /dev/null && USE_COCCIGREP="yes" + # The verbosity may be set by the environmental parameter V= # as for example with 'make V=1 coccicheck' @@ -89,6 +105,14 @@ else OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1" fi +if [ "$USE_GLIMPSE" = "yes" ]; then + OPTIONS="$OPTIONS --use-glimpse" +elif [ "$USE_GITGREP" = "yes" ]; then + OPTIONS="$OPTIONS --use-gitgrep" +elif [ "$USE_COCCIGREP" = "yes" ]; then + OPTIONS="$OPTIONS --use-coccigrep" +fi + run_cmd_paramap() { if [ $VERBOSE -ne 0 ] ; then echo "Running ($NPROC in parallel): $@" -- 2.8.2 ^ permalink raw reply related [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 20:42 ` [PATCH 4/4] coccicheck: add indexing enhancement options Luis R. Rodriguez @ 2016-06-10 21:02 ` Julia Lawall 2016-06-10 21:18 ` Luis R. Rodriguez 2016-06-11 5:55 ` [Cocci] " SF Markus Elfring 1 sibling, 1 reply; 45+ messages in thread From: Julia Lawall @ 2016-06-10 21:02 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > Enable indexing optimizations heuristics. Coccinelle has > support to make use of its own enhanced "grep" mechanisms > instead of using regular grep for searching code 'coccigrep', > in practice though this seems to not perform better than > regular grep however its expected to help with some use cases > so we use that if you have no other indexing options in place > available. > > Since git has its own index, support for using 'git grep' has been > added to Coccinelle, that should on average perform better than > using the internal cocci grep, and regular grep. Lastly, Coccinelle > has had support for glimpseindex for a long while, however the > tool was previously closed source, its now open sourced, and > provides the best performance, so support that if we can detect > you have a glimpse index. > > These tests have been run on an 8 core system: > > Before: > > $ export COCCI=scripts/coccinelle/free/kfree.cocci > $ time make coccicheck MODE=report > > Before this patch with no indexing or anything: > > real 16m22.435s > user 128m30.060s > sys 0m2.712s > > Using coccigrep (after this patch if you have no .git): > > real 16m27.650s > user 128m47.904s > sys 0m2.176s > > If you have .git and therefore use gitgrep: > > real 16m21.220s > user 129m30.940s > sys 0m2.060s > > And if you have a .glimpse_index: > > real 16m14.794s > user 128m42.356s > sys 0m1.880s I don't see any convincing differences in these times. I believe that Coccinelle's internal grep is always used, even with no option. I'm puzzled why glimpse gives no benefit. julia > > Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> > --- > scripts/coccicheck | 24 ++++++++++++++++++++++++ > 1 file changed, 24 insertions(+) > > diff --git a/scripts/coccicheck b/scripts/coccicheck > index eeb5fdc142ca..f31c9a152559 100755 > --- a/scripts/coccicheck > +++ b/scripts/coccicheck > @@ -5,6 +5,8 @@ > # version 1.0.0-rc11. > # > > +DIR=$(dirname $(readlink -f $0)) > +DIR="${DIR}/../" > SPATCH="`which ${SPATCH:=spatch}`" > > if [ ! -x "$SPATCH" ]; then > @@ -15,6 +17,20 @@ fi > USE_JOBS="no" > $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes" > > +# 0. --use-glimpse currently outperforms all. Refer > +# to scripts/glimpse.sh for details. > +# 1. Second best is --use-gitgrep, this is very comparable to --use-glimpse > +# 2. Use --use-coccigrep if no indexing options are available and your > +# version of coccinelle supports it > +USE_GLIMPSE="no" > +$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f $DIR/.glimpse_index ] && USE_GLIMPSE="yes" > + > +USE_GITGREP="no" > +$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && USE_GITGREP="yes" > + > +USE_COCCIGREP="no" > +$SPATCH --help | grep "\-\-use\-coccigrep" > /dev/null && USE_COCCIGREP="yes" > + > # The verbosity may be set by the environmental parameter V= > # as for example with 'make V=1 coccicheck' > > @@ -89,6 +105,14 @@ else > OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1" > fi > > +if [ "$USE_GLIMPSE" = "yes" ]; then > + OPTIONS="$OPTIONS --use-glimpse" > +elif [ "$USE_GITGREP" = "yes" ]; then > + OPTIONS="$OPTIONS --use-gitgrep" > +elif [ "$USE_COCCIGREP" = "yes" ]; then > + OPTIONS="$OPTIONS --use-coccigrep" > +fi > + > run_cmd_paramap() { > if [ $VERBOSE -ne 0 ] ; then > echo "Running ($NPROC in parallel): $@" > -- > 2.8.2 > > ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 21:02 ` Julia Lawall @ 2016-06-10 21:18 ` Luis R. Rodriguez 2016-06-10 21:21 ` Julia Lawall 0 siblings, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-10 21:18 UTC (permalink / raw) To: Julia Lawall Cc: Luis R. Rodriguez, Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote: > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > Enable indexing optimizations heuristics. Coccinelle has > > support to make use of its own enhanced "grep" mechanisms > > instead of using regular grep for searching code 'coccigrep', > > in practice though this seems to not perform better than > > regular grep however its expected to help with some use cases > > so we use that if you have no other indexing options in place > > available. > > > > Since git has its own index, support for using 'git grep' has been > > added to Coccinelle, that should on average perform better than > > using the internal cocci grep, and regular grep. Lastly, Coccinelle > > has had support for glimpseindex for a long while, however the > > tool was previously closed source, its now open sourced, and > > provides the best performance, so support that if we can detect > > you have a glimpse index. > > > > These tests have been run on an 8 core system: > > > > Before: > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci > > $ time make coccicheck MODE=report > > > > Before this patch with no indexing or anything: > > > > real 16m22.435s > > user 128m30.060s > > sys 0m2.712s > > > > Using coccigrep (after this patch if you have no .git): > > > > real 16m27.650s > > user 128m47.904s > > sys 0m2.176s > > > > If you have .git and therefore use gitgrep: > > > > real 16m21.220s > > user 129m30.940s > > sys 0m2.060s > > > > And if you have a .glimpse_index: > > > > real 16m14.794s > > user 128m42.356s > > sys 0m1.880s > > I don't see any convincing differences in these times. > > I believe that Coccinelle's internal grep is always used, even with no > option. Ah that would explain it. This uses coccinelle 1.0.5, is the default there to use --use-coccigrep if no other index is specified ? > I'm puzzled why glimpse gives no benefit. Well, slightly better. Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 21:18 ` Luis R. Rodriguez @ 2016-06-10 21:21 ` Julia Lawall 2016-06-10 21:43 ` [Cocci] " Wolfram Sang 2016-06-13 19:35 ` Luis R. Rodriguez 0 siblings, 2 replies; 45+ messages in thread From: Julia Lawall @ 2016-06-10 21:21 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote: > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > Enable indexing optimizations heuristics. Coccinelle has > > > support to make use of its own enhanced "grep" mechanisms > > > instead of using regular grep for searching code 'coccigrep', > > > in practice though this seems to not perform better than > > > regular grep however its expected to help with some use cases > > > so we use that if you have no other indexing options in place > > > available. > > > > > > Since git has its own index, support for using 'git grep' has been > > > added to Coccinelle, that should on average perform better than > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle > > > has had support for glimpseindex for a long while, however the > > > tool was previously closed source, its now open sourced, and > > > provides the best performance, so support that if we can detect > > > you have a glimpse index. > > > > > > These tests have been run on an 8 core system: > > > > > > Before: > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci > > > $ time make coccicheck MODE=report > > > > > > Before this patch with no indexing or anything: > > > > > > real 16m22.435s > > > user 128m30.060s > > > sys 0m2.712s > > > > > > Using coccigrep (after this patch if you have no .git): > > > > > > real 16m27.650s > > > user 128m47.904s > > > sys 0m2.176s > > > > > > If you have .git and therefore use gitgrep: > > > > > > real 16m21.220s > > > user 129m30.940s > > > sys 0m2.060s > > > > > > And if you have a .glimpse_index: > > > > > > real 16m14.794s > > > user 128m42.356s > > > sys 0m1.880s > > > > I don't see any convincing differences in these times. > > > > I believe that Coccinelle's internal grep is always used, even with no > > option. > > Ah that would explain it. This uses coccinelle 1.0.5, is the default > there to use --use-coccigrep if no other index is specified ? It has been the default for a long time. > > I'm puzzled why glimpse gives no benefit. > > Well, slightly better. No, it should be much better. You would have to look at the standard error to see if you are getting any benefit. There should be very few occurrences of Skipping if you are really using glimpse. In any case, if you asked for glimpse and it was not able to provide it, there should be warning messages at the top of stderr. julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 21:21 ` Julia Lawall @ 2016-06-10 21:43 ` Wolfram Sang 2016-06-10 21:49 ` Luis R. Rodriguez 2016-06-13 19:35 ` Luis R. Rodriguez 1 sibling, 1 reply; 45+ messages in thread From: Wolfram Sang @ 2016-06-10 21:43 UTC (permalink / raw) To: Julia Lawall; +Cc: Luis R. Rodriguez, cocci, mmarek, linux-kernel [-- Attachment #1: Type: text/plain, Size: 278 bytes --] > > Well, slightly better. > > No, it should be much better. You would have to look at the standard I use id-utils regularly and it is indeed at least a magnitude better. The indexing often pays off already with the first coccinelle run for me. Highly recommended. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 21:43 ` [Cocci] " Wolfram Sang @ 2016-06-10 21:49 ` Luis R. Rodriguez 2016-06-10 21:51 ` Wolfram Sang 2016-06-11 5:17 ` Julia Lawall 0 siblings, 2 replies; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-10 21:49 UTC (permalink / raw) To: Wolfram Sang; +Cc: Julia Lawall, Luis R. Rodriguez, cocci, mmarek, linux-kernel On Fri, Jun 10, 2016 at 11:43:57PM +0200, Wolfram Sang wrote: > > > Well, slightly better. > > > > No, it should be much better. You would have to look at the standard > > I use id-utils regularly and it is indeed at least a magnitude better. > The indexing often pays off already with the first coccinelle run for > me. Highly recommended. AFAICT coccinelle does not have integration support for id-utils though. Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 21:49 ` Luis R. Rodriguez @ 2016-06-10 21:51 ` Wolfram Sang 2016-06-10 22:08 ` Luis R. Rodriguez 2016-06-11 5:24 ` Julia Lawall 2016-06-11 5:17 ` Julia Lawall 1 sibling, 2 replies; 45+ messages in thread From: Wolfram Sang @ 2016-06-10 21:51 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Julia Lawall, cocci, mmarek, linux-kernel [-- Attachment #1: Type: text/plain, Size: 166 bytes --] > AFAICT coccinelle does not have integration support for id-utils though. I used it just today ;) -- "--use-idutils ./ID" ID was generated with simple 'mkid -s'. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 21:51 ` Wolfram Sang @ 2016-06-10 22:08 ` Luis R. Rodriguez 2016-06-10 22:25 ` Luis R. Rodriguez 2016-06-11 5:18 ` Julia Lawall 2016-06-11 5:24 ` Julia Lawall 1 sibling, 2 replies; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-10 22:08 UTC (permalink / raw) To: Wolfram Sang; +Cc: Luis R. Rodriguez, Julia Lawall, cocci, mmarek, linux-kernel On Fri, Jun 10, 2016 at 11:51:26PM +0200, Wolfram Sang wrote: > > AFAICT coccinelle does not have integration support for id-utils though. > > I used it just today ;) -- "--use-idutils ./ID" > > ID was generated with simple 'mkid -s'. > Sweet, testing that now. Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 22:08 ` Luis R. Rodriguez @ 2016-06-10 22:25 ` Luis R. Rodriguez 2016-06-11 5:46 ` Wolfram Sang 2016-06-11 5:18 ` Julia Lawall 1 sibling, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-10 22:25 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Wolfram Sang, Julia Lawall, cocci, mmarek, linux-kernel On Sat, Jun 11, 2016 at 12:08:32AM +0200, Luis R. Rodriguez wrote: > On Fri, Jun 10, 2016 at 11:51:26PM +0200, Wolfram Sang wrote: > > > AFAICT coccinelle does not have integration support for id-utils though. > > > > I used it just today ;) -- "--use-idutils ./ID" > > > > ID was generated with simple 'mkid -s'. > > > > Sweet, testing that now. Boooyah :D real 16m11.692s user 127m50.388s sys 0m2.168s Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 22:25 ` Luis R. Rodriguez @ 2016-06-11 5:46 ` Wolfram Sang 2016-06-11 5:54 ` Julia Lawall 0 siblings, 1 reply; 45+ messages in thread From: Wolfram Sang @ 2016-06-11 5:46 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Julia Lawall, cocci, mmarek, linux-kernel [-- Attachment #1: Type: text/plain, Size: 110 bytes --] > real 16m11.692s > user 127m50.388s > sys 0m2.168s That's better but not a magnitude, I wonder. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-11 5:46 ` Wolfram Sang @ 2016-06-11 5:54 ` Julia Lawall 2016-06-11 6:09 ` Wolfram Sang 2016-06-13 18:37 ` Luis R. Rodriguez 0 siblings, 2 replies; 45+ messages in thread From: Julia Lawall @ 2016-06-11 5:54 UTC (permalink / raw) To: Wolfram Sang; +Cc: Luis R. Rodriguez, cocci, mmarek, linux-kernel On Sat, 11 Jun 2016, Wolfram Sang wrote: > > > real 16m11.692s > > user 127m50.388s > > sys 0m2.168s > > That's better but not a magnitude, I wonder. I think that it is because the filtering that Coccinelle does already works pretty well, and there are quite a lot of files (7514) that contain kfree. julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-11 5:54 ` Julia Lawall @ 2016-06-11 6:09 ` Wolfram Sang 2016-06-13 18:37 ` Luis R. Rodriguez 1 sibling, 0 replies; 45+ messages in thread From: Wolfram Sang @ 2016-06-11 6:09 UTC (permalink / raw) To: Julia Lawall; +Cc: Luis R. Rodriguez, cocci, mmarek, linux-kernel [-- Attachment #1: Type: text/plain, Size: 141 bytes --] > works pretty well, and there are quite a lot of files (7514) that contain > kfree. Ah, kfree. That explains, I missed that info. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-11 5:54 ` Julia Lawall 2016-06-11 6:09 ` Wolfram Sang @ 2016-06-13 18:37 ` Luis R. Rodriguez 2016-06-13 18:55 ` Wolfram Sang 1 sibling, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-13 18:37 UTC (permalink / raw) To: Julia Lawall; +Cc: Wolfram Sang, Luis R. Rodriguez, cocci, mmarek, linux-kernel On Sat, Jun 11, 2016 at 07:54:39AM +0200, Julia Lawall wrote: > > > On Sat, 11 Jun 2016, Wolfram Sang wrote: > > > > > > real 16m11.692s > > > user 127m50.388s > > > sys 0m2.168s > > > > That's better but not a magnitude, I wonder. > > I think that it is because the filtering that Coccinelle does already > works pretty well, and there are quite a lot of files (7514) that contain > kfree. Is there another scripts/coccinelle/ file I can use to test against to demo against glimpse/idutils/gitgrep best? Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-13 18:37 ` Luis R. Rodriguez @ 2016-06-13 18:55 ` Wolfram Sang 2016-06-13 19:48 ` Julia Lawall 0 siblings, 1 reply; 45+ messages in thread From: Wolfram Sang @ 2016-06-13 18:55 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Julia Lawall, cocci, mmarek, linux-kernel [-- Attachment #1: Type: text/plain, Size: 257 bytes --] > Is there another scripts/coccinelle/ file I can use to test against to demo > against glimpse/idutils/gitgrep best? I'd think this one may be a candidate: scripts/coccinelle/misc/irqf_oneshot.cocci Not too many, but quite some matches over the tree. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-13 18:55 ` Wolfram Sang @ 2016-06-13 19:48 ` Julia Lawall 2016-06-13 21:22 ` Luis R. Rodriguez 0 siblings, 1 reply; 45+ messages in thread From: Julia Lawall @ 2016-06-13 19:48 UTC (permalink / raw) To: Wolfram Sang; +Cc: Luis R. Rodriguez, cocci, mmarek, linux-kernel On Mon, 13 Jun 2016, Wolfram Sang wrote: > > > Is there another scripts/coccinelle/ file I can use to test against to demo > > against glimpse/idutils/gitgrep best? > > I'd think this one may be a candidate: > > scripts/coccinelle/misc/irqf_oneshot.cocci > > Not too many, but quite some matches over the tree. Seems like a reasonable choice. julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-13 19:48 ` Julia Lawall @ 2016-06-13 21:22 ` Luis R. Rodriguez 2016-06-14 5:08 ` Julia Lawall 0 siblings, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-13 21:22 UTC (permalink / raw) To: Julia Lawall; +Cc: Wolfram Sang, Luis R. Rodriguez, cocci, mmarek, linux-kernel On Mon, Jun 13, 2016 at 09:48:30PM +0200, Julia Lawall wrote: > > > On Mon, 13 Jun 2016, Wolfram Sang wrote: > > > > > > Is there another scripts/coccinelle/ file I can use to test against to demo > > > against glimpse/idutils/gitgrep best? > > > > I'd think this one may be a candidate: > > > > scripts/coccinelle/misc/irqf_oneshot.cocci > > > > Not too many, but quite some matches over the tree. > > Seems like a reasonable choice. With this one on a 32-core system, I get: glimpse: real 0m6.549s user 0m49.136s sys 0m3.076s idutils: real 0m6.749s user 0m51.936s sys 0m3.876s gitgrep: real 0m6.805s user 0m51.572s sys 0m4.432s coccigrep: real 0m16.369s user 0m58.712s sys 0m5.064s I redirected stderr to stdout, and verified glimpse output has: glimpse request = request_threaded_irq Does this match expectations? Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-13 21:22 ` Luis R. Rodriguez @ 2016-06-14 5:08 ` Julia Lawall 0 siblings, 0 replies; 45+ messages in thread From: Julia Lawall @ 2016-06-14 5:08 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Wolfram Sang, cocci, mmarek, linux-kernel On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > On Mon, Jun 13, 2016 at 09:48:30PM +0200, Julia Lawall wrote: > > > > > > On Mon, 13 Jun 2016, Wolfram Sang wrote: > > > > > > > > > Is there another scripts/coccinelle/ file I can use to test against to demo > > > > against glimpse/idutils/gitgrep best? > > > > > > I'd think this one may be a candidate: > > > > > > scripts/coccinelle/misc/irqf_oneshot.cocci > > > > > > Not too many, but quite some matches over the tree. > > > > Seems like a reasonable choice. > > With this one on a 32-core system, I get: > > glimpse: > real 0m6.549s > user 0m49.136s > sys 0m3.076s > > idutils: > real 0m6.749s > user 0m51.936s > sys 0m3.876s > > gitgrep: > real 0m6.805s > user 0m51.572s > sys 0m4.432s > > coccigrep: > real 0m16.369s > user 0m58.712s > sys 0m5.064s > > I redirected stderr to stdout, and verified glimpse output has: > > glimpse request = request_threaded_irq > > Does this match expectations? Yes. I'm not sure that gitgrep would work as well when there are multiple keywords. It may descend to coccigrep. julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 22:08 ` Luis R. Rodriguez 2016-06-10 22:25 ` Luis R. Rodriguez @ 2016-06-11 5:18 ` Julia Lawall 2016-06-11 5:58 ` Wolfram Sang 1 sibling, 1 reply; 45+ messages in thread From: Julia Lawall @ 2016-06-11 5:18 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Wolfram Sang, cocci, mmarek, linux-kernel On Sat, 11 Jun 2016, Luis R. Rodriguez wrote: > On Fri, Jun 10, 2016 at 11:51:26PM +0200, Wolfram Sang wrote: > > > AFAICT coccinelle does not have integration support for id-utils though. > > > > I used it just today ;) -- "--use-idutils ./ID" > > > > ID was generated with simple 'mkid -s'. > > > > Sweet, testing that now. It's not as efficient as glimpse because the query language is simpler. So more filtering has to be done at the ocaml level. But it's probably fine in most cases. julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-11 5:18 ` Julia Lawall @ 2016-06-11 5:58 ` Wolfram Sang 2016-06-11 6:05 ` Julia Lawall 0 siblings, 1 reply; 45+ messages in thread From: Wolfram Sang @ 2016-06-11 5:58 UTC (permalink / raw) To: Julia Lawall; +Cc: Luis R. Rodriguez, cocci, mmarek, linux-kernel [-- Attachment #1: Type: text/plain, Size: 553 bytes --] > It's not as efficient as glimpse because the query language is simpler. Interesting, what is missing compared to glimpse? > So more filtering has to be done at the ocaml level. But it's probably > fine in most cases. For me, it has two advantages over glimpse: a) it is in the debian package repository b) the same database can be used with the code browser 'seascope' which can do nice things by feeding ctags on the fly with data from id-utils. Mileages vary, of course, just wanted to mention it to give pointers. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 819 bytes --] ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-11 5:58 ` Wolfram Sang @ 2016-06-11 6:05 ` Julia Lawall 0 siblings, 0 replies; 45+ messages in thread From: Julia Lawall @ 2016-06-11 6:05 UTC (permalink / raw) To: Wolfram Sang; +Cc: Luis R. Rodriguez, cocci, mmarek, linux-kernel On Sat, 11 Jun 2016, Wolfram Sang wrote: > > > It's not as efficient as glimpse because the query language is simpler. > > Interesting, what is missing compared to glimpse? Glimpse allows queries that are arbitrary formulas, up to a limited level of complexity, involving both && and ||. For idutils, Coccinelle runs lid on each of the tokens in the formula, and then does unions and intersections on the result. julia > > So more filtering has to be done at the ocaml level. But it's probably > > fine in most cases. > > For me, it has two advantages over glimpse: > > a) it is in the debian package repository > b) the same database can be used with the code browser 'seascope' > which can do nice things by feeding ctags on the fly with data > from id-utils. > > Mileages vary, of course, just wanted to mention it to give pointers. > > ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 21:51 ` Wolfram Sang 2016-06-10 22:08 ` Luis R. Rodriguez @ 2016-06-11 5:24 ` Julia Lawall 2016-06-13 18:39 ` Luis R. Rodriguez 1 sibling, 1 reply; 45+ messages in thread From: Julia Lawall @ 2016-06-11 5:24 UTC (permalink / raw) To: Wolfram Sang; +Cc: Luis R. Rodriguez, cocci, mmarek, linux-kernel On Fri, 10 Jun 2016, Wolfram Sang wrote: > > AFAICT coccinelle does not have integration support for id-utils though. > > I used it just today ;) -- "--use-idutils ./ID" > > ID was generated with simple 'mkid -s'. Coccinelle includes a script scripts/idutils_index.sh This does mkid -i C --output .id-utils.index * julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-11 5:24 ` Julia Lawall @ 2016-06-13 18:39 ` Luis R. Rodriguez 0 siblings, 0 replies; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-13 18:39 UTC (permalink / raw) To: Julia Lawall; +Cc: Wolfram Sang, Luis R. Rodriguez, cocci, mmarek, linux-kernel On Sat, Jun 11, 2016 at 07:24:51AM +0200, Julia Lawall wrote: > > > On Fri, 10 Jun 2016, Wolfram Sang wrote: > > > > AFAICT coccinelle does not have integration support for id-utils though. > > > > I used it just today ;) -- "--use-idutils ./ID" > > > > ID was generated with simple 'mkid -s'. > > Coccinelle includes a script scripts/idutils_index.sh > > This does mkid -i C --output .id-utils.index * I'll add support for detecting both. An issue of course is if any of these indexes grows stale. So I'll advise against these and recommend gitgrep unless the user has a hook to update index regularly. Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 21:49 ` Luis R. Rodriguez 2016-06-10 21:51 ` Wolfram Sang @ 2016-06-11 5:17 ` Julia Lawall 1 sibling, 0 replies; 45+ messages in thread From: Julia Lawall @ 2016-06-11 5:17 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: Wolfram Sang, cocci, mmarek, linux-kernel On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > On Fri, Jun 10, 2016 at 11:43:57PM +0200, Wolfram Sang wrote: > > > > Well, slightly better. > > > > > > No, it should be much better. You would have to look at the standard > > > > I use id-utils regularly and it is indeed at least a magnitude better. > > The indexing often pays off already with the first coccinelle run for > > me. Highly recommended. > > AFAICT coccinelle does not have integration support for id-utils though. Yes it does, for years. julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 21:21 ` Julia Lawall 2016-06-10 21:43 ` [Cocci] " Wolfram Sang @ 2016-06-13 19:35 ` Luis R. Rodriguez 2016-06-13 19:50 ` Julia Lawall 1 sibling, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-13 19:35 UTC (permalink / raw) To: Julia Lawall Cc: Luis R. Rodriguez, Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote: > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote: > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > Enable indexing optimizations heuristics. Coccinelle has > > > > support to make use of its own enhanced "grep" mechanisms > > > > instead of using regular grep for searching code 'coccigrep', > > > > in practice though this seems to not perform better than > > > > regular grep however its expected to help with some use cases > > > > so we use that if you have no other indexing options in place > > > > available. > > > > > > > > Since git has its own index, support for using 'git grep' has been > > > > added to Coccinelle, that should on average perform better than > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle > > > > has had support for glimpseindex for a long while, however the > > > > tool was previously closed source, its now open sourced, and > > > > provides the best performance, so support that if we can detect > > > > you have a glimpse index. > > > > > > > > These tests have been run on an 8 core system: > > > > > > > > Before: > > > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci > > > > $ time make coccicheck MODE=report > > > > > > > > Before this patch with no indexing or anything: > > > > > > > > real 16m22.435s > > > > user 128m30.060s > > > > sys 0m2.712s > > > > > > > > Using coccigrep (after this patch if you have no .git): > > > > > > > > real 16m27.650s > > > > user 128m47.904s > > > > sys 0m2.176s > > > > > > > > If you have .git and therefore use gitgrep: > > > > > > > > real 16m21.220s > > > > user 129m30.940s > > > > sys 0m2.060s > > > > > > > > And if you have a .glimpse_index: > > > > > > > > real 16m14.794s > > > > user 128m42.356s > > > > sys 0m1.880s > > > > > > I don't see any convincing differences in these times. > > > > > > I believe that Coccinelle's internal grep is always used, even with no > > > option. > > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default > > there to use --use-coccigrep if no other index is specified ? > > It has been the default for a long time. > > > > I'm puzzled why glimpse gives no benefit. > > > > Well, slightly better. > > No, it should be much better. You would have to look at the standard > error to see if you are getting any benefit. There should be very few > occurrences of Skipping if you are really using glimpse. In any case, if > you asked for glimpse and it was not able to provide it, there should be > warning messages at the top of stderr. I'll redirect stderr to stdout by default when parmap support is used then. Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-13 19:35 ` Luis R. Rodriguez @ 2016-06-13 19:50 ` Julia Lawall 2016-06-13 21:28 ` Luis R. Rodriguez 0 siblings, 1 reply; 45+ messages in thread From: Julia Lawall @ 2016-06-13 19:50 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote: > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote: > > > > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > Enable indexing optimizations heuristics. Coccinelle has > > > > > support to make use of its own enhanced "grep" mechanisms > > > > > instead of using regular grep for searching code 'coccigrep', > > > > > in practice though this seems to not perform better than > > > > > regular grep however its expected to help with some use cases > > > > > so we use that if you have no other indexing options in place > > > > > available. > > > > > > > > > > Since git has its own index, support for using 'git grep' has been > > > > > added to Coccinelle, that should on average perform better than > > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle > > > > > has had support for glimpseindex for a long while, however the > > > > > tool was previously closed source, its now open sourced, and > > > > > provides the best performance, so support that if we can detect > > > > > you have a glimpse index. > > > > > > > > > > These tests have been run on an 8 core system: > > > > > > > > > > Before: > > > > > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci > > > > > $ time make coccicheck MODE=report > > > > > > > > > > Before this patch with no indexing or anything: > > > > > > > > > > real 16m22.435s > > > > > user 128m30.060s > > > > > sys 0m2.712s > > > > > > > > > > Using coccigrep (after this patch if you have no .git): > > > > > > > > > > real 16m27.650s > > > > > user 128m47.904s > > > > > sys 0m2.176s > > > > > > > > > > If you have .git and therefore use gitgrep: > > > > > > > > > > real 16m21.220s > > > > > user 129m30.940s > > > > > sys 0m2.060s > > > > > > > > > > And if you have a .glimpse_index: > > > > > > > > > > real 16m14.794s > > > > > user 128m42.356s > > > > > sys 0m1.880s > > > > > > > > I don't see any convincing differences in these times. > > > > > > > > I believe that Coccinelle's internal grep is always used, even with no > > > > option. > > > > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default > > > there to use --use-coccigrep if no other index is specified ? > > > > It has been the default for a long time. > > > > > > I'm puzzled why glimpse gives no benefit. > > > > > > Well, slightly better. > > > > No, it should be much better. You would have to look at the standard > > error to see if you are getting any benefit. There should be very few > > occurrences of Skipping if you are really using glimpse. In any case, if > > you asked for glimpse and it was not able to provide it, there should be > > warning messages at the top of stderr. > > I'll redirect stderr to stdout by default when parmap support is used then. Usually I put them in different files. julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-13 19:50 ` Julia Lawall @ 2016-06-13 21:28 ` Luis R. Rodriguez 2016-06-14 5:22 ` Julia Lawall 0 siblings, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-13 21:28 UTC (permalink / raw) To: Julia Lawall Cc: Luis R. Rodriguez, Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote: > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > Enable indexing optimizations heuristics. Coccinelle has > > > > > > support to make use of its own enhanced "grep" mechanisms > > > > > > instead of using regular grep for searching code 'coccigrep', > > > > > > in practice though this seems to not perform better than > > > > > > regular grep however its expected to help with some use cases > > > > > > so we use that if you have no other indexing options in place > > > > > > available. > > > > > > > > > > > > Since git has its own index, support for using 'git grep' has been > > > > > > added to Coccinelle, that should on average perform better than > > > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle > > > > > > has had support for glimpseindex for a long while, however the > > > > > > tool was previously closed source, its now open sourced, and > > > > > > provides the best performance, so support that if we can detect > > > > > > you have a glimpse index. > > > > > > > > > > > > These tests have been run on an 8 core system: > > > > > > > > > > > > Before: > > > > > > > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci > > > > > > $ time make coccicheck MODE=report > > > > > > > > > > > > Before this patch with no indexing or anything: > > > > > > > > > > > > real 16m22.435s > > > > > > user 128m30.060s > > > > > > sys 0m2.712s > > > > > > > > > > > > Using coccigrep (after this patch if you have no .git): > > > > > > > > > > > > real 16m27.650s > > > > > > user 128m47.904s > > > > > > sys 0m2.176s > > > > > > > > > > > > If you have .git and therefore use gitgrep: > > > > > > > > > > > > real 16m21.220s > > > > > > user 129m30.940s > > > > > > sys 0m2.060s > > > > > > > > > > > > And if you have a .glimpse_index: > > > > > > > > > > > > real 16m14.794s > > > > > > user 128m42.356s > > > > > > sys 0m1.880s > > > > > > > > > > I don't see any convincing differences in these times. > > > > > > > > > > I believe that Coccinelle's internal grep is always used, even with no > > > > > option. > > > > > > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default > > > > there to use --use-coccigrep if no other index is specified ? > > > > > > It has been the default for a long time. > > > > > > > > I'm puzzled why glimpse gives no benefit. > > > > > > > > Well, slightly better. > > > > > > No, it should be much better. You would have to look at the standard > > > error to see if you are getting any benefit. There should be very few > > > occurrences of Skipping if you are really using glimpse. In any case, if > > > you asked for glimpse and it was not able to provide it, there should be > > > warning messages at the top of stderr. > > > > I'll redirect stderr to stdout by default when parmap support is used then. > > Usually I put them in different files. We can do that as well but I would only want to deal with parmap support case. Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of the shell script? Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-13 21:28 ` Luis R. Rodriguez @ 2016-06-14 5:22 ` Julia Lawall 2016-06-14 19:27 ` Luis R. Rodriguez 0 siblings, 1 reply; 45+ messages in thread From: Julia Lawall @ 2016-06-14 5:22 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote: > > > > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > > Enable indexing optimizations heuristics. Coccinelle has > > > > > > > support to make use of its own enhanced "grep" mechanisms > > > > > > > instead of using regular grep for searching code 'coccigrep', > > > > > > > in practice though this seems to not perform better than > > > > > > > regular grep however its expected to help with some use cases > > > > > > > so we use that if you have no other indexing options in place > > > > > > > available. > > > > > > > > > > > > > > Since git has its own index, support for using 'git grep' has been > > > > > > > added to Coccinelle, that should on average perform better than > > > > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle > > > > > > > has had support for glimpseindex for a long while, however the > > > > > > > tool was previously closed source, its now open sourced, and > > > > > > > provides the best performance, so support that if we can detect > > > > > > > you have a glimpse index. > > > > > > > > > > > > > > These tests have been run on an 8 core system: > > > > > > > > > > > > > > Before: > > > > > > > > > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci > > > > > > > $ time make coccicheck MODE=report > > > > > > > > > > > > > > Before this patch with no indexing or anything: > > > > > > > > > > > > > > real 16m22.435s > > > > > > > user 128m30.060s > > > > > > > sys 0m2.712s > > > > > > > > > > > > > > Using coccigrep (after this patch if you have no .git): > > > > > > > > > > > > > > real 16m27.650s > > > > > > > user 128m47.904s > > > > > > > sys 0m2.176s > > > > > > > > > > > > > > If you have .git and therefore use gitgrep: > > > > > > > > > > > > > > real 16m21.220s > > > > > > > user 129m30.940s > > > > > > > sys 0m2.060s > > > > > > > > > > > > > > And if you have a .glimpse_index: > > > > > > > > > > > > > > real 16m14.794s > > > > > > > user 128m42.356s > > > > > > > sys 0m1.880s > > > > > > > > > > > > I don't see any convincing differences in these times. > > > > > > > > > > > > I believe that Coccinelle's internal grep is always used, even with no > > > > > > option. > > > > > > > > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default > > > > > there to use --use-coccigrep if no other index is specified ? > > > > > > > > It has been the default for a long time. > > > > > > > > > > I'm puzzled why glimpse gives no benefit. > > > > > > > > > > Well, slightly better. > > > > > > > > No, it should be much better. You would have to look at the standard > > > > error to see if you are getting any benefit. There should be very few > > > > occurrences of Skipping if you are really using glimpse. In any case, if > > > > you asked for glimpse and it was not able to provide it, there should be > > > > warning messages at the top of stderr. > > > > > > I'll redirect stderr to stdout by default when parmap support is used then. > > > > Usually I put them in different files. > > We can do that as well but I would only want to deal with parmap support > case. Any preference? How about .coccicheck.stderr.$PID where PID would > be the PID of the shell script? I don't understand the connection with parmap. Originally our use of parmap made output files based on pids. Maybe this is the default for parmap. I found this completely unusable. I guess one could look at the dates to see which file is the most recent one, but it seems tedious. If you are putting the standard output in x.out, then put the standard error in x.err. julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-14 5:22 ` Julia Lawall @ 2016-06-14 19:27 ` Luis R. Rodriguez 2016-06-14 20:47 ` Julia Lawall 0 siblings, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-14 19:27 UTC (permalink / raw) To: Julia Lawall Cc: Luis R. Rodriguez, Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote: > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then. > > > > > > Usually I put them in different files. > > > > We can do that as well but I would only want to deal with parmap support > > case. Any preference? How about .coccicheck.stderr.$PID where PID would > > be the PID of the shell script? > > I don't understand the connection with parmap. When parmap support is not available the cocciscript will currently disregard stderr, output is provided as it comes to stdout from each thread I guess. > Originally our use of parmap made output files based on pids. Maybe this > is the default for parmap. I found this completely unusable. I guess one > could look at the dates to see which file is the most recent one, but it > seems tedious. If you are putting the standard output in x.out, then put > the standard error in x.err. I'll use ${DIR}/coccicheck.$$.err for stderr. Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-14 19:27 ` Luis R. Rodriguez @ 2016-06-14 20:47 ` Julia Lawall 2016-06-14 21:10 ` Luis R. Rodriguez 0 siblings, 1 reply; 45+ messages in thread From: Julia Lawall @ 2016-06-14 20:47 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote: > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then. > > > > > > > > Usually I put them in different files. > > > > > > We can do that as well but I would only want to deal with parmap support > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would > > > be the PID of the shell script? > > > > I don't understand the connection with parmap. > > When parmap support is not available the cocciscript will currently > disregard stderr, output is provided as it comes to stdout from each > thread I guess. Deepa's recent patch to coccicheck made apparent that Coccicheck uses --very-quiet, so there is standard error. > > Originally our use of parmap made output files based on pids. Maybe this > > is the default for parmap. I found this completely unusable. I guess one > > could look at the dates to see which file is the most recent one, but it > > seems tedious. If you are putting the standard output in x.out, then put > > the standard error in x.err. > > I'll use ${DIR}/coccicheck.$$.err for stderr. What is ${DIR}? and what is $$? julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-14 20:47 ` Julia Lawall @ 2016-06-14 21:10 ` Luis R. Rodriguez 2016-06-14 21:17 ` Julia Lawall 0 siblings, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-14 21:10 UTC (permalink / raw) To: Julia Lawall Cc: Luis R. Rodriguez, Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote: > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote: > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then. > > > > > > > > > > Usually I put them in different files. > > > > > > > > We can do that as well but I would only want to deal with parmap support > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would > > > > be the PID of the shell script? > > > > > > I don't understand the connection with parmap. > > > > When parmap support is not available the cocciscript will currently > > disregard stderr, output is provided as it comes to stdout from each > > thread I guess. > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses > --very-quiet, so there is standard error. OK I'm disegarding the redirect for non-parmap for now but we'd have to determine if we want to append or add one per PID... I rather leave that stuff as-is and encourage folks to upgrade coccinelle. > > > Originally our use of parmap made output files based on pids. Maybe this > > > is the default for parmap. I found this completely unusable. I guess one > > > could look at the dates to see which file is the most recent one, but it > > > seems tedious. If you are putting the standard output in x.out, then put > > > the standard error in x.err. > > > > I'll use ${DIR}/coccicheck.$$.err for stderr. > > What is ${DIR}? and what is $$? When you run scripts/coccicheck we take the absolute directory of it and then go down one level of directory, so in this case it would be the base directory of the Linux kernel. $$ is the PID of the bash script. Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-14 21:10 ` Luis R. Rodriguez @ 2016-06-14 21:17 ` Julia Lawall 2016-06-14 22:02 ` Luis R. Rodriguez 0 siblings, 1 reply; 45+ messages in thread From: Julia Lawall @ 2016-06-14 21:17 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote: > > > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote: > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then. > > > > > > > > > > > > Usually I put them in different files. > > > > > > > > > > We can do that as well but I would only want to deal with parmap support > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would > > > > > be the PID of the shell script? > > > > > > > > I don't understand the connection with parmap. > > > > > > When parmap support is not available the cocciscript will currently > > > disregard stderr, output is provided as it comes to stdout from each > > > thread I guess. > > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses > > --very-quiet, so there is standard error. > > OK I'm disegarding the redirect for non-parmap for now but we'd have to > determine if we want to append or add one per PID... I rather leave that > stuff as-is and encourage folks to upgrade coccinelle. If coccicheck is using --very-quiet, there will not be much stderr of interest when using parmap either. I'm not sure to understan the issue about appending. Appending what to what? While Coccinelle is running, you have a directory with the same name as your semantic patch (or a directory name that you specify with --tmp-dir) that has separate files for each core's standard output and standard error. At the end, when all the code has been treated, Coccinelle writes the successive stdouts to standard output, and the sucessive stderrs to standard error. > > > > Originally our use of parmap made output files based on pids. Maybe this > > > > is the default for parmap. I found this completely unusable. I guess one > > > > could look at the dates to see which file is the most recent one, but it > > > > seems tedious. If you are putting the standard output in x.out, then put > > > > the standard error in x.err. > > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr. > > > > What is ${DIR}? and what is $$? > > When you run scripts/coccicheck we take the absolute directory > of it and then go down one level of directory, so in this case it > would be the base directory of the Linux kernel. > > $$ is the PID of the bash script. OK. I still don't find PIDs useful, but I guess if we are talking about the entire output of coccicheck, there is not much else to do. Normally, I don't want these files accumulating, and just write over the old ones. julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-14 21:17 ` Julia Lawall @ 2016-06-14 22:02 ` Luis R. Rodriguez 2016-06-15 7:39 ` Julia Lawall 0 siblings, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-14 22:02 UTC (permalink / raw) To: Julia Lawall Cc: Luis R. Rodriguez, Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote: > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote: > > > > > > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then. > > > > > > > > > > > > > > Usually I put them in different files. > > > > > > > > > > > > We can do that as well but I would only want to deal with parmap support > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would > > > > > > be the PID of the shell script? > > > > > > > > > > I don't understand the connection with parmap. > > > > > > > > When parmap support is not available the cocciscript will currently > > > > disregard stderr, output is provided as it comes to stdout from each > > > > thread I guess. > > > > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses > > > --very-quiet, so there is standard error. > > > > OK I'm disegarding the redirect for non-parmap for now but we'd have to > > determine if we want to append or add one per PID... I rather leave that > > stuff as-is and encourage folks to upgrade coccinelle. > > If coccicheck is using --very-quiet, there will not be much stderr of > interest when using parmap either. OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet is used ? Or does using --very-quiet suppress stderr ? > I'm not sure to understan the issue about appending. Appending what to > what? > While Coccinelle is running, you have a directory with the same > name as your semantic patch (or a directory name that you specify with > --tmp-dir) that has separate files for each core's standard output and > standard error. At the end, when all the code has been treated, > Coccinelle writes the successive stdouts to standard output, and the > sucessive stderrs to standard error. Ah I see. OK yeah never mind about appending. > > > > > Originally our use of parmap made output, specia files based on pids. Maybe this > > > > > is the default for parmap. I found this completely unusable. I guess one > > > > > could look at the dates to see which file is the most recent one, but it > > > > > seems tedious. If you are putting the standard output in x.out, then put > > > > > the standard error in x.err. > > > > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr. > > > > > > What is ${DIR}? and what is $$? > > > > When you run scripts/coccicheck we take the absolute directory > > of it and then go down one level of directory, so in this case it > > would be the base directory of the Linux kernel. > > > > $$ is the PID of the bash script. > > OK. I still don't find PIDs useful, but I guess if we are talking about > the entire output of coccicheck, there is not much else to do. Normally, > I don't want these files accumulating, and just write over the old ones. Which is why I would much prefer to instead just redirect in coccicheck case stderr to stdout from coccinelle. Is that preferred? Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-14 22:02 ` Luis R. Rodriguez @ 2016-06-15 7:39 ` Julia Lawall 2016-06-15 15:36 ` Luis R. Rodriguez 0 siblings, 1 reply; 45+ messages in thread From: Julia Lawall @ 2016-06-15 7:39 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Wed, 15 Jun 2016, Luis R. Rodriguez wrote: > On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote: > > > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > > > > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote: > > > > > > > > > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then. > > > > > > > > > > > > > > > > Usually I put them in different files. > > > > > > > > > > > > > > We can do that as well but I would only want to deal with parmap support > > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would > > > > > > > be the PID of the shell script? > > > > > > > > > > > > I don't understand the connection with parmap. > > > > > > > > > > When parmap support is not available the cocciscript will currently > > > > > disregard stderr, output is provided as it comes to stdout from each > > > > > thread I guess. > > > > > > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses > > > > --very-quiet, so there is standard error. > > > > > > OK I'm disegarding the redirect for non-parmap for now but we'd have to > > > determine if we want to append or add one per PID... I rather leave that > > > stuff as-is and encourage folks to upgrade coccinelle. > > > > If coccicheck is using --very-quiet, there will not be much stderr of > > interest when using parmap either. > > OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet > is used ? Or does using --very-quiet suppress stderr ? --very-quiet suppresses most administrative messages that go to stderr. There are still actual errors. Bu you don't see eg what file is being currently processed. > > > > > > Originally our use of parmap made output, specia files based on pids. Maybe this > > > > > > is the default for parmap. I found this completely unusable. I guess one > > > > > > could look at the dates to see which file is the most recent one, but it > > > > > > seems tedious. If you are putting the standard output in x.out, then put > > > > > > the standard error in x.err. > > > > > > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr. > > > > > > > > What is ${DIR}? and what is $$? > > > > > > When you run scripts/coccicheck we take the absolute directory > > > of it and then go down one level of directory, so in this case it > > > would be the base directory of the Linux kernel. > > > > > > $$ is the PID of the bash script. > > > > OK. I still don't find PIDs useful, but I guess if we are talking about > > the entire output of coccicheck, there is not much else to do. Normally, > > I don't want these files accumulating, and just write over the old ones. > > Which is why I would much prefer to instead just redirect in coccicheck > case stderr to stdout from coccinelle. Is that preferred? Then things will be merged in strange ways. Why not just let the user decide what to do with these things? julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-15 7:39 ` Julia Lawall @ 2016-06-15 15:36 ` Luis R. Rodriguez 2016-06-15 15:44 ` Julia Lawall 0 siblings, 1 reply; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-15 15:36 UTC (permalink / raw) To: Julia Lawall Cc: Luis R. Rodriguez, Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Wed, Jun 15, 2016 at 09:39:49AM +0200, Julia Lawall wrote: > > > On Wed, 15 Jun 2016, Luis R. Rodriguez wrote: > > > On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote: > > > > > > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then. > > > > > > > > > > > > > > > > > > Usually I put them in different files. > > > > > > > > > > > > > > > > We can do that as well but I would only want to deal with parmap support > > > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would > > > > > > > > be the PID of the shell script? > > > > > > > > > > > > > > I don't understand the connection with parmap. > > > > > > > > > > > > When parmap support is not available the cocciscript will currently > > > > > > disregard stderr, output is provided as it comes to stdout from each > > > > > > thread I guess. > > > > > > > > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses > > > > > --very-quiet, so there is standard error. > > > > > > > > OK I'm disegarding the redirect for non-parmap for now but we'd have to > > > > determine if we want to append or add one per PID... I rather leave that > > > > stuff as-is and encourage folks to upgrade coccinelle. > > > > > > If coccicheck is using --very-quiet, there will not be much stderr of > > > interest when using parmap either. > > > > OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet > > is used ? Or does using --very-quiet suppress stderr ? > > --very-quiet suppresses most administrative messages that go to stderr. > There are still actual errors. Bu you don't see eg what file is being > currently processed. OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll extend this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you can specify extra options that the script doesn't specifically support. > > > > > > > Originally our use of parmap made output, specia files based on pids. Maybe this > > > > > > > is the default for parmap. I found this completely unusable. I guess one > > > > > > > could look at the dates to see which file is the most recent one, but it > > > > > > > seems tedious. If you are putting the standard output in x.out, then put > > > > > > > the standard error in x.err. > > > > > > > > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr. > > > > > > > > > > What is ${DIR}? and what is $$? > > > > > > > > When you run scripts/coccicheck we take the absolute directory > > > > of it and then go down one level of directory, so in this case it > > > > would be the base directory of the Linux kernel. > > > > > > > > $$ is the PID of the bash script. > > > > > > OK. I still don't find PIDs useful, but I guess if we are talking about > > > the entire output of coccicheck, there is not much else to do. Normally, > > > I don't want these files accumulating, and just write over the old ones. > > > > Which is why I would much prefer to instead just redirect in coccicheck > > case stderr to stdout from coccinelle. Is that preferred? > > Then things will be merged in strange ways. > > Why not just let the user decide what to do with these things? Sure, what should be the default? Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-15 15:36 ` Luis R. Rodriguez @ 2016-06-15 15:44 ` Julia Lawall 2016-06-15 17:53 ` Luis R. Rodriguez 0 siblings, 1 reply; 45+ messages in thread From: Julia Lawall @ 2016-06-15 15:44 UTC (permalink / raw) To: Luis R. Rodriguez Cc: Julia Lawall, Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Wed, 15 Jun 2016, Luis R. Rodriguez wrote: > On Wed, Jun 15, 2016 at 09:39:49AM +0200, Julia Lawall wrote: > > > > > > On Wed, 15 Jun 2016, Luis R. Rodriguez wrote: > > > > > On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote: > > > > > > > > > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote: > > > > > > > > > > > > > > > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then. > > > > > > > > > > > > > > > > > > > > Usually I put them in different files. > > > > > > > > > > > > > > > > > > We can do that as well but I would only want to deal with parmap support > > > > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would > > > > > > > > > be the PID of the shell script? > > > > > > > > > > > > > > > > I don't understand the connection with parmap. > > > > > > > > > > > > > > When parmap support is not available the cocciscript will currently > > > > > > > disregard stderr, output is provided as it comes to stdout from each > > > > > > > thread I guess. > > > > > > > > > > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses > > > > > > --very-quiet, so there is standard error. > > > > > > > > > > OK I'm disegarding the redirect for non-parmap for now but we'd have to > > > > > determine if we want to append or add one per PID... I rather leave that > > > > > stuff as-is and encourage folks to upgrade coccinelle. > > > > > > > > If coccicheck is using --very-quiet, there will not be much stderr of > > > > interest when using parmap either. > > > > > > OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet > > > is used ? Or does using --very-quiet suppress stderr ? > > > > --very-quiet suppresses most administrative messages that go to stderr. > > There are still actual errors. Bu you don't see eg what file is being > > currently processed. > > OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll extend > this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you can > specify extra options that the script doesn't specifically support. If it is more convenient, you don't actually have to remove --very-quiet. You can just put --quiet before --show-trying or --profile. --quiet will override --very-quiet. > > > > > > > > Originally our use of parmap made output, specia files based on pids. Maybe this > > > > > > > > is the default for parmap. I found this completely unusable. I guess one > > > > > > > > could look at the dates to see which file is the most recent one, but it > > > > > > > > seems tedious. If you are putting the standard output in x.out, then put > > > > > > > > the standard error in x.err. > > > > > > > > > > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr. > > > > > > > > > > > > What is ${DIR}? and what is $$? > > > > > > > > > > When you run scripts/coccicheck we take the absolute directory > > > > > of it and then go down one level of directory, so in this case it > > > > > would be the base directory of the Linux kernel. > > > > > > > > > > $$ is the PID of the bash script. > > > > > > > > OK. I still don't find PIDs useful, but I guess if we are talking about > > > > the entire output of coccicheck, there is not much else to do. Normally, > > > > I don't want these files accumulating, and just write over the old ones. > > > > > > Which is why I would much prefer to instead just redirect in coccicheck > > > case stderr to stdout from coccinelle. Is that preferred? > > > > Then things will be merged in strange ways. > > > > Why not just let the user decide what to do with these things? > > Sure, what should be the default? I would normally just expect standard output and standard error to appear randomly on the screen. That is, no management effort from the tool at all. julia ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-15 15:44 ` Julia Lawall @ 2016-06-15 17:53 ` Luis R. Rodriguez 0 siblings, 0 replies; 45+ messages in thread From: Luis R. Rodriguez @ 2016-06-15 17:53 UTC (permalink / raw) To: Julia Lawall Cc: Luis R. Rodriguez, Gilles Muller, nicolas.palix, mmarek, linux-kernel, cocci On Wed, Jun 15, 2016 at 05:44:01PM +0200, Julia Lawall wrote: > > > On Wed, 15 Jun 2016, Luis R. Rodriguez wrote: > > OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll extend > > this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you can > > specify extra options that the script doesn't specifically support. > > If it is more convenient, you don't actually have to remove --very-quiet. > You can just put --quiet before --show-trying or --profile. --quiet will > override --very-quiet. How about just: if [ "$SPFLAGS" == *"--profile"* -o "$SPFLAGS" == "--show-trying" ]; then FLAGS="--quiet $SPFLAGS" else FLAGS="--very-quiet $SPFLAGS" fi > > > > > > > > > Originally our use of parmap made output, specia files based on pids. Maybe this > > > > > > > > > is the default for parmap. I found this completely unusable. I guess one > > > > > > > > > could look at the dates to see which file is the most recent one, but it > > > > > > > > > seems tedious. If you are putting the standard output in x.out, then put > > > > > > > > > the standard error in x.err. > > > > > > > > > > > > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr. > > > > > > > > > > > > > > What is ${DIR}? and what is $$? > > > > > > > > > > > > When you run scripts/coccicheck we take the absolute directory > > > > > > of it and then go down one level of directory, so in this case it > > > > > > would be the base directory of the Linux kernel. > > > > > > > > > > > > $$ is the PID of the bash script. > > > > > > > > > > OK. I still don't find PIDs useful, but I guess if we are talking about > > > > > the entire output of coccicheck, there is not much else to do. Normally, > > > > > I don't want these files accumulating, and just write over the old ones. > > > > > > > > Which is why I would much prefer to instead just redirect in coccicheck > > > > case stderr to stdout from coccinelle. Is that preferred? > > > > > > Then things will be merged in strange ways. > > > > > > Why not just let the user decide what to do with these things? > > > > Sure, what should be the default? > > I would normally just expect standard output and standard error to appear > randomly on the screen. That is, no management effort from the tool at > all. But the thing is, stderr is ignored now given that a shell script is used wrapped over a Makefile so if we want what you describe I think we do have to by default do 2>&1 on the spatch run command. Luis ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options 2016-06-10 20:42 ` [PATCH 4/4] coccicheck: add indexing enhancement options Luis R. Rodriguez 2016-06-10 21:02 ` Julia Lawall @ 2016-06-11 5:55 ` SF Markus Elfring 1 sibling, 0 replies; 45+ messages in thread From: SF Markus Elfring @ 2016-06-11 5:55 UTC (permalink / raw) To: Luis R. Rodriguez; +Cc: cocci, linux-kernel > in practice though this seems to not perform better than > regular grep however its expected to help with some use cases > so we use that if you have no other indexing options in place > available. Would you like to fix a typo in this paragraph? Regards, Markus ^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [Cocci] [PATCH 0/4] scripts/coccicheck: add paramap and indexing options 2016-06-10 20:42 [PATCH 0/4] scripts/coccicheck: add paramap and indexing options Luis R. Rodriguez ` (3 preceding siblings ...) 2016-06-10 20:42 ` [PATCH 4/4] coccicheck: add indexing enhancement options Luis R. Rodriguez @ 2016-06-11 5:27 ` SF Markus Elfring 4 siblings, 0 replies; 45+ messages in thread From: SF Markus Elfring @ 2016-06-11 5:27 UTC (permalink / raw) To: Luis R. Rodriguez Cc: cocci, linux-kernel, Gilles Muller, Michal Marek, Nicolas Palix > we should be able to also take advantage of with coccicheck, > the most important one is paramap support. Do OCaml developers prefer to refer to the software "Parmap" here? https://rdicosmo.github.io/parmap/ Regards, Markus ^ permalink raw reply [flat|nested] 45+ messages in thread
end of thread, other threads:[~2016-06-15 17:53 UTC | newest] Thread overview: 45+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-06-10 20:42 [PATCH 0/4] scripts/coccicheck: add paramap and indexing options Luis R. Rodriguez 2016-06-10 20:42 ` [PATCH 1/4] coccicheck: move spatch binary check up Luis R. Rodriguez 2016-06-10 20:42 ` [PATCH 2/4] coccicheck: enable paramap support Luis R. Rodriguez 2016-06-11 5:45 ` [Cocci] " SF Markus Elfring 2016-06-11 5:55 ` Julia Lawall 2016-06-10 20:42 ` [PATCH 3/4] scripts: add glimpse.sh for indexing the kernel Luis R. Rodriguez 2016-06-11 17:09 ` [Cocci] " SF Markus Elfring 2016-06-10 20:42 ` [PATCH 4/4] coccicheck: add indexing enhancement options Luis R. Rodriguez 2016-06-10 21:02 ` Julia Lawall 2016-06-10 21:18 ` Luis R. Rodriguez 2016-06-10 21:21 ` Julia Lawall 2016-06-10 21:43 ` [Cocci] " Wolfram Sang 2016-06-10 21:49 ` Luis R. Rodriguez 2016-06-10 21:51 ` Wolfram Sang 2016-06-10 22:08 ` Luis R. Rodriguez 2016-06-10 22:25 ` Luis R. Rodriguez 2016-06-11 5:46 ` Wolfram Sang 2016-06-11 5:54 ` Julia Lawall 2016-06-11 6:09 ` Wolfram Sang 2016-06-13 18:37 ` Luis R. Rodriguez 2016-06-13 18:55 ` Wolfram Sang 2016-06-13 19:48 ` Julia Lawall 2016-06-13 21:22 ` Luis R. Rodriguez 2016-06-14 5:08 ` Julia Lawall 2016-06-11 5:18 ` Julia Lawall 2016-06-11 5:58 ` Wolfram Sang 2016-06-11 6:05 ` Julia Lawall 2016-06-11 5:24 ` Julia Lawall 2016-06-13 18:39 ` Luis R. Rodriguez 2016-06-11 5:17 ` Julia Lawall 2016-06-13 19:35 ` Luis R. Rodriguez 2016-06-13 19:50 ` Julia Lawall 2016-06-13 21:28 ` Luis R. Rodriguez 2016-06-14 5:22 ` Julia Lawall 2016-06-14 19:27 ` Luis R. Rodriguez 2016-06-14 20:47 ` Julia Lawall 2016-06-14 21:10 ` Luis R. Rodriguez 2016-06-14 21:17 ` Julia Lawall 2016-06-14 22:02 ` Luis R. Rodriguez 2016-06-15 7:39 ` Julia Lawall 2016-06-15 15:36 ` Luis R. Rodriguez 2016-06-15 15:44 ` Julia Lawall 2016-06-15 17:53 ` Luis R. Rodriguez 2016-06-11 5:55 ` [Cocci] " SF Markus Elfring 2016-06-11 5:27 ` [Cocci] [PATCH 0/4] scripts/coccicheck: add paramap and indexing options SF Markus Elfring
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).