[PATCH v2 0/8] coccicheck: modernize

[PATCH v2 0/8] coccicheck: modernize

[Luis R. Rodriguez]

This series gives coccicheck some love by modernizing it
to catch up to some features implemented in Coccinelle, of
most importance is parmap support and some indexing
enhancements. This should help speed things up considerably,
for me this now runs twice as fast in some situations.

This also takes advantage of the open sourcing of Glimpse,
thanks to the university of Arizona for doing and to Udi Manber
for helping push that out.

Since Coccinelle is ever evolving this also now allows developer
to require specific Coccinelle versions on top of the SmPL patches.

This is part of a larger series, all of which you can find here:

https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/log/?h=20160616-sysdata-v2

Since further development is done on top of this we may need to coordinate
with another maintainer for how these go in. The coccicheck changes will be
important to help speed up some of the new Coccinelle API checks and
transformations implemented in later series on the request firmware API.

Luis R. Rodriguez (8):
  coccicheck: move spatch binary check up
  coccicheck: enable parmap support
  coccicheck: add indexing enhancement options
  scripts: add glimpse.sh for indexing the kernel
  coccicheck: replace --very-quiet with --quit when debugging
  coccicheck: add support for requring a coccinelle version
  coccicheck: refer to coccicheck bottest wiki for documentation
  scripts/coccinelle: require coccinelle >= 1.0.4 on
    device_node_continue.cocci

 .gitignore                                         |   3 +
 scripts/coccicheck                                 | 242 ++++++++++++++++++++-
 .../iterators/device_node_continue.cocci           |   3 +
 scripts/glimpse.sh                                 |  11 +
 4 files changed, 248 insertions(+), 11 deletions(-)
 create mode 100755 scripts/glimpse.sh

-- 
2.8.2

[Cocci] [PATCH v2 0/8] coccicheck: modernize

[Luis R. Rodriguez]

This series gives coccicheck some love by modernizing it
to catch up to some features implemented in Coccinelle, of
most importance is parmap support and some indexing
enhancements. This should help speed things up considerably,
for me this now runs twice as fast in some situations.

This also takes advantage of the open sourcing of Glimpse,
thanks to the university of Arizona for doing and to Udi Manber
for helping push that out.

Since Coccinelle is ever evolving this also now allows developer
to require specific Coccinelle versions on top of the SmPL patches.

This is part of a larger series, all of which you can find here:

https://git.kernel.org/cgit/linux/kernel/git/mcgrof/linux-next.git/log/?h=20160616-sysdata-v2

Since further development is done on top of this we may need to coordinate
with another maintainer for how these go in. The coccicheck changes will be
important to help speed up some of the new Coccinelle API checks and
transformations implemented in later series on the request firmware API.

Luis R. Rodriguez (8):
  coccicheck: move spatch binary check up
  coccicheck: enable parmap support
  coccicheck: add indexing enhancement options
  scripts: add glimpse.sh for indexing the kernel
  coccicheck: replace --very-quiet with --quit when debugging
  coccicheck: add support for requring a coccinelle version
  coccicheck: refer to coccicheck bottest wiki for documentation
  scripts/coccinelle: require coccinelle >= 1.0.4 on
    device_node_continue.cocci

 .gitignore                                         |   3 +
 scripts/coccicheck                                 | 242 ++++++++++++++++++++-
 .../iterators/device_node_continue.cocci           |   3 +
 scripts/glimpse.sh                                 |  11 +
 4 files changed, 248 insertions(+), 11 deletions(-)
 create mode 100755 scripts/glimpse.sh

-- 
2.8.2

[PATCH v2 1/8] coccicheck: move spatch binary check up

[Luis R. Rodriguez]

This has no functional changes. This is being done
to enable us to later use spatch binary for some
flag checking for certain features early on.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index dd85a455b2ba..aa5e78fba270 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -7,6 +7,11 @@
 
 SPATCH="`which ${SPATCH:=spatch}`"
 
+if [ ! -x "$SPATCH" ]; then
+    echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/'
+    exit 1
+fi
+
 trap kill_running SIGTERM SIGINT
 declare -a SPATCH_PID
 
@@ -51,11 +56,6 @@ if [ "$KBUILD_EXTMOD" != "" ] ; then
     OPTIONS="--patch $srctree $OPTIONS"
 fi
 
-if [ ! -x "$SPATCH" ]; then
-    echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/'
-    exit 1
-fi
-
 if [ "$MODE" = "" ] ; then
     if [ "$ONLINE" = "0" ] ; then
 	echo 'You have not explicitly specified the mode to use. Using default "report" mode.'
-- 
2.8.2

[Cocci] [PATCH v2 1/8] coccicheck: move spatch binary check up

[Luis R. Rodriguez]

This has no functional changes. This is being done
to enable us to later use spatch binary for some
flag checking for certain features early on.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index dd85a455b2ba..aa5e78fba270 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -7,6 +7,11 @@
 
 SPATCH="`which ${SPATCH:=spatch}`"
 
+if [ ! -x "$SPATCH" ]; then
+    echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/'
+    exit 1
+fi
+
 trap kill_running SIGTERM SIGINT
 declare -a SPATCH_PID
 
@@ -51,11 +56,6 @@ if [ "$KBUILD_EXTMOD" != "" ] ; then
     OPTIONS="--patch $srctree $OPTIONS"
 fi
 
-if [ ! -x "$SPATCH" ]; then
-    echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/'
-    exit 1
-fi
-
 if [ "$MODE" = "" ] ; then
     if [ "$ONLINE" = "0" ] ; then
 	echo 'You have not explicitly specified the mode to use. Using default "report" mode.'
-- 
2.8.2

[PATCH v2 2/8] coccicheck: enable parmap support

[Luis R. Rodriguez]

Coccinelle has had parmap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported.

Also enable the load balancing to be dynamic, so that if a
thread finishes early we keep feeding it.

Note: now that we have all things handled for us, redirect stderr to
stdout as well to capture any possible errors or warnings issued by
coccinelle.

If --jobs is not supported we fallback to the old mechanism.
This also now accepts DEBUG_FILE= to specify where you want
stderr to be redirected to, by default we redirect stderr to
/dev/null.

Also since while at it propagate back into the shell script any
coccinelle error code. When used in serialized mode where all cocci
files are run this also stops processing if an error has occured.
This lets us handle some errors in coccinelle cocci files and if
they bail out we should inspect the errors. This will be more
useful later to help annotate coccinelle version dependency
requirements. This will let you run only SmPL files that your
system supports.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>

As a small example, prior to this change, on an 8-core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report DEBUG_FILE=cocci.err
...

real    29m14.912s
user    103m1.796s
sys     0m4.464s

After:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

v2:

o redirect coccinelle stderr to /dev/null by default and
  only if DEBUG_FILE is used do we pass it to a file
o fix typo of paramap/parmap

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 40 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 37 insertions(+), 3 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index aa5e78fba270..7acef3efc258 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
     exit 1
 fi
 
-trap kill_running SIGTERM SIGINT
-declare -a SPATCH_PID
+USE_JOBS="no"
+$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
@@ -82,7 +82,33 @@ if [ "$ONLINE" = "0" ] ; then
     echo ''
 fi
 
-run_cmd() {
+if [ "$USE_JOBS" = "no" ]; then
+	trap kill_running SIGTERM SIGINT
+	declare -a SPATCH_PID
+else
+	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
+fi
+
+run_cmd_parmap() {
+	if [ $VERBOSE -ne 0 ] ; then
+		echo "Running ($NPROC in parallel): $@"
+	fi
+	if [ "$DEBUG_FILE" != "/dev/null" -a "$DEBUG_FILE" != "" ]; then
+		if [ -f $DEBUG_FILE ]; then
+			echo "Debug file $DEBUG_FILE exists, bailing"
+			exit
+		fi
+	else
+		DEBUG_FILE="/dev/null"
+	fi
+	$@ 2>$DEBUG_FILE
+	if [[ $? -ne 0 ]]; then
+		echo "coccicheck failed"
+		exit $?
+	fi
+}
+
+run_cmd_old() {
 	local i
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
@@ -97,6 +123,14 @@ run_cmd() {
 	wait
 }
 
+run_cmd() {
+	if [ "$USE_JOBS" = "yes" ]; then
+		run_cmd_parmap $@
+	else
+		run_cmd_old $@
+	fi
+}
+
 kill_running() {
 	for i in $(seq 0 $(( NPROC - 1 )) ); do
 		if [ $VERBOSE -eq 2 ] ; then
-- 
2.8.2

[Cocci] [PATCH v2 2/8] coccicheck: enable parmap support

[Luis R. Rodriguez]

Coccinelle has had parmap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported.

Also enable the load balancing to be dynamic, so that if a
thread finishes early we keep feeding it.

Note: now that we have all things handled for us, redirect stderr to
stdout as well to capture any possible errors or warnings issued by
coccinelle.

If --jobs is not supported we fallback to the old mechanism.
This also now accepts DEBUG_FILE= to specify where you want
stderr to be redirected to, by default we redirect stderr to
/dev/null.

Also since while at it propagate back into the shell script any
coccinelle error code. When used in serialized mode where all cocci
files are run this also stops processing if an error has occured.
This lets us handle some errors in coccinelle cocci files and if
they bail out we should inspect the errors. This will be more
useful later to help annotate coccinelle version dependency
requirements. This will let you run only SmPL files that your
system supports.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>

As a small example, prior to this change, on an 8-core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report DEBUG_FILE=cocci.err
...

real    29m14.912s
user    103m1.796s
sys     0m4.464s

After:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

v2:

o redirect coccinelle stderr to /dev/null by default and
  only if DEBUG_FILE is used do we pass it to a file
o fix typo of paramap/parmap

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 40 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 37 insertions(+), 3 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index aa5e78fba270..7acef3efc258 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
     exit 1
 fi
 
-trap kill_running SIGTERM SIGINT
-declare -a SPATCH_PID
+USE_JOBS="no"
+$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
@@ -82,7 +82,33 @@ if [ "$ONLINE" = "0" ] ; then
     echo ''
 fi
 
-run_cmd() {
+if [ "$USE_JOBS" = "no" ]; then
+	trap kill_running SIGTERM SIGINT
+	declare -a SPATCH_PID
+else
+	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
+fi
+
+run_cmd_parmap() {
+	if [ $VERBOSE -ne 0 ] ; then
+		echo "Running ($NPROC in parallel): $@"
+	fi
+	if [ "$DEBUG_FILE" != "/dev/null" -a "$DEBUG_FILE" != "" ]; then
+		if [ -f $DEBUG_FILE ]; then
+			echo "Debug file $DEBUG_FILE exists, bailing"
+			exit
+		fi
+	else
+		DEBUG_FILE="/dev/null"
+	fi
+	$@ 2>$DEBUG_FILE
+	if [[ $? -ne 0 ]]; then
+		echo "coccicheck failed"
+		exit $?
+	fi
+}
+
+run_cmd_old() {
 	local i
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
@@ -97,6 +123,14 @@ run_cmd() {
 	wait
 }
 
+run_cmd() {
+	if [ "$USE_JOBS" = "yes" ]; then
+		run_cmd_parmap $@
+	else
+		run_cmd_old $@
+	fi
+}
+
 kill_running() {
 	for i in $(seq 0 $(( NPROC - 1 )) ); do
 		if [ $VERBOSE -eq 2 ] ; then
-- 
2.8.2

[PATCH v2 3/8] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

Coccinelle has support to make use of its own enhanced "grep"
mechanisms instead of using regular grep for searching code,
it calls this 'coccigrep'. In lack of any indexing optimization
information it uses --use-coccigrep by default.

This patch enable indexing optimizations heuristics so that coccigrep
can automatically detect what indexing options are available and use
them accordinly without any user input.

Since git has its own index, support for using 'git grep' has been
added to Coccinelle, that should on average perform better than
using the internal coccigrep. Coccinelle has had idutils support
as well for a while now, you however need to refer to the index
file. We support detecting two idutils index files by default,
ID and .id-utils.index, assuming you ran either of:

 # What you might have done:
mkid -s
 # as in coccinelle scripts/idutils_index.sh
mkid -i C --output .id-utils.index *

Lastly, Coccinelle has had support for glimpseindex for a long while,
however the glimpseindex tool, the agrep library were previously closed
source, its all now open sourced, and provides the best performance, so
support that if we can detect you have a glimpse index.

You can always override the index as follows:

$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ time make coccicheck MODE=report

coccigrep (default and without this patch):
real    0m16.369s
user    0m58.712s
sys     0m5.064s

After:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ time make coccicheck MODE=report

With glimpse:
real    0m6.549s
user    0m49.136s
sys     0m3.076s

With idutils:
real    0m6.749s
user    0m51.936s
sys     0m3.876s

With gitgrep:
real    0m6.805s
user    0m51.572s
sys     0m4.432s

v2 changes:

o simplify DIR assignment to 1 line
o detected a bug when KBUILD_EXTMOD is used other than the parent
  directory for both glimpse and idutils, so we avoid both when
  M=path/driver/ is used. This is being looked into upstream on
  Coccinelle.
o move indexing heuristics to a file
o document logic used for indexing
o add idutils support, supports two indexing files
o remove coccigrep heuristics as its the default anyway
o add COCCI_INDEX to enable overriding heuristics, you can use this
  as follows, for example:
o replace references to stderr file with DEBUG_FILE use instructions

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 150 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 150 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 7acef3efc258..30f5a531ad34 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -5,6 +5,7 @@
 # version 1.0.0-rc11.
 #
 
+DIR="$(dirname $(readlink -f $0))/.."
 SPATCH="`which ${SPATCH:=spatch}`"
 
 if [ ! -x "$SPATCH" ]; then
@@ -15,6 +16,134 @@ fi
 USE_JOBS="no"
 $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
+function can_use_glimpse()
+{
+	$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null
+	if [ $? -ne 0 ]; then
+		echo "no"
+		return
+	fi
+	if [ ! -f $DIR/.glimpse_index ]; then
+		echo "no"
+		return
+	fi
+
+	# As of coccinelle 1.0.5 --use-glimpse cannot be used with M= other
+	# than the base directory. We expect a future release will let us
+	# specify the full path of the glimpse index but even if that's
+	# supported, glimpse use seems to require an index per every
+	# directory parsed, so you'd have to generate a glimpse index
+	# per directory. If M=path is used (where epath is not the top level)
+	# we'll have to fallback to alternatives then.
+	if [ "$KBUILD_EXTMOD" != "./" -a "$KBUILD_EXTMOD" != "" ]; then
+		echo "no"
+		return
+	fi
+	echo yes
+}
+
+function can_use_idutils()
+{
+	$SPATCH --help | grep "\-\-use\-idutils" > /dev/null
+	if [ $? -ne 0 ]; then
+		echo "no"
+		return
+	fi
+	# As of coccinelle 1.0.5 --use-idutils will bust if one uses
+	# idutils with an index out of the main tree.
+	if [ "$KBUILD_EXTMOD" != "./" -a "$KBUILD_EXTMOD" != "" ]; then
+		echo "no"
+		return
+	fi
+	if [ -f $DIR/ID -o -f $DIR/.id-utils.index ]; then
+		echo "yes"
+		return
+	fi
+	echo "no"
+}
+
+function can_use_gitgrep()
+{
+	$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null
+	if [ $? -ne 0 ]; then
+		echo "no"
+		return
+	fi
+	if [ ! -d $DIR/.git ]; then
+		echo "no"
+		return
+	fi
+	echo "yes"
+}
+
+# Indexing USE_* optimizations heuristics.
+#
+# Linux runs on git (TM). However, if you have supplemental indexing options
+# you may use them to help Coccinelle further. If you are using Coccinelle
+# within a target git tree --use-gitrep will be used, and this should
+# suffice for most uses. If you however want optimal performance do
+# consider embracing a supplemental indexing as part of your development.
+# For instance glimpse, and idutils can be used, however you should
+# be sure to update the indexes as often as you update your git tree to
+# ensure your indexes are not stale.
+#
+# idutils is currently not as efficient as glimpse because the query language
+# for glimpse is simpler, so when idutils is used more filtering has to be
+# done at the ocaml level within Coccinelle. Glimpse allows queries that are
+# arbitrary formulas, up to a limited level of complexity, involving both
+# && and ||. For idutils, Coccinelle runs lid intersections on the result.
+#
+# You can override these heuristics with COCCI_INDEX="--use-gitgrep" for
+# example. This will force to use --use-gitgrep even if you have a glimpse
+# index. Other examples:
+#
+# $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"
+#
+# The order of heuristics for indexing used by coccicheck is listed below.
+#
+# 0. Glimpse currently should outperform all indexing options. so if a glimpse
+#    index is used we use it. Refer to Linux scripts/glimpse.sh for details.
+#    If you think you should be getting better performance with glimpse than
+#    what you would expect inspect the stderr log file for cocciecheck, you
+#    ask for a debug file with DEBUG_FILE="" parameter to coccicheck.
+#
+#    If glimpse is running correctly there should be very few occurrences
+#    of "Skipping", also coccinelle will inform you if it could not use
+#    glimpse. As an example an output of the following would indicate glimpse
+#    was properly used on the stderr log file:
+#
+#    There are matches to 1252 out of 47281 files
+#    glimpse request = request_threaded_irq
+#
+# 1. Use idutils next. You'll need to generate an index using either of these:
+#
+#	a) mkid -s
+#       By default this dumps the index into ./ID
+#
+#	b) mkid -i C --output .id-utils.index *
+#       This method is provided with coccinelle repo on
+#       scripts/idutils_index.sh
+#
+# 2. Next best is --use-gitgrep and if you are working within a git tree
+#    this will be used by default.
+#
+# 3. By default coccinelle internally uses --use-coccigrep if no indexing
+#    options are requested and your version of coccinelle supports it so we
+#    do not need to be specific about requesting that as a fallback mechanism.
+#    Use of --use-coccigrep is comparable to --use-gitgrep.
+#
+# XXX: Glimpse is not well maintained. See if we can add similar indexing
+# features and query language glimpse supports to git.
+if [ "$COCCI_INDEX" = "" ] ; then
+	USE_GLIMPSE=$(can_use_glimpse)
+	USE_IDUTILS=$(can_use_idutils)
+	USE_GITGREP=$(can_use_gitgrep)
+fi
+
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
 
@@ -89,6 +218,27 @@ else
 	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
 fi
 
+# Check COCCI_INDEX first to manual override, otherwise rely on
+# internal heuristics documented above.
+if [ "$COCCI_INDEX" != "" ] ; then
+	OPTIONS="$OPTIONS $COCCI_INDEX"
+elif [ "$USE_GLIMPSE" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-glimpse"
+elif [ "$USE_IDUTILS" = "yes" ]; then
+	index=""
+	if [ -f $DIR/ID ]; then
+		index="$DIR/ID"
+	elif [ -f $DIR/.id-utils.index ]; then
+		index="$DIR/.id-utils.index"
+	else
+		echo "idutils index not found, expected: $DIR/ID or $DIR/.id-utils.index"
+		exit 1
+	fi
+	OPTIONS="$OPTIONS --use-idutils $index"
+elif [ "$USE_GITGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-gitgrep"
+fi
+
 run_cmd_parmap() {
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
-- 
2.8.2

[Cocci] [PATCH v2 3/8] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

Coccinelle has support to make use of its own enhanced "grep"
mechanisms instead of using regular grep for searching code,
it calls this 'coccigrep'. In lack of any indexing optimization
information it uses --use-coccigrep by default.

This patch enable indexing optimizations heuristics so that coccigrep
can automatically detect what indexing options are available and use
them accordinly without any user input.

Since git has its own index, support for using 'git grep' has been
added to Coccinelle, that should on average perform better than
using the internal coccigrep. Coccinelle has had idutils support
as well for a while now, you however need to refer to the index
file. We support detecting two idutils index files by default,
ID and .id-utils.index, assuming you ran either of:

 # What you might have done:
mkid -s
 # as in coccinelle scripts/idutils_index.sh
mkid -i C --output .id-utils.index *

Lastly, Coccinelle has had support for glimpseindex for a long while,
however the glimpseindex tool, the agrep library were previously closed
source, its all now open sourced, and provides the best performance, so
support that if we can detect you have a glimpse index.

You can always override the index as follows:

$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ time make coccicheck MODE=report

coccigrep (default and without this patch):
real    0m16.369s
user    0m58.712s
sys     0m5.064s

After:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ time make coccicheck MODE=report

With glimpse:
real    0m6.549s
user    0m49.136s
sys     0m3.076s

With idutils:
real    0m6.749s
user    0m51.936s
sys     0m3.876s

With gitgrep:
real    0m6.805s
user    0m51.572s
sys     0m4.432s

v2 changes:

o simplify DIR assignment to 1 line
o detected a bug when KBUILD_EXTMOD is used other than the parent
  directory for both glimpse and idutils, so we avoid both when
  M=path/driver/ is used. This is being looked into upstream on
  Coccinelle.
o move indexing heuristics to a file
o document logic used for indexing
o add idutils support, supports two indexing files
o remove coccigrep heuristics as its the default anyway
o add COCCI_INDEX to enable overriding heuristics, you can use this
  as follows, for example:
o replace references to stderr file with DEBUG_FILE use instructions

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 150 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 150 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 7acef3efc258..30f5a531ad34 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -5,6 +5,7 @@
 # version 1.0.0-rc11.
 #
 
+DIR="$(dirname $(readlink -f $0))/.."
 SPATCH="`which ${SPATCH:=spatch}`"
 
 if [ ! -x "$SPATCH" ]; then
@@ -15,6 +16,134 @@ fi
 USE_JOBS="no"
 $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
+function can_use_glimpse()
+{
+	$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null
+	if [ $? -ne 0 ]; then
+		echo "no"
+		return
+	fi
+	if [ ! -f $DIR/.glimpse_index ]; then
+		echo "no"
+		return
+	fi
+
+	# As of coccinelle 1.0.5 --use-glimpse cannot be used with M= other
+	# than the base directory. We expect a future release will let us
+	# specify the full path of the glimpse index but even if that's
+	# supported, glimpse use seems to require an index per every
+	# directory parsed, so you'd have to generate a glimpse index
+	# per directory. If M=path is used (where epath is not the top level)
+	# we'll have to fallback to alternatives then.
+	if [ "$KBUILD_EXTMOD" != "./" -a "$KBUILD_EXTMOD" != "" ]; then
+		echo "no"
+		return
+	fi
+	echo yes
+}
+
+function can_use_idutils()
+{
+	$SPATCH --help | grep "\-\-use\-idutils" > /dev/null
+	if [ $? -ne 0 ]; then
+		echo "no"
+		return
+	fi
+	# As of coccinelle 1.0.5 --use-idutils will bust if one uses
+	# idutils with an index out of the main tree.
+	if [ "$KBUILD_EXTMOD" != "./" -a "$KBUILD_EXTMOD" != "" ]; then
+		echo "no"
+		return
+	fi
+	if [ -f $DIR/ID -o -f $DIR/.id-utils.index ]; then
+		echo "yes"
+		return
+	fi
+	echo "no"
+}
+
+function can_use_gitgrep()
+{
+	$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null
+	if [ $? -ne 0 ]; then
+		echo "no"
+		return
+	fi
+	if [ ! -d $DIR/.git ]; then
+		echo "no"
+		return
+	fi
+	echo "yes"
+}
+
+# Indexing USE_* optimizations heuristics.
+#
+# Linux runs on git (TM). However, if you have supplemental indexing options
+# you may use them to help Coccinelle further. If you are using Coccinelle
+# within a target git tree --use-gitrep will be used, and this should
+# suffice for most uses. If you however want optimal performance do
+# consider embracing a supplemental indexing as part of your development.
+# For instance glimpse, and idutils can be used, however you should
+# be sure to update the indexes as often as you update your git tree to
+# ensure your indexes are not stale.
+#
+# idutils is currently not as efficient as glimpse because the query language
+# for glimpse is simpler, so when idutils is used more filtering has to be
+# done at the ocaml level within Coccinelle. Glimpse allows queries that are
+# arbitrary formulas, up to a limited level of complexity, involving both
+# && and ||. For idutils, Coccinelle runs lid intersections on the result.
+#
+# You can override these heuristics with COCCI_INDEX="--use-gitgrep" for
+# example. This will force to use --use-gitgrep even if you have a glimpse
+# index. Other examples:
+#
+# $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"
+#
+# The order of heuristics for indexing used by coccicheck is listed below.
+#
+# 0. Glimpse currently should outperform all indexing options. so if a glimpse
+#    index is used we use it. Refer to Linux scripts/glimpse.sh for details.
+#    If you think you should be getting better performance with glimpse than
+#    what you would expect inspect the stderr log file for cocciecheck, you
+#    ask for a debug file with DEBUG_FILE="" parameter to coccicheck.
+#
+#    If glimpse is running correctly there should be very few occurrences
+#    of "Skipping", also coccinelle will inform you if it could not use
+#    glimpse. As an example an output of the following would indicate glimpse
+#    was properly used on the stderr log file:
+#
+#    There are matches to 1252 out of 47281 files
+#    glimpse request = request_threaded_irq
+#
+# 1. Use idutils next. You'll need to generate an index using either of these:
+#
+#	a) mkid -s
+#       By default this dumps the index into ./ID
+#
+#	b) mkid -i C --output .id-utils.index *
+#       This method is provided with coccinelle repo on
+#       scripts/idutils_index.sh
+#
+# 2. Next best is --use-gitgrep and if you are working within a git tree
+#    this will be used by default.
+#
+# 3. By default coccinelle internally uses --use-coccigrep if no indexing
+#    options are requested and your version of coccinelle supports it so we
+#    do not need to be specific about requesting that as a fallback mechanism.
+#    Use of --use-coccigrep is comparable to --use-gitgrep.
+#
+# XXX: Glimpse is not well maintained. See if we can add similar indexing
+# features and query language glimpse supports to git.
+if [ "$COCCI_INDEX" = "" ] ; then
+	USE_GLIMPSE=$(can_use_glimpse)
+	USE_IDUTILS=$(can_use_idutils)
+	USE_GITGREP=$(can_use_gitgrep)
+fi
+
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
 
@@ -89,6 +218,27 @@ else
 	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
 fi
 
+# Check COCCI_INDEX first to manual override, otherwise rely on
+# internal heuristics documented above.
+if [ "$COCCI_INDEX" != "" ] ; then
+	OPTIONS="$OPTIONS $COCCI_INDEX"
+elif [ "$USE_GLIMPSE" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-glimpse"
+elif [ "$USE_IDUTILS" = "yes" ]; then
+	index=""
+	if [ -f $DIR/ID ]; then
+		index="$DIR/ID"
+	elif [ -f $DIR/.id-utils.index ]; then
+		index="$DIR/.id-utils.index"
+	else
+		echo "idutils index not found, expected: $DIR/ID or $DIR/.id-utils.index"
+		exit 1
+	fi
+	OPTIONS="$OPTIONS --use-idutils $index"
+elif [ "$USE_GITGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-gitgrep"
+fi
+
 run_cmd_parmap() {
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
-- 
2.8.2

[PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

Glimpse is a tool you can use to index the kernel. The tool
was open sourced on 2014-09-26 [0] under the ISC license however
the original release still [1] does not compile. A fix for this
and a release that does compile is provided in a temporary
tree [2].

v2 changesl:
o simplify DIR in one line and add verbose release information
  to commit log about the open sourcing of glimpse and its
  current status.
o add index files to .gitignore

[0] http://webglimpse.net/
[1] https://github.com/gvelez17/glimpse/
[2] https://github.com/mcgrof/glimpse.git

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 .gitignore         |  3 +++
 scripts/glimpse.sh | 11 +++++++++++
 2 files changed, 14 insertions(+)
 create mode 100755 scripts/glimpse.sh

diff --git a/.gitignore b/.gitignore
index 2be25f771bd8..e6b40a6f3a7a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -113,3 +113,6 @@ all.config
 
 # Kdevelop4
 *.kdev4
+
+# Glimpse
+.glimpse_*
diff --git a/scripts/glimpse.sh b/scripts/glimpse.sh
new file mode 100755
index 000000000000..2e866228a88f
--- /dev/null
+++ b/scripts/glimpse.sh
@@ -0,0 +1,11 @@
+#!/bin/bash
+
+DIR="$(dirname $(readlink -f $0))/.."
+
+GLIMPSEINDEX="`which ${GLIMPSEINDEX:=glimpseindex}`"
+if [ ! -x "$GLIMPSEINDEX" ]; then
+	echo 'glimpseindex can be obtained at https://github.com/mcgrof/glimpse.git'
+	exit 1
+fi
+
+find $DIR/* -name "*.[ch]" | $GLIMPSEINDEX -o -H . -F
-- 
2.8.2

[Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

Glimpse is a tool you can use to index the kernel. The tool
was open sourced on 2014-09-26 [0] under the ISC license however
the original release still [1] does not compile. A fix for this
and a release that does compile is provided in a temporary
tree [2].

v2 changesl:
o simplify DIR in one line and add verbose release information
  to commit log about the open sourcing of glimpse and its
  current status.
o add index files to .gitignore

[0] http://webglimpse.net/
[1] https://github.com/gvelez17/glimpse/
[2] https://github.com/mcgrof/glimpse.git

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 .gitignore         |  3 +++
 scripts/glimpse.sh | 11 +++++++++++
 2 files changed, 14 insertions(+)
 create mode 100755 scripts/glimpse.sh

diff --git a/.gitignore b/.gitignore
index 2be25f771bd8..e6b40a6f3a7a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -113,3 +113,6 @@ all.config
 
 # Kdevelop4
 *.kdev4
+
+# Glimpse
+.glimpse_*
diff --git a/scripts/glimpse.sh b/scripts/glimpse.sh
new file mode 100755
index 000000000000..2e866228a88f
--- /dev/null
+++ b/scripts/glimpse.sh
@@ -0,0 +1,11 @@
+#!/bin/bash
+
+DIR="$(dirname $(readlink -f $0))/.."
+
+GLIMPSEINDEX="`which ${GLIMPSEINDEX:=glimpseindex}`"
+if [ ! -x "$GLIMPSEINDEX" ]; then
+	echo 'glimpseindex can be obtained at https://github.com/mcgrof/glimpse.git'
+	exit 1
+fi
+
+find $DIR/* -name "*.[ch]" | $GLIMPSEINDEX -o -H . -F
-- 
2.8.2

[PATCH v2 5/8] coccicheck: replace --very-quiet with --quit when debugging

[Luis R. Rodriguez]

When debugging (using --profile or --show-trying) you want to
avoid supressing output,  use --quiet instead. While at it, extend
documentation for SPFLAGS use.

For instance one can use:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ make coccicheck DEBUG_FILE="poo.err" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c

v2: use egrep instead of the *"=--option"* check, this doesn't work for
    disjunctions.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 30f5a531ad34..f3a7d9fe9bc9 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -159,7 +159,25 @@ else
 	NPROC="$J"
 fi
 
-FLAGS="$SPFLAGS --very-quiet"
+# You can use SPFLAGS to append extra arguments to coccicheck.
+# A good example is if you want to debug your cocci script, you can
+# for instance use the following:
+#
+# $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
+# $ time make coccicheck MODE=report DEBUG_FILE="all.err" SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c
+#
+# "--show-trying" should show you what rule is being processed as it goes to
+# stdout, you do not need a debug file for that. The profile output will be
+# be sent to stdout, if you provide a DEBUG_FILE the profiling data can be
+# inspected there.
+#
+# --profile will not output if --very-quiet is used, so avoid it.
+echo $SPFLAGS | egrep -e "--profile|--show-trying" 2>&1 > /dev/null
+if [ $? -eq 0 ]; then
+	FLAGS="--quiet $SPFLAGS"
+else
+	FLAGS="--very-quiet $SPFLAGS"
+fi
 
 # spatch only allows include directories with the syntax "-I include"
 # while gcc also allows "-Iinclude" and "-include include"
-- 
2.8.2

[Cocci] [PATCH v2 5/8] coccicheck: replace --very-quiet with --quit when debugging

[Luis R. Rodriguez]

When debugging (using --profile or --show-trying) you want to
avoid supressing output,  use --quiet instead. While at it, extend
documentation for SPFLAGS use.

For instance one can use:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ make coccicheck DEBUG_FILE="poo.err" MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c

v2: use egrep instead of the *"=--option"* check, this doesn't work for
    disjunctions.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 30f5a531ad34..f3a7d9fe9bc9 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -159,7 +159,25 @@ else
 	NPROC="$J"
 fi
 
-FLAGS="$SPFLAGS --very-quiet"
+# You can use SPFLAGS to append extra arguments to coccicheck.
+# A good example is if you want to debug your cocci script, you can
+# for instance use the following:
+#
+# $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
+# $ time make coccicheck MODE=report DEBUG_FILE="all.err" SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c
+#
+# "--show-trying" should show you what rule is being processed as it goes to
+# stdout, you do not need a debug file for that. The profile output will be
+# be sent to stdout, if you provide a DEBUG_FILE the profiling data can be
+# inspected there.
+#
+# --profile will not output if --very-quiet is used, so avoid it.
+echo $SPFLAGS | egrep -e "--profile|--show-trying" 2>&1 > /dev/null
+if [ $? -eq 0 ]; then
+	FLAGS="--quiet $SPFLAGS"
+else
+	FLAGS="--very-quiet $SPFLAGS"
+fi
 
 # spatch only allows include directories with the syntax "-I include"
 # while gcc also allows "-Iinclude" and "-include include"
-- 
2.8.2

[PATCH v2 6/8] coccicheck: add support for requring a coccinelle version

[Luis R. Rodriguez]

Enable Coccinelle SmPL patches to require a specific version of
Coccinelle. In the event that the version does not match we just
inform the user, if the user asked to go through all SmPL patches
we just inform them of the need for a new version of coccinelle for
the SmPL patch and continue on with the rest.

This uses the simple kernel scripts/ld-version.sh to create a weight
on the version provided by spatch. The -dirty attribute is ignored if
supplied, the benefit of scripts/ld-version.sh is it has a long history
and well tested.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index f3a7d9fe9bc9..268ea375273b 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -13,6 +13,9 @@ if [ ! -x "$SPATCH" ]; then
     exit 1
 fi
 
+SPATCH_VERSION=$($SPATCH --version | head -1 | awk '{print $3}')
+SPATCH_VERSION_NUM=$(echo $SPATCH_VERSION | ${DIR}/scripts/ld-version.sh)
+
 USE_JOBS="no"
 $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
@@ -312,6 +315,16 @@ coccinelle () {
     COCCI="$1"
 
     OPT=`grep "Option" $COCCI | cut -d':' -f2`
+    REQ=`grep "Requires" $COCCI | cut -d':' -f2 | sed "s| ||"`
+    REQ_NUM=$(echo $REQ | ${DIR}/scripts/ld-version.sh)
+    if [ "$REQ_NUM" != "0" ] ; then
+	    if [ "$SPATCH_VERSION_NUM" -lt "$REQ_NUM" ] ; then
+		    echo "Skipping coccinele SmPL patch: $COCCI"
+		    echo "You have coccinelle:           $SPATCH_VERSION"
+		    echo "This SmPL patch requires:      $REQ"
+		    return
+	    fi
+    fi
 
 #   The option '--parse-cocci' can be used to syntactically check the SmPL files.
 #
-- 
2.8.2

[Cocci] [PATCH v2 6/8] coccicheck: add support for requring a coccinelle version

[Luis R. Rodriguez]

Enable Coccinelle SmPL patches to require a specific version of
Coccinelle. In the event that the version does not match we just
inform the user, if the user asked to go through all SmPL patches
we just inform them of the need for a new version of coccinelle for
the SmPL patch and continue on with the rest.

This uses the simple kernel scripts/ld-version.sh to create a weight
on the version provided by spatch. The -dirty attribute is ignored if
supplied, the benefit of scripts/ld-version.sh is it has a long history
and well tested.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index f3a7d9fe9bc9..268ea375273b 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -13,6 +13,9 @@ if [ ! -x "$SPATCH" ]; then
     exit 1
 fi
 
+SPATCH_VERSION=$($SPATCH --version | head -1 | awk '{print $3}')
+SPATCH_VERSION_NUM=$(echo $SPATCH_VERSION | ${DIR}/scripts/ld-version.sh)
+
 USE_JOBS="no"
 $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
@@ -312,6 +315,16 @@ coccinelle () {
     COCCI="$1"
 
     OPT=`grep "Option" $COCCI | cut -d':' -f2`
+    REQ=`grep "Requires" $COCCI | cut -d':' -f2 | sed "s| ||"`
+    REQ_NUM=$(echo $REQ | ${DIR}/scripts/ld-version.sh)
+    if [ "$REQ_NUM" != "0" ] ; then
+	    if [ "$SPATCH_VERSION_NUM" -lt "$REQ_NUM" ] ; then
+		    echo "Skipping coccinele SmPL patch: $COCCI"
+		    echo "You have coccinelle:           $SPATCH_VERSION"
+		    echo "This SmPL patch requires:      $REQ"
+		    return
+	    fi
+    fi
 
 #   The option '--parse-cocci' can be used to syntactically check the SmPL files.
 #
-- 
2.8.2

[PATCH v2 7/8] coccicheck: refer to coccicheck bottest wiki for documentation

[Luis R. Rodriguez]

Sprinkling *tons* of documentation on the script is not a good
idea, instead refer to a wiki for further coccicheck documentation:

https://bottest.wiki.kernel.org/coccicheck

This page shall always refer to the linux-next iteration of
scripts/coccicheck.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 268ea375273b..46aeaad2847e 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -1,9 +1,14 @@
 #!/bin/bash
-
+# Linux kernel coccicheck
+#
+# For more detailed documentation refer to:
+#
+# https://bottest.wiki.kernel.org/coccicheck
+#
+# This documentation always refers to the linux-next version of the script.
 #
 # This script requires at least spatch
 # version 1.0.0-rc11.
-#
 
 DIR="$(dirname $(readlink -f $0))/.."
 SPATCH="`which ${SPATCH:=spatch}`"
-- 
2.8.2

[Cocci] [PATCH v2 7/8] coccicheck: refer to coccicheck bottest wiki for documentation

[Luis R. Rodriguez]

Sprinkling *tons* of documentation on the script is not a good
idea, instead refer to a wiki for further coccicheck documentation:

https://bottest.wiki.kernel.org/coccicheck

This page shall always refer to the linux-next iteration of
scripts/coccicheck.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 268ea375273b..46aeaad2847e 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -1,9 +1,14 @@
 #!/bin/bash
-
+# Linux kernel coccicheck
+#
+# For more detailed documentation refer to:
+#
+# https://bottest.wiki.kernel.org/coccicheck
+#
+# This documentation always refers to the linux-next version of the script.
 #
 # This script requires at least spatch
 # version 1.0.0-rc11.
-#
 
 DIR="$(dirname $(readlink -f $0))/.."
 SPATCH="`which ${SPATCH:=spatch}`"
-- 
2.8.2

[PATCH v2 8/8] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Luis R. Rodriguez]

Make use of the new Requires: tag to be able to specify coccinelle binary
version requirements. The cocci file device_node_continue.cocci requires at
least coccinelle 1.0.4.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccinelle/iterators/device_node_continue.cocci | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/coccinelle/iterators/device_node_continue.cocci b/scripts/coccinelle/iterators/device_node_continue.cocci
index 38ab744a4037..a36c16db171b 100644
--- a/scripts/coccinelle/iterators/device_node_continue.cocci
+++ b/scripts/coccinelle/iterators/device_node_continue.cocci
@@ -5,8 +5,11 @@
 // Copyright: (C) 2015 Julia Lawall, Inria. GPLv2.
 // URL: http://coccinelle.lip6.fr/
 // Options: --no-includes --include-headers
+// Requires: 1.0.4
 // Keywords: for_each_child_of_node, etc.
 
+// This uses a conjunction, which requires at least coccinelle >= 1.0.4
+
 virtual patch
 virtual context
 virtual org
-- 
2.8.2

[Cocci] [PATCH v2 8/8] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci

[Luis R. Rodriguez]

Make use of the new Requires: tag to be able to specify coccinelle binary
version requirements. The cocci file device_node_continue.cocci requires at
least coccinelle 1.0.4.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccinelle/iterators/device_node_continue.cocci | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/coccinelle/iterators/device_node_continue.cocci b/scripts/coccinelle/iterators/device_node_continue.cocci
index 38ab744a4037..a36c16db171b 100644
--- a/scripts/coccinelle/iterators/device_node_continue.cocci
+++ b/scripts/coccinelle/iterators/device_node_continue.cocci
@@ -5,8 +5,11 @@
 // Copyright: (C) 2015 Julia Lawall, Inria. GPLv2.
 // URL: http://coccinelle.lip6.fr/
 // Options: --no-includes --include-headers
+// Requires: 1.0.4
 // Keywords: for_each_child_of_node, etc.
 
+// This uses a conjunction, which requires at least coccinelle >= 1.0.4
+
 virtual patch
 virtual context
 virtual org
-- 
2.8.2

Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1034 bytes --]

I'm not sure that this is worth it.  It adds a dependency on a tool that
seems not to be well maintained.  In terms of Coccinelle, I'm not sure
that it gives a big benefit.

Attached is a graph showing the file selection time for Coccinelle for a
selection of fairly complex semantic patches.  Coccigrep is just a
line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
In most cases, glimpse is clearly faster.

On the other hand, it seems that glimpse often selects more files.
Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
vs 236.  I suspect that this is because glimpse considers _ to be a space,
and thus it can have many false positives.  There are, however, a few
cases where glimpse also selects fewer files.

The file processing time (ie parsing the file, searching for, matches of
the semantic patch in the file, and performing the transformation) is
normally much higher than the file selection time.

So it seems that git grep is currently a better option for the kernel.

julia

[-- Attachment #2: Type: APPLICATION/PDF, Size: 16124 bytes --]

[Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

I'm not sure that this is worth it.  It adds a dependency on a tool that
seems not to be well maintained.  In terms of Coccinelle, I'm not sure
that it gives a big benefit.

Attached is a graph showing the file selection time for Coccinelle for a
selection of fairly complex semantic patches.  Coccigrep is just a
line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
In most cases, glimpse is clearly faster.

On the other hand, it seems that glimpse often selects more files.
Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
vs 236.  I suspect that this is because glimpse considers _ to be a space,
and thus it can have many false positives.  There are, however, a few
cases where glimpse also selects fewer files.

The file processing time (ie parsing the file, searching for, matches of
the semantic patch in the file, and performing the transformation) is
normally much higher than the file selection time.

So it seems that git grep is currently a better option for the kernel.

julia
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fl.pdf
Type: application/pdf
Size: 16124 bytes
Desc: 
URL: <https://systeme.lip6.fr/pipermail/cocci/attachments/20160617/4455aab3/attachment-0001.pdf>

Re: [PATCH v2 3/8] coccicheck: add indexing enhancement options

[Julia Lawall]

On Thu, 16 Jun 2016, Luis R. Rodriguez wrote:

> Coccinelle has support to make use of its own enhanced "grep"
> mechanisms instead of using regular grep for searching code,
> it calls this 'coccigrep'. In lack of any indexing optimization
> information it uses --use-coccigrep by default.
>
> This patch enable indexing optimizations heuristics so that coccigrep
> can automatically detect what indexing options are available and use
> them accordinly without any user input.
>
> Since git has its own index, support for using 'git grep' has been
> added to Coccinelle, that should on average perform better than
> using the internal coccigrep. Coccinelle has had idutils support
> as well for a while now, you however need to refer to the index
> file. We support detecting two idutils index files by default,
> ID and .id-utils.index, assuming you ran either of:
>
>  # What you might have done:
> mkid -s
>  # as in coccinelle scripts/idutils_index.sh
> mkid -i C --output .id-utils.index *
>
> Lastly, Coccinelle has had support for glimpseindex for a long while,
> however the glimpseindex tool, the agrep library were previously closed
> source, its all now open sourced, and provides the best performance, so
> support that if we can detect you have a glimpse index.
>
> You can always override the index as follows:
>
> $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"

Why not just have a generic COCCI_ARGS argument?

julia


> These tests have been run on an 8 core system:
>
> Before:
>
> $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
> $ time make coccicheck MODE=report
>
> coccigrep (default and without this patch):
> real    0m16.369s
> user    0m58.712s
> sys     0m5.064s
>
> After:
>
> $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
> $ time make coccicheck MODE=report
>
> With glimpse:
> real    0m6.549s
> user    0m49.136s
> sys     0m3.076s
>
> With idutils:
> real    0m6.749s
> user    0m51.936s
> sys     0m3.876s
>
> With gitgrep:
> real    0m6.805s
> user    0m51.572s
> sys     0m4.432s
>
> v2 changes:
>
> o simplify DIR assignment to 1 line
> o detected a bug when KBUILD_EXTMOD is used other than the parent
>   directory for both glimpse and idutils, so we avoid both when
>   M=path/driver/ is used. This is being looked into upstream on
>   Coccinelle.
> o move indexing heuristics to a file
> o document logic used for indexing
> o add idutils support, supports two indexing files
> o remove coccigrep heuristics as its the default anyway
> o add COCCI_INDEX to enable overriding heuristics, you can use this
>   as follows, for example:
> o replace references to stderr file with DEBUG_FILE use instructions
>
> $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
> $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
> $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
> $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
> $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"
>
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>  scripts/coccicheck | 150 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 150 insertions(+)
>
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 7acef3efc258..30f5a531ad34 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -5,6 +5,7 @@
>  # version 1.0.0-rc11.
>  #
>
> +DIR="$(dirname $(readlink -f $0))/.."
>  SPATCH="`which ${SPATCH:=spatch}`"
>
>  if [ ! -x "$SPATCH" ]; then
> @@ -15,6 +16,134 @@ fi
>  USE_JOBS="no"
>  $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
>
> +function can_use_glimpse()
> +{
> +	$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null
> +	if [ $? -ne 0 ]; then
> +		echo "no"
> +		return
> +	fi
> +	if [ ! -f $DIR/.glimpse_index ]; then
> +		echo "no"
> +		return
> +	fi
> +
> +	# As of coccinelle 1.0.5 --use-glimpse cannot be used with M= other
> +	# than the base directory. We expect a future release will let us
> +	# specify the full path of the glimpse index but even if that's
> +	# supported, glimpse use seems to require an index per every
> +	# directory parsed, so you'd have to generate a glimpse index
> +	# per directory. If M=path is used (where epath is not the top level)
> +	# we'll have to fallback to alternatives then.
> +	if [ "$KBUILD_EXTMOD" != "./" -a "$KBUILD_EXTMOD" != "" ]; then
> +		echo "no"
> +		return
> +	fi
> +	echo yes
> +}
> +
> +function can_use_idutils()
> +{
> +	$SPATCH --help | grep "\-\-use\-idutils" > /dev/null
> +	if [ $? -ne 0 ]; then
> +		echo "no"
> +		return
> +	fi
> +	# As of coccinelle 1.0.5 --use-idutils will bust if one uses
> +	# idutils with an index out of the main tree.
> +	if [ "$KBUILD_EXTMOD" != "./" -a "$KBUILD_EXTMOD" != "" ]; then
> +		echo "no"
> +		return
> +	fi
> +	if [ -f $DIR/ID -o -f $DIR/.id-utils.index ]; then
> +		echo "yes"
> +		return
> +	fi
> +	echo "no"
> +}
> +
> +function can_use_gitgrep()
> +{
> +	$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null
> +	if [ $? -ne 0 ]; then
> +		echo "no"
> +		return
> +	fi
> +	if [ ! -d $DIR/.git ]; then
> +		echo "no"
> +		return
> +	fi
> +	echo "yes"
> +}
> +
> +# Indexing USE_* optimizations heuristics.
> +#
> +# Linux runs on git (TM). However, if you have supplemental indexing options
> +# you may use them to help Coccinelle further. If you are using Coccinelle
> +# within a target git tree --use-gitrep will be used, and this should
> +# suffice for most uses. If you however want optimal performance do
> +# consider embracing a supplemental indexing as part of your development.
> +# For instance glimpse, and idutils can be used, however you should
> +# be sure to update the indexes as often as you update your git tree to
> +# ensure your indexes are not stale.
> +#
> +# idutils is currently not as efficient as glimpse because the query language
> +# for glimpse is simpler, so when idutils is used more filtering has to be
> +# done at the ocaml level within Coccinelle. Glimpse allows queries that are
> +# arbitrary formulas, up to a limited level of complexity, involving both
> +# && and ||. For idutils, Coccinelle runs lid intersections on the result.
> +#
> +# You can override these heuristics with COCCI_INDEX="--use-gitgrep" for
> +# example. This will force to use --use-gitgrep even if you have a glimpse
> +# index. Other examples:
> +#
> +# $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
> +# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
> +# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
> +# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
> +# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"
> +#
> +# The order of heuristics for indexing used by coccicheck is listed below.
> +#
> +# 0. Glimpse currently should outperform all indexing options. so if a glimpse
> +#    index is used we use it. Refer to Linux scripts/glimpse.sh for details.
> +#    If you think you should be getting better performance with glimpse than
> +#    what you would expect inspect the stderr log file for cocciecheck, you
> +#    ask for a debug file with DEBUG_FILE="" parameter to coccicheck.
> +#
> +#    If glimpse is running correctly there should be very few occurrences
> +#    of "Skipping", also coccinelle will inform you if it could not use
> +#    glimpse. As an example an output of the following would indicate glimpse
> +#    was properly used on the stderr log file:
> +#
> +#    There are matches to 1252 out of 47281 files
> +#    glimpse request = request_threaded_irq
> +#
> +# 1. Use idutils next. You'll need to generate an index using either of these:
> +#
> +#	a) mkid -s
> +#       By default this dumps the index into ./ID
> +#
> +#	b) mkid -i C --output .id-utils.index *
> +#       This method is provided with coccinelle repo on
> +#       scripts/idutils_index.sh
> +#
> +# 2. Next best is --use-gitgrep and if you are working within a git tree
> +#    this will be used by default.
> +#
> +# 3. By default coccinelle internally uses --use-coccigrep if no indexing
> +#    options are requested and your version of coccinelle supports it so we
> +#    do not need to be specific about requesting that as a fallback mechanism.
> +#    Use of --use-coccigrep is comparable to --use-gitgrep.
> +#
> +# XXX: Glimpse is not well maintained. See if we can add similar indexing
> +# features and query language glimpse supports to git.
> +if [ "$COCCI_INDEX" = "" ] ; then
> +	USE_GLIMPSE=$(can_use_glimpse)
> +	USE_IDUTILS=$(can_use_idutils)
> +	USE_GITGREP=$(can_use_gitgrep)
> +fi
> +
>  # The verbosity may be set by the environmental parameter V=
>  # as for example with 'make V=1 coccicheck'
>
> @@ -89,6 +218,27 @@ else
>  	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
>  fi
>
> +# Check COCCI_INDEX first to manual override, otherwise rely on
> +# internal heuristics documented above.
> +if [ "$COCCI_INDEX" != "" ] ; then
> +	OPTIONS="$OPTIONS $COCCI_INDEX"
> +elif [ "$USE_GLIMPSE" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-glimpse"
> +elif [ "$USE_IDUTILS" = "yes" ]; then
> +	index=""
> +	if [ -f $DIR/ID ]; then
> +		index="$DIR/ID"
> +	elif [ -f $DIR/.id-utils.index ]; then
> +		index="$DIR/.id-utils.index"
> +	else
> +		echo "idutils index not found, expected: $DIR/ID or $DIR/.id-utils.index"
> +		exit 1
> +	fi
> +	OPTIONS="$OPTIONS --use-idutils $index"
> +elif [ "$USE_GITGREP" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-gitgrep"
> +fi
> +
>  run_cmd_parmap() {
>  	if [ $VERBOSE -ne 0 ] ; then
>  		echo "Running ($NPROC in parallel): $@"
> --
> 2.8.2
>
>

[Cocci] [PATCH v2 3/8] coccicheck: add indexing enhancement options

[Julia Lawall]

On Thu, 16 Jun 2016, Luis R. Rodriguez wrote:

> Coccinelle has support to make use of its own enhanced "grep"
> mechanisms instead of using regular grep for searching code,
> it calls this 'coccigrep'. In lack of any indexing optimization
> information it uses --use-coccigrep by default.
>
> This patch enable indexing optimizations heuristics so that coccigrep
> can automatically detect what indexing options are available and use
> them accordinly without any user input.
>
> Since git has its own index, support for using 'git grep' has been
> added to Coccinelle, that should on average perform better than
> using the internal coccigrep. Coccinelle has had idutils support
> as well for a while now, you however need to refer to the index
> file. We support detecting two idutils index files by default,
> ID and .id-utils.index, assuming you ran either of:
>
>  # What you might have done:
> mkid -s
>  # as in coccinelle scripts/idutils_index.sh
> mkid -i C --output .id-utils.index *
>
> Lastly, Coccinelle has had support for glimpseindex for a long while,
> however the glimpseindex tool, the agrep library were previously closed
> source, its all now open sourced, and provides the best performance, so
> support that if we can detect you have a glimpse index.
>
> You can always override the index as follows:
>
> $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"

Why not just have a generic COCCI_ARGS argument?

julia


> These tests have been run on an 8 core system:
>
> Before:
>
> $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
> $ time make coccicheck MODE=report
>
> coccigrep (default and without this patch):
> real    0m16.369s
> user    0m58.712s
> sys     0m5.064s
>
> After:
>
> $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
> $ time make coccicheck MODE=report
>
> With glimpse:
> real    0m6.549s
> user    0m49.136s
> sys     0m3.076s
>
> With idutils:
> real    0m6.749s
> user    0m51.936s
> sys     0m3.876s
>
> With gitgrep:
> real    0m6.805s
> user    0m51.572s
> sys     0m4.432s
>
> v2 changes:
>
> o simplify DIR assignment to 1 line
> o detected a bug when KBUILD_EXTMOD is used other than the parent
>   directory for both glimpse and idutils, so we avoid both when
>   M=path/driver/ is used. This is being looked into upstream on
>   Coccinelle.
> o move indexing heuristics to a file
> o document logic used for indexing
> o add idutils support, supports two indexing files
> o remove coccigrep heuristics as its the default anyway
> o add COCCI_INDEX to enable overriding heuristics, you can use this
>   as follows, for example:
> o replace references to stderr file with DEBUG_FILE use instructions
>
> $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
> $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
> $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
> $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
> $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"
>
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>  scripts/coccicheck | 150 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 150 insertions(+)
>
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 7acef3efc258..30f5a531ad34 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -5,6 +5,7 @@
>  # version 1.0.0-rc11.
>  #
>
> +DIR="$(dirname $(readlink -f $0))/.."
>  SPATCH="`which ${SPATCH:=spatch}`"
>
>  if [ ! -x "$SPATCH" ]; then
> @@ -15,6 +16,134 @@ fi
>  USE_JOBS="no"
>  $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
>
> +function can_use_glimpse()
> +{
> +	$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null
> +	if [ $? -ne 0 ]; then
> +		echo "no"
> +		return
> +	fi
> +	if [ ! -f $DIR/.glimpse_index ]; then
> +		echo "no"
> +		return
> +	fi
> +
> +	# As of coccinelle 1.0.5 --use-glimpse cannot be used with M= other
> +	# than the base directory. We expect a future release will let us
> +	# specify the full path of the glimpse index but even if that's
> +	# supported, glimpse use seems to require an index per every
> +	# directory parsed, so you'd have to generate a glimpse index
> +	# per directory. If M=path is used (where epath is not the top level)
> +	# we'll have to fallback to alternatives then.
> +	if [ "$KBUILD_EXTMOD" != "./" -a "$KBUILD_EXTMOD" != "" ]; then
> +		echo "no"
> +		return
> +	fi
> +	echo yes
> +}
> +
> +function can_use_idutils()
> +{
> +	$SPATCH --help | grep "\-\-use\-idutils" > /dev/null
> +	if [ $? -ne 0 ]; then
> +		echo "no"
> +		return
> +	fi
> +	# As of coccinelle 1.0.5 --use-idutils will bust if one uses
> +	# idutils with an index out of the main tree.
> +	if [ "$KBUILD_EXTMOD" != "./" -a "$KBUILD_EXTMOD" != "" ]; then
> +		echo "no"
> +		return
> +	fi
> +	if [ -f $DIR/ID -o -f $DIR/.id-utils.index ]; then
> +		echo "yes"
> +		return
> +	fi
> +	echo "no"
> +}
> +
> +function can_use_gitgrep()
> +{
> +	$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null
> +	if [ $? -ne 0 ]; then
> +		echo "no"
> +		return
> +	fi
> +	if [ ! -d $DIR/.git ]; then
> +		echo "no"
> +		return
> +	fi
> +	echo "yes"
> +}
> +
> +# Indexing USE_* optimizations heuristics.
> +#
> +# Linux runs on git (TM). However, if you have supplemental indexing options
> +# you may use them to help Coccinelle further. If you are using Coccinelle
> +# within a target git tree --use-gitrep will be used, and this should
> +# suffice for most uses. If you however want optimal performance do
> +# consider embracing a supplemental indexing as part of your development.
> +# For instance glimpse, and idutils can be used, however you should
> +# be sure to update the indexes as often as you update your git tree to
> +# ensure your indexes are not stale.
> +#
> +# idutils is currently not as efficient as glimpse because the query language
> +# for glimpse is simpler, so when idutils is used more filtering has to be
> +# done at the ocaml level within Coccinelle. Glimpse allows queries that are
> +# arbitrary formulas, up to a limited level of complexity, involving both
> +# && and ||. For idutils, Coccinelle runs lid intersections on the result.
> +#
> +# You can override these heuristics with COCCI_INDEX="--use-gitgrep" for
> +# example. This will force to use --use-gitgrep even if you have a glimpse
> +# index. Other examples:
> +#
> +# $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
> +# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
> +# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
> +# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
> +# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"
> +#
> +# The order of heuristics for indexing used by coccicheck is listed below.
> +#
> +# 0. Glimpse currently should outperform all indexing options. so if a glimpse
> +#    index is used we use it. Refer to Linux scripts/glimpse.sh for details.
> +#    If you think you should be getting better performance with glimpse than
> +#    what you would expect inspect the stderr log file for cocciecheck, you
> +#    ask for a debug file with DEBUG_FILE="" parameter to coccicheck.
> +#
> +#    If glimpse is running correctly there should be very few occurrences
> +#    of "Skipping", also coccinelle will inform you if it could not use
> +#    glimpse. As an example an output of the following would indicate glimpse
> +#    was properly used on the stderr log file:
> +#
> +#    There are matches to 1252 out of 47281 files
> +#    glimpse request = request_threaded_irq
> +#
> +# 1. Use idutils next. You'll need to generate an index using either of these:
> +#
> +#	a) mkid -s
> +#       By default this dumps the index into ./ID
> +#
> +#	b) mkid -i C --output .id-utils.index *
> +#       This method is provided with coccinelle repo on
> +#       scripts/idutils_index.sh
> +#
> +# 2. Next best is --use-gitgrep and if you are working within a git tree
> +#    this will be used by default.
> +#
> +# 3. By default coccinelle internally uses --use-coccigrep if no indexing
> +#    options are requested and your version of coccinelle supports it so we
> +#    do not need to be specific about requesting that as a fallback mechanism.
> +#    Use of --use-coccigrep is comparable to --use-gitgrep.
> +#
> +# XXX: Glimpse is not well maintained. See if we can add similar indexing
> +# features and query language glimpse supports to git.
> +if [ "$COCCI_INDEX" = "" ] ; then
> +	USE_GLIMPSE=$(can_use_glimpse)
> +	USE_IDUTILS=$(can_use_idutils)
> +	USE_GITGREP=$(can_use_gitgrep)
> +fi
> +
>  # The verbosity may be set by the environmental parameter V=
>  # as for example with 'make V=1 coccicheck'
>
> @@ -89,6 +218,27 @@ else
>  	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
>  fi
>
> +# Check COCCI_INDEX first to manual override, otherwise rely on
> +# internal heuristics documented above.
> +if [ "$COCCI_INDEX" != "" ] ; then
> +	OPTIONS="$OPTIONS $COCCI_INDEX"
> +elif [ "$USE_GLIMPSE" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-glimpse"
> +elif [ "$USE_IDUTILS" = "yes" ]; then
> +	index=""
> +	if [ -f $DIR/ID ]; then
> +		index="$DIR/ID"
> +	elif [ -f $DIR/.id-utils.index ]; then
> +		index="$DIR/.id-utils.index"
> +	else
> +		echo "idutils index not found, expected: $DIR/ID or $DIR/.id-utils.index"
> +		exit 1
> +	fi
> +	OPTIONS="$OPTIONS --use-idutils $index"
> +elif [ "$USE_GITGREP" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-gitgrep"
> +fi
> +
>  run_cmd_parmap() {
>  	if [ $VERBOSE -ne 0 ] ; then
>  		echo "Running ($NPROC in parallel): $@"
> --
> 2.8.2
>
>

Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote:
> I'm not sure that this is worth it.  It adds a dependency on a tool that
> seems not to be well maintained.  In terms of Coccinelle, I'm not sure
> that it gives a big benefit.
> 
> Attached is a graph showing the file selection time for Coccinelle for a
> selection of fairly complex semantic patches.  Coccigrep is just a
> line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
> In most cases, glimpse is clearly faster.
> 
> On the other hand, it seems that glimpse often selects more files.
> Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
> vs 236.  I suspect that this is because glimpse considers _ to be a space,
> and thus it can have many false positives.  There are, however, a few
> cases where glimpse also selects fewer files.
> 
> The file processing time (ie parsing the file, searching for, matches of
> the semantic patch in the file, and performing the transformation) is
> normally much higher than the file selection time.
> 
> So it seems that git grep is currently a better option for the kernel.

Great, thanks, consider this patch dropped, do we want the heuristics
for the cache index in place though or should I drop that as well ?

  Luis

[Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote:
> I'm not sure that this is worth it.  It adds a dependency on a tool that
> seems not to be well maintained.  In terms of Coccinelle, I'm not sure
> that it gives a big benefit.
> 
> Attached is a graph showing the file selection time for Coccinelle for a
> selection of fairly complex semantic patches.  Coccigrep is just a
> line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
> In most cases, glimpse is clearly faster.
> 
> On the other hand, it seems that glimpse often selects more files.
> Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
> vs 236.  I suspect that this is because glimpse considers _ to be a space,
> and thus it can have many false positives.  There are, however, a few
> cases where glimpse also selects fewer files.
> 
> The file processing time (ie parsing the file, searching for, matches of
> the semantic patch in the file, and performing the transformation) is
> normally much higher than the file selection time.
> 
> So it seems that git grep is currently a better option for the kernel.

Great, thanks, consider this patch dropped, do we want the heuristics
for the cache index in place though or should I drop that as well ?

  Luis

Re: [PATCH v2 3/8] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Fri, Jun 17, 2016 at 11:47:25AM +0200, Julia Lawall wrote:
> 
> 
> On Thu, 16 Jun 2016, Luis R. Rodriguez wrote:
> 
> > Coccinelle has support to make use of its own enhanced "grep"
> > mechanisms instead of using regular grep for searching code,
> > it calls this 'coccigrep'. In lack of any indexing optimization
> > information it uses --use-coccigrep by default.
> >
> > This patch enable indexing optimizations heuristics so that coccigrep
> > can automatically detect what indexing options are available and use
> > them accordinly without any user input.
> >
> > Since git has its own index, support for using 'git grep' has been
> > added to Coccinelle, that should on average perform better than
> > using the internal coccigrep. Coccinelle has had idutils support
> > as well for a while now, you however need to refer to the index
> > file. We support detecting two idutils index files by default,
> > ID and .id-utils.index, assuming you ran either of:
> >
> >  # What you might have done:
> > mkid -s
> >  # as in coccinelle scripts/idutils_index.sh
> > mkid -i C --output .id-utils.index *
> >
> > Lastly, Coccinelle has had support for glimpseindex for a long while,
> > however the glimpseindex tool, the agrep library were previously closed
> > source, its all now open sourced, and provides the best performance, so
> > support that if we can detect you have a glimpse index.
> >
> > You can always override the index as follows:
> >
> > $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
> 
> Why not just have a generic COCCI_ARGS argument?

Actually SPFLAGS exists already so we can just document to
override using that. Will fix.

  Luis

[Cocci] [PATCH v2 3/8] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Fri, Jun 17, 2016 at 11:47:25AM +0200, Julia Lawall wrote:
> 
> 
> On Thu, 16 Jun 2016, Luis R. Rodriguez wrote:
> 
> > Coccinelle has support to make use of its own enhanced "grep"
> > mechanisms instead of using regular grep for searching code,
> > it calls this 'coccigrep'. In lack of any indexing optimization
> > information it uses --use-coccigrep by default.
> >
> > This patch enable indexing optimizations heuristics so that coccigrep
> > can automatically detect what indexing options are available and use
> > them accordinly without any user input.
> >
> > Since git has its own index, support for using 'git grep' has been
> > added to Coccinelle, that should on average perform better than
> > using the internal coccigrep. Coccinelle has had idutils support
> > as well for a while now, you however need to refer to the index
> > file. We support detecting two idutils index files by default,
> > ID and .id-utils.index, assuming you ran either of:
> >
> >  # What you might have done:
> > mkid -s
> >  # as in coccinelle scripts/idutils_index.sh
> > mkid -i C --output .id-utils.index *
> >
> > Lastly, Coccinelle has had support for glimpseindex for a long while,
> > however the glimpseindex tool, the agrep library were previously closed
> > source, its all now open sourced, and provides the best performance, so
> > support that if we can detect you have a glimpse index.
> >
> > You can always override the index as follows:
> >
> > $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
> 
> Why not just have a generic COCCI_ARGS argument?

Actually SPFLAGS exists already so we can just document to
override using that. Will fix.

  Luis

Re: [Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[SF Markus Elfring]

> I'm not sure that this is worth it.  It adds a dependency on a tool
> that seems not to be well maintained.

Would any developers like to achieve further software improvements by
additional means?
How are the chances for corresponding progress from an approach like in
a repository by Luis?
https://github.com/mcgrof/glimpse


> In terms of Coccinelle, I'm not sure that it gives a big benefit.

Under which circumstances will the indexing efforts pay off?


> Attached is a graph showing the file selection time for Coccinelle for a
> selection of fairly complex semantic patches.

Would you like to share more information about test cases for this report?
Which scripts measured the shown data points?

Regards,
Markus

[Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[SF Markus Elfring]

> I'm not sure that this is worth it.  It adds a dependency on a tool
> that seems not to be well maintained.

Would any developers like to achieve further software improvements by
additional means?
How are the chances for corresponding progress from an approach like in
a repository by Luis?
https://github.com/mcgrof/glimpse


> In terms of Coccinelle, I'm not sure that it gives a big benefit.

Under which circumstances will the indexing efforts pay off?


> Attached is a graph showing the file selection time for Coccinelle for a
> selection of fairly complex semantic patches.

Would you like to share more information about test cases for this report?
Which scripts measured the shown data points?

Regards,
Markus

Re: [PATCH v2 3/8] coccicheck: add indexing enhancement options

[Julia Lawall]

On Fri, 17 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 17, 2016 at 11:47:25AM +0200, Julia Lawall wrote:
> >
> >
> > On Thu, 16 Jun 2016, Luis R. Rodriguez wrote:
> >
> > > Coccinelle has support to make use of its own enhanced "grep"
> > > mechanisms instead of using regular grep for searching code,
> > > it calls this 'coccigrep'. In lack of any indexing optimization
> > > information it uses --use-coccigrep by default.
> > >
> > > This patch enable indexing optimizations heuristics so that coccigrep
> > > can automatically detect what indexing options are available and use
> > > them accordinly without any user input.
> > >
> > > Since git has its own index, support for using 'git grep' has been
> > > added to Coccinelle, that should on average perform better than
> > > using the internal coccigrep. Coccinelle has had idutils support
> > > as well for a while now, you however need to refer to the index
> > > file. We support detecting two idutils index files by default,
> > > ID and .id-utils.index, assuming you ran either of:
> > >
> > >  # What you might have done:
> > > mkid -s
> > >  # as in coccinelle scripts/idutils_index.sh
> > > mkid -i C --output .id-utils.index *
> > >
> > > Lastly, Coccinelle has had support for glimpseindex for a long while,
> > > however the glimpseindex tool, the agrep library were previously closed
> > > source, its all now open sourced, and provides the best performance, so
> > > support that if we can detect you have a glimpse index.
> > >
> > > You can always override the index as follows:
> > >
> > > $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
> >
> > Why not just have a generic COCCI_ARGS argument?
>
> Actually SPFLAGS exists already so we can just document to
> override using that. Will fix.

That seems like a good solution.

julia

[Cocci] [PATCH v2 3/8] coccicheck: add indexing enhancement options

[Julia Lawall]

On Fri, 17 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 17, 2016 at 11:47:25AM +0200, Julia Lawall wrote:
> >
> >
> > On Thu, 16 Jun 2016, Luis R. Rodriguez wrote:
> >
> > > Coccinelle has support to make use of its own enhanced "grep"
> > > mechanisms instead of using regular grep for searching code,
> > > it calls this 'coccigrep'. In lack of any indexing optimization
> > > information it uses --use-coccigrep by default.
> > >
> > > This patch enable indexing optimizations heuristics so that coccigrep
> > > can automatically detect what indexing options are available and use
> > > them accordinly without any user input.
> > >
> > > Since git has its own index, support for using 'git grep' has been
> > > added to Coccinelle, that should on average perform better than
> > > using the internal coccigrep. Coccinelle has had idutils support
> > > as well for a while now, you however need to refer to the index
> > > file. We support detecting two idutils index files by default,
> > > ID and .id-utils.index, assuming you ran either of:
> > >
> > >  # What you might have done:
> > > mkid -s
> > >  # as in coccinelle scripts/idutils_index.sh
> > > mkid -i C --output .id-utils.index *
> > >
> > > Lastly, Coccinelle has had support for glimpseindex for a long while,
> > > however the glimpseindex tool, the agrep library were previously closed
> > > source, its all now open sourced, and provides the best performance, so
> > > support that if we can detect you have a glimpse index.
> > >
> > > You can always override the index as follows:
> > >
> > > $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
> >
> > Why not just have a generic COCCI_ARGS argument?
>
> Actually SPFLAGS exists already so we can just document to
> override using that. Will fix.

That seems like a good solution.

julia

Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

On Fri, 17 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote:
> > I'm not sure that this is worth it.  It adds a dependency on a tool that
> > seems not to be well maintained.  In terms of Coccinelle, I'm not sure
> > that it gives a big benefit.
> >
> > Attached is a graph showing the file selection time for Coccinelle for a
> > selection of fairly complex semantic patches.  Coccigrep is just a
> > line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
> > In most cases, glimpse is clearly faster.
> >
> > On the other hand, it seems that glimpse often selects more files.
> > Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
> > vs 236.  I suspect that this is because glimpse considers _ to be a space,
> > and thus it can have many false positives.  There are, however, a few
> > cases where glimpse also selects fewer files.
> >
> > The file processing time (ie parsing the file, searching for, matches of
> > the semantic patch in the file, and performing the transformation) is
> > normally much higher than the file selection time.
> >
> > So it seems that git grep is currently a better option for the kernel.
>
> Great, thanks, consider this patch dropped, do we want the heuristics
> for the cache index in place though or should I drop that as well ?

I assume you mean this patch:

[PATCH v2 3/8] coccicheck: add indexing enhancement options

I think it should be dropped.  It adds complexity and git grep works
pretty well.  If people want to use something else, they can use SPARGS,
or a .cocciconfig file, eg:

[spatch]
options = --use-glimpse

julia

[Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

On Fri, 17 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote:
> > I'm not sure that this is worth it.  It adds a dependency on a tool that
> > seems not to be well maintained.  In terms of Coccinelle, I'm not sure
> > that it gives a big benefit.
> >
> > Attached is a graph showing the file selection time for Coccinelle for a
> > selection of fairly complex semantic patches.  Coccigrep is just a
> > line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
> > In most cases, glimpse is clearly faster.
> >
> > On the other hand, it seems that glimpse often selects more files.
> > Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
> > vs 236.  I suspect that this is because glimpse considers _ to be a space,
> > and thus it can have many false positives.  There are, however, a few
> > cases where glimpse also selects fewer files.
> >
> > The file processing time (ie parsing the file, searching for, matches of
> > the semantic patch in the file, and performing the transformation) is
> > normally much higher than the file selection time.
> >
> > So it seems that git grep is currently a better option for the kernel.
>
> Great, thanks, consider this patch dropped, do we want the heuristics
> for the cache index in place though or should I drop that as well ?

I assume you mean this patch:

[PATCH v2 3/8] coccicheck: add indexing enhancement options

I think it should be dropped.  It adds complexity and git grep works
pretty well.  If people want to use something else, they can use SPARGS,
or a .cocciconfig file, eg:

[spatch]
options = --use-glimpse

julia

Re: [Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

> Under which circumstances will the indexing efforts pay off?

Indexing is beneficial when a semantic patch uses in an essential way some
words that appear rarely in the code base.  For example, the following
semantic patch will only do something useful on code that contains xyz:

- x = kmalloc();
  ..
- x = xyz();

There is no need to parse the code, match the thousands of occurrences of
kmalloc, nor to do the tracing of execution paths implied by ... if the
code does not contain any call to xyz anyway.

julia

[Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

> Under which circumstances will the indexing efforts pay off?

Indexing is beneficial when a semantic patch uses in an essential way some
words that appear rarely in the code base.  For example, the following
semantic patch will only do something useful on code that contains xyz:

- x = kmalloc();
  ..
- x = xyz();

There is no need to parse the code, match the thousands of occurrences of
kmalloc, nor to do the tracing of execution paths implied by ... if the
code does not contain any call to xyz anyway.

julia

Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

On Fri, Jun 17, 2016 at 05:35:26PM +0200, Julia Lawall wrote:
> On Fri, 17 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote:
> > > I'm not sure that this is worth it.  It adds a dependency on a tool that
> > > seems not to be well maintained.  In terms of Coccinelle, I'm not sure
> > > that it gives a big benefit.
> > >
> > > Attached is a graph showing the file selection time for Coccinelle for a
> > > selection of fairly complex semantic patches.  Coccigrep is just a
> > > line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
> > > In most cases, glimpse is clearly faster.
> > >
> > > On the other hand, it seems that glimpse often selects more files.
> > > Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
> > > vs 236.  I suspect that this is because glimpse considers _ to be a space,
> > > and thus it can have many false positives.  There are, however, a few
> > > cases where glimpse also selects fewer files.
> > >
> > > The file processing time (ie parsing the file, searching for, matches of
> > > the semantic patch in the file, and performing the transformation) is
> > > normally much higher than the file selection time.
> > >
> > > So it seems that git grep is currently a better option for the kernel.
> >
> > Great, thanks, consider this patch dropped, do we want the heuristics
> > for the cache index in place though or should I drop that as well ?
> 
> I assume you mean this patch:
> 
> [PATCH v2 3/8] coccicheck: add indexing enhancement options
> 
> I think it should be dropped.  It adds complexity and git grep works
> pretty well.

Hmm but coccicheck does not make use of --git-grep even.

> If people want to use something else, they can use SPARGS,
> or a .cocciconfig file, eg:
> 
> [spatch]
> options = --use-glimpse

Neat will these be used last and thus override anything? If so, what about just
adding an upstream .cocciconfig with --use-gitgrep -- only issue then is
what if a user wants to use idutils ? How do we let them override?

  Luis

[Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

On Fri, Jun 17, 2016 at 05:35:26PM +0200, Julia Lawall wrote:
> On Fri, 17 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote:
> > > I'm not sure that this is worth it.  It adds a dependency on a tool that
> > > seems not to be well maintained.  In terms of Coccinelle, I'm not sure
> > > that it gives a big benefit.
> > >
> > > Attached is a graph showing the file selection time for Coccinelle for a
> > > selection of fairly complex semantic patches.  Coccigrep is just a
> > > line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
> > > In most cases, glimpse is clearly faster.
> > >
> > > On the other hand, it seems that glimpse often selects more files.
> > > Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
> > > vs 236.  I suspect that this is because glimpse considers _ to be a space,
> > > and thus it can have many false positives.  There are, however, a few
> > > cases where glimpse also selects fewer files.
> > >
> > > The file processing time (ie parsing the file, searching for, matches of
> > > the semantic patch in the file, and performing the transformation) is
> > > normally much higher than the file selection time.
> > >
> > > So it seems that git grep is currently a better option for the kernel.
> >
> > Great, thanks, consider this patch dropped, do we want the heuristics
> > for the cache index in place though or should I drop that as well ?
> 
> I assume you mean this patch:
> 
> [PATCH v2 3/8] coccicheck: add indexing enhancement options
> 
> I think it should be dropped.  It adds complexity and git grep works
> pretty well.

Hmm but coccicheck does not make use of --git-grep even.

> If people want to use something else, they can use SPARGS,
> or a .cocciconfig file, eg:
> 
> [spatch]
> options = --use-glimpse

Neat will these be used last and thus override anything? If so, what about just
adding an upstream .cocciconfig with --use-gitgrep -- only issue then is
what if a user wants to use idutils ? How do we let them override?

  Luis

Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

On Sat, 18 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 17, 2016 at 05:35:26PM +0200, Julia Lawall wrote:
> > On Fri, 17 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote:
> > > > I'm not sure that this is worth it.  It adds a dependency on a tool that
> > > > seems not to be well maintained.  In terms of Coccinelle, I'm not sure
> > > > that it gives a big benefit.
> > > >
> > > > Attached is a graph showing the file selection time for Coccinelle for a
> > > > selection of fairly complex semantic patches.  Coccigrep is just a
> > > > line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
> > > > In most cases, glimpse is clearly faster.
> > > >
> > > > On the other hand, it seems that glimpse often selects more files.
> > > > Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
> > > > vs 236.  I suspect that this is because glimpse considers _ to be a space,
> > > > and thus it can have many false positives.  There are, however, a few
> > > > cases where glimpse also selects fewer files.
> > > >
> > > > The file processing time (ie parsing the file, searching for, matches of
> > > > the semantic patch in the file, and performing the transformation) is
> > > > normally much higher than the file selection time.
> > > >
> > > > So it seems that git grep is currently a better option for the kernel.
> > >
> > > Great, thanks, consider this patch dropped, do we want the heuristics
> > > for the cache index in place though or should I drop that as well ?
> > 
> > I assume you mean this patch:
> > 
> > [PATCH v2 3/8] coccicheck: add indexing enhancement options
> > 
> > I think it should be dropped.  It adds complexity and git grep works
> > pretty well.
> 
> Hmm but coccicheck does not make use of --git-grep even.
> 
> > If people want to use something else, they can use SPARGS,
> > or a .cocciconfig file, eg:
> > 
> > [spatch]
> > options = --use-glimpse
> 
> Neat will these be used last and thus override anything?

Good point.  If it is in the home directory, it is overrided by 
everything.  So make coccicheck shouldn't have an option related to this 
issue.

> If so, what 
> about just adding an upstream .cocciconfig with --use-gitgrep -- only 
> issue then is what if a user wants to use idutils ? How do we let them 
> override?

If we have an upstream .cocciconfig with --use-gitgrep, then the user can 
specify an SPARGS with --use-idutils and override.

If we are making an upstream .cocciconfig, I would put a timeout in it 
too.  In my experience, 120 (seconds) is fine.  Maybe 200 to give a little 
more margin.  Again, this can be overridden on the command line.

julia

[Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

On Sat, 18 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 17, 2016 at 05:35:26PM +0200, Julia Lawall wrote:
> > On Fri, 17 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote:
> > > > I'm not sure that this is worth it.  It adds a dependency on a tool that
> > > > seems not to be well maintained.  In terms of Coccinelle, I'm not sure
> > > > that it gives a big benefit.
> > > >
> > > > Attached is a graph showing the file selection time for Coccinelle for a
> > > > selection of fairly complex semantic patches.  Coccigrep is just a
> > > > line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
> > > > In most cases, glimpse is clearly faster.
> > > >
> > > > On the other hand, it seems that glimpse often selects more files.
> > > > Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
> > > > vs 236.  I suspect that this is because glimpse considers _ to be a space,
> > > > and thus it can have many false positives.  There are, however, a few
> > > > cases where glimpse also selects fewer files.
> > > >
> > > > The file processing time (ie parsing the file, searching for, matches of
> > > > the semantic patch in the file, and performing the transformation) is
> > > > normally much higher than the file selection time.
> > > >
> > > > So it seems that git grep is currently a better option for the kernel.
> > >
> > > Great, thanks, consider this patch dropped, do we want the heuristics
> > > for the cache index in place though or should I drop that as well ?
> > 
> > I assume you mean this patch:
> > 
> > [PATCH v2 3/8] coccicheck: add indexing enhancement options
> > 
> > I think it should be dropped.  It adds complexity and git grep works
> > pretty well.
> 
> Hmm but coccicheck does not make use of --git-grep even.
> 
> > If people want to use something else, they can use SPARGS,
> > or a .cocciconfig file, eg:
> > 
> > [spatch]
> > options = --use-glimpse
> 
> Neat will these be used last and thus override anything?

Good point.  If it is in the home directory, it is overrided by 
everything.  So make coccicheck shouldn't have an option related to this 
issue.

> If so, what 
> about just adding an upstream .cocciconfig with --use-gitgrep -- only 
> issue then is what if a user wants to use idutils ? How do we let them 
> override?

If we have an upstream .cocciconfig with --use-gitgrep, then the user can 
specify an SPARGS with --use-idutils and override.

If we are making an upstream .cocciconfig, I would put a timeout in it 
too.  In my experience, 120 (seconds) is fine.  Maybe 200 to give a little 
more margin.  Again, this can be overridden on the command line.

julia

Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

Overall, idutils seems to be a good choice.  As compared to a grep based 
solution, it knows what is code, so it doesn't report on files where the 
words of interest only occur in comments.  As compared to glimpse, it 
knows that foo_bar is a single word.  Indexing is faster than with 
glimpse, and looking things up in the index is also slightly faster, even 
though Coccinelle needs to make multiple calls because it doesn't support 
complex formulas.  It does support regexps, which could perhaps be even 
faster, but since the running time currently is mostly under 1 second and 
often under .1 seconds, it probably doesn't matter.

julia

[Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

Overall, idutils seems to be a good choice.  As compared to a grep based 
solution, it knows what is code, so it doesn't report on files where the 
words of interest only occur in comments.  As compared to glimpse, it 
knows that foo_bar is a single word.  Indexing is faster than with 
glimpse, and looking things up in the index is also slightly faster, even 
though Coccinelle needs to make multiple calls because it doesn't support 
complex formulas.  It does support regexps, which could perhaps be even 
faster, but since the running time currently is mostly under 1 second and 
often under .1 seconds, it probably doesn't matter.

julia

Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

On Sat, 18 Jun 2016, Julia Lawall wrote:

> Overall, idutils seems to be a good choice.  As compared to a grep based 
> solution, it knows what is code, so it doesn't report on files where the 
> words of interest only occur in comments.  As compared to glimpse, it 
> knows that foo_bar is a single word.  Indexing is faster than with 
> glimpse, and looking things up in the index is also slightly faster, even 
> though Coccinelle needs to make multiple calls because it doesn't support 
> complex formulas.  It does support regexps, which could perhaps be even 
> faster, but since the running time currently is mostly under 1 second and 
> often under .1 seconds, it probably doesn't matter.

I'm not suggesting that idutils should be the default.  Only that someone 
who wants to go to the trouble of indexing could find that idutils is a 
good choice.

julia

[Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Julia Lawall]

On Sat, 18 Jun 2016, Julia Lawall wrote:

> Overall, idutils seems to be a good choice.  As compared to a grep based 
> solution, it knows what is code, so it doesn't report on files where the 
> words of interest only occur in comments.  As compared to glimpse, it 
> knows that foo_bar is a single word.  Indexing is faster than with 
> glimpse, and looking things up in the index is also slightly faster, even 
> though Coccinelle needs to make multiple calls because it doesn't support 
> complex formulas.  It does support regexps, which could perhaps be even 
> faster, but since the running time currently is mostly under 1 second and 
> often under .1 seconds, it probably doesn't matter.

I'm not suggesting that idutils should be the default.  Only that someone 
who wants to go to the trouble of indexing could find that idutils is a 
good choice.

julia

Re: [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

On Sat, Jun 18, 2016 at 07:51:55AM +0200, Julia Lawall wrote:
> 
> 
> On Sat, 18 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Fri, Jun 17, 2016 at 05:35:26PM +0200, Julia Lawall wrote:
> > > On Fri, 17 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote:
> > > > > I'm not sure that this is worth it.  It adds a dependency on a tool that
> > > > > seems not to be well maintained.  In terms of Coccinelle, I'm not sure
> > > > > that it gives a big benefit.
> > > > >
> > > > > Attached is a graph showing the file selection time for Coccinelle for a
> > > > > selection of fairly complex semantic patches.  Coccigrep is just a
> > > > > line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
> > > > > In most cases, glimpse is clearly faster.
> > > > >
> > > > > On the other hand, it seems that glimpse often selects more files.
> > > > > Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
> > > > > vs 236.  I suspect that this is because glimpse considers _ to be a space,
> > > > > and thus it can have many false positives.  There are, however, a few
> > > > > cases where glimpse also selects fewer files.
> > > > >
> > > > > The file processing time (ie parsing the file, searching for, matches of
> > > > > the semantic patch in the file, and performing the transformation) is
> > > > > normally much higher than the file selection time.
> > > > >
> > > > > So it seems that git grep is currently a better option for the kernel.
> > > >
> > > > Great, thanks, consider this patch dropped, do we want the heuristics
> > > > for the cache index in place though or should I drop that as well ?
> > > 
> > > I assume you mean this patch:
> > > 
> > > [PATCH v2 3/8] coccicheck: add indexing enhancement options
> > > 
> > > I think it should be dropped.  It adds complexity and git grep works
> > > pretty well.
> > 
> > Hmm but coccicheck does not make use of --git-grep even.
> > 
> > > If people want to use something else, they can use SPARGS,
> > > or a .cocciconfig file, eg:
> > > 
> > > [spatch]
> > > options = --use-glimpse
> > 
> > Neat will these be used last and thus override anything?
> 
> Good point.  If it is in the home directory, it is overrided by 
> everything.  So make coccicheck shouldn't have an option related to this 
> issue.

Great.

> > If so, what 
> > about just adding an upstream .cocciconfig with --use-gitgrep -- only 
> > issue then is what if a user wants to use idutils ? How do we let them 
> > override?
> 
> If we have an upstream .cocciconfig with --use-gitgrep, then the user can 
> specify an SPARGS with --use-idutils and override.

I take it you meant SPFLAGS. I just read the order rules, I'll past them
for completeness:

-- from coccinelle/read_options.ml:

.cocciconfig files can be placed in the user's home directory, the              
directory from which spatch is called, and the directory provided with the      
--dir option.  The .cocciconfig file in the user's home directory is            
processed first, the .cocciconfig file in the directory from which spatch       
is called is processed next, and the .cocciconfig file in the directory         
provided with the --dir option is processed last.  In each case, the read       
options extend/override the previously read ones.  In all cases, the user       
can extend/override the options found in the .cocciconfig files on the          
command line.  
---

So order is:

0. $HOME/.cocciconfig
1. $PWD/.cocciconfig
2. --dir .cocciconfig

So indeed an upstream .cocciconfig would seem to work well.

Drivers can also have their own .cocciconfig if they would need it, but I
cannot see this being needed at this time though, but good to know and keep
in mind. In the future this fact might be a bit more useful if we added
support for instance of a rule namespace, then for instance if we know a
tweak is only needed for one driver we might for instance have something
like:

[spatch rule=scripts/coccinelle/api/d_find_alias.cocci]
  options = --opt1 --opt2                                                       
  ...    

But for now I think more than good with an upstream linux/.cocciconfig
then and SPFLAGs. I will have to do just one small adjustment to SPFLAGS
on coccicheck to ensure it does go at the end. I'll address that in the
re-spin of this series.

> If we are making an upstream .cocciconfig, I would put a timeout in it 
> too.  In my experience, 120 (seconds) is fine.  Maybe 200 to give a little 
> more margin.  Again, this can be overridden on the command line.

OK will use 200.

  Luis

[Cocci] [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

On Sat, Jun 18, 2016 at 07:51:55AM +0200, Julia Lawall wrote:
> 
> 
> On Sat, 18 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Fri, Jun 17, 2016 at 05:35:26PM +0200, Julia Lawall wrote:
> > > On Fri, 17 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > On Fri, Jun 17, 2016 at 11:44:26AM +0200, Julia Lawall wrote:
> > > > > I'm not sure that this is worth it.  It adds a dependency on a tool that
> > > > > seems not to be well maintained.  In terms of Coccinelle, I'm not sure
> > > > > that it gives a big benefit.
> > > > >
> > > > > Attached is a graph showing the file selection time for Coccinelle for a
> > > > > selection of fairly complex semantic patches.  Coccigrep is just a
> > > > > line-by-line regexp search implemented in ocaml, gitgrep uses git grep.
> > > > > In most cases, glimpse is clearly faster.
> > > > >
> > > > > On the other hand, it seems that glimpse often selects more files.
> > > > > Sometimes a few more, eg 16 vs 14, and sometimes quite a lot more, eg 538
> > > > > vs 236.  I suspect that this is because glimpse considers _ to be a space,
> > > > > and thus it can have many false positives.  There are, however, a few
> > > > > cases where glimpse also selects fewer files.
> > > > >
> > > > > The file processing time (ie parsing the file, searching for, matches of
> > > > > the semantic patch in the file, and performing the transformation) is
> > > > > normally much higher than the file selection time.
> > > > >
> > > > > So it seems that git grep is currently a better option for the kernel.
> > > >
> > > > Great, thanks, consider this patch dropped, do we want the heuristics
> > > > for the cache index in place though or should I drop that as well ?
> > > 
> > > I assume you mean this patch:
> > > 
> > > [PATCH v2 3/8] coccicheck: add indexing enhancement options
> > > 
> > > I think it should be dropped.  It adds complexity and git grep works
> > > pretty well.
> > 
> > Hmm but coccicheck does not make use of --git-grep even.
> > 
> > > If people want to use something else, they can use SPARGS,
> > > or a .cocciconfig file, eg:
> > > 
> > > [spatch]
> > > options = --use-glimpse
> > 
> > Neat will these be used last and thus override anything?
> 
> Good point.  If it is in the home directory, it is overrided by 
> everything.  So make coccicheck shouldn't have an option related to this 
> issue.

Great.

> > If so, what 
> > about just adding an upstream .cocciconfig with --use-gitgrep -- only 
> > issue then is what if a user wants to use idutils ? How do we let them 
> > override?
> 
> If we have an upstream .cocciconfig with --use-gitgrep, then the user can 
> specify an SPARGS with --use-idutils and override.

I take it you meant SPFLAGS. I just read the order rules, I'll past them
for completeness:

-- from coccinelle/read_options.ml:

.cocciconfig files can be placed in the user's home directory, the              
directory from which spatch is called, and the directory provided with the      
--dir option.  The .cocciconfig file in the user's home directory is            
processed first, the .cocciconfig file in the directory from which spatch       
is called is processed next, and the .cocciconfig file in the directory         
provided with the --dir option is processed last.  In each case, the read       
options extend/override the previously read ones.  In all cases, the user       
can extend/override the options found in the .cocciconfig files on the          
command line.  
---

So order is:

0. $HOME/.cocciconfig
1. $PWD/.cocciconfig
2. --dir .cocciconfig

So indeed an upstream .cocciconfig would seem to work well.

Drivers can also have their own .cocciconfig if they would need it, but I
cannot see this being needed at this time though, but good to know and keep
in mind. In the future this fact might be a bit more useful if we added
support for instance of a rule namespace, then for instance if we know a
tweak is only needed for one driver we might for instance have something
like:

[spatch rule=scripts/coccinelle/api/d_find_alias.cocci]
  options = --opt1 --opt2                                                       
  ...    

But for now I think more than good with an upstream linux/.cocciconfig
then and SPFLAGs. I will have to do just one small adjustment to SPFLAGS
on coccicheck to ensure it does go at the end. I'll address that in the
re-spin of this series.

> If we are making an upstream .cocciconfig, I would put a timeout in it 
> too.  In my experience, 120 (seconds) is fine.  Maybe 200 to give a little 
> more margin.  Again, this can be overridden on the command line.

OK will use 200.

  Luis