[PATCH v2 0/5] coccicheck: extend with parmap and indexing heuristics

[PATCH v2 0/5] coccicheck: extend with parmap and indexing heuristics

[Luis R. Rodriguez]

This v2 addresses the feedback from the v1 series. It also adds 2 more
patches, one to document how to add extra additional options and refers
to a wiki for further documentation.

Luis R. Rodriguez (5):
  coccicheck: enable parmap support
  scripts: add glimpse.sh for indexing the kernel
  coccicheck: add indexing enhancement options
  coccicheck: document how to pass extra options to coccinelle
  coccicheck: refer to coccicheck bottest wiki for documentation

 scripts/coccicheck | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++---
 scripts/glimpse.sh |  12 +++++
 2 files changed, 161 insertions(+), 6 deletions(-)
 create mode 100755 scripts/glimpse.sh

-- 
2.8.2

[PATCH v2 1/5] coccicheck: enable parmap support

[Luis R. Rodriguez]

Coccinelle has had parmap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported.

Also enable the load balancing to be dynamic, so that if a
thread finishes early we keep feeding it.

Note: now that we have all things handled for us, redirect stderr to
stdout as well to capture any possible errors or warnings issued by
coccinelle.

If --jobs is not supported we fallback to the old mechanism.

As a small example, prior to this change, on an 8-core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report
...

real    29m14.912s
user    103m1.796s
sys     0m4.464s

After:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

v2:

o redirect coccinelle stderr to ${DIR}/coccicheck.$$.err
o fix typo of paramap/parmap

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index aa5e78fba270..02602381be59 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
     exit 1
 fi
 
-trap kill_running SIGTERM SIGINT
-declare -a SPATCH_PID
+USE_JOBS="no"
+$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
@@ -82,7 +82,21 @@ if [ "$ONLINE" = "0" ] ; then
     echo ''
 fi
 
-run_cmd() {
+if [ "$USE_JOBS" = "no" ]; then
+	trap kill_running SIGTERM SIGINT
+	declare -a SPATCH_PID
+else
+	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
+fi
+
+run_cmd_parmap() {
+	if [ $VERBOSE -ne 0 ] ; then
+		echo "Running ($NPROC in parallel): $@"
+	fi
+	$@ 2> ${DIR}/coccicheck.$$.err
+}
+
+run_cmd_old() {
 	local i
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
@@ -97,6 +111,14 @@ run_cmd() {
 	wait
 }
 
+run_cmd() {
+	if [ "$USE_JOBS" = "yes" ]; then
+		run_cmd_parmap $@
+	else
+		run_cmd_old $@
+	fi
+}
+
 kill_running() {
 	for i in $(seq 0 $(( NPROC - 1 )) ); do
 		if [ $VERBOSE -eq 2 ] ; then
-- 
2.8.2

[PATCH v2 2/5] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

Glimpse is a tool you can use to index the kernel. The tool
was recently open sourced under the ISC license and can be
obtained at:

https://github.com/mcgrof/glimpse.git

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/glimpse.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100755 scripts/glimpse.sh

diff --git a/scripts/glimpse.sh b/scripts/glimpse.sh
new file mode 100755
index 000000000000..5fd24e49f31b
--- /dev/null
+++ b/scripts/glimpse.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+DIR=$(dirname $(readlink -f $0))
+DIR="${DIR}/../"
+
+GLIMPSEINDEX="`which ${GLIMPSEINDEX:=glimpseindex}`"
+if [ ! -x "$GLIMPSEINDEX" ]; then
+	echo 'glimpseindex can be obtained at https://github.com/mcgrof/glimpse.git'
+	exit 1
+fi
+
+find $DIR/* -name "*.[ch]" | $GLIMPSEINDEX -o -H . -F
-- 
2.8.2

[PATCH v2 3/5] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

Coccinelle has support to make use of its own enhanced "grep"
mechanisms instead of using regular grep for searching code,
it calls this 'coccigrep'. In lack of any indexing optimization
information it uses --use-coccigrep by default.

This patch enable indexing optimizations heuristics so that coccigrep
can automatically detect what indexing options are available and use
them accordinly without any user input.

Since git has its own index, support for using 'git grep' has been
added to Coccinelle, that should on average perform better than
using the internal coccigrep. Coccinelle has had idutils support
as well for a while now, you however need to refer to the index
file. We support detecting two idutils index files by default,
ID and .id-utils.index, assuming you ran either of:

 # What you might have done:
mkid -s
 # as in coccinelle scripts/idutils_index.sh
mkid -i C --output .id-utils.index *

Lastly, Coccinelle has had support for glimpseindex for a long while,
however the glimpseindex tool, the agrep library were previously closed
source, its all now open sourced, and provides the best performance, so
support that if we can detect you have a glimpse index.

You can always override the index as follows:

$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ time make coccicheck MODE=report

coccigrep (default and without this patch):
real    0m16.369s
user    0m58.712s
sys     0m5.064s

After:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ time make coccicheck MODE=report

With glimpse:
real    0m6.549s
user    0m49.136s
sys     0m3.076s

With idutils:
real    0m6.749s
user    0m51.936s
sys     0m3.876s

With gitgrep:
real    0m6.805s
user    0m51.572s
sys     0m4.432s

o document logic used for indexing
o add idutils support, supports two indexing files
o remove coccigrep heuristics as its the default anyway
o add COCCI_INDEX to enable overriding heuristics, you can use this
  as follows, for example:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"

o Refer users to ${DIR}/coccicheck.$$.err for indexing errors,
  document what is expected to ensure glimpse is working correctly

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 97 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 02602381be59..6d75fd8580a6 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -5,6 +5,8 @@
 # version 1.0.0-rc11.
 #
 
+DIR=$(dirname $(readlink -f $0))
+DIR="${DIR}/../"
 SPATCH="`which ${SPATCH:=spatch}`"
 
 if [ ! -x "$SPATCH" ]; then
@@ -15,6 +17,80 @@ fi
 USE_JOBS="no"
 $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
+# Indexing USE_* optimizations heuristics.
+#
+# Linux runs on git (TM). However, if you have supplemental indexing options
+# you may use them to help Coccinelle further. If you are using Coccinelle
+# within a target git tree --use-gitrep will be used, and this should
+# suffice for most uses. If you however want optimal performance do
+# consider embracing a supplemental indexing as part of your development.
+# For instance glimpse, and idutils can be used, however you should
+# be sure to update the indexes as often as you update your git tree to
+# ensure your indexes are not stale.
+#
+# idutils is currently not as efficient as glimpse because the query language
+# for glimpse is simpler, so when idutils is used more filtering has to be
+# done at the ocaml level within Coccinelle. Glimpse allows queries that are
+# arbitrary formulas, up to a limited level of complexity, involving both
+# && and ||. For idutils, Coccinelle runs lid intersections on the result.
+#
+# You can override these heuristics with COCCI_INDEX="--use-gitgrep" for
+# example. This will force to use --use-gitgrep even if you have a glimpse
+# index. Other examples:
+#
+# $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"
+#
+# The order of heuristics for indexing used by coccicheck is listed below.
+#
+# 0. Glimpse currently should outperform all indexing options. so if a glimpse
+#    index is used we use it. Refer to Linux scripts/glimpse.sh for details.
+#    If you think you should be getting better performance with glimpse than
+#    what you would expect inspect the stderr log file for cocciecheck:
+#
+#    ${DIR}/coccicheck.$$.err
+#
+#    If glimpse is running correctly there should be very few occurrences
+#    of "Skipping", also coccinelle will inform you if it could not use
+#    glimpse. As an example an output of the following would indicate glimpse
+#    was properly used on ${DIR}/coccicheck.$$.err :
+#
+#    There are matches to 1252 out of 47281 files
+#    glimpse request = request_threaded_irq
+#
+# 1. Use idutils next. You'll need to generate an index using either of these:
+#
+#	a) mkid -s
+#       By default this dumps the index into ./ID
+#
+#	b) mkid -i C --output .id-utils.index *
+#       This method is provided with coccinelle repo on
+#       scripts/idutils_index.sh
+#
+# 2. Next best is --use-gitgrep and if you are working within a git tree
+#    this will be used by default.
+#
+# 3. By default coccinelle internally uses --use-coccigrep if no indexing
+#    options are requested and your version of coccinelle supports it so we
+#    do not need to be specific about requesting that as a fallback mechanism.
+#    Use of --use-coccigrep is comparable to --use-gitgrep.
+#
+# XXX: Glimpse is not well maintained. See if we can add similar indexing
+# features and query language glimpse supports to git.
+if [ "$COCCI_INDEX" = "" ] ; then
+	USE_GLIMPSE="no"
+	$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f $DIR/.glimpse_index ] && USE_GLIMPSE="yes"
+
+	USE_IDUTILS="no"
+	$SPATCH --help | grep "\-\-use\-idutils" > /dev/null && [ -f $DIR/ID -o -f $DIR/.id-utils.index ] && USE_IDUTILS="yes"
+
+	USE_GITGREP="no"
+	$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && USE_GITGREP="yes"
+fi
+
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
 
@@ -89,6 +165,27 @@ else
 	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
 fi
 
+# Check COCCI_INDEX first to manual override, otherwise rely on
+# internal heuristics documented above.
+if [ "$COCCI_INDEX" != "" ] ; then
+	OPTIONS="$OPTIONS $COCCI_INDEX"
+elif [ "$USE_GLIMPSE" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-glimpse"
+elif [ "$USE_IDUTILS" = "yes" ]; then
+	index=""
+	if [ -f $DIR/ID ]; then
+		index="$DIR/ID"
+	elif [ -f $DIR/.id-utils.index ]; then
+		index="$DIR/.id-utils.index"
+	else
+		echo "idutils index not found, expected: $DIR/ID or $DIR/.id-utils.index"
+		exit 1
+	fi
+	OPTIONS="$OPTIONS --use-idutils $index"
+elif [ "$USE_GITGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-gitgrep"
+fi
+
 run_cmd_parmap() {
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
-- 
2.8.2

[PATCH v2 4/5] coccicheck: document how to pass extra options to coccinelle

[Luis R. Rodriguez]

As an example show how to debug your coccinelle script.
While at it, fix the case when --profile is passed, when
--profile is used we need to ensure --very-quiet is not used.

For instance one can use:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ make coccicheck MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 6d75fd8580a6..c6440138c6ea 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -106,7 +106,26 @@ else
 	NPROC="$J"
 fi
 
-FLAGS="$SPFLAGS --very-quiet"
+# You can use SPFLAGS to append extra arguments to coccicheck.
+# A good example is if you want to debug your cocci script, you can
+# for instance use the following:
+#
+# $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
+# $ time make coccicheck MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c
+#
+# "--show-trying" should show you what rule is being processed as it goes, if
+# you have issues with a rule getting stuck you can then inspect the file:
+#
+# ${DIR}/coccicheck.$$.err
+#
+# which will have the profile output.
+#
+# --profile will not output if --very-quiet is used, so avoid it.
+if [[ $SPFLAGS  == *"--profile"* ]]; then
+	FLAGS="$SPFLAGS"
+else
+	FLAGS="$SPFLAGS --very-quiet"
+fi
 
 # spatch only allows include directories with the syntax "-I include"
 # while gcc also allows "-Iinclude" and "-include include"
-- 
2.8.2

[PATCH v2 5/5] coccicheck: refer to coccicheck bottest wiki for documentation

[Luis R. Rodriguez]

Sprinkling *tons* of documentation on the script is not a good
idea, instead refer to a wiki for further coccicheck documentation:

https://bottest.wiki.kernel.org/coccicheck

This page shall always refer to the linux-next iteration of
scripts/coccicheck.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index c6440138c6ea..37db7ac1dfae 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -1,9 +1,14 @@
 #!/bin/bash
-
+# Linux kernel coccicheck
+#
+# For more detailed documentation refer to:
+#
+# https://bottest.wiki.kernel.org/coccicheck
+#
+# This documentation always refers to the linux-next version of the script.
 #
 # This script requires at least spatch
 # version 1.0.0-rc11.
-#
 
 DIR=$(dirname $(readlink -f $0))
 DIR="${DIR}/../"
-- 
2.8.2

Re: [PATCH v2 0/5] coccicheck: extend with parmap and indexing heuristics

[Luis R. Rodriguez]

On Tue, Jun 14, 2016 at 01:05:47PM -0700, Luis R. Rodriguez wrote:
> This v2 addresses the feedback from the v1 series. It also adds 2 more
> patches, one to document how to add extra additional options and refers
> to a wiki for further documentation.
> 
> Luis R. Rodriguez (5):
>   coccicheck: enable parmap support
>   scripts: add glimpse.sh for indexing the kernel
>   coccicheck: add indexing enhancement options
>   coccicheck: document how to pass extra options to coccinelle
>   coccicheck: refer to coccicheck bottest wiki for documentation
> 

Sorry I forgot the first patch so this should be 6 patches will send v3...

 Luis

[PATCH v3 0/6] coccicheck: extend with parmap and indexing heuristics

[Luis R. Rodriguez]

I fat fingered the v2 series and forgot the first patch, this just adds
the first patch... sorry about that.

Luis R. Rodriguez (6):
  coccicheck: move spatch binary check up
  coccicheck: enable parmap support
  scripts: add glimpse.sh for indexing the kernel
  coccicheck: add indexing enhancement options
  coccicheck: document how to pass extra options to coccinelle
  coccicheck: refer to coccicheck bottest wiki for documentation

 scripts/coccicheck | 165 +++++++++++++++++++++++++++++++++++++++++++++++++----
 scripts/glimpse.sh |  12 ++++
 2 files changed, 166 insertions(+), 11 deletions(-)
 create mode 100755 scripts/glimpse.sh

-- 
2.8.2

[PATCH v3 1/6] coccicheck: move spatch binary check up

[Luis R. Rodriguez]

This has no functional changes. This is being done
to enable us to later use spatch binary for some
flag checking for certain features early on.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index dd85a455b2ba..aa5e78fba270 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -7,6 +7,11 @@
 
 SPATCH="`which ${SPATCH:=spatch}`"
 
+if [ ! -x "$SPATCH" ]; then
+    echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/'
+    exit 1
+fi
+
 trap kill_running SIGTERM SIGINT
 declare -a SPATCH_PID
 
@@ -51,11 +56,6 @@ if [ "$KBUILD_EXTMOD" != "" ] ; then
     OPTIONS="--patch $srctree $OPTIONS"
 fi
 
-if [ ! -x "$SPATCH" ]; then
-    echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/'
-    exit 1
-fi
-
 if [ "$MODE" = "" ] ; then
     if [ "$ONLINE" = "0" ] ; then
 	echo 'You have not explicitly specified the mode to use. Using default "report" mode.'
-- 
2.8.2

[PATCH v3 2/6] coccicheck: enable parmap support

[Luis R. Rodriguez]

Coccinelle has had parmap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported.

Also enable the load balancing to be dynamic, so that if a
thread finishes early we keep feeding it.

Note: now that we have all things handled for us, redirect stderr to
stdout as well to capture any possible errors or warnings issued by
coccinelle.

If --jobs is not supported we fallback to the old mechanism.

As a small example, prior to this change, on an 8-core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report
...

real    29m14.912s
user    103m1.796s
sys     0m4.464s

After:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

v2:

o redirect coccinelle stderr to ${DIR}/coccicheck.$$.err
o fix typo of paramap/parmap

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index aa5e78fba270..02602381be59 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
     exit 1
 fi
 
-trap kill_running SIGTERM SIGINT
-declare -a SPATCH_PID
+USE_JOBS="no"
+$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
@@ -82,7 +82,21 @@ if [ "$ONLINE" = "0" ] ; then
     echo ''
 fi
 
-run_cmd() {
+if [ "$USE_JOBS" = "no" ]; then
+	trap kill_running SIGTERM SIGINT
+	declare -a SPATCH_PID
+else
+	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
+fi
+
+run_cmd_parmap() {
+	if [ $VERBOSE -ne 0 ] ; then
+		echo "Running ($NPROC in parallel): $@"
+	fi
+	$@ 2> ${DIR}/coccicheck.$$.err
+}
+
+run_cmd_old() {
 	local i
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
@@ -97,6 +111,14 @@ run_cmd() {
 	wait
 }
 
+run_cmd() {
+	if [ "$USE_JOBS" = "yes" ]; then
+		run_cmd_parmap $@
+	else
+		run_cmd_old $@
+	fi
+}
+
 kill_running() {
 	for i in $(seq 0 $(( NPROC - 1 )) ); do
 		if [ $VERBOSE -eq 2 ] ; then
-- 
2.8.2

[PATCH v3 3/6] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

Glimpse is a tool you can use to index the kernel. The tool
was recently open sourced under the ISC license and can be
obtained at:

https://github.com/mcgrof/glimpse.git

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/glimpse.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100755 scripts/glimpse.sh

diff --git a/scripts/glimpse.sh b/scripts/glimpse.sh
new file mode 100755
index 000000000000..5fd24e49f31b
--- /dev/null
+++ b/scripts/glimpse.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+DIR=$(dirname $(readlink -f $0))
+DIR="${DIR}/../"
+
+GLIMPSEINDEX="`which ${GLIMPSEINDEX:=glimpseindex}`"
+if [ ! -x "$GLIMPSEINDEX" ]; then
+	echo 'glimpseindex can be obtained at https://github.com/mcgrof/glimpse.git'
+	exit 1
+fi
+
+find $DIR/* -name "*.[ch]" | $GLIMPSEINDEX -o -H . -F
-- 
2.8.2

[PATCH v3 4/6] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

Coccinelle has support to make use of its own enhanced "grep"
mechanisms instead of using regular grep for searching code,
it calls this 'coccigrep'. In lack of any indexing optimization
information it uses --use-coccigrep by default.

This patch enable indexing optimizations heuristics so that coccigrep
can automatically detect what indexing options are available and use
them accordinly without any user input.

Since git has its own index, support for using 'git grep' has been
added to Coccinelle, that should on average perform better than
using the internal coccigrep. Coccinelle has had idutils support
as well for a while now, you however need to refer to the index
file. We support detecting two idutils index files by default,
ID and .id-utils.index, assuming you ran either of:

 # What you might have done:
mkid -s
 # as in coccinelle scripts/idutils_index.sh
mkid -i C --output .id-utils.index *

Lastly, Coccinelle has had support for glimpseindex for a long while,
however the glimpseindex tool, the agrep library were previously closed
source, its all now open sourced, and provides the best performance, so
support that if we can detect you have a glimpse index.

You can always override the index as follows:

$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ time make coccicheck MODE=report

coccigrep (default and without this patch):
real    0m16.369s
user    0m58.712s
sys     0m5.064s

After:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ time make coccicheck MODE=report

With glimpse:
real    0m6.549s
user    0m49.136s
sys     0m3.076s

With idutils:
real    0m6.749s
user    0m51.936s
sys     0m3.876s

With gitgrep:
real    0m6.805s
user    0m51.572s
sys     0m4.432s

o document logic used for indexing
o add idutils support, supports two indexing files
o remove coccigrep heuristics as its the default anyway
o add COCCI_INDEX to enable overriding heuristics, you can use this
  as follows, for example:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
$ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"

o Refer users to ${DIR}/coccicheck.$$.err for indexing errors,
  document what is expected to ensure glimpse is working correctly

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 97 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 02602381be59..6d75fd8580a6 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -5,6 +5,8 @@
 # version 1.0.0-rc11.
 #
 
+DIR=$(dirname $(readlink -f $0))
+DIR="${DIR}/../"
 SPATCH="`which ${SPATCH:=spatch}`"
 
 if [ ! -x "$SPATCH" ]; then
@@ -15,6 +17,80 @@ fi
 USE_JOBS="no"
 $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
+# Indexing USE_* optimizations heuristics.
+#
+# Linux runs on git (TM). However, if you have supplemental indexing options
+# you may use them to help Coccinelle further. If you are using Coccinelle
+# within a target git tree --use-gitrep will be used, and this should
+# suffice for most uses. If you however want optimal performance do
+# consider embracing a supplemental indexing as part of your development.
+# For instance glimpse, and idutils can be used, however you should
+# be sure to update the indexes as often as you update your git tree to
+# ensure your indexes are not stale.
+#
+# idutils is currently not as efficient as glimpse because the query language
+# for glimpse is simpler, so when idutils is used more filtering has to be
+# done at the ocaml level within Coccinelle. Glimpse allows queries that are
+# arbitrary formulas, up to a limited level of complexity, involving both
+# && and ||. For idutils, Coccinelle runs lid intersections on the result.
+#
+# You can override these heuristics with COCCI_INDEX="--use-gitgrep" for
+# example. This will force to use --use-gitgrep even if you have a glimpse
+# index. Other examples:
+#
+# $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-coccigrep"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-idutils ID"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-glimpse"
+# $ make coccicheck V=1 MODE=report COCCI_INDEX="--use-gitgrep"
+#
+# The order of heuristics for indexing used by coccicheck is listed below.
+#
+# 0. Glimpse currently should outperform all indexing options. so if a glimpse
+#    index is used we use it. Refer to Linux scripts/glimpse.sh for details.
+#    If you think you should be getting better performance with glimpse than
+#    what you would expect inspect the stderr log file for cocciecheck:
+#
+#    ${DIR}/coccicheck.$$.err
+#
+#    If glimpse is running correctly there should be very few occurrences
+#    of "Skipping", also coccinelle will inform you if it could not use
+#    glimpse. As an example an output of the following would indicate glimpse
+#    was properly used on ${DIR}/coccicheck.$$.err :
+#
+#    There are matches to 1252 out of 47281 files
+#    glimpse request = request_threaded_irq
+#
+# 1. Use idutils next. You'll need to generate an index using either of these:
+#
+#	a) mkid -s
+#       By default this dumps the index into ./ID
+#
+#	b) mkid -i C --output .id-utils.index *
+#       This method is provided with coccinelle repo on
+#       scripts/idutils_index.sh
+#
+# 2. Next best is --use-gitgrep and if you are working within a git tree
+#    this will be used by default.
+#
+# 3. By default coccinelle internally uses --use-coccigrep if no indexing
+#    options are requested and your version of coccinelle supports it so we
+#    do not need to be specific about requesting that as a fallback mechanism.
+#    Use of --use-coccigrep is comparable to --use-gitgrep.
+#
+# XXX: Glimpse is not well maintained. See if we can add similar indexing
+# features and query language glimpse supports to git.
+if [ "$COCCI_INDEX" = "" ] ; then
+	USE_GLIMPSE="no"
+	$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f $DIR/.glimpse_index ] && USE_GLIMPSE="yes"
+
+	USE_IDUTILS="no"
+	$SPATCH --help | grep "\-\-use\-idutils" > /dev/null && [ -f $DIR/ID -o -f $DIR/.id-utils.index ] && USE_IDUTILS="yes"
+
+	USE_GITGREP="no"
+	$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && USE_GITGREP="yes"
+fi
+
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
 
@@ -89,6 +165,27 @@ else
 	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
 fi
 
+# Check COCCI_INDEX first to manual override, otherwise rely on
+# internal heuristics documented above.
+if [ "$COCCI_INDEX" != "" ] ; then
+	OPTIONS="$OPTIONS $COCCI_INDEX"
+elif [ "$USE_GLIMPSE" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-glimpse"
+elif [ "$USE_IDUTILS" = "yes" ]; then
+	index=""
+	if [ -f $DIR/ID ]; then
+		index="$DIR/ID"
+	elif [ -f $DIR/.id-utils.index ]; then
+		index="$DIR/.id-utils.index"
+	else
+		echo "idutils index not found, expected: $DIR/ID or $DIR/.id-utils.index"
+		exit 1
+	fi
+	OPTIONS="$OPTIONS --use-idutils $index"
+elif [ "$USE_GITGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-gitgrep"
+fi
+
 run_cmd_parmap() {
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
-- 
2.8.2

[PATCH v3 5/6] coccicheck: document how to pass extra options to coccinelle

[Luis R. Rodriguez]

As an example show how to debug your coccinelle script.
While at it, fix the case when --profile is passed, when
--profile is used we need to ensure --very-quiet is not used.

For instance one can use:

$ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
$ make coccicheck MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index 6d75fd8580a6..c6440138c6ea 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -106,7 +106,26 @@ else
 	NPROC="$J"
 fi
 
-FLAGS="$SPFLAGS --very-quiet"
+# You can use SPFLAGS to append extra arguments to coccicheck.
+# A good example is if you want to debug your cocci script, you can
+# for instance use the following:
+#
+# $ export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci
+# $ time make coccicheck MODE=report SPFLAGS="--profile --show-trying" M=./drivers/mfd/arizona-irq.c
+#
+# "--show-trying" should show you what rule is being processed as it goes, if
+# you have issues with a rule getting stuck you can then inspect the file:
+#
+# ${DIR}/coccicheck.$$.err
+#
+# which will have the profile output.
+#
+# --profile will not output if --very-quiet is used, so avoid it.
+if [[ $SPFLAGS  == *"--profile"* ]]; then
+	FLAGS="$SPFLAGS"
+else
+	FLAGS="$SPFLAGS --very-quiet"
+fi
 
 # spatch only allows include directories with the syntax "-I include"
 # while gcc also allows "-Iinclude" and "-include include"
-- 
2.8.2

[PATCH v3 6/6] coccicheck: refer to coccicheck bottest wiki for documentation

[Luis R. Rodriguez]

Sprinkling *tons* of documentation on the script is not a good
idea, instead refer to a wiki for further coccicheck documentation:

https://bottest.wiki.kernel.org/coccicheck

This page shall always refer to the linux-next iteration of
scripts/coccicheck.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index c6440138c6ea..37db7ac1dfae 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -1,9 +1,14 @@
 #!/bin/bash
-
+# Linux kernel coccicheck
+#
+# For more detailed documentation refer to:
+#
+# https://bottest.wiki.kernel.org/coccicheck
+#
+# This documentation always refers to the linux-next version of the script.
 #
 # This script requires at least spatch
 # version 1.0.0-rc11.
-#
 
 DIR=$(dirname $(readlink -f $0))
 DIR="${DIR}/../"
-- 
2.8.2

Aw: [Cocci] [PATCH v3 3/6] scripts: add glimpse.sh for indexing the kernel

[SF Markus Elfring]

> Glimpse is a tool you can use to index the kernel.
> The tool was recently open sourced under the ISC license
> and can be obtained at:

Do you find any official announcement for this software evolution?

Is the wording "recent" really appropriate if you would eventually like to refer to a repository which was published in the year 2014?
https://github.com/gvelez17/glimpse/


Is it nice to mention the script addition also directly in the commit message?


> +DIR=$(dirname $(readlink -f $0))
> +DIR="${DIR}/../"

I would prefer that these variable assignments will be merged into one.

+DIR="$(dirname $(readlink -f $0))/../"

Regards,
Markus

Aw: [Cocci] [PATCH v3 4/6] coccicheck: add indexing enhancement options

[SF Markus Elfring]

> +DIR=$(dirname $(readlink -f $0))
> +DIR="${DIR}/../"

How do you think about to merge these variable assignments also into one?

+DIR="$(dirname $(readlink -f $0))/../"

Regards,
Markus

Re: [Cocci] [PATCH v3 3/6] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 12:26:19PM +0200, SF Markus Elfring wrote:
> > Glimpse is a tool you can use to index the kernel.
> > The tool was recently open sourced under the ISC license
> > and can be obtained at:
> 
> Do you find any official announcement for this software evolution?

I had reached out to Udi Manber in 2014, with the help of a few others we
helped persuade University of Arizona of an ISC release.

http://webglimpse.net/

Its vaguely mentioned there. That's as good as it gets...

> Is the wording "recent" really appropriate if you would eventually like to refer to a repository which was published in the year 2014?
> https://github.com/gvelez17/glimpse/

Sadly the original release didn't compile, I reported the issue in hopes it would
be fixed. Nothing has happened since and I got tired of waiting so I forked
and fixed it myself. I sent a pull request but I don't think its been merged
yet.

  Luis

Re: scripts: add glimpse.sh for indexing the kernel

[SF Markus Elfring]

> http://webglimpse.net/
> Its vaguely mentioned there. That's as good as it gets...

How do you think about to insert the date "2014-09-26" into your commit
message?


>> https://github.com/gvelez17/glimpse/
> Sadly the original release didn't compile, I reported the issue in hopes it would
> be fixed. Nothing has happened since and I got tired of waiting so I forked
> and fixed it myself.

I am also curious on further improvements for this software.
How will corresponding constructive feedback evolve?

Regards,
Markus

Re: scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 08:24:03PM +0200, SF Markus Elfring wrote:
> > http://webglimpse.net/
> > Its vaguely mentioned there. That's as good as it gets...
> 
> How do you think about to insert the date "2014-09-26" into your commit
> message?

Sure.

> >> https://github.com/gvelez17/glimpse/
> > Sadly the original release didn't compile, I reported the issue in hopes it would
> > be fixed. Nothing has happened since and I got tired of waiting so I forked
> > and fixed it myself.
> 
> I am also curious on further improvements for this software.
> How will corresponding constructive feedback evolve?

Beats me. As I see it no one cares over this software now, it would seem
some folks have used it in proprietary crap and those versions are much
better. To me what we can best do is see what we can take out of agrep,
and port it to proper GNU grep, then see if git can also enable complex
queries as Julia notes.

Patches to those projects welcomed I'm sure.

  Luis

Re: [Cocci] [PATCH v3 4/6] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 01:20:19PM +0200, SF Markus Elfring wrote:
> > +DIR=$(dirname $(readlink -f $0))
> > +DIR="${DIR}/../"
> 
> How do you think about to merge these variable assignments also into one?
> 
> +DIR="$(dirname $(readlink -f $0))/../"

Sure.

  Luis

Re: scripts: add glimpse.sh for indexing the kernel

[SF Markus Elfring]

> To me what we can best do is see what we can take out of agrep,
> and port it to proper GNU grep, then see if git can also enable
> complex queries as Julia notes.

How do you think about to improve such tools together with computation
tree logic?   ;-)

Regards,
Markus

Re: scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

On Fri, Jun 17, 2016 at 06:05:38PM +0200, SF Markus Elfring wrote:
> > To me what we can best do is see what we can take out of agrep,
> > and port it to proper GNU grep, then see if git can also enable
> > complex queries as Julia notes.
> 
> How do you think about to improve such tools together with computation
> tree logic?   ;-)

Patches are welcomed, I'm sure. Please send patches to git or GNU grep
on their respective upstream list.

  Luis