[PATCH 0/4] scripts/coccicheck: add paramap and indexing options

[PATCH 0/4] scripts/coccicheck: add paramap and indexing options

[Luis R. Rodriguez]

coccicheck hasn't been updated for a while. The backports
project has been using some features for a while now that
we should be able to also take advantage of with coccicheck,
the most important one is paramap support.

glimpseindex stuff wasn't even building but today I decided
to go tackle and fix that, the public open source release is
is now working and you can optionally use that. Note that
using git performs just as well, glimpseindex just shaves off
a bit of time, however if since we can support it now we do it.

Luis R. Rodriguez (4):
  coccicheck: move spatch binary check up
  coccicheck: enable paramap support
  scripts: add glimpse.sh for indexing the kernel
  coccicheck: add indexing enhancement options

 scripts/coccicheck | 62 +++++++++++++++++++++++++++++++++++++++++++++++-------
 scripts/glimpse.sh | 12 +++++++++++
 2 files changed, 66 insertions(+), 8 deletions(-)
 create mode 100755 scripts/glimpse.sh

-- 
2.8.2

[PATCH 1/4] coccicheck: move spatch binary check up

[Luis R. Rodriguez]

This has no functional changes. This is being done
to enable us to later use spatch binary for some
flag checking for certain features early on.

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index dd85a455b2ba..aa5e78fba270 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -7,6 +7,11 @@
 
 SPATCH="`which ${SPATCH:=spatch}`"
 
+if [ ! -x "$SPATCH" ]; then
+    echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/'
+    exit 1
+fi
+
 trap kill_running SIGTERM SIGINT
 declare -a SPATCH_PID
 
@@ -51,11 +56,6 @@ if [ "$KBUILD_EXTMOD" != "" ] ; then
     OPTIONS="--patch $srctree $OPTIONS"
 fi
 
-if [ ! -x "$SPATCH" ]; then
-    echo 'spatch is part of the Coccinelle project and is available at http://coccinelle.lip6.fr/'
-    exit 1
-fi
-
 if [ "$MODE" = "" ] ; then
     if [ "$ONLINE" = "0" ] ; then
 	echo 'You have not explicitly specified the mode to use. Using default "report" mode.'
-- 
2.8.2

[PATCH 2/4] coccicheck: enable paramap support

[Luis R. Rodriguez]

Coccinelle has had paramap support since 1.0.2, this means
it supports --jobs, enabling built-in multithreaded functionality,
instead of needing one to script it out. Just look for --jobs
in the help output to determine if this is supported.

Also enable the load balancing to be dynamic, so that if a
thread finishes early we keep feeding it.

If --jobs is not supported we fallback to the old mechanism.

As a small example, prior to this change, on an 8-core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report
...

real    29m14.912s
user    103m1.796s
sys     0m4.464s

After:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index aa5e78fba270..eeb5fdc142ca 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
     exit 1
 fi
 
-trap kill_running SIGTERM SIGINT
-declare -a SPATCH_PID
+USE_JOBS="no"
+$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
@@ -82,7 +82,21 @@ if [ "$ONLINE" = "0" ] ; then
     echo ''
 fi
 
-run_cmd() {
+if [ "$USE_JOBS" = "no" ]; then
+	trap kill_running SIGTERM SIGINT
+	declare -a SPATCH_PID
+else
+	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
+fi
+
+run_cmd_paramap() {
+	if [ $VERBOSE -ne 0 ] ; then
+		echo "Running ($NPROC in parallel): $@"
+	fi
+	$@
+}
+
+run_cmd_old() {
 	local i
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
@@ -97,6 +111,14 @@ run_cmd() {
 	wait
 }
 
+run_cmd() {
+	if [ "$USE_JOBS" = "yes" ]; then
+		run_cmd_paramap $@
+	else
+		run_cmd_old $@
+	fi
+}
+
 kill_running() {
 	for i in $(seq 0 $(( NPROC - 1 )) ); do
 		if [ $VERBOSE -eq 2 ] ; then
-- 
2.8.2

Re: [Cocci] [PATCH 2/4] coccicheck: enable paramap support

[SF Markus Elfring]

> Also enable the load balancing to be dynamic, so that
> if a thread finishes early we keep feeding it.

Is this functionality influenced by the parameter "chunksize"?

Regards,
Markus

Re: [Cocci] [PATCH 2/4] coccicheck: enable paramap support

[Julia Lawall]

On Sat, 11 Jun 2016, SF Markus Elfring wrote:

> > Also enable the load balancing to be dynamic, so that
> > if a thread finishes early we keep feeding it.
> 
> Is this functionality influenced by the parameter "chunksize"?

Yes, without chunksize the distribution of work to processes is static.

julia

[PATCH 3/4] scripts: add glimpse.sh for indexing the kernel

[Luis R. Rodriguez]

Glimpse is a tool you can use to index the kernel. The tool
was recently open sourced under the ISC license and can be
obtained at:

https://github.com/mcgrof/glimpse.git

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/glimpse.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100755 scripts/glimpse.sh

diff --git a/scripts/glimpse.sh b/scripts/glimpse.sh
new file mode 100755
index 000000000000..5fd24e49f31b
--- /dev/null
+++ b/scripts/glimpse.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+DIR=$(dirname $(readlink -f $0))
+DIR="${DIR}/../"
+
+GLIMPSEINDEX="`which ${GLIMPSEINDEX:=glimpseindex}`"
+if [ ! -x "$GLIMPSEINDEX" ]; then
+	echo 'glimpseindex can be obtained at https://github.com/mcgrof/glimpse.git'
+	exit 1
+fi
+
+find $DIR/* -name "*.[ch]" | $GLIMPSEINDEX -o -H . -F
-- 
2.8.2

Re: [Cocci] [PATCH 3/4] scripts: add glimpse.sh for indexing the kernel

[SF Markus Elfring]

> Glimpse is a tool you can use to index the kernel. The tool
> was recently open sourced under the ISC license and can be
> obtained at:

How do you think about to mention the script addition also directly in
the commit message?


> @@ -0,0 +1,12 @@
> +#!/bin/bash
> +
> +DIR=$(dirname $(readlink -f $0))
> +DIR="${DIR}/../"

Would you like to use the following variable assignment (instead of two
before)?

+DIR="$(dirname $(readlink -f $0))/../"


By the way: How are the chances to achieve further software improvements?
https://github.com/gvelez17/glimpse/issues

Regards,
Markus

[PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

Enable indexing optimizations heuristics. Coccinelle has
support to make use of its own enhanced "grep" mechanisms
instead of using regular grep for searching code 'coccigrep',
in practice though this seems to not perform better than
regular grep however its expected to help with some use cases
so we use that if you have no other indexing options in place
available.

Since git has its own index, support for using 'git grep' has been
added to Coccinelle, that should on average perform better than
using the internal cocci grep, and regular grep. Lastly, Coccinelle
has had support for glimpseindex for a long while, however the
tool was previously closed source, its now open sourced, and
provides the best performance, so support that if we can detect
you have a glimpse index.

These tests have been run on an 8 core system:

Before:

$ export COCCI=scripts/coccinelle/free/kfree.cocci
$ time make coccicheck MODE=report

Before this patch with no indexing or anything:

real    16m22.435s
user    128m30.060s
sys     0m2.712s

Using coccigrep (after this patch if you have no .git):

real    16m27.650s
user    128m47.904s
sys     0m2.176s

If you have .git and therefore use gitgrep:

real    16m21.220s
user    129m30.940s
sys     0m2.060s

And if you have a .glimpse_index:

real    16m14.794s
user    128m42.356s
sys     0m1.880s

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 scripts/coccicheck | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/scripts/coccicheck b/scripts/coccicheck
index eeb5fdc142ca..f31c9a152559 100755
--- a/scripts/coccicheck
+++ b/scripts/coccicheck
@@ -5,6 +5,8 @@
 # version 1.0.0-rc11.
 #
 
+DIR=$(dirname $(readlink -f $0))
+DIR="${DIR}/../"
 SPATCH="`which ${SPATCH:=spatch}`"
 
 if [ ! -x "$SPATCH" ]; then
@@ -15,6 +17,20 @@ fi
 USE_JOBS="no"
 $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
 
+# 0. --use-glimpse currently outperforms all. Refer
+#    to scripts/glimpse.sh for details.
+# 1. Second best is --use-gitgrep, this is very comparable to --use-glimpse
+# 2. Use --use-coccigrep if no indexing options are available and your
+#    version of coccinelle supports it
+USE_GLIMPSE="no"
+$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f $DIR/.glimpse_index ] && USE_GLIMPSE="yes"
+
+USE_GITGREP="no"
+$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && USE_GITGREP="yes"
+
+USE_COCCIGREP="no"
+$SPATCH --help | grep "\-\-use\-coccigrep" > /dev/null && USE_COCCIGREP="yes"
+
 # The verbosity may be set by the environmental parameter V=
 # as for example with 'make V=1 coccicheck'
 
@@ -89,6 +105,14 @@ else
 	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
 fi
 
+if [ "$USE_GLIMPSE" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-glimpse"
+elif [ "$USE_GITGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-gitgrep"
+elif [ "$USE_COCCIGREP" = "yes" ]; then
+	OPTIONS="$OPTIONS --use-coccigrep"
+fi
+
 run_cmd_paramap() {
 	if [ $VERBOSE -ne 0 ] ; then
 		echo "Running ($NPROC in parallel): $@"
-- 
2.8.2

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:

> Enable indexing optimizations heuristics. Coccinelle has
> support to make use of its own enhanced "grep" mechanisms
> instead of using regular grep for searching code 'coccigrep',
> in practice though this seems to not perform better than
> regular grep however its expected to help with some use cases
> so we use that if you have no other indexing options in place
> available.
> 
> Since git has its own index, support for using 'git grep' has been
> added to Coccinelle, that should on average perform better than
> using the internal cocci grep, and regular grep. Lastly, Coccinelle
> has had support for glimpseindex for a long while, however the
> tool was previously closed source, its now open sourced, and
> provides the best performance, so support that if we can detect
> you have a glimpse index.
> 
> These tests have been run on an 8 core system:
> 
> Before:
> 
> $ export COCCI=scripts/coccinelle/free/kfree.cocci
> $ time make coccicheck MODE=report
> 
> Before this patch with no indexing or anything:
> 
> real    16m22.435s
> user    128m30.060s
> sys     0m2.712s
> 
> Using coccigrep (after this patch if you have no .git):
> 
> real    16m27.650s
> user    128m47.904s
> sys     0m2.176s
> 
> If you have .git and therefore use gitgrep:
> 
> real    16m21.220s
> user    129m30.940s
> sys     0m2.060s
> 
> And if you have a .glimpse_index:
> 
> real    16m14.794s
> user    128m42.356s
> sys     0m1.880s

I don't see any convincing differences in these times.

I believe that Coccinelle's internal grep is always used, even with no 
option.

I'm puzzled why glimpse gives no benefit.

julia


> 
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
>  scripts/coccicheck | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
> 
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index eeb5fdc142ca..f31c9a152559 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -5,6 +5,8 @@
>  # version 1.0.0-rc11.
>  #
>  
> +DIR=$(dirname $(readlink -f $0))
> +DIR="${DIR}/../"
>  SPATCH="`which ${SPATCH:=spatch}`"
>  
>  if [ ! -x "$SPATCH" ]; then
> @@ -15,6 +17,20 @@ fi
>  USE_JOBS="no"
>  $SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
>  
> +# 0. --use-glimpse currently outperforms all. Refer
> +#    to scripts/glimpse.sh for details.
> +# 1. Second best is --use-gitgrep, this is very comparable to --use-glimpse
> +# 2. Use --use-coccigrep if no indexing options are available and your
> +#    version of coccinelle supports it
> +USE_GLIMPSE="no"
> +$SPATCH --help | grep "\-\-use\-glimpse" > /dev/null && [ -f $DIR/.glimpse_index ] && USE_GLIMPSE="yes"
> +
> +USE_GITGREP="no"
> +$SPATCH --help | grep "\-\-use\-gitgrep" > /dev/null && [ -d $DIR/.git ] && USE_GITGREP="yes"
> +
> +USE_COCCIGREP="no"
> +$SPATCH --help | grep "\-\-use\-coccigrep" > /dev/null && USE_COCCIGREP="yes"
> +
>  # The verbosity may be set by the environmental parameter V=
>  # as for example with 'make V=1 coccicheck'
>  
> @@ -89,6 +105,14 @@ else
>  	OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
>  fi
>  
> +if [ "$USE_GLIMPSE" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-glimpse"
> +elif [ "$USE_GITGREP" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-gitgrep"
> +elif [ "$USE_COCCIGREP" = "yes" ]; then
> +	OPTIONS="$OPTIONS --use-coccigrep"
> +fi
> +
>  run_cmd_paramap() {
>  	if [ $VERBOSE -ne 0 ] ; then
>  		echo "Running ($NPROC in parallel): $@"
> -- 
> 2.8.2
> 
> 

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> 
> 
> On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> 
> > Enable indexing optimizations heuristics. Coccinelle has
> > support to make use of its own enhanced "grep" mechanisms
> > instead of using regular grep for searching code 'coccigrep',
> > in practice though this seems to not perform better than
> > regular grep however its expected to help with some use cases
> > so we use that if you have no other indexing options in place
> > available.
> > 
> > Since git has its own index, support for using 'git grep' has been
> > added to Coccinelle, that should on average perform better than
> > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > has had support for glimpseindex for a long while, however the
> > tool was previously closed source, its now open sourced, and
> > provides the best performance, so support that if we can detect
> > you have a glimpse index.
> > 
> > These tests have been run on an 8 core system:
> > 
> > Before:
> > 
> > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > $ time make coccicheck MODE=report
> > 
> > Before this patch with no indexing or anything:
> > 
> > real    16m22.435s
> > user    128m30.060s
> > sys     0m2.712s
> > 
> > Using coccigrep (after this patch if you have no .git):
> > 
> > real    16m27.650s
> > user    128m47.904s
> > sys     0m2.176s
> > 
> > If you have .git and therefore use gitgrep:
> > 
> > real    16m21.220s
> > user    129m30.940s
> > sys     0m2.060s
> > 
> > And if you have a .glimpse_index:
> > 
> > real    16m14.794s
> > user    128m42.356s
> > sys     0m1.880s
> 
> I don't see any convincing differences in these times.
> 
> I believe that Coccinelle's internal grep is always used, even with no 
> option.

Ah that would explain it. This uses coccinelle 1.0.5, is the default
there to use --use-coccigrep if no other index is specified ?

> I'm puzzled why glimpse gives no benefit.

Well, slightly better.

  Luis

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > Enable indexing optimizations heuristics. Coccinelle has
> > > support to make use of its own enhanced "grep" mechanisms
> > > instead of using regular grep for searching code 'coccigrep',
> > > in practice though this seems to not perform better than
> > > regular grep however its expected to help with some use cases
> > > so we use that if you have no other indexing options in place
> > > available.
> > > 
> > > Since git has its own index, support for using 'git grep' has been
> > > added to Coccinelle, that should on average perform better than
> > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > has had support for glimpseindex for a long while, however the
> > > tool was previously closed source, its now open sourced, and
> > > provides the best performance, so support that if we can detect
> > > you have a glimpse index.
> > > 
> > > These tests have been run on an 8 core system:
> > > 
> > > Before:
> > > 
> > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > $ time make coccicheck MODE=report
> > > 
> > > Before this patch with no indexing or anything:
> > > 
> > > real    16m22.435s
> > > user    128m30.060s
> > > sys     0m2.712s
> > > 
> > > Using coccigrep (after this patch if you have no .git):
> > > 
> > > real    16m27.650s
> > > user    128m47.904s
> > > sys     0m2.176s
> > > 
> > > If you have .git and therefore use gitgrep:
> > > 
> > > real    16m21.220s
> > > user    129m30.940s
> > > sys     0m2.060s
> > > 
> > > And if you have a .glimpse_index:
> > > 
> > > real    16m14.794s
> > > user    128m42.356s
> > > sys     0m1.880s
> > 
> > I don't see any convincing differences in these times.
> > 
> > I believe that Coccinelle's internal grep is always used, even with no 
> > option.
> 
> Ah that would explain it. This uses coccinelle 1.0.5, is the default
> there to use --use-coccigrep if no other index is specified ?

It has been the default for a long time.

> > I'm puzzled why glimpse gives no benefit.
> 
> Well, slightly better.

No, it should be much better.  You would have to look at the standard 
error to see if you are getting any benefit.  There should be very few 
occurrences of Skipping if you are really using glimpse.  In any case, if 
you asked for glimpse and it was not able to provide it, there should be 
warning messages at the top of stderr.

julia

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

[-- Attachment #1: Type: text/plain, Size: 278 bytes --]

> > Well, slightly better.
> 
> No, it should be much better.  You would have to look at the standard 

I use id-utils regularly and it is indeed at least a magnitude better.
The indexing often pays off already with the first coccinelle run for
me. Highly recommended.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Fri, Jun 10, 2016 at 11:43:57PM +0200, Wolfram Sang wrote:
> > > Well, slightly better.
> > 
> > No, it should be much better.  You would have to look at the standard 
> 
> I use id-utils regularly and it is indeed at least a magnitude better.
> The indexing often pays off already with the first coccinelle run for
> me. Highly recommended.

AFAICT coccinelle does not have integration support for id-utils though.

  Luis

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

[-- Attachment #1: Type: text/plain, Size: 166 bytes --]

> AFAICT coccinelle does not have integration support for id-utils though.

I used it just today ;) -- "--use-idutils ./ID"

ID was generated with simple 'mkid -s'.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Fri, Jun 10, 2016 at 11:51:26PM +0200, Wolfram Sang wrote:
> > AFAICT coccinelle does not have integration support for id-utils though.
> 
> I used it just today ;) -- "--use-idutils ./ID"
> 
> ID was generated with simple 'mkid -s'.
> 

Sweet, testing that now.

  Luis

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Sat, Jun 11, 2016 at 12:08:32AM +0200, Luis R. Rodriguez wrote:
> On Fri, Jun 10, 2016 at 11:51:26PM +0200, Wolfram Sang wrote:
> > > AFAICT coccinelle does not have integration support for id-utils though.
> > 
> > I used it just today ;) -- "--use-idutils ./ID"
> > 
> > ID was generated with simple 'mkid -s'.
> > 
> 
> Sweet, testing that now.

Boooyah :D

real    16m11.692s
user    127m50.388s
sys     0m2.168s

  Luis

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

[-- Attachment #1: Type: text/plain, Size: 110 bytes --]


> real    16m11.692s
> user    127m50.388s
> sys     0m2.168s

That's better but not a magnitude, I wonder.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Sat, 11 Jun 2016, Wolfram Sang wrote:

> 
> > real    16m11.692s
> > user    127m50.388s
> > sys     0m2.168s
> 
> That's better but not a magnitude, I wonder.

I think that it is because the filtering that Coccinelle does already 
works pretty well, and there are quite a lot of files (7514) that contain 
kfree.

julia

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

[-- Attachment #1: Type: text/plain, Size: 141 bytes --]


> works pretty well, and there are quite a lot of files (7514) that contain 
> kfree.

Ah, kfree. That explains, I missed that info.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Sat, Jun 11, 2016 at 07:54:39AM +0200, Julia Lawall wrote:
> 
> 
> On Sat, 11 Jun 2016, Wolfram Sang wrote:
> 
> > 
> > > real    16m11.692s
> > > user    127m50.388s
> > > sys     0m2.168s
> > 
> > That's better but not a magnitude, I wonder.
> 
> I think that it is because the filtering that Coccinelle does already 
> works pretty well, and there are quite a lot of files (7514) that contain 
> kfree.

Is there another scripts/coccinelle/ file I can use to test against to demo
against glimpse/idutils/gitgrep best?

  Luis

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

[-- Attachment #1: Type: text/plain, Size: 257 bytes --]


> Is there another scripts/coccinelle/ file I can use to test against to demo
> against glimpse/idutils/gitgrep best?

I'd think this one may be a candidate:

scripts/coccinelle/misc/irqf_oneshot.cocci

Not too many, but quite some matches over the tree.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Mon, 13 Jun 2016, Wolfram Sang wrote:

> 
> > Is there another scripts/coccinelle/ file I can use to test against to demo
> > against glimpse/idutils/gitgrep best?
> 
> I'd think this one may be a candidate:
> 
> scripts/coccinelle/misc/irqf_oneshot.cocci
> 
> Not too many, but quite some matches over the tree.

Seems like a reasonable choice.

julia

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Mon, Jun 13, 2016 at 09:48:30PM +0200, Julia Lawall wrote:
> 
> 
> On Mon, 13 Jun 2016, Wolfram Sang wrote:
> 
> > 
> > > Is there another scripts/coccinelle/ file I can use to test against to demo
> > > against glimpse/idutils/gitgrep best?
> > 
> > I'd think this one may be a candidate:
> > 
> > scripts/coccinelle/misc/irqf_oneshot.cocci
> > 
> > Not too many, but quite some matches over the tree.
> 
> Seems like a reasonable choice.

With this one on a 32-core system, I get:

glimpse:
real    0m6.549s
user    0m49.136s
sys     0m3.076s

idutils:
real    0m6.749s
user    0m51.936s
sys     0m3.876s

gitgrep:
real    0m6.805s
user    0m51.572s
sys     0m4.432s

coccigrep:
real    0m16.369s
user    0m58.712s
sys     0m5.064s

I redirected stderr to stdout, and verified glimpse output has:

glimpse request = request_threaded_irq

Does this match expectations?

  Luis

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:

> On Mon, Jun 13, 2016 at 09:48:30PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Mon, 13 Jun 2016, Wolfram Sang wrote:
> > 
> > > 
> > > > Is there another scripts/coccinelle/ file I can use to test against to demo
> > > > against glimpse/idutils/gitgrep best?
> > > 
> > > I'd think this one may be a candidate:
> > > 
> > > scripts/coccinelle/misc/irqf_oneshot.cocci
> > > 
> > > Not too many, but quite some matches over the tree.
> > 
> > Seems like a reasonable choice.
> 
> With this one on a 32-core system, I get:
> 
> glimpse:
> real    0m6.549s
> user    0m49.136s
> sys     0m3.076s
> 
> idutils:
> real    0m6.749s
> user    0m51.936s
> sys     0m3.876s
> 
> gitgrep:
> real    0m6.805s
> user    0m51.572s
> sys     0m4.432s
> 
> coccigrep:
> real    0m16.369s
> user    0m58.712s
> sys     0m5.064s
> 
> I redirected stderr to stdout, and verified glimpse output has:
> 
> glimpse request = request_threaded_irq
> 
> Does this match expectations?

Yes.  I'm not sure that gitgrep would work as well when there are multiple 
keywords.  It may descend to coccigrep.

julia

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Sat, 11 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 10, 2016 at 11:51:26PM +0200, Wolfram Sang wrote:
> > > AFAICT coccinelle does not have integration support for id-utils though.
> > 
> > I used it just today ;) -- "--use-idutils ./ID"
> > 
> > ID was generated with simple 'mkid -s'.
> > 
> 
> Sweet, testing that now.

It's not as efficient as glimpse because the query language is simpler.  
So more filtering has to be done at the ocaml level.  But it's probably 
fine in most cases.

julia

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Wolfram Sang]

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]


> It's not as efficient as glimpse because the query language is simpler.  

Interesting, what is missing compared to glimpse?

> So more filtering has to be done at the ocaml level.  But it's probably 
> fine in most cases.

For me, it has two advantages over glimpse:

a) it is in the debian package repository
b) the same database can be used with the code browser 'seascope'
   which can do nice things by feeding ctags on the fly with data
   from id-utils.

Mileages vary, of course, just wanted to mention it to give pointers.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Sat, 11 Jun 2016, Wolfram Sang wrote:

> 
> > It's not as efficient as glimpse because the query language is simpler.  
> 
> Interesting, what is missing compared to glimpse?

Glimpse allows queries that are arbitrary formulas, up to a limited level 
of complexity, involving both && and ||.  For idutils, Coccinelle runs lid 
on each of the tokens in the formula, and then does unions and 
intersections on the result.

julia

> > So more filtering has to be done at the ocaml level.  But it's probably 
> > fine in most cases.
> 
> For me, it has two advantages over glimpse:
> 
> a) it is in the debian package repository
> b) the same database can be used with the code browser 'seascope'
>    which can do nice things by feeding ctags on the fly with data
>    from id-utils.
> 
> Mileages vary, of course, just wanted to mention it to give pointers.
> 
> 

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Fri, 10 Jun 2016, Wolfram Sang wrote:

> > AFAICT coccinelle does not have integration support for id-utils though.
> 
> I used it just today ;) -- "--use-idutils ./ID"
> 
> ID was generated with simple 'mkid -s'.

Coccinelle includes a script scripts/idutils_index.sh

This does mkid -i C --output .id-utils.index *

julia

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Sat, Jun 11, 2016 at 07:24:51AM +0200, Julia Lawall wrote:
> 
> 
> On Fri, 10 Jun 2016, Wolfram Sang wrote:
> 
> > > AFAICT coccinelle does not have integration support for id-utils though.
> > 
> > I used it just today ;) -- "--use-idutils ./ID"
> > 
> > ID was generated with simple 'mkid -s'.
> 
> Coccinelle includes a script scripts/idutils_index.sh
> 
> This does mkid -i C --output .id-utils.index *

I'll add support for detecting both. An issue of course is if any of these
indexes grows stale. So I'll advise against these and recommend gitgrep
unless the user has a hook to update index regularly.

  Luis

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 10, 2016 at 11:43:57PM +0200, Wolfram Sang wrote:
> > > > Well, slightly better.
> > > 
> > > No, it should be much better.  You would have to look at the standard 
> > 
> > I use id-utils regularly and it is indeed at least a magnitude better.
> > The indexing often pays off already with the first coccinelle run for
> > me. Highly recommended.
> 
> AFAICT coccinelle does not have integration support for id-utils though.

Yes it does, for years.

julia

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> 
> 
> On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > support to make use of its own enhanced "grep" mechanisms
> > > > instead of using regular grep for searching code 'coccigrep',
> > > > in practice though this seems to not perform better than
> > > > regular grep however its expected to help with some use cases
> > > > so we use that if you have no other indexing options in place
> > > > available.
> > > > 
> > > > Since git has its own index, support for using 'git grep' has been
> > > > added to Coccinelle, that should on average perform better than
> > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > > has had support for glimpseindex for a long while, however the
> > > > tool was previously closed source, its now open sourced, and
> > > > provides the best performance, so support that if we can detect
> > > > you have a glimpse index.
> > > > 
> > > > These tests have been run on an 8 core system:
> > > > 
> > > > Before:
> > > > 
> > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > $ time make coccicheck MODE=report
> > > > 
> > > > Before this patch with no indexing or anything:
> > > > 
> > > > real    16m22.435s
> > > > user    128m30.060s
> > > > sys     0m2.712s
> > > > 
> > > > Using coccigrep (after this patch if you have no .git):
> > > > 
> > > > real    16m27.650s
> > > > user    128m47.904s
> > > > sys     0m2.176s
> > > > 
> > > > If you have .git and therefore use gitgrep:
> > > > 
> > > > real    16m21.220s
> > > > user    129m30.940s
> > > > sys     0m2.060s
> > > > 
> > > > And if you have a .glimpse_index:
> > > > 
> > > > real    16m14.794s
> > > > user    128m42.356s
> > > > sys     0m1.880s
> > > 
> > > I don't see any convincing differences in these times.
> > > 
> > > I believe that Coccinelle's internal grep is always used, even with no 
> > > option.
> > 
> > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > there to use --use-coccigrep if no other index is specified ?
> 
> It has been the default for a long time.
> 
> > > I'm puzzled why glimpse gives no benefit.
> > 
> > Well, slightly better.
> 
> No, it should be much better.  You would have to look at the standard 
> error to see if you are getting any benefit.  There should be very few 
> occurrences of Skipping if you are really using glimpse.  In any case, if 
> you asked for glimpse and it was not able to provide it, there should be 
> warning messages at the top of stderr.

I'll redirect stderr to stdout by default when parmap support is used then.

  Luis

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:

> On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > > support to make use of its own enhanced "grep" mechanisms
> > > > > instead of using regular grep for searching code 'coccigrep',
> > > > > in practice though this seems to not perform better than
> > > > > regular grep however its expected to help with some use cases
> > > > > so we use that if you have no other indexing options in place
> > > > > available.
> > > > > 
> > > > > Since git has its own index, support for using 'git grep' has been
> > > > > added to Coccinelle, that should on average perform better than
> > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > > > has had support for glimpseindex for a long while, however the
> > > > > tool was previously closed source, its now open sourced, and
> > > > > provides the best performance, so support that if we can detect
> > > > > you have a glimpse index.
> > > > > 
> > > > > These tests have been run on an 8 core system:
> > > > > 
> > > > > Before:
> > > > > 
> > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > > $ time make coccicheck MODE=report
> > > > > 
> > > > > Before this patch with no indexing or anything:
> > > > > 
> > > > > real    16m22.435s
> > > > > user    128m30.060s
> > > > > sys     0m2.712s
> > > > > 
> > > > > Using coccigrep (after this patch if you have no .git):
> > > > > 
> > > > > real    16m27.650s
> > > > > user    128m47.904s
> > > > > sys     0m2.176s
> > > > > 
> > > > > If you have .git and therefore use gitgrep:
> > > > > 
> > > > > real    16m21.220s
> > > > > user    129m30.940s
> > > > > sys     0m2.060s
> > > > > 
> > > > > And if you have a .glimpse_index:
> > > > > 
> > > > > real    16m14.794s
> > > > > user    128m42.356s
> > > > > sys     0m1.880s
> > > > 
> > > > I don't see any convincing differences in these times.
> > > > 
> > > > I believe that Coccinelle's internal grep is always used, even with no 
> > > > option.
> > > 
> > > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > > there to use --use-coccigrep if no other index is specified ?
> > 
> > It has been the default for a long time.
> > 
> > > > I'm puzzled why glimpse gives no benefit.
> > > 
> > > Well, slightly better.
> > 
> > No, it should be much better.  You would have to look at the standard 
> > error to see if you are getting any benefit.  There should be very few 
> > occurrences of Skipping if you are really using glimpse.  In any case, if 
> > you asked for glimpse and it was not able to provide it, there should be 
> > warning messages at the top of stderr.
> 
> I'll redirect stderr to stdout by default when parmap support is used then.

Usually I put them in different files.

julia

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> 
> 
> On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > > > 
> > > > > 
> > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > > 
> > > > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > > > support to make use of its own enhanced "grep" mechanisms
> > > > > > instead of using regular grep for searching code 'coccigrep',
> > > > > > in practice though this seems to not perform better than
> > > > > > regular grep however its expected to help with some use cases
> > > > > > so we use that if you have no other indexing options in place
> > > > > > available.
> > > > > > 
> > > > > > Since git has its own index, support for using 'git grep' has been
> > > > > > added to Coccinelle, that should on average perform better than
> > > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > > > > has had support for glimpseindex for a long while, however the
> > > > > > tool was previously closed source, its now open sourced, and
> > > > > > provides the best performance, so support that if we can detect
> > > > > > you have a glimpse index.
> > > > > > 
> > > > > > These tests have been run on an 8 core system:
> > > > > > 
> > > > > > Before:
> > > > > > 
> > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > > > $ time make coccicheck MODE=report
> > > > > > 
> > > > > > Before this patch with no indexing or anything:
> > > > > > 
> > > > > > real    16m22.435s
> > > > > > user    128m30.060s
> > > > > > sys     0m2.712s
> > > > > > 
> > > > > > Using coccigrep (after this patch if you have no .git):
> > > > > > 
> > > > > > real    16m27.650s
> > > > > > user    128m47.904s
> > > > > > sys     0m2.176s
> > > > > > 
> > > > > > If you have .git and therefore use gitgrep:
> > > > > > 
> > > > > > real    16m21.220s
> > > > > > user    129m30.940s
> > > > > > sys     0m2.060s
> > > > > > 
> > > > > > And if you have a .glimpse_index:
> > > > > > 
> > > > > > real    16m14.794s
> > > > > > user    128m42.356s
> > > > > > sys     0m1.880s
> > > > > 
> > > > > I don't see any convincing differences in these times.
> > > > > 
> > > > > I believe that Coccinelle's internal grep is always used, even with no 
> > > > > option.
> > > > 
> > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > > > there to use --use-coccigrep if no other index is specified ?
> > > 
> > > It has been the default for a long time.
> > > 
> > > > > I'm puzzled why glimpse gives no benefit.
> > > > 
> > > > Well, slightly better.
> > > 
> > > No, it should be much better.  You would have to look at the standard 
> > > error to see if you are getting any benefit.  There should be very few 
> > > occurrences of Skipping if you are really using glimpse.  In any case, if 
> > > you asked for glimpse and it was not able to provide it, there should be 
> > > warning messages at the top of stderr.
> > 
> > I'll redirect stderr to stdout by default when parmap support is used then.
> 
> Usually I put them in different files.

We can do that as well but I would only want to deal with parmap support case.
Any preference? How about .coccicheck.stderr.$PID where PID would be the PID of
the shell script?

  Luis

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:

> On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Fri, Jun 10, 2016 at 11:21:28PM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > > On Fri, Jun 10, 2016 at 11:02:38PM +0200, Julia Lawall wrote:
> > > > > > 
> > > > > > 
> > > > > > On Fri, 10 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > 
> > > > > > > Enable indexing optimizations heuristics. Coccinelle has
> > > > > > > support to make use of its own enhanced "grep" mechanisms
> > > > > > > instead of using regular grep for searching code 'coccigrep',
> > > > > > > in practice though this seems to not perform better than
> > > > > > > regular grep however its expected to help with some use cases
> > > > > > > so we use that if you have no other indexing options in place
> > > > > > > available.
> > > > > > > 
> > > > > > > Since git has its own index, support for using 'git grep' has been
> > > > > > > added to Coccinelle, that should on average perform better than
> > > > > > > using the internal cocci grep, and regular grep. Lastly, Coccinelle
> > > > > > > has had support for glimpseindex for a long while, however the
> > > > > > > tool was previously closed source, its now open sourced, and
> > > > > > > provides the best performance, so support that if we can detect
> > > > > > > you have a glimpse index.
> > > > > > > 
> > > > > > > These tests have been run on an 8 core system:
> > > > > > > 
> > > > > > > Before:
> > > > > > > 
> > > > > > > $ export COCCI=scripts/coccinelle/free/kfree.cocci
> > > > > > > $ time make coccicheck MODE=report
> > > > > > > 
> > > > > > > Before this patch with no indexing or anything:
> > > > > > > 
> > > > > > > real    16m22.435s
> > > > > > > user    128m30.060s
> > > > > > > sys     0m2.712s
> > > > > > > 
> > > > > > > Using coccigrep (after this patch if you have no .git):
> > > > > > > 
> > > > > > > real    16m27.650s
> > > > > > > user    128m47.904s
> > > > > > > sys     0m2.176s
> > > > > > > 
> > > > > > > If you have .git and therefore use gitgrep:
> > > > > > > 
> > > > > > > real    16m21.220s
> > > > > > > user    129m30.940s
> > > > > > > sys     0m2.060s
> > > > > > > 
> > > > > > > And if you have a .glimpse_index:
> > > > > > > 
> > > > > > > real    16m14.794s
> > > > > > > user    128m42.356s
> > > > > > > sys     0m1.880s
> > > > > > 
> > > > > > I don't see any convincing differences in these times.
> > > > > > 
> > > > > > I believe that Coccinelle's internal grep is always used, even with no 
> > > > > > option.
> > > > > 
> > > > > Ah that would explain it. This uses coccinelle 1.0.5, is the default
> > > > > there to use --use-coccigrep if no other index is specified ?
> > > > 
> > > > It has been the default for a long time.
> > > > 
> > > > > > I'm puzzled why glimpse gives no benefit.
> > > > > 
> > > > > Well, slightly better.
> > > > 
> > > > No, it should be much better.  You would have to look at the standard 
> > > > error to see if you are getting any benefit.  There should be very few 
> > > > occurrences of Skipping if you are really using glimpse.  In any case, if 
> > > > you asked for glimpse and it was not able to provide it, there should be 
> > > > warning messages at the top of stderr.
> > > 
> > > I'll redirect stderr to stdout by default when parmap support is used then.
> > 
> > Usually I put them in different files.
> 
> We can do that as well but I would only want to deal with parmap support 
> case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> be the PID of the shell script?

I don't understand the connection with parmap.

Originally our use of parmap made output files based on pids.  Maybe this 
is the default for parmap.  I found this completely unusable.  I guess one 
could look at the dates to see which file is the most recent one, but it 
seems tedious.  If you are putting the standard output in x.out, then put 
the standard error in x.err.

julia

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> 
> 
> On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > 
> > > Usually I put them in different files.
> > 
> > We can do that as well but I would only want to deal with parmap support 
> > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > be the PID of the shell script?
> 
> I don't understand the connection with parmap.

When parmap support is not available the cocciscript will currently
disregard stderr, output is provided as it comes to stdout from each
thread I guess.

> Originally our use of parmap made output files based on pids.  Maybe this 
> is the default for parmap.  I found this completely unusable.  I guess one 
> could look at the dates to see which file is the most recent one, but it 
> seems tedious.  If you are putting the standard output in x.out, then put 
> the standard error in x.err.

I'll use ${DIR}/coccicheck.$$.err for stderr.

  Luis

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:

> On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > 
> > 
> > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > 
> > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > 
> > > > Usually I put them in different files.
> > > 
> > > We can do that as well but I would only want to deal with parmap support 
> > > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > > be the PID of the shell script?
> > 
> > I don't understand the connection with parmap.
> 
> When parmap support is not available the cocciscript will currently
> disregard stderr, output is provided as it comes to stdout from each
> thread I guess.

Deepa's recent patch to coccicheck made apparent that Coccicheck uses 
--very-quiet, so there is standard error.

> > Originally our use of parmap made output files based on pids.  Maybe this 
> > is the default for parmap.  I found this completely unusable.  I guess one 
> > could look at the dates to see which file is the most recent one, but it 
> > seems tedious.  If you are putting the standard output in x.out, then put 
> > the standard error in x.err.
> 
> I'll use ${DIR}/coccicheck.$$.err for stderr.

What is ${DIR}? and what is $$?

julia

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> 
> 
> On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > 
> > > > > 
> > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > 
> > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > 
> > > > > Usually I put them in different files.
> > > > 
> > > > We can do that as well but I would only want to deal with parmap support 
> > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > > > be the PID of the shell script?
> > > 
> > > I don't understand the connection with parmap.
> > 
> > When parmap support is not available the cocciscript will currently
> > disregard stderr, output is provided as it comes to stdout from each
> > thread I guess.
> 
> Deepa's recent patch to coccicheck made apparent that Coccicheck uses 
> --very-quiet, so there is standard error.

OK I'm disegarding the redirect for non-parmap for now but we'd have to
determine if we want to append or add one per PID... I rather leave that
stuff as-is and encourage folks to upgrade coccinelle.

> > > Originally our use of parmap made output files based on pids.  Maybe this 
> > > is the default for parmap.  I found this completely unusable.  I guess one 
> > > could look at the dates to see which file is the most recent one, but it 
> > > seems tedious.  If you are putting the standard output in x.out, then put 
> > > the standard error in x.err.
> > 
> > I'll use ${DIR}/coccicheck.$$.err for stderr.
> 
> What is ${DIR}? and what is $$?

When you run scripts/coccicheck we take the absolute directory
of it and then go down one level of directory, so in this case it
would be the base directory of the Linux kernel.

$$ is the PID of the bash script.

  Luis

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:

> On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > 
> > 
> > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > 
> > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > 
> > > > 
> > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > 
> > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > 
> > > > > > 
> > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > 
> > > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > > 
> > > > > > Usually I put them in different files.
> > > > > 
> > > > > We can do that as well but I would only want to deal with parmap support 
> > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > > > > be the PID of the shell script?
> > > > 
> > > > I don't understand the connection with parmap.
> > > 
> > > When parmap support is not available the cocciscript will currently
> > > disregard stderr, output is provided as it comes to stdout from each
> > > thread I guess.
> > 
> > Deepa's recent patch to coccicheck made apparent that Coccicheck uses 
> > --very-quiet, so there is standard error.
> 
> OK I'm disegarding the redirect for non-parmap for now but we'd have to
> determine if we want to append or add one per PID... I rather leave that
> stuff as-is and encourage folks to upgrade coccinelle.

If coccicheck is using --very-quiet, there will not be much stderr of 
interest when using parmap either.

I'm not sure to understan the issue about appending.  Appending what to 
what?  While Coccinelle is running, you have a directory with the same 
name as your semantic patch (or a directory name that you specify with 
--tmp-dir) that has separate files for each core's standard output and 
standard error.  At the end, when all the code has been treated, 
Coccinelle writes the successive stdouts to standard output, and the 
sucessive stderrs to standard error.

> > > > Originally our use of parmap made output files based on pids.  Maybe this 
> > > > is the default for parmap.  I found this completely unusable.  I guess one 
> > > > could look at the dates to see which file is the most recent one, but it 
> > > > seems tedious.  If you are putting the standard output in x.out, then put 
> > > > the standard error in x.err.
> > > 
> > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > 
> > What is ${DIR}? and what is $$?
> 
> When you run scripts/coccicheck we take the absolute directory
> of it and then go down one level of directory, so in this case it
> would be the base directory of the Linux kernel.
> 
> $$ is the PID of the bash script.

OK.  I still don't find PIDs useful, but I guess if we are talking about 
the entire output of coccicheck, there is not much else to do.  Normally, 
I don't want these files accumulating, and just write over the old ones.

julia

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote:
> 
> 
> On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > > 
> > > 
> > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > 
> > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > > 
> > > > > 
> > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > 
> > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > > 
> > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > > > 
> > > > > > > Usually I put them in different files.
> > > > > > 
> > > > > > We can do that as well but I would only want to deal with parmap support 
> > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would 
> > > > > > be the PID of the shell script?
> > > > > 
> > > > > I don't understand the connection with parmap.
> > > > 
> > > > When parmap support is not available the cocciscript will currently
> > > > disregard stderr, output is provided as it comes to stdout from each
> > > > thread I guess.
> > > 
> > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses 
> > > --very-quiet, so there is standard error.
> > 
> > OK I'm disegarding the redirect for non-parmap for now but we'd have to
> > determine if we want to append or add one per PID... I rather leave that
> > stuff as-is and encourage folks to upgrade coccinelle.
> 
> If coccicheck is using --very-quiet, there will not be much stderr of 
> interest when using parmap either.

OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet
is used ? Or does using --very-quiet suppress stderr ?

> I'm not sure to understan the issue about appending.  Appending what to 
> what?
> While Coccinelle is running, you have a directory with the same 
> name as your semantic patch (or a directory name that you specify with 
> --tmp-dir) that has separate files for each core's standard output and 
> standard error.  At the end, when all the code has been treated, 
> Coccinelle writes the successive stdouts to standard output, and the 
> sucessive stderrs to standard error.

Ah I see. OK yeah never mind about appending.

> > > > > Originally our use of parmap made output, specia files based on pids.  Maybe this 
> > > > > is the default for parmap.  I found this completely unusable.  I guess one 
> > > > > could look at the dates to see which file is the most recent one, but it 
> > > > > seems tedious.  If you are putting the standard output in x.out, then put 
> > > > > the standard error in x.err.
> > > > 
> > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > 
> > > What is ${DIR}? and what is $$?
> > 
> > When you run scripts/coccicheck we take the absolute directory
> > of it and then go down one level of directory, so in this case it
> > would be the base directory of the Linux kernel.
> > 
> > $$ is the PID of the bash script.
> 
> OK.  I still don't find PIDs useful, but I guess if we are talking about 
> the entire output of coccicheck, there is not much else to do.  Normally, 
> I don't want these files accumulating, and just write over the old ones.

Which is why I would much prefer to instead just redirect in coccicheck
case stderr to stdout from coccinelle. Is that preferred?

  Luis

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:

> On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote:
> >
> >
> > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> >
> > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > > >
> > > >
> > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > >
> > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > > >
> > > > > >
> > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > >
> > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > > >
> > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > > > >
> > > > > > > > Usually I put them in different files.
> > > > > > >
> > > > > > > We can do that as well but I would only want to deal with parmap support
> > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would
> > > > > > > be the PID of the shell script?
> > > > > >
> > > > > > I don't understand the connection with parmap.
> > > > >
> > > > > When parmap support is not available the cocciscript will currently
> > > > > disregard stderr, output is provided as it comes to stdout from each
> > > > > thread I guess.
> > > >
> > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses
> > > > --very-quiet, so there is standard error.
> > >
> > > OK I'm disegarding the redirect for non-parmap for now but we'd have to
> > > determine if we want to append or add one per PID... I rather leave that
> > > stuff as-is and encourage folks to upgrade coccinelle.
> >
> > If coccicheck is using --very-quiet, there will not be much stderr of
> > interest when using parmap either.
>
> OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet
> is used ? Or does using --very-quiet suppress stderr ?

--very-quiet suppresses most administrative messages that go to stderr.
There are still actual errors.  Bu you don't see eg what file is being
currently processed.

> > > > > > Originally our use of parmap made output, specia files based on pids.  Maybe this
> > > > > > is the default for parmap.  I found this completely unusable.  I guess one
> > > > > > could look at the dates to see which file is the most recent one, but it
> > > > > > seems tedious.  If you are putting the standard output in x.out, then put
> > > > > > the standard error in x.err.
> > > > >
> > > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > >
> > > > What is ${DIR}? and what is $$?
> > >
> > > When you run scripts/coccicheck we take the absolute directory
> > > of it and then go down one level of directory, so in this case it
> > > would be the base directory of the Linux kernel.
> > >
> > > $$ is the PID of the bash script.
> >
> > OK.  I still don't find PIDs useful, but I guess if we are talking about
> > the entire output of coccicheck, there is not much else to do.  Normally,
> > I don't want these files accumulating, and just write over the old ones.
>
> Which is why I would much prefer to instead just redirect in coccicheck
> case stderr to stdout from coccinelle. Is that preferred?

Then things will be merged in strange ways.

Why not just let the user decide what to do with these things?

julia

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 09:39:49AM +0200, Julia Lawall wrote:
> 
> 
> On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> 
> > On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote:
> > >
> > >
> > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > >
> > > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > > > >
> > > > >
> > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > > >
> > > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > >
> > > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > > > >
> > > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > > > > >
> > > > > > > > > Usually I put them in different files.
> > > > > > > >
> > > > > > > > We can do that as well but I would only want to deal with parmap support
> > > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would
> > > > > > > > be the PID of the shell script?
> > > > > > >
> > > > > > > I don't understand the connection with parmap.
> > > > > >
> > > > > > When parmap support is not available the cocciscript will currently
> > > > > > disregard stderr, output is provided as it comes to stdout from each
> > > > > > thread I guess.
> > > > >
> > > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses
> > > > > --very-quiet, so there is standard error.
> > > >
> > > > OK I'm disegarding the redirect for non-parmap for now but we'd have to
> > > > determine if we want to append or add one per PID... I rather leave that
> > > > stuff as-is and encourage folks to upgrade coccinelle.
> > >
> > > If coccicheck is using --very-quiet, there will not be much stderr of
> > > interest when using parmap either.
> >
> > OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet
> > is used ? Or does using --very-quiet suppress stderr ?
> 
> --very-quiet suppresses most administrative messages that go to stderr.
> There are still actual errors.  Bu you don't see eg what file is being
> currently processed.

OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll extend
this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you can
specify extra options that the script doesn't specifically support.

> > > > > > > Originally our use of parmap made output, specia files based on pids.  Maybe this
> > > > > > > is the default for parmap.  I found this completely unusable.  I guess one
> > > > > > > could look at the dates to see which file is the most recent one, but it
> > > > > > > seems tedious.  If you are putting the standard output in x.out, then put
> > > > > > > the standard error in x.err.
> > > > > >
> > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > > >
> > > > > What is ${DIR}? and what is $$?
> > > >
> > > > When you run scripts/coccicheck we take the absolute directory
> > > > of it and then go down one level of directory, so in this case it
> > > > would be the base directory of the Linux kernel.
> > > >
> > > > $$ is the PID of the bash script.
> > >
> > > OK.  I still don't find PIDs useful, but I guess if we are talking about
> > > the entire output of coccicheck, there is not much else to do.  Normally,
> > > I don't want these files accumulating, and just write over the old ones.
> >
> > Which is why I would much prefer to instead just redirect in coccicheck
> > case stderr to stdout from coccinelle. Is that preferred?
> 
> Then things will be merged in strange ways.
> 
> Why not just let the user decide what to do with these things?

Sure, what should be the default?

  Luis

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Julia Lawall]

On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:

> On Wed, Jun 15, 2016 at 09:39:49AM +0200, Julia Lawall wrote:
> >
> >
> > On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> >
> > > On Tue, Jun 14, 2016 at 11:17:13PM +0200, Julia Lawall wrote:
> > > >
> > > >
> > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > >
> > > > > On Tue, Jun 14, 2016 at 10:47:32PM +0200, Julia Lawall wrote:
> > > > > >
> > > > > >
> > > > > > On Tue, 14 Jun 2016, Luis R. Rodriguez wrote:
> > > > > >
> > > > > > > On Tue, Jun 14, 2016 at 07:22:03AM +0200, Julia Lawall wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > >
> > > > > > > > > On Mon, Jun 13, 2016 at 09:50:15PM +0200, Julia Lawall wrote:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, 13 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > > > > >
> > > > > > > > > > > I'll redirect stderr to stdout by default when parmap support is used then.
> > > > > > > > > >
> > > > > > > > > > Usually I put them in different files.
> > > > > > > > >
> > > > > > > > > We can do that as well but I would only want to deal with parmap support
> > > > > > > > > case. Any preference? How about .coccicheck.stderr.$PID where PID would
> > > > > > > > > be the PID of the shell script?
> > > > > > > >
> > > > > > > > I don't understand the connection with parmap.
> > > > > > >
> > > > > > > When parmap support is not available the cocciscript will currently
> > > > > > > disregard stderr, output is provided as it comes to stdout from each
> > > > > > > thread I guess.
> > > > > >
> > > > > > Deepa's recent patch to coccicheck made apparent that Coccicheck uses
> > > > > > --very-quiet, so there is standard error.
> > > > >
> > > > > OK I'm disegarding the redirect for non-parmap for now but we'd have to
> > > > > determine if we want to append or add one per PID... I rather leave that
> > > > > stuff as-is and encourage folks to upgrade coccinelle.
> > > >
> > > > If coccicheck is using --very-quiet, there will not be much stderr of
> > > > interest when using parmap either.
> > >
> > > OK I don't follow. Does coccinelle only direct error to stderr when --very-quiet
> > > is used ? Or does using --very-quiet suppress stderr ?
> >
> > --very-quiet suppresses most administrative messages that go to stderr.
> > There are still actual errors.  Bu you don't see eg what file is being
> > currently processed.
>
> OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll extend
> this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you can
> specify extra options that the script doesn't specifically support.

If it is more convenient, you don't actually have to remove --very-quiet.
You can just put --quiet before --show-trying or --profile.  --quiet will
override --very-quiet.

> > > > > > > > Originally our use of parmap made output, specia files based on pids.  Maybe this
> > > > > > > > is the default for parmap.  I found this completely unusable.  I guess one
> > > > > > > > could look at the dates to see which file is the most recent one, but it
> > > > > > > > seems tedious.  If you are putting the standard output in x.out, then put
> > > > > > > > the standard error in x.err.
> > > > > > >
> > > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > > > >
> > > > > > What is ${DIR}? and what is $$?
> > > > >
> > > > > When you run scripts/coccicheck we take the absolute directory
> > > > > of it and then go down one level of directory, so in this case it
> > > > > would be the base directory of the Linux kernel.
> > > > >
> > > > > $$ is the PID of the bash script.
> > > >
> > > > OK.  I still don't find PIDs useful, but I guess if we are talking about
> > > > the entire output of coccicheck, there is not much else to do.  Normally,
> > > > I don't want these files accumulating, and just write over the old ones.
> > >
> > > Which is why I would much prefer to instead just redirect in coccicheck
> > > case stderr to stdout from coccinelle. Is that preferred?
> >
> > Then things will be merged in strange ways.
> >
> > Why not just let the user decide what to do with these things?
>
> Sure, what should be the default?

I would normally just expect standard output and standard error to appear
randomly on the screen.  That is, no management effort from the tool at
all.

julia

Re: [PATCH 4/4] coccicheck: add indexing enhancement options

[Luis R. Rodriguez]

On Wed, Jun 15, 2016 at 05:44:01PM +0200, Julia Lawall wrote:
> 
> 
> On Wed, 15 Jun 2016, Luis R. Rodriguez wrote:
> > OK thanks. I remove --very-quiet now if --profile is used within SPFLAGS, I'll extend
> > this to also avoid --very-quiet if --show-trying is used. SPFLAGS is where you can
> > specify extra options that the script doesn't specifically support.
> 
> If it is more convenient, you don't actually have to remove --very-quiet.
> You can just put --quiet before --show-trying or --profile.  --quiet will
> override --very-quiet.


How about just:

if [ "$SPFLAGS"  == *"--profile"* -o "$SPFLAGS"  == "--show-trying" ]; then     
        FLAGS="--quiet $SPFLAGS"                                                
else                                                                            
        FLAGS="--very-quiet $SPFLAGS"                                           
fi 

> > > > > > > > > Originally our use of parmap made output, specia files based on pids.  Maybe this
> > > > > > > > > is the default for parmap.  I found this completely unusable.  I guess one
> > > > > > > > > could look at the dates to see which file is the most recent one, but it
> > > > > > > > > seems tedious.  If you are putting the standard output in x.out, then put
> > > > > > > > > the standard error in x.err.
> > > > > > > >
> > > > > > > > I'll use ${DIR}/coccicheck.$$.err for stderr.
> > > > > > >
> > > > > > > What is ${DIR}? and what is $$?
> > > > > >
> > > > > > When you run scripts/coccicheck we take the absolute directory
> > > > > > of it and then go down one level of directory, so in this case it
> > > > > > would be the base directory of the Linux kernel.
> > > > > >
> > > > > > $$ is the PID of the bash script.
> > > > >
> > > > > OK.  I still don't find PIDs useful, but I guess if we are talking about
> > > > > the entire output of coccicheck, there is not much else to do.  Normally,
> > > > > I don't want these files accumulating, and just write over the old ones.
> > > >
> > > > Which is why I would much prefer to instead just redirect in coccicheck
> > > > case stderr to stdout from coccinelle. Is that preferred?
> > >
> > > Then things will be merged in strange ways.
> > >
> > > Why not just let the user decide what to do with these things?
> >
> > Sure, what should be the default?
> 
> I would normally just expect standard output and standard error to appear
> randomly on the screen.  That is, no management effort from the tool at
> all.

But the thing is, stderr is ignored now given that a shell script is used
wrapped over a Makefile so if we want what you describe I think we do
have to by default do 2>&1 on the spatch run command.

  Luis

Re: [Cocci] [PATCH 4/4] coccicheck: add indexing enhancement options

[SF Markus Elfring]

> in practice though this seems to not perform better than
> regular grep however its expected to help with some use cases
> so we use that if you have no other indexing options in place
> available.

Would you like to fix a typo in this paragraph?

Regards,
Markus

Re: [Cocci] [PATCH 0/4] scripts/coccicheck: add paramap and indexing options

[SF Markus Elfring]

> we should be able to also take advantage of with coccicheck,
> the most important one is paramap support.

Do OCaml developers prefer to refer to the software "Parmap" here?
https://rdicosmo.github.io/parmap/

Regards,
Markus