From: Julia Lawall <julia.lawall@lip6.fr>
To: "Luis R. Rodriguez" <mcgrof@kernel.org>
Cc: Julia.Lawall@lip6.fr, Gilles.Muller@lip6.fr,
nicolas.palix@imag.fr, mmarek@suse.com,
linux-kernel@vger.kernel.org, akpm@linux-foundation.org,
gregkh@linuxfoundation.org, markivx@codeaurora.org,
stephen.boyd@linaro.org, zohar@linux.vnet.ibm.com,
broonie@kernel.org, ming.lei@canonical.com, tiwai@suse.de,
johannes@sipsolutions.net, chunkeey@googlemail.com,
hauke@hauke-m.de, jwboyer@fedoraproject.org,
dmitry.torokhov@gmail.com, dwmw2@infradead.org, jslaby@suse.com,
torvalds@linux-foundation.org, deepa.kernel@gmail.com,
cocci@systeme.lip6.fr
Subject: Re: [PATCH v3 3/8] coccicheck: enable parmap support
Date: Tue, 21 Jun 2016 22:17:38 +0200 (CEST) [thread overview]
Message-ID: <alpine.DEB.2.02.1606212217060.2040@localhost6.localdomain6> (raw)
In-Reply-To: <1466536893-23355-4-git-send-email-mcgrof@kernel.org>
On Tue, 21 Jun 2016, Luis R. Rodriguez wrote:
> Coccinelle has had parmap support since 1.0.2, this means
> it supports --jobs, enabling built-in multithreaded functionality,
> instead of needing one to script it out. Just look for --jobs
> in the help output to determine if this is supported.
>
> Also enable the load balancing to be dynamic, so that if a
> thread finishes early we keep feeding it.
>
> Note: now that we have all things handled for us, redirect stderr to
> stdout as well to capture any possible errors or warnings issued by
> coccinelle.
>
> If --jobs is not supported we fallback to the old mechanism.
> This also now accepts DEBUG_FILE= to specify where you want
> stderr to be redirected to, by default we redirect stderr to
> /dev/null.
Why do you want to do something different for standard error in the parmap
and nonparmap case?
julia
> Also since while at it propagate back into the shell script any
> coccinelle error code. When used in serialized mode where all cocci
> files are run this also stops processing if an error has occured.
> This lets us handle some errors in coccinelle cocci files and if
> they bail out we should inspect the errors. This will be more
> useful later to help annotate coccinelle version dependency
> requirements. This will let you run only SmPL files that your
> system supports.
>
> As a small example, prior to this change, on an 8-core system:
>
> Before:
>
> $ export COCCI=scripts/coccinelle/free/kfree.cocci
> $ time make coccicheck MODE=report DEBUG_FILE=cocci.err
> ...
>
> real 29m14.912s
> user 103m1.796s
> sys 0m4.464s
>
> After:
>
> real 16m22.435s
> user 128m30.060s
> sys 0m2.712s
>
> v3:
>
> o move USE_JOBS to avoid being overriden
>
> v2:
>
> o redirect coccinelle stderr to /dev/null by default and
> only if DEBUG_FILE is used do we pass it to a file
> o fix typo of paramap/parmap
>
> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
> ---
> scripts/coccicheck | 41 ++++++++++++++++++++++++++++++++++++++---
> 1 file changed, 38 insertions(+), 3 deletions(-)
>
> diff --git a/scripts/coccicheck b/scripts/coccicheck
> index 5319fae910b4..a77f0f246405 100755
> --- a/scripts/coccicheck
> +++ b/scripts/coccicheck
> @@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then
> exit 1
> fi
>
> -trap kill_running SIGTERM SIGINT
> -declare -a SPATCH_PID
> +USE_JOBS="no"
> +$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes"
>
> # The verbosity may be set by the environmental parameter V=
> # as for example with 'make V=1 coccicheck'
> @@ -56,6 +56,14 @@ if [ "$KBUILD_EXTMOD" != "" ] ; then
> OPTIONS="--patch $srctree $OPTIONS"
> fi
>
> +# You can override by using SPFLAGS
> +if [ "$USE_JOBS" = "no" ]; then
> + trap kill_running SIGTERM SIGINT
> + declare -a SPATCH_PID
> +else
> + OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1"
> +fi
> +
> if [ "$MODE" = "" ] ; then
> if [ "$ONLINE" = "0" ] ; then
> echo 'You have not explicitly specified the mode to use. Using default "report" mode.'
> @@ -82,7 +90,26 @@ if [ "$ONLINE" = "0" ] ; then
> echo ''
> fi
>
> -run_cmd() {
> +run_cmd_parmap() {
> + if [ $VERBOSE -ne 0 ] ; then
> + echo "Running ($NPROC in parallel): $@"
> + fi
> + if [ "$DEBUG_FILE" != "/dev/null" -a "$DEBUG_FILE" != "" ]; then
> + if [ -f $DEBUG_FILE ]; then
> + echo "Debug file $DEBUG_FILE exists, bailing"
> + exit
> + fi
> + else
> + DEBUG_FILE="/dev/null"
> + fi
> + $@ 2>$DEBUG_FILE
> + if [[ $? -ne 0 ]]; then
> + echo "coccicheck failed"
> + exit $?
> + fi
> +}
> +
> +run_cmd_old() {
> local i
> if [ $VERBOSE -ne 0 ] ; then
> echo "Running ($NPROC in parallel): $@"
> @@ -97,6 +124,14 @@ run_cmd() {
> wait
> }
>
> +run_cmd() {
> + if [ "$USE_JOBS" = "yes" ]; then
> + run_cmd_parmap $@
> + else
> + run_cmd_old $@
> + fi
> +}
> +
> kill_running() {
> for i in $(seq 0 $(( NPROC - 1 )) ); do
> if [ $VERBOSE -eq 2 ] ; then
> --
> 2.8.2
>
>
next prev parent reply other threads:[~2016-06-21 20:17 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-21 19:21 [PATCH v3 0/8] coccicheck: modernize Luis R. Rodriguez
2016-06-21 19:21 ` [PATCH v3 1/8] coccicheck: move spatch binary check up Luis R. Rodriguez
2016-06-21 20:34 ` Nicolas Palix (LIG)
2016-06-21 19:21 ` [PATCH v3 2/8] coccicheck: make SPFLAGS more useful Luis R. Rodriguez
2016-06-22 20:39 ` Nicolas Palix (LIG)
2016-06-21 19:21 ` [PATCH v3 3/8] coccicheck: enable parmap support Luis R. Rodriguez
2016-06-21 20:17 ` Julia Lawall [this message]
2016-06-21 20:39 ` Luis R. Rodriguez
2016-06-21 20:43 ` Julia Lawall
2016-06-21 20:54 ` Luis R. Rodriguez
2016-06-21 21:10 ` Julia Lawall
2016-06-21 21:27 ` Luis R. Rodriguez
2016-06-21 21:31 ` Julia Lawall
2016-06-21 21:00 ` Nicolas Palix (LIG)
2016-06-21 21:28 ` Luis R. Rodriguez
2016-06-21 21:32 ` Julia Lawall
2016-06-21 21:39 ` Luis R. Rodriguez
2016-06-21 21:44 ` Julia Lawall
2016-06-21 22:14 ` Luis R. Rodriguez
2016-06-22 5:25 ` Julia Lawall
2016-06-22 15:10 ` [Cocci] " Luis R. Rodriguez
2016-06-22 19:47 ` Julia Lawall
2016-06-21 19:21 ` [PATCH v3 4/8] coccicheck: replace --very-quiet with --quit when debugging Luis R. Rodriguez
2016-06-21 19:21 ` [PATCH v3 5/8] scripts: add Linux .cocciconfig for coccinelle Luis R. Rodriguez
2016-06-21 20:29 ` Julia Lawall
2016-06-21 23:43 ` Luis R. Rodriguez
2016-06-22 5:53 ` Julia Lawall
2016-06-21 21:07 ` Nicolas Palix (LIG)
2016-06-21 21:12 ` Julia Lawall
2016-06-21 23:38 ` Luis R. Rodriguez
2016-06-22 5:51 ` Julia Lawall
2016-06-21 19:21 ` [PATCH v3 6/8] coccicheck: add support for requring a coccinelle version Luis R. Rodriguez
2016-06-21 21:11 ` Nicolas Palix (LIG)
2016-06-21 19:21 ` [PATCH v3 7/8] coccicheck: refer to coccicheck bottest wiki for documentation Luis R. Rodriguez
2016-06-21 21:14 ` Nicolas Palix (LIG)
2016-06-21 21:18 ` Julia Lawall
2016-06-22 0:07 ` Luis R. Rodriguez
2016-06-22 5:56 ` Julia Lawall
2016-06-21 19:21 ` [PATCH v3 8/8] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci Luis R. Rodriguez
2016-06-21 20:36 ` Julia Lawall
2016-06-21 21:17 ` Nicolas Palix (LIG)
2016-06-21 20:13 ` [PATCH v3 0/8] coccicheck: modernize Julia Lawall
2016-06-21 20:51 ` [Cocci] " Luis R. Rodriguez
2016-06-21 21:02 ` Julia Lawall
2016-06-21 21:21 ` Luis R. Rodriguez
2016-06-21 21:30 ` Julia Lawall
2016-06-21 21:34 ` Luis R. Rodriguez
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.2.02.1606212217060.2040@localhost6.localdomain6 \
--to=julia.lawall@lip6.fr \
--cc=Gilles.Muller@lip6.fr \
--cc=akpm@linux-foundation.org \
--cc=broonie@kernel.org \
--cc=chunkeey@googlemail.com \
--cc=cocci@systeme.lip6.fr \
--cc=deepa.kernel@gmail.com \
--cc=dmitry.torokhov@gmail.com \
--cc=dwmw2@infradead.org \
--cc=gregkh@linuxfoundation.org \
--cc=hauke@hauke-m.de \
--cc=johannes@sipsolutions.net \
--cc=jslaby@suse.com \
--cc=jwboyer@fedoraproject.org \
--cc=linux-kernel@vger.kernel.org \
--cc=markivx@codeaurora.org \
--cc=mcgrof@kernel.org \
--cc=ming.lei@canonical.com \
--cc=mmarek@suse.com \
--cc=nicolas.palix@imag.fr \
--cc=stephen.boyd@linaro.org \
--cc=tiwai@suse.de \
--cc=torvalds@linux-foundation.org \
--cc=zohar@linux.vnet.ibm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).