From: "Luis R. Rodriguez" <mcgrof@kernel.org> To: Julia.Lawall@lip6.fr, Gilles.Muller@lip6.fr, nicolas.palix@imag.fr, mmarek@suse.com Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, gregkh@linuxfoundation.org, markivx@codeaurora.org, stephen.boyd@linaro.org, zohar@linux.vnet.ibm.com, broonie@kernel.org, ming.lei@canonical.com, tiwai@suse.de, johannes@sipsolutions.net, chunkeey@googlemail.com, hauke@hauke-m.de, jwboyer@fedoraproject.org, dmitry.torokhov@gmail.com, dwmw2@infradead.org, jslaby@suse.com, torvalds@linux-foundation.org, cocci@systeme.lip6.fr, "Luis R. Rodriguez" <mcgrof@kernel.org> Subject: [PATCH v2 2/8] coccicheck: enable parmap support Date: Thu, 16 Jun 2016 15:31:26 -0700 [thread overview] Message-ID: <1466116292-21843-3-git-send-email-mcgrof@kernel.org> (raw) In-Reply-To: <1466116292-21843-1-git-send-email-mcgrof@kernel.org> Coccinelle has had parmap support since 1.0.2, this means it supports --jobs, enabling built-in multithreaded functionality, instead of needing one to script it out. Just look for --jobs in the help output to determine if this is supported. Also enable the load balancing to be dynamic, so that if a thread finishes early we keep feeding it. Note: now that we have all things handled for us, redirect stderr to stdout as well to capture any possible errors or warnings issued by coccinelle. If --jobs is not supported we fallback to the old mechanism. This also now accepts DEBUG_FILE= to specify where you want stderr to be redirected to, by default we redirect stderr to /dev/null. Also since while at it propagate back into the shell script any coccinelle error code. When used in serialized mode where all cocci files are run this also stops processing if an error has occured. This lets us handle some errors in coccinelle cocci files and if they bail out we should inspect the errors. This will be more useful later to help annotate coccinelle version dependency requirements. This will let you run only SmPL files that your system supports. Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> As a small example, prior to this change, on an 8-core system: Before: $ export COCCI=scripts/coccinelle/free/kfree.cocci $ time make coccicheck MODE=report DEBUG_FILE=cocci.err ... real 29m14.912s user 103m1.796s sys 0m4.464s After: real 16m22.435s user 128m30.060s sys 0m2.712s v2: o redirect coccinelle stderr to /dev/null by default and only if DEBUG_FILE is used do we pass it to a file o fix typo of paramap/parmap Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> --- scripts/coccicheck | 40 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index aa5e78fba270..7acef3efc258 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then exit 1 fi -trap kill_running SIGTERM SIGINT -declare -a SPATCH_PID +USE_JOBS="no" +$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes" # The verbosity may be set by the environmental parameter V= # as for example with 'make V=1 coccicheck' @@ -82,7 +82,33 @@ if [ "$ONLINE" = "0" ] ; then echo '' fi -run_cmd() { +if [ "$USE_JOBS" = "no" ]; then + trap kill_running SIGTERM SIGINT + declare -a SPATCH_PID +else + OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1" +fi + +run_cmd_parmap() { + if [ $VERBOSE -ne 0 ] ; then + echo "Running ($NPROC in parallel): $@" + fi + if [ "$DEBUG_FILE" != "/dev/null" -a "$DEBUG_FILE" != "" ]; then + if [ -f $DEBUG_FILE ]; then + echo "Debug file $DEBUG_FILE exists, bailing" + exit + fi + else + DEBUG_FILE="/dev/null" + fi + $@ 2>$DEBUG_FILE + if [[ $? -ne 0 ]]; then + echo "coccicheck failed" + exit $? + fi +} + +run_cmd_old() { local i if [ $VERBOSE -ne 0 ] ; then echo "Running ($NPROC in parallel): $@" @@ -97,6 +123,14 @@ run_cmd() { wait } +run_cmd() { + if [ "$USE_JOBS" = "yes" ]; then + run_cmd_parmap $@ + else + run_cmd_old $@ + fi +} + kill_running() { for i in $(seq 0 $(( NPROC - 1 )) ); do if [ $VERBOSE -eq 2 ] ; then -- 2.8.2
WARNING: multiple messages have this Message-ID (diff)
From: mcgrof@kernel.org (Luis R. Rodriguez) To: cocci@systeme.lip6.fr Subject: [Cocci] [PATCH v2 2/8] coccicheck: enable parmap support Date: Thu, 16 Jun 2016 15:31:26 -0700 [thread overview] Message-ID: <1466116292-21843-3-git-send-email-mcgrof@kernel.org> (raw) In-Reply-To: <1466116292-21843-1-git-send-email-mcgrof@kernel.org> Coccinelle has had parmap support since 1.0.2, this means it supports --jobs, enabling built-in multithreaded functionality, instead of needing one to script it out. Just look for --jobs in the help output to determine if this is supported. Also enable the load balancing to be dynamic, so that if a thread finishes early we keep feeding it. Note: now that we have all things handled for us, redirect stderr to stdout as well to capture any possible errors or warnings issued by coccinelle. If --jobs is not supported we fallback to the old mechanism. This also now accepts DEBUG_FILE= to specify where you want stderr to be redirected to, by default we redirect stderr to /dev/null. Also since while at it propagate back into the shell script any coccinelle error code. When used in serialized mode where all cocci files are run this also stops processing if an error has occured. This lets us handle some errors in coccinelle cocci files and if they bail out we should inspect the errors. This will be more useful later to help annotate coccinelle version dependency requirements. This will let you run only SmPL files that your system supports. Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> As a small example, prior to this change, on an 8-core system: Before: $ export COCCI=scripts/coccinelle/free/kfree.cocci $ time make coccicheck MODE=report DEBUG_FILE=cocci.err ... real 29m14.912s user 103m1.796s sys 0m4.464s After: real 16m22.435s user 128m30.060s sys 0m2.712s v2: o redirect coccinelle stderr to /dev/null by default and only if DEBUG_FILE is used do we pass it to a file o fix typo of paramap/parmap Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org> --- scripts/coccicheck | 40 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/scripts/coccicheck b/scripts/coccicheck index aa5e78fba270..7acef3efc258 100755 --- a/scripts/coccicheck +++ b/scripts/coccicheck @@ -12,8 +12,8 @@ if [ ! -x "$SPATCH" ]; then exit 1 fi -trap kill_running SIGTERM SIGINT -declare -a SPATCH_PID +USE_JOBS="no" +$SPATCH --help | grep "\-\-jobs" > /dev/null && USE_JOBS="yes" # The verbosity may be set by the environmental parameter V= # as for example with 'make V=1 coccicheck' @@ -82,7 +82,33 @@ if [ "$ONLINE" = "0" ] ; then echo '' fi -run_cmd() { +if [ "$USE_JOBS" = "no" ]; then + trap kill_running SIGTERM SIGINT + declare -a SPATCH_PID +else + OPTIONS="$OPTIONS --jobs $NPROC --chunksize 1" +fi + +run_cmd_parmap() { + if [ $VERBOSE -ne 0 ] ; then + echo "Running ($NPROC in parallel): $@" + fi + if [ "$DEBUG_FILE" != "/dev/null" -a "$DEBUG_FILE" != "" ]; then + if [ -f $DEBUG_FILE ]; then + echo "Debug file $DEBUG_FILE exists, bailing" + exit + fi + else + DEBUG_FILE="/dev/null" + fi + $@ 2>$DEBUG_FILE + if [[ $? -ne 0 ]]; then + echo "coccicheck failed" + exit $? + fi +} + +run_cmd_old() { local i if [ $VERBOSE -ne 0 ] ; then echo "Running ($NPROC in parallel): $@" @@ -97,6 +123,14 @@ run_cmd() { wait } +run_cmd() { + if [ "$USE_JOBS" = "yes" ]; then + run_cmd_parmap $@ + else + run_cmd_old $@ + fi +} + kill_running() { for i in $(seq 0 $(( NPROC - 1 )) ); do if [ $VERBOSE -eq 2 ] ; then -- 2.8.2
next prev parent reply other threads:[~2016-06-16 22:31 UTC|newest] Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top 2016-06-16 22:31 [PATCH v2 0/8] coccicheck: modernize Luis R. Rodriguez 2016-06-16 22:31 ` [Cocci] " Luis R. Rodriguez 2016-06-16 22:31 ` [PATCH v2 1/8] coccicheck: move spatch binary check up Luis R. Rodriguez 2016-06-16 22:31 ` [Cocci] " Luis R. Rodriguez 2016-06-16 22:31 ` Luis R. Rodriguez [this message] 2016-06-16 22:31 ` [Cocci] [PATCH v2 2/8] coccicheck: enable parmap support Luis R. Rodriguez 2016-06-16 22:31 ` [PATCH v2 3/8] coccicheck: add indexing enhancement options Luis R. Rodriguez 2016-06-16 22:31 ` [Cocci] " Luis R. Rodriguez 2016-06-17 9:47 ` Julia Lawall 2016-06-17 9:47 ` [Cocci] " Julia Lawall 2016-06-17 15:12 ` Luis R. Rodriguez 2016-06-17 15:12 ` [Cocci] " Luis R. Rodriguez 2016-06-17 15:29 ` Julia Lawall 2016-06-17 15:29 ` [Cocci] " Julia Lawall 2016-06-16 22:31 ` [PATCH v2 4/8] scripts: add glimpse.sh for indexing the kernel Luis R. Rodriguez 2016-06-16 22:31 ` [Cocci] " Luis R. Rodriguez 2016-06-17 9:44 ` Julia Lawall 2016-06-17 9:44 ` [Cocci] " Julia Lawall 2016-06-17 15:10 ` Luis R. Rodriguez 2016-06-17 15:10 ` [Cocci] " Luis R. Rodriguez 2016-06-17 15:35 ` Julia Lawall 2016-06-17 15:35 ` [Cocci] " Julia Lawall 2016-06-18 1:22 ` Luis R. Rodriguez 2016-06-18 1:22 ` [Cocci] " Luis R. Rodriguez 2016-06-18 5:51 ` Julia Lawall 2016-06-18 5:51 ` [Cocci] " Julia Lawall 2016-06-20 19:18 ` Luis R. Rodriguez 2016-06-20 19:18 ` [Cocci] " Luis R. Rodriguez 2016-06-18 10:37 ` Julia Lawall 2016-06-18 10:37 ` [Cocci] " Julia Lawall 2016-06-18 10:38 ` Julia Lawall 2016-06-18 10:38 ` [Cocci] " Julia Lawall 2016-06-17 15:18 ` SF Markus Elfring 2016-06-17 15:18 ` SF Markus Elfring 2016-06-17 16:11 ` Julia Lawall 2016-06-17 16:11 ` Julia Lawall 2016-06-16 22:31 ` [PATCH v2 5/8] coccicheck: replace --very-quiet with --quit when debugging Luis R. Rodriguez 2016-06-16 22:31 ` [Cocci] " Luis R. Rodriguez 2016-06-16 22:31 ` [PATCH v2 6/8] coccicheck: add support for requring a coccinelle version Luis R. Rodriguez 2016-06-16 22:31 ` [Cocci] " Luis R. Rodriguez 2016-06-16 22:31 ` [PATCH v2 7/8] coccicheck: refer to coccicheck bottest wiki for documentation Luis R. Rodriguez 2016-06-16 22:31 ` [Cocci] " Luis R. Rodriguez 2016-06-16 22:31 ` [PATCH v2 8/8] scripts/coccinelle: require coccinelle >= 1.0.4 on device_node_continue.cocci Luis R. Rodriguez 2016-06-16 22:31 ` [Cocci] " Luis R. Rodriguez
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1466116292-21843-3-git-send-email-mcgrof@kernel.org \ --to=mcgrof@kernel.org \ --cc=Gilles.Muller@lip6.fr \ --cc=Julia.Lawall@lip6.fr \ --cc=akpm@linux-foundation.org \ --cc=broonie@kernel.org \ --cc=chunkeey@googlemail.com \ --cc=cocci@systeme.lip6.fr \ --cc=dmitry.torokhov@gmail.com \ --cc=dwmw2@infradead.org \ --cc=gregkh@linuxfoundation.org \ --cc=hauke@hauke-m.de \ --cc=johannes@sipsolutions.net \ --cc=jslaby@suse.com \ --cc=jwboyer@fedoraproject.org \ --cc=linux-kernel@vger.kernel.org \ --cc=markivx@codeaurora.org \ --cc=ming.lei@canonical.com \ --cc=mmarek@suse.com \ --cc=nicolas.palix@imag.fr \ --cc=stephen.boyd@linaro.org \ --cc=tiwai@suse.de \ --cc=torvalds@linux-foundation.org \ --cc=zohar@linux.vnet.ibm.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.