All of lore.kernel.org
 help / color / mirror / Atom feed
From: Duy Nguyen <pclouds@gmail.com>
To: Jeff King <peff@peff.net>
Cc: "Jacob Keller" <jacob.keller@gmail.com>,
	"SZEDER Gábor" <szeder.dev@gmail.com>,
	"Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
	"Denton Liu" <liu.denton@gmail.com>,
	"Git Mailing List" <git@vger.kernel.org>,
	"Junio C Hamano" <gitster@pobox.com>,
	"Emily Shaffer" <emilyshaffer@google.com>,
	"Thomas Gummerer" <t.gummerer@gmail.com>
Subject: Re: [PATCH] coccicheck: optionally batch spatch invocations
Date: Tue, 7 May 2019 17:20:41 +0700	[thread overview]
Message-ID: <CACsJy8BFxN9oL-ag2DH+mhySkOAXOzP1Q0wnb4bEHWTwgyGtfg@mail.gmail.com> (raw)
In-Reply-To: <20190506234334.GA13296@sigill.intra.peff.net>

On Tue, May 7, 2019 at 6:43 AM Jeff King <peff@peff.net> wrote:
>
> On Mon, May 06, 2019 at 04:34:09PM +0700, Duy Nguyen wrote:
>
> > > However, it comes at a cost. The RSS of each spatch process goes from
> > > ~50MB to ~1500MB (and peak memory usage may be even higher if make runs
> >
> > 1.5G should be fine. Trying...
> >
> > Even with no -j, my htop's RES column goes up 6GB and put my laptop in
> > "swap every bit of memory out, including the bits handling the screen"
> > mode :( I don't think it was even the peak.
>
> Interesting if you have a different version of spatch. I'm using 1.0.4
> from Debian unstable.

I'm using 1.0.6

> Ideally we'd just invoke a single instance of spatch per rule-file and
> feed it all source files. But spatch can be rather memory hungry when
> run in this way. I measured the peak RSS going from ~90MB for a single
> file to ~1900MB for all files. Multiplied by multiple rule files being
> processed at the same time (for "make -j"), this can make things slower
> or even cause them to fail (e.g., this is reported to happen on our
> Travis builds).
>
> Instead, let's provide a tunable knob. We'll leave the default at "1",
> but it can be cranked up to "999" for maximum CPU/memory tradeoff, or
> people can find points in between that serve their particular machines.
>
> Here are a few numbers running a single rule via:
>
>   SIZES='1 4 16 999'
>   RULE=contrib/coccinelle/object_id.cocci
>   for i in $SIZES; do
>     make clean
>     /usr/bin/time -o $i.out --format='%e | %U | %S | %M' \
>       make $RULE.patch SPATCH_BATCH_SIZE=$i
>   done
>   for i in $SIZES; do
>     printf '%4d | %s\n' $i "$(cat $i.out)"
>   done
>
> which yields:
>
>      1 | 97.73 | 93.38 | 4.33 | 100128
>      4 | 52.80 | 51.14 | 1.69 | 135204
>     16 | 35.82 | 35.09 | 0.76 | 284124
>    999 | 23.30 | 23.13 | 0.20 | 1903852

I think I'm going to settle around 32 which has peak at 2GB for me and
I can still make use of 4 hyperthread cores.

It's weird for me though that no SIZES=1 is actually the fastest. The
larger SPATCH_BATCH_SIZE is, the slower:

> ~/w/git/temp $ rm contrib/coccinelle/commit.cocci.patch
> ~/w/git/temp $ time make contrib/coccinelle/commit.cocci.patch
     SPATCH contrib/coccinelle/commit.cocci

real    1m20,242s
user    1m15,863s
sys     0m4,376s

> ~/w/git/temp $ rm contrib/coccinelle/commit.cocci.patch
> ~/w/git/temp $ time make contrib/coccinelle/commit.cocci.patch SPATCH_BATCH_SIZE=16
     SPATCH contrib/coccinelle/commit.cocci

real    3m3,131s
user    2m59,345s
sys     0m3,819s

> ~/w/git/temp $ rm contrib/coccinelle/commit.cocci.patch
> ~/w/git/temp $ time make contrib/coccinelle/commit.cocci.patch SPATCH_BATCH_SIZE=32
     SPATCH contrib/coccinelle/commit.cocci

real    5m10,966s
user    5m5,434s
sys     0m5,500s

But that might be just cocci or environmental problem. Since you and
Jacob both see good speedup, this patch is good.
-- 
Duy

  parent reply	other threads:[~2019-05-07 10:21 UTC|newest]

Thread overview: 103+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-12 23:48 [RFC PATCH] *.h: remove extern from function declarations Denton Liu
2019-04-13  1:24 ` Jeff King
2019-04-13  5:45   ` Junio C Hamano
2019-04-15 18:24 ` [PATCH v2 0/3] " Denton Liu
2019-04-15 18:24   ` [PATCH v2 1/3] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-15 19:19     ` Thomas Gummerer
2019-04-15 18:24   ` [PATCH v2 2/3] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-15 18:24   ` [PATCH v2 3/3] cocci: prevent extern function declarations Denton Liu
2019-04-17  7:58   ` [PATCH v3 0/4] remove extern from " Denton Liu
2019-04-17  7:58     ` [PATCH v3 1/4] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-17  7:58     ` [PATCH v3 2/4] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-17  7:58     ` [PATCH v3 3/4] *.[ch]: manually align parameter lists Denton Liu
2019-04-17  7:58     ` [PATCH v3 4/4] cocci: prevent extern function declarations Denton Liu
2019-04-22  5:44     ` [PATCH] cache.h: fix mismerge of 'dl/no-extern-in-func-decl' Denton Liu
2019-04-22  6:30       ` Junio C Hamano
2019-04-22 11:19         ` Junio C Hamano
2019-04-22 21:49     ` [PATCH v3 0/4] remove extern from function declarations Jeff King
2019-04-25 12:07       ` SZEDER Gábor
2019-04-25 18:05         ` Denton Liu
2019-04-30 23:21         ` Johannes Schindelin
2019-05-01 10:01           ` Denton Liu
2019-05-01 18:56             ` Jeff King
2019-05-02  0:04             ` SZEDER Gábor
2019-05-03  9:32               ` Johannes Schindelin
2019-05-03 14:42                 ` SZEDER Gábor
2019-05-03 14:58                   ` SZEDER Gábor
2019-05-03 17:45                   ` Jeff King
2019-05-03 18:44                     ` SZEDER Gábor
2019-05-05  5:28                     ` Junio C Hamano
2019-05-05 18:09                       ` Jacob Keller
2019-05-05 18:08                     ` Jacob Keller
2019-05-06  5:11                       ` [PATCH] coccicheck: optionally process every source file at once Jeff King
2019-05-06  9:34                         ` Duy Nguyen
2019-05-06 23:43                           ` [PATCH] coccicheck: optionally batch spatch invocations Jeff King
2019-05-07  1:41                             ` Jacob Keller
2019-05-07  2:04                               ` Jeff King
2019-05-07  2:42                             ` Junio C Hamano
2019-05-07  2:55                               ` Jeff King
2019-05-07  3:04                                 ` Jacob Keller
2019-05-07  4:52                                 ` Junio C Hamano
2019-05-08  7:07                                   ` Jeff King
2019-05-08 12:36                                     ` Denton Liu
2019-05-08 22:39                                       ` Jeff King
2019-05-07 10:20                             ` Duy Nguyen [this message]
2019-05-07 11:19                             ` SZEDER Gábor
2021-03-02 20:51                             ` [PATCH] Makefile: fix bugs in coccicheck and speed it up Ævar Arnfjörð Bjarmason
2021-03-03  9:43                               ` Denton Liu
2021-03-03 11:45                               ` Ævar Arnfjörð Bjarmason
2021-03-04 23:18                               ` Junio C Hamano
2021-03-05 11:17                                 ` Ævar Arnfjörð Bjarmason
2021-03-05 10:24                               ` Jeff King
2021-03-05 17:20                                 ` Ævar Arnfjörð Bjarmason
2021-03-06 10:59                                   ` Jeff King
2021-03-05 17:07                               ` [PATCH v2 0/4] Makefile/coccicheck: fix bugs " Ævar Arnfjörð Bjarmason
2021-03-05 19:10                                 ` René Scharfe.
     [not found]                                   ` <xmqqim659u57.fsf@gitster.c.googlers.com>
2021-03-06 11:26                                     ` René Scharfe.
2021-03-06 12:43                                       ` René Scharfe.
     [not found]                                       ` <xmqqft16914r.fsf@gitster.c.googlers.com>
2021-03-13 16:10                                         ` René Scharfe.
2021-03-06 17:27                                   ` Ævar Arnfjörð Bjarmason
2021-03-06 17:41                                     ` René Scharfe.
2021-03-06 17:52                                       ` Ævar Arnfjörð Bjarmason
2021-03-06 19:08                                         ` René Scharfe.
2021-03-05 17:07                               ` [PATCH v2 1/4] Makefile/coccicheck: add comment heading for all SPATCH flags Ævar Arnfjörð Bjarmason
2021-03-05 17:07                               ` [PATCH v2 2/4] Makefile/coccicheck: speed up and fix bug with duplicate hunks Ævar Arnfjörð Bjarmason
2021-03-06 10:45                                 ` Jeff King
2021-03-06 19:29                                   ` Ævar Arnfjörð Bjarmason
2021-03-05 17:07                               ` [PATCH v2 3/4] Makefile/coccicheck: allow for setting xargs concurrency Ævar Arnfjörð Bjarmason
2021-03-06 10:51                                 ` Jeff King
2021-03-05 17:07                               ` [PATCH v2 4/4] Makefile/coccicheck: set SPATCH_BATCH_SIZE to 8 Ævar Arnfjörð Bjarmason
2021-03-06 19:25                                 ` [PATCH v2 5/4] Makefile/coccicheck: use --include-headers-for-types Ævar Arnfjörð Bjarmason
2021-03-18 20:49                                   ` SZEDER Gábor
2021-03-19 10:32                                     ` Ævar Arnfjörð Bjarmason
2021-03-22 12:11                                   ` [PATCH v4 0/4] Makefile/coccicheck: fix bugs and speed it up Ævar Arnfjörð Bjarmason
2021-03-22 12:11                                     ` [PATCH v4 1/4] Makefile/coccicheck: add comment heading for all SPATCH flags Ævar Arnfjörð Bjarmason
2021-03-22 18:04                                       ` René Scharfe.
2021-03-22 12:11                                     ` [PATCH v4 2/4] Makefile/coccicheck: speed up and fix bug with duplicate hunks Ævar Arnfjörð Bjarmason
2021-03-22 18:05                                       ` René Scharfe.
2021-03-24 19:19                                         ` Jeff King
2021-03-22 19:09                                       ` Junio C Hamano
2021-03-22 12:11                                     ` [PATCH v4 3/4] Makefile/coccicheck: allow for setting xargs concurrency Ævar Arnfjörð Bjarmason
2021-03-24 19:26                                       ` Jeff King
2021-03-25  2:29                                         ` Ævar Arnfjörð Bjarmason
2021-03-26  4:11                                           ` Jeff King
2021-03-22 12:11                                     ` [PATCH v4 4/4] Makefile/coccicheck: set SPATCH_BATCH_SIZE to 8 Ævar Arnfjörð Bjarmason
2021-03-22 18:05                                       ` René Scharfe.
2021-03-24 19:27                                         ` Jeff King
2021-03-27 17:43                                     ` [PATCH v4 0/4] Makefile/coccicheck: fix bugs and speed it up Junio C Hamano
2021-03-27 19:46                                       ` Ævar Arnfjörð Bjarmason
2019-05-03  9:40               ` [PATCH v3 0/4] remove extern from function declarations Denton Liu
2019-04-23 23:40   ` [PATCH v4 " Denton Liu
2019-04-23 23:40     ` [PATCH v4 1/4] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-23 23:40     ` [PATCH v4 2/4] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-24  4:56       ` Junio C Hamano
2019-04-25 19:00         ` Denton Liu
2019-04-23 23:40     ` [PATCH v4 3/4] *.[ch]: manually align parameter lists Denton Liu
2019-04-23 23:40     ` [PATCH v4 4/4] cocci: prevent extern function declarations Denton Liu
2019-04-29  8:28     ` [PATCH v5 0/3] *** SUBJECT HERE *** Denton Liu
2019-04-29  8:28       ` [PATCH v5 1/3] *.[ch]: remove extern from function declarations using spatch Denton Liu
2019-04-29  8:28       ` [PATCH v5 2/3] *.[ch]: remove extern from function declarations using sed Denton Liu
2019-04-29  8:28       ` [PATCH v5 3/3] *.[ch]: manually align parameter lists Denton Liu
2019-04-29  8:30       ` [PATCH v5 0/3] *** SUBJECT HERE *** Denton Liu
2019-05-06 11:03         ` Ævar Arnfjörð Bjarmason
2019-05-06 15:34           ` Denton Liu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CACsJy8BFxN9oL-ag2DH+mhySkOAXOzP1Q0wnb4bEHWTwgyGtfg@mail.gmail.com \
    --to=pclouds@gmail.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=emilyshaffer@google.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=jacob.keller@gmail.com \
    --cc=liu.denton@gmail.com \
    --cc=peff@peff.net \
    --cc=szeder.dev@gmail.com \
    --cc=t.gummerer@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.