git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: ZheNing Hu <adlternative@gmail.com>
To: Jeff King <peff@peff.net>
Cc: Git List <git@vger.kernel.org>
Subject: Re: [hacky PATCH 0/2] speeding up trivial for-each-ref invocations
Date: Sun, 5 Sep 2021 16:19:53 +0800	[thread overview]
Message-ID: <CAOLTT8RzHb5zWUm1psZRTP=iST-jP4DBmG2Zh1UPdo6z+QyHcg@mail.gmail.com> (raw)
In-Reply-To: <YTNpQ7Od1U/5i0R7@coredump.intra.peff.net>

Jeff King <peff@peff.net> 于2021年9月4日周六 下午8:40写道:
>
> Somebody complained to me off-list recently that for-each-ref was slow
> when doing a trivial listing (like refname + objectname) of a large
> number of refs, even though you could get mostly the same output by
> just dumping the packed-refs file.
>
> So I was nerd-sniped into making this horrible series, which does speed
> up some cases. It was a combination of "how fast can we easily get it",
> plus I was curious _which_ parts of for-each-ref were actually slow.
>
> In this version there are 2 patches, tested against 'git for-each-ref
> --format="%(objectname) %(refname)"' on a fully packed repo with 500k
> refs:
>

Regarding this 500k refs, is there any way I can reproduce it?

>   - directly outputting items rather than building up a ref_array yields
>     about a 1.5x speedup
>
>   - avoiding the whole atom-formatting system yields a 2.5x speedup on
>     top of that
>
> I had originally written it against a slightly older version of Git,
> before show_ref_array_item() was inlined. In that version, there was a
> middle ground where we still created a ref_array_item for each ref, and
> then fed it to the "quick" formatter (so we could see the cost of
> having to malloc/free a bunch of ref_array_item structs). The numbers
> there wre:
>
>   - streaming out items gave a 1.5x speedup
>
>   - using the "quick" formatter gave a 1.8x speedup
>
>   - avoiding the extra malloc/free for each item gave a 1.4x
>
> which makes sense; 1.8 * 1.4 is ~2.5, so it's the same speedup just
> broken down further.
>
> I'm not sure I'm really advocating that we should do something like
> this, though it is tempting because it does make some common queries
> much faster. But I wanted to share it here to give some insight on where
> the time may be going in ref-filter / for-each-ref.
>

Well, this acceleration looks gratifying. But so far I haven't seen
its effect on
git cat-file --batch | --batch-check.


>   [1/2]: ref-filter: hacky "streaming" mode
>   [2/2]: ref-filter: implement "quick" formats
>
>  builtin/for-each-ref.c |   7 +++
>  ref-filter.c           | 113 +++++++++++++++++++++++++++++++++++++----
>  ref-filter.h           |  21 ++++++++
>  3 files changed, 132 insertions(+), 9 deletions(-)
>
> -Peff

  parent reply	other threads:[~2021-09-05  8:20 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-04 12:40 Jeff King
2021-09-04 12:41 ` [PATCH 1/2] ref-filter: hacky "streaming" mode Jeff King
2021-09-05  8:20   ` ZheNing Hu
2021-09-05 13:04     ` Jeff King
2021-09-07  5:28       ` ZheNing Hu
2021-09-07 18:01         ` Jeff King
2021-09-09 14:45           ` ZheNing Hu
2021-09-10 14:26             ` Jeff King
2021-09-15 12:27               ` ZheNing Hu
2021-09-15 14:23                 ` ZheNing Hu
2021-09-16 21:45                   ` Jeff King
2021-09-20  7:42                     ` ZheNing Hu
2021-09-16 21:31                 ` Jeff King
2021-09-05 13:15     ` Jeff King
2021-09-07  5:42       ` ZheNing Hu
2021-09-04 12:42 ` [PATCH 2/2] ref-filter: implement "quick" formats Jeff King
2021-09-05  8:20   ` ZheNing Hu
2021-09-05 13:07     ` Jeff King
2021-09-06 13:34       ` ZheNing Hu
2021-09-07 20:06       ` Junio C Hamano
2021-09-05  8:19 ` ZheNing Hu [this message]
2021-09-05 12:49   ` [hacky PATCH 0/2] speeding up trivial for-each-ref invocations Jeff King
2021-09-06 13:30     ` ZheNing Hu
2021-09-07 17:28       ` Jeff King
2021-09-09 13:20         ` ZheNing Hu
2021-09-06  6:54 ` Patrick Steinhardt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAOLTT8RzHb5zWUm1psZRTP=iST-jP4DBmG2Zh1UPdo6z+QyHcg@mail.gmail.com' \
    --to=adlternative@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=peff@peff.net \
    --subject='Re: [hacky PATCH 0/2] speeding up trivial for-each-ref invocations' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).