git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jeff King <peff@peff.net>
To: git@vger.kernel.org
Cc: ZheNing Hu <adlternative@gmail.com>
Subject: [hacky PATCH 0/2] speeding up trivial for-each-ref invocations
Date: Sat, 4 Sep 2021 08:40:35 -0400	[thread overview]
Message-ID: <YTNpQ7Od1U/5i0R7@coredump.intra.peff.net> (raw)

Somebody complained to me off-list recently that for-each-ref was slow
when doing a trivial listing (like refname + objectname) of a large
number of refs, even though you could get mostly the same output by
just dumping the packed-refs file.

So I was nerd-sniped into making this horrible series, which does speed
up some cases. It was a combination of "how fast can we easily get it",
plus I was curious _which_ parts of for-each-ref were actually slow.

In this version there are 2 patches, tested against 'git for-each-ref
--format="%(objectname) %(refname)"' on a fully packed repo with 500k
refs:

  - directly outputting items rather than building up a ref_array yields
    about a 1.5x speedup

  - avoiding the whole atom-formatting system yields a 2.5x speedup on
    top of that

I had originally written it against a slightly older version of Git,
before show_ref_array_item() was inlined. In that version, there was a
middle ground where we still created a ref_array_item for each ref, and
then fed it to the "quick" formatter (so we could see the cost of
having to malloc/free a bunch of ref_array_item structs). The numbers
there wre:

  - streaming out items gave a 1.5x speedup

  - using the "quick" formatter gave a 1.8x speedup

  - avoiding the extra malloc/free for each item gave a 1.4x

which makes sense; 1.8 * 1.4 is ~2.5, so it's the same speedup just
broken down further.

I'm not sure I'm really advocating that we should do something like
this, though it is tempting because it does make some common queries
much faster. But I wanted to share it here to give some insight on where
the time may be going in ref-filter / for-each-ref.

  [1/2]: ref-filter: hacky "streaming" mode
  [2/2]: ref-filter: implement "quick" formats

 builtin/for-each-ref.c |   7 +++
 ref-filter.c           | 113 +++++++++++++++++++++++++++++++++++++----
 ref-filter.h           |  21 ++++++++
 3 files changed, 132 insertions(+), 9 deletions(-)

-Peff

             reply	other threads:[~2021-09-04 12:40 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-04 12:40 Jeff King [this message]
2021-09-04 12:41 ` [PATCH 1/2] ref-filter: hacky "streaming" mode Jeff King
2021-09-05  8:20   ` ZheNing Hu
2021-09-05 13:04     ` Jeff King
2021-09-07  5:28       ` ZheNing Hu
2021-09-07 18:01         ` Jeff King
2021-09-09 14:45           ` ZheNing Hu
2021-09-10 14:26             ` Jeff King
2021-09-15 12:27               ` ZheNing Hu
2021-09-15 14:23                 ` ZheNing Hu
2021-09-16 21:45                   ` Jeff King
2021-09-20  7:42                     ` ZheNing Hu
2021-09-16 21:31                 ` Jeff King
2021-09-05 13:15     ` Jeff King
2021-09-07  5:42       ` ZheNing Hu
2021-09-04 12:42 ` [PATCH 2/2] ref-filter: implement "quick" formats Jeff King
2021-09-05  8:20   ` ZheNing Hu
2021-09-05 13:07     ` Jeff King
2021-09-06 13:34       ` ZheNing Hu
2021-09-07 20:06       ` Junio C Hamano
2021-09-05  8:19 ` [hacky PATCH 0/2] speeding up trivial for-each-ref invocations ZheNing Hu
2021-09-05 12:49   ` Jeff King
2021-09-06 13:30     ` ZheNing Hu
2021-09-07 17:28       ` Jeff King
2021-09-09 13:20         ` ZheNing Hu
2021-09-06  6:54 ` Patrick Steinhardt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YTNpQ7Od1U/5i0R7@coredump.intra.peff.net \
    --to=peff@peff.net \
    --cc=adlternative@gmail.com \
    --cc=git@vger.kernel.org \
    --subject='Re: [hacky PATCH 0/2] speeding up trivial for-each-ref invocations' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).