From: Taylor Blau <me@ttaylorr.com>
To: git@vger.kernel.org
Cc: peff@peff.net
Subject: [PATCH 0/8] repack: introduce `--write-midx`
Date: Fri, 10 Sep 2021 23:32:25 -0400 [thread overview]
Message-ID: <cover.1631331139.git.me@ttaylorr.com> (raw)
This series implements the final component of multi-pack reachability bitmaps:
the ability to write them from `git repack`.
The gist is that it is currently impossible to write a MIDX based on the result
of a `git repack` without the possibility that an existing MIDX may go away.
That's because `git repack` may have deleted a pack that was included in the
MIDX (either because it was rolled up when doing a `--geometric` repack, or the
repack was done all-into-one), and MIDXs cannot refer to deleted packs.
To solve this, `git repack` is taught a new option to write a MIDX between the
critical section: right after the new packs are written and moved into place,
but right before any old packs are deleted.
Unfortunately, there is quite a bit of auxiliary tooling we need to implement
before this is possible. The series is laid out like so:
- The first two patches lay the ground work for and expose `git
multi-pack-index --stdin-packs`, which allows callers to write a MIDX
containing only certain packs.
- The next patch implements preliminary support for `--refs-snapshot`, which
allows the MIDX bitmap selection code to reference the prior state of
reference tips, preventing a race.
- The next two patches prepare the `repack` builtin to write a MIDX, and the
following patch actually implements `--write-midx`.
- The final two patches make sure the MIDX generated has an appropriately
chosen preferred pack, and that the refs-snapshot is provided.
It is based on `tb/multi-pack-bitmaps`. Thanks in advance for your review!
Taylor Blau (8):
midx: expose 'write_midx_file_only()' publicly
builtin/multi-pack-index.c: support --stdin-packs mode
midx: preliminary support for `--refs-snapshot`
builtin/repack.c: keep track of existing packs unconditionally
builtin/repack.c: extract showing progress to a variable
builtin/repack.c: support writing a MIDX while repacking
builtin/repack.c: make largest pack preferred
builtin/repack.c: pass `--refs-snapshot` when writing bitmaps
Documentation/git-multi-pack-index.txt | 19 ++
Documentation/git-repack.txt | 18 +-
builtin/multi-pack-index.c | 35 +++-
builtin/repack.c | 251 ++++++++++++++++++++++---
midx.c | 102 ++++++++--
midx.h | 4 +-
pack-bitmap.c | 2 +-
pack-bitmap.h | 1 +
t/helper/test-read-midx.c | 25 ++-
t/lib-midx.sh | 8 +
t/t5319-multi-pack-index.sh | 15 ++
t/t5326-multi-pack-bitmaps.sh | 82 ++++++++
t/t7700-repack.sh | 96 ++++++++++
t/t7703-repack-geometric.sh | 23 +++
14 files changed, 630 insertions(+), 51 deletions(-)
create mode 100644 t/lib-midx.sh
--
2.33.0.96.g73915697e6
next reply other threads:[~2021-09-11 3:32 UTC|newest]
Thread overview: 76+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-11 3:32 Taylor Blau [this message]
2021-09-11 3:32 ` [PATCH 1/8] midx: expose 'write_midx_file_only()' publicly Taylor Blau
2021-09-11 5:00 ` Junio C Hamano
2021-09-11 16:17 ` Taylor Blau
2021-09-11 10:07 ` Ævar Arnfjörð Bjarmason
2021-09-11 16:21 ` Taylor Blau
2021-09-11 3:32 ` [PATCH 2/8] builtin/multi-pack-index.c: support --stdin-packs mode Taylor Blau
2021-09-11 10:05 ` Ævar Arnfjörð Bjarmason
2021-09-11 16:25 ` Taylor Blau
2021-09-11 16:28 ` Taylor Blau
2021-09-12 2:08 ` Eric Sunshine
2021-09-12 2:21 ` Taylor Blau
2021-09-12 15:15 ` Ævar Arnfjörð Bjarmason
2021-09-12 22:30 ` Junio C Hamano
2021-09-12 22:32 ` Ævar Arnfjörð Bjarmason
2021-09-14 19:02 ` Jeff King
2021-09-14 23:48 ` Taylor Blau
2021-09-15 1:55 ` Eric Sunshine
2021-09-11 3:32 ` [PATCH 3/8] midx: preliminary support for `--refs-snapshot` Taylor Blau
2021-09-11 3:32 ` [PATCH 4/8] builtin/repack.c: keep track of existing packs unconditionally Taylor Blau
2021-09-11 3:32 ` [PATCH 5/8] builtin/repack.c: extract showing progress to a variable Taylor Blau
2021-09-11 3:32 ` [PATCH 6/8] builtin/repack.c: support writing a MIDX while repacking Taylor Blau
2021-09-11 3:32 ` [PATCH 7/8] builtin/repack.c: make largest pack preferred Taylor Blau
2021-09-11 10:17 ` Ævar Arnfjörð Bjarmason
2021-09-11 16:35 ` Taylor Blau
2021-09-11 3:32 ` [PATCH 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps Taylor Blau
2021-09-11 10:27 ` Ævar Arnfjörð Bjarmason
2021-09-11 11:19 ` Ævar Arnfjörð Bjarmason
2021-09-11 16:51 ` Taylor Blau
2021-09-14 18:55 ` Jeff King
2021-09-14 23:34 ` Taylor Blau
2021-09-14 23:56 ` Ævar Arnfjörð Bjarmason
2021-09-15 4:31 ` Taylor Blau
2021-09-11 16:49 ` Taylor Blau
2021-09-15 18:24 ` [PATCH v2 0/8] repack: introduce `--write-midx` Taylor Blau
2021-09-15 18:24 ` [PATCH v2 1/8] midx: expose `write_midx_file_only()` publicly Taylor Blau
2021-09-22 23:14 ` Jonathan Tan
2021-09-23 3:09 ` Taylor Blau
2021-09-15 18:24 ` [PATCH v2 2/8] builtin/multi-pack-index.c: support `--stdin-packs` mode Taylor Blau
2021-09-22 22:29 ` Josh Steadmon
2021-09-23 2:03 ` Taylor Blau
2021-09-22 23:11 ` Jonathan Tan
2021-09-23 2:06 ` Taylor Blau
2021-09-15 18:24 ` [PATCH v2 3/8] midx: preliminary support for `--refs-snapshot` Taylor Blau
2021-09-22 22:34 ` Josh Steadmon
2021-09-23 2:08 ` Taylor Blau
2021-09-22 23:00 ` Jonathan Tan
2021-09-23 2:18 ` Taylor Blau
2021-09-15 18:24 ` [PATCH v2 4/8] builtin/repack.c: keep track of existing packs unconditionally Taylor Blau
2021-09-22 22:56 ` Jonathan Tan
2021-09-23 2:59 ` Taylor Blau
2021-09-15 18:24 ` [PATCH v2 5/8] builtin/repack.c: extract showing progress to a variable Taylor Blau
2021-09-15 18:24 ` [PATCH v2 6/8] builtin/repack.c: support writing a MIDX while repacking Taylor Blau
2021-09-22 22:39 ` Jonathan Tan
2021-09-23 2:40 ` Taylor Blau
2021-09-15 18:24 ` [PATCH v2 7/8] builtin/repack.c: make largest pack preferred Taylor Blau
2021-09-15 18:24 ` [PATCH v2 8/8] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps Taylor Blau
2021-09-24 18:22 ` Jonathan Tan
2021-10-01 22:38 ` Taylor Blau
2021-09-15 19:22 ` [PATCH v2 0/8] repack: introduce `--write-midx` Junio C Hamano
2021-09-15 19:29 ` Junio C Hamano
2021-09-15 21:19 ` Taylor Blau
2021-09-16 22:16 ` Junio C Hamano
2021-09-29 1:54 ` [PATCH v3 0/9] " Taylor Blau
2021-09-29 1:55 ` [PATCH v3 1/9] midx: expose `write_midx_file_only()` publicly Taylor Blau
2021-09-29 1:55 ` [PATCH v3 2/9] builtin/multi-pack-index.c: support `--stdin-packs` mode Taylor Blau
2021-09-29 1:55 ` [PATCH v3 3/9] midx: preliminary support for `--refs-snapshot` Taylor Blau
2021-09-29 1:55 ` [PATCH v3 4/9] builtin/repack.c: keep track of existing packs unconditionally Taylor Blau
2021-09-29 1:55 ` [PATCH v3 5/9] builtin/repack.c: rename variables that deal with non-kept packs Taylor Blau
2021-09-29 1:55 ` [PATCH v3 6/9] builtin/repack.c: extract showing progress to a variable Taylor Blau
2021-09-29 1:55 ` [PATCH v3 7/9] builtin/repack.c: support writing a MIDX while repacking Taylor Blau
2021-09-29 1:55 ` [PATCH v3 8/9] builtin/repack.c: make largest pack preferred Taylor Blau
2021-09-29 1:55 ` [PATCH v3 9/9] builtin/repack.c: pass `--refs-snapshot` when writing bitmaps Taylor Blau
2021-09-29 4:24 ` [PATCH v3 0/9] repack: introduce `--write-midx` Junio C Hamano
2021-10-01 20:01 ` Jonathan Tan
2021-10-01 22:40 ` Taylor Blau
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1631331139.git.me@ttaylorr.com \
--to=me@ttaylorr.com \
--cc=git@vger.kernel.org \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).