From: Derrick Stolee <dstolee@microsoft.com>
To: "git@vger.kernel.org" <git@vger.kernel.org>
Cc: Derrick Stolee <dstolee@microsoft.com>
Subject: [PATCH 0/9] multi-pack-index cleanups
Date: Mon, 20 Aug 2018 16:51:51 +0000 [thread overview]
Message-ID: <20180820165124.152146-1-dstolee@microsoft.com> (raw)
This series is based on ds/multi-pack-index and
jk/for-each-object-iteration.
The multi-pack-index indexes objects across multiple pack-files. To
speed up object lookups and abbreviations, we do not place the pack-
files covered by the multi-pack-index into the packed_git linked list
or the packed_git_mru list. Existing test coverage focused on typical
uses and the main consumers of the multi-pack-index.
To better understand the implications of the multi-pack-index with
other scenarios, I ran the test suite after adding a step to 'git repack'
to write a multi-pack-index, and to default core.multiPackIndex to 'true'.
This commit is available as [1].
The following issues were discovered, and are fixed by this series:
1. The multi-pack-index did not distinguish between local and non-local
pack-files.
2. A bad packed object was not inspected by object lookups in the multi-
pack-index, so would loop infinitely trying to load the same object.
3. 'git count-objects --verbose' would not see the objects in the multi-
pack-index and would report the multi-pack-index as garbage.
4. If the local object directory had a multi-pack-index but an alternate
did not, then the multi-pack-index would be dropped.
5. If the multi-pack-index covered a pack-file that was paired with a
reachability bitmap, then that bitmap would not be loaded.
Several issues were resolved simply by making a new 'all_packs' list in
the object store and replacing get_packed_git() calls with get_all_packs()
calls. The all_packs list is a linked list that starts with the pack-files
in multi-pack-indexes and then continues along the packed_git linked list.
Also: I simplified the usage reports in 'git multi-pack-index' to help
users who are entering parameters incorrectly.
[1] https://github.com/derrickstolee/git/commit/098dd1d515b592fb165a276241d7d68d1cde0036
DO-NOT-MERGE: compute multi-pack-index on repack.
I will send this commit as a separate patch so we can see the change
I made and the one test I needed to fix (because it moves a pack-file,
thereby making the multi-pack-index invalid).
[2] https://github.com/derrickstolee/git/pull/9
A GitHub pull request containing this series.
Derrick Stolee (9):
multi-pack-index: provide more helpful usage info
multi-pack-index: store local property
midx: mark bad packed objects
midx: stop reporting garbage
midx: fix bug that skips midx with alternates
packfile: add all_packs list
treewide: use get_all_packs
midx: test a few commands that use get_all_packs
pack-objects: consider packs in multi-pack-index
builtin/count-objects.c | 2 +-
builtin/fsck.c | 4 ++--
builtin/gc.c | 4 ++--
builtin/multi-pack-index.c | 16 +++++++-------
builtin/pack-objects.c | 42 +++++++++++++++++++++++++++++------
builtin/pack-redundant.c | 4 ++--
fast-import.c | 4 ++--
http-backend.c | 4 ++--
midx.c | 32 ++++++++++++++++++---------
midx.h | 7 ++++--
object-store.h | 6 +++++
pack-bitmap.c | 2 +-
pack-objects.c | 2 +-
packfile.c | 40 ++++++++++++++++++++++++++++-----
packfile.h | 1 +
server-info.c | 4 ++--
t/helper/test-read-midx.c | 2 +-
t/t5319-multi-pack-index.sh | 44 ++++++++++++++++++++++++++++++++++---
18 files changed, 168 insertions(+), 52 deletions(-)
--
2.18.0.118.gd4f65b8d14
next reply other threads:[~2018-08-20 16:51 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-20 16:51 Derrick Stolee [this message]
2018-08-20 16:51 ` [PATCH 1/9] multi-pack-index: provide more helpful usage info Derrick Stolee
2018-08-20 16:51 ` [PATCH 2/9] multi-pack-index: store local property Derrick Stolee
2018-08-20 21:14 ` Stefan Beller
2018-08-20 16:51 ` [PATCH 3/9] midx: mark bad packed objects Derrick Stolee
2018-08-20 21:23 ` Stefan Beller
2018-08-21 13:53 ` Derrick Stolee
2018-08-20 16:51 ` [PATCH 4/9] midx: stop reporting garbage Derrick Stolee
2018-08-20 16:52 ` [PATCH 5/9] midx: fix bug that skips midx with alternates Derrick Stolee
2018-08-20 16:52 ` [PATCH 6/9] packfile: add all_packs list Derrick Stolee
2018-08-20 16:52 ` [PATCH 7/9] treewide: use get_all_packs Derrick Stolee
2018-08-20 22:01 ` Stefan Beller
2018-08-21 13:56 ` Derrick Stolee
2018-08-20 16:52 ` [PATCH 8/9] midx: test a few commands that " Derrick Stolee
2018-08-20 22:03 ` Stefan Beller
2018-08-20 16:52 ` [PATCH 9/9] pack-objects: consider packs in multi-pack-index Derrick Stolee
2018-08-21 14:34 ` [PATCH 0/9] multi-pack-index cleanups Duy Nguyen
2018-08-21 14:44 ` Derrick Stolee
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180820165124.152146-1-dstolee@microsoft.com \
--to=dstolee@microsoft.com \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).