From: Taylor Blau <me@ttaylorr.com>
To: git@vger.kernel.org
Cc: dstolee@microsoft.com, gitster@pobox.com, peff@peff.net,
martin.agren@gmail.com, szeder.dev@gmail.com
Subject: [PATCH v2 24/24] pack-bitmap-write: better reuse bitmaps
Date: Tue, 17 Nov 2020 16:48:36 -0500 [thread overview]
Message-ID: <42399a1c2e52e1d055a2d0ad96af2ca4dce6b1a0.1605649533.git.me@ttaylorr.com> (raw)
In-Reply-To: <cover.1605649533.git.me@ttaylorr.com>
From: Derrick Stolee <dstolee@microsoft.com>
If the old bitmap file contains a bitmap for a given commit, then that
commit does not need help from intermediate commits in its history to
compute its final bitmap. Eject that commit from the walk and insert it
as a maximal commit in the list of commits for computing bitmaps.
This helps the repeat bitmap computation task, even if the selected
commits shift drastically. This helps when a previously-bitmapped commit
exists in the first-parent history of a newly-selected commit. Since we
stop the walk at these commits and we use a first-parent walk, it is
harder to walk "around" these bitmapped commits. It's not impossible,
but we can greatly reduce the computation time for many selected
commits.
| runtime (sec) | peak heap (GB) |
| | |
| from | with | from | with |
| scratch | existing | scratch | existing |
-----------+---------+----------+---------+-----------
last patch | 100.641 | 35.560 | 2.152 | 2.224 |
this patch | 99.720 | 11.696 | 2.152 | 2.217 |
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
pack-bitmap-write.c | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index b0493d971d..3ac90ae410 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -195,7 +195,8 @@ struct bitmap_builder {
};
static void bitmap_builder_init(struct bitmap_builder *bb,
- struct bitmap_writer *writer)
+ struct bitmap_writer *writer,
+ struct bitmap_index *old_bitmap)
{
struct rev_info revs;
struct commit *commit;
@@ -234,12 +235,26 @@ static void bitmap_builder_init(struct bitmap_builder *bb,
c_ent = bb_data_at(&bb->data, commit);
+ if (old_bitmap && bitmap_for_commit(old_bitmap, commit)) {
+ /*
+ * This commit has an existing bitmap, so we can
+ * get its bits immediately without an object
+ * walk. There is no need to continue walking
+ * beyond this commit.
+ */
+ c_ent->maximal = 1;
+ p = NULL;
+ }
+
if (c_ent->maximal) {
num_maximal++;
ALLOC_GROW(bb->commits, bb->commits_nr + 1, bb->commits_alloc);
bb->commits[bb->commits_nr++] = commit;
}
+ if (!c_ent->commit_mask)
+ continue;
+
if (p) {
struct bb_commit *p_ent = bb_data_at(&bb->data, p->item);
int c_not_p, p_not_c;
@@ -422,7 +437,7 @@ void bitmap_writer_build(struct packing_data *to_pack)
else
mapping = NULL;
- bitmap_builder_init(&bb, &writer);
+ bitmap_builder_init(&bb, &writer, old_bitmap);
for (i = bb.commits_nr; i > 0; i--) {
struct commit *commit = bb.commits[i-1];
struct bb_commit *ent = bb_data_at(&bb.data, commit);
--
2.29.2.312.gabc4d358d8
next prev parent reply other threads:[~2020-11-17 21:48 UTC|newest]
Thread overview: 174+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-11 19:41 [PATCH 00/23] pack-bitmap: bitmap generation improvements Taylor Blau
2020-11-11 19:41 ` [PATCH 01/23] ewah/ewah_bitmap.c: grow buffer past 1 Taylor Blau
2020-11-22 19:36 ` Junio C Hamano
2020-11-23 16:22 ` Taylor Blau
2020-11-24 2:48 ` Jeff King
2020-11-24 2:51 ` Jeff King
2020-12-01 22:56 ` Taylor Blau
2020-11-11 19:41 ` [PATCH 02/23] pack-bitmap: fix header size check Taylor Blau
2020-11-12 17:39 ` Martin Ågren
2020-11-11 19:42 ` [PATCH 03/23] pack-bitmap: bounds-check size of cache extension Taylor Blau
2020-11-12 17:47 ` Martin Ågren
2020-11-13 4:57 ` Jeff King
2020-11-13 5:26 ` Martin Ågren
2020-11-13 21:29 ` Taylor Blau
2020-11-13 21:39 ` Jeff King
2020-11-13 21:49 ` Taylor Blau
2020-11-13 22:11 ` Jeff King
2020-11-11 19:42 ` [PATCH 04/23] t5310: drop size of truncated ewah bitmap Taylor Blau
2020-11-11 19:42 ` [PATCH 05/23] rev-list: die when --test-bitmap detects a mismatch Taylor Blau
2020-11-11 19:42 ` [PATCH 06/23] ewah: factor out bitmap growth Taylor Blau
2020-11-11 19:42 ` [PATCH 07/23] ewah: make bitmap growth less aggressive Taylor Blau
2020-11-22 20:32 ` Junio C Hamano
2020-11-23 16:49 ` Taylor Blau
2020-11-24 3:00 ` Jeff King
2020-11-24 20:11 ` Junio C Hamano
2020-11-11 19:43 ` [PATCH 08/23] ewah: implement bitmap_or() Taylor Blau
2020-11-22 20:34 ` Junio C Hamano
2020-11-23 16:52 ` Taylor Blau
2020-11-11 19:43 ` [PATCH 09/23] ewah: add bitmap_dup() function Taylor Blau
2020-11-11 19:43 ` [PATCH 10/23] pack-bitmap-write: reimplement bitmap writing Taylor Blau
2020-11-11 19:43 ` [PATCH 11/23] pack-bitmap-write: pass ownership of intermediate bitmaps Taylor Blau
2020-11-11 19:43 ` [PATCH 12/23] pack-bitmap-write: fill bitmap with commit history Taylor Blau
2020-11-11 19:43 ` [PATCH 13/23] bitmap: add bitmap_diff_nonzero() Taylor Blau
2020-11-11 19:43 ` [PATCH 14/23] commit: implement commit_list_contains() Taylor Blau
2020-11-11 19:43 ` [PATCH 15/23] t5310: add branch-based checks Taylor Blau
2020-11-11 20:58 ` Derrick Stolee
2020-11-11 21:04 ` Junio C Hamano
2020-11-15 23:26 ` Johannes Schindelin
2020-11-11 19:43 ` [PATCH 16/23] pack-bitmap-write: rename children to reverse_edges Taylor Blau
2020-11-11 19:43 ` [PATCH 17/23] pack-bitmap-write: build fewer intermediate bitmaps Taylor Blau
2020-11-13 22:23 ` SZEDER Gábor
2020-11-13 23:03 ` Jeff King
2020-11-14 6:23 ` Jeff King
2020-11-11 19:43 ` [PATCH 18/23] pack-bitmap-write: ignore BITMAP_FLAG_REUSE Taylor Blau
2020-11-11 19:44 ` [PATCH 19/23] pack-bitmap: factor out 'bitmap_for_commit()' Taylor Blau
2020-11-11 19:44 ` [PATCH 20/23] pack-bitmap: factor out 'add_commit_to_bitmap()' Taylor Blau
2020-11-11 19:44 ` [PATCH 21/23] pack-bitmap-write: use existing bitmaps Taylor Blau
2020-11-11 19:44 ` [PATCH 22/23] pack-bitmap-write: relax unique rewalk condition Taylor Blau
2020-11-11 19:44 ` [PATCH 23/23] pack-bitmap-write: better reuse bitmaps Taylor Blau
2020-11-17 21:46 ` [PATCH v2 00/24] pack-bitmap: bitmap generation improvements Taylor Blau
2020-11-17 21:46 ` [PATCH v2 01/24] ewah/ewah_bitmap.c: grow buffer past 1 Taylor Blau
2020-11-17 21:46 ` [PATCH v2 02/24] pack-bitmap: fix header size check Taylor Blau
2020-11-17 21:46 ` [PATCH v2 03/24] pack-bitmap: bounds-check size of cache extension Taylor Blau
2020-11-17 21:46 ` [PATCH v2 04/24] t5310: drop size of truncated ewah bitmap Taylor Blau
2020-11-17 21:46 ` [PATCH v2 05/24] rev-list: die when --test-bitmap detects a mismatch Taylor Blau
2020-11-17 21:46 ` [PATCH v2 06/24] ewah: factor out bitmap growth Taylor Blau
2020-11-17 21:47 ` [PATCH v2 07/24] ewah: make bitmap growth less aggressive Taylor Blau
2020-11-17 21:47 ` [PATCH v2 08/24] ewah: implement bitmap_or() Taylor Blau
2020-11-17 21:47 ` [PATCH v2 09/24] ewah: add bitmap_dup() function Taylor Blau
2020-11-17 21:47 ` [PATCH v2 10/24] pack-bitmap-write: reimplement bitmap writing Taylor Blau
2020-11-25 0:53 ` Jonathan Tan
2020-11-28 17:27 ` Taylor Blau
2020-11-17 21:47 ` [PATCH v2 11/24] pack-bitmap-write: pass ownership of intermediate bitmaps Taylor Blau
2020-11-25 1:00 ` Jonathan Tan
2020-11-17 21:47 ` [PATCH v2 12/24] pack-bitmap-write: fill bitmap with commit history Taylor Blau
2020-11-22 21:50 ` Junio C Hamano
2020-11-23 14:54 ` Derrick Stolee
2020-11-25 1:14 ` Jonathan Tan
2020-11-28 17:21 ` Taylor Blau
2020-11-30 18:33 ` Jonathan Tan
2020-11-17 21:47 ` [PATCH v2 13/24] bitmap: add bitmap_diff_nonzero() Taylor Blau
2020-11-22 22:01 ` Junio C Hamano
2020-11-23 20:19 ` Taylor Blau
2020-11-17 21:47 ` [PATCH v2 14/24] commit: implement commit_list_contains() Taylor Blau
2020-11-17 21:47 ` [PATCH v2 15/24] t5310: add branch-based checks Taylor Blau
2020-11-25 1:17 ` Jonathan Tan
2020-11-28 17:30 ` Taylor Blau
2020-11-17 21:47 ` [PATCH v2 16/24] pack-bitmap-write: rename children to reverse_edges Taylor Blau
2020-11-17 21:47 ` [PATCH v2 17/24] pack-bitmap.c: check reads more aggressively when loading Taylor Blau
2020-11-17 21:48 ` [PATCH v2 18/24] pack-bitmap-write: build fewer intermediate bitmaps Taylor Blau
2020-11-24 6:07 ` Jonathan Tan
2020-11-25 1:46 ` Jonathan Tan
2020-11-30 18:41 ` Derrick Stolee
2020-11-17 21:48 ` [PATCH v2 19/24] pack-bitmap-write: ignore BITMAP_FLAG_REUSE Taylor Blau
2020-12-02 7:13 ` Jonathan Tan
2020-11-17 21:48 ` [PATCH v2 20/24] pack-bitmap: factor out 'bitmap_for_commit()' Taylor Blau
2020-12-02 7:17 ` Jonathan Tan
2020-11-17 21:48 ` [PATCH v2 21/24] pack-bitmap: factor out 'add_commit_to_bitmap()' Taylor Blau
2020-12-02 7:20 ` Jonathan Tan
2020-11-17 21:48 ` [PATCH v2 22/24] pack-bitmap-write: use existing bitmaps Taylor Blau
2020-12-02 7:28 ` Jonathan Tan
2020-12-02 16:21 ` Taylor Blau
2020-11-17 21:48 ` [PATCH v2 23/24] pack-bitmap-write: relax unique rewalk condition Taylor Blau
2020-12-02 7:44 ` Jonathan Tan
2020-12-02 16:30 ` Taylor Blau
2020-12-07 18:19 ` Jonathan Tan
2020-12-07 18:43 ` Derrick Stolee
2020-12-07 18:45 ` Derrick Stolee
2020-12-07 18:48 ` Jeff King
2020-11-17 21:48 ` Taylor Blau [this message]
2020-12-02 8:08 ` [PATCH v2 24/24] pack-bitmap-write: better reuse bitmaps Jonathan Tan
2020-12-02 16:35 ` Taylor Blau
2020-12-02 18:22 ` Derrick Stolee
2020-12-02 18:25 ` Taylor Blau
2020-12-07 18:26 ` Jonathan Tan
2020-12-07 18:24 ` Jonathan Tan
2020-12-07 19:20 ` Derrick Stolee
2020-11-18 18:32 ` [PATCH v2 00/24] pack-bitmap: bitmap generation improvements SZEDER Gábor
2020-11-18 19:51 ` Taylor Blau
2020-11-22 2:17 ` Taylor Blau
2020-11-22 2:28 ` Taylor Blau
2020-11-20 6:34 ` Martin Ågren
2020-11-21 19:37 ` Junio C Hamano
2020-11-21 20:11 ` Martin Ågren
2020-11-22 2:31 ` Taylor Blau
2020-11-24 2:43 ` Jeff King
2020-12-01 23:04 ` Taylor Blau
2020-12-01 23:37 ` Jonathan Tan
2020-12-01 23:43 ` Taylor Blau
2020-12-02 8:11 ` Jonathan Tan
2020-12-08 0:04 ` [PATCH v3 " Taylor Blau
2020-12-08 0:04 ` [PATCH v3 01/24] ewah/ewah_bitmap.c: avoid open-coding ALLOC_GROW() Taylor Blau
2020-12-08 0:04 ` [PATCH v3 02/24] pack-bitmap: fix header size check Taylor Blau
2020-12-08 0:04 ` [PATCH v3 03/24] pack-bitmap: bounds-check size of cache extension Taylor Blau
2020-12-08 0:04 ` [PATCH v3 04/24] t5310: drop size of truncated ewah bitmap Taylor Blau
2020-12-08 0:04 ` [PATCH v3 05/24] rev-list: die when --test-bitmap detects a mismatch Taylor Blau
2020-12-08 0:04 ` [PATCH v3 06/24] ewah: factor out bitmap growth Taylor Blau
2020-12-08 0:04 ` [PATCH v3 07/24] ewah: make bitmap growth less aggressive Taylor Blau
2020-12-08 0:04 ` [PATCH v3 08/24] ewah: implement bitmap_or() Taylor Blau
2020-12-08 0:04 ` [PATCH v3 09/24] ewah: add bitmap_dup() function Taylor Blau
2020-12-08 0:04 ` [PATCH v3 10/24] pack-bitmap-write: reimplement bitmap writing Taylor Blau
2020-12-08 0:05 ` [PATCH v3 11/24] pack-bitmap-write: pass ownership of intermediate bitmaps Taylor Blau
2020-12-08 0:05 ` [PATCH v3 12/24] pack-bitmap-write: fill bitmap with commit history Taylor Blau
2020-12-08 0:05 ` [PATCH v3 13/24] bitmap: implement bitmap_is_subset() Taylor Blau
2020-12-08 0:05 ` [PATCH v3 14/24] commit: implement commit_list_contains() Taylor Blau
2020-12-08 0:05 ` [PATCH v3 15/24] t5310: add branch-based checks Taylor Blau
2020-12-08 0:05 ` [PATCH v3 16/24] pack-bitmap-write: rename children to reverse_edges Taylor Blau
2020-12-08 0:05 ` [PATCH v3 17/24] pack-bitmap.c: check reads more aggressively when loading Taylor Blau
2020-12-08 0:05 ` [PATCH v3 18/24] pack-bitmap-write: build fewer intermediate bitmaps Taylor Blau
2020-12-08 0:05 ` [PATCH v3 19/24] pack-bitmap-write: ignore BITMAP_FLAG_REUSE Taylor Blau
2020-12-08 0:05 ` [PATCH v3 20/24] pack-bitmap: factor out 'bitmap_for_commit()' Taylor Blau
2020-12-08 0:05 ` [PATCH v3 21/24] pack-bitmap: factor out 'add_commit_to_bitmap()' Taylor Blau
2020-12-08 0:05 ` [PATCH v3 22/24] pack-bitmap-write: use existing bitmaps Taylor Blau
2020-12-08 0:05 ` [PATCH v3 23/24] pack-bitmap-write: relax unique rewalk condition Taylor Blau
2020-12-08 0:05 ` [PATCH v3 24/24] pack-bitmap-write: better reuse bitmaps Taylor Blau
2020-12-08 20:56 ` [PATCH v3 00/24] pack-bitmap: bitmap generation improvements Junio C Hamano
2020-12-08 21:03 ` Taylor Blau
2020-12-08 22:03 ` Junio C Hamano
2020-12-08 22:03 ` [PATCH v4 " Taylor Blau
2020-12-08 22:03 ` [PATCH v4 01/24] ewah/ewah_bitmap.c: avoid open-coding ALLOC_GROW() Taylor Blau
2020-12-08 22:03 ` [PATCH v4 02/24] pack-bitmap: fix header size check Taylor Blau
2020-12-08 22:03 ` [PATCH v4 03/24] pack-bitmap: bounds-check size of cache extension Taylor Blau
2020-12-08 22:03 ` [PATCH v4 04/24] t5310: drop size of truncated ewah bitmap Taylor Blau
2020-12-08 22:03 ` [PATCH v4 05/24] rev-list: die when --test-bitmap detects a mismatch Taylor Blau
2020-12-08 22:03 ` [PATCH v4 06/24] ewah: factor out bitmap growth Taylor Blau
2020-12-08 22:03 ` [PATCH v4 07/24] ewah: make bitmap growth less aggressive Taylor Blau
2020-12-08 22:03 ` [PATCH v4 08/24] ewah: implement bitmap_or() Taylor Blau
2020-12-08 22:03 ` [PATCH v4 09/24] ewah: add bitmap_dup() function Taylor Blau
2020-12-08 22:03 ` [PATCH v4 10/24] pack-bitmap-write: reimplement bitmap writing Taylor Blau
2020-12-08 22:03 ` [PATCH v4 11/24] pack-bitmap-write: pass ownership of intermediate bitmaps Taylor Blau
2020-12-08 22:04 ` [PATCH v4 12/24] pack-bitmap-write: fill bitmap with commit history Taylor Blau
2020-12-08 22:04 ` [PATCH v4 13/24] bitmap: implement bitmap_is_subset() Taylor Blau
2020-12-08 22:04 ` [PATCH v4 14/24] commit: implement commit_list_contains() Taylor Blau
2020-12-08 22:04 ` [PATCH v4 15/24] t5310: add branch-based checks Taylor Blau
2020-12-08 22:04 ` [PATCH v4 16/24] pack-bitmap-write: rename children to reverse_edges Taylor Blau
2020-12-08 22:04 ` [PATCH v4 17/24] pack-bitmap.c: check reads more aggressively when loading Taylor Blau
2020-12-08 22:04 ` [PATCH v4 18/24] pack-bitmap-write: build fewer intermediate bitmaps Taylor Blau
2020-12-08 22:04 ` [PATCH v4 19/24] pack-bitmap-write: ignore BITMAP_FLAG_REUSE Taylor Blau
2020-12-08 22:04 ` [PATCH v4 20/24] pack-bitmap: factor out 'bitmap_for_commit()' Taylor Blau
2020-12-08 22:05 ` Taylor Blau
2020-12-08 22:05 ` [PATCH v4 21/24] pack-bitmap: factor out 'add_commit_to_bitmap()' Taylor Blau
2020-12-08 22:05 ` [PATCH v4 22/24] pack-bitmap-write: use existing bitmaps Taylor Blau
2020-12-08 22:05 ` [PATCH v4 23/24] pack-bitmap-write: relax unique revwalk condition Taylor Blau
2020-12-08 22:05 ` [PATCH v4 24/24] pack-bitmap-write: better reuse bitmaps Taylor Blau
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=42399a1c2e52e1d055a2d0ad96af2ca4dce6b1a0.1605649533.git.me@ttaylorr.com \
--to=me@ttaylorr.com \
--cc=dstolee@microsoft.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=martin.agren@gmail.com \
--cc=peff@peff.net \
--cc=szeder.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).