From: Patrick Steinhardt <ps@pks.im>
To: git@vger.kernel.org
Cc: "Jeff King" <peff@peff.net>,
"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
"Junio C Hamano" <gitster@pobox.com>,
"Derrick Stolee" <stolee@gmail.com>,
"René Scharfe" <l.s.r@web.de>
Subject: [PATCH v3 0/7] Speed up mirror-fetches with many refs
Date: Wed, 1 Sep 2021 15:09:36 +0200 [thread overview]
Message-ID: <cover.1630501732.git.ps@pks.im> (raw)
In-Reply-To: <cover.1629452412.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 5048 bytes --]
Hi,
this is the third version of my patch series to speed up mirror-fetches
with many refs. This patch series applies on top of master with
ps/connectivity-optim merged into it.
There's only some smallish changes based on Stolee's feedback (thanks
for that!):
- A small typo in 1/7.
- A confict fix in 4/7 required now because it's based on master
instead of directly on my merged topic.
- I've adjusted patch 5/7 such that I don't have to re-touch the
logic in 6/7.
Patrick
Patrick Steinhardt (7):
fetch: speed up lookup of want refs via commit-graph
fetch: avoid unpacking headers in object existence check
connected: refactor iterator to return next object ID directly
fetch-pack: optimize loading of refs via commit graph
fetch: refactor fetch refs to be more extendable
fetch: merge fetching and consuming refs
fetch: avoid second connectivity check if we already have all objects
builtin/clone.c | 8 ++---
builtin/fetch.c | 74 +++++++++++++++++++++++-------------------
builtin/receive-pack.c | 17 ++++------
connected.c | 15 +++++----
connected.h | 2 +-
fetch-pack.c | 12 ++++---
6 files changed, 67 insertions(+), 61 deletions(-)
Range-diff against v2:
1: 4a819a6830 ! 1: 8214f04971 fetch: speed up lookup of want refs via commit-graph
@@ Commit message
that we repeatedly need to unpack object headers for each of the
referenced objects.
- Speed this up by opportunistcally trying to resolve object IDs via the
+ Speed this up by opportunistically trying to resolve object IDs via the
commit graph. We only do so for any refs which are not in "refs/tags":
more likely than not, these are going to be a commit anyway, and this
lets us avoid having to unpack object headers completely in case the
2: 81ebadabe8 = 2: 991a27cb82 fetch: avoid unpacking headers in object existence check
3: 98e981ced9 = 3: ba834803ab connected: refactor iterator to return next object ID directly
4: 6311203f08 ! 4: 99d3316d48 fetch-pack: optimize loading of refs via commit graph
@@ fetch-pack.c: static struct commit *deref_without_lazy_fetch(const struct object
while (1) {
if (oid_object_info_extended(the_repository, oid, &info,
-@@ fetch-pack.c: static struct commit *deref_without_lazy_fetch(const struct object_id *oid,
- }
-
- if (type == OBJ_COMMIT) {
-- struct commit *commit = lookup_commit(the_repository, oid);
-+ commit = lookup_commit(the_repository, oid);
- if (!commit || repo_parse_commit(the_repository, commit))
- return NULL;
- return commit;
5: 56a9158ac3 ! 5: d64888e072 fetch: refactor fetch refs to be more extendable
@@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
trace2_region_enter("fetch", "fetch_refs", the_repository);
ret = transport_fetch_refs(transport, ref_map);
trace2_region_leave("fetch", "fetch_refs", the_repository);
-+ if (ret) {
-+ transport_unlock_pack(transport);
-+ return ret;
-+ }
++ if (ret)
++ goto out;
}
- if (!ret)
- /*
@@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
- * time to update refs to reference the new objects.
- */
- return 0;
-- transport_unlock_pack(transport);
-- return ret;
+
+ /*
+ * Keep the new pack's ".keep" file around to allow the caller
+ * time to update refs to reference the new objects.
+ */
-+ return 0;
++ return ret;
++
++out:
+ transport_unlock_pack(transport);
+ return ret;
}
-
- /* Update local refs based on the ref values fetched from a remote */
6: 31d9f72edf ! 6: 56ecbfc9c3 fetch: merge fetching and consuming refs
@@ builtin/fetch.c: static int check_exist_and_connected(struct ref *ref_map)
/*
@@ builtin/fetch.c: static int fetch_refs(struct transport *transport, struct ref *ref_map)
- trace2_region_enter("fetch", "fetch_refs", the_repository);
- ret = transport_fetch_refs(transport, ref_map);
- trace2_region_leave("fetch", "fetch_refs", the_repository);
-- if (ret) {
-- transport_unlock_pack(transport);
-- return ret;
-- }
-+ if (ret)
-+ goto out;
+ goto out;
}
- /*
- * Keep the new pack's ".keep" file around to allow the caller
- * time to update refs to reference the new objects.
- */
-- return 0;
+- return ret;
+-
+-out:
+- transport_unlock_pack(transport);
+- return ret;
-}
-
-/* Update local refs based on the ref values fetched from a remote */
7: 84e39c847f = 7: c342fc0c69 fetch: avoid second connectivity check if we already have all objects
--
2.33.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2021-09-01 13:09 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-20 10:08 [PATCH 0/6] Speed up mirror-fetches with many refs Patrick Steinhardt
2021-08-20 10:08 ` [PATCH 1/6] fetch: speed up lookup of want refs via commit-graph Patrick Steinhardt
2021-08-20 14:27 ` Derrick Stolee
2021-08-20 17:18 ` Junio C Hamano
2021-08-23 6:46 ` Patrick Steinhardt
2021-08-25 14:12 ` Derrick Stolee
2021-08-20 10:08 ` [PATCH 2/6] fetch: avoid unpacking headers in object existence check Patrick Steinhardt
2021-08-25 23:44 ` Ævar Arnfjörð Bjarmason
2021-08-20 10:08 ` [PATCH 3/6] connected: refactor iterator to return next object ID directly Patrick Steinhardt
2021-08-20 14:32 ` Derrick Stolee
2021-08-20 17:43 ` Junio C Hamano
2021-08-20 17:43 ` René Scharfe
2021-08-23 6:47 ` Patrick Steinhardt
2021-08-20 10:08 ` [PATCH 4/6] fetch-pack: optimize loading of refs via commit graph Patrick Steinhardt
2021-08-20 14:37 ` Derrick Stolee
2021-08-20 10:08 ` [PATCH 5/6] fetch: refactor fetch refs to be more extendable Patrick Steinhardt
2021-08-20 14:41 ` Derrick Stolee
2021-08-20 10:08 ` [PATCH 6/6] fetch: avoid second connectivity check if we already have all objects Patrick Steinhardt
2021-08-20 14:47 ` Derrick Stolee
2021-08-23 6:52 ` Patrick Steinhardt
2021-08-20 14:50 ` [PATCH 0/6] Speed up mirror-fetches with many refs Derrick Stolee
2021-08-21 0:09 ` Junio C Hamano
2021-08-24 10:36 ` [PATCH v2 0/7] " Patrick Steinhardt
2021-08-24 10:36 ` [PATCH v2 1/7] fetch: speed up lookup of want refs via commit-graph Patrick Steinhardt
2021-08-25 14:16 ` Derrick Stolee
2021-08-24 10:37 ` [PATCH v2 2/7] fetch: avoid unpacking headers in object existence check Patrick Steinhardt
2021-08-24 10:37 ` [PATCH v2 3/7] connected: refactor iterator to return next object ID directly Patrick Steinhardt
2021-08-24 10:37 ` [PATCH v2 4/7] fetch-pack: optimize loading of refs via commit graph Patrick Steinhardt
2021-08-24 10:37 ` [PATCH v2 5/7] fetch: refactor fetch refs to be more extendable Patrick Steinhardt
2021-08-25 14:19 ` Derrick Stolee
2021-09-01 12:48 ` Patrick Steinhardt
2021-08-24 10:37 ` [PATCH v2 6/7] fetch: merge fetching and consuming refs Patrick Steinhardt
2021-08-25 14:26 ` Derrick Stolee
2021-09-01 12:49 ` Patrick Steinhardt
2021-08-24 10:37 ` [PATCH v2 7/7] fetch: avoid second connectivity check if we already have all objects Patrick Steinhardt
2021-08-24 22:48 ` [PATCH v2 0/7] Speed up mirror-fetches with many refs Junio C Hamano
2021-08-25 6:04 ` Patrick Steinhardt
2021-08-25 14:27 ` Derrick Stolee
2021-09-01 13:09 ` Patrick Steinhardt [this message]
2021-09-01 13:09 ` [PATCH v3 1/7] fetch: speed up lookup of want refs via commit-graph Patrick Steinhardt
2021-09-01 13:09 ` [PATCH v3 2/7] fetch: avoid unpacking headers in object existence check Patrick Steinhardt
2021-09-01 13:09 ` [PATCH v3 3/7] connected: refactor iterator to return next object ID directly Patrick Steinhardt
2021-09-01 13:09 ` [PATCH v3 4/7] fetch-pack: optimize loading of refs via commit graph Patrick Steinhardt
2021-09-01 13:09 ` [PATCH v3 5/7] fetch: refactor fetch refs to be more extendable Patrick Steinhardt
2021-09-01 13:10 ` [PATCH v3 6/7] fetch: merge fetching and consuming refs Patrick Steinhardt
2021-09-01 13:10 ` [PATCH v3 7/7] fetch: avoid second connectivity check if we already have all objects Patrick Steinhardt
2021-09-01 19:58 ` [PATCH v3 0/7] Speed up mirror-fetches with many refs Junio C Hamano
2021-09-08 0:08 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1630501732.git.ps@pks.im \
--to=ps@pks.im \
--cc=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=l.s.r@web.de \
--cc=peff@peff.net \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.