From: "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com,
avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com,
Derrick Stolee <derrickstolee@github.com>
Subject: [PATCH 0/7] Bundle URIs III: Parse and download from bundle lists
Date: Mon, 22 Aug 2022 15:12:47 +0000 [thread overview]
Message-ID: <pull.1333.git.1661181174.gitgitgadget@gmail.com> (raw)
This is the third series building the bundle URI feature. It is built on top
of ds/bundle-uri-clone, which introduced 'git clone --bundle-uri=' where is
a URI to a bundle file. This series adds the capability of downloading and
parsing a bundle list and then downloading the URIs in that list.
The core functionality of bundle lists is implemented by creating data
structures from a list of key-value pairs. These pairs can come from a
plain-text file in Git config format, but in the future, we will support the
list being supplied by packet lines over Git's protocol v2 in the
'bundle-uri' command (reserved for the next series).
The patches are organized in this way:
1. Patches 1-2 create the bundle list data structures and the logic for
populating the list from key-value pairs.
2. Patches 3-4 teach Git to parse "key=value" lines to construct a bundle
list. Add unit tests that ensure this logic constructs lists correctly.
These patches are adapted from Ævar's RFC [1] and were previously seen
in my combined RFC [2].
3. Patch 5 teaches Git to parse Git config files into bundle lists.
4. Patches 6-7 implement the ability to download a bundle list and
recursively download the contained bundles (and possibly the bundle
lists within). This is limited by a constant depth to avoid issues with
cycles or otherwise incorrectly configured bundle lists.
[1]
https://lore.kernel.org/git/RFC-cover-v2-00.36-00000000000-20220418T165545Z-avarab@gmail.com/
[2]
https://lore.kernel.org/git/pull.1234.git.1653072042.gitgitgadget@gmail.com/
At the end of this series, users can bootstrap clones using 'git clone
--bundle-uri= ' where points to a bundle list instead of a single bundle
file.
As outlined in the design document [1], the next steps after this are:
1. Implement the protocol v2 verb, re-using the bundle list logic from (2).
Use this to auto-discover bundle URIs during 'git clone' (behind a
config option). [2]
2. Implement the 'creationToken' heuristic, allowing incremental 'git
fetch' commands to download a bundle list from a configured URI, and
only download bundles that are new based on the creation token values.
[3]
I have prepared some of this work as pull requests on my personal fork so
curious readers can look ahead to where we are going:
[3]
https://lore.kernel.org/git/pull.1248.v3.git.1658757188.gitgitgadget@gmail.com
[4] https://github.com/derrickstolee/git/pull/21
[5] https://github.com/derrickstolee/git/pull/22
Thanks,
* Stolee
Derrick Stolee (5):
bundle-uri: create bundle_list struct and helpers
bundle-uri: create base key-value pair parsing
bundle-uri: parse bundle list in config format
bundle-uri: limit recursion depth for bundle lists
bundle-uri: fetch a list of bundles
Ævar Arnfjörð Bjarmason (2):
bundle-uri: create "key=value" line parsing
bundle-uri: unit test "key=value" parsing
Documentation/config.txt | 2 +
Documentation/config/bundle.txt | 22 ++
Makefile | 1 +
bundle-uri.c | 442 +++++++++++++++++++++++++++++++-
bundle-uri.h | 98 ++++++-
t/helper/test-bundle-uri.c | 90 +++++++
t/helper/test-tool.c | 1 +
t/helper/test-tool.h | 1 +
t/t5558-clone-bundle-uri.sh | 93 +++++++
t/t5750-bundle-uri-parse.sh | 141 ++++++++++
t/test-lib-functions.sh | 11 +
11 files changed, 889 insertions(+), 13 deletions(-)
create mode 100644 Documentation/config/bundle.txt
create mode 100644 t/helper/test-bundle-uri.c
create mode 100755 t/t5750-bundle-uri-parse.sh
base-commit: e21e663cd1942df29979d3e01f7eacb532727bb7
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1333%2Fderrickstolee%2Fbundle-redo%2Flist-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1333/derrickstolee/bundle-redo/list-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1333
--
gitgitgadget
next reply other threads:[~2022-08-22 15:19 UTC|newest]
Thread overview: 94+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-22 15:12 Derrick Stolee via GitGitGadget [this message]
2022-08-22 15:12 ` [PATCH 1/7] bundle-uri: create bundle_list struct and helpers Derrick Stolee via GitGitGadget
2022-08-22 17:57 ` Junio C Hamano
2022-08-22 15:12 ` [PATCH 2/7] bundle-uri: create base key-value pair parsing Derrick Stolee via GitGitGadget
2022-08-22 18:20 ` Junio C Hamano
2022-08-23 16:29 ` Derrick Stolee
2022-08-31 22:10 ` Jonathan Tan
2022-08-31 22:02 ` Glen Choo
2022-09-01 2:38 ` [PATCH 4/7] bundle-uri: unit test "key=value" parsing Teng Long
2022-08-22 15:12 ` [PATCH 3/7] bundle-uri: create "key=value" line parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-08-22 19:17 ` Junio C Hamano
2022-08-23 16:31 ` Derrick Stolee
2022-09-02 23:41 ` Josh Steadmon
2022-08-22 15:12 ` [PATCH 4/7] bundle-uri: unit test "key=value" parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-09-01 2:56 ` Teng Long
2022-08-22 15:12 ` [PATCH 5/7] bundle-uri: parse bundle list in config format Derrick Stolee via GitGitGadget
2022-08-22 19:25 ` Junio C Hamano
2022-08-23 16:43 ` Derrick Stolee
2022-08-31 22:18 ` Jonathan Tan
2022-09-01 8:05 ` Teng Long
2022-08-22 15:12 ` [PATCH 6/7] bundle-uri: limit recursion depth for bundle lists Derrick Stolee via GitGitGadget
2022-08-22 15:12 ` [PATCH 7/7] bundle-uri: fetch a list of bundles Derrick Stolee via GitGitGadget
2022-09-02 23:51 ` Josh Steadmon
2022-09-05 12:50 ` Teng Long
2022-09-08 17:10 ` Derrick Stolee
2022-09-09 14:33 ` [PATCH v2 0/9] Bundle URIs III: Parse and download from bundle lists Derrick Stolee via GitGitGadget
2022-09-09 14:33 ` [PATCH v2 1/9] bundle-uri: short-circuit capability parsing Derrick Stolee via GitGitGadget
2022-09-09 17:24 ` Junio C Hamano
2022-09-19 17:55 ` Derrick Stolee
2022-09-09 14:33 ` [PATCH v2 2/9] bundle-uri: use plain string in find_temp_filename() Derrick Stolee via GitGitGadget
2022-09-09 17:56 ` Junio C Hamano
2022-09-19 17:54 ` Derrick Stolee
2022-09-19 18:16 ` Junio C Hamano
2022-09-09 14:33 ` [PATCH v2 3/9] bundle-uri: create bundle_list struct and helpers Derrick Stolee via GitGitGadget
2022-09-09 14:33 ` [PATCH v2 4/9] bundle-uri: create base key-value pair parsing Derrick Stolee via GitGitGadget
2022-09-29 21:49 ` Jonathan Tan
2022-09-09 14:33 ` [PATCH v2 5/9] bundle-uri: create "key=value" line parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-09-09 14:33 ` [PATCH v2 6/9] bundle-uri: unit test "key=value" parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-09-09 14:33 ` [PATCH v2 7/9] bundle-uri: parse bundle list in config format Derrick Stolee via GitGitGadget
2022-09-09 14:33 ` [PATCH v2 8/9] bundle-uri: limit recursion depth for bundle lists Derrick Stolee via GitGitGadget
2022-09-09 14:33 ` [PATCH v2 9/9] bundle-uri: fetch a list of bundles Derrick Stolee via GitGitGadget
2022-09-29 21:58 ` Jonathan Tan
2022-09-30 12:49 ` Derrick Stolee
2022-09-26 13:19 ` [PATCH v2 0/9] Bundle URIs III: Parse and download from bundle lists Derrick Stolee
2022-09-26 19:10 ` Junio C Hamano
2022-09-29 22:00 ` Jonathan Tan
2022-09-30 13:21 ` Derrick Stolee
2022-10-04 12:34 ` [PATCH v3 " Derrick Stolee via GitGitGadget
2022-10-04 12:34 ` [PATCH v3 1/9] bundle-uri: use plain string in find_temp_filename() Derrick Stolee via GitGitGadget
2022-10-04 12:34 ` [PATCH v3 2/9] bundle-uri: create bundle_list struct and helpers Derrick Stolee via GitGitGadget
2022-10-04 12:34 ` [PATCH v3 3/9] bundle-uri: create base key-value pair parsing Derrick Stolee via GitGitGadget
2022-10-04 12:34 ` [PATCH v3 4/9] bundle-uri: create "key=value" line parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-04 12:34 ` [PATCH v3 5/9] bundle-uri: unit test "key=value" parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-04 12:34 ` [PATCH v3 6/9] bundle-uri: parse bundle list in config format Derrick Stolee via GitGitGadget
2022-10-04 12:34 ` [PATCH v3 7/9] bundle-uri: limit recursion depth for bundle lists Derrick Stolee via GitGitGadget
2022-10-04 12:34 ` [PATCH v3 8/9] bundle-uri: fetch a list of bundles Derrick Stolee via GitGitGadget
2022-10-04 21:44 ` Jonathan Tan
2022-10-07 13:29 ` Derrick Stolee
2022-10-04 12:34 ` [PATCH v3 9/9] bundle-uri: suppress stderr from remote-https Derrick Stolee via GitGitGadget
2022-10-10 16:04 ` [PATCH v4 00/11] Bundle URIs III: Parse and download from bundle lists Derrick Stolee via GitGitGadget
2022-10-10 16:04 ` [PATCH v4 01/11] bundle-uri: use plain string in find_temp_filename() Derrick Stolee via GitGitGadget
2022-10-10 16:04 ` [PATCH v4 02/11] bundle-uri: create bundle_list struct and helpers Derrick Stolee via GitGitGadget
2022-10-10 16:04 ` [PATCH v4 03/11] bundle-uri: create base key-value pair parsing Derrick Stolee via GitGitGadget
2022-10-10 16:04 ` [PATCH v4 04/11] bundle-uri: create "key=value" line parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-10 16:04 ` [PATCH v4 05/11] bundle-uri: unit test "key=value" parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-10 16:04 ` [PATCH v4 06/11] bundle-uri: parse bundle list in config format Derrick Stolee via GitGitGadget
2022-10-10 16:04 ` [PATCH v4 07/11] bundle-uri: limit recursion depth for bundle lists Derrick Stolee via GitGitGadget
2022-10-10 16:04 ` [PATCH v4 08/11] bundle: add flags to verify_bundle(), skip walk Derrick Stolee via GitGitGadget
2022-10-10 17:27 ` Junio C Hamano
2022-10-10 18:13 ` Derrick Stolee
2022-10-10 18:40 ` Junio C Hamano
2022-10-11 19:04 ` Derrick Stolee
2022-10-10 16:04 ` [PATCH v4 09/11] bundle-uri: fetch a list of bundles Derrick Stolee via GitGitGadget
2022-10-10 16:04 ` [PATCH v4 10/11] bundle-uri: quiet failed unbundlings Derrick Stolee via GitGitGadget
2022-10-10 16:04 ` [PATCH v4 11/11] bundle-uri: suppress stderr from remote-https Derrick Stolee via GitGitGadget
2022-10-12 12:52 ` [PATCH v5 00/12] Bundle URIs III: Parse and download from bundle lists Derrick Stolee via GitGitGadget
2022-10-12 12:52 ` [PATCH v5 01/12] bundle-uri: use plain string in find_temp_filename() Derrick Stolee via GitGitGadget
2022-10-12 12:52 ` [PATCH v5 02/12] bundle-uri: create bundle_list struct and helpers Derrick Stolee via GitGitGadget
2022-10-12 12:52 ` [PATCH v5 03/12] bundle-uri: create base key-value pair parsing Derrick Stolee via GitGitGadget
2022-10-12 12:52 ` [PATCH v5 04/12] bundle-uri: create "key=value" line parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-12 12:52 ` [PATCH v5 05/12] bundle-uri: unit test "key=value" parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-12 12:52 ` [PATCH v5 06/12] bundle-uri: parse bundle list in config format Derrick Stolee via GitGitGadget
2022-10-12 12:52 ` [PATCH v5 07/12] bundle-uri: limit recursion depth for bundle lists Derrick Stolee via GitGitGadget
2022-10-12 12:52 ` [PATCH v5 08/12] bundle: properly clear all revision flags Derrick Stolee via GitGitGadget
2022-10-12 16:17 ` Junio C Hamano
2022-10-12 12:52 ` [PATCH v5 09/12] bundle-uri: fetch a list of bundles Derrick Stolee via GitGitGadget
2022-10-26 19:06 ` Junio C Hamano
2022-10-12 12:52 ` [PATCH v5 10/12] bundle: add flags to verify_bundle() Derrick Stolee via GitGitGadget
2022-10-12 12:52 ` [PATCH v5 11/12] bundle-uri: quiet failed unbundlings Derrick Stolee via GitGitGadget
2022-10-12 16:32 ` Junio C Hamano
2022-10-12 12:52 ` [PATCH v5 12/12] bundle-uri: suppress stderr from remote-https Derrick Stolee via GitGitGadget
2022-10-26 18:54 ` Junio C Hamano
2022-10-26 14:34 ` [PATCH v5 00/12] Bundle URIs III: Parse and download from bundle lists Derrick Stolee
2022-10-26 16:06 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=pull.1333.git.1661181174.gitgitgadget@gmail.com \
--to=gitgitgadget@gmail.com \
--cc=avarab@gmail.com \
--cc=derrickstolee@github.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=me@ttaylorr.com \
--cc=mjcheetham@outlook.com \
--cc=newren@gmail.com \
--cc=steadmon@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).