git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 0/7] Bundle URIs III: Parse and download from bundle lists
@ 2022-08-22 15:12 Derrick Stolee via GitGitGadget
  2022-08-22 15:12 ` [PATCH 1/7] bundle-uri: create bundle_list struct and helpers Derrick Stolee via GitGitGadget
                   ` (7 more replies)
  0 siblings, 8 replies; 94+ messages in thread
From: Derrick Stolee via GitGitGadget @ 2022-08-22 15:12 UTC (permalink / raw)
  To: git; +Cc: gitster, me, newren, avarab, mjcheetham, steadmon, Derrick Stolee

This is the third series building the bundle URI feature. It is built on top
of ds/bundle-uri-clone, which introduced 'git clone --bundle-uri=' where is
a URI to a bundle file. This series adds the capability of downloading and
parsing a bundle list and then downloading the URIs in that list.

The core functionality of bundle lists is implemented by creating data
structures from a list of key-value pairs. These pairs can come from a
plain-text file in Git config format, but in the future, we will support the
list being supplied by packet lines over Git's protocol v2 in the
'bundle-uri' command (reserved for the next series).

The patches are organized in this way:

 1. Patches 1-2 create the bundle list data structures and the logic for
    populating the list from key-value pairs.

 2. Patches 3-4 teach Git to parse "key=value" lines to construct a bundle
    list. Add unit tests that ensure this logic constructs lists correctly.
    These patches are adapted from Ævar's RFC [1] and were previously seen
    in my combined RFC [2].

 3. Patch 5 teaches Git to parse Git config files into bundle lists.

 4. Patches 6-7 implement the ability to download a bundle list and
    recursively download the contained bundles (and possibly the bundle
    lists within). This is limited by a constant depth to avoid issues with
    cycles or otherwise incorrectly configured bundle lists.

[1]
https://lore.kernel.org/git/RFC-cover-v2-00.36-00000000000-20220418T165545Z-avarab@gmail.com/

[2]
https://lore.kernel.org/git/pull.1234.git.1653072042.gitgitgadget@gmail.com/

At the end of this series, users can bootstrap clones using 'git clone
--bundle-uri= ' where points to a bundle list instead of a single bundle
file.

As outlined in the design document [1], the next steps after this are:

 1. Implement the protocol v2 verb, re-using the bundle list logic from (2).
    Use this to auto-discover bundle URIs during 'git clone' (behind a
    config option). [2]
 2. Implement the 'creationToken' heuristic, allowing incremental 'git
    fetch' commands to download a bundle list from a configured URI, and
    only download bundles that are new based on the creation token values.
    [3]

I have prepared some of this work as pull requests on my personal fork so
curious readers can look ahead to where we are going:

[3]
https://lore.kernel.org/git/pull.1248.v3.git.1658757188.gitgitgadget@gmail.com

[4] https://github.com/derrickstolee/git/pull/21

[5] https://github.com/derrickstolee/git/pull/22

Thanks,

 * Stolee

Derrick Stolee (5):
  bundle-uri: create bundle_list struct and helpers
  bundle-uri: create base key-value pair parsing
  bundle-uri: parse bundle list in config format
  bundle-uri: limit recursion depth for bundle lists
  bundle-uri: fetch a list of bundles

Ævar Arnfjörð Bjarmason (2):
  bundle-uri: create "key=value" line parsing
  bundle-uri: unit test "key=value" parsing

 Documentation/config.txt        |   2 +
 Documentation/config/bundle.txt |  22 ++
 Makefile                        |   1 +
 bundle-uri.c                    | 442 +++++++++++++++++++++++++++++++-
 bundle-uri.h                    |  98 ++++++-
 t/helper/test-bundle-uri.c      |  90 +++++++
 t/helper/test-tool.c            |   1 +
 t/helper/test-tool.h            |   1 +
 t/t5558-clone-bundle-uri.sh     |  93 +++++++
 t/t5750-bundle-uri-parse.sh     | 141 ++++++++++
 t/test-lib-functions.sh         |  11 +
 11 files changed, 889 insertions(+), 13 deletions(-)
 create mode 100644 Documentation/config/bundle.txt
 create mode 100644 t/helper/test-bundle-uri.c
 create mode 100755 t/t5750-bundle-uri-parse.sh


base-commit: e21e663cd1942df29979d3e01f7eacb532727bb7
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1333%2Fderrickstolee%2Fbundle-redo%2Flist-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1333/derrickstolee/bundle-redo/list-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1333
-- 
gitgitgadget

^ permalink raw reply	[flat|nested] 94+ messages in thread

end of thread, other threads:[~2022-10-26 19:09 UTC | newest]

Thread overview: 94+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-22 15:12 [PATCH 0/7] Bundle URIs III: Parse and download from bundle lists Derrick Stolee via GitGitGadget
2022-08-22 15:12 ` [PATCH 1/7] bundle-uri: create bundle_list struct and helpers Derrick Stolee via GitGitGadget
2022-08-22 17:57   ` Junio C Hamano
2022-08-22 15:12 ` [PATCH 2/7] bundle-uri: create base key-value pair parsing Derrick Stolee via GitGitGadget
2022-08-22 18:20   ` Junio C Hamano
2022-08-23 16:29     ` Derrick Stolee
2022-08-31 22:10       ` Jonathan Tan
2022-08-31 22:02   ` Glen Choo
2022-09-01  2:38   ` [PATCH 4/7] bundle-uri: unit test "key=value" parsing Teng Long
2022-08-22 15:12 ` [PATCH 3/7] bundle-uri: create "key=value" line parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-08-22 19:17   ` Junio C Hamano
2022-08-23 16:31     ` Derrick Stolee
2022-09-02 23:41   ` Josh Steadmon
2022-08-22 15:12 ` [PATCH 4/7] bundle-uri: unit test "key=value" parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-09-01  2:56   ` Teng Long
2022-08-22 15:12 ` [PATCH 5/7] bundle-uri: parse bundle list in config format Derrick Stolee via GitGitGadget
2022-08-22 19:25   ` Junio C Hamano
2022-08-23 16:43     ` Derrick Stolee
2022-08-31 22:18     ` Jonathan Tan
2022-09-01  8:05   ` Teng Long
2022-08-22 15:12 ` [PATCH 6/7] bundle-uri: limit recursion depth for bundle lists Derrick Stolee via GitGitGadget
2022-08-22 15:12 ` [PATCH 7/7] bundle-uri: fetch a list of bundles Derrick Stolee via GitGitGadget
2022-09-02 23:51   ` Josh Steadmon
2022-09-05 12:50   ` Teng Long
2022-09-08 17:10     ` Derrick Stolee
2022-09-09 14:33 ` [PATCH v2 0/9] Bundle URIs III: Parse and download from bundle lists Derrick Stolee via GitGitGadget
2022-09-09 14:33   ` [PATCH v2 1/9] bundle-uri: short-circuit capability parsing Derrick Stolee via GitGitGadget
2022-09-09 17:24     ` Junio C Hamano
2022-09-19 17:55       ` Derrick Stolee
2022-09-09 14:33   ` [PATCH v2 2/9] bundle-uri: use plain string in find_temp_filename() Derrick Stolee via GitGitGadget
2022-09-09 17:56     ` Junio C Hamano
2022-09-19 17:54       ` Derrick Stolee
2022-09-19 18:16         ` Junio C Hamano
2022-09-09 14:33   ` [PATCH v2 3/9] bundle-uri: create bundle_list struct and helpers Derrick Stolee via GitGitGadget
2022-09-09 14:33   ` [PATCH v2 4/9] bundle-uri: create base key-value pair parsing Derrick Stolee via GitGitGadget
2022-09-29 21:49     ` Jonathan Tan
2022-09-09 14:33   ` [PATCH v2 5/9] bundle-uri: create "key=value" line parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-09-09 14:33   ` [PATCH v2 6/9] bundle-uri: unit test "key=value" parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-09-09 14:33   ` [PATCH v2 7/9] bundle-uri: parse bundle list in config format Derrick Stolee via GitGitGadget
2022-09-09 14:33   ` [PATCH v2 8/9] bundle-uri: limit recursion depth for bundle lists Derrick Stolee via GitGitGadget
2022-09-09 14:33   ` [PATCH v2 9/9] bundle-uri: fetch a list of bundles Derrick Stolee via GitGitGadget
2022-09-29 21:58     ` Jonathan Tan
2022-09-30 12:49       ` Derrick Stolee
2022-09-26 13:19   ` [PATCH v2 0/9] Bundle URIs III: Parse and download from bundle lists Derrick Stolee
2022-09-26 19:10     ` Junio C Hamano
2022-09-29 22:00       ` Jonathan Tan
2022-09-30 13:21         ` Derrick Stolee
2022-10-04 12:34   ` [PATCH v3 " Derrick Stolee via GitGitGadget
2022-10-04 12:34     ` [PATCH v3 1/9] bundle-uri: use plain string in find_temp_filename() Derrick Stolee via GitGitGadget
2022-10-04 12:34     ` [PATCH v3 2/9] bundle-uri: create bundle_list struct and helpers Derrick Stolee via GitGitGadget
2022-10-04 12:34     ` [PATCH v3 3/9] bundle-uri: create base key-value pair parsing Derrick Stolee via GitGitGadget
2022-10-04 12:34     ` [PATCH v3 4/9] bundle-uri: create "key=value" line parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-04 12:34     ` [PATCH v3 5/9] bundle-uri: unit test "key=value" parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-04 12:34     ` [PATCH v3 6/9] bundle-uri: parse bundle list in config format Derrick Stolee via GitGitGadget
2022-10-04 12:34     ` [PATCH v3 7/9] bundle-uri: limit recursion depth for bundle lists Derrick Stolee via GitGitGadget
2022-10-04 12:34     ` [PATCH v3 8/9] bundle-uri: fetch a list of bundles Derrick Stolee via GitGitGadget
2022-10-04 21:44       ` Jonathan Tan
2022-10-07 13:29         ` Derrick Stolee
2022-10-04 12:34     ` [PATCH v3 9/9] bundle-uri: suppress stderr from remote-https Derrick Stolee via GitGitGadget
2022-10-10 16:04     ` [PATCH v4 00/11] Bundle URIs III: Parse and download from bundle lists Derrick Stolee via GitGitGadget
2022-10-10 16:04       ` [PATCH v4 01/11] bundle-uri: use plain string in find_temp_filename() Derrick Stolee via GitGitGadget
2022-10-10 16:04       ` [PATCH v4 02/11] bundle-uri: create bundle_list struct and helpers Derrick Stolee via GitGitGadget
2022-10-10 16:04       ` [PATCH v4 03/11] bundle-uri: create base key-value pair parsing Derrick Stolee via GitGitGadget
2022-10-10 16:04       ` [PATCH v4 04/11] bundle-uri: create "key=value" line parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-10 16:04       ` [PATCH v4 05/11] bundle-uri: unit test "key=value" parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-10 16:04       ` [PATCH v4 06/11] bundle-uri: parse bundle list in config format Derrick Stolee via GitGitGadget
2022-10-10 16:04       ` [PATCH v4 07/11] bundle-uri: limit recursion depth for bundle lists Derrick Stolee via GitGitGadget
2022-10-10 16:04       ` [PATCH v4 08/11] bundle: add flags to verify_bundle(), skip walk Derrick Stolee via GitGitGadget
2022-10-10 17:27         ` Junio C Hamano
2022-10-10 18:13           ` Derrick Stolee
2022-10-10 18:40             ` Junio C Hamano
2022-10-11 19:04               ` Derrick Stolee
2022-10-10 16:04       ` [PATCH v4 09/11] bundle-uri: fetch a list of bundles Derrick Stolee via GitGitGadget
2022-10-10 16:04       ` [PATCH v4 10/11] bundle-uri: quiet failed unbundlings Derrick Stolee via GitGitGadget
2022-10-10 16:04       ` [PATCH v4 11/11] bundle-uri: suppress stderr from remote-https Derrick Stolee via GitGitGadget
2022-10-12 12:52       ` [PATCH v5 00/12] Bundle URIs III: Parse and download from bundle lists Derrick Stolee via GitGitGadget
2022-10-12 12:52         ` [PATCH v5 01/12] bundle-uri: use plain string in find_temp_filename() Derrick Stolee via GitGitGadget
2022-10-12 12:52         ` [PATCH v5 02/12] bundle-uri: create bundle_list struct and helpers Derrick Stolee via GitGitGadget
2022-10-12 12:52         ` [PATCH v5 03/12] bundle-uri: create base key-value pair parsing Derrick Stolee via GitGitGadget
2022-10-12 12:52         ` [PATCH v5 04/12] bundle-uri: create "key=value" line parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-12 12:52         ` [PATCH v5 05/12] bundle-uri: unit test "key=value" parsing Ævar Arnfjörð Bjarmason via GitGitGadget
2022-10-12 12:52         ` [PATCH v5 06/12] bundle-uri: parse bundle list in config format Derrick Stolee via GitGitGadget
2022-10-12 12:52         ` [PATCH v5 07/12] bundle-uri: limit recursion depth for bundle lists Derrick Stolee via GitGitGadget
2022-10-12 12:52         ` [PATCH v5 08/12] bundle: properly clear all revision flags Derrick Stolee via GitGitGadget
2022-10-12 16:17           ` Junio C Hamano
2022-10-12 12:52         ` [PATCH v5 09/12] bundle-uri: fetch a list of bundles Derrick Stolee via GitGitGadget
2022-10-26 19:06           ` Junio C Hamano
2022-10-12 12:52         ` [PATCH v5 10/12] bundle: add flags to verify_bundle() Derrick Stolee via GitGitGadget
2022-10-12 12:52         ` [PATCH v5 11/12] bundle-uri: quiet failed unbundlings Derrick Stolee via GitGitGadget
2022-10-12 16:32           ` Junio C Hamano
2022-10-12 12:52         ` [PATCH v5 12/12] bundle-uri: suppress stderr from remote-https Derrick Stolee via GitGitGadget
2022-10-26 18:54           ` Junio C Hamano
2022-10-26 14:34         ` [PATCH v5 00/12] Bundle URIs III: Parse and download from bundle lists Derrick Stolee
2022-10-26 16:06           ` Junio C Hamano

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).