archive mirror
 help / color / mirror / Atom feed
From: "Ævar Arnfjörð Bjarmason" <>
To: Jonathan Nieder <>
Cc:, Jeff King <>,
	Patrick Steinhardt <>,
	Christian Couder <>,
	Albert Cui <>,
	Jonathan Tan <>
Subject: Re: [RFC PATCH 00/13] Add bundle-uri: resumably clones, static "dumb" CDN etc.
Date: Fri, 06 Aug 2021 18:26:26 +0200	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <YQ1JW8hHmG8B/>

On Fri, Aug 06 2021, Jonathan Nieder wrote:

> Hi,
> Ævar Arnfjörð Bjarmason wrote:
>> We're in the 2.33.0 rc cycle, and I'd hoped to have some more prep
>> work for this integrated already, but for now here's something
>> interesting I've been working on for early commentary/feedback.
>> This adds the the ability to protocol v2 for servers to optimistically
>> pre-seed supporting clients with one or more bundles via a new
>> "bundle-uri" protocol extension.
> My initial thought here is that even though this includes a comparison
> to packfile URIs, I suspect you're underestimating them. :)
> Would it be possible to do the same pre-seeding using the packfile
> URIs protocol?  Nothing stops a server from sending more objects than
> the client asked for.  Is the issue that you want the client to be
> able to list "have"s based on that pack?  Can't the server obtain that
> same information at the same time as it obtains the bundle URL?

Hi. Thanks for taking a quick look.

I think that the changes to Documentation/technical/protocol-v2.txt in
01/13[1] and the Documentation/technical/bundle-uri.txt document added
in 13/13[2] should address these questions.

Or perhaps not, but they're my currently my best effort to explain the
differences between the two and how they interact. So I think it's best
to point to those instead of coming up with something in this reply,
which'll inevitably be an incomplete rewrite of much of that.

In short, there are use-cases that packfile-uri is inherently unsuitable
for, or rather changing the packfile-uri feature to support them would
pretty much make it indistinguishable from this bundle-uri mechanism,
which I think would just add more confusion to the protocol.

> The reason I ask is that this contains a number of differences
> relative to packfile URIs, most noticeably the use of bundles instead
> of packfiles as the format for the static content.  If we were
> starting from scratch and chose this design _instead_ of packfile URIs
> then that could make sense (though there are issues with the bundle
> format that we can also go into), but in a world where people are also
> using packfile URIs it makes for a kind of confusing UX.  Is a server
> operator expected to put both kinds of files on CDN and double their
> storage bill?  Is this meant as an alternative, a replacement, or
> something that combines well together with the packfile URIs feature?
> What does the intended end state look like?

The two are complimentary, i.e. it's meant as something that combines
well with packfile-uri. One (packfile-uri) is meant as a dumb
pre-seeding of objects from static URLs that happens pre-negotiation.

The other (packfile-uri) is a post-negotiation mechanism for giving a
client not only a PACK, but complimenting it with a URI, together they
two form one logical complete PACK. (I know that you know this, but for
anyone reading along...).

> Projects like chromium have been using packfile URIs in production for
> about 11 months now and it's been working well.  Because of that, I'd
> be interested in understanding its shortcomings and improving it in
> place --- or in other words, I want _you_ to benefit from them instead
> of having to create an alternative to them.  Alternatively, if the
> packfile URIs protocol is fundamentally flawed, then I'd like us to
> understand that early and act on it instead of creating a parallel
> alternative and waiting for it to bitrot.

I don't think there's any real shortcomings of packfile-uri in the senes
that it makes sense to improve it in the direction of this bundle-uri
mechanism, they're simply doing different things at different phases in
the dialog.

When packfile-uri was initially being discussed I was interested in
getting something like what this bundle-uri mechanism to be a part it,
but it was clear from discussions with Jonathan Tan that he was taking
packfile-uri in a very different direction. Links to those discussions:

A demo of this feature playing nice with a server that supports
packfile-uri, although I haven't actually gotten it to also send me a
packfile-uri if I pre-seed with bundles (perhaps it's only doing it on
full clones?):

    git -c fetch.uriprotocols=https \
        -c transfer.injectBundleURI="" \
        -c transfer.injectBundleURI="" \
        clone --bare /tmp/codesearch.git

Will emit:
    Cloning into bare repository '/tmp/codesearch.git'...
    Receiving bundle (1/2): 100% (322661/322661), 89.66 MiB | 0 bytes/s, done.
    Resolving deltas: 100% (142478/142478), done.
    Receiving bundle (2/2): 100% (69378/69378), 5.51 MiB | 0 bytes/s, done.
    Resolving deltas: 100% (52558/52558)completed with 4 local objects
    remote: Total 1958 (delta 4), reused 1958 (delta 4)
    Receiving objects: 100% (1958/1958), 1.60 MiB | 0 bytes/s, done.
    Resolving deltas: 100% (4/4)completed with 3 local objects
    Checking connectivity: 393997, done.

I.e. I produced a combination of bundles going up to a commit at the
start of this month, so that last "Receiving objects" is the only
dynamic fetching of objects via the negotiation.


  reply	other threads:[~2021-08-06 17:37 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-05 15:07 [RFC PATCH 00/13] Add bundle-uri: resumably clones, static "dumb" CDN etc Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 01/13] serve: add command to advertise bundle URIs Ævar Arnfjörð Bjarmason
2021-08-10 13:58   ` Derrick Stolee
2021-08-23 13:25     ` Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 02/13] bundle-uri client: add "bundle-uri" parsing + tests Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 03/13] connect.c: refactor sending of agent & object-format Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 04/13] bundle-uri client: add minimal NOOP client Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 05/13] bundle-uri client: add "git ls-remote-bundle-uri" Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 06/13] bundle-uri client: add transfer.injectBundleURI support Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 07/13] bundle-uri client: add boolean transfer.bundleURI setting Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 08/13] bundle.h: make "fd" version of read_bundle_header() public Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 09/13] fetch-pack: add a deref_without_lazy_fetch_extended() Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 10/13] fetch-pack: move --keep=* option filling to a function Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 11/13] index-pack: add --progress-title option Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 12/13] bundle-uri client: support for bundle-uri with "clone" Ævar Arnfjörð Bjarmason
2021-08-05 15:07 ` [RFC PATCH 13/13] bundle-uri docs: add design notes Ævar Arnfjörð Bjarmason
2021-08-24 21:48   ` brian m. carlson
2021-08-24 22:33     ` Ævar Arnfjörð Bjarmason
2021-08-06 14:38 ` [RFC PATCH 00/13] Add bundle-uri: resumably clones, static "dumb" CDN etc Jonathan Nieder
2021-08-06 16:26   ` Ævar Arnfjörð Bjarmason [this message]
2021-08-06 20:40     ` Jonathan Nieder
2021-08-07  2:19       ` Ævar Arnfjörð Bjarmason
2021-08-10 13:55 ` Derrick Stolee
2021-08-23 13:28   ` Ævar Arnfjörð Bjarmason
2021-08-24  2:03     ` Derrick Stolee
2021-08-24 22:00       ` Ævar Arnfjörð Bjarmason

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).