Re: RFC: Resumable clone based on hybrid "smart" and "dumb" HTTP

[Jonathan Nieder <jrnieder@gmail.com>]

Jeff King wrote:
> On Wed, Feb 10, 2016 at 12:11:46PM -0800, Shawn Pearce wrote:

>> Several of us at $DAY_JOB talked about this more today and thought a
>> variation makes more sense:
>>
>> 1. Clients attempting clone ask for /info/refs?service=git-upload-pack
>> like they do today.
>>
>> 2. Servers that support resumable clone include a "resumable"
>> capability in the advertisement.
>
> Because the magic happens in the git protocol, that would mean this does
> not have to be limited to git-over-http. It could be "resumable=<url>"
> to point the client anywhere (the same server over a different protocol,
> another server, etc).

Thanks for bringing this up.  A worry with putting the URL in the
capabilities line is that it makes it easy to run into the 1000-byte
limit.  It's been a while since v1.8.3-rc0~148^2~6 (pkt-line: provide
a LARGE_PACKET_MAX static buffer, 2013-02-20) but we still can't
rely on clients having that applied.

(I also haven't checked whether current versions of git are able to
handle longer capability strings with that patch applied.)

Another nice thing about using a 302 is that you can set cookies
during the redirect, which might make authenticated access easier.
(That said, authenticated access through e.g. signed URLs can work
fine without that.)

[...]
> Clients do not have to _just_ fetch a packfile. They could get a bundle
> file that contains the roots along with the packfile. I know that one of
> your goals is not duplicating the storage of the packfile on the server,
> but it would not be hard for the server to store the packfile and the
> bundle header separately, and concatenate them on the fly.

Doesn't that prevent using a git-unaware file transfer service to
serve the files?

It also means the client can't use the downloaded file as-is --- they
need to separate the root list from the packfile (that's not a big
deal; just some added complication to switch files at the appropriate
moment during download if you want to avoid temporarily using twice
the space).

That said, both these problems are avoided by serving the 'split
bundle' you described as-is instead of concatenating.

[...]
> And you'll notice, too, that all of the bundle-http magic kicks in
> during step 2 because the client sees they're grabbing a bundle. Which
> means that the <url> in step 1 doesn't _have_ to be a bundle. It can be
> "go fetch from kernel.org, then come back to me".

I think that use case brings in complications that make it not
necessarily worth it.  In this example, if kernel.org is serving pack
files, why shouldn't I point directly at the advertised pack CDN URL
instead of adding an extra hop that puts added load on kernel.org
servers?

Allowing an arbitrary "fetch from here first" capability is very
flexible.  I guess my fear comes from not knowing what the flexibility
buys beyond aesthetics.  (My motivation comes from the example of
alternates: it is pretty and very flexible and ended up as a support
and maintenance headache instead of being widely useful.  I think what
you are proposing is more harmless but I'd still want to have an
example of what it's used for before going in that direction.)

Thanks,
Jonathan