Re: Git-aware HTTP transport

[Tarmigan <tarmigan+git@gmail.com>]

On Mon, Sep 1, 2008 at 11:06 PM, Shawn O. Pearce <spearce@spearce.org> wrote:
>> What does the server do after a 000cgive-up ?  Does the server send
>> back a complete pack (like a new clone) or if not, how does clone work
>> over smart http?
>
> When the server receives a "give-up" it needs to create a pack
> based on "git rev-list --objects-boundary $WANT --not $COMMON".
> If the set $COMMON is non-empty then its a partial pack; if $COMMON
> is empty then its a full clone.  This is what the native protocol
> does when the client gives up.

OK, that makes sense now.

>> Does that mean that if I fall more than 256 commits
>> behind, I have to redownload the whole repo?
>
> You are thinking the wrong way.  If you have more than 256 commits
> that the other side doesn't have you may give up too early.
> For that to be true you need to create 256 commits locally that
> aren't on the remote peer and whose timestamps are all ahead of
> the commits you last fetched from the remote peer.
>
> Yes, it can happen.  But its less likely than you think because
> we're talking about you doing 256 commits worth of development and
> not picking up any new commits from remote peers in the middle of
> that time period.  Get just one and it resets the counter back to
> 0 and allows it to try another 256 commits before giving up.
>
> I should amend this section to talk about what giving up here
> really means.  If we have nothing sent in common yet or maybe
> very little sent in common we may have existing remote refs tied
> to this URL in .git/config that can send, and we may have one or
> more annotated tags that we know for a fact are in common as both
> peers have the same tag name pointing to the same tag object.
>
> A smart(er) client might try to toss some recently dated annotated
> tags at the server before throwing a give-up if it would otherwise
> throw a give-up.  Its likely to narrow the result set, and doesn't
> hurt if it doesn't.

Yes, throwing in tags and remotes as a last resort sounds like a good idea.

>> So again, if the client falls more than 1000 commits behind (not hard
>> to do for example during the linux merge window), and then the client
>> WANTs HEAD^1001, what happens?  Does the get nothing from the server,
>> or does the client essentially reclone, or I am missing something?
>
> Oh, this is a live-lock condition.  If the client grabs the list of
> refs from the server, then has to wait 100 ms to get back to the
> server and start upload-pack (due to latency) and in that 100ms
> window Linus shoves a 1001 commit merge into his tree then yes,
> the server may abort and tell the client "error invalid want".

Ahh, now I get it.  Somehow I forgot that the WANTs were only boundary
commits and not a list of all the commits that the client wants.

On Mon, Sep 1, 2008 at 11:13 PM, Shawn O. Pearce <spearce@spearce.org> wrote:
> "H. Peter Anvin" <hpa@zytor.com> wrote:
>> Shawn O. Pearce wrote:
>>>
>>> Correct.  Today _none_ of the transport protocols allow the server
>>> to force the client to use some sort of reference repository for an
>>> initial clone.  There are likely two reasons for this:
>>>
>>>  *) Its a lot simpler to program to just get everything from
>>>     one location.
>>>
>>>  *) If you really are forking an open source project then in
>>>     some cases you may need to distribute the full source,
>>>      not your delta.  You may just as well distribute the full
>>>      source and call it a day.
>>>
>>
>> 3) it encourages single points of failure.
>
> Or bad network usage, as I pointed out later about an India user
> unknowingly being forced into a US based mirror when another was
> closer to them.
>
> I didn't make it clear in my response but I'm really against our
> protocol having this sort of explicit redirect.  I'd rather put a
> requirement in that says "Unless you have X,Y,Z in common with me
> (directly or indirectly) I'm just not going to give you a pack".

OK, this all makes sense. http:// and git:// are probably the wrong
protocols to reduce bandwidth for the server for new clones.  Long
term, maybe gittorrent will be the right solution...

Thanks,
Tarmigan