git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* maybe breakage with latest git-pull and http protocol
@ 2005-10-13  0:53 Randal L. Schwartz
  2005-10-13  5:50 ` Junio C Hamano
                   ` (2 more replies)
  0 siblings, 3 replies; 17+ messages in thread
From: Randal L. Schwartz @ 2005-10-13  0:53 UTC (permalink / raw)
  To: git


I updated git to d06b689a933f6d2130f8afdf1ac0ddb83eeb59ab,
then compiled and installed.

When I went to "git-pull" on my cogito archive (which I had edited
to use HTTP instead of RSYNC), I got into trouble.  Unfortunately,
I changed it to rsync to force cogito into a sane state before
I realized that this would be a good bug report. :)

This is perhaps just a heads-up that the recent git-pull might be
broken with respect to http updates.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-13  0:53 maybe breakage with latest git-pull and http protocol Randal L. Schwartz
@ 2005-10-13  5:50 ` Junio C Hamano
  2005-10-14 10:58 ` Randal L. Schwartz
  2005-10-20 17:43 ` Nick Hengeveld
  2 siblings, 0 replies; 17+ messages in thread
From: Junio C Hamano @ 2005-10-13  5:50 UTC (permalink / raw)
  To: Randal L. Schwartz; +Cc: git

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> I updated git to d06b689a933f6d2130f8afdf1ac0ddb83eeb59ab,
> then compiled and installed.
>
> When I went to "git-pull" on my cogito archive (which I had edited
> to use HTTP instead of RSYNC), I got into trouble.  Unfortunately,
> I changed it to rsync to force cogito into a sane state before
> I realized that this would be a good bug report. :)

Indeed I wish we could see the set of refs you had and output
from fsck-objects before the failed git-pull and after.

One thing I am aware of is that cogito repository at kernel.org
is not set up to be HTTP friendly -- it lacks info/refs file
git-clone uses for discovery of the available refs.

Cogito's clone/fetch over HTTP uses recursive wget for
discovery, and I presume that is one of the reasons nobody
noticed this.  Another reason may probably be that more people
use rsync transport.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-13  0:53 maybe breakage with latest git-pull and http protocol Randal L. Schwartz
  2005-10-13  5:50 ` Junio C Hamano
@ 2005-10-14 10:58 ` Randal L. Schwartz
  2005-10-14 15:42   ` Junio C Hamano
  2005-10-20 17:43 ` Nick Hengeveld
  2 siblings, 1 reply; 17+ messages in thread
From: Randal L. Schwartz @ 2005-10-14 10:58 UTC (permalink / raw)
  To: git

>>>>> "Randal" == Randal L Schwartz <merlyn@stonehenge.com> writes:

Randal> I updated git to d06b689a933f6d2130f8afdf1ac0ddb83eeb59ab,
Randal> then compiled and installed.

Randal> When I went to "git-pull" on my cogito archive (which I had edited
Randal> to use HTTP instead of RSYNC), I got into trouble.  Unfortunately,
Randal> I changed it to rsync to force cogito into a sane state before
Randal> I realized that this would be a good bug report. :)

Randal> This is perhaps just a heads-up that the recent git-pull might be
Randal> broken with respect to http updates.

Even after updating git this morning, git-pull still seems to be broken
with respect to http://www.kernel.org/.
Is http pulling broken for good now?  Or is someone looking at this?

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-14 10:58 ` Randal L. Schwartz
@ 2005-10-14 15:42   ` Junio C Hamano
  2005-10-14 16:18     ` Randal L. Schwartz
  0 siblings, 1 reply; 17+ messages in thread
From: Junio C Hamano @ 2005-10-14 15:42 UTC (permalink / raw)
  To: Randal L. Schwartz; +Cc: git

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> Even after updating git this morning, git-pull still seems to be broken
> with respect to http://www.kernel.org/.
> Is http pulling broken for good now?  Or is someone looking at this?

Sorry, but this is not a description of your problem helpful
enough for someone who is willing to look at it, I am afraid.
http://www.kernel.org/ has 80 or so repos (I counted about a
month ago so it may probably have more by now) --- which ones?

I have local repositories used only to test pulling into them,
and I pull from Linus 2.6 kernel, and my own git repository,
every other day or so, but haven't seen breakage, so I do not
think it is http://www.kernel.org/. in general.  If some
particular repository is not set up HTTP friendly I would
understand.

Also how does it fail?  Does cloning from scratch succeed but
updating a repo that was in sync a few days ago fail?  Does it
die silently and you find the breakage by running fsck-object,
or does it fail loudly with error messages?  If the latter what
does it say?

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-14 15:42   ` Junio C Hamano
@ 2005-10-14 16:18     ` Randal L. Schwartz
  2005-10-14 19:56       ` Daniel Barkalow
  0 siblings, 1 reply; 17+ messages in thread
From: Randal L. Schwartz @ 2005-10-14 16:18 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: git

>>>>> "Junio" == Junio C Hamano <junkio@cox.net> writes:

Junio> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>> Even after updating git this morning, git-pull still seems to be broken
>> with respect to http://www.kernel.org/.
>> Is http pulling broken for good now?  Or is someone looking at this?

Junio> Sorry, but this is not a description of your problem helpful
Junio> enough for someone who is willing to look at it, I am afraid.
Junio> http://www.kernel.org/ has 80 or so repos (I counted about a
Junio> month ago so it may probably have more by now) --- which ones?

My previous message mentioned cogito.git breaking.  This message
was triggered because git.git itself broke.

Unfortunately, git.git hasn't been updated since the last time I
worked around the problem by switching to rsync again, but when it
does, I'll capture the breakage this time.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-14 16:18     ` Randal L. Schwartz
@ 2005-10-14 19:56       ` Daniel Barkalow
  2005-10-15 13:03         ` Randal L. Schwartz
  0 siblings, 1 reply; 17+ messages in thread
From: Daniel Barkalow @ 2005-10-14 19:56 UTC (permalink / raw)
  To: Randal L. Schwartz; +Cc: Junio C Hamano, git

On Fri, 14 Oct 2005, Randal L. Schwartz wrote:

> My previous message mentioned cogito.git breaking.  This message
> was triggered because git.git itself broke.
> 
> Unfortunately, git.git hasn't been updated since the last time I
> worked around the problem by switching to rsync again, but when it
> does, I'll capture the breakage this time.

Can you give a general description of what happens? I've noticed that I 
sometimes get spurious error messages that don't actually affect the 
download, which I haven't tracked down yet.

	-Daniel
*This .sig left intentionally blank*

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-14 19:56       ` Daniel Barkalow
@ 2005-10-15 13:03         ` Randal L. Schwartz
  2005-10-15 16:04           ` Junio C Hamano
                             ` (3 more replies)
  0 siblings, 4 replies; 17+ messages in thread
From: Randal L. Schwartz @ 2005-10-15 13:03 UTC (permalink / raw)
  To: Daniel Barkalow; +Cc: Junio C Hamano, git

>>>>> "Daniel" == Daniel Barkalow <barkalow@iabervon.org> writes:

Daniel> Can you give a general description of what happens? I've
Daniel> noticed that I sometimes get spurious error messages that
Daniel> don't actually affect the download, which I haven't tracked
Daniel> down yet.

OK, it happened this morning.  While syncing to update from
yesterday's version, I got:

    localhost:~/MIRROR/git-GIT % git-pull
    Fetching refs/heads/master from http://www.kernel.org/pub/scm/git/git.git using http
    Getting alternates list
    got 4546738b58a0134eef154231b07d60fc174d56e3
    walk 4546738b58a0134eef154231b07d60fc174d56e3
    got d402d5566fdf226697a386dfb9858e5d954e9b91
    got 873d8e5652c06c3891278f33546c437efc209c2d
    walk d402d5566fdf226697a386dfb9858e5d954e9b91
    error: 
    Getting pack list
    got 0207ab18a3876249a928e7539d8f594a4f6921f1
    got 9f7534accdf34b980a2de670cb1009dd84ee56c4
    error: Unable to find 5ad4a2766d34569f3a1278544ab64978fab14cc8 under http://www.kernel.org/pub/scm/git/git.git/

    Cannot obtain needed blob 5ad4a2766d34569f3a1278544ab64978fab14cc8
    while processing commit d402d5566fdf226697a386dfb9858e5d954e9b91.

Definitely broken.  But I can "rsync" just fine.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-15 13:03         ` Randal L. Schwartz
@ 2005-10-15 16:04           ` Junio C Hamano
  2005-10-15 16:22             ` Junio C Hamano
  2005-10-15 16:37           ` Junio C Hamano
                             ` (2 subsequent siblings)
  3 siblings, 1 reply; 17+ messages in thread
From: Junio C Hamano @ 2005-10-15 16:04 UTC (permalink / raw)
  To: Daniel Barkalow; +Cc: Randal L. Schwartz, git

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> OK, it happened this morning.  While syncing to update from
> yesterday's version, I got:

Thanks.

>     localhost:~/MIRROR/git-GIT % git-pull
>     Fetching refs/heads/master from http://www.kernel.org/pub/scm/git/git.git using http
>     Getting alternates list
>     got 4546738b58a0134eef154231b07d60fc174d56e3
>     walk 4546738b58a0134eef154231b07d60fc174d56e3
>     got d402d5566fdf226697a386dfb9858e5d954e9b91
>     got 873d8e5652c06c3891278f33546c437efc209c2d
>     walk d402d5566fdf226697a386dfb9858e5d954e9b91
>     error: 
>     Getting pack list
>     got 0207ab18a3876249a928e7539d8f594a4f6921f1

Here is the beginning of a session that succeeded:

    : siamese; GIT_DIR=. git-http-fetch -v -a heads/master \
               http://www.kernel.org/pub/scm/git/git.git/
    Getting alternates list
    got 4546738b58a0134eef154231b07d60fc174d56e3
    walk 4546738b58a0134eef154231b07d60fc174d56e3
    got d402d5566fdf226697a386dfb9858e5d954e9b91
    got 873d8e5652c06c3891278f33546c437efc209c2d
    got 5ad4a2766d34569f3a1278544ab64978fab14cc8
    walk d402d5566fdf226697a386dfb9858e5d954e9b91
    ...

The difference is that this log gets 5ad4a2 blob, before it
starts walking d402d5 commit, while Merlyn's log shows we tried
to walk that commit before getting the blob.  I think what is
happening is:

    - we request 454673 commit, and get it.

    - we start requesting trees, blobs, and parent commit
      reachable from it.  Especially, 5ad4a2 blob and d402d5
      commit are asked.

    - as soon as d402d5 commit arrives we walk and find out we
      need 5ad4a2 blob.  In the case that happened to work, that
      blob has already arrived because it was also part of the
      454673 commit, but in Merlyn's case that blob has not
      arrived yet. "Getting pack list" on the next line is an
      indication that the fetch_object incorrectly decided that
      the object we are waiting for is not available unpacked,
      which does not (and should not) happen in the case we got
      the blob object in time.

I have a suspicion that the recent multi-fetch work has some
interesting interaction with the assumption Sergey's fetch.c
optimization makes.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-15 16:04           ` Junio C Hamano
@ 2005-10-15 16:22             ` Junio C Hamano
  2005-10-15 19:41               ` Daniel Barkalow
  2005-10-15 21:21               ` Nick Hengeveld
  0 siblings, 2 replies; 17+ messages in thread
From: Junio C Hamano @ 2005-10-15 16:22 UTC (permalink / raw)
  To: Nick Hengeveld; +Cc: Randal L. Schwartz, git, Daniel Barkalow

Junio C Hamano <junkio@cox.net> writes:

> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>
>> OK, it happened this morning.  While syncing to update from
>> yesterday's version, I got:
>
> Thanks.
>
>>     localhost:~/MIRROR/git-GIT % git-pull
>>     Fetching refs/heads/master from http://www.kernel.org/pub/scm/git/git.git using http
>>     Getting alternates list
>>     got 4546738b58a0134eef154231b07d60fc174d56e3
>>     walk 4546738b58a0134eef154231b07d60fc174d56e3
>>     got d402d5566fdf226697a386dfb9858e5d954e9b91
>>     got 873d8e5652c06c3891278f33546c437efc209c2d
>>     walk d402d5566fdf226697a386dfb9858e5d954e9b91
>>     error: 
>>     Getting pack list
>>     got 0207ab18a3876249a928e7539d8f594a4f6921f1

More interesting is this "error:" without error message.
"Getting pack list" is a signal that we fell back to
fetch_pack(), so this is coming from fetch_object().

I see this line could emit an empty error message, if errorstr
is empty.

	if (request->curl_result != CURLE_OK && request->http_code != 416) {
		ret = error("%s", request->errorstr);
		release_request(request);
		return ret;
	}

So if that is the case maybe my previous speculation that we
sometimes forget to issue a necessary request was wrong.  We
asked for that object and got an error from cURL library...

BTW, I do not think this is related to git.git repository
problem, but I wonder why we do not do fetch_object() against
each altbase in http-fetch.c::fetch(); nobody said you cannot
borrow unpacked object from your neighbour.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-15 13:03         ` Randal L. Schwartz
  2005-10-15 16:04           ` Junio C Hamano
@ 2005-10-15 16:37           ` Junio C Hamano
  2005-10-15 17:13           ` Junio C Hamano
  2005-10-15 21:57           ` Nick Hengeveld
  3 siblings, 0 replies; 17+ messages in thread
From: Junio C Hamano @ 2005-10-15 16:37 UTC (permalink / raw)
  To: Randal L. Schwartz; +Cc: Daniel Barkalow, git

merlyn@stonehenge.com (Randal L. Schwartz) writes:

>>>>>> "Daniel" == Daniel Barkalow <barkalow@iabervon.org> writes:
>
> Daniel> Can you give a general description of what happens? I've
> Daniel> noticed that I sometimes get spurious error messages that
> Daniel> don't actually affect the download, which I haven't tracked
> Daniel> down yet.
>
> OK, it happened this morning.  While syncing to update from
> yesterday's version, I got:
>
>     localhost:~/MIRROR/git-GIT % git-pull
>     Fetching refs/heads/master from http://www.kernel.org/pub/scm/git/git.git using http
>     Getting alternates list
>     got 4546738b58a0134eef154231b07d60fc174d56e3
>     walk 4546738b58a0134eef154231b07d60fc174d56e3
>     got d402d5566fdf226697a386dfb9858e5d954e9b91
>     got 873d8e5652c06c3891278f33546c437efc209c2d
>     walk d402d5566fdf226697a386dfb9858e5d954e9b91
>     error: 
>     Getting pack list
>     got 0207ab18a3876249a928e7539d8f594a4f6921f1
>     got 9f7534accdf34b980a2de670cb1009dd84ee56c4
>     error: Unable to find 5ad4a2766d34569f3a1278544ab64978fab14cc8 under http://www.kernel.org/pub/scm/git/git.git/
>
>     Cannot obtain needed blob 5ad4a2766d34569f3a1278544ab64978fab14cc8
>     while processing commit d402d5566fdf226697a386dfb9858e5d954e9b91.
>
> Definitely broken.  But I can "rsync" just fine.
>
> -- 
> Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
> <merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
> Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
> See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-15 13:03         ` Randal L. Schwartz
  2005-10-15 16:04           ` Junio C Hamano
  2005-10-15 16:37           ` Junio C Hamano
@ 2005-10-15 17:13           ` Junio C Hamano
  2005-10-15 21:57           ` Nick Hengeveld
  3 siblings, 0 replies; 17+ messages in thread
From: Junio C Hamano @ 2005-10-15 17:13 UTC (permalink / raw)
  To: Randal L. Schwartz; +Cc: Daniel Barkalow, git

merlyn@stonehenge.com (Randal L. Schwartz) writes:

>>>>>> "Daniel" == Daniel Barkalow <barkalow@iabervon.org> writes:
>
> Daniel> Can you give a general description of what happens? I've
> Daniel> noticed that I sometimes get spurious error messages that
> Daniel> don't actually affect the download, which I haven't tracked
> Daniel> down yet.
>
> OK, it happened this morning.  While syncing to update from
> yesterday's version, I got:
>
>     localhost:~/MIRROR/git-GIT % git-pull
>     Fetching refs/heads/master from http://www.kernel.org/pub/scm/git/git.git using http
>     Getting alternates list
>     got 4546738b58a0134eef154231b07d60fc174d56e3
>     walk 4546738b58a0134eef154231b07d60fc174d56e3
>     got d402d5566fdf226697a386dfb9858e5d954e9b91
>     got 873d8e5652c06c3891278f33546c437efc209c2d
>     walk d402d5566fdf226697a386dfb9858e5d954e9b91
>     error: 
>     Getting pack list

Sorry for an empty message.

Could you do the following to establish a reliable way to
reproduce this problem in your setup, so we can diagnose it
better?

First please apply this patch to http-fetch.c and rebuild your
git.

diff --git a/http-fetch.c b/http-fetch.c
index 784aedf..2c1f72c 100644
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -1009,7 +1009,10 @@ static int fetch_object(struct alt_base 
 	}
 
 	if (request->curl_result != CURLE_OK && request->http_code != 416) {
-		ret = error("%s", request->errorstr);
+		ret = error("%s (curl_result = %d, http_code = %ld, sha1 = %s)",
+			    request->errorstr, request->curl_result,
+			    request->http_code,
+			    sha1_to_hex(request->sha1));
 		release_request(request);
 		return ret;
 	}

Then, using that:

$ cd ~/MIRROR
$ rm -fr git-BROKEN
$ git-clone -n http://www.kernel.org/pub/scm/git/git.git git-BROKEN

If you got any error here, then we know we already have a way
to reproduce.  Please give us a log.

If the above succeeds, then please run

$ git-fsck-objects --full

to make sure we got things OK.  This should not report
anything.

Now, I'd like to make this repository as close as I can make it
to "after yesterday's pull", by rewining its top commit back to 
this one:

    commit f7aac2eac96b637755c89c2405a41749c8f6033a
    Author: Linus Torvalds <torvalds@osdl.org>
    Date:   Thu Oct 13 15:38:29 2005 -0700
    Add "-l" flag for repacking only local packs

$ git-reset --hard f7aac2eac96b637755c89c2405a41749c8f6033a
$ git-update-ref refs/heads/origin f7aac2eac96b637755c89c2405a41749c8f6033a

Since proposed updates (pu) branch usually contains everything
that is in the master, we need to get rid of it for this
experiment:

$ rm -f .git/refs/heads/pu

and the we get rid of objects that were added since then:

$ git-prune
$ git-fsck-objects --full

Now we are hopefully as close as "after yesterday's pull" state.

$ git-http-fetch -a -v heads/master http://www.kernel.org/pub/scm/git/git.git/

This will start with something like:

        Getting alternates list
        got 4546738b58a0134eef154231b07d60fc174d56e3
        walk 4546738b58a0134eef154231b07d60fc174d56e3
        got d402d5566fdf226697a386dfb9858e5d954e9b91
        got 873d8e5652c06c3891278f33546c437efc209c2d
        walk d402d5566fdf226697a386dfb9858e5d954e9b91
        ...

if you see the same error, instead of the empty "error:" up
above, we may see a bit more information.

^ permalink raw reply related	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-15 16:22             ` Junio C Hamano
@ 2005-10-15 19:41               ` Daniel Barkalow
  2005-10-15 21:56                 ` Nick Hengeveld
  2005-10-15 21:21               ` Nick Hengeveld
  1 sibling, 1 reply; 17+ messages in thread
From: Daniel Barkalow @ 2005-10-15 19:41 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Nick Hengeveld, Randal L. Schwartz, git

On Sat, 15 Oct 2005, Junio C Hamano wrote:

> Junio C Hamano <junkio@cox.net> writes:
> 
> > merlyn@stonehenge.com (Randal L. Schwartz) writes:
> >
> >> OK, it happened this morning.  While syncing to update from
> >> yesterday's version, I got:
> >
> > Thanks.
> >
> >>     localhost:~/MIRROR/git-GIT % git-pull
> >>     Fetching refs/heads/master from http://www.kernel.org/pub/scm/git/git.git using http
> >>     Getting alternates list
> >>     got 4546738b58a0134eef154231b07d60fc174d56e3
> >>     walk 4546738b58a0134eef154231b07d60fc174d56e3
> >>     got d402d5566fdf226697a386dfb9858e5d954e9b91
> >>     got 873d8e5652c06c3891278f33546c437efc209c2d
> >>     walk d402d5566fdf226697a386dfb9858e5d954e9b91
> >>     error: 
> >>     Getting pack list
> >>     got 0207ab18a3876249a928e7539d8f594a4f6921f1
> 
> More interesting is this "error:" without error message.
> "Getting pack list" is a signal that we fell back to
> fetch_pack(), so this is coming from fetch_object().
> 
> I see this line could emit an empty error message, if errorstr
> is empty.
> 
> 	if (request->curl_result != CURLE_OK && request->http_code != 416) {
> 		ret = error("%s", request->errorstr);
> 		release_request(request);
> 		return ret;
> 	}
> 
> So if that is the case maybe my previous speculation that we
> sometimes forget to issue a necessary request was wrong.  We
> asked for that object and got an error from cURL library...

It looks like we didn't get an error from the cURL library, actually, or 
it would have printed something. My guess is that it is getting to the 
point about while the request is still in progress, but I'm not seeing how 
that could happen.

> BTW, I do not think this is related to git.git repository
> problem, but I wonder why we do not do fetch_object() against
> each altbase in http-fetch.c::fetch(); nobody said you cannot
> borrow unpacked object from your neighbour.

I believe the code is doing that, but elsewhere.

	-Daniel
*This .sig left intentionally blank*

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-15 16:22             ` Junio C Hamano
  2005-10-15 19:41               ` Daniel Barkalow
@ 2005-10-15 21:21               ` Nick Hengeveld
  1 sibling, 0 replies; 17+ messages in thread
From: Nick Hengeveld @ 2005-10-15 21:21 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Randal L. Schwartz, git, Daniel Barkalow

On Sat, Oct 15, 2005 at 09:22:25AM -0700, Junio C Hamano wrote:

> More interesting is this "error:" without error message.
> "Getting pack list" is a signal that we fell back to
> fetch_pack(), so this is coming from fetch_object().

I've seen that happen before fetch_pack used the active queue; open object
requests in the active queue would not be processed while fetch_pack was
transferring the pack, and in some cases a server had timed out a connection
from one of these requests by the time the active queue started processing 
again.  I worked around this at one point by detecting an empty server
response and retrying the request.

Changing all requests to run through the active queue seemed to fix the
problem, but it's possible that something is still holding up processing
long enough to cause a server timeout.

> BTW, I do not think this is related to git.git repository
> problem, but I wonder why we do not do fetch_object() against
> each altbase in http-fetch.c::fetch(); nobody said you cannot
> borrow unpacked object from your neighbour.

It doesn't look like there are any alternates defined in the git.git
repository.  I've seen alternates used during testing when I deliberately
removed objects and packs from the primary repository.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-15 19:41               ` Daniel Barkalow
@ 2005-10-15 21:56                 ` Nick Hengeveld
  0 siblings, 0 replies; 17+ messages in thread
From: Nick Hengeveld @ 2005-10-15 21:56 UTC (permalink / raw)
  To: Daniel Barkalow; +Cc: Junio C Hamano, Randal L. Schwartz, git

On Sat, Oct 15, 2005 at 03:41:35PM -0400, Daniel Barkalow wrote:

> > BTW, I do not think this is related to git.git repository
> > problem, but I wonder why we do not do fetch_object() against
> > each altbase in http-fetch.c::fetch(); nobody said you cannot
> > borrow unpacked object from your neighbour.
> 
> I believe the code is doing that, but elsewhere.

Er, never mind what I said in my previous message.  Daniel is correct,
process_curl_messages() will try the next altbase if a request 404s.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-15 13:03         ` Randal L. Schwartz
                             ` (2 preceding siblings ...)
  2005-10-15 17:13           ` Junio C Hamano
@ 2005-10-15 21:57           ` Nick Hengeveld
  2005-10-15 22:04             ` Randal L. Schwartz
  3 siblings, 1 reply; 17+ messages in thread
From: Nick Hengeveld @ 2005-10-15 21:57 UTC (permalink / raw)
  To: Randal L. Schwartz; +Cc: Daniel Barkalow, Junio C Hamano, git

On Sat, Oct 15, 2005 at 06:03:00AM -0700, Randal L. Schwartz wrote:

> OK, it happened this morning.  While syncing to update from
> yesterday's version, I got:
> ...
> Definitely broken.  But I can "rsync" just fine.

Which version of curl did you build with?

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-15 21:57           ` Nick Hengeveld
@ 2005-10-15 22:04             ` Randal L. Schwartz
  0 siblings, 0 replies; 17+ messages in thread
From: Randal L. Schwartz @ 2005-10-15 22:04 UTC (permalink / raw)
  To: Nick Hengeveld; +Cc: Daniel Barkalow, Junio C Hamano, git

>>>>> "Nick" == Nick Hengeveld <nickh@reactrix.com> writes:

Nick> On Sat, Oct 15, 2005 at 06:03:00AM -0700, Randal L. Schwartz wrote:
>> OK, it happened this morning.  While syncing to update from
>> yesterday's version, I got:
>> ...
>> Definitely broken.  But I can "rsync" just fine.

Nick> Which version of curl did you build with?

curl 7.13.1 (powerpc-apple-darwin8.0) libcurl/7.13.1 OpenSSL/0.9.7g zlib/1.2.3

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: maybe breakage with latest git-pull and http protocol
  2005-10-13  0:53 maybe breakage with latest git-pull and http protocol Randal L. Schwartz
  2005-10-13  5:50 ` Junio C Hamano
  2005-10-14 10:58 ` Randal L. Schwartz
@ 2005-10-20 17:43 ` Nick Hengeveld
  2 siblings, 0 replies; 17+ messages in thread
From: Nick Hengeveld @ 2005-10-20 17:43 UTC (permalink / raw)
  To: Randal L. Schwartz; +Cc: git

Can you try using the following patch?  This cleans up curl handles for
active request slots that are no longer in use.  If all other prefetched
requests finish while a large transfer is happening, the active slots
currently keep open connections to the server and I'm guessing that
perhaps by the time additional objects are prefetched the server has
timed out some of those keepalive connections.


---

 http-fetch.c |   26 ++++++++++++++++++++------
 1 files changed, 20 insertions(+), 6 deletions(-)

applies-to: ce9a5a0fdd52a29e370d849a132b4509c844aca1
04b4353279eaceb7e7c3d73a9565b219aa7a10a9
diff --git a/http-fetch.c b/http-fetch.c
index a7dc2cc..d26fae8 100644
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -291,11 +291,7 @@ static struct active_request_slot *get_a
 	}
 	if (slot == NULL) {
 		newslot = xmalloc(sizeof(*newslot));
-#ifdef NO_CURL_EASY_DUPHANDLE
-		newslot->curl = get_curl_handle();
-#else
-		newslot->curl = curl_easy_duphandle(curl_default);
-#endif
+		newslot->curl = NULL;
 		newslot->in_use = 0;
 		newslot->next = NULL;
 
@@ -311,6 +307,14 @@ static struct active_request_slot *get_a
 		slot = newslot;
 	}
 
+	if (slot->curl == NULL) {
+#ifdef NO_CURL_EASY_DUPHANDLE
+		slot->curl = get_curl_handle();
+#else
+		slot->curl = curl_easy_duphandle(curl_default);
+#endif
+	}
+
 	active_requests++;
 	slot->in_use = 1;
 	slot->done = 0;
@@ -612,6 +616,7 @@ void process_curl_messages(void)
 void process_request_queue(void)
 {
 	struct transfer_request *request = request_queue_head;
+	struct active_request_slot *slot = active_queue_head;
 	int num_transfers;
 
 	while (active_requests < max_requests && request != NULL) {
@@ -624,6 +629,14 @@ void process_request_queue(void)
 		}
 		request = request->next;
 	}
+
+	while (slot != NULL) {
+		if (!slot->in_use && slot->curl != NULL) {
+			curl_easy_cleanup(slot->curl);
+			slot->curl = NULL;
+		}
+		slot = slot->next;
+	}				
 }
 #endif
 
@@ -1297,7 +1310,8 @@ int main(int argc, char **argv)
 #endif
 	slot = active_queue_head;
 	while (slot != NULL) {
-		curl_easy_cleanup(slot->curl);
+		if (slot->curl != NULL)
+			curl_easy_cleanup(slot->curl);
 		slot = slot->next;
 	}
 #ifdef USE_CURL_MULTI
---
0.99.8.GIT

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

^ permalink raw reply related	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2005-10-20 17:43 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2005-10-13  0:53 maybe breakage with latest git-pull and http protocol Randal L. Schwartz
2005-10-13  5:50 ` Junio C Hamano
2005-10-14 10:58 ` Randal L. Schwartz
2005-10-14 15:42   ` Junio C Hamano
2005-10-14 16:18     ` Randal L. Schwartz
2005-10-14 19:56       ` Daniel Barkalow
2005-10-15 13:03         ` Randal L. Schwartz
2005-10-15 16:04           ` Junio C Hamano
2005-10-15 16:22             ` Junio C Hamano
2005-10-15 19:41               ` Daniel Barkalow
2005-10-15 21:56                 ` Nick Hengeveld
2005-10-15 21:21               ` Nick Hengeveld
2005-10-15 16:37           ` Junio C Hamano
2005-10-15 17:13           ` Junio C Hamano
2005-10-15 21:57           ` Nick Hengeveld
2005-10-15 22:04             ` Randal L. Schwartz
2005-10-20 17:43 ` Nick Hengeveld

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).