Is there a debug mode for git-clone https traffic?

Is there a debug mode for git-clone https traffic?

[Ævar Arnfjörð Bjarmason]

I''m trying to debug a problem where over a https proxy I have
"warning: remote HEAD refers to nonexistent ref, unable to checkout."
when doing git-clone.

I suspect that it might be an overzealous security scanner sitting in
the middle.

Is there some debugging mode for git-clone where it'll dump the
traffic being sent over the wire that I could use to confirm this?

Re: Is there a debug mode for git-clone https traffic?

[Richard Peterson]

On May 17, 2011, at 19:08, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

> I''m trying to debug a problem where over a https proxy I have
> "warning: remote HEAD refers to nonexistent ref, unable to checkout."
> when doing git-clone.

Last time I ran into that error, I was cloning from a bundle where the
remote head was simply not in the bundle. The branches I needed were
all there. Running "git branch -a" and then checking out one of the
existent branches worked fine.

Not sure if that's related to your situation at all.

Richard

Re: Is there a debug mode for git-clone https traffic?

[Tay Ray Chuan]

On Wed, May 18, 2011 at 7:08 AM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> I''m trying to debug a problem where over a https proxy I have
> "warning: remote HEAD refers to nonexistent ref, unable to checkout."
> when doing git-clone.
>
> I suspect that it might be an overzealous security scanner sitting in
> the middle.
>
> Is there some debugging mode for git-clone where it'll dump the
> traffic being sent over the wire that I could use to confirm this?

Run the process with GIT_CURL_VERBOSE=1. There's also another option
to debug communication between transport helpers but I forget what it
is.

--
Cheers,
Ray Chuan

Re: Is there a debug mode for git-clone https traffic?

[Junio C Hamano]

Tay Ray Chuan <rctay89@gmail.com> writes:

> On Wed, May 18, 2011 at 7:08 AM, Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>>
>> I''m trying to debug a problem where over a https proxy I have
>> "warning: remote HEAD refers to nonexistent ref, unable to checkout."
>> when doing git-clone.
>>
>> I suspect that it might be an overzealous security scanner sitting in
>> the middle.
>>
>> Is there some debugging mode for git-clone where it'll dump the
>> traffic being sent over the wire that I could use to confirm this?
>
> Run the process with GIT_CURL_VERBOSE=1. There's also another option
> to debug communication between transport helpers but I forget what it
> is.

GIT_TRACE_PACKET

Re: Is there a debug mode for git-clone https traffic?

[Jeff King]

On Wed, May 18, 2011 at 01:08:06AM +0200, Ævar Arnfjörð Bjarmason wrote:

> I''m trying to debug a problem where over a https proxy I have
> "warning: remote HEAD refers to nonexistent ref, unable to checkout."
> when doing git-clone.
> 
> I suspect that it might be an overzealous security scanner sitting in
> the middle.
> 
> Is there some debugging mode for git-clone where it'll dump the
> traffic being sent over the wire that I could use to confirm this?

Others have mentioned ways to dump data from clone. But you can more or
less see what's going on with "git ls-remote <remote>". The sha1 for
"HEAD" will try to match with the sha1 for some other ref. If it
doesn't, then we give up.

-Peff

Re: Is there a debug mode for git-clone https traffic?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Others have mentioned ways to dump data from clone. But you can more or
> less see what's going on with "git ls-remote <remote>". The sha1 for
> "HEAD" will try to match with the sha1 for some other ref. If it
> doesn't, then we give up.

One thing noticed by somebody at $work is that we perform poorly when
fetching from or pushing to a remote, which has an alternate, and when
that alternate has a dangling HEAD that does not point at a valid commit.

I don't recall the exact details, but the story goes like...

The remote in the middle would want to mark the objects that are available
from the alternate by internally running ls-remote against its alternate,
which returns with a failure status, and then the entire serving process
is aborted, and the original client does not get what it wanted from the
remote.

... or something like that.

As the internal discovery of what additional objects are available to the
remote from its alternate is primarily for improving network performance
by avoiding to transfer unnecessary objects, it is tempting to argue that
the remote could ignore the failure in ls-remote against the alternate.

The remote however _is_ relying on the alternate for some (and possibly
large part of) objects it _thinks_ it has via its own refs, and an
alternate that fails ls-remote because it has its refs messed up is a bad
sign that does not give us confidence that the remote itself is healthy,
so from that point of view, we might be doing the right thing by refusing
to serve the original requestor to prevent the damage from spreading.

Re: Is there a debug mode for git-clone https traffic?

[Ævar Arnfjörð Bjarmason]

On Wed, May 18, 2011 at 01:08, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> I''m trying to debug a problem where over a https proxy I have
> "warning: remote HEAD refers to nonexistent ref, unable to checkout."
> when doing git-clone.
>
> I suspect that it might be an overzealous security scanner sitting in
> the middle.
>
> Is there some debugging mode for git-clone where it'll dump the
> traffic being sent over the wire that I could use to confirm this?

I've debugged this, and the results are strange. Here it is on a
machine that doesn't work (proxy hostname manually munged):

    Cloning into magit...
    * Couldn't find host github.com in the .netrc file; using defaults
    * About to connect() to proxy proxy-BROKEN.example.net port 3128 (#0)
    *   Trying 10.146.207.60... * Connected to
proxy-BROKEN.example.net (10.146.207.60) port 3128 (#0)
    * Establish HTTP proxy tunnel to github.com:443
    > CONNECT github.com:443 HTTP/1.1
    Host: github.com:443
    User-Agent: git/1.7.4.4
    Proxy-Connection: Keep-Alive
    Pragma: no-cache

    < HTTP/1.0 200 Connection established
    <
    * Proxy replied OK to CONNECT request
    * found 158 certificates in /etc/ssl/certs/ca-certificates.crt
    > GET /magit/magit.git/info/refs?service=git-upload-pack HTTP/1.1
    User-Agent: git/1.7.4.4
    Host: github.com
    Accept: */*
    Pragma: no-cache

    * Connection #0 to host proxy-BROKEN.example.net left intact
    * Couldn't find host github.com in the .netrc file; using defaults
    * Connection #0 seems to be dead!
    * Closing connection #0
    * About to connect() to proxy proxy-BROKEN.example.net port 3128 (#0)
    *   Trying 10.146.207.60... * Connected to
proxy-BROKEN.example.net (10.146.207.60) port 3128 (#0)
    * Establish HTTP proxy tunnel to github.com:443
    > CONNECT github.com:443 HTTP/1.1
    Host: github.com:443
    User-Agent: git/1.7.4.4
    Proxy-Connection: Keep-Alive
    Pragma: no-cache

    < HTTP/1.0 200 Connection established
    <
    * Proxy replied OK to CONNECT request
    * found 158 certificates in /etc/ssl/certs/ca-certificates.crt
    > GET /magit/magit.git/HEAD HTTP/1.1
    User-Agent: git/1.7.4.4
    Host: github.com
    Accept: */*
    Pragma: no-cache

    * Connection #0 to host proxy-BROKEN.example.net left intact
    warning: remote HEAD refers to nonexistent ref, unable to checkout.

And here on a box with a different proxy where this does work:

    Cloning into magit...
    * Couldn't find host github.com in the .netrc file, using defaults
    * About to connect() to proxy proxy-OK.example.net port 3128
    *   Trying 10.147.82.1... * connected
    * Connected to proxy-OK.example.net (10.147.82.1) port 3128
    * Establish HTTP proxy tunnel to github.com:443
    > CONNECT github.com:443 HTTP/1.0
    Host: github.com:443
    User-Agent: git/1.7.2.1
    Proxy-Connection: Keep-Alive
    Pragma: no-cache

    < HTTP/1.0 200 Connection established
    <
    * Proxy replied OK to CONNECT request
    * successfully set certificate verify locations:
    *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
      CApath: none
    * SSL connection using AES256-SHA
    * Server certificate:
    *        subject: /O=*.github.com/OU=Domain Control
Validated/CN=*.github.com
    *        start date: 2009-12-11 05:02:36 GMT
    *        expire date: 2014-12-11 05:02:36 GMT
    *        subjectAltName: github.com matched
    *        issuer: /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com,
Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure
Certification Authority/serialNumber=07969287
    * SSL certificate verify ok.
    > GET /magit/magit.git/info/refs?service=git-upload-pack HTTP/1.1
    User-Agent: git/1.7.2.1
    Host: github.com
    Accept: */*
    Pragma: no-cache

    < HTTP/1.1 200 OK
    < Server: nginx/0.7.67
    < Date: Mon, 23 May 2011 08:14:22 GMT
    < Content-Type: application/x-git-upload-pack-advertisement
    < Transfer-Encoding: chunked
    < Connection: keep-alive
    < Expires: Fri, 01 Jan 1980 00:00:00 GMT
    < Pragma: no-cache
    < Cache-Control: no-cache, max-age=0, must-revalidate
    * Connection #0 to host proxy-OK.example.net left intact
    * Couldn't find host github.com in the .netrc file, using defaults
    * Re-using existing connection! (#0) with host proxy-OK.example.net
    * Connected to proxy-OK.example.net (10.147.82.1) port 3128
    > POST /magit/magit.git/git-upload-pack HTTP/1.1
    User-Agent: git/1.7.2.1
    Host: github.com
    Accept: */*
    Accept-Encoding: deflate, gzip
    Content-Type: application/x-git-upload-pack-request
    Accept: application/x-git-upload-pack-result
    Content-Length: 828

    0073want 761d6ad09bfce0d354c8fe958f5fc1b0fdde0a9a
multi_ack_detailed side-band-64k thin-pack no-progress ofs-delta
    0032want 292c83d70df3735ba3809dd946d99cf7aa49f6bd
    0032want 1984ce646f25ec073afe77f9d90c11f422659c60
    0032want fd6f6d5ea7b209f8968dfcd999a20f84e6e63edc
    0032want b60cb2bdb31d78670690e22a7fa9eb1a5fb6a40c
    0032want eb0da0582e6635e8b89636170a0de847a71d975e
    0032want 387969a8734377e4baabaf2533dfad8f427dce5f
    0032want cff8d04ef1fc0cce10a274e3737fbbe3ae9be43c
    0032want 8be54f67577478ce05cf68a44a377cab893e9ea0
    0032want 7b3473865309c8a6bc41d0e674ee648e124bf106
    0032want 01aa8d5874b4ad19ffd0423e2c0e0123c5393051
    0032want 00579d3e79504ce0be173b9cd0aec4100cbe8a76
    0032want be85320fabef7af16fb1a27fa74908f54a1f2403
    0032want b4b0a8bffc393137d3a52d62bc92c762a77067e9
    0032want e4d766306adbda64a974b5ea1daa9b7fffdc29d6
    00000009done
    < HTTP/1.1 200 OK
    < Server: nginx/0.7.67
    < Date: Mon, 23 May 2011 08:14:22 GMT
    < Content-Type: application/x-git-upload-pack-result
    < Transfer-Encoding: chunked
    < Connection: keep-alive
    < Expires: Fri, 01 Jan 1980 00:00:00 GMT
    < Pragma: no-cache
    < Cache-Control: no-cache, max-age=0, must-revalidate
    * Connection #0 to host proxy-OK.example.net left intact

Which is curious, because I can get that file manually with curl on
both of those boxes, i.e. the BROKEN and OK one, respectively:

    $ curl -v https://github.com/magit/magit.git/info/refs?service=git-upload-pack
| sha1sum
    * About to connect() to proxy proxy-BROKEN.example.net port 3128 (#0)
    *   Trying 10.146.207.60...   % Total    % Received % Xferd
Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent
Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:--
--:--:--     0connected
    * Connected to proxy-BROKEN.example.net (10.146.207.60) port 3128 (#0)
    * Establish HTTP proxy tunnel to github.com:443
    > CONNECT github.com:443 HTTP/1.1
    > Host: github.com:443
    > User-Agent: curl/7.21.6 (i486-pc-linux-gnu) libcurl/7.21.6
OpenSSL/1.0.0d zlib/1.2.3.4 libidn/1.20 libssh2/1.2.8 librtmp/2.3
    > Proxy-Connection: Keep-Alive
    >
    < HTTP/1.0 200 Connection established
    <
    * Proxy replied OK to CONNECT request
    * successfully set certificate verify locations:
    *   CAfile: none
      CApath: /etc/ssl/certs
    * SSLv3, TLS handshake, Client hello (1):
    } [data not shown]
    * SSLv3, TLS handshake, Server hello (2):
    { [data not shown]
    * SSLv3, TLS handshake, CERT (11):
    { [data not shown]
    * SSLv3, TLS handshake, Server finished (14):
    { [data not shown]
    * SSLv3, TLS handshake, Client key exchange (16):
    } [data not shown]
    * SSLv3, TLS change cipher, Client hello (1):
    } [data not shown]
    * SSLv3, TLS handshake, Finished (20):
    } [data not shown]
    * SSLv3, TLS change cipher, Client hello (1):
    { [data not shown]
    * SSLv3, TLS handshake, Finished (20):
    { [data not shown]
    * SSL connection using AES256-SHA
    * Server certificate:
    *        subject: O=*.github.com; OU=Domain Control Validated;
CN=*.github.com
    *        start date: 2009-12-11 05:02:36 GMT
    *        expire date: 2014-12-11 05:02:36 GMT
    *        subjectAltName: github.com matched
    *        issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com,
Inc.; OU=http://certificates.godaddy.com/repository; CN=Go Daddy
Secure Certification Authority; serialNumber=07969287
    *        SSL certificate verify ok.
    > GET /magit/magit.git/info/refs?service=git-upload-pack HTTP/1.1
    > User-Agent: curl/7.21.6 (i486-pc-linux-gnu) libcurl/7.21.6
OpenSSL/1.0.0d zlib/1.2.3.4 libidn/1.20 libssh2/1.2.8 librtmp/2.3
    > Host: github.com
    > Accept: */*
    >
      0     0    0     0    0     0      0      0 --:--:-- --:--:--
--:--:--     0< HTTP/1.1 200 OK
    < Server: nginx/0.7.67
    < Date: Mon, 23 May 2011 08:19:19 GMT
    < Content-Type: application/x-git-upload-pack-advertisement
    < Transfer-Encoding: chunked
    < Connection: keep-alive
    < Expires: Fri, 01 Jan 1980 00:00:00 GMT
    < Pragma: no-cache
    < Cache-Control: no-cache, max-age=0, must-revalidate
    <
    { [data not shown]
    100  1531    0  1531    0     0   2773      0 --:--:-- --:--:--
--:--:--  2877
    * Connection #0 to host proxy-BROKEN.example.net left intact
    * Closing connection #0
    * SSLv3, TLS alert, Client hello (1):
    } [data not shown]
    5b7eb0b5c25a8700bfc8376a5a38da78724dc1dd  -

    $ curl -v https://github.com/magit/magit.git/info/refs?service=git-upload-pack
| sha1sum
    * About to connect() to proxy proxy-OK.example.net port 3128
    *   Trying 10.147.82.5... connected
    * Connected to proxy-OK.example.net (10.147.82.5) port 3128
    * Establish HTTP proxy tunnel to github.com:443
    > CONNECT github.com:443 HTTP/1.0
    > Host: github.com:443
    > User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
    > Proxy-Connection: Keep-Alive
    >
    < HTTP/1.0 200 Connection established
    <
    * Proxy replied OK to CONNECT request
    * successfully set certificate verify locations:
    *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
      CApath: none
    * SSLv2, Client hello (1):
    SSLv3, TLS handshake, Server hello (2):
    SSLv3, TLS handshake, CERT (11):
    SSLv3, TLS handshake, Server finished (14):
    SSLv3, TLS handshake, Client key exchange (16):
    SSLv3, TLS change cipher, Client hello (1):
    SSLv3, TLS handshake, Finished (20):
    SSLv3, TLS change cipher, Client hello (1):
    SSLv3, TLS handshake, Finished (20):
    SSL connection using AES256-SHA
    * Server certificate:
    *        subject: /O=*.github.com/OU=Domain Control
Validated/CN=*.github.com
    *        start date: 2009-12-11 05:02:36 GMT
    *        expire date: 2014-12-11 05:02:36 GMT
    *        subjectAltName: github.com matched
    *        issuer: /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com,
Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure
Certification Authority/serialNumber=07969287
    * SSL certificate verify ok.
    > GET /magit/magit.git/info/refs?service=git-upload-pack HTTP/1.1
    > User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
    > Host: github.com
    > Accept: */*
    >
    < HTTP/1.1 200 OK
    < Server: nginx/0.7.67
    < Date: Mon, 23 May 2011 08:18:47 GMT
    < Content-Type: application/x-git-upload-pack-advertisement
    < Transfer-Encoding: chunked
    < Connection: keep-alive
    < Expires: Fri, 01 Jan 1980 00:00:00 GMT
    < Pragma: no-cache
    < Cache-Control: no-cache, max-age=0, must-revalidate
      % Total    % Received % Xferd  Average Speed   Time    Time
Time  Current
                                     Dload  Upload   Total   Spent
Left  Speed
    100  1531    0  1531    0     0   3173      0 --:--:-- --:--:--
--:--:--     0* Connection #0 to host proxy-OK.example.net left intact

    * Closing connection #0
    * SSLv3, TLS alert, Client hello (1):
    5b7eb0b5c25a8700bfc8376a5a38da78724dc1dd  -

The SSL part seems different though, but since curl(1) can get the
content perhaps there's some use of the curl library in Git that's
tripping me up?