Re: [msysGit] Git unable to access https repositories due to curl/OpenSSL 1.0.0 handshake issues

Re: [msysGit] Git unable to access https repositories due to curl/OpenSSL 1.0.0 handshake issues

[Erik Faye-Lund]

(CCing the mainline Git mailing list for insight)

On Tue, Jan 25, 2011 at 6:42 PM, Mika Fischer
<mika.a.fischer@googlemail.com> wrote:
> Hi,
>
> with the latest msysGit (1.7.3.1-preview20101002.exe), I cannot access
> git repositories via https, if they are served by an apache using
> OpenSSL 1.0.0
>
> The error is:
> ----
> error: error:14077458:SSL routines:SSL23_GET_SERVER_HELLO:reason(1112)
> while accessing https://server/repository/info/refs
>
> fatal: HTTP request failed
> ----
>
> An apache using OpenSSL 0.9.8 works fine.
>
> There seem to be some SSL handshake issues, when curl and apache use
> different versions of OpenSSL:
> http://bugs.gentoo.org/332661

This issue is listed as an issue with Gentoo's OpenSSL 1.0.0 builds,
and seems to be have resolved by adding back SSLv2 support.

> http://comments.gmane.org/gmane.comp.web.curl.general/11154

This seems to be an issue with Fedora's OpenSSL 1.0.0 builds.

> Any idea how to fix this? For instance, is it possible to configure
> parameters that are passed to curl (passing -3 would fix it)? I could
> also change the apache configuration if someone knows how to work
> around this issue, although I already tried playing around with the
> SSLProtocol option of Apache to no avail...
>

Git for Windows currently use OpenSSL 0.9.8k, so this sounds to me
like an issue in your server-end. Your server seems to simply be
incompatible with OpenSSL 0.9.8-clients, which is the vast majority of
SSL-clients out there.

AFAICT, Git does not run curl, but use libcurl instead. It doesn't set
CURLOPT_SSLVERSION, and Git for Windows use libcurl 7.21.1 where
either SSLv3 or TLSv1 seems to be the default. So I don't know if
there's anything we can do about this on the Git side. You could try
to set CURLOPT_SSLVERSION to work around the issue, but I don't think
this is something we'd want to do in a Git for Windows release.


But:
This issue seems like it might be related to this, which is a big issue:
http://www.openssl.org/news/secadv_20101202.txt

In other words: We should probably upgrade OpenSSL. If not, Git for
Windows will most likely be a security hole.

But we must do so while making sure we are compatible with 0.9.8. It
seems to me like either 0.9.8q or 1.0.0c and beyond fixes the
security-hole. If 1.0.x breaks 0.9.8 support (It's unclear to me if it
does or not - some sources say it does, some say it doesn't), perhaps
0.9.8q is the safest route? It's probably also the version that needs
the least amount of patching to run, as it's closer to what we're
already building. This probably means we COULD have it ready for Git
for Windows 1.7.4.

> Any help would be very much appreciated, as this makes git totally
> useless under Windows for us, as all our repositories are accessed
> through https...

As I said: I think this is a problem with your server, not Git for
Windows. But I'm far from an expert on the subject, so I could be
mistaken.

Re: [msysGit] Git unable to access https repositories due to curl/OpenSSL 1.0.0 handshake issues

[Mika Fischer]

Hi Erik,

many thanks for your comments!

On Tue, Jan 25, 2011 at 20:05, Erik Faye-Lund <kusmabite@gmail.com> wrote:
>> with the latest msysGit (1.7.3.1-preview20101002.exe), I cannot access
>> git repositories via https, if they are served by an apache using
>> OpenSSL 1.0.0
>>
>> The error is:
>> ----
>> error: error:14077458:SSL routines:SSL23_GET_SERVER_HELLO:reason(1112)
>> while accessing https://server/repository/info/refs
>>
>> fatal: HTTP request failed
>> ----
>>
>> An apache using OpenSSL 0.9.8 works fine.
>>
>> There seem to be some SSL handshake issues, when curl and apache use
>> different versions of OpenSSL:
>> http://bugs.gentoo.org/332661
>
> This issue is listed as an issue with Gentoo's OpenSSL 1.0.0 builds,
> and seems to be have resolved by adding back SSLv2 support.
>
>> http://comments.gmane.org/gmane.comp.web.curl.general/11154
>
> This seems to be an issue with Fedora's OpenSSL 1.0.0 builds.

I found something else, which seems to be more related to my exact issue:
http://www.mentby.com/Group/curl/curl-outputs-ssl23getserverhello.html

I've also tried doing "openssl s_client -connect <server>:443" with
the openssl command line tool that's included in the current msysgit,
and that works just fine.

> Git for Windows currently use OpenSSL 0.9.8k, so this sounds to me
> like an issue in your server-end. Your server seems to simply be
> incompatible with OpenSSL 0.9.8-clients, which is the vast majority of
> SSL-clients out there.

I find it hard to believe that something like this would have gone
unnoticed for such a long time. Most Linux distributions nowadays come
with OpenSSL 1.0.0 and that would mean that any webserver set up on
them could not communicate with any client using OpenSSL 0.9.8. Since
the openssl command line tool works fine, I think it must be something
much more specific and it must have something to do with curl.

I'll see if I can get a more current version of curl to compile under Windows.

I also wanted to try compiling a more current version of OpenSSL but
the build hangs at "Generating DLL definition files". Any idea what
might be the problem?

> AFAICT, Git does not run curl, but use libcurl instead. It doesn't set
> CURLOPT_SSLVERSION, and Git for Windows use libcurl 7.21.1 where
> either SSLv3 or TLSv1 seems to be the default. So I don't know if
> there's anything we can do about this on the Git side. You could try
> to set CURLOPT_SSLVERSION to work around the issue, but I don't think
> this is something we'd want to do in a Git for Windows release.

This environment variable has no effect for me, neither with git nor
with the command line client.

Best,
 Mika

Re: [msysGit] Git unable to access https repositories due to curl/OpenSSL 1.0.0 handshake issues

[Mika Fischer]

On Wed, Jan 26, 2011 at 10:15, Mika Fischer
<mika.a.fischer@googlemail.com> wrote:
> I'll see if I can get a more current version of curl to compile under Windows.

I did not manage to do do this.

But I noticed that the same problem occurs on OpenSuSE 11.2. So I
compiled the latest version of curl on OpenSuSE 11.2 and the problem
is still present...

Best,
 Mika

Re: [msysGit] Git unable to access https repositories due to curl/OpenSSL 1.0.0 handshake issues

[Daniel Stenberg]

On Wed, 26 Jan 2011, Mika Fischer wrote:

>> I'll see if I can get a more current version of curl to compile under 
>> Windows.
>
> I did not manage to do do this.
>
> But I noticed that the same problem occurs on OpenSuSE 11.2. So I compiled 
> the latest version of curl on OpenSuSE 11.2 and the problem is still 
> present...

Are these clean builds of libcurl and OpenSSL from source?

I'll try to look into this.

-- 

  / daniel.haxx.se

Re: [msysGit] Git unable to access https repositories due to curl/OpenSSL 1.0.0 handshake issues

[Mika Fischer]

On Wed, Jan 26, 2011 at 10:57, Daniel Stenberg <daniel@haxx.se> wrote:
> Are these clean builds of libcurl and OpenSSL from source?

I just compiled libcurl against the OpenSSL library of OpenSuSE 11.2 (0.9.8k).

If you want me to test other combinations/versions etc. just let me know.

> I'll try to look into this.

Thanks a lot!

Best,
 Mika

Re: [msysGit] Git unable to access https repositories due to curl/OpenSSL 1.0.0 handshake issues

[Daniel Stenberg]

On Wed, 26 Jan 2011, Mika Fischer wrote:

> But I noticed that the same problem occurs on OpenSuSE 11.2. So I compiled 
> the latest version of curl on OpenSuSE 11.2 and the problem is still 
> present...

Sorry, but I can't repeat this problem. I tried this:

1. I got OpenSSL 1.0.0c, built it and installed it.

2. I started OpenSSL's little HTTPS server with this version:

$ openssl s_server -HTTP -cert [PEM]

(I verified, and this binary uses the lib statically so there's no funny lib 
path issues here.)

3. I tried getting a file through this using curl 7.21.2 and my current dev 
version of 7.21.4-DEV (see below for full details). Both managed to get it 
fine.

>From this, I conclude that there's some additional factors necessary to make 
this problem trigger. Can you figure out what?

The full curl -V outputs from my two test versions:

curl 7.21.2 (i486-pc-linux-gnu) libcurl/7.21.2 OpenSSL/0.9.8o zlib/1.2.3.4 
libidn/1.18 libssh2/1.2.6
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 
pop3s rtsp scp sftp smtp smtps telnet tftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

curl 7.21.4-DEV (i686-pc-linux-gnu) libcurl/7.21.4-DEV OpenSSL/1.0.0c 
zlib/1.2.3.4 c-ares/1.7.5-DEV libidn/1.18 libssh2/1.2.8_DEV librtmp/2.2e
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 
pop3s rtmp rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS Debug TrackMemory GSS-Negotiate IDN IPv6 Largefile NTLM 
SSL libz

-- 

  / daniel.haxx.se

Re: Git unable to access https repositories due to curl/OpenSSL 1.0.0 handshake issues

[Erik Faye-Lund]

On Wed, Jan 26, 2011 at 2:29 PM, Daniel Stenberg <daniel@haxx.se> wrote:
> On Wed, 26 Jan 2011, Mika Fischer wrote:
>
>> But I noticed that the same problem occurs on OpenSuSE 11.2. So I compiled
>> the latest version of curl on OpenSuSE 11.2 and the problem is still
>> present...
>
> Sorry, but I can't repeat this problem. I tried this:
>
> 1. I got OpenSSL 1.0.0c, built it and installed it.
>
> 2. I started OpenSSL's little HTTPS server with this version:
>
> $ openssl s_server -HTTP -cert [PEM]
>
> (I verified, and this binary uses the lib statically so there's no funny lib
> path issues here.)
>
> 3. I tried getting a file through this using curl 7.21.2 and my current dev
> version of 7.21.4-DEV (see below for full details). Both managed to get it
> fine.
>
> From this, I conclude that there's some additional factors necessary to make
> this problem trigger. Can you figure out what?
>
> The full curl -V outputs from my two test versions:
>
> curl 7.21.2 (i486-pc-linux-gnu) libcurl/7.21.2 OpenSSL/0.9.8o zlib/1.2.3.4
> libidn/1.18 libssh2/1.2.6
> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3
> pop3s rtsp scp sftp smtp smtps telnet tftp
> Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
>
> curl 7.21.4-DEV (i686-pc-linux-gnu) libcurl/7.21.4-DEV OpenSSL/1.0.0c
> zlib/1.2.3.4 c-ares/1.7.5-DEV libidn/1.18 libssh2/1.2.8_DEV librtmp/2.2e
> Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3
> pop3s rtmp rtsp scp sftp smtp smtps telnet tftp
> Features: AsynchDNS Debug TrackMemory GSS-Negotiate IDN IPv6 Largefile NTLM
> SSL libz
>

Here's the output from Git for Windows' curl -V:

$ curl -V
curl 7.21.1 (i686-pc-mingw32) libcurl/7.21.1 OpenSSL/0.9.8k zlib/1.2.3
Protocols: dict file ftp ftps http https imap imaps ldap ldaps pop3
pop3s rtsp smtp smtps telnet tftp
Features: Largefile NTLM SSL SSPI libz

Re: [msysGit] Git unable to access https repositories due to curl/OpenSSL 1.0.0 handshake issues

[Mika Fischer]

On Wed, Jan 26, 2011 at 14:29, Daniel Stenberg <daniel@haxx.se> wrote:
> 3. I tried getting a file through this using curl 7.21.2 and my current dev
> version of 7.21.4-DEV (see below for full details). Both managed to get it
> fine.

Same here. curl can fetch fine from the openssl mini-httpd

> From this, I conclude that there's some additional factors necessary to make
> this problem trigger. Can you figure out what?

So to recap, we have the following situation:

curl/openssl 0.9.8k <-> apache/openssl 0.9.8k -> works
curl/openssl 1.0.0 <-> apache/openssl 0.9.8k -> works
curl/openssl 0.9.8k <-> apache/openssl 1.0.0 -> error
openssl 0.9.8k <-> apache/openssl 1.0.0 -> works
curl/openssl 0.9.8k <-> openssl 1.0.0 -> works
openssl 0.9.8k <-> openssl 1.0.0 -> works

I'm not sure what to take away from this. Maybe it's a problem that is
partly caused by both apache and curl?

Do you know how I could debug this? I won't have much time during the
day but I can test things in the evenings...

Just FYI: To get something running, I downgraded our apache server to
the OpenSuSE 11.2 packages. Now it works again fine, with the same
configuration as before...

The failing curl -V on OpenSuSE 11.2
curl 7.19.6 (x86_64-unknown-linux-gnu) libcurl/7.19.6 OpenSSL/0.9.8k
zlib/1.2.3 libidn/1.10
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

curl -V on the problematic apache host running OpenSuSE 11.3
curl 7.20.1 (x86_64-unknown-linux-gnu) libcurl/7.20.1 OpenSSL/1.0.0
zlib/1.2.3 libidn/1.15 libssh2/1.2.2_DEV
Protocols: dict file ftp ftps http https imap imaps ldap pop3 pop3s
rtsp scp sftp smtp smtps telnet tftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

Best,
 Mika

Re: [msysGit] Git unable to access https repositories due to curl/OpenSSL 1.0.0 handshake issues

[Daniel Stenberg]

On Wed, 26 Jan 2011, Mika Fischer wrote:

> curl/openssl 0.9.8k <-> apache/openssl 1.0.0 -> error

> I'm not sure what to take away from this. Maybe it's a problem that is 
> partly caused by both apache and curl?

Could be. I just have a hard time to see why code in curl that has worked for 
so many years suddenly would stop working. It just so feels like else changed.

This error (or something similar to it) is often seen when we try to talk 
plain HTTP to a HTTPS server or vice versa. Did you verify that HTTPS was 
working fine on that port when you ran the OpenSSL 1.0.0 version of the 
server?

Perhaps it is possible to add verbose level and further debug log stuff in the 
server to see what makes it suddenly decide the handshake is bad.

-- 

  / daniel.haxx.se

Re: [msysGit] Git unable to access https repositories due to curl/OpenSSL 1.0.0 handshake issues

[Mika Fischer]

On Wed, Jan 26, 2011 at 23:00, Daniel Stenberg <daniel@haxx.se> wrote:
>> curl/openssl 0.9.8k <-> apache/openssl 1.0.0 -> error
>
>> I'm not sure what to take away from this. Maybe it's a problem that is
>> partly caused by both apache and curl?
>
> Could be. I just have a hard time to see why code in curl that has worked
> for so many years suddenly would stop working. It just so feels like else
> changed.

Well, it's definitely a possibility that neither Apache nor Curl are
at fault but OpenSSL. The strange thing is that I could not produce
any failure with the OpenSSL command line tool. But that doesn't mean
that openssl is completely innocent.

> This error (or something similar to it) is often seen when we try to talk
> plain HTTP to a HTTPS server or vice versa. Did you verify that HTTPS was
> working fine on that port when you ran the OpenSSL 1.0.0 version of the
> server?

No, it definitely worked before with curl/openssl 1.0.0 and all kinds
of browsers. Also, as I wrote, it worked with the OpenSSL 0.9.8k
command line tool.

> Perhaps it is possible to add verbose level and further debug log stuff in
> the server to see what makes it suddenly decide the handshake is bad.

I didn't find much in this direction in the apache docs, unfortunately...
http://httpd.apache.org/docs/2.2/mod/mod_ssl.html

Next thing I'll try is to check whether I can reproduce this using a
different distribution that also uses OpenSSL 1.0.0. I'll let you know
when I have the results...

Best,
 Mika