Building Git with HTTPS support: avoiding libcurl?

Building Git with HTTPS support: avoiding libcurl?

[Paul Smith]

I'm trying to build Git (2.6.4) on GNU/Linux, but without any
requirements (other than basic libc etc.) on the local system.  This
works fine except for one thing: git-remote-https.

In order to build this I need to have libcurl, but libcurl is a MONSTER
library with an enormous number of prerequisites (see below).

Just wondering if anyone has considered an alternative to libcurl; maybe
I'm wrong but it seems to me that HTTPS support for Git would require
only a tiny fraction of the libcurl features and maybe there's an
alternative available which would be more targeted?

I realize this is not a short-term thing in that there won't be an API
compatible library that can just be dropped in.  This is more a forward
-looking question.  For now I'm looking to see if I can rebuild libcurl
myself without most of these dependencies such as Kerberos, LDAP, etc.


$ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4
        linux-vdso.so.1 =>  (0x00007fff37d81000)
        libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f682b921000)
        librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f682b704000)
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f682b49a000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f682b058000)
        libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f682ae0e000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f682abfe000)
        libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f682a9ac000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f682a792000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f682a573000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f682a1a9000)
        libgnutls-deb0.so.28 => /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28 (0x00007f6829e8d000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f6829c59000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f6829a23000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f68297a3000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f682959e000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f68292cc000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f682909d000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f6828e98000)
        libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f6828c8d000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f6828a71000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f6828855000)
        libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f6828615000)
        /lib64/ld-linux-x86-64.so.2 (0x0000559b03259000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f68283b0000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f682819c000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f6827f98000)
        libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f6827d8e000)
        libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f6827b04000)
        libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f6827861000)
        libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f682762d000)
        libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007f6827418000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f6827210000)
        libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007f6826fe6000)
        libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f6826dd7000)
        libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f6826b8c000)
        libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f68268be000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f6826686000)

Re: Building Git with HTTPS support: avoiding libcurl?

[Dave Borowitz]

On Tue, Dec 22, 2015 at 7:39 AM, Paul Smith <paul@mad-scientist.net> wrote:
> I'm trying to build Git (2.6.4) on GNU/Linux, but without any
> requirements (other than basic libc etc.) on the local system.  This
> works fine except for one thing: git-remote-https.
>
> In order to build this I need to have libcurl, but libcurl is a MONSTER
> library with an enormous number of prerequisites (see below).

Well, IIUC one of the reasons for Git's fork-everything strategy is to
avoid having to dynamically link the core git binary against large
libraries like libcurl, so the dependency size has been taken into
account at least in that sense.

> Just wondering if anyone has considered an alternative to libcurl; maybe
> I'm wrong but it seems to me that HTTPS support for Git would require
> only a tiny fraction of the libcurl features and maybe there's an
> alternative available which would be more targeted?
>
> I realize this is not a short-term thing in that there won't be an API
> compatible library that can just be dropped in.  This is more a forward
> -looking question.  For now I'm looking to see if I can rebuild libcurl
> myself without most of these dependencies such as Kerberos, LDAP, etc.
>
>
> $ ldd /usr/lib/x86_64-linux-gnu/libcurl.so.4
>         linux-vdso.so.1 =>  (0x00007fff37d81000)
>         libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f682b921000)
>         librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f682b704000)
>         libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f682b49a000)
>         libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f682b058000)
>         libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f682ae0e000)
>         liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f682abfe000)
>         libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f682a9ac000)
>         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f682a792000)
>         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f682a573000)
>         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f682a1a9000)
>         libgnutls-deb0.so.28 => /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28 (0x00007f6829e8d000)
>         libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f6829c59000)
>         libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f6829a23000)
>         libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f68297a3000)
>         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f682959e000)
>         libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f68292cc000)
>         libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f682909d000)
>         libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f6828e98000)
>         libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f6828c8d000)
>         libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f6828a71000)
>         libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f6828855000)
>         libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f6828615000)
>         /lib64/ld-linux-x86-64.so.2 (0x0000559b03259000)
>         libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f68283b0000)
>         libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f682819c000)
>         libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f6827f98000)
>         libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f6827d8e000)
>         libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f6827b04000)
>         libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f6827861000)
>         libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f682762d000)
>         libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007f6827418000)
>         libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f6827210000)
>         libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007f6826fe6000)
>         libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f6826dd7000)
>         libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f6826b8c000)
>         libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f68268be000)
>         libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f6826686000)

Maybe half of those dependencies are crypto libraries, so even if you
managed to eliminate libcurl, you'd have a hard time supporting HTTPS
without them. Or maybe use something like boringssl instead?

Re: Building Git with HTTPS support: avoiding libcurl?

[Paul Smith]

On Tue, 2015-12-22 at 09:08 -0800, Dave Borowitz wrote:
> Well, IIUC one of the reasons for Git's fork-everything strategy is to
> avoid having to dynamically link the core git binary against large
> libraries like libcurl, so the dependency size has been taken into
> account at least in that sense.

Sure, and it's great that I still get most Git even if I don't have
libcurl.

But without support for cloning https remotes there's a significant hole
in Git functionality...

I grok that Git doesn't want to re-invent the wheel and that libcurl is
convenient.  I just wonder if anyone knows of another wheel, that
doesn't come attached to an entire tractor-trailer, that could be used
instead :).

Re: Building Git with HTTPS support: avoiding libcurl?

[Daniel Stenberg]

On Tue, 22 Dec 2015, Paul Smith wrote:

> I grok that Git doesn't want to re-invent the wheel and that libcurl is 
> convenient.  I just wonder if anyone knows of another wheel, that doesn't 
> come attached to an entire tractor-trailer, that could be used instead :).

But if you would consider another lib, then you could just rebuild your own 
libcurl instead from source, entirely without any dependencies on other libs! 
That would be similar to finding another lib with less dependencies. (As 
already mentioned, you'd still need crypto and TLS support no doubt.)

That huge dependency collection is there much because your distro decided that 
having a libcurl with all that support is preferable. libcurl itself offers 
lots of customizability at build-time so you can strip out most of that if you 
wanted to.

But why do the distros build and provide a libcurl that can do all this?

I think you can look at this from a slightly higher altitude. By re-using a 
very widely used, well developed and properly documented library (yeah, I 
claim it is but you don't need to take my word for it) that is available 
everywhere - git benefits. By having many projects use the same lib, even if 
no two projects use the exact same feature set, we get more reliable software 
in the entire ecosystem - with less work.

I would guess that switching out libcurl in git would be a not insignificant 
amount of work, as no libcurl alternative I'm aware of is even close to this 
API.

-- 

  / daniel.haxx.se

Re: Building Git with HTTPS support: avoiding libcurl?

[Johannes Schindelin]

Hi Daniel,

On Wed, 23 Dec 2015, Daniel Stenberg wrote:

> By re-using a very widely used, well developed and properly documented
> library [libcurl] (yeah, I claim it is but you don't need to take my
> word for it) that is available everywhere - git benefits.

For what it's worth, I fully agree. My perspective: By using a pretty much
fully-configured cURL, Git for Windows has supported the largest number of
use cases with minimal requirements from myself. It has been a boon. If
every library was as easy to use and as high quality, anybody could
maintain Git for Windows ;-)

Ciao,
a very grateful Dscho

Re: Building Git with HTTPS support: avoiding libcurl?

[Thiago Farina]

On Tue, Dec 22, 2015 at 1:39 PM, Paul Smith <paul@mad-scientist.net> wrote:
> I'm trying to build Git (2.6.4) on GNU/Linux, but without any
> requirements (other than basic libc etc.) on the local system.  This
> works fine except for one thing: git-remote-https.
>
> In order to build this I need to have libcurl, but libcurl is a MONSTER
> library with an enormous number of prerequisites (see below).
>
I think Git would have to be changed to use raw sockets and implement
everything it needs on top of that, like libgit2 already does. Certainly this
won't be a trivial task.

-- 
Thiago Farina