Google Code: Support for Mercurial and Analysis of Git and Mercurial

Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Christian Couder]

Hi,

For information, now Google Code supports Mercurial for project hosting:

http://google-code-updates.blogspot.com/2009/04/mercurial-support-for-project-hosting.html

Mercurial was choosen over Git because of this (one year old) analysis:

http://code.google.com/p/support/wiki/DVCSAnalysis

There is this article on LWN about the analysis:

http://lwn.net/Articles/330138/

Regards,
Christian.

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Michael Witten]

On Sun, Apr 26, 2009 at 00:03, Christian Couder <chriscool@tuxfamily.org> wrote:
> Hi,
>
> For information, now Google Code supports Mercurial [and not git] for project hosting:

>From an 'advocacy' viewpoint, I think this is a major blow to git.

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Jakub Narebski]

Christian Couder <chriscool@tuxfamily.org> writes:

> For information, now Google Code supports Mercurial for project hosting:
> 
> http://google-code-updates.blogspot.com/2009/04/mercurial-support-for-project-hosting.html
> 
> Mercurial was choosen over Git because of this (one year old) analysis:
> 
> http://code.google.com/p/support/wiki/DVCSAnalysis
> 
> There is this article on LWN about the analysis:
> 
> http://lwn.net/Articles/330138/

It is a pity that the choice was based on year old analysis.  One year
for actively developed and fast moving targets such like Git and
Mercurial is ages in terms of development history.  But I guess this
is unavoidable.

For example periodic "maintenance" (garbage collecting) is nowadays
quite automatic in git, with fetching into pack, periodic repacking if
number of loose objects is above tthreshold, and "git gc --auto".

Whether Mercurial or Git has better UI and better documentation is
IMHO a matter of debate.  Git documentation is much better that it
was, with "Git User's Manual" and "Git Community Book"; UI also is
being improved.

I can't comment on MS Windows support, but AFAIK Mercurial has better
support here than Git.


The deciding feature (well, one of deciding features) was the fact
that Mercurial has better HTTP support... I guess (it was not obvious
from the analysis, but it was hinted at) that Mercurial uses its
custom protocol over HTTP, as opposed to "dumb" HTTP protocol support
in Git.

Perhaps it is time to restart work on _"smart" HTTP protocol_?

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Paolo Ciarrocchi]

On 4/26/09, Jakub Narebski <jnareb@gmail.com> wrote:

> The deciding feature (well, one of deciding features) was the fact
> that Mercurial has better HTTP support... I guess (it was not obvious
> from the analysis, but it was hinted at) that Mercurial uses its
> custom protocol over HTTP, as opposed to "dumb" HTTP protocol support
> in Git.
>
> Perhaps it is time to restart work on _"smart" HTTP protocol_?
>


wasn't Shawn working on it?

ciao,
-- 
Paolo
http://paolo.ciarrocchi.googlepages.com/
http://mypage.vodafone.it/

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Johannes Schindelin]

Hi,

On Sun, 26 Apr 2009, Paolo Ciarrocchi wrote:

> On 4/26/09, Jakub Narebski <jnareb@gmail.com> wrote:
> 
> > The deciding feature (well, one of deciding features) was the fact
> > that Mercurial has better HTTP support... I guess (it was not obvious
> > from the analysis, but it was hinted at) that Mercurial uses its
> > custom protocol over HTTP, as opposed to "dumb" HTTP protocol support
> > in Git.
> >
> > Perhaps it is time to restart work on _"smart" HTTP protocol_?
> >
> 
> 
> wasn't Shawn working on it?

GIVE HIM A BREAK!

Isn't Shawn doing enough for Git?  No need to offload the stuff on him 
_that you could very well tackle yourself_.

Ciao,
Dscho

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Jakub Narebski]

On Sun, 26 April 2009, Johannes Schindelin wrote:
> On Sun, 26 Apr 2009, Paolo Ciarrocchi wrote:
> > On 4/26/09, Jakub Narebski <jnareb@gmail.com> wrote:
> > 
> > > The deciding feature (well, one of deciding features) was the fact
> > > that Mercurial has better HTTP support... I guess (it was not obvious
> > > from the analysis, but it was hinted at) that Mercurial uses its
> > > custom protocol over HTTP, as opposed to "dumb" HTTP protocol support
> > > in Git.
> > >
> > > Perhaps it is time to restart work on _"smart" HTTP protocol_?
> > 
> > wasn't Shawn working on it?
> 
> GIVE HIM A BREAK!
> 
> Isn't Shawn doing enough for Git?  No need to offload the stuff on him 
> _that you could very well tackle yourself_.

It is a bit pity that we don't have "smart" HTTP protocol as one of
git projects for this year Google Summer of Code 2009.

-- 
Jakub Narebski
Poland

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Johannes Schindelin]

Hi,

On Sun, 26 Apr 2009, Johannes Schindelin wrote:

> On Sun, 26 Apr 2009, Paolo Ciarrocchi wrote:
> 
> > On 4/26/09, Jakub Narebski <jnareb@gmail.com> wrote:
> > 
> > > The deciding feature (well, one of deciding features) was the fact
> > > that Mercurial has better HTTP support... I guess (it was not obvious
> > > from the analysis, but it was hinted at) that Mercurial uses its
> > > custom protocol over HTTP, as opposed to "dumb" HTTP protocol support
> > > in Git.
> > >
> > > Perhaps it is time to restart work on _"smart" HTTP protocol_?
> > >
> > 
> > 
> > wasn't Shawn working on it?
> 
> GIVE HIM A BREAK!

Sorry.  While it reflects exactly what I felt reading your mail, I should 
have phrased it like this:

	Don't ask what Shawn can do for you.  Ask what you can do for 
	Shawn.

Ciao,
Dscho

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Alex Blewitt]

Johannes Schindelin <Johannes.Schindelin <at> gmx.de> writes:

> 
> Hi,
> 
> On Sun, 26 Apr 2009, Johannes Schindelin wrote:
> 
> > On Sun, 26 Apr 2009, Paolo Ciarrocchi wrote:
> > 
> > > On 4/26/09, Jakub Narebski <jnareb <at> gmail.com> wrote:
> > > 
> > > > Perhaps it is time to restart work on _"smart" HTTP protocol_?
> > > >
> > > 
> > > 
> > > wasn't Shawn working on it?
> > 
> > GIVE HIM A BREAK!
> 
> Sorry.  While it reflects exactly what I felt reading your mail, I should 
> have phrased it like this:
> 
> 	Don't ask what Shawn can do for you.  Ask what you can do for 
> 	Shawn.

It's something I raised a while ago with the eclipse.egit discussion; I'm happy
to step forward and see  what I can do to improve the HTTP access, since I 
think that's critical for adoption in organizations  who, through no fault of 
the end user, are either not directly connected to the internet or have to go 
via HTTP proxies due to firewall limitations.

Alex

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Shawn O. Pearce]

Alex Blewitt <Alex.Blewitt@gmail.com> wrote:
> Johannes Schindelin <Johannes.Schindelin <at> gmx.de> writes:
> > On Sun, 26 Apr 2009, Johannes Schindelin wrote:
> > > On Sun, 26 Apr 2009, Paolo Ciarrocchi wrote:
> > > > On 4/26/09, Jakub Narebski <jnareb <at> gmail.com> wrote:
> > > > 
> > > > > Perhaps it is time to restart work on _"smart" HTTP protocol_?
> > > > 
> > > > wasn't Shawn working on it?

Errh, uhm, yes, I had planned to work on it.  I failed to find
the time.  Gerrit Code Review and the necessary patches to JGit
have basically sucked up any time that I have, for anything.  Oh,
and so has "repo", the hack^W^W^W^tool Android uses to manage its
forrest of 140+ Git repositories.

> > > GIVE HIM A BREAK!
> > 
> > Sorry.  While it reflects exactly what I felt reading your mail, I should 
> > have phrased it like this:
> > 
> > 	Don't ask what Shawn can do for you.  Ask what you can do for 
> > 	Shawn.
> 
> It's something I raised a while ago with the eclipse.egit discussion; I'm happy
> to step forward and see  what I can do to improve the HTTP access, since I 
> think that's critical for adoption in organizations  who, through no fault of 
> the end user, are either not directly connected to the internet or have to go 
> via HTTP proxies due to firewall limitations.

I'm quite looking forward to this.  I'm thrilled to see another
person is interested in better HTTP support, and is trying to
work on it.

-- 
Shawn.

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Paolo Ciarrocchi]

On 4/26/09, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Sun, 26 Apr 2009, Paolo Ciarrocchi wrote:
>
>> On 4/26/09, Jakub Narebski <jnareb@gmail.com> wrote:
>>
>> > The deciding feature (well, one of deciding features) was the fact
>> > that Mercurial has better HTTP support... I guess (it was not obvious
>> > from the analysis, but it was hinted at) that Mercurial uses its
>> > custom protocol over HTTP, as opposed to "dumb" HTTP protocol support
>> > in Git.
>> >
>> > Perhaps it is time to restart work on _"smart" HTTP protocol_?
>> >
>>
>>
>> wasn't Shawn working on it?
>
> GIVE HIM A BREAK!

it was just a question.

> Isn't Shawn doing enough for Git?  No need to offload the stuff on him
> _that you could very well tackle yourself_.


shawn is a stellar developer and with my message i didn't want to
imply that he should do more than what is already doing.
It was just a question to better understand the issue.

ciao.
-- 
Paolo
http://paolo.ciarrocchi.googlepages.com/
http://mypage.vodafone.it/

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Matthias Andree]

Am 26.04.2009, 10:16 Uhr, schrieb Jakub Narebski <jnareb@gmail.com>:

> I can't comment on MS Windows support, but AFAIK Mercurial has better
> support here than Git.

I have some experience here, and with exception to the SVN 1.6 breaks  
git-svn for https:// (probably due to misbehaviour of APR or SVN stuff on  
Cygwin), it works flawless on Cygwin 1.5. (SVN 1.5 on Cygwin 1.5 or SVN  
1.6 on Cygwin 1.7 seem to work).

I wonder why people are always pissed at Cygwin - it's quite easy to setup  
and works.

With Mercurial, I've sometimes found that it's awkward to integrate with  
other tools such as Emacs. It only works if you use the hg.exe approach,  
which entails packing the whole module library into the .exe and is  
inefficient. Particularly, it seems slower on start than the native or  
byte-compiled Python code for many operations. Not benchmarked, just a gut  
feeling. And I'm talking about a rather swift laptop here, Intel Core Duo  
T2500 (2 x 2 GHz), 2 GB RAM, Win XP SP3.

> The deciding feature (well, one of deciding features) was the fact
> that Mercurial has better HTTP support... I guess (it was not obvious
> from the analysis, but it was hinted at) that Mercurial uses its
> custom protocol over HTTP, as opposed to "dumb" HTTP protocol support
> in Git.
>
> Perhaps it is time to restart work on _"smart" HTTP protocol_?

That would certainly be useful, but the "packs" approach is something that  
may make this more difficult than for Mercurial. Git+SSH works rather well  
though.

-- 
Matthias Andree

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Jakub Narebski]

On Sun, 26 April 2009, Matthias Andree wrote:
> Am 26.04.2009, 10:16 Uhr, schrieb Jakub Narebski <jnareb@gmail.com>:
> 
> > I can't comment on MS Windows support, but AFAIK Mercurial has better
> > support here than Git.
> 
> I have some experience here, and with exception to the SVN 1.6 breaks  
> git-svn for https:// (probably due to misbehaviour of APR or SVN stuff on  
> Cygwin), it works flawless on Cygwin 1.5. (SVN 1.5 on Cygwin 1.5 or SVN  
> 1.6 on Cygwin 1.7 seem to work).
> 
> I wonder why people are always pissed at Cygwin - it's quite easy to setup  
> and works.

Well, but you have to install Cygwin (or use MsysGit, which isn't there
yet).  On the other hand you need to install Python for Mercurial...
but perhaps it is bundled in Windows install package for Mercurial.

Beside it isn't only about being easy to install and use (and have
decent enough performance) given SCM on MS Windows, but also about
tools such like TortoiseHg and VisualHg (as Windows users are usually
not used to using CLI alone).  Although also this improves for Git,
with TortoiseGit, Git Extensions and git-cheetah.  And I think it was
much worse (for Git vs Mercurial) at the time the analysis in question
was conducted.

[...]
> > The deciding feature (well, one of deciding features) was the fact
> > that Mercurial has better HTTP support... I guess (it was not obvious
> > from the analysis, but it was hinted at) that Mercurial uses its
> > custom protocol over HTTP, as opposed to "dumb" HTTP protocol support
> > in Git.
> >
> > Perhaps it is time to restart work on _"smart" HTTP protocol_?
> 
> That would certainly be useful, but the "packs" approach is something that  
> may make this more difficult than for Mercurial. Git+SSH works rather well  
> though.

As you can find in mailing list archives the design part of "tunelling"
pack protocol over HTTP, using git-aware server (for example some CGI
script, or simple HTTP server like Mercurial's hg-serve), is done.  Even
taking into account the fact that HTTP protocol by itself is stateless.
Unfortunately development itself of "smart" HTTP server for Git got
stalled... if it was present, the conclusion of mentioned analysis might
have been different.  OTOH perhaps it would be not, as it is my impression
that Google Code stuff is either Python or Java...


P.S. I wonder what happened to GMane interface... seems stalled.

-- 
Jakub Narebski
Poland

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Matthias Andree]

Am 26.04.2009, 12:09 Uhr, schrieb Jakub Narebski <jnareb@gmail.com>:

> On Sun, 26 April 2009, Matthias Andree wrote:
>> Am 26.04.2009, 10:16 Uhr, schrieb Jakub Narebski <jnareb@gmail.com>:
>>
>> > I can't comment on MS Windows support, but AFAIK Mercurial has better
>> > support here than Git.
>>
>> I have some experience here, and with exception to the SVN 1.6 breaks
>> git-svn for https:// (probably due to misbehaviour of APR or SVN stuff  
>> on
>> Cygwin), it works flawless on Cygwin 1.5. (SVN 1.5 on Cygwin 1.5 or SVN
>> 1.6 on Cygwin 1.7 seem to work).
>>
>> I wonder why people are always pissed at Cygwin - it's quite easy to  
>> setup and works.
>
> Well, but you have to install Cygwin (or use MsysGit, which isn't there
> yet).  On the other hand you need to install Python for Mercurial...
> but perhaps it is bundled in Windows install package for Mercurial.

AFAIR Mercurial is compiled through py2exe, but I'd have to look again  
what that actually means WRT Python installations.

> Beside it isn't only about being easy to install and use (and have
> decent enough performance) given SCM on MS Windows, but also about
> tools such like TortoiseHg and VisualHg (as Windows users are usually
> not used to using CLI alone).  Although also this improves for Git,
> with TortoiseGit, Git Extensions and git-cheetah.  And I think it was
> much worse (for Git vs Mercurial) at the time the analysis in question
> was conducted.

I wonder why people always talk about "not being used to console" or  
whatever. These captive GUIs serialize work and tend to get in the way. It  
may be different for beasts like Eclipse, but I haven't tried the latter.

I have yet to see any TortoiseCrap that does not smell. Judging from  
Tortoise{CVS,SVN,Hg}, they are feature-limited, cumbersome-to-use  
frontends where a command line client would be much faster. However you  
can't mix Unix versions of SCM exes and Tortoise-compiled exes easily due  
to differing [CR-]LF conventions.

I've done away with all that TortoiseCrap and use SCM from the command  
line.

I acknowledge that code browsing as in gitk or git-gui is more concise  
than a shell at times, but that doesn't warrant Tortoise* Explorer  
extension stuff which only wraps the trivial operations anyhow.

My opinion is that if people can't be bothered to learn the few SCM  
commands, they won't fully understand what Tortoises creep to do, and then  
they shouldn't be let anywhere near any kind of shell - graphical or text  
console - anyways.

> As you can find in mailing list archives the design part of "tunelling"
> pack protocol over HTTP, using git-aware server (for example some CGI
> script, or simple HTTP server like Mercurial's hg-serve), is done.  Even
> taking into account the fact that HTTP protocol by itself is stateless.
> Unfortunately development itself of "smart" HTTP server for Git got
> stalled... if it was present, the conclusion of mentioned analysis might
> have been different.  OTOH perhaps it would be not, as it is my  
> impression that Google Code stuff is either Python or Java...

I don't care much about Google anything, to be frank.

> P.S. I wonder what happened to GMane interface... seems stalled.

What's so difficult about mailing "subscribe git MY@ADDRE.SS.example" to  
majordomo at vger dot kernel dot org?

-- 
Matthias Andree

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Jakub Narebski]

On Sun, 26 April 2009, Jakub Narebski wrote:
> On Sun, 26 April 2009, Matthias Andree wrote:
> > Am 26.04.2009, 10:16 Uhr, schrieb Jakub Narebski <jnareb@gmail.com>:

> [...]
> > > The deciding feature (well, one of deciding features) was the fact
> > > that Mercurial has better HTTP support... I guess (it was not obvious
> > > from the analysis, but it was hinted at) that Mercurial uses its
> > > custom protocol over HTTP, as opposed to "dumb" HTTP protocol support
> > > in Git.
> > >
> > > Perhaps it is time to restart work on _"smart" HTTP protocol_?
> > 
> > That would certainly be useful, but the "packs" approach is something that  
> > may make this more difficult than for Mercurial. Git+SSH works rather well  
> > though.
> 
> As you can find in mailing list archives the design part of "tunelling"
> pack protocol over HTTP, using git-aware server (for example some CGI
> script, or simple HTTP server like Mercurial's hg-serve), is done.
[...]

See thread named "More on git over HTTP POST" and its predecessor

  http://permalink.gmane.org/gmane.comp.version-control.git/91196
  http://thread.gmane.org/gmane.comp.version-control.git/91104/focus=91196

-- 
Jakub Narebski
Poland

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[A Large Angry SCM]

Jakub Narebski wrote:
> Christian Couder <chriscool@tuxfamily.org> writes:
> 
>> For information, now Google Code supports Mercurial for project hosting:
>>
>> http://google-code-updates.blogspot.com/2009/04/mercurial-support-for-project-hosting.html
>>
>> Mercurial was choosen over Git because of this (one year old) analysis:
>>
>> http://code.google.com/p/support/wiki/DVCSAnalysis
>>
>> There is this article on LWN about the analysis:
>>
>> http://lwn.net/Articles/330138/
> 
> It is a pity that the choice was based on year old analysis.  One year
> for actively developed and fast moving targets such like Git and
> Mercurial is ages in terms of development history.  But I guess this
> is unavoidable.
> 
> For example periodic "maintenance" (garbage collecting) is nowadays
> quite automatic in git, with fetching into pack, periodic repacking if
> number of loose objects is above tthreshold, and "git gc --auto".
> 
> Whether Mercurial or Git has better UI and better documentation is
> IMHO a matter of debate.  Git documentation is much better that it
> was, with "Git User's Manual" and "Git Community Book"; UI also is
> being improved.
> 
> I can't comment on MS Windows support, but AFAIK Mercurial has better
> support here than Git.
> 
> 
> The deciding feature (well, one of deciding features) was the fact
> that Mercurial has better HTTP support... I guess (it was not obvious
> from the analysis, but it was hinted at) that Mercurial uses its
> custom protocol over HTTP, as opposed to "dumb" HTTP protocol support
> in Git.
> 
> Perhaps it is time to restart work on _"smart" HTTP protocol_?
> 

Another important criteria was which, both or neither of Git and Hg 
would actually work and perform well on top of Google Code's underling 
storage system and except to mention they would be using Bigtable, the 
report did not discuss this. Git on top of Bigtable will not perform well.

Read the paper and do the math if you are interested.

	http://labs.google.com/papers/bigtable-osdi06.pdf

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Michael Witten]

On Sun, Apr 26, 2009 at 09:54, A Large Angry SCM <gitzilla@gmail.com> wrote:
> Another important criteria was which, both or neither of Git and Hg would
> actually work and perform well on top of Google Code's underling storage
> system and except to mention they would be using Bigtable, the report did
> not discuss this. Git on top of Bigtable will not perform well.
>
> Read the paper and do the math if you are interested.
>
>        http://labs.google.com/papers/bigtable-osdi06.pdf
>

If the math is so simple, then surely you could have supplied a quick
overview of the problem. Nobody likes "this is left as an exercise for
the reader".

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Johannes Schindelin]

Hi,

On Sun, 26 Apr 2009, A Large Angry SCM wrote:

> Another important criteria was which, both or neither of Git and Hg 
> would actually work and perform well on top of Google Code's underling 
> storage system and except to mention they would be using Bigtable, the 
> report did not discuss this. Git on top of Bigtable will not perform 
> well.

Actually, did we not arrive at the conclusion that it could perform well 
at least with the filesystem layer on top of big table, but even better if 
the big tables stored certain chunks (not really all that different from 
the chunks needed for mirror-sync!)?

Back when I discussed this with a Googler, it was all too obvious that 
they are not interested (and in the meantime I understand why, see my 
other mail).

Ciao,
Dscho

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[A Large Angry SCM]

Johannes Schindelin wrote:
> Hi,
> 
> On Sun, 26 Apr 2009, A Large Angry SCM wrote:
> 
>> Another important criteria was which, both or neither of Git and Hg 
>> would actually work and perform well on top of Google Code's underling 
>> storage system and except to mention they would be using Bigtable, the 
>> report did not discuss this. Git on top of Bigtable will not perform 
>> well.
> 
> Actually, did we not arrive at the conclusion that it could perform well 
> at least with the filesystem layer on top of big table, but even better if 
> the big tables stored certain chunks (not really all that different from 
> the chunks needed for mirror-sync!)?
> 
> Back when I discussed this with a Googler, it was all too obvious that 
> they are not interested (and in the meantime I understand why, see my 
> other mail).
> 

I don't remember the mirror-sync discussion. But I do remember that when 
the discussion turned to implementing a filesystem on top of Bigtable 
that would not cause performance problems for Git, my response was that 
you'd still be much better off going to GFS directly instead of faking a 
filesystem on top of Bigtable without all of the Bigtable limitations.

Bigtable _is_ appealing to implement the Git object store on. It's too 
bad the latency in Bigtable would make it horribly slow.

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Johannes Schindelin]

Hi,

On Sun, 26 Apr 2009, A Large Angry SCM wrote:

> Johannes Schindelin wrote:
> 
> > On Sun, 26 Apr 2009, A Large Angry SCM wrote:
> > 
> > > Another important criteria was which, both or neither of Git and Hg 
> > > would actually work and perform well on top of Google Code's 
> > > underling storage system and except to mention they would be using 
> > > Bigtable, the report did not discuss this. Git on top of Bigtable 
> > > will not perform well.
> > 
> > Actually, did we not arrive at the conclusion that it could perform 
> > well at least with the filesystem layer on top of big table, but even 
> > better if the big tables stored certain chunks (not really all that 
> > different from the chunks needed for mirror-sync!)?
> > 
> > Back when I discussed this with a Googler, it was all too obvious that 
> > they are not interested (and in the meantime I understand why, see my 
> > other mail).
> 
> I don't remember the mirror-sync discussion. But I do remember that when 
> the discussion turned to implementing a filesystem on top of Bigtable 
> that would not cause performance problems for Git, my response was that 
> you'd still be much better off going to GFS directly instead of faking a 
> filesystem on top of Bigtable without all of the Bigtable limitations.

Umm, GFS is built on top of Bigtable, no?

> Bigtable _is_ appealing to implement the Git object store on. It's too 
> bad the latency in Bigtable would make it horribly slow.

If you store one object per Bigtable, yes.  If you store a few undelta'd 
objects there, and then use the pack run to optimize those tables, I think 
it would not be horribly slow.  Of course, you'd need to do exactly the 
same optimizations necessary for mirror-sync, but I might have mentioned 
that already ;-)

Ciao,
Dscho

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[A Large Angry SCM]

Johannes Schindelin wrote:
> Hi,
> 
> On Sun, 26 Apr 2009, A Large Angry SCM wrote:
> 
>> Johannes Schindelin wrote:
>>
>>> On Sun, 26 Apr 2009, A Large Angry SCM wrote:
>>>
>>>> Another important criteria was which, both or neither of Git and Hg 
>>>> would actually work and perform well on top of Google Code's 
>>>> underling storage system and except to mention they would be using 
>>>> Bigtable, the report did not discuss this. Git on top of Bigtable 
>>>> will not perform well.
>>> Actually, did we not arrive at the conclusion that it could perform 
>>> well at least with the filesystem layer on top of big table, but even 
>>> better if the big tables stored certain chunks (not really all that 
>>> different from the chunks needed for mirror-sync!)?
>>>
>>> Back when I discussed this with a Googler, it was all too obvious that 
>>> they are not interested (and in the meantime I understand why, see my 
>>> other mail).
>> I don't remember the mirror-sync discussion. But I do remember that when 
>> the discussion turned to implementing a filesystem on top of Bigtable 
>> that would not cause performance problems for Git, my response was that 
>> you'd still be much better off going to GFS directly instead of faking a 
>> filesystem on top of Bigtable without all of the Bigtable limitations.
> 
> Umm, GFS is built on top of Bigtable, no?

Other way around.

>> Bigtable _is_ appealing to implement the Git object store on. It's too 
>> bad the latency in Bigtable would make it horribly slow.
> 
> If you store one object per Bigtable, yes.  If you store a few undelta'd 
> objects there, and then use the pack run to optimize those tables, I think 
> it would not be horribly slow.  Of course, you'd need to do exactly the 
> same optimizations necessary for mirror-sync, but I might have mentioned 
> that already ;-)

But now you have to find where you stored those "few undelta'd objects" 
and then go get the object you're interested in. The only way you can 
win with that scheme is if you can find groups of objects that are 
(almost) always accessed together, for all objects (and still not get 
tripped up by the other limitations of Bigtable).

One method would be to group all of the commit objects into one BT entry 
and then create a BT entry for each commit that contains all the trees 
and blobs. This may be fast enough for some operations but would cause 
the storage requirements to explode.

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[James Cloos]

>>>>> "Jakub" == Jakub Narebski <jnareb@gmail.com> writes:

Jakub> Perhaps it is time to restart work on _"smart" HTTP protocol_?

I had put together some ideas for how that could work, but didn’t post
them because it looked like smarter http transport was fait accompli.

The general idea was to use an http server as a proxy for git-daemon.
Sites running git-daemon could put up such a proxy which will only
accept proxy attempts to their own daemon, and sites or users behind
restrictive firewalls (or http proxies) could set up such a proxy which
requires http-level authentication but then will proxy for any git-daemon.

IIRC, I intended to suggest the name post_proxy for the config files.
One could add a post_proxy to a repo (for the former style) or in their
global config (for the latter style).  Either way, an http_proxy could
still be used, if necessary, to access the post_proxy.

Any stream the client would send to a remote git-daemon would be
encapsulted and sent via an HTTP POST to the post_proxy, which would
use the git protocol to send it to the specified git-daemon.  Any
reply back from the git-daemon would be sent to the client as the reply
to the POST.

The proxy can be readilly written as a CGI, as a mod_lang extension (for
one’s favourite lang), as a standalone server, or as an extension to
projects such as gitweb or cgit.

I never got past the rough design phase because, when I was preparing to
post the idea, it looked like alternate code was already written....

Is there any interest in this?

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Johannes Schindelin]

Hi,

On Sun, 26 Apr 2009, Christian Couder wrote:

> For information, now Google Code supports Mercurial for project hosting:
> 
> http://google-code-updates.blogspot.com/2009/04/mercurial-support-for-project-hosting.html
> 
> Mercurial was choosen over Git because of this (one year old) analysis:
> 
> http://code.google.com/p/support/wiki/DVCSAnalysis
> 
> There is this article on LWN about the analysis:
> 
> http://lwn.net/Articles/330138/

FWIW some little bird (yes, related to the Google Code team) told me that 
the real reason was because a certain person important in Git development 
was upsetting the Google Code team.  It _might_ be related to the fact 
that the original Googe Code team had a substantial involvement in 
Subversion...

So, don't believe that the reason Git is not supported by Google Code is a 
technical one (just like it was no technical reason at all for Python to 
choose Hg over Git).

Oh, and I have to be very clear on some important point: they are free to 
choose what they want.  I, for one, am happy that not everybody and her 
dog uses Git.  That way, we can steal cute ideas from other projects like 
Hg or Bazaar.

Ciao,
Dscho

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Michael Witten]

On Sun, Apr 26, 2009 at 05:13, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> So, don't believe that the reason Git is not supported by Google Code is a
> technical one (just like it was no technical reason at all for Python to
> choose Hg over Git).

Frankly, I think that it can be considered technical for Python
developers to choose an SCM written in Python. I can't fault them for
that.

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Miles Bader]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> So, don't believe that the reason Git is not supported by Google Code is a 
> technical one (just like it was no technical reason at all for Python to 
> choose Hg over Git).

I look at it this way:  having added another VCS to the previously
subversion-only google-code, they've probably generalized parts of their
framework in a way that should make it much easier to add git support in
the future!

Maybe it's time to file another git support request at google code...

:-)

-Miles

-- 
Patience, n. A minor form of despair, disguised as a virtue.

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Shawn O. Pearce]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On Sun, 26 Apr 2009, Christian Couder wrote:
> 
> > For information, now Google Code supports Mercurial [...]
> > 
> > Mercurial was choosen over Git because of this (one year old) analysis:
> > 
> > http://code.google.com/p/support/wiki/DVCSAnalysis
> 
> FWIW some little bird (yes, related to the Google Code team) told me that 
> the real reason was because [...]

There were certainly technical factors involved.

As the DVCSAnalysis document above describes, Hg's relatively
efficient custom Hg-in-HTTP protocol performs about as well as git://
does, but requires only stateless HTTP, rather than a stateful
direct TCP connection.

There is a fundemental reason why Google App Engine only supports
incoming HTTP connections on 80/443.  Its easy to stand up a new
application behind the existing load balancers.  Its quite a bit
more effort to add support for yet-another-protocol.

As the recent discussion on eclipse.egit

  http://www.eclipse.org/newsportal/article.php?id=34&group=eclipse.egit#34

showed, many users are stuck behind corporate firewalls where only
HTTP transit is available.

Tossing aside the Google server infrastructure and why HTTP might be
preferred there, Google also tries to target the widest user base
possible.  Running your VCS through HTTP, which can be easily run
through a corporate proxy, gives a wider user base than running your
VCS through an SSH tunnel, or a relatively new IANA assigned port.

A long-time GCC committer, and an old SVN committer, pointed out
to me the other day that the reason why SVN uses HTTP is so it
can get around corporate firewalls without involving the IT staff.
Because for the past 10 years, being "on the Internet" has meant
being "behind an HTTP and SMTP proxy".  And "tunneling through HTTP"
is somehow safe, no matter how insecure the protocol might be;
while opening an IANA assigned port causes the world to end.

IOW, if Git wants to expand into these user communities where the
individual is stuck behind a corporate proxy that only permits HTTP
"for security reasons" (but blindly winds up passing through whatever
it gets), we need to support a more efficient HTTP protocol.

-- 
Shawn.

Re: Google Code: Support for Mercurial and Analysis of Git and Mercurial

[Mark Lodato]

On Mon, Apr 27, 2009 at 5:15 PM, Shawn O. Pearce <spearce@spearce.org> wrote:
> IOW, if Git wants to expand into these user communities where the
> individual is stuck behind a corporate proxy that only permits HTTP
> "for security reasons" (but blindly winds up passing through whatever
> it gets), we need to support a more efficient HTTP protocol.

Another advantage of supporting HTTP is allowing HTTPS.  SSL can take
advantage of an existing PKI infrastructure, either the Internet's
existing server-side certificate system, or a corporate server- and
possibly client-side PKI infrastructure.

--
Mark