From P4 to Git

From P4 to Git

[Arnaud Bailly]

Hello,
I am investigating the use of Git in a setting where we:
 1. branches a lot
 2. have a very large codebase

Given Git is developed to handle these 2 issues, I suspect it would be a
very good choice, but I need to gather some experiments feedback and
hard figures on how Git performs (storage use, necessary
bandwidth/resources, maintainance of repositories, cleanup & gc...).

For the experiment part, I am started working on it but would be
interested in other people's experiences.

For the figures part, I think I read somewhere there exists some
statistics on Git usage for Linux kernel. Is this correct ? If true,
where could I find them ? 

Thanks in advance for answering my (maybe pointless) questions and for
producing such a nice piece of software.
-- 
Arnaud Bailly

Re: From P4 to Git

[david]

On Tue, 28 Jul 2009, Arnaud Bailly wrote:

> Hello,
> I am investigating the use of Git in a setting where we:
> 1. branches a lot
> 2. have a very large codebase
>
> Given Git is developed to handle these 2 issues, I suspect it would be a
> very good choice, but I need to gather some experiments feedback and
> hard figures on how Git performs (storage use, necessary
> bandwidth/resources, maintainance of repositories, cleanup & gc...).
>
> For the experiment part, I am started working on it but would be
> interested in other people's experiences.
>
> For the figures part, I think I read somewhere there exists some
> statistics on Git usage for Linux kernel. Is this correct ? If true,
> where could I find them ?
>
> Thanks in advance for answering my (maybe pointless) questions and for
> producing such a nice piece of software.

the one thing that I have seen P4 reported as handling better than git is 
the case where the repository is very large. this is typically not caused 
by lots of lines of code, but by checking in lots of binary blobs to the 
repository.

David Lang

Re: From P4 to Git

[Jakub Narebski]

Arnaud Bailly <abailly@oqube.com> writes:

> I am investigating the use of Git in a setting where we:
>  1. branches a lot

Git handles large number of branches very well.  

The workflow of topic branches, where each new feature is developed on
separate branch, which can be individually picked to be merged (or
not) is used by git itself, for example.

>  2. have a very large codebase

How large codebase?  What it means "large codebase"?  Large number of
files, or files large in size (usually binary)?


Git can deal comfortably with codebase of the size of Linux kernel.
Perl 5 core was converted from Perforce to Git.

But git is snapshot based, not changeset based, and treats project
(repository) as whole, not as a combination of single file histories.
This means that it would be unproductive to use 'everything in single
repository' approach.  If your codebase is of the size of whole KDE
tree, or the whole GNOME tree, you would need to organize it and split
it into smaller, reasonably sized repositories (you can urganize them
back together in a superproject using submodules).

If ou can't do that, you would probably be better with other version
control system, like Subversion (IIRC both KDE and OpenOffice.org
chosen this free centralized version control system).


Because Git was created to version control source code, it might not
work well with large binary files, meaning that performance would
suffer.  

Partial checkouts (where you check out only part of repository) were
proposed, but are not implemented yet.  Neither did lazy clone /
remote alternates idea.  You can do a bit with undocumented `delta`
gitattribute, and with putting large binary blobs into separate
packfile, which is 'kept' (using *.keep file) against repacking, and
perhaps available on networked filesystem.

I think you can use refs/replace/ mechanizm (IIRC currently in 'pu',
(proposed updates) branch) to have two versions of repository: one
with binary blobs and one without.

> 
> Given Git is developed to handle these 2 issues, I suspect it would be a
> very good choice, but I need to gather some experiments feedback and
> hard figures on how Git performs (storage use, necessary
> bandwidth/resources, maintainance of repositories, cleanup & gc...).
> 
> For the experiment part, I am started working on it but would be
> interested in other people's experiences.

Check out Sam Vilain (?) reports from converting Perl 5 repository
from Perforce to Git.

> 
> For the figures part, I think I read somewhere there exists some
> statistics on Git usage for Linux kernel. Is this correct ? If true,
> where could I find them ? 

There is GitStat project: http://mirror.celinuxforum.org/gitstat/

There was also Git-Statistics project at Google Summer of Code 2008
which repository can be found at http://repo.or.cz/w/git-stats.git
See http://git.or.cz/gitwiki/SoC2008Projects

> 
> Thanks in advance for answering my (maybe pointless) questions and for
> producing such a nice piece of software.
> -- 
> Arnaud Bailly

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: From P4 to Git

[Jakub Narebski]

I have re-added git mailing list to Cc.

On Fri, 31 July 2009, Arnaud Bailly wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
>>
>>>  2. have a very large codebase
>>
>> How large codebase?  What it means "large codebase"?  Large number of
>> files, or files large in size (usually binary)?
> 
> Mostly lot of source files, amounting to about 9 MLOCs of mixed
> languages source.
> 
>> Git can deal comfortably with codebase of the size of Linux kernel.
>> Perl 5 core was converted from Perforce to Git.
> 
> I guess we are in the same order of magnitude than kernel.

Linux kernel, for development of which git was initially designed,
is around 7.5M LoC of code (10M LoC with comments and blank lines)[1].
The performance for such large codebase has to be good (at least on
Linux and other POSIX systems), as good performance was one of goal
decisions of git.

Perl 5 core, which version control history was converted from Perforce
to Git in December 2008[1][2] by Sam Vilain (you might want to take
a look how it was done; unfortunately it seems that Sam Vilain blog
vanished from Internet), is 2.3M LoC of mixed language code (mainly
Perl and C, with twice as much Perl), so it is smaller than yours
codebase.

[1] According to OSS software metric site Ohloh (http://www.ohloh.net)
[2] http://news.perlfoundation.org/2008/12/perl_5_development_now_on_git.html
[3] http://use.perl.org/article.pl?sid=08/12/22/0830205

>> But git is snapshot based, not changeset based, and treats project
>> (repository) as whole, not as a combination of single file histories.
>> This means that it would be unproductive to use 'everything in single
>> repository' approach.  If your codebase is of the size of whole KDE
>> tree, or the whole GNOME tree, you would need to organize it and split
>> it into smaller, reasonably sized repositories (you can urganize them
>> back together in a superproject using submodules).
> 
> That's my biggest concern. We are actually using a single tree repository
> approach with lot of branches. What led me to Git at first was the ease
> of branching and merging. I used branching and merging with Subversion
> and its painful.

So it looks like you wouldn't _need_ to split source tree into separate
smaller repositories for performance reasons.  Still it might be good
idea to put separate (sub)projects into separate repositories.  But
I guess you can do that even at later time (although it would be best
to do this upfront).

Branching and merging in Git is very easy (with Subversion 1.5 merging
is supposedly to get easier).  Git itself uses 'topic branches' workflow,
where each feature (each series of patches) gets its own branch, and
branches are cherry-picked to be merged (or to be dropped, or replaced
by newer version of series).
 
>> There is GitStat project: http://mirror.celinuxforum.org/gitstat/

Which follows Linux kernel.  There are also some GitStat deployments
tracking other code.

>> There was also Git-Statistics project at Google Summer of Code 2008
>> which repository can be found at http://repo.or.cz/w/git-stats.gitSee http://git.or.cz/gitwiki/SoC2008Projects
>>
> 
> Great.

And there are tools such as Ohloh, which do software metric, and some
of code is available.  GitHub hosting site also offers some software 
metric / statistic tools in its web interface; I don't know about other
sites such as Gitorious or InDefero.  Gitweb currently doesn't offer any
statistics.

-- 
Jakub Narebski
Poland

Re: From P4 to Git

[Alex Riesen]

On Fri, Jul 31, 2009 at 11:22, Jakub Narebski<jnareb@gmail.com> wrote:
>> That's my biggest concern. We are actually using a single tree repository
>> approach with lot of branches. What led me to Git at first was the ease
>> of branching and merging. I used branching and merging with Subversion
>> and its painful.
>
> So it looks like you wouldn't _need_ to split source tree into separate
> smaller repositories for performance reasons.  Still it might be good
> idea to put separate (sub)projects into separate repositories.  But
> I guess you can do that even at later time (although it would be best
> to do this upfront).

It looks like they use P4 branches. Which are NOT branches as Git
understands it (a line of history). The P4 "branches" are just directories
with some metadata regarding recording where the data in the directory
were before an act of "branching" (just a server-side recursive copy).

In this case (and this must be, as there are no other branches in P4),
they'll have to split their repository.

Re: From P4 to Git

[Sam Vilain]

On Fri, 2009-07-31 at 11:22 +0200, Jakub Narebski wrote:
> Sam Vilain (you might want to take
> a look how it was done; unfortunately it seems that Sam Vilain blog
> vanished from Internet)

You think that was unfortunate?  Wow maybe I should pull finger and get
something back up again :-).  Didn't try since the drupal blog I had
finally bitrot, silly php.

So if you're looking for the actual importer ... google git-p4raw,
should be on http://utsl.gen.nz/gitweb/ or http://git.catalyst.net.nz/gw

Sam

Re: From P4 to Git

[Sam Vilain]

Alex Riesen wrote:
>> So it looks like you wouldn't _need_ to split source tree into separate
>> smaller repositories for performance reasons.  Still it might be good
>> idea to put separate (sub)projects into separate repositories.  But
>> I guess you can do that even at later time (although it would be best
>> to do this upfront).
>>     
>
> It looks like they use P4 branches. Which are NOT branches as Git
> understands it (a line of history). The P4 "branches" are just directories
> with some metadata regarding recording where the data in the directory
> were before an act of "branching" (just a server-side recursive copy).
>
> In this case (and this must be, as there are no other branches in P4),
> they'll have to split their repository.

p4's branches are 'close enough'.  My tool travels through the history
of the repository forwards, detects paths where new content was placed
and calls those roots.  When branching is detected (recorded
file-by-file in perforce), it puts the branch source as a new parent. 
When branches are 'integrated', it makes a fuzzy decision based on the
number of outstanding merges it thinks are due, also based on a merge
base calculation on the current view of the derived history.  It allows
manual correction of this fuzzy information, and arbitrary grafting
along the way.  Discrepancies between the fuzzy decision and the actual
integration records are recorded in the commit log along with other
metadata including the commit ID.

Sam

Re: From P4 to Git

[Alex Riesen]

On Mon, Aug 3, 2009 at 09:49, Sam Vilain<sam@vilain.net> wrote:
> Alex Riesen wrote:
>>> So it looks like you wouldn't _need_ to split source tree into separate
>>> smaller repositories for performance reasons.  Still it might be good
>>> idea to put separate (sub)projects into separate repositories.  But
>>> I guess you can do that even at later time (although it would be best
>>> to do this upfront).
>>>
>>
>> It looks like they use P4 branches. Which are NOT branches as Git
>> understands it (a line of history). The P4 "branches" are just directories
>> with some metadata regarding recording where the data in the directory
>> were before an act of "branching" (just a server-side recursive copy).
>>
>> In this case (and this must be, as there are no other branches in P4),
>> they'll have to split their repository.
>
> p4's branches are 'close enough'.  My tool travels through the history
> of the repository forwards, detects paths where new content was placed
> and calls those roots.  When branching is detected (recorded
> file-by-file in perforce), it puts the branch source as a new parent.
> When branches are 'integrated', it makes a fuzzy decision based on the
> number of outstanding merges it thinks are due, also based on a merge
> base calculation on the current view of the derived history.  It allows
> manual correction of this fuzzy information, and arbitrary grafting
> along the way.  Discrepancies between the fuzzy decision and the actual
> integration records are recorded in the commit log along with other
> metadata including the commit ID.

Sounds dangerous. And interesting...

Is it an import-once tool, or can the process be restarted? (because it looks
like the script needs a complicated setup).
Can it be used from a client machine? And more importantly:
can the branches be found from incomplete history,
restricted by path and changelist range? (because, in a corporate
setup, clients seldom have full access to all data).

Re: From P4 to Git

[Sam Vilain]

On Mon, 2009-08-03 at 10:47 +0200, Alex Riesen wrote:
> Is it an import-once tool, or can the process be restarted? (because it looks
> like the script needs a complicated setup).

It's fully restartable.  Not only that but it uses transaction
protection to make sure that its internal state doesn't get corrupted
when performing the various options.

> Can it be used from a client machine?
>  And more importantly:
> can the branches be found from incomplete history,
> restricted by path and changelist range? (because, in a corporate
> setup, clients seldom have full access to all data).

No, it's server only.  I think I did get around to implementing not having
to go through all the stages for branches you didn't care to import.  It's
difficult though, the stage which correlates those thousands of 'integrate'
records is never going to be fast.  Be prepared to tune your postgres - add
lots of shared_buffers and sort memory if your project is as large as Perl.

Sam.

Re: From P4 to Git

[Alex Riesen]

On Mon, Aug 3, 2009 at 13:30, Sam Vilain<sam@vilain.net> wrote:
> On Mon, 2009-08-03 at 10:47 +0200, Alex Riesen wrote:
>> Is it an import-once tool, or can the process be restarted? (because it looks
>> like the script needs a complicated setup).
>
> It's fully restartable.  Not only that but it uses transaction
> protection to make sure that its internal state doesn't get corrupted
> when performing the various options.

"varios options"? Operations? As when working on a live server?
Aren't P4 changelist numbers always increasing? Or you mean
the protection against multiple running instances of p4raw,
so it is also parallelizable?

>> Can it be used from a client machine?
>>  And more importantly:
>> can the branches be found from incomplete history,
>> restricted by path and changelist range? (because, in a corporate
>> setup, clients seldom have full access to all data).
>
> No, it's server only. ...

Darn. I hoped it wasn't. Can't play with it, then.

> ... I think I did get around to implementing not having
> to go through all the stages for branches you didn't care to import.  It's
> difficult though, the stage which correlates those thousands of 'integrate'
> records is never going to be fast.

Maybe if it is done locally, it can be improved? You seem to use the
Postgre for lookups, right?

> Be prepared to tune your postgres - add lots of shared_buffers and
> sort memory if your project is as large as Perl.

Mine isn't, but it is thrown on a server with a lot of others. Some branched
off mine, some integrate it, some just copied and re-introduced into repository
(these will probably worked with manually forever). There is also a small
problem of different P4 servers, hosting code from the same project (they
pull different repos together on client, imagine that!)

And all of them go back for 5-6 years, so it is kind of largish (not to mention
a mess of binaries).

Some branches I hope to merge back. I do that already, but it is a lot of
manual work (I use that git-p4-import.bat I posted early, which only can
import a sequence of changelists with files they touch).

That's were I hoped your project could help. I thought, if I pull in all the
needed changelists (selected by path/CL), there may be a way to
recreate a mergeable history out of this dump. At least, one involving less
labor then I have to do now.

Re: From P4 to Git

[Sam Vilain]

On Mon, 2009-08-03 at 15:50 +0200, Alex Riesen wrote:
> On Mon, Aug 3, 2009 at 13:30, Sam Vilain<sam@vilain.net> wrote:
> > On Mon, 2009-08-03 at 10:47 +0200, Alex Riesen wrote:
> >> Is it an import-once tool, or can the process be restarted? (because it looks
> >> like the script needs a complicated setup).
> >
> > It's fully restartable.  Not only that but it uses transaction
> > protection to make sure that its internal state doesn't get corrupted
> > when performing the various options.
> 
> "varios options"? Operations? As when working on a live server?
> Aren't P4 changelist numbers always increasing? Or you mean
> the protection against multiple running instances of p4raw,
> so it is also parallelizable?

The "live" parts are never touched - only the write-only files that
perforce writes; and the rcs files are read using rcs.

What I was referring to by "various options" was the commands in the
program which perform the migration.  It's a multi-step process - load
the metadata from perforce 'load', import the file images to git blobs
'export-blobs', trace through the path structure and decide which parts
are the roots, and make parents 'find-branches', etc.  All parts are
fully transaction protected, restartable and rewindable.  It also means
that if a command crashes that you are left in a sane state.  I had to
build it like that to keep my sanity writing and using it :-).  After
all it was something of a reverse engineering of Perforce's internals.

All you need is the RCS backing files, a 'checkpoint' and (optionally)
'journal' files.  No access to the live perforce DBs is required.

> > No, it's server only. ...
> Darn. I hoped it wasn't. Can't play with it, then.

Yeah, in theory you could derive the internal table information by
issuing zillions of 'p4 integrated', 'p4 filelog' commands etc.  I've
written p4raw sub-commands which can produce the output of those
commands but not tried to go the other way; I wasn't interested.

> > ... I think I did get around to implementing not having
> > to go through all the stages for branches you didn't care to import.  It's
> > difficult though, the stage which correlates those thousands of 'integrate'
> > records is never going to be fast.
> Maybe if it is done locally, it can be improved? You seem to use the
> Postgre for lookups, right?

I'm sure that the queries could be improved, but Postgres is actually
quite good at crunching the numbers.  It's just a lot of data to crunch.

> That's were I hoped your project could help. I thought, if I pull in all the
> needed changelists (selected by path/CL), there may be a way to
> recreate a mergeable history out of this dump. At least, one involving less
> labor then I have to do now.

Yeah... well the design of my tool is that it needs to have the perforce
internal information.  But really you can probably consider the
converter orphaned, I have no current need to work on it; it served a
purpose, which was converting perl's perforce to perl.git, and that's
history now.

Sam

Re: From P4 to Git

[John Tapsell]

2009/7/29 Jakub Narebski <jnareb@gmail.com>:
> If ou can't do that, you would probably be better with other version
> control system, like Subversion (IIRC both KDE and OpenOffice.org
> chosen this free centralized version control system).

Going a bit off topic - but to clarify, KDE hasn't really 'chosen' svn
over git, it's that we haven't switched to git _yet_. It just takes
time.  We already have trial projects (Amarok, for example, has
switched to git to lead the way) and I'm not sure if git submodule
support is mature enough, but baring any major problems we will
switch.

John

Re: From P4 to Git

[Alex Riesen]

Sam Vilain, Mon, Aug 03, 2009 22:32:20 +0200:
> On Mon, 2009-08-03 at 15:50 +0200, Alex Riesen wrote:
> > "varios options"? Operations? As when working on a live server?
> > Aren't P4 changelist numbers always increasing? Or you mean
> > the protection against multiple running instances of p4raw,
> > so it is also parallelizable?
> 
> The "live" parts are never touched - only the write-only files that
> perforce writes; and the rcs files are read using rcs.

I always felt Perforce had a touch of pure evil...

> All you need is the RCS backing files, a 'checkpoint' and (optionally)
> 'journal' files.  No access to the live perforce DBs is required.

Which in my case is all the same. I don't dare to ask, frankly.

> > That's were I hoped your project could help. I thought, if I pull in all the
> > needed changelists (selected by path/CL), there may be a way to
> > recreate a mergeable history out of this dump. At least, one involving less
> > labor then I have to do now.
> 
> Yeah... well the design of my tool is that it needs to have the perforce
> internal information.  But really you can probably consider the
> converter orphaned, I have no current need to work on it; it served a
> purpose, which was converting perl's perforce to perl.git, and that's
> history now.

Thanks anyway for bringing it to public attention. At least, it is
known to be done once. Could be an encouragement for next one to try.
A guide, even.

Re: From P4 to Git

[Sam Vilain]

Alex Riesen wrote:
>> The "live" parts are never touched - only the write-only files that
>> perforce writes; and the rcs files are read using rcs.
>>     
>
> I always felt Perforce had a touch of pure evil...
>   

Perforce is not really that evil imho. It uses rcs for what it's good at
- storing file revisions. All the metadata it kept in table stores which
could be checkpointed and journalled like a regular database. Simple,
elegant, fast, robust. Overall I'd say it was a well engineered piece of
software which delivered great benefits to its users, far less evil than
CVS or Subversion. Which is funny because Subversion rips off many
concepts from Perforce including the 'branching is just copying' idea.
But they left behind the integration system. Even now svn's merging
support can't even handle merging correctly in the face of branched
branches, or so I hear. That's evil - pretending to have a feature
solved when they haven't even met the standard set by bzr, hg, mtn, git,
etc.

Sam

Re: From P4 to Git

[Arnaud Bailly]

Thanks all for your detailed answers.

Playing with git on various environments, I have a couple more questions
that I could not (quickly) find answers on the web.
 1. when checkout-ing on NTFS, I got all my files marked as changed
 because of file permissions set from 100644 to 100755. I see there is a
 config flag about file permissions (core.sharedRepository ?) but not
 sure how to use it
 2. I am working with NTFS on a truecrypt managed USB HD and I noticed
 an order of magnitude difference in speed (ie. about 10x for checkout
 of 50k files) between this and my ext3 FS  on laptop HD. Is this
 expected ? Some people here reported slowness on non-encrypted NTFS and
 I think I read a couple of web fragments about speed difference between
 the 2. WDYT ?

Regards,
-- 
Arnaud Bailly -- OQube
<software engineering>
http://www.oqube.com/

Re: From P4 to Git

[Peter Baumann]

On Tue, Aug 04, 2009 at 02:31:42PM +0200, Arnaud Bailly wrote:
> Thanks all for your detailed answers.
> 
> Playing with git on various environments, I have a couple more questions
> that I could not (quickly) find answers on the web.
>  1. when checkout-ing on NTFS, I got all my files marked as changed
>  because of file permissions set from 100644 to 100755. I see there is a
>  config flag about file permissions (core.sharedRepository ?) but not
>  sure how to use it

No, that config flag is for sharing a repository with other people. You want
something like core.filemode = false

-Peter