[git-p4] Trouble importing all perforce branches into git repository

[git-p4] Trouble importing all perforce branches into git repository

[Grant Limberg]

[-- Attachment #1: Type: text/plain, Size: 4052 bytes --]

I'm trying to import a project from my company's Perforce depot into a git repository. I've been successful getting a single branch into a single git repo, but what I'd like to do is to pull in the entire project plus all branches, but I can't get it working correctly.

I have my .p4settings file:

> P4PORT=perforce-server.local:1666
> P4USER=my.username
> P4CLIENT=my.clientspec.name

Clientspec is fairly simple:

> //project/... //my.clientspec.name/project/...
> -//project/External/... //my.clientspec.name/project/External/...

The second line is to exclude the external library binaries from the checkout.

I use the following command line to start git-p4 importing:

> git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all
> 
This goes through and checks all the branch names, etc, and attempts to import starting at the first commit (in this case its changelists 771 for this project).

> Importing from //project/...@all into project
> Initialized empty Git repository in /Users/grant/Development/git_test/project/.git/
> Reading pipe: git config git-p4.useclientspec
> Reading pipe: git config git-p4.user
> Reading pipe: git config git-p4.password
> Reading pipe: git config git-p4.port
> Reading pipe: git config git-p4.host
> Reading pipe: git config git-p4.client
> p4 -G branches
> Opening pipe: p4 -G branches
> p4 -G branch -o branch-1
> Opening pipe: p4 -G branch -o branch-1
> Reading pipe: git config --bool core.ignorecase
> p4 -G branch -o branch-2
> ...
> Opening pipe: p4 -G branch -o branch-n
> p4-git branches: []
> initial parents: {}

I'm not sure if p4-git branches, and initial parents are supposed to be empty here, but they are.

Finally we get to the point where it's about to start importing changelists into git and the following happens:

> Getting p4 changes for //project/...
> p4 changes //project/...
> Reading pipe: p4 changes //project/...
> p4 -G describe 771
> Opening pipe: p4 -G describe 771
> Importing revision 771 (1%)branch is MAIN
> 
>     Importing new branch RCMerge/MAIN
> p4 changes //RCMerge/MAIN/...@1,770
> Reading pipe: p4 changes //RCMerge/MAIN/...@1,770
> 
>     Resuming with change 771
> parent determined through known branches: Branch-foo
> looking for initial parent for refs/remotes/p4/project/MAIN; current parent is refs/remotes/p4/project/Branch-foo
> commit into refs/remotes/p4/project/MAIN
> parent refs/remotes/p4/project/Branch-foo
> p4 -G -x - print
> Opening pipe: p4 -G -x - print
> Glue/source/.empty
> fatal: Invalid ref name or SHA1 expression: refs/remotes/p4/project/Branch-foo
> fast-import: dumping crash report to .git/fast_import_crash_26002

Here's the file referenced above:

> fast-import crash report:
>     fast-import process: 26002
>     parent process     : 26000
>     at Thu May 19 11:51:54 2011
> 
> fatal: Invalid ref name or SHA1 expression: refs/remotes/p4/project/Branch-foo
> 
> Most Recent Commands Before Crash
> ---------------------------------
>   checkpoint
>   commit refs/remotes/p4/project/MAIN
>   committer Some User <some.user@domain.com> 1253574589 -0800
>   data <<EOT
> * from refs/remotes/p4/project/Branch-foo
> 
> Active Branch LRU
> -----------------
>     active_branches = 0 cur, 5 max
> 
>   pos  clock name
>   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Inactive Branches
> -----------------
> refs/remotes/p4/project/MAIN:
>   status      : dirty
>   tip commit  : 0000000000000000000000000000000000000000
>   old tree    : 0000000000000000000000000000000000000000
>   cur tree    : 0000000000000000000000000000000000000000
>   commit clock: 0
>   last pack   : 
> 
> 
> Marks
> -----
> 
> -------------------
> END OF CRASH REPORT

Now, being a perforce novice, I have no idea what this means and have no idea how to resolve this issue, or if its even possible. Has anyone run into a similar issue before? If so, how did you resolve it?


Grant Limberg
glimberg@gmail.com




[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 881 bytes --]

Re: [git-p4] Trouble importing all perforce branches into git repository

[Pete Wyckoff]

glimberg@gmail.com wrote on Thu, 19 May 2011 19:32 +0000:
> [git-p4 automatic branch detection]
> > git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all

I worry that --max-changes prevents it from pulling in the
ref for Branch-foo, and this confuses the automatic branch
detection that assumes it has all its parents.

It would certainly be more reliable if you did "git-p4 sync
--branch foo //project/foo@all" for each branch you care about.

Vitor has some modifications to auto-branch detection that might
be useful, or maybe some thoughts as to what's going on.

		-- Pete

(Vitor: full mail is here:
http://article.gmane.org/gmane.comp.version-control.git/173996/ )

Re: [git-p4] Trouble importing all perforce branches into git repository

[Grant Limberg]

[-- Attachment #1: Type: text/plain, Size: 1167 bytes --]

On May 22, 2011, at 4:49 AM, Pete Wyckoff wrote:

> glimberg@gmail.com wrote on Thu, 19 May 2011 19:32 +0000:
>> [git-p4 automatic branch detection]
>>> git p4 clone --verbose --detect-branches --max-changes=100 //project/...@all
> 
> I worry that --max-changes prevents it from pulling in the
> ref for Branch-foo, and this confuses the automatic branch
> detection that assumes it has all its parents.
> 
> It would certainly be more reliable if you did "git-p4 sync
> --branch foo //project/foo@all" for each branch you care about.
> 
> Vitor has some modifications to auto-branch detection that might
> be useful, or maybe some thoughts as to what's going on.
> 
> 		-- Pete
> 
> (Vitor: full mail is here:
> http://article.gmane.org/gmane.comp.version-control.git/173996/ )
> 

Removing --max-changes=100 has no effect on the crash I'm experiencing.   Same crash still happens on the very first imported commit.

If i do "git-p4 sync --branch foo //project/foo@all" for each branch, does that convert 'p4 integrate' commands into git merges?  I"m trying to keep all history intact when importing from p4 into git.

Thanks,

-Grant


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 881 bytes --]

Re: [git-p4] Trouble importing all perforce branches into git repository

[Vitor Antunes]

Hi Grant,

I think I saw that happening when there is nothing to import or in
situations where the parent of a certain branch does not exist.

Perforce does not strictly require branch definitions to allow a branch
structure/flow. It is quite possible that you are integrating stuff
around without using "branch specs". When git-p4 is importing your P4
database it will look at all branches available in the server, including
ones from other depots. So it is possible that while you are seeing a
big list of branches being processed, none apply to the branch structure
you are trying to import.

Could you please confirm that you have Perforce branch specs for all
branches that you are trying to import?

Thanks,
Vitor

Re: [git-p4] Trouble importing all perforce branches into git repository

[Grant Limberg]

[-- Attachment #1: Type: text/plain, Size: 1205 bytes --]

On May 23, 2011, at 3:54 AM, Vitor Antunes wrote:

> Hi Grant,
> 
> I think I saw that happening when there is nothing to import or in
> situations where the parent of a certain branch does not exist.
> 
> Perforce does not strictly require branch definitions to allow a branch
> structure/flow. It is quite possible that you are integrating stuff
> around without using "branch specs". When git-p4 is importing your P4
> database it will look at all branches available in the server, including
> ones from other depots. So it is possible that while you are seeing a
> big list of branches being processed, none apply to the branch structure
> you are trying to import.
> 
> Could you please confirm that you have Perforce branch specs for all
> branches that you are trying to import?
> 
> Thanks,
> Vitor

Vitor, 

The "Branch-foo" that git-p4 is crashing on does have a branchspec in Perforce.  I cannot, however guarantee that branchspecs were correctly created and used for all integrates in the repository.  It was created long before I started working here.  Is there a way to limit --detect-branches to only certain branches that I can ensure do have branchspecs?

- Grant

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 881 bytes --]

Re: [git-p4] Trouble importing all perforce branches into git repository

[Vitor Antunes]

Hi Grant,

> The "Branch-foo" that git-p4 is crashing on does have a branchspec in
> Perforce.  I cannot, however guarantee that branchspecs were
> correctly created and used for all integrates in the repository.  It
> was created long before I started working here.

You can create the branch specs now. There is no requirement for them to
have been created at the time of the integration.

I've sent a patch to this mailing list [1] that adds the possibility of
creating these branches in a gitconfig file. If you feel brave enough,
please give it a try. The patch is missing a test case, which I still
did not have time to review and send

> Is there a way to limit --detect-branches to only certain branches
> that I can ensure do have branchspecs?

If you have an incremental structure you could limit the cloning to a
subset of the directories. I am not aware of any other way. Maybe
someone else in the mailing list can shed some light...? :)

Vitor

[1] http://thread.gmane.org/gmane.comp.version-control.git/167998/focus=168000

Re: [git-p4] Trouble importing all perforce branches into git repository

[Grant Limberg]

[-- Attachment #1: Type: text/plain, Size: 1678 bytes --]

On May 23, 2011, at 10:38 AM, Vitor Antunes wrote:

> Hi Grant,
> 
>> The "Branch-foo" that git-p4 is crashing on does have a branchspec in
>> Perforce.  I cannot, however guarantee that branchspecs were
>> correctly created and used for all integrates in the repository.  It
>> was created long before I started working here.
> 
> You can create the branch specs now. There is no requirement for them to
> have been created at the time of the integration.
> 
> I've sent a patch to this mailing list [1] that adds the possibility of
> creating these branches in a gitconfig file. If you feel brave enough,
> please give it a try. The patch is missing a test case, which I still
> did not have time to review and send
> 
>> Is there a way to limit --detect-branches to only certain branches
>> that I can ensure do have branchspecs?
> 
> If you have an incremental structure you could limit the cloning to a
> subset of the directories. I am not aware of any other way. Maybe
> someone else in the mailing list can shed some light...? :)
> 
> Vitor
> 
> [1] http://thread.gmane.org/gmane.comp.version-control.git/167998/focus=168000

I think I just figured out the problem.  The branch mapping between //project/MAIN/... and //project/Branch-foo/... was done backwards, thus git-p4 was expecting Branch-foo to be a parent of MAIN when the inverse was actually true.  MAIN's first changelist number was 771.  Branch-foo's first changelist number was 7652.  That's why git-p4 was getting confused.  I fixed the branch mapping in Perforce and the clone with --detect-branches seems to be working fine now. 

Grant Limberg
glimberg@gmail.com





[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 881 bytes --]

Re: [git-p4] Trouble importing all perforce branches into git repository

[tumik]

Grant Limberg wrote:
> 
> I think I just figured out the problem.  The branch mapping between
> //project/MAIN/... and //project/Branch-foo/... was done backwards, thus
> git-p4 was expecting Branch-foo to be a parent of MAIN when the inverse
> was actually true.
> 

Did you get it working so that the branch integrates at Perforce show up as
merges in git? I have been trying to do this for tens of hours now and just
can't get it working.. I have tried making the branch mapping both ways, and
even both at the same time but with no success.

I have succesfully imported the two testing branches to git, but if I look
at it in gitk for example, the history is linear and there are no merges
from the other branch (it shows up as a normal commit, without the parent
from other branch, just adding all lines that the merging did). in p4v it
shows up as it should!

--
View this message in context: http://git.661346.n2.nabble.com/git-p4-Trouble-importing-all-perforce-branches-into-git-repository-tp6383422p6482174.html
Sent from the git mailing list archive at Nabble.com.

Re: [git-p4] Trouble importing all perforce branches into git repository

[Thomas Berg]

On Thu, Jun 16, 2011 at 10:28 AM, tumik <tumik@tuomisalo.com> wrote:
> Did you get it working so that the branch integrates at Perforce show up as
> merges in git? I have been trying to do this for tens of hours now and just
> can't get it working.. I have tried making the branch mapping both ways, and
> even both at the same time but with no success.
>
> I have succesfully imported the two testing branches to git, but if I look
> at it in gitk for example, the history is linear and there are no merges
> from the other branch (it shows up as a normal commit, without the parent
> from other branch, just adding all lines that the merging did). in p4v it
> shows up as it should!

git-p4 only uses the branch mapping for the initial commit on a
branch. It does not attempt to detect merges after that point. After
importing to git, merges just appear as regular commits in the
history.

Unfortunately, implementing a general solution for this is
non-trivial. Perforce branches and merges per file, meaning that
partial merges are possible in Perforce. Such states can not be
correctly represented in Git, and depending on your needs, you may or
may not want a particular partial merge to be considered as a git
merge.

There was another script called git-p4raw for importing Perforce
history to git, which does merge detection. It solved the problem by
having configurable rules for what should be regarded as a merge, and
it supported manually reviewing the decisions afterwards (iirc).

One possibly useful feature in git-p4 would be an option to regard
every integration as a complete merge, which shouldn't be too hard to
implement. But this would break every time someone did a single-file
integration between two branches (which happens quite often in my
experience, for cherry-picking changes etc.).

Thomas

Re: [git-p4] Trouble importing all perforce branches into git repository

[tumik]

Thomas Berg wrote:
> 
> git-p4 only uses the branch mapping for the initial commit on a
> branch. It does not attempt to detect merges after that point. After
> importing to git, merges just appear as regular commits in the
> history.
> 
Thank you for the info, that's the way it seems to be working! It's a shame
if the cloning from perforce -> git can't be made succesfully keeping all
the history information :(


Thomas Berg wrote:
> 
> There was another script called git-p4raw for importing Perforce
> history to git, which does merge detection. It solved the problem by
> having configurable rules for what should be regarded as a merge, and
> it supported manually reviewing the decisions afterwards (iirc).
> 
Yes, I tried the git-p4raw too. Actually even several forks of it! But I
wasn't able to get any of those working, as none of them could load the
perforce database correctly. Maybe the perforce database has changed lately
or something, as git-p4raw gave errors that it's database tables don't have
as much columns as perforce's does. Tried also to change the code to add
those columns but there were so many of them and in many tables so I decided
it's too hard and time-consuming to start reverse-engineering them..

- tumik

--
View this message in context: http://git.661346.n2.nabble.com/git-p4-Trouble-importing-all-perforce-branches-into-git-repository-tp6383422p6482333.html
Sent from the git mailing list archive at Nabble.com.

Re: [git-p4] Trouble importing all perforce branches into git repository

[Thomas Berg]

On Thu, Jun 16, 2011 at 11:35 AM, tumik <tumik@tuomisalo.com> wrote:
> Yes, I tried the git-p4raw too. Actually even several forks of it! But I
> wasn't able to get any of those working, as none of them could load the
> perforce database correctly. Maybe the perforce database has changed lately
> or something, as git-p4raw gave errors that it's database tables don't have
> as much columns as perforce's does. Tried also to change the code to add
> those columns but there were so many of them and in many tables so I decided
> it's too hard and time-consuming to start reverse-engineering them..
>
> - tumik

I could maybe mention another approach: if your Perforce history isn't
too large so you can extract the info manually, or if you are able to
create a script to figure out your merges (changelist number, revision
of source and target branch in Perforce) it is easy to modify the git
history with this info afterwards.

You can add merge arrows to your git-p4 imported repository, either by
creating a .git/info/graft file and running git-filter-branch
afterwards (to make it permanent and distributable), or by using
git-replace. I may have a script somewhere that generates the graft
file from p4 changelist numbers and branch names, if you're
interested.

The graft + git-filter-branch is good for a one-time conversion, but
since it rewrites history it may not work well for branches that still
get updated in Perforce.

I have not tested git-replace myself, but it should be possible to use
this to add merge arrows too, without modifying the hashes of commits
later in the history. I'm not sure how well git-replace works with the
rest of git though.

Thomas