git-p4 workflow suggestions?

git-p4 workflow suggestions?

[Sam Hocevar]

   Dear list,

   I have modified git and git-p4 to a point where they are usable in
my work environment. I am now faced with a new problem: Perforce's
composite workspaces. They allow you to "mount" parts of the repo onto
other directories, even nonempty ones.

   Take the following example repository, where a "framework" project
contains an example subdirectory with build files and other directories,
and a "project1" project contains subdirectories that are meant to
replace the ones in "example":

   //work/framework/example/src/
                           /include/
			   /Makefile
			   /...
   //work/project1/src/
                  /include/

   Using the Perforce client, one can reorganise the workspace
so that project1/src/ transparently replaces the contents of
framework/example/src/ (same for */include). All the work is then done
in the framework/ local checkout, but commits may also affect project1/.

   I could not find a way to do the same with git and git-p4. My main
requirements are the following:

   - preserve the atomicity of commits affecting both project1/src and
   project1/include (having to commit from a different directory due to
   symlink hacks is acceptable to me, even if not terribly practical)

   - have git-p4 rebase work without requiring tedious merges

   - *if possible* (but not a strong requirement), preserve the
   atomicity of commits affecting both framework/ and project1/.

   If anyone ever ran into the problem, I'd like to hear from their
experience. Or maybe someone will have suggestions based on similar
requirements.

Cheers,
-- 
Sam.

Re: git-p4 workflow suggestions?

[Christian Couder]

Hi Sam,

Le lundi 9 mars 2009, Sam Hocevar a écrit :
>    Dear list,
>
>    I have modified git and git-p4 to a point where they are usable in
> my work environment. I am now faced with a new problem: Perforce's
> composite workspaces. They allow you to "mount" parts of the repo onto
> other directories, even nonempty ones.

It looks like SVN externals. So I think you should read about "git 
submodule".

There is this related link on the wiki:

http://blog.alieniloquent.com/2008/03/08/git-svn-with-svnexternals/

You may also want to search the mailing list as this subject has often been 
discussed.

Best regards,
Christian.

Re: git-p4 workflow suggestions?

[Sam Hocevar]

On Tue, Mar 10, 2009, Christian Couder wrote:
> >    I have modified git and git-p4 to a point where they are usable in
> > my work environment. I am now faced with a new problem: Perforce's
> > composite workspaces. They allow you to "mount" parts of the repo onto
> > other directories, even nonempty ones.
> 
> It looks like SVN externals. So I think you should read about "git 
> submodule".
> 
> There is this related link on the wiki:
> 
> http://blog.alieniloquent.com/2008/03/08/git-svn-with-svnexternals/

   Unfortunately submodules are considered separate repositories, so if
I have /include and /src as submodules, I cannot commit atomically to
both. Or can I? That's probably my strongest requirement.

> You may also want to search the mailing list as this subject has often been 
> discussed.

   I did skim through the archives, but couldn't find much. There was
this discussion: http://kerneltrap.org/mailarchive/git/2006/11/28/231515
where the idea of Perforce-like workspaces was apparently dismissed as
being "a mess".

Cheers,
-- 
Sam.

Re: git-p4 workflow suggestions?

[Christian Couder]

Le mardi 10 mars 2009, Sam Hocevar a écrit :
> On Tue, Mar 10, 2009, Christian Couder wrote:
> > >    I have modified git and git-p4 to a point where they are usable in
> > > my work environment. I am now faced with a new problem: Perforce's
> > > composite workspaces. They allow you to "mount" parts of the repo
> > > onto other directories, even nonempty ones.
> >
> > It looks like SVN externals. So I think you should read about "git
> > submodule".
> >
> > There is this related link on the wiki:
> >
> > http://blog.alieniloquent.com/2008/03/08/git-svn-with-svnexternals/
>
>    Unfortunately submodules are considered separate repositories, so if
> I have /include and /src as submodules, I cannot commit atomically to
> both. Or can I? That's probably my strongest requirement.

I thought that you can commit in /include then in /src and then in the 
supermodule. This way you should have an atomic commit in the supermodule 
with both changes. But I don't know much about submodules.

> > You may also want to search the mailing list as this subject has often
> > been discussed.
>
>    I did skim through the archives, but couldn't find much. There was
> this discussion: http://kerneltrap.org/mailarchive/git/2006/11/28/231515
> where the idea of Perforce-like workspaces was apparently dismissed as
> being "a mess".

Perhaps. Anyway I think there are patches about submodules quite often on 
the list and it seems a hot topic as there is an entry on the GSoC2009 idea 
page:

http://git.or.cz/gitwiki/SoC2009Ideas

On this page you may also be interested by the "Narrow and Sparse clone 
support" idea.

Best regards,
Christian.

PS: it would be nice to put me in the CC header of your emails when you 
reply to me.

Re: git-p4 workflow suggestions?

[Pete Wyckoff]

sam@zoy.org wrote on Mon, 09 Mar 2009 15:21 +0100:
>    I have modified git and git-p4 to a point where they are usable in
> my work environment. I am now faced with a new problem: Perforce's
> composite workspaces. They allow you to "mount" parts of the repo onto
> other directories, even nonempty ones.
> 
>    Take the following example repository, where a "framework" project
> contains an example subdirectory with build files and other directories,
> and a "project1" project contains subdirectories that are meant to
> replace the ones in "example":
> 
>    //work/framework/example/src/
>                            /include/
> 			   /Makefile
> 			   /...
>    //work/project1/src/
>                   /include/

In perforce terms, your "view mapping" looks like:

    //work/framework/example/src/... //client/src/...
    //work/project1/src/include/...  //client/src/include/...

?

I'm not a pro with p4, but do deal with many-line mappings like
this.  Stock git-p4 handles these, except doesn't map correctly to
the right-hand side.  I haven't tried to see if it would correctly
use the include files from project1 instead of framework in your
example.

If you can get git-p4 to figure out the mapping correctly, I don't
expect any problems with respect to atomicity of commits.  As far as
perforce goes, a server seems to manage its entire p4 space as one
big single project.  Similarly with the git side of things---it's
just a matter of getting this mapping correct.

I too hacked the getClientSpec() part of git-p4 to put files into
the correct directories in the git side.  My changes are a bit
messy, and may interfere with other usage models, hence not
submitted.  Maybe we should make an effort to get this right though.
Do you have any changes to show how you are modifying things?

		-- Pete

Re: git-p4 workflow suggestions?

[Sam Hocevar]

On Wed, Mar 11, 2009, Pete Wyckoff wrote:
> sam@zoy.org wrote on Mon, 09 Mar 2009 15:21 +0100:
> >    I have modified git and git-p4 to a point where they are usable in
> > my work environment. I am now faced with a new problem: Perforce's
> > composite workspaces. They allow you to "mount" parts of the repo onto
> > other directories, even nonempty ones.
> > 
> >    Take the following example repository, where a "framework" project
> > contains an example subdirectory with build files and other directories,
> > and a "project1" project contains subdirectories that are meant to
> > replace the ones in "example":
> > 
> >    //work/framework/example/src/
> >                            /include/
> > 			   /Makefile
> > 			   /...
> >    //work/project1/src/
> >                   /include/
> 
> In perforce terms, your "view mapping" looks like:
> 
>     //work/framework/example/src/... //client/src/...
>     //work/project1/src/include/...  //client/src/include/...

   Yes, like this. More precisely:

    //work/framework/example/... //client/...
    //work/project1/src/...  //client/src/...
    //work/project1/include/...  //client/include/...

> I'm not a pro with p4, but do deal with many-line mappings like
> this.  Stock git-p4 handles these, except doesn't map correctly to
> the right-hand side.  I haven't tried to see if it would correctly
> use the include files from project1 instead of framework in your
> example.

   No luck here. If I clone //work with git-p4, I get two separate
/framework and /project1 directories, and the mapping is not done.

   The "solution" I found so far was to clone //work and hack git-p4
so that it ignores //work/framework/example/src, and then symlink
//work/project1/src to //work/framework/example/src. This allows me to
pull changes with a single "git-p4 rebase" command. Unfortunately it
also requires me to clone a full, separate //work p4 workspace in order
to use "git-p4 submit" later, and that's more than 120 GiB wasted.

> If you can get git-p4 to figure out the mapping correctly, I don't
> expect any problems with respect to atomicity of commits.  As far as
> perforce goes, a server seems to manage its entire p4 space as one
> big single project.  Similarly with the git side of things---it's
> just a matter of getting this mapping correct.
> 
> I too hacked the getClientSpec() part of git-p4 to put files into
> the correct directories in the git side.  My changes are a bit
> messy, and may interfere with other usage models, hence not
> submitted.  Maybe we should make an effort to get this right though.
> Do you have any changes to show how you are modifying things?

   I'm curious to see your changes. I don't feel I completely understand
the p4 way to do things yet.

   My changes are extremely messy but I will refactor them as time goes.
There is at least one other important thing my git-p4 does, which is not
storing the whole commit in memory. Combined with the patches I sent
last week to this list, it's the only way I can import the p4 repository
we have at work. (See http://zoy.org/~sam/git/clumsily-hacked-git-p4)

   Feel free to contact me in private if you have questions or want
information that may not be mailing-list relevant. I'm all for cleaning
up things and getting a fully featured git-p4. I'm on that project for
at least three years, and there is absolutely no way my blood pressure
can stand that long with Perforce.

Cheers,
-- 
Sam.

Re: git-p4 workflow suggestions?

[Pete Wyckoff]

sam@zoy.org wrote on Mon, 16 Mar 2009 19:01 +0100:
>    Yes, like this. More precisely:
> 
>     //work/framework/example/... //client/...
>     //work/project1/src/...  //client/src/...
>     //work/project1/include/...  //client/include/...
[..]
>    My changes are extremely messy but I will refactor them as time goes.
> There is at least one other important thing my git-p4 does, which is not
> storing the whole commit in memory. Combined with the patches I sent
> last week to this list, it's the only way I can import the p4 repository
> we have at work. (See http://zoy.org/~sam/git/clumsily-hacked-git-p4)

Oh, that is, ahem, a bit site-specific.  The looping over chunks in
the import phase is important, and other people have been thinking
about that.  I'll ignore that aspect for now, though, and we'll see
if we can get the client-spec part worked out.

Can you take a look at the attached.  Its goal is purely to allow
you to clone a complex spec like yours above.  You may have to merge
this in with your perrformance changes.

Edit your ~/.gitconfig to add a section:

	[git-p4]
		useClientSpec = true

Then copy a good P4ENV from a p4 client that has a client spec
checked out as you like.  git-p4 clone will do "p4 client -o",
reading that spec, and use the results to import, hopefully as
you have things laid out in the spec.

If this seems to work for you, we can figure out how to clean up
the patch so it can be used generally by people with and without
client specs.

		-- Pete

---------
From 4a922efcef18e3bb740f88c31ed4d00fa66f4cce Mon Sep 17 00:00:00 2001
From: Pete Wyckoff <petew@netapp.com>
Date: Wed, 26 Nov 2008 12:28:09 -0500
Subject: [PATCH] honor git client spec

Destination directories for parts of the depot are specified in the
client spec.  Use them as given.  Also read the entire client spec to
figure out what to do.
---
 contrib/fast-import/git-p4 |  110 ++++++++++++++++++++++++++++++++++---------
 1 files changed, 87 insertions(+), 23 deletions(-)

diff --git a/contrib/fast-import/git-p4 b/contrib/fast-import/git-p4
index 332c7f8..d953a1e 100755
--- a/contrib/fast-import/git-p4
+++ b/contrib/fast-import/git-p4
@@ -451,6 +451,26 @@ def p4ChangesForPaths(depotPaths, changeRange):
     changelist.sort()
     return changelist
 
+#
+# Sort by number of slashes first:  more specific at the top.  Then
+# sort by alpha within a given number of path components.
+#
+def clientSortFunc(a, b):
+    asrc = a[0]
+    bsrc = b[0]
+    asrclen = asrc.count("/")
+    bsrclen = bsrc.count("/")
+    if asrclen > bsrclen:
+	return -1
+    elif asrclen < bsrclen:
+	return 1
+    elif asrc > bsrc:
+	return 1
+    elif asrc < bsrc:
+	return -1
+    else:
+	return 0
+
 class Command:
     def __init__(self):
         self.usage = "usage: %prog [options]"
@@ -959,8 +979,9 @@ class P4Sync(Command):
             includeFile = True
             for val in self.clientSpecDirs:
                 if f['path'].startswith(val[0]):
-                    if val[1] <= 0:
+                    if val[1] == '-':
                         includeFile = False
+		    f['pathmap'] = val
                     break
 
             if includeFile:
@@ -1056,7 +1077,12 @@ class P4Sync(Command):
                 print "\nfile %s is a strange apple file that forks. Ignoring!" % file['path']
                 continue
 
-            relPath = self.stripRepoPath(file['path'], branchPrefixes)
+	    if 'pathmap' in file:
+		relPath = file['pathmap'][1] + file['path'][len(file['pathmap'][0]):]
+	    else:
+		relPath = self.stripRepoPath(file['path'], branchPrefixes)
+
+	    print "path", file['path'], "->", relPath
             if file["action"] in ("delete", "purge"):
                 self.gitStream.write("D %s\n" % relPath)
             else:
@@ -1439,24 +1465,54 @@ class P4Sync(Command):
 
 
     def getClientSpec(self):
-        specList = p4CmdList( "client -o" )
+        specList = p4CmdList("client -o")
         temp = {}
+	client = ""
+        for entry in specList:
+            for k,v in entry.iteritems():
+		if k.startswith("Client"):
+		    client = v
+		    print "client is", client
+	if not client:
+	    sys.stderr.write("no client found\n")
+	    sys.exit(1)
+	client = "//" + client + "/"
         for entry in specList:
             for k,v in entry.iteritems():
                 if k.startswith("View"):
-                    if v.startswith('"'):
-                        start = 1
-                    else:
-                        start = 0
-                    index = v.find("...")
-                    v = v[start:index]
-                    if v.startswith("-"):
-                        v = v[1:]
-                        temp[v] = -len(v)
-                    else:
-                        temp[v] = len(v)
-        self.clientSpecDirs = temp.items()
-        self.clientSpecDirs.sort( lambda x, y: abs( y[1] ) - abs( x[1] ) )
+		    if v.startswith('"'):
+			v = v[1:]
+		    if v.endswith('"'):
+			v = v[:-1]
+		    d = v.split(" ");
+		    if len(d) != 2:
+			sys.stderr.write( \
+			    "expecting two fields in view, got: %s\n" % v)
+			sys.exit(1)
+		    if not d[0].endswith("..."):
+			sys.stderr.write(\
+			    "expecting trailing ..., got: %s\n" % d[0])
+			sys.exit(1)
+		    d[0] = d[0][:-3]
+		    if not d[1].endswith("..."):
+			sys.stderr.write(\
+			    "expecting trailing ..., got: %s\n" % d[1])
+			sys.exit(1)
+		    d[1] = d[1][:-3]
+		    if not d[1].startswith(client):
+			sys.stderr.write(\
+			    "expecting dest to start with %s, got: %s\n" % \
+			    (client, d[1]))
+			sys.exit(1)
+		    d[1] = d[1][len(client):]
+		    # negated items do not appear in tree
+		    if d[0].startswith("-"):
+			d[0] = d[0][1:]
+			d[1] = ""
+		    temp[d[0]] = d[1]
+
+	self.clientSpecDirs = temp.items()
+	self.clientSpecDirs.sort(clientSortFunc)
 
     def run(self, args):
         self.depotPaths = []
@@ -1718,7 +1774,7 @@ class P4Clone(P4Sync):
     def __init__(self):
         P4Sync.__init__(self)
         self.description = "Creates a new git repository and imports from Perforce into it"
-        self.usage = "usage: %prog [options] //depot/path[@revRange]"
+        self.usage = "usage: %prog [options] [//depot/path[@revRange]]"
         self.options += [
             optparse.make_option("--destination", dest="cloneDestination",
                                  action='store', default=None,
@@ -1746,18 +1802,26 @@ class P4Clone(P4Sync):
         return os.path.split(depotDir)[1]
 
     def run(self, args):
-        if len(args) < 1:
-            return False
-
         if self.keepRepoPath and not self.cloneDestination:
             sys.stderr.write("Must specify destination for --keep-path\n")
             sys.exit(1)
 
         depotPaths = args
 
-        if not self.cloneDestination and len(depotPaths) > 1:
-            self.cloneDestination = depotPaths[-1]
-            depotPaths = depotPaths[:-1]
+	if gitConfig("git-p4.useclientspec") == "true":
+	    if not os.path.exists("P4ENV"):
+		sys.stderr.write("Must copy P4ENV file from a valid client\n")
+		sys.exit(1)
+	    self.getClientSpec()
+	    if not depotPaths:
+		depotPaths = [p[0] for p in self.clientSpecDirs]
+	else:
+	    if not depotPaths:
+		sys.stderr.write("Must specify depot path if no client spec\n")
+		sys.exit(1)
+
+        if not self.cloneDestination:
+		self.cloneDestination = "."
 
         self.cloneExclude = ["/"+p for p in self.cloneExclude]
         for p in depotPaths:
-- 
1.6.0.6

Re: git-p4 workflow suggestions?

[Sam Hocevar]

On Tue, Mar 17, 2009, Pete Wyckoff wrote:

> Can you take a look at the attached.  Its goal is purely to allow
> you to clone a complex spec like yours above.  You may have to merge
> this in with your perrformance changes.
> 
> Edit your ~/.gitconfig to add a section:
> 
> 	[git-p4]
> 		useClientSpec = true
> 
> Then copy a good P4ENV from a p4 client that has a client spec
> checked out as you like.  git-p4 clone will do "p4 client -o",
> reading that spec, and use the results to import, hopefully as
> you have things laid out in the spec.
> 
> If this seems to work for you, we can figure out how to clean up
> the patch so it can be used generally by people with and without
> client specs.

   I am afraid I don't know what a "P4ENV" is. However, our P4
repository is set up in such a way that "p4 client -o" shows the
expected mappings, so I just commented out the P4ENV check and I finally
managed to clone my gigantic repository.

   One concern: git-p4 clone creates .git in the current directory and
it caused me to do at least one unfortunate "rm -rf .git". I would
expect clone to create a subdirectory.

   Apart from that, "clone" seems to work rather well. I haven't tried
to submit a commit yet, though.

-- 
Sam.