[PATCH v3] git-p4: recover from inconsistent perforce history

[PATCH v3] git-p4: recover from inconsistent perforce history

[Andrew Oakley]

Perforce allows you commit files and directories with the same name, so
you could have files //depot/foo and //depot/foo/bar both checked in.  A
p4 sync of a repository in this state fails.  Deleting one of the files
recovers the repository.

When this happens we want git-p4 to recover in the same way as perforce.

Signed-off-by: Andrew Oakley <andrew@adoakley.name>
---
 git-p4.py                      | 43 ++++++++++++++++++++-
 t/t9834-git-p4-file-dir-bug.sh | 70 ++++++++++++++++++++++++++++++++++
 2 files changed, 111 insertions(+), 2 deletions(-)
 create mode 100755 t/t9834-git-p4-file-dir-bug.sh

diff --git a/git-p4.py b/git-p4.py
index b8b2a1679e..d551efb0dd 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -3214,6 +3214,42 @@ def hasBranchPrefix(self, path):
             print('Ignoring file outside of prefix: {0}'.format(path))
         return hasPrefix
 
+    def findShadowedFiles(self, files, change):
+        # Perforce allows you commit files and directories with the same name,
+        # so you could have files //depot/foo and //depot/foo/bar both checked
+        # in.  A p4 sync of a repository in this state fails.  Deleting one of
+        # the files recovers the repository.
+        #
+        # Git will not allow the broken state to exist and only the most recent
+        # of the conflicting names is left in the repository.  When one of the
+        # conflicting files is deleted we need to re-add the other one to make
+        # sure the git repository recovers in the same way as perforce.
+        deleted = [f for f in files if f['action'] in self.delete_actions]
+        to_check = set()
+        for f in deleted:
+            path = decode_path(f['path'])
+            to_check.add(path + '/...')
+            while True:
+                path = path.rsplit("/", 1)[0]
+                if path == "/" or path in to_check:
+                    break
+                to_check.add(path)
+        to_check = ['%s@%s' % (wildcard_encode(p), change) for p in to_check
+            if self.hasBranchPrefix(p)]
+        if to_check:
+            stat_result = p4CmdList(["-x", "-", "fstat", "-T",
+                "depotFile,headAction,headRev,headType"], stdin=to_check)
+            for record in stat_result:
+                if record['code'] != 'stat':
+                    continue
+                if record['headAction'] in self.delete_actions:
+                    continue
+                files.append({
+                    'action': 'add',
+                    'path': record['depotFile'],
+                    'rev': record['headRev'],
+                    'type': record['headType']})
+
     def commit(self, details, files, branch, parent = "", allow_empty=False):
         epoch = details["time"]
         author = details["user"]
@@ -3222,11 +3258,14 @@ def commit(self, details, files, branch, parent = "", allow_empty=False):
         if self.verbose:
             print('commit into {0}'.format(branch))
 
+        files = [f for f in files
+            if self.hasBranchPrefix(decode_path(f['path']))]
+        self.findShadowedFiles(files, details['change'])
+
         if self.clientSpecDirs:
             self.clientSpecDirs.update_client_spec_path_cache(files)
 
-        files = [f for (f, path) in ((f, decode_path(f['path'])) for f in files)
-            if self.inClientSpec(path) and self.hasBranchPrefix(path)]
+        files = [f for f in files if self.inClientSpec(decode_path(f['path']))]
 
         if gitConfigBool('git-p4.keepEmptyCommits'):
             allow_empty = True
diff --git a/t/t9834-git-p4-file-dir-bug.sh b/t/t9834-git-p4-file-dir-bug.sh
new file mode 100755
index 0000000000..031e1f8668
--- /dev/null
+++ b/t/t9834-git-p4-file-dir-bug.sh
@@ -0,0 +1,70 @@
+#!/bin/sh
+
+test_description='git p4 directory/file bug handling
+
+This test creates files and directories with the same name in perforce and
+checks that git-p4 recovers from the error at the same time as the perforce
+repository.'
+
+. ./lib-git-p4.sh
+
+test_expect_success 'start p4d' '
+	start_p4d &&
+	test_might_fail p4 configure set submit.collision.check=0
+'
+
+test_expect_success 'init depot' '
+	(
+		cd "$cli" &&
+
+		touch add_file_add_dir_del_file add_file_add_dir_del_dir &&
+		p4 add add_file_add_dir_del_file add_file_add_dir_del_dir &&
+		mkdir add_dir_add_file_del_file add_dir_add_file_del_dir &&
+		touch add_dir_add_file_del_file/file add_dir_add_file_del_dir/file &&
+		p4 add add_dir_add_file_del_file/file add_dir_add_file_del_dir/file &&
+		p4 submit -d "add initial" &&
+
+		rm -f add_file_add_dir_del_file add_file_add_dir_del_dir &&
+		mkdir add_file_add_dir_del_file add_file_add_dir_del_dir &&
+		touch add_file_add_dir_del_file/file add_file_add_dir_del_dir/file &&
+		p4 add add_file_add_dir_del_file/file add_file_add_dir_del_dir/file &&
+		rm -rf add_dir_add_file_del_file add_dir_add_file_del_dir &&
+		touch add_dir_add_file_del_file add_dir_add_file_del_dir &&
+		p4 add add_dir_add_file_del_file add_dir_add_file_del_dir &&
+		p4 submit -d "add conflicting" &&
+
+		p4 delete -k add_file_add_dir_del_file &&
+		p4 delete -k add_file_add_dir_del_dir/file &&
+		p4 delete -k add_dir_add_file_del_file &&
+		p4 delete -k add_dir_add_file_del_dir/file &&
+		p4 submit -d "delete conflicting" &&
+
+		p4 delete -k "add_file_add_dir_del_file/file" &&
+		p4 delete -k "add_file_add_dir_del_dir" &&
+		p4 delete -k "add_dir_add_file_del_file/file" &&
+		p4 delete -k "add_dir_add_file_del_dir" &&
+		p4 submit -d "delete remaining"
+	)
+'
+
+test_expect_success 'clone with git-p4' '
+	git p4 clone --dest="$git" //depot/@1,3
+'
+
+test_expect_success 'check contents' '
+	test_path_is_dir "$git/add_file_add_dir_del_file" &&
+	test_path_is_file "$git/add_file_add_dir_del_dir" &&
+	test_path_is_dir "$git/add_dir_add_file_del_file" &&
+	test_path_is_file "$git/add_dir_add_file_del_dir"
+'
+
+test_expect_success 'rebase and check empty' '
+	git -C "$git" p4 rebase &&
+
+	test_path_is_missing "$git/add_file_add_dir_del_file" &&
+	test_path_is_missing "$git/add_file_add_dir_del_dir" &&
+	test_path_is_missing "$git/add_dir_add_file_del_file" &&
+	test_path_is_missing "$git/add_dir_add_file_del_dir"
+'
+
+test_done
-- 
2.24.1

Re: [PATCH v3] git-p4: recover from inconsistent perforce history

[Luke Diamand]

On Sun, 10 May 2020 at 11:17, Andrew Oakley <andrew@adoakley.name> wrote:
>
> Perforce allows you commit files and directories with the same name, so
> you could have files //depot/foo and //depot/foo/bar both checked in.  A
> p4 sync of a repository in this state fails.  Deleting one of the files
> recovers the repository.
>
> When this happens we want git-p4 to recover in the same way as perforce.

Looks good to me.

Perforce changed their server to reject this kind of thing in the
2017.1 version:

    Bugs fixed in 2017.1
    #1489051 (Job #2170) **
       Submitting a file with the same name as an existing depot
       directory path (or vice versa) will now be rejected.

(Of course people will still have damaged repos even today).

I tried your test with both the 2015.1 and the 2020.1 versions, and it
worked in both cases - shouldn't it be impossible to get into the
state that git-p4 now recovers from with a newer p4d?

Luke


>
>
> Signed-off-by: Andrew Oakley <andrew@adoakley.name>
> ---
>  git-p4.py                      | 43 ++++++++++++++++++++-
>  t/t9834-git-p4-file-dir-bug.sh | 70 ++++++++++++++++++++++++++++++++++
>  2 files changed, 111 insertions(+), 2 deletions(-)
>  create mode 100755 t/t9834-git-p4-file-dir-bug.sh
>
> diff --git a/git-p4.py b/git-p4.py
> index b8b2a1679e..d551efb0dd 100755
> --- a/git-p4.py
> +++ b/git-p4.py
> @@ -3214,6 +3214,42 @@ def hasBranchPrefix(self, path):
>              print('Ignoring file outside of prefix: {0}'.format(path))
>          return hasPrefix
>
> +    def findShadowedFiles(self, files, change):
> +        # Perforce allows you commit files and directories with the same name,
> +        # so you could have files //depot/foo and //depot/foo/bar both checked
> +        # in.  A p4 sync of a repository in this state fails.  Deleting one of
> +        # the files recovers the repository.
> +        #
> +        # Git will not allow the broken state to exist and only the most recent
> +        # of the conflicting names is left in the repository.  When one of the
> +        # conflicting files is deleted we need to re-add the other one to make
> +        # sure the git repository recovers in the same way as perforce.
> +        deleted = [f for f in files if f['action'] in self.delete_actions]
> +        to_check = set()
> +        for f in deleted:
> +            path = decode_path(f['path'])
> +            to_check.add(path + '/...')
> +            while True:
> +                path = path.rsplit("/", 1)[0]
> +                if path == "/" or path in to_check:
> +                    break
> +                to_check.add(path)
> +        to_check = ['%s@%s' % (wildcard_encode(p), change) for p in to_check
> +            if self.hasBranchPrefix(p)]
> +        if to_check:
> +            stat_result = p4CmdList(["-x", "-", "fstat", "-T",
> +                "depotFile,headAction,headRev,headType"], stdin=to_check)
> +            for record in stat_result:
> +                if record['code'] != 'stat':
> +                    continue
> +                if record['headAction'] in self.delete_actions:
> +                    continue
> +                files.append({
> +                    'action': 'add',
> +                    'path': record['depotFile'],
> +                    'rev': record['headRev'],
> +                    'type': record['headType']})
> +
>      def commit(self, details, files, branch, parent = "", allow_empty=False):
>          epoch = details["time"]
>          author = details["user"]
> @@ -3222,11 +3258,14 @@ def commit(self, details, files, branch, parent = "", allow_empty=False):
>          if self.verbose:
>              print('commit into {0}'.format(branch))
>
> +        files = [f for f in files
> +            if self.hasBranchPrefix(decode_path(f['path']))]
> +        self.findShadowedFiles(files, details['change'])
> +
>          if self.clientSpecDirs:
>              self.clientSpecDirs.update_client_spec_path_cache(files)
>
> -        files = [f for (f, path) in ((f, decode_path(f['path'])) for f in files)
> -            if self.inClientSpec(path) and self.hasBranchPrefix(path)]
> +        files = [f for f in files if self.inClientSpec(decode_path(f['path']))]
>
>          if gitConfigBool('git-p4.keepEmptyCommits'):
>              allow_empty = True
> diff --git a/t/t9834-git-p4-file-dir-bug.sh b/t/t9834-git-p4-file-dir-bug.sh
> new file mode 100755
> index 0000000000..031e1f8668
> --- /dev/null
> +++ b/t/t9834-git-p4-file-dir-bug.sh
> @@ -0,0 +1,70 @@
> +#!/bin/sh
> +
> +test_description='git p4 directory/file bug handling
> +
> +This test creates files and directories with the same name in perforce and
> +checks that git-p4 recovers from the error at the same time as the perforce
> +repository.'
> +
> +. ./lib-git-p4.sh
> +
> +test_expect_success 'start p4d' '
> +       start_p4d &&
> +       test_might_fail p4 configure set submit.collision.check=0
> +'
> +
> +test_expect_success 'init depot' '
> +       (
> +               cd "$cli" &&
> +
> +               touch add_file_add_dir_del_file add_file_add_dir_del_dir &&
> +               p4 add add_file_add_dir_del_file add_file_add_dir_del_dir &&
> +               mkdir add_dir_add_file_del_file add_dir_add_file_del_dir &&
> +               touch add_dir_add_file_del_file/file add_dir_add_file_del_dir/file &&
> +               p4 add add_dir_add_file_del_file/file add_dir_add_file_del_dir/file &&
> +               p4 submit -d "add initial" &&
> +
> +               rm -f add_file_add_dir_del_file add_file_add_dir_del_dir &&
> +               mkdir add_file_add_dir_del_file add_file_add_dir_del_dir &&
> +               touch add_file_add_dir_del_file/file add_file_add_dir_del_dir/file &&
> +               p4 add add_file_add_dir_del_file/file add_file_add_dir_del_dir/file &&
> +               rm -rf add_dir_add_file_del_file add_dir_add_file_del_dir &&
> +               touch add_dir_add_file_del_file add_dir_add_file_del_dir &&
> +               p4 add add_dir_add_file_del_file add_dir_add_file_del_dir &&
> +               p4 submit -d "add conflicting" &&
> +
> +               p4 delete -k add_file_add_dir_del_file &&
> +               p4 delete -k add_file_add_dir_del_dir/file &&
> +               p4 delete -k add_dir_add_file_del_file &&
> +               p4 delete -k add_dir_add_file_del_dir/file &&
> +               p4 submit -d "delete conflicting" &&
> +
> +               p4 delete -k "add_file_add_dir_del_file/file" &&
> +               p4 delete -k "add_file_add_dir_del_dir" &&
> +               p4 delete -k "add_dir_add_file_del_file/file" &&
> +               p4 delete -k "add_dir_add_file_del_dir" &&
> +               p4 submit -d "delete remaining"
> +       )
> +'
> +
> +test_expect_success 'clone with git-p4' '
> +       git p4 clone --dest="$git" //depot/@1,3
> +'
> +
> +test_expect_success 'check contents' '
> +       test_path_is_dir "$git/add_file_add_dir_del_file" &&
> +       test_path_is_file "$git/add_file_add_dir_del_dir" &&
> +       test_path_is_dir "$git/add_dir_add_file_del_file" &&
> +       test_path_is_file "$git/add_dir_add_file_del_dir"
> +'
> +
> +test_expect_success 'rebase and check empty' '
> +       git -C "$git" p4 rebase &&
> +
> +       test_path_is_missing "$git/add_file_add_dir_del_file" &&
> +       test_path_is_missing "$git/add_file_add_dir_del_dir" &&
> +       test_path_is_missing "$git/add_dir_add_file_del_file" &&
> +       test_path_is_missing "$git/add_dir_add_file_del_dir"
> +'
> +
> +test_done
> --
> 2.24.1
>

Re: [PATCH v3] git-p4: recover from inconsistent perforce history

[Andrew Oakley]

On Sun, 10 May 2020 13:03:11 +0100
Luke Diamand <luke@diamand.org> wrote:
> Perforce changed their server to reject this kind of thing in the
> 2017.1 version:
> 
>     Bugs fixed in 2017.1
>     #1489051 (Job #2170) **
>        Submitting a file with the same name as an existing depot
>        directory path (or vice versa) will now be rejected.
> 
> (Of course people will still have damaged repos even today).
> 
> I tried your test with both the 2015.1 and the 2020.1 versions, and it
> worked in both cases - shouldn't it be impossible to get into the
> state that git-p4 now recovers from with a newer p4d?

Yes, there is an option in perforce (submit.collision.check) that stops
new changelists from introducing this problem, and it is turned on by
default.  Unfortunately this option does *exactly* what the description
says, so you can't delete a directory and replace it with a file in the
same changelist - the delete has to happen first.  It's not clear which
behaviour is least bad.

The test case tries to turn the perforce option off so it works on both
old and new server versions.

Thanks

Re: [PATCH v3] git-p4: recover from inconsistent perforce history

[Junio C Hamano]

Luke Diamand <luke@diamand.org> writes:

> On Sun, 10 May 2020 at 11:17, Andrew Oakley <andrew@adoakley.name> wrote:
>>
>> Perforce allows you commit files and directories with the same name, so
>> you could have files //depot/foo and //depot/foo/bar both checked in.  A
>> p4 sync of a repository in this state fails.  Deleting one of the files
>> recovers the repository.
>>
>> When this happens we want git-p4 to recover in the same way as perforce.
>
> Looks good to me.
>
> Perforce changed their server to reject this kind of thing in the
> 2017.1 version:
>
>     Bugs fixed in 2017.1
>     #1489051 (Job #2170) **
>        Submitting a file with the same name as an existing depot
>        directory path (or vice versa) will now be rejected.
>
> (Of course people will still have damaged repos even today).

Perhaps it is worth describing the above in the log message?  E.g.

    Perforce allows you commit files and directories with the same name,
    so you could have files //depot/foo and //depot/foo/bar both checked
    in.  A p4 sync of a repository in this state fails.  Deleting one of
    the files recovers the repository.

    When this happens we want git-p4 to recover in the same way as
    perforce.

    Note that Perforce has this change in their 2017.1 version:

         Bugs fixed in 2017.1
         #1489051 (Job #2170) **
            Submitting a file with the same name as an existing depot
            directory path (or vice versa) will now be rejected.

    so people hopefully will not creating damaged Perforce repos
    anymore, but "git p4" needs to be able to interact with already
    corrupt ones.

    Signed-off-by: Andrew Oakley <andrew@adoakley.name>
    Reviewed-by: Luke Diamand <luke@diamand.org>
    Signed-off-by: Junio C Hamano <gitster@pobox.com>

Thanks.