Re: bug? round-trip through fast-import/fast-export loses files

[Elijah Newren <newren@gmail.com>]

Hi,

On Mon, Mar 20, 2023 at 11:23 AM Priedhorsky, Reid <reidpr@lanl.gov> wrote:
>
>   Hello,
>
>   I believe I’ve found a bug in Git. It seems that (1) round-tripping through
>   fast-export/fast-import a repository (2) that contains a commit that changes
>   a file to a directory (3) deletes the contents of that directory from the
>   repository.
>
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
>
> What did you do before the bug happened? (Steps to reproduce your issue)
>
>   Run this shell script:
>
>   ~~~~
>   #!/bin/bash
>
>   set -ex
>
>   mkdir -p /tmp/weirdal
>   cd /tmp/weirdal
>   git --version
>
>   # init repo
>   rm -Rf wd
>   mkdir wd
>   cd wd
>   git init -b main
>
>   # first commit - foo is a file
>   touch foo
>   git add -A
>   git commit -m 'file'
>
>   # second commit - foo is a directory
>   rm foo
>   mkdir foo
>   touch foo/bar
>   git add -A
>   git commit -m 'directory'
>
>   # the contents of foo are in the working dir and the repo
>   git status
>   ls -lR
>   git ls-tree --name-only -r HEAD
>
>   # import/export repository (add --full-tree to work around bug)
>   git fast-export --no-data -- --all > ../export
>   cat ../export
>   git fast-import --force --quiet < ../export
>
>   # bug: foo is still in the WD but not the repo; should still be both
>   git status
>   ls -lR
>   git ls-tree --name-only -r HEAD
>   #git fast-export --no-data -- --all | diff -u --text ../export - || true
>   ~~~~
>
> What did you expect to happen? (Expected behavior)
>
>   Repo should be unchanged, i.e.:
>
>   + git status
>   On branch main
>   nothing to commit, working tree clean
>
> What happened instead? (Actual behavior)
>
>   Git thinks foo/bar has been staged:
>
>   + git status
>   On branch main
>   Changes to be committed:
>     (use "git restore --staged <file>..." to unstage)
>           new file:   foo/bar
>
> What's different between what you expected and what actually happened?
>
>   File foo/bar is staged when it should be unchanged.
>
> Anything else you want to add:
>
>   This also happens in 2.38.1 built from source.
>
>   The bad behavior can be worked around with “--full-tree” on fast-export, but
>   the real repo where I want to do this is pretty large, so I’d prefer not to.
>
>   Note the “git fast-export” output:
>
>     commit refs/heads/main
>     mark :2
>     author Reid Priedhorsky <reidpr@lanl.gov> 1679330805 -0600
>     committer Reid Priedhorsky <reidpr@lanl.gov> 1679330805 -0600
>     data 10
>     directory
>     from :1
>     M 100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 foo/bar
>     D foo
>
>   It looks to me like the “M ... foo/bar” is being processed before “D foo”
>   when it should happen in the opposite order.

Thanks for the well-written bug report, including not only a testcase
but even the relevant bits of the fast-export output.  I thought I had
fixed D/F issues in fast-export & fast-import before, and indeed a
search turns up both of

253fb5f889 (fast-import: Improve robustness when D->F changes provided
in wrong order, 2010-07-09)
060df62422 (fast-export: Fix output order of D/F changes, 2010-07-09)

However, it looks like both of those only considered D->F (directory
becomes a file) changes, whereas you specifically have a case of F->D
(file becoming a directory).

Honestly, looking back at those two patches of mine, I think both were
rather suboptimal.  A better solution that would handle both F->D and
D->F would be having fast-export sort the diff_filepairs such that it
processes the deletes before the modifies.  Another improved solution
would be having fast-import sort the files given to it and handling
deletes first.  Either should fix this.

Might be a good task for a new contributor.  Any takers?  (Tagging as
#leftoverbits.)