Question about git merge -s ort command‏‏

Question about git merge -s ort command‏‏

[Ariel Man]

Hi,
following the below thread, led me to do some tests:
https://stackoverflow.com/a/69114710/11705021

One of the test can be seen here, in the last example in my question:
https://stackoverflow.com/questions/69150777/git-mv-doesnt-work-as-expected-wrong-files-location

I wanted to get confirmation for my case:
Does `git merge -s ort` command, doesn't know to detect new folder
location when merging it? It did worked for folders and files
renaming, and new files under existing folders (as describe here:
https://stackoverflow.com/a/52227201/6309)

Ariel

Re: Question about git merge -s ort command‏‏

[Elijah Newren]

On Sun, Sep 19, 2021 at 8:12 PM Ariel Man <arielman22@gmail.com> wrote:
>
> Hi,
> following the below thread, led me to do some tests:
> https://stackoverflow.com/a/69114710/11705021
>
> One of the test can be seen here, in the last example in my question:
> https://stackoverflow.com/questions/69150777/git-mv-doesnt-work-as-expected-wrong-files-location
>
> I wanted to get confirmation for my case:
> Does `git merge -s ort` command, doesn't know to detect new folder
> location when merging it? It did worked for folders and files
> renaming, and new files under existing folders (as describe here:
> https://stackoverflow.com/a/52227201/6309)

I mean, if you have a testcase and want to see how it works, you can
just try it.  merge-ort has been released and is part of git already.

But I'll take a stab...

Your testcase with new folders sounds like testcase 12f from t6423 in
our testsuite (from commit 902c521a35 ("t6423: more involved directory
rename test", 2020-10-15)), a case that works with the `ort` merge
strategy but not with `recursive`.  So, the `ort` strategy is likely
to be better for your usecase than `recursive`, but it's hard for me
to definitively state that it'll solve all your usecase(s) since you
didn't fully describe them.  This is especially important since you
appear to have discovered a segfault (in the first post); I was not
aware of any circumstances when the old merge machinery would do so.
I'd be interested to know if the same issue repeats with the `ort`
strategy, and, if so, whether you can get a stack trace, or find an
example that reproduces the problem, or provide us with access to your
repository.  Also, in your second link, you allude to problems in a
case that does not include new folders, in situations where I'd expect
`recursive` and `ort` to behave the same.  But it's hard to know the
expected behavior since you didn't provide that many details about the
setup -- e.g. you didn't cover the contents of any of the files (so
it's hard to know if files are more similar to ones with different
names than what you expected to be matched up), it's unclear if there
are other files involved (e.g. if you have multiple copies of some
library and the rename detection is detecting a different copy than
you expected, or if there are lots of additional files besides those
you are listing in your example and they are being renamed
differently, etc.), whether you have lots of large files also being
renamed (perhaps the VDI files you allude to somewhere?), etc., etc.
If you can provide access to a repository and the commands you
reported and what you expected vs. what you saw, I'd try to find some
time to take a look.

I hope that helps, though I'm not sure I've got enough details to
answer all your questions.  I guess if I were to summarize, I'd say:
  * the `ort` strategy would be better than `recursive` for you, for
one particular type of issue, but probably behave the same on other
areas
  * you seem to have useful testcases that might be different than
ones we have in our testsuite or that I know about
  * you haven't provided enough information for me to investigate or
answer any further, though.

You have gotten me really curious about this segfaulting testcase --
whether it's just in the recursive backend or also afflicts merge-ort,
and how to duplicate it.

Re: Question about git merge -s ort command‏‏

[Ariel Man]

Hi,
Thanks for the detailed answer, but just to clarify, my main question
is regarding ORT strategy, and not recursive.
Mainly in this case, where ORT didn't behave as I expected:
https://stackoverflow.com/questions/69150777/git-mv-doesnt-work-as-expected-wrong-files-location#:~:text=test3.py%0A%E2%94%82%20%20%20%E2%94%94%E2%94%80%E2%94%80%20dir3-,UPDATE,-I%27m%20adding%20one

As I wrote, this specific case is where new folder and files were
created after the consolidation (and not new files under the existing
folder). As you can see, when only new files were added to the
existing folder, the git ORT recognized it and moved the folder
python1 with its files test1.py and test4.py to the right place
(dir2). But when new folder was created on the target (new_test with
the file test5.py), when I ran the ORT command, the GIT didn't
recognized it and created again the folder python1 on the old path
(dir1-->python1-->new_test-->test5.py).
I believe you can reproduce it yourself on your repository and see the
behaviour.
Ariel

‫בתאריך יום ה׳, 30 בספט׳ 2021 ב-8:27 מאת ‪Elijah Newren‬‏
<‪newren@gmail.com‬‏>:‬
>
> On Sun, Sep 19, 2021 at 8:12 PM Ariel Man <arielman22@gmail.com> wrote:
> >
> > Hi,
> > following the below thread, led me to do some tests:
> > https://stackoverflow.com/a/69114710/11705021
> >
> > One of the test can be seen here, in the last example in my question:
> > https://stackoverflow.com/questions/69150777/git-mv-doesnt-work-as-expected-wrong-files-location
> >
> > I wanted to get confirmation for my case:
> > Does `git merge -s ort` command, doesn't know to detect new folder
> > location when merging it? It did worked for folders and files
> > renaming, and new files under existing folders (as describe here:
> > https://stackoverflow.com/a/52227201/6309)
>
> I mean, if you have a testcase and want to see how it works, you can
> just try it.  merge-ort has been released and is part of git already.
>
> But I'll take a stab...
>
> Your testcase with new folders sounds like testcase 12f from t6423 in
> our testsuite (from commit 902c521a35 ("t6423: more involved directory
> rename test", 2020-10-15)), a case that works with the `ort` merge
> strategy but not with `recursive`.  So, the `ort` strategy is likely
> to be better for your usecase than `recursive`, but it's hard for me
> to definitively state that it'll solve all your usecase(s) since you
> didn't fully describe them.  This is especially important since you
> appear to have discovered a segfault (in the first post); I was not
> aware of any circumstances when the old merge machinery would do so.
> I'd be interested to know if the same issue repeats with the `ort`
> strategy, and, if so, whether you can get a stack trace, or find an
> example that reproduces the problem, or provide us with access to your
> repository.  Also, in your second link, you allude to problems in a
> case that does not include new folders, in situations where I'd expect
> `recursive` and `ort` to behave the same.  But it's hard to know the
> expected behavior since you didn't provide that many details about the
> setup -- e.g. you didn't cover the contents of any of the files (so
> it's hard to know if files are more similar to ones with different
> names than what you expected to be matched up), it's unclear if there
> are other files involved (e.g. if you have multiple copies of some
> library and the rename detection is detecting a different copy than
> you expected, or if there are lots of additional files besides those
> you are listing in your example and they are being renamed
> differently, etc.), whether you have lots of large files also being
> renamed (perhaps the VDI files you allude to somewhere?), etc., etc.
> If you can provide access to a repository and the commands you
> reported and what you expected vs. what you saw, I'd try to find some
> time to take a look.
>
> I hope that helps, though I'm not sure I've got enough details to
> answer all your questions.  I guess if I were to summarize, I'd say:
>   * the `ort` strategy would be better than `recursive` for you, for
> one particular type of issue, but probably behave the same on other
> areas
>   * you seem to have useful testcases that might be different than
> ones we have in our testsuite or that I know about
>   * you haven't provided enough information for me to investigate or
> answer any further, though.
>
> You have gotten me really curious about this segfaulting testcase --
> whether it's just in the recursive backend or also afflicts merge-ort,
> and how to duplicate it.

Re: Question about git merge -s ort command‏‏

[Elijah Newren]

Hi,

Please do not top-post on this mailing list.

On Wed, Sep 29, 2021 at 11:58 PM Ariel Man <arielman22@gmail.com> wrote:
>
> Hi,
> Thanks for the detailed answer, but just to clarify, my main question
> is regarding ORT strategy, and not recursive.

Since you simply linked to three posts in your original email, and the
first showed a segfault, this wasn't at all clear.

> Mainly in this case, where ORT didn't behave as I expected:
> https://stackoverflow.com/questions/69150777/git-mv-doesnt-work-as-expected-wrong-files-location#:~:text=test3.py%0A%E2%94%82%20%20%20%E2%94%94%E2%94%80%E2%94%80%20dir3-,UPDATE,-I%27m%20adding%20one
>
> As I wrote, this specific case is where new folder and files were
> created after the consolidation (and not new files under the existing
> folder). As you can see, when only new files were added to the
> existing folder, the git ORT recognized it and moved the folder
> python1 with its files test1.py and test4.py to the right place
> (dir2). But when new folder was created on the target (new_test with
> the file test5.py), when I ran the ORT command, the GIT didn't
> recognized it and created again the folder python1 on the old path
> (dir1-->python1-->new_test-->test5.py).

Yes, I read through that and pointed out your examples weren't fully
specified.  You haven't stated what else exists in all these
directories, what kind of contents files had, if there were big
binaries, etc.

> I believe you can reproduce it yourself on your repository and see the
> behaviour.

Nope, I can't.  I went through the trouble of setting up the testcases
as close as I could from your diagrams.  When using the recursive
strategy, I do see that test4.py and test5.py are left behind in dir1,
but when using the ort strategy, it behaves exactly as you said you
wanted (with *both* test4.py and test5.py appearing somewhere under
dir2 rather than dir1).

Here's the output from my script showing all the files that exist in
each of the relevant commits:

=== Original ===
dir1/otherfile
dir1/python1/test1.py
dir2/otherfile
dir2/python2/test2.py
dir3/otherfile
dir3/python3/test3.py

=== Consolidated ===
dir1/otherfile
dir2/otherfile
dir2/python1/test1.py
dir2/python2/test2.py
dir2/python3/test3.py
dir3/otherfile

=== New files ===
dir1/otherfile
dir1/python1/new_test/test5.py
dir1/python1/test1.py
dir1/python1/test4.py
dir2/otherfile
dir2/python2/test2.py
dir3/otherfile
dir3/python3/test3.py

=== Merged result ===
dir1/otherfile
dir2/otherfile
dir2/python1/new_test/test5.py
dir2/python1/test1.py
dir2/python1/test4.py
dir2/python2/test2.py
dir2/python3/test3.py
dir3/otherfile


and here's the script I used to run this testcase and generate the above output:

#!/bin/bash

git init -b main nukeme
cd nukeme

echo '''
│   └── dir1
│   │   └── python1
│   │   │   └── test1.py
│   └── dir2
│   │   └── python2
│   │   │   └── test2.py
│   └── dir3
│   │   └── python3
│   │   │   └── test3.py
''' >/dev/null
for i in $(seq 1 3); do
    mkdir -p dir$i/python$i
    echo $i >dir$i/python$i/test$i.py
    >dir$i/otherfile
done

git add .
git commit -qm "Initial"

git branch consolidated
git branch newfiles

git checkout consolidated
echo '''
│   └── dir1
│   └── dir2
│   │   └── python1
│   │   │   └── test1.py
│   │   └── python2
│   │   │   └── test2.py
│   │   └── python3
│   │   │   └── test3.py
│   └── dir3
''' >/dev/null
git mv dir1/python1 dir2/
git mv dir3/python3 dir2/
git commit -qm consolidated

git checkout newfiles
echo 4 >dir1/python1/test4.py
git add dir1/python1/test4.py
mkdir dir1/python1/new_test
echo 5 >dir1/python1/new_test/test5.py
git add dir1/python1/new_test/test5.py
git commit -qm newfiles

git checkout -qb merged consolidated
git merge --no-edit -s ort newfiles

echo
echo "=== Original ==="
git ls-tree -r --name-only main
echo
echo "=== Consolidated ==="
git ls-tree -r --name-only consolidated
echo
echo "=== New files ==="
git ls-tree -r --name-only newfiles
echo
echo "=== Merged result ==="
git ls-tree -r --name-only merged