Regression in 'git pull --rebase --autostash' since v2.32.0

Regression in 'git pull --rebase --autostash' since v2.32.0

[Philippe Blain]

Hi Felipe,

Your recent clean-up of 'git pull --autostash' seems to unfortunately have made things
worse if the pull brings new files that conflict with existing untracked files,
which makes the pull abort,
and there are tracked files with modifications (so --autostash comes into play).

Before your change, 'git pull --no-rebase --autostash' did *not* apply the autostash
after the pull failed, thus loosing modifications to tracked files (and it did not save the
stash entry !). 'git pull --rebase --autostash' correctly applied the autostash, but ended with
a strange "error: could not detach HEAD".

After your change, both 'git pull --no-rebase --autostash' and 'git pull --rebase --autostash'
have the same buggy behaviour: they do not apply the autostash and do not save it in the stash list.

I had already documented the old behaviour at [1]. Here, I copy my reproducer script
(save it as "script"):

~~~bash
#!/bin/sh

# usage: ./script <'git pull' arguments>

set -x

rm -rf test
rm -rf clone

# Create origin repo
git init test
(
cd test
date>>file
git add file
git commit -m "add file"
)
# Clone
git clone test clone
# Create new file in origin
(
cd test
date>>other
git add other
git commit -m "add other"
)
# Create the same file in clone (untracked)
(
cd clone
date>>other
# If testing '--autostash', add some modifications to 'file'
if [[ "$@" =~ "--autostash" ]]; then
   date>>file
fi
# status before pull
git status
# Try to pull
git pull "$@"
# status after pull
git status
# see if the stash was saved
git stash list
)
~~~


Here are the buggy results :

** ./script --no-rebase --autostash **

~~~
+ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
   (use "git add <file>..." to update what will be committed)
   (use "git restore <file>..." to discard changes in working directory)
	modified:   file

Untracked files:
   (use "git add <file>..." to include in what will be committed)
	other

no changes added to commit (use "git add" and/or "git commit -a")
+ git pull --no-rebase --autostash
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), 283 bytes | 94.00 KiB/s, done.
 From /Users/Philippe/Code/GIT-devel/BUGS/ggg-759-pull-autotash-untracked/test
    4ebab2f..fc7a169  master     -> origin/master
Updating 4ebab2f..fc7a169
Created autostash: cfd51b5
error: The following untracked working tree files would be overwritten by merge:
	other
Please move or remove them before you merge.
Aborting
+ git status
On branch master
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
   (use "git pull" to update your local branch)

Untracked files:
   (use "git add <file>..." to include in what will be committed)
	other

nothing added to commit but untracked files present (use "git add" to track)
+ git stash list
# empty!
~~~

** ./script --rebase --autostash **

~~~
+ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
   (use "git add <file>..." to update what will be committed)
   (use "git restore <file>..." to discard changes in working directory)
	modified:   file

Untracked files:
   (use "git add <file>..." to include in what will be committed)
	other

no changes added to commit (use "git add" and/or "git commit -a")
+ git pull --rebase --autostash
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 2 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (2/2), 223 bytes | 13.00 KiB/s, done.
 From /Users/Philippe/Code/GIT-devel/BUGS/ggg-759-pull-autotash-untracked/test
    1aa91d4..4f8c34c  master     -> origin/master
Updating 1aa91d4..4f8c34c
Created autostash: d5dffd9
error: The following untracked working tree files would be overwritten by merge:
	other
Please move or remove them before you merge.
Aborting
+ git status
On branch master
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
   (use "git pull" to update your local branch)

Untracked files:
   (use "git add <file>..." to include in what will be committed)
	other

nothing added to commit but untracked files present (use "git add" to track)
+ git stash list
# empty!
~~~

Reverting 221ec24e9b (Merge branch 'fc/pull-cleanups', 2021-07-08) brings
back the old behaviour (which is still buggy for --no-rebase).

I noticed (by reading the code and checking)
that the autostash is not completely lost, it's still pointed to by the MERGE_AUTOSTASH
special ref, but this is ref is not documented (it's just mentioned without a
clear definition).

Cheers,
Philippe.


[1] https://github.com/gitgitgadget/git/issues/759

Re: Regression in 'git pull --rebase --autostash' since v2.32.0

[Philippe Blain]

Le 2021-07-17 à 11:29, Philippe Blain a écrit :
> Hi Felipe,
> 
> Your recent clean-up of 'git pull --autostash' seems to unfortunately have made things
> worse if the pull brings new files that conflict with existing untracked files,
> which makes the pull abort,
> and there are tracked files with modifications (so --autostash comes into play).
> 
> Before your change, 'git pull --no-rebase --autostash' did *not* apply the autostash
> after the pull failed, thus loosing modifications to tracked files (and it did not save the
> stash entry !). 'git pull --rebase --autostash' correctly applied the autostash, but ended with
> a strange "error: could not detach HEAD".
> 
> After your change, both 'git pull --no-rebase --autostash' and 'git pull --rebase --autostash'
> have the same buggy behaviour: they do not apply the autostash and do not save it in the stash list.

The change below seem to fix both cases:

diff --git a/builtin/merge.c b/builtin/merge.c
index a8a843b1f5..b2ad70c50a 100644
--- a/builtin/merge.c
+++ b/builtin/merge.c
@@ -1560,6 +1560,7 @@ int cmd_merge(int argc, const char **argv, const char *prefix)
  					  &head_commit->object.oid,
  					  &commit->object.oid,
  					  overwrite_ignore)) {
+			apply_autostash(git_path_merge_autostash(the_repository));
  			ret = 1;
  			goto done;
  		}


*But* from a quick audit of 'cmd_merge', there are still two code paths that
call 'create_autostash' but then fail to apply it before calling 'goto done':

1. the branch 'if (automerge_was_ok)' (line 1693)
2. the branch 'if (!best_strategy)' (line 1704)


Cheers,
Philippe.

RE: Regression in 'git pull --rebase --autostash' since v2.32.0

[Felipe Contreras]

Hello,

Philippe Blain wrote:
> Your recent clean-up of 'git pull --autostash' seems to unfortunately have made things
> worse if the pull brings new files that conflict with existing untracked files,
> which makes the pull abort,
> and there are tracked files with modifications (so --autostash comes into play).
> 
> Before your change, 'git pull --no-rebase --autostash' did *not* apply the autostash
> after the pull failed, thus loosing modifications to tracked files (and it did not save the
> stash entry !). 'git pull --rebase --autostash' correctly applied the autostash, but ended with
> a strange "error: could not detach HEAD".
> 
> After your change, both 'git pull --no-rebase --autostash' and 'git pull --rebase --autostash'
> have the same buggy behaviour: they do not apply the autostash and do not save it in the stash list.
> 
> I had already documented the old behaviour at [1]. Here, I copy my reproducer script
> (save it as "script"):

I cannot reproduce this. In my case the reproducer script never puts
anything in the stash list.

Moreover, this is not an issue of `git pull`, but `git merge`.

I can reproduce the problem that the modifications are lost like this:

  git init test
  (
    cd test
    date >> file
    git add file
    git commit -m 'add file'
    date >> other
    git add other
    git commit -m 'add other'
    git checkout -b topic @~
    date >> other
    date >> file
    git status
    git "$@" master
    git status
    git stash list
  )

Running this with 'rebase --autostash' fails and nothing is put in the
stash list, but the modifications to 'file' remain. I think this is the
correct behavior.

But with 'merge --autostash' the modifications to 'file' are lost. That
is a bug, and it's already present in v2.32.

Do you agree?

-- 
Felipe Contreras

Re: Regression in 'git pull --rebase --autostash' since v2.32.0

[Philippe Blain]

Hi Felipe,

Le 2021-07-17 à 15:34, Felipe Contreras a écrit :
> Hello,
> 
> Philippe Blain wrote:
>> Your recent clean-up of 'git pull --autostash' seems to unfortunately have made things
>> worse if the pull brings new files that conflict with existing untracked files,
>> which makes the pull abort,
>> and there are tracked files with modifications (so --autostash comes into play).
>>
>> Before your change, 'git pull --no-rebase --autostash' did *not* apply the autostash
>> after the pull failed, thus loosing modifications to tracked files (and it did not save the
>> stash entry !). 'git pull --rebase --autostash' correctly applied the autostash, but ended with
>> a strange "error: could not detach HEAD".
>>
>> After your change, both 'git pull --no-rebase --autostash' and 'git pull --rebase --autostash'
>> have the same buggy behaviour: they do not apply the autostash and do not save it in the stash list.
>>
>> I had already documented the old behaviour at [1]. Here, I copy my reproducer script
>> (save it as "script"):
> 
> I cannot reproduce this. In my case the reproducer script never puts
> anything in the stash list.

I tried again with GIT_CONFIG_GLOBAL=/dev/null on my end (and manually
setting GIT_AUTHOR_{NAME,EMAIL}) to be sure that my config was not playing
a role, and I still reproduce. Copying the output from my 1st email
(for --rebase, but it's the same thing for --no-rebase):


> ** ./script --rebase --autostash **
> 
> ~~~
> + git status
> On branch master
> Your branch is up to date with 'origin/master'.
> 
> Changes not staged for commit:
>   (use "git add <file>..." to update what will be committed)
>   (use "git restore <file>..." to discard changes in working directory)
>     modified:   file
> 
> Untracked files:
>   (use "git add <file>..." to include in what will be committed)
>     other
> 
> no changes added to commit (use "git add" and/or "git commit -a")
> + git pull --rebase --autostash
> remote: Enumerating objects: 4, done.
> remote: Counting objects: 100% (4/4), done.
> remote: Compressing objects: 100% (2/2), done.
> remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
> Unpacking objects: 100% (3/3), 283 bytes | 94.00 KiB/s, done.
> From /Users/Philippe/Code/GIT-devel/BUGS/ggg-759-pull-autotash-untracked/test
>    4ebab2f..fc7a169  master     -> origin/master
> Updating 4ebab2f..fc7a169
> Created autostash: cfd51b5
> error: The following untracked working tree files would be overwritten by merge:
>     other
> Please move or remove them before you merge.
> Aborting

Here is the bug: we should see "Applied autostash" after "Aborting". The 'git status'
below is to verify that the autostash was indeed not applied, and the
'git stash list' was to check if the autostash was at least available
in the stash list (it's not).

> + git status
> On branch master
> Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
>   (use "git pull" to update your local branch)
> 
> Untracked files:
>   (use "git add <file>..." to include in what will be committed)
>     other
> 
> nothing added to commit but untracked files present (use "git add" to track)
> + git stash list
> # empty!
> ~~~ 

Back to your reply:

> 
> Moreover, this is not an issue of `git pull`, but `git merge`.
> 
> I can reproduce the problem that the modifications are lost like this:
> 
>    git init test
>    (
>      cd test
>      date >> file
>      git add file
>      git commit -m 'add file'
>      date >> other
>      git add other
>      git commit -m 'add other'
>      git checkout -b topic @~
>      date >> other
>      date >> file
>      git status
>      git "$@" master
>      git status
>      git stash list
>    )
> 
> Running this with 'rebase --autostash' fails and nothing is put in the
> stash list, but the modifications to 'file' remain. I think this is the
> correct behavior.

Yes. We see the following output:

     Created autostash: 69775ee
     Applied autostash.
     Successfully rebased and updated refs/heads/topic.

The code correctly applied the autostash after the rebase, so it's
normal it's not kept in the stash list. I agree it's the correct behaviour.

> 
> But with 'merge --autostash' the modifications to 'file' are lost. That
> is a bug, and it's already present in v2.32.

That's also true. From what I understand it dates back to the introduction
of 'git merge --autostash'.

> 
> Do you agree?
> 

I agree that the bug with 'merge --autostash' was already present in v2.32.0.
But since your 340062243a (pull: cleanup autostash check, 2021-06-17),
'git pull --rebase --autostash' calls 'git merge --autostash' in the fast-forward
case, and so we hit the bug, which was not the case before since the fast-forward
merge shortcut was skipped if '--autostash' was given for 'git pull --rebase'.

The fix I suggested in [1] seems to fix both cases, but as I wrote there
it still leaves two code paths that might trigger the bug. I'm not familiar
at all with the code for these two code paths, so I'm not ready to send
a formal patch; I thought I'd send the diff anyway if you or Denton (or anyone
else) wants to start from there.

Thanks,

Philippe.

[1] https://lore.kernel.org/git/a0071549-73f6-9636-9279-3f01143a05de@gmail.com/T/#me6fd1cebbcbc91365bb1deff857c2fcf72db8d2d

Re: Regression in 'git pull --rebase --autostash' since v2.32.0

[Felipe Contreras]

Philippe Blain wrote:
> Le 2021-07-17 à 15:34, Felipe Contreras a écrit :
> > Hello,
> > 
> > Philippe Blain wrote:
> >> Your recent clean-up of 'git pull --autostash' seems to unfortunately have made things
> >> worse if the pull brings new files that conflict with existing untracked files,
> >> which makes the pull abort,
> >> and there are tracked files with modifications (so --autostash comes into play).
> >>
> >> Before your change, 'git pull --no-rebase --autostash' did *not* apply the autostash
> >> after the pull failed, thus loosing modifications to tracked files (and it did not save the
> >> stash entry !). 'git pull --rebase --autostash' correctly applied the autostash, but ended with
> >> a strange "error: could not detach HEAD".
> >>
> >> After your change, both 'git pull --no-rebase --autostash' and 'git pull --rebase --autostash'
> >> have the same buggy behaviour: they do not apply the autostash and do not save it in the stash list.
> >>
> >> I had already documented the old behaviour at [1]. Here, I copy my reproducer script
> >> (save it as "script"):
> > 
> > I cannot reproduce this. In my case the reproducer script never puts
> > anything in the stash list.
> 
> I tried again with GIT_CONFIG_GLOBAL=/dev/null on my end (and manually
> setting GIT_AUTHOR_{NAME,EMAIL}) to be sure that my config was not playing
> a role, and I still reproduce. Copying the output from my 1st email
> (for --rebase, but it's the same thing for --no-rebase):
> 
> 
> > ** ./script --rebase --autostash **
> > 
> > ~~~
> > + git status
> > On branch master
> > Your branch is up to date with 'origin/master'.
> > 
> > Changes not staged for commit:
> >   (use "git add <file>..." to update what will be committed)
> >   (use "git restore <file>..." to discard changes in working directory)
> >     modified:   file
> > 
> > Untracked files:
> >   (use "git add <file>..." to include in what will be committed)
> >     other
> > 
> > no changes added to commit (use "git add" and/or "git commit -a")
> > + git pull --rebase --autostash
> > remote: Enumerating objects: 4, done.
> > remote: Counting objects: 100% (4/4), done.
> > remote: Compressing objects: 100% (2/2), done.
> > remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
> > Unpacking objects: 100% (3/3), 283 bytes | 94.00 KiB/s, done.
> > From /Users/Philippe/Code/GIT-devel/BUGS/ggg-759-pull-autotash-untracked/test
> >    4ebab2f..fc7a169  master     -> origin/master
> > Updating 4ebab2f..fc7a169
> > Created autostash: cfd51b5
> > error: The following untracked working tree files would be overwritten by merge:
> >     other
> > Please move or remove them before you merge.
> > Aborting
> 
> Here is the bug: we should see "Applied autostash" after "Aborting". The 'git status'
> below is to verify that the autostash was indeed not applied, and the
> 'git stash list' was to check if the autostash was at least available
> in the stash list (it's not).

Ahh, I thought you expected `git stash list` to show something. If this
is the bug then yeah, I can reproduce.

> > + git status
> > On branch master
> > Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
> >   (use "git pull" to update your local branch)
> > 
> > Untracked files:
> >   (use "git add <file>..." to include in what will be committed)
> >     other
> > 
> > nothing added to commit but untracked files present (use "git add" to track)
> > + git stash list
> > # empty!
> > ~~~ 
> 
> Back to your reply:
> 
> > 
> > Moreover, this is not an issue of `git pull`, but `git merge`.
> > 
> > I can reproduce the problem that the modifications are lost like this:
> > 
> >    git init test
> >    (
> >      cd test
> >      date >> file
> >      git add file
> >      git commit -m 'add file'
> >      date >> other
> >      git add other
> >      git commit -m 'add other'
> >      git checkout -b topic @~
> >      date >> other
> >      date >> file
> >      git status
> >      git "$@" master
> >      git status
> >      git stash list
> >    )
> > 
> > Running this with 'rebase --autostash' fails and nothing is put in the
> > stash list, but the modifications to 'file' remain. I think this is the
> > correct behavior.
> 
> Yes. We see the following output:
> 
>      Created autostash: 69775ee
>      Applied autostash.
>      Successfully rebased and updated refs/heads/topic.
> 
> The code correctly applied the autostash after the rebase, so it's
> normal it's not kept in the stash list. I agree it's the correct behaviour.

Good.

> > But with 'merge --autostash' the modifications to 'file' are lost. That
> > is a bug, and it's already present in v2.32.
> 
> That's also true. From what I understand it dates back to the introduction
> of 'git merge --autostash'.

Yes, that's probably true.

> > Do you agree?
> 
> I agree that the bug with 'merge --autostash' was already present in v2.32.0.
> But since your 340062243a (pull: cleanup autostash check, 2021-06-17),
> 'git pull --rebase --autostash' calls 'git merge --autostash' in the fast-forward
> case, and so we hit the bug, which was not the case before since the fast-forward
> merge shortcut was skipped if '--autostash' was given for 'git pull --rebase'.

Yes we hit the bug with `git pull`, but the bug was there already when
people did `git merge`.

> The fix I suggested in [1] seems to fix both cases, but as I wrote there
> it still leaves two code paths that might trigger the bug. I'm not familiar
> at all with the code for these two code paths, so I'm not ready to send
> a formal patch; I thought I'd send the diff anyway if you or Denton (or anyone
> else) wants to start from there.

Unfortunately I'm not familiar with the `git merge` code either, I've
only been modifying the `git pull` code, so I would also be starting
from scratch if I tried to fix that myself.

But I think that's where it should be fixed.

Cheers.

-- 
Felipe Contreras

Re: Regression in 'git pull --rebase --autostash' since v2.32.0

[Philippe Blain]

Hi Felipe,

Le 2021-07-17 à 23:05, Felipe Contreras a écrit :
> Philippe Blain wrote:
>> Le 2021-07-17 à 15:34, Felipe Contreras a écrit :
> 
>> The fix I suggested in [1] seems to fix both cases, but as I wrote there
>> it still leaves two code paths that might trigger the bug. I'm not familiar
>> at all with the code for these two code paths, so I'm not ready to send
>> a formal patch; I thought I'd send the diff anyway if you or Denton (or anyone
>> else) wants to start from there.
> 
> Unfortunately I'm not familiar with the `git merge` code either, I've
> only been modifying the `git pull` code, so I would also be starting
> from scratch if I tried to fix that myself.
> 
> But I think that's where it should be fixed.

I've just sent a patch series that fixes it:

https://lore.kernel.org/git/pull.1002.git.1627042470.gitgitgadget@gmail.com/T/#t

Cheers,

Philippe.

Re: Regression in 'git pull --rebase --autostash' since v2.32.0

[Felipe Contreras]

Philippe Blain wrote:
> Le 2021-07-17 à 23:05, Felipe Contreras a écrit :
> > Philippe Blain wrote:
> >> Le 2021-07-17 à 15:34, Felipe Contreras a écrit :
> > 
> >> The fix I suggested in [1] seems to fix both cases, but as I wrote there
> >> it still leaves two code paths that might trigger the bug. I'm not familiar
> >> at all with the code for these two code paths, so I'm not ready to send
> >> a formal patch; I thought I'd send the diff anyway if you or Denton (or anyone
> >> else) wants to start from there.
> > 
> > Unfortunately I'm not familiar with the `git merge` code either, I've
> > only been modifying the `git pull` code, so I would also be starting
> > from scratch if I tried to fix that myself.
> > 
> > But I think that's where it should be fixed.
> 
> I've just sent a patch series that fixes it:
> 
> https://lore.kernel.org/git/pull.1002.git.1627042470.gitgitgadget@gmail.com/T/#t

Thank you. I've reviewed the series and looks fine to me.

-- 
Felipe Contreras