Lost files after git stash && git stash pop

Lost files after git stash && git stash pop

[Till Friebe]

Thank you for filling out a Git bug report!
Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)
```
git init
mkdir README
touch README/README
git add .
git commit -m "Init project"
echo "Test" > README/README
mv README/README README2
rmdir README
mv README2 README
git stash 
git stash pop
```

What did you expect to happen? (Expected behavior)
I expected that after the `git stash pop` the README file would be back.

What happened instead? (Actual behavior)
This README with "Test" file was deleted and I lost 5 hours of work.

What's different between what you expected and what actually happened?
The file doesn't exist anymore and I can't recover it.

Anything else you want to add:
This is just a reproducible example.

Please review the rest of the bug report below.
You can delete any lines you don't wish to share.


[System Info]
git version:
git version 2.39.2 (Apple Git-143)
cpu: arm64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon
uname: Darwin 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:20 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6000 arm64
compiler info: clang: 14.0.3 (clang-1403.0.22.14.1)
libc info: no libc information available
$SHELL (typically, interactive shell): /bin/zsh


[Enabled Hooks]

Re: Lost files after git stash && git stash pop

[Torsten Bögershausen]

On Fri, Jul 21, 2023 at 07:31:53PM +0200, Till Friebe wrote:
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
>
> What did you do before the bug happened? (Steps to reproduce your issue)
> ```
> git init
> mkdir README
> touch README/README
> git add .
> git commit -m "Init project"
> echo "Test" > README/README
> mv README/README README2
> rmdir README
> mv README2 README
> git stash
> git stash pop
> ```
>
> What did you expect to happen? (Expected behavior)
> I expected that after the `git stash pop` the README file would be back.
>
> What happened instead? (Actual behavior)
> This README with "Test" file was deleted and I lost 5 hours of work.

That is always sad to hear, when work is lost.

However, I personally wonder if this is a bug or not.
First, Git is told to track a file called README/README
Then the file is removed, without telling Git.
And a new, unkown file appers on disk (which collides with the name
of the directory)

Using this sequence could have told Git, what is going on:
git mv README/README README2
rmdir README
git mv README2 README

(a temporary branch may be checked out, with the option
 to merge-squash the final result)


An other alternative could be to tell `git stash` to care
about untracked file(s):

git stash -u
git stash pop

Which will refuse to apply the stash.

A third alternative could be to keep the file inside an
editor, to have the content still available.

However, it would/could be nice, if files are not simply deleted,
but saved into a "lost+found" folder, or a wastebasket kind of thing.

But which files ?
Those that are untracked ?
They may be important (local config files, passwords, help scripts, ...)
or not (.o files from a C compiler).

In some older discussions they had been named "precious" files.
But, as far as I remember, there was no easy solution.
In that sense I don't have a better answer.
Others may have.

Thanks for reporting, it make me read [1] and come to the conclusion
that it is sometimes safer to checkout out a temporary branch, commit
everything and clean up later, rather than relying too much on
`git stash`


<https://stackoverflow.com/questions/835501/how-do-you-stash-an-untracked-file>
>
> What's different between what you expected and what actually happened?
> The file doesn't exist anymore and I can't recover it.
>
> Anything else you want to add:
> This is just a reproducible example.
>

Re: Lost files after git stash && git stash pop

[Phillip Wood]

On 22/07/2023 22:44, Torsten Bögershausen wrote:
> On Fri, Jul 21, 2023 at 07:31:53PM +0200, Till Friebe wrote:
>> Thank you for filling out a Git bug report!
>> Please answer the following questions to help us understand your issue.
>>
>> What did you do before the bug happened? (Steps to reproduce your issue)
>> ```
>> git init
>> mkdir README
>> touch README/README
>> git add .
>> git commit -m "Init project"
>> echo "Test" > README/README
>> mv README/README README2
>> rmdir README
>> mv README2 README
>> git stash
>> git stash pop
>> ```
>>
>> What did you expect to happen? (Expected behavior)
>> I expected that after the `git stash pop` the README file would be back.
>>
>> What happened instead? (Actual behavior)
>> This README with "Test" file was deleted and I lost 5 hours of work.
> 
> That is always sad to hear, when work is lost.

Indeed it is. Thanks Till for providing an easy reproducer.

> However, I personally wonder if this is a bug or not.

I think whenever git overwrites an untracked file without the user 
passing some option indicating that they want to do so it is a bug. For 
example "git checkout" refuses to overwrite untracked files by default. 
Sadly this seems to be a known bug in do_push_stash() where we are using 
"git reset --hard" to remove the stashed changes from the working copy. 
This was documented in 94b7f1563a (Comment important codepaths regarding 
nuking untracked files/dirs, 2021-09-27). The stash implementation does 
a lot of necessary forking of subprocesses, in this case I think it 
would be better to call unpack_trees() directly with 
UNPACK_RESET_PROTECT_UNTRACKED.

Best Wishes

Phillip

> First, Git is told to track a file called README/README
> Then the file is removed, without telling Git.
> And a new, unkown file appers on disk (which collides with the name
> of the directory)
> 
> Using this sequence could have told Git, what is going on:
> git mv README/README README2
> rmdir README
> git mv README2 README
> 
> (a temporary branch may be checked out, with the option
>   to merge-squash the final result)
> 
> 
> An other alternative could be to tell `git stash` to care
> about untracked file(s):
> 
> git stash -u
> git stash pop
> 
> Which will refuse to apply the stash.
> 
> A third alternative could be to keep the file inside an
> editor, to have the content still available.
> 
> However, it would/could be nice, if files are not simply deleted,
> but saved into a "lost+found" folder, or a wastebasket kind of thing.
> 
> But which files ?
> Those that are untracked ?
> They may be important (local config files, passwords, help scripts, ...)
> or not (.o files from a C compiler).
> 
> In some older discussions they had been named "precious" files.
> But, as far as I remember, there was no easy solution.
> In that sense I don't have a better answer.
> Others may have.
> 
> Thanks for reporting, it make me read [1] and come to the conclusion
> that it is sometimes safer to checkout out a temporary branch, commit
> everything and clean up later, rather than relying too much on
> `git stash`
> 
> 
> <https://stackoverflow.com/questions/835501/how-do-you-stash-an-untracked-file>
>>
>> What's different between what you expected and what actually happened?
>> The file doesn't exist anymore and I can't recover it.
>>
>> Anything else you want to add:
>> This is just a reproducible example.
>>

Re: Lost files after git stash && git stash pop

[Torsten Bögershausen]

On Sun, Jul 23, 2023 at 11:01:29AM +0100, Phillip Wood wrote:
> On 22/07/2023 22:44, Torsten Bögershausen wrote:
> > On Fri, Jul 21, 2023 at 07:31:53PM +0200, Till Friebe wrote:
> > > Thank you for filling out a Git bug report!
> > > Please answer the following questions to help us understand your issue.
> > >
> > > What did you do before the bug happened? (Steps to reproduce your issue)
> > > ```
> > > git init
> > > mkdir README
> > > touch README/README
> > > git add .
> > > git commit -m "Init project"
> > > echo "Test" > README/README
> > > mv README/README README2
> > > rmdir README
> > > mv README2 README
> > > git stash
> > > git stash pop
> > > ```
> > >
> > > What did you expect to happen? (Expected behavior)
> > > I expected that after the `git stash pop` the README file would be back.
> > >
> > > What happened instead? (Actual behavior)
> > > This README with "Test" file was deleted and I lost 5 hours of work.
> >
> > That is always sad to hear, when work is lost.
>
> Indeed it is. Thanks Till for providing an easy reproducer.
>
> > However, I personally wonder if this is a bug or not.
>
> I think whenever git overwrites an untracked file without the user passing
> some option indicating that they want to do so it is a bug.

OK, agreed after reading the next sentence.

> For example "git
> checkout" refuses to overwrite untracked files by default. Sadly this seems
> to be a known bug in do_push_stash() where we are using "git reset --hard"
> to remove the stashed changes from the working copy. This was documented in
> 94b7f1563a (Comment important codepaths regarding nuking untracked
> files/dirs, 2021-09-27). The stash implementation does a lot of necessary
> forking of subprocesses, in this case I think it would be better to call
> unpack_trees() directly with UNPACK_RESET_PROTECT_UNTRACKED.

Thanks for the fast response.

This is not an area of Git, where I have much understanding of the code.
But is seems as if pop_stash() in builtin/stash.c
(and the called functions) seems to be the problem here ?

Re: Lost files after git stash && git stash pop

[Phillip Wood]

Hi Torsten

On 23/07/2023 21:52, Torsten Bögershausen wrote:
>> I think whenever git overwrites an untracked file without the user passing
>> some option indicating that they want to do so it is a bug.
> 
> OK, agreed after reading the next sentence.
> 
>> For example "git
>> checkout" refuses to overwrite untracked files by default. Sadly this seems
>> to be a known bug in do_push_stash() where we are using "git reset --hard"
>> to remove the stashed changes from the working copy. This was documented in
>> 94b7f1563a (Comment important codepaths regarding nuking untracked
>> files/dirs, 2021-09-27). The stash implementation does a lot of necessary
>> forking of subprocesses, in this case I think it would be better to call
>> unpack_trees() directly with UNPACK_RESET_PROTECT_UNTRACKED.
> 
> Thanks for the fast response.
> 
> This is not an area of Git, where I have much understanding of the code.
> But is seems as if pop_stash() in builtin/stash.c
> (and the called functions) seems to be the problem here ?

Confusingly it is creating the stash that deletes the untracked file because
it recreates README/README. do_push_stash() in builtin/stash.c is the culprit
I think. I had hoped the diff below would fix the problem but it does not
seem to and breaks half a dozen test cases that seem to rely on removing
untracked files. Unfortunately I don't really have time to dig any
further at the moment.

Best Wishes

Phillip


diff --git a/builtin/stash.c b/builtin/stash.c
index fe64cde9ce3..c8bbfe56d26 100644
--- a/builtin/stash.c
+++ b/builtin/stash.c
@@ -29,6 +29,8 @@
  #include "exec-cmd.h"
  #include "reflog.h"
  #include "add-interactive.h"
+#include "reset.h"
+#include "submodule.h"
  
  #define INCLUDE_ALL_FILES 2
  
@@ -336,7 +338,7 @@ static int apply_cached(struct strbuf *out)
         return pipe_command(&cp, out->buf, out->len, NULL, 0, NULL, 0);
  }
  
-static int reset_head(void)
+static int stash_reset_head(void)
  {
         struct child_process cp = CHILD_PROCESS_INIT;
  
@@ -569,7 +571,7 @@ static int do_apply_stash(const char *prefix, struct stash_info *info,
                                                 get_index_file(), 0, NULL))
                                 return error(_("could not save index tree"));
  
-                       reset_head();
+                       stash_reset_head();
                         discard_index(&the_index);
                         repo_read_index(the_repository);
                 }
@@ -1649,12 +1651,14 @@ static int do_push_stash(const struct pathspec *ps, const char *stash_msg, int q
                                 goto done;
                         }
                 } else {
-                       struct child_process cp = CHILD_PROCESS_INIT;
-                       cp.git_cmd = 1;
-                       /* BUG: this nukes untracked files in the way */
-                       strvec_pushl(&cp.args, "reset", "--hard", "-q",
-                                    "--no-recurse-submodules", NULL);
-                       if (run_command(&cp)) {
+                       struct reset_head_opts opts = {
+                               .flags = RESET_HEAD_HARD,
+                       };
+
+                       if (should_update_submodules())
+                               BUG("stash should not update submodules");
+
+                       if (reset_head(the_repository, &opts)) {
                                 ret = -1;
                                 goto done;
                         }

[PATCH v1 1/1] git stash needing mkdir deletes untracked file

[tboegi]

From: Torsten Bögershausen <tboegi@web.de>

The following sequence leads to loss of work:
 git init
 mkdir README
 touch README/README
 git add .
 git commit -m "Init project"
 echo "Test" > README/README
 mv README/README README2
 rmdir README
 mv README2 README
 git stash
 git stash pop

The problem is, that `git stash` needs to create the directory README/
and to be able to do this, the file README needs to be removed.
And this is, where the work was lost.
There are different possibilities preventing this loss of work:
a)
  `git stash` does refuse the removel of the untracked file,
   when a directory with the same name needs to be created
  There is a small problem here:
  In the ideal world, the stash would do nothing at all,
  and not do anything but complain.
  The current code makes this hard to achieve
  An other solution could be to do as much stash work as possible,
  but stop when the file/directory conflict is detected.
  This would create some inconsistent state.

b) Create the directory as needed, but rename the file before doing that.
  This would let the `git stash` proceed as usual and create a "new" file,
  which may be surprising for some worlflows.

This change goes for b), as it seems the most intuitive solution for
Git users.

Introdue a new function rename_to_untracked_or_warn() and use it
in create_directories() in entry.c

Reported-by: Till Friebe <friebetill@gmail.com>
Signed-off-by: Torsten Bögershausen <tboegi@web.de>
---
 entry.c          | 25 ++++++++++++++++++++++++-
 t/t3903-stash.sh | 23 +++++++++++++++++++++++
 2 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/entry.c b/entry.c
index 43767f9043..76d8a0762d 100644
--- a/entry.c
+++ b/entry.c
@@ -15,6 +15,28 @@
 #include "entry.h"
 #include "parallel-checkout.h"

+static int rename_to_untracked_or_warn(const char *file)
+{
+	const size_t file_name_len = strlen(file);
+	const static char *dot_untracked = ".untracked";
+	const size_t dot_un_len = strlen(dot_untracked);
+	struct strbuf sb;
+	int ret;
+
+	strbuf_init(&sb, file_name_len + dot_un_len);
+	strbuf_add(&sb, file, file_name_len);
+	strbuf_add(&sb, dot_untracked, dot_un_len);
+	ret = rename(file, sb.buf);
+
+	if (ret) {
+		int saved_errno = errno;
+		warning_errno(_("unable rename '%s' into '%s'"), file, sb.buf);
+		errno = saved_errno;
+	}
+	strbuf_release(&sb);
+	return ret;
+}
+
 static void create_directories(const char *path, int path_len,
 			       const struct checkout *state)
 {
@@ -48,7 +70,8 @@ static void create_directories(const char *path, int path_len,
 		 */
 		if (mkdir(buf, 0777)) {
 			if (errno == EEXIST && state->force &&
-			    !unlink_or_warn(buf) && !mkdir(buf, 0777))
+			    !rename_to_untracked_or_warn(buf) &&
+			    !mkdir(buf, 0777))
 				continue;
 			die_errno("cannot create directory at '%s'", buf);
 		}
diff --git a/t/t3903-stash.sh b/t/t3903-stash.sh
index 0b3dfeaea2..1a210f8a5a 100755
--- a/t/t3903-stash.sh
+++ b/t/t3903-stash.sh
@@ -1512,4 +1512,27 @@ test_expect_success 'restore untracked files even when we hit conflicts' '
 	)
 '

+test_expect_success 'stash mkdir README needed - README.untracked created' '
+	git init mkdir_needed_file_untracked &&
+	(
+		cd mkdir_needed_file_untracked &&
+		mkdir README &&
+		touch README/README &&
+		git add . &&
+		git commit -m "Add README/README" &&
+		echo Version2 > README/README &&
+		mv README/README README2 &&
+		rmdir README &&
+		mv README2 README &&
+		git stash &&
+		test_path_is_file README.untracked &&
+		echo Version2 >expect &&
+		test_cmp expect README.untracked &&
+		rm expect &&
+		git stash pop &&
+		test_path_is_file README.untracked &&
+		echo Version2 >expect &&
+		test_cmp expect README.untracked
+	)
+'
 test_done
--
2.41.0.394.ge43f4fd0bd

Re: [PATCH v1 1/1] git stash needing mkdir deletes untracked file

[Torsten Bögershausen]

On Tue, Aug 08, 2023 at 07:26:24PM +0200, tboegi@web.de wrote:
> From: Torsten Bögershausen <tboegi@web.de>
> .

... The following sequence leads to loss of work:

I just realized that this breaks other tests:
t1092-sparse-checkout-compatibility.sh not ok 17 - diff with renames and conflicts
t1092-sparse-checkout-compatibility.sh not ok 18 - diff with directory/file conflicts

However, comments are welcome:
Is it a good idea to create file called "filename.untracked" ?

Re: [PATCH v1 1/1] git stash needing mkdir deletes untracked file

[Eric Sunshine]

On Tue, Aug 8, 2023 at 3:15 PM <tboegi@web.de> wrote:
> The following sequence leads to loss of work:
>  git init
>  mkdir README
>  touch README/README
>  git add .
>  git commit -m "Init project"
>  echo "Test" > README/README
>  mv README/README README2
>  rmdir README
>  mv README2 README
>  git stash
>  git stash pop
>
> The problem is, that `git stash` needs to create the directory README/
> and to be able to do this, the file README needs to be removed.
> And this is, where the work was lost.
> There are different possibilities preventing this loss of work:
> a)
>   `git stash` does refuse the removel of the untracked file,

s/removel/removal/

>    when a directory with the same name needs to be created

s/$/./

>   There is a small problem here:
>   In the ideal world, the stash would do nothing at all,
>   and not do anything but complain.
>   The current code makes this hard to achieve

s/$/./

>   An other solution could be to do as much stash work as possible,

s/An other/Another/

>   but stop when the file/directory conflict is detected.
>   This would create some inconsistent state.
>
> b) Create the directory as needed, but rename the file before doing that.
>   This would let the `git stash` proceed as usual and create a "new" file,
>   which may be surprising for some worlflows.

s/worlflows/workflows/

> This change goes for b), as it seems the most intuitive solution for
> Git users.
>
> Introdue a new function rename_to_untracked_or_warn() and use it

s/Introdue/Introduce/

> in create_directories() in entry.c
>
> Reported-by: Till Friebe <friebetill@gmail.com>
> Signed-off-by: Torsten Bögershausen <tboegi@web.de>
> ---
> diff --git a/entry.c b/entry.c
> @@ -15,6 +15,28 @@
> +static int rename_to_untracked_or_warn(const char *file)
> +{
> +       const size_t file_name_len = strlen(file);
> +       const static char *dot_untracked = ".untracked";
> +       const size_t dot_un_len = strlen(dot_untracked);
> +       struct strbuf sb;
> +       int ret;
> +
> +       strbuf_init(&sb, file_name_len + dot_un_len);
> +       strbuf_add(&sb, file, file_name_len);
> +       strbuf_add(&sb, dot_untracked, dot_un_len);
> +       ret = rename(file, sb.buf);

This could probably all be simplified to:

    char *to = xstrfmt("%s.untracked", file);
    ret = rename(...);
    ...
    free(to);

If there is already a file named "foo.untracked", then this will
overwrite it, thus potentially losing work, right? I wonder if it
makes sense to be a bit more careful.

> +       if (ret) {
> +               int saved_errno = errno;
> +               warning_errno(_("unable rename '%s' into '%s'"), file, sb.buf);
> +               errno = saved_errno;
> +       }
> +       strbuf_release(&sb);
> +       return ret;
> +}

Do we want to give the user some warning/notification that their file,
as a safety precaution, got renamed to "foo.untracked"?

> diff --git a/t/t3903-stash.sh b/t/t3903-stash.sh
> @@ -1512,4 +1512,27 @@ test_expect_success 'restore untracked files even when we hit conflicts' '
> +test_expect_success 'stash mkdir README needed - README.untracked created' '
> +       git init mkdir_needed_file_untracked &&
> +       (
> +               cd mkdir_needed_file_untracked &&
> +               mkdir README &&
> +               touch README/README &&

s/touch/>/

> +               git add . &&
> +               git commit -m "Add README/README" &&
> +               echo Version2 > README/README &&

s/> R/>R/

> +               mv README/README README2 &&
> +               rmdir README &&
> +               mv README2 README &&
> +               git stash &&
> +               test_path_is_file README.untracked &&
> +               echo Version2 >expect &&
> +               test_cmp expect README.untracked &&
> +               rm expect &&
> +               git stash pop &&
> +               test_path_is_file README.untracked &&
> +               echo Version2 >expect &&
> +               test_cmp expect README.untracked
> +       )
> +'

Re: [PATCH v1 1/1] git stash needing mkdir deletes untracked file

[Phillip Wood]

Hi Torsten

Thanks for working on this. I've cc'd Junio for his unpack_trees() 
knowledge.

On 08/08/2023 18:26, tboegi@web.de wrote:
> From: Torsten Bögershausen <tboegi@web.de>
> 
> The following sequence leads to loss of work:
>   git init
>   mkdir README
>   touch README/README
>   git add .
>   git commit -m "Init project"
>   echo "Test" > README/README
>   mv README/README README2
>   rmdir README
>   mv README2 README
>   git stash
>   git stash pop
> 
> The problem is, that `git stash` needs to create the directory README/
> and to be able to do this, the file README needs to be removed.
> And this is, where the work was lost.
> There are different possibilities preventing this loss of work:
> a)
>    `git stash` does refuse the removel of the untracked file,
>     when a directory with the same name needs to be created
>    There is a small problem here:
>    In the ideal world, the stash would do nothing at all,
>    and not do anything but complain.
>    The current code makes this hard to achieve
>    An other solution could be to do as much stash work as possible,
>    but stop when the file/directory conflict is detected.
>    This would create some inconsistent state.
> 
> b) Create the directory as needed, but rename the file before doing that.
>    This would let the `git stash` proceed as usual and create a "new" file,
>    which may be surprising for some worlflows.
> 
> This change goes for b), as it seems the most intuitive solution for
> Git users.
> 
> Introdue a new function rename_to_untracked_or_warn() and use it
> in create_directories() in entry.c

Although this change is framed in terms of changes to "git stash push" I 
think the underlying issue and this patch actually affects all users of 
unpack_trees(). For example if "README" is untracked then

	git checkout <rev> README

will currently fail if <rev>:README is a blob but will succeed and 
remove the untracked file if <rev>:README is a tree.

I'm far from an expert in this area but I think we might want to 
understand why unpack_trees() sets state->force when it calls 
checkout_entry() before making any changes.

Best Wishes

Phillip

> Reported-by: Till Friebe <friebetill@gmail.com>
> Signed-off-by: Torsten Bögershausen <tboegi@web.de>
> ---
>   entry.c          | 25 ++++++++++++++++++++++++-
>   t/t3903-stash.sh | 23 +++++++++++++++++++++++
>   2 files changed, 47 insertions(+), 1 deletion(-)
> 
> diff --git a/entry.c b/entry.c
> index 43767f9043..76d8a0762d 100644
> --- a/entry.c
> +++ b/entry.c
> @@ -15,6 +15,28 @@
>   #include "entry.h"
>   #include "parallel-checkout.h"
> 
> +static int rename_to_untracked_or_warn(const char *file)
> +{
> +	const size_t file_name_len = strlen(file);
> +	const static char *dot_untracked = ".untracked";
> +	const size_t dot_un_len = strlen(dot_untracked);
> +	struct strbuf sb;
> +	int ret;
> +
> +	strbuf_init(&sb, file_name_len + dot_un_len);
> +	strbuf_add(&sb, file, file_name_len);
> +	strbuf_add(&sb, dot_untracked, dot_un_len);
> +	ret = rename(file, sb.buf);
> +
> +	if (ret) {
> +		int saved_errno = errno;
> +		warning_errno(_("unable rename '%s' into '%s'"), file, sb.buf);
> +		errno = saved_errno;
> +	}
> +	strbuf_release(&sb);
> +	return ret;
> +}
> +
>   static void create_directories(const char *path, int path_len,
>   			       const struct checkout *state)
>   {
> @@ -48,7 +70,8 @@ static void create_directories(const char *path, int path_len,
>   		 */
>   		if (mkdir(buf, 0777)) {
>   			if (errno == EEXIST && state->force &&
> -			    !unlink_or_warn(buf) && !mkdir(buf, 0777))
> +			    !rename_to_untracked_or_warn(buf) &&
> +			    !mkdir(buf, 0777))
>   				continue;
>   			die_errno("cannot create directory at '%s'", buf);
>   		}
> diff --git a/t/t3903-stash.sh b/t/t3903-stash.sh
> index 0b3dfeaea2..1a210f8a5a 100755
> --- a/t/t3903-stash.sh
> +++ b/t/t3903-stash.sh
> @@ -1512,4 +1512,27 @@ test_expect_success 'restore untracked files even when we hit conflicts' '
>   	)
>   '
> 
> +test_expect_success 'stash mkdir README needed - README.untracked created' '
> +	git init mkdir_needed_file_untracked &&
> +	(
> +		cd mkdir_needed_file_untracked &&
> +		mkdir README &&
> +		touch README/README &&
> +		git add . &&
> +		git commit -m "Add README/README" &&
> +		echo Version2 > README/README &&
> +		mv README/README README2 &&
> +		rmdir README &&
> +		mv README2 README &&
> +		git stash &&
> +		test_path_is_file README.untracked &&
> +		echo Version2 >expect &&
> +		test_cmp expect README.untracked &&
> +		rm expect &&
> +		git stash pop &&
> +		test_path_is_file README.untracked &&
> +		echo Version2 >expect &&
> +		test_cmp expect README.untracked
> +	)
> +'
>   test_done
> --
> 2.41.0.394.ge43f4fd0bd
> 

Re: [PATCH v1 1/1] git stash needing mkdir deletes untracked file

[Torsten Bögershausen]

On Wed, Aug 09, 2023 at 02:15:28PM +0100, Phillip Wood wrote:
> Hi Torsten
>
> Thanks for working on this. I've cc'd Junio for his unpack_trees()
> knowledge.

Thanks Eric for the review.

Hej Phillip,
I have been playing around with the whole thing some time.
At the end I had a version, which did fiddle the information
that we are doing a `git stash` (and not any other operation)
into entry.c, and all test cases passed.
So in principle I can dig out all changes, polish them
and send them out, after doing cleanups of course.

(And that could take a couple of days, or weeks ;-)

My main question is still open:
Is it a good idea, to create a "helper file" ?
The naming can be discussed, we may stick the date/time
into the filename to make it really unique, or so.

Reading the different reports and including own experience,
I still think that a directory called ".deleted-by-user"
or ".wastebin" or something in that style is a good idea.

What do others think ?

Re: [PATCH v1 1/1] git stash needing mkdir deletes untracked file

[Phillip Wood]

Hi Torsten

Sorry for the slow reply

On 09/08/2023 19:47, Torsten Bögershausen wrote:
> On Wed, Aug 09, 2023 at 02:15:28PM +0100, Phillip Wood wrote:
>> Hi Torsten
>>
>> Thanks for working on this. I've cc'd Junio for his unpack_trees()
>> knowledge.
> 
> Thanks Eric for the review.
> 
> Hej Phillip,
> I have been playing around with the whole thing some time.
> At the end I had a version, which did fiddle the information
> that we are doing a `git stash` (and not any other operation)
> into entry.c, and all test cases passed.
> So in principle I can dig out all changes, polish them
> and send them out, after doing cleanups of course.

I don't think we should be treating "git stash" as a special case here - 
commands like "git checkout" should not be removing untracked files 
unprompted either.

> (And that could take a couple of days, or weeks ;-)
> 
> My main question is still open:
> Is it a good idea, to create a "helper file" ?
> The naming can be discussed, we may stick the date/time
> into the filename to make it really unique, or so.

I think stopping and telling the user that the file would be overwritten 
as we do in other cases would be better.

> Reading the different reports and including own experience,
> I still think that a directory called ".deleted-by-user"
> or ".wastebin" or something in that style is a good idea.

I can see an argument for being able to opt-in to that for "git restore" 
and "git reset --hard" but that is a different problem to the one here.

Best Wishes

Phillip

Re: [PATCH v1 1/1] git stash needing mkdir deletes untracked file

[Torsten Bögershausen]

On Tue, Aug 15, 2023 at 10:15:37AM +0100, Phillip Wood wrote:
> Hi Torsten
>
> Sorry for the slow reply

No problem.
Thanks for the response, I think that we have an
agreement not to overwrite an untracked file, when a directory
with the same name needs to be created.

I try to come up with a patch series -
starting with the stash operation.

>
> On 09/08/2023 19:47, Torsten Bögershausen wrote:
> > On Wed, Aug 09, 2023 at 02:15:28PM +0100, Phillip Wood wrote:
> > > Hi Torsten
> > >
> > > Thanks for working on this. I've cc'd Junio for his unpack_trees()
> > > knowledge.
> >
> > Thanks Eric for the review.
> >
> > Hej Phillip,
> > I have been playing around with the whole thing some time.
> > At the end I had a version, which did fiddle the information
> > that we are doing a `git stash` (and not any other operation)
> > into entry.c, and all test cases passed.
> > So in principle I can dig out all changes, polish them
> > and send them out, after doing cleanups of course.
>
> I don't think we should be treating "git stash" as a special case here -
> commands like "git checkout" should not be removing untracked files
> unprompted either.
>
> > (And that could take a couple of days, or weeks ;-)
> >
> > My main question is still open:
> > Is it a good idea, to create a "helper file" ?
> > The naming can be discussed, we may stick the date/time
> > into the filename to make it really unique, or so.
>
> I think stopping and telling the user that the file would be overwritten as
> we do in other cases would be better.
>
> > Reading the different reports and including own experience,
> > I still think that a directory called ".deleted-by-user"
> > or ".wastebin" or something in that style is a good idea.
>
> I can see an argument for being able to opt-in to that for "git restore" and
> "git reset --hard" but that is a different problem to the one here.
>
> Best Wishes
>
> Phillip
>

Re: [PATCH v1 1/1] git stash needing mkdir deletes untracked file

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

> I don't think we should be treating "git stash" as a special case here
> - commands like "git checkout" should not be removing untracked files
> unprompted either.

Yeah, I tend to agree.  "git checkout branch path" should overwrite
a leftover "path" in the working tree in response to such an
explicit request, and that should equally apply for a request with
pathspec e.g. "git checkout branch .", as the latter is also an
explicit "please check out all paths out of the tree-ish of the
branch".

But "git checkout branch" in a working tree with untracked "path"
should not lose it if "branch" has it as a tracked file.

> I think stopping and telling the user that the file would be
> overwritten as we do in other cases would be better.

Yup, that is what we have done and probably one of the design
choices that made us successful.

>> Reading the different reports and including own experience,
>> I still think that a directory called ".deleted-by-user"
>> or ".wastebin" or something in that style is a good idea.
>
> I can see an argument for being able to opt-in to that for "git
> restore" and "git reset --hard" but that is a different problem to the
> one here.

Yeah, I tend to agree.  If anything, such a trash directory should
be kept out-of-line, not inside the working tree.  Perhaps in $HOME
or somewhere, and not necessarily tied to the use of Git, as the way
a file gets "deleted by user" is not necessarily limited to the use
of Git.

Re: [PATCH v1 1/1] git stash needing mkdir deletes untracked file

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

> Although this change is framed in terms of changes to "git stash push"
> I think the underlying issue and this patch actually affects all users
> of unpack_trees(). For example if "README" is untracked then
>
> 	git checkout <rev> README
>
> will currently fail if <rev>:README is a blob but will succeed and
> remove the untracked file if <rev>:README is a tree.

Very true, and with an .untracked file nobody asked Git to create,
presumably?  I am not sure if the updated behaviour is better than
the current behaviour.  

If "silent and unconditional removal" bothers us, I wonder if it is
a lot better approach to error out and have the user sort out the
mess, which is what we usually do when it gets tempting to "move it
away with an arbitrary rename" like this patch tries to do.  I
dunno.

Thanks.

Re: [PATCH v1 1/1] git stash needing mkdir deletes untracked file

[Phillip Wood]

On 09/08/2023 21:57, Junio C Hamano wrote:
> Phillip Wood <phillip.wood123@gmail.com> writes:
>
> If "silent and unconditional removal" bothers us, I wonder if it is
> a lot better approach to error out and have the user sort out the
> mess, which is what we usually 

Yes I think that would be a better approach.

Best Wishes

Phillip

> do when it gets tempting to "move it
> away with an arbitrary rename" like this patch tries to do.  I
> dunno.
> 
> Thanks.
> 
> 
>