Bug with .gitignore and branch switching

Bug with .gitignore and branch switching

[Nevada Sanchez]

Here's an easy to reproduce bug. It's the only example I know of where
git legitimately loses data in a way that is unrecoverable,
unexpected, and without warning.

```
git --version
# git version 2.12.0

mkdir git-demo
cd git-demo

git init

# Commit a file that will end up in .gitignore
echo 'original settings' > mine.conf
git add mine.conf
git commit -m "Unknowingly committed my settings."

echo '*.conf' > .gitignore
git add .gitignore
git commit -m "Users shouldn't commit their settings"

# Spin off a feature branch here (but don't check it out)
git branch feature

# Realize that we don't want that file committed
git rm mine.conf
git commit -m "Delete mine.conf"

echo 'Lots of laboriously tuned settings' > mine.conf

# Hop on the feature branch to do some work
git checkout feature

# Hmmm... My settings are gone
cat mine.conf
# original settings

# Lemme hop back
git checkout master

# Wait... they are gone for good!
cat mine.conf
# cat: mine.conf: No such file or directory
```

Re: Bug with .gitignore and branch switching

[Junio C Hamano]

Nevada Sanchez <sanchez.nevada@gmail.com> writes:

> Here's an easy to reproduce bug. It's the only example I know of where
> git legitimately loses data in a way that is unrecoverable,
> unexpected, and without warning.

This is an example of a user explicitly telling git to discard data
and git performing as it is told.

There is no "untracked but precious" vs "untracked and expendable"
difference in the current system.  An untracked file that matches
patterns listed in .gitignore is treated as the latter.

When you have an untracked file that .gitignore knows about in the
working tree while you are on "feature", if switching to another
branch requires to remove that file, the content there is deemed
expendable, because the user said so by listing it in .gitignore.

We've discussed the lack of "untracked but precious" class a few
times on the list in the past, but I do not recall the topic came up
in the recent past.  It perhaps is because nobody found that class
useful enough so far.

Re: Bug with .gitignore and branch switching

[Stefan Beller]

On Fri, Mar 17, 2017 at 2:23 PM, Junio C Hamano <gitster@pobox.com> wrote:
> We've discussed the lack of "untracked but precious" class a few
> times on the list in the past, but I do not recall the topic came up
> in the recent past.  It perhaps is because nobody found that class
> useful enough so far.

My gut reaction on reading the bug report was that the root cause is
git-checkout doing the wrong thing by default. (cf. Git-Merge-2017,
"What’s Wrong With Git?", I am not sure if the video is yet available)

One argument in that talk was that Git promises to do "work on multiple
branches in parallel (context-switched, single threaded)", and git-checkout
is the apparent command to switch to another context (branch).
However by putting away only tracked content, we miss
doing a proper context switch for untracked and ignored files.

That partial switch has advantages in the typical use case, e.g.
* compiled objects in the worktree may not need to be recompiled.
* no need to do work for the untracked files (e.g. move to a special
  location).

Both these reasons argue for performance, instead of "correctness"
in the sense of "easy-to-understand commands for top level principles".

And in that talk the presenter concluded that git-stash was only invented
to circumvent these "correctness" problems, such that if git-checkout
were to also (de)populate the untracked and ignored files on branch
switch we would not need git-stash, because git-checkout did it for you
already. And by the omission of git-stash and an apparent
easier-to-understand git-checkout the whole git suite would become
easier for users.

I further conclude that when git-checkout were to behave "correct" as
outlined above, then this class of bug reports would not occur.

Just food for thought.

Thanks,
Stefan

Re: Bug with .gitignore and branch switching

[Jonathan Nieder]

Junio C Hamano wrote:

> There is no "untracked but precious" vs "untracked and expendable"
> difference in the current system.  An untracked file that matches
> patterns listed in .gitignore is treated as the latter.
[...]
> We've discussed the lack of "untracked but precious" class a few
> times on the list in the past, but I do not recall the topic came up
> in the recent past.  It perhaps is because nobody found that class
> useful enough so far.

The most recent example I can find is 2010:
http://public-inbox.org/git/4C6A1C5B.4030304@workspacewhiz.com/.

It also came up in 2007:
http://public-inbox.org/git/C0E9F681E68D48EB8989022D11FEE3D1@ntdev.corp.microsoft.com/
Earlier in that year it even made the "What's not in 1.5.2" list.
http://public-inbox.org/git/11793556383977-git-send-email-junkio@cox.net/

Perhaps those references could be a useful starting point for an
interested person's thinking.

Jonathan

Re: Bug with .gitignore and branch switching

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> The most recent example I can find is 2010:
> http://public-inbox.org/git/4C6A1C5B.4030304@workspacewhiz.com/.
>
> It also came up in 2007:
> http://public-inbox.org/git/C0E9F681E68D48EB8989022D11FEE3D1@ntdev.corp.microsoft.com/
> Earlier in that year it even made the "What's not in 1.5.2" list.
> http://public-inbox.org/git/11793556383977-git-send-email-junkio@cox.net/
>
> Perhaps those references could be a useful starting point for an
> interested person's thinking.

Thanks for links.  It seems that my thinking back in 1.5.3 timeperiod
was to introduce "precious" attribute.

I noticed that among the four-message "What's not in 1.5.2" series,
3/4 has a large discussion that may be relevant to Brandon's
"submodule is-active" thing.

Re: Bug with .gitignore and branch switching

[Duy Nguyen]

On Sat, Mar 18, 2017 at 5:02 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Junio C Hamano wrote:
>
>> There is no "untracked but precious" vs "untracked and expendable"
>> difference in the current system.  An untracked file that matches
>> patterns listed in .gitignore is treated as the latter.
> [...]
>> We've discussed the lack of "untracked but precious" class a few
>> times on the list in the past, but I do not recall the topic came up
>> in the recent past.  It perhaps is because nobody found that class
>> useful enough so far.
>
> The most recent example I can find is 2010:
> http://public-inbox.org/git/4C6A1C5B.4030304@workspacewhiz.com/.
>
> It also came up in 2007:
> http://public-inbox.org/git/C0E9F681E68D48EB8989022D11FEE3D1@ntdev.corp.microsoft.com/
> Earlier in that year it even made the "What's not in 1.5.2" list.
> http://public-inbox.org/git/11793556383977-git-send-email-junkio@cox.net/
>
> Perhaps those references could be a useful starting point for an
> interested person's thinking.

I think I made it work in 2014 [1] using new "precious" attribute, but
never submitted it, probably because I was worried about the
interaction with untracked cache (adding .gitattributes as a new
dependency) though maybe we can avoid that by always checking for
preciousness after all the tree walking/filtering is done, either with
or without untracked cache. But I never addressed that loose end. Then
again, it could also be another useful starting point for interested
person's thinking ;-)

[1] https://github.com/pclouds/git/commit/0e7f7afa1879b055369ebd3f1224311c43c8a32b
-- 
Duy

Re: Bug with .gitignore and branch switching

[Nevada Sanchez]

On Fri, Mar 17, 2017 at 5:23 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Nevada Sanchez <sanchez.nevada@gmail.com> writes:
>
>> Here's an easy to reproduce bug. It's the only example I know of where
>> git legitimately loses data in a way that is unrecoverable,
>> unexpected, and without warning.
>
> This is an example of a user explicitly telling git to discard data
> and git performing as it is told.
>
> There is no "untracked but precious" vs "untracked and expendable"
> difference in the current system.  An untracked file that matches
> patterns listed in .gitignore is treated as the latter.
>
> When you have an untracked file that .gitignore knows about in the
> working tree while you are on "feature", if switching to another
> branch requires to remove that file, the content there is deemed
> expendable, because the user said so by listing it in .gitignore.
>
> We've discussed the lack of "untracked but precious" class a few
> times on the list in the past, but I do not recall the topic came up
> in the recent past.  It perhaps is because nobody found that class
> useful enough so far.

I must admit that I missed that attribute of .gitignore (i.e.
untracked and **expendable**). I have grown accustomed to Git being
rather conservative and erring on the side of not losing data unless
the user is doing something deliberate (for example, 'git clean' won't
work unless you force it, checkouts fail if they do anything that
might lose data... unless it is in .gitignore, as I just learned).
When I saw this behavior, I assumed that it was a bug.

This isn't necessarily a situation I need to have fixed--it is not
part of my workflow and since that fateful commit, all feature
branches checked out after the change to .gitignore will not have any
problems as I switch branches. It was an unfortunate surprise to one
of my co-workers, not long after I reassured him that git was
conservative and will almost never accidentally lose data (even if it
means going to 'git reflog').

In keeping with this spirit, I would tend to lean towards having
"untracked but precious" being the default behavior (more
conservative), and if a user wants "untracked but expendable"
behavior, then that case requires special effort from the user (like
learning about and using a new type of ignore file). My guess is that
if the user is both ignoring and committing something to their
repository, it is probably a mistake, and as that user, I would rather
discover that mistake early with loud warning messages (and/or a
suggestion to use an alternate ignore strategy, or config flag), than
learn about it by losing data.

In summary, I do not need this fixed for my workflow, but want to
bring it to light in case other users are being similarly surprised. I
struggle to guess how far reaching of an impact it would have on
existing users to change the default behavior, but it would probably
be less than that of the push default behavior change that happened
not too long ago.

A quick and easy immediate step is to make note of this behavior in
the very first sentence of gitignore(5):

> A gitignore file specifies intentionally untracked files that Git should ignore *and that Git is allowed to overwrite without warning*.

More details about untracked but expendable can be placed in the NOTES
section, but the last part of that sentence would be quite helpful.

Thank you,
-Nevada

Re: Bug with .gitignore and branch switching

[Jonathan Nieder]

Hi Nevada,

Nevada Sanchez wrote:

> # Commit a file that will end up in .gitignore
> echo 'original settings' > mine.conf
> git add mine.conf
> git commit -m "Unknowingly committed my settings."
>
> echo '*.conf' > .gitignore
> git add .gitignore
> git commit -m "Users shouldn't commit their settings"

Naming a file in .gitignore tells git that you do not want to track it
and are giving git permission to write over it.  This commonly happens
when people check in build products.  For example:

	git rm -f my-build-product
	echo /my-build-product >>.gitignore
	git commit -m "Remove generated my-build-product file"
	make my-build-product

	git checkout HEAD^

Without that rule, this 'git checkout' command would fail.

That said, there are some cases (e.g. the .conf file case you mention)
where a person would want git not to track a file but do not want to
give git permission to write over it.  As you've seen, .gitignore does
not work well for this. :/

Ideas for next steps:

 1. The gitignore(5) manpage does not do a good job of emphasizing
    that files named there are not precious and can be overwritten by
    git.  Do you have ideas for wording that would help with that?
    This would be especially welcome if you can phrase them in the
    form of a patch against Documentation/gitignore.txt.

 2. Occasionally people have mentioned the idea of a .gitprecious file
    listing precious files that git should not track and not overwrite
    (e.g., keys and other configuration files, IDE state, or metadata
    for another version control system being used in parallel).  Would
    you be interested in working on that?

Thanks and hope that helps,
Jonathan