git pull suggestion

git pull suggestion

[Aghiles]

Hello,

It would be nice to have _all_ the WIP conflicts listed when pulling.
As of now, one has to fix the currently showed conflict to see the next one.

If there is a way to do that, please advise.

Thanks,

  -- aghiles

Re: git pull suggestion

[Thomas Rast]

Aghiles wrote:
> 
> It would be nice to have _all_ the WIP conflicts listed when pulling.
> As of now, one has to fix the currently showed conflict to see the next one.

Are you using 'git pull --rebase' or the equivalent
branch.<name>.rebase setting?

If so, note that git-rebase (which does all the hard work) can't know
the later conflicts once it hits the first one: your resolution of the
first conflict constitutes the base onto which the further patches are
applied.  So depending on what changes you make during the resolution,
there may be more or fewer conflicts in the rest of the rebase.

If not, I can't see how your question makes sense as ordinary 'git
pull' does a merge, and during a 'git merge' there can only ever be
one conflict resolution phase.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: git pull suggestion

[Aghiles]

>>
>> It would be nice to have _all_ the WIP conflicts listed when pulling.
>> As of now, one has to fix the currently showed conflict to see the next one.
>
> Are you using 'git pull --rebase' or the equivalent
> branch.<name>.rebase setting?
>
> If so, note that git-rebase (which does all the hard work) can't know
> the later conflicts once it hits the first one: your resolution of the
> first conflict constitutes the base onto which the further patches are
> applied.  So depending on what changes you make during the resolution,
> there may be more or fewer conflicts in the rest of the rebase.
>
> If not, I can't see how your question makes sense as ordinary 'git
> pull' does a merge, and during a 'git merge' there can only ever be
> one conflict resolution phase.
>

Sorry, my explanation was not clear. I am talking about changes in the
working directory that are not in the index. So my working directory is
"dirty" and I just issue a 'git pull'. Because some files are not "up to date"
git would abort the pull, saying that a certain file is not "up to date".
So I was suggesting to list all the "problematic" files in one go instead.
Not a biggie of course.

  -- aghiles

Re: git pull suggestion

[Nicolas Sebrecht]

The 08/04/10, Aghiles wrote:
> >>
> >> It would be nice to have _all_ the WIP conflicts listed when pulling.
> >> As of now, one has to fix the currently showed conflict to see the next one.
> >
> > Are you using 'git pull --rebase' or the equivalent
> > branch.<name>.rebase setting?
> >
> > If so, note that git-rebase (which does all the hard work) can't know
> > the later conflicts once it hits the first one: your resolution of the
> > first conflict constitutes the base onto which the further patches are
> > applied.  So depending on what changes you make during the resolution,
> > there may be more or fewer conflicts in the rest of the rebase.
> >
> > If not, I can't see how your question makes sense as ordinary 'git
> > pull' does a merge, and during a 'git merge' there can only ever be
> > one conflict resolution phase.
> 
> Sorry, my explanation was not clear. I am talking about changes in the
> working directory that are not in the index. So my working directory is
> "dirty" and I just issue a 'git pull'. Because some files are not "up to date"
> git would abort the pull, saying that a certain file is not "up to date".
> So I was suggesting to list all the "problematic" files in one go instead.

Doesn't 'git status' ouput what you want ? Or am I out of the scope ?

-- 
Nicolas Sebrecht

Re: git pull suggestion

[Aghiles]

Nicolas wrote:
>>
>> Sorry, my explanation was not clear. I am talking about changes in the
>> working directory that are not in the index. So my working directory is
>> "dirty" and I just issue a 'git pull'. Because some files are not "up to date"
>> git would abort the pull, saying that a certain file is not "up to date".
>> So I was suggesting to list all the "problematic" files in one go instead.
>
> Doesn't 'git status' ouput what you want ? Or am I out of the scope ?
>

Kind of ! 'git status' will effectively tell you what files are modified but not
in the index. The problem is that there is no way to know what files are
in potential conflict with what is coming from 'git pull'. So basically, you can
have as many dirty files as you want in your working directory as long as
they are not conflicting with what's coming.
If dirty files are in conflict, then 'git pull' will complain, but slowly ... :)

Another way of looking at it: do we have a command to know if some files
that are _not_ in the index are in a conflict with some upstream repository.
I guess this would imply fetching and then doing some work but I am not a
git expert, not by a  stretch of the word!

  -- aghiles

Re: git pull suggestion

[Jeff King]

On Thu, Apr 08, 2010 at 11:06:42PM -0400, Aghiles wrote:

> Kind of ! 'git status' will effectively tell you what files are
> modified but not in the index. The problem is that there is no way to
> know what files are in potential conflict with what is coming from
> 'git pull'. So basically, you can have as many dirty files as you want
> in your working directory as long as they are not conflicting with
> what's coming.
> If dirty files are in conflict, then 'git pull' will complain, but
> slowly ... :)

I think I get what you are talking about now. Consider the following
example script, and let me know if it illustrates your problem:

  # define some handy functions
  content() {
    echo 1 $1 >file1
    echo 2 $1 >file2
  }
  commit() {
    git add file1 file2
    git commit -m "$1"
  }

  # make a new repo
  mkdir repo && cd repo && git init

  # make two diverged branches
  content one; commit one
  git checkout -b other
  content two; commit two
  git checkout master
  content three; commit three

  # now give us dirty working tree state on both files
  content four

  # and try to merge
  git merge other

I get:

  $ git merge other
  error: Your local changes to 'file1' would be overwritten by merge. Aborting.
  Please, commit your changes or stash them before you can merge.

But that's not the whole story. Once you fix that, you will see that
your local changes to 'file2' would be overwritten by the merge:

  $ git commit -m "commit file1" file1
  $ git merge other
  error: Your local changes to 'file2' would be overwritten by merge. Aborting.
  Please, commit your changes or stash them before you can merge.

And so on.

Notice that I didn't use "pull", but pull should invoke git-merge after
fetching from the remote. I assume this is the same message you are
talking about?

I agree it would be nicer if, on the error case, we kept going through
the index to find all of the other error cases, and printed them all.
The code for that is a bit tricky, though, as the unpack-trees merging
code which produces that message is a callback, and only sees one cache
entry at a time.

It is possible to manually get the answer you want, or close to it. You
are looking for the intersection of files modified by you and files
modified by the upstream. So:

  # unique list of modified working tree files and index entries
  $ (git diff-files --name-only;
     git diff-index --name-only HEAD
    ) | sort -u >us
  # files that will be changing as part of merge
  $ git diff-tree --name-only $HEAD_TO_MERGE_FROM | sort >them
  $ comm -12 us them

where $HEAD_TO_MERGE_FROM in my example would be "other", but in the
case of a pull, would probably be FETCH_HEAD.

It isn't 100% accurate, as there are some special cases that the merging
code handles (e.g., I think if the remote changed a file and we have the
same uncommitted change in our working tree, we may still allow that.
There are probably others, too). But it should give a general idea of
what is in conflict.

In practice, I have never actually wanted to this. The workflow goes
something like:

  (1) Run git merge foo (or git pull)

  (2) Oops, I have cruft in my working tree. What was it? Run git
      status.

  (3a) Oh, that cruft should have been committed. Make a commit (or
       commits). Go to (1), possibly still with some changes in
       the working tree.

       or

  (3b) Oh, that cruft is some change I want to carry forward in the
       working directory. Run git stash, repeat the pull, fix any
       merge conflicts, and then git stash apply.

So it doesn't really matter to me if there is 1 conflicting file or 100.
In most cases, the commits in (3a) will clean up all of it in one go.
Otherwise, I'll just stash it all and come back to it.

-Peff

Re: git pull suggestion

[Aghiles]

Jeff King <peff@peff.net> wrote:
>
> ...
>
> I get:
>
>  $ git merge other
>  error: Your local changes to 'file1' would be overwritten by merge. Aborting.
>  Please, commit your changes or stash them before you can merge.
>
> But that's not the whole story. Once you fix that, you will see that
> your local changes to 'file2' would be overwritten by the merge:
>
>  $ git commit -m "commit file1" file1
>  $ git merge other
>  error: Your local changes to 'file2' would be overwritten by merge. Aborting.
>  Please, commit your changes or stash them before you can merge.
>
> And so on.
>
> Notice that I didn't use "pull", but pull should invoke git-merge after
> fetching from the remote. I assume this is the same message you are
> talking about?

Exactly.

> It is possible to manually get the answer you want, or close to it. You
> are looking for the intersection of files modified by you and files
> modified by the upstream. So:
>
>  # unique list of modified working tree files and index entries
>  $ (git diff-files --name-only;
>     git diff-index --name-only HEAD
>    ) | sort -u >us
>  # files that will be changing as part of merge
>  $ git diff-tree --name-only $HEAD_TO_MERGE_FROM | sort >them
>  $ comm -12 us them
>
> where $HEAD_TO_MERGE_FROM in my example would be "other", but in the
> case of a pull, would probably be FETCH_HEAD.

Thanks a lot for this, I will try it.

> In practice, I have never actually wanted to this. The workflow goes
> something like:
>
>  (1) Run git merge foo (or git pull)
>
>  (2) Oops, I have cruft in my working tree. What was it? Run git
>      status.
>
>  (3a) Oh, that cruft should have been committed. Make a commit (or
>       commits). Go to (1), possibly still with some changes in
>       the working tree.
>
>       or
>
>  (3b) Oh, that cruft is some change I want to carry forward in the
>       working directory. Run git stash, repeat the pull, fix any
>       merge conflicts, and then git stash apply.
>
> So it doesn't really matter to me if there is 1 conflicting file or 100.
> In most cases, the commits in (3a) will clean up all of it in one go.
> Otherwise, I'll just stash it all and come back to it.
>

That is exactly my workflow and I am perfectly happy with that. The
problem is that I am putting git in the hands of svnites and
sometimes I have to address some usability issues like these.

It is another issue, but I feel that the 'dirty working directory' is
one of the major usability hurdles for people migrating from svn
and CVS (a git pull --merge-using-stash could address it, maybe).

  -- aghiles

Re: git pull suggestion

[Aghiles]

Jeff King <peff@peff.net> wrote:
> It is possible to manually get the answer you want, or close to it. You
> are looking for the intersection of files modified by you and files
> modified by the upstream. So:
>
>  # unique list of modified working tree files and index entries
>  $ (git diff-files --name-only;
>     git diff-index --name-only HEAD
>    ) | sort -u >us
>  # files that will be changing as part of merge
>  $ git diff-tree --name-only $HEAD_TO_MERGE_FROM | sort >them
>  $ comm -12 us them
>
> where $HEAD_TO_MERGE_FROM in my example would be "other", but in the
> case of a pull, would probably be FETCH_HEAD.

It works. You actually need the -r flag to 'git diff-tree' (recursive) in order
to list the actual files and not only their parent directories. So it becomes:

$ (git diff-files --name-only; git diff-index --name-only HEAD ) | sort -u>us
$ git diff-tree -r --name-only $HEAD_TO_MERGE_FROM | sort > them
$ comm -12 us them

Thank you very much,

  -- aghiles

Re: git pull suggestion

[Jeff King]

On Fri, Apr 09, 2010 at 03:33:35PM -0400, Aghiles wrote:

> It is another issue, but I feel that the 'dirty working directory' is
> one of the major usability hurdles for people migrating from svn
> and CVS (a git pull --merge-using-stash could address it, maybe).

I think this has been discussed before, but I couldn't find it in the
archives.

It is probably a little bit confusing because your pull will not
necessarily complete immediately. It may have conflicts, which you may
fix up, or you may "git reset --hard" to abort. But either way, you need
to remember that your dirty state was stashed and that you need to pull
it out after it's all done.

I think we would do better to tell the user about stash there, so they
can do it themselves. Then they know where their changes went and how to
get them back. Since v1.6.5.5, this error message now says:

  Your local changes to '%s' would be overwritten by merge.  Aborting.
  Please, commit your changes or stash them before you can merge.

What version of git are you using? If you (or others you are helping)
saw that message and it wasn't helpful, do you have any suggestions for
how to improve it?

-Peff

Re: git pull suggestion

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>   Your local changes to '%s' would be overwritten by merge.  Aborting.
>   Please, commit your changes or stash them before you can merge.
>
> What version of git are you using? If you (or others you are helping)
> saw that message and it wasn't helpful, do you have any suggestions for
> how to improve it?

I use 1.7.1-rc0 myself, but dropping comma after "Please" may make it a
bit easier to read IMHO ;-).

Re: git pull suggestion

[Aghiles]

King <peff@peff.net> writes:
> I think we would do better to tell the user about stash there, so they
> can do it themselves. Then they know where their changes went and how to
> get them back. Since v1.6.5.5, this error message now says:
>
>  Your local changes to '%s' would be overwritten by merge.  Aborting.
>  Please, commit your changes or stash them before you can merge.
>
> What version of git are you using? If you (or others you are helping)
> saw that message and it wasn't helpful, do you have any suggestions for
> how to improve it?

Yes we have the latest version and we do see this message. This helps
a bit. Although for people used to CVS/CVN the "stash" is yet another thing
to learn. There is also a high probability for new users to see this message
very early when using git and the question is always the same: why can't git
just merge with my files and show me the conflict?

(There was also some usability issues with the "stash", I remember people
loosing _untracked_ files but I am not sure if that was PEBKAC. First versions
of stash had a very friendly syntax that punished you by obliterating your files
if you made a typo and some are still traumatized.)

Simply put: in git, your working directory is a second class citizen and git
doesn't want to deal with it. Fundamentally, this is in a collision course with
what some users think about their work in progress.

I started a similar discussion had a couple years ago:
http://lists-archives.org/git/635926-git-pull-opinion.html

Back then, I was certain that 'git pull' should have an option to mix with a
dirty tree but now, after a couple years of using the tool, I am not certain
anymore. I am just reporting the biggest frustrations I see with new users.

Thanks,

  -- aghiles

Re: git pull suggestion

[Junio C Hamano]

Aghiles <aghilesk@gmail.com> writes:

> Although for people used to CVS/CVN the "stash" is yet another thing
> to learn. There is also a high probability for new users to see this message
> very early when using git and the question is always the same: why can't git
> just merge with my files and show me the conflict?

There actually are two answers to this question.

One is that it is not necessarily "can't", but rather "chooses not to".
If you are limiting yourself to CVS/SVN style of development, then we
certainly could do that.

Let's review what happens in CVS/SVN world when you "update" with dirty
working tree.  As you are familiar with CVS/SVN, you should be able to
follow this part quite easily.

You "updated" and were in sync with the central repository at some point,
let's call it O, and have some uncommitted changes in the work tree since
then.  In the meantime, other people worked on the project and made more
commits in the central repository.  The tip of the central repository is
at commit A.

         x
        / 
    ---O---X---X---X---A

You would want your "cvs update" to end up with a topology like this:

                         x'
                        / 
    ---O---X---X---X---A

where

        x' = merge-three-way(O, x, A)

That is, the files in the work tree are updated to contain whatever
was done by other people between O and A.

git does _not_ implement a handy Porcelain to do this, but we could script
it like this (I am only illustrating that it can be done, but I am leaving
the reason why git chooses not to to a later part of this message).

	#!/bin/sh
        # Usage: git-cvs-pull remote [refspec]

        # Fetch from the other
        git fetch "$@"
	# Figure out "A", i.e. the updated commit
	merge_head=$(sed -e '/	not-for-merge	/d' \
		-e 's/	.*//' .git/FETCH_HEAD | \
		tr '\012' ' ')
	# cvs/svn style pull will never have an octopus
        case "$merge_head" in
        ?*' '?*)	die "cannot merge more than one" ;;
        ?*)		;;
        *)		die "nothing to merge" ;;
        esac

	# Make sure it is cvs/svn-style pull.  That is, our commit must
        # be an ancestor of the updated commit
	test 0 = (git rev-list $merge_head..HEAD | wc -l) || die "you forked"

        # At this point, we know the topology is like this.
        #
        #         x
        #        / 
        #    ---O---X---X---X---A

	# Figure out the current branch
	branch=$(git symbolic-ref HEAD)

	# Checkout and detach to "A" while carrying the local changes.
        # This may leave conflicts but that is what the user is asking for.
        git checkout -m "$merge_head^0"

        # At this point, topology has become:
        #
        #                         x'
        #                        / 
        #    ---O---X---X---X---A
        #
        # We have detached HEAD at A but haven't updated the branch yet.

        case "$branch" in
        '')	;; # detached from the beginning
        ?*)	git update-ref -m "cvs-pull" "$branch" "$merge_head" ;;
	esac

Note that this was written in my MUA and is obviously untested ;-) but I
think you have also been around here long enough to understand the idea.

So that was one of the answers.  It's not "we can't do it", but is "in a
world with cvs/svn limitation, we could".

The other answer would initially appear a bit sad, but after you think
about it, it would turn into an enlightenment, especially for people whose
brains have rotten from years and years of CVS/SVN use.

If you are not limited to CVS/SVN style of development and have made
commits since you updated from the central repository the last time,
CVS/SVN style "update" is fundamentally impossible.

Again, you "updated" and were in sync with the central repository at some
point, let's call it O, and this time, made a few commits, ending with
commit B.  You further have some uncommitted changes in the work tree
since then.  In the meantime, other people worked on the project and made
more commits in the central repository.  Again, the tip of the central
repository is at commit A.

                   x
                  /
         Y---Y---B
        / 
    ---O---X---X---X---A

First, a simple question.

What kind of topology would you want to end up with?

Think.

	... you think for five minutes ...

	(page break)

	... and then you look at the answer ...

Yes, you want to have a merge between A and B, and then have your local
change relative to M in your working tree.  In other words, the topology
should look like this:

                           x'
                          /
         Y---Y---B-------M
        /               /  
    ---O---X---X---X---A

where

	M = merge-three-way(O, B, A)
        x' = merge-three-way(B, x, M)

Again, think.  How would you deal with conflicts while coming up with M?

You cannot leave files with conflict markers in the work tree and have the
user fix them up to record M.  Quite contrary to what you insinuated, your
working directory is not a second class citizen but is a very valuable
entity, and it already has important changes between B and x.  We cannot
afford to overwrite it with a half-merge result between A and B for the
purpose of conflict resolution between A and B.

Worse yet, even if we _could_ keep the changes between B and x in the same
file while showing conflicts between A and B (perhaps the changes you made
between B and x did not overlap the region conflicted between A and B), we
cannot still write such a thing out to the working tree.  Why?  Because
then you have to sift through the changes in that file and commit _only_
the parts that are relevant to the merge between A and B while finishing
the merge to produce M, while leaving the change between B and x (which is
going to become the difference between M and x' and left in the working
tree) alone.  And that is actually the best case.  What would you do if
the conflicted region between A and B were something that you changed in
the working tree between B and x?

So the "enlightenment" part is that once you have an ability to "fork" the
history, CVS/SVN style "edit, update, commit" cycle _fundamentally_ would
not work.  That is why "commit first and then merge" is the norm in DVCS
world.

Now how would one deal with this then?  The answer is actually quite
simple.  Let's go back to the first picture:

                   x
                  /
         Y---Y---B
        / 
    ---O---X---X---X---A

We want to come up with a merge between A and B first to produce M, and
while we do that, we do not want to lose the valuable change between B and
x, so we _stash it away_.  Then we can use the working tree to deal with
potential conflicts while finishing the merge to produce M.  In other
words, after stashing, we can safely run "git pull" to produce M.

                   (x) --- the change is stashed away
                  /
         Y---Y---B-------M
        /               /  
    ---O---X---X---X---A

And then we can replay the stash on top of M to produce x'

                           x'
                          /
         Y---Y---B-------M
        /               /  
    ---O---X---X---X---A


And the final answer (yes, I said there are two answers to the original
question, and I already gave two answers, but I let the above description
to raise another question "why does git choose not to implement the logic
of the first answer in a fast-forward case, aka cvs/svn style?") is that
it simply is not worth it to special case the "I didn't commit and ran
pull again".  The workflow to result in such a case would look like this:

	$ git pull
        ... you are in sync with the other end ...
        $ edit
        $ edit
        $ edit
        $ edit
        $ edit
        $ edit
        ... keep working forever _without ever committing_ ...
        $ git pull

which goes against the distributed nature of the system you are using.

Worse yet, once you have committed between these pulls, even once, then
the simple-minded "cvs/svn update" style will not fundamentally work.
Rather than training the users with "If you didn't commit, then you can do
"pull" many many times, but once you commit, then you have to do something
different", which is not very useful anyway, it is better to teach the
more general "forked" case, because the general case solution will also
work in the fast-forward case.

Now, the above inevitably solicits "then why doesn't 'pull' automatically
stash and then unstash?" question.  I think the answer is obvious if you
think about it, and it is getting late, so I'll leave that as an exercise
to the readers but will leave a pictorial hint.

                   C-------M
                  /       /
         Y---Y---B       / 
        /               /
    ---O---X---X---X---A

Re: git pull suggestion

[Matthieu Moy]

Junio C Hamano <gitster@pobox.com> writes:

> git does _not_ implement a handy Porcelain to do this, but we could script
> it like this (I am only illustrating that it can be done, but I am leaving
> the reason why git chooses not to to a later part of this message).

Actually, another way to do this is

git commit -a
git rebase origin/where-you-want-to-merge-from
git reset HEAD^

That would not necessarily be a good way to implement a command, but I
often use variants of this interactively.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: git pull suggestion

[Aghiles]

On Sun, Apr 11, Junio C Hamano <gitster@pobox.com> wrote:
> ...
>
> The other answer would initially appear a bit sad, but after you think
> about it, it would turn into an enlightenment, especially for people whose
> brains have rotten from years and years of CVS/SVN use.

My brain was rotten even _before_ CVS/SVN use, so you can picture
the damage.

Now, thank you very much for taking the time to explain everything. I had
a vague understanding but now things are clearer.

I think that there was a document back in the day, that was giving a
relationship between CVS/SVN commands and git. A posteriori, that
document did more harm than good: it made you believe that you could
use git as CVS/SVN. In practice, that is very difficult and error prone.

> Now, the above inevitably solicits "then why doesn't 'pull' automatically
> stash and then unstash?" question.  I think the answer is obvious if you
> think about it, and it is getting late, so I'll leave that as an exercise
> to the readers but will leave a pictorial hint.

Before I start thinking I already have a question (which says a lot about
my thinking capacity): can't git detect this  problematic case ? My feeling
is that an automatic stash/unstash will  work in most cases and could be
triggered by a --dirty flag.

  -- aghiles

Re: git pull suggestion

[Junio C Hamano]

Aghiles <aghilesk@gmail.com> writes:

> On Sun, Apr 11, Junio C Hamano <gitster@pobox.com> wrote:
>> ...
>>
>> The other answer would initially appear a bit sad, but after you think
>> about it, it would turn into an enlightenment, especially for people whose
>> brains have rotten from years and years of CVS/SVN use.
>
> My brain was rotten even _before_ CVS/SVN use, so you can picture
> the damage.

Please don't take that too seriously; it was just me imitating these:

    http://www.youtube.com/watch?v=4XpnKHJAok8
    https://git.wiki.kernel.org/index.php/LinusTalk200705Transcript

;-)

>> ... so I'll leave that as an exercise
>> to the readers but will leave a pictorial hint.
>
> Before I start thinking I already have a question (which says a lot about
> my thinking capacity): can't git detect this problematic case?

The pictorial hint is to illustrate that the case is not necessarily
problematic.