cherry picking and merge

cherry picking and merge

[Mike Stump]

Cherry picking doesn’t work as well as it should.  I was testing on git version 1.7.9.5.

Put in a line in a file, call it:

first version

then cherry pick this into your branch. Then update on master and transform that into:

second version

then, merge that branch back to master.  Death in the form of conflicts.

In gcc land, I do this sort of thing all the time, and I need a merging subsystem to actually keep track of things.  I can manage this will diff and patch and it works flawlessly.  The point of using something better than diff and patch is for it to be better than diff and patch.

I’d like for merge to merge in the work that has yet to be merged.  Not that, plus blindly try and apply or reapply cherry picked items.

Re: cherry picking and merge

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1598 bytes --]

On Thu, Jul 31, 2014 at 05:58:17PM -0700, Mike Stump wrote:
> Cherry picking doesn’t work as well as it should.  I was testing on
> git version 1.7.9.5.
> 
> Put in a line in a file, call it:
> 
> first version
> 
> then cherry pick this into your branch. Then update on master and transform that into:
> 
> second version
> 
> then, merge that branch back to master.  Death in the form of conflicts.
> 
> In gcc land, I do this sort of thing all the time, and I need a
> merging subsystem to actually keep track of things.  I can manage this
> will diff and patch and it works flawlessly.  The point of using
> something better than diff and patch is for it to be better than diff
> and patch.
> 
> I’d like for merge to merge in the work that has yet to be merged.
> Not that, plus blindly try and apply or reapply cherry picked items.

You're not the first person to be surprised by the way merge works.
From the git-merge manpage:

  [This behavior] occurs because only the heads and the merge base are
  considered when performing a merge, not the individual commits.

(That was added after 1.7.9.5.)

If you want the behavior of applying multiple patches in a row, you want
to use git rebase, not git merge.  Since rebase re-applies the patches
of each of your commits on top of another branch, the identical change
won't cause conflicts.

-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: cherry picking and merge

[Jakub Narębski]

W dniu 2014-08-01 04:43, brian m. carlson pisze:
> On Thu, Jul 31, 2014 at 05:58:17PM -0700, Mike Stump wrote:

>> Cherry picking doesn’t work as well as it should.  I was testing on
>> git version 1.7.9.5.
>>
>> Put in a line in a file, call it:
>>
>> first version
>>
>> then cherry pick this into your branch. Then update on master and transform that into:
>>
>> second version
>>
>> then, merge that branch back to master.  Death in the form of conflicts.
>>
>> In gcc land, I do this sort of thing all the time, and I need a
>> merging subsystem to actually keep track of things.  I can manage this
>> will diff and patch and it works flawlessly.  The point of using
>> something better than diff and patch is for it to be better than diff
>> and patch.
>>
>> I’d like for merge to merge in the work that has yet to be merged.
>> Not that, plus blindly try and apply or reapply cherry picked items.

Note that you should try to avoid cherry-picking, as they do not
leave trace in the graph of revisions.

For example if you are creating a bugfix, instead of putting it
directly on maint, and then cherry-picking to master, it is better
to create a separate feature branch for this fix (based at an early
version), and then merge said branch into maint, then into master.
It is described in blog post by Junio Hamano (which I cannot find now).

> You're not the first person to be surprised by the way merge works.
>  From the git-merge manpage:
>
>    [This behavior] occurs because only the heads and the merge base are
>    considered when performing a merge, not the individual commits.
>
> (That was added after 1.7.9.5.)
>
> If you want the behavior of applying multiple patches in a row, you want
> to use git rebase, not git merge.  Since rebase re-applies the patches
> of each of your commits on top of another branch, the identical change
> won't cause conflicts.

There is also git-imerge, third party tool that is intended to help
merging changes (and make it possible to do it in incremental way).

HTH
-- 
Jakub Narębski

Re: cherry picking and merge

[Mike Stump]

On Aug 1, 2014, at 9:27 AM, Jakub Narębski <jnareb@gmail.com> wrote:
> 
> Note that you should try to avoid cherry-picking, as they do not
> leave trace in the graph of revisions.

Fine, then I want a new command to merge in a change into my branch from another branch and I want merge to account for the motion and not duplicate it when I merge that branch back into master.  Funny thing is, cherry and merge seem to be documented mostly to do exactly what I want.

> For example if you are creating a bugfix, instead of putting it
> directly on maint, and then cherry-picking to master, it is better
> to create a separate feature branch for this fix

You’re assuming that I’m the author of master, I’m not, I’m merely a contributor.  This tail doesn’t wag that dog.  What that means is that I cannot change the world to work around a simple bug in git.

> There is also git-imerge, third party tool that is intended to help
> merging changes (and make it possible to do it in incremental way).

Then remove git merge and replace it with git-imerge.  :-)  Anyway, I read that, and I can see some beauty of that that might be nice in complex merges.  The problem is, I want git merge to work.


I was curious if svn handles this better the same or worse, and it did it just fine.  I know that a while ago, svn could not handle this, it would do what git does currently.  Apparently they figured out it was a bug and fixed it.  Have you guys figured out it is a bug yet?  The first step in solving a problem, is admitting you have a problem.

Re: cherry picking and merge

[Philip Oakley]

From: "Mike Stump" <mikestump@comcast.net>
> On Aug 1, 2014, at 9:27 AM, Jakub Narębski <jnareb@gmail.com> wrote:
>>
>> Note that you should try to avoid cherry-picking, as they do not
>> leave trace in the graph of revisions.
>
> Fine, then I want a new command to merge in a change into my branch 
> from another branch and I want merge to account for the motion and not 
> duplicate it when I merge that branch back into master.  Funny thing 
> is, cherry and merge seem to be documented mostly to do exactly what I 
> want.
>
>> For example if you are creating a bugfix, instead of putting it
>> directly on maint, and then cherry-picking to master, it is better
>> to create a separate feature branch for this fix
>
> You’re assuming that I’m the author of master, I’m not, I’m merely a 
> contributor.  This tail doesn’t wag that dog.  What that means is that 
> I cannot change the world to work around a simple bug in git.
>
>> There is also git-imerge, third party tool that is intended to help
>> merging changes (and make it possible to do it in incremental way).
>
> Then remove git merge and replace it with git-imerge.  :-)  Anyway, I 
> read that, and I can see some beauty of that that might be nice in 
> complex merges.  The problem is, I want git merge to work.
>
>
> I was curious if svn handles this better the same or worse, and it did 
> it just fine.  I know that a while ago, svn could not handle this, it 
> would do what git does currently.  Apparently they figured out it was 
> a bug and fixed it.  Have you guys figured out it is a bug yet?  The 
> first step in solving a problem, is admitting you have a problem.
--
But that goes both ways, and is a philosophical issue about what is to 
be expected in various cases. For some central control use styles, the 
ideas behind _distributed_ version control are anathema and (Git) just 
grinds away at the policies that are expected.

That said, Git doesn't claim to be perfect (and can't because of the 
'relativity' that comes with being distributed - truth has to give way 
to a web of trust). Also the artefacts that Git validates are at a 
different level of abstraction i.e. the whole project as a commit, 
rather than just a few/one file at a time.

In your example (when generalised) the problem is deciding when, in the 
change sequence, the cherry pick is to be backed out, especially if 
there are conflicts in the change sequence that would need fixing 
anyway, and in a long change sequence that would be a lot of conflict 
fix-ups, hence the current choice of getting the merge conflicts all 
resolved in the one go.

The alternate case, mentioned/implied by Brian, is to use a rebase 
(probably after duplicating the branch so as to retain the original if 
required) so as to see each patch/changeset being applied, and doing 
any/many conflit resolutions as they appear, before finally doing any 
merge of the new line of development back into the mainline (which again 
presumes your earlier resolutions don't cause more conflicts on that 
merge). But do note that I've hidden the problem of deciding where the 
rebase start point should be, relative to the merge point, because 
that's actually where the original problem is hidden (which bits merge 
with what!)

git-imerge is a visual tool to show which bits merge cleanly with what 
between two change sequences.

Selecting a compatible workflow is a problem of usage, rather than a 
problem in Git. If Git has a problem, it's that it has too many ways of 
doing things, leaving most of us with too much rope entangled round our 
neck.
--
Philip

Re: cherry picking and merge

[Mike Stump]

On Aug 1, 2014, at 11:57 AM, Philip Oakley <philipoakley@iee.org> wrote:
> But that goes both ways, and is a philosophical issue about what is to be expected in various cases.

The problem is, users expect merge to merge.  There isn’t a user that expects it to scramble the source code, because the command is called merge, not scramble.  That word has semantics that were not invented by your project.  You cannot change the semantic of the word.  Merge has a nice mathematical definition.  Merge branch into master means, place into into master the work from that branch.  git already does this 99% correct, it is missing one corner case.

This is not a philosophical issue.  It is a definitional one.

> For some central control use styles, the ideas behind _distributed_ version control are anathema and (Git) just grinds away at the policies that are expected.

This is irrelevant to the issue at hand.

> That said, Git doesn't claim to be perfect

Again, irrelevant.

> (and can't because

Do you mean, and can’t be?  If so, you are wrong in the case at hand.  svn is an existence proof that you are wrong.

> of the 'relativity' that comes with being distributed - truth has to give way to a web of trust). Also the artefacts that Git validates are at a different level of abstraction i.e. the whole project as a commit, rather than just a few/one file at a time.

Ah, so that gives me an idea.  [ pause ] If we try the cherry-pick as retroactively creating a feature branch, cherrying into that, then merge unconditionally so that no change happens that into trunk (thus killing those conflicts), and then git merge that feature branch into branch then it all works perfectly.  See, another existence proof that you are wrong, this time with git itself.

It was 13 lines of code, so, apparently, it is possible and easy to do, in git.  Now, we just want the cherry-pick to create a temporary cherry branch, cherry the pick into it, merge and drop into trunk and merge into branch…

I tested with the below and it worked just fine.  Things to clean up, we want the meta data on the cherry on the merge commit, but, you get the idea.

branch=b
master=master
base=$(git merge-base $branch $master)
cherry="$1"

git checkout -b cherry-$branch $base
git cherry-pick "$cherry"
git checkout $master
git merge -s ours cherry-$branch
git checkout $branch
git merge cherry-$branch
git branch -d cherry-$branch
git cherry-pick --strategy=ours --allow-empty "$cherry"
git commit --allow-empty

I tested that with two cherries with further changes on master to ensure that it works for more than a single one, no problem.  Wow, even tried a merge of master back into b, and it worked just fine, no conflicts, yet, all the code was jammed up together nicely.

So, if you wish to continue your position, please explain why it can’t get this better, given the existence proof above of it working better in git.

> In your example (when generalized)

I’m not interested in other bugs that I didn’t state, in this email.  I don’t care about those.  Please don’t detract from fixing this issue, because you can identify other things that might not be perfect.  We attain perfection one step at a time.

> the problem is deciding when, in the change sequence, the cherry pick is to be backed out, especially if there are conflicts in the change sequence that would need fixing anyway, and in a long change sequence that would be a lot of conflict fix-ups, hence the current choice of getting the merge conflicts all resolved in the one go.

I have two possible conflict fixups in the above.  In my case (I have a specific patch in gcc-land i wanted to cherry), those fixups were trivial (no conflicts).  When they are trivial, I don’t care much that there were two of them.  When non-trivial, well, I’m resigned to the idea that I have to explain what is going on.

> Selecting a compatible workflow is a problem of usage,

Not when the workflow is mandated on you to work around trivial little bugs that can be fixed but for which the author’s don't even comprehend the bug.

> rather than a problem in Git.

Re: cherry picking and merge

[Philip Oakley]

From: "Mike Stump" <mikestump@comcast.net>
Sent: Friday, August 01, 2014 11:10 PM
> On Aug 1, 2014, at 11:57 AM, Philip Oakley <philipoakley@iee.org> 
> wrote:
>> But that goes both ways, and is a philosophical issue about what is 
>> to be expected in various cases.
>
> The problem is, users expect merge to merge.  There isn’t a user that 
> expects it to scramble the source code, because the command is called 
> merge, not scramble.

Unfortunately we are back at the problem af what 'merge' means. Git uses 
a snapshot model, while most other version control systems uses a 
changeset model (which has been used since before the Titanic was built 
for various reasons [1]) for their storage. Thus most VCS users see 
merge as the addition of a delta "A + delta -> B", while Git sees merge 
as the union of snapshots "A + G -> Q".

The cherry-pick and rebase methods determine the change deltas between 
adjacent snaphots (i.e. patches) and then apply that to a different 
snapshot. In those cases we are not merging snapshots, rather applying 
patches (note my changed use of 'merge').

>That word has semantics that were not invented by your project.  You 
>cannot change the semantic of the word.  Merge has a nice mathematical 
>definition.  Merge branch into master means, place into into master the 
>work from that branch.  git already does this 99% correct, it is 
>missing one corner case.
>
> This is not a philosophical issue.  It is a definitional one.
>
>> For some central control use styles, the ideas behind _distributed_ 
>> version control are anathema and (Git) just grinds away at the 
>> policies that are expected.
>
> This is irrelevant to the issue at hand.
>
>> That said, Git doesn't claim to be perfect
>
> Again, irrelevant.
>
>> (and can't because
>
> Do you mean, and can’t be?  If so, you are wrong in the case at hand. 
> svn is an existence proof that you are wrong.
>
>> of the 'relativity' that comes with being distributed - truth has to 
>> give way to a web of trust). Also the artefacts that Git validates 
>> are at a different level of abstraction i.e. the whole project as a 
>> commit, rather than just a few/one file at a time.
>
<snipped remainder because of time limitations>

--
Philip

[1] Way back when blueprints really were blue, and real drawing were on 
kaolin & linen drawing sheets with indian ink, the 'master' drawings 
were the most prized items, that if damaged would stop production, so 
there were many layers of drawing office controls to avoid touching it 
(e.g. tracers to copy drawings) and keep the master drawings pristine.

From that, the ideas of Change requests, Change orders, Approved changes 
etc. became the norm. It was the changes that were recorded. These 
processes are still in place today in most engineering companies and the 
military, in fact anyone with real artefacts. These techniques were 
copied into the computing world when it was young.

The big change has been that Computing has reduced the cost of 
production (duplication, replication) to zero, so now the problem is in 
trying to validate and verify which alledged copy is actually the 
'master' - especially in a fully distributed environment, such as the 
many different Linux distributions and applications. It was into this 
gap that Git steps, with the use of the sha1 to both validate the 
history chain and verify any specific snapshot. It doesn't get hung up 
on what the change was, that can be determined easily after the fact by 
a simple diff between adjacent snapshots, though having lots of small 
changes and crisp commit messages does help.

That's the way I view it anyway. (When I first started work, they still 
had blue prints, but had moved to microfiche and melinex (polyester) 
drawing sheets, and the 8088 / IBM PC was new and powerful! I still work 
in engineering where the old VC model is ubiquitous)

Re: cherry picking and merge

[Philip Oakley]

From: "Mike Stump" <mikestump@comcast.net>
Sent: Friday, August 01, 2014 11:10 PM
(part 2)
> On Aug 1, 2014, at 11:57 AM, Philip Oakley <philipoakley@iee.org> 
> wrote:
>> For some central control use styles, the ideas behind _distributed_ 
>> version control are anathema and (Git) just grinds away at the 
>> policies that are expected.
> ...
>> of the 'relativity' that comes with being distributed - truth has to 
>> give way to a web of trust). Also the artefacts that Git validates 
>> are at a different level of abstraction i.e. the whole project as a 
>> commit, rather than just a few/one file at a time.
>
> Ah, so that gives me an idea.  [ pause ] If we try the cherry-pick as 
> retroactively creating a feature branch, cherrying into that, then 
> merge unconditionally so that no change happens that into trunk (thus 
> killing those conflicts), and then git merge that feature branch into 
> branch then it all works perfectly.  See, another existence proof that 
> you are wrong, this time with git itself.
>
> It was 13 lines of code, so, apparently, it is possible and easy to 
> do, in git.  Now, we just want the cherry-pick to create a temporary 
> cherry branch, cherry the pick into it, merge and drop into trunk and 
> merge into branch…
>
> I tested with the below and it worked just fine.  Things to clean up, 
> we want the meta data on the cherry on the merge commit, but, you get 
> the idea.
>

I've annotated some of the bits to make sure we are on the same 
wavelength as to what this does...

> branch=b
> master=master
> base=$(git merge-base $branch $master)

> cherry="$1"  # not quite sure where this commit is located relative to 
> either $branch or $master

>
# create a new branch, starting at base, for our cherry picked commit
> git checkout -b cherry-$branch $base
> git cherry-pick "$cherry" # which also commits onto our cherry pick 
> barnch

> git checkout $master
> git merge -s ours cherry-$branch # "mark/remember" the cherry branch, 
> its fix and it's base point, but don't actualy use it here on $master

> git checkout $branch
> git merge cherry-$branch # bring the 'fix' into $branch

> git branch -d cherry-$branch # remove the fix branch that started at 
> $base - branches are ephemeral anyway.

# still on $branch, which already has the change merged in (Git style) ?
> git cherry-pick --strategy=ours --allow-empty "$cherry" # check its 
> all already included?
> git commit --allow-empty
>

Does my annotation match your understanding? It wasn't clear to me where 
$1 had been hiding previously, nor why the common fix didn't use a 
"merge -s ours cherry-$branch" in both cases - that maybe my 
misunderstanding about how your workflow goes.

> I tested that with two cherries with further changes on master to 
> ensure that it works for more than a single one, no problem.  Wow, 
> even tried a merge of master back into b, and it worked just fine, no 
> conflicts, yet, all the code was jammed up together nicely.
>
> So, if you wish to continue your position, please explain why it can’t 
> get this better, given the existence proof above of it working better 
> in git.
>
...
> I have two possible conflict fixups in the above.  In my case (I have 
> a specific patch in gcc-land i wanted to cherry), those fixups were 
> trivial (no conflicts).  When they are trivial, I don’t care much that 
> there were two of them.  When non-trivial, well, I’m resigned to the 
> idea that I have to explain what is going on.
>
>> Selecting a compatible workflow is a problem of usage,
>
> Not when the workflow is mandated on you to work around trivial little 
> bugs that can be fixed but for which the author’s don't even 
> comprehend the bug.
>
>> rather than a problem in Git.
> --


Philip. 

Fwd: cherry picking and merge

[Jakub Narębski]

[sorry for duplicate sent as private mail; I forgot to turn off HTML
when sending to the git mailing list]

On Fri, Aug 1, 2014 at 7:48 PM, Mike Stump <mikestump@comcast.net> wrote:
[...]
>
> I was curious if svn handles this better the same or worse, and it did it just fine.  I know that a while ago, svn could not handle this, it would do what git does currently.  Apparently they figured out it was a bug and fixed it.  Have you guys figured out it is a bug yet?  The first step in solving a problem, is admitting you have a problem.


It can work in Subversion because Subversion stores information about
what was merged in (and this includes cherry-picks, or whatever it is
named in svn) in svn:mergeinfo property. Git does not track what was
merged in, instead it represent the history as the graph of revisions,
and tracks merges (by storing that it came from two or more commits)
and not merged-in information.

When merging Git uses only what is being merged and its common
ancestor (3-point merge). It is simple, and simple works!!!
Unfortunately, it does not see cherry-picked commits - it is invisible
to merge as being on the chain from one of merged commits to the
common ancestor.

The rebase command handles cherry-picked commits by detecting that the
change was already applied. I think that git-imerge does the same (but
I have not used it myself).

Have you tried git-imerge?

-- 
Jakub Narębski



-- 
Jakub Narebski

Re: cherry picking and merge

[Mike Stump]

On Aug 1, 2014, at 12:01 PM, Jakub Narębski <jnareb@gmail.com> wrote:
> It can work in Subversion because Subversion stores information about
> what was merged in (and this includes cherry-picks, or whatever it is
> named in svn) in svn:mergeinfo property. Git does not track what was
> merged in, instead it represent the history as the graph of revisions,
> and tracks merges (by storing that it came from two or more commits)
> and not merged-in information.

So, as a dumb user that just wants it to work, I am unsympathetic to the `but software is hard’ excuse.  I am aware that some bugs are harder to fix than others.  svn took a long time to fix this bug, but they did.  I can wait, the only question is, will it be a week, a month, a year, or a decade.

> When merging Git uses only what is being merged and its common
> ancestor (3-point merge). It is simple, and simple works!!!

I gave a solution for git using branches and it works just fine.  It retains the simple 3-point merge as well.

> Unfortunately, it does not see cherry-picked commits - it is invisible
> to merge as being on the chain from one of merged commits to the
> common ancestor.

Im the solution that I sketched in my previous email, that information is then exposed so that the right merge happens.

> The rebase command handles

I can’t use rebase as it is unfriendly to coworkers.

> cherry-picked commits by detecting that the
> change was already applied. I think that git-imerge does the same (but
> I have not used it myself).
> 
> Have you tried git-imerge?

No, not yet.  I’m not as interested in using it, as I would like git itself to just work.

Re: cherry picking and merge

[Philip Oakley]

From: "Mike Stump" <mikestump@comcast.net>
Sent: Friday, August 01, 2014 11:24 PM
> On Aug 1, 2014, at 12:01 PM, Jakub Narębski <jnareb@gmail.com> wrote:
>> It can work in Subversion because Subversion stores information about
>> what was merged in (and this includes cherry-picks, or whatever it is
>> named in svn) in svn:mergeinfo property. Git does not track what was
>> merged in, instead it represent the history as the graph of 
>> revisions,
>> and tracks merges (by storing that it came from two or more commits)
>> and not merged-in information.
>
> So, as a dumb user that just wants it to work, I am unsympathetic to 
> the `but software is hard’ excuse.  I am aware that some bugs are 
> harder to fix than others.  svn took a long time to fix this bug, but 
> they did.  I can wait, the only question is, will it be a week, a 
> month, a year, or a decade.
>
>> When merging Git uses only what is being merged and its common
>> ancestor (3-point merge). It is simple, and simple works!!!
>
> I gave a solution for git using branches and it works just fine.  It 
> retains the simple 3-point merge as well.

At the moment there is no formal way for Git to record within the commit 
metadata the inclusion of the cherry-picked diff (the 'merge' of the 
fix).

Thinking out of the box, the issue is that the commit parents list does 
not have a formal mechanism to allow the recording that the 'merged' 
change was the patch change from a specific commit fom somewhere else 
(which may be missing from the local repo).

Perhaps it needs a style of merging-rebase where a second (last) parent 
is added but it isn't the straight <sha1>, but says 'patch-<sha1>', such 
that readers with the capability could check if that <sha1> history is 
present locally, and if so if it's correct, so that you can now 'track' 
your fixes between releases, and (hopefully) older Gits don't barf on 
that extra 'fake' parent. Somehow I suspect that older Git's would 
barf.. (not enough time to create and test such a fake commit)

>
>> Unfortunately, it does not see cherry-picked commits - it is 
>> invisible
>> to merge as being on the chain from one of merged commits to the
>> common ancestor.
>
> Im the solution that I sketched in my previous email, that information 
> is then exposed so that the right merge happens.
>
>> The rebase command handles
>
> I can’t use rebase as it is unfriendly to coworkers.
>
>> cherry-picked commits by detecting that the
>> change was already applied. I think that git-imerge does the same 
>> (but
>> I have not used it myself).
>>
>> Have you tried git-imerge?
>
> No, not yet.  I’m not as interested in using it, as I would like git 
> itself to just work.

--
Philip 

Re: cherry picking and merge

[Jakub Narębski]

Philip Oakley wrote:
> From: "Mike Stump" <mikestump@comcast.net>
> Sent: Friday, August 01, 2014 11:24 PM
>> On Aug 1, 2014, at 12:01 PM, Jakub Narębski <jnareb@gmail.com> wrote:

>>> It can work in Subversion because Subversion stores information about
>>> what was merged in (and this includes cherry-picks, or whatever it is
>>> named in svn) in svn:mergeinfo property. Git does not track what was
>>> merged in, instead it represent the history as the graph of revisions,
>>> and tracks merges (by storing that it came from two or more commits)
>>> and not merged-in information.
>>
>> So, as a dumb user that just wants it to work, I am unsympathetic to
>> the `but software is hard’ excuse.  I am aware that some bugs are
>> harder to fix than others.  svn took a long time to fix this bug, but
>> they did.  I can wait, the only question is, will it be a week, a
>> month, a year, or a decade.

Here Git and Subversion went in different directions, and use
different mechanisms (merge tracking vs merged-on tracking).
Both have their advantages and disadvantages.

git-merge (in the most usual case) depends only on three revisions:
the revision you merge into (current branch, ours), the revision
you are merging (merged branch, theirs), and merge base (common
ancestor).  We could have another merge strategy that examines
contents of revisions to handle cherry-picks and reverts... but
it would be more complicated, and much slower.

>>> When merging Git uses only what is being merged and its common
>>> ancestor (3-point merge). It is simple, and simple works!!!
>>
>> I gave a solution for git using branches and it works just fine.  It
>> retains the simple 3-point merge as well.

It works for this simple case, but I think it has unfortunate potential
to go silently wrong.

Also, it prevents fully removing (commits, not only refs) the branch
you cherry-picked from.  The commit you cherry picked may no longer
be (or may no longer should be) in the repository.

> At the moment there is no formal way for Git to record within the commit
> metadata the inclusion of the cherry-picked diff (the 'merge' of the fix).
>
> Thinking out of the box, the issue is that the commit parents list does
> not have a formal mechanism to allow the recording that the 'merged'
> change was the patch change from a specific commit fom somewhere else
> (which may be missing from the local repo).
>
> Perhaps it needs a style of merging-rebase where a second (last) parent
> is added but it isn't the straight <sha1>, but says 'patch-<sha1>', such
> that readers with the capability could check if that <sha1> history is
> present locally, and if so if it's correct, so that you can now 'track'
> your fixes between releases, and (hopefully) older Gits don't barf on
> that extra 'fake' parent. Somehow I suspect that older Git's would
> barf.. (not enough time to create and test such a fake commit).

Sometime ago there was long discussion about adding 'weak' references
to commit object header.

Beside the problem of backward compatibility, there was also the problem
of semantics of said reference - what does it mean?  It should work as
well for cherry-picks, for interactive rebase (maybe?), and for reverts
(which are also a problem).

Also, this could be avoided by using feature branches and merging
instead of committing to one branch and cherry-picking to other
branches. Also, git-rerere is your friend... sometimes.

>>> Have you tried git-imerge?
>>
>> No, not yet.  I’m not as interested in using it, as I would like git
>> itself to just work.

Maybe this command would make it into git proper, though probably
not written in Python (there was once merge strategy written in Python,
but currently git does not depend on Python).

-- 
Jakub Narębski

Re: cherry picking and merge

[Mike Stump]

On Aug 6, 2014, at 8:43 AM, Jakub Narębski <jnareb@gmail.com> wrote:
>>> I gave a solution for git using branches and it works just fine.  It
>>> retains the simple 3-point merge as well.
> 
> It works for this simple case, but I think it has unfortunate potential
> to go silently wrong.

That just means that you want to have two commands.  One for the people that when the remove a patch, they want it gone.  The other for people that when they remove a patch, they want it to magically reappear.  I’m of the former class of individuals.  Now, I would argue that that is the wrong solution of course.  See below for uncherry-pick.

Now, if I needed a solution to the one problem that was mentioned, I would then request an uncherry-pick command to undo the cherry-pick.  The semantics of it are, the patch is removed from the tree, and when merged, that patch isn’t removed from the source.  See, we then retain the useful property that everything that should work does, and the system is predictable because it then does exactly what the user said to do.  Conceptually of course, it doesn’t have anything to do with cherry, if you merge a branch accidentally, and then remove it, and merge it, I think you still wind up with the work being removed.  Conceptually, it is just an undo a change, cherry, merge, file rename, whatever.

Now, why is this preferable?  Because the advanced user gets to explain what they want to git, and then git does what they want.  It also works for beginning users, it does what they ask it to do.  If you are afraid you know better what command that they really wanted to use instead of the command they are using, you can prompt them and ask, did you mean this or that?  After 20 times being asked, it would get old and then even a new user would just issue the commands they want.  I’m not in favor of that, I’d prefer that the system just do what they tell it to do.

> Also, it prevents fully removing (commits, not only refs) the branch
> you cherry-picked from.  The commit you cherry picked may no longer
> be (or may no longer should be) in the repository.

I’m picking from trunk, when it goes, I go.  :-)

> Also, this could be avoided by using feature branches and merging
> instead of committing to one branch and cherry-picking to other
> branches.

If the problem remains unfixed, at least the documentation should be changed to say cherry will mess up merge.  If you never merge, never a problem.  For me, I would read that, and say, well, trivially, cherry isn’t for me (til they fix the bug that causes it to mess up merges).  I can’t see anything on http://git-scm.com/docs/git-cherry-pick which says it will mess up merges.

Re: cherry picking and merge

[Sam Vilain]

On 08/01/2014 10:48 AM, Mike Stump wrote:
>> There is also git-imerge, third party tool that is intended to help
>> merging changes (and make it possible to do it in incremental way).
> Then remove git merge and replace it with git-imerge.  :-)  Anyway, I read that, and I can see some beauty of that that might be nice in complex merges.  The problem is, I want git merge to work.


Git merge has a notion of discrete "merge strategies".  The default,
"recursive" merge strategy isn't completely oblivious to history; in the
event that the two branches don't have a single merge bases, it performs
3-way merges (strangely enough) recursively, with the merge bases of the
branch you're trying to merge until it completes.  In general, this
works pretty well.  Some systems even simpler than that (eg, github's
green merge button) work acceptably as well.

There's no particular reason that you couldn't implement a merge
strategy which works more like SVN's approach, which essentially does an
internal rebase and then commits the result.  The advantages of a rebase
in this situation is that you get to eliminate changes which don't need
to be applied, either because (in SVN's case), it had some
metadata/hearsay information that told it that it could skip that
change, or (in git's case), because it found content/facts that the
change already was applied on one side.

However, there are corresponding disadvantages to this strategy.  It's
just as easy to contrive a situation where this "internal rebasing"
doesn't do the right thing, even without cheating by getting the
metadata wrong.  And besides, there's already a way to do this: do an
actual rebase.  You could also do a rebase, and then if, say, the
original branch you're rebasing is published and you don't want to
rewrite, then you can easily enough use squash merging, merge -s ours,
etc to make it look like the strategy you wanted was a built-in git
merge strategy.  Or, in the spirit of open source, you could contribute
the code required to make 'imerge' a built-in strategy.

> I was curious if svn handles this better the same or worse, and it did it just fine.  I know that a while ago, svn could not handle this, it would do what git does currently.  Apparently they figured out it was a bug and fixed it.  Have you guys figured out it is a bug yet?  The first step in solving a problem, is admitting you have a problem.

So, I have to chuckle when I read this indignant comment.  There's a
funny story to the "while ago" you refer to.  This refers to the time
period during which SVN was relevant; about versions 1.4 and earlier
(being generous).  Back in those days, SVN projects for the most part
avoided merging, because it was so problematic and not tracked at all. 
As one core SVN developer said to me, they found "teams collaborate more
closely if they're all working on the same branch".  Sure, you could do
it, and I even know of a few communities who did, but by and large, it
was avoided.  Then, the new wave of version control systems including
Git, bzr and Mercurial were cropping up, and their merges were actually
good enough that you could practically use them.

The SVN core team had to keep pace to match.  So, in 1.5 the "merge
tracking" system, previously only supplied as a "contrib" script, became
core.  This is ironic, because the version control system which SVN
imitated poorly--Perforce--had a very sophisticated, if
over-complicated, merge tracking system which was also based on
metadata.  Per-branch, per-patch, per-file entries for whether or not a
patch had been "integrated" into the target branch.  I can only guess
that the reason they didn't implement this in the original SVN version
was that it was something of a pain point for users in Perforce. 
Possibly something to do with the way that Perforce would store double
entries for each merge (yes: two rows in a relational store, one
representing the mirror image of the other), and differentiated between
many different forms of "integrated" (ie, 2 rows and 4 states instead
of, say, a single bit).  So the underlying data model wasn't as simple
as it could have been, and this was reflected in the difficult to use
command-line tools.  Plus, they were using BerkeleyDB for metadata
instead of the relational ISAM library, and debugging a rabbit's nest of
merge record as Perforce used would have been a nightmare.  They didn't
go there.  And besides, they found that often, detecting patches as
already applied based on content, like 'patch' did, worked.

Prior to 1.5, the Perl community developed SVK, an offline version of
SVN, and this had a far simpler model for merge tracking, more similar
to git's: just tracking whole-branch merges rather than individual
files, patches, and branches.  SVN eventually added two separate ways of
tracking merges: either a per-file, per-branch, per-commit or a
per-branch, per-commit model.

Anyway, I'm not sure where I'm going with this, but I guess a little
extra perspective would be useful!

Sam

Re: cherry picking and merge

[Mike Stump]

On Aug 1, 2014, at 1:12 PM, Sam Vilain <sam@vilain.net> wrote:
> Git merge has a notion of discrete "merge strategies”.

> There's no particular reason that you couldn't implement a merge
> strategy which works more like SVN's approach, which essentially does an
> internal rebase and then commits the result.

Well, being a simple user, wanting to do a simple thing, I want the default strategy to just work.  The expert that wants it to work faster, but less well, well, they can use a merge -s faster, or cherry-pick -s faster.

> However, there are corresponding disadvantages to this strategy.  It's
> just as easy to contrive a situation where this "internal rebasing"
> doesn't do the right thing, even without cheating by getting the
> metadata wrong.

I sketched a solution using branches…  A large portion of the failures that happen when a cherry is remerged are gone.  I feel that benefit easily swamps the problem you sketched.

> And besides, there's already a way to do this: do an actual rebase.

One problem is that rebase doesn’t work with co-workers nicely…  The other is that it isn’t spelled merge.  I am a simple user.

>> I was curious if svn handles this better the same or worse, and it did it just fine.  I know that a while ago, svn could not handle this, it would do what git does currently.  Apparently they figured out it was a bug and fixed it.  Have you guys figured out it is a bug yet?  The first step in solving a problem, is admitting you have a problem.
> 
> So, I have to chuckle when I read this indignant comment.

:-)  Yeah, a chuckle, good.  Actually, no anger is involved.  I’d just like for git to work better in this regard.

Re: cherry picking and merge

[Nico Williams]

On Fri, Aug 1, 2014 at 6:06 PM, Mike Stump <mikestump@comcast.net> wrote:
> On Aug 1, 2014, at 1:12 PM, Sam Vilain <sam@vilain.net> wrote:
>> Git merge has a notion of discrete "merge strategies”.
>
>> There's no particular reason that you couldn't implement a merge
>> strategy which works more like SVN's approach, which essentially does an
>> internal rebase and then commits the result.
>
> Well, being a simple user, wanting to do a simple thing, I want the default strategy to just work.  [...]

Different users want different defaults.  You can't always have the
one you want.

As for rebase, I still don't understand why it doesn't work for you.
You didn't really explain.  Suppose we're right and it's the right
solution for you, then you might be ecstatic, but you gotta try it
first.

My workflow is rebase-heavy.  It's long been so, and it was so before
git happened.  The only case where I can imagine not using a
rebase-heavy workflow is where I have to track multiple forked
upstreams and so I want to merge each into my branch.

If tracking multiple forked upstreams is not your case and yet rebase
can't work for you then I'd like to understand why.  Please help me
understand your use case.  OTOH, if your use case is amenable to
rebase, then I highly recommend that you try it.

(I find that many users are allergic to rebasing.  Many people have
told me that rebase is lying, that history must be immutable, and so
on, all ignoring that: git users don't rebase published branches, and
that other VCSes tend to squash (therefore lose) history anyways when
pushing merges upstream.  But this all seems theological rather than
rational.  It's true that I dislike merge commits, but that's a
different story; I'm not allergic to merging after all.)

Nico
--

Re: cherry picking and merge

[Alex Davidson]

On Fri, 2014-08-01 at 18:40 -0500, Nico Williams wrote:
> On Fri, Aug 1, 2014 at 6:06 PM, Mike Stump <mikestump@comcast.net> wrote:
> > On Aug 1, 2014, at 1:12 PM, Sam Vilain <sam@vilain.net> wrote:
> >> Git merge has a notion of discrete "merge strategies”.
> >
> >> There's no particular reason that you couldn't implement a merge
> >> strategy which works more like SVN's approach, which essentially does an
> >> internal rebase and then commits the result.
> >
> > Well, being a simple user, wanting to do a simple thing, I want the default strategy to just work.  [...]
> 
> Different users want different defaults.  You can't always have the
> one you want.

Or to put it another way: one man's bug is another man's feature.

> 
> As for rebase, I still don't understand why it doesn't work for you.
> You didn't really explain.  Suppose we're right and it's the right
> solution for you, then you might be ecstatic, but you gotta try it
> first.
> ...
> Nico
> --

Data point:

We've been using a rebase-centric workflow for a while at my current
employer. It's simple and generally straightforward for new development
on master.

However we need to maintain multiple 'released' version branches which
receive hotfixes and (sadly) features from later development, and merges
make it much easier to visualise which releases have received specific
fixes/shinies than cherry-picks do.

In a hybrid merge/rebase workflow it is convenient to have the option of
merges which yield rebase-like output. We have seen awkward merges
outside of master, but I mostly see that as an indication that we
shouldn't be mixing workflows so much, ie. hotfixing one thing with a
cherry-pick and another with a merge (usually only happens when the
branch predates our use of merges).

TL;DR: The option of a rebase-like merge would be a nice feature, but
the default system does not seem so onerous.

Alex

Re: cherry picking and merge

[Mike Stump]

On Aug 1, 2014, at 4:40 PM, Nico Williams <nico@cryptonector.com> wrote:
> As for rebase, I still don't understand why it doesn't work for you.

http://git-scm.com/docs/git-rebase says:

  Rebasing (or any other form of rewriting) a branch that others have based work on is a bad idea

If you read stack-overflow, you will discover a ton of people that like doing this, and they get hammered because of it.  My use case fits exactly (as near as I can tell) the hammer case.

If you make a different command that isn’t guaranteed to screw me and my co-workers over, and tell us to use that, then I’d be happy to use it.  Bet the farm that it won’t bite you, just to be bitten isn’t what I want to try and recover from.

> You didn't really explain.

If you say it will never bit you and then fix all the documentation to not say it will bite you…  I’d be happy to contemplate it again.  Now, I found the stack-overflow commentary first, and all the horrors of it, and all the nuances.  I carefully read what people were doing, how what I wanted to related to what they were doing, and it sure felt like I was in the, don’t go there camp.

> Suppose we're right and it's the right solution for you, then you might be ecstatic, but you gotta try it
> first.

So, I like to know if I’m driving off a cliff, before I do.  I’m the type of person that would rather know were the road goes, and merely avoid driving off the cliff.  When stack-overflow is littered with the bodies of people that thought it would be fun, I tend to just say, that’s not for me.

> The only case where I can imagine not using a
> rebase-heavy workflow is where I have to track multiple forked
> upstreams and so I want to merge each into my branch.

So, sounds like I fit that use case and rebase could be my friend.  How do I square what you said and:

  Rebasing (or any other form of rewriting) a branch that others have based work on is a bad idea

?

I want all old refs in old emails to work.  I want all refs in bugzilla to work.  I want to see the original dates of all the work.  I want git blame to report those artifacts in email and bugzilla.  I have coworkers that I push to, pull from (through a single sharing point, we call the master tree).  We work on gcc, we pull git gcc down to a local copy, then merge it into our tree.  I want to cherry pick changes from upstream.  I do work and push to our master, I pull work of coworkers from the master, my coworkers do the same.  Isn’t this the canonical open source use case?

> (I find that many users are allergic to rebasing.  Many people have
> told me that rebase is lying, that history must be immutable, and so
> on, all ignoring that: git users don't rebase published branches,

So, when I push, and someone else pulls, is that published?  I thought it was.

Rebase safely (Re: cherry picking and merge)

[Nico Williams]

On Wed, Aug 06, 2014 at 12:11:16PM -0700, Mike Stump wrote:
> On Aug 1, 2014, at 4:40 PM, Nico Williams <nico@cryptonector.com> wrote:
> > As for rebase, I still don't understand why it doesn't work for you.
> 
> http://git-scm.com/docs/git-rebase says:
> 
>   Rebasing (or any other form of rewriting) a branch that others have
>   based work on is a bad idea
> 
> If you read stack-overflow, you will discover a ton of people that
> like doing this, and they get hammered because of it.  My use case
> fits exactly (as near as I can tell) the hammer case.

It's not a good idea to rebase a branch in a repo that others pull from.

There's nothing wrong with rebasing your patches on that same branch in
your clone as long as the end result is a fast forward merge when you
push.

    $ git clone https://..../blah
    $ cd blah
    $ <do some work>
    $ git commit ...
    $ git fetch origin
--->$ git rebase origin/master
    $ git push origin master

There's NOTHING wrong with that rebase.  It's perfectly safe because
it's only in your *private* clone.

(If later you publish that clone and expect people to track your master
branch, then rebase becomes problematic, but that's not something most
people ever do with their clones.)

The only use-case I've seen where a rebase-based workflow doesn't work
is where you have multiple upstreams that you're following.  I.e., the
upstream forked and you want to take some commits from one, some from
the other, or otherwise keep a merge of both.

(Also, if an upstream is ever rebased you can usually recover on the
downstream side by rebasing with the --onto option, so it's not the end
of the world.)

> Now, I found the stack-overflow commentary first, and all the horrors
> of it, and all the nuances.  I carefully read what people were doing,
> how what I wanted to related to what they were doing, and it sure felt
> like I was in the, don’t go there camp.

A lot of people rant about rebase.  They're wrong.  They've distorted
your perspective.

> So, I like to know if I’m driving off a cliff, before I do.  I’m the
> [...]

There's just two simple rules to follow and you'll be safe:

1) NEVER git push -f (--force) to a "published" repo/branch.

   The upstream should enforce this with a receive hook.

2) NEVER work directly in a published repo.  Instead work in a private
   clone.  To help make sure of this, never publish a non-bare repo
   (bare == has no workspace; non-bare == has a workspace).

If you ever do a rebase that produces results you're unhappy with you
can undo that rebase like so:

 - use git reflog to find the branch's previous HEAD commit
 - reset the branch to point to that commit

It really helps to think of git as a pile of commits arranged in a
Merkle has tree.  Branches and tags are just symbolic names for specific
commits.  Rebase builds a new line of commits in the tree then it
changes the symbolic branch name's HEAD to point to the head of that new
line of commits, BUT NOTHING IS LOST in the pile of commits that is the
repo, not until you git-prune(1) to remove commits not reachable from
symbolic names (branches and tags).

> > The only case where I can imagine not using a
> > rebase-heavy workflow is where I have to track multiple forked
> > upstreams and so I want to merge each into my branch.
> 
> So, sounds like I fit that use case and rebase could be my friend.

Excellent.

> How do I square what you said and:
> 
>   Rebasing (or any other form of rewriting) a branch that others have
>   based work on is a bad idea
> 
> ?

See above.

> I want all old refs in old emails to work.  I want all refs in

They will if you stick to the two rules I mention above.

> bugzilla to work.  I want to see the original dates of all the work.

Ditto.

> I want git blame to report those artifacts in email and bugzilla.  I
> have coworkers that I push to, pull from (through a single sharing
> point, we call the master tree).  We work on gcc, we pull git gcc down
> to a local copy, then merge it into our tree.  I want to cherry pick
> changes from upstream.  I do work and push to our master, I pull work
> of coworkers from the master, my coworkers do the same.  Isn’t this
> the canonical open source use case?

That means that you have/maintain an intermediate upstream, yes?

This is a bit trickier since once in a while that intermediate upstream
and everyone downstream of it has to catch up with the real upstream.

Here you have two options:

 - the intermediate diverges from the real upstream, and then you
   merge/cherry-pick from the upstream as needed
   
   The intermediate's maintainer must still merge/rebase/cherry-pick
   from the intermediate branch and onto a branch of the upstream in
   order to push to the upstream.

or

 - the intermediate occasionally rebases onto the upstream, and then the
   repos downstream of the intermediate must also rebase with --onto.

   In this case the intermediate's maintainer must tell the downstreams
   what rebase command to execute.

   This makes it easier to push from the intermediate to the upstream:
   the intermediate's commits should always be on top of the upstream
   (at rebase time) so it's easy to push them.

The latter is the workflow we used at Sun for decades for large
projects.  We followed that workflow long before git, first with
Teamware, then later with Mercurial (even though Mercurial technically
had no support for rebasing at that time; we just made it work).

(We always left symbolic names for the pre-rebase branch HEADs, mind
you, to make life easier for everyone.)

> > (I find that many users are allergic to rebasing.  Many people have
> > told me that rebase is lying, that history must be immutable, and so
> > on, all ignoring that: git users don't rebase published branches,
> 
> So, when I push, and someone else pulls, is that published?  I thought
> it was.

Yes.  You shouldn't push -f.  As long as you don't there's no problem.

Nico
-- 

Re: Rebase safely (Re: cherry picking and merge)

[Nico Williams]

On Wed, Aug 06, 2014 at 02:44:59PM -0500, Nico Williams wrote:
> That means that you have/maintain an intermediate upstream, yes?
> 
> This is a bit trickier since once in a while that intermediate upstream
> and everyone downstream of it has to catch up with the real upstream.
> 
> Here you have two options:
> 
>  - the intermediate diverges from the real upstream, and then you
>    merge/cherry-pick from the upstream as needed
>    
>    The intermediate's maintainer must still merge/rebase/cherry-pick
>    from the intermediate branch and onto a branch of the upstream in
>    order to push to the upstream.

I should add something important here.

Rebasing makes life easier for the intermediate maintainer, and for any
upstream maintainer who has to merge "pull requests" or patches sent in
email.  Rebasing puts the onus for merging on the contributor, exactly
where it belongs!

(Granted, for an e-mail based workflow one's patches might have made for
a fast-forward merge when sent but not when the upstream gets to them.
With long enough latency this gets painful.  Which is why I don't
recommend an e-mail based commit integration workflow.)

Nico
-- 

Re: Rebase safely (Re: cherry picking and merge)

[Nico Williams]

On Wed, Aug 06, 2014 at 05:38:43PM -0700, Mike Stump wrote:
> Oh, wait, maybe I have misunderstood the prohibition.  I have:
> 
>        upstream  <—— fsf
>            |
>             \
>              |
>              v
> Me  <—>   master  <—> coworker.

This looks a lot like what I meant about project repos.

> Me is a git clone of master, coworker is a git clone of master.
> Master is a bare repo on a shared server where we put all of our work.
> upstream is a bare git repo of the fsf git tree for gcc.  fsf is a box

Yes, exactly.  We did used exactly this at Sun, with a rebase-only
workflow.  You won't believe it till you see it [below].

> owned by other that hosts the gcc git repository.  I do a git pull fsf
> in upstream from time to time, and a corresponding git merge fsf in Me
> from time to time.  When I like my work I do a git push (to master
> exclusively).  To go to upstream, we submit patches by hand, git is
> not really involved.  I never pull into master from upstream (don’t
> even think that’s possible since they are both bare).

I see.  Hey, if that works for you...  You could, of course, merge or
cherry-pick, or rebase your team's commits onto another copy of the FSF
(upstream) master and then send those commits: sending commits is better
than sending diffs, IMO, mostly because you get to have some metadata
and integrity protection, and because git can ensure lineage and so on.

But you could live purely with diff/patch, no question, and anywhere
between that and making full use of a VCS' powers.

Here now is what we did at Sun, mapped onto git, written as something of
a hardcopy to be more exact.

Remember, this was what we did for _all_ of Solaris.  You can probably
still find docs from the OpenSolaris days describing how to do it with
Mercurial, so you can see I'm not lying.  Thousands of engineers,
working on many discrete projects, with a large OS broken up into a few
"consolidations" (each with its own repo).

(Here the "project gate" is the team repo, that I think you call
"master" above.)

$ # on a bi-weekly (or whatever's best) basis:
$
$ git clone $foo_project_gate foo
$ cd foo
$ git remote add upstream ...
$ git fetch upstream
$ git checkout $current_master
$ new_snapshot=master-$(date +%Y-%m-%d)
$ git checkout -b $new_snapshot
$ git rebase upstream/master
$ git push origin $new_snapshot
$
$ mutt -s "PROJECT FOO: Rebase onto new master branch master-$(date +%Y-%m-%d)" foo-engineers < /dev/null

Then the engineers on this project do this (at their leisure):

$ old_snapshot=<YYYY-mm-dd from current master branch>
$ new_snapshot=<YYYY-mm-dd from new master branch>
$ cd $my_foo_project_clone
$ git fetch origin
$ for topic_branch in ...; do
    git checkout -b ${topic_branch%"-${old_snapshot}"}-$new_snapshot
    git rebase --onto master-$new_snapshot master-$old_snapshot
  done
$
$ # Ready to pick up where I left off!
...

Eventually engineers integrate commits into the project gate:

$ # I'm ready to push to the project gate!
$
$ git checkout some_topic_branch
$
$ # Note: no -f!
$ git push origin HEAD:master-$current_snapshot
...
$ # yay

Eventually the project is ready to push its commits upstream:

$ git clone $project_gate foo
$ cd foo
$ git remote add upstream ...
$ git checkout master-$current_snapshot
$ git push upstream HEAD:master

If you're not going to be sending all local commits upstream yet then
you can do an interactive rebase, put the commits you do want to send
immediately after the upstream's HEAD commit, all the others after, and
send just those.  If you do this you should create a new snapshot and
tell your team members to git rebase --onto it.

Note that we're always rebasing _new_ branches.  Never old ones.  The
project gate does plain rebases of those new branches.  Downstreams have
to rebase --onto to "recover" (it works fine).

This is a very rebase-happy workflow.  It keeps as-yet-not-contributed
commits "on top" relative to the immediate upstream of any repo.  This
makes them easy to identify, and it keeps the author/date/subject
metadata.  Because you rebase often, you don't lag the upstream by much.
Because they are "on top" it's always fast-forward merge to push --
you're always "merged", with some lag, yes, but merged.  And the person
doing the merging is the owner of the repo (team members, project
gatekeeper).

It's a bit more work each time you rebase than a merge-heavy workflow.
But it's also easier to contribute, and it's easier on each successive
upstream's maintainers.

(The upstream also kept "snapshot" branches.  Doing this has many good
side effects, not the least of which is that git prune (and gc, which I
knew about) doesn't go deleting the past of each rebase.)

> > The only use-case I've seen where a rebase-based workflow doesn't work
> 
> Well, and now mine, which I claim is a the canonical open source use
> [...]

Nah.  Sun managed this for decades without a hitch, and for products
much larger than GCC.  See above.

(It's true that it's difficult to sell some people on this workflow,
especially when their previous experiences are with VCSes that look down
on rebase.  You don't have to buy it either.  However, it works very
well.)

> I’m trying to envision how anyone could ever use rebase.  If you
> can’t share your work, it isn’t work.

Do some experiments based on the above hardcopy.  If that doesn't
convince you that it works, oh well, I'll have given it a good try.

Nico
-- 

Re: Rebase safely (Re: cherry picking and merge)

[Mike Stump]

On Aug 6, 2014, at 10:11 PM, Nico Williams <nico@cryptonector.com> wrote:
> Nah.  Sun managed this for decades without a hitch, and for products
> much larger than GCC.  See above.

Ok.  Ah, ok, perfect.  I see how that method of working would cure the cherry-pick and merge don’t work problem mentioned at the top of the thread.

> Do some experiments based on the above hardcopy.  If that doesn't
> convince you that it works, oh well, I'll have given it a good try.

Thank you for taking the time to diagram that as it appears to violate everyones how to use git guide.   I see the workflow does an onto, which was the ‘fix’ people talked about on stack overflow, and I see just how things would work.

If the old master branches are deleted and gc is run, then all the old references go away, and then the refs from email and bugzilla then don’t work.  Did you guys ever remove them and then prune (or gc)?

Now, the biggest issue, if that is recognized as `fixing’ the cherry-pick problem, then certainly the problem is understood to be a problem.  If one recognized it as a problem, then one can envision cherry-pick and merge working together so that the problem doesn’t need fixing in the first place.  And, if it doesn’t need fixing, then the cost of the solution isn’t needed either.  The biggest problem with git, is that two features don’t work nicely together when they could; in this case, cherry-pick and merge).  Because they don’t, it makes it hard for people to predict what will happen when they use it.  This makes it more expensive to use and less suitable than a system that is more predictable.  You improve git, by fixing the problem and making the features work nicely together and making it predicable.

I still favor fixing the underlying problem with cherry-pick and merge not working.  :-)  That said, I see how to work around the bug with rebase, if I need to.

I wish the top google hit were your guide and I wish I never saw all the other pages…  I see now your position, and I see why all the guides are wrong, if you know just how to use rebase.  I wish the git documentation were improved to say as the first sentence under cherry-pick, this feature sucks and doesn’t really work well, it can cause excess merge conflicts.  rebase can be used to work around the bugs in cherry-pick for now.  And under rebase, instead of saying what it said now, that how one can can trivially and effortlessly use git, instead of saying, Do not rebase commits that you have pushed to a public repository which I now see is wrong.

Re: Rebase safely (Re: cherry picking and merge)

[Nico Williams]

On Fri, Aug 08, 2014 at 10:34:43AM -0700, Mike Stump wrote:
> On Aug 6, 2014, at 10:11 PM, Nico Williams <nico@cryptonector.com> wrote:
> > Nah.  Sun managed this for decades without a hitch, and for products
> > much larger than GCC.  See above.
> 
> Ok.  Ah, ok, perfect.  I see how that method of working would cure the
> cherry-pick and merge don’t work problem mentioned at the top of the
> thread.
> 
> > Do some experiments based on the above hardcopy.  If that doesn't
> > convince you that it works, oh well, I'll have given it a good try.
> 
> Thank you for taking the time to diagram that as it appears to violate
> everyones how to use git guide.   I see the workflow does an onto,
> which was the ‘fix’ people talked about on stack overflow, and I see
> just how things would work.

There's nothing scary about --onto.  You're saying "figure out which are
my local commits (the ones on top of the previous upstream) and pick
them onto the new upstream".

We only need to do it manually (though it can be scripted[*]) because
git doesn't track rebase history so that it can be done automatically.

[*] And then there's Tony Finch's
    https://git.csx.cam.ac.uk/x/ucs/git/git-repub.git , which is kinda
    awesome!

> If the old master branches are deleted and gc is run, then all the old
> references go away, and then the refs from email and bugzilla then
> don’t work.  Did you guys ever remove them and then prune (or gc)?

Product gates' repos and snapshots stuck around forever, though it was
Teamware, and finding really old ones wasn't necessarily easy,
particularly since their names didn't always reflect product names.

Prominent project gate repos and their snapshots also stuck around
forever.

Lesser project gate repos tended to be as ephemeral as the project.

> Now, the biggest issue, if that is recognized as `fixing’ the
> cherry-pick problem, then certainly the problem is understood to be a
> problem.  If one recognized it as a problem, then one can envision

Not really.  This isn't about git.  We followed a rebase-only workflow
with VCSes that nominally didn't support rebase.  We did it because it
was easier on everyone and kept history in the upstream clean.  We
didn't do it because git has issues when combining merge and
cherry-pick.

> cherry-pick and merge working together so that the problem doesn’t
> need fixing in the first place.  And, if it doesn’t need fixing, then

If you buy into the Sun model then this is all a non-issue.  If you
don't then I think you have other problems (because I have bought into
the Sun model) :)

> the cost of the solution isn’t needed either.  The biggest problem
> with git, is that two features don’t work nicely together when they
> [...]

The Sun model has no additional cost.  It moves costs around so that the
people dealing with conflicts are the downstreams, not the upstreams,
and that's exactly as it should be.

(Keep in mind that Solaris gates tended to have large numbers of commits
on any given day, so it was quite common that one would have to rebase
multiple times before successfully pushing.  For large projects with
long test cycles the gates would close to avoid the need to rebase and
re-test.)

> I still favor fixing the underlying problem with cherry-pick and merge
> not working.  :-)  That said, I see how to work around the bug with
> rebase, if I need to.

IMO it could be done, but I can't help that.

> I wish the top google hit were your guide and I wish I never saw all
> the other pages…  I see now your position, and I see why all the

Me too!  I should blog it.

> guides are wrong, if you know just how to use rebase.  I wish the git
> documentation were improved to say as the first sentence under

The Sun model is not the only way to use git though.

> cherry-pick, this feature sucks and doesn’t really work well, it can
> cause excess merge conflicts.  rebase can be used to work around the
> bugs in cherry-pick for now.  And under rebase, instead of saying what
> it said now, that how one can can trivially and effortlessly use git,
> instead of saying, Do not rebase commits that you have pushed to a
> public repository which I now see is wrong.

I'm glad you understand the Sun model now.  You should evaluate its
applicability to your use case on its own merits.  Don't use it just to
workaround a problem in git; use it because it's good, or don't use it
because it doesn't fit your team's needs.

Nico
-- 

Fwd: Rebase safely (Re: cherry picking and merge)

[Mike Stump]

[ sorry for the dup ]

Begin forwarded message:

On Aug 6, 2014, at 12:44 PM, Nico Williams <nico@cryptonector.com> wrote:
> It's not a good idea to rebase a branch in a repo that others pull from.

Well, so rebase is then out, as I don’t want to rebase _my_ tree, I want to rebase _the_ tree.  Recall, I don’t want to cherry pick for my tree, I want to cherry pick for the tree.

[ reads rest of email ]

Oh, wait, maybe I have misunderstood the prohibition.  I have:

       upstream  <—— fsf
           |
            \
             |
             v
Me  <—>   master  <—> coworker.

Me is a git clone of master, coworker is a git clone of master.  Master is a bare repo on a shared server where we put all of our work.  upstream is a bare git repo of the fsf git tree for gcc.  fsf is a box owned by other that hosts the gcc git repository.  I do a git pull fsf in upstream from time to time, and a corresponding git merge fsf in Me from time to time.  When I like my work I do a git push (to master exclusively).  To go to upstream, we submit patches by hand, git is not really involved.  I never pull into master from upstream (don’t even think that’s possible since they are both bare).

I read the prohibition as don’t rebase my branch called master on Me and push it to master on master as others then pull master from master.  Did I misunderstand?  Instead, the prohibition is you can use push/pull freely and you can have as many coworkers as you want, just don’t use push -f and don’t let anyone push/pull from your own private clone.

I had envisions that the rebase of master on Me once pushed to master and then pull from master to coworker is the exact case that would screw us.

> The only use-case I've seen where a rebase-based workflow doesn't work

Well, and now mine, which I claim is a the canonical open source use case.  Can't use source, unless you import the source,  can’t be real unless you can change the source, once you do that, you then need to merge in newer sources, and if the company has or will have more than a single individual and these folks are ever to work together, then they need to share the source between them.

I’m trying to envision how anyone could ever use rebase.  If you can’t share your work, it isn’t work.

> is where you have multiple upstreams that you're following.

I only have a single (for this repo) upstream.

>> Now, I found the stack-overflow commentary first, and all the horrors
>> of it, and all the nuances.  I carefully read what people were doing,
>> how what I wanted to related to what they were doing, and it sure felt
>> like I was in the, don’t go there camp.
> 
> A lot of people rant about rebase.  They're wrong.  They've distorted
> your perspective.

What I saw were the people that screwed their world and were trying to recover.  It was a question, how do I recover, and what did I do wrong.  There was no rant.  Or, at least, I’m impervious to rants and don’t actually see them.  I deal in the cold hard facts and transform the rant into what happened, what they did wrong, and how to avoid doing it myself.

No, you’ve set my perspective, let me quote you:

  It's not a good idea to rebase a branch in a repo that others pull from.

this matches the doc, matches the experience of users on stack overflow and matches what what I think is true.  You are free to correct that if I am wrong.

I don’t know why you think my perspective is distorted.  Either, I can rebase all my patches, all my coworkers patches, and push those up to master and have all my coworkers pull from master and develop (meaning branches off master as well as patches to master) as normal, or I can’t.

> There's just two simple rules to follow and you'll be safe:
> 
> 1) NEVER git push -f (--force) to a "published" repo/branch.

I can never use push -f.  That seems trivial.  git config --system receive.denyNonFastForwards true seems to be exactly what I would do to my master to enforce this rule.  This then seems to permanently be a non-issue.

> 2) NEVER work directly in a published repo.  Instead work in a private
>   clone.

I only ever work in a private to me clone, so I’m safe.  The only published repo is a bare repo, which can’t be worked in by design, so, again, I think I’m perfectly safe.

So, if that is true, why do others write such things as (from http://ctoinsights.wordpress.com/2012/06/29/git-flow-with-rebase/):

> The way to get the best of both worlds is to be explicit about when to use one versus the other. For us the simple rules to follow are:
> 	• Rebase feature branches.
> 	• Never rebase develop or master branch. (Always merge into develop and master.)
> 	• Never rebase public branches.
master is public, I want to rebase master.  This violates rule 3 above, but not any of your rules?

I develop on master, thus violating rule 2.  This does’t violate any of your rules?

I want to rebase master, thus, violating rule 1.  This doesn’t violate any of your rules?

I violate every single rule, by his standard, I am screwed.

http://blog.experimentalworks.net/2009/03/merge-vs-rebase-a-deep-dive-into-the-mysteries-of-revision-control/ says:

> Never rebase branches or trees that you pulled. Only rebase local branches. 

I violate that.  This doesn’t violate ant of your rules?  I pull master rebase it, then push it.  master is not a local branch, or, more correctly, I have a local branch called master that is push to and pull from a bare repo that I’m calling master to/from a branch called master.  git diff origin/master master shows our work.

To me, a local branch is one call b, that doesn’t push or pull to any remote repo.

Linus in https://www.mail-archive.com/dri-devel@lists.sourceforge.net/msg39091.html said:

But never other peoples code. That's a "destroy 
history"

Thus violating his rule.  Recall why rebase was suggested.  I want to merge the upstream fsf tree into our tree (or our tree into a copy of the fsf tree) and instead of using merge, it was suggested to use rebase.  But I operate on our entire tree.  He further says:

- You must never EVER destroy other peoples history. You must not rebase 
   commits other people did.

I want to exactly rebase either the entire fsf tree onto mine, or mine on to the fsf tree.  Either result I am rebasing code I didn’t write, commits other than mine.  This violates his rule, not your?

He states:

- Minor clarification to the rule: once you've published your history in 
   some public site, other people may be using it, and so now it's clearly 
   not your _private_ history any more.

And I violate this rule, but it doesn’t violate any of your rules?

>  To help make sure of this, never publish a non-bare repo
>   (bare == has no workspace; non-bare == has a workspace).

We only have a bare repo (our master) and we only ever push/pull from it, so I’m safe.

> It really helps to think of git as a pile of commits arranged in a
> Merkle has tree.  Branches and tags are just symbolic names for specific
> commits.  Rebase builds a new line of commits in the tree then it
> changes the symbolic branch name's HEAD to point to the head of that new
> line of commits, BUT NOTHING IS LOST in the pile of commits that is the
> repo, not until you git-prune(1) to remove commits not reachable from
> symbolic names (branches and tags).

Wrong, let me introduce you to git gc:

       --prune=<date>
           Prune loose objects older than date (default is 2 weeks ago, overridable by the config variable
           gc.pruneExpire). --prune=all prunes loose objects regardless of their age. --prune is on by default.

which, I run every now and then as we work on stuff that is more than 5 lines long.  I bring in 60,000 changes, I push, I git gc on the bare repo.

>>> The only case where I can imagine not using a
>>> rebase-heavy workflow is where I have to track multiple forked
>>> upstreams and so I want to merge each into my branch.
>> 
>> So, sounds like I fit that use case and rebase could be my friend.
> 
> Excellent.

I’m optimistic.

>> How do I square what you said and:
>> 
>>  Rebasing (or any other form of rewriting) a branch that others have
>>  based work on is a bad idea
>> 
>> ?
> 
> See above.

So, rebasing will always work just fine, one just needs to never, ever use push -f and never ever share ones own private clone.  Sharing a bare repo that everyone works on (push/pull) is fine.  rebasing and pushing into it and others pulling from it will always just work fine.

Gosh, could you get the documentation to say that.  I’ve certainly been scared off even trying rebase.

Rebase should just say it always works perfectly (remove the warning entirely) and then burry into push -f, this option will destroy your world and put into it, the entire how it will screw it, how to recover from it, and then say in the clone documentation, you should never clone a none bare repo, because if you do, your world will end when you rebase.  In fact, I’ve make if a clone -f operation, and then fail the default is non-base, and then under clone -f explain this is a ver bad idea.

Articles like http://stackoverflow.com/questions/457927/git-workflow-and-rebase-vs-merge-questions:

Reason #2: With rebase, there is no undo!

makes me nervous.

>> I want all old refs in old emails to work.  I want all refs in
> 
> They will if you stick to the two rules I mention above.

Ah, excellent.

>> bugzilla to work.  I want to see the original dates of all the work.
> 
> Ditto.

Ah, nice.

>> I want git blame to report those artifacts in email and bugzilla.  I
>> have coworkers that I push to, pull from (through a single sharing
>> point, we call the master tree).  We work on gcc, we pull git gcc down
>> to a local copy, then merge it into our tree.  I want to cherry pick
>> changes from upstream.  I do work and push to our master, I pull work
>> of coworkers from the master, my coworkers do the same.  Isn’t this
>> the canonical open source use case?
> 
> That means that you have/maintain an intermediate upstream, yes?

Yes.  I have upstream which is a virgin copy of the fsf in a bare repo.

Our master only ever pulls from the upstream.

> This is a bit trickier since once in a while that intermediate upstream
> and everyone downstream of it has to catch up with the real upstream.
> 
> Here you have two options:
> 
> - the intermediate diverges from the real upstream, and then you
>   merge/cherry-pick from the upstream as needed

Virgin copy, it only can ever lag in time, no other way.  We can only pull or get work from our virgin copy, no other way.  To get a patch that just went in, I would like to pull from fsf into upstream, cherry-pick upstream into Me, push into master.  When I merge I same thing except I do a merge upstream.

>   The intermediate's maintainer must still merge/rebase/cherry-pick
>   from the intermediate branch and onto a branch of the upstream in
>   order to push to the upstream.

It never pushes up, it is unidirectional.  The only direction down is pull the entire bare repo, everything, or nothing, that’s it.

> (We always left symbolic names for the pre-rebase branch HEADs, mind
> you, to make life easier for everyone.)
> 
>>> (I find that many users are allergic to rebasing.  Many people have
>>> told me that rebase is lying, that history must be immutable, and so
>>> on, all ignoring that: git users don't rebase published branches,
>> 
>> So, when I push, and someone else pulls, is that published?  I thought
>> it was.
> 
> Yes.  You shouldn't push -f.  As long as you don't there's no problem.

So, you would like to withdraw your statement, git users don't rebase published branches, and instead say, git users can rebase published branches and it all works flawlessly well, provided they stay away from non-bare shared repos and stay away from push -f?  The later statement doesn’t make be nervous at all.  Though, I would like an option like receive.denyNonFastForwards to turn off the ability to push/pull from a non-bare repo.


So, thinking about it some more, would references to the fsf git repo from the fsf bugzilla work in our tree once I rebase the fsf work on ours?  Remember, I do want those to work.

Re: cherry picking and merge

[Mike Stump]

On Jul 31, 2014, at 7:43 PM, brian m. carlson <sandals@crustytoothpaste.net> wrote:
> 
> You're not the first person to be surprised by the way merge works.

I’m not the first, because the merge command is broken.  Once fixed, I would be happy to be the last.  Until then, the bug remains unfixed.  I’m sending the mail to petition for this bug to be fixed.

> From the git-merge manage:
> 
>  [This behavior] occurs because only the heads and the merge base are
>  considered when performing a merge, not the individual commits.

From:

google(”git merge”):

> git-merge - Join two or more development histories together

Either, the command should do as documented, or be fixed.  In that reference, there is no mention that merge will not merge.  There is no mention that merge isn’t the command I want to merge, but that I should use rebase.

Further, google(“git rebase”) says:

> There is no difference in the end product of the integration,

Clearly, this is a lie.  There is a difference.

Now, about rebase:

> Do not rebase commits that you have pushed to a public repository.
> 
> If you follow that guideline, you’ll be fine. If you don’t, people will hate you, and you’ll be scorned by friends and family.


Since everything I do goes up and down into repositories and I don’t want my friends and family to scorn me, rebase isn’t the command I want to use.

I want to use the simple, it works, named for the operation I want to perform, merge.  I’m a simple user, and the simple command I want to work.  You can name the old merge command, merge-mostly or merge-fast and the new one can be called merge.

> (That was added after 1.7.9.5.)

I don’t want bugs documented, I want them fixed.  I’m not reporting a doc bug, the doc is correct.

> If you want the behavior of applying multiple patches in a row, you want
> to use git rebase, not git merge.  Since rebase re-applies the patches
> of each of your commits on top of another branch, the identical change
> won't cause conflicts.

But, I don’t want the series of patches, I just want a simple,  merged feature X on trunk single commit that merge does.

Given branch B, master M, and cherry picked C, what I want merged is B-(M+C), not B-M.  The problem with B-M, is that when you do B += C (aka cherry pick from master onto your branch), then M += B-M (merge your branch into master), that C is then replicated.  This replication is wrong, always wrong, never right, incorrect, broken.  This is the bug I want fixed.

Re: cherry picking and merge

[Keller, Jacob E]

On Fri, 2014-08-01 at 09:56 -0700, Mike Stump wrote:
> Since everything I do goes up and down into repositories and I don’t want my friends and family to scorn me, rebase isn’t the command I want to use.

You completely mis-understand what "published" means. Published history
is history from which other people can pull right now.

That means it has to be in a publicly addressable repository (ie: just
like the remote that you are pulling from as upstream).

rebasing commits which are already in the upstream is bad. Rebasing
commits which you have created locally is NOT bad. These commits would
not be published until you do a push.

This is the fundamental issue with rebase, and it is infact easy to
avoid mis-using, especially if you don't publish changes. The key is
that a commit isn't published until it's something someone else can
depend on.

Doing "git pull --rebase" essentially doesn't ever get you into trouble.

Regards,
Jake

Re: cherry picking and merge

[Keller, Jacob E]

On Thu, 2014-08-21 at 17:36 +0000, Keller, Jacob E wrote:
> On Fri, 2014-08-01 at 09:56 -0700, Mike Stump wrote:
> > Since everything I do goes up and down into repositories and I don’t want my friends and family to scorn me, rebase isn’t the command I want to use.
> 
> You completely mis-understand what "published" means. Published history
> is history from which other people can pull right now.
> 
> That means it has to be in a publicly addressable repository (ie: just
> like the remote that you are pulling from as upstream).
> 
> rebasing commits which are already in the upstream is bad. Rebasing
> commits which you have created locally is NOT bad. These commits would
> not be published until you do a push.
> 
> This is the fundamental issue with rebase, and it is infact easy to
> avoid mis-using, especially if you don't publish changes. The key is
> that a commit isn't published until it's something someone else can
> depend on.
> 
> Doing "git pull --rebase" essentially doesn't ever get you into trouble.
> 
> Regards,
> Jake
> \x04�{.n�+�������+%��lzwm��b�맲��r��z\b��{ay�\x1dʇڙ�,j\a��f���h���z�\x1e�w���\f���j:+v���w�j�m����\a����zZ+�����ݢj"��!�i

Pardon me. You can actually ignore this post. I read through more of the
thread, and actually realize I completely misunderstood what your issue
was, and why rebase might not work.

Regards,
Jake

cherry picking and merge

[Nico Williams]

On Thursday, July 31, 2014, Mike Stump <mikestump@comcast.net> wrote:
>
> Cherry picking doesn’t work as well as it should.  I was testing on git version 1.7.9.5.
>
> Put in a line in a file, call it:
>
> first version
>
> then cherry pick this into your branch. Then update on master and transform that into:
>
> second version
>
> then, merge that branch back to master.  Death in the form of conflicts.

The problem is that cherry-picked commits lack the metadata that git
merge could use to avoid this spurious conflict report.  The reflog
has the metadata in question, but there's no guarantee that that
reflog will be available where you do the merge.  (IMO this is another
reason to want branches as objects, so such ancillary information can
be recorded somewhere, but in a way that can get dropped if desired
and without changing commit hashes, but I digress.)

If you always rebase your commits on top of the upstream, then this
problem goes away.  You can't always rebase your commits on top of the
upstream though, but wherever possible it's the best course of action
for this and other reasons.

Nico
--

Re: cherry picking and merge

[Mike Stump]

On Aug 1, 2014, at 12:22 PM, Nico Williams <nico@cryptonector.com> wrote:
> If you always rebase

I can’t use rebase unless you make rebase work with multiple users and pushing pulling.

Re: cherry picking and merge

[Nico Williams]

On Fri, Aug 1, 2014 at 5:13 PM, Mike Stump <mikestump@comcast.net> wrote:
> On Aug 1, 2014, at 12:22 PM, Nico Williams <nico@cryptonector.com> wrote:
>> If you always rebase
>
> I can’t use rebase unless you make rebase work with multiple users and pushing pulling.

That works now, and I do it all the time.  Have a single repo ("the
truth"), always rebase local commits on top of the latest upstream,
always do fast-forward pushes.  Done.

Re: cherry picking and merge

[Jonathan Nieder]

Hi Mike,

Mike Stump wrote:

> Cherry picking doesn’t work as well as it should.  I was testing on
> git version 1.7.9.5.
>
> Put in a line in a file, call it:
>
> first version
>
> then cherry pick this into your branch. Then update on master and
> transform that into:
>
> second version
>
> then, merge that branch back to master.  Death in the form of conflicts.

Do you mean that "git merge" should be aware of what changes you have
already cherry-picked?

It isn't, and that's deliberate ("git merge" is designed to be simple
as possible, though no more simple than that).  This way, if on a side
branch someone makes a change that would conflict with "master" and
then backs it out, then the branch can still merge cleanly.

Generally people do one of the following:

 * Use a merge-centric workflow.  Don't cherry-pick "forward" but
   merge instead.  (Do use cherry-pick for backports when you forgot
   to commit a fix on top of the oldest supported branch that would
   need it.)  The gitworkflows(7) manpage has more details on how
   this works.

 * Use a cherry-pick-centric workflow.  Never merge.  Notice when
   you're trying to apply a patch you already applied and skip it.
   (Others in the thread have covered this workflow a little.)

Even in those workflows, it's possible to have conflicts due to
genuinely conflicting changes, even with no cherry-pick involved.  I
find the '[merge] conflictstyle = diff3' setting (see git-config(1))
and git-rerere(1) to be helpful in making that less painful.

Hope that helps,
Jonathan

Re: cherry picking and merge

[Jonathan Nieder]

Jonathan Nieder wrote:

> Do you mean that "git merge" should be aware of what changes you have
> already cherry-picked?
>
> It isn't, and that's deliberate

That said, when today's "git merge" fails to resolve conflicts, it's
easily possible that we could do better at resolving the merge by
walking through both sides and understanding what happened.

The detailed history lets you

   i) Present conflicts in an easier to resolve way.

      "Patch #1 which tries to do X conflicted with patch #2 which
      tries to do Y; please reconcile them" can be less painful to
      deal with than "Something in this pile conflicted with something
      in that pile".

  ii) Break a seeming conflict into pieces that can be automatically
      resolved more easily.

      X vs X'+Y may conflict where X' is a cherry-pick of X, if X and
      Y touch the same code.  Meanwhile if we're lucky then X vs X'
      will not conflict because they make the same change, and Y can
      apply on top.

 iii) Handle cherry-picked changes in a *different* way.  For example,
      if patch X was applied on one side and applied and then reverted
      on the other side, this could show up as a conflict.  After all,
      the two sides don't agree on whether patch X is a good change or
      not.

These features have corresponding downsides:

   i') (Speaking from experience of using git-imerge) Too many tiny
       conflicts can sometimes be more painful to resolve than all the
       conflicts at once.  When X, Y, Z, and W had various conflicts,
       how to reconcile X and Y alone or Z and W alone are academic
       questions that don't actually need to be answered to produce
       the merge result.

  ii') This kind of clean, broken-down merge can produce a "clean"
       but wrong result.

       For example, if the following sequence of events occured:

         1. Build fancy new feature X on "master".

	 2. Cherry-pick X to the bugfixes-only branch "maint".
	    Whoops.
	 3. Correct the mistake: revert X on "maint".  Now "maint"
	    is bugfixes-only again!

	 4. Merge "maint" to "master".

       Then a naive, 3-way merge will notice there is no change
       on "maint" since it was last merged to master and the
       merge will bring in no change (good).

       And on the other hand a one-patch-at-a-time merge would
       try to apply X (with no effect, since it's already applied)
       and then try to apply the revert of X.  The net effect would
       be to revert X from "master" (bad)!

 iii') See (ii').

git-imerge from https://github.com/mhagger/git-imerge can help with
(i) and (ii) but not (iii).

Hoping that clarifies,
Jonathan

Re: cherry picking and merge

[Nico Williams]

On Fri, Aug 1, 2014 at 3:50 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Jonathan Nieder wrote:
>
>> Do you mean that "git merge" should be aware of what changes you have
>> already cherry-picked?
>>
>> It isn't, and that's deliberate
>
> That said, when today's "git merge" fails to resolve conflicts, it's
> easily possible that we could do better at resolving the merge by
> walking through both sides and understanding what happened.

It would help if cherry-pick history where recorded somewhere (beyond
the reflog)...

Cherry-picks should record two parents, like merges.

(Of course, it does no good to know about an unreachable parent, when
a commit with two parents is pushed to a repo that doesn't have one of
those parents, which can happen when topic branches aren't pushed
upstream.)

Nico
--

Re: cherry picking and merge

[Junio C Hamano]

Nico Williams <nico@cryptonector.com> writes:

> Cherry-picks should record two parents, like merges.

No.

It is OK to record where it came from, and we let you do so with the
"-x" option.

But the "where it came from" commit is very different from being
parent, which implies "all the history behind it".  The whole point
of a cherry-pick is that you do not want to grab the changes behind
the commit you are cherry-picking and you want the _change_ the
cherry-picked commit (and that commit alone) brings in.  It should
never record "two parents, like merges."

Re: cherry picking and merge

[Nico Williams]

On Fri, Aug 1, 2014 at 4:44 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Nico Williams <nico@cryptonector.com> writes:
>
>> Cherry-picks should record two parents, like merges.
>
> No.
>
> It is OK to record where it came from, and we let you do so with the
> "-x" option.
>
> But the "where it came from" commit is very different from being
> parent, which implies "all the history behind it".  The whole point
> of a cherry-pick is that you do not want to grab the changes behind
> the commit you are cherry-picking and you want the _change_ the
> cherry-picked commit (and that commit alone) brings in.  It should
> never record "two parents, like merges."

I didn't mean to imply all that.  s/parent/where it came from/, but -x
edits the commit message, not the metadata...

The point remains: to do what the OP wants git merge would have to be
able to notice that a given commit was cherry-picked from the other
branch, and what commit it was on that other branch, and right now the
only place where that information is available is in the reflog.
Recording that metadata somewhere in the commit resulting from the
cherry-pick would be better.

Nico
--

Re: cherry picking and merge

[Junio C Hamano]

Nico Williams <nico@cryptonector.com> writes:

> branch, and what commit it was on that other branch, and right now the
> only place where that information is available is in the reflog.

... or the line in "-x".

We do not add random unstructured cruft in the commit object
header.  Check the list archive.

Re: cherry picking and merge

[Jakub Narębski]

W dniu 2014-08-01 22:55, Nico Williams pisze:
> On Fri, Aug 1, 2014 at 3:50 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> Jonathan Nieder wrote:
>>
>>> Do you mean that "git merge" should be aware of what changes you have
>>> already cherry-picked?
>>>
>>> It isn't, and that's deliberate
>>
>> That said, when today's "git merge" fails to resolve conflicts, it's
>> easily possible that we could do better at resolving the merge by
>> walking through both sides and understanding what happened.
>
> It would help if cherry-pick history where recorded somewhere (beyond
> the reflog)...
>
> Cherry-picks should record two parents, like merges.
>
> (Of course, it does no good to know about an unreachable parent, when
> a commit with two parents is pushed to a repo that doesn't have one of
> those parents, which can happen when topic branches aren't pushed
> upstream.)

There was (long time ago) a long thread about idea of adding some
kind of 'weak' references (links), 'weakparent' that can be 
automatically used by Git but do not pollute the commit message,
and do not affect reachability calculations.  Ultimately it went
nowhere (as you can see) - there were many problems.

For example: how it would work for reverts and rebases?

-- 
Jakub Narębski

Re: cherry picking and merge

[Nico Williams]

On Wed, Aug 6, 2014 at 10:58 AM, Jakub Narębski <jnareb@gmail.com> wrote:
> W dniu 2014-08-01 22:55, Nico Williams pisze:
>> It would help if cherry-pick history where recorded somewhere (beyond
>> the reflog)...
>>
>> Cherry-picks should record two parents, like merges.
>>
>> (Of course, it does no good to know about an unreachable parent, when
>> a commit with two parents is pushed to a repo that doesn't have one of
>> those parents, which can happen when topic branches aren't pushed
>> upstream.)
>
> There was (long time ago) a long thread about idea of adding some
> kind of 'weak' references (links), 'weakparent' that can be automatically
> used by Git but do not pollute the commit message,
> and do not affect reachability calculations.  Ultimately it went
> nowhere (as you can see) - there were many problems.

My proposal was to put this sort of ancillary history info in a
"branch object" (and that branches should be objects).  This would
have a number of benefits, not the least of which is that at push time
you can drop such ancillary history without having to alter the
commits being pushed.

> For example: how it would work for reverts and rebases?

Reverts upstream?  The revert should record the commit hash of the
commit it reverts (but file-level reverts lose), so that this could be
noticed.

Rebases upstream?  Well, that shouldn't happen, but if it does then
you must rebase --onto and any cherry-picks of upstream rebased
commits lose their ties to those (but this can be detected).

In general recording more metadata (assuming there's not privacy
issues to consider) can't hurt.  Using it might, but having the option
to can also help.

Nico
--

Re: cherry picking and merge

[Junio C Hamano]

Jakub Narębski <jnareb@gmail.com> writes:

> There was (long time ago) a long thread about idea of adding some
> kind of 'weak' references (links), 'weakparent' that can be
> automatically used by Git but do not pollute the commit message,
> and do not affect reachability calculations.  Ultimately it went
> nowhere (as you can see) - there were many problems.
>
> For example: how it would work for reverts and rebases?

Perhaps some digging in the list archive before typing is in order.
This may be a good starting point.

http://thread.gmane.org/gmane.comp.version-control.git/46770/focus=46799

Re: cherry picking and merge

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Jakub Narębski <jnareb@gmail.com> writes:
>
>> There was (long time ago) a long thread about idea of adding some
>> kind of 'weak' references (links), 'weakparent' that can be
>> automatically used by Git but do not pollute the commit message,
>> and do not affect reachability calculations.  Ultimately it went
>> nowhere (as you can see) - there were many problems.
>>
>> For example: how it would work for reverts and rebases?
>
> Perhaps some digging in the list archive before typing is in order.
> This may be a good starting point.
>
> http://thread.gmane.org/gmane.comp.version-control.git/46770/focus=46799

Here is another.

http://thread.gmane.org/gmane.comp.version-control.git/19126/focus=19149

Re: cherry picking and merge

[Mike Stump]

On Aug 1, 2014, at 1:50 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> 
>       And on the other hand a one-patch-at-a-time merge would
>       try to apply X (with no effect, since it's already applied)
>       and then try to apply the revert of X.  The net effect would
>       be to revert X from "master" (bad)!

Yeah, I appreciate that.  I know the type of user that would do that, and I understand why you would want to do that and even do that by default.

However, as an expert user, I don’t need that particular type of hand holding given the cost of more conflicts and would like an option to let me to choose to have fewer conflicts when merging.

Re: cherry picking and merge

[Mike Stump]

On Aug 1, 2014, at 1:02 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> 
> Do you mean that "git merge" should be aware of what changes you have
> already cherry-picked?

Yes, it amounts to that.

> It isn't, and that's deliberate

Deliberate bugs are still bugs.  In time, users will either wear you down until you fix it, or they will move on to other systems that work better.

> ("git merge" is designed to be simple as possible, though no more simple than that).

I sketched a solution that retains a simple git merge…

> This way, if on a side branch someone makes a change that would conflict with "master" and
> then backs it out, then the branch can still merge cleanly.

Yeah, my solution doesn’t impinge upon that working nicely.  In it, I make cherry use scratch branches to record meta information so that the existing git merge just works.  git cherry is free to do the same.  Having a git cherry that fully interoperates with git merge, is a feature.

> Even in those workflows, it's possible to have conflicts due to

> genuinely conflicting changes, even with no cherry-pick involved.  I
> find the '[merge] conflictstyle = diff3' setting (see git-config(1))
> and git-rerere(1) to be helpful in making that less painful.

I think those two should be the default, but it is easy enough to turn them on that it doesn’t matter much.  In my environment, I have both on.

Re: cherry picking and merge

[Jonathan Nieder]

Mike Stump wrote:
> On Aug 1, 2014, at 1:02 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:

>> It isn't, and that's deliberate
>
> Deliberate bugs are still bugs.

Yes, you and I disagree about what the behavior should be.

I actively prefer the current behavior over the one you proposed,
unless I'm misunderstanding the one you proposed.  As I understand it,
there would be no way to undo the mistake of cherry-picking a change
that didn't belong on "maint".

You said that 3-way merge doesn't fit your idea of what a merge is.
It does fit mine.

I'm even slightly against there being an option for the 'git cherry'
based thing.  I think it would be a support burden.  But if someone
else wants to do the work of implementing it, I wouldn't stop them
(and would instead just help make sure the documentation is crystal
clear).

Hoping that clarifies,
Jonathan