rethinking patch management with GIT / topgit

rethinking patch management with GIT / topgit

[Thomas Koch]

Hi,

I'd like to argue, that topgit (tg) fails to fullfill it's task and propose a 
different approach to the problem of patch management on top of git.

IMHO tg fails due to the following reasons:

- it's to complicate
- AFAIK nobody has solved the problem of managing different patchsets with tg
- it pollutes the patch branches with metadata (.topmsg, .topdeps)
- even after 1 1/2 years topgit isn't feature complete and development seems 
to stagnate
- there also aren't best practices documented
- it quickly fills the list of branches
- it's very easy to break something

Although the above is quite a harsh judgement, I'd like to note, that tg has 
had its merrit to promote one right idea: Patches should be managed in the 
form of branches by the means of the underlying VCS and not as simple 
patchfiles.

The new approach I'd like to propose is based on two core ideas:

1) merge commits to save history

git allows the free creation of merge commits with an arbitrary content tree. 
So we can create an octupus merge combining all patch-branches while the 
content of this merge can contain meta data about a patchset instead of the 
content of the merged commits.
Such a merge commit thus provides a pointer to all the history of all patches 
and can contain all metadata about the merged patch branches. Pushing only a 
branch or tag with this commit to a central repository thus pushes all the 
history of all contained patches.

2) collapse / expand branches

Managing a Debian package in stable, unstable and experimental can quickly 
doom you to manage at least three different patchsets with possibly three 
different roots. The list of branches grows in the douzens. Which branches 
belong to which patchset? Which branches are already pushed or pulled?
It may be an advantage to see only some main branches and the branches of one 
patchset I'm currently working on.
The tool I propose would manage each patchset in one branch per patchset. This 
branch has two roles:
- keep the metadata of the patchset as files in the content tree
- keep pointers to the top of the patch-branches in the parent pointers of the 
commit
With the help of such a patchset-metadata-branch I can:
- delete the patch-branches of one patchset while the commits are kept save
- recreate the patch-branches of one patchset

For Debian packaging one last function is needed: export a patchset as quilt 
series.

Is my explanation understandable? Could this approach work or did I miss 
something? Who has time to implement it (GSOC?)?

Best regards,

Thomas Koch, http://www.koch.ro

Re: rethinking patch management with GIT / topgit

[Petr Baudis]

  Hi!

On Sat, Mar 20, 2010 at 06:02:51PM +0100, Thomas Koch wrote:
> I'd like to argue, that topgit (tg) fails to fullfill it's task and propose a 
> different approach to the problem of patch management on top of git.
> 
> IMHO tg fails due to the following reasons:
> 
> - it pollutes the patch branches with metadata (.topmsg, .topdeps)

I'd like to single this out - this is a very arbitrary decision and has
its distinct pros and cons compared to alternative approaches, and was
not taken lightly. It wouldn't be that difficult to go in a different
way and it's not a fundamental limitation.

> - AFAIK nobody has solved the problem of managing different patchsets with tg

Why not just manage them in separate repositories? If you want to pick
patches between patchsets, that should be easily posible through remotes.

> - it's to complicate
> - even after 1 1/2 years topgit isn't feature complete and development seems 
> to stagnate
> - there also aren't best practices documented
> - it quickly fills the list of branches
> - it's very easy to break something
> 
> Although the above is quite a harsh judgement, I'd like to note, that tg has 
> had its merrit to promote one right idea: Patches should be managed in the 
> form of branches by the means of the underlying VCS and not as simple 
> patchfiles.

  I agree that topgit really sucks in many areas. :) While it is usable
in practice if you know what you are doing, in some aspects it never got
out of the proof-of-concept stage.

> The new approach I'd like to propose is based on two core ideas:
> 
> 1) merge commits to save history
> 
> git allows the free creation of merge commits with an arbitrary content tree. 
> So we can create an octupus merge combining all patch-branches while the 
> content of this merge can contain meta data about a patchset instead of the 
> content of the merged commits.
> Such a merge commit thus provides a pointer to all the history of all patches 
> and can contain all metadata about the merged patch branches. Pushing only a 
> branch or tag with this commit to a central repository thus pushes all the 
> history of all contained patches.

  But how is this different from TopGit's approach? TopGit doesn't use
octopus merges for various reasons (I mainly guess since I was too lazy
to implement that), but that's more of an implementation detail than
anything conceptual.

> 2) collapse / expand branches
> 
> Managing a Debian package in stable, unstable and experimental can quickly 
> doom you to manage at least three different patchsets with possibly three 
> different roots. The list of branches grows in the douzens. Which branches 
> belong to which patchset? Which branches are already pushed or pulled?
> It may be an advantage to see only some main branches and the branches of one 
> patchset I'm currently working on.
> The tool I propose would manage each patchset in one branch per patchset. This 
> branch has two roles:
> - keep the metadata of the patchset as files in the content tree
> - keep pointers to the top of the patch-branches in the parent pointers of the 
> commit
> With the help of such a patchset-metadata-branch I can:
> - delete the patch-branches of one patchset while the commits are kept save
> - recreate the patch-branches of one patchset

  The disadvantage is that you will have to invent a lot of arbitrary
metadata and wide range of commands to manipulate and work with this
metadata, all to accomplish something that _is actually one of the main
Git functions to do_!

  Wouldn't it be better to do the collapsing/expanding instead, e.g.
have a convention for patchset/stage branch tying up all patchset/*
branches, and an alias that lists only */stage branches and another that
lists only patchset/* minus patchset/stage branches.

> For Debian packaging one last function is needed: export a patchset as quilt 
> series.
> 
> Is my explanation understandable? Could this approach work or did I miss 
> something? Who has time to implement it (GSOC?)?

  It's especially not clear at all whether you propose to start over or
just make two improvements to TopGit, and if the former, how would your
approach differ from TopGit in all the other aspects. It's all a bit
ambiguous.

-- 
				Petr "Pasky" Baudis
http://pasky.or.cz/ | "Ars longa, vita brevis." -- Hippocrates

Re: rethinking patch management with GIT / topgit

[Thomas Koch]

Petr Baudis:
> > - it pollutes the patch branches with metadata (.topmsg, .topdeps)
> 
> I'd like to single this out - this is a very arbitrary decision and has
> its distinct pros and cons compared to alternative approaches, and was
> not taken lightly. It wouldn't be that difficult to go in a different
> way and it's not a fundamental limitation.
It's also the most minor problem I see, rather a matter of taste.

> > - AFAIK nobody has solved the problem of managing different patchsets
> > with tg
> 
> Why not just manage them in separate repositories? If you want to pick
> patches between patchsets, that should be easily posible through remotes.
You mean having multiple local working directories per project? That's not 
nice. If you also work on the project with an IDE, it'd most probably mean 
setting up the same project multiple times.

>   I agree that topgit really sucks in many areas. :) While it is usable
> in practice if you know what you are doing, in some aspects it never got
> out of the proof-of-concept stage.
Thank you for not taking it personaly! :-)

> > 1) merge commits to save history
> >
> > git allows the free creation of merge commits with an arbitrary content
> > tree. So we can create an octupus merge combining all patch-branches
> > while the content of this merge can contain meta data about a patchset
> > instead of the content of the merged commits.
> > Such a merge commit thus provides a pointer to all the history of all
> > patches and can contain all metadata about the merged patch branches.
> > Pushing only a branch or tag with this commit to a central repository
> > thus pushes all the history of all contained patches.
> 
>   But how is this different from TopGit's approach? TopGit doesn't use
> octopus merges for various reasons (I mainly guess since I was too lazy
> to implement that), but that's more of an implementation detail than
> anything conceptual.
Maybe I didn't make clear that the commits on the patchset branch should not 
necessarily contain the merged content of all patches. The role of the merges 
is solely to keep pointers to the commits making up the patch branches.
Now that you're asking there may however be a similarity to topgit in that you 
can have one topgit controlled branch that combines all topgit branches 
belonging to the same patchset.
However there still remain some differences:

- in tg you need to push all tg branches while in my approach you'd need to 
push only one branch per patchset
- As far as I understand tg, all commands are repository global while for the 
Debian packaging use case you'd need patchset local commands:
  - tg export <patchset>
    export <patchset> as a quilt series

  - tg summary <patchset>
    gives summary information for the branches of <patchset>

  - tg delete <patchset> <branch>
    removes <branch> from <patchset>. However old history of <branch> is still 
recorded in the history of the <patchset>-branch

  - tg add <patchset> <branch>
    adds <branch> to <patchset>
    (The question is open, where to record the merge resolution of all 
patches)

  - tg recreate <patchset> <newbase> <new patchset name>
    Creates a new patchset with root <newbase> by creating new patch branches 
for each patch branch in <patchset>
    This command is useful if you need to keep the old patchset to maintain an 
older version of your Debian package.
    
> > 2) collapse / expand branches
> >
> > Managing a Debian package in stable, unstable and experimental can
> > quickly doom you to manage at least three different patchsets with
> > possibly three different roots. The list of branches grows in the
> > douzens. Which branches belong to which patchset? Which branches are
> > already pushed or pulled? It may be an advantage to see only some main
> > branches and the branches of one patchset I'm currently working on.
> > The tool I propose would manage each patchset in one branch per patchset.
> > This branch has two roles:
> > - keep the metadata of the patchset as files in the content tree
> > - keep pointers to the top of the patch-branches in the parent pointers
> > of the commit
> > With the help of such a patchset-metadata-branch I can:
> > - delete the patch-branches of one patchset while the commits are kept
> > save - recreate the patch-branches of one patchset
> 
>   The disadvantage is that you will have to invent a lot of arbitrary
> metadata and wide range of commands to manipulate and work with this
> metadata, all to accomplish something that _is actually one of the main
> Git functions to do_!
I don't see this. What do I miss? All metadata I'd need to manage is:
- one file with the name of each branch, it's last commit and the names of its 
dependencies (the root of the patchset, if empty)
- one message file for each patch
- the root of the patchset

The example commands given above would manipulate or read the patchset branch 
in the background much like pristine-tar does it with its metadata branch.
 
>   Wouldn't it be better to do the collapsing/expanding instead, e.g.
> have a convention for patchset/stage branch tying up all patchset/*
> branches, and an alias that lists only */stage branches and another that
> lists only patchset/* minus patchset/stage branches.
So you propose not to delete/recreate the patch branches but to provide extra 
commands to list only the desired subset of branches? This would still mean 
that I'd see douzens of patch branches in gitweb and that I't need to push 
douzens of branches to my co-packagers. - That doesn't solve it for me.
It may also be of interest, that Guido Günther, the author of git-
buildpackage[1] wrote a command git-pq to "maintain debian/patches on a patch 
queue branch". - There seems to be a need to have few branches and that it 
seems to be practical to create and destroy patch branches in your workflow.

[1] https://honk.sigxcpu.org/piki/projects/git-buildpackage/

> > Is my explanation understandable? Could this approach work or did I miss
> > something? Who has time to implement it (GSOC?)?
> 
>   It's especially not clear at all whether you propose to start over or
> just make two improvements to TopGit, and if the former, how would your
> approach differ from TopGit in all the other aspects. It's all a bit
> ambiguous.
I hope I managed to make it clearer this time. I believe my proposals are 
incompatible to topgit and thus would require a new project from scratch. 
However it may well be that some code from topgit could be reused.

Best regards,

Thomas Koch, http://www.koch.ro

Re: rethinking patch management with GIT / topgit

[Petr Baudis]

On Sat, Mar 20, 2010 at 07:53:34PM +0100, Thomas Koch wrote:
> Petr Baudis:
>   - tg recreate <patchset> <newbase> <new patchset name>
>     Creates a new patchset with root <newbase> by creating new patch branches 
> for each patch branch in <patchset>
>     This command is useful if you need to keep the old patchset to maintain an 
> older version of your Debian package.

This means wiping out history again; in TopGit, you would ideally
checkpoint all the branches within the patchset, then just tg update
your branches. It's another matter that the former is now difficult to
do easily.

> I don't see this. What do I miss? All metadata I'd need to manage is:
> - one file with the name of each branch, it's last commit and the names of its 
> dependencies (the root of the patchset, if empty)
> - one message file for each patch
> - the root of the patchset
> 
> The example commands given above would manipulate or read the patchset branch 
> in the background much like pristine-tar does it with its metadata branch.

Hmm, to a degree I misunderstood your idea. You would still need quirky
commands to update the references when you make a new commit, to go to a
certain patch (at which point git will start acting a bit annoyed since
it's not on a branch), etc. Other than that, I can offer only my gut
feeling.  ;-)

> >   Wouldn't it be better to do the collapsing/expanding instead, e.g.
> > have a convention for patchset/stage branch tying up all patchset/*
> > branches, and an alias that lists only */stage branches and another that
> > lists only patchset/* minus patchset/stage branches.
> So you propose not to delete/recreate the patch branches but to provide extra 
> commands to list only the desired subset of branches? This would still mean 
> that I'd see douzens of patch branches in gitweb and that I't need to push 
> douzens of branches to my co-packagers. - That doesn't solve it for me.

There are already some patches in the wild to make gitweb topgit-aware;
I don't see why is the latter a problem.

> I hope I managed to make it clearer this time. I believe my proposals are 
> incompatible to topgit and thus would require a new project from scratch. 

Yes, I finally understood what do you mean, sorry for being a bit dense.

-- 
				Petr "Pasky" Baudis
http://pasky.or.cz/ | "Ars longa, vita brevis." -- Hippocrates

Re: rethinking patch management with GIT / topgit

[Thomas Koch]

Hi Petr,

Petr Baudis:
> On Sat, Mar 20, 2010 at 07:53:34PM +0100, Thomas Koch wrote:
> > Petr Baudis:
> >   - tg recreate <patchset> <newbase> <new patchset name>
> >     Creates a new patchset with root <newbase> by creating new patch
> > branches for each patch branch in <patchset>
> >     This command is useful if you need to keep the old patchset to
> > maintain an older version of your Debian package.
> 
> This means wiping out history again; in TopGit, you would ideally
> checkpoint all the branches within the patchset, then just tg update
> your branches. It's another matter that the former is now difficult to
> do easily.
It's not necessary to lose history. Creating a new patchset from a given 
patchset could be done in at least three different ways. Given that you've a 
patchset named patches-x.y.z/* and you'd like to create patches-x.y.z+1/* from 
this, you could

- checkout new patch branches from the top of the old patch branches and merge 
upstream into each of them

- recreate (like rebase) the full history of the patch branches on top of the 
new upstream

- collapse the branch history and create one commit per patch branch on top of 
the new upstream

In all three cases you're free to either keep or throw away the old patchset.

Best regards,

Thomas Koch, http://www.koch.ro

Re: rethinking patch management with GIT / topgit

[Petr Baudis]

On Mon, Mar 22, 2010 at 08:59:40AM +0100, Thomas Koch wrote:
> In all three cases you're free to either keep or throw away the old patchset.

Yes, but to the same degree e.g. with StGIT I'm free to keep the head of
the old patch series. That does not mean the operation *preserves* the
history, only that the history is still around somewhere in the
repository, however it won't be around in other incarnations of the
repository and it will not be connected in any way to the current
version of the patchset.

Yes, if you are lucky you can figure out the name of the previous
version, but it's like starting development of each new kernel version
by a clean import of the sources.

-- 
				Petr "Pasky" Baudis
http://pasky.or.cz/ | "Ars longa, vita brevis." -- Hippocrates

Re: rethinking patch management with GIT / topgit

[Thomas Koch]

Petr Baudis:
> On Mon, Mar 22, 2010 at 08:59:40AM +0100, Thomas Koch wrote:
> > In all three cases you're free to either keep or throw away the old
> > patchset.
> 
> Yes, but to the same degree e.g. with StGIT I'm free to keep the head of
> the old patch series. That does not mean the operation *preserves* the
> history, only that the history is still around somewhere in the
> repository, however it won't be around in other incarnations of the
> repository and it will not be connected in any way to the current
> version of the patchset.
> 
> Yes, if you are lucky you can figure out the name of the previous
> version, but it's like starting development of each new kernel version
> by a clean import of the sources.

Hi Petr,

let me copy the list of methods from my last mail:

1) checkout new patch branches from the top of the old patch branches and 
merge upstream into each of them

2) recreate (like rebase) the full history of the patch branches on top of the 
new upstream

3) collapse the branch history and create one commit per patch branch on top 
of the new upstream

>From these methods, 3) loses all history, 2) loses some history but preserves 
the individual history of one patch branch on a new base and 1) preserves all 
history. Let me give an example for method 1). 
You've got a patchset identified by the prefix debian/. Now you want to 
package a new upstream but need to retain the old patchset in case of security 
updates in Debian stable. Debian stable has version 0.1, new upstream is 0.2.
So you
- rename the old patchset from debian/ to debian-0.1/
- clone/copy/recreate (pick a name) a new patchset debian/ on top of 
upstream/0.2. This is done by merging upstream/0.2 into each debian/* branch.
- Once you don't maintain version 0.1 anymore, you can delete the debian-0.1/* 
branches

After these steps, the debian/ branch still contains pointers to all commits 
from the debian-0.1/* branches.
It's an additional question, how to deal with commits that are done in 
debian-0.1/* after the new upstream merge.

Best regards,

Thomas Koch, http://www.koch.ro