[Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

I'd like to announce a project I've been working on for a while.  I sent
this announcement to LKML, but since many people don't subscribe to LKML
directly, and since ksummit-discuss has had several discussions
specifically about patch workflow and development processes, I thought
I'd send the announcement here as well, in case anyone found it useful
for their workflow.

git-series provides a tool for managing patch series with git, tracking
the "history of history". git series tracks changes to the patch series
over time, including rebases and other non-fast-forwarding changes. git
series also tracks a cover letter for the patch series, formats the
series for email, and prepares pull requests.

This makes it easier to collaborate on a patch series, distribution
package, backport, or any other development process that includes
rebasing or non-fast-forward development.

A patch series typically goes through multiple iterations before
submission; the path from idea to RFC to [PATCHv12 1/8] includes many
invocations of git rebase -i. However, while Git tracks and organizes
commits quite well, it doesn't actually track changes to a patch series
at all, outside of the ephemeral reflog. This makes it a challenge to
collaborate on a patch series, distribution package, backport, or any
other development process that includes rebasing or non-fast-forward
development.

Typically, tracking the evolution of a patch series over time involves
moving part of the version control outside of git. You can move the
patch series from git into quilt or a distribution package, and then
version the patch files with git, losing the power of git's tools. Or,
you can keep the patch series in git, and version it via multiple named
branches; however, names like feature-v2, feature-v3-typofix, and
feature-v8-rebased-4.6-alice-fix sound like filenames from corporate
email, not modern version control. And either way, git doesn't track
your cover letter at all.

git-series tracks both a patch series and its evolution within the same
git repository. git-series works entirely with existing git features,
allowing git to push and pull a series to any git repository along with
other branches and tags. Each time you change the patch series, whether
fast-forwarding or not, you can "git series commit" a new version of the
patch series, complete with commit message.

You can rebase a patch series with "git series rebase -i", format it for
submission with "git series format", or send a "please pull" request with
"git series req".  git-series knows the base of your series, so you
don't need to count patches or find a commit hash to run rebase or
format.

If you're interested in trying git-series, see
https://github.com/git-series/git-series for installation instructions
and a "getting started" guide.

I've also documented the internal storage format of git-series at
https://github.com/git-series/git-series/blob/master/INTERNALS.md ,
including the details for how git-series ensures git can always reach,
push, and pull a series.

I'd welcome any feedback, whether on the interface and workflow, the
internals and collaboration, ideas on presenting diffs of patch series,
or anything else.

- Josh Triplett

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[David Howells]

Josh Triplett <josh@joshtriplett.org> wrote:

> I'd like to announce a project I've been working on for a while.  I sent
> this announcement to LKML, but since many people don't subscribe to LKML
> directly, and since ksummit-discuss has had several discussions
> specifically about patch workflow and development processes, I thought
> I'd send the announcement here as well, in case anyone found it useful
> for their workflow.

Can this be used as a direct substitute for stgit for maintaining a patch
series?

David

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Fri, Jul 29, 2016 at 01:20:12PM +0100, David Howells wrote:
> Josh Triplett <josh@joshtriplett.org> wrote:
> > I'd like to announce a project I've been working on for a while.  I sent
> > this announcement to LKML, but since many people don't subscribe to LKML
> > directly, and since ksummit-discuss has had several discussions
> > specifically about patch workflow and development processes, I thought
> > I'd send the announcement here as well, in case anyone found it useful
> > for their workflow.
> 
> Can this be used as a direct substitute for stgit for maintaining a patch
> series?

Yes, that's exactly what I designed it for.  git-series has the added
advantage of tracking the versions of the patch series across rewrites.
stgit just directly rewrites history, like rebase -i does; as far as I
know, it doesn't remember the old history.  You'd have to go to the
reflog for that.  git-series lets you rewrite history (using rebase -i
or any other tool you like), but actually tracks the history of your
patch series across rewrites, complete with commit messages.

Note that git-series doesn't provide a quilt-style push/pop workflow,
with applied and unapplied patches; it just looks at HEAD.

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[David Howells]

Josh Triplett <josh@joshtriplett.org> wrote:

> Note that git-series doesn't provide a quilt-style push/pop workflow,
> with applied and unapplied patches; it just looks at HEAD.

Ah...  In that case it's probably not a sufficient substitute for how I use
stgit.

David

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Christoph Lameter]

On Fri, 29 Jul 2016, David Howells wrote:

> Josh Triplett <josh@joshtriplett.org> wrote:
>
> > Note that git-series doesn't provide a quilt-style push/pop workflow,
> > with applied and unapplied patches; it just looks at HEAD.
>
> Ah...  In that case it's probably not a sufficient substitute for how I use
> stgit.

Did not know that there was stgit. Still stuck on quilt since
I like editing the files directly (its often easier to edit the diffs if
you want to rename things etc).

Interesting projects.

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Fri, Jul 29, 2016 at 03:06:52PM +0100, David Howells wrote:
> Josh Triplett <josh@joshtriplett.org> wrote:
> 
> > Note that git-series doesn't provide a quilt-style push/pop workflow,
> > with applied and unapplied patches; it just looks at HEAD.
> 
> Ah...  In that case it's probably not a sufficient substitute for how I use
> stgit.

Fair enough.  I normally manipulate patches via "git rebase -i".

You could use the two together, though, if you'd like to track the
history you've changed with stgit.  You'd just need to make sure you'd
applied all the patches before doing a git series commit.

- Josh Triplett

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[David Howells]

Josh Triplett <josh@joshtriplett.org> wrote:

> You could use the two together, though, if you'd like to track the
> history you've changed with stgit.  You'd just need to make sure you'd
> applied all the patches before doing a git series commit.

Would it be possible to make stgit run on top of git series?  Or extend git
series to also do the push/pop thing?

David

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Fri, Jul 29, 2016 at 09:21:02AM -0500, Christoph Lameter wrote:
> On Fri, 29 Jul 2016, David Howells wrote:
> 
> > Josh Triplett <josh@joshtriplett.org> wrote:
> >
> > > Note that git-series doesn't provide a quilt-style push/pop workflow,
> > > with applied and unapplied patches; it just looks at HEAD.
> >
> > Ah...  In that case it's probably not a sufficient substitute for how I use
> > stgit.
> 
> Did not know that there was stgit. Still stuck on quilt since
> I like editing the files directly (its often easier to edit the diffs if
> you want to rename things etc).

git series format, edit patches, git reset --hard $(git series base),
git am? :)

More seriously, though: I don't expect git-series to address all
possible workflows for developing a patch series.  It's intentionally a
bit opinionated about some things.  I designed it for a family of
workflows that I regularly encountered and that I regularly saw other
people encounter.

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[James Bottomley]

On Fri, 2016-07-29 at 00:50 -0700, Josh Triplett wrote:
> I'd welcome any feedback, whether on the interface and workflow, the
> internals and collaboration, ideas on presenting diffs of patch 
> series, or anything else.

I've got an offer for you:  The Plumbers PC has been kicking around the
idea of a git MC at Plumbers.  We got as far ask asking Junio Hamano,
who said he'd commit to coming, but didn't want to run the MC.  Would
you be interested in running it?  Since it's a barely formed idea at
the moment, it would still have to go through the usual Plumbers
process of proposing it on

http://wiki.linuxplumbersconf.org/2016:topics

And then getting formal plumbers PC acceptance (Paul McKenney shepherds
this).  You can put me down as a committed attendee because, like a lot
of people, I have quite a bit of scripting around my git infrastructure
which, I suspect, could be shared.

James

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Fri, Jul 29, 2016 at 03:37:24PM +0100, David Howells wrote:
> Josh Triplett <josh@joshtriplett.org> wrote:
> > You could use the two together, though, if you'd like to track the
> > history you've changed with stgit.  You'd just need to make sure you'd
> > applied all the patches before doing a git series commit.
> 
> Would it be possible to make stgit run on top of git series?

Possibly; it would depend on what you mean by "run on top of".  I'm open
to suggestion, though.

> Or extend git series to also do the push/pop thing?

I'll have to give that workflow some thought.  The concept of a "patch
that isn't applied" seems somewhat orthogonal to git-series.

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Daniel Vetter]

On Fri, Jul 29, 2016 at 4:21 PM, Christoph Lameter <cl@linux.com> wrote:
> On Fri, 29 Jul 2016, David Howells wrote:
>
>> Josh Triplett <josh@joshtriplett.org> wrote:
>>
>> > Note that git-series doesn't provide a quilt-style push/pop workflow,
>> > with applied and unapplied patches; it just looks at HEAD.
>>
>> Ah...  In that case it's probably not a sufficient substitute for how I use
>> stgit.
>
> Did not know that there was stgit. Still stuck on quilt since
> I like editing the files directly (its often easier to edit the diffs if
> you want to rename things etc).
>
> Interesting projects.

Shameless plug of our own tooling for maintainer a quilt pile and
tracking it in git:

https://cgit.freedesktop.org/drm-intel/tree/qf?h=maintainer-tools

It even tracks the baseline sha1 and pulls/pushes it in hidden remote
refs/. Which all together allows you to git bisect on the quilt
branch, which is a really powerful thing for a long-lived rebasing
patch pile. quilt+git was the only thing that allowed me to glue
something git bisect capable together.

And the other reason is the same you have: Editing raw patches is
really powerful for doing rebases over mechanical changes. Function
renames become a trivial quilt pop -a ; sed; while quilt push ; do
make ; done.

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Fri, Jul 29, 2016 at 07:55:44AM -0700, James Bottomley wrote:
> On Fri, 2016-07-29 at 00:50 -0700, Josh Triplett wrote:
> > I'd welcome any feedback, whether on the interface and workflow, the
> > internals and collaboration, ideas on presenting diffs of patch 
> > series, or anything else.
> 
> I've got an offer for you:  The Plumbers PC has been kicking around the
> idea of a git MC at Plumbers.  We got as far ask asking Junio Hamano,
> who said he'd commit to coming, but didn't want to run the MC.  Would
> you be interested in running it?  Since it's a barely formed idea at
> the moment, it would still have to go through the usual Plumbers
> process of proposing it on
> 
> http://wiki.linuxplumbersconf.org/2016:topics
> 
> And then getting formal plumbers PC acceptance (Paul McKenney shepherds
> this).  You can put me down as a committed attendee because, like a lot
> of people, I have quite a bit of scripting around my git infrastructure
> which, I suspect, could be shared.

I'd mentioned the idea of a git MC to Paul, and I'd be happy to give a
git-series presentation/BoF at one, but I don't have the bandwidth to
run one.

- Josh Triplett

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Fri, Jul 29, 2016 at 05:00:06PM +0200, Daniel Vetter wrote:
> On Fri, Jul 29, 2016 at 4:21 PM, Christoph Lameter <cl@linux.com> wrote:
> > On Fri, 29 Jul 2016, David Howells wrote:
> >
> >> Josh Triplett <josh@joshtriplett.org> wrote:
> >>
> >> > Note that git-series doesn't provide a quilt-style push/pop workflow,
> >> > with applied and unapplied patches; it just looks at HEAD.
> >>
> >> Ah...  In that case it's probably not a sufficient substitute for how I use
> >> stgit.
> >
> > Did not know that there was stgit. Still stuck on quilt since
> > I like editing the files directly (its often easier to edit the diffs if
> > you want to rename things etc).
> >
> > Interesting projects.
> 
> Shameless plug of our own tooling for maintainer a quilt pile and
> tracking it in git:
> 
> https://cgit.freedesktop.org/drm-intel/tree/qf?h=maintainer-tools
> 
> It even tracks the baseline sha1 and pulls/pushes it in hidden remote
> refs/. Which all together allows you to git bisect on the quilt
> branch, which is a really powerful thing for a long-lived rebasing
> patch pile. quilt+git was the only thing that allowed me to glue
> something git bisect capable together.

By "bisect on the quilt branch", do you mean bisecting between versions
of the overall patch series?

That does seem quite helpful, if you need to figure out at what point a
change occurred in the patch series to break something.  I've filed
https://github.com/git-series/git-series/issues/3 about supporting that.

> And the other reason is the same you have: Editing raw patches is
> really powerful for doing rebases over mechanical changes. Function
> renames become a trivial quilt pop -a ; sed; while quilt push ; do
> make ; done.

Interesting!  I'll have to give that some thought, to figure out if
I can support workflows like that.

What other kinds of changes do you tend to make by editing patches
directly?

I'm tempted to add a "git series filter" that applies filter-branch to
the commits in the series, or something similar to that.  I'll also have
to think about the use case of testing each commit in the series, to see
if that needs support from git-series or just from underlying git tools.

- Josh Triplett

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 4633 bytes --]

Hi Josh,

On Fri, Jul 29, 2016 at 12:50:39AM -0700, Josh Triplett wrote:
> I'd like to announce a project I've been working on for a while.  I sent
> this announcement to LKML, but since many people don't subscribe to LKML
> directly, and since ksummit-discuss has had several discussions
> specifically about patch workflow and development processes, I thought
> I'd send the announcement here as well, in case anyone found it useful
> for their workflow.
> 
> git-series provides a tool for managing patch series with git, tracking
> the "history of history". git series tracks changes to the patch series
> over time, including rebases and other non-fast-forwarding changes. git
> series also tracks a cover letter for the patch series, formats the
> series for email, and prepares pull requests.
> 
> This makes it easier to collaborate on a patch series, distribution
> package, backport, or any other development process that includes
> rebasing or non-fast-forward development.
> 
> A patch series typically goes through multiple iterations before
> submission; the path from idea to RFC to [PATCHv12 1/8] includes many
> invocations of git rebase -i. However, while Git tracks and organizes
> commits quite well, it doesn't actually track changes to a patch series
> at all, outside of the ephemeral reflog. This makes it a challenge to
> collaborate on a patch series, distribution package, backport, or any
> other development process that includes rebasing or non-fast-forward
> development.
> 
> Typically, tracking the evolution of a patch series over time involves
> moving part of the version control outside of git. You can move the
> patch series from git into quilt or a distribution package, and then
> version the patch files with git, losing the power of git's tools. Or,
> you can keep the patch series in git, and version it via multiple named
> branches; however, names like feature-v2, feature-v3-typofix, and
> feature-v8-rebased-4.6-alice-fix sound like filenames from corporate
> email, not modern version control. And either way, git doesn't track
> your cover letter at all.
> 
> git-series tracks both a patch series and its evolution within the same
> git repository. git-series works entirely with existing git features,
> allowing git to push and pull a series to any git repository along with
> other branches and tags. Each time you change the patch series, whether
> fast-forwarding or not, you can "git series commit" a new version of the
> patch series, complete with commit message.
> 
> You can rebase a patch series with "git series rebase -i", format it for
> submission with "git series format", or send a "please pull" request with
> "git series req".  git-series knows the base of your series, so you
> don't need to count patches or find a commit hash to run rebase or
> format.
> 
> If you're interested in trying git-series, see
> https://github.com/git-series/git-series for installation instructions
> and a "getting started" guide.
> 
> I've also documented the internal storage format of git-series at
> https://github.com/git-series/git-series/blob/master/INTERNALS.md ,
> including the details for how git-series ensures git can always reach,
> push, and pull a series.
> 
> I'd welcome any feedback, whether on the interface and workflow, the
> internals and collaboration, ideas on presenting diffs of patch series,
> or anything else.

Thats pretty neat, thanks! I do loads of rebasing when preparing
patchsets, and tend to just rely on reflog if something goes wrong and I
need to look back at the history.

It looks like it could work quite well for tracking linux-next style
trees (not necessarily linux-next itself) that automatically regenerate
a bunch of merges of different branches each time one of them changes
(if you ignore git series format & git series rebase, since they
wouldn't be relevant and don't like merge commits anyway).

It may be helpful to be able to easily tag versions of the series, e.g.
rfc/v1/v2 etc when you actually submit them (sure, the git-series ref
could be tagged manually according to some naming convention, or perhaps
storing it in the metadata would make it more self-contained).

I use various git aliases / scripts for diffing patchsets, mainly based
around vimdiff'ing git log A...B --{left,right}-only | grep to filter
out irrelevant lines of diff (like index lines), but that all sort of
breaks down as soon as you reorder commits, requiring me to e.g.
manually edit one side and :diffupdate to be able to check the changes.

Cheers
James

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Daniel Vetter]

On Fri, Jul 29, 2016 at 5:18 PM, Josh Triplett <josh@joshtriplett.org> wrote:
> On Fri, Jul 29, 2016 at 05:00:06PM +0200, Daniel Vetter wrote:
>> On Fri, Jul 29, 2016 at 4:21 PM, Christoph Lameter <cl@linux.com> wrote:
>> > On Fri, 29 Jul 2016, David Howells wrote:
>> >
>> >> Josh Triplett <josh@joshtriplett.org> wrote:
>> >>
>> >> > Note that git-series doesn't provide a quilt-style push/pop workflow,
>> >> > with applied and unapplied patches; it just looks at HEAD.
>> >>
>> >> Ah...  In that case it's probably not a sufficient substitute for how I use
>> >> stgit.
>> >
>> > Did not know that there was stgit. Still stuck on quilt since
>> > I like editing the files directly (its often easier to edit the diffs if
>> > you want to rename things etc).
>> >
>> > Interesting projects.
>>
>> Shameless plug of our own tooling for maintainer a quilt pile and
>> tracking it in git:
>>
>> https://cgit.freedesktop.org/drm-intel/tree/qf?h=maintainer-tools
>>
>> It even tracks the baseline sha1 and pulls/pushes it in hidden remote
>> refs/. Which all together allows you to git bisect on the quilt
>> branch, which is a really powerful thing for a long-lived rebasing
>> patch pile. quilt+git was the only thing that allowed me to glue
>> something git bisect capable together.
>
> By "bisect on the quilt branch", do you mean bisecting between versions
> of the overall patch series?

Yes. I can't show you a live branch because it's internal, but in the
patches/ directory managed by quilt there's 2 additional things:
- .git dir to manage just teh patches directory as a git branch
- config file which only stores the baseline sha1

The parent source dir is any git repo you want really.

There's convenience commands to rebase to a new baseline sha1 in the
parent repo and adjust the quilt branch accordingly. And there's a
special checkout command to check out the current baseline into the
parent repo and then apply the quil patches on top. As long as you run
that before you test a revision git bisect wants you to test, you can
bisect on the quilt branch.

The other magic bit is that it stores the quilt branch in
refs/quilt-branches (also synced in the remote) and the baselines as
tags in refs/quilt-tags, plus it keeps the two .git directories
somewhat in sync. That way you can git push/pull and both the quilt
branch _and_ all the baseline tags needed to reconstruct each commit
get transferred. But since it's a special refs/quilt-* namespace this
is only done when you run the setup command. Anyone using plain git
only observes a normal rebasing git branch and none of the underlying
magic.

> That does seem quite helpful, if you need to figure out at what point a
> change occurred in the patch series to break something.  I've filed
> https://github.com/git-series/git-series/issues/3 about supporting that.
>
>> And the other reason is the same you have: Editing raw patches is
>> really powerful for doing rebases over mechanical changes. Function
>> renames become a trivial quilt pop -a ; sed; while quilt push ; do
>> make ; done.
>
> Interesting!  I'll have to give that some thought, to figure out if
> I can support workflows like that.
>
> What other kinds of changes do you tend to make by editing patches
> directly?

The other upshot of raw patches is that you can use horrible risky
tools like wiggle to force a patch to apply. git is a lot more strict
and ime wiggle helps you with a lot of simple rebase conflicts. I have
small helper scripts to integrate wiggle both into that quilt flow
script, but also into normal git rebase.

> I'm tempted to add a "git series filter" that applies filter-branch to
> the commits in the series, or something similar to that.  I'll also have
> to think about the use case of testing each commit in the series, to see
> if that needs support from git-series or just from underlying git tools.

Another one I really started to like is that you can visually compare
changes to a patch. It takes a bit of experience reading them, but
after a while diffs-of-patches start to make as much sense as a plain
patch, e.g.
- Context movements easily stand out, they only touch @@ lines or
lines not starting in +/-.
- Rebasing adjustements in the patch itself (in the + lines of the patch).
- Direct code conflicts with upstream (anything that changes a - line
in the patch).

I think that's the other massive benefit of working on a pile of patches.

But the problem is that all three (bisecting, sed on raw patches and
patch-diffs) are only really useful if you have to maintain a rebasing
tree over a really long time (months/years). I think for a normal
feature patch series your git series is more than sufficient.
-Daniel

PS: Since we work for the same company I can actually show you the
internal branch if you're interested in how this works ;-)
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Fri, Jul 29, 2016 at 05:40:05PM +0200, Daniel Vetter wrote:
> On Fri, Jul 29, 2016 at 5:18 PM, Josh Triplett <josh@joshtriplett.org> wrote:
> > On Fri, Jul 29, 2016 at 05:00:06PM +0200, Daniel Vetter wrote:
> >> And the other reason is the same you have: Editing raw patches is
> >> really powerful for doing rebases over mechanical changes. Function
> >> renames become a trivial quilt pop -a ; sed; while quilt push ; do
> >> make ; done.
> >
> > Interesting!  I'll have to give that some thought, to figure out if
> > I can support workflows like that.
> >
> > What other kinds of changes do you tend to make by editing patches
> > directly?
> 
> The other upshot of raw patches is that you can use horrible risky
> tools like wiggle to force a patch to apply. git is a lot more strict
> and ime wiggle helps you with a lot of simple rebase conflicts. I have
> small helper scripts to integrate wiggle both into that quilt flow
> script, but also into normal git rebase.

That does sound useful.  It'd be nice to have "git apply --wiggle" and
"git am --wiggle", too.

> > I'm tempted to add a "git series filter" that applies filter-branch to
> > the commits in the series, or something similar to that.  I'll also have
> > to think about the use case of testing each commit in the series, to see
> > if that needs support from git-series or just from underlying git tools.
> 
> Another one I really started to like is that you can visually compare
> changes to a patch. It takes a bit of experience reading them, but
> after a while diffs-of-patches start to make as much sense as a plain
> patch, e.g.
> - Context movements easily stand out, they only touch @@ lines or
> lines not starting in +/-.
> - Rebasing adjustements in the patch itself (in the + lines of the patch).
> - Direct code conflicts with upstream (anything that changes a - line
> in the patch).
> 
> I think that's the other massive benefit of working on a pile of patches.

That's actually the output I'm planning to provide for "git series diff"
or "git series log -p" to show the diff between patch series.

> But the problem is that all three (bisecting, sed on raw patches and
> patch-diffs) are only really useful if you have to maintain a rebasing
> tree over a really long time (months/years). I think for a normal
> feature patch series your git series is more than sufficient.

I'd like git-series to work for both workflows.  The feature patch case
is definitely easier, but I'd like to support long-standing rebasing
trees too.  Use cases include maintainer trees like yours, as well as
distribution packaging repositories with patches, and vendor/product
kernels.

> PS: Since we work for the same company I can actually show you the
> internal branch if you're interested in how this works ;-)

Definitely interested; perhaps when I'm back from sabbatical. :)

- Josh Triplett

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Luis R. Rodriguez]

On Fri, Jul 29, 2016 at 09:21:04AM -0700, Josh Triplett wrote:
> On Fri, Jul 29, 2016 at 05:40:05PM +0200, Daniel Vetter wrote:
> > On Fri, Jul 29, 2016 at 5:18 PM, Josh Triplett <josh@joshtriplett.org> wrote:
> > > On Fri, Jul 29, 2016 at 05:00:06PM +0200, Daniel Vetter wrote:
> > >> And the other reason is the same you have: Editing raw patches is
> > >> really powerful for doing rebases over mechanical changes. Function
> > >> renames become a trivial quilt pop -a ; sed; while quilt push ; do
> > >> make ; done.
> > >
> > > Interesting!  I'll have to give that some thought, to figure out if
> > > I can support workflows like that.
> > >
> > > What other kinds of changes do you tend to make by editing patches
> > > directly?
> > 
> > The other upshot of raw patches is that you can use horrible risky
> > tools like wiggle to force a patch to apply. git is a lot more strict
> > and ime wiggle helps you with a lot of simple rebase conflicts. I have
> > small helper scripts to integrate wiggle both into that quilt flow
> > script, but also into normal git rebase.
> 
> That does sound useful.  It'd be nice to have "git apply --wiggle" and
> "git am --wiggle", too.

A bigger pipe dream is inference of SmPL grammar of a patch, then use
SmPL to address the rebase. Inference work is in theory possible and
R&D is ongoing in that direction. That should make this scale.

  Luis

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Bird, Timothy]

> -----Original Message-----
> From: ksummit-discuss-bounces@lists.linuxfoundation.org [mailto:ksummit-
> discuss-bounces@lists.linuxfoundation.org] On Behalf Of Christoph Lameter
> Sent: Friday, July 29, 2016 7:21 AM
> To: David Howells <dhowells@redhat.com>
> Cc: ksummit-discuss@lists.linuxfoundation.org
> Subject: Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch
> series over time
> 
> On Fri, 29 Jul 2016, David Howells wrote:
> 
> > Josh Triplett <josh@joshtriplett.org> wrote:
> >
> > > Note that git-series doesn't provide a quilt-style push/pop workflow,
> > > with applied and unapplied patches; it just looks at HEAD.
> >
> > Ah...  In that case it's probably not a sufficient substitute for how I use
> > stgit.
> 
> Did not know that there was stgit. Still stuck on quilt since
> I like editing the files directly (its often easier to edit the diffs if
> you want to rename things etc).
> 
> Interesting projects.

Indeed.  For many years I used a quilt-based workflow.  Recently
I've been trying to convert to a pure git-based workflow, but for
some reason git rebase -I always seems to give me problems.  It
always takes me much longer to just move some hunk from one
commit to another than it did in quilt using the patches and vi.

I don't know if git-series addresses that particular problem, but
Josh mentioned enough other problems that I have every time I
prepare patches for upstream, that I definitely plan to check out
the tool.
 -- Tim

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[James Bottomley]

On Fri, 2016-07-29 at 17:52 +0000, Bird, Timothy wrote:
> 
> > -----Original Message-----
> > From: ksummit-discuss-bounces@lists.linuxfoundation.org [mailto:ksu
> > mmit-
> > discuss-bounces@lists.linuxfoundation.org] On Behalf Of Christoph
> > Lameter
> > Sent: Friday, July 29, 2016 7:21 AM
> > To: David Howells <dhowells@redhat.com>
> > Cc: ksummit-discuss@lists.linuxfoundation.org
> > Subject: Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes
> > to a patch
> > series over time
> > 
> > On Fri, 29 Jul 2016, David Howells wrote:
> > 
> > > Josh Triplett <josh@joshtriplett.org> wrote:
> > > 
> > > > Note that git-series doesn't provide a quilt-style push/pop
> > > > workflow,
> > > > with applied and unapplied patches; it just looks at HEAD.
> > > 
> > > Ah...  In that case it's probably not a sufficient substitute for
> > > how I use
> > > stgit.
> > 
> > Did not know that there was stgit. Still stuck on quilt since
> > I like editing the files directly (its often easier to edit the
> > diffs if
> > you want to rename things etc).
> > 
> > Interesting projects.
> 
> Indeed.  For many years I used a quilt-based workflow.  Recently
> I've been trying to convert to a pure git-based workflow, but for
> some reason git rebase -I always seems to give me problems.  It
> always takes me much longer to just move some hunk from one
> commit to another than it did in quilt using the patches and vi.

I do this too, but I also work with the diff when moving hunks.  What I
do is:

git checkout <removal commit>
git show <removal commit> > tmp.diff
vi tmp.diff so it only has the hunks I want to remove
patch -p1 -R < ~/tmp.diff
git commit --amend -a
git checkout <add commit>
patch -p1 < ~/tmp.diff
git commit --amend -a

It's a bit cumbersome, but you can script it.  If there are better ways
to do it, I'm interested.

James

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[David Howells]

James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> I do this too, but I also work with the diff when moving hunks.  What I
> do is:
> 
> git checkout <removal commit>
> git show <removal commit> > tmp.diff
> vi tmp.diff so it only has the hunks I want to remove
> patch -p1 -R < ~/tmp.diff
> git commit --amend -a
> git checkout <add commit>
> patch -p1 < ~/tmp.diff
> git commit --amend -a
> 
> It's a bit cumbersome, but you can script it.  If there are better ways
> to do it, I'm interested.

One of the reasons I use stgit is when it comes to excising changes into a
separate patch, it makes it a lot easier:

	 stg show >tmp.diff	# Export contents of current patch
	 emacs tmp.diff		# Edit so it only has the hunks I want to remove
	 stg pop		# Pop the patch I want to remove part of
	 stg new foo		# Create a new patch
	 stg fold tmp.diff	# ... and apply the patch created above
	 emacs ...		# Fix up any bits
	 stg refresh		# ... and make a commit out of it
	 stg push		# Push the patch I wanted to remove from

The push at the end then attempts to modify the patch you previously popped so
as to effect the excision.  If it can't quite manage it, you'll get <<<<<<<
and suchlike markers indicating the bit it had difficulty with.

I can then very easily switch the order of the patches:

	stg float foo

to put the excised patch last.

I can also:

	 stg pop		# Pop the patch I want to remove part of
	 stg new foo		# Create a new patch
	 emacs ...		# Make the changes I want directly
	 stg refresh		# ... and make a commit out of it
	 stg push		# Reapply the previously popped patch.

As before, the last line will attempt to modify the previously popped patch,
and if it can't it will insert merge failure markers.

David

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 1895 bytes --]

On Fri, Jul 29, 2016 at 10:57:26AM -0700, James Bottomley wrote:
> On Fri, 2016-07-29 at 17:52 +0000, Bird, Timothy wrote:
> > I've been trying to convert to a pure git-based workflow, but for
> > some reason git rebase -I always seems to give me problems.  It
> > always takes me much longer to just move some hunk from one
> > commit to another than it did in quilt using the patches and vi.
> 
> I do this too, but I also work with the diff when moving hunks.  What I
> do is:
> 
> git checkout <removal commit>

> git show <removal commit> > tmp.diff
> vi tmp.diff so it only has the hunks I want to remove
> patch -p1 -R < ~/tmp.diff

another alternative:
git checkout -p HEAD~

and...

> git commit --amend -a
> git checkout <add commit>

> patch -p1 < ~/tmp.diff
> git commit --amend -a

git checkout -p <pre-rebase commit id> (assuming you're moving the hunk
further towards the head of the branch).

thats easy to do from a rebase -i, as you have all the commit ids right
there and can just add a line:
x git checkout -p <commit_id>

> 
> It's a bit cumbersome, but you can script it.  If there are better ways
> to do it, I'm interested.

I often find myself just splitting it up with rebase -i. I.e. just
duplicate the removal commit, and add "x git checkout -p HEAD~" in
between them. The second pick will end up just with the hunk you've
removed.

Then you can mark it as a fixup on a second run of rebase -i, or even on
the first run using commit --fixup and rebase --autosquash:

pick abc removal commit
x git checkout -p HEAD~
pick abc removal commit
x git commit --amend --fixup add_commit

For more complicated extractions where checkout -p HEAD~ gets confusing
or when you want to move a hunk backwards while avoiding nearby
conflicts, precede the removal commit with:
x git checkout -p <removal commit>

Cheers
James

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Steven Rostedt]

On Fri, 29 Jul 2016 10:57:26 -0700
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:


> I do this too, but I also work with the diff when moving hunks.  What I
> do is:
> 
> git checkout <removal commit>
> git show <removal commit> > tmp.diff
> vi tmp.diff so it only has the hunks I want to remove
> patch -p1 -R < ~/tmp.diff
> git commit --amend -a
> git checkout <add commit>
> patch -p1 < ~/tmp.diff
> git commit --amend -a
> 
> It's a bit cumbersome, but you can script it.  If there are better ways
> to do it, I'm interested.

I do that quite a lot too. But I also use a git quilt mix a bit.

When I get a patch that is based on my work and fails to apply, I do:

quilt import /tmp/ftrace.patch
quilt push -f

see what breaks, and fix it up. Sometimes I have to go and change the
change log, or subject, especially if the patch is from a non native
English speaker, and the change log simply doesn't make much sense
(unless you know what the patch is already doing). Then:

quilt refresh
do-git-am.pl patches/ftrace.patch

Where do-git-am.pl is my script that will add Cc's and Link: tags.

-- Steve
 

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Fri, Jul 29, 2016 at 09:13:58PM +0100, David Howells wrote:
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> 
> > I do this too, but I also work with the diff when moving hunks.  What I
> > do is:
> > 
> > git checkout <removal commit>
> > git show <removal commit> > tmp.diff
> > vi tmp.diff so it only has the hunks I want to remove
> > patch -p1 -R < ~/tmp.diff
> > git commit --amend -a
> > git checkout <add commit>
> > patch -p1 < ~/tmp.diff
> > git commit --amend -a
> > 
> > It's a bit cumbersome, but you can script it.  If there are better ways
> > to do it, I'm interested.
> 
> One of the reasons I use stgit is when it comes to excising changes into a
> separate patch, it makes it a lot easier:
> 
> 	 stg show >tmp.diff	# Export contents of current patch
> 	 emacs tmp.diff		# Edit so it only has the hunks I want to remove
> 	 stg pop		# Pop the patch I want to remove part of
> 	 stg new foo		# Create a new patch
> 	 stg fold tmp.diff	# ... and apply the patch created above
> 	 emacs ...		# Fix up any bits
> 	 stg refresh		# ... and make a commit out of it
> 	 stg push		# Push the patch I wanted to remove from

Typically, I'd do the same thing by using `git rebase -i` (or `git
series rebase -i`), marking the patch I want to split as 'e' for edit to
stop with that patch on top of the stack, resetting it, and using add -p
and committing bits incrementally into multiple commits.  (I can grab
bits of the commit message from ORIG_HEAD as needed.)

- Josh Triplett

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Arnd Bergmann]

On Friday, July 29, 2016 10:02:42 PM CEST Josh Triplett wrote:
> On Fri, Jul 29, 2016 at 09:13:58PM +0100, David Howells wrote:
> > James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> > 
> > > I do this too, but I also work with the diff when moving hunks.  What I
> > > do is:
> > > 
> > > git checkout <removal commit>
> > > git show <removal commit> > tmp.diff
> > > vi tmp.diff so it only has the hunks I want to remove
> > > patch -p1 -R < ~/tmp.diff
> > > git commit --amend -a
> > > git checkout <add commit>
> > > patch -p1 < ~/tmp.diff
> > > git commit --amend -a
> > > 
> > > It's a bit cumbersome, but you can script it.  If there are better ways
> > > to do it, I'm interested.
> > 
> > One of the reasons I use stgit is when it comes to excising changes into a
> > separate patch, it makes it a lot easier:
> > 
> >        stg show >tmp.diff     # Export contents of current patch
> >        emacs tmp.diff         # Edit so it only has the hunks I want to remove
> >        stg pop                # Pop the patch I want to remove part of
> >        stg new foo            # Create a new patch
> >        stg fold tmp.diff      # ... and apply the patch created above
> >        emacs ...              # Fix up any bits
> >        stg refresh            # ... and make a commit out of it
> >        stg push               # Push the patch I wanted to remove from
> 
> Typically, I'd do the same thing by using `git rebase -i` (or `git
> series rebase -i`), marking the patch I want to split as 'e' for edit to
> stop with that patch on top of the stack, resetting it, and using add -p
> and committing bits incrementally into multiple commits.  (I can grab
> bits of the commit message from ORIG_HEAD as needed.)

I tend to use a variation of this where I save the HEAD of the existing
branch in a temporary branch, then rebase as you do to undo parts of a patch
and finally merge+rebase the original branch on top of it to get the
remaining pieces as patches on top.

Also, saving the previous HEAD means you can easily do 'git diff' of the
old and new branch to see if the resulting commit is identical or you
forgot something.

	Arnd

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Jani Nikula]

On Sat, 30 Jul 2016, Josh Triplett <josh@joshtriplett.org> wrote:
> On Fri, Jul 29, 2016 at 09:13:58PM +0100, David Howells wrote:
>> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
>> 
>> > I do this too, but I also work with the diff when moving hunks.  What I
>> > do is:
>> > 
>> > git checkout <removal commit>
>> > git show <removal commit> > tmp.diff
>> > vi tmp.diff so it only has the hunks I want to remove
>> > patch -p1 -R < ~/tmp.diff
>> > git commit --amend -a
>> > git checkout <add commit>
>> > patch -p1 < ~/tmp.diff
>> > git commit --amend -a
>> > 
>> > It's a bit cumbersome, but you can script it.  If there are better ways
>> > to do it, I'm interested.
>> 
>> One of the reasons I use stgit is when it comes to excising changes into a
>> separate patch, it makes it a lot easier:
>> 
>> 	 stg show >tmp.diff	# Export contents of current patch
>> 	 emacs tmp.diff		# Edit so it only has the hunks I want to remove
>> 	 stg pop		# Pop the patch I want to remove part of
>> 	 stg new foo		# Create a new patch
>> 	 stg fold tmp.diff	# ... and apply the patch created above
>> 	 emacs ...		# Fix up any bits
>> 	 stg refresh		# ... and make a commit out of it
>> 	 stg push		# Push the patch I wanted to remove from
>
> Typically, I'd do the same thing by using `git rebase -i` (or `git
> series rebase -i`), marking the patch I want to split as 'e' for edit to
> stop with that patch on top of the stack, resetting it, and using add -p
> and committing bits incrementally into multiple commits.  (I can grab
> bits of the commit message from ORIG_HEAD as needed.)

Same here. Very powerful, and saves you from editing diffs
directly.

It's also sometimes useful to combine this with temporary reverts of
other commits for cleanly moving conflicting hunks earlier in the
series. It may conflict later anyway, but at least the reordering
doesn't have to conflict every step of the way.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Catalin Marinas]

On 29 Jul 2016, at 15:12, Josh Triplett <josh@joshtriplett.org> wrote:
> On Fri, Jul 29, 2016 at 01:20:12PM +0100, David Howells wrote:
>> Josh Triplett <josh@joshtriplett.org> wrote:
>>> I'd like to announce a project I've been working on for a while.  I sent
>>> this announcement to LKML, but since many people don't subscribe to LKML
>>> directly, and since ksummit-discuss has had several discussions
>>> specifically about patch workflow and development processes, I thought
>>> I'd send the announcement here as well, in case anyone found it useful
>>> for their workflow.
>> 
>> Can this be used as a direct substitute for stgit for maintaining a patch
>> series?
> 
> Yes, that's exactly what I designed it for.  git-series has the added
> advantage of tracking the versions of the patch series across rewrites.
> stgit just directly rewrites history, like rebase -i does; as far as I
> know, it doesn't remember the old history.  You'd have to go to the
> reflog for that.  

I haven't looked at git-series yet (I actually have a git "series" alias 
to list the current commits against a parent/tracking branch) 
but StGit does remember the series history. It stores all the past states 
of a series in a <branch>.stgit branch and you can inspect the 
changes, get unlimited undo/redo, even show a diff of diffs for 
a given patch. 

> Note that git-series doesn't provide a quilt-style push/pop workflow,
> with applied and unapplied patches; it just looks at HEAD.

Even though I'm the original author of StGit, I find myself using it less 
and less these days as I'm busier integrating others' patches than 
creating my own series from scratch. But what I miss though in plain git is the patch "pop" functionality. At 
some point I may add a 'git stash head' feature to git which 
would stash away the HEAD commit without losing its content (and 
the corresponding 'git stash apply' restoring the original commit). 

-- 
Catalin

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Michael S. Tsirkin]

On Fri, Aug 05, 2016 at 12:46:47AM +0200, Catalin Marinas wrote:
> On 29 Jul 2016, at 15:12, Josh Triplett <josh@joshtriplett.org> wrote:
> > On Fri, Jul 29, 2016 at 01:20:12PM +0100, David Howells wrote:
> >> Josh Triplett <josh@joshtriplett.org> wrote:
> >>> I'd like to announce a project I've been working on for a while.  I sent
> >>> this announcement to LKML, but since many people don't subscribe to LKML
> >>> directly, and since ksummit-discuss has had several discussions
> >>> specifically about patch workflow and development processes, I thought
> >>> I'd send the announcement here as well, in case anyone found it useful
> >>> for their workflow.
> >> 
> >> Can this be used as a direct substitute for stgit for maintaining a patch
> >> series?
> > 
> > Yes, that's exactly what I designed it for.  git-series has the added
> > advantage of tracking the versions of the patch series across rewrites.
> > stgit just directly rewrites history, like rebase -i does; as far as I
> > know, it doesn't remember the old history.  You'd have to go to the
> > reflog for that.  
> 
> I haven't looked at git-series yet (I actually have a git "series" alias 
> to list the current commits against a parent/tracking branch) 
> but StGit does remember the series history. It stores all the past states 
> of a series in a <branch>.stgit branch and you can inspect the 
> changes, get unlimited undo/redo, even show a diff of diffs for 
> a given patch. 
> 
> > Note that git-series doesn't provide a quilt-style push/pop workflow,
> > with applied and unapplied patches; it just looks at HEAD.
> 
> Even though I'm the original author of StGit, I find myself using it less 
> and less these days as I'm busier integrating others' patches than 
> creating my own series from scratch. But what I miss though in plain git is the patch "pop" functionality. At 
> some point I may add a 'git stash head' feature to git which 
> would stash away the HEAD commit without losing its content (and 
> the corresponding 'git stash apply' restoring the original commit). 

This will stash an arbitrary commit:
	#!/bin/sh

	commit="${1}";
	git tag -f stash ${commit}
	git rebase --onto ${commit}~1 ${commit}

and

 git cherry-pick stash

will unstash


> -- 
> Catalin
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Fri, Aug 05, 2016 at 12:46:47AM +0200, Catalin Marinas wrote:
> On 29 Jul 2016, at 15:12, Josh Triplett <josh@joshtriplett.org> wrote:
> > On Fri, Jul 29, 2016 at 01:20:12PM +0100, David Howells wrote:
> >> Josh Triplett <josh@joshtriplett.org> wrote:
> >>> I'd like to announce a project I've been working on for a while.  I sent
> >>> this announcement to LKML, but since many people don't subscribe to LKML
> >>> directly, and since ksummit-discuss has had several discussions
> >>> specifically about patch workflow and development processes, I thought
> >>> I'd send the announcement here as well, in case anyone found it useful
> >>> for their workflow.
> >> 
> >> Can this be used as a direct substitute for stgit for maintaining a patch
> >> series?
> > 
> > Yes, that's exactly what I designed it for.  git-series has the added
> > advantage of tracking the versions of the patch series across rewrites.
> > stgit just directly rewrites history, like rebase -i does; as far as I
> > know, it doesn't remember the old history.  You'd have to go to the
> > reflog for that.  
> 
> I haven't looked at git-series yet (I actually have a git "series" alias 
> to list the current commits against a parent/tracking branch) 
> but StGit does remember the series history. It stores all the past states 
> of a series in a <branch>.stgit branch and you can inspect the 
> changes, get unlimited undo/redo, even show a diff of diffs for 
> a given patch. 

Interesting!  I didn't find that when I last looked at stgit.  Does the
format used on that branch have documentation anywhere?  I didn't find
any in the stgit source or manpages.

- Josh Triplett

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Michael S. Tsirkin]

On Fri, Jul 29, 2016 at 12:50:39AM -0700, Josh Triplett wrote:
> I'd welcome any feedback, whether on the interface and workflow, the
> internals and collaboration, ideas on presenting diffs of patch series,
> or anything else.

Hi Josh!
A couple of ideas, I hope these will be helpful:

I often need to move series between unrelated trees.
I sometimes use cherry-pick for that (if they are from my tree)
or am (if not).

How about "git series am" - get patch series from email,
including a cover letter, and create a series from that?

And how about "git series cherry-pick" - to apply
series to the current head?

Kind of going in the reverse direction from git series format.

-- 
MST

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Fri, Aug 05, 2016 at 02:52:18AM +0300, Michael S. Tsirkin wrote:
> On Fri, Jul 29, 2016 at 12:50:39AM -0700, Josh Triplett wrote:
> > I'd welcome any feedback, whether on the interface and workflow, the
> > internals and collaboration, ideas on presenting diffs of patch series,
> > or anything else.
> 
> Hi Josh!
> A couple of ideas, I hope these will be helpful:
> 
> I often need to move series between unrelated trees.
> I sometimes use cherry-pick for that (if they are from my tree)
> or am (if not).
> 
> How about "git series am" - get patch series from email,
> including a cover letter, and create a series from that?

Would you expect to feed this one mbox containing all the mails, or a
set of files containing one patch each (including the 0000 cover
letter), or both?

> And how about "git series cherry-pick" - to apply
> series to the current head?

Could you elaborate on the semantic of this?  You have a series in your
tree, and you want to take all the patches in the series and apply them
to HEAD?  Or, do you want to copy the series and rebase the copy on HEAD?

For the former, once I add support for formatting a series other than
the current one, you could do that with "git series format seriesname
--stdout | git am".

> Kind of going in the reverse direction from git series format.

I like the idea; I'd like to offer more "import" mechanisms for any
other patch-series format people find useful.

- Josh Triplett

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Catalin Marinas]

On Fri, Aug 05, 2016 at 02:07:15AM +0300, Michael S. Tsirkin wrote:
> On Fri, Aug 05, 2016 at 12:46:47AM +0200, Catalin Marinas wrote:
> > On 29 Jul 2016, at 15:12, Josh Triplett <josh@joshtriplett.org> wrote:
> > > On Fri, Jul 29, 2016 at 01:20:12PM +0100, David Howells wrote:
> > >> Josh Triplett <josh@joshtriplett.org> wrote:
> > >>> I'd like to announce a project I've been working on for a while.  I sent
> > >>> this announcement to LKML, but since many people don't subscribe to LKML
> > >>> directly, and since ksummit-discuss has had several discussions
> > >>> specifically about patch workflow and development processes, I thought
> > >>> I'd send the announcement here as well, in case anyone found it useful
> > >>> for their workflow.
> > >> 
> > >> Can this be used as a direct substitute for stgit for maintaining a patch
> > >> series?
> > > 
> > > Yes, that's exactly what I designed it for.  git-series has the added
> > > advantage of tracking the versions of the patch series across rewrites.
> > > stgit just directly rewrites history, like rebase -i does; as far as I
> > > know, it doesn't remember the old history.  You'd have to go to the
> > > reflog for that.  
> > 
> > I haven't looked at git-series yet (I actually have a git "series" alias 
> > to list the current commits against a parent/tracking branch) 
> > but StGit does remember the series history. It stores all the past states 
> > of a series in a <branch>.stgit branch and you can inspect the 
> > changes, get unlimited undo/redo, even show a diff of diffs for 
> > a given patch. 
> > 
> > > Note that git-series doesn't provide a quilt-style push/pop workflow,
> > > with applied and unapplied patches; it just looks at HEAD.
> > 
> > Even though I'm the original author of StGit, I find myself using it less 
> > and less these days as I'm busier integrating others' patches than 
> > creating my own series from scratch. But what I miss though in plain git is the patch "pop" functionality. At 
> > some point I may add a 'git stash head' feature to git which 
> > would stash away the HEAD commit without losing its content (and 
> > the corresponding 'git stash apply' restoring the original commit). 
> 
> This will stash an arbitrary commit:
> 	#!/bin/sh
> 
> 	commit="${1}";
> 	git tag -f stash ${commit}
> 	git rebase --onto ${commit}~1 ${commit}
> 
> and
> 
>  git cherry-pick stash
> 
> will unstash

Thanks for the tip. All it needs is some refining to be able to stash
multiple commits (linking into a branch similar to the 'stash' would do;
I had a plan to do something similar for stgit but we ended up with a
more metadata).

-- 
Catalin

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Catalin Marinas]

On Thu, Aug 04, 2016 at 01:46:23PM -1000, Josh Triplett wrote:
> On Fri, Aug 05, 2016 at 12:46:47AM +0200, Catalin Marinas wrote:
> > On 29 Jul 2016, at 15:12, Josh Triplett <josh@joshtriplett.org> wrote:
> > > On Fri, Jul 29, 2016 at 01:20:12PM +0100, David Howells wrote:
> > >> Josh Triplett <josh@joshtriplett.org> wrote:
> > >>> I'd like to announce a project I've been working on for a while.  I sent
> > >>> this announcement to LKML, but since many people don't subscribe to LKML
> > >>> directly, and since ksummit-discuss has had several discussions
> > >>> specifically about patch workflow and development processes, I thought
> > >>> I'd send the announcement here as well, in case anyone found it useful
> > >>> for their workflow.
> > >> 
> > >> Can this be used as a direct substitute for stgit for maintaining a patch
> > >> series?
> > > 
> > > Yes, that's exactly what I designed it for.  git-series has the added
> > > advantage of tracking the versions of the patch series across rewrites.
> > > stgit just directly rewrites history, like rebase -i does; as far as I
> > > know, it doesn't remember the old history.  You'd have to go to the
> > > reflog for that.  
> > 
> > I haven't looked at git-series yet (I actually have a git "series" alias 
> > to list the current commits against a parent/tracking branch) 
> > but StGit does remember the series history. It stores all the past states 
> > of a series in a <branch>.stgit branch and you can inspect the 
> > changes, get unlimited undo/redo, even show a diff of diffs for 
> > a given patch. 
> 
> Interesting!  I didn't find that when I last looked at stgit.  Does the
> format used on that branch have documentation anywhere?  I didn't find
> any in the stgit source or manpages.

It is documented in stgit/lib/log.py (or if you generate the docs from
the source files). It is relatively simple, a 'meta' file with details
about the applied/unapplied/hidden patches with the name and commit id
and a patches directory with the actual diffs. The latter are not needed
for patch operations like reordering since git only needs a commit id
but you can use it to easily do a diff of diffs.

BTW, the stgit metadata is normally meant to be accessed via stgit
commands but you can always checkout <branch>.stgit. Long time ago I had
a plan to make stgit more useful for collaborating between multiple
people working on a set of patches but I can't find enough spare time
these days.

-- 
Catalin

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Paul E. McKenney]

On Fri, Jul 29, 2016 at 08:05:22AM -0700, Josh Triplett wrote:
> On Fri, Jul 29, 2016 at 07:55:44AM -0700, James Bottomley wrote:
> > On Fri, 2016-07-29 at 00:50 -0700, Josh Triplett wrote:
> > > I'd welcome any feedback, whether on the interface and workflow, the
> > > internals and collaboration, ideas on presenting diffs of patch 
> > > series, or anything else.
> > 
> > I've got an offer for you:  The Plumbers PC has been kicking around the
> > idea of a git MC at Plumbers.  We got as far ask asking Junio Hamano,
> > who said he'd commit to coming, but didn't want to run the MC.  Would
> > you be interested in running it?  Since it's a barely formed idea at
> > the moment, it would still have to go through the usual Plumbers
> > process of proposing it on
> > 
> > http://wiki.linuxplumbersconf.org/2016:topics
> > 
> > And then getting formal plumbers PC acceptance (Paul McKenney shepherds
> > this).  You can put me down as a committed attendee because, like a lot
> > of people, I have quite a bit of scripting around my git infrastructure
> > which, I suspect, could be shared.
> 
> I'd mentioned the idea of a git MC to Paul, and I'd be happy to give a
> git-series presentation/BoF at one, but I don't have the bandwidth to
> run one.

Which is exactly how the MC proposal came to be.  ;-)

Kate Stewart took over the shepherding and has found a runner, so here is
hoping that this will actually happen!

							Thanx, Paul

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 1983 bytes --]

Hi Josh,

On Fri, Jul 29, 2016 at 12:50:39AM -0700, Josh Triplett wrote:
> I'd welcome any feedback, whether on the interface and workflow, the
> internals and collaboration, ideas on presenting diffs of patch series,
> or anything else.

I often have patch series which are dependent on one another. It is then
very useful to be able to see all the branch names in logs (i.e. with
--decorate or using tig). This isn't possible out of the box with
git-series however since it seems to prefer to work on a detached head.

I sort of worked around this with a hacky script (see below) to update
my branches (which start with e.g. "4.8/") to point to the corresponding
git series latest commit.

Do you think it'd be practical and make sense for git-series to learn to
(optionally?) automatically remain on a particular branch (in my case
with the same name as the series) and keep it updated with rebases etc?
I realise the semantics of how it might work may be a little unclear at
the moment since the SHEAD doesn't get updated until git series commit.

Cheers
James

#!/bin/bash

V=4.8
if [ -n "$1" ]; then
	V=$(echo "$1" | sed 's/^v//')
fi

set -e

branches=$(git branch | grep -v 'detached from' | sed 's/^. //g')
for b in $(git series | grep '^. '$V'\/.*$' | sed 's/^. //g'); do
	if echo "$branches" | fgrep -qx "$b"; then
		brev=$(git rev-parse "$b")
		ref=$(echo $b | sed 's/^\(.*\)$/git-series\/\1^2/g')
		refrev=$(git rev-parse --verify --quiet "$ref" 2>&1 || true)
		if [ -z "$refrev" ]; then
			ref=$(echo $b | sed 's/^\(.*\)$/git-series\/\1^/g')
			refrev=$(git rev-parse --verify --quiet "$ref" 2>&1 || true)
			if [ -z "$refrev" ]; then
				continue
			fi
		fi
		if [ "$brev" != "$refrev" ]; then
			echo "Moving $b $brev...$refrev ($ref)"
			echo "  git branch -D \"$b\""
			echo "  git branch \"$b\" \"$ref\""
			echo -n "Go ahead [Y|n]? "
			read yes
			if [ "$yes" == "y" ]; then
				git branch -D "$b"
				git branch "$b" "$ref"
				echo "  done"
			fi
		fi
	fi
done

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 1299 bytes --]

On Fri, Aug 05, 2016 at 10:26:15AM -1000, Josh Triplett wrote:
> On Fri, Aug 05, 2016 at 02:52:18AM +0300, Michael S. Tsirkin wrote:
> > And how about "git series cherry-pick" - to apply
> > series to the current head?
> 
> Could you elaborate on the semantic of this?  You have a series in your
> tree, and you want to take all the patches in the series and apply them
> to HEAD?  Or, do you want to copy the series and rebase the copy on HEAD?
> 
> For the former, once I add support for formatting a series other than
> the current one, you could do that with "git series format seriesname
> --stdout | git am".

FWIW I had to do the latter the other day to check which stable branches
needed distinct backports. Since git series doesn't seem to allow
"forking" of a series yes (i.e. analagous to git checkout -b b1 b2), i
ended up doing something along the lines of this repeatedly:

git branch git-series/$feature/stable/4.7 git-series/$feature/staging
git series checkout git-series/$feature/stable/4.7
git series rebase v4.7
etc

(so the new series inherits / branches the history of the old one)

So it'd be nice to be able to do something like this instead:
git series checkout -b new_series [old_series = current_series]
git series rebase v4.7

Cheers
James

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Mon, Aug 15, 2016 at 02:20:04PM +0100, James Hogan wrote:
> On Fri, Aug 05, 2016 at 10:26:15AM -1000, Josh Triplett wrote:
> > On Fri, Aug 05, 2016 at 02:52:18AM +0300, Michael S. Tsirkin wrote:
> > > And how about "git series cherry-pick" - to apply
> > > series to the current head?
> > 
> > Could you elaborate on the semantic of this?  You have a series in your
> > tree, and you want to take all the patches in the series and apply them
> > to HEAD?  Or, do you want to copy the series and rebase the copy on HEAD?
> > 
> > For the former, once I add support for formatting a series other than
> > the current one, you could do that with "git series format seriesname
> > --stdout | git am".
> 
> FWIW I had to do the latter the other day to check which stable branches
> needed distinct backports. Since git series doesn't seem to allow
> "forking" of a series yes (i.e. analagous to git checkout -b b1 b2), i
> ended up doing something along the lines of this repeatedly:
> 
> git branch git-series/$feature/stable/4.7 git-series/$feature/staging
> git series checkout git-series/$feature/stable/4.7
> git series rebase v4.7
> etc
> 
> (so the new series inherits / branches the history of the old one)
> 
> So it'd be nice to be able to do something like this instead:
> git series checkout -b new_series [old_series = current_series]
> git series rebase v4.7

That seems completely reasonable; thanks for the suggestion!

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Mon, Aug 15, 2016 at 01:53:09PM +0100, James Hogan wrote:
> Hi Josh,
> 
> On Fri, Jul 29, 2016 at 12:50:39AM -0700, Josh Triplett wrote:
> > I'd welcome any feedback, whether on the interface and workflow, the
> > internals and collaboration, ideas on presenting diffs of patch series,
> > or anything else.
> 
> I often have patch series which are dependent on one another. It is then
> very useful to be able to see all the branch names in logs (i.e. with
> --decorate or using tig). This isn't possible out of the box with
> git-series however since it seems to prefer to work on a detached head.
C> 
> I sort of worked around this with a hacky script (see below) to update
> my branches (which start with e.g. "4.8/") to point to the corresponding
> git series latest commit.
> 
> Do you think it'd be practical and make sense for git-series to learn to
> (optionally?) automatically remain on a particular branch (in my case
> with the same name as the series) and keep it updated with rebases etc?
> I realise the semantics of how it might work may be a little unclear at
> the moment since the SHEAD doesn't get updated until git series commit.

That sounds entirely reasonable.  I had git-series use a detached HEAD
to avoid the potentially surprising behavior of continuing to update the
branch you started the series from (or a branch you re-attached HEAD
to).  However, intentionally specifying a branch to keep updated with
the current state of the series seems fine, and useful for exactly the
reason you mentioned.

I'm tempted to introduce a default branch name for this, involving the
series name and "current", which would make the --decorate use case work
out-of-the-box.  Naming suggestions welcome.

Given such a default, would you still want to specify a specific branch
to keep updated, or would the default branch name suffice?

Either way, I don't think I'd store the branch name within the series
itself (as it represents entirely local metadata that shouldn't get
transmitted with the series).  Easy enough to store local metadata
separately.

Also, your mention of dependent patch series makes me tempted to try to
add some kind of dependency mechanism between series, to make it easier
to notice when you need to rebase a family of series, and to work out
what patches you need to submit together.  (That would also allow using
the new format-patch metadata for "prerequisite-patch-id".)  I don't
want to get as complex as topgit, but I'd welcome suggestions for how
this should work, since you have a workflow that motivates it.

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 5320 bytes --]

On Mon, Aug 15, 2016 at 09:34:43AM -0700, Josh Triplett wrote:
> On Mon, Aug 15, 2016 at 01:53:09PM +0100, James Hogan wrote:
> > Hi Josh,
> > 
> > On Fri, Jul 29, 2016 at 12:50:39AM -0700, Josh Triplett wrote:
> > > I'd welcome any feedback, whether on the interface and workflow, the
> > > internals and collaboration, ideas on presenting diffs of patch series,
> > > or anything else.
> > 
> > I often have patch series which are dependent on one another. It is then
> > very useful to be able to see all the branch names in logs (i.e. with
> > --decorate or using tig). This isn't possible out of the box with
> > git-series however since it seems to prefer to work on a detached head.
> C> 
> > I sort of worked around this with a hacky script (see below) to update
> > my branches (which start with e.g. "4.8/") to point to the corresponding
> > git series latest commit.
> > 
> > Do you think it'd be practical and make sense for git-series to learn to
> > (optionally?) automatically remain on a particular branch (in my case
> > with the same name as the series) and keep it updated with rebases etc?
> > I realise the semantics of how it might work may be a little unclear at
> > the moment since the SHEAD doesn't get updated until git series commit.
> 
> That sounds entirely reasonable.  I had git-series use a detached HEAD
> to avoid the potentially surprising behavior of continuing to update the
> branch you started the series from (or a branch you re-attached HEAD
> to).  However, intentionally specifying a branch to keep updated with
> the current state of the series seems fine, and useful for exactly the
> reason you mentioned.
> 
> I'm tempted to introduce a default branch name for this, involving the
> series name and "current", which would make the --decorate use case work
> out-of-the-box.  Naming suggestions welcome.
> 
> Given such a default, would you still want to specify a specific branch
> to keep updated, or would the default branch name suffice?

Personally I'd want it to be able to match the name of the series as
closely as possible (handy to be able to copy branch name and paste into
"git series checkout" command).

> 
> Either way, I don't think I'd store the branch name within the series
> itself (as it represents entirely local metadata that shouldn't get
> transmitted with the series). Easy enough to store local metadata
> separately.
> 
> Also, your mention of dependent patch series makes me tempted to try to
> add some kind of dependency mechanism between series, to make it easier
> to notice when you need to rebase a family of series, and to work out
> what patches you need to submit together.  (That would also allow using
> the new format-patch metadata for "prerequisite-patch-id".)  I don't
> want to get as complex as topgit, but I'd welcome suggestions for how
> this should work, since you have a workflow that motivates it.

I haven't tried topgit tbh (I probably should).

The metadata for my current workflow would be a set of other series that
each series is dependent on (usually 1, but sometmes more if there are
series for different subsystems, which need merging together before the
base of the new series).

If I update one of the earlier series, I'd normally just rebase all the
others on top one by one (git rebase -p --onto HEAD <base> <branch>). It
gets a bit repetative, but with tig alongside showing the graph with
commit ids, and -p to preserve merges when necessary, and diffing to
sanity check changes, its doable.

git-series could make that easier as I could just "git series rebase
otherbranch" without having to check the commit id for the base, asside
from when it contains merges of course.

So I suppose it'd be nice to be able to do something roughly like:

$ git series create kvm/a/main v4.8-rc2
...
$ git series create mips/a/main v4.8-rc2
...
$ git series create kvm/b/main kvm/a/main
(Implicitly depends on "kvm/a/main" branch / series)
...
$ git series depend add mips/a/main
(Adds [sequence of] distinct merges at the beginning of the series)
...
$ git series create kvm/c/main kvm/b/main
...
$ git series checkout mips/a/main
... hack a bit on that branch
$ git series update
 It'd probably be necessary to analyse the graph of dependencies to
 figure out the order, and for each series regenerate the merges and
 rebase on top of them:
   checkout dependency 1
   merge dependency 2
   ...
   rebase --onto HEAD <last merge at beginning of current series> series

it'd probably be convenient to be able to autocommit each rebased
series too, which I suppose raises the question of conflicts, and how
hard it'd be to have --abort-all, --abort, & --continue options.

git series rebase -i should obviously go back to the last merge after
the bases, since you can't meaningfully rebase -i merges.

git series rebase onto... perhaps that should require a dependent branch
or series that is being replaced (previously implicitly the current
base), and I suppose require regenerating the merges too, to avoid
storing more metadata.

Sounds like it'd certainly need a fair bit of complexity to do that
though, although if number of dependencies was limited to 1 it could be
a lot simpler.

Cheers
James

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Mon, Aug 15, 2016 at 07:46:47PM +0100, James Hogan wrote:
> On Mon, Aug 15, 2016 at 09:34:43AM -0700, Josh Triplett wrote:
> > On Mon, Aug 15, 2016 at 01:53:09PM +0100, James Hogan wrote:
> > > Hi Josh,
> > > 
> > > On Fri, Jul 29, 2016 at 12:50:39AM -0700, Josh Triplett wrote:
> > > > I'd welcome any feedback, whether on the interface and workflow, the
> > > > internals and collaboration, ideas on presenting diffs of patch series,
> > > > or anything else.
> > > 
> > > I often have patch series which are dependent on one another. It is then
> > > very useful to be able to see all the branch names in logs (i.e. with
> > > --decorate or using tig). This isn't possible out of the box with
> > > git-series however since it seems to prefer to work on a detached head.
> > C> 
> > > I sort of worked around this with a hacky script (see below) to update
> > > my branches (which start with e.g. "4.8/") to point to the corresponding
> > > git series latest commit.
> > > 
> > > Do you think it'd be practical and make sense for git-series to learn to
> > > (optionally?) automatically remain on a particular branch (in my case
> > > with the same name as the series) and keep it updated with rebases etc?
> > > I realise the semantics of how it might work may be a little unclear at
> > > the moment since the SHEAD doesn't get updated until git series commit.
> > 
> > That sounds entirely reasonable.  I had git-series use a detached HEAD
> > to avoid the potentially surprising behavior of continuing to update the
> > branch you started the series from (or a branch you re-attached HEAD
> > to).  However, intentionally specifying a branch to keep updated with
> > the current state of the series seems fine, and useful for exactly the
> > reason you mentioned.
> > 
> > I'm tempted to introduce a default branch name for this, involving the
> > series name and "current", which would make the --decorate use case work
> > out-of-the-box.  Naming suggestions welcome.
> > 
> > Given such a default, would you still want to specify a specific branch
> > to keep updated, or would the default branch name suffice?
> 
> Personally I'd want it to be able to match the name of the series as
> closely as possible (handy to be able to copy branch name and paste into
> "git series checkout" command).

That makes sense; adding a suffix like "/current" would make that less
trivial.

I could provide a branch named the same as the series, if one doesn't
already exist.  I'd need to think about the implications of doing that
by *default*, though.  But an option to create and use that branch,
defaulting to the name of the series (without the "git-series/" prefix),
seems fine.

> > Either way, I don't think I'd store the branch name within the series
> > itself (as it represents entirely local metadata that shouldn't get
> > transmitted with the series). Easy enough to store local metadata
> > separately.
> > 
> > Also, your mention of dependent patch series makes me tempted to try to
> > add some kind of dependency mechanism between series, to make it easier
> > to notice when you need to rebase a family of series, and to work out
> > what patches you need to submit together.  (That would also allow using
> > the new format-patch metadata for "prerequisite-patch-id".)  I don't
> > want to get as complex as topgit, but I'd welcome suggestions for how
> > this should work, since you have a workflow that motivates it.
> 
> I haven't tried topgit tbh (I probably should).

I tried it briefly, but it had such a fundamentally *different* workflow
from git itself that I didn't really find it sufficiently worthwhile to
stick with.

> The metadata for my current workflow would be a set of other series that
> each series is dependent on (usually 1, but sometmes more if there are
> series for different subsystems, which need merging together before the
> base of the new series).
> 
> If I update one of the earlier series, I'd normally just rebase all the
> others on top one by one (git rebase -p --onto HEAD <base> <branch>). It
> gets a bit repetative, but with tig alongside showing the graph with
> commit ids, and -p to preserve merges when necessary, and diffing to
> sanity check changes, its doable.
> 
> git-series could make that easier as I could just "git series rebase
> otherbranch" without having to check the commit id for the base, asside
> from when it contains merges of course.

Would it help to suport "git series rebase seriesname" to rebase on top
of a series?

Actually you can do that today with "git series rebase
git-series/seriesname:series", thanks to git's "extended" syntax.
Still, probably a good idea to support a shorter syntax like "git series
rebase seriesname" as well.

> So I suppose it'd be nice to be able to do something roughly like:
> 
> $ git series create kvm/a/main v4.8-rc2

I do plan to add a second argument to start to provide a base.  Sounds
like I should also consider providing an alias "create" for "start". :)

> ...
> $ git series create mips/a/main v4.8-rc2
> ...
> $ git series create kvm/b/main kvm/a/main
> (Implicitly depends on "kvm/a/main" branch / series)
> ...
> $ git series depend add mips/a/main
> (Adds [sequence of] distinct merges at the beginning of the series)
> ...
> $ git series create kvm/c/main kvm/b/main
> ...
> $ git series checkout mips/a/main
> ... hack a bit on that branch
> $ git series update
>  It'd probably be necessary to analyse the graph of dependencies to
>  figure out the order, and for each series regenerate the merges and
>  rebase on top of them:
>    checkout dependency 1
>    merge dependency 2
>    ...
>    rebase --onto HEAD <last merge at beginning of current series> series
> 
> it'd probably be convenient to be able to autocommit each rebased
> series too, which I suppose raises the question of conflicts, and how
> hard it'd be to have --abort-all, --abort, & --continue options.
> 
> git series rebase -i should obviously go back to the last merge after
> the bases, since you can't meaningfully rebase -i merges.
> 
> git series rebase onto... perhaps that should require a dependent branch
> or series that is being replaced (previously implicitly the current
> base), and I suppose require regenerating the merges too, to avoid
> storing more metadata.
> 
> Sounds like it'd certainly need a fair bit of complexity to do that
> though, although if number of dependencies was limited to 1 it could be
> a lot simpler.

Yeah, I could imagine several possible workflows here, but it would
definitely increase complexity quite a bit.

If it would help people with various interdependent maintainer trees,
I'd definitely consider it, especially if the complexity remains limited
to people who actually declare series dependencies.

As an alternative to doing all of that completely automatically, I could
imagine tracking the dependencies similar to how git tracks upstream
"tracking" branches, and then providing guided next steps but still
requiring you to rebase the series individually.  For instance, if
you have a series 4.7/base, and then another series 4.7/kvm that depends
on 4.7/base, "git series status" on 4.7/kvm could notice if you've made
changes in 4.7/base since the version you based 4.7/kvm on, like this:

$ git series status
On series 4.7/kvm
Base series 4.7/base updated (rebased N commits ahead)
  (use "git series rebase 4.7/base" to update)

And conversely, "git series status" on 4.7/base could say:

$ git series status
On series 4.7/base
Dependent series 4.7/kvm (and N more) needs update
  ("git series checkout 4.7/kvm" then "git rebase 4.7/base" to update)

Would that help simplify the process, to avoid having to carefully
orchestrate it while watching a repository browser?

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[James Hogan]

[-- Attachment #1: Type: text/plain, Size: 5317 bytes --]

On Mon, Aug 15, 2016 at 02:35:37PM -0700, Josh Triplett wrote:
> On Mon, Aug 15, 2016 at 07:46:47PM +0100, James Hogan wrote:
> > The metadata for my current workflow would be a set of other series that
> > each series is dependent on (usually 1, but sometmes more if there are
> > series for different subsystems, which need merging together before the
> > base of the new series).
> > 
> > If I update one of the earlier series, I'd normally just rebase all the
> > others on top one by one (git rebase -p --onto HEAD <base> <branch>). It
> > gets a bit repetative, but with tig alongside showing the graph with
> > commit ids, and -p to preserve merges when necessary, and diffing to
> > sanity check changes, its doable.
> > 
> > git-series could make that easier as I could just "git series rebase
> > otherbranch" without having to check the commit id for the base, asside
> > from when it contains merges of course.
> 
> Would it help to suport "git series rebase seriesname" to rebase on top
> of a series?
> 
> Actually you can do that today with "git series rebase
> git-series/seriesname:series", thanks to git's "extended" syntax.
> Still, probably a good idea to support a shorter syntax like "git series
> rebase seriesname" as well.

Right, I've already done it using a branch with the same name as the
series, so unless the default branch name is different I guess it
wouldn't help much.

> 
> > So I suppose it'd be nice to be able to do something roughly like:
> > 
> > $ git series create kvm/a/main v4.8-rc2
> 
> I do plan to add a second argument to start to provide a base.  Sounds
> like I should also consider providing an alias "create" for "start". :)

Yep, I meant start :-).

> 
> > ...
> > $ git series create mips/a/main v4.8-rc2
> > ...
> > $ git series create kvm/b/main kvm/a/main
> > (Implicitly depends on "kvm/a/main" branch / series)
> > ...
> > $ git series depend add mips/a/main
> > (Adds [sequence of] distinct merges at the beginning of the series)
> > ...
> > $ git series create kvm/c/main kvm/b/main
> > ...
> > $ git series checkout mips/a/main
> > ... hack a bit on that branch
> > $ git series update
> >  It'd probably be necessary to analyse the graph of dependencies to
> >  figure out the order, and for each series regenerate the merges and
> >  rebase on top of them:
> >    checkout dependency 1
> >    merge dependency 2
> >    ...
> >    rebase --onto HEAD <last merge at beginning of current series> series
> > 
> > it'd probably be convenient to be able to autocommit each rebased
> > series too, which I suppose raises the question of conflicts, and how
> > hard it'd be to have --abort-all, --abort, & --continue options.
> > 
> > git series rebase -i should obviously go back to the last merge after
> > the bases, since you can't meaningfully rebase -i merges.
> > 
> > git series rebase onto... perhaps that should require a dependent branch
> > or series that is being replaced (previously implicitly the current
> > base), and I suppose require regenerating the merges too, to avoid
> > storing more metadata.
> > 
> > Sounds like it'd certainly need a fair bit of complexity to do that
> > though, although if number of dependencies was limited to 1 it could be
> > a lot simpler.
> 
> Yeah, I could imagine several possible workflows here, but it would
> definitely increase complexity quite a bit.
> 
> If it would help people with various interdependent maintainer trees,
> I'd definitely consider it, especially if the complexity remains limited
> to people who actually declare series dependencies.
> 
> As an alternative to doing all of that completely automatically, I could
> imagine tracking the dependencies similar to how git tracks upstream
> "tracking" branches, and then providing guided next steps but still
> requiring you to rebase the series individually.  For instance, if
> you have a series 4.7/base, and then another series 4.7/kvm that depends
> on 4.7/base, "git series status" on 4.7/kvm could notice if you've made
> changes in 4.7/base since the version you based 4.7/kvm on, like this:
> 
> $ git series status
> On series 4.7/kvm
> Base series 4.7/base updated (rebased N commits ahead)
>   (use "git series rebase 4.7/base" to update)
> 
> And conversely, "git series status" on 4.7/base could say:
> 
> $ git series status
> On series 4.7/base
> Dependent series 4.7/kvm (and N more) needs update
>   ("git series checkout 4.7/kvm" then "git rebase 4.7/base" to update)
> 
> Would that help simplify the process, to avoid having to carefully
> orchestrate it while watching a repository browser?

I could see that being useful, although personally I'm usually quite
aware of the overall commit graph I'm dealing with, so it might be more
handy for when I forget that some other random WIP branch is based on
it.

I suppose though once you have git-series taking away the need to find
the base commit, its much simpler to script a sequence of rebases in the
right order, so the problem may just fade away. Even redundant rebases
should be harmless (although I just tried one and "Base unchanged" seems
to be treated as an error which necessitates a "git rebase --continue"
after it).

Cheers
James

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Michael S. Tsirkin]

On Fri, Aug 05, 2016 at 10:26:15AM -1000, Josh Triplett wrote:
> On Fri, Aug 05, 2016 at 02:52:18AM +0300, Michael S. Tsirkin wrote:
> > On Fri, Jul 29, 2016 at 12:50:39AM -0700, Josh Triplett wrote:
> > > I'd welcome any feedback, whether on the interface and workflow, the
> > > internals and collaboration, ideas on presenting diffs of patch series,
> > > or anything else.
> > 
> > Hi Josh!
> > A couple of ideas, I hope these will be helpful:
> > 
> > I often need to move series between unrelated trees.
> > I sometimes use cherry-pick for that (if they are from my tree)
> > or am (if not).
> > 
> > How about "git series am" - get patch series from email,
> > including a cover letter, and create a series from that?
> 
> Would you expect to feed this one mbox containing all the mails, or a
> set of files containing one patch each (including the 0000 cover
> letter), or both?

If piping from mutt, a set of files. If saving to an mbox
from mutt and piping from command line, one mbox.
So both.

> > And how about "git series cherry-pick" - to apply
> > series to the current head?
> 
> Could you elaborate on the semantic of this?  You have a series in your
> tree, and you want to take all the patches in the series and apply them
> to HEAD?  Or, do you want to copy the series and rebase the copy on HEAD?


What I had in mind is that I might have multiple patchsets under test
on an unstable branch X.
Now I decide to copy one patchset to the stable branch.

With a single patch I would just cherry-pick.




> For the former, once I add support for formatting a series other than
> the current one, you could do that with "git series format seriesname
> --stdout | git am".

Yes, except cherry-pick has the -x flag to track where did
I get the patch from. And I would like this info recorded
for the cover as well. Maybe format can export that info.
In fact this might be a handy flag for format-patch too:

ability to add (cherry-picked from commit XYZ "subject")

automatically.

> > Kind of going in the reverse direction from git series format.
> 
> I like the idea; I'd like to offer more "import" mechanisms for any
> other patch-series format people find useful.
> 
> - Josh Triplett

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Michael S. Tsirkin]

On Mon, Aug 08, 2016 at 06:27:11PM +0100, Catalin Marinas wrote:
> On Fri, Aug 05, 2016 at 02:07:15AM +0300, Michael S. Tsirkin wrote:
> > On Fri, Aug 05, 2016 at 12:46:47AM +0200, Catalin Marinas wrote:
> > > On 29 Jul 2016, at 15:12, Josh Triplett <josh@joshtriplett.org> wrote:
> > > > On Fri, Jul 29, 2016 at 01:20:12PM +0100, David Howells wrote:
> > > >> Josh Triplett <josh@joshtriplett.org> wrote:
> > > >>> I'd like to announce a project I've been working on for a while.  I sent
> > > >>> this announcement to LKML, but since many people don't subscribe to LKML
> > > >>> directly, and since ksummit-discuss has had several discussions
> > > >>> specifically about patch workflow and development processes, I thought
> > > >>> I'd send the announcement here as well, in case anyone found it useful
> > > >>> for their workflow.
> > > >> 
> > > >> Can this be used as a direct substitute for stgit for maintaining a patch
> > > >> series?
> > > > 
> > > > Yes, that's exactly what I designed it for.  git-series has the added
> > > > advantage of tracking the versions of the patch series across rewrites.
> > > > stgit just directly rewrites history, like rebase -i does; as far as I
> > > > know, it doesn't remember the old history.  You'd have to go to the
> > > > reflog for that.  
> > > 
> > > I haven't looked at git-series yet (I actually have a git "series" alias 
> > > to list the current commits against a parent/tracking branch) 
> > > but StGit does remember the series history. It stores all the past states 
> > > of a series in a <branch>.stgit branch and you can inspect the 
> > > changes, get unlimited undo/redo, even show a diff of diffs for 
> > > a given patch. 
> > > 
> > > > Note that git-series doesn't provide a quilt-style push/pop workflow,
> > > > with applied and unapplied patches; it just looks at HEAD.
> > > 
> > > Even though I'm the original author of StGit, I find myself using it less 
> > > and less these days as I'm busier integrating others' patches than 
> > > creating my own series from scratch. But what I miss though in plain git is the patch "pop" functionality. At 
> > > some point I may add a 'git stash head' feature to git which 
> > > would stash away the HEAD commit without losing its content (and 
> > > the corresponding 'git stash apply' restoring the original commit). 
> > 
> > This will stash an arbitrary commit:
> > 	#!/bin/sh
> > 
> > 	commit="${1}";
> > 	git tag -f stash ${commit}
> > 	git rebase --onto ${commit}~1 ${commit}
> > 
> > and
> > 
> >  git cherry-pick stash
> > 
> > will unstash
> 
> Thanks for the tip. All it needs is some refining to be able to stash
> multiple commits (linking into a branch similar to the 'stash' would do;
> I had a plan to do something similar for stgit but we ended up with a
> more metadata).

Yes - generally a new kind of object that is more like
a tag than a commit but can point at
a list of commits and not just a single one would
solve many problems like this.


> -- 
> Catalin

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Josh Triplett]

On Mon, Aug 15, 2016 at 11:06:47PM +0100, James Hogan wrote:
> On Mon, Aug 15, 2016 at 02:35:37PM -0700, Josh Triplett wrote:
> > On Mon, Aug 15, 2016 at 07:46:47PM +0100, James Hogan wrote:
> > > ...
> > > $ git series create mips/a/main v4.8-rc2
> > > ...
> > > $ git series create kvm/b/main kvm/a/main
> > > (Implicitly depends on "kvm/a/main" branch / series)
> > > ...
> > > $ git series depend add mips/a/main
> > > (Adds [sequence of] distinct merges at the beginning of the series)
> > > ...
> > > $ git series create kvm/c/main kvm/b/main
> > > ...
> > > $ git series checkout mips/a/main
> > > ... hack a bit on that branch
> > > $ git series update
> > >  It'd probably be necessary to analyse the graph of dependencies to
> > >  figure out the order, and for each series regenerate the merges and
> > >  rebase on top of them:
> > >    checkout dependency 1
> > >    merge dependency 2
> > >    ...
> > >    rebase --onto HEAD <last merge at beginning of current series> series
> > > 
> > > it'd probably be convenient to be able to autocommit each rebased
> > > series too, which I suppose raises the question of conflicts, and how
> > > hard it'd be to have --abort-all, --abort, & --continue options.
> > > 
> > > git series rebase -i should obviously go back to the last merge after
> > > the bases, since you can't meaningfully rebase -i merges.
> > > 
> > > git series rebase onto... perhaps that should require a dependent branch
> > > or series that is being replaced (previously implicitly the current
> > > base), and I suppose require regenerating the merges too, to avoid
> > > storing more metadata.
> > > 
> > > Sounds like it'd certainly need a fair bit of complexity to do that
> > > though, although if number of dependencies was limited to 1 it could be
> > > a lot simpler.
> > 
> > Yeah, I could imagine several possible workflows here, but it would
> > definitely increase complexity quite a bit.
> > 
> > If it would help people with various interdependent maintainer trees,
> > I'd definitely consider it, especially if the complexity remains limited
> > to people who actually declare series dependencies.
> > 
> > As an alternative to doing all of that completely automatically, I could
> > imagine tracking the dependencies similar to how git tracks upstream
> > "tracking" branches, and then providing guided next steps but still
> > requiring you to rebase the series individually.  For instance, if
> > you have a series 4.7/base, and then another series 4.7/kvm that depends
> > on 4.7/base, "git series status" on 4.7/kvm could notice if you've made
> > changes in 4.7/base since the version you based 4.7/kvm on, like this:
> > 
> > $ git series status
> > On series 4.7/kvm
> > Base series 4.7/base updated (rebased N commits ahead)
> >   (use "git series rebase 4.7/base" to update)
> > 
> > And conversely, "git series status" on 4.7/base could say:
> > 
> > $ git series status
> > On series 4.7/base
> > Dependent series 4.7/kvm (and N more) needs update
> >   ("git series checkout 4.7/kvm" then "git rebase 4.7/base" to update)
> > 
> > Would that help simplify the process, to avoid having to carefully
> > orchestrate it while watching a repository browser?
> 
> I could see that being useful, although personally I'm usually quite
> aware of the overall commit graph I'm dealing with, so it might be more
> handy for when I forget that some other random WIP branch is based on
> it.

Fair enough.  I might still implement this at some point just to make it
easier to carry less state in your head. :)

> I suppose though once you have git-series taking away the need to find
> the base commit, its much simpler to script a sequence of rebases in the
> right order, so the problem may just fade away.

I was quite pleased to see that you've already started scripting
git-series. :)

> Even redundant rebases
> should be harmless (although I just tried one and "Base unchanged" seems
> to be treated as an error which necessitates a "git rebase --continue"
> after it).

Oops.  Fixed in 0.8.9, thanks.  (I also changed rebase itself to not do
redundant rebases; if the base hasn't changed, a non-interactive rebase
has nothing to do.)

- Josh Triplett

Re: [Ksummit-discuss] [ANNOUNCE] git-series: track changes to a patch series over time

[Michael S. Tsirkin]

On Mon, Aug 15, 2016 at 04:59:21PM -0700, Josh Triplett wrote:
> (I also changed rebase itself to not do
> redundant rebases; if the base hasn't changed, a non-interactive rebase
> has nothing to do.)
> 
> - Josh Triplett

It's actually a minor bug in git rebase that --autosquash has
no effect unless -i is supplied. With --autosquash rebase
with an unchanged base could in theory do something useful
even without -i.

-- 
MST