RFC: group maintainership for misc drm trees

RFC: group maintainership for misc drm trees

[Daniel Vetter]

Hi all,

I've already chatted with some of you in private, here's the entire idea with a
bit more thought. My motiviation for group maintainership of drm-misc was that I
got a bit a guilty feeling the last few vacations/conferences when folks pinged
about reviewed/tested pretty patches not landing. But also just increasing the
bus factor and sharing the load better is good. And finally a shared misc drm
tree would allow fringe drivers to get faster into Dave's drm-next by
piggy-packing on top of the one pull request train. And it would also reduce a
bit tree proliferation (at one point we had drm-misc/-bridge/-panel/-trivial and
may more even). And at least everyone I chatted with seems to like the idea in
principle.

But what's still open is how to do it exactly. One big change with group
maintainership is that you can't rebase a tree anymore. And right now I need to
rebase drm-misc fairly often to throw out bad apples again. I think solving that
is the important bit to make a shared drm-misc work. A few ideas:

- I think CI is super-important. We're starting to finally roll that out for
  real for i915, and it's catching an awful lot of stuff already. Not yet ready
  for prime-time on public mailing lists, and for misc we probably can't test
  every patch before they land like we do for i915. But CI should have veto
  power before a pull request goes to Dave imo.

  For non-i915 Daniel Stone and others are working on ARM CI using the generic
  igt testcases for kms. And I'm open to merging driver-specific tests into igt
  too, if it makes sense, and e.g. Eric has already pushed some vc4 tests.

- Stuff needs to at least compile cleanly before pushing. I've been really bad
  at that wrt arm drivers with my own drm-misc, but turns out it's fairly easy
  to get this right: http://blog.ffwll.ch/2016/02/arm-kernel-cross-compiling.html

- Bad apples need to be kicked out with reverts, not rebases. I think that's
  fine for simple patches, and hence those can go directly to drm-misc. But a
  bunch of subsystem-wide refactorings go in through mis trees, and for those
  constantly mass-reverting until it's all solid is silly. And ime you need some
  soak time in a shared tree to iron out bugs with those kind of endeavours. We
  can address that with ad-hoc&short-lived topic branches which are then again
  owned by a single maintainer, but automatically pulled into an integration
  tree. After some soaking time to give CI systems time to crunch through those
  topic branches can then be merged into the main drm-misc and removed.

- Also we can't roll forward to latest drm-next easily with rebases any more. So
  that needs to happen either right after the pull request lands (when no patch
  has been merged meanwhile). Or with backmerges, which then need a short commit
  message as to way the backmerge is needed ("Backmerge because we need
  $feature" is what I usually type), plus sob line from the committer. And of
  course the backmerged treed must be stable/non-rebasing and preferrably the
  backmerge should be a release/pull-request tag.

- For tooling I suggest we just go with drm-intel maintainer-tools for a start.
  Picking dim has a few reasons:
  * Well tested, documented and fairly complete (includes e.g. bash-completion).
  * Integrated support for topic branches, including support for git worktree.
  * Well-exercised and robust integration tree support.
  Short-term we could add some convenience functions (e.g. for creating/merging
  drm-misc topic branches). Long-term we might or might not want to have this
  separated from drm-intel - that should be possible but a bit of work. Also,
  this way drm-misc would stay at it's current location while we figure things
  out. Longer term we might also want to look into adding other big drivers
  into an over drm integration.

Of course group maintainership needs an initial group. My experience from
drm-intel is that a bigger group of maintainer has benefits: It's clear that
part-time maintaining is ok too, with maintainers focusing on their area of
interest/expertise and only helping out in other places when there's a gap (due
to e.g. vacations). Anyway, here's my thoughts for the starting group:
Archit, Jani, Thierry & me as existing maintainers of drm-misc/bridge/panel,
Alex&Rob as maintainers of big drivers and engaged in core drm stuff,
Daniel Stone so that he has no more excuses to stall on arm drm ci.

I think if this goes well we can extend it to more driver maintainers, e.g.
Patrick would like to just push gma500 patches to some tree and not fiddle with
pull requests all the time himself.

Thoughts? Glaring ommisssions? Other topics we should discuss before we get
started? Should we do a MAINTAINERS change right away, or leave things as-is
until we're confident this will work and makes sense?

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: group maintainership for misc drm trees

[Eric Anholt]

[-- Attachment #1.1: Type: text/plain, Size: 1889 bytes --]

Daniel Vetter <daniel.vetter@ffwll.ch> writes:

> Hi all,
>
> I've already chatted with some of you in private, here's the entire idea with a
> bit more thought. My motiviation for group maintainership of drm-misc was that I
> got a bit a guilty feeling the last few vacations/conferences when folks pinged
> about reviewed/tested pretty patches not landing. But also just increasing the
> bus factor and sharing the load better is good. And finally a shared misc drm
> tree would allow fringe drivers to get faster into Dave's drm-next by
> piggy-packing on top of the one pull request train. And it would also reduce a
> bit tree proliferation (at one point we had drm-misc/-bridge/-panel/-trivial and
> may more even). And at least everyone I chatted with seems to like the idea in
> principle.

[...]

> Of course group maintainership needs an initial group. My experience from
> drm-intel is that a bigger group of maintainer has benefits: It's clear that
> part-time maintaining is ok too, with maintainers focusing on their area of
> interest/expertise and only helping out in other places when there's a gap (due
> to e.g. vacations). Anyway, here's my thoughts for the starting group:
> Archit, Jani, Thierry & me as existing maintainers of drm-misc/bridge/panel,
> Alex&Rob as maintainers of big drivers and engaged in core drm stuff,
> Daniel Stone so that he has no more excuses to stall on arm drm ci.
>
> I think if this goes well we can extend it to more driver maintainers, e.g.
> Patrick would like to just push gma500 patches to some tree and not fiddle with
> pull requests all the time himself.

I would love to be able to push vc4 changes to an "automatically" pull
requested tree with some amount of CI (at least of the
build-all-the-options variety like kbuild test robot does).  Pull
requests are one of the more error-prone and time-consuming parts of my
process.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: group maintainership for misc drm trees

[Dave Airlie]

On 15 February 2016 at 20:06, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> Hi all,
>
> I've already chatted with some of you in private, here's the entire idea with a
> bit more thought. My motiviation for group maintainership of drm-misc was that I
> got a bit a guilty feeling the last few vacations/conferences when folks pinged
> about reviewed/tested pretty patches not landing. But also just increasing the
> bus factor and sharing the load better is good. And finally a shared misc drm
> tree would allow fringe drivers to get faster into Dave's drm-next by
> piggy-packing on top of the one pull request train. And it would also reduce a
> bit tree proliferation (at one point we had drm-misc/-bridge/-panel/-trivial and
> may more even). And at least everyone I chatted with seems to like the idea in
> principle.
>
> But what's still open is how to do it exactly. One big change with group
> maintainership is that you can't rebase a tree anymore. And right now I need to
> rebase drm-misc fairly often to throw out bad apples again. I think solving that
> is the important bit to make a shared drm-misc work. A few ideas:

This is kinda what I don't like. I don't want a tree that can't rebase
out bad stuff
coming to me that often. If we are being too over eager in merging
stuff the answer
is to merge less not try and merge more. If we have a tree where things
are thrown until they stick I'm not sure the history will ever be nice
to pull from.

I'd rather consider a staging tree where everything from patchwork can
get thrown
into, CIed, but that we cherry-pick fixes things to go back to me from the what
works category.

I'm finding with i915 for example there is a massive latency in the pipeline now
waiting for fixes, and the pipeline to the end of drm-intel-next is
very long and
hard to figure out what fixes should be pulled back, and how much of the
driver has been rewritten between the -next and the -fixes pulls.

> - I think CI is super-important. We're starting to finally roll that out for
>   real for i915, and it's catching an awful lot of stuff already. Not yet ready
>   for prime-time on public mailing lists, and for misc we probably can't test
>   every patch before they land like we do for i915. But CI should have veto
>   power before a pull request goes to Dave imo.
>
>   For non-i915 Daniel Stone and others are working on ARM CI using the generic
>   igt testcases for kms. And I'm open to merging driver-specific tests into igt
>   too, if it makes sense, and e.g. Eric has already pushed some vc4 tests.
>
> - Stuff needs to at least compile cleanly before pushing. I've been really bad
>   at that wrt arm drivers with my own drm-misc, but turns out it's fairly easy
>   to get this right: http://blog.ffwll.ch/2016/02/arm-kernel-cross-compiling.html
>
> - Bad apples need to be kicked out with reverts, not rebases. I think that's
>   fine for simple patches, and hence those can go directly to drm-misc. But a
>   bunch of subsystem-wide refactorings go in through mis trees, and for those
>   constantly mass-reverting until it's all solid is silly. And ime you need some
>   soak time in a shared tree to iron out bugs with those kind of endeavours. We
>   can address that with ad-hoc&short-lived topic branches which are then again
>   owned by a single maintainer, but automatically pulled into an integration
>   tree. After some soaking time to give CI systems time to crunch through those
>   topic branches can then be merged into the main drm-misc and removed.

I think integration trees are probably the way forward, but I
understand they come
with a massive overhead for constant merging, I like the idea of topic
branches until
it becomes my turn to spaghetti merge 3-4 features at once. At that
point I still
fallback to the slow things down, pick one feature merge it, back the
others off,
and I think this is something we should do more off, the pile everything in and
hope that magical CI will stabilise things doesn't seem to be a responsible
process to me.

so I'd like CI to be happening but I'd like it to be happening before the git
history is baked into stone.

> - Also we can't roll forward to latest drm-next easily with rebases any more. So
>   that needs to happen either right after the pull request lands (when no patch
>   has been merged meanwhile). Or with backmerges, which then need a short commit
>   message as to way the backmerge is needed ("Backmerge because we need
>   $feature" is what I usually type), plus sob line from the committer. And of
>   course the backmerged treed must be stable/non-rebasing and preferrably the
>   backmerge should be a release/pull-request tag.
>
> - For tooling I suggest we just go with drm-intel maintainer-tools for a start.
>   Picking dim has a few reasons:
>   * Well tested, documented and fairly complete (includes e.g. bash-completion).
>   * Integrated support for topic branches, including support for git worktree.
>   * Well-exercised and robust integration tree support.
>   Short-term we could add some convenience functions (e.g. for creating/merging
>   drm-misc topic branches). Long-term we might or might not want to have this
>   separated from drm-intel - that should be possible but a bit of work. Also,
>   this way drm-misc would stay at it's current location while we figure things
>   out. Longer term we might also want to look into adding other big drivers
>   into an over drm integration.
>
> Of course group maintainership needs an initial group. My experience from
> drm-intel is that a bigger group of maintainer has benefits: It's clear that
> part-time maintaining is ok too, with maintainers focusing on their area of
> interest/expertise and only helping out in other places when there's a gap (due
> to e.g. vacations). Anyway, here's my thoughts for the starting group:
> Archit, Jani, Thierry & me as existing maintainers of drm-misc/bridge/panel,
> Alex&Rob as maintainers of big drivers and engaged in core drm stuff,
> Daniel Stone so that he has no more excuses to stall on arm drm ci.
>
> I think if this goes well we can extend it to more driver maintainers, e.g.
> Patrick would like to just push gma500 patches to some tree and not fiddle with
> pull requests all the time himself.

Is pull request sending a big issue? I don't really want a misc tree
as a way to avoid sending distinct pull requests for drivers. If we do
this I want
it to be drm core patches and subsequent fallout only and if we want
an unmaintained
drivers patches tree to do that separately.

I don't really like the misc tree replacing panel/bridge tree either,
those were nicely
compartmentalised in my head as trees I pulled but didn't really care
about as Thierry
did a good job and I don't have ARM hw that I personally care about.

It's important to me that stuff still comes in some sort of
compartments, as it decides
for each pull request how much time/effort I put into testing it on my
end, I'm pretty much
at the mercy of ARM hw maintainers, but with x86 I still like to smoke
test it on some
hardware as much as possible, also with DRM core stuff.

So I'm still not sure where I stand on this, personally I don't care
if you take holidays
and stuff stalls, delays aren't the big enemy here, merging crap is
much worse than
delaying.

Dave.
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: group maintainership for misc drm trees

[Daniel Vetter]

On Tue, Feb 16, 2016 at 2:44 AM, Dave Airlie <airlied@gmail.com> wrote:
> I'm finding with i915 for example there is a massive latency in the pipeline now
> waiting for fixes, and the pipeline to the end of drm-intel-next is
> very long and
> hard to figure out what fixes should be pulled back, and how much of the
> driver has been rewritten between the -next and the -fixes pulls.

I think this is a separate topic, so I'm making that into my own reply.

What's your target here, or what would you like to see changed? More
transparency in how -fixes flow to drm-fixes ultimately? For you, or
for the fix submittter?

We've discussed this already on irc in the context of Lyude's patches,
and those showed up on intel-gfx on 2nd/4th Feb and landed in your
inbox as a pull request on 14th, so about 10 days. And most of that
was because I was at LCA. Some intel-written fixes got delayed a bit
more because CI was down in-between, too. But now that we have CI
(even though just basic) I really don't want to ignore it, since the
past two times it was down an unrelated overlapping regression crept
into drm-intel-next. And that's just pain. Given that I think we're
definitely not doing things perfectly (Jani needs a reminder for
timely pull requests), but overall it seems to work ok. And lots more
patches seem to flow to -fixes I think.

Or do you have some other patches in mind?

Thanks, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: group maintainership for misc drm trees

[Jani Nikula]

On Tue, 16 Feb 2016, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> On Tue, Feb 16, 2016 at 2:44 AM, Dave Airlie <airlied@gmail.com> wrote:
>> I'm finding with i915 for example there is a massive latency in the pipeline now
>> waiting for fixes, and the pipeline to the end of drm-intel-next is
>> very long and
>> hard to figure out what fixes should be pulled back, and how much of the
>> driver has been rewritten between the -next and the -fixes pulls.
>
> I think this is a separate topic, so I'm making that into my own reply.
>
> What's your target here, or what would you like to see changed? More
> transparency in how -fixes flow to drm-fixes ultimately? For you, or
> for the fix submittter?
>
> We've discussed this already on irc in the context of Lyude's patches,
> and those showed up on intel-gfx on 2nd/4th Feb and landed in your
> inbox as a pull request on 14th, so about 10 days. And most of that
> was because I was at LCA. Some intel-written fixes got delayed a bit
> more because CI was down in-between, too. But now that we have CI
> (even though just basic) I really don't want to ignore it, since the
> past two times it was down an unrelated overlapping regression crept
> into drm-intel-next. And that's just pain. Given that I think we're
> definitely not doing things perfectly (Jani needs a reminder for
> timely pull requests), but overall it seems to work ok. And lots more
> patches seem to flow to -fixes I think.

I certainly have room for improvement on sending timely fixes pull
requests, no question.

Perhaps we did get some fixes through faster before. But we also dropped
more fixes on the floor and we had less confidence in them. Now they all
go through the CI and get applied to our next, meaning even the ones
that aren't cherry-picked for the fixes fast path will eventually make
it. At least I have more confidence in the fixes pulls I send now than
six months ago, and any latency introduced by waiting for CI is IMO
worth it.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: group maintainership for misc drm trees

[Daniel Vetter]

On Tue, Feb 16, 2016 at 11:44:52AM +1000, Dave Airlie wrote:
> On 15 February 2016 at 20:06, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> > Hi all,
> >
> > I've already chatted with some of you in private, here's the entire idea with a
> > bit more thought. My motiviation for group maintainership of drm-misc was that I
> > got a bit a guilty feeling the last few vacations/conferences when folks pinged
> > about reviewed/tested pretty patches not landing. But also just increasing the
> > bus factor and sharing the load better is good. And finally a shared misc drm
> > tree would allow fringe drivers to get faster into Dave's drm-next by
> > piggy-packing on top of the one pull request train. And it would also reduce a
> > bit tree proliferation (at one point we had drm-misc/-bridge/-panel/-trivial and
> > may more even). And at least everyone I chatted with seems to like the idea in
> > principle.
> >
> > But what's still open is how to do it exactly. One big change with group
> > maintainership is that you can't rebase a tree anymore. And right now I need to
> > rebase drm-misc fairly often to throw out bad apples again. I think solving that
> > is the important bit to make a shared drm-misc work. A few ideas:
> 
> This is kinda what I don't like. I don't want a tree that can't rebase
> out bad stuff
> coming to me that often. If we are being too over eager in merging
> stuff the answer
> is to merge less not try and merge more. If we have a tree where things
> are thrown until they stick I'm not sure the history will ever be nice
> to pull from.
> 
> I'd rather consider a staging tree where everything from patchwork can
> get thrown
> into, CIed, but that we cherry-pick fixes things to go back to me from the what
> works category.

I fully agree on your concern that drm-misc could become a mess - that's
why I think everything bigger than e.g. 2 patches or touching drivers
needs to soak first in a topic branch. We could still get it into drm-misc
with cherry-picks to avoid too many merges.

The other part where I often had to squash in fixups is build fail on arm.
But I fixed myself with some decent cross-compile toolchain, so I'm
positive that won't happen any more.

I think a good idea would be that I test-drive this process a bit without
adding more maintainers, i.e.
- no more rebases for drm-misc
- topic branches for everything big
- dutifully testing arm before pushing

I'll do that over the next few weeks, then we can see how bad it would
look and whether we need more to avoid bad history.

> I'm finding with i915 for example there is a massive latency in the pipeline now
> waiting for fixes, and the pipeline to the end of drm-intel-next is
> very long and
> hard to figure out what fixes should be pulled back, and how much of the
> driver has been rewritten between the -next and the -fixes pulls.
> 
> > - I think CI is super-important. We're starting to finally roll that out for
> >   real for i915, and it's catching an awful lot of stuff already. Not yet ready
> >   for prime-time on public mailing lists, and for misc we probably can't test
> >   every patch before they land like we do for i915. But CI should have veto
> >   power before a pull request goes to Dave imo.
> >
> >   For non-i915 Daniel Stone and others are working on ARM CI using the generic
> >   igt testcases for kms. And I'm open to merging driver-specific tests into igt
> >   too, if it makes sense, and e.g. Eric has already pushed some vc4 tests.
> >
> > - Stuff needs to at least compile cleanly before pushing. I've been really bad
> >   at that wrt arm drivers with my own drm-misc, but turns out it's fairly easy
> >   to get this right: http://blog.ffwll.ch/2016/02/arm-kernel-cross-compiling.html
> >
> > - Bad apples need to be kicked out with reverts, not rebases. I think that's
> >   fine for simple patches, and hence those can go directly to drm-misc. But a
> >   bunch of subsystem-wide refactorings go in through mis trees, and for those
> >   constantly mass-reverting until it's all solid is silly. And ime you need some
> >   soak time in a shared tree to iron out bugs with those kind of endeavours. We
> >   can address that with ad-hoc&short-lived topic branches which are then again
> >   owned by a single maintainer, but automatically pulled into an integration
> >   tree. After some soaking time to give CI systems time to crunch through those
> >   topic branches can then be merged into the main drm-misc and removed.
> 
> I think integration trees are probably the way forward, but I
> understand they come
> with a massive overhead for constant merging, I like the idea of topic
> branches until
> it becomes my turn to spaghetti merge 3-4 features at once. At that
> point I still
> fallback to the slow things down, pick one feature merge it, back the
> others off,
> and I think this is something we should do more off, the pile everything in and
> hope that magical CI will stabilise things doesn't seem to be a responsible
> process to me.

My idea is that you won't see the integration tree mess ever, but that
tested stuff would land in drm-misc first to resolve conflicts. But really
there never should be, since when a series takes longer than 1-2 weeks to
get ready it's just not ready for merging. And the entire topic branch
should be kicked out.

Also integration tree mangling is why I suggested we start out using dim
scripts - we're integrating 5-10 branches for drm-intel-nightly since
years, since a few months with multiple maintainers pushing conflict
resolutions, and it all works really well.

Also agreed that CI won't magically make stuff stabilise - if CI
complaints the rule needs to be to revert or drop the topic branch, not
try to wrestle it until it's less unhappy. That indeed just doesn't work,
and we're learning all that with drm-intel right now (before we had no CI
but ran on hope, which works even less).

> so I'd like CI to be happening but I'd like it to be happening before the git
> history is baked into stone.

Yeah that's what we do for drm-intel. At least right now we don't even
have anyone doing arm CI, much less CI before merging. Daniel Stone is
working on it though, so I'm hopeful. But yeah for my own stuff or patches
from intel folks we already throw them all at CI before they land
anywhere.

> > - Also we can't roll forward to latest drm-next easily with rebases any more. So
> >   that needs to happen either right after the pull request lands (when no patch
> >   has been merged meanwhile). Or with backmerges, which then need a short commit
> >   message as to way the backmerge is needed ("Backmerge because we need
> >   $feature" is what I usually type), plus sob line from the committer. And of
> >   course the backmerged treed must be stable/non-rebasing and preferrably the
> >   backmerge should be a release/pull-request tag.
> >
> > - For tooling I suggest we just go with drm-intel maintainer-tools for a start.
> >   Picking dim has a few reasons:
> >   * Well tested, documented and fairly complete (includes e.g. bash-completion).
> >   * Integrated support for topic branches, including support for git worktree.
> >   * Well-exercised and robust integration tree support.
> >   Short-term we could add some convenience functions (e.g. for creating/merging
> >   drm-misc topic branches). Long-term we might or might not want to have this
> >   separated from drm-intel - that should be possible but a bit of work. Also,
> >   this way drm-misc would stay at it's current location while we figure things
> >   out. Longer term we might also want to look into adding other big drivers
> >   into an over drm integration.
> >
> > Of course group maintainership needs an initial group. My experience from
> > drm-intel is that a bigger group of maintainer has benefits: It's clear that
> > part-time maintaining is ok too, with maintainers focusing on their area of
> > interest/expertise and only helping out in other places when there's a gap (due
> > to e.g. vacations). Anyway, here's my thoughts for the starting group:
> > Archit, Jani, Thierry & me as existing maintainers of drm-misc/bridge/panel,
> > Alex&Rob as maintainers of big drivers and engaged in core drm stuff,
> > Daniel Stone so that he has no more excuses to stall on arm drm ci.
> >
> > I think if this goes well we can extend it to more driver maintainers, e.g.
> > Patrick would like to just push gma500 patches to some tree and not fiddle with
> > pull requests all the time himself.
> 
> Is pull request sending a big issue? I don't really want a misc tree
> as a way to avoid sending distinct pull requests for drivers. If we do
> this I want
> it to be drm core patches and subsequent fallout only and if we want
> an unmaintained
> drivers patches tree to do that separately.

Eric and Patrick mentioned in private that pull reqs for drivers is
somewhat a nuisance for them. And iirc Thierry also said he's drowning, so
often only gets around to stuff very late.

But really driver trees for small drivers is an entire different topic
that I'd like to maybe keep in mind. But certainly not for initial.

> I don't really like the misc tree replacing panel/bridge tree either,
> those were nicely
> compartmentalised in my head as trees I pulled but didn't really care
> about as Thierry
> did a good job and I don't have ARM hw that I personally care about.

Hm, I mentioned that since I've heard complaints from ARM folks about
tree proliferation, so figured we could try something to get things
together a bit more. But since I care about x86 first I don't mind at all
if we keep drm-misc restricted to what I pulled in thus far if you don't
want this.

> It's important to me that stuff still comes in some sort of
> compartments, as it decides
> for each pull request how much time/effort I put into testing it on my
> end, I'm pretty much
> at the mercy of ARM hw maintainers, but with x86 I still like to smoke
> test it on some
> hardware as much as possible, also with DRM core stuff.

If we keep bridge/panel out I don't expect stuff will change at all in
that regard for you. drm-misc will still be misc random patches, plus the
oddball treewide refactor. And I'll still try to pace the bigger refactors
reasonable. That needs some coordination among maintainers, but I think
the bigger upside here is to get the oddball ones in with less overhead -
big patch series are much harder to forget about ;-)

> So I'm still not sure where I stand on this, personally I don't care
> if you take holidays
> and stuff stalls, delays aren't the big enemy here, merging crap is
> much worse than
> delaying.

One part of my motivation is that this went much, much better than I ever
hoped for in drm-intel. There's teething issues for sure, but it helped
massively in increasing the bus factor, and in reducing people pinging me
directly for every little tiny patch. So much more time for me to look at
patches that are actually important for me to look at wrt "should this go
in or not".

Anyway no rush at all here for drm-misc, and I think next step is for me
to test-run the "no rebasing" to figure that part out. But still would
like comments/ideas from others, too.

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: group maintainership for misc drm trees

[Patrik Jakobsson]

On Tue, Feb 16, 2016 at 10:44 PM, Daniel Vetter <daniel@ffwll.ch> wrote:
> On Tue, Feb 16, 2016 at 11:44:52AM +1000, Dave Airlie wrote:
>> On 15 February 2016 at 20:06, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
>> > Hi all,
>> >
>> > I've already chatted with some of you in private, here's the entire idea with a
>> > bit more thought. My motiviation for group maintainership of drm-misc was that I
>> > got a bit a guilty feeling the last few vacations/conferences when folks pinged
>> > about reviewed/tested pretty patches not landing. But also just increasing the
>> > bus factor and sharing the load better is good. And finally a shared misc drm
>> > tree would allow fringe drivers to get faster into Dave's drm-next by
>> > piggy-packing on top of the one pull request train. And it would also reduce a
>> > bit tree proliferation (at one point we had drm-misc/-bridge/-panel/-trivial and
>> > may more even). And at least everyone I chatted with seems to like the idea in
>> > principle.
>> >
>> > But what's still open is how to do it exactly. One big change with group
>> > maintainership is that you can't rebase a tree anymore. And right now I need to
>> > rebase drm-misc fairly often to throw out bad apples again. I think solving that
>> > is the important bit to make a shared drm-misc work. A few ideas:
>>
>> This is kinda what I don't like. I don't want a tree that can't rebase
>> out bad stuff
>> coming to me that often. If we are being too over eager in merging
>> stuff the answer
>> is to merge less not try and merge more. If we have a tree where things
>> are thrown until they stick I'm not sure the history will ever be nice
>> to pull from.
>>
>> I'd rather consider a staging tree where everything from patchwork can
>> get thrown
>> into, CIed, but that we cherry-pick fixes things to go back to me from the what
>> works category.
>
> I fully agree on your concern that drm-misc could become a mess - that's
> why I think everything bigger than e.g. 2 patches or touching drivers
> needs to soak first in a topic branch. We could still get it into drm-misc
> with cherry-picks to avoid too many merges.
>
> The other part where I often had to squash in fixups is build fail on arm.
> But I fixed myself with some decent cross-compile toolchain, so I'm
> positive that won't happen any more.
>
> I think a good idea would be that I test-drive this process a bit without
> adding more maintainers, i.e.
> - no more rebases for drm-misc
> - topic branches for everything big
> - dutifully testing arm before pushing
>
> I'll do that over the next few weeks, then we can see how bad it would
> look and whether we need more to avoid bad history.
>
>> I'm finding with i915 for example there is a massive latency in the pipeline now
>> waiting for fixes, and the pipeline to the end of drm-intel-next is
>> very long and
>> hard to figure out what fixes should be pulled back, and how much of the
>> driver has been rewritten between the -next and the -fixes pulls.
>>
>> > - I think CI is super-important. We're starting to finally roll that out for
>> >   real for i915, and it's catching an awful lot of stuff already. Not yet ready
>> >   for prime-time on public mailing lists, and for misc we probably can't test
>> >   every patch before they land like we do for i915. But CI should have veto
>> >   power before a pull request goes to Dave imo.
>> >
>> >   For non-i915 Daniel Stone and others are working on ARM CI using the generic
>> >   igt testcases for kms. And I'm open to merging driver-specific tests into igt
>> >   too, if it makes sense, and e.g. Eric has already pushed some vc4 tests.
>> >
>> > - Stuff needs to at least compile cleanly before pushing. I've been really bad
>> >   at that wrt arm drivers with my own drm-misc, but turns out it's fairly easy
>> >   to get this right: http://blog.ffwll.ch/2016/02/arm-kernel-cross-compiling.html
>> >
>> > - Bad apples need to be kicked out with reverts, not rebases. I think that's
>> >   fine for simple patches, and hence those can go directly to drm-misc. But a
>> >   bunch of subsystem-wide refactorings go in through mis trees, and for those
>> >   constantly mass-reverting until it's all solid is silly. And ime you need some
>> >   soak time in a shared tree to iron out bugs with those kind of endeavours. We
>> >   can address that with ad-hoc&short-lived topic branches which are then again
>> >   owned by a single maintainer, but automatically pulled into an integration
>> >   tree. After some soaking time to give CI systems time to crunch through those
>> >   topic branches can then be merged into the main drm-misc and removed.
>>
>> I think integration trees are probably the way forward, but I
>> understand they come
>> with a massive overhead for constant merging, I like the idea of topic
>> branches until
>> it becomes my turn to spaghetti merge 3-4 features at once. At that
>> point I still
>> fallback to the slow things down, pick one feature merge it, back the
>> others off,
>> and I think this is something we should do more off, the pile everything in and
>> hope that magical CI will stabilise things doesn't seem to be a responsible
>> process to me.
>
> My idea is that you won't see the integration tree mess ever, but that
> tested stuff would land in drm-misc first to resolve conflicts. But really
> there never should be, since when a series takes longer than 1-2 weeks to
> get ready it's just not ready for merging. And the entire topic branch
> should be kicked out.
>
> Also integration tree mangling is why I suggested we start out using dim
> scripts - we're integrating 5-10 branches for drm-intel-nightly since
> years, since a few months with multiple maintainers pushing conflict
> resolutions, and it all works really well.
>
> Also agreed that CI won't magically make stuff stabilise - if CI
> complaints the rule needs to be to revert or drop the topic branch, not
> try to wrestle it until it's less unhappy. That indeed just doesn't work,
> and we're learning all that with drm-intel right now (before we had no CI
> but ran on hope, which works even less).
>
>> so I'd like CI to be happening but I'd like it to be happening before the git
>> history is baked into stone.
>
> Yeah that's what we do for drm-intel. At least right now we don't even
> have anyone doing arm CI, much less CI before merging. Daniel Stone is
> working on it though, so I'm hopeful. But yeah for my own stuff or patches
> from intel folks we already throw them all at CI before they land
> anywhere.
>
>> > - Also we can't roll forward to latest drm-next easily with rebases any more. So
>> >   that needs to happen either right after the pull request lands (when no patch
>> >   has been merged meanwhile). Or with backmerges, which then need a short commit
>> >   message as to way the backmerge is needed ("Backmerge because we need
>> >   $feature" is what I usually type), plus sob line from the committer. And of
>> >   course the backmerged treed must be stable/non-rebasing and preferrably the
>> >   backmerge should be a release/pull-request tag.
>> >
>> > - For tooling I suggest we just go with drm-intel maintainer-tools for a start.
>> >   Picking dim has a few reasons:
>> >   * Well tested, documented and fairly complete (includes e.g. bash-completion).
>> >   * Integrated support for topic branches, including support for git worktree.
>> >   * Well-exercised and robust integration tree support.
>> >   Short-term we could add some convenience functions (e.g. for creating/merging
>> >   drm-misc topic branches). Long-term we might or might not want to have this
>> >   separated from drm-intel - that should be possible but a bit of work. Also,
>> >   this way drm-misc would stay at it's current location while we figure things
>> >   out. Longer term we might also want to look into adding other big drivers
>> >   into an over drm integration.
>> >
>> > Of course group maintainership needs an initial group. My experience from
>> > drm-intel is that a bigger group of maintainer has benefits: It's clear that
>> > part-time maintaining is ok too, with maintainers focusing on their area of
>> > interest/expertise and only helping out in other places when there's a gap (due
>> > to e.g. vacations). Anyway, here's my thoughts for the starting group:
>> > Archit, Jani, Thierry & me as existing maintainers of drm-misc/bridge/panel,
>> > Alex&Rob as maintainers of big drivers and engaged in core drm stuff,
>> > Daniel Stone so that he has no more excuses to stall on arm drm ci.
>> >
>> > I think if this goes well we can extend it to more driver maintainers, e.g.
>> > Patrick would like to just push gma500 patches to some tree and not fiddle with
>> > pull requests all the time himself.
>>
>> Is pull request sending a big issue? I don't really want a misc tree
>> as a way to avoid sending distinct pull requests for drivers. If we do
>> this I want
>> it to be drm core patches and subsequent fallout only and if we want
>> an unmaintained
>> drivers patches tree to do that separately.
>
> Eric and Patrick mentioned in private that pull reqs for drivers is
> somewhat a nuisance for them. And iirc Thierry also said he's drowning, so
> often only gets around to stuff very late.

I don't mind sending pull requests. It just felt overkill to do a PR
for a few trivial fixes. If keeping gma500 stuff separated from
drm-misc is easier for you and Dave I don't mind doing that.

-Patrik

>
> But really driver trees for small drivers is an entire different topic
> that I'd like to maybe keep in mind. But certainly not for initial.
>
>> I don't really like the misc tree replacing panel/bridge tree either,
>> those were nicely
>> compartmentalised in my head as trees I pulled but didn't really care
>> about as Thierry
>> did a good job and I don't have ARM hw that I personally care about.
>
> Hm, I mentioned that since I've heard complaints from ARM folks about
> tree proliferation, so figured we could try something to get things
> together a bit more. But since I care about x86 first I don't mind at all
> if we keep drm-misc restricted to what I pulled in thus far if you don't
> want this.
>
>> It's important to me that stuff still comes in some sort of
>> compartments, as it decides
>> for each pull request how much time/effort I put into testing it on my
>> end, I'm pretty much
>> at the mercy of ARM hw maintainers, but with x86 I still like to smoke
>> test it on some
>> hardware as much as possible, also with DRM core stuff.
>
> If we keep bridge/panel out I don't expect stuff will change at all in
> that regard for you. drm-misc will still be misc random patches, plus the
> oddball treewide refactor. And I'll still try to pace the bigger refactors
> reasonable. That needs some coordination among maintainers, but I think
> the bigger upside here is to get the oddball ones in with less overhead -
> big patch series are much harder to forget about ;-)
>
>> So I'm still not sure where I stand on this, personally I don't care
>> if you take holidays
>> and stuff stalls, delays aren't the big enemy here, merging crap is
>> much worse than
>> delaying.
>
> One part of my motivation is that this went much, much better than I ever
> hoped for in drm-intel. There's teething issues for sure, but it helped
> massively in increasing the bus factor, and in reducing people pinging me
> directly for every little tiny patch. So much more time for me to look at
> patches that are actually important for me to look at wrt "should this go
> in or not".
>
> Anyway no rush at all here for drm-misc, and I think next step is for me
> to test-run the "no rebasing" to figure that part out. But still would
> like comments/ideas from others, too.
>
> Cheers, Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: RFC: group maintainership for misc drm trees

[Thierry Reding]

[-- Attachment #1.1: Type: text/plain, Size: 7831 bytes --]

On Tue, Feb 16, 2016 at 11:44:52AM +1000, Dave Airlie wrote:
> On 15 February 2016 at 20:06, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> > Hi all,
> >
> > I've already chatted with some of you in private, here's the entire idea with a
> > bit more thought. My motiviation for group maintainership of drm-misc was that I
> > got a bit a guilty feeling the last few vacations/conferences when folks pinged
> > about reviewed/tested pretty patches not landing. But also just increasing the
> > bus factor and sharing the load better is good. And finally a shared misc drm
> > tree would allow fringe drivers to get faster into Dave's drm-next by
> > piggy-packing on top of the one pull request train. And it would also reduce a
> > bit tree proliferation (at one point we had drm-misc/-bridge/-panel/-trivial and
> > may more even). And at least everyone I chatted with seems to like the idea in
> > principle.
> >
> > But what's still open is how to do it exactly. One big change with group
> > maintainership is that you can't rebase a tree anymore. And right now I need to
> > rebase drm-misc fairly often to throw out bad apples again. I think solving that
> > is the important bit to make a shared drm-misc work. A few ideas:
> 
> This is kinda what I don't like. I don't want a tree that can't rebase
> out bad stuff
> coming to me that often. If we are being too over eager in merging
> stuff the answer
> is to merge less not try and merge more. If we have a tree where things
> are thrown until they stick I'm not sure the history will ever be nice
> to pull from.
> 
> I'd rather consider a staging tree where everything from patchwork can
> get thrown
> into, CIed, but that we cherry-pick fixes things to go back to me from the what
> works category.

This is kind of what drm-trivial was meant to be (at least for drivers
not necessarily core stuff), although I didn't see much benefit in (and
hence gave up on) maintaining such a tree if at the end of the day you
went and picked up the patches manually anyway.

My main pet peeve at the time had been that whenever you make changes
across all of DRM we're not very good at picking things up. Technically
maintainers would need to ACK such patches and then we'd have to wait
on either you or Daniel to pick them up.

I think it'd be really great if we had a tree where we can make these
kinds of patches available earlier. My experience has been that most of
the trivial errors are detected once patches hit linux-next. The
alternative to drm-trivial is to establish -next trees for all drivers.
At the moment there's only a handful of those, and not even radeon/amd
or nouveau feed into linux-next. With all the CI infrastructure that's
been created over the last few years I think we're missing out on much
of this goodness by not getting more code into linux-next early on. And
there's also additional testers which use linux-next in day to day use
and will report back when things break (beyond build issues or obvious
failures such as those that would panic unconditionally on boot).

Of course for separate driver trees to work well they do need to be in
a well-maintained state, and I'll be the first to admit that that's not
at all as easy as it may sound.

Feeding code into linux-next comes with an increase in responsibility as
well, because if we merge patches that break builds we make life
miserable for everyone else. Of course waiting until the merge window
before merging code is not a good solution either because you'll simply
postpone the misery. Probably the simplest answer here is to have some
standard sanity build scripts that will give us some confidence in that
what we merge at least builds fine. I've been maintaining a set of
scripts to do exactly that, and I'm hoping to find the time to polish
them a little more and perhaps write something up so that more people
can use them.

> I'm finding with i915 for example there is a massive latency in the pipeline now
> waiting for fixes, and the pipeline to the end of drm-intel-next is
> very long and
> hard to figure out what fixes should be pulled back, and how much of the
> driver has been rewritten between the -next and the -fixes pulls.
> 
> > - I think CI is super-important. We're starting to finally roll that out for
> >   real for i915, and it's catching an awful lot of stuff already. Not yet ready
> >   for prime-time on public mailing lists, and for misc we probably can't test
> >   every patch before they land like we do for i915. But CI should have veto
> >   power before a pull request goes to Dave imo.
> >
> >   For non-i915 Daniel Stone and others are working on ARM CI using the generic
> >   igt testcases for kms. And I'm open to merging driver-specific tests into igt
> >   too, if it makes sense, and e.g. Eric has already pushed some vc4 tests.
> >
> > - Stuff needs to at least compile cleanly before pushing. I've been really bad
> >   at that wrt arm drivers with my own drm-misc, but turns out it's fairly easy
> >   to get this right: http://blog.ffwll.ch/2016/02/arm-kernel-cross-compiling.html
> >
> > - Bad apples need to be kicked out with reverts, not rebases. I think that's
> >   fine for simple patches, and hence those can go directly to drm-misc. But a
> >   bunch of subsystem-wide refactorings go in through mis trees, and for those
> >   constantly mass-reverting until it's all solid is silly. And ime you need some
> >   soak time in a shared tree to iron out bugs with those kind of endeavours. We
> >   can address that with ad-hoc&short-lived topic branches which are then again
> >   owned by a single maintainer, but automatically pulled into an integration
> >   tree. After some soaking time to give CI systems time to crunch through those
> >   topic branches can then be merged into the main drm-misc and removed.
> 
> I think integration trees are probably the way forward, but I
> understand they come
> with a massive overhead for constant merging, I like the idea of topic
> branches until
> it becomes my turn to spaghetti merge 3-4 features at once. At that
> point I still
> fallback to the slow things down, pick one feature merge it, back the
> others off,
> and I think this is something we should do more off, the pile everything in and
> hope that magical CI will stabilise things doesn't seem to be a responsible
> process to me.
> 
> so I'd like CI to be happening but I'd like it to be happening before the git
> history is baked into stone.

I think topic branches work very well for this case. For Tegra I've been
maintaining an assortment of topic branches in a development tree that I
rebase onto linux-next (almost) on a daily basis. With the right scripts
this is fairly easy to do. It has the big advantage that you can keep
topic branches separate and merge them into one tree to test out the
result. Once the feature is ready you can "promote" it to one of the
stable branches.

I think for DRM this could work very well. There could be a single tree
for integration where almost everything gets pulled in. There could be
separate branches for the drivers, plus topic branches that track the
development of new features. Once a feature is ready, or maintainers
think that the driver code is done, the branches can either be cherry-
picked or merged into some stable branch that is the base for all other
development branches. Once patches are promoted to the stable branch(es)
the whole tree can be rebased and the promoted patches will simply be
rebased out of the respective branches.

This is all quite a bit of maintenance work, but I think with the right
tooling it could be made fairly painless. Group maintainership will also
help spread the load.

Thierry

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel