Proposal for a new Committer model

Proposal for a new Committer model

[Mcnamara, John]

Repost from the moving@dpdk.org mailing list to get a wider audience.
Original thread: http://dpdk.org/ml/archives/moving/2016-November/000059.html


Hi,

I'd like to propose a change to the DPDK committer model. Currently we have one committer for the master branch of the DPDK project. 

One committer to master represents a single point of failure and at times can be inefficient. There is also no agreed cover for times when the committer is unavailable such as vacation, public holidays, etc. I propose that we change to a multi-committer model for the DPDK project. We should have three committers for each release that can commit changes to the master branch.
 
There are a number of benefits:
 
1. Greater capacity to commit patches.
2. No single points of failure - a committer should always be available if we have three.
3. A more timely committing of patches. More committers should equal a faster turnaround - ideally, maintainers should also provide feedback on patches submitted within a 2-3 day period, as much as possible, to facilitate this. 
4. It follows best practice in creating a successful multi-vendor community - to achieve this we must ensure there is a level playing field for all participants, no single person should be required to make all of the decisions on patches to be included in the release.  

Having multiple committers will require some degree of co-ordination but there are a number of other communities successfully following this model such as Apache, OVS, FD.io, OpenStack etc. so the approach is workable.

John

Re: Proposal for a new Committer model

[Matthew Hall]

On Thu, Nov 17, 2016 at 09:20:50AM +0000, Mcnamara, John wrote:
> One committer to master represents a single point of failure and at times can be inefficient.

I have a lot more issues because of slow or inconclusive review of patches 
than I do because of committers. Often times they just get rejected in 
Patchwork with no feedback. Or it takes forever to get reviews.

I don't think the committer is the right place to point to the single point of 
failure.

Matthew.

Re: Proposal for a new Committer model

[Neil Horman]

On Thu, Nov 17, 2016 at 09:20:50AM +0000, Mcnamara, John wrote:
> Repost from the moving at dpdk.org mailing list to get a wider audience.
> Original thread: http://dpdk.org/ml/archives/moving/2016-November/000059.html
> 
> 
> Hi,
> 
> I'd like to propose a change to the DPDK committer model. Currently we have one committer for the master branch of the DPDK project. 
> 
> One committer to master represents a single point of failure and at times can be inefficient. There is also no agreed cover for times when the committer is unavailable such as vacation, public holidays, etc. I propose that we change to a multi-committer model for the DPDK project. We should have three committers for each release that can commit changes to the master branch.
>  
> There are a number of benefits:
>  
> 1. Greater capacity to commit patches.
> 2. No single points of failure - a committer should always be available if we have three.
> 3. A more timely committing of patches. More committers should equal a faster turnaround - ideally, maintainers should also provide feedback on patches submitted within a 2-3 day period, as much as possible, to facilitate this. 
> 4. It follows best practice in creating a successful multi-vendor community - to achieve this we must ensure there is a level playing field for all participants, no single person should be required to make all of the decisions on patches to be included in the release.  
> 
> Having multiple committers will require some degree of co-ordination but there are a number of other communities successfully following this model such as Apache, OVS, FD.io, OpenStack etc. so the approach is workable.
> 
> John

I agree that the problems you are attempting to address exist and are
worth finding a solution for.  That said, I don't think the solution you
are proposing is the ideal, or complete fix for any of the issues being
addressed.

If I may, I'd like to ennumerate the issues I think you are trying to
address based on your comments above, then make a counter-proposal for a
solution:

Problems to address:

1) high-availability - There is a desire to make sure that, when patches
are proposed, they are integrated in a timely fashion.

2) high-throughput - DPDK has a large volume of patches, more than one
person can normally integrate.  There is a desire to shard that work such
that it is handled by multiple individuals

3) Multi-Vendor fairness - There is a desire for multiple vendors to feel
as though the project tree maintainer isn't biased toward any individual
vendor.

To solve these I would propose the following solution (which is simmilar
to, but not quite identical, to yours).

A) Further promote subtree maintainership.  This was a conversation that I
proposed some time ago, but my proposed granularity was discarded in favor
of something that hasn't worked as well (in my opinion).  That is to say a
few driver pmds (i40e and fm10k come to mind) have their own tree that
send pull requests to Thomas.  We should be sharding that at a much higher
granularity and using it much more consistently.  That is to say, that we
should have a maintainer for all the ethernet pmds, and another for the
crypto pmds, another for the core eal layer, another for misc libraries
that have low patch volumes, etc.  Each of those subdivisions should have
their own list to communicate on, and each should have a tree that
integrates patches for their own subsystem, and they should on a regular
cycle send pull requests to Thomas.  Thomas in turn should by and large,
only be integrating pull requests.  This should address our high-
throughput issue, in that it will allow multiple maintainers to share the
workload, and integration should be relatively easy.

B) Designate alternates to serve as backups for the maintainer when they
are unavailable.  This provides high-availablility, and sounds very much
like your proposal, but in the interests of clarity, there is still a
single maintainer at any one time, it just may change to ensure the
continued merging of patches, if the primary maintainer isn't available.
Ideally however, those backup alternates arent needed, because most of the
primary maintainers work in merging pull requests, which are done based on
the trust of the submaintainer, and done during a very limited window of
time.  This also partially addreses multi-vendor fairness if your subtree
maintainers come from multiple participating companies.

Regards
Neil

Re: Proposal for a new Committer model

[Jerin Jacob]

On Fri, Nov 18, 2016 at 01:09:35PM -0500, Neil Horman wrote:
> On Thu, Nov 17, 2016 at 09:20:50AM +0000, Mcnamara, John wrote:
> > Repost from the moving at dpdk.org mailing list to get a wider audience.
> > Original thread: http://dpdk.org/ml/archives/moving/2016-November/000059.html
> > 
> > 
> > Hi,
> > 
> > I'd like to propose a change to the DPDK committer model. Currently we have one committer for the master branch of the DPDK project. 
> > 
> > One committer to master represents a single point of failure and at times can be inefficient. There is also no agreed cover for times when the committer is unavailable such as vacation, public holidays, etc. I propose that we change to a multi-committer model for the DPDK project. We should have three committers for each release that can commit changes to the master branch.
> >  
> > There are a number of benefits:
> >  
> > 1. Greater capacity to commit patches.
> > 2. No single points of failure - a committer should always be available if we have three.
> > 3. A more timely committing of patches. More committers should equal a faster turnaround - ideally, maintainers should also provide feedback on patches submitted within a 2-3 day period, as much as possible, to facilitate this. 
> > 4. It follows best practice in creating a successful multi-vendor community - to achieve this we must ensure there is a level playing field for all participants, no single person should be required to make all of the decisions on patches to be included in the release.  
> > 
> > Having multiple committers will require some degree of co-ordination but there are a number of other communities successfully following this model such as Apache, OVS, FD.io, OpenStack etc. so the approach is workable.
> > 
> > John
> 
> I agree that the problems you are attempting to address exist and are
> worth finding a solution for.  That said, I don't think the solution you
> are proposing is the ideal, or complete fix for any of the issues being
> addressed.
> 
> If I may, I'd like to ennumerate the issues I think you are trying to
> address based on your comments above, then make a counter-proposal for a
> solution:
> 
> Problems to address:
> 
> 1) high-availability - There is a desire to make sure that, when patches
> are proposed, they are integrated in a timely fashion.
> 
> 2) high-throughput - DPDK has a large volume of patches, more than one
> person can normally integrate.  There is a desire to shard that work such
> that it is handled by multiple individuals
> 
> 3) Multi-Vendor fairness - There is a desire for multiple vendors to feel
> as though the project tree maintainer isn't biased toward any individual
> vendor.
> 
> To solve these I would propose the following solution (which is simmilar
> to, but not quite identical, to yours).
> 
> A) Further promote subtree maintainership.  This was a conversation that I
> proposed some time ago, but my proposed granularity was discarded in favor
> of something that hasn't worked as well (in my opinion).  That is to say a
> few driver pmds (i40e and fm10k come to mind) have their own tree that
> send pull requests to Thomas.  We should be sharding that at a much higher
> granularity and using it much more consistently.  That is to say, that we
> should have a maintainer for all the ethernet pmds, and another for the
> crypto pmds, another for the core eal layer, another for misc libraries
> that have low patch volumes, etc.  Each of those subdivisions should have
> their own list to communicate on, and each should have a tree that
> integrates patches for their own subsystem, and they should on a regular
> cycle send pull requests to Thomas.  Thomas in turn should by and large,
> only be integrating pull requests.  This should address our high-
> throughput issue, in that it will allow multiple maintainers to share the
> workload, and integration should be relatively easy.

+1

> 
> B) Designate alternates to serve as backups for the maintainer when they
> are unavailable.  This provides high-availablility, and sounds very much
> like your proposal, but in the interests of clarity, there is still a
> single maintainer at any one time, it just may change to ensure the
> continued merging of patches, if the primary maintainer isn't available.
> Ideally however, those backup alternates arent needed, because most of the
> primary maintainers work in merging pull requests, which are done based on
> the trust of the submaintainer, and done during a very limited window of
> time.  This also partially addreses multi-vendor fairness if your subtree
> maintainers come from multiple participating companies.

+1

> 
> Regards
> Neil
> 
> 

Re: Proposal for a new Committer model

[Stephen Hemminger]

why aren't some patches as marked trivial and accepted right away.

On Fri, Nov 18, 2016 at 11:06 AM, Jerin Jacob <
jerin.jacob@caviumnetworks.com> wrote:

> On Fri, Nov 18, 2016 at 01:09:35PM -0500, Neil Horman wrote:
> > On Thu, Nov 17, 2016 at 09:20:50AM +0000, Mcnamara, John wrote:
> > > Repost from the moving at dpdk.org mailing list to get a wider
> audience.
> > > Original thread: http://dpdk.org/ml/archives/
> moving/2016-November/000059.html
> > >
> > >
> > > Hi,
> > >
> > > I'd like to propose a change to the DPDK committer model. Currently we
> have one committer for the master branch of the DPDK project.
> > >
> > > One committer to master represents a single point of failure and at
> times can be inefficient. There is also no agreed cover for times when the
> committer is unavailable such as vacation, public holidays, etc. I propose
> that we change to a multi-committer model for the DPDK project. We should
> have three committers for each release that can commit changes to the
> master branch.
> > >
> > > There are a number of benefits:
> > >
> > > 1. Greater capacity to commit patches.
> > > 2. No single points of failure - a committer should always be
> available if we have three.
> > > 3. A more timely committing of patches. More committers should equal a
> faster turnaround - ideally, maintainers should also provide feedback on
> patches submitted within a 2-3 day period, as much as possible, to
> facilitate this.
> > > 4. It follows best practice in creating a successful multi-vendor
> community - to achieve this we must ensure there is a level playing field
> for all participants, no single person should be required to make all of
> the decisions on patches to be included in the release.
> > >
> > > Having multiple committers will require some degree of co-ordination
> but there are a number of other communities successfully following this
> model such as Apache, OVS, FD.io, OpenStack etc. so the approach is
> workable.
> > >
> > > John
> >
> > I agree that the problems you are attempting to address exist and are
> > worth finding a solution for.  That said, I don't think the solution you
> > are proposing is the ideal, or complete fix for any of the issues being
> > addressed.
> >
> > If I may, I'd like to ennumerate the issues I think you are trying to
> > address based on your comments above, then make a counter-proposal for a
> > solution:
> >
> > Problems to address:
> >
> > 1) high-availability - There is a desire to make sure that, when patches
> > are proposed, they are integrated in a timely fashion.
> >
> > 2) high-throughput - DPDK has a large volume of patches, more than one
> > person can normally integrate.  There is a desire to shard that work such
> > that it is handled by multiple individuals
> >
> > 3) Multi-Vendor fairness - There is a desire for multiple vendors to feel
> > as though the project tree maintainer isn't biased toward any individual
> > vendor.
> >
> > To solve these I would propose the following solution (which is simmilar
> > to, but not quite identical, to yours).
> >
> > A) Further promote subtree maintainership.  This was a conversation that
> I
> > proposed some time ago, but my proposed granularity was discarded in
> favor
> > of something that hasn't worked as well (in my opinion).  That is to say
> a
> > few driver pmds (i40e and fm10k come to mind) have their own tree that
> > send pull requests to Thomas.  We should be sharding that at a much
> higher
> > granularity and using it much more consistently.  That is to say, that we
> > should have a maintainer for all the ethernet pmds, and another for the
> > crypto pmds, another for the core eal layer, another for misc libraries
> > that have low patch volumes, etc.  Each of those subdivisions should have
> > their own list to communicate on, and each should have a tree that
> > integrates patches for their own subsystem, and they should on a regular
> > cycle send pull requests to Thomas.  Thomas in turn should by and large,
> > only be integrating pull requests.  This should address our high-
> > throughput issue, in that it will allow multiple maintainers to share the
> > workload, and integration should be relatively easy.
>
> +1
>
> >
> > B) Designate alternates to serve as backups for the maintainer when they
> > are unavailable.  This provides high-availablility, and sounds very much
> > like your proposal, but in the interests of clarity, there is still a
> > single maintainer at any one time, it just may change to ensure the
> > continued merging of patches, if the primary maintainer isn't available.
> > Ideally however, those backup alternates arent needed, because most of
> the
> > primary maintainers work in merging pull requests, which are done based
> on
> > the trust of the submaintainer, and done during a very limited window of
> > time.  This also partially addreses multi-vendor fairness if your subtree
> > maintainers come from multiple participating companies.
>
> +1
>
> >
> > Regards
> > Neil
> >
> >
>

Re: Proposal for a new Committer model

[Thomas Monjalon]

2016-11-18 13:09, Neil Horman:
> A) Further promote subtree maintainership.  This was a conversation that I
> proposed some time ago, but my proposed granularity was discarded in favor
> of something that hasn't worked as well (in my opinion).  That is to say a
> few driver pmds (i40e and fm10k come to mind) have their own tree that
> send pull requests to Thomas.

Yes we tried this fine granularity and stated that it was not working well.
We are now using the bigger granularity that you describe below.

> We should be sharding that at a much higher
> granularity and using it much more consistently.  That is to say, that we
> should have a maintainer for all the ethernet pmds, and another for the
> crypto pmds, another for the core eal layer, another for misc libraries
> that have low patch volumes, etc.

Yes we could open a tree for EAL and another one for the core libraries.

> Each of those subdivisions should have
> their own list to communicate on, and each should have a tree that
> integrates patches for their own subsystem, and they should on a regular
> cycle send pull requests to Thomas.

Yes I think it is now a good idea to split the mailing list traffic,
at least for netdev and cryptodev.

> Thomas in turn should by and large,
> only be integrating pull requests.  This should address our high-
> throughput issue, in that it will allow multiple maintainers to share the
> workload, and integration should be relatively easy.

Yes in an ideal organization, the last committer does only a last check
that technical plan and fairness are respected.
So it gives more time to coordinate the plans :)

> B) Designate alternates to serve as backups for the maintainer when they
> are unavailable.  This provides high-availablility, and sounds very much
> like your proposal, but in the interests of clarity, there is still a
> single maintainer at any one time, it just may change to ensure the
> continued merging of patches, if the primary maintainer isn't available.
> Ideally however, those backup alternates arent needed, because most of the
> primary maintainers work in merging pull requests, which are done based on
> the trust of the submaintainer, and done during a very limited window of
> time.  This also partially addreses multi-vendor fairness if your subtree
> maintainers come from multiple participating companies.

About the merge window, I do not have a strong opinion about how it can be
improved. However, I know that closing the window too early makes developer
unhappy because it makes wait - between development start and its release -
longer.

Re: Proposal for a new Committer model

[Neil Horman]

On Mon, Nov 21, 2016 at 09:52:41AM +0100, Thomas Monjalon wrote:
> 2016-11-18 13:09, Neil Horman:
> > A) Further promote subtree maintainership.  This was a conversation that I
> > proposed some time ago, but my proposed granularity was discarded in favor
> > of something that hasn't worked as well (in my opinion).  That is to say a
> > few driver pmds (i40e and fm10k come to mind) have their own tree that
> > send pull requests to Thomas.
> 
> Yes we tried this fine granularity and stated that it was not working well.
> We are now using the bigger granularity that you describe below.
> 
Ok, thats good, but that must be _very_ new.  Looking at your git tree, I see no
merge commits.  How are you pulling from those subtrees?


> > We should be sharding that at a much higher
> > granularity and using it much more consistently.  That is to say, that we
> > should have a maintainer for all the ethernet pmds, and another for the
> > crypto pmds, another for the core eal layer, another for misc libraries
> > that have low patch volumes, etc.
> 
> Yes we could open a tree for EAL and another one for the core libraries.
> 
That could be worthwhile.  Lets see how the net and crypto subtrees work out
(assuming again that these trees are newly founded)


> > Each of those subdivisions should have
> > their own list to communicate on, and each should have a tree that
> > integrates patches for their own subsystem, and they should on a regular
> > cycle send pull requests to Thomas.
> 
> Yes I think it is now a good idea to split the mailing list traffic,
> at least for netdev and cryptodev.
> 
Agreed, that serves two purposes, it lowers the volume for people with a
specific interest (i.e. its a rudimentary filter), and it avoids confusion
between you and the subtree maintainer (that is to say, you don't have to even
consider pulling patches that go to the crypo and net lists, you just have to
trust that they pull those patches in and send you appropriate pull requests).

> > Thomas in turn should by and large,
> > only be integrating pull requests.  This should address our high-
> > throughput issue, in that it will allow multiple maintainers to share the
> > workload, and integration should be relatively easy.
> 
> Yes in an ideal organization, the last committer does only a last check
> that technical plan and fairness are respected.
> So it gives more time to coordinate the plans :)
> 
Correct.  Thats never 100% accurate of course, some things will still have to
come to you directly, simply by virtue of the fact that they don't completely
fit anywhere else, but thats ok, the goal is really just to get your total patch
volume lower, and replace it with pull requests that you can either trivialy
mere or figure out with the help of the subtree maintainer.

> > B) Designate alternates to serve as backups for the maintainer when they
> > are unavailable.  This provides high-availablility, and sounds very much
> > like your proposal, but in the interests of clarity, there is still a
> > single maintainer at any one time, it just may change to ensure the
> > continued merging of patches, if the primary maintainer isn't available.
> > Ideally however, those backup alternates arent needed, because most of the
> > primary maintainers work in merging pull requests, which are done based on
> > the trust of the submaintainer, and done during a very limited window of
> > time.  This also partially addreses multi-vendor fairness if your subtree
> > maintainers come from multiple participating companies.
> 
> About the merge window, I do not have a strong opinion about how it can be
> improved. However, I know that closing the window too early makes developer
> unhappy because it makes wait - between development start and its release -
> longer.

This is a fair point, but I'm not talking about closing it early here, all
I'm suggesting is that, if you do proper pull requests from subtrees, your tree
Thomas will only need a reasonably small window of time to accept new features,
because you'll just merge the subtrees, rather than integrating individual
patches.  E.g. you won't be constantly merging patches over the course of a
development cycle, your tree's HEAD will mostly consist of merge commits as
subtree maintainers send you pull requests, and ideally they will send those
near the start of a window.  How long you keep your merge window open after that
is up to you.

Neil


 
> 

Re: Proposal for a new Committer model

[Ferruh Yigit]

On 11/22/2016 7:52 PM, Neil Horman wrote:
> On Mon, Nov 21, 2016 at 09:52:41AM +0100, Thomas Monjalon wrote:
>> 2016-11-18 13:09, Neil Horman:
>>> A) Further promote subtree maintainership.  This was a conversation that I
>>> proposed some time ago, but my proposed granularity was discarded in favor
>>> of something that hasn't worked as well (in my opinion).  That is to say a
>>> few driver pmds (i40e and fm10k come to mind) have their own tree that
>>> send pull requests to Thomas.
>>
>> Yes we tried this fine granularity and stated that it was not working well.
>> We are now using the bigger granularity that you describe below.
>>
> Ok, thats good, but that must be _very_ new.  Looking at your git tree, I see no
> merge commits.  How are you pulling from those subtrees?

next-net tree is active for last three releases.

I guess following is the first commit to the sub-tree:
http://dpdk.org/ml/archives/dev/2016-February/032580.html

sub-trees rebase on top of main tree regularly, that is why there is no
merge commit.

> 
> 
>>> We should be sharding that at a much higher
>>> granularity and using it much more consistently.  That is to say, that we
>>> should have a maintainer for all the ethernet pmds, and another for the
>>> crypto pmds, another for the core eal layer, another for misc libraries
>>> that have low patch volumes, etc.
>>
>> Yes we could open a tree for EAL and another one for the core libraries.
>>
> That could be worthwhile.  Lets see how the net and crypto subtrees work out
> (assuming again that these trees are newly founded)
> 
> 
>>> Each of those subdivisions should have
>>> their own list to communicate on, and each should have a tree that
>>> integrates patches for their own subsystem, and they should on a regular
>>> cycle send pull requests to Thomas.
>>
>> Yes I think it is now a good idea to split the mailing list traffic,
>> at least for netdev and cryptodev.
>>
> Agreed, that serves two purposes, it lowers the volume for people with a
> specific interest (i.e. its a rudimentary filter), and it avoids confusion
> between you and the subtree maintainer (that is to say, you don't have to even
> consider pulling patches that go to the crypo and net lists, you just have to
> trust that they pull those patches in and send you appropriate pull requests).

I still find single mail list more useful.
Also with current process, after -rc2 release, patches directly merged
into main tree instead of sub-trees...

> 
>>> Thomas in turn should by and large,
>>> only be integrating pull requests.  This should address our high-
>>> throughput issue, in that it will allow multiple maintainers to share the
>>> workload, and integration should be relatively easy.
>>
>> Yes in an ideal organization, the last committer does only a last check
>> that technical plan and fairness are respected.
>> So it gives more time to coordinate the plans :)
>>
> Correct.  Thats never 100% accurate of course, some things will still have to
> come to you directly, simply by virtue of the fact that they don't completely
> fit anywhere else, but thats ok, the goal is really just to get your total patch
> volume lower, and replace it with pull requests that you can either trivialy
> mere or figure out with the help of the subtree maintainer.
> 
>>> B) Designate alternates to serve as backups for the maintainer when they
>>> are unavailable.  This provides high-availablility, and sounds very much
>>> like your proposal, but in the interests of clarity, there is still a
>>> single maintainer at any one time, it just may change to ensure the
>>> continued merging of patches, if the primary maintainer isn't available.
>>> Ideally however, those backup alternates arent needed, because most of the
>>> primary maintainers work in merging pull requests, which are done based on
>>> the trust of the submaintainer, and done during a very limited window of
>>> time.  This also partially addreses multi-vendor fairness if your subtree
>>> maintainers come from multiple participating companies.
>>
>> About the merge window, I do not have a strong opinion about how it can be
>> improved. However, I know that closing the window too early makes developer
>> unhappy because it makes wait - between development start and its release -
>> longer.
> 
> This is a fair point, but I'm not talking about closing it early here, all
> I'm suggesting is that, if you do proper pull requests from subtrees, your tree
> Thomas will only need a reasonably small window of time to accept new features,
> because you'll just merge the subtrees, rather than integrating individual
> patches.  E.g. you won't be constantly merging patches over the course of a
> development cycle, your tree's HEAD will mostly consist of merge commits as
> subtree maintainers send you pull requests, and ideally they will send those
> near the start of a window.  How long you keep your merge window open after that
> is up to you.
> 
> Neil
> 
> 
>  
>>

Re: Proposal for a new Committer model

[Mcnamara, John]

> -----Original Message-----
> From: Neil Horman [mailto:nhorman@tuxdriver.com]
> Sent: Tuesday, November 22, 2016 7:52 PM
> To: Thomas Monjalon <thomas.monjalon@6wind.com>
> Cc: dev@dpdk.org; Mcnamara, John <john.mcnamara@intel.com>
> Subject: Re: [dpdk-dev] Proposal for a new Committer model
> 
> On Mon, Nov 21, 2016 at 09:52:41AM +0100, Thomas Monjalon wrote:
> > 2016-11-18 13:09, Neil Horman:
> > > A) Further promote subtree maintainership.  This was a conversation
> > > that I proposed some time ago, but my proposed granularity was
> > > discarded in favor of something that hasn't worked as well (in my
> > > opinion).  That is to say a few driver pmds (i40e and fm10k come to
> > > mind) have their own tree that send pull requests to Thomas.
> >
> > Yes we tried this fine granularity and stated that it was not working
> well.
> > We are now using the bigger granularity that you describe below.
> >
> Ok, thats good, but that must be _very_ new.  Looking at your git tree, I
> see no merge commits.  How are you pulling from those subtrees?
> 


Hi Neil,

It seems like the weight of consensus in around your proposal for further subtree maintainers and backups. If you don't mind I'll take your text and redraft it as a potential section on maintainership for a future Project Charter document. Or at least so that we have a documented maintainship process.

 
> > > We should be sharding that at a much higher granularity and using it
> > > much more consistently.  That is to say, that we should have a
> > > maintainer for all the ethernet pmds, and another for the crypto
> > > pmds, another for the core eal layer, another for misc libraries
> > > that have low patch volumes, etc.
> >
> > Yes we could open a tree for EAL and another one for the core libraries.
> >
> That could be worthwhile.  Lets see how the net and crypto subtrees work
> out (assuming again that these trees are newly founded)

Could we define some of the potential subtrees now and look to introduce them in the this release cycle? EAL and the Core libs, as suggested by Thomas, seem like 2 obvious ones.


> 
> > > Each of those subdivisions should have their own list to communicate
> > > on, and each should have a tree that integrates patches for their
> > > own subsystem, and they should on a regular cycle send pull requests
> > > to Thomas.
> >
> > Yes I think it is now a good idea to split the mailing list traffic,
> > at least for netdev and cryptodev.

I'd prefer not to have split dev lists, for now at least. We can reevaluate that again in a few months though.


> >
> 
> > > B) Designate alternates to serve as backups for the maintainer when
> > > they are unavailable.  This provides high-availablility, and sounds
> > > very much like your proposal, but in the interests of clarity, there
> > > is still a single maintainer at any one time, it just may change to
> > > ensure the continued merging of patches, if the primary maintainer
> isn't available.
> > > Ideally however, those backup alternates arent needed, because most
> > > of the primary maintainers work in merging pull requests, which are
> > > done based on the trust of the submaintainer, and done during a very
> > > limited window of time.  This also partially addreses multi-vendor
> > > fairness if your subtree maintainers come from multiple participating
> companies.


+1 on this apart from the limited merge window (for reasons similar to Thomas).

Should we have a call for volunteers for backup, on master and the sub-trees, followed by a simple +1 from community members to endorse them?


John

Re: Proposal for a new Committer model

[Neil Horman]

On Tue, Nov 22, 2016 at 08:56:23PM +0000, Ferruh Yigit wrote:
> On 11/22/2016 7:52 PM, Neil Horman wrote:
> > On Mon, Nov 21, 2016 at 09:52:41AM +0100, Thomas Monjalon wrote:
> >> 2016-11-18 13:09, Neil Horman:
> >>> A) Further promote subtree maintainership.  This was a conversation that I
> >>> proposed some time ago, but my proposed granularity was discarded in favor
> >>> of something that hasn't worked as well (in my opinion).  That is to say a
> >>> few driver pmds (i40e and fm10k come to mind) have their own tree that
> >>> send pull requests to Thomas.
> >>
> >> Yes we tried this fine granularity and stated that it was not working well.
> >> We are now using the bigger granularity that you describe below.
> >>
> > Ok, thats good, but that must be _very_ new.  Looking at your git tree, I see no
> > merge commits.  How are you pulling from those subtrees?
> 
> next-net tree is active for last three releases.
> 
What!?  What is the purpose of holding patches in a subtree for multiple
releases?  If a given changeset isn't ready for merge to Thomas tree the people
working on it should clone the subtree to some place they can all collaborate on
it.  Once it goes into a subtree there needs to be a defined workflow to get it
into the canonical tree that Thomas maintains on a regular, short time frame.
to do less is to confuse the process for everyone involved, and slow people
down, rather than accelerate their work.

> I guess following is the first commit to the sub-tree:
> http://dpdk.org/ml/archives/dev/2016-February/032580.html
> 
> sub-trees rebase on top of main tree regularly, that is why there is no
> merge commit.
> 
I'm not asking about merge commits in the sub-tree, I'm asking about merge
commits in thomas's tree.  There should be a merge commit every time he pulls
from a sub-tree (unless its a fast-forward I think, but with multiple subtrees
and commits going to thomas directly, that should never really happen).  I don't
see any Merge commits in the master branch of his tree, so I'm left wondering
what mechanic is being used to migrate patches from net-next or crypo-next to
his tree.  Thomas, can you comment here?

> > 
> > 
> >>> We should be sharding that at a much higher
> >>> granularity and using it much more consistently.  That is to say, that we
> >>> should have a maintainer for all the ethernet pmds, and another for the
> >>> crypto pmds, another for the core eal layer, another for misc libraries
> >>> that have low patch volumes, etc.
> >>
> >> Yes we could open a tree for EAL and another one for the core libraries.
> >>
> > That could be worthwhile.  Lets see how the net and crypto subtrees work out
> > (assuming again that these trees are newly founded)
> > 
> > 
> >>> Each of those subdivisions should have
> >>> their own list to communicate on, and each should have a tree that
> >>> integrates patches for their own subsystem, and they should on a regular
> >>> cycle send pull requests to Thomas.
> >>
> >> Yes I think it is now a good idea to split the mailing list traffic,
> >> at least for netdev and cryptodev.
> >>
> > Agreed, that serves two purposes, it lowers the volume for people with a
> > specific interest (i.e. its a rudimentary filter), and it avoids confusion
> > between you and the subtree maintainer (that is to say, you don't have to even
> > consider pulling patches that go to the crypo and net lists, you just have to
> > trust that they pull those patches in and send you appropriate pull requests).
> 
> I still find single mail list more useful.
Why?  If you have interest in all the subsystems of a project, then its a small
amount of overhead to subscribe to a set of mailing lists and dump them all to a
single mail folder.  If you only have interest in a subset, its much more
difficult to filter them out, given that we have a plethora of prefix tags for
patches to define subsystems that aren't always used consistently.  Given that
this thread is here because we've identified the patch volume as a problem, it
seems fragmenting the list is the better solution.

> Also with current process, after -rc2 release, patches directly merged
> into main tree instead of sub-trees...
> 
Thats fine, at that point, if everything works properly, Thomas should only be
getting low volume patch flow for stabilization/bug fixing.  If thats not the
case, then perhaps we need to consider doing extra merges from the subtrees
later in the cycle.

Neil

Re: Proposal for a new Committer model

[Ferruh Yigit]

On 11/23/2016 1:48 PM, Neil Horman wrote:
> On Tue, Nov 22, 2016 at 08:56:23PM +0000, Ferruh Yigit wrote:
>> On 11/22/2016 7:52 PM, Neil Horman wrote:
>>> On Mon, Nov 21, 2016 at 09:52:41AM +0100, Thomas Monjalon wrote:
>>>> 2016-11-18 13:09, Neil Horman:
>>>>> A) Further promote subtree maintainership.  This was a conversation that I
>>>>> proposed some time ago, but my proposed granularity was discarded in favor
>>>>> of something that hasn't worked as well (in my opinion).  That is to say a
>>>>> few driver pmds (i40e and fm10k come to mind) have their own tree that
>>>>> send pull requests to Thomas.
>>>>
>>>> Yes we tried this fine granularity and stated that it was not working well.
>>>> We are now using the bigger granularity that you describe below.
>>>>
>>> Ok, thats good, but that must be _very_ new.  Looking at your git tree, I see no
>>> merge commits.  How are you pulling from those subtrees?
>>
>> next-net tree is active for last three releases.
>>
> What!?  What is the purpose of holding patches in a subtree for multiple
> releases?  

:) Of course not holding them in the sub-tree.

Briefly, process is:
- sub-tree gets patches during merge window
- sub-tree first merged into main tree in -rc1 and later in -r2

next-net tree is actively in use for last three releases, and driver/net
patches delegated to this tree. You can see different commiters in main
tree.

> If a given changeset isn't ready for merge to Thomas tree the people
> working on it should clone the subtree to some place they can all collaborate on
> it.  Once it goes into a subtree there needs to be a defined workflow to get it
> into the canonical tree that Thomas maintains on a regular, short time frame.
> to do less is to confuse the process for everyone involved, and slow people
> down, rather than accelerate their work.
> 
>> I guess following is the first commit to the sub-tree:
>> http://dpdk.org/ml/archives/dev/2016-February/032580.html
>>
>> sub-trees rebase on top of main tree regularly, that is why there is no
>> merge commit.
>>
> I'm not asking about merge commits in the sub-tree, I'm asking about merge
> commits in thomas's tree.

Same, talking about Thomas' tree.

>  There should be a merge commit every time he pulls
> from a sub-tree (unless its a fast-forward I think, but with multiple subtrees
> and commits going to thomas directly, that should never really happen).  

That is what happening. Since sub-tree's rebase on top of main tree,
when Thomas merges, it is just plain fast-forward. So it is allowed to
re-write to history in sub-trees.

> I don't
> see any Merge commits in the master branch of his tree, so I'm left wondering
> what mechanic is being used to migrate patches from net-next or crypo-next to
> his tree.  Thomas, can you comment here?
> 
>>>
>>>
>>>>> We should be sharding that at a much higher
>>>>> granularity and using it much more consistently.  That is to say, that we
>>>>> should have a maintainer for all the ethernet pmds, and another for the
>>>>> crypto pmds, another for the core eal layer, another for misc libraries
>>>>> that have low patch volumes, etc.
>>>>
>>>> Yes we could open a tree for EAL and another one for the core libraries.
>>>>
>>> That could be worthwhile.  Lets see how the net and crypto subtrees work out
>>> (assuming again that these trees are newly founded)
>>>
>>>
>>>>> Each of those subdivisions should have
>>>>> their own list to communicate on, and each should have a tree that
>>>>> integrates patches for their own subsystem, and they should on a regular
>>>>> cycle send pull requests to Thomas.
>>>>
>>>> Yes I think it is now a good idea to split the mailing list traffic,
>>>> at least for netdev and cryptodev.
>>>>
>>> Agreed, that serves two purposes, it lowers the volume for people with a
>>> specific interest (i.e. its a rudimentary filter), and it avoids confusion
>>> between you and the subtree maintainer (that is to say, you don't have to even
>>> consider pulling patches that go to the crypo and net lists, you just have to
>>> trust that they pull those patches in and send you appropriate pull requests).
>>
>> I still find single mail list more useful.
> Why?  If you have interest in all the subsystems of a project, then its a small
> amount of overhead to subscribe to a set of mailing lists and dump them all to a
> single mail folder.  If you only have interest in a subset, its much more
> difficult to filter them out, given that we have a plethora of prefix tags for
> patches to define subsystems that aren't always used consistently.  Given that
> this thread is here because we've identified the patch volume as a problem, it
> seems fragmenting the list is the better solution.
> 
>> Also with current process, after -rc2 release, patches directly merged
>> into main tree instead of sub-trees...
>>
> Thats fine, at that point, if everything works properly, Thomas should only be
> getting low volume patch flow for stabilization/bug fixing.  If thats not the
> case, then perhaps we need to consider doing extra merges from the subtrees
> later in the cycle.
> 
> Neil
> 

Re: Proposal for a new Committer model

[Neil Horman]

On Wed, Nov 23, 2016 at 08:21:39AM +0000, Mcnamara, John wrote:
> 
> 
> > -----Original Message-----
> > From: Neil Horman [mailto:nhorman@tuxdriver.com]
> > Sent: Tuesday, November 22, 2016 7:52 PM
> > To: Thomas Monjalon <thomas.monjalon@6wind.com>
> > Cc: dev@dpdk.org; Mcnamara, John <john.mcnamara@intel.com>
> > Subject: Re: [dpdk-dev] Proposal for a new Committer model
> > 
> > On Mon, Nov 21, 2016 at 09:52:41AM +0100, Thomas Monjalon wrote:
> > > 2016-11-18 13:09, Neil Horman:
> > > > A) Further promote subtree maintainership.  This was a conversation
> > > > that I proposed some time ago, but my proposed granularity was
> > > > discarded in favor of something that hasn't worked as well (in my
> > > > opinion).  That is to say a few driver pmds (i40e and fm10k come to
> > > > mind) have their own tree that send pull requests to Thomas.
> > >
> > > Yes we tried this fine granularity and stated that it was not working
> > well.
> > > We are now using the bigger granularity that you describe below.
> > >
> > Ok, thats good, but that must be _very_ new.  Looking at your git tree, I
> > see no merge commits.  How are you pulling from those subtrees?
> > 
> 
> 
> Hi Neil,
> 
> It seems like the weight of consensus in around your proposal for further subtree maintainers and backups. If you don't mind I'll take your text and redraft it as a potential section on maintainership for a future Project Charter document. Or at least so that we have a documented maintainship process.
> 
Sure, that sounds good to me.  Thank you.

>  
> > > > We should be sharding that at a much higher granularity and using it
> > > > much more consistently.  That is to say, that we should have a
> > > > maintainer for all the ethernet pmds, and another for the crypto
> > > > pmds, another for the core eal layer, another for misc libraries
> > > > that have low patch volumes, etc.
> > >
> > > Yes we could open a tree for EAL and another one for the core libraries.
> > >
> > That could be worthwhile.  Lets see how the net and crypto subtrees work
> > out (assuming again that these trees are newly founded)
> 
> Could we define some of the potential subtrees now and look to introduce them in the this release cycle? EAL and the Core libs, as suggested by Thomas, seem like 2 obvious ones.
> 
Sure, I'd suggest the following:
	* net-pmds:
		- all network pmds located under drivers/net
		- librte_net
		- libtre_ether
		- librte_ip_frag
		- librte_pdump
		- librte_port
	* crypto-pmds:
		- all crypto pmds located under drivers/crypto
		- librte_cryptodev
	* eal:
		- librte_eal
	* core:
		- librte_cfgfile
		- librte_cmdline
		- librte_compat
		- librte_kvargs
		- librte_kni
		- librte_compat
	* misc:
		- librte_acl
		- librte_distributor
		- librte_hash
		- librte_jobstats
		- librte_lpm
		- librte_meter
		- librte_pipeline
		- librte_power
		- librte_reorder
		- librte_ring
		- librte_sched
		- librte_table
		- librte_timer
		- librte_vhost

Thats just a rough stab mind, but perhaps it would get the ball rolling.  I'd be
willing to take maintainership of one of these subtrees if there is consensus
around my doing so.

> 
> > 
> > > > Each of those subdivisions should have their own list to communicate
> > > > on, and each should have a tree that integrates patches for their
> > > > own subsystem, and they should on a regular cycle send pull requests
> > > > to Thomas.
> > >
> > > Yes I think it is now a good idea to split the mailing list traffic,
> > > at least for netdev and cryptodev.
> 
> I'd prefer not to have split dev lists, for now at least. We can reevaluate that again in a few months though.
> 
So, if thats a deal breaker, we can talk about it, but I need to point out that
multiple subtrees with a single development list creates a wide range of
problems that will later give the perception that creating subtrees in the first
place wasn't worthwhile.  If you have multiple maintainers working on multiple
subtrees in parallel with the expectation that we will all merge to thomas's
tree during the devel cycle, then those maintainers absolutely must have a way
to identify which patches are destined for their tree, and which can be ignored.
To not have that invites duplicated work at best (as multiple maintainers apply
the same patch), and serious merge breakage at worst (as multiple massaged patch
applications conflict during the merge).  We can manage it with tagged
submissions, but relying on developers to use those tags consistently is very
hit or miss.  It would be much more desireable to split development into lists
that correspond with the subtrees, so that posting to a list has a 1:1
correlation with the tree it was intended to be merged to.  As I noted
previously, subscribing to several lists and treating them as a single inbox is
a trivial operation.

> 
> > >
> > 
> > > > B) Designate alternates to serve as backups for the maintainer when
> > > > they are unavailable.  This provides high-availablility, and sounds
> > > > very much like your proposal, but in the interests of clarity, there
> > > > is still a single maintainer at any one time, it just may change to
> > > > ensure the continued merging of patches, if the primary maintainer
> > isn't available.
> > > > Ideally however, those backup alternates arent needed, because most
> > > > of the primary maintainers work in merging pull requests, which are
> > > > done based on the trust of the submaintainer, and done during a very
> > > > limited window of time.  This also partially addreses multi-vendor
> > > > fairness if your subtree maintainers come from multiple participating
> > companies.
> 
> 
> +1 on this apart from the limited merge window (for reasons similar to Thomas).
> 
Yeah, apparently I misspoke here.  I never meant to indicate that the merge
window could be smaller/shorter/what have you.  I only meant to indicate that,
if Thomas's primary role is merging pull requests from subtrees, then his job
takes much less time than it otherwise would if he's applying individual
patches, which in turn makes the need for backups (due to
vacation/illness/whatever), less of a pressing issue.


> Should we have a call for volunteers for backup, on master and the sub-trees, followed by a simple +1 from community members to endorse them?
> 
I would think that we could merge the roles into one.  That is to say, if we
assume that sub-tree maintainers are implicitly trusted by Thomas to send good
pul requests, then they also would make good backups for him.  i.e. if you
volunteer to be a sub-tree maintainer, you get added to the list of master tree
backups, in some order to be determined by Thomas should he need to take an
absence for any reason.  Just my $0.02 on that, we can organize that however you
would like.

I'd be willing to be a subtree maintainer.  I'll handle any tree, once we have
consensus on the segmentation, and list relationships.

Regards
Neil
> 
> John
> 
> 
> 

Re: Proposal for a new Committer model

[Neil Horman]

On Wed, Nov 23, 2016 at 02:01:44PM +0000, Ferruh Yigit wrote:
> On 11/23/2016 1:48 PM, Neil Horman wrote:
> > On Tue, Nov 22, 2016 at 08:56:23PM +0000, Ferruh Yigit wrote:
> >> On 11/22/2016 7:52 PM, Neil Horman wrote:
> >>> On Mon, Nov 21, 2016 at 09:52:41AM +0100, Thomas Monjalon wrote:
> >>>> 2016-11-18 13:09, Neil Horman:
> >>>>> A) Further promote subtree maintainership.  This was a conversation that I
> >>>>> proposed some time ago, but my proposed granularity was discarded in favor
> >>>>> of something that hasn't worked as well (in my opinion).  That is to say a
> >>>>> few driver pmds (i40e and fm10k come to mind) have their own tree that
> >>>>> send pull requests to Thomas.
> >>>>
> >>>> Yes we tried this fine granularity and stated that it was not working well.
> >>>> We are now using the bigger granularity that you describe below.
> >>>>
> >>> Ok, thats good, but that must be _very_ new.  Looking at your git tree, I see no
> >>> merge commits.  How are you pulling from those subtrees?
> >>
> >> next-net tree is active for last three releases.
> >>
> > What!?  What is the purpose of holding patches in a subtree for multiple
> > releases?  
> 
> :) Of course not holding them in the sub-tree.
> 
Ok, glad that we're on the same page.

> Briefly, process is:
> - sub-tree gets patches during merge window
> - sub-tree first merged into main tree in -rc1 and later in -r2
> 
> next-net tree is actively in use for last three releases, and driver/net
> patches delegated to this tree. You can see different commiters in main
> tree.
> 
> > If a given changeset isn't ready for merge to Thomas tree the people
> > working on it should clone the subtree to some place they can all collaborate on
> > it.  Once it goes into a subtree there needs to be a defined workflow to get it
> > into the canonical tree that Thomas maintains on a regular, short time frame.
> > to do less is to confuse the process for everyone involved, and slow people
> > down, rather than accelerate their work.
> > 
> >> I guess following is the first commit to the sub-tree:
> >> http://dpdk.org/ml/archives/dev/2016-February/032580.html
> >>
> >> sub-trees rebase on top of main tree regularly, that is why there is no
> >> merge commit.
> >>
> > I'm not asking about merge commits in the sub-tree, I'm asking about merge
> > commits in thomas's tree.
> 
> Same, talking about Thomas' tree.
> 
> >  There should be a merge commit every time he pulls
> > from a sub-tree (unless its a fast-forward I think, but with multiple subtrees
> > and commits going to thomas directly, that should never really happen).  
> 
> That is what happening. Since sub-tree's rebase on top of main tree,
> when Thomas merges, it is just plain fast-forward. So it is allowed to
> re-write to history in sub-trees.
> 
ok, I see what you're saying here, but I still don't see how this results in no
merge commits.  From what I can see we have at least 4 subtrees (next-crypto,
next-net, next-eventdev, next-virtio).  If you rebase all on lastest version of
thomas's tree, and then they all make changes and send a pull request, the first
to be merged will, by definition be a fast forward.  The remaining three
however, cannot be, because the prior merge has advanced the HEAD commit in
Thomas's tree such that its divergent from the subsequent tree to be merged.  So there
should be some merge commits in Thomas's tree.

The only way I see around that, is if the merges are serialized (i.e. if you
provide an order to the subtrees and each subtree rebases from thomas prior to
applying its patches, then merges back to thomas's tree).  Thats rather defeating
of the purpose of parallel trees though.  You can all rebase, but then you
operate independently of each other.  Thats how we realize a speedup here


Neil

Re: Proposal for a new Committer model

[Yuanhan Liu]

On Wed, Nov 23, 2016 at 09:11:54AM -0500, Neil Horman wrote:
> > Could we define some of the potential subtrees now and look to introduce them in the this release cycle? EAL and the Core libs, as suggested by Thomas, seem like 2 obvious ones.
> > 
> Sure, I'd suggest the following:

I would pull the git history to see which components are in
active status in last release (or even, in last few release).
And try to make a sub-tree if corresponding component is hot.

# the 2nd volume shows how many patches prefixed with a related component
[yliu@yliu-dev ~/dpdk]$ git log --oneline v16.07..v16.11 | awk '{print $2}' | \
		        sort | uniq -c  | sort -nr | head -30 | nl
     1       52 doc:
     2       40 net/ixgbe/base:
     3       38 app/test:
     4       37 kni:
     5       27 vhost:
     6       27 net/virtio:
     7       27 net/mlx5:
     8       26 app/testpmd:
     9       25 net/i40e:
    10       23 net/pcap:
    11       22 net/bnxt:
    12       20 net/enic:
    13       18 net/qede:
    14       17 net/thunderx:
    15       16 net/qede/base:
    16       16 eal:
    17       15 net/ixgbe:
    18       14 net:
    19       14 crypto/qat:
    20       13 scripts:
    21       13 net/bnx2x:
    22       12 net/i40e/base:
    23       12 examples/ipsec-secgw:
    24       11 mbuf:
    25       11 hash:
    26       10 lib:
    27       10 examples/ip_pipeline:
    28       10 ethdev:
    29        9 pci:
    30        7 net/vmxnet3:
    ...
    46        3 pdump:
    47        3 net/virtio_user:
    48        3 net/ring:
    49        3 net/nfp:
    50        3 net/mlx:
    51        3 net/ena:
    52        3 net/e1000:
    53        3 net/bonding:
    ...
    56        2 sched:
    57        2 port:
    ...
    65        1 timer:
    66        1 remove
    67        1 pmdinfogen:
    68        1 net/igb:
    69        1 net/enic/base:
    70        1 meter:
    ...
    84        1 cfgfile:
    85        1 app/procinfo:
    86        1 app/proc_info:
    87        1 acl:

Something obvious is that:

- "doc" deserves a sub-tree, and John is a perfect committer for that
  if he's willing to.

- generally, I'd agree with Neil that most (if not all) pmds may need
  a sub-tree. While, some others may not, for example, net/ring, net/pcap.

  For those non-active pmds, I think it's okay to let the generic
  pmd committer to cover them.

- it's not that wise to me to list all the components we have so far
  and make a sub-tree for each of them.

  For example, some components like librte_{port, pdump, cfgfile, acl,
  and etc} just have few (or even, just one) commits in last release.
  It makes no sense to me to introduce a tree for each of them.

Another thought is we could also create sub-trees based on category
but not on components like Neil suggested, especially that EAL looks
way too big to be maintained in one tree. Instead, it could be something
like:

- a tree for BSD

- a tree for ARM (and some other trees for other platforms)

- a tree for mem related (mempool, mbuf, hugepage, etc)

- a tree for BUS

- ...


Last but not the least, I think it's general good to have more and
more trees in the end. But I don't think it's a good idea to go
radically and create all those trees once (say in one release).

Something I would like to suggest is one or two (or a bit more) at
a release. For example, if I remember them well, we have next-net
tree at 16.04, and next-virtio (including vhost) at 16.07, and a
recent one, next-crypto at 16.11.

	--yliu


> 	* net-pmds:
> 		- all network pmds located under drivers/net
> 		- librte_net
> 		- libtre_ether
> 		- librte_ip_frag
> 		- librte_pdump
> 		- librte_port
> 	* crypto-pmds:
> 		- all crypto pmds located under drivers/crypto
> 		- librte_cryptodev
> 	* eal:
> 		- librte_eal
> 	* core:
> 		- librte_cfgfile
> 		- librte_cmdline
> 		- librte_compat
> 		- librte_kvargs
> 		- librte_kni
> 		- librte_compat
> 	* misc:
> 		- librte_acl
> 		- librte_distributor
> 		- librte_hash
> 		- librte_jobstats
> 		- librte_lpm
> 		- librte_meter
> 		- librte_pipeline
> 		- librte_power
> 		- librte_reorder
> 		- librte_ring
> 		- librte_sched
> 		- librte_table
> 		- librte_timer
> 		- librte_vhost
> 
> Thats just a rough stab mind, but perhaps it would get the ball rolling.  I'd be
> willing to take maintainership of one of these subtrees if there is consensus
> around my doing so.

Re: Proposal for a new Committer model

[Ferruh Yigit]

On 11/23/2016 3:33 PM, Neil Horman wrote:
> On Wed, Nov 23, 2016 at 02:01:44PM +0000, Ferruh Yigit wrote:
>> On 11/23/2016 1:48 PM, Neil Horman wrote:
>>> On Tue, Nov 22, 2016 at 08:56:23PM +0000, Ferruh Yigit wrote:
>>>> On 11/22/2016 7:52 PM, Neil Horman wrote:
>>>>> On Mon, Nov 21, 2016 at 09:52:41AM +0100, Thomas Monjalon wrote:
>>>>>> 2016-11-18 13:09, Neil Horman:
>>>>>>> A) Further promote subtree maintainership.  This was a conversation that I
>>>>>>> proposed some time ago, but my proposed granularity was discarded in favor
>>>>>>> of something that hasn't worked as well (in my opinion).  That is to say a
>>>>>>> few driver pmds (i40e and fm10k come to mind) have their own tree that
>>>>>>> send pull requests to Thomas.
>>>>>>
>>>>>> Yes we tried this fine granularity and stated that it was not working well.
>>>>>> We are now using the bigger granularity that you describe below.
>>>>>>
>>>>> Ok, thats good, but that must be _very_ new.  Looking at your git tree, I see no
>>>>> merge commits.  How are you pulling from those subtrees?
>>>>
>>>> next-net tree is active for last three releases.
>>>>
>>> What!?  What is the purpose of holding patches in a subtree for multiple
>>> releases?  
>>
>> :) Of course not holding them in the sub-tree.
>>
> Ok, glad that we're on the same page.
> 
>> Briefly, process is:
>> - sub-tree gets patches during merge window
>> - sub-tree first merged into main tree in -rc1 and later in -r2
>>
>> next-net tree is actively in use for last three releases, and driver/net
>> patches delegated to this tree. You can see different commiters in main
>> tree.
>>
>>> If a given changeset isn't ready for merge to Thomas tree the people
>>> working on it should clone the subtree to some place they can all collaborate on
>>> it.  Once it goes into a subtree there needs to be a defined workflow to get it
>>> into the canonical tree that Thomas maintains on a regular, short time frame.
>>> to do less is to confuse the process for everyone involved, and slow people
>>> down, rather than accelerate their work.
>>>
>>>> I guess following is the first commit to the sub-tree:
>>>> http://dpdk.org/ml/archives/dev/2016-February/032580.html
>>>>
>>>> sub-trees rebase on top of main tree regularly, that is why there is no
>>>> merge commit.
>>>>
>>> I'm not asking about merge commits in the sub-tree, I'm asking about merge
>>> commits in thomas's tree.
>>
>> Same, talking about Thomas' tree.
>>
>>>  There should be a merge commit every time he pulls
>>> from a sub-tree (unless its a fast-forward I think, but with multiple subtrees
>>> and commits going to thomas directly, that should never really happen).  
>>
>> That is what happening. Since sub-tree's rebase on top of main tree,
>> when Thomas merges, it is just plain fast-forward. So it is allowed to
>> re-write to history in sub-trees.
>>
> ok, I see what you're saying here, but I still don't see how this results in no
> merge commits.  From what I can see we have at least 4 subtrees (next-crypto,
> next-net, next-eventdev, next-virtio).  If you rebase all on lastest version of
> thomas's tree, and then they all make changes and send a pull request, the first
> to be merged will, by definition be a fast forward.  The remaining three
> however, cannot be, because the prior merge has advanced the HEAD commit in
> Thomas's tree such that its divergent from the subsequent tree to be merged.  So there
> should be some merge commits in Thomas's tree.

This is simple indeed, all can do fast-forward, because all sub-trees
touch to different files.

Currently:
next-net: drivers/net/* [except virtio and vhost]
next-crypto: drivers/crypto/*
next-virtio: drivers/net/virtio/*, drivers/net/vhost/*

Common files are in main tree, when all rebased on top of it, they all
can be merged as fast-forward.

> 
> The only way I see around that, is if the merges are serialized (i.e. if you
> provide an order to the subtrees and each subtree rebases from thomas prior to
> applying its patches, then merges back to thomas's tree).  Thats rather defeating
> of the purpose of parallel trees though.  You can all rebase, but then you
> operate independently of each other.  Thats how we realize a speedup here
> 
> 
> Neil
> 

Re: Proposal for a new Committer model

[Neil Horman]

On Wed, Nov 23, 2016 at 04:21:00PM +0000, Ferruh Yigit wrote:
> On 11/23/2016 3:33 PM, Neil Horman wrote:
> > On Wed, Nov 23, 2016 at 02:01:44PM +0000, Ferruh Yigit wrote:
> >> On 11/23/2016 1:48 PM, Neil Horman wrote:
> >>> On Tue, Nov 22, 2016 at 08:56:23PM +0000, Ferruh Yigit wrote:
> >>>> On 11/22/2016 7:52 PM, Neil Horman wrote:
> >>>>> On Mon, Nov 21, 2016 at 09:52:41AM +0100, Thomas Monjalon wrote:
> >>>>>> 2016-11-18 13:09, Neil Horman:
> >>>>>>> A) Further promote subtree maintainership.  This was a conversation that I
> >>>>>>> proposed some time ago, but my proposed granularity was discarded in favor
> >>>>>>> of something that hasn't worked as well (in my opinion).  That is to say a
> >>>>>>> few driver pmds (i40e and fm10k come to mind) have their own tree that
> >>>>>>> send pull requests to Thomas.
> >>>>>>
> >>>>>> Yes we tried this fine granularity and stated that it was not working well.
> >>>>>> We are now using the bigger granularity that you describe below.
> >>>>>>
> >>>>> Ok, thats good, but that must be _very_ new.  Looking at your git tree, I see no
> >>>>> merge commits.  How are you pulling from those subtrees?
> >>>>
> >>>> next-net tree is active for last three releases.
> >>>>
> >>> What!?  What is the purpose of holding patches in a subtree for multiple
> >>> releases?  
> >>
> >> :) Of course not holding them in the sub-tree.
> >>
> > Ok, glad that we're on the same page.
> > 
> >> Briefly, process is:
> >> - sub-tree gets patches during merge window
> >> - sub-tree first merged into main tree in -rc1 and later in -r2
> >>
> >> next-net tree is actively in use for last three releases, and driver/net
> >> patches delegated to this tree. You can see different commiters in main
> >> tree.
> >>
> >>> If a given changeset isn't ready for merge to Thomas tree the people
> >>> working on it should clone the subtree to some place they can all collaborate on
> >>> it.  Once it goes into a subtree there needs to be a defined workflow to get it
> >>> into the canonical tree that Thomas maintains on a regular, short time frame.
> >>> to do less is to confuse the process for everyone involved, and slow people
> >>> down, rather than accelerate their work.
> >>>
> >>>> I guess following is the first commit to the sub-tree:
> >>>> http://dpdk.org/ml/archives/dev/2016-February/032580.html
> >>>>
> >>>> sub-trees rebase on top of main tree regularly, that is why there is no
> >>>> merge commit.
> >>>>
> >>> I'm not asking about merge commits in the sub-tree, I'm asking about merge
> >>> commits in thomas's tree.
> >>
> >> Same, talking about Thomas' tree.
> >>
> >>>  There should be a merge commit every time he pulls
> >>> from a sub-tree (unless its a fast-forward I think, but with multiple subtrees
> >>> and commits going to thomas directly, that should never really happen).  
> >>
> >> That is what happening. Since sub-tree's rebase on top of main tree,
> >> when Thomas merges, it is just plain fast-forward. So it is allowed to
> >> re-write to history in sub-trees.
> >>
> > ok, I see what you're saying here, but I still don't see how this results in no
> > merge commits.  From what I can see we have at least 4 subtrees (next-crypto,
> > next-net, next-eventdev, next-virtio).  If you rebase all on lastest version of
> > thomas's tree, and then they all make changes and send a pull request, the first
> > to be merged will, by definition be a fast forward.  The remaining three
> > however, cannot be, because the prior merge has advanced the HEAD commit in
> > Thomas's tree such that its divergent from the subsequent tree to be merged.  So there
> > should be some merge commits in Thomas's tree.
> 
> This is simple indeed, all can do fast-forward, because all sub-trees
> touch to different files.
> 
> Currently:
> next-net: drivers/net/* [except virtio and vhost]
> next-crypto: drivers/crypto/*
> next-virtio: drivers/net/virtio/*, drivers/net/vhost/*
> 
> Common files are in main tree, when all rebased on top of it, they all
> can be merged as fast-forward.
> 
Fast forward-ability isn't determined by only touching orthogonal files, its
completely dependent on the relative HEAD and base pointers of each branch.  If
you send a pull request for your subtree on branch A from commit B to commit C,
the merge will resolve as a fast forward, if and only if, the HEAD commit of the
branch that the receiver is pulling to matches the base commit of the branch
being pulled (that is to say, commit B). If the receiving branch has any
additional commits on top of it, then the branches have diverged (even if the
applied patches don't touch a common set of files), and as a result a merge
commit is created (which may be empty if there are no conflicts).  Given that
there are no merge commits in Thomas tree, I conclude that you are either:

1) Not sending pull requests to Thomas - ie, just sending him a series of
patches to apply

2) Rebasing your work immediately prior to sending a pull request, resulting in
a fast-forward

Can either you or thomas provide some detail as to how you are doing patch
management between trees (details of the commands you use are what I would be
interested in). It sounds to me like there may be some optimization to be made
here before we even make changes to the subtrees.

Best
Neil

> > 
> > The only way I see around that, is if the merges are serialized (i.e. if you
> > provide an order to the subtrees and each subtree rebases from thomas prior to
> > applying its patches, then merges back to thomas's tree).  Thats rather defeating
> > of the purpose of parallel trees though.  You can all rebase, but then you
> > operate independently of each other.  Thats how we realize a speedup here
> > 
> > 
> > Neil
> > 
> 
> 

Re: Proposal for a new Committer model

[Neil Horman]

On Wed, Nov 23, 2016 at 11:41:20PM +0800, Yuanhan Liu wrote:
> On Wed, Nov 23, 2016 at 09:11:54AM -0500, Neil Horman wrote:
> > > Could we define some of the potential subtrees now and look to introduce them in the this release cycle? EAL and the Core libs, as suggested by Thomas, seem like 2 obvious ones.
> > > 
> > Sure, I'd suggest the following:
> 
> I would pull the git history to see which components are in
> active status in last release (or even, in last few release).
> And try to make a sub-tree if corresponding component is hot.
> 
> # the 2nd volume shows how many patches prefixed with a related component
> [yliu@yliu-dev ~/dpdk]$ git log --oneline v16.07..v16.11 | awk '{print $2}' | \
> 		        sort | uniq -c  | sort -nr | head -30 | nl
>      1       52 doc:
>      2       40 net/ixgbe/base:
>      3       38 app/test:
>      4       37 kni:
>      5       27 vhost:
>      6       27 net/virtio:
>      7       27 net/mlx5:
>      8       26 app/testpmd:
>      9       25 net/i40e:
>     10       23 net/pcap:
>     11       22 net/bnxt:
>     12       20 net/enic:
>     13       18 net/qede:
>     14       17 net/thunderx:
>     15       16 net/qede/base:
>     16       16 eal:
>     17       15 net/ixgbe:
>     18       14 net:
>     19       14 crypto/qat:
>     20       13 scripts:
>     21       13 net/bnx2x:
>     22       12 net/i40e/base:
>     23       12 examples/ipsec-secgw:
>     24       11 mbuf:
>     25       11 hash:
>     26       10 lib:
>     27       10 examples/ip_pipeline:
>     28       10 ethdev:
>     29        9 pci:
>     30        7 net/vmxnet3:
>     ...
>     46        3 pdump:
>     47        3 net/virtio_user:
>     48        3 net/ring:
>     49        3 net/nfp:
>     50        3 net/mlx:
>     51        3 net/ena:
>     52        3 net/e1000:
>     53        3 net/bonding:
>     ...
>     56        2 sched:
>     57        2 port:
>     ...
>     65        1 timer:
>     66        1 remove
>     67        1 pmdinfogen:
>     68        1 net/igb:
>     69        1 net/enic/base:
>     70        1 meter:
>     ...
>     84        1 cfgfile:
>     85        1 app/procinfo:
>     86        1 app/proc_info:
>     87        1 acl:
> 
> Something obvious is that:
> 
> - "doc" deserves a sub-tree, and John is a perfect committer for that
>   if he's willing to.
> 
> - generally, I'd agree with Neil that most (if not all) pmds may need
>   a sub-tree. While, some others may not, for example, net/ring, net/pcap.
> 
No, thats the opposite of what I think.  I think all net pmds should flow
through a single subtree, all crypto pmds through another, etc.

>   For those non-active pmds, I think it's okay to let the generic
>   pmd committer to cover them.
> 
Not sure what you're getting at here.  Low volume pms (or any library) can still
go through a subtree.  The goal is to fragmet the commit work so one person
doesn't have to do it all.

> - it's not that wise to me to list all the components we have so far
>   and make a sub-tree for each of them.
> 
I think you misunderstood the organization of my last note.  I agree with you
here.  When I listed the core and listed several libraries under it, my intent
was to create a core subtree that accepted patches for all of those libraries.

>   For example, some components like librte_{port, pdump, cfgfile, acl,
>   and etc} just have few (or even, just one) commits in last release.
>   It makes no sense to me to introduce a tree for each of them.
> 
Yes, this is what I was saying in my last note.

> Another thought is we could also create sub-trees based on category
> but not on components like Neil suggested, especially that EAL looks
> way too big to be maintained in one tree. Instead, it could be something
> like:
> 
> - a tree for BSD
> 
This gets tricky, because then several libraries will be covered by multiple
trees, and that leads to merge conflicts.

> - a tree for ARM (and some other trees for other platforms)
> 
> - a tree for mem related (mempool, mbuf, hugepage, etc)
> 
> - a tree for BUS
> 
> - ...
> 
> 
> Last but not the least, I think it's general good to have more and
> more trees in the end. But I don't think it's a good idea to go
> radically and create all those trees once (say in one release).
> 
> Something I would like to suggest is one or two (or a bit more) at
> a release. For example, if I remember them well, we have next-net
> tree at 16.04, and next-virtio (including vhost) at 16.07, and a
> recent one, next-crypto at 16.11.
> 
I'm not sure what you mean by this.  -next trees rather by definition should e
rebased on a release to start at the head of thomas's tree and add commits from
there based on their subject area.

Neil

> 	--yliu
> 
> 
> > 	* net-pmds:
> > 		- all network pmds located under drivers/net
> > 		- librte_net
> > 		- libtre_ether
> > 		- librte_ip_frag
> > 		- librte_pdump
> > 		- librte_port
> > 	* crypto-pmds:
> > 		- all crypto pmds located under drivers/crypto
> > 		- librte_cryptodev
> > 	* eal:
> > 		- librte_eal
> > 	* core:
> > 		- librte_cfgfile
> > 		- librte_cmdline
> > 		- librte_compat
> > 		- librte_kvargs
> > 		- librte_kni
> > 		- librte_compat
> > 	* misc:
> > 		- librte_acl
> > 		- librte_distributor
> > 		- librte_hash
> > 		- librte_jobstats
> > 		- librte_lpm
> > 		- librte_meter
> > 		- librte_pipeline
> > 		- librte_power
> > 		- librte_reorder
> > 		- librte_ring
> > 		- librte_sched
> > 		- librte_table
> > 		- librte_timer
> > 		- librte_vhost
> > 
> > Thats just a rough stab mind, but perhaps it would get the ball rolling.  I'd be
> > willing to take maintainership of one of these subtrees if there is consensus
> > around my doing so.
> 

Re: Proposal for a new Committer model

[Yuanhan Liu]

On Wed, Nov 23, 2016 at 03:19:19PM -0500, Neil Horman wrote:
> On Wed, Nov 23, 2016 at 11:41:20PM +0800, Yuanhan Liu wrote:
> > On Wed, Nov 23, 2016 at 09:11:54AM -0500, Neil Horman wrote:
> > > > Could we define some of the potential subtrees now and look to introduce them in the this release cycle? EAL and the Core libs, as suggested by Thomas, seem like 2 obvious ones.
> > > > 
> > > Sure, I'd suggest the following:
> > 
> > I would pull the git history to see which components are in
> > active status in last release (or even, in last few release).
> > And try to make a sub-tree if corresponding component is hot.
> > 
> > # the 2nd volume shows how many patches prefixed with a related component
> > [yliu@yliu-dev ~/dpdk]$ git log --oneline v16.07..v16.11 | awk '{print $2}' | \
> > 		        sort | uniq -c  | sort -nr | head -30 | nl
> >      1       52 doc:
> >      2       40 net/ixgbe/base:
> >      3       38 app/test:
> >      4       37 kni:
> >      5       27 vhost:
> >      6       27 net/virtio:
> >      7       27 net/mlx5:
> >      8       26 app/testpmd:
> >      9       25 net/i40e:
> >     10       23 net/pcap:
> >     11       22 net/bnxt:
> >     12       20 net/enic:
> >     13       18 net/qede:
> >     14       17 net/thunderx:
> >     15       16 net/qede/base:
> >     16       16 eal:
> >     17       15 net/ixgbe:
> >     18       14 net:
> >     19       14 crypto/qat:
> >     20       13 scripts:
> >     21       13 net/bnx2x:
> >     22       12 net/i40e/base:
> >     23       12 examples/ipsec-secgw:
> >     24       11 mbuf:
> >     25       11 hash:
> >     26       10 lib:
> >     27       10 examples/ip_pipeline:
> >     28       10 ethdev:
> >     29        9 pci:
> >     30        7 net/vmxnet3:
> >     ...
> >     46        3 pdump:
> >     47        3 net/virtio_user:
> >     48        3 net/ring:
> >     49        3 net/nfp:
> >     50        3 net/mlx:
> >     51        3 net/ena:
> >     52        3 net/e1000:
> >     53        3 net/bonding:
> >     ...
> >     56        2 sched:
> >     57        2 port:
> >     ...
> >     65        1 timer:
> >     66        1 remove
> >     67        1 pmdinfogen:
> >     68        1 net/igb:
> >     69        1 net/enic/base:
> >     70        1 meter:
> >     ...
> >     84        1 cfgfile:
> >     85        1 app/procinfo:
> >     86        1 app/proc_info:
> >     87        1 acl:
> > 
> > Something obvious is that:
> > 
> > - "doc" deserves a sub-tree, and John is a perfect committer for that
> >   if he's willing to.
> > 
> > - generally, I'd agree with Neil that most (if not all) pmds may need
> >   a sub-tree. While, some others may not, for example, net/ring, net/pcap.
> > 
> No, thats the opposite of what I think.  I think all net pmds should flow
> through a single subtree, all crypto pmds through another, etc.

I misunderstood it. I was think you were suggesting to create a sub-tree
for most (or all) pmds. Some of my comments didn't apply then.

But yes, we have already done that: we have next-net and next-crypto.

> >   For those non-active pmds, I think it's okay to let the generic
> >   pmd committer to cover them.
> > 
> Not sure what you're getting at here.  Low volume pms (or any library) can still
> go through a subtree.  The goal is to fragmet the commit work so one person
> doesn't have to do it all.
> 
> > - it's not that wise to me to list all the components we have so far
> >   and make a sub-tree for each of them.
> > 
> I think you misunderstood the organization of my last note.  I agree with you
> here.  When I listed the core and listed several libraries under it, my intent
> was to create a core subtree that accepted patches for all of those libraries.
> 
> >   For example, some components like librte_{port, pdump, cfgfile, acl,
> >   and etc} just have few (or even, just one) commits in last release.
> >   It makes no sense to me to introduce a tree for each of them.
> > 
> Yes, this is what I was saying in my last note.
> 
> > Another thought is we could also create sub-trees based on category
> > but not on components like Neil suggested, especially that EAL looks
> > way too big to be maintained in one tree. Instead, it could be something
> > like:
> > 
> > - a tree for BSD
> > 
> This gets tricky, because then several libraries will be covered by multiple
> trees, and that leads to merge conflicts.

If we go that way, I meant a sub-sub-tree under EAL sub-tree. And conflicts
is almost impossible to avoid when we have multiple trees.

> > - a tree for ARM (and some other trees for other platforms)
> > 
> > - a tree for mem related (mempool, mbuf, hugepage, etc)
> > 
> > - a tree for BUS
> > 
> > - ...
> > 
> > 
> > Last but not the least, I think it's general good to have more and
> > more trees in the end. But I don't think it's a good idea to go
> > radically and create all those trees once (say in one release).
> > 
> > Something I would like to suggest is one or two (or a bit more) at
> > a release. For example, if I remember them well, we have next-net
> > tree at 16.04, and next-virtio (including vhost) at 16.07, and a
> > recent one, next-crypto at 16.11.
> > 
> I'm not sure what you mean by this.

I meant we already add more and more trees, from 0 and 1, and then
from 1 to 3 (and more), a bit slowly but not radically.

>  -next trees rather by definition should e
> rebased on a release to start at the head of thomas's tree and add commits from
> there based on their subject area.

Yep, and that's we are doing.

And maybe we could revisit your suggested list:

> > > 	* net-pmds:
> > > 		- all network pmds located under drivers/net
> > > 		- librte_net
> > > 		- libtre_ether
> > > 		- librte_ip_frag
> > > 		- librte_pdump
> > > 		- librte_port
> > > 	* crypto-pmds:
> > > 		- all crypto pmds located under drivers/crypto
> > > 		- librte_cryptodev

We already have the two.

> > > 	* eal:
> > > 		- librte_eal

I think EAL deserves to have a sub-tree.

> > > 	* core:
> > > 		- librte_cfgfile
> > > 		- librte_cmdline
> > > 		- librte_compat
> > > 		- librte_kvargs
> > > 		- librte_kni
> > > 		- librte_compat
> > > 	* misc:

It may be vague to define which belongs to core and which belongs to
misc. It might be better to have a lib sub-tree, to hold all others
that don't belong to other sub-trees.

	--yliu

> > > 		- librte_acl
> > > 		- librte_distributor
> > > 		- librte_hash
> > > 		- librte_jobstats
> > > 		- librte_lpm
> > > 		- librte_meter
> > > 		- librte_pipeline
> > > 		- librte_power
> > > 		- librte_reorder
> > > 		- librte_ring
> > > 		- librte_sched
> > > 		- librte_table
> > > 		- librte_timer
> > > 		- librte_vhost
> > > 
> > > Thats just a rough stab mind, but perhaps it would get the ball rolling.  I'd be
> > > willing to take maintainership of one of these subtrees if there is consensus
> > > around my doing so.
> > 

Re: Proposal for a new Committer model

[Thomas Monjalon]

2016-11-23 15:13, Neil Horman:
> Can either you or thomas provide some detail as to how you are doing patch
> management between trees (details of the commands you use are what I would be
> interested in). It sounds to me like there may be some optimization to be made
> here before we even make changes to the subtrees.

Until now, we preferred avoiding merge commits.
That's why I rebase sub-trees to integrate them in the mainline.
As Ferruh explained, it does not require more work because sub-trees are
regularly rebased anyway.
And I use a script to keep original committer name and date.

Hope it's clear now

Re: Proposal for a new Committer model

[Neil Horman]

On Thu, Nov 24, 2016 at 10:17:09AM +0100, Thomas Monjalon wrote:
> 2016-11-23 15:13, Neil Horman:
> > Can either you or thomas provide some detail as to how you are doing patch
> > management between trees (details of the commands you use are what I would be
> > interested in). It sounds to me like there may be some optimization to be made
> > here before we even make changes to the subtrees.
> 
> Until now, we preferred avoiding merge commits.
> That's why I rebase sub-trees to integrate them in the mainline.
> As Ferruh explained, it does not require more work because sub-trees are
> regularly rebased anyway.
> And I use a script to keep original committer name and date.
> 
> Hope it's clear now
> 
It is clear, but I would make the assertion that performing that rebase
yourselves (and arguably having the subtree maintainers do that too), you are
undoing all the speedup that you would otherwise have gained.

To illustrate, from what it sounds like to me, this is your workflow, from the
moment that a merge window starts:

1) Each subtree maintainer syncrhonizes with your tree (either via a rebase, or
a simple pull and creation of a new next version branch), the specific method
can really be left up to the preference of the subtree maintainer

2) Developent procedes in parallel on multiple subtrees, and to your tree (for
those patches that have no other designated subtree)

3) At some point, you opt to merge subtrees to your trees.  For each tree you
ostensibly then need to:

3a) Inform the subtree maintainer (or otherwise record the point up to which you
decided to integrate)

3b) Grab a copy of that tree from the start to the end point (either via a
remote branch clone or some other method)

3c) Rebase the branch in 3(b) to the HEAD of your tree, correcting any errors
along the way

3d) preform a merge to your master branch, which, after 3(c), will by definition
be a fast forward branch

While thats a workable solution, it suffers from many drawbacks:

a) It looses your commit history.  That is to say, with a true merge, you record
several other relevant pieces of information, including:
	* The branch/tree that the series came from
	* The changes that needed to be made to make the series fit (conflict
	  tracking)
	* The sha1 sums of the commits (immutable changeset tracking).
   By rebasing you loose all of that, especially that last piece because the
rebase you preform changes your shasums.  With a non-merge, you keep all that
information properly.

b) You increase your error risk.  With a true merge, all the changes you need to
make to get a merge to resolve properly are contained in a single commit (the
merge commit),  providing easy review of your updates (not that they should
happen frequently).  By rebasing, you are left with potentially changed commits
that aren't trackable or easily comparable to the origional submission.

c) Perhaps most importantly, you serialize the entire process.  To follow the
above, you need to merge one branch at a time, rebasing each until it works
properly, then merging it.  With a true merge, you are afforded the opportunity
to merge any number of branches in parallel (see git-octopus-merge).


Apologies if this was discussed on list previously, but what was the percieved
advantage to avoiding merge commits?  It seems like alot of lost opportunity to
avoid something that is actually fairly informative in the toolchain.

Regards
Neil

Re: Proposal for a new Committer model

[Neil Horman]

On Thu, Nov 24, 2016 at 01:53:55PM +0800, Yuanhan Liu wrote:
> On Wed, Nov 23, 2016 at 03:19:19PM -0500, Neil Horman wrote:
> > On Wed, Nov 23, 2016 at 11:41:20PM +0800, Yuanhan Liu wrote:
> > > On Wed, Nov 23, 2016 at 09:11:54AM -0500, Neil Horman wrote:
> > > > > Could we define some of the potential subtrees now and look to introduce them in the this release cycle? EAL and the Core libs, as suggested by Thomas, seem like 2 obvious ones.
> > > > > 
> > > > Sure, I'd suggest the following:
> > > 
> > > I would pull the git history to see which components are in
> > > active status in last release (or even, in last few release).
> > > And try to make a sub-tree if corresponding component is hot.
> > > 
> > > # the 2nd volume shows how many patches prefixed with a related component
> > > [yliu@yliu-dev ~/dpdk]$ git log --oneline v16.07..v16.11 | awk '{print $2}' | \
> > > 		        sort | uniq -c  | sort -nr | head -30 | nl
> > >      1       52 doc:
> > >      2       40 net/ixgbe/base:
> > >      3       38 app/test:
> > >      4       37 kni:
> > >      5       27 vhost:
> > >      6       27 net/virtio:
> > >      7       27 net/mlx5:
> > >      8       26 app/testpmd:
> > >      9       25 net/i40e:
> > >     10       23 net/pcap:
> > >     11       22 net/bnxt:
> > >     12       20 net/enic:
> > >     13       18 net/qede:
> > >     14       17 net/thunderx:
> > >     15       16 net/qede/base:
> > >     16       16 eal:
> > >     17       15 net/ixgbe:
> > >     18       14 net:
> > >     19       14 crypto/qat:
> > >     20       13 scripts:
> > >     21       13 net/bnx2x:
> > >     22       12 net/i40e/base:
> > >     23       12 examples/ipsec-secgw:
> > >     24       11 mbuf:
> > >     25       11 hash:
> > >     26       10 lib:
> > >     27       10 examples/ip_pipeline:
> > >     28       10 ethdev:
> > >     29        9 pci:
> > >     30        7 net/vmxnet3:
> > >     ...
> > >     46        3 pdump:
> > >     47        3 net/virtio_user:
> > >     48        3 net/ring:
> > >     49        3 net/nfp:
> > >     50        3 net/mlx:
> > >     51        3 net/ena:
> > >     52        3 net/e1000:
> > >     53        3 net/bonding:
> > >     ...
> > >     56        2 sched:
> > >     57        2 port:
> > >     ...
> > >     65        1 timer:
> > >     66        1 remove
> > >     67        1 pmdinfogen:
> > >     68        1 net/igb:
> > >     69        1 net/enic/base:
> > >     70        1 meter:
> > >     ...
> > >     84        1 cfgfile:
> > >     85        1 app/procinfo:
> > >     86        1 app/proc_info:
> > >     87        1 acl:
> > > 
> > > Something obvious is that:
> > > 
> > > - "doc" deserves a sub-tree, and John is a perfect committer for that
> > >   if he's willing to.
> > > 
> > > - generally, I'd agree with Neil that most (if not all) pmds may need
> > >   a sub-tree. While, some others may not, for example, net/ring, net/pcap.
> > > 
> > No, thats the opposite of what I think.  I think all net pmds should flow
> > through a single subtree, all crypto pmds through another, etc.
> 
> I misunderstood it. I was think you were suggesting to create a sub-tree
> for most (or all) pmds. Some of my comments didn't apply then.
> 
> But yes, we have already done that: we have next-net and next-crypto.
> 
Yes, I'm speaking elsewhere in this thread on the topic of how that merge
process works, and how it can be improved.  But this granularity I think is
correct.

> > >   For those non-active pmds, I think it's okay to let the generic
> > >   pmd committer to cover them.
> > > 
> > Not sure what you're getting at here.  Low volume pms (or any library) can still
> > go through a subtree.  The goal is to fragmet the commit work so one person
> > doesn't have to do it all.
> > 
> > > - it's not that wise to me to list all the components we have so far
> > >   and make a sub-tree for each of them.
> > > 
> > I think you misunderstood the organization of my last note.  I agree with you
> > here.  When I listed the core and listed several libraries under it, my intent
> > was to create a core subtree that accepted patches for all of those libraries.
> > 
> > >   For example, some components like librte_{port, pdump, cfgfile, acl,
> > >   and etc} just have few (or even, just one) commits in last release.
> > >   It makes no sense to me to introduce a tree for each of them.
> > > 
> > Yes, this is what I was saying in my last note.
> > 
> > > Another thought is we could also create sub-trees based on category
> > > but not on components like Neil suggested, especially that EAL looks
> > > way too big to be maintained in one tree. Instead, it could be something
> > > like:
> > > 
> > > - a tree for BSD
> > > 
> > This gets tricky, because then several libraries will be covered by multiple
> > trees, and that leads to merge conflicts.
> 
> If we go that way, I meant a sub-sub-tree under EAL sub-tree. And conflicts
Hmmm.....I'm not sure we're quite there yet.  I'm not opposed to such a subtree
per-se, but I'd be somewhat curious to see how much BSD specific change or
linux-specific change was going into EAL before we broke that out separately.
Perhaps the best approach is to break out EAL and let that subtree maintainer
choose to further subdivide it as needed

> is almost impossible to avoid when we have multiple trees.
> 
I don't think thats particularly true.  Especially if the divisions are made on
a rough file basis.  A high percentage of  files should rarely if ever conflict.

Of course, there is always the possibility of a conflict, but we can minimize
it.

> > > - a tree for ARM (and some other trees for other platforms)
> > > 
> > > - a tree for mem related (mempool, mbuf, hugepage, etc)
> > > 
> > > - a tree for BUS
> > > 
> > > - ...
> > > 
> > > 
> > > Last but not the least, I think it's general good to have more and
> > > more trees in the end. But I don't think it's a good idea to go
> > > radically and create all those trees once (say in one release).
> > > 
> > > Something I would like to suggest is one or two (or a bit more) at
> > > a release. For example, if I remember them well, we have next-net
> > > tree at 16.04, and next-virtio (including vhost) at 16.07, and a
> > > recent one, next-crypto at 16.11.
> > > 
> > I'm not sure what you mean by this.
> 
> I meant we already add more and more trees, from 0 and 1, and then
> from 1 to 3 (and more), a bit slowly but not radically.
> 
Ah, you're commenting on the frequency with which we create new trees.  I tend
to think that we shouldn't really limit ourselves artificially there.  That is
to say, lets do what makes sense in terms of community organization, and in
terms of resources.  Creating new trees at release boundaries seems natural, but
lets not assert that there is a 'right' number of trees to create in any single
given release.

> >  -next trees rather by definition should e
> > rebased on a release to start at the head of thomas's tree and add commits from
> > there based on their subject area.
> 
> Yep, and that's we are doing.
> 
Ok

> And maybe we could revisit your suggested list:
> 
> > > > 	* net-pmds:
> > > > 		- all network pmds located under drivers/net
> > > > 		- librte_net
> > > > 		- libtre_ether
> > > > 		- librte_ip_frag
> > > > 		- librte_pdump
> > > > 		- librte_port
> > > > 	* crypto-pmds:
> > > > 		- all crypto pmds located under drivers/crypto
> > > > 		- librte_cryptodev
> 
> We already have the two.
> 
> > > > 	* eal:
> > > > 		- librte_eal
> 
> I think EAL deserves to have a sub-tree.
> 
> > > > 	* core:
> > > > 		- librte_cfgfile
> > > > 		- librte_cmdline
> > > > 		- librte_compat
> > > > 		- librte_kvargs
> > > > 		- librte_kni
> > > > 		- librte_compat
> > > > 	* misc:
> 
> It may be vague to define which belongs to core and which belongs to
> misc. It might be better to have a lib sub-tree, to hold all others
> that don't belong to other sub-trees.
> 
Sure, I'm not tied to any particular names, just trying to find a sane
subdivision more than anything else

Neil

Re: Proposal for a new Committer model

[Neil Horman]

On Fri, Nov 18, 2016 at 01:09:35PM -0500, Neil Horman wrote:
> On Thu, Nov 17, 2016 at 09:20:50AM +0000, Mcnamara, John wrote:
> > Repost from the moving at dpdk.org mailing list to get a wider audience.
> > Original thread: http://dpdk.org/ml/archives/moving/2016-November/000059.html
> > 
> > 
> > Hi,
> > 
> > I'd like to propose a change to the DPDK committer model. Currently we have one committer for the master branch of the DPDK project. 
> > 
> > One committer to master represents a single point of failure and at times can be inefficient. There is also no agreed cover for times when the committer is unavailable such as vacation, public holidays, etc. I propose that we change to a multi-committer model for the DPDK project. We should have three committers for each release that can commit changes to the master branch.
> >  
> > There are a number of benefits:
> >  
> > 1. Greater capacity to commit patches.
> > 2. No single points of failure - a committer should always be available if we have three.
> > 3. A more timely committing of patches. More committers should equal a faster turnaround - ideally, maintainers should also provide feedback on patches submitted within a 2-3 day period, as much as possible, to facilitate this. 
> > 4. It follows best practice in creating a successful multi-vendor community - to achieve this we must ensure there is a level playing field for all participants, no single person should be required to make all of the decisions on patches to be included in the release.  
> > 
> > Having multiple committers will require some degree of co-ordination but there are a number of other communities successfully following this model such as Apache, OVS, FD.io, OpenStack etc. so the approach is workable.
> > 
> > John
> 
> I agree that the problems you are attempting to address exist and are
> worth finding a solution for.  That said, I don't think the solution you
> are proposing is the ideal, or complete fix for any of the issues being
> addressed.
> 
> If I may, I'd like to ennumerate the issues I think you are trying to
> address based on your comments above, then make a counter-proposal for a
> solution:
> 
> Problems to address:
> 
> 1) high-availability - There is a desire to make sure that, when patches
> are proposed, they are integrated in a timely fashion.
> 
> 2) high-throughput - DPDK has a large volume of patches, more than one
> person can normally integrate.  There is a desire to shard that work such
> that it is handled by multiple individuals
> 
> 3) Multi-Vendor fairness - There is a desire for multiple vendors to feel
> as though the project tree maintainer isn't biased toward any individual
> vendor.
> 
> To solve these I would propose the following solution (which is simmilar
> to, but not quite identical, to yours).
> 
> A) Further promote subtree maintainership.  This was a conversation that I
> proposed some time ago, but my proposed granularity was discarded in favor
> of something that hasn't worked as well (in my opinion).  That is to say a
> few driver pmds (i40e and fm10k come to mind) have their own tree that
> send pull requests to Thomas.  We should be sharding that at a much higher
> granularity and using it much more consistently.  That is to say, that we
> should have a maintainer for all the ethernet pmds, and another for the
> crypto pmds, another for the core eal layer, another for misc libraries
> that have low patch volumes, etc.  Each of those subdivisions should have
> their own list to communicate on, and each should have a tree that
> integrates patches for their own subsystem, and they should on a regular
> cycle send pull requests to Thomas.  Thomas in turn should by and large,
> only be integrating pull requests.  This should address our high-
> throughput issue, in that it will allow multiple maintainers to share the
> workload, and integration should be relatively easy.
> 
> B) Designate alternates to serve as backups for the maintainer when they
> are unavailable.  This provides high-availablility, and sounds very much
> like your proposal, but in the interests of clarity, there is still a
> single maintainer at any one time, it just may change to ensure the
> continued merging of patches, if the primary maintainer isn't available.
> Ideally however, those backup alternates arent needed, because most of the
> primary maintainers work in merging pull requests, which are done based on
> the trust of the submaintainer, and done during a very limited window of
> time.  This also partially addreses multi-vendor fairness if your subtree
> maintainers come from multiple participating companies.
> 
> Regards
> Neil
> 
> 
> 

Soo, I feel like we're wandering away from this thread.  Are you going to make a
change to the comitter model?

Neil

Re: Proposal for a new Committer model

[Mcnamara, John]

> -----Original Message-----
> From: Neil Horman [mailto:nhorman@tuxdriver.com]
> Sent: Tuesday, November 29, 2016 7:12 PM
> To: Mcnamara, John <john.mcnamara@intel.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] Proposal for a new Committer model
> 
> > ...
> >
> > B) Designate alternates to serve as backups for the maintainer when
> > they are unavailable.  This provides high-availablility, and sounds
> > very much like your proposal, but in the interests of clarity, there
> > is still a single maintainer at any one time, it just may change to
> > ensure the continued merging of patches, if the primary maintainer isn't
> available.
> > Ideally however, those backup alternates arent needed, because most of
> > the primary maintainers work in merging pull requests, which are done
> > based on the trust of the submaintainer, and done during a very
> > limited window of time.  This also partially addreses multi-vendor
> > fairness if your subtree maintainers come from multiple participating
> companies.
> >
> > Regards
> > Neil
> >
> >
> >
> 
> Soo, I feel like we're wandering away from this thread.  Are you going to
> make a change to the comitter model?

Hi,

Yes. I think we have consensus on the main parts. I'll re-draft a proposal that we can discuss and then add to the contributors guide.

John

Re: Proposal for a new Committer model

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Mcnamara, John
> Sent: Wednesday, November 30, 2016 9:59 AM
> To: Neil Horman <nhorman@tuxdriver.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] Proposal for a new Committer model
> 
> > -----Original Message-----
> > From: Neil Horman [mailto:nhorman@tuxdriver.com]
> > Sent: Tuesday, November 29, 2016 7:12 PM
> > To: Mcnamara, John <john.mcnamara@intel.com>
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] Proposal for a new Committer model
> >
> > > ...
> > >
> > > B) Designate alternates to serve as backups for the maintainer when
> > > they are unavailable.  This provides high-availablility, and sounds
> > > very much like your proposal, but in the interests of clarity, there
> > > is still a single maintainer at any one time, it just may change to
> > > ensure the continued merging of patches, if the primary maintainer
> > > isn't
> > available.
> > > Ideally however, those backup alternates arent needed, because most
> > > of the primary maintainers work in merging pull requests, which are
> > > done based on the trust of the submaintainer, and done during a very
> > > limited window of time.  This also partially addreses multi-vendor
> > > fairness if your subtree maintainers come from multiple
> > > participating
> > companies.
> > >
> > > Regards
> > > Neil
> > >
> > >
> > >
> >
> > Soo, I feel like we're wandering away from this thread.  Are you going
> > to make a change to the comitter model?
> 
> Hi,
> 
> Yes. I think we have consensus on the main parts. I'll re-draft a proposal
> that we can discuss and then add to the contributors guide.
> 
> John
> 

I'll submit a draft update to the DPDK Code Contributors guide shortly.

John