OpenBMC Project metrics

OpenBMC Project metrics

[Kurt Taylor]

All, I just posted the project merge metrics for September and
October. I will be updating the company/developer lists for November
and posting those shortly. Enjoy.

https://github.com/openbmc/openbmc/wiki/Project-Metrics

NOTE: these metrics should be used *very carefully*. They do not
represent the total contributions to the project. We value
contributions many that do not show up in these charts, including
reviews, mail list involvement, IRC involvement, etc.

Kurt Taylor (krtaylor)

Re: OpenBMC Project metrics

[Andrew Jeffery]

On Thu, 5 Dec 2019, at 05:14, Kurt Taylor wrote:
> All, I just posted the project merge metrics for September and
> October. I will be updating the company/developer lists for November
> and posting those shortly. Enjoy.
> 
> https://github.com/openbmc/openbmc/wiki/Project-Metrics
> 
> NOTE: these metrics should be used *very carefully*. They do not
> represent the total contributions to the project. We value
> contributions many that do not show up in these charts, including
> reviews, mail list involvement, IRC involvement, etc.

Given all the caveats and the lopsided view the graphs display, what
are we trying to achieve by graphing the metric of commits per company?

It's also not clear to me what the inputs to these graphs are, for instance
whether changes to Linux, u-boot, qemu or other major projects that we
consume and contribute to are included or whether it's just repositories
under the openbmc org on github. If we're excluding upstream projects,
why?

Where are the scripts to reproduce the graphs? Can you contribute them
to openbmc-tools?

Andrew

Re: OpenBMC Project metrics

[krtaylor]

On 12/4/19 4:33 PM, Andrew Jeffery wrote:
> 
> 
> On Thu, 5 Dec 2019, at 05:14, Kurt Taylor wrote:
>> All, I just posted the project merge metrics for September and
>> October. I will be updating the company/developer lists for November
>> and posting those shortly. Enjoy.
>>
>> https://github.com/openbmc/openbmc/wiki/Project-Metrics
>>
>> NOTE: these metrics should be used *very carefully*. They do not
>> represent the total contributions to the project. We value
>> contributions many that do not show up in these charts, including
>> reviews, mail list involvement, IRC involvement, etc.
> 
> Given all the caveats and the lopsided view the graphs display, what
> are we trying to achieve by graphing the metric of commits per company?

"What gets measured, gets managed" I am a firm believer of this simple 
quote. Measuring a project always improves it. That, and I have been 
asked to start gathering metrics from several of our contributing 
companies. They appreciate it.

> It's also not clear to me what the inputs to these graphs are, for instance
> whether changes to Linux, u-boot, qemu or other major projects that we
> consume and contribute to are included or whether it's just repositories
> under the openbmc org on github. If we're excluding upstream projects,
> why?

It is only for contributions under openbmc. Other projects have been 
excluded simply because they have their own project metrics. For example:

Linux: 
https://www.linuxfoundation.org/blog/2016/08/the-top-10-developers-and-companies-contributing-to-the-linux-kernel-in-2015-2016/

uboot: 
https://osfc.io/uploads/talk/paper/31/2019_State_U-Boot_development_report.pdf

*Really nice* interactive openstack stats gui: https://www.stackalytics.com/

> Where are the scripts to reproduce the graphs? Can you contribute them
> to openbmc-tools?

Eventually yes, if my employer will let me do more upstream. :) But, the 
data is publicly available, you can get it yourself from gerrit. Simply 
go to our gerrit dashboard and search something like: " status:merged 
AND after:<date> AND before:<date> AND NOT topic:autobump AND 
owner:<gerrit id> "

I appreciate your feedback, I will make the specifics of what is 
measured and how it is done more clear on the project metrics wiki page.

Kurt Taylor (krtaylor)

> Andrew
> 

Re: OpenBMC Project metrics

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 3901 bytes --]

On Fri, Dec 06, 2019 at 07:33:26AM -0600, krtaylor wrote:
> On 12/4/19 4:33 PM, Andrew Jeffery wrote:
> > On Thu, 5 Dec 2019, at 05:14, Kurt Taylor wrote:
> > > 
> > > NOTE: these metrics should be used *very carefully*. They do not
> > > represent the total contributions to the project. We value
> > > contributions many that do not show up in these charts, including
> > > reviews, mail list involvement, IRC involvement, etc.
> > 
> > Given all the caveats and the lopsided view the graphs display, what
> > are we trying to achieve by graphing the metric of commits per company?
> 
> "What gets measured, gets managed" I am a firm believer of this simple
> quote. Measuring a project always improves it. That, and I have been asked
> to start gathering metrics from several of our contributing companies. They
> appreciate it.

I recognize some other projects publish statistics like this and it is
all publicly available information but I personally have slight
apprehension about this data.  This project is no where as mature as the
Linux kernel and the data is highly skewed towards one company.  I have
some concern that this data could be used for political purposes, both
externally in community interaction and internally to member companies
w.r.t. their decisions on future involvement.

The data is public (from Gerrit), no doubt about it, but I think it is
reasonable to question if it is a net-positive or net-negative for the
community to gather the data and put it on Github, to put it on Github
and advertise it, or to put it on Github and the advertisement coming from
the Community Manager.  (ie. there is a spectrum of possible ways to
deal with this data with different pros/cons)

> "Measuring a project always improves it."

Maybe a first step here is answering what is the desired change by
publishing this data?  And who's desire is it?  That isn't obvious to me.

> > It's also not clear to me what the inputs to these graphs are, for instance
> > whether changes to Linux, u-boot, qemu or other major projects that we
> > consume and contribute to are included or whether it's just repositories
> > under the openbmc org on github. If we're excluding upstream projects,
> > why?
> 
> It is only for contributions under openbmc. Other projects have been
> excluded simply because they have their own project metrics. For example:

The commit-count-from-Gerrit approach is slightly disappointing to me for
two reasons:

    1. Commit count does little to assess the impact of the
       contribution.  Ex. a one-line recipe update to add a dependency
       counts the same as a feature.

    2. There are significant contributions on the kernel side done by
       and pretty much exclusively for this project.  The effort
       involved with getting kernel patches upstream is at least an
       order of magnitude higher than userspace changes (see also
       "impact").

> > Where are the scripts to reproduce the graphs? Can you contribute them
> > to openbmc-tools?
> 
> Eventually yes, if my employer will let me do more upstream. :) But, the
> data is publicly available, you can get it yourself from gerrit. Simply go
> to our gerrit dashboard and search something like: " status:merged AND
> after:<date> AND before:<date> AND NOT topic:autobump AND owner:<gerrit id>
> "

One aspect that isn't immediately obvious, since it isn't available via
source code, is how you've done the company assignment.  I suspect the
ones for your employer are correct but for other companies there might be
mistakes or oversights when people are using personal email addresses.

I think this concern also ties into the ask a month ago with the
"computer readable CLA database."  If we had a CLA database and this
tool used it, we would have one place to audit for correctness.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: OpenBMC Project metrics

[Andrew Jeffery]

On Sat, 7 Dec 2019, at 00:03, krtaylor wrote:
> On 12/4/19 4:33 PM, Andrew Jeffery wrote:
> > 
> > 
> > On Thu, 5 Dec 2019, at 05:14, Kurt Taylor wrote:
> >> All, I just posted the project merge metrics for September and
> >> October. I will be updating the company/developer lists for November
> >> and posting those shortly. Enjoy.
> >>
> >> https://github.com/openbmc/openbmc/wiki/Project-Metrics
> >>
> >> NOTE: these metrics should be used *very carefully*. They do not
> >> represent the total contributions to the project. We value
> >> contributions many that do not show up in these charts, including
> >> reviews, mail list involvement, IRC involvement, etc.
> > 
> > Given all the caveats and the lopsided view the graphs display, what
> > are we trying to achieve by graphing the metric of commits per company?
> 
> "What gets measured, gets managed" I am a firm believer of this simple 
> quote.

Okay, but my concern is the lack of context. I think we're putting the cart
before the horse in that we as a project need to decide what we want
to manage, determine the appropriate metrics and then perform the
measurements rather than cherry-pick things to measure and present
the data without context. We need to know what questions we're trying to
answer and there is no context available for your data as far as I'm aware,
certainly not at the current revision of the wiki page:

https://github.com/openbmc/openbmc/wiki/Project-Metrics/f15759a7ff5c61fa6713b5602dd0f40820b84d0e

> Measuring a project always improves it. That, and I have been 
> asked to start gathering metrics from several of our contributing 
> companies.

Where did this discussion occur? Can you provide a link?

> They appreciate it.
> 
> > It's also not clear to me what the inputs to these graphs are, for instance
> > whether changes to Linux, u-boot, qemu or other major projects that we
> > consume and contribute to are included or whether it's just repositories
> > under the openbmc org on github. If we're excluding upstream projects,
> > why?
> 
> It is only for contributions under openbmc. Other projects have been 
> excluded simply because they have their own project metrics. For example:
> 
> Linux: 
> https://www.linuxfoundation.org/blog/2016/08/the-top-10-developers-and-companies-contributing-to-the-linux-kernel-in-2015-2016/
> 
> uboot: 
> https://osfc.io/uploads/talk/paper/31/2019_State_U-Boot_development_report.pdf

Sure, but delegating to the upstream projects' statistics buries the
contributions that some are making that are driven by working on OpenBMC.
Often contribution to OpenBMC is by way of improving the kernel. Disregarding
this contributes to the lopsided view that your graphs present.

I'm concerned that we're trying to create a stick to beat contributing companies with
rather than working to find ways increase contributions for mutual benefit. Competition
works as a motivator when community members feel safe to take it on but I'm not sure
the community is mature enough for that to be true. Adding the context for your
statistics might help remove my concerns.

> 
> *Really nice* interactive openstack stats gui: https://www.stackalytics.com/
> 
> > Where are the scripts to reproduce the graphs? Can you contribute them
> > to openbmc-tools?
> 
> Eventually yes, if my employer will let me do more upstream. :) But, the 
> data is publicly available, you can get it yourself from gerrit. Simply 
> go to our gerrit dashboard and search something like: " status:merged 
> AND after:<date> AND before:<date> AND NOT topic:autobump AND 
> owner:<gerrit id> "
> 
> I appreciate your feedback, I will make the specifics of what is 
> measured and how it is done more clear on the project metrics wiki page.

Thanks, but please make sure to address the critical issue: What is the
question whose answer the data from these metrics supports?

Andrew

Re: OpenBMC Project metrics

[krtaylor]

On 12/8/19 6:06 PM, Andrew Jeffery wrote:
> 
> 
> On Sat, 7 Dec 2019, at 00:03, krtaylor wrote:
>> On 12/4/19 4:33 PM, Andrew Jeffery wrote:
>>>
>>>
>>> On Thu, 5 Dec 2019, at 05:14, Kurt Taylor wrote:
>>>> All, I just posted the project merge metrics for September and
>>>> October. I will be updating the company/developer lists for November
>>>> and posting those shortly. Enjoy.
>>>>
>>>> https://github.com/openbmc/openbmc/wiki/Project-Metrics
>>>>
>>>> NOTE: these metrics should be used *very carefully*. They do not
>>>> represent the total contributions to the project. We value
>>>> contributions many that do not show up in these charts, including
>>>> reviews, mail list involvement, IRC involvement, etc.
>>>
>>> Given all the caveats and the lopsided view the graphs display, what
>>> are we trying to achieve by graphing the metric of commits per company?
>>
>> "What gets measured, gets managed" I am a firm believer of this simple
>> quote.
> 
> Okay, but my concern is the lack of context. I think we're putting the cart
> before the horse in that we as a project need to decide what we want
> to manage, determine the appropriate metrics and then perform the
> measurements rather than cherry-pick things to measure and present

Thanks for your opinions! As I have said, this is the first in several 
data points that will be gathered and presented, hence the caveats. I 
had to start somewhere.

> the data without context. We need to know what questions we're trying to
> answer and there is no context available for your data as far as I'm aware,
> certainly not at the current revision of the wiki page:
> 
> https://github.com/openbmc/openbmc/wiki/Project-Metrics/f15759a7ff5c61fa6713b5602dd0f40820b84d0e
> 
>> Measuring a project always improves it. That, and I have been
>> asked to start gathering metrics from several of our contributing
>> companies.
> 
> Where did this discussion occur? Can you provide a link?

The conversations have happen many times over the last 2 years.

At TSC and community meetings (not recorded in the meeting minutes that 
I could find, but it was discussed)

At release planning meetings (see minutes):
https://github.com/openbmc/openbmc/wiki/Release-Planning

I would rather not disclose email (without consent) that I have received 
privately from several companies supporting this work.

No one has ever had any objection to gathering this information (until 
now). Remember, anyone can go see this information any time they want.

> 
>> They appreciate it.
>>
>>> It's also not clear to me what the inputs to these graphs are, for instance
>>> whether changes to Linux, u-boot, qemu or other major projects that we
>>> consume and contribute to are included or whether it's just repositories
>>> under the openbmc org on github. If we're excluding upstream projects,
>>> why?
>>
>> It is only for contributions under openbmc. Other projects have been
>> excluded simply because they have their own project metrics. For example:
>>
>> Linux:
>> https://www.linuxfoundation.org/blog/2016/08/the-top-10-developers-and-companies-contributing-to-the-linux-kernel-in-2015-2016/
>>
>> uboot:
>> https://osfc.io/uploads/talk/paper/31/2019_State_U-Boot_development_report.pdf
> 
> Sure, but delegating to the upstream projects' statistics buries the
> contributions that some are making that are driven by working on OpenBMC.
> Often contribution to OpenBMC is by way of improving the kernel. Disregarding
> this contributes to the lopsided view that your graphs present.

Exactly my point. See carefully wording.
> 
> I'm concerned that we're trying to create a stick to beat contributing companies with
> rather than working to find ways increase contributions for mutual benefit. Competition
> works as a motivator when community members feel safe to take it on but I'm not sure
> the community is mature enough for that to be true. Adding the context for your
> statistics might help remove my concerns.

Honestly I'm surprised at this reaction to a *potential* situation I 
have never witnessed, but, I am willing to add any wording that you feel 
is necessary to create a safe development environment. I have 
participated in open source projects for 20+ years and personally was 
very motivated by contribution metrics, it made it really fun to see if 
I could do better in the community next month!

I value your feedback. When do you feel we as a community are mature 
enough to start monitoring reviews, commits, and other project data? 
Should we hide this "early" data until some future time when it 
represents everyone equally? Personally, I don't feel like we will ever 
get to that place. There will always be people that contribute more in 
one particular area than others and they just can't be upset that they 
may have done less. Open Source requires thick skin. Again, I've never 
seen project metrics reduce productivity/contributions.

Eventually, I'd like to break this data down by project and individual, 
not just company. And, also show data on other extremely valuable 
project contributions such as reviews, IRC meeting participation, 
testing/results, bug reports, maillist involvement and more. Its all 
valuable!

Thanks again!
Kurt Taylor (krtaylor)

>>
>> *Really nice* interactive openstack stats gui: https://www.stackalytics.com/
>>
>>> Where are the scripts to reproduce the graphs? Can you contribute them
>>> to openbmc-tools?
>>
>> Eventually yes, if my employer will let me do more upstream. :) But, the
>> data is publicly available, you can get it yourself from gerrit. Simply
>> go to our gerrit dashboard and search something like: " status:merged
>> AND after:<date> AND before:<date> AND NOT topic:autobump AND
>> owner:<gerrit id> "
>>
>> I appreciate your feedback, I will make the specifics of what is
>> measured and how it is done more clear on the project metrics wiki page.
> 
> Thanks, but please make sure to address the critical issue: What is the
> question whose answer the data from these metrics supports > Andrew
> 

Re: OpenBMC Project metrics

[krtaylor]

On 12/6/19 10:51 AM, Patrick Williams wrote:
> On Fri, Dec 06, 2019 at 07:33:26AM -0600, krtaylor wrote:
>> On 12/4/19 4:33 PM, Andrew Jeffery wrote:
>>> On Thu, 5 Dec 2019, at 05:14, Kurt Taylor wrote:
>>>>
>>>> NOTE: these metrics should be used *very carefully*. They do not
>>>> represent the total contributions to the project. We value
>>>> contributions many that do not show up in these charts, including
>>>> reviews, mail list involvement, IRC involvement, etc.
>>>
>>> Given all the caveats and the lopsided view the graphs display, what
>>> are we trying to achieve by graphing the metric of commits per company?
>>
>> "What gets measured, gets managed" I am a firm believer of this simple
>> quote. Measuring a project always improves it. That, and I have been asked
>> to start gathering metrics from several of our contributing companies. They
>> appreciate it.
> 
> I recognize some other projects publish statistics like this and it is
> all publicly available information but I personally have slight
> apprehension about this data.  This project is no where as mature as the
> Linux kernel and the data is highly skewed towards one company.  I have
> some concern that this data could be used for political purposes, both
> externally in community interaction and internally to member companies
> w.r.t. their decisions on future involvement.
> 
> The data is public (from Gerrit), no doubt about it, but I think it is
> reasonable to question if it is a net-positive or net-negative for the
> community to gather the data and put it on Github, to put it on Github
> and advertise it, or to put it on Github and the advertisement coming from
> the Community Manager.  (ie. there is a spectrum of possible ways to
> deal with this data with different pros/cons)
> 
>> "Measuring a project always improves it."
> 
> Maybe a first step here is answering what is the desired change by
> publishing this data?  And who's desire is it?  That isn't obvious to me.

Thanks again for the comments! See the previous reply to Andrew where I 
address some of these points.

> 
>>> It's also not clear to me what the inputs to these graphs are, for instance
>>> whether changes to Linux, u-boot, qemu or other major projects that we
>>> consume and contribute to are included or whether it's just repositories
>>> under the openbmc org on github. If we're excluding upstream projects,
>>> why?
>>
>> It is only for contributions under openbmc. Other projects have been
>> excluded simply because they have their own project metrics. For example:
> 
> The commit-count-from-Gerrit approach is slightly disappointing to me for
> two reasons:
> 
>      1. Commit count does little to assess the impact of the
>         contribution.  Ex. a one-line recipe update to add a dependency
>         counts the same as a feature.
> 
>      2. There are significant contributions on the kernel side done by
>         and pretty much exclusively for this project.  The effort
>         involved with getting kernel patches upstream is at least an
>         order of magnitude higher than userspace changes (see also
>         "impact").
> 
>>> Where are the scripts to reproduce the graphs? Can you contribute them
>>> to openbmc-tools?
>>
>> Eventually yes, if my employer will let me do more upstream. :) But, the
>> data is publicly available, you can get it yourself from gerrit. Simply go
>> to our gerrit dashboard and search something like: " status:merged AND
>> after:<date> AND before:<date> AND NOT topic:autobump AND owner:<gerrit id>
>> "
> 
> One aspect that isn't immediately obvious, since it isn't available via
> source code, is how you've done the company assignment.  I suspect the
> ones for your employer are correct but for other companies there might be
> mistakes or oversights when people are using personal email addresses.

I have been using the lists of developers that have access to run tests 
(ci-authorized) maintained per company, and do not use the email address 
(error prone) instead using the gerrit id. You are right that it may not 
be a complete or accurate list, but its all we have at the moment. The 
groups are listed here:

https://gerrit.openbmc-project.xyz/admin/groups

As you can see, some contributors listed don't even have email addresses 
in their gerrit profile.

> I think this concern also ties into the ask a month ago with the
> "computer readable CLA database."  If we had a CLA database and this
> tool used it, we would have one place to audit for correctness.

Absolutely! I have been working with the LF for a tool, but found out 
that the tool they have developed costs money and requires a LF Id. It 
is an option for the future for us to use this system and have 
CLA/CCLA/Id/Developer ACLs all in one place, but that will require 
"member" companies and monetary contributions to the project. That, or 
develop our own. All good topics for discussion.

Thanks again!
Kurt Taylor (krtaylor)

Re: OpenBMC Project metrics

[Andrew Jeffery]

On Tue, 10 Dec 2019, at 04:54, krtaylor wrote:
> On 12/8/19 6:06 PM, Andrew Jeffery wrote:
> > On Sat, 7 Dec 2019, at 00:03, krtaylor wrote:
> >> Measuring a project always improves it. That, and I have been
> >> asked to start gathering metrics from several of our contributing
> >> companies.
> > 
> > Where did this discussion occur? Can you provide a link?
> 
> The conversations have happen many times over the last 2 years.
> 
> At TSC and community meetings (not recorded in the meeting minutes that 
> I could find, but it was discussed)
> 
> At release planning meetings (see minutes):
> https://github.com/openbmc/openbmc/wiki/Release-Planning

Ah, okay, but it just briefly talks about metrics directly and not about the
questions we're trying to answer with the metrics?

> 
> I would rather not disclose email (without consent) that I have received 
> privately from several companies supporting this work.

That's okay, but I'm still trying to understand motivations here.

> 
> No one has ever had any objection to gathering this information (until 
> now). Remember, anyone can go see this information any time they want.

I'm not objecting to gathering it at all, I'm objecting to the lack of context in
presentation of the data. Why are we gathering it? What questions are we
answering? "Who commits the most" is the first order question, but why are
we asking it? Are we trying to establish the breadth of contributing companies?
Are we trying to identify first time contributions from companies and work with
them to drive further participation?

> > 
> > I'm concerned that we're trying to create a stick to beat contributing companies with
> > rather than working to find ways increase contributions for mutual benefit. Competition
> > works as a motivator when community members feel safe to take it on but I'm not sure
> > the community is mature enough for that to be true. Adding the context for your
> > statistics might help remove my concerns.
> 
> Honestly I'm surprised at this reaction to a *potential* situation I 
> have never witnessed

This is the problem with the lack of context. We can both have different
ideas about the motivations because they aren't written down anywhere.
It was just a concern of mine, I'm not claiming that it _is_ the motivation,
and I certainly hope that it's not. My concern could be eliminated if we
wrote down why we're gathering the metrics.

> but, I am willing to add any wording that you feel 
> is necessary to create a safe development environment.

I don't think it's about adding words here to create a safe environment, just
I don't expect everyone engaging with the project has 20+ years of experience
in open source software development. Open source work can be intimidating
with scrutiny that can be applied through code review and other interactions,
and not everyone is comfortable with that out of the gate. Certainly we've
done a lot of work internal to IBM to help people become comfortable with
working in open source. What I'd hate to see is people being discouraged from
interacting in the upstream community through metrics that don't have clear
goals.

OpenBMC is flipping the switch on what was a very propriety ecosystem, and
we will have contributors that don't have strong backgrounds in open source.
If the metrics have been discussed in meetings that's fine, but I'd hope the
context would make its way out as well and be attached to the data that we've
collected.

> 
> I value your feedback. When do you feel we as a community are mature 
> enough to start monitoring reviews, commits, and other project data? 

As above I don't think that we can pick a point on a timeline, and I don't
think that's even the point. I think that there should be engagement through
the general project channels (mailing list, IRC, not targeted meetings with
limited audiences) to determine what we're trying to measure, gather the
data, and then _present the data in the context of the question we're trying
to answer_. My consistent complaint is the lack of context.

> Should we hide this "early" data until some future time when it 
> represents everyone equally? 

No, I think you have a misunderstanding of my point here. We just need
to make the question that we're answering is provided with the view of
the data. Context is important.

> Personally, I don't feel like we will ever 
> get to that place. There will always be people that contribute more in 
> one particular area than others and they just can't be upset that they 
> may have done less. Open Source requires thick skin.

It shouldn't necessarily, and we need to work at making sure we're
approachable as a community. Providing context with metrics will give
people the information they need to know how the metrics are being
used by the project.

> 
> Eventually, I'd like to break this data down by project and individual, 
> not just company.

But why? I keep asking this. It's not because there's not a valid answer
and I'm trying to trap you, I just want to understand what the motivations
are.

Anyway, Github provides some of this information for us already. For
example:

https://github.com/openbmc/openbmc/graphs/contributors

Cheers,

Andrew

Re: OpenBMC Project metrics

[James Mihm]

[-- Attachment #1: Type: text/plain, Size: 7722 bytes --]

Getting caught up on emails and ARs.

I agree with the others that the context of metrics are not clear nor is
the goal in presenting this data as is.

When I look at the chart, it's not clear to me what it represents either. I
see from Kurts comment in this email that this graph represents the 'merge
metrics', but that is nowhere on the chart. There's also a disclaimer on
the wiki page that these metrics may not be a true representation of
project contribution. So, what's the true value of the metrics then?

What does the y-axis represent in units?
What repositories out of the 140 possible are represented in the graph?

I see charts like this as having one main purpose which is to communicate
who are making contributions, and how do they rank amongst the others. With
the target audience for the metrics being managers. Which bothers me,
because it's too subjective and ambiguous.

We do need to have a clear statement on the goal in providing a chart on
metrics. Here are my thoughts for what I would like to see in the metrics.
Which are focused on the what and not the who.

The primary goal is to represent a holistic view the openbmc project by
providing insight to the progress and health of the openbmc project.

Where progress is represented by the code that attains sufficient quality
to be merged upstream. I would include any upstream projects that were made
on behalf of the openbmc project (i.e., linux, u-boot, etc…).

Health as measured by amount pull requests/code commits being made, reviews
that are in progress, the amount of code added versus deleted, abandoned
code, and aging of commits/reviews.

1 - To show source code contributions that make it into the upstream
repository (i.e., merged)
2 - To show incoming code contributions via pull requests or commits (i.e.,
being reviewed)
3 - To show lines of code additions, deletions, or abandoned
4 - To show contributions via review comments (i.e., number of review
comments)
5 - Snapshots in time weekly, bi-weekly, or monthly (x-axis)

Thanks,
James.

On Tue, Dec 10, 2019 at 3:14 PM Andrew Jeffery <andrew@aj.id.au> wrote:

> On Tue, 10 Dec 2019, at 04:54, krtaylor wrote:
> > On 12/8/19 6:06 PM, Andrew Jeffery wrote:
> > > On Sat, 7 Dec 2019, at 00:03, krtaylor wrote:
> > >> Measuring a project always improves it. That, and I have been
> > >> asked to start gathering metrics from several of our contributing
> > >> companies.
> > >
> > > Where did this discussion occur? Can you provide a link?
> >
> > The conversations have happen many times over the last 2 years.
> >
> > At TSC and community meetings (not recorded in the meeting minutes that
> > I could find, but it was discussed)
> >
> > At release planning meetings (see minutes):
> > https://github.com/openbmc/openbmc/wiki/Release-Planning
>
> Ah, okay, but it just briefly talks about metrics directly and not about
> the
> questions we're trying to answer with the metrics?
>
> >
> > I would rather not disclose email (without consent) that I have received
> > privately from several companies supporting this work.
>
> That's okay, but I'm still trying to understand motivations here.
>
> >
> > No one has ever had any objection to gathering this information (until
> > now). Remember, anyone can go see this information any time they want.
>
> I'm not objecting to gathering it at all, I'm objecting to the lack of
> context in
> presentation of the data. Why are we gathering it? What questions are we
> answering? "Who commits the most" is the first order question, but why are
> we asking it? Are we trying to establish the breadth of contributing
> companies?
> Are we trying to identify first time contributions from companies and work
> with
> them to drive further participation?
>
> > >
> > > I'm concerned that we're trying to create a stick to beat contributing
> companies with
> > > rather than working to find ways increase contributions for mutual
> benefit. Competition
> > > works as a motivator when community members feel safe to take it on
> but I'm not sure
> > > the community is mature enough for that to be true. Adding the context
> for your
> > > statistics might help remove my concerns.
> >
> > Honestly I'm surprised at this reaction to a *potential* situation I
> > have never witnessed
>
> This is the problem with the lack of context. We can both have different
> ideas about the motivations because they aren't written down anywhere.
> It was just a concern of mine, I'm not claiming that it _is_ the
> motivation,
> and I certainly hope that it's not. My concern could be eliminated if we
> wrote down why we're gathering the metrics.
>
> > but, I am willing to add any wording that you feel
> > is necessary to create a safe development environment.
>
> I don't think it's about adding words here to create a safe environment,
> just
> I don't expect everyone engaging with the project has 20+ years of
> experience
> in open source software development. Open source work can be intimidating
> with scrutiny that can be applied through code review and other
> interactions,
> and not everyone is comfortable with that out of the gate. Certainly we've
> done a lot of work internal to IBM to help people become comfortable with
> working in open source. What I'd hate to see is people being discouraged
> from
> interacting in the upstream community through metrics that don't have clear
> goals.
>
> OpenBMC is flipping the switch on what was a very propriety ecosystem, and
> we will have contributors that don't have strong backgrounds in open
> source.
> If the metrics have been discussed in meetings that's fine, but I'd hope
> the
> context would make its way out as well and be attached to the data that
> we've
> collected.
>
> >
> > I value your feedback. When do you feel we as a community are mature
> > enough to start monitoring reviews, commits, and other project data?
>
> As above I don't think that we can pick a point on a timeline, and I don't
> think that's even the point. I think that there should be engagement
> through
> the general project channels (mailing list, IRC, not targeted meetings with
> limited audiences) to determine what we're trying to measure, gather the
> data, and then _present the data in the context of the question we're
> trying
> to answer_. My consistent complaint is the lack of context.
>
> > Should we hide this "early" data until some future time when it
> > represents everyone equally?
>
> No, I think you have a misunderstanding of my point here. We just need
> to make the question that we're answering is provided with the view of
> the data. Context is important.
>
> > Personally, I don't feel like we will ever
> > get to that place. There will always be people that contribute more in
> > one particular area than others and they just can't be upset that they
> > may have done less. Open Source requires thick skin.
>
> It shouldn't necessarily, and we need to work at making sure we're
> approachable as a community. Providing context with metrics will give
> people the information they need to know how the metrics are being
> used by the project.
>
> >
> > Eventually, I'd like to break this data down by project and individual,
> > not just company.
>
> But why? I keep asking this. It's not because there's not a valid answer
> and I'm trying to trap you, I just want to understand what the motivations
> are.
>
> Anyway, Github provides some of this information for us already. For
> example:
>
> https://github.com/openbmc/openbmc/graphs/contributors
>
> Cheers,
>
> Andrew
>

[-- Attachment #2: Type: text/html, Size: 8998 bytes --]

Re: OpenBMC Project metrics

[Brad Bishop]

FYI - The Yocto project put up a project metrics page:
https://lists.yoctoproject.org/g/yocto/message/48893

The stated goal is possibly this:
> we have put in place a dashboard that shows our community engagement and  
> stats

-brad

Re: OpenBMC Project metrics

[Kurt Taylor]

Thanks Brad. I talked to the LF about Chaoss last summer and it was
still in the works. Looks like they have had a release, and that I
should look into it again.

It is great to see Yocto bring it all brought together into a clear
dashboard. I've added it to the community infrastructure list [1].

Kurt Taylor (krtaylor)

[1] https://docs.google.com/document/d/1QCFRGCRofcR3K8clSLtJHw10-Cu9zkp0dvwXPWzQSCc/edit?usp=sharing

On Mon, Mar 23, 2020 at 7:44 AM Brad Bishop <bradleyb@fuzziesquirrel.com> wrote:
>
> FYI - The Yocto project put up a project metrics page:
> https://lists.yoctoproject.org/g/yocto/message/48893
>
> The stated goal is possibly this:
> > we have put in place a dashboard that shows our community engagement and
> > stats
>
> -brad