Better organizing ext4 development community

Better organizing ext4 development community

[Theodore Ts'o]

There seems to be an increasing number of people interested in ext4 who
have been contributing to ext4, and so I thought this might be a good
time to pen a note about how we might be able to organize things a
little better.

The weekly conference calls
===========================

We have a weekly conference call at Monday 8am US/Pacific and 11am
US/Eastern.  The people who participate on this call are primarily from
the following companies (alphabetically): Google, HP, IBM, Red Hat, and
Whamcloud.  The attendees are primarily mostly from the US, due to
history of who has known about the calls, but also due to the time of
the conference calls.

Given there are more people who are participating on this call from
Asia, especially from Tao Bao and other companies, I think it's
important that we revisit whether that time is best, and to try to
invite some folks from the other countries to participate.   I know that
language may be a barrier in some cases, but I think it's important that
we try to get a wider representation that has a chance to hear what's
going.

I'd like to hear what folks from Asia think about this; if we started
having some conference calls that alternated between being convenient
for folks based out of Asia time zones and folks based out of US time
zones, would that be helpful?

Getting together for a face to face meeting
===========================================

Again, because of the fact that we quite a few newcomers to the ext4
development community, it seems that it might be a good idea if we could
get together so we could know each other better.  I'd like to propose
that we try doing so either immediately before or immediately after the
Linux Storage, File System, and MM workshop in San Fracisco, which will
be taking place during the first week of April next year.  I'm hoping
that we can get good representation from all of the companies who have
an interest in ext4 development, and if we start early, we can hopefully
get people thinking about some kind of discussion topic to propose for
the LSF workshop (proposing a discussion topic that would be of general
interest is how you get an invite to the LSF), and so people have ample
time to work out the logistics --- getting travel approval, getting
Visa's, etc.

If you would be interested in attending an ext4 get together next year
in April, please let me know, so I can start guaging interesting and
numbers.

Review Bottleneck
=================

Currently, patches, especially large patch series which introduce some
new feature, have become bottlenecked on my time to review them.   It
would be very helpful if we had more people reviewing patches.  And it
needs to be substantive reviews, and preferably from people who work at
companies other than the developer who has submitted the patches.

So some way that we can get more people reviewing patches would
certainly be helpful.  There have been some people who have suggested
different ways that we might do things, from the method used in XFS
(where no patch gets submitted until it gets an independent review;
which would be a bit scary since at the moment so little review takes
place I'm concerned it would hold back development significantly), to
giving people patchwork accounts and formally delegating work to people
(it has worked for some subsystems, and utterly failed for others).

Or we could keep going with the current method, with people
understanding that if you review other people's patches, it makes it
more likely I will have time to integrate your patches (and if some
folks do more review work, I'll take that into account about which
patches series I'm more likely to review myself for integration).


What do you think?  This, like all of the other parts of this note, was
meant to start a discussion.  I've been extraordinarily pleased with
Ext4 development: with what we've been able to achieve, and the people
we've managed to attract to use and to work on this project.   With your
help, we can make things even better!

Cheers,

						- Ted

Re: Better organizing ext4 development community

[Amir Goldstein]

On Mon, Nov 21, 2011 at 5:58 PM, Theodore Ts'o <tytso@mit.edu> wrote:
>
> There seems to be an increasing number of people interested in ext4 who
> have been contributing to ext4, and so I thought this might be a good
> time to pen a note about how we might be able to organize things a
> little better.
>
> The weekly conference calls
> ===========================
>
> We have a weekly conference call at Monday 8am US/Pacific and 11am
> US/Eastern.  The people who participate on this call are primarily from
> the following companies (alphabetically): Google, HP, IBM, Red Hat, and
> Whamcloud.  The attendees are primarily mostly from the US, due to
> history of who has known about the calls, but also due to the time of
> the conference calls.
>
> Given there are more people who are participating on this call from
> Asia, especially from Tao Bao and other companies, I think it's
> important that we revisit whether that time is best, and to try to
> invite some folks from the other countries to participate.   I know that
> language may be a barrier in some cases, but I think it's important that
> we try to get a wider representation that has a chance to hear what's
> going.

Hi Ted,

As someone who participated in some calls (which lately don't fit in
my schedule),
I can say that the information shared on these call can be very
interesting to the community
but rarely leaves the "walls" of the conference.

It would be nice to get minutes of the call posted to the list
(there are some from 2007 on ext4 wiki)
I know it's about who will do that, but even the minimal updates could
be useful.


>
> I'd like to hear what folks from Asia think about this; if we started
> having some conference calls that alternated between being convenient
> for folks based out of Asia time zones and folks based out of US time
> zones, would that be helpful?
>
> Getting together for a face to face meeting
> ===========================================
>
> Again, because of the fact that we quite a few newcomers to the ext4
> development community, it seems that it might be a good idea if we could
> get together so we could know each other better.  I'd like to propose
> that we try doing so either immediately before or immediately after the
> Linux Storage, File System, and MM workshop in San Fracisco, which will
> be taking place during the first week of April next year.  I'm hoping
> that we can get good representation from all of the companies who have
> an interest in ext4 development, and if we start early, we can hopefully
> get people thinking about some kind of discussion topic to propose for
> the LSF workshop (proposing a discussion topic that would be of general
> interest is how you get an invite to the LSF), and so people have ample
> time to work out the logistics --- getting travel approval, getting
> Visa's, etc.

Wouldn't it be more inviting to hold such a meeting adjacent to an open event?
After all, not everyone can get invited to LSF on the same year.

>
> If you would be interested in attending an ext4 get together next year
> in April, please let me know, so I can start guaging interesting and
> numbers.
>
> Review Bottleneck
> =================
>
> Currently, patches, especially large patch series which introduce some
> new feature, have become bottlenecked on my time to review them.   It
> would be very helpful if we had more people reviewing patches.  And it
> needs to be substantive reviews, and preferably from people who work at
> companies other than the developer who has submitted the patches.
>
> So some way that we can get more people reviewing patches would
> certainly be helpful.  There have been some people who have suggested
> different ways that we might do things, from the method used in XFS
> (where no patch gets submitted until it gets an independent review;
> which would be a bit scary since at the moment so little review takes
> place I'm concerned it would hold back development significantly), to
> giving people patchwork accounts and formally delegating work to people
> (it has worked for some subsystems, and utterly failed for others).
>
> Or we could keep going with the current method, with people
> understanding that if you review other people's patches, it makes it
> more likely I will have time to integrate your patches (and if some
> folks do more review work, I'll take that into account about which
> patches series I'm more likely to review myself for integration).
>

I am going to throw a crazy idea in the air -
Let the "long patch series" stand in line for review by the maintainer.
On every merge window, the maintainer may have time to review one (or less)
such long series.
If people know their patches need to stand in line for review, they will
have several ways to expedite the review/merge of their patches:
1. make the case why their feature is more valuable then others
2. make the case why a feature in front of them is less valuable then others
3. get someone else to do an independent review on their feature
4. review the patches in front of them, so they will be returned
    for another round of improvements (and come back at the end of the line)
    or better yet, be acked and merged.

don't know if this can work in practice, but it intrigues me as an experiment
in game theory ;-)

Amir.

>
> What do you think?  This, like all of the other parts of this note, was
> meant to start a discussion.  I've been extraordinarily pleased with
> Ext4 development: with what we've been able to achieve, and the people
> we've managed to attract to use and to work on this project.   With your
> help, we can make things even better!
>
> Cheers,
>
>                                                - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Better organizing ext4 development community

[Andreas Dilger]

On 2011-11-21, at 8:58 AM, Theodore Ts'o wrote:
> The weekly conference calls
> ===========================
> 
> We have a weekly conference call at Monday 8am US/Pacific and 11am
> US/Eastern.  The people who participate on this call are primarily from
> the following companies (alphabetically): Google, HP, IBM, Red Hat, and
> Whamcloud.  The attendees are primarily mostly from the US, due to
> history of who has known about the calls, but also due to the time of
> the conference calls.
> 
> I'd like to hear what folks from Asia think about this; if we started
> having some conference calls that alternated between being convenient
> for folks based out of Asia time zones and folks based out of US time
> zones, would that be helpful?

At Whamcloud we have regularly scheduled concalls that cross a lot of
timezones, and realistically the only time that allows people to join
is early PT in the USA, end of day in Europe, and ~midnight in Asia.

I don't personally have an objection to attending concalls at midnight
in my timezone, but that makes it 2AM ET which was usually a deal breaker
for our scheduling.  Going any earlier means before 7AM in Europe.

> Getting together for a face to face meeting
> ===========================================
> 
> Again, because of the fact that we quite a few newcomers to the ext4
> development community, it seems that it might be a good idea if we could
> get together so we could know each other better.  I'd like to propose
> that we try doing so either immediately before or immediately after the
> Linux Storage, File System, and MM workshop in San Fracisco, which will
> be taking place during the first week of April next year.  I'm hoping
> that we can get good representation from all of the companies who have
> an interest in ext4 development, and if we start early, we can hopefully
> get people thinking about some kind of discussion topic to propose for
> the LSF workshop (proposing a discussion topic that would be of general
> interest is how you get an invite to the LSF), and so people have ample
> time to work out the logistics --- getting travel approval, getting
> Visa's, etc.
> 
> If you would be interested in attending an ext4 get together next year
> in April, please let me know, so I can start guaging interesting and
> numbers.

I'd be OK to attend such a meeting, but I agree with Amir that relatively
few ext4 developers are invited to KS so it doesn't necessarily reduce the
travel.  In years past we would attend OLS, but that is (AFAIK) largely
unattended by ext4 developers these days.

> Review Bottleneck
> =================
> So some way that we can get more people reviewing patches would
> certainly be helpful.  There have been some people who have suggested
> different ways that we might do things, from the method used in XFS
> (where no patch gets submitted until it gets an independent review;
> which would be a bit scary since at the moment so little review takes
> place I'm concerned it would hold back development significantly), to
> giving people patchwork accounts and formally delegating work to people
> (it has worked for some subsystems, and utterly failed for others).
> 
> Or we could keep going with the current method, with people
> understanding that if you review other people's patches, it makes it
> more likely I will have time to integrate your patches (and if some
> folks do more review work, I'll take that into account about which
> patches series I'm more likely to review myself for integration).

I've been doing reviews on a subset of the patches that pass the list
(e.g. metadata checksums, bigalloc extent size, inline data, online
resize).  I'll admit I've also been lax in terms of making positive
comments when I've completed a review without finding any problems, and
sometimes my inspections are not as detailed as they need to be.  Since
code inspection makes up a significant fraction of my day job, it is hard
to get excited about it later on.

I wouldn't object to using Gerrit to do inspections for the ext4 patches.
We use that internally, and it allows tracking of patches, easily comparing
new patches against the old ones to speed up re-inspection, and hooks into
Jenkins and other automated testing tools to allow automated building.

> What do you think?  This, like all of the other parts of this note, was
> meant to start a discussion.  I've been extraordinarily pleased with
> Ext4 development: with what we've been able to achieve, and the people
> we've managed to attract to use and to work on this project.   With your
> help, we can make things even better!


Cheers, Andreas

Re: Better organizing ext4 development community

[Theodore Tso]

On Nov 21, 2011, at 4:51 PM, Andreas Dilger wrote:

> At Whamcloud we have regularly scheduled concalls that cross a lot of
> timezones, and realistically the only time that allows people to join
> is early PT in the USA, end of day in Europe, and ~midnight in Asia.

What I've done (which has never been pleasant) is to alternate between 11am US/Eastern and 11pm US/Pacific.  That distributes the pain a bit more equally, but yeah, it's not great.


> I'd be OK to attend such a meeting, but I agree with Amir that relatively
> few ext4 developers are invited to KS so it doesn't necessarily reduce the
> travel.  In years past we would attend OLS, but that is (AFAIK) largely
> unattended by ext4 developers these days.

This is not the Kernel Summit, but LSF.   And the LSF is held at the same week as the Collaboration Summit, which is a open event.   Also, getting an invite to LSF really isn't that hard, if you plan ahead…

-- Ted


--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Better organizing ext4 development community

[Tao Ma]

Hi Ted,
On 11/21/2011 11:58 PM, Theodore Ts'o wrote:
> There seems to be an increasing number of people interested in ext4 who
> have been contributing to ext4, and so I thought this might be a good
> time to pen a note about how we might be able to organize things a
> little better.
> 
> The weekly conference calls
> ===========================
> 
> We have a weekly conference call at Monday 8am US/Pacific and 11am
> US/Eastern.  The people who participate on this call are primarily from
> the following companies (alphabetically): Google, HP, IBM, Red Hat, and
> Whamcloud.  The attendees are primarily mostly from the US, due to
> history of who has known about the calls, but also due to the time of
> the conference calls.
> 
> Given there are more people who are participating on this call from
> Asia, especially from Tao Bao and other companies, I think it's
> important that we revisit whether that time is best, and to try to
> invite some folks from the other countries to participate.   I know that
> language may be a barrier in some cases, but I think it's important that
> we try to get a wider representation that has a chance to hear what's
> going.
> 
> I'd like to hear what folks from Asia think about this; if we started
> having some conference calls that alternated between being convenient
> for folks based out of Asia time zones and folks based out of US time
> zones, would that be helpful?
We are glad to attend this conference call, but have 2 concerns here:
1. about the time. Are there any guys from Europe attend this concall?
If not, I would recommend 4-5PM PT, which means 7-8PM east time and
aroudn 8am here in Asia. We have the concall at that time when I was in
my previous company. If there are some guys from Europe, I think it
would be a pain for them and then I am OK with the current time.
2. about the access number. Are there any local provider for the concall
service? I don't think the company(or us) can cover the long-distance
call :) and a local access number or a toll free one would be great.
> 
> Getting together for a face to face meeting
> ===========================================
> 
> Again, because of the fact that we quite a few newcomers to the ext4
> development community, it seems that it might be a good idea if we could
> get together so we could know each other better.  I'd like to propose
> that we try doing so either immediately before or immediately after the
> Linux Storage, File System, and MM workshop in San Fracisco, which will
> be taking place during the first week of April next year.  I'm hoping
> that we can get good representation from all of the companies who have
> an interest in ext4 development, and if we start early, we can hopefully
> get people thinking about some kind of discussion topic to propose for
> the LSF workshop (proposing a discussion topic that would be of general
> interest is how you get an invite to the LSF), and so people have ample
> time to work out the logistics --- getting travel approval, getting
> Visa's, etc.
> 
> If you would be interested in attending an ext4 get together next year
> in April, please let me know, so I can start guaging interesting and
> numbers.
Coly and I had attended this year's LSF and it would be really
appreciated if we will get invited next year also.
> 
> Review Bottleneck
> =================
> 
> Currently, patches, especially large patch series which introduce some
> new feature, have become bottlenecked on my time to review them.   It
> would be very helpful if we had more people reviewing patches.  And it
> needs to be substantive reviews, and preferably from people who work at
> companies other than the developer who has submitted the patches.
> 
> So some way that we can get more people reviewing patches would
> certainly be helpful.  There have been some people who have suggested
> different ways that we might do things, from the method used in XFS
> (where no patch gets submitted until it gets an independent review;
> which would be a bit scary since at the moment so little review takes
> place I'm concerned it would hold back development significantly), to
> giving people patchwork accounts and formally delegating work to people
> (it has worked for some subsystems, and utterly failed for others).
> 
> Or we could keep going with the current method, with people
> understanding that if you review other people's patches, it makes it
> more likely I will have time to integrate your patches (and if some
> folks do more review work, I'll take that into account about which
> patches series I'm more likely to review myself for integration).
Fine, we will try to spare some time to review others work if we
consider ourselves qualified.
> 
> 
> What do you think?  This, like all of the other parts of this note, was
> meant to start a discussion.  I've been extraordinarily pleased with
> Ext4 development: with what we've been able to achieve, and the people
> we've managed to attract to use and to work on this project.   With your
> help, we can make things even better!
yeah, actually Tao Bao had began to migrate our production systems to be
based on ext4 earlier this year and we will continue to put more guys in
the ext4 development. And we would be glad to see ext4 be more stable
and feature complete in the near future.

Thanks
Tao

Re: Better organizing ext4 development community

[Dave Chinner]

On Mon, Nov 21, 2011 at 10:58:36AM -0500, Theodore Ts'o wrote:
> Review Bottleneck
> =================
> 
> Currently, patches, especially large patch series which introduce some
> new feature, have become bottlenecked on my time to review them.   It
> would be very helpful if we had more people reviewing patches.  And it
> needs to be substantive reviews, and preferably from people who work at
> companies other than the developer who has submitted the patches.
> 
> So some way that we can get more people reviewing patches would
> certainly be helpful.  There have been some people who have suggested
> different ways that we might do things, from the method used in XFS
> (where no patch gets submitted until it gets an independent review;
> which would be a bit scary since at the moment so little review takes
> place I'm concerned it would hold back development significantly),

I think that's only a short-term concern, Ted.

Indeed, look at the numbers. For XFS, we have a core group of 4-5
active developers doing all the work and all the review for both the
userspace and kernel changes[*] - ext4 has more than 2x the active
developer base compared to XFS.

However, it's worth a look at the amount of code this XFS dev group
has changed and reviewed since 2.6.32 under the mandatory review
policy.  Kernel code [**]:

$ git diff --find-renames --stat v2.6.32.. -- fs/xfs/ |tail -1
169 files changed, 24897 insertions(+), 30340 deletions(-)

and commit count:

$ glo v2.6.32.. -- fs/xfs | wc -l
816

In the same timeframe, the ext4 kernel code:

$ git diff --find-renames --stat v2.6.32.. -- fs/ext4 fs/jdb2 |tail
-1
 34 files changed, 12080 insertions(+), 6979 deletions(-)
$ glo v2.6.32.. -- fs/ext4 fs/jdb2 | wc -l
691

The amount of change made to the ext4 kernel vs XFS kernel code
bases in terms of LOC is 3-4x less, and the commit count is about
20% less despite having 2x the developer code base than for XFS.
That's a big differential in productivity when you consider it on
a per-developer basis, regardless of the size of the code base.

My point here is that mandatory code review does not actually slow
down the rate of development. In fact, I'll argue that it speeds up
the rate of development over the medium to long term because of
several factors:

1. it forces people to understand code they are not familiar with
   and ask questions about it which the developer has to respond to,
   thereby speeding up the learning cycle for both developers and
   reviewers.

2. it encourages people to review code, because if you don't review
   patches as they are posted, then nobody is going to review your
   patches promptly in return. This speeds up the review process.

3. from 1) developer expertise improves, leading to improved quality
   of submitted patches, which also then has the effect of making
   review easier.

4. from 2) and 3), it reduces the review burden on the maintainer,
   because over time the maintainer can begin to trust that
   Reviewed-by tags indicate that the change is good. This will take
   time to build such trust, but without it there is no way for the
   maintainer to know how hard to look at any given change before
   deciding whether to merge it.

5. from 2) and 3), the maintainer no longer needs to be a gate
   keeper deciding whether changes are good or not. The maintainer
   participates in that discussion just like any other developer,
   including giving out Reviewed-by tags when they are satisfied a
   change is good. IOWs, the bar to getting a change merged is
   consensus and having sufficient reviewed-by tags on a patch
   [series] to convince the maintainer that the change is
   acceptible.

6) from 4) and 5), the trust built into reviewed-by tag when used
   in this manner removes the need for the maintainer to be the
   ultimate expert about everything. In the XFS world the maintainer
   is the person that maintains the tree that is pushed to Linus,
   but otherwise they are just another developer that proposes
   changes and reviews code like anyone else.

7) from 6), it frees experts to develop and review code and improve
   processes rather than having to spend most of their time being
   patch monkeys, cat herders, gate keepers and, ultimately, a
   bottleneck in the development process. Distributed review as
   encoded by gathering Reviewed-by tags generates consensus about
   whether changes are good or not, hence removing the need for the
   maintainer to be the sole decision maker on a project.

8) From 7), experts have more time available to mentor new
   developers and also spend more time explaining complexities to
   more experienced developers during review cycles.  This feeds
   back into improving the knowledge base of the development group
   via 1), 2) and 3).

9) from all of the above, code and product quality improves at a
   faster rate because of the continuously improving knowledge base
   and increasing level of confidence that developers have in the
   capability of their peers.

I'm not going to argue that this policy the right solution for the
problems Ted has brought up - that's for you guys to decide. All I'm
doing is explaining why I think the mandatory review policy works
for XFS, and how it could also benefit the ext4 community....

Cheers,

Dave.

[*] FYI, Lucas had a great slide describing the relative developer
breakdown between the filesystems at his talk at Linuxcon Europe.
It's the slides 5-7 of this presentation:

https://events.linuxfoundation.org/images/stories/pdf/lceu11_czerner1.pdf

[**] I've ignored userspace because I don't have a ext4 userspace
tree handy right now. The XFS xfsprogs changes (excluding
translations) over the last 2 years (since 3.0.4 was release) are
also significantly larger than the ext4 kernel codebase changes:

$ git diff --find-renames --stat v3.0.4 -- [!p]* |tail -1
 227 files changed, 11729 insertions(+), 8788 deletions(-)
$ glo v3.0.4 -- [!p]* | wc -l
934

-- 
Dave Chinner
david@fromorbit.com