[ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Doug Ledford]

Sending to list and changing subject because my answer raises an issue
that I need to make others aware of...

On Tue, 2017-09-19 at 18:34 +0000, Saleem, Shiraz wrote:
> Hi Doug,
> 
> Hope you're doing well and had a good vacation.
> 
> With respect to i40iw patch submissions, the typical workflow we have
> followed is to submit i40iw patches and patch series to the mailing
> list. And you pick them up from the mailing list after feedback.
> 
> If it makes sense and makes your life easier, I could send a pull
> request for the i40iw patches submitted to the mailing list.
> 
> The pull request would be sent at some point in the future (example
> +1 week) from submission to mailing list; after the patches have got
> a chance for community feedback. I could send it on a day of the week
> specified by you. 
> 
> Let me know what you think about this suggested workflow.

That won't make much difference.  I'm pretty efficient at patchworks. 
The only time it really helps is when you need to prepare a shared pull
request, and in that case, it *must* be a pull request.

So, what am I talking about with a shared pull request?

That's my answer to people not necessarily wanting to wait until kernel
n+1 to use features they put in their net driver in kernel n.  So, this
applies to the mlx4, mlx5, cxgb3, cxgb4, bnxt_re, hns_roce, and i40iw
drivers off the top of my head.

I have no intention of pulling Dave Miller's net-next tree on any sort
of regular basis.  I will admit that there are certain circumstances
where it can't be avoided (a specific case being if someone introduces
a new core net feature, and the only consumer of that feature is their
net driver, so they are required to submit both the feature and the
first consumer together, and then if their dependent rdma driver also
needs updating to work with the modified net driver, then I have no
choice but to take a snapshot of net-next with the new core feature and
update the rdma driver or else we might end up with a broken tree). 
Waiting on things to land in net-next takes too much time and
coordination, pulling his tree pollutes mine with tons of additional
patches beyond just what's needed, and doing so played a direct role in
me missing the 4.13 merge window (there were other factors too, so I'm
not saying it was the sole reason, but it was certainly a significant
one) so I'm reluctant to do this any more than absolutely necessary.

So, what I want as the normal, day to day workflow for drivers that
need to have dependent code in the RDMA stack based upon code in the
net-next area is shared pull requests instead.

There are requirements for submitting pull requests.  It is preferred
to have a tree either on k.o or some similar place, and it is required
by Linus that a pull request from any place other than kernel.org
*must* be a PGP signed tag (but even for pull requests on trees hosted
on kernel.org, he prefers to still use PGP signed tags), so any pull
request to Dave Miller and myself will have the same requirements.

Mellanox has been doing shared pull requests for a year or so now, so
feel free to look at the mailing list archives and find one of their
pull requests to see how they do it.  They list the pull request in the
cover letter and still send the patch series (unlike the pull request I
send to Linus which is just a cover letter and none of the patches).

I want any shared pull requests to be based on nothing newer than an
rc2 kernel.  I branch my for-next branch at rc2, so if your pull
request is on something later it will pollute my main for-next branch
to merge it.  I plan to merge my own -rc pull requests after rc2 into
my for-next branch, so you won't be missing code that goes into later
rcs if you base your work on my for-next branch, so that's a second
alternative starting point for your shared pull requests should you
need it.

All of this causes me to bring up another point as well.  For-next
testing.  I've sent an email to Stephen to change his setup.  Right
now, he only pulls my for-next tag, so in order to get for-next
testing, I have to put things under my for-next tag, and this can cause
confusion and make people think I've accepted things I haven't if I
want to get early testing on code that's still waiting to be approved. 
I've requested that he change his setup to pull all of my branches on
k.o that start with k.o/for-next, so I might have one official k.o/for-
next branch for the code that's accepted, then I might create a
k.o/for-next-mlx5-pending for a branch that isn't yet approved and
merged into for-next, but it would still get for-next testing.  So, my
point here is that people shouldn't make assumptions about branches on
k.o.  Unless I've mailed the list to say the patches are accepted,
their presence in a branch on k.o is simply for advanced for-next
testing.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Jason Gunthorpe]

On Sat, Sep 23, 2017 at 11:03:25AM -0400, Doug Ledford wrote:

> That's my answer to people not necessarily wanting to wait until kernel
> n+1 to use features they put in their net driver in kernel n.  So, this
> applies to the mlx4, mlx5, cxgb3, cxgb4, bnxt_re, hns_roce, and i40iw
> drivers off the top of my head.

A different possible model would be to more broadly accept pull
requests for drivers and push the work to synchronize with other parts
of the tree directly to the driver folks, instead of trying to
coordinate it all yourself.

When a driver sends you a PR it will have to be 'clean' and be based
on accepted shared commits already in Linus's tree, in the net tree
and in the rdma-core tree. Ignoring shared commits, the PR can only
touch drivers/infiniband/hw/xxx. No uAPIs. Each driver team can decide
how they want to construct their PR within those guidelines.

Set a cut off of rc6 or something for these driver PRs to give time to
sort out any RDMA merge issues.

Run the core and ULP parts as a normal tree with a normal work flow,
don't integrate tricky driver PRs into the normal tree until the merge
window. Make sure this flow runs fast enough that the driver people can
pull from here when constructing their driver branch.

Run a single 'volatile' for-next tree that merges all pending
branches, including driver PRs. This is the early warning for
merge problems.

When the merge window comes around decide what to branches to send based
on what is in Linus's tree already. Maybe two PRs per merge window
will be needed to sort out dependencies, depends how fast net/etc are
at getting merged. I think other subsystems do this already?

The advantage is *you* are largely no longer worrying about timing or
monitoring the net tree or anything like that. The driver teams take
full responsibility to do that in a set time window. Either they post
an acceptable PR in time or they miss the merge window. Some scripting
is possible so you can determine on a daily basis when driver PRs become
'clean' and thus sendable to Linus. Eliminates some of the manual
coordination.

Another advantage is QA. When the driver teams QA their PR, they are
now QA'ing something with all dependent changes, and the configuration
they QA'd is recorded in the git history instead of being lost when
patches get rebased as they flow through patchworks. They have more
certainty what they will QA and they do not have to run so many
various speculative combinations.

I belive this is broadly similar to what arm-soc does for how they
manage the SOC maintainers, and deal with their crazy cross-tree
dependencies.

Review one of their presentations:

http://events.linuxfoundation.org/sites/events/files/slides/Maintaining%20a%20large%20kernel%20subsystem_0.pdf

And imagine things like SOC's are our drivers (eg omap == mlx)

arm-soc does a bit more than just this, their downstreams split
patches eg into fixes/fixes-non-critical, etc. RDMA could do that
do.

People are asking for more certainty, the best way I can think to
provide that is to let them run their own sub-tree within certain
fairly tight boundaries.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

On Sat, Sep 23, 2017 at 11:03:25AM -0400, Doug Ledford wrote:

<...>

>
> I want any shared pull requests to be based on nothing newer than an
> rc2 kernel.  I branch my for-next branch at rc2, so if your pull
> request is on something later it will pollute my main for-next branch
> to merge it.  I plan to merge my own -rc pull requests after rc2 into
> my for-next branch, so you won't be missing code that goes into later
> rcs if you base your work on my for-next branch, so that's a second
> alternative starting point for your shared pull requests should you
> need it.

From our experience, it is not feasible to demand shared pull
requests on "newer than an rc2 kernel". The magic of shared pull request
is in the fact that it is based on the same origin for both trees.

In our case the same origin is Linus's tree and his official -rcX, which
for sure will be in both trees.

Once shared pull request is based on that tree, it will apply cleanly
on both netdev and RDMA without pollution of the trees.

And again, shared pull request shouldn't be based on any patches
accepted to netdev and/or RDMA which are not in Linus's tree.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 4316 bytes --]

On Sun, Sep 24, 2017 at 01:56:27PM -0600, Jason Gunthorpe wrote:
> On Sat, Sep 23, 2017 at 11:03:25AM -0400, Doug Ledford wrote:
>
> > That's my answer to people not necessarily wanting to wait until kernel
> > n+1 to use features they put in their net driver in kernel n.  So, this
> > applies to the mlx4, mlx5, cxgb3, cxgb4, bnxt_re, hns_roce, and i40iw
> > drivers off the top of my head.
>
> A different possible model would be to more broadly accept pull
> requests for drivers and push the work to synchronize with other parts
> of the tree directly to the driver folks, instead of trying to
> coordinate it all yourself.
>
> When a driver sends you a PR it will have to be 'clean' and be based
> on accepted shared commits already in Linus's tree, in the net tree
> and in the rdma-core tree. Ignoring shared commits, the PR can only
> touch drivers/infiniband/hw/xxx. No uAPIs. Each driver team can decide
> how they want to construct their PR within those guidelines.
>
> Set a cut off of rc6 or something for these driver PRs to give time to
> sort out any RDMA merge issues.
>
> Run the core and ULP parts as a normal tree with a normal work flow,
> don't integrate tricky driver PRs into the normal tree until the merge
> window. Make sure this flow runs fast enough that the driver people can
> pull from here when constructing their driver branch.
>
> Run a single 'volatile' for-next tree that merges all pending
> branches, including driver PRs. This is the early warning for
> merge problems.
>
> When the merge window comes around decide what to branches to send based
> on what is in Linus's tree already. Maybe two PRs per merge window
> will be needed to sort out dependencies, depends how fast net/etc are
> at getting merged. I think other subsystems do this already?
>
> The advantage is *you* are largely no longer worrying about timing or
> monitoring the net tree or anything like that. The driver teams take
> full responsibility to do that in a set time window. Either they post
> an acceptable PR in time or they miss the merge window. Some scripting
> is possible so you can determine on a daily basis when driver PRs become
> 'clean' and thus sendable to Linus. Eliminates some of the manual
> coordination.
>
> Another advantage is QA. When the driver teams QA their PR, they are
> now QA'ing something with all dependent changes, and the configuration
> they QA'd is recorded in the git history instead of being lost when
> patches get rebased as they flow through patchworks. They have more
> certainty what they will QA and they do not have to run so many
> various speculative combinations.
>
> I belive this is broadly similar to what arm-soc does for how they
> manage the SOC maintainers, and deal with their crazy cross-tree
> dependencies.
>
> Review one of their presentations:
>
> http://events.linuxfoundation.org/sites/events/files/slides/Maintaining%20a%20large%20kernel%20subsystem_0.pdf
>
> And imagine things like SOC's are our drivers (eg omap == mlx)
>
> arm-soc does a bit more than just this, their downstreams split
> patches eg into fixes/fixes-non-critical, etc. RDMA could do that
> do.
>
> People are asking for more certainty, the best way I can think to
> provide that is to let them run their own sub-tree within certain
> fairly tight boundaries.

Jason,

I reread it a couple of times and still didn't find an answer for
myself. Why do we need it?

After LPC, I left with an impression that we have three main issues:
1. Overloaded maintainer who can't go without worries to PTO, especially
in this subsystem which is backed by distributed companies which work
around the globe and have a little in common with US holidays and vacations.
Next major holiday will be Christmas and at least two companies here
(and from previous years, Linus too) will continue to work.
2. Unclear guidelines to the QA.
3. Contributors to this subsystem are paid developers and not hobbyists.
It creates certain expectations from all participants.

So I believe that once we will solve the first issue, the rest will be easy.

Thanks

>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Christoph Hellwig]

On Sun, Sep 24, 2017 at 01:56:27PM -0600, Jason Gunthorpe wrote:
> Review one of their presentations:
> 
> http://events.linuxfoundation.org/sites/events/files/slides/Maintaining%20a%20large%20kernel%20subsystem_0.pdf
> 
> And imagine things like SOC's are our drivers (eg omap == mlx)

In case of hns it even is part of such a SOC :)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Doug Ledford]

[-- Attachment #1.1: Type: text/plain, Size: 1989 bytes --]

On 9/24/2017 11:59 PM, Leon Romanovsky wrote:
> On Sat, Sep 23, 2017 at 11:03:25AM -0400, Doug Ledford wrote:
> 
> <...>
> 
>>
>> I want any shared pull requests to be based on nothing newer than an
>> rc2 kernel.  I branch my for-next branch at rc2, so if your pull
>> request is on something later it will pollute my main for-next branch
>> to merge it.  I plan to merge my own -rc pull requests after rc2 into
>> my for-next branch, so you won't be missing code that goes into later
>> rcs if you base your work on my for-next branch, so that's a second
>> alternative starting point for your shared pull requests should you
>> need it.
> 
> From our experience, it is not feasible to demand shared pull
> requests on "newer than an rc2 kernel".

I said "nothing newer than an rc2 kernel".  Did you mean to leave out
the nothing?

> The magic of shared pull request
> is in the fact that it is based on the same origin for both trees.

Correct.  So, for instance, I'm opening up my for-next area today and it
will be based on a clean v4.14-rc2.  What I'm then asking for is that
subsequent driver shared pull requests be based on a v4.14-rc2 tree.
Your last shared pull request was mostly OK, but it was based on a
v4.13-rc4 kernel and so it would have simultaneously brought in your
patches and also all the changes between 4.13-rc2 and 4.13-rc4.

> In our case the same origin is Linus's tree and his official -rcX, which
> for sure will be in both trees.

Right, that what I want.

> Once shared pull request is based on that tree, it will apply cleanly
> on both netdev and RDMA without pollution of the trees.
> 
> And again, shared pull request shouldn't be based on any patches
> accepted to netdev and/or RDMA which are not in Linus's tree.

Exactly.


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG Key ID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 2453 bytes --]

On Mon, Sep 25, 2017 at 10:53:01AM -0400, Doug Ledford wrote:
> On 9/24/2017 11:59 PM, Leon Romanovsky wrote:
> > On Sat, Sep 23, 2017 at 11:03:25AM -0400, Doug Ledford wrote:
> >
> > <...>
> >
> >>
> >> I want any shared pull requests to be based on nothing newer than an
> >> rc2 kernel.  I branch my for-next branch at rc2, so if your pull
> >> request is on something later it will pollute my main for-next branch
> >> to merge it.  I plan to merge my own -rc pull requests after rc2 into
> >> my for-next branch, so you won't be missing code that goes into later
> >> rcs if you base your work on my for-next branch, so that's a second
> >> alternative starting point for your shared pull requests should you
> >> need it.
> >
> > From our experience, it is not feasible to demand shared pull
> > requests on "newer than an rc2 kernel".
>
> I said "nothing newer than an rc2 kernel".  Did you mean to leave out
> the nothing?

Most probably I translated it incorrectly. I wanted to say that all
shared pull requests will be -rc2, -rc3, e.t.c. and probably will never
be -rc1. Did you mean the same?

>
> > The magic of shared pull request
> > is in the fact that it is based on the same origin for both trees.
>
> Correct.  So, for instance, I'm opening up my for-next area today and it
> will be based on a clean v4.14-rc2.  What I'm then asking for is that
> subsequent driver shared pull requests be based on a v4.14-rc2 tree.
> Your last shared pull request was mostly OK, but it was based on a
> v4.13-rc4 kernel and so it would have simultaneously brought in your
> patches and also all the changes between 4.13-rc2 and 4.13-rc4.

Why is it "undesired behavior"? Anyhow git request-pull to Linus will
filter all patches which already exist in Linus's tree and you will get
merge of -rc fixes for free in your for-next.

>
> > In our case the same origin is Linus's tree and his official -rcX, which
> > for sure will be in both trees.
>
> Right, that what I want.
>
> > Once shared pull request is based on that tree, it will apply cleanly
> > on both netdev and RDMA without pollution of the trees.
> >
> > And again, shared pull request shouldn't be based on any patches
> > accepted to netdev and/or RDMA which are not in Linus's tree.
>
> Exactly.
>
>
> --
> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>     GPG Key ID: B826A3330E572FDD
>     Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
>




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Doug Ledford]

[-- Attachment #1.1: Type: text/plain, Size: 2953 bytes --]

On 9/25/2017 11:57 AM, Leon Romanovsky wrote:
> On Mon, Sep 25, 2017 at 10:53:01AM -0400, Doug Ledford wrote:
>> On 9/24/2017 11:59 PM, Leon Romanovsky wrote:
>>> On Sat, Sep 23, 2017 at 11:03:25AM -0400, Doug Ledford wrote:
>>>
>>> <...>
>>>
>>>>
>>>> I want any shared pull requests to be based on nothing newer than an
>>>> rc2 kernel.  I branch my for-next branch at rc2, so if your pull
>>>> request is on something later it will pollute my main for-next branch
>>>> to merge it.  I plan to merge my own -rc pull requests after rc2 into
>>>> my for-next branch, so you won't be missing code that goes into later
>>>> rcs if you base your work on my for-next branch, so that's a second
>>>> alternative starting point for your shared pull requests should you
>>>> need it.
>>>
>>> From our experience, it is not feasible to demand shared pull
>>> requests on "newer than an rc2 kernel".
>>
>> I said "nothing newer than an rc2 kernel".  Did you mean to leave out
>> the nothing?
> 
> Most probably I translated it incorrectly. I wanted to say that all
> shared pull requests will be -rc2, -rc3, e.t.c. and probably will never
> be -rc1. Did you mean the same?
> 
>>
>>> The magic of shared pull request
>>> is in the fact that it is based on the same origin for both trees.
>>
>> Correct.  So, for instance, I'm opening up my for-next area today and it
>> will be based on a clean v4.14-rc2.  What I'm then asking for is that
>> subsequent driver shared pull requests be based on a v4.14-rc2 tree.
>> Your last shared pull request was mostly OK, but it was based on a
>> v4.13-rc4 kernel and so it would have simultaneously brought in your
>> patches and also all the changes between 4.13-rc2 and 4.13-rc4.
> 
> Why is it "undesired behavior"? Anyhow git request-pull to Linus will
> filter all patches which already exist in Linus's tree and you will get
> merge of -rc fixes for free in your for-next.

You obviously have not been paying attention when Linus yells at me.
This is something he specifically does *not* want, and if you send me a
shared pull request that is based on, say, -rc4, then I can't merge it
into my for-next branch and instead of I have to carry a separate branch
and send a separate pull request just for that branch.

The reason is that Linus wants to be able to pull up gitk on my pull
request and see the changes I am submitting for easy review.  If I merge
a shared pull request that also includes an update to the -rc level of
my tree, then all those -rc patches get mixed into the gitk ordering and
it makes it hard for Linus to find just the patches he wants.  So, this
is a Linus issue, not so much my issue.  If he didn't care, I wouldn't
either, but this is the way it is.


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG Key ID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Jason Gunthorpe]

On Mon, Sep 25, 2017 at 07:23:55AM +0300, Leon Romanovsky wrote:

> I reread it a couple of times and still didn't find an answer for
> myself. Why do we need it?

The point is to make it easier for everyone to understand what is
going on by clearly documenting via git the actual dependencies that
are required, and to enable early QA on something that will actually
become merged without being rebased.

Handling cross tree dependencies is hard, and if Doug is saying it is
caused so much trouble he wants to basically stop doing it, then
exploring another way to handle it does make sense.

I think it would work better to send a complete workable PR than to
try and wrangle all the manual coordination on what should be merged
on top of what tree and so on.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 4350 bytes --]

On Mon, Sep 25, 2017 at 10:31:47PM -0400, Doug Ledford wrote:
> On 9/25/2017 11:57 AM, Leon Romanovsky wrote:
> > On Mon, Sep 25, 2017 at 10:53:01AM -0400, Doug Ledford wrote:
> >> On 9/24/2017 11:59 PM, Leon Romanovsky wrote:
> >>> On Sat, Sep 23, 2017 at 11:03:25AM -0400, Doug Ledford wrote:
> >>>
> >>> <...>
> >>>
> >>>>
> >>>> I want any shared pull requests to be based on nothing newer than an
> >>>> rc2 kernel.  I branch my for-next branch at rc2, so if your pull
> >>>> request is on something later it will pollute my main for-next branch
> >>>> to merge it.  I plan to merge my own -rc pull requests after rc2 into
> >>>> my for-next branch, so you won't be missing code that goes into later
> >>>> rcs if you base your work on my for-next branch, so that's a second
> >>>> alternative starting point for your shared pull requests should you
> >>>> need it.
> >>>
> >>> From our experience, it is not feasible to demand shared pull
> >>> requests on "newer than an rc2 kernel".
> >>
> >> I said "nothing newer than an rc2 kernel".  Did you mean to leave out
> >> the nothing?
> >
> > Most probably I translated it incorrectly. I wanted to say that all
> > shared pull requests will be -rc2, -rc3, e.t.c. and probably will never
> > be -rc1. Did you mean the same?
> >
> >>
> >>> The magic of shared pull request
> >>> is in the fact that it is based on the same origin for both trees.
> >>
> >> Correct.  So, for instance, I'm opening up my for-next area today and it
> >> will be based on a clean v4.14-rc2.  What I'm then asking for is that
> >> subsequent driver shared pull requests be based on a v4.14-rc2 tree.
> >> Your last shared pull request was mostly OK, but it was based on a
> >> v4.13-rc4 kernel and so it would have simultaneously brought in your
> >> patches and also all the changes between 4.13-rc2 and 4.13-rc4.
> >
> > Why is it "undesired behavior"? Anyhow git request-pull to Linus will
> > filter all patches which already exist in Linus's tree and you will get
> > merge of -rc fixes for free in your for-next.
>
> You obviously have not been paying attention when Linus yells at me.

I do, but probably we are understanding Linus's responses differently.

You probably mean the conversation after that pull request.
 * First round of RC fixes for 4.13
   https://marc.info/?l=linux-rdma&m=150039306716272&w=2

He started to complain about empty merge commit with v4.13-rc1, and continued with explanations
about difference between back and forward merges. From my point of view, my intention to base
shared pull request on latest -rcX is exactly forward (good) merge, because it ensures that
all our future submissions for net/net-next/rdma-rc/rdma-next are completely in sync and
ready to go.

> This is something he specifically does *not* want, and if you send me a
> shared pull request that is based on, say, -rc4, then I can't merge it
> into my for-next branch and instead of I have to carry a separate branch
> and send a separate pull request just for that branch.

I don't think so, but we can probably catch Linus at KS next month and
ask him directly.

>
> The reason is that Linus wants to be able to pull up gitk on my pull
> request and see the changes I am submitting for easy review.  If I merge
> a shared pull request that also includes an update to the -rc level of
> my tree, then all those -rc patches get mixed into the gitk ordering and
> it makes it hard for Linus to find just the patches he wants.  So, this
> is a Linus issue, not so much my issue.  If he didn't care, I wouldn't
> either, but this is the way it is.
>

I don't know how gitk presents git history, but the following command
"git log --graph --no-merges -- drivers/infiniband include/rdma/ include/uapi/rdma" works like
a charm and presents only RDMA related commits.

Other replies from Linus about late submission, compiler warnings, tree-wide changes,
request to separate pull requests, fix with strange name ("Add ...") are completely unrelated.

I read upto 19 Mar 2016, in that pull request Linus explained to us how we should create shared pull request.
Everything before is not relevant for shared pull requests.

Thanks

>
> --
> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>     GPG Key ID: B826A3330E572FDD
>     Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
>




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

On Mon, Sep 25, 2017 at 09:33:43PM -0600, Jason Gunthorpe wrote:
> On Mon, Sep 25, 2017 at 07:23:55AM +0300, Leon Romanovsky wrote:
>
> > I reread it a couple of times and still didn't find an answer for
> > myself. Why do we need it?
>
> The point is to make it easier for everyone to understand what is
> going on by clearly documenting via git the actual dependencies that
> are required, and to enable early QA on something that will actually
> become merged without being rebased.
>
> Handling cross tree dependencies is hard, and if Doug is saying it is
> caused so much trouble he wants to basically stop doing it, then
> exploring another way to handle it does make sense.
>
> I think it would work better to send a complete workable PR than to
> try and wrangle all the manual coordination on what should be merged
> on top of what tree and so on.

Right, it will work better on Linus's level, but it won't work on Doug
vs. Dave level. The complete PR means that your code should apply
cleanly on netdev and RDMA at the same time.

At some point of time, usually after -rc4 for bleeding edge devices,
the netdev and RDAM trees are so different in that area (for example mlx5_core)
that the chances to have clean PR are not high.

Thanks

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 5224 bytes --]

On Tue, Sep 26, 2017 at 08:13:49AM +0300, Leon Romanovsky wrote:
> On Mon, Sep 25, 2017 at 10:31:47PM -0400, Doug Ledford wrote:
> > On 9/25/2017 11:57 AM, Leon Romanovsky wrote:
> > > On Mon, Sep 25, 2017 at 10:53:01AM -0400, Doug Ledford wrote:
> > >> On 9/24/2017 11:59 PM, Leon Romanovsky wrote:
> > >>> On Sat, Sep 23, 2017 at 11:03:25AM -0400, Doug Ledford wrote:
> > >>>
> > >>> <...>
> > >>>
> > >>>>
> > >>>> I want any shared pull requests to be based on nothing newer than an
> > >>>> rc2 kernel.  I branch my for-next branch at rc2, so if your pull
> > >>>> request is on something later it will pollute my main for-next branch
> > >>>> to merge it.  I plan to merge my own -rc pull requests after rc2 into
> > >>>> my for-next branch, so you won't be missing code that goes into later
> > >>>> rcs if you base your work on my for-next branch, so that's a second
> > >>>> alternative starting point for your shared pull requests should you
> > >>>> need it.
> > >>>
> > >>> From our experience, it is not feasible to demand shared pull
> > >>> requests on "newer than an rc2 kernel".
> > >>
> > >> I said "nothing newer than an rc2 kernel".  Did you mean to leave out
> > >> the nothing?
> > >
> > > Most probably I translated it incorrectly. I wanted to say that all
> > > shared pull requests will be -rc2, -rc3, e.t.c. and probably will never
> > > be -rc1. Did you mean the same?
> > >
> > >>
> > >>> The magic of shared pull request
> > >>> is in the fact that it is based on the same origin for both trees.
> > >>
> > >> Correct.  So, for instance, I'm opening up my for-next area today and it
> > >> will be based on a clean v4.14-rc2.  What I'm then asking for is that
> > >> subsequent driver shared pull requests be based on a v4.14-rc2 tree.
> > >> Your last shared pull request was mostly OK, but it was based on a
> > >> v4.13-rc4 kernel and so it would have simultaneously brought in your
> > >> patches and also all the changes between 4.13-rc2 and 4.13-rc4.
> > >
> > > Why is it "undesired behavior"? Anyhow git request-pull to Linus will
> > > filter all patches which already exist in Linus's tree and you will get
> > > merge of -rc fixes for free in your for-next.
> >
> > You obviously have not been paying attention when Linus yells at me.
>
> I do, but probably we are understanding Linus's responses differently.
>
> You probably mean the conversation after that pull request.
>  * First round of RC fixes for 4.13
>    https://marc.info/?l=linux-rdma&m=150039306716272&w=2
>
> He started to complain about empty merge commit with v4.13-rc1, and continued with explanations
> about difference between back and forward merges. From my point of view, my intention to base
> shared pull request on latest -rcX is exactly forward (good) merge, because it ensures that
> all our future submissions for net/net-next/rdma-rc/rdma-next are completely in sync and
> ready to go.

And if we are talking about back-forward merges, the following merge is
a back one and it was mentioned by Linus as a bad type of merge which
should be avoided:
 * Merge tag 'v4.14-rc2' into k.o/for-next
   https://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma.git/commit/?h=k.o/for-next&id=0d9c2ff1c9f7f8b339fc42ac9763b28c71f1c115

Because your previous for-next was fully merged and new one was opened after
-rc2 and your initial pull request with fixes was accepted during in -rc1 too,
you was supposed simply reset your for-next to v4.14-rc2 tag
git reset --hard v4.14-rc2) as a starting point of for-next.

Thanks

>
> > This is something he specifically does *not* want, and if you send me a
> > shared pull request that is based on, say, -rc4, then I can't merge it
> > into my for-next branch and instead of I have to carry a separate branch
> > and send a separate pull request just for that branch.
>
> I don't think so, but we can probably catch Linus at KS next month and
> ask him directly.
>
> >
> > The reason is that Linus wants to be able to pull up gitk on my pull
> > request and see the changes I am submitting for easy review.  If I merge
> > a shared pull request that also includes an update to the -rc level of
> > my tree, then all those -rc patches get mixed into the gitk ordering and
> > it makes it hard for Linus to find just the patches he wants.  So, this
> > is a Linus issue, not so much my issue.  If he didn't care, I wouldn't
> > either, but this is the way it is.
> >
>
> I don't know how gitk presents git history, but the following command
> "git log --graph --no-merges -- drivers/infiniband include/rdma/ include/uapi/rdma" works like
> a charm and presents only RDMA related commits.
>
> Other replies from Linus about late submission, compiler warnings, tree-wide changes,
> request to separate pull requests, fix with strange name ("Add ...") are completely unrelated.
>
> I read upto 19 Mar 2016, in that pull request Linus explained to us how we should create shared pull request.
> Everything before is not relevant for shared pull requests.
>
> Thanks
>
> >
> > --
> > Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> >     GPG Key ID: B826A3330E572FDD
> >     Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
> >
>
>
>



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Doug Ledford]

[-- Attachment #1.1: Type: text/plain, Size: 6618 bytes --]

On 9/26/2017 2:56 AM, Leon Romanovsky wrote:
> On Tue, Sep 26, 2017 at 08:13:49AM +0300, Leon Romanovsky wrote:
>> On Mon, Sep 25, 2017 at 10:31:47PM -0400, Doug Ledford wrote:
>>> On 9/25/2017 11:57 AM, Leon Romanovsky wrote:
>>>> On Mon, Sep 25, 2017 at 10:53:01AM -0400, Doug Ledford wrote:
>>>>> On 9/24/2017 11:59 PM, Leon Romanovsky wrote:
>>>>>> On Sat, Sep 23, 2017 at 11:03:25AM -0400, Doug Ledford wrote:
>>>>>>
>>>>>> <...>
>>>>>>
>>>>>>>
>>>>>>> I want any shared pull requests to be based on nothing newer than an
>>>>>>> rc2 kernel.  I branch my for-next branch at rc2, so if your pull
>>>>>>> request is on something later it will pollute my main for-next branch
>>>>>>> to merge it.  I plan to merge my own -rc pull requests after rc2 into
>>>>>>> my for-next branch, so you won't be missing code that goes into later
>>>>>>> rcs if you base your work on my for-next branch, so that's a second
>>>>>>> alternative starting point for your shared pull requests should you
>>>>>>> need it.
>>>>>>
>>>>>> From our experience, it is not feasible to demand shared pull
>>>>>> requests on "newer than an rc2 kernel".
>>>>>
>>>>> I said "nothing newer than an rc2 kernel".  Did you mean to leave out
>>>>> the nothing?
>>>>
>>>> Most probably I translated it incorrectly. I wanted to say that all
>>>> shared pull requests will be -rc2, -rc3, e.t.c. and probably will never
>>>> be -rc1. Did you mean the same?
>>>>
>>>>>
>>>>>> The magic of shared pull request
>>>>>> is in the fact that it is based on the same origin for both trees.
>>>>>
>>>>> Correct.  So, for instance, I'm opening up my for-next area today and it
>>>>> will be based on a clean v4.14-rc2.  What I'm then asking for is that
>>>>> subsequent driver shared pull requests be based on a v4.14-rc2 tree.
>>>>> Your last shared pull request was mostly OK, but it was based on a
>>>>> v4.13-rc4 kernel and so it would have simultaneously brought in your
>>>>> patches and also all the changes between 4.13-rc2 and 4.13-rc4.
>>>>
>>>> Why is it "undesired behavior"? Anyhow git request-pull to Linus will
>>>> filter all patches which already exist in Linus's tree and you will get
>>>> merge of -rc fixes for free in your for-next.
>>>
>>> You obviously have not been paying attention when Linus yells at me.
>>
>> I do, but probably we are understanding Linus's responses differently.
>>
>> You probably mean the conversation after that pull request.
>>  * First round of RC fixes for 4.13
>>    https://marc.info/?l=linux-rdma&m=150039306716272&w=2

More than just that conversation.  There have been offline conversations
too.

>> He started to complain about empty merge commit with v4.13-rc1, and continued with explanations
>> about difference between back and forward merges. From my point of view, my intention to base
>> shared pull request on latest -rcX is exactly forward (good) merge, because it ensures that
>> all our future submissions for net/net-next/rdma-rc/rdma-next are completely in sync and
>> ready to go.

This is why, when I submit a pull request for -rc, I also merge it into
for-next.  It keeps this up to date as you say, without needing to pull
in all of the -rc, so it gives Linus an easier history to look at.  It
does not, however, sync up the net side of things.

> And if we are talking about back-forward merges, the following merge is
> a back one and it was mentioned by Linus as a bad type of merge which
> should be avoided:
>  * Merge tag 'v4.14-rc2' into k.o/for-next
>    https://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma.git/commit/?h=k.o/for-next&id=0d9c2ff1c9f7f8b339fc42ac9763b28c71f1c115
> 
> Because your previous for-next was fully merged

It was...

> and new one was opened after
> -rc2

It was...

> and your initial pull request with fixes was accepted during in -rc1 too,

It was...

> you was supposed simply reset your for-next to v4.14-rc2 tag
> git reset --hard v4.14-rc2) as a starting point of for-next.

I didn't think this would work on k.o (and it's why I did the merge),
but it does work.  The reason I didn't think it would work is that even
though v4.14-rc2 and my branch merged with v4.14-rc2 have the same final
image, their path to get there (as shown by git log --oneline
--topo-order) would be different.  As such, I expected the k.o git
server to kick it as a non-fast-forward update when I attempted to push
after having used git reset --hard v4.14-rc2.  But, it didn't, it took
it, and a git pull of the tree before the reset and after the reset was
totally non-eventful.  So, I won't do that next time.

> Thanks
> 
>>
>>> This is something he specifically does *not* want, and if you send me a
>>> shared pull request that is based on, say, -rc4, then I can't merge it
>>> into my for-next branch and instead of I have to carry a separate branch
>>> and send a separate pull request just for that branch.
>>
>> I don't think so, but we can probably catch Linus at KS next month and
>> ask him directly.
>>
>>>
>>> The reason is that Linus wants to be able to pull up gitk on my pull
>>> request and see the changes I am submitting for easy review.  If I merge
>>> a shared pull request that also includes an update to the -rc level of
>>> my tree, then all those -rc patches get mixed into the gitk ordering and
>>> it makes it hard for Linus to find just the patches he wants.  So, this
>>> is a Linus issue, not so much my issue.  If he didn't care, I wouldn't
>>> either, but this is the way it is.
>>>
>>
>> I don't know how gitk presents git history, but the following command
>> "git log --graph --no-merges -- drivers/infiniband include/rdma/ include/uapi/rdma" works like
>> a charm and presents only RDMA related commits.
>>
>> Other replies from Linus about late submission, compiler warnings, tree-wide changes,
>> request to separate pull requests, fix with strange name ("Add ...") are completely unrelated.
>>
>> I read upto 19 Mar 2016, in that pull request Linus explained to us how we should create shared pull request.
>> Everything before is not relevant for shared pull requests.
>>
>> Thanks
>>
>>>
>>> --
>>> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>>>     GPG Key ID: B826A3330E572FDD
>>>     Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
>>>
>>
>>
>>
> 
> 


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG Key ID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Doug Ledford]

[-- Attachment #1.1: Type: text/plain, Size: 302 bytes --]

On 9/26/2017 12:03 PM, Doug Ledford wrote:
> So, I won't do that next time.

That meaning the useless merge commit.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG Key ID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

On Tue, Sep 26, 2017 at 12:03:01PM -0400, Doug Ledford wrote:

<...>

> >> He started to complain about empty merge commit with v4.13-rc1, and continued with explanations
> >> about difference between back and forward merges. From my point of view, my intention to base
> >> shared pull request on latest -rcX is exactly forward (good) merge, because it ensures that
> >> all our future submissions for net/net-next/rdma-rc/rdma-next are completely in sync and
> >> ready to go.
>
> This is why, when I submit a pull request for -rc, I also merge it into
> for-next.  It keeps this up to date as you say, without needing to pull
> in all of the -rc, so it gives Linus an easier history to look at.  It
> does not, however, sync up the net side of things.
>

Doug,

What is the bottom line for the bases for shared pull requests?

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Doug Ledford]

[-- Attachment #1.1: Type: text/plain, Size: 1178 bytes --]

On 9/26/2017 12:21 PM, Leon Romanovsky wrote:
> On Tue, Sep 26, 2017 at 12:03:01PM -0400, Doug Ledford wrote:
> 
> <...>
> 
>>>> He started to complain about empty merge commit with v4.13-rc1, and continued with explanations
>>>> about difference between back and forward merges. From my point of view, my intention to base
>>>> shared pull request on latest -rcX is exactly forward (good) merge, because it ensures that
>>>> all our future submissions for net/net-next/rdma-rc/rdma-next are completely in sync and
>>>> ready to go.
>>
>> This is why, when I submit a pull request for -rc, I also merge it into
>> for-next.  It keeps this up to date as you say, without needing to pull
>> in all of the -rc, so it gives Linus an easier history to look at.  It
>> does not, however, sync up the net side of things.
>>
> 
> Doug,
> 
> What is the bottom line for the bases for shared pull requests?
> 
> Thanks
> 

Just base it on either A) my current for-next or B) -rc2.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG Key ID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1361 bytes --]

On Tue, Sep 26, 2017 at 12:23:35PM -0400, Doug Ledford wrote:
> On 9/26/2017 12:21 PM, Leon Romanovsky wrote:
> > On Tue, Sep 26, 2017 at 12:03:01PM -0400, Doug Ledford wrote:
> >
> > <...>
> >
> >>>> He started to complain about empty merge commit with v4.13-rc1, and continued with explanations
> >>>> about difference between back and forward merges. From my point of view, my intention to base
> >>>> shared pull request on latest -rcX is exactly forward (good) merge, because it ensures that
> >>>> all our future submissions for net/net-next/rdma-rc/rdma-next are completely in sync and
> >>>> ready to go.
> >>
> >> This is why, when I submit a pull request for -rc, I also merge it into
> >> for-next.  It keeps this up to date as you say, without needing to pull
> >> in all of the -rc, so it gives Linus an easier history to look at.  It
> >> does not, however, sync up the net side of things.
> >>
> >
> > Doug,
> >
> > What is the bottom line for the bases for shared pull requests?
> >
> > Thanks
> >
>
> Just base it on either A) my current for-next or B) -rc2.

OK, in this cycle, we will have it on -rc2, but I don't know if it is always
feasible.

Thanks

>
> --
> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>     GPG Key ID: B826A3330E572FDD
>     Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD
>




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Jason Gunthorpe]

On Tue, Sep 26, 2017 at 08:28:05AM +0300, Leon Romanovsky wrote:

> At some point of time, usually after -rc4 for bleeding edge devices,
> the netdev and RDAM trees are so different in that area (for example mlx5_core)
> that the chances to have clean PR are not high.

Well, exactly. That is the entire problem.

And it is solvable, with work and understanding. You need to use git
properly, and create a PR that starts out with a merge of the netdev
and rdma trees from the 'right point'.

But you absolutely can technically construct a driver PR that passes
QA, includes the rdma and netdev components, and all of the driver
related patches.

And I think the result would be better if the driver teams did this
directly instead of trying to explain to Doug what it is they need and
then having him do it for them.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ANNOUNCE] Shared pull requests (was Re: Workflow for i40iw patch submissions)

[Jason Gunthorpe]

On Tue, Sep 26, 2017 at 12:03:01PM -0400, Doug Ledford wrote:
> >> You probably mean the conversation after that pull request.
> >>  * First round of RC fixes for 4.13
> >>    https://marc.info/?l=linux-rdma&m=150039306716272&w=2
> 
> More than just that conversation.  There have been offline conversations
> too.

Other trees are certainly not doing something as restrictive as 'only
base branches off rc2'...

branches should be based off a point that makes the patches work, and
ideally, the point at which the patches were actually tested.

Patches should never, ever be applied against a branch where they are
known to be broken (either compile wise, or functionality wise)
because of missing dependent patches from other trees.

This is all done to make future bisection work properly.

Overall, the only solution is to use a variety of bases on a variety
of branches. This is what other trees are doing, and AFAIK Linus
accepts it.

So, in your new multi-branch work flow, when patches come to you, you
have to decide if they can be safely applied to an existing branch (eg
'fixes' in arm-soc nomenclature), or if a new feature branch, with a
new base is required.

You *will* find you have branch bases all over the place due to the
co-ordination with other parts of the tree.. And you may find you have
feature branches based on your own branches too if things get a bit
complex.

> I didn't think this would work on k.o (and it's why I did the merge),
> but it does work.  The reason I didn't think it would work is that
> even

The multi-branch work flow is less of a 'force push' more of a 'delete
the banch once merged', 'create a new branch at a choosen starting
point'. The fact the two branch have the same name means most people
just handle it with 'git reset --hard' and 'git push -f'..

But it is worth while to have in mind the overall picture that the
multi-branch work flow is 'create a branch at good base & apply
patches', 'merge the branch', 'discard the branch'.. repeat..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html