* Re: [ofa-general] Further 2.6.23 merge plans...
2007-07-12 23:15 ` Further 2.6.23 merge plans Roland Dreier
@ 2007-07-13 0:17 ` Hal Rosenstock
2007-07-13 1:14 ` Sean Hefty
` (4 subsequent siblings)
5 siblings, 0 replies; 31+ messages in thread
From: Hal Rosenstock @ 2007-07-13 0:17 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-kernel, general
On Thu, 2007-07-12 at 19:15, Roland Dreier wrote:
> As you can see, I just sent my first 2.6.23 pull request for Linus.
> There are still a few more things I plan to do in before the merge
> window closes (in ~10 days):
>
> - Write a patch to add P_Key handling to user_mad in the way we
> discussed (add an ioctl to enable P_Key mode without breaking old
> apps) -- I hope to do this tomorrow so we can get some review and
> testing before merging it.
Unfortunately, I'll mostly just be able to review it. Not sure how much
testing I will be able to do but we'll see...
-- Hal
> - Take a look at Sean's local SA caching patches. I merged
> everything else from Sean's tree, but I'm still undecided about
> these. I haven't read them carefully yet, but even aside from that
> I don't have a good feeling about whether there's consensus about
> this yet. Any opinions about merging, for or against, would be
> appreciated here.
>
> - Merge up pending hardware driver changes, including the cxgb3 and
> ehca patches I have in my queue, plus Jack's catastrophic error
> patch for mlx4.
>
> - Try to get to resolution on the IPoIB "CM without SRQ" solution.
>
> Also, if there's something I didn't list and didn't already include in
> the tree I asked Linus to pull, please remind me. I probably dropped it.
>
> - R.
> _______________________________________________
> general mailing list
> general@lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Further 2.6.23 merge plans...
2007-07-12 23:15 ` Further 2.6.23 merge plans Roland Dreier
2007-07-13 0:17 ` [ofa-general] " Hal Rosenstock
@ 2007-07-13 1:14 ` Sean Hefty
[not found] ` <15ddcffd0707172020j5b68fcb2v7d3ca77863998020@mail.gmail.com>
2007-07-13 5:47 ` Michael S. Tsirkin
` (3 subsequent siblings)
5 siblings, 1 reply; 31+ messages in thread
From: Sean Hefty @ 2007-07-13 1:14 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-kernel, general
> - Take a look at Sean's local SA caching patches. I merged
> everything else from Sean's tree, but I'm still undecided about
> these. I haven't read them carefully yet, but even aside from that
> I don't have a good feeling about whether there's consensus about
> this yet. Any opinions about merging, for or against, would be
> appreciated here.
Obviously I'm biased here, but we've definitely seen local caching of
path records (PR) greatly improve performance for large MPI job runs.
(Our largest jobs wouldn't run without it.) The development of the
feature was requested and paid for by the US national labs.
Infinicon/Silverstorm/QLogic also had this feature in their IB stack for
scalability reasons as well. PR caching is done in the stack today by
IPoIB.
The implementation is hidden under the current kernel ib_sa interface,
is disabled by default, and automatically fails over to standard PR
queries if needed. Removing the cache later should be fairly easy.
But to be fair, it will be difficult to enable both QoS and local PR
caching. To me, this would be the strongest reason against using it.
However, QoS places additional burden on the SA, which will make scaling
even more challenging.
- Sean
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Further 2.6.23 merge plans...
2007-07-12 23:15 ` Further 2.6.23 merge plans Roland Dreier
2007-07-13 0:17 ` [ofa-general] " Hal Rosenstock
2007-07-13 1:14 ` Sean Hefty
@ 2007-07-13 5:47 ` Michael S. Tsirkin
2007-07-13 18:14 ` Roland Dreier
2007-07-13 18:56 ` [ofa-general] " Shirley Ma
` (2 subsequent siblings)
5 siblings, 1 reply; 31+ messages in thread
From: Michael S. Tsirkin @ 2007-07-13 5:47 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-kernel, general
> Also, if there's something I didn't list and didn't already include in
> the tree I asked Linus to pull, please remind me. I probably dropped it.
Any plans to do something with multiple EQ support in mthca?
--
MST
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Further 2.6.23 merge plans...
2007-07-13 5:47 ` Michael S. Tsirkin
@ 2007-07-13 18:14 ` Roland Dreier
2007-07-13 18:50 ` [ofa-general] " Shirley Ma
2007-07-14 17:54 ` Michael S. Tsirkin
0 siblings, 2 replies; 31+ messages in thread
From: Roland Dreier @ 2007-07-13 18:14 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: linux-kernel, general
> Any plans to do something with multiple EQ support in mthca?
I haven't done any work on it or seen anything from anyone else, so I
expect this will have to wait for 2.6.24.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Re: Further 2.6.23 merge plans...
2007-07-13 18:14 ` Roland Dreier
@ 2007-07-13 18:50 ` Shirley Ma
2007-07-17 18:06 ` Roland Dreier
2007-07-14 17:54 ` Michael S. Tsirkin
1 sibling, 1 reply; 31+ messages in thread
From: Shirley Ma @ 2007-07-13 18:50 UTC (permalink / raw)
To: Roland Dreier; +Cc: general, general-bounces, linux-kernel, Michael S. Tsirkin
Hello Roland,
> > Any plans to do something with multiple EQ support in mthca?
>
> I haven't done any work on it or seen anything from anyone else, so I
> expect this will have to wait for 2.6.24.
We are working on IPoIB to use multiple EQ for multiple
links/connetions scalability. Does this mean this will wait for 2.6.24?
Thanks
Shirley
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Re: Further 2.6.23 merge plans...
2007-07-13 18:50 ` [ofa-general] " Shirley Ma
@ 2007-07-17 18:06 ` Roland Dreier
0 siblings, 0 replies; 31+ messages in thread
From: Roland Dreier @ 2007-07-17 18:06 UTC (permalink / raw)
To: Shirley Ma; +Cc: general, general-bounces, linux-kernel, Michael S. Tsirkin
> We are working on IPoIB to use multiple EQ for multiple
> links/connetions scalability. Does this mean this will wait for 2.6.24?
I think so -- I don't want to merge something that first appears in
the last few days of the merge window. The idea is to get your stuff
queued up *before* the merge window opens.
- R.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Further 2.6.23 merge plans...
2007-07-13 18:14 ` Roland Dreier
2007-07-13 18:50 ` [ofa-general] " Shirley Ma
@ 2007-07-14 17:54 ` Michael S. Tsirkin
[not found] ` <OF72F6B9D1.F60C4EEF-ON8725731A.00506757-8825731A.0024BD1C@us.ibm.com>
2007-07-16 16:42 ` Roland Dreier
1 sibling, 2 replies; 31+ messages in thread
From: Michael S. Tsirkin @ 2007-07-14 17:54 UTC (permalink / raw)
To: Roland Dreier; +Cc: Michael S. Tsirkin, linux-kernel, general
> Quoting Roland Dreier <rdreier@cisco.com>:
> Subject: Re: Further 2.6.23 merge plans...
>
> > Any plans to do something with multiple EQ support in mthca?
>
> I haven't done any work on it or seen anything from anyone else, so I
> expect this will have to wait for 2.6.24.
I'm surprised to hear this. How about this:
http://lists.openfabrics.org/pipermail/general/2007-May/035757.html
--
MST
^ permalink raw reply [flat|nested] 31+ messages in thread
[parent not found: <OF72F6B9D1.F60C4EEF-ON8725731A.00506757-8825731A.0024BD1C@us.ibm.com>]
* Re: Further 2.6.23 merge plans...
2007-07-14 17:54 ` Michael S. Tsirkin
[not found] ` <OF72F6B9D1.F60C4EEF-ON8725731A.00506757-8825731A.0024BD1C@us.ibm.com>
@ 2007-07-16 16:42 ` Roland Dreier
2007-07-16 20:05 ` Michael S. Tsirkin
1 sibling, 1 reply; 31+ messages in thread
From: Roland Dreier @ 2007-07-16 16:42 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: linux-kernel, general
> > I haven't done any work on it or seen anything from anyone else, so I
> > expect this will have to wait for 2.6.24.
> I'm surprised to hear this. How about this:
> http://lists.openfabrics.org/pipermail/general/2007-May/035757.html
Sure, I remember that. But I haven't seen anything to suggest that
anyone has given any further thought to the issues that were raised in
that thread.
- R.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Further 2.6.23 merge plans...
2007-07-16 16:42 ` Roland Dreier
@ 2007-07-16 20:05 ` Michael S. Tsirkin
2007-07-17 17:53 ` Roland Dreier
0 siblings, 1 reply; 31+ messages in thread
From: Michael S. Tsirkin @ 2007-07-16 20:05 UTC (permalink / raw)
To: Roland Dreier; +Cc: Michael S. Tsirkin, linux-kernel, general
> Quoting Roland Dreier <rdreier@cisco.com>:
> Subject: Re: Further 2.6.23 merge plans...
>
> > > I haven't done any work on it or seen anything from anyone else, so I
> > > expect this will have to wait for 2.6.24.
>
> > I'm surprised to hear this. How about this:
> > http://lists.openfabrics.org/pipermail/general/2007-May/035757.html
>
> Sure, I remember that. But I haven't seen anything to suggest that
> anyone has given any further thought to the issues that were raised in
> that thread.
Well, the only issue I recall is about the # of EQs we want to allocate.
Was there something else?
Maybe code can be merged as-is (2 EQs) and the number be tuned
later as applications start using vectors?
--
MST
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Further 2.6.23 merge plans...
2007-07-12 23:15 ` Further 2.6.23 merge plans Roland Dreier
` (2 preceding siblings ...)
2007-07-13 5:47 ` Michael S. Tsirkin
@ 2007-07-13 18:56 ` Shirley Ma
2007-07-16 16:47 ` Roland Dreier
2007-07-15 12:26 ` Tziporet Koren
2007-07-17 18:07 ` Roland Dreier
5 siblings, 1 reply; 31+ messages in thread
From: Shirley Ma @ 2007-07-13 18:56 UTC (permalink / raw)
To: Roland Dreier; +Cc: general, general-bounces, linux-kernel
Hello Roland,
FYI, we are working on several IPoIB performance improvement
patches which are not on the list. Some of the patches are under test,
some of the patches are going to be submitted soon. They are:
1. skb aggregations for both dev xmit(networking layer) and IPoIB send
(it will be submitted soon, for both UD and RC mode)
2. multiple interrupt vectors in IPoIB for multiple links scalability
(working on patch for both UD and RC mode)
3. split CQ and send completion aggregation (for both UD and RC mode)
4. LRO for IPoIB when generic LRO is available in networking layer. (UD
mode only)
Some of them might be made in 2.6.23 timeline, some of them might
not, it depends on our test progress and community review feedback.
Thanks
Shirley
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Further 2.6.23 merge plans...
2007-07-13 18:56 ` [ofa-general] " Shirley Ma
@ 2007-07-16 16:47 ` Roland Dreier
0 siblings, 0 replies; 31+ messages in thread
From: Roland Dreier @ 2007-07-16 16:47 UTC (permalink / raw)
To: Shirley Ma; +Cc: general, general-bounces, linux-kernel
> FYI, we are working on several IPoIB performance improvement
> patches which are not on the list. Some of the patches are under test,
> some of the patches are going to be submitted soon. They are:
There is less than a week left in the merge window, and none of these
changes has been reviewed yet. So being realistic, I don't think we
can expect to get any of this into 2.6.23.
- R.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Further 2.6.23 merge plans...
2007-07-12 23:15 ` Further 2.6.23 merge plans Roland Dreier
` (3 preceding siblings ...)
2007-07-13 18:56 ` [ofa-general] " Shirley Ma
@ 2007-07-15 12:26 ` Tziporet Koren
2007-07-16 16:42 ` Roland Dreier
2007-07-17 18:07 ` Roland Dreier
5 siblings, 1 reply; 31+ messages in thread
From: Tziporet Koren @ 2007-07-15 12:26 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-kernel, general
Roland Dreier wrote:
> As you can see, I just sent my first 2.6.23 pull request for Linus.
> There are still a few more things I plan to do in before the merge
> window closes (in ~10 days):
>
>
Till when can we insert mlx4 with FMRs?
Tziporet
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Further 2.6.23 merge plans...
2007-07-12 23:15 ` Further 2.6.23 merge plans Roland Dreier
` (4 preceding siblings ...)
2007-07-15 12:26 ` Tziporet Koren
@ 2007-07-17 18:07 ` Roland Dreier
2007-07-17 20:43 ` Matt Leininger
2007-07-17 21:44 ` Michael S. Tsirkin
5 siblings, 2 replies; 31+ messages in thread
From: Roland Dreier @ 2007-07-17 18:07 UTC (permalink / raw)
To: linux-kernel; +Cc: general
> - Take a look at Sean's local SA caching patches. I merged
> everything else from Sean's tree, but I'm still undecided about
> these. I haven't read them carefully yet, but even aside from that
> I don't have a good feeling about whether there's consensus about
> this yet. Any opinions about merging, for or against, would be
> appreciated here.
Does anyone other than Sean have an opinion here? If you want this
feature, if you've tested it, if you don't think it's ready yet,
whatever, please speak up -- I don't feel comfortable making a
decision on my own here (although I will if I have to).
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Further 2.6.23 merge plans...
2007-07-17 18:07 ` Roland Dreier
@ 2007-07-17 20:43 ` Matt Leininger
2007-07-17 20:45 ` Roland Dreier
2007-07-17 21:44 ` Michael S. Tsirkin
1 sibling, 1 reply; 31+ messages in thread
From: Matt Leininger @ 2007-07-17 20:43 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-kernel, general
On Tue, 2007-07-17 at 11:07 -0700, Roland Dreier wrote:
> > - Take a look at Sean's local SA caching patches. I merged
> > everything else from Sean's tree, but I'm still undecided about
> > these. I haven't read them carefully yet, but even aside from that
> > I don't have a good feeling about whether there's consensus about
> > this yet. Any opinions about merging, for or against, would be
> > appreciated here.
>
> Does anyone other than Sean have an opinion here? If you want this
> feature, if you've tested it, if you don't think it's ready yet,
> whatever, please speak up -- I don't feel comfortable making a
> decision on my own here (although I will if I have to).
Roland,
I would like to see these features moved upstream. DOE funded this
work as part of the items we see needing on our large scale IB
deployment (both present and future). So from at least one big customer
perspective we see this as useful.
I'll let others comment on specific code/implementation issues.
Thanks,
- Matt
> _______________________________________________
> general mailing list
> general@lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
--
Matt Leininger, Ph.D.
Lawrence Livermore National Laboratory
leininger2@llnl.gov
V 925-422-4110
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Further 2.6.23 merge plans...
2007-07-17 20:43 ` Matt Leininger
@ 2007-07-17 20:45 ` Roland Dreier
2007-07-18 6:36 ` Or Gerlitz
0 siblings, 1 reply; 31+ messages in thread
From: Roland Dreier @ 2007-07-17 20:45 UTC (permalink / raw)
To: leininger2; +Cc: linux-kernel, general
> I would like to see these features moved upstream. DOE funded this
> work as part of the items we see needing on our large scale IB
> deployment (both present and future). So from at least one big customer
> perspective we see this as useful.
Does your reference to "present deployment" mean you are running this
code now?
- R.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Further 2.6.23 merge plans...
2007-07-17 20:45 ` Roland Dreier
@ 2007-07-18 6:36 ` Or Gerlitz
0 siblings, 0 replies; 31+ messages in thread
From: Or Gerlitz @ 2007-07-18 6:36 UTC (permalink / raw)
To: Roland Dreier; +Cc: leininger2, linux-kernel, general
Roland Dreier wrote:
> > I would like to see these features moved upstream. DOE funded this
> > work as part of the items we see needing on our large scale IB
> > deployment (both present and future). So from at least one big customer
> > perspective we see this as useful.
>
> Does your reference to "present deployment" mean you are running this
> code now?
Indeed, my understanding is that the DOE uses an Open MPI device (I
think its called PTE) which is implemented directly over libibverbs and
hence no path queries are issued at all, if this is indeed the case, for
them its more of a "for-the-future" thing.
Or.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Further 2.6.23 merge plans...
2007-07-17 18:07 ` Roland Dreier
2007-07-17 20:43 ` Matt Leininger
@ 2007-07-17 21:44 ` Michael S. Tsirkin
2007-07-18 7:34 ` [ofa-general] " Tziporet Koren
1 sibling, 1 reply; 31+ messages in thread
From: Michael S. Tsirkin @ 2007-07-17 21:44 UTC (permalink / raw)
To: Roland Dreier; +Cc: linux-kernel, general
> Quoting Roland Dreier <rdreier@cisco.com>:
> Subject: Re: Further 2.6.23 merge plans...
>
> > - Take a look at Sean's local SA caching patches. I merged
> > everything else from Sean's tree, but I'm still undecided about
> > these. I haven't read them carefully yet, but even aside from that
> > I don't have a good feeling about whether there's consensus about
> > this yet. Any opinions about merging, for or against, would be
> > appreciated here.
>
> Does anyone other than Sean have an opinion here? If you want this
> feature, if you've tested it, if you don't think it's ready yet,
> whatever, please speak up -- I don't feel comfortable making a
> decision on my own here (although I will if I have to).
We have the patches applied in ofed 1.2.c with default module parameter set to
caching disabled (ofed 1.2 had a different version of the patches, but caching
is disabled by default there, too). At least in this configuration
(caching disabled), all issues I've seen seem to be fixed now, and tests seem to
be running smoothly.
So I think it's safe to merge it up if the module parameter
is set to cache disabled by default.
No idea what happens if it's enabled though :)
--
MST
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Re: Further 2.6.23 merge plans...
2007-07-17 21:44 ` Michael S. Tsirkin
@ 2007-07-18 7:34 ` Tziporet Koren
2007-07-18 7:38 ` Michael S. Tsirkin
0 siblings, 1 reply; 31+ messages in thread
From: Tziporet Koren @ 2007-07-18 7:34 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Roland Dreier, linux-kernel, general
Michael S. Tsirkin wrote:
> We have the patches applied in ofed 1.2.c with default module parameter set to
> caching disabled (ofed 1.2 had a different version of the patches, but caching
> is disabled by default there, too). At least in this configuration
> (caching disabled), all issues I've seen seem to be fixed now, and tests seem to
> be running smoothly.
>
As far as I know Intel run with SA cache enabled on large clusters with
Intel MPI
Tziporet
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Re: Further 2.6.23 merge plans...
2007-07-18 7:34 ` [ofa-general] " Tziporet Koren
@ 2007-07-18 7:38 ` Michael S. Tsirkin
2007-07-18 8:48 ` Tziporet Koren
0 siblings, 1 reply; 31+ messages in thread
From: Michael S. Tsirkin @ 2007-07-18 7:38 UTC (permalink / raw)
To: Tziporet Koren; +Cc: Michael S. Tsirkin, Roland Dreier, linux-kernel, general
> Quoting Tziporet Koren <tziporet@dev.mellanox.co.il>:
> Subject: Re: [ofa-general] Re: Further 2.6.23 merge plans...
>
> Michael S. Tsirkin wrote:
> >We have the patches applied in ofed 1.2.c with default module parameter set
> >to caching disabled (ofed 1.2 had a different version of the patches, but
> >caching is disabled by default there, too). At least
> >in this configuration (caching disabled), all issues I've seen seem to be
> >fixed now, and tests seem to be running smoothly.
>
> As far as I know Intel run with SA cache enabled on large clusters with
> Intel MPI
With OFED 1.2 version of the code, right?
--
MST
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Re: Further 2.6.23 merge plans...
2007-07-18 7:38 ` Michael S. Tsirkin
@ 2007-07-18 8:48 ` Tziporet Koren
2007-07-18 16:16 ` Sean Hefty
0 siblings, 1 reply; 31+ messages in thread
From: Tziporet Koren @ 2007-07-18 8:48 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: Tziporet Koren, Roland Dreier, linux-kernel, general
Michael S. Tsirkin wrote:
>> As far as I know Intel run with SA cache enabled on large clusters with
>> Intel MPI
>>
>
> With OFED 1.2 version of the code, right?
>
>
Yes.
But maybe they also used the new module - Sean?
^ permalink raw reply [flat|nested] 31+ messages in thread
* RE: [ofa-general] Re: Further 2.6.23 merge plans...
2007-07-18 8:48 ` Tziporet Koren
@ 2007-07-18 16:16 ` Sean Hefty
2007-07-18 16:20 ` Roland Dreier
0 siblings, 1 reply; 31+ messages in thread
From: Sean Hefty @ 2007-07-18 16:16 UTC (permalink / raw)
To: 'Tziporet Koren', Michael S. Tsirkin
Cc: Roland Dreier, general, linux-kernel
>> With OFED 1.2 version of the code, right?
>>
>>
>Yes.
>But maybe they also used the new module - Sean?
We actually use the OFED 1.2 version. So, this feature is in use, but not this
specific implementation.
- Sean
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [ofa-general] Re: Further 2.6.23 merge plans...
2007-07-18 16:16 ` Sean Hefty
@ 2007-07-18 16:20 ` Roland Dreier
0 siblings, 0 replies; 31+ messages in thread
From: Roland Dreier @ 2007-07-18 16:20 UTC (permalink / raw)
To: Sean Hefty
Cc: 'Tziporet Koren', Michael S. Tsirkin, general, linux-kernel
> We actually use the OFED 1.2 version. So, this feature is in use, but not this
> specific implementation.
Hmm... how much testing has the implementation being proposed for
merging actually had?
It might still be OK if the answer is that it hasn't been tested at
scale but that the basic code works and should behave the same as the
code that was tested because the underlying design is the same... is
at least that much true?
^ permalink raw reply [flat|nested] 31+ messages in thread