[GIT PULL] please pull infiniband.git

[GIT PULL] please pull infiniband.git

[Roland Dreier]

Linus, please pull from

    master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus

This tree is also available from kernel.org mirrors at:

    git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus

This will get the first batch of changes for the 2.6.23 merge window:

Andrew Morton (1):
      IB: Fix ib_umem_get() when npages == 0

Arthur Jones (3):
      IB/ipath: Update MAINTAINERS entry
      IB/ipath: Test interrupts at driver startup
      IB/ipath: Remove bogus RD_ATOMIC checks from modify_qp

Bryan O'Sullivan (1):
      IB/ipath: Include <linux/vmalloc.h> to fix ppc64 build

Dave Olson (5):
      IB/ipath: Support the IBA6110 revision 4
      IB/ipath: Fix the mtrr_add args for chips with 2 buffer sizes
      IB/ipath: Use S_ABORT not cancel and abort on exit freeze mode after recovery
      IB/ipath: Be more cautious about coming out of freeze mode
      IB/ipath: Change version wording to be less confusing with release number

Dotan Barak (2):
      mlx4_core: Get the maximum message size from reported device capabilities
      IB/core: Take sizeof the correct pointer when calling kmalloc()

Hal Rosenstock (1):
      IB/mad: Enhance SMI for switch support

Hoang-Nam Nguyen (3):
      IB/ehca: Change scaling_code parameter description to match default value
      IB/ehca: Report RDMA atomic attributes in query_qp()
      IB/ehca: Improve latency by unlocking after triggering the hardware

Jack Morgenstein (2):
      IB/mlx4: Implement query QP
      IB/mlx4: Implement query SRQ

Jan Engelhardt (1):
      IB: Use menuconfig for InfiniBand menu

Joachim Fenkes (9):
      IB/ehca: Refactor "maybe missed event" code
      IB/ehca: HW level, HW caps and MTU autodetection
      IB/ehca: QP code restructuring in preparation for SRQ
      IB/ehca: add Shared Receive Queue support
      IB/ehca: Lock renaming, static initializers
      IB/ehca: Refactor sync between completions and destroy_cq using atomic_t
      IB/ehca: Change idr spinlocks into rwlocks
      IB/ehca: Return QP pointer in poll_cq()
      IB/ehca: Notify consumers of LID/PKEY/SM changes after nondisruptive events

Joan Eslinger (1):
      IB/ipath: Change use of constants for TID type to defined values

John Gregor (2):
      IB/ipath: Remove incompletely implemented ipath_runtime flags and code
      IB/ipath: Update copyright dates

Mark Debbage (2):
      IB/ipath: Correct checking of swminor version field when using subports
      IB/ipath: Make handling of one subport consistent

Michael Albaugh (4):
      IB/ipath: Support blinking LEDs with an led_override file
      IB/ipath: Lock and always use shadow copies of GPIO register
      IB/ipath: Log "active" time and some errors to EEPROM
      IB/ipath: Add capability to modify PBC word

Michael S. Tsirkin (2):
      IB/mlx4: Include linux/mutex.h from mlx4_ib.h
      mlx4_core: Include linux/mutex.h from mlx4.h

Ralph Campbell (10):
      IB/ipath: Fix problem with next WQE after a UC completion
      IB/ipath: Fix local loopback bug when waiting for resources
      IB/ipath: Set M bit in BTH according to IB spec
      IB/ipath: Fix RDMA read retry code
      IB/ipath: Wait for PIO available interrupt
      IB/ipath: Fix possible data corruption if multiple SGEs used for receive
      IB/ipath: Duplicate RDMA reads can cause responder to NAK inappropriately
      IB/ipath: Add barrier before updating WC head in shared memory
      IB/ipath: Lower default number of kernel send buffers
      IB/ipath: Remove support for preproduction HTX InfiniPath cards

Robert Walsh (5):
      IB/ipath: Fix maximum MTU reporting
      IB/ipath: Fill in some missing FMR-related fields in query_device
      IB/ipath: Send ACK invalid where appropriate
      IB/ipath: ipath_poll fixups and enhancements
      IB/ipath: Clean send flags properly on QP reset

Roland Dreier (5):
      IB: Remove garbage non-ASCII characters from comments
      IB: Update mailing list address
      IPoIB/cm: Fix warning if IPV6 is not enabled
      IPoIB: Recycle loopback skbs instead of freeing and reallocating
      IB: Update MAINTAINERS with Hal's new email address

Sean Hefty (7):
      IB/ipath: return correct PortGUID in NodeInfo
      IB/sa: Make sure SA queries use default P_Key
      IB/cm: Use spin_lock_irq() instead of spin_lock_irqsave() when possible
      IB/cm: Include HCA ACK delay in local ACK timeout
      IB/cm: cm_msgs.h should include ib_cm.h
      IB/cm: Fix handling of duplicate SIDR REQs
      IB/cm: Send no match if a SIDR REQ does not match a listen

Shani Moideen (1):
      IB/mthca: Replace memset(<addr>, 0, PAGE_SIZE) with clear_page(<addr>)

Stefan Roscher (2):
      IB/ehca: Support UD low-latency QPs
      IB/ehca: Set SEND_GRH flag for all non-LL UD QPs on eHCA2

Steve Wise (6):
      RDMA/cxgb3: Streaming -> RDMA mode transition fixes
      RDMA/cxgb3: TERMINATE WRs can hang the tx ofld queue
      RDMA/cxgb3: Don't count neg_adv abort_req_rss messages as real aborts
      RDMA/cxgb3: ctrl-qp init/clear shouldn't set the gen bit
      RDMA/cxgb3: Don't post TID_RELEASE message
      RDMA/cxgb3: Don't abort after failures sending the mpa reply

WANG Cong (1):
      RDMA/cxgb3: Check return of kmalloc() in iwch_register_device()

 MAINTAINERS                                       |   15 +-
 drivers/infiniband/Kconfig                        |   15 +-
 drivers/infiniband/core/agent.c                   |   19 +-
 drivers/infiniband/core/cm.c                      |  247 ++++---
 drivers/infiniband/core/cm_msgs.h                 |    1 +
 drivers/infiniband/core/cma.c                     |    1 -
 drivers/infiniband/core/mad.c                     |   50 ++-
 drivers/infiniband/core/multicast.c               |    2 +-
 drivers/infiniband/core/sa.h                      |    2 +-
 drivers/infiniband/core/sa_query.c                |   87 ++-
 drivers/infiniband/core/smi.c                     |   16 +-
 drivers/infiniband/core/smi.h                     |    2 +
 drivers/infiniband/core/sysfs.c                   |    2 +-
 drivers/infiniband/core/ucm.c                     |    1 -
 drivers/infiniband/core/umem.c                    |    1 +
 drivers/infiniband/hw/amso1100/Kconfig            |    2 +-
 drivers/infiniband/hw/cxgb3/Kconfig               |    2 +-
 drivers/infiniband/hw/cxgb3/cxio_hal.c            |    6 +-
 drivers/infiniband/hw/cxgb3/cxio_wr.h             |    3 +-
 drivers/infiniband/hw/cxgb3/iwch_cm.c             |  108 ++--
 drivers/infiniband/hw/cxgb3/iwch_cm.h             |    1 +
 drivers/infiniband/hw/cxgb3/iwch_provider.c       |    7 +-
 drivers/infiniband/hw/cxgb3/iwch_qp.c             |    7 +-
 drivers/infiniband/hw/ehca/Kconfig                |    2 +-
 drivers/infiniband/hw/ehca/ehca_av.c              |    6 +-
 drivers/infiniband/hw/ehca/ehca_classes.h         |   75 ++-
 drivers/infiniband/hw/ehca/ehca_classes_pSeries.h |    4 +-
 drivers/infiniband/hw/ehca/ehca_cq.c              |   50 +-
 drivers/infiniband/hw/ehca/ehca_hca.c             |   61 ++-
 drivers/infiniband/hw/ehca/ehca_irq.c             |  140 +++--
 drivers/infiniband/hw/ehca/ehca_irq.h             |    1 -
 drivers/infiniband/hw/ehca/ehca_iverbs.h          |   18 +
 drivers/infiniband/hw/ehca/ehca_main.c            |   98 +++-
 drivers/infiniband/hw/ehca/ehca_qp.c              |  751 +++++++++++++++------
 drivers/infiniband/hw/ehca/ehca_reqs.c            |   85 ++-
 drivers/infiniband/hw/ehca/ehca_tools.h           |    1 +
 drivers/infiniband/hw/ehca/ehca_uverbs.c          |   13 +-
 drivers/infiniband/hw/ehca/hcp_if.c               |   58 +-
 drivers/infiniband/hw/ehca/hcp_if.h               |    1 -
 drivers/infiniband/hw/ehca/hipz_hw.h              |   19 +
 drivers/infiniband/hw/ehca/ipz_pt_fn.h            |   28 +-
 drivers/infiniband/hw/ipath/Kconfig               |    2 +-
 drivers/infiniband/hw/ipath/ipath_common.h        |   33 +-
 drivers/infiniband/hw/ipath/ipath_cq.c            |    7 +-
 drivers/infiniband/hw/ipath/ipath_debug.h         |    2 +-
 drivers/infiniband/hw/ipath/ipath_diag.c          |   41 +-
 drivers/infiniband/hw/ipath/ipath_driver.c        |  187 +++++-
 drivers/infiniband/hw/ipath/ipath_eeprom.c        |  303 ++++++++-
 drivers/infiniband/hw/ipath/ipath_file_ops.c      |  205 ++++--
 drivers/infiniband/hw/ipath/ipath_fs.c            |    9 +-
 drivers/infiniband/hw/ipath/ipath_iba6110.c       |  101 ++--
 drivers/infiniband/hw/ipath/ipath_iba6120.c       |   92 ++-
 drivers/infiniband/hw/ipath/ipath_init_chip.c     |   26 +-
 drivers/infiniband/hw/ipath/ipath_intr.c          |  141 ++++-
 drivers/infiniband/hw/ipath/ipath_kernel.h        |   85 +++-
 drivers/infiniband/hw/ipath/ipath_keys.c          |    2 +-
 drivers/infiniband/hw/ipath/ipath_layer.c         |    2 +-
 drivers/infiniband/hw/ipath/ipath_layer.h         |    2 +-
 drivers/infiniband/hw/ipath/ipath_mad.c           |   11 +-
 drivers/infiniband/hw/ipath/ipath_mmap.c          |    2 +-
 drivers/infiniband/hw/ipath/ipath_mr.c            |    2 +-
 drivers/infiniband/hw/ipath/ipath_qp.c            |   19 +-
 drivers/infiniband/hw/ipath/ipath_rc.c            |  116 +++-
 drivers/infiniband/hw/ipath/ipath_registers.h     |    2 +-
 drivers/infiniband/hw/ipath/ipath_ruc.c           |   36 +-
 drivers/infiniband/hw/ipath/ipath_srq.c           |    4 +-
 drivers/infiniband/hw/ipath/ipath_stats.c         |   25 +-
 drivers/infiniband/hw/ipath/ipath_sysfs.c         |   43 ++-
 drivers/infiniband/hw/ipath/ipath_uc.c            |    9 +-
 drivers/infiniband/hw/ipath/ipath_ud.c            |    6 +-
 drivers/infiniband/hw/ipath/ipath_user_pages.c    |    2 +-
 drivers/infiniband/hw/ipath/ipath_verbs.c         |   29 +-
 drivers/infiniband/hw/ipath/ipath_verbs.h         |    3 +-
 drivers/infiniband/hw/ipath/ipath_verbs_mcast.c   |    2 +-
 drivers/infiniband/hw/ipath/ipath_wc_ppc64.c      |    2 +-
 drivers/infiniband/hw/ipath/ipath_wc_x86_64.c     |   29 +-
 drivers/infiniband/hw/mlx4/Kconfig                |    1 -
 drivers/infiniband/hw/mlx4/main.c                 |    6 +-
 drivers/infiniband/hw/mlx4/mlx4_ib.h              |    4 +
 drivers/infiniband/hw/mlx4/qp.c                   |  137 ++++
 drivers/infiniband/hw/mlx4/srq.c                  |   18 +
 drivers/infiniband/hw/mthca/Kconfig               |    2 +-
 drivers/infiniband/hw/mthca/mthca_allocator.c     |    2 +-
 drivers/infiniband/hw/mthca/mthca_eq.c            |    2 +-
 drivers/infiniband/ulp/ipoib/Kconfig              |    2 +-
 drivers/infiniband/ulp/ipoib/ipoib_cm.c           |    4 +-
 drivers/infiniband/ulp/ipoib/ipoib_ib.c           |   33 +-
 drivers/infiniband/ulp/iser/Kconfig               |    2 +-
 drivers/infiniband/ulp/srp/Kconfig                |    2 +-
 drivers/net/cxgb3/version.h                       |    2 +-
 drivers/net/mlx4/fw.c                             |    3 +
 drivers/net/mlx4/fw.h                             |    1 +
 drivers/net/mlx4/main.c                           |    1 +
 drivers/net/mlx4/mlx4.h                           |    1 +
 drivers/net/mlx4/qp.c                             |   21 +
 drivers/net/mlx4/srq.c                            |   30 +
 include/linux/mlx4/device.h                       |    2 +
 include/linux/mlx4/qp.h                           |    3 +
 include/rdma/ib_cm.h                              |    1 -
 include/rdma/ib_mad.h                             |    3 +
 100 files changed, 2812 insertions(+), 1061 deletions(-)

Further 2.6.23 merge plans...

[Roland Dreier]

As you can see, I just sent my first 2.6.23 pull request for Linus.
There are still a few more things I plan to do in before the merge
window closes (in ~10 days):

 - Write a patch to add P_Key handling to user_mad in the way we
   discussed (add an ioctl to enable P_Key mode without breaking old
   apps) -- I hope to do this tomorrow so we can get some review and
   testing before merging it.

 - Take a look at Sean's local SA caching patches.  I merged
   everything else from Sean's tree, but I'm still undecided about
   these.  I haven't read them carefully yet, but even aside from that
   I don't have a good feeling about whether there's consensus about
   this yet.  Any opinions about merging, for or against, would be
   appreciated here.

 - Merge up pending hardware driver changes, including the cxgb3 and
   ehca patches I have in my queue, plus Jack's catastrophic error
   patch for mlx4.

 - Try to get to resolution on the IPoIB "CM without SRQ" solution.

Also, if there's something I didn't list and didn't already include in
the tree I asked Linus to pull, please remind me.  I probably dropped it.

 - R.

Re: [ofa-general] Further 2.6.23 merge plans...

[Hal Rosenstock]

On Thu, 2007-07-12 at 19:15, Roland Dreier wrote:
> As you can see, I just sent my first 2.6.23 pull request for Linus.
> There are still a few more things I plan to do in before the merge
> window closes (in ~10 days):
> 
>  - Write a patch to add P_Key handling to user_mad in the way we
>    discussed (add an ioctl to enable P_Key mode without breaking old
>    apps) -- I hope to do this tomorrow so we can get some review and
>    testing before merging it.

Unfortunately, I'll mostly just be able to review it. Not sure how much
testing I will be able to do but we'll see...

-- Hal

>  - Take a look at Sean's local SA caching patches.  I merged
>    everything else from Sean's tree, but I'm still undecided about
>    these.  I haven't read them carefully yet, but even aside from that
>    I don't have a good feeling about whether there's consensus about
>    this yet.  Any opinions about merging, for or against, would be
>    appreciated here.
> 
>  - Merge up pending hardware driver changes, including the cxgb3 and
>    ehca patches I have in my queue, plus Jack's catastrophic error
>    patch for mlx4.
> 
>  - Try to get to resolution on the IPoIB "CM without SRQ" solution.
> 
> Also, if there's something I didn't list and didn't already include in
> the tree I asked Linus to pull, please remind me.  I probably dropped it.
> 
>  - R.
> _______________________________________________
> general mailing list
> general@lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [ofa-general] Further 2.6.23 merge plans...

[Sean Hefty]

>  - Take a look at Sean's local SA caching patches.  I merged
>    everything else from Sean's tree, but I'm still undecided about
>    these.  I haven't read them carefully yet, but even aside from that
>    I don't have a good feeling about whether there's consensus about
>    this yet.  Any opinions about merging, for or against, would be
>    appreciated here.

Obviously I'm biased here, but we've definitely seen local caching of 
path records (PR) greatly improve performance for large MPI job runs. 
(Our largest jobs wouldn't run without it.)  The development of the 
feature was requested and paid for by the US national labs. 
Infinicon/Silverstorm/QLogic also had this feature in their IB stack for 
scalability reasons as well.  PR caching is done in the stack today by 
IPoIB.

The implementation is hidden under the current kernel ib_sa interface, 
is disabled by default, and automatically fails over to standard PR 
queries if needed.  Removing the cache later should be fairly easy.

But to be fair, it will be difficult to enable both QoS and local PR 
caching.  To me, this would be the strongest reason against using it. 
However, QoS places additional burden on the SA, which will make scaling 
even more challenging.

- Sean

Re: Further 2.6.23 merge plans...

[Michael S. Tsirkin]

> Also, if there's something I didn't list and didn't already include in
> the tree I asked Linus to pull, please remind me.  I probably dropped it.

Any plans to do something with multiple EQ support in mthca?

-- 
MST

Re: Further 2.6.23 merge plans...

[Roland Dreier]

 > Any plans to do something with multiple EQ support in mthca?

I haven't done any work on it or seen anything from anyone else, so I
expect this will have to wait for 2.6.24.

Re: [ofa-general] Re: Further 2.6.23 merge plans...

[Shirley Ma]

Hello Roland,

>  > Any plans to do something with multiple EQ support in mthca?
> 
> I haven't done any work on it or seen anything from anyone else, so I
> expect this will have to wait for 2.6.24.

        We are working on IPoIB to use multiple EQ for multiple 
links/connetions scalability. Does this mean this will wait for 2.6.24?

Thanks
Shirley

Re: [ofa-general] Further 2.6.23 merge plans...

[Shirley Ma]

Hello Roland,

        FYI, we are working on several IPoIB performance improvement 
patches which are not on the list. Some of the patches are under test, 
some of the patches are going to be submitted soon. They are:

1.  skb aggregations for both dev xmit(networking layer) and IPoIB send 
(it will be submitted soon, for both UD and RC mode)
2.  multiple interrupt vectors in IPoIB for multiple links scalability 
(working on patch for both UD and RC mode)
3.  split CQ and send completion aggregation (for both UD and RC mode)
4.  LRO for IPoIB when generic LRO is available in networking layer. (UD 
mode only)

        Some of them might be made in 2.6.23 timeline, some of them might 
not, it depends on our test progress and community review feedback.

Thanks
Shirley

Re: Further 2.6.23 merge plans...

[Michael S. Tsirkin]

> Quoting Roland Dreier <rdreier@cisco.com>:
> Subject: Re: Further 2.6.23 merge plans...
> 
>  > Any plans to do something with multiple EQ support in mthca?
> 
> I haven't done any work on it or seen anything from anyone else, so I
> expect this will have to wait for 2.6.24.

I'm surprised to hear this. How about this:
http://lists.openfabrics.org/pipermail/general/2007-May/035757.html

-- 
MST

Re: [ofa-general] Further 2.6.23 merge plans...

[Tziporet Koren]

Roland Dreier wrote:
> As you can see, I just sent my first 2.6.23 pull request for Linus.
> There are still a few more things I plan to do in before the merge
> window closes (in ~10 days):
>
>   
Till when can we insert mlx4 with FMRs?

Tziporet

Re: [ofa-general] Re: Further 2.6.23 merge plans...

[Michael S. Tsirkin]

> Quoting Shirley Ma <xma@us.ibm.com>:
> Subject: Re: [ofa-general] Re: Further 2.6.23 merge plans...
> 
> Michael,
> 
> I would like to try this patch for one adapter/2 ports scalability performance
> for IPoIB. Is this patch appliable to OFED-1.2?

Most likely yes.

-- 
MST

Re: Further 2.6.23 merge plans...

[Roland Dreier]

 > > I haven't done any work on it or seen anything from anyone else, so I
 > > expect this will have to wait for 2.6.24.

 > I'm surprised to hear this. How about this:
 > http://lists.openfabrics.org/pipermail/general/2007-May/035757.html

Sure, I remember that.  But I haven't seen anything to suggest that
anyone has given any further thought to the issues that were raised in
that thread.

 - R.

Re: [ofa-general] Further 2.6.23 merge plans...

[Roland Dreier]

 > Till when can we insert mlx4 with FMRs?

2.6.22 came out on July 8, so I would expect 2.6.23-rc1 (the end of
the merge window) to be July 22.

Re: [ofa-general] Further 2.6.23 merge plans...

[Roland Dreier]

 >         FYI, we are working on several IPoIB performance improvement 
 > patches which are not on the list. Some of the patches are under test, 
 > some of the patches are going to be submitted soon. They are:

There is less than a week left in the merge window, and none of these
changes has been reviewed yet.  So being realistic, I don't think we
can expect to get any of this into 2.6.23.

 - R.

Re: Further 2.6.23 merge plans...

[Michael S. Tsirkin]

> Quoting Roland Dreier <rdreier@cisco.com>:
> Subject: Re: Further 2.6.23 merge plans...
> 
>  > > I haven't done any work on it or seen anything from anyone else, so I
>  > > expect this will have to wait for 2.6.24.
> 
>  > I'm surprised to hear this. How about this:
>  > http://lists.openfabrics.org/pipermail/general/2007-May/035757.html
> 
> Sure, I remember that.  But I haven't seen anything to suggest that
> anyone has given any further thought to the issues that were raised in
> that thread.

Well, the only issue I recall is about the # of EQs we want to allocate.
Was there something else?

Maybe code can be merged as-is (2 EQs) and the number be tuned
later as applications start using vectors?

-- 
MST

Re: Further 2.6.23 merge plans...

[Roland Dreier]

 > Well, the only issue I recall is about the # of EQs we want to allocate.
 > Was there something else?

Yes, some ideas about how applications should pick which EQ to use.
And how to handle CPU affinity.  And whether we want to try to do
something NUMA-aware.

 - R.

Re: [ofa-general] Re: Further 2.6.23 merge plans...

[Roland Dreier]

 >         We are working on IPoIB to use multiple EQ for multiple 
 > links/connetions scalability. Does this mean this will wait for 2.6.24?

I think so -- I don't want to merge something that first appears in
the last few days of the merge window.  The idea is to get your stuff
queued up *before* the merge window opens.

 - R.

Re: [ofa-general] Further 2.6.23 merge plans...

[Roland Dreier]

 >  - Take a look at Sean's local SA caching patches.  I merged
 >    everything else from Sean's tree, but I'm still undecided about
 >    these.  I haven't read them carefully yet, but even aside from that
 >    I don't have a good feeling about whether there's consensus about
 >    this yet.  Any opinions about merging, for or against, would be
 >    appreciated here.

Does anyone other than Sean have an opinion here?  If you want this
feature, if you've tested it, if you don't think it's ready yet,
whatever, please speak up -- I don't feel comfortable making a
decision on my own here (although I will if I have to).

Re: [ofa-general] Further 2.6.23 merge plans...

[Matt Leininger]

On Tue, 2007-07-17 at 11:07 -0700, Roland Dreier wrote:
> >  - Take a look at Sean's local SA caching patches.  I merged
>  >    everything else from Sean's tree, but I'm still undecided about
>  >    these.  I haven't read them carefully yet, but even aside from that
>  >    I don't have a good feeling about whether there's consensus about
>  >    this yet.  Any opinions about merging, for or against, would be
>  >    appreciated here.
> 
> Does anyone other than Sean have an opinion here?  If you want this
> feature, if you've tested it, if you don't think it's ready yet,
> whatever, please speak up -- I don't feel comfortable making a
> decision on my own here (although I will if I have to).
  
  Roland,
 
     I would like to see these features moved upstream.  DOE funded this
work as part of the items we see needing on our large scale IB
deployment (both present and future).  So from at least one big customer
perspective we see this as useful.  

    I'll let others comment on specific code/implementation issues.

  Thanks,

	- Matt

> _______________________________________________
> general mailing list
> general@lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

-- 
Matt Leininger, Ph.D.
Lawrence Livermore National Laboratory
leininger2@llnl.gov
V 925-422-4110

Re: [ofa-general] Further 2.6.23 merge plans...

[Roland Dreier]

 >      I would like to see these features moved upstream.  DOE funded this
 > work as part of the items we see needing on our large scale IB
 > deployment (both present and future).  So from at least one big customer
 > perspective we see this as useful.  

Does your reference to "present deployment" mean you are running this
code now?

 - R.

Re: Further 2.6.23 merge plans...

[Michael S. Tsirkin]

> Quoting Roland Dreier <rdreier@cisco.com>:
> Subject: Re: Further 2.6.23 merge plans...
> 
>  >  - Take a look at Sean's local SA caching patches.  I merged
>  >    everything else from Sean's tree, but I'm still undecided about
>  >    these.  I haven't read them carefully yet, but even aside from that
>  >    I don't have a good feeling about whether there's consensus about
>  >    this yet.  Any opinions about merging, for or against, would be
>  >    appreciated here.
> 
> Does anyone other than Sean have an opinion here?  If you want this
> feature, if you've tested it, if you don't think it's ready yet,
> whatever, please speak up -- I don't feel comfortable making a
> decision on my own here (although I will if I have to).

We have the patches applied in ofed 1.2.c with default module parameter set to
caching disabled (ofed 1.2 had a different version of the patches, but caching
is disabled by default there, too). At least in this configuration
(caching disabled), all issues I've seen seem to be fixed now, and tests seem to
be running smoothly.

So I think it's safe to merge it up if the module parameter
is set to cache disabled by default.
No idea what happens if it's enabled though :)

-- 
MST

Re: [ofa-general] Further 2.6.23 merge plans...

[Roland Dreier]

 > > But to be fair, it will be difficult to enable both QoS and local PR
 > > caching.  To me, this would be the strongest reason against using it.
 > > However, QoS places additional burden on the SA, which will make scaling
 > > even more challenging.
 > 
 > my understanding is that the local sa does a path-query where all the fields
 > except for the SGID are wildcard-ed. This means we expect the result to be a
 > table of all the paths from this port to every other port on the fabrics for
 > every pkey which this port is a member of etc, correct?
 > 
 > How do you plug here  the QoS concept of SID in the path query? are you
 > expecting the SA to realize what are all the services for which this port is
 > a "member"? does the proposed definision for QoS management at the SA
 > defines "services per gids" isn't it "what SL to user per Service"?

Or, thanks for rescuing this post.

I think this is an important question.  If we merge the local SA
stuff, then are we creating a problem for dealing with QoS?  Are we
going to have to revert the local SA stuff once the QoS stuff is
available?  Or is there at least a sketch of a plan on how to handle
this?

 - R.

RE: [ofa-general] Further 2.6.23 merge plans...

[Sean Hefty]

>I think this is an important question.  If we merge the local SA
>stuff, then are we creating a problem for dealing with QoS?

Yes - I do believe that merging PR caching and QoS together will be difficult.
I don't think the problems are insurmountable, but I can't say that I have a
definite solution for how to deal with this. 

My current thoughts are that the purpose of the cache is to increase SA
scalability on large clusters.  We've seen issues running MPI, trying to
establish all-to-all connections, on our 256 node cluster.  (With 4 processes
per node, this results in about 500,000+ PR queries hitting the SA.)  The SA was
swamped with work, and it wasn't trying to enforce QoS requirements across the
cluster.

I just don't see how an SA that is already having trouble scaling to this number
of nodes will be able to perform the additional task of providing QoS across the
cluster.  It may be that, at least initially, an administrator may need to
select between enabling PR caching or QoS.

>Are we going to have to revert the local SA stuff once the QoS stuff is
>available?

In the best case, the local SA will need enhancements added to the base support.
In the worst case, a user would have to choose between QoS or PR caching.  If
all users choose QoS, then it would make sense to remove the local SA. 

>Or is there at least a sketch of a plan on how to handle this?

This is only a rough idea, and it depends on how the QoS is implemented.  The
idea is to create a local QoS module on each node.  The local QoS modules would
be programmed with basic QoS information.  For example, which types of queries
to handle locally, versus which ones to forward to the SA.  Locally handled
queries would return PRs based on some QoS mapping table.  (I haven't looked
into any details of this.)

Ideally, local QoS modules would be programmed by a QoS master.  This would
require a new vendor-specific protocol, but would allow for a simple distributed
QoS manager.

We will have a better idea of the issues and possible solutions once the QoS
spec is released, and we can hold discussions on it.  I will be working more
details on QoS enhancements starting in the next couple of weeks. 

- Sean

Re: [ofa-general] Further 2.6.23 merge plans...

[Or Gerlitz]

Roland Dreier wrote:
>  >      I would like to see these features moved upstream.  DOE funded this
>  > work as part of the items we see needing on our large scale IB
>  > deployment (both present and future).  So from at least one big customer
>  > perspective we see this as useful.  
> 
> Does your reference to "present deployment" mean you are running this
> code now?

Indeed, my understanding is that the DOE uses an Open MPI device (I 
think its called PTE) which is implemented directly over libibverbs and 
hence no path queries are issued at all, if this is indeed the case, for 
them its more of a "for-the-future" thing.

Or.

Re: [ofa-general] Re: Further 2.6.23 merge plans...

[Tziporet Koren]

Michael S. Tsirkin wrote:
> We have the patches applied in ofed 1.2.c with default module parameter set to
> caching disabled (ofed 1.2 had a different version of the patches, but caching
> is disabled by default there, too). At least in this configuration
> (caching disabled), all issues I've seen seem to be fixed now, and tests seem to
> be running smoothly.
>   
As far as I know Intel run with SA cache enabled on large clusters with 
Intel MPI

Tziporet

Re: [ofa-general] Re: Further 2.6.23 merge plans...

[Michael S. Tsirkin]

> Quoting Tziporet Koren <tziporet@dev.mellanox.co.il>:
> Subject: Re: [ofa-general] Re: Further 2.6.23 merge plans...
> 
> Michael S. Tsirkin wrote:
> >We have the patches applied in ofed 1.2.c with default module parameter set
> >to caching disabled (ofed 1.2 had a different version of the patches, but
> >caching is disabled by default there, too). At least
> >in this configuration (caching disabled), all issues I've seen seem to be
> >fixed now, and tests seem to be running smoothly.
>
> As far as I know Intel run with SA cache enabled on large clusters with 
> Intel MPI

With OFED 1.2 version of the code, right?

-- 
MST

Re: [ofa-general] Re: Further 2.6.23 merge plans...

[Tziporet Koren]

Michael S. Tsirkin wrote:
>> As far as I know Intel run with SA cache enabled on large clusters with
>> Intel MPI
>>     
>
> With OFED 1.2 version of the code, right?
>
>   
Yes.
But maybe they also used the new module - Sean?

RE: [ofa-general] Re: Further 2.6.23 merge plans...

[Sean Hefty]

>> With OFED 1.2 version of the code, right?
>>
>>
>Yes.
>But maybe they also used the new module - Sean?

We actually use the OFED 1.2 version.  So, this feature is in use, but not this
specific implementation.

- Sean

Re: [ofa-general] Re: Further 2.6.23 merge plans...

[Roland Dreier]

 > We actually use the OFED 1.2 version.  So, this feature is in use, but not this
 > specific implementation.

Hmm... how much testing has the implementation being proposed for
merging actually had?

It might still be OK if the answer is that it hasn't been tested at
scale but that the basic code works and should behave the same as the
code that was tested because the underlying design is the same... is
at least that much true?

Re: [ofa-general] Further 2.6.23 merge plans...

[Sean Hefty]

> We will have a better idea of the issues and possible solutions once the QoS
> spec is released, and we can hold discussions on it.  I will be working more
> details on QoS enhancements starting in the next couple of weeks. 

Based on discussions so far, maybe the best path forward from here is to
delay until 2.6.24.  This will let us add this version to OFED 1.3 for
more widespread testing, plus give us the time that we need to come up
with a plan to integrate QoS with the local SA.  I don't think we'll
have a final implementation for QoS support by that time, but at least
we'll have a better idea of the problems.

These patches are based on the same design used with OFED 1.2, but a
fair number of lines of code still changed, plus it added InformInfo 
registration.  I don't believe anyone other than me has tested these 
patches with the local SA enabled.  It's typically running on my 
systems, but because it automatically fails over to standard SA queries, 
it would be easy for me to miss problems.

- Sean

Re: [ofa-general] Further 2.6.23 merge plans...

[Sean Hefty]

> Based on discussions so far, maybe the best path forward from here is to
> delay until 2.6.24.  This will let us add this version to OFED 1.3 for
> more widespread testing, plus give us the time that we need to come up
> with a plan to integrate QoS with the local SA.

I spoke with Matt on this, and he agreed with this plan.

- Sean