net-next pull request: RDS

net-next pull request: RDS

[Andy Grover]

Hi Dave,

Sorry for the "release late, release seldom". We've made a lot of 
changes since spring but it never really stabilized..

Summary:
   * Add atomic operations (cswp and fadd) and masked versions
   * New RDS_RDMA_SILENT flag will try to execute RDMA op without
     causing an interrupt on the target machine
   * Massive performance improvements on big boxes (Chris Mason)
   * Many rmmod refcounting fixes (Zach Brown)
   * Do much more from tasklets instead of the worker thread
   * Add rds.h to kernel exported headers list
   * Other cleanups

Thanks -- Regards -- Andy

------
The following changes since commit db40980fcdb560d7992b0511df16cdd3f7e381f3:
   Eric Dumazet (1):
         net: poll() optimizations

are available in the git repository at:

   git://oss.oracle.com/git/agrover/linux-2.6.git for-davem

Andy Grover (66):
       RDS: Fix BUG_ONs to not fire when in a tasklet
       RDS: Rewrite rds_send_drop_to() for clarity
       RDS: Fix locking in send on m_rs_lock
       RDS: move rds_shutdown_worker impl. to rds_conn_shutdown
       RDS: cleanup: remove "== NULL"s and "!= NULL"s in ptr comparisons
       RDS: break out rdma and data ops into nested structs in rds_message
       RDS: cleanup/fix rds_rdma_unuse
       RDS: Explicitly allocate rm in sendmsg()
       RDS: fold rdma.h into rds.h
       RDS: make m_rdma_op a member of rds_message
       RDS: make sure all sgs alloced are initialized
       RDS: Clear up some confusing code in send_remove_from_sock
       RDS: Implement atomic operations
       RDS: Base init_depth and responder_resources on hw values
       RDS: Do not mask address when pinning pages
       RDS: Whitespace
       RDS: inc_purge() transport function unused - remove it
       RDS/IB: eliminate duplicate code
       RDS/IB: Disallow connections less than RDS 3.1
       RDS/IB: Remove dead code
       RDS/IB: Remove ib_[header/data]_sge() functions
       RDS: rewrite rds_ib_xmit
       RDS: Remove unsignaled_bytes sysctl
       RDS/IB: Make all flow control code conditional on i_flowctl
       RDS: Move loop-only function to loop.c
       RDS: Implement silent atomics
       RDS: Inline rdma_prepare into cmsg_rdma_args
       RDS: purge atomic resources too in rds_message_purge()
       RDS: Remove struct rds_rdma_op
       RDS: Rename data op members prefix from m_ to op_
       RDS: Rewrite rds_send_xmit
       RDS: Do not set op_active in r_m_copy_from_user().
       RDS: Add a warning if trying to allocate 0 sgs
       RDS: queue failure notifications for dropped atomic ops
       RDS: Move some variables around for consistency
       RDS: Add flag for silent ops. Do atomic op before RDMA
       RDS: Make sure cmsgs aren't used in improper ways
       RDS: Perform unmapping ops in stages
       RDS/IB: Do not wait for send ring to be empty on conn shutdown
       RDS: Stop supporting old cong map sending method
       RDS: Refill recv ring directly from tasklet
       RDS: Change send lock from a mutex to a spinlock
       RDS: rds_send_xmit() locking/irq fixes
       RDS: Call rds_send_xmit() directly from sendmsg()
       RDS: Bypass workqueue when queueing cong updates
       RDS: Use NOWAIT in message_map_pages()
       RDS: Fix locking in send on m_rs_lock
       RDS: rds_message_unmapped() doesn't need to check if queue active
       RDS: Move atomic stats from general to ib-specific area
       RDS: Remove send_quota from send_xmit()
       RDS: Do wait_event_interruptible instead of wait_event
       RDS: Get pong working again
       RDS: Update comments in rds_send_xmit()
       RDS/IB: Remove unused variable in ib_remove_addr()
       RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node()
       RDS: Use page_remainder_alloc() for recv bufs
       RDS: Assume recv->r_frag is always NULL in refill_one()
       RDS/IB: Remove ib_recv_unmap_page()
       RDS: whitespace
       RDS: Remove unused XLIST_PTR_TAIL and xlist_protect()
       RDS/IB: Make ib_recv_refill return void
       RDS/IB: Use SLAB_HWCACHE_ALIGN flag for kmem_cache_create()
       RDS: Implement masked atomic operations
       RDS: Add rds.h to exported headers list
       RDS: rds.h: Replace u_int[size]_t with uint[size]_t
       RDS: Remove dead struct from rds.h

Chris Mason (17):
       RDS: Use a generation counter to avoid rds_send_xmit loop
       rds: switch to rwlock on bind_lock
       rds: per-rm flush_wait waitq
       rds: rcu-ize rds_ib_get_device()
       rds: Use RCU for the bind lookup searches
       rds: don't let RDS shutdown a connection while senders are present
       rds: Fix RDMA message reference counting
       RDS: use locking on the connection hash list
       rds: use RCU to protect the connection hash
       rds: Fix reference counting on the for xmit_atomic and xmit_rdma
       RDS/IB: Add caching of frags and incs
       RDS: use friendly gfp masks for prefill
       rds: recycle FMRs through lockless lists
       rds: more FMRs are faster
       RDS: use delayed work for the FMR flushes
       RDS: properly use sg_init_table
       RDS: flush fmrs before allocating new ones

Tina Yang (1):
       RDS: Fix corrupted rds_mrs

Zach Brown (23):
       RDS/IB: rds_ib_cm_handle_connect() forgot to unlock c_cm_lock
       RDS/IB: get the xmit max_sge from the RDS IB device on the connection
       RDS/IB: add refcount tracking to struct rds_ib_device
       RDS: introduce rds_conn_connect_if_down()
       RDS/IB: disconnect when IB devices are removed
       rds: remove unused rds_send_acked_before()
       rds: block ints when acquiring c_lock in rds_conn_message_info()
       rds: fix rds_send_xmit() serialization
       RDS/IB: wait for IB dev freeing work to finish during rmmod
       RDS/IB: destroy connections on rmmod
       RDS/IB: create a work queue for FMR flushing
       RDS: return to a single-threaded krdsd
       RDS/IB: always process recv completions
       RDS: remove __init and __exit annotation
       RDS/IB: track signaled sends
       RDS/IB: print IB event strings as well as their number
       RDS/IB: protect the list of IB devices
       RDS: lock rds_conn_count decrement in rds_conn_destroy()
       RDS: remove old rs_transport comment
       RDS: have sockets get transport module references
       RDS: don't call rds_conn_shutdown() from rds_conn_destroy()
       RDS: cancel connection work structs as we shut down
       RDS/IB: print string constants in more places

  include/linux/Kbuild     |    1 +
  include/linux/rds.h      |  106 +++++---
  net/rds/af_rds.c         |   26 ++-
  net/rds/bind.c           |   82 +++---
  net/rds/cong.c           |    8 +-
  net/rds/connection.c     |  157 ++++++++---
  net/rds/ib.c             |  194 +++++++++++---
  net/rds/ib.h             |  100 ++++----
  net/rds/ib_cm.c          |  183 +++++++++----
  net/rds/ib_rdma.c        |  314 +++++++++++++++++-----
  net/rds/ib_recv.c        |  549 ++++++++++++++++++++++---------------
  net/rds/ib_send.c        |  682 
+++++++++++++++++++++++++++-------------------
  net/rds/ib_stats.c       |    2 +
  net/rds/ib_sysctl.c      |   17 +-
  net/rds/info.c           |   12 +-
  net/rds/iw.c             |    4 +-
  net/rds/iw.h             |   11 +-
  net/rds/iw_cm.c          |   14 +-
  net/rds/iw_rdma.c        |    1 -
  net/rds/iw_recv.c        |   24 +-
  net/rds/iw_send.c        |   93 ++++---
  net/rds/iw_sysctl.c      |    4 +-
  net/rds/loop.c           |   31 ++-
  net/rds/message.c        |  118 +++++----
  net/rds/page.c           |    5 +-
  net/rds/rdma.c           |  339 +++++++++++++++--------
  net/rds/rdma.h           |   85 ------
  net/rds/rdma_transport.c |   42 +++-
  net/rds/rds.h            |  187 +++++++++++---
  net/rds/recv.c           |    9 +-
  net/rds/send.c           |  544 ++++++++++++++++++++++---------------
  net/rds/stats.c          |    6 +-
  net/rds/sysctl.c         |    4 +-
  net/rds/tcp.c            |    8 +-
  net/rds/tcp.h            |    9 +-
  net/rds/tcp_connect.c    |    2 +-
  net/rds/tcp_listen.c     |    6 +-
  net/rds/tcp_recv.c       |   14 +-
  net/rds/tcp_send.c       |   66 +----
  net/rds/threads.c        |   69 +----
  net/rds/transport.c      |   19 +-
  net/rds/xlist.h          |   80 ++++++
  42 files changed, 2614 insertions(+), 1613 deletions(-)
  delete mode 100644 net/rds/rdma.h
  create mode 100644 net/rds/xlist.h
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: net-next pull request: RDS

[David Miller]

From: Andy Grover <andy.grover-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Date: Thu, 09 Sep 2010 11:51:08 -0700

> Hi Dave,
> 
> Sorry for the "release late, release seldom". We've made a lot of
> changes since spring but it never really stabilized..
> 
> Summary:
>   * Add atomic operations (cswp and fadd) and masked versions
>   * New RDS_RDMA_SILENT flag will try to execute RDMA op without
>     causing an interrupt on the target machine
>   * Massive performance improvements on big boxes (Chris Mason)
>   * Many rmmod refcounting fixes (Zach Brown)
>   * Do much more from tasklets instead of the worker thread
>   * Add rds.h to kernel exported headers list
>   * Other cleanups

I'll pull, but yeah you guys have to sync with me a lot more
often.

It would be nice if you guys CC:'d netdev for every patch that ends
up in your tree as well.

Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: net-next pull request: RDS

[Or Gerlitz]

Hi Andy, 

Some clarifications/questions from whatever quick look one can have over 107 patches...

Zach Brown's "RDS/IB: print IB event strings as well as their number" - commit
1bde04a63d532c2540d6fdee0a661530a62b1686 in net-next-2.6 looks perfect to reside as a helper function in the core IB stack which can be in use by other rdma drivers (e.g ipoib, iser, srp, etc). 

Chris Mason's "rds: recycle FMRs through lockless lists added net/rds/xlist.h" - 6fa70da6081bbcf948801fd5ee0be4d222298a43 adds net/rds/xlist.h - isn't this something that better be placed under include/linux/. etc?

And last, your "RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node()" patch looks interesting, 1st, it has some macros which could be placed in more general locations e.g pcidev_to_node and ibdev_to_node, your "significantly helps performance" comment is interesting, I'll send a separate note about that to the rdma mailing list.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node()

[Or Gerlitz]

Hi Andy, looking on this net-next-2.6 patch, I wonder if you can elaborate on your "significantly helps performance" comment - what improvement you see with this patch? 

What about the QP/CQ memory, are they better be placed in node-local to the HCA manner?

Or.

commit e4c52c98e04937ea87b0979a81354d0040d284f9
Author: Andy Grover <andy.grover-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Date:   Fri Apr 23 10:49:53 2010 -0700

    RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node()
    
    Allocate send/recv rings in memory that is node-local to the HCA.
    This significantly helps performance.
    
    Signed-off-by: Andy Grover <andy.grover-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

diff --git a/net/rds/ib.c b/net/rds/ib.c
index 7a2131d..7d289d7 100644
--- a/net/rds/ib.c
+++ b/net/rds/ib.c
@@ -77,7 +77,7 @@ void rds_ib_add_one(struct ib_device *device)
 		goto free_attr;
 	}
 
-	rds_ibdev = kmalloc(sizeof *rds_ibdev, GFP_KERNEL);
+	rds_ibdev = kmalloc_node(sizeof *rds_ibdev, GFP_KERNEL, ibdev_to_node(device));
 	if (!rds_ibdev)
 		goto free_attr;
 
diff --git a/net/rds/ib.h b/net/rds/ib.h
index c506604..4bc3e2f 100644
--- a/net/rds/ib.h
+++ b/net/rds/ib.h
@@ -3,6 +3,8 @@
 
 #include <rdma/ib_verbs.h>
 #include <rdma/rdma_cm.h>
+#include <linux/pci.h>
+#include <linux/slab.h>
 #include "rds.h"
 #include "rdma_transport.h"
 
@@ -167,6 +169,10 @@ struct rds_ib_device {
 	spinlock_t		spinlock;	/* protect the above */
 };
 
+#define pcidev_to_node(pcidev) pcibus_to_node(pcidev->bus)
+#define ibdev_to_node(ibdev) pcidev_to_node(to_pci_dev(ibdev->dma_device))
+#define rdsibdev_to_node(rdsibdev) ibdev_to_node(rdsibdev->dev)
+
 /* bits for i_ack_flags */
 #define IB_ACK_IN_FLIGHT	0
 #define IB_ACK_REQUESTED	1
diff --git a/net/rds/ib_cm.c b/net/rds/ib_cm.c
index 75eda9c..b5d0b60 100644
--- a/net/rds/ib_cm.c
+++ b/net/rds/ib_cm.c
@@ -347,7 +347,8 @@ static int rds_ib_setup_qp(struct rds_connection *conn)
 		goto out;
 	}
 
-	ic->i_sends = vmalloc(ic->i_send_ring.w_nr * sizeof(struct rds_ib_send_work));
+	ic->i_sends = vmalloc_node(ic->i_send_ring.w_nr * sizeof(struct rds_ib_send_work),
+				   ibdev_to_node(dev));
 	if (!ic->i_sends) {
 		ret = -ENOMEM;
 		rdsdebug("send allocation failed\n");
@@ -355,7 +356,8 @@ static int rds_ib_setup_qp(struct rds_connection *conn)
 	}
 	memset(ic->i_sends, 0, ic->i_send_ring.w_nr * sizeof(struct rds_ib_send_work));
 
-	ic->i_recvs = vmalloc(ic->i_recv_ring.w_nr * sizeof(struct rds_ib_recv_work));
+	ic->i_recvs = vmalloc_node(ic->i_recv_ring.w_nr * sizeof(struct rds_ib_recv_work),
+				   ibdev_to_node(dev));
 	if (!ic->i_recvs) {
 		ret = -ENOMEM;
 		rdsdebug("recv allocation failed\n");
diff --git a/net/rds/ib_rdma.c b/net/rds/ib_rdma.c
index 7315fff..cc341cd 100644
--- a/net/rds/ib_rdma.c
+++ b/net/rds/ib_rdma.c
@@ -297,7 +297,7 @@ static struct rds_ib_mr *rds_ib_alloc_fmr(struct rds_ib_device *rds_ibdev)
 		rds_ib_flush_mr_pool(pool, 0);
 	}
 
-	ibmr = kzalloc(sizeof(*ibmr), GFP_KERNEL);
+	ibmr = kzalloc_node(sizeof(*ibmr), GFP_KERNEL, rdsibdev_to_node(rds_ibdev));
 	if (!ibmr) {
 		err = -ENOMEM;
 		goto out_no_cigar;
@@ -376,7 +376,8 @@ static int rds_ib_map_fmr(struct rds_ib_device *rds_ibdev, struct rds_ib_mr *ibm
 	if (page_cnt > fmr_message_size)
 		return -EINVAL;
 
-	dma_pages = kmalloc(sizeof(u64) * page_cnt, GFP_ATOMIC);
+	dma_pages = kmalloc_node(sizeof(u64) * page_cnt, GFP_ATOMIC,
+				 rdsibdev_to_node(rds_ibdev));
 	if (!dma_pages)
 		return -ENOMEM;
 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [rds-devel] net-next pull request: RDS

[Chris Mason]

On Sun, Sep 12, 2010 at 10:49:38AM +0200, Or Gerlitz wrote:
> Hi Andy, 
> 
> Some clarifications/questions from whatever quick look one can have over 107 patches...
> 
> Zach Brown's "RDS/IB: print IB event strings as well as their number" - commit
> 1bde04a63d532c2540d6fdee0a661530a62b1686 in net-next-2.6 looks perfect to reside as a helper function in the core IB stack which can be in use by other rdma drivers (e.g ipoib, iser, srp, etc). 
> 
> Chris Mason's "rds: recycle FMRs through lockless lists added net/rds/xlist.h" - 6fa70da6081bbcf948801fd5ee0be4d222298a43 adds net/rds/xlist.h - isn't this something that better be placed under include/linux/. etc?

Yes, I plan on expanding xlist.h a little and trying to use it in a few
other places.  But since RDS is currently the only consumer it made sense
to me to keep it in rds until I had more code in line to take advantage
of it.

-chris

Re: [rds-devel] net-next pull request: RDS

[Zach Brown]

> Zach Brown's "RDS/IB: print IB event strings as well as their number" - commit
> 1bde04a63d532c2540d6fdee0a661530a62b1686 in net-next-2.6 looks perfect to
> reside as a helper function in the core IB stack which can be in use by other
> rdma drivers (e.g ipoib, iser, srp, etc). 

Yeah, if those other drivers were interested in the little helpers then it
would make sense to bring the functionality up into the core stack.

Note that there's another later commit in net-next that gives other event
strings the same treatment: 59f740a6aeb2cde2f79fe0df38262d4c1ef35cd8 .

- z
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [rds-devel] net-next pull request: RDS

[Andrew Grover]

On Tue, Sep 14, 2010 at 11:35 AM, Zach Brown <zach.brown-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
>> Zach Brown's "RDS/IB: print IB event strings as well as their number" - commit
>> 1bde04a63d532c2540d6fdee0a661530a62b1686 in net-next-2.6 looks perfect to
>> reside as a helper function in the core IB stack which can be in use by other
>> rdma drivers (e.g ipoib, iser, srp, etc).
>
> Yeah, if those other drivers were interested in the little helpers then it
> would make sense to bring the functionality up into the core stack.
>
> Note that there's another later commit in net-next that gives other event
> strings the same treatment: 59f740a6aeb2cde2f79fe0df38262d4c1ef35cd8 .

Since rds goes upstream via netdev not linux-rdma, it's a little
tricky.. any core ib helpers we added would not be in net-next so rds
wouldn't build. :) Once net-next gets pushed to mainline and Roland
pulls from that, then we'll be in a good position to put these helpers
where they should go, and change other ULPs to use them.

-- Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [rds-devel] net-next pull request: RDS

[Roland Dreier]

 > Since rds goes upstream via netdev not linux-rdma, it's a little
 > tricky.. any core ib helpers we added would not be in net-next so rds
 > wouldn't build. :) Once net-next gets pushed to mainline and Roland
 > pulls from that, then we'll be in a good position to put these helpers
 > where they should go, and change other ULPs to use them.

In cases where cross-tree dependencies exist, I'm happy to review core
IB stack changes but then have them go upstream via davem after
linux-rdma is happy with them.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node()

[Andy Grover]

On 09/12/2010 01:52 AM, Or Gerlitz wrote:
> Hi Andy, looking on this net-next-2.6 patch, I wonder if you can
> elaborate on your "significantly helps performance" comment - what
> improvement you see with this patch?

 > What about the QP/CQ memory, are they better be placed in node-local
 > to the HCA manner?

Hi Or, I'm afraid I don't have exact numbers for you.. those changes 
were part of a series of tweaks and fixes for bottlenecks in rds code.

On our numa box, the HCA is attached to a particular package of CPUs & 
memory. Having the in-memory structures that the hca is accessing in 
node-local RAM seemed to measurably help, although reducing contention 
helped even more. Chris Mason says it seemed like init-time allocations 
seemed to benefit while runtime allocations were not helped, or even hurt.

Clearly this is not enough data to go on, beyond the conclusion that yes 
it's something that should be looked at more closely. Big machines are 
still rather scarce, which makes it harder.

Regards -- Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [rds-devel] net-next pull request: RDS

[Or Gerlitz]

Andrew Grover wrote:
> Once net-next gets pushed to mainline and Roland pulls from that, 
> then we'll be in a good position to put these helpers where they should go, 
> and change other ULPs to use them.

Andy, as Roland commented, you can push such helpers through Dave once Roland made a review of them, in a similar manner to a situation where an iser patch is merged by the iscsi maintainer, etc. if going to the review now... Roland - what's your take on the below patch net-next-2.6 and also on its net-next-2.6  "RDS/IB: print IB event strings as well as their number" 1bde04a63d532c2540d6fdee0a661530a62b1686 buddy?

Or.


commit 59f740a6aeb2cde2f79fe0df38262d4c1ef35cd8
Author: Zach Brown <zach.brown-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Date:   Tue Aug 3 13:52:47 2010 -0700

    RDS/IB: print string constants in more places
    
    This prints the constant identifier for work completion status and rdma
    cm event types, like we already do for IB event types.
    
    A core string array helper is added that each string type uses.
    
    Signed-off-by: Zach Brown <zach.brown-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

diff --git a/net/rds/af_rds.c b/net/rds/af_rds.c
index 8e3886d..bb6ad81 100644
--- a/net/rds/af_rds.c
+++ b/net/rds/af_rds.c
@@ -40,6 +40,15 @@
 
 #include "rds.h"
 
+char *rds_str_array(char **array, size_t elements, size_t index)
+{
+	if ((index < elements) && array[index])
+		return array[index];
+	else
+		return "unknown";
+}
+EXPORT_SYMBOL(rds_str_array);
+
 /* this is just used for stats gathering :/ */
 static DEFINE_SPINLOCK(rds_sock_lock);
 static unsigned long rds_sock_count;
diff --git a/net/rds/ib.h b/net/rds/ib.h
index 2189fd4..7ad3d57 100644
--- a/net/rds/ib.h
+++ b/net/rds/ib.h
@@ -345,6 +345,7 @@ u32 rds_ib_ring_completed(struct rds_ib_work_ring *ring, u32 wr_id, u32 oldest);
 extern wait_queue_head_t rds_ib_ring_empty_wait;
 
 /* ib_send.c */
+char *rds_ib_wc_status_str(enum ib_wc_status status);
 void rds_ib_xmit_complete(struct rds_connection *conn);
 int rds_ib_xmit(struct rds_connection *conn, struct rds_message *rm,
 		unsigned int hdr_off, unsigned int sg, unsigned int off);
diff --git a/net/rds/ib_cm.c b/net/rds/ib_cm.c
index 0e2fea8..bc3dbc1 100644
--- a/net/rds/ib_cm.c
+++ b/net/rds/ib_cm.c
@@ -39,7 +39,8 @@
 #include "ib.h"
 
 static char *rds_ib_event_type_strings[] = {
-#define RDS_IB_EVENT_STRING(foo) [IB_EVENT_##foo] = __stringify(foo)
+#define RDS_IB_EVENT_STRING(foo) \
+		[IB_EVENT_##foo] = __stringify(IB_EVENT_##foo)
 	RDS_IB_EVENT_STRING(CQ_ERR),
 	RDS_IB_EVENT_STRING(QP_FATAL),
 	RDS_IB_EVENT_STRING(QP_REQ_ERR),
@@ -63,11 +64,8 @@ static char *rds_ib_event_type_strings[] = {
 
 static char *rds_ib_event_str(enum ib_event_type type)
 {
-	if (type < ARRAY_SIZE(rds_ib_event_type_strings) &&
-	    rds_ib_event_type_strings[type])
-		return rds_ib_event_type_strings[type];
-	else
-		return "unknown";
+	return rds_str_array(rds_ib_event_type_strings,
+			     ARRAY_SIZE(rds_ib_event_type_strings), type);
 };
 
 /*
diff --git a/net/rds/ib_recv.c b/net/rds/ib_recv.c
index a2f5f6f..e29e0ca 100644
--- a/net/rds/ib_recv.c
+++ b/net/rds/ib_recv.c
@@ -966,8 +966,9 @@ static inline void rds_poll_cq(struct rds_ib_connection *ic,
 	struct rds_ib_recv_work *recv;
 
 	while (ib_poll_cq(ic->i_recv_cq, 1, &wc) > 0) {
-		rdsdebug("wc wr_id 0x%llx status %u byte_len %u imm_data %u\n",
-			 (unsigned long long)wc.wr_id, wc.status, wc.byte_len,
+		rdsdebug("wc wr_id 0x%llx status %u (%s) byte_len %u imm_data %u\n",
+			 (unsigned long long)wc.wr_id, wc.status,
+			 rds_ib_wc_status_str(wc.status), wc.byte_len,
 			 be32_to_cpu(wc.ex.imm_data));
 		rds_ib_stats_inc(s_ib_rx_cq_event);
 
@@ -985,10 +986,11 @@ static inline void rds_poll_cq(struct rds_ib_connection *ic,
 		} else {
 			/* We expect errors as the qp is drained during shutdown */
 			if (rds_conn_up(conn) || rds_conn_connecting(conn))
-				rds_ib_conn_error(conn, "recv completion on "
-						  "%pI4 had status %u, disconnecting and "
+				rds_ib_conn_error(conn, "recv completion on %pI4 had "
+						  "status %u (%s), disconnecting and "
 						  "reconnecting\n", &conn->c_faddr,
-						  wc.status);
+						  wc.status,
+						  rds_ib_wc_status_str(wc.status));
 		}
 
 		/*
diff --git a/net/rds/ib_send.c b/net/rds/ib_send.c
index 15f7569..808544a 100644
--- a/net/rds/ib_send.c
+++ b/net/rds/ib_send.c
@@ -38,6 +38,40 @@
 #include "rds.h"
 #include "ib.h"
 
+static char *rds_ib_wc_status_strings[] = {
+#define RDS_IB_WC_STATUS_STR(foo) \
+		[IB_WC_##foo] = __stringify(IB_WC_##foo)
+	RDS_IB_WC_STATUS_STR(SUCCESS),
+	RDS_IB_WC_STATUS_STR(LOC_LEN_ERR),
+	RDS_IB_WC_STATUS_STR(LOC_QP_OP_ERR),
+	RDS_IB_WC_STATUS_STR(LOC_EEC_OP_ERR),
+	RDS_IB_WC_STATUS_STR(LOC_PROT_ERR),
+	RDS_IB_WC_STATUS_STR(WR_FLUSH_ERR),
+	RDS_IB_WC_STATUS_STR(MW_BIND_ERR),
+	RDS_IB_WC_STATUS_STR(BAD_RESP_ERR),
+	RDS_IB_WC_STATUS_STR(LOC_ACCESS_ERR),
+	RDS_IB_WC_STATUS_STR(REM_INV_REQ_ERR),
+	RDS_IB_WC_STATUS_STR(REM_ACCESS_ERR),
+	RDS_IB_WC_STATUS_STR(REM_OP_ERR),
+	RDS_IB_WC_STATUS_STR(RETRY_EXC_ERR),
+	RDS_IB_WC_STATUS_STR(RNR_RETRY_EXC_ERR),
+	RDS_IB_WC_STATUS_STR(LOC_RDD_VIOL_ERR),
+	RDS_IB_WC_STATUS_STR(REM_INV_RD_REQ_ERR),
+	RDS_IB_WC_STATUS_STR(REM_ABORT_ERR),
+	RDS_IB_WC_STATUS_STR(INV_EECN_ERR),
+	RDS_IB_WC_STATUS_STR(INV_EEC_STATE_ERR),
+	RDS_IB_WC_STATUS_STR(FATAL_ERR),
+	RDS_IB_WC_STATUS_STR(RESP_TIMEOUT_ERR),
+	RDS_IB_WC_STATUS_STR(GENERAL_ERR),
+#undef RDS_IB_WC_STATUS_STR
+};
+
+char *rds_ib_wc_status_str(enum ib_wc_status status)
+{
+	return rds_str_array(rds_ib_wc_status_strings,
+			     ARRAY_SIZE(rds_ib_wc_status_strings), status);
+}
+
 /*
  * Convert IB-specific error message to RDS error message and call core
  * completion handler.
@@ -257,8 +291,9 @@ void rds_ib_send_cq_comp_handler(struct ib_cq *cq, void *context)
 		rdsdebug("ib_req_notify_cq send failed: %d\n", ret);
 
 	while (ib_poll_cq(cq, 1, &wc) > 0) {
-		rdsdebug("wc wr_id 0x%llx status %u byte_len %u imm_data %u\n",
-			 (unsigned long long)wc.wr_id, wc.status, wc.byte_len,
+		rdsdebug("wc wr_id 0x%llx status %u (%s) byte_len %u imm_data %u\n",
+			 (unsigned long long)wc.wr_id, wc.status,
+			 rds_ib_wc_status_str(wc.status), wc.byte_len,
 			 be32_to_cpu(wc.ex.imm_data));
 		rds_ib_stats_inc(s_ib_tx_cq_event);
 
@@ -306,10 +341,10 @@ void rds_ib_send_cq_comp_handler(struct ib_cq *cq, void *context)
 
 		/* We expect errors as the qp is drained during shutdown */
 		if (wc.status != IB_WC_SUCCESS && rds_conn_up(conn)) {
-			rds_ib_conn_error(conn,
-				"send completion on %pI4 "
-				"had status %u, disconnecting and reconnecting\n",
-				&conn->c_faddr, wc.status);
+			rds_ib_conn_error(conn, "send completion on %pI4 had status "
+					  "%u (%s), disconnecting and reconnecting\n",
+					  &conn->c_faddr, wc.status,
+					  rds_ib_wc_status_str(wc.status));
 		}
 	}
 }
diff --git a/net/rds/rdma_transport.c b/net/rds/rdma_transport.c
index 550d348..e6ed10a 100644
--- a/net/rds/rdma_transport.c
+++ b/net/rds/rdma_transport.c
@@ -36,6 +36,34 @@
 
 static struct rdma_cm_id *rds_rdma_listen_id;
 
+static char *rds_cm_event_strings[] = {
+#define RDS_CM_EVENT_STRING(foo) \
+		[RDMA_CM_EVENT_##foo] = __stringify(RDMA_CM_EVENT_##foo)
+	RDS_CM_EVENT_STRING(ADDR_RESOLVED),
+	RDS_CM_EVENT_STRING(ADDR_ERROR),
+	RDS_CM_EVENT_STRING(ROUTE_RESOLVED),
+	RDS_CM_EVENT_STRING(ROUTE_ERROR),
+	RDS_CM_EVENT_STRING(CONNECT_REQUEST),
+	RDS_CM_EVENT_STRING(CONNECT_RESPONSE),
+	RDS_CM_EVENT_STRING(CONNECT_ERROR),
+	RDS_CM_EVENT_STRING(UNREACHABLE),
+	RDS_CM_EVENT_STRING(REJECTED),
+	RDS_CM_EVENT_STRING(ESTABLISHED),
+	RDS_CM_EVENT_STRING(DISCONNECTED),
+	RDS_CM_EVENT_STRING(DEVICE_REMOVAL),
+	RDS_CM_EVENT_STRING(MULTICAST_JOIN),
+	RDS_CM_EVENT_STRING(MULTICAST_ERROR),
+	RDS_CM_EVENT_STRING(ADDR_CHANGE),
+	RDS_CM_EVENT_STRING(TIMEWAIT_EXIT),
+#undef RDS_CM_EVENT_STRING
+};
+
+static char *rds_cm_event_str(enum rdma_cm_event_type type)
+{
+	return rds_str_array(rds_cm_event_strings,
+			     ARRAY_SIZE(rds_cm_event_strings), type);
+};
+
 int rds_rdma_cm_event_handler(struct rdma_cm_id *cm_id,
 			      struct rdma_cm_event *event)
 {
@@ -44,8 +72,8 @@ int rds_rdma_cm_event_handler(struct rdma_cm_id *cm_id,
 	struct rds_transport *trans;
 	int ret = 0;
 
-	rdsdebug("conn %p id %p handling event %u\n", conn, cm_id,
-		 event->event);
+	rdsdebug("conn %p id %p handling event %u (%s)\n", conn, cm_id,
+		 event->event, rds_cm_event_str(event->event));
 
 	if (cm_id->device->node_type == RDMA_NODE_RNIC)
 		trans = &rds_iw_transport;
@@ -109,7 +137,8 @@ int rds_rdma_cm_event_handler(struct rdma_cm_id *cm_id,
 
 	default:
 		/* things like device disconnect? */
-		printk(KERN_ERR "RDS: unknown event %u!\n", event->event);
+		printk(KERN_ERR "RDS: unknown event %u (%s)!\n",
+		       event->event, rds_cm_event_str(event->event));
 		break;
 	}
 
@@ -117,7 +146,8 @@ out:
 	if (conn)
 		mutex_unlock(&conn->c_cm_lock);
 
-	rdsdebug("id %p event %u handling ret %d\n", cm_id, event->event, ret);
+	rdsdebug("id %p event %u (%s) handling ret %d\n", cm_id, event->event,
+		 rds_cm_event_str(event->event), ret);
 
 	return ret;
 }
diff --git a/net/rds/rds.h b/net/rds/rds.h
index aab5e94..aadaddb 100644
--- a/net/rds/rds.h
+++ b/net/rds/rds.h
@@ -566,6 +566,7 @@ struct rds_statistics {
 };
 
 /* af_rds.c */
+char *rds_str_array(char **array, size_t elements, size_t index);
 void rds_sock_addref(struct rds_sock *rs);
 void rds_sock_put(struct rds_sock *rs);
 void rds_wake_sk_sleep(struct rds_sock *rs);
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html