* net-next pull request: RDS @ 2010-09-09 18:51 Andy Grover [not found] ` <4C892C9C.4060304-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 0 siblings, 1 reply; 10+ messages in thread From: Andy Grover @ 2010-09-09 18:51 UTC (permalink / raw) To: netdev-u79uwXL29TY76Z2rM5mHXA Cc: RDS Devel, linux-rdma-u79uwXL29TY76Z2rM5mHXA Hi Dave, Sorry for the "release late, release seldom". We've made a lot of changes since spring but it never really stabilized.. Summary: * Add atomic operations (cswp and fadd) and masked versions * New RDS_RDMA_SILENT flag will try to execute RDMA op without causing an interrupt on the target machine * Massive performance improvements on big boxes (Chris Mason) * Many rmmod refcounting fixes (Zach Brown) * Do much more from tasklets instead of the worker thread * Add rds.h to kernel exported headers list * Other cleanups Thanks -- Regards -- Andy ------ The following changes since commit db40980fcdb560d7992b0511df16cdd3f7e381f3: Eric Dumazet (1): net: poll() optimizations are available in the git repository at: git://oss.oracle.com/git/agrover/linux-2.6.git for-davem Andy Grover (66): RDS: Fix BUG_ONs to not fire when in a tasklet RDS: Rewrite rds_send_drop_to() for clarity RDS: Fix locking in send on m_rs_lock RDS: move rds_shutdown_worker impl. to rds_conn_shutdown RDS: cleanup: remove "== NULL"s and "!= NULL"s in ptr comparisons RDS: break out rdma and data ops into nested structs in rds_message RDS: cleanup/fix rds_rdma_unuse RDS: Explicitly allocate rm in sendmsg() RDS: fold rdma.h into rds.h RDS: make m_rdma_op a member of rds_message RDS: make sure all sgs alloced are initialized RDS: Clear up some confusing code in send_remove_from_sock RDS: Implement atomic operations RDS: Base init_depth and responder_resources on hw values RDS: Do not mask address when pinning pages RDS: Whitespace RDS: inc_purge() transport function unused - remove it RDS/IB: eliminate duplicate code RDS/IB: Disallow connections less than RDS 3.1 RDS/IB: Remove dead code RDS/IB: Remove ib_[header/data]_sge() functions RDS: rewrite rds_ib_xmit RDS: Remove unsignaled_bytes sysctl RDS/IB: Make all flow control code conditional on i_flowctl RDS: Move loop-only function to loop.c RDS: Implement silent atomics RDS: Inline rdma_prepare into cmsg_rdma_args RDS: purge atomic resources too in rds_message_purge() RDS: Remove struct rds_rdma_op RDS: Rename data op members prefix from m_ to op_ RDS: Rewrite rds_send_xmit RDS: Do not set op_active in r_m_copy_from_user(). RDS: Add a warning if trying to allocate 0 sgs RDS: queue failure notifications for dropped atomic ops RDS: Move some variables around for consistency RDS: Add flag for silent ops. Do atomic op before RDMA RDS: Make sure cmsgs aren't used in improper ways RDS: Perform unmapping ops in stages RDS/IB: Do not wait for send ring to be empty on conn shutdown RDS: Stop supporting old cong map sending method RDS: Refill recv ring directly from tasklet RDS: Change send lock from a mutex to a spinlock RDS: rds_send_xmit() locking/irq fixes RDS: Call rds_send_xmit() directly from sendmsg() RDS: Bypass workqueue when queueing cong updates RDS: Use NOWAIT in message_map_pages() RDS: Fix locking in send on m_rs_lock RDS: rds_message_unmapped() doesn't need to check if queue active RDS: Move atomic stats from general to ib-specific area RDS: Remove send_quota from send_xmit() RDS: Do wait_event_interruptible instead of wait_event RDS: Get pong working again RDS: Update comments in rds_send_xmit() RDS/IB: Remove unused variable in ib_remove_addr() RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node() RDS: Use page_remainder_alloc() for recv bufs RDS: Assume recv->r_frag is always NULL in refill_one() RDS/IB: Remove ib_recv_unmap_page() RDS: whitespace RDS: Remove unused XLIST_PTR_TAIL and xlist_protect() RDS/IB: Make ib_recv_refill return void RDS/IB: Use SLAB_HWCACHE_ALIGN flag for kmem_cache_create() RDS: Implement masked atomic operations RDS: Add rds.h to exported headers list RDS: rds.h: Replace u_int[size]_t with uint[size]_t RDS: Remove dead struct from rds.h Chris Mason (17): RDS: Use a generation counter to avoid rds_send_xmit loop rds: switch to rwlock on bind_lock rds: per-rm flush_wait waitq rds: rcu-ize rds_ib_get_device() rds: Use RCU for the bind lookup searches rds: don't let RDS shutdown a connection while senders are present rds: Fix RDMA message reference counting RDS: use locking on the connection hash list rds: use RCU to protect the connection hash rds: Fix reference counting on the for xmit_atomic and xmit_rdma RDS/IB: Add caching of frags and incs RDS: use friendly gfp masks for prefill rds: recycle FMRs through lockless lists rds: more FMRs are faster RDS: use delayed work for the FMR flushes RDS: properly use sg_init_table RDS: flush fmrs before allocating new ones Tina Yang (1): RDS: Fix corrupted rds_mrs Zach Brown (23): RDS/IB: rds_ib_cm_handle_connect() forgot to unlock c_cm_lock RDS/IB: get the xmit max_sge from the RDS IB device on the connection RDS/IB: add refcount tracking to struct rds_ib_device RDS: introduce rds_conn_connect_if_down() RDS/IB: disconnect when IB devices are removed rds: remove unused rds_send_acked_before() rds: block ints when acquiring c_lock in rds_conn_message_info() rds: fix rds_send_xmit() serialization RDS/IB: wait for IB dev freeing work to finish during rmmod RDS/IB: destroy connections on rmmod RDS/IB: create a work queue for FMR flushing RDS: return to a single-threaded krdsd RDS/IB: always process recv completions RDS: remove __init and __exit annotation RDS/IB: track signaled sends RDS/IB: print IB event strings as well as their number RDS/IB: protect the list of IB devices RDS: lock rds_conn_count decrement in rds_conn_destroy() RDS: remove old rs_transport comment RDS: have sockets get transport module references RDS: don't call rds_conn_shutdown() from rds_conn_destroy() RDS: cancel connection work structs as we shut down RDS/IB: print string constants in more places include/linux/Kbuild | 1 + include/linux/rds.h | 106 +++++--- net/rds/af_rds.c | 26 ++- net/rds/bind.c | 82 +++--- net/rds/cong.c | 8 +- net/rds/connection.c | 157 ++++++++--- net/rds/ib.c | 194 +++++++++++--- net/rds/ib.h | 100 ++++---- net/rds/ib_cm.c | 183 +++++++++---- net/rds/ib_rdma.c | 314 +++++++++++++++++----- net/rds/ib_recv.c | 549 ++++++++++++++++++++++--------------- net/rds/ib_send.c | 682 +++++++++++++++++++++++++++------------------- net/rds/ib_stats.c | 2 + net/rds/ib_sysctl.c | 17 +- net/rds/info.c | 12 +- net/rds/iw.c | 4 +- net/rds/iw.h | 11 +- net/rds/iw_cm.c | 14 +- net/rds/iw_rdma.c | 1 - net/rds/iw_recv.c | 24 +- net/rds/iw_send.c | 93 ++++--- net/rds/iw_sysctl.c | 4 +- net/rds/loop.c | 31 ++- net/rds/message.c | 118 +++++---- net/rds/page.c | 5 +- net/rds/rdma.c | 339 +++++++++++++++-------- net/rds/rdma.h | 85 ------ net/rds/rdma_transport.c | 42 +++- net/rds/rds.h | 187 +++++++++++--- net/rds/recv.c | 9 +- net/rds/send.c | 544 ++++++++++++++++++++++--------------- net/rds/stats.c | 6 +- net/rds/sysctl.c | 4 +- net/rds/tcp.c | 8 +- net/rds/tcp.h | 9 +- net/rds/tcp_connect.c | 2 +- net/rds/tcp_listen.c | 6 +- net/rds/tcp_recv.c | 14 +- net/rds/tcp_send.c | 66 +---- net/rds/threads.c | 69 +---- net/rds/transport.c | 19 +- net/rds/xlist.h | 80 ++++++ 42 files changed, 2614 insertions(+), 1613 deletions(-) delete mode 100644 net/rds/rdma.h create mode 100644 net/rds/xlist.h -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
[parent not found: <4C892C9C.4060304-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>]
* Re: net-next pull request: RDS [not found] ` <4C892C9C.4060304-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> @ 2010-09-09 21:58 ` David Miller 2010-09-12 8:49 ` Or Gerlitz 1 sibling, 0 replies; 10+ messages in thread From: David Miller @ 2010-09-09 21:58 UTC (permalink / raw) To: andy.grover-QHcLZuEGTsvQT0dZR+AlfA Cc: netdev-u79uwXL29TY76Z2rM5mHXA, rds-devel-N0ozoZBvEnrZJqsBc5GL+g, linux-rdma-u79uwXL29TY76Z2rM5mHXA From: Andy Grover <andy.grover-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> Date: Thu, 09 Sep 2010 11:51:08 -0700 > Hi Dave, > > Sorry for the "release late, release seldom". We've made a lot of > changes since spring but it never really stabilized.. > > Summary: > * Add atomic operations (cswp and fadd) and masked versions > * New RDS_RDMA_SILENT flag will try to execute RDMA op without > causing an interrupt on the target machine > * Massive performance improvements on big boxes (Chris Mason) > * Many rmmod refcounting fixes (Zach Brown) > * Do much more from tasklets instead of the worker thread > * Add rds.h to kernel exported headers list > * Other cleanups I'll pull, but yeah you guys have to sync with me a lot more often. It would be nice if you guys CC:'d netdev for every patch that ends up in your tree as well. Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: net-next pull request: RDS [not found] ` <4C892C9C.4060304-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2010-09-09 21:58 ` David Miller @ 2010-09-12 8:49 ` Or Gerlitz [not found] ` <4C8C9422.2070306-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> 2010-09-13 17:14 ` Chris Mason 1 sibling, 2 replies; 10+ messages in thread From: Or Gerlitz @ 2010-09-12 8:49 UTC (permalink / raw) To: Andy Grover Cc: netdev-u79uwXL29TY76Z2rM5mHXA, RDS Devel, linux-rdma-u79uwXL29TY76Z2rM5mHXA, Roland Dreier Hi Andy, Some clarifications/questions from whatever quick look one can have over 107 patches... Zach Brown's "RDS/IB: print IB event strings as well as their number" - commit 1bde04a63d532c2540d6fdee0a661530a62b1686 in net-next-2.6 looks perfect to reside as a helper function in the core IB stack which can be in use by other rdma drivers (e.g ipoib, iser, srp, etc). Chris Mason's "rds: recycle FMRs through lockless lists added net/rds/xlist.h" - 6fa70da6081bbcf948801fd5ee0be4d222298a43 adds net/rds/xlist.h - isn't this something that better be placed under include/linux/. etc? And last, your "RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node()" patch looks interesting, 1st, it has some macros which could be placed in more general locations e.g pcidev_to_node and ibdev_to_node, your "significantly helps performance" comment is interesting, I'll send a separate note about that to the rdma mailing list. Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
[parent not found: <4C8C9422.2070306-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>]
* Re: RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node() [not found] ` <4C8C9422.2070306-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> @ 2010-09-12 8:52 ` Or Gerlitz [not found] ` <4C8C94B8.9030107-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> 2010-09-14 18:35 ` [rds-devel] net-next pull request: RDS Zach Brown 1 sibling, 1 reply; 10+ messages in thread From: Or Gerlitz @ 2010-09-12 8:52 UTC (permalink / raw) To: Andy Grover; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Roland Dreier Hi Andy, looking on this net-next-2.6 patch, I wonder if you can elaborate on your "significantly helps performance" comment - what improvement you see with this patch? What about the QP/CQ memory, are they better be placed in node-local to the HCA manner? Or. commit e4c52c98e04937ea87b0979a81354d0040d284f9 Author: Andy Grover <andy.grover-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> Date: Fri Apr 23 10:49:53 2010 -0700 RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node() Allocate send/recv rings in memory that is node-local to the HCA. This significantly helps performance. Signed-off-by: Andy Grover <andy.grover-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> diff --git a/net/rds/ib.c b/net/rds/ib.c index 7a2131d..7d289d7 100644 --- a/net/rds/ib.c +++ b/net/rds/ib.c @@ -77,7 +77,7 @@ void rds_ib_add_one(struct ib_device *device) goto free_attr; } - rds_ibdev = kmalloc(sizeof *rds_ibdev, GFP_KERNEL); + rds_ibdev = kmalloc_node(sizeof *rds_ibdev, GFP_KERNEL, ibdev_to_node(device)); if (!rds_ibdev) goto free_attr; diff --git a/net/rds/ib.h b/net/rds/ib.h index c506604..4bc3e2f 100644 --- a/net/rds/ib.h +++ b/net/rds/ib.h @@ -3,6 +3,8 @@ #include <rdma/ib_verbs.h> #include <rdma/rdma_cm.h> +#include <linux/pci.h> +#include <linux/slab.h> #include "rds.h" #include "rdma_transport.h" @@ -167,6 +169,10 @@ struct rds_ib_device { spinlock_t spinlock; /* protect the above */ }; +#define pcidev_to_node(pcidev) pcibus_to_node(pcidev->bus) +#define ibdev_to_node(ibdev) pcidev_to_node(to_pci_dev(ibdev->dma_device)) +#define rdsibdev_to_node(rdsibdev) ibdev_to_node(rdsibdev->dev) + /* bits for i_ack_flags */ #define IB_ACK_IN_FLIGHT 0 #define IB_ACK_REQUESTED 1 diff --git a/net/rds/ib_cm.c b/net/rds/ib_cm.c index 75eda9c..b5d0b60 100644 --- a/net/rds/ib_cm.c +++ b/net/rds/ib_cm.c @@ -347,7 +347,8 @@ static int rds_ib_setup_qp(struct rds_connection *conn) goto out; } - ic->i_sends = vmalloc(ic->i_send_ring.w_nr * sizeof(struct rds_ib_send_work)); + ic->i_sends = vmalloc_node(ic->i_send_ring.w_nr * sizeof(struct rds_ib_send_work), + ibdev_to_node(dev)); if (!ic->i_sends) { ret = -ENOMEM; rdsdebug("send allocation failed\n"); @@ -355,7 +356,8 @@ static int rds_ib_setup_qp(struct rds_connection *conn) } memset(ic->i_sends, 0, ic->i_send_ring.w_nr * sizeof(struct rds_ib_send_work)); - ic->i_recvs = vmalloc(ic->i_recv_ring.w_nr * sizeof(struct rds_ib_recv_work)); + ic->i_recvs = vmalloc_node(ic->i_recv_ring.w_nr * sizeof(struct rds_ib_recv_work), + ibdev_to_node(dev)); if (!ic->i_recvs) { ret = -ENOMEM; rdsdebug("recv allocation failed\n"); diff --git a/net/rds/ib_rdma.c b/net/rds/ib_rdma.c index 7315fff..cc341cd 100644 --- a/net/rds/ib_rdma.c +++ b/net/rds/ib_rdma.c @@ -297,7 +297,7 @@ static struct rds_ib_mr *rds_ib_alloc_fmr(struct rds_ib_device *rds_ibdev) rds_ib_flush_mr_pool(pool, 0); } - ibmr = kzalloc(sizeof(*ibmr), GFP_KERNEL); + ibmr = kzalloc_node(sizeof(*ibmr), GFP_KERNEL, rdsibdev_to_node(rds_ibdev)); if (!ibmr) { err = -ENOMEM; goto out_no_cigar; @@ -376,7 +376,8 @@ static int rds_ib_map_fmr(struct rds_ib_device *rds_ibdev, struct rds_ib_mr *ibm if (page_cnt > fmr_message_size) return -EINVAL; - dma_pages = kmalloc(sizeof(u64) * page_cnt, GFP_ATOMIC); + dma_pages = kmalloc_node(sizeof(u64) * page_cnt, GFP_ATOMIC, + rdsibdev_to_node(rds_ibdev)); if (!dma_pages) return -ENOMEM; -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 10+ messages in thread
[parent not found: <4C8C94B8.9030107-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org>]
* Re: RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node() [not found] ` <4C8C94B8.9030107-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> @ 2010-09-14 23:53 ` Andy Grover 0 siblings, 0 replies; 10+ messages in thread From: Andy Grover @ 2010-09-14 23:53 UTC (permalink / raw) To: Or Gerlitz; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA, Roland Dreier, Chris Mason On 09/12/2010 01:52 AM, Or Gerlitz wrote: > Hi Andy, looking on this net-next-2.6 patch, I wonder if you can > elaborate on your "significantly helps performance" comment - what > improvement you see with this patch? > What about the QP/CQ memory, are they better be placed in node-local > to the HCA manner? Hi Or, I'm afraid I don't have exact numbers for you.. those changes were part of a series of tweaks and fixes for bottlenecks in rds code. On our numa box, the HCA is attached to a particular package of CPUs & memory. Having the in-memory structures that the hca is accessing in node-local RAM seemed to measurably help, although reducing contention helped even more. Chris Mason says it seemed like init-time allocations seemed to benefit while runtime allocations were not helped, or even hurt. Clearly this is not enough data to go on, beyond the conclusion that yes it's something that should be looked at more closely. Big machines are still rather scarce, which makes it harder. Regards -- Andy -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [rds-devel] net-next pull request: RDS [not found] ` <4C8C9422.2070306-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> 2010-09-12 8:52 ` RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node() Or Gerlitz @ 2010-09-14 18:35 ` Zach Brown [not found] ` <20100914183514.GA3417-Mg0Rv1oTIzK+NQz5YmTpAAHTzkgPINtM@public.gmane.org> 1 sibling, 1 reply; 10+ messages in thread From: Zach Brown @ 2010-09-14 18:35 UTC (permalink / raw) To: Or Gerlitz Cc: Andy Grover, linux-rdma-u79uwXL29TY76Z2rM5mHXA, netdev-u79uwXL29TY76Z2rM5mHXA, RDS Devel, Roland Dreier > Zach Brown's "RDS/IB: print IB event strings as well as their number" - commit > 1bde04a63d532c2540d6fdee0a661530a62b1686 in net-next-2.6 looks perfect to > reside as a helper function in the core IB stack which can be in use by other > rdma drivers (e.g ipoib, iser, srp, etc). Yeah, if those other drivers were interested in the little helpers then it would make sense to bring the functionality up into the core stack. Note that there's another later commit in net-next that gives other event strings the same treatment: 59f740a6aeb2cde2f79fe0df38262d4c1ef35cd8 . - z -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
[parent not found: <20100914183514.GA3417-Mg0Rv1oTIzK+NQz5YmTpAAHTzkgPINtM@public.gmane.org>]
* Re: [rds-devel] net-next pull request: RDS [not found] ` <20100914183514.GA3417-Mg0Rv1oTIzK+NQz5YmTpAAHTzkgPINtM@public.gmane.org> @ 2010-09-14 22:28 ` Andrew Grover [not found] ` <AANLkTin69m0kK6bqUiwYeHRQT0Ehso16xA6RBtBFFgW3-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 10+ messages in thread From: Andrew Grover @ 2010-09-14 22:28 UTC (permalink / raw) To: Zach Brown Cc: Or Gerlitz, Andy Grover, linux-rdma-u79uwXL29TY76Z2rM5mHXA, netdev-u79uwXL29TY76Z2rM5mHXA, RDS Devel, Roland Dreier On Tue, Sep 14, 2010 at 11:35 AM, Zach Brown <zach.brown-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote: >> Zach Brown's "RDS/IB: print IB event strings as well as their number" - commit >> 1bde04a63d532c2540d6fdee0a661530a62b1686 in net-next-2.6 looks perfect to >> reside as a helper function in the core IB stack which can be in use by other >> rdma drivers (e.g ipoib, iser, srp, etc). > > Yeah, if those other drivers were interested in the little helpers then it > would make sense to bring the functionality up into the core stack. > > Note that there's another later commit in net-next that gives other event > strings the same treatment: 59f740a6aeb2cde2f79fe0df38262d4c1ef35cd8 . Since rds goes upstream via netdev not linux-rdma, it's a little tricky.. any core ib helpers we added would not be in net-next so rds wouldn't build. :) Once net-next gets pushed to mainline and Roland pulls from that, then we'll be in a good position to put these helpers where they should go, and change other ULPs to use them. -- Andy -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
[parent not found: <AANLkTin69m0kK6bqUiwYeHRQT0Ehso16xA6RBtBFFgW3-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: [rds-devel] net-next pull request: RDS [not found] ` <AANLkTin69m0kK6bqUiwYeHRQT0Ehso16xA6RBtBFFgW3-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2010-09-14 23:10 ` Roland Dreier 2010-09-15 9:22 ` Or Gerlitz 1 sibling, 0 replies; 10+ messages in thread From: Roland Dreier @ 2010-09-14 23:10 UTC (permalink / raw) To: Andrew Grover Cc: Zach Brown, Or Gerlitz, Andy Grover, linux-rdma-u79uwXL29TY76Z2rM5mHXA, netdev-u79uwXL29TY76Z2rM5mHXA, RDS Devel > Since rds goes upstream via netdev not linux-rdma, it's a little > tricky.. any core ib helpers we added would not be in net-next so rds > wouldn't build. :) Once net-next gets pushed to mainline and Roland > pulls from that, then we'll be in a good position to put these helpers > where they should go, and change other ULPs to use them. In cases where cross-tree dependencies exist, I'm happy to review core IB stack changes but then have them go upstream via davem after linux-rdma is happy with them. - R. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [rds-devel] net-next pull request: RDS [not found] ` <AANLkTin69m0kK6bqUiwYeHRQT0Ehso16xA6RBtBFFgW3-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2010-09-14 23:10 ` Roland Dreier @ 2010-09-15 9:22 ` Or Gerlitz 1 sibling, 0 replies; 10+ messages in thread From: Or Gerlitz @ 2010-09-15 9:22 UTC (permalink / raw) To: Andy Grover, Roland Dreier Cc: Zach Brown, linux-rdma-u79uwXL29TY76Z2rM5mHXA, RDS Devel Andrew Grover wrote: > Once net-next gets pushed to mainline and Roland pulls from that, > then we'll be in a good position to put these helpers where they should go, > and change other ULPs to use them. Andy, as Roland commented, you can push such helpers through Dave once Roland made a review of them, in a similar manner to a situation where an iser patch is merged by the iscsi maintainer, etc. if going to the review now... Roland - what's your take on the below patch net-next-2.6 and also on its net-next-2.6 "RDS/IB: print IB event strings as well as their number" 1bde04a63d532c2540d6fdee0a661530a62b1686 buddy? Or. commit 59f740a6aeb2cde2f79fe0df38262d4c1ef35cd8 Author: Zach Brown <zach.brown-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> Date: Tue Aug 3 13:52:47 2010 -0700 RDS/IB: print string constants in more places This prints the constant identifier for work completion status and rdma cm event types, like we already do for IB event types. A core string array helper is added that each string type uses. Signed-off-by: Zach Brown <zach.brown-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> diff --git a/net/rds/af_rds.c b/net/rds/af_rds.c index 8e3886d..bb6ad81 100644 --- a/net/rds/af_rds.c +++ b/net/rds/af_rds.c @@ -40,6 +40,15 @@ #include "rds.h" +char *rds_str_array(char **array, size_t elements, size_t index) +{ + if ((index < elements) && array[index]) + return array[index]; + else + return "unknown"; +} +EXPORT_SYMBOL(rds_str_array); + /* this is just used for stats gathering :/ */ static DEFINE_SPINLOCK(rds_sock_lock); static unsigned long rds_sock_count; diff --git a/net/rds/ib.h b/net/rds/ib.h index 2189fd4..7ad3d57 100644 --- a/net/rds/ib.h +++ b/net/rds/ib.h @@ -345,6 +345,7 @@ u32 rds_ib_ring_completed(struct rds_ib_work_ring *ring, u32 wr_id, u32 oldest); extern wait_queue_head_t rds_ib_ring_empty_wait; /* ib_send.c */ +char *rds_ib_wc_status_str(enum ib_wc_status status); void rds_ib_xmit_complete(struct rds_connection *conn); int rds_ib_xmit(struct rds_connection *conn, struct rds_message *rm, unsigned int hdr_off, unsigned int sg, unsigned int off); diff --git a/net/rds/ib_cm.c b/net/rds/ib_cm.c index 0e2fea8..bc3dbc1 100644 --- a/net/rds/ib_cm.c +++ b/net/rds/ib_cm.c @@ -39,7 +39,8 @@ #include "ib.h" static char *rds_ib_event_type_strings[] = { -#define RDS_IB_EVENT_STRING(foo) [IB_EVENT_##foo] = __stringify(foo) +#define RDS_IB_EVENT_STRING(foo) \ + [IB_EVENT_##foo] = __stringify(IB_EVENT_##foo) RDS_IB_EVENT_STRING(CQ_ERR), RDS_IB_EVENT_STRING(QP_FATAL), RDS_IB_EVENT_STRING(QP_REQ_ERR), @@ -63,11 +64,8 @@ static char *rds_ib_event_type_strings[] = { static char *rds_ib_event_str(enum ib_event_type type) { - if (type < ARRAY_SIZE(rds_ib_event_type_strings) && - rds_ib_event_type_strings[type]) - return rds_ib_event_type_strings[type]; - else - return "unknown"; + return rds_str_array(rds_ib_event_type_strings, + ARRAY_SIZE(rds_ib_event_type_strings), type); }; /* diff --git a/net/rds/ib_recv.c b/net/rds/ib_recv.c index a2f5f6f..e29e0ca 100644 --- a/net/rds/ib_recv.c +++ b/net/rds/ib_recv.c @@ -966,8 +966,9 @@ static inline void rds_poll_cq(struct rds_ib_connection *ic, struct rds_ib_recv_work *recv; while (ib_poll_cq(ic->i_recv_cq, 1, &wc) > 0) { - rdsdebug("wc wr_id 0x%llx status %u byte_len %u imm_data %u\n", - (unsigned long long)wc.wr_id, wc.status, wc.byte_len, + rdsdebug("wc wr_id 0x%llx status %u (%s) byte_len %u imm_data %u\n", + (unsigned long long)wc.wr_id, wc.status, + rds_ib_wc_status_str(wc.status), wc.byte_len, be32_to_cpu(wc.ex.imm_data)); rds_ib_stats_inc(s_ib_rx_cq_event); @@ -985,10 +986,11 @@ static inline void rds_poll_cq(struct rds_ib_connection *ic, } else { /* We expect errors as the qp is drained during shutdown */ if (rds_conn_up(conn) || rds_conn_connecting(conn)) - rds_ib_conn_error(conn, "recv completion on " - "%pI4 had status %u, disconnecting and " + rds_ib_conn_error(conn, "recv completion on %pI4 had " + "status %u (%s), disconnecting and " "reconnecting\n", &conn->c_faddr, - wc.status); + wc.status, + rds_ib_wc_status_str(wc.status)); } /* diff --git a/net/rds/ib_send.c b/net/rds/ib_send.c index 15f7569..808544a 100644 --- a/net/rds/ib_send.c +++ b/net/rds/ib_send.c @@ -38,6 +38,40 @@ #include "rds.h" #include "ib.h" +static char *rds_ib_wc_status_strings[] = { +#define RDS_IB_WC_STATUS_STR(foo) \ + [IB_WC_##foo] = __stringify(IB_WC_##foo) + RDS_IB_WC_STATUS_STR(SUCCESS), + RDS_IB_WC_STATUS_STR(LOC_LEN_ERR), + RDS_IB_WC_STATUS_STR(LOC_QP_OP_ERR), + RDS_IB_WC_STATUS_STR(LOC_EEC_OP_ERR), + RDS_IB_WC_STATUS_STR(LOC_PROT_ERR), + RDS_IB_WC_STATUS_STR(WR_FLUSH_ERR), + RDS_IB_WC_STATUS_STR(MW_BIND_ERR), + RDS_IB_WC_STATUS_STR(BAD_RESP_ERR), + RDS_IB_WC_STATUS_STR(LOC_ACCESS_ERR), + RDS_IB_WC_STATUS_STR(REM_INV_REQ_ERR), + RDS_IB_WC_STATUS_STR(REM_ACCESS_ERR), + RDS_IB_WC_STATUS_STR(REM_OP_ERR), + RDS_IB_WC_STATUS_STR(RETRY_EXC_ERR), + RDS_IB_WC_STATUS_STR(RNR_RETRY_EXC_ERR), + RDS_IB_WC_STATUS_STR(LOC_RDD_VIOL_ERR), + RDS_IB_WC_STATUS_STR(REM_INV_RD_REQ_ERR), + RDS_IB_WC_STATUS_STR(REM_ABORT_ERR), + RDS_IB_WC_STATUS_STR(INV_EECN_ERR), + RDS_IB_WC_STATUS_STR(INV_EEC_STATE_ERR), + RDS_IB_WC_STATUS_STR(FATAL_ERR), + RDS_IB_WC_STATUS_STR(RESP_TIMEOUT_ERR), + RDS_IB_WC_STATUS_STR(GENERAL_ERR), +#undef RDS_IB_WC_STATUS_STR +}; + +char *rds_ib_wc_status_str(enum ib_wc_status status) +{ + return rds_str_array(rds_ib_wc_status_strings, + ARRAY_SIZE(rds_ib_wc_status_strings), status); +} + /* * Convert IB-specific error message to RDS error message and call core * completion handler. @@ -257,8 +291,9 @@ void rds_ib_send_cq_comp_handler(struct ib_cq *cq, void *context) rdsdebug("ib_req_notify_cq send failed: %d\n", ret); while (ib_poll_cq(cq, 1, &wc) > 0) { - rdsdebug("wc wr_id 0x%llx status %u byte_len %u imm_data %u\n", - (unsigned long long)wc.wr_id, wc.status, wc.byte_len, + rdsdebug("wc wr_id 0x%llx status %u (%s) byte_len %u imm_data %u\n", + (unsigned long long)wc.wr_id, wc.status, + rds_ib_wc_status_str(wc.status), wc.byte_len, be32_to_cpu(wc.ex.imm_data)); rds_ib_stats_inc(s_ib_tx_cq_event); @@ -306,10 +341,10 @@ void rds_ib_send_cq_comp_handler(struct ib_cq *cq, void *context) /* We expect errors as the qp is drained during shutdown */ if (wc.status != IB_WC_SUCCESS && rds_conn_up(conn)) { - rds_ib_conn_error(conn, - "send completion on %pI4 " - "had status %u, disconnecting and reconnecting\n", - &conn->c_faddr, wc.status); + rds_ib_conn_error(conn, "send completion on %pI4 had status " + "%u (%s), disconnecting and reconnecting\n", + &conn->c_faddr, wc.status, + rds_ib_wc_status_str(wc.status)); } } } diff --git a/net/rds/rdma_transport.c b/net/rds/rdma_transport.c index 550d348..e6ed10a 100644 --- a/net/rds/rdma_transport.c +++ b/net/rds/rdma_transport.c @@ -36,6 +36,34 @@ static struct rdma_cm_id *rds_rdma_listen_id; +static char *rds_cm_event_strings[] = { +#define RDS_CM_EVENT_STRING(foo) \ + [RDMA_CM_EVENT_##foo] = __stringify(RDMA_CM_EVENT_##foo) + RDS_CM_EVENT_STRING(ADDR_RESOLVED), + RDS_CM_EVENT_STRING(ADDR_ERROR), + RDS_CM_EVENT_STRING(ROUTE_RESOLVED), + RDS_CM_EVENT_STRING(ROUTE_ERROR), + RDS_CM_EVENT_STRING(CONNECT_REQUEST), + RDS_CM_EVENT_STRING(CONNECT_RESPONSE), + RDS_CM_EVENT_STRING(CONNECT_ERROR), + RDS_CM_EVENT_STRING(UNREACHABLE), + RDS_CM_EVENT_STRING(REJECTED), + RDS_CM_EVENT_STRING(ESTABLISHED), + RDS_CM_EVENT_STRING(DISCONNECTED), + RDS_CM_EVENT_STRING(DEVICE_REMOVAL), + RDS_CM_EVENT_STRING(MULTICAST_JOIN), + RDS_CM_EVENT_STRING(MULTICAST_ERROR), + RDS_CM_EVENT_STRING(ADDR_CHANGE), + RDS_CM_EVENT_STRING(TIMEWAIT_EXIT), +#undef RDS_CM_EVENT_STRING +}; + +static char *rds_cm_event_str(enum rdma_cm_event_type type) +{ + return rds_str_array(rds_cm_event_strings, + ARRAY_SIZE(rds_cm_event_strings), type); +}; + int rds_rdma_cm_event_handler(struct rdma_cm_id *cm_id, struct rdma_cm_event *event) { @@ -44,8 +72,8 @@ int rds_rdma_cm_event_handler(struct rdma_cm_id *cm_id, struct rds_transport *trans; int ret = 0; - rdsdebug("conn %p id %p handling event %u\n", conn, cm_id, - event->event); + rdsdebug("conn %p id %p handling event %u (%s)\n", conn, cm_id, + event->event, rds_cm_event_str(event->event)); if (cm_id->device->node_type == RDMA_NODE_RNIC) trans = &rds_iw_transport; @@ -109,7 +137,8 @@ int rds_rdma_cm_event_handler(struct rdma_cm_id *cm_id, default: /* things like device disconnect? */ - printk(KERN_ERR "RDS: unknown event %u!\n", event->event); + printk(KERN_ERR "RDS: unknown event %u (%s)!\n", + event->event, rds_cm_event_str(event->event)); break; } @@ -117,7 +146,8 @@ out: if (conn) mutex_unlock(&conn->c_cm_lock); - rdsdebug("id %p event %u handling ret %d\n", cm_id, event->event, ret); + rdsdebug("id %p event %u (%s) handling ret %d\n", cm_id, event->event, + rds_cm_event_str(event->event), ret); return ret; } diff --git a/net/rds/rds.h b/net/rds/rds.h index aab5e94..aadaddb 100644 --- a/net/rds/rds.h +++ b/net/rds/rds.h @@ -566,6 +566,7 @@ struct rds_statistics { }; /* af_rds.c */ +char *rds_str_array(char **array, size_t elements, size_t index); void rds_sock_addref(struct rds_sock *rs); void rds_sock_put(struct rds_sock *rs); void rds_wake_sk_sleep(struct rds_sock *rs); -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [rds-devel] net-next pull request: RDS 2010-09-12 8:49 ` Or Gerlitz [not found] ` <4C8C9422.2070306-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> @ 2010-09-13 17:14 ` Chris Mason 1 sibling, 0 replies; 10+ messages in thread From: Chris Mason @ 2010-09-13 17:14 UTC (permalink / raw) To: Or Gerlitz; +Cc: Andy Grover, linux-rdma, netdev, RDS Devel, Roland Dreier On Sun, Sep 12, 2010 at 10:49:38AM +0200, Or Gerlitz wrote: > Hi Andy, > > Some clarifications/questions from whatever quick look one can have over 107 patches... > > Zach Brown's "RDS/IB: print IB event strings as well as their number" - commit > 1bde04a63d532c2540d6fdee0a661530a62b1686 in net-next-2.6 looks perfect to reside as a helper function in the core IB stack which can be in use by other rdma drivers (e.g ipoib, iser, srp, etc). > > Chris Mason's "rds: recycle FMRs through lockless lists added net/rds/xlist.h" - 6fa70da6081bbcf948801fd5ee0be4d222298a43 adds net/rds/xlist.h - isn't this something that better be placed under include/linux/. etc? Yes, I plan on expanding xlist.h a little and trying to use it in a few other places. But since RDS is currently the only consumer it made sense to me to keep it in rds until I had more code in line to take advantage of it. -chris ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2010-09-15 9:22 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-09-09 18:51 net-next pull request: RDS Andy Grover [not found] ` <4C892C9C.4060304-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2010-09-09 21:58 ` David Miller 2010-09-12 8:49 ` Or Gerlitz [not found] ` <4C8C9422.2070306-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> 2010-09-12 8:52 ` RDS/IB: add _to_node() macros for numa and use {k,v}malloc_node() Or Gerlitz [not found] ` <4C8C94B8.9030107-hKgKHo2Ms0FWk0Htik3J/w@public.gmane.org> 2010-09-14 23:53 ` Andy Grover 2010-09-14 18:35 ` [rds-devel] net-next pull request: RDS Zach Brown [not found] ` <20100914183514.GA3417-Mg0Rv1oTIzK+NQz5YmTpAAHTzkgPINtM@public.gmane.org> 2010-09-14 22:28 ` Andrew Grover [not found] ` <AANLkTin69m0kK6bqUiwYeHRQT0Ehso16xA6RBtBFFgW3-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2010-09-14 23:10 ` Roland Dreier 2010-09-15 9:22 ` Or Gerlitz 2010-09-13 17:14 ` Chris Mason
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.