[PATCH v4 0/9] rdma/cm: Add support for native Infiniband addressing

[PATCH v4 0/9] rdma/cm: Add support for native Infiniband addressing

[sean.hefty-ral2JQCrhuEAvxtiuMwx3w]

From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

This is the first group of patches in a series that adds the
ability to handle native Infiniband addressing to the rdma_cm.
I'm breaking the submission up to limit the number of patches
that need to be reviewed at once, hoping that this helps with
merging the code upstream.  The full patch series is available
from:

        git://git.openfabrics.org/~shefty/rdma-dev.git for-next

Allowing the use of native IB addresses removes the requirement
for IPoIB, which in turn allows us to offload name and/or address
translation services to a user space daemon.  The primary motivation
is to support large scale fabrics, with address and name services
either cached or bypassed completely.  For example, IB GIDs are
known or the information is exchanged out of band by an MPI process
manager.  However, another use case involves load balancing software.
Currently the rdma cm cannot establish rdma connections through
a load balancer, since the IP -> GID mapping is not well defined.
An out of band mechanism could be used in such situations to
determine the correct mapping, with the rdma cm still managing
the connection.

The patch set introduces af_ib and sockaddr_ib.  The kernel
rdma_cm is updated accordingly, mainly to make its handling of
addresses more generic.  However, since sockaddr_ib is larger
than sockaddr_in6, the rdma_ucm requires changes to its user to
kernel interface.  To provide backwards compatibility, the userspace
ABI is extended to support the larger address size.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Changes from v3:
Updated to 3.8-rc4 kernel

Sean Hefty (9):
  rdma/cm: define native IB address
  rdma/cm: Include AF_IB in loopback and any address checks
  ib/addr: Add AF_IB support to ip_addr_size
  rdma/cm: Update port reservation to support AF_IB
  rdma/cm: Allow user to specify AF_IB when binding
  rdma/cm: Do not modify sa_family when setting loopback address
  rdma/cm: Add helper functions to return id address information
  rdma/cm: Restrict AF_IB loopback to binding to IB devices only
  rdma/cm: Verify that source and dest sa_family are the same

 drivers/infiniband/core/addr.c |   20 ++-
 drivers/infiniband/core/cma.c  |  374 ++++++++++++++++++++++++++--------------
 include/linux/socket.h         |    2 +
 include/rdma/ib.h              |   89 ++++++++++
 include/rdma/ib_addr.h         |    6 +-
 include/rdma/rdma_cm.h         |    5 +
 6 files changed, 359 insertions(+), 137 deletions(-)
 create mode 100644 include/rdma/ib.h

-- 
1.7.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4 1/9] rdma/cm: define native IB address

[sean.hefty-ral2JQCrhuEAvxtiuMwx3w]

From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Define AF_IB and sockaddr_ib to allow the rdma_cm to use native IB
addressing.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 include/linux/socket.h |    2 +
 include/rdma/ib.h      |   89 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+), 0 deletions(-)
 create mode 100644 include/rdma/ib.h

diff --git a/include/linux/socket.h b/include/linux/socket.h
index 9a546ff..17a33f7 100644
--- a/include/linux/socket.h
+++ b/include/linux/socket.h
@@ -167,6 +167,7 @@ struct ucred {
 #define AF_PPPOX	24	/* PPPoX sockets		*/
 #define AF_WANPIPE	25	/* Wanpipe API Sockets */
 #define AF_LLC		26	/* Linux LLC			*/
+#define AF_IB		27	/* Native InfiniBand address	*/
 #define AF_CAN		29	/* Controller Area Network      */
 #define AF_TIPC		30	/* TIPC sockets			*/
 #define AF_BLUETOOTH	31	/* Bluetooth sockets 		*/
@@ -210,6 +211,7 @@ struct ucred {
 #define PF_PPPOX	AF_PPPOX
 #define PF_WANPIPE	AF_WANPIPE
 #define PF_LLC		AF_LLC
+#define PF_IB		AF_IB
 #define PF_CAN		AF_CAN
 #define PF_TIPC		AF_TIPC
 #define PF_BLUETOOTH	AF_BLUETOOTH
diff --git a/include/rdma/ib.h b/include/rdma/ib.h
new file mode 100644
index 0000000..cf8f9e7
--- /dev/null
+++ b/include/rdma/ib.h
@@ -0,0 +1,89 @@
+/*
+ * Copyright (c) 2010 Intel Corporation.  All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses.  You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ *     Redistribution and use in source and binary forms, with or
+ *     without modification, are permitted provided that the following
+ *     conditions are met:
+ *
+ *      - Redistributions of source code must retain the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer.
+ *
+ *      - Redistributions in binary form must reproduce the above
+ *        copyright notice, this list of conditions and the following
+ *        disclaimer in the documentation and/or other materials
+ *        provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#if !defined(_RDMA_IB_H)
+#define _RDMA_IB_H
+
+#include <linux/types.h>
+
+struct ib_addr {
+	union {
+		__u8		uib_addr8[16];
+		__be16		uib_addr16[8];
+		__be32		uib_addr32[4];
+		__be64		uib_addr64[2];
+	} ib_u;
+#define sib_addr8		ib_u.uib_addr8
+#define sib_addr16		ib_u.uib_addr16
+#define sib_addr32		ib_u.uib_addr32
+#define sib_addr64		ib_u.uib_addr64
+#define sib_raw			ib_u.uib_addr8
+#define sib_subnet_prefix	ib_u.uib_addr64[0]
+#define sib_interface_id	ib_u.uib_addr64[1]
+};
+
+static inline int ib_addr_any(const struct ib_addr *a)
+{
+	return ((a->sib_addr64[0] | a->sib_addr64[1]) == 0);
+}
+
+static inline int ib_addr_loopback(const struct ib_addr *a)
+{
+	return ((a->sib_addr32[0] | a->sib_addr32[1] |
+		 a->sib_addr32[2] | (a->sib_addr32[3] ^ htonl(1))) == 0);
+}
+
+static inline void ib_addr_set(struct ib_addr *addr,
+			       __be32 w1, __be32 w2, __be32 w3, __be32 w4)
+{
+	addr->sib_addr32[0] = w1;
+	addr->sib_addr32[1] = w2;
+	addr->sib_addr32[2] = w3;
+	addr->sib_addr32[3] = w4;
+}
+
+static inline int ib_addr_cmp(const struct ib_addr *a1, const struct ib_addr *a2)
+{
+	return memcmp(a1, a2, sizeof(struct ib_addr));
+}
+
+struct sockaddr_ib {
+	unsigned short int	sib_family;	/* AF_IB */
+	__be16			sib_pkey;
+	__be32			sib_flowinfo;
+	struct ib_addr		sib_addr;
+	__be64			sib_sid;
+	__be64			sib_sid_mask;
+	__u64			sib_scope_id;
+};
+
+#endif /* _RDMA_IB_H */
-- 
1.7.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4 2/9] rdma/cm: Include AF_IB in loopback and any address checks

[sean.hefty-ral2JQCrhuEAvxtiuMwx3w]

From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Enhance checks for loopback and any address to support AF_IB
in addition to AF_INET and AF_INT6.  This will allow future
patches to use AF_IB when binding and resolving addresses.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/core/cma.c |   40 ++++++++++++++++++++++++----------------
 1 files changed, 24 insertions(+), 16 deletions(-)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index d789eea..9e3dc7f 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -50,6 +50,7 @@
 #include <rdma/rdma_cm.h>
 #include <rdma/rdma_cm_ib.h>
 #include <rdma/rdma_netlink.h>
+#include <rdma/ib.h>
 #include <rdma/ib_cache.h>
 #include <rdma/ib_cm.h>
 #include <rdma/ib_sa.h>
@@ -679,26 +680,30 @@ EXPORT_SYMBOL(rdma_init_qp_attr);
 
 static inline int cma_zero_addr(struct sockaddr *addr)
 {
-	struct in6_addr *ip6;
-
-	if (addr->sa_family == AF_INET)
-		return ipv4_is_zeronet(
-			((struct sockaddr_in *)addr)->sin_addr.s_addr);
-	else {
-		ip6 = &((struct sockaddr_in6 *) addr)->sin6_addr;
-		return (ip6->s6_addr32[0] | ip6->s6_addr32[1] |
-			ip6->s6_addr32[2] | ip6->s6_addr32[3]) == 0;
+	switch (addr->sa_family) {
+	case AF_INET:
+		return ipv4_is_zeronet(((struct sockaddr_in *)addr)->sin_addr.s_addr);
+	case AF_INET6:
+		return ipv6_addr_any(&((struct sockaddr_in6 *) addr)->sin6_addr);
+	case AF_IB:
+		return ib_addr_any(&((struct sockaddr_ib *) addr)->sib_addr);
+	default:
+		return 0;
 	}
 }
 
 static inline int cma_loopback_addr(struct sockaddr *addr)
 {
-	if (addr->sa_family == AF_INET)
-		return ipv4_is_loopback(
-			((struct sockaddr_in *) addr)->sin_addr.s_addr);
-	else
-		return ipv6_addr_loopback(
-			&((struct sockaddr_in6 *) addr)->sin6_addr);
+	switch (addr->sa_family) {
+	case AF_INET:
+		return ipv4_is_loopback(((struct sockaddr_in *) addr)->sin_addr.s_addr);
+	case AF_INET6:
+		return ipv6_addr_loopback(&((struct sockaddr_in6 *) addr)->sin6_addr);
+	case AF_IB:
+		return ib_addr_loopback(&((struct sockaddr_ib *) addr)->sib_addr);
+	default:
+		return 0;
+	}
 }
 
 static inline int cma_any_addr(struct sockaddr *addr)
@@ -715,9 +720,12 @@ static int cma_addr_cmp(struct sockaddr *src, struct sockaddr *dst)
 	case AF_INET:
 		return ((struct sockaddr_in *) src)->sin_addr.s_addr !=
 		       ((struct sockaddr_in *) dst)->sin_addr.s_addr;
-	default:
+	case AF_INET6:
 		return ipv6_addr_cmp(&((struct sockaddr_in6 *) src)->sin6_addr,
 				     &((struct sockaddr_in6 *) dst)->sin6_addr);
+	default:
+		return ib_addr_cmp(&((struct sockaddr_ib *) src)->sib_addr,
+				   &((struct sockaddr_ib *) dst)->sib_addr);
 	}
 }
 
-- 
1.7.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4 3/9] ib/addr: Add AF_IB support to ip_addr_size

[sean.hefty]

From: Sean Hefty <sean.hefty@intel.com>

Add support for AF_IB to ip_addr_size, and rename the function
to account for the change.  Give the compiler more control over
whether the call should be inline or not by moving the definition
into the .c file, removing the static inline, and exporting it.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
---
 drivers/infiniband/core/addr.c |   20 ++++++++++++++++++--
 drivers/infiniband/core/cma.c  |   12 ++++++------
 include/rdma/ib_addr.h         |    6 +-----
 3 files changed, 25 insertions(+), 13 deletions(-)

diff --git a/drivers/infiniband/core/addr.c b/drivers/infiniband/core/addr.c
index eaec8d7..e90f2b2 100644
--- a/drivers/infiniband/core/addr.c
+++ b/drivers/infiniband/core/addr.c
@@ -45,6 +45,7 @@
 #include <net/addrconf.h>
 #include <net/ip6_route.h>
 #include <rdma/ib_addr.h>
+#include <rdma/ib.h>
 
 MODULE_AUTHOR("Sean Hefty");
 MODULE_DESCRIPTION("IB Address Translation");
@@ -70,6 +71,21 @@ static LIST_HEAD(req_list);
 static DECLARE_DELAYED_WORK(work, process_req);
 static struct workqueue_struct *addr_wq;
 
+int rdma_addr_size(struct sockaddr *addr)
+{
+	switch (addr->sa_family) {
+	case AF_INET:
+		return sizeof(struct sockaddr_in);
+	case AF_INET6:
+		return sizeof(struct sockaddr_in6);
+	case AF_IB:
+		return sizeof(struct sockaddr_ib);
+	default:
+		return 0;
+	}
+}
+EXPORT_SYMBOL(rdma_addr_size);
+
 void rdma_addr_register_client(struct rdma_addr_client *client)
 {
 	atomic_set(&client->refcount, 1);
@@ -369,12 +385,12 @@ int rdma_resolve_ip(struct rdma_addr_client *client,
 			goto err;
 		}
 
-		memcpy(src_in, src_addr, ip_addr_size(src_addr));
+		memcpy(src_in, src_addr, rdma_addr_size(src_addr));
 	} else {
 		src_in->sa_family = dst_addr->sa_family;
 	}
 
-	memcpy(dst_in, dst_addr, ip_addr_size(dst_addr));
+	memcpy(dst_in, dst_addr, rdma_addr_size(dst_addr));
 	req->addr = addr;
 	req->callback = callback;
 	req->context = context;
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 9e3dc7f..e08f830 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -1584,7 +1584,7 @@ static void cma_listen_on_dev(struct rdma_id_private *id_priv,
 
 	dev_id_priv->state = RDMA_CM_ADDR_BOUND;
 	memcpy(&id->route.addr.src_addr, &id_priv->id.route.addr.src_addr,
-	       ip_addr_size((struct sockaddr *) &id_priv->id.route.addr.src_addr));
+	       rdma_addr_size((struct sockaddr *) &id_priv->id.route.addr.src_addr));
 
 	cma_attach_to_dev(dev_id_priv, cma_dev);
 	list_add_tail(&dev_id_priv->listen_list, &id_priv->listen_list);
@@ -1989,7 +1989,7 @@ static void addr_handler(int status, struct sockaddr *src_addr,
 		event.status = status;
 	} else {
 		memcpy(&id_priv->id.route.addr.src_addr, src_addr,
-		       ip_addr_size(src_addr));
+		       rdma_addr_size(src_addr));
 		event.event = RDMA_CM_EVENT_ADDR_RESOLVED;
 	}
 
@@ -2079,7 +2079,7 @@ int rdma_resolve_addr(struct rdma_cm_id *id, struct sockaddr *src_addr,
 		return -EINVAL;
 
 	atomic_inc(&id_priv->refcount);
-	memcpy(&id->route.addr.dst_addr, dst_addr, ip_addr_size(dst_addr));
+	memcpy(&id->route.addr.dst_addr, dst_addr, rdma_addr_size(dst_addr));
 	if (cma_any_addr(dst_addr))
 		ret = cma_resolve_loopback(id_priv);
 	else
@@ -2410,7 +2410,7 @@ int rdma_bind_addr(struct rdma_cm_id *id, struct sockaddr *addr)
 			goto err1;
 	}
 
-	memcpy(&id->route.addr.src_addr, addr, ip_addr_size(addr));
+	memcpy(&id->route.addr.src_addr, addr, rdma_addr_size(addr));
 	if (!(id_priv->options & (1 << CMA_OPTION_AFONLY))) {
 		if (addr->sa_family == AF_INET)
 			id_priv->afonly = 1;
@@ -3189,7 +3189,7 @@ int rdma_join_multicast(struct rdma_cm_id *id, struct sockaddr *addr,
 	if (!mc)
 		return -ENOMEM;
 
-	memcpy(&mc->addr, addr, ip_addr_size(addr));
+	memcpy(&mc->addr, addr, rdma_addr_size(addr));
 	mc->context = context;
 	mc->id_priv = id_priv;
 
@@ -3234,7 +3234,7 @@ void rdma_leave_multicast(struct rdma_cm_id *id, struct sockaddr *addr)
 	id_priv = container_of(id, struct rdma_id_private, id);
 	spin_lock_irq(&id_priv->lock);
 	list_for_each_entry(mc, &id_priv->mc_list, list) {
-		if (!memcmp(&mc->addr, addr, ip_addr_size(addr))) {
+		if (!memcmp(&mc->addr, addr, rdma_addr_size(addr))) {
 			list_del(&mc->list);
 			spin_unlock_irq(&id_priv->lock);
 
diff --git a/include/rdma/ib_addr.h b/include/rdma/ib_addr.h
index 9996539..f3ac0f2 100644
--- a/include/rdma/ib_addr.h
+++ b/include/rdma/ib_addr.h
@@ -102,11 +102,7 @@ void rdma_addr_cancel(struct rdma_dev_addr *addr);
 int rdma_copy_addr(struct rdma_dev_addr *dev_addr, struct net_device *dev,
 	      const unsigned char *dst_dev_addr);
 
-static inline int ip_addr_size(struct sockaddr *addr)
-{
-	return addr->sa_family == AF_INET6 ?
-	       sizeof(struct sockaddr_in6) : sizeof(struct sockaddr_in);
-}
+int rdma_addr_size(struct sockaddr *addr);
 
 static inline u16 ib_addr_get_pkey(struct rdma_dev_addr *dev_addr)
 {
-- 
1.7.3

[PATCH v4 4/9] rdma/cm: Update port reservation to support AF_IB

[sean.hefty-ral2JQCrhuEAvxtiuMwx3w]

From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

The AF_IB uses a 64-bit service id (SID), which the
user can control through the use of a mask.  The rdma_cm
will assign values to the unmasked portions of the SID
based on the selected port space and port number.

Because the IB spec divides the SID range into several regions,
a SID/mask combination may fall into one of the existing
port space ranges as defined by the RDMA CM IP Annex.  Map
the AF_IB SID to the correct RDMA port space.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/core/cma.c |  107 +++++++++++++++++++++++++++++++++--------
 include/rdma/rdma_cm.h        |    5 ++
 2 files changed, 91 insertions(+), 21 deletions(-)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index e08f830..7eed4ee 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -729,12 +729,22 @@ static int cma_addr_cmp(struct sockaddr *src, struct sockaddr *dst)
 	}
 }
 
-static inline __be16 cma_port(struct sockaddr *addr)
+static __be16 cma_port(struct sockaddr *addr)
 {
-	if (addr->sa_family == AF_INET)
+	struct sockaddr_ib *sib;
+
+	switch (addr->sa_family) {
+	case AF_INET:
 		return ((struct sockaddr_in *) addr)->sin_port;
-	else
+	case AF_INET6:
 		return ((struct sockaddr_in6 *) addr)->sin6_port;
+	case AF_IB:
+		sib = (struct sockaddr_ib *) addr;
+		return htons((u16) (be64_to_cpu(sib->sib_sid) &
+				    be64_to_cpu(sib->sib_sid_mask)));
+	default:
+		return 0;
+	}
 }
 
 static inline int cma_any_port(struct sockaddr *addr)
@@ -2139,10 +2149,29 @@ EXPORT_SYMBOL(rdma_set_afonly);
 static void cma_bind_port(struct rdma_bind_list *bind_list,
 			  struct rdma_id_private *id_priv)
 {
-	struct sockaddr_in *sin;
+	struct sockaddr *addr;
+	struct sockaddr_ib *sib;
+	u64 sid, mask;
+	__be16 port;
 
-	sin = (struct sockaddr_in *) &id_priv->id.route.addr.src_addr;
-	sin->sin_port = htons(bind_list->port);
+	addr = (struct sockaddr *) &id_priv->id.route.addr.src_addr;
+	port = htons(bind_list->port);
+
+	switch (addr->sa_family) {
+	case AF_INET:
+		((struct sockaddr_in *) addr)->sin_port = port;
+		break;
+	case AF_INET6:
+		((struct sockaddr_in6 *) addr)->sin6_port = port;
+		break;
+	case AF_IB:
+		sib = (struct sockaddr_ib *) addr;
+		sid = be64_to_cpu(sib->sib_sid);
+		mask = be64_to_cpu(sib->sib_sid_mask);
+		sib->sib_sid = cpu_to_be64((sid & mask) | (u64) ntohs(port));
+		sib->sib_sid_mask = cpu_to_be64(~0ULL);
+		break;
+	}
 	id_priv->bind_list = bind_list;
 	hlist_add_head(&id_priv->node, &bind_list->owners);
 }
@@ -2280,31 +2309,67 @@ static int cma_bind_listen(struct rdma_id_private *id_priv)
 	return ret;
 }
 
-static int cma_get_port(struct rdma_id_private *id_priv)
+static struct idr *cma_select_inet_ps(struct rdma_id_private *id_priv)
 {
-	struct idr *ps;
-	int ret;
-
 	switch (id_priv->id.ps) {
 	case RDMA_PS_SDP:
-		ps = &sdp_ps;
-		break;
+		return &sdp_ps;
 	case RDMA_PS_TCP:
-		ps = &tcp_ps;
-		break;
+		return &tcp_ps;
 	case RDMA_PS_UDP:
-		ps = &udp_ps;
-		break;
+		return &udp_ps;
 	case RDMA_PS_IPOIB:
-		ps = &ipoib_ps;
-		break;
+		return &ipoib_ps;
 	case RDMA_PS_IB:
-		ps = &ib_ps;
-		break;
+		return &ib_ps;
 	default:
-		return -EPROTONOSUPPORT;
+		return NULL;
+	}
+}
+
+static struct idr *cma_select_ib_ps(struct rdma_id_private *id_priv)
+{
+	struct idr *ps = NULL;
+	struct sockaddr_ib *sib;
+	u64 sid_ps, mask, sid;
+
+	sib = (struct sockaddr_ib *) &id_priv->id.route.addr.src_addr;
+	mask = be64_to_cpu(sib->sib_sid_mask) & RDMA_IB_IP_PS_MASK;
+	sid = be64_to_cpu(sib->sib_sid) & mask;
+
+	if ((id_priv->id.ps == RDMA_PS_IB) && (sid == (RDMA_IB_IP_PS_IB & mask))) {
+		sid_ps = RDMA_IB_IP_PS_IB;
+		ps = &ib_ps;
+	} else if (((id_priv->id.ps == RDMA_PS_IB) || (id_priv->id.ps == RDMA_PS_TCP)) &&
+		   (sid == (RDMA_IB_IP_PS_TCP & mask))) {
+		sid_ps = RDMA_IB_IP_PS_TCP;
+		ps = &tcp_ps;
+	} else if (((id_priv->id.ps == RDMA_PS_IB) || (id_priv->id.ps == RDMA_PS_UDP)) &&
+		   (sid == (RDMA_IB_IP_PS_UDP & mask))) {
+		sid_ps = RDMA_IB_IP_PS_UDP;
+		ps = &udp_ps;
 	}
 
+	if (ps) {
+		sib->sib_sid = cpu_to_be64(sid_ps | ntohs(cma_port((struct sockaddr *) sib)));
+		sib->sib_sid_mask = cpu_to_be64(RDMA_IB_IP_PS_MASK |
+						be64_to_cpu(sib->sib_sid_mask));
+	}
+	return ps;
+}
+
+static int cma_get_port(struct rdma_id_private *id_priv)
+{
+	struct idr *ps;
+	int ret;
+
+	if (id_priv->id.route.addr.src_addr.ss_family != AF_IB)
+		ps = cma_select_inet_ps(id_priv);
+	else
+		ps = cma_select_ib_ps(id_priv);
+	if (!ps)
+		return -EPROTONOSUPPORT;
+
 	mutex_lock(&lock);
 	if (cma_any_port((struct sockaddr *) &id_priv->id.route.addr.src_addr))
 		ret = cma_alloc_any_port(ps, id_priv);
diff --git a/include/rdma/rdma_cm.h b/include/rdma/rdma_cm.h
index ad3a314..1e6c3c7 100644
--- a/include/rdma/rdma_cm.h
+++ b/include/rdma/rdma_cm.h
@@ -70,6 +70,11 @@ enum rdma_port_space {
 	RDMA_PS_UDP   = 0x0111,
 };
 
+#define RDMA_IB_IP_PS_MASK   0xFFFFFFFFFFFF0000ULL
+#define RDMA_IB_IP_PS_TCP    0x0000000001060000ULL
+#define RDMA_IB_IP_PS_UDP    0x0000000001110000ULL
+#define RDMA_IB_IP_PS_IB     0x00000000013F0000ULL
+
 struct rdma_addr {
 	struct sockaddr_storage src_addr;
 	struct sockaddr_storage dst_addr;
-- 
1.7.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4 5/9] rdma/cm: Allow user to specify AF_IB when binding

[sean.hefty]

From: Sean Hefty <sean.hefty@intel.com>

Modify rdma_bind_addr to allow the user to specify AF_IB when
binding to a device.  AF_IB indicates that the user is not
mapping an IP address to the native IB addressing.  (The mapping
may have already been done, or is not needed.)

Signed-off-by: Sean Hefty <sean.hefty@intel.com>
---
 drivers/infiniband/core/cma.c |   34 ++++++++++++++++++++++++++++------
 1 files changed, 28 insertions(+), 6 deletions(-)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 7eed4ee..caea4ef3 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -359,6 +359,27 @@ static int find_gid_port(struct ib_device *device, union ib_gid *gid, u8 port_nu
 	return -EADDRNOTAVAIL;
 }
 
+static void cma_translate_ib(struct sockaddr_ib *sib, struct rdma_dev_addr *dev_addr)
+{
+	dev_addr->dev_type = ARPHRD_INFINIBAND;
+	rdma_addr_set_sgid(dev_addr, (union ib_gid *) &sib->sib_addr);
+	ib_addr_set_pkey(dev_addr, ntohs(sib->sib_pkey));
+}
+
+static int cma_translate_addr(struct sockaddr *addr, struct rdma_dev_addr *dev_addr)
+{
+	int ret;
+
+	if (addr->sa_family != AF_IB) {
+		ret = rdma_translate_ip(addr, dev_addr);
+	} else {
+		cma_translate_ib((struct sockaddr_ib *) addr, dev_addr);
+		ret = 0;
+	}
+
+	return ret;
+}
+
 static int cma_acquire_dev(struct rdma_id_private *id_priv)
 {
 	struct rdma_dev_addr *dev_addr = &id_priv->id.route.addr.dev_addr;
@@ -1136,8 +1157,8 @@ static struct rdma_id_private *cma_new_conn_id(struct rdma_cm_id *listen_id,
 		rdma_addr_set_sgid(&rt->addr.dev_addr, &rt->path_rec[0].sgid);
 		ib_addr_set_pkey(&rt->addr.dev_addr, be16_to_cpu(rt->path_rec[0].pkey));
 	} else {
-		ret = rdma_translate_ip((struct sockaddr *) &rt->addr.src_addr,
-					&rt->addr.dev_addr);
+		ret = cma_translate_addr((struct sockaddr *) &rt->addr.src_addr,
+					 &rt->addr.dev_addr);
 		if (ret)
 			goto err;
 	}
@@ -1176,8 +1197,8 @@ static struct rdma_id_private *cma_new_udp_id(struct rdma_cm_id *listen_id,
 			  ip_ver, port, src, dst);
 
 	if (!cma_any_addr((struct sockaddr *) &id->route.addr.src_addr)) {
-		ret = rdma_translate_ip((struct sockaddr *) &id->route.addr.src_addr,
-					&id->route.addr.dev_addr);
+		ret = cma_translate_addr((struct sockaddr *) &id->route.addr.src_addr,
+					 &id->route.addr.dev_addr);
 		if (ret)
 			goto err;
 	}
@@ -2454,7 +2475,8 @@ int rdma_bind_addr(struct rdma_cm_id *id, struct sockaddr *addr)
 	struct rdma_id_private *id_priv;
 	int ret;
 
-	if (addr->sa_family != AF_INET && addr->sa_family != AF_INET6)
+	if (addr->sa_family != AF_INET && addr->sa_family != AF_INET6 &&
+	    addr->sa_family != AF_IB)
 		return -EAFNOSUPPORT;
 
 	id_priv = container_of(id, struct rdma_id_private, id);
@@ -2466,7 +2488,7 @@ int rdma_bind_addr(struct rdma_cm_id *id, struct sockaddr *addr)
 		goto err1;
 
 	if (!cma_any_addr(addr)) {
-		ret = rdma_translate_ip(addr, &id->route.addr.dev_addr);
+		ret = cma_translate_addr(addr, &id->route.addr.dev_addr);
 		if (ret)
 			goto err1;
 
-- 
1.7.3

[PATCH v4 6/9] rdma/cm: Do not modify sa_family when setting loopback address

[sean.hefty-ral2JQCrhuEAvxtiuMwx3w]

From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

cma_resolve_loopback is called after an rdma_cm_id has been
bound to a specific sa_family and port.  Once the
source sa_family for the id has been set, do not modify it.
Only the actual IP address portion of the source address
needs to be set.

As part of this fix, we can simplify setting the source address
by moving the loopback address assignment from cma_resolve_loopback
to cma_bind_loopback.  cma_bind_loopback is only invoked when
the source address is the loopback address.

Finally, add loopback support for AF_IB as part of the change.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/core/cma.c |   31 ++++++++++++++++++-------------
 1 files changed, 18 insertions(+), 13 deletions(-)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index caea4ef3..bb22885 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -1952,6 +1952,23 @@ err:
 }
 EXPORT_SYMBOL(rdma_resolve_route);
 
+static void cma_set_loopback(struct sockaddr *addr)
+{
+	switch (addr->sa_family) {
+	case AF_INET:
+		((struct sockaddr_in *) addr)->sin_addr.s_addr = htonl(INADDR_LOOPBACK);
+		break;
+	case AF_INET6:
+		ipv6_addr_set(&((struct sockaddr_in6 *) addr)->sin6_addr,
+			      0, 0, 0, htonl(1));
+		break;
+	default:
+		ib_addr_set(&((struct sockaddr_ib *) addr)->sib_addr,
+			    0, 0, 0, htonl(1));
+		break;
+	}
+}
+
 static int cma_bind_loopback(struct rdma_id_private *id_priv)
 {
 	struct cma_device *cma_dev;
@@ -1992,6 +2009,7 @@ port_found:
 	ib_addr_set_pkey(&id_priv->id.route.addr.dev_addr, pkey);
 	id_priv->id.port_num = p;
 	cma_attach_to_dev(id_priv, cma_dev);
+	cma_set_loopback((struct sockaddr *) &id_priv->id.route.addr.src_addr);
 out:
 	mutex_unlock(&lock);
 	return ret;
@@ -2039,7 +2057,6 @@ out:
 static int cma_resolve_loopback(struct rdma_id_private *id_priv)
 {
 	struct cma_work *work;
-	struct sockaddr *src, *dst;
 	union ib_gid gid;
 	int ret;
 
@@ -2056,18 +2073,6 @@ static int cma_resolve_loopback(struct rdma_id_private *id_priv)
 	rdma_addr_get_sgid(&id_priv->id.route.addr.dev_addr, &gid);
 	rdma_addr_set_dgid(&id_priv->id.route.addr.dev_addr, &gid);
 
-	src = (struct sockaddr *) &id_priv->id.route.addr.src_addr;
-	if (cma_zero_addr(src)) {
-		dst = (struct sockaddr *) &id_priv->id.route.addr.dst_addr;
-		if ((src->sa_family = dst->sa_family) == AF_INET) {
-			((struct sockaddr_in *)src)->sin_addr =
-				((struct sockaddr_in *)dst)->sin_addr;
-		} else {
-			((struct sockaddr_in6 *)src)->sin6_addr =
-				((struct sockaddr_in6 *)dst)->sin6_addr;
-		}
-	}
-
 	work->id = id_priv;
 	INIT_WORK(&work->work, cma_work_handler);
 	work->old_state = RDMA_CM_ADDR_QUERY;
-- 
1.7.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4 7/9] rdma/cm: Add helper functions to return id address information

[sean.hefty-ral2JQCrhuEAvxtiuMwx3w]

From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Provide inline helpers to extract source and destination address
data from the rdma_cm_id.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/core/cma.c |  138 +++++++++++++++++++++--------------------
 1 files changed, 71 insertions(+), 67 deletions(-)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index bb22885..0ebf123 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -311,6 +311,21 @@ static void cma_release_dev(struct rdma_id_private *id_priv)
 	mutex_unlock(&lock);
 }
 
+static inline struct sockaddr *cma_src_addr(struct rdma_id_private *id_priv)
+{
+	return (struct sockaddr *) &id_priv->id.route.addr.src_addr;
+}
+
+static inline struct sockaddr *cma_dst_addr(struct rdma_id_private *id_priv)
+{
+	return (struct sockaddr *) &id_priv->id.route.addr.dst_addr;
+}
+
+static inline unsigned short cma_family(struct rdma_id_private *id_priv)
+{
+	return id_priv->id.route.addr.src_addr.ss_family;
+}
+
 static int cma_set_qkey(struct rdma_id_private *id_priv)
 {
 	struct ib_sa_mcmember_rec rec;
@@ -900,8 +915,7 @@ static void cma_cancel_operation(struct rdma_id_private *id_priv,
 		cma_cancel_route(id_priv);
 		break;
 	case RDMA_CM_LISTEN:
-		if (cma_any_addr((struct sockaddr *) &id_priv->id.route.addr.src_addr)
-				&& !id_priv->cma_dev)
+		if (cma_any_addr(cma_src_addr(id_priv)) && !id_priv->cma_dev)
 			cma_cancel_listens(id_priv);
 		break;
 	default:
@@ -1138,6 +1152,7 @@ static struct rdma_id_private *cma_new_conn_id(struct rdma_cm_id *listen_id,
 	if (IS_ERR(id))
 		return NULL;
 
+	id_priv = container_of(id, struct rdma_id_private, id);
 	cma_save_net_info(&id->route.addr, &listen_id->route.addr,
 			  ip_ver, port, src, dst);
 
@@ -1152,19 +1167,17 @@ static struct rdma_id_private *cma_new_conn_id(struct rdma_cm_id *listen_id,
 	if (rt->num_paths == 2)
 		rt->path_rec[1] = *ib_event->param.req_rcvd.alternate_path;
 
-	if (cma_any_addr((struct sockaddr *) &rt->addr.src_addr)) {
+	if (cma_any_addr(cma_src_addr(id_priv))) {
 		rt->addr.dev_addr.dev_type = ARPHRD_INFINIBAND;
 		rdma_addr_set_sgid(&rt->addr.dev_addr, &rt->path_rec[0].sgid);
 		ib_addr_set_pkey(&rt->addr.dev_addr, be16_to_cpu(rt->path_rec[0].pkey));
 	} else {
-		ret = cma_translate_addr((struct sockaddr *) &rt->addr.src_addr,
-					 &rt->addr.dev_addr);
+		ret = cma_translate_addr(cma_src_addr(id_priv), &rt->addr.dev_addr);
 		if (ret)
 			goto err;
 	}
 	rdma_addr_set_dgid(&rt->addr.dev_addr, &rt->path_rec[0].dgid);
 
-	id_priv = container_of(id, struct rdma_id_private, id);
 	id_priv->state = RDMA_CM_CONNECT;
 	return id_priv;
 
@@ -1188,7 +1201,7 @@ static struct rdma_id_private *cma_new_udp_id(struct rdma_cm_id *listen_id,
 	if (IS_ERR(id))
 		return NULL;
 
-
+	id_priv = container_of(id, struct rdma_id_private, id);
 	if (cma_get_net_info(ib_event->private_data, listen_id->ps,
 			     &ip_ver, &port, &src, &dst))
 		goto err;
@@ -1197,13 +1210,11 @@ static struct rdma_id_private *cma_new_udp_id(struct rdma_cm_id *listen_id,
 			  ip_ver, port, src, dst);
 
 	if (!cma_any_addr((struct sockaddr *) &id->route.addr.src_addr)) {
-		ret = cma_translate_addr((struct sockaddr *) &id->route.addr.src_addr,
-					 &id->route.addr.dev_addr);
+		ret = cma_translate_addr(cma_src_addr(id_priv), &id->route.addr.dev_addr);
 		if (ret)
 			goto err;
 	}
 
-	id_priv = container_of(id, struct rdma_id_private, id);
 	id_priv->state = RDMA_CM_CONNECT;
 	return id_priv;
 err:
@@ -1386,9 +1397,9 @@ static int cma_iw_handler(struct iw_cm_id *iw_id, struct iw_cm_event *iw_event)
 		event.event = RDMA_CM_EVENT_DISCONNECTED;
 		break;
 	case IW_CM_EVENT_CONNECT_REPLY:
-		sin = (struct sockaddr_in *) &id_priv->id.route.addr.src_addr;
+		sin = (struct sockaddr_in *) cma_src_addr(id_priv);
 		*sin = iw_event->local_addr;
-		sin = (struct sockaddr_in *) &id_priv->id.route.addr.dst_addr;
+		sin = (struct sockaddr_in *) cma_dst_addr(id_priv);
 		*sin = iw_event->remote_addr;
 		switch (iw_event->status) {
 		case 0:
@@ -1486,9 +1497,9 @@ static int iw_conn_req_handler(struct iw_cm_id *cm_id,
 	cm_id->context = conn_id;
 	cm_id->cm_handler = cma_iw_handler;
 
-	sin = (struct sockaddr_in *) &new_cm_id->route.addr.src_addr;
+	sin = (struct sockaddr_in *) cma_src_addr(conn_id);
 	*sin = iw_event->local_addr;
-	sin = (struct sockaddr_in *) &new_cm_id->route.addr.dst_addr;
+	sin = (struct sockaddr_in *) cma_dst_addr(conn_id);
 	*sin = iw_event->remote_addr;
 
 	ret = ib_query_device(conn_id->id.device, &attr);
@@ -1545,7 +1556,7 @@ static int cma_ib_listen(struct rdma_id_private *id_priv)
 
 	id_priv->cm_id.ib = id;
 
-	addr = (struct sockaddr *) &id_priv->id.route.addr.src_addr;
+	addr = cma_src_addr(id_priv);
 	svc_id = cma_get_service_id(id_priv->id.ps, addr);
 	if (cma_any_addr(addr) && !id_priv->afonly)
 		ret = ib_cm_listen(id_priv->cm_id.ib, svc_id, 0, NULL);
@@ -1576,7 +1587,7 @@ static int cma_iw_listen(struct rdma_id_private *id_priv, int backlog)
 
 	id_priv->cm_id.iw = id;
 
-	sin = (struct sockaddr_in *) &id_priv->id.route.addr.src_addr;
+	sin = (struct sockaddr_in *) cma_src_addr(id_priv);
 	id_priv->cm_id.iw->local_addr = *sin;
 
 	ret = iw_cm_listen(id_priv->cm_id.iw, backlog);
@@ -1614,8 +1625,8 @@ static void cma_listen_on_dev(struct rdma_id_private *id_priv,
 	dev_id_priv = container_of(id, struct rdma_id_private, id);
 
 	dev_id_priv->state = RDMA_CM_ADDR_BOUND;
-	memcpy(&id->route.addr.src_addr, &id_priv->id.route.addr.src_addr,
-	       rdma_addr_size((struct sockaddr *) &id_priv->id.route.addr.src_addr));
+	memcpy(cma_src_addr(dev_id_priv), cma_src_addr(id_priv),
+	       rdma_addr_size(cma_src_addr(id_priv)));
 
 	cma_attach_to_dev(dev_id_priv, cma_dev);
 	list_add_tail(&dev_id_priv->listen_list, &id_priv->listen_list);
@@ -1673,29 +1684,28 @@ static void cma_query_handler(int status, struct ib_sa_path_rec *path_rec,
 static int cma_query_ib_route(struct rdma_id_private *id_priv, int timeout_ms,
 			      struct cma_work *work)
 {
-	struct rdma_addr *addr = &id_priv->id.route.addr;
+	struct rdma_dev_addr *dev_addr = &id_priv->id.route.addr.dev_addr;
 	struct ib_sa_path_rec path_rec;
 	ib_sa_comp_mask comp_mask;
 	struct sockaddr_in6 *sin6;
 
 	memset(&path_rec, 0, sizeof path_rec);
-	rdma_addr_get_sgid(&addr->dev_addr, &path_rec.sgid);
-	rdma_addr_get_dgid(&addr->dev_addr, &path_rec.dgid);
-	path_rec.pkey = cpu_to_be16(ib_addr_get_pkey(&addr->dev_addr));
+	rdma_addr_get_sgid(dev_addr, &path_rec.sgid);
+	rdma_addr_get_dgid(dev_addr, &path_rec.dgid);
+	path_rec.pkey = cpu_to_be16(ib_addr_get_pkey(dev_addr));
 	path_rec.numb_path = 1;
 	path_rec.reversible = 1;
-	path_rec.service_id = cma_get_service_id(id_priv->id.ps,
-							(struct sockaddr *) &addr->dst_addr);
+	path_rec.service_id = cma_get_service_id(id_priv->id.ps, cma_dst_addr(id_priv));
 
 	comp_mask = IB_SA_PATH_REC_DGID | IB_SA_PATH_REC_SGID |
 		    IB_SA_PATH_REC_PKEY | IB_SA_PATH_REC_NUMB_PATH |
 		    IB_SA_PATH_REC_REVERSIBLE | IB_SA_PATH_REC_SERVICE_ID;
 
-	if (addr->src_addr.ss_family == AF_INET) {
+	if (cma_family(id_priv) == AF_INET) {
 		path_rec.qos_class = cpu_to_be16((u16) id_priv->tos);
 		comp_mask |= IB_SA_PATH_REC_QOS_CLASS;
 	} else {
-		sin6 = (struct sockaddr_in6 *) &addr->src_addr;
+		sin6 = (struct sockaddr_in6 *) cma_src_addr(id_priv);
 		path_rec.traffic_class = (u8) (be32_to_cpu(sin6->sin6_flowinfo) >> 20);
 		comp_mask |= IB_SA_PATH_REC_TRAFFIC_CLASS;
 	}
@@ -2009,7 +2019,7 @@ port_found:
 	ib_addr_set_pkey(&id_priv->id.route.addr.dev_addr, pkey);
 	id_priv->id.port_num = p;
 	cma_attach_to_dev(id_priv, cma_dev);
-	cma_set_loopback((struct sockaddr *) &id_priv->id.route.addr.src_addr);
+	cma_set_loopback(cma_src_addr(id_priv));
 out:
 	mutex_unlock(&lock);
 	return ret;
@@ -2037,8 +2047,7 @@ static void addr_handler(int status, struct sockaddr *src_addr,
 		event.event = RDMA_CM_EVENT_ADDR_ERROR;
 		event.status = status;
 	} else {
-		memcpy(&id_priv->id.route.addr.src_addr, src_addr,
-		       rdma_addr_size(src_addr));
+		memcpy(cma_src_addr(id_priv), src_addr, rdma_addr_size(src_addr));
 		event.event = RDMA_CM_EVENT_ADDR_RESOLVED;
 	}
 
@@ -2115,11 +2124,11 @@ int rdma_resolve_addr(struct rdma_cm_id *id, struct sockaddr *src_addr,
 		return -EINVAL;
 
 	atomic_inc(&id_priv->refcount);
-	memcpy(&id->route.addr.dst_addr, dst_addr, rdma_addr_size(dst_addr));
+	memcpy(cma_dst_addr(id_priv), dst_addr, rdma_addr_size(dst_addr));
 	if (cma_any_addr(dst_addr))
 		ret = cma_resolve_loopback(id_priv);
 	else
-		ret = rdma_resolve_ip(&addr_client, (struct sockaddr *) &id->route.addr.src_addr,
+		ret = rdma_resolve_ip(&addr_client, cma_src_addr(id_priv),
 				      dst_addr, &id->route.addr.dev_addr,
 				      timeout_ms, addr_handler, id_priv);
 	if (ret)
@@ -2180,7 +2189,7 @@ static void cma_bind_port(struct rdma_bind_list *bind_list,
 	u64 sid, mask;
 	__be16 port;
 
-	addr = (struct sockaddr *) &id_priv->id.route.addr.src_addr;
+	addr = cma_src_addr(id_priv);
 	port = htons(bind_list->port);
 
 	switch (addr->sa_family) {
@@ -2279,7 +2288,7 @@ static int cma_check_port(struct rdma_bind_list *bind_list,
 	struct sockaddr *addr, *cur_addr;
 	struct hlist_node *node;
 
-	addr = (struct sockaddr *) &id_priv->id.route.addr.src_addr;
+	addr = cma_src_addr(id_priv);
 	hlist_for_each_entry(cur_id, node, &bind_list->owners, node) {
 		if (id_priv == cur_id)
 			continue;
@@ -2288,7 +2297,7 @@ static int cma_check_port(struct rdma_bind_list *bind_list,
 		    cur_id->reuseaddr)
 			continue;
 
-		cur_addr = (struct sockaddr *) &cur_id->id.route.addr.src_addr;
+		cur_addr = cma_src_addr(cur_id);
 		if (id_priv->afonly && cur_id->afonly &&
 		    (addr->sa_family != cur_addr->sa_family))
 			continue;
@@ -2308,7 +2317,7 @@ static int cma_use_port(struct idr *ps, struct rdma_id_private *id_priv)
 	unsigned short snum;
 	int ret;
 
-	snum = ntohs(cma_port((struct sockaddr *) &id_priv->id.route.addr.src_addr));
+	snum = ntohs(cma_port(cma_src_addr(id_priv)));
 	if (snum < PROT_SOCK && !capable(CAP_NET_BIND_SERVICE))
 		return -EACCES;
 
@@ -2359,7 +2368,7 @@ static struct idr *cma_select_ib_ps(struct rdma_id_private *id_priv)
 	struct sockaddr_ib *sib;
 	u64 sid_ps, mask, sid;
 
-	sib = (struct sockaddr_ib *) &id_priv->id.route.addr.src_addr;
+	sib = (struct sockaddr_ib *) cma_src_addr(id_priv);
 	mask = be64_to_cpu(sib->sib_sid_mask) & RDMA_IB_IP_PS_MASK;
 	sid = be64_to_cpu(sib->sib_sid) & mask;
 
@@ -2389,7 +2398,7 @@ static int cma_get_port(struct rdma_id_private *id_priv)
 	struct idr *ps;
 	int ret;
 
-	if (id_priv->id.route.addr.src_addr.ss_family != AF_IB)
+	if (cma_family(id_priv) != AF_IB)
 		ps = cma_select_inet_ps(id_priv);
 	else
 		ps = cma_select_ib_ps(id_priv);
@@ -2397,7 +2406,7 @@ static int cma_get_port(struct rdma_id_private *id_priv)
 		return -EPROTONOSUPPORT;
 
 	mutex_lock(&lock);
-	if (cma_any_port((struct sockaddr *) &id_priv->id.route.addr.src_addr))
+	if (cma_any_port(cma_src_addr(id_priv)))
 		ret = cma_alloc_any_port(ps, id_priv);
 	else
 		ret = cma_use_port(ps, id_priv);
@@ -2432,8 +2441,8 @@ int rdma_listen(struct rdma_cm_id *id, int backlog)
 
 	id_priv = container_of(id, struct rdma_id_private, id);
 	if (id_priv->state == RDMA_CM_IDLE) {
-		((struct sockaddr *) &id->route.addr.src_addr)->sa_family = AF_INET;
-		ret = rdma_bind_addr(id, (struct sockaddr *) &id->route.addr.src_addr);
+		id->route.addr.src_addr.ss_family = AF_INET;
+		ret = rdma_bind_addr(id, cma_src_addr(id_priv));
 		if (ret)
 			return ret;
 	}
@@ -2502,7 +2511,7 @@ int rdma_bind_addr(struct rdma_cm_id *id, struct sockaddr *addr)
 			goto err1;
 	}
 
-	memcpy(&id->route.addr.src_addr, addr, rdma_addr_size(addr));
+	memcpy(cma_src_addr(id_priv), addr, rdma_addr_size(addr));
 	if (!(id_priv->options & (1 << CMA_OPTION_AFONLY))) {
 		if (addr->sa_family == AF_INET)
 			id_priv->afonly = 1;
@@ -2525,19 +2534,18 @@ err1:
 }
 EXPORT_SYMBOL(rdma_bind_addr);
 
-static int cma_format_hdr(void *hdr, enum rdma_port_space ps,
-			  struct rdma_route *route)
+static int cma_format_hdr(void *hdr, struct rdma_id_private *id_priv)
 {
 	struct cma_hdr *cma_hdr;
 	struct sdp_hh *sdp_hdr;
 
-	if (route->addr.src_addr.ss_family == AF_INET) {
+	if (cma_family(id_priv) == AF_INET) {
 		struct sockaddr_in *src4, *dst4;
 
-		src4 = (struct sockaddr_in *) &route->addr.src_addr;
-		dst4 = (struct sockaddr_in *) &route->addr.dst_addr;
+		src4 = (struct sockaddr_in *) cma_src_addr(id_priv);
+		dst4 = (struct sockaddr_in *) cma_dst_addr(id_priv);
 
-		switch (ps) {
+		switch (id_priv->id.ps) {
 		case RDMA_PS_SDP:
 			sdp_hdr = hdr;
 			if (sdp_get_majv(sdp_hdr->sdp_version) != SDP_MAJ_VERSION)
@@ -2559,10 +2567,10 @@ static int cma_format_hdr(void *hdr, enum rdma_port_space ps,
 	} else {
 		struct sockaddr_in6 *src6, *dst6;
 
-		src6 = (struct sockaddr_in6 *) &route->addr.src_addr;
-		dst6 = (struct sockaddr_in6 *) &route->addr.dst_addr;
+		src6 = (struct sockaddr_in6 *) cma_src_addr(id_priv);
+		dst6 = (struct sockaddr_in6 *) cma_dst_addr(id_priv);
 
-		switch (ps) {
+		switch (id_priv->id.ps) {
 		case RDMA_PS_SDP:
 			sdp_hdr = hdr;
 			if (sdp_get_majv(sdp_hdr->sdp_version) != SDP_MAJ_VERSION)
@@ -2653,7 +2661,6 @@ static int cma_resolve_ib_udp(struct rdma_id_private *id_priv,
 			      struct rdma_conn_param *conn_param)
 {
 	struct ib_cm_sidr_req_param req;
-	struct rdma_route *route;
 	struct ib_cm_id	*id;
 	int ret;
 
@@ -2670,8 +2677,7 @@ static int cma_resolve_ib_udp(struct rdma_id_private *id_priv,
 		memcpy((void *) req.private_data + sizeof(struct cma_hdr),
 		       conn_param->private_data, conn_param->private_data_len);
 
-	route = &id_priv->id.route;
-	ret = cma_format_hdr((void *) req.private_data, id_priv->id.ps, route);
+	ret = cma_format_hdr((void *) req.private_data, id_priv);
 	if (ret)
 		goto out;
 
@@ -2683,9 +2689,8 @@ static int cma_resolve_ib_udp(struct rdma_id_private *id_priv,
 	}
 	id_priv->cm_id.ib = id;
 
-	req.path = route->path_rec;
-	req.service_id = cma_get_service_id(id_priv->id.ps,
-					    (struct sockaddr *) &route->addr.dst_addr);
+	req.path = id_priv->id.route.path_rec;
+	req.service_id = cma_get_service_id(id_priv->id.ps, cma_dst_addr(id_priv));
 	req.timeout_ms = 1 << (CMA_CM_RESPONSE_TIMEOUT - 8);
 	req.max_cm_retries = CMA_MAX_CM_RETRIES;
 
@@ -2730,7 +2735,7 @@ static int cma_connect_ib(struct rdma_id_private *id_priv,
 	id_priv->cm_id.ib = id;
 
 	route = &id_priv->id.route;
-	ret = cma_format_hdr(private_data, id_priv->id.ps, route);
+	ret = cma_format_hdr(private_data, id_priv);
 	if (ret)
 		goto out;
 	req.private_data = private_data;
@@ -2739,8 +2744,7 @@ static int cma_connect_ib(struct rdma_id_private *id_priv,
 	if (route->num_paths == 2)
 		req.alternate_path = &route->path_rec[1];
 
-	req.service_id = cma_get_service_id(id_priv->id.ps,
-					    (struct sockaddr *) &route->addr.dst_addr);
+	req.service_id = cma_get_service_id(id_priv->id.ps, cma_dst_addr(id_priv));
 	req.qp_num = id_priv->qp_num;
 	req.qp_type = id_priv->id.qp_type;
 	req.starting_psn = id_priv->seq_num;
@@ -2779,10 +2783,10 @@ static int cma_connect_iw(struct rdma_id_private *id_priv,
 
 	id_priv->cm_id.iw = cm_id;
 
-	sin = (struct sockaddr_in*) &id_priv->id.route.addr.src_addr;
+	sin = (struct sockaddr_in *) cma_src_addr(id_priv);
 	cm_id->local_addr = *sin;
 
-	sin = (struct sockaddr_in*) &id_priv->id.route.addr.dst_addr;
+	sin = (struct sockaddr_in *) cma_dst_addr(id_priv);
 	cm_id->remote_addr = *sin;
 
 	ret = cma_modify_qp_rtr(id_priv, conn_param);
@@ -3547,29 +3551,29 @@ static int cma_get_id_stats(struct sk_buff *skb, struct netlink_callback *cb)
 			id_stats->bound_dev_if =
 				id->route.addr.dev_addr.bound_dev_if;
 
-			if (id->route.addr.src_addr.ss_family == AF_INET) {
+			if (cma_family(id_priv) == AF_INET) {
 				if (ibnl_put_attr(skb, nlh,
 						  sizeof(struct sockaddr_in),
-						  &id->route.addr.src_addr,
+						  cma_src_addr(id_priv),
 						  RDMA_NL_RDMA_CM_ATTR_SRC_ADDR)) {
 					goto out;
 				}
 				if (ibnl_put_attr(skb, nlh,
 						  sizeof(struct sockaddr_in),
-						  &id->route.addr.dst_addr,
+						  cma_dst_addr(id_priv),
 						  RDMA_NL_RDMA_CM_ATTR_DST_ADDR)) {
 					goto out;
 				}
-			} else if (id->route.addr.src_addr.ss_family == AF_INET6) {
+			} else if (cma_family(id_priv) == AF_INET6) {
 				if (ibnl_put_attr(skb, nlh,
 						  sizeof(struct sockaddr_in6),
-						  &id->route.addr.src_addr,
+						  cma_src_addr(id_priv),
 						  RDMA_NL_RDMA_CM_ATTR_SRC_ADDR)) {
 					goto out;
 				}
 				if (ibnl_put_attr(skb, nlh,
 						  sizeof(struct sockaddr_in6),
-						  &id->route.addr.dst_addr,
+						  cma_dst_addr(id_priv),
 						  RDMA_NL_RDMA_CM_ATTR_DST_ADDR)) {
 					goto out;
 				}
-- 
1.7.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4 8/9] rdma/cm: Restrict AF_IB loopback to binding to IB devices only

[sean.hefty-ral2JQCrhuEAvxtiuMwx3w]

From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

If a user specifies AF_IB as the source address for a loopback
connection, limit the resolution to IB devices only.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/core/cma.c |   28 ++++++++++++++++++++--------
 1 files changed, 20 insertions(+), 8 deletions(-)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 0ebf123..e7409b4 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -1981,26 +1981,38 @@ static void cma_set_loopback(struct sockaddr *addr)
 
 static int cma_bind_loopback(struct rdma_id_private *id_priv)
 {
-	struct cma_device *cma_dev;
+	struct cma_device *cma_dev, *cur_dev;
 	struct ib_port_attr port_attr;
 	union ib_gid gid;
 	u16 pkey;
 	int ret;
 	u8 p;
 
+	cma_dev = NULL;
 	mutex_lock(&lock);
-	if (list_empty(&dev_list)) {
+	list_for_each_entry(cur_dev, &dev_list, list) {
+		if (cma_family(id_priv) == AF_IB &&
+		    rdma_node_get_transport(cur_dev->device->node_type) != RDMA_TRANSPORT_IB)
+			continue;
+
+		if (!cma_dev)
+			cma_dev = cur_dev;
+
+		for (p = 1; p <= cur_dev->device->phys_port_cnt; ++p) {
+			if (!ib_query_port(cur_dev->device, p, &port_attr) &&
+			    port_attr.state == IB_PORT_ACTIVE) {
+				cma_dev = cur_dev;
+				goto port_found;
+			}
+		}
+	}
+
+	if (!cma_dev) {
 		ret = -ENODEV;
 		goto out;
 	}
-	list_for_each_entry(cma_dev, &dev_list, list)
-		for (p = 1; p <= cma_dev->device->phys_port_cnt; ++p)
-			if (!ib_query_port(cma_dev->device, p, &port_attr) &&
-			    port_attr.state == IB_PORT_ACTIVE)
-				goto port_found;
 
 	p = 1;
-	cma_dev = list_entry(dev_list.next, struct cma_device, list);
 
 port_found:
 	ret = ib_get_cached_gid(cma_dev->device, p, 0, &gid);
-- 
1.7.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v4 9/9] rdma/cm: Verify that source and dest sa_family are the same

[sean.hefty-ral2JQCrhuEAvxtiuMwx3w]

From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 drivers/infiniband/core/cma.c |    8 +++-----
 1 files changed, 3 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index e7409b4..64799ea 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -1849,14 +1849,9 @@ static int cma_resolve_iboe_route(struct rdma_id_private *id_priv)
 	struct rdma_addr *addr = &route->addr;
 	struct cma_work *work;
 	int ret;
-	struct sockaddr_in *src_addr = (struct sockaddr_in *)&route->addr.src_addr;
-	struct sockaddr_in *dst_addr = (struct sockaddr_in *)&route->addr.dst_addr;
 	struct net_device *ndev = NULL;
 	u16 vid;
 
-	if (src_addr->sin_family != dst_addr->sin_family)
-		return -EINVAL;
-
 	work = kzalloc(sizeof *work, GFP_KERNEL);
 	if (!work)
 		return -ENOMEM;
@@ -2132,6 +2127,9 @@ int rdma_resolve_addr(struct rdma_cm_id *id, struct sockaddr *src_addr,
 			return ret;
 	}
 
+	if (cma_family(id_priv) != dst_addr->sa_family)
+		return -EINVAL;
+
 	if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_ADDR_QUERY))
 		return -EINVAL;
 
-- 
1.7.3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v4 1/9] rdma/cm: define native IB address

[Hefty, Sean]

> Define AF_IB and sockaddr_ib to allow the rdma_cm to use native IB
> addressing.
> 
> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
> ---
>  include/linux/socket.h |    2 +
>  include/rdma/ib.h      |   89 ++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 91 insertions(+), 0 deletions(-)
>  create mode 100644 include/rdma/ib.h
> 
> diff --git a/include/linux/socket.h b/include/linux/socket.h
> index 9a546ff..17a33f7 100644
> --- a/include/linux/socket.h
> +++ b/include/linux/socket.h
> @@ -167,6 +167,7 @@ struct ucred {
>  #define AF_PPPOX	24	/* PPPoX sockets		*/
>  #define AF_WANPIPE	25	/* Wanpipe API Sockets */
>  #define AF_LLC		26	/* Linux LLC			*/
> +#define AF_IB		27	/* Native InfiniBand address	*/

...

> diff --git a/include/rdma/ib.h b/include/rdma/ib.h

...

> +struct sockaddr_ib {
> +	unsigned short int	sib_family;	/* AF_IB */
> +	__be16			sib_pkey;
> +	__be32			sib_flowinfo;
> +	struct ib_addr		sib_addr;
> +	__be64			sib_sid;
> +	__be64			sib_sid_mask;
> +	__u64			sib_scope_id;
> +};

Dave/Roland/anyone, is there any feedback on this approach?

If there's hesitation to add new address families to socket.h, I could instead use definitions local to the rdma_cm.  This has the potential to result in conflicts if the rdma_cm is expanded for other address families, though such conflicts seem unlikely.

- Sean

Re: [PATCH v4 4/9] rdma/cm: Update port reservation to support AF_IB

[Or Gerlitz]

On 22/01/2013 23:56, sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org wrote:
> The AF_IB uses a 64-bit service id (SID), which the
> user can control through the use of a mask.  The rdma_cm
> will assign values to the unmasked portions of the SID
> based on the selected port space and port number.

Something here I am not sure to follow -- if AF_IB consumers are allowed 
to provide 64-bit SID, how
the IB port space is expressed in their SIDs?

Wouldn't it make sense to use a simpler approach that has a dedicated 
IB_PS in the port space
portion of the RDMA-CM IP IBTA annex and let the consumer provide 16 
bits for their part of the SID?

Or.


> Because the IB spec divides the SID range into several regions,
> a SID/mask combination may fall into one of the existing
> port space ranges as defined by the RDMA CM IP Annex.  Map
> the AF_IB SID to the correct RDMA port space.

> [...]
> --- a/include/rdma/rdma_cm.h
> +++ b/include/rdma/rdma_cm.h
> @@ -70,6 +70,11 @@ enum rdma_port_space {
>   	RDMA_PS_UDP   = 0x0111,
>   };
>   
> +#define RDMA_IB_IP_PS_MASK   0xFFFFFFFFFFFF0000ULL
> +#define RDMA_IB_IP_PS_TCP    0x0000000001060000ULL
> +#define RDMA_IB_IP_PS_UDP    0x0000000001110000ULL
> +#define RDMA_IB_IP_PS_IB     0x00000000013F0000ULL
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 4/9] rdma/cm: Update port reservation to support AF_IB

[Or Gerlitz]

On 22/01/2013 23:56, sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org wrote:
> The AF_IB uses a 64-bit service id (SID), which the
> user can control through the use of a mask.  The rdma_cm
> will assign values to the unmasked portions of the SID
> based on the selected port space and port number.

Something here I am not sure to follow -- if AF_IB consumers are allowed 
to provide 64-bit SID, how
the IB port space is expressed in their SIDs?

Wouldn't it make sense to use a simpler approach that has a dedicated 
IB_PS in the port space
portion of the RDMA-CM IP IBTA annex and let the consumer provide 16 
bits for their part of the SID?

Or.


> Because the IB spec divides the SID range into several regions,
> a SID/mask combination may fall into one of the existing
> port space ranges as defined by the RDMA CM IP Annex.  Map
> the AF_IB SID to the correct RDMA port space.

> [...]
> --- a/include/rdma/rdma_cm.h
> +++ b/include/rdma/rdma_cm.h
> @@ -70,6 +70,11 @@ enum rdma_port_space {
>   	RDMA_PS_UDP   = 0x0111,
>   };
>   
> +#define RDMA_IB_IP_PS_MASK   0xFFFFFFFFFFFF0000ULL
> +#define RDMA_IB_IP_PS_TCP    0x0000000001060000ULL
> +#define RDMA_IB_IP_PS_UDP    0x0000000001110000ULL
> +#define RDMA_IB_IP_PS_IB     0x00000000013F0000ULL
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 7/9] rdma/cm: Add helper functions to return id address information

[Or Gerlitz]

On 22/01/2013 23:56, sean.hefty@intel.com wrote:
> Provide inline helpers to extract source and destination address data from the rdma_cm_id.

Sean,

FWIW - this patch and most of patch #6 can (and worth doing) be 
introduced as cleanup/maintenance patches to the rdma-cm without direct 
relation to the AF_IB effort.

Or.

Re: [PATCH v4 7/9] rdma/cm: Add helper functions to return id address information

[Or Gerlitz]

On 22/01/2013 23:56, sean.hefty@intel.com wrote:
> Provide inline helpers to extract source and destination address data from the rdma_cm_id.

Sean,

FWIW - this patch and most of patch #6 can (and worth doing) be 
introduced as cleanup/maintenance patches to the rdma-cm without direct 
relation to the AF_IB effort.

Or.

Re: [PATCH v4 1/9] rdma/cm: define native IB address

[Or Gerlitz]

On 11/02/2013 20:02, Hefty, Sean wrote:
>> Define AF_IB and sockaddr_ib to allow the rdma_cm to use native IB addressing.
>>
>> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
>> ---
>>   include/linux/socket.h |    2 +
>>   include/rdma/ib.h      |   89 ++++++++++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 91 insertions(+), 0 deletions(-)
>>   create mode 100644 include/rdma/ib.h
>>
>> diff --git a/include/linux/socket.h b/include/linux/socket.h
>> index 9a546ff..17a33f7 100644
>> --- a/include/linux/socket.h
>> +++ b/include/linux/socket.h
>> @@ -167,6 +167,7 @@ struct ucred {
>>   #define AF_PPPOX	24	/* PPPoX sockets		*/
>>   #define AF_WANPIPE	25	/* Wanpipe API Sockets */
>>   #define AF_LLC		26	/* Linux LLC			*/
>> +#define AF_IB		27	/* Native InfiniBand address	*/
> ...
>
>> diff --git a/include/rdma/ib.h b/include/rdma/ib.h
> ...
>
>> +struct sockaddr_ib {
>> +	unsigned short int	sib_family;	/* AF_IB */
>> +	__be16			sib_pkey;
>> +	__be32			sib_flowinfo;
>> +	struct ib_addr		sib_addr;
>> +	__be64			sib_sid;
>> +	__be64			sib_sid_mask;
>> +	__u64			sib_scope_id;
>> +};
> Dave/Roland/anyone, is there any feedback on this approach?
>
> If there's hesitation to add new address families to socket.h, I could instead use definitions local to the rdma_cm.  This has the potential to result in conflicts if the rdma_cm is expanded for other address families, though such conflicts seem unlikely.
>
>

I don't see why not add new address family if it comes to serve a real 
world use case, which seems to be the case from the description you 
provided in the cover letter.

Or.

Re: [PATCH v4 1/9] rdma/cm: define native IB address

[Or Gerlitz]

On 22/01/2013 23:56, sean.hefty@intel.com wrote:
> +
> +struct sockaddr_ib {
> +	unsigned short int	sib_family;	/* AF_IB */
> +	__be16			sib_pkey;
> +	__be32			sib_flowinfo;
> +	struct ib_addr		sib_addr;
> +	__be64			sib_sid;
> +	__be64			sib_sid_mask;
> +	__u64			sib_scope_id;
> +};

just a nit, maybe reorder the fields to better cope with their IPv6 
buddies (where  there is
such) from sockaddr_in6?

Also I see that both IPv6 header and IB GRH have a traffic class field 
which is skipped in both
cases for the related sockaddr_ structure, not sure why, is this 
something the kernel stack decides on and uses but not available for 
applications to read/modify?

struct sockaddr_in6 {
         unsigned short int      sin6_family;    /* AF_INET6 */
         __be16                  sin6_port;      /* Transport layer port 
# */
         __be32                  sin6_flowinfo;  /* IPv6 flow information */
         struct in6_addr         sin6_addr;      /* IPv6 address */
         __u32                   sin6_scope_id;  /* scope id (new in 
RFC2553) */
};

Re: [PATCH v4 1/9] rdma/cm: define native IB address

[Or Gerlitz]

On 22/01/2013 23:56, sean.hefty@intel.com wrote:
> +
> +struct sockaddr_ib {
> +	unsigned short int	sib_family;	/* AF_IB */
> +	__be16			sib_pkey;
> +	__be32			sib_flowinfo;
> +	struct ib_addr		sib_addr;
> +	__be64			sib_sid;
> +	__be64			sib_sid_mask;
> +	__u64			sib_scope_id;
> +};

just a nit, maybe reorder the fields to better cope with their IPv6 
buddies (where  there is
such) from sockaddr_in6?

Also I see that both IPv6 header and IB GRH have a traffic class field 
which is skipped in both
cases for the related sockaddr_ structure, not sure why, is this 
something the kernel stack decides on and uses but not available for 
applications to read/modify?

struct sockaddr_in6 {
         unsigned short int      sin6_family;    /* AF_INET6 */
         __be16                  sin6_port;      /* Transport layer port 
# */
         __be32                  sin6_flowinfo;  /* IPv6 flow information */
         struct in6_addr         sin6_addr;      /* IPv6 address */
         __u32                   sin6_scope_id;  /* scope id (new in 
RFC2553) */
};

RE: [PATCH v4 4/9] rdma/cm: Update port reservation to support AF_IB

[Hefty, Sean]

> > The AF_IB uses a 64-bit service id (SID), which the
> > user can control through the use of a mask.  The rdma_cm
> > will assign values to the unmasked portions of the SID
> > based on the selected port space and port number.
> 
> Something here I am not sure to follow -- if AF_IB consumers are allowed
> to provide 64-bit SID, how
> the IB port space is expressed in their SIDs?

The spec, notably the RDMA CM Annex, provides fixed values for the upper 48-bits of the service ID.  Only the lower 16-bits are available to users.  The upper bits end up identifying the various 'port spaces'. 

(Note that there's no actual association between an IB port space and port spaces in the IP network stack.)

> Wouldn't it make sense to use a simpler approach that has a dedicated
> IB_PS in the port space
> portion of the RDMA-CM IP IBTA annex and let the consumer provide 16
> bits for their part of the SID?

Ultimately, IB establishes communication using 64-bit service IDs, which is what sockaddr_ib exposes.

The SID mask was proposed by Jason as a way to simplify things.  Conceptually, AF_IB has a single port space range, the 64-bit SID.  The mask helps us break the SID into the various ranges, so we can clearly know which parts of the SID are fixed, and which can be variable.  E.g. when connecting, the entire SID must be provided.  However, when binding we want to support binding to a specific SID or only to a SID range (i.e. port space).

(I'm having trouble finding the original email threads where this was discussed.)

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v4 4/9] rdma/cm: Update port reservation to support AF_IB

[Jason Gunthorpe]

On Wed, Feb 13, 2013 at 05:44:58PM +0000, Hefty, Sean wrote:

> Ultimately, IB establishes communication using 64-bit service IDs, which is what sockaddr_ib exposes.
> 
> The SID mask was proposed by Jason as a way to simplify things.
> Conceptually, AF_IB has a single port space range, the 64-bit SID.
> The mask helps us break the SID into the various ranges, so we can
> clearly know which parts of the SID are fixed, and which can be
> variable.  E.g. when connecting, the entire SID must be provided.
> However, when binding we want to support binding to a specific SID
> or only to a SID range (i.e. port space).

Right. AF_IB is about exposing the native IB CM, rather than a cooked
version of it, so allowing user space to control the entire SID is
desirable.

The intent of the masking was to provide a means for the kernel to
atomically select a process-unique ID (such as the 16 bits the IP
varient requires) for listening.

Other uses of IB CM would use a different prefix, but will still
fundamentally require a machine unique portion.

This addresses a fundamental problem in ib_ucm - there is no way for
multiple processes to co-ordinate on unique SIDs.

Sean, I've lost track of all this over time, but just to check in,
with this new interface, would it be possible to obsolete ib_ucm? That
seems desirable to me, as it is so flawed..

Jason

RE: [PATCH v4 4/9] rdma/cm: Update port reservation to support AF_IB

[Hefty, Sean]

> Sean, I've lost track of all this over time, but just to check in,
> with this new interface, would it be possible to obsolete ib_ucm? That
> seems desirable to me, as it is so flawed..

I'd love to obsolete ib_ucm.  This interface should make it possible, though some additional implementation would be needed (e.g. for automatic path migration).

I have an immediate need to connect using native IB addresses.  So without this, I'll be forced to use ib_ucm or roll a new user space CM.

Re: [PATCH v4 4/9] rdma/cm: Update port reservation to support AF_IB

[Jason Gunthorpe]

On Wed, Feb 13, 2013 at 06:19:51PM +0000, Hefty, Sean wrote:
> > Sean, I've lost track of all this over time, but just to check in,
> > with this new interface, would it be possible to obsolete ib_ucm? That
> > seems desirable to me, as it is so flawed..
> 
> I'd love to obsolete ib_ucm.  This interface should make it
> possible, though some additional implementation would be needed
> (e.g. for automatic path migration).

Okay - but there are some nice gaps in the API to cleanly place such
things in future?

A long time ago we discussed having different paths for CM GMP,
inbound data and outbound data (ie the full IB asymetric pathing
model), did that get included in the API as well?

> I have an immediate need to connect using native IB addresses.  So
> without this, I'll be forced to use ib_ucm or roll a new user space
> CM.

Using ucm is alot of trouble. Many sites don't have it setup properly,
don't have proper device node permissions and the issue with unique
IDs ends up being critical..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH v4 4/9] rdma/cm: Update port reservation to support AF_IB

[Hefty, Sean]

> A long time ago we discussed having different paths for CM GMP,
> inbound data and outbound data (ie the full IB asymetric pathing
> model), did that get included in the API as well?

Yes - this support made it upstream.

RE: [PATCH v4 1/9] rdma/cm: define native IB address

[Hefty, Sean]

> > Define AF_IB and sockaddr_ib to allow the rdma_cm to use native IB
> > addressing.
> >
> > Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > ---
> >  include/linux/socket.h |    2 +
> >  include/rdma/ib.h      |   89
> ++++++++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 91 insertions(+), 0 deletions(-)
> >  create mode 100644 include/rdma/ib.h
> >
> > diff --git a/include/linux/socket.h b/include/linux/socket.h
> > index 9a546ff..17a33f7 100644
> > --- a/include/linux/socket.h
> > +++ b/include/linux/socket.h
> > @@ -167,6 +167,7 @@ struct ucred {
> >  #define AF_PPPOX	24	/* PPPoX sockets		*/
> >  #define AF_WANPIPE	25	/* Wanpipe API Sockets */
> >  #define AF_LLC		26	/* Linux LLC			*/
> > +#define AF_IB		27	/* Native InfiniBand address	*/
> 
> ...
> 
> > diff --git a/include/rdma/ib.h b/include/rdma/ib.h
> 
> ...
> 
> > +struct sockaddr_ib {
> > +	unsigned short int	sib_family;	/* AF_IB */
> > +	__be16			sib_pkey;
> > +	__be32			sib_flowinfo;
> > +	struct ib_addr		sib_addr;
> > +	__be64			sib_sid;
> > +	__be64			sib_sid_mask;
> > +	__u64			sib_scope_id;
> > +};
> 
> Dave/Roland/anyone, is there any feedback on this approach?

ping...

Seriously, there is a need to establish connections using native IB GIDs.  It is preferable to add this functionality to the rdma_cm.  Although the ib_cm can exchange IB CM messages based on GIDs, the rdma_cm is the only interface that provides applications with dynamic service IDs, path record queries, and usable multicast support. 

The rdma_cm uses sockaddr for addressing information.  The sockaddr_ib structure is defined within the RDMA tree, but I believe that the address family value belongs in socket.h.  This helps to avoid any future conflicts.  However, I can change the rdma_cm to use an internal enum for address values if there is an objection.

sockaddr_ib/AF_IB is usable with any IB network card.  It allows an application to establish connections over IB without IBoIP being present, which is a real need.  It also allows an application to use out of band mechanisms for discovering remote GIDs.  Such mechanisms are needed to work through IP load balancing software and for MPI scalability.

I'm not trying to be impatient, but it's been 3 years... 

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Qperf and APM

[Suresh Shelvapille]

Folks:
The qperf utility has -ap, -lap, -rap options in support of APM. Unfortunately they don't seem to work. Is there a way
to make this work, or if you have any other ideas on testing failover using rdma please let me know. I have successfully
used tcp_bw tests for this purpose already.

Thanks,
Suri

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Qperf and APM

[Luick, Dean]

Hi,

Can you be more specific?  What are you running and how is it failing?

Thanks,
Dean

> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Suresh Shelvapille
> Sent: Wednesday, March 06, 2013 9:40 AM
> To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Cc: Hefty, Sean
> Subject: Qperf and APM
> 
> Folks:
> The qperf utility has -ap, -lap, -rap options in support of APM. Unfortunately
> they don't seem to work. Is there a way
> to make this work, or if you have any other ideas on testing failover using
> rdma please let me know. I have successfully
> used tcp_bw tests for this purpose already.
> 
> Thanks,
> Suri
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Qperf and APM

[Suresh Shelvapille]

This is the error I see on the client(the same warning is seen with -lap and -rap options):

------------------------------
qperf <server-ip-addr> -ap 2 rc_bw
rc_bw:
warning: -ap set but not used in test rc_bw
    bw  =  3.39 GB/sec

> -----Original Message-----
> From: Luick, Dean [mailto:dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org]
> Sent: Wednesday, March 06, 2013 1:52 PM
> To: Suresh Shelvapille; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Cc: Hefty, Sean
> Subject: RE: Qperf and APM
> 
> Hi,
> 
> Can you be more specific?  What are you running and how is it failing?
> 
> Thanks,
> Dean
> 
> > -----Original Message-----
> > From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Suresh Shelvapille
> > Sent: Wednesday, March 06, 2013 9:40 AM
> > To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Cc: Hefty, Sean
> > Subject: Qperf and APM
> >
> > Folks:
> > The qperf utility has -ap, -lap, -rap options in support of APM. Unfortunately
> > they don't seem to work. Is there a way
> > to make this work, or if you have any other ideas on testing failover using
> > rdma please let me know. I have successfully
> > used tcp_bw tests for this purpose already.
> >
> > Thanks,
> > Suri
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Qperf and APM

[Luick, Dean]

Have you tried ignoring the warning and checking of the alternate port works?  The qperf code looks to be using the alternate port, but incorrectly fails to mark the argument as used.

Dean

> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Suresh Shelvapille
> Sent: Wednesday, March 06, 2013 1:00 PM
> To: Luick, Dean; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: RE: Qperf and APM
> 
> This is the error I see on the client(the same warning is seen with -lap and -
> rap options):
> 
> ------------------------------
> qperf <server-ip-addr> -ap 2 rc_bw
> rc_bw:
> warning: -ap set but not used in test rc_bw
>     bw  =  3.39 GB/sec
> 
> > -----Original Message-----
> > From: Luick, Dean [mailto:dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org]
> > Sent: Wednesday, March 06, 2013 1:52 PM
> > To: Suresh Shelvapille; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Cc: Hefty, Sean
> > Subject: RE: Qperf and APM
> >
> > Hi,
> >
> > Can you be more specific?  What are you running and how is it failing?
> >
> > Thanks,
> > Dean
> >
> > > -----Original Message-----
> > > From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> > > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Suresh Shelvapille
> > > Sent: Wednesday, March 06, 2013 9:40 AM
> > > To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > Cc: Hefty, Sean
> > > Subject: Qperf and APM
> > >
> > > Folks:
> > > The qperf utility has -ap, -lap, -rap options in support of APM.
> Unfortunately
> > > they don't seem to work. Is there a way
> > > to make this work, or if you have any other ideas on testing failover using
> > > rdma please let me know. I have successfully
> > > used tcp_bw tests for this purpose already.
> > >
> > > Thanks,
> > > Suri
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Qperf and APM

[Suresh Shelvapille]

I can see the packets going on the alternate path when the first link fails, but at the end of the test, the result is a
failure, so I am not sure!
I tried this with both '-ap' and "-lap -rap" options. I am assuming they are equivalent.

------------------------------
[root]# qperf <server-ipaddr> -ap 2 -t 30 rc_bw
rc_bw:
warning: -ap set but not used in test rc_bw
failed to receive results: timed out



-------------------------
[root]# qperf <server-ipaddr> -lap 2 -rap 2 -t 30 rc_bw
rc_bw:
warning: -lap set but not used in test rc_bw
warning: -rap set but not used in test rc_bw
failed to receive results: timed out

thanks,
Suri

> -----Original Message-----
> From: Luick, Dean [mailto:dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org]
> Sent: Wednesday, March 06, 2013 3:57 PM
> To: Suresh Shelvapille; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: RE: Qperf and APM
> 
> Have you tried ignoring the warning and checking of the alternate port works?  The qperf code looks
> to be using the alternate port, but incorrectly fails to mark the argument as used.
> 
> Dean
> 
> > -----Original Message-----
> > From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Suresh Shelvapille
> > Sent: Wednesday, March 06, 2013 1:00 PM
> > To: Luick, Dean; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Subject: RE: Qperf and APM
> >
> > This is the error I see on the client(the same warning is seen with -lap and -
> > rap options):
> >
> > ------------------------------
> > qperf <server-ip-addr> -ap 2 rc_bw
> > rc_bw:
> > warning: -ap set but not used in test rc_bw
> >     bw  =  3.39 GB/sec
> >
> > > -----Original Message-----
> > > From: Luick, Dean [mailto:dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org]
> > > Sent: Wednesday, March 06, 2013 1:52 PM
> > > To: Suresh Shelvapille; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > Cc: Hefty, Sean
> > > Subject: RE: Qperf and APM
> > >
> > > Hi,
> > >
> > > Can you be more specific?  What are you running and how is it failing?
> > >
> > > Thanks,
> > > Dean
> > >
> > > > -----Original Message-----
> > > > From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> > > > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Suresh Shelvapille
> > > > Sent: Wednesday, March 06, 2013 9:40 AM
> > > > To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > Cc: Hefty, Sean
> > > > Subject: Qperf and APM
> > > >
> > > > Folks:
> > > > The qperf utility has -ap, -lap, -rap options in support of APM.
> > Unfortunately
> > > > they don't seem to work. Is there a way
> > > > to make this work, or if you have any other ideas on testing failover using
> > > > rdma please let me know. I have successfully
> > > > used tcp_bw tests for this purpose already.
> > > >
> > > > Thanks,
> > > > Suri
> > > >
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Qperf and APM

[Luick, Dean]

Using QLogic dual port HCAs, this works for me when I disable port 1 on the client, server, or both while the test is running:

# qperf <server ipaddr> -ap 2 -t 30 rc_bw
rc_bw:
warning: -ap set but not used in test rc_bw
    bw  =  272 MB/sec

The warning can be ignored.  Using -ap is equivalent to using -rap and -lap with the same port.

The "failed to receive results" is qperf saying that once the test finished, the server did not contact the client.


Dean

> -----Original Message-----
> From: Suresh Shelvapille [mailto:suri-lFb5ksp6isWhEniVeURVKkEOCMrvLtNR@public.gmane.org]
> Sent: Wednesday, March 06, 2013 4:24 PM
> To: Luick, Dean; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: RE: Qperf and APM
> 
> I can see the packets going on the alternate path when the first link fails, but
> at the end of the test, the result is a
> failure, so I am not sure!
> I tried this with both '-ap' and "-lap -rap" options. I am assuming they are
> equivalent.
> 
> ------------------------------
> [root]# qperf <server-ipaddr> -ap 2 -t 30 rc_bw
> rc_bw:
> warning: -ap set but not used in test rc_bw
> failed to receive results: timed out
> 
> 
> 
> -------------------------
> [root]# qperf <server-ipaddr> -lap 2 -rap 2 -t 30 rc_bw
> rc_bw:
> warning: -lap set but not used in test rc_bw
> warning: -rap set but not used in test rc_bw
> failed to receive results: timed out
> 
> thanks,
> Suri
> 
> > -----Original Message-----
> > From: Luick, Dean [mailto:dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org]
> > Sent: Wednesday, March 06, 2013 3:57 PM
> > To: Suresh Shelvapille; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Subject: RE: Qperf and APM
> >
> > Have you tried ignoring the warning and checking of the alternate port
> works?  The qperf code looks
> > to be using the alternate port, but incorrectly fails to mark the argument as
> used.
> >
> > Dean
> >
> > > -----Original Message-----
> > > From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> > > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Suresh Shelvapille
> > > Sent: Wednesday, March 06, 2013 1:00 PM
> > > To: Luick, Dean; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > Subject: RE: Qperf and APM
> > >
> > > This is the error I see on the client(the same warning is seen with -lap and
> -
> > > rap options):
> > >
> > > ------------------------------
> > > qperf <server-ip-addr> -ap 2 rc_bw
> > > rc_bw:
> > > warning: -ap set but not used in test rc_bw
> > >     bw  =  3.39 GB/sec
> > >
> > > > -----Original Message-----
> > > > From: Luick, Dean [mailto:dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org]
> > > > Sent: Wednesday, March 06, 2013 1:52 PM
> > > > To: Suresh Shelvapille; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > Cc: Hefty, Sean
> > > > Subject: RE: Qperf and APM
> > > >
> > > > Hi,
> > > >
> > > > Can you be more specific?  What are you running and how is it failing?
> > > >
> > > > Thanks,
> > > > Dean
> > > >
> > > > > -----Original Message-----
> > > > > From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> > > > > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Suresh Shelvapille
> > > > > Sent: Wednesday, March 06, 2013 9:40 AM
> > > > > To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > > Cc: Hefty, Sean
> > > > > Subject: Qperf and APM
> > > > >
> > > > > Folks:
> > > > > The qperf utility has -ap, -lap, -rap options in support of APM.
> > > Unfortunately
> > > > > they don't seem to work. Is there a way
> > > > > to make this work, or if you have any other ideas on testing failover
> using
> > > > > rdma please let me know. I have successfully
> > > > > used tcp_bw tests for this purpose already.
> > > > >
> > > > > Thanks,
> > > > > Suri
> > > > >
> > > > > --
> > > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> in
> > > > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > > More majordomo info at  http://vger.kernel.org/majordomo-
> info.html
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Qperf and APM

[Suresh Shelvapille]

Dean:

The difference is, you got a BW number where as I don't! So, I don't know whether the test was successful or not. Do you
have a Mellanox dual port card?

Mellanox folks: have you guys tried this on your dual port cards?

Thanks,
Suri

> -----Original Message-----
> From: Luick, Dean [mailto:dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org]
> Sent: Wednesday, March 13, 2013 6:25 PM
> To: Suresh Shelvapille; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: RE: Qperf and APM
> 
> Using QLogic dual port HCAs, this works for me when I disable port 1 on the client, server, or both
> while the test is running:
> 
> # qperf <server ipaddr> -ap 2 -t 30 rc_bw
> rc_bw:
> warning: -ap set but not used in test rc_bw
>     bw  =  272 MB/sec
> 
> The warning can be ignored.  Using -ap is equivalent to using -rap and -lap with the same port.
> 
> The "failed to receive results" is qperf saying that once the test finished, the server did not
> contact the client.
> 
> 
> Dean
> 
> > -----Original Message-----
> > From: Suresh Shelvapille [mailto:suri-lFb5ksp6isWhEniVeURVKkEOCMrvLtNR@public.gmane.org]
> > Sent: Wednesday, March 06, 2013 4:24 PM
> > To: Luick, Dean; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Subject: RE: Qperf and APM
> >
> > I can see the packets going on the alternate path when the first link fails, but
> > at the end of the test, the result is a
> > failure, so I am not sure!
> > I tried this with both '-ap' and "-lap -rap" options. I am assuming they are
> > equivalent.
> >
> > ------------------------------
> > [root]# qperf <server-ipaddr> -ap 2 -t 30 rc_bw
> > rc_bw:
> > warning: -ap set but not used in test rc_bw
> > failed to receive results: timed out
> >
> >
> >
> > -------------------------
> > [root]# qperf <server-ipaddr> -lap 2 -rap 2 -t 30 rc_bw
> > rc_bw:
> > warning: -lap set but not used in test rc_bw
> > warning: -rap set but not used in test rc_bw
> > failed to receive results: timed out
> >
> > thanks,
> > Suri
> >
> > > -----Original Message-----
> > > From: Luick, Dean [mailto:dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org]
> > > Sent: Wednesday, March 06, 2013 3:57 PM
> > > To: Suresh Shelvapille; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > Subject: RE: Qperf and APM
> > >
> > > Have you tried ignoring the warning and checking of the alternate port
> > works?  The qperf code looks
> > > to be using the alternate port, but incorrectly fails to mark the argument as
> > used.
> > >
> > > Dean
> > >
> > > > -----Original Message-----
> > > > From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> > > > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Suresh Shelvapille
> > > > Sent: Wednesday, March 06, 2013 1:00 PM
> > > > To: Luick, Dean; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > Subject: RE: Qperf and APM
> > > >
> > > > This is the error I see on the client(the same warning is seen with -lap and
> > -
> > > > rap options):
> > > >
> > > > ------------------------------
> > > > qperf <server-ip-addr> -ap 2 rc_bw
> > > > rc_bw:
> > > > warning: -ap set but not used in test rc_bw
> > > >     bw  =  3.39 GB/sec
> > > >
> > > > > -----Original Message-----
> > > > > From: Luick, Dean [mailto:dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org]
> > > > > Sent: Wednesday, March 06, 2013 1:52 PM
> > > > > To: Suresh Shelvapille; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > > Cc: Hefty, Sean
> > > > > Subject: RE: Qperf and APM
> > > > >
> > > > > Hi,
> > > > >
> > > > > Can you be more specific?  What are you running and how is it failing?
> > > > >
> > > > > Thanks,
> > > > > Dean
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-
> > > > > > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Suresh Shelvapille
> > > > > > Sent: Wednesday, March 06, 2013 9:40 AM
> > > > > > To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > > > Cc: Hefty, Sean
> > > > > > Subject: Qperf and APM
> > > > > >
> > > > > > Folks:
> > > > > > The qperf utility has -ap, -lap, -rap options in support of APM.
> > > > Unfortunately
> > > > > > they don't seem to work. Is there a way
> > > > > > to make this work, or if you have any other ideas on testing failover
> > using
> > > > > > rdma please let me know. I have successfully
> > > > > > used tcp_bw tests for this purpose already.
> > > > > >
> > > > > > Thanks,
> > > > > > Suri
> > > > > >
> > > > > > --
> > > > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> > in
> > > > > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > > > More majordomo info at  http://vger.kernel.org/majordomo-
> > info.html
> > > >
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html