[PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Leon Romanovsky]

Hi Doug,

This patchset mostly comes from Erez with one exception in the first patch.

That patch origins from two different commits, first from Niranjana who added
RDMA netdev interface and second from Erez who added IPoIB support.

During the preparation to submission, I squashed their commits into one
and refactored code to allow submission as a standalone topic without creating
dependecies between different submissions. This caused to change in author line.
I hope that it doesn't really matter and it won't stop you from merging it.

Per-your request, I based this patch set on v4.11-rc3.

Thanks,
	Leon

CC: Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
CC: Alex Vesker <valex-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
---

The rest comes from Erez:

    The IPoIB protocol encapsulates IP packets over Infiniband datagrams.
    As a direct RDMA Upper Layer Protocol (ULP), IPoIB cannot support HW
    features that are specific to the IP protocol stack.

    Nevertheless, RDMA interfaces have been extended to support some of the
    prominent IP offload features, such as TCP/UDP checksum and TSO.
    This provided reasonable performance gain for IPoIB but is still
    insufficient to cope with the increasing network bandwidth demand.

    However, New features are exisiting in common network interfaces that
    are very hard to implement in IPoIB interfaces while it uses the RDMA
    layer, examples include TSS and RSS, tunneling offloads, and XDP.
    Rather than continuously porting IP network interface developments into
    the RDMA stack, we propose adding an abstract network data-path
    interfaces to RDMA devices.

    In order to present a consistent interface to users, the IPoIB ULP
    continues to represent the network device to the IP stack.
    The common code also manages the IPoIB control plane, such as resolving
    path queries and registering to multicast groups.
    Data path operations are forwarded to devices that implement the new
    API, or fallback to the standard implementation otherwise.
    Using the forgoing approach, we show how IPoIB closes the performance
    gap compared to state-of-the-art Ethernet network interfaces.

    The implementation idea is to use the api of
    alloc_rdma_netdev/free_rdma_netdev to expose a struct that has data
    members and set of functions that are used for IB network interfaces,
    like attach/detach multicast to qp, and send IB packet.

    The functions are specific for IB operations and are not part of the
    common api the the netdev struct exposes via the ndo functions.
    1. multicast handling - attach/detach
    2. send operation - the ndo start_xmit has only 2 parameters and the IB
    send needs the destination qp and the ah object, there were few options
    to handle it via the netdev ndo, but they don't make more sense than
    using a specific send function (we are rdma_netdev after all)

    The IPoIB code will be adapted to enable the option of accelerating the
    network interface, but the code will work as before if the HW below
    doesn't support the acceleration.
    That means that in the default mode of ipoib I tried to keep it as much
    as it was before, not to force it to adopt the new api, where there is
    no code sharing between the ipoib and the vnic/hfi.
    The default code uses the controll and the data, the accelerator uses
    only the control flows andstructors.
    The changes of the default ipoib can be made on top of that series.

    Each HW vendor can supply the acceleration for the IPoIB or to leave
    IPoIB to work as before.

Thanks,
	Erez

Erez Shitrit (5):
  IB/IPoIB: Separate control and data related initializations
  IB/IPoIB: Separate control from HW operation on ipoib_open/stop ndo
  IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions
  IB/IPoIB: Formatting before adding accelerator to driver
  IB/IPoIB: Support acceleration options callbacks

Leon Romanovsky (1):
  IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs

 drivers/infiniband/ulp/ipoib/ipoib.h           |  40 +--
 drivers/infiniband/ulp/ipoib/ipoib_cm.c        |  66 ++---
 drivers/infiniband/ulp/ipoib/ipoib_ethtool.c   |   6 +-
 drivers/infiniband/ulp/ipoib/ipoib_fs.c        |   4 +-
 drivers/infiniband/ulp/ipoib/ipoib_ib.c        | 339 +++++++++++++------------
 drivers/infiniband/ulp/ipoib/ipoib_main.c      | 312 +++++++++++++++++------
 drivers/infiniband/ulp/ipoib/ipoib_multicast.c |  39 +--
 drivers/infiniband/ulp/ipoib/ipoib_netlink.c   |  12 +-
 drivers/infiniband/ulp/ipoib/ipoib_verbs.c     |  64 ++---
 drivers/infiniband/ulp/ipoib/ipoib_vlan.c      |   9 +-
 include/rdma/ib_verbs.h                        |  41 +++
 11 files changed, 565 insertions(+), 367 deletions(-)

--
2.12.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 1/6] IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs

[Leon Romanovsky]

Add RDMA netdev interface to ib device structure allowing RDMA netdev
devices to be allocated by ib clients.

The idea is to allow to providers to optimize IPoIB data path.
New struct that includes functions and data member is exposed.
It exposes set of callback functions for handling data path flows
in IPoIB driver.

Each provider can support these set of functions in order to optimize its
specific data path, and let IPoIB to leverage its data path.

There is an assumption, that providers should give the full set of functions
and not only part of them, in order to work properly.

Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 include/rdma/ib_verbs.h | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 0f1813c13687..d3b104a96b58 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -55,6 +55,7 @@
 #include <net/ip.h>
 #include <linux/string.h>
 #include <linux/slab.h>
+#include <linux/netdevice.h>
 
 #include <linux/if_link.h>
 #include <linux/atomic.h>
@@ -1862,7 +1863,38 @@ struct ib_port_immutable {
 	u32                           max_mad_size;
 };
 
+/* rdma netdev type - specifies protocol type */
+enum rdma_netdev_t {
+	RDMA_NETDEV_IPOIB,
+};
+
+/*
+ * struct rdma_netdev - rdma netdev
+ * For cases where netstack interfacing is required.
+ */
+struct rdma_netdev {
+	void *clnt_priv;
+
+	/* control functions */
+	void (*set_id)(struct net_device *netdev, int id);
+	/* send packet */
+	int (*send)(struct net_device *dev, struct sk_buff *skb,
+		    struct ib_ah *address, u32 dqpn);
+	/* multicast */
+	int (*attach_mcast)(struct net_device *dev, struct ib_device *hca,
+			    union ib_gid *gid, u16 mlid,
+			    int set_qkey, u32 qkey);
+	int (*detach_mcast)(struct net_device *dev, struct ib_device *hca,
+			    union ib_gid *gid, u16 mlid);
+
+	/* rdma-netdev private members */
+	struct ib_device *hca;
+};
+
 struct ib_device {
+	/* Do not access @dma_device directly from ULP nor from HW drivers. */
+	struct device                *dma_device;
+
 	char                          name[IB_DEVICE_NAME_MAX];
 
 	struct list_head              event_handler_list;
@@ -2112,6 +2144,15 @@ struct ib_device {
 							   struct ib_rwq_ind_table_init_attr *init_attr,
 							   struct ib_udata *udata);
 	int                        (*destroy_rwq_ind_table)(struct ib_rwq_ind_table *wq_ind_table);
+	/* rdma netdev operations */
+	struct net_device *(*alloc_rdma_netdev)(
+					struct ib_device *device,
+					u8 port_num,
+					enum rdma_netdev_t type,
+					const char *name,
+					unsigned char name_assign_type,
+					void (*setup)(struct net_device *));
+	void (*free_rdma_netdev)(struct net_device *netdev);
 
 	struct module               *owner;
 	struct device                dev;
-- 
2.12.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 2/6] IB/IPoIB: Separate control and data related initializations

[Leon Romanovsky]

From: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

This patch prepares init and teardown flows so we can call them
through ipoib_options function pointers.

It arranges that area of code as the following:
 * All operations which deal with the resource allocation/deletion
   are performed in one place.
 * All operations that are control oriented, meaning that they are not
   connected to a specific hardware, are performed in a separate place.

The operations for allocation of hardware resources are now in the
function ipoib_dev_init_default, and the deletion of all the resources
are in ipoib_dev_uninit_default

The only exception is the creation of the PD object, which is used both
for resource allocation (create QP etc.) and for control flows like creating AH.

It also does:
 * Move creation of rx_ring and tx_ring to be in the resources allocation area.
 * Move the function ipoib_ib_dev_open that does the open device to the control
   area instead of the dev_init which creates resources.

Signed-off-by: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/ulp/ipoib/ipoib.h       |   5 +-
 drivers/infiniband/ulp/ipoib/ipoib_ib.c    |  38 +++-------
 drivers/infiniband/ulp/ipoib/ipoib_main.c  | 116 ++++++++++++++++++++++++-----
 drivers/infiniband/ulp/ipoib/ipoib_verbs.c |  40 +---------
 4 files changed, 109 insertions(+), 90 deletions(-)

diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h
index bed233bf45c3..7cd9befd7d54 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib.h
+++ b/drivers/infiniband/ulp/ipoib/ipoib.h
@@ -491,14 +491,13 @@ void ipoib_mark_paths_invalid(struct net_device *dev);
 void ipoib_flush_paths(struct net_device *dev);
 int ipoib_check_sm_sendonly_fullmember_support(struct ipoib_dev_priv *priv);
 struct ipoib_dev_priv *ipoib_intf_alloc(const char *format);
-
-int ipoib_ib_dev_init(struct net_device *dev, struct ib_device *ca, int port);
+void ipoib_ib_tx_timer_func(unsigned long ctx);
 void ipoib_ib_dev_flush_light(struct work_struct *work);
 void ipoib_ib_dev_flush_normal(struct work_struct *work);
 void ipoib_ib_dev_flush_heavy(struct work_struct *work);
 void ipoib_pkey_event(struct work_struct *work);
 void ipoib_ib_dev_cleanup(struct net_device *dev);
-
+void ipoib_dev_uninit_default(struct net_device *dev);
 int ipoib_ib_dev_open(struct net_device *dev);
 void ipoib_ib_dev_up(struct net_device *dev);
 void ipoib_ib_dev_down(struct net_device *dev);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index 12c4f84a6639..f9dd2bb676aa 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -692,7 +692,7 @@ static void ipoib_stop_ah(struct net_device *dev)
 	ipoib_flush_ah(dev);
 }
 
-static void ipoib_ib_tx_timer_func(unsigned long ctx)
+void ipoib_ib_tx_timer_func(unsigned long ctx)
 {
 	drain_tx_cq((struct net_device *)ctx);
 }
@@ -913,32 +913,6 @@ void ipoib_ib_dev_stop(struct net_device *dev)
 	ib_req_notify_cq(priv->recv_cq, IB_CQ_NEXT_COMP);
 }
 
-int ipoib_ib_dev_init(struct net_device *dev, struct ib_device *ca, int port)
-{
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
-
-	priv->ca = ca;
-	priv->port = port;
-	priv->qp = NULL;
-
-	if (ipoib_transport_dev_init(dev, ca)) {
-		printk(KERN_WARNING "%s: ipoib_transport_dev_init failed\n", ca->name);
-		return -ENODEV;
-	}
-
-	setup_timer(&priv->poll_timer, ipoib_ib_tx_timer_func,
-		    (unsigned long) dev);
-
-	if (dev->flags & IFF_UP) {
-		if (ipoib_ib_dev_open(dev)) {
-			ipoib_transport_dev_cleanup(dev);
-			return -ENODEV;
-		}
-	}
-
-	return 0;
-}
-
 /*
  * Takes whatever value which is in pkey index 0 and updates priv->pkey
  * returns 0 if the pkey value was changed.
@@ -1236,7 +1210,13 @@ void ipoib_ib_dev_cleanup(struct net_device *dev)
 	 */
 	ipoib_stop_ah(dev);
 
-	ipoib_transport_dev_cleanup(dev);
-}
+	clear_bit(IPOIB_PKEY_ASSIGNED, &priv->flags);
+
+	ipoib_dev_uninit_default(dev);
 
+	if (priv->pd) {
+		ib_dealloc_pd(priv->pd);
+		priv->pd = NULL;
+	}
+}
 
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c
index d1d3fb7a6127..ef8e51ccb1cd 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -1622,8 +1622,23 @@ static void ipoib_neigh_hash_uninit(struct net_device *dev)
 	wait_for_completion(&priv->ntbl.deleted);
 }
 
+void ipoib_dev_uninit_default(struct net_device *dev)
+{
+	struct ipoib_dev_priv *priv = netdev_priv(dev);
 
-int ipoib_dev_init(struct net_device *dev, struct ib_device *ca, int port)
+	ipoib_transport_dev_cleanup(dev);
+
+	ipoib_cm_dev_cleanup(dev);
+
+	kfree(priv->rx_ring);
+	vfree(priv->tx_ring);
+
+	priv->rx_ring = NULL;
+	priv->tx_ring = NULL;
+}
+
+static int ipoib_dev_init_default(struct net_device *dev, struct ib_device *ca,
+				  int port)
 {
 	struct ipoib_dev_priv *priv = netdev_priv(dev);
 
@@ -1642,29 +1657,95 @@ int ipoib_dev_init(struct net_device *dev, struct ib_device *ca, int port)
 
 	/* priv->tx_head, tx_tail & tx_outstanding are already 0 */
 
-	if (ipoib_ib_dev_init(dev, ca, port))
+	if (ipoib_transport_dev_init(dev, ca)) {
+		pr_warn("%s: ipoib_transport_dev_init failed\n", ca->name);
 		goto out_tx_ring_cleanup;
+	}
+
+	setup_timer(&priv->poll_timer, ipoib_ib_tx_timer_func,
+		    (unsigned long)dev);
+
+	return 0;
+
+out_tx_ring_cleanup:
+	vfree(priv->tx_ring);
+
+out_rx_ring_cleanup:
+	kfree(priv->rx_ring);
+
+out:
+	return -ENOMEM;
+}
+
+int ipoib_dev_init(struct net_device *dev, struct ib_device *ca, int port)
+{
+	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	int ret = -ENOMEM;
+
+	priv->ca = ca;
+	priv->port = port;
+	priv->qp = NULL;
 
 	/*
-	 * Must be after ipoib_ib_dev_init so we can allocate a per
-	 * device wq there and use it here
+	 * the various IPoIB tasks assume they will never race against
+	 * themselves, so always use a single thread workqueue
 	 */
-	if (ipoib_neigh_hash_init(priv) < 0)
+	priv->wq = alloc_ordered_workqueue("ipoib_wq", WQ_MEM_RECLAIM);
+	if (!priv->wq) {
+		pr_warn("%s: failed to allocate device WQ\n", dev->name);
+		goto out;
+	}
+
+	/* create pd, which used both for control and datapath*/
+	priv->pd = ib_alloc_pd(priv->ca, 0);
+	if (IS_ERR(priv->pd)) {
+		pr_warn("%s: failed to allocate PD\n", ca->name);
+		goto clean_wq;
+	}
+
+	ret = ipoib_dev_init_default(dev, ca, port);
+	if (ret) {
+		pr_warn("%s failed to init HW resource\n", dev->name);
+		goto out_free_pd;
+	}
+
+	/* after qp created set dev address */
+	priv->dev->dev_addr[1] = (priv->qp->qp_num >> 16) & 0xff;
+	priv->dev->dev_addr[2] = (priv->qp->qp_num >>  8) & 0xff;
+	priv->dev->dev_addr[3] = (priv->qp->qp_num) & 0xff;
+
+	if (ipoib_neigh_hash_init(priv) < 0) {
+		pr_warn("%s failed to init neigh hash\n", dev->name);
 		goto out_dev_uninit;
+	}
+
+	if (dev->flags & IFF_UP) {
+		if (ipoib_ib_dev_open(dev)) {
+			pr_warn("%s failed to open device\n", dev->name);
+			ret = -ENODEV;
+			goto out_dev_uninit;
+		}
+	}
 
 	return 0;
 
 out_dev_uninit:
 	ipoib_ib_dev_cleanup(dev);
 
-out_tx_ring_cleanup:
-	vfree(priv->tx_ring);
+out_free_pd:
+	if (priv->pd) {
+		ib_dealloc_pd(priv->pd);
+		priv->pd = NULL;
+	}
 
-out_rx_ring_cleanup:
-	kfree(priv->rx_ring);
+clean_wq:
+	if (priv->wq) {
+		destroy_workqueue(priv->wq);
+		priv->wq = NULL;
+	}
 
 out:
-	return -ENOMEM;
+	return ret;
 }
 
 void ipoib_dev_cleanup(struct net_device *dev)
@@ -1685,19 +1766,16 @@ void ipoib_dev_cleanup(struct net_device *dev)
 	}
 	unregister_netdevice_many(&head);
 
-	/*
-	 * Must be before ipoib_ib_dev_cleanup or we delete an in use
-	 * work queue
-	 */
 	ipoib_neigh_hash_uninit(dev);
 
 	ipoib_ib_dev_cleanup(dev);
 
-	kfree(priv->rx_ring);
-	vfree(priv->tx_ring);
-
-	priv->rx_ring = NULL;
-	priv->tx_ring = NULL;
+	/* no more works over the priv->wq */
+	if (priv->wq) {
+		flush_workqueue(priv->wq);
+		destroy_workqueue(priv->wq);
+		priv->wq = NULL;
+	}
 }
 
 static int ipoib_set_vf_link_state(struct net_device *dev, int vf, int link_state)
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
index 189dcd1709d2..529b9509a2c4 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
@@ -147,22 +147,6 @@ int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca)
 	int ret, size;
 	int i;
 
-	priv->pd = ib_alloc_pd(priv->ca, 0);
-	if (IS_ERR(priv->pd)) {
-		printk(KERN_WARNING "%s: failed to allocate PD\n", ca->name);
-		return -ENODEV;
-	}
-
-	/*
-	 * the various IPoIB tasks assume they will never race against
-	 * themselves, so always use a single thread workqueue
-	 */
-	priv->wq = alloc_ordered_workqueue("ipoib_wq", WQ_MEM_RECLAIM);
-	if (!priv->wq) {
-		printk(KERN_WARNING "ipoib: failed to allocate device WQ\n");
-		goto out_free_pd;
-	}
-
 	size = ipoib_recvq_size + 1;
 	ret = ipoib_cm_dev_init(dev);
 	if (!ret) {
@@ -173,7 +157,7 @@ int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca)
 			size += ipoib_recvq_size * ipoib_max_conn_qp;
 	} else
 		if (ret != -ENOSYS)
-			goto out_free_wq;
+			return -ENODEV;
 
 	cq_attr.cqe = size;
 	priv->recv_cq = ib_create_cq(priv->ca, ipoib_ib_completion, NULL,
@@ -212,10 +196,6 @@ int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca)
 		goto out_free_send_cq;
 	}
 
-	priv->dev->dev_addr[1] = (priv->qp->qp_num >> 16) & 0xff;
-	priv->dev->dev_addr[2] = (priv->qp->qp_num >>  8) & 0xff;
-	priv->dev->dev_addr[3] = (priv->qp->qp_num      ) & 0xff;
-
 	for (i = 0; i < MAX_SKB_FRAGS + 1; ++i)
 		priv->tx_sge[i].lkey = priv->pd->local_dma_lkey;
 
@@ -247,13 +227,6 @@ int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca)
 out_cm_dev_cleanup:
 	ipoib_cm_dev_cleanup(dev);
 
-out_free_wq:
-	destroy_workqueue(priv->wq);
-	priv->wq = NULL;
-
-out_free_pd:
-	ib_dealloc_pd(priv->pd);
-
 	return -ENODEV;
 }
 
@@ -266,7 +239,6 @@ void ipoib_transport_dev_cleanup(struct net_device *dev)
 			ipoib_warn(priv, "ib_qp_destroy failed\n");
 
 		priv->qp = NULL;
-		clear_bit(IPOIB_PKEY_ASSIGNED, &priv->flags);
 	}
 
 	if (ib_destroy_cq(priv->send_cq))
@@ -274,16 +246,6 @@ void ipoib_transport_dev_cleanup(struct net_device *dev)
 
 	if (ib_destroy_cq(priv->recv_cq))
 		ipoib_warn(priv, "ib_cq_destroy (recv) failed\n");
-
-	ipoib_cm_dev_cleanup(dev);
-
-	if (priv->wq) {
-		flush_workqueue(priv->wq);
-		destroy_workqueue(priv->wq);
-		priv->wq = NULL;
-	}
-
-	ib_dealloc_pd(priv->pd);
 }
 
 void ipoib_event(struct ib_event_handler *handler,
-- 
2.12.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 3/6] IB/IPoIB: Separate control from HW operation on ipoib_open/stop ndo

[Leon Romanovsky]

From: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

This patch is preparing the netdev part at the IPoIB driver to be able
to use the ipoib_options.

It deals with the two flows from the .ndo: ipoib_open and ipoib_stop.

The code is rearranged as follows:
 * All operations which deal with the hardware resources, (for example
   change QP state, post-receive etc.) are performed in one place.
 * All operations that are control oriented (like restart multicast task,
   start the reap_ah etc.) are performed in separate place.

The functions that deal with the hardware resources now located at
__ipoib_ib_dev_open for the ipoib_open flow and __ipoib_ib_dev_stop
for ipoib_stop.

Signed-off-by: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/ulp/ipoib/ipoib.h      |   2 +-
 drivers/infiniband/ulp/ipoib/ipoib_ib.c   | 228 +++++++++++++++++-------------
 drivers/infiniband/ulp/ipoib/ipoib_main.c |   2 +-
 3 files changed, 129 insertions(+), 103 deletions(-)

diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h
index 7cd9befd7d54..b3900b253ad5 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib.h
+++ b/drivers/infiniband/ulp/ipoib/ipoib.h
@@ -501,7 +501,7 @@ void ipoib_dev_uninit_default(struct net_device *dev);
 int ipoib_ib_dev_open(struct net_device *dev);
 void ipoib_ib_dev_up(struct net_device *dev);
 void ipoib_ib_dev_down(struct net_device *dev);
-void ipoib_ib_dev_stop(struct net_device *dev);
+int ipoib_ib_dev_stop_default(struct net_device *dev);
 void ipoib_pkey_dev_check_presence(struct net_device *dev);
 
 int ipoib_dev_init(struct net_device *dev, struct ib_device *ca, int port);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index f9dd2bb676aa..5eaa4e6cebcd 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -692,24 +692,113 @@ static void ipoib_stop_ah(struct net_device *dev)
 	ipoib_flush_ah(dev);
 }
 
-void ipoib_ib_tx_timer_func(unsigned long ctx)
+static int recvs_pending(struct net_device *dev)
 {
-	drain_tx_cq((struct net_device *)ctx);
+	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	int pending = 0;
+	int i;
+
+	for (i = 0; i < ipoib_recvq_size; ++i)
+		if (priv->rx_ring[i].skb)
+			++pending;
+
+	return pending;
 }
 
-int ipoib_ib_dev_open(struct net_device *dev)
+int ipoib_ib_dev_stop_default(struct net_device *dev)
 {
 	struct ipoib_dev_priv *priv = netdev_priv(dev);
-	int ret;
+	struct ib_qp_attr qp_attr;
+	unsigned long begin;
+	struct ipoib_tx_buf *tx_req;
+	int i;
 
-	ipoib_pkey_dev_check_presence(dev);
+	if (test_and_clear_bit(IPOIB_FLAG_INITIALIZED, &priv->flags))
+		napi_disable(&priv->napi);
 
-	if (!test_bit(IPOIB_PKEY_ASSIGNED, &priv->flags)) {
-		ipoib_warn(priv, "P_Key 0x%04x is %s\n", priv->pkey,
-			   (!(priv->pkey & 0x7fff) ? "Invalid" : "not found"));
-		return -1;
+	ipoib_cm_dev_stop(dev);
+
+	/*
+	 * Move our QP to the error state and then reinitialize in
+	 * when all work requests have completed or have been flushed.
+	 */
+	qp_attr.qp_state = IB_QPS_ERR;
+	if (ib_modify_qp(priv->qp, &qp_attr, IB_QP_STATE))
+		ipoib_warn(priv, "Failed to modify QP to ERROR state\n");
+
+	/* Wait for all sends and receives to complete */
+	begin = jiffies;
+
+	while (priv->tx_head != priv->tx_tail || recvs_pending(dev)) {
+		if (time_after(jiffies, begin + 5 * HZ)) {
+			ipoib_warn(priv, "timing out; %d sends %d receives not completed\n",
+				   priv->tx_head - priv->tx_tail, recvs_pending(dev));
+
+			/*
+			 * assume the HW is wedged and just free up
+			 * all our pending work requests.
+			 */
+			while ((int) priv->tx_tail - (int) priv->tx_head < 0) {
+				tx_req = &priv->tx_ring[priv->tx_tail &
+							(ipoib_sendq_size - 1)];
+				ipoib_dma_unmap_tx(priv, tx_req);
+				dev_kfree_skb_any(tx_req->skb);
+				++priv->tx_tail;
+				--priv->tx_outstanding;
+			}
+
+			for (i = 0; i < ipoib_recvq_size; ++i) {
+				struct ipoib_rx_buf *rx_req;
+
+				rx_req = &priv->rx_ring[i];
+				if (!rx_req->skb)
+					continue;
+				ipoib_ud_dma_unmap_rx(priv,
+						      priv->rx_ring[i].mapping);
+				dev_kfree_skb_any(rx_req->skb);
+				rx_req->skb = NULL;
+			}
+
+			goto timeout;
+		}
+
+		ipoib_drain_cq(dev);
+
+		msleep(1);
 	}
 
+	ipoib_dbg(priv, "All sends and receives done.\n");
+
+timeout:
+	del_timer_sync(&priv->poll_timer);
+	qp_attr.qp_state = IB_QPS_RESET;
+	if (ib_modify_qp(priv->qp, &qp_attr, IB_QP_STATE))
+		ipoib_warn(priv, "Failed to modify QP to RESET state\n");
+
+	ib_req_notify_cq(priv->recv_cq, IB_CQ_NEXT_COMP);
+
+	return 0;
+}
+
+int ipoib_ib_dev_stop(struct net_device *dev)
+{
+	ipoib_ib_dev_stop_default(dev);
+
+	ipoib_flush_ah(dev);
+
+	return 0;
+}
+
+void ipoib_ib_tx_timer_func(unsigned long ctx)
+{
+	drain_tx_cq((struct net_device *)ctx);
+}
+
+int ipoib_ib_dev_open_default(struct net_device *dev)
+{
+	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	int ret;
+
 	ret = ipoib_init_qp(dev);
 	if (ret) {
 		ipoib_warn(priv, "ipoib_init_qp returned %d\n", ret);
@@ -728,10 +817,6 @@ int ipoib_ib_dev_open(struct net_device *dev)
 		goto dev_stop;
 	}
 
-	clear_bit(IPOIB_STOP_REAPER, &priv->flags);
-	queue_delayed_work(priv->wq, &priv->ah_reap_task,
-			   round_jiffies_relative(HZ));
-
 	if (!test_and_set_bit(IPOIB_FLAG_INITIALIZED, &priv->flags))
 		napi_enable(&priv->napi);
 
@@ -743,6 +828,35 @@ int ipoib_ib_dev_open(struct net_device *dev)
 	return -1;
 }
 
+int ipoib_ib_dev_open(struct net_device *dev)
+{
+	struct ipoib_dev_priv *priv = netdev_priv(dev);
+
+	ipoib_pkey_dev_check_presence(dev);
+
+	if (!test_bit(IPOIB_PKEY_ASSIGNED, &priv->flags)) {
+		ipoib_warn(priv, "P_Key 0x%04x is %s\n", priv->pkey,
+			   (!(priv->pkey & 0x7fff) ? "Invalid" : "not found"));
+		return -1;
+	}
+
+	clear_bit(IPOIB_STOP_REAPER, &priv->flags);
+	queue_delayed_work(priv->wq, &priv->ah_reap_task,
+			   round_jiffies_relative(HZ));
+
+	if (ipoib_ib_dev_open_default(dev)) {
+		pr_warn("%s: Failed to open dev\n", dev->name);
+		goto stop_ah_reap;
+	}
+
+	return 0;
+
+stop_ah_reap:
+	set_bit(IPOIB_STOP_REAPER, &priv->flags);
+	cancel_delayed_work(&priv->ah_reap_task);
+	return -1;
+}
+
 void ipoib_pkey_dev_check_presence(struct net_device *dev)
 {
 	struct ipoib_dev_priv *priv = netdev_priv(dev);
@@ -786,19 +900,6 @@ void ipoib_ib_dev_down(struct net_device *dev)
 	ipoib_flush_paths(dev);
 }
 
-static int recvs_pending(struct net_device *dev)
-{
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
-	int pending = 0;
-	int i;
-
-	for (i = 0; i < ipoib_recvq_size; ++i)
-		if (priv->rx_ring[i].skb)
-			++pending;
-
-	return pending;
-}
-
 void ipoib_drain_cq(struct net_device *dev)
 {
 	struct ipoib_dev_priv *priv = netdev_priv(dev);
@@ -838,81 +939,6 @@ void ipoib_drain_cq(struct net_device *dev)
 	local_bh_enable();
 }
 
-void ipoib_ib_dev_stop(struct net_device *dev)
-{
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
-	struct ib_qp_attr qp_attr;
-	unsigned long begin;
-	struct ipoib_tx_buf *tx_req;
-	int i;
-
-	if (test_and_clear_bit(IPOIB_FLAG_INITIALIZED, &priv->flags))
-		napi_disable(&priv->napi);
-
-	ipoib_cm_dev_stop(dev);
-
-	/*
-	 * Move our QP to the error state and then reinitialize in
-	 * when all work requests have completed or have been flushed.
-	 */
-	qp_attr.qp_state = IB_QPS_ERR;
-	if (ib_modify_qp(priv->qp, &qp_attr, IB_QP_STATE))
-		ipoib_warn(priv, "Failed to modify QP to ERROR state\n");
-
-	/* Wait for all sends and receives to complete */
-	begin = jiffies;
-
-	while (priv->tx_head != priv->tx_tail || recvs_pending(dev)) {
-		if (time_after(jiffies, begin + 5 * HZ)) {
-			ipoib_warn(priv, "timing out; %d sends %d receives not completed\n",
-				   priv->tx_head - priv->tx_tail, recvs_pending(dev));
-
-			/*
-			 * assume the HW is wedged and just free up
-			 * all our pending work requests.
-			 */
-			while ((int) priv->tx_tail - (int) priv->tx_head < 0) {
-				tx_req = &priv->tx_ring[priv->tx_tail &
-							(ipoib_sendq_size - 1)];
-				ipoib_dma_unmap_tx(priv, tx_req);
-				dev_kfree_skb_any(tx_req->skb);
-				++priv->tx_tail;
-				--priv->tx_outstanding;
-			}
-
-			for (i = 0; i < ipoib_recvq_size; ++i) {
-				struct ipoib_rx_buf *rx_req;
-
-				rx_req = &priv->rx_ring[i];
-				if (!rx_req->skb)
-					continue;
-				ipoib_ud_dma_unmap_rx(priv,
-						      priv->rx_ring[i].mapping);
-				dev_kfree_skb_any(rx_req->skb);
-				rx_req->skb = NULL;
-			}
-
-			goto timeout;
-		}
-
-		ipoib_drain_cq(dev);
-
-		msleep(1);
-	}
-
-	ipoib_dbg(priv, "All sends and receives done.\n");
-
-timeout:
-	del_timer_sync(&priv->poll_timer);
-	qp_attr.qp_state = IB_QPS_RESET;
-	if (ib_modify_qp(priv->qp, &qp_attr, IB_QP_STATE))
-		ipoib_warn(priv, "Failed to modify QP to RESET state\n");
-
-	ipoib_flush_ah(dev);
-
-	ib_req_notify_cq(priv->recv_cq, IB_CQ_NEXT_COMP);
-}
-
 /*
  * Takes whatever value which is in pkey index 0 and updates priv->pkey
  * returns 0 if the pkey value was changed.
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c
index ef8e51ccb1cd..08c85084d8f5 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -166,7 +166,7 @@ static int ipoib_stop(struct net_device *dev)
 	netif_stop_queue(dev);
 
 	ipoib_ib_dev_down(dev);
-	ipoib_ib_dev_stop(dev);
+	ipoib_ib_dev_stop_default(dev);
 
 	if (!test_bit(IPOIB_FLAG_SUBINTERFACE, &priv->flags)) {
 		struct ipoib_dev_priv *cpriv;
-- 
2.12.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 4/6] IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions

[Leon Romanovsky]

From: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Change of function parameter name from qpn to be dqpn.

Signed-off-by: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/ulp/ipoib/ipoib.h    | 2 +-
 drivers/infiniband/ulp/ipoib/ipoib_ib.c | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h
index b3900b253ad5..d94a7a953338 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib.h
+++ b/drivers/infiniband/ulp/ipoib/ipoib.h
@@ -483,7 +483,7 @@ int ipoib_add_pkey_attr(struct net_device *dev);
 int ipoib_add_umcast_attr(struct net_device *dev);
 
 void ipoib_send(struct net_device *dev, struct sk_buff *skb,
-		struct ipoib_ah *address, u32 qpn);
+		struct ipoib_ah *address, u32 dqpn);
 void ipoib_reap_ah(struct work_struct *work);
 
 struct ipoib_path *__path_find(struct net_device *dev, void *gid);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index 5eaa4e6cebcd..688688c21ef7 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -513,7 +513,7 @@ void ipoib_send_comp_handler(struct ib_cq *cq, void *dev_ptr)
 
 static inline int post_send(struct ipoib_dev_priv *priv,
 			    unsigned int wr_id,
-			    struct ib_ah *address, u32 qpn,
+			    struct ib_ah *address, u32 dqpn,
 			    struct ipoib_tx_buf *tx_req,
 			    void *head, int hlen)
 {
@@ -523,7 +523,7 @@ static inline int post_send(struct ipoib_dev_priv *priv,
 	ipoib_build_sge(priv, tx_req);
 
 	priv->tx_wr.wr.wr_id	= wr_id;
-	priv->tx_wr.remote_qpn	= qpn;
+	priv->tx_wr.remote_qpn	= dqpn;
 	priv->tx_wr.ah		= address;
 
 	if (head) {
@@ -538,7 +538,7 @@ static inline int post_send(struct ipoib_dev_priv *priv,
 }
 
 void ipoib_send(struct net_device *dev, struct sk_buff *skb,
-		struct ipoib_ah *address, u32 qpn)
+		struct ipoib_ah *address, u32 dqpn)
 {
 	struct ipoib_dev_priv *priv = netdev_priv(dev);
 	struct ipoib_tx_buf *tx_req;
@@ -620,7 +620,7 @@ void ipoib_send(struct net_device *dev, struct sk_buff *skb,
 	skb_dst_drop(skb);
 
 	rc = post_send(priv, priv->tx_head & (ipoib_sendq_size - 1),
-		       address->ah, qpn, tx_req, phead, hlen);
+		       address->ah, dqpn, tx_req, phead, hlen);
 	if (unlikely(rc)) {
 		ipoib_warn(priv, "post_send failed, error %d\n", rc);
 		++dev->stats.tx_errors;
-- 
2.12.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 5/6] IB/IPoIB: Formatting before adding accelerator to driver

[Leon Romanovsky]

From: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

Make ipoib_priv point to netdev_priv where the code calls netdev_priv.

Signed-off-by: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/ulp/ipoib/ipoib.h           | 11 +++--
 drivers/infiniband/ulp/ipoib/ipoib_cm.c        | 66 ++++++++++++-------------
 drivers/infiniband/ulp/ipoib/ipoib_ethtool.c   |  6 +--
 drivers/infiniband/ulp/ipoib/ipoib_fs.c        |  4 +-
 drivers/infiniband/ulp/ipoib/ipoib_ib.c        | 42 ++++++++--------
 drivers/infiniband/ulp/ipoib/ipoib_main.c      | 68 +++++++++++++-------------
 drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 18 +++----
 drivers/infiniband/ulp/ipoib/ipoib_netlink.c   | 10 ++--
 drivers/infiniband/ulp/ipoib/ipoib_verbs.c     |  4 +-
 drivers/infiniband/ulp/ipoib/ipoib_vlan.c      |  4 +-
 10 files changed, 118 insertions(+), 115 deletions(-)

diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h
index d94a7a953338..055adacd6d06 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib.h
+++ b/drivers/infiniband/ulp/ipoib/ipoib.h
@@ -153,6 +153,9 @@ static inline void skb_add_pseudo_hdr(struct sk_buff *skb)
 	skb_pull(skb, IPOIB_HARD_LEN);
 }
 
+/* Keep the refactoring compile able */
+#define ipoib_priv netdev_priv
+
 /* Used for all multicast joins (broadcast, IPv4 mcast and IPv6 mcast) */
 struct ipoib_mcast {
 	struct ib_sa_mcmember_rec mcmember;
@@ -606,14 +609,14 @@ extern int ipoib_max_conn_qp;
 
 static inline int ipoib_cm_admin_enabled(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	return IPOIB_CM_SUPPORTED(dev->dev_addr) &&
 		test_bit(IPOIB_FLAG_ADMIN_CM, &priv->flags);
 }
 
 static inline int ipoib_cm_enabled(struct net_device *dev, u8 *hwaddr)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	return IPOIB_CM_SUPPORTED(hwaddr) &&
 		test_bit(IPOIB_FLAG_ADMIN_CM, &priv->flags);
 }
@@ -636,13 +639,13 @@ static inline void ipoib_cm_set(struct ipoib_neigh *neigh, struct ipoib_cm_tx *t
 
 static inline int ipoib_cm_has_srq(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	return !!priv->cm.srq;
 }
 
 static inline unsigned int ipoib_cm_max_mtu(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	return priv->cm.max_cm_mtu;
 }
 
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
index 0cdf2b7f272f..052a60494f9b 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
@@ -92,7 +92,7 @@ static void ipoib_cm_dma_unmap_rx(struct ipoib_dev_priv *priv, int frags,
 
 static int ipoib_cm_post_receive_srq(struct net_device *dev, int id)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_recv_wr *bad_wr;
 	int i, ret;
 
@@ -118,7 +118,7 @@ static int ipoib_cm_post_receive_nonsrq(struct net_device *dev,
 					struct ib_recv_wr *wr,
 					struct ib_sge *sge, int id)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_recv_wr *bad_wr;
 	int i, ret;
 
@@ -145,7 +145,7 @@ static struct sk_buff *ipoib_cm_alloc_rx_skb(struct net_device *dev,
 					     u64 mapping[IPOIB_CM_RX_SG],
 					     gfp_t gfp)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct sk_buff *skb;
 	int i;
 
@@ -196,7 +196,7 @@ static struct sk_buff *ipoib_cm_alloc_rx_skb(struct net_device *dev,
 static void ipoib_cm_free_rx_ring(struct net_device *dev,
 				  struct ipoib_cm_rx_buf *rx_ring)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int i;
 
 	for (i = 0; i < ipoib_recvq_size; ++i)
@@ -235,7 +235,7 @@ static void ipoib_cm_start_rx_drain(struct ipoib_dev_priv *priv)
 static void ipoib_cm_rx_event_handler(struct ib_event *event, void *ctx)
 {
 	struct ipoib_cm_rx *p = ctx;
-	struct ipoib_dev_priv *priv = netdev_priv(p->dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(p->dev);
 	unsigned long flags;
 
 	if (event->event != IB_EVENT_QP_LAST_WQE_REACHED)
@@ -251,7 +251,7 @@ static void ipoib_cm_rx_event_handler(struct ib_event *event, void *ctx)
 static struct ib_qp *ipoib_cm_create_rx_qp(struct net_device *dev,
 					   struct ipoib_cm_rx *p)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_qp_init_attr attr = {
 		.event_handler = ipoib_cm_rx_event_handler,
 		.send_cq = priv->recv_cq, /* For drain WR */
@@ -276,7 +276,7 @@ static int ipoib_cm_modify_rx_qp(struct net_device *dev,
 				 struct ib_cm_id *cm_id, struct ib_qp *qp,
 				 unsigned psn)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_qp_attr qp_attr;
 	int qp_attr_mask, ret;
 
@@ -331,7 +331,7 @@ static void ipoib_cm_init_rx_wr(struct net_device *dev,
 				struct ib_recv_wr *wr,
 				struct ib_sge *sge)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int i;
 
 	for (i = 0; i < priv->cm.num_frags; ++i)
@@ -349,7 +349,7 @@ static void ipoib_cm_init_rx_wr(struct net_device *dev,
 static int ipoib_cm_nonsrq_init_rx(struct net_device *dev, struct ib_cm_id *cm_id,
 				   struct ipoib_cm_rx *rx)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct {
 		struct ib_recv_wr wr;
 		struct ib_sge sge[IPOIB_CM_RX_SG];
@@ -422,7 +422,7 @@ static int ipoib_cm_send_rep(struct net_device *dev, struct ib_cm_id *cm_id,
 			     struct ib_qp *qp, struct ib_cm_req_event_param *req,
 			     unsigned psn)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_cm_data data = {};
 	struct ib_cm_rep_param rep = {};
 
@@ -442,7 +442,7 @@ static int ipoib_cm_send_rep(struct net_device *dev, struct ib_cm_id *cm_id,
 static int ipoib_cm_req_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
 {
 	struct net_device *dev = cm_id->context;
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_cm_rx *p;
 	unsigned psn;
 	int ret;
@@ -515,7 +515,7 @@ static int ipoib_cm_rx_handler(struct ib_cm_id *cm_id,
 		/* Fall through */
 	case IB_CM_REJ_RECEIVED:
 		p = cm_id->context;
-		priv = netdev_priv(p->dev);
+		priv = ipoib_priv(p->dev);
 		if (ib_modify_qp(p->qp, &ipoib_cm_err_attr, IB_QP_STATE))
 			ipoib_warn(priv, "unable to move qp to error state\n");
 		/* Fall through */
@@ -559,7 +559,7 @@ static void skb_put_frags(struct sk_buff *skb, unsigned int hdr_space,
 
 void ipoib_cm_handle_rx_wc(struct net_device *dev, struct ib_wc *wc)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_cm_rx_buf *rx_ring;
 	unsigned int wr_id = wc->wr_id & ~(IPOIB_OP_CM | IPOIB_OP_RECV);
 	struct sk_buff *skb, *newskb;
@@ -708,7 +708,7 @@ static inline int post_send(struct ipoib_dev_priv *priv,
 
 void ipoib_cm_send(struct net_device *dev, struct sk_buff *skb, struct ipoib_cm_tx *tx)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_tx_buf *tx_req;
 	int rc;
 	unsigned usable_sge = tx->max_send_sge - !!skb_headlen(skb);
@@ -786,7 +786,7 @@ void ipoib_cm_send(struct net_device *dev, struct sk_buff *skb, struct ipoib_cm_
 
 void ipoib_cm_handle_tx_wc(struct net_device *dev, struct ib_wc *wc)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_cm_tx *tx = wc->qp->qp_context;
 	unsigned int wr_id = wc->wr_id & ~IPOIB_OP_CM;
 	struct ipoib_tx_buf *tx_req;
@@ -855,7 +855,7 @@ void ipoib_cm_handle_tx_wc(struct net_device *dev, struct ib_wc *wc)
 
 int ipoib_cm_dev_open(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int ret;
 
 	if (!IPOIB_CM_SUPPORTED(dev->dev_addr))
@@ -887,7 +887,7 @@ int ipoib_cm_dev_open(struct net_device *dev)
 
 static void ipoib_cm_free_rx_reap_list(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_cm_rx *rx, *n;
 	LIST_HEAD(list);
 
@@ -910,7 +910,7 @@ static void ipoib_cm_free_rx_reap_list(struct net_device *dev)
 
 void ipoib_cm_dev_stop(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_cm_rx *p;
 	unsigned long begin;
 	int ret;
@@ -969,7 +969,7 @@ void ipoib_cm_dev_stop(struct net_device *dev)
 static int ipoib_cm_rep_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event)
 {
 	struct ipoib_cm_tx *p = cm_id->context;
-	struct ipoib_dev_priv *priv = netdev_priv(p->dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(p->dev);
 	struct ipoib_cm_data *data = event->private_data;
 	struct sk_buff_head skqueue;
 	struct ib_qp_attr qp_attr;
@@ -1037,7 +1037,7 @@ static int ipoib_cm_rep_handler(struct ib_cm_id *cm_id, struct ib_cm_event *even
 
 static struct ib_qp *ipoib_cm_create_tx_qp(struct net_device *dev, struct ipoib_cm_tx *tx)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_qp_init_attr attr = {
 		.send_cq		= priv->recv_cq,
 		.recv_cq		= priv->recv_cq,
@@ -1070,7 +1070,7 @@ static int ipoib_cm_send_req(struct net_device *dev,
 			     u32 qpn,
 			     struct ib_sa_path_rec *pathrec)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_cm_data data = {};
 	struct ib_cm_req_param req = {};
 
@@ -1105,7 +1105,7 @@ static int ipoib_cm_send_req(struct net_device *dev,
 static int ipoib_cm_modify_tx_init(struct net_device *dev,
 				  struct ib_cm_id *cm_id, struct ib_qp *qp)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_qp_attr qp_attr;
 	int qp_attr_mask, ret;
 	ret = ib_find_pkey(priv->ca, priv->port, priv->pkey, &qp_attr.pkey_index);
@@ -1130,7 +1130,7 @@ static int ipoib_cm_modify_tx_init(struct net_device *dev,
 static int ipoib_cm_tx_init(struct ipoib_cm_tx *p, u32 qpn,
 			    struct ib_sa_path_rec *pathrec)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(p->dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(p->dev);
 	int ret;
 
 	p->tx_ring = __vmalloc(ipoib_sendq_size * sizeof *p->tx_ring,
@@ -1186,7 +1186,7 @@ static int ipoib_cm_tx_init(struct ipoib_cm_tx *p, u32 qpn,
 
 static void ipoib_cm_tx_destroy(struct ipoib_cm_tx *p)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(p->dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(p->dev);
 	struct ipoib_tx_buf *tx_req;
 	unsigned long begin;
 
@@ -1236,7 +1236,7 @@ static int ipoib_cm_tx_handler(struct ib_cm_id *cm_id,
 			       struct ib_cm_event *event)
 {
 	struct ipoib_cm_tx *tx = cm_id->context;
-	struct ipoib_dev_priv *priv = netdev_priv(tx->dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(tx->dev);
 	struct net_device *dev = priv->dev;
 	struct ipoib_neigh *neigh;
 	unsigned long flags;
@@ -1287,7 +1287,7 @@ static int ipoib_cm_tx_handler(struct ib_cm_id *cm_id,
 struct ipoib_cm_tx *ipoib_cm_create_tx(struct net_device *dev, struct ipoib_path *path,
 				       struct ipoib_neigh *neigh)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_cm_tx *tx;
 
 	tx = kzalloc(sizeof *tx, GFP_ATOMIC);
@@ -1306,7 +1306,7 @@ struct ipoib_cm_tx *ipoib_cm_create_tx(struct net_device *dev, struct ipoib_path
 
 void ipoib_cm_destroy_tx(struct ipoib_cm_tx *tx)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(tx->dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(tx->dev);
 	unsigned long flags;
 	if (test_and_clear_bit(IPOIB_FLAG_INITIALIZED, &tx->flags)) {
 		spin_lock_irqsave(&priv->lock, flags);
@@ -1441,7 +1441,7 @@ static void ipoib_cm_skb_reap(struct work_struct *work)
 void ipoib_cm_skb_too_long(struct net_device *dev, struct sk_buff *skb,
 			   unsigned int mtu)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int e = skb_queue_empty(&priv->cm.skb_queue);
 
 	if (skb_dst(skb))
@@ -1490,7 +1490,7 @@ static void ipoib_cm_stale_task(struct work_struct *work)
 static ssize_t show_mode(struct device *d, struct device_attribute *attr,
 			 char *buf)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(to_net_dev(d));
+	struct ipoib_dev_priv *priv = ipoib_priv(to_net_dev(d));
 
 	if (test_bit(IPOIB_FLAG_ADMIN_CM, &priv->flags))
 		return sprintf(buf, "connected\n");
@@ -1503,7 +1503,7 @@ static ssize_t set_mode(struct device *d, struct device_attribute *attr,
 {
 	struct net_device *dev = to_net_dev(d);
 	int ret;
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	if (test_bit(IPOIB_FLAG_GOING_DOWN, &priv->flags))
 		return -EPERM;
@@ -1532,7 +1532,7 @@ int ipoib_cm_add_mode_attr(struct net_device *dev)
 
 static void ipoib_cm_create_srq(struct net_device *dev, int max_sge)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_srq_init_attr srq_init_attr = {
 		.srq_type = IB_SRQT_BASIC,
 		.attr = {
@@ -1561,7 +1561,7 @@ static void ipoib_cm_create_srq(struct net_device *dev, int max_sge)
 
 int ipoib_cm_dev_init(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int max_srq_sge, i;
 
 	INIT_LIST_HEAD(&priv->cm.passive_ids);
@@ -1622,7 +1622,7 @@ int ipoib_cm_dev_init(struct net_device *dev)
 
 void ipoib_cm_dev_cleanup(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int ret;
 
 	if (!priv->cm.srq)
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c
index bac455a1942d..379c02fb4181 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ethtool.c
@@ -60,7 +60,7 @@ static const struct ipoib_stats ipoib_gstrings_stats[] = {
 static void ipoib_get_drvinfo(struct net_device *netdev,
 			      struct ethtool_drvinfo *drvinfo)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(netdev);
+	struct ipoib_dev_priv *priv = ipoib_priv(netdev);
 
 	ib_get_device_fw_str(priv->ca, drvinfo->fw_version,
 			     sizeof(drvinfo->fw_version));
@@ -77,7 +77,7 @@ static void ipoib_get_drvinfo(struct net_device *netdev,
 static int ipoib_get_coalesce(struct net_device *dev,
 			      struct ethtool_coalesce *coal)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	coal->rx_coalesce_usecs = priv->ethtool.coalesce_usecs;
 	coal->rx_max_coalesced_frames = priv->ethtool.max_coalesced_frames;
@@ -88,7 +88,7 @@ static int ipoib_get_coalesce(struct net_device *dev,
 static int ipoib_set_coalesce(struct net_device *dev,
 			      struct ethtool_coalesce *coal)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int ret;
 
 	/*
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_fs.c b/drivers/infiniband/ulp/ipoib/ipoib_fs.c
index 6bd5740e2691..6e73450241bf 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_fs.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_fs.c
@@ -261,7 +261,7 @@ static const struct file_operations ipoib_path_fops = {
 
 void ipoib_create_debug_files(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	char name[IFNAMSIZ + sizeof "_path"];
 
 	snprintf(name, sizeof name, "%s_mcg", dev->name);
@@ -279,7 +279,7 @@ void ipoib_create_debug_files(struct net_device *dev)
 
 void ipoib_delete_debug_files(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	debugfs_remove(priv->mcg_dentry);
 	debugfs_remove(priv->path_dentry);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index 688688c21ef7..374533ce82fd 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -71,7 +71,7 @@ struct ipoib_ah *ipoib_create_ah(struct net_device *dev,
 		ah = (struct ipoib_ah *)vah;
 	} else {
 		ah->ah = vah;
-		ipoib_dbg(netdev_priv(dev), "Created ah %p\n", ah->ah);
+		ipoib_dbg(ipoib_priv(dev), "Created ah %p\n", ah->ah);
 	}
 
 	return ah;
@@ -80,7 +80,7 @@ struct ipoib_ah *ipoib_create_ah(struct net_device *dev,
 void ipoib_free_ah(struct kref *kref)
 {
 	struct ipoib_ah *ah = container_of(kref, struct ipoib_ah, ref);
-	struct ipoib_dev_priv *priv = netdev_priv(ah->dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(ah->dev);
 
 	unsigned long flags;
 
@@ -99,7 +99,7 @@ static void ipoib_ud_dma_unmap_rx(struct ipoib_dev_priv *priv,
 
 static int ipoib_ib_post_receive(struct net_device *dev, int id)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_recv_wr *bad_wr;
 	int ret;
 
@@ -121,7 +121,7 @@ static int ipoib_ib_post_receive(struct net_device *dev, int id)
 
 static struct sk_buff *ipoib_alloc_rx_skb(struct net_device *dev, int id)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct sk_buff *skb;
 	int buf_size;
 	u64 *mapping;
@@ -153,7 +153,7 @@ static struct sk_buff *ipoib_alloc_rx_skb(struct net_device *dev, int id)
 
 static int ipoib_ib_post_receives(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int i;
 
 	for (i = 0; i < ipoib_recvq_size; ++i) {
@@ -172,7 +172,7 @@ static int ipoib_ib_post_receives(struct net_device *dev)
 
 static void ipoib_ib_handle_rx_wc(struct net_device *dev, struct ib_wc *wc)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	unsigned int wr_id = wc->wr_id & ~IPOIB_OP_RECV;
 	struct sk_buff *skb;
 	u64 mapping[IPOIB_UD_RX_SG];
@@ -381,7 +381,7 @@ static void ipoib_qp_state_validate_work(struct work_struct *work)
 
 static void ipoib_ib_handle_tx_wc(struct net_device *dev, struct ib_wc *wc)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	unsigned int wr_id = wc->wr_id;
 	struct ipoib_tx_buf *tx_req;
 
@@ -485,14 +485,14 @@ int ipoib_poll(struct napi_struct *napi, int budget)
 void ipoib_ib_completion(struct ib_cq *cq, void *dev_ptr)
 {
 	struct net_device *dev = dev_ptr;
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	napi_schedule(&priv->napi);
 }
 
 static void drain_tx_cq(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	netif_tx_lock(dev);
 	while (poll_tx(priv))
@@ -506,7 +506,7 @@ static void drain_tx_cq(struct net_device *dev)
 
 void ipoib_send_comp_handler(struct ib_cq *cq, void *dev_ptr)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev_ptr);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev_ptr);
 
 	mod_timer(&priv->poll_timer, jiffies);
 }
@@ -677,7 +677,7 @@ void ipoib_reap_ah(struct work_struct *work)
 
 static void ipoib_flush_ah(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	cancel_delayed_work(&priv->ah_reap_task);
 	flush_workqueue(priv->wq);
@@ -686,7 +686,7 @@ static void ipoib_flush_ah(struct net_device *dev)
 
 static void ipoib_stop_ah(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	set_bit(IPOIB_STOP_REAPER, &priv->flags);
 	ipoib_flush_ah(dev);
@@ -694,7 +694,7 @@ static void ipoib_stop_ah(struct net_device *dev)
 
 static int recvs_pending(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int pending = 0;
 	int i;
 
@@ -707,7 +707,7 @@ static int recvs_pending(struct net_device *dev)
 
 int ipoib_ib_dev_stop_default(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_qp_attr qp_attr;
 	unsigned long begin;
 	struct ipoib_tx_buf *tx_req;
@@ -796,7 +796,7 @@ void ipoib_ib_tx_timer_func(unsigned long ctx)
 
 int ipoib_ib_dev_open_default(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int ret;
 
 	ret = ipoib_init_qp(dev);
@@ -830,7 +830,7 @@ int ipoib_ib_dev_open_default(struct net_device *dev)
 
 int ipoib_ib_dev_open(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	ipoib_pkey_dev_check_presence(dev);
 
@@ -859,7 +859,7 @@ int ipoib_ib_dev_open(struct net_device *dev)
 
 void ipoib_pkey_dev_check_presence(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	if (!(priv->pkey & 0x7fff) ||
 	    ib_find_pkey(priv->ca, priv->port, priv->pkey,
@@ -871,7 +871,7 @@ void ipoib_pkey_dev_check_presence(struct net_device *dev)
 
 void ipoib_ib_dev_up(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	ipoib_pkey_dev_check_presence(dev);
 
@@ -887,7 +887,7 @@ void ipoib_ib_dev_up(struct net_device *dev)
 
 void ipoib_ib_dev_down(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	ipoib_dbg(priv, "downing ib_dev\n");
 
@@ -902,7 +902,7 @@ void ipoib_ib_dev_down(struct net_device *dev)
 
 void ipoib_drain_cq(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int i, n;
 
 	/*
@@ -1216,7 +1216,7 @@ void ipoib_ib_dev_flush_heavy(struct work_struct *work)
 
 void ipoib_ib_dev_cleanup(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	ipoib_dbg(priv, "cleaning up ib_dev\n");
 	/*
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c
index 08c85084d8f5..a88df83677b1 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -110,7 +110,7 @@ static struct ib_client ipoib_client = {
 
 int ipoib_open(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	ipoib_dbg(priv, "bringing up interface\n");
 
@@ -157,7 +157,7 @@ int ipoib_open(struct net_device *dev)
 
 static int ipoib_stop(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	ipoib_dbg(priv, "stopping interface\n");
 
@@ -195,7 +195,7 @@ static void ipoib_uninit(struct net_device *dev)
 
 static netdev_features_t ipoib_fix_features(struct net_device *dev, netdev_features_t features)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	if (test_bit(IPOIB_FLAG_ADMIN_CM, &priv->flags))
 		features &= ~(NETIF_F_IP_CSUM | NETIF_F_TSO);
@@ -205,7 +205,7 @@ static netdev_features_t ipoib_fix_features(struct net_device *dev, netdev_featu
 
 static int ipoib_change_mtu(struct net_device *dev, int new_mtu)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	/* dev->mtu > 2K ==> connected mode */
 	if (ipoib_cm_admin_enabled(dev)) {
@@ -468,7 +468,7 @@ static struct net_device *ipoib_get_net_dev_by_params(
 
 int ipoib_set_mode(struct net_device *dev, const char *buf)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	if ((test_bit(IPOIB_FLAG_ADMIN_CM, &priv->flags) &&
 	     !strcmp(buf, "connected\n")) ||
@@ -505,7 +505,7 @@ int ipoib_set_mode(struct net_device *dev, const char *buf)
 
 struct ipoib_path *__path_find(struct net_device *dev, void *gid)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct rb_node *n = priv->path_tree.rb_node;
 	struct ipoib_path *path;
 	int ret;
@@ -529,7 +529,7 @@ struct ipoib_path *__path_find(struct net_device *dev, void *gid)
 
 static int __path_add(struct net_device *dev, struct ipoib_path *path)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct rb_node **n = &priv->path_tree.rb_node;
 	struct rb_node *pn = NULL;
 	struct ipoib_path *tpath;
@@ -564,7 +564,7 @@ static void path_free(struct net_device *dev, struct ipoib_path *path)
 	while ((skb = __skb_dequeue(&path->queue)))
 		dev_kfree_skb_irq(skb);
 
-	ipoib_dbg(netdev_priv(dev), "path_free\n");
+	ipoib_dbg(ipoib_priv(dev), "path_free\n");
 
 	/* remove all neigh connected to this path */
 	ipoib_del_neighs_by_gid(dev, path->pathrec.dgid.raw);
@@ -598,7 +598,7 @@ struct ipoib_path_iter *ipoib_path_iter_init(struct net_device *dev)
 
 int ipoib_path_iter_next(struct ipoib_path_iter *iter)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(iter->dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(iter->dev);
 	struct rb_node *n;
 	struct ipoib_path *path;
 	int ret = 1;
@@ -635,7 +635,7 @@ void ipoib_path_iter_read(struct ipoib_path_iter *iter,
 
 void ipoib_mark_paths_invalid(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_path *path, *tp;
 
 	spin_lock_irq(&priv->lock);
@@ -731,7 +731,7 @@ static void push_pseudo_header(struct sk_buff *skb, const char *daddr)
 
 void ipoib_flush_paths(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_path *path, *tp;
 	LIST_HEAD(remove_list);
 	unsigned long flags;
@@ -765,7 +765,7 @@ static void path_rec_completion(int status,
 {
 	struct ipoib_path *path = path_ptr;
 	struct net_device *dev = path->dev;
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_ah *ah = NULL;
 	struct ipoib_ah *old_ah = NULL;
 	struct ipoib_neigh *neigh, *tn;
@@ -858,7 +858,7 @@ static void path_rec_completion(int status,
 
 static struct ipoib_path *path_rec_create(struct net_device *dev, void *gid)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_path *path;
 
 	if (!priv->broadcast)
@@ -886,7 +886,7 @@ static struct ipoib_path *path_rec_create(struct net_device *dev, void *gid)
 static int path_rec_start(struct net_device *dev,
 			  struct ipoib_path *path)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	ipoib_dbg(priv, "Start path record lookup for %pI6\n",
 		  path->pathrec.dgid.raw);
@@ -1144,7 +1144,7 @@ static int ipoib_start_xmit(struct sk_buff *skb, struct net_device *dev)
 
 static void ipoib_timeout(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	ipoib_warn(priv, "transmit timeout: latency %d msecs\n",
 		   jiffies_to_msecs(jiffies - dev_trans_start(dev)));
@@ -1178,7 +1178,7 @@ static int ipoib_hard_header(struct sk_buff *skb,
 
 static void ipoib_set_mcast_list(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	if (!test_bit(IPOIB_FLAG_OPER_UP, &priv->flags)) {
 		ipoib_dbg(priv, "IPOIB_FLAG_OPER_UP not set");
@@ -1190,7 +1190,7 @@ static void ipoib_set_mcast_list(struct net_device *dev)
 
 static int ipoib_get_iflink(const struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	/* parent interface */
 	if (!test_bit(IPOIB_FLAG_SUBINTERFACE, &priv->flags))
@@ -1218,7 +1218,7 @@ static u32 ipoib_addr_hash(struct ipoib_neigh_hash *htbl, u8 *daddr)
 
 struct ipoib_neigh *ipoib_neigh_get(struct net_device *dev, u8 *daddr)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_neigh_table *ntbl = &priv->ntbl;
 	struct ipoib_neigh_hash *htbl;
 	struct ipoib_neigh *neigh = NULL;
@@ -1347,7 +1347,7 @@ static struct ipoib_neigh *ipoib_neigh_ctor(u8 *daddr,
 struct ipoib_neigh *ipoib_neigh_alloc(u8 *daddr,
 				      struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_neigh_table *ntbl = &priv->ntbl;
 	struct ipoib_neigh_hash *htbl;
 	struct ipoib_neigh *neigh;
@@ -1404,7 +1404,7 @@ void ipoib_neigh_dtor(struct ipoib_neigh *neigh)
 {
 	/* neigh reference count was dropprd to zero */
 	struct net_device *dev = neigh->dev;
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct sk_buff *skb;
 	if (neigh->ah)
 		ipoib_put_ah(neigh->ah);
@@ -1414,7 +1414,7 @@ void ipoib_neigh_dtor(struct ipoib_neigh *neigh)
 	}
 	if (ipoib_cm_get(neigh))
 		ipoib_cm_destroy_tx(ipoib_cm_get(neigh));
-	ipoib_dbg(netdev_priv(dev),
+	ipoib_dbg(ipoib_priv(dev),
 		  "neigh free for %06x %pI6\n",
 		  IPOIB_QPN(neigh->daddr),
 		  neigh->daddr + 4);
@@ -1436,7 +1436,7 @@ static void ipoib_neigh_reclaim(struct rcu_head *rp)
 void ipoib_neigh_free(struct ipoib_neigh *neigh)
 {
 	struct net_device *dev = neigh->dev;
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_neigh_table *ntbl = &priv->ntbl;
 	struct ipoib_neigh_hash *htbl;
 	struct ipoib_neigh __rcu **np;
@@ -1519,7 +1519,7 @@ static void neigh_hash_free_rcu(struct rcu_head *head)
 
 void ipoib_del_neighs_by_gid(struct net_device *dev, u8 *gid)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_neigh_table *ntbl = &priv->ntbl;
 	struct ipoib_neigh_hash *htbl;
 	unsigned long flags;
@@ -1605,7 +1605,7 @@ static void ipoib_flush_neighs(struct ipoib_dev_priv *priv)
 
 static void ipoib_neigh_hash_uninit(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int stopped;
 
 	ipoib_dbg(priv, "ipoib_neigh_hash_uninit\n");
@@ -1624,7 +1624,7 @@ static void ipoib_neigh_hash_uninit(struct net_device *dev)
 
 void ipoib_dev_uninit_default(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	ipoib_transport_dev_cleanup(dev);
 
@@ -1750,7 +1750,7 @@ int ipoib_dev_init(struct net_device *dev, struct ib_device *ca, int port)
 
 void ipoib_dev_cleanup(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev), *cpriv, *tcpriv;
+	struct ipoib_dev_priv *priv = ipoib_priv(dev), *cpriv, *tcpriv;
 	LIST_HEAD(head);
 
 	ASSERT_RTNL();
@@ -1780,7 +1780,7 @@ void ipoib_dev_cleanup(struct net_device *dev)
 
 static int ipoib_set_vf_link_state(struct net_device *dev, int vf, int link_state)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	return ib_set_vf_link_state(priv->ca, vf, priv->port, link_state);
 }
@@ -1788,7 +1788,7 @@ static int ipoib_set_vf_link_state(struct net_device *dev, int vf, int link_stat
 static int ipoib_get_vf_config(struct net_device *dev, int vf,
 			       struct ifla_vf_info *ivf)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int err;
 
 	err = ib_get_vf_config(priv->ca, vf, priv->port, ivf);
@@ -1802,7 +1802,7 @@ static int ipoib_get_vf_config(struct net_device *dev, int vf,
 
 static int ipoib_set_vf_guid(struct net_device *dev, int vf, u64 guid, int type)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	if (type != IFLA_VF_IB_NODE_GUID && type != IFLA_VF_IB_PORT_GUID)
 		return -EINVAL;
@@ -1813,7 +1813,7 @@ static int ipoib_set_vf_guid(struct net_device *dev, int vf, u64 guid, int type)
 static int ipoib_get_vf_stats(struct net_device *dev, int vf,
 			      struct ifla_vf_stats *vf_stats)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	return ib_get_vf_stats(priv->ca, vf, priv->port, vf_stats);
 }
@@ -1925,14 +1925,14 @@ static DEVICE_ATTR(pkey, S_IRUGO, show_pkey, NULL);
 static ssize_t show_umcast(struct device *dev,
 			   struct device_attribute *attr, char *buf)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(to_net_dev(dev));
+	struct ipoib_dev_priv *priv = ipoib_priv(to_net_dev(dev));
 
 	return sprintf(buf, "%d\n", test_bit(IPOIB_FLAG_UMCAST, &priv->flags));
 }
 
 void ipoib_set_umcast(struct net_device *ndev, int umcast_val)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(ndev);
+	struct ipoib_dev_priv *priv = ipoib_priv(ndev);
 
 	if (umcast_val > 0) {
 		set_bit(IPOIB_FLAG_UMCAST, &priv->flags);
@@ -2005,7 +2005,7 @@ static int ipoib_check_lladdr(struct net_device *dev,
 
 static int ipoib_set_mac(struct net_device *dev, void *addr)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct sockaddr_storage *ss = addr;
 	int ret;
 
@@ -2224,7 +2224,7 @@ static void ipoib_add_one(struct ib_device *device)
 			continue;
 		dev = ipoib_add_port("ib%d", device, p);
 		if (!IS_ERR(dev)) {
-			priv = netdev_priv(dev);
+			priv = ipoib_priv(dev);
 			list_add_tail(&priv->list, dev_list);
 			count++;
 		}
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
index 69e146cdc306..6707fd703dca 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
@@ -114,7 +114,7 @@ static void ipoib_mcast_free(struct ipoib_mcast *mcast)
 	struct net_device *dev = mcast->dev;
 	int tx_dropped = 0;
 
-	ipoib_dbg_mcast(netdev_priv(dev), "deleting multicast group %pI6\n",
+	ipoib_dbg_mcast(ipoib_priv(dev), "deleting multicast group %pI6\n",
 			mcast->mcmember.mgid.raw);
 
 	/* remove all neigh connected to this mcast */
@@ -158,7 +158,7 @@ static struct ipoib_mcast *ipoib_mcast_alloc(struct net_device *dev,
 
 static struct ipoib_mcast *__ipoib_mcast_find(struct net_device *dev, void *mgid)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct rb_node *n = priv->multicast_tree.rb_node;
 
 	while (n) {
@@ -182,7 +182,7 @@ static struct ipoib_mcast *__ipoib_mcast_find(struct net_device *dev, void *mgid
 
 static int __ipoib_mcast_add(struct net_device *dev, struct ipoib_mcast *mcast)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct rb_node **n = &priv->multicast_tree.rb_node, *pn = NULL;
 
 	while (*n) {
@@ -375,7 +375,7 @@ static int ipoib_mcast_join_complete(int status,
 {
 	struct ipoib_mcast *mcast = multicast->context;
 	struct net_device *dev = mcast->dev;
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	ipoib_dbg_mcast(priv, "%sjoin completion for %pI6 (status %d)\n",
 			test_bit(IPOIB_MCAST_FLAG_SENDONLY, &mcast->flags) ?
@@ -477,7 +477,7 @@ static int ipoib_mcast_join_complete(int status,
  */
 static int ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_sa_multicast *multicast;
 	struct ib_sa_mcmember_rec rec = {
 		.join_state = 1
@@ -678,7 +678,7 @@ void ipoib_mcast_join_task(struct work_struct *work)
 
 void ipoib_mcast_start_thread(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	unsigned long flags;
 
 	ipoib_dbg_mcast(priv, "starting multicast thread\n");
@@ -690,7 +690,7 @@ void ipoib_mcast_start_thread(struct net_device *dev)
 
 int ipoib_mcast_stop_thread(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	unsigned long flags;
 
 	ipoib_dbg_mcast(priv, "stopping multicast thread\n");
@@ -837,7 +837,7 @@ void ipoib_mcast_send(struct net_device *dev, u8 *daddr, struct sk_buff *skb)
 
 void ipoib_mcast_dev_flush(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	LIST_HEAD(remove_list);
 	struct ipoib_mcast *mcast, *tmcast;
 	unsigned long flags;
@@ -1029,7 +1029,7 @@ struct ipoib_mcast_iter *ipoib_mcast_iter_init(struct net_device *dev)
 
 int ipoib_mcast_iter_next(struct ipoib_mcast_iter *iter)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(iter->dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(iter->dev);
 	struct rb_node *n;
 	struct ipoib_mcast *mcast;
 	int ret = 1;
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_netlink.c b/drivers/infiniband/ulp/ipoib/ipoib_netlink.c
index cdc7df4fdb8a..406e7b039b79 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_netlink.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_netlink.c
@@ -44,7 +44,7 @@ static const struct nla_policy ipoib_policy[IFLA_IPOIB_MAX + 1] = {
 
 static int ipoib_fill_info(struct sk_buff *skb, const struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	u16 val;
 
 	if (nla_put_u16(skb, IFLA_IPOIB_PKEY, priv->pkey))
@@ -107,7 +107,7 @@ static int ipoib_new_child_link(struct net *src_net, struct net_device *dev,
 	if (!pdev || pdev->type != ARPHRD_INFINIBAND)
 		return -ENODEV;
 
-	ppriv = netdev_priv(pdev);
+	ppriv = ipoib_priv(pdev);
 
 	if (test_bit(IPOIB_FLAG_SUBINTERFACE, &ppriv->flags)) {
 		ipoib_warn(ppriv, "child creation disallowed for child devices\n");
@@ -129,7 +129,7 @@ static int ipoib_new_child_link(struct net *src_net, struct net_device *dev,
 	 */
 	child_pkey |= 0x8000;
 
-	err = __ipoib_vlan_add(ppriv, netdev_priv(dev), child_pkey, IPOIB_RTNL_CHILD);
+	err = __ipoib_vlan_add(ppriv, ipoib_priv(dev), child_pkey, IPOIB_RTNL_CHILD);
 
 	if (!err && data)
 		err = ipoib_changelink(dev, tb, data);
@@ -140,8 +140,8 @@ static void ipoib_unregister_child_dev(struct net_device *dev, struct list_head
 {
 	struct ipoib_dev_priv *priv, *ppriv;
 
-	priv = netdev_priv(dev);
-	ppriv = netdev_priv(priv->parent);
+	priv = ipoib_priv(dev);
+	ppriv = ipoib_priv(priv->parent);
 
 	down_write(&ppriv->vlan_rwsem);
 	unregister_netdevice_queue(dev, head);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
index 529b9509a2c4..e66e7081644d 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
@@ -130,7 +130,7 @@ int ipoib_init_qp(struct net_device *dev)
 
 int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_qp_init_attr init_attr = {
 		.cap = {
 			.max_send_wr  = ipoib_sendq_size,
@@ -232,7 +232,7 @@ int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca)
 
 void ipoib_transport_dev_cleanup(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	if (priv->qp) {
 		if (ib_destroy_qp(priv->qp))
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_vlan.c b/drivers/infiniband/ulp/ipoib/ipoib_vlan.c
index 3e10e3dac2e7..b4de1737986d 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_vlan.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_vlan.c
@@ -44,7 +44,7 @@ static ssize_t show_parent(struct device *d, struct device_attribute *attr,
 			   char *buf)
 {
 	struct net_device *dev = to_net_dev(d);
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
 	return sprintf(buf, "%s\n", priv->parent->name);
 }
@@ -183,7 +183,7 @@ int ipoib_vlan_delete(struct net_device *pdev, unsigned short pkey)
 	if (!capable(CAP_NET_ADMIN))
 		return -EPERM;
 
-	ppriv = netdev_priv(pdev);
+	ppriv = ipoib_priv(pdev);
 
 	if (test_bit(IPOIB_FLAG_GOING_DOWN, &ppriv->flags))
 		return -EPERM;
-- 
2.12.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH rdma-next 6/6] IB/IPoIB: Support acceleration options callbacks

[Leon Romanovsky]

From: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>

IPoIB driver now uses the new set of callback functions.

If the hardware provider supports the new ipoib_options implementation,
the driver uses the callbacks in its data path flows, otherwise it uses the
driver default implementation for all data flows in its code.

The default implementation wasn't change and it is exactly as it was before
introduction of acceleration support.

Signed-off-by: Erez Shitrit <erezsh-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
Signed-off-by: Leon Romanovsky <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
---
 drivers/infiniband/ulp/ipoib/ipoib.h           |  28 +++--
 drivers/infiniband/ulp/ipoib/ipoib_ib.c        |  57 +++++----
 drivers/infiniband/ulp/ipoib/ipoib_main.c      | 154 ++++++++++++++++++-------
 drivers/infiniband/ulp/ipoib/ipoib_multicast.c |  21 ++--
 drivers/infiniband/ulp/ipoib/ipoib_netlink.c   |   2 +-
 drivers/infiniband/ulp/ipoib/ipoib_verbs.c     |  20 +++-
 drivers/infiniband/ulp/ipoib/ipoib_vlan.c      |   5 +-
 7 files changed, 198 insertions(+), 89 deletions(-)

diff --git a/drivers/infiniband/ulp/ipoib/ipoib.h b/drivers/infiniband/ulp/ipoib/ipoib.h
index 055adacd6d06..addbdd88bb32 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib.h
+++ b/drivers/infiniband/ulp/ipoib/ipoib.h
@@ -52,7 +52,6 @@
 #include <rdma/ib_pack.h>
 #include <rdma/ib_sa.h>
 #include <linux/sched.h>
-
 /* constants */
 
 enum ipoib_flush_level {
@@ -153,8 +152,12 @@ static inline void skb_add_pseudo_hdr(struct sk_buff *skb)
 	skb_pull(skb, IPOIB_HARD_LEN);
 }
 
-/* Keep the refactoring compile able */
-#define ipoib_priv netdev_priv
+static inline void *ipoib_priv(const struct net_device *dev)
+{
+	struct rdma_netdev *rn = netdev_priv(dev);
+
+	return rn->clnt_priv;
+}
 
 /* Used for all multicast joins (broadcast, IPv4 mcast and IPv6 mcast) */
 struct ipoib_mcast {
@@ -407,6 +410,7 @@ struct ipoib_dev_priv {
 	struct timer_list poll_timer;
 	unsigned max_send_sge;
 	bool sm_fullmember_sendonly_support;
+	const struct net_device_ops	*rn_ops;
 };
 
 struct ipoib_ah {
@@ -485,23 +489,25 @@ int ipoib_open(struct net_device *dev);
 int ipoib_add_pkey_attr(struct net_device *dev);
 int ipoib_add_umcast_attr(struct net_device *dev);
 
-void ipoib_send(struct net_device *dev, struct sk_buff *skb,
-		struct ipoib_ah *address, u32 dqpn);
+int ipoib_send(struct net_device *dev, struct sk_buff *skb,
+	       struct ib_ah *address, u32 dqpn);
 void ipoib_reap_ah(struct work_struct *work);
 
 struct ipoib_path *__path_find(struct net_device *dev, void *gid);
 void ipoib_mark_paths_invalid(struct net_device *dev);
 void ipoib_flush_paths(struct net_device *dev);
 int ipoib_check_sm_sendonly_fullmember_support(struct ipoib_dev_priv *priv);
-struct ipoib_dev_priv *ipoib_intf_alloc(const char *format);
+struct ipoib_dev_priv *ipoib_intf_alloc(struct ib_device *hca, u8 port,
+					const char *format);
 void ipoib_ib_tx_timer_func(unsigned long ctx);
 void ipoib_ib_dev_flush_light(struct work_struct *work);
 void ipoib_ib_dev_flush_normal(struct work_struct *work);
 void ipoib_ib_dev_flush_heavy(struct work_struct *work);
 void ipoib_pkey_event(struct work_struct *work);
 void ipoib_ib_dev_cleanup(struct net_device *dev);
-void ipoib_dev_uninit_default(struct net_device *dev);
+int ipoib_ib_dev_open_default(struct net_device *dev);
 int ipoib_ib_dev_open(struct net_device *dev);
+int ipoib_ib_dev_stop(struct net_device *dev);
 void ipoib_ib_dev_up(struct net_device *dev);
 void ipoib_ib_dev_down(struct net_device *dev);
 int ipoib_ib_dev_stop_default(struct net_device *dev);
@@ -564,8 +570,10 @@ void ipoib_path_iter_read(struct ipoib_path_iter *iter,
 			  struct ipoib_path *path);
 #endif
 
-int ipoib_mcast_attach(struct net_device *dev, u16 mlid,
-		       union ib_gid *mgid, int set_qkey);
+int ipoib_mcast_attach(struct net_device *dev, struct ib_device *hca,
+		       union ib_gid *mgid, u16 mlid, int set_qkey, u32 qkey);
+int ipoib_mcast_detach(struct net_device *dev, struct ib_device *hca,
+		       union ib_gid *mgid, u16 mlid);
 void ipoib_mcast_remove_list(struct list_head *remove_list);
 void ipoib_check_and_add_mcast_sendonly(struct ipoib_dev_priv *priv, u8 *mgid,
 				struct list_head *remove_list);
@@ -589,7 +597,7 @@ void __exit ipoib_netlink_fini(void);
 void ipoib_set_umcast(struct net_device *ndev, int umcast_val);
 int  ipoib_set_mode(struct net_device *dev, const char *buf);
 
-void ipoib_setup(struct net_device *dev);
+void ipoib_setup_common(struct net_device *dev);
 
 void ipoib_pkey_open(struct ipoib_dev_priv *priv);
 void ipoib_drain_cq(struct net_device *dev);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index 374533ce82fd..997a9dba748a 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -537,10 +537,10 @@ static inline int post_send(struct ipoib_dev_priv *priv,
 	return ib_post_send(priv->qp, &priv->tx_wr.wr, &bad_wr);
 }
 
-void ipoib_send(struct net_device *dev, struct sk_buff *skb,
-		struct ipoib_ah *address, u32 dqpn)
+int ipoib_send(struct net_device *dev, struct sk_buff *skb,
+	       struct ib_ah *address, u32 dqpn)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_tx_buf *tx_req;
 	int hlen, rc;
 	void *phead;
@@ -554,7 +554,7 @@ void ipoib_send(struct net_device *dev, struct sk_buff *skb,
 			++dev->stats.tx_dropped;
 			++dev->stats.tx_errors;
 			dev_kfree_skb_any(skb);
-			return;
+			return -1;
 		}
 	} else {
 		if (unlikely(skb->len > priv->mcast_mtu + IPOIB_ENCAP_LEN)) {
@@ -563,7 +563,7 @@ void ipoib_send(struct net_device *dev, struct sk_buff *skb,
 			++dev->stats.tx_dropped;
 			++dev->stats.tx_errors;
 			ipoib_cm_skb_too_long(dev, skb, priv->mcast_mtu);
-			return;
+			return -1;
 		}
 		phead = NULL;
 		hlen  = 0;
@@ -574,7 +574,7 @@ void ipoib_send(struct net_device *dev, struct sk_buff *skb,
 			++dev->stats.tx_dropped;
 			++dev->stats.tx_errors;
 			dev_kfree_skb_any(skb);
-			return;
+			return -1;
 		}
 		/* Does skb_linearize return ok without reducing nr_frags? */
 		if (skb_shinfo(skb)->nr_frags > usable_sge) {
@@ -582,7 +582,7 @@ void ipoib_send(struct net_device *dev, struct sk_buff *skb,
 			++dev->stats.tx_dropped;
 			++dev->stats.tx_errors;
 			dev_kfree_skb_any(skb);
-			return;
+			return -1;
 		}
 	}
 
@@ -601,7 +601,7 @@ void ipoib_send(struct net_device *dev, struct sk_buff *skb,
 	if (unlikely(ipoib_dma_map_tx(priv->ca, tx_req))) {
 		++dev->stats.tx_errors;
 		dev_kfree_skb_any(skb);
-		return;
+		return -1;
 	}
 
 	if (skb->ip_summed == CHECKSUM_PARTIAL)
@@ -620,7 +620,7 @@ void ipoib_send(struct net_device *dev, struct sk_buff *skb,
 	skb_dst_drop(skb);
 
 	rc = post_send(priv, priv->tx_head & (ipoib_sendq_size - 1),
-		       address->ah, dqpn, tx_req, phead, hlen);
+		       address, dqpn, tx_req, phead, hlen);
 	if (unlikely(rc)) {
 		ipoib_warn(priv, "post_send failed, error %d\n", rc);
 		++dev->stats.tx_errors;
@@ -629,21 +629,23 @@ void ipoib_send(struct net_device *dev, struct sk_buff *skb,
 		dev_kfree_skb_any(skb);
 		if (netif_queue_stopped(dev))
 			netif_wake_queue(dev);
+		rc = 0;
 	} else {
 		netif_trans_update(dev);
-
-		address->last_send = priv->tx_head;
+		rc = priv->tx_head;
 		++priv->tx_head;
 	}
 
 	if (unlikely(priv->tx_outstanding > MAX_SEND_CQE))
 		while (poll_tx(priv))
 			; /* nothing */
+
+	return rc;
 }
 
 static void __ipoib_reap_ah(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ipoib_ah *ah, *tah;
 	LIST_HEAD(remove_list);
 	unsigned long flags;
@@ -713,7 +715,7 @@ int ipoib_ib_dev_stop_default(struct net_device *dev)
 	struct ipoib_tx_buf *tx_req;
 	int i;
 
-	if (test_and_clear_bit(IPOIB_FLAG_INITIALIZED, &priv->flags))
+	if (test_bit(IPOIB_FLAG_INITIALIZED, &priv->flags))
 		napi_disable(&priv->napi);
 
 	ipoib_cm_dev_stop(dev);
@@ -782,8 +784,11 @@ int ipoib_ib_dev_stop_default(struct net_device *dev)
 
 int ipoib_ib_dev_stop(struct net_device *dev)
 {
-	ipoib_ib_dev_stop_default(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
+
+	priv->rn_ops->ndo_stop(dev);
 
+	clear_bit(IPOIB_FLAG_INITIALIZED, &priv->flags);
 	ipoib_flush_ah(dev);
 
 	return 0;
@@ -808,23 +813,20 @@ int ipoib_ib_dev_open_default(struct net_device *dev)
 	ret = ipoib_ib_post_receives(dev);
 	if (ret) {
 		ipoib_warn(priv, "ipoib_ib_post_receives returned %d\n", ret);
-		goto dev_stop;
+		goto out;
 	}
 
 	ret = ipoib_cm_dev_open(dev);
 	if (ret) {
 		ipoib_warn(priv, "ipoib_cm_dev_open returned %d\n", ret);
-		goto dev_stop;
+		goto out;
 	}
 
-	if (!test_and_set_bit(IPOIB_FLAG_INITIALIZED, &priv->flags))
+	if (!test_bit(IPOIB_FLAG_INITIALIZED, &priv->flags))
 		napi_enable(&priv->napi);
 
 	return 0;
-dev_stop:
-	if (!test_and_set_bit(IPOIB_FLAG_INITIALIZED, &priv->flags))
-		napi_enable(&priv->napi);
-	ipoib_ib_dev_stop(dev);
+out:
 	return -1;
 }
 
@@ -844,16 +846,21 @@ int ipoib_ib_dev_open(struct net_device *dev)
 	queue_delayed_work(priv->wq, &priv->ah_reap_task,
 			   round_jiffies_relative(HZ));
 
-	if (ipoib_ib_dev_open_default(dev)) {
+	if (priv->rn_ops->ndo_open(dev)) {
 		pr_warn("%s: Failed to open dev\n", dev->name);
-		goto stop_ah_reap;
+		goto dev_stop;
 	}
 
+	set_bit(IPOIB_FLAG_INITIALIZED, &priv->flags);
+
 	return 0;
 
-stop_ah_reap:
+dev_stop:
 	set_bit(IPOIB_STOP_REAPER, &priv->flags);
 	cancel_delayed_work(&priv->ah_reap_task);
+	set_bit(IPOIB_FLAG_INITIALIZED, &priv->flags);
+	napi_enable(&priv->napi);
+	ipoib_ib_dev_stop(dev);
 	return -1;
 }
 
@@ -1238,7 +1245,7 @@ void ipoib_ib_dev_cleanup(struct net_device *dev)
 
 	clear_bit(IPOIB_PKEY_ASSIGNED, &priv->flags);
 
-	ipoib_dev_uninit_default(dev);
+	priv->rn_ops->ndo_uninit(dev);
 
 	if (priv->pd) {
 		ib_dealloc_pd(priv->pd);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c
index a88df83677b1..0b1186964738 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -166,7 +166,7 @@ static int ipoib_stop(struct net_device *dev)
 	netif_stop_queue(dev);
 
 	ipoib_ib_dev_down(dev);
-	ipoib_ib_dev_stop_default(dev);
+	ipoib_ib_dev_stop(dev);
 
 	if (!test_bit(IPOIB_FLAG_SUBINTERFACE, &priv->flags)) {
 		struct ipoib_dev_priv *cpriv;
@@ -917,7 +917,8 @@ static int path_rec_start(struct net_device *dev,
 static void neigh_add_path(struct sk_buff *skb, u8 *daddr,
 			   struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
+	struct rdma_netdev *rn = netdev_priv(dev);
 	struct ipoib_path *path;
 	struct ipoib_neigh *neigh;
 	unsigned long flags;
@@ -964,7 +965,8 @@ static void neigh_add_path(struct sk_buff *skb, u8 *daddr,
 			}
 		} else {
 			spin_unlock_irqrestore(&priv->lock, flags);
-			ipoib_send(dev, skb, path->ah, IPOIB_QPN(daddr));
+			path->ah->last_send = rn->send(dev, skb, path->ah->ah,
+						       IPOIB_QPN(daddr));
 			ipoib_neigh_put(neigh);
 			return;
 		}
@@ -998,7 +1000,8 @@ static void neigh_add_path(struct sk_buff *skb, u8 *daddr,
 static void unicast_arp_send(struct sk_buff *skb, struct net_device *dev,
 			     struct ipoib_pseudo_header *phdr)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
+	struct rdma_netdev *rn = netdev_priv(dev);
 	struct ipoib_path *path;
 	unsigned long flags;
 
@@ -1042,7 +1045,8 @@ static void unicast_arp_send(struct sk_buff *skb, struct net_device *dev,
 			  be16_to_cpu(path->pathrec.dlid));
 
 		spin_unlock_irqrestore(&priv->lock, flags);
-		ipoib_send(dev, skb, path->ah, IPOIB_QPN(phdr->hwaddr));
+		path->ah->last_send = rn->send(dev, skb, path->ah->ah,
+					       IPOIB_QPN(phdr->hwaddr));
 		return;
 	} else if ((path->query || !path_rec_start(dev, path)) &&
 		   skb_queue_len(&path->queue) < IPOIB_MAX_PATH_REC_QUEUE) {
@@ -1058,7 +1062,8 @@ static void unicast_arp_send(struct sk_buff *skb, struct net_device *dev,
 
 static int ipoib_start_xmit(struct sk_buff *skb, struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
+	struct rdma_netdev *rn = netdev_priv(dev);
 	struct ipoib_neigh *neigh;
 	struct ipoib_pseudo_header *phdr;
 	struct ipoib_header *header;
@@ -1122,7 +1127,8 @@ static int ipoib_start_xmit(struct sk_buff *skb, struct net_device *dev)
 			goto unref;
 		}
 	} else if (neigh->ah) {
-		ipoib_send(dev, skb, neigh->ah, IPOIB_QPN(phdr->hwaddr));
+		neigh->ah->last_send = rn->send(dev, skb, neigh->ah->ah,
+						IPOIB_QPN(phdr->hwaddr));
 		goto unref;
 	}
 
@@ -1637,10 +1643,11 @@ void ipoib_dev_uninit_default(struct net_device *dev)
 	priv->tx_ring = NULL;
 }
 
-static int ipoib_dev_init_default(struct net_device *dev, struct ib_device *ca,
-				  int port)
+static int ipoib_dev_init_default(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
+
+	netif_napi_add(dev, &priv->napi, ipoib_poll, NAPI_POLL_WEIGHT);
 
 	/* Allocate RX/TX "rings" to hold queued skbs */
 	priv->rx_ring =	kzalloc(ipoib_recvq_size * sizeof *priv->rx_ring,
@@ -1651,17 +1658,22 @@ static int ipoib_dev_init_default(struct net_device *dev, struct ib_device *ca,
 	priv->tx_ring = vzalloc(ipoib_sendq_size * sizeof *priv->tx_ring);
 	if (!priv->tx_ring) {
 		printk(KERN_WARNING "%s: failed to allocate TX ring (%d entries)\n",
-		       ca->name, ipoib_sendq_size);
+		       priv->ca->name, ipoib_sendq_size);
 		goto out_rx_ring_cleanup;
 	}
 
 	/* priv->tx_head, tx_tail & tx_outstanding are already 0 */
 
-	if (ipoib_transport_dev_init(dev, ca)) {
-		pr_warn("%s: ipoib_transport_dev_init failed\n", ca->name);
+	if (ipoib_transport_dev_init(dev, priv->ca)) {
+		pr_warn("%s: ipoib_transport_dev_init failed\n", priv->ca->name);
 		goto out_tx_ring_cleanup;
 	}
 
+	/* after qp created set dev address */
+	priv->dev->dev_addr[1] = (priv->qp->qp_num >> 16) & 0xff;
+	priv->dev->dev_addr[2] = (priv->qp->qp_num >>  8) & 0xff;
+	priv->dev->dev_addr[3] = (priv->qp->qp_num) & 0xff;
+
 	setup_timer(&priv->poll_timer, ipoib_ib_tx_timer_func,
 		    (unsigned long)dev);
 
@@ -1679,7 +1691,7 @@ static int ipoib_dev_init_default(struct net_device *dev, struct ib_device *ca,
 
 int ipoib_dev_init(struct net_device *dev, struct ib_device *ca, int port)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int ret = -ENOMEM;
 
 	priv->ca = ca;
@@ -1703,17 +1715,12 @@ int ipoib_dev_init(struct net_device *dev, struct ib_device *ca, int port)
 		goto clean_wq;
 	}
 
-	ret = ipoib_dev_init_default(dev, ca, port);
+	ret = priv->rn_ops->ndo_init(dev);
 	if (ret) {
 		pr_warn("%s failed to init HW resource\n", dev->name);
 		goto out_free_pd;
 	}
 
-	/* after qp created set dev address */
-	priv->dev->dev_addr[1] = (priv->qp->qp_num >> 16) & 0xff;
-	priv->dev->dev_addr[2] = (priv->qp->qp_num >>  8) & 0xff;
-	priv->dev->dev_addr[3] = (priv->qp->qp_num) & 0xff;
-
 	if (ipoib_neigh_hash_init(priv) < 0) {
 		pr_warn("%s failed to init neigh hash\n", dev->name);
 		goto out_dev_uninit;
@@ -1851,21 +1858,12 @@ static const struct net_device_ops ipoib_netdev_ops_vf = {
 	.ndo_get_iflink		 = ipoib_get_iflink,
 };
 
-void ipoib_setup(struct net_device *dev)
+void ipoib_setup_common(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
-
-	if (priv->hca_caps & IB_DEVICE_VIRTUAL_FUNCTION)
-		dev->netdev_ops	= &ipoib_netdev_ops_vf;
-	else
-		dev->netdev_ops	= &ipoib_netdev_ops_pf;
-
 	dev->header_ops		 = &ipoib_header_ops;
 
 	ipoib_set_ethtool_ops(dev);
 
-	netif_napi_add(dev, &priv->napi, ipoib_poll, NAPI_POLL_WEIGHT);
-
 	dev->watchdog_timeo	 = HZ;
 
 	dev->flags		|= IFF_BROADCAST | IFF_MULTICAST;
@@ -1879,11 +1877,14 @@ void ipoib_setup(struct net_device *dev)
 	netif_keep_dst(dev);
 
 	memcpy(dev->broadcast, ipv4_bcast_addr, INFINIBAND_ALEN);
+}
 
-	priv->dev = dev;
+void ipoib_build_priv(struct net_device *dev)
+{
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 
+	priv->dev = dev;
 	spin_lock_init(&priv->lock);
-
 	init_rwsem(&priv->vlan_rwsem);
 
 	INIT_LIST_HEAD(&priv->path_list);
@@ -1901,22 +1902,96 @@ void ipoib_setup(struct net_device *dev)
 	INIT_DELAYED_WORK(&priv->neigh_reap_task, ipoib_reap_neigh);
 }
 
-struct ipoib_dev_priv *ipoib_intf_alloc(const char *name)
+static const struct net_device_ops ipoib_netdev_default_pf = {
+	.ndo_init		 = ipoib_dev_init_default,
+	.ndo_uninit		 = ipoib_dev_uninit_default,
+	.ndo_open		 = ipoib_ib_dev_open_default,
+	.ndo_stop		 = ipoib_ib_dev_stop_default,
+};
+
+static struct net_device *ipoib_create_netdev_default(struct ib_device *hca,
+						      const char *name,
+						      unsigned char name_assign_type,
+						      void (*setup)(struct net_device *))
 {
 	struct net_device *dev;
+	struct rdma_netdev *rn;
 
-	dev = alloc_netdev((int)sizeof(struct ipoib_dev_priv), name,
-			   NET_NAME_UNKNOWN, ipoib_setup);
+	dev = alloc_netdev((int)sizeof(struct rdma_netdev),
+			   name,
+			   name_assign_type, setup);
 	if (!dev)
 		return NULL;
 
-	return netdev_priv(dev);
+	rn = netdev_priv(dev);
+
+	rn->send = ipoib_send;
+	rn->attach_mcast = ipoib_mcast_attach;
+	rn->detach_mcast = ipoib_mcast_detach;
+
+	dev->netdev_ops = &ipoib_netdev_default_pf;
+
+	return dev;
+}
+
+static struct net_device *ipoib_get_netdev(struct ib_device *hca, u8 port,
+					   const char *name)
+{
+	struct net_device *dev;
+
+	if (hca->alloc_rdma_netdev) {
+		dev = hca->alloc_rdma_netdev(hca, port,
+					     RDMA_NETDEV_IPOIB, name,
+					     NET_NAME_UNKNOWN,
+					     ipoib_setup_common);
+		if (IS_ERR_OR_NULL(dev) && PTR_ERR(dev) != ENOTSUPP)
+			return NULL;
+	}
+
+	if (!hca->alloc_rdma_netdev || PTR_ERR(dev) == ENOTSUPP)
+		dev = ipoib_create_netdev_default(hca, name, NET_NAME_UNKNOWN,
+						  ipoib_setup_common);
+
+	return dev;
+}
+
+struct ipoib_dev_priv *ipoib_intf_alloc(struct ib_device *hca, u8 port,
+					const char *name)
+{
+	struct net_device *dev;
+	struct ipoib_dev_priv *priv;
+	struct rdma_netdev *rn;
+
+	priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+	if (!priv)
+		return NULL;
+
+	dev = ipoib_get_netdev(hca, port, name);
+	if (!dev)
+		goto free_priv;
+
+	priv->rn_ops = dev->netdev_ops;
+
+	/* fixme : should be after the query_cap */
+	if (priv->hca_caps & IB_DEVICE_VIRTUAL_FUNCTION)
+		dev->netdev_ops	= &ipoib_netdev_ops_vf;
+	else
+		dev->netdev_ops	= &ipoib_netdev_ops_pf;
+
+	rn = netdev_priv(dev);
+	rn->clnt_priv = priv;
+	ipoib_build_priv(dev);
+
+	return priv;
+free_priv:
+	kfree(priv);
+	return NULL;
 }
 
 static ssize_t show_pkey(struct device *dev,
 			 struct device_attribute *attr, char *buf)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(to_net_dev(dev));
+	struct ipoib_dev_priv *priv = ipoib_priv(to_net_dev(dev));
 
 	return sprintf(buf, "0x%04x\n", priv->pkey);
 }
@@ -2078,7 +2153,7 @@ void ipoib_set_dev_features(struct ipoib_dev_priv *priv, struct ib_device *hca)
 	priv->hca_caps = hca->attrs.device_cap_flags;
 
 	if (priv->hca_caps & IB_DEVICE_UD_IP_CSUM) {
-		priv->dev->hw_features = NETIF_F_IP_CSUM | NETIF_F_RXCSUM;
+		priv->dev->hw_features |= NETIF_F_IP_CSUM | NETIF_F_RXCSUM;
 
 		if (priv->hca_caps & IB_DEVICE_UD_TSO)
 			priv->dev->hw_features |= NETIF_F_TSO;
@@ -2094,7 +2169,7 @@ static struct net_device *ipoib_add_port(const char *format,
 	struct ib_port_attr attr;
 	int result = -ENOMEM;
 
-	priv = ipoib_intf_alloc(format);
+	priv = ipoib_intf_alloc(hca, port, format);
 	if (!priv)
 		goto alloc_mem_failed;
 
@@ -2264,6 +2339,7 @@ static void ipoib_remove_one(struct ib_device *device, void *client_data)
 
 		unregister_netdev(priv->dev);
 		free_netdev(priv->dev);
+		kfree(priv);
 	}
 
 	kfree(dev_list);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
index 6707fd703dca..8da1955e0003 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_multicast.c
@@ -212,7 +212,8 @@ static int ipoib_mcast_join_finish(struct ipoib_mcast *mcast,
 				   struct ib_sa_mcmember_rec *mcmember)
 {
 	struct net_device *dev = mcast->dev;
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
+	struct rdma_netdev *rn = netdev_priv(dev);
 	struct ipoib_ah *ah;
 	int ret;
 	int set_qkey = 0;
@@ -260,8 +261,9 @@ static int ipoib_mcast_join_finish(struct ipoib_mcast *mcast,
 			return 0;
 		}
 
-		ret = ipoib_mcast_attach(dev, be16_to_cpu(mcast->mcmember.mlid),
-					 &mcast->mcmember.mgid, set_qkey);
+		ret = rn->attach_mcast(dev, priv->ca, &mcast->mcmember.mgid,
+				       be16_to_cpu(mcast->mcmember.mlid),
+				       set_qkey, priv->qkey);
 		if (ret < 0) {
 			ipoib_warn(priv, "couldn't attach QP to multicast group %pI6\n",
 				   mcast->mcmember.mgid.raw);
@@ -706,7 +708,8 @@ int ipoib_mcast_stop_thread(struct net_device *dev)
 
 static int ipoib_mcast_leave(struct net_device *dev, struct ipoib_mcast *mcast)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
+	struct rdma_netdev *rn = netdev_priv(dev);
 	int ret = 0;
 
 	if (test_and_clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags))
@@ -720,8 +723,8 @@ static int ipoib_mcast_leave(struct net_device *dev, struct ipoib_mcast *mcast)
 				mcast->mcmember.mgid.raw);
 
 		/* Remove ourselves from the multicast group */
-		ret = ib_detach_mcast(priv->qp, &mcast->mcmember.mgid,
-				      be16_to_cpu(mcast->mcmember.mlid));
+		ret = rn->detach_mcast(dev, priv->ca, &mcast->mcmember.mgid,
+				       be16_to_cpu(mcast->mcmember.mlid));
 		if (ret)
 			ipoib_warn(priv, "ib_detach_mcast failed (result = %d)\n", ret);
 	} else if (!test_bit(IPOIB_MCAST_FLAG_SENDONLY, &mcast->flags))
@@ -762,7 +765,8 @@ void ipoib_mcast_remove_list(struct list_head *remove_list)
 
 void ipoib_mcast_send(struct net_device *dev, u8 *daddr, struct sk_buff *skb)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
+	struct rdma_netdev *rn = netdev_priv(dev);
 	struct ipoib_mcast *mcast;
 	unsigned long flags;
 	void *mgid = daddr + 4;
@@ -825,7 +829,8 @@ void ipoib_mcast_send(struct net_device *dev, u8 *daddr, struct sk_buff *skb)
 			}
 		}
 		spin_unlock_irqrestore(&priv->lock, flags);
-		ipoib_send(dev, skb, mcast->ah, IB_MULTICAST_QPN);
+		mcast->ah->last_send = rn->send(dev, skb, mcast->ah->ah,
+						IB_MULTICAST_QPN);
 		if (neigh)
 			ipoib_neigh_put(neigh);
 		return;
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_netlink.c b/drivers/infiniband/ulp/ipoib/ipoib_netlink.c
index 406e7b039b79..5d795aaea92c 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_netlink.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_netlink.c
@@ -161,7 +161,7 @@ static struct rtnl_link_ops ipoib_link_ops __read_mostly = {
 	.maxtype	= IFLA_IPOIB_MAX,
 	.policy		= ipoib_policy,
 	.priv_size	= sizeof(struct ipoib_dev_priv),
-	.setup		= ipoib_setup,
+	.setup		= ipoib_setup_common,
 	.newlink	= ipoib_new_child_link,
 	.changelink	= ipoib_changelink,
 	.dellink	= ipoib_unregister_child_dev,
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
index e66e7081644d..bb64baf25309 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_verbs.c
@@ -35,9 +35,10 @@
 
 #include "ipoib.h"
 
-int ipoib_mcast_attach(struct net_device *dev, u16 mlid, union ib_gid *mgid, int set_qkey)
+int ipoib_mcast_attach(struct net_device *dev, struct ib_device *hca,
+		       union ib_gid *mgid, u16 mlid, int set_qkey, u32 qkey)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	struct ib_qp_attr *qp_attr = NULL;
 	int ret;
 	u16 pkey_index;
@@ -56,7 +57,7 @@ int ipoib_mcast_attach(struct net_device *dev, u16 mlid, union ib_gid *mgid, int
 			goto out;
 
 		/* set correct QKey for QP */
-		qp_attr->qkey = priv->qkey;
+		qp_attr->qkey = qkey;
 		ret = ib_modify_qp(priv->qp, qp_attr, IB_QP_QKEY);
 		if (ret) {
 			ipoib_warn(priv, "failed to modify QP, ret = %d\n", ret);
@@ -74,9 +75,20 @@ int ipoib_mcast_attach(struct net_device *dev, u16 mlid, union ib_gid *mgid, int
 	return ret;
 }
 
+int ipoib_mcast_detach(struct net_device *dev, struct ib_device *hca,
+		       union ib_gid *mgid, u16 mlid)
+{
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
+	int ret;
+
+	ret = ib_detach_mcast(priv->qp, mgid, mlid);
+
+	return ret;
+}
+
 int ipoib_init_qp(struct net_device *dev)
 {
-	struct ipoib_dev_priv *priv = netdev_priv(dev);
+	struct ipoib_dev_priv *priv = ipoib_priv(dev);
 	int ret;
 	struct ib_qp_attr qp_attr;
 	int attr_mask;
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_vlan.c b/drivers/infiniband/ulp/ipoib/ipoib_vlan.c
index b4de1737986d..b97839a803e9 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_vlan.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_vlan.c
@@ -128,14 +128,15 @@ int ipoib_vlan_add(struct net_device *pdev, unsigned short pkey)
 	if (!capable(CAP_NET_ADMIN))
 		return -EPERM;
 
-	ppriv = netdev_priv(pdev);
+	ppriv = ipoib_priv(pdev);
 
 	if (test_bit(IPOIB_FLAG_GOING_DOWN, &ppriv->flags))
 		return -EPERM;
 
 	snprintf(intf_name, sizeof intf_name, "%s.%04x",
 		 ppriv->dev->name, pkey);
-	priv = ipoib_intf_alloc(intf_name);
+
+	priv = ipoib_intf_alloc(ppriv->ca, ppriv->port, intf_name);
 	if (!priv)
 		return -ENOMEM;
 
-- 
2.12.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 1/6] IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs

[Jason Gunthorpe]

On Tue, Apr 04, 2017 at 10:17:27PM +0300, Leon Romanovsky wrote:
>  struct ib_device {
> +	/* Do not access @dma_device directly from ULP nor from HW drivers. */
> +	struct device                *dma_device;
> +

Why is this hunk in this patch? I thought Bart was taking care of this
topic?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 6/6] IB/IPoIB: Support acceleration options callbacks

[Jason Gunthorpe]

On Tue, Apr 04, 2017 at 10:17:32PM +0300, Leon Romanovsky wrote:
> -/* Keep the refactoring compile able */
> -#define ipoib_priv netdev_priv
> +static inline void *ipoib_priv(const struct net_device *dev)
> +{
> +	struct rdma_netdev *rn = netdev_priv(dev);
> +
> +	return rn->clnt_priv;
> +}

This should return struct ipoib_dev_priv *

>  static ssize_t show_pkey(struct device *dev,
>  			 struct device_attribute *attr, char *buf)
>  {
> -	struct ipoib_dev_priv *priv = netdev_priv(to_net_dev(dev));
> +	struct ipoib_dev_priv *priv = ipoib_priv(to_net_dev(dev));

Several of these seem to be in the wrong patch

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 6/6] IB/IPoIB: Support acceleration options callbacks

[Jason Gunthorpe]

On Tue, Apr 04, 2017 at 10:17:32PM +0300, Leon Romanovsky wrote:

> +static struct net_device *ipoib_create_netdev_default(struct ib_device *hca,
> +						      const char *name,
> +						      unsigned char name_assign_type,
> +						      void (*setup)(struct net_device *))
>  {
>  	struct net_device *dev;
> +	struct rdma_netdev *rn;
>  
> -	dev = alloc_netdev((int)sizeof(struct ipoib_dev_priv), name,
> -			   NET_NAME_UNKNOWN, ipoib_setup);
> +	dev = alloc_netdev((int)sizeof(struct rdma_netdev),
> +			   name,
> +			   name_assign_type, setup);
>  	if (!dev)
>  		return NULL;
>  
> -	return netdev_priv(dev);
> +	rn = netdev_priv(dev);
> +
> +	rn->send = ipoib_send;
> +	rn->attach_mcast = ipoib_mcast_attach;
> +	rn->detach_mcast = ipoib_mcast_detach;

This should really set rn->hca as well, don't forget the kref

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Jason Gunthorpe]

On Tue, Apr 04, 2017 at 10:17:26PM +0300, Leon Romanovsky wrote:
> Hi Doug,
> 
> This patchset mostly comes from Erez with one exception in the first patch.
> 
> That patch origins from two different commits, first from Niranjana who added
> RDMA netdev interface and second from Erez who added IPoIB support.
> 
> During the preparation to submission, I squashed their commits into one
> and refactored code to allow submission as a standalone topic without creating
> dependecies between different submissions. This caused to change in author line.
> I hope that it doesn't really matter and it won't stop you from merging it.
> 
> Per-your request, I based this patch set on v4.11-rc3.

I didn't dig into the details of the ipoib part, but I think the
core change and the overall look is now what I would expect. modulo
the little comments

It would still have been nicer to isolate the 'default' implementation
so those functions did not touch ipoib_priv, but that is also
something that could be done in a followup..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 1/6] IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs

[Bart Van Assche]

On Tue, 2017-04-04 at 13:37 -0600, Jason Gunthorpe wrote:
> On Tue, Apr 04, 2017 at 10:17:27PM +0300, Leon Romanovsky wrote:
> >  struct ib_device {
> > +	/* Do not access @dma_device directly from ULP nor from HW drivers. */
> > +	struct device                *dma_device;
> > +
> 
> Why is this hunk in this patch? I thought Bart was taking care of this
> topic?

That hunk is already upstream. See also commit 0957c29f78af.

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Weiny, Ira]

> 
> Hi Doug,
> 
> This patchset mostly comes from Erez with one exception in the first patch.
> 
> That patch origins from two different commits, first from Niranjana who added
> RDMA netdev interface and second from Erez who added IPoIB support.
> 
> During the preparation to submission, I squashed their commits into one and
> refactored code to allow submission as a standalone topic without creating
> dependecies between different submissions. This caused to change in author
> line.
> I hope that it doesn't really matter and it won't stop you from merging it.

I'm confused by how you expect Niranjana's patch to be merged in this situation?

Doug, will you take care of the conflict between these 2 submitted patches?

[PATCH 02/11] IB/opa-vnic: Virtual Network Interface Controller (VNIC) interface
And
[PATCH rdma-next 1/6] IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs

Ira

> 
> Per-your request, I based this patch set on v4.11-rc3.
> 
> Thanks,
> 	Leon
> 
> CC: Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> CC: Alex Vesker <valex-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> ---
> 
> The rest comes from Erez:
> 
>     The IPoIB protocol encapsulates IP packets over Infiniband datagrams.
>     As a direct RDMA Upper Layer Protocol (ULP), IPoIB cannot support HW
>     features that are specific to the IP protocol stack.
> 
>     Nevertheless, RDMA interfaces have been extended to support some of the
>     prominent IP offload features, such as TCP/UDP checksum and TSO.
>     This provided reasonable performance gain for IPoIB but is still
>     insufficient to cope with the increasing network bandwidth demand.
> 
>     However, New features are exisiting in common network interfaces that
>     are very hard to implement in IPoIB interfaces while it uses the RDMA
>     layer, examples include TSS and RSS, tunneling offloads, and XDP.
>     Rather than continuously porting IP network interface developments into
>     the RDMA stack, we propose adding an abstract network data-path
>     interfaces to RDMA devices.
> 
>     In order to present a consistent interface to users, the IPoIB ULP
>     continues to represent the network device to the IP stack.
>     The common code also manages the IPoIB control plane, such as resolving
>     path queries and registering to multicast groups.
>     Data path operations are forwarded to devices that implement the new
>     API, or fallback to the standard implementation otherwise.
>     Using the forgoing approach, we show how IPoIB closes the performance
>     gap compared to state-of-the-art Ethernet network interfaces.
> 
>     The implementation idea is to use the api of
>     alloc_rdma_netdev/free_rdma_netdev to expose a struct that has data
>     members and set of functions that are used for IB network interfaces,
>     like attach/detach multicast to qp, and send IB packet.
> 
>     The functions are specific for IB operations and are not part of the
>     common api the the netdev struct exposes via the ndo functions.
>     1. multicast handling - attach/detach
>     2. send operation - the ndo start_xmit has only 2 parameters and the IB
>     send needs the destination qp and the ah object, there were few options
>     to handle it via the netdev ndo, but they don't make more sense than
>     using a specific send function (we are rdma_netdev after all)
> 
>     The IPoIB code will be adapted to enable the option of accelerating the
>     network interface, but the code will work as before if the HW below
>     doesn't support the acceleration.
>     That means that in the default mode of ipoib I tried to keep it as much
>     as it was before, not to force it to adopt the new api, where there is
>     no code sharing between the ipoib and the vnic/hfi.
>     The default code uses the controll and the data, the accelerator uses
>     only the control flows andstructors.
>     The changes of the default ipoib can be made on top of that series.
> 
>     Each HW vendor can supply the acceleration for the IPoIB or to leave
>     IPoIB to work as before.
> 
> Thanks,
> 	Erez
> 
> Erez Shitrit (5):
>   IB/IPoIB: Separate control and data related initializations
>   IB/IPoIB: Separate control from HW operation on ipoib_open/stop ndo
>   IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions
>   IB/IPoIB: Formatting before adding accelerator to driver
>   IB/IPoIB: Support acceleration options callbacks
> 
> Leon Romanovsky (1):
>   IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs
> 
>  drivers/infiniband/ulp/ipoib/ipoib.h           |  40 +--
>  drivers/infiniband/ulp/ipoib/ipoib_cm.c        |  66 ++---
>  drivers/infiniband/ulp/ipoib/ipoib_ethtool.c   |   6 +-
>  drivers/infiniband/ulp/ipoib/ipoib_fs.c        |   4 +-
>  drivers/infiniband/ulp/ipoib/ipoib_ib.c        | 339 +++++++++++++------------
>  drivers/infiniband/ulp/ipoib/ipoib_main.c      | 312 +++++++++++++++++------
>  drivers/infiniband/ulp/ipoib/ipoib_multicast.c |  39 +--
>  drivers/infiniband/ulp/ipoib/ipoib_netlink.c   |  12 +-
>  drivers/infiniband/ulp/ipoib/ipoib_verbs.c     |  64 ++---
>  drivers/infiniband/ulp/ipoib/ipoib_vlan.c      |   9 +-
>  include/rdma/ib_verbs.h                        |  41 +++
>  11 files changed, 565 insertions(+), 367 deletions(-)
> 
> --
> 2.12.0
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body
> of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 1/6] IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

On Tue, Apr 04, 2017 at 01:37:30PM -0600, Jason Gunthorpe wrote:
> On Tue, Apr 04, 2017 at 10:17:27PM +0300, Leon Romanovsky wrote:
> >  struct ib_device {
> > +	/* Do not access @dma_device directly from ULP nor from HW drivers. */
> > +	struct device                *dma_device;
> > +
>
> Why is this hunk in this patch? I thought Bart was taking care of this
> topic?

This patch set is based on -rc3, Bart's commit is available in -rc4.
➜  linux-rdma git:(topic/ipoib-accel) git describe 0957c29f78af
v4.11-rc3-10-g0957c29f78af

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]

On Tue, Apr 04, 2017 at 01:47:50PM -0600, Jason Gunthorpe wrote:
> On Tue, Apr 04, 2017 at 10:17:26PM +0300, Leon Romanovsky wrote:
> > Hi Doug,
> >
> > This patchset mostly comes from Erez with one exception in the first patch.
> >
> > That patch origins from two different commits, first from Niranjana who added
> > RDMA netdev interface and second from Erez who added IPoIB support.
> >
> > During the preparation to submission, I squashed their commits into one
> > and refactored code to allow submission as a standalone topic without creating
> > dependecies between different submissions. This caused to change in author line.
> > I hope that it doesn't really matter and it won't stop you from merging it.
> >
> > Per-your request, I based this patch set on v4.11-rc3.
>
> I didn't dig into the details of the ipoib part, but I think the
> core change and the overall look is now what I would expect. modulo
> the little comments
>
> It would still have been nicer to isolate the 'default' implementation
> so those functions did not touch ipoib_priv, but that is also
> something that could be done in a followup..

Thanks, Jason

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 6366 bytes --]

On Tue, Apr 04, 2017 at 07:50:21PM +0000, Weiny, Ira wrote:
> >
> > Hi Doug,
> >
> > This patchset mostly comes from Erez with one exception in the first patch.
> >
> > That patch origins from two different commits, first from Niranjana who added
> > RDMA netdev interface and second from Erez who added IPoIB support.
> >
> > During the preparation to submission, I squashed their commits into one and
> > refactored code to allow submission as a standalone topic without creating
> > dependecies between different submissions. This caused to change in author
> > line.
> > I hope that it doesn't really matter and it won't stop you from merging it.
>
> I'm confused by how you expect Niranjana's patch to be merged in this situation?
>
> Doug, will you take care of the conflict between these 2 submitted patches?
>
> [PATCH 02/11] IB/opa-vnic: Virtual Network Interface Controller (VNIC) interface
> And
> [PATCH rdma-next 1/6] IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs

Ira,

The IPoIB and HFI-VNIC are independent series, so topics should be
self-contained. If the merge conflict worries you, Niranjana or me could
easily base our patch sets on another topic, but we need to know
the acceptance (yes/no) and order between them.

Thanks

>
> Ira
>
> >
> > Per-your request, I based this patch set on v4.11-rc3.
> >
> > Thanks,
> > 	Leon
> >
> > CC: Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > CC: Alex Vesker <valex-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > ---
> >
> > The rest comes from Erez:
> >
> >     The IPoIB protocol encapsulates IP packets over Infiniband datagrams.
> >     As a direct RDMA Upper Layer Protocol (ULP), IPoIB cannot support HW
> >     features that are specific to the IP protocol stack.
> >
> >     Nevertheless, RDMA interfaces have been extended to support some of the
> >     prominent IP offload features, such as TCP/UDP checksum and TSO.
> >     This provided reasonable performance gain for IPoIB but is still
> >     insufficient to cope with the increasing network bandwidth demand.
> >
> >     However, New features are exisiting in common network interfaces that
> >     are very hard to implement in IPoIB interfaces while it uses the RDMA
> >     layer, examples include TSS and RSS, tunneling offloads, and XDP.
> >     Rather than continuously porting IP network interface developments into
> >     the RDMA stack, we propose adding an abstract network data-path
> >     interfaces to RDMA devices.
> >
> >     In order to present a consistent interface to users, the IPoIB ULP
> >     continues to represent the network device to the IP stack.
> >     The common code also manages the IPoIB control plane, such as resolving
> >     path queries and registering to multicast groups.
> >     Data path operations are forwarded to devices that implement the new
> >     API, or fallback to the standard implementation otherwise.
> >     Using the forgoing approach, we show how IPoIB closes the performance
> >     gap compared to state-of-the-art Ethernet network interfaces.
> >
> >     The implementation idea is to use the api of
> >     alloc_rdma_netdev/free_rdma_netdev to expose a struct that has data
> >     members and set of functions that are used for IB network interfaces,
> >     like attach/detach multicast to qp, and send IB packet.
> >
> >     The functions are specific for IB operations and are not part of the
> >     common api the the netdev struct exposes via the ndo functions.
> >     1. multicast handling - attach/detach
> >     2. send operation - the ndo start_xmit has only 2 parameters and the IB
> >     send needs the destination qp and the ah object, there were few options
> >     to handle it via the netdev ndo, but they don't make more sense than
> >     using a specific send function (we are rdma_netdev after all)
> >
> >     The IPoIB code will be adapted to enable the option of accelerating the
> >     network interface, but the code will work as before if the HW below
> >     doesn't support the acceleration.
> >     That means that in the default mode of ipoib I tried to keep it as much
> >     as it was before, not to force it to adopt the new api, where there is
> >     no code sharing between the ipoib and the vnic/hfi.
> >     The default code uses the controll and the data, the accelerator uses
> >     only the control flows andstructors.
> >     The changes of the default ipoib can be made on top of that series.
> >
> >     Each HW vendor can supply the acceleration for the IPoIB or to leave
> >     IPoIB to work as before.
> >
> > Thanks,
> > 	Erez
> >
> > Erez Shitrit (5):
> >   IB/IPoIB: Separate control and data related initializations
> >   IB/IPoIB: Separate control from HW operation on ipoib_open/stop ndo
> >   IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions
> >   IB/IPoIB: Formatting before adding accelerator to driver
> >   IB/IPoIB: Support acceleration options callbacks
> >
> > Leon Romanovsky (1):
> >   IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs
> >
> >  drivers/infiniband/ulp/ipoib/ipoib.h           |  40 +--
> >  drivers/infiniband/ulp/ipoib/ipoib_cm.c        |  66 ++---
> >  drivers/infiniband/ulp/ipoib/ipoib_ethtool.c   |   6 +-
> >  drivers/infiniband/ulp/ipoib/ipoib_fs.c        |   4 +-
> >  drivers/infiniband/ulp/ipoib/ipoib_ib.c        | 339 +++++++++++++------------
> >  drivers/infiniband/ulp/ipoib/ipoib_main.c      | 312 +++++++++++++++++------
> >  drivers/infiniband/ulp/ipoib/ipoib_multicast.c |  39 +--
> >  drivers/infiniband/ulp/ipoib/ipoib_netlink.c   |  12 +-
> >  drivers/infiniband/ulp/ipoib/ipoib_verbs.c     |  64 ++---
> >  drivers/infiniband/ulp/ipoib/ipoib_vlan.c      |   9 +-
> >  include/rdma/ib_verbs.h                        |  41 +++
> >  11 files changed, 565 insertions(+), 367 deletions(-)
> >
> > --
> > 2.12.0
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body
> > of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at
> > http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Jason Gunthorpe]

On Tue, Apr 04, 2017 at 10:57:20PM +0300, Leon Romanovsky wrote:
> > It would still have been nicer to isolate the 'default' implementation
> > so those functions did not touch ipoib_priv, but that is also
> > something that could be done in a followup..
> 
> Thanks, Jason

I think before merging it would also be good to see the benchmark
numbers - we all assume this is making things a lot faster..

Does it now exceed RC performance?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 4/6] IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 3228 bytes --]

Hi Erez,

[auto build test ERROR on rdma/master]
[also build test ERROR on v4.11-rc5 next-20170404]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Leon-Romanovsky/Enhanced-mode-for-IPoIB-driver/20170405-063335
base:   https://git.kernel.org/pub/scm/linux/kernel/git/dledford/rdma.git master
config: i386-allmodconfig (attached as .config)
compiler: gcc-6 (Debian 6.2.0-3) 6.2.0 20160901
reproduce:
        # save the attached .config to linux build tree
        make ARCH=i386 

All errors (new ones prefixed by >>):

   In file included from drivers/infiniband/ulp/ipoib/ipoib_ib.c:44:0:
   drivers/infiniband/ulp/ipoib/ipoib_ib.c: In function 'ipoib_send':
>> drivers/infiniband/ulp/ipoib/ipoib_ib.c:590:29: error: 'qpn' undeclared (first use in this function)
             skb->len, address, qpn);
                                ^
   drivers/infiniband/ulp/ipoib/ipoib.h:780:79: note: in definition of macro 'ipoib_printk'
     printk(level "%s: " format, ((struct ipoib_dev_priv *) priv)->dev->name , ## arg)
                                                                                  ^~~
   drivers/infiniband/ulp/ipoib/ipoib_ib.c:589:2: note: in expansion of macro 'ipoib_dbg_data'
     ipoib_dbg_data(priv, "sending packet, length=%d address=%p qpn=0x%06x\n",
     ^~~~~~~~~~~~~~
   drivers/infiniband/ulp/ipoib/ipoib_ib.c:590:29: note: each undeclared identifier is reported only once for each function it appears in
             skb->len, address, qpn);
                                ^
   drivers/infiniband/ulp/ipoib/ipoib.h:780:79: note: in definition of macro 'ipoib_printk'
     printk(level "%s: " format, ((struct ipoib_dev_priv *) priv)->dev->name , ## arg)
                                                                                  ^~~
   drivers/infiniband/ulp/ipoib/ipoib_ib.c:589:2: note: in expansion of macro 'ipoib_dbg_data'
     ipoib_dbg_data(priv, "sending packet, length=%d address=%p qpn=0x%06x\n",
     ^~~~~~~~~~~~~~

vim +/qpn +590 drivers/infiniband/ulp/ipoib/ipoib_ib.c

78a50a5e Hans Westgaard Ry 2016-03-02  584  			dev_kfree_skb_any(skb);
78a50a5e Hans Westgaard Ry 2016-03-02  585  			return;
78a50a5e Hans Westgaard Ry 2016-03-02  586  		}
78a50a5e Hans Westgaard Ry 2016-03-02  587  	}
^1da177e Linus Torvalds    2005-04-16  588  
^1da177e Linus Torvalds    2005-04-16  589  	ipoib_dbg_data(priv, "sending packet, length=%d address=%p qpn=0x%06x\n",
^1da177e Linus Torvalds    2005-04-16 @590  		       skb->len, address, qpn);
^1da177e Linus Torvalds    2005-04-16  591  
^1da177e Linus Torvalds    2005-04-16  592  	/*
^1da177e Linus Torvalds    2005-04-16  593  	 * We put the skb into the tx_ring _before_ we call post_send()

:::::: The code at line 590 was first introduced by commit
:::::: 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 Linux-2.6.12-rc2

:::::: TO: Linus Torvalds <torvalds-gWtpgVMusWVb5UGfqNBoRg@public.gmane.org>
:::::: CC: Linus Torvalds <torvalds-gWtpgVMusWVb5UGfqNBoRg@public.gmane.org>

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 59004 bytes --]

Re: [PATCH rdma-next 6/6] IB/IPoIB: Support acceleration options callbacks

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 785 bytes --]

On Tue, Apr 04, 2017 at 01:42:34PM -0600, Jason Gunthorpe wrote:
> On Tue, Apr 04, 2017 at 10:17:32PM +0300, Leon Romanovsky wrote:
> > -/* Keep the refactoring compile able */
> > -#define ipoib_priv netdev_priv
> > +static inline void *ipoib_priv(const struct net_device *dev)
> > +{
> > +	struct rdma_netdev *rn = netdev_priv(dev);
> > +
> > +	return rn->clnt_priv;
> > +}
>
> This should return struct ipoib_dev_priv *

You are right, this is the whole purpose of ipoib_priv().

>
> >  static ssize_t show_pkey(struct device *dev,
> >  			 struct device_attribute *attr, char *buf)
> >  {
> > -	struct ipoib_dev_priv *priv = netdev_priv(to_net_dev(dev));
> > +	struct ipoib_dev_priv *priv = ipoib_priv(to_net_dev(dev));
>
> Several of these seem to be in the wrong patch
>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 6/6] IB/IPoIB: Support acceleration options callbacks

[Erez Shitrit]

On Tue, Apr 4, 2017 at 10:42 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Tue, Apr 04, 2017 at 10:17:32PM +0300, Leon Romanovsky wrote:
>> -/* Keep the refactoring compile able */
>> -#define ipoib_priv netdev_priv
>> +static inline void *ipoib_priv(const struct net_device *dev)
>> +{
>> +     struct rdma_netdev *rn = netdev_priv(dev);
>> +
>> +     return rn->clnt_priv;
>> +}
>
> This should return struct ipoib_dev_priv *

agree. will fix.

>
>>  static ssize_t show_pkey(struct device *dev,
>>                        struct device_attribute *attr, char *buf)
>>  {
>> -     struct ipoib_dev_priv *priv = netdev_priv(to_net_dev(dev));
>> +     struct ipoib_dev_priv *priv = ipoib_priv(to_net_dev(dev));
>
> Several of these seem to be in the wrong patch
>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 4/6] IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 335 bytes --]

On Wed, Apr 05, 2017 at 08:56:29AM +0800, kbuild test robot wrote:
> Hi Erez,
>
> [auto build test ERROR on rdma/master]
> [also build test ERROR on v4.11-rc5 next-20170404]
> [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

It should be applied to v4.11-rc3 and not to -rc5.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

On Tue, Apr 04, 2017 at 02:14:08PM -0600, Jason Gunthorpe wrote:
> On Tue, Apr 04, 2017 at 10:57:20PM +0300, Leon Romanovsky wrote:
> > > It would still have been nicer to isolate the 'default' implementation
> > > so those functions did not touch ipoib_priv, but that is also
> > > something that could be done in a followup..
> >
> > Thanks, Jason
>
> I think before merging it would also be good to see the benchmark
> numbers - we all assume this is making things a lot faster..
>
> Does it now exceed RC performance?

In our tests, prior to submission, It achieved almost line rate (~93-95Gb/sec) with 4 streams.
Erez is supposed to continue his work and optimize it more till full line rate.

Thanks

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 6/6] IB/IPoIB: Support acceleration options callbacks

[Erez Shitrit]

On Tue, Apr 4, 2017 at 10:42 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Tue, Apr 04, 2017 at 10:17:32PM +0300, Leon Romanovsky wrote:
>> -/* Keep the refactoring compile able */
>> -#define ipoib_priv netdev_priv
>> +static inline void *ipoib_priv(const struct net_device *dev)
>> +{
>> +     struct rdma_netdev *rn = netdev_priv(dev);
>> +
>> +     return rn->clnt_priv;
>> +}
>
> This should return struct ipoib_dev_priv *
>
>>  static ssize_t show_pkey(struct device *dev,
>>                        struct device_attribute *attr, char *buf)
>>  {
>> -     struct ipoib_dev_priv *priv = netdev_priv(to_net_dev(dev));
>> +     struct ipoib_dev_priv *priv = ipoib_priv(to_net_dev(dev));
>
> Several of these seem to be in the wrong patch

will fix that.
thanks !

>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 6/6] IB/IPoIB: Support acceleration options callbacks

[Erez Shitrit]

On Tue, Apr 4, 2017 at 10:46 PM, Jason Gunthorpe
<jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> On Tue, Apr 04, 2017 at 10:17:32PM +0300, Leon Romanovsky wrote:
>
>> +static struct net_device *ipoib_create_netdev_default(struct ib_device *hca,
>> +                                                   const char *name,
>> +                                                   unsigned char name_assign_type,
>> +                                                   void (*setup)(struct net_device *))
>>  {
>>       struct net_device *dev;
>> +     struct rdma_netdev *rn;
>>
>> -     dev = alloc_netdev((int)sizeof(struct ipoib_dev_priv), name,
>> -                        NET_NAME_UNKNOWN, ipoib_setup);
>> +     dev = alloc_netdev((int)sizeof(struct rdma_netdev),
>> +                        name,
>> +                        name_assign_type, setup);
>>       if (!dev)
>>               return NULL;
>>
>> -     return netdev_priv(dev);
>> +     rn = netdev_priv(dev);
>> +
>> +     rn->send = ipoib_send;
>> +     rn->attach_mcast = ipoib_mcast_attach;
>> +     rn->detach_mcast = ipoib_mcast_detach;
>
> This should really set rn->hca as well, don't forget the kref

ok, will add.
thanks.

>
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 4/6] IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions

[Bart Van Assche]

On Wed, 2017-04-05 at 13:38 +0300, Leon Romanovsky wrote:
> On Wed, Apr 05, 2017 at 08:56:29AM +0800, kbuild test robot wrote:
> > Hi Erez,
> > 
> > [auto build test ERROR on rdma/master]
> > [also build test ERROR on v4.11-rc5 next-20170404]
> > [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
> 
> It should be applied to v4.11-rc3 and not to -rc5.

Hello Leon,

Will it be possible for Doug to apply this series to his tree without
rebasing this patch series on rc5 instead of rc3?

Thanks,

Bart.--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 4/6] IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

On Wed, Apr 05, 2017 at 03:00:48PM +0000, Bart Van Assche wrote:
> On Wed, 2017-04-05 at 13:38 +0300, Leon Romanovsky wrote:
> > On Wed, Apr 05, 2017 at 08:56:29AM +0800, kbuild test robot wrote:
> > > Hi Erez,
> > >
> > > [auto build test ERROR on rdma/master]
> > > [also build test ERROR on v4.11-rc5 next-20170404]
> > > [if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
> >
> > It should be applied to v4.11-rc3 and not to -rc5.
>
> Hello Leon,
>
> Will it be possible for Doug to apply this series to his tree without
> rebasing this patch series on rc5 instead of rc3?

Yes, I tried it now and it worked in both directions without conflicts.
I merged these patches into rc5 and rc5 into them.

However, I asked from Doug explicitly during the OFA on which rc should
I base my submissions and he asked to submit based on rc3, so I did.

Thanks

>
> Thanks,
>
> Bart.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Weiny, Ira]

> 
> On Tue, Apr 04, 2017 at 07:50:21PM +0000, Weiny, Ira wrote:
> > >
> > > Hi Doug,
> > >
> > > This patchset mostly comes from Erez with one exception in the first patch.
> > >
> > > That patch origins from two different commits, first from Niranjana
> > > who added RDMA netdev interface and second from Erez who added IPoIB
> support.
> > >
> > > During the preparation to submission, I squashed their commits into
> > > one and refactored code to allow submission as a standalone topic
> > > without creating dependecies between different submissions. This
> > > caused to change in author line.
> > > I hope that it doesn't really matter and it won't stop you from merging it.
> >
> > I'm confused by how you expect Niranjana's patch to be merged in this
> situation?
> >
> > Doug, will you take care of the conflict between these 2 submitted patches?
> >
> > [PATCH 02/11] IB/opa-vnic: Virtual Network Interface Controller (VNIC)
> > interface And [PATCH rdma-next 1/6] IB/IPoIB: Introduce RDMA netdev
> > interface and IPoIB structs
> 
> Ira,
> 
> The IPoIB and HFI-VNIC are independent series, so topics should be self-
> contained. If the merge conflict worries you, Niranjana or me could easily base
> our patch sets on another topic, but we need to know the acceptance (yes/no)
> and order between them.

I guess my expectation was that upon your agreement of this new interface, Doug would accept Niranjanas change and you could build on top of that rather than just taking part of Niranjanas patch and expecting Doug to see there are common hunks.  Perhaps I'm missing something and git is smart enough to deal with it?

Does this mean you have reviewed and are comfortable with Niranjanas new rdma_netdev changes?  If so it seems Doug could accept them.

Ira

> 
> Thanks
> 
> >
> > Ira
> >
> > >
> > > Per-your request, I based this patch set on v4.11-rc3.
> > >
> > > Thanks,
> > > 	Leon
> > >
> > > CC: Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > > CC: Alex Vesker <valex-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > ---
> > >
> > > The rest comes from Erez:
> > >
> > >     The IPoIB protocol encapsulates IP packets over Infiniband datagrams.
> > >     As a direct RDMA Upper Layer Protocol (ULP), IPoIB cannot support HW
> > >     features that are specific to the IP protocol stack.
> > >
> > >     Nevertheless, RDMA interfaces have been extended to support some of
> the
> > >     prominent IP offload features, such as TCP/UDP checksum and TSO.
> > >     This provided reasonable performance gain for IPoIB but is still
> > >     insufficient to cope with the increasing network bandwidth demand.
> > >
> > >     However, New features are exisiting in common network interfaces that
> > >     are very hard to implement in IPoIB interfaces while it uses the RDMA
> > >     layer, examples include TSS and RSS, tunneling offloads, and XDP.
> > >     Rather than continuously porting IP network interface developments into
> > >     the RDMA stack, we propose adding an abstract network data-path
> > >     interfaces to RDMA devices.
> > >
> > >     In order to present a consistent interface to users, the IPoIB ULP
> > >     continues to represent the network device to the IP stack.
> > >     The common code also manages the IPoIB control plane, such as resolving
> > >     path queries and registering to multicast groups.
> > >     Data path operations are forwarded to devices that implement the new
> > >     API, or fallback to the standard implementation otherwise.
> > >     Using the forgoing approach, we show how IPoIB closes the performance
> > >     gap compared to state-of-the-art Ethernet network interfaces.
> > >
> > >     The implementation idea is to use the api of
> > >     alloc_rdma_netdev/free_rdma_netdev to expose a struct that has data
> > >     members and set of functions that are used for IB network interfaces,
> > >     like attach/detach multicast to qp, and send IB packet.
> > >
> > >     The functions are specific for IB operations and are not part of the
> > >     common api the the netdev struct exposes via the ndo functions.
> > >     1. multicast handling - attach/detach
> > >     2. send operation - the ndo start_xmit has only 2 parameters and the IB
> > >     send needs the destination qp and the ah object, there were few options
> > >     to handle it via the netdev ndo, but they don't make more sense than
> > >     using a specific send function (we are rdma_netdev after all)
> > >
> > >     The IPoIB code will be adapted to enable the option of accelerating the
> > >     network interface, but the code will work as before if the HW below
> > >     doesn't support the acceleration.
> > >     That means that in the default mode of ipoib I tried to keep it as much
> > >     as it was before, not to force it to adopt the new api, where there is
> > >     no code sharing between the ipoib and the vnic/hfi.
> > >     The default code uses the controll and the data, the accelerator uses
> > >     only the control flows andstructors.
> > >     The changes of the default ipoib can be made on top of that series.
> > >
> > >     Each HW vendor can supply the acceleration for the IPoIB or to leave
> > >     IPoIB to work as before.
> > >
> > > Thanks,
> > > 	Erez
> > >
> > > Erez Shitrit (5):
> > >   IB/IPoIB: Separate control and data related initializations
> > >   IB/IPoIB: Separate control from HW operation on ipoib_open/stop ndo
> > >   IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions
> > >   IB/IPoIB: Formatting before adding accelerator to driver
> > >   IB/IPoIB: Support acceleration options callbacks
> > >
> > > Leon Romanovsky (1):
> > >   IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs
> > >
> > >  drivers/infiniband/ulp/ipoib/ipoib.h           |  40 +--
> > >  drivers/infiniband/ulp/ipoib/ipoib_cm.c        |  66 ++---
> > >  drivers/infiniband/ulp/ipoib/ipoib_ethtool.c   |   6 +-
> > >  drivers/infiniband/ulp/ipoib/ipoib_fs.c        |   4 +-
> > >  drivers/infiniband/ulp/ipoib/ipoib_ib.c        | 339 +++++++++++++------------
> > >  drivers/infiniband/ulp/ipoib/ipoib_main.c      | 312 +++++++++++++++++---
> ---
> > >  drivers/infiniband/ulp/ipoib/ipoib_multicast.c |  39 +--
> > >  drivers/infiniband/ulp/ipoib/ipoib_netlink.c   |  12 +-
> > >  drivers/infiniband/ulp/ipoib/ipoib_verbs.c     |  64 ++---
> > >  drivers/infiniband/ulp/ipoib/ipoib_vlan.c      |   9 +-
> > >  include/rdma/ib_verbs.h                        |  41 +++
> > >  11 files changed, 565 insertions(+), 367 deletions(-)
> > >
> > > --
> > > 2.12.0
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe
> > > linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> > in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo
> > info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Dennis Dalessandro]

On 04/04/2017 03:17 PM, Leon Romanovsky wrote:
> Hi Doug,
>
> This patchset mostly comes from Erez with one exception in the first patch.
>
> That patch origins from two different commits, first from Niranjana who added
> RDMA netdev interface and second from Erez who added IPoIB support.
>
> During the preparation to submission, I squashed their commits into one
> and refactored code to allow submission as a standalone topic without creating
> dependecies between different submissions. This caused to change in author line.
> I hope that it doesn't really matter and it won't stop you from merging it.

I think that it does indeed matter. The patch was authored by Niranjana 
it should reflect as such. Why not take Niranjana's patch(s), the fact 
that you folded them in seems to mean you at least agree with the patch.

-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 8717 bytes --]

On Thu, Apr 06, 2017 at 12:54:02AM +0000, Weiny, Ira wrote:
> >
> > On Tue, Apr 04, 2017 at 07:50:21PM +0000, Weiny, Ira wrote:
> > > >
> > > > Hi Doug,
> > > >
> > > > This patchset mostly comes from Erez with one exception in the first patch.
> > > >
> > > > That patch origins from two different commits, first from Niranjana
> > > > who added RDMA netdev interface and second from Erez who added IPoIB
> > support.
> > > >
> > > > During the preparation to submission, I squashed their commits into
> > > > one and refactored code to allow submission as a standalone topic
> > > > without creating dependecies between different submissions. This
> > > > caused to change in author line.
> > > > I hope that it doesn't really matter and it won't stop you from merging it.
> > >
> > > I'm confused by how you expect Niranjana's patch to be merged in this
> > situation?
> > >
> > > Doug, will you take care of the conflict between these 2 submitted patches?
> > >
> > > [PATCH 02/11] IB/opa-vnic: Virtual Network Interface Controller (VNIC)
> > > interface And [PATCH rdma-next 1/6] IB/IPoIB: Introduce RDMA netdev
> > > interface and IPoIB structs
> >
> > Ira,
> >
> > The IPoIB and HFI-VNIC are independent series, so topics should be self-
> > contained. If the merge conflict worries you, Niranjana or me could easily base
> > our patch sets on another topic, but we need to know the acceptance (yes/no)
> > and order between them.
>
> I guess my expectation was that upon your agreement of this new interface, Doug would accept Niranjanas change and you could build on top of that rather than just taking part of Niranjanas patch and expecting Doug to see there are common hunks.  Perhaps I'm missing something and git is smart enough to deal with it?

It is not related to smart git. Parallel submissions should be independent,
this is the why I'm submitting all our topics based on one common ground (usually -rcX),
despite the fact that sometimes there is merge conflict between them.

As long Doug works in this mode, we are obligated to continue.

>
> Does this mean you have reviewed and are comfortable with Niranjanas new rdma_netdev changes?  If so it seems Doug could accept them.

We already discussed it in email and privately, I'm agree with you that IPoIB and HFI-VNIC
are better to be separated, due to conceptually difference and future code maintenance.
The proposed rdma_netdev suits well both patch sets, so I'm very comfortable with them.

However, I'm not comfortable with ULP location, for the code which is not interoperable
between different systems (nodes). After OFA, I realized that it wouldn't work in Susan's
LANL mixed cluster, which is pretty awful for the common ULP.

The code itself needs proprietary software (ethernet manager) to be
operable and it placed in ULP based on some promise that future drivers
will be added to hfi family. Right now, it is not the case.

You asked, I answered :)

>
> Ira
>
> >
> > Thanks
> >
> > >
> > > Ira
> > >
> > > >
> > > > Per-your request, I based this patch set on v4.11-rc3.
> > > >
> > > > Thanks,
> > > > 	Leon
> > > >
> > > > CC: Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > > > CC: Alex Vesker <valex-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > > ---
> > > >
> > > > The rest comes from Erez:
> > > >
> > > >     The IPoIB protocol encapsulates IP packets over Infiniband datagrams.
> > > >     As a direct RDMA Upper Layer Protocol (ULP), IPoIB cannot support HW
> > > >     features that are specific to the IP protocol stack.
> > > >
> > > >     Nevertheless, RDMA interfaces have been extended to support some of
> > the
> > > >     prominent IP offload features, such as TCP/UDP checksum and TSO.
> > > >     This provided reasonable performance gain for IPoIB but is still
> > > >     insufficient to cope with the increasing network bandwidth demand.
> > > >
> > > >     However, New features are exisiting in common network interfaces that
> > > >     are very hard to implement in IPoIB interfaces while it uses the RDMA
> > > >     layer, examples include TSS and RSS, tunneling offloads, and XDP.
> > > >     Rather than continuously porting IP network interface developments into
> > > >     the RDMA stack, we propose adding an abstract network data-path
> > > >     interfaces to RDMA devices.
> > > >
> > > >     In order to present a consistent interface to users, the IPoIB ULP
> > > >     continues to represent the network device to the IP stack.
> > > >     The common code also manages the IPoIB control plane, such as resolving
> > > >     path queries and registering to multicast groups.
> > > >     Data path operations are forwarded to devices that implement the new
> > > >     API, or fallback to the standard implementation otherwise.
> > > >     Using the forgoing approach, we show how IPoIB closes the performance
> > > >     gap compared to state-of-the-art Ethernet network interfaces.
> > > >
> > > >     The implementation idea is to use the api of
> > > >     alloc_rdma_netdev/free_rdma_netdev to expose a struct that has data
> > > >     members and set of functions that are used for IB network interfaces,
> > > >     like attach/detach multicast to qp, and send IB packet.
> > > >
> > > >     The functions are specific for IB operations and are not part of the
> > > >     common api the the netdev struct exposes via the ndo functions.
> > > >     1. multicast handling - attach/detach
> > > >     2. send operation - the ndo start_xmit has only 2 parameters and the IB
> > > >     send needs the destination qp and the ah object, there were few options
> > > >     to handle it via the netdev ndo, but they don't make more sense than
> > > >     using a specific send function (we are rdma_netdev after all)
> > > >
> > > >     The IPoIB code will be adapted to enable the option of accelerating the
> > > >     network interface, but the code will work as before if the HW below
> > > >     doesn't support the acceleration.
> > > >     That means that in the default mode of ipoib I tried to keep it as much
> > > >     as it was before, not to force it to adopt the new api, where there is
> > > >     no code sharing between the ipoib and the vnic/hfi.
> > > >     The default code uses the controll and the data, the accelerator uses
> > > >     only the control flows andstructors.
> > > >     The changes of the default ipoib can be made on top of that series.
> > > >
> > > >     Each HW vendor can supply the acceleration for the IPoIB or to leave
> > > >     IPoIB to work as before.
> > > >
> > > > Thanks,
> > > > 	Erez
> > > >
> > > > Erez Shitrit (5):
> > > >   IB/IPoIB: Separate control and data related initializations
> > > >   IB/IPoIB: Separate control from HW operation on ipoib_open/stop ndo
> > > >   IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions
> > > >   IB/IPoIB: Formatting before adding accelerator to driver
> > > >   IB/IPoIB: Support acceleration options callbacks
> > > >
> > > > Leon Romanovsky (1):
> > > >   IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs
> > > >
> > > >  drivers/infiniband/ulp/ipoib/ipoib.h           |  40 +--
> > > >  drivers/infiniband/ulp/ipoib/ipoib_cm.c        |  66 ++---
> > > >  drivers/infiniband/ulp/ipoib/ipoib_ethtool.c   |   6 +-
> > > >  drivers/infiniband/ulp/ipoib/ipoib_fs.c        |   4 +-
> > > >  drivers/infiniband/ulp/ipoib/ipoib_ib.c        | 339 +++++++++++++------------
> > > >  drivers/infiniband/ulp/ipoib/ipoib_main.c      | 312 +++++++++++++++++---
> > ---
> > > >  drivers/infiniband/ulp/ipoib/ipoib_multicast.c |  39 +--
> > > >  drivers/infiniband/ulp/ipoib/ipoib_netlink.c   |  12 +-
> > > >  drivers/infiniband/ulp/ipoib/ipoib_verbs.c     |  64 ++---
> > > >  drivers/infiniband/ulp/ipoib/ipoib_vlan.c      |   9 +-
> > > >  include/rdma/ib_verbs.h                        |  41 +++
> > > >  11 files changed, 565 insertions(+), 367 deletions(-)
> > > >
> > > > --
> > > > 2.12.0
> > > >
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe
> > > > linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> > > in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo
> > > info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1274 bytes --]

On Wed, Apr 05, 2017 at 09:50:59PM -0400, Dennis Dalessandro wrote:
> On 04/04/2017 03:17 PM, Leon Romanovsky wrote:
> > Hi Doug,
> >
> > This patchset mostly comes from Erez with one exception in the first patch.
> >
> > That patch origins from two different commits, first from Niranjana who added
> > RDMA netdev interface and second from Erez who added IPoIB support.
> >
> > During the preparation to submission, I squashed their commits into one
> > and refactored code to allow submission as a standalone topic without creating
> > dependecies between different submissions. This caused to change in author line.
> > I hope that it doesn't really matter and it won't stop you from merging it.
>
> I think that it does indeed matter. The patch was authored by Niranjana it
> should reflect as such. Why not take Niranjana's patch(s), the fact that you
> folded them in seems to mean you at least agree with the patch.

Sure, I agree, but I was supposed to remove all VNIC code from that
patch, which is unrelated to IPoIB.

If Niranjana feels comfortable to be authored on IPoIB code, the
following code will do the trick.

git commit --amend --author "Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>"

Thanks

>
> -Denny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Vishwanathapura, Niranjana]

On Thu, Apr 06, 2017 at 08:28:20AM +0300, Leon Romanovsky wrote:
>On Thu, Apr 06, 2017 at 12:54:02AM +0000, Weiny, Ira wrote:
>> >
>> > On Tue, Apr 04, 2017 at 07:50:21PM +0000, Weiny, Ira wrote:
>> > > >
>> > > > Hi Doug,
>> > > >
>> > > > This patchset mostly comes from Erez with one exception in the first patch.
>> > > >
>> > > > That patch origins from two different commits, first from Niranjana
>> > > > who added RDMA netdev interface and second from Erez who added IPoIB
>> > support.
>> > > >
>> > > > During the preparation to submission, I squashed their commits into
>> > > > one and refactored code to allow submission as a standalone topic
>> > > > without creating dependecies between different submissions. This
>> > > > caused to change in author line.
>> > > > I hope that it doesn't really matter and it won't stop you from merging it.
>> > >
>> > > I'm confused by how you expect Niranjana's patch to be merged in this
>> > situation?
>> > >
>> > > Doug, will you take care of the conflict between these 2 submitted patches?
>> > >
>> > > [PATCH 02/11] IB/opa-vnic: Virtual Network Interface Controller (VNIC)
>> > > interface And [PATCH rdma-next 1/6] IB/IPoIB: Introduce RDMA netdev
>> > > interface and IPoIB structs
>> >
>> > Ira,
>> >
>> > The IPoIB and HFI-VNIC are independent series, so topics should be self-
>> > contained. If the merge conflict worries you, Niranjana or me could easily base
>> > our patch sets on another topic, but we need to know the acceptance (yes/no)
>> > and order between them.
>>
>> I guess my expectation was that upon your agreement of this new interface, Doug would accept Niranjanas change and you could build on top of that rather than just taking part of Niranjanas patch and expecting Doug to see there are common hunks.  Perhaps I'm missing something and git is smart enough to deal with it?
>
>It is not related to smart git. Parallel submissions should be independent,
>this is the why I'm submitting all our topics based on one common ground (usually -rcX),
>despite the fact that sometimes there is merge conflict between them.
>
>As long Doug works in this mode, we are obligated to continue.
>
>>
>> Does this mean you have reviewed and are comfortable with Niranjanas new rdma_netdev changes?  If so it seems Doug could accept them.
>
>We already discussed it in email and privately, I'm agree with you that IPoIB and HFI-VNIC
>are better to be separated, due to conceptually difference and future code maintenance.
>The proposed rdma_netdev suits well both patch sets, so I'm very comfortable with them.
>

Thanks.

>However, I'm not comfortable with ULP location, for the code which is not interoperable
>between different systems (nodes). After OFA, I realized that it wouldn't work in Susan's
>LANL mixed cluster, which is pretty awful for the common ULP.
>
>The code itself needs proprietary software (ethernet manager) to be
>operable and it placed in ULP based on some promise that future drivers
>will be added to hfi family. Right now, it is not the case.
>
>You asked, I answered :)
>

These have been already discussed out before.
That is not what Ira is asking at all.
We have been patiently waiting for a month and a half to ensure this IPoIB 
patches can fit will with the rdma netdev interface as Jason and Doug wanted.

Now that IPoIB is comfortable with the rdma netdev interface, I think OPA VNIC 
patch can move forward.

Niranjana

>>
>> Ira
>>
>> >
>> > Thanks
>> >
>> > >
>> > > Ira
>> > >
>> > > >
>> > > > Per-your request, I based this patch set on v4.11-rc3.
>> > > >
>> > > > Thanks,
>> > > > 	Leon
>> > > >
>> > > > CC: Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
>> > > > CC: Alex Vesker <valex-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
>> > > > ---
>> > > >
>> > > > The rest comes from Erez:
>> > > >
>> > > >     The IPoIB protocol encapsulates IP packets over Infiniband datagrams.
>> > > >     As a direct RDMA Upper Layer Protocol (ULP), IPoIB cannot support HW
>> > > >     features that are specific to the IP protocol stack.
>> > > >
>> > > >     Nevertheless, RDMA interfaces have been extended to support some of
>> > the
>> > > >     prominent IP offload features, such as TCP/UDP checksum and TSO.
>> > > >     This provided reasonable performance gain for IPoIB but is still
>> > > >     insufficient to cope with the increasing network bandwidth demand.
>> > > >
>> > > >     However, New features are exisiting in common network interfaces that
>> > > >     are very hard to implement in IPoIB interfaces while it uses the RDMA
>> > > >     layer, examples include TSS and RSS, tunneling offloads, and XDP.
>> > > >     Rather than continuously porting IP network interface developments into
>> > > >     the RDMA stack, we propose adding an abstract network data-path
>> > > >     interfaces to RDMA devices.
>> > > >
>> > > >     In order to present a consistent interface to users, the IPoIB ULP
>> > > >     continues to represent the network device to the IP stack.
>> > > >     The common code also manages the IPoIB control plane, such as resolving
>> > > >     path queries and registering to multicast groups.
>> > > >     Data path operations are forwarded to devices that implement the new
>> > > >     API, or fallback to the standard implementation otherwise.
>> > > >     Using the forgoing approach, we show how IPoIB closes the performance
>> > > >     gap compared to state-of-the-art Ethernet network interfaces.
>> > > >
>> > > >     The implementation idea is to use the api of
>> > > >     alloc_rdma_netdev/free_rdma_netdev to expose a struct that has data
>> > > >     members and set of functions that are used for IB network interfaces,
>> > > >     like attach/detach multicast to qp, and send IB packet.
>> > > >
>> > > >     The functions are specific for IB operations and are not part of the
>> > > >     common api the the netdev struct exposes via the ndo functions.
>> > > >     1. multicast handling - attach/detach
>> > > >     2. send operation - the ndo start_xmit has only 2 parameters and the IB
>> > > >     send needs the destination qp and the ah object, there were few options
>> > > >     to handle it via the netdev ndo, but they don't make more sense than
>> > > >     using a specific send function (we are rdma_netdev after all)
>> > > >
>> > > >     The IPoIB code will be adapted to enable the option of accelerating the
>> > > >     network interface, but the code will work as before if the HW below
>> > > >     doesn't support the acceleration.
>> > > >     That means that in the default mode of ipoib I tried to keep it as much
>> > > >     as it was before, not to force it to adopt the new api, where there is
>> > > >     no code sharing between the ipoib and the vnic/hfi.
>> > > >     The default code uses the controll and the data, the accelerator uses
>> > > >     only the control flows andstructors.
>> > > >     The changes of the default ipoib can be made on top of that series.
>> > > >
>> > > >     Each HW vendor can supply the acceleration for the IPoIB or to leave
>> > > >     IPoIB to work as before.
>> > > >
>> > > > Thanks,
>> > > > 	Erez
>> > > >
>> > > > Erez Shitrit (5):
>> > > >   IB/IPoIB: Separate control and data related initializations
>> > > >   IB/IPoIB: Separate control from HW operation on ipoib_open/stop ndo
>> > > >   IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions
>> > > >   IB/IPoIB: Formatting before adding accelerator to driver
>> > > >   IB/IPoIB: Support acceleration options callbacks
>> > > >
>> > > > Leon Romanovsky (1):
>> > > >   IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs
>> > > >
>> > > >  drivers/infiniband/ulp/ipoib/ipoib.h           |  40 +--
>> > > >  drivers/infiniband/ulp/ipoib/ipoib_cm.c        |  66 ++---
>> > > >  drivers/infiniband/ulp/ipoib/ipoib_ethtool.c   |   6 +-
>> > > >  drivers/infiniband/ulp/ipoib/ipoib_fs.c        |   4 +-
>> > > >  drivers/infiniband/ulp/ipoib/ipoib_ib.c        | 339 +++++++++++++------------
>> > > >  drivers/infiniband/ulp/ipoib/ipoib_main.c      | 312 +++++++++++++++++---
>> > ---
>> > > >  drivers/infiniband/ulp/ipoib/ipoib_multicast.c |  39 +--
>> > > >  drivers/infiniband/ulp/ipoib/ipoib_netlink.c   |  12 +-
>> > > >  drivers/infiniband/ulp/ipoib/ipoib_verbs.c     |  64 ++---
>> > > >  drivers/infiniband/ulp/ipoib/ipoib_vlan.c      |   9 +-
>> > > >  include/rdma/ib_verbs.h                        |  41 +++
>> > > >  11 files changed, 565 insertions(+), 367 deletions(-)
>> > > >
>> > > > --
>> > > > 2.12.0
>> > > >
>> > > > --
>> > > > To unsubscribe from this list: send the line "unsubscribe
>> > > > linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
>> > > --
>> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
>> > > in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo
>> > > info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 10495 bytes --]

On Thu, Apr 06, 2017 at 08:10:14AM -0700, Vishwanathapura, Niranjana wrote:
> On Thu, Apr 06, 2017 at 08:28:20AM +0300, Leon Romanovsky wrote:
> > On Thu, Apr 06, 2017 at 12:54:02AM +0000, Weiny, Ira wrote:
> > > >
> > > > On Tue, Apr 04, 2017 at 07:50:21PM +0000, Weiny, Ira wrote:
> > > > > >
> > > > > > Hi Doug,
> > > > > >
> > > > > > This patchset mostly comes from Erez with one exception in the first patch.
> > > > > >
> > > > > > That patch origins from two different commits, first from Niranjana
> > > > > > who added RDMA netdev interface and second from Erez who added IPoIB
> > > > support.
> > > > > >
> > > > > > During the preparation to submission, I squashed their commits into
> > > > > > one and refactored code to allow submission as a standalone topic
> > > > > > without creating dependecies between different submissions. This
> > > > > > caused to change in author line.
> > > > > > I hope that it doesn't really matter and it won't stop you from merging it.
> > > > >
> > > > > I'm confused by how you expect Niranjana's patch to be merged in this
> > > > situation?
> > > > >
> > > > > Doug, will you take care of the conflict between these 2 submitted patches?
> > > > >
> > > > > [PATCH 02/11] IB/opa-vnic: Virtual Network Interface Controller (VNIC)
> > > > > interface And [PATCH rdma-next 1/6] IB/IPoIB: Introduce RDMA netdev
> > > > > interface and IPoIB structs
> > > >
> > > > Ira,
> > > >
> > > > The IPoIB and HFI-VNIC are independent series, so topics should be self-
> > > > contained. If the merge conflict worries you, Niranjana or me could easily base
> > > > our patch sets on another topic, but we need to know the acceptance (yes/no)
> > > > and order between them.
> > >
> > > I guess my expectation was that upon your agreement of this new interface, Doug would accept Niranjanas change and you could build on top of that rather than just taking part of Niranjanas patch and expecting Doug to see there are common hunks.  Perhaps I'm missing something and git is smart enough to deal with it?
> >
> > It is not related to smart git. Parallel submissions should be independent,
> > this is the why I'm submitting all our topics based on one common ground (usually -rcX),
> > despite the fact that sometimes there is merge conflict between them.
> >
> > As long Doug works in this mode, we are obligated to continue.
> >
> > >
> > > Does this mean you have reviewed and are comfortable with Niranjanas new rdma_netdev changes?  If so it seems Doug could accept them.
> >
> > We already discussed it in email and privately, I'm agree with you that IPoIB and HFI-VNIC
> > are better to be separated, due to conceptually difference and future code maintenance.
> > The proposed rdma_netdev suits well both patch sets, so I'm very comfortable with them.
> >
>
> Thanks.
>
> > However, I'm not comfortable with ULP location, for the code which is not interoperable
> > between different systems (nodes). After OFA, I realized that it wouldn't work in Susan's
> > LANL mixed cluster, which is pretty awful for the common ULP.
> >
> > The code itself needs proprietary software (ethernet manager) to be
> > operable and it placed in ULP based on some promise that future drivers
> > will be added to hfi family. Right now, it is not the case.
> >
> > You asked, I answered :)
> >
>
> These have been already discussed out before.
> That is not what Ira is asking at all.
> We have been patiently waiting for a month and a half to ensure this IPoIB
> patches can fit will with the rdma netdev interface as Jason and Doug
> wanted.
>
> Now that IPoIB is comfortable with the rdma netdev interface, I think OPA
> VNIC patch can move forward.

Sure, you can and I have no plans to stop it. It works for you, your
customers and future devices are coming.

I just want to be sure that once any vendor in this susbsystem
will use the same magic sentence "it gives best performance for our
hardware and future devices will come". There will be no questions about
interoperability between various devices, no more questions about
standards, no more requests to implement for other drivers (hardware and
software).

Or we are requiring it from everyone, or we are not requiring it from
anyone.

Thanks

>
> Niranjana
>
> > >
> > > Ira
> > >
> > > >
> > > > Thanks
> > > >
> > > > >
> > > > > Ira
> > > > >
> > > > > >
> > > > > > Per-your request, I based this patch set on v4.11-rc3.
> > > > > >
> > > > > > Thanks,
> > > > > > 	Leon
> > > > > >
> > > > > > CC: Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> > > > > > CC: Alex Vesker <valex-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> > > > > > ---
> > > > > >
> > > > > > The rest comes from Erez:
> > > > > >
> > > > > >     The IPoIB protocol encapsulates IP packets over Infiniband datagrams.
> > > > > >     As a direct RDMA Upper Layer Protocol (ULP), IPoIB cannot support HW
> > > > > >     features that are specific to the IP protocol stack.
> > > > > >
> > > > > >     Nevertheless, RDMA interfaces have been extended to support some of
> > > > the
> > > > > >     prominent IP offload features, such as TCP/UDP checksum and TSO.
> > > > > >     This provided reasonable performance gain for IPoIB but is still
> > > > > >     insufficient to cope with the increasing network bandwidth demand.
> > > > > >
> > > > > >     However, New features are exisiting in common network interfaces that
> > > > > >     are very hard to implement in IPoIB interfaces while it uses the RDMA
> > > > > >     layer, examples include TSS and RSS, tunneling offloads, and XDP.
> > > > > >     Rather than continuously porting IP network interface developments into
> > > > > >     the RDMA stack, we propose adding an abstract network data-path
> > > > > >     interfaces to RDMA devices.
> > > > > >
> > > > > >     In order to present a consistent interface to users, the IPoIB ULP
> > > > > >     continues to represent the network device to the IP stack.
> > > > > >     The common code also manages the IPoIB control plane, such as resolving
> > > > > >     path queries and registering to multicast groups.
> > > > > >     Data path operations are forwarded to devices that implement the new
> > > > > >     API, or fallback to the standard implementation otherwise.
> > > > > >     Using the forgoing approach, we show how IPoIB closes the performance
> > > > > >     gap compared to state-of-the-art Ethernet network interfaces.
> > > > > >
> > > > > >     The implementation idea is to use the api of
> > > > > >     alloc_rdma_netdev/free_rdma_netdev to expose a struct that has data
> > > > > >     members and set of functions that are used for IB network interfaces,
> > > > > >     like attach/detach multicast to qp, and send IB packet.
> > > > > >
> > > > > >     The functions are specific for IB operations and are not part of the
> > > > > >     common api the the netdev struct exposes via the ndo functions.
> > > > > >     1. multicast handling - attach/detach
> > > > > >     2. send operation - the ndo start_xmit has only 2 parameters and the IB
> > > > > >     send needs the destination qp and the ah object, there were few options
> > > > > >     to handle it via the netdev ndo, but they don't make more sense than
> > > > > >     using a specific send function (we are rdma_netdev after all)
> > > > > >
> > > > > >     The IPoIB code will be adapted to enable the option of accelerating the
> > > > > >     network interface, but the code will work as before if the HW below
> > > > > >     doesn't support the acceleration.
> > > > > >     That means that in the default mode of ipoib I tried to keep it as much
> > > > > >     as it was before, not to force it to adopt the new api, where there is
> > > > > >     no code sharing between the ipoib and the vnic/hfi.
> > > > > >     The default code uses the controll and the data, the accelerator uses
> > > > > >     only the control flows andstructors.
> > > > > >     The changes of the default ipoib can be made on top of that series.
> > > > > >
> > > > > >     Each HW vendor can supply the acceleration for the IPoIB or to leave
> > > > > >     IPoIB to work as before.
> > > > > >
> > > > > > Thanks,
> > > > > > 	Erez
> > > > > >
> > > > > > Erez Shitrit (5):
> > > > > >   IB/IPoIB: Separate control and data related initializations
> > > > > >   IB/IPoIB: Separate control from HW operation on ipoib_open/stop ndo
> > > > > >   IB/IPoIB: Rename qpn to be dqpn in ipoib_send and post_send functions
> > > > > >   IB/IPoIB: Formatting before adding accelerator to driver
> > > > > >   IB/IPoIB: Support acceleration options callbacks
> > > > > >
> > > > > > Leon Romanovsky (1):
> > > > > >   IB/IPoIB: Introduce RDMA netdev interface and IPoIB structs
> > > > > >
> > > > > >  drivers/infiniband/ulp/ipoib/ipoib.h           |  40 +--
> > > > > >  drivers/infiniband/ulp/ipoib/ipoib_cm.c        |  66 ++---
> > > > > >  drivers/infiniband/ulp/ipoib/ipoib_ethtool.c   |   6 +-
> > > > > >  drivers/infiniband/ulp/ipoib/ipoib_fs.c        |   4 +-
> > > > > >  drivers/infiniband/ulp/ipoib/ipoib_ib.c        | 339 +++++++++++++------------
> > > > > >  drivers/infiniband/ulp/ipoib/ipoib_main.c      | 312 +++++++++++++++++---
> > > > ---
> > > > > >  drivers/infiniband/ulp/ipoib/ipoib_multicast.c |  39 +--
> > > > > >  drivers/infiniband/ulp/ipoib/ipoib_netlink.c   |  12 +-
> > > > > >  drivers/infiniband/ulp/ipoib/ipoib_verbs.c     |  64 ++---
> > > > > >  drivers/infiniband/ulp/ipoib/ipoib_vlan.c      |   9 +-
> > > > > >  include/rdma/ib_verbs.h                        |  41 +++
> > > > > >  11 files changed, 565 insertions(+), 367 deletions(-)
> > > > > >
> > > > > > --
> > > > > > 2.12.0
> > > > > >
> > > > > > --
> > > > > > To unsubscribe from this list: send the line "unsubscribe
> > > > > > linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > > > --
> > > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> > > > > in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo
> > > > > info at  http://vger.kernel.org/majordomo-info.html
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Jason Gunthorpe]

On Mon, Apr 10, 2017 at 11:02:28AM +0300, Leon Romanovsky wrote:

> I just want to be sure that once any vendor in this susbsystem
> will use the same magic sentence "it gives best performance for our
> hardware and future devices will come". There will be no questions about
> interoperability between various devices, no more questions about
> standards, no more requests to implement for other drivers (hardware and
> software).

Don't be absurd..

I repeatedly chewed out intel for having their driver act like a
subsystem in of itself, keeping user-facing code that is not linked to
hardware out of driver is the right thing to do.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 982 bytes --]

On Mon, Apr 10, 2017 at 09:40:39AM -0600, Jason Gunthorpe wrote:
> On Mon, Apr 10, 2017 at 11:02:28AM +0300, Leon Romanovsky wrote:
>
> > I just want to be sure that once any vendor in this susbsystem
> > will use the same magic sentence "it gives best performance for our
> > hardware and future devices will come". There will be no questions about
> > interoperability between various devices, no more questions about
> > standards, no more requests to implement for other drivers (hardware and
> > software).
>
> Don't be absurd..
>
> I repeatedly chewed out intel for having their driver act like a
> subsystem in of itself, keeping user-facing code that is not linked to
> hardware out of driver is the right thing to do.

Jason,

Yes, you did, however for many reasons hfi ended to be subsystem on its own,
and now we are moving this subsystem one layer up. I really failed to understand
your point of view, why hfi-vnic is not linked to specific hardware.

Thanks

>
> Jason

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH rdma-next 0/6] Enhanced mode for IPoIB driver

[Jason Gunthorpe]

On Wed, Apr 12, 2017 at 08:22:23AM +0300, Leon Romanovsky wrote:

> Yes, you did, however for many reasons hfi ended to be subsystem on
> its own, and now we are moving this subsystem one layer up. I really
> failed to understand your point of view, why hfi-vnic is not linked
> to specific hardware.

Well, the ULP part is clearly not hardware specific, it uses generic
interfaces like mad and rdma netdev (eg the same generic interfaces as
ipoib) - that was part of the transformation since the first series
was presented.

We have been treating OPA as a top level protocol on par with
rocee,ib,iwarp - so an OPA specific ULP that only uses core interfaces
seems reasonable.

It has changed alot since the first series, it is no longer tightly
linked to hfi1. But it does use OPA specific MADs and requires the
driver implement the opa_vnic rdma_netdev API.

It very clearly has nothing to do with hfi1, other than hfi1 is the
only driver implementing OPA.

I think there is a path to make this work on IB if someone cared to do
it, but I think it is unreasonable to demand Intel do that work.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html