RE: [PATCH] IB/ipoib: improve latency in ipoib/cm connection formation

[Manjunath Patil <manjunath.b.patil@oracle.com>]

> On Thu, Jan 21, 2021 at 09:50:03AM -0800, Manjunath Patil wrote:
> > ipoib connected mode presently queries the device[HCA] to get P_Key
> > table entry during connection formation. This will increase the time
> > taken to form the connection, especially when limited P_Keys are in use.
> > This gets worse when multiple connection attempts are done in parallel.
> > Improve this by using the cached version of ib_find_pkey().
> >
> > This improved the latency from 500ms to 3ms on an internal setup.
> >
> > Suggested-by: Haakon Bugge <haakon.bugge@oracle.com>
> > Signed-off-by: Manjunath Patil <manjunath.b.patil@oracle.com>
> >  drivers/infiniband/ulp/ipoib/ipoib_cm.c |    4 +++-
> >  1 files changed, 3 insertions(+), 1 deletions(-)
> >
> > diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
> > b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
> > index 8f0b598..013a1d8 100644
> > +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
> > @@ -40,6 +40,7 @@
> >  #include <linux/moduleparam.h>
> >  #include <linux/sched/signal.h>
> >  #include <linux/sched/mm.h>
> > +#include <rdma/ib_cache.h>
> >
> >  #include "ipoib.h"
> >
> > @@ -1122,7 +1123,8 @@ static int ipoib_cm_modify_tx_init(struct
> net_device *dev,
> >  	struct ipoib_dev_priv *priv = ipoib_priv(dev);
> >  	struct ib_qp_attr qp_attr;
> >  	int qp_attr_mask, ret;
> > -	ret = ib_find_pkey(priv->ca, priv->port, priv->pkey,
> &qp_attr.pkey_index);
> > +	ret = ib_find_cached_pkey(priv->ca, priv->port, priv->pkey,
> > +						&qp_attr.pkey_index);
> 
> ipoib interfaces are locked to a single pkey, you should be able to get the
> pkey index that was determined at link up time and use it here instead of
> searching anything
> 
> Jason

Thank you Jason for your input. I will explore and get back to you.

-Manjunath