* [PATCH] IB/ipoib: improve latency in ipoib/cm connection formation
@ 2021-01-21 17:50 Manjunath Patil
2021-01-21 18:16 ` Jason Gunthorpe
0 siblings, 1 reply; 6+ messages in thread
From: Manjunath Patil @ 2021-01-21 17:50 UTC (permalink / raw)
To: dledford, jgg, leon, valentinef, gustavoars
Cc: haakon.bugge, manjunath.b.patil, linux-rdma
ipoib connected mode presently queries the device[HCA] to get P_Key
table entry during connection formation. This will increase the time
taken to form the connection, especially when limited P_Keys are in use.
This gets worse when multiple connection attempts are done in parallel.
Improve this by using the cached version of ib_find_pkey().
This improved the latency from 500ms to 3ms on an internal setup.
Suggested-by: Haakon Bugge <haakon.bugge@oracle.com>
Signed-off-by: Manjunath Patil <manjunath.b.patil@oracle.com>
---
drivers/infiniband/ulp/ipoib/ipoib_cm.c | 4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
index 8f0b598..013a1d8 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
@@ -40,6 +40,7 @@
#include <linux/moduleparam.h>
#include <linux/sched/signal.h>
#include <linux/sched/mm.h>
+#include <rdma/ib_cache.h>
#include "ipoib.h"
@@ -1122,7 +1123,8 @@ static int ipoib_cm_modify_tx_init(struct net_device *dev,
struct ipoib_dev_priv *priv = ipoib_priv(dev);
struct ib_qp_attr qp_attr;
int qp_attr_mask, ret;
- ret = ib_find_pkey(priv->ca, priv->port, priv->pkey, &qp_attr.pkey_index);
+ ret = ib_find_cached_pkey(priv->ca, priv->port, priv->pkey,
+ &qp_attr.pkey_index);
if (ret) {
ipoib_warn(priv, "pkey 0x%x not found: %d\n", priv->pkey, ret);
return ret;
--
1.7.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH] IB/ipoib: improve latency in ipoib/cm connection formation
2021-01-21 17:50 [PATCH] IB/ipoib: improve latency in ipoib/cm connection formation Manjunath Patil
@ 2021-01-21 18:16 ` Jason Gunthorpe
2021-01-25 18:49 ` Manjunath Patil
0 siblings, 1 reply; 6+ messages in thread
From: Jason Gunthorpe @ 2021-01-21 18:16 UTC (permalink / raw)
To: Manjunath Patil
Cc: dledford, leon, valentinef, gustavoars, haakon.bugge, linux-rdma
On Thu, Jan 21, 2021 at 09:50:03AM -0800, Manjunath Patil wrote:
> ipoib connected mode presently queries the device[HCA] to get P_Key
> table entry during connection formation. This will increase the time
> taken to form the connection, especially when limited P_Keys are in use.
> This gets worse when multiple connection attempts are done in parallel.
> Improve this by using the cached version of ib_find_pkey().
>
> This improved the latency from 500ms to 3ms on an internal setup.
>
> Suggested-by: Haakon Bugge <haakon.bugge@oracle.com>
> Signed-off-by: Manjunath Patil <manjunath.b.patil@oracle.com>
> drivers/infiniband/ulp/ipoib/ipoib_cm.c | 4 +++-
> 1 files changed, 3 insertions(+), 1 deletions(-)
>
> diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
> index 8f0b598..013a1d8 100644
> +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
> @@ -40,6 +40,7 @@
> #include <linux/moduleparam.h>
> #include <linux/sched/signal.h>
> #include <linux/sched/mm.h>
> +#include <rdma/ib_cache.h>
>
> #include "ipoib.h"
>
> @@ -1122,7 +1123,8 @@ static int ipoib_cm_modify_tx_init(struct net_device *dev,
> struct ipoib_dev_priv *priv = ipoib_priv(dev);
> struct ib_qp_attr qp_attr;
> int qp_attr_mask, ret;
> - ret = ib_find_pkey(priv->ca, priv->port, priv->pkey, &qp_attr.pkey_index);
> + ret = ib_find_cached_pkey(priv->ca, priv->port, priv->pkey,
> + &qp_attr.pkey_index);
ipoib interfaces are locked to a single pkey, you should be able to
get the pkey index that was determined at link up time and use it here
instead of searching anything
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: [PATCH] IB/ipoib: improve latency in ipoib/cm connection formation
2021-01-21 18:16 ` Jason Gunthorpe
@ 2021-01-25 18:49 ` Manjunath Patil
2021-01-26 20:01 ` Haakon Bugge
2021-01-26 20:28 ` Haakon Bugge
0 siblings, 2 replies; 6+ messages in thread
From: Manjunath Patil @ 2021-01-25 18:49 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: dledford, leon, valentinef, gustavoars, Haakon Bugge, linux-rdma
> On Thu, Jan 21, 2021 at 09:50:03AM -0800, Manjunath Patil wrote:
> > ipoib connected mode presently queries the device[HCA] to get P_Key
> > table entry during connection formation. This will increase the time
> > taken to form the connection, especially when limited P_Keys are in use.
> > This gets worse when multiple connection attempts are done in parallel.
> > Improve this by using the cached version of ib_find_pkey().
> >
> > This improved the latency from 500ms to 3ms on an internal setup.
> >
> > Suggested-by: Haakon Bugge <haakon.bugge@oracle.com>
> > Signed-off-by: Manjunath Patil <manjunath.b.patil@oracle.com>
> > drivers/infiniband/ulp/ipoib/ipoib_cm.c | 4 +++-
> > 1 files changed, 3 insertions(+), 1 deletions(-)
> >
> > diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
> > b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
> > index 8f0b598..013a1d8 100644
> > +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
> > @@ -40,6 +40,7 @@
> > #include <linux/moduleparam.h>
> > #include <linux/sched/signal.h>
> > #include <linux/sched/mm.h>
> > +#include <rdma/ib_cache.h>
> >
> > #include "ipoib.h"
> >
> > @@ -1122,7 +1123,8 @@ static int ipoib_cm_modify_tx_init(struct
> net_device *dev,
> > struct ipoib_dev_priv *priv = ipoib_priv(dev);
> > struct ib_qp_attr qp_attr;
> > int qp_attr_mask, ret;
> > - ret = ib_find_pkey(priv->ca, priv->port, priv->pkey,
> &qp_attr.pkey_index);
> > + ret = ib_find_cached_pkey(priv->ca, priv->port, priv->pkey,
> > + &qp_attr.pkey_index);
>
> ipoib interfaces are locked to a single pkey, you should be able to get the
> pkey index that was determined at link up time and use it here instead of
> searching anything
>
> Jason
Thank you Jason for your input. I will explore and get back to you.
-Manjunath
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] IB/ipoib: improve latency in ipoib/cm connection formation
2021-01-25 18:49 ` Manjunath Patil
@ 2021-01-26 20:01 ` Haakon Bugge
2021-01-26 20:28 ` Haakon Bugge
1 sibling, 0 replies; 6+ messages in thread
From: Haakon Bugge @ 2021-01-26 20:01 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: dledford, leon, valentinef, gustavoars, OFED mailing list,
Manjunath Patil
> On 25 Jan 2021, at 19:49, Manjunath Patil <manjunath.b.patil@oracle.com> wrote:
>
>> On Thu, Jan 21, 2021 at 09:50:03AM -0800, Manjunath Patil wrote:
>>> ipoib connected mode presently queries the device[HCA] to get P_Key
>>> table entry during connection formation. This will increase the time
>>> taken to form the connection, especially when limited P_Keys are in use.
>>> This gets worse when multiple connection attempts are done in parallel.
>>> Improve this by using the cached version of ib_find_pkey().
>>>
>>> This improved the latency from 500ms to 3ms on an internal setup.
>>>
>>> Suggested-by: Haakon Bugge <haakon.bugge@oracle.com>
>>> Signed-off-by: Manjunath Patil <manjunath.b.patil@oracle.com>
>>> drivers/infiniband/ulp/ipoib/ipoib_cm.c | 4 +++-
>>> 1 files changed, 3 insertions(+), 1 deletions(-)
>>>
>>> diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
>>> b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
>>> index 8f0b598..013a1d8 100644
>>> +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
>>> @@ -40,6 +40,7 @@
>>> #include <linux/moduleparam.h>
>>> #include <linux/sched/signal.h>
>>> #include <linux/sched/mm.h>
>>> +#include <rdma/ib_cache.h>
>>>
>>> #include "ipoib.h"
>>>
>>> @@ -1122,7 +1123,8 @@ static int ipoib_cm_modify_tx_init(struct
>> net_device *dev,
>>> struct ipoib_dev_priv *priv = ipoib_priv(dev);
>>> struct ib_qp_attr qp_attr;
>>> int qp_attr_mask, ret;
>>> - ret = ib_find_pkey(priv->ca, priv->port, priv->pkey,
>> &qp_attr.pkey_index);
>>> + ret = ib_find_cached_pkey(priv->ca, priv->port, priv->pkey,
>>> + &qp_attr.pkey_index);
>>
>> ipoib interfaces are locked to a single pkey, you should be able to get the
>> pkey index that was determined at link up time and use it here instead of
>> searching anything
Isn't possible to:
# ip link add DEVICE name NAME type ipoib [ pkey PKEY ]
?
Thxs, Håkon
>>
>> Jason
>
> Thank you Jason for your input. I will explore and get back to you.
>
> -Manjunath
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] IB/ipoib: improve latency in ipoib/cm connection formation
2021-01-25 18:49 ` Manjunath Patil
2021-01-26 20:01 ` Haakon Bugge
@ 2021-01-26 20:28 ` Haakon Bugge
2021-01-27 0:16 ` Jason Gunthorpe
1 sibling, 1 reply; 6+ messages in thread
From: Haakon Bugge @ 2021-01-26 20:28 UTC (permalink / raw)
To: Jason Gunthorpe
Cc: dledford, leon, valentinef, gustavoars, OFED mailing list,
Manjunath Patil
> On 25 Jan 2021, at 19:49, Manjunath Patil <manjunath.b.patil@oracle.com> wrote:
>
>> On Thu, Jan 21, 2021 at 09:50:03AM -0800, Manjunath Patil wrote:
>>> ipoib connected mode presently queries the device[HCA] to get P_Key
>>> table entry during connection formation. This will increase the time
>>> taken to form the connection, especially when limited P_Keys are in use.
>>> This gets worse when multiple connection attempts are done in parallel.
>>> Improve this by using the cached version of ib_find_pkey().
>>>
>>> This improved the latency from 500ms to 3ms on an internal setup.
>>>
>>> Suggested-by: Haakon Bugge <haakon.bugge@oracle.com>
>>> Signed-off-by: Manjunath Patil <manjunath.b.patil@oracle.com>
>>> drivers/infiniband/ulp/ipoib/ipoib_cm.c | 4 +++-
>>> 1 files changed, 3 insertions(+), 1 deletions(-)
>>>
>>> diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
>>> b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
>>> index 8f0b598..013a1d8 100644
>>> +++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
>>> @@ -40,6 +40,7 @@
>>> #include <linux/moduleparam.h>
>>> #include <linux/sched/signal.h>
>>> #include <linux/sched/mm.h>
>>> +#include <rdma/ib_cache.h>
>>>
>>> #include "ipoib.h"
>>>
>>> @@ -1122,7 +1123,8 @@ static int ipoib_cm_modify_tx_init(struct
>> net_device *dev,
>>> struct ipoib_dev_priv *priv = ipoib_priv(dev);
>>> struct ib_qp_attr qp_attr;
>>> int qp_attr_mask, ret;
>>> - ret = ib_find_pkey(priv->ca, priv->port, priv->pkey,
>> &qp_attr.pkey_index);
>>> + ret = ib_find_cached_pkey(priv->ca, priv->port, priv->pkey,
>>> + &qp_attr.pkey_index);
>>
>> ipoib interfaces are locked to a single pkey, you should be able to get the
>> pkey index that was determined at link up time and use it here instead of
>> searching anything
Isn't possible to:
# ip link add DEVICE name NAME type ipoib [ pkey PKEY ]
?
Thxs, Håkon
>>
>> Jason
>
> Thank you Jason for your input. I will explore and get back to you.
>
> -Manjunath
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] IB/ipoib: improve latency in ipoib/cm connection formation
2021-01-26 20:28 ` Haakon Bugge
@ 2021-01-27 0:16 ` Jason Gunthorpe
0 siblings, 0 replies; 6+ messages in thread
From: Jason Gunthorpe @ 2021-01-27 0:16 UTC (permalink / raw)
To: Haakon Bugge
Cc: dledford, leon, valentinef, gustavoars, OFED mailing list,
Manjunath Patil
On Tue, Jan 26, 2021 at 08:28:18PM +0000, Haakon Bugge wrote:
> >>> @@ -1122,7 +1123,8 @@ static int ipoib_cm_modify_tx_init(struct
> >> net_device *dev,
> >>> struct ipoib_dev_priv *priv = ipoib_priv(dev);
> >>> struct ib_qp_attr qp_attr;
> >>> int qp_attr_mask, ret;
> >>> - ret = ib_find_pkey(priv->ca, priv->port, priv->pkey,
> >> &qp_attr.pkey_index);
> >>> + ret = ib_find_cached_pkey(priv->ca, priv->port, priv->pkey,
> >>> + &qp_attr.pkey_index);
> >>
> >> ipoib interfaces are locked to a single pkey, you should be able to get the
> >> pkey index that was determined at link up time and use it here instead of
> >> searching anything
>
> Isn't possible to:
>
> # ip link add DEVICE name NAME type ipoib [ pkey PKEY ]
>
> ?
Yes, and each new netdev that spawns has a fixed pkey that doesn't
change for the life of the netdev
Jason
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-01-27 4:22 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-21 17:50 [PATCH] IB/ipoib: improve latency in ipoib/cm connection formation Manjunath Patil
2021-01-21 18:16 ` Jason Gunthorpe
2021-01-25 18:49 ` Manjunath Patil
2021-01-26 20:01 ` Haakon Bugge
2021-01-26 20:28 ` Haakon Bugge
2021-01-27 0:16 ` Jason Gunthorpe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.