[PATCH 0/4] Generate UDP src port with flow label or lqpn/rqpn

[PATCH 0/4] Generate UDP src port with flow label or lqpn/rqpn

[yanjun.zhu]

From: Zhu Yanjun <yanjun.zhu@linux.dev>

Follow the advice from Leon Romanovsky, rdma_get_udp_sport is moved to
ib_verbs.h. several drivers generate udp source port with this function.

Zhu Yanjun (4):
  RDMA/core: Calculate UDP source port based on flow label or lqpn/rqpn
  RDMA/hns: Replace get_udp_sport with rdma_get_udp_sport
  RDMA/irdma: Make the source udp port vary
  RDMA/rxe: Use the standard method to produce udp source port

 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 12 ++----------
 drivers/infiniband/hw/irdma/verbs.c        |  5 +++++
 drivers/infiniband/sw/rxe/rxe_verbs.c      | 10 ++++++++++
 include/rdma/ib_verbs.h                    | 17 +++++++++++++++++
 4 files changed, 34 insertions(+), 10 deletions(-)

-- 
2.27.0

[PATCH 1/4] RDMA/core: Calculate UDP source port based on flow label or lqpn/rqpn

[yanjun.zhu]

From: Zhu Yanjun <yanjun.zhu@linux.dev>

Calculate and set UDP source port based on the flow label. If flow label
is not defined in GRH then calculate it based on lqpn/rqpn.

Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
---
 include/rdma/ib_verbs.h | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 6e9ad656ecb7..2f122aa81f0f 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -4749,6 +4749,23 @@ static inline u32 rdma_calc_flow_label(u32 lqpn, u32 rqpn)
 	return (u32)(v & IB_GRH_FLOWLABEL_MASK);
 }
 
+/**
+ * rdma_get_udp_sport - Calculate and set UDP source port based on the flow
+ *                      label. If flow label is not defined in GRH then
+ *                      calculate it based on lqpn/rqpn.
+ *
+ * @fl:			flow label from GRH
+ * @lqpn:		local qp number
+ * @rqpn:		remote qp number
+ */
+static inline u16 rdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
+{
+	if (!fl)
+		fl = rdma_calc_flow_label(lqpn, rqpn);
+
+	return rdma_flow_label_to_udp_sport(fl);
+}
+
 const struct ib_port_immutable*
 ib_port_immutable_read(struct ib_device *dev, unsigned int port);
 #endif /* IB_VERBS_H */
-- 
2.27.0

Re: [PATCH 1/4] RDMA/core: Calculate UDP source port based on flow label or lqpn/rqpn

[Mark Zhang]

On 1/5/2022 4:07 PM, yanjun.zhu@linux.dev wrote:
> From: Zhu Yanjun <yanjun.zhu@linux.dev>
> 
> Calculate and set UDP source port based on the flow label. If flow label
> is not defined in GRH then calculate it based on lqpn/rqpn.
> 
> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
> ---
>   include/rdma/ib_verbs.h | 17 +++++++++++++++++
>   1 file changed, 17 insertions(+)
> 
> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
> index 6e9ad656ecb7..2f122aa81f0f 100644
> --- a/include/rdma/ib_verbs.h
> +++ b/include/rdma/ib_verbs.h
> @@ -4749,6 +4749,23 @@ static inline u32 rdma_calc_flow_label(u32 lqpn, u32 rqpn)
>   	return (u32)(v & IB_GRH_FLOWLABEL_MASK);
>   }
>   
> +/**
> + * rdma_get_udp_sport - Calculate and set UDP source port based on the flow
> + *                      label. If flow label is not defined in GRH then
> + *                      calculate it based on lqpn/rqpn.
> + *
> + * @fl:			flow label from GRH

Indent:
+ * @fl:		flow label from GRH

> + * @lqpn:		local qp number
> + * @rqpn:		remote qp number
> + */
> +static inline u16 rdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
> +{
> +	if (!fl)
> +		fl = rdma_calc_flow_label(lqpn, rqpn);
> +
> +	return rdma_flow_label_to_udp_sport(fl);
> +}
> + >   const struct ib_port_immutable*
>   ib_port_immutable_read(struct ib_device *dev, unsigned int port);
>   #endif /* IB_VERBS_H */

Maybe this and next patch can be squashed into one?

Re: [PATCH 1/4] RDMA/core: Calculate UDP source port based on flow label or lqpn/rqpn

[Yanjun Zhu]

在 2022/1/5 21:51, Mark Zhang 写道:
> On 1/5/2022 4:07 PM, yanjun.zhu@linux.dev wrote:
>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>
>> Calculate and set UDP source port based on the flow label. If flow label
>> is not defined in GRH then calculate it based on lqpn/rqpn.
>>
>> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
>> ---
>>   include/rdma/ib_verbs.h | 17 +++++++++++++++++
>>   1 file changed, 17 insertions(+)
>>
>> diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
>> index 6e9ad656ecb7..2f122aa81f0f 100644
>> --- a/include/rdma/ib_verbs.h
>> +++ b/include/rdma/ib_verbs.h
>> @@ -4749,6 +4749,23 @@ static inline u32 rdma_calc_flow_label(u32 
>> lqpn, u32 rqpn)
>>       return (u32)(v & IB_GRH_FLOWLABEL_MASK);
>>   }
>>   +/**
>> + * rdma_get_udp_sport - Calculate and set UDP source port based on 
>> the flow
>> + *                      label. If flow label is not defined in GRH then
>> + *                      calculate it based on lqpn/rqpn.
>> + *
>> + * @fl:            flow label from GRH
>
> Indent:
> + * @fl:        flow label from GRH

Thanks. In V2, this problem is fixed.

>
>> + * @lqpn:        local qp number
>> + * @rqpn:        remote qp number
>> + */
>> +static inline u16 rdma_get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
>> +{
>> +    if (!fl)
>> +        fl = rdma_calc_flow_label(lqpn, rqpn);
>> +
>> +    return rdma_flow_label_to_udp_sport(fl);
>> +}
>> + >   const struct ib_port_immutable*
>>   ib_port_immutable_read(struct ib_device *dev, unsigned int port);
>>   #endif /* IB_VERBS_H */
>
> Maybe this and next patch can be squashed into one?

Why do you think this commit and next commit should be squashed into one?

Best Regards,

Zhu Yanjun

[PATCH 2/4] RDMA/hns: Replace get_udp_sport with rdma_get_udp_sport

[yanjun.zhu]

From: Zhu Yanjun <yanjun.zhu@linux.dev>

Several drivers have the same function xxx_get_udp_sport. So this
function is moved to ib_verbs.h.

Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
---
 drivers/infiniband/hw/hns/hns_roce_hw_v2.c | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)

diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
index eb0defa80d0d..cb795663b813 100644
--- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
+++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
@@ -4488,14 +4488,6 @@ static int modify_qp_rtr_to_rts(struct ib_qp *ibqp,
 	return 0;
 }
 
-static inline u16 get_udp_sport(u32 fl, u32 lqpn, u32 rqpn)
-{
-	if (!fl)
-		fl = rdma_calc_flow_label(lqpn, rqpn);
-
-	return rdma_flow_label_to_udp_sport(fl);
-}
-
 static int get_dip_ctx_idx(struct ib_qp *ibqp, const struct ib_qp_attr *attr,
 			   u32 *dip_idx)
 {
@@ -4712,8 +4704,8 @@ static int hns_roce_v2_set_path(struct ib_qp *ibqp,
 	}
 
 	hr_reg_write(context, QPC_UDPSPN,
-		     is_udp ? get_udp_sport(grh->flow_label, ibqp->qp_num,
-					    attr->dest_qp_num) : 0);
+		     is_udp ? rdma_get_udp_sport(grh->flow_label, ibqp->qp_num,
+						 attr->dest_qp_num) : 0);
 
 	hr_reg_clear(qpc_mask, QPC_UDPSPN);
 
-- 
2.27.0

[PATCH 3/4] RDMA/irdma: Make the source udp port vary

[yanjun.zhu]

From: Zhu Yanjun <yanjun.zhu@linux.dev>

Get the source udp port number for a QP based on the grh.flow_label or
lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues.

Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
---
 drivers/infiniband/hw/irdma/verbs.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c
index 8cd5f9261692..09dba7ed5ab9 100644
--- a/drivers/infiniband/hw/irdma/verbs.c
+++ b/drivers/infiniband/hw/irdma/verbs.c
@@ -1167,6 +1167,11 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 
 		memset(&iwqp->roce_ah, 0, sizeof(iwqp->roce_ah));
 		if (attr->ah_attr.ah_flags & IB_AH_GRH) {
+			u32 fl = attr->ah_attr.grh.flow_label;
+			u32 lqp = ibqp->qp_num;
+			u32 rqp = roce_info->dest_qp;
+
+			udp_info->src_port = rdma_get_udp_sport(fl, lqp, rqp);
 			udp_info->ttl = attr->ah_attr.grh.hop_limit;
 			udp_info->flow_label = attr->ah_attr.grh.flow_label;
 			udp_info->tos = attr->ah_attr.grh.traffic_class;
-- 
2.27.0

RE: [PATCH 3/4] RDMA/irdma: Make the source udp port vary

[Saleem, Shiraz]

> Subject: [PATCH 3/4] RDMA/irdma: Make the source udp port vary
> 
> From: Zhu Yanjun <yanjun.zhu@linux.dev>
> 
> Get the source udp port number for a QP based on the grh.flow_label or
> lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues.
> 
> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
> ---
>  drivers/infiniband/hw/irdma/verbs.c | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c
> index 8cd5f9261692..09dba7ed5ab9 100644
> --- a/drivers/infiniband/hw/irdma/verbs.c
> +++ b/drivers/infiniband/hw/irdma/verbs.c
> @@ -1167,6 +1167,11 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct
> ib_qp_attr *attr,
> 
>  		memset(&iwqp->roce_ah, 0, sizeof(iwqp->roce_ah));
>  		if (attr->ah_attr.ah_flags & IB_AH_GRH) {
> +			u32 fl = attr->ah_attr.grh.flow_label;
> +			u32 lqp = ibqp->qp_num;
> +			u32 rqp = roce_info->dest_qp;
> +
> +	
Do you really need these locals?

		udp_info->src_port = rdma_get_udp_sport(fl, lqp, rqp);
>  			udp_info->ttl = attr->ah_attr.grh.hop_limit;
>  			udp_info->flow_label = attr->ah_attr.grh.flow_label;
>  			udp_info->tos = attr->ah_attr.grh.traffic_class;
> --
> 2.27.0

Re: [PATCH 3/4] RDMA/irdma: Make the source udp port vary

[Leon Romanovsky]

On Tue, Jan 04, 2022 at 05:18:01PM +0000, Saleem, Shiraz wrote:
> > Subject: [PATCH 3/4] RDMA/irdma: Make the source udp port vary
> > 
> > From: Zhu Yanjun <yanjun.zhu@linux.dev>
> > 
> > Get the source udp port number for a QP based on the grh.flow_label or
> > lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues.
> > 
> > Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
> > ---
> >  drivers/infiniband/hw/irdma/verbs.c | 5 +++++
> >  1 file changed, 5 insertions(+)
> > 
> > diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c
> > index 8cd5f9261692..09dba7ed5ab9 100644
> > --- a/drivers/infiniband/hw/irdma/verbs.c
> > +++ b/drivers/infiniband/hw/irdma/verbs.c
> > @@ -1167,6 +1167,11 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct
> > ib_qp_attr *attr,
> > 
> >  		memset(&iwqp->roce_ah, 0, sizeof(iwqp->roce_ah));
> >  		if (attr->ah_attr.ah_flags & IB_AH_GRH) {
> > +			u32 fl = attr->ah_attr.grh.flow_label;
> > +			u32 lqp = ibqp->qp_num;
> > +			u32 rqp = roce_info->dest_qp;
> > +
> > +	
> Do you really need these locals?

I asked same question in previous revision.

Zhu, please remove them.

Thanks

Re: [PATCH 3/4] RDMA/irdma: Make the source udp port vary

[Yanjun Zhu]

在 2022/1/5 3:26, Leon Romanovsky 写道:
> On Tue, Jan 04, 2022 at 05:18:01PM +0000, Saleem, Shiraz wrote:
>>> Subject: [PATCH 3/4] RDMA/irdma: Make the source udp port vary
>>>
>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>>
>>> Get the source udp port number for a QP based on the grh.flow_label or
>>> lqpn/rqrpn. This provides a better spread of traffic across NIC RX queues.
>>>
>>> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
>>> ---
>>>   drivers/infiniband/hw/irdma/verbs.c | 5 +++++
>>>   1 file changed, 5 insertions(+)
>>>
>>> diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c
>>> index 8cd5f9261692..09dba7ed5ab9 100644
>>> --- a/drivers/infiniband/hw/irdma/verbs.c
>>> +++ b/drivers/infiniband/hw/irdma/verbs.c
>>> @@ -1167,6 +1167,11 @@ int irdma_modify_qp_roce(struct ib_qp *ibqp, struct
>>> ib_qp_attr *attr,
>>>
>>>   		memset(&iwqp->roce_ah, 0, sizeof(iwqp->roce_ah));
>>>   		if (attr->ah_attr.ah_flags & IB_AH_GRH) {
>>> +			u32 fl = attr->ah_attr.grh.flow_label;
>>> +			u32 lqp = ibqp->qp_num;
>>> +			u32 rqp = roce_info->dest_qp;
>>> +
>>> +	
>> Do you really need these locals?
> 
> I asked same question in previous revision.
> 
> Zhu, please remove them.

Hi, Leon Romanovsky && Saleem, Shiraz

Without these local variables, the line "udp_info->src_port = 
rdma_get_udp_sport(fl, lqp, rqp);" will exceed 80. This will cause 
warning when this commit is checked by script/checkpatch.pl.

Let me have a try to find a better way to avoid these local variables.

Zhu Yanjun

> 
> Thanks

[PATCH 4/4] RDMA/rxe: Use the standard method to produce udp source port

[yanjun.zhu]

From: Zhu Yanjun <yanjun.zhu@linux.dev>

Use the standard method to produce udp source port.

Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
---
 drivers/infiniband/sw/rxe/rxe_verbs.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index 0aa0d7e52773..f30d98ad13cd 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -469,6 +469,16 @@ static int rxe_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 	if (err)
 		goto err1;
 
+	if (mask & IB_QP_AV) {
+		if (attr->ah_attr.ah_flags & IB_AH_GRH) {
+			u32 fl = attr->ah_attr.grh.flow_label;
+			u32 lqp = qp->ibqp.qp_num;
+			u32 rqp = qp->attr.dest_qp_num;
+
+			qp->src_port = rdma_get_udp_sport(fl, lqp, rqp);
+		}
+	}
+
 	return 0;
 
 err1:
-- 
2.27.0

RE: [PATCH 4/4] RDMA/rxe: Use the standard method to produce udp source port

[Saleem, Shiraz]

> Subject: [PATCH 4/4] RDMA/rxe: Use the standard method to produce udp source
> port
> 
> From: Zhu Yanjun <yanjun.zhu@linux.dev>
> 
> Use the standard method to produce udp source port.
> 
> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
> ---
>  drivers/infiniband/sw/rxe/rxe_verbs.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c
> b/drivers/infiniband/sw/rxe/rxe_verbs.c
> index 0aa0d7e52773..f30d98ad13cd 100644
> --- a/drivers/infiniband/sw/rxe/rxe_verbs.c
> +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
> @@ -469,6 +469,16 @@ static int rxe_modify_qp(struct ib_qp *ibqp, struct
> ib_qp_attr *attr,
>  	if (err)
>  		goto err1;
> 
> +	if (mask & IB_QP_AV) {
> +		if (attr->ah_attr.ah_flags & IB_AH_GRH) {
> +			u32 fl = attr->ah_attr.grh.flow_label;
> +			u32 lqp = qp->ibqp.qp_num;
> +			u32 rqp = qp->attr.dest_qp_num;
> +
Isn't the randomization for src_port done in rxe_qp_init_req redundant then?

https://elixir.bootlin.com/linux/v5.16-rc8/source/drivers/infiniband/sw/rxe/rxe_qp.c#L220

Can we remove it?

> +			qp->src_port = rdma_get_udp_sport(fl, lqp, rqp);
> +		}
> +	}
> +
>  	return 0;
> 
>  err1:
> --
> 2.27.0

Re: [PATCH 4/4] RDMA/rxe: Use the standard method to produce udp source port

[Yanjun Zhu]

在 2022/1/5 1:17, Saleem, Shiraz 写道:
>> Subject: [PATCH 4/4] RDMA/rxe: Use the standard method to produce udp source
>> port
>>
>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>
>> Use the standard method to produce udp source port.
>>
>> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
>> ---
>>   drivers/infiniband/sw/rxe/rxe_verbs.c | 10 ++++++++++
>>   1 file changed, 10 insertions(+)
>>
>> diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c
>> b/drivers/infiniband/sw/rxe/rxe_verbs.c
>> index 0aa0d7e52773..f30d98ad13cd 100644
>> --- a/drivers/infiniband/sw/rxe/rxe_verbs.c
>> +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
>> @@ -469,6 +469,16 @@ static int rxe_modify_qp(struct ib_qp *ibqp, struct
>> ib_qp_attr *attr,
>>   	if (err)
>>   		goto err1;
>>
>> +	if (mask & IB_QP_AV) {
>> +		if (attr->ah_attr.ah_flags & IB_AH_GRH) {
>> +			u32 fl = attr->ah_attr.grh.flow_label;
>> +			u32 lqp = qp->ibqp.qp_num;
>> +			u32 rqp = qp->attr.dest_qp_num;
>> +
> Isn't the randomization for src_port done in rxe_qp_init_req redundant then?
> 
> https://elixir.bootlin.com/linux/v5.16-rc8/source/drivers/infiniband/sw/rxe/rxe_qp.c#L220
> 
> Can we remove it?

Yes. We can remove it.
Because this "randomization for src_port done in  rxe_qp_init_req" is 
replaced by rdma_get_udp_sport in rxe_modify_qp, I do not remove it.

I will remove it in the latest commits soon.

Zhu Yanjun

> 
>> +			qp->src_port = rdma_get_udp_sport(fl, lqp, rqp);
>> +		}
>> +	}
>> +
>>   	return 0;
>>
>>   err1:
>> --
>> 2.27.0
> 

Re: [PATCH 4/4] RDMA/rxe: Use the standard method to produce udp source port

[Yanjun Zhu]

在 2022/1/5 8:33, Yanjun Zhu 写道:
> 在 2022/1/5 1:17, Saleem, Shiraz 写道:
>>> Subject: [PATCH 4/4] RDMA/rxe: Use the standard method to produce 
>>> udp source
>>> port
>>>
>>> From: Zhu Yanjun <yanjun.zhu@linux.dev>
>>>
>>> Use the standard method to produce udp source port.
>>>
>>> Signed-off-by: Zhu Yanjun <yanjun.zhu@linux.dev>
>>> ---
>>>   drivers/infiniband/sw/rxe/rxe_verbs.c | 10 ++++++++++
>>>   1 file changed, 10 insertions(+)
>>>
>>> diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c
>>> b/drivers/infiniband/sw/rxe/rxe_verbs.c
>>> index 0aa0d7e52773..f30d98ad13cd 100644
>>> --- a/drivers/infiniband/sw/rxe/rxe_verbs.c
>>> +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
>>> @@ -469,6 +469,16 @@ static int rxe_modify_qp(struct ib_qp *ibqp, 
>>> struct
>>> ib_qp_attr *attr,
>>>       if (err)
>>>           goto err1;
>>>
>>> +    if (mask & IB_QP_AV) {
>>> +        if (attr->ah_attr.ah_flags & IB_AH_GRH) {
>>> +            u32 fl = attr->ah_attr.grh.flow_label;
>>> +            u32 lqp = qp->ibqp.qp_num;
>>> +            u32 rqp = qp->attr.dest_qp_num;
>>> +
>> Isn't the randomization for src_port done in rxe_qp_init_req 
>> redundant then?
>>
>> https://elixir.bootlin.com/linux/v5.16-rc8/source/drivers/infiniband/sw/rxe/rxe_qp.c#L220 
>>
>>
>> Can we remove it?

Based on the discussion with Leon Romanovsky, when in-subnet 
communications, GRH is optional.

So It is possible that GRH is not set. Without the randomization for 
src_port done in rxe_qp_init_req, udp source

port will be 0xC000 in that case.

So it is better to keep this randomization for src_port done in 
rxe_qp_init_req.

Zhu Yanjun

>
> Yes. We can remove it.
> Because this "randomization for src_port done in  rxe_qp_init_req" is 
> replaced by rdma_get_udp_sport in rxe_modify_qp, I do not remove it.
>
> I will remove it in the latest commits soon.
>
> Zhu Yanjun
>
>>
>>> +            qp->src_port = rdma_get_udp_sport(fl, lqp, rqp);
>>> +        }
>>> +    }
>>> +
>>>       return 0;
>>>
>>>   err1:
>>> -- 
>>> 2.27.0
>>
>