[RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Xiao Yang]

Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
module cannot set imm_data in WC when the related WR with imm_data
finished on SQ.

Signed-off-by: Xiao Yang <yangx.jy@cn.fujitsu.com>
---

Current rxe module and other rdma modules(e.g. mlx5) only set
imm_data in WC when the related WR with imm_data finished on RQ.
I am not sure if it is a expected behavior.

 drivers/infiniband/sw/rxe/rxe_comp.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c
index 0a1e6393250b..e380ff63ab2d 100644
--- a/drivers/infiniband/sw/rxe/rxe_comp.c
+++ b/drivers/infiniband/sw/rxe/rxe_comp.c
@@ -384,8 +384,10 @@ static void make_send_cqe(struct rxe_qp *qp, struct rxe_send_wqe *wqe,
 		wc->status		= wqe->status;
 		wc->opcode		= wr_to_wc_opcode(wqe->wr.opcode);
 		if (wqe->wr.opcode == IB_WR_RDMA_WRITE_WITH_IMM ||
-		    wqe->wr.opcode == IB_WR_SEND_WITH_IMM)
+		    wqe->wr.opcode == IB_WR_SEND_WITH_IMM) {
 			wc->wc_flags = IB_WC_WITH_IMM;
+			wc->ex.imm_data = wqe->wr.ex.imm_data;
+		}
 		wc->byte_len		= wqe->dma.length;
 		wc->qp			= &qp->ibqp;
 	} else {
@@ -395,8 +397,10 @@ static void make_send_cqe(struct rxe_qp *qp, struct rxe_send_wqe *wqe,
 		uwc->status		= wqe->status;
 		uwc->opcode		= wr_to_wc_opcode(wqe->wr.opcode);
 		if (wqe->wr.opcode == IB_WR_RDMA_WRITE_WITH_IMM ||
-		    wqe->wr.opcode == IB_WR_SEND_WITH_IMM)
+		    wqe->wr.opcode == IB_WR_SEND_WITH_IMM) {
 			uwc->wc_flags = IB_WC_WITH_IMM;
+			uwc->ex.imm_data = wqe->wr.ex.imm_data;
+		}
 		uwc->byte_len		= wqe->dma.length;
 		uwc->qp_num		= qp->ibqp.qp_num;
 	}
-- 
2.23.0

Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Leon Romanovsky]

On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
> module cannot set imm_data in WC when the related WR with imm_data
> finished on SQ.
>
> Signed-off-by: Xiao Yang <yangx.jy@cn.fujitsu.com>
> ---
>
> Current rxe module and other rdma modules(e.g. mlx5) only set
> imm_data in WC when the related WR with imm_data finished on RQ.
> I am not sure if it is a expected behavior.

This is IBTA behavior.

5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
"Immediate Data (ImmDt) contains data that is placed in the receive
 Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
 RDMA WRITE packets with Immediate Data."

If I understand the spec, you shouldn't set imm_data in SQ.

Thanks

Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Zhu Yanjun]

On Wed, Jan 27, 2021 at 8:10 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
> > Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
> > module cannot set imm_data in WC when the related WR with imm_data
> > finished on SQ.
> >
> > Signed-off-by: Xiao Yang <yangx.jy@cn.fujitsu.com>
> > ---
> >
> > Current rxe module and other rdma modules(e.g. mlx5) only set
> > imm_data in WC when the related WR with imm_data finished on RQ.
> > I am not sure if it is a expected behavior.
>
> This is IBTA behavior.
>
> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
> "Immediate Data (ImmDt) contains data that is placed in the receive
>  Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>  RDMA WRITE packets with Immediate Data."

Cool!

Zhu Yanjun

>
> If I understand the spec, you shouldn't set imm_data in SQ.
>
> Thanks

Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Bob Pearson]

On 1/27/21 6:04 AM, Leon Romanovsky wrote:
> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
>> module cannot set imm_data in WC when the related WR with imm_data
>> finished on SQ.
>>
>> Signed-off-by: Xiao Yang <yangx.jy@cn.fujitsu.com>
>> ---
>>
>> Current rxe module and other rdma modules(e.g. mlx5) only set
>> imm_data in WC when the related WR with imm_data finished on RQ.
>> I am not sure if it is a expected behavior.
> 
> This is IBTA behavior.
> 
> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
> "Immediate Data (ImmDt) contains data that is placed in the receive
>  Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>  RDMA WRITE packets with Immediate Data."
> 
> If I understand the spec, you shouldn't set imm_data in SQ.
> 
> Thanks
> 

This seems a little confused to me. wc.imm_data is set in rxe_resp.c in response to an incoming request packet that contains an IMMDT extension header. I.e. a write with immediate or send with immediate opcode from the remote end of the wire. This wc is delivered to the receive completion queue when the message is complete. It should not have anything to do with the local send work queue entries.

Bob Pearson

Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Xiao Yang]

On 2021/1/27 20:04, Leon Romanovsky wrote:
> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
>> module cannot set imm_data in WC when the related WR with imm_data
>> finished on SQ.
>>
>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
>> ---
>>
>> Current rxe module and other rdma modules(e.g. mlx5) only set
>> imm_data in WC when the related WR with imm_data finished on RQ.
>> I am not sure if it is a expected behavior.
> This is IBTA behavior.
>
> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
> "Immediate Data (ImmDt) contains data that is placed in the receive
>   Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>   RDMA WRITE packets with Immediate Data."
>
> If I understand the spec, you shouldn't set imm_data in SQ.
Hi Leon,

Got it, thanks a lot for your explanation.

Best Regards,
Xiao Yang
> Thanks
>
>
> .
>

Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Xiao Yang]

On 2021/1/27 20:04, Leon Romanovsky wrote:
> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
>> module cannot set imm_data in WC when the related WR with imm_data
>> finished on SQ.
>>
>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
>> ---
>>
>> Current rxe module and other rdma modules(e.g. mlx5) only set
>> imm_data in WC when the related WR with imm_data finished on RQ.
>> I am not sure if it is a expected behavior.
> This is IBTA behavior.
>
> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
> "Immediate Data (ImmDt) contains data that is placed in the receive
>   Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>   RDMA WRITE packets with Immediate Data."
>
> If I understand the spec, you shouldn't set imm_data in SQ.
Hi Leon,

About the behavior, I have another question:
For send operation with imm_data, we can verify if the delivered 
imm_data is correct by CQE on RQ.
For rdma write operation with imm_data, how to verify if the delivered 
imm_data is correct? :-)

Best Regards,
Xiao Yang
> Thanks
>
>
> .
>

Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Leon Romanovsky]

On Thu, Jan 28, 2021 at 02:29:43PM +0800, Xiao Yang wrote:
> On 2021/1/27 20:04, Leon Romanovsky wrote:
> > On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
> > > Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
> > > module cannot set imm_data in WC when the related WR with imm_data
> > > finished on SQ.
> > >
> > > Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
> > > ---
> > >
> > > Current rxe module and other rdma modules(e.g. mlx5) only set
> > > imm_data in WC when the related WR with imm_data finished on RQ.
> > > I am not sure if it is a expected behavior.
> > This is IBTA behavior.
> >
> > 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
> > "Immediate Data (ImmDt) contains data that is placed in the receive
> >   Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
> >   RDMA WRITE packets with Immediate Data."
> >
> > If I understand the spec, you shouldn't set imm_data in SQ.
> Hi Leon,
>
> About the behavior, I have another question:
> For send operation with imm_data, we can verify if the delivered imm_data is
> correct by CQE on RQ.
> For rdma write operation with imm_data, how to verify if the delivered
> imm_data is correct? :-)

Probably that I didn't understand the question, but the RDMA WRITE is
marked with special opcode in the BTH that indicates imm_data.

Thanks

>
> Best Regards,
> Xiao Yang
> > Thanks
> >
> >
> > .
> >
>
>
>

Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Xiao Yang]

On 2021/1/28 20:54, Leon Romanovsky wrote:
> On Thu, Jan 28, 2021 at 02:29:43PM +0800, Xiao Yang wrote:
>> On 2021/1/27 20:04, Leon Romanovsky wrote:
>>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
>>>> module cannot set imm_data in WC when the related WR with imm_data
>>>> finished on SQ.
>>>>
>>>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
>>>> ---
>>>>
>>>> Current rxe module and other rdma modules(e.g. mlx5) only set
>>>> imm_data in WC when the related WR with imm_data finished on RQ.
>>>> I am not sure if it is a expected behavior.
>>> This is IBTA behavior.
>>>
>>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
>>> "Immediate Data (ImmDt) contains data that is placed in the receive
>>>    Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>>>    RDMA WRITE packets with Immediate Data."
>>>
>>> If I understand the spec, you shouldn't set imm_data in SQ.
>> Hi Leon,
>>
>> About the behavior, I have another question:
>> For send operation with imm_data, we can verify if the delivered imm_data is
>> correct by CQE on RQ.
>> For rdma write operation with imm_data, how to verify if the delivered
>> imm_data is correct? :-)
> Probably that I didn't understand the question, but the RDMA WRITE is
> marked with special opcode in the BTH that indicates imm_data.
Hi Leon,

The quesion is about how to get the imm_data in applications(programs in 
user space)
1) If client program does send operation with imm_data, server program 
can get the delivered imm_data by calling ibv_poll_cq(&wc)
2) If client program does rdma write operation with imm_data, server 
program cannot get the delivered imm_data by calling ibv_poll_cq(&wc).
     In this case, how does server program get the delivered imm_data?

Best Regards,
Xiao Yang
> Thanks
>
>> Best Regards,
>> Xiao Yang
>>> Thanks
>>>
>>>
>>> .
>>>
>>
>>
>
> .
>

RE: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Pearson, Robert B]

-----Original Message-----
From: Xiao Yang <yangx.jy@cn.fujitsu.com> 
Sent: Thursday, January 28, 2021 8:15 PM
To: Leon Romanovsky <leon@kernel.org>
Cc: linux-rdma@vger.kernel.org; jgg@nvidia.com
Subject: Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

On 2021/1/28 20:54, Leon Romanovsky wrote:
> On Thu, Jan 28, 2021 at 02:29:43PM +0800, Xiao Yang wrote:
>> On 2021/1/27 20:04, Leon Romanovsky wrote:
>>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe 
>>>> module cannot set imm_data in WC when the related WR with imm_data 
>>>> finished on SQ.
>>>>
>>>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
>>>> ---
>>>>
>>>> Current rxe module and other rdma modules(e.g. mlx5) only set 
>>>> imm_data in WC when the related WR with imm_data finished on RQ.
>>>> I am not sure if it is a expected behavior.
>>> This is IBTA behavior.
>>>
>>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES 
>>> "Immediate Data (ImmDt) contains data that is placed in the receive
>>>    Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>>>    RDMA WRITE packets with Immediate Data."
>>>
>>> If I understand the spec, you shouldn't set imm_data in SQ.
>> Hi Leon,
>>
>> About the behavior, I have another question:
>> For send operation with imm_data, we can verify if the delivered 
>> imm_data is correct by CQE on RQ.
>> For rdma write operation with imm_data, how to verify if the 
>> delivered imm_data is correct? :-)
> Probably that I didn't understand the question, but the RDMA WRITE is 
> marked with special opcode in the BTH that indicates imm_data.
Hi Leon,

The quesion is about how to get the imm_data in applications(programs in user space)
1) If client program does send operation with imm_data, server program can get the delivered imm_data by calling ibv_poll_cq(&wc)
2) If client program does rdma write operation with imm_data, server program cannot get the delivered imm_data by calling ibv_poll_cq(&wc).
     In this case, how does server program get the delivered imm_data?

Best Regards,
Xiao Yang
> Thanks
>
>> Best Regards,
>> Xiao Yang
>>> Thanks
>>>
>>>
>>> .
>>>
>>
>>
>
> .
>

Write with immediate deposits the payload into an RDMA memory region and also consumes a receive work request and generates a receive completion event. The immediate data part sort of looks like a send but without any data in the receive buffer. The WR ID is copied into the completion event along with the immediate data so software can correlate the immediate with the work request.

Bob Pearson
rpearson@hpe.com

Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Xiao Yang]

On 2021/1/28 3:40, Bob Pearson wrote:
> On 1/27/21 6:04 AM, Leon Romanovsky wrote:
>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
>>> module cannot set imm_data in WC when the related WR with imm_data
>>> finished on SQ.
>>>
>>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
>>> ---
>>>
>>> Current rxe module and other rdma modules(e.g. mlx5) only set
>>> imm_data in WC when the related WR with imm_data finished on RQ.
>>> I am not sure if it is a expected behavior.
>> This is IBTA behavior.
>>
>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
>> "Immediate Data (ImmDt) contains data that is placed in the receive
>>   Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>>   RDMA WRITE packets with Immediate Data."
>>
>> If I understand the spec, you shouldn't set imm_data in SQ.
>>
>> Thanks
>>
> This seems a little confused to me. wc.imm_data is set in rxe_resp.c in response to an incoming request packet that contains an IMMDT extension header. I.e. a write with immediate or send with immediate opcode from the remote end of the wire. This wc is delivered to the receive completion queue when the message is complete. It should not have anything to do with the local send work queue entries.
Hi Bob,

Current rdma modules(e.g softroce, mlx5) only set wc_flags to 
IBV_WC_WITH_IMM for the completed send work queue entries.
I am not sure if it is also the IBTA behavior.

Best Regards,
Xiao Yang
> Bob Pearson
>
>
> .
>

RE: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Pearson, Robert B]

-----Original Message-----
From: Xiao Yang <yangx.jy@cn.fujitsu.com> 
Sent: Thursday, January 28, 2021 11:42 PM
To: Bob Pearson <rpearsonhpe@gmail.com>
Cc: Leon Romanovsky <leon@kernel.org>; linux-rdma@vger.kernel.org; jgg@nvidia.com
Subject: Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

On 2021/1/28 3:40, Bob Pearson wrote:
> On 1/27/21 6:04 AM, Leon Romanovsky wrote:
>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe 
>>> module cannot set imm_data in WC when the related WR with imm_data 
>>> finished on SQ.
>>>
>>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
>>> ---
>>>
>>> Current rxe module and other rdma modules(e.g. mlx5) only set 
>>> imm_data in WC when the related WR with imm_data finished on RQ.
>>> I am not sure if it is a expected behavior.
>> This is IBTA behavior.
>>
>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES 
>> "Immediate Data (ImmDt) contains data that is placed in the receive
>>   Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>>   RDMA WRITE packets with Immediate Data."
>>
>> If I understand the spec, you shouldn't set imm_data in SQ.
>>
>> Thanks
>>
> This seems a little confused to me. wc.imm_data is set in rxe_resp.c in response to an incoming request packet that contains an IMMDT extension header. I.e. a write with immediate or send with immediate opcode from the remote end of the wire. This wc is delivered to the receive completion queue when the message is complete. It should not have anything to do with the local send work queue entries.
Hi Bob,

Current rdma modules(e.g softroce, mlx5) only set wc_flags to IBV_WC_WITH_IMM for the completed send work queue entries.
I am not sure if it is also the IBTA behavior.

Best Regards,
Xiao Yang
> Bob Pearson
>
>
> .
>


Ah. That might be an issue. I'll check it in the morning and fix it if its wrong. Not sure what the purpose is for the send completion event since the verbs consumer knows what was put in the send work queue entry. On the other hand the receive completion event consumer needs to know whether the imm_data field is valid.

Bob

Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Zhu Yanjun]

On Fri, Jan 29, 2021 at 10:16 AM Xiao Yang <yangx.jy@cn.fujitsu.com> wrote:
>
> On 2021/1/28 20:54, Leon Romanovsky wrote:
> > On Thu, Jan 28, 2021 at 02:29:43PM +0800, Xiao Yang wrote:
> >> On 2021/1/27 20:04, Leon Romanovsky wrote:
> >>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
> >>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
> >>>> module cannot set imm_data in WC when the related WR with imm_data
> >>>> finished on SQ.
> >>>>
> >>>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
> >>>> ---
> >>>>
> >>>> Current rxe module and other rdma modules(e.g. mlx5) only set
> >>>> imm_data in WC when the related WR with imm_data finished on RQ.
> >>>> I am not sure if it is a expected behavior.
> >>> This is IBTA behavior.
> >>>
> >>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
> >>> "Immediate Data (ImmDt) contains data that is placed in the receive
> >>>    Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
> >>>    RDMA WRITE packets with Immediate Data."
> >>>
> >>> If I understand the spec, you shouldn't set imm_data in SQ.
> >> Hi Leon,
> >>
> >> About the behavior, I have another question:
> >> For send operation with imm_data, we can verify if the delivered imm_data is
> >> correct by CQE on RQ.
> >> For rdma write operation with imm_data, how to verify if the delivered
> >> imm_data is correct? :-)
> > Probably that I didn't understand the question, but the RDMA WRITE is
> > marked with special opcode in the BTH that indicates imm_data.
> Hi Leon,
>
> The quesion is about how to get the imm_data in applications(programs in
> user space)

Any steps or simple method to reproduce this problem?
I want to delve into this problem.

Thanks a lot.
Zhu Yanjun

> 1) If client program does send operation with imm_data, server program
> can get the delivered imm_data by calling ibv_poll_cq(&wc)
> 2) If client program does rdma write operation with imm_data, server
> program cannot get the delivered imm_data by calling ibv_poll_cq(&wc).
>      In this case, how does server program get the delivered imm_data?
>
> Best Regards,
> Xiao Yang
> > Thanks
> >
> >> Best Regards,
> >> Xiao Yang
> >>> Thanks
> >>>
> >>>
> >>> .
> >>>
> >>
> >>
> >
> > .
> >
>
>
>

RE: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Pearson, Robert B]

-----Original Message-----
From: Xiao Yang <yangx.jy@cn.fujitsu.com> 
Sent: Thursday, January 28, 2021 11:42 PM
To: Bob Pearson <rpearsonhpe@gmail.com>
Cc: Leon Romanovsky <leon@kernel.org>; linux-rdma@vger.kernel.org; jgg@nvidia.com
Subject: Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

On 2021/1/28 3:40, Bob Pearson wrote:
> On 1/27/21 6:04 AM, Leon Romanovsky wrote:
>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe 
>>> module cannot set imm_data in WC when the related WR with imm_data 
>>> finished on SQ.
>>>
>>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
>>> ---
>>>
>>> Current rxe module and other rdma modules(e.g. mlx5) only set 
>>> imm_data in WC when the related WR with imm_data finished on RQ.
>>> I am not sure if it is a expected behavior.
>> This is IBTA behavior.
>>
>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES 
>> "Immediate Data (ImmDt) contains data that is placed in the receive
>>   Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>>   RDMA WRITE packets with Immediate Data."
>>
>> If I understand the spec, you shouldn't set imm_data in SQ.
>>
>> Thanks
>>
> This seems a little confused to me. wc.imm_data is set in rxe_resp.c in response to an incoming request packet that contains an IMMDT extension header. I.e. a write with immediate or send with immediate opcode from the remote end of the wire. This wc is delivered to the receive completion queue when the message is complete. It should not have anything to do with the local send work queue entries.
Hi Bob,

Current rdma modules(e.g softroce, mlx5) only set wc_flags to IBV_WC_WITH_IMM for the completed send work queue entries.
I am not sure if it is also the IBTA behavior.

Best Regards,
Xiao Yang
> Bob Pearson
>
>
> .
>

On inspection, the IBV_EC_WITH_IMM flag is set in rxe_comp.c for send completion queue entries

		if (wqe->wr.opcode == IB_WR_RDMA_WRITE_WITH_IMM ||
		    wqe->wr.opcode == IB_WR_SEND_WITH_IMM)
			wc->wc_flags = IB_WC_WITH_IMM;
(There are two copies of this because the user space and kernel space WC structs are different.)
As I said earlier this seems of little value to me because the wc consumer wrote the send wqe and knows
whether it is an immediate operation but doesn't hurt anything. In particular it does not imply that
immediate data is returned in the wc. 

It is also set in rxe_resp.c for receive completion entries
		if (pkt->mask & RXE_IMMDT_MASK) {
			uwc->wc_flags |= IB_WC_WITH_IMM;
			uwc->ex.imm_data = immdt_imm(pkt);
		}
(Again there are two copies.)

In addition the receive wc opcode is set for write with immediate

		wc->opcode = (pkt->mask & RXE_IMMDT_MASK &&
				pkt->mask & RXE_WRITE_MASK) ?
					IB_WC_RECV_RDMA_WITH_IMM : IB_WC_RECV;
(This is required by IBTA chapter 11.)

The receive wc processing is triggered in the normal case when the last packet of a message is received (i.e. write last with immediate, etc.) which will set the flags above or when an error occurs which may not return the immediate data.

Bob

Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Xiao Yang]

On 2021/1/29 14:28, Zhu Yanjun wrote:
> On Fri, Jan 29, 2021 at 10:16 AM Xiao Yang<yangx.jy@cn.fujitsu.com>  wrote:
>> On 2021/1/28 20:54, Leon Romanovsky wrote:
>>> On Thu, Jan 28, 2021 at 02:29:43PM +0800, Xiao Yang wrote:
>>>> On 2021/1/27 20:04, Leon Romanovsky wrote:
>>>>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
>>>>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
>>>>>> module cannot set imm_data in WC when the related WR with imm_data
>>>>>> finished on SQ.
>>>>>>
>>>>>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
>>>>>> ---
>>>>>>
>>>>>> Current rxe module and other rdma modules(e.g. mlx5) only set
>>>>>> imm_data in WC when the related WR with imm_data finished on RQ.
>>>>>> I am not sure if it is a expected behavior.
>>>>> This is IBTA behavior.
>>>>>
>>>>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
>>>>> "Immediate Data (ImmDt) contains data that is placed in the receive
>>>>>     Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
>>>>>     RDMA WRITE packets with Immediate Data."
>>>>>
>>>>> If I understand the spec, you shouldn't set imm_data in SQ.
>>>> Hi Leon,
>>>>
>>>> About the behavior, I have another question:
>>>> For send operation with imm_data, we can verify if the delivered imm_data is
>>>> correct by CQE on RQ.
>>>> For rdma write operation with imm_data, how to verify if the delivered
>>>> imm_data is correct? :-)
>>> Probably that I didn't understand the question, but the RDMA WRITE is
>>> marked with special opcode in the BTH that indicates imm_data.
>> Hi Leon,
>>
>> The quesion is about how to get the imm_data in applications(programs in
>> user space)
> Any steps or simple method to reproduce this problem?
> I want to delve into this problem.
Hi Yanjun,

Sorry for the wrong question. :-)
For rdma write operation with imm_data, I can get the delivered imm_data 
by CQE on RQ.

Best Regards,
Xiao Yang
> Thanks a lot.
> Zhu Yanjun
>
>> 1) If client program does send operation with imm_data, server program
>> can get the delivered imm_data by calling ibv_poll_cq(&wc)
>> 2) If client program does rdma write operation with imm_data, server
>> program cannot get the delivered imm_data by calling ibv_poll_cq(&wc).
>>       In this case, how does server program get the delivered imm_data?
>>
>> Best Regards,
>> Xiao Yang
>>> Thanks
>>>
>>>> Best Regards,
>>>> Xiao Yang
>>>>> Thanks
>>>>>
>>>>>
>>>>> .
>>>>>
>>>>
>>> .
>>>
>>
>>
>
> .
>

Re: [RFC PATCH] RDMA/rxe: Export imm_data to WC when the related WR with imm_data finished on SQ

[Zhu Yanjun]

On Wed, Feb 3, 2021 at 1:25 PM Xiao Yang <yangx.jy@cn.fujitsu.com> wrote:
>
> On 2021/1/29 14:28, Zhu Yanjun wrote:
> > On Fri, Jan 29, 2021 at 10:16 AM Xiao Yang<yangx.jy@cn.fujitsu.com>  wrote:
> >> On 2021/1/28 20:54, Leon Romanovsky wrote:
> >>> On Thu, Jan 28, 2021 at 02:29:43PM +0800, Xiao Yang wrote:
> >>>> On 2021/1/27 20:04, Leon Romanovsky wrote:
> >>>>> On Wed, Jan 27, 2021 at 04:24:31PM +0800, Xiao Yang wrote:
> >>>>>> Even if we enable sq_sig_all or IBV_SEND_SIGNALED, current rxe
> >>>>>> module cannot set imm_data in WC when the related WR with imm_data
> >>>>>> finished on SQ.
> >>>>>>
> >>>>>> Signed-off-by: Xiao Yang<yangx.jy@cn.fujitsu.com>
> >>>>>> ---
> >>>>>>
> >>>>>> Current rxe module and other rdma modules(e.g. mlx5) only set
> >>>>>> imm_data in WC when the related WR with imm_data finished on RQ.
> >>>>>> I am not sure if it is a expected behavior.
> >>>>> This is IBTA behavior.
> >>>>>
> >>>>> 5.2.11 IMMEDIATE DATA EXTENDED TRANSPORT HEADER (ImmDt) - 4 BYTES
> >>>>> "Immediate Data (ImmDt) contains data that is placed in the receive
> >>>>>     Completion Queue Element (CQE). The ImmDt is only allowed in SEND or
> >>>>>     RDMA WRITE packets with Immediate Data."
> >>>>>
> >>>>> If I understand the spec, you shouldn't set imm_data in SQ.
> >>>> Hi Leon,
> >>>>
> >>>> About the behavior, I have another question:
> >>>> For send operation with imm_data, we can verify if the delivered imm_data is
> >>>> correct by CQE on RQ.
> >>>> For rdma write operation with imm_data, how to verify if the delivered
> >>>> imm_data is correct? :-)
> >>> Probably that I didn't understand the question, but the RDMA WRITE is
> >>> marked with special opcode in the BTH that indicates imm_data.
> >> Hi Leon,
> >>
> >> The quesion is about how to get the imm_data in applications(programs in
> >> user space)
> > Any steps or simple method to reproduce this problem?
> > I want to delve into this problem.
> Hi Yanjun,
>
> Sorry for the wrong question. :-)
> For rdma write operation with imm_data, I can get the delivered imm_data
> by CQE on RQ.

Wonderful.

Zhu Yanjun
>
> Best Regards,
> Xiao Yang
> > Thanks a lot.
> > Zhu Yanjun
> >
> >> 1) If client program does send operation with imm_data, server program
> >> can get the delivered imm_data by calling ibv_poll_cq(&wc)
> >> 2) If client program does rdma write operation with imm_data, server
> >> program cannot get the delivered imm_data by calling ibv_poll_cq(&wc).
> >>       In this case, how does server program get the delivered imm_data?
> >>
> >> Best Regards,
> >> Xiao Yang
> >>> Thanks
> >>>
> >>>> Best Regards,
> >>>> Xiao Yang
> >>>>> Thanks
> >>>>>
> >>>>>
> >>>>> .
> >>>>>
> >>>>
> >>> .
> >>>
> >>
> >>
> >
> > .
> >
>
>
>