Why RXE_COMP_MASK on RDMA SEND without immediate?

Why RXE_COMP_MASK on RDMA SEND without immediate?

[Josh Beavers]

Hello,

I am still new to RoCEv2 and have created a test utility patterned
after the rc_pingpong.c libibverbs example, but using RDMA WRITE
instead of RDMA SEND.

I discovered that when one machine sends a read, the other machine
acknowledges but does not indicate any completion on the CQ.  This
means userspace never wakes up to do its "pong" code.


The cause of this is that IB_OPCODE_RC_SEND_LAST has RXE_COMP_MASK
associated (http://lxr.free-electrons.com/source/drivers/infiniband/sw/rxe/rxe_opcode.c#L149),
but IB_OPCODE_RC_RDMA_WRITE_LAST does not.  Can someone please help me
understand why SEND LAST would trigger the CQ (creating a CQE, if I
understand correctly) even in the non-immediate case?

Page 170 of the Infiniband spec (vol 1, release 1.3) suggests the CQE
is only created for such packets with immediate data (ImmDt header).

Also for reference, page 531 of the Infiniband spec talks about
signaled completions and the conditions which cause a CQE to be
generated.


Thanks,
-J
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why RXE_COMP_MASK on RDMA SEND without immediate?

[Majd Dibbiny]

Sent from my iPhone
> On Apr 20, 2017, at 1:53 PM, Josh Beavers <josh.beavers-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> Hello,
Hi Josh,
> 
> I am still new to RoCEv2 and have created a test utility patterned
> after the rc_pingpong.c libibverbs example, but using RDMA WRITE
> instead of RDMA SEND.
> 
> I discovered that when one machine sends a read, the other machine
> acknowledges but does not indicate any completion on the CQ.  This
> means userspace never wakes up to do its "pong" code.
> 
> 
> The cause of this is that IB_OPCODE_RC_SEND_LAST has RXE_COMP_MASK
> associated (http://lxr.free-electrons.com/source/drivers/infiniband/sw/rxe/rxe_opcode.c#L149),
> but IB_OPCODE_RC_RDMA_WRITE_LAST does not.  Can someone please help me
> understand why SEND LAST would trigger the CQ (creating a CQE, if I
> understand correctly) even in the non-immediate case?
> 
> Page 170 of the Infiniband spec (vol 1, release 1.3) suggests the CQE
> is only created for such packets with immediate data (ImmDt header).
This is basically the reason.
RDMA READ and WRITE, and ATOMIC operations do not consume any receive buffers in the receiver side, and thus no completion is generated.
> 
> Also for reference, page 531 of the Infiniband spec talks about
> signaled completions and the conditions which cause a CQE to be
> generated.
> 
> 
> Thanks,
> -J
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Why RXE_COMP_MASK on RDMA SEND without immediate?

[Josh Beavers]

Hi Majd,

 >> Page 170 of the Infiniband spec (vol 1, release 1.3) suggests the CQE
>> is only created for such packets with immediate data (ImmDt header).
> This is basically the reason.
> RDMA READ and WRITE, and ATOMIC operations do not consume any receive buffers in the receiver side, and thus no completion is generated.

I had not realized SEND always needed a receive WQE to operate, even
in the plain SEND without immediate case.  Page 100:

"For a SEND operation, the WQE specifies a block of data in the
consumer’s memory space for the hardware to send to the desti-
nation, letting a receive WQE already queued at the destination
specify where to place that data."

>From that, I think I now understand that SEND always requires a
receive WQE, and that is why the CQ gets activated.

Thanks,
-J
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html