RE: Send with immediate data completion

RE: Send with immediate data completion

[Hefty, Sean]

> From my interpretation of the spec, it looks like a bug and the immediate data
> should be available in the send wc, since it's not explicitly excluded.  I'll
> verify that this is the intent with the IBTA.

I'm still waiting on feedback from the IBTA, but they are looking into the matter.
 
> I haven't traced through the library and driver code to see if mlx4 driver can
> report it.  I'll see if I can do this within the next couple of days.

The following patch to libmlx4 added the immediate data to the send work completion for me.  A similar change looks like it would be needed in the kernel.

Report immediate data with send work completions

From: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 src/cq.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/src/cq.c b/src/cq.c
index 8226b6b..ad1d20c 100644
--- a/src/cq.c
+++ b/src/cq.c
@@ -46,6 +46,7 @@
 
 #include "mlx4.h"
 #include "doorbell.h"
+#include "wqe.h"
 
 enum {
 	MLX4_CQ_DOORBELL			= 0x20
@@ -194,6 +195,7 @@ static int mlx4_poll_one(struct mlx4_cq *cq,
 {
 	struct mlx4_wq *wq;
 	struct mlx4_cqe *cqe;
+	struct mlx4_wqe_ctrl_seg *ctrl;
 	struct mlx4_srq *srq;
 	uint32_t qpn;
 	uint32_t g_mlpath_rqpn;
@@ -240,6 +242,8 @@ static int mlx4_poll_one(struct mlx4_cq *cq,
 		wq = &(*cur_qp)->sq;
 		wqe_index = ntohs(cqe->wqe_index);
 		wq->tail += (uint16_t) (wqe_index - (uint16_t) wq->tail);
+		ctrl = (*cur_qp)->buf.buf + (*cur_qp)->sq.offset +
+			((wq->tail & (wq->wqe_cnt - 1)) << (*cur_qp)->sq.wqe_shift);
 		wc->wr_id = wq->wrid[wq->tail & (wq->wqe_cnt - 1)];
 		++wq->tail;
 	} else if ((*cur_qp)->ibv_qp.srq) {
@@ -265,11 +269,13 @@ static int mlx4_poll_one(struct mlx4_cq *cq,
 		switch (cqe->owner_sr_opcode & MLX4_CQE_OPCODE_MASK) {
 		case MLX4_OPCODE_RDMA_WRITE_IMM:
 			wc->wc_flags |= IBV_WC_WITH_IMM;
+			wc->imm_data = ctrl->imm;
 		case MLX4_OPCODE_RDMA_WRITE:
 			wc->opcode    = IBV_WC_RDMA_WRITE;
 			break;
 		case MLX4_OPCODE_SEND_IMM:
 			wc->wc_flags |= IBV_WC_WITH_IMM;
+			wc->imm_data = ctrl->imm;
 		case MLX4_OPCODE_SEND:
 			wc->opcode    = IBV_WC_SEND;
 			break;


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Send with immediate data completion

[Hefty, Sean]

> I'm still waiting on feedback from the IBTA, but they are looking into the
> matter.

The intent is for immediate data only to be provided on receive work completions.  The IBTA will clarify the spec on this.  I'll submit patches that remove setting the wc flag, which may help avoid this confusion some.

Thanks,
- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Send with immediate data completion

[Roland Dreier]

On Wed, Jan 11, 2012 at 2:22 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> The intent is for immediate data only to be provided on receive work completions.  The IBTA will clarify the spec on this.  I'll submit patches that remove setting the wc flag, which may help avoid this confusion some.

The unfortunate thing is that we never defined enum values like

IBV_WC_RDMA_WRITE_WITH_IMM

to go along with

IBV_WR_RDMA_WRITE_WITH_IMM

If there was a good reason for that, I've long since forgotten it.

But in any case we don't have a good way to return the opcode
in the completion queue entry for an RDMA write with immediate
request, except the way we do it now.

Perhaps the only way forward is with a documentation update?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Send with immediate data completion

[Jason Gunthorpe]

On Wed, Jan 11, 2012 at 03:35:58PM -0800, Roland Dreier wrote:
> On Wed, Jan 11, 2012 at 2:22 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> > The intent is for immediate data only to be provided on receive
> > work completions. ?The IBTA will clarify the spec on this. ?I'll
> > submit patches that remove setting the wc flag, which may help
> > avoid this confusion some.
> 
> The unfortunate thing is that we never defined enum values like
> 
> IBV_WC_RDMA_WRITE_WITH_IMM
> 
> to go along with
> 
> IBV_WR_RDMA_WRITE_WITH_IMM
> 
> If there was a good reason for that, I've long since forgotten it.

Do you think there is a need to have a WC discern if there was an
attached immediate data? There is no resource attached to the
immediate data that needs special handling.

That is about the only argument I can see for continuing to set
the IBV_WC_WITH_IMM flag on the WC.

All I think is really needed here is a firm note someplace that
imm_data is only valid if IBV_WC_RECV is set, while IBV_WC_WITH_IMM
is set based on the opcode.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Send with immediate data completion

[Atchley, Scott]

On Jan 11, 2012, at 5:22 PM, Hefty, Sean wrote:

>> I'm still waiting on feedback from the IBTA, but they are looking into the
>> matter.
> 
> The intent is for immediate data only to be provided on receive work completions.  The IBTA will clarify the spec on this.  I'll submit patches that remove setting the wc flag, which may help avoid this confusion some.

Sean,

Thanks for looking into this.

Scott--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Send with immediate data completion

[Hefty, Sean]

> Any thoughts?
> 
> Personally, I would like to have it in the send completion, but it might not
> be possible for all drivers to implement. If not, then the spec should be
> clarified.

>From my interpretation of the spec, it looks like a bug and the immediate data should be available in the send wc, since it's not explicitly excluded.  I'll verify that this is the intent with the IBTA.

I haven't traced through the library and driver code to see if mlx4 driver can report it.  I'll see if I can do this within the next couple of days.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Send with immediate data completion

[Atchley, Scott]

On Jan 3, 2012, at 12:35 PM, Atchley, Scott wrote:

> On Jan 3, 2012, at 11:55 AM, Hefty, Sean wrote:
> 
>>> I have a question about a completion for a send with immediate data. The IB
>>> spec (1.2.1) only mentions that the WC's immediate data be present at the
>>> receiver. It is silent on the value on the sender at completion. It does say
>>> that it is only valid if the WC's immediate data indicator is set.
>> 
>> Can you provide a section reference to the spec on the areas that you're looking at?  Looking quickly, section 11.4.2.1 reads like immediate data should be available in either case.
>> 
>> I've never checked imm data on the send wc.  I'm just trying to determine if there's an issue in the spec that should be addressed, or if this is simply a bug in the hca/driver.
> 
> There is the definition in the glossary:
> 
> Immediate Data
> 
> Data contained in a Work Queue Element that is sent along with the payload to the remote Channel Adapter and placed in a Receive Work Completion.
> 
> Section 3.7.4 Transport Layer:
> 
> The Immediate Data (IMMDT) field is optionally present in RDMA WRITE and SEND messages. It contains data that the consumer placed in the Send or RDMA Write request and the receiving QP will place that value in the current receive WQE. An RDMA Write with immediate data will consume a receive WQE even though the QP did not place any data into the receive buffer since the IMMDT is placed in a CQE that references the receive WQE and indicates that the WQE has completed.
> 
> Section 11.4.1.1 Post Send Request has:
> 
> Immediate Data Indicator. This is set if Immediate Data is to
> be included in the outgoing request. Valid only for Send or
> Write RDMA operations.
> 
> 4-byte Immediate Data. Valid only for Send or Write RDMA operations.
> 
> 11.4.2.1 Poll for Completion
> 
> Immediate data indicator. This is set if immediate data is present.
> 
> 4-byte immediate data.
> 
> 
> None specifically mention the sender's completion event.

Sean,

Any thoughts?

Personally, I would like to have it in the send completion, but it might not be possible for all drivers to implement. If not, then the spec should be clarified.

Scott--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Send with immediate data completion

[Atchley, Scott]

On Jan 3, 2012, at 11:55 AM, Hefty, Sean wrote:

>> I have a question about a completion for a send with immediate data. The IB
>> spec (1.2.1) only mentions that the WC's immediate data be present at the
>> receiver. It is silent on the value on the sender at completion. It does say
>> that it is only valid if the WC's immediate data indicator is set.
> 
> Can you provide a section reference to the spec on the areas that you're looking at?  Looking quickly, section 11.4.2.1 reads like immediate data should be available in either case.
> 
> I've never checked imm data on the send wc.  I'm just trying to determine if there's an issue in the spec that should be addressed, or if this is simply a bug in the hca/driver.

For the record, I am using:

hca_id: mlx4_0
        transport:                      InfiniBand (0)
        fw_ver:                         2.7.626
        node_guid:                      0002:c903:000b:64e8
        sys_image_guid:                 0002:c903:000b:64eb
        vendor_id:                      0x02c9
        vendor_part_id:                 26428
        hw_ver:                         0xB0
        board_id:                       MT_0D90110009

Scott--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Send with immediate data completion

[Atchley, Scott]

On Jan 3, 2012, at 11:55 AM, Hefty, Sean wrote:

>> I have a question about a completion for a send with immediate data. The IB
>> spec (1.2.1) only mentions that the WC's immediate data be present at the
>> receiver. It is silent on the value on the sender at completion. It does say
>> that it is only valid if the WC's immediate data indicator is set.
> 
> Can you provide a section reference to the spec on the areas that you're looking at?  Looking quickly, section 11.4.2.1 reads like immediate data should be available in either case.
> 
> I've never checked imm data on the send wc.  I'm just trying to determine if there's an issue in the spec that should be addressed, or if this is simply a bug in the hca/driver.

There is the definition in the glossary:

Immediate Data

Data contained in a Work Queue Element that is sent along with the payload to the remote Channel Adapter and placed in a Receive Work Completion.

Section 3.7.4 Transport Layer:

The Immediate Data (IMMDT) field is optionally present in RDMA WRITE and SEND messages. It contains data that the consumer placed in the Send or RDMA Write request and the receiving QP will place that value in the current receive WQE. An RDMA Write with immediate data will consume a receive WQE even though the QP did not place any data into the receive buffer since the IMMDT is placed in a CQE that references the receive WQE and indicates that the WQE has completed.

Section 11.4.1.1 Post Send Request has:

Immediate Data Indicator. This is set if Immediate Data is to
be included in the outgoing request. Valid only for Send or
Write RDMA operations.

4-byte Immediate Data. Valid only for Send or Write RDMA operations.

11.4.2.1 Poll for Completion

Immediate data indicator. This is set if immediate data is present.

4-byte immediate data.


None specifically mention the sender's completion event.

Scott--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Send with immediate data completion

[Hefty, Sean]

> I have a question about a completion for a send with immediate data. The IB
> spec (1.2.1) only mentions that the WC's immediate data be present at the
> receiver. It is silent on the value on the sender at completion. It does say
> that it is only valid if the WC's immediate data indicator is set.

Can you provide a section reference to the spec on the areas that you're looking at?  Looking quickly, section 11.4.2.1 reads like immediate data should be available in either case.

I've never checked imm data on the send wc.  I'm just trying to determine if there's an issue in the spec that should be addressed, or if this is simply a bug in the hca/driver.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Send with immediate data completion

[Atchley, Scott]

Hi all,

I have a question about a completion for a send with immediate data. The IB spec (1.2.1) only mentions that the WC's immediate data be present at the receiver. It is silent on the value on the sender at completion. It does say that it is only valid if the WC's immediate data indicator is set.

When I test using a 2.6.38 kernel with the kernel.org libibverbs git tree, I see a send completion's wc_flags set with IBV_WC_WITH_IMM yet the imm_data is not what I passed in. Since the spec is silent on setting imm_data on the sender, I assume that I should not rely on looking at the imm_data on a send completion.

Given that, should IBV_WC_WITH_IMM ever be set on the sender?

Thanks,

Scott

-------------
Scott Atchley
HPC Systems Engineer
Center for Computational Sciences
Oak Ridge National Laboratory
atchleyes-1Heg1YXhbW8@public.gmane.org



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html