IBV_SEND_INLINE Behavior Differing from Spec

IBV_SEND_INLINE Behavior Differing from Spec

[Christopher Mitchell]

Hey all,

The manpage for ibv_post_send indicates that if you specify the
IBV_SEND_INLINE flag, that you can immediately re-use the buffer after
ibv_post_send() returns. However, I have found that if I modify the
buffer immediately after ibv_post_send(), the changes are rarely
reflected in the presumably already-posted message. I'm using
ConnectX-3 cards and RC connections, if that's relevant. Is this a
known issue, and/or is it possible I'm missing something obvious?

Cheers,
Christopher
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IBV_SEND_INLINE Behavior Differing from Spec

[Roland Dreier]

On Fri, Jul 3, 2015 at 2:50 PM, Christopher Mitchell
<christopher-1z5WdJkP5Frk1uMJSBkQmQ@public.gmane.org> wrote:
> The manpage for ibv_post_send indicates that if you specify the
> IBV_SEND_INLINE flag, that you can immediately re-use the buffer after
> ibv_post_send() returns. However, I have found that if I modify the
> buffer immediately after ibv_post_send(), the changes are rarely
> reflected in the presumably already-posted message. I'm using
> ConnectX-3 cards and RC connections, if that's relevant. Is this a
> known issue, and/or is it possible I'm missing something obvious?

I can't see any way to get the behavior you're seeing.  What version
of libmlx4 are you using?

The reason I say I don't see how that can happen is that in
mlx4_post_send() in libmlx4, if IBV_SEND_INLINE is set, the code
memcpys the data into the descriptor and does not keep any reference
to the pointers passed in through the sg ilst.  So I don't see how the
hardware could even detect that you change the data after
ibv_post_send().

Or what do you mean about "after ibv_post_send()"?  Are you possibly
changing the data from another thread and racing with the memcpy
before ibv_post_send() returns?

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IBV_SEND_INLINE Behavior Differing from Spec

[Christopher Mitchell]

Roland,

My libmlx4 is named libmlx4-rdmav2.so and timestamped 2014-01-01 (from
the package libmlx4-1, 1.0.5-1ubuntu1 for Ubuntu Trusty 14.04). I use
a mutex-protected FIFO of send buffers to make sure only one thread is
able to acquire a particular send buffer at a time, and my application
does not modify a send buffer after it passes it to my message-sending
method. I've also replicated this with a single-threaded version of
the application; something as simple as zeroing out the first byte of
the send buffer on the line after ibv_post_send() is enough to
occasionally trigger this behavior.

Thanks,
Christopher

On Sun, Jul 5, 2015 at 12:28 PM, Roland Dreier <roland-BHEL68pLQRGGvPXPguhicg@public.gmane.org> wrote:
> On Fri, Jul 3, 2015 at 2:50 PM, Christopher Mitchell
> <christopher-1z5WdJkP5Frk1uMJSBkQmQ@public.gmane.org> wrote:
>> The manpage for ibv_post_send indicates that if you specify the
>> IBV_SEND_INLINE flag, that you can immediately re-use the buffer after
>> ibv_post_send() returns. However, I have found that if I modify the
>> buffer immediately after ibv_post_send(), the changes are rarely
>> reflected in the presumably already-posted message. I'm using
>> ConnectX-3 cards and RC connections, if that's relevant. Is this a
>> known issue, and/or is it possible I'm missing something obvious?
>
> I can't see any way to get the behavior you're seeing.  What version
> of libmlx4 are you using?
>
> The reason I say I don't see how that can happen is that in
> mlx4_post_send() in libmlx4, if IBV_SEND_INLINE is set, the code
> memcpys the data into the descriptor and does not keep any reference
> to the pointers passed in through the sg ilst.  So I don't see how the
> hardware could even detect that you change the data after
> ibv_post_send().
>
> Or what do you mean about "after ibv_post_send()"?  Are you possibly
> changing the data from another thread and racing with the memcpy
> before ibv_post_send() returns?
>
>  - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: IBV_SEND_INLINE Behavior Differing from Spec

[Shachar Raindel]

Hi,

> From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma-
> owner@vger.kernel.org] On Behalf Of Christopher Mitchell
> Sent: Tuesday, July 07, 2015 5:21 AM
> 
> Roland,
> 
> My libmlx4 is named libmlx4-rdmav2.so and timestamped 2014-01-01 (from
> the package libmlx4-1, 1.0.5-1ubuntu1 for Ubuntu Trusty 14.04).

This is good to align us on the source code. Reading there, I see the
following code handling inline data copy:
				while (len >= MLX4_INLINE_ALIGN - off) {
					to_copy = MLX4_INLINE_ALIGN - off;
					memcpy(wqe, addr, to_copy);
					len -= to_copy;
					wqe += to_copy;
					addr += to_copy;
					seg_len += to_copy;
					wmb(); /* see comment below */
					seg->byte_count = htonl(MLX4_INLINE_SEG | seg_len);
					seg_len = 0;
					seg = wqe;
					wqe += sizeof *seg;
					off = sizeof *seg;
					++num_seg;
				}

				memcpy(wqe, addr, len);
				wqe += len;
				seg_len += len;
				off += len;

Note the memcpy to the work queue element ("wqe"). This buffer is completely
independent of the original data buffer. Unless memcpy started behaving
differently recently, this does not seem like something that can produce
the symptoms you are describing.

> I use
> a mutex-protected FIFO of send buffers to make sure only one thread is
> able to acquire a particular send buffer at a time, and my application
> does not modify a send buffer after it passes it to my message-sending
> method.

While not directly related to issue at hand, you might want to consider
a design where each thread is using an independent work queue, to prevent
data dependencies and locking between threads.

> I've also replicated this with a single-threaded version of
> the application; something as simple as zeroing out the first byte of
> the send buffer on the line after ibv_post_send() is enough to
> occasionally trigger this behavior.
>

Can you provide a short reproducer code for this?

Could it be that you are calling "fork" (or a variant of it, i.e. system())
in your code after doing the memory registration and before calling
the post send verb? This can lead to similar behaviors, as the memory page
is CoWed and the registered memory pages of the work queue might differ from
what libmlx4 is writing into.

Thanks,
--Shachar

N.B. you might want to avoid top-posting in OSS mailing lists, as it makes it
hard to maintain context.