Re: [PATCH 3/4] rdma ioengine improvement

[Yufei Ren <yufei.ren@stonybrook.edu>]

On Mon, Oct 22, 2012 at 4:06 AM, Jens Axboe <axboe@kernel.dk> wrote:
> On 2012-10-20 05:11, Yufei Ren wrote:
>> From: Yufei Ren <renyufei83@gmail.com>
>
> This one needs a bit of explaining. What is the point of using a
> re-entrant variant of rand(), if you are using a shared static variable
> anyway?
>

It's my buggy :-)

> Would it be better to use the fio shipped rand, now we're in there
> anyway?
>

Thanks for remind me the fio shipped rand. Let me send you another patch.

>> @@ -790,6 +792,13 @@ static int fio_rdmaio_connect(struct thread_data *td, struct fio_file *f)
>>       /* wait for remote MR info from server side */
>>       rdma_poll_wait(td, IBV_WC_RECV);
>>
>> +     /* In SEND/RECV test, iodepth in RECV side is deeper
>> +      * in SEND side. RECV needs more time to construct the
>> +      * buffer blocks, so the server side may need to stop
                                                   ~~~~~~~~~ should be SEND side
>> +      * some time before transfer data.
>> +      */
>> +     usleep(500000);
>> +
>>       return 0;
>
> Hmm?

After rdma connection(queue pair) established, data source and data
sink commit data blocks into send queue and receive queue,
respectively. If data source posts so many io requests that over the
capacity of the receiver queue. Receiver Not Ready (RNR) error comes
out. To avoid this, it's better that data source waits for some time
until data sink posts sufficient number of recv buffers into recv
queue.

Maybe it is a better way to add a `receiver ready notification' from
the sink to the source for addressing this problem.

Anyway, I will send you another patch with clear comments.

>
> --
> Jens Axboe
>