On Jul 09, 2010 07:13 PM, "Hefty, Sean" <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> > Is librdmacm version 1.0.12 stable?
> 
> Yes - but you are using APIs that are new to this release. But those
> calls are wrappers around existing libibverbs calls.
>  
> > The flag IBV_SEND_INLINE supports buffer size until 64 bytes (cards'
> > feature).
> > 
> > I know that RDMA is a protocol connected with 10 Gigabit/s Ethernet.
> > How can I see this trasfer speed? Can you suggest the "path"?
> > I attached the starting latency_RDMA.cpp code, which works with
> > librdmacm version 1.0.12 with BUF_SIZE < = 64 bytes (thanks to
> > Chien).
> > I would like to explore transfer speed in range 1 bytes < BUF_SIZE <
> > 4
> > Mbytes.
> 
> The use of INLINE allows you to post a send message without the buffer
> being registered. As soon as you try to post a send with a buffer
> larger than what can be supported as inline, the buffer must be
> registered, and the memory region should be passed into the send
> command. Add something like this to both the client and server:
> 
> struct ibv_mr *send_mr;
> 
> ...
> 
> send_mr = rdma_reg_msgs(id, send_msg, BUF_SIZE);
> // add matching rdma_dereg_mr(send_mr); at end
> 
> and change your rdma_post_send calls to:
> 
> ret = rdma_post_send(id, NULL, send_msg, BUF_SIZE, send_mr,
> IBV_SEND_INLINE);
> 
> - Sean

Hi Sean,

I see that this rdma_post_send call gives a big contribute to CPU use on
client side. Now the CPU usage (%) is about 95%-99%.
I consider the formula:
CPU = (user_time + system_time)/total_time_real

Test on latency and speed are OK, but I'm surprised by CPU use data.
How - if it's possible - can I reduce the CPU use with these calls?

Could you please send me a comment also about the "plateau non stable"
in graph speed versus buffer size with buffer size > 10^5 bytes?
(attached files)

Thank you very much for your time.
Regards,
Andrea
Andrea Gozzelino

INFN - Laboratori Nazionali di Legnaro	(LNL)
Viale dell'Universita' 2 -I-35020 - Legnaro (PD)- ITALIA
Office: E-101
Tel: +39 049 8068346
Fax: +39 049 641925
Mail: andrea.gozzelino-PK20h7lG/Rc1GQ1Ptb7lUw@public.gmane.org
Cell: +39 3488245552