Re: IPoIB performance

["Atchley, Scott" <atchleyes-1Heg1YXhbW8@public.gmane.org>]

On Sep 5, 2012, at 2:20 PM, Christoph Lameter wrote:

> On Wed, 5 Sep 2012, Atchley, Scott wrote:
> 
>> # ethtool -k ib0
>> Offload parameters for ib0:
>> rx-checksumming: off
>> tx-checksumming: off
>> scatter-gather: off
>> tcp segmentation offload: off
>> udp fragmentation offload: off
>> generic segmentation offload: on
>> generic-receive-offload: off
>> 
>> There is no checksum support which I would expect to lower performance.
>> Since checksums need to be calculated in the host, I would expect faster
>> processors to help performance some.
> 
> K that is a major problem. Both are on by default here. What NIC is this?

These are Mellanox QDR HCAs (board id is MT_0D90110009). The full output of ibv_devinfo is in my original post.

>>> A) increase the block size handled by the socket layer
>> 
>> Do you mean altering sysctl with something like:
> 
> Nope increase mtu. Connected mode supports up to 64k mtu size I believe.

Yes, I am using the max MTU (65520).

>> or something increasing the SO_SNFBUF and SO_RCVBUF sizes or something else?
> 
> That does nothing for performance. The problem is that the handling of the
> data by the kernel causes too much latency so that you cannot reach the
> full bw of the hardware.
> 
>> We actually want to test the socket stack and not bypass it.
> 
> AFAICT the network stack is useful up to 1Gbps and
> after that more and more band-aid comes into play.

Hmm, many 10G Ethernet NICs can reach line rate. I have not yet tested any 40G Ethernet NICs, but I hope that they will get close to line rate. If not, what is the point? ;-)

Scott--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html