XioMessenger (RDMA) Performance results

XioMessenger (RDMA) Performance results

[Yaron Haviv]

Im happy to share test results we run in the lab with Matt's latest XioMessenger code which implements Ceph messaging over Accelio RDMA library 
Results look pretty encouraging, demonstrating a * 20x * performance boost

Below is a table comparing XioMessenger (RDMA) with SimpleMessanger (TCP) and various interconnects (56G InfiniBand and 40GbE/RoCE)
Note that we tested with CRC on/off, in RDMA there is no need for software CRC, its done by the hardware  
Tests below use a single communication thread, using more threads would produce higher performance (on 64KB IO the Link/PCIe is saturated with RDMA using a single thread)   

Matt has some more small IO optimizations in the pipe, and we hope to share perf results using librados soon, I assume they will be disk bound 

protocol	CRC	msg depth	IO size	Msg/sec	bandwidth (MB/s)	CPU % server	CPU% client
eth	crc	50	4K	          16,262 	64	100%	100%
eth	no_crc	50	4K	          15,637 	61	100%	100%
eth	crc	50	64K	            5,960 	373	93%	100%
eth	no_crc	50	64K	            7,678 	480	93%	100%
ipoib	no_crc	50	4K	          16,003 	63	100%	100%
ipoib	no_crc	50	64K	            7,375 	461	93%	100%
							
IB	no_crc	50	4K	        334,088 	1305	98%	98%
IB	no_crc	50	64K	          95,078 	5942	98%	98%
							
roce	no_crc	50	4K	        332,388 	1298	95%	100%
roce	no_crc	50	64K	          69,445 	4340	91%	87%
							
roce	crc	50	4K	        172,756 	675	97%	100%
roce	crc	50	64K	          19,657 	1229	100%	48%

Regards, Yaron

Re: XioMessenger (RDMA) Performance results

[Gregory Farnum]

On Tue, Mar 18, 2014 at 1:05 PM, Yaron Haviv <yaronh@mellanox.com> wrote:
> Im happy to share test results we run in the lab with Matt's latest XioMessenger code which implements Ceph messaging over Accelio RDMA library
> Results look pretty encouraging, demonstrating a * 20x * performance boost

Very nice! I'm excited. :)
What did you use to generate these numbers?

> Below is a table comparing XioMessenger (RDMA) with SimpleMessanger (TCP) and various interconnects (56G InfiniBand and 40GbE/RoCE)
> Note that we tested with CRC on/off, in RDMA there is no need for software CRC, its done by the hardware
> Tests below use a single communication thread, using more threads would produce higher performance (on 64KB IO the Link/PCIe is saturated with RDMA using a single thread)
>
> Matt has some more small IO optimizations in the pipe, and we hope to share perf results using librados soon, I assume they will be disk bound
>
> protocol        CRC     msg depth       IO size Msg/sec bandwidth (MB/s)        CPU % server    CPU% client
> eth     crc     50      4K                16,262        64      100%    100%
> eth     no_crc  50      4K                15,637        61      100%    100%

Are these numbers correct? no_crc is slower?
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

> eth     crc     50      64K                 5,960       373     93%     100%
> eth     no_crc  50      64K                 7,678       480     93%     100%
> ipoib   no_crc  50      4K                16,003        63      100%    100%
> ipoib   no_crc  50      64K                 7,375       461     93%     100%
>
> IB      no_crc  50      4K              334,088         1305    98%     98%
> IB      no_crc  50      64K               95,078        5942    98%     98%
>
> roce    no_crc  50      4K              332,388         1298    95%     100%
> roce    no_crc  50      64K               69,445        4340    91%     87%
>
> roce    crc     50      4K              172,756         675     97%     100%
> roce    crc     50      64K               19,657        1229    100%    48%
>
> Regards, Yaron
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: XioMessenger (RDMA) Performance results

[Mark Nelson]

On 03/18/2014 03:05 PM, Yaron Haviv wrote:
> Im happy to share test results we run in the lab with Matt's latest XioMessenger code which implements Ceph messaging over Accelio RDMA library
> Results look pretty encouraging, demonstrating a * 20x * performance boost
>
> Below is a table comparing XioMessenger (RDMA) with SimpleMessanger (TCP) and various interconnects (56G InfiniBand and 40GbE/RoCE)
> Note that we tested with CRC on/off, in RDMA there is no need for software CRC, its done by the hardware
> Tests below use a single communication thread, using more threads would produce higher performance (on 64KB IO the Link/PCIe is saturated with RDMA using a single thread)
>
> Matt has some more small IO optimizations in the pipe, and we hope to share perf results using librados soon, I assume they will be disk bound
>
> protocol	CRC	msg depth	IO size	Msg/sec	bandwidth (MB/s)	CPU % server	CPU% client
> eth	crc	50	4K	          16,262 	64	100%	100%
> eth	no_crc	50	4K	          15,637 	61	100%	100%
> eth	crc	50	64K	            5,960 	373	93%	100%
> eth	no_crc	50	64K	            7,678 	480	93%	100%
> ipoib	no_crc	50	4K	          16,003 	63	100%	100%
> ipoib	no_crc	50	64K	            7,375 	461	93%	100%
> 							
> IB	no_crc	50	4K	        334,088 	1305	98%	98%
> IB	no_crc	50	64K	          95,078 	5942	98%	98%
> 							
> roce	no_crc	50	4K	        332,388 	1298	95%	100%
> roce	no_crc	50	64K	          69,445 	4340	91%	87%
> 							
> roce	crc	50	4K	        172,756 	675	97%	100%
> roce	crc	50	64K	          19,657 	1229	100%	48%

Very nice!  were you just writing to memory on the backend?

Mark

>
> Regards, Yaron
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: XioMessenger (RDMA) Performance results

[Matt W. Benjamin]

Hi Greg,

----- "Gregory Farnum" <greg@inktank.com> wrote:

> On Tue, Mar 18, 2014 at 1:05 PM, Yaron Haviv <yaronh@mellanox.com>
> wrote:
> > Im happy to share test results we run in the lab with Matt's latest
> XioMessenger code which implements Ceph messaging over Accelio RDMA
> library
> > Results look pretty encouraging, demonstrating a * 20x * performance
> boost
> 
> Very nice! I'm excited. :)
> What did you use to generate these numbers?

The two workload sets are created using the simple_client/simple_server vs
xio_client/xio_server client/server pairs, on...one of the latest XioMessenger
branches.  Eyal, was it xio-rados-exp or xio-rados-exp-noreg?

I -think- it would be xio-rados-exp.  The new stuff from us (including a couple
of librados fixups by Marcus and myself) is on xio-rados-exp-noreg.

> 
> > Below is a table comparing XioMessenger (RDMA) with SimpleMessanger
> (TCP) and various interconnects (56G InfiniBand and 40GbE/RoCE)
> > Note that we tested with CRC on/off, in RDMA there is no need for
> software CRC, its done by the hardware
> > Tests below use a single communication thread, using more threads
> would produce higher performance (on 64KB IO the Link/PCIe is
> saturated with RDMA using a single thread)
> >
> > Matt has some more small IO optimizations in the pipe, and we hope
> to share perf results using librados soon, I assume they will be disk
> bound
> >
> > protocol        CRC     msg depth       IO size Msg/sec bandwidth
> (MB/s)        CPU % server    CPU% client
> > eth     crc     50      4K                16,262        64      100%
>    100%
> > eth     no_crc  50      4K                15,637        61      100%
>    100%
> 
> Are these numbers correct? no_crc is slower?
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com

These numbers were from Mellanox, so I defer to Eyal.  I have seen
some inconsistency with crc vs no_crc in informal runs, myself.

> 
> > eth     crc     50      64K                 5,960       373     93% 
>    100%
> > eth     no_crc  50      64K                 7,678       480     93% 
>    100%
> > ipoib   no_crc  50      4K                16,003        63      100%
>    100%
> > ipoib   no_crc  50      64K                 7,375       461     93% 
>    100%
> >
> > IB      no_crc  50      4K              334,088         1305    98% 
>    98%
> > IB      no_crc  50      64K               95,078        5942    98% 
>    98%
> >
> > roce    no_crc  50      4K              332,388         1298    95% 
>    100%
> > roce    no_crc  50      64K               69,445        4340    91% 
>    87%
> >
> > roce    crc     50      4K              172,756         675     97% 
>    100%
> > roce    crc     50      64K               19,657        1229    100%
>    48%
> >
> > Regards, Yaron
> >

Regards,

Matt

-- 
Matt Benjamin
CohortFS, LLC.
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://cohortfs.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Re: XioMessenger (RDMA) Performance results

[Matt W. Benjamin]

Hi Mark,

----- "Mark Nelson" <mark.nelson@inktank.com> wrote:

> 
> Very nice!  were you just writing to memory on the backend?

These are just passing 64K message payloads across the sender/receiver,
it's a message-passing workload comparison only.

> 
> Mark
> 
> >
> > Regards, Yaron
> >
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe
> ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >

-- 
Matt Benjamin
CohortFS, LLC.
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://cohortfs.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309