Preliminary RDMA vs TCP numbers

Preliminary RDMA vs TCP numbers

[Somnath Roy]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

Hi,
Please find the preliminary performance numbers of TCP Vs RDMA (XIO) implementation (on top of SSDs) attached here.

Mark,
If we have time, I can present it in tomorrow's performance meeting.

Thanks & Regards
Somnath

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).


[-- Attachment #2: ceph_on_rdma.pptx --]
[-- Type: application/vnd.openxmlformats-officedocument.presentationml.presentation, Size: 893733 bytes --]

[-- Attachment #3: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Preliminary RDMA vs TCP numbers

[Andrei Mikhailovsky]

[-- Attachment #1.1: Type: text/plain, Size: 5521 bytes --]

Mike, yeah, I wouldn't switch to rdma until it is fully supported in a stable release ))) 

Andrei 

----- Original Message -----

> From: "Andrei Mikhailovsky" <andrei-930XJYlnu5nQT0dZR+AlfA@public.gmane.org>
> To: "Somnath Roy" <Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, "ceph-devel"
> <ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, 8 April, 2015 7:16:40 PM
> Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers

> Somnath,

> Sounds very promising! I can't wait to try it on my cluster as I am
> currently using IPOIB instread of the native rdma.

> Cheers

> Andrei

> ----- Original Message -----

> > From: "Somnath Roy" <Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> 
> > To: "Andrei Mikhailovsky" <andrei-930XJYlnu5nQT0dZR+AlfA@public.gmane.org>, "Andrey Korolyov"
> > <andrey-5vqebrSIFTo@public.gmane.org>
> 
> > Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, "ceph-devel"
> > <ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> 
> > Sent: Wednesday, 8 April, 2015 5:23:23 PM
> 
> > Subject: RE: [ceph-users] Preliminary RDMA vs TCP numbers
> 

> > Andrei,
> 
> > Yes, I see it has lot of potential and I believe fixing the
> > performance bottlenecks inside XIO messenger it should go further.
> 
> > We are working on it and will keep community posted..
> 

> > Thanks & Regards
> 
> > Somnath
> 

> > From: Andrei Mikhailovsky [mailto:andrei-930XJYlnu5nQT0dZR+AlfA@public.gmane.org]
> 
> > Sent: Wednesday, April 08, 2015 2:22 AM
> 
> > To: Andrey Korolyov
> 
> > Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org; ceph-devel; Somnath Roy
> 
> > Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers
> 

> > Hi,
> 

> > Am I the only person noticing disappointing results from the
> > preliminary RDMA testing, or am I reading the numbers wrong?
> 

> > Yes, it's true that on a very small cluster you do see a great
> > improvement in rdma, but in real life rdma is used in large
> > infrastructure projects, not on a few servers with a handful of
> > osds. In fact, from what i've seen from the slides, the rdma
> > implementation scales horribly to the point that it becomes slower
> > the more osds you through at it.
> 

> > From my limited knowledge, i have expected a much higher
> > performance
> > gains with rdma, taking into account that you should have much
> > lower
> > latency and overhead and lower cpu utilisation when using this
> > transport in comparison with tcp.
> 

> > Are we likely to see a great deal of improvement with ceph and rdma
> > in a near future? Is there a roadmap for having a stable and
> > reliable rdma protocol support?
> 

> > Thanks
> 

> > Andrei
> 
> > ----- Original Message -----
> 

> > > From: "Andrey Korolyov" < andrey-5vqebrSIFTo@public.gmane.org >
> > 
> 
> > > To: "Somnath Roy" < Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org >
> > 
> 
> > > Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org , "ceph-devel" <
> > > ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org >
> > 
> 
> > > Sent: Wednesday, 8 April, 2015 9:28:12 AM
> > 
> 
> > > Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers
> > 
> 

> > > On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy <
> > > Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org > wrote:
> > 
> 
> > > >
> > 
> 
> > > > Hi,
> > 
> 
> > > > Please find the preliminary performance numbers of TCP Vs RDMA
> > > > (XIO) implementation (on top of SSDs) in the following link.
> > 
> 
> > > >
> > 
> 
> > > > http://www.slideshare.net/somnathroy7568/ceph-on-rdma
> > 
> 
> > > >
> > 
> 
> > > > The attachment didn't go through it seems, so, I had to use
> > > > slideshare.
> > 
> 
> > > >
> > 
> 
> > > > Mark,
> > 
> 
> > > > If we have time, I can present it in tomorrow's performance
> > > > meeting.
> > 
> 
> > > >
> > 
> 
> > > > Thanks & Regards
> > 
> 
> > > > Somnath
> > 
> 
> > > >
> > 
> 

> > > Those numbers are really impressive (for small numbers at least)!
> > > What
> > 
> 
> > > are TCP settings you using?For example, difference can be lowered
> > > on
> > 
> 
> > > scale due to less intensive per-connection acceleration on CUBIC
> > > on
> > > a
> > 
> 
> > > larger number of nodes, though I do not believe that it was a
> > > main
> > 
> 
> > > reason for an observed TCP catchup on a relatively flat workload
> > > such
> > 
> 
> > > as fio generates.
> > 
> 
> > > _______________________________________________
> > 
> 
> > > ceph-users mailing list
> > 
> 
> > > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > 
> 
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > 
> 
> > PLEASE NOTE: The information contained in this electronic mail
> > message is intended only for the use of the designated recipient(s)
> > named above. If the reader of this message is not the intended
> > recipient, you are hereby notified that you have received this
> > message in error and that any review, dissemination, distribution,
> > or copying of this message is strictly prohibited. If you have
> > received this communication in error, please notify the sender by
> > telephone or e-mail (as shown above) immediately and destroy any
> > and
> > all copies of this message in your possession (whether hard copies
> > or electronically stored copies).
> 

> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[-- Attachment #1.2: Type: text/html, Size: 10759 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Preliminary RDMA vs TCP numbers

[Andrei Mikhailovsky]

[-- Attachment #1.1: Type: text/plain, Size: 4393 bytes --]

Somnath, 

Sounds very promising! I can't wait to try it on my cluster as I am currently using IPOIB instread of the native rdma. 

Cheers 

Andrei 

----- Original Message -----

> From: "Somnath Roy" <Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: "Andrei Mikhailovsky" <andrei-930XJYlnu5nQT0dZR+AlfA@public.gmane.org>, "Andrey Korolyov"
> <andrey-5vqebrSIFTo@public.gmane.org>
> Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, "ceph-devel"
> <ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, 8 April, 2015 5:23:23 PM
> Subject: RE: [ceph-users] Preliminary RDMA vs TCP numbers

> Andrei,
> Yes, I see it has lot of potential and I believe fixing the
> performance bottlenecks inside XIO messenger it should go further.
> We are working on it and will keep community posted..

> Thanks & Regards
> Somnath

> From: Andrei Mikhailovsky [mailto:andrei-930XJYlnu5nQT0dZR+AlfA@public.gmane.org]
> Sent: Wednesday, April 08, 2015 2:22 AM
> To: Andrey Korolyov
> Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org; ceph-devel; Somnath Roy
> Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers

> Hi,

> Am I the only person noticing disappointing results from the
> preliminary RDMA testing, or am I reading the numbers wrong?

> Yes, it's true that on a very small cluster you do see a great
> improvement in rdma, but in real life rdma is used in large
> infrastructure projects, not on a few servers with a handful of
> osds. In fact, from what i've seen from the slides, the rdma
> implementation scales horribly to the point that it becomes slower
> the more osds you through at it.

> From my limited knowledge, i have expected a much higher performance
> gains with rdma, taking into account that you should have much lower
> latency and overhead and lower cpu utilisation when using this
> transport in comparison with tcp.

> Are we likely to see a great deal of improvement with ceph and rdma
> in a near future? Is there a roadmap for having a stable and
> reliable rdma protocol support?

> Thanks

> Andrei
> ----- Original Message -----

> > From: "Andrey Korolyov" < andrey-5vqebrSIFTo@public.gmane.org >
> 
> > To: "Somnath Roy" < Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org >
> 
> > Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org , "ceph-devel" <
> > ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org >
> 
> > Sent: Wednesday, 8 April, 2015 9:28:12 AM
> 
> > Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers
> 

> > On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy <
> > Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org > wrote:
> 
> > >
> 
> > > Hi,
> 
> > > Please find the preliminary performance numbers of TCP Vs RDMA
> > > (XIO) implementation (on top of SSDs) in the following link.
> 
> > >
> 
> > > http://www.slideshare.net/somnathroy7568/ceph-on-rdma
> 
> > >
> 
> > > The attachment didn't go through it seems, so, I had to use
> > > slideshare.
> 
> > >
> 
> > > Mark,
> 
> > > If we have time, I can present it in tomorrow's performance
> > > meeting.
> 
> > >
> 
> > > Thanks & Regards
> 
> > > Somnath
> 
> > >
> 

> > Those numbers are really impressive (for small numbers at least)!
> > What
> 
> > are TCP settings you using?For example, difference can be lowered
> > on
> 
> > scale due to less intensive per-connection acceleration on CUBIC on
> > a
> 
> > larger number of nodes, though I do not believe that it was a main
> 
> > reason for an observed TCP catchup on a relatively flat workload
> > such
> 
> > as fio generates.
> 
> > _______________________________________________
> 
> > ceph-users mailing list
> 
> > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> PLEASE NOTE: The information contained in this electronic mail
> message is intended only for the use of the designated recipient(s)
> named above. If the reader of this message is not the intended
> recipient, you are hereby notified that you have received this
> message in error and that any review, dissemination, distribution,
> or copying of this message is strictly prohibited. If you have
> received this communication in error, please notify the sender by
> telephone or e-mail (as shown above) immediately and destroy any and
> all copies of this message in your possession (whether hard copies
> or electronically stored copies).

[-- Attachment #1.2: Type: text/html, Size: 9660 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Preliminary RDMA vs TCP numbers

[Somnath Roy]

[-- Attachment #1.1: Type: text/plain, Size: 3519 bytes --]

Andrei,
Yes, I see it has lot of potential and I believe fixing the performance bottlenecks inside XIO messenger it should go further.
We are working on it and will keep community posted..

Thanks & Regards
Somnath

From: Andrei Mikhailovsky [mailto:andrei@arhont.com]
Sent: Wednesday, April 08, 2015 2:22 AM
To: Andrey Korolyov
Cc: ceph-users@lists.ceph.com; ceph-devel; Somnath Roy
Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers

Hi,

Am I the only person noticing disappointing results from the preliminary RDMA testing, or am I reading the numbers wrong?

Yes, it's true that on a very small cluster you do see a great improvement in rdma, but in real life rdma is used in large infrastructure projects, not on a few servers with a handful of osds. In fact, from what i've seen from the slides, the rdma implementation scales horribly to the point that it becomes slower the more osds you through at it.

From my limited knowledge, i have expected a much higher performance gains with rdma, taking into account that you should have much lower latency and overhead and lower cpu utilisation when using this transport in comparison with tcp.

Are we likely to see a great deal of improvement with ceph and rdma in a near future? Is there a roadmap for having a stable and reliable rdma protocol support?

Thanks

Andrei
________________________________
From: "Andrey Korolyov" <andrey@xdel.ru<mailto:andrey@xdel.ru>>
To: "Somnath Roy" <Somnath.Roy@sandisk.com<mailto:Somnath.Roy@sandisk.com>>
Cc: ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>, "ceph-devel" <ceph-devel@vger.kernel.org<mailto:ceph-devel@vger.kernel.org>>
Sent: Wednesday, 8 April, 2015 9:28:12 AM
Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers

On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy <Somnath.Roy@sandisk.com<mailto:Somnath.Roy@sandisk.com>> wrote:
>
> Hi,
> Please find the preliminary performance numbers of TCP Vs RDMA (XIO) implementation (on top of SSDs) in the following link.
>
> http://www.slideshare.net/somnathroy7568/ceph-on-rdma
>
> The attachment didn't go through it seems, so, I had to use slideshare.
>
> Mark,
> If we have time, I can present it in tomorrow's performance meeting.
>
> Thanks & Regards
> Somnath
>

Those numbers are really impressive (for small numbers at least)! What
are TCP settings you using?For example, difference can be lowered on
scale due to less intensive per-connection acceleration on CUBIC on a
larger number of nodes, though I do not believe that it was a main
reason for an observed TCP catchup on a relatively flat workload such
as fio generates.
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).


[-- Attachment #1.2: Type: text/html, Size: 8807 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Preliminary RDMA vs TCP numbers

[Somnath Roy]

[-- Attachment #1.1: Type: text/plain, Size: 1756 bytes --]

I am using Mellanox 40GbE, I think it is TCP offloaded.

From: Viral Mehta [mailto:viral.vkm@gmail.com]
Sent: Wednesday, April 08, 2015 1:30 AM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-devel
Subject: Re: Preliminary RDMA vs TCP numbers

I am sorry, I am new to the discussion.
But, is it TCP offloaded (TOE) or otherwise ?

On Wed, Apr 8, 2015 at 1:47 PM, Somnath Roy <Somnath.Roy@sandisk.com<mailto:Somnath.Roy@sandisk.com>> wrote:

Hi,
Please find the preliminary performance numbers of TCP Vs RDMA (XIO) implementation (on top of SSDs) in the following link.

http://www.slideshare.net/somnathroy7568/ceph-on-rdma

The attachment didn't go through it seems, so, I had to use slideshare.

Mark,
If we have time, I can present it in tomorrow's performance meeting.

Thanks & Regards
Somnath

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org<mailto:majordomo@vger.kernel.org>
More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
Thanks,
Viral Mehta

[-- Attachment #1.2: Type: text/html, Size: 4668 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

RE: Preliminary RDMA vs TCP numbers

[Somnath Roy]

I used the default TCP setting in Ubuntu 14.04.

-----Original Message-----
From: Andrey Korolyov [mailto:andrey@xdel.ru]
Sent: Wednesday, April 08, 2015 1:28 AM
To: Somnath Roy
Cc: ceph-users@lists.ceph.com; ceph-devel
Subject: Re: Preliminary RDMA vs TCP numbers

On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy <Somnath.Roy@sandisk.com> wrote:
>
> Hi,
> Please find the preliminary performance numbers of TCP Vs RDMA (XIO) implementation (on top of SSDs) in the following link.
>
> http://www.slideshare.net/somnathroy7568/ceph-on-rdma
>
> The attachment didn't go through it seems, so, I had to use slideshare.
>
> Mark,
> If we have time, I can present it in tomorrow's performance meeting.
>
> Thanks & Regards
> Somnath
>

Those numbers are really impressive (for small numbers at least)! What are TCP settings you using?For example, difference can be lowered on scale due to less intensive per-connection acceleration on CUBIC on a larger number of nodes, though I do not believe that it was a main reason for an observed TCP catchup on a relatively flat workload such as fio generates.

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: Preliminary RDMA vs TCP numbers

[Andrei Mikhailovsky]

[-- Attachment #1.1: Type: text/plain, Size: 2394 bytes --]

Hi, 

Am I the only person noticing disappointing results from the preliminary RDMA testing, or am I reading the numbers wrong? 

Yes, it's true that on a very small cluster you do see a great improvement in rdma, but in real life rdma is used in large infrastructure projects, not on a few servers with a handful of osds. In fact, from what i've seen from the slides, the rdma implementation scales horribly to the point that it becomes slower the more osds you through at it. 

From my limited knowledge, i have expected a much higher performance gains with rdma, taking into account that you should have much lower latency and overhead and lower cpu utilisation when using this transport in comparison with tcp. 

Are we likely to see a great deal of improvement with ceph and rdma in a near future? Is there a roadmap for having a stable and reliable rdma protocol support? 

Thanks 

Andrei 
----- Original Message -----

> From: "Andrey Korolyov" <andrey-5vqebrSIFTo@public.gmane.org>
> To: "Somnath Roy" <Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, "ceph-devel"
> <ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
> Sent: Wednesday, 8 April, 2015 9:28:12 AM
> Subject: Re: [ceph-users] Preliminary RDMA vs TCP numbers

> On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy
> <Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:
> >
> > Hi,
> > Please find the preliminary performance numbers of TCP Vs RDMA
> > (XIO) implementation (on top of SSDs) in the following link.
> >
> > http://www.slideshare.net/somnathroy7568/ceph-on-rdma
> >
> > The attachment didn't go through it seems, so, I had to use
> > slideshare.
> >
> > Mark,
> > If we have time, I can present it in tomorrow's performance
> > meeting.
> >
> > Thanks & Regards
> > Somnath
> >

> Those numbers are really impressive (for small numbers at least)!
> What
> are TCP settings you using?For example, difference can be lowered on
> scale due to less intensive per-connection acceleration on CUBIC on a
> larger number of nodes, though I do not believe that it was a main
> reason for an observed TCP catchup on a relatively flat workload such
> as fio generates.
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[-- Attachment #1.2: Type: text/html, Size: 2976 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Preliminary RDMA vs TCP numbers

[Viral Mehta]

[-- Attachment #1.1: Type: text/plain, Size: 1513 bytes --]

I am sorry, I am new to the discussion.

But, is it TCP offloaded (TOE) or otherwise ?

On Wed, Apr 8, 2015 at 1:47 PM, Somnath Roy <Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:

>
> Hi,
> Please find the preliminary performance numbers of TCP Vs RDMA (XIO)
> implementation (on top of SSDs) in the following link.
>
> http://www.slideshare.net/somnathroy7568/ceph-on-rdma
>
> The attachment didn't go through it seems, so, I had to use slideshare.
>
> Mark,
> If we have time, I can present it in tomorrow's performance meeting.
>
> Thanks & Regards
> Somnath
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is
> intended only for the use of the designated recipient(s) named above. If
> the reader of this message is not the intended recipient, you are hereby
> notified that you have received this message in error and that any review,
> dissemination, distribution, or copying of this message is strictly
> prohibited. If you have received this communication in error, please notify
> the sender by telephone or e-mail (as shown above) immediately and destroy
> any and all copies of this message in your possession (whether hard copies
> or electronically stored copies).
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Thanks,
Viral Mehta

[-- Attachment #1.2: Type: text/html, Size: 2224 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Preliminary RDMA vs TCP numbers

[Andrey Korolyov]

On Wed, Apr 8, 2015 at 11:17 AM, Somnath Roy <Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:
>
> Hi,
> Please find the preliminary performance numbers of TCP Vs RDMA (XIO) implementation (on top of SSDs) in the following link.
>
> http://www.slideshare.net/somnathroy7568/ceph-on-rdma
>
> The attachment didn't go through it seems, so, I had to use slideshare.
>
> Mark,
> If we have time, I can present it in tomorrow's performance meeting.
>
> Thanks & Regards
> Somnath
>

Those numbers are really impressive (for small numbers at least)! What
are TCP settings you using?For example, difference can be lowered on
scale due to less intensive per-connection acceleration on CUBIC on a
larger number of nodes, though I do not believe that it was a main
reason for an observed TCP catchup on a relatively flat workload such
as fio generates.

Preliminary RDMA vs TCP numbers

[Somnath Roy]

Hi,
Please find the preliminary performance numbers of TCP Vs RDMA (XIO) implementation (on top of SSDs) in the following link.

http://www.slideshare.net/somnathroy7568/ceph-on-rdma

The attachment didn't go through it seems, so, I had to use slideshare.

Mark,
If we have time, I can present it in tomorrow's performance meeting.

Thanks & Regards
Somnath

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).