Re: 25G RDMA networking thoughts???

Re: 25G RDMA networking thoughts???

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1788 bytes --]

On Wed, 9 Nov 2016, LIU, Fei wrote:
> Hi Sage，
>    Yes, Totally understood.  25G RDMA network for Ceph cluster is built for
> interal test. Xio messenger and Async messenger(But it can only support
> infiniband,right) are going to be two options. We are carefully evaluate
> both of these two options. But the most important goal in the end is to see
> how bluestore works with rdma to bring down the total latency for workload
> like OLTP.  
> 
> Hi Haomai,
>    Would you mind let us know when the async messenger is going to support
> ethernet if not support yet?

The default async backend is PosixStack which is all TCP-based.  (And 
async is now the default messenger in kraken.)

sage

> 
>    Regards,
>    James
>       ------------------------------------------------------------------
> From:Sage Weil <sweil@redhat.com>
> Time:2016 Nov 8 (Tue) 13:19
> To:James <james.liu@alibaba-inc.com>
> Subject:Re: 25G RDMA networking thoughts???
> 
> [adding ceph-devel]
> 
> On Wed, 9 Nov 2016, LIU, Fei wrote:
> > Hi Sage,
> >    I was wondering do you have any thoughts of 25G RDMA networking
> > construction besides of xio-messenger/async?  Is there any guidance to bu
> ild
> > 25G RDMA netowrk to better control the whole Ceph cluster latency?
> 
> The only RDMA options right now are XioMessenger and AsyncMessenger's new 
> RDMA backend. Both are experimental, but we'd be very interested in 
> hearing about your experience.
> 
> I wouldn't assume that latency is network-related, though.  More often 
> than not we're finding it's the OSD backend or the OSD request internals 
> (e.g., request scheduling or peering) that's the culprit...
> 
> sage
> 
> 
> 
> 

Re: 25G RDMA networking thoughts???

[Haomai Wang]

On Wed, Nov 9, 2016 at 5:51 AM, LIU, Fei <james.liu@alibaba-inc.com> wrote:
> Hi Sage，
>    thanks，Sorry for confusing you.  What i am trying to say is async with
> RDMA over ethernet.  I understand that async messenger support TCP based
> ethernet well.
> Secondly, would be possible to have data traffice among osds categorized
> into different service level to better provide QoS for the whole Ceph
> cluster service if we can within the 25G RDMA faclitiis?
> Thirdly, would be possible to provide unified network for both storage and
> compute within QoS control? We don't expect the replication/recovery/refill
> to have bad impact to the latency of application.

Currently I only use rdma over ib to test. I don't have eth over ib
nic by hand. I don't know what need to do for eth rdma nic...

>
>
>    Regards,
>    James
>
> ------------------------------------------------------------------
> From:Sage Weil <sweil@redhat.com>
> Time:2016 Nov 8 (Tue) 13:45
> To:James <james.liu@alibaba-inc.com>
> Cc:Haomai Wang <haomai@xsky.com>; ceph-devel <ceph-devel@vger.kernel.org>
> Subject:Re: 25G RDMA networking thoughts???
>
> On Wed, 9 Nov 2016, LIU, Fei wrote:
>> Hi Sage，
>>    Yes, Totally understood.  25G RDMA network for Ceph cluster is built
>> for
>> interal test. Xio messenger and Async messenger(But it can only support
>> infiniband,right) are going to be two options. We are carefully evaluate
>> both of these two options. But the most important goal in the end is to
>> see
>> how bluestore works with rdma to bring down the total latency for workload
>> like OLTP.
>>
>> Hi Haomai,
>>    Would you mind let us know when the async messenger is going to support
>> ethernet if not support yet?
>
> The default async backend is PosixStack which is all TCP-based.  (And
> async is now the default messenger in kraken.)
>
> sage
>
>>
>>    Regards,
>>    James
>>       ------------------------------------------------------------------
>> From:Sage Weil <sweil@redhat.com>
>> Time:2016 Nov 8 (Tue) 13:19
>> To:James <james.liu@alibaba-inc.com>
>> Subject:Re: 25G RDMA networking thoughts???
>>
>> [adding ceph-devel]
>>
>> On Wed, 9 Nov 2016, LIU, Fei wrote:
>> > Hi Sage,
>> >    I was wondering do you have any thoughts of 25G RDMA networking
>> > construction besides of xio-messenger/async?  Is there any guidance to
>> > bu
>> ild
>> > 25G RDMA netowrk to better control the whole Ceph cluster latency?
>>
>> The only RDMA options right now are XioMessenger and AsyncMessenger's new
>> RDMA backend. Both are experimental, but we'd be very interested in
>> hearing about your experience.
>>
>> I wouldn't assume that latency is network-related, though.  More often
>> than not we're finding it's the OSD backend or the OSD request internals
>> (e.g., request scheduling or peering) that's the culprit...
>>
>> sage
>>
>>
>>
>>
>
>