AsyncMessenger maturity

AsyncMessenger maturity

[Avner Ben Hanoch]

Hi All,

I tried to evaluate AsyncMessenger (together with its sub classes) and I found its code great in my eyes. Still, when using it at runtime, Ceph will warn you that this module is dangerous.

*I have two questions:*
1. Considering pros and cons, are there any cons when using AsyncMessenger instead of SimpleMessenger (like stability, or cpu/memory consumption, .)

2. I am considering a variant of AsyncMessenger that might require that both sides will be same messenger, I wonder, is it possible to use AsyncMessenger for private network between OSDs only and to remain with SimpleMessenger for all other messengers (for example, by just changing the lines that creates the 6 Messengers in ceph_osd.cc::main) 

TIA,
  Avner

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: AsyncMessenger maturity

[Sage Weil]

On Wed, 15 Jun 2016, Avner Ben Hanoch wrote:
> Hi All,
> 
> I tried to evaluate AsyncMessenger (together with its sub classes) and I 
> found its code great in my eyes. Still, when using it at runtime, Ceph 
> will warn you that this module is dangerous.

In jewel we've removed the 'experimental' flag.  Upgrade!
 
> *I have two questions:*
> 1. Considering pros and cons, are there any cons when using 
> AsyncMessenger instead of SimpleMessenger (like stability, or cpu/memory 
> consumption, .)

Less CPU, fewer threads, nicer code.

> 2. I am considering a variant of AsyncMessenger that might require that 
> both sides will be same messenger, I wonder, is it possible to use 
> AsyncMessenger for private network between OSDs only and to remain with 
> SimpleMessenger for all other messengers (for example, by just changing 
> the lines that creates the 6 Messengers in ceph_osd.cc::main)

It's the same protocol as SimpleMessenger, so there is no need to run 
SimpleMessenger at all.  (You can't currently use two types in the same 
process, but I don't think there's any need to.)

sage

Re: AsyncMessenger maturity

[Haomai Wang]

On Thu, Jun 16, 2016 at 2:16 AM, Sage Weil <sage@newdream.net> wrote:
> On Wed, 15 Jun 2016, Avner Ben Hanoch wrote:
>> Hi All,
>>
>> I tried to evaluate AsyncMessenger (together with its sub classes) and I
>> found its code great in my eyes. Still, when using it at runtime, Ceph
>> will warn you that this module is dangerous.
>
> In jewel we've removed the 'experimental' flag.  Upgrade!
>
>> *I have two questions:*
>> 1. Considering pros and cons, are there any cons when using
>> AsyncMessenger instead of SimpleMessenger (like stability, or cpu/memory
>> consumption, .)
>
> Less CPU, fewer threads, nicer code.
>
>> 2. I am considering a variant of AsyncMessenger that might require that
>> both sides will be same messenger, I wonder, is it possible to use
>> AsyncMessenger for private network between OSDs only and to remain with
>> SimpleMessenger for all other messengers (for example, by just changing
>> the lines that creates the 6 Messengers in ceph_osd.cc::main)
>
> It's the same protocol as SimpleMessenger, so there is no need to run
> SimpleMessenger at all.  (You can't currently use two types in the same
> process, but I don't think there's any need to.)

Recently we tested on a ~70 osds cluster used as rgw case. We can
obverse ~20% performance improvement compared to simple while do
cosbench benchmark. I'm not sure the actual reason now, but I guess in
a resource-limited env, async behavior much better.

>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: AsyncMessenger maturity

[Avner Ben Hanoch]

Cool! Thanks!

Async code looks really apropriate with that findings.

Thank you very much for this fantastic module.  I liked it on first sight!

Avner


-----Original Message-----
From: Haomai Wang [mailto:haomai@xsky.com] 
Sent: Thursday, June 16, 2016 04:46
To: Sage Weil <sage@newdream.net>
Cc: Avner Ben Hanoch <avnerb@mellanox.com>; ceph-devel@vger.kernel.org
Subject: Re: AsyncMessenger maturity

On Thu, Jun 16, 2016 at 2:16 AM, Sage Weil <sage@newdream.net> wrote:
> On Wed, 15 Jun 2016, Avner Ben Hanoch wrote:
>> Hi All,
>>
>> I tried to evaluate AsyncMessenger (together with its sub classes) 
>> and I found its code great in my eyes. Still, when using it at 
>> runtime, Ceph will warn you that this module is dangerous.
>
> In jewel we've removed the 'experimental' flag.  Upgrade!
>
>> *I have two questions:*
>> 1. Considering pros and cons, are there any cons when using 
>> AsyncMessenger instead of SimpleMessenger (like stability, or 
>> cpu/memory consumption, .)
>
> Less CPU, fewer threads, nicer code.
>
>> 2. I am considering a variant of AsyncMessenger that might require 
>> that both sides will be same messenger, I wonder, is it possible to 
>> use AsyncMessenger for private network between OSDs only and to 
>> remain with SimpleMessenger for all other messengers (for example, by 
>> just changing the lines that creates the 6 Messengers in 
>> ceph_osd.cc::main)
>
> It's the same protocol as SimpleMessenger, so there is no need to run 
> SimpleMessenger at all.  (You can't currently use two types in the 
> same process, but I don't think there's any need to.)

Recently we tested on a ~70 osds cluster used as rgw case. We can obverse ~20% performance improvement compared to simple while do cosbench benchmark. I'm not sure the actual reason now, but I guess in a resource-limited env, async behavior much better.

>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html

Re: AsyncMessenger maturity

[Mark Nelson]

> Recently we tested on a ~70 osds cluster used as rgw case. We can
> obverse ~20% performance improvement compared to simple while do
> cosbench benchmark. I'm not sure the actual reason now, but I guess in
> a resource-limited env, async behavior much better.
>

I've noticed that async messenger especially does much better when the 
memory allocator is not specifically optimized.  When it is, sometimes 
simplemessenger did better in my old tests.  I need to retest now though 
to see if that is still the case.  It's good to hear you are seeing 
better performance with async messenger now!

Mark