* [RFC]New Message Implementation Based on Event
@ 2014-09-12 3:09 Haomai Wang
2014-09-15 15:51 ` Sage Weil
0 siblings, 1 reply; 7+ messages in thread
From: Haomai Wang @ 2014-09-12 3:09 UTC (permalink / raw)
To: Sage Weil, Samuel Just, Gregory Farnum; +Cc: ceph-devel
Hi all,
Recently, I did some basic work on new message implementation based on
event(https://github.com/yuyuyu101/ceph/tree/msg-event). The basic
idea is that we use a Processor thread for each Messenger to monitor
all sockets and dispatch fd to threadpool. The event mechanism can be
epoll, kqueue, poll or select. The thread in threadpool will
read/write with this socket and dispatch message later.
Now the branch has passed basic tests and before make it more stable
and pass more QA suites. I want to do some benchmark tests compared to
pipe implementation with large-scale cluster. I would like to use at
least 100 OSDs(SSD) and hundreds of clients to test it. And now the
benchmark for only one OSD, the client can get the same latency with
pipe implementation and the latency stdev will be smaller.
The background for this implementation is that pipe implementation
consumes too much overhead on context switch and thread resource. In
our env, several ceph-osd is running on compute node which also runs
KVM process.
Do you have any ideas about this, or some serious concerns compared to pipe.
--
Best Regards,
Wheat
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC]New Message Implementation Based on Event
2014-09-12 3:09 [RFC]New Message Implementation Based on Event Haomai Wang
@ 2014-09-15 15:51 ` Sage Weil
2014-09-16 2:33 ` Haomai Wang
0 siblings, 1 reply; 7+ messages in thread
From: Sage Weil @ 2014-09-15 15:51 UTC (permalink / raw)
To: Haomai Wang; +Cc: Samuel Just, Gregory Farnum, ceph-devel
Hi Haomai,
On Fri, 12 Sep 2014, Haomai Wang wrote:
> Hi all,
>
> Recently, I did some basic work on new message implementation based on
> event(https://github.com/yuyuyu101/ceph/tree/msg-event). The basic
> idea is that we use a Processor thread for each Messenger to monitor
> all sockets and dispatch fd to threadpool. The event mechanism can be
> epoll, kqueue, poll or select. The thread in threadpool will
> read/write with this socket and dispatch message later.
>
> Now the branch has passed basic tests and before make it more stable
> and pass more QA suites. I want to do some benchmark tests compared to
> pipe implementation with large-scale cluster. I would like to use at
> least 100 OSDs(SSD) and hundreds of clients to test it. And now the
> benchmark for only one OSD, the client can get the same latency with
> pipe implementation and the latency stdev will be smaller.
>
> The background for this implementation is that pipe implementation
> consumes too much overhead on context switch and thread resource. In
> our env, several ceph-osd is running on compute node which also runs
> KVM process.
>
> Do you have any ideas about this, or some serious concerns compared to pipe.
I haven't had time to look at this in much detail yet, but at a high
level, this looks awesome! It sounds like using an event lib for this is
a good approach, and from a quick skim it looks like you've already done
the hard work of breaking all of the logic in Pipe.cc into a state
machine.
How much testing have you done with this?
I hope to find more time this week to look in more detail, but wanted to
let you know I didn't miss this before that :)
Cheers-
sage
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC]New Message Implementation Based on Event
2014-09-15 15:51 ` Sage Weil
@ 2014-09-16 2:33 ` Haomai Wang
2014-10-07 14:59 ` Sage Weil
0 siblings, 1 reply; 7+ messages in thread
From: Haomai Wang @ 2014-09-16 2:33 UTC (permalink / raw)
To: Sage Weil; +Cc: Samuel Just, Gregory Farnum, ceph-devel
As for testing, now I mainly passed tests in src/tests such as
ceph_test_rados. Because of the lack of Messenger's unittest, I have
to deploy this branch into my dev cluster to test. I'm thinking in
make ms_inject* options available in this Messenger for failure
coverage.
On Mon, Sep 15, 2014 at 11:51 PM, Sage Weil <sweil@redhat.com> wrote:
> Hi Haomai,
>
> On Fri, 12 Sep 2014, Haomai Wang wrote:
>> Hi all,
>>
>> Recently, I did some basic work on new message implementation based on
>> event(https://github.com/yuyuyu101/ceph/tree/msg-event). The basic
>> idea is that we use a Processor thread for each Messenger to monitor
>> all sockets and dispatch fd to threadpool. The event mechanism can be
>> epoll, kqueue, poll or select. The thread in threadpool will
>> read/write with this socket and dispatch message later.
>>
>> Now the branch has passed basic tests and before make it more stable
>> and pass more QA suites. I want to do some benchmark tests compared to
>> pipe implementation with large-scale cluster. I would like to use at
>> least 100 OSDs(SSD) and hundreds of clients to test it. And now the
>> benchmark for only one OSD, the client can get the same latency with
>> pipe implementation and the latency stdev will be smaller.
>>
>> The background for this implementation is that pipe implementation
>> consumes too much overhead on context switch and thread resource. In
>> our env, several ceph-osd is running on compute node which also runs
>> KVM process.
>>
>> Do you have any ideas about this, or some serious concerns compared to pipe.
>
> I haven't had time to look at this in much detail yet, but at a high
> level, this looks awesome! It sounds like using an event lib for this is
> a good approach, and from a quick skim it looks like you've already done
> the hard work of breaking all of the logic in Pipe.cc into a state
> machine.
>
> How much testing have you done with this?
>
> I hope to find more time this week to look in more detail, but wanted to
> let you know I didn't miss this before that :)
>
> Cheers-
> sage
--
Best Regards,
Wheat
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC]New Message Implementation Based on Event
2014-09-16 2:33 ` Haomai Wang
@ 2014-10-07 14:59 ` Sage Weil
2014-10-07 17:50 ` Mark Nelson
2014-10-08 1:46 ` Haomai Wang
0 siblings, 2 replies; 7+ messages in thread
From: Sage Weil @ 2014-10-07 14:59 UTC (permalink / raw)
To: Haomai Wang; +Cc: Samuel Just, Gregory Farnum, ceph-devel
Hi Haomai,
A branch cleaning up the messenger interface a bit more just merged.
Everything is now using the Messenger::create() factory method, and the
type of messenger instantiated is controlled by the ms_type config option.
There's also a reorg of the SimpleMessenger files into msg/simple/.
Do you mind rebasing your series onto the latest master?
Since this is an optional backend I think we can take a similar route as
KeyValueStore and merge it early so that it is easier to test and
improve.
Will you be able to join the performance call tomorrow?
sage
On Tue, 16 Sep 2014, Haomai Wang wrote:
> As for testing, now I mainly passed tests in src/tests such as
> ceph_test_rados. Because of the lack of Messenger's unittest, I have
> to deploy this branch into my dev cluster to test. I'm thinking in
> make ms_inject* options available in this Messenger for failure
> coverage.
>
> On Mon, Sep 15, 2014 at 11:51 PM, Sage Weil <sweil@redhat.com> wrote:
> > Hi Haomai,
> >
> > On Fri, 12 Sep 2014, Haomai Wang wrote:
> >> Hi all,
> >>
> >> Recently, I did some basic work on new message implementation based on
> >> event(https://github.com/yuyuyu101/ceph/tree/msg-event). The basic
> >> idea is that we use a Processor thread for each Messenger to monitor
> >> all sockets and dispatch fd to threadpool. The event mechanism can be
> >> epoll, kqueue, poll or select. The thread in threadpool will
> >> read/write with this socket and dispatch message later.
> >>
> >> Now the branch has passed basic tests and before make it more stable
> >> and pass more QA suites. I want to do some benchmark tests compared to
> >> pipe implementation with large-scale cluster. I would like to use at
> >> least 100 OSDs(SSD) and hundreds of clients to test it. And now the
> >> benchmark for only one OSD, the client can get the same latency with
> >> pipe implementation and the latency stdev will be smaller.
> >>
> >> The background for this implementation is that pipe implementation
> >> consumes too much overhead on context switch and thread resource. In
> >> our env, several ceph-osd is running on compute node which also runs
> >> KVM process.
> >>
> >> Do you have any ideas about this, or some serious concerns compared to pipe.
> >
> > I haven't had time to look at this in much detail yet, but at a high
> > level, this looks awesome! It sounds like using an event lib for this is
> > a good approach, and from a quick skim it looks like you've already done
> > the hard work of breaking all of the logic in Pipe.cc into a state
> > machine.
> >
> > How much testing have you done with this?
> >
> > I hope to find more time this week to look in more detail, but wanted to
> > let you know I didn't miss this before that :)
> >
> > Cheers-
> > sage
>
>
>
> --
> Best Regards,
>
> Wheat
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC]New Message Implementation Based on Event
2014-10-07 14:59 ` Sage Weil
@ 2014-10-07 17:50 ` Mark Nelson
2014-10-07 18:52 ` M Ranga Swami Reddy
2014-10-08 1:46 ` Haomai Wang
1 sibling, 1 reply; 7+ messages in thread
From: Mark Nelson @ 2014-10-07 17:50 UTC (permalink / raw)
To: Sage Weil, Haomai Wang; +Cc: Samuel Just, Gregory Farnum, ceph-devel
Btw, just wanted to say that this is fantastic and exactly what I was
hoping we'd eventually do. :)
Mark
On 10/07/2014 09:59 AM, Sage Weil wrote:
> Hi Haomai,
>
> A branch cleaning up the messenger interface a bit more just merged.
> Everything is now using the Messenger::create() factory method, and the
> type of messenger instantiated is controlled by the ms_type config option.
> There's also a reorg of the SimpleMessenger files into msg/simple/.
>
> Do you mind rebasing your series onto the latest master?
>
> Since this is an optional backend I think we can take a similar route as
> KeyValueStore and merge it early so that it is easier to test and
> improve.
>
> Will you be able to join the performance call tomorrow?
> sage
>
>
> On Tue, 16 Sep 2014, Haomai Wang wrote:
>
>> As for testing, now I mainly passed tests in src/tests such as
>> ceph_test_rados. Because of the lack of Messenger's unittest, I have
>> to deploy this branch into my dev cluster to test. I'm thinking in
>> make ms_inject* options available in this Messenger for failure
>> coverage.
>>
>> On Mon, Sep 15, 2014 at 11:51 PM, Sage Weil <sweil@redhat.com> wrote:
>>> Hi Haomai,
>>>
>>> On Fri, 12 Sep 2014, Haomai Wang wrote:
>>>> Hi all,
>>>>
>>>> Recently, I did some basic work on new message implementation based on
>>>> event(https://github.com/yuyuyu101/ceph/tree/msg-event). The basic
>>>> idea is that we use a Processor thread for each Messenger to monitor
>>>> all sockets and dispatch fd to threadpool. The event mechanism can be
>>>> epoll, kqueue, poll or select. The thread in threadpool will
>>>> read/write with this socket and dispatch message later.
>>>>
>>>> Now the branch has passed basic tests and before make it more stable
>>>> and pass more QA suites. I want to do some benchmark tests compared to
>>>> pipe implementation with large-scale cluster. I would like to use at
>>>> least 100 OSDs(SSD) and hundreds of clients to test it. And now the
>>>> benchmark for only one OSD, the client can get the same latency with
>>>> pipe implementation and the latency stdev will be smaller.
>>>>
>>>> The background for this implementation is that pipe implementation
>>>> consumes too much overhead on context switch and thread resource. In
>>>> our env, several ceph-osd is running on compute node which also runs
>>>> KVM process.
>>>>
>>>> Do you have any ideas about this, or some serious concerns compared to pipe.
>>>
>>> I haven't had time to look at this in much detail yet, but at a high
>>> level, this looks awesome! It sounds like using an event lib for this is
>>> a good approach, and from a quick skim it looks like you've already done
>>> the hard work of breaking all of the logic in Pipe.cc into a state
>>> machine.
>>>
>>> How much testing have you done with this?
>>>
>>> I hope to find more time this week to look in more detail, but wanted to
>>> let you know I didn't miss this before that :)
>>>
>>> Cheers-
>>> sage
>>
>>
>>
>> --
>> Best Regards,
>>
>> Wheat
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC]New Message Implementation Based on Event
2014-10-07 17:50 ` Mark Nelson
@ 2014-10-07 18:52 ` M Ranga Swami Reddy
0 siblings, 0 replies; 7+ messages in thread
From: M Ranga Swami Reddy @ 2014-10-07 18:52 UTC (permalink / raw)
To: Mark Nelson
Cc: Sage Weil, Haomai Wang, Samuel Just, Gregory Farnum, ceph-devel
Hi,
Could you please clarify, if this event message can be used with
radosgw and this message can be populated to outside of ceph (ie like
openstack metering services)?
Thanks
Swami
On Tue, Oct 7, 2014 at 11:20 PM, Mark Nelson <mark.nelson@inktank.com> wrote:
> Btw, just wanted to say that this is fantastic and exactly what I was hoping
> we'd eventually do. :)
>
> Mark
>
>
> On 10/07/2014 09:59 AM, Sage Weil wrote:
>>
>> Hi Haomai,
>>
>> A branch cleaning up the messenger interface a bit more just merged.
>> Everything is now using the Messenger::create() factory method, and the
>> type of messenger instantiated is controlled by the ms_type config option.
>> There's also a reorg of the SimpleMessenger files into msg/simple/.
>>
>> Do you mind rebasing your series onto the latest master?
>>
>> Since this is an optional backend I think we can take a similar route as
>> KeyValueStore and merge it early so that it is easier to test and
>> improve.
>>
>> Will you be able to join the performance call tomorrow?
>> sage
>>
>>
>> On Tue, 16 Sep 2014, Haomai Wang wrote:
>>
>>> As for testing, now I mainly passed tests in src/tests such as
>>> ceph_test_rados. Because of the lack of Messenger's unittest, I have
>>> to deploy this branch into my dev cluster to test. I'm thinking in
>>> make ms_inject* options available in this Messenger for failure
>>> coverage.
>>>
>>> On Mon, Sep 15, 2014 at 11:51 PM, Sage Weil <sweil@redhat.com> wrote:
>>>>
>>>> Hi Haomai,
>>>>
>>>> On Fri, 12 Sep 2014, Haomai Wang wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Recently, I did some basic work on new message implementation based on
>>>>> event(https://github.com/yuyuyu101/ceph/tree/msg-event). The basic
>>>>> idea is that we use a Processor thread for each Messenger to monitor
>>>>> all sockets and dispatch fd to threadpool. The event mechanism can be
>>>>> epoll, kqueue, poll or select. The thread in threadpool will
>>>>> read/write with this socket and dispatch message later.
>>>>>
>>>>> Now the branch has passed basic tests and before make it more stable
>>>>> and pass more QA suites. I want to do some benchmark tests compared to
>>>>> pipe implementation with large-scale cluster. I would like to use at
>>>>> least 100 OSDs(SSD) and hundreds of clients to test it. And now the
>>>>> benchmark for only one OSD, the client can get the same latency with
>>>>> pipe implementation and the latency stdev will be smaller.
>>>>>
>>>>> The background for this implementation is that pipe implementation
>>>>> consumes too much overhead on context switch and thread resource. In
>>>>> our env, several ceph-osd is running on compute node which also runs
>>>>> KVM process.
>>>>>
>>>>> Do you have any ideas about this, or some serious concerns compared to
>>>>> pipe.
>>>>
>>>>
>>>> I haven't had time to look at this in much detail yet, but at a high
>>>> level, this looks awesome! It sounds like using an event lib for this
>>>> is
>>>> a good approach, and from a quick skim it looks like you've already done
>>>> the hard work of breaking all of the logic in Pipe.cc into a state
>>>> machine.
>>>>
>>>> How much testing have you done with this?
>>>>
>>>> I hope to find more time this week to look in more detail, but wanted to
>>>> let you know I didn't miss this before that :)
>>>>
>>>> Cheers-
>>>> sage
>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>>
>>> Wheat
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC]New Message Implementation Based on Event
2014-10-07 14:59 ` Sage Weil
2014-10-07 17:50 ` Mark Nelson
@ 2014-10-08 1:46 ` Haomai Wang
1 sibling, 0 replies; 7+ messages in thread
From: Haomai Wang @ 2014-10-08 1:46 UTC (permalink / raw)
To: Sage Weil; +Cc: Samuel Just, Gregory Farnum, ceph-devel
No problem, I just came from a long vacation. I will create a PR ASAP
On Tue, Oct 7, 2014 at 10:59 PM, Sage Weil <sweil@redhat.com> wrote:
> Hi Haomai,
>
> A branch cleaning up the messenger interface a bit more just merged.
> Everything is now using the Messenger::create() factory method, and the
> type of messenger instantiated is controlled by the ms_type config option.
> There's also a reorg of the SimpleMessenger files into msg/simple/.
>
> Do you mind rebasing your series onto the latest master?
>
> Since this is an optional backend I think we can take a similar route as
> KeyValueStore and merge it early so that it is easier to test and
> improve.
>
> Will you be able to join the performance call tomorrow?
> sage
>
>
> On Tue, 16 Sep 2014, Haomai Wang wrote:
>
>> As for testing, now I mainly passed tests in src/tests such as
>> ceph_test_rados. Because of the lack of Messenger's unittest, I have
>> to deploy this branch into my dev cluster to test. I'm thinking in
>> make ms_inject* options available in this Messenger for failure
>> coverage.
>>
>> On Mon, Sep 15, 2014 at 11:51 PM, Sage Weil <sweil@redhat.com> wrote:
>> > Hi Haomai,
>> >
>> > On Fri, 12 Sep 2014, Haomai Wang wrote:
>> >> Hi all,
>> >>
>> >> Recently, I did some basic work on new message implementation based on
>> >> event(https://github.com/yuyuyu101/ceph/tree/msg-event). The basic
>> >> idea is that we use a Processor thread for each Messenger to monitor
>> >> all sockets and dispatch fd to threadpool. The event mechanism can be
>> >> epoll, kqueue, poll or select. The thread in threadpool will
>> >> read/write with this socket and dispatch message later.
>> >>
>> >> Now the branch has passed basic tests and before make it more stable
>> >> and pass more QA suites. I want to do some benchmark tests compared to
>> >> pipe implementation with large-scale cluster. I would like to use at
>> >> least 100 OSDs(SSD) and hundreds of clients to test it. And now the
>> >> benchmark for only one OSD, the client can get the same latency with
>> >> pipe implementation and the latency stdev will be smaller.
>> >>
>> >> The background for this implementation is that pipe implementation
>> >> consumes too much overhead on context switch and thread resource. In
>> >> our env, several ceph-osd is running on compute node which also runs
>> >> KVM process.
>> >>
>> >> Do you have any ideas about this, or some serious concerns compared to pipe.
>> >
>> > I haven't had time to look at this in much detail yet, but at a high
>> > level, this looks awesome! It sounds like using an event lib for this is
>> > a good approach, and from a quick skim it looks like you've already done
>> > the hard work of breaking all of the logic in Pipe.cc into a state
>> > machine.
>> >
>> > How much testing have you done with this?
>> >
>> > I hope to find more time this week to look in more detail, but wanted to
>> > let you know I didn't miss this before that :)
>> >
>> > Cheers-
>> > sage
>>
>>
>>
>> --
>> Best Regards,
>>
>> Wheat
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
--
Best Regards,
Wheat
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2014-10-08 1:46 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-09-12 3:09 [RFC]New Message Implementation Based on Event Haomai Wang
2014-09-15 15:51 ` Sage Weil
2014-09-16 2:33 ` Haomai Wang
2014-10-07 14:59 ` Sage Weil
2014-10-07 17:50 ` Mark Nelson
2014-10-07 18:52 ` M Ranga Swami Reddy
2014-10-08 1:46 ` Haomai Wang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.