OOB message roll into Messenger interface

OOB message roll into Messenger interface

[Haomai Wang]

Background:
Each osd has two heartbeat messenger instances to maintain front/back
network available. It brings lots of connections and messages overhead
in scale out cluster. Actually we can combine these heartbeat
exchanges to public/cluster messengers to reduce tons of
connections(resources).

Then heartbeat message should be OOB and shared the same thread/socket
with normal message channel. So it can exactly represent the heartbeat
role for real IO message. Otherwise, heartbeat channel's status can't
indicate the real IO message channel status. Because different socket
uses different send buffer/recv buffer, if real io message blocked,
oob message may be healthy.

Besides OSD's heartbeat things, we have logic PING/PONG lived in
Objecter Ping/WatchNotify Ping etc. For the same goal, they could
share the heartbeat message.

In a real rbd use case env, if we combines these ping/pong messages,
thousands of messages could be avoided which means lots of resources.

As we reduce the heartbeat overhead, we can reduce heartbeat interval
and increase frequency which help a lot to the accurate of cluster
failure detection!

Design:

As discussed in Raleigh, we could defines these interfaces:

int Connection::register_oob_message(identitfy_op, callback, interval);

Users like Objecter linger ping could register a "callback" which
generate bufferlist used to be carried by heartbeat message.
"interval" indicate the user's oob message's send interval.

"identitfy_op" indicates who can handle the oob info in peer side.
Like "Ping", "OSDPing" or "LingerPing" as the current message define.

void Dispatcher::ms_dispatch_oob(Message*)

handle the oob message with parsing each oob part.

So lots of timer control in user's side could be avoided via callback
generator. When sending, OOB message could insert the front of send
message queue but we can't get any help from kernel oob flag since
it's really useless..

Any suggestion is welcomed!

Re: OOB message roll into Messenger interface

[Sage Weil]

Hi Haomai!

On Sun, 4 Sep 2016, Haomai Wang wrote:
> Background:
> Each osd has two heartbeat messenger instances to maintain front/back
> network available. It brings lots of connections and messages overhead
> in scale out cluster. Actually we can combine these heartbeat
> exchanges to public/cluster messengers to reduce tons of
> connections(resources).
> 
> Then heartbeat message should be OOB and shared the same thread/socket
> with normal message channel. So it can exactly represent the heartbeat
> role for real IO message. Otherwise, heartbeat channel's status can't
> indicate the real IO message channel status. Because different socket
> uses different send buffer/recv buffer, if real io message blocked,
> oob message may be healthy.
> 
> Besides OSD's heartbeat things, we have logic PING/PONG lived in
> Objecter Ping/WatchNotify Ping etc. For the same goal, they could
> share the heartbeat message.
> 
> In a real rbd use case env, if we combines these ping/pong messages,
> thousands of messages could be avoided which means lots of resources.
> 
> As we reduce the heartbeat overhead, we can reduce heartbeat interval
> and increase frequency which help a lot to the accurate of cluster
> failure detection!

I'm very excited to see this move forward!
 
> Design:
> 
> As discussed in Raleigh, we could defines these interfaces:
> 
> int Connection::register_oob_message(identitfy_op, callback, interval);
> 
> Users like Objecter linger ping could register a "callback" which
> generate bufferlist used to be carried by heartbeat message.
> "interval" indicate the user's oob message's send interval.
> 
> "identitfy_op" indicates who can handle the oob info in peer side.
> Like "Ping", "OSDPing" or "LingerPing" as the current message define.

This looks convenient for the simpler callers, but I worry it won't work 
as well for OSDPing. There's a bunch of odd locking around the heartbeat 
info and the code already exists to do the the heartbeat sends.  I'm not 
sure it will simplify to a simple interval.

An easier first step would be to just define a 
Connection::send_message_oob(Message*).  That would require almost no 
changes to the calling code, and avoid having to create the timing 
infrastructure inside AsyncMessenger...

sage

> void Dispatcher::ms_dispatch_oob(Message*)
> 
> handle the oob message with parsing each oob part.
> 
> So lots of timer control in user's side could be avoided via callback
> generator. When sending, OOB message could insert the front of send
> message queue but we can't get any help from kernel oob flag since
> it's really useless..
> 
> Any suggestion is welcomed!
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: OOB message roll into Messenger interface

[Haomai Wang]

On Tue, Sep 6, 2016 at 9:17 PM, Sage Weil <sage@newdream.net> wrote:
> Hi Haomai!
>
> On Sun, 4 Sep 2016, Haomai Wang wrote:
>> Background:
>> Each osd has two heartbeat messenger instances to maintain front/back
>> network available. It brings lots of connections and messages overhead
>> in scale out cluster. Actually we can combine these heartbeat
>> exchanges to public/cluster messengers to reduce tons of
>> connections(resources).
>>
>> Then heartbeat message should be OOB and shared the same thread/socket
>> with normal message channel. So it can exactly represent the heartbeat
>> role for real IO message. Otherwise, heartbeat channel's status can't
>> indicate the real IO message channel status. Because different socket
>> uses different send buffer/recv buffer, if real io message blocked,
>> oob message may be healthy.
>>
>> Besides OSD's heartbeat things, we have logic PING/PONG lived in
>> Objecter Ping/WatchNotify Ping etc. For the same goal, they could
>> share the heartbeat message.
>>
>> In a real rbd use case env, if we combines these ping/pong messages,
>> thousands of messages could be avoided which means lots of resources.
>>
>> As we reduce the heartbeat overhead, we can reduce heartbeat interval
>> and increase frequency which help a lot to the accurate of cluster
>> failure detection!
>
> I'm very excited to see this move forward!
>
>> Design:
>>
>> As discussed in Raleigh, we could defines these interfaces:
>>
>> int Connection::register_oob_message(identitfy_op, callback, interval);
>>
>> Users like Objecter linger ping could register a "callback" which
>> generate bufferlist used to be carried by heartbeat message.
>> "interval" indicate the user's oob message's send interval.
>>
>> "identitfy_op" indicates who can handle the oob info in peer side.
>> Like "Ping", "OSDPing" or "LingerPing" as the current message define.
>
> This looks convenient for the simpler callers, but I worry it won't work
> as well for OSDPing. There's a bunch of odd locking around the heartbeat
> info and the code already exists to do the the heartbeat sends.  I'm not
> sure it will simplify to a simple interval.

Hmm, I'm not sure what's the odd locking thing refer to. As we can
register callback when adding new peer and unregister callback when
removing peer from "heartbeat_peers".

The main send message construct callback extract from this loop:
  for (map<int,HeartbeatInfo>::iterator i = heartbeat_peers.begin();
       i != heartbeat_peers.end();
       ++i) {
    int peer = i->first;
    i->second.last_tx = now;
    if (i->second.first_tx == utime_t())
      i->second.first_tx = now;
    dout(30) << "heartbeat sending ping to osd." << peer << dendl;
    i->second.con_back->send_message(new MOSDPing(monc->get_fsid(),
 service.get_osdmap()->get_epoch(),
 MOSDPing::PING,
 now));

    if (i->second.con_front)
      i->second.con_front->send_message(new MOSDPing(monc->get_fsid(),
    service.get_osdmap()->get_epoch(),
    MOSDPing::PING,
    now));
  }

Only "fsid", "osdmap epoch" are required, I don't think it will block.
Then I think lots of locking/odding things exists on heartbeat
dispatch/handle process. sending process is clear I guess.

The advantage to register callback is we can combine multi layers oob
messages to one.

>
> An easier first step would be to just define a
> Connection::send_message_oob(Message*).  That would require almost no
> changes to the calling code, and avoid having to create the timing
> infrastructure inside AsyncMessenger...
>
> sage
>
>> void Dispatcher::ms_dispatch_oob(Message*)
>>
>> handle the oob message with parsing each oob part.
>>
>> So lots of timer control in user's side could be avoided via callback
>> generator. When sending, OOB message could insert the front of send
>> message queue but we can't get any help from kernel oob flag since
>> it's really useless..
>>
>> Any suggestion is welcomed!
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>

Re: OOB message roll into Messenger interface

[Sage Weil]

On Tue, 6 Sep 2016, Haomai Wang wrote:
> On Tue, Sep 6, 2016 at 9:17 PM, Sage Weil <sage@newdream.net> wrote:
> > Hi Haomai!
> >
> > On Sun, 4 Sep 2016, Haomai Wang wrote:
> >> Background:
> >> Each osd has two heartbeat messenger instances to maintain front/back
> >> network available. It brings lots of connections and messages overhead
> >> in scale out cluster. Actually we can combine these heartbeat
> >> exchanges to public/cluster messengers to reduce tons of
> >> connections(resources).
> >>
> >> Then heartbeat message should be OOB and shared the same thread/socket
> >> with normal message channel. So it can exactly represent the heartbeat
> >> role for real IO message. Otherwise, heartbeat channel's status can't
> >> indicate the real IO message channel status. Because different socket
> >> uses different send buffer/recv buffer, if real io message blocked,
> >> oob message may be healthy.
> >>
> >> Besides OSD's heartbeat things, we have logic PING/PONG lived in
> >> Objecter Ping/WatchNotify Ping etc. For the same goal, they could
> >> share the heartbeat message.
> >>
> >> In a real rbd use case env, if we combines these ping/pong messages,
> >> thousands of messages could be avoided which means lots of resources.
> >>
> >> As we reduce the heartbeat overhead, we can reduce heartbeat interval
> >> and increase frequency which help a lot to the accurate of cluster
> >> failure detection!
> >
> > I'm very excited to see this move forward!
> >
> >> Design:
> >>
> >> As discussed in Raleigh, we could defines these interfaces:
> >>
> >> int Connection::register_oob_message(identitfy_op, callback, interval);
> >>
> >> Users like Objecter linger ping could register a "callback" which
> >> generate bufferlist used to be carried by heartbeat message.
> >> "interval" indicate the user's oob message's send interval.
> >>
> >> "identitfy_op" indicates who can handle the oob info in peer side.
> >> Like "Ping", "OSDPing" or "LingerPing" as the current message define.
> >
> > This looks convenient for the simpler callers, but I worry it won't work
> > as well for OSDPing. There's a bunch of odd locking around the heartbeat
> > info and the code already exists to do the the heartbeat sends.  I'm not
> > sure it will simplify to a simple interval.
> 
> Hmm, I'm not sure what's the odd locking thing refer to. As we can
> register callback when adding new peer and unregister callback when
> removing peer from "heartbeat_peers".
> 
> The main send message construct callback extract from this loop:
>   for (map<int,HeartbeatInfo>::iterator i = heartbeat_peers.begin();
>        i != heartbeat_peers.end();
>        ++i) {
>     int peer = i->first;
>     i->second.last_tx = now;
>     if (i->second.first_tx == utime_t())
>       i->second.first_tx = now;
>     dout(30) << "heartbeat sending ping to osd." << peer << dendl;
>     i->second.con_back->send_message(new MOSDPing(monc->get_fsid(),
>  service.get_osdmap()->get_epoch(),
>  MOSDPing::PING,
>  now));
> 
>     if (i->second.con_front)
>       i->second.con_front->send_message(new MOSDPing(monc->get_fsid(),
>     service.get_osdmap()->get_epoch(),
>     MOSDPing::PING,
>     now));
>   }
> 
> Only "fsid", "osdmap epoch" are required, I don't think it will block.
> Then I think lots of locking/odding things exists on heartbeat
> dispatch/handle process. sending process is clear I guess.

Yeah, I guess that's fine.  I was worried about some dependency between 
who we ping and the osdmap epoch in the message (and races adding/removing 
heartbeat peers), but I think it doesn't matter.

Even so, I think it would be good to expose the send_message_oob() 
interface, and do this in 2 stages so the two changes are decoupled.  
Unless there is some implementation reason why the oob message scheduling 
needs to be done inside the messenger?

sage

> The advantage to register callback is we can combine multi layers oob
> messages to one.
> 
> >
> > An easier first step would be to just define a
> > Connection::send_message_oob(Message*).  That would require almost no
> > changes to the calling code, and avoid having to create the timing
> > infrastructure inside AsyncMessenger...
> >
> > sage
> >
> >> void Dispatcher::ms_dispatch_oob(Message*)
> >>
> >> handle the oob message with parsing each oob part.
> >>
> >> So lots of timer control in user's side could be avoided via callback
> >> generator. When sending, OOB message could insert the front of send
> >> message queue but we can't get any help from kernel oob flag since
> >> it's really useless..
> >>
> >> Any suggestion is welcomed!
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: OOB message roll into Messenger interface

[Haomai Wang]

On Tue, Sep 6, 2016 at 10:06 PM, Sage Weil <sage@newdream.net> wrote:
> On Tue, 6 Sep 2016, Haomai Wang wrote:
>> On Tue, Sep 6, 2016 at 9:17 PM, Sage Weil <sage@newdream.net> wrote:
>> > Hi Haomai!
>> >
>> > On Sun, 4 Sep 2016, Haomai Wang wrote:
>> >> Background:
>> >> Each osd has two heartbeat messenger instances to maintain front/back
>> >> network available. It brings lots of connections and messages overhead
>> >> in scale out cluster. Actually we can combine these heartbeat
>> >> exchanges to public/cluster messengers to reduce tons of
>> >> connections(resources).
>> >>
>> >> Then heartbeat message should be OOB and shared the same thread/socket
>> >> with normal message channel. So it can exactly represent the heartbeat
>> >> role for real IO message. Otherwise, heartbeat channel's status can't
>> >> indicate the real IO message channel status. Because different socket
>> >> uses different send buffer/recv buffer, if real io message blocked,
>> >> oob message may be healthy.
>> >>
>> >> Besides OSD's heartbeat things, we have logic PING/PONG lived in
>> >> Objecter Ping/WatchNotify Ping etc. For the same goal, they could
>> >> share the heartbeat message.
>> >>
>> >> In a real rbd use case env, if we combines these ping/pong messages,
>> >> thousands of messages could be avoided which means lots of resources.
>> >>
>> >> As we reduce the heartbeat overhead, we can reduce heartbeat interval
>> >> and increase frequency which help a lot to the accurate of cluster
>> >> failure detection!
>> >
>> > I'm very excited to see this move forward!
>> >
>> >> Design:
>> >>
>> >> As discussed in Raleigh, we could defines these interfaces:
>> >>
>> >> int Connection::register_oob_message(identitfy_op, callback, interval);
>> >>
>> >> Users like Objecter linger ping could register a "callback" which
>> >> generate bufferlist used to be carried by heartbeat message.
>> >> "interval" indicate the user's oob message's send interval.
>> >>
>> >> "identitfy_op" indicates who can handle the oob info in peer side.
>> >> Like "Ping", "OSDPing" or "LingerPing" as the current message define.
>> >
>> > This looks convenient for the simpler callers, but I worry it won't work
>> > as well for OSDPing. There's a bunch of odd locking around the heartbeat
>> > info and the code already exists to do the the heartbeat sends.  I'm not
>> > sure it will simplify to a simple interval.
>>
>> Hmm, I'm not sure what's the odd locking thing refer to. As we can
>> register callback when adding new peer and unregister callback when
>> removing peer from "heartbeat_peers".
>>
>> The main send message construct callback extract from this loop:
>>   for (map<int,HeartbeatInfo>::iterator i = heartbeat_peers.begin();
>>        i != heartbeat_peers.end();
>>        ++i) {
>>     int peer = i->first;
>>     i->second.last_tx = now;
>>     if (i->second.first_tx == utime_t())
>>       i->second.first_tx = now;
>>     dout(30) << "heartbeat sending ping to osd." << peer << dendl;
>>     i->second.con_back->send_message(new MOSDPing(monc->get_fsid(),
>>  service.get_osdmap()->get_epoch(),
>>  MOSDPing::PING,
>>  now));
>>
>>     if (i->second.con_front)
>>       i->second.con_front->send_message(new MOSDPing(monc->get_fsid(),
>>     service.get_osdmap()->get_epoch(),
>>     MOSDPing::PING,
>>     now));
>>   }
>>
>> Only "fsid", "osdmap epoch" are required, I don't think it will block.
>> Then I think lots of locking/odding things exists on heartbeat
>> dispatch/handle process. sending process is clear I guess.
>
> Yeah, I guess that's fine.  I was worried about some dependency between
> who we ping and the osdmap epoch in the message (and races adding/removing
> heartbeat peers), but I think it doesn't matter.
>
> Even so, I think it would be good to expose the send_message_oob()
> interface, and do this in 2 stages so the two changes are decoupled.
> Unless there is some implementation reason why the oob message scheduling
> needs to be done inside the messenger?

Agreed! we could remove heartbeat messenger firstly!

>
> sage
>
>> The advantage to register callback is we can combine multi layers oob
>> messages to one.
>>
>> >
>> > An easier first step would be to just define a
>> > Connection::send_message_oob(Message*).  That would require almost no
>> > changes to the calling code, and avoid having to create the timing
>> > infrastructure inside AsyncMessenger...
>> >
>> > sage
>> >
>> >> void Dispatcher::ms_dispatch_oob(Message*)
>> >>
>> >> handle the oob message with parsing each oob part.
>> >>
>> >> So lots of timer control in user's side could be avoided via callback
>> >> generator. When sending, OOB message could insert the front of send
>> >> message queue but we can't get any help from kernel oob flag since
>> >> it's really useless..
>> >>
>> >> Any suggestion is welcomed!
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >>
>> >>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>

Re: OOB message roll into Messenger interface

[Gregory Farnum]

On Tue, Sep 6, 2016 at 7:15 AM, Haomai Wang <haomai@xsky.com> wrote:
> On Tue, Sep 6, 2016 at 10:06 PM, Sage Weil <sage@newdream.net> wrote:
>> On Tue, 6 Sep 2016, Haomai Wang wrote:
>>> On Tue, Sep 6, 2016 at 9:17 PM, Sage Weil <sage@newdream.net> wrote:
>>> > Hi Haomai!
>>> >
>>> > On Sun, 4 Sep 2016, Haomai Wang wrote:
>>> >> Background:
>>> >> Each osd has two heartbeat messenger instances to maintain front/back
>>> >> network available. It brings lots of connections and messages overhead
>>> >> in scale out cluster. Actually we can combine these heartbeat
>>> >> exchanges to public/cluster messengers to reduce tons of
>>> >> connections(resources).
>>> >>
>>> >> Then heartbeat message should be OOB and shared the same thread/socket
>>> >> with normal message channel. So it can exactly represent the heartbeat
>>> >> role for real IO message. Otherwise, heartbeat channel's status can't
>>> >> indicate the real IO message channel status. Because different socket
>>> >> uses different send buffer/recv buffer, if real io message blocked,
>>> >> oob message may be healthy.
>>> >>
>>> >> Besides OSD's heartbeat things, we have logic PING/PONG lived in
>>> >> Objecter Ping/WatchNotify Ping etc. For the same goal, they could
>>> >> share the heartbeat message.
>>> >>
>>> >> In a real rbd use case env, if we combines these ping/pong messages,
>>> >> thousands of messages could be avoided which means lots of resources.
>>> >>
>>> >> As we reduce the heartbeat overhead, we can reduce heartbeat interval
>>> >> and increase frequency which help a lot to the accurate of cluster
>>> >> failure detection!
>>> >
>>> > I'm very excited to see this move forward!
>>> >
>>> >> Design:
>>> >>
>>> >> As discussed in Raleigh, we could defines these interfaces:
>>> >>
>>> >> int Connection::register_oob_message(identitfy_op, callback, interval);
>>> >>
>>> >> Users like Objecter linger ping could register a "callback" which
>>> >> generate bufferlist used to be carried by heartbeat message.
>>> >> "interval" indicate the user's oob message's send interval.
>>> >>
>>> >> "identitfy_op" indicates who can handle the oob info in peer side.
>>> >> Like "Ping", "OSDPing" or "LingerPing" as the current message define.
>>> >
>>> > This looks convenient for the simpler callers, but I worry it won't work
>>> > as well for OSDPing. There's a bunch of odd locking around the heartbeat
>>> > info and the code already exists to do the the heartbeat sends.  I'm not
>>> > sure it will simplify to a simple interval.
>>>
>>> Hmm, I'm not sure what's the odd locking thing refer to. As we can
>>> register callback when adding new peer and unregister callback when
>>> removing peer from "heartbeat_peers".
>>>
>>> The main send message construct callback extract from this loop:
>>>   for (map<int,HeartbeatInfo>::iterator i = heartbeat_peers.begin();
>>>        i != heartbeat_peers.end();
>>>        ++i) {
>>>     int peer = i->first;
>>>     i->second.last_tx = now;
>>>     if (i->second.first_tx == utime_t())
>>>       i->second.first_tx = now;
>>>     dout(30) << "heartbeat sending ping to osd." << peer << dendl;
>>>     i->second.con_back->send_message(new MOSDPing(monc->get_fsid(),
>>>  service.get_osdmap()->get_epoch(),
>>>  MOSDPing::PING,
>>>  now));
>>>
>>>     if (i->second.con_front)
>>>       i->second.con_front->send_message(new MOSDPing(monc->get_fsid(),
>>>     service.get_osdmap()->get_epoch(),
>>>     MOSDPing::PING,
>>>     now));
>>>   }
>>>
>>> Only "fsid", "osdmap epoch" are required, I don't think it will block.
>>> Then I think lots of locking/odding things exists on heartbeat
>>> dispatch/handle process. sending process is clear I guess.
>>
>> Yeah, I guess that's fine.  I was worried about some dependency between
>> who we ping and the osdmap epoch in the message (and races adding/removing
>> heartbeat peers), but I think it doesn't matter.
>>
>> Even so, I think it would be good to expose the send_message_oob()
>> interface, and do this in 2 stages so the two changes are decoupled.
>> Unless there is some implementation reason why the oob message scheduling
>> needs to be done inside the messenger?
>
> Agreed! we could remove heartbeat messenger firstly!
>
>>
>> sage
>>
>>> The advantage to register callback is we can combine multi layers oob
>>> messages to one.
>>>
>>> >
>>> > An easier first step would be to just define a
>>> > Connection::send_message_oob(Message*).  That would require almost no
>>> > changes to the calling code, and avoid having to create the timing
>>> > infrastructure inside AsyncMessenger...
>>> >
>>> > sage
>>> >
>>> >> void Dispatcher::ms_dispatch_oob(Message*)
>>> >>
>>> >> handle the oob message with parsing each oob part.
>>> >>
>>> >> So lots of timer control in user's side could be avoided via callback
>>> >> generator. When sending, OOB message could insert the front of send
>>> >> message queue but we can't get any help from kernel oob flag since
>>> >> it's really useless..
>>> >>
>>> >> Any suggestion is welcomed!

Let's keep in mind the challenges of out-of-band messaging over TCP/IP.

Namely, when we discussed this we couldn't figure out any way
(including the TCP priority stuff, which doesn't work with the
required semantics — even when it does function) to get traffic to
actually go out-of-band. IB messaging systems actually have a
"channels" concept that lets you do genuine OOB transmission that
skips over queues and other data; TCP doesn't. In fact the best we
came up with for doing this with Simple/AsyncMessenger was giving the
Messenger duplicate sockets/queues/etc, which is hardly ideal.

So, maybe we can remove the heartbeat messenger by giving each
Connection two sockets and queues. That might even work better for the
AsyncMessenger than it does for SimpleMessenger?
But any implementation that orders OSD heartbeat messages behind
ordinary data traffic in kernel or router buffers is probably going to
fail us. :(
-Greg

Re: OOB message roll into Messenger interface

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 6505 bytes --]

On Tue, 6 Sep 2016, Gregory Farnum wrote:
> On Tue, Sep 6, 2016 at 7:15 AM, Haomai Wang <haomai@xsky.com> wrote:
> > On Tue, Sep 6, 2016 at 10:06 PM, Sage Weil <sage@newdream.net> wrote:
> >> On Tue, 6 Sep 2016, Haomai Wang wrote:
> >>> On Tue, Sep 6, 2016 at 9:17 PM, Sage Weil <sage@newdream.net> wrote:
> >>> > Hi Haomai!
> >>> >
> >>> > On Sun, 4 Sep 2016, Haomai Wang wrote:
> >>> >> Background:
> >>> >> Each osd has two heartbeat messenger instances to maintain front/back
> >>> >> network available. It brings lots of connections and messages overhead
> >>> >> in scale out cluster. Actually we can combine these heartbeat
> >>> >> exchanges to public/cluster messengers to reduce tons of
> >>> >> connections(resources).
> >>> >>
> >>> >> Then heartbeat message should be OOB and shared the same thread/socket
> >>> >> with normal message channel. So it can exactly represent the heartbeat
> >>> >> role for real IO message. Otherwise, heartbeat channel's status can't
> >>> >> indicate the real IO message channel status. Because different socket
> >>> >> uses different send buffer/recv buffer, if real io message blocked,
> >>> >> oob message may be healthy.
> >>> >>
> >>> >> Besides OSD's heartbeat things, we have logic PING/PONG lived in
> >>> >> Objecter Ping/WatchNotify Ping etc. For the same goal, they could
> >>> >> share the heartbeat message.
> >>> >>
> >>> >> In a real rbd use case env, if we combines these ping/pong messages,
> >>> >> thousands of messages could be avoided which means lots of resources.
> >>> >>
> >>> >> As we reduce the heartbeat overhead, we can reduce heartbeat interval
> >>> >> and increase frequency which help a lot to the accurate of cluster
> >>> >> failure detection!
> >>> >
> >>> > I'm very excited to see this move forward!
> >>> >
> >>> >> Design:
> >>> >>
> >>> >> As discussed in Raleigh, we could defines these interfaces:
> >>> >>
> >>> >> int Connection::register_oob_message(identitfy_op, callback, interval);
> >>> >>
> >>> >> Users like Objecter linger ping could register a "callback" which
> >>> >> generate bufferlist used to be carried by heartbeat message.
> >>> >> "interval" indicate the user's oob message's send interval.
> >>> >>
> >>> >> "identitfy_op" indicates who can handle the oob info in peer side.
> >>> >> Like "Ping", "OSDPing" or "LingerPing" as the current message define.
> >>> >
> >>> > This looks convenient for the simpler callers, but I worry it won't work
> >>> > as well for OSDPing. There's a bunch of odd locking around the heartbeat
> >>> > info and the code already exists to do the the heartbeat sends.  I'm not
> >>> > sure it will simplify to a simple interval.
> >>>
> >>> Hmm, I'm not sure what's the odd locking thing refer to. As we can
> >>> register callback when adding new peer and unregister callback when
> >>> removing peer from "heartbeat_peers".
> >>>
> >>> The main send message construct callback extract from this loop:
> >>>   for (map<int,HeartbeatInfo>::iterator i = heartbeat_peers.begin();
> >>>        i != heartbeat_peers.end();
> >>>        ++i) {
> >>>     int peer = i->first;
> >>>     i->second.last_tx = now;
> >>>     if (i->second.first_tx == utime_t())
> >>>       i->second.first_tx = now;
> >>>     dout(30) << "heartbeat sending ping to osd." << peer << dendl;
> >>>     i->second.con_back->send_message(new MOSDPing(monc->get_fsid(),
> >>>  service.get_osdmap()->get_epoch(),
> >>>  MOSDPing::PING,
> >>>  now));
> >>>
> >>>     if (i->second.con_front)
> >>>       i->second.con_front->send_message(new MOSDPing(monc->get_fsid(),
> >>>     service.get_osdmap()->get_epoch(),
> >>>     MOSDPing::PING,
> >>>     now));
> >>>   }
> >>>
> >>> Only "fsid", "osdmap epoch" are required, I don't think it will block.
> >>> Then I think lots of locking/odding things exists on heartbeat
> >>> dispatch/handle process. sending process is clear I guess.
> >>
> >> Yeah, I guess that's fine.  I was worried about some dependency between
> >> who we ping and the osdmap epoch in the message (and races adding/removing
> >> heartbeat peers), but I think it doesn't matter.
> >>
> >> Even so, I think it would be good to expose the send_message_oob()
> >> interface, and do this in 2 stages so the two changes are decoupled.
> >> Unless there is some implementation reason why the oob message scheduling
> >> needs to be done inside the messenger?
> >
> > Agreed! we could remove heartbeat messenger firstly!
> >
> >>
> >> sage
> >>
> >>> The advantage to register callback is we can combine multi layers oob
> >>> messages to one.
> >>>
> >>> >
> >>> > An easier first step would be to just define a
> >>> > Connection::send_message_oob(Message*).  That would require almost no
> >>> > changes to the calling code, and avoid having to create the timing
> >>> > infrastructure inside AsyncMessenger...
> >>> >
> >>> > sage
> >>> >
> >>> >> void Dispatcher::ms_dispatch_oob(Message*)
> >>> >>
> >>> >> handle the oob message with parsing each oob part.
> >>> >>
> >>> >> So lots of timer control in user's side could be avoided via callback
> >>> >> generator. When sending, OOB message could insert the front of send
> >>> >> message queue but we can't get any help from kernel oob flag since
> >>> >> it's really useless..
> >>> >>
> >>> >> Any suggestion is welcomed!
> 
> Let's keep in mind the challenges of out-of-band messaging over TCP/IP.
> 
> Namely, when we discussed this we couldn't figure out any way
> (including the TCP priority stuff, which doesn't work with the
> required semantics — even when it does function) to get traffic to
> actually go out-of-band. IB messaging systems actually have a
> "channels" concept that lets you do genuine OOB transmission that
> skips over queues and other data; TCP doesn't. In fact the best we
> came up with for doing this with Simple/AsyncMessenger was giving the
> Messenger duplicate sockets/queues/etc, which is hardly ideal.
> 
> So, maybe we can remove the heartbeat messenger by giving each
> Connection two sockets and queues. That might even work better for the
> AsyncMessenger than it does for SimpleMessenger?
> But any implementation that orders OSD heartbeat messages behind
> ordinary data traffic in kernel or router buffers is probably going to
> fail us. :(

Oh, good point.  I didn't read that paragraph carefully.  I think we 
should use a second socket connected to the same address for OOB messages.  
Or possibly push them over UDP... but we'd need to define retry semantics 
in that case.

sage

Re: OOB message roll into Messenger interface

[Haomai Wang]

On Wed, Sep 7, 2016 at 1:42 AM, Gregory Farnum <gfarnum@redhat.com> wrote:
> On Tue, Sep 6, 2016 at 7:15 AM, Haomai Wang <haomai@xsky.com> wrote:
>> On Tue, Sep 6, 2016 at 10:06 PM, Sage Weil <sage@newdream.net> wrote:
>>> On Tue, 6 Sep 2016, Haomai Wang wrote:
>>>> On Tue, Sep 6, 2016 at 9:17 PM, Sage Weil <sage@newdream.net> wrote:
>>>> > Hi Haomai!
>>>> >
>>>> > On Sun, 4 Sep 2016, Haomai Wang wrote:
>>>> >> Background:
>>>> >> Each osd has two heartbeat messenger instances to maintain front/back
>>>> >> network available. It brings lots of connections and messages overhead
>>>> >> in scale out cluster. Actually we can combine these heartbeat
>>>> >> exchanges to public/cluster messengers to reduce tons of
>>>> >> connections(resources).
>>>> >>
>>>> >> Then heartbeat message should be OOB and shared the same thread/socket
>>>> >> with normal message channel. So it can exactly represent the heartbeat
>>>> >> role for real IO message. Otherwise, heartbeat channel's status can't
>>>> >> indicate the real IO message channel status. Because different socket
>>>> >> uses different send buffer/recv buffer, if real io message blocked,
>>>> >> oob message may be healthy.
>>>> >>
>>>> >> Besides OSD's heartbeat things, we have logic PING/PONG lived in
>>>> >> Objecter Ping/WatchNotify Ping etc. For the same goal, they could
>>>> >> share the heartbeat message.
>>>> >>
>>>> >> In a real rbd use case env, if we combines these ping/pong messages,
>>>> >> thousands of messages could be avoided which means lots of resources.
>>>> >>
>>>> >> As we reduce the heartbeat overhead, we can reduce heartbeat interval
>>>> >> and increase frequency which help a lot to the accurate of cluster
>>>> >> failure detection!
>>>> >
>>>> > I'm very excited to see this move forward!
>>>> >
>>>> >> Design:
>>>> >>
>>>> >> As discussed in Raleigh, we could defines these interfaces:
>>>> >>
>>>> >> int Connection::register_oob_message(identitfy_op, callback, interval);
>>>> >>
>>>> >> Users like Objecter linger ping could register a "callback" which
>>>> >> generate bufferlist used to be carried by heartbeat message.
>>>> >> "interval" indicate the user's oob message's send interval.
>>>> >>
>>>> >> "identitfy_op" indicates who can handle the oob info in peer side.
>>>> >> Like "Ping", "OSDPing" or "LingerPing" as the current message define.
>>>> >
>>>> > This looks convenient for the simpler callers, but I worry it won't work
>>>> > as well for OSDPing. There's a bunch of odd locking around the heartbeat
>>>> > info and the code already exists to do the the heartbeat sends.  I'm not
>>>> > sure it will simplify to a simple interval.
>>>>
>>>> Hmm, I'm not sure what's the odd locking thing refer to. As we can
>>>> register callback when adding new peer and unregister callback when
>>>> removing peer from "heartbeat_peers".
>>>>
>>>> The main send message construct callback extract from this loop:
>>>>   for (map<int,HeartbeatInfo>::iterator i = heartbeat_peers.begin();
>>>>        i != heartbeat_peers.end();
>>>>        ++i) {
>>>>     int peer = i->first;
>>>>     i->second.last_tx = now;
>>>>     if (i->second.first_tx == utime_t())
>>>>       i->second.first_tx = now;
>>>>     dout(30) << "heartbeat sending ping to osd." << peer << dendl;
>>>>     i->second.con_back->send_message(new MOSDPing(monc->get_fsid(),
>>>>  service.get_osdmap()->get_epoch(),
>>>>  MOSDPing::PING,
>>>>  now));
>>>>
>>>>     if (i->second.con_front)
>>>>       i->second.con_front->send_message(new MOSDPing(monc->get_fsid(),
>>>>     service.get_osdmap()->get_epoch(),
>>>>     MOSDPing::PING,
>>>>     now));
>>>>   }
>>>>
>>>> Only "fsid", "osdmap epoch" are required, I don't think it will block.
>>>> Then I think lots of locking/odding things exists on heartbeat
>>>> dispatch/handle process. sending process is clear I guess.
>>>
>>> Yeah, I guess that's fine.  I was worried about some dependency between
>>> who we ping and the osdmap epoch in the message (and races adding/removing
>>> heartbeat peers), but I think it doesn't matter.
>>>
>>> Even so, I think it would be good to expose the send_message_oob()
>>> interface, and do this in 2 stages so the two changes are decoupled.
>>> Unless there is some implementation reason why the oob message scheduling
>>> needs to be done inside the messenger?
>>
>> Agreed! we could remove heartbeat messenger firstly!
>>
>>>
>>> sage
>>>
>>>> The advantage to register callback is we can combine multi layers oob
>>>> messages to one.
>>>>
>>>> >
>>>> > An easier first step would be to just define a
>>>> > Connection::send_message_oob(Message*).  That would require almost no
>>>> > changes to the calling code, and avoid having to create the timing
>>>> > infrastructure inside AsyncMessenger...
>>>> >
>>>> > sage
>>>> >
>>>> >> void Dispatcher::ms_dispatch_oob(Message*)
>>>> >>
>>>> >> handle the oob message with parsing each oob part.
>>>> >>
>>>> >> So lots of timer control in user's side could be avoided via callback
>>>> >> generator. When sending, OOB message could insert the front of send
>>>> >> message queue but we can't get any help from kernel oob flag since
>>>> >> it's really useless..
>>>> >>
>>>> >> Any suggestion is welcomed!
>
> Let's keep in mind the challenges of out-of-band messaging over TCP/IP.
>
> Namely, when we discussed this we couldn't figure out any way
> (including the TCP priority stuff, which doesn't work with the
> required semantics — even when it does function) to get traffic to
> actually go out-of-band. IB messaging systems actually have a
> "channels" concept that lets you do genuine OOB transmission that
> skips over queues and other data; TCP doesn't. In fact the best we
> came up with for doing this with Simple/AsyncMessenger was giving the
> Messenger duplicate sockets/queues/etc, which is hardly ideal.

hmm, I also think about oob message may not deliver when the socket is
busy,  do we consider to add oob udp socket associated to
Connection(hb msg should be lossy, so we don't need to handle errors).
So in the upper layer, we don't explicit maintain heartbeat
connection. I can see we can achieve the same goal mentioned above.

Then when we target to msgr v2, we still want to move heartbeat logic
to messenger layer to aggregate hb msg in multi entity case.

>
> So, maybe we can remove the heartbeat messenger by giving each
> Connection two sockets and queues. That might even work better for the
> AsyncMessenger than it does for SimpleMessenger?

actually asyncmessenger also plays a lightweight role compared to
simplemessenger, so we may not need to do this.

> But any implementation that orders OSD heartbeat messages behind
> ordinary data traffic in kernel or router buffers is probably going to
> fail us. :(
> -Greg

Re: OOB message roll into Messenger interface

[Haomai Wang]

On Wed, Sep 7, 2016 at 2:06 AM, Sage Weil <sage@newdream.net> wrote:
> On Tue, 6 Sep 2016, Gregory Farnum wrote:
>> On Tue, Sep 6, 2016 at 7:15 AM, Haomai Wang <haomai@xsky.com> wrote:
>> > On Tue, Sep 6, 2016 at 10:06 PM, Sage Weil <sage@newdream.net> wrote:
>> >> On Tue, 6 Sep 2016, Haomai Wang wrote:
>> >>> On Tue, Sep 6, 2016 at 9:17 PM, Sage Weil <sage@newdream.net> wrote:
>> >>> > Hi Haomai!
>> >>> >
>> >>> > On Sun, 4 Sep 2016, Haomai Wang wrote:
>> >>> >> Background:
>> >>> >> Each osd has two heartbeat messenger instances to maintain front/back
>> >>> >> network available. It brings lots of connections and messages overhead
>> >>> >> in scale out cluster. Actually we can combine these heartbeat
>> >>> >> exchanges to public/cluster messengers to reduce tons of
>> >>> >> connections(resources).
>> >>> >>
>> >>> >> Then heartbeat message should be OOB and shared the same thread/socket
>> >>> >> with normal message channel. So it can exactly represent the heartbeat
>> >>> >> role for real IO message. Otherwise, heartbeat channel's status can't
>> >>> >> indicate the real IO message channel status. Because different socket
>> >>> >> uses different send buffer/recv buffer, if real io message blocked,
>> >>> >> oob message may be healthy.
>> >>> >>
>> >>> >> Besides OSD's heartbeat things, we have logic PING/PONG lived in
>> >>> >> Objecter Ping/WatchNotify Ping etc. For the same goal, they could
>> >>> >> share the heartbeat message.
>> >>> >>
>> >>> >> In a real rbd use case env, if we combines these ping/pong messages,
>> >>> >> thousands of messages could be avoided which means lots of resources.
>> >>> >>
>> >>> >> As we reduce the heartbeat overhead, we can reduce heartbeat interval
>> >>> >> and increase frequency which help a lot to the accurate of cluster
>> >>> >> failure detection!
>> >>> >
>> >>> > I'm very excited to see this move forward!
>> >>> >
>> >>> >> Design:
>> >>> >>
>> >>> >> As discussed in Raleigh, we could defines these interfaces:
>> >>> >>
>> >>> >> int Connection::register_oob_message(identitfy_op, callback, interval);
>> >>> >>
>> >>> >> Users like Objecter linger ping could register a "callback" which
>> >>> >> generate bufferlist used to be carried by heartbeat message.
>> >>> >> "interval" indicate the user's oob message's send interval.
>> >>> >>
>> >>> >> "identitfy_op" indicates who can handle the oob info in peer side.
>> >>> >> Like "Ping", "OSDPing" or "LingerPing" as the current message define.
>> >>> >
>> >>> > This looks convenient for the simpler callers, but I worry it won't work
>> >>> > as well for OSDPing. There's a bunch of odd locking around the heartbeat
>> >>> > info and the code already exists to do the the heartbeat sends.  I'm not
>> >>> > sure it will simplify to a simple interval.
>> >>>
>> >>> Hmm, I'm not sure what's the odd locking thing refer to. As we can
>> >>> register callback when adding new peer and unregister callback when
>> >>> removing peer from "heartbeat_peers".
>> >>>
>> >>> The main send message construct callback extract from this loop:
>> >>>   for (map<int,HeartbeatInfo>::iterator i = heartbeat_peers.begin();
>> >>>        i != heartbeat_peers.end();
>> >>>        ++i) {
>> >>>     int peer = i->first;
>> >>>     i->second.last_tx = now;
>> >>>     if (i->second.first_tx == utime_t())
>> >>>       i->second.first_tx = now;
>> >>>     dout(30) << "heartbeat sending ping to osd." << peer << dendl;
>> >>>     i->second.con_back->send_message(new MOSDPing(monc->get_fsid(),
>> >>>  service.get_osdmap()->get_epoch(),
>> >>>  MOSDPing::PING,
>> >>>  now));
>> >>>
>> >>>     if (i->second.con_front)
>> >>>       i->second.con_front->send_message(new MOSDPing(monc->get_fsid(),
>> >>>     service.get_osdmap()->get_epoch(),
>> >>>     MOSDPing::PING,
>> >>>     now));
>> >>>   }
>> >>>
>> >>> Only "fsid", "osdmap epoch" are required, I don't think it will block.
>> >>> Then I think lots of locking/odding things exists on heartbeat
>> >>> dispatch/handle process. sending process is clear I guess.
>> >>
>> >> Yeah, I guess that's fine.  I was worried about some dependency between
>> >> who we ping and the osdmap epoch in the message (and races adding/removing
>> >> heartbeat peers), but I think it doesn't matter.
>> >>
>> >> Even so, I think it would be good to expose the send_message_oob()
>> >> interface, and do this in 2 stages so the two changes are decoupled.
>> >> Unless there is some implementation reason why the oob message scheduling
>> >> needs to be done inside the messenger?
>> >
>> > Agreed! we could remove heartbeat messenger firstly!
>> >
>> >>
>> >> sage
>> >>
>> >>> The advantage to register callback is we can combine multi layers oob
>> >>> messages to one.
>> >>>
>> >>> >
>> >>> > An easier first step would be to just define a
>> >>> > Connection::send_message_oob(Message*).  That would require almost no
>> >>> > changes to the calling code, and avoid having to create the timing
>> >>> > infrastructure inside AsyncMessenger...
>> >>> >
>> >>> > sage
>> >>> >
>> >>> >> void Dispatcher::ms_dispatch_oob(Message*)
>> >>> >>
>> >>> >> handle the oob message with parsing each oob part.
>> >>> >>
>> >>> >> So lots of timer control in user's side could be avoided via callback
>> >>> >> generator. When sending, OOB message could insert the front of send
>> >>> >> message queue but we can't get any help from kernel oob flag since
>> >>> >> it's really useless..
>> >>> >>
>> >>> >> Any suggestion is welcomed!
>>
>> Let's keep in mind the challenges of out-of-band messaging over TCP/IP.
>>
>> Namely, when we discussed this we couldn't figure out any way
>> (including the TCP priority stuff, which doesn't work with the
>> required semantics — even when it does function) to get traffic to
>> actually go out-of-band. IB messaging systems actually have a
>> "channels" concept that lets you do genuine OOB transmission that
>> skips over queues and other data; TCP doesn't. In fact the best we
>> came up with for doing this with Simple/AsyncMessenger was giving the
>> Messenger duplicate sockets/queues/etc, which is hardly ideal.
>>
>> So, maybe we can remove the heartbeat messenger by giving each
>> Connection two sockets and queues. That might even work better for the
>> AsyncMessenger than it does for SimpleMessenger?
>> But any implementation that orders OSD heartbeat messages behind
>> ordinary data traffic in kernel or router buffers is probably going to
>> fail us. :(
>
> Oh, good point.  I didn't read that paragraph carefully.  I think we
> should use a second socket connected to the same address for OOB messages.
> Or possibly push them over UDP... but we'd need to define retry semantics
> in that case.

if udp, I think udp hb interval should be less, caller should delegate
send logic to connection...

>
> sage

Re: OOB message roll into Messenger interface

[Matt Benjamin]

ifaict, sending over UDP brings lossy delivery

will that in general be acceptable?

Matt

----- Original Message -----
> From: "Haomai Wang" <haomai@xsky.com>
> To: "Sage Weil" <sage@newdream.net>
> Cc: "Gregory Farnum" <gfarnum@redhat.com>, ceph-devel@vger.kernel.org
> Sent: Tuesday, September 6, 2016 10:46:11 PM
> Subject: Re: OOB message roll into Messenger interface
> 
> On Wed, Sep 7, 2016 at 2:06 AM, Sage Weil <sage@newdream.net> wrote:
> > On Tue, 6 Sep 2016, Gregory Farnum wrote:
> >> On Tue, Sep 6, 2016 at 7:15 AM, Haomai Wang <haomai@xsky.com> wrote:
> >> > On Tue, Sep 6, 2016 at 10:06 PM, Sage Weil <sage@newdream.net> wrote:
> >> >> On Tue, 6 Sep 2016, Haomai Wang wrote:
> >> >>> On Tue, Sep 6, 2016 at 9:17 PM, Sage Weil <sage@newdream.net> wrote:
> >> >>> > Hi Haomai!
> >> >>> >
> >> >>> > On Sun, 4 Sep 2016, Haomai Wang wrote:
> >> >>> >> Background:
> >> >>> >> Each osd has two heartbeat messenger instances to maintain
> >> >>> >> front/back
> >> >>> >> network available. It brings lots of connections and messages
> >> >>> >> overhead
> >> >>> >> in scale out cluster. Actually we can combine these heartbeat
> >> >>> >> exchanges to public/cluster messengers to reduce tons of
> >> >>> >> connections(resources).
> >> >>> >>
> >> >>> >> Then heartbeat message should be OOB and shared the same
> >> >>> >> thread/socket
> >> >>> >> with normal message channel. So it can exactly represent the
> >> >>> >> heartbeat
> >> >>> >> role for real IO message. Otherwise, heartbeat channel's status
> >> >>> >> can't
> >> >>> >> indicate the real IO message channel status. Because different
> >> >>> >> socket
> >> >>> >> uses different send buffer/recv buffer, if real io message blocked,
> >> >>> >> oob message may be healthy.
> >> >>> >>
> >> >>> >> Besides OSD's heartbeat things, we have logic PING/PONG lived in
> >> >>> >> Objecter Ping/WatchNotify Ping etc. For the same goal, they could
> >> >>> >> share the heartbeat message.
> >> >>> >>
> >> >>> >> In a real rbd use case env, if we combines these ping/pong
> >> >>> >> messages,
> >> >>> >> thousands of messages could be avoided which means lots of
> >> >>> >> resources.
> >> >>> >>
> >> >>> >> As we reduce the heartbeat overhead, we can reduce heartbeat
> >> >>> >> interval
> >> >>> >> and increase frequency which help a lot to the accurate of cluster
> >> >>> >> failure detection!
> >> >>> >
> >> >>> > I'm very excited to see this move forward!
> >> >>> >
> >> >>> >> Design:
> >> >>> >>
> >> >>> >> As discussed in Raleigh, we could defines these interfaces:
> >> >>> >>
> >> >>> >> int Connection::register_oob_message(identitfy_op, callback,
> >> >>> >> interval);
> >> >>> >>
> >> >>> >> Users like Objecter linger ping could register a "callback" which
> >> >>> >> generate bufferlist used to be carried by heartbeat message.
> >> >>> >> "interval" indicate the user's oob message's send interval.
> >> >>> >>
> >> >>> >> "identitfy_op" indicates who can handle the oob info in peer side.
> >> >>> >> Like "Ping", "OSDPing" or "LingerPing" as the current message
> >> >>> >> define.
> >> >>> >
> >> >>> > This looks convenient for the simpler callers, but I worry it won't
> >> >>> > work
> >> >>> > as well for OSDPing. There's a bunch of odd locking around the
> >> >>> > heartbeat
> >> >>> > info and the code already exists to do the the heartbeat sends.  I'm
> >> >>> > not
> >> >>> > sure it will simplify to a simple interval.
> >> >>>
> >> >>> Hmm, I'm not sure what's the odd locking thing refer to. As we can
> >> >>> register callback when adding new peer and unregister callback when
> >> >>> removing peer from "heartbeat_peers".
> >> >>>
> >> >>> The main send message construct callback extract from this loop:
> >> >>>   for (map<int,HeartbeatInfo>::iterator i = heartbeat_peers.begin();
> >> >>>        i != heartbeat_peers.end();
> >> >>>        ++i) {
> >> >>>     int peer = i->first;
> >> >>>     i->second.last_tx = now;
> >> >>>     if (i->second.first_tx == utime_t())
> >> >>>       i->second.first_tx = now;
> >> >>>     dout(30) << "heartbeat sending ping to osd." << peer << dendl;
> >> >>>     i->second.con_back->send_message(new MOSDPing(monc->get_fsid(),
> >> >>>  service.get_osdmap()->get_epoch(),
> >> >>>  MOSDPing::PING,
> >> >>>  now));
> >> >>>
> >> >>>     if (i->second.con_front)
> >> >>>       i->second.con_front->send_message(new MOSDPing(monc->get_fsid(),
> >> >>>     service.get_osdmap()->get_epoch(),
> >> >>>     MOSDPing::PING,
> >> >>>     now));
> >> >>>   }
> >> >>>
> >> >>> Only "fsid", "osdmap epoch" are required, I don't think it will block.
> >> >>> Then I think lots of locking/odding things exists on heartbeat
> >> >>> dispatch/handle process. sending process is clear I guess.
> >> >>
> >> >> Yeah, I guess that's fine.  I was worried about some dependency between
> >> >> who we ping and the osdmap epoch in the message (and races
> >> >> adding/removing
> >> >> heartbeat peers), but I think it doesn't matter.
> >> >>
> >> >> Even so, I think it would be good to expose the send_message_oob()
> >> >> interface, and do this in 2 stages so the two changes are decoupled.
> >> >> Unless there is some implementation reason why the oob message
> >> >> scheduling
> >> >> needs to be done inside the messenger?
> >> >
> >> > Agreed! we could remove heartbeat messenger firstly!
> >> >
> >> >>
> >> >> sage
> >> >>
> >> >>> The advantage to register callback is we can combine multi layers oob
> >> >>> messages to one.
> >> >>>
> >> >>> >
> >> >>> > An easier first step would be to just define a
> >> >>> > Connection::send_message_oob(Message*).  That would require almost
> >> >>> > no
> >> >>> > changes to the calling code, and avoid having to create the timing
> >> >>> > infrastructure inside AsyncMessenger...
> >> >>> >
> >> >>> > sage
> >> >>> >
> >> >>> >> void Dispatcher::ms_dispatch_oob(Message*)
> >> >>> >>
> >> >>> >> handle the oob message with parsing each oob part.
> >> >>> >>
> >> >>> >> So lots of timer control in user's side could be avoided via
> >> >>> >> callback
> >> >>> >> generator. When sending, OOB message could insert the front of send
> >> >>> >> message queue but we can't get any help from kernel oob flag since
> >> >>> >> it's really useless..
> >> >>> >>
> >> >>> >> Any suggestion is welcomed!
> >>
> >> Let's keep in mind the challenges of out-of-band messaging over TCP/IP.
> >>
> >> Namely, when we discussed this we couldn't figure out any way
> >> (including the TCP priority stuff, which doesn't work with the
> >> required semantics — even when it does function) to get traffic to
> >> actually go out-of-band. IB messaging systems actually have a
> >> "channels" concept that lets you do genuine OOB transmission that
> >> skips over queues and other data; TCP doesn't. In fact the best we
> >> came up with for doing this with Simple/AsyncMessenger was giving the
> >> Messenger duplicate sockets/queues/etc, which is hardly ideal.
> >>
> >> So, maybe we can remove the heartbeat messenger by giving each
> >> Connection two sockets and queues. That might even work better for the
> >> AsyncMessenger than it does for SimpleMessenger?
> >> But any implementation that orders OSD heartbeat messages behind
> >> ordinary data traffic in kernel or router buffers is probably going to
> >> fail us. :(
> >
> > Oh, good point.  I didn't read that paragraph carefully.  I think we
> > should use a second socket connected to the same address for OOB messages.
> > Or possibly push them over UDP... but we'd need to define retry semantics
> > in that case.
> 
> if udp, I think udp hb interval should be less, caller should delegate
> send logic to connection...
> 
> >
> > sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309