From mboxrd@z Thu Jan 1 00:00:00 1970 From: Haomai Wang Subject: Re: OOB message roll into Messenger interface Date: Tue, 6 Sep 2016 22:15:24 +0800 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Return-path: Received: from mail-pu1apc01on0107.outbound.protection.outlook.com ([104.47.126.107]:54965 "EHLO APC01-PU1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1756034AbcIFOPl (ORCPT ); Tue, 6 Sep 2016 10:15:41 -0400 Received: by mail-vk0-f46.google.com with SMTP id v189so101131404vkv.1 for ; Tue, 06 Sep 2016 07:15:34 -0700 (PDT) In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Sage Weil Cc: "ceph-devel@vger.kernel.org" On Tue, Sep 6, 2016 at 10:06 PM, Sage Weil wrote: > On Tue, 6 Sep 2016, Haomai Wang wrote: >> On Tue, Sep 6, 2016 at 9:17 PM, Sage Weil wrote: >> > Hi Haomai! >> > >> > On Sun, 4 Sep 2016, Haomai Wang wrote: >> >> Background: >> >> Each osd has two heartbeat messenger instances to maintain front/back >> >> network available. It brings lots of connections and messages overhead >> >> in scale out cluster. Actually we can combine these heartbeat >> >> exchanges to public/cluster messengers to reduce tons of >> >> connections(resources). >> >> >> >> Then heartbeat message should be OOB and shared the same thread/socket >> >> with normal message channel. So it can exactly represent the heartbeat >> >> role for real IO message. Otherwise, heartbeat channel's status can't >> >> indicate the real IO message channel status. Because different socket >> >> uses different send buffer/recv buffer, if real io message blocked, >> >> oob message may be healthy. >> >> >> >> Besides OSD's heartbeat things, we have logic PING/PONG lived in >> >> Objecter Ping/WatchNotify Ping etc. For the same goal, they could >> >> share the heartbeat message. >> >> >> >> In a real rbd use case env, if we combines these ping/pong messages, >> >> thousands of messages could be avoided which means lots of resources. >> >> >> >> As we reduce the heartbeat overhead, we can reduce heartbeat interval >> >> and increase frequency which help a lot to the accurate of cluster >> >> failure detection! >> > >> > I'm very excited to see this move forward! >> > >> >> Design: >> >> >> >> As discussed in Raleigh, we could defines these interfaces: >> >> >> >> int Connection::register_oob_message(identitfy_op, callback, interval); >> >> >> >> Users like Objecter linger ping could register a "callback" which >> >> generate bufferlist used to be carried by heartbeat message. >> >> "interval" indicate the user's oob message's send interval. >> >> >> >> "identitfy_op" indicates who can handle the oob info in peer side. >> >> Like "Ping", "OSDPing" or "LingerPing" as the current message define. >> > >> > This looks convenient for the simpler callers, but I worry it won't work >> > as well for OSDPing. There's a bunch of odd locking around the heartbeat >> > info and the code already exists to do the the heartbeat sends. I'm not >> > sure it will simplify to a simple interval. >> >> Hmm, I'm not sure what's the odd locking thing refer to. As we can >> register callback when adding new peer and unregister callback when >> removing peer from "heartbeat_peers". >> >> The main send message construct callback extract from this loop: >> for (map::iterator i = heartbeat_peers.begin(); >> i != heartbeat_peers.end(); >> ++i) { >> int peer = i->first; >> i->second.last_tx = now; >> if (i->second.first_tx == utime_t()) >> i->second.first_tx = now; >> dout(30) << "heartbeat sending ping to osd." << peer << dendl; >> i->second.con_back->send_message(new MOSDPing(monc->get_fsid(), >> service.get_osdmap()->get_epoch(), >> MOSDPing::PING, >> now)); >> >> if (i->second.con_front) >> i->second.con_front->send_message(new MOSDPing(monc->get_fsid(), >> service.get_osdmap()->get_epoch(), >> MOSDPing::PING, >> now)); >> } >> >> Only "fsid", "osdmap epoch" are required, I don't think it will block. >> Then I think lots of locking/odding things exists on heartbeat >> dispatch/handle process. sending process is clear I guess. > > Yeah, I guess that's fine. I was worried about some dependency between > who we ping and the osdmap epoch in the message (and races adding/removing > heartbeat peers), but I think it doesn't matter. > > Even so, I think it would be good to expose the send_message_oob() > interface, and do this in 2 stages so the two changes are decoupled. > Unless there is some implementation reason why the oob message scheduling > needs to be done inside the messenger? Agreed! we could remove heartbeat messenger firstly! > > sage > >> The advantage to register callback is we can combine multi layers oob >> messages to one. >> >> > >> > An easier first step would be to just define a >> > Connection::send_message_oob(Message*). That would require almost no >> > changes to the calling code, and avoid having to create the timing >> > infrastructure inside AsyncMessenger... >> > >> > sage >> > >> >> void Dispatcher::ms_dispatch_oob(Message*) >> >> >> >> handle the oob message with parsing each oob part. >> >> >> >> So lots of timer control in user's side could be avoided via callback >> >> generator. When sending, OOB message could insert the front of send >> >> message queue but we can't get any help from kernel oob flag since >> >> it's really useless.. >> >> >> >> Any suggestion is welcomed! >> >> -- >> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> >> the body of a message to majordomo@vger.kernel.org >> >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> >> >> >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >>