Re: crc error when decode_message?

From: Sage Weil <sage@newdream.net>
To: Ning Yao <zay11022@gmail.com>
Cc: Haomai Wang <haomaiwang@gmail.com>, Xinze Chi <xmdxcxz@gmail.com>,
	"ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: crc error when decode_message?
Date: Tue, 17 Mar 2015 06:46:13 -0700 (PDT)	[thread overview]
Message-ID: <alpine.DEB.2.00.1503170643410.7043@cobra.newdream.net> (raw)
In-Reply-To: <CALZt5jyiiM2NcurGLSVYYkNn+bPuwhWX8+zb0bBndKQ92p2KNg@mail.gmail.com>

On Tue, 17 Mar 2015, Ning Yao wrote:
> 2015-03-16 22:06 GMT+08:00 Haomai Wang <haomaiwang@gmail.com>:
> > On Mon, Mar 16, 2015 at 10:04 PM, Xinze Chi <xmdxcxz@gmail.com> wrote:
> >> How to process the write request in primary?
> >>
> >> Thanks.
> >>
> >> 2015-03-16 22:01 GMT+08:00 Haomai Wang <haomaiwang@gmail.com>:
> >>> AFAR Pipe and AsyncConnection both will mark self fault and shutdown
> >>> socket and peer will detect this reset. So each side has chance to
> >>> rebuild the session.
> >>>
> >>> On Mon, Mar 16, 2015 at 9:19 PM, Xinze Chi <xmdxcxz@gmail.com> wrote:
> >>>> Such as, Client send write request to osd.0 (primary), osd.0 send
> >>>> MOSDSubOp to osd.1 and osd.2
> >>>>
> >>>> osd.1 send reply to osd.0 (primary), but accident happened:
> >>>>
> >>>> 1. decode_message crc error when decode reply msg
> >>>> or
> >>>> 2. the reply msg is lost when send to osd.0, so osd.0 do not receive replay msg
> >>>>
> >>>> Could anyone tell me what is the behavior if osd.0 (primary)?
> >>>>
> >
> > osd.0 and osd.1 both will try to reconnect peer side, and the lost
> > message will be resend to osd.0 from osd.1
> So I wonder if different routing path delays the arrival of one
> message, then the in_seq would be setting ahead, then based on the
> logic. Later, if the delaying message arrives, it will be dropping and
> discard. Thus, if it is just a sub_op reply message as xinze
> describes, how ceph works after that? It seems repop of the write Op
> will be waiting infinite times until the osd restart?

These sorts of scenarios are why src/msg/simple/Pipe.cc (an in particular, 
accept()) is not so simple.  The case you describe is 

 https://github.com/ceph/ceph/blob/master/src/msg/simple/Pipe.cc#L492
or
 https://github.com/ceph/ceph/blob/master/src/msg/simple/Pipe.cc#L492

In other words, this is all masked by the Messenger layer so that the 
higher layers (OSD.cc etc) see a single, ordered, reliable stream of 
messages and all of the failure/retry/reconnect logic is hidden.

sage