[Ocfs2-devel] [PATCH] ocfs2: re-queue AST or BAST if sending is failed to improve the reliability

From: Junxiao Bi <junxiao.bi@oracle.com>
To: ocfs2-devel@oss.oracle.com
Subject: [Ocfs2-devel] [PATCH] ocfs2: re-queue AST or BAST if sending is failed to improve the reliability
Date: Wed, 23 Aug 2017 10:23:20 +0800	[thread overview]
Message-ID: <b2d93241-cf33-9a31-224d-5cfc983f7ff9@oracle.com> (raw)
In-Reply-To: <63ADC13FD55D6546B7DECE290D39E373AC2CBD22@H3CMLB14-EX.srv.huawei-3com.com>

On 08/10/2017 06:49 PM, Changwei Ge wrote:
> Hi Joseph,
> 
> 
> On 2017/8/10 17:53, Joseph Qi wrote:
>> Hi Changwei,
>>
>> On 17/8/9 23:24, ge changwei wrote:
>>> Hi
>>>
>>>
>>> On 2017/8/9 ??7:32, Joseph Qi wrote:
>>>> Hi,
>>>>
>>>> On 17/8/7 15:13, Changwei Ge wrote:
>>>>> Hi,
>>>>>
>>>>> In current code, while flushing AST, we don't handle an exception that
>>>>> sending AST or BAST is failed.
>>>>> But it is indeed possible that AST or BAST is lost due to some kind of
>>>>> networks fault.
>>>>>
>>>> Could you please describe this issue more clearly? It is better analyze
>>>> issue along with the error message and the status of related nodes.
>>>> IMO, if network is down, one of the two nodes will be fenced. So what's
>>>> your case here?
>>>>
>>>> Thanks,
>>>> Joseph
>>> I have posted the status of related lock resource in my preceding email. 
>>> Please check them out.
>>>
>>> Moreover, network is not down forever even not longer than threshold  to 
>>> be fenced.
>>> So no node will be fenced.
>>>
>>> This issue happens in terrible network environment. Some messages may be 
>>> abandoned by switch due to various conditions.
>>> And even frequent and fast link up and down will also cause this issue.
>>>
>>> In a nutshell,  re-queuing AST and BAST is crucial when link between 
>>> nodes recover quickly. It prevents cluster from hanging.
>>> So you mean the tcp packet is lost due to connection reset? IIRC,
> Yes, it's something like that exception which I think is deserved to be
> fixed within OCFS2.
>> Junxiao has posted a patchset to fix this issue.
>> If you are using the way of re-queuing, how to make sure the original
>> message is *truly* lost and the same ast/bast won't be sent twice?
> With regards to TCP layer, if it returns error to OCFS2, packets must
> not be sent successfully. So no node will obtain such an AST or BAST.
Right, but not only AST/BAST, other messages pending in tcp queue will
also lost if tcp return error to ocfs2, this can also caused hung.
Besides, your fix may introduce duplicated ast/bast message Joseph
mentioned.
Ocfs2 depends tcp a lot, it can't work well if tcp return error to it.
To fix it, maybe ocfs2 should maintain its own message queue and ack
messages while not depend on TCP.

Thanks,
Junxiao.

> With regards to OCFS2, my patch can guarantee that one AST/BAST can't be
> queued on pending list twice of which are both sent successfully.
> 
> Thanks,
> Changwei
>>
>> Thanks,
>> Joseph
>>  
>>> Thanks,
>>> Changwei
>>>>> If above exception happens, the requesting node will never obtain an AST
>>>>> back, hence, it will never acquire the lock or abort current locking.
>>>>>
>>>>> With this patch, I'd like to fix this issue by re-queuing the AST or
>>>>> BAST if sending is failed due to networks fault.
>>>>>
>>>>> And the re-queuing AST or BAST will be dropped if the requesting node is
>>>>> dead!
>>>>>
>>>>> It will improve the reliability a lot.
>>>>>
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Changwei.
>>>> _______________________________________________
>>>> Ocfs2-devel mailing list
>>>> Ocfs2-devel at oss.oracle.com
>>>> https://oss.oracle.com/mailman/listinfo/ocfs2-devel
> 