Re: [PATCH for-next] RDMA/hns: Bugfix for flush cqe in case softirq and multi-process

["Liuyixian (Eason)" <liuyixian@huawei.com>]

On 2019/9/23 13:01, Leon Romanovsky wrote:
> On Fri, Sep 20, 2019 at 11:55:56AM +0800, Liuyixian (Eason) wrote:
>>
>>
>> On 2019/9/11 21:17, Liuyixian (Eason) wrote:
>>>
>>>
>>> On 2019/9/10 15:52, Leon Romanovsky wrote:
>>>> On Tue, Sep 10, 2019 at 02:40:20PM +0800, Liuyixian (Eason) wrote:
>>>>>
>>>>>
>>>>> On 2019/9/8 16:03, Leon Romanovsky wrote:
>>>>>> On Thu, Sep 05, 2019 at 08:31:11PM +0800, Weihang Li wrote:
>>>>>>> From: Yixian Liu <liuyixian@huawei.com>
>>>>>>>
>>>>>>> Hip08 has the feature flush cqe, which help to flush wqe in workqueue
>>>>>>> (sq and rq) when error happened by transmitting producer index with
>>>>>>> mailbox to hardware. Flush cqe is emplemented in post send and recv
>>>>>>> verbs. However, under NVMe cases, these verbs will be called under
>>>>>>> softirq context, and it will lead to following calltrace with
>>>>>>> current driver as mailbox used by flush cqe can go to sleep.
>>>>>>>
>>>>>>> This patch solves this problem by using workqueue to do flush cqe,
>>>>>>
>>>>>> Unbelievable, almost every bug in this driver is solved by introducing
>>>>>> workqueue. You should fix "sleep in flush path" issue and not by adding
>>>>>> new workqueue.
>>>>>>
>>>>> Hi Leon,
>>>>>
>>>>> Thanks for the comment.
>>>>> Up to now, for hip08, only one place use workqueue in hns_roce_hw_v2.c
>>>>> where for irq prints.
>>>>
>>>> Thanks to our lack of desire to add more workqueues and previous patches
>>>> which removed extra workqueues from the driver.
>>>>
>>> Thanks, I see.
>>>
>>>>>
>>>>> The solution for flush cqe in this patch is as follow:
>>>>> While flush cqe should be implement, the driver should modify qp to error state
>>>>> through mailbox with the newest product index of sq and rq, the hardware then
>>>>> can flush all outstanding wqes in sq and rq.
>>>>>
>>>>> That's the whole mechanism of flush cqe, also is the flush path. We can't
>>>>> change neither mailbox sleep attribute or flush cqe occurred in post send/recv.
>>>>> To avoid the calltrace of flush cqe in post verbs under NVMe softirq,
>>>>> use workqueue for flush cqe seems reasonable.
>>>>>
>>>>> As far as I know, there is no other alternative solution for this situation.
>>>>> I will be very grateful if you reminder me more information.
>>>>
>>>> ib_drain_rq/ib_drain_sq/ib_drain_qp????
>>>>
>>> Hi Leon,
>>>
>>> I think these interfaces are designed for application to check that all wqes
>>> have been processed by hardware, so called drain or flush. However, it is not
>>> the same as the flush in this patch. The solution in this patch is used
>>> to help the hardware generate flush cqes for outstanding wqes while qp error.
>>>
>> Hi Leon,
>>
>> What's your opinion about above? Do you have any further comments?
> 
> My opinion didn't change, you need to read discussions about ib_drain_*()
> functions, how and why they were introduced. It is a way to go.
> 
> Thanks

Hi Leon,

Thanks a lot! I will dig those functions for my problem.

> 
>>
>> Thanks.
>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>> _______________________________________________
>>>>>> Linuxarm mailing list
>>>>>> Linuxarm@huawei.com
>>>>>> http://hulk.huawei.com/mailman/listinfo/linuxarm
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> .
>>>>
>>>
>>> _______________________________________________
>>> Linuxarm mailing list
>>> Linuxarm@huawei.com
>>> http://hulk.huawei.com/mailman/listinfo/linuxarm
>>>
>>> .
>>>
>>
> 
> .
>