Re: [EXT] Re: [PATCH] qedi: Fix cmd_cleanup_cmpl counter mismatch issue.

[michael.christie@oracle.com]

On 11/24/21 12:05 AM, Manish Rangankar wrote:
>>>
>>>  check_cleanup_reqs:
>>>  	if (qedi_conn->cmd_cleanup_req > 0) {
>>> -		QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_TID,
>>> -			  "Freeing tid=0x%x for cid=0x%x\n",
>>> -			  cqe->itid, qedi_conn->iscsi_conn_id);
>>> -		qedi_conn->cmd_cleanup_cmpl++;
>>> +		++qedi_conn->cmd_cleanup_cmpl;
>>> +		QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_SCSI_TM,
>>> +			  "Freeing tid=0x%x for cid=0x%x cleanup count=%d\n",
>>> +			  cqe->itid, qedi_conn->iscsi_conn_id,
>>> +			  qedi_conn->cmd_cleanup_cmpl);
>>
>> Is the issue that cmd_cleanup_cmpl's increment is not seen by
>> qedi_cleanup_all_io's wait_event_interruptible_timeout call when it wakes up,
>> and your patch fixes this by doing a pre increment?
>>
> 
> Yes, cmd_cleanup_cmpl's increment is not seen by qedi_cleanup_all_io's 
> wait_event_interruptible_timeout call when it wakes up, even after firmware 
> post all the ISCSI_CQE_TYPE_TASK_CLEANUP events for requested cmd_cleanup_req.
> Yes, pre increment did addressed this issue. Do you feel otherwise ?
> 
>> Does doing a pre increment give you barrier like behavior and is that why this
>> works? I thought if wake_up ends up waking up the other thread it does a barrier
>> already, so it's not clear to me how changing to a pre-increment helps.
>>
>> Is doing a pre-increment a common way to handle this? It looks like we do a
>> post increment and wake_up* in other places. However, like in the scsi layer we
>> do wake_up_process and memory-barriers.txt says that always does a general
>> barrier, so is that why we can do a post increment there?
>>
>> Does pre-increment give you barrier like behavior, and is the wake_up call not
>> waking up the process so we didn't get a barrier from that, and so that's why this
>> works?
>>
> 
> Issue happen before calling wake_up. When we gets a ISCSI_CQE_TYPE_TASK_CLEANUP surge on
> multiple Rx threads, cmd_cleanup_cmpl tend to miss the increment. The scenario is more similar to
> multiple threads access cmd_cleanup_cmpl causing race during postfix increment. This could be because of 
> thread reading the same value at a time.
> 
> Now that I am explaining it, it felt instead of pre-incrementing cmd_cleanup_cmpl, 
> it should be atomic variable. Do see any issue ? 
> 

Yeah, atomic.

And then I guess for this:

        if (qedi_conn->cmd_cleanup_req > 0) {
                QEDI_INFO(&qedi->dbg_ctx, QEDI_LOG_TID,
                          "Freeing tid=0x%x for cid=0x%x\n",
                          cqe->itid, qedi_conn->iscsi_conn_id);
                qedi_conn->cmd_cleanup_cmpl++;
                wake_up(&qedi_conn->wait_queue);


we might only want to do the wake_up once:

if (atomic_inc_return(&qedi_conn->cmd_cleanup_cmpl) ==
    qedi_conn->cmd_cleanup_req) {

?