Re: nvme deadlock with ANA

[Sagi Grimberg <sagi@grimberg.me>]

>>>> I want to consult with you guys on a deadlock condition I'm able to
>>>> hit with a test that incorporate controller reconnect, ana updates
>>>> and live I/O with timeouts.
>>>>
>>>> This is true for NVMe/TCP, but can also happen in rdma or pci 
>>>> drivers as
>>>> well.
>>>>
>>>> The deadlock combines 4 flows in parallel:
>>>> - ns scanning (triggered from reconnect)
>>>> - request timeout
>>>> - ANA update (triggered from reconnect)
>>>> - FS I/O coming into the mpath device
>>>>
>>>> (1) ns scanning triggers disk revalidation -> update disk info ->
>>>>      freeze queue -> but blocked, why?
>>>
>>> What does -> but blocked mean?
>>
>> It is blocked and cannot complete, because of (2)
>>
>>>> (2) timeout handler reference the g_usage_counter - > but blocks in
>>>>      the timeout handler, why?
>>>
>>> The timeout handler obviously needs to keep the queue alive while
>>> running.  We could think of doing a try_get, though?
>>
>> It is keeping the queue alive, that is not the issue. it is blocked in
>> the driver .timeout() handler (i.e. nvme_tcp_timeout).
>>
>> The reason that it blocked and cannot make forward progress is because
>> the driver timeout handler will call nvme_stop_queues(), which is
>> blocked as this takes namespaces_rwsem...
>>
>> There is a chain of dependency that is deadlocking with circular
>> dependency.
> 
> Can't you simply call 'nvme_reset_ctrl()' ?
> Seems to work reasonably well on the fc side, so I wonder what's 
> different for tcp ...

As I mentioned, this is not specific for tcp, it pci timeout handler
can block just as well because it calls nvme_dev_disable which calls
nvme_stop_queues. The rest of the flows are not related to the
transport.

There is no expectation from the driver to always defer handling the
timeout to a different context, but should we make that operation for
nvme transprots at least?

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme