Re: nvme deadlock with ANA

[Sagi Grimberg <sagi@grimberg.me>]

>> Hey,
>>
>> I want to consult with you guys on a deadlock condition I'm able to
>> hit with a test that incorporate controller reconnect, ana updates
>> and live I/O with timeouts.
>>
>> This is true for NVMe/TCP, but can also happen in rdma or pci drivers as
>> well.
>>
>> The deadlock combines 4 flows in parallel:
>> - ns scanning (triggered from reconnect)
>> - request timeout
>> - ANA update (triggered from reconnect)
>> - FS I/O coming into the mpath device
>>
>> (1) ns scanning triggers disk revalidation -> update disk info ->
>>      freeze queue -> but blocked, why?
>>
>> (2) timeout handler reference the g_usage_counter - > but blocks in
>>      the timeout handler, why?
>>
>> (3) the timeout handler (indirectly) calls nvme_stop_queue() -> which
>>      takes the namespaces_rwsem - > but blocks, why?
>>
>> (4) ANA update takes the namespaces_rwsem -> calls nvme_mpath_set_live()
>>      -> which synchronize the ns_head srcu (see commit 504db087aacc) ->
>>      but it blocks, why?
> 
> 
> Why is ana updating occuring under a write lock? AFAIK, that should be
> necessary only if it's manipulating the ctrl->namespaces list, but this
> is just iterating it. What's the harm in using the read lock here?

Yes you are absolutely correct!

I think this should resolve the deadlock, as nvme_stop_queues can make
forward progress, and then the transport timeout handler will complete,
the timeout handler can decrement the g_usage_counter, and the queue
freeze in update_disk_info will complete.

I'll send a patch.

Thanks!

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme