blk_mq_reinit_tagset during NVMEoF port toggling

blk_mq_reinit_tagset during NVMEoF port toggling

[Sagi Grimberg]

> Hi guys,

Hi Max, CCing linux-nvme.

> we have encountered a bug during our port toggling test with MP using 
> NVMEoF over RDMA (1 IO queue repro it quickly).
> We have been receiving local protection errors dumps after failing back 
> to the port that became active again (it's not the retransmission issue 
> we fixed in the past). After debugging it we saw that the requests have 
> been doing a reinit process (dereg_mr/alloc_mr).
> But somehow the req->mr->need_inval is still true in the beginning of 
> nvme_rdma_queue_rq function. This shouldn't happen since we should have 
> perform the dereg_mr/alloc_mr in the reinit func and set it to false.
> We don't see this issue in kernel older than 4.11 so before bisecting:

Which code base is this max?

is commit 842594c8775b585c58459e044708c0335b6aa6b7 applied?

if so, maybe it is possible that not all requests are being reinitialized.
Can you reproduce with the following applied:
--
diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index d0be72ccb091..420ef106057e 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -302,8 +302,10 @@ int blk_mq_reinit_tagset(struct blk_mq_tag_set *set)
                         continue;

                 for (j = 0; j < tags->nr_tags; j++) {
-                       if (!tags->static_rqs[j])
+                       if (!tags->static_rqs[j]) {
+                               pr_info("passing rq %d\n", j);
                                 continue;
+                       }

                         ret = set->ops->reinit_request(set->driver_data,
                                                 tags->static_rqs[j]);
--

blk_mq_reinit_tagset during NVMEoF port toggling

[Israel Rukshin]

On 8/28/2017 10:40 AM, Sagi Grimberg wrote:
>> Hi guys,
>
> Hi Max, CCing linux-nvme.
Hi Sagi,
>
>> we have encountered a bug during our port toggling test with MP using 
>> NVMEoF over RDMA (1 IO queue repro it quickly).
>> We have been receiving local protection errors dumps after failing 
>> back to the port that became active again (it's not the 
>> retransmission issue we fixed in the past). After debugging it we saw 
>> that the requests have been doing a reinit process (dereg_mr/alloc_mr).
>> But somehow the req->mr->need_inval is still true in the beginning of 
>> nvme_rdma_queue_rq function. This shouldn't happen since we should 
>> have perform the dereg_mr/alloc_mr in the reinit func and set it to 
>> false.
>> We don't see this issue in kernel older than 4.11 so before bisecting:
>
> Which code base is this max?
The code base is kernel 4.13.0-rc3.
>
> is commit 842594c8775b585c58459e044708c0335b6aa6b7 applied?
Yes.
>
> if so, maybe it is possible that not all requests are being 
> reinitialized.
> Can you reproduce with the following applied:
We reproduced this issue with similar prints and we didn't see them.
blk_mq_reinit_tagset() went over all the the static requests.

Thanks,
Israel.

blk_mq_reinit_tagset during NVMEoF port toggling

[Sagi Grimberg]

>>> Hi guys,
>>
>> Hi Max, CCing linux-nvme.
> Hi Sagi,
>>
>>> we have encountered a bug during our port toggling test with MP using 
>>> NVMEoF over RDMA (1 IO queue repro it quickly).
>>> We have been receiving local protection errors dumps after failing 
>>> back to the port that became active again (it's not the 
>>> retransmission issue we fixed in the past). After debugging it we saw 
>>> that the requests have been doing a reinit process (dereg_mr/alloc_mr).
>>> But somehow the req->mr->need_inval is still true in the beginning of 
>>> nvme_rdma_queue_rq function. This shouldn't happen since we should 
>>> have perform the dereg_mr/alloc_mr in the reinit func and set it to 
>>> false.
>>> We don't see this issue in kernel older than 4.11 so before bisecting:
>>
>> Which code base is this max?
> The code base is kernel 4.13.0-rc3.
>>
>> is commit 842594c8775b585c58459e044708c0335b6aa6b7 applied?
> Yes.
>>
>> if so, maybe it is possible that not all requests are being 
>> reinitialized.
>> Can you reproduce with the following applied:
> We reproduced this issue with similar prints and we didn't see them.
> blk_mq_reinit_tagset() went over all the the static requests.

If mr->need_inval is true in queue_rq it means that reinit was not
called on it, did you see a request that performed reinit but
still had need_inval == true?

blk_mq_reinit_tagset during NVMEoF port toggling

[Max Gurtovoy]

On 8/28/2017 4:30 PM, Sagi Grimberg wrote:
> 
>>>> Hi guys,
>>>
>>> Hi Max, CCing linux-nvme.
>> Hi Sagi,
>>>
>>>> we have encountered a bug during our port toggling test with MP 
>>>> using NVMEoF over RDMA (1 IO queue repro it quickly).
>>>> We have been receiving local protection errors dumps after failing 
>>>> back to the port that became active again (it's not the 
>>>> retransmission issue we fixed in the past). After debugging it we 
>>>> saw that the requests have been doing a reinit process 
>>>> (dereg_mr/alloc_mr).
>>>> But somehow the req->mr->need_inval is still true in the beginning 
>>>> of nvme_rdma_queue_rq function. This shouldn't happen since we 
>>>> should have perform the dereg_mr/alloc_mr in the reinit func and set 
>>>> it to false.
>>>> We don't see this issue in kernel older than 4.11 so before bisecting:
>>>
>>> Which code base is this max?
>> The code base is kernel 4.13.0-rc3.
>>>
>>> is commit 842594c8775b585c58459e044708c0335b6aa6b7 applied?
>> Yes.
>>>
>>> if so, maybe it is possible that not all requests are being 
>>> reinitialized.
>>> Can you reproduce with the following applied:
>> We reproduced this issue with similar prints and we didn't see them.
>> blk_mq_reinit_tagset() went over all the the static requests.
> 
> If mr->need_inval is true in queue_rq it means that reinit was not
> called on it, did you see a request that performed reinit but
> still had need_inval == true?

No. The requests that are "reinited" are the static_rqs.
 From what I saw in the code, the MP has a cloned request queue that it 
uses but those requests are not "reinited" (only the original requests).