From: Hannes Reinecke <hare@suse.de>
To: Chao Leng <lengchao@huawei.com>, Sagi Grimberg <sagi@grimberg.me>,
Daniel Wagner <dwagner@suse.de>
Cc: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org,
Jens Axboe <axboe@fb.com>, Keith Busch <kbusch@kernel.org>,
Christoph Hellwig <hch@lst.de>
Subject: Re: [PATCH v2] nvme-multipath: Early exit if no path is available
Date: Mon, 1 Feb 2021 08:29:57 +0100 [thread overview]
Message-ID: <114751ac-1f7d-ce5e-12c5-7d6303bdb999@suse.de> (raw)
In-Reply-To: <6ceff3cb-c9e9-7e74-92f0-dd745987c943@huawei.com>
On 2/1/21 3:16 AM, Chao Leng wrote:
>
>
> On 2021/1/29 17:20, Hannes Reinecke wrote:
>> On 1/29/21 9:46 AM, Chao Leng wrote:
>>>
>>>
>>> On 2021/1/29 16:33, Hannes Reinecke wrote:
>>>> On 1/29/21 8:45 AM, Chao Leng wrote:
>>>>>
>>>>>
>>>>> On 2021/1/29 15:06, Hannes Reinecke wrote:
>>>>>> On 1/29/21 4:07 AM, Chao Leng wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2021/1/29 9:42, Sagi Grimberg wrote:
>>>>>>>>
>>>>>>>>>> You can't see exactly where it dies but I followed the
>>>>>>>>>> assembly to
>>>>>>>>>> nvme_round_robin_path(). Maybe it's not the initial
>>>>>>>>>> nvme_next_ns(head,
>>>>>>>>>> old) which returns NULL but nvme_next_ns() is returning NULL
>>>>>>>>>> eventually
>>>>>>>>>> (list_next_or_null_rcu()).
>>>>>>>>> So there is other bug cause nvme_next_ns abormal.
>>>>>>>>> I review the code about head->list and head->current_path, I
>>>>>>>>> find 2 bugs
>>>>>>>>> may cause the bug:
>>>>>>>>> First, I already send the patch. see:
>>>>>>>>> https://lore.kernel.org/linux-nvme/20210128033351.22116-1-lengchao@huawei.com/
>>>>>>>>>
>>>>>>>>> Second, in nvme_ns_remove, list_del_rcu is before
>>>>>>>>> nvme_mpath_clear_current_path. This may cause "old" is deleted
>>>>>>>>> from the
>>>>>>>>> "head", but still use "old". I'm not sure there's any other
>>>>>>>>> consideration here, I will check it and try to fix it.
>>>>>>>>
>>>>>>>> The reason why we first remove from head->list and only then clear
>>>>>>>> current_path is because the other way around there is no way
>>>>>>>> to guarantee that that the ns won't be assigned as current_path
>>>>>>>> again (because it is in head->list).
>>>>>>> ok, I see.
>>>>>>>>
>>>>>>>> nvme_ns_remove fences continue of deletion of the ns by
>>>>>>>> synchronizing
>>>>>>>> the srcu such that for sure the current_path clearance is visible.
>>>>>>> The list will be like this:
>>>>>>> head->next = ns1;
>>>>>>> ns1->next = head;
>>>>>>> old->next = ns1;
>>>>>>
>>>>>> Where does 'old' pointing to?
>>>>>>
>>>>>>> This may cause infinite loop in nvme_round_robin_path.
>>>>>>> for (ns = nvme_next_ns(head, old);
>>>>>>> ns != old;
>>>>>>> ns = nvme_next_ns(head, ns))
>>>>>>> The ns will always be ns1, and then infinite loop.
>>>>>>
>>>>>> No. nvme_next_ns() will return NULL.
>>>>> If there is just one path(the "old") and the "old" is deleted,
>>>>> nvme_next_ns() will return NULL.
>>>>> The list like this:
>>>>> head->next = head;
>>>>> old->next = head;
>>>>> If there is two or more path and the "old" is deleted,
>>>>> "for" will be infinite loop. because nvme_next_ns() will return
>>>>> the path which in the list except the "old", check condition will
>>>>> be true for ever.
>>>>
>>>> But that will be caught by the statement above:
>>>>
>>>> if (list_is_singular(&head->list))
>>>>
>>>> no?
>>> Two path just a sample example.
>>> If there is just two path, will enter it, may cause no path but there is
>>> actually one path. It is falsely assumed that the "old" must be not
>>> deleted.
>>> If there is more than two path, will cause infinite loop.
>> So you mean we'll need something like this?
>>
>> diff --git a/drivers/nvme/host/multipath.c
>> b/drivers/nvme/host/multipath.c
>> index 71696819c228..8ffccaf9c19a 100644
>> --- a/drivers/nvme/host/multipath.c
>> +++ b/drivers/nvme/host/multipath.c
>> @@ -202,10 +202,12 @@ static struct nvme_ns *__nvme_find_path(struct
>> nvme_ns_head *head, int node)
>> static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head,
>> struct nvme_ns *ns)
>> {
>> - ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct
>> nvme_ns,
>> - siblings);
>> - if (ns)
>> - return ns;
>> + if (ns) {
>> + ns = list_next_or_null_rcu(&head->list, &ns->siblings,
>> + struct nvme_ns, siblings);
>> + if (ns)
>> + return ns;
>> + }
> No, in the scenario, ns should not be NULL.
Why not? 'ns == NULL' is precisely the corner-case this is trying to fix...
> May be we can do like this:
>
> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
> index 282b7a4ea9a9..b895011a2cbd 100644
> --- a/drivers/nvme/host/multipath.c
> +++ b/drivers/nvme/host/multipath.c
> @@ -199,30 +199,24 @@ static struct nvme_ns *__nvme_find_path(struct
> nvme_ns_head *head, int node)
> return found;
> }
>
> -static struct nvme_ns *nvme_next_ns(struct nvme_ns_head *head,
> - struct nvme_ns *ns)
> -{
> - ns = list_next_or_null_rcu(&head->list, &ns->siblings, struct
> nvme_ns,
> - siblings);
> - if (ns)
> - return ns;
> - return list_first_or_null_rcu(&head->list, struct nvme_ns,
> siblings);
> -}
> +#define nvme_next_ns_condition(head, current, condition) \
> +({ \
> + struct nvme_ns *__ptr = list_next_or_null_rcu(&(head)->list, \
> + &(current)->siblings, struct nvme_ns, siblings); \
> + __ptr ? __ptr : (condition) ? (condition) = false, \
> + list_first_or_null_rcu(&(head)->list, struct nvme_ns, \
> + siblings) : NULL; \
> +})
>
Urgh. Please, no. That is well impossible to debug.
Can you please open-code it to demonstrate where the difference to the
current (and my fixed) versions is?
I'm still not clear where the problem is once we applied both patches.
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
next prev parent reply other threads:[~2021-02-01 7:30 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-27 10:30 [PATCH v2] nvme-multipath: Early exit if no path is available Daniel Wagner
2021-01-27 10:34 ` Hannes Reinecke
2021-01-27 16:49 ` Christoph Hellwig
2021-01-28 1:31 ` Chao Leng
2021-01-28 7:58 ` Daniel Wagner
2021-01-28 9:18 ` Chao Leng
2021-01-28 9:23 ` Hannes Reinecke
2021-01-29 1:18 ` Chao Leng
2021-01-28 9:40 ` Daniel Wagner
2021-01-29 1:23 ` Chao Leng
2021-01-29 1:42 ` Sagi Grimberg
2021-01-29 3:07 ` Chao Leng
2021-01-29 3:30 ` Sagi Grimberg
2021-01-29 3:36 ` Chao Leng
2021-01-29 7:06 ` Hannes Reinecke
[not found] ` <81b22bbf-4dd3-6161-e63a-9699690a4e4f@huawei.com>
[not found] ` <715dd943-0587-be08-2840-e0948cf0bc62@suse.de>
[not found] ` <eb131d8f-f009-42e7-105d-58b84060f0dd@huawei.com>
[not found] ` <ac019690-7f02-d28c-ed58-bfc8c1d48879@suse.de>
2021-02-01 2:16 ` Chao Leng
2021-02-01 7:29 ` Hannes Reinecke [this message]
2021-02-01 8:47 ` Chao Leng
2021-02-01 8:57 ` Hannes Reinecke
2021-02-01 9:40 ` Chao Leng
2021-02-01 10:45 ` Hannes Reinecke
2021-02-02 1:12 ` Chao Leng
2021-01-28 1:36 ` Chao Leng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=114751ac-1f7d-ce5e-12c5-7d6303bdb999@suse.de \
--to=hare@suse.de \
--cc=axboe@fb.com \
--cc=dwagner@suse.de \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=lengchao@huawei.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).