issue with ubi_wl_put_peb

issue with ubi_wl_put_peb

[Nikhilesh Reddy]

Hi


I am currently running into a hard to reproduce issue( takes upto a week 
to reproduce) with ubi_wl_put_peb.

I would appreciate any help you can give me.

In the following stack
-000|__list_del_entry()
-001|list_del()
-002|prot_queue_del()
-003|ubi_wl_put_peb()
-004|ubi_eba_unmap_leb()
-005|ubifs_leb_unmap()
-006|ubifs_gc_start_commit()
-007|do_commit()
-008|run_bg_commit(inline)
-008|ubifs_bg_thread()
-009|kthread()

This issue was seen with CONFIG_DEBUG_LIST=y

What we see is that the wl entry that is being put is not in the 
protection queue but seems to be in free/used trees.

In the code  below:
http://git.infradead.org/linux-ubifs.git/blob/HEAD:/drivers/mtd/ubi/wl.c#l1231 


We see that the the wl_entry is checked to be in used/scrub and 
erroneous trees .. and then it simply assumes that it is in the 
protection queue if it is not in any of the rb trees.

There appears to be a race where the  wl_entry is being put before it 
actually has a chance to be inserted into the protection queue.

This seems to occur when ubi_io_write_vid_hdr takes long to write the 
VID header.


I noticed that a check for a similar situation is 
http://git.infradead.org/linux-ubifs.git/blob/HEAD:/drivers/mtd/ubi/wl.c#l761

Is the same check needed in the case of ubi_wl_put_peb as well?

Can you please tell me if this sounds like a known issue?

I am a little lost on when |ubi_eba_unmap_leb would be called before the 
PEB has a chance to be inserted into a another tree from the free tree.

I would be grateful any pointers that you can give me so i can get to 
the root cause.


-- 
Thanks
Nikhilesh Reddy

Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project.

Re: issue with ubi_wl_put_peb

[Richard Weinberger]

Nikhilesh Reddy,

On 23.08.2016 20:55, Nikhilesh Reddy wrote:
> Hi
> 
> 
> I am currently running into a hard to reproduce issue( takes upto a week to reproduce) with ubi_wl_put_peb.
> 
> I would appreciate any help you can give me.
> 
> In the following stack
> -000|__list_del_entry()
> -001|list_del()
> -002|prot_queue_del()
> -003|ubi_wl_put_peb()
> -004|ubi_eba_unmap_leb()
> -005|ubifs_leb_unmap()
> -006|ubifs_gc_start_commit()
> -007|do_commit()
> -008|run_bg_commit(inline)
> -008|ubifs_bg_thread()
> -009|kthread()
> 
> This issue was seen with CONFIG_DEBUG_LIST=y
> 
> What we see is that the wl entry that is being put is not in the protection queue but seems to be in free/used trees.

So, you trigger a list assert?
Please share the kernel log.

> In the code  below:
> http://git.infradead.org/linux-ubifs.git/blob/HEAD:/drivers/mtd/ubi/wl.c#l1231
> 
> We see that the the wl_entry is checked to be in used/scrub and erroneous trees .. and then it simply assumes that it is in the protection queue if it is not in any of the rb trees.
> 
> There appears to be a race where the  wl_entry is being put before it actually has a chance to be inserted into the protection queue.
> 
> This seems to occur when ubi_io_write_vid_hdr takes long to write the VID header.
> 
> 
> I noticed that a check for a similar situation is http://git.infradead.org/linux-ubifs.git/blob/HEAD:/drivers/mtd/ubi/wl.c#l761
> 
> Is the same check needed in the case of ubi_wl_put_peb as well?
> 
> Can you please tell me if this sounds like a known issue?
> 
> I am a little lost on when |ubi_eba_unmap_leb would be called before the PEB has a chance to be inserted into a another tree from the free tree.
> 
> I would be grateful any pointers that you can give me so i can get to the root cause.

Hmmmm, is Fastmap involved?

Thanks,
//richard

Re: issue with ubi_wl_put_peb

[Nikhilesh Reddy]

On Wed 24 Aug 2016 01:48:29 PM PDT, Richard Weinberger wrote:
> Nikhilesh Reddy,
>
> On 23.08.2016 20:55, Nikhilesh Reddy wrote:
>> Hi
>>
>>
>> I am currently running into a hard to reproduce issue( takes upto a week to reproduce) with ubi_wl_put_peb.
>>
>> I would appreciate any help you can give me.
>>
>> In the following stack
>> -000|__list_del_entry()
>> -001|list_del()
>> -002|prot_queue_del()
>> -003|ubi_wl_put_peb()
>> -004|ubi_eba_unmap_leb()
>> -005|ubifs_leb_unmap()
>> -006|ubifs_gc_start_commit()
>> -007|do_commit()
>> -008|run_bg_commit(inline)
>> -008|ubifs_bg_thread()
>> -009|kthread()
>>
>> This issue was seen with CONFIG_DEBUG_LIST=y
>>
>> What we see is that the wl entry that is being put is not in the protection queue but seems to be in free/used trees.
>
> So, you trigger a list assert?
> Please share the kernel log.
>
>> In the code  below:
>> http://git.infradead.org/linux-ubifs.git/blob/HEAD:/drivers/mtd/ubi/wl.c#l1231
>>
>> We see that the the wl_entry is checked to be in used/scrub and erroneous trees .. and then it simply assumes that it is in the protection queue if it is not in any of the rb trees.
>>
>> There appears to be a race where the  wl_entry is being put before it actually has a chance to be inserted into the protection queue.
>>
>> This seems to occur when ubi_io_write_vid_hdr takes long to write the VID header.
>>
>>
>> I noticed that a check for a similar situation is http://git.infradead.org/linux-ubifs.git/blob/HEAD:/drivers/mtd/ubi/wl.c#l761
>>
>> Is the same check needed in the case of ubi_wl_put_peb as well?
>>
>> Can you please tell me if this sounds like a known issue?
>>
>> I am a little lost on when |ubi_eba_unmap_leb would be called before the PEB has a chance to be inserted into a another tree from the free tree.
>>
>> I would be grateful any pointers that you can give me so i can get to the root cause.
>
> Hmmmm, is Fastmap involved?
>
> Thanks,
> //richard

Hi
Actually  FASTMAP was not involved.
This issue is pretty hard to reproduce ... just had reports of one 
instance where this occured.
If i see it again i will try to get all the logs.

Thanks So much for your help

--
Thanks
Nikhilesh Reddy

Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora 
Forum,
a Linux Foundation Collaborative Project.

Re: issue with ubi_wl_put_peb

[Richard Weinberger]

Nikhilesh Reddy,

On 30.08.2016 19:53, Nikhilesh Reddy wrote:
> Hi
> Actually  FASTMAP was not involved.

So, CONFIG_MTD_UBI_FASTMAP=n?

> This issue is pretty hard to reproduce ... just had reports of one instance where this occured.
> If i see it again i will try to get all the logs.

Always collect logs. :-)
At least the kernel trace/oops.

Thanks,
//richard