Regarding UBI/UBIFS corruptions during random power-cuts.

Regarding UBI/UBIFS corruptions during random power-cuts.

[Ritesh Harjani]

Hi Richard,

Recently we also encountered a similar problem which is discussed in 
this thread. This was encountered on board with kernel 3.18.
This is happening with external power supply power cut testing.

https://patchwork.ozlabs.org/patch/719544/

Can you kindly help with what happened on this issue?
Were we able to identify any bug in kernel for this? Since I don't have 
any later conversations, it will be good to know if this problem is 
fixed in the latest kernel or not.
If yes, then could you please point me to the patches/discussion on 
this, if any.


Issue logs :-
https://pastebin.com/TL2b0ZbH


Regards
Ritesh

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: Regarding UBI/UBIFS corruptions during random power-cuts.

[Richard Weinberger]

Ritesh,

Am 25.07.2017 um 13:16 schrieb Ritesh Harjani:
> Hi Richard,
> 
> Recently we also encountered a similar problem which is discussed in this thread. This was encountered on board with kernel 3.18.
> This is happening with external power supply power cut testing.
> 
> https://patchwork.ozlabs.org/patch/719544/
> 
> Can you kindly help with what happened on this issue?
> Were we able to identify any bug in kernel for this? Since I don't have any later conversations, it will be good to know if this problem is fixed in the latest kernel or not.

Bhuvan and I are still looking into this.

> If yes, then could you please point me to the patches/discussion on this, if any.
> 
> 
> Issue logs :-
> https://pastebin.com/TL2b0ZbH

Do you see this in the same hardware?
The error you're facing is rather generic.
In our case we can reproduce the issue very reliable when we enable the UBIFS /sys/kernel/debug/ubifs/chk_fs debugfs knob.
Can you please also try? Please note that chk_fs will make mounting UBIFS very slow.

I'm very interested in your results, maybe it helps tracking down the root cause.

Thanks,
//richard

Re: Regarding UBI/UBIFS corruptions during random power-cuts.

[Ritesh Harjani]

On 7/27/2017 1:35 AM, Richard Weinberger wrote:
> Ritesh,
> 
> Am 25.07.2017 um 13:16 schrieb Ritesh Harjani:
>> Hi Richard,
>>
>> Recently we also encountered a similar problem which is discussed in this thread. This was encountered on board with kernel 3.18.
>> This is happening with external power supply power cut testing.
>>
>> https://patchwork.ozlabs.org/patch/719544/
>>
>> Can you kindly help with what happened on this issue?
>> Were we able to identify any bug in kernel for this? Since I don't have any later conversations, it will be good to know if this problem is fixed in the latest kernel or not.
> 
> Bhuvan and I are still looking into this.

Thanks for the updating us.

> 
>> If yes, then could you please point me to the patches/discussion on this, if any.
>>
>>
>> Issue logs :-
>> https://pastebin.com/TL2b0ZbH
> 
> Do you see this in the same hardware?
No, the HW is different. It is some MSM Hardware.

> The error you're facing is rather generic.
> In our case we can reproduce the issue very reliable when we enable the UBIFS /sys/kernel/debug/ubifs/chk_fs debugfs knob.
> Can you please also try? Please note that chk_fs will make mounting UBIFS very slow.
> 
> I'm very interested in your results, maybe it helps tracking down the root cause.
Sure, I will give this a shot and let you know the outcome.


Thanks
Ritesh

> 
> Thanks,
> //richard
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: Regarding UBI/UBIFS corruptions during random power-cuts.

[Richard Weinberger]

Ritesh,

Am 27.07.2017 um 16:11 schrieb Ritesh Harjani:
>> The error you're facing is rather generic.
>> In our case we can reproduce the issue very reliable when we enable the UBIFS /sys/kernel/debug/ubifs/chk_fs debugfs knob.
>> Can you please also try? Please note that chk_fs will make mounting UBIFS very slow.
>>
>> I'm very interested in your results, maybe it helps tracking down the root cause.
> Sure, I will give this a shot and let you know the outcome.

Have you been able to trigger it again on your target?

Thanks,
//richard

Re: Regarding UBI/UBIFS corruptions during random power-cuts.

[Ritesh Harjani]

Hi Richard,

On 8/6/2017 2:35 PM, Richard Weinberger wrote:
> Ritesh,
> 
> Am 27.07.2017 um 16:11 schrieb Ritesh Harjani:
>>> The error you're facing is rather generic.
>>> In our case we can reproduce the issue very reliable when we enable the UBIFS /sys/kernel/debug/ubifs/chk_fs debugfs knob.
>>> Can you please also try? Please note that chk_fs will make mounting UBIFS very slow.
>>>
>>> I'm very interested in your results, maybe it helps tracking down the root cause.
>> Sure, I will give this a shot and let you know the outcome.
> 
> Have you been able to trigger it again on your target?
Yes, we tried reproducing with chk_fs enabled. But could not reproduce 
it yet (earlier we were trying without chk_fs enabled). We will continue 
to retry for more time. Is there anything specific scenario you think I 
should include in power on/off to reproduce this issue. The test(with 
chk_fs enabled) is still running since 6-7 hours.

> 
> Thanks,
> //richard
> 

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: Regarding UBI/UBIFS corruptions during random power-cuts.

[Ritesh Harjani]

Hi Richard,

On 8/7/2017 11:39 PM, Ritesh Harjani wrote:
> Hi Richard,
> 
> On 8/6/2017 2:35 PM, Richard Weinberger wrote:
>> Ritesh,
>>
>> Am 27.07.2017 um 16:11 schrieb Ritesh Harjani:
>>>> The error you're facing is rather generic.
>>>> In our case we can reproduce the issue very reliable when we enable 
>>>> the UBIFS /sys/kernel/debug/ubifs/chk_fs debugfs knob.
>>>> Can you please also try? Please note that chk_fs will make mounting 
>>>> UBIFS very slow.
>>>>
>>>> I'm very interested in your results, maybe it helps tracking down 
>>>> the root cause.
>>> Sure, I will give this a shot and let you know the outcome.
>>
>> Have you been able to trigger it again on your target?
> Yes, we tried reproducing with chk_fs enabled. But could not reproduce 
> it yet (earlier we were trying without chk_fs enabled). We will continue 
> to retry for more time. Is there anything specific scenario you think I 
> should include in power on/off to reproduce this issue. The test(with 
> chk_fs enabled) is still running since 6-7 hours.

This could not be reproduced yet. Ran tests for overnight.
Do you have anything else for us to try? Also kindly loops us in 
whenever you have any updates on the fix of this issue :)

Thanks
Ritesh

> 
>>
>> Thanks,
>> //richard
>>
> 

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: Regarding UBI/UBIFS corruptions during random power-cuts.

[Richard Weinberger]

Ritesh,

Am 09.08.2017 um 17:23 schrieb Ritesh Harjani:
>>> Have you been able to trigger it again on your target?
>> Yes, we tried reproducing with chk_fs enabled. But could not reproduce it yet (earlier we were trying without chk_fs enabled). We will continue to retry for more time. Is there
>> anything specific scenario you think I should include in power on/off to reproduce this issue. The test(with chk_fs enabled) is still running since 6-7 hours.
> 
> This could not be reproduced yet. Ran tests for overnight.
> Do you have anything else for us to try? Also kindly loops us in whenever you have any updates on the fix of this issue :)

First of all, we don't know whether this issue is related in any kind.
99% of all UBIFS issue I get in touch with (mostly off list) have their roots in driver and hardware specific issues.

Make sure that you do a "real" power-cut. i.e. make sure that the board really loses all power.
A plain chip reset often does not trigger these issues because the NAND still has power to complete.

Thanks,
//richard