mt8173-evb: failing to boot in linux-next and mainline

mt8173-evb: failing to boot in linux-next and mainline

[Kevin Hilman]

Hello,

The mt8173-evb has been failing to boot in linux-next since 29 Feb,
and is now causing failures in mainline[2] since the merge window is
open and things are getting merged.

Bisecting this has led to inconclusive results, so I'm still not sure
what the real culprit is.  The last few kernel for the failures are:

[    0.570979] NET: Registered protocol family 17
[    0.571615] 9pnet: Installing 9P2000 support
[    0.572211] Key type dns_resolver registered
[    0.573195] registered taskstats version 1
[    0.574170] hctosys: unable to open rtc device (rtc0)
[    0.581122] usb_vbus: disabling

Thinking it was this regulator getting prematurely disabled, I set the
'regulator-always-on' property, which doesn't do much better, but does
report a CPU stall:

[    0.556186] NET: Registered protocol family 17
[    0.556829] 9pnet: Installing 9P2000 support
[    0.557496] Key type dns_resolver registered
[    0.558669] registered taskstats version 1
[    0.559611] hctosys: unable to open rtc device (rtc0)
[   21.573504] INFO: rcu_preempt detected stalls on CPUs/tasks:
[   21.574227]  1-...: (1 GPs behind) idle=483/140000000000000/0
softirq=221/222 fqs=2638
[   21.575227]  (detected by 3, t=5252 jiffies, g=-238, c=-239, q=204)
[   21.576017] Task dump for CPU 1:
[   21.576424] swapper/0       R  running task        0     1      0 0x00000002
[   21.577316] Call trace:
[   21.577634] [<ffffff8008086c20>] __switch_to+0xd0/0xdc
[   21.578280] [<ffffffc077096900>] 0xffffffc077096900
[   33.321443] random: nonblocking pool is initialized
[   84.593487] INFO: rcu_preempt detected stalls on CPUs/tasks:
[   84.594207]  1-...: (1 GPs behind) idle=483/140000000000000/0
softirq=221/222 fqs=10861
[   84.595216]  (detected by 3, t=21007 jiffies, g=-238, c=-239, q=204)
[   84.596017] Task dump for CPU 1:
[   84.596424] swapper/0       R  running task        0     1      0 0x00000002
[   84.597315] Call trace:
[   84.597628] [<ffffff8008086c20>] __switch_to+0xd0/0xdc
[   84.598273] [<ffffffc077096900>] 0xffffffc077096900

Anyways, I gave up on this for now hoping someone here would take a closer look.

Thanks,

Kevin

[1] https://kernelci.org/boot/mt8173-evb/job/next/kernel/next-20160324/defconfig/defconfig/lab/lab-khilman/?_id=56f3b10759b51423934b0500
[2] https://kernelci.org/boot/mt8173-evb/job/mainline/kernel/v4.5-12149-gaca04ce5dbda/defconfig/defconfig/lab/lab-khilman/?_id=56f3c33f59b5145aa14b04f2

Re: mt8173-evb: failing to boot in linux-next and mainline

[Eddie Huang]

Hi Kevin,

On Thu, 2016-03-24 at 11:42 -0700, Kevin Hilman wrote:
> Hello,
> 
> The mt8173-evb has been failing to boot in linux-next since 29 Feb,
> and is now causing failures in mainline[2] since the merge window is
> open and things are getting merged.
> 
> Bisecting this has led to inconclusive results, so I'm still not sure
> what the real culprit is.  The last few kernel for the failures are:
> 
> [    0.570979] NET: Registered protocol family 17
> [    0.571615] 9pnet: Installing 9P2000 support
> [    0.572211] Key type dns_resolver registered
> [    0.573195] registered taskstats version 1
> [    0.574170] hctosys: unable to open rtc device (rtc0)
> [    0.581122] usb_vbus: disabling
> 
> Thinking it was this regulator getting prematurely disabled, I set the
> 'regulator-always-on' property, which doesn't do much better, but does
> report a CPU stall:
> 
> [    0.556186] NET: Registered protocol family 17
> [    0.556829] 9pnet: Installing 9P2000 support
> [    0.557496] Key type dns_resolver registered
> [    0.558669] registered taskstats version 1
> [    0.559611] hctosys: unable to open rtc device (rtc0)
> [   21.573504] INFO: rcu_preempt detected stalls on CPUs/tasks:
> [   21.574227]  1-...: (1 GPs behind) idle=483/140000000000000/0
> softirq=221/222 fqs=2638
> [   21.575227]  (detected by 3, t=5252 jiffies, g=-238, c=-239, q=204)
> [   21.576017] Task dump for CPU 1:
> [   21.576424] swapper/0       R  running task        0     1      0 0x00000002
> [   21.577316] Call trace:
> [   21.577634] [<ffffff8008086c20>] __switch_to+0xd0/0xdc
> [   21.578280] [<ffffffc077096900>] 0xffffffc077096900
> [   33.321443] random: nonblocking pool is initialized
> [   84.593487] INFO: rcu_preempt detected stalls on CPUs/tasks:
> [   84.594207]  1-...: (1 GPs behind) idle=483/140000000000000/0
> softirq=221/222 fqs=10861
> [   84.595216]  (detected by 3, t=21007 jiffies, g=-238, c=-239, q=204)
> [   84.596017] Task dump for CPU 1:
> [   84.596424] swapper/0       R  running task        0     1      0 0x00000002
> [   84.597315] Call trace:
> [   84.597628] [<ffffff8008086c20>] __switch_to+0xd0/0xdc
> [   84.598273] [<ffffffc077096900>] 0xffffffc077096900
> 
> Anyways, I gave up on this for now hoping someone here would take a closer look.
> 
> Thanks,
> 
> Kevin
> 
> [1] https://kernelci.org/boot/mt8173-evb/job/next/kernel/next-20160324/defconfig/defconfig/lab/lab-khilman/?_id=56f3b10759b51423934b0500
> [2] https://kernelci.org/boot/mt8173-evb/job/mainline/kernel/v4.5-12149-gaca04ce5dbda/defconfig/defconfig/lab/lab-khilman/?_id=56f3c33f59b5145aa14b04f2
> 

Thanks your testing again. I can reproduce your case in my local
environment using 4.6-rc1. After checking, there are two ways to avoid
the hang:
1. set clk_ignore_unused to kernel command line
2. Revert patch "cc8ed76 soc: mediatek: SCPSYS: Fix double enabling of
regulators"

It seems there is a problem between turn on/off clock and power domain.
We will try to fix this and send patch to public once have a solution.

Thanks
Eddie

Re: mt8173-evb: failing to boot in linux-next and mainline

[Kevin Hilman]

Hi Eddie,

Eddie Huang <eddie.huang-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org> writes:

[...]

> Thanks your testing again. I can reproduce your case in my local
> environment using 4.6-rc1. After checking, there are two ways to avoid
> the hang:
> 1. set clk_ignore_unused to kernel command line
> 2. Revert patch "cc8ed76 soc: mediatek: SCPSYS: Fix double enabling of
> regulators"
>
> It seems there is a problem between turn on/off clock and power domain.
> We will try to fix this and send patch to public once have a solution.

Thanks for verifying.

I've confirmed that reverting the patch suggested above fixes the
problem.

Since -rc1 is out, and this problem still exists in mainline, if a
proper fix is not found very soon, I suggest reverting that patch in mainline.

Kevin

Re: mt8173-evb: failing to boot in linux-next and mainline

[James Liao]

Hi Sascha,

On Mon, 2016-03-28 at 08:15 -0700, Kevin Hilman wrote:
> Hi Eddie,
> 
> Eddie Huang <eddie.huang-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org> writes:
> 
> [...]
> 
> > Thanks your testing again. I can reproduce your case in my local
> > environment using 4.6-rc1. After checking, there are two ways to avoid
> > the hang:
> > 1. set clk_ignore_unused to kernel command line
> > 2. Revert patch "cc8ed76 soc: mediatek: SCPSYS: Fix double enabling of
> > regulators"
> >
> > It seems there is a problem between turn on/off clock and power domain.
> > We will try to fix this and send patch to public once have a solution.
> 
> Thanks for verifying.
> 
> I've confirmed that reverting the patch suggested above fixes the
> problem.
> 
> Since -rc1 is out, and this problem still exists in mainline, if a
> proper fix is not found very soon, I suggest reverting that patch in mainline.
> 
> Kevin

According to Eddie and Kevin's test, I think some clocks
(mm/mfg/venc/venclt) may be turned off and there are drivers need these
clocks without clk_prepare_enable(). But I have no idea to find out what
driver causes this issue. So I prefer to revert it on v4.6-rc1. Do you
have comments about this issue?


Best regards,

James

Re: mt8173-evb: failing to boot in linux-next and mainline

[Kevin Hilman]

On Mon, Mar 28, 2016 at 10:40 PM, James Liao <jamesjj.liao-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org> wrote:
> Hi Sascha,
>
> On Mon, 2016-03-28 at 08:15 -0700, Kevin Hilman wrote:
>> Hi Eddie,
>>
>> Eddie Huang <eddie.huang-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org> writes:
>>
>> [...]
>>
>> > Thanks your testing again. I can reproduce your case in my local
>> > environment using 4.6-rc1. After checking, there are two ways to avoid
>> > the hang:
>> > 1. set clk_ignore_unused to kernel command line
>> > 2. Revert patch "cc8ed76 soc: mediatek: SCPSYS: Fix double enabling of
>> > regulators"
>> >
>> > It seems there is a problem between turn on/off clock and power domain.
>> > We will try to fix this and send patch to public once have a solution.
>>
>> Thanks for verifying.
>>
>> I've confirmed that reverting the patch suggested above fixes the
>> problem.
>>
>> Since -rc1 is out, and this problem still exists in mainline, if a
>> proper fix is not found very soon, I suggest reverting that patch in mainline.
>>
>> Kevin
>
> According to Eddie and Kevin's test, I think some clocks
> (mm/mfg/venc/venclt) may be turned off and there are drivers need these
> clocks without clk_prepare_enable(). But I have no idea to find out what
> driver causes this issue. So I prefer to revert it on v4.6-rc1. Do you
> have comments about this issue?

FYI... mt8173-evb is still failing in mainline v4.6-rc3:

   https://kernelci.org/boot/all/job/mainline/kernel/v4.6-rc3/

please fix and/or revert ASAP so this boot regression is not in v4.6 final.

Kevin

Re: mt8173-evb: failing to boot in linux-next and mainline

[Matthias Brugger]

On 12/04/16 01:12, Kevin Hilman wrote:
> On Mon, Mar 28, 2016 at 10:40 PM, James Liao <jamesjj.liao-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org> wrote:
>> Hi Sascha,
>>
>> On Mon, 2016-03-28 at 08:15 -0700, Kevin Hilman wrote:
>>> Hi Eddie,
>>>
>>> Eddie Huang <eddie.huang-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org> writes:
>>>
>>> [...]
>>>
>>>> Thanks your testing again. I can reproduce your case in my local
>>>> environment using 4.6-rc1. After checking, there are two ways to avoid
>>>> the hang:
>>>> 1. set clk_ignore_unused to kernel command line
>>>> 2. Revert patch "cc8ed76 soc: mediatek: SCPSYS: Fix double enabling of
>>>> regulators"
>>>>
>>>> It seems there is a problem between turn on/off clock and power domain.
>>>> We will try to fix this and send patch to public once have a solution.
>>>
>>> Thanks for verifying.
>>>
>>> I've confirmed that reverting the patch suggested above fixes the
>>> problem.
>>>
>>> Since -rc1 is out, and this problem still exists in mainline, if a
>>> proper fix is not found very soon, I suggest reverting that patch in mainline.
>>>
>>> Kevin
>>
>> According to Eddie and Kevin's test, I think some clocks
>> (mm/mfg/venc/venclt) may be turned off and there are drivers need these
>> clocks without clk_prepare_enable(). But I have no idea to find out what
>> driver causes this issue. So I prefer to revert it on v4.6-rc1. Do you
>> have comments about this issue?
>
> FYI... mt8173-evb is still failing in mainline v4.6-rc3:
>
>     https://kernelci.org/boot/all/job/mainline/kernel/v4.6-rc3/
>
> please fix and/or revert ASAP so this boot regression is not in v4.6 final.
>

Sorry for the late reply. I'm quite busy in my day job ATM.
I just send the revert James proposed as fix.

Regards,
Matthias

Re: mt8173-evb: failing to boot in linux-next and mainline

[Kevin Hilman]

On Wed, Apr 13, 2016 at 3:15 AM, Matthias Brugger
<matthias.bgg-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>
> On 12/04/16 01:12, Kevin Hilman wrote:
>>
>> On Mon, Mar 28, 2016 at 10:40 PM, James Liao <jamesjj.liao-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
>> wrote:
>>>
>>> Hi Sascha,
>>>
>>> On Mon, 2016-03-28 at 08:15 -0700, Kevin Hilman wrote:
>>>>
>>>> Hi Eddie,
>>>>
>>>> Eddie Huang <eddie.huang-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org> writes:
>>>>
>>>> [...]
>>>>
>>>>> Thanks your testing again. I can reproduce your case in my local
>>>>> environment using 4.6-rc1. After checking, there are two ways to avoid
>>>>> the hang:
>>>>> 1. set clk_ignore_unused to kernel command line
>>>>> 2. Revert patch "cc8ed76 soc: mediatek: SCPSYS: Fix double enabling of
>>>>> regulators"
>>>>>
>>>>> It seems there is a problem between turn on/off clock and power domain.
>>>>> We will try to fix this and send patch to public once have a solution.
>>>>
>>>>
>>>> Thanks for verifying.
>>>>
>>>> I've confirmed that reverting the patch suggested above fixes the
>>>> problem.
>>>>
>>>> Since -rc1 is out, and this problem still exists in mainline, if a
>>>> proper fix is not found very soon, I suggest reverting that patch in
>>>> mainline.
>>>>
>>>> Kevin
>>>
>>>
>>> According to Eddie and Kevin's test, I think some clocks
>>> (mm/mfg/venc/venclt) may be turned off and there are drivers need these
>>> clocks without clk_prepare_enable(). But I have no idea to find out what
>>> driver causes this issue. So I prefer to revert it on v4.6-rc1. Do you
>>> have comments about this issue?
>>
>>
>> FYI... mt8173-evb is still failing in mainline v4.6-rc3:
>>
>>     https://kernelci.org/boot/all/job/mainline/kernel/v4.6-rc3/
>>
>> please fix and/or revert ASAP so this boot regression is not in v4.6
>> final.
>>
>
> Sorry for the late reply. I'm quite busy in my day job ATM.
> I just send the revert James proposed as fix.
>

Thanks, I just tried your branch with arm-soc/for-next and mt8173-evb
is booting again.  Thanks.

Kevin

Re: mt8173-evb: failing to boot in linux-next and mainline

[Kevin Hilman]

On Thu, Apr 14, 2016 at 2:58 PM, Kevin Hilman <khilman-rdvid1DuHRBWk0Htik3J/w@public.gmane.org> wrote:
> On Wed, Apr 13, 2016 at 3:15 AM, Matthias Brugger
> <matthias.bgg-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>
>>
>> On 12/04/16 01:12, Kevin Hilman wrote:
>>>
>>> On Mon, Mar 28, 2016 at 10:40 PM, James Liao <jamesjj.liao-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
>>> wrote:
>>>>
>>>> Hi Sascha,
>>>>
>>>> On Mon, 2016-03-28 at 08:15 -0700, Kevin Hilman wrote:
>>>>>
>>>>> Hi Eddie,
>>>>>
>>>>> Eddie Huang <eddie.huang-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org> writes:
>>>>>
>>>>> [...]
>>>>>
>>>>>> Thanks your testing again. I can reproduce your case in my local
>>>>>> environment using 4.6-rc1. After checking, there are two ways to avoid
>>>>>> the hang:
>>>>>> 1. set clk_ignore_unused to kernel command line
>>>>>> 2. Revert patch "cc8ed76 soc: mediatek: SCPSYS: Fix double enabling of
>>>>>> regulators"
>>>>>>
>>>>>> It seems there is a problem between turn on/off clock and power domain.
>>>>>> We will try to fix this and send patch to public once have a solution.
>>>>>
>>>>>
>>>>> Thanks for verifying.
>>>>>
>>>>> I've confirmed that reverting the patch suggested above fixes the
>>>>> problem.
>>>>>
>>>>> Since -rc1 is out, and this problem still exists in mainline, if a
>>>>> proper fix is not found very soon, I suggest reverting that patch in
>>>>> mainline.
>>>>>
>>>>> Kevin
>>>>
>>>>
>>>> According to Eddie and Kevin's test, I think some clocks
>>>> (mm/mfg/venc/venclt) may be turned off and there are drivers need these
>>>> clocks without clk_prepare_enable(). But I have no idea to find out what
>>>> driver causes this issue. So I prefer to revert it on v4.6-rc1. Do you
>>>> have comments about this issue?
>>>
>>>
>>> FYI... mt8173-evb is still failing in mainline v4.6-rc3:
>>>
>>>     https://kernelci.org/boot/all/job/mainline/kernel/v4.6-rc3/
>>>
>>> please fix and/or revert ASAP so this boot regression is not in v4.6
>>> final.
>>>
>>
>> Sorry for the late reply. I'm quite busy in my day job ATM.
>> I just send the revert James proposed as fix.
>>
>
> Thanks, I just tried your branch with arm-soc/for-next and mt8173-evb
> is booting again.  Thanks.

FYI... mt8173-evb still failing in linux-next, and not sure if it's a
new issue or not.

https://kernelci.org/boot/mt8173-evb/job/next/kernel/next-20160419/defconfig/defconfig/lab/lab-khilman/?_id=57163f9459b514ffc46e78d72

Re: mt8173-evb: failing to boot in linux-next and mainline

[Eddie Huang]

Hi Kevin,

On Tue, 2016-04-19 at 15:57 -0700, Kevin Hilman wrote:
> On Thu, Apr 14, 2016 at 2:58 PM, Kevin Hilman <khilman-rdvid1DuHRBWk0Htik3J/w@public.gmane.org> wrote:
> > On Wed, Apr 13, 2016 at 3:15 AM, Matthias Brugger
> > <matthias.bgg-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >>
> >>
> >> On 12/04/16 01:12, Kevin Hilman wrote:
> >>>
> >>> On Mon, Mar 28, 2016 at 10:40 PM, James Liao <jamesjj.liao-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org>
> >>> wrote:
> >>>>
> >>>> Hi Sascha,
> >>>>
> >>>> On Mon, 2016-03-28 at 08:15 -0700, Kevin Hilman wrote:
> >>>>>
> >>>>> Hi Eddie,
> >>>>>
> >>>>> Eddie Huang <eddie.huang-NuS5LvNUpcJWk0Htik3J/w@public.gmane.org> writes:
> >>>>>
> >>>>> [...]
> >>>>>
> >>>>>> Thanks your testing again. I can reproduce your case in my local
> >>>>>> environment using 4.6-rc1. After checking, there are two ways to avoid
> >>>>>> the hang:
> >>>>>> 1. set clk_ignore_unused to kernel command line
> >>>>>> 2. Revert patch "cc8ed76 soc: mediatek: SCPSYS: Fix double enabling of
> >>>>>> regulators"
> >>>>>>
> >>>>>> It seems there is a problem between turn on/off clock and power domain.
> >>>>>> We will try to fix this and send patch to public once have a solution.
> >>>>>
> >>>>>
> >>>>> Thanks for verifying.
> >>>>>
> >>>>> I've confirmed that reverting the patch suggested above fixes the
> >>>>> problem.
> >>>>>
> >>>>> Since -rc1 is out, and this problem still exists in mainline, if a
> >>>>> proper fix is not found very soon, I suggest reverting that patch in
> >>>>> mainline.
> >>>>>
> >>>>> Kevin
> >>>>
> >>>>
> >>>> According to Eddie and Kevin's test, I think some clocks
> >>>> (mm/mfg/venc/venclt) may be turned off and there are drivers need these
> >>>> clocks without clk_prepare_enable(). But I have no idea to find out what
> >>>> driver causes this issue. So I prefer to revert it on v4.6-rc1. Do you
> >>>> have comments about this issue?
> >>>
> >>>
> >>> FYI... mt8173-evb is still failing in mainline v4.6-rc3:
> >>>
> >>>     https://kernelci.org/boot/all/job/mainline/kernel/v4.6-rc3/
> >>>
> >>> please fix and/or revert ASAP so this boot regression is not in v4.6
> >>> final.
> >>>
> >>
> >> Sorry for the late reply. I'm quite busy in my day job ATM.
> >> I just send the revert James proposed as fix.
> >>
> >
> > Thanks, I just tried your branch with arm-soc/for-next and mt8173-evb
> > is booting again.  Thanks.
> 
> FYI... mt8173-evb still failing in linux-next, and not sure if it's a
> new issue or not.
> 
> https://kernelci.org/boot/mt8173-evb/job/next/kernel/next-20160419/defconfig/defconfig/lab/lab-khilman/?_id=57163f9459b514ffc46e78d72

Old issue. It seems [1] still not in linux-next. After apply, my
mt8173-evb boot to shell successfully.

[1]
http://lists.infradead.org/pipermail/linux-arm-kernel/2016-April/421735.html

Eddie