Regression: v4.0-rc1: iMX6: Hummingboard fails due to drivers/clk changes

Regression: v4.0-rc1: iMX6: Hummingboard fails due to drivers/clk changes

[Russell King - ARM Linux]

All,

I just tried my SolidRun patch set on top of v4.0-rc1, and it fails to
boot.  The kernel itself finishes booting, but it is unable to DHCP (for
root-NFS) - nothing appears on the network.

At first, I thought this would be a network driver bug, but reverting
the mainline changes there didn't fix the regression.  Then I wondered
if it was something in arch/arm/mach-imx, but no.

When reverting all drivers/clk changes (and corresponding include/linux/*.h)
between v3.19 and v4.0-rc1, I find that the regression is fixed.

It seems that some CCF change is causing a regression somewhere in the
clock tree on iMX6 CPUs.

Unfortunately, without networking running, I can't dump the clock tree
to find out what's different, and bisecting will be a major pain (needing
each bisect point to be merged with my patch set, built and tested.)

Has anyone else seen this?

Thanks.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Regression: v4.0-rc1: iMX6: Hummingboard fails due to drivers/clk changes

[Olof Johansson]

On Mon, Feb 23, 2015 at 2:32 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> All,
>
> I just tried my SolidRun patch set on top of v4.0-rc1, and it fails to
> boot.  The kernel itself finishes booting, but it is unable to DHCP (for
> root-NFS) - nothing appears on the network.
>
> At first, I thought this would be a network driver bug, but reverting
> the mainline changes there didn't fix the regression.  Then I wondered
> if it was something in arch/arm/mach-imx, but no.
>
> When reverting all drivers/clk changes (and corresponding include/linux/*.h)
> between v3.19 and v4.0-rc1, I find that the regression is fixed.
>
> It seems that some CCF change is causing a regression somewhere in the
> clock tree on iMX6 CPUs.
>
> Unfortunately, without networking running, I can't dump the clock tree
> to find out what's different, and bisecting will be a major pain (needing
> each bisect point to be merged with my patch set, built and tested.)
>
> Has anyone else seen this?

FWIW, my hummingboard comes up with networking on v4.0-rc1 with
imx_v6_v7_defconfig and multi_v7:

http://arm-soc.lixom.net/bootlogs/mainline/v4.0-rc1/hummingboard-arm-imx_v6_v7_defconfig.html
http://arm-soc.lixom.net/bootlogs/mainline/v4.0-rc1/hummingboard-arm-multi_v7_defconfig.html

Which is unfortunate, since it'd be easy for me to bisect. :(


-Olof

Regression: v4.0-rc1: iMX6: Hummingboard fails due to drivers/clk changes

[Russell King - ARM Linux]

On Mon, Feb 23, 2015 at 02:54:02PM -0800, Olof Johansson wrote:
> On Mon, Feb 23, 2015 at 2:32 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > All,
> >
> > I just tried my SolidRun patch set on top of v4.0-rc1, and it fails to
> > boot.  The kernel itself finishes booting, but it is unable to DHCP (for
> > root-NFS) - nothing appears on the network.
> >
> > At first, I thought this would be a network driver bug, but reverting
> > the mainline changes there didn't fix the regression.  Then I wondered
> > if it was something in arch/arm/mach-imx, but no.
> >
> > When reverting all drivers/clk changes (and corresponding include/linux/*.h)
> > between v3.19 and v4.0-rc1, I find that the regression is fixed.
> >
> > It seems that some CCF change is causing a regression somewhere in the
> > clock tree on iMX6 CPUs.
> >
> > Unfortunately, without networking running, I can't dump the clock tree
> > to find out what's different, and bisecting will be a major pain (needing
> > each bisect point to be merged with my patch set, built and tested.)
> >
> > Has anyone else seen this?
> 
> FWIW, my hummingboard comes up with networking on v4.0-rc1 with
> imx_v6_v7_defconfig and multi_v7:
> 
> http://arm-soc.lixom.net/bootlogs/mainline/v4.0-rc1/hummingboard-arm-imx_v6_v7_defconfig.html
> http://arm-soc.lixom.net/bootlogs/mainline/v4.0-rc1/hummingboard-arm-multi_v7_defconfig.html
> 
> Which is unfortunate, since it'd be easy for me to bisect. :(

Bisect comes out with this commit as the culpret:

commit 035a61c314eb3dab5bcc5683afaf4d412689858a
Author: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Date:   Fri Jan 23 12:03:30 2015 +0100

    clk: Make clk API return per-user struct clk instances

Indeed, having the tree merged with af0f349b2996 works, then
applying the above commit on top causes it to fail.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Regression: v4.0-rc1: iMX6: Hummingboard fails due to drivers/clk changes

[Fabio Estevam]

Hi Russell,

On Mon, Feb 23, 2015 at 7:32 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> All,
>
> I just tried my SolidRun patch set on top of v4.0-rc1, and it fails to
> boot.  The kernel itself finishes booting, but it is unable to DHCP (for
> root-NFS) - nothing appears on the network.
>
> At first, I thought this would be a network driver bug, but reverting
> the mainline changes there didn't fix the regression.  Then I wondered
> if it was something in arch/arm/mach-imx, but no.
>
> When reverting all drivers/clk changes (and corresponding include/linux/*.h)
> between v3.19 and v4.0-rc1, I find that the regression is fixed.
>
> It seems that some CCF change is causing a regression somewhere in the
> clock tree on iMX6 CPUs.
>
> Unfortunately, without networking running, I can't dump the clock tree
> to find out what's different, and bisecting will be a major pain (needing
> each bisect point to be merged with my patch set, built and tested.)
>
> Has anyone else seen this?

Just tried it and I confirm that NFS fails to mount on a imx6q-hummingboard:

[    1.882023] fec 2188000.ethernet eth0: Freescale FEC PHY driver
[Atheros 8035 ethernet] (mii_bus:phy_addr=2188000.ethernet:00, irq)
[    1.894409] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    1.954662] hub 2-1:1.0: USB hub found
[    1.958864] hub 2-1:1.0: 4 ports detected
[    3.882249] fec 2188000.ethernet eth0: Link is Up - 100Mbps/Full -
flow control rx/tx
[    3.890900] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    3.912131] Sending DHCP requests ...

(Gets stuck here)

I haven't seen this problem on other imx6 boards though running the same kernel.

Regards,

Fabio Estevam

Regression: v4.0-rc1: iMX6: Hummingboard fails due to drivers/clk changes

[Russell King - ARM Linux]

On Mon, Feb 23, 2015 at 08:42:55PM -0300, Fabio Estevam wrote:
> Hi Russell,
> 
> On Mon, Feb 23, 2015 at 7:32 PM, Russell King - ARM Linux
> <linux@arm.linux.org.uk> wrote:
> > All,
> >
> > I just tried my SolidRun patch set on top of v4.0-rc1, and it fails to
> > boot.  The kernel itself finishes booting, but it is unable to DHCP (for
> > root-NFS) - nothing appears on the network.
> >
> > At first, I thought this would be a network driver bug, but reverting
> > the mainline changes there didn't fix the regression.  Then I wondered
> > if it was something in arch/arm/mach-imx, but no.
> >
> > When reverting all drivers/clk changes (and corresponding include/linux/*.h)
> > between v3.19 and v4.0-rc1, I find that the regression is fixed.
> >
> > It seems that some CCF change is causing a regression somewhere in the
> > clock tree on iMX6 CPUs.
> >
> > Unfortunately, without networking running, I can't dump the clock tree
> > to find out what's different, and bisecting will be a major pain (needing
> > each bisect point to be merged with my patch set, built and tested.)
> >
> > Has anyone else seen this?
> 
> Just tried it and I confirm that NFS fails to mount on a imx6q-hummingboard:
> 
> [    1.882023] fec 2188000.ethernet eth0: Freescale FEC PHY driver
> [Atheros 8035 ethernet] (mii_bus:phy_addr=2188000.ethernet:00, irq)
> [    1.894409] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> [    1.954662] hub 2-1:1.0: USB hub found
> [    1.958864] hub 2-1:1.0: 4 ports detected
> [    3.882249] fec 2188000.ethernet eth0: Link is Up - 100Mbps/Full -
> flow control rx/tx
> [    3.890900] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [    3.912131] Sending DHCP requests ...
> 
> (Gets stuck here)
> 
> I haven't seen this problem on other imx6 boards though running the
> same kernel.

The reason appears to be a lack of packets coming out of the ethernet.
Why that is, I'm not sure (maybe a gated GPIO16 clock to the phy?)
Given that the phy state seems frozen - it doesn't detect the cable
being disconnected - this seems highly likely.

Looking at imx6q_1588_init(), this could be a problem:

        ptp_clk = of_clk_get(np, 2);
        enet_ref = clk_get_sys(NULL, "enet_ref");
        clksel = ptp_clk == enet_ref ? IMX6Q_GPR1_ENET_CLK_SEL_ANATOP :
                                       IMX6Q_GPR1_ENET_CLK_SEL_PAD;

since ptp_clk and enet_ref will _never_ be the same pointer.  Forcing
that test to be always true fixes the problem.

-- 
FTTC broadband for 0.8mile line: currently at 10.5Mbps down 400kbps up
according to speedtest.net.

Regression: v4.0-rc1: iMX6: Hummingboard fails due to drivers/clk changes

[Fabio Estevam]

On Mon, Feb 23, 2015 at 8:53 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:

> The reason appears to be a lack of packets coming out of the ethernet.
> Why that is, I'm not sure (maybe a gated GPIO16 clock to the phy?)
> Given that the phy state seems frozen - it doesn't detect the cable
> being disconnected - this seems highly likely.
>
> Looking at imx6q_1588_init(), this could be a problem:
>
>         ptp_clk = of_clk_get(np, 2);
>         enet_ref = clk_get_sys(NULL, "enet_ref");
>         clksel = ptp_clk == enet_ref ? IMX6Q_GPR1_ENET_CLK_SEL_ANATOP :
>                                        IMX6Q_GPR1_ENET_CLK_SEL_PAD;
>
> since ptp_clk and enet_ref will _never_ be the same pointer.  Forcing
> that test to be always true fixes the problem.

You are right. With such change imx6q-hummingboard can NFS mount (and
so does imx6q-sabresd).

Regression: v4.0-rc1: iMX6: Hummingboard fails due to drivers/clk changes

[Fabio Estevam]

On Mon, Feb 23, 2015 at 8:27 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Mon, Feb 23, 2015 at 02:54:02PM -0800, Olof Johansson wrote:
>> On Mon, Feb 23, 2015 at 2:32 PM, Russell King - ARM Linux
>> <linux@arm.linux.org.uk> wrote:
>> > All,
>> >
>> > I just tried my SolidRun patch set on top of v4.0-rc1, and it fails to
>> > boot.  The kernel itself finishes booting, but it is unable to DHCP (for
>> > root-NFS) - nothing appears on the network.
>> >
>> > At first, I thought this would be a network driver bug, but reverting
>> > the mainline changes there didn't fix the regression.  Then I wondered
>> > if it was something in arch/arm/mach-imx, but no.
>> >
>> > When reverting all drivers/clk changes (and corresponding include/linux/*.h)
>> > between v3.19 and v4.0-rc1, I find that the regression is fixed.
>> >
>> > It seems that some CCF change is causing a regression somewhere in the
>> > clock tree on iMX6 CPUs.
>> >
>> > Unfortunately, without networking running, I can't dump the clock tree
>> > to find out what's different, and bisecting will be a major pain (needing
>> > each bisect point to be merged with my patch set, built and tested.)
>> >

> Bisect comes out with this commit as the culpret:
>
> commit 035a61c314eb3dab5bcc5683afaf4d412689858a
> Author: Tomeu Vizoso <tomeu.vizoso@collabora.com>
> Date:   Fri Jan 23 12:03:30 2015 +0100
>
>     clk: Make clk API return per-user struct clk instances
>
> Indeed, having the tree merged with af0f349b2996 works, then
> applying the above commit on top causes it to fail.

Tomeu, could you please take a look at this regression?

Regression: v4.0-rc1: iMX6: Hummingboard fails due to drivers/clk changes

[Robert Gadsdon]

I have applied the change to mach-imx6q.c for 4.0-rc1 on a CuBox-i4Pro,
and eth0/networking now works correctly, again.

Thanks!

Robert Gadsdon.