msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up

msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up

[Paolo Pisati]

From time to time, my apq8096-db820c fails to boot to userspace, reset back to
fastboot and locks up: to easily reproduce the issue, i'm boot looping using a
cron job with a 1 min reboot entry on the board while leaving a "while 1; do
fastboot boot boot.img; done" on the host pc.

The issue is present in mainline up to 5.2-rc4, using defconfig and:

CONFIG_SCSI_UFS_QCOM=y
CONFIG_PHY_QCOM_QMP=y
CONFIG_PHY_QCOM_UFS=y

but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or
mainline), where it's even easier to reproduce (e.g. takes way less reboots to
trigger it).

These are the last lines printed out:
...
[    7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
[    7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
[    7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
[    7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out
[    7.538596] phy phy-34000.phy.0: phy init failed --> -110
[    7.550891] qcom-pcie: probe of 600000.pcie failed with error -110
[    7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,
using dummy regulator

Log Type: B - Since Boot(Power On Reset),  D - Delta,  S - Statistic
S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
S - IMAGE_VARIANT_STRING=M8996LAB
S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
S - Boot Interface: UFS
S - Secure Boot: Off
...

Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/

Any idea what is going on? Am i doing something wrong?
-- 
bye,
p.

Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up

[Niklas Cassel]

On Mon, Jun 10, 2019 at 03:44:01PM +0200, Paolo Pisati wrote:
> From time to time, my apq8096-db820c fails to boot to userspace, reset back to
> fastboot and locks up: to easily reproduce the issue, i'm boot looping using a
> cron job with a 1 min reboot entry on the board while leaving a "while 1; do
> fastboot boot boot.img; done" on the host pc.
> 
> The issue is present in mainline up to 5.2-rc4, using defconfig and:
> 
> CONFIG_SCSI_UFS_QCOM=y
> CONFIG_PHY_QCOM_QMP=y
> CONFIG_PHY_QCOM_UFS=y
> 
> but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or
> mainline), where it's even easier to reproduce (e.g. takes way less reboots to
> trigger it).

Hello Paolo,

I have a guess of what is going on.
db820c has 3 PCIe controllers,
that shares a singe QMP block (that has clocks, regulators, and resets).
The QMP block has 3 PCIe PHYs, that have their own clocks and resets.

> 
> These are the last lines printed out:
> ...
> [    7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
> [    7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
> [    7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
> [    7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out

^^ here the phy_init() called from pcie-qcom.c
which ends up to a call to qcom_qmp_phy_enable()

which has this code:

        ret = qcom_qmp_phy_com_init(qphy);
        if (ret)
                return ret;

qcom_qmp_phy_com_init() has this code:

        if (qmp->init_count++) {
                mutex_unlock(&qmp->phy_mutex);
                return 0;
        }

qcom_qmp_phy_com_init() later fails,
since the common block init time out, so the qmp driver
disables clocks, asserts reset, and disables regulators


> [    7.538596] phy phy-34000.phy.0: phy init failed --> -110
> [    7.550891] qcom-pcie: probe of 600000.pcie failed with error -110

^^ here the first PCIe controller instance fails to probe

> [    7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,
> using dummy regulator

^^ here the second PCIe controller is probed.

it will call phy_init()

which will again call qcom_qmp_phy_enable() which will call
qcom_qmp_phy_com_init()

where this code:

        if (qmp->init_count++) {
                mutex_unlock(&qmp->phy_mutex);
                return 0;
        }

now will return 0,

so clocks will never be enabled, resets never deasserted, regulators
never enabled.

since qcom_qmp_phy_com_init() returns success in this case,
qcom_qmp_phy_enable() will try to continue with the init,
and writes to disabled hardware is usually not a good idea.

I think the proper fix for this is:

diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c b/drivers/phy/qualcomm/phy-qcom-qmp.c
index cd91b4179b10..22352e3b0ec5 100644
--- a/drivers/phy/qualcomm/phy-qcom-qmp.c
+++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
@@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
 
        ret = qcom_qmp_phy_com_init(qphy);
        if (ret)
-               return ret;
+               goto err_lane_rst;
 
        if (cfg->has_lane_rst) {
                ret = reset_control_deassert(qphy->lane_rst);



Kind regards,
Niklas

> 
> Log Type: B - Since Boot(Power On Reset),  D - Delta,  S - Statistic
> S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
> S - IMAGE_VARIANT_STRING=M8996LAB
> S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
> S - Boot Interface: UFS
> S - Secure Boot: Off
> ...
> 
> Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
> 
> Any idea what is going on? Am i doing something wrong?
> -- 
> bye,
> p.

Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up

[Niklas Cassel]

On Tue, Jun 11, 2019 at 07:12:25PM +0200, Niklas Cassel wrote:
> On Mon, Jun 10, 2019 at 03:44:01PM +0200, Paolo Pisati wrote:
> > From time to time, my apq8096-db820c fails to boot to userspace, reset back to
> > fastboot and locks up: to easily reproduce the issue, i'm boot looping using a
> > cron job with a 1 min reboot entry on the board while leaving a "while 1; do
> > fastboot boot boot.img; done" on the host pc.
> > 
> > The issue is present in mainline up to 5.2-rc4, using defconfig and:
> > 
> > CONFIG_SCSI_UFS_QCOM=y
> > CONFIG_PHY_QCOM_QMP=y
> > CONFIG_PHY_QCOM_UFS=y
> > 
> > but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or
> > mainline), where it's even easier to reproduce (e.g. takes way less reboots to
> > trigger it).
> 
> Hello Paolo,
> 
> I have a guess of what is going on.
> db820c has 3 PCIe controllers,
> that shares a singe QMP block (that has clocks, regulators, and resets).
> The QMP block has 3 PCIe PHYs, that have their own clocks and resets.
> 
> > 
> > These are the last lines printed out:
> > ...
> > [    7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
> > [    7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
> > [    7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
> > [    7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out
> 
> ^^ here the phy_init() called from pcie-qcom.c
> which ends up to a call to qcom_qmp_phy_enable()
> 
> which has this code:
> 
>         ret = qcom_qmp_phy_com_init(qphy);
>         if (ret)
>                 return ret;
> 
> qcom_qmp_phy_com_init() has this code:
> 
>         if (qmp->init_count++) {
>                 mutex_unlock(&qmp->phy_mutex);
>                 return 0;
>         }
> 
> qcom_qmp_phy_com_init() later fails,
> since the common block init time out, so the qmp driver
> disables clocks, asserts reset, and disables regulators
> 
> 
> > [    7.538596] phy phy-34000.phy.0: phy init failed --> -110
> > [    7.550891] qcom-pcie: probe of 600000.pcie failed with error -110
> 
> ^^ here the first PCIe controller instance fails to probe
> 
> > [    7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,
> > using dummy regulator
> 
> ^^ here the second PCIe controller is probed.
> 
> it will call phy_init()
> 
> which will again call qcom_qmp_phy_enable() which will call
> qcom_qmp_phy_com_init()
> 
> where this code:
> 
>         if (qmp->init_count++) {
>                 mutex_unlock(&qmp->phy_mutex);
>                 return 0;
>         }
> 
> now will return 0,
> 
> so clocks will never be enabled, resets never deasserted, regulators
> never enabled.
> 
> since qcom_qmp_phy_com_init() returns success in this case,
> qcom_qmp_phy_enable() will try to continue with the init,
> and writes to disabled hardware is usually not a good idea.
> 
> I think the proper fix for this is:
> 
> diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c b/drivers/phy/qualcomm/phy-qcom-qmp.c
> index cd91b4179b10..22352e3b0ec5 100644
> --- a/drivers/phy/qualcomm/phy-qcom-qmp.c
> +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
> @@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
>  
>         ret = qcom_qmp_phy_com_init(qphy);
>         if (ret)
> -               return ret;
> +               goto err_lane_rst;
>  
>         if (cfg->has_lane_rst) {
>                 ret = reset_control_deassert(qphy->lane_rst);
> 
> 
> 
> Kind regards,
> Niklas
> 
> > 
> > Log Type: B - Since Boot(Power On Reset),  D - Delta,  S - Statistic
> > S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
> > S - IMAGE_VARIANT_STRING=M8996LAB
> > S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
> > S - Boot Interface: UFS
> > S - Secure Boot: Off
> > ...
> > 
> > Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
> > 
> > Any idea what is going on? Am i doing something wrong?
> > -- 
> > bye,
> > p.

Adding Vivek.

Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up

[Paolo Pisati]

On Wed, Jun 12, 2019 at 03:17:35PM +0200, Niklas Cassel wrote:
> > diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c b/drivers/phy/qualcomm/phy-qcom-qmp.c
> > index cd91b4179b10..22352e3b0ec5 100644
> > --- a/drivers/phy/qualcomm/phy-qcom-qmp.c
> > +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
> > @@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
> >  
> >         ret = qcom_qmp_phy_com_init(qphy);
> >         if (ret)
> > -               return ret;
> > +               goto err_lane_rst;
> >  
> >         if (cfg->has_lane_rst) {
> >                 ret = reset_control_deassert(qphy->lane_rst);

Hi Niklas,
unfortunately, it didn't help - i added a printk, to highlight when it failed:

--- a/drivers/phy/qualcomm/phy-qcom-qmp.c
+++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
@@ -1489,8 +1489,10 @@ static int qcom_qmp_phy_enable(struct phy *phy)
        }
 
        ret = qcom_qmp_phy_com_init(qphy);
-       if (ret)
-               return ret;
+       if (ret) {
+               dev_err(qmp->dev, "qphy initialization failed\n");
+               goto err_lane_rst;
+       }
 
        if (cfg->has_lane_rst) {
                ret = reset_control_deassert(qphy->lane_rst);

After several reboots i was able to trigger the phy init failure again:

...
[    2.223999] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
[    2.224956] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
[    2.228798] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
[    2.237271] qcom-qmp-phy 34000.phy: phy common block init timed-out
[    2.240315] qcom-qmp-phy 34000.phy: qphy initialization failed
...

these are the last lines printed, before rebooting in fastboot and
locking up there (as before[*]).

So, as far as i understand there are two distinct problems:

1) sometimes, qcom-qmp-phy fails to initialize

2) and when that happens, the failure is fatal and it led to a reboot & lockup
in fastboot

1: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
-- 
bye,
p.

Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up

[Niklas Cassel]

On Wed, Jun 12, 2019 at 04:09:11PM +0200, Paolo Pisati wrote:
> On Wed, Jun 12, 2019 at 03:17:35PM +0200, Niklas Cassel wrote:
> > > diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c b/drivers/phy/qualcomm/phy-qcom-qmp.c
> > > index cd91b4179b10..22352e3b0ec5 100644
> > > --- a/drivers/phy/qualcomm/phy-qcom-qmp.c
> > > +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
> > > @@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy)
> > >  
> > >         ret = qcom_qmp_phy_com_init(qphy);
> > >         if (ret)
> > > -               return ret;
> > > +               goto err_lane_rst;
> > >  
> > >         if (cfg->has_lane_rst) {
> > >                 ret = reset_control_deassert(qphy->lane_rst);
> 
> Hi Niklas,
> unfortunately, it didn't help - i added a printk, to highlight when it failed:
> 
> --- a/drivers/phy/qualcomm/phy-qcom-qmp.c
> +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c
> @@ -1489,8 +1489,10 @@ static int qcom_qmp_phy_enable(struct phy *phy)
>         }
>  
>         ret = qcom_qmp_phy_com_init(qphy);
> -       if (ret)
> -               return ret;
> +       if (ret) {
> +               dev_err(qmp->dev, "qphy initialization failed\n");
> +               goto err_lane_rst;
> +       }
>  
>         if (cfg->has_lane_rst) {
>                 ret = reset_control_deassert(qphy->lane_rst);
> 
> After several reboots i was able to trigger the phy init failure again:
> 
> ...
> [    2.223999] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
> [    2.224956] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
> [    2.228798] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
> [    2.237271] qcom-qmp-phy 34000.phy: phy common block init timed-out
> [    2.240315] qcom-qmp-phy 34000.phy: qphy initialization failed
> ...

I still think that the above patch is correct,
even though it didn't fix your problem.

If you try to disable the two other PCIe controllers:

diff --git a/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi b/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi
index 943f69912074..95900fe99f89 100644
--- a/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi
+++ b/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi
@@ -455,12 +455,12 @@
                        };
 
                        pcie@608000 {
-                               status = "okay";
+                               status = "disabled";
                                perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>;
                        };
 
                        pcie@610000 {
-                               status = "okay";
+                               status = "disabled";
                                perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>;
                        };
                };


Can you still reproduce the reboot?


Kind regards,
Niklas

> 
> these are the last lines printed, before rebooting in fastboot and
> locking up there (as before[*]).
> 
> So, as far as i understand there are two distinct problems:
> 
> 1) sometimes, qcom-qmp-phy fails to initialize
> 
> 2) and when that happens, the failure is fatal and it led to a reboot & lockup
> in fastboot
> 
> 1: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
> -- 
> bye,
> p.

Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up

[Paolo Pisati]

On Wed, Jun 12, 2019 at 06:20:48PM +0200, Niklas Cassel wrote:
> 
> Can you still reproduce the reboot?

--- a/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi
+++ b/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi
@@ -447,12 +447,12 @@
                        };
 
                        pcie@608000 {
-                               status = "okay";
+                               status = "disabled";
                                perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>;
                        };
 
                        pcie@610000 {
-                               status = "okay";
+                               status = "disabled";
                                perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>;
                        };
                };

disabling the two other pci controllers (and leaving your previous patch
applied), made the issue more frequent: i was never able to boot to user space
three times in a row without experiencing a crash, previously it was more
sporadic.

On the other hand, now i can reproduce the issue more easily this way.
-- 
bye,
p.

Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up

[Marc Gonzalez]

On 10/06/2019 15:44, Paolo Pisati wrote:

> From time to time, my apq8096-db820c fails to boot to userspace, reset back to
> fastboot and locks up: to easily reproduce the issue, i'm boot looping using a
> cron job with a 1 min reboot entry on the board while leaving a "while 1; do
> fastboot boot boot.img; done" on the host pc.
> 
> The issue is present in mainline up to 5.2-rc4, using defconfig and:
> 
> CONFIG_SCSI_UFS_QCOM=y
> CONFIG_PHY_QCOM_QMP=y
> CONFIG_PHY_QCOM_UFS=y
> 
> but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or
> mainline), where it's even easier to reproduce (e.g. takes way less reboots to
> trigger it).
> 
> These are the last lines printed out:
> ...
> [    7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy
> [    7.448058] qcom-qmp-phy 7410000.phy: Registered Qcom-QMP phy
> [    7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource
> [    7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out
> [    7.538596] phy phy-34000.phy.0: phy init failed --> -110
> [    7.550891] qcom-pcie: probe of 600000.pcie failed with error -110
> [    7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found,
> using dummy regulator
> 
> Log Type: B - Since Boot(Power On Reset),  D - Delta,  S - Statistic
> S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301
> S - IMAGE_VARIANT_STRING=M8996LAB
> S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68
> S - Boot Interface: UFS
> S - Secure Boot: Off
> ...
> 
> Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/
> 
> Any idea what is going on? Am i doing something wrong?

Hmmm... this might be related to:

https://patchwork.kernel.org/patch/10976217/

Regards.