msm8992-lg-bullhead-rev-101 resets in initramfs

msm8992-lg-bullhead-rev-101 resets in initramfs

[Petr Vorel]

Hi all,

[ sorry for sending mail twice, now as plain text. I should move away
from gmail.com to something which allows to use mutt... ]

Some releases after 5.17 bullhead rev 1.01 started to reset.
I'm not sure which kernel version it started, because it might be
caused by kernel config or dts changes.

On current qcom/for-next (e89c75d437ca292d3ef377de341892f571d5d87a) it
resets immediately when loading sdhci controller (before reaching
initramfs).
I tried to disable sdhc1 and whole  rpm_requests in
arch/arm64/boot/dts/qcom/msm8992-lg-bullhead.dtsi, which helped to
reach initramfs (from postmarketOS) and then it got reset in initramfs
during touching network [1] either in setup_usb_network() or later.
NOTE I'm using debug-shell hook [3], which probably calls
setup_usb_network().

I also tried to add tlmm { gpio-reserved-ranges = <85 4>; };
(inspiration from angler), but that obviously didn't help.

Maybe I have something harmful in config [4].

Any idea what could be wrong, what to try?

Kind regards,
Petr

Re: msm8992-lg-bullhead-rev-101 resets in initramfs

[Petr Vorel]

I'm sorry, forget to add links.
Petr

[1] https://gitlab.com/-/snippets/2420133
[2] https://gitlab.com/postmarketOS/pmaports/-/blob/master/main/postmarketos-mkinitfs/init_functions.sh#L514-523
[3] https://gitlab.com/postmarketOS/pmaports/-/blob/master/main/postmarketos-mkinitfs-hook-debug-shell/20-debug-shell.sh
[4] https://gitlab.com/-/snippets/2420128

On Fri, 30 Sept 2022 at 23:30, Petr Vorel <petr.vorel@gmail.com> wrote:
>
> Hi all,
>
> [ sorry for sending mail twice, now as plain text. I should move away
> from gmail.com to something which allows to use mutt... ]
>
> Some releases after 5.17 bullhead rev 1.01 started to reset.
> I'm not sure which kernel version it started, because it might be
> caused by kernel config or dts changes.
>
> On current qcom/for-next (e89c75d437ca292d3ef377de341892f571d5d87a) it
> resets immediately when loading sdhci controller (before reaching
> initramfs).
> I tried to disable sdhc1 and whole  rpm_requests in
> arch/arm64/boot/dts/qcom/msm8992-lg-bullhead.dtsi, which helped to
> reach initramfs (from postmarketOS) and then it got reset in initramfs
> during touching network [1] either in setup_usb_network() or later.
> NOTE I'm using debug-shell hook [3], which probably calls
> setup_usb_network().
>
> I also tried to add tlmm { gpio-reserved-ranges = <85 4>; };
> (inspiration from angler), but that obviously didn't help.
>
> Maybe I have something harmful in config [4].
>
> Any idea what could be wrong, what to try?
>
> Kind regards,
> Petr

Re: msm8992-lg-bullhead-rev-101 resets in initramfs

[Krzysztof Kozlowski]

On 30/09/2022 23:39, Petr Vorel wrote:
> I'm sorry, forget to add links.
> Petr
> 
> [1] https://gitlab.com/-/snippets/2420133
> [2] https://gitlab.com/postmarketOS/pmaports/-/blob/master/main/postmarketos-mkinitfs/init_functions.sh#L514-523
> [3] https://gitlab.com/postmarketOS/pmaports/-/blob/master/main/postmarketos-mkinitfs-hook-debug-shell/20-debug-shell.sh
> [4] https://gitlab.com/-/snippets/2420128
> 
> On Fri, 30 Sept 2022 at 23:30, Petr Vorel <petr.vorel@gmail.com> wrote:
>>
>> Hi all,
>>
>> [ sorry for sending mail twice, now as plain text. I should move away
>> from gmail.com to something which allows to use mutt... ]
>>
>> Some releases after 5.17 bullhead rev 1.01 started to reset.
>> I'm not sure which kernel version it started, because it might be
>> caused by kernel config or dts changes.

I propose to try to bisect it to specific commit in Linux kernel.

Best regards,
Krzysztof

Re: msm8992-lg-bullhead-rev-101 resets in initramfs

[Petr Vorel]

On Sat, 1 Oct 2022 at 10:52, Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 30/09/2022 23:39, Petr Vorel wrote:
> > I'm sorry, forget to add links.
> > Petr
> >
> > [1] https://gitlab.com/-/snippets/2420133
> > [2] https://gitlab.com/postmarketOS/pmaports/-/blob/master/main/postmarketos-mkinitfs/init_functions.sh#L514-523
> > [3] https://gitlab.com/postmarketOS/pmaports/-/blob/master/main/postmarketos-mkinitfs-hook-debug-shell/20-debug-shell.sh
> > [4] https://gitlab.com/-/snippets/2420128
> >
> > On Fri, 30 Sept 2022 at 23:30, Petr Vorel <petr.vorel@gmail.com> wrote:
> >>
> >> Hi all,
> >>
> >> [ sorry for sending mail twice, now as plain text. I should move away
> >> from gmail.com to something which allows to use mutt... ]
> >>
> >> Some releases after 5.17 bullhead rev 1.01 started to reset.
> >> I'm not sure which kernel version it started, because it might be
> >> caused by kernel config or dts changes.
>
> I propose to try to bisect it to specific commit in Linux kernel.
Hi Krzysztof,

thanks for a suggestion, that's what I do for a non-embedded kernel (e.g.
x86_64, ppc64le, or even aarch64 used on server), because these works on
defconfig. But on qcom bullhead and angler devices, last few months I've come
across to few broken boots, although some of them were regressions others
depends a lot on particular config.  And since last 2 or 3 kernel releases it
does not even boot on defconfig. It's quite frustrating to bisect with cross
compilation (takes time) to realize that it boots with custom config. Last time
it took a week and I tried to find what actually caused phone reset,
unsuccessfully. That's why I'm asking before wasting more time.

Of course with no suggestion I there is no other option than to do
bisecting again.


Kind regards,
Petr

> Best regards,
> Krzysztof
>

Re: msm8992-lg-bullhead-rev-101 resets in initramfs

[Krzysztof Kozlowski]

On 01/10/2022 19:10, Petr Vorel wrote:
> On Sat, 1 Oct 2022 at 10:52, Krzysztof Kozlowski
> <krzysztof.kozlowski@linaro.org> wrote:
>>
>> On 30/09/2022 23:39, Petr Vorel wrote:
>>> I'm sorry, forget to add links.
>>> Petr
>>>
>>> [1] https://gitlab.com/-/snippets/2420133
>>> [2] https://gitlab.com/postmarketOS/pmaports/-/blob/master/main/postmarketos-mkinitfs/init_functions.sh#L514-523
>>> [3] https://gitlab.com/postmarketOS/pmaports/-/blob/master/main/postmarketos-mkinitfs-hook-debug-shell/20-debug-shell.sh
>>> [4] https://gitlab.com/-/snippets/2420128
>>>
>>> On Fri, 30 Sept 2022 at 23:30, Petr Vorel <petr.vorel@gmail.com> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> [ sorry for sending mail twice, now as plain text. I should move away
>>>> from gmail.com to something which allows to use mutt... ]
>>>>
>>>> Some releases after 5.17 bullhead rev 1.01 started to reset.
>>>> I'm not sure which kernel version it started, because it might be
>>>> caused by kernel config or dts changes.
>>
>> I propose to try to bisect it to specific commit in Linux kernel.
> Hi Krzysztof,
> 
> thanks for a suggestion, that's what I do for a non-embedded kernel (e.g.
> x86_64, ppc64le, or even aarch64 used on server), because these works on
> defconfig. But on qcom bullhead and angler devices, last few months I've come
> across to few broken boots, although some of them were regressions others
> depends a lot on particular config.  And since last 2 or 3 kernel releases it
> does not even boot on defconfig. It's quite frustrating to bisect with cross
> compilation (takes time) to realize that it boots with custom config. Last time
> it took a week and I tried to find what actually caused phone reset,
> unsuccessfully. That's why I'm asking before wasting more time.
> 
> Of course with no suggestion I there is no other option than to do
> bisecting again.

I understand that false positives/negatives due to config changes are
frustrating, but what stops you from using all the time your custom
config? I was bisecting on embedded devices many times and the only slow
parts in my case was booting device over network and few builds (like 6
out of 12 steps, rest is fast due to ccache). There is basically no
difference (except that booting over network) when bisecting on VM or my
boards. Cross compilation does not take any more time than regular
build, so maybe your setup could be improved?


Best regards,
Krzysztof

Re: msm8992-lg-bullhead-rev-101 resets in initramfs

[Petr Vorel]

On Sun, 2 Oct 2022 at 09:51, Krzysztof Kozlowski
<krzysztof.kozlowski@linaro.org> wrote:
>
> On 01/10/2022 19:10, Petr Vorel wrote:
> > On Sat, 1 Oct 2022 at 10:52, Krzysztof Kozlowski
> > <krzysztof.kozlowski@linaro.org> wrote:
> >>
> >> On 30/09/2022 23:39, Petr Vorel wrote:
> >>> I'm sorry, forget to add links.
> >>> Petr
> >>>
> >>> [1] https://gitlab.com/-/snippets/2420133
> >>> [2] https://gitlab.com/postmarketOS/pmaports/-/blob/master/main/postmarketos-mkinitfs/init_functions.sh#L514-523
> >>> [3] https://gitlab.com/postmarketOS/pmaports/-/blob/master/main/postmarketos-mkinitfs-hook-debug-shell/20-debug-shell.sh
> >>> [4] https://gitlab.com/-/snippets/2420128
> >>>
> >>> On Fri, 30 Sept 2022 at 23:30, Petr Vorel <petr.vorel@gmail.com> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> [ sorry for sending mail twice, now as plain text. I should move away
> >>>> from gmail.com to something which allows to use mutt... ]
> >>>>
> >>>> Some releases after 5.17 bullhead rev 1.01 started to reset.
> >>>> I'm not sure which kernel version it started, because it might be
> >>>> caused by kernel config or dts changes.
> >>
> >> I propose to try to bisect it to specific commit in Linux kernel.
> > Hi Krzysztof,
> >
> > thanks for a suggestion, that's what I do for a non-embedded kernel (e.g.
> > x86_64, ppc64le, or even aarch64 used on server), because these works on
> > defconfig. But on qcom bullhead and angler devices, last few months I've come
> > across to few broken boots, although some of them were regressions others
> > depends a lot on particular config.  And since last 2 or 3 kernel releases it
> > does not even boot on defconfig. It's quite frustrating to bisect with cross
> > compilation (takes time) to realize that it boots with custom config. Last time
> > it took a week and I tried to find what actually caused phone reset,
> > unsuccessfully. That's why I'm asking before wasting more time.
> >
> > Of course with no suggestion I there is no other option than to do
> > bisecting again.
>
> I understand that false positives/negatives due to config changes are
> frustrating, but what stops you from using all the time your custom
> config? I was bisecting on embedded devices many times and the only slow
> parts in my case was booting device over network and few builds (like 6
> out of 12 steps, rest is fast due to ccache). There is basically no
> difference (except that booting over network) when bisecting on VM or my
> boards. Cross compilation does not take any more time than regular
> build, so maybe your setup could be improved?

Thank you for pointing out ccache, I'm using it as well.

Well, last time I wanted to speedup, thus I used custom config (with make
oldconfig) in the middle of bisecting.which was a mistake (obviously it's
important to keep the same config), which in the end took more time than
compiling with the defconfig.  The reason for not using defconfig was first
speedup (disabled unused soc etc).

Other time I found that problematic commit is a merge commit where both children
are merges, which means to test all commits of one merge on the top of the
other merge only to find that it's some kconfig option, but which?

My frustration was also that already working things are broken (probably due
wrong dts, which nobody found), but that's life (testing depends on the users
of particular hw).

OK, lesson learned + I need more patience :). (I've done several bisecting, but
not for embedded, bisecting for some kernel non-drive subsystem is way easier,
one can concentrate on the development).

Kind regards,
Petr


>
>
> Best regards,
> Krzysztof
>

Re: msm8992-lg-bullhead-rev-101 resets in initramfs

[Petr Vorel]

Hi Krzysztof, all,

FYI bisecting showed that reset happens on 86588296acbf ("fdt: Properly handle
"no-map" field in the memory region").  Obviously some memory mapping is wrong,
on angler the same commit also caused reset, some time ago I fixed it in
0e5ded926f2a ("arm64: dts: qcom: msm8994-angler: Disable cont_splash_mem").
Therefore I'll try to find problematic mapping in bullhead dts.

Bullhead also contains oops in most of the commits (no matter if phone
resets during initramfs or keeps running):
[    2.001953] Call trace:
[    2.007244]  clk_branch_wait+0x140/0x160
[    2.009415]  clk_branch2_disable+0x2c/0x40
[    2.013581]  clk_core_disable+0x60/0xc0
[    2.017486]  clk_disable+0x34/0x50
[    2.021219]  dwc3_qcom_probe+0xbec/0xf6c
[    2.024691]  platform_probe+0x68/0xc0
[    2.028772]  really_probe+0xe8/0x4a0
[    2.032330]  driver_probe_device+0x58/0xc0
[    2.035976]  device_driver_attach+0xc0/0xcc
[    2.039883]  __driver_attach+0x84/0x114
[    2.043962]  bus_for_each_dev+0x70/0xd0
[    2.047782]  driver_attach+0x24/0x30
[    2.051600]  bus_add_driver+0x104/0x1ec
[    2.055421]  driver_register+0x78/0x130
[    2.058980]  __platform_driver_register+0x28/0x34
[    2.062801]  dwc3_qcom_driver_init+0x1c/0x28
[    2.067661]  do_one_initcall+0x50/0x1b0
[    2.072000]  kernel_init_freeable+0x1d4/0x23c
[    2.075562]  kernel_init+0x14/0x118
[    2.080073]  ret_from_fork+0x10/0x34
[    2.083379] ---[ end trace fdaf735b91bb478f ]---

If I remember correctly angler didn't have this oops.

Kind regards,
Petr