Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting

Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting

[Bjorn Helgaas]

On Sat, Apr 30, 2022 at 2:53 PM <bugzilla-daemon@kernel.org> wrote:
>
> https://bugzilla.kernel.org/show_bug.cgi?id=215925
>
>             Bug ID: 215925
>            Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4)
>                     breaks booting
>            Product: Drivers
>            Version: 2.5
>     Kernel Version: v5.17-rc1
>           Hardware: ARM
>                 OS: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: PCI
>           Assignee: drivers_pci@kernel-bugs.osdl.org
>           Reporter: kibi@debian.org
>         Regression: No
>
> Catching up with latest kernel releases in Debian, it turned out that my
> Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO
> Board,
> and booting from an SD card, no longer boots: this means a black screen on the
> HDMI output, and no output on the serial console.
>
> Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the
> first (pre)release that wasn't.
>
> After some git bisect, it turns out the cause seems to be the following commit
> (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):
>
> ```
> commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21
> Author: Jim Quinlan <jim2101024@gmail.com>
> Date:   Thu Jan 6 11:03:27 2022 -0500
>
>     PCI: brcmstb: Split brcm_pcie_setup() into two funcs
> ```
>
> Starting with this commit, the kernel panics early (before 0.30 seconds), with
> an `Asynchronous SError Interrupt`. The backtrace references various
> `brcm_pcie_*` functions; I can share a picture or try and transcribe it
> manually if that helps (nothing on the serial console…).
>
> This commit is part of a branch that was ultimately merged as
> d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's
> not even a backtrace anymore, the screen stays black after the usual “boot-up
> rainbow”, and there's still nothing on the serial console.
>
> I confirmed that 88db8458086b1dcf20b56682504bdb34d2bca0e2 (on the master side)
> was still booting properly, and that 87c71931633bd15e9cfd51d4a4d9cd685e8cdb55
> (from the branch being merged into master) is the last commit showing the
> panic.
>
> Since d0a231f01e5b25bacd23e6edc7c979a18a517b2b is a merge commit that includes
> conflict resolutions in drivers/pci/controller/pcie-brcmstb.c, I suppose this
> could be consistent with the initial panic being “upgraded” into an even more
> serious issue.
>
> I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still
> affected by this issue.
>
> The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the
> exact same image on the same SD card (with latest master) boots fine on it.

Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting

[Thorsten Leemhuis]

Hi, this is your Linux kernel regression tracker. Partly top-posting to
mnake this easily accessible.

Jim, what's up here? The regression was reported more than a week ago
and it seems nothing happened since then. Or was there progress and I
just missed it?

Anyway:

[TLDR: I'm adding this regression report to the list of tracked
regressions; all text from me you find below is based on a few templates
paragraphs you might have encountered already already in similar form.]

On 02.05.22 20:38, Bjorn Helgaas wrote:
> On Sat, Apr 30, 2022 at 2:53 PM <bugzilla-daemon@kernel.org> wrote:
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=215925
>>
>>             Bug ID: 215925
>>            Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4)
>>                     breaks booting
>>            Product: Drivers
>>            Version: 2.5
>>     Kernel Version: v5.17-rc1
>>           Hardware: ARM
>>                 OS: Linux
>>               Tree: Mainline
>>             Status: NEW
>>           Severity: normal
>>           Priority: P1
>>          Component: PCI
>>           Assignee: drivers_pci@kernel-bugs.osdl.org
>>           Reporter: kibi@debian.org
>>         Regression: No
>>
>> Catching up with latest kernel releases in Debian, it turned out that my
>> Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO
>> Board,
>> and booting from an SD card, no longer boots: this means a black screen on the
>> HDMI output, and no output on the serial console.
>>
>> Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the
>> first (pre)release that wasn't.
>>
>> After some git bisect, it turns out the cause seems to be the following commit
>> (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):
>>
>> ```
>> commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21
>> Author: Jim Quinlan <jim2101024@gmail.com>
>> Date:   Thu Jan 6 11:03:27 2022 -0500
>>
>>     PCI: brcmstb: Split brcm_pcie_setup() into two funcs
>> ```
>>
>> Starting with this commit, the kernel panics early (before 0.30 seconds), with
>> an `Asynchronous SError Interrupt`. The backtrace references various
>> `brcm_pcie_*` functions; I can share a picture or try and transcribe it
>> manually if that helps (nothing on the serial console…).
>>
>> This commit is part of a branch that was ultimately merged as
>> d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's
>> not even a backtrace anymore, the screen stays black after the usual “boot-up
>> rainbow”, and there's still nothing on the serial console.
>>
>> I confirmed that 88db8458086b1dcf20b56682504bdb34d2bca0e2 (on the master side)
>> was still booting properly, and that 87c71931633bd15e9cfd51d4a4d9cd685e8cdb55
>> (from the branch being merged into master) is the last commit showing the
>> panic.
>>
>> Since d0a231f01e5b25bacd23e6edc7c979a18a517b2b is a merge commit that includes
>> conflict resolutions in drivers/pci/controller/pcie-brcmstb.c, I suppose this
>> could be consistent with the initial panic being “upgraded” into an even more
>> serious issue.
>>
>> I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still
>> affected by this issue.
>>
>> The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the
>> exact same image on the same SD card (with latest master) boots fine on it.

CCing the regression mailing list, as it should be in the loop for all
regressions, as explained here:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html

To be sure below issue doesn't fall through the cracks unnoticed, I'm
adding it to regzbot, my Linux kernel regression tracking bot:

#regzbot ^introduced 830aa6f29f07a4e2f1a
#regzbot title pci: brcmstb: CM4 no longer boots from SD card
#regzbot ignore-activity
#regzbot from: Cyril Brulebois <kibi@debian.org>
#regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=215925

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply -- ideally with also
telling regzbot about it, as explained here:
https://linux-regtracking.leemhuis.info/tracked-regression/

Reminder for developers: When fixing the issue, add 'Link:' tags
pointing to the report (the mail this one replied to), as the kernel's
documentation call for; above page explains why this is important for
tracked regressions.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting

[Bjorn Helgaas]

On Mon, May 09, 2022 at 09:44:29AM +0200, Thorsten Leemhuis wrote:
> Hi, this is your Linux kernel regression tracker. Partly top-posting to
> mnake this easily accessible.
> 
> Jim, what's up here? The regression was reported more than a week ago
> and it seems nothing happened since then. Or was there progress and I
> just missed it?
> 
> Anyway:
> 
> [TLDR: I'm adding this regression report to the list of tracked
> regressions; all text from me you find below is based on a few templates
> paragraphs you might have encountered already already in similar form.]
> 
> On 02.05.22 20:38, Bjorn Helgaas wrote:
> > On Sat, Apr 30, 2022 at 2:53 PM <bugzilla-daemon@kernel.org> wrote:
> >>
> >> https://bugzilla.kernel.org/show_bug.cgi?id=215925
> >>
> >>             Bug ID: 215925
> >>            Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4)
> >>                     breaks booting
> >>            Product: Drivers
> >>            Version: 2.5
> >>     Kernel Version: v5.17-rc1
> >>           Hardware: ARM
> >>                 OS: Linux
> >>               Tree: Mainline
> >>             Status: NEW
> >>           Severity: normal
> >>           Priority: P1
> >>          Component: PCI
> >>           Assignee: drivers_pci@kernel-bugs.osdl.org
> >>           Reporter: kibi@debian.org
> >>         Regression: No
> >>
> >> Catching up with latest kernel releases in Debian, it turned out that my
> >> Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO
> >> Board,
> >> and booting from an SD card, no longer boots: this means a black screen on the
> >> HDMI output, and no output on the serial console.
> >>
> >> Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the
> >> first (pre)release that wasn't.
> >>
> >> After some git bisect, it turns out the cause seems to be the following commit
> >> (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):
> >>
> >> ```
> >> commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21
> >> Author: Jim Quinlan <jim2101024@gmail.com>
> >> Date:   Thu Jan 6 11:03:27 2022 -0500
> >>
> >>     PCI: brcmstb: Split brcm_pcie_setup() into two funcs
> >> ```
> >>
> >> Starting with this commit, the kernel panics early (before 0.30 seconds), with
> >> an `Asynchronous SError Interrupt`. The backtrace references various
> >> `brcm_pcie_*` functions; I can share a picture or try and transcribe it
> >> manually if that helps (nothing on the serial console…).
> >>
> >> This commit is part of a branch that was ultimately merged as
> >> d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's
> >> not even a backtrace anymore, the screen stays black after the usual “boot-up
> >> rainbow”, and there's still nothing on the serial console.
> >>
> >> I confirmed that 88db8458086b1dcf20b56682504bdb34d2bca0e2 (on the master side)
> >> was still booting properly, and that 87c71931633bd15e9cfd51d4a4d9cd685e8cdb55
> >> (from the branch being merged into master) is the last commit showing the
> >> panic.
> >>
> >> Since d0a231f01e5b25bacd23e6edc7c979a18a517b2b is a merge commit that includes
> >> conflict resolutions in drivers/pci/controller/pcie-brcmstb.c, I suppose this
> >> could be consistent with the initial panic being “upgraded” into an even more
> >> serious issue.
> >>
> >> I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still
> >> affected by this issue.
> >>
> >> The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the
> >> exact same image on the same SD card (with latest master) boots fine on it.

Cyril, 830aa6f29f07 ("PCI: brcmstb: Split brcm_pcie_setup() into two
funcs") reverts cleanly as of 57ae8a492116.  Does reverting it avoid
the regression?

Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting

[Cyril Brulebois]

[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]

Hi Bjorn,

Bjorn Helgaas <helgaas@kernel.org> (2022-05-09):
> Cyril, 830aa6f29f07 ("PCI: brcmstb: Split brcm_pcie_setup() into two
> funcs") reverts cleanly as of 57ae8a492116.  Does reverting it avoid
> the regression?

I didn't even try and revert this commit before you suggested doing so
since it was supposed to be some preliminary work. Quoting a part of
its commit message:

    In future commits the brcm_pcie_linkup() function will be called
    indirectly by pci_host_probe() as opposed to the host driver
    invoking it directly.


Anyway, the patch can indeed be reverted on top of v5.18-rc4 or
v5.18-rc6 but the build fails due to the former function being removed,
while being still called from other places:

      CC      drivers/pci/controller/pcie-brcmstb.o
    drivers/pci/controller/pcie-brcmstb.c:199:12: warning: ‘brcm_pcie_linkup’ used but never defined
      199 | static int brcm_pcie_linkup(struct brcm_pcie *pcie);
          |            ^~~~~~~~~~~~~~~~
    …
    aarch64-linux-gnu-ld: drivers/pci/controller/pcie-brcmstb.o: in function `brcm_pcie_add_bus':
    /home/kibi/hack/linux.git/drivers/pci/controller/pcie-brcmstb.c:527: undefined reference to `brcm_pcie_linkup'


See for example:

    commit 93e41f3fca3d4a0f927b784012338c37f80a8a80
    Author: Jim Quinlan <jim2101024@gmail.com>
    Date:   Thu Jan 6 11:03:29 2022 -0500
    
        PCI: brcmstb: Add control of subdevice voltage regulators

(And that one cannot be trivially reverted.)


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting

[Bjorn Helgaas]

On Mon, May 09, 2022 at 07:45:27PM +0200, Cyril Brulebois wrote:
> Hi Bjorn,
> 
> Bjorn Helgaas <helgaas@kernel.org> (2022-05-09):
> > Cyril, 830aa6f29f07 ("PCI: brcmstb: Split brcm_pcie_setup() into two
> > funcs") reverts cleanly as of 57ae8a492116.  Does reverting it avoid
> > the regression?
> 
> I didn't even try and revert this commit before you suggested doing so
> since it was supposed to be some preliminary work. Quoting a part of
> its commit message:
> 
>     In future commits the brcm_pcie_linkup() function will be called
>     indirectly by pci_host_probe() as opposed to the host driver
>     invoking it directly.
> 
> 
> Anyway, the patch can indeed be reverted on top of v5.18-rc4 or
> v5.18-rc6 but the build fails due to the former function being removed,
> while being still called from other places:
> 
>       CC      drivers/pci/controller/pcie-brcmstb.o
>     drivers/pci/controller/pcie-brcmstb.c:199:12: warning: ‘brcm_pcie_linkup’ used but never defined
>       199 | static int brcm_pcie_linkup(struct brcm_pcie *pcie);
>           |            ^~~~~~~~~~~~~~~~
>     …
>     aarch64-linux-gnu-ld: drivers/pci/controller/pcie-brcmstb.o: in function `brcm_pcie_add_bus':
>     /home/kibi/hack/linux.git/drivers/pci/controller/pcie-brcmstb.c:527: undefined reference to `brcm_pcie_linkup'
> 
> 
> See for example:
> 
>     commit 93e41f3fca3d4a0f927b784012338c37f80a8a80
>     Author: Jim Quinlan <jim2101024@gmail.com>
>     Date:   Thu Jan 6 11:03:29 2022 -0500
>     
>         PCI: brcmstb: Add control of subdevice voltage regulators
> 
> (And that one cannot be trivially reverted.)

What if you revert 830aa6f29f07 and the subsequent brcmstb patches?

  11ed8b8624b8 ("PCI: brcmstb: Do not turn off WOL regulators on suspend")
  93e41f3fca3d ("PCI: brcmstb: Add control of subdevice voltage regulators")
  67211aadcb4b ("PCI: brcmstb: Add mechanism to turn on subdev regulators")
  830aa6f29f07 ("PCI: brcmstb: Split brcm_pcie_setup() into two funcs")

  $ git revert 11ed8b8624b8 93e41f3fca3d 67211aadcb4b 830aa6f29f07

I did that on current upstream: 9be9ed2612b5 ("Merge tag
'platform-drivers-x86-v5.18-4' of
git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86")
and it built fine on x86.

Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting

[Cyril Brulebois]

[-- Attachment #1: Type: text/plain, Size: 2944 bytes --]

Bjorn Helgaas <helgaas@kernel.org> (2022-05-10):
> What if you revert 830aa6f29f07 and the subsequent brcmstb patches?
> 
>   11ed8b8624b8 ("PCI: brcmstb: Do not turn off WOL regulators on suspend")
>   93e41f3fca3d ("PCI: brcmstb: Add control of subdevice voltage regulators")
>   67211aadcb4b ("PCI: brcmstb: Add mechanism to turn on subdev regulators")
>   830aa6f29f07 ("PCI: brcmstb: Split brcm_pcie_setup() into two funcs")
> 
>   $ git revert 11ed8b8624b8 93e41f3fca3d 67211aadcb4b 830aa6f29f07
> 
> I did that on current upstream: 9be9ed2612b5 ("Merge tag
> 'platform-drivers-x86-v5.18-4' of
> git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86")
> and it built fine on x86.

I've done exactly that, and it boots again. Comparing kernel messages
against an older version (5.10.106), I'm getting the same output on that
bare CM4 on CM4 IO Board setup:

    # dmesg|grep -i pcie
    [    3.368620] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
    [    3.368654] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
    [    3.368703] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
    [    3.368748] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x003fffffff -> 0x0400000000
    [    3.421094] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
    [    3.421341] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00

And with a PCIe → quad-USB board plugged in, I'm getting those
additional lines:

    [    3.426842] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
    [    3.427072] pcieport 0000:00:00.0: PME: Signaling with IRQ 51
    [    3.427472] pcieport 0000:00:00.0: AER: enabled with IRQ 51

(It seems to be consistently IRQ 51 with 5.18.0-rc6+ while it seems to
be consistently IRQ 52 on 5.10.106, but the output is very similar in
both cases.)

And plugging a keyboard on one of those USB ports works fine:

    [   13.406351] input: Logitech USB Keyboard as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/0003:046D:C31C.0001/input/input0
    [   13.510144] input: Logitech USB Keyboard Consumer Control as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.1/0003:046D:C31C.0002/input/input2
    [   13.591345] input: Logitech USB Keyboard System Control as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.1/0003:046D:C31C.0002/input/input3


Wrapping up: it boots again (with or without PCIe equipment plugged in),
and PCIe seems functional.

I'm happy to test more patches (e.g. trying to fix the actual issue
without going for a set of reverts).


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting

[Bjorn Helgaas]

[+to Jim, heads-up about reverting commits to avoid the regression]

On Tue, May 10, 2022 at 10:07:09PM +0200, Cyril Brulebois wrote:
> Bjorn Helgaas <helgaas@kernel.org> (2022-05-10):
> > What if you revert 830aa6f29f07 and the subsequent brcmstb patches?
> > 
> >   11ed8b8624b8 ("PCI: brcmstb: Do not turn off WOL regulators on suspend")
> >   93e41f3fca3d ("PCI: brcmstb: Add control of subdevice voltage regulators")
> >   67211aadcb4b ("PCI: brcmstb: Add mechanism to turn on subdev regulators")
> >   830aa6f29f07 ("PCI: brcmstb: Split brcm_pcie_setup() into two funcs")
> > 
> >   $ git revert 11ed8b8624b8 93e41f3fca3d 67211aadcb4b 830aa6f29f07
> > 
> > I did that on current upstream: 9be9ed2612b5 ("Merge tag
> > 'platform-drivers-x86-v5.18-4' of
> > git://git.kernel.org/pub/scm/linux/kernel/git/pdx86/platform-drivers-x86")
> > and it built fine on x86.
> 
> I've done exactly that, and it boots again. Comparing kernel messages
> against an older version (5.10.106), I'm getting the same output on that
> bare CM4 on CM4 IO Board setup:
> 
>     # dmesg|grep -i pcie
>     [    3.368620] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
>     [    3.368654] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
>     [    3.368703] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
>     [    3.368748] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x003fffffff -> 0x0400000000
>     [    3.421094] brcm-pcie fd500000.pcie: link up, 5.0 GT/s PCIe x1 (SSC)
>     [    3.421341] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
> 
> And with a PCIe → quad-USB board plugged in, I'm getting those
> additional lines:
> 
>     [    3.426842] pcieport 0000:00:00.0: enabling device (0000 -> 0002)
>     [    3.427072] pcieport 0000:00:00.0: PME: Signaling with IRQ 51
>     [    3.427472] pcieport 0000:00:00.0: AER: enabled with IRQ 51
> 
> (It seems to be consistently IRQ 51 with 5.18.0-rc6+ while it seems to
> be consistently IRQ 52 on 5.10.106, but the output is very similar in
> both cases.)
> 
> And plugging a keyboard on one of those USB ports works fine:
> 
>     [   13.406351] input: Logitech USB Keyboard as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.0/0003:046D:C31C.0001/input/input0
>     [   13.510144] input: Logitech USB Keyboard Consumer Control as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.1/0003:046D:C31C.0002/input/input2
>     [   13.591345] input: Logitech USB Keyboard System Control as /devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.2/1-1.2:1.1/0003:046D:C31C.0002/input/input3
> 
> 
> Wrapping up: it boots again (with or without PCIe equipment plugged in),
> and PCIe seems functional.
> 
> I'm happy to test more patches (e.g. trying to fix the actual issue
> without going for a set of reverts).

We're at -rc6 already, but no word from Jim yet.  I think all we can
do at this point is queue up reverts of these patches for v5.18.  The
reverts on now on my for-linus branch.

If we get a fix, I can easily drop the reverts, but I don't want v5.18
to release with PCI completely broken on the Compute Module 4.

Bjorn

Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting

[Jim Quinlan]

Hi Bjorn, Thorsten,

I apologize -- I did not see this email until now; I think I have to
work on my gmail filters and labels.

I've just made a post on the Bugzilla website regarding this
regression and have ideas on what may be causing the problem.
Unfortunately, the error cannot be reproduced on my RPi4 or Broadcom
STB version of the 2711.
Hopefully Cyril can help me identify the issue.

I will try to get a Fixup ASAP.

Regards,
Jim Quinlan
Broadcom STB

On Mon, May 9, 2022 at 3:44 AM Thorsten Leemhuis
<regressions@leemhuis.info> wrote:
>
> Hi, this is your Linux kernel regression tracker. Partly top-posting to
> mnake this easily accessible.
>
> Jim, what's up here? The regression was reported more than a week ago
> and it seems nothing happened since then. Or was there progress and I
> just missed it?
>
> Anyway:
>
> [TLDR: I'm adding this regression report to the list of tracked
> regressions; all text from me you find below is based on a few templates
> paragraphs you might have encountered already already in similar form.]
>
> On 02.05.22 20:38, Bjorn Helgaas wrote:
> > On Sat, Apr 30, 2022 at 2:53 PM <bugzilla-daemon@kernel.org> wrote:
> >>
> >> https://bugzilla.kernel.org/show_bug.cgi?id=215925
> >>
> >>             Bug ID: 215925
> >>            Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4)
> >>                     breaks booting
> >>            Product: Drivers
> >>            Version: 2.5
> >>     Kernel Version: v5.17-rc1
> >>           Hardware: ARM
> >>                 OS: Linux
> >>               Tree: Mainline
> >>             Status: NEW
> >>           Severity: normal
> >>           Priority: P1
> >>          Component: PCI
> >>           Assignee: drivers_pci@kernel-bugs.osdl.org
> >>           Reporter: kibi@debian.org
> >>         Regression: No
> >>
> >> Catching up with latest kernel releases in Debian, it turned out that my
> >> Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO
> >> Board,
> >> and booting from an SD card, no longer boots: this means a black screen on the
> >> HDMI output, and no output on the serial console.
> >>
> >> Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the
> >> first (pre)release that wasn't.
> >>
> >> After some git bisect, it turns out the cause seems to be the following commit
> >> (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):
> >>
> >> ```
> >> commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21
> >> Author: Jim Quinlan <jim2101024@gmail.com>
> >> Date:   Thu Jan 6 11:03:27 2022 -0500
> >>
> >>     PCI: brcmstb: Split brcm_pcie_setup() into two funcs
> >> ```
> >>
> >> Starting with this commit, the kernel panics early (before 0.30 seconds), with
> >> an `Asynchronous SError Interrupt`. The backtrace references various
> >> `brcm_pcie_*` functions; I can share a picture or try and transcribe it
> >> manually if that helps (nothing on the serial console…).
> >>
> >> This commit is part of a branch that was ultimately merged as
> >> d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's
> >> not even a backtrace anymore, the screen stays black after the usual “boot-up
> >> rainbow”, and there's still nothing on the serial console.
> >>
> >> I confirmed that 88db8458086b1dcf20b56682504bdb34d2bca0e2 (on the master side)
> >> was still booting properly, and that 87c71931633bd15e9cfd51d4a4d9cd685e8cdb55
> >> (from the branch being merged into master) is the last commit showing the
> >> panic.
> >>
> >> Since d0a231f01e5b25bacd23e6edc7c979a18a517b2b is a merge commit that includes
> >> conflict resolutions in drivers/pci/controller/pcie-brcmstb.c, I suppose this
> >> could be consistent with the initial panic being “upgraded” into an even more
> >> serious issue.
> >>
> >> I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still
> >> affected by this issue.
> >>
> >> The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the
> >> exact same image on the same SD card (with latest master) boots fine on it.
>
> CCing the regression mailing list, as it should be in the loop for all
> regressions, as explained here:
> https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
>
> To be sure below issue doesn't fall through the cracks unnoticed, I'm
> adding it to regzbot, my Linux kernel regression tracking bot:
>
> #regzbot ^introduced 830aa6f29f07a4e2f1a
> #regzbot title pci: brcmstb: CM4 no longer boots from SD card
> #regzbot ignore-activity
> #regzbot from: Cyril Brulebois <kibi@debian.org>
> #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=215925
>
> This isn't a regression? This issue or a fix for it are already
> discussed somewhere else? It was fixed already? You want to clarify when
> the regression started to happen? Or point out I got the title or
> something else totally wrong? Then just reply -- ideally with also
> telling regzbot about it, as explained here:
> https://linux-regtracking.leemhuis.info/tracked-regression/
>
> Reminder for developers: When fixing the issue, add 'Link:' tags
> pointing to the report (the mail this one replied to), as the kernel's
> documentation call for; above page explains why this is important for
> tracked regressions.
>
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
>
> P.S.: As the Linux kernel's regression tracker I deal with a lot of
> reports and sometimes miss something important when writing mails like
> this. If that's the case here, don't hesitate to tell me in a public
> reply, it's in everyone's interest to set the public record straight.

Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting

[Jim Quinlan]

Hello,
I have just sent a pullrequest to linu-pci@vger@kernel.org to address
this regression.
Please let me know if I have to do anything else besides addressing
reviewers concerns
for this pullreq.
Thanks,
Jim Quinlan
Broadcom STB



On Mon, May 16, 2022 at 5:05 PM Jim Quinlan <jim2101024@gmail.com> wrote:
>
> Hi Bjorn, Thorsten,
>
> I apologize -- I did not see this email until now; I think I have to
> work on my gmail filters and labels.
>
> I've just made a post on the Bugzilla website regarding this
> regression and have ideas on what may be causing the problem.
> Unfortunately, the error cannot be reproduced on my RPi4 or Broadcom
> STB version of the 2711.
> Hopefully Cyril can help me identify the issue.
>
> I will try to get a Fixup ASAP.
>
> Regards,
> Jim Quinlan
> Broadcom STB
>
> On Mon, May 9, 2022 at 3:44 AM Thorsten Leemhuis
> <regressions@leemhuis.info> wrote:
> >
> > Hi, this is your Linux kernel regression tracker. Partly top-posting to
> > mnake this easily accessible.
> >
> > Jim, what's up here? The regression was reported more than a week ago
> > and it seems nothing happened since then. Or was there progress and I
> > just missed it?
> >
> > Anyway:
> >
> > [TLDR: I'm adding this regression report to the list of tracked
> > regressions; all text from me you find below is based on a few templates
> > paragraphs you might have encountered already already in similar form.]
> >
> > On 02.05.22 20:38, Bjorn Helgaas wrote:
> > > On Sat, Apr 30, 2022 at 2:53 PM <bugzilla-daemon@kernel.org> wrote:
> > >>
> > >> https://bugzilla.kernel.org/show_bug.cgi?id=215925
> > >>
> > >>             Bug ID: 215925
> > >>            Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4)
> > >>                     breaks booting
> > >>            Product: Drivers
> > >>            Version: 2.5
> > >>     Kernel Version: v5.17-rc1
> > >>           Hardware: ARM
> > >>                 OS: Linux
> > >>               Tree: Mainline
> > >>             Status: NEW
> > >>           Severity: normal
> > >>           Priority: P1
> > >>          Component: PCI
> > >>           Assignee: drivers_pci@kernel-bugs.osdl.org
> > >>           Reporter: kibi@debian.org
> > >>         Regression: No
> > >>
> > >> Catching up with latest kernel releases in Debian, it turned out that my
> > >> Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO
> > >> Board,
> > >> and booting from an SD card, no longer boots: this means a black screen on the
> > >> HDMI output, and no output on the serial console.
> > >>
> > >> Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the
> > >> first (pre)release that wasn't.
> > >>
> > >> After some git bisect, it turns out the cause seems to be the following commit
> > >> (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):
> > >>
> > >> ```
> > >> commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21
> > >> Author: Jim Quinlan <jim2101024@gmail.com>
> > >> Date:   Thu Jan 6 11:03:27 2022 -0500
> > >>
> > >>     PCI: brcmstb: Split brcm_pcie_setup() into two funcs
> > >> ```
> > >>
> > >> Starting with this commit, the kernel panics early (before 0.30 seconds), with
> > >> an `Asynchronous SError Interrupt`. The backtrace references various
> > >> `brcm_pcie_*` functions; I can share a picture or try and transcribe it
> > >> manually if that helps (nothing on the serial console…).
> > >>
> > >> This commit is part of a branch that was ultimately merged as
> > >> d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's
> > >> not even a backtrace anymore, the screen stays black after the usual “boot-up
> > >> rainbow”, and there's still nothing on the serial console.
> > >>
> > >> I confirmed that 88db8458086b1dcf20b56682504bdb34d2bca0e2 (on the master side)
> > >> was still booting properly, and that 87c71931633bd15e9cfd51d4a4d9cd685e8cdb55
> > >> (from the branch being merged into master) is the last commit showing the
> > >> panic.
> > >>
> > >> Since d0a231f01e5b25bacd23e6edc7c979a18a517b2b is a merge commit that includes
> > >> conflict resolutions in drivers/pci/controller/pcie-brcmstb.c, I suppose this
> > >> could be consistent with the initial panic being “upgraded” into an even more
> > >> serious issue.
> > >>
> > >> I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still
> > >> affected by this issue.
> > >>
> > >> The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the
> > >> exact same image on the same SD card (with latest master) boots fine on it.
> >
> > CCing the regression mailing list, as it should be in the loop for all
> > regressions, as explained here:
> > https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
> >
> > To be sure below issue doesn't fall through the cracks unnoticed, I'm
> > adding it to regzbot, my Linux kernel regression tracking bot:
> >
> > #regzbot ^introduced 830aa6f29f07a4e2f1a
> > #regzbot title pci: brcmstb: CM4 no longer boots from SD card
> > #regzbot ignore-activity
> > #regzbot from: Cyril Brulebois <kibi@debian.org>
> > #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=215925
> >
> > This isn't a regression? This issue or a fix for it are already
> > discussed somewhere else? It was fixed already? You want to clarify when
> > the regression started to happen? Or point out I got the title or
> > something else totally wrong? Then just reply -- ideally with also
> > telling regzbot about it, as explained here:
> > https://linux-regtracking.leemhuis.info/tracked-regression/
> >
> > Reminder for developers: When fixing the issue, add 'Link:' tags
> > pointing to the report (the mail this one replied to), as the kernel's
> > documentation call for; above page explains why this is important for
> > tracked regressions.
> >
> > Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> >
> > P.S.: As the Linux kernel's regression tracker I deal with a lot of
> > reports and sometimes miss something important when writing mails like
> > this. If that's the case here, don't hesitate to tell me in a public
> > reply, it's in everyone's interest to set the public record straight.

Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting

[Thorsten Leemhuis]

On 18.05.22 21:47, Jim Quinlan wrote:

> I have just sent a pullrequest to linu-pci@vger@kernel.org to address
> this regression.

Great, thx

> Please let me know if I have to do anything else besides addressing
> reviewers concerns
> for this pullreq.

No need to, everything fine. I just have to tell regzbot manually about
the patch (and the commit-id once it landed), as it doesn't handle links
to bugzilla ticket to well yet, but that hopefully should be history in
a week or two (or three).

#regzbot monitor:
https://lore.kernel.org/linux-pci/CANCKTBuz9FG-aGuwM-thFgkAyTi480=1j78DpjWUkgU4UDuurg@mail.gmail.com/

Ciao, Thorsten

> On Mon, May 16, 2022 at 5:05 PM Jim Quinlan <jim2101024@gmail.com> wrote:
>>
>> Hi Bjorn, Thorsten,
>>
>> I apologize -- I did not see this email until now; I think I have to
>> work on my gmail filters and labels.
>>
>> I've just made a post on the Bugzilla website regarding this
>> regression and have ideas on what may be causing the problem.
>> Unfortunately, the error cannot be reproduced on my RPi4 or Broadcom
>> STB version of the 2711.
>> Hopefully Cyril can help me identify the issue.
>>
>> I will try to get a Fixup ASAP.
>>
>> Regards,
>> Jim Quinlan
>> Broadcom STB
>>
>> On Mon, May 9, 2022 at 3:44 AM Thorsten Leemhuis
>> <regressions@leemhuis.info> wrote:
>>>
>>> Hi, this is your Linux kernel regression tracker. Partly top-posting to
>>> mnake this easily accessible.
>>>
>>> Jim, what's up here? The regression was reported more than a week ago
>>> and it seems nothing happened since then. Or was there progress and I
>>> just missed it?
>>>
>>> Anyway:
>>>
>>> [TLDR: I'm adding this regression report to the list of tracked
>>> regressions; all text from me you find below is based on a few templates
>>> paragraphs you might have encountered already already in similar form.]
>>>
>>> On 02.05.22 20:38, Bjorn Helgaas wrote:
>>>> On Sat, Apr 30, 2022 at 2:53 PM <bugzilla-daemon@kernel.org> wrote:
>>>>>
>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=215925
>>>>>
>>>>>             Bug ID: 215925
>>>>>            Summary: PCIe regression on Raspberry Pi Compute Module 4 (CM4)
>>>>>                     breaks booting
>>>>>            Product: Drivers
>>>>>            Version: 2.5
>>>>>     Kernel Version: v5.17-rc1
>>>>>           Hardware: ARM
>>>>>                 OS: Linux
>>>>>               Tree: Mainline
>>>>>             Status: NEW
>>>>>           Severity: normal
>>>>>           Priority: P1
>>>>>          Component: PCI
>>>>>           Assignee: drivers_pci@kernel-bugs.osdl.org
>>>>>           Reporter: kibi@debian.org
>>>>>         Regression: No
>>>>>
>>>>> Catching up with latest kernel releases in Debian, it turned out that my
>>>>> Raspberry Pi Compute Module 4, mounted on an official Compute Module 4 IO
>>>>> Board,
>>>>> and booting from an SD card, no longer boots: this means a black screen on the
>>>>> HDMI output, and no output on the serial console.
>>>>>
>>>>> Trying various releases, I confirmed that v5.16 was fine, and v5.17-rc1 was the
>>>>> first (pre)release that wasn't.
>>>>>
>>>>> After some git bisect, it turns out the cause seems to be the following commit
>>>>> (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21):
>>>>>
>>>>> ```
>>>>> commit 830aa6f29f07a4e2f1a947dfa72b3ccddb46dd21
>>>>> Author: Jim Quinlan <jim2101024@gmail.com>
>>>>> Date:   Thu Jan 6 11:03:27 2022 -0500
>>>>>
>>>>>     PCI: brcmstb: Split brcm_pcie_setup() into two funcs
>>>>> ```
>>>>>
>>>>> Starting with this commit, the kernel panics early (before 0.30 seconds), with
>>>>> an `Asynchronous SError Interrupt`. The backtrace references various
>>>>> `brcm_pcie_*` functions; I can share a picture or try and transcribe it
>>>>> manually if that helps (nothing on the serial console…).
>>>>>
>>>>> This commit is part of a branch that was ultimately merged as
>>>>> d0a231f01e5b25bacd23e6edc7c979a18a517b2b; starting with this commit, there's
>>>>> not even a backtrace anymore, the screen stays black after the usual “boot-up
>>>>> rainbow”, and there's still nothing on the serial console.
>>>>>
>>>>> I confirmed that 88db8458086b1dcf20b56682504bdb34d2bca0e2 (on the master side)
>>>>> was still booting properly, and that 87c71931633bd15e9cfd51d4a4d9cd685e8cdb55
>>>>> (from the branch being merged into master) is the last commit showing the
>>>>> panic.
>>>>>
>>>>> Since d0a231f01e5b25bacd23e6edc7c979a18a517b2b is a merge commit that includes
>>>>> conflict resolutions in drivers/pci/controller/pcie-brcmstb.c, I suppose this
>>>>> could be consistent with the initial panic being “upgraded” into an even more
>>>>> serious issue.
>>>>>
>>>>> I've also verified that latest master (v5.18-rc4-396-g57ae8a492116) is still
>>>>> affected by this issue.
>>>>>
>>>>> The regular Raspberry Pi 4 B doesn't seem to be affected by this issue: the
>>>>> exact same image on the same SD card (with latest master) boots fine on it.
>>>
>>> CCing the regression mailing list, as it should be in the loop for all
>>> regressions, as explained here:
>>> https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
>>>
>>> To be sure below issue doesn't fall through the cracks unnoticed, I'm
>>> adding it to regzbot, my Linux kernel regression tracking bot:
>>>
>>> #regzbot ^introduced 830aa6f29f07a4e2f1a
>>> #regzbot title pci: brcmstb: CM4 no longer boots from SD card
>>> #regzbot ignore-activity
>>> #regzbot from: Cyril Brulebois <kibi@debian.org>
>>> #regzbot link: https://bugzilla.kernel.org/show_bug.cgi?id=215925
>>>
>>> This isn't a regression? This issue or a fix for it are already
>>> discussed somewhere else? It was fixed already? You want to clarify when
>>> the regression started to happen? Or point out I got the title or
>>> something else totally wrong? Then just reply -- ideally with also
>>> telling regzbot about it, as explained here:
>>> https://linux-regtracking.leemhuis.info/tracked-regression/
>>>
>>> Reminder for developers: When fixing the issue, add 'Link:' tags
>>> pointing to the report (the mail this one replied to), as the kernel's
>>> documentation call for; above page explains why this is important for
>>> tracked regressions.
>>>
>>> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
>>>
>>> P.S.: As the Linux kernel's regression tracker I deal with a lot of
>>> reports and sometimes miss something important when writing mails like
>>> this. If that's the case here, don't hesitate to tell me in a public
>>> reply, it's in everyone's interest to set the public record straight.
> 

Re: [Bug 215925] New: PCIe regression on Raspberry Pi Compute Module 4 (CM4) breaks booting #forregzbot

[Thorsten Leemhuis]

TWIMC: this mail is primarily send for documentation purposes and for
regzbot, my Linux kernel regression tracking bot. These mails usually
contain '#forregzbot' in the subject, to make them easy to spot and filter.

On 09.05.22 09:44, Thorsten Leemhuis wrote:
> [TLDR: I'm adding this regression report to the list of tracked
> regressions; all text from me you find below is based on a few templates
> paragraphs you might have encountered already already in similar form.]
> 
> On 02.05.22 20:38, Bjorn Helgaas wrote:
>> On Sat, Apr 30, 2022 at 2:53 PM <bugzilla-daemon@kernel.org> wrote:
>>>
>>> https://bugzilla.kernel.org/show_bug.cgi?id=215925

#regzbot fixed-by: f4fd559de3434c44