[Bug] SD card reader in Acer Aspire S5 broken in 4.20-rc

[Bug] SD card reader in Acer Aspire S5 broken in 4.20-rc

[Rafael J. Wysocki]

[-- Attachment #1: Type: text/plain, Size: 1906 bytes --]

Hi Bjorn,

The SD card reader in my Acer Aspire S5 doesn't work with 4.20-rc.

Here's what lspci -v says about it (in a bad kernel):

02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5209 PCI Express Card Reader 
(rev 01)
        Subsystem: Acer Incorporated [ALI] Device 0704
        Flags: bus master, fast devsel, latency 0, IRQ 35
        Memory at d9001000 (32-bit, non-prefetchable) [size=4K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-00-00-01-00-4c-e0-00
        Kernel driver in use: rtsx_pci
        Kernel modules: rtsx_pci

When it doesn't work, it doesn't generate any interrupts on device insertion
and removal and this seems to be reproducible 100% of the time.

Bisection turned up

commit de468b755464426c276df2daf1e54bcd64186020
Merge: b1801bf05964 d6112f8def51
Author: Bjorn Helgaas <bhelgaas@google.com>
Date:   Sat Oct 20 11:45:28 2018 -0500

    Merge branch 'pci/enumeration'
    
      - Remove x86 and arm64 node-local allocation for host bridge structures
        (Punit Agrawal)
    
      - Pay attention to device-specific _PXM node values (Jonathan Cameron)
    
      - Support new Immediate Readiness bit (Felipe Balbi)
    
    * pci/enumeration:
      PCI: Add support for Immediate Readiness
      ACPI/PCI: Pay attention to device-specific _PXM node values
      x86/PCI: Remove node-local allocation when initialising host controller
      arm64: PCI: Remove node-local allocations when initialising host controller

as the first bad commit, but the "PCI: Add support for Immediate Readiness"
one was tested as "good" (full bisect log is attached).

I wonder if you have any ideas on what to check?

Cheers,
Rafael

[-- Attachment #2: aspire-rtsx-bisect.log --]
[-- Type: text/x-log, Size: 2624 bytes --]

git bisect start
# good: [84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d] Linux 4.19
git bisect good 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d
# bad: [2e6e902d185027f8e3cb8b7305238f7e35d6a436] Linux 4.20-rc4
git bisect bad 2e6e902d185027f8e3cb8b7305238f7e35d6a436
# bad: [18d0eae30e6a4f8644d589243d7ac1d70d29203d] Merge tag 'char-misc-4.20-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc
git bisect bad 18d0eae30e6a4f8644d589243d7ac1d70d29203d
# good: [50b825d7e87f4cff7070df6eb26390152bb29537] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
git bisect good 50b825d7e87f4cff7070df6eb26390152bb29537
# bad: [3acbd2de6bc3af215c6ed7732dfc097d1e238503] Merge tag 'sound-4.20-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect bad 3acbd2de6bc3af215c6ed7732dfc097d1e238503
# good: [a41efc2a0f68cea26665ab9e6d991c9bf33b3f59] Merge tag 'dmaengine-4.20-rc1' of git://git.infradead.org/users/vkoul/slave-dma
git bisect good a41efc2a0f68cea26665ab9e6d991c9bf33b3f59
# bad: [d49f8a52b15bf35db778035340d8a673149f9f93] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
git bisect bad d49f8a52b15bf35db778035340d8a673149f9f93
# good: [6c404a68bf83b4135a8a9aa1c388ebdf98e8ba7f] scsi: dc395x: fix DMA API usage in sg_update_list
git bisect good 6c404a68bf83b4135a8a9aa1c388ebdf98e8ba7f
# bad: [61ce5809570b38710f08ebfcd44a5b067eeee77d] Merge branch 'remotes/lorenzo/pci/cadence'
git bisect bad 61ce5809570b38710f08ebfcd44a5b067eeee77d
# good: [e51cd9ce5dd3b10f9e67a30a4dc00fc1fa80c673] PCI/AER: Refactor error injection fallbacks
git bisect good e51cd9ce5dd3b10f9e67a30a4dc00fc1fa80c673
# bad: [ee8360fdafac54eefd0df69fbd99338896cf806b] Merge branch 'pci/misc'
git bisect bad ee8360fdafac54eefd0df69fbd99338896cf806b
# good: [fb513f60ea58f096be7006f899e2181762af37cb] NTB: switchtec_ntb: Update switchtec documentation with prerequisites for NTB
git bisect good fb513f60ea58f096be7006f899e2181762af37cb
# good: [b1801bf05964321d79fbbeae96c8ab09da2e2e49] Merge branch 'pci/aspm'
git bisect good b1801bf05964321d79fbbeae96c8ab09da2e2e49
# good: [d6112f8def514e019658bcc9b57d53acdb71ca3f] PCI: Add support for Immediate Readiness
git bisect good d6112f8def514e019658bcc9b57d53acdb71ca3f
# bad: [20634dc361e1c5fe2dae380a7d0a21ca7f32c4f7] Merge branch 'pci/hotplug'
git bisect bad 20634dc361e1c5fe2dae380a7d0a21ca7f32c4f7
# bad: [de468b755464426c276df2daf1e54bcd64186020] Merge branch 'pci/enumeration'
git bisect bad de468b755464426c276df2daf1e54bcd64186020
# first bad commit: [de468b755464426c276df2daf1e54bcd64186020] Merge branch 'pci/enumeration'

Re: [Bug] SD card reader in Acer Aspire S5 broken in 4.20-rc

[Rafael J. Wysocki]

On Monday, November 26, 2018 7:03:58 PM CET Rafael J. Wysocki wrote:
> Hi Bjorn,
> 
> The SD card reader in my Acer Aspire S5 doesn't work with 4.20-rc.
> 
> Here's what lspci -v says about it (in a bad kernel):
> 
> 02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5209 PCI Express Card Reader 
> (rev 01)
>         Subsystem: Acer Incorporated [ALI] Device 0704
>         Flags: bus master, fast devsel, latency 0, IRQ 35
>         Memory at d9001000 (32-bit, non-prefetchable) [size=4K]
>         Capabilities: [40] Power Management version 3
>         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
>         Capabilities: [70] Express Endpoint, MSI 00
>         Capabilities: [100] Advanced Error Reporting
>         Capabilities: [140] Device Serial Number 00-00-00-01-00-4c-e0-00
>         Kernel driver in use: rtsx_pci
>         Kernel modules: rtsx_pci
> 
> When it doesn't work, it doesn't generate any interrupts on device insertion
> and removal and this seems to be reproducible 100% of the time.
> 
> Bisection turned up
> 
> commit de468b755464426c276df2daf1e54bcd64186020
> Merge: b1801bf05964 d6112f8def51
> Author: Bjorn Helgaas <bhelgaas@google.com>
> Date:   Sat Oct 20 11:45:28 2018 -0500
> 
>     Merge branch 'pci/enumeration'
>     
>       - Remove x86 and arm64 node-local allocation for host bridge structures
>         (Punit Agrawal)
>     
>       - Pay attention to device-specific _PXM node values (Jonathan Cameron)
>     
>       - Support new Immediate Readiness bit (Felipe Balbi)
>     
>     * pci/enumeration:
>       PCI: Add support for Immediate Readiness
>       ACPI/PCI: Pay attention to device-specific _PXM node values
>       x86/PCI: Remove node-local allocation when initialising host controller
>       arm64: PCI: Remove node-local allocations when initialising host controller
> 
> as the first bad commit, but the "PCI: Add support for Immediate Readiness"
> one was tested as "good" (full bisect log is attached).
> 
> I wonder if you have any ideas on what to check?

So reverting 17c91487364f (PCI/ASPM: Do not initialize link state when
aspm_disabled is set) on top of 4.20-rc4 makes the problem go away.

I guess that the device in question needs pcie_aspm_cap_init() to be
called for it even though the FADT says "NO_ASPM".

Thanks,
Rafael

Re: [Bug] SD card reader in Acer Aspire S5 broken in 4.20-rc

[Bjorn Helgaas]

On Mon, Nov 26, 2018 at 11:37:20PM +0100, Rafael J. Wysocki wrote:
> On Monday, November 26, 2018 7:03:58 PM CET Rafael J. Wysocki wrote:
> > Hi Bjorn,
> > 
> > The SD card reader in my Acer Aspire S5 doesn't work with 4.20-rc.
> > 
> > Here's what lspci -v says about it (in a bad kernel):
> > 
> > 02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5209 PCI Express Card Reader 
> > (rev 01)
> >         Subsystem: Acer Incorporated [ALI] Device 0704
> >         Flags: bus master, fast devsel, latency 0, IRQ 35
> >         Memory at d9001000 (32-bit, non-prefetchable) [size=4K]
> >         Capabilities: [40] Power Management version 3
> >         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
> >         Capabilities: [70] Express Endpoint, MSI 00
> >         Capabilities: [100] Advanced Error Reporting
> >         Capabilities: [140] Device Serial Number 00-00-00-01-00-4c-e0-00
> >         Kernel driver in use: rtsx_pci
> >         Kernel modules: rtsx_pci

Thanks a lot for bisecting this!

With a good kernel (v4.19 or v4.20-rc with 17c91487364f reverted),
would you mind collecting "lspci -vv" output, the dmesg log with
"pci=earlydump", and the FADT dump?

I'm interested in the initial state of the device at handoff from
BIOS, and what Linux changes even when aspm_disabled is set.

If we can't figure out a way to fix both this issue and the one fixed
by 17c91487364f, I guess the fallback will be to revert 17c91487364f
since it's better to allow a system that was previously broken to
remain broken than it is to break a system that previously worked.

But obviously I hope we can figure out a solution that fixes both
cases.

> > When it doesn't work, it doesn't generate any interrupts on device insertion
> > and removal and this seems to be reproducible 100% of the time.
> 
> So reverting 17c91487364f (PCI/ASPM: Do not initialize link state when
> aspm_disabled is set) on top of 4.20-rc4 makes the problem go away.
> 
> I guess that the device in question needs pcie_aspm_cap_init() to be
> called for it even though the FADT says "NO_ASPM".

Re: [Bug] SD card reader in Acer Aspire S5 broken in 4.20-rc

[Rafael J. Wysocki]

On Tuesday, November 27, 2018 9:25:14 PM CET Bjorn Helgaas wrote:
> On Mon, Nov 26, 2018 at 11:37:20PM +0100, Rafael J. Wysocki wrote:
> > On Monday, November 26, 2018 7:03:58 PM CET Rafael J. Wysocki wrote:
> > > Hi Bjorn,
> > > 
> > > The SD card reader in my Acer Aspire S5 doesn't work with 4.20-rc.
> > > 
> > > Here's what lspci -v says about it (in a bad kernel):
> > > 
> > > 02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5209 PCI Express Card Reader 
> > > (rev 01)
> > >         Subsystem: Acer Incorporated [ALI] Device 0704
> > >         Flags: bus master, fast devsel, latency 0, IRQ 35
> > >         Memory at d9001000 (32-bit, non-prefetchable) [size=4K]
> > >         Capabilities: [40] Power Management version 3
> > >         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
> > >         Capabilities: [70] Express Endpoint, MSI 00
> > >         Capabilities: [100] Advanced Error Reporting
> > >         Capabilities: [140] Device Serial Number 00-00-00-01-00-4c-e0-00
> > >         Kernel driver in use: rtsx_pci
> > >         Kernel modules: rtsx_pci
> 
> Thanks a lot for bisecting this!
> 
> With a good kernel (v4.19 or v4.20-rc with 17c91487364f reverted),
> would you mind collecting "lspci -vv" output, the dmesg log with
> "pci=earlydump", and the FADT dump?

I'll do that tomorrow.

> I'm interested in the initial state of the device at handoff from
> BIOS, and what Linux changes even when aspm_disabled is set.

OK

> If we can't figure out a way to fix both this issue and the one fixed
> by 17c91487364f, I guess the fallback will be to revert 17c91487364f
> since it's better to allow a system that was previously broken to
> remain broken than it is to break a system that previously worked.

Well, depending on how many systems are affected by the issues fixed by
17c91487364f IMO.

Arguably, if the FADT says "NO_ASPM", then the platform should not need the
OS to initialize ASPM to work.  I guess a manual workaround (like an extra
kernel command line option or similar) should suffice in this particular
case.

> But obviously I hope we can figure out a solution that fixes both
> cases.

Of course.

Re: [Bug] SD card reader in Acer Aspire S5 broken in 4.20-rc

[Rafael J. Wysocki]

On Tuesday, November 27, 2018 9:25:14 PM CET Bjorn Helgaas wrote:
> On Mon, Nov 26, 2018 at 11:37:20PM +0100, Rafael J. Wysocki wrote:
> > On Monday, November 26, 2018 7:03:58 PM CET Rafael J. Wysocki wrote:
> > > Hi Bjorn,
> > > 
> > > The SD card reader in my Acer Aspire S5 doesn't work with 4.20-rc.
> > > 
> > > Here's what lspci -v says about it (in a bad kernel):
> > > 
> > > 02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5209 PCI Express Card Reader 
> > > (rev 01)
> > >         Subsystem: Acer Incorporated [ALI] Device 0704
> > >         Flags: bus master, fast devsel, latency 0, IRQ 35
> > >         Memory at d9001000 (32-bit, non-prefetchable) [size=4K]
> > >         Capabilities: [40] Power Management version 3
> > >         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
> > >         Capabilities: [70] Express Endpoint, MSI 00
> > >         Capabilities: [100] Advanced Error Reporting
> > >         Capabilities: [140] Device Serial Number 00-00-00-01-00-4c-e0-00
> > >         Kernel driver in use: rtsx_pci
> > >         Kernel modules: rtsx_pci
> 
> Thanks a lot for bisecting this!
> 
> With a good kernel (v4.19 or v4.20-rc with 17c91487364f reverted),
> would you mind collecting "lspci -vv" output, the dmesg log with
> "pci=earlydump", and the FADT dump?

All of the information is attached to the BZ entry at

https://bugzilla.kernel.org/show_bug.cgi?id=201801

I have created for the tracking of this issue.  Let's continue in the BZ.

Cheers,
Rafael

Re: [Bug] SD card reader in Acer Aspire S5 broken in 4.20-rc

[Bjorn Helgaas]

On Wed, Nov 28, 2018 at 6:13 AM Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
>
> On Tuesday, November 27, 2018 9:25:14 PM CET Bjorn Helgaas wrote:
> > On Mon, Nov 26, 2018 at 11:37:20PM +0100, Rafael J. Wysocki wrote:
> > > On Monday, November 26, 2018 7:03:58 PM CET Rafael J. Wysocki wrote:
> > > > Hi Bjorn,
> > > >
> > > > The SD card reader in my Acer Aspire S5 doesn't work with 4.20-rc.
> > > >
> > > > Here's what lspci -v says about it (in a bad kernel):
> > > >
> > > > 02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5209 PCI Express Card Reader
> > > > (rev 01)
> > > >         Subsystem: Acer Incorporated [ALI] Device 0704
> > > >         Flags: bus master, fast devsel, latency 0, IRQ 35
> > > >         Memory at d9001000 (32-bit, non-prefetchable) [size=4K]
> > > >         Capabilities: [40] Power Management version 3
> > > >         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
> > > >         Capabilities: [70] Express Endpoint, MSI 00
> > > >         Capabilities: [100] Advanced Error Reporting
> > > >         Capabilities: [140] Device Serial Number 00-00-00-01-00-4c-e0-00
> > > >         Kernel driver in use: rtsx_pci
> > > >         Kernel modules: rtsx_pci
> >
> > Thanks a lot for bisecting this!
> >
> > With a good kernel (v4.19 or v4.20-rc with 17c91487364f reverted),
> > would you mind collecting "lspci -vv" output, the dmesg log with
> > "pci=earlydump", and the FADT dump?
>
> All of the information is attached to the BZ entry at
>
> https://bugzilla.kernel.org/show_bug.cgi?id=201801

Thanks!  I hope Patrick has a chance to look at this.  Per the
bugzilla mentioned in 17c91487364f, it fixes a problem with a custom
proprietary PCIe device, and there's a lot of good detailed analysis
there, so hopefully we can figure out a way to address both
situations.

Re: [Bug] SD card reader in Acer Aspire S5 broken in 4.20-rc

[Bjorn Helgaas]

On Wed, Nov 28, 2018 at 02:05:21PM -0600, Bjorn Helgaas wrote:
> On Wed, Nov 28, 2018 at 6:13 AM Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > On Tuesday, November 27, 2018 9:25:14 PM CET Bjorn Helgaas wrote:
> > > On Mon, Nov 26, 2018 at 11:37:20PM +0100, Rafael J. Wysocki wrote:
> > > > On Monday, November 26, 2018 7:03:58 PM CET Rafael J. Wysocki wrote:
> > > > > Hi Bjorn,
> > > > >
> > > > > The SD card reader in my Acer Aspire S5 doesn't work with 4.20-rc.
> > > > >
> > > > > Here's what lspci -v says about it (in a bad kernel):
> > > > >
> > > > > 02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5209 PCI Express Card Reader
> > > > > (rev 01)
> > > > >         Subsystem: Acer Incorporated [ALI] Device 0704
> > > > >         Flags: bus master, fast devsel, latency 0, IRQ 35
> > > > >         Memory at d9001000 (32-bit, non-prefetchable) [size=4K]
> > > > >         Capabilities: [40] Power Management version 3
> > > > >         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
> > > > >         Capabilities: [70] Express Endpoint, MSI 00
> > > > >         Capabilities: [100] Advanced Error Reporting
> > > > >         Capabilities: [140] Device Serial Number 00-00-00-01-00-4c-e0-00
> > > > >         Kernel driver in use: rtsx_pci
> > > > >         Kernel modules: rtsx_pci
> > >
> > > Thanks a lot for bisecting this!
> > >
> > > With a good kernel (v4.19 or v4.20-rc with 17c91487364f reverted),
> > > would you mind collecting "lspci -vv" output, the dmesg log with
> > > "pci=earlydump", and the FADT dump?
> >
> > All of the information is attached to the BZ entry at
> >
> > https://bugzilla.kernel.org/show_bug.cgi?id=201801
> 
> Thanks!  I hope Patrick has a chance to look at this.  Per the
> bugzilla mentioned in 17c91487364f, it fixes a problem with a custom
> proprietary PCIe device, and there's a lot of good detailed analysis
> there, so hopefully we can figure out a way to address both
> situations.

I queued up a revert on for-linus, since we haven't made any progress on
this yet.  I'll be on vacation much of this week, but I want to get
the revert (or better, a fix if we can find one) in before -rc6 comes
out next Sunday.

If we figure out a fix before then, I'll drop the revert and use the
fix instead.

Bjorn

Re: [Bug] SD card reader in Acer Aspire S5 broken in 4.20-rc

[Rafael J. Wysocki]

On Tuesday, December 4, 2018 1:10:09 AM CET Bjorn Helgaas wrote:
> On Wed, Nov 28, 2018 at 02:05:21PM -0600, Bjorn Helgaas wrote:
> > On Wed, Nov 28, 2018 at 6:13 AM Rafael J. Wysocki <rjw@rjwysocki.net> wrote:
> > > On Tuesday, November 27, 2018 9:25:14 PM CET Bjorn Helgaas wrote:
> > > > On Mon, Nov 26, 2018 at 11:37:20PM +0100, Rafael J. Wysocki wrote:
> > > > > On Monday, November 26, 2018 7:03:58 PM CET Rafael J. Wysocki wrote:
> > > > > > Hi Bjorn,
> > > > > >
> > > > > > The SD card reader in my Acer Aspire S5 doesn't work with 4.20-rc.
> > > > > >
> > > > > > Here's what lspci -v says about it (in a bad kernel):
> > > > > >
> > > > > > 02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5209 PCI Express Card Reader
> > > > > > (rev 01)
> > > > > >         Subsystem: Acer Incorporated [ALI] Device 0704
> > > > > >         Flags: bus master, fast devsel, latency 0, IRQ 35
> > > > > >         Memory at d9001000 (32-bit, non-prefetchable) [size=4K]
> > > > > >         Capabilities: [40] Power Management version 3
> > > > > >         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
> > > > > >         Capabilities: [70] Express Endpoint, MSI 00
> > > > > >         Capabilities: [100] Advanced Error Reporting
> > > > > >         Capabilities: [140] Device Serial Number 00-00-00-01-00-4c-e0-00
> > > > > >         Kernel driver in use: rtsx_pci
> > > > > >         Kernel modules: rtsx_pci
> > > >
> > > > Thanks a lot for bisecting this!
> > > >
> > > > With a good kernel (v4.19 or v4.20-rc with 17c91487364f reverted),
> > > > would you mind collecting "lspci -vv" output, the dmesg log with
> > > > "pci=earlydump", and the FADT dump?
> > >
> > > All of the information is attached to the BZ entry at
> > >
> > > https://bugzilla.kernel.org/show_bug.cgi?id=201801
> > 
> > Thanks!  I hope Patrick has a chance to look at this.  Per the
> > bugzilla mentioned in 17c91487364f, it fixes a problem with a custom
> > proprietary PCIe device, and there's a lot of good detailed analysis
> > there, so hopefully we can figure out a way to address both
> > situations.
> 
> I queued up a revert on for-linus, since we haven't made any progress on
> this yet.  I'll be on vacation much of this week, but I want to get
> the revert (or better, a fix if we can find one) in before -rc6 comes
> out next Sunday.

Thanks!

> If we figure out a fix before then, I'll drop the revert and use the
> fix instead.