[RFC] pcie hotplug doesn't work with kernel 4.19

[RFC] pcie hotplug doesn't work with kernel 4.19

[Jinpu Wang]

Hi folks,

We are testing PCIe nvme SSD hotplug, it works out of box with kernel 5.4.62,
dmesg during the hotplug:

[  605.734513] pcieport 0000:16:00.0: pciehp: Slot(0-3): Link Down
[  605.734516] pcieport 0000:16:00.0: pciehp: Slot(0-3): Card not present
[  605.842634] blk_update_request: I/O error, dev nvme0n1, sector
205976576 op 0x1:(WRITE) flags 0x0 phys_seg 112 prio class 0
[  608.908764] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
command 0x15e1 (issued 3030 msec ago)
[  609.988759] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
command 0x15e1 (issued 4110 msec ago)
[  683.218554] pcieport 0000:16:00.0: pciehp: Slot(0-3): Card present
[  683.218555] pcieport 0000:16:00.0: pciehp: Slot(0-3): Link Up
[  683.271702] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
command 0x17e1 (issued 73280 msec ago)
[  686.301874] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
command 0x13e1 (issued 3030 msec ago)
[  686.361894] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
command 0x13e1 (issued 3090 msec ago)
[  686.521911] pci 0000:17:00.0: [1b96:2400] type 00 class 0x010802
[  686.521924] pci 0000:17:00.0: reg 0x10: [mem 0x00000000-0x00007fff 64bit]
[  686.521934] pci 0000:17:00.0: reg 0x20: [mem 0x00000000-0x00000fff
64bit pref]
[  686.521937] pci 0000:17:00.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
[  686.521941] pci 0000:17:00.0: enabling Extended Tags
[  686.522045] pci 0000:17:00.0: BAR 6: assigned [mem
0xc5e00000-0xc5e1ffff pref]
[  686.522046] pci 0000:17:00.0: BAR 0: assigned [mem
0xc5e20000-0xc5e27fff 64bit]
[  686.522051] pci 0000:17:00.0: BAR 4: assigned [mem
0x387ffff00000-0x387ffff00fff 64bit pref]
[  686.522055] pcieport 0000:16:00.0: PCI bridge to [bus 17]
[  686.522057] pcieport 0000:16:00.0:   bridge window [io  0x4000-0x4fff]
[  686.522059] pcieport 0000:16:00.0:   bridge window [mem
0xc5e00000-0xc5efffff]
[  686.522060] pcieport 0000:16:00.0:   bridge window [mem
0x387ffff00000-0x387fffffffff 64bit pref]
[  686.522302] nvme nvme2: pci function 0000:17:00.0
[  686.522355] nvme 0000:17:00.0: enabling device (0100 -> 0102)
[  689.072008] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
command 0x12e1 (issued 2710 msec ago)
[  690.373707] nvme nvme2: 40/0/0 default/read/poll queues

But with kernel 4.19.133, pcieport core doesn't print anything, is
there known problem with kernel 4.19 support for pcie hotplug, do we
need to backport some fixes from newer kernel to make it work?

In both kernel 4.19.133 and kernel 5.4.62 following configs are enabled.

CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
CONFIG_HOTPLUG_PCI_SHPC=y
CONFIG_HOTPLUG_PCI_PCIE=y

Thanks in advance!
-- 
Jinpu Wang
Linux Kernel Developer

Application Support (IONOS Cloud)

1&1 IONOS SE | Greifswalder Str. 207 | 10405 Berlin | Germany
Phone:
E-mail: jinpu.wang@cloud.ionos.com | Web: www.ionos.de

Hauptsitz Montabaur, Amtsgericht Montabaur, HRB 24498

Vorstand: Dr. Christian Böing, Hüseyin Dogan, Dr. Martin Endreß,
Hans-Henning Kettler, Arthur Mai, Matthias Steinberg, Achim Weiß
Aufsichtsratsvorsitzender: Markus Kadelke


Member of United Internet

Diese E-Mail kann vertrauliche und/oder gesetzlich geschützte
Informationen enthalten. Wenn Sie nicht der bestimmungsgemäße Adressat
sind oder diese E-Mail irrtümlich erhalten haben, unterrichten Sie
bitte den Absender und vernichten Sie diese E-Mail. Anderen als dem
bestimmungsgemäßen Adressaten ist untersagt, diese E-Mail zu
speichern, weiterzuleiten oder ihren Inhalt auf welche Weise auch
immer zu verwenden.

This e-mail may contain confidential and/or privileged information. If
you are not the intended recipient of this e-mail, you are hereby
notified that saving, distribution or use of the content of this
e-mail in any way is prohibited. If you have received this e-mail in
error, please notify the sender and delete the e-mail.

Re: [RFC] pcie hotplug doesn't work with kernel 4.19

[Lukas Wunner]

On Tue, Sep 15, 2020 at 04:15:15PM +0200, Jinpu Wang wrote:
> We are testing PCIe nvme SSD hotplug, it works out of box with kernel 5.4.62,
> dmesg during the hotplug:
[...]
> But with kernel 4.19.133, pcieport core doesn't print anything, is
> there known problem with kernel 4.19 support for pcie hotplug, do we
> need to backport some fixes from newer kernel to make it work?

No known problem.  Please open a bug on bugzilla.kernel.org and attach
full dmesg output for 5.4.62 and 4.19.133, as well as lspci -vv output.
You may want to add the following to the kernel command line:

ignore_loglevel log_bug_len=10M "dyndbg=file drivers/pci/* +p"
pciehp.pciehp_debug=1


> [  683.218554] pcieport 0000:16:00.0: pciehp: Slot(0-3): Card present
> [  683.218555] pcieport 0000:16:00.0: pciehp: Slot(0-3): Link Up
> [  683.271702] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
> command 0x17e1 (issued 73280 msec ago)
> [  686.301874] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
> command 0x13e1 (issued 3030 msec ago)
> [  686.361894] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
> command 0x13e1 (issued 3090 msec ago)

Those timeouts look suspicious.  Perhaps the hotplug controller
claims to support Command Completed Support, but in reality does not?


> CONFIG_HOTPLUG_PCI=y
> CONFIG_HOTPLUG_PCI_ACPI=y
> CONFIG_HOTPLUG_PCI_ACPI_IBM=m
> CONFIG_HOTPLUG_PCI_CPCI=y
> CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
> CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
> CONFIG_HOTPLUG_PCI_SHPC=y
> CONFIG_HOTPLUG_PCI_PCIE=y

You may not need ACPI_IBM, CPCI, SHPC.

Thanks,

Lukas

Re: [RFC] pcie hotplug doesn't work with kernel 4.19

[Jinpu Wang]

Hi Lukas,

see reply below
On Wed, Sep 16, 2020 at 7:48 AM Lukas Wunner <lukas@wunner.de> wrote:
>
> On Tue, Sep 15, 2020 at 04:15:15PM +0200, Jinpu Wang wrote:
> > We are testing PCIe nvme SSD hotplug, it works out of box with kernel 5.4.62,
> > dmesg during the hotplug:
> [...]
> > But with kernel 4.19.133, pcieport core doesn't print anything, is
> > there known problem with kernel 4.19 support for pcie hotplug, do we
> > need to backport some fixes from newer kernel to make it work?
>
> No known problem.  Please open a bug on bugzilla.kernel.org and attach
> full dmesg output for 5.4.62 and 4.19.133, as well as lspci -vv output.
> You may want to add the following to the kernel command line:
>
> ignore_loglevel log_bug_len=10M "dyndbg=file drivers/pci/* +p"
> pciehp.pciehp_debug=1
https://bugzilla.kernel.org/show_bug.cgi?id=209283
I added you as cc.
I will gather other logs as suggested, and upload them to bugzilla.
>
>
> > [  683.218554] pcieport 0000:16:00.0: pciehp: Slot(0-3): Card present
> > [  683.218555] pcieport 0000:16:00.0: pciehp: Slot(0-3): Link Up
> > [  683.271702] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
> > command 0x17e1 (issued 73280 msec ago)
> > [  686.301874] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
> > command 0x13e1 (issued 3030 msec ago)
> > [  686.361894] pcieport 0000:16:00.0: pciehp: Timeout on hotplug
> > command 0x13e1 (issued 3090 msec ago)
>
> Those timeouts look suspicious.  Perhaps the hotplug controller
> claims to support Command Completed Support, but in reality does not?
>
>
> > CONFIG_HOTPLUG_PCI=y
> > CONFIG_HOTPLUG_PCI_ACPI=y
> > CONFIG_HOTPLUG_PCI_ACPI_IBM=m
> > CONFIG_HOTPLUG_PCI_CPCI=y
> > CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
> > CONFIG_HOTPLUG_PCI_CPCI_GENERIC=m
> > CONFIG_HOTPLUG_PCI_SHPC=y
> > CONFIG_HOTPLUG_PCI_PCIE=y
>
> You may not need ACPI_IBM, CPCI, SHPC.
will consider this.
>
> Thanks,
>
> Lukas
Thanks!
Jinpu