* [3.8-rc7] PCI hotplug wakeup oops @ 2013-02-11 13:03 Daniel J Blueman 2013-02-11 17:08 ` Daniel J Blueman 0 siblings, 1 reply; 12+ messages in thread From: Daniel J Blueman @ 2013-02-11 13:03 UTC (permalink / raw) To: Linux Kernel, Linux PCI With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released: pciehp 0000:06:03.0:pcie24: Card not present on Slot(3) tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff tg3 0000:0a:00.0 eth0: No firmware running tg3 0000:0a:00.0 eth0: Link is down [sched_delayed] sched: RT throttling activated pcieport 0000:00:01.1: System wakeup enabled by ACPI pciehp 0000:09:00.0:pcie24: unloading service driver pciehp pci_bus 0000:0a: busn_res: [bus 0a] is released pci_bus 0000:09: busn_res: [bus 09-0a] is released After some activity later (eg I can reproduce this by switching to a text console and back), often we'll see an oops: Unable to handle kernel paging request at 0000000000001070 pci_pme_list_scan+0x3d/0xe0 Call Trace: process_one_work+0x193 ? process_one_work+0x131 ? pci_pme_wakeup+0x60 worker_thread+0x15d (gdb) list *(pci_pme_list_scan+0x3d) 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556). 1551 /* 1552 * If bridge is in low power state, the 1553 * configuration space of subordinate devices 1554 * may be not accessible 1555 */ 1556 if (bridge && bridge->current_state != PCI_D0) 1557 continue; 1558 pci_pme_wakeup(pme_dev->dev, NULL); 1559 } else { 1560 list_del(&pme_dev->list); Since a panic in vsnprintf happens after the oops (hence I can't catch it with EFI pstore), it is almost certainly significant heap corruption; this would explain why pme_dev became null (the load has been ordered ahead). I'll see what I can find out with memory poisoning and list debugging. Thanks, Daniel --- [1] $ lspci 00:00.0 Host bridge: Intel Corporation 3rd Gen Core processor DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09) 00:02.0 VGA compatible controller: Intel Corporation 3rd Gen Core processor Graphics Controller (rev 09) 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04) 00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04) 00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04) 00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04) 00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4) 00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4) 00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04) 00:1f.0 ISA bridge: Intel Corporation HM77 Express Chipset LPC Controller (rev 04) 00:1f.2 SATA controller: Intel Corporation 7 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) 00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04) 01:00.0 VGA compatible controller: NVIDIA Corporation Device 0fd5 (rev ff) 01:00.1 Audio device: NVIDIA Corporation Device 0e1b (rev ff) 03:00.0 Ethernet controller: Broadcom Corporation Device 16a3 (rev 10) 03:00.1 SD Host controller: Broadcom Corporation NetXtreme BCM57765 Memory Card Reader (rev 10) 04:00.0 Network controller: Broadcom Corporation BCM4331 802.11a/b/g/n (rev 02) 05:00.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus Ridge] (rev 03) 06:00.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus Ridge] (rev 03) 06:03.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus Ridge] (rev 03) 06:04.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus Ridge] (rev 03) 06:05.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus Ridge] (rev 03) 06:06.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus Ridge] (rev 03) 07:00.0 System peripheral: Intel Corporation DSL3510 Thunderbolt Port [Cactus Ridge] (rev 03) 08:00.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] 09:00.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] 0a:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe -- Daniel J Blueman ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops 2013-02-11 13:03 [3.8-rc7] PCI hotplug wakeup oops Daniel J Blueman @ 2013-02-11 17:08 ` Daniel J Blueman 2013-02-11 19:01 ` Bjorn Helgaas 0 siblings, 1 reply; 12+ messages in thread From: Daniel J Blueman @ 2013-02-11 17:08 UTC (permalink / raw) To: Linux Kernel, Linux PCI; +Cc: Yijing Wang, Bjorn Helgaas On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote: > With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a > [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released: > > pciehp 0000:06:03.0:pcie24: Card not present on Slot(3) > tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not > clear MAC_TX_MODE=ffffffff > tg3 0000:0a:00.0 eth0: No firmware running > tg3 0000:0a:00.0 eth0: Link is down > [sched_delayed] sched: RT throttling activated > pcieport 0000:00:01.1: System wakeup enabled by ACPI > pciehp 0000:09:00.0:pcie24: unloading service driver pciehp > pci_bus 0000:0a: busn_res: [bus 0a] is released > pci_bus 0000:09: busn_res: [bus 09-0a] is released > > After some activity later (eg I can reproduce this by switching to a > text console and back), often we'll see an oops: > > Unable to handle kernel paging request at 0000000000001070 > pci_pme_list_scan+0x3d/0xe0 > Call Trace: > process_one_work+0x193 > ? process_one_work+0x131 > ? pci_pme_wakeup+0x60 > worker_thread+0x15d > > (gdb) list *(pci_pme_list_scan+0x3d) > 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556). > 1551 /* > 1552 * If bridge is in low power state, the > 1553 * configuration space of subordinate devices > 1554 * may be not accessible > 1555 */ > 1556 if (bridge && bridge->current_state != PCI_D0) > 1557 continue; > 1558 pci_pme_wakeup(pme_dev->dev, NULL); > 1559 } else { > 1560 list_del(&pme_dev->list); > > Since a panic in vsnprintf happens after the oops (hence I can't catch > it with EFI pstore), it is almost certainly significant heap > corruption; this would explain why pme_dev became null (the load has > been ordered ahead). > > I'll see what I can find out with memory poisoning and list debugging. Enabling a bunch of related debugging, we see pme_dev is non-null and: BUG: Unable to handle NULL pointer dereference at pci_bus_read_config_word+0x6c PGD 26314c067 PUD 2633f9067 PMD 0 Oops: 0000 [#1] SMP pci_check_pme_status+0x4f pci_pme_wakeup+0x21 pci_pme_list_scan+0xd5 process_one_work+0x1ca ? process_one_work+0x160 ? pci_pme_wakeup+0x60 worker_thread+0x14e Anyway, it looks like the device being unplugged wasn't removed from pci_pme_list as pci_pme_active(dev, false) wasn't called. >From a quick review, I wasn't able to find the right place in the call-chain which I only see releases the child busses and PCIe port drivers. Anyone? Thanks, Daniel -- Daniel J Blueman ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops 2013-02-11 17:08 ` Daniel J Blueman @ 2013-02-11 19:01 ` Bjorn Helgaas 2013-02-11 19:27 ` Rafael J. Wysocki 0 siblings, 1 reply; 12+ messages in thread From: Bjorn Helgaas @ 2013-02-11 19:01 UTC (permalink / raw) To: Daniel J Blueman; +Cc: Linux Kernel, Linux PCI, Yijing Wang, Rafael J. Wysocki [+cc Rafael] On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote: > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote: >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released: >> >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3) >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not >> clear MAC_TX_MODE=ffffffff >> tg3 0000:0a:00.0 eth0: No firmware running >> tg3 0000:0a:00.0 eth0: Link is down >> [sched_delayed] sched: RT throttling activated >> pcieport 0000:00:01.1: System wakeup enabled by ACPI >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp >> pci_bus 0000:0a: busn_res: [bus 0a] is released >> pci_bus 0000:09: busn_res: [bus 09-0a] is released >> >> After some activity later (eg I can reproduce this by switching to a >> text console and back), often we'll see an oops: >> >> Unable to handle kernel paging request at 0000000000001070 >> pci_pme_list_scan+0x3d/0xe0 >> Call Trace: >> process_one_work+0x193 >> ? process_one_work+0x131 >> ? pci_pme_wakeup+0x60 >> worker_thread+0x15d >> >> (gdb) list *(pci_pme_list_scan+0x3d) >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556). >> 1551 /* >> 1552 * If bridge is in low power state, the >> 1553 * configuration space of subordinate devices >> 1554 * may be not accessible >> 1555 */ >> 1556 if (bridge && bridge->current_state != PCI_D0) >> 1557 continue; >> 1558 pci_pme_wakeup(pme_dev->dev, NULL); >> 1559 } else { >> 1560 list_del(&pme_dev->list); >> >> Since a panic in vsnprintf happens after the oops (hence I can't catch >> it with EFI pstore), it is almost certainly significant heap >> corruption; this would explain why pme_dev became null (the load has >> been ordered ahead). >> >> I'll see what I can find out with memory poisoning and list debugging. > > Enabling a bunch of related debugging, we see pme_dev is non-null and: > > BUG: Unable to handle NULL pointer dereference at > pci_bus_read_config_word+0x6c > PGD 26314c067 PUD 2633f9067 PMD 0 > Oops: 0000 [#1] SMP > pci_check_pme_status+0x4f > pci_pme_wakeup+0x21 > pci_pme_list_scan+0xd5 > process_one_work+0x1ca > ? process_one_work+0x160 > ? pci_pme_wakeup+0x60 > worker_thread+0x14e > > Anyway, it looks like the device being unplugged wasn't removed from > pci_pme_list as pci_pme_active(dev, false) wasn't called. > > From a quick review, I wasn't able to find the right place in the > call-chain which I only see releases the child busses and PCIe port > drivers. Anyone? It looks like drivers *add* devices to pci_pme_list when they use pci_enable_wake() or pci_wake_from_d3(). But many drivers never remove their devices, and I don't see any place where the core does it either. My guess is we need to remove it in pci_stop_dev() (we already do pcie_aspm_exit_link_state() there) or somewhere similar. Bjorn ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops 2013-02-11 19:01 ` Bjorn Helgaas @ 2013-02-11 19:27 ` Rafael J. Wysocki 2013-02-11 19:49 ` Rafael J. Wysocki 0 siblings, 1 reply; 12+ messages in thread From: Rafael J. Wysocki @ 2013-02-11 19:27 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Daniel J Blueman, Linux Kernel, Linux PCI, Yijing Wang On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote: > [+cc Rafael] > > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote: > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote: > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released: > >> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3) > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not > >> clear MAC_TX_MODE=ffffffff > >> tg3 0000:0a:00.0 eth0: No firmware running > >> tg3 0000:0a:00.0 eth0: Link is down > >> [sched_delayed] sched: RT throttling activated > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp > >> pci_bus 0000:0a: busn_res: [bus 0a] is released > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released > >> > >> After some activity later (eg I can reproduce this by switching to a > >> text console and back), often we'll see an oops: > >> > >> Unable to handle kernel paging request at 0000000000001070 > >> pci_pme_list_scan+0x3d/0xe0 > >> Call Trace: > >> process_one_work+0x193 > >> ? process_one_work+0x131 > >> ? pci_pme_wakeup+0x60 > >> worker_thread+0x15d > >> > >> (gdb) list *(pci_pme_list_scan+0x3d) > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556). > >> 1551 /* > >> 1552 * If bridge is in low power state, the > >> 1553 * configuration space of subordinate devices > >> 1554 * may be not accessible > >> 1555 */ > >> 1556 if (bridge && bridge->current_state != PCI_D0) > >> 1557 continue; > >> 1558 pci_pme_wakeup(pme_dev->dev, NULL); > >> 1559 } else { > >> 1560 list_del(&pme_dev->list); > >> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch > >> it with EFI pstore), it is almost certainly significant heap > >> corruption; this would explain why pme_dev became null (the load has > >> been ordered ahead). > >> > >> I'll see what I can find out with memory poisoning and list debugging. > > > > Enabling a bunch of related debugging, we see pme_dev is non-null and: > > > > BUG: Unable to handle NULL pointer dereference at > > pci_bus_read_config_word+0x6c > > PGD 26314c067 PUD 2633f9067 PMD 0 > > Oops: 0000 [#1] SMP > > pci_check_pme_status+0x4f > > pci_pme_wakeup+0x21 > > pci_pme_list_scan+0xd5 > > process_one_work+0x1ca > > ? process_one_work+0x160 > > ? pci_pme_wakeup+0x60 > > worker_thread+0x14e > > > > Anyway, it looks like the device being unplugged wasn't removed from > > pci_pme_list as pci_pme_active(dev, false) wasn't called. > > > > From a quick review, I wasn't able to find the right place in the > > call-chain which I only see releases the child busses and PCIe port > > drivers. Anyone? > > It looks like drivers *add* devices to pci_pme_list when they use > pci_enable_wake() or pci_wake_from_d3(). But many drivers never > remove their devices, and I don't see any place where the core does it > either. My guess is we need to remove it in pci_stop_dev() (we > already do pcie_aspm_exit_link_state() there) or somewhere similar. Yes, we should call pci_pme_active(dev, false) somewhere in there I think. It's fine to call that even if PME was not "active" before. Thanks, Rafael -- I speak only for myself. Rafael J. Wysocki, Intel Open Source Technology Center. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops 2013-02-11 19:27 ` Rafael J. Wysocki @ 2013-02-11 19:49 ` Rafael J. Wysocki 2013-02-12 2:18 ` Daniel J Blueman 2013-02-13 14:32 ` Daniel J Blueman 0 siblings, 2 replies; 12+ messages in thread From: Rafael J. Wysocki @ 2013-02-11 19:49 UTC (permalink / raw) To: Bjorn Helgaas, Daniel J Blueman; +Cc: Linux Kernel, Linux PCI, Yijing Wang On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote: > On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote: > > [+cc Rafael] > > > > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote: > > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote: > > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a > > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released: > > >> > > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3) > > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not > > >> clear MAC_TX_MODE=ffffffff > > >> tg3 0000:0a:00.0 eth0: No firmware running > > >> tg3 0000:0a:00.0 eth0: Link is down > > >> [sched_delayed] sched: RT throttling activated > > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI > > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp > > >> pci_bus 0000:0a: busn_res: [bus 0a] is released > > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released > > >> > > >> After some activity later (eg I can reproduce this by switching to a > > >> text console and back), often we'll see an oops: > > >> > > >> Unable to handle kernel paging request at 0000000000001070 > > >> pci_pme_list_scan+0x3d/0xe0 > > >> Call Trace: > > >> process_one_work+0x193 > > >> ? process_one_work+0x131 > > >> ? pci_pme_wakeup+0x60 > > >> worker_thread+0x15d > > >> > > >> (gdb) list *(pci_pme_list_scan+0x3d) > > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556). > > >> 1551 /* > > >> 1552 * If bridge is in low power state, the > > >> 1553 * configuration space of subordinate devices > > >> 1554 * may be not accessible > > >> 1555 */ > > >> 1556 if (bridge && bridge->current_state != PCI_D0) > > >> 1557 continue; > > >> 1558 pci_pme_wakeup(pme_dev->dev, NULL); > > >> 1559 } else { > > >> 1560 list_del(&pme_dev->list); > > >> > > >> Since a panic in vsnprintf happens after the oops (hence I can't catch > > >> it with EFI pstore), it is almost certainly significant heap > > >> corruption; this would explain why pme_dev became null (the load has > > >> been ordered ahead). > > >> > > >> I'll see what I can find out with memory poisoning and list debugging. > > > > > > Enabling a bunch of related debugging, we see pme_dev is non-null and: > > > > > > BUG: Unable to handle NULL pointer dereference at > > > pci_bus_read_config_word+0x6c > > > PGD 26314c067 PUD 2633f9067 PMD 0 > > > Oops: 0000 [#1] SMP > > > pci_check_pme_status+0x4f > > > pci_pme_wakeup+0x21 > > > pci_pme_list_scan+0xd5 > > > process_one_work+0x1ca > > > ? process_one_work+0x160 > > > ? pci_pme_wakeup+0x60 > > > worker_thread+0x14e > > > > > > Anyway, it looks like the device being unplugged wasn't removed from > > > pci_pme_list as pci_pme_active(dev, false) wasn't called. > > > > > > From a quick review, I wasn't able to find the right place in the > > > call-chain which I only see releases the child busses and PCIe port > > > drivers. Anyone? > > > > It looks like drivers *add* devices to pci_pme_list when they use > > pci_enable_wake() or pci_wake_from_d3(). But many drivers never > > remove their devices, and I don't see any place where the core does it > > either. My guess is we need to remove it in pci_stop_dev() (we > > already do pcie_aspm_exit_link_state() there) or somewhere similar. > > Yes, we should call pci_pme_active(dev, false) somewhere in there I think. > It's fine to call that even if PME was not "active" before. Daniel, I wonder if the patch below helps? Rafael Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> --- drivers/pci/remove.c | 2 ++ 1 file changed, 2 insertions(+) Index: test/drivers/pci/remove.c =================================================================== --- test.orig/drivers/pci/remove.c +++ test/drivers/pci/remove.c @@ -19,6 +19,8 @@ static void pci_free_resources(struct pc static void pci_stop_dev(struct pci_dev *dev) { + pci_pme_active(dev, false); + if (dev->is_added) { pci_proc_detach_device(dev); pci_remove_sysfs_dev_files(dev); -- I speak only for myself. Rafael J. Wysocki, Intel Open Source Technology Center. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops 2013-02-11 19:49 ` Rafael J. Wysocki @ 2013-02-12 2:18 ` Daniel J Blueman 2013-02-12 20:53 ` Rafael J. Wysocki 2013-02-13 14:32 ` Daniel J Blueman 1 sibling, 1 reply; 12+ messages in thread From: Daniel J Blueman @ 2013-02-12 2:18 UTC (permalink / raw) To: Rafael J. Wysocki; +Cc: Bjorn Helgaas, Linux Kernel, Linux PCI, Yijing Wang On 12 February 2013 03:49, Rafael J. Wysocki <rjw@sisk.pl> wrote: > On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote: >> On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote: >> > [+cc Rafael] >> > >> > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote: >> > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote: >> > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a >> > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released: >> > >> >> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3) >> > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not >> > >> clear MAC_TX_MODE=ffffffff >> > >> tg3 0000:0a:00.0 eth0: No firmware running >> > >> tg3 0000:0a:00.0 eth0: Link is down >> > >> [sched_delayed] sched: RT throttling activated >> > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI >> > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp >> > >> pci_bus 0000:0a: busn_res: [bus 0a] is released >> > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released >> > >> >> > >> After some activity later (eg I can reproduce this by switching to a >> > >> text console and back), often we'll see an oops: >> > >> >> > >> Unable to handle kernel paging request at 0000000000001070 >> > >> pci_pme_list_scan+0x3d/0xe0 >> > >> Call Trace: >> > >> process_one_work+0x193 >> > >> ? process_one_work+0x131 >> > >> ? pci_pme_wakeup+0x60 >> > >> worker_thread+0x15d >> > >> >> > >> (gdb) list *(pci_pme_list_scan+0x3d) >> > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556). >> > >> 1551 /* >> > >> 1552 * If bridge is in low power state, the >> > >> 1553 * configuration space of subordinate devices >> > >> 1554 * may be not accessible >> > >> 1555 */ >> > >> 1556 if (bridge && bridge->current_state != PCI_D0) >> > >> 1557 continue; >> > >> 1558 pci_pme_wakeup(pme_dev->dev, NULL); >> > >> 1559 } else { >> > >> 1560 list_del(&pme_dev->list); >> > >> >> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch >> > >> it with EFI pstore), it is almost certainly significant heap >> > >> corruption; this would explain why pme_dev became null (the load has >> > >> been ordered ahead). >> > >> >> > >> I'll see what I can find out with memory poisoning and list debugging. >> > > >> > > Enabling a bunch of related debugging, we see pme_dev is non-null and: >> > > >> > > BUG: Unable to handle NULL pointer dereference at >> > > pci_bus_read_config_word+0x6c >> > > PGD 26314c067 PUD 2633f9067 PMD 0 >> > > Oops: 0000 [#1] SMP >> > > pci_check_pme_status+0x4f >> > > pci_pme_wakeup+0x21 >> > > pci_pme_list_scan+0xd5 >> > > process_one_work+0x1ca >> > > ? process_one_work+0x160 >> > > ? pci_pme_wakeup+0x60 >> > > worker_thread+0x14e >> > > >> > > Anyway, it looks like the device being unplugged wasn't removed from >> > > pci_pme_list as pci_pme_active(dev, false) wasn't called. >> > > >> > > From a quick review, I wasn't able to find the right place in the >> > > call-chain which I only see releases the child busses and PCIe port >> > > drivers. Anyone? >> > >> > It looks like drivers *add* devices to pci_pme_list when they use >> > pci_enable_wake() or pci_wake_from_d3(). But many drivers never >> > remove their devices, and I don't see any place where the core does it >> > either. My guess is we need to remove it in pci_stop_dev() (we >> > already do pcie_aspm_exit_link_state() there) or somewhere similar. >> >> Yes, we should call pci_pme_active(dev, false) somewhere in there I think. >> It's fine to call that even if PME was not "active" before. > > Daniel, I wonder if the patch below helps? I had tried with the pci_pme_active call inside the is_added condition (it'll always hold here); it resolves the issue, though it introduces a new lockdep warning [1]. All said, I don't see any other way except for a patch to abstract the list entry removal to avoid the unnecessary reads and writes (as it is called for four devices here), though I don't see how that would alter the locking behaviour and why we didn't see this lockdep warning before. What do you think? Dan --- [1] kworker/0:0/4 is trying to acquire lock: (name){++++.+}, at: [<ffffffff8105ac70>] flush_workqueue+0x0/0x4d0 but task is already holding lock: (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(name); lock(name); *** DEADLOCK *** May be due to missing lock nesting notation 4 locks held by kworker/0:0/4: #0: (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0 #1: ((&info->work)#2){+.+.+.}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0 #2: (&__lockdep_no_validate__){......}, at: [<ffffffff813022a1>] device_release_driver+0x21/0x40 #3: (&__lockdep_no_validate__){......}, at: [<ffffffff813022a1>] device_release_driver+0x21/0x40 stack backtrace: Pid: 4, comm: kworker/0:0 Not tainted 3.8.0-rc7-ninja+ #21 Call Trace: [<ffffffff81090213>] validate_chain.isra.33+0xda3/0x1240 [<ffffffff8108ea90>] ? print_shortest_lock_dependencies+0x1c0/0x1c0 [<ffffffff810909d5>] ? mark_lock+0x215/0x5d0 [<ffffffff8109284f>] ? debug_check_no_locks_freed+0x8f/0x160 [<ffffffff8109113c>] __lock_acquire+0x3ac/0xb30 [<ffffffff810927bd>] ? trace_hardirqs_on+0xd/0x10 [<ffffffff8108dc0c>] ? lockdep_init_map+0x9c/0x4d0 [<ffffffff81091d8a>] lock_acquire+0x5a/0x70 [<ffffffff8105ac70>] ? flush_workqueue_prep_cwqs+0x200/0x200 [<ffffffff8105ad58>] flush_workqueue+0xe8/0x4d0 [<ffffffff8105ac70>] ? flush_workqueue_prep_cwqs+0x200/0x200 [<ffffffff8105b1c8>] drain_workqueue+0x68/0x1f0 [<ffffffff8105b363>] destroy_workqueue+0x13/0x160 [<ffffffff8125ad0a>] pciehp_release_ctrl+0x3a/0x90 [<ffffffff81257ca5>] pciehp_remove+0x25/0x30 [<ffffffff81251f72>] pcie_port_remove_service+0x52/0x70 [<ffffffff81302217>] __device_release_driver+0x77/0xe0 [<ffffffff813022a9>] device_release_driver+0x29/0x40 [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140 [<ffffffff812ff847>] device_del+0x127/0x1c0 [<ffffffff812520f0>] ? resume_iter+0x40/0x40 [<ffffffff812ff8f1>] device_unregister+0x11/0x20 [<ffffffff81252125>] remove_iter+0x35/0x40 [<ffffffff812fe716>] device_for_each_child+0x36/0x70 [<ffffffff812526c1>] pcie_port_device_remove+0x21/0x40 [<ffffffff81252908>] pcie_portdrv_remove+0x28/0x50 [<ffffffff81246cb1>] pci_device_remove+0x41/0xc0 [<ffffffff81302217>] __device_release_driver+0x77/0xe0 [<ffffffff813022a9>] device_release_driver+0x29/0x40 [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140 [<ffffffff812ff847>] device_del+0x127/0x1c0 [<ffffffff812ff8f1>] device_unregister+0x11/0x20 [<ffffffff81241b74>] pci_stop_bus_device+0xb4/0xc0 [<ffffffff81241af5>] pci_stop_bus_device+0x35/0xc0 [<ffffffff81241cd1>] pci_stop_and_remove_bus_device+0x11/0x20 [<ffffffff81259021>] pciehp_unconfigure_device+0x91/0x190 [<ffffffff81258921>] pciehp_disable_slot+0x71/0x220 [<ffffffff81258bb6>] pciehp_power_thread+0xe6/0x110 [<ffffffff8105c84a>] process_one_work+0x1ca/0x4e0 [<ffffffff8105c7e0>] ? process_one_work+0x160/0x4e0 [<ffffffff81258ad0>] ? pciehp_disable_slot+0x220/0x220 [<ffffffff8105cefe>] worker_thread+0x14e/0x3f0 [<ffffffff810927bd>] ? trace_hardirqs_on+0xd/0x10 [<ffffffff8105cdb0>] ? rescuer_thread+0x210/0x210 [<ffffffff81063086>] kthread+0xd6/0xe0 [<ffffffff8154cf2b>] ? _raw_spin_unlock_irq+0x2b/0x50 [<ffffffff81062fb0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff8154dc2c>] ret_from_fork+0x7c/0xb0 [<ffffffff81062fb0>] ? __init_kthread_worker+0x70/0x70 -- Daniel J Blueman ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops 2013-02-12 2:18 ` Daniel J Blueman @ 2013-02-12 20:53 ` Rafael J. Wysocki 2013-02-12 20:50 ` Tejun Heo 0 siblings, 1 reply; 12+ messages in thread From: Rafael J. Wysocki @ 2013-02-12 20:53 UTC (permalink / raw) To: Daniel J Blueman, Tejun Heo Cc: Bjorn Helgaas, Linux Kernel, Linux PCI, Yijing Wang On Tuesday, February 12, 2013 10:18:57 AM Daniel J Blueman wrote: > On 12 February 2013 03:49, Rafael J. Wysocki <rjw@sisk.pl> wrote: > > On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote: > >> On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote: > >> > [+cc Rafael] > >> > > >> > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote: > >> > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote: > >> > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a > >> > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released: > >> > >> > >> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3) > >> > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not > >> > >> clear MAC_TX_MODE=ffffffff > >> > >> tg3 0000:0a:00.0 eth0: No firmware running > >> > >> tg3 0000:0a:00.0 eth0: Link is down > >> > >> [sched_delayed] sched: RT throttling activated > >> > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI > >> > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp > >> > >> pci_bus 0000:0a: busn_res: [bus 0a] is released > >> > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released > >> > >> > >> > >> After some activity later (eg I can reproduce this by switching to a > >> > >> text console and back), often we'll see an oops: > >> > >> > >> > >> Unable to handle kernel paging request at 0000000000001070 > >> > >> pci_pme_list_scan+0x3d/0xe0 > >> > >> Call Trace: > >> > >> process_one_work+0x193 > >> > >> ? process_one_work+0x131 > >> > >> ? pci_pme_wakeup+0x60 > >> > >> worker_thread+0x15d > >> > >> > >> > >> (gdb) list *(pci_pme_list_scan+0x3d) > >> > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556). > >> > >> 1551 /* > >> > >> 1552 * If bridge is in low power state, the > >> > >> 1553 * configuration space of subordinate devices > >> > >> 1554 * may be not accessible > >> > >> 1555 */ > >> > >> 1556 if (bridge && bridge->current_state != PCI_D0) > >> > >> 1557 continue; > >> > >> 1558 pci_pme_wakeup(pme_dev->dev, NULL); > >> > >> 1559 } else { > >> > >> 1560 list_del(&pme_dev->list); > >> > >> > >> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch > >> > >> it with EFI pstore), it is almost certainly significant heap > >> > >> corruption; this would explain why pme_dev became null (the load has > >> > >> been ordered ahead). > >> > >> > >> > >> I'll see what I can find out with memory poisoning and list debugging. > >> > > > >> > > Enabling a bunch of related debugging, we see pme_dev is non-null and: > >> > > > >> > > BUG: Unable to handle NULL pointer dereference at > >> > > pci_bus_read_config_word+0x6c > >> > > PGD 26314c067 PUD 2633f9067 PMD 0 > >> > > Oops: 0000 [#1] SMP > >> > > pci_check_pme_status+0x4f > >> > > pci_pme_wakeup+0x21 > >> > > pci_pme_list_scan+0xd5 > >> > > process_one_work+0x1ca > >> > > ? process_one_work+0x160 > >> > > ? pci_pme_wakeup+0x60 > >> > > worker_thread+0x14e > >> > > > >> > > Anyway, it looks like the device being unplugged wasn't removed from > >> > > pci_pme_list as pci_pme_active(dev, false) wasn't called. > >> > > > >> > > From a quick review, I wasn't able to find the right place in the > >> > > call-chain which I only see releases the child busses and PCIe port > >> > > drivers. Anyone? > >> > > >> > It looks like drivers *add* devices to pci_pme_list when they use > >> > pci_enable_wake() or pci_wake_from_d3(). But many drivers never > >> > remove their devices, and I don't see any place where the core does it > >> > either. My guess is we need to remove it in pci_stop_dev() (we > >> > already do pcie_aspm_exit_link_state() there) or somewhere similar. > >> > >> Yes, we should call pci_pme_active(dev, false) somewhere in there I think. > >> It's fine to call that even if PME was not "active" before. > > > > Daniel, I wonder if the patch below helps? > > I had tried with the pci_pme_active call inside the is_added condition > (it'll always hold here); it resolves the issue, though it introduces > a new lockdep warning [1]. > > All said, I don't see any other way except for a patch to abstract the > list entry removal to avoid the unnecessary reads and writes (as it is > called for four devices here), though I don't see how that would alter > the locking behaviour and why we didn't see this lockdep warning > before. > > What do you think? This looks fishy, but I wonder if Tejun has any ideas. Tejun, can you please have a look at the call trace below? It looks like the workqueues code is involved heavily. Thanks, Rafael > --- [1] > > kworker/0:0/4 is trying to acquire lock: > (name){++++.+}, at: [<ffffffff8105ac70>] flush_workqueue+0x0/0x4d0 > > but task is already holding lock: > (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0 > > other info that might help us debug this: > Possible unsafe locking scenario: > > CPU0 > ---- > lock(name); > lock(name); > > *** DEADLOCK *** > > May be due to missing lock nesting notation > > 4 locks held by kworker/0:0/4: > #0: (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0 > #1: ((&info->work)#2){+.+.+.}, at: [<ffffffff8105c7e0>] > process_one_work+0x160/0x4e0 > #2: (&__lockdep_no_validate__){......}, at: [<ffffffff813022a1>] > device_release_driver+0x21/0x40 > #3: (&__lockdep_no_validate__){......}, at: [<ffffffff813022a1>] > device_release_driver+0x21/0x40 > > stack backtrace: > Pid: 4, comm: kworker/0:0 Not tainted 3.8.0-rc7-ninja+ #21 > Call Trace: > [<ffffffff81090213>] validate_chain.isra.33+0xda3/0x1240 > [<ffffffff8108ea90>] ? print_shortest_lock_dependencies+0x1c0/0x1c0 > [<ffffffff810909d5>] ? mark_lock+0x215/0x5d0 > [<ffffffff8109284f>] ? debug_check_no_locks_freed+0x8f/0x160 > [<ffffffff8109113c>] __lock_acquire+0x3ac/0xb30 > [<ffffffff810927bd>] ? trace_hardirqs_on+0xd/0x10 > [<ffffffff8108dc0c>] ? lockdep_init_map+0x9c/0x4d0 > [<ffffffff81091d8a>] lock_acquire+0x5a/0x70 > [<ffffffff8105ac70>] ? flush_workqueue_prep_cwqs+0x200/0x200 > [<ffffffff8105ad58>] flush_workqueue+0xe8/0x4d0 > [<ffffffff8105ac70>] ? flush_workqueue_prep_cwqs+0x200/0x200 > [<ffffffff8105b1c8>] drain_workqueue+0x68/0x1f0 > [<ffffffff8105b363>] destroy_workqueue+0x13/0x160 > [<ffffffff8125ad0a>] pciehp_release_ctrl+0x3a/0x90 > [<ffffffff81257ca5>] pciehp_remove+0x25/0x30 > [<ffffffff81251f72>] pcie_port_remove_service+0x52/0x70 > [<ffffffff81302217>] __device_release_driver+0x77/0xe0 > [<ffffffff813022a9>] device_release_driver+0x29/0x40 > [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140 > [<ffffffff812ff847>] device_del+0x127/0x1c0 > [<ffffffff812520f0>] ? resume_iter+0x40/0x40 > [<ffffffff812ff8f1>] device_unregister+0x11/0x20 > [<ffffffff81252125>] remove_iter+0x35/0x40 > [<ffffffff812fe716>] device_for_each_child+0x36/0x70 > [<ffffffff812526c1>] pcie_port_device_remove+0x21/0x40 > [<ffffffff81252908>] pcie_portdrv_remove+0x28/0x50 > [<ffffffff81246cb1>] pci_device_remove+0x41/0xc0 > [<ffffffff81302217>] __device_release_driver+0x77/0xe0 > [<ffffffff813022a9>] device_release_driver+0x29/0x40 > [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140 > [<ffffffff812ff847>] device_del+0x127/0x1c0 > [<ffffffff812ff8f1>] device_unregister+0x11/0x20 > [<ffffffff81241b74>] pci_stop_bus_device+0xb4/0xc0 > [<ffffffff81241af5>] pci_stop_bus_device+0x35/0xc0 > [<ffffffff81241cd1>] pci_stop_and_remove_bus_device+0x11/0x20 > [<ffffffff81259021>] pciehp_unconfigure_device+0x91/0x190 > [<ffffffff81258921>] pciehp_disable_slot+0x71/0x220 > [<ffffffff81258bb6>] pciehp_power_thread+0xe6/0x110 > [<ffffffff8105c84a>] process_one_work+0x1ca/0x4e0 > [<ffffffff8105c7e0>] ? process_one_work+0x160/0x4e0 > [<ffffffff81258ad0>] ? pciehp_disable_slot+0x220/0x220 > [<ffffffff8105cefe>] worker_thread+0x14e/0x3f0 > [<ffffffff810927bd>] ? trace_hardirqs_on+0xd/0x10 > [<ffffffff8105cdb0>] ? rescuer_thread+0x210/0x210 > [<ffffffff81063086>] kthread+0xd6/0xe0 > [<ffffffff8154cf2b>] ? _raw_spin_unlock_irq+0x2b/0x50 > [<ffffffff81062fb0>] ? __init_kthread_worker+0x70/0x70 > [<ffffffff8154dc2c>] ret_from_fork+0x7c/0xb0 > [<ffffffff81062fb0>] ? __init_kthread_worker+0x70/0x70 > -- I speak only for myself. Rafael J. Wysocki, Intel Open Source Technology Center. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops 2013-02-12 20:53 ` Rafael J. Wysocki @ 2013-02-12 20:50 ` Tejun Heo 2013-02-12 21:00 ` Rafael J. Wysocki 0 siblings, 1 reply; 12+ messages in thread From: Tejun Heo @ 2013-02-12 20:50 UTC (permalink / raw) To: Rafael J. Wysocki Cc: Daniel J Blueman, Bjorn Helgaas, Linux Kernel, Linux PCI, Yijing Wang Hey, Rafael. On Tue, Feb 12, 2013 at 09:53:08PM +0100, Rafael J. Wysocki wrote: > This looks fishy, but I wonder if Tejun has any ideas. > > Tejun, can you please have a look at the call trace below? It looks like > the workqueues code is involved heavily. > > > > kworker/0:0/4 is trying to acquire lock: > > (name){++++.+}, at: [<ffffffff8105ac70>] flush_workqueue+0x0/0x4d0 > > > > but task is already holding lock: > > (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0 It's basically saying that a work item is trying to flush the workqueue it's currently executing on, at least in lockdep's eyes. > > stack backtrace: > > Pid: 4, comm: kworker/0:0 Not tainted 3.8.0-rc7-ninja+ #21 > > Call Trace: > > [<ffffffff81090213>] validate_chain.isra.33+0xda3/0x1240 > > [<ffffffff8109113c>] __lock_acquire+0x3ac/0xb30 > > [<ffffffff81091d8a>] lock_acquire+0x5a/0x70 > > [<ffffffff8105ad58>] flush_workqueue+0xe8/0x4d0 > > [<ffffffff8105b1c8>] drain_workqueue+0x68/0x1f0 > > [<ffffffff8105b363>] destroy_workqueue+0x13/0x160 And the flush is from workqueue destruction > > [<ffffffff8125ad0a>] pciehp_release_ctrl+0x3a/0x90 > > [<ffffffff81257ca5>] pciehp_remove+0x25/0x30 > > [<ffffffff81251f72>] pcie_port_remove_service+0x52/0x70 > > [<ffffffff81302217>] __device_release_driver+0x77/0xe0 > > [<ffffffff813022a9>] device_release_driver+0x29/0x40 > > [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140 > > [<ffffffff812ff847>] device_del+0x127/0x1c0 > > [<ffffffff812ff8f1>] device_unregister+0x11/0x20 > > [<ffffffff81252125>] remove_iter+0x35/0x40 > > [<ffffffff812fe716>] device_for_each_child+0x36/0x70 > > [<ffffffff812526c1>] pcie_port_device_remove+0x21/0x40 > > [<ffffffff81252908>] pcie_portdrv_remove+0x28/0x50 > > [<ffffffff81246cb1>] pci_device_remove+0x41/0xc0 > > [<ffffffff81302217>] __device_release_driver+0x77/0xe0 > > [<ffffffff813022a9>] device_release_driver+0x29/0x40 > > [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140 > > [<ffffffff812ff847>] device_del+0x127/0x1c0 > > [<ffffffff812ff8f1>] device_unregister+0x11/0x20 > > [<ffffffff81241b74>] pci_stop_bus_device+0xb4/0xc0 > > [<ffffffff81241af5>] pci_stop_bus_device+0x35/0xc0 > > [<ffffffff81241cd1>] pci_stop_and_remove_bus_device+0x11/0x20 > > [<ffffffff81259021>] pciehp_unconfigure_device+0x91/0x190 > > [<ffffffff81258921>] pciehp_disable_slot+0x71/0x220 > > [<ffffffff81258bb6>] pciehp_power_thread+0xe6/0x110 > > [<ffffffff8105c84a>] process_one_work+0x1ca/0x4e0 running from a workqueue which probably is at least transitively related to the workqueue being destroyed. Does this lead to an actual deadlock? Thanks. -- tejun ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops 2013-02-12 20:50 ` Tejun Heo @ 2013-02-12 21:00 ` Rafael J. Wysocki 0 siblings, 0 replies; 12+ messages in thread From: Rafael J. Wysocki @ 2013-02-12 21:00 UTC (permalink / raw) To: Tejun Heo, Daniel J Blueman Cc: Bjorn Helgaas, Linux Kernel, Linux PCI, Yijing Wang On Tuesday, February 12, 2013 12:50:38 PM Tejun Heo wrote: > Hey, Rafael. > > On Tue, Feb 12, 2013 at 09:53:08PM +0100, Rafael J. Wysocki wrote: > > This looks fishy, but I wonder if Tejun has any ideas. > > > > Tejun, can you please have a look at the call trace below? It looks like > > the workqueues code is involved heavily. > > > > > > kworker/0:0/4 is trying to acquire lock: > > > (name){++++.+}, at: [<ffffffff8105ac70>] flush_workqueue+0x0/0x4d0 > > > > > > but task is already holding lock: > > > (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0 > > It's basically saying that a work item is trying to flush the > workqueue it's currently executing on, at least in lockdep's eyes. > > > > stack backtrace: > > > Pid: 4, comm: kworker/0:0 Not tainted 3.8.0-rc7-ninja+ #21 > > > Call Trace: > > > [<ffffffff81090213>] validate_chain.isra.33+0xda3/0x1240 > > > [<ffffffff8109113c>] __lock_acquire+0x3ac/0xb30 > > > [<ffffffff81091d8a>] lock_acquire+0x5a/0x70 > > > [<ffffffff8105ad58>] flush_workqueue+0xe8/0x4d0 > > > [<ffffffff8105b1c8>] drain_workqueue+0x68/0x1f0 > > > [<ffffffff8105b363>] destroy_workqueue+0x13/0x160 > > And the flush is from workqueue destruction > > > > [<ffffffff8125ad0a>] pciehp_release_ctrl+0x3a/0x90 > > > [<ffffffff81257ca5>] pciehp_remove+0x25/0x30 > > > [<ffffffff81251f72>] pcie_port_remove_service+0x52/0x70 > > > [<ffffffff81302217>] __device_release_driver+0x77/0xe0 > > > [<ffffffff813022a9>] device_release_driver+0x29/0x40 > > > [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140 > > > [<ffffffff812ff847>] device_del+0x127/0x1c0 > > > [<ffffffff812ff8f1>] device_unregister+0x11/0x20 > > > [<ffffffff81252125>] remove_iter+0x35/0x40 > > > [<ffffffff812fe716>] device_for_each_child+0x36/0x70 > > > [<ffffffff812526c1>] pcie_port_device_remove+0x21/0x40 > > > [<ffffffff81252908>] pcie_portdrv_remove+0x28/0x50 > > > [<ffffffff81246cb1>] pci_device_remove+0x41/0xc0 > > > [<ffffffff81302217>] __device_release_driver+0x77/0xe0 > > > [<ffffffff813022a9>] device_release_driver+0x29/0x40 > > > [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140 > > > [<ffffffff812ff847>] device_del+0x127/0x1c0 > > > [<ffffffff812ff8f1>] device_unregister+0x11/0x20 > > > [<ffffffff81241b74>] pci_stop_bus_device+0xb4/0xc0 > > > [<ffffffff81241af5>] pci_stop_bus_device+0x35/0xc0 > > > [<ffffffff81241cd1>] pci_stop_and_remove_bus_device+0x11/0x20 > > > [<ffffffff81259021>] pciehp_unconfigure_device+0x91/0x190 > > > [<ffffffff81258921>] pciehp_disable_slot+0x71/0x220 > > > [<ffffffff81258bb6>] pciehp_power_thread+0xe6/0x110 > > > [<ffffffff8105c84a>] process_one_work+0x1ca/0x4e0 > > running from a workqueue which probably is at least transitively > related to the workqueue being destroyed. Does this lead to an actual > deadlock? Might be. I need to have a deeper look at things in the acpiphp land. Daniel, I'm quite sure it isn't related to the addition of the pci_pme_active() call. Thanks, Rafael -- I speak only for myself. Rafael J. Wysocki, Intel Open Source Technology Center. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops 2013-02-11 19:49 ` Rafael J. Wysocki 2013-02-12 2:18 ` Daniel J Blueman @ 2013-02-13 14:32 ` Daniel J Blueman 2013-02-13 19:29 ` Bjorn Helgaas 1 sibling, 1 reply; 12+ messages in thread From: Daniel J Blueman @ 2013-02-13 14:32 UTC (permalink / raw) To: Rafael J. Wysocki; +Cc: Bjorn Helgaas, Linux Kernel, Linux PCI, Yijing Wang On 12 February 2013 03:49, Rafael J. Wysocki <rjw@sisk.pl> wrote: > On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote: >> On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote: >> > [+cc Rafael] >> > >> > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote: >> > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote: >> > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a >> > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released: >> > >> >> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3) >> > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not >> > >> clear MAC_TX_MODE=ffffffff >> > >> tg3 0000:0a:00.0 eth0: No firmware running >> > >> tg3 0000:0a:00.0 eth0: Link is down >> > >> [sched_delayed] sched: RT throttling activated >> > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI >> > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp >> > >> pci_bus 0000:0a: busn_res: [bus 0a] is released >> > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released >> > >> >> > >> After some activity later (eg I can reproduce this by switching to a >> > >> text console and back), often we'll see an oops: >> > >> >> > >> Unable to handle kernel paging request at 0000000000001070 >> > >> pci_pme_list_scan+0x3d/0xe0 >> > >> Call Trace: >> > >> process_one_work+0x193 >> > >> ? process_one_work+0x131 >> > >> ? pci_pme_wakeup+0x60 >> > >> worker_thread+0x15d >> > >> >> > >> (gdb) list *(pci_pme_list_scan+0x3d) >> > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556). >> > >> 1551 /* >> > >> 1552 * If bridge is in low power state, the >> > >> 1553 * configuration space of subordinate devices >> > >> 1554 * may be not accessible >> > >> 1555 */ >> > >> 1556 if (bridge && bridge->current_state != PCI_D0) >> > >> 1557 continue; >> > >> 1558 pci_pme_wakeup(pme_dev->dev, NULL); >> > >> 1559 } else { >> > >> 1560 list_del(&pme_dev->list); >> > >> >> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch >> > >> it with EFI pstore), it is almost certainly significant heap >> > >> corruption; this would explain why pme_dev became null (the load has >> > >> been ordered ahead). >> > >> >> > >> I'll see what I can find out with memory poisoning and list debugging. >> > > >> > > Enabling a bunch of related debugging, we see pme_dev is non-null and: >> > > >> > > BUG: Unable to handle NULL pointer dereference at >> > > pci_bus_read_config_word+0x6c >> > > PGD 26314c067 PUD 2633f9067 PMD 0 >> > > Oops: 0000 [#1] SMP >> > > pci_check_pme_status+0x4f >> > > pci_pme_wakeup+0x21 >> > > pci_pme_list_scan+0xd5 >> > > process_one_work+0x1ca >> > > ? process_one_work+0x160 >> > > ? pci_pme_wakeup+0x60 >> > > worker_thread+0x14e >> > > >> > > Anyway, it looks like the device being unplugged wasn't removed from >> > > pci_pme_list as pci_pme_active(dev, false) wasn't called. >> > > >> > > From a quick review, I wasn't able to find the right place in the >> > > call-chain which I only see releases the child busses and PCIe port >> > > drivers. Anyone? >> > >> > It looks like drivers *add* devices to pci_pme_list when they use >> > pci_enable_wake() or pci_wake_from_d3(). But many drivers never >> > remove their devices, and I don't see any place where the core does it >> > either. My guess is we need to remove it in pci_stop_dev() (we >> > already do pcie_aspm_exit_link_state() there) or somewhere similar. >> >> Yes, we should call pci_pme_active(dev, false) somewhere in there I think. >> It's fine to call that even if PME was not "active" before. > > Daniel, I wonder if the patch below helps? > > Rafael > > > Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > --- > drivers/pci/remove.c | 2 ++ > 1 file changed, 2 insertions(+) > > Index: test/drivers/pci/remove.c > =================================================================== > --- test.orig/drivers/pci/remove.c > +++ test/drivers/pci/remove.c > @@ -19,6 +19,8 @@ static void pci_free_resources(struct pc > > static void pci_stop_dev(struct pci_dev *dev) > { > + pci_pme_active(dev, false); > + > if (dev->is_added) { > pci_proc_detach_device(dev); > pci_remove_sysfs_dev_files(dev); Indeed, the oops would prevent hitting the lockdep warning, which is a secondary matter. As it stands, this fix prevents the fatal oops, so IMHO is urgent material for 3.8-final. I can't prove that is_added will always be true, so let's stick with your patch. Tested-by: Daniel J Blueman <daniel@quora.org> -- Daniel J Blueman ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops 2013-02-13 14:32 ` Daniel J Blueman @ 2013-02-13 19:29 ` Bjorn Helgaas 2013-02-13 20:55 ` Rafael J. Wysocki 0 siblings, 1 reply; 12+ messages in thread From: Bjorn Helgaas @ 2013-02-13 19:29 UTC (permalink / raw) To: Daniel J Blueman; +Cc: Rafael J. Wysocki, Linux Kernel, Linux PCI, Yijing Wang On Wed, Feb 13, 2013 at 7:32 AM, Daniel J Blueman <daniel@quora.org> wrote: > On 12 February 2013 03:49, Rafael J. Wysocki <rjw@sisk.pl> wrote: >> On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote: >>> On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote: >>> > [+cc Rafael] >>> > >>> > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote: >>> > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote: >>> > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a >>> > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released: >>> > >> >>> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3) >>> > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not >>> > >> clear MAC_TX_MODE=ffffffff >>> > >> tg3 0000:0a:00.0 eth0: No firmware running >>> > >> tg3 0000:0a:00.0 eth0: Link is down >>> > >> [sched_delayed] sched: RT throttling activated >>> > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI >>> > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp >>> > >> pci_bus 0000:0a: busn_res: [bus 0a] is released >>> > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released >>> > >> >>> > >> After some activity later (eg I can reproduce this by switching to a >>> > >> text console and back), often we'll see an oops: >>> > >> >>> > >> Unable to handle kernel paging request at 0000000000001070 >>> > >> pci_pme_list_scan+0x3d/0xe0 >>> > >> Call Trace: >>> > >> process_one_work+0x193 >>> > >> ? process_one_work+0x131 >>> > >> ? pci_pme_wakeup+0x60 >>> > >> worker_thread+0x15d >>> > >> >>> > >> (gdb) list *(pci_pme_list_scan+0x3d) >>> > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556). >>> > >> 1551 /* >>> > >> 1552 * If bridge is in low power state, the >>> > >> 1553 * configuration space of subordinate devices >>> > >> 1554 * may be not accessible >>> > >> 1555 */ >>> > >> 1556 if (bridge && bridge->current_state != PCI_D0) >>> > >> 1557 continue; >>> > >> 1558 pci_pme_wakeup(pme_dev->dev, NULL); >>> > >> 1559 } else { >>> > >> 1560 list_del(&pme_dev->list); >>> > >> >>> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch >>> > >> it with EFI pstore), it is almost certainly significant heap >>> > >> corruption; this would explain why pme_dev became null (the load has >>> > >> been ordered ahead). >>> > >> >>> > >> I'll see what I can find out with memory poisoning and list debugging. >>> > > >>> > > Enabling a bunch of related debugging, we see pme_dev is non-null and: >>> > > >>> > > BUG: Unable to handle NULL pointer dereference at >>> > > pci_bus_read_config_word+0x6c >>> > > PGD 26314c067 PUD 2633f9067 PMD 0 >>> > > Oops: 0000 [#1] SMP >>> > > pci_check_pme_status+0x4f >>> > > pci_pme_wakeup+0x21 >>> > > pci_pme_list_scan+0xd5 >>> > > process_one_work+0x1ca >>> > > ? process_one_work+0x160 >>> > > ? pci_pme_wakeup+0x60 >>> > > worker_thread+0x14e >>> > > >>> > > Anyway, it looks like the device being unplugged wasn't removed from >>> > > pci_pme_list as pci_pme_active(dev, false) wasn't called. >>> > > >>> > > From a quick review, I wasn't able to find the right place in the >>> > > call-chain which I only see releases the child busses and PCIe port >>> > > drivers. Anyone? >>> > >>> > It looks like drivers *add* devices to pci_pme_list when they use >>> > pci_enable_wake() or pci_wake_from_d3(). But many drivers never >>> > remove their devices, and I don't see any place where the core does it >>> > either. My guess is we need to remove it in pci_stop_dev() (we >>> > already do pcie_aspm_exit_link_state() there) or somewhere similar. >>> >>> Yes, we should call pci_pme_active(dev, false) somewhere in there I think. >>> It's fine to call that even if PME was not "active" before. >> >> Daniel, I wonder if the patch below helps? >> >> Rafael >> >> >> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> >> --- >> drivers/pci/remove.c | 2 ++ >> 1 file changed, 2 insertions(+) >> >> Index: test/drivers/pci/remove.c >> =================================================================== >> --- test.orig/drivers/pci/remove.c >> +++ test/drivers/pci/remove.c >> @@ -19,6 +19,8 @@ static void pci_free_resources(struct pc >> >> static void pci_stop_dev(struct pci_dev *dev) >> { >> + pci_pme_active(dev, false); >> + >> if (dev->is_added) { >> pci_proc_detach_device(dev); >> pci_remove_sysfs_dev_files(dev); > > Indeed, the oops would prevent hitting the lockdep warning, which is a > secondary matter. > > As it stands, this fix prevents the fatal oops, so IMHO is urgent > material for 3.8-final. I can't prove that is_added will always be > true, so let's stick with your patch. > > Tested-by: Daniel J Blueman <daniel@quora.org> I wrote a changelog and applied this to my for-linus branch. Please take a look and confirm that it makes sense before I ask Linus to pull it. http://git.kernel.org/?p=linux/kernel/git/helgaas/pci.git;a=commitdiff;h=249bfb83cf8ba658955f0245ac3981d941f746ee ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops 2013-02-13 19:29 ` Bjorn Helgaas @ 2013-02-13 20:55 ` Rafael J. Wysocki 0 siblings, 0 replies; 12+ messages in thread From: Rafael J. Wysocki @ 2013-02-13 20:55 UTC (permalink / raw) To: Bjorn Helgaas; +Cc: Daniel J Blueman, Linux Kernel, Linux PCI, Yijing Wang On Wednesday, February 13, 2013 12:29:59 PM Bjorn Helgaas wrote: > On Wed, Feb 13, 2013 at 7:32 AM, Daniel J Blueman <daniel@quora.org> wrote: > > On 12 February 2013 03:49, Rafael J. Wysocki <rjw@sisk.pl> wrote: > >> On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote: > >>> On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote: > >>> > [+cc Rafael] > >>> > > >>> > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote: > >>> > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote: > >>> > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a > >>> > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released: > >>> > >> > >>> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3) > >>> > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not > >>> > >> clear MAC_TX_MODE=ffffffff > >>> > >> tg3 0000:0a:00.0 eth0: No firmware running > >>> > >> tg3 0000:0a:00.0 eth0: Link is down > >>> > >> [sched_delayed] sched: RT throttling activated > >>> > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI > >>> > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp > >>> > >> pci_bus 0000:0a: busn_res: [bus 0a] is released > >>> > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released > >>> > >> > >>> > >> After some activity later (eg I can reproduce this by switching to a > >>> > >> text console and back), often we'll see an oops: > >>> > >> > >>> > >> Unable to handle kernel paging request at 0000000000001070 > >>> > >> pci_pme_list_scan+0x3d/0xe0 > >>> > >> Call Trace: > >>> > >> process_one_work+0x193 > >>> > >> ? process_one_work+0x131 > >>> > >> ? pci_pme_wakeup+0x60 > >>> > >> worker_thread+0x15d > >>> > >> > >>> > >> (gdb) list *(pci_pme_list_scan+0x3d) > >>> > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556). > >>> > >> 1551 /* > >>> > >> 1552 * If bridge is in low power state, the > >>> > >> 1553 * configuration space of subordinate devices > >>> > >> 1554 * may be not accessible > >>> > >> 1555 */ > >>> > >> 1556 if (bridge && bridge->current_state != PCI_D0) > >>> > >> 1557 continue; > >>> > >> 1558 pci_pme_wakeup(pme_dev->dev, NULL); > >>> > >> 1559 } else { > >>> > >> 1560 list_del(&pme_dev->list); > >>> > >> > >>> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch > >>> > >> it with EFI pstore), it is almost certainly significant heap > >>> > >> corruption; this would explain why pme_dev became null (the load has > >>> > >> been ordered ahead). > >>> > >> > >>> > >> I'll see what I can find out with memory poisoning and list debugging. > >>> > > > >>> > > Enabling a bunch of related debugging, we see pme_dev is non-null and: > >>> > > > >>> > > BUG: Unable to handle NULL pointer dereference at > >>> > > pci_bus_read_config_word+0x6c > >>> > > PGD 26314c067 PUD 2633f9067 PMD 0 > >>> > > Oops: 0000 [#1] SMP > >>> > > pci_check_pme_status+0x4f > >>> > > pci_pme_wakeup+0x21 > >>> > > pci_pme_list_scan+0xd5 > >>> > > process_one_work+0x1ca > >>> > > ? process_one_work+0x160 > >>> > > ? pci_pme_wakeup+0x60 > >>> > > worker_thread+0x14e > >>> > > > >>> > > Anyway, it looks like the device being unplugged wasn't removed from > >>> > > pci_pme_list as pci_pme_active(dev, false) wasn't called. > >>> > > > >>> > > From a quick review, I wasn't able to find the right place in the > >>> > > call-chain which I only see releases the child busses and PCIe port > >>> > > drivers. Anyone? > >>> > > >>> > It looks like drivers *add* devices to pci_pme_list when they use > >>> > pci_enable_wake() or pci_wake_from_d3(). But many drivers never > >>> > remove their devices, and I don't see any place where the core does it > >>> > either. My guess is we need to remove it in pci_stop_dev() (we > >>> > already do pcie_aspm_exit_link_state() there) or somewhere similar. > >>> > >>> Yes, we should call pci_pme_active(dev, false) somewhere in there I think. > >>> It's fine to call that even if PME was not "active" before. > >> > >> Daniel, I wonder if the patch below helps? > >> > >> Rafael > >> > >> > >> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com> > >> --- > >> drivers/pci/remove.c | 2 ++ > >> 1 file changed, 2 insertions(+) > >> > >> Index: test/drivers/pci/remove.c > >> =================================================================== > >> --- test.orig/drivers/pci/remove.c > >> +++ test/drivers/pci/remove.c > >> @@ -19,6 +19,8 @@ static void pci_free_resources(struct pc > >> > >> static void pci_stop_dev(struct pci_dev *dev) > >> { > >> + pci_pme_active(dev, false); > >> + > >> if (dev->is_added) { > >> pci_proc_detach_device(dev); > >> pci_remove_sysfs_dev_files(dev); > > > > Indeed, the oops would prevent hitting the lockdep warning, which is a > > secondary matter. > > > > As it stands, this fix prevents the fatal oops, so IMHO is urgent > > material for 3.8-final. I can't prove that is_added will always be > > true, so let's stick with your patch. > > > > Tested-by: Daniel J Blueman <daniel@quora.org> > > I wrote a changelog and applied this to my for-linus branch. Please > take a look and confirm that it makes sense before I ask Linus to pull > it. > > http://git.kernel.org/?p=linux/kernel/git/helgaas/pci.git;a=commitdiff;h=249bfb83cf8ba658955f0245ac3981d941f746ee Yes, it does, thanks a lot! Rafael -- I speak only for myself. Rafael J. Wysocki, Intel Open Source Technology Center. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2013-02-13 20:48 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-02-11 13:03 [3.8-rc7] PCI hotplug wakeup oops Daniel J Blueman 2013-02-11 17:08 ` Daniel J Blueman 2013-02-11 19:01 ` Bjorn Helgaas 2013-02-11 19:27 ` Rafael J. Wysocki 2013-02-11 19:49 ` Rafael J. Wysocki 2013-02-12 2:18 ` Daniel J Blueman 2013-02-12 20:53 ` Rafael J. Wysocki 2013-02-12 20:50 ` Tejun Heo 2013-02-12 21:00 ` Rafael J. Wysocki 2013-02-13 14:32 ` Daniel J Blueman 2013-02-13 19:29 ` Bjorn Helgaas 2013-02-13 20:55 ` Rafael J. Wysocki
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).