linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [3.8-rc7] PCI hotplug wakeup oops
@ 2013-02-11 13:03 Daniel J Blueman
  2013-02-11 17:08 ` Daniel J Blueman
  0 siblings, 1 reply; 12+ messages in thread
From: Daniel J Blueman @ 2013-02-11 13:03 UTC (permalink / raw)
  To: Linux Kernel, Linux PCI

With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a
[1]) on a Macbook Pro 10,1, we see the PCIe port correctly released:

pciehp 0000:06:03.0:pcie24: Card not present on Slot(3)
tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not
clear MAC_TX_MODE=ffffffff
tg3 0000:0a:00.0 eth0: No firmware running
tg3 0000:0a:00.0 eth0: Link is down
[sched_delayed] sched: RT throttling activated
pcieport 0000:00:01.1: System wakeup enabled by ACPI
pciehp 0000:09:00.0:pcie24: unloading service driver pciehp
pci_bus 0000:0a: busn_res: [bus 0a] is released
pci_bus 0000:09: busn_res: [bus 09-0a] is released

After some activity later (eg I can reproduce this by switching to a
text console and back), often we'll see an oops:

Unable to handle kernel paging request at 0000000000001070
pci_pme_list_scan+0x3d/0xe0
Call Trace:
process_one_work+0x193
? process_one_work+0x131
? pci_pme_wakeup+0x60
worker_thread+0x15d

(gdb) list *(pci_pme_list_scan+0x3d)
0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556).
1551					/*
1552					 * If bridge is in low power state, the
1553					 * configuration space of subordinate devices
1554					 * may be not accessible
1555					 */
1556					if (bridge && bridge->current_state != PCI_D0)
1557						continue;
1558					pci_pme_wakeup(pme_dev->dev, NULL);
1559				} else {
1560					list_del(&pme_dev->list);

Since a panic in vsnprintf happens after the oops (hence I can't catch
it with EFI pstore), it is almost certainly significant heap
corruption; this would explain why pme_dev became null (the load has
been ordered ahead).

I'll see what I can find out with memory poisoning and list debugging.

Thanks,
  Daniel

--- [1]

$ lspci
00:00.0 Host bridge: Intel Corporation 3rd Gen Core processor DRAM
Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core
processor PCI Express Root Port (rev 09)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core
processor PCI Express Root Port (rev 09)
00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core
processor PCI Express Root Port (rev 09)
00:02.0 VGA compatible controller: Intel Corporation 3rd Gen Core
processor Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C210
Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset
Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset
Family PCI Express Root Port 1 (rev c4)
00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset
Family PCI Express Root Port 2 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
Family USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation HM77 Express Chipset LPC
Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 7 Series Chipset Family
6-port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family
SMBus Controller (rev 04)
01:00.0 VGA compatible controller: NVIDIA Corporation Device 0fd5 (rev ff)
01:00.1 Audio device: NVIDIA Corporation Device 0e1b (rev ff)
03:00.0 Ethernet controller: Broadcom Corporation Device 16a3 (rev 10)
03:00.1 SD Host controller: Broadcom Corporation NetXtreme BCM57765
Memory Card Reader (rev 10)
04:00.0 Network controller: Broadcom Corporation BCM4331 802.11a/b/g/n (rev 02)
05:00.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus
Ridge] (rev 03)
06:00.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus
Ridge] (rev 03)
06:03.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus
Ridge] (rev 03)
06:04.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus
Ridge] (rev 03)
06:05.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus
Ridge] (rev 03)
06:06.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Port [Cactus
Ridge] (rev 03)
07:00.0 System peripheral: Intel Corporation DSL3510 Thunderbolt Port
[Cactus Ridge] (rev 03)
08:00.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Controller
[Cactus Ridge]
09:00.0 PCI bridge: Intel Corporation DSL3510 Thunderbolt Controller
[Cactus Ridge]
0a:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM57762
Gigabit Ethernet PCIe
-- 
Daniel J Blueman

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops
  2013-02-11 13:03 [3.8-rc7] PCI hotplug wakeup oops Daniel J Blueman
@ 2013-02-11 17:08 ` Daniel J Blueman
  2013-02-11 19:01   ` Bjorn Helgaas
  0 siblings, 1 reply; 12+ messages in thread
From: Daniel J Blueman @ 2013-02-11 17:08 UTC (permalink / raw)
  To: Linux Kernel, Linux PCI; +Cc: Yijing Wang, Bjorn Helgaas

On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote:
> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a
> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released:
>
> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3)
> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not
> clear MAC_TX_MODE=ffffffff
> tg3 0000:0a:00.0 eth0: No firmware running
> tg3 0000:0a:00.0 eth0: Link is down
> [sched_delayed] sched: RT throttling activated
> pcieport 0000:00:01.1: System wakeup enabled by ACPI
> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp
> pci_bus 0000:0a: busn_res: [bus 0a] is released
> pci_bus 0000:09: busn_res: [bus 09-0a] is released
>
> After some activity later (eg I can reproduce this by switching to a
> text console and back), often we'll see an oops:
>
> Unable to handle kernel paging request at 0000000000001070
> pci_pme_list_scan+0x3d/0xe0
> Call Trace:
> process_one_work+0x193
> ? process_one_work+0x131
> ? pci_pme_wakeup+0x60
> worker_thread+0x15d
>
> (gdb) list *(pci_pme_list_scan+0x3d)
> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556).
> 1551                                    /*
> 1552                                     * If bridge is in low power state, the
> 1553                                     * configuration space of subordinate devices
> 1554                                     * may be not accessible
> 1555                                     */
> 1556                                    if (bridge && bridge->current_state != PCI_D0)
> 1557                                            continue;
> 1558                                    pci_pme_wakeup(pme_dev->dev, NULL);
> 1559                            } else {
> 1560                                    list_del(&pme_dev->list);
>
> Since a panic in vsnprintf happens after the oops (hence I can't catch
> it with EFI pstore), it is almost certainly significant heap
> corruption; this would explain why pme_dev became null (the load has
> been ordered ahead).
>
> I'll see what I can find out with memory poisoning and list debugging.

Enabling a bunch of related debugging, we see pme_dev is non-null and:

BUG: Unable to handle NULL pointer dereference at
pci_bus_read_config_word+0x6c
PGD 26314c067 PUD 2633f9067 PMD 0
Oops: 0000 [#1] SMP
pci_check_pme_status+0x4f
pci_pme_wakeup+0x21
pci_pme_list_scan+0xd5
process_one_work+0x1ca
? process_one_work+0x160
? pci_pme_wakeup+0x60
worker_thread+0x14e

Anyway, it looks like the device being unplugged wasn't removed from
pci_pme_list as pci_pme_active(dev, false) wasn't called.

>From a quick review, I wasn't able to find the right place in the
call-chain which I only see releases the child busses and PCIe port
drivers. Anyone?

Thanks,
  Daniel
-- 
Daniel J Blueman

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops
  2013-02-11 17:08 ` Daniel J Blueman
@ 2013-02-11 19:01   ` Bjorn Helgaas
  2013-02-11 19:27     ` Rafael J. Wysocki
  0 siblings, 1 reply; 12+ messages in thread
From: Bjorn Helgaas @ 2013-02-11 19:01 UTC (permalink / raw)
  To: Daniel J Blueman; +Cc: Linux Kernel, Linux PCI, Yijing Wang, Rafael J. Wysocki

[+cc Rafael]

On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote:
> On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote:
>> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a
>> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released:
>>
>> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3)
>> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not
>> clear MAC_TX_MODE=ffffffff
>> tg3 0000:0a:00.0 eth0: No firmware running
>> tg3 0000:0a:00.0 eth0: Link is down
>> [sched_delayed] sched: RT throttling activated
>> pcieport 0000:00:01.1: System wakeup enabled by ACPI
>> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp
>> pci_bus 0000:0a: busn_res: [bus 0a] is released
>> pci_bus 0000:09: busn_res: [bus 09-0a] is released
>>
>> After some activity later (eg I can reproduce this by switching to a
>> text console and back), often we'll see an oops:
>>
>> Unable to handle kernel paging request at 0000000000001070
>> pci_pme_list_scan+0x3d/0xe0
>> Call Trace:
>> process_one_work+0x193
>> ? process_one_work+0x131
>> ? pci_pme_wakeup+0x60
>> worker_thread+0x15d
>>
>> (gdb) list *(pci_pme_list_scan+0x3d)
>> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556).
>> 1551                                    /*
>> 1552                                     * If bridge is in low power state, the
>> 1553                                     * configuration space of subordinate devices
>> 1554                                     * may be not accessible
>> 1555                                     */
>> 1556                                    if (bridge && bridge->current_state != PCI_D0)
>> 1557                                            continue;
>> 1558                                    pci_pme_wakeup(pme_dev->dev, NULL);
>> 1559                            } else {
>> 1560                                    list_del(&pme_dev->list);
>>
>> Since a panic in vsnprintf happens after the oops (hence I can't catch
>> it with EFI pstore), it is almost certainly significant heap
>> corruption; this would explain why pme_dev became null (the load has
>> been ordered ahead).
>>
>> I'll see what I can find out with memory poisoning and list debugging.
>
> Enabling a bunch of related debugging, we see pme_dev is non-null and:
>
> BUG: Unable to handle NULL pointer dereference at
> pci_bus_read_config_word+0x6c
> PGD 26314c067 PUD 2633f9067 PMD 0
> Oops: 0000 [#1] SMP
> pci_check_pme_status+0x4f
> pci_pme_wakeup+0x21
> pci_pme_list_scan+0xd5
> process_one_work+0x1ca
> ? process_one_work+0x160
> ? pci_pme_wakeup+0x60
> worker_thread+0x14e
>
> Anyway, it looks like the device being unplugged wasn't removed from
> pci_pme_list as pci_pme_active(dev, false) wasn't called.
>
> From a quick review, I wasn't able to find the right place in the
> call-chain which I only see releases the child busses and PCIe port
> drivers. Anyone?

It looks like drivers *add* devices to pci_pme_list when they use
pci_enable_wake() or pci_wake_from_d3().  But many drivers never
remove their devices, and I don't see any place where the core does it
either.  My guess is we need to remove it in pci_stop_dev() (we
already do pcie_aspm_exit_link_state() there) or somewhere similar.

Bjorn

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops
  2013-02-11 19:01   ` Bjorn Helgaas
@ 2013-02-11 19:27     ` Rafael J. Wysocki
  2013-02-11 19:49       ` Rafael J. Wysocki
  0 siblings, 1 reply; 12+ messages in thread
From: Rafael J. Wysocki @ 2013-02-11 19:27 UTC (permalink / raw)
  To: Bjorn Helgaas; +Cc: Daniel J Blueman, Linux Kernel, Linux PCI, Yijing Wang

On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote:
> [+cc Rafael]
> 
> On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote:
> > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote:
> >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a
> >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released:
> >>
> >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3)
> >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not
> >> clear MAC_TX_MODE=ffffffff
> >> tg3 0000:0a:00.0 eth0: No firmware running
> >> tg3 0000:0a:00.0 eth0: Link is down
> >> [sched_delayed] sched: RT throttling activated
> >> pcieport 0000:00:01.1: System wakeup enabled by ACPI
> >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp
> >> pci_bus 0000:0a: busn_res: [bus 0a] is released
> >> pci_bus 0000:09: busn_res: [bus 09-0a] is released
> >>
> >> After some activity later (eg I can reproduce this by switching to a
> >> text console and back), often we'll see an oops:
> >>
> >> Unable to handle kernel paging request at 0000000000001070
> >> pci_pme_list_scan+0x3d/0xe0
> >> Call Trace:
> >> process_one_work+0x193
> >> ? process_one_work+0x131
> >> ? pci_pme_wakeup+0x60
> >> worker_thread+0x15d
> >>
> >> (gdb) list *(pci_pme_list_scan+0x3d)
> >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556).
> >> 1551                                    /*
> >> 1552                                     * If bridge is in low power state, the
> >> 1553                                     * configuration space of subordinate devices
> >> 1554                                     * may be not accessible
> >> 1555                                     */
> >> 1556                                    if (bridge && bridge->current_state != PCI_D0)
> >> 1557                                            continue;
> >> 1558                                    pci_pme_wakeup(pme_dev->dev, NULL);
> >> 1559                            } else {
> >> 1560                                    list_del(&pme_dev->list);
> >>
> >> Since a panic in vsnprintf happens after the oops (hence I can't catch
> >> it with EFI pstore), it is almost certainly significant heap
> >> corruption; this would explain why pme_dev became null (the load has
> >> been ordered ahead).
> >>
> >> I'll see what I can find out with memory poisoning and list debugging.
> >
> > Enabling a bunch of related debugging, we see pme_dev is non-null and:
> >
> > BUG: Unable to handle NULL pointer dereference at
> > pci_bus_read_config_word+0x6c
> > PGD 26314c067 PUD 2633f9067 PMD 0
> > Oops: 0000 [#1] SMP
> > pci_check_pme_status+0x4f
> > pci_pme_wakeup+0x21
> > pci_pme_list_scan+0xd5
> > process_one_work+0x1ca
> > ? process_one_work+0x160
> > ? pci_pme_wakeup+0x60
> > worker_thread+0x14e
> >
> > Anyway, it looks like the device being unplugged wasn't removed from
> > pci_pme_list as pci_pme_active(dev, false) wasn't called.
> >
> > From a quick review, I wasn't able to find the right place in the
> > call-chain which I only see releases the child busses and PCIe port
> > drivers. Anyone?
> 
> It looks like drivers *add* devices to pci_pme_list when they use
> pci_enable_wake() or pci_wake_from_d3().  But many drivers never
> remove their devices, and I don't see any place where the core does it
> either.  My guess is we need to remove it in pci_stop_dev() (we
> already do pcie_aspm_exit_link_state() there) or somewhere similar.

Yes, we should call pci_pme_active(dev, false) somewhere in there I think.
It's fine to call that even if PME was not "active" before.

Thanks,
Rafael


-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops
  2013-02-11 19:27     ` Rafael J. Wysocki
@ 2013-02-11 19:49       ` Rafael J. Wysocki
  2013-02-12  2:18         ` Daniel J Blueman
  2013-02-13 14:32         ` Daniel J Blueman
  0 siblings, 2 replies; 12+ messages in thread
From: Rafael J. Wysocki @ 2013-02-11 19:49 UTC (permalink / raw)
  To: Bjorn Helgaas, Daniel J Blueman; +Cc: Linux Kernel, Linux PCI, Yijing Wang

On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote:
> On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote:
> > [+cc Rafael]
> > 
> > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote:
> > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote:
> > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a
> > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released:
> > >>
> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3)
> > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not
> > >> clear MAC_TX_MODE=ffffffff
> > >> tg3 0000:0a:00.0 eth0: No firmware running
> > >> tg3 0000:0a:00.0 eth0: Link is down
> > >> [sched_delayed] sched: RT throttling activated
> > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI
> > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp
> > >> pci_bus 0000:0a: busn_res: [bus 0a] is released
> > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released
> > >>
> > >> After some activity later (eg I can reproduce this by switching to a
> > >> text console and back), often we'll see an oops:
> > >>
> > >> Unable to handle kernel paging request at 0000000000001070
> > >> pci_pme_list_scan+0x3d/0xe0
> > >> Call Trace:
> > >> process_one_work+0x193
> > >> ? process_one_work+0x131
> > >> ? pci_pme_wakeup+0x60
> > >> worker_thread+0x15d
> > >>
> > >> (gdb) list *(pci_pme_list_scan+0x3d)
> > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556).
> > >> 1551                                    /*
> > >> 1552                                     * If bridge is in low power state, the
> > >> 1553                                     * configuration space of subordinate devices
> > >> 1554                                     * may be not accessible
> > >> 1555                                     */
> > >> 1556                                    if (bridge && bridge->current_state != PCI_D0)
> > >> 1557                                            continue;
> > >> 1558                                    pci_pme_wakeup(pme_dev->dev, NULL);
> > >> 1559                            } else {
> > >> 1560                                    list_del(&pme_dev->list);
> > >>
> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch
> > >> it with EFI pstore), it is almost certainly significant heap
> > >> corruption; this would explain why pme_dev became null (the load has
> > >> been ordered ahead).
> > >>
> > >> I'll see what I can find out with memory poisoning and list debugging.
> > >
> > > Enabling a bunch of related debugging, we see pme_dev is non-null and:
> > >
> > > BUG: Unable to handle NULL pointer dereference at
> > > pci_bus_read_config_word+0x6c
> > > PGD 26314c067 PUD 2633f9067 PMD 0
> > > Oops: 0000 [#1] SMP
> > > pci_check_pme_status+0x4f
> > > pci_pme_wakeup+0x21
> > > pci_pme_list_scan+0xd5
> > > process_one_work+0x1ca
> > > ? process_one_work+0x160
> > > ? pci_pme_wakeup+0x60
> > > worker_thread+0x14e
> > >
> > > Anyway, it looks like the device being unplugged wasn't removed from
> > > pci_pme_list as pci_pme_active(dev, false) wasn't called.
> > >
> > > From a quick review, I wasn't able to find the right place in the
> > > call-chain which I only see releases the child busses and PCIe port
> > > drivers. Anyone?
> > 
> > It looks like drivers *add* devices to pci_pme_list when they use
> > pci_enable_wake() or pci_wake_from_d3().  But many drivers never
> > remove their devices, and I don't see any place where the core does it
> > either.  My guess is we need to remove it in pci_stop_dev() (we
> > already do pcie_aspm_exit_link_state() there) or somewhere similar.
> 
> Yes, we should call pci_pme_active(dev, false) somewhere in there I think.
> It's fine to call that even if PME was not "active" before.

Daniel, I wonder if the patch below helps?

Rafael


Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
---
 drivers/pci/remove.c |    2 ++
 1 file changed, 2 insertions(+)

Index: test/drivers/pci/remove.c
===================================================================
--- test.orig/drivers/pci/remove.c
+++ test/drivers/pci/remove.c
@@ -19,6 +19,8 @@ static void pci_free_resources(struct pc
 
 static void pci_stop_dev(struct pci_dev *dev)
 {
+	pci_pme_active(dev, false);
+
 	if (dev->is_added) {
 		pci_proc_detach_device(dev);
 		pci_remove_sysfs_dev_files(dev);



-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops
  2013-02-11 19:49       ` Rafael J. Wysocki
@ 2013-02-12  2:18         ` Daniel J Blueman
  2013-02-12 20:53           ` Rafael J. Wysocki
  2013-02-13 14:32         ` Daniel J Blueman
  1 sibling, 1 reply; 12+ messages in thread
From: Daniel J Blueman @ 2013-02-12  2:18 UTC (permalink / raw)
  To: Rafael J. Wysocki; +Cc: Bjorn Helgaas, Linux Kernel, Linux PCI, Yijing Wang

On 12 February 2013 03:49, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote:
>> On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote:
>> > [+cc Rafael]
>> >
>> > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote:
>> > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote:
>> > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a
>> > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released:
>> > >>
>> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3)
>> > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not
>> > >> clear MAC_TX_MODE=ffffffff
>> > >> tg3 0000:0a:00.0 eth0: No firmware running
>> > >> tg3 0000:0a:00.0 eth0: Link is down
>> > >> [sched_delayed] sched: RT throttling activated
>> > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI
>> > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp
>> > >> pci_bus 0000:0a: busn_res: [bus 0a] is released
>> > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released
>> > >>
>> > >> After some activity later (eg I can reproduce this by switching to a
>> > >> text console and back), often we'll see an oops:
>> > >>
>> > >> Unable to handle kernel paging request at 0000000000001070
>> > >> pci_pme_list_scan+0x3d/0xe0
>> > >> Call Trace:
>> > >> process_one_work+0x193
>> > >> ? process_one_work+0x131
>> > >> ? pci_pme_wakeup+0x60
>> > >> worker_thread+0x15d
>> > >>
>> > >> (gdb) list *(pci_pme_list_scan+0x3d)
>> > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556).
>> > >> 1551                                    /*
>> > >> 1552                                     * If bridge is in low power state, the
>> > >> 1553                                     * configuration space of subordinate devices
>> > >> 1554                                     * may be not accessible
>> > >> 1555                                     */
>> > >> 1556                                    if (bridge && bridge->current_state != PCI_D0)
>> > >> 1557                                            continue;
>> > >> 1558                                    pci_pme_wakeup(pme_dev->dev, NULL);
>> > >> 1559                            } else {
>> > >> 1560                                    list_del(&pme_dev->list);
>> > >>
>> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch
>> > >> it with EFI pstore), it is almost certainly significant heap
>> > >> corruption; this would explain why pme_dev became null (the load has
>> > >> been ordered ahead).
>> > >>
>> > >> I'll see what I can find out with memory poisoning and list debugging.
>> > >
>> > > Enabling a bunch of related debugging, we see pme_dev is non-null and:
>> > >
>> > > BUG: Unable to handle NULL pointer dereference at
>> > > pci_bus_read_config_word+0x6c
>> > > PGD 26314c067 PUD 2633f9067 PMD 0
>> > > Oops: 0000 [#1] SMP
>> > > pci_check_pme_status+0x4f
>> > > pci_pme_wakeup+0x21
>> > > pci_pme_list_scan+0xd5
>> > > process_one_work+0x1ca
>> > > ? process_one_work+0x160
>> > > ? pci_pme_wakeup+0x60
>> > > worker_thread+0x14e
>> > >
>> > > Anyway, it looks like the device being unplugged wasn't removed from
>> > > pci_pme_list as pci_pme_active(dev, false) wasn't called.
>> > >
>> > > From a quick review, I wasn't able to find the right place in the
>> > > call-chain which I only see releases the child busses and PCIe port
>> > > drivers. Anyone?
>> >
>> > It looks like drivers *add* devices to pci_pme_list when they use
>> > pci_enable_wake() or pci_wake_from_d3().  But many drivers never
>> > remove their devices, and I don't see any place where the core does it
>> > either.  My guess is we need to remove it in pci_stop_dev() (we
>> > already do pcie_aspm_exit_link_state() there) or somewhere similar.
>>
>> Yes, we should call pci_pme_active(dev, false) somewhere in there I think.
>> It's fine to call that even if PME was not "active" before.
>
> Daniel, I wonder if the patch below helps?

I had tried with the pci_pme_active call inside the is_added condition
(it'll always hold here); it resolves the issue, though it introduces
a new lockdep warning [1].

All said, I don't see any other way except for a patch to abstract the
list entry removal to avoid the unnecessary reads and writes (as it is
called for four devices here), though I don't see how that would alter
the locking behaviour and why we didn't see this lockdep warning
before.

What do you think?

Dan

--- [1]

kworker/0:0/4 is trying to acquire lock:
 (name){++++.+}, at: [<ffffffff8105ac70>] flush_workqueue+0x0/0x4d0

but task is already holding lock:
 (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0

other info that might help us debug this:
 Possible unsafe locking scenario:

    CPU0
    ----
 lock(name);
 lock(name);

 *** DEADLOCK ***

 May be due to missing lock nesting notation

4 locks held by kworker/0:0/4:
 #0: (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0
 #1: ((&info->work)#2){+.+.+.}, at: [<ffffffff8105c7e0>]
process_one_work+0x160/0x4e0
 #2: (&__lockdep_no_validate__){......}, at: [<ffffffff813022a1>]
device_release_driver+0x21/0x40
 #3: (&__lockdep_no_validate__){......}, at: [<ffffffff813022a1>]
device_release_driver+0x21/0x40

stack backtrace:
Pid: 4, comm: kworker/0:0 Not tainted 3.8.0-rc7-ninja+ #21
Call Trace:
 [<ffffffff81090213>] validate_chain.isra.33+0xda3/0x1240
 [<ffffffff8108ea90>] ? print_shortest_lock_dependencies+0x1c0/0x1c0
 [<ffffffff810909d5>] ? mark_lock+0x215/0x5d0
 [<ffffffff8109284f>] ? debug_check_no_locks_freed+0x8f/0x160
 [<ffffffff8109113c>] __lock_acquire+0x3ac/0xb30
 [<ffffffff810927bd>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff8108dc0c>] ? lockdep_init_map+0x9c/0x4d0
 [<ffffffff81091d8a>] lock_acquire+0x5a/0x70
 [<ffffffff8105ac70>] ? flush_workqueue_prep_cwqs+0x200/0x200
 [<ffffffff8105ad58>] flush_workqueue+0xe8/0x4d0
 [<ffffffff8105ac70>] ? flush_workqueue_prep_cwqs+0x200/0x200
 [<ffffffff8105b1c8>] drain_workqueue+0x68/0x1f0
 [<ffffffff8105b363>] destroy_workqueue+0x13/0x160
 [<ffffffff8125ad0a>] pciehp_release_ctrl+0x3a/0x90
 [<ffffffff81257ca5>] pciehp_remove+0x25/0x30
 [<ffffffff81251f72>] pcie_port_remove_service+0x52/0x70
 [<ffffffff81302217>] __device_release_driver+0x77/0xe0
 [<ffffffff813022a9>] device_release_driver+0x29/0x40
 [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140
 [<ffffffff812ff847>] device_del+0x127/0x1c0
 [<ffffffff812520f0>] ? resume_iter+0x40/0x40
 [<ffffffff812ff8f1>] device_unregister+0x11/0x20
 [<ffffffff81252125>] remove_iter+0x35/0x40
 [<ffffffff812fe716>] device_for_each_child+0x36/0x70
 [<ffffffff812526c1>] pcie_port_device_remove+0x21/0x40
 [<ffffffff81252908>] pcie_portdrv_remove+0x28/0x50
 [<ffffffff81246cb1>] pci_device_remove+0x41/0xc0
 [<ffffffff81302217>] __device_release_driver+0x77/0xe0
 [<ffffffff813022a9>] device_release_driver+0x29/0x40
 [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140
 [<ffffffff812ff847>] device_del+0x127/0x1c0
 [<ffffffff812ff8f1>] device_unregister+0x11/0x20
 [<ffffffff81241b74>] pci_stop_bus_device+0xb4/0xc0
 [<ffffffff81241af5>] pci_stop_bus_device+0x35/0xc0
 [<ffffffff81241cd1>] pci_stop_and_remove_bus_device+0x11/0x20
 [<ffffffff81259021>] pciehp_unconfigure_device+0x91/0x190
 [<ffffffff81258921>] pciehp_disable_slot+0x71/0x220
 [<ffffffff81258bb6>] pciehp_power_thread+0xe6/0x110
 [<ffffffff8105c84a>] process_one_work+0x1ca/0x4e0
 [<ffffffff8105c7e0>] ? process_one_work+0x160/0x4e0
 [<ffffffff81258ad0>] ? pciehp_disable_slot+0x220/0x220
 [<ffffffff8105cefe>] worker_thread+0x14e/0x3f0
 [<ffffffff810927bd>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff8105cdb0>] ? rescuer_thread+0x210/0x210
 [<ffffffff81063086>] kthread+0xd6/0xe0
 [<ffffffff8154cf2b>] ? _raw_spin_unlock_irq+0x2b/0x50
 [<ffffffff81062fb0>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8154dc2c>] ret_from_fork+0x7c/0xb0
 [<ffffffff81062fb0>] ? __init_kthread_worker+0x70/0x70
-- 
Daniel J Blueman

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops
  2013-02-12 20:53           ` Rafael J. Wysocki
@ 2013-02-12 20:50             ` Tejun Heo
  2013-02-12 21:00               ` Rafael J. Wysocki
  0 siblings, 1 reply; 12+ messages in thread
From: Tejun Heo @ 2013-02-12 20:50 UTC (permalink / raw)
  To: Rafael J. Wysocki
  Cc: Daniel J Blueman, Bjorn Helgaas, Linux Kernel, Linux PCI, Yijing Wang

Hey, Rafael.

On Tue, Feb 12, 2013 at 09:53:08PM +0100, Rafael J. Wysocki wrote:
> This looks fishy, but I wonder if Tejun has any ideas.
> 
> Tejun, can you please have a look at the call trace below?  It looks like
> the workqueues code is involved heavily.
> > 
> > kworker/0:0/4 is trying to acquire lock:
> >  (name){++++.+}, at: [<ffffffff8105ac70>] flush_workqueue+0x0/0x4d0
> > 
> > but task is already holding lock:
> >  (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0

It's basically saying that a work item is trying to flush the
workqueue it's currently executing on, at least in lockdep's eyes.

> > stack backtrace:
> > Pid: 4, comm: kworker/0:0 Not tainted 3.8.0-rc7-ninja+ #21
> > Call Trace:
> >  [<ffffffff81090213>] validate_chain.isra.33+0xda3/0x1240
> >  [<ffffffff8109113c>] __lock_acquire+0x3ac/0xb30
> >  [<ffffffff81091d8a>] lock_acquire+0x5a/0x70
> >  [<ffffffff8105ad58>] flush_workqueue+0xe8/0x4d0
> >  [<ffffffff8105b1c8>] drain_workqueue+0x68/0x1f0
> >  [<ffffffff8105b363>] destroy_workqueue+0x13/0x160

And the flush is from workqueue destruction

> >  [<ffffffff8125ad0a>] pciehp_release_ctrl+0x3a/0x90
> >  [<ffffffff81257ca5>] pciehp_remove+0x25/0x30
> >  [<ffffffff81251f72>] pcie_port_remove_service+0x52/0x70
> >  [<ffffffff81302217>] __device_release_driver+0x77/0xe0
> >  [<ffffffff813022a9>] device_release_driver+0x29/0x40
> >  [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140
> >  [<ffffffff812ff847>] device_del+0x127/0x1c0
> >  [<ffffffff812ff8f1>] device_unregister+0x11/0x20
> >  [<ffffffff81252125>] remove_iter+0x35/0x40
> >  [<ffffffff812fe716>] device_for_each_child+0x36/0x70
> >  [<ffffffff812526c1>] pcie_port_device_remove+0x21/0x40
> >  [<ffffffff81252908>] pcie_portdrv_remove+0x28/0x50
> >  [<ffffffff81246cb1>] pci_device_remove+0x41/0xc0
> >  [<ffffffff81302217>] __device_release_driver+0x77/0xe0
> >  [<ffffffff813022a9>] device_release_driver+0x29/0x40
> >  [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140
> >  [<ffffffff812ff847>] device_del+0x127/0x1c0
> >  [<ffffffff812ff8f1>] device_unregister+0x11/0x20
> >  [<ffffffff81241b74>] pci_stop_bus_device+0xb4/0xc0
> >  [<ffffffff81241af5>] pci_stop_bus_device+0x35/0xc0
> >  [<ffffffff81241cd1>] pci_stop_and_remove_bus_device+0x11/0x20
> >  [<ffffffff81259021>] pciehp_unconfigure_device+0x91/0x190
> >  [<ffffffff81258921>] pciehp_disable_slot+0x71/0x220
> >  [<ffffffff81258bb6>] pciehp_power_thread+0xe6/0x110
> >  [<ffffffff8105c84a>] process_one_work+0x1ca/0x4e0

running from a workqueue which probably is at least transitively
related to the workqueue being destroyed.  Does this lead to an actual
deadlock?

Thanks.

-- 
tejun

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops
  2013-02-12  2:18         ` Daniel J Blueman
@ 2013-02-12 20:53           ` Rafael J. Wysocki
  2013-02-12 20:50             ` Tejun Heo
  0 siblings, 1 reply; 12+ messages in thread
From: Rafael J. Wysocki @ 2013-02-12 20:53 UTC (permalink / raw)
  To: Daniel J Blueman, Tejun Heo
  Cc: Bjorn Helgaas, Linux Kernel, Linux PCI, Yijing Wang

On Tuesday, February 12, 2013 10:18:57 AM Daniel J Blueman wrote:
> On 12 February 2013 03:49, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> > On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote:
> >> On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote:
> >> > [+cc Rafael]
> >> >
> >> > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote:
> >> > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote:
> >> > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a
> >> > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released:
> >> > >>
> >> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3)
> >> > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not
> >> > >> clear MAC_TX_MODE=ffffffff
> >> > >> tg3 0000:0a:00.0 eth0: No firmware running
> >> > >> tg3 0000:0a:00.0 eth0: Link is down
> >> > >> [sched_delayed] sched: RT throttling activated
> >> > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI
> >> > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp
> >> > >> pci_bus 0000:0a: busn_res: [bus 0a] is released
> >> > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released
> >> > >>
> >> > >> After some activity later (eg I can reproduce this by switching to a
> >> > >> text console and back), often we'll see an oops:
> >> > >>
> >> > >> Unable to handle kernel paging request at 0000000000001070
> >> > >> pci_pme_list_scan+0x3d/0xe0
> >> > >> Call Trace:
> >> > >> process_one_work+0x193
> >> > >> ? process_one_work+0x131
> >> > >> ? pci_pme_wakeup+0x60
> >> > >> worker_thread+0x15d
> >> > >>
> >> > >> (gdb) list *(pci_pme_list_scan+0x3d)
> >> > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556).
> >> > >> 1551                                    /*
> >> > >> 1552                                     * If bridge is in low power state, the
> >> > >> 1553                                     * configuration space of subordinate devices
> >> > >> 1554                                     * may be not accessible
> >> > >> 1555                                     */
> >> > >> 1556                                    if (bridge && bridge->current_state != PCI_D0)
> >> > >> 1557                                            continue;
> >> > >> 1558                                    pci_pme_wakeup(pme_dev->dev, NULL);
> >> > >> 1559                            } else {
> >> > >> 1560                                    list_del(&pme_dev->list);
> >> > >>
> >> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch
> >> > >> it with EFI pstore), it is almost certainly significant heap
> >> > >> corruption; this would explain why pme_dev became null (the load has
> >> > >> been ordered ahead).
> >> > >>
> >> > >> I'll see what I can find out with memory poisoning and list debugging.
> >> > >
> >> > > Enabling a bunch of related debugging, we see pme_dev is non-null and:
> >> > >
> >> > > BUG: Unable to handle NULL pointer dereference at
> >> > > pci_bus_read_config_word+0x6c
> >> > > PGD 26314c067 PUD 2633f9067 PMD 0
> >> > > Oops: 0000 [#1] SMP
> >> > > pci_check_pme_status+0x4f
> >> > > pci_pme_wakeup+0x21
> >> > > pci_pme_list_scan+0xd5
> >> > > process_one_work+0x1ca
> >> > > ? process_one_work+0x160
> >> > > ? pci_pme_wakeup+0x60
> >> > > worker_thread+0x14e
> >> > >
> >> > > Anyway, it looks like the device being unplugged wasn't removed from
> >> > > pci_pme_list as pci_pme_active(dev, false) wasn't called.
> >> > >
> >> > > From a quick review, I wasn't able to find the right place in the
> >> > > call-chain which I only see releases the child busses and PCIe port
> >> > > drivers. Anyone?
> >> >
> >> > It looks like drivers *add* devices to pci_pme_list when they use
> >> > pci_enable_wake() or pci_wake_from_d3().  But many drivers never
> >> > remove their devices, and I don't see any place where the core does it
> >> > either.  My guess is we need to remove it in pci_stop_dev() (we
> >> > already do pcie_aspm_exit_link_state() there) or somewhere similar.
> >>
> >> Yes, we should call pci_pme_active(dev, false) somewhere in there I think.
> >> It's fine to call that even if PME was not "active" before.
> >
> > Daniel, I wonder if the patch below helps?
> 
> I had tried with the pci_pme_active call inside the is_added condition
> (it'll always hold here); it resolves the issue, though it introduces
> a new lockdep warning [1].
> 
> All said, I don't see any other way except for a patch to abstract the
> list entry removal to avoid the unnecessary reads and writes (as it is
> called for four devices here), though I don't see how that would alter
> the locking behaviour and why we didn't see this lockdep warning
> before.
> 
> What do you think?

This looks fishy, but I wonder if Tejun has any ideas.

Tejun, can you please have a look at the call trace below?  It looks like
the workqueues code is involved heavily.

Thanks,
Rafael


> --- [1]
> 
> kworker/0:0/4 is trying to acquire lock:
>  (name){++++.+}, at: [<ffffffff8105ac70>] flush_workqueue+0x0/0x4d0
> 
> but task is already holding lock:
>  (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0
> 
> other info that might help us debug this:
>  Possible unsafe locking scenario:
> 
>     CPU0
>     ----
>  lock(name);
>  lock(name);
> 
>  *** DEADLOCK ***
> 
>  May be due to missing lock nesting notation
> 
> 4 locks held by kworker/0:0/4:
>  #0: (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0
>  #1: ((&info->work)#2){+.+.+.}, at: [<ffffffff8105c7e0>]
> process_one_work+0x160/0x4e0
>  #2: (&__lockdep_no_validate__){......}, at: [<ffffffff813022a1>]
> device_release_driver+0x21/0x40
>  #3: (&__lockdep_no_validate__){......}, at: [<ffffffff813022a1>]
> device_release_driver+0x21/0x40
> 
> stack backtrace:
> Pid: 4, comm: kworker/0:0 Not tainted 3.8.0-rc7-ninja+ #21
> Call Trace:
>  [<ffffffff81090213>] validate_chain.isra.33+0xda3/0x1240
>  [<ffffffff8108ea90>] ? print_shortest_lock_dependencies+0x1c0/0x1c0
>  [<ffffffff810909d5>] ? mark_lock+0x215/0x5d0
>  [<ffffffff8109284f>] ? debug_check_no_locks_freed+0x8f/0x160
>  [<ffffffff8109113c>] __lock_acquire+0x3ac/0xb30
>  [<ffffffff810927bd>] ? trace_hardirqs_on+0xd/0x10
>  [<ffffffff8108dc0c>] ? lockdep_init_map+0x9c/0x4d0
>  [<ffffffff81091d8a>] lock_acquire+0x5a/0x70
>  [<ffffffff8105ac70>] ? flush_workqueue_prep_cwqs+0x200/0x200
>  [<ffffffff8105ad58>] flush_workqueue+0xe8/0x4d0
>  [<ffffffff8105ac70>] ? flush_workqueue_prep_cwqs+0x200/0x200
>  [<ffffffff8105b1c8>] drain_workqueue+0x68/0x1f0
>  [<ffffffff8105b363>] destroy_workqueue+0x13/0x160
>  [<ffffffff8125ad0a>] pciehp_release_ctrl+0x3a/0x90
>  [<ffffffff81257ca5>] pciehp_remove+0x25/0x30
>  [<ffffffff81251f72>] pcie_port_remove_service+0x52/0x70
>  [<ffffffff81302217>] __device_release_driver+0x77/0xe0
>  [<ffffffff813022a9>] device_release_driver+0x29/0x40
>  [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140
>  [<ffffffff812ff847>] device_del+0x127/0x1c0
>  [<ffffffff812520f0>] ? resume_iter+0x40/0x40
>  [<ffffffff812ff8f1>] device_unregister+0x11/0x20
>  [<ffffffff81252125>] remove_iter+0x35/0x40
>  [<ffffffff812fe716>] device_for_each_child+0x36/0x70
>  [<ffffffff812526c1>] pcie_port_device_remove+0x21/0x40
>  [<ffffffff81252908>] pcie_portdrv_remove+0x28/0x50
>  [<ffffffff81246cb1>] pci_device_remove+0x41/0xc0
>  [<ffffffff81302217>] __device_release_driver+0x77/0xe0
>  [<ffffffff813022a9>] device_release_driver+0x29/0x40
>  [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140
>  [<ffffffff812ff847>] device_del+0x127/0x1c0
>  [<ffffffff812ff8f1>] device_unregister+0x11/0x20
>  [<ffffffff81241b74>] pci_stop_bus_device+0xb4/0xc0
>  [<ffffffff81241af5>] pci_stop_bus_device+0x35/0xc0
>  [<ffffffff81241cd1>] pci_stop_and_remove_bus_device+0x11/0x20
>  [<ffffffff81259021>] pciehp_unconfigure_device+0x91/0x190
>  [<ffffffff81258921>] pciehp_disable_slot+0x71/0x220
>  [<ffffffff81258bb6>] pciehp_power_thread+0xe6/0x110
>  [<ffffffff8105c84a>] process_one_work+0x1ca/0x4e0
>  [<ffffffff8105c7e0>] ? process_one_work+0x160/0x4e0
>  [<ffffffff81258ad0>] ? pciehp_disable_slot+0x220/0x220
>  [<ffffffff8105cefe>] worker_thread+0x14e/0x3f0
>  [<ffffffff810927bd>] ? trace_hardirqs_on+0xd/0x10
>  [<ffffffff8105cdb0>] ? rescuer_thread+0x210/0x210
>  [<ffffffff81063086>] kthread+0xd6/0xe0
>  [<ffffffff8154cf2b>] ? _raw_spin_unlock_irq+0x2b/0x50
>  [<ffffffff81062fb0>] ? __init_kthread_worker+0x70/0x70
>  [<ffffffff8154dc2c>] ret_from_fork+0x7c/0xb0
>  [<ffffffff81062fb0>] ? __init_kthread_worker+0x70/0x70
> 
-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops
  2013-02-12 20:50             ` Tejun Heo
@ 2013-02-12 21:00               ` Rafael J. Wysocki
  0 siblings, 0 replies; 12+ messages in thread
From: Rafael J. Wysocki @ 2013-02-12 21:00 UTC (permalink / raw)
  To: Tejun Heo, Daniel J Blueman
  Cc: Bjorn Helgaas, Linux Kernel, Linux PCI, Yijing Wang

On Tuesday, February 12, 2013 12:50:38 PM Tejun Heo wrote:
> Hey, Rafael.
> 
> On Tue, Feb 12, 2013 at 09:53:08PM +0100, Rafael J. Wysocki wrote:
> > This looks fishy, but I wonder if Tejun has any ideas.
> > 
> > Tejun, can you please have a look at the call trace below?  It looks like
> > the workqueues code is involved heavily.
> > > 
> > > kworker/0:0/4 is trying to acquire lock:
> > >  (name){++++.+}, at: [<ffffffff8105ac70>] flush_workqueue+0x0/0x4d0
> > > 
> > > but task is already holding lock:
> > >  (name){++++.+}, at: [<ffffffff8105c7e0>] process_one_work+0x160/0x4e0
> 
> It's basically saying that a work item is trying to flush the
> workqueue it's currently executing on, at least in lockdep's eyes.
> 
> > > stack backtrace:
> > > Pid: 4, comm: kworker/0:0 Not tainted 3.8.0-rc7-ninja+ #21
> > > Call Trace:
> > >  [<ffffffff81090213>] validate_chain.isra.33+0xda3/0x1240
> > >  [<ffffffff8109113c>] __lock_acquire+0x3ac/0xb30
> > >  [<ffffffff81091d8a>] lock_acquire+0x5a/0x70
> > >  [<ffffffff8105ad58>] flush_workqueue+0xe8/0x4d0
> > >  [<ffffffff8105b1c8>] drain_workqueue+0x68/0x1f0
> > >  [<ffffffff8105b363>] destroy_workqueue+0x13/0x160
> 
> And the flush is from workqueue destruction
> 
> > >  [<ffffffff8125ad0a>] pciehp_release_ctrl+0x3a/0x90
> > >  [<ffffffff81257ca5>] pciehp_remove+0x25/0x30
> > >  [<ffffffff81251f72>] pcie_port_remove_service+0x52/0x70
> > >  [<ffffffff81302217>] __device_release_driver+0x77/0xe0
> > >  [<ffffffff813022a9>] device_release_driver+0x29/0x40
> > >  [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140
> > >  [<ffffffff812ff847>] device_del+0x127/0x1c0
> > >  [<ffffffff812ff8f1>] device_unregister+0x11/0x20
> > >  [<ffffffff81252125>] remove_iter+0x35/0x40
> > >  [<ffffffff812fe716>] device_for_each_child+0x36/0x70
> > >  [<ffffffff812526c1>] pcie_port_device_remove+0x21/0x40
> > >  [<ffffffff81252908>] pcie_portdrv_remove+0x28/0x50
> > >  [<ffffffff81246cb1>] pci_device_remove+0x41/0xc0
> > >  [<ffffffff81302217>] __device_release_driver+0x77/0xe0
> > >  [<ffffffff813022a9>] device_release_driver+0x29/0x40
> > >  [<ffffffff81301cb1>] bus_remove_device+0xf1/0x140
> > >  [<ffffffff812ff847>] device_del+0x127/0x1c0
> > >  [<ffffffff812ff8f1>] device_unregister+0x11/0x20
> > >  [<ffffffff81241b74>] pci_stop_bus_device+0xb4/0xc0
> > >  [<ffffffff81241af5>] pci_stop_bus_device+0x35/0xc0
> > >  [<ffffffff81241cd1>] pci_stop_and_remove_bus_device+0x11/0x20
> > >  [<ffffffff81259021>] pciehp_unconfigure_device+0x91/0x190
> > >  [<ffffffff81258921>] pciehp_disable_slot+0x71/0x220
> > >  [<ffffffff81258bb6>] pciehp_power_thread+0xe6/0x110
> > >  [<ffffffff8105c84a>] process_one_work+0x1ca/0x4e0
> 
> running from a workqueue which probably is at least transitively
> related to the workqueue being destroyed.  Does this lead to an actual
> deadlock?

Might be.  I need to have a deeper look at things in the acpiphp land.

Daniel, I'm quite sure it isn't related to the addition of the pci_pme_active()
call.

Thanks,
Rafael


-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops
  2013-02-11 19:49       ` Rafael J. Wysocki
  2013-02-12  2:18         ` Daniel J Blueman
@ 2013-02-13 14:32         ` Daniel J Blueman
  2013-02-13 19:29           ` Bjorn Helgaas
  1 sibling, 1 reply; 12+ messages in thread
From: Daniel J Blueman @ 2013-02-13 14:32 UTC (permalink / raw)
  To: Rafael J. Wysocki; +Cc: Bjorn Helgaas, Linux Kernel, Linux PCI, Yijing Wang

On 12 February 2013 03:49, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote:
>> On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote:
>> > [+cc Rafael]
>> >
>> > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote:
>> > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote:
>> > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a
>> > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released:
>> > >>
>> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3)
>> > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not
>> > >> clear MAC_TX_MODE=ffffffff
>> > >> tg3 0000:0a:00.0 eth0: No firmware running
>> > >> tg3 0000:0a:00.0 eth0: Link is down
>> > >> [sched_delayed] sched: RT throttling activated
>> > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI
>> > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp
>> > >> pci_bus 0000:0a: busn_res: [bus 0a] is released
>> > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released
>> > >>
>> > >> After some activity later (eg I can reproduce this by switching to a
>> > >> text console and back), often we'll see an oops:
>> > >>
>> > >> Unable to handle kernel paging request at 0000000000001070
>> > >> pci_pme_list_scan+0x3d/0xe0
>> > >> Call Trace:
>> > >> process_one_work+0x193
>> > >> ? process_one_work+0x131
>> > >> ? pci_pme_wakeup+0x60
>> > >> worker_thread+0x15d
>> > >>
>> > >> (gdb) list *(pci_pme_list_scan+0x3d)
>> > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556).
>> > >> 1551                                    /*
>> > >> 1552                                     * If bridge is in low power state, the
>> > >> 1553                                     * configuration space of subordinate devices
>> > >> 1554                                     * may be not accessible
>> > >> 1555                                     */
>> > >> 1556                                    if (bridge && bridge->current_state != PCI_D0)
>> > >> 1557                                            continue;
>> > >> 1558                                    pci_pme_wakeup(pme_dev->dev, NULL);
>> > >> 1559                            } else {
>> > >> 1560                                    list_del(&pme_dev->list);
>> > >>
>> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch
>> > >> it with EFI pstore), it is almost certainly significant heap
>> > >> corruption; this would explain why pme_dev became null (the load has
>> > >> been ordered ahead).
>> > >>
>> > >> I'll see what I can find out with memory poisoning and list debugging.
>> > >
>> > > Enabling a bunch of related debugging, we see pme_dev is non-null and:
>> > >
>> > > BUG: Unable to handle NULL pointer dereference at
>> > > pci_bus_read_config_word+0x6c
>> > > PGD 26314c067 PUD 2633f9067 PMD 0
>> > > Oops: 0000 [#1] SMP
>> > > pci_check_pme_status+0x4f
>> > > pci_pme_wakeup+0x21
>> > > pci_pme_list_scan+0xd5
>> > > process_one_work+0x1ca
>> > > ? process_one_work+0x160
>> > > ? pci_pme_wakeup+0x60
>> > > worker_thread+0x14e
>> > >
>> > > Anyway, it looks like the device being unplugged wasn't removed from
>> > > pci_pme_list as pci_pme_active(dev, false) wasn't called.
>> > >
>> > > From a quick review, I wasn't able to find the right place in the
>> > > call-chain which I only see releases the child busses and PCIe port
>> > > drivers. Anyone?
>> >
>> > It looks like drivers *add* devices to pci_pme_list when they use
>> > pci_enable_wake() or pci_wake_from_d3().  But many drivers never
>> > remove their devices, and I don't see any place where the core does it
>> > either.  My guess is we need to remove it in pci_stop_dev() (we
>> > already do pcie_aspm_exit_link_state() there) or somewhere similar.
>>
>> Yes, we should call pci_pme_active(dev, false) somewhere in there I think.
>> It's fine to call that even if PME was not "active" before.
>
> Daniel, I wonder if the patch below helps?
>
> Rafael
>
>
> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> ---
>  drivers/pci/remove.c |    2 ++
>  1 file changed, 2 insertions(+)
>
> Index: test/drivers/pci/remove.c
> ===================================================================
> --- test.orig/drivers/pci/remove.c
> +++ test/drivers/pci/remove.c
> @@ -19,6 +19,8 @@ static void pci_free_resources(struct pc
>
>  static void pci_stop_dev(struct pci_dev *dev)
>  {
> +       pci_pme_active(dev, false);
> +
>         if (dev->is_added) {
>                 pci_proc_detach_device(dev);
>                 pci_remove_sysfs_dev_files(dev);

Indeed, the oops would prevent hitting the lockdep warning, which is a
secondary matter.

As it stands, this fix prevents the fatal oops, so IMHO is urgent
material for 3.8-final. I can't prove that is_added will always be
true, so let's stick with your patch.

Tested-by: Daniel J Blueman <daniel@quora.org>
-- 
Daniel J Blueman

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops
  2013-02-13 14:32         ` Daniel J Blueman
@ 2013-02-13 19:29           ` Bjorn Helgaas
  2013-02-13 20:55             ` Rafael J. Wysocki
  0 siblings, 1 reply; 12+ messages in thread
From: Bjorn Helgaas @ 2013-02-13 19:29 UTC (permalink / raw)
  To: Daniel J Blueman; +Cc: Rafael J. Wysocki, Linux Kernel, Linux PCI, Yijing Wang

On Wed, Feb 13, 2013 at 7:32 AM, Daniel J Blueman <daniel@quora.org> wrote:
> On 12 February 2013 03:49, Rafael J. Wysocki <rjw@sisk.pl> wrote:
>> On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote:
>>> On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote:
>>> > [+cc Rafael]
>>> >
>>> > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote:
>>> > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote:
>>> > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a
>>> > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released:
>>> > >>
>>> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3)
>>> > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not
>>> > >> clear MAC_TX_MODE=ffffffff
>>> > >> tg3 0000:0a:00.0 eth0: No firmware running
>>> > >> tg3 0000:0a:00.0 eth0: Link is down
>>> > >> [sched_delayed] sched: RT throttling activated
>>> > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI
>>> > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp
>>> > >> pci_bus 0000:0a: busn_res: [bus 0a] is released
>>> > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released
>>> > >>
>>> > >> After some activity later (eg I can reproduce this by switching to a
>>> > >> text console and back), often we'll see an oops:
>>> > >>
>>> > >> Unable to handle kernel paging request at 0000000000001070
>>> > >> pci_pme_list_scan+0x3d/0xe0
>>> > >> Call Trace:
>>> > >> process_one_work+0x193
>>> > >> ? process_one_work+0x131
>>> > >> ? pci_pme_wakeup+0x60
>>> > >> worker_thread+0x15d
>>> > >>
>>> > >> (gdb) list *(pci_pme_list_scan+0x3d)
>>> > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556).
>>> > >> 1551                                    /*
>>> > >> 1552                                     * If bridge is in low power state, the
>>> > >> 1553                                     * configuration space of subordinate devices
>>> > >> 1554                                     * may be not accessible
>>> > >> 1555                                     */
>>> > >> 1556                                    if (bridge && bridge->current_state != PCI_D0)
>>> > >> 1557                                            continue;
>>> > >> 1558                                    pci_pme_wakeup(pme_dev->dev, NULL);
>>> > >> 1559                            } else {
>>> > >> 1560                                    list_del(&pme_dev->list);
>>> > >>
>>> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch
>>> > >> it with EFI pstore), it is almost certainly significant heap
>>> > >> corruption; this would explain why pme_dev became null (the load has
>>> > >> been ordered ahead).
>>> > >>
>>> > >> I'll see what I can find out with memory poisoning and list debugging.
>>> > >
>>> > > Enabling a bunch of related debugging, we see pme_dev is non-null and:
>>> > >
>>> > > BUG: Unable to handle NULL pointer dereference at
>>> > > pci_bus_read_config_word+0x6c
>>> > > PGD 26314c067 PUD 2633f9067 PMD 0
>>> > > Oops: 0000 [#1] SMP
>>> > > pci_check_pme_status+0x4f
>>> > > pci_pme_wakeup+0x21
>>> > > pci_pme_list_scan+0xd5
>>> > > process_one_work+0x1ca
>>> > > ? process_one_work+0x160
>>> > > ? pci_pme_wakeup+0x60
>>> > > worker_thread+0x14e
>>> > >
>>> > > Anyway, it looks like the device being unplugged wasn't removed from
>>> > > pci_pme_list as pci_pme_active(dev, false) wasn't called.
>>> > >
>>> > > From a quick review, I wasn't able to find the right place in the
>>> > > call-chain which I only see releases the child busses and PCIe port
>>> > > drivers. Anyone?
>>> >
>>> > It looks like drivers *add* devices to pci_pme_list when they use
>>> > pci_enable_wake() or pci_wake_from_d3().  But many drivers never
>>> > remove their devices, and I don't see any place where the core does it
>>> > either.  My guess is we need to remove it in pci_stop_dev() (we
>>> > already do pcie_aspm_exit_link_state() there) or somewhere similar.
>>>
>>> Yes, we should call pci_pme_active(dev, false) somewhere in there I think.
>>> It's fine to call that even if PME was not "active" before.
>>
>> Daniel, I wonder if the patch below helps?
>>
>> Rafael
>>
>>
>> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
>> ---
>>  drivers/pci/remove.c |    2 ++
>>  1 file changed, 2 insertions(+)
>>
>> Index: test/drivers/pci/remove.c
>> ===================================================================
>> --- test.orig/drivers/pci/remove.c
>> +++ test/drivers/pci/remove.c
>> @@ -19,6 +19,8 @@ static void pci_free_resources(struct pc
>>
>>  static void pci_stop_dev(struct pci_dev *dev)
>>  {
>> +       pci_pme_active(dev, false);
>> +
>>         if (dev->is_added) {
>>                 pci_proc_detach_device(dev);
>>                 pci_remove_sysfs_dev_files(dev);
>
> Indeed, the oops would prevent hitting the lockdep warning, which is a
> secondary matter.
>
> As it stands, this fix prevents the fatal oops, so IMHO is urgent
> material for 3.8-final. I can't prove that is_added will always be
> true, so let's stick with your patch.
>
> Tested-by: Daniel J Blueman <daniel@quora.org>

I wrote a changelog and applied this to my for-linus branch.  Please
take a look and confirm that it makes sense before I ask Linus to pull
it.

http://git.kernel.org/?p=linux/kernel/git/helgaas/pci.git;a=commitdiff;h=249bfb83cf8ba658955f0245ac3981d941f746ee

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: [3.8-rc7] PCI hotplug wakeup oops
  2013-02-13 19:29           ` Bjorn Helgaas
@ 2013-02-13 20:55             ` Rafael J. Wysocki
  0 siblings, 0 replies; 12+ messages in thread
From: Rafael J. Wysocki @ 2013-02-13 20:55 UTC (permalink / raw)
  To: Bjorn Helgaas; +Cc: Daniel J Blueman, Linux Kernel, Linux PCI, Yijing Wang

On Wednesday, February 13, 2013 12:29:59 PM Bjorn Helgaas wrote:
> On Wed, Feb 13, 2013 at 7:32 AM, Daniel J Blueman <daniel@quora.org> wrote:
> > On 12 February 2013 03:49, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> >> On Monday, February 11, 2013 08:27:49 PM Rafael J. Wysocki wrote:
> >>> On Monday, February 11, 2013 12:01:37 PM Bjorn Helgaas wrote:
> >>> > [+cc Rafael]
> >>> >
> >>> > On Mon, Feb 11, 2013 at 10:08 AM, Daniel J Blueman <daniel@quora.org> wrote:
> >>> > > On 11 February 2013 21:03, Daniel J Blueman <daniel@quora.org> wrote:
> >>> > >> With 3.8-rc7, when unplugging the Thunderbolt ethernet adapter (bus 0a
> >>> > >> [1]) on a Macbook Pro 10,1, we see the PCIe port correctly released:
> >>> > >>
> >>> > >> pciehp 0000:06:03.0:pcie24: Card not present on Slot(3)
> >>> > >> tg3 0000:0a:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not
> >>> > >> clear MAC_TX_MODE=ffffffff
> >>> > >> tg3 0000:0a:00.0 eth0: No firmware running
> >>> > >> tg3 0000:0a:00.0 eth0: Link is down
> >>> > >> [sched_delayed] sched: RT throttling activated
> >>> > >> pcieport 0000:00:01.1: System wakeup enabled by ACPI
> >>> > >> pciehp 0000:09:00.0:pcie24: unloading service driver pciehp
> >>> > >> pci_bus 0000:0a: busn_res: [bus 0a] is released
> >>> > >> pci_bus 0000:09: busn_res: [bus 09-0a] is released
> >>> > >>
> >>> > >> After some activity later (eg I can reproduce this by switching to a
> >>> > >> text console and back), often we'll see an oops:
> >>> > >>
> >>> > >> Unable to handle kernel paging request at 0000000000001070
> >>> > >> pci_pme_list_scan+0x3d/0xe0
> >>> > >> Call Trace:
> >>> > >> process_one_work+0x193
> >>> > >> ? process_one_work+0x131
> >>> > >> ? pci_pme_wakeup+0x60
> >>> > >> worker_thread+0x15d
> >>> > >>
> >>> > >> (gdb) list *(pci_pme_list_scan+0x3d)
> >>> > >> 0xffffffff8123f6dd is in pci_pme_list_scan (drivers/pci/pci.c:1556).
> >>> > >> 1551                                    /*
> >>> > >> 1552                                     * If bridge is in low power state, the
> >>> > >> 1553                                     * configuration space of subordinate devices
> >>> > >> 1554                                     * may be not accessible
> >>> > >> 1555                                     */
> >>> > >> 1556                                    if (bridge && bridge->current_state != PCI_D0)
> >>> > >> 1557                                            continue;
> >>> > >> 1558                                    pci_pme_wakeup(pme_dev->dev, NULL);
> >>> > >> 1559                            } else {
> >>> > >> 1560                                    list_del(&pme_dev->list);
> >>> > >>
> >>> > >> Since a panic in vsnprintf happens after the oops (hence I can't catch
> >>> > >> it with EFI pstore), it is almost certainly significant heap
> >>> > >> corruption; this would explain why pme_dev became null (the load has
> >>> > >> been ordered ahead).
> >>> > >>
> >>> > >> I'll see what I can find out with memory poisoning and list debugging.
> >>> > >
> >>> > > Enabling a bunch of related debugging, we see pme_dev is non-null and:
> >>> > >
> >>> > > BUG: Unable to handle NULL pointer dereference at
> >>> > > pci_bus_read_config_word+0x6c
> >>> > > PGD 26314c067 PUD 2633f9067 PMD 0
> >>> > > Oops: 0000 [#1] SMP
> >>> > > pci_check_pme_status+0x4f
> >>> > > pci_pme_wakeup+0x21
> >>> > > pci_pme_list_scan+0xd5
> >>> > > process_one_work+0x1ca
> >>> > > ? process_one_work+0x160
> >>> > > ? pci_pme_wakeup+0x60
> >>> > > worker_thread+0x14e
> >>> > >
> >>> > > Anyway, it looks like the device being unplugged wasn't removed from
> >>> > > pci_pme_list as pci_pme_active(dev, false) wasn't called.
> >>> > >
> >>> > > From a quick review, I wasn't able to find the right place in the
> >>> > > call-chain which I only see releases the child busses and PCIe port
> >>> > > drivers. Anyone?
> >>> >
> >>> > It looks like drivers *add* devices to pci_pme_list when they use
> >>> > pci_enable_wake() or pci_wake_from_d3().  But many drivers never
> >>> > remove their devices, and I don't see any place where the core does it
> >>> > either.  My guess is we need to remove it in pci_stop_dev() (we
> >>> > already do pcie_aspm_exit_link_state() there) or somewhere similar.
> >>>
> >>> Yes, we should call pci_pme_active(dev, false) somewhere in there I think.
> >>> It's fine to call that even if PME was not "active" before.
> >>
> >> Daniel, I wonder if the patch below helps?
> >>
> >> Rafael
> >>
> >>
> >> Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> >> ---
> >>  drivers/pci/remove.c |    2 ++
> >>  1 file changed, 2 insertions(+)
> >>
> >> Index: test/drivers/pci/remove.c
> >> ===================================================================
> >> --- test.orig/drivers/pci/remove.c
> >> +++ test/drivers/pci/remove.c
> >> @@ -19,6 +19,8 @@ static void pci_free_resources(struct pc
> >>
> >>  static void pci_stop_dev(struct pci_dev *dev)
> >>  {
> >> +       pci_pme_active(dev, false);
> >> +
> >>         if (dev->is_added) {
> >>                 pci_proc_detach_device(dev);
> >>                 pci_remove_sysfs_dev_files(dev);
> >
> > Indeed, the oops would prevent hitting the lockdep warning, which is a
> > secondary matter.
> >
> > As it stands, this fix prevents the fatal oops, so IMHO is urgent
> > material for 3.8-final. I can't prove that is_added will always be
> > true, so let's stick with your patch.
> >
> > Tested-by: Daniel J Blueman <daniel@quora.org>
> 
> I wrote a changelog and applied this to my for-linus branch.  Please
> take a look and confirm that it makes sense before I ask Linus to pull
> it.
> 
> http://git.kernel.org/?p=linux/kernel/git/helgaas/pci.git;a=commitdiff;h=249bfb83cf8ba658955f0245ac3981d941f746ee

Yes, it does, thanks a lot!

Rafael


-- 
I speak only for myself.
Rafael J. Wysocki, Intel Open Source Technology Center.

^ permalink raw reply	[flat|nested] 12+ messages in thread
end of thread, other threads:[~2013-02-13 20:48 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-02-11 13:03 [3.8-rc7] PCI hotplug wakeup oops Daniel J Blueman
2013-02-11 17:08 ` Daniel J Blueman
2013-02-11 19:01   ` Bjorn Helgaas
2013-02-11 19:27     ` Rafael J. Wysocki
2013-02-11 19:49       ` Rafael J. Wysocki
2013-02-12  2:18         ` Daniel J Blueman
2013-02-12 20:53           ` Rafael J. Wysocki
2013-02-12 20:50             ` Tejun Heo
2013-02-12 21:00               ` Rafael J. Wysocki
2013-02-13 14:32         ` Daniel J Blueman
2013-02-13 19:29           ` Bjorn Helgaas
2013-02-13 20:55             ` Rafael J. Wysocki

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).