* Re: Thunderbolt driver hotplug not working correctly
[not found] <6486222925419988442@unknownmsgid>
@ 2014-08-15 11:24 ` Steven Noonan
[not found] ` <CAMxnaaXwY1oyaws-SG3=DzO0qo12bT0-6V9mti7fO11--sp7Zw@mail.gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Steven Noonan @ 2014-08-15 11:24 UTC (permalink / raw)
To: Andreas Noever; +Cc: Greg KH, Linux Kernel mailing List
On Wed, Aug 13, 2014 at 4:05 PM, Andreas Noever
<andreas.noever@gmail.com> wrote:
> Hello Steven,
>
> I think that there are two problems:
> - The Kernel does not notice that the device is gone.
> - The first hotplug operation, after removing a coldplugged device fails.
>
> For the first one could you check whether thie pciehp (sub)-driver is loaded?
> (dmesg | grep pciehp should show something, the config option is
> CONFIG_HOTPLUG_PCI_PCIE).
>
> I was able reproduce the second problem on my machine. Could you test whether
> this patch fixes the problem?
>
With the patch I see that PCI bridge 09:00.0 survives the hotplug
events, but the bridge at 0a:00.0 and the Ethernet controller don't
survive.
>
> ---
> drivers/thunderbolt/path.c | 21 ++++++++++++++++++++-
> 1 file changed, 20 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/thunderbolt/path.c b/drivers/thunderbolt/path.c
> index 8fcf8a7..9562cd0 100644
> --- a/drivers/thunderbolt/path.c
> +++ b/drivers/thunderbolt/path.c
> @@ -150,7 +150,26 @@ int tb_path_activate(struct tb_path *path)
>
> /* Activate hops. */
> for (i = path->path_length - 1; i >= 0; i--) {
> - struct tb_regs_hop hop;
> + struct tb_regs_hop hop = { 0 };
> +
> + /*
> + * We do (currently) not tear down paths setup by the firmeware.
> + * If a firmware device is unplugged and plugged in again then
> + * it can happen that we reuse some of the hops from the (now
> + * defunct) firmeware path. This causes the hotplug operation to
> + * fail (the pci device does not show up). Clearing the hop
> + * before overwriting it fixes the problem.
> + *
> + * Should be removed once we discover and tear down firmeware
> + * paths.
> + */
> + res = tb_port_write(path->hops[i].in_port, &hop, TB_CFG_HOPS,
> + 2 * path->hops[i].in_hop_index, 2);
> + if (res) {
> + __tb_path_deactivate_hops(path, i);
> + __tb_path_deallocate_nfc(path, 0);
> + goto err;
> + }
>
> /* dword 0 */
> hop.next_hop = path->hops[i].next_hop_index;
> --
> 2.0.4
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Thunderbolt driver hotplug not working correctly
[not found] ` <20140815173539.GA2012@twoflower.us-west-2.compute.internal>
@ 2014-08-15 21:14 ` Andreas Noever
2014-08-26 15:58 ` Andreas Noever
0 siblings, 1 reply; 8+ messages in thread
From: Andreas Noever @ 2014-08-15 21:14 UTC (permalink / raw)
To: Steven Noonan; +Cc: Greg KH, Linux Kernel mailing List
(apparently I hit "reply" instead of "reply all" sometime back, sorry
for that. Readding ccs)
On Fri, Aug 15, 2014 at 7:35 PM, Steven Noonan <steven@uplinklabs.net> wrote:
> On Fri, Aug 15, 2014 at 04:03:08PM +0200, Andreas Noever wrote:
>> On Fri, Aug 15, 2014 at 2:48 PM, Steven Noonan <steven@uplinklabs.net> wrote:
>> > On Fri, Aug 15, 2014 at 5:41 AM, Andreas Noever
>> > <andreas.noever@gmail.com> wrote:
>> >> On Fri, Aug 15, 2014 at 1:24 PM, Steven Noonan <steven@uplinklabs.net> wrote:
>> >>> On Wed, Aug 13, 2014 at 4:05 PM, Andreas Noever
>> >>> <andreas.noever@gmail.com> wrote:
>> >>>> Hello Steven,
>> >>>>
>> >>>> I think that there are two problems:
>> >>>> - The Kernel does not notice that the device is gone.
>> >>>> - The first hotplug operation, after removing a coldplugged device fails.
>> >>>>
>> >>>> For the first one could you check whether thie pciehp (sub)-driver is loaded?
>> >>>> (dmesg | grep pciehp should show something, the config option is
>> >>>> CONFIG_HOTPLUG_PCI_PCIE).
>> >>>>
>> >>>> I was able reproduce the second problem on my machine. Could you test whether
>> >>>> this patch fixes the problem?
>> >>>>
>> >>>
>> >>> With the patch I see that PCI bridge 09:00.0 survives the hotplug
>> >>> events, but the bridge at 0a:00.0 and the Ethernet controller don't
>> >>> survive.
>> >>
>> >> Is CONFIG_HOTPLUG_PCI_PCIE set? Any output from pciehp?
>> >
>> > CONFIG_HOTPLUG_PCI_PCIE=y
>> >
>> > Aug 15 04:17:55 twoflower kernel: pci_hotplug: PCI Hot Plug PCI Core
>> > version: 0.5
>> > Aug 15 04:17:55 twoflower kernel: pciehp: Using ACPI for slot detection.
>> > Aug 15 04:17:55 twoflower kernel: pciehp 0000:07:00.0:pcie24: Slot #0
>> > AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> > Aug 15 04:17:55 twoflower kernel: pciehp 0000:07:00.0:pcie24: service
>> > driver pciehp loaded
>> > Aug 15 04:17:55 twoflower kernel: pciehp: PCI Express Hot Plug
>> > Controller Driver version: 0.4
>> >
>> > And that's all I get from pciehp.
>>
>> 07:00 is not one of the downstream ports. The driver should bind to
>> 07:03-06. (On my system :00 does not even have the hotplug cap set).
>>
>> Does pciehp.pciehp_force=1 help?
>
> That looks more sensible.
>
> Aug 15 10:20:18 twoflower kernel: Command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
> Aug 15 10:20:18 twoflower kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
> Aug 15 10:20:18 twoflower kernel: pciehp: Using ACPI for slot detection.
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Bypassing BIOS check for pciehp use on 0000:00:1c.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:00.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Device 0000:08:00.0 already exists at 0000:08:00, cannot hot-add
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Cannot add device at 0000:08:00
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:03.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Slot #3 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Device 0000:09:00.0 already exists at 0000:09:00, cannot hot-add
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot add device at 0000:09:00
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:04.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: Slot #4 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:05.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: Slot #5 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:06.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: Slot #6 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:0a:00.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Slot #9 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl- LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00000000 (issued 0 msec ago)
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Device 0000:0b:00.0 already exists at 0000:0b:00, cannot hot-add
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Cannot add device at 0000:0b:00
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>
> Though the "cannot hot-add" lines are worrying. The above is a boot with
> the Ethernet dongle attached at boot.
Yes this is strange. Either the hp driver is getting spurious hotplug
events or the thunderbolt driver tries to hotplug the already
configured device. Can you send me the full dmesg and lspci -vvnn
output for this scenario? Please also pass pciehp.pciehp_debug=1 to
the kernel.
> And here's a hotplug attempt (which at least successfully *removes* the device
> from the tg3 driver's perspective, but hot-adding the device still fails):
>
> Aug 15 10:24:03 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
> Aug 15 10:24:03 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot remove display device 0000:09:00.0
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Up event
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Device 0000:09:00.0 already exists at 0000:09:00, cannot hot-add
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot add device at 0000:09:00
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot remove display device 0000:09:00.0
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card not present on Slot(3-1)
> Aug 15 10:24:06 twoflower kernel: pciehp 0000:0a:00.0:pcie24: unloading service driver pciehp
> Aug 15 10:24:06 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00001038 (issued 232550 msec ago)
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card present on Slot(3-1)
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Up event
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: Link Up event ignored on slot(3-1): already powering on
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:0a:00.0
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Slot #9 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl- LLActRep+
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00000000 (issued 0 msec ago)
> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card not present on Slot(3-1)
> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: Link Down event ignored on slot(3-1): already powering off
"Cannot remove display device 0000:09:00.0"... The message comes from
http://lxr.free-electrons.com/source/drivers/pci/hotplug/pciehp_pci.c#L112
The pciehp driver tries to read from the removed device (which returns
0xffff) and thus it thinks that the VGA flag is set. I have no idea
why presence is true here (it is read a few lines earlier). This is of
course a little bit racy..
> Without the dongle attached at boot, the thunderbolt driver (and rest of the
> kernel, for that matter) still stays silent when hotplugging it:
>
> Aug 15 10:26:24 twoflower kernel: Command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
> Aug 15 10:26:24 twoflower kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Bypassing BIOS check for pciehp use on 0000:00:1c.0
> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: service driver pciehp loaded
> Aug 15 10:26:24 twoflower kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>
> Looking in lspci, it appears a bunch of devices (all of 06:00.0 and up) are
> missing, which explains the thunderbolt driver's silence. Does Apple's firmware
> only announce that the thunderbolt bus exists when a device is attached at
> boot?
Yes, you can try passing acpi_osi=Darwin. If that makes 06:00 etc.
appear then I would also be interested in dmesg and lspci -vvnn.
Thanks,
Andreas
>>
>> >>>>
>> >>>> ---
>> >>>> drivers/thunderbolt/path.c | 21 ++++++++++++++++++++-
>> >>>> 1 file changed, 20 insertions(+), 1 deletion(-)
>> >>>>
>> >>>> diff --git a/drivers/thunderbolt/path.c b/drivers/thunderbolt/path.c
>> >>>> index 8fcf8a7..9562cd0 100644
>> >>>> --- a/drivers/thunderbolt/path.c
>> >>>> +++ b/drivers/thunderbolt/path.c
>> >>>> @@ -150,7 +150,26 @@ int tb_path_activate(struct tb_path *path)
>> >>>>
>> >>>> /* Activate hops. */
>> >>>> for (i = path->path_length - 1; i >= 0; i--) {
>> >>>> - struct tb_regs_hop hop;
>> >>>> + struct tb_regs_hop hop = { 0 };
>> >>>> +
>> >>>> + /*
>> >>>> + * We do (currently) not tear down paths setup by the firmeware.
>> >>>> + * If a firmware device is unplugged and plugged in again then
>> >>>> + * it can happen that we reuse some of the hops from the (now
>> >>>> + * defunct) firmeware path. This causes the hotplug operation to
>> >>>> + * fail (the pci device does not show up). Clearing the hop
>> >>>> + * before overwriting it fixes the problem.
>> >>>> + *
>> >>>> + * Should be removed once we discover and tear down firmeware
>> >>>> + * paths.
>> >>>> + */
>> >>>> + res = tb_port_write(path->hops[i].in_port, &hop, TB_CFG_HOPS,
>> >>>> + 2 * path->hops[i].in_hop_index, 2);
>> >>>> + if (res) {
>> >>>> + __tb_path_deactivate_hops(path, i);
>> >>>> + __tb_path_deallocate_nfc(path, 0);
>> >>>> + goto err;
>> >>>> + }
>> >>>>
>> >>>> /* dword 0 */
>> >>>> hop.next_hop = path->hops[i].next_hop_index;
>> >>>> --
>> >>>> 2.0.4
>> >>>>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Thunderbolt driver hotplug not working correctly
2014-08-15 21:14 ` Andreas Noever
@ 2014-08-26 15:58 ` Andreas Noever
0 siblings, 0 replies; 8+ messages in thread
From: Andreas Noever @ 2014-08-26 15:58 UTC (permalink / raw)
To: Steven Noonan; +Cc: Greg KH, Linux Kernel mailing List
On Fri, Aug 15, 2014 at 11:14 PM, Andreas Noever
<andreas.noever@gmail.com> wrote:
> (apparently I hit "reply" instead of "reply all" sometime back, sorry
> for that. Readding ccs)
>
> On Fri, Aug 15, 2014 at 7:35 PM, Steven Noonan <steven@uplinklabs.net> wrote:
>> On Fri, Aug 15, 2014 at 04:03:08PM +0200, Andreas Noever wrote:
>>> On Fri, Aug 15, 2014 at 2:48 PM, Steven Noonan <steven@uplinklabs.net> wrote:
>>> > On Fri, Aug 15, 2014 at 5:41 AM, Andreas Noever
>>> > <andreas.noever@gmail.com> wrote:
>>> >> On Fri, Aug 15, 2014 at 1:24 PM, Steven Noonan <steven@uplinklabs.net> wrote:
>>> >>> On Wed, Aug 13, 2014 at 4:05 PM, Andreas Noever
>>> >>> <andreas.noever@gmail.com> wrote:
>>> >>>> Hello Steven,
>>> >>>>
>>> >>>> I think that there are two problems:
>>> >>>> - The Kernel does not notice that the device is gone.
>>> >>>> - The first hotplug operation, after removing a coldplugged device fails.
>>> >>>>
>>> >>>> For the first one could you check whether thie pciehp (sub)-driver is loaded?
>>> >>>> (dmesg | grep pciehp should show something, the config option is
>>> >>>> CONFIG_HOTPLUG_PCI_PCIE).
>>> >>>>
>>> >>>> I was able reproduce the second problem on my machine. Could you test whether
>>> >>>> this patch fixes the problem?
>>> >>>>
>>> >>>
>>> >>> With the patch I see that PCI bridge 09:00.0 survives the hotplug
>>> >>> events, but the bridge at 0a:00.0 and the Ethernet controller don't
>>> >>> survive.
>>> >>
>>> >> Is CONFIG_HOTPLUG_PCI_PCIE set? Any output from pciehp?
>>> >
>>> > CONFIG_HOTPLUG_PCI_PCIE=y
>>> >
>>> > Aug 15 04:17:55 twoflower kernel: pci_hotplug: PCI Hot Plug PCI Core
>>> > version: 0.5
>>> > Aug 15 04:17:55 twoflower kernel: pciehp: Using ACPI for slot detection.
>>> > Aug 15 04:17:55 twoflower kernel: pciehp 0000:07:00.0:pcie24: Slot #0
>>> > AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>>> > Aug 15 04:17:55 twoflower kernel: pciehp 0000:07:00.0:pcie24: service
>>> > driver pciehp loaded
>>> > Aug 15 04:17:55 twoflower kernel: pciehp: PCI Express Hot Plug
>>> > Controller Driver version: 0.4
>>> >
>>> > And that's all I get from pciehp.
>>>
>>> 07:00 is not one of the downstream ports. The driver should bind to
>>> 07:03-06. (On my system :00 does not even have the hotplug cap set).
>>>
>>> Does pciehp.pciehp_force=1 help?
>>
>> That looks more sensible.
>>
>> Aug 15 10:20:18 twoflower kernel: Command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
>> Aug 15 10:20:18 twoflower kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
>> Aug 15 10:20:18 twoflower kernel: pciehp: Using ACPI for slot detection.
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Bypassing BIOS check for pciehp use on 0000:00:1c.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:00.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Device 0000:08:00.0 already exists at 0000:08:00, cannot hot-add
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Cannot add device at 0000:08:00
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:03.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Slot #3 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Device 0000:09:00.0 already exists at 0000:09:00, cannot hot-add
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot add device at 0000:09:00
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:04.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: Slot #4 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:05.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: Slot #5 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:06.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: Slot #6 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:0a:00.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Slot #9 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl- LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00000000 (issued 0 msec ago)
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Device 0000:0b:00.0 already exists at 0000:0b:00, cannot hot-add
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Cannot add device at 0000:0b:00
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>
>> Though the "cannot hot-add" lines are worrying. The above is a boot with
>> the Ethernet dongle attached at boot.
>
> Yes this is strange. Either the hp driver is getting spurious hotplug
> events or the thunderbolt driver tries to hotplug the already
> configured device. Can you send me the full dmesg and lspci -vvnn
> output for this scenario? Please also pass pciehp.pciehp_debug=1 to
> the kernel.
>
>> And here's a hotplug attempt (which at least successfully *removes* the device
>> from the tg3 driver's perspective, but hot-adding the device still fails):
>>
>> Aug 15 10:24:03 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
>> Aug 15 10:24:03 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot remove display device 0000:09:00.0
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Up event
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Device 0000:09:00.0 already exists at 0000:09:00, cannot hot-add
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot add device at 0000:09:00
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot remove display device 0000:09:00.0
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card not present on Slot(3-1)
>> Aug 15 10:24:06 twoflower kernel: pciehp 0000:0a:00.0:pcie24: unloading service driver pciehp
>> Aug 15 10:24:06 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00001038 (issued 232550 msec ago)
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card present on Slot(3-1)
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Up event
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: Link Up event ignored on slot(3-1): already powering on
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:0a:00.0
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Slot #9 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl- LLActRep+
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00000000 (issued 0 msec ago)
>> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card not present on Slot(3-1)
>> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
>> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: Link Down event ignored on slot(3-1): already powering off
>
>
> "Cannot remove display device 0000:09:00.0"... The message comes from
> http://lxr.free-electrons.com/source/drivers/pci/hotplug/pciehp_pci.c#L112
>
> The pciehp driver tries to read from the removed device (which returns
> 0xffff) and thus it thinks that the VGA flag is set. I have no idea
> why presence is true here (it is read a few lines earlier). This is of
> course a little bit racy..
>
>> Without the dongle attached at boot, the thunderbolt driver (and rest of the
>> kernel, for that matter) still stays silent when hotplugging it:
>>
>> Aug 15 10:26:24 twoflower kernel: Command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
>> Aug 15 10:26:24 twoflower kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
>> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Bypassing BIOS check for pciehp use on 0000:00:1c.0
>> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: service driver pciehp loaded
>> Aug 15 10:26:24 twoflower kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>
>> Looking in lspci, it appears a bunch of devices (all of 06:00.0 and up) are
>> missing, which explains the thunderbolt driver's silence. Does Apple's firmware
>> only announce that the thunderbolt bus exists when a device is attached at
>> boot?
>
> Yes, you can try passing acpi_osi=Darwin. If that makes 06:00 etc.
> appear then I would also be interested in dmesg and lspci -vvnn.
If you have time can you also run a test with the acpi patches
applied? These would be the last four patches from
https://github.com/anoever/thunderbolt/tree/acpi_rebased
Try applying those and booting without a TB device attached and
without acpi/pciehp parameters. Check that the TB controller is
present (06:00.0 and below) and that pciehp gets loeaded for 07:03-06.
Then plug in a TB device.
> Thanks,
> Andreas
>
>>>
>>> >>>>
>>> >>>> ---
>>> >>>> drivers/thunderbolt/path.c | 21 ++++++++++++++++++++-
>>> >>>> 1 file changed, 20 insertions(+), 1 deletion(-)
>>> >>>>
>>> >>>> diff --git a/drivers/thunderbolt/path.c b/drivers/thunderbolt/path.c
>>> >>>> index 8fcf8a7..9562cd0 100644
>>> >>>> --- a/drivers/thunderbolt/path.c
>>> >>>> +++ b/drivers/thunderbolt/path.c
>>> >>>> @@ -150,7 +150,26 @@ int tb_path_activate(struct tb_path *path)
>>> >>>>
>>> >>>> /* Activate hops. */
>>> >>>> for (i = path->path_length - 1; i >= 0; i--) {
>>> >>>> - struct tb_regs_hop hop;
>>> >>>> + struct tb_regs_hop hop = { 0 };
>>> >>>> +
>>> >>>> + /*
>>> >>>> + * We do (currently) not tear down paths setup by the firmeware.
>>> >>>> + * If a firmware device is unplugged and plugged in again then
>>> >>>> + * it can happen that we reuse some of the hops from the (now
>>> >>>> + * defunct) firmeware path. This causes the hotplug operation to
>>> >>>> + * fail (the pci device does not show up). Clearing the hop
>>> >>>> + * before overwriting it fixes the problem.
>>> >>>> + *
>>> >>>> + * Should be removed once we discover and tear down firmeware
>>> >>>> + * paths.
>>> >>>> + */
>>> >>>> + res = tb_port_write(path->hops[i].in_port, &hop, TB_CFG_HOPS,
>>> >>>> + 2 * path->hops[i].in_hop_index, 2);
>>> >>>> + if (res) {
>>> >>>> + __tb_path_deactivate_hops(path, i);
>>> >>>> + __tb_path_deallocate_nfc(path, 0);
>>> >>>> + goto err;
>>> >>>> + }
>>> >>>>
>>> >>>> /* dword 0 */
>>> >>>> hop.next_hop = path->hops[i].next_hop_index;
>>> >>>> --
>>> >>>> 2.0.4
>>> >>>>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Thunderbolt driver hotplug not working correctly
2014-08-12 22:55 ` Andreas Noever
2014-08-13 23:05 ` Andreas Noever
@ 2014-08-15 11:19 ` Steven Noonan
1 sibling, 0 replies; 8+ messages in thread
From: Steven Noonan @ 2014-08-15 11:19 UTC (permalink / raw)
To: Andreas Noever; +Cc: Linux Kernel mailing List
On Tue, Aug 12, 2014 at 3:55 PM, Andreas Noever
<andreas.noever@gmail.com> wrote:
> On Tue, Aug 12, 2014 at 11:01 AM, Steven Noonan <steven@uplinklabs.net> wrote:
>> Hello Andreas,
>>
>> I'm trying out the new Thunderbolt driver in Linus' tree and I'm
>> noticing device hotplug isn't quite working correctly.
>>
>> I've got a Haswell 2013 MacBook Pro Retina and the Apple-vended
>> Thunderbolt Ethernet adapter, which uses the tg3 driver.
>>
>> Here's what dmesg says when I unplug:
>>
>> [ 1866.359118] thunderbolt 0000:08:00.0: resetting error on 0:3.
>> [ 1866.359150] thunderbolt 0000:08:00.0: 0:3: unplugged
>>
>> When I re-plug the device:
>>
>> [ 1872.481675] thunderbolt 0000:08:00.0: resetting error on 0:3.
>> [ 1872.481695] thunderbolt 0000:08:00.0: 0:3: hotplug: scanning
>> [ 1872.481764] thunderbolt 0000:08:00.0: 0:3: is connected, link is up (state: 2)
>> [ 1872.482078] thunderbolt 0000:08:00.0: initializing Switch at 0x3 (depth: 1, up port: 1)
>> [ 1872.482084] thunderbolt 0000:08:00.0: old switch config:
>> [ 1872.482089] thunderbolt 0000:08:00.0: Switch: 8086:1549 (Revision: 0, TB Version: 1)
>> [ 1872.482092] thunderbolt 0000:08:00.0: Max Port Number: 2
>> [ 1872.482094] thunderbolt 0000:08:00.0: Config:
>> [ 1872.482098] thunderbolt 0000:08:00.0: Upstream Port Number: 0 Depth: 0 Route String: 0x0 Enabled: 0, PlugEventsDelay: 10ms
>> [ 1872.482102] thunderbolt 0000:08:00.0: unknown1: 0x0 unknown4: 0x0
>> [ 1872.496213] thunderbolt 0000:08:00.0: 3: reading drom (length: 0x7b)
>> [ 1872.821969] thunderbolt 0000:08:00.0: 3: uid: 0x10002014eed70
>> [ 1872.822067] thunderbolt 0000:08:00.0: Port 0: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
>> [ 1872.822069] thunderbolt 0000:08:00.0: Max hop id (in/out): 7/7
>> [ 1872.822071] thunderbolt 0000:08:00.0: Max counters: 8
>> [ 1872.822072] thunderbolt 0000:08:00.0: NFC Credits: 0x700000
>> [ 1872.822566] thunderbolt 0000:08:00.0: Port 1: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
>> [ 1872.822567] thunderbolt 0000:08:00.0: Max hop id (in/out): 8/8
>> [ 1872.822569] thunderbolt 0000:08:00.0: Max counters: 4
>> [ 1872.822570] thunderbolt 0000:08:00.0: NFC Credits: 0x3c00000
>> [ 1872.822666] thunderbolt 0000:08:00.0: Port 2: 8086:1549 (Revision: 0, TB Version: 1, Type: PCIe (0x100102))
>> [ 1872.822667] thunderbolt 0000:08:00.0: Max hop id (in/out): 8/8
>> [ 1872.822668] thunderbolt 0000:08:00.0: Max counters: 2
>> [ 1872.822670] thunderbolt 0000:08:00.0: NFC Credits: 0x700000
>> [ 1872.822982] thunderbolt 0000:08:00.0: 3: hotplug: activating pcie devices
>> [ 1872.823576] thunderbolt 0000:08:00.0: 0:6 <-> 3:2 (PCI): activating
>> [ 1872.823580] thunderbolt 0000:08:00.0: activating path from 0:6 to 3:2
>> [ 1872.823582] thunderbolt 0000:08:00.0: 3:1: Writing hop 1, index 8
>> [ 1872.823583] thunderbolt 0000:08:00.0: 3:1: Hop through port 2 to hop 8 (enabled)
>> [ 1872.823585] thunderbolt 0000:08:00.0: 3:1: Weight: 1 Priority: 3 Credits: 16 Drop: 0
>> [ 1872.823587] thunderbolt 0000:08:00.0: 3:1: Counter enabled: 0 Counter index: 2047
>> [ 1872.823589] thunderbolt 0000:08:00.0: 3:1: Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
>> [ 1872.823591] thunderbolt 0000:08:00.0: 3:1: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
>> [ 1872.823664] thunderbolt 0000:08:00.0: 0:6: Writing hop 0, index 8
>> [ 1872.823666] thunderbolt 0000:08:00.0: 0:6: Hop through port 3 to hop 8 (enabled)
>> [ 1872.823667] thunderbolt 0000:08:00.0: 0:6: Weight: 1 Priority: 3 Credits: 7 Drop: 0
>> [ 1872.823669] thunderbolt 0000:08:00.0: 0:6: Counter enabled: 0 Counter index: 2047
>> [ 1872.823670] thunderbolt 0000:08:00.0: 0:6: Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
>> [ 1872.823672] thunderbolt 0000:08:00.0: 0:6: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
>> [ 1872.823764] thunderbolt 0000:08:00.0: path activation complete
>> [ 1872.823765] thunderbolt 0000:08:00.0: activating path from 3:2 to 0:6
>> [ 1872.823767] thunderbolt 0000:08:00.0: 0:3: Writing hop 1, index 8
>> [ 1872.823768] thunderbolt 0000:08:00.0: 0:3: Hop through port 6 to hop 8 (enabled)
>> [ 1872.823770] thunderbolt 0000:08:00.0: 0:3: Weight: 1 Priority: 3 Credits: 16 Drop: 0
>> [ 1872.823771] thunderbolt 0000:08:00.0: 0:3: Counter enabled: 0 Counter index: 2047
>> [ 1872.823773] thunderbolt 0000:08:00.0: 0:3: Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
>> [ 1872.823774] thunderbolt 0000:08:00.0: 0:3: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
>> [ 1872.823864] thunderbolt 0000:08:00.0: 3:2: Writing hop 0, index 8
>> [ 1872.823865] thunderbolt 0000:08:00.0: 3:2: Hop through port 1 to hop 8 (enabled)
>> [ 1872.823867] thunderbolt 0000:08:00.0: 3:2: Weight: 1 Priority: 3 Credits: 7 Drop: 0
>> [ 1872.823868] thunderbolt 0000:08:00.0: 3:2: Counter enabled: 0 Counter index: 2047
>> [ 1872.823870] thunderbolt 0000:08:00.0: 3:2: Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
>> [ 1872.823871] thunderbolt 0000:08:00.0: 3:2: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
>> [ 1872.823963] thunderbolt 0000:08:00.0: path activation complete
>>
>>
>> And the tg3 driver didn't notice anything happened at all during that
>> process, so I did an rmmod/modprobe:
>>
>> [ 1894.412903] tg3 0000:0b:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
>> [ 1894.795333] cfg80211: Calling CRDA to update world regulatory domain
>> [ 1895.295757] tg3.c:v3.137 (May 11, 2014)
>> [ 1895.315622] tg3 0000:0b:00.0: Refused to change power state, currently in D3
>> [ 1895.463145] tg3 0000:0b:00.0: phy probe failed, err -19
>> [ 1895.463485] tg3 0000:0b:00.0: Problem fetching invariants of chip, aborting
>>
>> Uh oh.
>>
>> Here's the diff in 'lspci -nnvv' between unplug/replug:
>>
>> --- before 2014-08-12 01:46:09.313859543 -0700
>> +++ after 2014-08-12 01:46:30.493241976 -0700
>> @@ -814,89 +814,89 @@
>> 07:03.0 PCI bridge [0604]: Intel Corporation Device [8086:156d] (prog-if 00 [Normal decode])
>> Physical Slot: 2
>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>> Latency: 0, Cache Line Size: 128 bytes
>> Bus: primary=07, secondary=09, subordinate=39, sec-latency=0
>> I/O behind bridge: 00003000-00003fff
>> Memory behind bridge: c2000000-c5ffffff
>> Prefetchable memory behind bridge: 00000000ce000000-00000000d20fffff
>> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
>> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>> Capabilities: [80] Power Management version 3
>> Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
>> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>> Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
>> Address: 00000000fee00378 Data: 0000
>> Capabilities: [ac] Subsystem: Device [2222:1111]
>> Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
>> DevCap: MaxPayload 128 bytes, PhantFunc 0
>> ExtTag+ RBE+
>> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
>> MaxPayload 128 bytes, MaxReadReq 512 bytes
>> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
>> LnkCap: Port #3, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
>> ClockPM- Surprise- LLActRep+ BwNot+
>> LnkCtl: ASPM Disabled; Disabled- CommClk+
>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>> - LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
>> + LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt-
>> SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
>> Slot #3, PowerLimit 0.000W; Interlock- NoCompl+
>> SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
>> Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
>> - SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
>> - Changed: MRL- PresDet- LinkState-
>> + SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
>> + Changed: MRL- PresDet+ LinkState+
>> DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd-
>> - DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
>> + DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
>> LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
>> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>> Compliance De-emphasis: -6dB
>> LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
>> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>> Capabilities: [100 v1] Device Serial Number 2e-dc-31-20-d3-f1-0c-00
>> Capabilities: [200 v1] Advanced Error Reporting
>> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
>> Capabilities: [300 v1] Virtual Channel
>> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>> Arb: Fixed- WRR32- WRR64- WRR128-
>> Ctrl: ArbSelect=Fixed
>> Status: InProgress-
>> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>> Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
>> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>> - Status: NegoPending- InProgress-
>> + Status: NegoPending+ InProgress-
>> Capabilities: [400 v1] Power Budgeting <?>
>> Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=050 <?>
>> Kernel driver in use: pcieport
>>
>> @@ -1122,193 +1122,16 @@
>> Kernel driver in use: thunderbolt
>> Kernel modules: thunderbolt
>>
>> -09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
>> - Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>> - Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>> - Latency: 0, Cache Line Size: 128 bytes
>> - Bus: primary=09, secondary=0a, subordinate=0b, sec-latency=0
>> - I/O behind bridge: 00003000-00003fff
>> - Memory behind bridge: c2000000-c24fffff
>> - Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
>> - Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
>> - BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>> - PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>> - Capabilities: [80] Power Management version 3
>> - Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
>> - Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>> - Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
>> - Address: 00000000fee003f8 Data: 0000
>> - Capabilities: [c0] Express (v2) Upstream Port, MSI 00
>> - DevCap: MaxPayload 128 bytes, PhantFunc 0
>> - ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
>> - DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>> - RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>> - MaxPayload 128 bytes, MaxReadReq 512 bytes
>> - DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
>> - LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
>> - ClockPM+ Surprise- LLActRep- BwNot-
>> - LnkCtl: ASPM Disabled; Disabled- CommClk+
>> - ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>> - LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>> - DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
>> - DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
>> - LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
>> - Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>> - Compliance De-emphasis: -6dB
>> - LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
>> - EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>> - Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
>> - Capabilities: [200 v1] Advanced Error Reporting
>> - UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> - UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> - UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>> - CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> - CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> - AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
>> - Capabilities: [300 v1] Virtual Channel
>> - Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>> - Arb: Fixed- WRR32- WRR64- WRR128-
>> - Ctrl: ArbSelect=Fixed
>> - Status: InProgress-
>> - VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>> - Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
>> - Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>> - Status: NegoPending- InProgress-
>> - Capabilities: [400 v1] Power Budgeting <?>
>> - Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
>> +09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
>> + !!! Unknown header type 7f
>> Kernel driver in use: pcieport
>>
>> -0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
>> - Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>> - Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>> - Latency: 0, Cache Line Size: 128 bytes
>> - Bus: primary=0a, secondary=0b, subordinate=0b, sec-latency=0
>> - I/O behind bridge: 00003000-00003fff
>> - Memory behind bridge: c2000000-c24fffff
>> - Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
>> - Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
>> - BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>> - PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>> - Capabilities: [80] Power Management version 3
>> - Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
>> - Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>> - Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
>> - Address: 00000000fee00418 Data: 0000
>> - Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
>> - DevCap: MaxPayload 128 bytes, PhantFunc 0
>> - ExtTag+ RBE+
>> - DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>> - RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>> - MaxPayload 128 bytes, MaxReadReq 512 bytes
>> - DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
>> - LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
>> - ClockPM- Surprise- LLActRep+ BwNot+
>> - LnkCtl: ASPM Disabled; Disabled- CommClk+
>> - ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>> - LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
>> - SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
>> - Slot #9, PowerLimit 0.000W; Interlock- NoCompl-
>> - SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
>> - Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
>> - SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
>> - Changed: MRL- PresDet- LinkState-
>> - DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
>> - DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
>> - LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
>> - Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>> - Compliance De-emphasis: -6dB
>> - LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
>> - EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>> - Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
>> - Capabilities: [200 v1] Advanced Error Reporting
>> - UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> - UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> - UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>> - CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>> - CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> - AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
>> - Capabilities: [300 v1] Virtual Channel
>> - Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>> - Arb: Fixed- WRR32- WRR64- WRR128-
>> - Ctrl: ArbSelect=Fixed
>> - Status: InProgress-
>> - VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>> - Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
>> - Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>> - Status: NegoPending- InProgress-
>> - Capabilities: [400 v1] Power Budgeting <?>
>> - Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
>> +0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
>> + !!! Unknown header type 7f
>> Kernel driver in use: pcieport
>>
>> -0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682]
>> - Subsystem: Apple Inc. Device [106b:00f6]
>> - Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>> - Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>> - Latency: 0, Cache Line Size: 128 bytes
>> - Interrupt: pin A routed to IRQ 16
>> - Region 0: Memory at ce000000 (64-bit, prefetchable) [size=64K]
>> - Region 2: Memory at ce010000 (64-bit, prefetchable) [size=64K]
>> - Expansion ROM at c2000000 [disabled] [size=64K]
>> - Capabilities: [48] Power Management version 3
>> - Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
>> - Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>> - Capabilities: [50] Vital Product Data
>> - Product Name: Thunderbolt Ethernet
>> - Read-only fields:
>> - [PN] Part number: BCM957762
>> - [EC] Engineering changes: 106679-15
>> - [SN] Serial number: 0123456789
>> - [MN] Manufacture ID: 31 34 65 34
>> - [RV] Reserved: checksum good, 53 byte(s) reserved
>> - Read/write fields:
>> - [YA] Asset tag: XYZ01234567
>> - [RW] Read-write area: 107 byte(s) free
>> - End
>> - Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>> - Address: 0000000000000000 Data: 0000
>> - Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
>> - Vector table: BAR=2 offset=00000000
>> - PBA: BAR=2 offset=00000120
>> - Capabilities: [ac] Express (v2) Endpoint, MSI 00
>> - DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
>> - ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>> - DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>> - RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop-
>> - MaxPayload 128 bytes, MaxReadReq 4096 bytes
>> - DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
>> - LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
>> - ClockPM+ Surprise- LLActRep- BwNot-
>> - LnkCtl: ASPM Disableparentd; RCB 64 bytes Disabled- CommClk+
>> - ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>> - LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>> - DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
>> - DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
>> - LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
>> - Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>> - Compliance De-emphasis: -6dB
>> - LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
>> - EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>> - Capabilities: [100 v1] Advanced Error Reporting
>> - UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> - UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> - UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>> - CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>> - CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> - AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
>> - Capabilities: [13c v1] Device Serial Number 00-00-a8-20-66-3c-64-4a
>> - Capabilities: [150 v1] Power Budgeting <?>
>> - Capabilities: [160 v1] Virtual Channel
>> - Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
>> - Arb: Fixed- WRR32- WRR64- WRR128-
>> - Ctrl: ArbSelect=Fixed
>> - Status: InProgress-
>> - VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>> - Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
>> - Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>> - Status: NegoPending- InProgress-
>> - Capabilities: [1b0 v1] Latency Tolerance Reporting
>> - Max snoop latency: 0ns
>> - Max no snoop latency: 0ns
>> +0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682] (rev ff) (prog-if ff)
>> + !!! Unknown header type 7f
>> Kernel driver in use: tg3
>> Kernel modules: tg3
>>
>>
>> So now I've got 3 confused devices that have fallen off the bus.
>>
>> If I ask the surviving parent bridge (07:03.0) to rescan for devices via
>> sysfs, this is what I get in dmesg:
>>
>> [ 3985.833685] pcieport 0000:09:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
>> [ 3985.833739] pcieport 0000:07:00.0: bridge window [io 0x1000-0x0fff] to [bus 08] add_size 1000
>> [ 3985.833743] pcieport 0000:07:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 08] add_size 200000
>> [ 3985.833767] pcieport 0000:07:04.0: bridge window [io 0x1000-0x0fff] to [bus 3a] add_size 1000
>> [ 3985.833770] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 3a] add_size 200000
>> [ 3985.833771] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff] to [bus 3a] add_size 200000
>> [ 3985.833784] pcieport 0000:07:06.0: bridge window [io 0x1000-0x0fff] to [bus 6c] add_size 1000
>> [ 3985.833786] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 6c] add_size 200000
>> [ 3985.833788] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff] to [bus 6c] add_size 200000
>> [ 3985.833792] pcieport 0000:07:00.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
>> [ 3985.833794] pcieport 0000:07:04.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
>> [ 3985.833796] pcieport 0000:07:04.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
>> [ 3985.833797] pcieport 0000:07:06.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
>> [ 3985.833799] pcieport 0000:07:06.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
>> [ 3985.833801] pcieport 0000:07:00.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000
>> [ 3985.833802] pcieport 0000:07:04.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000
>> [ 3985.833804] pcieport 0000:07:06.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000
>> [ 3985.833810] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833811] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833814] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
>> [ 3985.833815] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
>> [ 3985.833818] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833820] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833821] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
>> [ 3985.833823] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
>> [ 3985.833825] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833827] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833829] pcieport 0000:07:00.0: BAR 13: no space for [io size 0x1000]
>> [ 3985.833831] pcieport 0000:07:00.0: BAR 13: failed to assign [io size 0x1000]
>> [ 3985.833832] pcieport 0000:07:04.0: BAR 13: no space for [io size 0x1000]
>> [ 3985.833834] pcieport 0000:07:04.0: BAR 13: failed to assign [io size 0x1000]
>> [ 3985.833835] pcieport 0000:07:06.0: BAR 13: no space for [io size 0x1000]
>> [ 3985.833837] pcieport 0000:07:06.0: BAR 13: failed to assign [io size 0x1000]
>> [ 3985.833840] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
>> [ 3985.833841] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
>> [ 3985.833844] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833845] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833847] pcieport 0000:07:06.0: BAR 13: no space for [io size 0x1000]
>> [ 3985.833848] pcieport 0000:07:06.0: BAR 13: failed to assign [io size 0x1000]
>> [ 3985.833850] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
>> [ 3985.833852] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
>> [ 3985.833854] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833856] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833858] pcieport 0000:07:04.0: BAR 13: no space for [io size 0x1000]
>> [ 3985.833859] pcieport 0000:07:04.0: BAR 13: failed to assign [io size 0x1000]
>> [ 3985.833862] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833863] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833865] pcieport 0000:07:00.0: BAR 13: no space for [io size 0x1000]
>> [ 3985.833866] pcieport 0000:07:00.0: BAR 13: failed to assign [io size 0x1000]
>>
>> Hrm. Any idea what to do here? I also tried some 'setpci' hackery on the
>> surviving bridge with a secondary bus reset and a link retrain. Neither
>> seems to have any effect.
>>
>> - Steven
>
> Are you booting with the device plugged in at boot? The driver does
> not yet try to discover paths which were setup by the firmeware and we
> do not do everything we should if a boot time device is unplugged.
> Could you try to replug it a second time? If that does not work try
> booting without the device attached and add the acpi_osi=Darwin to the
> kernel command line.
>
> Andreas
Good point. I tried leaving it unplugged until the system booted but
then *nothing* happened with the thunderbolt driver. No messages at
boot or anything. Unloading/reloading the driver, still no messages.
Hrmph.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Thunderbolt driver hotplug not working correctly
2014-08-12 22:55 ` Andreas Noever
@ 2014-08-13 23:05 ` Andreas Noever
2014-08-15 11:19 ` Steven Noonan
1 sibling, 0 replies; 8+ messages in thread
From: Andreas Noever @ 2014-08-13 23:05 UTC (permalink / raw)
To: Steven Noonan; +Cc: Andreas Noever, Greg KH, Linux Kernel mailing List
Hello Steven,
I think that there are two problems:
- The Kernel does not notice that the device is gone.
- The first hotplug operation, after removing a coldplugged device fails.
For the first one could you check whether thie pciehp (sub)-driver is loaded?
(dmesg | grep pciehp should show something, the config option is
CONFIG_HOTPLUG_PCI_PCIE).
I was able reproduce the second problem on my machine. Could you test whether
this patch fixes the problem?
Thanks,
Andreas
---
drivers/thunderbolt/path.c | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/drivers/thunderbolt/path.c b/drivers/thunderbolt/path.c
index 8fcf8a7..9562cd0 100644
--- a/drivers/thunderbolt/path.c
+++ b/drivers/thunderbolt/path.c
@@ -150,7 +150,26 @@ int tb_path_activate(struct tb_path *path)
/* Activate hops. */
for (i = path->path_length - 1; i >= 0; i--) {
- struct tb_regs_hop hop;
+ struct tb_regs_hop hop = { 0 };
+
+ /*
+ * We do (currently) not tear down paths setup by the firmeware.
+ * If a firmware device is unplugged and plugged in again then
+ * it can happen that we reuse some of the hops from the (now
+ * defunct) firmeware path. This causes the hotplug operation to
+ * fail (the pci device does not show up). Clearing the hop
+ * before overwriting it fixes the problem.
+ *
+ * Should be removed once we discover and tear down firmeware
+ * paths.
+ */
+ res = tb_port_write(path->hops[i].in_port, &hop, TB_CFG_HOPS,
+ 2 * path->hops[i].in_hop_index, 2);
+ if (res) {
+ __tb_path_deactivate_hops(path, i);
+ __tb_path_deallocate_nfc(path, 0);
+ goto err;
+ }
/* dword 0 */
hop.next_hop = path->hops[i].next_hop_index;
--
2.0.4
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: Thunderbolt driver hotplug not working correctly
2014-08-12 9:01 Steven Noonan
2014-08-12 9:05 ` Greg KH
@ 2014-08-12 22:55 ` Andreas Noever
2014-08-13 23:05 ` Andreas Noever
2014-08-15 11:19 ` Steven Noonan
1 sibling, 2 replies; 8+ messages in thread
From: Andreas Noever @ 2014-08-12 22:55 UTC (permalink / raw)
To: Steven Noonan; +Cc: Linux Kernel mailing List
On Tue, Aug 12, 2014 at 11:01 AM, Steven Noonan <steven@uplinklabs.net> wrote:
> Hello Andreas,
>
> I'm trying out the new Thunderbolt driver in Linus' tree and I'm
> noticing device hotplug isn't quite working correctly.
>
> I've got a Haswell 2013 MacBook Pro Retina and the Apple-vended
> Thunderbolt Ethernet adapter, which uses the tg3 driver.
>
> Here's what dmesg says when I unplug:
>
> [ 1866.359118] thunderbolt 0000:08:00.0: resetting error on 0:3.
> [ 1866.359150] thunderbolt 0000:08:00.0: 0:3: unplugged
>
> When I re-plug the device:
>
> [ 1872.481675] thunderbolt 0000:08:00.0: resetting error on 0:3.
> [ 1872.481695] thunderbolt 0000:08:00.0: 0:3: hotplug: scanning
> [ 1872.481764] thunderbolt 0000:08:00.0: 0:3: is connected, link is up (state: 2)
> [ 1872.482078] thunderbolt 0000:08:00.0: initializing Switch at 0x3 (depth: 1, up port: 1)
> [ 1872.482084] thunderbolt 0000:08:00.0: old switch config:
> [ 1872.482089] thunderbolt 0000:08:00.0: Switch: 8086:1549 (Revision: 0, TB Version: 1)
> [ 1872.482092] thunderbolt 0000:08:00.0: Max Port Number: 2
> [ 1872.482094] thunderbolt 0000:08:00.0: Config:
> [ 1872.482098] thunderbolt 0000:08:00.0: Upstream Port Number: 0 Depth: 0 Route String: 0x0 Enabled: 0, PlugEventsDelay: 10ms
> [ 1872.482102] thunderbolt 0000:08:00.0: unknown1: 0x0 unknown4: 0x0
> [ 1872.496213] thunderbolt 0000:08:00.0: 3: reading drom (length: 0x7b)
> [ 1872.821969] thunderbolt 0000:08:00.0: 3: uid: 0x10002014eed70
> [ 1872.822067] thunderbolt 0000:08:00.0: Port 0: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
> [ 1872.822069] thunderbolt 0000:08:00.0: Max hop id (in/out): 7/7
> [ 1872.822071] thunderbolt 0000:08:00.0: Max counters: 8
> [ 1872.822072] thunderbolt 0000:08:00.0: NFC Credits: 0x700000
> [ 1872.822566] thunderbolt 0000:08:00.0: Port 1: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
> [ 1872.822567] thunderbolt 0000:08:00.0: Max hop id (in/out): 8/8
> [ 1872.822569] thunderbolt 0000:08:00.0: Max counters: 4
> [ 1872.822570] thunderbolt 0000:08:00.0: NFC Credits: 0x3c00000
> [ 1872.822666] thunderbolt 0000:08:00.0: Port 2: 8086:1549 (Revision: 0, TB Version: 1, Type: PCIe (0x100102))
> [ 1872.822667] thunderbolt 0000:08:00.0: Max hop id (in/out): 8/8
> [ 1872.822668] thunderbolt 0000:08:00.0: Max counters: 2
> [ 1872.822670] thunderbolt 0000:08:00.0: NFC Credits: 0x700000
> [ 1872.822982] thunderbolt 0000:08:00.0: 3: hotplug: activating pcie devices
> [ 1872.823576] thunderbolt 0000:08:00.0: 0:6 <-> 3:2 (PCI): activating
> [ 1872.823580] thunderbolt 0000:08:00.0: activating path from 0:6 to 3:2
> [ 1872.823582] thunderbolt 0000:08:00.0: 3:1: Writing hop 1, index 8
> [ 1872.823583] thunderbolt 0000:08:00.0: 3:1: Hop through port 2 to hop 8 (enabled)
> [ 1872.823585] thunderbolt 0000:08:00.0: 3:1: Weight: 1 Priority: 3 Credits: 16 Drop: 0
> [ 1872.823587] thunderbolt 0000:08:00.0: 3:1: Counter enabled: 0 Counter index: 2047
> [ 1872.823589] thunderbolt 0000:08:00.0: 3:1: Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
> [ 1872.823591] thunderbolt 0000:08:00.0: 3:1: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
> [ 1872.823664] thunderbolt 0000:08:00.0: 0:6: Writing hop 0, index 8
> [ 1872.823666] thunderbolt 0000:08:00.0: 0:6: Hop through port 3 to hop 8 (enabled)
> [ 1872.823667] thunderbolt 0000:08:00.0: 0:6: Weight: 1 Priority: 3 Credits: 7 Drop: 0
> [ 1872.823669] thunderbolt 0000:08:00.0: 0:6: Counter enabled: 0 Counter index: 2047
> [ 1872.823670] thunderbolt 0000:08:00.0: 0:6: Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
> [ 1872.823672] thunderbolt 0000:08:00.0: 0:6: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
> [ 1872.823764] thunderbolt 0000:08:00.0: path activation complete
> [ 1872.823765] thunderbolt 0000:08:00.0: activating path from 3:2 to 0:6
> [ 1872.823767] thunderbolt 0000:08:00.0: 0:3: Writing hop 1, index 8
> [ 1872.823768] thunderbolt 0000:08:00.0: 0:3: Hop through port 6 to hop 8 (enabled)
> [ 1872.823770] thunderbolt 0000:08:00.0: 0:3: Weight: 1 Priority: 3 Credits: 16 Drop: 0
> [ 1872.823771] thunderbolt 0000:08:00.0: 0:3: Counter enabled: 0 Counter index: 2047
> [ 1872.823773] thunderbolt 0000:08:00.0: 0:3: Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
> [ 1872.823774] thunderbolt 0000:08:00.0: 0:3: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
> [ 1872.823864] thunderbolt 0000:08:00.0: 3:2: Writing hop 0, index 8
> [ 1872.823865] thunderbolt 0000:08:00.0: 3:2: Hop through port 1 to hop 8 (enabled)
> [ 1872.823867] thunderbolt 0000:08:00.0: 3:2: Weight: 1 Priority: 3 Credits: 7 Drop: 0
> [ 1872.823868] thunderbolt 0000:08:00.0: 3:2: Counter enabled: 0 Counter index: 2047
> [ 1872.823870] thunderbolt 0000:08:00.0: 3:2: Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
> [ 1872.823871] thunderbolt 0000:08:00.0: 3:2: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
> [ 1872.823963] thunderbolt 0000:08:00.0: path activation complete
>
>
> And the tg3 driver didn't notice anything happened at all during that
> process, so I did an rmmod/modprobe:
>
> [ 1894.412903] tg3 0000:0b:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
> [ 1894.795333] cfg80211: Calling CRDA to update world regulatory domain
> [ 1895.295757] tg3.c:v3.137 (May 11, 2014)
> [ 1895.315622] tg3 0000:0b:00.0: Refused to change power state, currently in D3
> [ 1895.463145] tg3 0000:0b:00.0: phy probe failed, err -19
> [ 1895.463485] tg3 0000:0b:00.0: Problem fetching invariants of chip, aborting
>
> Uh oh.
>
> Here's the diff in 'lspci -nnvv' between unplug/replug:
>
> --- before 2014-08-12 01:46:09.313859543 -0700
> +++ after 2014-08-12 01:46:30.493241976 -0700
> @@ -814,89 +814,89 @@
> 07:03.0 PCI bridge [0604]: Intel Corporation Device [8086:156d] (prog-if 00 [Normal decode])
> Physical Slot: 2
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 128 bytes
> Bus: primary=07, secondary=09, subordinate=39, sec-latency=0
> I/O behind bridge: 00003000-00003fff
> Memory behind bridge: c2000000-c5ffffff
> Prefetchable memory behind bridge: 00000000ce000000-00000000d20fffff
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> Capabilities: [80] Power Management version 3
> Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
> Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
> Address: 00000000fee00378 Data: 0000
> Capabilities: [ac] Subsystem: Device [2222:1111]
> Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
> DevCap: MaxPayload 128 bytes, PhantFunc 0
> ExtTag+ RBE+
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
> LnkCap: Port #3, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
> ClockPM- Surprise- LLActRep+ BwNot+
> LnkCtl: ASPM Disabled; Disabled- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> - LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
> + LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt-
> SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
> Slot #3, PowerLimit 0.000W; Interlock- NoCompl+
> SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
> Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
> - SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
> - Changed: MRL- PresDet- LinkState-
> + SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
> + Changed: MRL- PresDet+ LinkState+
> DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd-
> - DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
> + DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
> LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> Compliance De-emphasis: -6dB
> LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> Capabilities: [100 v1] Device Serial Number 2e-dc-31-20-d3-f1-0c-00
> Capabilities: [200 v1] Advanced Error Reporting
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
> Capabilities: [300 v1] Virtual Channel
> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
> Arb: Fixed- WRR32- WRR64- WRR128-
> Ctrl: ArbSelect=Fixed
> Status: InProgress-
> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> - Status: NegoPending- InProgress-
> + Status: NegoPending+ InProgress-
> Capabilities: [400 v1] Power Budgeting <?>
> Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=050 <?>
> Kernel driver in use: pcieport
>
> @@ -1122,193 +1122,16 @@
> Kernel driver in use: thunderbolt
> Kernel modules: thunderbolt
>
> -09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
> - Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> - Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> - Latency: 0, Cache Line Size: 128 bytes
> - Bus: primary=09, secondary=0a, subordinate=0b, sec-latency=0
> - I/O behind bridge: 00003000-00003fff
> - Memory behind bridge: c2000000-c24fffff
> - Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
> - Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> - BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> - PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> - Capabilities: [80] Power Management version 3
> - Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
> - Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
> - Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
> - Address: 00000000fee003f8 Data: 0000
> - Capabilities: [c0] Express (v2) Upstream Port, MSI 00
> - DevCap: MaxPayload 128 bytes, PhantFunc 0
> - ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
> - DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> - RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
> - MaxPayload 128 bytes, MaxReadReq 512 bytes
> - DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
> - LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
> - ClockPM+ Surprise- LLActRep- BwNot-
> - LnkCtl: ASPM Disabled; Disabled- CommClk+
> - ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> - LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> - DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
> - DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
> - LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> - Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> - Compliance De-emphasis: -6dB
> - LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
> - EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> - Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
> - Capabilities: [200 v1] Advanced Error Reporting
> - UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> - UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> - UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> - CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> - CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> - AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
> - Capabilities: [300 v1] Virtual Channel
> - Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
> - Arb: Fixed- WRR32- WRR64- WRR128-
> - Ctrl: ArbSelect=Fixed
> - Status: InProgress-
> - VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> - Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
> - Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> - Status: NegoPending- InProgress-
> - Capabilities: [400 v1] Power Budgeting <?>
> - Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
> +09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
> + !!! Unknown header type 7f
> Kernel driver in use: pcieport
>
> -0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
> - Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> - Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> - Latency: 0, Cache Line Size: 128 bytes
> - Bus: primary=0a, secondary=0b, subordinate=0b, sec-latency=0
> - I/O behind bridge: 00003000-00003fff
> - Memory behind bridge: c2000000-c24fffff
> - Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
> - Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> - BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> - PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> - Capabilities: [80] Power Management version 3
> - Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
> - Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
> - Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
> - Address: 00000000fee00418 Data: 0000
> - Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
> - DevCap: MaxPayload 128 bytes, PhantFunc 0
> - ExtTag+ RBE+
> - DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> - RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
> - MaxPayload 128 bytes, MaxReadReq 512 bytes
> - DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
> - LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
> - ClockPM- Surprise- LLActRep+ BwNot+
> - LnkCtl: ASPM Disabled; Disabled- CommClk+
> - ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> - LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
> - SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
> - Slot #9, PowerLimit 0.000W; Interlock- NoCompl-
> - SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
> - Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
> - SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
> - Changed: MRL- PresDet- LinkState-
> - DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
> - DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
> - LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
> - Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> - Compliance De-emphasis: -6dB
> - LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
> - EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> - Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
> - Capabilities: [200 v1] Advanced Error Reporting
> - UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> - UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> - UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> - CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
> - CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> - AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
> - Capabilities: [300 v1] Virtual Channel
> - Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
> - Arb: Fixed- WRR32- WRR64- WRR128-
> - Ctrl: ArbSelect=Fixed
> - Status: InProgress-
> - VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> - Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
> - Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> - Status: NegoPending- InProgress-
> - Capabilities: [400 v1] Power Budgeting <?>
> - Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
> +0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
> + !!! Unknown header type 7f
> Kernel driver in use: pcieport
>
> -0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682]
> - Subsystem: Apple Inc. Device [106b:00f6]
> - Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> - Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> - Latency: 0, Cache Line Size: 128 bytes
> - Interrupt: pin A routed to IRQ 16
> - Region 0: Memory at ce000000 (64-bit, prefetchable) [size=64K]
> - Region 2: Memory at ce010000 (64-bit, prefetchable) [size=64K]
> - Expansion ROM at c2000000 [disabled] [size=64K]
> - Capabilities: [48] Power Management version 3
> - Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
> - Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
> - Capabilities: [50] Vital Product Data
> - Product Name: Thunderbolt Ethernet
> - Read-only fields:
> - [PN] Part number: BCM957762
> - [EC] Engineering changes: 106679-15
> - [SN] Serial number: 0123456789
> - [MN] Manufacture ID: 31 34 65 34
> - [RV] Reserved: checksum good, 53 byte(s) reserved
> - Read/write fields:
> - [YA] Asset tag: XYZ01234567
> - [RW] Read-write area: 107 byte(s) free
> - End
> - Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
> - Address: 0000000000000000 Data: 0000
> - Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
> - Vector table: BAR=2 offset=00000000
> - PBA: BAR=2 offset=00000120
> - Capabilities: [ac] Express (v2) Endpoint, MSI 00
> - DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
> - ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> - DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> - RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop-
> - MaxPayload 128 bytes, MaxReadReq 4096 bytes
> - DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
> - LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
> - ClockPM+ Surprise- LLActRep- BwNot-
> - LnkCtl: ASPM Disableparentd; RCB 64 bytes Disabled- CommClk+
> - ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> - LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> - DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
> - DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
> - LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> - Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> - Compliance De-emphasis: -6dB
> - LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
> - EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> - Capabilities: [100 v1] Advanced Error Reporting
> - UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> - UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> - UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> - CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
> - CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> - AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
> - Capabilities: [13c v1] Device Serial Number 00-00-a8-20-66-3c-64-4a
> - Capabilities: [150 v1] Power Budgeting <?>
> - Capabilities: [160 v1] Virtual Channel
> - Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
> - Arb: Fixed- WRR32- WRR64- WRR128-
> - Ctrl: ArbSelect=Fixed
> - Status: InProgress-
> - VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> - Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
> - Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> - Status: NegoPending- InProgress-
> - Capabilities: [1b0 v1] Latency Tolerance Reporting
> - Max snoop latency: 0ns
> - Max no snoop latency: 0ns
> +0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682] (rev ff) (prog-if ff)
> + !!! Unknown header type 7f
> Kernel driver in use: tg3
> Kernel modules: tg3
>
>
> So now I've got 3 confused devices that have fallen off the bus.
>
> If I ask the surviving parent bridge (07:03.0) to rescan for devices via
> sysfs, this is what I get in dmesg:
>
> [ 3985.833685] pcieport 0000:09:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
> [ 3985.833739] pcieport 0000:07:00.0: bridge window [io 0x1000-0x0fff] to [bus 08] add_size 1000
> [ 3985.833743] pcieport 0000:07:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 08] add_size 200000
> [ 3985.833767] pcieport 0000:07:04.0: bridge window [io 0x1000-0x0fff] to [bus 3a] add_size 1000
> [ 3985.833770] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 3a] add_size 200000
> [ 3985.833771] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff] to [bus 3a] add_size 200000
> [ 3985.833784] pcieport 0000:07:06.0: bridge window [io 0x1000-0x0fff] to [bus 6c] add_size 1000
> [ 3985.833786] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 6c] add_size 200000
> [ 3985.833788] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff] to [bus 6c] add_size 200000
> [ 3985.833792] pcieport 0000:07:00.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
> [ 3985.833794] pcieport 0000:07:04.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
> [ 3985.833796] pcieport 0000:07:04.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
> [ 3985.833797] pcieport 0000:07:06.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
> [ 3985.833799] pcieport 0000:07:06.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
> [ 3985.833801] pcieport 0000:07:00.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000
> [ 3985.833802] pcieport 0000:07:04.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000
> [ 3985.833804] pcieport 0000:07:06.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000
> [ 3985.833810] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833811] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833814] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
> [ 3985.833815] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
> [ 3985.833818] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833820] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833821] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
> [ 3985.833823] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
> [ 3985.833825] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833827] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833829] pcieport 0000:07:00.0: BAR 13: no space for [io size 0x1000]
> [ 3985.833831] pcieport 0000:07:00.0: BAR 13: failed to assign [io size 0x1000]
> [ 3985.833832] pcieport 0000:07:04.0: BAR 13: no space for [io size 0x1000]
> [ 3985.833834] pcieport 0000:07:04.0: BAR 13: failed to assign [io size 0x1000]
> [ 3985.833835] pcieport 0000:07:06.0: BAR 13: no space for [io size 0x1000]
> [ 3985.833837] pcieport 0000:07:06.0: BAR 13: failed to assign [io size 0x1000]
> [ 3985.833840] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
> [ 3985.833841] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
> [ 3985.833844] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833845] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833847] pcieport 0000:07:06.0: BAR 13: no space for [io size 0x1000]
> [ 3985.833848] pcieport 0000:07:06.0: BAR 13: failed to assign [io size 0x1000]
> [ 3985.833850] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
> [ 3985.833852] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
> [ 3985.833854] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833856] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833858] pcieport 0000:07:04.0: BAR 13: no space for [io size 0x1000]
> [ 3985.833859] pcieport 0000:07:04.0: BAR 13: failed to assign [io size 0x1000]
> [ 3985.833862] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833863] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833865] pcieport 0000:07:00.0: BAR 13: no space for [io size 0x1000]
> [ 3985.833866] pcieport 0000:07:00.0: BAR 13: failed to assign [io size 0x1000]
>
> Hrm. Any idea what to do here? I also tried some 'setpci' hackery on the
> surviving bridge with a secondary bus reset and a link retrain. Neither
> seems to have any effect.
>
> - Steven
Are you booting with the device plugged in at boot? The driver does
not yet try to discover paths which were setup by the firmeware and we
do not do everything we should if a boot time device is unplugged.
Could you try to replug it a second time? If that does not work try
booting without the device attached and add the acpi_osi=Darwin to the
kernel command line.
Andreas
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Thunderbolt driver hotplug not working correctly
2014-08-12 9:01 Steven Noonan
@ 2014-08-12 9:05 ` Greg KH
2014-08-12 22:55 ` Andreas Noever
1 sibling, 0 replies; 8+ messages in thread
From: Greg KH @ 2014-08-12 9:05 UTC (permalink / raw)
To: Steven Noonan, Matthew Garrett; +Cc: Linux Kernel mailing List, Andreas Noever
On Tue, Aug 12, 2014 at 02:01:38AM -0700, Steven Noonan wrote:
> Hello Andreas,
>
> I'm trying out the new Thunderbolt driver in Linus' tree and I'm
> noticing device hotplug isn't quite working correctly.
>
> I've got a Haswell 2013 MacBook Pro Retina and the Apple-vended
> Thunderbolt Ethernet adapter, which uses the tg3 driver.
I don't think that the relevant ACPI changes have landed in Linus's tree
yet to make this all work.
Matthew, have your changes be merged yet?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 8+ messages in thread
* Thunderbolt driver hotplug not working correctly
@ 2014-08-12 9:01 Steven Noonan
2014-08-12 9:05 ` Greg KH
2014-08-12 22:55 ` Andreas Noever
0 siblings, 2 replies; 8+ messages in thread
From: Steven Noonan @ 2014-08-12 9:01 UTC (permalink / raw)
To: Linux Kernel mailing List, Andreas Noever
Hello Andreas,
I'm trying out the new Thunderbolt driver in Linus' tree and I'm
noticing device hotplug isn't quite working correctly.
I've got a Haswell 2013 MacBook Pro Retina and the Apple-vended
Thunderbolt Ethernet adapter, which uses the tg3 driver.
Here's what dmesg says when I unplug:
[ 1866.359118] thunderbolt 0000:08:00.0: resetting error on 0:3.
[ 1866.359150] thunderbolt 0000:08:00.0: 0:3: unplugged
When I re-plug the device:
[ 1872.481675] thunderbolt 0000:08:00.0: resetting error on 0:3.
[ 1872.481695] thunderbolt 0000:08:00.0: 0:3: hotplug: scanning
[ 1872.481764] thunderbolt 0000:08:00.0: 0:3: is connected, link is up (state: 2)
[ 1872.482078] thunderbolt 0000:08:00.0: initializing Switch at 0x3 (depth: 1, up port: 1)
[ 1872.482084] thunderbolt 0000:08:00.0: old switch config:
[ 1872.482089] thunderbolt 0000:08:00.0: Switch: 8086:1549 (Revision: 0, TB Version: 1)
[ 1872.482092] thunderbolt 0000:08:00.0: Max Port Number: 2
[ 1872.482094] thunderbolt 0000:08:00.0: Config:
[ 1872.482098] thunderbolt 0000:08:00.0: Upstream Port Number: 0 Depth: 0 Route String: 0x0 Enabled: 0, PlugEventsDelay: 10ms
[ 1872.482102] thunderbolt 0000:08:00.0: unknown1: 0x0 unknown4: 0x0
[ 1872.496213] thunderbolt 0000:08:00.0: 3: reading drom (length: 0x7b)
[ 1872.821969] thunderbolt 0000:08:00.0: 3: uid: 0x10002014eed70
[ 1872.822067] thunderbolt 0000:08:00.0: Port 0: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
[ 1872.822069] thunderbolt 0000:08:00.0: Max hop id (in/out): 7/7
[ 1872.822071] thunderbolt 0000:08:00.0: Max counters: 8
[ 1872.822072] thunderbolt 0000:08:00.0: NFC Credits: 0x700000
[ 1872.822566] thunderbolt 0000:08:00.0: Port 1: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
[ 1872.822567] thunderbolt 0000:08:00.0: Max hop id (in/out): 8/8
[ 1872.822569] thunderbolt 0000:08:00.0: Max counters: 4
[ 1872.822570] thunderbolt 0000:08:00.0: NFC Credits: 0x3c00000
[ 1872.822666] thunderbolt 0000:08:00.0: Port 2: 8086:1549 (Revision: 0, TB Version: 1, Type: PCIe (0x100102))
[ 1872.822667] thunderbolt 0000:08:00.0: Max hop id (in/out): 8/8
[ 1872.822668] thunderbolt 0000:08:00.0: Max counters: 2
[ 1872.822670] thunderbolt 0000:08:00.0: NFC Credits: 0x700000
[ 1872.822982] thunderbolt 0000:08:00.0: 3: hotplug: activating pcie devices
[ 1872.823576] thunderbolt 0000:08:00.0: 0:6 <-> 3:2 (PCI): activating
[ 1872.823580] thunderbolt 0000:08:00.0: activating path from 0:6 to 3:2
[ 1872.823582] thunderbolt 0000:08:00.0: 3:1: Writing hop 1, index 8
[ 1872.823583] thunderbolt 0000:08:00.0: 3:1: Hop through port 2 to hop 8 (enabled)
[ 1872.823585] thunderbolt 0000:08:00.0: 3:1: Weight: 1 Priority: 3 Credits: 16 Drop: 0
[ 1872.823587] thunderbolt 0000:08:00.0: 3:1: Counter enabled: 0 Counter index: 2047
[ 1872.823589] thunderbolt 0000:08:00.0: 3:1: Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
[ 1872.823591] thunderbolt 0000:08:00.0: 3:1: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
[ 1872.823664] thunderbolt 0000:08:00.0: 0:6: Writing hop 0, index 8
[ 1872.823666] thunderbolt 0000:08:00.0: 0:6: Hop through port 3 to hop 8 (enabled)
[ 1872.823667] thunderbolt 0000:08:00.0: 0:6: Weight: 1 Priority: 3 Credits: 7 Drop: 0
[ 1872.823669] thunderbolt 0000:08:00.0: 0:6: Counter enabled: 0 Counter index: 2047
[ 1872.823670] thunderbolt 0000:08:00.0: 0:6: Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
[ 1872.823672] thunderbolt 0000:08:00.0: 0:6: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
[ 1872.823764] thunderbolt 0000:08:00.0: path activation complete
[ 1872.823765] thunderbolt 0000:08:00.0: activating path from 3:2 to 0:6
[ 1872.823767] thunderbolt 0000:08:00.0: 0:3: Writing hop 1, index 8
[ 1872.823768] thunderbolt 0000:08:00.0: 0:3: Hop through port 6 to hop 8 (enabled)
[ 1872.823770] thunderbolt 0000:08:00.0: 0:3: Weight: 1 Priority: 3 Credits: 16 Drop: 0
[ 1872.823771] thunderbolt 0000:08:00.0: 0:3: Counter enabled: 0 Counter index: 2047
[ 1872.823773] thunderbolt 0000:08:00.0: 0:3: Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
[ 1872.823774] thunderbolt 0000:08:00.0: 0:3: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
[ 1872.823864] thunderbolt 0000:08:00.0: 3:2: Writing hop 0, index 8
[ 1872.823865] thunderbolt 0000:08:00.0: 3:2: Hop through port 1 to hop 8 (enabled)
[ 1872.823867] thunderbolt 0000:08:00.0: 3:2: Weight: 1 Priority: 3 Credits: 7 Drop: 0
[ 1872.823868] thunderbolt 0000:08:00.0: 3:2: Counter enabled: 0 Counter index: 2047
[ 1872.823870] thunderbolt 0000:08:00.0: 3:2: Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
[ 1872.823871] thunderbolt 0000:08:00.0: 3:2: Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
[ 1872.823963] thunderbolt 0000:08:00.0: path activation complete
And the tg3 driver didn't notice anything happened at all during that
process, so I did an rmmod/modprobe:
[ 1894.412903] tg3 0000:0b:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
[ 1894.795333] cfg80211: Calling CRDA to update world regulatory domain
[ 1895.295757] tg3.c:v3.137 (May 11, 2014)
[ 1895.315622] tg3 0000:0b:00.0: Refused to change power state, currently in D3
[ 1895.463145] tg3 0000:0b:00.0: phy probe failed, err -19
[ 1895.463485] tg3 0000:0b:00.0: Problem fetching invariants of chip, aborting
Uh oh.
Here's the diff in 'lspci -nnvv' between unplug/replug:
--- before 2014-08-12 01:46:09.313859543 -0700
+++ after 2014-08-12 01:46:30.493241976 -0700
@@ -814,89 +814,89 @@
07:03.0 PCI bridge [0604]: Intel Corporation Device [8086:156d] (prog-if 00 [Normal decode])
Physical Slot: 2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 128 bytes
Bus: primary=07, secondary=09, subordinate=39, sec-latency=0
I/O behind bridge: 00003000-00003fff
Memory behind bridge: c2000000-c5ffffff
Prefetchable memory behind bridge: 00000000ce000000-00000000d20fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00378 Data: 0000
Capabilities: [ac] Subsystem: Device [2222:1111]
Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag+ RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #3, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
ClockPM- Surprise- LLActRep+ BwNot+
LnkCtl: ASPM Disabled; Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
- LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
+ LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
Slot #3, PowerLimit 0.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
- Changed: MRL- PresDet- LinkState-
+ SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
+ Changed: MRL- PresDet+ LinkState+
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd-
- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 2e-dc-31-20-d3-f1-0c-00
Capabilities: [200 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [300 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
- Status: NegoPending- InProgress-
+ Status: NegoPending+ InProgress-
Capabilities: [400 v1] Power Budgeting <?>
Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=050 <?>
Kernel driver in use: pcieport
@@ -1122,193 +1122,16 @@
Kernel driver in use: thunderbolt
Kernel modules: thunderbolt
-09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
- Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
- Latency: 0, Cache Line Size: 128 bytes
- Bus: primary=09, secondary=0a, subordinate=0b, sec-latency=0
- I/O behind bridge: 00003000-00003fff
- Memory behind bridge: c2000000-c24fffff
- Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
- Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
- Capabilities: [80] Power Management version 3
- Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
- Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
- Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
- Address: 00000000fee003f8 Data: 0000
- Capabilities: [c0] Express (v2) Upstream Port, MSI 00
- DevCap: MaxPayload 128 bytes, PhantFunc 0
- ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
- MaxPayload 128 bytes, MaxReadReq 512 bytes
- DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
- LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
- ClockPM+ Surprise- LLActRep- BwNot-
- LnkCtl: ASPM Disabled; Disabled- CommClk+
- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
- LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
- Compliance De-emphasis: -6dB
- LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
- Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
- Capabilities: [200 v1] Advanced Error Reporting
- UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
- Capabilities: [300 v1] Virtual Channel
- Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
- Arb: Fixed- WRR32- WRR64- WRR128-
- Ctrl: ArbSelect=Fixed
- Status: InProgress-
- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
- Status: NegoPending- InProgress-
- Capabilities: [400 v1] Power Budgeting <?>
- Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
+09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
+ !!! Unknown header type 7f
Kernel driver in use: pcieport
-0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
- Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
- Latency: 0, Cache Line Size: 128 bytes
- Bus: primary=0a, secondary=0b, subordinate=0b, sec-latency=0
- I/O behind bridge: 00003000-00003fff
- Memory behind bridge: c2000000-c24fffff
- Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
- Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
- Capabilities: [80] Power Management version 3
- Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
- Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
- Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
- Address: 00000000fee00418 Data: 0000
- Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
- DevCap: MaxPayload 128 bytes, PhantFunc 0
- ExtTag+ RBE+
- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
- MaxPayload 128 bytes, MaxReadReq 512 bytes
- DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
- LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
- ClockPM- Surprise- LLActRep+ BwNot+
- LnkCtl: ASPM Disabled; Disabled- CommClk+
- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
- Slot #9, PowerLimit 0.000W; Interlock- NoCompl-
- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
- Changed: MRL- PresDet- LinkState-
- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
- Compliance De-emphasis: -6dB
- LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
- Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
- Capabilities: [200 v1] Advanced Error Reporting
- UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
- CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
- Capabilities: [300 v1] Virtual Channel
- Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
- Arb: Fixed- WRR32- WRR64- WRR128-
- Ctrl: ArbSelect=Fixed
- Status: InProgress-
- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
- Status: NegoPending- InProgress-
- Capabilities: [400 v1] Power Budgeting <?>
- Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
+0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
+ !!! Unknown header type 7f
Kernel driver in use: pcieport
-0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682]
- Subsystem: Apple Inc. Device [106b:00f6]
- Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
- Latency: 0, Cache Line Size: 128 bytes
- Interrupt: pin A routed to IRQ 16
- Region 0: Memory at ce000000 (64-bit, prefetchable) [size=64K]
- Region 2: Memory at ce010000 (64-bit, prefetchable) [size=64K]
- Expansion ROM at c2000000 [disabled] [size=64K]
- Capabilities: [48] Power Management version 3
- Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
- Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
- Capabilities: [50] Vital Product Data
- Product Name: Thunderbolt Ethernet
- Read-only fields:
- [PN] Part number: BCM957762
- [EC] Engineering changes: 106679-15
- [SN] Serial number: 0123456789
- [MN] Manufacture ID: 31 34 65 34
- [RV] Reserved: checksum good, 53 byte(s) reserved
- Read/write fields:
- [YA] Asset tag: XYZ01234567
- [RW] Read-write area: 107 byte(s) free
- End
- Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
- Address: 0000000000000000 Data: 0000
- Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
- Vector table: BAR=2 offset=00000000
- PBA: BAR=2 offset=00000120
- Capabilities: [ac] Express (v2) Endpoint, MSI 00
- DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
- ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
- RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop-
- MaxPayload 128 bytes, MaxReadReq 4096 bytes
- DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
- ClockPM+ Surprise- LLActRep- BwNot-
- LnkCtl: ASPM Disableparentd; RCB 64 bytes Disabled- CommClk+
- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
- Compliance De-emphasis: -6dB
- LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
- Capabilities: [100 v1] Advanced Error Reporting
- UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
- AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
- Capabilities: [13c v1] Device Serial Number 00-00-a8-20-66-3c-64-4a
- Capabilities: [150 v1] Power Budgeting <?>
- Capabilities: [160 v1] Virtual Channel
- Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
- Arb: Fixed- WRR32- WRR64- WRR128-
- Ctrl: ArbSelect=Fixed
- Status: InProgress-
- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
- Status: NegoPending- InProgress-
- Capabilities: [1b0 v1] Latency Tolerance Reporting
- Max snoop latency: 0ns
- Max no snoop latency: 0ns
+0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682] (rev ff) (prog-if ff)
+ !!! Unknown header type 7f
Kernel driver in use: tg3
Kernel modules: tg3
So now I've got 3 confused devices that have fallen off the bus.
If I ask the surviving parent bridge (07:03.0) to rescan for devices via
sysfs, this is what I get in dmesg:
[ 3985.833685] pcieport 0000:09:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
[ 3985.833739] pcieport 0000:07:00.0: bridge window [io 0x1000-0x0fff] to [bus 08] add_size 1000
[ 3985.833743] pcieport 0000:07:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 08] add_size 200000
[ 3985.833767] pcieport 0000:07:04.0: bridge window [io 0x1000-0x0fff] to [bus 3a] add_size 1000
[ 3985.833770] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 3a] add_size 200000
[ 3985.833771] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff] to [bus 3a] add_size 200000
[ 3985.833784] pcieport 0000:07:06.0: bridge window [io 0x1000-0x0fff] to [bus 6c] add_size 1000
[ 3985.833786] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 6c] add_size 200000
[ 3985.833788] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff] to [bus 6c] add_size 200000
[ 3985.833792] pcieport 0000:07:00.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[ 3985.833794] pcieport 0000:07:04.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
[ 3985.833796] pcieport 0000:07:04.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[ 3985.833797] pcieport 0000:07:06.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
[ 3985.833799] pcieport 0000:07:06.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[ 3985.833801] pcieport 0000:07:00.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000
[ 3985.833802] pcieport 0000:07:04.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000
[ 3985.833804] pcieport 0000:07:06.0: res[13]=[io 0x1000-0x0fff] get_res_add_size add_size 1000
[ 3985.833810] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833811] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833814] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
[ 3985.833815] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
[ 3985.833818] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833820] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833821] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
[ 3985.833823] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
[ 3985.833825] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833827] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833829] pcieport 0000:07:00.0: BAR 13: no space for [io size 0x1000]
[ 3985.833831] pcieport 0000:07:00.0: BAR 13: failed to assign [io size 0x1000]
[ 3985.833832] pcieport 0000:07:04.0: BAR 13: no space for [io size 0x1000]
[ 3985.833834] pcieport 0000:07:04.0: BAR 13: failed to assign [io size 0x1000]
[ 3985.833835] pcieport 0000:07:06.0: BAR 13: no space for [io size 0x1000]
[ 3985.833837] pcieport 0000:07:06.0: BAR 13: failed to assign [io size 0x1000]
[ 3985.833840] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
[ 3985.833841] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
[ 3985.833844] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833845] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833847] pcieport 0000:07:06.0: BAR 13: no space for [io size 0x1000]
[ 3985.833848] pcieport 0000:07:06.0: BAR 13: failed to assign [io size 0x1000]
[ 3985.833850] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
[ 3985.833852] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
[ 3985.833854] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833856] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833858] pcieport 0000:07:04.0: BAR 13: no space for [io size 0x1000]
[ 3985.833859] pcieport 0000:07:04.0: BAR 13: failed to assign [io size 0x1000]
[ 3985.833862] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833863] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833865] pcieport 0000:07:00.0: BAR 13: no space for [io size 0x1000]
[ 3985.833866] pcieport 0000:07:00.0: BAR 13: failed to assign [io size 0x1000]
Hrm. Any idea what to do here? I also tried some 'setpci' hackery on the
surviving bridge with a secondary bus reset and a link retrain. Neither
seems to have any effect.
- Steven
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2014-08-26 15:58 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <6486222925419988442@unknownmsgid>
2014-08-15 11:24 ` Thunderbolt driver hotplug not working correctly Steven Noonan
[not found] ` <CAMxnaaXwY1oyaws-SG3=DzO0qo12bT0-6V9mti7fO11--sp7Zw@mail.gmail.com>
[not found] ` <CAKbGBLivQ1cP_EN=maQ0fHAUKfmpWyEY1T5frqv4nAqTdXw5GQ@mail.gmail.com>
[not found] ` <CAMxnaaXwrgh4XVUEhEVDfUJ=9n+OWD3Ncj-=FOHUQ1Jw5xwSvw@mail.gmail.com>
[not found] ` <20140815173539.GA2012@twoflower.us-west-2.compute.internal>
2014-08-15 21:14 ` Andreas Noever
2014-08-26 15:58 ` Andreas Noever
2014-08-12 9:01 Steven Noonan
2014-08-12 9:05 ` Greg KH
2014-08-12 22:55 ` Andreas Noever
2014-08-13 23:05 ` Andreas Noever
2014-08-15 11:19 ` Steven Noonan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.