All of lore.kernel.org
 help / color / mirror / Atom feed
* Re: Thunderbolt driver hotplug not working correctly
       [not found] <6486222925419988442@unknownmsgid>
@ 2014-08-15 11:24 ` Steven Noonan
       [not found]   ` <CAMxnaaXwY1oyaws-SG3=DzO0qo12bT0-6V9mti7fO11--sp7Zw@mail.gmail.com>
  0 siblings, 1 reply; 8+ messages in thread
From: Steven Noonan @ 2014-08-15 11:24 UTC (permalink / raw)
  To: Andreas Noever; +Cc: Greg KH, Linux Kernel mailing List

On Wed, Aug 13, 2014 at 4:05 PM, Andreas Noever
<andreas.noever@gmail.com> wrote:
> Hello Steven,
>
> I think that there are two problems:
>  - The Kernel does not notice that the device is gone.
>  - The first hotplug operation, after removing a coldplugged device fails.
>
> For the first one could you check whether thie pciehp (sub)-driver is loaded?
> (dmesg | grep pciehp should show something, the config option is
> CONFIG_HOTPLUG_PCI_PCIE).
>
> I was able reproduce the second problem on my machine. Could you test whether
> this patch fixes the problem?
>

With the patch I see that PCI bridge 09:00.0 survives the hotplug
events, but the bridge at 0a:00.0 and the Ethernet controller don't
survive.

>
> ---
>  drivers/thunderbolt/path.c | 21 ++++++++++++++++++++-
>  1 file changed, 20 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/thunderbolt/path.c b/drivers/thunderbolt/path.c
> index 8fcf8a7..9562cd0 100644
> --- a/drivers/thunderbolt/path.c
> +++ b/drivers/thunderbolt/path.c
> @@ -150,7 +150,26 @@ int tb_path_activate(struct tb_path *path)
>
>         /* Activate hops. */
>         for (i = path->path_length - 1; i >= 0; i--) {
> -               struct tb_regs_hop hop;
> +               struct tb_regs_hop hop = { 0 };
> +
> +               /*
> +                * We do (currently) not tear down paths setup by the firmeware.
> +                * If a firmware device is unplugged and plugged in again then
> +                * it can happen that we reuse some of the hops from the (now
> +                * defunct) firmeware path. This causes the hotplug operation to
> +                * fail (the pci device does not show up). Clearing the hop
> +                * before overwriting it fixes the problem.
> +                *
> +                * Should be removed once we discover and tear down firmeware
> +                * paths.
> +                */
> +               res = tb_port_write(path->hops[i].in_port, &hop, TB_CFG_HOPS,
> +                                   2 * path->hops[i].in_hop_index, 2);
> +               if (res) {
> +                       __tb_path_deactivate_hops(path, i);
> +                       __tb_path_deallocate_nfc(path, 0);
> +                       goto err;
> +               }
>
>                 /* dword 0 */
>                 hop.next_hop = path->hops[i].next_hop_index;
> --
> 2.0.4
>

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Thunderbolt driver hotplug not working correctly
       [not found]         ` <20140815173539.GA2012@twoflower.us-west-2.compute.internal>
@ 2014-08-15 21:14           ` Andreas Noever
  2014-08-26 15:58             ` Andreas Noever
  0 siblings, 1 reply; 8+ messages in thread
From: Andreas Noever @ 2014-08-15 21:14 UTC (permalink / raw)
  To: Steven Noonan; +Cc: Greg KH, Linux Kernel mailing List

(apparently I hit "reply" instead of "reply all" sometime back, sorry
for that. Readding ccs)

On Fri, Aug 15, 2014 at 7:35 PM, Steven Noonan <steven@uplinklabs.net> wrote:
> On Fri, Aug 15, 2014 at 04:03:08PM +0200, Andreas Noever wrote:
>> On Fri, Aug 15, 2014 at 2:48 PM, Steven Noonan <steven@uplinklabs.net> wrote:
>> > On Fri, Aug 15, 2014 at 5:41 AM, Andreas Noever
>> > <andreas.noever@gmail.com> wrote:
>> >> On Fri, Aug 15, 2014 at 1:24 PM, Steven Noonan <steven@uplinklabs.net> wrote:
>> >>> On Wed, Aug 13, 2014 at 4:05 PM, Andreas Noever
>> >>> <andreas.noever@gmail.com> wrote:
>> >>>> Hello Steven,
>> >>>>
>> >>>> I think that there are two problems:
>> >>>>  - The Kernel does not notice that the device is gone.
>> >>>>  - The first hotplug operation, after removing a coldplugged device fails.
>> >>>>
>> >>>> For the first one could you check whether thie pciehp (sub)-driver is loaded?
>> >>>> (dmesg | grep pciehp should show something, the config option is
>> >>>> CONFIG_HOTPLUG_PCI_PCIE).
>> >>>>
>> >>>> I was able reproduce the second problem on my machine. Could you test whether
>> >>>> this patch fixes the problem?
>> >>>>
>> >>>
>> >>> With the patch I see that PCI bridge 09:00.0 survives the hotplug
>> >>> events, but the bridge at 0a:00.0 and the Ethernet controller don't
>> >>> survive.
>> >>
>> >> Is CONFIG_HOTPLUG_PCI_PCIE set? Any output from pciehp?
>> >
>> > CONFIG_HOTPLUG_PCI_PCIE=y
>> >
>> > Aug 15 04:17:55 twoflower kernel: pci_hotplug: PCI Hot Plug PCI Core
>> > version: 0.5
>> > Aug 15 04:17:55 twoflower kernel: pciehp: Using ACPI for slot detection.
>> > Aug 15 04:17:55 twoflower kernel: pciehp 0000:07:00.0:pcie24: Slot #0
>> > AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> > Aug 15 04:17:55 twoflower kernel: pciehp 0000:07:00.0:pcie24: service
>> > driver pciehp loaded
>> > Aug 15 04:17:55 twoflower kernel: pciehp: PCI Express Hot Plug
>> > Controller Driver version: 0.4
>> >
>> > And that's all I get from pciehp.
>>
>> 07:00 is not one of the downstream ports. The driver should bind to
>> 07:03-06. (On my system :00 does not even have the hotplug cap set).
>>
>> Does pciehp.pciehp_force=1 help?
>
> That looks more sensible.
>
> Aug 15 10:20:18 twoflower kernel: Command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
> Aug 15 10:20:18 twoflower kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
> Aug 15 10:20:18 twoflower kernel: pciehp: Using ACPI for slot detection.
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Bypassing BIOS check for pciehp use on 0000:00:1c.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:00.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Device 0000:08:00.0 already exists at 0000:08:00, cannot hot-add
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Cannot add device at 0000:08:00
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:03.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Slot #3 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Device 0000:09:00.0 already exists at 0000:09:00, cannot hot-add
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot add device at 0000:09:00
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:04.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: Slot #4 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:05.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: Slot #5 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:06.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: Slot #6 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:0a:00.0
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Slot #9 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl- LLActRep+
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00000000 (issued 0 msec ago)
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Device 0000:0b:00.0 already exists at 0000:0b:00, cannot hot-add
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Cannot add device at 0000:0b:00
> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: service driver pciehp loaded
> Aug 15 10:20:18 twoflower kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>
> Though the "cannot hot-add" lines are worrying. The above is a boot with
> the Ethernet dongle attached at boot.

Yes this is strange. Either the hp driver is getting spurious hotplug
events or the thunderbolt driver tries to hotplug the already
configured device. Can you send me the full dmesg and lspci -vvnn
output for this scenario? Please also pass pciehp.pciehp_debug=1 to
the kernel.

> And here's a hotplug attempt (which at least successfully *removes* the device
> from the tg3 driver's perspective, but hot-adding the device still fails):
>
> Aug 15 10:24:03 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
> Aug 15 10:24:03 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot remove display device 0000:09:00.0
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Up event
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Device 0000:09:00.0 already exists at 0000:09:00, cannot hot-add
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot add device at 0000:09:00
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot remove display device 0000:09:00.0
> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card not present on Slot(3-1)
> Aug 15 10:24:06 twoflower kernel: pciehp 0000:0a:00.0:pcie24: unloading service driver pciehp
> Aug 15 10:24:06 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00001038 (issued 232550 msec ago)
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card present on Slot(3-1)
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Up event
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: Link Up event ignored on slot(3-1): already powering on
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:0a:00.0
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Slot #9 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl- LLActRep+
> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00000000 (issued 0 msec ago)
> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card not present on Slot(3-1)
> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: Link Down event ignored on slot(3-1): already powering off


"Cannot remove display device 0000:09:00.0"... The message comes from
http://lxr.free-electrons.com/source/drivers/pci/hotplug/pciehp_pci.c#L112

The pciehp driver tries to read from the removed device (which returns
0xffff) and thus it thinks that the VGA flag is set. I have no idea
why presence is true here (it is read a few lines earlier). This is of
course a little bit racy..

> Without the dongle attached at boot, the thunderbolt driver (and rest of the
> kernel, for that matter) still stays silent when hotplugging it:
>
> Aug 15 10:26:24 twoflower kernel: Command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
> Aug 15 10:26:24 twoflower kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Bypassing BIOS check for pciehp use on 0000:00:1c.0
> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: service driver pciehp loaded
> Aug 15 10:26:24 twoflower kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>
> Looking in lspci, it appears a bunch of devices (all of 06:00.0 and up) are
> missing, which explains the thunderbolt driver's silence. Does Apple's firmware
> only announce that the thunderbolt bus exists when a device is attached at
> boot?

Yes, you can try passing acpi_osi=Darwin. If that makes 06:00 etc.
appear then I would also be interested in dmesg and lspci -vvnn.

Thanks,
Andreas

>>
>> >>>>
>> >>>> ---
>> >>>>  drivers/thunderbolt/path.c | 21 ++++++++++++++++++++-
>> >>>>  1 file changed, 20 insertions(+), 1 deletion(-)
>> >>>>
>> >>>> diff --git a/drivers/thunderbolt/path.c b/drivers/thunderbolt/path.c
>> >>>> index 8fcf8a7..9562cd0 100644
>> >>>> --- a/drivers/thunderbolt/path.c
>> >>>> +++ b/drivers/thunderbolt/path.c
>> >>>> @@ -150,7 +150,26 @@ int tb_path_activate(struct tb_path *path)
>> >>>>
>> >>>>         /* Activate hops. */
>> >>>>         for (i = path->path_length - 1; i >= 0; i--) {
>> >>>> -               struct tb_regs_hop hop;
>> >>>> +               struct tb_regs_hop hop = { 0 };
>> >>>> +
>> >>>> +               /*
>> >>>> +                * We do (currently) not tear down paths setup by the firmeware.
>> >>>> +                * If a firmware device is unplugged and plugged in again then
>> >>>> +                * it can happen that we reuse some of the hops from the (now
>> >>>> +                * defunct) firmeware path. This causes the hotplug operation to
>> >>>> +                * fail (the pci device does not show up). Clearing the hop
>> >>>> +                * before overwriting it fixes the problem.
>> >>>> +                *
>> >>>> +                * Should be removed once we discover and tear down firmeware
>> >>>> +                * paths.
>> >>>> +                */
>> >>>> +               res = tb_port_write(path->hops[i].in_port, &hop, TB_CFG_HOPS,
>> >>>> +                                   2 * path->hops[i].in_hop_index, 2);
>> >>>> +               if (res) {
>> >>>> +                       __tb_path_deactivate_hops(path, i);
>> >>>> +                       __tb_path_deallocate_nfc(path, 0);
>> >>>> +                       goto err;
>> >>>> +               }
>> >>>>
>> >>>>                 /* dword 0 */
>> >>>>                 hop.next_hop = path->hops[i].next_hop_index;
>> >>>> --
>> >>>> 2.0.4
>> >>>>

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Thunderbolt driver hotplug not working correctly
  2014-08-15 21:14           ` Andreas Noever
@ 2014-08-26 15:58             ` Andreas Noever
  0 siblings, 0 replies; 8+ messages in thread
From: Andreas Noever @ 2014-08-26 15:58 UTC (permalink / raw)
  To: Steven Noonan; +Cc: Greg KH, Linux Kernel mailing List

On Fri, Aug 15, 2014 at 11:14 PM, Andreas Noever
<andreas.noever@gmail.com> wrote:
> (apparently I hit "reply" instead of "reply all" sometime back, sorry
> for that. Readding ccs)
>
> On Fri, Aug 15, 2014 at 7:35 PM, Steven Noonan <steven@uplinklabs.net> wrote:
>> On Fri, Aug 15, 2014 at 04:03:08PM +0200, Andreas Noever wrote:
>>> On Fri, Aug 15, 2014 at 2:48 PM, Steven Noonan <steven@uplinklabs.net> wrote:
>>> > On Fri, Aug 15, 2014 at 5:41 AM, Andreas Noever
>>> > <andreas.noever@gmail.com> wrote:
>>> >> On Fri, Aug 15, 2014 at 1:24 PM, Steven Noonan <steven@uplinklabs.net> wrote:
>>> >>> On Wed, Aug 13, 2014 at 4:05 PM, Andreas Noever
>>> >>> <andreas.noever@gmail.com> wrote:
>>> >>>> Hello Steven,
>>> >>>>
>>> >>>> I think that there are two problems:
>>> >>>>  - The Kernel does not notice that the device is gone.
>>> >>>>  - The first hotplug operation, after removing a coldplugged device fails.
>>> >>>>
>>> >>>> For the first one could you check whether thie pciehp (sub)-driver is loaded?
>>> >>>> (dmesg | grep pciehp should show something, the config option is
>>> >>>> CONFIG_HOTPLUG_PCI_PCIE).
>>> >>>>
>>> >>>> I was able reproduce the second problem on my machine. Could you test whether
>>> >>>> this patch fixes the problem?
>>> >>>>
>>> >>>
>>> >>> With the patch I see that PCI bridge 09:00.0 survives the hotplug
>>> >>> events, but the bridge at 0a:00.0 and the Ethernet controller don't
>>> >>> survive.
>>> >>
>>> >> Is CONFIG_HOTPLUG_PCI_PCIE set? Any output from pciehp?
>>> >
>>> > CONFIG_HOTPLUG_PCI_PCIE=y
>>> >
>>> > Aug 15 04:17:55 twoflower kernel: pci_hotplug: PCI Hot Plug PCI Core
>>> > version: 0.5
>>> > Aug 15 04:17:55 twoflower kernel: pciehp: Using ACPI for slot detection.
>>> > Aug 15 04:17:55 twoflower kernel: pciehp 0000:07:00.0:pcie24: Slot #0
>>> > AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>>> > Aug 15 04:17:55 twoflower kernel: pciehp 0000:07:00.0:pcie24: service
>>> > driver pciehp loaded
>>> > Aug 15 04:17:55 twoflower kernel: pciehp: PCI Express Hot Plug
>>> > Controller Driver version: 0.4
>>> >
>>> > And that's all I get from pciehp.
>>>
>>> 07:00 is not one of the downstream ports. The driver should bind to
>>> 07:03-06. (On my system :00 does not even have the hotplug cap set).
>>>
>>> Does pciehp.pciehp_force=1 help?
>>
>> That looks more sensible.
>>
>> Aug 15 10:20:18 twoflower kernel: Command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
>> Aug 15 10:20:18 twoflower kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
>> Aug 15 10:20:18 twoflower kernel: pciehp: Using ACPI for slot detection.
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Bypassing BIOS check for pciehp use on 0000:00:1c.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:00:1c.0:pcie04: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:00.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Device 0000:08:00.0 already exists at 0000:08:00, cannot hot-add
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: Cannot add device at 0000:08:00
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:00.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:03.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Slot #3 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Device 0000:09:00.0 already exists at 0000:09:00, cannot hot-add
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot add device at 0000:09:00
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:03.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:04.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: Slot #4 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:04.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:05.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: Slot #5 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:05.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: Bypassing BIOS check for pciehp use on 0000:07:06.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: Slot #6 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:07:06.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:0a:00.0
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Slot #9 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl- LLActRep+
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00000000 (issued 0 msec ago)
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Device 0000:0b:00.0 already exists at 0000:0b:00, cannot hot-add
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Cannot add device at 0000:0b:00
>> Aug 15 10:20:18 twoflower kernel: pciehp 0000:0a:00.0:pcie24: service driver pciehp loaded
>> Aug 15 10:20:18 twoflower kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>
>> Though the "cannot hot-add" lines are worrying. The above is a boot with
>> the Ethernet dongle attached at boot.
>
> Yes this is strange. Either the hp driver is getting spurious hotplug
> events or the thunderbolt driver tries to hotplug the already
> configured device. Can you send me the full dmesg and lspci -vvnn
> output for this scenario? Please also pass pciehp.pciehp_debug=1 to
> the kernel.
>
>> And here's a hotplug attempt (which at least successfully *removes* the device
>> from the tg3 driver's perspective, but hot-adding the device still fails):
>>
>> Aug 15 10:24:03 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
>> Aug 15 10:24:03 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot remove display device 0000:09:00.0
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Up event
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Device 0000:09:00.0 already exists at 0000:09:00, cannot hot-add
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot add device at 0000:09:00
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Cannot remove display device 0000:09:00.0
>> Aug 15 10:24:04 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card not present on Slot(3-1)
>> Aug 15 10:24:06 twoflower kernel: pciehp 0000:0a:00.0:pcie24: unloading service driver pciehp
>> Aug 15 10:24:06 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00001038 (issued 232550 msec ago)
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card present on Slot(3-1)
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Up event
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:07:03.0:pcie24: Link Up event ignored on slot(3-1): already powering on
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Bypassing BIOS check for pciehp use on 0000:0a:00.0
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Slot #9 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl- LLActRep+
>> Aug 15 10:24:27 twoflower kernel: pciehp 0000:0a:00.0:pcie24: Timeout on hotplug command 0x00000000 (issued 0 msec ago)
>> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: Card not present on Slot(3-1)
>> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: slot(3-1): Link Down event
>> Aug 15 10:24:47 twoflower kernel: pciehp 0000:07:03.0:pcie24: Link Down event ignored on slot(3-1): already powering off
>
>
> "Cannot remove display device 0000:09:00.0"... The message comes from
> http://lxr.free-electrons.com/source/drivers/pci/hotplug/pciehp_pci.c#L112
>
> The pciehp driver tries to read from the removed device (which returns
> 0xffff) and thus it thinks that the VGA flag is set. I have no idea
> why presence is true here (it is read a few lines earlier). This is of
> course a little bit racy..
>
>> Without the dongle attached at boot, the thunderbolt driver (and rest of the
>> kernel, for that matter) still stays silent when hotplugging it:
>>
>> Aug 15 10:26:24 twoflower kernel: Command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
>> Aug 15 10:26:24 twoflower kernel: Kernel command line: BOOT_IMAGE=/vmlinuz-3.16.0-ec2-11383-gc9d2642-dirty root=UUID=6146fd5a-e8b0-449f-8ba4-36676f089aae rw earlyprintk=verbose loglevel=5 libata.force=noncq rootflags=data=writeback intel_pstate=disable i915.lvds_channel_mode=2 pciehp.pciehp_force=1
>> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Bypassing BIOS check for pciehp use on 0000:00:1c.0
>> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: Slot #0 AttnBtn- AttnInd- PwrInd- PwrCtrl- MRL- Interlock- NoCompl+ LLActRep+
>> Aug 15 10:26:24 twoflower kernel: pciehp 0000:00:1c.0:pcie04: service driver pciehp loaded
>> Aug 15 10:26:24 twoflower kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
>>
>> Looking in lspci, it appears a bunch of devices (all of 06:00.0 and up) are
>> missing, which explains the thunderbolt driver's silence. Does Apple's firmware
>> only announce that the thunderbolt bus exists when a device is attached at
>> boot?
>
> Yes, you can try passing acpi_osi=Darwin. If that makes 06:00 etc.
> appear then I would also be interested in dmesg and lspci -vvnn.

If you have time can you also run a test with the acpi patches
applied? These would be the last four patches from
https://github.com/anoever/thunderbolt/tree/acpi_rebased

Try applying those and booting without a TB device attached and
without acpi/pciehp parameters. Check that the TB controller is
present (06:00.0 and below) and that pciehp gets loeaded for 07:03-06.
Then plug in a TB device.


> Thanks,
> Andreas
>
>>>
>>> >>>>
>>> >>>> ---
>>> >>>>  drivers/thunderbolt/path.c | 21 ++++++++++++++++++++-
>>> >>>>  1 file changed, 20 insertions(+), 1 deletion(-)
>>> >>>>
>>> >>>> diff --git a/drivers/thunderbolt/path.c b/drivers/thunderbolt/path.c
>>> >>>> index 8fcf8a7..9562cd0 100644
>>> >>>> --- a/drivers/thunderbolt/path.c
>>> >>>> +++ b/drivers/thunderbolt/path.c
>>> >>>> @@ -150,7 +150,26 @@ int tb_path_activate(struct tb_path *path)
>>> >>>>
>>> >>>>         /* Activate hops. */
>>> >>>>         for (i = path->path_length - 1; i >= 0; i--) {
>>> >>>> -               struct tb_regs_hop hop;
>>> >>>> +               struct tb_regs_hop hop = { 0 };
>>> >>>> +
>>> >>>> +               /*
>>> >>>> +                * We do (currently) not tear down paths setup by the firmeware.
>>> >>>> +                * If a firmware device is unplugged and plugged in again then
>>> >>>> +                * it can happen that we reuse some of the hops from the (now
>>> >>>> +                * defunct) firmeware path. This causes the hotplug operation to
>>> >>>> +                * fail (the pci device does not show up). Clearing the hop
>>> >>>> +                * before overwriting it fixes the problem.
>>> >>>> +                *
>>> >>>> +                * Should be removed once we discover and tear down firmeware
>>> >>>> +                * paths.
>>> >>>> +                */
>>> >>>> +               res = tb_port_write(path->hops[i].in_port, &hop, TB_CFG_HOPS,
>>> >>>> +                                   2 * path->hops[i].in_hop_index, 2);
>>> >>>> +               if (res) {
>>> >>>> +                       __tb_path_deactivate_hops(path, i);
>>> >>>> +                       __tb_path_deallocate_nfc(path, 0);
>>> >>>> +                       goto err;
>>> >>>> +               }
>>> >>>>
>>> >>>>                 /* dword 0 */
>>> >>>>                 hop.next_hop = path->hops[i].next_hop_index;
>>> >>>> --
>>> >>>> 2.0.4
>>> >>>>

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Thunderbolt driver hotplug not working correctly
  2014-08-12 22:55 ` Andreas Noever
  2014-08-13 23:05   ` Andreas Noever
@ 2014-08-15 11:19   ` Steven Noonan
  1 sibling, 0 replies; 8+ messages in thread
From: Steven Noonan @ 2014-08-15 11:19 UTC (permalink / raw)
  To: Andreas Noever; +Cc: Linux Kernel mailing List

On Tue, Aug 12, 2014 at 3:55 PM, Andreas Noever
<andreas.noever@gmail.com> wrote:
> On Tue, Aug 12, 2014 at 11:01 AM, Steven Noonan <steven@uplinklabs.net> wrote:
>> Hello Andreas,
>>
>> I'm trying out the new Thunderbolt driver in Linus' tree and I'm
>> noticing device hotplug isn't quite working correctly.
>>
>> I've got a Haswell 2013 MacBook Pro Retina and the Apple-vended
>> Thunderbolt Ethernet adapter, which uses the tg3 driver.
>>
>> Here's what dmesg says when I unplug:
>>
>> [ 1866.359118] thunderbolt 0000:08:00.0: resetting error on 0:3.
>> [ 1866.359150] thunderbolt 0000:08:00.0: 0:3: unplugged
>>
>> When I re-plug the device:
>>
>> [ 1872.481675] thunderbolt 0000:08:00.0: resetting error on 0:3.
>> [ 1872.481695] thunderbolt 0000:08:00.0: 0:3: hotplug: scanning
>> [ 1872.481764] thunderbolt 0000:08:00.0: 0:3: is connected, link is up (state: 2)
>> [ 1872.482078] thunderbolt 0000:08:00.0: initializing Switch at 0x3 (depth: 1, up port: 1)
>> [ 1872.482084] thunderbolt 0000:08:00.0: old switch config:
>> [ 1872.482089] thunderbolt 0000:08:00.0:  Switch: 8086:1549 (Revision: 0, TB Version: 1)
>> [ 1872.482092] thunderbolt 0000:08:00.0:   Max Port Number: 2
>> [ 1872.482094] thunderbolt 0000:08:00.0:   Config:
>> [ 1872.482098] thunderbolt 0000:08:00.0:    Upstream Port Number: 0 Depth: 0 Route String: 0x0 Enabled: 0, PlugEventsDelay: 10ms
>> [ 1872.482102] thunderbolt 0000:08:00.0:    unknown1: 0x0 unknown4: 0x0
>> [ 1872.496213] thunderbolt 0000:08:00.0: 3: reading drom (length: 0x7b)
>> [ 1872.821969] thunderbolt 0000:08:00.0: 3: uid: 0x10002014eed70
>> [ 1872.822067] thunderbolt 0000:08:00.0:  Port 0: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
>> [ 1872.822069] thunderbolt 0000:08:00.0:   Max hop id (in/out): 7/7
>> [ 1872.822071] thunderbolt 0000:08:00.0:   Max counters: 8
>> [ 1872.822072] thunderbolt 0000:08:00.0:   NFC Credits: 0x700000
>> [ 1872.822566] thunderbolt 0000:08:00.0:  Port 1: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
>> [ 1872.822567] thunderbolt 0000:08:00.0:   Max hop id (in/out): 8/8
>> [ 1872.822569] thunderbolt 0000:08:00.0:   Max counters: 4
>> [ 1872.822570] thunderbolt 0000:08:00.0:   NFC Credits: 0x3c00000
>> [ 1872.822666] thunderbolt 0000:08:00.0:  Port 2: 8086:1549 (Revision: 0, TB Version: 1, Type: PCIe (0x100102))
>> [ 1872.822667] thunderbolt 0000:08:00.0:   Max hop id (in/out): 8/8
>> [ 1872.822668] thunderbolt 0000:08:00.0:   Max counters: 2
>> [ 1872.822670] thunderbolt 0000:08:00.0:   NFC Credits: 0x700000
>> [ 1872.822982] thunderbolt 0000:08:00.0: 3: hotplug: activating pcie devices
>> [ 1872.823576] thunderbolt 0000:08:00.0: 0:6 <-> 3:2 (PCI): activating
>> [ 1872.823580] thunderbolt 0000:08:00.0: activating path from 0:6 to 3:2
>> [ 1872.823582] thunderbolt 0000:08:00.0: 3:1: Writing hop 1, index 8
>> [ 1872.823583] thunderbolt 0000:08:00.0: 3:1:  Hop through port 2 to hop 8 (enabled)
>> [ 1872.823585] thunderbolt 0000:08:00.0: 3:1:   Weight: 1 Priority: 3 Credits: 16 Drop: 0
>> [ 1872.823587] thunderbolt 0000:08:00.0: 3:1:    Counter enabled: 0 Counter index: 2047
>> [ 1872.823589] thunderbolt 0000:08:00.0: 3:1:   Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
>> [ 1872.823591] thunderbolt 0000:08:00.0: 3:1:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
>> [ 1872.823664] thunderbolt 0000:08:00.0: 0:6: Writing hop 0, index 8
>> [ 1872.823666] thunderbolt 0000:08:00.0: 0:6:  Hop through port 3 to hop 8 (enabled)
>> [ 1872.823667] thunderbolt 0000:08:00.0: 0:6:   Weight: 1 Priority: 3 Credits: 7 Drop: 0
>> [ 1872.823669] thunderbolt 0000:08:00.0: 0:6:    Counter enabled: 0 Counter index: 2047
>> [ 1872.823670] thunderbolt 0000:08:00.0: 0:6:   Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
>> [ 1872.823672] thunderbolt 0000:08:00.0: 0:6:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
>> [ 1872.823764] thunderbolt 0000:08:00.0: path activation complete
>> [ 1872.823765] thunderbolt 0000:08:00.0: activating path from 3:2 to 0:6
>> [ 1872.823767] thunderbolt 0000:08:00.0: 0:3: Writing hop 1, index 8
>> [ 1872.823768] thunderbolt 0000:08:00.0: 0:3:  Hop through port 6 to hop 8 (enabled)
>> [ 1872.823770] thunderbolt 0000:08:00.0: 0:3:   Weight: 1 Priority: 3 Credits: 16 Drop: 0
>> [ 1872.823771] thunderbolt 0000:08:00.0: 0:3:    Counter enabled: 0 Counter index: 2047
>> [ 1872.823773] thunderbolt 0000:08:00.0: 0:3:   Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
>> [ 1872.823774] thunderbolt 0000:08:00.0: 0:3:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
>> [ 1872.823864] thunderbolt 0000:08:00.0: 3:2: Writing hop 0, index 8
>> [ 1872.823865] thunderbolt 0000:08:00.0: 3:2:  Hop through port 1 to hop 8 (enabled)
>> [ 1872.823867] thunderbolt 0000:08:00.0: 3:2:   Weight: 1 Priority: 3 Credits: 7 Drop: 0
>> [ 1872.823868] thunderbolt 0000:08:00.0: 3:2:    Counter enabled: 0 Counter index: 2047
>> [ 1872.823870] thunderbolt 0000:08:00.0: 3:2:   Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
>> [ 1872.823871] thunderbolt 0000:08:00.0: 3:2:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
>> [ 1872.823963] thunderbolt 0000:08:00.0: path activation complete
>>
>>
>> And the tg3 driver didn't notice anything happened at all during that
>> process, so I did an rmmod/modprobe:
>>
>> [ 1894.412903] tg3 0000:0b:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
>> [ 1894.795333] cfg80211: Calling CRDA to update world regulatory domain
>> [ 1895.295757] tg3.c:v3.137 (May 11, 2014)
>> [ 1895.315622] tg3 0000:0b:00.0: Refused to change power state, currently in D3
>> [ 1895.463145] tg3 0000:0b:00.0: phy probe failed, err -19
>> [ 1895.463485] tg3 0000:0b:00.0: Problem fetching invariants of chip, aborting
>>
>> Uh oh.
>>
>> Here's the diff in 'lspci -nnvv' between unplug/replug:
>>
>> --- before      2014-08-12 01:46:09.313859543 -0700
>> +++ after       2014-08-12 01:46:30.493241976 -0700
>> @@ -814,89 +814,89 @@
>>  07:03.0 PCI bridge [0604]: Intel Corporation Device [8086:156d] (prog-if 00 [Normal decode])
>>         Physical Slot: 2
>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>>         Latency: 0, Cache Line Size: 128 bytes
>>         Bus: primary=07, secondary=09, subordinate=39, sec-latency=0
>>         I/O behind bridge: 00003000-00003fff
>>         Memory behind bridge: c2000000-c5ffffff
>>         Prefetchable memory behind bridge: 00000000ce000000-00000000d20fffff
>>         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
>>         BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>         Capabilities: [80] Power Management version 3
>>                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
>>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>>         Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
>>                 Address: 00000000fee00378  Data: 0000
>>         Capabilities: [ac] Subsystem: Device [2222:1111]
>>         Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
>>                 DevCap: MaxPayload 128 bytes, PhantFunc 0
>>                         ExtTag+ RBE+
>>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>>                         RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
>>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
>>                 LnkCap: Port #3, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
>>                         ClockPM- Surprise- LLActRep+ BwNot+
>>                 LnkCtl: ASPM Disabled; Disabled- CommClk+
>>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>> -               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
>> +               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt-
>>                 SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
>>                         Slot #3, PowerLimit 0.000W; Interlock- NoCompl+
>>                 SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
>>                         Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
>> -               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
>> -                       Changed: MRL- PresDet- LinkState-
>> +               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
>> +                       Changed: MRL- PresDet+ LinkState+
>>                 DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd-
>> -               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
>> +               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
>>                 LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
>>                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>>                          Compliance De-emphasis: -6dB
>>                 LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
>>                          EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>>         Capabilities: [100 v1] Device Serial Number 2e-dc-31-20-d3-f1-0c-00
>>         Capabilities: [200 v1] Advanced Error Reporting
>>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>>                 AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
>>         Capabilities: [300 v1] Virtual Channel
>>                 Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
>>                 Arb:    Fixed- WRR32- WRR64- WRR128-
>>                 Ctrl:   ArbSelect=Fixed
>>                 Status: InProgress-
>>                 VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>>                         Arb:    Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
>>                         Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>> -                       Status: NegoPending- InProgress-
>> +                       Status: NegoPending+ InProgress-
>>         Capabilities: [400 v1] Power Budgeting <?>
>>         Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=050 <?>
>>         Kernel driver in use: pcieport
>>
>> @@ -1122,193 +1122,16 @@
>>         Kernel driver in use: thunderbolt
>>         Kernel modules: thunderbolt
>>
>> -09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
>> -       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>> -       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>> -       Latency: 0, Cache Line Size: 128 bytes
>> -       Bus: primary=09, secondary=0a, subordinate=0b, sec-latency=0
>> -       I/O behind bridge: 00003000-00003fff
>> -       Memory behind bridge: c2000000-c24fffff
>> -       Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
>> -       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
>> -       BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>> -               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>> -       Capabilities: [80] Power Management version 3
>> -               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
>> -               Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>> -       Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
>> -               Address: 00000000fee003f8  Data: 0000
>> -       Capabilities: [c0] Express (v2) Upstream Port, MSI 00
>> -               DevCap: MaxPayload 128 bytes, PhantFunc 0
>> -                       ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
>> -               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>> -                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>> -                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>> -               DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
>> -               LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
>> -                       ClockPM+ Surprise- LLActRep- BwNot-
>> -               LnkCtl: ASPM Disabled; Disabled- CommClk+
>> -                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>> -               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>> -               DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
>> -               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
>> -               LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
>> -                        Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>> -                        Compliance De-emphasis: -6dB
>> -               LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
>> -                        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>> -       Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
>> -       Capabilities: [200 v1] Advanced Error Reporting
>> -               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> -               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> -               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>> -               CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> -               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> -               AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
>> -       Capabilities: [300 v1] Virtual Channel
>> -               Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
>> -               Arb:    Fixed- WRR32- WRR64- WRR128-
>> -               Ctrl:   ArbSelect=Fixed
>> -               Status: InProgress-
>> -               VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>> -                       Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
>> -                       Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>> -                       Status: NegoPending- InProgress-
>> -       Capabilities: [400 v1] Power Budgeting <?>
>> -       Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
>> +09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
>> +       !!! Unknown header type 7f
>>         Kernel driver in use: pcieport
>>
>> -0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
>> -       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>> -       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>> -       Latency: 0, Cache Line Size: 128 bytes
>> -       Bus: primary=0a, secondary=0b, subordinate=0b, sec-latency=0
>> -       I/O behind bridge: 00003000-00003fff
>> -       Memory behind bridge: c2000000-c24fffff
>> -       Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
>> -       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
>> -       BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>> -               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>> -       Capabilities: [80] Power Management version 3
>> -               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
>> -               Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>> -       Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
>> -               Address: 00000000fee00418  Data: 0000
>> -       Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
>> -               DevCap: MaxPayload 128 bytes, PhantFunc 0
>> -                       ExtTag+ RBE+
>> -               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>> -                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>> -                       MaxPayload 128 bytes, MaxReadReq 512 bytes
>> -               DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
>> -               LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
>> -                       ClockPM- Surprise- LLActRep+ BwNot+
>> -               LnkCtl: ASPM Disabled; Disabled- CommClk+
>> -                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>> -               LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
>> -               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
>> -                       Slot #9, PowerLimit 0.000W; Interlock- NoCompl-
>> -               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
>> -                       Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
>> -               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
>> -                       Changed: MRL- PresDet- LinkState-
>> -               DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
>> -               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
>> -               LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
>> -                        Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>> -                        Compliance De-emphasis: -6dB
>> -               LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
>> -                        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>> -       Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
>> -       Capabilities: [200 v1] Advanced Error Reporting
>> -               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> -               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> -               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>> -               CESta:  RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>> -               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> -               AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
>> -       Capabilities: [300 v1] Virtual Channel
>> -               Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
>> -               Arb:    Fixed- WRR32- WRR64- WRR128-
>> -               Ctrl:   ArbSelect=Fixed
>> -               Status: InProgress-
>> -               VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>> -                       Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
>> -                       Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>> -                       Status: NegoPending- InProgress-
>> -       Capabilities: [400 v1] Power Budgeting <?>
>> -       Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
>> +0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
>> +       !!! Unknown header type 7f
>>         Kernel driver in use: pcieport
>>
>> -0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682]
>> -       Subsystem: Apple Inc. Device [106b:00f6]
>> -       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>> -       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>> -       Latency: 0, Cache Line Size: 128 bytes
>> -       Interrupt: pin A routed to IRQ 16
>> -       Region 0: Memory at ce000000 (64-bit, prefetchable) [size=64K]
>> -       Region 2: Memory at ce010000 (64-bit, prefetchable) [size=64K]
>> -       Expansion ROM at c2000000 [disabled] [size=64K]
>> -       Capabilities: [48] Power Management version 3
>> -               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
>> -               Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>> -       Capabilities: [50] Vital Product Data
>> -               Product Name: Thunderbolt Ethernet
>> -               Read-only fields:
>> -                       [PN] Part number: BCM957762
>> -                       [EC] Engineering changes: 106679-15
>> -                       [SN] Serial number: 0123456789
>> -                       [MN] Manufacture ID: 31 34 65 34
>> -                       [RV] Reserved: checksum good, 53 byte(s) reserved
>> -               Read/write fields:
>> -                       [YA] Asset tag: XYZ01234567
>> -                       [RW] Read-write area: 107 byte(s) free
>> -               End
>> -       Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
>> -               Address: 0000000000000000  Data: 0000
>> -       Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
>> -               Vector table: BAR=2 offset=00000000
>> -               PBA: BAR=2 offset=00000120
>> -       Capabilities: [ac] Express (v2) Endpoint, MSI 00
>> -               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
>> -                       ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>> -               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>> -                       RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop-
>> -                       MaxPayload 128 bytes, MaxReadReq 4096 bytes
>> -               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
>> -               LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
>> -                       ClockPM+ Surprise- LLActRep- BwNot-
>> -               LnkCtl: ASPM Disableparentd; RCB 64 bytes Disabled- CommClk+
>> -                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>> -               LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>> -               DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
>> -               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
>> -               LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
>> -                        Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>> -                        Compliance De-emphasis: -6dB
>> -               LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
>> -                        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>> -       Capabilities: [100 v1] Advanced Error Reporting
>> -               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> -               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> -               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>> -               CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>> -               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> -               AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
>> -       Capabilities: [13c v1] Device Serial Number 00-00-a8-20-66-3c-64-4a
>> -       Capabilities: [150 v1] Power Budgeting <?>
>> -       Capabilities: [160 v1] Virtual Channel
>> -               Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
>> -               Arb:    Fixed- WRR32- WRR64- WRR128-
>> -               Ctrl:   ArbSelect=Fixed
>> -               Status: InProgress-
>> -               VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>> -                       Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
>> -                       Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
>> -                       Status: NegoPending- InProgress-
>> -       Capabilities: [1b0 v1] Latency Tolerance Reporting
>> -               Max snoop latency: 0ns
>> -               Max no snoop latency: 0ns
>> +0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682] (rev ff) (prog-if ff)
>> +       !!! Unknown header type 7f
>>         Kernel driver in use: tg3
>>         Kernel modules: tg3
>>
>>
>> So now I've got 3 confused devices that have fallen off the bus.
>>
>> If I ask the surviving parent bridge (07:03.0) to rescan for devices via
>> sysfs, this is what I get in dmesg:
>>
>> [ 3985.833685] pcieport 0000:09:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
>> [ 3985.833739] pcieport 0000:07:00.0: bridge window [io  0x1000-0x0fff] to [bus 08] add_size 1000
>> [ 3985.833743] pcieport 0000:07:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 08] add_size 200000
>> [ 3985.833767] pcieport 0000:07:04.0: bridge window [io  0x1000-0x0fff] to [bus 3a] add_size 1000
>> [ 3985.833770] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 3a] add_size 200000
>> [ 3985.833771] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff] to [bus 3a] add_size 200000
>> [ 3985.833784] pcieport 0000:07:06.0: bridge window [io  0x1000-0x0fff] to [bus 6c] add_size 1000
>> [ 3985.833786] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 6c] add_size 200000
>> [ 3985.833788] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff] to [bus 6c] add_size 200000
>> [ 3985.833792] pcieport 0000:07:00.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
>> [ 3985.833794] pcieport 0000:07:04.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
>> [ 3985.833796] pcieport 0000:07:04.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
>> [ 3985.833797] pcieport 0000:07:06.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
>> [ 3985.833799] pcieport 0000:07:06.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
>> [ 3985.833801] pcieport 0000:07:00.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
>> [ 3985.833802] pcieport 0000:07:04.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
>> [ 3985.833804] pcieport 0000:07:06.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
>> [ 3985.833810] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833811] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833814] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
>> [ 3985.833815] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
>> [ 3985.833818] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833820] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833821] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
>> [ 3985.833823] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
>> [ 3985.833825] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833827] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833829] pcieport 0000:07:00.0: BAR 13: no space for [io  size 0x1000]
>> [ 3985.833831] pcieport 0000:07:00.0: BAR 13: failed to assign [io  size 0x1000]
>> [ 3985.833832] pcieport 0000:07:04.0: BAR 13: no space for [io  size 0x1000]
>> [ 3985.833834] pcieport 0000:07:04.0: BAR 13: failed to assign [io  size 0x1000]
>> [ 3985.833835] pcieport 0000:07:06.0: BAR 13: no space for [io  size 0x1000]
>> [ 3985.833837] pcieport 0000:07:06.0: BAR 13: failed to assign [io  size 0x1000]
>> [ 3985.833840] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
>> [ 3985.833841] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
>> [ 3985.833844] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833845] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833847] pcieport 0000:07:06.0: BAR 13: no space for [io  size 0x1000]
>> [ 3985.833848] pcieport 0000:07:06.0: BAR 13: failed to assign [io  size 0x1000]
>> [ 3985.833850] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
>> [ 3985.833852] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
>> [ 3985.833854] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833856] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833858] pcieport 0000:07:04.0: BAR 13: no space for [io  size 0x1000]
>> [ 3985.833859] pcieport 0000:07:04.0: BAR 13: failed to assign [io  size 0x1000]
>> [ 3985.833862] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
>> [ 3985.833863] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
>> [ 3985.833865] pcieport 0000:07:00.0: BAR 13: no space for [io  size 0x1000]
>> [ 3985.833866] pcieport 0000:07:00.0: BAR 13: failed to assign [io  size 0x1000]
>>
>> Hrm. Any idea what to do here? I also tried some 'setpci' hackery on the
>> surviving bridge with a secondary bus reset and a link retrain. Neither
>> seems to have any effect.
>>
>> - Steven
>
> Are you booting with the device plugged in at boot? The driver does
> not yet try to discover paths which were setup by the firmeware and we
> do not do everything we should if a boot time device is unplugged.
> Could you try to replug it a second time? If that does not work try
> booting without the device attached and add the acpi_osi=Darwin to the
> kernel command line.
>
> Andreas

Good point. I tried leaving it unplugged until the system booted but
then *nothing* happened with the thunderbolt driver. No messages at
boot or anything. Unloading/reloading the driver, still no messages.
Hrmph.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Thunderbolt driver hotplug not working correctly
  2014-08-12 22:55 ` Andreas Noever
@ 2014-08-13 23:05   ` Andreas Noever
  2014-08-15 11:19   ` Steven Noonan
  1 sibling, 0 replies; 8+ messages in thread
From: Andreas Noever @ 2014-08-13 23:05 UTC (permalink / raw)
  To: Steven Noonan; +Cc: Andreas Noever, Greg KH, Linux Kernel mailing List

Hello Steven,

I think that there are two problems:
 - The Kernel does not notice that the device is gone.
 - The first hotplug operation, after removing a coldplugged device fails.

For the first one could you check whether thie pciehp (sub)-driver is loaded?
(dmesg | grep pciehp should show something, the config option is
CONFIG_HOTPLUG_PCI_PCIE).

I was able reproduce the second problem on my machine. Could you test whether
this patch fixes the problem?

Thanks,
Andreas

---
 drivers/thunderbolt/path.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/drivers/thunderbolt/path.c b/drivers/thunderbolt/path.c
index 8fcf8a7..9562cd0 100644
--- a/drivers/thunderbolt/path.c
+++ b/drivers/thunderbolt/path.c
@@ -150,7 +150,26 @@ int tb_path_activate(struct tb_path *path)
 
 	/* Activate hops. */
 	for (i = path->path_length - 1; i >= 0; i--) {
-		struct tb_regs_hop hop;
+		struct tb_regs_hop hop = { 0 };
+
+		/*
+		 * We do (currently) not tear down paths setup by the firmeware.
+		 * If a firmware device is unplugged and plugged in again then
+		 * it can happen that we reuse some of the hops from the (now
+		 * defunct) firmeware path. This causes the hotplug operation to
+		 * fail (the pci device does not show up). Clearing the hop
+		 * before overwriting it fixes the problem.
+		 *
+		 * Should be removed once we discover and tear down firmeware
+		 * paths.
+		 */
+		res = tb_port_write(path->hops[i].in_port, &hop, TB_CFG_HOPS,
+				    2 * path->hops[i].in_hop_index, 2);
+		if (res) {
+			__tb_path_deactivate_hops(path, i);
+			__tb_path_deallocate_nfc(path, 0);
+			goto err;
+		}
 
 		/* dword 0 */
 		hop.next_hop = path->hops[i].next_hop_index;
-- 
2.0.4


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: Thunderbolt driver hotplug not working correctly
  2014-08-12  9:01 Steven Noonan
  2014-08-12  9:05 ` Greg KH
@ 2014-08-12 22:55 ` Andreas Noever
  2014-08-13 23:05   ` Andreas Noever
  2014-08-15 11:19   ` Steven Noonan
  1 sibling, 2 replies; 8+ messages in thread
From: Andreas Noever @ 2014-08-12 22:55 UTC (permalink / raw)
  To: Steven Noonan; +Cc: Linux Kernel mailing List

On Tue, Aug 12, 2014 at 11:01 AM, Steven Noonan <steven@uplinklabs.net> wrote:
> Hello Andreas,
>
> I'm trying out the new Thunderbolt driver in Linus' tree and I'm
> noticing device hotplug isn't quite working correctly.
>
> I've got a Haswell 2013 MacBook Pro Retina and the Apple-vended
> Thunderbolt Ethernet adapter, which uses the tg3 driver.
>
> Here's what dmesg says when I unplug:
>
> [ 1866.359118] thunderbolt 0000:08:00.0: resetting error on 0:3.
> [ 1866.359150] thunderbolt 0000:08:00.0: 0:3: unplugged
>
> When I re-plug the device:
>
> [ 1872.481675] thunderbolt 0000:08:00.0: resetting error on 0:3.
> [ 1872.481695] thunderbolt 0000:08:00.0: 0:3: hotplug: scanning
> [ 1872.481764] thunderbolt 0000:08:00.0: 0:3: is connected, link is up (state: 2)
> [ 1872.482078] thunderbolt 0000:08:00.0: initializing Switch at 0x3 (depth: 1, up port: 1)
> [ 1872.482084] thunderbolt 0000:08:00.0: old switch config:
> [ 1872.482089] thunderbolt 0000:08:00.0:  Switch: 8086:1549 (Revision: 0, TB Version: 1)
> [ 1872.482092] thunderbolt 0000:08:00.0:   Max Port Number: 2
> [ 1872.482094] thunderbolt 0000:08:00.0:   Config:
> [ 1872.482098] thunderbolt 0000:08:00.0:    Upstream Port Number: 0 Depth: 0 Route String: 0x0 Enabled: 0, PlugEventsDelay: 10ms
> [ 1872.482102] thunderbolt 0000:08:00.0:    unknown1: 0x0 unknown4: 0x0
> [ 1872.496213] thunderbolt 0000:08:00.0: 3: reading drom (length: 0x7b)
> [ 1872.821969] thunderbolt 0000:08:00.0: 3: uid: 0x10002014eed70
> [ 1872.822067] thunderbolt 0000:08:00.0:  Port 0: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
> [ 1872.822069] thunderbolt 0000:08:00.0:   Max hop id (in/out): 7/7
> [ 1872.822071] thunderbolt 0000:08:00.0:   Max counters: 8
> [ 1872.822072] thunderbolt 0000:08:00.0:   NFC Credits: 0x700000
> [ 1872.822566] thunderbolt 0000:08:00.0:  Port 1: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
> [ 1872.822567] thunderbolt 0000:08:00.0:   Max hop id (in/out): 8/8
> [ 1872.822569] thunderbolt 0000:08:00.0:   Max counters: 4
> [ 1872.822570] thunderbolt 0000:08:00.0:   NFC Credits: 0x3c00000
> [ 1872.822666] thunderbolt 0000:08:00.0:  Port 2: 8086:1549 (Revision: 0, TB Version: 1, Type: PCIe (0x100102))
> [ 1872.822667] thunderbolt 0000:08:00.0:   Max hop id (in/out): 8/8
> [ 1872.822668] thunderbolt 0000:08:00.0:   Max counters: 2
> [ 1872.822670] thunderbolt 0000:08:00.0:   NFC Credits: 0x700000
> [ 1872.822982] thunderbolt 0000:08:00.0: 3: hotplug: activating pcie devices
> [ 1872.823576] thunderbolt 0000:08:00.0: 0:6 <-> 3:2 (PCI): activating
> [ 1872.823580] thunderbolt 0000:08:00.0: activating path from 0:6 to 3:2
> [ 1872.823582] thunderbolt 0000:08:00.0: 3:1: Writing hop 1, index 8
> [ 1872.823583] thunderbolt 0000:08:00.0: 3:1:  Hop through port 2 to hop 8 (enabled)
> [ 1872.823585] thunderbolt 0000:08:00.0: 3:1:   Weight: 1 Priority: 3 Credits: 16 Drop: 0
> [ 1872.823587] thunderbolt 0000:08:00.0: 3:1:    Counter enabled: 0 Counter index: 2047
> [ 1872.823589] thunderbolt 0000:08:00.0: 3:1:   Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
> [ 1872.823591] thunderbolt 0000:08:00.0: 3:1:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
> [ 1872.823664] thunderbolt 0000:08:00.0: 0:6: Writing hop 0, index 8
> [ 1872.823666] thunderbolt 0000:08:00.0: 0:6:  Hop through port 3 to hop 8 (enabled)
> [ 1872.823667] thunderbolt 0000:08:00.0: 0:6:   Weight: 1 Priority: 3 Credits: 7 Drop: 0
> [ 1872.823669] thunderbolt 0000:08:00.0: 0:6:    Counter enabled: 0 Counter index: 2047
> [ 1872.823670] thunderbolt 0000:08:00.0: 0:6:   Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
> [ 1872.823672] thunderbolt 0000:08:00.0: 0:6:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
> [ 1872.823764] thunderbolt 0000:08:00.0: path activation complete
> [ 1872.823765] thunderbolt 0000:08:00.0: activating path from 3:2 to 0:6
> [ 1872.823767] thunderbolt 0000:08:00.0: 0:3: Writing hop 1, index 8
> [ 1872.823768] thunderbolt 0000:08:00.0: 0:3:  Hop through port 6 to hop 8 (enabled)
> [ 1872.823770] thunderbolt 0000:08:00.0: 0:3:   Weight: 1 Priority: 3 Credits: 16 Drop: 0
> [ 1872.823771] thunderbolt 0000:08:00.0: 0:3:    Counter enabled: 0 Counter index: 2047
> [ 1872.823773] thunderbolt 0000:08:00.0: 0:3:   Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
> [ 1872.823774] thunderbolt 0000:08:00.0: 0:3:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
> [ 1872.823864] thunderbolt 0000:08:00.0: 3:2: Writing hop 0, index 8
> [ 1872.823865] thunderbolt 0000:08:00.0: 3:2:  Hop through port 1 to hop 8 (enabled)
> [ 1872.823867] thunderbolt 0000:08:00.0: 3:2:   Weight: 1 Priority: 3 Credits: 7 Drop: 0
> [ 1872.823868] thunderbolt 0000:08:00.0: 3:2:    Counter enabled: 0 Counter index: 2047
> [ 1872.823870] thunderbolt 0000:08:00.0: 3:2:   Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
> [ 1872.823871] thunderbolt 0000:08:00.0: 3:2:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
> [ 1872.823963] thunderbolt 0000:08:00.0: path activation complete
>
>
> And the tg3 driver didn't notice anything happened at all during that
> process, so I did an rmmod/modprobe:
>
> [ 1894.412903] tg3 0000:0b:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
> [ 1894.795333] cfg80211: Calling CRDA to update world regulatory domain
> [ 1895.295757] tg3.c:v3.137 (May 11, 2014)
> [ 1895.315622] tg3 0000:0b:00.0: Refused to change power state, currently in D3
> [ 1895.463145] tg3 0000:0b:00.0: phy probe failed, err -19
> [ 1895.463485] tg3 0000:0b:00.0: Problem fetching invariants of chip, aborting
>
> Uh oh.
>
> Here's the diff in 'lspci -nnvv' between unplug/replug:
>
> --- before      2014-08-12 01:46:09.313859543 -0700
> +++ after       2014-08-12 01:46:30.493241976 -0700
> @@ -814,89 +814,89 @@
>  07:03.0 PCI bridge [0604]: Intel Corporation Device [8086:156d] (prog-if 00 [Normal decode])
>         Physical Slot: 2
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 128 bytes
>         Bus: primary=07, secondary=09, subordinate=39, sec-latency=0
>         I/O behind bridge: 00003000-00003fff
>         Memory behind bridge: c2000000-c5ffffff
>         Prefetchable memory behind bridge: 00000000ce000000-00000000d20fffff
>         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
>         BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>         Capabilities: [80] Power Management version 3
>                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
>         Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
>                 Address: 00000000fee00378  Data: 0000
>         Capabilities: [ac] Subsystem: Device [2222:1111]
>         Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
>                 DevCap: MaxPayload 128 bytes, PhantFunc 0
>                         ExtTag+ RBE+
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>                         RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
>                 LnkCap: Port #3, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
>                         ClockPM- Surprise- LLActRep+ BwNot+
>                 LnkCtl: ASPM Disabled; Disabled- CommClk+
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> -               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
> +               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt-
>                 SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
>                         Slot #3, PowerLimit 0.000W; Interlock- NoCompl+
>                 SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
>                         Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
> -               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
> -                       Changed: MRL- PresDet- LinkState-
> +               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
> +                       Changed: MRL- PresDet+ LinkState+
>                 DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd-
> -               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
> +               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
>                 LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
>                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>                          Compliance De-emphasis: -6dB
>                 LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
>                          EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>         Capabilities: [100 v1] Device Serial Number 2e-dc-31-20-d3-f1-0c-00
>         Capabilities: [200 v1] Advanced Error Reporting
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>                 AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
>         Capabilities: [300 v1] Virtual Channel
>                 Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
>                 Arb:    Fixed- WRR32- WRR64- WRR128-
>                 Ctrl:   ArbSelect=Fixed
>                 Status: InProgress-
>                 VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
>                         Arb:    Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
>                         Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> -                       Status: NegoPending- InProgress-
> +                       Status: NegoPending+ InProgress-
>         Capabilities: [400 v1] Power Budgeting <?>
>         Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=050 <?>
>         Kernel driver in use: pcieport
>
> @@ -1122,193 +1122,16 @@
>         Kernel driver in use: thunderbolt
>         Kernel modules: thunderbolt
>
> -09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
> -       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> -       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> -       Latency: 0, Cache Line Size: 128 bytes
> -       Bus: primary=09, secondary=0a, subordinate=0b, sec-latency=0
> -       I/O behind bridge: 00003000-00003fff
> -       Memory behind bridge: c2000000-c24fffff
> -       Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
> -       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> -       BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> -               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> -       Capabilities: [80] Power Management version 3
> -               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
> -               Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
> -       Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
> -               Address: 00000000fee003f8  Data: 0000
> -       Capabilities: [c0] Express (v2) Upstream Port, MSI 00
> -               DevCap: MaxPayload 128 bytes, PhantFunc 0
> -                       ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
> -               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> -                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
> -                       MaxPayload 128 bytes, MaxReadReq 512 bytes
> -               DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
> -               LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
> -                       ClockPM+ Surprise- LLActRep- BwNot-
> -               LnkCtl: ASPM Disabled; Disabled- CommClk+
> -                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> -               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> -               DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
> -               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
> -               LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> -                        Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> -                        Compliance De-emphasis: -6dB
> -               LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
> -                        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> -       Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
> -       Capabilities: [200 v1] Advanced Error Reporting
> -               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> -               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> -               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> -               CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> -               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> -               AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
> -       Capabilities: [300 v1] Virtual Channel
> -               Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
> -               Arb:    Fixed- WRR32- WRR64- WRR128-
> -               Ctrl:   ArbSelect=Fixed
> -               Status: InProgress-
> -               VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> -                       Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
> -                       Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> -                       Status: NegoPending- InProgress-
> -       Capabilities: [400 v1] Power Budgeting <?>
> -       Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
> +09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
> +       !!! Unknown header type 7f
>         Kernel driver in use: pcieport
>
> -0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
> -       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> -       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> -       Latency: 0, Cache Line Size: 128 bytes
> -       Bus: primary=0a, secondary=0b, subordinate=0b, sec-latency=0
> -       I/O behind bridge: 00003000-00003fff
> -       Memory behind bridge: c2000000-c24fffff
> -       Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
> -       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> -       BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> -               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> -       Capabilities: [80] Power Management version 3
> -               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
> -               Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
> -       Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
> -               Address: 00000000fee00418  Data: 0000
> -       Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
> -               DevCap: MaxPayload 128 bytes, PhantFunc 0
> -                       ExtTag+ RBE+
> -               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> -                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
> -                       MaxPayload 128 bytes, MaxReadReq 512 bytes
> -               DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
> -               LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
> -                       ClockPM- Surprise- LLActRep+ BwNot+
> -               LnkCtl: ASPM Disabled; Disabled- CommClk+
> -                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> -               LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
> -               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
> -                       Slot #9, PowerLimit 0.000W; Interlock- NoCompl-
> -               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
> -                       Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
> -               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
> -                       Changed: MRL- PresDet- LinkState-
> -               DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
> -               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
> -               LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
> -                        Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> -                        Compliance De-emphasis: -6dB
> -               LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
> -                        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> -       Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
> -       Capabilities: [200 v1] Advanced Error Reporting
> -               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> -               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> -               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> -               CESta:  RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
> -               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> -               AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
> -       Capabilities: [300 v1] Virtual Channel
> -               Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
> -               Arb:    Fixed- WRR32- WRR64- WRR128-
> -               Ctrl:   ArbSelect=Fixed
> -               Status: InProgress-
> -               VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> -                       Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
> -                       Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> -                       Status: NegoPending- InProgress-
> -       Capabilities: [400 v1] Power Budgeting <?>
> -       Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
> +0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
> +       !!! Unknown header type 7f
>         Kernel driver in use: pcieport
>
> -0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682]
> -       Subsystem: Apple Inc. Device [106b:00f6]
> -       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> -       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> -       Latency: 0, Cache Line Size: 128 bytes
> -       Interrupt: pin A routed to IRQ 16
> -       Region 0: Memory at ce000000 (64-bit, prefetchable) [size=64K]
> -       Region 2: Memory at ce010000 (64-bit, prefetchable) [size=64K]
> -       Expansion ROM at c2000000 [disabled] [size=64K]
> -       Capabilities: [48] Power Management version 3
> -               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
> -               Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
> -       Capabilities: [50] Vital Product Data
> -               Product Name: Thunderbolt Ethernet
> -               Read-only fields:
> -                       [PN] Part number: BCM957762
> -                       [EC] Engineering changes: 106679-15
> -                       [SN] Serial number: 0123456789
> -                       [MN] Manufacture ID: 31 34 65 34
> -                       [RV] Reserved: checksum good, 53 byte(s) reserved
> -               Read/write fields:
> -                       [YA] Asset tag: XYZ01234567
> -                       [RW] Read-write area: 107 byte(s) free
> -               End
> -       Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
> -               Address: 0000000000000000  Data: 0000
> -       Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
> -               Vector table: BAR=2 offset=00000000
> -               PBA: BAR=2 offset=00000120
> -       Capabilities: [ac] Express (v2) Endpoint, MSI 00
> -               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
> -                       ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> -               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> -                       RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop-
> -                       MaxPayload 128 bytes, MaxReadReq 4096 bytes
> -               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
> -               LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
> -                       ClockPM+ Surprise- LLActRep- BwNot-
> -               LnkCtl: ASPM Disableparentd; RCB 64 bytes Disabled- CommClk+
> -                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> -               LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> -               DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
> -               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
> -               LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> -                        Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> -                        Compliance De-emphasis: -6dB
> -               LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
> -                        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> -       Capabilities: [100 v1] Advanced Error Reporting
> -               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> -               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> -               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
> -               CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
> -               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
> -               AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
> -       Capabilities: [13c v1] Device Serial Number 00-00-a8-20-66-3c-64-4a
> -       Capabilities: [150 v1] Power Budgeting <?>
> -       Capabilities: [160 v1] Virtual Channel
> -               Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
> -               Arb:    Fixed- WRR32- WRR64- WRR128-
> -               Ctrl:   ArbSelect=Fixed
> -               Status: InProgress-
> -               VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> -                       Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
> -                       Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> -                       Status: NegoPending- InProgress-
> -       Capabilities: [1b0 v1] Latency Tolerance Reporting
> -               Max snoop latency: 0ns
> -               Max no snoop latency: 0ns
> +0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682] (rev ff) (prog-if ff)
> +       !!! Unknown header type 7f
>         Kernel driver in use: tg3
>         Kernel modules: tg3
>
>
> So now I've got 3 confused devices that have fallen off the bus.
>
> If I ask the surviving parent bridge (07:03.0) to rescan for devices via
> sysfs, this is what I get in dmesg:
>
> [ 3985.833685] pcieport 0000:09:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
> [ 3985.833739] pcieport 0000:07:00.0: bridge window [io  0x1000-0x0fff] to [bus 08] add_size 1000
> [ 3985.833743] pcieport 0000:07:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 08] add_size 200000
> [ 3985.833767] pcieport 0000:07:04.0: bridge window [io  0x1000-0x0fff] to [bus 3a] add_size 1000
> [ 3985.833770] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 3a] add_size 200000
> [ 3985.833771] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff] to [bus 3a] add_size 200000
> [ 3985.833784] pcieport 0000:07:06.0: bridge window [io  0x1000-0x0fff] to [bus 6c] add_size 1000
> [ 3985.833786] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 6c] add_size 200000
> [ 3985.833788] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff] to [bus 6c] add_size 200000
> [ 3985.833792] pcieport 0000:07:00.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
> [ 3985.833794] pcieport 0000:07:04.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
> [ 3985.833796] pcieport 0000:07:04.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
> [ 3985.833797] pcieport 0000:07:06.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
> [ 3985.833799] pcieport 0000:07:06.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
> [ 3985.833801] pcieport 0000:07:00.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
> [ 3985.833802] pcieport 0000:07:04.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
> [ 3985.833804] pcieport 0000:07:06.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
> [ 3985.833810] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833811] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833814] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
> [ 3985.833815] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
> [ 3985.833818] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833820] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833821] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
> [ 3985.833823] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
> [ 3985.833825] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833827] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833829] pcieport 0000:07:00.0: BAR 13: no space for [io  size 0x1000]
> [ 3985.833831] pcieport 0000:07:00.0: BAR 13: failed to assign [io  size 0x1000]
> [ 3985.833832] pcieport 0000:07:04.0: BAR 13: no space for [io  size 0x1000]
> [ 3985.833834] pcieport 0000:07:04.0: BAR 13: failed to assign [io  size 0x1000]
> [ 3985.833835] pcieport 0000:07:06.0: BAR 13: no space for [io  size 0x1000]
> [ 3985.833837] pcieport 0000:07:06.0: BAR 13: failed to assign [io  size 0x1000]
> [ 3985.833840] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
> [ 3985.833841] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
> [ 3985.833844] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833845] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833847] pcieport 0000:07:06.0: BAR 13: no space for [io  size 0x1000]
> [ 3985.833848] pcieport 0000:07:06.0: BAR 13: failed to assign [io  size 0x1000]
> [ 3985.833850] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
> [ 3985.833852] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
> [ 3985.833854] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833856] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833858] pcieport 0000:07:04.0: BAR 13: no space for [io  size 0x1000]
> [ 3985.833859] pcieport 0000:07:04.0: BAR 13: failed to assign [io  size 0x1000]
> [ 3985.833862] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
> [ 3985.833863] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
> [ 3985.833865] pcieport 0000:07:00.0: BAR 13: no space for [io  size 0x1000]
> [ 3985.833866] pcieport 0000:07:00.0: BAR 13: failed to assign [io  size 0x1000]
>
> Hrm. Any idea what to do here? I also tried some 'setpci' hackery on the
> surviving bridge with a secondary bus reset and a link retrain. Neither
> seems to have any effect.
>
> - Steven

Are you booting with the device plugged in at boot? The driver does
not yet try to discover paths which were setup by the firmeware and we
do not do everything we should if a boot time device is unplugged.
Could you try to replug it a second time? If that does not work try
booting without the device attached and add the acpi_osi=Darwin to the
kernel command line.

Andreas

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Thunderbolt driver hotplug not working correctly
  2014-08-12  9:01 Steven Noonan
@ 2014-08-12  9:05 ` Greg KH
  2014-08-12 22:55 ` Andreas Noever
  1 sibling, 0 replies; 8+ messages in thread
From: Greg KH @ 2014-08-12  9:05 UTC (permalink / raw)
  To: Steven Noonan, Matthew Garrett; +Cc: Linux Kernel mailing List, Andreas Noever

On Tue, Aug 12, 2014 at 02:01:38AM -0700, Steven Noonan wrote:
> Hello Andreas,
> 
> I'm trying out the new Thunderbolt driver in Linus' tree and I'm
> noticing device hotplug isn't quite working correctly.
> 
> I've got a Haswell 2013 MacBook Pro Retina and the Apple-vended
> Thunderbolt Ethernet adapter, which uses the tg3 driver.

I don't think that the relevant ACPI changes have landed in Linus's tree
yet to make this all work.

Matthew, have your changes be merged yet?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Thunderbolt driver hotplug not working correctly
@ 2014-08-12  9:01 Steven Noonan
  2014-08-12  9:05 ` Greg KH
  2014-08-12 22:55 ` Andreas Noever
  0 siblings, 2 replies; 8+ messages in thread
From: Steven Noonan @ 2014-08-12  9:01 UTC (permalink / raw)
  To: Linux Kernel mailing List, Andreas Noever

Hello Andreas,

I'm trying out the new Thunderbolt driver in Linus' tree and I'm
noticing device hotplug isn't quite working correctly.

I've got a Haswell 2013 MacBook Pro Retina and the Apple-vended
Thunderbolt Ethernet adapter, which uses the tg3 driver.

Here's what dmesg says when I unplug:

[ 1866.359118] thunderbolt 0000:08:00.0: resetting error on 0:3.
[ 1866.359150] thunderbolt 0000:08:00.0: 0:3: unplugged

When I re-plug the device:

[ 1872.481675] thunderbolt 0000:08:00.0: resetting error on 0:3.
[ 1872.481695] thunderbolt 0000:08:00.0: 0:3: hotplug: scanning
[ 1872.481764] thunderbolt 0000:08:00.0: 0:3: is connected, link is up (state: 2)
[ 1872.482078] thunderbolt 0000:08:00.0: initializing Switch at 0x3 (depth: 1, up port: 1)
[ 1872.482084] thunderbolt 0000:08:00.0: old switch config:
[ 1872.482089] thunderbolt 0000:08:00.0:  Switch: 8086:1549 (Revision: 0, TB Version: 1)
[ 1872.482092] thunderbolt 0000:08:00.0:   Max Port Number: 2
[ 1872.482094] thunderbolt 0000:08:00.0:   Config:
[ 1872.482098] thunderbolt 0000:08:00.0:    Upstream Port Number: 0 Depth: 0 Route String: 0x0 Enabled: 0, PlugEventsDelay: 10ms
[ 1872.482102] thunderbolt 0000:08:00.0:    unknown1: 0x0 unknown4: 0x0
[ 1872.496213] thunderbolt 0000:08:00.0: 3: reading drom (length: 0x7b)
[ 1872.821969] thunderbolt 0000:08:00.0: 3: uid: 0x10002014eed70
[ 1872.822067] thunderbolt 0000:08:00.0:  Port 0: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
[ 1872.822069] thunderbolt 0000:08:00.0:   Max hop id (in/out): 7/7
[ 1872.822071] thunderbolt 0000:08:00.0:   Max counters: 8
[ 1872.822072] thunderbolt 0000:08:00.0:   NFC Credits: 0x700000
[ 1872.822566] thunderbolt 0000:08:00.0:  Port 1: 8086:1549 (Revision: 0, TB Version: 1, Type: Port (0x1))
[ 1872.822567] thunderbolt 0000:08:00.0:   Max hop id (in/out): 8/8
[ 1872.822569] thunderbolt 0000:08:00.0:   Max counters: 4
[ 1872.822570] thunderbolt 0000:08:00.0:   NFC Credits: 0x3c00000
[ 1872.822666] thunderbolt 0000:08:00.0:  Port 2: 8086:1549 (Revision: 0, TB Version: 1, Type: PCIe (0x100102))
[ 1872.822667] thunderbolt 0000:08:00.0:   Max hop id (in/out): 8/8
[ 1872.822668] thunderbolt 0000:08:00.0:   Max counters: 2
[ 1872.822670] thunderbolt 0000:08:00.0:   NFC Credits: 0x700000
[ 1872.822982] thunderbolt 0000:08:00.0: 3: hotplug: activating pcie devices
[ 1872.823576] thunderbolt 0000:08:00.0: 0:6 <-> 3:2 (PCI): activating
[ 1872.823580] thunderbolt 0000:08:00.0: activating path from 0:6 to 3:2
[ 1872.823582] thunderbolt 0000:08:00.0: 3:1: Writing hop 1, index 8
[ 1872.823583] thunderbolt 0000:08:00.0: 3:1:  Hop through port 2 to hop 8 (enabled)
[ 1872.823585] thunderbolt 0000:08:00.0: 3:1:   Weight: 1 Priority: 3 Credits: 16 Drop: 0
[ 1872.823587] thunderbolt 0000:08:00.0: 3:1:    Counter enabled: 0 Counter index: 2047
[ 1872.823589] thunderbolt 0000:08:00.0: 3:1:   Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
[ 1872.823591] thunderbolt 0000:08:00.0: 3:1:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
[ 1872.823664] thunderbolt 0000:08:00.0: 0:6: Writing hop 0, index 8
[ 1872.823666] thunderbolt 0000:08:00.0: 0:6:  Hop through port 3 to hop 8 (enabled)
[ 1872.823667] thunderbolt 0000:08:00.0: 0:6:   Weight: 1 Priority: 3 Credits: 7 Drop: 0
[ 1872.823669] thunderbolt 0000:08:00.0: 0:6:    Counter enabled: 0 Counter index: 2047
[ 1872.823670] thunderbolt 0000:08:00.0: 0:6:   Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
[ 1872.823672] thunderbolt 0000:08:00.0: 0:6:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
[ 1872.823764] thunderbolt 0000:08:00.0: path activation complete
[ 1872.823765] thunderbolt 0000:08:00.0: activating path from 3:2 to 0:6
[ 1872.823767] thunderbolt 0000:08:00.0: 0:3: Writing hop 1, index 8
[ 1872.823768] thunderbolt 0000:08:00.0: 0:3:  Hop through port 6 to hop 8 (enabled)
[ 1872.823770] thunderbolt 0000:08:00.0: 0:3:   Weight: 1 Priority: 3 Credits: 16 Drop: 0
[ 1872.823771] thunderbolt 0000:08:00.0: 0:3:    Counter enabled: 0 Counter index: 2047
[ 1872.823773] thunderbolt 0000:08:00.0: 0:3:   Flow Control (In/Eg): 1/0 Shared Buffer (In/Eg): 0/0
[ 1872.823774] thunderbolt 0000:08:00.0: 0:3:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
[ 1872.823864] thunderbolt 0000:08:00.0: 3:2: Writing hop 0, index 8
[ 1872.823865] thunderbolt 0000:08:00.0: 3:2:  Hop through port 1 to hop 8 (enabled)
[ 1872.823867] thunderbolt 0000:08:00.0: 3:2:   Weight: 1 Priority: 3 Credits: 7 Drop: 0
[ 1872.823868] thunderbolt 0000:08:00.0: 3:2:    Counter enabled: 0 Counter index: 2047
[ 1872.823870] thunderbolt 0000:08:00.0: 3:2:   Flow Control (In/Eg): 1/1 Shared Buffer (In/Eg): 0/0
[ 1872.823871] thunderbolt 0000:08:00.0: 3:2:   Unknown1: 0x0 Unknown2: 0x0 Unknown3: 0x0
[ 1872.823963] thunderbolt 0000:08:00.0: path activation complete


And the tg3 driver didn't notice anything happened at all during that
process, so I did an rmmod/modprobe:

[ 1894.412903] tg3 0000:0b:00.0: tg3_abort_hw timed out, TX_MODE_ENABLE will not clear MAC_TX_MODE=ffffffff
[ 1894.795333] cfg80211: Calling CRDA to update world regulatory domain
[ 1895.295757] tg3.c:v3.137 (May 11, 2014)
[ 1895.315622] tg3 0000:0b:00.0: Refused to change power state, currently in D3
[ 1895.463145] tg3 0000:0b:00.0: phy probe failed, err -19
[ 1895.463485] tg3 0000:0b:00.0: Problem fetching invariants of chip, aborting

Uh oh.

Here's the diff in 'lspci -nnvv' between unplug/replug:

--- before      2014-08-12 01:46:09.313859543 -0700
+++ after       2014-08-12 01:46:30.493241976 -0700
@@ -814,89 +814,89 @@
 07:03.0 PCI bridge [0604]: Intel Corporation Device [8086:156d] (prog-if 00 [Normal decode])
        Physical Slot: 2
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 128 bytes
        Bus: primary=07, secondary=09, subordinate=39, sec-latency=0
        I/O behind bridge: 00003000-00003fff
        Memory behind bridge: c2000000-c5ffffff
        Prefetchable memory behind bridge: 00000000ce000000-00000000d20fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee00378  Data: 0000
        Capabilities: [ac] Subsystem: Device [2222:1111]
        Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #3, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot+
                LnkCtl: ASPM Disabled; Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
-               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
+               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
                        Slot #3, PowerLimit 0.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
-               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
-                       Changed: MRL- PresDet- LinkState-
+               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
+                       Changed: MRL- PresDet+ LinkState+
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd-
-               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
+               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 2e-dc-31-20-d3-f1-0c-00
        Capabilities: [200 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [300 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
-                       Status: NegoPending- InProgress-
+                       Status: NegoPending+ InProgress-
        Capabilities: [400 v1] Power Budgeting <?>
        Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=050 <?>
        Kernel driver in use: pcieport

@@ -1122,193 +1122,16 @@
        Kernel driver in use: thunderbolt
        Kernel modules: thunderbolt
 
-09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
-       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
-       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
-       Latency: 0, Cache Line Size: 128 bytes
-       Bus: primary=09, secondary=0a, subordinate=0b, sec-latency=0
-       I/O behind bridge: 00003000-00003fff
-       Memory behind bridge: c2000000-c24fffff
-       Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
-       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
-       BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
-               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
-       Capabilities: [80] Power Management version 3
-               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
-               Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
-       Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
-               Address: 00000000fee003f8  Data: 0000
-       Capabilities: [c0] Express (v2) Upstream Port, MSI 00
-               DevCap: MaxPayload 128 bytes, PhantFunc 0
-                       ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W
-               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
-                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
-                       MaxPayload 128 bytes, MaxReadReq 512 bytes
-               DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
-               LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
-                       ClockPM+ Surprise- LLActRep- BwNot-
-               LnkCtl: ASPM Disabled; Disabled- CommClk+
-                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
-               LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
-               DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
-               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
-               LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
-                        Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
-                        Compliance De-emphasis: -6dB
-               LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
-                        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
-       Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
-       Capabilities: [200 v1] Advanced Error Reporting
-               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
-               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
-               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
-               CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
-               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
-               AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
-       Capabilities: [300 v1] Virtual Channel
-               Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
-               Arb:    Fixed- WRR32- WRR64- WRR128-
-               Ctrl:   ArbSelect=Fixed
-               Status: InProgress-
-               VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
-                       Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
-                       Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
-                       Status: NegoPending- InProgress-
-       Capabilities: [400 v1] Power Budgeting <?>
-       Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
+09:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
+       !!! Unknown header type 7f
        Kernel driver in use: pcieport
 
-0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (prog-if 00 [Normal decode])
-       Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
-       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
-       Latency: 0, Cache Line Size: 128 bytes
-       Bus: primary=0a, secondary=0b, subordinate=0b, sec-latency=0
-       I/O behind bridge: 00003000-00003fff
-       Memory behind bridge: c2000000-c24fffff
-       Prefetchable memory behind bridge: 00000000ce000000-00000000ce0fffff
-       Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
-       BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
-               PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
-       Capabilities: [80] Power Management version 3
-               Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
-               Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
-       Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
-               Address: 00000000fee00418  Data: 0000
-       Capabilities: [c0] Express (v2) Downstream Port (Slot+), MSI 00
-               DevCap: MaxPayload 128 bytes, PhantFunc 0
-                       ExtTag+ RBE+
-               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
-                       RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
-                       MaxPayload 128 bytes, MaxReadReq 512 bytes
-               DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
-               LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
-                       ClockPM- Surprise- LLActRep+ BwNot+
-               LnkCtl: ASPM Disabled; Disabled- CommClk+
-                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
-               LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
-               SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+
-                       Slot #9, PowerLimit 0.000W; Interlock- NoCompl-
-               SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
-                       Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
-               SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
-                       Changed: MRL- PresDet- LinkState-
-               DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd-
-               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
-               LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
-                        Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
-                        Compliance De-emphasis: -6dB
-               LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
-                        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
-       Capabilities: [100 v1] Device Serial Number 01-00-00-00-00-c9-a0-00
-       Capabilities: [200 v1] Advanced Error Reporting
-               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
-               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
-               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
-               CESta:  RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
-               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
-               AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
-       Capabilities: [300 v1] Virtual Channel
-               Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
-               Arb:    Fixed- WRR32- WRR64- WRR128-
-               Ctrl:   ArbSelect=Fixed
-               Status: InProgress-
-               VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
-                       Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
-                       Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
-                       Status: NegoPending- InProgress-
-       Capabilities: [400 v1] Power Budgeting <?>
-       Capabilities: [500 v1] Vendor Specific Information: ID=1234 Rev=1 Len=01c <?>
+0a:00.0 PCI bridge [0604]: Intel Corporation DSL3510 Thunderbolt Controller [Cactus Ridge] [8086:1549] (rev ff) (prog-if ff)
+       !!! Unknown header type 7f
        Kernel driver in use: pcieport
 
-0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682]
-       Subsystem: Apple Inc. Device [106b:00f6]
-       Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
-       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
-       Latency: 0, Cache Line Size: 128 bytes
-       Interrupt: pin A routed to IRQ 16
-       Region 0: Memory at ce000000 (64-bit, prefetchable) [size=64K]
-       Region 2: Memory at ce010000 (64-bit, prefetchable) [size=64K]
-       Expansion ROM at c2000000 [disabled] [size=64K]
-       Capabilities: [48] Power Management version 3
-               Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
-               Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
-       Capabilities: [50] Vital Product Data
-               Product Name: Thunderbolt Ethernet
-               Read-only fields:
-                       [PN] Part number: BCM957762
-                       [EC] Engineering changes: 106679-15
-                       [SN] Serial number: 0123456789
-                       [MN] Manufacture ID: 31 34 65 34
-                       [RV] Reserved: checksum good, 53 byte(s) reserved
-               Read/write fields:
-                       [YA] Asset tag: XYZ01234567
-                       [RW] Read-write area: 107 byte(s) free
-               End
-       Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
-               Address: 0000000000000000  Data: 0000
-       Capabilities: [a0] MSI-X: Enable+ Count=6 Masked-
-               Vector table: BAR=2 offset=00000000
-               PBA: BAR=2 offset=00000120
-       Capabilities: [ac] Express (v2) Endpoint, MSI 00
-               DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
-                       ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
-               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
-                       RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop-
-                       MaxPayload 128 bytes, MaxReadReq 4096 bytes
-               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
-               LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
-                       ClockPM+ Surprise- LLActRep- BwNot-
-               LnkCtl: ASPM Disableparentd; RCB 64 bytes Disabled- CommClk+
-                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
-               LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
-               DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
-               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
-               LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
-                        Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
-                        Compliance De-emphasis: -6dB
-               LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
-                        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
-       Capabilities: [100 v1] Advanced Error Reporting
-               UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
-               UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
-               UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
-               CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
-               CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
-               AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
-       Capabilities: [13c v1] Device Serial Number 00-00-a8-20-66-3c-64-4a
-       Capabilities: [150 v1] Power Budgeting <?>
-       Capabilities: [160 v1] Virtual Channel
-               Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
-               Arb:    Fixed- WRR32- WRR64- WRR128-
-               Ctrl:   ArbSelect=Fixed
-               Status: InProgress-
-               VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
-                       Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
-                       Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
-                       Status: NegoPending- InProgress-
-       Capabilities: [1b0 v1] Latency Tolerance Reporting
-               Max snoop latency: 0ns
-               Max no snoop latency: 0ns
+0b:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM57762 Gigabit Ethernet PCIe [14e4:1682] (rev ff) (prog-if ff)
+       !!! Unknown header type 7f
        Kernel driver in use: tg3
        Kernel modules: tg3


So now I've got 3 confused devices that have fallen off the bus.

If I ask the surviving parent bridge (07:03.0) to rescan for devices via
sysfs, this is what I get in dmesg:

[ 3985.833685] pcieport 0000:09:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
[ 3985.833739] pcieport 0000:07:00.0: bridge window [io  0x1000-0x0fff] to [bus 08] add_size 1000
[ 3985.833743] pcieport 0000:07:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 08] add_size 200000
[ 3985.833767] pcieport 0000:07:04.0: bridge window [io  0x1000-0x0fff] to [bus 3a] add_size 1000
[ 3985.833770] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 3a] add_size 200000
[ 3985.833771] pcieport 0000:07:04.0: bridge window [mem 0x00100000-0x000fffff] to [bus 3a] add_size 200000
[ 3985.833784] pcieport 0000:07:06.0: bridge window [io  0x1000-0x0fff] to [bus 6c] add_size 1000
[ 3985.833786] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 6c] add_size 200000
[ 3985.833788] pcieport 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff] to [bus 6c] add_size 200000
[ 3985.833792] pcieport 0000:07:00.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[ 3985.833794] pcieport 0000:07:04.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
[ 3985.833796] pcieport 0000:07:04.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[ 3985.833797] pcieport 0000:07:06.0: res[14]=[mem 0x00100000-0x000fffff] get_res_add_size add_size 200000
[ 3985.833799] pcieport 0000:07:06.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] get_res_add_size add_size 200000
[ 3985.833801] pcieport 0000:07:00.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
[ 3985.833802] pcieport 0000:07:04.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
[ 3985.833804] pcieport 0000:07:06.0: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
[ 3985.833810] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833811] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833814] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
[ 3985.833815] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
[ 3985.833818] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833820] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833821] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
[ 3985.833823] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
[ 3985.833825] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833827] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833829] pcieport 0000:07:00.0: BAR 13: no space for [io  size 0x1000]
[ 3985.833831] pcieport 0000:07:00.0: BAR 13: failed to assign [io  size 0x1000]
[ 3985.833832] pcieport 0000:07:04.0: BAR 13: no space for [io  size 0x1000]
[ 3985.833834] pcieport 0000:07:04.0: BAR 13: failed to assign [io  size 0x1000]
[ 3985.833835] pcieport 0000:07:06.0: BAR 13: no space for [io  size 0x1000]
[ 3985.833837] pcieport 0000:07:06.0: BAR 13: failed to assign [io  size 0x1000]
[ 3985.833840] pcieport 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
[ 3985.833841] pcieport 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
[ 3985.833844] pcieport 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833845] pcieport 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833847] pcieport 0000:07:06.0: BAR 13: no space for [io  size 0x1000]
[ 3985.833848] pcieport 0000:07:06.0: BAR 13: failed to assign [io  size 0x1000]
[ 3985.833850] pcieport 0000:07:04.0: BAR 14: no space for [mem size 0x00200000]
[ 3985.833852] pcieport 0000:07:04.0: BAR 14: failed to assign [mem size 0x00200000]
[ 3985.833854] pcieport 0000:07:04.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833856] pcieport 0000:07:04.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833858] pcieport 0000:07:04.0: BAR 13: no space for [io  size 0x1000]
[ 3985.833859] pcieport 0000:07:04.0: BAR 13: failed to assign [io  size 0x1000]
[ 3985.833862] pcieport 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[ 3985.833863] pcieport 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[ 3985.833865] pcieport 0000:07:00.0: BAR 13: no space for [io  size 0x1000]
[ 3985.833866] pcieport 0000:07:00.0: BAR 13: failed to assign [io  size 0x1000]

Hrm. Any idea what to do here? I also tried some 'setpci' hackery on the
surviving bridge with a secondary bus reset and a link retrain. Neither
seems to have any effect.

- Steven

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2014-08-26 15:58 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <6486222925419988442@unknownmsgid>
2014-08-15 11:24 ` Thunderbolt driver hotplug not working correctly Steven Noonan
     [not found]   ` <CAMxnaaXwY1oyaws-SG3=DzO0qo12bT0-6V9mti7fO11--sp7Zw@mail.gmail.com>
     [not found]     ` <CAKbGBLivQ1cP_EN=maQ0fHAUKfmpWyEY1T5frqv4nAqTdXw5GQ@mail.gmail.com>
     [not found]       ` <CAMxnaaXwrgh4XVUEhEVDfUJ=9n+OWD3Ncj-=FOHUQ1Jw5xwSvw@mail.gmail.com>
     [not found]         ` <20140815173539.GA2012@twoflower.us-west-2.compute.internal>
2014-08-15 21:14           ` Andreas Noever
2014-08-26 15:58             ` Andreas Noever
2014-08-12  9:01 Steven Noonan
2014-08-12  9:05 ` Greg KH
2014-08-12 22:55 ` Andreas Noever
2014-08-13 23:05   ` Andreas Noever
2014-08-15 11:19   ` Steven Noonan

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.