Possible problem with thunderbolt 4

Possible problem with thunderbolt 4

[Christian Schaubschläger]

[-- Attachment #1: Type: text/plain, Size: 2923 bytes --]

Hello list,

I'm having an issue which I guess might be related to thunderbolt 4.

I have an HP EliteBook 630 G9 notebook and an HP Thunderbolt Dock G4, both with Thunderbolt 4. I run linux on it and need a working network interface (the one on the dock) in the UEFI firmware for PXE boot.

When I come from a cold boot, the thunderbolt connection works well both in the UEFI firmware (I can do PXE), and also later in linux. After a reboot from linux, the dock disappears from the PCI bus and is no longer accessible in the UEFI firmware. Hence a PXE boot is not possible. When I then boot into linux again, the dock is there again, working just fine.

On my machine the thunderbolt controller has the PCI address 0000.00.0d.2, and the PCI bridge to the dock has the address 0000.00.07.0.
I've attached the PCI config spaces of these two devices as they are seen from the UEFI firmeware from two different states:

 1. When the machine comes from a cold boot. In that state the UEFI firmware sees the dock and all devices on the dock.
 2. When the machine comes from a linux reboot. In that state the dock is not visible on the PCI bus.

The config spaces of the mentioned two devices are different in the two states.

Note: once the machine is in state 2, it is necessary to remove the power supply from the dock (or physically disconnect and re-connect the thunderbolt cable) in order to get it working in UEFI again. That's what "cold boot" above actually means.

Also, when the machine is in state 2 and boots into Windows the dock does not not become visible on the PCI bus. Interestingly, after a subsequent reboot from Windows it does become avialable in UEFI again (no need to disconnect the power supply or thunderbolt cable in this case!!)

So I guess the linux kernel does something on shutdown (or misses to do something) that prevents the dock to wake up again after the reboot in the UEFI firmware.

I'm observing this on all kernels I've tried (5.18.x, 6.0.x, 6.1.x; also when I run a vanilla Ubuntu 22.04 this happens); the logs below are from a pre-release kernel from today (which will be 6.2-rc1 in a few days). I've also experimented with some powersaving related settings on the kernel command line, unfortunately without success.

Can anyone confirm this behaviour?

Note2: when I use this TB4 Dock on a laptop with an older Thunderbolt 3 controller, none of these problems occur.

Files attached:

pci_config_space_000700_cold.txt: 07.00's config space from within UEFI after a cold boot
pci_config_space_000700_lnx.txt: 07.00's config space after a reboot from linux
pci_config_space_000D02_cold.txt: 0D.02's config space after a cold boot
pci_config_space_000D02_lnx.txt: 0D.02's config space after a reboot from linux

lscpi*: lspci output from linux

dmesg_cold.txt: dmesg output after a cold boot
dmesg_after_lnx.txt: dmesg output after a reboot from linux

Thanks and best regards,
Christian


[-- Attachment #2: pci_config_space_000D02_lnx.txt.xz --]
[-- Type: application/x-xz, Size: 404 bytes --]

[-- Attachment #3: pci_config_space_000D02_cold.txt.xz --]
[-- Type: application/x-xz, Size: 408 bytes --]

[-- Attachment #4: pci_config_space_000700_lnx.txt.xz --]
[-- Type: application/x-xz, Size: 1200 bytes --]

[-- Attachment #5: pci_config_space_000700_cold.txt.xz --]
[-- Type: application/x-xz, Size: 1244 bytes --]

[-- Attachment #6: lspci-vv.txt.xz --]
[-- Type: application/x-xz, Size: 5916 bytes --]

[-- Attachment #7: lspci-t.txt.xz --]
[-- Type: application/x-xz, Size: 204 bytes --]

[-- Attachment #8: dmesg_cold.txt.xz --]
[-- Type: application/x-xz, Size: 20096 bytes --]

[-- Attachment #9: dmesg_after_lnx.txt.xz --]
[-- Type: application/x-xz, Size: 20448 bytes --]

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi,

On Fri, Dec 23, 2022 at 12:24:35PM +0100, Christian Schaubschläger wrote:
> Hello list,
> 
> I'm having an issue which I guess might be related to thunderbolt 4.
> 
> I have an HP EliteBook 630 G9 notebook and an HP Thunderbolt Dock G4, both with Thunderbolt 4. I run linux on it and need a working network interface (the one on the dock) in the UEFI firmware for PXE boot.
> 
> When I come from a cold boot, the thunderbolt connection works well both in the UEFI firmware (I can do PXE), and also later in linux. After a reboot from linux, the dock disappears from the PCI bus and is no longer accessible in the UEFI firmware. Hence a PXE boot is not possible. When I then boot into linux again, the dock is there again, working just fine.
> 
> On my machine the thunderbolt controller has the PCI address 0000.00.0d.2, and the PCI bridge to the dock has the address 0000.00.07.0.
> I've attached the PCI config spaces of these two devices as they are seen from the UEFI firmeware from two different states:
> 
>  1. When the machine comes from a cold boot. In that state the UEFI firmware sees the dock and all devices on the dock.
>  2. When the machine comes from a linux reboot. In that state the dock is not visible on the PCI bus.
> 
> The config spaces of the mentioned two devices are different in the two states.
> 
> Note: once the machine is in state 2, it is necessary to remove the power supply from the dock (or physically disconnect and re-connect the thunderbolt cable) in order to get it working in UEFI again. That's what "cold boot" above actually means.
> 
> Also, when the machine is in state 2 and boots into Windows the dock does not not become visible on the PCI bus. Interestingly, after a subsequent reboot from Windows it does become avialable in UEFI again (no need to disconnect the power supply or thunderbolt cable in this case!!)
> 
> So I guess the linux kernel does something on shutdown (or misses to do something) that prevents the dock to wake up again after the reboot in the UEFI firmware.
> 
> I'm observing this on all kernels I've tried (5.18.x, 6.0.x, 6.1.x; also when I run a vanilla Ubuntu 22.04 this happens); the logs below are from a pre-release kernel from today (which will be 6.2-rc1 in a few days). I've also experimented with some powersaving related settings on the kernel command line, unfortunately without success.
> 
> Can anyone confirm this behaviour?

First of all can you check if you are running Intel or Microsoft driver
for the Thunderbolt controller? It can be seen in Device Manager
somehow. It is possible that Windows and Linux use different "connection
manager" so that explains why there is a difference in behaviour.

In case of Linux this is software connection manager so it is Linux that
does all the tunneling. In case of Windows it may be also firmware
connection manager so it is handled in the firmware (and this might
explain why it magically works after rebooting from Windows).

In general this depends on the BIOS setting whether there is PCIe tunnel
or not. Typically there is something like "boot from Thunderbolt" or
similar option that turns it on so I suggest checking if you have
such option.

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi,

Am 27.12.22 um 15:28 schrieb Mika Westerberg:
> Hi,
>
> On Fri, Dec 23, 2022 at 12:24:35PM +0100, Christian Schaubschläger wrote:
>> Hello list,
>>
>> I'm having an issue which I guess might be related to thunderbolt 4.
>>
>> I have an HP EliteBook 630 G9 notebook and an HP Thunderbolt Dock G4, both with Thunderbolt 4. I run linux on it and need a working network interface (the one on the dock) in the UEFI firmware for PXE boot.
>>
>> When I come from a cold boot, the thunderbolt connection works well both in the UEFI firmware (I can do PXE), and also later in linux. After a reboot from linux, the dock disappears from the PCI bus and is no longer accessible in the UEFI firmware. Hence a PXE boot is not possible. When I then boot into linux again, the dock is there again, working just fine.
>>
>> On my machine the thunderbolt controller has the PCI address 0000.00.0d.2, and the PCI bridge to the dock has the address 0000.00.07.0.
>> I've attached the PCI config spaces of these two devices as they are seen from the UEFI firmeware from two different states:
>>
>>  1. When the machine comes from a cold boot. In that state the UEFI firmware sees the dock and all devices on the dock.
>>  2. When the machine comes from a linux reboot. In that state the dock is not visible on the PCI bus.
>>
>> The config spaces of the mentioned two devices are different in the two states.
>>
>> Note: once the machine is in state 2, it is necessary to remove the power supply from the dock (or physically disconnect and re-connect the thunderbolt cable) in order to get it working in UEFI again. That's what "cold boot" above actually means.
>>
>> Also, when the machine is in state 2 and boots into Windows the dock does not not become visible on the PCI bus. Interestingly, after a subsequent reboot from Windows it does become avialable in UEFI again (no need to disconnect the power supply or thunderbolt cable in this case!!)
>>
>> So I guess the linux kernel does something on shutdown (or misses to do something) that prevents the dock to wake up again after the reboot in the UEFI firmware.
>>
>> I'm observing this on all kernels I've tried (5.18.x, 6.0.x, 6.1.x; also when I run a vanilla Ubuntu 22.04 this happens); the logs below are from a pre-release kernel from today (which will be 6.2-rc1 in a few days). I've also experimented with some powersaving related settings on the kernel command line, unfortunately without success.
>>
>> Can anyone confirm this behaviour?
> First of all can you check if you are running Intel or Microsoft driver
> for the Thunderbolt controller? It can be seen in Device Manager
> somehow. It is possible that Windows and Linux use different "connection
> manager" so that explains why there is a difference in behaviour.
The TB contoller in Windows uses the Intel driver, the dock itself a driver from Microsoft.

> In case of Linux this is software connection manager so it is Linux that
> does all the tunneling. In case of Windows it may be also firmware
> connection manager so it is handled in the firmware (and this might
> explain why it magically works after rebooting from Windows).
>
> In general this depends on the BIOS setting whether there is PCIe tunnel
> or not. Typically there is something like "boot from Thunderbolt" or
> similar option that turns it on so I suggest checking if you have
> such option.
Unfortunately there is no such BIOS setting on this machine...

But as described above: when the laptop comes from a cold boot (with power supply removed before, etc.), then there _is_ a PCIe tunnel in the UEFI firmware; so the firmware can do that. Only after a reboot from Linux something prohibits the firmware from re-establishing the tunnel again. And I'm not sure if this is a kernel issue or a firmware issue, but it clearly makes a valid use case (pxe boot after linux) impossible.

Can I do anything to bring more light to this problem?

Thanks and best regards,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Him

On Fri, Dec 30, 2022 at 08:57:37AM +0100, Christian Schaubschläger wrote:
> Hi,
> 
> Am 27.12.22 um 15:28 schrieb Mika Westerberg:
> > Hi,
> >
> > On Fri, Dec 23, 2022 at 12:24:35PM +0100, Christian Schaubschläger wrote:
> >> Hello list,
> >>
> >> I'm having an issue which I guess might be related to thunderbolt 4.
> >>
> >> I have an HP EliteBook 630 G9 notebook and an HP Thunderbolt Dock G4, both with Thunderbolt 4. I run linux on it and need a working network interface (the one on the dock) in the UEFI firmware for PXE boot.
> >>
> >> When I come from a cold boot, the thunderbolt connection works well both in the UEFI firmware (I can do PXE), and also later in linux. After a reboot from linux, the dock disappears from the PCI bus and is no longer accessible in the UEFI firmware. Hence a PXE boot is not possible. When I then boot into linux again, the dock is there again, working just fine.
> >>
> >> On my machine the thunderbolt controller has the PCI address 0000.00.0d.2, and the PCI bridge to the dock has the address 0000.00.07.0.
> >> I've attached the PCI config spaces of these two devices as they are seen from the UEFI firmeware from two different states:
> >>
> >>  1. When the machine comes from a cold boot. In that state the UEFI firmware sees the dock and all devices on the dock.
> >>  2. When the machine comes from a linux reboot. In that state the dock is not visible on the PCI bus.
> >>
> >> The config spaces of the mentioned two devices are different in the two states.
> >>
> >> Note: once the machine is in state 2, it is necessary to remove the power supply from the dock (or physically disconnect and re-connect the thunderbolt cable) in order to get it working in UEFI again. That's what "cold boot" above actually means.
> >>
> >> Also, when the machine is in state 2 and boots into Windows the dock does not not become visible on the PCI bus. Interestingly, after a subsequent reboot from Windows it does become avialable in UEFI again (no need to disconnect the power supply or thunderbolt cable in this case!!)
> >>
> >> So I guess the linux kernel does something on shutdown (or misses to do something) that prevents the dock to wake up again after the reboot in the UEFI firmware.
> >>
> >> I'm observing this on all kernels I've tried (5.18.x, 6.0.x, 6.1.x; also when I run a vanilla Ubuntu 22.04 this happens); the logs below are from a pre-release kernel from today (which will be 6.2-rc1 in a few days). I've also experimented with some powersaving related settings on the kernel command line, unfortunately without success.
> >>
> >> Can anyone confirm this behaviour?
> > First of all can you check if you are running Intel or Microsoft driver
> > for the Thunderbolt controller? It can be seen in Device Manager
> > somehow. It is possible that Windows and Linux use different "connection
> > manager" so that explains why there is a difference in behaviour.
> The TB contoller in Windows uses the Intel driver, the dock itself a driver from Microsoft.

Okay then it is using "Firmware Connection Manager" whereas Linux is
using "Software Connection Manager" so completely diffrent things.

> > In case of Linux this is software connection manager so it is Linux that
> > does all the tunneling. In case of Windows it may be also firmware
> > connection manager so it is handled in the firmware (and this might
> > explain why it magically works after rebooting from Windows).
> >
> > In general this depends on the BIOS setting whether there is PCIe tunnel
> > or not. Typically there is something like "boot from Thunderbolt" or
> > similar option that turns it on so I suggest checking if you have
> > such option.
> Unfortunately there is no such BIOS setting on this machine...
> 
> But as described above: when the laptop comes from a cold boot (with
> power supply removed before, etc.), then there _is_ a PCIe tunnel in
> the UEFI firmware; so the firmware can do that. Only after a reboot
> from Linux something prohibits the firmware from re-establishing the
> tunnel again. And I'm not sure if this is a kernel issue or a firmware
> issue, but it clearly makes a valid use case (pxe boot after linux)
> impossible.
> 
> Can I do anything to bring more light to this problem?

One thing you can try is to "force" Linux to use the same FW CM path
than Windows. This is done by compiling your kernel with CONFIG_USB4=n.
This should enable the firmware CM in Linux side as well. All the
tunneling (except software/networking/P2P) should work with it even if
the Thunderbolt driver is not loaded. I wonder if you can try that and
see if the PXE starts working better?

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

[-- Attachment #1: Type: text/plain, Size: 5808 bytes --]

Hello,

Am 30.12.22 um 10:28 schrieb Mika Westerberg:
> Him
>
> On Fri, Dec 30, 2022 at 08:57:37AM +0100, Christian Schaubschläger wrote:
>> Hi,
>>
>> Am 27.12.22 um 15:28 schrieb Mika Westerberg:
>>> Hi,
>>>
>>> On Fri, Dec 23, 2022 at 12:24:35PM +0100, Christian Schaubschläger wrote:
>>>> Hello list,
>>>>
>>>> I'm having an issue which I guess might be related to thunderbolt 4.
>>>>
>>>> I have an HP EliteBook 630 G9 notebook and an HP Thunderbolt Dock G4, both with Thunderbolt 4. I run linux on it and need a working network interface (the one on the dock) in the UEFI firmware for PXE boot.
>>>>
>>>> When I come from a cold boot, the thunderbolt connection works well both in the UEFI firmware (I can do PXE), and also later in linux. After a reboot from linux, the dock disappears from the PCI bus and is no longer accessible in the UEFI firmware. Hence a PXE boot is not possible. When I then boot into linux again, the dock is there again, working just fine.
>>>>
>>>> On my machine the thunderbolt controller has the PCI address 0000.00.0d.2, and the PCI bridge to the dock has the address 0000.00.07.0.
>>>> I've attached the PCI config spaces of these two devices as they are seen from the UEFI firmeware from two different states:
>>>>
>>>>  1. When the machine comes from a cold boot. In that state the UEFI firmware sees the dock and all devices on the dock.
>>>>  2. When the machine comes from a linux reboot. In that state the dock is not visible on the PCI bus.
>>>>
>>>> The config spaces of the mentioned two devices are different in the two states.
>>>>
>>>> Note: once the machine is in state 2, it is necessary to remove the power supply from the dock (or physically disconnect and re-connect the thunderbolt cable) in order to get it working in UEFI again. That's what "cold boot" above actually means.
>>>>
>>>> Also, when the machine is in state 2 and boots into Windows the dock does not not become visible on the PCI bus. Interestingly, after a subsequent reboot from Windows it does become avialable in UEFI again (no need to disconnect the power supply or thunderbolt cable in this case!!)
>>>>
>>>> So I guess the linux kernel does something on shutdown (or misses to do something) that prevents the dock to wake up again after the reboot in the UEFI firmware.
>>>>
>>>> I'm observing this on all kernels I've tried (5.18.x, 6.0.x, 6.1.x; also when I run a vanilla Ubuntu 22.04 this happens); the logs below are from a pre-release kernel from today (which will be 6.2-rc1 in a few days). I've also experimented with some powersaving related settings on the kernel command line, unfortunately without success.
>>>>
>>>> Can anyone confirm this behaviour?
>>> First of all can you check if you are running Intel or Microsoft driver
>>> for the Thunderbolt controller? It can be seen in Device Manager
>>> somehow. It is possible that Windows and Linux use different "connection
>>> manager" so that explains why there is a difference in behaviour.
>> The TB contoller in Windows uses the Intel driver, the dock itself a driver from Microsoft.
> Okay then it is using "Firmware Connection Manager" whereas Linux is
> using "Software Connection Manager" so completely diffrent things.
>
>>> In case of Linux this is software connection manager so it is Linux that
>>> does all the tunneling. In case of Windows it may be also firmware
>>> connection manager so it is handled in the firmware (and this might
>>> explain why it magically works after rebooting from Windows).
>>>
>>> In general this depends on the BIOS setting whether there is PCIe tunnel
>>> or not. Typically there is something like "boot from Thunderbolt" or
>>> similar option that turns it on so I suggest checking if you have
>>> such option.
>> Unfortunately there is no such BIOS setting on this machine...
>>
>> But as described above: when the laptop comes from a cold boot (with
>> power supply removed before, etc.), then there _is_ a PCIe tunnel in
>> the UEFI firmware; so the firmware can do that. Only after a reboot
>> from Linux something prohibits the firmware from re-establishing the
>> tunnel again. And I'm not sure if this is a kernel issue or a firmware
>> issue, but it clearly makes a valid use case (pxe boot after linux)
>> impossible.
>>
>> Can I do anything to bring more light to this problem?
> One thing you can try is to "force" Linux to use the same FW CM path
> than Windows. This is done by compiling your kernel with CONFIG_USB4=n.
> This should enable the firmware CM in Linux side as well. All the
> tunneling (except software/networking/P2P) should work with it even if
> the Thunderbolt driver is not loaded. I wonder if you can try that and
> see if the PXE starts working better?
Ok, setting CONFIG_USB4=n makes everything work as expected! Even hotplugging the dock works fine.

Now I've tried this: CONFIG_USB4=m and blacklist the thunderbolt module (because disabling CONFIG_USB4 in general is not an option for me (I guess?) There's probably hardware out there where the firmware doesn't set up the PCIe tunnels. Then I need the linux thunderbolt driver to do this I suppose).

So with CONFIG_USB=m and blacklisting the thunderbolt module this happens: the tunnel is there before and after linux (PXE works), but in linux none of the devices that sit behind the PCIe tunnel (network, external display) are there... that's strange, because I would have expected that CONFIG_USB4=m + blacklist thunderbolt would be the same as CONFIG_USB4=n. Which obviously isn't the case. I've attached lspci output from either case.

Is there any other option to decide at boottime whether or not to use the USB4 driver (besides using two different kernels)?

Thanks and best regards,
Christian

[-- Attachment #2: lspci_vv_no_usb4.txt.xz --]
[-- Type: application/x-xz, Size: 5932 bytes --]

[-- Attachment #3: lspci_vv_blacklist_usb4.txt.xz --]
[-- Type: application/x-xz, Size: 5052 bytes --]

[-- Attachment #4: lspci_t_blacklist_usb4.txt.xz --]
[-- Type: application/x-xz, Size: 160 bytes --]

[-- Attachment #5: lspci_t_no_usb4.txt.xz --]
[-- Type: application/x-xz, Size: 204 bytes --]

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

On Fri, Dec 30, 2022 at 12:38:27PM +0100, Christian Schaubschläger wrote:
> Ok, setting CONFIG_USB4=n makes everything work as expected! Even
> hotplugging the dock works fine.

Yes but things like power management does not.

> Now I've tried this: CONFIG_USB4=m and blacklist the thunderbolt
> module (because disabling CONFIG_USB4 in general is not an option for
> me (I guess?) There's probably hardware out there where the firmware
> doesn't set up the PCIe tunnels. Then I need the linux thunderbolt
> driver to do this I suppose).

You must have CONFIG_USB4=n because that makes the Linux ACPI core not
to negotiate software connection manager support. Blaclisting the
Thunderbolt driver does not do that as that code is always built-in.

> So with CONFIG_USB=m and blacklisting the thunderbolt module this
> happens: the tunnel is there before and after linux (PXE works), but
> in linux none of the devices that sit behind the PCIe tunnel (network,
> external display) are there... that's strange, because I would have
> expected that CONFIG_USB4=m + blacklist thunderbolt would be the same
> as CONFIG_USB4=n. Which obviously isn't the case. I've attached lspci
> output from either case.

See above.

> Is there any other option to decide at boottime whether or not to use
> the USB4 driver (besides using two different kernels)?

No but this is not the "final" solution - just an experiment.

Okay so we have this:

- When both Linux and Windows uses the same firmware connection manager
  all work.

- If you install Windows 11 (or whatever the fresh one is that actually
  supports USB4 and software connection manager) you end up in the
  exact same situation.

Can you try following:

1. Go back to CONFIG_USB4=m
2. Boot the system up
3. Check that the PCIe tunneling is up and things work as expected
4. Unplug the dock
5. Unload the Thunderbolt driver

  # rmmod thunderbolt

6. Plug the dock back
7. Soft reboot the system

Does this make the PXE boot see the connected device?

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

> No but this is not the "final" solution - just an experiment.
>
> Okay so we have this:
>
> - When both Linux and Windows uses the same firmware connection manager
>   all work.
>
> - If you install Windows 11 (or whatever the fresh one is that actually
>   supports USB4 and software connection manager) you end up in the
>   exact same situation.
I'm running Win10 22H2 on this machine... I could upgrade to Win11 if this helps.

> Can you try following:
>
> 1. Go back to CONFIG_USB4=m
> 2. Boot the system up
> 3. Check that the PCIe tunneling is up and things work as expected
> 4. Unplug the dock
> 5. Unload the Thunderbolt driver
>
>   # rmmod thunderbolt
>
> 6. Plug the dock back
> 7. Soft reboot the system
>
> Does this make the PXE boot see the connected device?
After these steps the firmware sees the connected device!

If I just remove the module without un- and re-plugging the cable, the connection is not there in the firmware after the reboot.

Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

On Fri, Dec 30, 2022 at 01:08:36PM +0100, Christian Schaubschläger wrote:
> 
> > No but this is not the "final" solution - just an experiment.
> >
> > Okay so we have this:
> >
> > - When both Linux and Windows uses the same firmware connection manager
> >   all work.
> >
> > - If you install Windows 11 (or whatever the fresh one is that actually
> >   supports USB4 and software connection manager) you end up in the
> >   exact same situation.
> I'm running Win10 22H2 on this machine... I could upgrade to Win11 if this helps.

Most likely it will just make the Windows work like Linux (so PXE does
not work) therere I don't suggest doing that.

> > Can you try following:
> >
> > 1. Go back to CONFIG_USB4=m
> > 2. Boot the system up
> > 3. Check that the PCIe tunneling is up and things work as expected
> > 4. Unplug the dock
> > 5. Unload the Thunderbolt driver
> >
> >   # rmmod thunderbolt
> >
> > 6. Plug the dock back
> > 7. Soft reboot the system
> >
> > Does this make the PXE boot see the connected device?
> After these steps the firmware sees the connected device!

OK.

> If I just remove the module without un- and re-plugging the cable, the
> connection is not there in the firmware after the reboot.

Right because the driver leaves all the tunnels up so the boot firmware
then finds the PCIe tunnels established.

One more experiment if you will.

Same steps 1-3 as above but then do this:

4. Disconnect the PCIe tunnel directly:

  # echo 0 > /sys/bus/thunderbolt/devices/DEVICE/authorized

(where DEVICE is typically 0-1 or 0-3 in Intel platforms)

5. Check in lspci that it is not visible anymore
6. Soft boot the system.

In other words this leaves the USB4 link and other tunnels up but not
PCIe.

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi, (and happy new year!)

>> If I just remove the module without un- and re-plugging the cable, the
>> connection is not there in the firmware after the reboot.
> Right because the driver leaves all the tunnels up so the boot firmware
> then finds the PCIe tunnels established.

So maybe it's a firmware issue? If the kernel leaves the tunnels in place, but the firmware doesn't see them?

> One more experiment if you will.
>
> Same steps 1-3 as above but then do this:
>
> 4. Disconnect the PCIe tunnel directly:
>
>   # echo 0 > /sys/bus/thunderbolt/devices/DEVICE/authorized
>
> (where DEVICE is typically 0-1 or 0-3 in Intel platforms)
>
> 5. Check in lspci that it is not visible anymore
> 6. Soft boot the system.
>
> In other words this leaves the USB4 link and other tunnels up but not
> PCIe.
After echoing 0 to the 'authorized' file, the devices disappear in linux from the PCI bus, but unfortunately don't show up in the firmware after the reboot...

Thanks & regards,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

On Mon, Jan 02, 2023 at 09:56:18AM +0100, Christian Schaubschläger wrote:
> Hi, (and happy new year!)
> 
> >> If I just remove the module without un- and re-plugging the cable, the
> >> connection is not there in the firmware after the reboot.
> > Right because the driver leaves all the tunnels up so the boot firmware
> > then finds the PCIe tunnels established.
> 
> So maybe it's a firmware issue? If the kernel leaves the tunnels in
> place, but the firmware doesn't see them?

It probably sees them and that's the issue.

> > One more experiment if you will.
> >
> > Same steps 1-3 as above but then do this:
> >
> > 4. Disconnect the PCIe tunnel directly:
> >
> >   # echo 0 > /sys/bus/thunderbolt/devices/DEVICE/authorized
> >
> > (where DEVICE is typically 0-1 or 0-3 in Intel platforms)
> >
> > 5. Check in lspci that it is not visible anymore
> > 6. Soft boot the system.
> >
> > In other words this leaves the USB4 link and other tunnels up but not
> > PCIe.
> After echoing 0 to the 'authorized' file, the devices disappear in
> linux from the PCI bus, but unfortunately don't show up in the
> firmware after the reboot...

Okay it may be that the USB 3 tunnel that is still up makes the CM in
BIOS to think the hardware is in unexected state or so.

This was Thunderbolt 4 dock, right? Do you happen to have Thunderbolt 3
device or Thunderbolt 3 active cable around? If yes then there is only
the PCIe tunnel so doing the above de-authorization should in theory
work. If no then one option is to add the ->shutdown() hook to tear down
the tunnels.

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi,

>> After echoing 0 to the 'authorized' file, the devices disappear in
>> linux from the PCI bus, but unfortunately don't show up in the
>> firmware after the reboot...
> Okay it may be that the USB 3 tunnel that is still up makes the CM in
> BIOS to think the hardware is in unexected state or so.
>
> This was Thunderbolt 4 dock, right? Do you happen to have Thunderbolt 3
> device or Thunderbolt 3 active cable around? If yes then there is only
> the PCIe tunnel so doing the above de-authorization should in theory
> work. If no then one option is to add the ->shutdown() hook to tear down
> the tunnels.
I have a slightly older EliteBook with a TB3 controller. The TB4 Dock works flawlessly with this.
And I have the predecessor of the TB4 Dock which also uses TB3. This works fine on the new EliteBook with the TB4 controller.

Both docks have their thunderbolt cables 'built-in', so I cannot switch cables...

Any more tests I could make?

Thanks and best regards,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi,

On Mon, Jan 02, 2023 at 12:45:40PM +0100, Christian Schaubschläger wrote:
> Hi,
> 
> >> After echoing 0 to the 'authorized' file, the devices disappear in
> >> linux from the PCI bus, but unfortunately don't show up in the
> >> firmware after the reboot...
> > Okay it may be that the USB 3 tunnel that is still up makes the CM in
> > BIOS to think the hardware is in unexected state or so.
> >
> > This was Thunderbolt 4 dock, right? Do you happen to have Thunderbolt 3
> > device or Thunderbolt 3 active cable around? If yes then there is only
> > the PCIe tunnel so doing the above de-authorization should in theory
> > work. If no then one option is to add the ->shutdown() hook to tear down
> > the tunnels.
> I have a slightly older EliteBook with a TB3 controller. The TB4 Dock works flawlessly with this.
> And I have the predecessor of the TB4 Dock which also uses TB3. This
> works fine on the new EliteBook with the TB4 controller.
> 
> Both docks have their thunderbolt cables 'built-in', so I cannot switch cables...

I see.

> Any more tests I could make?

Can you try the below hack? It should tear down all tunnels during
reboot so the firmware should see pristine path configuration spaces
(assuming it is looking at them and failing because if already configure
paths).

diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
index 1711dc19b1e2..e0544843e242 100644
--- a/drivers/thunderbolt/tb.c
+++ b/drivers/thunderbolt/tb.c
@@ -1982,13 +1982,7 @@ static void tb_stop(struct tb *tb)
 	cancel_delayed_work(&tcm->remove_work);
 	/* tunnels are only present after everything has been initialized */
 	list_for_each_entry_safe(tunnel, n, &tcm->tunnel_list, list) {
-		/*
-		 * DMA tunnels require the driver to be functional so we
-		 * tear them down. Other protocol tunnels can be left
-		 * intact.
-		 */
-		if (tb_tunnel_is_dma(tunnel))
-			tb_tunnel_deactivate(tunnel);
+		tb_tunnel_deactivate(tunnel);
 		tb_tunnel_free(tunnel);
 	}
 	tb_switch_remove(tb->root_switch);

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Good morning,

> Can you try the below hack? It should tear down all tunnels during
> reboot so the firmware should see pristine path configuration spaces
> (assuming it is looking at them and failing because if already configure
> paths).
>
> diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
> index 1711dc19b1e2..e0544843e242 100644
> --- a/drivers/thunderbolt/tb.c
> +++ b/drivers/thunderbolt/tb.c
> @@ -1982,13 +1982,7 @@ static void tb_stop(struct tb *tb)
>  	cancel_delayed_work(&tcm->remove_work);
>  	/* tunnels are only present after everything has been initialized */
>  	list_for_each_entry_safe(tunnel, n, &tcm->tunnel_list, list) {
> -		/*
> -		 * DMA tunnels require the driver to be functional so we
> -		 * tear them down. Other protocol tunnels can be left
> -		 * intact.
> -		 */
> -		if (tb_tunnel_is_dma(tunnel))
> -			tb_tunnel_deactivate(tunnel);
> +		tb_tunnel_deactivate(tunnel);
>  		tb_tunnel_free(tunnel);
>  	}
>  	tb_switch_remove(tb->root_switch);
That doesn't change the behaviour unfortunately.

I did find out two (confusing) things, though. The network interface on the dock is an Intel I255-LMvP, in linux it's eth1. When I do this before the reboot:

echo 1 > /sys/class/net/eth1/device/remove

the device is removed and then added again immedidately:

[   31.930950] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   77.496059] igc 0000:2c:00.0 eth1: PHC removed
[   77.670531] pci 0000:2c:00.0: Removing from iommu group 21
[   78.551553] pci 0000:2c:00.0: [8086:5502] type 00 class 0x020000
[   78.552412] pci 0000:2c:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
[   78.553194] pci 0000:2c:00.0: reg 0x1c: [mem 0x00000000-0x00003fff]
[   78.554128] pci 0000:2c:00.0: PME# supported from D0 D3hot D3cold
[   78.555110] pci 0000:2c:00.0: Adding to iommu group 21
[   78.555913] pcieport 0000:04:04.0: ASPM: current common clock configuration is inconsistent, reconfiguring
[   78.566265] pci_bus 0000:05: Allocating resources
[   78.566285] pci_bus 0000:2c: Allocating resources
[   78.566296] pci 0000:2c:00.0: BAR 0: assigned [mem 0x82000000-0x820fffff]
[   78.567075] pci 0000:2c:00.0: BAR 3: assigned [mem 0x82100000-0x82103fff]
[   78.567862] igc 0000:2c:00.0: enabling device (0000 -> 0002)
[   78.568671] igc 0000:2c:00.0: PTM enabled, 4ns granularity
[   78.626297] pps pps0: new PPS source ptp1
[   78.627161] igc 0000:2c:00.0 (unnamed net_device) (uninitialized): PHC added
[   78.653444] igc 0000:2c:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
[   78.654448] igc 0000:2c:00.0 eth1: MAC: 5c:60:ba:71:ea:be
[   78.655307] pci_bus 0000:05: Allocating resources
[   78.655327] pci_bus 0000:2c: Allocating resources
[   81.598703] igc 0000:2c:00.0 eth1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   81.599729] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready


If I reboot now, I can see the network interface in the firmware!

And now for the confusing part: I do exactly the same steps as above, but whith an external display connected to the dock. Even if I don't touch this display in Linux (the i915 driver isn't loaded, just efi framebuffer), then the net interface is gone in the firmware after the reboot...

Also, after the

echo 1 > /sys/class/net/eth1/device/remove


the device doesn't show up in linux again:

[   10.158596] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   42.773958] igc 0000:2c:00.0 eth1: PHC removed
[   42.962208] pci 0000:2c:00.0: Removing from iommu group 21


I'm not sure if this is some weird coincidence or if it can help to bring ligth to this issue...

Thanks and best regards,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi,

On Tue, Jan 03, 2023 at 10:08:16AM +0100, Christian Schaubschläger wrote:
> Good morning,
> 
> > Can you try the below hack? It should tear down all tunnels during
> > reboot so the firmware should see pristine path configuration spaces
> > (assuming it is looking at them and failing because if already configure
> > paths).
> >
> > diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
> > index 1711dc19b1e2..e0544843e242 100644
> > --- a/drivers/thunderbolt/tb.c
> > +++ b/drivers/thunderbolt/tb.c
> > @@ -1982,13 +1982,7 @@ static void tb_stop(struct tb *tb)
> >  	cancel_delayed_work(&tcm->remove_work);
> >  	/* tunnels are only present after everything has been initialized */
> >  	list_for_each_entry_safe(tunnel, n, &tcm->tunnel_list, list) {
> > -		/*
> > -		 * DMA tunnels require the driver to be functional so we
> > -		 * tear them down. Other protocol tunnels can be left
> > -		 * intact.
> > -		 */
> > -		if (tb_tunnel_is_dma(tunnel))
> > -			tb_tunnel_deactivate(tunnel);
> > +		tb_tunnel_deactivate(tunnel);
> >  		tb_tunnel_free(tunnel);
> >  	}
> >  	tb_switch_remove(tb->root_switch);
> That doesn't change the behaviour unfortunately.

Okay then it is not about the paths.

> I did find out two (confusing) things, though. The network interface
> on the dock is an Intel I255-LMvP, in linux it's eth1. When I do this
> before the reboot:
> 
> echo 1 > /sys/class/net/eth1/device/remove
> 
> the device is removed and then added again immedidately:
> 
> [   31.930950] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
> [   77.496059] igc 0000:2c:00.0 eth1: PHC removed
> [   77.670531] pci 0000:2c:00.0: Removing from iommu group 21
> [   78.551553] pci 0000:2c:00.0: [8086:5502] type 00 class 0x020000
> [   78.552412] pci 0000:2c:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
> [   78.553194] pci 0000:2c:00.0: reg 0x1c: [mem 0x00000000-0x00003fff]
> [   78.554128] pci 0000:2c:00.0: PME# supported from D0 D3hot D3cold
> [   78.555110] pci 0000:2c:00.0: Adding to iommu group 21
> [   78.555913] pcieport 0000:04:04.0: ASPM: current common clock configuration is inconsistent, reconfiguring
> [   78.566265] pci_bus 0000:05: Allocating resources
> [   78.566285] pci_bus 0000:2c: Allocating resources
> [   78.566296] pci 0000:2c:00.0: BAR 0: assigned [mem 0x82000000-0x820fffff]
> [   78.567075] pci 0000:2c:00.0: BAR 3: assigned [mem 0x82100000-0x82103fff]
> [   78.567862] igc 0000:2c:00.0: enabling device (0000 -> 0002)
> [   78.568671] igc 0000:2c:00.0: PTM enabled, 4ns granularity
> [   78.626297] pps pps0: new PPS source ptp1
> [   78.627161] igc 0000:2c:00.0 (unnamed net_device) (uninitialized): PHC added
> [   78.653444] igc 0000:2c:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
> [   78.654448] igc 0000:2c:00.0 eth1: MAC: 5c:60:ba:71:ea:be
> [   78.655307] pci_bus 0000:05: Allocating resources
> [   78.655327] pci_bus 0000:2c: Allocating resources
> [   81.598703] igc 0000:2c:00.0 eth1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> [   81.599729] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
> 
> 
> If I reboot now, I can see the network interface in the firmware!

Does this happens 100% or it happened just once?

> And now for the confusing part: I do exactly the same steps as above,
> but whith an external display connected to the dock. Even if I don't
> touch this display in Linux (the i915 driver isn't loaded, just efi
> framebuffer), then the net interface is gone in the firmware after the
> reboot...

It is enough if there is monitor plugged the DP tunnel is then up even
if there is no i915.

> Also, after the
> 
> echo 1 > /sys/class/net/eth1/device/remove
> 
> 
> the device doesn't show up in linux again:
> 
> [   10.158596] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
> [   42.773958] igc 0000:2c:00.0 eth1: PHC removed
> [   42.962208] pci 0000:2c:00.0: Removing from iommu group 21

You mean with display connected it does not show up again?

I think what is expected is that when you do

 # echo 1 > /sys/class/net/eth1/device/remove

the network interface is removed and it is not expected to come back but
I realized that when you have the DP connected the TCSS (Type-C
SubsSystem) does not enter D3.

So I wonder if you can run another test still?

0. Remove the hack patch if not already.

Steps 1-3 same as before.

4. Disable runtime PM from the PCIe root ports:

  # echo on > /sys/bus/pci/devices/0000:00:07.0/power/control
  # echo on > /sys/bus/pci/devices/0000:00:07.1/power/control
  # echo on > /sys/bus/pci/devices/0000:00:07.2/power/control
  # echo on > /sys/bus/pci/devices/0000:00:07.3/power/control

(or how many Thunderbolt PCIE root ports you have, disable it from all of them).

5. De-authorize the PCIe tunnel

  # echo 0 > /sys/bus/thunderbolt/devices/.../authorized

6. Soft reboot

Does this change anything or the behavior is the same?

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

[-- Attachment #1: Type: text/plain, Size: 5337 bytes --]

Hi,

>> I did find out two (confusing) things, though. The network interface
>> on the dock is an Intel I255-LMvP, in linux it's eth1. When I do this
>> before the reboot:
>>
>> echo 1 > /sys/class/net/eth1/device/remove
>>
>> the device is removed and then added again immedidately:
>>
>> [   31.930950] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
>> [   77.496059] igc 0000:2c:00.0 eth1: PHC removed
>> [   77.670531] pci 0000:2c:00.0: Removing from iommu group 21
>> [   78.551553] pci 0000:2c:00.0: [8086:5502] type 00 class 0x020000
>> [   78.552412] pci 0000:2c:00.0: reg 0x10: [mem 0x00000000-0x000fffff]
>> [   78.553194] pci 0000:2c:00.0: reg 0x1c: [mem 0x00000000-0x00003fff]
>> [   78.554128] pci 0000:2c:00.0: PME# supported from D0 D3hot D3cold
>> [   78.555110] pci 0000:2c:00.0: Adding to iommu group 21
>> [   78.555913] pcieport 0000:04:04.0: ASPM: current common clock configuration is inconsistent, reconfiguring
>> [   78.566265] pci_bus 0000:05: Allocating resources
>> [   78.566285] pci_bus 0000:2c: Allocating resources
>> [   78.566296] pci 0000:2c:00.0: BAR 0: assigned [mem 0x82000000-0x820fffff]
>> [   78.567075] pci 0000:2c:00.0: BAR 3: assigned [mem 0x82100000-0x82103fff]
>> [   78.567862] igc 0000:2c:00.0: enabling device (0000 -> 0002)
>> [   78.568671] igc 0000:2c:00.0: PTM enabled, 4ns granularity
>> [   78.626297] pps pps0: new PPS source ptp1
>> [   78.627161] igc 0000:2c:00.0 (unnamed net_device) (uninitialized): PHC added
>> [   78.653444] igc 0000:2c:00.0: 4.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x1 link)
>> [   78.654448] igc 0000:2c:00.0 eth1: MAC: 5c:60:ba:71:ea:be
>> [   78.655307] pci_bus 0000:05: Allocating resources
>> [   78.655327] pci_bus 0000:2c: Allocating resources
>> [   81.598703] igc 0000:2c:00.0 eth1: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
>> [   81.599729] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
>>
>>
>> If I reboot now, I can see the network interface in the firmware!
> Does this happens 100% or it happened just once?
This happens always.
>> Also, after the
>>
>> echo 1 > /sys/class/net/eth1/device/remove
>>
>>
>> the device doesn't show up in linux again:
>>
>> [   10.158596] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
>> [   42.773958] igc 0000:2c:00.0 eth1: PHC removed
>> [   42.962208] pci 0000:2c:00.0: Removing from iommu group 21
> You mean with display connected it does not show up again?
Correct! This is also always the case.
> So I wonder if you can run another test still?
>
> 0. Remove the hack patch if not already.
>
> Steps 1-3 same as before.
>
> 4. Disable runtime PM from the PCIe root ports:
>
>   # echo on > /sys/bus/pci/devices/0000:00:07.0/power/control
>   # echo on > /sys/bus/pci/devices/0000:00:07.1/power/control
>   # echo on > /sys/bus/pci/devices/0000:00:07.2/power/control
>   # echo on > /sys/bus/pci/devices/0000:00:07.3/power/control
>
> (or how many Thunderbolt PCIE root ports you have, disable it from all of them).
>
> 5. De-authorize the PCIe tunnel
>
>   # echo 0 > /sys/bus/thunderbolt/devices/.../authorized
>
> 6. Soft reboot
>
> Does this change anything or the behavior is the same?
No, that doesn't change anything; though I'm not sure how many Thunderbolt PCIe root ports I have; I guess only one: 0000:00:07.0 - please have a look into the attached lspci output.

dmesg after the above two steps:

[   91.655320] pcieport 0000:00:07.0: pciehp: Slot(3): Link Down
[   91.657353] pcieport 0000:00:07.0: pciehp: Slot(3): Card not present
[   91.659525] igc 0000:2c:00.0 eth1: PHC removed
[   91.786204] pcieport 0000:04:03.0: Unable to change power state from D3hot to D0, device inaccessible
[   91.788645] pcieport 0000:04:03.0: Runtime PM usage count underflow!
[   91.790766] pcieport 0000:04:02.0: Unable to change power state from D3hot to D0, device inaccessible
[   91.793156] pcieport 0000:04:02.0: Runtime PM usage count underflow!
[   91.795341] pcieport 0000:04:01.0: Unable to change power state from D3hot to D0, device inaccessible
[   91.797797] pcieport 0000:04:01.0: Runtime PM usage count underflow!
[   91.800031] pcieport 0000:04:00.0: Unable to change power state from D3hot to D0, device inaccessible
[   91.802671] pci_bus 0000:05: busn_res: [bus 05] is released
[   91.805004] pci 0000:04:00.0: Removing from iommu group 16
[   91.807261] pci_bus 0000:06: busn_res: [bus 06-12] is released
[   91.809582] pci 0000:04:01.0: Removing from iommu group 17
[   91.811925] pci_bus 0000:13: busn_res: [bus 13-1f] is released
[   91.814303] pci 0000:04:02.0: Removing from iommu group 18
[   91.816604] pci_bus 0000:20: busn_res: [bus 20-2b] is released
[   91.818985] pci 0000:04:03.0: Removing from iommu group 19
[   91.821412] pci 0000:2c:00.0: Removing from iommu group 21
[   91.823875] pci_bus 0000:2c: busn_res: [bus 2c] is released
[   91.826241] pci 0000:04:04.0: Removing from iommu group 20
[   91.828592] pci_bus 0000:04: busn_res: [bus 04-2c] is released
[   91.831018] pci 0000:03:00.0: Removing from iommu group 15

Still, no network interface after the reboot in the firmware...

Best regards,
Christian

[-- Attachment #2: lspci.txt --]
[-- Type: text/plain, Size: 20194 bytes --]

00:00.0 Host bridge: Intel Corporation Device 4601 (rev 04)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: bus master, fast devsel, latency 0, IOMMU group 0
	Capabilities: [e0] Vendor Specific Information: Len=14 <?>

00:02.0 VGA compatible controller: Intel Corporation Device 4628 (rev 0c) (prog-if 00 [VGA controller])
	DeviceName: Onboard IGD
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: bus master, fast devsel, latency 0, IRQ 255, IOMMU group 1
	Memory at 601c000000 (64-bit, non-prefetchable) [size=16M]
	Memory at 4000000000 (64-bit, prefetchable) [size=256M]
	I/O ports at 3000 [size=64]
	Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
	Capabilities: [40] Vendor Specific Information: Len=0c <?>
	Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
	Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit-
	Capabilities: [d0] Power Management version 2
	Capabilities: [100] Process Address Space ID (PASID)
	Capabilities: [200] Address Translation Service (ATS)
	Capabilities: [300] Page Request Interface (PRI)
	Capabilities: [320] Single Root I/O Virtualization (SR-IOV)
	Kernel modules: i915

00:04.0 Signal processing controller: Intel Corporation Device 461d (rev 04)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: fast devsel, IRQ 255, IOMMU group 2
	Memory at 601d140000 (64-bit, non-prefetchable) [disabled] [size=128K]
	Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [d0] Power Management version 3
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>

00:06.0 PCI bridge: Intel Corporation Device 464d (rev 04) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 122, IOMMU group 3
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: [disabled]
	Memory behind bridge: [disabled]
	Prefetchable memory behind bridge: [disabled]
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Hewlett-Packard Company Device 897a
	Capabilities: [a0] Power Management version 3
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [220] Access Control Services
	Capabilities: [150] Precision Time Measurement
	Capabilities: [280] Virtual Channel
	Capabilities: [a30] Secondary PCI Express
	Capabilities: [a90] Data Link Feature <?>
	Capabilities: [a9c] Physical Layer 16.0 GT/s <?>
	Capabilities: [edc] Lane Margining at the Receiver <?>
	Kernel driver in use: pcieport

00:06.2 PCI bridge: Intel Corporation Device 463d (rev 04) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 123, IOMMU group 4
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: [disabled]
	Memory behind bridge: 82200000-822fffff [size=1M]
	Prefetchable memory behind bridge: [disabled]
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Hewlett-Packard Company Device 897a
	Capabilities: [a0] Power Management version 3
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [220] Access Control Services
	Capabilities: [200] L1 PM Substates
	Capabilities: [150] Precision Time Measurement
	Capabilities: [280] Virtual Channel
	Capabilities: [a30] Secondary PCI Express
	Capabilities: [a90] Data Link Feature <?>
	Capabilities: [a9c] Physical Layer 16.0 GT/s <?>
	Capabilities: [edc] Lane Margining at the Receiver <?>
	Kernel driver in use: pcieport

00:07.0 PCI bridge: Intel Corporation Device 466e (rev 04) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 124, IOMMU group 5
	Bus: primary=00, secondary=03, subordinate=2c, sec-latency=0
	I/O behind bridge: 00004000-00004fff [size=4K]
	Memory behind bridge: 76000000-821fffff [size=194M]
	Prefetchable memory behind bridge: 0000006000000000-000000601bffffff [size=448M]
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Hewlett-Packard Company Device 897a
	Capabilities: [a0] Power Management version 3
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [220] Access Control Services
	Capabilities: [150] Precision Time Measurement
	Capabilities: [a00] Downstream Port Containment
	Capabilities: [a30] Secondary PCI Express
	Capabilities: [a90] Data Link Feature <?>
	Capabilities: [ba0] Physical Resizable BAR
	Kernel driver in use: pcieport

00:08.0 System peripheral: Intel Corporation Device 464f (rev 04)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: fast devsel, IRQ 255, IOMMU group 6
	Memory at 601d193000 (64-bit, non-prefetchable) [disabled] [size=4K]
	Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [a0] Vendor Specific Information: Len=14 <?>
	Capabilities: [dc] Power Management version 2
	Capabilities: [f0] PCI Advanced Features

00:0d.0 USB controller: Intel Corporation Device 461e (rev 04) (prog-if 30 [XHCI])
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: bus master, medium devsel, latency 0, IRQ 164, IOMMU group 7
	Memory at 601d170000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [70] Power Management version 2
	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
	Capabilities: [90] Vendor Specific Information: Len=14 <?>
	Capabilities: [b0] Vendor Specific Information: Len=00 <?>
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

00:0d.2 USB controller: Intel Corporation Device 463e (rev 04) (prog-if 40 [USB4 Host Interface])
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 7
	Memory at 601d100000 (64-bit, non-prefetchable) [size=256K]
	Memory at 601d192000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [80] Power Management version 3
	Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [a0] MSI-X: Enable+ Count=16 Masked-
	Kernel driver in use: thunderbolt
	Kernel modules: thunderbolt

00:14.0 USB controller: Intel Corporation Device 51ed (rev 01) (prog-if 30 [XHCI])
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: bus master, medium devsel, latency 0, IRQ 163, IOMMU group 8
	Memory at 601d160000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [70] Power Management version 2
	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
	Capabilities: [90] Vendor Specific Information: Len=14 <?>
	Capabilities: [b0] Vendor Specific Information: Len=00 <?>
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

00:14.2 RAM memory: Intel Corporation Device 51ef (rev 01)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: fast devsel, IOMMU group 8
	Memory at 601d188000 (64-bit, non-prefetchable) [disabled] [size=16K]
	Memory at 601d191000 (64-bit, non-prefetchable) [disabled] [size=4K]
	Capabilities: [80] Power Management version 3

00:14.3 Network controller: Intel Corporation Device 51f0 (rev 01)
	Subsystem: Intel Corporation Device 0090
	Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 9
	Memory at 601d184000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [c8] Power Management version 3
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [40] Express Root Complex Integrated Endpoint, MSI 00
	Capabilities: [80] MSI-X: Enable+ Count=16 Masked-
	Capabilities: [100] Latency Tolerance Reporting
	Capabilities: [164] Vendor Specific Information: ID=0010 Rev=0 Len=014 <?>
	Kernel driver in use: iwlwifi
	Kernel modules: iwlwifi

00:15.0 Serial bus controller [0c80]: Intel Corporation Device 51e8 (rev 01)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: bus master, fast devsel, latency 0, IRQ 27, IOMMU group 10
	Memory at 4017000000 (64-bit, non-prefetchable) [virtual] [size=4K]
	Capabilities: [80] Power Management version 3
	Capabilities: [90] Vendor Specific Information: Len=14 <?>
	Kernel driver in use: intel-lpss

00:16.0 Communication controller: Intel Corporation Device 51e0 (rev 01)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: bus master, fast devsel, latency 0, IRQ 135, IOMMU group 11
	Memory at 601d18f000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [50] Power Management version 3
	Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [a4] Vendor Specific Information: Len=14 <?>
	Kernel driver in use: mei_me
	Kernel modules: mei_me

00:16.3 Serial controller: Intel Corporation Device 51e3 (rev 01) (prog-if 02 [16550])
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: 66MHz, fast devsel, IRQ 19, IOMMU group 11
	I/O ports at 3060 [size=8]
	Memory at 82321000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [40] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [50] Power Management version 3
	Kernel driver in use: serial
	Kernel modules: 8250_pci

00:1e.0 Communication controller: Intel Corporation Device 51a8 (rev 01)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 12
	Memory at 4017001000 (64-bit, non-prefetchable) [virtual] [size=4K]
	Capabilities: [80] Power Management version 3
	Capabilities: [90] Vendor Specific Information: Len=14 <?>
	Kernel driver in use: intel-lpss

00:1e.2 Serial bus controller [0c80]: Intel Corporation Device 51aa (rev 01)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: bus master, fast devsel, latency 0, IRQ 36, IOMMU group 12
	Memory at 4017002000 (64-bit, non-prefetchable) [virtual] [size=4K]
	Capabilities: [80] Power Management version 3
	Capabilities: [90] Vendor Specific Information: Len=14 <?>
	Kernel driver in use: intel-lpss

00:1f.0 ISA bridge: Intel Corporation Device 5182 (rev 01)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: bus master, fast devsel, latency 0, IOMMU group 13

00:1f.3 Multimedia audio controller: Intel Corporation Device 51c8 (rev 01)
	DeviceName: Onboard Audio
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: fast devsel, IRQ 255, IOMMU group 13
	Memory at 601d180000 (64-bit, non-prefetchable) [disabled] [size=16K]
	Memory at 601d000000 (64-bit, non-prefetchable) [disabled] [size=1M]
	Capabilities: [50] Power Management version 3
	Capabilities: [80] Vendor Specific Information: Len=14 <?>
	Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
	Kernel modules: snd_hda_intel

00:1f.4 SMBus: Intel Corporation Device 51a3 (rev 01)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: medium devsel, IRQ 16, IOMMU group 13
	Memory at 601d18c000 (64-bit, non-prefetchable) [size=256]
	I/O ports at efa0 [size=32]
	Kernel driver in use: i801_smbus
	Kernel modules: i2c_i801

00:1f.5 Serial bus controller [0c80]: Intel Corporation Device 51a4 (rev 01)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: fast devsel, IOMMU group 13
	Memory at 74800000 (32-bit, non-prefetchable) [size=4K]

00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (16) I219-LM (rev 01)
	Subsystem: Hewlett-Packard Company Device 897a
	Flags: bus master, fast devsel, latency 0, IRQ 152, IOMMU group 13
	Memory at 82300000 (32-bit, non-prefetchable) [size=128K]
	Capabilities: [c8] Power Management version 3
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Kernel driver in use: e1000e
	Kernel modules: e1000e

02:00.0 Non-Volatile memory controller: Micron Technology Inc Device 5407 (prog-if 02 [NVM Express])
	Subsystem: Micron Technology Inc Device 0100
	Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 14
	Memory at 82200000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI-X: Enable+ Count=32 Masked-
	Capabilities: [60] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [2a0] Secondary PCI Express
	Capabilities: [2d0] Latency Tolerance Reporting
	Capabilities: [320] Data Link Feature <?>
	Capabilities: [330] Physical Layer 16.0 GT/s <?>
	Capabilities: [360] Lane Margining at the Receiver <?>
	Capabilities: [700] L1 PM Substates
	Kernel driver in use: nvme

03:00.0 PCI bridge: Intel Corporation Device 0b26 (rev 03) (prog-if 00 [Normal decode])
	Physical Slot: 3
	Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 15
	Bus: primary=03, secondary=04, subordinate=2c, sec-latency=0
	I/O behind bridge: 00004000-00004fff [size=4K]
	Memory behind bridge: 76000000-821fffff [size=194M]
	Prefetchable memory behind bridge: 0000006000000000-000000601bffffff [size=448M]
	Capabilities: [80] Power Management version 3
	Capabilities: [88] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [ac] Subsystem: Intel Corporation Device 0000
	Capabilities: [c0] Express Upstream Port, MSI 00
	Capabilities: [50] Capability ID 0x15 [0000]
	Capabilities: [100] Device Serial Number ad-65-8c-6a-36-23-04-00
	Capabilities: [200] Advanced Error Reporting
	Capabilities: [300] Virtual Channel
	Capabilities: [400] Power Budgeting <?>
	Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
	Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
	Capabilities: [700] Secondary PCI Express
	Capabilities: [800] Latency Tolerance Reporting
	Capabilities: [a00] L1 PM Substates
	Capabilities: [b00] Precision Time Measurement
	Kernel driver in use: pcieport

04:00.0 PCI bridge: Intel Corporation Device 0b26 (rev 03) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 165, IOMMU group 16
	Bus: primary=04, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 00004000-00004fff [size=4K]
	Memory behind bridge: 76000000-760fffff [size=1M]
	Prefetchable memory behind bridge: 0000006000000000-00000060000fffff [size=1M]
	Capabilities: [80] Power Management version 3
	Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [ac] Subsystem: Intel Corporation Device 0000
	Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
	Capabilities: [50] Capability ID 0x15 [0000]
	Capabilities: [100] Device Serial Number ad-65-8c-6a-36-23-04-00
	Capabilities: [200] Advanced Error Reporting
	Capabilities: [300] Virtual Channel
	Capabilities: [400] Power Budgeting <?>
	Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
	Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
	Capabilities: [700] Secondary PCI Express
	Capabilities: [900] Access Control Services
	Capabilities: [c00] Designated Vendor-Specific: Vendor=8086 ID=0006 Rev=0 Len=16 <?>
	Kernel driver in use: pcieport

04:01.0 PCI bridge: Intel Corporation Device 0b26 (rev 03) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 166, IOMMU group 17
	Bus: primary=04, secondary=06, subordinate=12, sec-latency=0
	I/O behind bridge: [disabled]
	Memory behind bridge: 76100000-7a0fffff [size=64M]
	Prefetchable memory behind bridge: 0000006000100000-00000060095fffff [size=149M]
	Capabilities: [80] Power Management version 3
	Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [ac] Subsystem: Intel Corporation Device 0000
	Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
	Capabilities: [50] Capability ID 0x15 [0000]
	Capabilities: [100] Device Serial Number ad-65-8c-6a-36-23-04-00
	Capabilities: [200] Advanced Error Reporting
	Capabilities: [300] Virtual Channel
	Capabilities: [400] Power Budgeting <?>
	Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
	Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
	Capabilities: [700] Secondary PCI Express
	Capabilities: [900] Access Control Services
	Capabilities: [c00] Designated Vendor-Specific: Vendor=8086 ID=0006 Rev=0 Len=16 <?>
	Kernel driver in use: pcieport

04:02.0 PCI bridge: Intel Corporation Device 0b26 (rev 03) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 167, IOMMU group 18
	Bus: primary=04, secondary=13, subordinate=1f, sec-latency=0
	I/O behind bridge: [disabled]
	Memory behind bridge: 7a100000-7e0fffff [size=64M]
	Prefetchable memory behind bridge: 0000006009600000-0000006012afffff [size=149M]
	Capabilities: [80] Power Management version 3
	Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [ac] Subsystem: Intel Corporation Device 0000
	Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
	Capabilities: [50] Capability ID 0x15 [0000]
	Capabilities: [100] Device Serial Number ad-65-8c-6a-36-23-04-00
	Capabilities: [200] Advanced Error Reporting
	Capabilities: [300] Virtual Channel
	Capabilities: [400] Power Budgeting <?>
	Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
	Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
	Capabilities: [700] Secondary PCI Express
	Capabilities: [900] Access Control Services
	Capabilities: [c00] Designated Vendor-Specific: Vendor=8086 ID=0006 Rev=0 Len=16 <?>
	Kernel driver in use: pcieport

04:03.0 PCI bridge: Intel Corporation Device 0b26 (rev 03) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 168, IOMMU group 19
	Bus: primary=04, secondary=20, subordinate=2b, sec-latency=0
	I/O behind bridge: [disabled]
	Memory behind bridge: 7e100000-81ffffff [size=63M]
	Prefetchable memory behind bridge: 0000006012b00000-000000601befffff [size=148M]
	Capabilities: [80] Power Management version 3
	Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [ac] Subsystem: Intel Corporation Device 0000
	Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
	Capabilities: [50] Capability ID 0x15 [0000]
	Capabilities: [100] Device Serial Number ad-65-8c-6a-36-23-04-00
	Capabilities: [200] Advanced Error Reporting
	Capabilities: [300] Virtual Channel
	Capabilities: [400] Power Budgeting <?>
	Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
	Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
	Capabilities: [700] Secondary PCI Express
	Capabilities: [900] Access Control Services
	Capabilities: [c00] Designated Vendor-Specific: Vendor=8086 ID=0006 Rev=0 Len=16 <?>
	Kernel driver in use: pcieport

04:04.0 PCI bridge: Intel Corporation Device 0b26 (rev 03) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 169, IOMMU group 20
	Bus: primary=04, secondary=2c, subordinate=2c, sec-latency=0
	I/O behind bridge: [disabled]
	Memory behind bridge: 82000000-821fffff [size=2M]
	Prefetchable memory behind bridge: 000000601bf00000-000000601bffffff [size=1M]
	Capabilities: [80] Power Management version 3
	Capabilities: [88] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [ac] Subsystem: Intel Corporation Device 0000
	Capabilities: [c0] Express Downstream Port (Slot+), MSI 00
	Capabilities: [50] Capability ID 0x15 [0000]
	Capabilities: [100] Device Serial Number ad-65-8c-6a-36-23-04-00
	Capabilities: [200] Advanced Error Reporting
	Capabilities: [300] Virtual Channel
	Capabilities: [400] Power Budgeting <?>
	Capabilities: [500] Vendor Specific Information: ID=1234 Rev=1 Len=100 <?>
	Capabilities: [600] Vendor Specific Information: ID=8086 Rev=2 Len=04c <?>
	Capabilities: [700] Secondary PCI Express
	Capabilities: [900] Access Control Services
	Capabilities: [c00] Designated Vendor-Specific: Vendor=8086 ID=0006 Rev=0 Len=16 <?>
	Kernel driver in use: pcieport

2c:00.0 Ethernet controller: Intel Corporation Ethernet Controller (2) I225-LMvP (rev 03)
	Subsystem: Hewlett-Packard Company Device 87b9
	Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 21
	Memory at 82000000 (32-bit, non-prefetchable) [size=1M]
	Memory at 82100000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 84-3a-5b-ff-ff-6c-0d-a5
	Capabilities: [1c0] Latency Tolerance Reporting
	Capabilities: [1f0] Precision Time Measurement
	Capabilities: [1e0] L1 PM Substates
	Kernel driver in use: igc
	Kernel modules: igc

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi,

On Tue, Jan 03, 2023 at 01:14:09PM +0100, Christian Schaubschläger wrote:
> >
> > 0. Remove the hack patch if not already.
> >
> > Steps 1-3 same as before.
> >
> > 4. Disable runtime PM from the PCIe root ports:
> >
> >   # echo on > /sys/bus/pci/devices/0000:00:07.0/power/control
> >   # echo on > /sys/bus/pci/devices/0000:00:07.1/power/control
> >   # echo on > /sys/bus/pci/devices/0000:00:07.2/power/control
> >   # echo on > /sys/bus/pci/devices/0000:00:07.3/power/control
> >
> > (or how many Thunderbolt PCIE root ports you have, disable it from all of them).
> >
> > 5. De-authorize the PCIe tunnel
> >
> >   # echo 0 > /sys/bus/thunderbolt/devices/.../authorized
> >
> > 6. Soft reboot
> >
> > Does this change anything or the behavior is the same?
> No, that doesn't change anything; though I'm not sure how many
> Thunderbolt PCIe root ports I have; I guess only one: 0000:00:07.0 -
> please have a look into the attached lspci output.
> 
> dmesg after the above two steps:
> 
> [   91.655320] pcieport 0000:00:07.0: pciehp: Slot(3): Link Down
> [   91.657353] pcieport 0000:00:07.0: pciehp: Slot(3): Card not present
> [   91.659525] igc 0000:2c:00.0 eth1: PHC removed
> [   91.786204] pcieport 0000:04:03.0: Unable to change power state from D3hot to D0, device inaccessible
> [   91.788645] pcieport 0000:04:03.0: Runtime PM usage count underflow!
> [   91.790766] pcieport 0000:04:02.0: Unable to change power state from D3hot to D0, device inaccessible
> [   91.793156] pcieport 0000:04:02.0: Runtime PM usage count underflow!
> [   91.795341] pcieport 0000:04:01.0: Unable to change power state from D3hot to D0, device inaccessible
> [   91.797797] pcieport 0000:04:01.0: Runtime PM usage count underflow!
> [   91.800031] pcieport 0000:04:00.0: Unable to change power state from D3hot to D0, device inaccessible
> [   91.802671] pci_bus 0000:05: busn_res: [bus 05] is released
> [   91.805004] pci 0000:04:00.0: Removing from iommu group 16
> [   91.807261] pci_bus 0000:06: busn_res: [bus 06-12] is released
> [   91.809582] pci 0000:04:01.0: Removing from iommu group 17
> [   91.811925] pci_bus 0000:13: busn_res: [bus 13-1f] is released
> [   91.814303] pci 0000:04:02.0: Removing from iommu group 18
> [   91.816604] pci_bus 0000:20: busn_res: [bus 20-2b] is released
> [   91.818985] pci 0000:04:03.0: Removing from iommu group 19
> [   91.821412] pci 0000:2c:00.0: Removing from iommu group 21
> [   91.823875] pci_bus 0000:2c: busn_res: [bus 2c] is released
> [   91.826241] pci 0000:04:04.0: Removing from iommu group 20
> [   91.828592] pci_bus 0000:04: busn_res: [bus 04-2c] is released
> [   91.831018] pci 0000:03:00.0: Removing from iommu group 15
> 
> Still, no network interface after the reboot in the firmware...

Okay you have only one Thunderbolt PCIe root port just as you correctly
guessed (07.0). At this point I'm running out of ideas what to check or
try. We have seen similar but only when there is no battery on the
device which is not your case, I suppose. In this case the initial USB4
link state is kind of "open" so it may come up or not.

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi,


> Okay you have only one Thunderbolt PCIe root port just as you correctly
> guessed (07.0). At this point I'm running out of ideas what to check or
> try. We have seen similar but only when there is no battery on the
> device which is not your case, I suppose. In this case the initial USB4
> link state is kind of "open" so it may come up or not.

Hm, maybe it _is_ a firmware issue then? Would it, in your opinion, make any sense to confront HP with that issue? Not that I'd have any contacts to HP, but just thinking about options...

Earlier you said that with Windows 11 the behaviour might be the same as in Linux. I'll try that next week (out of office until then). If Win 11 indeed has the same problem, then I guess someone at HP or wherever will address this sooner or later. I mean, the requirement for a working network device/stack after a reboot from an OS is not that unusual I'd say...

Thanks,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi,

On Tue, Jan 03, 2023 at 06:50:02PM +0100, Christian Schaubschläger wrote:
> Hi,
> 
> 
> > Okay you have only one Thunderbolt PCIe root port just as you correctly
> > guessed (07.0). At this point I'm running out of ideas what to check or
> > try. We have seen similar but only when there is no battery on the
> > device which is not your case, I suppose. In this case the initial USB4
> > link state is kind of "open" so it may come up or not.
> 
> Hm, maybe it _is_ a firmware issue then? Would it, in your opinion,
> make any sense to confront HP with that issue? Not that I'd have any
> contacts to HP, but just thinking about options...

It may also be a "feature" nobody considered ;-)

> Earlier you said that with Windows 11 the behaviour might be the same
> as in Linux. I'll try that next week (out of office until then). If
> Win 11 indeed has the same problem, then I guess someone at HP or
> wherever will address this sooner or later. I mean, the requirement
> for a working network device/stack after a reboot from an OS is not
> that unusual I'd say...

I agree. If it turns out that Windows 11 works wrt. this (please check
that it is using Windows "inbox" driver for TBT) then we need to figure
out what it is doing differently.

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hello again,

>> Earlier you said that with Windows 11 the behaviour might be the same
>> as in Linux. I'll try that next week (out of office until then). If
>> Win 11 indeed has the same problem, then I guess someone at HP or
>> wherever will address this sooner or later. I mean, the requirement
>> for a working network device/stack after a reboot from an OS is not
>> that unusual I'd say...
> I agree. If it turns out that Windows 11 works wrt. this (please check
> that it is using Windows "inbox" driver for TBT) then we need to figure
> out what it is doing differently.

So today I've installed Windows 11 22H2 on the EliteBook, and I can report this:

* The thunderbolt controller is run by a driver from Microsoft (in the Win 11 device manager it's no longer called "Thunderbolt Controller", but "USB4 Host Router".
* I think there isn't even a thunderbolt driver from Intel for Windows 11 - at least on the HP support site for the EliteBook they don't have one (but they do have an Intel driver for Windows 10).
* And regarding functionality: after rebooting Windows 11, the network device is visible in the firmware, and working fine...

Any more test I could make?

Thanks,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi,

On Mon, Jan 09, 2023 at 01:17:30PM +0100, Christian Schaubschläger wrote:
> Hello again,
> 
> >> Earlier you said that with Windows 11 the behaviour might be the same
> >> as in Linux. I'll try that next week (out of office until then). If
> >> Win 11 indeed has the same problem, then I guess someone at HP or
> >> wherever will address this sooner or later. I mean, the requirement
> >> for a working network device/stack after a reboot from an OS is not
> >> that unusual I'd say...
> > I agree. If it turns out that Windows 11 works wrt. this (please check
> > that it is using Windows "inbox" driver for TBT) then we need to figure
> > out what it is doing differently.
> 
> So today I've installed Windows 11 22H2 on the EliteBook, and I can report this:
> 
> * The thunderbolt controller is run by a driver from Microsoft (in the Win 11 device manager it's no longer called "Thunderbolt Controller", but "USB4 Host Router".
> * I think there isn't even a thunderbolt driver from Intel for Windows 11 - at least on the HP support site for the EliteBook they don't have one (but they do have an Intel driver for Windows 10).

Yes, Windows 11 has "native" support so there is no need for a vendor
driver.

> * And regarding functionality: after rebooting Windows 11, the network device is visible in the firmware, and working fine...
> 
> Any more test I could make?

Okay it could be that Windows CM is doing something during the reboot,
like resetting the ports. Can you try the below hack patch and see if
that makes it work the same in Linux?

diff --git a/drivers/thunderbolt/tb.c b/drivers/thunderbolt/tb.c
index 1711dc19b1e2..584d839c7943 100644
--- a/drivers/thunderbolt/tb.c
+++ b/drivers/thunderbolt/tb.c
@@ -1977,22 +1977,24 @@ static void tb_stop(struct tb *tb)
 {
 	struct tb_cm *tcm = tb_priv(tb);
 	struct tb_tunnel *tunnel;
+	struct tb_port *port;
 	struct tb_tunnel *n;
 
 	cancel_delayed_work(&tcm->remove_work);
 	/* tunnels are only present after everything has been initialized */
 	list_for_each_entry_safe(tunnel, n, &tcm->tunnel_list, list) {
-		/*
-		 * DMA tunnels require the driver to be functional so we
-		 * tear them down. Other protocol tunnels can be left
-		 * intact.
-		 */
-		if (tb_tunnel_is_dma(tunnel))
-			tb_tunnel_deactivate(tunnel);
+		tb_tunnel_deactivate(tunnel);
 		tb_tunnel_free(tunnel);
 	}
-	tb_switch_remove(tb->root_switch);
+
 	tcm->hotplug_active = false; /* signal tb_handle_hotplug to quit */
+
+	tb_switch_for_each_port(tb->root_switch, port) {
+		if (tb_port_is_null(port))
+			usb4_port_reset(port);
+	}
+
+	tb_switch_remove(tb->root_switch);
 }
 
 static int tb_scan_finalize_switch(struct device *dev, void *data)
diff --git a/drivers/thunderbolt/tb.h b/drivers/thunderbolt/tb.h
index 3095c206ea8a..c4cf8eee6771 100644
--- a/drivers/thunderbolt/tb.h
+++ b/drivers/thunderbolt/tb.h
@@ -1243,6 +1243,7 @@ void usb4_switch_remove_ports(struct tb_switch *sw);
 
 int usb4_port_unlock(struct tb_port *port);
 int usb4_port_hotplug_enable(struct tb_port *port);
+int usb4_port_reset(struct tb_port *port);
 int usb4_port_configure(struct tb_port *port);
 void usb4_port_unconfigure(struct tb_port *port);
 int usb4_port_configure_xdomain(struct tb_port *port, struct tb_xdomain *xd);
diff --git a/drivers/thunderbolt/tb_regs.h b/drivers/thunderbolt/tb_regs.h
index 4ecbe811ea43..8797a7c0316e 100644
--- a/drivers/thunderbolt/tb_regs.h
+++ b/drivers/thunderbolt/tb_regs.h
@@ -384,6 +384,7 @@ struct tb_regs_port_header {
 #define PORT_CS_18_WODS				BIT(17)
 #define PORT_CS_18_WOU4S			BIT(18)
 #define PORT_CS_19				0x13
+#define PORT_CS_19_DPR				BIT(0)
 #define PORT_CS_19_PC				BIT(3)
 #define PORT_CS_19_PID				BIT(4)
 #define PORT_CS_19_WOC				BIT(16)
diff --git a/drivers/thunderbolt/usb4.c b/drivers/thunderbolt/usb4.c
index b4fbc692ffc3..6a632ace4530 100644
--- a/drivers/thunderbolt/usb4.c
+++ b/drivers/thunderbolt/usb4.c
@@ -1111,6 +1111,39 @@ int usb4_port_hotplug_enable(struct tb_port *port)
 	return tb_port_write(port, &val, TB_CFG_PORT, ADP_CS_5, 1);
 }
 
+int usb4_port_reset(struct tb_port *port)
+{
+	int ret;
+	u32 val;
+
+	if (!port->cap_usb4)
+		return -EINVAL;
+
+	ret = tb_port_read(port, &val, TB_CFG_PORT,
+			   port->cap_usb4 + PORT_CS_19, 1);
+	if (ret)
+		return ret;
+
+	val |= PORT_CS_19_DPR;
+
+	ret = tb_port_write(port, &val, TB_CFG_PORT,
+			    port->cap_usb4 + PORT_CS_19, 1);
+	if (ret)
+		return ret;
+
+	msleep(10);
+
+	ret = tb_port_read(port, &val, TB_CFG_PORT,
+			   port->cap_usb4 + PORT_CS_19, 1);
+	if (ret)
+		return ret;
+
+	val &= ~PORT_CS_19_DPR;
+
+	return tb_port_write(port, &val, TB_CFG_PORT,
+			     port->cap_usb4 + PORT_CS_19, 1);
+}
+
 static int usb4_port_set_configured(struct tb_port *port, bool configured)
 {
 	int ret;

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi,

> Okay it could be that Windows CM is doing something during the reboot,
> like resetting the ports. Can you try the below hack patch and see if
> that makes it work the same in Linux?

I'm afraid that doesn't make it work. I can see that usb4_port_reset is called 4 times when the machine is rebooted, but still the net device isn't there afterwards.

Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi,

On Mon, Jan 09, 2023 at 03:21:15PM +0100, Christian Schaubschläger wrote:
> Hi,
> 
> > Okay it could be that Windows CM is doing something during the reboot,
> > like resetting the ports. Can you try the below hack patch and see if
> > that makes it work the same in Linux?
> 
> I'm afraid that doesn't make it work. I can see that usb4_port_reset
> is called 4 times when the machine is rebooted, but still the net
> device isn't there afterwards.

Okay thanks for checking. I'm discussing with our Windows folks if there
is something done differently in Windows side. They do similar reset in
certain flows but I'm not sure if it is exactly on "reboot" and it is
not clear whether Windows and Linux "reboot" in general is doing the
same thing either.

In the meantime, did we try this already?

0. Drop all kernel changes
1. Boot the system up, nothing connected
2. Plug in the dock
3. Check that it works (lspci etc.)
4. Unload the Thunderbolt driver

  # rmmod thunderbolt

5. Reboot the system

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi,
> Okay thanks for checking. I'm discussing with our Windows folks if there
> is something done differently in Windows side. They do similar reset in
> certain flows but I'm not sure if it is exactly on "reboot" and it is
> not clear whether Windows and Linux "reboot" in general is doing the
> same thing either.
>
> In the meantime, did we try this already?
>
> 0. Drop all kernel changes
> 1. Boot the system up, nothing connected
> 2. Plug in the dock
> 3. Check that it works (lspci etc.)
> 4. Unload the Thunderbolt driver
>
>   # rmmod thunderbolt
>
> 5. Reboot the system

I think we did not try this one so far. But I did now -> same result...

What I _did_ find out is this: if I disable ACPI on the kernel command line (acpi=off), things work fine. I can even boot with the dock unplugged, then plug it when linux is up, and it works in linux, and also after the reboot in the firmware!

Of course switching off ACPI is not a real option... but maybe this brings some light into the dark...

Regards,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

On Wed, Jan 11, 2023 at 11:38:02AM +0100, Christian Schaubschläger wrote:
> Hi,
> > Okay thanks for checking. I'm discussing with our Windows folks if there
> > is something done differently in Windows side. They do similar reset in
> > certain flows but I'm not sure if it is exactly on "reboot" and it is
> > not clear whether Windows and Linux "reboot" in general is doing the
> > same thing either.
> >
> > In the meantime, did we try this already?
> >
> > 0. Drop all kernel changes
> > 1. Boot the system up, nothing connected
> > 2. Plug in the dock
> > 3. Check that it works (lspci etc.)
> > 4. Unload the Thunderbolt driver
> >
> >   # rmmod thunderbolt
> >
> > 5. Reboot the system
> 
> I think we did not try this one so far. But I did now -> same result...

Okay :(

> What I _did_ find out is this: if I disable ACPI on the kernel command
> line (acpi=off), things work fine. I can even boot with the dock
> unplugged, then plug it when linux is up, and it works in linux, and
> also after the reboot in the firmware!
> 
> Of course switching off ACPI is not a real option... but maybe this
> brings some light into the dark...

It pretty much disables a lot of functionality so I don't think it helps
here, unfortunately.

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi Mika,

>> What I _did_ find out is this: if I disable ACPI on the kernel command
>> line (acpi=off), things work fine. I can even boot with the dock
>> unplugged, then plug it when linux is up, and it works in linux, and
>> also after the reboot in the firmware!
>>
>> Of course switching off ACPI is not a real option... but maybe this
>> brings some light into the dark...
> It pretty much disables a lot of functionality so I don't think it helps
> here, unfortunately.

I may have some more information on this issue, though I'm not sure if this will help... the thing is this:

this week I got a brandnew Lenovo Thinkpad Yoga X1 along with a Lenovo Thunderbolt 4 Workstation Dock, all thunderbolt 4. And at first it seemed to behave exactly like the HP combo; namely after booting Ubuntu 22.04 (which ships with a  5.15 kernel) the dock is gone in the firmware after the reboot, and subsequently later in Windows. Same behaviour with 5.18.9. And to my big surprise, with 5.18.11 the issue is gone! I've bisected it down to this commit:

8beb71759cc8fddd937cadf9ec482e524d4f0f1c is the first fixed commit
commit 8beb71759cc8fddd937cadf9ec482e524d4f0f1c
Author: Pierre Gondois <pierre.gondois@arm.com>
Date:   Wed May 18 11:08:58 2022 +0200

    ACPI: bus: Set CPPC _OSC bits for all and when CPPC_LIB is supported
    
    [ Upstream commit 72f2ecb7ece7c1d89758d4929d98e95d95fe7199 ]
    
    The _OSC method allows the OS and firmware to communicate about
    supported features/capabitlities. It also allows the OS to take
    control of some features.
    
    In ACPI 6.4, s6.2.11.2 Platform-Wide OSPM Capabilities, the CPPC
    (resp. v2) bit should be set by the OS if it 'supports controlling
    processor performance via the interfaces described in the _CPC
    object'.
    
    The OS supports CPPC and parses the _CPC object only if
    CONFIG_ACPI_CPPC_LIB is set. Replace the x86 specific
    boot_cpu_has(X86_FEATURE_HWP) dynamic check with an arch
    generic CONFIG_ACPI_CPPC_LIB build-time check.
    
    Note:
    CONFIG_X86_INTEL_PSTATE selects CONFIG_ACPI_CPPC_LIB.
    
    Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
    Reviewed-by: Sudeep Holla <sudeep.holla@arm.com>
    Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
    Signed-off-by: Sasha Levin <sashal@kernel.org>

 drivers/acpi/bus.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

I've no idea why this solves the issue on the Lenovo, and obviously it doesn't solve it on the HP, but maybe this rings a bell somewhere? (Note: in my woking kernel config (on the Lenovo) I have neither CONFIG_X86_INTEL_PSTATE nor CONFIG_ACPI_CPPC_LIB set...)

Previously you said you'd talk with your Windows folks about this; any news from there?

Thanks and best regards,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi,

On Thu, Jan 19, 2023 at 10:41:26AM +0100, Christian Schaubschläger wrote:
> Hi Mika,
> 
> >> What I _did_ find out is this: if I disable ACPI on the kernel command
> >> line (acpi=off), things work fine. I can even boot with the dock
> >> unplugged, then plug it when linux is up, and it works in linux, and
> >> also after the reboot in the firmware!
> >>
> >> Of course switching off ACPI is not a real option... but maybe this
> >> brings some light into the dark...
> > It pretty much disables a lot of functionality so I don't think it helps
> > here, unfortunately.
> 
> I may have some more information on this issue, though I'm not sure if this will help... the thing is this:
> 
> this week I got a brandnew Lenovo Thinkpad Yoga X1 along with a Lenovo Thunderbolt 4 Workstation Dock, all thunderbolt 4. And at first it seemed to behave exactly like the HP combo; namely after booting Ubuntu 22.04 (which ships with a  5.15 kernel) the dock is gone in the firmware after the reboot, and subsequently later in Windows. Same behaviour with 5.18.9. And to my big surprise, with 5.18.11 the issue is gone! I've bisected it down to this commit:
> 
> 8beb71759cc8fddd937cadf9ec482e524d4f0f1c is the first fixed commit
> commit 8beb71759cc8fddd937cadf9ec482e524d4f0f1c
> Author: Pierre Gondois <pierre.gondois@arm.com>
> Date:   Wed May 18 11:08:58 2022 +0200
> 
>     ACPI: bus: Set CPPC _OSC bits for all and when CPPC_LIB is supported
>     
>     [ Upstream commit 72f2ecb7ece7c1d89758d4929d98e95d95fe7199 ]
>     
>     The _OSC method allows the OS and firmware to communicate about
>     supported features/capabitlities. It also allows the OS to take
>     control of some features.
>     
>     In ACPI 6.4, s6.2.11.2 Platform-Wide OSPM Capabilities, the CPPC
>     (resp. v2) bit should be set by the OS if it 'supports controlling
>     processor performance via the interfaces described in the _CPC
>     object'.
>     
>     The OS supports CPPC and parses the _CPC object only if
>     CONFIG_ACPI_CPPC_LIB is set. Replace the x86 specific
>     boot_cpu_has(X86_FEATURE_HWP) dynamic check with an arch
>     generic CONFIG_ACPI_CPPC_LIB build-time check.
>     
>     Note:
>     CONFIG_X86_INTEL_PSTATE selects CONFIG_ACPI_CPPC_LIB.
>     
>     Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
>     Reviewed-by: Sudeep Holla <sudeep.holla@arm.com>
>     Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
>     Signed-off-by: Sasha Levin <sashal@kernel.org>
> 
>  drivers/acpi/bus.c | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
> 
> I've no idea why this solves the issue on the Lenovo, and obviously it doesn't solve it on the HP, but maybe this rings a bell somewhere? (Note: in my woking kernel config (on the Lenovo) I have neither CONFIG_X86_INTEL_PSTATE nor CONFIG_ACPI_CPPC_LIB set...)

Can you check if with this commit reverted does Thunderbolt use software
or firmware connection manager? (You can see this in the logs when
thunderbolt.dyndbg=+p is in the command line).

> Previously you said you'd talk with your Windows folks about this; any
> news from there?

I've talked to them and still in talks with the UEFI folks but the
current undestanding is that Windows does not do anything special when
the system is rebooted (so equal to what Linux does). There is one
"development" system in Israel lab that should be pretty similar to what
the HP system of yours is but the person who was going to try to
reproduce is in sick leave now.

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi,
>> 8beb71759cc8fddd937cadf9ec482e524d4f0f1c is the first fixed commit
>> commit 8beb71759cc8fddd937cadf9ec482e524d4f0f1c
>> Author: Pierre Gondois <pierre.gondois@arm.com>
>> Date:   Wed May 18 11:08:58 2022 +0200
>>
>>     ACPI: bus: Set CPPC _OSC bits for all and when CPPC_LIB is supported
>>     
>>     [ Upstream commit 72f2ecb7ece7c1d89758d4929d98e95d95fe7199 ]
>>     
>>     The _OSC method allows the OS and firmware to communicate about
>>     supported features/capabitlities. It also allows the OS to take
>>     control of some features.
>>     
>>     In ACPI 6.4, s6.2.11.2 Platform-Wide OSPM Capabilities, the CPPC
>>     (resp. v2) bit should be set by the OS if it 'supports controlling
>>     processor performance via the interfaces described in the _CPC
>>     object'.
>>     
>>     The OS supports CPPC and parses the _CPC object only if
>>     CONFIG_ACPI_CPPC_LIB is set. Replace the x86 specific
>>     boot_cpu_has(X86_FEATURE_HWP) dynamic check with an arch
>>     generic CONFIG_ACPI_CPPC_LIB build-time check.
>>     
>>     Note:
>>     CONFIG_X86_INTEL_PSTATE selects CONFIG_ACPI_CPPC_LIB.
>>     
>>     Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
>>     Reviewed-by: Sudeep Holla <sudeep.holla@arm.com>
>>     Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
>>     Signed-off-by: Sasha Levin <sashal@kernel.org>
>>
>>  drivers/acpi/bus.c | 16 ++++++++--------
>>  1 file changed, 8 insertions(+), 8 deletions(-)
>>
>> I've no idea why this solves the issue on the Lenovo, and obviously it doesn't solve it on the HP, but maybe this rings a bell somewhere? (Note: in my woking kernel config (on the Lenovo) I have neither CONFIG_X86_INTEL_PSTATE nor CONFIG_ACPI_CPPC_LIB set...)
> Can you check if with this commit reverted does Thunderbolt use software
> or firmware connection manager? (You can see this in the logs when
> thunderbolt.dyndbg=+p is in the command line).
>
You're right! With this commit reverted it uses the software connection manager, with the commit applied it uses the firmware connection manager.
>> Previously you said you'd talk with your Windows folks about this; any
>> news from there?
> I've talked to them and still in talks with the UEFI folks but the
> current undestanding is that Windows does not do anything special when
> the system is rebooted (so equal to what Linux does). There is one
> "development" system in Israel lab that should be pretty similar to what
> the HP system of yours is but the person who was going to try to
> reproduce is in sick leave now.
Ok...

Regards,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

On Thu, Jan 19, 2023 at 03:37:52PM +0100, Christian Schaubschläger wrote:
> Hi,
> >> 8beb71759cc8fddd937cadf9ec482e524d4f0f1c is the first fixed commit
> >> commit 8beb71759cc8fddd937cadf9ec482e524d4f0f1c
> >> Author: Pierre Gondois <pierre.gondois@arm.com>
> >> Date:   Wed May 18 11:08:58 2022 +0200
> >>
> >>     ACPI: bus: Set CPPC _OSC bits for all and when CPPC_LIB is supported
> >>     
> >>     [ Upstream commit 72f2ecb7ece7c1d89758d4929d98e95d95fe7199 ]
> >>     
> >>     The _OSC method allows the OS and firmware to communicate about
> >>     supported features/capabitlities. It also allows the OS to take
> >>     control of some features.
> >>     
> >>     In ACPI 6.4, s6.2.11.2 Platform-Wide OSPM Capabilities, the CPPC
> >>     (resp. v2) bit should be set by the OS if it 'supports controlling
> >>     processor performance via the interfaces described in the _CPC
> >>     object'.
> >>     
> >>     The OS supports CPPC and parses the _CPC object only if
> >>     CONFIG_ACPI_CPPC_LIB is set. Replace the x86 specific
> >>     boot_cpu_has(X86_FEATURE_HWP) dynamic check with an arch
> >>     generic CONFIG_ACPI_CPPC_LIB build-time check.
> >>     
> >>     Note:
> >>     CONFIG_X86_INTEL_PSTATE selects CONFIG_ACPI_CPPC_LIB.
> >>     
> >>     Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
> >>     Reviewed-by: Sudeep Holla <sudeep.holla@arm.com>
> >>     Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
> >>     Signed-off-by: Sasha Levin <sashal@kernel.org>
> >>
> >>  drivers/acpi/bus.c | 16 ++++++++--------
> >>  1 file changed, 8 insertions(+), 8 deletions(-)
> >>
> >> I've no idea why this solves the issue on the Lenovo, and obviously it doesn't solve it on the HP, but maybe this rings a bell somewhere? (Note: in my woking kernel config (on the Lenovo) I have neither CONFIG_X86_INTEL_PSTATE nor CONFIG_ACPI_CPPC_LIB set...)
> > Can you check if with this commit reverted does Thunderbolt use software
> > or firmware connection manager? (You can see this in the logs when
> > thunderbolt.dyndbg=+p is in the command line).
> >
> You're right! With this commit reverted it uses the software connection manager, with the commit applied it uses the firmware connection manager.

Hmm, so reverting this commit enables software connection manager and
makes the Lenovo system work? Are you sure? that's completely opposite
how the HP system behaves ;-)

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi,
>> You're right! With this commit reverted it uses the software connection manager, with the commit applied it uses the firmware connection manager.
> Hmm, so reverting this commit enables software connection manager and
> makes the Lenovo system work? Are you sure? that's completely opposite
> how the HP system behaves ;-)

No, with the software CM active it does _not_ work! The patch introduced in 5.18.11 somehow prevents the software CM to be used...

5.18.9, without the comit:

bash-5.0# uname -r
5.18.9-superschaf-uefi-bisect
bash-5.0# dmesg |grep manager
[    2.028144] thunderbolt 0000:00:0d.2: using software connection manager
[    2.220380] thunderbolt 0000:00:0d.3: using software connection manager

and 5.18.11, including the commit:

bash-5.0# uname -r
5.18.11-superschaf-uefi-bisect
bash-5.0# dmesg | grep manager
[    1.827993] thunderbolt 0000:00:0d.2: using firmware connection manager
[    2.113706] thunderbolt 0000:00:0d.3: using firmware connection manager

On the HP the software CM is used always (unless I run a kernel with CONFIG_USB4=n).

Regards,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

On Fri, Jan 20, 2023 at 12:19:18PM +0100, Christian Schaubschläger wrote:
> Hi,
> >> You're right! With this commit reverted it uses the software connection manager, with the commit applied it uses the firmware connection manager.
> > Hmm, so reverting this commit enables software connection manager and
> > makes the Lenovo system work? Are you sure? that's completely opposite
> > how the HP system behaves ;-)
> 
> No, with the software CM active it does _not_ work! The patch
> introduced in 5.18.11 somehow prevents the software CM to be used...
> 
> 5.18.9, without the comit:
> 
> bash-5.0# uname -r
> 5.18.9-superschaf-uefi-bisect
> bash-5.0# dmesg |grep manager
> [    2.028144] thunderbolt 0000:00:0d.2: using software connection manager
> [    2.220380] thunderbolt 0000:00:0d.3: using software connection manager
> 
> and 5.18.11, including the commit:
> 
> bash-5.0# uname -r
> 5.18.11-superschaf-uefi-bisect
> bash-5.0# dmesg | grep manager
> [    1.827993] thunderbolt 0000:00:0d.2: using firmware connection manager
> [    2.113706] thunderbolt 0000:00:0d.3: using firmware connection manager
> 
> On the HP the software CM is used always (unless I run a kernel with CONFIG_USB4=n).

Okay then it is consistent. Thanks for confirming.

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi Mika,

>>> Previously you said you'd talk with your Windows folks about this; any
>>> news from there?
>> I've talked to them and still in talks with the UEFI folks but the
>> current undestanding is that Windows does not do anything special when
>> the system is rebooted (so equal to what Linux does). There is one
>> "development" system in Israel lab that should be pretty similar to what
>> the HP system of yours is but the person who was going to try to
>> reproduce is in sick leave now.
> Ok...
I was just wondering if there are already any news from your lab in Israel with the similar HP system. And if the issue can be reproduced there...

Another thought of mine was, if it wouldn't it be useful to have a kernel commandline switch (or thunderbolt module option) that allows the user to select whether the software connection manager should be used, or the firmware connection manager (if avialable)? I understand that in general using the sw connection manager is favorable because the OS then has control over the connection, but maybe there are use cases where letting the firmware doing the job is just fine (like in my case ;-)

Thanks and best regards,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi,

On Fri, Jan 27, 2023 at 01:07:43PM +0100, Christian Schaubschläger wrote:
> Hi Mika,
> 
> >>> Previously you said you'd talk with your Windows folks about this; any
> >>> news from there?
> >> I've talked to them and still in talks with the UEFI folks but the
> >> current undestanding is that Windows does not do anything special when
> >> the system is rebooted (so equal to what Linux does). There is one
> >> "development" system in Israel lab that should be pretty similar to what
> >> the HP system of yours is but the person who was going to try to
> >> reproduce is in sick leave now.
> > Ok...
> I was just wondering if there are already any news from your lab in
> Israel with the similar HP system. And if the issue can be reproduced
> there...

Yeah, I just got report from the folks and unfortunately we were not
able to reproduce the issue in the lab. The PCIe tunnel stays up (or
gets re-created) after reboot from Linux and is visible in the UEFI
shell (so in the BIOS).

So at the moment I'm out of ideas why this happens since Windows flows
should be the same. And we have no means to repro it locally to
investigate further :/

> Another thought of mine was, if it wouldn't it be useful to have a kernel commandline switch (or thunderbolt module option) that allows the user to select whether the software connection manager should be used, or the firmware connection manager (if avialable)? I understand that in general using the sw connection manager is favorable because the OS then has control over the connection, but maybe there are use cases where letting the firmware doing the job is just fine (like in my case ;-)

Right, we may add one but I think it should be part of the USB4 _OSC
negotiation like the below patch (acpi.native_usb4_disable=1 in the
command line should switch to FW CM). We could also make it "automatic"
by adding a DMI quirks table that then sets this.

diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c
index 0c05ccde1f7a..7da4d56fb936 100644
--- a/drivers/acpi/bus.c
+++ b/drivers/acpi/bus.c
@@ -296,6 +296,9 @@ EXPORT_SYMBOL_GPL(osc_cpc_flexible_adr_space_confirmed);
 /*
  * ACPI 6.4 Operating System Capabilities for USB.
  */
+bool native_usb4_disable;
+module_param(native_usb4_disable, bool, 0644);
+
 bool osc_sb_native_usb4_support_confirmed;
 EXPORT_SYMBOL_GPL(osc_sb_native_usb4_support_confirmed);
 
@@ -344,7 +347,7 @@ static void acpi_bus_osc_negotiate_platform_control(void)
 	if (IS_ENABLED(CONFIG_SCHED_MC_PRIO))
 		capbuf[OSC_SUPPORT_DWORD] |= OSC_SB_CPC_DIVERSE_HIGH_SUPPORT;
 
-	if (IS_ENABLED(CONFIG_USB4))
+	if (IS_ENABLED(CONFIG_USB4) && !native_usb4_disable)
 		capbuf[OSC_SUPPORT_DWORD] |= OSC_SB_NATIVE_USB4_SUPPORT;
 
 	if (!ghes_disable)

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi,

Am 27.01.23 um 13:27 schrieb Mika Westerberg:
> Yeah, I just got report from the folks and unfortunately we were not
> able to reproduce the issue in the lab. The PCIe tunnel stays up (or
> gets re-created) after reboot from Linux and is visible in the UEFI
> shell (so in the BIOS).
>
> So at the moment I'm out of ideas why this happens since Windows flows
> should be the same. And we have no means to repro it locally to
> investigate further :/

Ok. I try to get access to some more TB4 devices. Currently I have two (HP, Lenovo) that show the issue when the software connection manager is used.

>> Another thought of mine was, if it wouldn't it be useful to have a kernel commandline switch (or thunderbolt module option) that allows the user to select whether the software connection manager should be used, or the firmware connection manager (if avialable)? I understand that in general using the sw connection manager is favorable because the OS then has control over the connection, but maybe there are use cases where letting the firmware doing the job is just fine (like in my case ;-)
> Right, we may add one but I think it should be part of the USB4 _OSC
> negotiation like the below patch (acpi.native_usb4_disable=1 in the
> command line should switch to FW CM). We could also make it "automatic"
> by adding a DMI quirks table that then sets this.
>
> diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c
> index 0c05ccde1f7a..7da4d56fb936 100644
> --- a/drivers/acpi/bus.c
> +++ b/drivers/acpi/bus.c
> @@ -296,6 +296,9 @@ EXPORT_SYMBOL_GPL(osc_cpc_flexible_adr_space_confirmed);
>  /*
>   * ACPI 6.4 Operating System Capabilities for USB.
>   */
> +bool native_usb4_disable;
> +module_param(native_usb4_disable, bool, 0644);
> +
>  bool osc_sb_native_usb4_support_confirmed;
>  EXPORT_SYMBOL_GPL(osc_sb_native_usb4_support_confirmed);
>
> @@ -344,7 +347,7 @@ static void acpi_bus_osc_negotiate_platform_control(void)
>  	if (IS_ENABLED(CONFIG_SCHED_MC_PRIO))
>  		capbuf[OSC_SUPPORT_DWORD] |= OSC_SB_CPC_DIVERSE_HIGH_SUPPORT;
>
> -	if (IS_ENABLED(CONFIG_USB4))
> +	if (IS_ENABLED(CONFIG_USB4) && !native_usb4_disable)
>  		capbuf[OSC_SUPPORT_DWORD] |= OSC_SB_NATIVE_USB4_SUPPORT;
>
>  	if (!ghes_disable)
That works fine on the HP, with that active the firmware connection manger is used! To make it 'automatic' would mean that on machines listed in this new DMI quirks table the software connection manager would be disabled, correct?

Regards,
Christian

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi,

On Mon, Jan 30, 2023 at 11:52:59AM +0100, Christian Schaubschläger wrote:
> Hi,
> 
> Am 27.01.23 um 13:27 schrieb Mika Westerberg:
> > Yeah, I just got report from the folks and unfortunately we were not
> > able to reproduce the issue in the lab. The PCIe tunnel stays up (or
> > gets re-created) after reboot from Linux and is visible in the UEFI
> > shell (so in the BIOS).
> >
> > So at the moment I'm out of ideas why this happens since Windows flows
> > should be the same. And we have no means to repro it locally to
> > investigate further :/
> 
> Ok. I try to get access to some more TB4 devices. Currently I have two
> (HP, Lenovo) that show the issue when the software connection manager
> is used.

We should be getting one HP EliteBook 630 G9 soon (I guess this is one
of the systems you see the issue). Once we get it we will try to repro
and investigate further.

> >> Another thought of mine was, if it wouldn't it be useful to have a kernel commandline switch (or thunderbolt module option) that allows the user to select whether the software connection manager should be used, or the firmware connection manager (if avialable)? I understand that in general using the sw connection manager is favorable because the OS then has control over the connection, but maybe there are use cases where letting the firmware doing the job is just fine (like in my case ;-)
> > Right, we may add one but I think it should be part of the USB4 _OSC
> > negotiation like the below patch (acpi.native_usb4_disable=1 in the
> > command line should switch to FW CM). We could also make it "automatic"
> > by adding a DMI quirks table that then sets this.
> >
> > diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c
> > index 0c05ccde1f7a..7da4d56fb936 100644
> > --- a/drivers/acpi/bus.c
> > +++ b/drivers/acpi/bus.c
> > @@ -296,6 +296,9 @@ EXPORT_SYMBOL_GPL(osc_cpc_flexible_adr_space_confirmed);
> >  /*
> >   * ACPI 6.4 Operating System Capabilities for USB.
> >   */
> > +bool native_usb4_disable;
> > +module_param(native_usb4_disable, bool, 0644);
> > +
> >  bool osc_sb_native_usb4_support_confirmed;
> >  EXPORT_SYMBOL_GPL(osc_sb_native_usb4_support_confirmed);
> >
> > @@ -344,7 +347,7 @@ static void acpi_bus_osc_negotiate_platform_control(void)
> >  	if (IS_ENABLED(CONFIG_SCHED_MC_PRIO))
> >  		capbuf[OSC_SUPPORT_DWORD] |= OSC_SB_CPC_DIVERSE_HIGH_SUPPORT;
> >
> > -	if (IS_ENABLED(CONFIG_USB4))
> > +	if (IS_ENABLED(CONFIG_USB4) && !native_usb4_disable)
> >  		capbuf[OSC_SUPPORT_DWORD] |= OSC_SB_NATIVE_USB4_SUPPORT;
> >
> >  	if (!ghes_disable)
> That works fine on the HP, with that active the firmware connection manger is used! To make it 'automatic' would mean that on machines listed in this new DMI quirks table the software connection manager would be disabled, correct?

Yes, correct. However, since the software CM works as well except for
this case, I'm having some second thougths on adding a DMI quirk because
it makes certain things not be available, such as lane bonding for the
networking, and de-authorization of the PCIe tunnels among other things.
I think the command line option can be added regardless.

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi Christian,

On Mon, Jan 30, 2023 at 04:10:12PM +0200, Mika Westerberg wrote:
> Hi,
> 
> On Mon, Jan 30, 2023 at 11:52:59AM +0100, Christian Schaubschläger wrote:
> > Hi,
> > 
> > Am 27.01.23 um 13:27 schrieb Mika Westerberg:
> > > Yeah, I just got report from the folks and unfortunately we were not
> > > able to reproduce the issue in the lab. The PCIe tunnel stays up (or
> > > gets re-created) after reboot from Linux and is visible in the UEFI
> > > shell (so in the BIOS).
> > >
> > > So at the moment I'm out of ideas why this happens since Windows flows
> > > should be the same. And we have no means to repro it locally to
> > > investigate further :/
> > 
> > Ok. I try to get access to some more TB4 devices. Currently I have two
> > (HP, Lenovo) that show the issue when the software connection manager
> > is used.
> 
> We should be getting one HP EliteBook 630 G9 soon (I guess this is one
> of the systems you see the issue). Once we get it we will try to repro
> and investigate further.

We are able to reproduce the issue and it was root caused to be missing
UNSET_INBOUND_SBTX port operation during retimer enumeration. I posted a
series of fixes few minutes ago (with you Cc'd) and one of them should
fix this issue:

https://lore.kernel.org/linux-usb/20230306113605.46137-3-mika.westerberg@linux.intel.com/

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi Mika,

> We are able to reproduce the issue and it was root caused to be missing
> UNSET_INBOUND_SBTX port operation during retimer enumeration. I posted a
> series of fixes few minutes ago (with you Cc'd) and one of them should
> fix this issue:
>
> https://lore.kernel.org/linux-usb/20230306113605.46137-3-mika.westerberg@linux.intel.com/

That's great news, thank you!
I'll apply this patch to a 6.1.15 kernel and give it a try. Could take a day or two, though, beacause I don't have the hardware here at the moment. I'll let you know.

Thanks and best regards,
Christian

Re: Possible problem with thunderbolt 4

[Christian Schaubschläger]

Hi Mika,

it seems that I have an issue here which seems similar to the one on the HP earlier this year.

This time the hardware is a Dell Latitude 7440 with a rather new Intel i5-1345U (13th Gen.) and TB4. I have three Docks, one older Dell WD19TB, a newer Dell WD22TB4, and a ThinkPad Universal USB-C Dock.
With the TB4 dock everything works fine (meaning all hardware is functioning in the UEFI firmware as well as in Windows after booting Linux first).

With the older TB dock, as well as with the USB-C dock the network interface on the dock is dead after Linux. But in contrast to the HP case from earlier this year, this time the network interface remains dead when I boot Linux again. After replugging the dock it becomes alive again.

This can easilsy be reproduced by booting Ubuntu 22.04 from an USB device. Using a newer kernel with the UNSET_INBOUND_SBTX fix does not solve the issue.

I'm not sure if this is a USB or Thunderbolt issue, could be an issue with the network device, too (all three docks have the same Realtek 8153). But I don't see this problem with these docks on other Laptops I have access to, so probably it's not the network device...

Can I provide some logs, etc. to dive deeper into this?

Thanks and best regards,
Christian



Am 06.03.23 um 13:27 schrieb Christian Schaubschläger:
> Hi Mika,
>
>> We are able to reproduce the issue and it was root caused to be missing
>> UNSET_INBOUND_SBTX port operation during retimer enumeration. I posted a
>> series of fixes few minutes ago (with you Cc'd) and one of them should
>> fix this issue:
>>
>> https://lore.kernel.org/linux-usb/20230306113605.46137-3-mika.westerberg@linux.intel.com/
> That's great news, thank you!
> I'll apply this patch to a 6.1.15 kernel and give it a try. Could take a day or two, though, beacause I don't have the hardware here at the moment. I'll let you know.
>
> Thanks and best regards,
> Christian
>

Re: Possible problem with thunderbolt 4

[Mika Westerberg]

Hi,

On Tue, May 23, 2023 at 10:22:28AM +0200, Christian Schaubschläger wrote:
> Hi Mika,
> 
> it seems that I have an issue here which seems similar to the one on the HP earlier this year.
> 
> This time the hardware is a Dell Latitude 7440 with a rather new Intel i5-1345U (13th Gen.) and TB4. I have three Docks, one older Dell WD19TB, a newer Dell WD22TB4, and a ThinkPad Universal USB-C Dock.
> With the TB4 dock everything works fine (meaning all hardware is functioning in the UEFI firmware as well as in Windows after booting Linux first).
> 
> With the older TB dock, as well as with the USB-C dock the network interface on the dock is dead after Linux. But in contrast to the HP case from earlier this year, this time the network interface remains dead when I boot Linux again. After replugging the dock it becomes alive again.
> 
> This can easilsy be reproduced by booting Ubuntu 22.04 from an USB device. Using a newer kernel with the UNSET_INBOUND_SBTX fix does not solve the issue.
> 
> I'm not sure if this is a USB or Thunderbolt issue, could be an issue with the network device, too (all three docks have the same Realtek 8153). But I don't see this problem with these docks on other Laptops I have access to, so probably it's not the network device...
> 
> Can I provide some logs, etc. to dive deeper into this?

Sure, I suggest also start a new email thread or file a kernel.org
bugzilla ticket.

Can you add "thunderbolt.dyndbg=+p" in the kernel command line and
reproduce the issue and share the full dmesg?