Missing Thunderbolt 3 PCI-E atomics support

Missing Thunderbolt 3 PCI-E atomics support

[Timur Kristóf]

Hi,

I was sent here by Greg KH from the Linux USB mailing list, I hope this
is the right place to ask.

PCI-E atomics don't work for me with Thunderbolt 3.
I see the following message from my Thunderbolt 3 eGPU in dmesg:

kfd: skipped device 1002:67df, PCI rejects atomics

Hardware is a Dell XPS 13 9370 (with i7-8550U CPU) connected to a Zotac
AMP mini, with an AMD RX 570 graphics card. Due to this, I cannot use
the GPU for OpenCL, because the compute stack requires PCI-E atomics
support [1].

What could be the problem? Is this a hardware limitation or a missing
feature in the Linux TB driver?

[1] https://github.com/RadeonOpenCompute/ROCm/issues/157

Thanks & best regards,
Tim

ps. I'm not subscribed to the list, please CC me on your replies.

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Missing Thunderbolt 3 PCI-E atomics support

[Mika Westerberg]

On Wed, Mar 13, 2019 at 07:09:26PM +0100, Timur Kristóf wrote:
> Hi,

Hi,

> I was sent here by Greg KH from the Linux USB mailing list, I hope this
> is the right place to ask.
> 
> PCI-E atomics don't work for me with Thunderbolt 3.
> I see the following message from my Thunderbolt 3 eGPU in dmesg:
> 
> kfd: skipped device 1002:67df, PCI rejects atomics
> 
> Hardware is a Dell XPS 13 9370 (with i7-8550U CPU) connected to a Zotac
> AMP mini, with an AMD RX 570 graphics card. Due to this, I cannot use
> the GPU for OpenCL, because the compute stack requires PCI-E atomics
> support [1].
> 
> What could be the problem? Is this a hardware limitation or a missing
> feature in the Linux TB driver?

I don't think it has anything to do with TBT itself. AtomicOps is a PCIe
feature.

What does 'sudo lspci -vv' show for the ports in question?
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Missing Thunderbolt 3 PCI-E atomics support

[Timur Kristóf]

On Thu, 2019-03-14 at 12:30 +0200, Mika Westerberg wrote:
> On Wed, Mar 13, 2019 at 07:09:26PM +0100, Timur Kristóf wrote:
> > Hi,
> 
> Hi,
> 
> > I was sent here by Greg KH from the Linux USB mailing list, I hope
> > this
> > is the right place to ask.
> > 
> > PCI-E atomics don't work for me with Thunderbolt 3.
> > I see the following message from my Thunderbolt 3 eGPU in dmesg:
> > 
> > kfd: skipped device 1002:67df, PCI rejects atomics
> > 
> > Hardware is a Dell XPS 13 9370 (with i7-8550U CPU) connected to a
> > Zotac
> > AMP mini, with an AMD RX 570 graphics card. Due to this, I cannot
> > use
> > the GPU for OpenCL, because the compute stack requires PCI-E
> > atomics
> > support [1].
> > 
> > What could be the problem? Is this a hardware limitation or a
> > missing
> > feature in the Linux TB driver?
> 
> I don't think it has anything to do with TBT itself. AtomicOps is a
> PCIe
> feature.
> 
> What does 'sudo lspci -vv' show for the ports in question?

Hi Mika,

I know atomics is a PCIe feature, but in this case the PCIe goes
through TB3, so I would assume it has something to do with it.

Here is the output of 'lspci -vv':
https://pastebin.com/Qt5RUFVc

Best regards,
Tim


_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Missing Thunderbolt 3 PCI-E atomics support

[Mika Westerberg]

On Thu, Mar 14, 2019 at 06:26:00PM +0100, Timur Kristóf wrote:
> I know atomics is a PCIe feature, but in this case the PCIe goes
> through TB3, so I would assume it has something to do with it.

Does it work if you plug the graphics card directly to the PCIe slot?

> Here is the output of 'lspci -vv':
> https://pastebin.com/Qt5RUFVc

The root port (1c.4) says this:

      DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd+
	   AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-

Not knowing much about AtomicOps but to me this looks like the root port
does not support the feature.
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Missing Thunderbolt 3 PCI-E atomics support

[Timur Kristóf]

On Thu, 2019-03-14 at 19:40 +0200, Mika Westerberg wrote:
> On Thu, Mar 14, 2019 at 06:26:00PM +0100, Timur Kristóf wrote:
> > I know atomics is a PCIe feature, but in this case the PCIe goes
> > through TB3, so I would assume it has something to do with it.
> 
> Does it work if you plug the graphics card directly to the PCIe slot?

There is no PCIe slot in which I could plug the graphics card.
At least I'm not aware of there being one on this laptop.
Please correct me if I'm wrong.

> 
> > Here is the output of 'lspci -vv':
> > https://pastebin.com/Qt5RUFVc
> 
> The root port (1c.4) says this:
> 
>       DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF
> Not Supported ARIFwd+
> 	   AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> 
> Not knowing much about AtomicOps but to me this looks like the root
> port
> does not support the feature.

What kind of output should lspci show if the feature were supported?

As far as I understand the root port is integrated in the CPU, or in
the chipset maybe? It says it's a Sunrise Point-LP, and I googled it
but was unable to find a spec sheet.


Thanks,
Tim

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Missing Thunderbolt 3 PCI-E atomics support

[Mika Westerberg]

On Thu, Mar 14, 2019 at 06:54:21PM +0100, Timur Kristóf wrote:
> On Thu, 2019-03-14 at 19:40 +0200, Mika Westerberg wrote:
> > On Thu, Mar 14, 2019 at 06:26:00PM +0100, Timur Kristóf wrote:
> > > I know atomics is a PCIe feature, but in this case the PCIe goes
> > > through TB3, so I would assume it has something to do with it.
> > 
> > Does it work if you plug the graphics card directly to the PCIe slot?
> 
> There is no PCIe slot in which I could plug the graphics card.
> At least I'm not aware of there being one on this laptop.
> Please correct me if I'm wrong.

No, you are right. I forgot that this is a laptop.

> > 
> > > Here is the output of 'lspci -vv':
> > > https://pastebin.com/Qt5RUFVc
> > 
> > The root port (1c.4) says this:
> > 
> >       DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF
> > Not Supported ARIFwd+
> > 	   AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> > 
> > Not knowing much about AtomicOps but to me this looks like the root
> > port
> > does not support the feature.
> 
> What kind of output should lspci show if the feature were supported?

The AMD card has this:

         DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported
	              AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-

so I would expect something similar on the root port side as
pci_enable_atomic_ops_to_root() fails otherwise with mask of
PCI_EXP_DEVCAP2_ATOMIC_COMP32 | PCI_EXP_DEVCAP2_ATOMIC_COMP64 that the
AMD driver requests.

> As far as I understand the root port is integrated in the CPU, or in
> the chipset maybe? It says it's a Sunrise Point-LP, and I googled it
> but was unable to find a spec sheet.

You can find it here:

  https://www.intel.com/content/www/us/en/products/docs/processors/core/6th-gen-core-pch-u-y-io-datasheet-vol-2.html

Pages 845-826 show the DEVCAP2 register for the 1c.4 (D28/F4) and it
does not seem to have AtomicOps caps set.
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Missing Thunderbolt 3 PCI-E atomics support

[Timur Kristóf]

On Thu, 2019-03-14 at 20:17 +0200, Mika Westerberg wrote:
> > > > Here is the output of 'lspci -vv':
> > > > https://pastebin.com/Qt5RUFVc
> > > 
> > > The root port (1c.4) says this:
> > > 
> > >       DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+,
> > > OBFF
> > > Not Supported ARIFwd+
> > > 	   AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> > > 
> > > Not knowing much about AtomicOps but to me this looks like the
> > > root
> > > port
> > > does not support the feature.
> > 
> > What kind of output should lspci show if the feature were
> > supported?
> 
> The AMD card has this:
> 
>          DevCap2: Completion Timeout: Not Supported, TimeoutDis-,
> LTR+, OBFF Not Supported
> 	              AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
> 
> so I would expect something similar on the root port side as
> pci_enable_atomic_ops_to_root() fails otherwise with mask of
> PCI_EXP_DEVCAP2_ATOMIC_COMP32 | PCI_EXP_DEVCAP2_ATOMIC_COMP64 that
> the
> AMD driver requests.
> 
> > As far as I understand the root port is integrated in the CPU, or
> > in
> > the chipset maybe? It says it's a Sunrise Point-LP, and I googled
> > it
> > but was unable to find a spec sheet.
> 
> You can find it here:
> 
>   
> https://www.intel.com/content/www/us/en/products/docs/processors/core/6th-gen-core-pch-u-y-io-datasheet-vol-2.html
> 
> Pages 845-826 show the DEVCAP2 register for the 1c.4 (D28/F4) and it
> does not seem to have AtomicOps caps set.

This would be the 8th gen (8550U) but I assume it has similar
capabilities to the 6th gen. So, it seems that this is a hardware
limitation of the chipset.

Thanks Mika for clearing that up.

Best regards,
Tim

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Missing Thunderbolt 3 PCI-E atomics support

[Dieter Nützel]

Hello to both of you, Tim and Mika,

sorry that I step in so late, but after I've read that Tim's laptop 
didn't have PCIe atomics (is it realy PCIe 2.xx?) I have some hints, 
too. Question: Is PCIe 3.xx+ needed together with Thunderbold 3 or is 
PCIe 2.xx enaugth (like Tim's laptop)?

If Tim's laptop is realy only PCIe 2.xx (_without_ atomics) like mine 
'old' Intel Xeon X3470 (3400 Series Chipset) _without_ atomics you/we 
only have the option running Clover or AMD-pro's OpenCL. ROCm isn't an 
option with Polaris (20/RX580, here), currently. But we had some 
questions about AMD (Polaris) running as eGPU on one of our lists 
(mesa-devel / dri-devel / amd-gfx), lately.

I've CC'ed Alex.

Have a look here (ROCm 2.2), too:
https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/1086267-radeon-rocm-2-2-released-with-vega-20-optimization-caffe2-multi-gpu-training?p=1086387#post1086387

Greetings,
Dieter

Am 14.03.2019 19:36, schrieb Timur Kristóf:
> On Thu, 2019-03-14 at 20:17 +0200, Mika Westerberg wrote:
>> > > > Here is the output of 'lspci -vv':
>> > > > https://pastebin.com/Qt5RUFVc
>> > >
>> > > The root port (1c.4) says this:
>> > >
>> > >       DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+,
>> > > OBFF
>> > > Not Supported ARIFwd+
>> > > 	   AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
>> > >
>> > > Not knowing much about AtomicOps but to me this looks like the
>> > > root
>> > > port
>> > > does not support the feature.
>> >
>> > What kind of output should lspci show if the feature were
>> > supported?
>> 
>> The AMD card has this:
>> 
>>          DevCap2: Completion Timeout: Not Supported, TimeoutDis-,
>> LTR+, OBFF Not Supported
>> 	              AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
>> 
>> so I would expect something similar on the root port side as
>> pci_enable_atomic_ops_to_root() fails otherwise with mask of
>> PCI_EXP_DEVCAP2_ATOMIC_COMP32 | PCI_EXP_DEVCAP2_ATOMIC_COMP64 that
>> the
>> AMD driver requests.
>> 
>> > As far as I understand the root port is integrated in the CPU, or
>> > in
>> > the chipset maybe? It says it's a Sunrise Point-LP, and I googled
>> > it
>> > but was unable to find a spec sheet.
>> 
>> You can find it here:
>> 
>> 
>> https://www.intel.com/content/www/us/en/products/docs/processors/core/6th-gen-core-pch-u-y-io-datasheet-vol-2.html
>> 
>> Pages 845-826 show the DEVCAP2 register for the 1c.4 (D28/F4) and it
>> does not seem to have AtomicOps caps set.
> 
> This would be the 8th gen (8550U) but I assume it has similar
> capabilities to the 6th gen. So, it seems that this is a hardware
> limitation of the chipset.
> 
> Thanks Mika for clearing that up.
> 
> Best regards,
> Tim
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel

Re: Missing Thunderbolt 3 PCI-E atomics support

[Alex Deucher]

As Mika said pci atomics have to be supported by the bridge for the
device to be able to use them.

Alex

On Fri, Mar 15, 2019 at 3:46 PM Dieter Nützel <Dieter@nuetzel-hh.de> wrote:
>
> Hello to both of you, Tim and Mika,
>
> sorry that I step in so late, but after I've read that Tim's laptop
> didn't have PCIe atomics (is it realy PCIe 2.xx?) I have some hints,
> too. Question: Is PCIe 3.xx+ needed together with Thunderbold 3 or is
> PCIe 2.xx enaugth (like Tim's laptop)?
>
> If Tim's laptop is realy only PCIe 2.xx (_without_ atomics) like mine
> 'old' Intel Xeon X3470 (3400 Series Chipset) _without_ atomics you/we
> only have the option running Clover or AMD-pro's OpenCL. ROCm isn't an
> option with Polaris (20/RX580, here), currently. But we had some
> questions about AMD (Polaris) running as eGPU on one of our lists
> (mesa-devel / dri-devel / amd-gfx), lately.
>
> I've CC'ed Alex.
>
> Have a look here (ROCm 2.2), too:
> https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/1086267-radeon-rocm-2-2-released-with-vega-20-optimization-caffe2-multi-gpu-training?p=1086387#post1086387
>
> Greetings,
> Dieter
>
> Am 14.03.2019 19:36, schrieb Timur Kristóf:
> > On Thu, 2019-03-14 at 20:17 +0200, Mika Westerberg wrote:
> >> > > > Here is the output of 'lspci -vv':
> >> > > > https://pastebin.com/Qt5RUFVc
> >> > >
> >> > > The root port (1c.4) says this:
> >> > >
> >> > >       DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+,
> >> > > OBFF
> >> > > Not Supported ARIFwd+
> >> > >     AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-
> >> > >
> >> > > Not knowing much about AtomicOps but to me this looks like the
> >> > > root
> >> > > port
> >> > > does not support the feature.
> >> >
> >> > What kind of output should lspci show if the feature were
> >> > supported?
> >>
> >> The AMD card has this:
> >>
> >>          DevCap2: Completion Timeout: Not Supported, TimeoutDis-,
> >> LTR+, OBFF Not Supported
> >>                    AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
> >>
> >> so I would expect something similar on the root port side as
> >> pci_enable_atomic_ops_to_root() fails otherwise with mask of
> >> PCI_EXP_DEVCAP2_ATOMIC_COMP32 | PCI_EXP_DEVCAP2_ATOMIC_COMP64 that
> >> the
> >> AMD driver requests.
> >>
> >> > As far as I understand the root port is integrated in the CPU, or
> >> > in
> >> > the chipset maybe? It says it's a Sunrise Point-LP, and I googled
> >> > it
> >> > but was unable to find a spec sheet.
> >>
> >> You can find it here:
> >>
> >>
> >> https://www.intel.com/content/www/us/en/products/docs/processors/core/6th-gen-core-pch-u-y-io-datasheet-vol-2.html
> >>
> >> Pages 845-826 show the DEVCAP2 register for the 1c.4 (D28/F4) and it
> >> does not seem to have AtomicOps caps set.
> >
> > This would be the 8th gen (8550U) but I assume it has similar
> > capabilities to the 6th gen. So, it seems that this is a hardware
> > limitation of the chipset.
> >
> > Thanks Mika for clearing that up.
> >
> > Best regards,
> > Tim
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel