* Missing Thunderbolt 3 PCI-E atomics support @ 2019-03-13 18:09 Timur Kristóf 2019-03-14 10:30 ` Mika Westerberg 0 siblings, 1 reply; 9+ messages in thread From: Timur Kristóf @ 2019-03-13 18:09 UTC (permalink / raw) To: michael.jamet, Mika Westerberg; +Cc: dri-devel Hi, I was sent here by Greg KH from the Linux USB mailing list, I hope this is the right place to ask. PCI-E atomics don't work for me with Thunderbolt 3. I see the following message from my Thunderbolt 3 eGPU in dmesg: kfd: skipped device 1002:67df, PCI rejects atomics Hardware is a Dell XPS 13 9370 (with i7-8550U CPU) connected to a Zotac AMP mini, with an AMD RX 570 graphics card. Due to this, I cannot use the GPU for OpenCL, because the compute stack requires PCI-E atomics support [1]. What could be the problem? Is this a hardware limitation or a missing feature in the Linux TB driver? [1] https://github.com/RadeonOpenCompute/ROCm/issues/157 Thanks & best regards, Tim ps. I'm not subscribed to the list, please CC me on your replies. _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Missing Thunderbolt 3 PCI-E atomics support 2019-03-13 18:09 Missing Thunderbolt 3 PCI-E atomics support Timur Kristóf @ 2019-03-14 10:30 ` Mika Westerberg 2019-03-14 17:26 ` Timur Kristóf 0 siblings, 1 reply; 9+ messages in thread From: Mika Westerberg @ 2019-03-14 10:30 UTC (permalink / raw) To: Timur Kristóf; +Cc: michael.jamet, dri-devel On Wed, Mar 13, 2019 at 07:09:26PM +0100, Timur Kristóf wrote: > Hi, Hi, > I was sent here by Greg KH from the Linux USB mailing list, I hope this > is the right place to ask. > > PCI-E atomics don't work for me with Thunderbolt 3. > I see the following message from my Thunderbolt 3 eGPU in dmesg: > > kfd: skipped device 1002:67df, PCI rejects atomics > > Hardware is a Dell XPS 13 9370 (with i7-8550U CPU) connected to a Zotac > AMP mini, with an AMD RX 570 graphics card. Due to this, I cannot use > the GPU for OpenCL, because the compute stack requires PCI-E atomics > support [1]. > > What could be the problem? Is this a hardware limitation or a missing > feature in the Linux TB driver? I don't think it has anything to do with TBT itself. AtomicOps is a PCIe feature. What does 'sudo lspci -vv' show for the ports in question? _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Missing Thunderbolt 3 PCI-E atomics support 2019-03-14 10:30 ` Mika Westerberg @ 2019-03-14 17:26 ` Timur Kristóf 2019-03-14 17:40 ` Mika Westerberg 0 siblings, 1 reply; 9+ messages in thread From: Timur Kristóf @ 2019-03-14 17:26 UTC (permalink / raw) To: Mika Westerberg; +Cc: michael.jamet, dri-devel On Thu, 2019-03-14 at 12:30 +0200, Mika Westerberg wrote: > On Wed, Mar 13, 2019 at 07:09:26PM +0100, Timur Kristóf wrote: > > Hi, > > Hi, > > > I was sent here by Greg KH from the Linux USB mailing list, I hope > > this > > is the right place to ask. > > > > PCI-E atomics don't work for me with Thunderbolt 3. > > I see the following message from my Thunderbolt 3 eGPU in dmesg: > > > > kfd: skipped device 1002:67df, PCI rejects atomics > > > > Hardware is a Dell XPS 13 9370 (with i7-8550U CPU) connected to a > > Zotac > > AMP mini, with an AMD RX 570 graphics card. Due to this, I cannot > > use > > the GPU for OpenCL, because the compute stack requires PCI-E > > atomics > > support [1]. > > > > What could be the problem? Is this a hardware limitation or a > > missing > > feature in the Linux TB driver? > > I don't think it has anything to do with TBT itself. AtomicOps is a > PCIe > feature. > > What does 'sudo lspci -vv' show for the ports in question? Hi Mika, I know atomics is a PCIe feature, but in this case the PCIe goes through TB3, so I would assume it has something to do with it. Here is the output of 'lspci -vv': https://pastebin.com/Qt5RUFVc Best regards, Tim _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Missing Thunderbolt 3 PCI-E atomics support 2019-03-14 17:26 ` Timur Kristóf @ 2019-03-14 17:40 ` Mika Westerberg 2019-03-14 17:54 ` Timur Kristóf 0 siblings, 1 reply; 9+ messages in thread From: Mika Westerberg @ 2019-03-14 17:40 UTC (permalink / raw) To: Timur Kristóf; +Cc: michael.jamet, dri-devel On Thu, Mar 14, 2019 at 06:26:00PM +0100, Timur Kristóf wrote: > I know atomics is a PCIe feature, but in this case the PCIe goes > through TB3, so I would assume it has something to do with it. Does it work if you plug the graphics card directly to the PCIe slot? > Here is the output of 'lspci -vv': > https://pastebin.com/Qt5RUFVc The root port (1c.4) says this: DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd+ AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- Not knowing much about AtomicOps but to me this looks like the root port does not support the feature. _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Missing Thunderbolt 3 PCI-E atomics support 2019-03-14 17:40 ` Mika Westerberg @ 2019-03-14 17:54 ` Timur Kristóf 2019-03-14 18:17 ` Mika Westerberg 0 siblings, 1 reply; 9+ messages in thread From: Timur Kristóf @ 2019-03-14 17:54 UTC (permalink / raw) To: Mika Westerberg; +Cc: michael.jamet, dri-devel On Thu, 2019-03-14 at 19:40 +0200, Mika Westerberg wrote: > On Thu, Mar 14, 2019 at 06:26:00PM +0100, Timur Kristóf wrote: > > I know atomics is a PCIe feature, but in this case the PCIe goes > > through TB3, so I would assume it has something to do with it. > > Does it work if you plug the graphics card directly to the PCIe slot? There is no PCIe slot in which I could plug the graphics card. At least I'm not aware of there being one on this laptop. Please correct me if I'm wrong. > > > Here is the output of 'lspci -vv': > > https://pastebin.com/Qt5RUFVc > > The root port (1c.4) says this: > > DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF > Not Supported ARIFwd+ > AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- > > Not knowing much about AtomicOps but to me this looks like the root > port > does not support the feature. What kind of output should lspci show if the feature were supported? As far as I understand the root port is integrated in the CPU, or in the chipset maybe? It says it's a Sunrise Point-LP, and I googled it but was unable to find a spec sheet. Thanks, Tim _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Missing Thunderbolt 3 PCI-E atomics support 2019-03-14 17:54 ` Timur Kristóf @ 2019-03-14 18:17 ` Mika Westerberg 2019-03-14 18:36 ` Timur Kristóf 0 siblings, 1 reply; 9+ messages in thread From: Mika Westerberg @ 2019-03-14 18:17 UTC (permalink / raw) To: Timur Kristóf; +Cc: michael.jamet, dri-devel On Thu, Mar 14, 2019 at 06:54:21PM +0100, Timur Kristóf wrote: > On Thu, 2019-03-14 at 19:40 +0200, Mika Westerberg wrote: > > On Thu, Mar 14, 2019 at 06:26:00PM +0100, Timur Kristóf wrote: > > > I know atomics is a PCIe feature, but in this case the PCIe goes > > > through TB3, so I would assume it has something to do with it. > > > > Does it work if you plug the graphics card directly to the PCIe slot? > > There is no PCIe slot in which I could plug the graphics card. > At least I'm not aware of there being one on this laptop. > Please correct me if I'm wrong. No, you are right. I forgot that this is a laptop. > > > > > Here is the output of 'lspci -vv': > > > https://pastebin.com/Qt5RUFVc > > > > The root port (1c.4) says this: > > > > DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF > > Not Supported ARIFwd+ > > AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- > > > > Not knowing much about AtomicOps but to me this looks like the root > > port > > does not support the feature. > > What kind of output should lspci show if the feature were supported? The AMD card has this: DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported AtomicOpsCap: 32bit+ 64bit+ 128bitCAS- so I would expect something similar on the root port side as pci_enable_atomic_ops_to_root() fails otherwise with mask of PCI_EXP_DEVCAP2_ATOMIC_COMP32 | PCI_EXP_DEVCAP2_ATOMIC_COMP64 that the AMD driver requests. > As far as I understand the root port is integrated in the CPU, or in > the chipset maybe? It says it's a Sunrise Point-LP, and I googled it > but was unable to find a spec sheet. You can find it here: https://www.intel.com/content/www/us/en/products/docs/processors/core/6th-gen-core-pch-u-y-io-datasheet-vol-2.html Pages 845-826 show the DEVCAP2 register for the 1c.4 (D28/F4) and it does not seem to have AtomicOps caps set. _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Missing Thunderbolt 3 PCI-E atomics support 2019-03-14 18:17 ` Mika Westerberg @ 2019-03-14 18:36 ` Timur Kristóf 2019-03-15 19:46 ` Dieter Nützel 0 siblings, 1 reply; 9+ messages in thread From: Timur Kristóf @ 2019-03-14 18:36 UTC (permalink / raw) To: Mika Westerberg; +Cc: michael.jamet, dri-devel On Thu, 2019-03-14 at 20:17 +0200, Mika Westerberg wrote: > > > > Here is the output of 'lspci -vv': > > > > https://pastebin.com/Qt5RUFVc > > > > > > The root port (1c.4) says this: > > > > > > DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, > > > OBFF > > > Not Supported ARIFwd+ > > > AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- > > > > > > Not knowing much about AtomicOps but to me this looks like the > > > root > > > port > > > does not support the feature. > > > > What kind of output should lspci show if the feature were > > supported? > > The AMD card has this: > > DevCap2: Completion Timeout: Not Supported, TimeoutDis-, > LTR+, OBFF Not Supported > AtomicOpsCap: 32bit+ 64bit+ 128bitCAS- > > so I would expect something similar on the root port side as > pci_enable_atomic_ops_to_root() fails otherwise with mask of > PCI_EXP_DEVCAP2_ATOMIC_COMP32 | PCI_EXP_DEVCAP2_ATOMIC_COMP64 that > the > AMD driver requests. > > > As far as I understand the root port is integrated in the CPU, or > > in > > the chipset maybe? It says it's a Sunrise Point-LP, and I googled > > it > > but was unable to find a spec sheet. > > You can find it here: > > > https://www.intel.com/content/www/us/en/products/docs/processors/core/6th-gen-core-pch-u-y-io-datasheet-vol-2.html > > Pages 845-826 show the DEVCAP2 register for the 1c.4 (D28/F4) and it > does not seem to have AtomicOps caps set. This would be the 8th gen (8550U) but I assume it has similar capabilities to the 6th gen. So, it seems that this is a hardware limitation of the chipset. Thanks Mika for clearing that up. Best regards, Tim _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Missing Thunderbolt 3 PCI-E atomics support 2019-03-14 18:36 ` Timur Kristóf @ 2019-03-15 19:46 ` Dieter Nützel 2019-03-15 20:03 ` Alex Deucher 0 siblings, 1 reply; 9+ messages in thread From: Dieter Nützel @ 2019-03-15 19:46 UTC (permalink / raw) To: Timur Kristóf; +Cc: michael.jamet, Mika Westerberg, dri-devel Hello to both of you, Tim and Mika, sorry that I step in so late, but after I've read that Tim's laptop didn't have PCIe atomics (is it realy PCIe 2.xx?) I have some hints, too. Question: Is PCIe 3.xx+ needed together with Thunderbold 3 or is PCIe 2.xx enaugth (like Tim's laptop)? If Tim's laptop is realy only PCIe 2.xx (_without_ atomics) like mine 'old' Intel Xeon X3470 (3400 Series Chipset) _without_ atomics you/we only have the option running Clover or AMD-pro's OpenCL. ROCm isn't an option with Polaris (20/RX580, here), currently. But we had some questions about AMD (Polaris) running as eGPU on one of our lists (mesa-devel / dri-devel / amd-gfx), lately. I've CC'ed Alex. Have a look here (ROCm 2.2), too: https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/1086267-radeon-rocm-2-2-released-with-vega-20-optimization-caffe2-multi-gpu-training?p=1086387#post1086387 Greetings, Dieter Am 14.03.2019 19:36, schrieb Timur Kristóf: > On Thu, 2019-03-14 at 20:17 +0200, Mika Westerberg wrote: >> > > > Here is the output of 'lspci -vv': >> > > > https://pastebin.com/Qt5RUFVc >> > > >> > > The root port (1c.4) says this: >> > > >> > > DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, >> > > OBFF >> > > Not Supported ARIFwd+ >> > > AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- >> > > >> > > Not knowing much about AtomicOps but to me this looks like the >> > > root >> > > port >> > > does not support the feature. >> > >> > What kind of output should lspci show if the feature were >> > supported? >> >> The AMD card has this: >> >> DevCap2: Completion Timeout: Not Supported, TimeoutDis-, >> LTR+, OBFF Not Supported >> AtomicOpsCap: 32bit+ 64bit+ 128bitCAS- >> >> so I would expect something similar on the root port side as >> pci_enable_atomic_ops_to_root() fails otherwise with mask of >> PCI_EXP_DEVCAP2_ATOMIC_COMP32 | PCI_EXP_DEVCAP2_ATOMIC_COMP64 that >> the >> AMD driver requests. >> >> > As far as I understand the root port is integrated in the CPU, or >> > in >> > the chipset maybe? It says it's a Sunrise Point-LP, and I googled >> > it >> > but was unable to find a spec sheet. >> >> You can find it here: >> >> >> https://www.intel.com/content/www/us/en/products/docs/processors/core/6th-gen-core-pch-u-y-io-datasheet-vol-2.html >> >> Pages 845-826 show the DEVCAP2 register for the 1c.4 (D28/F4) and it >> does not seem to have AtomicOps caps set. > > This would be the 8th gen (8550U) but I assume it has similar > capabilities to the 6th gen. So, it seems that this is a hardware > limitation of the chipset. > > Thanks Mika for clearing that up. > > Best regards, > Tim _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Missing Thunderbolt 3 PCI-E atomics support 2019-03-15 19:46 ` Dieter Nützel @ 2019-03-15 20:03 ` Alex Deucher 0 siblings, 0 replies; 9+ messages in thread From: Alex Deucher @ 2019-03-15 20:03 UTC (permalink / raw) To: Dieter Nützel Cc: Michael Jamet, Mika Westerberg, Maling list - DRI developers, Timur Kristóf As Mika said pci atomics have to be supported by the bridge for the device to be able to use them. Alex On Fri, Mar 15, 2019 at 3:46 PM Dieter Nützel <Dieter@nuetzel-hh.de> wrote: > > Hello to both of you, Tim and Mika, > > sorry that I step in so late, but after I've read that Tim's laptop > didn't have PCIe atomics (is it realy PCIe 2.xx?) I have some hints, > too. Question: Is PCIe 3.xx+ needed together with Thunderbold 3 or is > PCIe 2.xx enaugth (like Tim's laptop)? > > If Tim's laptop is realy only PCIe 2.xx (_without_ atomics) like mine > 'old' Intel Xeon X3470 (3400 Series Chipset) _without_ atomics you/we > only have the option running Clover or AMD-pro's OpenCL. ROCm isn't an > option with Polaris (20/RX580, here), currently. But we had some > questions about AMD (Polaris) running as eGPU on one of our lists > (mesa-devel / dri-devel / amd-gfx), lately. > > I've CC'ed Alex. > > Have a look here (ROCm 2.2), too: > https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/1086267-radeon-rocm-2-2-released-with-vega-20-optimization-caffe2-multi-gpu-training?p=1086387#post1086387 > > Greetings, > Dieter > > Am 14.03.2019 19:36, schrieb Timur Kristóf: > > On Thu, 2019-03-14 at 20:17 +0200, Mika Westerberg wrote: > >> > > > Here is the output of 'lspci -vv': > >> > > > https://pastebin.com/Qt5RUFVc > >> > > > >> > > The root port (1c.4) says this: > >> > > > >> > > DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, > >> > > OBFF > >> > > Not Supported ARIFwd+ > >> > > AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS- > >> > > > >> > > Not knowing much about AtomicOps but to me this looks like the > >> > > root > >> > > port > >> > > does not support the feature. > >> > > >> > What kind of output should lspci show if the feature were > >> > supported? > >> > >> The AMD card has this: > >> > >> DevCap2: Completion Timeout: Not Supported, TimeoutDis-, > >> LTR+, OBFF Not Supported > >> AtomicOpsCap: 32bit+ 64bit+ 128bitCAS- > >> > >> so I would expect something similar on the root port side as > >> pci_enable_atomic_ops_to_root() fails otherwise with mask of > >> PCI_EXP_DEVCAP2_ATOMIC_COMP32 | PCI_EXP_DEVCAP2_ATOMIC_COMP64 that > >> the > >> AMD driver requests. > >> > >> > As far as I understand the root port is integrated in the CPU, or > >> > in > >> > the chipset maybe? It says it's a Sunrise Point-LP, and I googled > >> > it > >> > but was unable to find a spec sheet. > >> > >> You can find it here: > >> > >> > >> https://www.intel.com/content/www/us/en/products/docs/processors/core/6th-gen-core-pch-u-y-io-datasheet-vol-2.html > >> > >> Pages 845-826 show the DEVCAP2 register for the 1c.4 (D28/F4) and it > >> does not seem to have AtomicOps caps set. > > > > This would be the 8th gen (8550U) but I assume it has similar > > capabilities to the 6th gen. So, it seems that this is a hardware > > limitation of the chipset. > > > > Thanks Mika for clearing that up. > > > > Best regards, > > Tim _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/dri-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-03-15 20:04 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-03-13 18:09 Missing Thunderbolt 3 PCI-E atomics support Timur Kristóf 2019-03-14 10:30 ` Mika Westerberg 2019-03-14 17:26 ` Timur Kristóf 2019-03-14 17:40 ` Mika Westerberg 2019-03-14 17:54 ` Timur Kristóf 2019-03-14 18:17 ` Mika Westerberg 2019-03-14 18:36 ` Timur Kristóf 2019-03-15 19:46 ` Dieter Nützel 2019-03-15 20:03 ` Alex Deucher
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).