PCIe3 atomics requirement for amdkfd

PCIe3 atomics requirement for amdkfd

[Tom Stellard]

Hi,

How hard of a requirement is PCIe3 atomics for dGPUs with the amdkfd
kernel driver?  Is it possible to make modifications to the runtime/kernel
driver to drop this requirement?

-Tom
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: PCIe3 atomics requirement for amdkfd

[Felix Kühling]

[-- Attachment #1.1.1: Type: text/plain, Size: 726 bytes --]

As I understand it, it would require changes in the ROCr Runtime and in
the firmware (MEC microcode). It also changes the programming model, so
it may affect certain applications or higher level language runtimes
that rely on atomic operations.

Regards,
  Felix


Am 19.12.2017 um 16:04 schrieb Tom Stellard:
> Hi,
>
> How hard of a requirement is PCIe3 atomics for dGPUs with the amdkfd
> kernel driver?  Is it possible to make modifications to the runtime/kernel
> driver to drop this requirement?
>
> -Tom
> _______________________________________________
> amd-gfx mailing list
> amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: PCIe3 atomics requirement for amdkfd

[Alexander Kjeldaas]

[-- Attachment #1.1: Type: text/plain, Size: 1147 bytes --]

 Can the atomic ops work on the modified risers used for mining?

On Dec 23, 2017 16:41, "Felix Kühling" <felix.kuehling-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> As I understand it, it would require changes in the ROCr Runtime and in
> the firmware (MEC microcode). It also changes the programming model, so
> it may affect certain applications or higher level language runtimes
> that rely on atomic operations.
>
> Regards,
>   Felix
>
>
> Am 19.12.2017 um 16:04 schrieb Tom Stellard:
> > Hi,
> >
> > How hard of a requirement is PCIe3 atomics for dGPUs with the amdkfd
> > kernel driver?  Is it possible to make modifications to the
> runtime/kernel
> > driver to drop this requirement?
> >
> > -Tom
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>
>
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>
>

[-- Attachment #1.2: Type: text/html, Size: 1933 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: PCIe3 atomics requirement for amdkfd

[Felix Kühling]

[-- Attachment #1.1.1: Type: text/plain, Size: 1746 bytes --]

Am 23.12.2017 um 17:26 schrieb Alexander Kjeldaas:
>  Can the atomic ops work on the modified risers used for mining?

If the PCI bridge on the riser supports PCIe 3 atomic routing, then yes.

Regards,
  Felix

>
> On Dec 23, 2017 16:41, "Felix Kühling" <felix.kuehling-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> <mailto:felix.kuehling-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> wrote:
>
>     As I understand it, it would require changes in the ROCr Runtime
>     and in
>     the firmware (MEC microcode). It also changes the programming
>     model, so
>     it may affect certain applications or higher level language runtimes
>     that rely on atomic operations.
>
>     Regards,
>       Felix
>
>
>     Am 19.12.2017 um 16:04 schrieb Tom Stellard:
>     > Hi,
>     >
>     > How hard of a requirement is PCIe3 atomics for dGPUs with the amdkfd
>     > kernel driver?  Is it possible to make modifications to the
>     runtime/kernel
>     > driver to drop this requirement?
>     >
>     > -Tom
>     > _______________________________________________
>     > amd-gfx mailing list
>     > amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org <mailto:amd-gfx-PD4FTy7X32lNgt0PjOBp93rCq3LdnpKM@public.gmane.orgrg>
>     > https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>     <https://lists.freedesktop.org/mailman/listinfo/amd-gfx>
>
>
>
>     _______________________________________________
>     amd-gfx mailing list
>     amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org <mailto:amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org>
>     https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>     <https://lists.freedesktop.org/mailman/listinfo/amd-gfx>
>



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 154 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: PCIe3 atomics requirement for amdkfd

[Luke A. Guest]

Hi,

I have to agree here. At least there should be a non-PCIe-3.0 pathway
which implements them on the CPU, I mean, they're fairly simple atomics,
CAS, SWAP, FetchAdd.

What AMD have actually done is royally screwed over anyone with an FX
chipset, i.e. no OpenCL - the open source AMD one requires ROCm, they
abandoned Clover, maybe PoCL will work? Who knows.


On 19/12/17 15:04, Tom Stellard wrote:
> Hi,
>
> How hard of a requirement is PCIe3 atomics for dGPUs with the amdkfd
> kernel driver?  Is it possible to make modifications to the runtime/kernel
> driver to drop this requirement?
>
> -Tom
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: PCIe3 atomics requirement for amdkfd

[Bridgman, John]

Let's separate out OpenCL from HCC/HIP and the rest of the ROCm stack. 

We are working on a solution to deliver OpenCL without requiring atomics, but not the rest of the ROCm stack.

>-----Original Message-----
>From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf Of
>Luke A. Guest
>Sent: Monday, December 25, 2017 9:55 AM
>To: amd-gfx@lists.freedesktop.org
>Subject: Re: PCIe3 atomics requirement for amdkfd
>
>Hi,
>
>I have to agree here. At least there should be a non-PCIe-3.0 pathway which
>implements them on the CPU, I mean, they're fairly simple atomics, CAS,
>SWAP, FetchAdd.
>
>What AMD have actually done is royally screwed over anyone with an FX
>chipset, i.e. no OpenCL - the open source AMD one requires ROCm, they
>abandoned Clover, maybe PoCL will work? Who knows.
>
>
>On 19/12/17 15:04, Tom Stellard wrote:
>> Hi,
>>
>> How hard of a requirement is PCIe3 atomics for dGPUs with the amdkfd
>> kernel driver?  Is it possible to make modifications to the
>> runtime/kernel driver to drop this requirement?
>>
>> -Tom
>> _______________________________________________
>> amd-gfx mailing list
>> amd-gfx@lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>
>_______________________________________________
>amd-gfx mailing list
>amd-gfx@lists.freedesktop.org
>https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: PCIe3 atomics requirement for amdkfd

[Tom Stellard]

On 12/23/2017 07:40 AM, Felix Kühling wrote:
> As I understand it, it would require changes in the ROCr Runtime and in
> the firmware (MEC microcode). It also changes the programming model, so
> it may affect certain applications or higher level language runtimes
> that rely on atomic operations.
> 

How does the MEC microcode know that it is running a ROCm workload as opposed
to a graphics workload that doesn't require PCIe3 atomics.  Is there a specific
configuration bit that is set to indicate the ROCm programming model is needed?

-Tom

> Regards,
>   Felix
> 
> 
> Am 19.12.2017 um 16:04 schrieb Tom Stellard:
>> Hi,
>>
>> How hard of a requirement is PCIe3 atomics for dGPUs with the amdkfd
>> kernel driver?  Is it possible to make modifications to the runtime/kernel
>> driver to drop this requirement?
>>
>> -Tom
>> _______________________________________________
>> amd-gfx mailing list
>> amd-gfx@lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> 
> 

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: PCIe3 atomics requirement for amdkfd

[Liu, Shaoyun]

I think currently atomic  Ops are only used in AQL package which is only available for ROCm , graphics workload will not use AQL package. 

Regards
Shaoyun.liu

-----Original Message-----
From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf Of Tom Stellard
Sent: Wednesday, January 03, 2018 9:57 AM
To: Felix Kühling; amd-gfx@lists.freedesktop.org
Cc: Kuehling, Felix
Subject: Re: PCIe3 atomics requirement for amdkfd

On 12/23/2017 07:40 AM, Felix Kühling wrote:
> As I understand it, it would require changes in the ROCr Runtime and 
> in the firmware (MEC microcode). It also changes the programming 
> model, so it may affect certain applications or higher level language 
> runtimes that rely on atomic operations.
> 

How does the MEC microcode know that it is running a ROCm workload as opposed to a graphics workload that doesn't require PCIe3 atomics.  Is there a specific configuration bit that is set to indicate the ROCm programming model is needed?

-Tom

> Regards,
>   Felix
> 
> 
> Am 19.12.2017 um 16:04 schrieb Tom Stellard:
>> Hi,
>>
>> How hard of a requirement is PCIe3 atomics for dGPUs with the amdkfd 
>> kernel driver?  Is it possible to make modifications to the 
>> runtime/kernel driver to drop this requirement?
>>
>> -Tom
>> _______________________________________________
>> amd-gfx mailing list
>> amd-gfx@lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
> 
> 

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: PCIe3 atomics requirement for amdkfd

[Bridgman, John]

Agreed - MEC microcode uses atomics when the queue type is set to AQL (rather than PM4).

>-----Original Message-----
>From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf Of
>Liu, Shaoyun
>Sent: Wednesday, January 03, 2018 11:24 AM
>To: tstellar@redhat.com; Felix Kühling; amd-gfx@lists.freedesktop.org
>Cc: Kuehling, Felix
>Subject: RE: PCIe3 atomics requirement for amdkfd
>
>I think currently atomic  Ops are only used in AQL package which is only
>available for ROCm , graphics workload will not use AQL package.
>
>Regards
>Shaoyun.liu
>
>-----Original Message-----
>From: amd-gfx [mailto:amd-gfx-bounces@lists.freedesktop.org] On Behalf Of
>Tom Stellard
>Sent: Wednesday, January 03, 2018 9:57 AM
>To: Felix Kühling; amd-gfx@lists.freedesktop.org
>Cc: Kuehling, Felix
>Subject: Re: PCIe3 atomics requirement for amdkfd
>
>On 12/23/2017 07:40 AM, Felix Kühling wrote:
>> As I understand it, it would require changes in the ROCr Runtime and
>> in the firmware (MEC microcode). It also changes the programming
>> model, so it may affect certain applications or higher level language
>> runtimes that rely on atomic operations.
>>
>
>How does the MEC microcode know that it is running a ROCm workload as
>opposed to a graphics workload that doesn't require PCIe3 atomics.  Is there a
>specific configuration bit that is set to indicate the ROCm programming model
>is needed?
>
>-Tom
>
>> Regards,
>>   Felix
>>
>>
>> Am 19.12.2017 um 16:04 schrieb Tom Stellard:
>>> Hi,
>>>
>>> How hard of a requirement is PCIe3 atomics for dGPUs with the amdkfd
>>> kernel driver?  Is it possible to make modifications to the
>>> runtime/kernel driver to drop this requirement?
>>>
>>> -Tom
>>> _______________________________________________
>>> amd-gfx mailing list
>>> amd-gfx@lists.freedesktop.org
>>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>>
>>
>
>_______________________________________________
>amd-gfx mailing list
>amd-gfx@lists.freedesktop.org
>https://lists.freedesktop.org/mailman/listinfo/amd-gfx
>_______________________________________________
>amd-gfx mailing list
>amd-gfx@lists.freedesktop.org
>https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx