* [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-24 19:15 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-24 19:15 UTC (permalink / raw)
To: amd-gfx
KFD has been verified to function on POWER systems (Talos II / Vega 64).
It should be available as a kernel configuration option on these systems.
Signed-off-by: Timothy Pearson <tpearson@raptorengineering.com>
---
drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
index a1a35d4d594b..ba0e68057a89 100644
--- a/drivers/gpu/drm/amd/amdkfd/Kconfig
+++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
@@ -5,7 +5,7 @@
config HSA_AMD
bool "HSA kernel driver for AMD GPU devices"
- depends on DRM_AMDGPU && (X86_64 || ARM64)
+ depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
imply AMD_IOMMU_V2 if X86_64
select MMU_NOTIFIER
help
--
2.20.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-24 19:15 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-24 19:15 UTC (permalink / raw)
To: amd-gfx
KFD has been verified to function on POWER systems (Talos II / Vega 64).
It should be available as a kernel configuration option on these systems.
Signed-off-by: Timothy Pearson <tpearson@raptorengineering.com>
---
drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
index a1a35d4d594b..ba0e68057a89 100644
--- a/drivers/gpu/drm/amd/amdkfd/Kconfig
+++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
@@ -5,7 +5,7 @@
config HSA_AMD
bool "HSA kernel driver for AMD GPU devices"
- depends on DRM_AMDGPU && (X86_64 || ARM64)
+ depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
imply AMD_IOMMU_V2 if X86_64
select MMU_NOTIFIER
help
--
2.20.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 18+ messages in thread
[parent not found: <2068915477.1923611.1574622916022.JavaMail.zimbra-z0qzliK6Om0mgXJStvpl+u2xl0RS173n@public.gmane.org>]
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-25 14:59 ` Alex Deucher
0 siblings, 0 replies; 18+ messages in thread
From: Alex Deucher @ 2019-11-25 14:59 UTC (permalink / raw)
To: Timothy Pearson; +Cc: amd-gfx
On Sun, Nov 24, 2019 at 2:15 PM Timothy Pearson
<tpearson@raptorengineering.com> wrote:
>
> KFD has been verified to function on POWER systems (Talos II / Vega 64).
> It should be available as a kernel configuration option on these systems.
>
> Signed-off-by: Timothy Pearson <tpearson@raptorengineering.com>
Applied. Thanks!
Alex
> ---
> drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
> index a1a35d4d594b..ba0e68057a89 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Kconfig
> +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
> @@ -5,7 +5,7 @@
>
> config HSA_AMD
> bool "HSA kernel driver for AMD GPU devices"
> - depends on DRM_AMDGPU && (X86_64 || ARM64)
> + depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
> imply AMD_IOMMU_V2 if X86_64
> select MMU_NOTIFIER
> help
> --
> 2.20.1
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-25 14:59 ` Alex Deucher
0 siblings, 0 replies; 18+ messages in thread
From: Alex Deucher @ 2019-11-25 14:59 UTC (permalink / raw)
To: Timothy Pearson; +Cc: amd-gfx
On Sun, Nov 24, 2019 at 2:15 PM Timothy Pearson
<tpearson@raptorengineering.com> wrote:
>
> KFD has been verified to function on POWER systems (Talos II / Vega 64).
> It should be available as a kernel configuration option on these systems.
>
> Signed-off-by: Timothy Pearson <tpearson@raptorengineering.com>
Applied. Thanks!
Alex
> ---
> drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
> index a1a35d4d594b..ba0e68057a89 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Kconfig
> +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
> @@ -5,7 +5,7 @@
>
> config HSA_AMD
> bool "HSA kernel driver for AMD GPU devices"
> - depends on DRM_AMDGPU && (X86_64 || ARM64)
> + depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
> imply AMD_IOMMU_V2 if X86_64
> select MMU_NOTIFIER
> help
> --
> 2.20.1
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-25 17:07 ` Felix Kuehling
0 siblings, 0 replies; 18+ messages in thread
From: Felix Kuehling @ 2019-11-25 17:07 UTC (permalink / raw)
To: Timothy Pearson, amd-gfx
Hi Timothy,
Thank you for the patch and for confirming that it works. We did some
experimental work on Power8 a few years ago. I see that Talos II is Power9.
At the time we were working on Power8 we had to add some #ifdef
CONFIG_ACPI guards around some ACPI-specific code in KFD. Do you know to
what extent ACPI is available and working on the Power architecture?
Another problem we ran into with Power, is the physical address map.
System memory can be a physical addresses outside the range accessible
by the GPU. Vega has 44-bit physical addressing. Older Polaris GPUs only
have 40-bits. Did you run into any such problems? Do you need an IOMMU
to make system memory accessible to the GPU?
Regards,
Felix
On 2019-11-24 2:15 p.m., Timothy Pearson wrote:
> KFD has been verified to function on POWER systems (Talos II / Vega 64).
> It should be available as a kernel configuration option on these systems.
>
> Signed-off-by: Timothy Pearson <tpearson@raptorengineering.com>
> ---
> drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
> index a1a35d4d594b..ba0e68057a89 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Kconfig
> +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
> @@ -5,7 +5,7 @@
>
> config HSA_AMD
> bool "HSA kernel driver for AMD GPU devices"
> - depends on DRM_AMDGPU && (X86_64 || ARM64)
> + depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
> imply AMD_IOMMU_V2 if X86_64
> select MMU_NOTIFIER
> help
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-25 17:07 ` Felix Kuehling
0 siblings, 0 replies; 18+ messages in thread
From: Felix Kuehling @ 2019-11-25 17:07 UTC (permalink / raw)
To: Timothy Pearson, amd-gfx
Hi Timothy,
Thank you for the patch and for confirming that it works. We did some
experimental work on Power8 a few years ago. I see that Talos II is Power9.
At the time we were working on Power8 we had to add some #ifdef
CONFIG_ACPI guards around some ACPI-specific code in KFD. Do you know to
what extent ACPI is available and working on the Power architecture?
Another problem we ran into with Power, is the physical address map.
System memory can be a physical addresses outside the range accessible
by the GPU. Vega has 44-bit physical addressing. Older Polaris GPUs only
have 40-bits. Did you run into any such problems? Do you need an IOMMU
to make system memory accessible to the GPU?
Regards,
Felix
On 2019-11-24 2:15 p.m., Timothy Pearson wrote:
> KFD has been verified to function on POWER systems (Talos II / Vega 64).
> It should be available as a kernel configuration option on these systems.
>
> Signed-off-by: Timothy Pearson <tpearson@raptorengineering.com>
> ---
> drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
> index a1a35d4d594b..ba0e68057a89 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Kconfig
> +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
> @@ -5,7 +5,7 @@
>
> config HSA_AMD
> bool "HSA kernel driver for AMD GPU devices"
> - depends on DRM_AMDGPU && (X86_64 || ARM64)
> + depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
> imply AMD_IOMMU_V2 if X86_64
> select MMU_NOTIFIER
> help
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <44175d39-e507-83f9-6596-cbde108dd65a-5C7GfCeVMHo@public.gmane.org>]
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-25 21:06 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-25 21:06 UTC (permalink / raw)
To: Felix Kuehling; +Cc: amd-gfx
----- Original Message -----
> From: "Felix Kuehling" <felix.kuehling@amd.com>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "amd-gfx" <amd-gfx@lists.freedesktop.org>
> Sent: Monday, November 25, 2019 11:07:31 AM
> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
> Hi Timothy,
>
> Thank you for the patch and for confirming that it works. We did some
> experimental work on Power8 a few years ago. I see that Talos II is Power9.
>
> At the time we were working on Power8 we had to add some #ifdef
> CONFIG_ACPI guards around some ACPI-specific code in KFD. Do you know to
> what extent ACPI is available and working on the Power architecture?
>
> Another problem we ran into with Power, is the physical address map.
> System memory can be a physical addresses outside the range accessible
> by the GPU. Vega has 44-bit physical addressing. Older Polaris GPUs only
> have 40-bits. Did you run into any such problems? Do you need an IOMMU
> to make system memory accessible to the GPU?
>
> Regards,
> Felix
Yes, we are POWER9. It looks like the ACPI guards are no longer required; as you have surmised, POWER does not use ACPI (the equivalent is OPAL, which is a different interface entirely). What were the APCI calls used for? There may be OPAL equivalents that could be added in to replace them and provide similar functionality.
Kernel 5.4 enables a > 32-bit and <=64-bit bypass mode for POWER. This is one reason we came back and revisited the KFD/ROCm functionality on POWER; as it turns out, after fixing up the userspace tools KFD is indeed functional on POWER with 5.4-rc8 and above. My understanding is that the POWER IOMMU is used as a lightweight translation layer between the 64-bit host and the 40/44-bit GPU.
I'm working on getting a Debian PPA set up for POWER to make the userspace tools easier to obtain for testing, but progress is slow due to lack of Debian source packages. Probably the easiest way to replicate / test this with HIP is to use the AOMP repository with my modifications; pull requests are already in place on Github for most of the userspace tooling updates.
Thank you!
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-25 21:06 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-25 21:06 UTC (permalink / raw)
To: Felix Kuehling; +Cc: amd-gfx
----- Original Message -----
> From: "Felix Kuehling" <felix.kuehling@amd.com>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "amd-gfx" <amd-gfx@lists.freedesktop.org>
> Sent: Monday, November 25, 2019 11:07:31 AM
> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
> Hi Timothy,
>
> Thank you for the patch and for confirming that it works. We did some
> experimental work on Power8 a few years ago. I see that Talos II is Power9.
>
> At the time we were working on Power8 we had to add some #ifdef
> CONFIG_ACPI guards around some ACPI-specific code in KFD. Do you know to
> what extent ACPI is available and working on the Power architecture?
>
> Another problem we ran into with Power, is the physical address map.
> System memory can be a physical addresses outside the range accessible
> by the GPU. Vega has 44-bit physical addressing. Older Polaris GPUs only
> have 40-bits. Did you run into any such problems? Do you need an IOMMU
> to make system memory accessible to the GPU?
>
> Regards,
> Felix
Yes, we are POWER9. It looks like the ACPI guards are no longer required; as you have surmised, POWER does not use ACPI (the equivalent is OPAL, which is a different interface entirely). What were the APCI calls used for? There may be OPAL equivalents that could be added in to replace them and provide similar functionality.
Kernel 5.4 enables a > 32-bit and <=64-bit bypass mode for POWER. This is one reason we came back and revisited the KFD/ROCm functionality on POWER; as it turns out, after fixing up the userspace tools KFD is indeed functional on POWER with 5.4-rc8 and above. My understanding is that the POWER IOMMU is used as a lightweight translation layer between the 64-bit host and the 40/44-bit GPU.
I'm working on getting a Debian PPA set up for POWER to make the userspace tools easier to obtain for testing, but progress is slow due to lack of Debian source packages. Probably the easiest way to replicate / test this with HIP is to use the AOMP repository with my modifications; pull requests are already in place on Github for most of the userspace tooling updates.
Thank you!
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <535694171.2118031.1574716002431.JavaMail.zimbra-z0qzliK6Om0mgXJStvpl+u2xl0RS173n@public.gmane.org>]
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-25 21:34 ` Felix Kuehling
0 siblings, 0 replies; 18+ messages in thread
From: Felix Kuehling @ 2019-11-25 21:34 UTC (permalink / raw)
To: Timothy Pearson; +Cc: amd-gfx
On 2019-11-25 4:06 p.m., Timothy Pearson wrote:
>
> ----- Original Message -----
>> From: "Felix Kuehling" <felix.kuehling@amd.com>
>> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "amd-gfx" <amd-gfx@lists.freedesktop.org>
>> Sent: Monday, November 25, 2019 11:07:31 AM
>> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
>> Hi Timothy,
>>
>> Thank you for the patch and for confirming that it works. We did some
>> experimental work on Power8 a few years ago. I see that Talos II is Power9.
>>
>> At the time we were working on Power8 we had to add some #ifdef
>> CONFIG_ACPI guards around some ACPI-specific code in KFD. Do you know to
>> what extent ACPI is available and working on the Power architecture?
>>
>> Another problem we ran into with Power, is the physical address map.
>> System memory can be a physical addresses outside the range accessible
>> by the GPU. Vega has 44-bit physical addressing. Older Polaris GPUs only
>> have 40-bits. Did you run into any such problems? Do you need an IOMMU
>> to make system memory accessible to the GPU?
>>
>> Regards,
>> Felix
> Yes, we are POWER9. It looks like the ACPI guards are no longer required; as you have surmised, POWER does not use ACPI (the equivalent is OPAL, which is a different interface entirely). What were the APCI calls used for? There may be OPAL equivalents that could be added in to replace them and provide similar functionality.
There are some ACPI calls (e.g. acpi_get_table) in kfd_crat.c for
getting a CRAT table from ACPI. This is only useful for AMD APUs, which
are x86_64. We don't need this for discrete GPUs because on non-APU
systems there is no CRAT table and we build our own. If you can compile
the code without problems on Power and with CONFIG_ACPI not defined,
then I guess this is no longer an issue.
> Kernel 5.4 enables a > 32-bit and <=64-bit bypass mode for POWER. This is one reason we came back and revisited the KFD/ROCm functionality on POWER; as it turns out, after fixing up the userspace tools KFD is indeed functional on POWER with 5.4-rc8 and above. My understanding is that the POWER IOMMU is used as a lightweight translation layer between the 64-bit host and the 40/44-bit GPU.
>
> I'm working on getting a Debian PPA set up for POWER to make the userspace tools easier to obtain for testing, but progress is slow due to lack of Debian source packages. Probably the easiest way to replicate / test this with HIP is to use the AOMP repository with my modifications; pull requests are already in place on Github for most of the userspace tooling updates.
>
> Thank you!
Thanks,
Felix
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-25 21:34 ` Felix Kuehling
0 siblings, 0 replies; 18+ messages in thread
From: Felix Kuehling @ 2019-11-25 21:34 UTC (permalink / raw)
To: Timothy Pearson; +Cc: amd-gfx
On 2019-11-25 4:06 p.m., Timothy Pearson wrote:
>
> ----- Original Message -----
>> From: "Felix Kuehling" <felix.kuehling@amd.com>
>> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "amd-gfx" <amd-gfx@lists.freedesktop.org>
>> Sent: Monday, November 25, 2019 11:07:31 AM
>> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
>> Hi Timothy,
>>
>> Thank you for the patch and for confirming that it works. We did some
>> experimental work on Power8 a few years ago. I see that Talos II is Power9.
>>
>> At the time we were working on Power8 we had to add some #ifdef
>> CONFIG_ACPI guards around some ACPI-specific code in KFD. Do you know to
>> what extent ACPI is available and working on the Power architecture?
>>
>> Another problem we ran into with Power, is the physical address map.
>> System memory can be a physical addresses outside the range accessible
>> by the GPU. Vega has 44-bit physical addressing. Older Polaris GPUs only
>> have 40-bits. Did you run into any such problems? Do you need an IOMMU
>> to make system memory accessible to the GPU?
>>
>> Regards,
>> Felix
> Yes, we are POWER9. It looks like the ACPI guards are no longer required; as you have surmised, POWER does not use ACPI (the equivalent is OPAL, which is a different interface entirely). What were the APCI calls used for? There may be OPAL equivalents that could be added in to replace them and provide similar functionality.
There are some ACPI calls (e.g. acpi_get_table) in kfd_crat.c for
getting a CRAT table from ACPI. This is only useful for AMD APUs, which
are x86_64. We don't need this for discrete GPUs because on non-APU
systems there is no CRAT table and we build our own. If you can compile
the code without problems on Power and with CONFIG_ACPI not defined,
then I guess this is no longer an issue.
> Kernel 5.4 enables a > 32-bit and <=64-bit bypass mode for POWER. This is one reason we came back and revisited the KFD/ROCm functionality on POWER; as it turns out, after fixing up the userspace tools KFD is indeed functional on POWER with 5.4-rc8 and above. My understanding is that the POWER IOMMU is used as a lightweight translation layer between the 64-bit host and the 40/44-bit GPU.
>
> I'm working on getting a Debian PPA set up for POWER to make the userspace tools easier to obtain for testing, but progress is slow due to lack of Debian source packages. Probably the easiest way to replicate / test this with HIP is to use the AOMP repository with my modifications; pull requests are already in place on Github for most of the userspace tooling updates.
>
> Thank you!
Thanks,
Felix
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <5dcfdba7-dc14-99d4-d6e5-fc1768fcc5c0-5C7GfCeVMHo@public.gmane.org>]
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-25 21:52 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-25 21:52 UTC (permalink / raw)
To: Felix Kuehling; +Cc: amd-gfx
----- Original Message -----
> From: "Felix Kuehling" <felix.kuehling@amd.com>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>
> Cc: "amd-gfx" <amd-gfx@lists.freedesktop.org>
> Sent: Monday, November 25, 2019 3:34:20 PM
> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
> On 2019-11-25 4:06 p.m., Timothy Pearson wrote:
>>
>> ----- Original Message -----
>>> From: "Felix Kuehling" <felix.kuehling@amd.com>
>>> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "amd-gfx"
>>> <amd-gfx@lists.freedesktop.org>
>>> Sent: Monday, November 25, 2019 11:07:31 AM
>>> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
>>> Hi Timothy,
>>>
>>> Thank you for the patch and for confirming that it works. We did some
>>> experimental work on Power8 a few years ago. I see that Talos II is Power9.
>>>
>>> At the time we were working on Power8 we had to add some #ifdef
>>> CONFIG_ACPI guards around some ACPI-specific code in KFD. Do you know to
>>> what extent ACPI is available and working on the Power architecture?
>>>
>>> Another problem we ran into with Power, is the physical address map.
>>> System memory can be a physical addresses outside the range accessible
>>> by the GPU. Vega has 44-bit physical addressing. Older Polaris GPUs only
>>> have 40-bits. Did you run into any such problems? Do you need an IOMMU
>>> to make system memory accessible to the GPU?
>>>
>>> Regards,
>>> Felix
>> Yes, we are POWER9. It looks like the ACPI guards are no longer required; as
>> you have surmised, POWER does not use ACPI (the equivalent is OPAL, which is a
>> different interface entirely). What were the APCI calls used for? There may
>> be OPAL equivalents that could be added in to replace them and provide similar
>> functionality.
>
> There are some ACPI calls (e.g. acpi_get_table) in kfd_crat.c for
> getting a CRAT table from ACPI. This is only useful for AMD APUs, which
> are x86_64. We don't need this for discrete GPUs because on non-APU
> systems there is no CRAT table and we build our own. If you can compile
> the code without problems on Power and with CONFIG_ACPI not defined,
> then I guess this is no longer an issue.
Sounds reasonable -- yes, it compiles without issue so I think we're good to go.
>
>> Kernel 5.4 enables a > 32-bit and <=64-bit bypass mode for POWER. This is one
>> reason we came back and revisited the KFD/ROCm functionality on POWER; as it
>> turns out, after fixing up the userspace tools KFD is indeed functional on
>> POWER with 5.4-rc8 and above. My understanding is that the POWER IOMMU is used
>> as a lightweight translation layer between the 64-bit host and the 40/44-bit
>> GPU.
>>
>> I'm working on getting a Debian PPA set up for POWER to make the userspace tools
>> easier to obtain for testing, but progress is slow due to lack of Debian source
>> packages. Probably the easiest way to replicate / test this with HIP is to use
>> the AOMP repository with my modifications; pull requests are already in place
>> on Github for most of the userspace tooling updates.
>>
>> Thank you!
> Thanks,
> Felix
>
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-25 21:52 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-25 21:52 UTC (permalink / raw)
To: Felix Kuehling; +Cc: amd-gfx
----- Original Message -----
> From: "Felix Kuehling" <felix.kuehling@amd.com>
> To: "Timothy Pearson" <tpearson@raptorengineering.com>
> Cc: "amd-gfx" <amd-gfx@lists.freedesktop.org>
> Sent: Monday, November 25, 2019 3:34:20 PM
> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
> On 2019-11-25 4:06 p.m., Timothy Pearson wrote:
>>
>> ----- Original Message -----
>>> From: "Felix Kuehling" <felix.kuehling@amd.com>
>>> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "amd-gfx"
>>> <amd-gfx@lists.freedesktop.org>
>>> Sent: Monday, November 25, 2019 11:07:31 AM
>>> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
>>> Hi Timothy,
>>>
>>> Thank you for the patch and for confirming that it works. We did some
>>> experimental work on Power8 a few years ago. I see that Talos II is Power9.
>>>
>>> At the time we were working on Power8 we had to add some #ifdef
>>> CONFIG_ACPI guards around some ACPI-specific code in KFD. Do you know to
>>> what extent ACPI is available and working on the Power architecture?
>>>
>>> Another problem we ran into with Power, is the physical address map.
>>> System memory can be a physical addresses outside the range accessible
>>> by the GPU. Vega has 44-bit physical addressing. Older Polaris GPUs only
>>> have 40-bits. Did you run into any such problems? Do you need an IOMMU
>>> to make system memory accessible to the GPU?
>>>
>>> Regards,
>>> Felix
>> Yes, we are POWER9. It looks like the ACPI guards are no longer required; as
>> you have surmised, POWER does not use ACPI (the equivalent is OPAL, which is a
>> different interface entirely). What were the APCI calls used for? There may
>> be OPAL equivalents that could be added in to replace them and provide similar
>> functionality.
>
> There are some ACPI calls (e.g. acpi_get_table) in kfd_crat.c for
> getting a CRAT table from ACPI. This is only useful for AMD APUs, which
> are x86_64. We don't need this for discrete GPUs because on non-APU
> systems there is no CRAT table and we build our own. If you can compile
> the code without problems on Power and with CONFIG_ACPI not defined,
> then I guess this is no longer an issue.
Sounds reasonable -- yes, it compiles without issue so I think we're good to go.
>
>> Kernel 5.4 enables a > 32-bit and <=64-bit bypass mode for POWER. This is one
>> reason we came back and revisited the KFD/ROCm functionality on POWER; as it
>> turns out, after fixing up the userspace tools KFD is indeed functional on
>> POWER with 5.4-rc8 and above. My understanding is that the POWER IOMMU is used
>> as a lightweight translation layer between the 64-bit host and the 40/44-bit
>> GPU.
>>
>> I'm working on getting a Debian PPA set up for POWER to make the userspace tools
>> easier to obtain for testing, but progress is slow due to lack of Debian source
>> packages. Probably the easiest way to replicate / test this with HIP is to use
>> the AOMP repository with my modifications; pull requests are already in place
>> on Github for most of the userspace tooling updates.
>>
>> Thank you!
> Thanks,
> Felix
>
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-24 19:10 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-24 19:10 UTC (permalink / raw)
To: amd-gfx
KFD has been verified to function on POWER systems (Talos II / Vega 64).
It should be available as a kernel configuration option on these systems.
---
drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
index a1a35d4d594b..ba0e68057a89 100644
--- a/drivers/gpu/drm/amd/amdkfd/Kconfig
+++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
@@ -5,7 +5,7 @@
config HSA_AMD
bool "HSA kernel driver for AMD GPU devices"
- depends on DRM_AMDGPU && (X86_64 || ARM64)
+ depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
imply AMD_IOMMU_V2 if X86_64
select MMU_NOTIFIER
help
--
2.20.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-24 19:10 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-24 19:10 UTC (permalink / raw)
To: amd-gfx
KFD has been verified to function on POWER systems (Talos II / Vega 64).
It should be available as a kernel configuration option on these systems.
---
drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
index a1a35d4d594b..ba0e68057a89 100644
--- a/drivers/gpu/drm/amd/amdkfd/Kconfig
+++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
@@ -5,7 +5,7 @@
config HSA_AMD
bool "HSA kernel driver for AMD GPU devices"
- depends on DRM_AMDGPU && (X86_64 || ARM64)
+ depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
imply AMD_IOMMU_V2 if X86_64
select MMU_NOTIFIER
help
--
2.20.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 18+ messages in thread
[parent not found: <443989900.1923547.1574622652551.JavaMail.zimbra-z0qzliK6Om0mgXJStvpl+u2xl0RS173n@public.gmane.org>]
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-24 19:19 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-24 19:19 UTC (permalink / raw)
To: amd-gfx
Sorry for the spam -- I had thought I was signed up for this list but apparently was not, and resent a couple times.
----- Original Message -----
> From: "Timothy Pearson" <tpearson@raptorengineering.com>
> To: "amd-gfx" <amd-gfx@lists.freedesktop.org>
> Sent: Sunday, November 24, 2019 1:10:52 PM
> Subject: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
> KFD has been verified to function on POWER systems (Talos II / Vega 64).
> It should be available as a kernel configuration option on these systems.
> ---
> drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig
> b/drivers/gpu/drm/amd/amdkfd/Kconfig
> index a1a35d4d594b..ba0e68057a89 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Kconfig
> +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
> @@ -5,7 +5,7 @@
>
> config HSA_AMD
> bool "HSA kernel driver for AMD GPU devices"
> - depends on DRM_AMDGPU && (X86_64 || ARM64)
> + depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
> imply AMD_IOMMU_V2 if X86_64
> select MMU_NOTIFIER
> help
> --
> 2.20.1
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-24 19:19 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-24 19:19 UTC (permalink / raw)
To: amd-gfx
Sorry for the spam -- I had thought I was signed up for this list but apparently was not, and resent a couple times.
----- Original Message -----
> From: "Timothy Pearson" <tpearson@raptorengineering.com>
> To: "amd-gfx" <amd-gfx@lists.freedesktop.org>
> Sent: Sunday, November 24, 2019 1:10:52 PM
> Subject: [PATCH 1/1] amdgpu: Enable KFD on POWER systems
> KFD has been verified to function on POWER systems (Talos II / Vega 64).
> It should be available as a kernel configuration option on these systems.
> ---
> drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig
> b/drivers/gpu/drm/amd/amdkfd/Kconfig
> index a1a35d4d594b..ba0e68057a89 100644
> --- a/drivers/gpu/drm/amd/amdkfd/Kconfig
> +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
> @@ -5,7 +5,7 @@
>
> config HSA_AMD
> bool "HSA kernel driver for AMD GPU devices"
> - depends on DRM_AMDGPU && (X86_64 || ARM64)
> + depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
> imply AMD_IOMMU_V2 if X86_64
> select MMU_NOTIFIER
> help
> --
> 2.20.1
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-24 19:09 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-24 19:09 UTC (permalink / raw)
To: amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW
KFD has been verified to function on POWER systems (Talos II / Vega 64).
It should be available as a kernel configuration option on these systems.
---
drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
index a1a35d4d594b..ba0e68057a89 100644
--- a/drivers/gpu/drm/amd/amdkfd/Kconfig
+++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
@@ -5,7 +5,7 @@
config HSA_AMD
bool "HSA kernel driver for AMD GPU devices"
- depends on DRM_AMDGPU && (X86_64 || ARM64)
+ depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
imply AMD_IOMMU_V2 if X86_64
select MMU_NOTIFIER
help
--
2.20.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 1/1] amdgpu: Enable KFD on POWER systems
@ 2019-11-24 19:09 ` Timothy Pearson
0 siblings, 0 replies; 18+ messages in thread
From: Timothy Pearson @ 2019-11-24 19:09 UTC (permalink / raw)
To: amd-gfx
KFD has been verified to function on POWER systems (Talos II / Vega 64).
It should be available as a kernel configuration option on these systems.
---
drivers/gpu/drm/amd/amdkfd/Kconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig
index a1a35d4d594b..ba0e68057a89 100644
--- a/drivers/gpu/drm/amd/amdkfd/Kconfig
+++ b/drivers/gpu/drm/amd/amdkfd/Kconfig
@@ -5,7 +5,7 @@
config HSA_AMD
bool "HSA kernel driver for AMD GPU devices"
- depends on DRM_AMDGPU && (X86_64 || ARM64)
+ depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64)
imply AMD_IOMMU_V2 if X86_64
select MMU_NOTIFIER
help
--
2.20.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 18+ messages in thread
end of thread, other threads:[~2019-11-25 21:52 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-24 19:15 [PATCH 1/1] amdgpu: Enable KFD on POWER systems Timothy Pearson
2019-11-24 19:15 ` Timothy Pearson
[not found] ` <2068915477.1923611.1574622916022.JavaMail.zimbra-z0qzliK6Om0mgXJStvpl+u2xl0RS173n@public.gmane.org>
2019-11-25 14:59 ` Alex Deucher
2019-11-25 14:59 ` Alex Deucher
2019-11-25 17:07 ` Felix Kuehling
2019-11-25 17:07 ` Felix Kuehling
[not found] ` <44175d39-e507-83f9-6596-cbde108dd65a-5C7GfCeVMHo@public.gmane.org>
2019-11-25 21:06 ` Timothy Pearson
2019-11-25 21:06 ` Timothy Pearson
[not found] ` <535694171.2118031.1574716002431.JavaMail.zimbra-z0qzliK6Om0mgXJStvpl+u2xl0RS173n@public.gmane.org>
2019-11-25 21:34 ` Felix Kuehling
2019-11-25 21:34 ` Felix Kuehling
[not found] ` <5dcfdba7-dc14-99d4-d6e5-fc1768fcc5c0-5C7GfCeVMHo@public.gmane.org>
2019-11-25 21:52 ` Timothy Pearson
2019-11-25 21:52 ` Timothy Pearson
-- strict thread matches above, loose matches on Subject: below --
2019-11-24 19:10 Timothy Pearson
2019-11-24 19:10 ` Timothy Pearson
[not found] ` <443989900.1923547.1574622652551.JavaMail.zimbra-z0qzliK6Om0mgXJStvpl+u2xl0RS173n@public.gmane.org>
2019-11-24 19:19 ` Timothy Pearson
2019-11-24 19:19 ` Timothy Pearson
2019-11-24 19:09 Timothy Pearson
2019-11-24 19:09 ` Timothy Pearson
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.