From: Felix Kuehling <felix.kuehling-5C7GfCeVMHo@public.gmane.org> To: Timothy Pearson <tpearson-z0qzliK6Om0mgXJStvpl+vpXobYPEAuW@public.gmane.org> Cc: amd-gfx <amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems Date: Mon, 25 Nov 2019 16:34:20 -0500 [thread overview] Message-ID: <5dcfdba7-dc14-99d4-d6e5-fc1768fcc5c0@amd.com> (raw) In-Reply-To: <535694171.2118031.1574716002431.JavaMail.zimbra-z0qzliK6Om0mgXJStvpl+u2xl0RS173n@public.gmane.org> On 2019-11-25 4:06 p.m., Timothy Pearson wrote: > > ----- Original Message ----- >> From: "Felix Kuehling" <felix.kuehling@amd.com> >> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "amd-gfx" <amd-gfx@lists.freedesktop.org> >> Sent: Monday, November 25, 2019 11:07:31 AM >> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems >> Hi Timothy, >> >> Thank you for the patch and for confirming that it works. We did some >> experimental work on Power8 a few years ago. I see that Talos II is Power9. >> >> At the time we were working on Power8 we had to add some #ifdef >> CONFIG_ACPI guards around some ACPI-specific code in KFD. Do you know to >> what extent ACPI is available and working on the Power architecture? >> >> Another problem we ran into with Power, is the physical address map. >> System memory can be a physical addresses outside the range accessible >> by the GPU. Vega has 44-bit physical addressing. Older Polaris GPUs only >> have 40-bits. Did you run into any such problems? Do you need an IOMMU >> to make system memory accessible to the GPU? >> >> Regards, >> Felix > Yes, we are POWER9. It looks like the ACPI guards are no longer required; as you have surmised, POWER does not use ACPI (the equivalent is OPAL, which is a different interface entirely). What were the APCI calls used for? There may be OPAL equivalents that could be added in to replace them and provide similar functionality. There are some ACPI calls (e.g. acpi_get_table) in kfd_crat.c for getting a CRAT table from ACPI. This is only useful for AMD APUs, which are x86_64. We don't need this for discrete GPUs because on non-APU systems there is no CRAT table and we build our own. If you can compile the code without problems on Power and with CONFIG_ACPI not defined, then I guess this is no longer an issue. > Kernel 5.4 enables a > 32-bit and <=64-bit bypass mode for POWER. This is one reason we came back and revisited the KFD/ROCm functionality on POWER; as it turns out, after fixing up the userspace tools KFD is indeed functional on POWER with 5.4-rc8 and above. My understanding is that the POWER IOMMU is used as a lightweight translation layer between the 64-bit host and the 40/44-bit GPU. > > I'm working on getting a Debian PPA set up for POWER to make the userspace tools easier to obtain for testing, but progress is slow due to lack of Debian source packages. Probably the easiest way to replicate / test this with HIP is to use the AOMP repository with my modifications; pull requests are already in place on Github for most of the userspace tooling updates. > > Thank you! Thanks, Felix _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx
WARNING: multiple messages have this Message-ID (diff)
From: Felix Kuehling <felix.kuehling@amd.com> To: Timothy Pearson <tpearson@raptorengineering.com> Cc: amd-gfx <amd-gfx@lists.freedesktop.org> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems Date: Mon, 25 Nov 2019 16:34:20 -0500 [thread overview] Message-ID: <5dcfdba7-dc14-99d4-d6e5-fc1768fcc5c0@amd.com> (raw) Message-ID: <20191125213420.AoPIqzI9D_hq1OQDyozkGc3rtY_LZH3PIOfkxpbCxkM@z> (raw) In-Reply-To: <535694171.2118031.1574716002431.JavaMail.zimbra@raptorengineeringinc.com> On 2019-11-25 4:06 p.m., Timothy Pearson wrote: > > ----- Original Message ----- >> From: "Felix Kuehling" <felix.kuehling@amd.com> >> To: "Timothy Pearson" <tpearson@raptorengineering.com>, "amd-gfx" <amd-gfx@lists.freedesktop.org> >> Sent: Monday, November 25, 2019 11:07:31 AM >> Subject: Re: [PATCH 1/1] amdgpu: Enable KFD on POWER systems >> Hi Timothy, >> >> Thank you for the patch and for confirming that it works. We did some >> experimental work on Power8 a few years ago. I see that Talos II is Power9. >> >> At the time we were working on Power8 we had to add some #ifdef >> CONFIG_ACPI guards around some ACPI-specific code in KFD. Do you know to >> what extent ACPI is available and working on the Power architecture? >> >> Another problem we ran into with Power, is the physical address map. >> System memory can be a physical addresses outside the range accessible >> by the GPU. Vega has 44-bit physical addressing. Older Polaris GPUs only >> have 40-bits. Did you run into any such problems? Do you need an IOMMU >> to make system memory accessible to the GPU? >> >> Regards, >> Felix > Yes, we are POWER9. It looks like the ACPI guards are no longer required; as you have surmised, POWER does not use ACPI (the equivalent is OPAL, which is a different interface entirely). What were the APCI calls used for? There may be OPAL equivalents that could be added in to replace them and provide similar functionality. There are some ACPI calls (e.g. acpi_get_table) in kfd_crat.c for getting a CRAT table from ACPI. This is only useful for AMD APUs, which are x86_64. We don't need this for discrete GPUs because on non-APU systems there is no CRAT table and we build our own. If you can compile the code without problems on Power and with CONFIG_ACPI not defined, then I guess this is no longer an issue. > Kernel 5.4 enables a > 32-bit and <=64-bit bypass mode for POWER. This is one reason we came back and revisited the KFD/ROCm functionality on POWER; as it turns out, after fixing up the userspace tools KFD is indeed functional on POWER with 5.4-rc8 and above. My understanding is that the POWER IOMMU is used as a lightweight translation layer between the 64-bit host and the 40/44-bit GPU. > > I'm working on getting a Debian PPA set up for POWER to make the userspace tools easier to obtain for testing, but progress is slow due to lack of Debian source packages. Probably the easiest way to replicate / test this with HIP is to use the AOMP repository with my modifications; pull requests are already in place on Github for most of the userspace tooling updates. > > Thank you! Thanks, Felix _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx
next prev parent reply other threads:[~2019-11-25 21:34 UTC|newest] Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-11-24 19:15 [PATCH 1/1] amdgpu: Enable KFD on POWER systems Timothy Pearson 2019-11-24 19:15 ` Timothy Pearson [not found] ` <2068915477.1923611.1574622916022.JavaMail.zimbra-z0qzliK6Om0mgXJStvpl+u2xl0RS173n@public.gmane.org> 2019-11-25 14:59 ` Alex Deucher 2019-11-25 14:59 ` Alex Deucher 2019-11-25 17:07 ` Felix Kuehling 2019-11-25 17:07 ` Felix Kuehling [not found] ` <44175d39-e507-83f9-6596-cbde108dd65a-5C7GfCeVMHo@public.gmane.org> 2019-11-25 21:06 ` Timothy Pearson 2019-11-25 21:06 ` Timothy Pearson [not found] ` <535694171.2118031.1574716002431.JavaMail.zimbra-z0qzliK6Om0mgXJStvpl+u2xl0RS173n@public.gmane.org> 2019-11-25 21:34 ` Felix Kuehling [this message] 2019-11-25 21:34 ` Felix Kuehling [not found] ` <5dcfdba7-dc14-99d4-d6e5-fc1768fcc5c0-5C7GfCeVMHo@public.gmane.org> 2019-11-25 21:52 ` Timothy Pearson 2019-11-25 21:52 ` Timothy Pearson -- strict thread matches above, loose matches on Subject: below -- 2019-11-24 19:10 Timothy Pearson 2019-11-24 19:10 ` Timothy Pearson [not found] ` <443989900.1923547.1574622652551.JavaMail.zimbra-z0qzliK6Om0mgXJStvpl+u2xl0RS173n@public.gmane.org> 2019-11-24 19:19 ` Timothy Pearson 2019-11-24 19:19 ` Timothy Pearson 2019-11-24 19:09 Timothy Pearson 2019-11-24 19:09 ` Timothy Pearson
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=5dcfdba7-dc14-99d4-d6e5-fc1768fcc5c0@amd.com \ --to=felix.kuehling-5c7gfcevmho@public.gmane.org \ --cc=amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org \ --cc=tpearson-z0qzliK6Om0mgXJStvpl+vpXobYPEAuW@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.