* [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active @ 2020-08-24 10:54 Joerg Roedel 2020-08-24 10:54 ` [PATCH 1/2] iommu/amd: Do not force direct mapping " Joerg Roedel ` (2 more replies) 0 siblings, 3 replies; 19+ messages in thread From: Joerg Roedel @ 2020-08-24 10:54 UTC (permalink / raw) To: iommu; +Cc: Tom Lendacky, jroedel, linux-kernel, Alexander.Deucher From: Joerg Roedel <jroedel@suse.de> Hi, Some IOMMUv2 capable devices do not work correctly when SME is active, because their DMA mask does not include the encryption bit, so that they can not DMA to encrypted memory directly. The IOMMU can jump in here, but the AMD IOMMU driver puts IOMMUv2 capable devices into an identity mapped domain. Fix that by not forcing an identity mapped domain on devices when SME is active and forbid using their IOMMUv2 functionality. Please review. Thanks, Joerg Joerg Roedel (2): iommu/amd: Do not force direct mapping when SME is active iommu/amd: Do not use IOMMUv2 functionality when SME is active drivers/iommu/amd/iommu.c | 7 ++++++- drivers/iommu/amd/iommu_v2.c | 7 +++++++ 2 files changed, 13 insertions(+), 1 deletion(-) -- 2.28.0 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 1/2] iommu/amd: Do not force direct mapping when SME is active 2020-08-24 10:54 [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active Joerg Roedel @ 2020-08-24 10:54 ` Joerg Roedel 2020-08-26 14:23 ` Deucher, Alexander 2020-08-24 10:54 ` [PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality " Joerg Roedel 2020-08-26 14:14 ` [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices " Deucher, Alexander 2 siblings, 1 reply; 19+ messages in thread From: Joerg Roedel @ 2020-08-24 10:54 UTC (permalink / raw) To: iommu; +Cc: Tom Lendacky, jroedel, linux-kernel, Alexander.Deucher From: Joerg Roedel <jroedel@suse.de> Do not force devices supporting IOMMUv2 to be direct mapped when memory encryption is active. This might cause them to be unusable because their DMA mask does not include the encryption bit. Signed-off-by: Joerg Roedel <jroedel@suse.de> --- drivers/iommu/amd/iommu.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c index ba9f3dbc5b94..77e4268e41cf 100644 --- a/drivers/iommu/amd/iommu.c +++ b/drivers/iommu/amd/iommu.c @@ -2659,7 +2659,12 @@ static int amd_iommu_def_domain_type(struct device *dev) if (!dev_data) return 0; - if (dev_data->iommu_v2) + /* + * Do not identity map IOMMUv2 capable devices when memory encryption is + * active, because some of those devices (AMD GPUs) don't have the + * encryption bit in their DMA-mask and require remapping. + */ + if (!mem_encrypt_active() && dev_data->iommu_v2) return IOMMU_DOMAIN_IDENTITY; return 0; -- 2.28.0 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 19+ messages in thread
* RE: [PATCH 1/2] iommu/amd: Do not force direct mapping when SME is active 2020-08-24 10:54 ` [PATCH 1/2] iommu/amd: Do not force direct mapping " Joerg Roedel @ 2020-08-26 14:23 ` Deucher, Alexander 0 siblings, 0 replies; 19+ messages in thread From: Deucher, Alexander @ 2020-08-26 14:23 UTC (permalink / raw) To: Joerg Roedel, iommu, Kuehling, Felix, Koenig, Christian Cc: Lendacky, Thomas, jroedel, linux-kernel [AMD Public Use] + Felix, Christian > -----Original Message----- > From: Joerg Roedel <joro@8bytes.org> > Sent: Monday, August 24, 2020 6:54 AM > To: iommu@lists.linux-foundation.org > Cc: Joerg Roedel <joro@8bytes.org>; jroedel@suse.de; Lendacky, Thomas > <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee > <Suravee.Suthikulpanit@amd.com>; Deucher, Alexander > <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org > Subject: [PATCH 1/2] iommu/amd: Do not force direct mapping when SME is > active > > From: Joerg Roedel <jroedel@suse.de> > > Do not force devices supporting IOMMUv2 to be direct mapped when > memory encryption is active. This might cause them to be unusable because > their DMA mask does not include the encryption bit. > > Signed-off-by: Joerg Roedel <jroedel@suse.de> > --- > drivers/iommu/amd/iommu.c | 7 ++++++- > 1 file changed, 6 insertions(+), 1 deletion(-) > > diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c > index ba9f3dbc5b94..77e4268e41cf 100644 > --- a/drivers/iommu/amd/iommu.c > +++ b/drivers/iommu/amd/iommu.c > @@ -2659,7 +2659,12 @@ static int amd_iommu_def_domain_type(struct > device *dev) > if (!dev_data) > return 0; > > - if (dev_data->iommu_v2) > + /* > + * Do not identity map IOMMUv2 capable devices when memory > encryption is > + * active, because some of those devices (AMD GPUs) don't have the > + * encryption bit in their DMA-mask and require remapping. > + */ I think on the integrated GPUs in APUs I'd prefer to have the identity mapping over SME, but I guess this is fine because you have to explicitly enable SME and if you do that you know what you are getting into. Alex > + if (!mem_encrypt_active() && dev_data->iommu_v2) > return IOMMU_DOMAIN_IDENTITY; > > return 0; > -- > 2.28.0 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality when SME is active 2020-08-24 10:54 [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active Joerg Roedel 2020-08-24 10:54 ` [PATCH 1/2] iommu/amd: Do not force direct mapping " Joerg Roedel @ 2020-08-24 10:54 ` Joerg Roedel 2020-08-26 14:20 ` Deucher, Alexander 2020-08-26 14:14 ` [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices " Deucher, Alexander 2 siblings, 1 reply; 19+ messages in thread From: Joerg Roedel @ 2020-08-24 10:54 UTC (permalink / raw) To: iommu; +Cc: Tom Lendacky, jroedel, linux-kernel, Alexander.Deucher From: Joerg Roedel <jroedel@suse.de> When memory encryption is active the device is likely not in a direct mapped domain. Forbid using IOMMUv2 functionality for now until finer grained checks for this have been implemented. Signed-off-by: Joerg Roedel <jroedel@suse.de> --- drivers/iommu/amd/iommu_v2.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/iommu/amd/iommu_v2.c b/drivers/iommu/amd/iommu_v2.c index c259108ab6dd..0d175aed1d92 100644 --- a/drivers/iommu/amd/iommu_v2.c +++ b/drivers/iommu/amd/iommu_v2.c @@ -737,6 +737,13 @@ int amd_iommu_init_device(struct pci_dev *pdev, int pasids) might_sleep(); + /* + * When memory encryption is active the device is likely not in a + * direct-mapped domain. Forbid using IOMMUv2 functionality for now. + */ + if (mem_encrypt_active()) + return -ENODEV; + if (!amd_iommu_v2_supported()) return -ENODEV; -- 2.28.0 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 19+ messages in thread
* RE: [PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality when SME is active 2020-08-24 10:54 ` [PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality " Joerg Roedel @ 2020-08-26 14:20 ` Deucher, Alexander 0 siblings, 0 replies; 19+ messages in thread From: Deucher, Alexander @ 2020-08-26 14:20 UTC (permalink / raw) To: Joerg Roedel, iommu, Kuehling, Felix, Koenig, Christian Cc: Lendacky, Thomas, jroedel, linux-kernel [AMD Public Use] + Felix, Christian > -----Original Message----- > From: Joerg Roedel <joro@8bytes.org> > Sent: Monday, August 24, 2020 6:54 AM > To: iommu@lists.linux-foundation.org > Cc: Joerg Roedel <joro@8bytes.org>; jroedel@suse.de; Lendacky, Thomas > <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee > <Suravee.Suthikulpanit@amd.com>; Deucher, Alexander > <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org > Subject: [PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality when > SME is active > > From: Joerg Roedel <jroedel@suse.de> > > When memory encryption is active the device is likely not in a direct mapped > domain. Forbid using IOMMUv2 functionality for now until finer grained > checks for this have been implemented. > > Signed-off-by: Joerg Roedel <jroedel@suse.de> > --- > drivers/iommu/amd/iommu_v2.c | 7 +++++++ > 1 file changed, 7 insertions(+) > > diff --git a/drivers/iommu/amd/iommu_v2.c > b/drivers/iommu/amd/iommu_v2.c index c259108ab6dd..0d175aed1d92 > 100644 > --- a/drivers/iommu/amd/iommu_v2.c > +++ b/drivers/iommu/amd/iommu_v2.c > @@ -737,6 +737,13 @@ int amd_iommu_init_device(struct pci_dev *pdev, > int pasids) > > might_sleep(); > > + /* > + * When memory encryption is active the device is likely not in a > + * direct-mapped domain. Forbid using IOMMUv2 functionality for > now. > + */ > + if (mem_encrypt_active()) > + return -ENODEV; > + > if (!amd_iommu_v2_supported()) > return -ENODEV; > > -- > 2.28.0 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-08-24 10:54 [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active Joerg Roedel 2020-08-24 10:54 ` [PATCH 1/2] iommu/amd: Do not force direct mapping " Joerg Roedel 2020-08-24 10:54 ` [PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality " Joerg Roedel @ 2020-08-26 14:14 ` Deucher, Alexander 2020-08-26 15:22 ` Felix Kuehling 2 siblings, 1 reply; 19+ messages in thread From: Deucher, Alexander @ 2020-08-26 14:14 UTC (permalink / raw) To: Joerg Roedel, iommu, Kuehling, Felix Cc: Lendacky, Thomas, jroedel, linux-kernel [-- Attachment #1.1: Type: text/plain, Size: 1434 bytes --] [AMD Official Use Only - Internal Distribution Only] + Felix ________________________________ From: Joerg Roedel <joro@8bytes.org> Sent: Monday, August 24, 2020 6:54 AM To: iommu@lists.linux-foundation.org <iommu@lists.linux-foundation.org> Cc: Joerg Roedel <joro@8bytes.org>; jroedel@suse.de <jroedel@suse.de>; Lendacky, Thomas <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org <linux-kernel@vger.kernel.org> Subject: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active From: Joerg Roedel <jroedel@suse.de> Hi, Some IOMMUv2 capable devices do not work correctly when SME is active, because their DMA mask does not include the encryption bit, so that they can not DMA to encrypted memory directly. The IOMMU can jump in here, but the AMD IOMMU driver puts IOMMUv2 capable devices into an identity mapped domain. Fix that by not forcing an identity mapped domain on devices when SME is active and forbid using their IOMMUv2 functionality. Please review. Thanks, Joerg Joerg Roedel (2): iommu/amd: Do not force direct mapping when SME is active iommu/amd: Do not use IOMMUv2 functionality when SME is active drivers/iommu/amd/iommu.c | 7 ++++++- drivers/iommu/amd/iommu_v2.c | 7 +++++++ 2 files changed, 13 insertions(+), 1 deletion(-) -- 2.28.0 [-- Attachment #1.2: Type: text/html, Size: 2510 bytes --] [-- Attachment #2: Type: text/plain, Size: 156 bytes --] _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-08-26 14:14 ` [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices " Deucher, Alexander @ 2020-08-26 15:22 ` Felix Kuehling 2020-08-26 15:25 ` Deucher, Alexander 0 siblings, 1 reply; 19+ messages in thread From: Felix Kuehling @ 2020-08-26 15:22 UTC (permalink / raw) To: Deucher, Alexander, Joerg Roedel, iommu, Huang, Ray Cc: Lendacky, Thomas, jroedel, linux-kernel [+Ray] Thanks for the heads up. Currently KFD won't work on APUs when IOMMUv2 is disabled. But Ray is working on fallbacks that will allow KFD to work on APUs even without IOMMUv2, similar to our dGPUs. Along with changes in ROCm user mode, those fallbacks are necessary for making ROCm on APUs generally useful. How common is SME on typical PCs or laptops that would use AMD APUs? Alex, do you know if anyone has tested amdgpu on an APU with SME enabled? Is this considered something we support? Thanks, Felix Am 2020-08-26 um 10:14 a.m. schrieb Deucher, Alexander: > > [AMD Official Use Only - Internal Distribution Only] > > > + Felix > ------------------------------------------------------------------------ > *From:* Joerg Roedel <joro@8bytes.org> > *Sent:* Monday, August 24, 2020 6:54 AM > *To:* iommu@lists.linux-foundation.org <iommu@lists.linux-foundation.org> > *Cc:* Joerg Roedel <joro@8bytes.org>; jroedel@suse.de > <jroedel@suse.de>; Lendacky, Thomas <Thomas.Lendacky@amd.com>; > Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; Deucher, > Alexander <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org > <linux-kernel@vger.kernel.org> > *Subject:* [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active > > From: Joerg Roedel <jroedel@suse.de> > > Hi, > > Some IOMMUv2 capable devices do not work correctly when SME is > active, because their DMA mask does not include the encryption bit, so > that they can not DMA to encrypted memory directly. > > The IOMMU can jump in here, but the AMD IOMMU driver puts IOMMUv2 > capable devices into an identity mapped domain. Fix that by not > forcing an identity mapped domain on devices when SME is active and > forbid using their IOMMUv2 functionality. > > Please review. > > Thanks, > > Joerg > > Joerg Roedel (2): > iommu/amd: Do not force direct mapping when SME is active > iommu/amd: Do not use IOMMUv2 functionality when SME is active > > drivers/iommu/amd/iommu.c | 7 ++++++- > drivers/iommu/amd/iommu_v2.c | 7 +++++++ > 2 files changed, 13 insertions(+), 1 deletion(-) > > -- > 2.28.0 > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-08-26 15:22 ` Felix Kuehling @ 2020-08-26 15:25 ` Deucher, Alexander 2020-08-28 13:46 ` jroedel 0 siblings, 1 reply; 19+ messages in thread From: Deucher, Alexander @ 2020-08-26 15:25 UTC (permalink / raw) To: Kuehling, Felix, Joerg Roedel, iommu, Huang, Ray, Koenig, Christian Cc: Lendacky, Thomas, jroedel, linux-kernel [AMD Public Use] + Christian > -----Original Message----- > From: Kuehling, Felix <Felix.Kuehling@amd.com> > Sent: Wednesday, August 26, 2020 11:22 AM > To: Deucher, Alexander <Alexander.Deucher@amd.com>; Joerg Roedel > <joro@8bytes.org>; iommu@lists.linux-foundation.org; Huang, Ray > <Ray.Huang@amd.com> > Cc: jroedel@suse.de; Lendacky, Thomas <Thomas.Lendacky@amd.com>; > Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; linux- > kernel@vger.kernel.org > Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is > active > > [+Ray] > > > Thanks for the heads up. Currently KFD won't work on APUs when IOMMUv2 > is disabled. But Ray is working on fallbacks that will allow KFD to work on > APUs even without IOMMUv2, similar to our dGPUs. Along with changes in > ROCm user mode, those fallbacks are necessary for making ROCm on APUs > generally useful. > > > How common is SME on typical PCs or laptops that would use AMD APUs? I think the hw supports it, but it as far as I know it's not formally productized on client parts. > > > Alex, do you know if anyone has tested amdgpu on an APU with SME > enabled? Is this considered something we support? It's not something we've tested. I'm not even sure the GPU portion of APUs will work properly without an identity mapping. SME should work properly with dGPUs however, so this is a proper fix for them. We don't use the IOMMUv2 path on dGPUs at all. Alex > > > Thanks, > Felix > > > Am 2020-08-26 um 10:14 a.m. schrieb Deucher, Alexander: > > > > [AMD Official Use Only - Internal Distribution Only] > > > > > > + Felix > > ---------------------------------------------------------------------- > > -- > > *From:* Joerg Roedel <joro@8bytes.org> > > *Sent:* Monday, August 24, 2020 6:54 AM > > *To:* iommu@lists.linux-foundation.org > > <iommu@lists.linux-foundation.org> > > *Cc:* Joerg Roedel <joro@8bytes.org>; jroedel@suse.de > > <jroedel@suse.de>; Lendacky, Thomas <Thomas.Lendacky@amd.com>; > > Suthikulpanit, Suravee <Suravee.Suthikulpanit@amd.com>; Deucher, > > Alexander <Alexander.Deucher@amd.com>; linux-kernel@vger.kernel.org > > <linux-kernel@vger.kernel.org> > > *Subject:* [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is > > active > > > > From: Joerg Roedel <jroedel@suse.de> > > > > Hi, > > > > Some IOMMUv2 capable devices do not work correctly when SME is active, > > because their DMA mask does not include the encryption bit, so that > > they can not DMA to encrypted memory directly. > > > > The IOMMU can jump in here, but the AMD IOMMU driver puts IOMMUv2 > > capable devices into an identity mapped domain. Fix that by not > > forcing an identity mapped domain on devices when SME is active and > > forbid using their IOMMUv2 functionality. > > > > Please review. > > > > Thanks, > > > > Joerg > > > > Joerg Roedel (2): > > iommu/amd: Do not force direct mapping when SME is active > > iommu/amd: Do not use IOMMUv2 functionality when SME is active > > > > drivers/iommu/amd/iommu.c | 7 ++++++- > > drivers/iommu/amd/iommu_v2.c | 7 +++++++ > > 2 files changed, 13 insertions(+), 1 deletion(-) > > > > -- > > 2.28.0 > > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-08-26 15:25 ` Deucher, Alexander @ 2020-08-28 13:46 ` jroedel 2020-08-28 13:54 ` Felix Kuehling 0 siblings, 1 reply; 19+ messages in thread From: jroedel @ 2020-08-28 13:46 UTC (permalink / raw) To: Deucher, Alexander Cc: Lendacky, Thomas, Kuehling, Felix, linux-kernel, Koenig, Christian, iommu, Huang, Ray On Wed, Aug 26, 2020 at 03:25:58PM +0000, Deucher, Alexander wrote: > > Alex, do you know if anyone has tested amdgpu on an APU with SME > > enabled? Is this considered something we support? > > It's not something we've tested. I'm not even sure the GPU portion of > APUs will work properly without an identity mapping. SME should work > properly with dGPUs however, so this is a proper fix for them. We > don't use the IOMMUv2 path on dGPUs at all. Is it possible to make the IOMMUv2 paths optional on iGPUs as well when SME is active (or better, when the GPU is not identity mapped)? Regards, Joerg _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-08-28 13:46 ` jroedel @ 2020-08-28 13:54 ` Felix Kuehling 2020-08-28 15:11 ` Deucher, Alexander 2020-08-28 15:27 ` jroedel 0 siblings, 2 replies; 19+ messages in thread From: Felix Kuehling @ 2020-08-28 13:54 UTC (permalink / raw) To: jroedel, Deucher, Alexander Cc: Lendacky, Thomas, linux-kernel, iommu, Huang, Ray, Koenig, Christian Am 2020-08-28 um 9:46 a.m. schrieb jroedel@suse.de: > On Wed, Aug 26, 2020 at 03:25:58PM +0000, Deucher, Alexander wrote: >>> Alex, do you know if anyone has tested amdgpu on an APU with SME >>> enabled? Is this considered something we support? >> It's not something we've tested. I'm not even sure the GPU portion of >> APUs will work properly without an identity mapping. SME should work >> properly with dGPUs however, so this is a proper fix for them. We >> don't use the IOMMUv2 path on dGPUs at all. > Is it possible to make the IOMMUv2 paths optional on iGPUs as well when > SME is active (or better, when the GPU is not identity mapped)? Yes, we're working on this. IOMMUv2 is only needed for KFD. It's not needed for graphics. And we're making it optional for KFD as well. The question Alex and I raised here is more general. We may have some assumptions in the amdgpu driver that are broken when the framebuffer is not identity mapped. This would break the iGPU in a more general sense, regardless of KFD and IOMMUv2. In that case, we don't really need to worry about breaking KFD because we have a much bigger problem. Regards, Felix > > Regards, > > Joerg _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-08-28 13:54 ` Felix Kuehling @ 2020-08-28 15:11 ` Deucher, Alexander 2020-08-28 15:29 ` jroedel 2020-08-28 15:27 ` jroedel 1 sibling, 1 reply; 19+ messages in thread From: Deucher, Alexander @ 2020-08-28 15:11 UTC (permalink / raw) To: Kuehling, Felix, jroedel Cc: Lendacky, Thomas, linux-kernel, iommu, Huang, Ray, Koenig, Christian [AMD Public Use] > -----Original Message----- > From: Kuehling, Felix <Felix.Kuehling@amd.com> > Sent: Friday, August 28, 2020 9:55 AM > To: jroedel@suse.de; Deucher, Alexander <Alexander.Deucher@amd.com> > Cc: Joerg Roedel <joro@8bytes.org>; iommu@lists.linux-foundation.org; > Huang, Ray <Ray.Huang@amd.com>; Koenig, Christian > <Christian.Koenig@amd.com>; Lendacky, Thomas > <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee > <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org > Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is > active > > Am 2020-08-28 um 9:46 a.m. schrieb jroedel@suse.de: > > On Wed, Aug 26, 2020 at 03:25:58PM +0000, Deucher, Alexander wrote: > >>> Alex, do you know if anyone has tested amdgpu on an APU with SME > >>> enabled? Is this considered something we support? > >> It's not something we've tested. I'm not even sure the GPU portion > >> of APUs will work properly without an identity mapping. SME should > >> work properly with dGPUs however, so this is a proper fix for them. > >> We don't use the IOMMUv2 path on dGPUs at all. > > Is it possible to make the IOMMUv2 paths optional on iGPUs as well > > when SME is active (or better, when the GPU is not identity mapped)? > > Yes, we're working on this. IOMMUv2 is only needed for KFD. It's not needed > for graphics. And we're making it optional for KFD as well. > > The question Alex and I raised here is more general. We may have some > assumptions in the amdgpu driver that are broken when the framebuffer is > not identity mapped. This would break the iGPU in a more general sense, > regardless of KFD and IOMMUv2. In that case, we don't really need to worry > about breaking KFD because we have a much bigger problem. There are hw bugs on Raven and probably Carrizo/Stoney where they need 1:1 mapping to avoid bugs in some corner cases with the displays. Other GPUs should be fine. The VIDs is 0x1002 and the DIDs are 0x15dd and 0x15d8 for raven variants and 0x9870, 0x9874, 0x9875, 0x9876, 0x9877 and 0x98e4 for carrizo and stoney. As long as we preserve the 1:1 mapping for those asics, we should be fine. Alex > > Regards, > Felix > > > > > > Regards, > > > > Joerg _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-08-28 15:11 ` Deucher, Alexander @ 2020-08-28 15:29 ` jroedel 2020-08-28 15:47 ` Deucher, Alexander 0 siblings, 1 reply; 19+ messages in thread From: jroedel @ 2020-08-28 15:29 UTC (permalink / raw) To: Deucher, Alexander Cc: Lendacky, Thomas, Kuehling, Felix, linux-kernel, Koenig, Christian, iommu, Huang, Ray On Fri, Aug 28, 2020 at 03:11:32PM +0000, Deucher, Alexander wrote: > There are hw bugs on Raven and probably Carrizo/Stoney where they need > 1:1 mapping to avoid bugs in some corner cases with the displays. > Other GPUs should be fine. The VIDs is 0x1002 and the DIDs are 0x15dd > and 0x15d8 for raven variants and 0x9870, 0x9874, 0x9875, 0x9876, > 0x9877 and 0x98e4 for carrizo and stoney. As long as we > preserve the 1:1 mapping for those asics, we should be fine. Okay, Stoney at least has no Zen-based CPU, so no support for memory encryption anyway. How about Raven, is it paired with a Zen CPU? Regards, Joerg _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-08-28 15:29 ` jroedel @ 2020-08-28 15:47 ` Deucher, Alexander 2020-09-04 10:05 ` Joerg Roedel 0 siblings, 1 reply; 19+ messages in thread From: Deucher, Alexander @ 2020-08-28 15:47 UTC (permalink / raw) To: jroedel Cc: Lendacky, Thomas, Kuehling, Felix, linux-kernel, Koenig, Christian, iommu, Huang, Ray [AMD Public Use] > -----Original Message----- > From: jroedel@suse.de <jroedel@suse.de> > Sent: Friday, August 28, 2020 11:30 AM > To: Deucher, Alexander <Alexander.Deucher@amd.com> > Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; Joerg Roedel > <joro@8bytes.org>; iommu@lists.linux-foundation.org; Huang, Ray > <Ray.Huang@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>; > Lendacky, Thomas <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee > <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org > Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is > active > > On Fri, Aug 28, 2020 at 03:11:32PM +0000, Deucher, Alexander wrote: > > There are hw bugs on Raven and probably Carrizo/Stoney where they need > > 1:1 mapping to avoid bugs in some corner cases with the displays. > > Other GPUs should be fine. The VIDs is 0x1002 and the DIDs are 0x15dd > > and 0x15d8 for raven variants and 0x9870, 0x9874, 0x9875, 0x9876, > > 0x9877 and 0x98e4 for carrizo and stoney. As long as we preserve the > > 1:1 mapping for those asics, we should be fine. > > Okay, Stoney at least has no Zen-based CPU, so no support for memory > encryption anyway. How about Raven, is it paired with a Zen CPU? Ah, right, So CZ and ST are not an issue. Raven is paired with Zen based CPUs. Thanks, Alex _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-08-28 15:47 ` Deucher, Alexander @ 2020-09-04 10:05 ` Joerg Roedel 2020-09-06 16:08 ` Deucher, Alexander 0 siblings, 1 reply; 19+ messages in thread From: Joerg Roedel @ 2020-09-04 10:05 UTC (permalink / raw) To: Deucher, Alexander Cc: Lendacky, Thomas, jroedel, Kuehling, Felix, linux-kernel, iommu, Huang, Ray, Koenig, Christian On Fri, Aug 28, 2020 at 03:47:07PM +0000, Deucher, Alexander wrote: > Ah, right, So CZ and ST are not an issue. Raven is paired with Zen based CPUs. Okay, so for the Raven case, can you add code to the amdgpu driver which makes it fail to initialize on Raven when SME is active? There is a global checking function for that, so that shouldn't be hard to do. Regards, Joerg _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* RE: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-09-04 10:05 ` Joerg Roedel @ 2020-09-06 16:08 ` Deucher, Alexander 2020-09-07 10:44 ` Joerg Roedel 2020-09-08 3:38 ` Felix Kuehling 0 siblings, 2 replies; 19+ messages in thread From: Deucher, Alexander @ 2020-09-06 16:08 UTC (permalink / raw) To: Joerg Roedel Cc: Lendacky, Thomas, jroedel, Kuehling, Felix, linux-kernel, iommu, Huang, Ray, Koenig, Christian [-- Attachment #1: Type: text/plain, Size: 1060 bytes --] [AMD Official Use Only - Internal Distribution Only] > -----Original Message----- > From: Joerg Roedel <joro@8bytes.org> > Sent: Friday, September 4, 2020 6:06 AM > To: Deucher, Alexander <Alexander.Deucher@amd.com> > Cc: jroedel@suse.de; Kuehling, Felix <Felix.Kuehling@amd.com>; > iommu@lists.linux-foundation.org; Huang, Ray <Ray.Huang@amd.com>; > Koenig, Christian <Christian.Koenig@amd.com>; Lendacky, Thomas > <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee > <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org > Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is > active > > On Fri, Aug 28, 2020 at 03:47:07PM +0000, Deucher, Alexander wrote: > > Ah, right, So CZ and ST are not an issue. Raven is paired with Zen based > CPUs. > > Okay, so for the Raven case, can you add code to the amdgpu driver which > makes it fail to initialize on Raven when SME is active? There is a global > checking function for that, so that shouldn't be hard to do. > Sure. How about the attached patch? Alex [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-drm-amdgpu-Fail-to-load-on-RAVEN-if-SME-is-active.patch --] [-- Type: text/x-patch; name="0001-drm-amdgpu-Fail-to-load-on-RAVEN-if-SME-is-active.patch", Size: 1438 bytes --] From f479b9da353c2547c26ebac8930a5dcd9a134eb7 Mon Sep 17 00:00:00 2001 From: Alex Deucher <alexander.deucher@amd.com> Date: Sun, 6 Sep 2020 12:05:12 -0400 Subject: [PATCH] drm/amdgpu: Fail to load on RAVEN if SME is active Due to hardware bugs, scatter/gather display on raven requires a 1:1 IOMMU mapping, however, SME (System Memory Encryption) requires an indirect IOMMU mapping because the encryption bit is beyond the DMA mask of the chip. As such, the two are incompatible. Signed-off-by: Alex Deucher <alexander.deucher@amd.com> --- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index 12e16445df7c..d87d37c25329 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c @@ -1102,6 +1102,16 @@ static int amdgpu_pci_probe(struct pci_dev *pdev, return -ENODEV; } + /* Due to hardware bugs, S/G Display on raven requires a 1:1 IOMMU mapping, + * however, SME requires an indirect IOMMU mapping because the encryption + * bit is beyond the DMA mask of the chip. + */ + if (mem_encrypt_active() && ((flags & AMD_ASIC_MASK) == CHIP_RAVEN)) { + dev_info(&pdev->dev, + "SME is not compatible with RAVEN\n"); + return -ENOTSUPP; + } + #ifdef CONFIG_DRM_AMDGPU_SI if (!amdgpu_si_support) { switch (flags & AMD_ASIC_MASK) { -- 2.25.4 [-- Attachment #3: Type: text/plain, Size: 156 bytes --] _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-09-06 16:08 ` Deucher, Alexander @ 2020-09-07 10:44 ` Joerg Roedel 2020-09-07 11:10 ` Christian König 2020-09-08 3:38 ` Felix Kuehling 1 sibling, 1 reply; 19+ messages in thread From: Joerg Roedel @ 2020-09-07 10:44 UTC (permalink / raw) To: Deucher, Alexander Cc: Lendacky, Thomas, jroedel, Kuehling, Felix, linux-kernel, iommu, Huang, Ray, Koenig, Christian On Sun, Sep 06, 2020 at 04:08:58PM +0000, Deucher, Alexander wrote: > From f479b9da353c2547c26ebac8930a5dcd9a134eb7 Mon Sep 17 00:00:00 2001 > From: Alex Deucher <alexander.deucher@amd.com> > Date: Sun, 6 Sep 2020 12:05:12 -0400 > Subject: [PATCH] drm/amdgpu: Fail to load on RAVEN if SME is active > > Due to hardware bugs, scatter/gather display on raven requires > a 1:1 IOMMU mapping, however, SME (System Memory Encryption) > requires an indirect IOMMU mapping because the encryption bit > is beyond the DMA mask of the chip. As such, the two are > incompatible. > > Signed-off-by: Alex Deucher <alexander.deucher@amd.com> > --- > drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 10 ++++++++++ > 1 file changed, 10 insertions(+) > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > index 12e16445df7c..d87d37c25329 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > @@ -1102,6 +1102,16 @@ static int amdgpu_pci_probe(struct pci_dev *pdev, > return -ENODEV; > } > > + /* Due to hardware bugs, S/G Display on raven requires a 1:1 IOMMU mapping, > + * however, SME requires an indirect IOMMU mapping because the encryption > + * bit is beyond the DMA mask of the chip. > + */ > + if (mem_encrypt_active() && ((flags & AMD_ASIC_MASK) == CHIP_RAVEN)) { > + dev_info(&pdev->dev, > + "SME is not compatible with RAVEN\n"); > + return -ENOTSUPP; > + } > + > #ifdef CONFIG_DRM_AMDGPU_SI > if (!amdgpu_si_support) { > switch (flags & AMD_ASIC_MASK) { > -- > 2.25.4 > Looks good to me, thanks. Acked-by: Joerg Roedel <jroedel@suse.de> _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-09-07 10:44 ` Joerg Roedel @ 2020-09-07 11:10 ` Christian König 0 siblings, 0 replies; 19+ messages in thread From: Christian König @ 2020-09-07 11:10 UTC (permalink / raw) To: Joerg Roedel, Deucher, Alexander Cc: Lendacky, Thomas, jroedel, Kuehling, Felix, linux-kernel, iommu, Huang, Ray Am 07.09.20 um 12:44 schrieb Joerg Roedel: > On Sun, Sep 06, 2020 at 04:08:58PM +0000, Deucher, Alexander wrote: >> From f479b9da353c2547c26ebac8930a5dcd9a134eb7 Mon Sep 17 00:00:00 2001 >> From: Alex Deucher <alexander.deucher@amd.com> >> Date: Sun, 6 Sep 2020 12:05:12 -0400 >> Subject: [PATCH] drm/amdgpu: Fail to load on RAVEN if SME is active >> >> Due to hardware bugs, scatter/gather display on raven requires >> a 1:1 IOMMU mapping, however, SME (System Memory Encryption) >> requires an indirect IOMMU mapping because the encryption bit >> is beyond the DMA mask of the chip. As such, the two are >> incompatible. >> >> Signed-off-by: Alex Deucher <alexander.deucher@amd.com> >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 10 ++++++++++ >> 1 file changed, 10 insertions(+) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >> index 12e16445df7c..d87d37c25329 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >> @@ -1102,6 +1102,16 @@ static int amdgpu_pci_probe(struct pci_dev *pdev, >> return -ENODEV; >> } >> >> + /* Due to hardware bugs, S/G Display on raven requires a 1:1 IOMMU mapping, >> + * however, SME requires an indirect IOMMU mapping because the encryption >> + * bit is beyond the DMA mask of the chip. >> + */ >> + if (mem_encrypt_active() && ((flags & AMD_ASIC_MASK) == CHIP_RAVEN)) { >> + dev_info(&pdev->dev, >> + "SME is not compatible with RAVEN\n"); >> + return -ENOTSUPP; >> + } >> + >> #ifdef CONFIG_DRM_AMDGPU_SI >> if (!amdgpu_si_support) { >> switch (flags & AMD_ASIC_MASK) { >> -- >> 2.25.4 >> > Looks good to me, thanks. > > Acked-by: Joerg Roedel <jroedel@suse.de> This is really unfortunate, but I don't see any other solution either. Reviewed-by: Christian König <christian.koenig@amd.com> _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-09-06 16:08 ` Deucher, Alexander 2020-09-07 10:44 ` Joerg Roedel @ 2020-09-08 3:38 ` Felix Kuehling 1 sibling, 0 replies; 19+ messages in thread From: Felix Kuehling @ 2020-09-08 3:38 UTC (permalink / raw) To: Deucher, Alexander, Joerg Roedel Cc: Lendacky, Thomas, jroedel, linux-kernel, iommu, Huang, Ray, Koenig, Christian Am 2020-09-06 um 12:08 p.m. schrieb Deucher, Alexander: > [AMD Official Use Only - Internal Distribution Only] > >> -----Original Message----- >> From: Joerg Roedel <joro@8bytes.org> >> Sent: Friday, September 4, 2020 6:06 AM >> To: Deucher, Alexander <Alexander.Deucher@amd.com> >> Cc: jroedel@suse.de; Kuehling, Felix <Felix.Kuehling@amd.com>; >> iommu@lists.linux-foundation.org; Huang, Ray <Ray.Huang@amd.com>; >> Koenig, Christian <Christian.Koenig@amd.com>; Lendacky, Thomas >> <Thomas.Lendacky@amd.com>; Suthikulpanit, Suravee >> <Suravee.Suthikulpanit@amd.com>; linux-kernel@vger.kernel.org >> Subject: Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is >> active >> >> On Fri, Aug 28, 2020 at 03:47:07PM +0000, Deucher, Alexander wrote: >>> Ah, right, So CZ and ST are not an issue. Raven is paired with Zen based >> CPUs. >> >> Okay, so for the Raven case, can you add code to the amdgpu driver which >> makes it fail to initialize on Raven when SME is active? There is a global >> checking function for that, so that shouldn't be hard to do. >> > Sure. How about the attached patch? The patch is Acked-by: Felix Kuehling <Felix.Kuehling@amd.com> Thanks, Felix > > Alex > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active 2020-08-28 13:54 ` Felix Kuehling 2020-08-28 15:11 ` Deucher, Alexander @ 2020-08-28 15:27 ` jroedel 1 sibling, 0 replies; 19+ messages in thread From: jroedel @ 2020-08-28 15:27 UTC (permalink / raw) To: Felix Kuehling Cc: Lendacky, Thomas, linux-kernel, iommu, Huang, Ray, Deucher, Alexander, Koenig, Christian Hi Felix, On Fri, Aug 28, 2020 at 09:54:59AM -0400, Felix Kuehling wrote: > Yes, we're working on this. IOMMUv2 is only needed for KFD. It's not > needed for graphics. And we're making it optional for KFD as well. Okay, KFD should fail gracefully because it can't initialize the device's iommuv2 functionality. Regards, Joerg _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2020-09-08 3:38 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-08-24 10:54 [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices when SME is active Joerg Roedel 2020-08-24 10:54 ` [PATCH 1/2] iommu/amd: Do not force direct mapping " Joerg Roedel 2020-08-26 14:23 ` Deucher, Alexander 2020-08-24 10:54 ` [PATCH 2/2] iommu/amd: Do not use IOMMUv2 functionality " Joerg Roedel 2020-08-26 14:20 ` Deucher, Alexander 2020-08-26 14:14 ` [PATCH 0/2] iommu/amd: Fix IOMMUv2 devices " Deucher, Alexander 2020-08-26 15:22 ` Felix Kuehling 2020-08-26 15:25 ` Deucher, Alexander 2020-08-28 13:46 ` jroedel 2020-08-28 13:54 ` Felix Kuehling 2020-08-28 15:11 ` Deucher, Alexander 2020-08-28 15:29 ` jroedel 2020-08-28 15:47 ` Deucher, Alexander 2020-09-04 10:05 ` Joerg Roedel 2020-09-06 16:08 ` Deucher, Alexander 2020-09-07 10:44 ` Joerg Roedel 2020-09-07 11:10 ` Christian König 2020-09-08 3:38 ` Felix Kuehling 2020-08-28 15:27 ` jroedel
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).