[PATCH 1/2] drm/amdkfd: remove redundant iommu cleanup code

[PATCH 1/2] drm/amdkfd: remove redundant iommu cleanup code

[Yifan Zhang]

kfd_resume doesn't involve iommu operation, remove
redundant iommu cleanup code.

Signed-off-by: Yifan Zhang <yifan1.zhang@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_device.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index c2a4d920da40..4a416231b24c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -1085,18 +1085,12 @@ static int kfd_resume(struct kfd_dev *kfd)
 	int err = 0;
 
 	err = kfd->dqm->ops.start(kfd->dqm);
-	if (err) {
+	if (err)
 		dev_err(kfd_device,
 			"Error starting queue manager for device %x:%x\n",
 			kfd->pdev->vendor, kfd->pdev->device);
-		goto dqm_start_error;
-	}
 
 	return err;
-
-dqm_start_error:
-	kfd_iommu_suspend(kfd);
-	return err;
 }
 
 static inline void kfd_queue_work(struct workqueue_struct *wq,
-- 
2.25.1

[PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[Yifan Zhang]

This patch is to fix clinfo failure in Raven/Picasso:

Number of platforms: 1
  Platform Profile: FULL_PROFILE
  Platform Version: OpenCL 2.2 AMD-APP (3364.0)
  Platform Name: AMD Accelerated Parallel Processing
  Platform Vendor: Advanced Micro Devices, Inc.
  Platform Extensions: cl_khr_icd cl_amd_event_callback

  Platform Name: AMD Accelerated Parallel Processing Number of devices: 0

Signed-off-by: Yifan Zhang <yifan1.zhang@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 4c8f2f4647c0..89ed9b091386 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2393,10 +2393,6 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 	if (r)
 		goto init_failed;
 
-	r = amdgpu_amdkfd_resume_iommu(adev);
-	if (r)
-		goto init_failed;
-
 	r = amdgpu_device_ip_hw_init_phase1(adev);
 	if (r)
 		goto init_failed;
@@ -2435,6 +2431,10 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 	if (!adev->gmc.xgmi.pending_reset)
 		amdgpu_amdkfd_device_init(adev);
 
+	r = amdgpu_amdkfd_resume_iommu(adev);
+	if (r)
+		goto init_failed;
+
 	amdgpu_fru_get_product_info(adev);
 
 init_failed:
-- 
2.25.1

Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[Felix Kuehling]

[+James]

This basically undoes James's change "drm/amdgpu: move iommu_resume
before ip init/resume". I assume James made his change for a reason. Can
you please discuss the issue with him and determine a solution that
solves both your problem and his?

If James' patch series was a mistake, I'd prefer to revert his patches,
because his patches complicated the initialization sequence and exposed
the iommu init sequence in amdgpu.

Thanks,
  Felix


Am 2021-09-28 um 4:28 a.m. schrieb Yifan Zhang:
> This patch is to fix clinfo failure in Raven/Picasso:
>
> Number of platforms: 1
>   Platform Profile: FULL_PROFILE
>   Platform Version: OpenCL 2.2 AMD-APP (3364.0)
>   Platform Name: AMD Accelerated Parallel Processing
>   Platform Vendor: Advanced Micro Devices, Inc.
>   Platform Extensions: cl_khr_icd cl_amd_event_callback
>
>   Platform Name: AMD Accelerated Parallel Processing Number of devices: 0
>
> Signed-off-by: Yifan Zhang <yifan1.zhang@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 4c8f2f4647c0..89ed9b091386 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -2393,10 +2393,6 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
>  	if (r)
>  		goto init_failed;
>  
> -	r = amdgpu_amdkfd_resume_iommu(adev);
> -	if (r)
> -		goto init_failed;
> -
>  	r = amdgpu_device_ip_hw_init_phase1(adev);
>  	if (r)
>  		goto init_failed;
> @@ -2435,6 +2431,10 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
>  	if (!adev->gmc.xgmi.pending_reset)
>  		amdgpu_amdkfd_device_init(adev);
>  
> +	r = amdgpu_amdkfd_resume_iommu(adev);
> +	if (r)
> +		goto init_failed;
> +
>  	amdgpu_fru_get_product_info(adev);
>  
>  init_failed:

Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[Zhu, James]

[-- Attachment #1: Type: text/plain, Size: 2636 bytes --]

[AMD Official Use Only]

H Felix,

Since the previous patch can help on PCO suspend/resume hung issue. Let me work with YiFan to see if
there is proper way to cover both cases.


Thanks & Best Regards!


James Zhu

________________________________
From: Kuehling, Felix <Felix.Kuehling@amd.com>
Sent: Tuesday, September 28, 2021 11:41 AM
To: Zhang, Yifan <Yifan1.Zhang@amd.com>; amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>; Zhu, James <James.Zhu@amd.com>
Subject: Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[+James]

This basically undoes James's change "drm/amdgpu: move iommu_resume
before ip init/resume". I assume James made his change for a reason. Can
you please discuss the issue with him and determine a solution that
solves both your problem and his?

If James' patch series was a mistake, I'd prefer to revert his patches,
because his patches complicated the initialization sequence and exposed
the iommu init sequence in amdgpu.

Thanks,
  Felix


Am 2021-09-28 um 4:28 a.m. schrieb Yifan Zhang:
> This patch is to fix clinfo failure in Raven/Picasso:
>
> Number of platforms: 1
>   Platform Profile: FULL_PROFILE
>   Platform Version: OpenCL 2.2 AMD-APP (3364.0)
>   Platform Name: AMD Accelerated Parallel Processing
>   Platform Vendor: Advanced Micro Devices, Inc.
>   Platform Extensions: cl_khr_icd cl_amd_event_callback
>
>   Platform Name: AMD Accelerated Parallel Processing Number of devices: 0
>
> Signed-off-by: Yifan Zhang <yifan1.zhang@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 4c8f2f4647c0..89ed9b091386 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -2393,10 +2393,6 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
>        if (r)
>                goto init_failed;
>
> -     r = amdgpu_amdkfd_resume_iommu(adev);
> -     if (r)
> -             goto init_failed;
> -
>        r = amdgpu_device_ip_hw_init_phase1(adev);
>        if (r)
>                goto init_failed;
> @@ -2435,6 +2431,10 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
>        if (!adev->gmc.xgmi.pending_reset)
>                amdgpu_amdkfd_device_init(adev);
>
> +     r = amdgpu_amdkfd_resume_iommu(adev);
> +     if (r)
> +             goto init_failed;
> +
>        amdgpu_fru_get_product_info(adev);
>
>  init_failed:

[-- Attachment #2: Type: text/html, Size: 5371 bytes --]

Re: [PATCH 1/2] drm/amdkfd: remove redundant iommu cleanup code

[Zhu, James]

[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]

[AMD Official Use Only]


Reviewed-by: James Zhu <James.Zhu@amd.com> for the series



James Zhu

________________________________
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> on behalf of Yifan Zhang <yifan1.zhang@amd.com>
Sent: Tuesday, September 28, 2021 4:28 AM
To: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>
Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; Zhang, Yifan <Yifan1.Zhang@amd.com>
Subject: [PATCH 1/2] drm/amdkfd: remove redundant iommu cleanup code

kfd_resume doesn't involve iommu operation, remove
redundant iommu cleanup code.

Signed-off-by: Yifan Zhang <yifan1.zhang@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_device.c | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index c2a4d920da40..4a416231b24c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -1085,18 +1085,12 @@ static int kfd_resume(struct kfd_dev *kfd)
         int err = 0;

         err = kfd->dqm->ops.start(kfd->dqm);
-       if (err) {
+       if (err)
                 dev_err(kfd_device,
                         "Error starting queue manager for device %x:%x\n",
                         kfd->pdev->vendor, kfd->pdev->device);
-               goto dqm_start_error;
-       }

         return err;
-
-dqm_start_error:
-       kfd_iommu_suspend(kfd);
-       return err;
 }

 static inline void kfd_queue_work(struct workqueue_struct *wq,
--
2.25.1


[-- Attachment #2: Type: text/html, Size: 4097 bytes --]

RE: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[Zhang, Yifan]

[-- Attachment #1: Type: text/plain, Size: 3391 bytes --]

[AMD Official Use Only]

Hi Felix,

After sync w/ James, we agree that this patch series could fix both our problems, and he verified this patch series will not cause regression of his previous issue. Do you have more comments regarding this patch series ? Thanks.

BRs,
Yifan

From: Zhu, James <James.Zhu@amd.com>
Sent: Wednesday, September 29, 2021 9:19 PM
To: Kuehling, Felix <Felix.Kuehling@amd.com>; Zhang, Yifan <Yifan1.Zhang@amd.com>; amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init


[AMD Official Use Only]

H Felix,

Since the previous patch can help on PCO suspend/resume hung issue. Let me work with YiFan to see if
there is proper way to cover both cases.


Thanks & Best Regards!



James Zhu

________________________________
From: Kuehling, Felix <Felix.Kuehling@amd.com<mailto:Felix.Kuehling@amd.com>>
Sent: Tuesday, September 28, 2021 11:41 AM
To: Zhang, Yifan <Yifan1.Zhang@amd.com<mailto:Yifan1.Zhang@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>>; Zhu, James <James.Zhu@amd.com<mailto:James.Zhu@amd.com>>
Subject: Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[+James]

This basically undoes James's change "drm/amdgpu: move iommu_resume
before ip init/resume". I assume James made his change for a reason. Can
you please discuss the issue with him and determine a solution that
solves both your problem and his?

If James' patch series was a mistake, I'd prefer to revert his patches,
because his patches complicated the initialization sequence and exposed
the iommu init sequence in amdgpu.

Thanks,
  Felix


Am 2021-09-28 um 4:28 a.m. schrieb Yifan Zhang:
> This patch is to fix clinfo failure in Raven/Picasso:
>
> Number of platforms: 1
>   Platform Profile: FULL_PROFILE
>   Platform Version: OpenCL 2.2 AMD-APP (3364.0)
>   Platform Name: AMD Accelerated Parallel Processing
>   Platform Vendor: Advanced Micro Devices, Inc.
>   Platform Extensions: cl_khr_icd cl_amd_event_callback
>
>   Platform Name: AMD Accelerated Parallel Processing Number of devices: 0
>
> Signed-off-by: Yifan Zhang <yifan1.zhang@amd.com<mailto:yifan1.zhang@amd.com>>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 4c8f2f4647c0..89ed9b091386 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -2393,10 +2393,6 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
>        if (r)
>                goto init_failed;
>
> -     r = amdgpu_amdkfd_resume_iommu(adev);
> -     if (r)
> -             goto init_failed;
> -
>        r = amdgpu_device_ip_hw_init_phase1(adev);
>        if (r)
>                goto init_failed;
> @@ -2435,6 +2431,10 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
>        if (!adev->gmc.xgmi.pending_reset)
>                amdgpu_amdkfd_device_init(adev);
>
> +     r = amdgpu_amdkfd_resume_iommu(adev);
> +     if (r)
> +             goto init_failed;
> +
>        amdgpu_fru_get_product_info(adev);
>
>  init_failed:

[-- Attachment #2: Type: text/html, Size: 8749 bytes --]

Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[Felix Kuehling]

I'm trying to understand what the end result is after James' and your
patches. If I'm reading it correctly, we now initialize IOMMUv2 after
kfd_device_init during initialization, but before kfd_device_init during
resume from S3. Is  that the correct understanding?

My concern is, that we may run into related problems again if the root
cause and the mechanism of the fix are poorly understood. Do you have an
explanation why this different sequence is needed during init and resume?

That said, given that your patches are tested and reviewed by James, you
can add my Acked-by to both patches. The series is

Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>

Thanks,
  Felix


Am 2021-10-04 um 5:42 a.m. schrieb Zhang, Yifan:
>
> [AMD Official Use Only]
>
>  
>
> Hi Felix,
>
>  
>
> After sync w/ James, we agree that this patch series could fix both
> our problems, and he verified this patch series will not cause
> regression of his previous issue. Do you have more comments regarding
> this patch series ? Thanks.
>
>  
>
> BRs,
>
> Yifan
>
>  
>
> *From:* Zhu, James <James.Zhu@amd.com>
> *Sent:* Wednesday, September 29, 2021 9:19 PM
> *To:* Kuehling, Felix <Felix.Kuehling@amd.com>; Zhang, Yifan
> <Yifan1.Zhang@amd.com>; amd-gfx@lists.freedesktop.org
> *Subject:* Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init
>
>  
>
> [AMD Official Use Only]
>
>  
>
> H Felix,
>
>  
>
> Since the previous patch can help on PCO suspend/resume hung issue.
> Let me work with YiFan to see if
>
> there is proper way to cover both cases.
>
>  
>
> Thanks & Best Regards!
>
>  
>
> James Zhu
>
> ------------------------------------------------------------------------
>
> *From:*Kuehling, Felix <Felix.Kuehling@amd.com
> <mailto:Felix.Kuehling@amd.com>>
> *Sent:* Tuesday, September 28, 2021 11:41 AM
> *To:* Zhang, Yifan <Yifan1.Zhang@amd.com
> <mailto:Yifan1.Zhang@amd.com>>; amd-gfx@lists.freedesktop.org
> <mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org
> <mailto:amd-gfx@lists.freedesktop.org>>; Zhu, James <James.Zhu@amd.com
> <mailto:James.Zhu@amd.com>>
> *Subject:* Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device
> init
>
>  
>
> [+James]
>
> This basically undoes James's change "drm/amdgpu: move iommu_resume
> before ip init/resume". I assume James made his change for a reason. Can
> you please discuss the issue with him and determine a solution that
> solves both your problem and his?
>
> If James' patch series was a mistake, I'd prefer to revert his patches,
> because his patches complicated the initialization sequence and exposed
> the iommu init sequence in amdgpu.
>
> Thanks,
>   Felix
>
>
> Am 2021-09-28 um 4:28 a.m. schrieb Yifan Zhang:
> > This patch is to fix clinfo failure in Raven/Picasso:
> >
> > Number of platforms: 1
> >   Platform Profile: FULL_PROFILE
> >   Platform Version: OpenCL 2.2 AMD-APP (3364.0)
> >   Platform Name: AMD Accelerated Parallel Processing
> >   Platform Vendor: Advanced Micro Devices, Inc.
> >   Platform Extensions: cl_khr_icd cl_amd_event_callback
> >
> >   Platform Name: AMD Accelerated Parallel Processing Number of
> devices: 0
> >
> > Signed-off-by: Yifan Zhang <yifan1.zhang@amd.com
> <mailto:yifan1.zhang@amd.com>>
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 ++++----
> >  1 file changed, 4 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > index 4c8f2f4647c0..89ed9b091386 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > @@ -2393,10 +2393,6 @@ static int amdgpu_device_ip_init(struct
> amdgpu_device *adev)
> >        if (r)
> >                goto init_failed;
> > 
> > -     r = amdgpu_amdkfd_resume_iommu(adev);
> > -     if (r)
> > -             goto init_failed;
> > -
> >        r = amdgpu_device_ip_hw_init_phase1(adev);
> >        if (r)
> >                goto init_failed;
> > @@ -2435,6 +2431,10 @@ static int amdgpu_device_ip_init(struct
> amdgpu_device *adev)
> >        if (!adev->gmc.xgmi.pending_reset)
> >                amdgpu_amdkfd_device_init(adev);
> > 
> > +     r = amdgpu_amdkfd_resume_iommu(adev);
> > +     if (r)
> > +             goto init_failed;
> > +
> >        amdgpu_fru_get_product_info(adev);
> > 
> >  init_failed:
>

Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[Zhu, James]

[-- Attachment #1: Type: text/plain, Size: 5195 bytes --]

[AMD Official Use Only]

Hi Flelix,

I have tested YiFan's patch and pco s3 test can pass.

The original s3 issue always occurred on resume, And YiFan's patches
are only related to initialization.

Hi YiFan,

Reviewed-by: James Zhu <James.Zhu@amd.com> for the series

Tested-by: James Zhu <James.Zhu@amd.com> for the series



Thanks & Best Regards!


James Zhu

________________________________
From: Kuehling, Felix <Felix.Kuehling@amd.com>
Sent: Monday, October 4, 2021 11:00 AM
To: Zhang, Yifan <Yifan1.Zhang@amd.com>; Zhu, James <James.Zhu@amd.com>; amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>
Subject: Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

I'm trying to understand what the end result is after James' and your
patches. If I'm reading it correctly, we now initialize IOMMUv2 after
kfd_device_init during initialization, but before kfd_device_init during
resume from S3. Is  that the correct understanding?

My concern is, that we may run into related problems again if the root
cause and the mechanism of the fix are poorly understood. Do you have an
explanation why this different sequence is needed during init and resume?

That said, given that your patches are tested and reviewed by James, you
can add my Acked-by to both patches. The series is

Acked-by: Felix Kuehling <Felix.Kuehling@amd.com>

Thanks,
  Felix


Am 2021-10-04 um 5:42 a.m. schrieb Zhang, Yifan:
>
> [AMD Official Use Only]
>
>
>
> Hi Felix,
>
>
>
> After sync w/ James, we agree that this patch series could fix both
> our problems, and he verified this patch series will not cause
> regression of his previous issue. Do you have more comments regarding
> this patch series ? Thanks.
>
>
>
> BRs,
>
> Yifan
>
>
>
> *From:* Zhu, James <James.Zhu@amd.com>
> *Sent:* Wednesday, September 29, 2021 9:19 PM
> *To:* Kuehling, Felix <Felix.Kuehling@amd.com>; Zhang, Yifan
> <Yifan1.Zhang@amd.com>; amd-gfx@lists.freedesktop.org
> *Subject:* Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init
>
>
>
> [AMD Official Use Only]
>
>
>
> H Felix,
>
>
>
> Since the previous patch can help on PCO suspend/resume hung issue.
> Let me work with YiFan to see if
>
> there is proper way to cover both cases.
>
>
>
> Thanks & Best Regards!
>
>
>
> James Zhu
>
> ------------------------------------------------------------------------
>
> *From:*Kuehling, Felix <Felix.Kuehling@amd.com
> <mailto:Felix.Kuehling@amd.com>>
> *Sent:* Tuesday, September 28, 2021 11:41 AM
> *To:* Zhang, Yifan <Yifan1.Zhang@amd.com
> <mailto:Yifan1.Zhang@amd.com>>; amd-gfx@lists.freedesktop.org
> <mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org
> <mailto:amd-gfx@lists.freedesktop.org>>; Zhu, James <James.Zhu@amd.com
> <mailto:James.Zhu@amd.com>>
> *Subject:* Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device
> init
>
>
>
> [+James]
>
> This basically undoes James's change "drm/amdgpu: move iommu_resume
> before ip init/resume". I assume James made his change for a reason. Can
> you please discuss the issue with him and determine a solution that
> solves both your problem and his?
>
> If James' patch series was a mistake, I'd prefer to revert his patches,
> because his patches complicated the initialization sequence and exposed
> the iommu init sequence in amdgpu.
>
> Thanks,
>   Felix
>
>
> Am 2021-09-28 um 4:28 a.m. schrieb Yifan Zhang:
> > This patch is to fix clinfo failure in Raven/Picasso:
> >
> > Number of platforms: 1
> >   Platform Profile: FULL_PROFILE
> >   Platform Version: OpenCL 2.2 AMD-APP (3364.0)
> >   Platform Name: AMD Accelerated Parallel Processing
> >   Platform Vendor: Advanced Micro Devices, Inc.
> >   Platform Extensions: cl_khr_icd cl_amd_event_callback
> >
> >   Platform Name: AMD Accelerated Parallel Processing Number of
> devices: 0
> >
> > Signed-off-by: Yifan Zhang <yifan1.zhang@amd.com
> <mailto:yifan1.zhang@amd.com>>
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 ++++----
> >  1 file changed, 4 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > index 4c8f2f4647c0..89ed9b091386 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > @@ -2393,10 +2393,6 @@ static int amdgpu_device_ip_init(struct
> amdgpu_device *adev)
> >        if (r)
> >                goto init_failed;
> >
> > -     r = amdgpu_amdkfd_resume_iommu(adev);
> > -     if (r)
> > -             goto init_failed;
> > -
> >        r = amdgpu_device_ip_hw_init_phase1(adev);
> >        if (r)
> >                goto init_failed;
> > @@ -2435,6 +2431,10 @@ static int amdgpu_device_ip_init(struct
> amdgpu_device *adev)
> >        if (!adev->gmc.xgmi.pending_reset)
> >                amdgpu_amdkfd_device_init(adev);
> >
> > +     r = amdgpu_amdkfd_resume_iommu(adev);
> > +     if (r)
> > +             goto init_failed;
> > +
> >        amdgpu_fru_get_product_info(adev);
> >
> >  init_failed:
>

[-- Attachment #2: Type: text/html, Size: 10388 bytes --]

Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[youling257]

my kernel config CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_V2=y.
linux kernel 5.15rc2 "drm/amdgpu: move iommu_resume before ip init/resume" cause my amd 3400g suspend to disk resume failed, have to press power button to force shutdown.
linux kernel 5.15rc5 "drm/amdgpu: init iommu after amdkfd device init" cause my amd 3400g blackscreen when boot enter my userspace.
i need revert "drm/amdgpu: init iommu after amdkfd device init" and "drm/amdgpu: move iommu_resume before ip init/resume" for my userspace, running androidx86 with mesa21.3 on amdgpu.

Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[youling 257]

drm/amdgpu: init iommu after amdkfd device init
but CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_V2=y
[    0.203386] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    0.203387] AMD-Vi: AMD IOMMUv2 functionality not available on this system
[    7.622052] kfd kfd: amdgpu: Allocated 3969056 bytes on gart
[    7.622128] kfd kfd: amdgpu: error getting iommu info. is the iommu enabled?
[    7.622129] kfd kfd: amdgpu: Error initializing iommuv2
[    7.622430] kfd kfd: amdgpu: device 1002:15d8 NOT added due to errors

2021-10-11 14:13 GMT+08:00, youling257 <youling257@gmail.com>:
> my kernel config CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_V2=y.
> linux kernel 5.15rc2 "drm/amdgpu: move iommu_resume before ip init/resume"
> cause my amd 3400g suspend to disk resume failed, have to press power button
> to force shutdown.
> linux kernel 5.15rc5 "drm/amdgpu: init iommu after amdkfd device init" cause
> my amd 3400g blackscreen when boot enter my userspace.
> i need revert "drm/amdgpu: init iommu after amdkfd device init" and
> "drm/amdgpu: move iommu_resume before ip init/resume" for my userspace,
> running androidx86 with mesa21.3 on amdgpu.
>

RE: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[Zhang, Yifan]

[AMD Official Use Only]

Hi youling,

Would you please send out the full dmesg ? 

BRs,
Yifan

-----Original Message-----
From: youling 257 <youling257@gmail.com> 
Sent: Monday, October 11, 2021 2:18 PM
To: Zhang, Yifan <Yifan1.Zhang@amd.com>
Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

drm/amdgpu: init iommu after amdkfd device init but CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_V2=y
[    0.203386] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    0.203387] AMD-Vi: AMD IOMMUv2 functionality not available on this system
[    7.622052] kfd kfd: amdgpu: Allocated 3969056 bytes on gart
[    7.622128] kfd kfd: amdgpu: error getting iommu info. is the iommu enabled?
[    7.622129] kfd kfd: amdgpu: Error initializing iommuv2
[    7.622430] kfd kfd: amdgpu: device 1002:15d8 NOT added due to errors

2021-10-11 14:13 GMT+08:00, youling257 <youling257@gmail.com>:
> my kernel config CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_V2=y.
> linux kernel 5.15rc2 "drm/amdgpu: move iommu_resume before ip init/resume"
> cause my amd 3400g suspend to disk resume failed, have to press power 
> button to force shutdown.
> linux kernel 5.15rc5 "drm/amdgpu: init iommu after amdkfd device init" 
> cause my amd 3400g blackscreen when boot enter my userspace.
> i need revert "drm/amdgpu: init iommu after amdkfd device init" and
> "drm/amdgpu: move iommu_resume before ip init/resume" for my 
> userspace, running androidx86 with mesa21.3 on amdgpu.
>

RE: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[Zhang, Yifan]

[-- Attachment #1: Type: text/plain, Size: 1557 bytes --]

[Public]

Hi youling,

Would you pls try this patch ? 

BRs,
Yifan

-----Original Message-----
From: youling 257 <youling257@gmail.com> 
Sent: Monday, October 11, 2021 2:18 PM
To: Zhang, Yifan <Yifan1.Zhang@amd.com>
Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

drm/amdgpu: init iommu after amdkfd device init but CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_V2=y
[    0.203386] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    0.203387] AMD-Vi: AMD IOMMUv2 functionality not available on this system
[    7.622052] kfd kfd: amdgpu: Allocated 3969056 bytes on gart
[    7.622128] kfd kfd: amdgpu: error getting iommu info. is the iommu enabled?
[    7.622129] kfd kfd: amdgpu: Error initializing iommuv2
[    7.622430] kfd kfd: amdgpu: device 1002:15d8 NOT added due to errors

2021-10-11 14:13 GMT+08:00, youling257 <youling257@gmail.com>:
> my kernel config CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_V2=y.
> linux kernel 5.15rc2 "drm/amdgpu: move iommu_resume before ip init/resume"
> cause my amd 3400g suspend to disk resume failed, have to press power 
> button to force shutdown.
> linux kernel 5.15rc5 "drm/amdgpu: init iommu after amdkfd device init" 
> cause my amd 3400g blackscreen when boot enter my userspace.
> i need revert "drm/amdgpu: init iommu after amdkfd device init" and
> "drm/amdgpu: move iommu_resume before ip init/resume" for my 
> userspace, running androidx86 with mesa21.3 on amdgpu.
>

[-- Attachment #2: 0001-drm-amdkfd-fix-boot-resume-error-when-iommuv2-disabl.patch --]
[-- Type: application/octet-stream, Size: 1644 bytes --]

From 5fe678b940c95608b52ff9145bc8fe761e0cbaec Mon Sep 17 00:00:00 2001
From: Yifan Zhang <yifan1.zhang@amd.com>
Date: Mon, 11 Oct 2021 17:53:15 +0800
Subject: [PATCH] drm/amdkfd: fix boot/resume error when iommuv2 disabled in
 bios

draft patch to fix the issue.

Signed-off-by: Yifan Zhang <yifan1.zhang@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 ----
 drivers/gpu/drm/amd/amdkfd/kfd_device.c    | 4 ++++
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index af9bdf16eefd..9dfcef2015c8 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2432,10 +2432,6 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 	if (!adev->gmc.xgmi.pending_reset)
 		amdgpu_amdkfd_device_init(adev);
 
-	r = amdgpu_amdkfd_resume_iommu(adev);
-	if (r)
-		goto init_failed;
-
 	amdgpu_fru_get_product_info(adev);
 
 init_failed:
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index 4a416231b24c..1fadc9fb168d 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -916,10 +916,14 @@ bool kgd2kfd_device_init(struct kfd_dev *kfd,
 	kfd_double_confirm_iommu_support(kfd);
 
 	if (kfd_iommu_device_init(kfd)) {
+		kfd->use_iommu_v2 = false;
 		dev_err(kfd_device, "Error initializing iommuv2\n");
 		goto device_iommu_error;
 	}
 
+	if(kgd2kfd_resume_iommu(kfd))
+		goto device_iommu_error;
+
 	kfd_cwsr_init(kfd);
 
 	svm_migrate_init((struct amdgpu_device *)kfd->kgd);
-- 
2.25.1

Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[youling 257]

test this patch can fix my boot and suspend problem.

2021-10-11 18:03 GMT+08:00, Zhang, Yifan <Yifan1.Zhang@amd.com>:
> [Public]
>
> Hi youling,
>
> Would you pls try this patch ?
>
> BRs,
> Yifan
>
> -----Original Message-----
> From: youling 257 <youling257@gmail.com>
> Sent: Monday, October 11, 2021 2:18 PM
> To: Zhang, Yifan <Yifan1.Zhang@amd.com>
> Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; amd-gfx@lists.freedesktop.org
> Subject: Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init
>
> drm/amdgpu: init iommu after amdkfd device init but CONFIG_AMD_IOMMU=y
> CONFIG_AMD_IOMMU_V2=y
> [    0.203386] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
> [    0.203387] AMD-Vi: AMD IOMMUv2 functionality not available on this
> system
> [    7.622052] kfd kfd: amdgpu: Allocated 3969056 bytes on gart
> [    7.622128] kfd kfd: amdgpu: error getting iommu info. is the iommu
> enabled?
> [    7.622129] kfd kfd: amdgpu: Error initializing iommuv2
> [    7.622430] kfd kfd: amdgpu: device 1002:15d8 NOT added due to errors
>
> 2021-10-11 14:13 GMT+08:00, youling257 <youling257@gmail.com>:
>> my kernel config CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_V2=y.
>> linux kernel 5.15rc2 "drm/amdgpu: move iommu_resume before ip
>> init/resume"
>> cause my amd 3400g suspend to disk resume failed, have to press power
>> button to force shutdown.
>> linux kernel 5.15rc5 "drm/amdgpu: init iommu after amdkfd device init"
>> cause my amd 3400g blackscreen when boot enter my userspace.
>> i need revert "drm/amdgpu: init iommu after amdkfd device init" and
>> "drm/amdgpu: move iommu_resume before ip init/resume" for my
>> userspace, running androidx86 with mesa21.3 on amdgpu.
>>
>

RE: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

[Zhang, Yifan]

[AMD Official Use Only]

Great. Thanks for testing.

-----Original Message-----
From: youling 257 <youling257@gmail.com> 
Sent: Monday, October 11, 2021 6:20 PM
To: Zhang, Yifan <Yifan1.Zhang@amd.com>
Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; amd-gfx@lists.freedesktop.org; Zhu, James <James.Zhu@amd.com>
Subject: Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device init

test this patch can fix my boot and suspend problem.

2021-10-11 18:03 GMT+08:00, Zhang, Yifan <Yifan1.Zhang@amd.com>:
> [Public]
>
> Hi youling,
>
> Would you pls try this patch ?
>
> BRs,
> Yifan
>
> -----Original Message-----
> From: youling 257 <youling257@gmail.com>
> Sent: Monday, October 11, 2021 2:18 PM
> To: Zhang, Yifan <Yifan1.Zhang@amd.com>
> Cc: Kuehling, Felix <Felix.Kuehling@amd.com>; 
> amd-gfx@lists.freedesktop.org
> Subject: Re: [PATCH 2/2] drm/amdgpu: init iommu after amdkfd device 
> init
>
> drm/amdgpu: init iommu after amdkfd device init but CONFIG_AMD_IOMMU=y 
> CONFIG_AMD_IOMMU_V2=y
> [    0.203386] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
> [    0.203387] AMD-Vi: AMD IOMMUv2 functionality not available on this
> system
> [    7.622052] kfd kfd: amdgpu: Allocated 3969056 bytes on gart
> [    7.622128] kfd kfd: amdgpu: error getting iommu info. is the iommu
> enabled?
> [    7.622129] kfd kfd: amdgpu: Error initializing iommuv2
> [    7.622430] kfd kfd: amdgpu: device 1002:15d8 NOT added due to errors
>
> 2021-10-11 14:13 GMT+08:00, youling257 <youling257@gmail.com>:
>> my kernel config CONFIG_AMD_IOMMU=y CONFIG_AMD_IOMMU_V2=y.
>> linux kernel 5.15rc2 "drm/amdgpu: move iommu_resume before ip 
>> init/resume"
>> cause my amd 3400g suspend to disk resume failed, have to press power 
>> button to force shutdown.
>> linux kernel 5.15rc5 "drm/amdgpu: init iommu after amdkfd device init"
>> cause my amd 3400g blackscreen when boot enter my userspace.
>> i need revert "drm/amdgpu: init iommu after amdkfd device init" and
>> "drm/amdgpu: move iommu_resume before ip init/resume" for my 
>> userspace, running androidx86 with mesa21.3 on amdgpu.
>>
>