[PATCH 1/3] drm/ttm: Helper function to get TTM mem limit

[PATCH 1/3] drm/ttm: Helper function to get TTM mem limit

[Mukul Joshi]

Add a helper function to get TTM memory limit. This is
needed by KFD to set its own internal memory limits.

Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
---
 drivers/gpu/drm/ttm/ttm_tt.c | 6 ++++++
 include/drm/ttm/ttm_tt.h     | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index d505603930a7..1f765dd7792c 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -449,3 +449,9 @@ ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
 	return &iter_tt->base;
 }
 EXPORT_SYMBOL(ttm_kmap_iter_tt_init);
+
+unsigned long ttm_tt_pages_limit(void)
+{
+	return ttm_pages_limit;
+}
+EXPORT_SYMBOL(ttm_tt_pages_limit);
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index b7d3f3843f1e..d54b2dc05d71 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -222,7 +222,7 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
 
 struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
 					    struct ttm_tt *tt);
-
+unsigned long ttm_tt_pages_limit(void);
 #if IS_ENABLED(CONFIG_AGP)
 #include <linux/agp_backend.h>
 
-- 
2.35.1

[PATCH 2/3] drm/amdgpu: Set GTT size equal to TTM mem limit

[Mukul Joshi]

Use the helper function in TTM to get TTM mem limit and
set GTT size to be equal to TTL mem limit.

Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 25 ++++++-------------------
 1 file changed, 6 insertions(+), 19 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index ce34b73d05bc..ac220c779fc8 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -1807,26 +1807,13 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
 	DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
 		 (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
 
-	/* Compute GTT size, either based on 1/2 the size of RAM size
-	 * or whatever the user passed on module init */
-	if (amdgpu_gtt_size == -1) {
-		struct sysinfo si;
-
-		si_meminfo(&si);
-		/* Certain GL unit tests for large textures can cause problems
-		 * with the OOM killer since there is no way to link this memory
-		 * to a process.  This was originally mitigated (but not necessarily
-		 * eliminated) by limiting the GTT size.  The problem is this limit
-		 * is often too low for many modern games so just make the limit 1/2
-		 * of system memory which aligns with TTM. The OOM accounting needs
-		 * to be addressed, but we shouldn't prevent common 3D applications
-		 * from being usable just to potentially mitigate that corner case.
-		 */
-		gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
-			       (u64)si.totalram * si.mem_unit / 2);
-	} else {
+	/* Compute GTT size, either based on TTM limit
+	 * or whatever the user passed on module init.
+	 */
+	if (amdgpu_gtt_size == -1)
+		gtt_size = ttm_tt_pages_limit() << PAGE_SHIFT;
+	else
 		gtt_size = (uint64_t)amdgpu_gtt_size << 20;
-	}
 
 	/* Initialize GTT memory pool */
 	r = amdgpu_gtt_mgr_init(adev, gtt_size);
-- 
2.35.1

[PATCH 3/3] drm/amdkfd: Update KFD TTM mem limit

[Mukul Joshi]

Use the helper function in TTM to get TTM memory
limit and set KFD's internal mem limit. This ensures
that KFD's TTM mem limit and actual TTM mem limit are
exactly same.

Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c       | 3 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 +++++-
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
index fed8bb9a721f..a46285841d17 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
@@ -53,7 +53,6 @@ int amdgpu_amdkfd_init(void)
 	amdgpu_amdkfd_total_mem_size *= si.mem_unit;
 
 	ret = kgd2kfd_init();
-	amdgpu_amdkfd_gpuvm_init_mem_limits();
 	kfd_initialized = !ret;
 
 	return ret;
@@ -143,6 +142,8 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev)
 	int i;
 	int last_valid_bit;
 
+	amdgpu_amdkfd_gpuvm_init_mem_limits();
+
 	if (adev->kfd.dev) {
 		struct kgd2kfd_shared_resources gpu_resources = {
 			.compute_vmid_bitmap =
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index 862e94fbf53c..1002c7834386 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -35,6 +35,7 @@
 #include <uapi/linux/kfd_ioctl.h>
 #include "amdgpu_xgmi.h"
 #include "kfd_smi_events.h"
+#include <drm/ttm/ttm_tt.h>
 
 /* Userptr restore delay, just long enough to allow consecutive VM
  * changes to accumulate
@@ -109,13 +110,16 @@ void amdgpu_amdkfd_gpuvm_init_mem_limits(void)
 	struct sysinfo si;
 	uint64_t mem;
 
+	if (kfd_mem_limit.max_system_mem_limit)
+		return;
+
 	si_meminfo(&si);
 	mem = si.freeram - si.freehigh;
 	mem *= si.mem_unit;
 
 	spin_lock_init(&kfd_mem_limit.mem_limit_lock);
 	kfd_mem_limit.max_system_mem_limit = mem - (mem >> 4);
-	kfd_mem_limit.max_ttm_mem_limit = (mem >> 1) - (mem >> 3);
+	kfd_mem_limit.max_ttm_mem_limit = ttm_tt_pages_limit() << PAGE_SHIFT;
 	pr_debug("Kernel memory limit %lluM, TTM limit %lluM\n",
 		(kfd_mem_limit.max_system_mem_limit >> 20),
 		(kfd_mem_limit.max_ttm_mem_limit >> 20));
-- 
2.35.1

RE: [PATCH 2/3] drm/amdgpu: Set GTT size equal to TTM mem limit

[Chen, Guchun]

Looks you can drop macro 'AMDGPU_DEFAULT_GTT_SIZE_MB' as well.

Regards,
Guchun

> -----Original Message-----
> From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of
> Mukul Joshi
> Sent: Wednesday, April 26, 2023 9:53 AM
> To: amd-gfx@lists.freedesktop.org; dri-devel@lists.freedesktop.org
> Cc: Joshi, Mukul <Mukul.Joshi@amd.com>; Kuehling, Felix
> <Felix.Kuehling@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>
> Subject: [PATCH 2/3] drm/amdgpu: Set GTT size equal to TTM mem limit
> 
> Use the helper function in TTM to get TTM mem limit and set GTT size to be
> equal to TTL mem limit.
> 
> Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 25 ++++++-------------------
>  1 file changed, 6 insertions(+), 19 deletions(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> index ce34b73d05bc..ac220c779fc8 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> @@ -1807,26 +1807,13 @@ int amdgpu_ttm_init(struct amdgpu_device
> *adev)
>  	DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
>  		 (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
> 
> -	/* Compute GTT size, either based on 1/2 the size of RAM size
> -	 * or whatever the user passed on module init */
> -	if (amdgpu_gtt_size == -1) {
> -		struct sysinfo si;
> -
> -		si_meminfo(&si);
> -		/* Certain GL unit tests for large textures can cause problems
> -		 * with the OOM killer since there is no way to link this
> memory
> -		 * to a process.  This was originally mitigated (but not
> necessarily
> -		 * eliminated) by limiting the GTT size.  The problem is this
> limit
> -		 * is often too low for many modern games so just make the
> limit 1/2
> -		 * of system memory which aligns with TTM. The OOM
> accounting needs
> -		 * to be addressed, but we shouldn't prevent common 3D
> applications
> -		 * from being usable just to potentially mitigate that corner
> case.
> -		 */
> -		gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
> -			       (u64)si.totalram * si.mem_unit / 2);
> -	} else {
> +	/* Compute GTT size, either based on TTM limit
> +	 * or whatever the user passed on module init.
> +	 */
> +	if (amdgpu_gtt_size == -1)
> +		gtt_size = ttm_tt_pages_limit() << PAGE_SHIFT;
> +	else
>  		gtt_size = (uint64_t)amdgpu_gtt_size << 20;
> -	}
> 
>  	/* Initialize GTT memory pool */
>  	r = amdgpu_gtt_mgr_init(adev, gtt_size);
> --
> 2.35.1

Re: [PATCH 1/3] drm/ttm: Helper function to get TTM mem limit

[Christian König]

Am 26.04.23 um 03:52 schrieb Mukul Joshi:
> Add a helper function to get TTM memory limit. This is
> needed by KFD to set its own internal memory limits.
>
> Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>

Reviewed-by: Christian König <christian.koenig@amd.com> for the series.

> ---
>   drivers/gpu/drm/ttm/ttm_tt.c | 6 ++++++
>   include/drm/ttm/ttm_tt.h     | 2 +-
>   2 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> index d505603930a7..1f765dd7792c 100644
> --- a/drivers/gpu/drm/ttm/ttm_tt.c
> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> @@ -449,3 +449,9 @@ ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
>   	return &iter_tt->base;
>   }
>   EXPORT_SYMBOL(ttm_kmap_iter_tt_init);
> +
> +unsigned long ttm_tt_pages_limit(void)
> +{
> +	return ttm_pages_limit;
> +}
> +EXPORT_SYMBOL(ttm_tt_pages_limit);
> diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
> index b7d3f3843f1e..d54b2dc05d71 100644
> --- a/include/drm/ttm/ttm_tt.h
> +++ b/include/drm/ttm/ttm_tt.h
> @@ -222,7 +222,7 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
>   
>   struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
>   					    struct ttm_tt *tt);
> -
> +unsigned long ttm_tt_pages_limit(void);
>   #if IS_ENABLED(CONFIG_AGP)
>   #include <linux/agp_backend.h>
>   

RE: [PATCH 2/3] drm/amdgpu: Set GTT size equal to TTM mem limit

[Joshi, Mukul]

[AMD Official Use Only - General]



> -----Original Message-----
> From: Chen, Guchun <Guchun.Chen@amd.com>
> Sent: Wednesday, April 26, 2023 2:00 AM
> To: Joshi, Mukul <Mukul.Joshi@amd.com>; amd-gfx@lists.freedesktop.org;
> dri-devel@lists.freedesktop.org
> Cc: Joshi, Mukul <Mukul.Joshi@amd.com>; Kuehling, Felix
> <Felix.Kuehling@amd.com>; Koenig, Christian <Christian.Koenig@amd.com>
> Subject: RE: [PATCH 2/3] drm/amdgpu: Set GTT size equal to TTM mem limit
> 
> Looks you can drop macro 'AMDGPU_DEFAULT_GTT_SIZE_MB' as well.
> 
Thanks for catching this. Sorry I missed it. I will remove and send a v2.

Regards,
Mukul
> Regards,
> Guchun
> 
> > -----Original Message-----
> > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of
> > Mukul Joshi
> > Sent: Wednesday, April 26, 2023 9:53 AM
> > To: amd-gfx@lists.freedesktop.org; dri-devel@lists.freedesktop.org
> > Cc: Joshi, Mukul <Mukul.Joshi@amd.com>; Kuehling, Felix
> > <Felix.Kuehling@amd.com>; Koenig, Christian
> <Christian.Koenig@amd.com>
> > Subject: [PATCH 2/3] drm/amdgpu: Set GTT size equal to TTM mem limit
> >
> > Use the helper function in TTM to get TTM mem limit and set GTT size
> > to be equal to TTL mem limit.
> >
> > Signed-off-by: Mukul Joshi <mukul.joshi@amd.com>
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 25
> > ++++++-------------------
> >  1 file changed, 6 insertions(+), 19 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > index ce34b73d05bc..ac220c779fc8 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
> > @@ -1807,26 +1807,13 @@ int amdgpu_ttm_init(struct amdgpu_device
> > *adev)
> >  	DRM_INFO("amdgpu: %uM of VRAM memory ready\n",
> >  		 (unsigned) (adev->gmc.real_vram_size / (1024 * 1024)));
> >
> > -	/* Compute GTT size, either based on 1/2 the size of RAM size
> > -	 * or whatever the user passed on module init */
> > -	if (amdgpu_gtt_size == -1) {
> > -		struct sysinfo si;
> > -
> > -		si_meminfo(&si);
> > -		/* Certain GL unit tests for large textures can cause problems
> > -		 * with the OOM killer since there is no way to link this
> > memory
> > -		 * to a process.  This was originally mitigated (but not
> > necessarily
> > -		 * eliminated) by limiting the GTT size.  The problem is this
> > limit
> > -		 * is often too low for many modern games so just make the
> > limit 1/2
> > -		 * of system memory which aligns with TTM. The OOM
> > accounting needs
> > -		 * to be addressed, but we shouldn't prevent common 3D
> > applications
> > -		 * from being usable just to potentially mitigate that corner
> > case.
> > -		 */
> > -		gtt_size = max((AMDGPU_DEFAULT_GTT_SIZE_MB << 20),
> > -			       (u64)si.totalram * si.mem_unit / 2);
> > -	} else {
> > +	/* Compute GTT size, either based on TTM limit
> > +	 * or whatever the user passed on module init.
> > +	 */
> > +	if (amdgpu_gtt_size == -1)
> > +		gtt_size = ttm_tt_pages_limit() << PAGE_SHIFT;
> > +	else
> >  		gtt_size = (uint64_t)amdgpu_gtt_size << 20;
> > -	}
> >
> >  	/* Initialize GTT memory pool */
> >  	r = amdgpu_gtt_mgr_init(adev, gtt_size);
> > --
> > 2.35.1