[PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

[PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

[Alex Sierra]

By enabling this parameter, the system will be forced to use pcie
interface only for p2p transactions.

Signed-off-by: Alex Sierra <alex.sierra@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
 3 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index ba65d4f2ab67..3645f00e9f61 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -188,6 +188,7 @@ extern int amdgpu_discovery;
 extern int amdgpu_mes;
 extern int amdgpu_noretry;
 extern int amdgpu_force_asic_type;
+extern int amdgpu_force_no_xgmi;
 #ifdef CONFIG_HSA_AMD
 extern int sched_policy;
 extern bool debug_evictions;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 1fe850e0a94d..0a5d97a84017 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 	if (r)
 		goto init_failed;
 
-	if (adev->gmc.xgmi.num_physical_nodes > 1)
+	if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1)
 		amdgpu_xgmi_add_device(adev);
 	amdgpu_amdkfd_device_init(adev);
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 4b78ecfd35f7..22485067cf31 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
 int amdgpu_tmz = 0;
 int amdgpu_reset_method = -1; /* auto */
 int amdgpu_num_kcq = -1;
+int amdgpu_force_no_xgmi = 0;
 
 struct amdgpu_mgpu_info mgpu_info = {
 	.mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
@@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);
 MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1");
 module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
 
+/**
+ * DOC: force_no_xgmi (uint)
+ * Forces not to use xgmi interface (0 = disable, 1 = enable).
+ * Default is 0 (disabled).
+ */
+MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface");
+module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600);
+
 /**
  * DOC: si_support (int)
  * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
-- 
2.17.1

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

[Christian König]

Am 28.10.20 um 15:55 schrieb Alex Sierra:
> By enabling this parameter, the system will be forced to use pcie
> interface only for p2p transactions.

Better name that amdgpu_xgmi with a default value of enabled.

Or maybe add another bit value for amdgpu_vm_debug instead.

>
> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
> ---
>   drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
>   3 files changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index ba65d4f2ab67..3645f00e9f61 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -188,6 +188,7 @@ extern int amdgpu_discovery;
>   extern int amdgpu_mes;
>   extern int amdgpu_noretry;
>   extern int amdgpu_force_asic_type;
> +extern int amdgpu_force_no_xgmi;
>   #ifdef CONFIG_HSA_AMD
>   extern int sched_policy;
>   extern bool debug_evictions;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 1fe850e0a94d..0a5d97a84017 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
>   	if (r)
>   		goto init_failed;
>   
> -	if (adev->gmc.xgmi.num_physical_nodes > 1)
> +	if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1)

Mhm, this will most likely cause problems. You still need to add the 
device to the hive because otherwise GPU won't work.

Apart from that sounds like a good idea in general.

Christian.

>   		amdgpu_xgmi_add_device(adev);
>   	amdgpu_amdkfd_device_init(adev);
>   
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 4b78ecfd35f7..22485067cf31 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
>   int amdgpu_tmz = 0;
>   int amdgpu_reset_method = -1; /* auto */
>   int amdgpu_num_kcq = -1;
> +int amdgpu_force_no_xgmi = 0;
>   
>   struct amdgpu_mgpu_info mgpu_info = {
>   	.mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
> @@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);
>   MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1");
>   module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
>   
> +/**
> + * DOC: force_no_xgmi (uint)
> + * Forces not to use xgmi interface (0 = disable, 1 = enable).
> + * Default is 0 (disabled).
> + */
> +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface");
> +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600);
> +
>   /**
>    * DOC: si_support (int)
>    * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

[Luben Tuikov]

On 2020-10-28 10:55, Alex Sierra wrote:
> By enabling this parameter, the system will be forced to use pcie
> interface only for p2p transactions.
> 
> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
>  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
>  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
>  3 files changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> index ba65d4f2ab67..3645f00e9f61 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> @@ -188,6 +188,7 @@ extern int amdgpu_discovery;
>  extern int amdgpu_mes;
>  extern int amdgpu_noretry;
>  extern int amdgpu_force_asic_type;
> +extern int amdgpu_force_no_xgmi;
>  #ifdef CONFIG_HSA_AMD
>  extern int sched_policy;
>  extern bool debug_evictions;
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> index 1fe850e0a94d..0a5d97a84017 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
>  	if (r)
>  		goto init_failed;
>  
> -	if (adev->gmc.xgmi.num_physical_nodes > 1)
> +	if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1)
>  		amdgpu_xgmi_add_device(adev);
>  	amdgpu_amdkfd_device_init(adev);
>  
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> index 4b78ecfd35f7..22485067cf31 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
>  int amdgpu_tmz = 0;
>  int amdgpu_reset_method = -1; /* auto */
>  int amdgpu_num_kcq = -1;
> +int amdgpu_force_no_xgmi = 0;
>  
>  struct amdgpu_mgpu_info mgpu_info = {
>  	.mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
> @@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444);
>  MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1");
>  module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
>  
> +/**
> + * DOC: force_no_xgmi (uint)
> + * Forces not to use xgmi interface (0 = disable, 1 = enable).

How about using human English here? Perhaps something like,

"Forces not to use" ==> "Disables the use of the XGMI interface ..."

Or, if you use the suggestion Christian posted, you could say something like,

"Enable XGMI for P2P transactions."

Regards,
Luben

> + * Default is 0 (disabled).
> + */
> +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface");
> +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600);
> +
>  /**
>   * DOC: si_support (int)
>   * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
> 

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

[Kim, Jonathan]

[AMD Official Use Only - Internal Distribution Only]

> -----Original Message-----
> From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of
> Luben Tuikov
> Sent: Wednesday, October 28, 2020 11:12 AM
> To: Sierra Guiza, Alejandro (Alex) <Alex.Sierra@amd.com>; amd-
> gfx@lists.freedesktop.org
> Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi
>
> [CAUTION: External Email]
>
> On 2020-10-28 10:55, Alex Sierra wrote:
> > By enabling this parameter, the system will be forced to use pcie
> > interface only for p2p transactions.
> >
> > Signed-off-by: Alex Sierra <alex.sierra@amd.com>
> > ---
> >  drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
> >  3 files changed, 11 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> > index ba65d4f2ab67..3645f00e9f61 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> > @@ -188,6 +188,7 @@ extern int amdgpu_discovery;  extern int
> > amdgpu_mes;  extern int amdgpu_noretry;  extern int
> > amdgpu_force_asic_type;
> > +extern int amdgpu_force_no_xgmi;
> >  #ifdef CONFIG_HSA_AMD
> >  extern int sched_policy;
> >  extern bool debug_evictions;
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > index 1fe850e0a94d..0a5d97a84017 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> > @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct
> amdgpu_device *adev)
> >       if (r)
> >               goto init_failed;
> >
> > -     if (adev->gmc.xgmi.num_physical_nodes > 1)
> > +     if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes
> >
> > + 1)
> >               amdgpu_xgmi_add_device(adev);
> >       amdgpu_amdkfd_device_init(adev);
> >
> > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > index 4b78ecfd35f7..22485067cf31 100644
> > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;  int amdgpu_tmz
> =
> > 0;  int amdgpu_reset_method = -1; /* auto */  int amdgpu_num_kcq = -1;
> > +int amdgpu_force_no_xgmi = 0;
> >
> >  struct amdgpu_mgpu_info mgpu_info = {
> >       .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
> > @@ -522,6 +523,14 @@ module_param_named(ras_enable,
> amdgpu_ras_enable,
> > int, 0444);  MODULE_PARM_DESC(ras_mask, "Mask of RAS features to
> > enable (default 0xffffffff), only valid when ras_enable == 1");
> > module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
> >
> > +/**
> > + * DOC: force_no_xgmi (uint)
> > + * Forces not to use xgmi interface (0 = disable, 1 = enable).
>
> How about using human English here? Perhaps something like,
>
> "Forces not to use" ==> "Disables the use of the XGMI interface ..."
>
> Or, if you use the suggestion Christian posted, you could say something like,
>
> "Enable XGMI for P2P transactions."

I second the use of enable.  The fact that we can prevent xGMI init means we enable it by default.

Thanks,

Jon

>
> Regards,
> Luben
>
> > + * Default is 0 (disabled).
> > + */
> > +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi
> interface");
> > +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int,
> 0600);
> > +
> >  /**
> >   * DOC: si_support (int)
> >   * Set SI support driver. This parameter works after set config
> > CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
> >
>
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
> freedesktop.org%2Fmailman%2Flistinfo%2Famd-
> gfx&amp;data=04%7C01%7Cjonathan.kim%40amd.com%7C7e0b2ac9a2ac4c7
> 8353008d87b53e1b3%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7
> C637394947518556523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sda
> ta=ve9KxUmOXYYYS8MQJBkzW7V%2FRxCORSVWxlUzhFRYxzw%3D&amp;re
> served=0
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

[Sierra Guiza, Alejandro (Alex)]

On 10/28/2020 9:58 AM, Christian König wrote:
> Am 28.10.20 um 15:55 schrieb Alex Sierra:
>> By enabling this parameter, the system will be forced to use pcie
>> interface only for p2p transactions.
>
> Better name that amdgpu_xgmi with a default value of enabled.
>
> Or maybe add another bit value for amdgpu_vm_debug instead.

Ack

Regards,
Alex Sierra

>
>
>>
>> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
>>   3 files changed, 11 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h 
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>> index ba65d4f2ab67..3645f00e9f61 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>> @@ -188,6 +188,7 @@ extern int amdgpu_discovery;
>>   extern int amdgpu_mes;
>>   extern int amdgpu_noretry;
>>   extern int amdgpu_force_asic_type;
>> +extern int amdgpu_force_no_xgmi;
>>   #ifdef CONFIG_HSA_AMD
>>   extern int sched_policy;
>>   extern bool debug_evictions;
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c 
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> index 1fe850e0a94d..0a5d97a84017 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>> @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct 
>> amdgpu_device *adev)
>>       if (r)
>>           goto init_failed;
>>   -    if (adev->gmc.xgmi.num_physical_nodes > 1)
>> +    if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1)
>
> Mhm, this will most likely cause problems. You still need to add the 
> device to the hive because otherwise GPU won't work.

What kind of problems? So far, I have validated this using a system with 
multiple devices and running ./rocm_bandwidth_test -t. With and without 
the parameter set.

Regards,
Alex Sierra

>
> Apart from that sounds like a good idea in general.
>
> Christian.
>
>>           amdgpu_xgmi_add_device(adev);
>>       amdgpu_amdkfd_device_init(adev);
>>   diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c 
>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> index 4b78ecfd35f7..22485067cf31 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>> @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
>>   int amdgpu_tmz = 0;
>>   int amdgpu_reset_method = -1; /* auto */
>>   int amdgpu_num_kcq = -1;
>> +int amdgpu_force_no_xgmi = 0;
>>     struct amdgpu_mgpu_info mgpu_info = {
>>       .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
>> @@ -522,6 +523,14 @@ module_param_named(ras_enable, 
>> amdgpu_ras_enable, int, 0444);
>>   MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 
>> 0xffffffff), only valid when ras_enable == 1");
>>   module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
>>   +/**
>> + * DOC: force_no_xgmi (uint)
>> + * Forces not to use xgmi interface (0 = disable, 1 = enable).
>> + * Default is 0 (disabled).
>> + */
>> +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface");
>> +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600);
>> +
>>   /**
>>    * DOC: si_support (int)
>>    * Set SI support driver. This parameter works after set config 
>> CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
>
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

[Sierra Guiza, Alejandro (Alex)]

[AMD Public Use]

Please ignore this patch, it should be in a different branch. As PCIe p2p is not supported in upstream.

Regards,
Alex Sierra

> -----Original Message-----
> From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of
> Sierra Guiza, Alejandro (Alex)
> Sent: Wednesday, October 28, 2020 1:09 PM
> To: Koenig, Christian <Christian.Koenig@amd.com>; amd-
> gfx@lists.freedesktop.org
> Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi
> 
> 
> On 10/28/2020 9:58 AM, Christian König wrote:
> > Am 28.10.20 um 15:55 schrieb Alex Sierra:
> >> By enabling this parameter, the system will be forced to use pcie
> >> interface only for p2p transactions.
> >
> > Better name that amdgpu_xgmi with a default value of enabled.
> >
> > Or maybe add another bit value for amdgpu_vm_debug instead.
> 
> Ack
> 
> Regards,
> Alex Sierra
> 
> >
> >
> >>
> >> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
> >> ---
> >>   drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
> >>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
> >>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
> >>   3 files changed, 11 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> >> b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> >> index ba65d4f2ab67..3645f00e9f61 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> >> @@ -188,6 +188,7 @@ extern int amdgpu_discovery;
> >>   extern int amdgpu_mes;
> >>   extern int amdgpu_noretry;
> >>   extern int amdgpu_force_asic_type;
> >> +extern int amdgpu_force_no_xgmi;
> >>   #ifdef CONFIG_HSA_AMD
> >>   extern int sched_policy;
> >>   extern bool debug_evictions;
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> >> index 1fe850e0a94d..0a5d97a84017 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> >> @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct
> >> amdgpu_device *adev)
> >>       if (r)
> >>           goto init_failed;
> >>   -    if (adev->gmc.xgmi.num_physical_nodes > 1)
> >> +    if (!amdgpu_force_no_xgmi && adev-
> >gmc.xgmi.num_physical_nodes >
> >> +1)
> >
> > Mhm, this will most likely cause problems. You still need to add the
> > device to the hive because otherwise GPU won't work.
> 
> What kind of problems? So far, I have validated this using a system with
> multiple devices and running ./rocm_bandwidth_test -t. With and without
> the parameter set.
> 
> Regards,
> Alex Sierra
> 
> >
> > Apart from that sounds like a good idea in general.
> >
> > Christian.
> >
> >>           amdgpu_xgmi_add_device(adev);
> >>       amdgpu_amdkfd_device_init(adev);
> >>   diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> >> index 4b78ecfd35f7..22485067cf31 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> >> @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
> >>   int amdgpu_tmz = 0;
> >>   int amdgpu_reset_method = -1; /* auto */
> >>   int amdgpu_num_kcq = -1;
> >> +int amdgpu_force_no_xgmi = 0;
> >>     struct amdgpu_mgpu_info mgpu_info = {
> >>       .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
> >> @@ -522,6 +523,14 @@ module_param_named(ras_enable,
> >> amdgpu_ras_enable, int, 0444);
> >>   MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable
> (default
> >> 0xffffffff), only valid when ras_enable == 1");
> >>   module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
> >>   +/**
> >> + * DOC: force_no_xgmi (uint)
> >> + * Forces not to use xgmi interface (0 = disable, 1 = enable).
> >> + * Default is 0 (disabled).
> >> + */
> >> +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi
> interface");
> >> +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int,
> 0600);
> >> +
> >>   /**
> >>    * DOC: si_support (int)
> >>    * Set SI support driver. This parameter works after set config
> >> CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
> >
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
> freedesktop.org%2Fmailman%2Flistinfo%2Famd-
> gfx&amp;data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865
> 91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63
> 7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=
> s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&amp;reserved=0
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

[Luben Tuikov]

On 2020-10-28 15:09, Sierra Guiza, Alejandro (Alex) wrote:
> [AMD Public Use]
> 
> Please ignore this patch, it should be in a different branch. As PCIe p2p is not supported in upstream.

No problem, but if you do add it elsewhere, please use something more specific, like

	amdgpu_xgmi_p2p

as the (positive-controlled) flag, since more generic flags could be added later,
to control a more encompassing behaviour.

Regards,
Luben

> 
> Regards,
> Alex Sierra
> 
>> -----Original Message-----
>> From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of
>> Sierra Guiza, Alejandro (Alex)
>> Sent: Wednesday, October 28, 2020 1:09 PM
>> To: Koenig, Christian <Christian.Koenig@amd.com>; amd-
>> gfx@lists.freedesktop.org
>> Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi
>>
>>
>> On 10/28/2020 9:58 AM, Christian König wrote:
>>> Am 28.10.20 um 15:55 schrieb Alex Sierra:
>>>> By enabling this parameter, the system will be forced to use pcie
>>>> interface only for p2p transactions.
>>>
>>> Better name that amdgpu_xgmi with a default value of enabled.
>>>
>>> Or maybe add another bit value for amdgpu_vm_debug instead.
>>
>> Ack
>>
>> Regards,
>> Alex Sierra
>>
>>>
>>>
>>>>
>>>> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
>>>> ---
>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
>>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
>>>>   3 files changed, 11 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>>>> index ba65d4f2ab67..3645f00e9f61 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
>>>> @@ -188,6 +188,7 @@ extern int amdgpu_discovery;
>>>>   extern int amdgpu_mes;
>>>>   extern int amdgpu_noretry;
>>>>   extern int amdgpu_force_asic_type;
>>>> +extern int amdgpu_force_no_xgmi;
>>>>   #ifdef CONFIG_HSA_AMD
>>>>   extern int sched_policy;
>>>>   extern bool debug_evictions;
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>> index 1fe850e0a94d..0a5d97a84017 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
>>>> @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct
>>>> amdgpu_device *adev)
>>>>       if (r)
>>>>           goto init_failed;
>>>>   -    if (adev->gmc.xgmi.num_physical_nodes > 1)
>>>> +    if (!amdgpu_force_no_xgmi && adev-
>>> gmc.xgmi.num_physical_nodes >
>>>> +1)
>>>
>>> Mhm, this will most likely cause problems. You still need to add the
>>> device to the hive because otherwise GPU won't work.
>>
>> What kind of problems? So far, I have validated this using a system with
>> multiple devices and running ./rocm_bandwidth_test -t. With and without
>> the parameter set.
>>
>> Regards,
>> Alex Sierra
>>
>>>
>>> Apart from that sounds like a good idea in general.
>>>
>>> Christian.
>>>
>>>>           amdgpu_xgmi_add_device(adev);
>>>>       amdgpu_amdkfd_device_init(adev);
>>>>   diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>>>> index 4b78ecfd35f7..22485067cf31 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
>>>> @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
>>>>   int amdgpu_tmz = 0;
>>>>   int amdgpu_reset_method = -1; /* auto */
>>>>   int amdgpu_num_kcq = -1;
>>>> +int amdgpu_force_no_xgmi = 0;
>>>>     struct amdgpu_mgpu_info mgpu_info = {
>>>>       .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
>>>> @@ -522,6 +523,14 @@ module_param_named(ras_enable,
>>>> amdgpu_ras_enable, int, 0444);
>>>>   MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable
>> (default
>>>> 0xffffffff), only valid when ras_enable == 1");
>>>>   module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
>>>>   +/**
>>>> + * DOC: force_no_xgmi (uint)
>>>> + * Forces not to use xgmi interface (0 = disable, 1 = enable).
>>>> + * Default is 0 (disabled).
>>>> + */
>>>> +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi
>> interface");
>>>> +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int,
>> 0600);
>>>> +
>>>>   /**
>>>>    * DOC: si_support (int)
>>>>    * Set SI support driver. This parameter works after set config
>>>> CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled,
>>>
>> _______________________________________________
>> amd-gfx mailing list
>> amd-gfx@lists.freedesktop.org
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
>> freedesktop.org%2Fmailman%2Flistinfo%2Famd-
>> gfx&amp;data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865
>> 91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63
>> 7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
>> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=
>> s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&amp;reserved=0
> _______________________________________________
> amd-gfx mailing list
> amd-gfx@lists.freedesktop.org
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=04%7C01%7Cluben.tuikov%40amd.com%7Cedd479f495ff42c3059408d87b75070b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637395089882966375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=K3AKD5tbr82wVMZDKlCiVO1P3MkV%2FyryqxF3KyOl1uU%3D&amp;reserved=0
> 

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

RE: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi

[Sierra Guiza, Alejandro (Alex)]

[AMD Public Use]

Will do Luben. Thanks for the recommendation.

> -----Original Message-----
> From: Tuikov, Luben <Luben.Tuikov@amd.com>
> Sent: Thursday, October 29, 2020 12:15 PM
> To: Sierra Guiza, Alejandro (Alex) <Alex.Sierra@amd.com>; Koenig, Christian
> <Christian.Koenig@amd.com>; amd-gfx@lists.freedesktop.org
> Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi
> 
> On 2020-10-28 15:09, Sierra Guiza, Alejandro (Alex) wrote:
> > [AMD Public Use]
> >
> > Please ignore this patch, it should be in a different branch. As PCIe p2p is
> not supported in upstream.
> 
> No problem, but if you do add it elsewhere, please use something more
> specific, like
> 
> 	amdgpu_xgmi_p2p
> 
> as the (positive-controlled) flag, since more generic flags could be added
> later, to control a more encompassing behaviour.
> 
> Regards,
> Luben
> 
> >
> > Regards,
> > Alex Sierra
> >
> >> -----Original Message-----
> >> From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of
> >> Sierra Guiza, Alejandro (Alex)
> >> Sent: Wednesday, October 28, 2020 1:09 PM
> >> To: Koenig, Christian <Christian.Koenig@amd.com>; amd-
> >> gfx@lists.freedesktop.org
> >> Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no
> >> xgmi
> >>
> >>
> >> On 10/28/2020 9:58 AM, Christian König wrote:
> >>> Am 28.10.20 um 15:55 schrieb Alex Sierra:
> >>>> By enabling this parameter, the system will be forced to use pcie
> >>>> interface only for p2p transactions.
> >>>
> >>> Better name that amdgpu_xgmi with a default value of enabled.
> >>>
> >>> Or maybe add another bit value for amdgpu_vm_debug instead.
> >>
> >> Ack
> >>
> >> Regards,
> >> Alex Sierra
> >>
> >>>
> >>>
> >>>>
> >>>> Signed-off-by: Alex Sierra <alex.sierra@amd.com>
> >>>> ---
> >>>>   drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
> >>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +-
> >>>>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 9 +++++++++
> >>>>   3 files changed, 11 insertions(+), 1 deletion(-)
> >>>>
> >>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> >>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> >>>> index ba65d4f2ab67..3645f00e9f61 100644
> >>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> >>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> >>>> @@ -188,6 +188,7 @@ extern int amdgpu_discovery;
> >>>>   extern int amdgpu_mes;
> >>>>   extern int amdgpu_noretry;
> >>>>   extern int amdgpu_force_asic_type;
> >>>> +extern int amdgpu_force_no_xgmi;
> >>>>   #ifdef CONFIG_HSA_AMD
> >>>>   extern int sched_policy;
> >>>>   extern bool debug_evictions;
> >>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> >>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> >>>> index 1fe850e0a94d..0a5d97a84017 100644
> >>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> >>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
> >>>> @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct
> >>>> amdgpu_device *adev)
> >>>>       if (r)
> >>>>           goto init_failed;
> >>>>   -    if (adev->gmc.xgmi.num_physical_nodes > 1)
> >>>> +    if (!amdgpu_force_no_xgmi && adev-
> >>> gmc.xgmi.num_physical_nodes >
> >>>> +1)
> >>>
> >>> Mhm, this will most likely cause problems. You still need to add the
> >>> device to the hive because otherwise GPU won't work.
> >>
> >> What kind of problems? So far, I have validated this using a system
> >> with multiple devices and running ./rocm_bandwidth_test -t. With and
> >> without the parameter set.
> >>
> >> Regards,
> >> Alex Sierra
> >>
> >>>
> >>> Apart from that sounds like a good idea in general.
> >>>
> >>> Christian.
> >>>
> >>>>           amdgpu_xgmi_add_device(adev);
> >>>>       amdgpu_amdkfd_device_init(adev);
> >>>>   diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> >>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> >>>> index 4b78ecfd35f7..22485067cf31 100644
> >>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> >>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> >>>> @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1;
> >>>>   int amdgpu_tmz = 0;
> >>>>   int amdgpu_reset_method = -1; /* auto */
> >>>>   int amdgpu_num_kcq = -1;
> >>>> +int amdgpu_force_no_xgmi = 0;
> >>>>     struct amdgpu_mgpu_info mgpu_info = {
> >>>>       .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex),
> >>>> @@ -522,6 +523,14 @@ module_param_named(ras_enable,
> >>>> amdgpu_ras_enable, int, 0444);
> >>>>   MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable
> >> (default
> >>>> 0xffffffff), only valid when ras_enable == 1");
> >>>>   module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444);
> >>>>   +/**
> >>>> + * DOC: force_no_xgmi (uint)
> >>>> + * Forces not to use xgmi interface (0 = disable, 1 = enable).
> >>>> + * Default is 0 (disabled).
> >>>> + */
> >>>> +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi
> >> interface");
> >>>> +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi,
> int,
> >> 0600);
> >>>> +
> >>>>   /**
> >>>>    * DOC: si_support (int)
> >>>>    * Set SI support driver. This parameter works after set config
> >>>> CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is
> enabled,
> >>>
> >> _______________________________________________
> >> amd-gfx mailing list
> >> amd-gfx@lists.freedesktop.org
> >>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.
> >> freedesktop.org%2Fmailman%2Flistinfo%2Famd-
> >>
> gfx&amp;data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865
> >>
> 91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63
> >>
> 7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> >>
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=
> >>
> s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&amp;reserved=0
> > _______________________________________________
> > amd-gfx mailing list
> > amd-gfx@lists.freedesktop.org
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
> > s.freedesktop.org%2Fmailman%2Flistinfo%2Famd-
> gfx&amp;data=04%7C01%7Clu
> >
> ben.tuikov%40amd.com%7Cedd479f495ff42c3059408d87b75070b%7C3dd896
> 1fe488
> >
> 4e608e11a82d994e183d%7C0%7C0%7C637395089882966375%7CUnknown%7
> CTWFpbGZs
> >
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
> 0%3D
> >
> %7C1000&amp;sdata=K3AKD5tbr82wVMZDKlCiVO1P3MkV%2FyryqxF3KyOl1
> uU%3D&amp
> > ;reserved=0
> >
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx