* [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi @ 2020-10-28 14:55 Alex Sierra 2020-10-28 14:58 ` Christian König 2020-10-28 15:12 ` Luben Tuikov 0 siblings, 2 replies; 8+ messages in thread From: Alex Sierra @ 2020-10-28 14:55 UTC (permalink / raw) To: amd-gfx; +Cc: Alex Sierra By enabling this parameter, the system will be forced to use pcie interface only for p2p transactions. Signed-off-by: Alex Sierra <alex.sierra@amd.com> --- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h index ba65d4f2ab67..3645f00e9f61 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h @@ -188,6 +188,7 @@ extern int amdgpu_discovery; extern int amdgpu_mes; extern int amdgpu_noretry; extern int amdgpu_force_asic_type; +extern int amdgpu_force_no_xgmi; #ifdef CONFIG_HSA_AMD extern int sched_policy; extern bool debug_evictions; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 1fe850e0a94d..0a5d97a84017 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev) if (r) goto init_failed; - if (adev->gmc.xgmi.num_physical_nodes > 1) + if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1) amdgpu_xgmi_add_device(adev); amdgpu_amdkfd_device_init(adev); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c index 4b78ecfd35f7..22485067cf31 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; int amdgpu_tmz = 0; int amdgpu_reset_method = -1; /* auto */ int amdgpu_num_kcq = -1; +int amdgpu_force_no_xgmi = 0; struct amdgpu_mgpu_info mgpu_info = { .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex), @@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444); MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1"); module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); +/** + * DOC: force_no_xgmi (uint) + * Forces not to use xgmi interface (0 = disable, 1 = enable). + * Default is 0 (disabled). + */ +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface"); +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600); + /** * DOC: si_support (int) * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled, -- 2.17.1 _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi 2020-10-28 14:55 [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi Alex Sierra @ 2020-10-28 14:58 ` Christian König 2020-10-28 18:08 ` Sierra Guiza, Alejandro (Alex) 2020-10-28 15:12 ` Luben Tuikov 1 sibling, 1 reply; 8+ messages in thread From: Christian König @ 2020-10-28 14:58 UTC (permalink / raw) To: Alex Sierra, amd-gfx Am 28.10.20 um 15:55 schrieb Alex Sierra: > By enabling this parameter, the system will be forced to use pcie > interface only for p2p transactions. Better name that amdgpu_xgmi with a default value of enabled. Or maybe add another bit value for amdgpu_vm_debug instead. > > Signed-off-by: Alex Sierra <alex.sierra@amd.com> > --- > drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + > drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ > 3 files changed, 11 insertions(+), 1 deletion(-) > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > index ba65d4f2ab67..3645f00e9f61 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > @@ -188,6 +188,7 @@ extern int amdgpu_discovery; > extern int amdgpu_mes; > extern int amdgpu_noretry; > extern int amdgpu_force_asic_type; > +extern int amdgpu_force_no_xgmi; > #ifdef CONFIG_HSA_AMD > extern int sched_policy; > extern bool debug_evictions; > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > index 1fe850e0a94d..0a5d97a84017 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev) > if (r) > goto init_failed; > > - if (adev->gmc.xgmi.num_physical_nodes > 1) > + if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1) Mhm, this will most likely cause problems. You still need to add the device to the hive because otherwise GPU won't work. Apart from that sounds like a good idea in general. Christian. > amdgpu_xgmi_add_device(adev); > amdgpu_amdkfd_device_init(adev); > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > index 4b78ecfd35f7..22485067cf31 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; > int amdgpu_tmz = 0; > int amdgpu_reset_method = -1; /* auto */ > int amdgpu_num_kcq = -1; > +int amdgpu_force_no_xgmi = 0; > > struct amdgpu_mgpu_info mgpu_info = { > .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex), > @@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444); > MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1"); > module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); > > +/** > + * DOC: force_no_xgmi (uint) > + * Forces not to use xgmi interface (0 = disable, 1 = enable). > + * Default is 0 (disabled). > + */ > +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface"); > +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600); > + > /** > * DOC: si_support (int) > * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled, _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi 2020-10-28 14:58 ` Christian König @ 2020-10-28 18:08 ` Sierra Guiza, Alejandro (Alex) 2020-10-28 19:09 ` Sierra Guiza, Alejandro (Alex) 0 siblings, 1 reply; 8+ messages in thread From: Sierra Guiza, Alejandro (Alex) @ 2020-10-28 18:08 UTC (permalink / raw) To: christian.koenig, amd-gfx On 10/28/2020 9:58 AM, Christian König wrote: > Am 28.10.20 um 15:55 schrieb Alex Sierra: >> By enabling this parameter, the system will be forced to use pcie >> interface only for p2p transactions. > > Better name that amdgpu_xgmi with a default value of enabled. > > Or maybe add another bit value for amdgpu_vm_debug instead. Ack Regards, Alex Sierra > > >> >> Signed-off-by: Alex Sierra <alex.sierra@amd.com> >> --- >> drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + >> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- >> drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ >> 3 files changed, 11 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h >> b/drivers/gpu/drm/amd/amdgpu/amdgpu.h >> index ba65d4f2ab67..3645f00e9f61 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h >> @@ -188,6 +188,7 @@ extern int amdgpu_discovery; >> extern int amdgpu_mes; >> extern int amdgpu_noretry; >> extern int amdgpu_force_asic_type; >> +extern int amdgpu_force_no_xgmi; >> #ifdef CONFIG_HSA_AMD >> extern int sched_policy; >> extern bool debug_evictions; >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >> index 1fe850e0a94d..0a5d97a84017 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >> @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct >> amdgpu_device *adev) >> if (r) >> goto init_failed; >> - if (adev->gmc.xgmi.num_physical_nodes > 1) >> + if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1) > > Mhm, this will most likely cause problems. You still need to add the > device to the hive because otherwise GPU won't work. What kind of problems? So far, I have validated this using a system with multiple devices and running ./rocm_bandwidth_test -t. With and without the parameter set. Regards, Alex Sierra > > Apart from that sounds like a good idea in general. > > Christian. > >> amdgpu_xgmi_add_device(adev); >> amdgpu_amdkfd_device_init(adev); >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >> index 4b78ecfd35f7..22485067cf31 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >> @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; >> int amdgpu_tmz = 0; >> int amdgpu_reset_method = -1; /* auto */ >> int amdgpu_num_kcq = -1; >> +int amdgpu_force_no_xgmi = 0; >> struct amdgpu_mgpu_info mgpu_info = { >> .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex), >> @@ -522,6 +523,14 @@ module_param_named(ras_enable, >> amdgpu_ras_enable, int, 0444); >> MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default >> 0xffffffff), only valid when ras_enable == 1"); >> module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); >> +/** >> + * DOC: force_no_xgmi (uint) >> + * Forces not to use xgmi interface (0 = disable, 1 = enable). >> + * Default is 0 (disabled). >> + */ >> +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface"); >> +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600); >> + >> /** >> * DOC: si_support (int) >> * Set SI support driver. This parameter works after set config >> CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled, > _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi 2020-10-28 18:08 ` Sierra Guiza, Alejandro (Alex) @ 2020-10-28 19:09 ` Sierra Guiza, Alejandro (Alex) 2020-10-29 17:14 ` Luben Tuikov 0 siblings, 1 reply; 8+ messages in thread From: Sierra Guiza, Alejandro (Alex) @ 2020-10-28 19:09 UTC (permalink / raw) To: Sierra Guiza, Alejandro (Alex), Koenig, Christian, amd-gfx [AMD Public Use] Please ignore this patch, it should be in a different branch. As PCIe p2p is not supported in upstream. Regards, Alex Sierra > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of > Sierra Guiza, Alejandro (Alex) > Sent: Wednesday, October 28, 2020 1:09 PM > To: Koenig, Christian <Christian.Koenig@amd.com>; amd- > gfx@lists.freedesktop.org > Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi > > > On 10/28/2020 9:58 AM, Christian König wrote: > > Am 28.10.20 um 15:55 schrieb Alex Sierra: > >> By enabling this parameter, the system will be forced to use pcie > >> interface only for p2p transactions. > > > > Better name that amdgpu_xgmi with a default value of enabled. > > > > Or maybe add another bit value for amdgpu_vm_debug instead. > > Ack > > Regards, > Alex Sierra > > > > > > >> > >> Signed-off-by: Alex Sierra <alex.sierra@amd.com> > >> --- > >> drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + > >> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- > >> drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ > >> 3 files changed, 11 insertions(+), 1 deletion(-) > >> > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h > >> b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > >> index ba65d4f2ab67..3645f00e9f61 100644 > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > >> @@ -188,6 +188,7 @@ extern int amdgpu_discovery; > >> extern int amdgpu_mes; > >> extern int amdgpu_noretry; > >> extern int amdgpu_force_asic_type; > >> +extern int amdgpu_force_no_xgmi; > >> #ifdef CONFIG_HSA_AMD > >> extern int sched_policy; > >> extern bool debug_evictions; > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > >> index 1fe850e0a94d..0a5d97a84017 100644 > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > >> @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct > >> amdgpu_device *adev) > >> if (r) > >> goto init_failed; > >> - if (adev->gmc.xgmi.num_physical_nodes > 1) > >> + if (!amdgpu_force_no_xgmi && adev- > >gmc.xgmi.num_physical_nodes > > >> +1) > > > > Mhm, this will most likely cause problems. You still need to add the > > device to the hive because otherwise GPU won't work. > > What kind of problems? So far, I have validated this using a system with > multiple devices and running ./rocm_bandwidth_test -t. With and without > the parameter set. > > Regards, > Alex Sierra > > > > > Apart from that sounds like a good idea in general. > > > > Christian. > > > >> amdgpu_xgmi_add_device(adev); > >> amdgpu_amdkfd_device_init(adev); > >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > >> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > >> index 4b78ecfd35f7..22485067cf31 100644 > >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > >> @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; > >> int amdgpu_tmz = 0; > >> int amdgpu_reset_method = -1; /* auto */ > >> int amdgpu_num_kcq = -1; > >> +int amdgpu_force_no_xgmi = 0; > >> struct amdgpu_mgpu_info mgpu_info = { > >> .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex), > >> @@ -522,6 +523,14 @@ module_param_named(ras_enable, > >> amdgpu_ras_enable, int, 0444); > >> MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable > (default > >> 0xffffffff), only valid when ras_enable == 1"); > >> module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); > >> +/** > >> + * DOC: force_no_xgmi (uint) > >> + * Forces not to use xgmi interface (0 = disable, 1 = enable). > >> + * Default is 0 (disabled). > >> + */ > >> +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi > interface"); > >> +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, > 0600); > >> + > >> /** > >> * DOC: si_support (int) > >> * Set SI support driver. This parameter works after set config > >> CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled, > > > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists. > freedesktop.org%2Fmailman%2Flistinfo%2Famd- > gfx&data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865 > 91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63 > 7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD > AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata= > s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&reserved=0 _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi 2020-10-28 19:09 ` Sierra Guiza, Alejandro (Alex) @ 2020-10-29 17:14 ` Luben Tuikov 2020-10-29 18:08 ` Sierra Guiza, Alejandro (Alex) 0 siblings, 1 reply; 8+ messages in thread From: Luben Tuikov @ 2020-10-29 17:14 UTC (permalink / raw) To: Sierra Guiza, Alejandro (Alex), Koenig, Christian, amd-gfx On 2020-10-28 15:09, Sierra Guiza, Alejandro (Alex) wrote: > [AMD Public Use] > > Please ignore this patch, it should be in a different branch. As PCIe p2p is not supported in upstream. No problem, but if you do add it elsewhere, please use something more specific, like amdgpu_xgmi_p2p as the (positive-controlled) flag, since more generic flags could be added later, to control a more encompassing behaviour. Regards, Luben > > Regards, > Alex Sierra > >> -----Original Message----- >> From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of >> Sierra Guiza, Alejandro (Alex) >> Sent: Wednesday, October 28, 2020 1:09 PM >> To: Koenig, Christian <Christian.Koenig@amd.com>; amd- >> gfx@lists.freedesktop.org >> Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi >> >> >> On 10/28/2020 9:58 AM, Christian König wrote: >>> Am 28.10.20 um 15:55 schrieb Alex Sierra: >>>> By enabling this parameter, the system will be forced to use pcie >>>> interface only for p2p transactions. >>> >>> Better name that amdgpu_xgmi with a default value of enabled. >>> >>> Or maybe add another bit value for amdgpu_vm_debug instead. >> >> Ack >> >> Regards, >> Alex Sierra >> >>> >>> >>>> >>>> Signed-off-by: Alex Sierra <alex.sierra@amd.com> >>>> --- >>>> drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + >>>> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- >>>> drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ >>>> 3 files changed, 11 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h >>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu.h >>>> index ba65d4f2ab67..3645f00e9f61 100644 >>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h >>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h >>>> @@ -188,6 +188,7 @@ extern int amdgpu_discovery; >>>> extern int amdgpu_mes; >>>> extern int amdgpu_noretry; >>>> extern int amdgpu_force_asic_type; >>>> +extern int amdgpu_force_no_xgmi; >>>> #ifdef CONFIG_HSA_AMD >>>> extern int sched_policy; >>>> extern bool debug_evictions; >>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>>> index 1fe850e0a94d..0a5d97a84017 100644 >>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c >>>> @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct >>>> amdgpu_device *adev) >>>> if (r) >>>> goto init_failed; >>>> - if (adev->gmc.xgmi.num_physical_nodes > 1) >>>> + if (!amdgpu_force_no_xgmi && adev- >>> gmc.xgmi.num_physical_nodes > >>>> +1) >>> >>> Mhm, this will most likely cause problems. You still need to add the >>> device to the hive because otherwise GPU won't work. >> >> What kind of problems? So far, I have validated this using a system with >> multiple devices and running ./rocm_bandwidth_test -t. With and without >> the parameter set. >> >> Regards, >> Alex Sierra >> >>> >>> Apart from that sounds like a good idea in general. >>> >>> Christian. >>> >>>> amdgpu_xgmi_add_device(adev); >>>> amdgpu_amdkfd_device_init(adev); >>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >>>> index 4b78ecfd35f7..22485067cf31 100644 >>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c >>>> @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; >>>> int amdgpu_tmz = 0; >>>> int amdgpu_reset_method = -1; /* auto */ >>>> int amdgpu_num_kcq = -1; >>>> +int amdgpu_force_no_xgmi = 0; >>>> struct amdgpu_mgpu_info mgpu_info = { >>>> .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex), >>>> @@ -522,6 +523,14 @@ module_param_named(ras_enable, >>>> amdgpu_ras_enable, int, 0444); >>>> MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable >> (default >>>> 0xffffffff), only valid when ras_enable == 1"); >>>> module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); >>>> +/** >>>> + * DOC: force_no_xgmi (uint) >>>> + * Forces not to use xgmi interface (0 = disable, 1 = enable). >>>> + * Default is 0 (disabled). >>>> + */ >>>> +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi >> interface"); >>>> +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, >> 0600); >>>> + >>>> /** >>>> * DOC: si_support (int) >>>> * Set SI support driver. This parameter works after set config >>>> CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled, >>> >> _______________________________________________ >> amd-gfx mailing list >> amd-gfx@lists.freedesktop.org >> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists. >> freedesktop.org%2Fmailman%2Flistinfo%2Famd- >> gfx&data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865 >> 91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63 >> 7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD >> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata= >> s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&reserved=0 > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=04%7C01%7Cluben.tuikov%40amd.com%7Cedd479f495ff42c3059408d87b75070b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637395089882966375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=K3AKD5tbr82wVMZDKlCiVO1P3MkV%2FyryqxF3KyOl1uU%3D&reserved=0 > _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi 2020-10-29 17:14 ` Luben Tuikov @ 2020-10-29 18:08 ` Sierra Guiza, Alejandro (Alex) 0 siblings, 0 replies; 8+ messages in thread From: Sierra Guiza, Alejandro (Alex) @ 2020-10-29 18:08 UTC (permalink / raw) To: Tuikov, Luben, Koenig, Christian, amd-gfx [AMD Public Use] Will do Luben. Thanks for the recommendation. > -----Original Message----- > From: Tuikov, Luben <Luben.Tuikov@amd.com> > Sent: Thursday, October 29, 2020 12:15 PM > To: Sierra Guiza, Alejandro (Alex) <Alex.Sierra@amd.com>; Koenig, Christian > <Christian.Koenig@amd.com>; amd-gfx@lists.freedesktop.org > Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi > > On 2020-10-28 15:09, Sierra Guiza, Alejandro (Alex) wrote: > > [AMD Public Use] > > > > Please ignore this patch, it should be in a different branch. As PCIe p2p is > not supported in upstream. > > No problem, but if you do add it elsewhere, please use something more > specific, like > > amdgpu_xgmi_p2p > > as the (positive-controlled) flag, since more generic flags could be added > later, to control a more encompassing behaviour. > > Regards, > Luben > > > > > Regards, > > Alex Sierra > > > >> -----Original Message----- > >> From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of > >> Sierra Guiza, Alejandro (Alex) > >> Sent: Wednesday, October 28, 2020 1:09 PM > >> To: Koenig, Christian <Christian.Koenig@amd.com>; amd- > >> gfx@lists.freedesktop.org > >> Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no > >> xgmi > >> > >> > >> On 10/28/2020 9:58 AM, Christian König wrote: > >>> Am 28.10.20 um 15:55 schrieb Alex Sierra: > >>>> By enabling this parameter, the system will be forced to use pcie > >>>> interface only for p2p transactions. > >>> > >>> Better name that amdgpu_xgmi with a default value of enabled. > >>> > >>> Or maybe add another bit value for amdgpu_vm_debug instead. > >> > >> Ack > >> > >> Regards, > >> Alex Sierra > >> > >>> > >>> > >>>> > >>>> Signed-off-by: Alex Sierra <alex.sierra@amd.com> > >>>> --- > >>>> drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + > >>>> drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- > >>>> drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ > >>>> 3 files changed, 11 insertions(+), 1 deletion(-) > >>>> > >>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h > >>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > >>>> index ba65d4f2ab67..3645f00e9f61 100644 > >>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h > >>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > >>>> @@ -188,6 +188,7 @@ extern int amdgpu_discovery; > >>>> extern int amdgpu_mes; > >>>> extern int amdgpu_noretry; > >>>> extern int amdgpu_force_asic_type; > >>>> +extern int amdgpu_force_no_xgmi; > >>>> #ifdef CONFIG_HSA_AMD > >>>> extern int sched_policy; > >>>> extern bool debug_evictions; > >>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > >>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > >>>> index 1fe850e0a94d..0a5d97a84017 100644 > >>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > >>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > >>>> @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct > >>>> amdgpu_device *adev) > >>>> if (r) > >>>> goto init_failed; > >>>> - if (adev->gmc.xgmi.num_physical_nodes > 1) > >>>> + if (!amdgpu_force_no_xgmi && adev- > >>> gmc.xgmi.num_physical_nodes > > >>>> +1) > >>> > >>> Mhm, this will most likely cause problems. You still need to add the > >>> device to the hive because otherwise GPU won't work. > >> > >> What kind of problems? So far, I have validated this using a system > >> with multiple devices and running ./rocm_bandwidth_test -t. With and > >> without the parameter set. > >> > >> Regards, > >> Alex Sierra > >> > >>> > >>> Apart from that sounds like a good idea in general. > >>> > >>> Christian. > >>> > >>>> amdgpu_xgmi_add_device(adev); > >>>> amdgpu_amdkfd_device_init(adev); > >>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > >>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > >>>> index 4b78ecfd35f7..22485067cf31 100644 > >>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > >>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > >>>> @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; > >>>> int amdgpu_tmz = 0; > >>>> int amdgpu_reset_method = -1; /* auto */ > >>>> int amdgpu_num_kcq = -1; > >>>> +int amdgpu_force_no_xgmi = 0; > >>>> struct amdgpu_mgpu_info mgpu_info = { > >>>> .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex), > >>>> @@ -522,6 +523,14 @@ module_param_named(ras_enable, > >>>> amdgpu_ras_enable, int, 0444); > >>>> MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable > >> (default > >>>> 0xffffffff), only valid when ras_enable == 1"); > >>>> module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); > >>>> +/** > >>>> + * DOC: force_no_xgmi (uint) > >>>> + * Forces not to use xgmi interface (0 = disable, 1 = enable). > >>>> + * Default is 0 (disabled). > >>>> + */ > >>>> +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi > >> interface"); > >>>> +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, > int, > >> 0600); > >>>> + > >>>> /** > >>>> * DOC: si_support (int) > >>>> * Set SI support driver. This parameter works after set config > >>>> CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is > enabled, > >>> > >> _______________________________________________ > >> amd-gfx mailing list > >> amd-gfx@lists.freedesktop.org > >> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists. > >> freedesktop.org%2Fmailman%2Flistinfo%2Famd- > >> > gfx&data=04%7C01%7Calex.sierra%40amd.com%7C6a2e34427fb449865 > >> > 91208d87b6c8c05%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C63 > >> > 7395053457347633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD > >> > AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata= > >> > s2hXVAVFtHJsNNBwhzzKDjMlEjES9uNGbYi6GdeD5cc%3D&reserved=0 > > _______________________________________________ > > amd-gfx mailing list > > amd-gfx@lists.freedesktop.org > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist > > s.freedesktop.org%2Fmailman%2Flistinfo%2Famd- > gfx&data=04%7C01%7Clu > > > ben.tuikov%40amd.com%7Cedd479f495ff42c3059408d87b75070b%7C3dd896 > 1fe488 > > > 4e608e11a82d994e183d%7C0%7C0%7C637395089882966375%7CUnknown%7 > CTWFpbGZs > > > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn > 0%3D > > > %7C1000&sdata=K3AKD5tbr82wVMZDKlCiVO1P3MkV%2FyryqxF3KyOl1 > uU%3D& > > ;reserved=0 > > _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi 2020-10-28 14:55 [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi Alex Sierra 2020-10-28 14:58 ` Christian König @ 2020-10-28 15:12 ` Luben Tuikov 2020-10-28 15:36 ` Kim, Jonathan 1 sibling, 1 reply; 8+ messages in thread From: Luben Tuikov @ 2020-10-28 15:12 UTC (permalink / raw) To: Alex Sierra, amd-gfx On 2020-10-28 10:55, Alex Sierra wrote: > By enabling this parameter, the system will be forced to use pcie > interface only for p2p transactions. > > Signed-off-by: Alex Sierra <alex.sierra@amd.com> > --- > drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + > drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ > 3 files changed, 11 insertions(+), 1 deletion(-) > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > index ba65d4f2ab67..3645f00e9f61 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > @@ -188,6 +188,7 @@ extern int amdgpu_discovery; > extern int amdgpu_mes; > extern int amdgpu_noretry; > extern int amdgpu_force_asic_type; > +extern int amdgpu_force_no_xgmi; > #ifdef CONFIG_HSA_AMD > extern int sched_policy; > extern bool debug_evictions; > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > index 1fe850e0a94d..0a5d97a84017 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev) > if (r) > goto init_failed; > > - if (adev->gmc.xgmi.num_physical_nodes > 1) > + if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > 1) > amdgpu_xgmi_add_device(adev); > amdgpu_amdkfd_device_init(adev); > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > index 4b78ecfd35f7..22485067cf31 100644 > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; > int amdgpu_tmz = 0; > int amdgpu_reset_method = -1; /* auto */ > int amdgpu_num_kcq = -1; > +int amdgpu_force_no_xgmi = 0; > > struct amdgpu_mgpu_info mgpu_info = { > .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex), > @@ -522,6 +523,14 @@ module_param_named(ras_enable, amdgpu_ras_enable, int, 0444); > MODULE_PARM_DESC(ras_mask, "Mask of RAS features to enable (default 0xffffffff), only valid when ras_enable == 1"); > module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); > > +/** > + * DOC: force_no_xgmi (uint) > + * Forces not to use xgmi interface (0 = disable, 1 = enable). How about using human English here? Perhaps something like, "Forces not to use" ==> "Disables the use of the XGMI interface ..." Or, if you use the suggestion Christian posted, you could say something like, "Enable XGMI for P2P transactions." Regards, Luben > + * Default is 0 (disabled). > + */ > +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi interface"); > +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, 0600); > + > /** > * DOC: si_support (int) > * Set SI support driver. This parameter works after set config CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled, > _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi 2020-10-28 15:12 ` Luben Tuikov @ 2020-10-28 15:36 ` Kim, Jonathan 0 siblings, 0 replies; 8+ messages in thread From: Kim, Jonathan @ 2020-10-28 15:36 UTC (permalink / raw) To: Tuikov, Luben, Sierra Guiza, Alejandro (Alex), amd-gfx [AMD Official Use Only - Internal Distribution Only] > -----Original Message----- > From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of > Luben Tuikov > Sent: Wednesday, October 28, 2020 11:12 AM > To: Sierra Guiza, Alejandro (Alex) <Alex.Sierra@amd.com>; amd- > gfx@lists.freedesktop.org > Subject: Re: [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi > > [CAUTION: External Email] > > On 2020-10-28 10:55, Alex Sierra wrote: > > By enabling this parameter, the system will be forced to use pcie > > interface only for p2p transactions. > > > > Signed-off-by: Alex Sierra <alex.sierra@amd.com> > > --- > > drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + > > drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- > > drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 9 +++++++++ > > 3 files changed, 11 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h > > b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > > index ba65d4f2ab67..3645f00e9f61 100644 > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h > > @@ -188,6 +188,7 @@ extern int amdgpu_discovery; extern int > > amdgpu_mes; extern int amdgpu_noretry; extern int > > amdgpu_force_asic_type; > > +extern int amdgpu_force_no_xgmi; > > #ifdef CONFIG_HSA_AMD > > extern int sched_policy; > > extern bool debug_evictions; > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > index 1fe850e0a94d..0a5d97a84017 100644 > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c > > @@ -2257,7 +2257,7 @@ static int amdgpu_device_ip_init(struct > amdgpu_device *adev) > > if (r) > > goto init_failed; > > > > - if (adev->gmc.xgmi.num_physical_nodes > 1) > > + if (!amdgpu_force_no_xgmi && adev->gmc.xgmi.num_physical_nodes > > > > + 1) > > amdgpu_xgmi_add_device(adev); > > amdgpu_amdkfd_device_init(adev); > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > > index 4b78ecfd35f7..22485067cf31 100644 > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c > > @@ -160,6 +160,7 @@ int amdgpu_force_asic_type = -1; int amdgpu_tmz > = > > 0; int amdgpu_reset_method = -1; /* auto */ int amdgpu_num_kcq = -1; > > +int amdgpu_force_no_xgmi = 0; > > > > struct amdgpu_mgpu_info mgpu_info = { > > .mutex = __MUTEX_INITIALIZER(mgpu_info.mutex), > > @@ -522,6 +523,14 @@ module_param_named(ras_enable, > amdgpu_ras_enable, > > int, 0444); MODULE_PARM_DESC(ras_mask, "Mask of RAS features to > > enable (default 0xffffffff), only valid when ras_enable == 1"); > > module_param_named(ras_mask, amdgpu_ras_mask, uint, 0444); > > > > +/** > > + * DOC: force_no_xgmi (uint) > > + * Forces not to use xgmi interface (0 = disable, 1 = enable). > > How about using human English here? Perhaps something like, > > "Forces not to use" ==> "Disables the use of the XGMI interface ..." > > Or, if you use the suggestion Christian posted, you could say something like, > > "Enable XGMI for P2P transactions." I second the use of enable. The fact that we can prevent xGMI init means we enable it by default. Thanks, Jon > > Regards, > Luben > > > + * Default is 0 (disabled). > > + */ > > +MODULE_PARM_DESC(force_no_xgmi, "Force not to use xgmi > interface"); > > +module_param_named(force_no_xgmi, amdgpu_force_no_xgmi, int, > 0600); > > + > > /** > > * DOC: si_support (int) > > * Set SI support driver. This parameter works after set config > > CONFIG_DRM_AMDGPU_SI. For SI asic, when radeon driver is enabled, > > > > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists. > freedesktop.org%2Fmailman%2Flistinfo%2Famd- > gfx&data=04%7C01%7Cjonathan.kim%40amd.com%7C7e0b2ac9a2ac4c7 > 8353008d87b53e1b3%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7 > C637394947518556523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw > MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sda > ta=ve9KxUmOXYYYS8MQJBkzW7V%2FRxCORSVWxlUzhFRYxzw%3D&re > served=0 _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-10-29 18:08 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-10-28 14:55 [PATCH] drm/amdgpu: Add kernel parameter to force no xgmi Alex Sierra 2020-10-28 14:58 ` Christian König 2020-10-28 18:08 ` Sierra Guiza, Alejandro (Alex) 2020-10-28 19:09 ` Sierra Guiza, Alejandro (Alex) 2020-10-29 17:14 ` Luben Tuikov 2020-10-29 18:08 ` Sierra Guiza, Alejandro (Alex) 2020-10-28 15:12 ` Luben Tuikov 2020-10-28 15:36 ` Kim, Jonathan
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.