* [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
@ 2021-03-11 17:15 shaoyunl
2021-03-12 0:47 ` Liu, Shaoyun
2021-03-12 3:42 ` Lazar, Lijo
0 siblings, 2 replies; 9+ messages in thread
From: shaoyunl @ 2021-03-11 17:15 UTC (permalink / raw)
To: amd-gfx; +Cc: shaoyunl
This is to fix the commit dda9bbb26c7 where it only enable the light SMU
on normal device init. This feature actually need to be enabled after ASIC
been reset as well.
Signed-off-by: shaoyunl <shaoyun.liu@amd.com>
Change-Id: Ie7ee02cd3ccdab3522aad9a02f681963e211ed44
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index cada3e77c7d5..fb775a9c0db1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2513,6 +2513,9 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
if (r)
DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
+ /* For XGMI + passthrough configuration , enable light SBR */
+ if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
+ smu_set_light_sbr(&adev->smu, true);
if (adev->gmc.xgmi.num_physical_nodes > 1) {
mutex_lock(&mgpu_info.mutex);
@@ -3615,10 +3618,6 @@ int amdgpu_device_init(struct amdgpu_device *adev,
if (amdgpu_device_cache_pci_state(adev->pdev))
pci_restore_state(pdev);
- /* Enable lightSBR on SMU in passthrough + xgmi configuration */
- if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
- smu_set_light_sbr(&adev->smu, true);
-
if (adev->gmc.xgmi.pending_reset)
queue_delayed_work(system_wq, &mgpu_info.delayed_reset_work,
msecs_to_jiffies(AMDGPU_RESUME_MS));
--
2.17.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 9+ messages in thread
* RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
2021-03-11 17:15 [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration shaoyunl
@ 2021-03-12 0:47 ` Liu, Shaoyun
2021-03-12 3:42 ` Lazar, Lijo
1 sibling, 0 replies; 9+ messages in thread
From: Liu, Shaoyun @ 2021-03-12 0:47 UTC (permalink / raw)
To: amd-gfx
[AMD Official Use Only - Internal Distribution Only]
Ping .
-----Original Message-----
From: Liu, Shaoyun <Shaoyun.Liu@amd.com>
Sent: Thursday, March 11, 2021 12:16 PM
To: amd-gfx@lists.freedesktop.org
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com>
Subject: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
This is to fix the commit dda9bbb26c7 where it only enable the light SMU on normal device init. This feature actually need to be enabled after ASIC been reset as well.
Signed-off-by: shaoyunl <shaoyun.liu@amd.com>
Change-Id: Ie7ee02cd3ccdab3522aad9a02f681963e211ed44
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index cada3e77c7d5..fb775a9c0db1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2513,6 +2513,9 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
if (r)
DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
+ /* For XGMI + passthrough configuration , enable light SBR */
+ if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
+ smu_set_light_sbr(&adev->smu, true);
if (adev->gmc.xgmi.num_physical_nodes > 1) {
mutex_lock(&mgpu_info.mutex);
@@ -3615,10 +3618,6 @@ int amdgpu_device_init(struct amdgpu_device *adev,
if (amdgpu_device_cache_pci_state(adev->pdev))
pci_restore_state(pdev);
- /* Enable lightSBR on SMU in passthrough + xgmi configuration */
- if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
- smu_set_light_sbr(&adev->smu, true);
-
if (adev->gmc.xgmi.pending_reset)
queue_delayed_work(system_wq, &mgpu_info.delayed_reset_work,
msecs_to_jiffies(AMDGPU_RESUME_MS));
--
2.17.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 9+ messages in thread
* RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
2021-03-11 17:15 [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration shaoyunl
2021-03-12 0:47 ` Liu, Shaoyun
@ 2021-03-12 3:42 ` Lazar, Lijo
2021-03-12 13:51 ` Liu, Shaoyun
1 sibling, 1 reply; 9+ messages in thread
From: Lazar, Lijo @ 2021-03-12 3:42 UTC (permalink / raw)
To: Liu, Shaoyun, amd-gfx; +Cc: Liu, Shaoyun
[AMD Public Use]
We don't need this as a generic ppt_func. Reset functionalities are changing over programs and this could be valid only for Arcturus. Please move it to Arcturus swsmu late init.
Thanks,
Lijo
-----Original Message-----
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of shaoyunl
Sent: Thursday, March 11, 2021 10:46 PM
To: amd-gfx@lists.freedesktop.org
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com>
Subject: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
This is to fix the commit dda9bbb26c7 where it only enable the light SMU on normal device init. This feature actually need to be enabled after ASIC been reset as well.
Signed-off-by: shaoyunl <shaoyun.liu@amd.com>
Change-Id: Ie7ee02cd3ccdab3522aad9a02f681963e211ed44
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index cada3e77c7d5..fb775a9c0db1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2513,6 +2513,9 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
if (r)
DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
+ /* For XGMI + passthrough configuration , enable light SBR */
+ if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
+ smu_set_light_sbr(&adev->smu, true);
if (adev->gmc.xgmi.num_physical_nodes > 1) {
mutex_lock(&mgpu_info.mutex);
@@ -3615,10 +3618,6 @@ int amdgpu_device_init(struct amdgpu_device *adev,
if (amdgpu_device_cache_pci_state(adev->pdev))
pci_restore_state(pdev);
- /* Enable lightSBR on SMU in passthrough + xgmi configuration */
- if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
- smu_set_light_sbr(&adev->smu, true);
-
if (adev->gmc.xgmi.pending_reset)
queue_delayed_work(system_wq, &mgpu_info.delayed_reset_work,
msecs_to_jiffies(AMDGPU_RESUME_MS));
--
2.17.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=04%7C01%7Clijo.lazar%40amd.com%7Cc5aedb6c2d9d49d2fee408d8e4b15b5c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637510797685776785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7ndhtOVmyZcRMe3UQiGvF%2BprCdBVgo7f6IATXSbQNg4%3D&reserved=0
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
2021-03-12 3:42 ` Lazar, Lijo
@ 2021-03-12 13:51 ` Liu, Shaoyun
2021-03-12 13:54 ` Lazar, Lijo
0 siblings, 1 reply; 9+ messages in thread
From: Liu, Shaoyun @ 2021-03-12 13:51 UTC (permalink / raw)
To: Lazar, Lijo, amd-gfx
[-- Attachment #1.1: Type: text/plain, Size: 3481 bytes --]
Thanks for the comments. This light sbr solution could be applied to other asic as well. In swsmu code, It will check whether the function pointer set_light_sbr is valid before real call the function. So for other asics if the smu apply the same change, just add the ppt function pointer and we will have this support without further code change.
Thanks
Shaoyun.liu
________________________________
From: Lazar, Lijo <Lijo.Lazar@amd.com>
Sent: March 11, 2021 10:42 PM
To: Liu, Shaoyun <Shaoyun.Liu@amd.com>; amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com>
Subject: RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
[AMD Public Use]
We don't need this as a generic ppt_func. Reset functionalities are changing over programs and this could be valid only for Arcturus. Please move it to Arcturus swsmu late init.
Thanks,
Lijo
-----Original Message-----
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org> On Behalf Of shaoyunl
Sent: Thursday, March 11, 2021 10:46 PM
To: amd-gfx@lists.freedesktop.org
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com>
Subject: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
This is to fix the commit dda9bbb26c7 where it only enable the light SMU on normal device init. This feature actually need to be enabled after ASIC been reset as well.
Signed-off-by: shaoyunl <shaoyun.liu@amd.com>
Change-Id: Ie7ee02cd3ccdab3522aad9a02f681963e211ed44
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index cada3e77c7d5..fb775a9c0db1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2513,6 +2513,9 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
if (r)
DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
+ /* For XGMI + passthrough configuration , enable light SBR */
+ if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
+ smu_set_light_sbr(&adev->smu, true);
if (adev->gmc.xgmi.num_physical_nodes > 1) {
mutex_lock(&mgpu_info.mutex);
@@ -3615,10 +3618,6 @@ int amdgpu_device_init(struct amdgpu_device *adev,
if (amdgpu_device_cache_pci_state(adev->pdev))
pci_restore_state(pdev);
- /* Enable lightSBR on SMU in passthrough + xgmi configuration */
- if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
- smu_set_light_sbr(&adev->smu, true);
-
if (adev->gmc.xgmi.pending_reset)
queue_delayed_work(system_wq, &mgpu_info.delayed_reset_work,
msecs_to_jiffies(AMDGPU_RESUME_MS));
--
2.17.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=04%7C01%7Clijo.lazar%40amd.com%7Cc5aedb6c2d9d49d2fee408d8e4b15b5c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637510797685776785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7ndhtOVmyZcRMe3UQiGvF%2BprCdBVgo7f6IATXSbQNg4%3D&reserved=0
[-- Attachment #1.2: Type: text/html, Size: 5758 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 9+ messages in thread
* RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
2021-03-12 13:51 ` Liu, Shaoyun
@ 2021-03-12 13:54 ` Lazar, Lijo
2021-03-12 15:27 ` Liu, Shaoyun
0 siblings, 1 reply; 9+ messages in thread
From: Lazar, Lijo @ 2021-03-12 13:54 UTC (permalink / raw)
To: Liu, Shaoyun, amd-gfx
[-- Attachment #1.1: Type: text/plain, Size: 4236 bytes --]
[AMD Public Use]
We want to keep ppt_funcs minimal. Adding everything to ppt_funcs and keeping as NULL is not the right way. Please keep the code to arcturus.
Thanks,
Lijo
From: Liu, Shaoyun <Shaoyun.Liu@amd.com>
Sent: Friday, March 12, 2021 7:21 PM
To: Lazar, Lijo <Lijo.Lazar@amd.com>; amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
Thanks for the comments. This light sbr solution could be applied to other asic as well. In swsmu code, It will check whether the function pointer set_light_sbr is valid before real call the function. So for other asics if the smu apply the same change, just add the ppt function pointer and we will have this support without further code change.
Thanks
Shaoyun.liu
________________________________
From: Lazar, Lijo <Lijo.Lazar@amd.com<mailto:Lijo.Lazar@amd.com>>
Sent: March 11, 2021 10:42 PM
To: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>>
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Subject: RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
[AMD Public Use]
We don't need this as a generic ppt_func. Reset functionalities are changing over programs and this could be valid only for Arcturus. Please move it to Arcturus swsmu late init.
Thanks,
Lijo
-----Original Message-----
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of shaoyunl
Sent: Thursday, March 11, 2021 10:46 PM
To: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Subject: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
This is to fix the commit dda9bbb26c7 where it only enable the light SMU on normal device init. This feature actually need to be enabled after ASIC been reset as well.
Signed-off-by: shaoyunl <shaoyun.liu@amd.com<mailto:shaoyun.liu@amd.com>>
Change-Id: Ie7ee02cd3ccdab3522aad9a02f681963e211ed44
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index cada3e77c7d5..fb775a9c0db1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2513,6 +2513,9 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
if (r)
DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
+ /* For XGMI + passthrough configuration , enable light SBR */
+ if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
+ smu_set_light_sbr(&adev->smu, true);
if (adev->gmc.xgmi.num_physical_nodes > 1) {
mutex_lock(&mgpu_info.mutex);
@@ -3615,10 +3618,6 @@ int amdgpu_device_init(struct amdgpu_device *adev,
if (amdgpu_device_cache_pci_state(adev->pdev))
pci_restore_state(pdev);
- /* Enable lightSBR on SMU in passthrough + xgmi configuration */
- if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
- smu_set_light_sbr(&adev->smu, true);
-
if (adev->gmc.xgmi.pending_reset)
queue_delayed_work(system_wq, &mgpu_info.delayed_reset_work,
msecs_to_jiffies(AMDGPU_RESUME_MS));
--
2.17.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=04%7C01%7Clijo.lazar%40amd.com%7Cc5aedb6c2d9d49d2fee408d8e4b15b5c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637510797685776785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7ndhtOVmyZcRMe3UQiGvF%2BprCdBVgo7f6IATXSbQNg4%3D&reserved=0
[-- Attachment #1.2: Type: text/html, Size: 9327 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 9+ messages in thread
* RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
2021-03-12 13:54 ` Lazar, Lijo
@ 2021-03-12 15:27 ` Liu, Shaoyun
2021-03-12 16:54 ` Lazar, Lijo
0 siblings, 1 reply; 9+ messages in thread
From: Liu, Shaoyun @ 2021-03-12 15:27 UTC (permalink / raw)
To: Lazar, Lijo, amd-gfx, Quan, Evan, Zhang, Hawking
[-- Attachment #1.1: Type: text/plain, Size: 5201 bytes --]
[AMD Public Use]
I don't like to add this set_light_sbr into ppt_funcs either , but please check current swsmu code structure , there is no asic specific swsmu late init function and there is no direct routine form amdgpu_smu.c to smu_v11_0.c either . It requires smu common code ->ppt_func -> smu_v11_0 for Arcturus specific function . So unless SMU and PPT have a major re-structure , set_light_sbr need to go through ppt_func for now, I think I better leave this re-structure task to SMU and PPT owner in the future .
Add SMU and PPT code owner Hawking and Quan for comments .
Regards
Shaoyun.liu
From: Lazar, Lijo <Lijo.Lazar@amd.com>
Sent: Friday, March 12, 2021 8:55 AM
To: Liu, Shaoyun <Shaoyun.Liu@amd.com>; amd-gfx@lists.freedesktop.org
Subject: RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
[AMD Public Use]
We want to keep ppt_funcs minimal. Adding everything to ppt_funcs and keeping as NULL is not the right way. Please keep the code to arcturus.
Thanks,
Lijo
From: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Sent: Friday, March 12, 2021 7:21 PM
To: Lazar, Lijo <Lijo.Lazar@amd.com<mailto:Lijo.Lazar@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
Thanks for the comments. This light sbr solution could be applied to other asic as well. In swsmu code, It will check whether the function pointer set_light_sbr is valid before real call the function. So for other asics if the smu apply the same change, just add the ppt function pointer and we will have this support without further code change.
Thanks
Shaoyun.liu
________________________________
From: Lazar, Lijo <Lijo.Lazar@amd.com<mailto:Lijo.Lazar@amd.com>>
Sent: March 11, 2021 10:42 PM
To: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>>
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Subject: RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
[AMD Public Use]
We don't need this as a generic ppt_func. Reset functionalities are changing over programs and this could be valid only for Arcturus. Please move it to Arcturus swsmu late init.
Thanks,
Lijo
-----Original Message-----
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of shaoyunl
Sent: Thursday, March 11, 2021 10:46 PM
To: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Subject: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
This is to fix the commit dda9bbb26c7 where it only enable the light SMU on normal device init. This feature actually need to be enabled after ASIC been reset as well.
Signed-off-by: shaoyunl <shaoyun.liu@amd.com<mailto:shaoyun.liu@amd.com>>
Change-Id: Ie7ee02cd3ccdab3522aad9a02f681963e211ed44
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index cada3e77c7d5..fb775a9c0db1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2513,6 +2513,9 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
if (r)
DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
+ /* For XGMI + passthrough configuration , enable light SBR */
+ if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
+ smu_set_light_sbr(&adev->smu, true);
if (adev->gmc.xgmi.num_physical_nodes > 1) {
mutex_lock(&mgpu_info.mutex);
@@ -3615,10 +3618,6 @@ int amdgpu_device_init(struct amdgpu_device *adev,
if (amdgpu_device_cache_pci_state(adev->pdev))
pci_restore_state(pdev);
- /* Enable lightSBR on SMU in passthrough + xgmi configuration */
- if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
- smu_set_light_sbr(&adev->smu, true);
-
if (adev->gmc.xgmi.pending_reset)
queue_delayed_work(system_wq, &mgpu_info.delayed_reset_work,
msecs_to_jiffies(AMDGPU_RESUME_MS));
--
2.17.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=04%7C01%7Clijo.lazar%40amd.com%7Cc5aedb6c2d9d49d2fee408d8e4b15b5c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637510797685776785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7ndhtOVmyZcRMe3UQiGvF%2BprCdBVgo7f6IATXSbQNg4%3D&reserved=0
[-- Attachment #1.2: Type: text/html, Size: 11598 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 9+ messages in thread
* RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
2021-03-12 15:27 ` Liu, Shaoyun
@ 2021-03-12 16:54 ` Lazar, Lijo
2021-03-12 18:00 ` Liu, Shaoyun
0 siblings, 1 reply; 9+ messages in thread
From: Lazar, Lijo @ 2021-03-12 16:54 UTC (permalink / raw)
To: Liu, Shaoyun, amd-gfx, Quan, Evan, Zhang, Hawking
[-- Attachment #1.1: Type: text/plain, Size: 5850 bytes --]
[AMD Public Use]
Looks like this can be handled during post_init. It will be called as smu_post_init() happening during late_init part of smu block. You can check vangogh or navi examples on how to add your implementation.
Thanks,
Lijo
From: Liu, Shaoyun <Shaoyun.Liu@amd.com>
Sent: Friday, March 12, 2021 8:57 PM
To: Lazar, Lijo <Lijo.Lazar@amd.com>; amd-gfx@lists.freedesktop.org; Quan, Evan <Evan.Quan@amd.com>; Zhang, Hawking <Hawking.Zhang@amd.com>
Subject: RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
[AMD Public Use]
I don't like to add this set_light_sbr into ppt_funcs either , but please check current swsmu code structure , there is no asic specific swsmu late init function and there is no direct routine form amdgpu_smu.c to smu_v11_0.c either . It requires smu common code ->ppt_func -> smu_v11_0 for Arcturus specific function . So unless SMU and PPT have a major re-structure , set_light_sbr need to go through ppt_func for now, I think I better leave this re-structure task to SMU and PPT owner in the future .
Add SMU and PPT code owner Hawking and Quan for comments .
Regards
Shaoyun.liu
From: Lazar, Lijo <Lijo.Lazar@amd.com<mailto:Lijo.Lazar@amd.com>>
Sent: Friday, March 12, 2021 8:55 AM
To: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
[AMD Public Use]
We want to keep ppt_funcs minimal. Adding everything to ppt_funcs and keeping as NULL is not the right way. Please keep the code to arcturus.
Thanks,
Lijo
From: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Sent: Friday, March 12, 2021 7:21 PM
To: Lazar, Lijo <Lijo.Lazar@amd.com<mailto:Lijo.Lazar@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
Thanks for the comments. This light sbr solution could be applied to other asic as well. In swsmu code, It will check whether the function pointer set_light_sbr is valid before real call the function. So for other asics if the smu apply the same change, just add the ppt function pointer and we will have this support without further code change.
Thanks
Shaoyun.liu
________________________________
From: Lazar, Lijo <Lijo.Lazar@amd.com<mailto:Lijo.Lazar@amd.com>>
Sent: March 11, 2021 10:42 PM
To: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>>
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Subject: RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
[AMD Public Use]
We don't need this as a generic ppt_func. Reset functionalities are changing over programs and this could be valid only for Arcturus. Please move it to Arcturus swsmu late init.
Thanks,
Lijo
-----Original Message-----
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of shaoyunl
Sent: Thursday, March 11, 2021 10:46 PM
To: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Subject: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
This is to fix the commit dda9bbb26c7 where it only enable the light SMU on normal device init. This feature actually need to be enabled after ASIC been reset as well.
Signed-off-by: shaoyunl <shaoyun.liu@amd.com<mailto:shaoyun.liu@amd.com>>
Change-Id: Ie7ee02cd3ccdab3522aad9a02f681963e211ed44
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index cada3e77c7d5..fb775a9c0db1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2513,6 +2513,9 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
if (r)
DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
+ /* For XGMI + passthrough configuration , enable light SBR */
+ if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
+ smu_set_light_sbr(&adev->smu, true);
if (adev->gmc.xgmi.num_physical_nodes > 1) {
mutex_lock(&mgpu_info.mutex);
@@ -3615,10 +3618,6 @@ int amdgpu_device_init(struct amdgpu_device *adev,
if (amdgpu_device_cache_pci_state(adev->pdev))
pci_restore_state(pdev);
- /* Enable lightSBR on SMU in passthrough + xgmi configuration */
- if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
- smu_set_light_sbr(&adev->smu, true);
-
if (adev->gmc.xgmi.pending_reset)
queue_delayed_work(system_wq, &mgpu_info.delayed_reset_work,
msecs_to_jiffies(AMDGPU_RESUME_MS));
--
2.17.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=04%7C01%7Clijo.lazar%40amd.com%7Cc5aedb6c2d9d49d2fee408d8e4b15b5c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637510797685776785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7ndhtOVmyZcRMe3UQiGvF%2BprCdBVgo7f6IATXSbQNg4%3D&reserved=0
[-- Attachment #1.2: Type: text/html, Size: 12745 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 9+ messages in thread
* RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
2021-03-12 16:54 ` Lazar, Lijo
@ 2021-03-12 18:00 ` Liu, Shaoyun
0 siblings, 0 replies; 9+ messages in thread
From: Liu, Shaoyun @ 2021-03-12 18:00 UTC (permalink / raw)
To: Lazar, Lijo, amd-gfx, Quan, Evan, Zhang, Hawking
[-- Attachment #1.1: Type: text/plain, Size: 6957 bytes --]
[AMD Public Use]
Hi , Lijo
For now we only enable this light sbr feature for SMU on XGMI + passthrough for Arcturus since this is the use case our customer is required and we only verified in this configuration . I feel it's more reasonable to keep the logic of enable/disable in amdgpu side instead in SMU internally. We might need to support the SBR in bare-metal mode in the future . Also we plan to use this feature in future ASIC if the default HW reset method doesn't works stable. Your suggestion will need to move the enable/disable logic into SMU internally which valid our original design.
Regards
Shaoyun.liu
From: Lazar, Lijo <Lijo.Lazar@amd.com>
Sent: Friday, March 12, 2021 11:54 AM
To: Liu, Shaoyun <Shaoyun.Liu@amd.com>; amd-gfx@lists.freedesktop.org; Quan, Evan <Evan.Quan@amd.com>; Zhang, Hawking <Hawking.Zhang@amd.com>
Subject: RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
[AMD Public Use]
Looks like this can be handled during post_init. It will be called as smu_post_init() happening during late_init part of smu block. You can check vangogh or navi examples on how to add your implementation.
Thanks,
Lijo
From: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Sent: Friday, March 12, 2021 8:57 PM
To: Lazar, Lijo <Lijo.Lazar@amd.com<mailto:Lijo.Lazar@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>; Quan, Evan <Evan.Quan@amd.com<mailto:Evan.Quan@amd.com>>; Zhang, Hawking <Hawking.Zhang@amd.com<mailto:Hawking.Zhang@amd.com>>
Subject: RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
[AMD Public Use]
I don't like to add this set_light_sbr into ppt_funcs either , but please check current swsmu code structure , there is no asic specific swsmu late init function and there is no direct routine form amdgpu_smu.c to smu_v11_0.c either . It requires smu common code ->ppt_func -> smu_v11_0 for Arcturus specific function . So unless SMU and PPT have a major re-structure , set_light_sbr need to go through ppt_func for now, I think I better leave this re-structure task to SMU and PPT owner in the future .
Add SMU and PPT code owner Hawking and Quan for comments .
Regards
Shaoyun.liu
From: Lazar, Lijo <Lijo.Lazar@amd.com<mailto:Lijo.Lazar@amd.com>>
Sent: Friday, March 12, 2021 8:55 AM
To: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
[AMD Public Use]
We want to keep ppt_funcs minimal. Adding everything to ppt_funcs and keeping as NULL is not the right way. Please keep the code to arcturus.
Thanks,
Lijo
From: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Sent: Friday, March 12, 2021 7:21 PM
To: Lazar, Lijo <Lijo.Lazar@amd.com<mailto:Lijo.Lazar@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Subject: Re: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
Thanks for the comments. This light sbr solution could be applied to other asic as well. In swsmu code, It will check whether the function pointer set_light_sbr is valid before real call the function. So for other asics if the smu apply the same change, just add the ppt function pointer and we will have this support without further code change.
Thanks
Shaoyun.liu
________________________________
From: Lazar, Lijo <Lijo.Lazar@amd.com<mailto:Lijo.Lazar@amd.com>>
Sent: March 11, 2021 10:42 PM
To: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>; amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org> <amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>>
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Subject: RE: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
[AMD Public Use]
We don't need this as a generic ppt_func. Reset functionalities are changing over programs and this could be valid only for Arcturus. Please move it to Arcturus swsmu late init.
Thanks,
Lijo
-----Original Message-----
From: amd-gfx <amd-gfx-bounces@lists.freedesktop.org<mailto:amd-gfx-bounces@lists.freedesktop.org>> On Behalf Of shaoyunl
Sent: Thursday, March 11, 2021 10:46 PM
To: amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
Cc: Liu, Shaoyun <Shaoyun.Liu@amd.com<mailto:Shaoyun.Liu@amd.com>>
Subject: [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
This is to fix the commit dda9bbb26c7 where it only enable the light SMU on normal device init. This feature actually need to be enabled after ASIC been reset as well.
Signed-off-by: shaoyunl <shaoyun.liu@amd.com<mailto:shaoyun.liu@amd.com>>
Change-Id: Ie7ee02cd3ccdab3522aad9a02f681963e211ed44
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index cada3e77c7d5..fb775a9c0db1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2513,6 +2513,9 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
if (r)
DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
+ /* For XGMI + passthrough configuration , enable light SBR */
+ if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
+ smu_set_light_sbr(&adev->smu, true);
if (adev->gmc.xgmi.num_physical_nodes > 1) {
mutex_lock(&mgpu_info.mutex);
@@ -3615,10 +3618,6 @@ int amdgpu_device_init(struct amdgpu_device *adev,
if (amdgpu_device_cache_pci_state(adev->pdev))
pci_restore_state(pdev);
- /* Enable lightSBR on SMU in passthrough + xgmi configuration */
- if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
- smu_set_light_sbr(&adev->smu, true);
-
if (adev->gmc.xgmi.pending_reset)
queue_delayed_work(system_wq, &mgpu_info.delayed_reset_work,
msecs_to_jiffies(AMDGPU_RESUME_MS));
--
2.17.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org<mailto:amd-gfx@lists.freedesktop.org>
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=04%7C01%7Clijo.lazar%40amd.com%7Cc5aedb6c2d9d49d2fee408d8e4b15b5c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637510797685776785%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7ndhtOVmyZcRMe3UQiGvF%2BprCdBVgo7f6IATXSbQNg4%3D&reserved=0
[-- Attachment #1.2: Type: text/html, Size: 14869 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration
@ 2021-03-12 22:02 shaoyunl
0 siblings, 0 replies; 9+ messages in thread
From: shaoyunl @ 2021-03-12 22:02 UTC (permalink / raw)
To: amd-gfx; +Cc: shaoyunl
This is to fix the commit dda9bbb26c7 where it only enable the light SMU
on normal device init. This feature actually need to be enabled after ASIC
been reset as well.
Signed-off-by: shaoyunl <shaoyun.liu@amd.com>
Change-Id: Ie7ee02cd3ccdab3522aad9a02f681963e211ed44
---
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index cada3e77c7d5..34472c9d73f1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2513,6 +2513,11 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
if (r)
DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
+ /* For XGMI + passthrough configuration on arcturus, enable light SBR */
+ if (adev->asic_type == CHIP_ARCTURUS &&
+ amdgpu_passthrough(adev) &&
+ adev->gmc.xgmi.num_physical_nodes > 1)
+ smu_set_light_sbr(&adev->smu, true);
if (adev->gmc.xgmi.num_physical_nodes > 1) {
mutex_lock(&mgpu_info.mutex);
@@ -3615,10 +3620,6 @@ int amdgpu_device_init(struct amdgpu_device *adev,
if (amdgpu_device_cache_pci_state(adev->pdev))
pci_restore_state(pdev);
- /* Enable lightSBR on SMU in passthrough + xgmi configuration */
- if (amdgpu_passthrough(adev) && adev->gmc.xgmi.num_physical_nodes > 1)
- smu_set_light_sbr(&adev->smu, true);
-
if (adev->gmc.xgmi.pending_reset)
queue_delayed_work(system_wq, &mgpu_info.delayed_reset_work,
msecs_to_jiffies(AMDGPU_RESUME_MS));
--
2.17.1
_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-03-12 22:02 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-11 17:15 [PATCH] drm/amdgpu: Enable light SBR in XGMI+passthrough configuration shaoyunl
2021-03-12 0:47 ` Liu, Shaoyun
2021-03-12 3:42 ` Lazar, Lijo
2021-03-12 13:51 ` Liu, Shaoyun
2021-03-12 13:54 ` Lazar, Lijo
2021-03-12 15:27 ` Liu, Shaoyun
2021-03-12 16:54 ` Lazar, Lijo
2021-03-12 18:00 ` Liu, Shaoyun
2021-03-12 22:02 shaoyunl
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.