All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 1/3] drm/amdgpu: change pstate only after all XGMI device initialized
@ 2019-10-31  7:22 ` Quan, Evan
  0 siblings, 0 replies; 8+ messages in thread
From: Quan, Evan @ 2019-10-31  7:22 UTC (permalink / raw)
  To: amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW
  Cc: Deucher, Alexander, Kim, Jonathan, Quan, Evan

Pstate settings should be performed after all device of the
XGMI setup get initialized.

Change-Id: I5c4b3f79fbd60a5ccfb4dc6f94d9e1db6faec694
Signed-off-by: Evan Quan <evan.quan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 08e9d85d22f4..93720c4790f8 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2081,9 +2081,6 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
 	if (r)
 		DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
 
-	/* set to low pstate by default */
-	amdgpu_xgmi_set_pstate(adev, 0);
-
 	return 0;
 }
 
@@ -2195,6 +2192,18 @@ static void amdgpu_device_delayed_init_work_handler(struct work_struct *work)
 	r = amdgpu_ib_ring_tests(adev);
 	if (r)
 		DRM_ERROR("ib ring test failed (%d).\n", r);
+
+	/*
+	 * set to low pstate by default
+	 * This should be performed after all devices from
+	 * XGMI finish their initializations. Thus it's moved
+	 * to here.
+	 * The time delay is 2S. TODO: confirm whether that
+	 * is enough for all possible XGMI setups.
+	 */
+	r = amdgpu_xgmi_set_pstate(adev, 0);
+	if (r)
+		DRM_ERROR("pstate setting failed (%d).\n", r);
 }
 
 static void amdgpu_device_delay_enable_gfx_off(struct work_struct *work)
-- 
2.23.0

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [PATCH 1/3] drm/amdgpu: change pstate only after all XGMI device initialized
@ 2019-10-31  7:22 ` Quan, Evan
  0 siblings, 0 replies; 8+ messages in thread
From: Quan, Evan @ 2019-10-31  7:22 UTC (permalink / raw)
  To: amd-gfx; +Cc: Deucher, Alexander, Kim, Jonathan, Quan, Evan

Pstate settings should be performed after all device of the
XGMI setup get initialized.

Change-Id: I5c4b3f79fbd60a5ccfb4dc6f94d9e1db6faec694
Signed-off-by: Evan Quan <evan.quan@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index 08e9d85d22f4..93720c4790f8 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2081,9 +2081,6 @@ static int amdgpu_device_ip_late_init(struct amdgpu_device *adev)
 	if (r)
 		DRM_ERROR("enable mgpu fan boost failed (%d).\n", r);
 
-	/* set to low pstate by default */
-	amdgpu_xgmi_set_pstate(adev, 0);
-
 	return 0;
 }
 
@@ -2195,6 +2192,18 @@ static void amdgpu_device_delayed_init_work_handler(struct work_struct *work)
 	r = amdgpu_ib_ring_tests(adev);
 	if (r)
 		DRM_ERROR("ib ring test failed (%d).\n", r);
+
+	/*
+	 * set to low pstate by default
+	 * This should be performed after all devices from
+	 * XGMI finish their initializations. Thus it's moved
+	 * to here.
+	 * The time delay is 2S. TODO: confirm whether that
+	 * is enough for all possible XGMI setups.
+	 */
+	r = amdgpu_xgmi_set_pstate(adev, 0);
+	if (r)
+		DRM_ERROR("pstate setting failed (%d).\n", r);
 }
 
 static void amdgpu_device_delay_enable_gfx_off(struct work_struct *work)
-- 
2.23.0

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [PATCH 2/3] drm/amd/powerplay: update is_sw_smu_xgmi check
@ 2019-10-31  7:22     ` Quan, Evan
  0 siblings, 0 replies; 8+ messages in thread
From: Quan, Evan @ 2019-10-31  7:22 UTC (permalink / raw)
  To: amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW
  Cc: Deucher, Alexander, Kim, Jonathan, Quan, Evan

Add check for is_sw_smu routine and drop check
for amdgpu_dpm which seems non-sense.

Change-Id: I2b694a6255a76d35305fc64ca39625730e3463db
Signed-off-by: Evan Quan <evan.quan@amd.com>
---
 drivers/gpu/drm/amd/powerplay/amdgpu_smu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/powerplay/amdgpu_smu.c b/drivers/gpu/drm/amd/powerplay/amdgpu_smu.c
index 3ce01e1994fc..dc2130a3168c 100644
--- a/drivers/gpu/drm/amd/powerplay/amdgpu_smu.c
+++ b/drivers/gpu/drm/amd/powerplay/amdgpu_smu.c
@@ -526,7 +526,7 @@ bool is_support_sw_smu(struct amdgpu_device *adev)
 
 bool is_support_sw_smu_xgmi(struct amdgpu_device *adev)
 {
-	if (amdgpu_dpm != 1)
+	if (!is_support_sw_smu(adev))
 		return false;
 
 	if (adev->asic_type == CHIP_VEGA20)
-- 
2.23.0

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [PATCH 2/3] drm/amd/powerplay: update is_sw_smu_xgmi check
@ 2019-10-31  7:22     ` Quan, Evan
  0 siblings, 0 replies; 8+ messages in thread
From: Quan, Evan @ 2019-10-31  7:22 UTC (permalink / raw)
  To: amd-gfx; +Cc: Deucher, Alexander, Kim, Jonathan, Quan, Evan

Add check for is_sw_smu routine and drop check
for amdgpu_dpm which seems non-sense.

Change-Id: I2b694a6255a76d35305fc64ca39625730e3463db
Signed-off-by: Evan Quan <evan.quan@amd.com>
---
 drivers/gpu/drm/amd/powerplay/amdgpu_smu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/amd/powerplay/amdgpu_smu.c b/drivers/gpu/drm/amd/powerplay/amdgpu_smu.c
index 3ce01e1994fc..dc2130a3168c 100644
--- a/drivers/gpu/drm/amd/powerplay/amdgpu_smu.c
+++ b/drivers/gpu/drm/amd/powerplay/amdgpu_smu.c
@@ -526,7 +526,7 @@ bool is_support_sw_smu(struct amdgpu_device *adev)
 
 bool is_support_sw_smu_xgmi(struct amdgpu_device *adev)
 {
-	if (amdgpu_dpm != 1)
+	if (!is_support_sw_smu(adev))
 		return false;
 
 	if (adev->asic_type == CHIP_VEGA20)
-- 
2.23.0

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [PATCH 3/3] drm/amd/powerplay: support xgmi pstate setting on powerplay routine V2
@ 2019-10-31  7:22     ` Quan, Evan
  0 siblings, 0 replies; 8+ messages in thread
From: Quan, Evan @ 2019-10-31  7:22 UTC (permalink / raw)
  To: amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW
  Cc: Deucher, Alexander, Kim, Jonathan, Quan, Evan

Add xgmi pstate setting on powerplay routine.

Change-Id: If1a49aa14c16f133e43ac1298c6b14eaeb44d79d
Signed-off-by: Evan Quan <evan.quan@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

V2: split the change of is_support_sw_smu_xgmi into a separate patch
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c       |  5 +++++
 drivers/gpu/drm/amd/include/kgd_pp_interface.h |  4 ++++
 drivers/gpu/drm/amd/powerplay/amd_powerplay.c  | 18 ++++++++++++++++++
 .../gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c | 15 +++++++++++++++
 drivers/gpu/drm/amd/powerplay/inc/hwmgr.h      |  1 +
 drivers/gpu/drm/amd/powerplay/smu_v11_0.c      |  5 +----
 6 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
index 00371713c671..167d9fbd2c4f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
@@ -285,6 +285,11 @@ int amdgpu_xgmi_set_pstate(struct amdgpu_device *adev, int pstate)
 
 	if (is_support_sw_smu_xgmi(adev))
 		ret = smu_set_xgmi_pstate(&adev->smu, pstate);
+	else if (adev->powerplay.pp_funcs &&
+		 adev->powerplay.pp_funcs->set_xgmi_pstate)
+		ret = adev->powerplay.pp_funcs->set_xgmi_pstate(adev->powerplay.pp_handle,
+								pstate);
+
 	if (ret)
 		dev_err(adev->dev,
 			"XGMI: Set pstate failure on device %llx, hive %llx, ret %d",
diff --git a/drivers/gpu/drm/amd/include/kgd_pp_interface.h b/drivers/gpu/drm/amd/include/kgd_pp_interface.h
index 5902f80d1fce..a7f92d0b3a90 100644
--- a/drivers/gpu/drm/amd/include/kgd_pp_interface.h
+++ b/drivers/gpu/drm/amd/include/kgd_pp_interface.h
@@ -220,6 +220,9 @@ enum pp_df_cstate {
 		((group) << PP_GROUP_SHIFT | (block) << PP_BLOCK_SHIFT | \
 		(support) << PP_STATE_SUPPORT_SHIFT | (state) << PP_STATE_SHIFT)
 
+#define XGMI_MODE_PSTATE_D3 0
+#define XGMI_MODE_PSTATE_D0 1
+
 struct seq_file;
 enum amd_pp_clock_type;
 struct amd_pp_simple_clock_info;
@@ -318,6 +321,7 @@ struct amd_pm_funcs {
 	int (*set_ppfeature_status)(void *handle, uint64_t ppfeature_masks);
 	int (*asic_reset_mode_2)(void *handle);
 	int (*set_df_cstate)(void *handle, enum pp_df_cstate state);
+	int (*set_xgmi_pstate)(void *handle, uint32_t pstate);
 };
 
 #endif
diff --git a/drivers/gpu/drm/amd/powerplay/amd_powerplay.c b/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
index f4ff15378e61..031447675203 100644
--- a/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
+++ b/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
@@ -1566,6 +1566,23 @@ static int pp_set_df_cstate(void *handle, enum pp_df_cstate state)
 	return 0;
 }
 
+static int pp_set_xgmi_pstate(void *handle, uint32_t pstate)
+{
+	struct pp_hwmgr *hwmgr = handle;
+
+	if (!hwmgr)
+		return -EINVAL;
+
+	if (!hwmgr->pm_en || !hwmgr->hwmgr_func->set_xgmi_pstate)
+		return 0;
+
+	mutex_lock(&hwmgr->smu_lock);
+	hwmgr->hwmgr_func->set_xgmi_pstate(hwmgr, pstate);
+	mutex_unlock(&hwmgr->smu_lock);
+
+	return 0;
+}
+
 static const struct amd_pm_funcs pp_dpm_funcs = {
 	.load_firmware = pp_dpm_load_fw,
 	.wait_for_fw_loading_complete = pp_dpm_fw_loading_complete,
@@ -1625,4 +1642,5 @@ static const struct amd_pm_funcs pp_dpm_funcs = {
 	.asic_reset_mode_2 = pp_asic_reset_mode_2,
 	.smu_i2c_bus_access = pp_smu_i2c_bus_access,
 	.set_df_cstate = pp_set_df_cstate,
+	.set_xgmi_pstate = pp_set_xgmi_pstate,
 };
diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c b/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
index 9295bd90b792..5bcf0d684151 100644
--- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
+++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
@@ -4176,6 +4176,20 @@ static int vega20_set_df_cstate(struct pp_hwmgr *hwmgr,
 	return ret;
 }
 
+static int vega20_set_xgmi_pstate(struct pp_hwmgr *hwmgr,
+				  uint32_t pstate)
+{
+	int ret;
+
+	ret = smum_send_msg_to_smc_with_parameter(hwmgr,
+						  PPSMC_MSG_SetXgmiMode,
+						  pstate ? XGMI_MODE_PSTATE_D0 : XGMI_MODE_PSTATE_D3);
+	if (ret)
+		pr_err("SetXgmiPstate failed!\n");
+
+	return ret;
+}
+
 static const struct pp_hwmgr_func vega20_hwmgr_funcs = {
 	/* init/fini related */
 	.backend_init = vega20_hwmgr_backend_init,
@@ -4245,6 +4259,7 @@ static const struct pp_hwmgr_func vega20_hwmgr_funcs = {
 	.set_mp1_state = vega20_set_mp1_state,
 	.smu_i2c_bus_access = vega20_smu_i2c_bus_access,
 	.set_df_cstate = vega20_set_df_cstate,
+	.set_xgmi_pstate = vega20_set_xgmi_pstate,
 };
 
 int vega20_hwmgr_init(struct pp_hwmgr *hwmgr)
diff --git a/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h b/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
index bd8c922dfd3e..40403bc76f1b 100644
--- a/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
+++ b/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
@@ -356,6 +356,7 @@ struct pp_hwmgr_func {
 	int (*asic_reset)(struct pp_hwmgr *hwmgr, enum SMU_ASIC_RESET_MODE mode);
 	int (*smu_i2c_bus_access)(struct pp_hwmgr *hwmgr, bool aquire);
 	int (*set_df_cstate)(struct pp_hwmgr *hwmgr, enum pp_df_cstate state);
+	int (*set_xgmi_pstate)(struct pp_hwmgr *hwmgr, uint32_t pstate);
 };
 
 struct pp_table_func {
diff --git a/drivers/gpu/drm/amd/powerplay/smu_v11_0.c b/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
index 7e882999abad..5877857760be 100644
--- a/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
+++ b/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
@@ -1463,16 +1463,13 @@ int smu_v11_0_set_fan_speed_rpm(struct smu_context *smu,
 	return ret;
 }
 
-#define XGMI_STATE_D0 1
-#define XGMI_STATE_D3 0
-
 int smu_v11_0_set_xgmi_pstate(struct smu_context *smu,
 				     uint32_t pstate)
 {
 	int ret = 0;
 	ret = smu_send_smc_msg_with_param(smu,
 					  SMU_MSG_SetXgmiMode,
-					  pstate ? XGMI_STATE_D0 : XGMI_STATE_D3);
+					  pstate ? XGMI_MODE_PSTATE_D0 : XGMI_MODE_PSTATE_D3);
 	return ret;
 }
 
-- 
2.23.0

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [PATCH 3/3] drm/amd/powerplay: support xgmi pstate setting on powerplay routine V2
@ 2019-10-31  7:22     ` Quan, Evan
  0 siblings, 0 replies; 8+ messages in thread
From: Quan, Evan @ 2019-10-31  7:22 UTC (permalink / raw)
  To: amd-gfx; +Cc: Deucher, Alexander, Kim, Jonathan, Quan, Evan

Add xgmi pstate setting on powerplay routine.

Change-Id: If1a49aa14c16f133e43ac1298c6b14eaeb44d79d
Signed-off-by: Evan Quan <evan.quan@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

V2: split the change of is_support_sw_smu_xgmi into a separate patch
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c       |  5 +++++
 drivers/gpu/drm/amd/include/kgd_pp_interface.h |  4 ++++
 drivers/gpu/drm/amd/powerplay/amd_powerplay.c  | 18 ++++++++++++++++++
 .../gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c | 15 +++++++++++++++
 drivers/gpu/drm/amd/powerplay/inc/hwmgr.h      |  1 +
 drivers/gpu/drm/amd/powerplay/smu_v11_0.c      |  5 +----
 6 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
index 00371713c671..167d9fbd2c4f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
@@ -285,6 +285,11 @@ int amdgpu_xgmi_set_pstate(struct amdgpu_device *adev, int pstate)
 
 	if (is_support_sw_smu_xgmi(adev))
 		ret = smu_set_xgmi_pstate(&adev->smu, pstate);
+	else if (adev->powerplay.pp_funcs &&
+		 adev->powerplay.pp_funcs->set_xgmi_pstate)
+		ret = adev->powerplay.pp_funcs->set_xgmi_pstate(adev->powerplay.pp_handle,
+								pstate);
+
 	if (ret)
 		dev_err(adev->dev,
 			"XGMI: Set pstate failure on device %llx, hive %llx, ret %d",
diff --git a/drivers/gpu/drm/amd/include/kgd_pp_interface.h b/drivers/gpu/drm/amd/include/kgd_pp_interface.h
index 5902f80d1fce..a7f92d0b3a90 100644
--- a/drivers/gpu/drm/amd/include/kgd_pp_interface.h
+++ b/drivers/gpu/drm/amd/include/kgd_pp_interface.h
@@ -220,6 +220,9 @@ enum pp_df_cstate {
 		((group) << PP_GROUP_SHIFT | (block) << PP_BLOCK_SHIFT | \
 		(support) << PP_STATE_SUPPORT_SHIFT | (state) << PP_STATE_SHIFT)
 
+#define XGMI_MODE_PSTATE_D3 0
+#define XGMI_MODE_PSTATE_D0 1
+
 struct seq_file;
 enum amd_pp_clock_type;
 struct amd_pp_simple_clock_info;
@@ -318,6 +321,7 @@ struct amd_pm_funcs {
 	int (*set_ppfeature_status)(void *handle, uint64_t ppfeature_masks);
 	int (*asic_reset_mode_2)(void *handle);
 	int (*set_df_cstate)(void *handle, enum pp_df_cstate state);
+	int (*set_xgmi_pstate)(void *handle, uint32_t pstate);
 };
 
 #endif
diff --git a/drivers/gpu/drm/amd/powerplay/amd_powerplay.c b/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
index f4ff15378e61..031447675203 100644
--- a/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
+++ b/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
@@ -1566,6 +1566,23 @@ static int pp_set_df_cstate(void *handle, enum pp_df_cstate state)
 	return 0;
 }
 
+static int pp_set_xgmi_pstate(void *handle, uint32_t pstate)
+{
+	struct pp_hwmgr *hwmgr = handle;
+
+	if (!hwmgr)
+		return -EINVAL;
+
+	if (!hwmgr->pm_en || !hwmgr->hwmgr_func->set_xgmi_pstate)
+		return 0;
+
+	mutex_lock(&hwmgr->smu_lock);
+	hwmgr->hwmgr_func->set_xgmi_pstate(hwmgr, pstate);
+	mutex_unlock(&hwmgr->smu_lock);
+
+	return 0;
+}
+
 static const struct amd_pm_funcs pp_dpm_funcs = {
 	.load_firmware = pp_dpm_load_fw,
 	.wait_for_fw_loading_complete = pp_dpm_fw_loading_complete,
@@ -1625,4 +1642,5 @@ static const struct amd_pm_funcs pp_dpm_funcs = {
 	.asic_reset_mode_2 = pp_asic_reset_mode_2,
 	.smu_i2c_bus_access = pp_smu_i2c_bus_access,
 	.set_df_cstate = pp_set_df_cstate,
+	.set_xgmi_pstate = pp_set_xgmi_pstate,
 };
diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c b/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
index 9295bd90b792..5bcf0d684151 100644
--- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
+++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
@@ -4176,6 +4176,20 @@ static int vega20_set_df_cstate(struct pp_hwmgr *hwmgr,
 	return ret;
 }
 
+static int vega20_set_xgmi_pstate(struct pp_hwmgr *hwmgr,
+				  uint32_t pstate)
+{
+	int ret;
+
+	ret = smum_send_msg_to_smc_with_parameter(hwmgr,
+						  PPSMC_MSG_SetXgmiMode,
+						  pstate ? XGMI_MODE_PSTATE_D0 : XGMI_MODE_PSTATE_D3);
+	if (ret)
+		pr_err("SetXgmiPstate failed!\n");
+
+	return ret;
+}
+
 static const struct pp_hwmgr_func vega20_hwmgr_funcs = {
 	/* init/fini related */
 	.backend_init = vega20_hwmgr_backend_init,
@@ -4245,6 +4259,7 @@ static const struct pp_hwmgr_func vega20_hwmgr_funcs = {
 	.set_mp1_state = vega20_set_mp1_state,
 	.smu_i2c_bus_access = vega20_smu_i2c_bus_access,
 	.set_df_cstate = vega20_set_df_cstate,
+	.set_xgmi_pstate = vega20_set_xgmi_pstate,
 };
 
 int vega20_hwmgr_init(struct pp_hwmgr *hwmgr)
diff --git a/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h b/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
index bd8c922dfd3e..40403bc76f1b 100644
--- a/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
+++ b/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
@@ -356,6 +356,7 @@ struct pp_hwmgr_func {
 	int (*asic_reset)(struct pp_hwmgr *hwmgr, enum SMU_ASIC_RESET_MODE mode);
 	int (*smu_i2c_bus_access)(struct pp_hwmgr *hwmgr, bool aquire);
 	int (*set_df_cstate)(struct pp_hwmgr *hwmgr, enum pp_df_cstate state);
+	int (*set_xgmi_pstate)(struct pp_hwmgr *hwmgr, uint32_t pstate);
 };
 
 struct pp_table_func {
diff --git a/drivers/gpu/drm/amd/powerplay/smu_v11_0.c b/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
index 7e882999abad..5877857760be 100644
--- a/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
+++ b/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
@@ -1463,16 +1463,13 @@ int smu_v11_0_set_fan_speed_rpm(struct smu_context *smu,
 	return ret;
 }
 
-#define XGMI_STATE_D0 1
-#define XGMI_STATE_D3 0
-
 int smu_v11_0_set_xgmi_pstate(struct smu_context *smu,
 				     uint32_t pstate)
 {
 	int ret = 0;
 	ret = smu_send_smc_msg_with_param(smu,
 					  SMU_MSG_SetXgmiMode,
-					  pstate ? XGMI_STATE_D0 : XGMI_STATE_D3);
+					  pstate ? XGMI_MODE_PSTATE_D0 : XGMI_MODE_PSTATE_D3);
 	return ret;
 }
 
-- 
2.23.0

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* Re: [PATCH 3/3] drm/amd/powerplay: support xgmi pstate setting on powerplay routine V2
@ 2019-10-31 14:23         ` Deucher, Alexander
  0 siblings, 0 replies; 8+ messages in thread
From: Deucher, Alexander @ 2019-10-31 14:23 UTC (permalink / raw)
  To: Quan, Evan, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW; +Cc: Kim, Jonathan


[-- Attachment #1.1: Type: text/plain, Size: 7274 bytes --]

Series is:
Reviewed-by: Alex Deucher <alexander.deucher-5C7GfCeVMHo@public.gmane.org>
________________________________
From: Quan, Evan <Evan.Quan-5C7GfCeVMHo@public.gmane.org>
Sent: Thursday, October 31, 2019 3:22 AM
To: amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org <amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org>
Cc: Kim, Jonathan <Jonathan.Kim-5C7GfCeVMHo@public.gmane.org>; Deucher, Alexander <Alexander.Deucher-5C7GfCeVMHo@public.gmane.org>; Quan, Evan <Evan.Quan-5C7GfCeVMHo@public.gmane.org>
Subject: [PATCH 3/3] drm/amd/powerplay: support xgmi pstate setting on powerplay routine V2

Add xgmi pstate setting on powerplay routine.

Change-Id: If1a49aa14c16f133e43ac1298c6b14eaeb44d79d
Signed-off-by: Evan Quan <evan.quan-5C7GfCeVMHo@public.gmane.org>
Reviewed-by: Alex Deucher <alexander.deucher-5C7GfCeVMHo@public.gmane.org>

V2: split the change of is_support_sw_smu_xgmi into a separate patch
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c       |  5 +++++
 drivers/gpu/drm/amd/include/kgd_pp_interface.h |  4 ++++
 drivers/gpu/drm/amd/powerplay/amd_powerplay.c  | 18 ++++++++++++++++++
 .../gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c | 15 +++++++++++++++
 drivers/gpu/drm/amd/powerplay/inc/hwmgr.h      |  1 +
 drivers/gpu/drm/amd/powerplay/smu_v11_0.c      |  5 +----
 6 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
index 00371713c671..167d9fbd2c4f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
@@ -285,6 +285,11 @@ int amdgpu_xgmi_set_pstate(struct amdgpu_device *adev, int pstate)

         if (is_support_sw_smu_xgmi(adev))
                 ret = smu_set_xgmi_pstate(&adev->smu, pstate);
+       else if (adev->powerplay.pp_funcs &&
+                adev->powerplay.pp_funcs->set_xgmi_pstate)
+               ret = adev->powerplay.pp_funcs->set_xgmi_pstate(adev->powerplay.pp_handle,
+                                                               pstate);
+
         if (ret)
                 dev_err(adev->dev,
                         "XGMI: Set pstate failure on device %llx, hive %llx, ret %d",
diff --git a/drivers/gpu/drm/amd/include/kgd_pp_interface.h b/drivers/gpu/drm/amd/include/kgd_pp_interface.h
index 5902f80d1fce..a7f92d0b3a90 100644
--- a/drivers/gpu/drm/amd/include/kgd_pp_interface.h
+++ b/drivers/gpu/drm/amd/include/kgd_pp_interface.h
@@ -220,6 +220,9 @@ enum pp_df_cstate {
                 ((group) << PP_GROUP_SHIFT | (block) << PP_BLOCK_SHIFT | \
                 (support) << PP_STATE_SUPPORT_SHIFT | (state) << PP_STATE_SHIFT)

+#define XGMI_MODE_PSTATE_D3 0
+#define XGMI_MODE_PSTATE_D0 1
+
 struct seq_file;
 enum amd_pp_clock_type;
 struct amd_pp_simple_clock_info;
@@ -318,6 +321,7 @@ struct amd_pm_funcs {
         int (*set_ppfeature_status)(void *handle, uint64_t ppfeature_masks);
         int (*asic_reset_mode_2)(void *handle);
         int (*set_df_cstate)(void *handle, enum pp_df_cstate state);
+       int (*set_xgmi_pstate)(void *handle, uint32_t pstate);
 };

 #endif
diff --git a/drivers/gpu/drm/amd/powerplay/amd_powerplay.c b/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
index f4ff15378e61..031447675203 100644
--- a/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
+++ b/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
@@ -1566,6 +1566,23 @@ static int pp_set_df_cstate(void *handle, enum pp_df_cstate state)
         return 0;
 }

+static int pp_set_xgmi_pstate(void *handle, uint32_t pstate)
+{
+       struct pp_hwmgr *hwmgr = handle;
+
+       if (!hwmgr)
+               return -EINVAL;
+
+       if (!hwmgr->pm_en || !hwmgr->hwmgr_func->set_xgmi_pstate)
+               return 0;
+
+       mutex_lock(&hwmgr->smu_lock);
+       hwmgr->hwmgr_func->set_xgmi_pstate(hwmgr, pstate);
+       mutex_unlock(&hwmgr->smu_lock);
+
+       return 0;
+}
+
 static const struct amd_pm_funcs pp_dpm_funcs = {
         .load_firmware = pp_dpm_load_fw,
         .wait_for_fw_loading_complete = pp_dpm_fw_loading_complete,
@@ -1625,4 +1642,5 @@ static const struct amd_pm_funcs pp_dpm_funcs = {
         .asic_reset_mode_2 = pp_asic_reset_mode_2,
         .smu_i2c_bus_access = pp_smu_i2c_bus_access,
         .set_df_cstate = pp_set_df_cstate,
+       .set_xgmi_pstate = pp_set_xgmi_pstate,
 };
diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c b/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
index 9295bd90b792..5bcf0d684151 100644
--- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
+++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
@@ -4176,6 +4176,20 @@ static int vega20_set_df_cstate(struct pp_hwmgr *hwmgr,
         return ret;
 }

+static int vega20_set_xgmi_pstate(struct pp_hwmgr *hwmgr,
+                                 uint32_t pstate)
+{
+       int ret;
+
+       ret = smum_send_msg_to_smc_with_parameter(hwmgr,
+                                                 PPSMC_MSG_SetXgmiMode,
+                                                 pstate ? XGMI_MODE_PSTATE_D0 : XGMI_MODE_PSTATE_D3);
+       if (ret)
+               pr_err("SetXgmiPstate failed!\n");
+
+       return ret;
+}
+
 static const struct pp_hwmgr_func vega20_hwmgr_funcs = {
         /* init/fini related */
         .backend_init = vega20_hwmgr_backend_init,
@@ -4245,6 +4259,7 @@ static const struct pp_hwmgr_func vega20_hwmgr_funcs = {
         .set_mp1_state = vega20_set_mp1_state,
         .smu_i2c_bus_access = vega20_smu_i2c_bus_access,
         .set_df_cstate = vega20_set_df_cstate,
+       .set_xgmi_pstate = vega20_set_xgmi_pstate,
 };

 int vega20_hwmgr_init(struct pp_hwmgr *hwmgr)
diff --git a/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h b/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
index bd8c922dfd3e..40403bc76f1b 100644
--- a/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
+++ b/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
@@ -356,6 +356,7 @@ struct pp_hwmgr_func {
         int (*asic_reset)(struct pp_hwmgr *hwmgr, enum SMU_ASIC_RESET_MODE mode);
         int (*smu_i2c_bus_access)(struct pp_hwmgr *hwmgr, bool aquire);
         int (*set_df_cstate)(struct pp_hwmgr *hwmgr, enum pp_df_cstate state);
+       int (*set_xgmi_pstate)(struct pp_hwmgr *hwmgr, uint32_t pstate);
 };

 struct pp_table_func {
diff --git a/drivers/gpu/drm/amd/powerplay/smu_v11_0.c b/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
index 7e882999abad..5877857760be 100644
--- a/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
+++ b/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
@@ -1463,16 +1463,13 @@ int smu_v11_0_set_fan_speed_rpm(struct smu_context *smu,
         return ret;
 }

-#define XGMI_STATE_D0 1
-#define XGMI_STATE_D3 0
-
 int smu_v11_0_set_xgmi_pstate(struct smu_context *smu,
                                      uint32_t pstate)
 {
         int ret = 0;
         ret = smu_send_smc_msg_with_param(smu,
                                           SMU_MSG_SetXgmiMode,
-                                         pstate ? XGMI_STATE_D0 : XGMI_STATE_D3);
+                                         pstate ? XGMI_MODE_PSTATE_D0 : XGMI_MODE_PSTATE_D3);
         return ret;
 }

--
2.23.0


[-- Attachment #1.2: Type: text/html, Size: 13614 bytes --]

[-- Attachment #2: Type: text/plain, Size: 153 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* Re: [PATCH 3/3] drm/amd/powerplay: support xgmi pstate setting on powerplay routine V2
@ 2019-10-31 14:23         ` Deucher, Alexander
  0 siblings, 0 replies; 8+ messages in thread
From: Deucher, Alexander @ 2019-10-31 14:23 UTC (permalink / raw)
  To: Quan, Evan, amd-gfx; +Cc: Kim, Jonathan


[-- Attachment #1.1: Type: text/plain, Size: 7071 bytes --]

Series is:
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
________________________________
From: Quan, Evan <Evan.Quan@amd.com>
Sent: Thursday, October 31, 2019 3:22 AM
To: amd-gfx@lists.freedesktop.org <amd-gfx@lists.freedesktop.org>
Cc: Kim, Jonathan <Jonathan.Kim@amd.com>; Deucher, Alexander <Alexander.Deucher@amd.com>; Quan, Evan <Evan.Quan@amd.com>
Subject: [PATCH 3/3] drm/amd/powerplay: support xgmi pstate setting on powerplay routine V2

Add xgmi pstate setting on powerplay routine.

Change-Id: If1a49aa14c16f133e43ac1298c6b14eaeb44d79d
Signed-off-by: Evan Quan <evan.quan@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>

V2: split the change of is_support_sw_smu_xgmi into a separate patch
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c       |  5 +++++
 drivers/gpu/drm/amd/include/kgd_pp_interface.h |  4 ++++
 drivers/gpu/drm/amd/powerplay/amd_powerplay.c  | 18 ++++++++++++++++++
 .../gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c | 15 +++++++++++++++
 drivers/gpu/drm/amd/powerplay/inc/hwmgr.h      |  1 +
 drivers/gpu/drm/amd/powerplay/smu_v11_0.c      |  5 +----
 6 files changed, 44 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
index 00371713c671..167d9fbd2c4f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c
@@ -285,6 +285,11 @@ int amdgpu_xgmi_set_pstate(struct amdgpu_device *adev, int pstate)

         if (is_support_sw_smu_xgmi(adev))
                 ret = smu_set_xgmi_pstate(&adev->smu, pstate);
+       else if (adev->powerplay.pp_funcs &&
+                adev->powerplay.pp_funcs->set_xgmi_pstate)
+               ret = adev->powerplay.pp_funcs->set_xgmi_pstate(adev->powerplay.pp_handle,
+                                                               pstate);
+
         if (ret)
                 dev_err(adev->dev,
                         "XGMI: Set pstate failure on device %llx, hive %llx, ret %d",
diff --git a/drivers/gpu/drm/amd/include/kgd_pp_interface.h b/drivers/gpu/drm/amd/include/kgd_pp_interface.h
index 5902f80d1fce..a7f92d0b3a90 100644
--- a/drivers/gpu/drm/amd/include/kgd_pp_interface.h
+++ b/drivers/gpu/drm/amd/include/kgd_pp_interface.h
@@ -220,6 +220,9 @@ enum pp_df_cstate {
                 ((group) << PP_GROUP_SHIFT | (block) << PP_BLOCK_SHIFT | \
                 (support) << PP_STATE_SUPPORT_SHIFT | (state) << PP_STATE_SHIFT)

+#define XGMI_MODE_PSTATE_D3 0
+#define XGMI_MODE_PSTATE_D0 1
+
 struct seq_file;
 enum amd_pp_clock_type;
 struct amd_pp_simple_clock_info;
@@ -318,6 +321,7 @@ struct amd_pm_funcs {
         int (*set_ppfeature_status)(void *handle, uint64_t ppfeature_masks);
         int (*asic_reset_mode_2)(void *handle);
         int (*set_df_cstate)(void *handle, enum pp_df_cstate state);
+       int (*set_xgmi_pstate)(void *handle, uint32_t pstate);
 };

 #endif
diff --git a/drivers/gpu/drm/amd/powerplay/amd_powerplay.c b/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
index f4ff15378e61..031447675203 100644
--- a/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
+++ b/drivers/gpu/drm/amd/powerplay/amd_powerplay.c
@@ -1566,6 +1566,23 @@ static int pp_set_df_cstate(void *handle, enum pp_df_cstate state)
         return 0;
 }

+static int pp_set_xgmi_pstate(void *handle, uint32_t pstate)
+{
+       struct pp_hwmgr *hwmgr = handle;
+
+       if (!hwmgr)
+               return -EINVAL;
+
+       if (!hwmgr->pm_en || !hwmgr->hwmgr_func->set_xgmi_pstate)
+               return 0;
+
+       mutex_lock(&hwmgr->smu_lock);
+       hwmgr->hwmgr_func->set_xgmi_pstate(hwmgr, pstate);
+       mutex_unlock(&hwmgr->smu_lock);
+
+       return 0;
+}
+
 static const struct amd_pm_funcs pp_dpm_funcs = {
         .load_firmware = pp_dpm_load_fw,
         .wait_for_fw_loading_complete = pp_dpm_fw_loading_complete,
@@ -1625,4 +1642,5 @@ static const struct amd_pm_funcs pp_dpm_funcs = {
         .asic_reset_mode_2 = pp_asic_reset_mode_2,
         .smu_i2c_bus_access = pp_smu_i2c_bus_access,
         .set_df_cstate = pp_set_df_cstate,
+       .set_xgmi_pstate = pp_set_xgmi_pstate,
 };
diff --git a/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c b/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
index 9295bd90b792..5bcf0d684151 100644
--- a/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
+++ b/drivers/gpu/drm/amd/powerplay/hwmgr/vega20_hwmgr.c
@@ -4176,6 +4176,20 @@ static int vega20_set_df_cstate(struct pp_hwmgr *hwmgr,
         return ret;
 }

+static int vega20_set_xgmi_pstate(struct pp_hwmgr *hwmgr,
+                                 uint32_t pstate)
+{
+       int ret;
+
+       ret = smum_send_msg_to_smc_with_parameter(hwmgr,
+                                                 PPSMC_MSG_SetXgmiMode,
+                                                 pstate ? XGMI_MODE_PSTATE_D0 : XGMI_MODE_PSTATE_D3);
+       if (ret)
+               pr_err("SetXgmiPstate failed!\n");
+
+       return ret;
+}
+
 static const struct pp_hwmgr_func vega20_hwmgr_funcs = {
         /* init/fini related */
         .backend_init = vega20_hwmgr_backend_init,
@@ -4245,6 +4259,7 @@ static const struct pp_hwmgr_func vega20_hwmgr_funcs = {
         .set_mp1_state = vega20_set_mp1_state,
         .smu_i2c_bus_access = vega20_smu_i2c_bus_access,
         .set_df_cstate = vega20_set_df_cstate,
+       .set_xgmi_pstate = vega20_set_xgmi_pstate,
 };

 int vega20_hwmgr_init(struct pp_hwmgr *hwmgr)
diff --git a/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h b/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
index bd8c922dfd3e..40403bc76f1b 100644
--- a/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
+++ b/drivers/gpu/drm/amd/powerplay/inc/hwmgr.h
@@ -356,6 +356,7 @@ struct pp_hwmgr_func {
         int (*asic_reset)(struct pp_hwmgr *hwmgr, enum SMU_ASIC_RESET_MODE mode);
         int (*smu_i2c_bus_access)(struct pp_hwmgr *hwmgr, bool aquire);
         int (*set_df_cstate)(struct pp_hwmgr *hwmgr, enum pp_df_cstate state);
+       int (*set_xgmi_pstate)(struct pp_hwmgr *hwmgr, uint32_t pstate);
 };

 struct pp_table_func {
diff --git a/drivers/gpu/drm/amd/powerplay/smu_v11_0.c b/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
index 7e882999abad..5877857760be 100644
--- a/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
+++ b/drivers/gpu/drm/amd/powerplay/smu_v11_0.c
@@ -1463,16 +1463,13 @@ int smu_v11_0_set_fan_speed_rpm(struct smu_context *smu,
         return ret;
 }

-#define XGMI_STATE_D0 1
-#define XGMI_STATE_D3 0
-
 int smu_v11_0_set_xgmi_pstate(struct smu_context *smu,
                                      uint32_t pstate)
 {
         int ret = 0;
         ret = smu_send_smc_msg_with_param(smu,
                                           SMU_MSG_SetXgmiMode,
-                                         pstate ? XGMI_STATE_D0 : XGMI_STATE_D3);
+                                         pstate ? XGMI_MODE_PSTATE_D0 : XGMI_MODE_PSTATE_D3);
         return ret;
 }

--
2.23.0


[-- Attachment #1.2: Type: text/html, Size: 13411 bytes --]

[-- Attachment #2: Type: text/plain, Size: 153 bytes --]

_______________________________________________
amd-gfx mailing list
amd-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

^ permalink raw reply related	[flat|nested] 8+ messages in thread
end of thread, other threads:[~2019-10-31 14:23 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-31  7:22 [PATCH 1/3] drm/amdgpu: change pstate only after all XGMI device initialized Quan, Evan
2019-10-31  7:22 ` Quan, Evan
     [not found] ` <20191031072205.27537-1-evan.quan-5C7GfCeVMHo@public.gmane.org>
2019-10-31  7:22   ` [PATCH 2/3] drm/amd/powerplay: update is_sw_smu_xgmi check Quan, Evan
2019-10-31  7:22     ` Quan, Evan
2019-10-31  7:22   ` [PATCH 3/3] drm/amd/powerplay: support xgmi pstate setting on powerplay routine V2 Quan, Evan
2019-10-31  7:22     ` Quan, Evan
     [not found]     ` <20191031072205.27537-3-evan.quan-5C7GfCeVMHo@public.gmane.org>
2019-10-31 14:23       ` Deucher, Alexander
2019-10-31 14:23         ` Deucher, Alexander

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.