* [PATCH] drm/amdgpu: Default disable GDS for compute VMIDs @ 2019-07-17 17:11 Greathouse, Joseph [not found] ` <20190717171112.4962-1-Joseph.Greathouse-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Greathouse, Joseph @ 2019-07-17 17:11 UTC (permalink / raw) To: amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW; +Cc: Greathouse, Joseph The GDS and GWS blocks default to allowing all VMIDs to access all entries. Graphics VMIDs can handle setting these limits when the driver launches work. However, compute workloads under HWS control don't go through the kernel driver. Instead, HWS firmware should set these limits when a process is put into a VMID slot. Disable access to these devices by default by turning off all mask bits (for OA) and setting BASE=SIZE=0 (for GDS and GWS) for all compute VMIDs. If a process wants to use these resources, they can request this from the HWS firmware (when such capabilities are enabled). HWS will then handle setting the base and limit for the process when it is assigned to a VMID. This will also prevent user kernels from getting 'stuck' in GWS by accident if they write GWS-using code but HWS firmware is not set up to handle GWS reset. Until HWS is enabled to handle GWS properly, all GWS accesses will MEM_VIOL fault the kernel. Change-Id: I2a709fdcb2468511754f2e5eae75546751c0e6f0 Signed-off-by: Joseph Greathouse <Joseph.Greathouse@amd.com> --- drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 6 ++++++ drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c | 6 ++++++ drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c | 6 ++++++ drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 6 ++++++ 4 files changed, 24 insertions(+) diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c index 618291df659b..32da5a91abfb 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c @@ -1513,6 +1513,12 @@ static void gfx_v10_0_init_compute_vmid(struct amdgpu_device *adev) /* CP and shaders */ WREG32_SOC15(GC, 0, mmSH_MEM_CONFIG, DEFAULT_SH_MEM_CONFIG); WREG32_SOC15(GC, 0, mmSH_MEM_BASES, sh_mem_bases); + /* Initialize all compute VMIDs to have no GDS, GWS, or OA + acccess. These should be enabled by FW for target VMID. */ + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_BASE, 2 * i, 0x00000000); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_SIZE, 2 * i, 0x00000000); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_GWS_VMID0, i, 0x00000000); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_OA_VMID0, i, 0x00000000); } nv_grbm_select(adev, 0, 0, 0, 0); mutex_unlock(&adev->srbm_mutex); diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c index e1e2a44ee13c..89ea0d799c12 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c @@ -1874,6 +1874,12 @@ static void gfx_v7_0_init_compute_vmid(struct amdgpu_device *adev) WREG32(mmSH_MEM_APE1_BASE, 1); WREG32(mmSH_MEM_APE1_LIMIT, 0); WREG32(mmSH_MEM_BASES, sh_mem_bases); + /* Initialize all compute VMIDs to have no GDS, GWS, or OA + acccess. These should be enabled by FW for target VMID. */ + WREG32(amdgpu_gds_reg_offset[i].mem_base, 0); + WREG32(amdgpu_gds_reg_offset[i].mem_size, 0); + WREG32(amdgpu_gds_reg_offset[i].gws, 0); + WREG32(amdgpu_gds_reg_offset[i].oa, 0); } cik_srbm_select(adev, 0, 0, 0, 0); mutex_unlock(&adev->srbm_mutex); diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c index 8c590a554675..53d9a223f8c1 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c @@ -3699,6 +3699,12 @@ static void gfx_v8_0_init_compute_vmid(struct amdgpu_device *adev) WREG32(mmSH_MEM_APE1_BASE, 1); WREG32(mmSH_MEM_APE1_LIMIT, 0); WREG32(mmSH_MEM_BASES, sh_mem_bases); + /* Initialize all compute VMIDs to have no GDS, GWS, or OA + acccess. These should be enabled by FW for target VMID. */ + WREG32(amdgpu_gds_reg_offset[i].mem_base, 0); + WREG32(amdgpu_gds_reg_offset[i].mem_size, 0); + WREG32(amdgpu_gds_reg_offset[i].gws, 0); + WREG32(amdgpu_gds_reg_offset[i].oa, 0); } vi_srbm_select(adev, 0, 0, 0, 0); mutex_unlock(&adev->srbm_mutex); diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c index 5af60e1c735a..b9ed03a8a561 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c @@ -2022,6 +2022,12 @@ static void gfx_v9_0_init_compute_vmid(struct amdgpu_device *adev) /* CP and shaders */ WREG32_SOC15_RLC(GC, 0, mmSH_MEM_CONFIG, sh_mem_config); WREG32_SOC15_RLC(GC, 0, mmSH_MEM_BASES, sh_mem_bases); + /* Initialize all compute VMIDs to have no GDS, GWS, or OA + acccess. These should be enabled by FW for target VMID. */ + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_BASE, 2 * i, 0x00000000); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_SIZE, 2 * i, 0x00000000); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_GWS_VMID0, i, 0x00000000); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_OA_VMID0, i, 0x00000000); } soc15_grbm_select(adev, 0, 0, 0, 0); mutex_unlock(&adev->srbm_mutex); -- 2.19.1 _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply related [flat|nested] 6+ messages in thread
[parent not found: <20190717171112.4962-1-Joseph.Greathouse-5C7GfCeVMHo@public.gmane.org>]
* Re: [PATCH] drm/amdgpu: Default disable GDS for compute VMIDs [not found] ` <20190717171112.4962-1-Joseph.Greathouse-5C7GfCeVMHo@public.gmane.org> @ 2019-07-17 17:23 ` Kuehling, Felix [not found] ` <28783441-080b-1696-b4ea-f6ec24901fb1-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Kuehling, Felix @ 2019-07-17 17:23 UTC (permalink / raw) To: Greathouse, Joseph, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW On 2019-07-17 13:11, Greathouse, Joseph wrote: > The GDS and GWS blocks default to allowing all VMIDs to > access all entries. Graphics VMIDs can handle setting > these limits when the driver launches work. However, > compute workloads under HWS control don't go through the > kernel driver. Instead, HWS firmware should set these > limits when a process is put into a VMID slot. > > Disable access to these devices by default by turning off > all mask bits (for OA) and setting BASE=SIZE=0 (for GDS > and GWS) for all compute VMIDs. If a process wants to use > these resources, they can request this from the HWS > firmware (when such capabilities are enabled). HWS will > then handle setting the base and limit for the process when > it is assigned to a VMID. > > This will also prevent user kernels from getting 'stuck' in > GWS by accident if they write GWS-using code but HWS > firmware is not set up to handle GWS reset. Until HWS is > enabled to handle GWS properly, all GWS accesses will > MEM_VIOL fault the kernel. > > Change-Id: I2a709fdcb2468511754f2e5eae75546751c0e6f0 > Signed-off-by: Joseph Greathouse <Joseph.Greathouse@amd.com> > --- > drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 6 ++++++ > drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c | 6 ++++++ > drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c | 6 ++++++ > drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 6 ++++++ > 4 files changed, 24 insertions(+) > > diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c > index 618291df659b..32da5a91abfb 100644 > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c > @@ -1513,6 +1513,12 @@ static void gfx_v10_0_init_compute_vmid(struct amdgpu_device *adev) > /* CP and shaders */ > WREG32_SOC15(GC, 0, mmSH_MEM_CONFIG, DEFAULT_SH_MEM_CONFIG); > WREG32_SOC15(GC, 0, mmSH_MEM_BASES, sh_mem_bases); > + /* Initialize all compute VMIDs to have no GDS, GWS, or OA > + acccess. These should be enabled by FW for target VMID. */ > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_BASE, 2 * i, 0x00000000); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_SIZE, 2 * i, 0x00000000); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_GWS_VMID0, i, 0x00000000); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_OA_VMID0, i, 0x00000000); I'd prefer these register initializations in a separate loop outside the srbm_mutex, because they don't depend on the srbm/grbm_select bits. Same for the other GFX IP versions. Regards, Felix > } > nv_grbm_select(adev, 0, 0, 0, 0); > mutex_unlock(&adev->srbm_mutex); > diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c > index e1e2a44ee13c..89ea0d799c12 100644 > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c > @@ -1874,6 +1874,12 @@ static void gfx_v7_0_init_compute_vmid(struct amdgpu_device *adev) > WREG32(mmSH_MEM_APE1_BASE, 1); > WREG32(mmSH_MEM_APE1_LIMIT, 0); > WREG32(mmSH_MEM_BASES, sh_mem_bases); > + /* Initialize all compute VMIDs to have no GDS, GWS, or OA > + acccess. These should be enabled by FW for target VMID. */ > + WREG32(amdgpu_gds_reg_offset[i].mem_base, 0); > + WREG32(amdgpu_gds_reg_offset[i].mem_size, 0); > + WREG32(amdgpu_gds_reg_offset[i].gws, 0); > + WREG32(amdgpu_gds_reg_offset[i].oa, 0); > } > cik_srbm_select(adev, 0, 0, 0, 0); > mutex_unlock(&adev->srbm_mutex); > diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c > index 8c590a554675..53d9a223f8c1 100644 > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c > @@ -3699,6 +3699,12 @@ static void gfx_v8_0_init_compute_vmid(struct amdgpu_device *adev) > WREG32(mmSH_MEM_APE1_BASE, 1); > WREG32(mmSH_MEM_APE1_LIMIT, 0); > WREG32(mmSH_MEM_BASES, sh_mem_bases); > + /* Initialize all compute VMIDs to have no GDS, GWS, or OA > + acccess. These should be enabled by FW for target VMID. */ > + WREG32(amdgpu_gds_reg_offset[i].mem_base, 0); > + WREG32(amdgpu_gds_reg_offset[i].mem_size, 0); > + WREG32(amdgpu_gds_reg_offset[i].gws, 0); > + WREG32(amdgpu_gds_reg_offset[i].oa, 0); > } > vi_srbm_select(adev, 0, 0, 0, 0); > mutex_unlock(&adev->srbm_mutex); > diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c > index 5af60e1c735a..b9ed03a8a561 100644 > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c > @@ -2022,6 +2022,12 @@ static void gfx_v9_0_init_compute_vmid(struct amdgpu_device *adev) > /* CP and shaders */ > WREG32_SOC15_RLC(GC, 0, mmSH_MEM_CONFIG, sh_mem_config); > WREG32_SOC15_RLC(GC, 0, mmSH_MEM_BASES, sh_mem_bases); > + /* Initialize all compute VMIDs to have no GDS, GWS, or OA > + acccess. These should be enabled by FW for target VMID. */ > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_BASE, 2 * i, 0x00000000); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_SIZE, 2 * i, 0x00000000); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_GWS_VMID0, i, 0x00000000); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_OA_VMID0, i, 0x00000000); > } > soc15_grbm_select(adev, 0, 0, 0, 0); > mutex_unlock(&adev->srbm_mutex); _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <28783441-080b-1696-b4ea-f6ec24901fb1-5C7GfCeVMHo@public.gmane.org>]
* [PATCH v2] drm/amdgpu: Default disable GDS for compute VMIDs [not found] ` <28783441-080b-1696-b4ea-f6ec24901fb1-5C7GfCeVMHo@public.gmane.org> @ 2019-07-17 18:23 ` Greathouse, Joseph [not found] ` <20190717182233.93031-1-Joseph.Greathouse-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Greathouse, Joseph @ 2019-07-17 18:23 UTC (permalink / raw) To: amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW; +Cc: Greathouse, Joseph The GDS and GWS blocks default to allowing all VMIDs to access all entries. Graphics VMIDs can handle setting these limits when the driver launches work. However, compute workloads under HWS control don't go through the kernel driver. Instead, HWS firmware should set these limits when a process is put into a VMID slot. Disable access to these devices by default by turning off all mask bits (for OA) and setting BASE=SIZE=0 (for GDS and GWS) for all compute VMIDs. If a process wants to use these resources, they can request this from the HWS firmware (when such capabilities are enabled). HWS will then handle setting the base and limit for the process when it is assigned to a VMID. This will also prevent user kernels from getting 'stuck' in GWS by accident if they write GWS-using code but HWS firmware is not set up to handle GWS reset. Until HWS is enabled to handle GWS properly, all GWS accesses will MEM_VIOL fault the kernel. v2: Move initialization outside of SRBM mutex Change-Id: I8edcea9d0b14d16a7444bcf9fbf9451aef8b707d Signed-off-by: Joseph Greathouse <Joseph.Greathouse@amd.com> --- drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 9 +++++++++ drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c | 9 +++++++++ drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c | 9 +++++++++ drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 9 +++++++++ 4 files changed, 36 insertions(+) diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c index 618291df659b..73dcb632a3ce 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c @@ -1516,6 +1516,15 @@ static void gfx_v10_0_init_compute_vmid(struct amdgpu_device *adev) } nv_grbm_select(adev, 0, 0, 0, 0); mutex_unlock(&adev->srbm_mutex); + + /* Initialize all compute VMIDs to have no GDS, GWS, or OA + acccess. These should be enabled by FW for target VMIDs. */ + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_BASE, 2 * i, 0); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_SIZE, 2 * i, 0); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_GWS_VMID0, i, 0); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_OA_VMID0, i, 0); + } } static void gfx_v10_0_tcp_harvest(struct amdgpu_device *adev) diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c index e1e2a44ee13c..3f98624772a4 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c @@ -1877,6 +1877,15 @@ static void gfx_v7_0_init_compute_vmid(struct amdgpu_device *adev) } cik_srbm_select(adev, 0, 0, 0, 0); mutex_unlock(&adev->srbm_mutex); + + /* Initialize all compute VMIDs to have no GDS, GWS, or OA + acccess. These should be enabled by FW for target VMIDs. */ + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { + WREG32(amdgpu_gds_reg_offset[i].mem_base, 0); + WREG32(amdgpu_gds_reg_offset[i].mem_size, 0); + WREG32(amdgpu_gds_reg_offset[i].gws, 0); + WREG32(amdgpu_gds_reg_offset[i].oa, 0); + } } static void gfx_v7_0_config_init(struct amdgpu_device *adev) diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c index 8c590a554675..e4028d54f8f7 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c @@ -3702,6 +3702,15 @@ static void gfx_v8_0_init_compute_vmid(struct amdgpu_device *adev) } vi_srbm_select(adev, 0, 0, 0, 0); mutex_unlock(&adev->srbm_mutex); + + /* Initialize all compute VMIDs to have no GDS, GWS, or OA + acccess. These should be enabled by FW for target VMIDs. */ + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { + WREG32(amdgpu_gds_reg_offset[i].mem_base, 0); + WREG32(amdgpu_gds_reg_offset[i].mem_size, 0); + WREG32(amdgpu_gds_reg_offset[i].gws, 0); + WREG32(amdgpu_gds_reg_offset[i].oa, 0); + } } static void gfx_v8_0_config_init(struct amdgpu_device *adev) diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c index 5af60e1c735a..259a35395fec 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c @@ -2025,6 +2025,15 @@ static void gfx_v9_0_init_compute_vmid(struct amdgpu_device *adev) } soc15_grbm_select(adev, 0, 0, 0, 0); mutex_unlock(&adev->srbm_mutex); + + /* Initialize all compute VMIDs to have no GDS, GWS, or OA + acccess. These should be enabled by FW for target VMIDs. */ + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_BASE, 2 * i, 0); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_SIZE, 2 * i, 0); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_GWS_VMID0, i, 0); + WREG32_SOC15_OFFSET(GC, 0, mmGDS_OA_VMID0, i, 0); + } } static void gfx_v9_0_constants_init(struct amdgpu_device *adev) -- 2.19.1 _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply related [flat|nested] 6+ messages in thread
[parent not found: <20190717182233.93031-1-Joseph.Greathouse-5C7GfCeVMHo@public.gmane.org>]
* Re: [PATCH v2] drm/amdgpu: Default disable GDS for compute VMIDs [not found] ` <20190717182233.93031-1-Joseph.Greathouse-5C7GfCeVMHo@public.gmane.org> @ 2019-07-17 20:09 ` Kuehling, Felix [not found] ` <268d3673-da7f-8e75-6131-3de9291d77d4-5C7GfCeVMHo@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Kuehling, Felix @ 2019-07-17 20:09 UTC (permalink / raw) To: Greathouse, Joseph, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW On 2019-07-17 14:23, Greathouse, Joseph wrote: > The GDS and GWS blocks default to allowing all VMIDs to > access all entries. Graphics VMIDs can handle setting > these limits when the driver launches work. However, > compute workloads under HWS control don't go through the > kernel driver. Instead, HWS firmware should set these > limits when a process is put into a VMID slot. > > Disable access to these devices by default by turning off > all mask bits (for OA) and setting BASE=SIZE=0 (for GDS > and GWS) for all compute VMIDs. If a process wants to use > these resources, they can request this from the HWS > firmware (when such capabilities are enabled). HWS will > then handle setting the base and limit for the process when > it is assigned to a VMID. > > This will also prevent user kernels from getting 'stuck' in > GWS by accident if they write GWS-using code but HWS > firmware is not set up to handle GWS reset. Until HWS is > enabled to handle GWS properly, all GWS accesses will > MEM_VIOL fault the kernel. > > v2: Move initialization outside of SRBM mutex > > Change-Id: I8edcea9d0b14d16a7444bcf9fbf9451aef8b707d > Signed-off-by: Joseph Greathouse <Joseph.Greathouse@amd.com> Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com> > --- > drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 9 +++++++++ > drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c | 9 +++++++++ > drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c | 9 +++++++++ > drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 9 +++++++++ > 4 files changed, 36 insertions(+) > > diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c > index 618291df659b..73dcb632a3ce 100644 > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c > @@ -1516,6 +1516,15 @@ static void gfx_v10_0_init_compute_vmid(struct amdgpu_device *adev) > } > nv_grbm_select(adev, 0, 0, 0, 0); > mutex_unlock(&adev->srbm_mutex); > + > + /* Initialize all compute VMIDs to have no GDS, GWS, or OA > + acccess. These should be enabled by FW for target VMIDs. */ > + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_BASE, 2 * i, 0); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_SIZE, 2 * i, 0); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_GWS_VMID0, i, 0); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_OA_VMID0, i, 0); > + } > } > > static void gfx_v10_0_tcp_harvest(struct amdgpu_device *adev) > diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c > index e1e2a44ee13c..3f98624772a4 100644 > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c > @@ -1877,6 +1877,15 @@ static void gfx_v7_0_init_compute_vmid(struct amdgpu_device *adev) > } > cik_srbm_select(adev, 0, 0, 0, 0); > mutex_unlock(&adev->srbm_mutex); > + > + /* Initialize all compute VMIDs to have no GDS, GWS, or OA > + acccess. These should be enabled by FW for target VMIDs. */ > + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { > + WREG32(amdgpu_gds_reg_offset[i].mem_base, 0); > + WREG32(amdgpu_gds_reg_offset[i].mem_size, 0); > + WREG32(amdgpu_gds_reg_offset[i].gws, 0); > + WREG32(amdgpu_gds_reg_offset[i].oa, 0); > + } > } > > static void gfx_v7_0_config_init(struct amdgpu_device *adev) > diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c > index 8c590a554675..e4028d54f8f7 100644 > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c > @@ -3702,6 +3702,15 @@ static void gfx_v8_0_init_compute_vmid(struct amdgpu_device *adev) > } > vi_srbm_select(adev, 0, 0, 0, 0); > mutex_unlock(&adev->srbm_mutex); > + > + /* Initialize all compute VMIDs to have no GDS, GWS, or OA > + acccess. These should be enabled by FW for target VMIDs. */ > + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { > + WREG32(amdgpu_gds_reg_offset[i].mem_base, 0); > + WREG32(amdgpu_gds_reg_offset[i].mem_size, 0); > + WREG32(amdgpu_gds_reg_offset[i].gws, 0); > + WREG32(amdgpu_gds_reg_offset[i].oa, 0); > + } > } > > static void gfx_v8_0_config_init(struct amdgpu_device *adev) > diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c > index 5af60e1c735a..259a35395fec 100644 > --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c > +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c > @@ -2025,6 +2025,15 @@ static void gfx_v9_0_init_compute_vmid(struct amdgpu_device *adev) > } > soc15_grbm_select(adev, 0, 0, 0, 0); > mutex_unlock(&adev->srbm_mutex); > + > + /* Initialize all compute VMIDs to have no GDS, GWS, or OA > + acccess. These should be enabled by FW for target VMIDs. */ > + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_BASE, 2 * i, 0); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_SIZE, 2 * i, 0); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_GWS_VMID0, i, 0); > + WREG32_SOC15_OFFSET(GC, 0, mmGDS_OA_VMID0, i, 0); > + } > } > > static void gfx_v9_0_constants_init(struct amdgpu_device *adev) _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <268d3673-da7f-8e75-6131-3de9291d77d4-5C7GfCeVMHo@public.gmane.org>]
* Re: [PATCH v2] drm/amdgpu: Default disable GDS for compute VMIDs [not found] ` <268d3673-da7f-8e75-6131-3de9291d77d4-5C7GfCeVMHo@public.gmane.org> @ 2019-07-18 8:14 ` Christian König [not found] ` <42627d7f-2431-6b17-72aa-e448b4937c53-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Christian König @ 2019-07-18 8:14 UTC (permalink / raw) To: Kuehling, Felix, Greathouse, Joseph, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW Am 17.07.19 um 22:09 schrieb Kuehling, Felix: > On 2019-07-17 14:23, Greathouse, Joseph wrote: >> The GDS and GWS blocks default to allowing all VMIDs to >> access all entries. Graphics VMIDs can handle setting >> these limits when the driver launches work. However, >> compute workloads under HWS control don't go through the >> kernel driver. Instead, HWS firmware should set these >> limits when a process is put into a VMID slot. >> >> Disable access to these devices by default by turning off >> all mask bits (for OA) and setting BASE=SIZE=0 (for GDS >> and GWS) for all compute VMIDs. If a process wants to use >> these resources, they can request this from the HWS >> firmware (when such capabilities are enabled). HWS will >> then handle setting the base and limit for the process when >> it is assigned to a VMID. >> >> This will also prevent user kernels from getting 'stuck' in >> GWS by accident if they write GWS-using code but HWS >> firmware is not set up to handle GWS reset. Until HWS is >> enabled to handle GWS properly, all GWS accesses will >> MEM_VIOL fault the kernel. >> >> v2: Move initialization outside of SRBM mutex >> >> Change-Id: I8edcea9d0b14d16a7444bcf9fbf9451aef8b707d >> Signed-off-by: Joseph Greathouse <Joseph.Greathouse@amd.com> > Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com> Might be a good idea to do this for all VMIDs during initialization and not just for the ones used for compute. But anyway patch is Reviewed-by: Christian König <christian.koenig@amd.com>. > > >> --- >> drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 9 +++++++++ >> drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c | 9 +++++++++ >> drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c | 9 +++++++++ >> drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 9 +++++++++ >> 4 files changed, 36 insertions(+) >> >> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c >> index 618291df659b..73dcb632a3ce 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c >> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c >> @@ -1516,6 +1516,15 @@ static void gfx_v10_0_init_compute_vmid(struct amdgpu_device *adev) >> } >> nv_grbm_select(adev, 0, 0, 0, 0); >> mutex_unlock(&adev->srbm_mutex); >> + >> + /* Initialize all compute VMIDs to have no GDS, GWS, or OA >> + acccess. These should be enabled by FW for target VMIDs. */ >> + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { >> + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_BASE, 2 * i, 0); >> + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_SIZE, 2 * i, 0); >> + WREG32_SOC15_OFFSET(GC, 0, mmGDS_GWS_VMID0, i, 0); >> + WREG32_SOC15_OFFSET(GC, 0, mmGDS_OA_VMID0, i, 0); >> + } >> } >> >> static void gfx_v10_0_tcp_harvest(struct amdgpu_device *adev) >> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c >> index e1e2a44ee13c..3f98624772a4 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c >> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v7_0.c >> @@ -1877,6 +1877,15 @@ static void gfx_v7_0_init_compute_vmid(struct amdgpu_device *adev) >> } >> cik_srbm_select(adev, 0, 0, 0, 0); >> mutex_unlock(&adev->srbm_mutex); >> + >> + /* Initialize all compute VMIDs to have no GDS, GWS, or OA >> + acccess. These should be enabled by FW for target VMIDs. */ >> + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { >> + WREG32(amdgpu_gds_reg_offset[i].mem_base, 0); >> + WREG32(amdgpu_gds_reg_offset[i].mem_size, 0); >> + WREG32(amdgpu_gds_reg_offset[i].gws, 0); >> + WREG32(amdgpu_gds_reg_offset[i].oa, 0); >> + } >> } >> >> static void gfx_v7_0_config_init(struct amdgpu_device *adev) >> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c >> index 8c590a554675..e4028d54f8f7 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c >> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c >> @@ -3702,6 +3702,15 @@ static void gfx_v8_0_init_compute_vmid(struct amdgpu_device *adev) >> } >> vi_srbm_select(adev, 0, 0, 0, 0); >> mutex_unlock(&adev->srbm_mutex); >> + >> + /* Initialize all compute VMIDs to have no GDS, GWS, or OA >> + acccess. These should be enabled by FW for target VMIDs. */ >> + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { >> + WREG32(amdgpu_gds_reg_offset[i].mem_base, 0); >> + WREG32(amdgpu_gds_reg_offset[i].mem_size, 0); >> + WREG32(amdgpu_gds_reg_offset[i].gws, 0); >> + WREG32(amdgpu_gds_reg_offset[i].oa, 0); >> + } >> } >> >> static void gfx_v8_0_config_init(struct amdgpu_device *adev) >> diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c >> index 5af60e1c735a..259a35395fec 100644 >> --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c >> +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c >> @@ -2025,6 +2025,15 @@ static void gfx_v9_0_init_compute_vmid(struct amdgpu_device *adev) >> } >> soc15_grbm_select(adev, 0, 0, 0, 0); >> mutex_unlock(&adev->srbm_mutex); >> + >> + /* Initialize all compute VMIDs to have no GDS, GWS, or OA >> + acccess. These should be enabled by FW for target VMIDs. */ >> + for (i = FIRST_COMPUTE_VMID; i < LAST_COMPUTE_VMID; i++) { >> + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_BASE, 2 * i, 0); >> + WREG32_SOC15_OFFSET(GC, 0, mmGDS_VMID0_SIZE, 2 * i, 0); >> + WREG32_SOC15_OFFSET(GC, 0, mmGDS_GWS_VMID0, i, 0); >> + WREG32_SOC15_OFFSET(GC, 0, mmGDS_OA_VMID0, i, 0); >> + } >> } >> >> static void gfx_v9_0_constants_init(struct amdgpu_device *adev) > _______________________________________________ > amd-gfx mailing list > amd-gfx@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/amd-gfx _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <42627d7f-2431-6b17-72aa-e448b4937c53-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]
* RE: [PATCH v2] drm/amdgpu: Default disable GDS for compute VMIDs [not found] ` <42627d7f-2431-6b17-72aa-e448b4937c53-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> @ 2019-07-18 20:46 ` Greathouse, Joseph 0 siblings, 0 replies; 6+ messages in thread From: Greathouse, Joseph @ 2019-07-18 20:46 UTC (permalink / raw) To: Koenig, Christian, Kuehling, Felix, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW > -----Original Message----- > From: Christian König <ckoenig.leichtzumerken@gmail.com> > Sent: Thursday, July 18, 2019 3:14 AM > To: Kuehling, Felix <Felix.Kuehling@amd.com>; Greathouse, Joseph > <Joseph.Greathouse@amd.com>; amd-gfx@lists.freedesktop.org > Subject: Re: [PATCH v2] drm/amdgpu: Default disable GDS for compute > VMIDs > > Am 17.07.19 um 22:09 schrieb Kuehling, Felix: > > On 2019-07-17 14:23, Greathouse, Joseph wrote: > >> The GDS and GWS blocks default to allowing all VMIDs to > >> access all entries. Graphics VMIDs can handle setting > >> these limits when the driver launches work. However, > >> compute workloads under HWS control don't go through the > >> kernel driver. Instead, HWS firmware should set these > >> limits when a process is put into a VMID slot. > >> > >> Disable access to these devices by default by turning off > >> all mask bits (for OA) and setting BASE=SIZE=0 (for GDS > >> and GWS) for all compute VMIDs. If a process wants to use > >> these resources, they can request this from the HWS > >> firmware (when such capabilities are enabled). HWS will > >> then handle setting the base and limit for the process when > >> it is assigned to a VMID. > >> > >> This will also prevent user kernels from getting 'stuck' in > >> GWS by accident if they write GWS-using code but HWS > >> firmware is not set up to handle GWS reset. Until HWS is > >> enabled to handle GWS properly, all GWS accesses will > >> MEM_VIOL fault the kernel. > >> > >> v2: Move initialization outside of SRBM mutex > >> > >> Change-Id: I8edcea9d0b14d16a7444bcf9fbf9451aef8b707d > >> Signed-off-by: Joseph Greathouse <Joseph.Greathouse@amd.com> > > Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com> > > Might be a good idea to do this for all VMIDs during initialization and > not just for the ones used for compute. > > But anyway patch is Reviewed-by: Christian König > <christian.koenig@amd.com>. Hmm, good point. It looks like graphics jobs will eventually call through to emit_gds_switch() to set these when launching a job, but it may be worthwhile to set these to zero as a default. I didn't want to step on any toes on the graphics side without checking first. Do you have opinions on the most reasonable location to do this? early_init(), late_init()? The various gfx_v*_set_gds_init() might be a good place -- a quick test of setting all 16 VMIDs in gfx_v9_0_set_gds_init() appears to work fine on my Vega 20. -Joe _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2019-07-18 20:46 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-07-17 17:11 [PATCH] drm/amdgpu: Default disable GDS for compute VMIDs Greathouse, Joseph [not found] ` <20190717171112.4962-1-Joseph.Greathouse-5C7GfCeVMHo@public.gmane.org> 2019-07-17 17:23 ` Kuehling, Felix [not found] ` <28783441-080b-1696-b4ea-f6ec24901fb1-5C7GfCeVMHo@public.gmane.org> 2019-07-17 18:23 ` [PATCH v2] " Greathouse, Joseph [not found] ` <20190717182233.93031-1-Joseph.Greathouse-5C7GfCeVMHo@public.gmane.org> 2019-07-17 20:09 ` Kuehling, Felix [not found] ` <268d3673-da7f-8e75-6131-3de9291d77d4-5C7GfCeVMHo@public.gmane.org> 2019-07-18 8:14 ` Christian König [not found] ` <42627d7f-2431-6b17-72aa-e448b4937c53-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> 2019-07-18 20:46 ` Greathouse, Joseph
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.